Introduction
Consider a fleet of autonomous vehicles serving delivery requests in a city. At each decision time, the operator must simultaneously decide which vehicle should serve which request and how each vehicle should navigate to its assigned destination. These decisions are tightly coupled. The best assignment depends on how long each vehicle will actually take to reach each target, while routing itself depends on the assignment. Assignments based only on nominal shortest paths can therefore be poor when travel times depend on traffic lights, congestion, and random delays. Yet solving this full joint assignment-routing problem over the fleet quickly becomes computationally prohibitive. Beyond delivery fleets, the same structure arises across a wide range of cooperative systems: inspection drones achieving coverage patterns [1 ] , warehouse robots repositioning to incoming orders [2 ] , emergency response units relocating to optimal positions [3 ] , ride-hailing platforms balancing supply and demand [4 ] , and electric-vehicle fleets sharing charging infrastructure [5 ] . In all these settings, the goal is to optimally coordinate a fleet of decoupled agents so that their collective terminal state matches or approximates a target configuration, such as uniform coverage, a set of delivery locations, or a demand-weighted positioning pattern. These problems are typically solved in a receding-horizon fashion to account for changing conditions and request patterns and, thus, require modular approaches that generalize to different fleet sizes or target distributions.
Fleet assignment and routing are classically addressed via combinatorial solvers [6 , 7 ] and, more recently, learned routing heuristics [8 ] . Since the joint assignment-routing problem is combinatorial in the fleet size, these methods decouple the two steps using nominal travel costs, such as shortest-path distances, to solve the assignment problem. These heuristics, however, ignore the stochasticity of the costs and may favor targets that “look close” but are expensive to reach in expectation. A natural learning-based alternative is multi-agent reinforcement learning (MARL) [9 , 10 , 11 , 12 ] , often with centralized training and decentralized execution. Policy-gradient methods such as MAPPO [13 ] and MADDPG [14 ] , and value-decomposition methods such as VDN [15 ] , QMIX [16 ] , and QPLEX [17 ] are effective in many settings [18 , 19 ] , but learn over a joint state-action space whose size grows exponentially with the number of agents [20 ] , limiting scalability and generalization to unseen fleet sizes. Value-decomposition methods gain tractability by imposing architectural assumptions on the joint value function, such as additivity or monotonicity, which are provably too restrictive even for simple coordination tasks [21 , 16 ] . Mean-field reinforcement learning [22 , 23 ] instead replaces pairwise interactions with an aggregate representation of the population. Although practical in large systems [4 , 24 ] , its approximations can lose fidelity when modeling finite populations [25 ] . Closer to our motivation, some methods learn latent subtasks and dynamically assign them among agents [26 ] , since in general cooperative MARL no such decomposition is available a priori. We instead study a structured subclass, with decoupled dynamics and a terminal optimal transport cost, in which the assignment-routing separation can be exploited directly.
In this work, we seek a principled separation of population-level assignment and single-agent control (Figure Figure 1 ). Inspired by the “deterministic” separation principle for multi-agent control in [27 ] , we formulate the cooperative MARL problem as an optimal control problem over the space of probability measures where terminal deviation from a desired target measure is quantified via an optimal transport cost. By construction, this representation is permutation invariant and captures both finite populations (as empirical measures) and a continuum of agents (as continuous measures). In the deterministic setting, Terpin et al. [27 ] show that the population cost-to-go is exactly an optimal transport cost whose transportation cost is the single-agent cost-to-go, thus separating assignment from single-agent control. This equality does not extend directly to the stochastic setting, where the population measure is random and future assignments can adapt to realized noise. Nevertheless, we show that a carefully designed separation principle, where noise is “averaged out”, continues to upper bound the population cost-to-go. This analysis is non-trivial: the randomness of the population measure requires handling probability measures over probability measures, a modeling challenge in itself, compounded by delicate measurability issues. Finally, Terpin et al. [27 ] provide a purely theoretical analysis, whereas we leverage our separation principle to design scalable algorithms for MARL.
Figure 1. Overview of SALT. (1) Cooperative MARL learns a joint fleet policy and scales poorly with N N N . (2) Formulating the problem over probability measures replaces agent labels with an empirical population μ t \mu_t μ t and target distribution ν \nu ν , but remains intractable. (3) The separation principle decouples assignment from routing, upper bounding the population cost-to-go by an optimal transport assignment with cost j ^ ( s , z ) \hat{j}(s,z) j ^ ( s , z ) . SALT assigns targets through this optimal transport problem and routes agents with a shared single-agent policy π \pi π , yielding scalable and sample-efficient learning.
Contributions. Our contributions are as follows:
We cast the cooperative multi-agent control problem with homogeneous agents, decoupled stochastic dynamics, and a terminal optimal transport formation cost as an optimal control problem over probability measures in the spirit of the work of Terpin et al. [27 ] . This removes agent labels, makes the formulation permutation invariant, and applies to both finite populations and a continuum of agents.
We prove a separation principle for this optimal control problem (Theorem 3.1, Part I). The population cost-to-go is upper bounded by an optimal transport problem whose transportation cost is the target-conditioned single-agent cost-to-go. This separates target allocation from routing, which can be solved independently of the fleet size. The upper bound becomes exact in the deterministic setting, recovering [27 Corollary 4.2] . We then bound the gap between the two sides of Theorem 3.1, Part I in terms of the noise of the single-agent dynamics (Theorem D.4 ), and the loss of optimality under weak population interactions (Theorem E.6 ). Importantly, our proof is constructive and yields an implementable policy.
This separation principle yields SALT, a family of MARL algorithms that learn a shared target-conditioned single-agent policy and coordinate agents via optimal transport. SALT is agnostic to the single-agent algorithm and generalizes to population sizes not seen during training.
We validate SALT on a time-varying southern Manhattan road network with 2087 intersections and 4255 directed roads. SALT trained with N = 20 N = 20 N = 20 vehicles transfers without retraining to fleets of 1–100 vehicles and reduces travel time by 15–21% relative to a nominal shortest-path heuristic. In controlled stochastic grid worlds, the SALT variants reduce the average travel time by 23–61% relative to the standard MARL baselines, consistently across different single-agent reinforcement learning (RL) algorithms. Additionally, SALT improves sample efficiency: with PPO [28 ] , it reaches the same reach-rate milestones as MAPPO with up to 480 × 480\times 480 × fewer environment interactions, and scales to N = 128 N = 128 N = 128 , while MAPPO fails at N = 8 N = 8 N = 8 within a 2 × 10 9 2 \times10^{9} 2 × 1 0 9 -interaction budget.
Notation. We work with Polish spaces, Borel functions, and Borel probability measures. For a Polish space X X X , P ( X ) \mathcal{P}(X) P ( X ) denotes the space of probability measures on X X X . We write δ x \delta_x δ x for the Dirac measure at x ∈ X x \in X x ∈ X . The projection from X × Y X \times Y X × Y onto X X X is denoted by proj X X × Y \operatorname{proj}_{X}^{X \times Y} proj X X × Y . For μ ∈ P ( X ) \mu\in\mathcal{P}(X) μ ∈ P ( X ) and F : X → Y F : X \to Y F : X → Y , the pushforward of μ \mu μ through F F F , F # μ ∈ P ( Y ) F_{\#}\mu\in\mathcal{P}(Y) F # μ ∈ P ( Y ) , is defined by F # μ ( B ) = μ ( F − 1 ( B ) ) F_{\#}\mu(B) = \mu(F^{-1}(B)) F # μ ( B ) = μ ( F − 1 ( B )) for all Borel sets B ⊆ Y B \subseteq Y B ⊆ Y . A coupling between the probability measures μ ∈ P ( X ) \mu\in\mathcal{P}(X) μ ∈ P ( X ) and ν ∈ P ( Y ) \nu\in\mathcal{P}(Y) ν ∈ P ( Y ) is a joint probability measure with marginals μ \mu μ and ν \nu ν ; the set of couplings between μ \mu μ and ν \nu ν is Γ ( μ , ν ) \Gamma(\mu,\nu) Γ ( μ , ν ) . For a nonnegative cost c : X × Y → R ≥ 0 c : X \times Y \to\mathbb{R}_{\ge0} c : X × Y → R ≥ 0 , the optimal transport cost between μ \mu μ and ν \nu ν is
K [ c ] ( μ , ν ) : = inf γ ∈ Γ ( μ , ν ) ∫ X × Y c ( x , y ) d γ ( x , y ) . \mathcal{K}[c](\mu,\nu) := \inf_{\gamma\in\Gamma(\mu,\nu)} \int_{X \times Y} c(x,y)\,\mathrm{d}\gamma(x,y). K [ c ] ( μ , ν ) := γ ∈ Γ ( μ , ν ) inf ∫ X × Y c ( x , y ) d γ ( x , y ) .
If k : X → P ( Y ) k : X \to\mathcal{P}(Y) k : X → P ( Y ) is a Markov kernel, μ ⊗ k \mu\otimes k μ ⊗ k denotes the induced joint law on X × Y X \times Y X × Y . In particular, if π ( d x , d y ) = μ ( d x ) k ( x ; d y ) \pi(\mathrm{d}x,\mathrm{d}y) = \mu(\mathrm{d}x)k(x;\mathrm{d}y) π ( d x , d y ) = μ ( d x ) k ( x ; d y ) is a disintegration, we write π = μ ⊗ k x \pi= \mu\otimes k^x π = μ ⊗ k x .
Problem setting
We consider cooperative MARL problems in which N N N homogeneous agents coordinate to reach a desired configuration over a finite horizon T T T , while incurring agent-level stage costs and a population-level terminal cost. Let S S S , A A A , W W W , and Z Z Z denote the state, action, noise, and target spaces. While Z = S Z = S Z = S for canonical target matching, we keep Z Z Z distinct to account for lower-dimensional targets or task-level descriptors, such as destinations when the state also includes velocity or battery level.
Agent-based formulation
Each agent i i i has state s t i ∈ S s_t^i \in S s t i ∈ S and chooses action a t i ∈ A a_t^i \in A a t i ∈ A . Its noisy dynamics are s t + 1 i = f t ( s t i , a t i , w t i ) s_{t+1}^i = f_t(s_t^i, a_t^i, w_t^i) s t + 1 i = f t ( s t i , a t i , w t i ) , where w t i ∼ ζ t ( ⋅ ∣ s t i , a t i ) w_t^i \sim\zeta_t(\cdot\mid s_t^i, a_t^i) w t i ∼ ζ t ( ⋅ ∣ s t i , a t i ) and ξ t ( ⋅ ∣ s t i , a t i ) ∈ P ( W ) \xi_t(\cdot\mid s_t^i, a_t^i) \in\mathcal{P}(W) ξ t ( ⋅ ∣ s t i , a t i ) ∈ P ( W ) gives the conditional law of the noise affecting an agent in state s s s taking action a a a . We consider three noise models, which differ in how noise is correlated across agents: individual noise , where noise is sampled independently across agents; local noise , where agents with the same state-action pair share the same noise realization; and global noise , where all agents share a common noise realization (see Appendix § 9 ). Let s t : = ( s t 1 , … , s t N ) s_t := (s_t^1,\ldots,s_t^N) s t := ( s t 1 , … , s t N ) and a t : = ( a t 1 , … , a t N ) a_t := (a_t^1,\ldots,a_t^N) a t := ( a t 1 , … , a t N ) denote the ordered states and actions of the agents. We take the control cost to be the sum of agent-level stage costs g t : S × A × W → R ‾ ≥ 0 g_t:S \times A \times\mathbb{W} \to\overline{\mathbb{R}}_{\ge0} g t : S × A × W → R ≥ 0 and a population-level terminal formation cost. Given a transportation cost g T : S × Z → R ‾ ≥ 0 g_T:S \times Z \to\overline{\mathbb{R}}_{\ge0} g T : S × Z → R ≥ 0 , the terminal cost compares the final population distribution μ T = 1 N ∑ i = 1 N δ s T i \mu_T = \frac{1}{N}\sum_{i=1}^{N}\delta_{s_T^i} μ T = N 1 ∑ i = 1 N δ s T i to a target configuration ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) through the optimal transport cost
K [ g T ] ( μ T , ν ) = inf γ ∈ Γ ( μ T , ν ) ∫ S × Z g T ( s , z ) d γ ( s , z ) . (1) \mathcal{K}[g_T](\mu_T,\nu)=\inf_{\gamma\in\Gamma(\mu_T,\nu)}\int_{S\times Z}g_T(s,z)\mathrm{d}\gamma(s,z). \tag*{(1)} K [ g T ] ( μ T , ν ) = γ ∈ Γ ( μ T , ν ) inf ∫ S × Z g T ( s , z ) d γ ( s , z ) . (1)
For g T ( s , z ) = ∥ s − z ∥ p g_T(s,z)=\lVert s-z\rVert^p g T ( s , z ) = ∥ s − z ∥ p with p ≥ 1 p\ge1 p ≥ 1 , K [ g T ] ( μ T , ν ) \mathcal{K}[g_T](\mu_T,\nu) K [ g T ] ( μ T , ν ) is the p p p -th power of the type-p p p Wasserstein distance. Since this is a fully observed cooperative problem, it can be cast as a multi-agent Markov decision process (MDP) [29 ] , which reduces to a single-agent MDP over the joint action space. By standard MDP theory [30 , 31 ] , we can restrict attention, without loss of optimality, to deterministic Markov policies π = ( π t ) t = 0 T − 1 \pi=(\pi_t)_{t=0}^{T-1} π = ( π t ) t = 0 T − 1 , universally measurable on general Borel state and action spaces [31 Ch. 8] . We denote this class by Π \Pi Π . Throughout, we assume that, given the current states and actions, the noise is independent of the past, so the law of the noise sequence ( w t ) t = 0 T − 1 (w_t)_{t=0}^{T-1} ( w t ) t = 0 T − 1 , w t : = ( w t 1 , … , w t N ) w_t := (w_t^1,\ldots,w_t^N) w t := ( w t 1 , … , w t N ) , under which expectations are taken, is determined by the dynamics, the policy, and the noise model.
Problem 1. Given initial states s ˉ 0 = ( s ˉ 0 1 , … , s ˉ 0 N ) \bar{s}_0=(\bar{s}_0^1,\ldots,\bar{s}_0^N) s ˉ 0 = ( s ˉ 0 1 , … , s ˉ 0 N ) and a target distribution ν \nu ν , learn a policy π ∈ Π \pi\in\Pi π ∈ Π minimizing the expected cumulative cost
J N ( s ˉ 0 , ν ) = inf π ∈ Π E ( w t ) t = 0 T − 1 [ 1 N ∑ i = 1 N ∑ t = 0 T − 1 g t ( s t i , a t i , w t i ) + K [ g T ] ( μ T , ν ) ] J^N(\bar{s}_0,\nu)=\inf_{\pi\in\Pi}\mathbb{E}_{(w_t)_{t=0}^{T-1}}\left[\frac{1}{N}\sum_{i=1}^{N}\sum_{t=0}^{T-1}g_t(s_t^i,a_t^i,w_t^i)+\mathcal{K}[g_T](\mu_T,\nu)\right] J N ( s ˉ 0 , ν ) = π ∈ Π inf E ( w t ) t = 0 T − 1 [ N 1 i = 1 ∑ N t = 0 ∑ T − 1 g t ( s t i , a t i , w t i ) + K [ g T ] ( μ T , ν ) ] s.t. s t + 1 i = f t ( s t i , a t i , w t i ) , s 0 = s ˉ 0 . \ s_{t+1}^i=f_t(s_t^i,a_t^i,w_t^i),\quad s_0=\bar{s}_0. s t + 1 i = f t ( s t i , a t i , w t i ) , s 0 = s ˉ 0 .
Population-based formulation
We generalize Problem 1 at the population level by replacing the ordered state s t s_t s t with the empirical measure μ t = 1 N ∑ i = 1 N δ s t i ∈ P ( S ) \mu_t=\frac{1}{N}\sum_{i=1}^{N}\delta_{s_t^i}\in\mathcal{P}(S) μ t = N 1 ∑ i = 1 N δ s t i ∈ P ( S ) , which discards agent labels and makes the permutation invariance in the problem explicit. We initially allow general population states μ t ∈ P ( S ) \mu_t\in\mathcal{P}(S) μ t ∈ P ( S ) and we will later specialize this formulation back to the finite-agent setting.
Population control. A population-level control must specify how the mass at each state is split among actions. Such a control is a state-action distribution λ t ≡ λ t ( μ t ) ∈ P ( S × A ) \lambda_t\equiv\lambda_t(\mu_t)\in\mathcal{P}(S\times A) λ t ≡ λ t ( μ t ) ∈ P ( S × A ) whose S S S -marginal is the current population state: ( proj S ) # λ t = μ t (\operatorname{proj}_{S})_{\#}\lambda_t=\mu_t ( proj S ) # λ t = μ t . For example, if all agents follow a common deterministic policy π t : S → A \pi_t:S\to A π t : S → A , then λ t = ( Id S , π t ) # μ t \lambda_t=(\operatorname{Id}_S,\pi_t)_{\#}\mu_t λ t = ( Id S , π t ) # μ t , that is, all the mass that μ t \mu_t μ t places at state s s s is assigned to the state-action pair ( s , π t ( s ) ) (s,\pi_t(s)) ( s , π t ( s )) .
Noise and population dynamics. When dynamics are deterministic, the next population state is the pushforward of λ t \lambda_t λ t under the dynamics, i.e., μ t + 1 = f t # λ t \mu_{t+1}=f_{t\#}\lambda_t μ t + 1 = f t # λ t . The stochastic case instead requires more care, as the transition also depends on the specific noise realizations across the population. In the agent-based formulation, each agent i i i draws w t i ∼ ξ t ( ⋅ ∣ s t i , a t i ) w_t^i\sim\xi_t(\cdot\mid s_t^i,a_t^i) w t i ∼ ξ t ( ⋅ ∣ s t i , a t i ) and transitions to f t ( s t i , a t i , w t i ) f_t(s_t^i,a_t^i,w_t^i) f t ( s t i , a t i , w t i ) . At the population level, these individual state-action-dependent noise draws cannot simply be replaced by their underlying distribution ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) . Indeed, the next population state depends on the collection of noise draws actually realized across all agents, which is itself a random object. We therefore represent the realized noise draws at each state-action pair by a state-action dependent empirical noise distribution ξ ^ t s , a ∈ P ( W ) \hat{\xi}_t^{s,a}\in\mathcal{P}(W) ξ ^ t s , a ∈ P ( W ) . The distinction between ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) and ξ ^ t s , a \hat{\xi}_t^{s,a} ξ ^ t s , a is crucial: the former is the single-agent noise law, while the latter collects the noise realizations that drive the population transition. Collecting the realized empirical kernels ξ ^ t s , a \hat{\xi}_t^{s,a} ξ ^ t s , a across all state-action pairs, we define the joint state-action-noise distribution ρ t \rho_t ρ t and the population dynamics as
ρ t = λ t ⊗ ξ ^ t s , a ∈ P ( S × A × W ) , μ t + 1 = f t # ρ t , \rho_t=\lambda_t\otimes\hat{\xi}_t^{s,a}\in\mathcal{P}(S\times A\times W),\qquad\mu_{t+1}=f_{t\#}\rho_t, ρ t = λ t ⊗ ξ ^ t s , a ∈ P ( S × A × W ) , μ t + 1 = f t # ρ t ,
where ρ t \rho_t ρ t records the fraction of the population experiencing each triple ( s , a , w ) (s,a,w) ( s , a , w ) .
Example 2.1. Consider three agents with state-action distribution λ t = 2 3 δ ( 0 , a ) + 1 3 δ ( 1 , a ) \lambda_t=\frac{2}{3}\delta_{(0,a)}+\frac{1}{3}\delta_{(1,a)} λ t = 3 2 δ ( 0 , a ) + 3 1 δ ( 1 , a ) , i.e., two agents are in state 0 0 0 and use action a a a , and one agent is in state 1 1 1 and uses action a a a . Assume that the agents are subject to individual noise and that the noise realizations for the two agents at ( 0 , a ) (0,a) ( 0 , a ) are 0 0 0 and 1 1 1 , while the noise realization for the agent at ( 1 , a ) (1,a) ( 1 , a ) is 1 1 1 . Then, ξ ^ t 0 , a = 1 2 δ 0 + 1 2 δ 1 \hat{\xi}_t^{0,a}=\frac{1}{2}\delta_0+\frac{1}{2}\delta_1 ξ ^ t 0 , a = 2 1 δ 0 + 2 1 δ 1 and ξ ^ t 1 , a = δ 1 \hat{\xi}_t^{1,a}=\delta_1 ξ ^ t 1 , a = δ 1 . Accordingly, the state-action-noise distribution is ρ t : = λ t ⊗ ξ ^ t s , a = 1 3 δ ( 0 , a , 0 ) + 1 3 δ ( 0 , a , 1 ) + 1 3 δ ( 1 , a , 1 ) \rho_t:=\lambda_t\otimes\hat{\xi}_t^{s,a}=\frac{1}{3}\delta_{(0,a,0)}+\frac{1}{3}\delta_{(0,a,1)}+\frac{1}{3}\delta_{(1,a,1)} ρ t := λ t ⊗ ξ ^ t s , a = 3 1 δ ( 0 , a , 0 ) + 3 1 δ ( 0 , a , 1 ) + 3 1 δ ( 1 , a , 1 ) and the next population distribution is μ t + 1 = ( f t ) # ρ t = 1 3 δ f t ( 0 , a , 0 ) + 1 3 δ f t ( 0 , a , 1 ) + 1 3 δ f t ( 1 , a , 1 ) \mu_{t+1}=(f_t)_{\#}\rho_t=\frac{1}{3}\delta_{f_t(0,a,0)}+\frac{1}{3}\delta_{f_t(0,a,1)}+\frac{1}{3}\delta_{f_t(1,a,1)} μ t + 1 = ( f t ) # ρ t = 3 1 δ f t ( 0 , a , 0 ) + 3 1 δ f t ( 0 , a , 1 ) + 3 1 δ f t ( 1 , a , 1 ) .
Cost. Since 1 N ∑ i = 1 N g t ( s t i , a t i , w t i ) = ∫ S × A × W g t ( s , a , w ) d ρ t ( s , a , w ) \frac{1}{N}\sum_{i=1}^{N}g_t(s_t^i,a_t^i,w_t^i)=\int_{S\times A\times W}g_t(s,a,w)d\rho_t(s,a,w) N 1 ∑ i = 1 N g t ( s t i , a t i , w t i ) = ∫ S × A × W g t ( s , a , w ) d ρ t ( s , a , w ) , the cost of a given control λ = ( λ t ) t = 0 T − 1 \lambda=(\lambda_t)_{t=0}^{T-1} λ = ( λ t ) t = 0 T − 1 is
J λ ( μ ˉ 0 , ν ) = E ( ξ ^ t ) t = 0 T − 1 [ ∑ t = 0 T − 1 ∫ S × A × W g t ( s , a , w ) d ρ t ( s , a , w ) + K [ g T ] ( μ T , ν ) ] . J^\lambda(\bar{\mu}_0,\nu)=\mathbb{E}_{(\hat{\xi}_t)_{t=0}^{T-1}}\left[\sum_{t=0}^{T-1}\int_{S\times A\times W}g_t(s,a,w)d\rho_t(s,a,w)+\mathcal{K}[g_T](\mu_T,\nu)\right]. J λ ( μ ˉ 0 , ν ) = E ( ξ ^ t ) t = 0 T − 1 [ t = 0 ∑ T − 1 ∫ S × A × W g t ( s , a , w ) d ρ t ( s , a , w ) + K [ g T ] ( μ T , ν ) ] .
where ξ ^ t = ( ξ ^ t s , a ) ( s , a ) ∈ S × A \hat{\xi}_t=(\hat{\xi}_t^{s,a})_{(s,a)\in S\times A} ξ ^ t = ( ξ ^ t s , a ) ( s , a ) ∈ S × A collects all realized noise kernels at time t t t . The expectation is then taken over the randomness of these profiles. This keeps finite-population fluctuations and noise correlations explicit while reducing the population-level randomness to the single object ξ ^ t \hat{\xi}_t ξ ^ t . We refer the reader to Appendix A for additional details and explicit construction for canonical noise models. Since μ t \mu_t μ t is random for t ≥ 1 t\geq1 t ≥ 1 , an admissible population control is a sequence of (universally measurable) feedback maps μ ↦ λ t ( μ ) \mu\mapsto\lambda_t(\mu) μ ↦ λ t ( μ ) with S S S -marginal μ \mu μ , applied before ξ ^ t \hat{\xi}_t ξ ^ t is realized. We write λ t \lambda_t λ t for λ t ( μ t ) \lambda_t(\mu_t) λ t ( μ t ) .
Problem 2. Given an initial population distribution μ ˉ 0 \bar{\mu}_0 μ ˉ 0 and a target distribution ν \nu ν , find an admissible population control ( λ t ) t = 0 T − 1 (\lambda_t)_{t=0}^{T-1} ( λ t ) t = 0 T − 1 that minimizes the expected cumulative cost
J ( μ ˉ 0 , ν ) = inf ( λ t ) t = 0 T − 1 E ( ξ ^ t ) t = 0 T − 1 [ ∑ t = 0 T − 1 ∫ S × A × W g t ( s , a , w ) d ρ t ( s , a , w ) + K [ g T ] ( μ T , ν ) ] J(\bar{\mu}_0,\nu)=\inf_{(\lambda_t)_{t=0}^{T-1}}\mathbb{E}_{(\hat{\xi}_t)_{t=0}^{T-1}}\left[\sum_{t=0}^{T-1}\int_{S\times A\times W}g_t(s,a,w)d\rho_t(s,a,w)+\mathcal{K}[g_T](\mu_T,\nu)\right] J ( μ ˉ 0 , ν ) = ( λ t ) t = 0 T − 1 inf E ( ξ ^ t ) t = 0 T − 1 [ t = 0 ∑ T − 1 ∫ S × A × W g t ( s , a , w ) d ρ t ( s , a , w ) + K [ g T ] ( μ T , ν ) ] s.t. μ 0 = μ ˉ 0 , μ t + 1 = f t # ρ t , ( proj S × A S × A ) # λ t = μ t , ρ t = λ t ⊗ ξ ^ t s , a \mu_0=\bar{\mu}_0,\qquad\mu_{t+1}=f_{t\#}\rho_t,\qquad(\operatorname{proj}_{S\times A}^{S\times A})_{\#}\lambda_t=\mu_t,\qquad\rho_t=\lambda_t\otimes\hat{\xi}_t^{s,a} μ 0 = μ ˉ 0 , μ t + 1 = f t # ρ t , ( proj S × A S × A ) # λ t = μ t , ρ t = λ t ⊗ ξ ^ t s , a .
The population formulation removes the artificial labels from the state description and generalizes Problem 1. Indeed, any finite-agent policy induces an empirical state-action distribution λ t = 1 N ∑ i = 1 N δ ( s t i , a t i ) \lambda_t=\frac{1}{N}\sum_{i=1}^{N}\delta_{(s_t^i,a_t^i)} λ t = N 1 ∑ i = 1 N δ ( s t i , a t i ) with S S S -marginal μ t \mu_t μ t , thus every policy admissible for Problem 1 is feasible in Problem 2 and J ( μ ˉ 0 , ν ) ≤ J N ( s ˉ 0 , ν ) J(\bar{\mu}_0,\nu)\leq J^N(\bar{s}_0,\nu) J ( μ ˉ 0 , ν ) ≤ J N ( s ˉ 0 , ν ) . The converse inequality does not hold in general. Problem 2 allows for arbitrary population state-action distributions, including those with more than N N N atoms, which are not realizable by N N N agents. However, we show in Corollary 3.3 that, for empirical target distributions ν N \nu^N ν N (i.e., N N N targets with equal mass), the separation bound derived for Problem 2 directly upper bounds the agent-based cost-to-go J N ( s ˉ 0 , ν N ) J^N(\bar{s}_0,\nu^N) J N ( s ˉ 0 , ν N ) , yielding a deployable policy for Problem 1.
Separation principle
In this section, we prove a separation principle for Problem 2, showing that the population cost-to-go function separates into optimal single-agent control toward a target and optimal target assignment across the population; see Theorem 3.1 below. We then specialize the result to the finite-agent value J N J^N J N of Problem 1 and obtain a deployable policy that we use in § 4 for algorithm design.
Target-conditioned single-agent problem. Fix a target z ∈ Z z\in\mathcal{Z} z ∈ Z . For an agent in state s ∈ S s\in S s ∈ S at time t t t , define j t ( s , z ) j_t(s,z) j t ( s , z ) as the optimal cost-to-go for reaching z z z over the remaining horizon:
j t ( s , z ) = inf ( π τ ) τ = t T − 1 E ( w τ ) τ = t T − 1 [ ∑ τ = t T − 1 g τ ( s τ , a τ , w τ ) + g T ( s T , z ) ] j_t(s,z)=\inf_{(\pi_\tau)_{\tau=t}^{T-1}}\mathbb{E}_{(w_\tau)_{\tau=t}^{T-1}}\left[\sum_{\tau=t}^{T-1}g_\tau(s_\tau,a_\tau,w_\tau)+g_T(s_T,z)\right] j t ( s , z ) = ( π τ ) τ = t T − 1 inf E ( w τ ) τ = t T − 1 [ τ = t ∑ T − 1 g τ ( s τ , a τ , w τ ) + g T ( s T , z ) ]
s.t. s t = s , s τ + 1 = f τ ( s τ , a τ , w τ ) , w τ ∼ ξ ^ τ ( ⋅ ∣ s τ , a τ ) . s_t=s,\qquad s_{\tau+1}=f_\tau(s_\tau,a_\tau,w_\tau),\qquad w_\tau\sim\hat{\xi}_\tau(\,\cdot\mid s_\tau,a_\tau). s t = s , s τ + 1 = f τ ( s τ , a τ , w τ ) , w τ ∼ ξ ^ τ ( ⋅ ∣ s τ , a τ ) .
{#eq:2}
This is a standard single-agent stochastic optimal control problem, whose complexity is independent of the population size N N N . Its Bellman recursion is
j t ( s t , z ) = inf a t ∈ A E w t [ g t ( s t , a t , w t ) + j t + 1 ( f t ( s t , a t , w t ) , z ) ] , j T ( s T , z ) = g T ( s T , z ) . (2) j_t(s_t,z)=\inf_{a_t\in A}\mathbb{E}_{w_t}\left[g_t(s_t,a_t,w_t)+j_{t+1}(f_t(s_t,a_t,w_t),z)\right],\qquad j_T(s_T,z)=g_T(s_T,z). \tag*{(2)} j t ( s t , z ) = a t ∈ A inf E w t [ g t ( s t , a t , w t ) + j t + 1 ( f t ( s t , a t , w t ) , z ) ] , j T ( s T , z ) = g T ( s T , z ) . (2)
A separation principle. For μ t ∈ P ( S ) \mu_t\in\mathcal{P}(S) μ t ∈ P ( S ) , let J t ( μ t , ν ) J_t(\mu_t,\nu) J t ( μ t , ν ) denote the optimal expected cost at time t t t , when the population μ t \mu_t μ t aims to match the target distribution ν \nu ν . At t = T t=T t = T , J T ( μ , ν ) = K [ g T ] ( μ , ν ) J_T(\mu,\nu)=\mathcal{K}[g_T](\mu,\nu) J T ( μ , ν ) = K [ g T ] ( μ , ν ) . Fix t < T t<T t < T . Since P ( S ) \mathcal{P}(S) P ( S ) is a Polish space whenever S S S is Polish [32 Theorem 15.15] , the cost is additive over time, and the population dynamics are Markovian, standard dynamic programming yields the population-level Bellman recursion [31 Ch. 8]
J t ( μ t , ν ) = inf λ t : ( proj S S × A ) # λ t = μ t E ( ξ ^ t ) t = 0 T − 1 [ ∫ S × A × W g t ( s , a , w ) d ρ t ( s , a , w ) + J t + 1 ( f t # ρ t , ν ) ] . (3) J_t(\mu_t,\nu)=\inf_{\lambda_t:(\operatorname{proj}_{S}^{S\times A})_{\#}\lambda_t=\mu_t}\mathbb{E}_{(\hat{\xi}_t)_{t=0}^{T-1}}\left[\int_{S\times A\times W}g_t(s,a,w)\,d\rho_t(s,a,w)+J_{t+1}(f_{t\#}\rho_t,\nu)\right]. \tag*{(3)} J t ( μ t , ν ) = λ t : ( proj S S × A ) # λ t = μ t inf E ( ξ ^ t ) t = 0 T − 1 [ ∫ S × A × W g t ( s , a , w ) d ρ t ( s , a , w ) + J t + 1 ( f t # ρ t , ν ) ] . (3)
Theorem 3.1 (Separation principle, Part I: upper bound for the population cost-to-go). Consider Problem 2. For all t ∈ { 0 , … , T } t \in\{0,\ldots,T\} t ∈ { 0 , … , T } , the population cost-to-go function (3) is upper bounded by an optimal transport problem whose transportation cost is the single-agent cost-to-go function j t j_t j t in (2):
J t ( μ t , ν ) ≤ K [ j t ] ( μ t , ν ) = inf γ ∈ Γ ( μ t , ν ) ∫ S × Z j t ( s , z ) d γ ( s , z ) . (4) J_t(\mu_t,\nu) \leq\mathcal{K}[j_t](\mu_t,\nu) = \inf_{\gamma\in\Gamma(\mu_t,\nu)} \int_{S \times Z} j_t(s,z)\,\mathrm{d}\gamma(s,z). \tag*{(4)} J t ( μ t , ν ) ≤ K [ j t ] ( μ t , ν ) = γ ∈ Γ ( μ t , ν ) inf ∫ S × Z j t ( s , z ) d γ ( s , z ) . (4) In particular, combined with the admissible control constructed in Part II below, the same upper bound holds for the optimal cost J ( μ ˉ 0 , ν ) J(\bar{\mu}_0,\nu) J ( μ ˉ 0 , ν ) of Problem 2.
Discussion. The upper bound (4) separates the stochastic population control problem into target-conditioned single-agent control and population-level assignment, in which j t j_t j t serves as the transport cost. This separation generally comes at the price of an inequality because the two problems account for future noise differently. In the optimal transport cost, K [ j t ] ( μ t , ν ) \mathcal{K}[j_t](\mu_t,\nu) K [ j t ] ( μ t , ν ) , mass is assigned to targets based on j t ( s , z ) j_t(s,z) j t ( s , z ) , which averages over future noise realizations. By contrast, the Bellman recursion (3) evaluates the remaining cost J t + 1 ( f t # ρ t , ν ) J_{t+1}(f_{t\#}\rho_t,\nu) J t + 1 ( f t # ρ t , ν ) at the realized future population f t # ρ t f_{t\#}\rho_t f t # ρ t , allowing the population-level assignment to adapt to the observed transition. The resulting minimization and expectation do not commute, which is the cause of the inequality (4) . At the same time, averaging over the noise is necessary for a meaningful separation, since it preserves the structure of a standard stochastic optimal control problem in (2). Appendix D quantifies the tightness of this inequality: if j τ + 1 j_{\tau+1} j τ + 1 is L τ + 1 L_{\tau+1} L τ + 1 -Lipschitz in the state, the gap K [ j t ] ( μ t , ν ) − J t ( μ t , ν ) \mathcal{K}[j_t](\mu_t,\nu)-J_t(\mu_t,\nu) K [ j t ] ( μ t , ν ) − J t ( μ t , ν ) in (4) is bounded by ∑ τ = t T − 1 L τ + 1 D τ \sum_{\tau=t}^{T-1} L_{\tau+1}D_\tau ∑ τ = t T − 1 L τ + 1 D τ , where D τ D_\tau D τ is the worst-case expected distance between two (conditionally independent) next states generated from the same state-action pair. The gap therefore depends linearly on the dispersion, and vanishes whenever dynamics are deterministic. In particular, under deterministic dynamics, the next population state is fully determined by ( μ t , λ t ) (\mu_t,\lambda_t) ( μ t , λ t ) , the expectation in (3) disappears, and the upper bound in Theorem 3.1, Part I becomes exact, recovering the deterministic separation of [27 ] .
Corollary 3.2 (Exact separation in the deterministic setting [27 Corollary 4.2] ). When the dynamics are deterministic, the population cost-to-go function (3) is an optimal transport cost with the single-agent cost-to-go function as its transportation cost:
J t ( μ t , ν ) = inf γ ∈ Γ ( μ t , ν ) ∫ S × Z j t ( s , z ) d γ ( s , z ) . J_t(\mu_t,\nu)=\inf_{\gamma\in\Gamma(\mu_t,\nu)}\int_{S\times Z}j_t(s,z)\,\mathrm{d}\gamma(s,z). J t ( μ t , ν ) = γ ∈ Γ ( μ t , ν ) inf ∫ S × Z j t ( s , z ) d γ ( s , z ) .
From a bound to a policy. For any ε > 0 \varepsilon>0 ε > 0 , Theorem 3.1, Part II constructs an explicit population-level control ( λ t ε ) t = 0 T − 1 (\lambda_t^\varepsilon)_{t=0}^{T-1} ( λ t ε ) t = 0 T − 1 whose cost exceeds the bound by at most ε \varepsilon ε . Here, J λ ( μ ˉ 0 , ν ) J^\lambda(\bar{\mu}_0,\nu) J λ ( μ ˉ 0 , ν ) denotes the expected cumulative cost of Problem 2 when the population follows the control ( λ t ) t = 0 T − 1 (\lambda_t)_{t=0}^{T-1} ( λ t ) t = 0 T − 1 .
Theorem 3.1 continued (Separation principle, Part II: constructive separation-based control). For every ε > 0 \varepsilon>0 ε > 0 and each time step t t t , let γ t ε ∈ Γ ( μ t , ν ) \gamma_t^\varepsilon\in\Gamma(\mu_t,\nu) γ t ε ∈ Γ ( μ t , ν ) be an ε 2 T \frac{\varepsilon}{2T} 2 T ε -optimal coupling for the transport problem K [ j t ] ( μ t , ν ) \mathcal{K}[j_t](\mu_t,\nu) K [ j t ] ( μ t , ν ) and let a t ε : S × Z → A a_t^\varepsilon:S\times Z\to A a t ε : S × Z → A be an ε 2 T \frac{\varepsilon}{2T} 2 T ε -optimal single-agent action for the Bellman recursion (2) . Define the separated state-action distribution λ t ε : = ( proj S S × Z , a t ε ) # γ t ε \lambda_t^\varepsilon:= (\operatorname{proj}^{S\times Z}_S,a_t^\varepsilon)_\#\gamma_t^\varepsilon λ t ε := ( proj S S × Z , a t ε ) # γ t ε . Then,
J λ ε ( μ ˉ 0 , ν ) ≤ inf γ ∈ Γ ( μ ˉ 0 , ν ) ∫ S × Z j 0 ( s , z ) d γ ( s , z ) + ε . J^{\lambda^\varepsilon}(\bar{\mu}_0,\nu)\leq\inf_{\gamma\in\Gamma(\bar{\mu}_0,\nu)}\int_{S\times Z}j_0(s,z)\,\mathrm{d}\gamma(s,z)+\varepsilon. J λ ε ( μ ˉ 0 , ν ) ≤ γ ∈ Γ ( μ ˉ 0 , ν ) inf ∫ S × Z j 0 ( s , z ) d γ ( s , z ) + ε . In particular, if optimal couplings γ t ⋆ \gamma_t^\star γ t ⋆ and optimal single-agent actions a t ⋆ a_t^\star a t ⋆ exist and can be selected measurably, we can choose ε = 0 \varepsilon=0 ε = 0 .
This construction has a natural interpretation. Assuming, for simplicity, that optimal couplings and actions exist (e.g., all spaces compact and all functions continuous; see the remark in Appendix C), the coupling γ t ⋆ ∈ Γ ( μ t , ν ) \gamma_t^\star\in\Gamma(\mu_t,\nu) γ t ⋆ ∈ Γ ( μ t , ν ) optimally assigns the population mass at each state s s s to targets, while a t ⋆ a_t^\star a t ⋆ selects the optimal action for each state-target pair. This is the core of our separation principle: the coupling γ t ⋆ \gamma_t^\star γ t ⋆ indicates how to dispatch agents to targets, while a t ⋆ a_t^\star a t ⋆ indicates how to route the single agent to the assigned target. Part II of Theorem 3.1 guarantees that the resulting population control has expected cost no greater than the transport upper bound established in Part I.
Algorithm 1 SALT: Separation-based Assignment and Learning via optimal Transport
Require: Initial target-conditioned policy π ( 0 ) ( a ∣ s , z ) \pi^{(0)}(a \mid s,z) π ( 0 ) ( a ∣ s , z ) , cost-to-go function estimates j ^ t ( 0 ) ( s , z ) \hat{j}^{(0)}_t(s,z) j ^ t ( 0 ) ( s , z ) , horizon T T T , exploration schedule ( α k ) k ≥ 0 (\alpha_k)_{k\geq0} ( α k ) k ≥ 0
for episodes k = 0 , 1 , 2 , … k=0,1,2,\ldots k = 0 , 1 , 2 , … do
Sample the episode initial population μ 0 ( k ) \mu^{(k)}_0 μ 0 ( k ) and target distribution ν k \nu_k ν k .
Initial population-level assignment. With probability 1 − α k 1-\alpha_k 1 − α k , assign agents to targets via optimal transport with cost j ^ 0 ( k ) \hat{j}^{(k)}_0 j ^ 0 ( k ) . Otherwise assign randomly.
Single-agent learning. Roll out the shared target-conditioned policy π ( k ) \pi^{(k)} π ( k ) in parallel for all agents over the horizon T T T , with the assigned targets kept fixed.
Update π ( k + 1 ) \pi^{(k+1)} π ( k + 1 ) and j ^ t ( k + 1 ) ( s , z ) \hat{j}^{(k+1)}_t(s,z) j ^ t ( k + 1 ) ( s , z ) with the chosen single-agent RL algorithm from the N N N trajectories collected in Step 4.
end for
Deployment (online-assignment variant).
for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 do
Assign agents to targets via optimal transport with cost j ^ t \hat{j}_t j ^ t .
Execute one step of π t \pi_t π t with the current target assignments.
end for
Robustness to weakly coupled dynamics. While Theorem 3.1 assumes decoupled dynamics and stage costs, Appendix E shows that the separation-based policy remains near-optimal when both depend weakly on the population. In particular, we construct the separation-based policy of Theorem 3.1, Part II using a decoupled surrogate and deploy it in the coupled system. Let ε τ f \varepsilon^\mathsf{f}_\tau ε τ f bound the distance between the next states the surrogate and the coupled system produce from the same state, action, and noise, let δ τ \delta_\tau δ τ the expected difference between their stage costs, and let L τ L_\tau L τ as above. The loss of optimality is then at most ∑ τ = 0 T − 1 L τ + 1 D τ + 2 ∑ τ = 0 T − 1 ( δ τ + L τ + 1 ε τ f ) \sum_{\tau=0}^{T-1}L_{\tau+1}D_\tau+2\sum_{\tau=0}^{T-1}(\delta_\tau+L_{\tau+1}\varepsilon^\mathsf{f}_\tau) ∑ τ = 0 T − 1 L τ + 1 D τ + 2 ∑ τ = 0 T − 1 ( δ τ + L τ + 1 ε τ f ) . The first term is the bound on the gap in (5) derived above, while the second isolates the loss caused by population interactions, and vanishes with them.
Implications for agent-based formulation J N J^N J N . Theorem 3.1 is stated for the population formulation in Problem 2. Corollary 3.3 bridges this to the original agent-based setting: for N N N agents and N N N target masses, the same transport surrogate K [ j t ] \mathcal{K}[j_t] K [ j t ] upper bounds J t N J^N_t J t N .
Corollary 3.3 (Agent-based upper bound). Let s t = ( s t 1 , … , s t N ) ∈ S N s_t=(s^1_t,\ldots,s^N_t)\in S^N s t = ( s t 1 , … , s t N ) ∈ S N be the agent states at time t t t , with empirical distribution μ t N = 1 N ∑ i = 1 N δ s t i \mu^N_t=\frac{1}{N}\sum_{i=1}^{N}\delta_{s^i_t} μ t N = N 1 ∑ i = 1 N δ s t i . Let z = ( z 1 , … , z N ) ∈ Z N z=(z^1,\ldots,z^N)\in Z^N z = ( z 1 , … , z N ) ∈ Z N be fixed targets with empirical distribution ν N = 1 N ∑ i = 1 N δ z i \nu^N=\frac{1}{N}\sum_{i=1}^{N}\delta_{z^i} ν N = N 1 ∑ i = 1 N δ z i . Then, the cost-to-go function for the optimal control problem in Problem 1, denoted by J t N ( s t , ν N ) J^N_t(s_t,\nu^N) J t N ( s t , ν N ) , satisfies
J t N ( s t , ν N ) ≤ K [ j t ] ( μ t N , ν N ) . (5) J^N_t(s_t,\nu^N)\leq\mathcal{K}[j_t](\mu^N_t,\nu^N). \tag*{(5)} J t N ( s t , ν N ) ≤ K [ j t ] ( μ t N , ν N ) . (5) In particular, the inequality holds true for the cost-to-go J N ( s 0 , ν N ) J^N(s_0,\nu^N) J N ( s 0 , ν N ) in Problem 1.
We defer the proofs of Theorem 3.1, Part I, Theorem 3.1, Part II, and Corollary 3.3 to Appendix C.
SALT: Separation-based Assignment and Learning via optimal Transport
Corollary 3.3 suggests using the upper bound
K [ j t ] ( μ t , ν ) = inf γ ∈ Γ ( μ t , ν ) ∫ j t ( s , z ) d γ ( s , z ) \mathcal{K}[j_t](\mu_t,\nu)=\inf_{\gamma\in\Gamma(\mu_t,\nu)}\int j_t(s,z)\,d\gamma(s,z) K [ j t ] ( μ t , ν ) = γ ∈ Γ ( μ t , ν ) inf ∫ j t ( s , z ) d γ ( s , z )
as a tractable surrogate for the agent-based cost-to-go J t N ( s t , ν N ) J^N_t(s_t,\nu^N) J t N ( s t , ν N ) . The resulting learning paradigm, summarized in Algorithm 1, has two phases: (i) learn the single-agent target-conditioned cost-to-go function j ^ t \hat{j}_t j ^ t , and (ii) use it as the transport cost to assign agents to targets. The choice of single-agent algorithm is entirely open: any method capable of learning a target-conditioned cost-to-go estimate j ^ t \hat{j}_t j ^ t (Q-learning [33 ] , DDQN [34 ] , PPO [28 ] , A2C [35 ] , etc. ) can be plugged in as the base learner. We refer to this family of methods as SALT, for Separation-based Assignment and Learning via optimal Transport. An instance using a single-agent RL algorithm is denoted SALT-ALG; e.g., SALT-PPO.
Discussion. SALT replaces learning over the joint population state-action space with a target-conditioned single-agent policy π ( ⋅ ∣ s , z ) \pi(\cdot\mid s,z) π ( ⋅ ∣ s , z ) and cost-to-go estimate j ^ ( s , z ) \hat{j}(s,z) j ^ ( s , z ) shared across all agents. Thus, population-level coordination is handled only through the optimal transport assignment, while both learned functions depend only on a single agent’s state–target pair. This has three main benefits. (i) Scalability: learning occurs entirely at the target-conditioned single-agent level, avoiding the curse of dimensionality of the joint state-action space and generalizing naturally across population sizes. (ii) Trajectory pooling: the same single-agent model is updated from every agent’s trajectory simultaneously. (iii) Modularity: the coordination layer requires only j ^ ( s , z ) \hat{j}(s,z) j ^ ( s , z ) as its transport cost, making SALT compatible with any single-agent RL algorithm.
The time-indexed bound in Theorem 3.1, Part I makes the assignment frequency a deployment choice. The transport assignment can be recomputed at every time step with j ^ t \hat{j}_t j ^ t , as in Algorithm 1, or solved once at t = 0 t=0 t = 0 and kept fixed for the entire episode. Both variants use the same target-conditioned learner and differ only in when the transport assignment is updated.
Training details. At each episode k k k , we sample the initial population μ 0 ( k ) \mu_0^{(k)} μ 0 ( k ) and target distribution ν k \nu_k ν k from the environment, and assign agents to targets by solving the optimal transport problem K [ j ^ 0 ( k ) ] ( μ 0 ( k ) , ν k ) \mathcal{K}[\hat{j}_0^{(k)}](\mu_0^{(k)},\nu_k) K [ j ^ 0 ( k ) ] ( μ 0 ( k ) , ν k ) with the current initial-time estimate j ^ 0 ( k ) ( s , z ) \hat{j}_0^{(k)}(s,z) j ^ 0 ( k ) ( s , z ) as cost. For empirical population and target measures with N N N equally weighted samples, a coupling γ ∈ Γ ( μ t , ν ) \gamma\in\Gamma(\mu_t,\nu) γ ∈ Γ ( μ t , ν ) can be represented as a doubly stochastic matrix (up to the factor 1 / N 1/N 1/ N ). By the Birkhoff–von Neumann theorem an optimal coupling can be taken to be a permutation (see the proof of Corollary 3.3), and thus the optimal transport problem reduces to a one-to-one assignment problem. In particular, given a cost matrix C i j = j ^ t ( k ) ( s i , z j ) C_{ij}=\hat{j}_t^{(k)}(s^i,z^j) C ij = j ^ t ( k ) ( s i , z j ) , we solve min σ ∈ S N ∑ i = 1 N j ^ t ( k ) ( s i , z σ ( i ) ) \min_{\sigma\in S_N}\sum_{i=1}^{N}\hat{j}_t^{(k)}(s^i,z^{\sigma(i)}) min σ ∈ S N ∑ i = 1 N j ^ t ( k ) ( s i , z σ ( i ) ) , where S N S_N S N is the set of permutations of { 1 , … , N } \{1,\ldots,N\} { 1 , … , N } , and assign target z σ ( i ) z^{\sigma(i)} z σ ( i ) to agent i i i . Such an assignment is computed exactly in O ( N 3 ) O(N^3) O ( N 3 ) with the Hungarian algorithm, or approximately using entropic regularization [36 ] . During training we apply this construction at t = 0 t=0 t = 0 with probability 1 − α k 1-\alpha_k 1 − α k , else we assign targets uniformly at random to ensure broad coverage of ( s , z ) (s,z) ( s , z ) pairs and prevent the cost-to-go estimate from overfitting to its own assignments. We then keep the assignments fixed for the entire episode. Each agent therefore faces a single-agent target-conditioned task with terminal cost g T ( s T , z i ) g_T(s_T,z^i) g T ( s T , z i ) , ensuring that the collected transitions are consistent with the cost-to-go j ^ t ( s , z i ) \hat{j}_t(s,z^i) j ^ t ( s , z i ) being learned. Once targets are assigned, agents roll out the shared target-conditioned policy in parallel, producing N N N trajectories that are pooled to update the shared policy and cost-to-go estimate using the chosen single-agent algorithm.
Experimental results
We evaluate SALT against MARL baselines in stochastic grid worlds, and test scalability and fleet-size transfer in a Manhattan urban logistics task. More ablations and experiment details are in Appendix F. The code for our experiments is available at https://github.com/lucia-pezzetti/SALT.
Stochastic grid-world experiments
Separation leads to better performance. We first evaluate SALT on a stochastic grid-world environment. A population of N = 7 N = 7 N = 7 homogeneous agents must reach N N N target locations within a horizon T = 100 T = 100 T = 100 on an 11 × 30 11 \times30 11 × 30 grid. Agents start in the first column, while targets are uniformly sampled from the last ten columns. At each step, the intended action is replaced uniformly by one of the other four actions with probability p p p . We evaluate the individual , local , and global noise models of § 2 , according to which replacement draws are independent across agents, shared among agents with the same state-action pair, or shared across the population. The objective is to minimize average travel time and terminal mismatch to the target distribution. We instantiate SALT with Q-learning [33 ] (SALT-Q), DDQN [34 ] (SALT-DDQN), PPO [28 ] (SALT-PPO), and A2C [35 ] (SALT-A2C), and compare them against six MARL baselines: MAPPO [13 ] , IPPO [37 ] , HAPPO [38 ] , VDN [15 ] , QMIX [16 ] , and mean-field Q-learning (MFQ) [22 ] . At evaluation, SALT uses the online-assignment variant in Algorithm 1, recomputing the transport assignment at every time step. All methods are trained with 5 seeds and evaluated on 50 evaluation seeds per trained policy. Figure 2 shows that SALT consistently outperforms the MARL baselines across all noise levels and noise models. The four SALT variants perform similarly, suggesting that the gain comes from the separation structure rather than from the particular single-agent RL algorithm. Complementary metrics, as well as the complete training, evaluation, and uncertainty details, are reported in Appendix § 14 .§ 1 .
Figure 2. Average travel time to targets as a function of noise probability p p p under individual, local, and global noise. Cells show the pooled arithmetic mean over 50 evaluation rollouts from each of 5 independently trained policies. Figure 7 shows the corresponding distributions. Brighter cells indicate worse relative performance. SALT variants consistently outperform standard MARL baselines.
Separation leads to better scalability. We next study sample complexity as the number of agents increases from N = 2 N = 2 N = 2 to N = 128 N = 128 N = 128 . We use the same grid-world dynamics with individual noise, and increase grid area to keep the target density roughly constant and isolate the effects of scaling from the complexity of the task. During training, agents start from independently sampled rows of the first column, and targets are sampled uniformly across the grid (see Appendix § 14 .§ 2 ). We select PPO as the single-agent learner in SALT, yielding SALT-PPO, and compare it against MAPPO. We report the number of environment interactions required to attain reach-rate milestones of 70%, 80%, 90%, 95%, and 99%, up to a maximum training budget of 2 × 10 9 2 \times10^9 2 × 1 0 9 interactions. Figure 3 shows that SALT-PPO attains the same milestones with up to 480 × 480 \times 480 × fewer interactions than MAPPO, and reaches the 99% milestone for populations of up to N = 128 N = 128 N = 128 agents, while MAPPO fails to reach the 70% milestone at N = 8 N = 8 N = 8 within the configured budget. This gap directly reflects their different scalability properties. MAPPO learns over a joint multi-agent state-action space, whose size grows exponentially with N N N , whereas SALT-PPO learns a target-conditioned policy that is independent of N N N .
Figure 3. Training environment interactions required by SALT-PPO and MAPPO to attain each reach-rate milestone (70–99%) for 10 consecutive batches as population size N N N increases (grid area scaled proportionally to maintain target density). Each cell reports the mean over 5 training seeds, a dash (–) indicates that no seed attained the milestone within the training budget.
Towards a realistic time-varying stochastic urban logistics task
A fleet of delivery vehicles must serve requests across Manhattan while jointly minimizing total travel time. This task couples vehicle-to-request assignment with time-efficient routing through a stochastic, time-varying network and is precisely the setting that motivated this work. SALT separates the problem into a coordination step (assigning vehicles to requests via optimal transport) and a routing step (navigating each vehicle to its current target via a shared target-conditioned learned policy). As motivated in § 4 , assignments remain fixed during each training episode to learn the target-conditioned subproblem, and are recomputed at every routing step during evaluation. The environment is a directed southern Manhattan road graph extracted from OpenStreetMap [39 ] , with 2087 intersections and 4255 directed roads (see Figure 4 ). Travel times depend on road length, road type, traffic-light cycles of 90 seconds (with a road-dependent green-phase duration and a random offset), and stochastic congestion effects. Vehicles start from three depot locations in random proportions and serve delivery requests sampled uniformly across the city. Further environment details are provided in Appendix § 14 .§ 4 . We instantiate SALT with tabular Q-learning (SALT-Q) and compare it against a static baseline that assigns vehicles once at t = 0 t = 0 t = 0 using nominal shortest-path distances and then follows the resulting paths. This standard operational heuristic ignores traffic-light timing and stochastic congestion, since incorporating them exactly requires a much larger spatiotemporal routing problem. We exclude the deep MARL baselines of § 5.1 because, as demonstrated, they become computationally prohibitive on a graph of this scale and are tied to a fixed training fleet size. SALT-Q policies are trained with N = 20 N = 20 N = 20 agents and evaluated, without retraining, on fleets of 1, 5, 10, 20, and 100 vehicles.
Figure 4. Relative travel-time improvement of SALT-Q over static shortest-path routing on the time-varying Manhattan road network, as a function of fleet size. Boxplots pool 500 evaluation instances from each of 8 training seeds. Boxes span the interquartile range, center lines show medians, and whiskers follow the Tukey 1.5-interquartile-range rule. The dashed line indicates zero improvement. SALT-Q is trained with 20 vehicles and evaluated without retraining on fleets of 1–100 vehicles.
Figure 4 reports the relative improvement in travel time over the static shortest-path baseline as a function of fleet size. SALT-Q consistently outperforms the shortest-path baseline, with median improvements ranging from 14.8% (N = 1 N = 1 N = 1 ) to 21.3% (N = 100 N = 100 N = 100 ), reflecting its ability to anticipate traffic-light timing and adapt to stochastic congestion. Performance remains stable across fleet sizes, confirming that SALT generalizes naturally to fleet sizes unseen at training time. The residual variance arises from a natural bias of distributional training: the policy concentrates on high-probability origin–destination pairs, improving sample efficiency at the cost of reduced accuracy on rare assignments. Additional ablations regarding SALT fleet-size transferability and reassignment variants of the shortest-path baseline are reported in Section F.3 and Section F.4, respectively.
Conclusion and limitations
We established a separation principle for cooperative MARL and used it to design SALT, a family of algorithms that learn a single-agent policy and use optimal transport to coordinate agents. SALT outperforms existing MARL baselines in grid worlds, both in terms of performance and scalability. In the southern Manhattan road network, where MARL baselines are impractical, SALT improves travel time by 15–21% over a shortest-path heuristic and generalizes to population sizes not seen during training. Our work does not address several important challenges, which we leave to future work. From a theoretical perspective, our separation principle requires decoupled dynamics and an optimal transport terminal cost, ruling out settings with strong interactions, collisions, or non-transport objectives. Our bound on the optimal cost is also not tight in general. From a practical perspective, the target-conditioned single-agent problem must generalize across the target space, which is challenging when targets are high-dimensional or sampled from distributions far from the training one.
Acknowledgments and Disclosure of Funding
This publication was made possible by the ETH AI Center doctoral fellowship to Lucia Pezzetti. This work was also supported as a part of NCCR Automation, a National Centre of Competence in Research, funded by the Swiss National Science Foundation (grant number 51NF40 225155). Nicolas Lanzetti was in part supported by the PIMCO Fellows Program, by the National Science Foundation under Grant #CCF-2326609, and by the Resnick Sustainability Institute.
References
Noise models and averaging property
This appendix develops the measure-theoretic framework used to handle stochasticity in the multi-agent system. We first introduce the relevant general notation, and illustrate how common noise models fit into this framework through a series of examples. We conclude with Lemma A.10, which will be a key technical tool used in the proof of Theorem 3.1, Part I in Appendix C. Throughout, the space of probability measures on a Polish space X X X , P ( X ) \mathcal{P}(X) P ( X ) , is endowed with the topology of weak convergence and its Borel σ \sigma σ -algebra. Since X X X is a Polish space, P ( X ) \mathcal{P}(X) P ( X ) is itself a Polish space [32 Theorem 15.15] . The product measure of two independent random variables with marginals μ ∈ P ( X ) \mu\in\mathcal{P}(X) μ ∈ P ( X ) and ν ∈ P ( Y ) \nu\in\mathcal{P}(Y) ν ∈ P ( Y ) is the unique measure μ ⊗ ν ∈ P ( X × Y ) \mu\otimes\nu\in\mathcal{P}(X \times Y) μ ⊗ ν ∈ P ( X × Y ) [40 Theorem B, Section 35] such that ( μ ⊗ ν ) ( A × B ) = μ ( A ) ν ( B ) (\mu\otimes\nu)(A \times B) = \mu(A) \nu(B) ( μ ⊗ ν ) ( A × B ) = μ ( A ) ν ( B ) for all Borel sets A ⊆ X A \subseteq X A ⊆ X , B ⊆ Y B \subseteq Y B ⊆ Y .
General framework
In multi-agent systems, randomness enters at the individual level, through the noise realizations for each individual agent. At the population level, these realizations induce randomness in the population distribution and therefore in the next population state. To capture this in a unified way, we work with distributions over noise distributions. Concretely, following the notation in § 2.2 , let λ t ∈ P ( S × A ) \lambda_t \in\mathcal{P}(S \times A) λ t ∈ P ( S × A ) denote the joint state-action distribution of the population at time t t t , and denote by ξ t ( ⋅ ∣ s , a ) ∈ P ( W ) \xi_t(\cdot\mid s,a) \in\mathcal{P}(W) ξ t ( ⋅ ∣ s , a ) ∈ P ( W ) the nominal kernel , i.e., the law of the noise experienced by an agent in state s s s taking action a a a . For agents sharing the same state-action pair ( s , a ) (s,a) ( s , a ) , their noise realizations form the realized kernel ξ ^ t s , a ∈ P ( W ) \hat{\xi}_{t}^{s,a} \in\mathcal{P}(W) ξ ^ t s , a ∈ P ( W ) . Unlike the nominal kernel ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) , which is a fixed object, ξ ^ t s , a \hat{\xi}_{t}^{s,a} ξ ^ t s , a is random, fluctuating across episodes depending on the specific noise draws of the agents in that bin. Therefore, we can talk about the distribution of ξ ^ t s , a \hat{\xi}_{t}^{s,a} ξ ^ t s , a , which we denote by η t s , a ∈ P ( P ( W ) ) \eta_t^{s,a} \in\mathcal{P}(\mathcal{P}(W)) η t s , a ∈ P ( P ( W )) , and collect these across all state-action pairs into the joint object η t ∈ P ( P ( W ) S × A ) \eta_t \in\mathcal{P}(\mathcal{P}(W)^{S \times A}) η t ∈ P ( P ( W ) S × A ) . Finally, to describe the full population transition in this stochastic setting, it is not enough to track where agents are and what actions they take; one also needs to account for the noise they experience. This motivates the introduction of the joint state-action-noise distribution, defined as the gluing
ρ t = λ t ⊗ ξ ^ t s , a ∈ P ( S × A × W ) . \rho_t = \lambda_t \otimes\hat{\xi}_{t}^{s,a} \in\mathcal{P}(S \times A \times W). ρ t = λ t ⊗ ξ ^ t s , a ∈ P ( S × A × W ) .
Intuitively, ρ t ( { ( s , a , w ) } ) \rho_t(\{(s,a,w)\}) ρ t ({( s , a , w )}) is the fraction of agents that are at state s s s , use action a a a , and experience the noise realization w w w . Since the noise realizations are random, ξ ^ t s , a \hat{\xi}_{t}^{s,a} ξ ^ t s , a is random and, consequently, so is ρ t \rho_t ρ t . At this point, the next population is a pushforward of ρ t \rho_t ρ t under the system dynamics f t f_t f t :
μ t + 1 = f t # ρ t . \mu_{t+1} = f_{t\#}\rho_t. μ t + 1 = f t # ρ t .
Different noise models below correspond to different distributional assumptions on ξ ^ t s , a \hat{\xi}_{t}^{s,a} ξ ^ t s , a , and hence on ρ t \rho_t ρ t . The objects introduced above are summarized in Table 1 .
Table 1. Notation for Appendices A–F.
The gluing ρ t = λ t ⊗ ξ ^ t s , a \rho_t = \lambda_t \otimes\hat{\xi}_{t}^{s,a} ρ t = λ t ⊗ ξ ^ t s , a is well-defined per realization only if the realized field is a measurable kernel. We impose this as a standing assumption. It holds directly for the individual and global models, while the only subtle case is discussed in Example A.6.
Assumption A.1 (Well-posedness of the realized noise field). Fix a time step t t t and an admissible state-action distribution λ t \lambda_t λ t . The realized noise field, defined on a probability space ( Ω , F , P ) (\Omega, \mathcal{F}, \mathbb{P}) ( Ω , F , P ) , admits
a version for which there exists a Borel set D t ⊆ S × A D_t \subseteq S \times A D t ⊆ S × A with λ t ( D t ) = 1 \lambda_t(D_t) = 1 λ t ( D t ) = 1 such that
( ( s , a ) , ω ) ↦ ξ ^ t s , a ( ω ) ( B ) ((s,a),\omega) \mapsto\hat{\xi}_t^{s,a}(\omega)(B) (( s , a ) , ω ) ↦ ξ ^ t s , a ( ω ) ( B )
is jointly measurable on D t × Ω D_t \times\Omega D t × Ω for every Borel set B ⊆ W B \subseteq W B ⊆ W .
In particular, almost surely, ( s , a ) ↦ ξ ^ t s , a (s,a) \mapsto\hat{\xi}_t^{s,a} ( s , a ) ↦ ξ ^ t s , a is a Borel kernel on D t D_t D t , so ρ t \rho_t ρ t is a well-defined random element of P ( S × A × W ) \mathcal{P}(S \times A \times W) P ( S × A × W ) , and expectations over the field can be interchanged with λ t \lambda_t λ t -integrals by Tonelli’s theorem.
We additionally impose the following consistency condition on the realized kernel.
Assumption A.2. Fix a time step t t t and an admissible state-action distribution λ t \lambda_t λ t , together with its realized noise field. There exists a Borel set E t ⊆ S × A E_t \subseteq S \times A E t ⊆ S × A with λ t ( E t ) = 1 \lambda_t(E_t) = 1 λ t ( E t ) = 1 such that for all ( s , a ) ∈ E t (s,a) \in E_t ( s , a ) ∈ E t and for every bounded Borel measurable φ : W → R \varphi: W \to\mathbb{R} φ : W → R
E η t s , a [ ∫ W φ ( w ) d ξ ^ t s , a ( w ) ] = ∫ W φ ( w ) d ξ t ( w ∣ s , a ) . (6) \mathbb{E}_{\eta_t^{s,a}}\left[\int_W \varphi(w)d\hat{\xi}_t^{s,a}(w)\right] = \int_W \varphi(w)d\xi_t(w \mid s,a). \tag*{(6)} E η t s , a [ ∫ W φ ( w ) d ξ ^ t s , a ( w ) ] = ∫ W φ ( w ) d ξ t ( w ∣ s , a ) . (6)
Condition (6) should be read as an unbiased or barycenter condition between the realized kernel ξ ^ t s , a \hat{\xi}_t^{s,a} ξ ^ t s , a and the nominal kernel ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) . More precisely, the assumption requires that these two descriptions agree after averaging over the realization of ξ ^ t s , a \hat{\xi}_t^{s,a} ξ ^ t s , a . As we will show below, Assumption A.2 is satisfied by several standard realized-kernel constructions. Specifically, in Proposition A.9 we show that it holds true for the noise models presented in Examples A.4–A.7. More generally, Assumption A.2 is satisfied whenever the realized kernel is chosen from a family of probability kernels whose mixture is the nominal kernel ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) .
What (6) excludes are biased realized-kernel constructions, as we show in the next example.
Example A.3. Let W = { 0 , 1 } W = \{0,1\} W = { 0 , 1 } and ξ t = 1 2 δ 0 + 1 2 δ 1 \xi_t = \frac{1}{2}\delta_0 + \frac{1}{2}\delta_1 ξ t = 2 1 δ 0 + 2 1 δ 1 . Suppose that we artificially bias all noise realizations towards 1 1 1 , so that
ξ ^ t s , a = δ 1 \hat{\xi}_t^{s,a} = \delta_1 ξ ^ t s , a = δ 1 almost surely. Then
E η t s , a [ ∫ W w d ξ ^ t s , a ( w ) ] = 1 ≠ 1 2 = ∫ W w d ξ t ( w ∣ s , a ) . \mathbb{E}_{\eta_t^{s,a}}\left[\int_W w d\hat{\xi}_t^{s,a}(w)\right] = 1 \ne\frac{1}{2} = \int_W w d\xi_t(w \mid s,a). E η t s , a [ ∫ W w d ξ ^ t s , a ( w ) ] = 1 = 2 1 = ∫ W w d ξ t ( w ∣ s , a ) . Hence, (6) fails.
More generally, (6) of Assumption A.2 fails whenever the realized population-level kernel systematically overweights some noise realizations relative to the nominal kernel used in the single-agent Bellman equation. In the remainder of this section, we specialize this general framework to the individual, local, and global noise models considered in this work (Appendix A.2). Finally, in Appendix A.3, we prove a key property of all these noise models: we establish, in Lemma A.10, that the expected cost under the realized kernel ξ ^ t s , a \hat{\xi}_t^{s,a} ξ ^ t s , a equals the cost under the nominal kernel ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) . Ultimately, this unbiasedness property, verified for all three noise models in Proposition A.9, is what allows us to eliminate randomness from the backward recursion in Appendix C and recover the single-agent Bellman equation.
Noise models
We start with the case where the noise is i.i.d. over all agents with the same state-action pair. We start with the case of finitely many agents:
Example A.4. (Individual noise, finitely many agents). Consider N N N finite agents, each with its “own” noise sample
w t i ∼ ξ t ( ⋅ ∣ s t i , a t i ) . w_t^i \sim\xi_t(\cdot\mid s_t^i,a_t^i). w t i ∼ ξ t ( ⋅ ∣ s t i , a t i ) .
We assume noise draws to be independent across agents and across state-action pairs. Given the state-action distribution λ t \lambda_t λ t , its support supp λ t \operatorname{supp}\lambda_t supp λ t is finite (at most N N N atoms, since λ t = 1 N ∑ i = 1 N δ ( s t i , a t i ) \lambda_t = \frac{1}{N}\sum_{i=1}^{N}\delta_{(s_t^i,a_t^i)} λ t = N 1 ∑ i = 1 N δ ( s t i , a t i ) ). For each bin ( s , a ) ∈ supp λ t (s,a) \in\operatorname{supp}\lambda_t ( s , a ) ∈ supp λ t , let
N s , a : = N ⋅ λ t ( { ( s , a ) } ) ≥ 1 (7) N_{s,a} := N \cdot\lambda_t(\{(s,a)\}) \ge1 \tag*{(7)} N s , a := N ⋅ λ t ({( s , a )}) ≥ 1 (7)
be the number of agents in that bin. This is a strictly positive integer, determined by λ t \lambda_t λ t . Define the empirical measure map
Ψ N s , a s , a : W N s , a → P ( W ) \Psi^{s,a}_{N_{s,a}}: W^{N_{s,a}} \to\mathcal{P}(W) Ψ N s , a s , a : W N s , a → P ( W )
w s , a ↦ 1 N s , a ∑ i = 1 N s , a δ w s , a , i , w^{s,a} \mapsto\frac{1}{N_{s,a}}\sum_{i=1}^{N_{s,a}}\delta_{w^{s,a,i}}, w s , a ↦ N s , a 1 i = 1 ∑ N s , a δ w s , a , i ,
where w s , a = ( w s , a , i ) i ∈ [ N s , a ] w^{s,a}=(w^{s,a,i})_{i\in[N_{s,a}]} w s , a = ( w s , a , i ) i ∈ [ N s , a ] collects the noise draws of the agents in bin ( s , a ) (s,a) ( s , a ) . The realized noise distribution for the state-action pair ( s , a ) (s,a) ( s , a ) is
ξ ^ t s , a = Ψ N s , a s , a ( w s , a ) ∈ P ( W ) , \hat{\xi}^{s,a}_t=\Psi^{s,a}_{N_{s,a}}(w^{s,a})\in\mathcal{P}(W), ξ ^ t s , a = Ψ N s , a s , a ( w s , a ) ∈ P ( W ) ,
with distribution
η t s , a = Ψ N s , a # s , a ( ⨂ i ∈ [ N s , a ] ξ t ( ⋅ ∣ s , a ) ) ∈ P ( P ( W ) ) . \eta^{s,a}_t=\Psi^{s,a}_{N_{s,a}\#}\left(\bigotimes_{i\in[N_{s,a}]}\xi_t(\cdot\mid s,a)\right)\in\mathcal{P}(\mathcal{P}(W)). η t s , a = Ψ N s , a # s , a i ∈ [ N s , a ] ⨂ ξ t ( ⋅ ∣ s , a ) ∈ P ( P ( W )) .
Since supp λ t \operatorname{supp}\lambda_t supp λ t is finite and noise realizations are independent across state-action pairs, the joint law is the finite product
η t = ⨂ ( s , a ) ∈ supp λ t η t s , a ∈ P ( P ( W ) supp λ t ) . \eta_t=\bigotimes_{(s,a)\in\operatorname{supp}\lambda_t}\eta^{s,a}_t\in\mathcal{P}\left(\mathcal{P}(W)^{\operatorname{supp}\lambda_t}\right). η t = ( s , a ) ∈ supp λ t ⨂ η t s , a ∈ P ( P ( W ) supp λ t ) .
We extend η t \eta_t η t to all state-action pairs by setting, for ( s , a ) ∉ supp λ t (s,a)\notin\operatorname{supp}\lambda_t ( s , a ) ∈ / supp λ t , ξ ^ t s , a = δ w 0 \hat{\xi}^{s,a}_t=\delta_{w^0} ξ ^ t s , a = δ w 0 , for an arbitrarily chosen w 0 ∈ W w^0\in W w 0 ∈ W . Clearly, the specific choice of w 0 w^0 w 0 is inconsequential: since λ t \lambda_t λ t assigns zero mass to these bins, they do not contribute to λ t \lambda_t λ t -weighted integrals, which are exactly the type of integrals in our problem setting. The map ( s , a ) ↦ η t s , a (s,a)\mapsto\eta^{s,a}_t ( s , a ) ↦ η t s , a is Borel measurable: it takes finitely many distinct values (one per element of supp λ t \operatorname{supp}\lambda_t supp λ t , plus the constant extension outside), and the preimage of each value is a finite union of singletons or the complement of supp λ t \operatorname{supp}\lambda_t supp λ t , all of which are Borel sets.
Example A.5 (Individual noise, infinitely many agents). In the limit case of infinitely many agents, the two-level randomization (each agent drawing a private noise sample, and the collection of noise realizations across the population) collapses into a single draw from the deterministic coupling
ρ t = λ t ⊗ ξ t ( ⋅ ∣ s , a ) . \rho_t=\lambda_t\otimes\xi_t(\cdot\mid s,a). ρ t = λ t ⊗ ξ t ( ⋅ ∣ s , a ) .
Intuitively, the population is so numerous that its empirical profile of noise realizations matches the underlying noise kernel itself. Rigorously, in the case of finite state and action spaces, we can assume that the number of agents in each state-action pair ( s , a ) (s,a) ( s , a ) grows to infinity and Varadarajan’s theorem [41 Theorem 11.4.1.] in the space P ( W ) \mathcal{P}(W) P ( W ) guarantees
ξ ^ t s , a → a . s . ξ t ( ⋅ ∣ s , a ) , as N s , a → ∞ . \hat{\xi}^{s,a}_t\xrightarrow{\mathrm{a.s.}}\xi_t(\cdot\mid s,a),\qquad\text{as }N_{s,a}\to\infty. ξ ^ t s , a a.s. ξ t ( ⋅ ∣ s , a ) , as N s , a → ∞.
Consequently, the laws of the realized kernels η t s , a \eta^{s,a}_t η t s , a converge weakly to δ ξ t ( ⋅ ∣ s , a ) \delta_{\xi_t(\cdot\mid s,a)} δ ξ t ( ⋅ ∣ s , a ) [41 Theorem 9.2.1, Proposition 9.3.5] . We therefore set ξ ^ t s , a = ξ t ( ⋅ ∣ s , a ) \hat{\xi}^{s,a}_t=\xi_t(\cdot\mid s,a) ξ ^ t s , a = ξ t ( ⋅ ∣ s , a ) almost surely and
η t s , a = δ ξ t ( ⋅ ∣ s , a ) ∈ P ( P ( W ) ) . \eta^{s,a}_t=\delta_{\xi_t(\cdot\mid s,a)}\in\mathcal{P}(\mathcal{P}(W)). η t s , a = δ ξ t ( ⋅ ∣ s , a ) ∈ P ( P ( W )) .
For general (possibly uncountable) Polish spaces S S S and A A A , we construct the joint law via Kolmogorov’s extension theorem [42 Theorems 36.1 and 36.2] . Specifically, for each finite subset J ⊂ S × A J\subset S\times A J ⊂ S × A , define a probability measure
η t J = ⨂ ( s , a ) ∈ J δ ξ t ( ⋅ ∣ s , a ) ∈ P ( P ( W ) J ) . \eta^J_t=\bigotimes_{(s,a)\in J}\delta_{\xi_t(\cdot\mid s,a)}\in\mathcal{P}(\mathcal{P}(W)^J). η t J = ( s , a ) ∈ J ⨂ δ ξ t ( ⋅ ∣ s , a ) ∈ P ( P ( W ) J ) .
These finite-dimensional marginals are consistent (the marginal of η t J \eta^J_t η t J onto any J ′ ⊂ J J'\subset J J ′ ⊂ J equals η t J ′ \eta^{J'}_t η t J ′ , since products of Dirac masses marginalize correctly). Thus, Kolmogorov’s extension theorem guarantees that there exists a unique probability measure
η t = ⨂ ( s , a ) ∈ S × A δ ξ t ( ⋅ ∣ s , a ) , \eta_t=\bigotimes_{(s,a)\in S\times A}\delta_{\xi_t(\cdot\mid s,a)}, η t = ( s , a ) ∈ S × A ⨂ δ ξ t ( ⋅ ∣ s , a ) ,
and the realized joint state-action-noise distribution collapses to the deterministic coupling ρ t = λ t ⊗ ξ t ( ⋅ ∣ s , a ) \rho_t=\lambda_t\otimes\xi_t(\cdot\mid s,a) ρ t = λ t ⊗ ξ t ( ⋅ ∣ s , a ) .
:::
Example A.6 (Local noise). Suppose the noise realizations are shared among agents with the same state-action pair. For each fixed ( s , a ) ∈ S × A (s,a) \in S \times A ( s , a ) ∈ S × A we draw
w t s , a ∼ ξ t ( ⋅ ∣ s , a ) , w_t^{s,a} \sim\xi_t(\cdot\mid s,a), w t s , a ∼ ξ t ( ⋅ ∣ s , a ) , independently across state-action pairs. Then, all agents in bin ( s , a ) (s,a) ( s , a ) experience the same noise realization w t s , a = w t s , a w_t^{s,a} = w_t^{s,a} w t s , a = w t s , a . In this framework, we define the empirical measure map
Ψ s , a : W → P ( W ) \Psi^{s,a}: W \to\mathcal{P}(W) Ψ s , a : W → P ( W ) w t s , a ↦ δ w t s , a . w_t^{s,a} \mapsto\delta_{w_t^{s,a}}. w t s , a ↦ δ w t s , a . The realized noise distribution for bin ( s , a ) (s,a) ( s , a ) collapses to
ξ ^ t s , a = δ w t s , a , \hat{\xi}_t^{s,a} = \delta_{w_t^{s,a}}, ξ ^ t s , a = δ w t s , a , with distribution
η t s , a = Ψ # s , a ξ t ( ⋅ ∣ s , a ) ∈ P ( P ( W ) ) . \eta_t^{s,a} = \Psi_{\#}^{s,a}\xi_t(\cdot\mid s,a) \in\mathcal{P}(\mathcal{P}(W)). η t s , a = Ψ # s , a ξ t ( ⋅ ∣ s , a ) ∈ P ( P ( W )) . Since noise samples are independent across state-action pairs, the joint law is constructed by the same Kolmogorov extension argument as in Example A.5 (applied with η t s , a \eta_t^{s,a} η t s , a in place of δ ξ t ( ⋅ ∣ s , a ) \delta_{\xi_t(\cdot\mid s,a)} δ ξ t ( ⋅ ∣ s , a ) ). For every finite J ⊂ S × A J \subset S \times A J ⊂ S × A , the marginal η t J = ⨂ ( s , a ) ∈ J η t s , a \eta_t^J = \bigotimes_{(s,a)\in J}\eta_t^{s,a} η t J = ⨂ ( s , a ) ∈ J η t s , a is consistent, yielding the unique extension1
η t = ⨂ ( s , a ) ∈ S × A η t s , a ∈ P ( P ( W ) S × A ) . \eta_t = \bigotimes_{(s,a)\in S\times A}\eta_t^{s,a} \in\mathcal{P}\left(\mathcal{P}(W)^{S\times A}\right). η t = ( s , a ) ∈ S × A ⨂ η t s , a ∈ P ( P ( W ) S × A ) .
Example A.7 (Global noise). Consider the case in which all agents are subject to the same noise draw w t ∼ ξ t ∈ P ( W ) w_t \sim\xi_t \in\mathcal{P}(W) w t ∼ ξ t ∈ P ( W ) , regardless of their state and action. Since every state-action pair ( s , a ) (s,a) ( s , a ) experiences the same noise realization, we have ξ ^ t s , a = δ w t \hat{\xi}_t^{s,a} = \delta_{w_t} ξ ^ t s , a = δ w t for all ( s , a ) (s,a) ( s , a ) . We then define the diagonal embedding
Φ : W → P ( W ) S × A \Phi: W \to\mathcal{P}(W)^{S\times A} Φ : W → P ( W ) S × A w ↦ ( ( s , a ) ↦ δ w ) . w \mapsto((s,a) \mapsto\delta_w). w ↦ (( s , a ) ↦ δ w ) . The entire random family ( ξ ^ t s , a ) ( s , a ) ∈ S × A = Φ ( w t ) \left(\hat{\xi}_t^{s,a}\right)_{(s,a)\in S\times A} = \Phi(w_t) ( ξ ^ t s , a ) ( s , a ) ∈ S × A = Φ ( w t ) is a P ( W ) S × A \mathcal{P}(W)^{S\times A} P ( W ) S × A -valued random variable and its law is
η t = Φ # ξ t ∈ P ( P ( W ) S × A ) . \eta_t = \Phi_{\#}\xi_t \in\mathcal{P}\left(\mathcal{P}(W)^{S\times A}\right). η t = Φ # ξ t ∈ P ( P ( W ) S × A ) . In words, ξ t ∈ P ( W ) \xi_t \in\mathcal{P}(W) ξ t ∈ P ( W ) is pushed forward through Φ \Phi Φ to a measure on P ( W ) S × A \mathcal{P}(W)^{S\times A} P ( W ) S × A , concentrated on the diagonal { Φ ( w ) : w ∈ W } \{\Phi(w): w \in W\} { Φ ( w ) : w ∈ W } . We refer the reader to Remark A.8 for a discussion on the well-posedness of Φ \Phi Φ .
instead, S × A S \times A S × A is uncountable, then P ( W ) S × A \mathcal{P}(W)^{S \times A} P ( W ) S × A might fail to be Polish. However, this does not pose issues as the image of Φ \Phi Φ is contained in the diagonal subset
D = { Φ ( w ) : w ∈ W } = { ( ( s , a ) ↦ δ w ) : w ∈ W } ⊂ P ( W ) S × A . \mathcal{D} = \{\Phi(w): w \in W\} = \{((s,a) \mapsto\delta_w): w \in W\} \subset\mathcal{P}(W)^{S \times A}. D = { Φ ( w ) : w ∈ W } = {(( s , a ) ↦ δ w ) : w ∈ W } ⊂ P ( W ) S × A .
The map Φ : W → D \Phi: W \to\mathcal{D} Φ : W → D is a homeomorphism: both Φ \Phi Φ and its inverse Φ − 1 : D → W \Phi^{-1}: \mathcal{D} \to W Φ − 1 : D → W (which evaluates any coordinate and then inverts w ↦ δ w w \mapsto\delta_w w ↦ δ w ) are continuous. Hence, D \mathcal{D} D is homeomorphic to W W W , which is Polish. In particular, D \mathcal{D} D is a Polish space in the subspace topology inherited from P ( W ) S × A \mathcal{P}(W)^{S \times A} P ( W ) S × A . Since Φ : W → D \Phi: W \to\mathcal{D} Φ : W → D is continuous between Polish spaces it is Borel measurable, and η t = Φ # ξ t \eta_t = \Phi_{\#}\xi_t η t = Φ # ξ t is a well-defined Borel probability measure on D \mathcal{D} D and so also on P ( W ) S × A \mathcal{P}(W)^{S \times A} P ( W ) S × A . We finally observe that η t \eta_t η t is fully concentrated on D \mathcal{D} D , so all subsequent integrals only involve elements of D \mathcal{D} D and the uncountability of S × A S \times A S × A does not pose an issue.
We now show that these canonical noise models satisfy Assumption A.2.
Proposition A.9. The noise models satisfy Assumption A.2. For each of the noise models in Examples A.4–A.7, the realized kernel ξ ^ t s , a \hat{\xi}_t^{s,a} ξ ^ t s , a and its distribution η t \eta_t η t satisfy Assumption A.2. That is, there exists a Borel set E t ⊆ S × A E_t \subseteq S \times A E t ⊆ S × A with λ t ( E t ) = 1 \lambda_t(E_t) = 1 λ t ( E t ) = 1 such that for all ( s , a ) ∈ E t (s,a) \in E_t ( s , a ) ∈ E t and for every bounded Borel measurable φ : W → R \varphi: W \to\mathbb{R} φ : W → R simultaneously,
E η t s , a [ ∫ W φ ( w ) d ξ ^ t s , a ( w ) ] = ∫ W φ ( w ) d ξ t ( w ∣ s , a ) . \mathbb{E}_{\eta_t^{s,a}}\left[\int_W \varphi(w)\mathrm{d}\hat{\xi}_t^{s,a}(w)\right] = \int_W \varphi(w)\mathrm{d}\xi_t(w \mid s,a). E η t s , a [ ∫ W φ ( w ) d ξ ^ t s , a ( w ) ] = ∫ W φ ( w ) d ξ t ( w ∣ s , a ) .
Proof. We prove the statement for each noise model separately.
Individual noise (Example A.4). Fix ( s , a ) ∈ supp λ t (s,a) \in\operatorname{supp}\lambda_t ( s , a ) ∈ supp λ t , hence N s , a ≥ 1 N_{s,a} \ge1 N s , a ≥ 1 . By construction,
ξ ^ t s , a = 1 N s , a ∑ i = 1 N s , a δ w t s , a , i . \hat{\xi}_t^{s,a} = \frac{1}{N_{s,a}}\sum_{i=1}^{N_{s,a}}\delta_{w_t^{s,a,i}}. ξ ^ t s , a = N s , a 1 i = 1 ∑ N s , a δ w t s , a , i .
where w t s , a , i w_t^{s,a,i} w t s , a , i are i.i.d. with law ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) . Hence, for any bounded measurable φ : W → R \varphi: W \to\mathbb{R} φ : W → R ,
∫ W φ ( w ) d ξ ^ t s , a ( w ) = 1 N s , a ∑ i = 1 N s , a φ ( w t s , a , i ) . \int_W \varphi(w)\mathrm{d}\hat{\xi}_t^{s,a}(w) = \frac{1}{N_{s,a}}\sum_{i=1}^{N_{s,a}}\varphi(w_t^{s,a,i}). ∫ W φ ( w ) d ξ ^ t s , a ( w ) = N s , a 1 i = 1 ∑ N s , a φ ( w t s , a , i ) .
and therefore, by linearity of expectation and since each w t s , a , i w_t^{s,a,i} w t s , a , i has law ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) ,
E η t s , a [ ∫ W φ ( w ) d ξ ^ t s , a ( w ) ] = 1 N s , a ∑ i = 1 N s , a E ξ t ( ⋅ ∣ s , a ) [ φ ( w ) ] = ∫ W φ ( w ) d ξ t ( w ∣ s , a ) . \mathbb{E}_{\eta_t^{s,a}}\left[\int_W \varphi(w)\mathrm{d}\hat{\xi}_t^{s,a}(w)\right] = \frac{1}{N_{s,a}}\sum_{i=1}^{N_{s,a}}\mathbb{E}_{\xi_t(\cdot\mid s,a)}[\varphi(w)] = \int_W \varphi(w)\mathrm{d}\xi_t(w \mid s,a). E η t s , a [ ∫ W φ ( w ) d ξ ^ t s , a ( w ) ] = N s , a 1 i = 1 ∑ N s , a E ξ t ( ⋅ ∣ s , a ) [ φ ( w )] = ∫ W φ ( w ) d ξ t ( w ∣ s , a ) .
Thus (7) holds on E t = supp λ t E_t = \operatorname{supp}\lambda_t E t = supp λ t , which clearly satisfies λ t ( E t ) = 1 \lambda_t(E_t) = 1 λ t ( E t ) = 1 .
Infinite-population model (Example A.5). By construction ξ ^ t s , a = ξ t ( ⋅ ∣ s , a ) \hat{\xi}_t^{s,a} = \xi_t(\cdot\mid s,a) ξ ^ t s , a = ξ t ( ⋅ ∣ s , a ) almost surely for every ( s , a ) ∈ S × A (s,a) \in S \times A ( s , a ) ∈ S × A . Therefore, for every bounded measurable φ \varphi φ
E η t s , a [ ∫ W φ ( w ) d ξ ^ t s , a ( w ) ] = ∫ W φ ( w ) d ξ t ( w ∣ s , a ) . \mathbb{E}_{\eta_t^{s,a}}\left[\int_W \varphi(w)\mathrm{d}\hat{\xi}_t^{s,a}(w)\right] = \int_W \varphi(w)\mathrm{d}\xi_t(w \mid s,a). E η t s , a [ ∫ W φ ( w ) d ξ ^ t s , a ( w ) ] = ∫ W φ ( w ) d ξ t ( w ∣ s , a ) .
Thus, (7) holds with E t = S × A E_t = S \times A E t = S × A .
Local noise (Example A.6). For each fixed ( s , a ) (s,a) ( s , a ) , a single draw w t s , a ∼ ξ t ( ⋅ ∣ s , a ) w_t^{s,a} \sim\xi_t(\cdot\mid s,a) w t s , a ∼ ξ t ( ⋅ ∣ s , a ) is taken, independently across state-action pairs, and the realized noise distribution is ξ ^ t s , a = δ w t s , a \hat{\xi}_t^{s,a} = \delta_{w_t^{s,a}} ξ ^ t s , a = δ w t s , a . So, for every bounded measurable φ \varphi φ , the inner integral equals φ ( w t s , a ) \varphi(w_t^{s,a}) φ ( w t s , a ) and, since w t s , a ∼ ξ t ( ⋅ ∣ s , a ) w_t^{s,a} \sim\xi_t(\cdot\mid s,a) w t s , a ∼ ξ t ( ⋅ ∣ s , a ) ,
E η t s , a [ ∫ W φ ( w ) d ξ ^ t s , a ( w ) ] = E ξ t ( ⋅ ∣ s , a ) [ φ ( w ) ] = ∫ W φ ( w ) d ξ t ( w ∣ s , a ) . \mathbb{E}_{\eta_t^{s,a}}\left[\int_W \varphi(w)\mathrm{d}\hat{\xi}_t^{s,a}(w)\right] = \mathbb{E}_{\xi_t(\cdot\mid s,a)}[\varphi(w)] = \int_W \varphi(w)\mathrm{d}\xi_t(w \mid s,a). E η t s , a [ ∫ W φ ( w ) d ξ ^ t s , a ( w ) ] = E ξ t ( ⋅ ∣ s , a ) [ φ ( w )] = ∫ W φ ( w ) d ξ t ( w ∣ s , a ) .
Thus, (7) holds again with E t = S × A E_t = S \times A E t = S × A .
Global-noise (Example A.7). This case is identical to the local noise one, with the same draw w t w_t w t shared across all ( s , a ) (s,a) ( s , a ) .
Averaging property of realized noise kernels
We now show that even if ξ ^ t s , a \widehat{\xi}^{s,a}_t ξ t s , a is random, the expected cost computed under its law η t s , a \eta^{s,a}_t η t s , a equals the cost computed directly under the nominal kernel ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) . In other words, replacing the realized kernel ξ ^ t s , a \widehat{\xi}^{s,a}_t ξ t s , a by the nominal kernel ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) is unbiased in expectation. In particular, this lemma will allow us to remove randomness from the backward recursion in the proof of Theorem 3.1, Part I.
Lemma A.10 (Averaging realized kernels). Let S S S , A A A , W W W be Polish spaces and fix a time step t t t . Let λ t \lambda_t λ t denote the joint state-action distribution and ξ t : S × A → P ( W ) \xi_t:S \times A \to\mathcal{P}(W) ξ t : S × A → P ( W ) be a Borel stochastic kernel. For each ( s , a ) (s,a) ( s , a ) , let ξ ^ t s , a \widehat{\xi}^{s,a}_t ξ t s , a be a P ( W ) \mathcal{P}(W) P ( W ) -valued random variable with distribution η t s , a ∈ P ( P ( W ) ) \eta^{s,a}_t \in\mathcal{P}(\mathcal{P}(W)) η t s , a ∈ P ( P ( W )) , such that Assumption A.2 holds and ( s , a ) ↦ η t s , a (s,a) \mapsto\eta^{s,a}_t ( s , a ) ↦ η t s , a is a Borel stochastic kernel. Then, for every nonnegative lower semianalytic c : S × A × W → R ‾ ≥ 0 c:S \times A \times W \to\overline{\mathbb{R}}_{\ge0} c : S × A × W → R ≥ 0 (in particular, for every nonnegative Borel measurable c c c ),
∫ S × A E η t s , a [ ∫ W c ( s , a , w ) d ξ ^ t s , a ( w ) ] d λ t ( s , a ) = ∫ S × A ∫ W c ( s , a , w ) d ξ t ( w ∣ s , a ) d λ t ( s , a ) . (8) \int_{S \times A} \mathbb{E}_{\eta^{s,a}_t}\left[\int_W c(s,a,w)d\widehat{\xi}^{s,a}_t(w)\right]d\lambda_t(s,a)=\int_{S \times A}\int_W c(s,a,w)d\xi_t(w\mid s,a)d\lambda_t(s,a). \tag*{(8)} ∫ S × A E η t s , a [ ∫ W c ( s , a , w ) d ξ t s , a ( w ) ] d λ t ( s , a ) = ∫ S × A ∫ W c ( s , a , w ) d ξ t ( w ∣ s , a ) d λ t ( s , a ) . (8)
Proof. For n ∈ N n \in\mathbb{N} n ∈ N , define
c n ( s , a , w ) : = min { c ( s , a , w ) , n } . c_n(s,a,w):=\min\{c(s,a,w),n\}. c n ( s , a , w ) := min { c ( s , a , w ) , n } .
By construction, each c n c_n c n is bounded and lower semianalytic, and c n ↑ c c_n \uparrow c c n ↑ c pointwise.
To start, we prove that the two integrands in (8) are (universally) measurable. For the left-hand side, for each n n n , the map ( s , a , ζ ) ↦ ∫ W c n ( s , a , w ) d ζ ( w ) (s,a,\zeta) \mapsto\int_W c_n(s,a,w)d\zeta(w) ( s , a , ζ ) ↦ ∫ W c n ( s , a , w ) d ζ ( w ) is lower semianalytic on S × A × P ( W ) S \times A \times\mathcal{P}(W) S × A × P ( W ) : the evaluation map q ( d w ∣ s , a , ζ ) : = ζ ( d w ) q(dw\mid s,a,\zeta):=\zeta(dw) q ( d w ∣ s , a , ζ ) := ζ ( d w ) is a Borel stochastic kernel (since ζ ↦ ζ ( B ) \zeta\mapsto\zeta(B) ζ ↦ ζ ( B ) is Borel measurable for every Borel set B ⊆ W B \subseteq W B ⊆ W [31 Proposition 7.28] ), and integration of a lower semianalytic function against a Borel stochastic kernel preserves lower semianalyticity [31 Proposition 7.48] . Since ( s , a ) ↦ η t s , a (s,a) \mapsto\eta^{s,a}_t ( s , a ) ↦ η t s , a is a Borel stochastic kernel, a second application of [31 Proposition 7.48] yields lower semianalyticity, hence universal measurability, of
( s , a ) ↦ E η t s , a [ ∫ W c n ( s , a , w ) d ξ ^ t s , a ( w ) ] . (s,a) \mapsto\mathbb{E}_{\eta^{s,a}_t}\left[\int_W c_n(s,a,w)d\widehat{\xi}^{s,a}_t(w)\right]. ( s , a ) ↦ E η t s , a [ ∫ W c n ( s , a , w ) d ξ t s , a ( w ) ] .
Letting n ↑ ∞ n \uparrow\infty n ↑ ∞ , and using that a pointwise increasing limit of universally measurable functions is universally measurable, we obtain the measurability of
( s , a ) ↦ E η t s , a [ ∫ W c ( s , a , w ) d ξ ^ t s , a ( w ) ] . (s,a) \mapsto\mathbb{E}_{\eta^{s,a}_t}\left[\int_W c(s,a,w)d\widehat{\xi}^{s,a}_t(w)\right]. ( s , a ) ↦ E η t s , a [ ∫ W c ( s , a , w ) d ξ t s , a ( w ) ] .
For the right-hand side, instead, the measurability of
( s , a ) ↦ ∫ W c ( s , a , w ) d ξ t ( w ∣ s , a ) , (s,a) \mapsto\int_W c(s,a,w)d\xi_t(w\mid s,a), ( s , a ) ↦ ∫ W c ( s , a , w ) d ξ t ( w ∣ s , a ) ,
follows from the same argument, applied with the Borel stochastic kernel ξ t \xi_t ξ t [31 Proposition 7.48] .
We now prove the equality (8) . Fix n n n and ( s , a ) ∈ E t (s,a) \in E_t ( s , a ) ∈ E t , and consider the bounded test function φ s , a ( w ) : = c n ( s , a , w ) \varphi_{s,a}(w):=c_n(s,a,w) φ s , a ( w ) := c n ( s , a , w ) , viewed as a function of w w w alone for fixed ( s , a ) (s,a) ( s , a ) . Assumption A.2 is stated for bounded Borel test functions, while φ s , a \varphi_{s,a} φ s , a is only lower semianalytic, hence universally measurable; we extend (7) to φ s , a \varphi_{s,a} φ s , a as follows. Let m : = ξ t ( ⋅ ∣ s , a ) m:=\xi_t(\cdot\mid s,a) m := ξ t ( ⋅ ∣ s , a ) . Since φ s , a \varphi_{s,a} φ s , a is universally measurable, it is measurable with respect to the m m m -completion of the Borel σ \sigma σ -algebra, so there exist a bounded Borel function φ ′ \varphi' φ ′ and a Borel set N ⊆ W \mathcal{N}\subseteq W N ⊆ W with m ( N ) = 0 m(\mathcal{N})=0 m ( N ) = 0 and { φ s , a ≠ φ ′ } ⊆ N \{\varphi_{s,a}\ne\varphi'\}\subseteq\mathcal{N} { φ s , a = φ ′ } ⊆ N . Applying (7) to the bounded Borel function 1 N \mathbf{1}_{\mathcal{N}} 1 N gives E η t s , a [ ξ ^ t s , a ( N ) ] = m ( N ) = 0 \mathbb{E}_{\eta^{s,a}_t}[\widehat{\xi}^{s,a}_t(\mathcal{N})]=m(\mathcal{N})=0 E η t s , a [ ξ t s , a ( N )] = m ( N ) = 0 , and since ξ ^ t s , a ( N ) ≥ 0 \widehat{\xi}^{s,a}_t(\mathcal{N})\ge0 ξ t s , a ( N ) ≥ 0 , we conclude ξ ^ t s , a ( N ) = 0 \widehat{\xi}^{s,a}_t(\mathcal{N})=0 ξ t s , a ( N ) = 0 almost surely. Consequently, ∫ φ s , a d ξ ^ t s , a = ∫ φ ′ d ξ ^ t s , a \int\varphi_{s,a}d\widehat{\xi}^{s,a}_t=\int\varphi'd\widehat{\xi}^{s,a}_t ∫ φ s , a d ξ t s , a = ∫ φ ′ d ξ t s , a almost surely and ∫ φ s , a d m = ∫ φ ′ d m \int\varphi_{s,a}dm=\int\varphi'dm ∫ φ s , a d m = ∫ φ ′ d m ; applying (7) to φ ′ \varphi' φ ′ then yields (7) for φ s , a \varphi_{s,a} φ s , a itself. This gives
E η t s , a [ ∫ W c n ( s , a , w ) d ξ ^ t s , a ( w ) ] = ∫ W c n ( s , a , w ) d ξ t ( w ∣ s , a ) \mathbb{E}_{\eta^{s,a}_t}\left[\int_W c_n(s,a,w)d\widehat{\xi}^{s,a}_t(w)\right]=\int_W c_n(s,a,w)d\xi_t(w\mid s,a) E η t s , a [ ∫ W c n ( s , a , w ) d ξ t s , a ( w ) ] = ∫ W c n ( s , a , w ) d ξ t ( w ∣ s , a )
for all ( s , a ) ∈ E t (s,a)\in E_t ( s , a ) ∈ E t . Since λ t ( E t ) = 1 \lambda_t(E_t)=1 λ t ( E t ) = 1 , integrating both sides with respect to λ t \lambda_t λ t gives
∫ S × A E η t s , a [ ∫ W c n ( s , a , w ) d ξ ^ t s , a ( w ) ] d λ t ( s , a ) = ∫ S × A ∫ W c n ( s , a , w ) d ξ t ( w ∣ s , a ) d λ t ( s , a ) . \int_{S \times A}\mathbb{E}_{\eta^{s,a}_t}\left[\int_W c_n(s,a,w)d\widehat{\xi}^{s,a}_t(w)\right]d\lambda_t(s,a)=\int_{S \times A}\int_W c_n(s,a,w)d\xi_t(w\mid s,a)d\lambda_t(s,a). ∫ S × A E η t s , a [ ∫ W c n ( s , a , w ) d ξ t s , a ( w ) ] d λ t ( s , a ) = ∫ S × A ∫ W c n ( s , a , w ) d ξ t ( w ∣ s , a ) d λ t ( s , a ) .
Finally, let n → ∞ n\to\infty n → ∞ . Since c n ↑ c c_n\uparrow c c n ↑ c , the monotone convergence theorem yields (8). □ \square □
Preliminary lemmata
We now collect the auxiliary results required for the main proof in Appendix § 11 .
Measurability convention. Throughout the appendices, primitive data such as the dynamics, stage costs, and terminal cost are assumed to be Borel measurable, as in the main text. Since every nonnegative Borel measurable function is lower semianalytic, these data also satisfy the analytic-measurability assumptions used in the auxiliary results below. We use lower semianalytic functions because the dynamic programming recursion involves pointwise infima over the action space, and such infima need not be Borel measurable on general Polish spaces; see [31 ] for a detailed discussion. Accordingly, value functions and pointwise Bellman infima are interpreted as lower semianalytic, hence universally measurable, functions. This is sufficient for the integrals and measurable-selection arguments used in the proof of Theorem 3.1. For this reason, the new auxiliary lemmata below (Lemmas B.3, B.5, and B.6) state their regularity requirements explicitly in terms of lower semianalytic costs: this is the regularity actually available when they are invoked (the costs there involve the value functions j t j_t j t ). The results recalled from [27 ] are stated and proved there under the standing assumption that all spaces are Polish and all maps are Borel; their proofs (change-of-variables, gluing, and disintegration arguments) apply verbatim when the transportation cost is only lower semianalytic, hence universally measurable (all integrals remain well-defined by nonnegativity), and we use them in this form. We work at this level of generality to simultaneously capture systems with finite and continuous state and action spaces. Nevertheless, when all spaces are finite, measurability is automatic and all measurability arguments can therefore be skipped.
We begin by recalling from [27 ] a result that lets us “move” the pushforward from the set of couplings into the transportation cost, and vice versa.
Lemma B.1 (Pushforward and optimal transport [27 Lemma 6.1] ). Given a transportation cost j t + 1 : S × Z → R ‾ ≥ 0 j_{t+1}: S \times Z \to\overline{\mathbb{R}}_{\ge0} j t + 1 : S × Z → R ≥ 0 , a map f t : S × A × W → S f_t: S \times A \times W \to S f t : S × A × W → S , and a probability measure ρ t ∈ P ( S × A × W ) \rho_t \in\mathcal{P}(S \times A \times W) ρ t ∈ P ( S × A × W ) , it holds that
inf γ : ( proj S × A × W S × A × W × Z ) # γ = ρ t ∫ S × A × W × Z j t + 1 ∘ ( f t × Id Z ) d γ = inf γ ~ : ( proj S S × Z ) # γ ~ = f t # ρ t ∫ S × Z j t + 1 d γ ~ , \inf_{\substack{\gamma\colon(\operatorname{proj}_{S \times A \times W}^{S \times A \times W \times Z})_{\#}\gamma=\rho_t}} \int_{S \times A \times W \times Z} j_{t+1}\circ(f_t\times\operatorname{Id}_Z)\,\mathrm{d}\gamma
=
\inf_{\substack{\widetilde{\gamma} \colon(\operatorname{proj}_{S}^{S \times Z})_{\#}\widetilde{\gamma}=f_{t\#}\rho_t}} \int_{S \times Z}j_{t+1}\,\mathrm{d}\widetilde{\gamma}, γ : ( proj S × A × W S × A × W × Z ) # γ = ρ t inf ∫ S × A × W × Z j t + 1 ∘ ( f t × Id Z ) d γ = γ : ( proj S S × Z ) # γ = f t # ρ t inf ∫ S × Z j t + 1 d γ ,
where Id Z \operatorname{Id}_Z Id Z is the identity map over Z Z Z .
In our setting, this lemma tells us that evaluating the cost after applying the system dynamics is equivalent to pushing forward the distribution first: we can “apply the dynamics first” and then evaluate the population-level cost without changing the result.
Next, we recall how to merge additive costs into a single transport objective.
Lemma B.2 (Sum of optimal transport discrepancies [27 Lemma 6.2] ). Given transportation costs c 1 : X → R ‾ ≥ 0 c_1:X\to\overline{\mathbb{R}}_{\ge0} c 1 : X → R ≥ 0 , c 2 : X × Z → R ‾ ≥ 0 c_2:X\times Z\to\overline{\mathbb{R}}_{\ge0} c 2 : X × Z → R ≥ 0 , and probability measures ρ ∈ P ( X ) \rho\in\mathcal{P}(X) ρ ∈ P ( X ) , ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) , it holds that
E ρ [ c 1 ] + K [ c 2 ] ( ρ , ν ) = K [ c ] ( ρ , ν ) , \mathbb{E}_{\rho}[c_1]+\mathcal{K}[c_2](\rho,\nu)=\mathcal{K}[c](\rho,\nu), E ρ [ c 1 ] + K [ c 2 ] ( ρ , ν ) = K [ c ] ( ρ , ν ) ,
with c : X × Z → R ‾ ≥ 0 c:X\times Z\to\overline{\mathbb{R}}_{\ge0} c : X × Z → R ≥ 0 defined as c ( x , z ) = c 1 ( x ) + c 2 ( x , z ) c(x,z)=c_1(x)+c_2(x,z) c ( x , z ) = c 1 ( x ) + c 2 ( x , z ) .
This result allows us to collapse stage costs and terminal transport costs into a unified expression, which is crucial for dynamic programming formulations.
We now introduce stochasticity which, in our setting, will appear due to the stochastic dynamics. Through a standard argument, we show that we can safely exchange expectation over random noise with the infimum over couplings, at the price of an inequality.
Lemma B.3 (Infimum and expectation). *Let X X X , W W W , Z Z Z be Polish spaces, λ ∈ P ( X ) \lambda\in\mathcal{P}(X) λ ∈ P ( X ) , and ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) . Let ξ ^ = ( ξ ^ x ) x ∈ X \hat{\xi}=(\hat{\xi}^{x})_{x\in X} ξ ^ = ( ξ ^ x ) x ∈ X be a random family of probability measures ξ ^ ∈ P ( W ) \hat{\xi}\in\mathcal{P}(W) ξ ^ ∈ P ( W ) , defined on a common probability space such that, almost surely, x ↦ ξ ^ x x\mapsto\hat{\xi}^{x} x ↦ ξ ^ x is Borel measurable kernel on a set of full λ \lambda λ -measure (cf. Assumption A.1). Denote its distribution by η \eta η . Let c : X × W × Z → R ‾ ≥ 0 c:X\times W\times Z\to\overline{\mathbb{R}}_{\ge0} c : X × W × Z → R ≥ 0 be measurable. Whenever the integrand of the left-hand side below fails to be measurable in ξ ^ \hat{\xi} ξ ^ , the outer expectation is understood; in our applications, it is universally measurable (see the proof of Theorem 3.1). Then
E ξ ^ [ inf γ ∈ Γ ( λ , ν ) ∫ X × Z ∫ W c ( x , w , z ) d ξ ^ x ( w ) d γ ( x , z ) ] ≤ inf γ ∈ Γ ( λ , ν ) E ξ ^ [ ∫ X × Z ∫ W c ( x , w , z ) d ξ ^ x ( w ) d γ ( x , z ) ] . \mathbb{E}_{\hat{\xi}}\left[\inf_{\gamma\in\Gamma(\lambda,\nu)}\int_{X\times Z}\int_W c(x,w,z)\,\mathrm{d}\hat{\xi}^{x}(w)\mathrm{d}\gamma(x,z)\right]
\leq\inf_{\gamma\in\Gamma(\lambda,\nu)}\mathbb{E}_{\hat{\xi}}\left[\int_{X\times Z}\int_W c(x,w,z)\,\mathrm{d}\hat{\xi}^{x}(w)\mathrm{d}\gamma(x,z)\right]. E ξ ^ [ γ ∈ Γ ( λ , ν ) inf ∫ X × Z ∫ W c ( x , w , z ) d ξ ^ x ( w ) d γ ( x , z ) ] ≤ γ ∈ Γ ( λ , ν ) inf E ξ ^ [ ∫ X × Z ∫ W c ( x , w , z ) d ξ ^ x ( w ) d γ ( x , z ) ] .
Proof. Fix any γ ˉ ∈ Γ ( λ , ν ) \bar{\gamma}\in\Gamma(\lambda,\nu) γ ˉ ∈ Γ ( λ , ν ) . Pointwise in ξ ^ \hat{\xi} ξ ^ ,
inf γ ∈ Γ ( λ , ν ) ∫ X × Z ∫ W c ( x , w , z ) d ξ ^ x ( w ) d γ ( x , z ) ≤ ∫ X × Z ∫ W c ( x , w , z ) d ξ ^ x ( w ) d γ ˉ ( x , z ) . \inf_{\gamma\in\Gamma(\lambda,\nu)}\int_{X\times Z}\int_W c(x,w,z)\,\mathrm{d}\hat{\xi}^{x}(w)\mathrm{d}\gamma(x,z)
\leq\int_{X\times Z}\int_W c(x,w,z)\,\mathrm{d}\hat{\xi}^{x}(w)\mathrm{d}\bar{\gamma}(x,z). γ ∈ Γ ( λ , ν ) inf ∫ X × Z ∫ W c ( x , w , z ) d ξ ^ x ( w ) d γ ( x , z ) ≤ ∫ X × Z ∫ W c ( x , w , z ) d ξ ^ x ( w ) d γ ˉ ( x , z ) .
Taking expectations over ξ ^ \hat{\xi} ξ ^ preserves the inequality (also when the left-hand side is interpreted as an outer expectation, by its monotonicity). Since the resulting inequality holds for every coupling γ ˉ \bar{\gamma} γ ˉ , taking the infimum over γ ˉ ∈ Γ ( λ , ν ) \bar{\gamma}\in\Gamma(\lambda,\nu) γ ˉ ∈ Γ ( λ , ν ) yields the claim.
With this inequality, randomness can be “pushed inside” the transport problem, a step that will later allow us to handle noisy transitions.
The next tool, recalled from [27 ] , refines our handling of couplings: it shows how an optimizer can be disintegrated with respect to one marginal, yielding conditional distributions.
Lemma B.4 (Disintegration of the optimizer [27 Lemma 6.4] ). Given a transportation cost c : S × A × Z → R ‾ ≥ 0 c:S\times A\times Z\to\overline{\mathbb{R}}_{\geq0} c : S × A × Z → R ≥ 0 and probability measures μ ∈ P ( S ) \mu\in\mathcal{P}(S) μ ∈ P ( S ) , ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) , it holds that
inf γ ∈ Γ ( μ , ν ) ∫ S × A × Z c ( s , a , z ) d γ ( s , a , z ) = inf γ ′ ∈ Γ ( μ , ν ) inf { α s , z } ∈ Λ ( A ) ∫ S × Z ∫ A c ( s , a , z ) d α s , z ( a ) d γ ′ ( s , z ) , (9) \inf_{\gamma\in\Gamma(\mu,\nu)}\int_{S\times A\times Z}c(s,a,z)\,\mathrm{d}\gamma(s,a,z)
=
\inf_{\gamma'\in\Gamma(\mu,\nu)}\inf_{\{\alpha^{s,z}\}\in\Lambda(A)}\int_{S\times Z}\int_A c(s,a,z)\,\mathrm{d}\alpha^{s,z}(a)\mathrm{d}\gamma'(s,z), \tag*{(9)} γ ∈ Γ ( μ , ν ) inf ∫ S × A × Z c ( s , a , z ) d γ ( s , a , z ) = γ ′ ∈ Γ ( μ , ν ) inf { α s , z } ∈ Λ ( A ) inf ∫ S × Z ∫ A c ( s , a , z ) d α s , z ( a ) d γ ′ ( s , z ) , (9) where Λ ( A ) : = { { α s , z } ( s , z ) ∈ S × Z ⊆ P ( A ) ∣ S × Z ∋ ( s , z ) ↦ α s , z ∈ P ( A ) Borel } \Lambda(A):=\left\{\{\alpha^{s,z}\}_{(s,z)\in S\times Z}\subseteq\mathcal{P}(A)\mid S\times Z\ni(s,z)\mapsto\alpha^{s,z}\in\mathcal{P}(A)\text{ Borel}\right\} Λ ( A ) := { { α s , z } ( s , z ) ∈ S × Z ⊆ P ( A ) ∣ S × Z ∋ ( s , z ) ↦ α s , z ∈ P ( A ) Borel } .
This separation will allow us to express the population-level policy as a distribution over conditional agent-level actions.
Next, we study more in detail the right-hand side in (9) , showing that the infimum over the family { α s , z } ∈ Λ ( A ) \{\alpha^{s,z}\}\in\Lambda(A) { α s , z } ∈ Λ ( A ) can be “pushed” into the integral.
Lemma B.5 (Reduction to pointwise infimum over deterministic policies). Let S S S , A A A , Z Z Z be Polish spaces and let c : S × A × Z → R ‾ ≥ 0 c:S\times A\times Z\to\overline{\mathbb{R}}_{\geq0} c : S × A × Z → R ≥ 0 be lower semianalytic. Let μ ∈ P ( S ) \mu\in\mathcal{P}(S) μ ∈ P ( S ) and ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) . Define
Λ ( A ) : = { { α s , z } ( s , z ) ∈ S × Z ⊆ P ( A ) ∣ S × Z ∋ ( s , z ) ↦ α s , z ∈ P ( A ) is Borel measurable } . \Lambda(A):=\left\{\{\alpha^{s,z}\}_{(s,z)\in S\times Z}\subseteq\mathcal{P}(A)\mid S\times Z\ni(s,z)\mapsto\alpha^{s,z}\in\mathcal{P}(A)\text{ is Borel measurable}\right\}. Λ ( A ) := { { α s , z } ( s , z ) ∈ S × Z ⊆ P ( A ) ∣ S × Z ∋ ( s , z ) ↦ α s , z ∈ P ( A ) is Borel measurable } . Then, the function ( s , z ) ↦ inf a ∈ A c ( s , a , z ) (s,z)\mapsto\inf_{a\in A}c(s,a,z) ( s , z ) ↦ inf a ∈ A c ( s , a , z ) is universally measurable, and for every coupling γ ∈ Γ ( μ , ν ) \gamma\in\Gamma(\mu,\nu) γ ∈ Γ ( μ , ν ) ,
inf { α s , z } ∈ Λ ( A ) ∫ S × Z ∫ A c ( s , a , z ) d α s , z ( a ) d γ ( s , z ) = ∫ S × Z inf a ∈ A c ( s , a , z ) d γ ( s , z ) . (10) \inf_{\{\alpha^{s,z}\}\in\Lambda(A)}\int_{S\times Z}\int_A c(s,a,z)\,\mathrm{d}\alpha^{s,z}(a)\mathrm{d}\gamma(s,z)
=
\int_{S\times Z}\inf_{a\in A}c(s,a,z)\mathrm{d}\gamma(s,z). \tag*{(10)} { α s , z } ∈ Λ ( A ) inf ∫ S × Z ∫ A c ( s , a , z ) d α s , z ( a ) d γ ( s , z ) = ∫ S × Z a ∈ A inf c ( s , a , z ) d γ ( s , z ) . (10) Proof. We start by assessing the measurability of the pointwise infimum. Since c c c is lower semianalytic and A A A is a Polish space, the function ( s , z ) ↦ inf a ∈ A c ( s , a , z ) (s,z)\mapsto\inf_{a\in A}c(s,a,z) ( s , z ) ↦ inf a ∈ A c ( s , a , z ) is lower semianalytic by [31 Proposition 7.47] , hence universally measurable. In particular, the right-hand side of (10) is a well-defined element of [ 0 , + ∞ ] [0,+\infty] [ 0 , + ∞ ] .
We then prove (10) by showing “≤ \leq ≤ ” and “≥ \geq ≥ ” separately.
Lower bound “≥ \geq ≥ ”. For any { α s , z } ∈ Λ ( A ) \{\alpha^{s,z}\}\in\Lambda(A) { α s , z } ∈ Λ ( A ) and every ( s , z ) ∈ S × Z (s,z)\in S\times Z ( s , z ) ∈ S × Z , since α s , z ∈ P ( A ) \alpha^{s,z}\in\mathcal{P}(A) α s , z ∈ P ( A ) is a probability measure
∫ A c ( s , a , z ) d α s , z ( a ) ≥ inf a ∈ A c ( s , a , z ) . \int_A c(s,a,z)\,\mathrm{d}\alpha^{s,z}(a)\geq\inf_{a\in A}c(s,a,z). ∫ A c ( s , a , z ) d α s , z ( a ) ≥ a ∈ A inf c ( s , a , z ) .
Integrating both sides with respect to γ \gamma γ and taking the infimum over all { α s , z } ∈ Λ ( A ) \{\alpha^{s,z}\} \in\Lambda(A) { α s , z } ∈ Λ ( A ) leads to
inf { α s , z } ∈ Λ ( A ) ∫ S × Z ∫ A c ( s , a , z ) d α s , z ( a ) d γ ( s , z ) ≥ ∫ S × Z inf a ∈ A c ( s , a , z ) d γ ( s , z ) . \inf_{\{\alpha^{s,z}\}\in\Lambda(A)}\int_{S\times Z}\int_A c(s,a,z)d\alpha^{s,z}(a)d\gamma(s,z)\geq\int_{S\times Z}\inf_{a\in A}c(s,a,z)d\gamma(s,z). { α s , z } ∈ Λ ( A ) inf ∫ S × Z ∫ A c ( s , a , z ) d α s , z ( a ) d γ ( s , z ) ≥ ∫ S × Z a ∈ A inf c ( s , a , z ) d γ ( s , z ) .
Upper bound “≤ \leq ≤ ”. Let ε > 0 \varepsilon>0 ε > 0 . Since c ≥ 0 c\geq0 c ≥ 0 is lower semianalytic and A A A is a Polish space, [31 Proposition 7.50] guarantees the existence of a universally measurable ε \varepsilon ε -optimal selector a ε : S × Z → A a^\varepsilon:S\times Z\to A a ε : S × Z → A satisfying
c ( s , a ε ( s , z ) , z ) ≤ inf a ∈ A c ( s , a , z ) + ε ∀ ( s , z ) ∈ S × Z . c(s,a^\varepsilon(s,z),z)\leq\inf_{a\in A}c(s,a,z)+\varepsilon\qquad\forall(s,z)\in S\times Z. c ( s , a ε ( s , z ) , z ) ≤ a ∈ A inf c ( s , a , z ) + ε ∀ ( s , z ) ∈ S × Z .
Now, since γ \gamma γ is a σ \sigma σ -finite probability measure on S × Z S\times Z S × Z , by [31 Lemma 7.27] there exists a Borel measurable map b ε : S × Z → A b^\varepsilon:S\times Z\to A b ε : S × Z → A with b ε = a ε b^\varepsilon=a^\varepsilon b ε = a ε γ \gamma γ -almost everywhere, so that
c ( s , b ε ( s , z ) , z ) ≤ inf a ∈ A c ( s , a , z ) + ε γ -almost everywhere . c(s,b^\varepsilon(s,z),z)\leq\inf_{a\in A}c(s,a,z)+\varepsilon\quad\gamma\text{-almost everywhere}. c ( s , b ε ( s , z ) , z ) ≤ a ∈ A inf c ( s , a , z ) + ε γ -almost everywhere .
The Dirac embedding A → P ( A ) A\to\mathcal{P}(A) A → P ( A ) defined by a ↦ δ a a\mapsto\delta_a a ↦ δ a is continuous in the weak topology [32 Theorem 15.8] , hence Borel measurable. Therefore, the composition ( s , z ) ↦ δ b ε ( s , z ) (s,z)\mapsto\delta_{b^\varepsilon(s,z)} ( s , z ) ↦ δ b ε ( s , z ) is Borel measurable, and so is the family { δ b ε ( s , z ) } ( s , z ) ∈ Λ ( A ) \{\delta_{b^\varepsilon(s,z)}\}_{(s,z)}\in\Lambda(A) { δ b ε ( s , z ) } ( s , z ) ∈ Λ ( A ) . Thus,
inf { α s , z } ∈ Λ ( A ) ∫ S × Z ∫ A c ( s , a , z ) d α s , z ( a ) d γ ( s , z ) ≤ ∫ S × Z c ( s , b ε ( s , z ) , z ) d γ ( s , z ) ≤ ∫ S × Z inf a ∈ A c ( s , a , z ) d γ ( s , z ) + ε . \begin{aligned}
\inf_{\{\alpha^{s,z}\}\in\Lambda(A)}\int_{S\times Z}\int_A c(s,a,z)d\alpha^{s,z}(a)d\gamma(s,z)\leq\int_{S\times Z}c(s,b^\varepsilon(s,z),z)d\gamma(s,z) \\
&\leq\int_{S\times Z}\inf_{a\in A}c(s,a,z)d\gamma(s,z)+\varepsilon.
\end{aligned} { α s , z } ∈ Λ ( A ) inf ∫ S × Z ∫ A c ( s , a , z ) d α s , z ( a ) d γ ( s , z ) ≤ ∫ S × Z c ( s , b ε ( s , z ) , z ) d γ ( s , z ) ≤ ∫ S × Z a ∈ A inf c ( s , a , z ) d γ ( s , z ) + ε .
Since ε > 0 \varepsilon>0 ε > 0 is arbitrary, we conclude
inf { α s , z } ∈ Λ ( A ) ∫ S × Z ∫ A c ( s , a , z ) d α s , z ( a ) d γ ( s , z ) ≤ ∫ S × Z inf a ∈ A c ( s , a , z ) d γ ( s , z ) . \inf_{\{\alpha^{s,z}\}\in\Lambda(A)}\int_{S\times Z}\int_A c(s,a,z)d\alpha^{s,z}(a)d\gamma(s,z)\leq\int_{S\times Z}\inf_{a\in A}c(s,a,z)d\gamma(s,z). { α s , z } ∈ Λ ( A ) inf ∫ S × Z ∫ A c ( s , a , z ) d α s , z ( a ) d γ ( s , z ) ≤ ∫ S × Z a ∈ A inf c ( s , a , z ) d γ ( s , z ) .
This completes the proof of the upper bound and, with it, the proof of the lemma. □
This lemma will allow us to relate the dynamic programming over probability measures with the standard dynamic programming over the state and action spaces.
Finally, we present a lemma that upper bounds the transport cost when replacing “full couplings” with “marginal plans”. This inequality will be used to simplify the optimization by relaxing the coupling constraints, enabling us to separate noise from the population-level allocation.
Lemma B.6 (Suboptimality of marginal plans). Let X X X , W W W , Z Z Z be Polish spaces. Given a lower semianalytic transportation cost c : X × W × Z → R ‾ ≥ 0 c:X\times W\times Z\to\overline{\mathbb{R}}_{\geq0} c : X × W × Z → R ≥ 0 and probability measures ρ ∈ P ( X × W ) \rho\in\mathcal{P}(X\times W) ρ ∈ P ( X × W ) , ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) , let ρ = λ ⊗ q x \rho=\lambda\otimes q^x ρ = λ ⊗ q x be the disintegration of ρ \rho ρ with respect to its first marginal λ : = ( proj X X × W ) # ρ \lambda:=(\operatorname{proj}^{X\times W}_X)_{\#}\rho λ := ( proj X X × W ) # ρ , where q x ∈ P ( W ) q^x\in\mathcal{P}(W) q x ∈ P ( W ) is a (deterministic) Borel kernel (we use the symbol q q q rather than ξ ^ \hat{\xi} ξ ^ to stress that no randomness is involved here). Then,
inf γ ∈ Γ ( ρ , ν ) ∫ X × W × Z c ( x , w , z ) d γ ( x , w , z ) ≤ inf γ 1 ∈ Γ ( λ , ν ) ∫ X × Z ∫ W c ( x , w , z ) d q x ( w ) d γ 1 ( x , z ) . (11) \inf_{\gamma\in\Gamma(\rho,\nu)}\int_{X\times W\times Z}c(x,w,z)d\gamma(x,w,z)\leq\inf_{\gamma_1\in\Gamma(\lambda,\nu)}\int_{X\times Z}\int_W c(x,w,z)dq^x(w)d\gamma_1(x,z). \tag*{(11)} γ ∈ Γ ( ρ , ν ) inf ∫ X × W × Z c ( x , w , z ) d γ ( x , w , z ) ≤ γ 1 ∈ Γ ( λ , ν ) inf ∫ X × Z ∫ W c ( x , w , z ) d q x ( w ) d γ 1 ( x , z ) . (11)
Proof. Consider the disintegration of ρ \rho ρ with respect to its first marginal λ = ( proj X X × W ) # ρ ∈ P ( X ) \lambda=(\operatorname{proj}^{X\times W}_X)_{\#}\rho\in\mathcal{P}(X) λ = ( proj X X × W ) # ρ ∈ P ( X ) , ρ = λ ⊗ q x \rho=\lambda\otimes q^x ρ = λ ⊗ q x . Let γ 1 ∈ Γ ( λ , ν ) \gamma_1\in\Gamma(\lambda,\nu) γ 1 ∈ Γ ( λ , ν ) and consider the gluing γ ˉ = γ 1 ⊗ q x ∈ P ( X × Z × W ) \bar{\gamma}=\gamma_1\otimes q^x\in\mathcal{P}(X\times Z\times W) γ ˉ = γ 1 ⊗ q x ∈ P ( X × Z × W ) . To “reorder” marginals, consider the permutation
σ : X × Z × W → X × W × Z , σ ( x , z , w ) = ( x , w , z ) , \sigma:X\times Z\times W\to X\times W\times Z,\qquad\sigma(x,z,w)=(x,w,z), σ : X × Z × W → X × W × Z , σ ( x , z , w ) = ( x , w , z ) ,
and set γ = σ # ( γ 1 ⊗ q x ) ∈ P ( X × W × Z ) \gamma=\sigma_{\#}(\gamma_1\otimes q^x)\in\mathcal{P}(X\times W\times Z) γ = σ # ( γ 1 ⊗ q x ) ∈ P ( X × W × Z ) .
We now show that γ \gamma γ is a coupling between ρ \rho ρ and ν \nu ν and, thus, a valid upper bound for the infimum in (11) . We check each marginal by testing against an arbitrary bounded measurable function. Let φ : X × W → R \varphi:X\times W\to\mathbb{R} φ : X × W → R be bounded and measurable, we have
∫ X × W φ ( x , w ) d ( ( proj X × W X × W × Z ) # γ ) ( x , w ) = ∫ X × W × Z φ ( proj X × W X × W × Z ( x , w , z ) ) d γ ( x , w , z ) = ∫ X × W × Z φ ( proj X × W X × W × Z ( x , w , z ) ) d ( σ # ( γ 1 ⊗ q x ) ) ( x , w , z ) = ∫ X × Z × W φ ( proj X × W X × W × Z ( σ ( x , z , w ) ) ) d ( γ 1 ⊗ q x ) ( x , z , w ) = ∫ X × Z ∫ W φ ( x , w ) d q x ( w ) d γ 1 ( x , z ) = ∫ X ∫ W φ ( x , w ) d q x ( w ) d λ ( x ) . \begin{aligned}
\int_{X\times W}\varphi(x,w)\,\mathrm{d}\left(\left(\operatorname{proj}_{X\times W}^{X\times W\times Z}\right)_{\#}\gamma\right)(x,w)
&=\int_{X\times W\times Z}\varphi\left(\operatorname{proj}_{X\times W}^{X\times W\times Z}(x,w,z)\right)\,\mathrm{d}\gamma(x,w,z)\\
&=\int_{X\times W\times Z}\varphi\left(\operatorname{proj}_{X\times W}^{X\times W\times Z}(x,w,z)\right)\,\mathrm{d}\left(\sigma_{\#}(\gamma_1\otimes q^x)\right)(x,w,z)\\
&=\int_{X\times Z\times W}\varphi\left(\operatorname{proj}_{X\times W}^{X\times W\times Z}(\sigma(x,z,w))\right)\,\mathrm{d}(\gamma_1\otimes q^x)(x,z,w)\\
&=\int_{X\times Z}\int_W\varphi(x,w)\,\mathrm{d}q^x(w)\,\mathrm{d}\gamma_1(x,z)\\
&=\int_X\int_W\varphi(x,w)\,\mathrm{d}q^x(w)\,\mathrm{d}\lambda(x).
\end{aligned} ∫ X × W φ ( x , w ) d ( ( proj X × W X × W × Z ) # γ ) ( x , w ) = ∫ X × W × Z φ ( proj X × W X × W × Z ( x , w , z ) ) d γ ( x , w , z ) = ∫ X × W × Z φ ( proj X × W X × W × Z ( x , w , z ) ) d ( σ # ( γ 1 ⊗ q x ) ) ( x , w , z ) = ∫ X × Z × W φ ( proj X × W X × W × Z ( σ ( x , z , w )) ) d ( γ 1 ⊗ q x ) ( x , z , w ) = ∫ X × Z ∫ W φ ( x , w ) d q x ( w ) d γ 1 ( x , z ) = ∫ X ∫ W φ ( x , w ) d q x ( w ) d λ ( x ) .
For every bounded and measurable φ \varphi φ , since q x q^x q x is the disintegration of ρ \rho ρ on W W W given x x x , and γ 1 ∈ Γ ( λ , ν ) \gamma_1\in\Gamma(\lambda,\nu) γ 1 ∈ Γ ( λ , ν ) has first marginal λ \lambda λ , Fubini–Tonelli [42 Theorem 18.3] gives
∫ X ∫ W φ ( x , w ) d q x ( w ) d λ ( x ) = ∫ X × W φ ( x , w ) d ( λ ⊗ q x ) ( x , w ) = ∫ X × W φ ( x , w ) d ρ ( x , w ) . \int_X\int_W\varphi(x,w)\,\mathrm{d}q^x(w)\,\mathrm{d}\lambda(x)=\int_{X\times W}\varphi(x,w)\,\mathrm{d}(\lambda\otimes q^x)(x,w)=\int_{X\times W}\varphi(x,w)\,\mathrm{d}\rho(x,w). ∫ X ∫ W φ ( x , w ) d q x ( w ) d λ ( x ) = ∫ X × W φ ( x , w ) d ( λ ⊗ q x ) ( x , w ) = ∫ X × W φ ( x , w ) d ρ ( x , w ) .
Hence ( proj X × W X × W × Z ) # γ = ρ \left(\operatorname{proj}_{X\times W}^{X\times W\times Z}\right)_{\#}\gamma=\rho ( proj X × W X × W × Z ) # γ = ρ . Similarly, for the marginal over Z Z Z , let ψ : Z → R \psi:Z\to\mathbb{R} ψ : Z → R be bounded and measurable. Then,
∫ Z ψ ( z ) d ( ( proj Z X × W × Z ) # γ ) ( z ) = ∫ X × W × Z ψ ( proj Z X × W × Z ( x , w , z ) ) d γ ( x , w , z ) = ∫ X × W × Z ψ ( proj Z X × W × Z ( x , w , z ) ) d ( σ # ( γ 1 ⊗ q x ) ) ( x , w , z ) = ∫ X × Z × W ψ ( proj Z X × W × Z ( σ ( x , z , w ) ) ) d ( γ 1 ⊗ q x ) ( x , z , w ) = ∫ X × Z ∫ W ψ ( z ) d q x ( w ) d γ 1 ( x , z ) . \begin{aligned}
\int_Z\psi(z)\,\mathrm{d}\left(\left(\operatorname{proj}_{Z}^{X\times W\times Z}\right)_{\#}\gamma\right)(z)
&=\int_{X\times W\times Z}\psi\left(\operatorname{proj}_{Z}^{X\times W\times Z}(x,w,z)\right)\,\mathrm{d}\gamma(x,w,z)\\
&=\int_{X\times W\times Z}\psi\left(\operatorname{proj}_{Z}^{X\times W\times Z}(x,w,z)\right)\,\mathrm{d}\left(\sigma_{\#}(\gamma_1\otimes q^x)\right)(x,w,z)\\
&=\int_{X\times Z\times W}\psi\left(\operatorname{proj}_{Z}^{X\times W\times Z}(\sigma(x,z,w))\right)\,\mathrm{d}(\gamma_1\otimes q^x)(x,z,w)\\
&=\int_{X\times Z}\int_W\psi(z)\,\mathrm{d}q^x(w)\,\mathrm{d}\gamma_1(x,z).
\end{aligned} ∫ Z ψ ( z ) d ( ( proj Z X × W × Z ) # γ ) ( z ) = ∫ X × W × Z ψ ( proj Z X × W × Z ( x , w , z ) ) d γ ( x , w , z ) = ∫ X × W × Z ψ ( proj Z X × W × Z ( x , w , z ) ) d ( σ # ( γ 1 ⊗ q x ) ) ( x , w , z ) = ∫ X × Z × W ψ ( proj Z X × W × Z ( σ ( x , z , w )) ) d ( γ 1 ⊗ q x ) ( x , z , w ) = ∫ X × Z ∫ W ψ ( z ) d q x ( w ) d γ 1 ( x , z ) .
Since ∫ W d q x ( w ) = 1 \int_W\mathrm{d}q^x(w)=1 ∫ W d q x ( w ) = 1 for all x x x and ν \nu ν is the marginal over Z Z Z of γ 1 ∈ Γ ( λ , ν ) \gamma_1\in\Gamma(\lambda,\nu) γ 1 ∈ Γ ( λ , ν ) , this reduces to
∫ X × Z ψ ( z ) d γ 1 ( x , z ) = ∫ Z ψ ( z ) d ν ( z ) . \int_{X\times Z}\psi(z)\,\mathrm{d}\gamma_1(x,z)=\int_Z\psi(z)\,\mathrm{d}\nu(z). ∫ X × Z ψ ( z ) d γ 1 ( x , z ) = ∫ Z ψ ( z ) d ν ( z ) .
Combining the two chains of equalities, ∫ Z ψ d ( ( proj Z X × W × Z ) # γ ) = ∫ Z ψ d ν \int_Z\psi\,\mathrm{d}\left(\left(\operatorname{proj}_{Z}^{X\times W\times Z}\right)_{\#}\gamma\right)=\int_Z\psi\,\mathrm{d}\nu ∫ Z ψ d ( ( proj Z X × W × Z ) # γ ) = ∫ Z ψ d ν for every bounded measurable ψ \psi ψ , and hence ( proj Z X × W × Z ) # γ = ν \left(\operatorname{proj}_{Z}^{X\times W\times Z}\right)_{\#}\gamma=\nu ( proj Z X × W × Z ) # γ = ν . Overall, we have proven that, for all γ 1 ∈ Γ ( λ , ν ) \gamma_1\in\Gamma(\lambda,\nu) γ 1 ∈ Γ ( λ , ν ) , γ = σ # ( γ 1 ⊗ q x ) ∈ Γ ( ρ , ν ) \gamma=\sigma_{\#}(\gamma_1\otimes q^x)\in\Gamma(\rho,\nu) γ = σ # ( γ 1 ⊗ q x ) ∈ Γ ( ρ , ν ) .
We are now ready to prove (11). Fix γ 1 ∈ Γ ( λ , ν ) \gamma_1\in\Gamma(\lambda,\nu) γ 1 ∈ Γ ( λ , ν ) . The coupling γ = σ # ( γ 1 ⊗ q x ) ∈ Γ ( ρ , ν ) \gamma=\sigma_{\#}(\gamma_1\otimes q^x)\in\Gamma(\rho,\nu) γ = σ # ( γ 1 ⊗ q x ) ∈ Γ ( ρ , ν ) is a valid candidate for the left-hand side. Thus, we obtain the upper bound
inf γ ~ ∈ Γ ( ρ , ν ) ∫ X × W × Z c ( x , w , z ) d γ ~ ( x , w , z ) ≤ ∫ X × W × Z c ( x , w , z ) d γ ( x , w , z ) = ∫ X × W × Z c ( x , w , z ) d ( σ # ( γ 1 ⊗ q x ) ) ( x , w , z ) = ∫ X × Z × W c ( x , w , z ) d ( γ 1 ⊗ q x ) ( x , z , w ) = ∫ X × Z ∫ W c ( x , w , z ) d q x ( w ) d γ 1 ( x , z ) , \begin{aligned}
\inf_{\widetilde{\gamma}\in\Gamma(\rho,\nu)}\int_{X\times W\times Z}c(x,w,z)\,\mathrm{d}\widetilde{\gamma}(x,w,z)
&\leq\int_{X\times W\times Z}c(x,w,z)\,\mathrm{d}\gamma(x,w,z)\\
&=\int_{X\times W\times Z}c(x,w,z)\,\mathrm{d}\left(\sigma_{\#}(\gamma_1\otimes q^x)\right)(x,w,z)\\
&=\int_{X\times Z\times W}c(x,w,z)\,\mathrm{d}(\gamma_1\otimes q^x)(x,z,w)\\
&=\int_{X\times Z}\int_W c(x,w,z)\,\mathrm{d}q^x(w)\,\mathrm{d}\gamma_1(x,z),
\end{aligned} γ ∈ Γ ( ρ , ν ) inf ∫ X × W × Z c ( x , w , z ) d γ ( x , w , z ) ≤ ∫ X × W × Z c ( x , w , z ) d γ ( x , w , z ) = ∫ X × W × Z c ( x , w , z ) d ( σ # ( γ 1 ⊗ q x ) ) ( x , w , z ) = ∫ X × Z × W c ( x , w , z ) d ( γ 1 ⊗ q x ) ( x , z , w ) = ∫ X × Z ∫ W c ( x , w , z ) d q x ( w ) d γ 1 ( x , z ) ,
where the last equality uses the disintegration of γ 1 ⊗ q x \gamma_1 \otimes q^x γ 1 ⊗ q x and Tonelli’s theorem. Since γ 1 ∈ Γ ( λ , ν ) \gamma_1 \in\Gamma(\lambda,\nu) γ 1 ∈ Γ ( λ , ν ) is arbitrary, we may take infima to obtain
inf γ ∈ Γ ( ρ , ν ) ∫ c ( x , w , z ) d γ ( x , w , z ) ≤ inf γ 1 ∈ Γ ( λ , ν ) ∫ X × Z ∫ W c ( x , w , z ) d q x ( w ) d γ 1 ( x , z ) , \inf_{\gamma\in\Gamma(\rho,\nu)} \int c(x,w,z)\,\mathrm{d}\gamma(x,w,z) \le\inf_{\gamma_1 \in\Gamma(\lambda,\nu)} \int_{X \times Z} \int_W c(x,w,z)\,\mathrm{d}q^x(w)\,\mathrm{d}\gamma_1(x,z), γ ∈ Γ ( ρ , ν ) inf ∫ c ( x , w , z ) d γ ( x , w , z ) ≤ γ 1 ∈ Γ ( λ , ν ) inf ∫ X × Z ∫ W c ( x , w , z ) d q x ( w ) d γ 1 ( x , z ) ,
which is the desired statement.
Through the following example, we show that the inequality (11) is generally strict.
Example B.7 (Strictness of Lemma B.6). Let X = W = Z = R X = W = Z = \mathbb{R} X = W = Z = R . Set λ = N ( 0 , 1 ) \lambda= \mathcal{N}(0,1) λ = N ( 0 , 1 ) on X X X . For each x ∈ X x \in X x ∈ X , let q = q x = N ( 0 , 1 ) q = q^x = \mathcal{N}(0,1) q = q x = N ( 0 , 1 ) on W W W , independent of x x x . Then, the product measure ρ = λ ⊗ q \rho= \lambda\otimes q ρ = λ ⊗ q is the law of a pair ( x , w ) (x,w) ( x , w ) with x , w ∼ i . i . d . N ( 0 , 1 ) x,w \mathrel{\overset{\mathrm{i.i.d.}}{\sim}} \mathcal{N}(0,1) x , w ∼ i.i.d. N ( 0 , 1 ) . Let ν = N ( 0 , 1 ) \nu= \mathcal{N}(0,1) ν = N ( 0 , 1 ) on Z Z Z and define the cost
c ( x , w , z ) = ( w − z ) 2 . c(x,w,z) = (w-z)^2. c ( x , w , z ) = ( w − z ) 2 . To evaluate the left-hand side of (11) , we aim to solve
inf γ ∈ Γ ( ρ , ν ) ∫ R 3 c ( x , w , z ) d γ ( x , w , z ) . \inf_{\gamma\in\Gamma(\rho,\nu)} \int_{\mathbb{R}^3} c(x,w,z)\,\mathrm{d}\gamma(x,w,z). γ ∈ Γ ( ρ , ν ) inf ∫ R 3 c ( x , w , z ) d γ ( x , w , z ) . To do so, we show that there exists a plan γ ~ ∈ Γ ( ρ , ν ) \widetilde{\gamma} \in\Gamma(\rho,\nu) γ ∈ Γ ( ρ , ν ) that attains a cost of 0 0 0 , which, by non-negativity of the cost, implies optimality. Consider T : R 2 → R 3 T:\mathbb{R}^2 \to\mathbb{R}^3 T : R 2 → R 3 defined by T ( x , w ) = ( x , w , w ) T(x,w) = (x,w,w) T ( x , w ) = ( x , w , w ) and set γ ~ = T # ρ \widetilde{\gamma} = T_{\#}\rho γ = T # ρ . By construction, its ( x , w ) (x,w) ( x , w ) -marginal is ρ \rho ρ and its z z z -marginal is given by w ∼ N ( 0 , 1 ) = ν w \sim\mathcal{N}(0,1) = \nu w ∼ N ( 0 , 1 ) = ν , hence γ ~ ∈ Γ ( ρ , ν ) \widetilde{\gamma} \in\Gamma(\rho,\nu) γ ∈ Γ ( ρ , ν ) . Under this coupling
∫ R 3 ( w − z ) 2 d γ ~ ( x , w , z ) = ∫ R 2 ( w − w ) 2 d ρ ( x , w ) = 0 \int_{\mathbb{R}^3} (w-z)^2\,\mathrm{d}\widetilde{\gamma}(x,w,z) = \int_{\mathbb{R}^2} (w-w)^2\,\mathrm{d}\rho(x,w) = 0 ∫ R 3 ( w − z ) 2 d γ ( x , w , z ) = ∫ R 2 ( w − w ) 2 d ρ ( x , w ) = 0 and so
inf γ ∈ Γ ( ρ , ν ) ∫ R 3 c ( x , w , z ) d γ ( x , w , z ) = 0. \inf_{\gamma\in\Gamma(\rho,\nu)} \int_{\mathbb{R}^3} c(x,w,z)\,\mathrm{d}\gamma(x,w,z) = 0. γ ∈ Γ ( ρ , ν ) inf ∫ R 3 c ( x , w , z ) d γ ( x , w , z ) = 0. We now consider the right-hand side in (11) . For any γ 1 ∈ Γ ( λ , ν ) \gamma_1 \in\Gamma(\lambda,\nu) γ 1 ∈ Γ ( λ , ν ) , we evaluate
∫ X × Z ∫ W c ( x , w , z ) d q x ( w ) d γ 1 ( x , z ) . \int_{X \times Z} \int_W c(x,w,z)\,\mathrm{d}q^x(w)\,\mathrm{d}\gamma_1(x,z). ∫ X × Z ∫ W c ( x , w , z ) d q x ( w ) d γ 1 ( x , z ) . Since the disintegration of ρ \rho ρ is q = N ( 0 , 1 ) q = \mathcal{N}(0,1) q = N ( 0 , 1 ) (in particular, it is independent of x x x ), we have
∫ R c ( x , w , z ) d q ( w ) = ∫ R ( w − z ) 2 d q ( w ) = ∫ R w 2 d q ( w ) − 2 z ∫ R w d q ( w ) + z 2 = 1 + z 2 , \int_{\mathbb{R}} c(x,w,z)\,\mathrm{d}q(w) = \int_{\mathbb{R}} (w-z)^2\,\mathrm{d}q(w) = \int_{\mathbb{R}} w^2\,\mathrm{d}q(w) - 2z\int_{\mathbb{R}} w\,\mathrm{d}q(w) + z^2 = 1 + z^2, ∫ R c ( x , w , z ) d q ( w ) = ∫ R ( w − z ) 2 d q ( w ) = ∫ R w 2 d q ( w ) − 2 z ∫ R w d q ( w ) + z 2 = 1 + z 2 , and so
inf γ 1 ∈ Γ ( λ , ν ) ∫ R 2 ∫ R c ( x , w , z ) d q ( w ) d γ 1 ( x , z ) = inf γ 1 ∈ Γ ( λ , ν ) ∫ R 2 1 + z 2 d γ 1 ( x , z ) = 1 + ∫ R z 2 d ν ( z ) = 2 , \begin{aligned}
\inf_{\gamma_1 \in\Gamma(\lambda,\nu)} \int_{\mathbb{R}^2} \int_{\mathbb{R}} c(x,w,z)\,\mathrm{d}q(w)\,\mathrm{d}\gamma_1(x,z)
&= \inf_{\gamma_1 \in\Gamma(\lambda,\nu)} \int_{\mathbb{R}^2} 1+z^2\,\mathrm{d}\gamma_1(x,z) \\
&= 1 + \int_{\mathbb{R}} z^2\,\mathrm{d}\nu(z) \\
&= 2,
\end{aligned} γ 1 ∈ Γ ( λ , ν ) inf ∫ R 2 ∫ R c ( x , w , z ) d q ( w ) d γ 1 ( x , z ) = γ 1 ∈ Γ ( λ , ν ) inf ∫ R 2 1 + z 2 d γ 1 ( x , z ) = 1 + ∫ R z 2 d ν ( z ) = 2 , where the last equality follows from the fact that the second marginal of γ 1 \gamma_1 γ 1 is ν = N ( 0 , 1 ) \nu= \mathcal{N}(0,1) ν = N ( 0 , 1 ) and thus its second moment is 1 1 1 . Clearly, the left- and right-hand sides do not agree, and the inequality (11) is strict in general.
Example B.7 shows that the inequality of Lemma B.6, and hence the bound of Theorem 3.1, Part I, is generally strict. Informally, the gap corresponds to the value of being able to match noise realizations directly to targets, rather than averaging over them. In control problems where noise is small relative to the nominal cost (as in the low-noise regime of our grid-world experiments), we expect this gap to be small.
Proofs of Section 3
We now prove the main results presented in § 3 . Throughout the following appendices, we write for the set of admissible state-action distributions at μ ∈ P ( S ) \mu\in\mathcal{P}(S) μ ∈ P ( S )
M ( μ ) : = { λ ∈ P ( S × A ) : ( proj S S × A ) # λ = μ } . \mathcal{M}(\mu) := \left\{ \lambda\in\mathcal{P}(S \times A) : \left(\operatorname{proj}_{S}^{S \times A}\right)_{\#}\lambda= \mu\right\}. M ( μ ) := { λ ∈ P ( S × A ) : ( proj S S × A ) # λ = μ } .
that is, the state-action distributions whose S S S -marginal is μ \mu μ . An admissible population control in the sense of Problem 2 thus satisfies λ t ( μ ) ∈ M ( μ ) \lambda_t(\mu) \in\mathcal{M}(\mu) λ t ( μ ) ∈ M ( μ ) for every μ \mu μ . Recall the single-agent problem and the dynamic programming for the cost-to-go function in (2):
j T ( s T , z ) = g T ( s T , z ) ; j_T(s_T,z)=g_T(s_T,z); j T ( s T , z ) = g T ( s T , z ) ;
j t ( s t , z ) = inf a t ∈ A E w t [ g t ( s t , a t , w t ) + j t + 1 ( f t ( s t , a t , w t ) , z ) ] . j_t(s_t,z)=\inf_{a_t\in A}\mathbb{E}_{w_t}\left[g_t(s_t,a_t,w_t)+j_{t+1}(f_t(s_t,a_t,w_t),z)\right]. j t ( s t , z ) = a t ∈ A inf E w t [ g t ( s t , a t , w t ) + j t + 1 ( f t ( s t , a t , w t ) , z ) ] .
Proof of Theorem 3.1
We aim to show that the population-level cost-to-go function in Problem 2 is upper bounded by an optimal transport problem whose cost is the single-agent cost-to-go function in (2). Concretely, we show by backward induction that
J t ( μ t , ν ) ≤ inf γ ∈ Γ ( μ t , ν ) ∫ j t ( s , z ) d γ ( s , z ) . J_t(\mu_t,\nu)\leq\inf_{\gamma\in\Gamma(\mu_t,\nu)}\int j_t(s,z)\,\mathrm{d}\gamma(s,z). J t ( μ t , ν ) ≤ γ ∈ Γ ( μ t , ν ) inf ∫ j t ( s , z ) d γ ( s , z ) .
We first prove the theorem under the field-based setting of Assumptions A.1 and A.2. The constructions in Examples A.4–A.7 and Proposition A.9 cover individual noise with finitely or infinitely many agents, global noise, and local noise on countably supported state-action distributions. For atomless local noise, the footnote in Example A.6 gives the corresponding aggregate random-measure construction and explains how the same proof carries over. In the field-based setting, Lemma A.10 allows us to replace the realized noise kernels by the nominal kernel after taking expectations. Throughout, Assumption A.1 makes the gluing ρ t = λ t ⊗ ξ ^ t s , a \rho_t=\lambda_t\otimes\hat{\xi}_t^{s,a} ρ t = λ t ⊗ ξ ^ t s , a a well-defined random measure and justifies interchanging expectations over the realized field with λ t \lambda_t λ t -integrals. All integrands below are lower semianalytic (see Appendix § 10 ), so all expectations are well-defined.
We start with the proof of Part I of the theorem.
Proof. The proof proceeds by induction, in three steps. First, we use the definition of the cost-to-go function and the induction hypothesis at time t + 1 t+1 t + 1 . Second, we express the conditional expectations over noise realizations in terms of the averaged single-agent cost. Finally, we reorganize the infima over couplings by means of Lemma B.4, which allows us to recover the desired distributional dynamic program and conclude the induction.
The base case of the induction is straightforward. By definition J T ( μ T , ν ) = K [ g T ] ( μ T , ν ) J_T(\mu_T,\nu)=\mathcal{K}[g_T](\mu_T,\nu) J T ( μ T , ν ) = K [ g T ] ( μ T , ν ) . Since j T ( s , z ) = g T ( s , z ) j_T(s,z)=g_T(s,z) j T ( s , z ) = g T ( s , z ) , we have K [ j T ] ( μ T , ν ) = K [ g T ] ( μ T , ν ) = J T ( μ T , ν ) \mathcal{K}[j_T](\mu_T,\nu)=\mathcal{K}[g_T](\mu_T,\nu)=J_T(\mu_T,\nu) K [ j T ] ( μ T , ν ) = K [ g T ] ( μ T , ν ) = J T ( μ T , ν ) . Fix t < T t<T t < T , using the induction hypothesis at time t + 1 t+1 t + 1 , i.e.,
J t + 1 ( f t # ρ t , ν ) ≤ inf γ ∈ Γ ( f t # ρ t , ν ) ∫ S × Z j t + 1 ( s + , z ) d γ , J_{t+1}(f_{t\#}\rho_t,\nu)\leq\inf_{\gamma\in\Gamma(f_{t\#}\rho_t,\nu)}\int_{S\times Z}j_{t+1}(s^+,z)\,\mathrm{d}\gamma, J t + 1 ( f t # ρ t , ν ) ≤ γ ∈ Γ ( f t # ρ t , ν ) inf ∫ S × Z j t + 1 ( s + , z ) d γ ,
the backward recursion for the cost-to-go function at time t t t gives
J t ( μ t , ν ) = inf ( proj S S × A ) # λ t = μ t E η t [ E ρ t [ g t ( s , a , w ) ] + J t + 1 ( f t # ρ t , ν ) ] ≤ inf ( proj S S × A ) # λ t = μ t E η t [ ∫ S × A × W g t ( s , a , w ) d ρ t + inf γ ∈ Γ ( f t # ρ t , ν ) ∫ S × Z j t + 1 ( s + , z ) d γ ] . (12) \begin{aligned}
J_t(\mu_t,\nu)=\inf_{\left(\operatorname{proj}_{S}^{S\times A}\right)_{\#}\lambda_t=\mu_t}\mathbb{E}_{\eta_t}\left[\mathbb{E}_{\rho_t}\left[g_t(s,a,w)\right]+J_{t+1}(f_{t\#}\rho_t,\nu)\right] \\
&\leq\inf_{\left(\operatorname{proj}_{S}^{S\times A}\right)_{\#}\lambda_t=\mu_t}\mathbb{E}_{\eta_t}\left[\int_{S\times A\times W}g_t(s,a,w)\,\mathrm{d}\rho_t+\inf_{\gamma\in\Gamma(f_{t\#}\rho_t,\nu)}\int_{S\times Z}j_{t+1}(s^+,z)\,\mathrm{d}\gamma\right]. \tag*{(12)}
\end{aligned} J t ( μ t , ν ) = ( proj S S × A ) # λ t = μ t inf E η t [ E ρ t [ g t ( s , a , w ) ] + J t + 1 ( f t # ρ t , ν ) ] ≤ ( proj S S × A ) # λ t = μ t inf E η t [ ∫ S × A × W g t ( s , a , w ) d ρ t + γ ∈ Γ ( f t # ρ t , ν ) inf ∫ S × Z j t + 1 ( s + , z ) d γ ] . (12)
We now focus on the term denoted by ( ∗ ) (*) ( ∗ ) . We analyze ( ∗ ) (*) ( ∗ ) pointwise for each realization of ξ ^ t \hat{\xi}_t ξ ^ t before taking expectations. By Lemma B.1, we can “push” the dynamics into the transportation cost, so that
( ∗ ) = ∫ S × A × W g t ( s , a , w ) d ρ t + inf γ ′ ∈ Γ ( ρ t , ν ) ∫ S × A × W × Z j t + 1 ( f t ( s , a , w ) , z ) d γ ′ . (*) = \int_{S \times A \times W} g_t(s,a,w)\,\mathrm{d}\rho_t + \inf_{\gamma' \in\Gamma(\rho_t,\nu)} \int_{S \times A \times W \times Z} j_{t+1}(f_t(s,a,w),z)\,\mathrm{d}\gamma'. ( ∗ ) = ∫ S × A × W g t ( s , a , w ) d ρ t + γ ′ ∈ Γ ( ρ t , ν ) inf ∫ S × A × W × Z j t + 1 ( f t ( s , a , w ) , z ) d γ ′ .
Here we use that the coupling correspondence in Lemma B.1 preserves the target marginal: since f t × Id Z f_t \times\operatorname{Id}_Z f t × Id Z acts as the identity on the Z Z Z -coordinate, it restricts to a bijection between the couplings with Z Z Z -marginal ν \nu ν , so imposing γ ′ ∈ Γ ( ρ t , ν ) \gamma' \in\Gamma(\rho_t,\nu) γ ′ ∈ Γ ( ρ t , ν ) (with ν \nu ν fixed) is equivalent to the corresponding fixed-ν \nu ν constraint on ( f t ) # ρ t (f_t)_{\#}\rho_t ( f t ) # ρ t . Since ρ t \rho_t ρ t is the first marginal of γ ′ \gamma' γ ′ , Lemma B.2, with X = S × A × W X = S \times A \times W X = S × A × W , c 1 = g t c_1 = g_t c 1 = g t and c 2 = j t + 1 ∘ ( f t × Id Z ) c_2 = j_{t+1} \circ(f_t \times\operatorname{Id}_Z) c 2 = j t + 1 ∘ ( f t × Id Z ) , allows us to merge the two costs into a single transport objective on S × A × W × Z S \times A \times W \times Z S × A × W × Z . We therefore have
( ∗ ) = inf γ ′ ∈ Γ ( ρ t , ν ) ∫ S × A × W × Z [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ′ . (*) = \inf_{\gamma' \in\Gamma(\rho_t,\nu)} \int_{S \times A \times W \times Z} \left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\gamma'. ( ∗ ) = γ ′ ∈ Γ ( ρ t , ν ) inf ∫ S × A × W × Z [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ′ .
We now use Lemma B.6 with X = S × A X = S \times A X = S × A , ρ t = λ t ⊗ ξ ^ t s , a \rho_t = \lambda_t \otimes\hat{\xi}_t^{s,a} ρ t = λ t ⊗ ξ ^ t s , a , and c ( s , a , w , z ) = g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) c(s,a,w,z) = g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z) c ( s , a , w , z ) = g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) , to replace the couplings of the full state-action-noise distribution ρ t \rho_t ρ t with couplings of its ( S × A ) (S \times A) ( S × A ) -marginal λ t \lambda_t λ t . This yields the inequality
( ∗ ) ≤ inf γ ~ ∈ Γ ( λ t , ν ) ∫ S × A × Z ∫ W [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d ξ ^ t s , a d γ ~ . (*) \le\inf_{\widetilde{\gamma} \in\Gamma(\lambda_t,\nu)} \int_{S \times A \times Z} \int_W \left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\hat{\xi}_t^{s,a}\,\mathrm{d}\widetilde{\gamma}. ( ∗ ) ≤ γ ∈ Γ ( λ t , ν ) inf ∫ S × A × Z ∫ W [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d ξ ^ t s , a d γ .
This relaxation removes the dependence on the realized noise from the optimal assignment, thus restricting it to depend only on ( s , a ) (s,a) ( s , a ) .
We now plug the upper bound obtained for ( ∗ ) (*) ( ∗ ) inside the expectation over ξ ^ t s , a \hat{\xi}_t^{s,a} ξ ^ t s , a , as in (12) . Since λ t \lambda_t λ t is fixed and the infimum is now over couplings γ ~ ∈ Γ ( λ t , ν ) \widetilde{\gamma} \in\Gamma(\lambda_t,\nu) γ ∈ Γ ( λ t , ν ) that do not depend on ξ ^ t s , a \hat{\xi}_t^{s,a} ξ ^ t s , a , Lemma B.3 applies, allowing us to interchange expectation and infimum and yielding the upper bound
E η t [ ( ∗ ) ] ≤ E η t [ inf γ ~ ∈ Γ ( λ t , ν ) ∫ S × A × Z ∫ W [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d ξ ^ t s , a d γ ~ ] ≤ inf γ ~ ∈ Γ ( λ t , ν ) E η t [ ∫ S × A × Z ∫ W [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d ξ ^ t s , a d γ ~ ] = inf γ ~ ∈ Γ ( λ t , ν ) ∫ S × A × Z E η t s , a [ ∫ W [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d ξ ^ t s , a ] d γ ~ , \begin{aligned}
\mathbb{E}_{\eta_t}[(*)] \le\mathbb{E}_{\eta_t}\left[\inf_{\widetilde{\gamma} \in\Gamma(\lambda_t,\nu)} \int_{S \times A \times Z} \int_W \left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\hat{\xi}_t^{s,a}\,\mathrm{d}\widetilde{\gamma}\right] \\
&\le\inf_{\widetilde{\gamma} \in\Gamma(\lambda_t,\nu)} \mathbb{E}_{\eta_t}\left[\int_{S \times A \times Z} \int_W \left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\hat{\xi}_t^{s,a}\,\mathrm{d}\widetilde{\gamma}\right] \\
&= \inf_{\widetilde{\gamma} \in\Gamma(\lambda_t,\nu)} \int_{S \times A \times Z} \mathbb{E}_{\eta_t}^{s,a}\left[\int_W \left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\hat{\xi}_t^{s,a}\right]\,\mathrm{d}\widetilde{\gamma},
\end{aligned} E η t [( ∗ )] ≤ E η t [ γ ∈ Γ ( λ t , ν ) inf ∫ S × A × Z ∫ W [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d ξ ^ t s , a d γ ] ≤ γ ∈ Γ ( λ t , ν ) inf E η t [ ∫ S × A × Z ∫ W [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d ξ ^ t s , a d γ ] = γ ∈ Γ ( λ t , ν ) inf ∫ S × A × Z E η t s , a [ ∫ W [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d ξ ^ t s , a ] d γ ,
where the last equality follows from Tonelli’s theorem, which applies by nonnegativity of the costs and joint measurability of the realized field (Assumption A.1). Informally, the second inequality is precisely where we transition from “choose the assignment after seeing the realized noise” to “choose one assignment against average noise”. We now focus on the inner expectation. In particular, consider a fixed γ ~ ∈ Γ ( λ t , ν ) \widetilde{\gamma} \in\Gamma(\lambda_t,\nu) γ ∈ Γ ( λ t , ν ) . We can disintegrate the coupling with respect to its ( S × A ) (S \times A) ( S × A ) -marginal λ t \lambda_t λ t as γ ~ = λ t ⊗ κ s , a \widetilde{\gamma} = \lambda_t \otimes\kappa_{s,a} γ = λ t ⊗ κ s , a , where κ s , a \kappa_{s,a} κ s , a is a (conditional) probability over Z Z Z . By non-negativity of all functions, Tonelli’s theorem gives
∫ S × A × Z E η t s , a [ ∫ W [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d ξ ^ t s , a ] d γ ~ = ∫ S × A E η t s , a [ ∫ W ∫ Z [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d κ s , a d ξ ^ t s , a ] d λ t . \int_{S \times A \times Z} \mathbb{E}_{\eta_t}^{s,a}\left[\int_W \left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\hat{\xi}_t^{s,a}\right]\,\mathrm{d}\widetilde{\gamma}
=
\int_{S \times A} \mathbb{E}_{\eta_t}^{s,a}\left[\int_W \int_Z \left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\kappa_{s,a}\,\mathrm{d}\hat{\xi}_t^{s,a}\right]\,\mathrm{d}\lambda_t. ∫ S × A × Z E η t s , a [ ∫ W [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d ξ ^ t s , a ] d γ = ∫ S × A E η t s , a [ ∫ W ∫ Z [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d κ s , a d ξ ^ t s , a ] d λ t .
We can apply Lemma A.10 (treating ( s , a ) (s,a) ( s , a ) as parameters) to the function
c ~ ( s , a , w ) : = ∫ Z [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d κ s , a ( z ) . \widetilde{c}(s,a,w) := \int_Z \left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\kappa_{s,a}(z). c ( s , a , w ) := ∫ Z [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d κ s , a ( z ) .
Note, in particular, that Lemma A.10 can be applied since κ s , a \kappa_{s,a} κ s , a is fixed and not a function of the noise distribution ξ ^ t s , a \hat{\xi}_t^{s,a} ξ ^ t s , a . Thus, we get
∫ S × A E η t s , a [ ∫ W ∫ Z [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d κ s , a d ξ ^ t s , a ] d λ t = ∫ S × A E w [ ∫ Z [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d κ s , a ] d λ t = ∫ S × A × Z E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ~ . \begin{aligned}
\int_{S \times A} \mathbb{E}_{\eta_t}^{s,a}\left[\int_W \int_Z \left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\kappa_{s,a}\,\mathrm{d}\hat{\xi}_t^{s,a}\right]\,\mathrm{d}\lambda_t
=
\int_{S \times A} \mathbb{E}_w\left[\int_Z \left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\kappa_{s,a}\right]\,\mathrm{d}\lambda_t \\
&= \int_{S \times A \times Z} \mathbb{E}_w\left[g_t(s,a,w) + j_{t+1}(f_t(s,a,w),z)\right]\,\mathrm{d}\widetilde{\gamma}.
\end{aligned} ∫ S × A E η t s , a [ ∫ W ∫ Z [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d κ s , a d ξ ^ t s , a ] d λ t = ∫ S × A E w [ ∫ Z [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d κ s , a ] d λ t = ∫ S × A × Z E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ .
where the last step follows again from Tonelli’s theorem and the definition of γ ~ \tilde{\gamma} γ ~ . Combined with the previous chain of inequalities, we therefore have
E η t [ ( ∗ ) ] ≤ inf γ ~ ∈ Γ ( λ t , ν ) ∫ S × A × Z E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ~ . \mathbb{E}_{\eta_t}[(\ast)] \le\inf_{\tilde{\gamma}\in\Gamma(\lambda_t,\nu)} \int_{S\times A\times Z} \mathbb{E}_w\left[g_t(s,a,w)+j_{t+1}(f_t(s,a,w),z)\right]\mathrm{d}\tilde{\gamma}. E η t [( ∗ )] ≤ γ ~ ∈ Γ ( λ t , ν ) inf ∫ S × A × Z E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ~ .
We can now plug back our bound in (12) to get
J t ( μ t , ν ) ≤ inf ( proj S × A ) # λ t = μ t γ ~ ∈ Γ ( λ t , ν ) ∫ S × A × Z E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ~ = inf γ ~ ∈ Γ ( μ t , ν ) ∫ S × A × Z E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ~ = : ( ⋆ ) . \begin{aligned}
J_t(\mu_t,\nu) \le\inf_{\substack{(\operatorname{proj}_{S\times A})_{\#}\lambda_t=\mu_t\\ \tilde{\gamma}\in\Gamma(\lambda_t,\nu)}} \int_{S\times A\times Z}\mathbb{E}_w\left[g_t(s,a,w)+j_{t+1}(f_t(s,a,w),z)\right]\mathrm{d}\tilde{\gamma} \\
&= \inf_{\tilde{\gamma}\in\Gamma(\mu_t,\nu)} \int_{S\times A\times Z}\mathbb{E}_w\left[g_t(s,a,w)+j_{t+1}(f_t(s,a,w),z)\right]\mathrm{d}\tilde{\gamma} \\
&=: (\star).
\end{aligned} J t ( μ t , ν ) ≤ ( proj S × A ) # λ t = μ t γ ~ ∈ Γ ( λ t , ν ) inf ∫ S × A × Z E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ~ = γ ~ ∈ Γ ( μ t , ν ) inf ∫ S × A × Z E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ~ =: ( ⋆ ) .
Now, we can use Lemma B.4 to disintegrate the optimizer in ( ⋆ ) (\star) ( ⋆ ) , which gives
( ⋆ ) = inf γ ~ ∈ Γ ( μ t , ν ) α s , z ∈ Λ ( A ) ∫ S × Z ∫ A E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d α s , z d γ ~ . (\star)=\inf_{\substack{\tilde{\gamma}\in\Gamma(\mu_t,\nu)\\ \alpha^{s,z}\in\Lambda(A)}}\int_{S\times Z}\int_A\mathbb{E}_w\left[g_t(s,a,w)+j_{t+1}(f_t(s,a,w),z)\right]\mathrm{d}\alpha^{s,z}\mathrm{d}\tilde{\gamma}. ( ⋆ ) = γ ~ ∈ Γ ( μ t , ν ) α s , z ∈ Λ ( A ) inf ∫ S × Z ∫ A E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d α s , z d γ ~ .
The integrand is lower semianalytic: g t g_t g t , g T g_T g T , and f t f_t f t are Borel, and, by backward induction, j t + 1 j_{t+1} j t + 1 is lower semianalytic, since lower semianalyticity is preserved by composition with Borel maps [31 Lemma 7.30] , integration against the Borel kernel ξ t \xi_t ξ t [31 Proposition 7.48] , and pointwise infima over A A A [31 Proposition 7.47] . Therefore, we can apply Lemma B.5, which gives
( ⋆ ) = inf γ ~ ∈ Γ ( μ t , ν ) ∫ S × Z inf a ∈ A E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ~ . (\star)=\inf_{\tilde{\gamma}\in\Gamma(\mu_t,\nu)}\int_{S\times Z}\inf_{a\in A}\mathbb{E}_w\left[g_t(s,a,w)+j_{t+1}(f_t(s,a,w),z)\right]\mathrm{d}\tilde{\gamma}. ( ⋆ ) = γ ~ ∈ Γ ( μ t , ν ) inf ∫ S × Z a ∈ A inf E w [ g t ( s , a , w ) + j t + 1 ( f t ( s , a , w ) , z ) ] d γ ~ .
At this point, we recognize that the term in the integral coincides with the Bellman recursion for j t j_t j t :
j t ( s t , z ) = inf a t ∈ A E w t [ g t ( s t , a t , w t ) + j t + 1 ( f t ( s t , a t , w t ) , z ) ] . j_t(s_t,z)=\inf_{a_t\in A}\mathbb{E}_{w_t}\left[g_t(s_t,a_t,w_t)+j_{t+1}(f_t(s_t,a_t,w_t),z)\right]. j t ( s t , z ) = a t ∈ A inf E w t [ g t ( s t , a t , w t ) + j t + 1 ( f t ( s t , a t , w t ) , z ) ] .
Hence,
J t ( μ t , ν ) ≤ ( ⋆ ) = inf γ ~ ∈ Γ ( μ t , ν ) ∫ S × Z j t ( s , z ) d γ ~ . J_t(\mu_t,\nu)\le(\star)=\inf_{\tilde{\gamma}\in\Gamma(\mu_t,\nu)}\int_{S\times Z}j_t(s,z)\mathrm{d}\tilde{\gamma}. J t ( μ t , ν ) ≤ ( ⋆ ) = γ ~ ∈ Γ ( μ t , ν ) inf ∫ S × Z j t ( s , z ) d γ ~ .
This concludes the inductive step and the proof of the theorem.
Two comments on our proof are in order. First, while the proof follows the inductive strategy of the proof of the deterministic separation in [27 ] , there is a key difference: the next population state is random and depends on the realized empirical noise kernel ξ ^ t \hat{\xi}_t ξ ^ t . This requires various additional steps (and preliminary lemmata) in the proof. All these steps trivialize in the absence of noise and, thus, our proof recovers the deterministic separation in [27 ] when the dynamics are deterministic. Second, our result is an upper bound on the cost-to-go function. Intuitively, this conservatism originates from the following fact: The population state-action distribution can, in principle, choose future couplings after observing this realization, while our upper bound posits that couplings are chosen before the noise is realized, averaging over the kernel ξ t ( ⋅ ∣ s , a ) \xi_t(\cdot\mid s,a) ξ t ( ⋅ ∣ s , a ) . This is precisely where Lemmas B.3 and B.6 yield inequalities rather than equalities. In the deterministic case, all these steps trivialize and we obtain equality as in [27 ] .
We now prove Part II of the theorem.
Proof. For each time step t t t , consider the function
K [ j t ] ( μ t , ν ) = inf γ ∈ Γ ( μ t , ν ) ∫ S × Z j t ( s , z ) d γ . \mathcal{K}[j_t](\mu_t,\nu)=\inf_{\gamma\in\Gamma(\mu_t,\nu)}\int_{S\times Z}j_t(s,z)\mathrm{d}\gamma. K [ j t ] ( μ t , ν ) = γ ∈ Γ ( μ t , ν ) inf ∫ S × Z j t ( s , z ) d γ .
Let ε > 0 \varepsilon>0 ε > 0 . At each time t t t , we can construct a “separation-based” state-action distribution λ t ε \lambda_t^\varepsilon λ t ε as follows: given a population distribution μ t \mu_t μ t , choose an ε 2 T \frac{\varepsilon}{2T} 2 T ε -optimal coupling γ t ε ∈ Γ ( μ t , ν ) \gamma_t^\varepsilon\in\Gamma(\mu_t,\nu) γ t ε ∈ Γ ( μ t , ν ) satisfying
∫ S × Z j t ( s , z ) d γ t ε ≤ K [ j t ] ( μ t , ν ) + ε 2 T . (13) \int_{S\times Z}j_t(s,z)\mathrm{d}\gamma_t^\varepsilon\le\mathcal{K}[j_t](\mu_t,\nu)+\frac{\varepsilon}{2T}. \tag*{(13)} ∫ S × Z j t ( s , z ) d γ t ε ≤ K [ j t ] ( μ t , ν ) + 2 T ε . (13)
and a universally measurable ε 2 T \frac{\varepsilon}{2T} 2 T ε -optimal action selector a t ε : S × Z → A a_t^\varepsilon\colon S \times Z \to A a t ε : S × Z → A satisfying, for all ( s , z ) (s,z) ( s , z ) ,
E w [ g t ( s , a t ε ( s , z ) , w ) + j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) ] ≤ j t ( s , z ) + ε 2 T . (14) \mathbb{E}_w\left[g_t(s,a_t^\varepsilon(s,z),w)+j_{t+1}(f_t(s,a_t^\varepsilon(s,z),w),z)\right]\leq j_t(s,z)+\frac{\varepsilon}{2T}. \tag*{(14)} E w [ g t ( s , a t ε ( s , z ) , w ) + j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) ] ≤ j t ( s , z ) + 2 T ε . (14)
Formally, we have
λ t ε : = ( proj S S × Z , a t ε ) # γ t ε . \lambda_t^\varepsilon:=\left(\operatorname{proj}_{S}^{S\times Z},a_t^\varepsilon\right)_{\#}\gamma_t^\varepsilon. λ t ε := ( proj S S × Z , a t ε ) # γ t ε .
In words, this state-action distribution assigns agents to targets through γ t ε \gamma_t^\varepsilon γ t ε and controls the individual agents through a t ε a_t^\varepsilon a t ε . Since the first marginal of γ t ε \gamma_t^\varepsilon γ t ε is μ t \mu_t μ t , we have λ t ε ∈ M ( μ t ) \lambda_t^\varepsilon\in M(\mu_t) λ t ε ∈ M ( μ t ) . In the rest of the proof, we will show that the resulting control sequence satisfies the bound,
J 0 ε ( μ 0 , ν ) : = J 0 λ 0 ε , … , λ T − 1 ε ( μ 0 , ν ) ≤ K [ j 0 ] ( μ 0 , ν ) + ε . (15) J_0^\varepsilon(\mu_0,\nu):=J_0^{\lambda_0^\varepsilon,\ldots,\lambda_{T-1}^\varepsilon}(\mu_0,\nu)\leq\mathcal{K}[j_0](\mu_0,\nu)+\varepsilon. \tag*{(15)} J 0 ε ( μ 0 , ν ) := J 0 λ 0 ε , … , λ T − 1 ε ( μ 0 , ν ) ≤ K [ j 0 ] ( μ 0 , ν ) + ε . (15)
Before doing so, we verify well-posedness of all the objects. The ε 2 T \frac{\varepsilon}{2T} 2 T ε -optimal universally measurable selector a t ε a_t^\varepsilon a t ε exists by [31 Proposition 7.50] , since the Bellman integrand in (14) is lower semianalytic and nonnegative (see the induction argument in the proof of Part I). For the existence of the ε 2 T \frac{\varepsilon}{2T} 2 T ε -optimal coupling we work as follows. Since j t j_t j t is lower semianalytic (by backward induction on (2)), the map γ ↦ ∫ S × Z j t d γ \gamma\mapsto\int_{S\times Z}j_t\mathrm{d}\gamma γ ↦ ∫ S × Z j t d γ is lower semianalytic by [31 Corollary 7.48.1] (jointly in ( μ , γ ) (\mu,\gamma) ( μ , γ ) , as it does not depend on μ \mu μ ). Further, the set { ( μ , γ ) : γ ∈ Γ ( μ , ν ) } \{(\mu,\gamma):\gamma\in\Gamma(\mu,\nu)\} {( μ , γ ) : γ ∈ Γ ( μ , ν )} is closed. Indeed, fixing ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) , define
G Z : = { ( μ , γ ) ∈ P ( S ) × P ( S × Z ) : γ ∈ Γ ( μ , ν ) } . G_Z:=\{(\mu,\gamma)\in\mathcal{P}(S)\times\mathcal{P}(S\times Z):\gamma\in\Gamma(\mu,\nu)\}. G Z := {( μ , γ ) ∈ P ( S ) × P ( S × Z ) : γ ∈ Γ ( μ , ν )} .
The maps F S ( γ ) : = ( proj S S × Z ) # γ F_S(\gamma):=(\operatorname{proj}_{S}^{S\times Z})_{\#}\gamma F S ( γ ) := ( proj S S × Z ) # γ and F Z ( γ ) : = ( proj Z S × Z ) # γ F_Z(\gamma):=(\operatorname{proj}_{Z}^{S\times Z})_{\#}\gamma F Z ( γ ) := ( proj Z S × Z ) # γ are continuous for the weak topologies, being the pushforward under a continuous map. Hence,
G Z = { ( μ , γ ) : F S ( γ ) = μ } ∩ { ( μ , γ ) : F Z ( γ ) = ν } . G_Z=\{(\mu,\gamma):F_S(\gamma)=\mu\}\cap\{(\mu,\gamma):F_Z(\gamma)=\nu\}. G Z = {( μ , γ ) : F S ( γ ) = μ } ∩ {( μ , γ ) : F Z ( γ ) = ν } .
The first set is the preimage of the diagonal in P ( S ) × P ( S ) \mathcal{P}(S)\times\mathcal{P}(S) P ( S ) × P ( S ) under the continuous map ( μ , γ ) ↦ ( μ , F S ( γ ) ) (\mu,\gamma)\mapsto(\mu,F_S(\gamma)) ( μ , γ ) ↦ ( μ , F S ( γ )) , and the second is the preimage of the closed singleton { ν } \{\nu\} { ν } under the continuous map ( μ , γ ) ↦ F Z ( γ ) (\mu,\gamma)\mapsto F_Z(\gamma) ( μ , γ ) ↦ F Z ( γ ) . Since these spaces are Polish, the diagonal and singleton are closed. By continuity of F S F_S F S and F Z F_Z F Z , the set G Z G_Z G Z is closed, hence Borel and in particular analytic. Thus, we can apply [31 Proposition 7.50] to obtain the existence of a universally measurable map
μ ↦ γ t ε ( μ ) ∈ Γ ( μ , ν ) with ∫ S × Z j t d γ t ε ( μ ) ≤ K [ j t ] ( μ t , ν ) + ε 2 T . \mu\mapsto\gamma_t^\varepsilon(\mu)\in\Gamma(\mu,\nu)\quad\text{with}\quad\int_{S\times Z}j_t\mathrm{d}\gamma_t^\varepsilon(\mu)\leq\mathcal{K}[j_t](\mu_t,\nu)+\frac{\varepsilon}{2T}. μ ↦ γ t ε ( μ ) ∈ Γ ( μ , ν ) with ∫ S × Z j t d γ t ε ( μ ) ≤ K [ j t ] ( μ t , ν ) + 2 T ε .
This shows the existence of a universally measurable selector γ t ε ( μ ) \gamma_t^\varepsilon(\mu) γ t ε ( μ ) . In particular, for fixed μ \mu μ , γ t ε ( μ ) \gamma_t^\varepsilon(\mu) γ t ε ( μ ) is a Borel probability measure, and λ t ε \lambda_t^\varepsilon λ t ε is a well-defined Borel probability measure: it is the pushforward of γ t ε ( μ ) \gamma_t^\varepsilon(\mu) γ t ε ( μ ) under the universally measurable map ( proj S S × Z , a t ε ) (\operatorname{proj}_{S}^{S\times Z},a_t^\varepsilon) ( proj S S × Z , a t ε ) , which coincides γ t ε ( μ ) \gamma_t^\varepsilon(\mu) γ t ε ( μ ) -almost everywhere with a Borel map [31 Lemma 7.27] .
We now turn our attention to the desired bound (15) . We now show by backward induction the more general bound
J t ε ( μ t , ν ) : = J t λ t ε , … , λ T − 1 ε ( μ t , ν ) ≤ K [ j t ] ( μ t , ν ) + T − t T ε . (16) J_t^\varepsilon(\mu_t,\nu):=J_t^{\lambda_t^\varepsilon,\ldots,\lambda_{T-1}^\varepsilon}(\mu_t,\nu)\leq\mathcal{K}[j_t](\mu_t,\nu)+\frac{T-t}{T}\varepsilon. \tag*{(16)} J t ε ( μ t , ν ) := J t λ t ε , … , λ T − 1 ε ( μ t , ν ) ≤ K [ j t ] ( μ t , ν ) + T T − t ε . (16)
which, evaluated at time t = 0 t=0 t = 0 , readily implies (15) . The base case t = T t=T t = T holds exactly by definition of terminal cost and j T = g T j_T=g_T j T = g T :
J T ε ( μ T , ν ) = K [ g T ] ( μ T , ν ) = inf γ ∈ Γ ( μ T , ν ) ∫ S × Z g T ( s , z ) d γ = K [ j T ] ( μ T , ν ) . J_T^\varepsilon(\mu_T,\nu)=\mathcal{K}[g_T](\mu_T,\nu)=\inf_{\gamma\in\Gamma(\mu_T,\nu)}\int_{S\times Z}g_T(s,z)\mathrm{d}\gamma=\mathcal{K}[j_T](\mu_T,\nu). J T ε ( μ T , ν ) = K [ g T ] ( μ T , ν ) = γ ∈ Γ ( μ T , ν ) inf ∫ S × Z g T ( s , z ) d γ = K [ j T ] ( μ T , ν ) .
Now fix t < T t<T t < T and assume (16) holds at t + 1 t+1 t + 1 . By the policy-cost recursion (which uses the independence of the noise across time) and the induction hypothesis at t + 1 t+1 t + 1 applied inside the expectation, with ρ t ε = λ t ε ⊗ ξ ^ t s , a \rho_t^\varepsilon=\lambda_t^\varepsilon\otimes\widehat{\xi}_t^{s,a} ρ t ε = λ t ε ⊗ ξ t s , a ,
J t ε ( μ t , ν ) ≤ E η t [ ∫ S × A × W g t ( s , a , w ) d ρ t ε + K [ j t + 1 ] ( μ t + 1 , ν ) ] + T − t − 1 T ε . (17) J_t^\varepsilon(\mu_t,\nu)\leq\mathbb{E}_{\eta_t}\left[\int_{S\times A\times W}g_t(s,a,w)\mathrm{d}\rho_t^\varepsilon+\mathcal{K}[j_{t+1}](\mu_{t+1},\nu)\right]+\frac{T-t-1}{T}\varepsilon. \tag*{(17)} J t ε ( μ t , ν ) ≤ E η t [ ∫ S × A × W g t ( s , a , w ) d ρ t ε + K [ j t + 1 ] ( μ t + 1 , ν ) ] + T T − t − 1 ε . (17)
Next, we define
γ ~ t ε : = ( f t , Id Z ) # ( σ # γ ~ t ε ) ∈ P ( S × Z ) \widetilde{\gamma}_t^\varepsilon:=(f_t,\operatorname{Id}_Z)_{\#}(\sigma_{\#}\widetilde{\gamma}_t^\varepsilon)\in\mathcal{P}(S\times Z) γ t ε := ( f t , Id Z ) # ( σ # γ t ε ) ∈ P ( S × Z )
where
γ ~ t ε : = ( Id S × Z , a t ε ) # γ t ε ⊗ ξ ^ t s , a t ε ( s , z ) ∈ P ( S × Z × A × W ) . \widetilde{\gamma}_t^\varepsilon:=(\operatorname{Id}_{S\times Z},a_t^\varepsilon)_{\#}\gamma_t^\varepsilon\otimes\widehat{\xi}_t^{s,a_t^\varepsilon(s,z)}\in\mathcal{P}(S\times Z\times A\times W). γ t ε := ( Id S × Z , a t ε ) # γ t ε ⊗ ξ t s , a t ε ( s , z ) ∈ P ( S × Z × A × W ) .
Where we use the permutation
σ : S × Z × A × W → S × A × W × Z , σ ( s , z , a , w ) = ( s , a , w , z ) , \sigma:S \times Z \times A \times W \to S \times A \times W \times Z,\quad\sigma(s,z,a,w)=(s,a,w,z), σ : S × Z × A × W → S × A × W × Z , σ ( s , z , a , w ) = ( s , a , w , z ) ,
to “re-order” the marginals. (The gluing is well-posed: ( s , z ) ↦ ξ ^ t s , a t ε ( s , z ) (s,z)\mapsto\hat{\xi}^{s,a_t^\varepsilon(s,z)}_{t} ( s , z ) ↦ ξ ^ t s , a t ε ( s , z ) is a universally measurable kernel, as the composition of the field of Assumption A.1 with the universally measurable map ( Id S , a t ε ) (\operatorname{Id}_S,a_t^\varepsilon) ( Id S , a t ε ) .) Informally, γ ˉ t ε \bar{\gamma}_t^\varepsilon γ ˉ t ε is the law of ( f t ( s , a t ε ( s , z ) , w ) , z ) (f_t(s,a_t^\varepsilon(s,z),w),z) ( f t ( s , a t ε ( s , z ) , w ) , z ) where ( s , z ) ∼ γ t ε (s,z)\sim\gamma_t^\varepsilon ( s , z ) ∼ γ t ε and w ∼ ξ ^ t s , a t ε ( s , z ) w\sim\hat{\xi}^{s,a_t^\varepsilon(s,z)}_t w ∼ ξ ^ t s , a t ε ( s , z ) . We now show γ ˉ t ε ∈ Γ ( μ t + 1 , ν ) \bar{\gamma}_t^\varepsilon\in\Gamma(\mu_{t+1},\nu) γ ˉ t ε ∈ Γ ( μ t + 1 , ν ) . Specifically, the second marginal is ν \nu ν , since the z z z -coordinate is unchanged by σ \sigma σ and by ( f t , Id Z ) (f_t,\operatorname{Id}_Z) ( f t , Id Z ) , and ( proj Z S × Z ) # γ ˉ t ε = ν (\operatorname{proj}_{Z}^{S\times Z})_{\#}\bar{\gamma}_t^\varepsilon=\nu ( proj Z S × Z ) # γ ˉ t ε = ν . For the first marginal, the ( S × A × W ) (S\times A\times W) ( S × A × W ) -marginal of σ # γ ~ t ε \sigma_{\#}\widetilde{\gamma}_t^\varepsilon σ # γ t ε equals ρ t ε \rho_t^\varepsilon ρ t ε , since the noise kernel depends on ( s , z ) (s,z) ( s , z ) only through the state-action pair ( s , a t ε ( s , z ) ) (s,a_t^\varepsilon(s,z)) ( s , a t ε ( s , z )) , whose law under ( Id S × Z , a t ε ) # γ t ε (\operatorname{Id}_{S\times Z},a_t^\varepsilon)_{\#}\gamma_t^\varepsilon ( Id S × Z , a t ε ) # γ t ε is λ t ε \lambda_t^\varepsilon λ t ε by definition. Hence the first marginal of γ ˉ t ε \bar{\gamma}_t^\varepsilon γ ˉ t ε is f t # ρ t ε = μ t + 1 f_{t\#}\rho_t^\varepsilon=\mu_{t+1} f t # ρ t ε = μ t + 1 , completing the proof of γ ˉ t ε ∈ Γ ( μ t + 1 , ν ) \bar{\gamma}_t^\varepsilon\in\Gamma(\mu_{t+1},\nu) γ ˉ t ε ∈ Γ ( μ t + 1 , ν ) . By definition of the infimum, we then obtain
K [ j t + 1 ] ( μ t + 1 , ν ) ≤ ∫ S × Z j t + 1 ( s , z ) d γ t ε = ∫ S × Z ∫ W j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) d ξ ^ t s , a t ε ( s , z ) d γ t ε . \mathcal{K}[j_{t+1}](\mu_{t+1},\nu)\leq\int_{S\times Z}j_{t+1}(s,z)\,\mathrm{d}\gamma_t^\varepsilon=\int_{S\times Z}\int_W j_{t+1}(f_t(s,a_t^\varepsilon(s,z),w),z)\,\mathrm{d}\hat{\xi}^{s,a_t^\varepsilon(s,z)}_t\,\mathrm{d}\gamma_t^\varepsilon. K [ j t + 1 ] ( μ t + 1 , ν ) ≤ ∫ S × Z j t + 1 ( s , z ) d γ t ε = ∫ S × Z ∫ W j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) d ξ ^ t s , a t ε ( s , z ) d γ t ε .
Taking expectations over the realized noise field and applying Tonelli’s theorem and Lemma A.10 as in the proof of Part I (with λ t ε \lambda_t^\varepsilon λ t ε in place of λ t \lambda_t λ t ), we get
E η t [ ∫ S × A × W g t ( s , a , w ) d ρ t ε + K [ j t + 1 ] ( μ t + 1 , ν ) ] ≤ E η t [ ∫ S × Z ∫ W [ g t ( s , a t ε ( s , z ) , w ) + j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) ] d ξ ^ t s , a t ε ( s , z ) d γ t ε ] = ∫ S × Z E w [ g t ( s , a t ε ( s , z ) , w ) + j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) ] d γ t ε . \begin{aligned}
\mathbb{E}_{\eta_t}\left[\int_{S\times A\times W}g_t(s,a,w)\,\mathrm{d}\rho_t^\varepsilon+\mathcal{K}[j_{t+1}](\mu_{t+1},\nu)\right]
\leq\mathbb{E}_{\eta_t}\left[\int_{S\times Z}\int_W\left[g_t(s,a_t^\varepsilon(s,z),w)+j_{t+1}(f_t(s,a_t^\varepsilon(s,z),w),z)\right]\,\mathrm{d}\hat{\xi}^{s,a_t^\varepsilon(s,z)}_t\,\mathrm{d}\gamma_t^\varepsilon\right] \\
&=\int_{S\times Z}\mathbb{E}_w\left[g_t(s,a_t^\varepsilon(s,z),w)+j_{t+1}(f_t(s,a_t^\varepsilon(s,z),w),z)\right]\,\mathrm{d}\gamma_t^\varepsilon.
\end{aligned} E η t [ ∫ S × A × W g t ( s , a , w ) d ρ t ε + K [ j t + 1 ] ( μ t + 1 , ν ) ] ≤ E η t [ ∫ S × Z ∫ W [ g t ( s , a t ε ( s , z ) , w ) + j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) ] d ξ ^ t s , a t ε ( s , z ) d γ t ε ] = ∫ S × Z E w [ g t ( s , a t ε ( s , z ) , w ) + j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) ] d γ t ε .
Then, we use the ε 2 T \frac{\varepsilon}{2T} 2 T ε -optimality of a t ε a_t^\varepsilon a t ε in (14) and of γ t ε \gamma_t^\varepsilon γ t ε in (13) to get
∫ S × Z E w [ g t ( s , a t ε ( s , z ) , w ) + j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) ] ⏟ ≤ j t ( s , z ) + ε / ( 2 T ) d γ t ε ≤ ∫ S × Z j t ( s , z ) d γ t ε + ε 2 T ≤ K [ j t ] ( μ t , ν ) + ε 2 T + ε 2 T . \int_{S\times Z}\underbrace{\mathbb{E}_w\left[g_t(s,a_t^\varepsilon(s,z),w)+j_{t+1}(f_t(s,a_t^\varepsilon(s,z),w),z)\right]}_{\leq j_t(s,z)+\varepsilon/(2T)}\,\mathrm{d}\gamma_t^\varepsilon
\leq\int_{S\times Z}j_t(s,z)\,\mathrm{d}\gamma_t^\varepsilon+\frac{\varepsilon}{2T}
\leq\mathcal{K}[j_t](\mu_t,\nu)+\frac{\varepsilon}{2T}+\frac{\varepsilon}{2T}. ∫ S × Z ≤ j t ( s , z ) + ε / ( 2 T ) E w [ g t ( s , a t ε ( s , z ) , w ) + j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) ] d γ t ε ≤ ∫ S × Z j t ( s , z ) d γ t ε + 2 T ε ≤ K [ j t ] ( μ t , ν ) + 2 T ε + 2 T ε .
We can now combine this bound with (17) and get
J t ε ( μ t , ν ) ≤ K [ j t ] ( μ t , ν ) + ε T + T − t − 1 T ε = K [ j t ] ( μ t , ν ) + T − t T ε . J_t^\varepsilon(\mu_t,\nu)\leq\mathcal{K}[j_t](\mu_t,\nu)+\frac{\varepsilon}{T}+\frac{T-t-1}{T}\varepsilon=\mathcal{K}[j_t](\mu_t,\nu)+\frac{T-t}{T}\varepsilon. J t ε ( μ t , ν ) ≤ K [ j t ] ( μ t , ν ) + T ε + T T − t − 1 ε = K [ j t ] ( μ t , ν ) + T T − t ε .
This concludes the inductive step. The bound at t = 0 t=0 t = 0 gives J 0 ε ( μ 0 , ν ) ≤ K [ j 0 ] ( μ 0 , ν ) + ε J_0^\varepsilon(\mu_0,\nu)\leq\mathcal{K}[j_0](\mu_0,\nu)+\varepsilon J 0 ε ( μ 0 , ν ) ≤ K [ j 0 ] ( μ 0 , ν ) + ε .
Remark C.1 (Choosing ε = 0 \varepsilon=0 ε = 0 under stronger regularity). The proof of Part II only requires universally measurable ε \varepsilon ε -optimal coupling selectors, which exist under the lower semianalytic assumptions used throughout Appendix B. Under the stronger assumption that the cost-to-go functions j t : S × Z → R ‾ ≥ 0 j_t:S\times Z\to\overline{\mathbb{R}}_{\geq0} j t : S × Z → R ≥ 0 are lower semicontinuous and the spaces are compact, the infimum in K [ j t ] ( μ , ν ) \mathcal{K}[j_t](\mu,\nu) K [ j t ] ( μ , ν ) is attained for each fixed ( μ , ν ) (\mu,\nu) ( μ , ν ) by the standard existence theorem for optimal couplings [46 Theorem 4.1] . This pointwise existence is, however, not sufficient to choose ε = 0 \varepsilon=0 ε = 0 in Theorem 3.1, Part II: the construction requires measurable selectors of exact optimizers. These exist when all spaces are compact, g t , g T , f t g_t,g_T,f_t g t , g T , f t are continuous, and ξ t \xi_t ξ t is weakly continuous: then j t j_t j t and the Bellman integrand are lower semicontinuous by backward induction, and exact Borel minimizing selectors exist [31 Proposition 7.33] (see also Theorem 18.19); the same argument applies to the coupling problem, where μ ↦ Γ ( μ , ν ) \mu\mapsto\Gamma(\mu,\nu) μ ↦ Γ ( μ , ν ) is a compact-valued continuous correspondence.
Proof of Corollary 3.3
Proof. We construct, for every ε > 0 \varepsilon>0 ε > 0 , an admissible separation policy whose cost is within ( T − t ) ε T \frac{(T-t)\varepsilon}{T} T ( T − t ) ε of K [ j t ] ( μ t N , ν N ) \mathcal{K}[j_t](\mu_t^N,\nu^N) K [ j t ] ( μ t N , ν N ) . Since ε > 0 \varepsilon>0 ε > 0 is arbitrary, and J t N J_t^N J t N is the infimum of the expected cost over all admissible policies, the claim follows. We start by giving an ε \varepsilon ε -optimal separation policy π s e p , t \pi_{\mathrm{sep},t} π sep , t which attains the upper bound. We then show that this policy is admissible, from which we directly obtain the desired inequality.
To define π s e p , t ε \pi_{\mathrm{sep},t}^{\varepsilon} π sep , t ε we proceed as follows. First, since μ t N \mu_t^N μ t N and ν N \nu^N ν N are both empirical measures over N N N atoms, the optimal transport problem K [ j t ] ( μ t N , ν N ) \mathcal{K}[j_t](\mu_t^N,\nu^N) K [ j t ] ( μ t N , ν N ) is a finite-dimensional transportation linear program. We work at the level of agent and target indices, i.e., we replace μ t N \mu_t^N μ t N and ν N \nu^N ν N by the uniform measures on { 1 , … , N } \{1,\ldots,N\} { 1 , … , N } and transport the cost c i j = j t ( s t i , z j ) c_{ij}=j_t(s_t^i,z^j) c ij = j t ( s t i , z j ) . With this identification, every coupling between μ t N \mu_t^N μ t N and ν N \nu^N ν N can be represented by a matrix Γ = ( γ i j ) i , j = 1 N \Gamma=(\gamma_{ij})_{i,j=1}^N Γ = ( γ ij ) i , j = 1 N satisfying γ i j ≥ 0 \gamma_{ij}\ge0 γ ij ≥ 0 , ∑ i = 1 N γ i j = ∑ j = 1 N γ i j = 1 N \sum_{i=1}^N\gamma_{ij}=\sum_{j=1}^N\gamma_{ij}=\frac{1}{N} ∑ i = 1 N γ ij = ∑ j = 1 N γ ij = N 1 ; equivalently, N Γ N\Gamma N Γ is doubly stochastic. By the Birkhoff–von Neumann theorem [47 ] applied to N Γ N\Gamma N Γ , every such matrix is a convex combination of 1 N \frac{1}{N} N 1 -scaled permutation matrices; since the objective is linear, the minimum is attained by a 1 N \frac{1}{N} N 1 -scaled permutation matrix. Thus,
K [ j t ] ( μ t N , ν N ) = 1 N min σ t ∈ S N ∑ i = 1 N j t ( s t i , z σ t ( i ) ) . \mathcal{K}[j_t](\mu_t^N,\nu^N)=\frac{1}{N}\min_{\sigma_t\in S_N}\sum_{i=1}^N j_t(s_t^i,z^{\sigma_t(i)}). K [ j t ] ( μ t N , ν N ) = N 1 σ t ∈ S N min i = 1 ∑ N j t ( s t i , z σ t ( i ) ) .
where S N S_N S N denotes the group of permutations of { 1 , … , N } \{1,\ldots,N\} { 1 , … , N } . Since S N S_N S N is finite and each map s ↦ ∑ i = 1 N j t ( s i , z σ ( i ) ) s\mapsto\sum_{i=1}^N j_t(s^i,z^{\sigma(i)}) s ↦ ∑ i = 1 N j t ( s i , z σ ( i ) ) is universally measurable, a minimizing permutation σ t ∗ : = σ t ∗ ( s t ) \sigma_t^*:=\sigma_t^*(s_t) σ t ∗ := σ t ∗ ( s t ) can be selected measurably (e.g., the first minimizer in a fixed ordering of S N S_N S N ); write z t ∗ , i : = z σ t ∗ ( i ) z_t^{*,i}:=z^{\sigma_t^*(i)} z t ∗ , i := z σ t ∗ ( i ) for the target assigned to agent i i i . Given that agents are now assigned to targets, we can now consider the single-agent optimal control problem from s t i s_t^i s t i to the corresponding target z t ∗ , i z_t^{*,i} z t ∗ , i . Specifically, for ε > 0 \varepsilon>0 ε > 0 , there exists a universally measurable ε T \frac{\varepsilon}{T} T ε -optimal action selector a t ε : S × Z → A a_t^\varepsilon:S\times Z\to A a t ε : S × Z → A , satisfying, for all ( s , z ) ∈ S × Z (s,z)\in S\times Z ( s , z ) ∈ S × Z :
E w [ g t ( s , a t ε ( s , z ) , w ) + j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) ] ≤ j t ( s , z ) + ε T . \mathbb{E}_w\left[g_t(s,a_t^\varepsilon(s,z),w)+j_{t+1}(f_t(s,a_t^\varepsilon(s,z),w),z)\right]\le j_t(s,z)+\frac{\varepsilon}{T}. E w [ g t ( s , a t ε ( s , z ) , w ) + j t + 1 ( f t ( s , a t ε ( s , z ) , w ) , z ) ] ≤ j t ( s , z ) + T ε .
At this point, we define the separation policy π s e p , t ε \pi_{\mathrm{sep},t}^{\varepsilon} π sep , t ε at time t t t . An agent at s t i s_t^i s t i executes the action a t ε ( s t i , z t ∗ , i ) a_t^\varepsilon(s_t^i,z_t^{*,i}) a t ε ( s t i , z t ∗ , i ) . Informally, the optimal transport problem indicates the target, whereas the single-agent optimal control problem determines the optimal action to reach that target. This policy is deterministic, Markovian (it depends only on s t s_t s t ), and universally measurable, being the composition of a t ε a_t^\varepsilon a t ε with the assignment map s t ↦ ( s t i , z t ∗ , i ) s_t\mapsto(s_t^i,z_t^{*,i}) s t ↦ ( s t i , z t ∗ , i ) ; hence π s e p ε ∈ Π \pi_{\mathrm{sep}}^\varepsilon\in\Pi π sep ε ∈ Π .
At this point, it suffices to verify that this policy yields the bound on the cost. In particular, we show that the expected cost-to-go function under this policy, denoted by J t N , ε ( s t , ν N ) J_t^{N,\varepsilon}(s_t,\nu^N) J t N , ε ( s t , ν N ) , is upper bounded as follows:
J t N , ε ( s t , ν N ) ≤ K [ j t ] ( μ t N , ν N ) + T − t T ε . J_t^{N,\varepsilon}(s_t,\nu^N)\le\mathcal{K}[j_t](\mu_t^N,\nu^N)+\frac{T-t}{T}\varepsilon. J t N , ε ( s t , ν N ) ≤ K [ j t ] ( μ t N , ν N ) + T T − t ε .
which gives the desired upper bound. We proceed by induction. The equality at time t = T t=T t = T is trivial, as the terminal cost does not depend on the policy but only on the terminal state. Suppose then the statement holds at time t + 1 t+1 t + 1 . By the policy-cost recursion (using the independence of the noise across time) and the induction hypothesis inside the expectation,
J t N , ε ( s t , ν N ) ≤ E w t [ 1 N ∑ i = 1 N g t ( s t i , a t i , w t i ) + K [ j t + 1 ] ( μ t + 1 N , ν N ) ] + T − t − 1 T ε = 1 N E w t [ ∑ i = 1 N g t ( s t i , a t i , w t i ) + min σ t + 1 ∈ S N ∑ i = 1 N j t + 1 ( f t ( s t i , a t i , w t i ) , z σ t + 1 ( i ) ) ] + T − t − 1 T ε ≤ 1 N ∑ i = 1 N E w t i [ g t ( s t i , a t i , w t i ) + j t + 1 ( f t ( s t i , a t i , w t i ) , z σ t ∗ ( i ) ) ] + T − t − 1 T ε ≤ 1 N ∑ i = 1 N j t ( s t i , z σ t ∗ ( i ) ) + ε T + T − t − 1 T ε = 1 N min σ ∈ S N ∑ i = 1 N j t ( s t i , z σ ( i ) ) + T − t T ε , \begin{aligned}
J_t^{N,\varepsilon}(s_t,\nu^N)
&\le\mathbb{E}_{w_t}\left[\frac{1}{N}\sum_{i=1}^N g_t(s_t^i,a_t^i,w_t^i)+\mathcal{K}[j_{t+1}](\mu_{t+1}^N,\nu^N)\right]+\frac{T-t-1}{T}\varepsilon\\
&=\frac{1}{N}\mathbb{E}_{w_t}\left[\sum_{i=1}^N g_t(s_t^i,a_t^i,w_t^i)+\min_{\sigma_{t+1}\in S_N}\sum_{i=1}^N j_{t+1}(f_t(s_t^i,a_t^i,w_t^i),z^{\sigma_{t+1}(i)})\right]+\frac{T-t-1}{T}\varepsilon\\
&\le\frac{1}{N}\sum_{i=1}^N\mathbb{E}_{w_t^i}\left[g_t(s_t^i,a_t^i,w_t^i)+j_{t+1}(f_t(s_t^i,a_t^i,w_t^i),z^{\sigma_t^*(i)})\right]+\frac{T-t-1}{T}\varepsilon\\
&\le\frac{1}{N}\sum_{i=1}^N j_t(s_t^i,z^{\sigma_t^*(i)})+\frac{\varepsilon}{T}+\frac{T-t-1}{T}\varepsilon\\
&=\frac{1}{N}\min_{\sigma\in S_N}\sum_{i=1}^N j_t(s_t^i,z^{\sigma(i)})+\frac{T-t}{T}\varepsilon,
\end{aligned} J t N , ε ( s t , ν N ) ≤ E w t [ N 1 i = 1 ∑ N g t ( s t i , a t i , w t i ) + K [ j t + 1 ] ( μ t + 1 N , ν N ) ] + T T − t − 1 ε = N 1 E w t [ i = 1 ∑ N g t ( s t i , a t i , w t i ) + σ t + 1 ∈ S N min i = 1 ∑ N j t + 1 ( f t ( s t i , a t i , w t i ) , z σ t + 1 ( i ) ) ] + T T − t − 1 ε ≤ N 1 i = 1 ∑ N E w t i [ g t ( s t i , a t i , w t i ) + j t + 1 ( f t ( s t i , a t i , w t i ) , z σ t ∗ ( i ) ) ] + T T − t − 1 ε ≤ N 1 i = 1 ∑ N j t ( s t i , z σ t ∗ ( i ) ) + T ε + T T − t − 1 ε = N 1 σ ∈ S N min i = 1 ∑ N j t ( s t i , z σ ( i ) ) + T T − t ε ,
where the first inequality combines the policy-cost recursion with the inductive assumption, the second inequality follows since σ t ∗ \sigma_t^* σ t ∗ is a potentially suboptimal permutation for the assignment at time t + 1 t+1 t + 1 , the last inequality follows by the ε T \frac{\varepsilon}{T} T ε -optimality of the action selector a t ε a_t^\varepsilon a t ε used by the policy π s e p , t ε \pi_{\mathrm{sep},t}^\varepsilon π sep , t ε , and the last equality follows from the definition of σ t ∗ \sigma_t^* σ t ∗ . Note that the expectation step above only uses the marginal law w t i ∼ ξ t ( ⋅ ∣ s t i , a t i ) w_t^i\sim\xi_t(\mathord\cdot\mid s_t^i,a_t^i) w t i ∼ ξ t ( ⋅ ∣ s t i , a t i ) of each agent’s noise together with the linearity of expectation; it is therefore valid under all noise models considered in this work (individual, local, and global), regardless of the correlation of the noise across agents. This gives
J t N , ε ( s t , ν N ) ≤ K [ j t ] ( μ t N , ν N ) + T − t T ε , J_t^{N,\varepsilon}(s_t,\nu^N) \leq\mathcal{K}[j_t](\mu_t^N,\nu^N)+\frac{T-t}{T}\varepsilon, J t N , ε ( s t , ν N ) ≤ K [ j t ] ( μ t N , ν N ) + T T − t ε ,
which completes the induction.
Since ε > 0 \varepsilon>0 ε > 0 is arbitrary, and J t N J_t^N J t N is the infimum of the expected cost over all admissible policies, letting ε ↓ 0 \varepsilon\downarrow0 ε ↓ 0 gives
J t N ( s t , ν N ) ≤ K [ j t ] ( μ t N , ν N ) . J_t^N(s_t,\nu^N)\leq\mathcal{K}[j_t](\mu_t^N,\nu^N). J t N ( s t , ν N ) ≤ K [ j t ] ( μ t N , ν N ) .
Quantifying the tightness of the optimal transport upper bound
The inequality in the proof of Theorem 3.1, Part I emerges from the fact that we remove the dependence of the target assignment on the realized noise. More precisely, we first apply Lemma B.6 to remove the transport plan dependence on the noise in the term ( ∗ ) (*) ( ∗ ) of (12), and then apply Lemma B.3 to allow the transport plan to be chosen before the noise is realized. In this appendix, we quantify the loss introduced by these two steps.
Throughout the appendix, we retain Assumptions A.1 and A.2 and write ξ ^ τ = ( ξ ^ τ s , a ) ( s , a ) \hat{\xi}_\tau=(\hat{\xi}_\tau^{s,a})_{(s,a)} ξ ^ τ = ( ξ ^ τ s , a ) ( s , a ) for the realized noise field. Recall that Assumption A.2 reads
E η τ [ ∫ W φ ( w ) d ξ ^ τ s , a ( w ) ] = ∫ W φ ( w ) d ξ τ ( w ∣ s , a ) \mathbb{E}_{\eta_\tau}\left[\int_W\varphi(w)\,\mathrm{d}\hat{\xi}_\tau^{s,a}(w)\right]=\int_W\varphi(w)\,\mathrm{d}\xi_\tau(w\mid s,a) E η τ [ ∫ W φ ( w ) d ξ ^ τ s , a ( w ) ] = ∫ W φ ( w ) d ξ τ ( w ∣ s , a )
for every bounded Borel function φ \varphi φ and almost every ( s , a ) (s,a) ( s , a ) . By truncation and monotone convergence, the same identity holds for every nonnegative Borel φ \varphi φ , which is the form we use below.
Assumption D.1 (Lipschitz single-agent cost-to-go). Equip S \mathcal{S} S with a metric d S d_{\mathcal{S}} d S . We assume that for every τ ∈ { 1 , … , T } \tau\in\{1,\ldots,T\} τ ∈ { 1 , … , T } and for some L τ < ∞ L_\tau<\infty L τ < ∞ ,
∣ j τ ( s , z ) − j τ ( s ′ , z ) ∣ ≤ L τ d S ( s , s ′ ) (18) \lvert j_\tau(s,z)-j_\tau(s',z)\rvert\leq L_\tau d_{\mathcal{S}}(s,s') \tag*{(18)} ∣ j τ ( s , z ) − j τ ( s ′ , z )∣ ≤ L τ d S ( s , s ′ ) (18)
for all s , s ′ ∈ S s,s'\in\mathcal{S} s , s ′ ∈ S and z ∈ Z z\in\mathcal{Z} z ∈ Z .
We need the constant to be uniform in z z z because the target assigned to an agent may itself be reassigned once the noise has been observed. The assumption holds, for instance, whenever all the underlying spaces are finite and the single-agent cost-to-go functions are finite-valued.
For each τ < T \tau<T τ < T , we measure the spread of the transition by
D τ : = sup ( s , a ) ∈ S × A ∫ W × W d S ( f τ ( s , a , w ) , f τ ( s , a , w ~ ) ) d ξ τ ( w ∣ s , a ) d ξ τ ( w ~ ∣ s , a ) , (19) D_\tau:=\sup_{(s,a)\in\mathcal{S}\times\mathcal{A}}\int_{W\times W}d_{\mathcal{S}}\bigl(f_\tau(s,a,w),f_\tau(s,a,\widetilde{w})\bigr)\,\mathrm{d}\xi_\tau(w\mid s,a)\mathrm{d}\xi_\tau(\widetilde{w}\mid s,a), \tag*{(19)} D τ := ( s , a ) ∈ S × A sup ∫ W × W d S ( f τ ( s , a , w ) , f τ ( s , a , w ) ) d ξ τ ( w ∣ s , a ) d ξ τ ( w ∣ s , a ) , (19)
that is, by the expected distance between two conditionally independent next states generated from the same state-action pair. We assume throughout that D τ D_\tau D τ , together with the expectations and transport costs appearing in the comparison below, is finite. This is the case, for example, when w ↦ f τ ( s , a , w ) w\mapsto f_\tau(s,a,w) w ↦ f τ ( s , a , w ) is Lipschitz uniformly in ( s , a ) (s,a) ( s , a ) , the nominal kernels have uniformly bounded first moments, and K [ j τ ] ( μ , ν ) < ∞ \mathcal{K}[j_\tau](\mu,\nu)<\infty K [ j τ ] ( μ , ν ) < ∞ , or more simply when all the underlying spaces are compact and data are continuous.
For the following analysis, it is convenient to write the recursion (4) in operator form. For a next-stage function F F F and ρ τ λ : = λ ⊗ ξ ^ τ s , a \rho_\tau^\lambda:=\lambda\otimes\hat{\xi}_\tau^{s,a} ρ τ λ := λ ⊗ ξ ^ τ s , a , we hence set
( T τ F ) ( μ , ν ) : = inf λ ∈ M ( μ ) E η τ [ ∫ S × A × W g τ ( s , a , w ) d ρ τ λ ( s , a , w ) + F ( f τ # ρ τ λ , ν ) ] , (\mathcal{T}_\tau F)(\mu,\nu):=\inf_{\lambda\in\mathcal{M}(\mu)}\mathbb{E}_{\eta_\tau}\left[\int_{\mathcal{S}\times\mathcal{A}\times W}g_\tau(s,a,w)\,\mathrm{d}\rho_\tau^\lambda(s,a,w)+F\bigl(f_{\tau\#}\rho_\tau^\lambda,\nu\bigr)\right], ( T τ F ) ( μ , ν ) := λ ∈ M ( μ ) inf E η τ [ ∫ S × A × W g τ ( s , a , w ) d ρ τ λ ( s , a , w ) + F ( f τ # ρ τ λ , ν ) ] ,
so that J τ = T τ J τ + 1 J_\tau=\mathcal{T}_\tau J_{\tau+1} J τ = T τ J τ + 1 by (4). The population Bellman operator T τ \mathcal{T}_\tau T τ satisfies the monotonicity and constant-shift properties that are standard for dynamic programming operators [31 Ch. 8] . More precisely, the following lemma holds.
Lemma D.2 (Monotonicity and constant shift of the population Bellman operator). Let F , F ~ : P ( S ) × P ( Z ) → R ‾ F,\widetilde{F}:\mathcal{P}(S)\times\mathcal{P}(Z)\to\overline{\mathbb{R}} F , F : P ( S ) × P ( Z ) → R be universally measurable and bounded below, and let c ∈ R c\in\mathbb{R} c ∈ R . Then
(i) F ≤ F ~ F\leq\widetilde{F} F ≤ F pointwise implies T τ F ≤ T τ F ~ \mathcal{T}_{\tau}F\leq\mathcal{T}_{\tau}\widetilde{F} T τ F ≤ T τ F pointwise;
(ii) T τ [ F + c ] = T τ [ F ] + c \mathcal{T}_{\tau}[F+c]=\mathcal{T}_{\tau}[F]+c T τ [ F + c ] = T τ [ F ] + c .
Proof. Both properties are immediate from the definition of T τ \mathcal{T}_{\tau} T τ : the next-stage function enters the right-hand side additively and monotonically, while a constant depends neither on the realized next population nor on λ \lambda λ , so it factors out of the expectation and out of the infimum. The argument uses only the structure of the definition.
Finally, we also recall that
K [ j τ ] ( μ , ν ) = inf γ ∈ Γ ( μ , ν ) ∫ S × Z j τ ( s , z ) d γ ( s , z ) . \mathcal{K}[j_{\tau}](\mu,\nu)=\inf_{\gamma\in\Gamma(\mu,\nu)}\int_{S\times Z}j_{\tau}(s,z)\,d\gamma(s,z). K [ j τ ] ( μ , ν ) = γ ∈ Γ ( μ , ν ) inf ∫ S × Z j τ ( s , z ) d γ ( s , z ) .
Here, γ \gamma γ assigns the mass in the current state distribution μ \mu μ to the target distribution ν \nu ν , and j τ ( s , z ) j_{\tau}(s,z) j τ ( s , z ) is the single-agent cost associated with each state–target pair. This assignment is made before the noise at time τ \tau τ is observed. In contrast, T τ [ K [ j τ + 1 ] ] \mathcal{T}_{\tau}[\mathcal{K}[j_{\tau+1}]] T τ [ K [ j τ + 1 ]] assigns targets after the realized noise has determined the next population. The difference in (20) is the cost of not being able to use that realized noise when assigning targets.
Lemma D.3 (One-step Bellman defect). Under Assumptions A.1, A.2, and D.1 , for every τ < T \tau<T τ < T , μ ∈ P ( S ) \mu\in\mathcal{P}(S) μ ∈ P ( S ) , and ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) , it holds that
0 ≤ K [ j τ ] ( μ , ν ) − T τ [ K [ j τ + 1 ] ] ( μ , ν ) ≤ L τ + 1 D τ . (20) 0\leq\mathcal{K}[j_{\tau}](\mu,\nu)-\mathcal{T}_{\tau}[\mathcal{K}[j_{\tau+1}]](\mu,\nu)\leq L_{\tau+1}D_{\tau}. \tag*{(20)} 0 ≤ K [ j τ ] ( μ , ν ) − T τ [ K [ j τ + 1 ]] ( μ , ν ) ≤ L τ + 1 D τ . (20)
Proof. Throughout this proof we abbreviate the state-action pair as x = ( s , a ) x=(s,a) x = ( s , a ) . We start by showing that the two quantities in (20) have the same expected stage cost. As a consequence, we only need to compare how the next-stage single-agent cost-to-go j τ + 1 ( f τ ( s , a , w ) , z ) j_{\tau+1}(f_{\tau}(s,a,w),z) j τ + 1 ( f τ ( s , a , w ) , z ) is minimized in the two transport problems. In K [ j τ ] ( μ , ν ) \mathcal{K}[j_{\tau}](\mu,\nu) K [ j τ ] ( μ , ν ) , the target assignment depends only on x x x , whereas in T τ [ K [ j τ + 1 ] ] ( μ , ν ) \mathcal{T}_{\tau}[\mathcal{K}[j_{\tau+1}]](\mu,\nu) T τ [ K [ j τ + 1 ]] ( μ , ν ) it may also depend on the realized noise. The second minimization is therefore over a larger set of assignments and cannot have a larger value. In the following proof, we show that the difference between the two values is at most L τ + 1 D τ L_{\tau+1}D_{\tau} L τ + 1 D τ .
Step 1: Define the two next-stage transport problems. Fix λ ∈ M ( μ ) \lambda\in M(\mu) λ ∈ M ( μ ) . If the target is assigned using x x x alone, the expected next-stage cost-to-go is
A τ ( λ ) : = inf γ ∈ Γ ( λ , ν ) ∫ ( S × A ) × Z ∫ W j τ + 1 ( f τ ( s , a , w ~ ) , z ) d ξ τ ( w ~ ∣ x ) d γ ( x , z ) . A_{\tau}(\lambda):=\inf_{\gamma\in\Gamma(\lambda,\nu)}\int_{(S\times A)\times Z}\int_{W}j_{\tau+1}(f_{\tau}(s,a,\widetilde{w}),z)\,d\xi_{\tau}(\widetilde{w}\mid x)\,d\gamma(x,z). A τ ( λ ) := γ ∈ Γ ( λ , ν ) inf ∫ ( S × A ) × Z ∫ W j τ + 1 ( f τ ( s , a , w ) , z ) d ξ τ ( w ∣ x ) d γ ( x , z ) .
Here, the plan γ \gamma γ assigns a target z z z to each x = ( s , a ) x=(s,a) x = ( s , a ) , before the noise w ~ \widetilde{w} w is drawn from the nominal kernel. Hence, this target assignment does not depend on w ~ \widetilde{w} w .
On the other hand, if the noise field is observed before the target is assigned, the transport plan may couple the target with the full pair ( x , w ) (x,w) ( x , w ) . For a fixed realization of the noise field, the smallest next-stage cost-to-go is then
A τ ( λ ; ξ ^ τ ) : = inf γ ′ ∈ Γ ( λ ⊗ ξ ^ τ x , ν ) ∫ ( S × A ) × W × Z j τ + 1 ( f τ ( s , a , w ) , z ) d γ ′ ( x , w , z ) . A_{\tau}(\lambda;\widehat{\xi}_{\tau}):=\inf_{\gamma'\in\Gamma(\lambda\otimes\widehat{\xi}_{\tau}^{x},\nu)}\int_{(S\times A)\times W\times Z}j_{\tau+1}(f_{\tau}(s,a,w),z)\,d\gamma'(x,w,z). A τ ( λ ; ξ τ ) := γ ′ ∈ Γ ( λ ⊗ ξ τ x , ν ) inf ∫ ( S × A ) × W × Z j τ + 1 ( f τ ( s , a , w ) , z ) d γ ′ ( x , w , z ) .
Indeed, since the first marginal of γ ′ \gamma' γ ′ is λ ⊗ ξ ^ τ x \lambda\otimes\widehat{\xi}_{\tau}^{x} λ ⊗ ξ τ x , the transport plan γ ′ \gamma' γ ′ assigns a target after both x x x and its realized noise w w w are known.
Step 2: Decompose the transport cost. We now verify that these two quantities are exactly the next-stage terms that appear in (20). We begin with K [ j τ ] ( μ , ν ) \mathcal{K}[j_{\tau}](\mu,\nu) K [ j τ ] ( μ , ν ) . Define
c τ ( s , a , z ) : = ∫ W g τ ( s , a , w ) + j τ + 1 ( f τ ( s , a , w ) , z ) d ξ τ ( w ∣ s , a ) . c_{\tau}(s,a,z):=\int_{W}g_{\tau}(s,a,w)+j_{\tau+1}(f_{\tau}(s,a,w),z)\,d\xi_{\tau}(w\mid s,a). c τ ( s , a , z ) := ∫ W g τ ( s , a , w ) + j τ + 1 ( f τ ( s , a , w ) , z ) d ξ τ ( w ∣ s , a ) .
By the single-agent Bellman recursion we have
j τ ( s , z ) = inf a ∈ A c τ ( s , a , z ) . (21) j_{\tau}(s,z)=\inf_{a\in A}c_{\tau}(s,a,z). \tag*{(21)} j τ ( s , z ) = a ∈ A inf c τ ( s , a , z ) . (21)
and plugging it into the definition of the transport cost gives
K [ j τ ] ( μ , ν ) = inf γ 1 ∈ Γ ( μ , ν ) ∫ S × Z inf a ∈ A c τ ( s , a , z ) d γ 1 ( s , z ) . K[j_\tau](\mu,\nu)=\inf_{\gamma_1\in\Gamma(\mu,\nu)}\int_{S\times Z}\inf_{a\in A}c_\tau(s,a,z)\,\mathrm{d}\gamma_1(s,z). K [ j τ ] ( μ , ν ) = γ 1 ∈ Γ ( μ , ν ) inf ∫ S × Z a ∈ A inf c τ ( s , a , z ) d γ 1 ( s , z ) .
The function c τ c_\tau c τ is lower semianalytic under the conventions of Appendix B, therefore we can apply Lemma B.5. In particular, for every fixed γ 1 ∈ Γ ( μ , ν ) \gamma_1\in\Gamma(\mu,\nu) γ 1 ∈ Γ ( μ , ν ) ,
∫ S × Z inf a ∈ A c τ ( s , a , z ) d γ 1 ( s , z ) = inf { α s , z } ∈ Λ ( A ) ∫ S × Z ∫ A c τ ( s , a , z ) d α s , z ( a ) d γ 1 ( s , z ) . \int_{S\times Z}\inf_{a\in A}c_\tau(s,a,z)\,\mathrm{d}\gamma_1(s,z)=\inf_{\{\alpha^{s,z}\}\in\Lambda(A)}\int_{S\times Z}\int_A c_\tau(s,a,z)\,\mathrm{d}\alpha^{s,z}(a)\,\mathrm{d}\gamma_1(s,z). ∫ S × Z a ∈ A inf c τ ( s , a , z ) d γ 1 ( s , z ) = { α s , z } ∈ Λ ( A ) inf ∫ S × Z ∫ A c τ ( s , a , z ) d α s , z ( a ) d γ 1 ( s , z ) .
Here, α s , z \alpha^{s,z} α s , z is a Borel kernel that chooses an action for each state–target pair. Now, the pair ( γ 1 , α s , z ) (\gamma_1,\alpha^{s,z}) ( γ 1 , α s , z ) defines a measure γ \gamma γ on ( S × A ) × Z (S\times A)\times Z ( S × A ) × Z by
d γ ( s , a , z ) = d α s , z ( a ) d γ 1 ( s , z ) , \mathrm{d}\gamma(s,a,z)=\mathrm{d}\alpha^{s,z}(a)\mathrm{d}\gamma_1(s,z), d γ ( s , a , z ) = d α s , z ( a ) d γ 1 ( s , z ) ,
and we can rewrite its marginals in the form used by the population Bellman problem. To do so, let λ \lambda λ be the ( s , a ) (s,a) ( s , a ) -marginal of γ \gamma γ . Since the state marginal of γ 1 \gamma_1 γ 1 is μ \mu μ , the state marginal of λ \lambda λ is also μ \mu μ , and therefore λ ∈ M ( μ ) \lambda\in M(\mu) λ ∈ M ( μ ) . Combining this with the fact that the target marginal of γ \gamma γ is ν \nu ν , we obtain γ ∈ Γ ( λ , ν ) \gamma\in\Gamma(\lambda,\nu) γ ∈ Γ ( λ , ν ) . Conversely, from every pair consisting of λ ∈ M ( μ ) \lambda\in M(\mu) λ ∈ M ( μ ) and γ ∈ Γ ( λ , ν ) \gamma\in\Gamma(\lambda,\nu) γ ∈ Γ ( λ , ν ) , we can obtain a coupling γ 1 ∈ Γ ( μ , ν ) \gamma_1\in\Gamma(\mu,\nu) γ 1 ∈ Γ ( μ , ν ) by taking the ( s , z ) (s,z) ( s , z ) -marginal of γ \gamma γ , and a Borel action kernel α s , z \alpha^{s,z} α s , z by disintegrating γ \gamma γ with respect to ( s , z ) (s,z) ( s , z ) . Therefore, optimizing over ( γ 1 , α s , z ) (\gamma_1,\alpha^{s,z}) ( γ 1 , α s , z ) is equivalent to optimizing over λ ∈ M ( μ ) \lambda\in M(\mu) λ ∈ M ( μ ) and γ ∈ Γ ( λ , ν ) \gamma\in\Gamma(\lambda,\nu) γ ∈ Γ ( λ , ν ) .
The transport problem can therefore be written as
K [ j τ ] ( μ , ν ) = inf λ ∈ M ( μ ) inf γ ∈ Γ ( λ , ν ) ∫ ( S × A ) × Z c τ ( s , a , z ) d γ ( s , a , z ) . K[j_\tau](\mu,\nu)=\inf_{\lambda\in M(\mu)}\inf_{\gamma\in\Gamma(\lambda,\nu)}\int_{(S\times A)\times Z}c_\tau(s,a,z)\,\mathrm{d}\gamma(s,a,z). K [ j τ ] ( μ , ν ) = λ ∈ M ( μ ) inf γ ∈ Γ ( λ , ν ) inf ∫ ( S × A ) × Z c τ ( s , a , z ) d γ ( s , a , z ) .
The outer infimum chooses an admissible state-action distribution λ \lambda λ , while the inner infimum assigns its mass to the target distribution ν \nu ν .
We can now expand the integral of c τ c_\tau c τ . Since the stage cost does not depend on the target z z z , its integral depends only on the ( s , a ) (s,a) ( s , a ) -marginal λ \lambda λ . We denote it as
G τ ( λ ) : = ∫ S × A ∫ W g τ ( s , a , w ) d ξ τ ( w ∣ x ) d λ ( x ) . G_\tau(\lambda):=\int_{S\times A}\int_W g_\tau(s,a,w)\,\mathrm{d}\xi_\tau(w\mid x)\,\mathrm{d}\lambda(x). G τ ( λ ) := ∫ S × A ∫ W g τ ( s , a , w ) d ξ τ ( w ∣ x ) d λ ( x ) .
Therefore
∫ ( S × A ) × Z c τ ( s , a , z ) d γ ( x , z ) = G τ ( λ ) + ∫ ( S × A ) × Z ∫ W j τ + 1 ( f τ ( s , a , w ~ ) , z ) d ξ τ ( w ~ ∣ x ) d γ ( x , z ) . \int_{(S\times A)\times Z}c_\tau(s,a,z)\,\mathrm{d}\gamma(x,z)=G_\tau(\lambda)+\int_{(S\times A)\times Z}\int_W j_{\tau+1}(f_\tau(s,a,\widetilde{w}),z)\,\mathrm{d}\xi_\tau(\widetilde{w}\mid x)\,\mathrm{d}\gamma(x,z). ∫ ( S × A ) × Z c τ ( s , a , z ) d γ ( x , z ) = G τ ( λ ) + ∫ ( S × A ) × Z ∫ W j τ + 1 ( f τ ( s , a , w ) , z ) d ξ τ ( w ∣ x ) d γ ( x , z ) .
For a given λ \lambda λ , the first term is fixed. Minimizing the second term over γ ∈ Γ ( λ , ν ) \gamma\in\Gamma(\lambda,\nu) γ ∈ Γ ( λ , ν ) gives A τ ( λ ) A_\tau(\lambda) A τ ( λ ) , and minimizing again over λ ∈ M ( μ ) \lambda\in M(\mu) λ ∈ M ( μ ) gives
K [ j τ ] ( μ , ν ) = inf λ ∈ M ( μ ) { G τ ( λ ) + A τ ( λ ) } . (22) K[j_\tau](\mu,\nu)=\inf_{\lambda\in M(\mu)}\{G_\tau(\lambda)+A_\tau(\lambda)\}. \tag*{(22)} K [ j τ ] ( μ , ν ) = λ ∈ M ( μ ) inf { G τ ( λ ) + A τ ( λ )} . (22)
Step 3: Decompose the population Bellman term. We now turn to the second quantity in (20). From the definition of T τ T_\tau T τ ,
T τ [ K [ j τ + 1 ] ] ( μ , ν ) = inf λ ∈ M ( μ ) E η τ [ ∫ S × A × W g τ ( s , a , w ) d ( λ ⊗ ξ ^ τ x ) ( x , w ) + K [ j τ + 1 ] ( f τ # λ ⊗ ξ ^ τ x , ν ) ] . T_\tau[K[j_{\tau+1}]](\mu,\nu)=\inf_{\lambda\in M(\mu)}\mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}g_\tau(s,a,w)\,\mathrm{d}\left(\lambda\otimes\widehat{\xi}_\tau^x\right)(x,w)+K[j_{\tau+1}]\left(f_{\tau\#}\lambda\otimes\widehat{\xi}_\tau^x,\nu\right)\right]. T τ [ K [ j τ + 1 ]] ( μ , ν ) = λ ∈ M ( μ ) inf E η τ [ ∫ S × A × W g τ ( s , a , w ) d ( λ ⊗ ξ τ x ) ( x , w ) + K [ j τ + 1 ] ( f τ # λ ⊗ ξ τ x , ν ) ] .
We look at the two terms in the expectation separately. The expectation of the stage cost can be rewritten using Assumption A.2 as
E η τ [ ∫ S × A × W g τ ( s , a , w ) d ( λ ⊗ ξ ^ τ x ) ( x , w ) ] = G τ ( λ ) . \mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}g_\tau(s,a,w)\,\mathrm{d}\left(\lambda\otimes\widehat{\xi}_\tau^x\right)(x,w)\right]=G_\tau(\lambda). E η τ [ ∫ S × A × W g τ ( s , a , w ) d ( λ ⊗ ξ τ x ) ( x , w ) ] = G τ ( λ ) .
For the transport term, we fix λ ∈ M ( μ ) \lambda\in M(\mu) λ ∈ M ( μ ) and a realization of the noise field. Since the term couples the next-state distribution f τ # λ ⊗ ξ ^ τ x f_{\tau\#}\lambda\otimes\widehat{\xi}_\tau^x f τ # λ ⊗ ξ τ x with the target distribution ν \nu ν , we can apply Lemma B.1. The lemma allows us to describe the same transport problem before applying the dynamics by coupling ( x , w ) (x,w) ( x , w ) directly with a target z z z and evaluating the cost at the resulting next state f τ ( s , a , w ) f_\tau(s,a,w) f τ ( s , a , w ) . By doing so we directly obtain A τ ( λ ; ξ ^ τ ) A_\tau(\lambda;\hat{\xi}_\tau) A τ ( λ ; ξ ^ τ ) . In particular, we have
K [ J τ + 1 ] ( f τ # λ ⊗ ξ ^ τ x , ν ) = inf γ ′ ∈ Γ ( λ ⊗ ξ ^ τ x , ν ) ∫ ( S × A ) × W × Z J τ + 1 ( f τ ( s , a , w ) , z ) d γ ′ ( x , w , z ) = A τ ( λ ; ξ ^ τ ) . \mathcal{K}[\mathcal{J}_{\tau+1}](f_\tau\#\lambda\otimes\hat{\xi}_\tau^x,\nu)
=
\inf_{\gamma'\in\Gamma(\lambda\otimes\hat{\xi}_\tau^x,\nu)}
\int_{(S\times A)\times W\times Z}\mathcal{J}_{\tau+1}(f_\tau(s,a,w),z)\,\mathrm{d}\gamma'(x,w,z)
=
A_\tau(\lambda;\hat{\xi}_\tau). K [ J τ + 1 ] ( f τ # λ ⊗ ξ ^ τ x , ν ) = γ ′ ∈ Γ ( λ ⊗ ξ ^ τ x , ν ) inf ∫ ( S × A ) × W × Z J τ + 1 ( f τ ( s , a , w ) , z ) d γ ′ ( x , w , z ) = A τ ( λ ; ξ ^ τ ) .
Substituting these two equalities into the definition of T τ \mathcal{T}_\tau T τ gives
T τ [ K [ J τ + 1 ] ] ( μ , ν ) = inf λ ∈ M ( μ ) { G τ ( λ ) + E η τ [ A τ ( λ ; ξ ^ τ ) ] } . (23) \mathcal{T}_\tau[\mathcal{K}[\mathcal{J}_{\tau+1}]](\mu,\nu)
=
\inf_{\lambda\in\mathcal{M}(\mu)}
\left\{
G_\tau(\lambda)+\mathbb{E}_{\eta_\tau}\left[A_\tau(\lambda;\hat{\xi}_\tau)\right]
\right\}. \tag*{(23)} T τ [ K [ J τ + 1 ]] ( μ , ν ) = λ ∈ M ( μ ) inf { G τ ( λ ) + E η τ [ A τ ( λ ; ξ ^ τ ) ] } . (23)
Step 4: Compare the next-stage costs for a fixed λ \lambda λ . Now, since (21) and (22) contain the same stage-cost integrals G τ ( λ ) G_\tau(\lambda) G τ ( λ ) , it is enough to show that, for every λ ∈ M ( μ ) \lambda\in\mathcal{M}(\mu) λ ∈ M ( μ ) ,
0 ≤ A τ ( λ ) − E η τ [ A τ ( λ ; ξ ^ τ ) ] ≤ L τ + 1 D τ . (24) 0\leq A_\tau(\lambda)-\mathbb{E}_{\eta_\tau}\left[A_\tau(\lambda;\hat{\xi}_\tau)\right]\leq L_{\tau+1}D_\tau. \tag*{(24)} 0 ≤ A τ ( λ ) − E η τ [ A τ ( λ ; ξ ^ τ ) ] ≤ L τ + 1 D τ . (24)
Indeed, after adding the common stage-cost term, the inequalities remain valid for every λ \lambda λ , and taking the infimum over λ ∈ M ( μ ) \lambda\in\mathcal{M}(\mu) λ ∈ M ( μ ) gives (20).
Upper bound. We first prove the upper bound. Fix a realization of the noise field and take any γ ′ \gamma' γ ′ admissible for A τ ( λ ; ξ ^ τ ) A_\tau(\lambda;\hat{\xi}_\tau) A τ ( λ ; ξ ^ τ ) . Since x = ( s , a ) x=(s,a) x = ( s , a ) and the ( x , z ) (x,z) ( x , z ) -marginal of γ ′ \gamma' γ ′ has marginals λ \lambda λ and ν \nu ν , respectively, it is admissible for A τ ( λ ) A_\tau(\lambda) A τ ( λ ) . We can therefore evaluate A τ ( λ ) A_\tau(\lambda) A τ ( λ ) at this ( x , z ) (x,z) ( x , z ) -marginal and draw w ~ \widetilde{w} w independently from ξ τ ( ⋅ ∣ x ) \xi_\tau(\cdot\mid x) ξ τ ( ⋅ ∣ x ) , which gives
A τ ( λ ) ≤ ∫ ( S × A ) × W × Z ∫ W J τ + 1 ( f τ ( s , a , w ~ ) , z ) d ξ τ ( w ~ ∣ x ) d γ ′ ( x , w , z ) ≤ ∫ ( S × A ) × W × Z J τ + 1 ( f τ ( s , a , w ) , z ) d γ ′ ( x , w , z ) + L τ + 1 ∫ ( S × A ) × W ∫ W d S ( f τ ( s , a , w ) , f τ ( s , a , w ~ ) ) d ξ τ ( w ~ ∣ x ) d ( λ ⊗ ξ ^ τ x ) ( x , w ) , \begin{aligned}
A_\tau(\lambda)
&\leq\int_{(S\times A)\times W\times Z}\int_W
\mathcal{J}_{\tau+1}(f_\tau(s,a,\widetilde{w}),z)\,\mathrm{d}\xi_\tau(\widetilde{w}\mid x)\,\mathrm{d}\gamma'(x,w,z)\\
&\leq\int_{(S\times A)\times W\times Z}
\mathcal{J}_{\tau+1}(f_\tau(s,a,w),z)\,\mathrm{d}\gamma'(x,w,z)\\
&\quad+L_{\tau+1}\int_{(S\times A)\times W}\int_W
d_S(f_\tau(s,a,w),f_\tau(s,a,\widetilde{w}))\,\mathrm{d}\xi_\tau(\widetilde{w}\mid x)\,\mathrm{d}(\lambda\otimes\hat{\xi}_\tau^x)(x,w),
\end{aligned} A τ ( λ ) ≤ ∫ ( S × A ) × W × Z ∫ W J τ + 1 ( f τ ( s , a , w ) , z ) d ξ τ ( w ∣ x ) d γ ′ ( x , w , z ) ≤ ∫ ( S × A ) × W × Z J τ + 1 ( f τ ( s , a , w ) , z ) d γ ′ ( x , w , z ) + L τ + 1 ∫ ( S × A ) × W ∫ W d S ( f τ ( s , a , w ) , f τ ( s , a , w )) d ξ τ ( w ∣ x ) d ( λ ⊗ ξ ^ τ x ) ( x , w ) ,
where the second inequality follows from Assumption D.1.
Looking at the obtained upper bound, we can observe that the first integral is the value of a fixed γ ′ \gamma' γ ′ in the definition of A τ ( λ ; ξ ^ τ ) A_\tau(\lambda;\hat{\xi}_\tau) A τ ( λ ; ξ ^ τ ) . The last integral, instead, does not contain z z z , and so does not depend on how γ ′ \gamma' γ ′ assigns targets. We can therefore minimize over γ ′ \gamma' γ ′ and obtain
A τ ( λ ) ≤ A τ ( λ ; ξ ^ τ ) + L τ + 1 ∫ ( S × A ) × W ∫ W d S ( f τ ( s , a , w ) , f τ ( s , a , w ~ ) ) d ξ τ ( w ~ ∣ x ) d ( λ ⊗ ξ ^ τ x ) ( x , w ) . \begin{aligned}
A_\tau(\lambda)
&\leq A_\tau(\lambda;\hat{\xi}_\tau)\\
&\quad+L_{\tau+1}\int_{(S\times A)\times W}\int_W
d_S(f_\tau(s,a,w),f_\tau(s,a,\widetilde{w}))\,\mathrm{d}\xi_\tau(\widetilde{w}\mid x)\,\mathrm{d}(\lambda\otimes\hat{\xi}_\tau^x)(x,w).
\end{aligned} A τ ( λ ) ≤ A τ ( λ ; ξ ^ τ ) + L τ + 1 ∫ ( S × A ) × W ∫ W d S ( f τ ( s , a , w ) , f τ ( s , a , w )) d ξ τ ( w ∣ x ) d ( λ ⊗ ξ ^ τ x ) ( x , w ) .
Averaging over the noise field and applying Assumption A.2 to replace ξ ^ τ x \hat{\xi}_\tau^x ξ ^ τ x by ξ τ ( ⋅ ∣ x ) \xi_\tau(\cdot\mid x) ξ τ ( ⋅ ∣ x ) in the last integral gives
E η τ [ ∫ ( S × A ) × W ∫ W d S ( f τ ( s , a , w ) , f τ ( s , a , w ~ ) ) d ξ τ ( w ~ ∣ x ) d ( λ ⊗ ξ ^ τ x ) ( x , w ) ] = ∫ S × A ∫ W × W d S ( f τ ( s , a , w ) , f τ ( s , a , w ~ ) ) d ξ τ ( w ∣ x ) d ξ τ ( w ~ ∣ x ) d λ ( x ) ≤ D τ , \begin{aligned}
\mathbb{E}_{\eta_\tau}\left[
\int_{(S\times A)\times W}\int_W
d_S(f_\tau(s,a,w),f_\tau(s,a,\widetilde{w}))\,\mathrm{d}\xi_\tau(\widetilde{w}\mid x)\,\mathrm{d}(\lambda\otimes\hat{\xi}_\tau^x)(x,w)
\right]
&=
\int_{S\times A}\int_{W\times W}
d_S(f_\tau(s,a,w),f_\tau(s,a,\widetilde{w}))\,\mathrm{d}\xi_\tau(w\mid x)\,\mathrm{d}\xi_\tau(\widetilde{w}\mid x)\,\mathrm{d}\lambda(x)\\
&\leq D_\tau,
\end{aligned} E η τ [ ∫ ( S × A ) × W ∫ W d S ( f τ ( s , a , w ) , f τ ( s , a , w )) d ξ τ ( w ∣ x ) d ( λ ⊗ ξ ^ τ x ) ( x , w ) ] = ∫ S × A ∫ W × W d S ( f τ ( s , a , w ) , f τ ( s , a , w )) d ξ τ ( w ∣ x ) d ξ τ ( w ∣ x ) d λ ( x ) ≤ D τ ,
where the last inequality follows from the supremum in (19). We have therefore proved the upper bound
A τ ( λ ) ≤ E η τ [ A τ ( λ ; ξ ^ τ ) ] + L τ + 1 D τ . A_\tau(\lambda)\leq\mathbb{E}_{\eta_\tau}\left[A_\tau(\lambda;\hat{\xi}_\tau)\right]+L_{\tau+1}D_\tau. A τ ( λ ) ≤ E η τ [ A τ ( λ ; ξ ^ τ ) ] + L τ + 1 D τ .
Lower bound. For the lower bound in (24) , fix any γ ∈ Γ ( λ , ν ) \gamma\in\Gamma(\lambda,\nu) γ ∈ Γ ( λ , ν ) . After the noise field is realized, we can couple this plan with the realized noise kernel by setting
d γ τ ′ ξ ^ ( x , w , z ) : = d γ y ( x , z ) d ξ ^ τ x ( w ) . \mathrm{d}\gamma_{\tau}^{\prime\hat{\xi}}(x,w,z) := \mathrm{d}\gamma^{y}(x,z)\mathrm{d}\hat{\xi}_{\tau}^{x}(w). d γ τ ′ ξ ^ ( x , w , z ) := d γ y ( x , z ) d ξ ^ τ x ( w ) .
The resulting plan is admissible for A τ ( λ ; ξ ^ τ ) A_{\tau}(\lambda;\hat{\xi}_{\tau}) A τ ( λ ; ξ ^ τ ) since its ( x , w ) (x,w) ( x , w ) -marginal is λ ⊗ ξ ^ τ x \lambda\otimes\hat{\xi}_{\tau}^{x} λ ⊗ ξ ^ τ x , because the x x x -marginal of γ \gamma γ is λ \lambda λ , and its target marginal is ν \nu ν . Evaluating the infimum at this plan, averaging over the noise field, and then applying Lemma A.10 with ( x , z ) (x,z) ( x , z ) as the base variable and j τ + 1 ( f τ ( s , a , w ) , z ) j_{\tau+1}(f_{\tau}(s,a,w),z) j τ + 1 ( f τ ( s , a , w ) , z ) as the cost, gives
E η τ [ A τ ( λ ; ξ ^ τ ) ] ≤ E η τ [ ∫ ( S × A ) × W × Z j τ + 1 ( f τ ( s , a , w ) , z ) d γ τ ′ ξ ^ τ ( x , w , z ) ] = ∫ ( S × A ) × Z ∫ W j τ + 1 ( f τ ( s , a , w ~ ) , z ) d ξ τ ( w ~ ∣ x ) d γ ( x , z ) . \begin{aligned}
\mathbb{E}_{\eta_{\tau}}\left[A_{\tau}(\lambda;\hat{\xi}_{\tau})\right]\leq\mathbb{E}_{\eta_{\tau}}\left[\int_{(S\times A)\times W\times Z}j_{\tau+1}(f_{\tau}(s,a,w),z)\mathrm{d}\gamma_{\tau}^{\prime\hat{\xi}_{\tau}}(x,w,z)\right] \\
&=\int_{(S\times A)\times Z}\int_{W}j_{\tau+1}(f_{\tau}(s,a,\widetilde{w}),z)\mathrm{d}\xi_{\tau}(\widetilde{w}\mid x)\mathrm{d}\gamma(x,z).
\end{aligned} E η τ [ A τ ( λ ; ξ ^ τ ) ] ≤ E η τ [ ∫ ( S × A ) × W × Z j τ + 1 ( f τ ( s , a , w ) , z ) d γ τ ′ ξ ^ τ ( x , w , z ) ] = ∫ ( S × A ) × Z ∫ W j τ + 1 ( f τ ( s , a , w ) , z ) d ξ τ ( w ∣ x ) d γ ( x , z ) .
The right-hand side corresponds exactly to the value of the fixed γ \gamma γ in the definition of A τ ( λ ) A_{\tau}(\lambda) A τ ( λ ) , and, since the inequality holds for every γ ∈ Γ ( λ , ν ) \gamma\in\Gamma(\lambda,\nu) γ ∈ Γ ( λ , ν ) , minimizing over γ \gamma γ yields
E η τ [ A τ ( λ ; ξ ^ τ ) ] ≤ A τ ( λ ) . \mathbb{E}_{\eta_{\tau}}\left[A_{\tau}(\lambda;\hat{\xi}_{\tau})\right]\leq A_{\tau}(\lambda). E η τ [ A τ ( λ ; ξ ^ τ ) ] ≤ A τ ( λ ) .
Step 5: Return to the two Bellman quantities. This proves (24) . To conclude, for every λ ∈ M ( μ ) \lambda\in M(\mu) λ ∈ M ( μ ) , adding the common stage-cost expectation to (24) gives
G τ ( λ ) + E η τ [ A τ ( λ ; ξ ^ τ ) ] ≤ G τ ( λ ) + A τ ( λ ) ≤ G τ ( λ ) + E η τ [ A τ ( λ ; ξ ^ τ ) ] + L τ + 1 D τ . \begin{aligned}
G_{\tau}(\lambda)+\mathbb{E}_{\eta_{\tau}}\left[A_{\tau}(\lambda;\hat{\xi}_{\tau})\right]\leq G_{\tau}(\lambda)+A_{\tau}(\lambda) \\
&\leq G_{\tau}(\lambda)+\mathbb{E}_{\eta_{\tau}}\left[A_{\tau}(\lambda;\hat{\xi}_{\tau})\right]+L_{\tau+1}D_{\tau}.
\end{aligned} G τ ( λ ) + E η τ [ A τ ( λ ; ξ ^ τ ) ] ≤ G τ ( λ ) + A τ ( λ ) ≤ G τ ( λ ) + E η τ [ A τ ( λ ; ξ ^ τ ) ] + L τ + 1 D τ .
Taking the infimum over λ \lambda λ in all three terms and using (22) and (23) gives exactly (20) . □
Lemma D.3 bounds, for one time step, the difference between assigning targets before and after the noise is observed. Applying this bound successively from time t t t to time T − 1 T-1 T − 1 gives the total difference stated in the following theorem.
Theorem D.4 (Quantification of the separation gap). Under Assumptions A.1, A.2, and D.1, for every t ∈ { 0 , … , T } t \in\{0,\ldots,T\} t ∈ { 0 , … , T } , every μ t ∈ P ( S ) \mu_t \in\mathcal{P}(S) μ t ∈ P ( S ) , and every ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) ,
0 ≤ K [ j t ] ( μ t , ν ) − J t ( μ t , ν ) ≤ ∑ τ = t T − 1 L τ + 1 D τ . (25) 0\leq K[j_t](\mu_t,\nu)-J_t(\mu_t,\nu)\leq\sum_{\tau=t}^{T-1}L_{\tau+1}D_{\tau}. \tag*{(25)} 0 ≤ K [ j t ] ( μ t , ν ) − J t ( μ t , ν ) ≤ τ = t ∑ T − 1 L τ + 1 D τ . (25)
Proof. We proceed by backward induction. At the horizon the two sides agree, since j T = g T j_T=g_T j T = g T and therefore K [ j T ] ( μ , ν ) = J T ( μ , ν ) K[j_T](\mu,\nu)=J_T(\mu,\nu) K [ j T ] ( μ , ν ) = J T ( μ , ν ) . Assume next that (25) holds at time τ + 1 \tau+1 τ + 1 , pointwise in the population and target distributions, so that
0 ≤ K [ j τ + 1 ] − J τ + 1 ≤ ∑ r = τ + 1 T − 1 L r + 1 D r . 0\leq K[j_{\tau+1}]-J_{\tau+1}\leq\sum_{r=\tau+1}^{T-1}L_{r+1}D_r. 0 ≤ K [ j τ + 1 ] − J τ + 1 ≤ r = τ + 1 ∑ T − 1 L r + 1 D r .
Since T τ \mathcal{T}_{\tau} T τ is monotone and commutes with the addition of constants by Lemma D.2 , applying it to both sides gives
0 ≤ T τ [ K [ j τ + 1 ] ] − T τ J τ + 1 ≤ ∑ r = τ + 1 T − 1 L r + 1 D r . (26) 0\leq\mathcal{T}_{\tau}[K[j_{\tau+1}]]-\mathcal{T}_{\tau}J_{\tau+1}\leq\sum_{r=\tau+1}^{T-1}L_{r+1}D_r. \tag*{(26)} 0 ≤ T τ [ K [ j τ + 1 ]] − T τ J τ + 1 ≤ r = τ + 1 ∑ T − 1 L r + 1 D r . (26)
We can now compare K [ j τ ] − J τ K[j_{\tau}]-J_{\tau} K [ j τ ] − J τ . In particular, using J τ = T τ J τ + 1 J_{\tau}=\mathcal{T}_{\tau}J_{\tau+1} J τ = T τ J τ + 1 and (20) and (26) , we obtain
0 ≤ K [ j τ ] − J τ ≤ ( K [ j τ ] − T τ [ K [ j τ + 1 ] ] ) + ( T τ [ K [ j τ + 1 ] ] − T τ J τ + 1 ) ≤ L τ + 1 D τ + ∑ r = τ + 1 T − 1 L r + 1 D r = ∑ r = τ T − 1 L r + 1 D r , \begin{aligned}
0\leq K[j_{\tau}]-J_{\tau} \\
&\leq\left(K[j_{\tau}]-\mathcal{T}_{\tau}[K[j_{\tau+1}]]\right)+\left(\mathcal{T}_{\tau}[K[j_{\tau+1}]]-\mathcal{T}_{\tau}J_{\tau+1}\right) \\
&\leq L_{\tau+1}D_{\tau}+\sum_{r=\tau+1}^{T-1}L_{r+1}D_r=\sum_{r=\tau}^{T-1}L_{r+1}D_r,
\end{aligned} 0 ≤ K [ j τ ] − J τ ≤ ( K [ j τ ] − T τ [ K [ j τ + 1 ]] ) + ( T τ [ K [ j τ + 1 ]] − T τ J τ + 1 ) ≤ L τ + 1 D τ + r = τ + 1 ∑ T − 1 L r + 1 D r = r = τ ∑ T − 1 L r + 1 D r ,
which is (25) at time τ \tau τ and completes the induction. □
Corollary D.5 (Finite-agent consequence). Let s t = ( s t 1 , … , s t N ) ∈ S N s_t=(s_t^1,\ldots,s_t^N)\in S^N s t = ( s t 1 , … , s t N ) ∈ S N be the agent states at time t t t , with empirical distribution μ t N = 1 N ∑ i = 1 N δ s t i \mu_t^N=\frac{1}{N}\sum_{i=1}^N\delta_{s_t^i} μ t N = N 1 ∑ i = 1 N δ s t i , and let ν N \nu^N ν N be an N N N -atom target distribution as in Corollary 3.3. Under the assumptions of Theorem D.4, it holds that
0 ≤ K [ j t ] ( μ t N , ν N ) − J t N ( s t , ν N ) ≤ ∑ τ = t T − 1 L τ + 1 D τ . (27) 0\leq\mathcal{K}[j_t](\mu_t^N,\nu^N)-J_t^N(s_t,\nu^N)\leq\sum_{\tau=t}^{T-1}L_{\tau+1}D_\tau. \tag*{(27)} 0 ≤ K [ j t ] ( μ t N , ν N ) − J t N ( s t , ν N ) ≤ τ = t ∑ T − 1 L τ + 1 D τ . (27)
Proof. For a joint noise law and costs that are invariant under permutations of agent labels, the finite-agent value depends only on the empirical state. We may therefore choose an ε \varepsilon ε -optimal policy whose empirical state–action measure is a feedback function of that state. This measure is admissible for the population problem and has the same cost, so the population problem is a relaxation of the agent-based one. Together with Corollary 3.3, this gives
J t ( μ t N , ν N ) ≤ J t N ( s t , ν N ) ≤ K [ j t ] ( μ t N , ν N ) . J_t(\mu_t^N,\nu^N)\leq J_t^N(s_t,\nu^N)\leq\mathcal{K}[j_t](\mu_t^N,\nu^N). J t ( μ t N , ν N ) ≤ J t N ( s t , ν N ) ≤ K [ j t ] ( μ t N , ν N ) .
Subtracting the middle term from the transport cost and applying (24) at ( μ t N , ν N ) (\mu_t^N,\nu^N) ( μ t N , ν N ) yields (26).
Interpretation. The two sides of (20) differ because the population Bellman recursion may reassign targets after seeing the realized next population, whereas the transport bound commits to an assignment before the nominal noise is integrated. The bound in Theorem D.4 gives a clear interpretation of this loss. In particular, L τ + 1 L_{\tau+1} L τ + 1 measures how sensitive the remaining cost is to a displacement of an agent, while D τ D_\tau D τ measures how far the noise can push apart two agents that start from the same state-action pair. The gap therefore closes for deterministic dynamics, and more generally whenever the optimal assignment does not depend on the realized noise.
Bounding the dispersion term. The definition of D τ D_\tau D τ involves the distance between next states. This quantity can be bounded directly in terms of the noise when W W W carries a metric d W d_W d W and the dynamics are K τ K_\tau K τ -Lipschitz in their noise argument, uniformly in ( s , a ) (s,a) ( s , a ) :
d S ( f τ ( s , a , w ) , f τ ( s , a , w ~ ) ) ≤ K τ d W ( w , w ~ ) . d_S(f_\tau(s,a,w),f_\tau(s,a,\widetilde{w}))\leq K_\tau d_W(w,\widetilde{w}). d S ( f τ ( s , a , w ) , f τ ( s , a , w )) ≤ K τ d W ( w , w ) .
Indeed, applying this inequality to the two independent noise draws in the definition of D τ D_\tau D τ gives
D τ ≤ K τ sup ( s , a ) E ξ τ ( ⋅ ∣ s , a ) ⊗ 2 [ d W ( w τ , w ~ τ ) ] . D_\tau\leq K_\tau\sup_{(s,a)}\mathbb{E}_{\xi_\tau(\cdot\mid s,a)^{\otimes2}}\left[d_W(w_\tau,\widetilde{w}_\tau)\right]. D τ ≤ K τ ( s , a ) sup E ξ τ ( ⋅ ∣ s , a ) ⊗ 2 [ d W ( w τ , w τ ) ] .
If W W W is also a normed space and the conditional mean exists, we can apply the triangle inequality through that mean and, since the two noise draws have the same conditional distribution, obtain
D τ ≤ 2 K τ sup ( s , a ) E ξ τ ( ⋅ ∣ s , a ) [ ∥ w τ − E ξ τ ( ⋅ ∣ s , a ) [ w τ ] ∥ ] . D_\tau\leq2K_\tau\sup_{(s,a)}\mathbb{E}_{\xi_\tau(\cdot\mid s,a)}\left[\left\|w_\tau-\mathbb{E}_{\xi_\tau(\cdot\mid s,a)}[w_\tau]\right\|\right]. D τ ≤ 2 K τ ( s , a ) sup E ξ τ ( ⋅ ∣ s , a ) [ w τ − E ξ τ ( ⋅ ∣ s , a ) [ w τ ] ] .
Finally, for additive dynamics f τ ( s , a , w ) = f ~ τ ( s , a ) + w f_\tau(s,a,w)=\widetilde{f}_\tau(s,a)+w f τ ( s , a , w ) = f τ ( s , a ) + w , with S S S and W W W carrying the same norm metric, the distance between the two next states is exactly ∥ w − w ~ ∥ \|w-\widetilde{w}\| ∥ w − w ∥ , and the first estimate holds with equality and K τ = 1 K_\tau=1 K τ = 1 .
Tightness. We here show through an example that the linear dependence on D τ D_\tau D τ in Lemma D.3 and Theorem D.4 cannot be improved without further assumptions.
Example D.6 (The linear coefficient is tight). Let τ = T − 1 \tau=T-1 τ = T − 1 , A = { 0 } A=\{0\} A = { 0 } , and S = Z = { 0 , 1 } S=Z=\{0,1\} S = Z = { 0 , 1 } , equipped with the discrete metric. Set zero stage cost g t ( s , a , z ) = 0 g_t(s,a,z)=0 g t ( s , a , z ) = 0 , and the following terminal cost and dynamics:
g T ( s , z ) = ∣ s − z ∣ , f τ ( s , a , w ) = w , g_T(s,z)=|s-z|,\qquad f_\tau(s,a,w)=w, g T ( s , z ) = ∣ s − z ∣ , f τ ( s , a , w ) = w , where w ∼ Bernoulli ( 1 2 ) w\sim\operatorname{Bernoulli}\left(\frac{1}{2}\right) w ∼ Bernoulli ( 2 1 ) . Consider an infinite population and use the individual noise model in Example A.5. Suppose that initially the population is concentrated at s = 0 s=0 s = 0 , i.e. μ 0 = δ 0 \mu_0=\delta_0 μ 0 = δ 0 , and the target distribution is selected as ν = 1 2 δ 0 + 1 2 δ 1 \nu=\frac{1}{2}\delta_0+\frac{1}{2}\delta_1 ν = 2 1 δ 0 + 2 1 δ 1 . Before this is observed, the only population coupling assigns half the mass to each target. Moreover, since the next state equals w w w regardless of the current state, the single-agent Bellman recursion gives
j τ ( 0 , z ) = E w [ ∣ w − z ∣ ] = 1 2 ∣ 0 − z ∣ + 1 2 ∣ 1 − z ∣ = 1 2 , j_\tau(0,z)=\mathbb{E}_w[|w-z|]=\frac{1}{2}|0-z|+\frac{1}{2}|1-z|=\frac{1}{2}, j τ ( 0 , z ) = E w [ ∣ w − z ∣ ] = 2 1 ∣0 − z ∣ + 2 1 ∣1 − z ∣ = 2 1 , for every z ∈ Z z\in Z z ∈ Z . Hence j τ ( 0 , 0 ) = j τ ( 0 , 1 ) = 1 2 j_\tau(0,0)=j_\tau(0,1)=\frac{1}{2} j τ ( 0 , 0 ) = j τ ( 0 , 1 ) = 2 1 , and
K [ j τ ] ( μ , ν ) = 1 2 . \mathcal{K}[j_\tau](\mu,\nu)=\frac{1}{2}. K [ j τ ] ( μ , ν ) = 2 1 .
After the transition, on the other hand, the realized population is exactly ν \nu ν , and the population Bellman step matches equal states and targets at zero cost:
T τ [ K [ j T ] ] ( μ , ν ) = 0. \mathcal{T}_{\tau}[\mathcal{K}[j_T]](\mu,\nu)=0. T τ [ K [ j T ]] ( μ , ν ) = 0.
Since L T = 1 L_T=1 L T = 1 and two independent Bernoulli ( 1 2 ) (\frac{1}{2}) ( 2 1 ) draws differ with probability 1 / 2 1/2 1/2 , we have D τ = 1 / 2 D_\tau=1/2 D τ = 1/2 , so equality holds in (20) .
Robustness to weakly coupled dynamics
In this appendix, we study what happens when the separation-based policy is applied to a model in which the agents are weakly coupled, that is, a system whose dynamics and stage cost may depend on the current population distribution. Since the separation principle of § 3 requires decoupled agents, we construct the policy from a decoupled surrogate model, deploy it in the coupled system, and bound the resulting loss of optimality. Because the admissible controls remain M ( μ ) \mathcal{M}(\mu) M ( μ ) in both models, the following result covers soft population effects, such as congestion or queueing, but not hard capacity or collision constraints that change the set of feasible controls.
Models and assumptions. Throughout this appendix, we denote the decoupled surrogate with a superscript D D D , so its dynamics, stage cost, single-agent value, and population cost-to-go are written as
f τ D ( s , a , w ) , g τ D ( s , a , w ) , j τ D ( s , z ) , J τ D ( μ , ν ) . f_\tau^D(s,a,w),\qquad g_\tau^D(s,a,w),\qquad j_\tau^D(s,z),\qquad J_\tau^D(\mu,\nu). f τ D ( s , a , w ) , g τ D ( s , a , w ) , j τ D ( s , z ) , J τ D ( μ , ν ) .
The corresponding quantities for the coupled model are denoted by a superscript C C C :
f τ C ( s , a , μ , w ) , g τ C ( s , a , μ , w ) , J τ C ( μ , ν ) . f_\tau^C(s,a,\mu,w),\qquad g_\tau^C(s,a,\mu,w),\qquad J_\tau^C(\mu,\nu). f τ C ( s , a , μ , w ) , g τ C ( s , a , μ , w ) , J τ C ( μ , ν ) .
Here, the dynamics and stage costs also depend on the population, so there is no direct counterpart of j τ D j_\tau^D j τ D . Indeed, because f τ C f_\tau^C f τ C depends on μ \mu μ , the cost-to-go of a single agent cannot be defined without fixing the trajectory of the population. At the terminal time T T T , we denote by J T C ( μ , ν ) J_T^C(\mu,\nu) J T C ( μ , ν ) the terminal objective, evaluated at the terminal fleet state μ \mu μ and target distribution ν \nu ν . Apart from the population dependence displayed above, the two models share the same basic setting. The models have the same state, action, noise, and target spaces, the same nominal noise kernel, and the same admissible controls M ( μ ) \mathcal{M}(\mu) M ( μ ) . We also drive them with the same realized noise field, which satisfies Assumptions A.1 and A.2 , so their next populations are
μ τ + 1 C , λ : = f τ C ( ⋅ , ⋅ , μ , ⋅ ) # ρ τ λ , μ τ + 1 D , λ : = f τ D # ρ τ λ . (28) \mu_{\tau+1}^{C,\lambda}:=f_\tau^C(\cdot,\cdot,\mu,\cdot)_{\#}\rho_\tau^\lambda,\qquad
\mu_{\tau+1}^{D,\lambda}:=f_\tau^D{}_{\#}\rho_\tau^\lambda. \tag*{(28)} μ τ + 1 C , λ := f τ C ( ⋅ , ⋅ , μ , ⋅ ) # ρ τ λ , μ τ + 1 D , λ := f τ D # ρ τ λ . (28)
where ρ τ λ : = λ ⊗ ξ ^ τ s , a \rho_\tau^\lambda:=\lambda\otimes\hat{\xi}_\tau^{s,a} ρ τ λ := λ ⊗ ξ ^ τ s , a denotes the joint state-action-noise distribution for a fixed population μ \mu μ and a control λ ∈ M ( μ ) \lambda\in\mathcal{M}(\mu) λ ∈ M ( μ ) . Finally, the terminal objective of the coupled and decoupled models is common and denoted by
J T C ( μ , ν ) = J T D ( μ , ν ) = K [ g T ] ( μ , ν ) . (29) J_T^C(\mu,\nu)=J_T^D(\mu,\nu)=\mathcal{K}[g_T](\mu,\nu). \tag*{(29)} J T C ( μ , ν ) = J T D ( μ , ν ) = K [ g T ] ( μ , ν ) . (29)
We next specify how far the coupled model may be from the decoupled one at each time step.
Assumption E.1 (Bounded one-step model mismatch). Fix a reference point s 0 ∈ S s_0\in S s 0 ∈ S and assume that every population distribution μ ∈ P ( S ) \mu\in\mathcal{P}(S) μ ∈ P ( S ) at which the coupled and decoupled models are compared below has finite first moment,
∫ S d S ( s , s 0 ) d μ ( s ) < ∞ . \int_S d_S(s,s_0)\,d\mu(s)<\infty. ∫ S d S ( s , s 0 ) d μ ( s ) < ∞. For every τ < T \tau<T τ < T , let ε τ , δ τ ≥ 0 \varepsilon_\tau,\delta_\tau\ge0 ε τ , δ τ ≥ 0 be constants such that the following hold:
(i) Dynamics mismatch. When the coupled and decoupled dynamics are evaluated at the same state, action, and noise, their outputs satisfy
d S ( f τ C ( s , a , μ , w ) , f τ D ( s , a , w ) ) ≤ ε τ (30) d_S\left(f_\tau^C(s,a,\mu,w),f_\tau^D(s,a,w)\right)\le\varepsilon_\tau \tag*{(30)} d S ( f τ C ( s , a , μ , w ) , f τ D ( s , a , w ) ) ≤ ε τ (30) uniformly in μ ∈ P ( S ) \mu\in\mathcal{P}(S) μ ∈ P ( S ) and ( s , a , w ) ∈ S × A × W (s,a,w)\in S\times A\times W ( s , a , w ) ∈ S × A × W .
(ii) Stage-cost mismatch. The expected stage costs, evaluated at the same population and the same admissible state-action distribution, satisfy
∣ E η τ [ ∫ S × A × W ( g τ C ( s , a , μ , w ) − g τ D ( s , a , w ) ) d ρ τ λ ( s , a , w ) ] ∣ ≤ δ τ (31) \left|\mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}\left(g_\tau^C(s,a,\mu,w)-g_\tau^D(s,a,w)\right)\,d\rho_\tau^\lambda(s,a,w)\right]\right|\le\delta_\tau \tag*{(31)} E η τ [ ∫ S × A × W ( g τ C ( s , a , μ , w ) − g τ D ( s , a , w ) ) d ρ τ λ ( s , a , w ) ] ≤ δ τ (31) uniformly in μ ∈ P ( S ) \mu\in\mathcal{P}(S) μ ∈ P ( S ) and λ ∈ M ( μ ) \lambda\in\mathcal{M}(\mu) λ ∈ M ( μ ) .
These two constants control the two parts of one Bellman step. Indeed, δ τ \delta_\tau δ τ bounds the difference in the current stage cost, while ε τ \varepsilon_\tau ε τ bounds the difference between the next populations.
Note also that condition (30) is satisfied if there exists a pointwise bound ∣ g τ C − g τ D ∣ ≤ δ τ \lvert g_\tau^C-g_\tau^D\rvert\leq\delta_\tau ∣ g τ C − g τ D ∣ ≤ δ τ . Additionally, for a fixed initial population, then it is sufficient that (29) and (30) hold on the population states reachable under any admissible control.
These one-step bounds enter the comparison through the Bellman recursions. Recall the Bellman operator T τ \mathcal{T}_\tau T τ introduced in Appendix D. We now rewrite the two population recursions, for the coupled and decoupled models, in a common form. For m ∈ { C , D } m\in\{C,D\} m ∈ { C , D } , define
( T τ m F ) ( μ , ν ) : = inf λ ∈ M ( μ ) E η τ [ ∫ S × A × W g τ m ( s , a , μ , w ) d ρ λ μ ( s , a , w ) + F ( μ τ + 1 m , λ , ν ) ] . (\mathcal{T}_\tau^m F)(\mu,\nu):=\inf_{\lambda\in\mathcal{M}(\mu)}\mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}g_\tau^m(s,a,\mu,w)\,d\rho_\lambda^\mu(s,a,w)+F(\mu_{\tau+1}^{m,\lambda},\nu)\right]. ( T τ m F ) ( μ , ν ) := λ ∈ M ( μ ) inf E η τ [ ∫ S × A × W g τ m ( s , a , μ , w ) d ρ λ μ ( s , a , w ) + F ( μ τ + 1 m , λ , ν ) ] .
Here and below, we use the convention g τ D ( s , a , μ , w ) : = g τ D ( s , a , w ) g_\tau^D(s,a,\mu,w):=g_\tau^D(s,a,w) g τ D ( s , a , μ , w ) := g τ D ( s , a , w ) , thus neither f τ D f_\tau^D f τ D nor g τ D g_\tau^D g τ D depends on the population argument. With this notation, we have the Bellman recursions J τ m = T τ m J τ + 1 m J_\tau^m=\mathcal{T}_\tau^m J_{\tau+1}^m J τ m = T τ m J τ + 1 m . In particular, T τ D = T τ \mathcal{T}_\tau^D=\mathcal{T}_\tau T τ D = T τ , since the decoupled stage cost and dynamics are exactly the data g τ g_\tau g τ and f τ f_\tau f τ used to define T τ \mathcal{T}_\tau T τ in Appendix D, so every result proved there for T τ \mathcal{T}_\tau T τ applies verbatim to T τ D \mathcal{T}_\tau^D T τ D . Additionally, if π \pi π is a feedback policy with π τ ( μ ) ∈ M ( μ ) \pi_\tau(\mu)\in\mathcal{M}(\mu) π τ ( μ ) ∈ M ( μ ) , we drop the infimum and denote the same expression evaluated at λ = π τ ( μ ) \lambda=\pi_\tau(\mu) λ = π τ ( μ ) as
( T τ m , π F ) ( μ , ν ) : = E η τ [ ∫ S × A × W g τ m ( s , a , μ , w ) d ρ π τ ( μ ) π ( s , a , w ) + F ( μ τ + 1 m , π τ ( μ ) , ν ) ] . (\mathcal{T}_\tau^{m,\pi}F)(\mu,\nu):=\mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}g_\tau^m(s,a,\mu,w)\,d\rho_{\pi_\tau(\mu)}^\pi(s,a,w)+F(\mu_{\tau+1}^{m,\pi_\tau(\mu)},\nu)\right]. ( T τ m , π F ) ( μ , ν ) := E η τ [ ∫ S × A × W g τ m ( s , a , μ , w ) d ρ π τ ( μ ) π ( s , a , w ) + F ( μ τ + 1 m , π τ ( μ ) , ν ) ] .
The cost of this policy satisfies
J T m , π = K [ g T ] , J τ m , π = T τ m , π J τ + 1 m , π . J_T^{m,\pi}=\mathcal{K}[g_T],\qquad J_\tau^{m,\pi}=\mathcal{T}_\tau^{m,\pi}J_{\tau+1}^{m,\pi}. J T m , π = K [ g T ] , J τ m , π = T τ m , π J τ + 1 m , π .
The next-stage terms in these recursions may be evaluated at different population distributions, so to compare those distributions, we use the 1-Wasserstein distance induced by d S d_S d S . Explicitly, for any μ , ν ∈ P ( S ) \mu,\nu\in\mathcal{P}(S) μ , ν ∈ P ( S ) with finite first moments, the 1-Wasserstein distance induced by d S d_S d S is denoted by W 1 ( μ , ν ) W_1(\mu,\nu) W 1 ( μ , ν ) and defined as
W 1 ( μ , ν ) : = inf q ∈ Γ ( μ , ν ) ∫ S × S d S ( s , s ~ ) d q ( s , s ~ ) . W_1(\mu,\nu):=\inf_{q\in\Gamma(\mu,\nu)}\int_{S\times S}d_S(s,\widetilde{s})\,dq(s,\widetilde{s}). W 1 ( μ , ν ) := q ∈ Γ ( μ , ν ) inf ∫ S × S d S ( s , s ) d q ( s , s ) .
To compare the next populations in (27), we need an estimate for two maps applied to the same measure. We formalize this in the following lemma. Note that this differs from [48 Proposition 3] , which considers one map applied to two measures.
Lemma E.2 (Wasserstein bound for pushforwards of a common measure). Let U U U be a measurable space, let μ ∈ P ( U ) \mu\in\mathcal{P}(U) μ ∈ P ( U ) , and let F , G : U → S F,G:U\to S F , G : U → S be measurable. Then
W 1 ( F # μ , G # μ ) ≤ ∫ U d S ( F ( u ) , G ( u ) ) d μ ( u ) . (32) W_1(F_{\#}\mu,G_{\#}\mu)\leq\int_U d_S(F(u),G(u))\,d\mu(u). \tag*{(32)} W 1 ( F # μ , G # μ ) ≤ ∫ U d S ( F ( u ) , G ( u )) d μ ( u ) . (32)
Proof. Consider the coupling q F , G : = ( F , G ) # μ q_{F,G}:=(F,G)_{\#}\mu q F , G := ( F , G ) # μ , where ( F , G ) (F,G) ( F , G ) maps u u u to ( F ( u ) , G ( u ) ) (F(u),G(u)) ( F ( u ) , G ( u )) . Its first marginal is F # μ F_{\#}\mu F # μ , since for every Borel set A ⊆ S A\subseteq S A ⊆ S ,
q F , G ( A × S ) = μ ( F − 1 ( A ) ) = ( F # μ ) ( A ) . q_{F,G}(A\times S)=\mu(F^{-1}(A))=(F_{\#}\mu)(A). q F , G ( A × S ) = μ ( F − 1 ( A )) = ( F # μ ) ( A ) .
The same argument shows that its second marginal is G # μ G_{\#}\mu G # μ , so q F , G ∈ Γ ( F # μ , G # μ ) q_{F,G}\in\Gamma(F_{\#}\mu,G_{\#}\mu) q F , G ∈ Γ ( F # μ , G # μ ) . Evaluating the definition of W 1 W_1 W 1 at this plan and applying the change-of-variables formula for the pushforward q F , G = ( F , G ) # μ q_{F,G}=(F,G)_{\#}\mu q F , G = ( F , G ) # μ gives
W 1 ( F # μ , G # μ ) ≤ ∫ S × S d S ( s , s ~ ) d q F , G ( s , s ~ ) = ∫ U d S ( F ( u ) , G ( u ) ) d μ ( u ) . W_1(F_{\#}\mu,G_{\#}\mu)\leq\int_{S\times S}d_S(s,\widetilde{s})\,dq_{F,G}(s,\widetilde{s})
=\int_U d_S(F(u),G(u))\,d\mu(u). W 1 ( F # μ , G # μ ) ≤ ∫ S × S d S ( s , s ) d q F , G ( s , s ) = ∫ U d S ( F ( u ) , G ( u )) d μ ( u ) .
This concludes the proof.
In other words, when two distributions are obtained from the same measure through two different maps, their Wasserstein distance can be bounded by comparing the outputs of the maps pointwise.
It remains to relate this distance between populations to the next-stage cost. To do so, we assume that the decoupled single-agent cost-to-go j τ D j_{\tau}^{D} j τ D satisfies Assumption D.1, with constants L τ L_{\tau} L τ , and use D τ D_{\tau} D τ (19) for the dispersion of the decoupled dynamics f τ D f_{\tau}^{D} f τ D . As in Appendix § 12 , we assume that all transport costs appearing below are finite.
We can now proceed with the study of the robustness of the separation-based policy. The first step is to show that the transport cost is Lipschitz in the population distribution. This will allow us, in Lemma E.4 below, to turn the state mismatch in (30) into a bound on the next-stage cost.
Lemma E.3 (Population sensitivity of the transport cost). Suppose Assumption D.1 holds for j τ D j_{\tau}^{D} j τ D with constant L τ L_{\tau} L τ . Then, for every τ ∈ { 1 , … , T } \tau\in\{1,\ldots,T\} τ ∈ { 1 , … , T } , every μ , μ ~ ∈ P ( S ) \mu,\widetilde{\mu} \in\mathcal{P}(S) μ , μ ∈ P ( S ) with finite first moments, and every ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) ,
∣ K [ j τ D ] ( μ , ν ) − K [ j τ D ] ( μ ~ , ν ) ∣ ≤ L τ W 1 ( μ , μ ~ ) . (33) \left|\mathcal{K}[j_{\tau}^{D}](\mu,\nu)-\mathcal{K}[j_{\tau}^{D}](\widetilde{\mu},\nu)\right| \le L_{\tau}W_{1}(\mu,\widetilde{\mu}). \tag*{(33)} K [ j τ D ] ( μ , ν ) − K [ j τ D ] ( μ , ν ) ≤ L τ W 1 ( μ , μ ) . (33)
Proof. The proof consists of transferring a nearly optimal target assignment for μ \mu μ to μ ~ \widetilde{\mu} μ . Fix ε > 0 \varepsilon>0 ε > 0 . By the definition of the transport cost and the Wasserstein distance, we can choose γ ε ∈ Γ ( μ , ν ) \gamma_{\varepsilon} \in\Gamma(\mu,\nu) γ ε ∈ Γ ( μ , ν ) and q ε ∈ Γ ( μ , μ ~ ) q_{\varepsilon} \in\Gamma(\mu,\widetilde{\mu}) q ε ∈ Γ ( μ , μ ) such that
∫ S × Z j τ D ( s , z ) d γ ε ( s , z ) ≤ K [ j τ D ] ( μ , ν ) + ε , \int_{S \times Z} j_{\tau}^{D}(s,z)\,\mathrm{d}\gamma_{\varepsilon}(s,z) \le\mathcal{K}[j_{\tau}^{D}](\mu,\nu)+\varepsilon, ∫ S × Z j τ D ( s , z ) d γ ε ( s , z ) ≤ K [ j τ D ] ( μ , ν ) + ε ,
∫ S × S d S ( s , s ~ ) d q ε ( s , s ~ ) ≤ W 1 ( μ , μ ~ ) + ε . \int_{S \times S} d_{S}(s,\widetilde{s})\,\mathrm{d}q_{\varepsilon}(s,\widetilde{s}) \le W_{1}(\mu,\widetilde{\mu})+\varepsilon. ∫ S × S d S ( s , s ) d q ε ( s , s ) ≤ W 1 ( μ , μ ) + ε .
Since the two plans have the same s s s -marginal μ \mu μ , the gluing lemma gives a measure r ε r_{\varepsilon} r ε on S × S × Z S \times S \times Z S × S × Z with ( s , z ) (s,z) ( s , z ) -marginal γ ε \gamma_{\varepsilon} γ ε and ( s , s ~ ) (s,\widetilde{s}) ( s , s ) -marginal q ε q_{\varepsilon} q ε . Note, in particular, that its ( s ~ , z ) (\widetilde{s},z) ( s , z ) -marginal couples μ ~ \widetilde{\mu} μ with ν \nu ν , so we can use it in the transport problem for K [ j τ D ] ( μ ~ , ν ) \mathcal{K}[j_{\tau}^{D}](\widetilde{\mu},\nu) K [ j τ D ] ( μ , ν ) . The first inequality below evaluates the transport cost at the ( s ~ , z ) (\widetilde{s},z) ( s , z ) -marginal of r ε r_{\varepsilon} r ε , the second applies (18) pointwise, and then we use the ( s , z ) (s,z) ( s , z ) and ( s , s ~ ) (s,\widetilde{s}) ( s , s ) -marginals of r ε r_{\varepsilon} r ε to separate the two integrals:
K [ j τ D ] ( μ ~ , ν ) ≤ ∫ S × S × Z j τ D ( s ~ , z ) d r ε ( s , s ~ , z ) ≤ ∫ S × S × Z [ j τ D ( s , z ) + L τ d S ( s , s ~ ) ] d r ε ( s , s ~ , z ) = ∫ S × Z j τ D ( s , z ) d γ ε ( s , z ) + L τ ∫ S × S d S ( s , s ~ ) d q ε ( s , s ~ ) ≤ K [ j τ D ] ( μ , ν ) + ε + L τ ( W 1 ( μ , μ ~ ) + ε ) = K [ j τ D ] ( μ , ν ) + L τ W 1 ( μ , μ ~ ) + ( 1 + L τ ) ε . (34) \begin{aligned}
\mathcal{K}[j_{\tau}^{D}](\widetilde{\mu},\nu)
&\le\int_{S \times S \times Z} j_{\tau}^{D}(\widetilde{s},z)\,\mathrm{d}r_{\varepsilon}(s,\widetilde{s},z)\\
&\le\int_{S \times S \times Z}\left[j_{\tau}^{D}(s,z)+L_{\tau}d_{S}(s,\widetilde{s})\right]\,\mathrm{d}r_{\varepsilon}(s,\widetilde{s},z)\\
&= \int_{S \times Z}j_{\tau}^{D}(s,z)\,\mathrm{d}\gamma_{\varepsilon}(s,z)+L_{\tau}\int_{S \times S}d_{S}(s,\widetilde{s})\,\mathrm{d}q_{\varepsilon}(s,\widetilde{s})\\
&\le\mathcal{K}[j_{\tau}^{D}](\mu,\nu)+\varepsilon+L_{\tau}\left(W_{1}(\mu,\widetilde{\mu})+\varepsilon\right)\\
&= \mathcal{K}[j_{\tau}^{D}](\mu,\nu)+L_{\tau}W_{1}(\mu,\widetilde{\mu})+(1+L_{\tau})\varepsilon. \tag*{(34)}
\end{aligned} K [ j τ D ] ( μ , ν ) ≤ ∫ S × S × Z j τ D ( s , z ) d r ε ( s , s , z ) ≤ ∫ S × S × Z [ j τ D ( s , z ) + L τ d S ( s , s ) ] d r ε ( s , s , z ) = ∫ S × Z j τ D ( s , z ) d γ ε ( s , z ) + L τ ∫ S × S d S ( s , s ) d q ε ( s , s ) ≤ K [ j τ D ] ( μ , ν ) + ε + L τ ( W 1 ( μ , μ ) + ε ) = K [ j τ D ] ( μ , ν ) + L τ W 1 ( μ , μ ) + ( 1 + L τ ) ε . (34)
where the last inequality uses the two ε \varepsilon ε -optimality bounds in the choice of γ ε \gamma_{\varepsilon} γ ε and q ε q_{\varepsilon} q ε . Since the construction holds for all ε > 0 \varepsilon>0 ε > 0 , letting ε ↓ 0 \varepsilon\downarrow0 ε ↓ 0 gives
K [ j τ D ] ( μ ~ , ν ) − K [ j τ D ] ( μ , ν ) ≤ L τ W 1 ( μ , μ ~ ) . \mathcal{K}[j_{\tau}^{D}](\widetilde{\mu},\nu)-\mathcal{K}[j_{\tau}^{D}](\mu,\nu)\le L_{\tau}W_{1}(\mu,\widetilde{\mu}). K [ j τ D ] ( μ , ν ) − K [ j τ D ] ( μ , ν ) ≤ L τ W 1 ( μ , μ ) .
Exchanging the roles of μ \mu μ and μ ~ \widetilde{\mu} μ throughout the construction above, that is, choosing γ ε ∈ Γ ( μ ~ , ν ) \gamma_{\varepsilon} \in\Gamma(\widetilde{\mu},\nu) γ ε ∈ Γ ( μ , ν ) that is ε \varepsilon ε -optimal for K [ j τ D ] ( μ ~ , ν ) \mathcal{K}[j_{\tau}^{D}](\widetilde{\mu},\nu) K [ j τ D ] ( μ , ν ) and q ε ∈ Γ ( μ ~ , μ ) q_{\varepsilon} \in\Gamma(\widetilde{\mu},\mu) q ε ∈ Γ ( μ , μ ) that is ε \varepsilon ε -optimal for W 1 ( μ ~ , μ ) W_{1}(\widetilde{\mu},\mu) W 1 ( μ , μ ) , and then repeating the gluing step and (34) with s s s and s ~ \widetilde{s} s swapped, gives
K [ j τ D ] ( μ , ν ) − K [ j τ D ] ( μ ~ , ν ) ≤ L τ W 1 ( μ ~ , μ ) . \mathcal{K}[j_{\tau}^{D}](\mu,\nu)-\mathcal{K}[j_{\tau}^{D}](\widetilde{\mu},\nu)\le L_{\tau}W_{1}(\widetilde{\mu},\mu). K [ j τ D ] ( μ , ν ) − K [ j τ D ] ( μ , ν ) ≤ L τ W 1 ( μ , μ ) .
Since W 1 ( μ ~ , μ ) = W 1 ( μ , μ ~ ) W_{1}(\widetilde{\mu},\mu)=W_{1}(\mu,\widetilde{\mu}) W 1 ( μ , μ ) = W 1 ( μ , μ ) , the two bounds give (33) . □ \square □
Lemma E.3 allows us to compare one Bellman step in the two models. To do so, we first keep the control λ \lambda λ fixed, so that the only differences come from the stage cost and the next population, and then minimize over λ \lambda λ .
Lemma E.4 (One-step model perturbation). Under Assumptions A.1, A.2, D.1, and E.1, for every τ < T \tau<T τ < T , every μ ∈ P ( S ) \mu\in\mathcal{P}(S) μ ∈ P ( S ) with finite first moment, every ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) , and every admissible feedback policy π \pi π ,
∣ T τ C [ K [ j τ + 1 D ] ] ( μ , ν ) − T τ D [ K [ j τ + 1 D ] ] ( μ , ν ) ∣ ≤ δ τ + L τ + 1 ε τ f , (35) \left|\mathcal{T}_{\tau}^{C}\left[\mathcal{K}[j_{\tau+1}^{D}]\right](\mu,\nu)-\mathcal{T}_{\tau}^{D}\left[\mathcal{K}[j_{\tau+1}^{D}]\right](\mu,\nu)\right| \le\delta_{\tau}+L_{\tau+1}\varepsilon_{\tau}^{f}, \tag*{(35)} T τ C [ K [ j τ + 1 D ] ] ( μ , ν ) − T τ D [ K [ j τ + 1 D ] ] ( μ , ν ) ≤ δ τ + L τ + 1 ε τ f , (35) ∣ T τ C , π [ K [ j τ + 1 D ] ] ( μ , ν ) − T τ D , π [ K [ j τ + 1 D ] ] ( μ , ν ) ∣ ≤ δ τ + L τ + 1 ε τ f . (36) \left|\mathcal{T}_{\tau}^{C,\pi}\left[\mathcal{K}[j_{\tau+1}^{D}]\right](\mu,\nu)-\mathcal{T}_{\tau}^{D,\pi}\left[\mathcal{K}[j_{\tau+1}^{D}]\right](\mu,\nu)\right| \le\delta_{\tau}+L_{\tau+1}\varepsilon_{\tau}^{f}. \tag*{(36)} T τ C , π [ K [ j τ + 1 D ] ] ( μ , ν ) − T τ D , π [ K [ j τ + 1 D ] ] ( μ , ν ) ≤ δ τ + L τ + 1 ε τ f . (36)
Proof. Step 1: Compare the next populations for a fixed control. Fix λ ∈ M ( μ ) \lambda\in M(\mu) λ ∈ M ( μ ) and a realization of the noise field. By (28) , we can obtain the two next populations by pushing forward the same state-action-noise distribution ρ τ λ \rho^\lambda_\tau ρ τ λ , once through f τ C f^C_\tau f τ C and once through f τ D f^D_\tau f τ D . We can then apply (32) with μ = ρ τ λ \mu=\rho^\lambda_\tau μ = ρ τ λ , F = f τ C ( ⋅ , ⋅ , μ , ⋅ ) F=f^C_\tau(\cdot,\cdot,\mu,\cdot) F = f τ C ( ⋅ , ⋅ , μ , ⋅ ) , and G = f τ D G=f^D_\tau G = f τ D and obtain
W 1 ( μ τ + 1 C , λ , μ τ + 1 D , λ ) ≤ ∫ S × A × W d S ( f τ C ( s , a , μ , w ) , f τ D ( s , a , w ) ) d ρ τ λ ( s , a , w ) ≤ ε τ f . (37) W_1\left(\mu^{C,\lambda}_{\tau+1},\mu^{D,\lambda}_{\tau+1}\right) \le\int_{S\times A\times W} d_S\left(f^C_\tau(s,a,\mu,w),f^D_\tau(s,a,w)\right)\,d\rho^\lambda_\tau(s,a,w) \le\varepsilon^f_\tau. \tag*{(37)} W 1 ( μ τ + 1 C , λ , μ τ + 1 D , λ ) ≤ ∫ S × A × W d S ( f τ C ( s , a , μ , w ) , f τ D ( s , a , w ) ) d ρ τ λ ( s , a , w ) ≤ ε τ f . (37)
The last inequality follows directly from the uniform dynamics-mismatch bound (30) in Assumption E.1 .
Step 2: Compare the next-stage transport costs. For the fixed realization of the noise field selected in the previous step, Lemma E.3 applied at time τ + 1 \tau+1 τ + 1 , together with (37) , gives
∣ K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) − K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ∣ ≤ L τ + 1 W 1 ( μ τ + 1 C , λ , μ τ + 1 D , λ ) ≤ L τ + 1 ε τ f . \left|\mathcal{K}[j^D_{\tau+1}](\mu^{C,\lambda}_{\tau+1},\nu)-\mathcal{K}[j^D_{\tau+1}](\mu^{D,\lambda}_{\tau+1},\nu)\right| \le L_{\tau+1}W_1\left(\mu^{C,\lambda}_{\tau+1},\mu^{D,\lambda}_{\tau+1}\right) \le L_{\tau+1}\varepsilon^f_\tau. K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) − K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ≤ L τ + 1 W 1 ( μ τ + 1 C , λ , μ τ + 1 D , λ ) ≤ L τ + 1 ε τ f .
Since this bound holds for every realization of the field, we can average it out and obtain
E η τ [ ∣ K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) − K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ∣ ] ≤ L τ + 1 ε τ f . \mathbb{E}_{\eta_\tau}\left[\left|\mathcal{K}[j^D_{\tau+1}](\mu^{C,\lambda}_{\tau+1},\nu)-\mathcal{K}[j^D_{\tau+1}](\mu^{D,\lambda}_{\tau+1},\nu)\right|\right] \le L_{\tau+1}\varepsilon^f_\tau. E η τ [ K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) − K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ] ≤ L τ + 1 ε τ f .
Step 3: Add the stage-cost difference. By (31) in Assumption E.1 , the absolute difference between the two expected stage costs is at most δ τ \delta_\tau δ τ . We now add this difference to the difference between the next-stage transport costs. The triangle inequality and the estimate from Step 2 give
∣ E η τ [ ∫ S × A × W ( g τ C ( s , a , μ , w ) − g τ D ( s , a , w ) ) d ρ τ λ ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) − K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ] ∣ ≤ ∣ E η τ [ ∫ S × A × W ( g τ C ( s , a , μ , w ) − g τ D ( s , a , w ) ) d ρ τ λ ( s , a , w ) ] ∣ + E η τ [ ∣ K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) − K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ∣ ] ≤ δ τ + L τ + 1 ε τ f . (38) \begin{aligned}
&\left|\mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}\left(g^C_\tau(s,a,\mu,w)-g^D_\tau(s,a,w)\right)d\rho^\lambda_\tau(s,a,w)\right.\right.\\
&\qquad\left.\left.{}+\mathcal{K}[j^D_{\tau+1}](\mu^{C,\lambda}_{\tau+1},\nu)-\mathcal{K}[j^D_{\tau+1}](\mu^{D,\lambda}_{\tau+1},\nu)\right]\right|\\
&\le\left|\mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}\left(g^C_\tau(s,a,\mu,w)-g^D_\tau(s,a,w)\right)d\rho^\lambda_\tau(s,a,w)\right]\right|\\
&\qquad+\mathbb{E}_{\eta_\tau}\left[\left|\mathcal{K}[j^D_{\tau+1}](\mu^{C,\lambda}_{\tau+1},\nu)-\mathcal{K}[j^D_{\tau+1}](\mu^{D,\lambda}_{\tau+1},\nu)\right|\right]\\
&\le\delta_\tau+L_{\tau+1}\varepsilon^f_\tau. \tag*{(38)}
\end{aligned} E η τ [ ∫ S × A × W ( g τ C ( s , a , μ , w ) − g τ D ( s , a , w ) ) d ρ τ λ ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) − K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ] ≤ E η τ [ ∫ S × A × W ( g τ C ( s , a , μ , w ) − g τ D ( s , a , w ) ) d ρ τ λ ( s , a , w ) ] + E η τ [ K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) − K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ] ≤ δ τ + L τ + 1 ε τ f . (38)
Step 4: Pass to the Bellman operators. By linearity of expectation, one of the two one-sided inequalities implied by (38) is
E η τ [ ∫ S × A × W g τ C ( s , a , μ , w ) d ρ τ λ ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) ] ≤ E η τ [ ∫ S × A × W g τ D ( s , a , w ) d ρ τ λ ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ] + δ τ + L τ + 1 ε τ f . (39) \mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}g^C_\tau(s,a,\mu,w)d\rho^\lambda_\tau(s,a,w)+\mathcal{K}[j^D_{\tau+1}](\mu^{C,\lambda}_{\tau+1},\nu)\right]
\le\mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}g^D_\tau(s,a,w)d\rho^\lambda_\tau(s,a,w)+\mathcal{K}[j^D_{\tau+1}](\mu^{D,\lambda}_{\tau+1},\nu)\right]+\delta_\tau+L_{\tau+1}\varepsilon^f_\tau. \tag*{(39)} E η τ [ ∫ S × A × W g τ C ( s , a , μ , w ) d ρ τ λ ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) ] ≤ E η τ [ ∫ S × A × W g τ D ( s , a , w ) d ρ τ λ ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ] + δ τ + L τ + 1 ε τ f . (39)
The left- and right-hand sides are precisely the fixed-control objectives minimized in the definitions of T τ C T^C_\tau T τ C and T τ D T^D_\tau T τ D . Since (39) holds for every admissible λ \lambda λ , we can take the infimum over the controls in M ( μ ) M(\mu) M ( μ ) on both sides, and obtain
T τ C [ K [ j τ + 1 D ] ] ( μ , ν ) = inf λ ∈ M ( μ ) E η τ [ ∫ S × A × W g τ C ( s , a , μ , w ) d ρ τ λ ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) ] ≤ inf λ ∈ M ( μ ) E η τ [ ∫ S × A × W g τ D ( s , a , w ) d ρ τ λ ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ] + δ τ + L τ + 1 ε τ f = T τ D [ K [ j τ + 1 D ] ] ( μ , ν ) + δ τ + L τ + 1 ε τ f . \begin{aligned}
T^C_\tau\left[\mathcal{K}[j^D_{\tau+1}]\right](\mu,\nu)
&=\inf_{\lambda\in M(\mu)}\mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}g^C_\tau(s,a,\mu,w)d\rho^\lambda_\tau(s,a,w)+\mathcal{K}[j^D_{\tau+1}](\mu^{C,\lambda}_{\tau+1},\nu)\right]\\
&\le\inf_{\lambda\in M(\mu)}\mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W}g^D_\tau(s,a,w)d\rho^\lambda_\tau(s,a,w)+\mathcal{K}[j^D_{\tau+1}](\mu^{D,\lambda}_{\tau+1},\nu)\right]+\delta_\tau+L_{\tau+1}\varepsilon^f_\tau\\
&=T^D_\tau\left[\mathcal{K}[j^D_{\tau+1}]\right](\mu,\nu)+\delta_\tau+L_{\tau+1}\varepsilon^f_\tau.
\end{aligned} T τ C [ K [ j τ + 1 D ] ] ( μ , ν ) = λ ∈ M ( μ ) inf E η τ [ ∫ S × A × W g τ C ( s , a , μ , w ) d ρ τ λ ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 C , λ , ν ) ] ≤ λ ∈ M ( μ ) inf E η τ [ ∫ S × A × W g τ D ( s , a , w ) d ρ τ λ ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 D , λ , ν ) ] + δ τ + L τ + 1 ε τ f = T τ D [ K [ j τ + 1 D ] ] ( μ , ν ) + δ τ + L τ + 1 ε τ f .
Similarly, taking the infimum over M ( μ ) M(\mu) M ( μ ) in the reverse fixed-control inequality of (37) gives the other one-sided bound
T τ D [ K τ + 1 D ] ( μ , ν ) ≤ T τ C [ K τ + 1 D ] ( μ , ν ) + δ τ + L τ + 1 ε τ f . T_{\tau}^{D}\left[\mathcal{K}_{\tau+1}^{D}\right](\mu,\nu) \leq T_{\tau}^{C}\left[\mathcal{K}_{\tau+1}^{D}\right](\mu,\nu)+\delta_{\tau}+L_{\tau+1}\varepsilon_{\tau}^{f}. T τ D [ K τ + 1 D ] ( μ , ν ) ≤ T τ C [ K τ + 1 D ] ( μ , ν ) + δ τ + L τ + 1 ε τ f .
Combining these two inequalities proves (34). Finally, for the fixed-policy operators, it is enough to evaluate the same objectives at the same control λ = π τ ( μ ) \lambda=\pi_{\tau}(\mu) λ = π τ ( μ ) . Indeed, substituting this value into (37) gives directly (35).
The separation-based policy. We now construct the separation-based policy from the decoupled model, following the same steps as in Section 4 and Theorem 3.1, Part II. We will then evaluate this policy in the coupled model, so to bound the performance loss due to the model mismatch. Briefly, the policy construction proceeds as follows: at an observed population μ \mu μ , the policy first chooses a θ τ / 2 \theta_{\tau}/2 θ τ /2 -optimal target assignment γ τ θ ∈ Γ ( μ , ν ) \gamma_{\tau}^{\theta}\in\Gamma(\mu,\nu) γ τ θ ∈ Γ ( μ , ν ) for the decoupled transport problem, so that
∫ S × Z j τ D ( s , z ) d γ τ θ ( s , z ) ≤ K [ j τ D ] ( μ , ν ) + θ τ 2 . (40) \int_{S\times Z}j_{\tau}^{D}(s,z)\,d\gamma_{\tau}^{\theta}(s,z)\leq\mathcal{K}\left[j_{\tau}^{D}\right](\mu,\nu)+\frac{\theta_{\tau}}{2}. \tag*{(40)} ∫ S × Z j τ D ( s , z ) d γ τ θ ( s , z ) ≤ K [ j τ D ] ( μ , ν ) + 2 θ τ . (40)
It then chooses a measurable action a τ θ ( s , z ) a_{\tau}^{\theta}(s,z) a τ θ ( s , z ) whose single-agent cost is within θ τ / 2 \theta_{\tau}/2 θ τ /2 of the decoupled optimum. Writing a = a τ θ ( s , z ) a=a_{\tau}^{\theta}(s,z) a = a τ θ ( s , z ) , this means that
∫ W [ g τ D ( s , a , w ) + j τ + 1 D ( f D ( s , a , w ) , z ) ] d ξ τ ( w ∣ s , a ) ≤ j τ D ( s , z ) + θ τ 2 . (41) \int_{W}\left[g_{\tau}^{D}(s,a,w)+j_{\tau+1}^{D}\left(f^{D}(s,a,w),z\right)\right]d\xi_{\tau}(w\mid s,a)\leq j_{\tau}^{D}(s,z)+\frac{\theta_{\tau}}{2}. \tag*{(41)} ∫ W [ g τ D ( s , a , w ) + j τ + 1 D ( f D ( s , a , w ) , z ) ] d ξ τ ( w ∣ s , a ) ≤ j τ D ( s , z ) + 2 θ τ . (41)
Since this inequality holds for every ( s , z ) ∈ S × Z (s,z)\in S\times Z ( s , z ) ∈ S × Z , the target plan and the selected actions together induce the following population control
π s e p , τ θ ( μ ) : = ( proj S × Z S , a τ θ ) # γ τ θ ∈ M ( μ ) . (42) \pi_{\mathrm{sep},\tau}^{\theta}(\mu):=\left(\operatorname{proj}_{S\times Z}^{S},a_{\tau}^{\theta}\right)_{\#}\gamma_{\tau}^{\theta}\in M(\mu). \tag*{(42)} π sep , τ θ ( μ ) := ( proj S × Z S , a τ θ ) # γ τ θ ∈ M ( μ ) . (42)
Note that the control π s e p , τ θ ( μ ) \pi_{\mathrm{sep},\tau}^{\theta}(\mu) π sep , τ θ ( μ ) is a measurable map from the population space to the set of admissible controls, as required by the definition of a feedback policy. The construction ensures that the policy is well-defined and can be implemented in practice, as it relies on measurable selections of near-optimal target assignments and actions. The two errors of size θ τ / 2 \theta_{\tau}/2 θ τ /2 add up to the per-stage tolerance θ τ \theta_{\tau} θ τ . If, additionally, the target plan and the actions can both be chosen as measurable exact minimizers, then the construction holds when we take θ τ = 0 \theta_{\tau}=0 θ τ = 0 .
Finally, recall that ρ τ λ \rho_{\tau}^{\lambda} ρ τ λ was defined immediately after (27), for every λ ∈ M ( μ ) \lambda\in M(\mu) λ ∈ M ( μ ) , as the corresponding state-action-noise distribution. Since π s e p , τ θ ( μ ) ∈ M ( μ ) \pi_{\mathrm{sep},\tau}^{\theta}(\mu)\in M(\mu) π sep , τ θ ( μ ) ∈ M ( μ ) , we can apply that definition with λ = π s e p , τ θ ( μ ) \lambda=\pi_{\mathrm{sep},\tau}^{\theta}(\mu) λ = π sep , τ θ ( μ ) , obtaining ρ τ π s e p , τ θ ( μ ) \rho_{\tau}^{\pi_{\mathrm{sep},\tau}^{\theta}(\mu)} ρ τ π sep , τ θ ( μ ) .
Before evaluating this separation-based policy in the coupled model, we show that when working in the decoupled model, one Bellman step under this policy costs at most the decoupled transport surrogate plus θ τ \theta_{\tau} θ τ . This is the estimate that will enter the backward argument of the main theorem.
Lemma E.5 (One-step bound for the separation-based policy). Under Assumptions A.1 and A.2, the separation-based policy defined by (39)–(41) satisfies, for every τ < T \tau<T τ < T , every μ ∈ P ( S ) \mu\in\mathcal{P}(S) μ ∈ P ( S ) , and every ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) ,
T τ D , π s e p , τ θ [ K τ + 1 D ] ( μ , ν ) ≤ K [ j τ D ] ( μ , ν ) + θ τ . (43) T_{\tau}^{D,\pi_{\mathrm{sep},\tau}^{\theta}}\left[\mathcal{K}_{\tau+1}^{D}\right](\mu,\nu)\leq\mathcal{K}\left[j_{\tau}^{D}\right](\mu,\nu)+\theta_{\tau}. \tag*{(43)} T τ D , π sep , τ θ [ K τ + 1 D ] ( μ , ν ) ≤ K [ j τ D ] ( μ , ν ) + θ τ . (43)
Proof. Step 1: Lift the policy to a measure on ( s , z , w ) (s,z,w) ( s , z , w ) . We start by fixing μ \mu μ and a realization of the noise field and, to obtain an admissible target assignment at the next time, we keep the target attached to each state while the state moves according to its selected action and noise. More precisely, consider the measure
d γ τ θ ( s , z ) d ξ ^ τ s , a τ θ ( s , z ) ( w ) d\gamma_{\tau}^{\theta}(s,z)d\widehat{\xi}_{\tau}^{s,a_{\tau}^{\theta}(s,z)}(w) d γ τ θ ( s , z ) d ξ τ s , a τ θ ( s , z ) ( w )
on ( s , z , w ) (s,z,w) ( s , z , w ) . It generates the same state, action, and noise distribution ρ τ π s e p , τ θ ( μ ) \rho_{\tau}^{\pi_{\mathrm{sep},\tau}^{\theta}(\mu)} ρ τ π sep , τ θ ( μ ) as the separation-based policy. More precisely, for every nonnegative measurable function h h h on S × A × W S\times A\times W S × A × W ,
∫ S × A × W h ( s , a , w ) d ρ τ π s e p , τ θ ( μ ) ( s , a , w ) = ∫ S × Z ∫ W h ( s , a τ θ ( s , z ) , w ) d ξ ^ τ s , a τ θ ( s , z ) ( w ) d γ τ θ ( s , z ) . (44) \int_{S\times A\times W}h(s,a,w)d\rho_{\tau}^{\pi_{\mathrm{sep},\tau}^{\theta}(\mu)}(s,a,w)
=
\int_{S\times Z}\int_{W}h\left(s,a_{\tau}^{\theta}(s,z),w\right)d\widehat{\xi}_{\tau}^{s,a_{\tau}^{\theta}(s,z)}(w)d\gamma_{\tau}^{\theta}(s,z). \tag*{(44)} ∫ S × A × W h ( s , a , w ) d ρ τ π sep , τ θ ( μ ) ( s , a , w ) = ∫ S × Z ∫ W h ( s , a τ θ ( s , z ) , w ) d ξ τ s , a τ θ ( s , z ) ( w ) d γ τ θ ( s , z ) . (44)
Indeed, (41) says that applying ( s , z ) ↦ ( s , a τ θ ( s , z ) ) (s,z) \mapsto(s,a^\theta_\tau(s,z)) ( s , z ) ↦ ( s , a τ θ ( s , z )) to γ τ θ \gamma^\theta_\tau γ τ θ gives the state-action distribution chosen by the policy. Adding the realized noise kernel then gives ρ τ , s e p , τ θ ( μ ) \rho^{\theta}_{\tau,\mathrm{sep},\tau}(\mu) ρ τ , sep , τ θ ( μ ) , which proves the identity.
Step 2: Build a feasible next-stage transport plan. We now push the measure on ( s , z , w ) (s,z,w) ( s , z , w ) forward under the following map:
( s , z , w ) ⟼ ( f τ D ( s , a τ θ ( s , z ) , w ) , z ) . (s,z,w) \longmapsto(f^D_\tau(s,a^\theta_\tau(s,z),w),z). ( s , z , w ) ⟼ ( f τ D ( s , a τ θ ( s , z ) , w ) , z ) .
Its first marginal is μ τ + 1 D , π s e p , τ θ ( μ ) \mu^{D,\pi^\theta_{\mathrm{sep},\tau}(\mu)}_{\tau+1} μ τ + 1 D , π sep , τ θ ( μ ) by (27) and the identity (43), while its second marginal is ν \nu ν , because the map leaves the target coordinate unchanged. The pushforward is therefore an admissible coupling in the transport problem
K [ j τ + 1 D ] ( μ τ + 1 D , π s e p , τ θ ( μ ) , ν ) . \mathcal{K}[j^D_{\tau+1}]\left(\mu^{D,\pi^\theta_{\mathrm{sep},\tau}(\mu)}_{\tau+1},\nu\right). K [ j τ + 1 D ] ( μ τ + 1 D , π sep , τ θ ( μ ) , ν ) .
Hence, this coupling handles the next-stage term of the Bellman objective, it remains to show that the coupling also accounts for the current stage cost. To do so, take h = g τ D h=g^D_\tau h = g τ D in (43) and write the stage cost using the same measure on ( s , z , w ) (s,z,w) ( s , z , w ) :
∫ S × A × W g τ D ( s , a , w ) d ρ τ , s e p , τ θ ( μ ) ( s , a , w ) = ∫ S × Z ∫ W g τ D ( s , a τ θ ( s , z ) , w ) d ξ ^ τ s , a τ θ ( s , z ) ( w ) d γ τ θ ( s , z ) . \int_{S\times A\times W} g^D_\tau(s,a,w)\,\mathrm{d}\rho^{\theta}_{\tau,\mathrm{sep},\tau}(\mu)(s,a,w)
=
\int_{S\times Z}\int_W g^D_\tau(s,a^\theta_\tau(s,z),w)\,\mathrm{d}\hat{\xi}^{s,a^\theta_\tau(s,z)}_\tau(w)\,\mathrm{d}\gamma^\theta_\tau(s,z). ∫ S × A × W g τ D ( s , a , w ) d ρ τ , sep , τ θ ( μ ) ( s , a , w ) = ∫ S × Z ∫ W g τ D ( s , a τ θ ( s , z ) , w ) d ξ ^ τ s , a τ θ ( s , z ) ( w ) d γ τ θ ( s , z ) .
Thus, both parts of the Bellman objective can be bounded using one integral with respect to the measure constructed above.
Step 3: Bound the cost of this coupling. We now combine the two parts of the Bellman objective and use the near-optimality of the target assignment and the actions to bound the total cost. Start by observing that the next-stage transport cost K [ j τ + 1 D ] ( μ τ + 1 D , π s e p , τ θ ( μ ) , ν ) \mathcal{K}[j^D_{\tau+1}]\left(\mu^{D,\pi^\theta_{\mathrm{sep},\tau}(\mu)}_{\tau+1},\nu\right) K [ j τ + 1 D ] ( μ τ + 1 D , π sep , τ θ ( μ ) , ν ) is the infimum over all admissible couplings, so evaluating it at the coupling just constructed gives an upper bound. Consequently, for the fixed realization of the noise field in the previous step, and writing a = a τ θ ( s , z ) a=a^\theta_\tau(s,z) a = a τ θ ( s , z ) , the Bellman objective of the separation-based policy is bounded by
∫ S × A × W g τ D ( s , a , w ) d ρ τ , s e p , τ θ ( μ ) ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 D , π s e p , τ θ ( μ ) , ν ) ≤ ∫ S × Z ∫ W [ g τ D ( s , a , w ) + j τ + 1 D ( f τ D ( s , a , w ) , z ) ] d ξ ^ τ s , a ( w ) d γ τ θ ( s , z ) . \int_{S\times A\times W} g^D_\tau(s,a,w)\,\mathrm{d}\rho^{\theta}_{\tau,\mathrm{sep},\tau}(\mu)(s,a,w)+\mathcal{K}[j^D_{\tau+1}]\left(\mu^{D,\pi^\theta_{\mathrm{sep},\tau}(\mu)}_{\tau+1},\nu\right)
\le\int_{S\times Z}\int_W \left[g^D_\tau(s,a,w)+j^D_{\tau+1}(f^D_\tau(s,a,w),z)\right]\,\mathrm{d}\hat{\xi}^{s,a}_\tau(w)\,\mathrm{d}\gamma^\theta_\tau(s,z). ∫ S × A × W g τ D ( s , a , w ) d ρ τ , sep , τ θ ( μ ) ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 D , π sep , τ θ ( μ ) , ν ) ≤ ∫ S × Z ∫ W [ g τ D ( s , a , w ) + j τ + 1 D ( f τ D ( s , a , w ) , z ) ] d ξ ^ τ s , a ( w ) d γ τ θ ( s , z ) .
To average this bound over the noise field, fix μ \mu μ and γ τ θ \gamma^\theta_\tau γ τ θ . The universally measurable selector a τ θ a^\theta_\tau a τ θ has a Borel version b : S × Z → A b:S\times Z\to A b : S × Z → A that agrees with it γ τ θ \gamma^\theta_\tau γ τ θ -almost everywhere. Such a version follows by applying [31 Lemma 7.28(c)] to the selector’s Dirac kernel and inverting the Dirac embedding on the resulting full-measure set. Replacing the selector by b b b changes neither the control nor the integrals. With a = b ( s , z ) a=b(s,z) a = b ( s , z ) , the cost in square brackets is nonnegative and lower semianalytic. The nominal kernel and the law of the realized kernel are Borel after this substitution on a γ τ θ \gamma^\theta_\tau γ τ θ -full set, where Assumptions A.1 and A.2 hold. Extending the kernels off that set, we may apply Lemma A.10 with base measure γ τ θ \gamma^\theta_\tau γ τ θ on S × Z S\times Z S × Z and a singleton action space. Since the coupling and selector were fixed before the noise realization, taking expectations and recognizing the Bellman operator gives
T τ D , π s e p θ [ K [ j τ + 1 D ] ] ( μ , ν ) = E η τ [ ∫ S × A × W g τ D ( s , a , w ) d ρ τ , s e p , τ θ ( μ ) ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 D , π s e p , τ θ ( μ ) , ν ) ] ≤ E η τ [ ∫ S × Z ∫ W [ g τ D ( s , a , w ) + j τ + 1 D ( f τ D ( s , a , w ) , z ) ] d ξ ^ τ s , a ( w ) d γ τ θ ( s , z ) ] = ∫ S × Z ∫ W [ g τ D ( s , a , w ) + j τ + 1 D ( f τ D ( s , a , w ) , z ) ] d ξ τ ( w ∣ s , a ) d γ τ θ ( s , z ) , \begin{aligned}
\mathcal{T}^{D,\pi^\theta_{\mathrm{sep}}}_{\tau}\left[\mathcal{K}[j^D_{\tau+1}]\right](\mu,\nu)
=
\mathbb{E}_{\eta_\tau}\left[\int_{S\times A\times W} g^D_\tau(s,a,w)\,\mathrm{d}\rho^{\theta}_{\tau,\mathrm{sep},\tau}(\mu)(s,a,w)+\mathcal{K}[j^D_{\tau+1}]\left(\mu^{D,\pi^\theta_{\mathrm{sep},\tau}(\mu)}_{\tau+1},\nu\right)\right] \\
&\le\mathbb{E}_{\eta_\tau}\left[\int_{S\times Z}\int_W \left[g^D_\tau(s,a,w)+j^D_{\tau+1}(f^D_\tau(s,a,w),z)\right]\,\mathrm{d}\hat{\xi}^{s,a}_\tau(w)\,\mathrm{d}\gamma^\theta_\tau(s,z)\right] \\
&= \int_{S\times Z}\int_W \left[g^D_\tau(s,a,w)+j^D_{\tau+1}(f^D_\tau(s,a,w),z)\right]\,\mathrm{d}\xi_\tau(w\mid s,a)\,\mathrm{d}\gamma^\theta_\tau(s,z),
\end{aligned} T τ D , π sep θ [ K [ j τ + 1 D ] ] ( μ , ν ) = E η τ [ ∫ S × A × W g τ D ( s , a , w ) d ρ τ , sep , τ θ ( μ ) ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 D , π sep , τ θ ( μ ) , ν ) ] ≤ E η τ [ ∫ S × Z ∫ W [ g τ D ( s , a , w ) + j τ + 1 D ( f τ D ( s , a , w ) , z ) ] d ξ ^ τ s , a ( w ) d γ τ θ ( s , z ) ] = ∫ S × Z ∫ W [ g τ D ( s , a , w ) + j τ + 1 D ( f τ D ( s , a , w ) , z ) ] d ξ τ ( w ∣ s , a ) d γ τ θ ( s , z ) ,
where the last equality is exactly the application of Lemma A.10 with the expression in square brackets as the cost.
By (40), the inner integral is at most j τ D ( s , z ) + θ τ / 2 j^{D}_{\tau}(s,z)+\theta_{\tau}/2 j τ D ( s , z ) + θ τ /2 . Therefore,
T τ D , π s e p θ [ K [ j τ + 1 D ] ] ( μ , ν ) ≤ ∫ S × Z [ j τ D ( s , z ) + θ τ 2 ] d γ θ ( s , z ) ≤ K [ j τ D ] ( μ , ν ) + θ τ , \mathcal{T}^{D,\pi^{\theta}_{\mathrm{sep}}}_{\tau}\left[\mathcal{K}[j^{D}_{\tau+1}]\right](\mu,\nu)\leq\int_{S\times Z}\left[j^{D}_{\tau}(s,z)+\frac{\theta_{\tau}}{2}\right]\mathrm{d}\gamma^{\theta}(s,z)\leq\mathcal{K}[j^{D}_{\tau}](\mu,\nu)+\theta_{\tau}, T τ D , π sep θ [ K [ j τ + 1 D ] ] ( μ , ν ) ≤ ∫ S × Z [ j τ D ( s , z ) + 2 θ τ ] d γ θ ( s , z ) ≤ K [ j τ D ] ( μ , ν ) + θ τ ,
where the last inequality uses the near-optimality of the target assignment in (39) to contribute another θ τ / 2 \theta_{\tau}/2 θ τ /2 . This is exactly (42).
The previous lemma controls the cost of the chosen policy relative to the decoupled transport surrogate. To compare that surrogate with the decoupled optimal Bellman step, we also use the result proved in Appendix D. In particular, applying Lemma D.3 to the decoupled model gives
0 ≤ K [ j τ D ] ( μ , ν ) − T τ D [ K [ j τ + 1 D ] ] ( μ , ν ) ≤ L τ + 1 D τ . (45) 0\leq\mathcal{K}[j^{D}_{\tau}](\mu,\nu)-\mathcal{T}^{D}_{\tau}\left[\mathcal{K}[j^{D}_{\tau+1}]\right](\mu,\nu)\leq L_{\tau+1}D_{\tau}. \tag*{(45)} 0 ≤ K [ j τ D ] ( μ , ν ) − T τ D [ K [ j τ + 1 D ] ] ( μ , ν ) ≤ L τ + 1 D τ . (45)
We now have the three one-step estimates needed to derive the full performance bound. (i) Lemma E.5 controls the separation-based policy in the decoupled model, (ii) Lemma E.4 transfers a Bellman step between the coupled and decoupled models, and (iii) (44) compares the decoupled transport surrogate with the decoupled optimal Bellman step. The theorem below accumulates these errors over the remaining time steps.
Theorem E.6 (Performance of separation under weak coupling). Consider the coupled and decoupled models defined above. Assume that the field satisfies Assumptions A.1 and A.2, that the decoupled single-agent cost-to-go satisfies Assumption D.1 with constants L τ L_{\tau} L τ , and that Assumption E.1 holds with constants ε τ f \varepsilon^{f}_{\tau} ε τ f and δ τ \delta_{\tau} δ τ . Let π s e p θ \pi^{\theta}_{\mathrm{sep}} π sep θ be the policy defined in (39)–(41). Then, for every τ ∈ { 0 , … , T } \tau\in\{0,\ldots,T\} τ ∈ { 0 , … , T } , μ ∈ P ( S ) \mu\in\mathcal{P}(S) μ ∈ P ( S ) with finite first moment, and ν ∈ P ( Z ) \nu\in\mathcal{P}(Z) ν ∈ P ( Z ) ,
0 ≤ J τ C , π s e p θ ( μ , ν ) − J τ C ( μ , ν ) ≤ ∑ r = τ T − 1 [ θ r + L r + 1 D r + 2 δ r + 2 L r + 1 ε r f ] . (46) 0\leq J^{C,\pi^{\theta}_{\mathrm{sep}}}_{\tau}(\mu,\nu)-J^{C}_{\tau}(\mu,\nu)\leq\sum_{r=\tau}^{T-1}\left[\theta_{r}+L_{r+1}D_{r}+2\delta_{r}+2L_{r+1}\varepsilon^{f}_{r}\right]. \tag*{(46)} 0 ≤ J τ C , π sep θ ( μ , ν ) − J τ C ( μ , ν ) ≤ r = τ ∑ T − 1 [ θ r + L r + 1 D r + 2 δ r + 2 L r + 1 ε r f ] . (46) When the transport and action minima are attained measurably, we can take θ r = 0 \theta_{r}=0 θ r = 0 for every r r r . In this case, at the initial population μ ˉ 0 \bar{\mu}_{0} μ ˉ 0 , the bound becomes
J 0 C , π s e p 0 ( μ ˉ 0 , ν ) − J 0 C ( μ ˉ 0 , ν ) ≤ ∑ r = 0 T − 1 L r + 1 D r + 2 ∑ r = 0 T − 1 ( δ r + L r + 1 ε r f ) . (47) J^{C,\pi^{0}_{\mathrm{sep}}}_{0}(\bar{\mu}_{0},\nu)-J^{C}_{0}(\bar{\mu}_{0},\nu)\leq\sum_{r=0}^{T-1}L_{r+1}D_{r}+2\sum_{r=0}^{T-1}\left(\delta_{r}+L_{r+1}\varepsilon^{f}_{r}\right). \tag*{(47)} J 0 C , π sep 0 ( μ ˉ 0 , ν ) − J 0 C ( μ ˉ 0 , ν ) ≤ r = 0 ∑ T − 1 L r + 1 D r + 2 r = 0 ∑ T − 1 ( δ r + L r + 1 ε r f ) . (47)
Proof. We use the decoupled transport cost as a common reference. First, we bound from above the cost incurred by the separation-based policy in the coupled model. Second, we bound the coupled optimal value from below using the same reference. Finally, we subtract the second estimate from the first to obtain the claimed performance bound. Both estimates are proved backward in time. At τ = T \tau=T τ = T , the policy value, the coupled optimum, and K [ j T D ] \mathcal{K}[j^{D}_{T}] K [ j T D ] all equal the common terminal cost in (28), so both inductions have the same base case. Explicitly, for every ( μ , ν ) (\mu,\nu) ( μ , ν ) ,
J T C , π s e p θ ( μ , ν ) = J T C ( μ , ν ) = K [ j T D ] ( μ , ν ) = K [ g T ] ( μ , ν ) . J^{C,\pi^{\theta}_{\mathrm{sep}}}_{T}(\mu,\nu)=J^{C}_{T}(\mu,\nu)=\mathcal{K}[j^{D}_{T}](\mu,\nu)=\mathcal{K}[g_{T}](\mu,\nu). J T C , π sep θ ( μ , ν ) = J T C ( μ , ν ) = K [ j T D ] ( μ , ν ) = K [ g T ] ( μ , ν ) .
Since the sums in (47) and (48) are empty at τ = T \tau=T τ = T , these equalities directly prove both base cases.
In both inductions that follow, we use the monotonicity and constant-shift properties of the Bellman operators (Lemma D.2). Indeed, the lemma is stated for T τ D = T τ \mathcal{T}^{D}_{\tau}=\mathcal{T}_{\tau} T τ D = T τ , and since its proof uses only the structure of the definition, it also applies verbatim to T τ C \mathcal{T}^{C}_{\tau} T τ C and to the fixed-policy operators T τ m , π \mathcal{T}^{m,\pi}_{\tau} T τ m , π .
Step 1: Upper bound for the separation-based policy. We first show that the policy cost in the coupled model is at most the decoupled transport cost plus the near-optimality and model-mismatch errors. More precisely, for every ( μ , ν ) (\mu,\nu) ( μ , ν ) ,
J τ C , π s e p θ ( μ , ν ) ≤ K [ j τ D ] ( μ , ν ) + ∑ r = τ T − 1 ( θ r + δ r + L r + 1 ε r f ) . (48) J^{C,\pi^{\theta}_{\mathrm{sep}}}_{\tau}(\mu,\nu)\leq\mathcal{K}[j^{D}_{\tau}](\mu,\nu)+\sum_{r=\tau}^{T-1}\left(\theta_{r}+\delta_{r}+L_{r+1}\varepsilon^{f}_{r}\right). \tag*{(48)} J τ C , π sep θ ( μ , ν ) ≤ K [ j τ D ] ( μ , ν ) + r = τ ∑ T − 1 ( θ r + δ r + L r + 1 ε r f ) . (48)
Assume, by backward induction, that (47) holds at time τ + 1 \tau+1 τ + 1 for every population. The induction hypothesis says that, from any such population, the cost of following the separation-based policy is at most the corresponding decoupled transport cost plus the errors accumulated from times τ + 1 \tau+1 τ + 1 to T − 1 T-1 T − 1 .
Now fix ( μ , ν ) (\mu,\nu) ( μ , ν ) and apply the induction hypothesis at the realized next population μ τ + 1 C , π s e p , τ θ ( μ ) \mu_{\tau+1}^{C,\pi_{\mathrm{sep},\tau}^{\theta}(\mu)} μ τ + 1 C , π sep , τ θ ( μ ) . We obtain
J τ C , π s e p θ ( μ , ν ) = E η τ [ ∫ S × A × W q τ C ( s , a , μ , w ) d ρ τ π s e p , τ θ ( μ ) ( s , a , w ) + J τ + 1 C , π s e p θ ( μ τ + 1 C , π s e p , τ θ ( μ ) , ν ) ] ≤ E η τ [ ∫ S × A × W q τ C ( s , a , μ , w ) d ρ τ π s e p , τ θ ( μ ) ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 C , π s e p , τ θ ( μ ) , ν ) ] + ∑ r = τ + 1 T − 1 ( θ r + δ r + L r + 1 ε r f ) = T τ C , π s e p θ ( K [ j τ + 1 D ] ( μ , ν ) + ∑ r = τ + 1 T − 1 ( θ r + δ r + L r + 1 ε r f ) ) ≤ T τ D , π s e p θ ( K [ j τ + 1 D ] ( μ , ν ) + δ τ + L τ + 1 ε τ f + ∑ r = τ + 1 T − 1 ( θ r + δ r + L r + 1 ε r f ) ] ≤ K [ j τ D ] ( μ , ν ) + ∑ r = τ T − 1 ( θ r + δ r + L r + 1 ε r f ) . \begin{aligned}
J_{\tau}^{C,\pi_{\mathrm{sep}}^{\theta}}(\mu,\nu)
&= \mathbb{E}_{\eta_{\tau}}\left[\int_{S\times A\times W}q_{\tau}^{C}(s,a,\mu,w)\,\mathrm{d}\rho_{\tau}^{\pi_{\mathrm{sep},\tau}^{\theta}(\mu)}(s,a,w)+J_{\tau+1}^{C,\pi_{\mathrm{sep}}^{\theta}}\left(\mu_{\tau+1}^{C,\pi_{\mathrm{sep},\tau}^{\theta}(\mu)},\nu\right)\right]\\
&\leq\mathbb{E}_{\eta_{\tau}}\left[\int_{S\times A\times W}q_{\tau}^{C}(s,a,\mu,w)\,\mathrm{d}\rho_{\tau}^{\pi_{\mathrm{sep},\tau}^{\theta}(\mu)}(s,a,w)+\mathcal{K}\left[j_{\tau+1}^{D}\right]\left(\mu_{\tau+1}^{C,\pi_{\mathrm{sep},\tau}^{\theta}(\mu)},\nu\right)\right]\\
&\quad+\sum_{r=\tau+1}^{T-1}\left(\theta_r+\delta_r+L_{r+1}\varepsilon_r^f\right)\\
&=\mathcal{T}_{\tau}^{C,\pi_{\mathrm{sep}}^{\theta}}\left(\mathcal{K}\left[j_{\tau+1}^{D}\right](\mu,\nu)+\sum_{r=\tau+1}^{T-1}\left(\theta_r+\delta_r+L_{r+1}\varepsilon_r^f\right)\right)\\
&\leq\mathcal{T}_{\tau}^{D,\pi_{\mathrm{sep}}^{\theta}}\left(\mathcal{K}\left[j_{\tau+1}^{D}\right](\mu,\nu)+\delta_{\tau}+L_{\tau+1}\varepsilon_{\tau}^f+\sum_{r=\tau+1}^{T-1}\left(\theta_r+\delta_r+L_{r+1}\varepsilon_r^f\right)\right]\\
&\leq\mathcal{K}\left[j_{\tau}^{D}\right](\mu,\nu)+\sum_{r=\tau}^{T-1}\left(\theta_r+\delta_r+L_{r+1}\varepsilon_r^f\right).
\end{aligned} J τ C , π sep θ ( μ , ν ) = E η τ [ ∫ S × A × W q τ C ( s , a , μ , w ) d ρ τ π sep , τ θ ( μ ) ( s , a , w ) + J τ + 1 C , π sep θ ( μ τ + 1 C , π sep , τ θ ( μ ) , ν ) ] ≤ E η τ [ ∫ S × A × W q τ C ( s , a , μ , w ) d ρ τ π sep , τ θ ( μ ) ( s , a , w ) + K [ j τ + 1 D ] ( μ τ + 1 C , π sep , τ θ ( μ ) , ν ) ] + r = τ + 1 ∑ T − 1 ( θ r + δ r + L r + 1 ε r f ) = T τ C , π sep θ ( K [ j τ + 1 D ] ( μ , ν ) + r = τ + 1 ∑ T − 1 ( θ r + δ r + L r + 1 ε r f ) ) ≤ T τ D , π sep θ ( K [ j τ + 1 D ] ( μ , ν ) + δ τ + L τ + 1 ε τ f + r = τ + 1 ∑ T − 1 ( θ r + δ r + L r + 1 ε r f ) ] ≤ K [ j τ D ] ( μ , ν ) + r = τ ∑ T − 1 ( θ r + δ r + L r + 1 ε r f ) .
The second inequality is (35) in Lemma E.4, which costs δ τ + L τ + 1 ε τ f \delta_{\tau}+L_{\tau+1}\varepsilon_{\tau}^f δ τ + L τ + 1 ε τ f , while the last inequality is (42) in Lemma E.5, which contributes θ τ \theta_{\tau} θ τ . After adding these current errors to the errors from later times, we obtain (47) at time τ \tau τ and prove the upper bound.
Step 2: Lower bound for the coupled optimum. We now compare the coupled optimal value with the same decoupled transport cost. Here the transport cost provides a lower bound after we subtract the separation and model-mismatch errors:
J τ C ( μ , ν ) ≥ K [ j τ D ] ( μ , ν ) − ∑ r = τ T − 1 [ L r + 1 D r + δ r + L r + 1 ε r f ] . (49) J_{\tau}^{C}(\mu,\nu)\geq\mathcal{K}\left[j_{\tau}^{D}\right](\mu,\nu)-\sum_{r=\tau}^{T-1}\left[L_{r+1}D_r+\delta_r+L_{r+1}\varepsilon_r^f\right]. \tag*{(49)} J τ C ( μ , ν ) ≥ K [ j τ D ] ( μ , ν ) − r = τ ∑ T − 1 [ L r + 1 D r + δ r + L r + 1 ε r f ] . (49)
Now fix ( μ , ν ) (\mu,\nu) ( μ , ν ) . Since the induction hypothesis is a pointwise lower bound on J τ + 1 C J_{\tau+1}^{C} J τ + 1 C , applying it inside T τ C \mathcal{T}_{\tau}^{C} T τ C and using Lemma D.2 gives
J τ C ( μ , ν ) = T τ C [ J τ + 1 C ] ( μ , ν ) ≥ T τ C [ K [ j τ + 1 D ] ( μ , ν ) − ∑ r = τ + 1 T − 1 [ L r + 1 D r + δ r + L r + 1 ε r f ] ] ≥ T τ D [ K [ j τ + 1 D ] ( μ , ν ) ] − δ τ − L τ + 1 ε τ f − ∑ r = τ + 1 T − 1 [ L r + 1 D r + δ r + L r + 1 ε r f ] ≥ K [ j τ D ] ( μ , ν ) − ∑ r = τ T − 1 [ L r + 1 D r + δ r + L r + 1 ε r f ] . \begin{aligned}
J_{\tau}^{C}(\mu,\nu)
&=\mathcal{T}_{\tau}^{C}\left[J_{\tau+1}^{C}\right](\mu,\nu)\\
&\geq\mathcal{T}_{\tau}^{C}\left[\mathcal{K}\left[j_{\tau+1}^{D}\right](\mu,\nu)-\sum_{r=\tau+1}^{T-1}\left[L_{r+1}D_r+\delta_r+L_{r+1}\varepsilon_r^f\right]\right]\\
&\geq\mathcal{T}_{\tau}^{D}\left[\mathcal{K}\left[j_{\tau+1}^{D}\right](\mu,\nu)\right]-\delta_{\tau}-L_{\tau+1}\varepsilon_{\tau}^f-\sum_{r=\tau+1}^{T-1}\left[L_{r+1}D_r+\delta_r+L_{r+1}\varepsilon_r^f\right]\\
&\geq\mathcal{K}\left[j_{\tau}^{D}\right](\mu,\nu)-\sum_{r=\tau}^{T-1}\left[L_{r+1}D_r+\delta_r+L_{r+1}\varepsilon_r^f\right].
\end{aligned} J τ C ( μ , ν ) = T τ C [ J τ + 1 C ] ( μ , ν ) ≥ T τ C [ K [ j τ + 1 D ] ( μ , ν ) − r = τ + 1 ∑ T − 1 [ L r + 1 D r + δ r + L r + 1 ε r f ] ] ≥ T τ D [ K [ j τ + 1 D ] ( μ , ν ) ] − δ τ − L τ + 1 ε τ f − r = τ + 1 ∑ T − 1 [ L r + 1 D r + δ r + L r + 1 ε r f ] ≥ K [ j τ D ] ( μ , ν ) − r = τ ∑ T − 1 [ L r + 1 D r + δ r + L r + 1 ε r f ] .
Here, the second inequality is (34) in Lemma E.4, which contributes δ τ + L τ + 1 ε τ f \delta_{\tau}+L_{\tau+1}\varepsilon_{\tau}^f δ τ + L τ + 1 ε τ f . Rearranging (44) gives
T τ D [ K [ j τ + 1 D ] ] ( μ , ν ) ≥ K [ j τ D ] ( μ , ν ) − L τ + 1 D τ , \mathcal{T}_{\tau}^{D}\left[\mathcal{K}\left[j_{\tau+1}^{D}\right]\right](\mu,\nu)\geq\mathcal{K}\left[j_{\tau}^{D}\right](\mu,\nu)-L_{\tau+1}D_{\tau}, T τ D [ K [ j τ + 1 D ] ] ( μ , ν ) ≥ K [ j τ D ] ( μ , ν ) − L τ + 1 D τ ,
which is the last inequality in the display. This proves (48) at time τ \tau τ .
Step 3: Combine the two bounds. Subtracting the obtained lower bound (49) from the upper bound (48) cancels the common decoupled transport cost and gives
J T C , π s e p θ ( μ , ν ) − J T C ( μ , ν ) ≤ ∑ r = τ T − 1 ( θ r + δ r + L r + 1 ε r f ) + ∑ r = τ T − 1 ( L r + 1 D r + δ r + L r + 1 ε r f ) = ∑ r = τ T − 1 [ θ r + L r + 1 D r + 2 δ r + 2 L r + 1 ε r f ] . J_T^{C,\pi_{\mathrm{sep}}^\theta}(\mu,\nu)-J_T^C(\mu,\nu) \le\sum_{r=\tau}^{T-1}\left(\theta_r+\delta_r+L_{r+1}\varepsilon_r^f\right)+\sum_{r=\tau}^{T-1}\left(L_{r+1}D_r+\delta_r+L_{r+1}\varepsilon_r^f\right)
= \sum_{r=\tau}^{T-1}\left[\theta_r+L_{r+1}D_r+2\delta_r+2L_{r+1}\varepsilon_r^f\right]. J T C , π sep θ ( μ , ν ) − J T C ( μ , ν ) ≤ r = τ ∑ T − 1 ( θ r + δ r + L r + 1 ε r f ) + r = τ ∑ T − 1 ( L r + 1 D r + δ r + L r + 1 ε r f ) = r = τ ∑ T − 1 [ θ r + L r + 1 D r + 2 δ r + 2 L r + 1 ε r f ] .
That corresponds exactly to the upper bound in (46) . To conclude the proof, it is enough to note that the zero lower bound is immediate because π s e p θ \pi_{\mathrm{sep}}^\theta π sep θ is admissible in the coupled model and J T C J_T^C J T C is the infimum of the costs over all admissible policies there. Finally, if exact minimizers can be selected measurably, setting τ = 0 \tau=0 τ = 0 , μ = μ ˉ 0 \mu=\bar{\mu}_0 μ = μ ˉ 0 , and θ r = 0 \theta_r=0 θ r = 0 gives (47) .
Interpretation. The three terms in the bound correspond to the three comparisons used in the proof. The tolerance θ r \theta_r θ r comes from the near-optimal target plan and action used to define the separation-based policy in (40) –(42) . The term L r + 1 D r L_{r+1}D_r L r + 1 D r is already present in Lemma D.3 for the decoupled problem and bounds the difference between assigning targets before the noise is observed and optimizing the population Bellman step after the resulting population is known. Finally, δ r + L r + 1 ε r f \delta_r+L_{r+1}\varepsilon_r^f δ r + L r + 1 ε r f is the cost of replacing one coupled Bellman step by its decoupled counterpart, as bounded in Lemma E.4 . This last term appears twice because the proof makes that replacement once for the deployed policy and once for the coupled optimal value.
Bounding the model mismatch. Note that Assumption E.1 only quantifies the model mismatch in terms of the abstract mismatch ε τ f \varepsilon_\tau^f ε τ f . To use the theorem in a concrete model, it is often more convenient to relate the quantity to the strength of the population interaction. This is the case, for example, when the coupled dynamics are Lipschitz in the population distribution:
d S ( f τ C ( s , a , μ , w ) , f τ C ( s , a , μ ~ , w ) ) ≤ ι τ W 1 ( μ , μ ~ ) , (50) d_S\left(f_\tau^C(s,a,\mu,w),f_\tau^C(s,a,\widetilde{\mu},w)\right)\le\iota_\tau W_1(\mu,\widetilde{\mu}), \tag*{(50)} d S ( f τ C ( s , a , μ , w ) , f τ C ( s , a , μ , w ) ) ≤ ι τ W 1 ( μ , μ ) , (50)
uniformly in ( s , a , w ) (s,a,w) ( s , a , w ) . In this case, indeed, the constant ι τ \iota_\tau ι τ measures how strongly the population affects the dynamics. If, in addition, the decoupled surrogate is obtained by freezing that population argument at a reference profile μ τ r e f \mu_\tau^{\mathrm{ref}} μ τ ref :
f τ D ( s , a , w ) = f τ C ( s , a , μ τ r e f , w ) , f_\tau^D(s,a,w)=f_\tau^C(s,a,\mu_\tau^{\mathrm{ref}},w), f τ D ( s , a , w ) = f τ C ( s , a , μ τ ref , w ) ,
and it holds that W 1 ( μ , μ τ r e f ) ≤ R τ W_1(\mu,\mu_\tau^{\mathrm{ref}})\le R_\tau W 1 ( μ , μ τ ref ) ≤ R τ at every population where the two models are compared, then we can apply (50) with μ ~ = μ τ r e f \widetilde{\mu}=\mu_\tau^{\mathrm{ref}} μ = μ τ ref . By the definition of f τ D f_\tau^D f τ D , this allows us to bound the general model mismatch in Assumption E.1 by the more informative ε τ f ≤ ι τ R τ \varepsilon_\tau^f\le\iota_\tau R_\tau ε τ f ≤ ι τ R τ . Under this class of models, (47) becomes
J 0 C , π s e p 0 ( μ ˉ 0 , ν ) − J 0 C ( μ ˉ 0 , ν ) ≤ ∑ τ = 0 T − 1 L τ + 1 D τ + 2 ∑ τ = 0 T − 1 ( δ τ + L τ + 1 ι τ R τ ) . J_0^{C,\pi_{\mathrm{sep}}^0}(\bar{\mu}_0,\nu)-J_0^C(\bar{\mu}_0,\nu)\le\sum_{\tau=0}^{T-1}L_{\tau+1}D_\tau+2\sum_{\tau=0}^{T-1}\left(\delta_\tau+L_{\tau+1}\iota_\tau R_\tau\right). J 0 C , π sep 0 ( μ ˉ 0 , ν ) − J 0 C ( μ ˉ 0 , ν ) ≤ τ = 0 ∑ T − 1 L τ + 1 D τ + 2 τ = 0 ∑ T − 1 ( δ τ + L τ + 1 ι τ R τ ) .
If, in addition, the stage costs coincide, so that δ τ = 0 \delta_\tau=0 δ τ = 0 , then
J 0 C , π s e p 0 ( μ ˉ 0 , ν ) − J 0 C ( μ ˉ 0 , ν ) ≤ ∑ τ = 0 T − 1 L τ + 1 D τ + 2 ∑ τ = 0 T − 1 L τ + 1 ι τ R τ . (51) J_0^{C,\pi_{\mathrm{sep}}^0}(\bar{\mu}_0,\nu)-J_0^C(\bar{\mu}_0,\nu)\le\sum_{\tau=0}^{T-1}L_{\tau+1}D_\tau+2\sum_{\tau=0}^{T-1}L_{\tau+1}\iota_\tau R_\tau. \tag*{(51)} J 0 C , π sep 0 ( μ ˉ 0 , ν ) − J 0 C ( μ ˉ 0 , ν ) ≤ τ = 0 ∑ T − 1 L τ + 1 D τ + 2 τ = 0 ∑ T − 1 L τ + 1 ι τ R τ . (51)
The second sum in (51) corresponds to the additional error due to weak coupling: it grows linearly in ι τ \iota_\tau ι τ and vanishes when the dynamics stop depending on the population, in which case (51) reduces to the bound of Theorem D.4 .
Additional numerical results
This appendix provides the implementation details and complementary results for the experiments in Section 5. We first describe the grid-world experiments and report the additional performance metrics used in our evaluation (Appendix F.1). We then give details of the scalability study (Appendix F.2) and examine how the origin–destination pairs covered during training affect transfer across population sizes (Appendix F.3). Finally, we describe the Manhattan road network (Appendix F.4) and report the computational resources used in the experiments.
Additional details for the grid-world experiments in Section 5.1
Environment. We use an 11 × 30 grid with horizon T = 100 T = 100 T = 100 and N = 7 N = 7 N = 7 agents. At the beginning of each episode, the agents are placed uniformly at random in the first column, while the N N N targets are sampled uniformly from the last ten columns. The action space is A = { up , down , left , right , stay } \mathcal{A} = \{\text{up}, \text{down}, \text{left}, \text{right}, \text{stay}\} A = { up , down , left , right , stay } . At each time step, the selected action is executed with probability 1 − p 1 - p 1 − p and is replaced uniformly by one of the other four actions with probability p p p . We consider p ∈ { 0 , 0.1 , 0.25 } p \in\{0, 0.1, 0.25\} p ∈ { 0 , 0.1 , 0.25 } under the individual, local, and global noise models defined in Appendix A. Specifically, under individual noise, each agent receives an independent noise draw, under local noise, agents with the same state-action pair share the same draw, while under global noise, the entire population shares one draw. We finally clip the resulting positions of the agents to make them remain in the grid after each transition.
An agent becomes inactive after reaching a relevant target and enters an absorbing state. In the case of a MARL agent, a relevant target for an agent is any previously unoccupied target, in the case of SALT, instead, each agent is assigned a target z z z before the rollout, and the agent becomes inactive when it reaches that specific target.2 For SALT with a fixed assignment, both the routing policy and the stopping rule depend only on each agent’s own state and target. Conditional on the initial states and assignments, the trajectories are independent under individual noise, but may be correlated under local or global noise.
Reward structure. An active agent incurs a stage cost of 1 on every transition until it does not reach a target. At the horizon, the MARL training objective also includes the empirical optimal transport cost K [ c ] ( μ T , ν ) \mathcal{K}[c](\mu_T, \nu) K [ c ] ( μ T , ν ) , where c c c is the Manhattan distance. In the target-conditioned single-agent problems used by SALT, the corresponding terminal penalty is the Manhattan distance from the agent to its assigned target. At evaluation, we report travel time, reach rate, and empirical terminal optimal transport cost separately.
Baselines. We compare SALT with MAPPO, IPPO, HAPPO, VDN, QMIX, and MFQ. All baselines use the same episode budget and training seeds. All neural policies have two hidden layers with 64 units and tanh activations. The separation-based neural methods are trained as goal-conditioned single-agent controllers. For SALT-PPO, SALT-A2C, and SALT-DDQN, the policy receives the input
( r H g , c W g , r z H g , c z W g , t T − 1 , ∥ s − z ∥ 1 H g + W g ) , \left(\frac{r}{H_g},\frac{c}{W_g},\frac{r_z}{H_g},\frac{c_z}{W_g},\frac{t}{T-1},\frac{\lVert s-z\rVert_1}{H_g+W_g}\right), ( H g r , W g c , H g r z , W g c z , T − 1 t , H g + W g ∥ s − z ∥ 1 ) ,
where s = ( r , c ) s = (r,c) s = ( r , c ) is the current agent position, z = ( r z , c z ) z = (r_z,c_z) z = ( r z , c z ) is its assigned target, t t t is the time step, and H g = 10 H_g = 10 H g = 10 and W g = 29 W_g = 29 W g = 29 are the maximum row and column indices. The separation policy therefore selects an action without observing the full population. Coordination between agents occurs only through solving the optimal transport problem, whose solution selects z z z .
MAPPO, IPPO, HAPPO, QMIX, VDN, and MFQ instead receive target-relative local observations. For agent i i i , this observation has dimension 5 + 3 N 5 + 3N 5 + 3 N :
( r i H g , c i W g , i N − 1 , t T − 1 , a i , Δ r i 1 H g , Δ c i 1 W g , b 1 , … , Δ r i N H g , Δ c i N W g , b N ) . \left(\frac{r_i}{H_g},\frac{c_i}{W_g},\frac{i}{N-1},\frac{t}{T-1},a_i,\frac{\Delta r_{i1}}{H_g},\frac{\Delta c_{i1}}{W_g},b_1,\ldots,\frac{\Delta r_{iN}}{H_g},\frac{\Delta c_{iN}}{W_g},b_N\right). ( H g r i , W g c i , N − 1 i , T − 1 t , a i , H g Δ r i 1 , W g Δ c i 1 , b 1 , … , H g Δ r i N , W g Δ c i N , b N ) .
Here a i a_i a i indicates whether agent i i i is active, Δ r i j = r z j − r i \Delta r_{ij} = r_{z_j} - r_i Δ r ij = r z j − r i and Δ c i j = c z j − c i \Delta c_{ij} = c_{z_j} - c_i Δ c ij = c z j − c i are the coordinates of target j j j relative to agent i i i , and b j b_j b j indicates whether target j j j remains unoccupied. Thus, each agent observes the target set as an ordered list of relative offsets and availability flags. MAPPO also uses a centralized critic state of dimension 6 N + 1 6N + 1 6 N + 1 . This state contains all normalized agent positions, all active-agent flags, the time step, all normalized target positions, and all target-availability flags. HAPPO uses the same centralized critic state, but has one non-shared actor per agent and updates the actors sequentially. QMIX passes the global state to its monotone mixing network, whose embedding dimension is 32. IPPO uses a local critic, VDN uses additive value decomposition, and MFQ augments each agent’s action value with the global mean action distribution of the currently active agents. All neural methods are optimized with Adam [49 ] .
Training and evaluation. For each algorithm, noise model, and noise probability, we train five policies using seeds 0–4 and evaluate each policy on 50 rollout instances. Within a given condition, evaluation seed s ∈ { 0 , … , 49 } s \in\{0,\ldots,49\} s ∈ { 0 , … , 49 } fixes the starts, targets, and initial state of the transition-noise generator for every algorithm.
For each metric, we pool the 50 outcomes from each training seed, and report the arithmetic mean of the resulting 250 outcomes in the heatmaps in Figure 2 , Figure 5 , and Figure 6 . We then summarize all the results and include the corresponding confidence intervals in Figure 7 . In particular, the reach-rate bars in Figure 7 show the pooled means together with two-sided 95% confidence intervals, while the terminal-cost and travel-time boxplots show the median and interquartile range (from the boxes), and observed minimum and maximum of the pooled outcomes (from the whiskers).
Figure 5. Terminal optimal transport cost as a function of the noise probability p p p . Each cell shows the pooled arithmetic mean under the training and evaluation protocol described above. Figure 7 shows the corresponding distributions. Brighter cells indicate higher cost.
Figure 6. Reach rate as a function of the noise probability p p p . Each cell shows the pooled arithmetic mean under the training and evaluation protocol described above. Figure 7 reports the corresponding 95% confidence intervals. Darker cells indicate higher reach rate.
Figure 7. Reach rate, terminal optimal transport cost, and average travel time under the evaluation protocol described above. Reach-rate bars show pooled means and two-sided 95% crossed two-way bootstrap confidence intervals. The remaining panels show the pooled outcome distributions. Boxes span the interquartile range, center lines are medians, and whiskers show the observed minimum and maximum.
Hyperparameters. Table 2 , Table 3 , and Table 4 report all hyperparameters used in the grid-world experiments.
Table 2. Shared grid-world training budget and evaluation settings for Section 5.1.
Table 3. Hyperparameters used for SALT in Section 5.1.
Table 4. Hyperparameters used for MARL baselines in § 5.1 .
For SALT-PPO and SALT-A2C, the ϵ \epsilon ϵ schedule in Table 3 controls random exploration of the target assignment, while action exploration is induced by the stochastic actor. SALT-Q and SALT-DDQN use the same schedule for random assignments and ϵ \epsilon ϵ -greedy action exploration.
Additional performance metrics. In addition to average travel time, we report two further metrics. Figure 5 reports the empirical optimal transport cost between the final agent positions and the targets, that is, the minimum average Manhattan distance obtained by matching agents to targets. Figure 6 reports the fraction of agents that reach a target before the horizon. More precisely, let
μ T N = 1 N ∑ i = 1 N δ s T i , ν N = 1 N ∑ j = 1 N δ z j . \mu_T^N = \frac{1}{N}\sum_{i=1}^{N}\delta_{s_T^i}, \qquad\nu^N = \frac{1}{N}\sum_{j=1}^{N}\delta_{z^j}. μ T N = N 1 i = 1 ∑ N δ s T i , ν N = N 1 j = 1 ∑ N δ z j .
With this notation, the empirical terminal optimal transport cost is
K [ d ] ( μ T N , ν N ) = min σ ∈ S N 1 N ∑ i = 1 N d ( s T i , z σ ( i ) ) . \mathcal{K}[d](\mu_T^N,\nu^N) = \min_{\sigma\in S_N}\frac{1}{N}\sum_{i=1}^{N}d\left(s_T^i,z^{\sigma(i)}\right). K [ d ] ( μ T N , ν N ) = σ ∈ S N min N 1 i = 1 ∑ N d ( s T i , z σ ( i ) ) .
where S N S_N S N is the set of permutations of { 1 , … , N } \{1,\ldots,N\} { 1 , … , N } , s T i s_T^i s T i is the final position of agent i i i , z j z^j z j is target j j j , d ( ⋅ , ⋅ ) d(\cdot,\cdot) d ( ⋅ , ⋅ ) is the Manhattan distance, and σ \sigma σ assigns agents to targets. This cost is zero exactly when every target is occupied and, unlike the reach rate, it also distinguishes near misses from terminal configurations that remain far from the target set.
The three metrics complement each other in describing performance and should therefore be read together. Average travel time tells us how long the agents travel, but an unsuccessful trajectory may have the same travel time as a slow but successful one. The reach rate tells us how many agents actually reach a target, but treats all unsuccessful agents in the same way. The terminal optimal transport cost adds the remaining information by distinguishing final configurations that are close to full target coverage from those that remain far from it. Figure 7 reports the three metrics together. To interpret it recall that higher values indicate better performance for the reach rate, while lower values indicate better performance for terminal optimal transport cost and average travel time. We can observe that across all three metrics, the SALT variants achieve higher reach rates, lower terminal costs, and shorter travel times than the MARL baselines.
Additional details for the scalability experiment
We next study how the number of environment interactions required for training changes with the population size. For this experiment, we use square grids and individual noise with probability p = 0.1 p = 0.1 p = 0.1 . Each agent starts from an independently sampled row of the first column, while the targets are sampled independently and uniformly over the full grid.
To avoid confounding population size with target density, we scale the grid area approximately in proportion to N N N , the resulting grid sizes are reported in Table 5 . We select PPO as the single-agent learner in SALT and compare the resulting method with MAPPO. The hyperparameters are the same as in Table 3 and Table 4 . For each population size in Table 5 , we choose the maximum number of batches so that the per-agent-transition budget is approximately 2 × 10 9 2 \times10^{9} 2 × 1 0 9 . We record a reach-rate milestone at the first of 10 consecutive batches whose 64-episode reach rate meets the stated threshold in Figure 3 .
Table 5. Population sizes and grid sizes used in the scalability experiment.
Origin–destination coverage under population-size transfer
The transportation cost used by SALT is learned from target-conditioned trajectories. Consequently, an origin–destination (OD) pair selected only a few times during training may therefore have a less accurate cost-to-go estimate than a pair selected more often, and this can affect performance at evaluation especially when the population size differs from the one used during training. Indeed, as training progresses, a larger population gives the assignment layer more opportunities to select low-cost matchings, whereas some OD pairs may then be selected less often and become underrepresented in the training data. With fewer agents, however, we have a limited number of available matchings, and some unfavorable assignments cannot be avoided. As a result, the OD pairs encountered at evaluation may differ from those for which the transportation cost was learned accurately.
Here, we examine this effect. To do so, we use a grid world in which the start and target distributions are known and remain the same during training and evaluation, and keep track of the visitation frequency of each OD pair during training.
Setup. We use a 30 × 30 30 \times30 30 × 30 grid with horizon T = 100 T = 100 T = 100 , unit stage cost, individual action noise with probability p = 0.1 p = 0.1 p = 0.1 , and uniform start and target distributions over the full grid. We compare training with N = 20 N = 20 N = 20 and N = 5 N = 5 N = 5 and evaluate the performance across fleets of N ∈ { 1 , 5 , 10 , 20 , 50 } N \in\{1, 5, 10, 20, 50\} N ∈ { 1 , 5 , 10 , 20 , 50 } agents. Table 6 reports the training settings for the two population sizes.
Table 6. Training settings for the origin–destination coverage experiment.
Since an episode produces four times as many assignment observations when N = 20 N = 20 N = 20 as when N = 5 N = 5 N = 5 , the decay periods for N = 5 N = 5 N = 5 are four times longer. The schedules are therefore the same when measured in assignment observations.
At the beginning of each training episode, follow the assignment exploration rule of Algorithm 1, with ϵ m a t c h \epsilon_{\mathrm{match}} ϵ match corresponding to α k \alpha_k α k . With probability ϵ m a t c h \epsilon_{\mathrm{match}} ϵ match , we choose a uniformly random one-to-one agent–target matching, we solve the empirical optimal transport problem using the current Q-based cost-to-go estimate. To measure the coverage of the training data, we record how often each OD pair is selected:
n ( s , z ) = # { training episodes in which an agent starting at s is assigned to z } . n(s,z) = \#\{\text{training episodes in which an agent starting at }s\text{ is assigned to }z\}. n ( s , z ) = # { training episodes in which an agent starting at s is assigned to z } .
Although the learned matching concentrates on lower-cost pairs, assignment exploration mitigates this and gives nonzero support to at least 809,801 of the 810,000 possible OD pairs. The grid-world Q-table is initialized at zero.
For each trained Q-table, we evaluate fleets of N ∈ { 1 , 5 , 10 , 20 , 50 } N \in\{1, 5, 10, 20, 50\} N ∈ { 1 , 5 , 10 , 20 , 50 } agents for 5,000 episodes. During evaluation, we compute the initial optimal transport assignment from the learned Q-table and keep it fixed for the entire rollout, so to better relate training coverage to evaluation performance.
When classifying coverage, we exclude the trivial pairs for which s = z s = z s = z and group every remaining pair ( s , z ) (s, z) ( s , z ) according to the number of times it was selected during training: n ( s , z ) ≤ 5 n(s, z) \le5 n ( s , z ) ≤ 5 , 6 ≤ n ( s , z ) ≤ 20 6 \le n(s, z) \le20 6 ≤ n ( s , z ) ≤ 20 , or n ( s , z ) > 20 n(s, z) > 20 n ( s , z ) > 20 . In Figure 8 c–Figure 8 d, the horizontal coordinate is the fraction of nontrivial assignments in an episode that satisfy n ( s , z ) ≤ 20 n(s, z) \le20 n ( s , z ) ≤ 20 . We divide this fraction into the bins 0%, (0, 20%], (20, 40%], (40, 60%], (60, 80%], and (80, 100%] and a bin is shown only if each of the five training seeds contributes at least 20 evaluation episodes. In this case completion corresponds to the fraction of agents that reach their assigned target by the horizon.
Figure 8. OD-pair coverage and population-size transfer after training SALT-Q with N = 20 N = 20 N = 20 or N = 5 N = 5 N = 5 agents at the same nominal interaction budget. (a) Share of nontrivial evaluation assignments grouped by training count; within each evaluation fleet, the left and right bars correspond to training with N = 5 N = 5 N = 5 and N = 20 N = 20 N = 20 . (b) Overall per-agent completion; lines show pooled episode means and shaded regions the episode-level fifth–ninety-fifth percentiles. (c–d) Per-agent completion conditioned on the fraction of assignments satisfying n ( s , z ) ≤ 20 n(s, z) \le20 n ( s , z ) ≤ 20 , shown separately for the two training fleets. Points are means and error bars span the minimum and maximum over five training seeds. The x x x coordinate is the mean realized fraction within each bin.
Results. Figure 8 a reports the percentage of evaluation OD pairs in each of the three training-count ranges described above. For both training fleets, the shares satisfying n ( s , z ) ≤ 5 n(s, z) \le5 n ( s , z ) ≤ 5 and 6 ≤ n ( s , z ) ≤ 20 6 \le n(s, z) \le20 6 ≤ n ( s , z ) ≤ 20 decrease as the evaluation fleet grows. We can see that, with the same interaction budget, training with N = 5 N = 5 N = 5 agents gives broader coverage with respect to training with N = 20 N = 20 N = 20 agents. For example, for an evaluation fleet of size N = 1 N = 1 N = 1 , the share satisfying n ( s , z ) > 20 n(s, z) > 20 n ( s , z ) > 20 increases from 21.3% after training with N = 20 N = 20 N = 20 to 39.7% after training with N = 5 N = 5 N = 5 . At the same time, the share satisfying n ( s , z ) ≤ 5 n(s, z) \le5 n ( s , z ) ≤ 5 decreases from 11.1% to 0.9%.
fig:8b shows how this difference in coverage impacts the completion rate. Training with N = 20 N = 20 N = 20 leads to completion rates that increase from 59.6% for an evaluation fleet of size N = 1 N = 1 N = 1 to 96.9% for N = 50 N = 50 N = 50 . Training with N = 5 N = 5 N = 5 raises these values to 74.5% and 98.4%, respectively, and also improves completion at every intermediate fleet size. The shaded areas show the episode-level fifth–ninety-fifth percentiles after pooling the evaluation episodes from all five training seeds.
Figures 8c–d examine the relation more directly by plotting completion against the fraction of evaluation assignments satisfying n ( s , z ) ≤ 20 n(s, z) \le20 n ( s , z ) ≤ 20 . For each fleet size and under both training regimes, completion decreases as this fraction increases. For an evaluation fleet of size N = 1 N = 1 N = 1 , the assignment layer has no alternative target. In this case, in episodes involving an under-covered OD pair, the agent reaches its target in 48.8% of cases after training with N = 20 N = 20 N = 20 , compared with 59.6% after training with N = 5 N = 5 N = 5 . A similar difference is present for N = 5 N = 5 N = 5 : when 20% of the assignments are under-covered, completion increases from 90.7% to 92.7%, while when their mean fraction is approximately 60%, it increases from 79.0% to 79.7%. The minimum–maximum bars show that these trends are consistent across the five training seeds.
These results show that the OD-pair coverage obtained during training has an important impact on the transfer of SALT across fleet sizes. When the evaluation fleet is smaller than the training fleet, the assignment layer may select pairs that were underrepresented in the training data, leading to worse performance. At a fixed interaction budget, training with fewer agents improves coverage and reduces this mismatch. However, it requires proportionally more population episodes and therefore a longer training time. We believe that adapting the fleet size during training may help balance these two effects, and point it out as a promising direction for future work.
Towards a realistic, time-varying urban logistics task
Manhattan environment and dynamics. We model southern Manhattan as a directed road graph extracted from OpenStreetMap [39 ] , whose data are available under the Open Data Commons Open Database License (ODbL). We restrict the graph to 38 taxi zones in southern Manhattan, retain its largest strongly connected component, and remove self-loops. The resulting graph contains ∣ V ∣ = 2087 |V| = 2087 ∣ V ∣ = 2087 nodes and 4255 directed edges. Each node in the graph represents an intersection, while each directed edge represents an admissible road segment. The nominal edge travel times are computed from the road length and a road-type-dependent speed limit.
To model variable traffic conditions, we include traffic signals and stochastic congestion noise. We model edge-level traffic signals using a 90-second cycle whose green-phase duration depends on the road type. Primary roads are green for 50% of the cycle, whereas secondary and residential roads are green for 30%. The remaining road types (motorway, trunk, tertiary, and unclassified roads) are treated as always green. Finally, the signal offset of each intersection is randomly initialized. The impact of traffic signals is that an agent may have to wait at an intersection if it arrives during a red phase. More precisely, when an agent traverses edge e e e at time t t t , its realized transition time is the sum of the travel time and the waiting time determined by the signal phase.
To represent time-varying traffic conditions, we also add stochastic congestion noise. When edge e e e is traversed, its travel time is multiplied by
1 + u , u ∼ Unif ( 0 , η e ) , 1 + u,\qquad u \sim\operatorname{Unif}(0,\eta_e), 1 + u , u ∼ Unif ( 0 , η e ) ,
where η e = 0.1 \eta_e = 0.1 η e = 0.1 for primary roads, η e = 0.15 \eta_e = 0.15 η e = 0.15 for secondary roads, and η e = 0 \eta_e = 0 η e = 0 otherwise. A new value of u u u is drawn for every agent-edge traversal. We exclude operationally unrealistic immediate U-turns from the urban routing task, in both training and evaluation. After a vehicle traverses u → v u \to v u → v , the reverse action v → u v \to u v → u is masked whenever v v v has another admissible successor.
Time discretization and Q-table. The Manhattan simulator evolves in event time, thus, one decision step corresponds to traversing one outgoing road segment, rather than advancing time by a fixed one-second increment. The tabular Q-learner instead uses a discrete time coordinate with resolution Δ t = 1 \Delta t = 1 Δ t = 1 second. Since all traffic lights follow a 90-second cycle, we record in the time coordinate of the Q-table the traffic phase instead of unbounded physical time. Thus, the Q-table contains T d i s c = 90 / Δ t = 90 T_{\mathrm{disc}} = 90/\Delta t = 90 T disc = 90/Δ t = 90 time slices. At physical time t t t , the corresponding table index is
k ( t ) = clip ( ⌊ t / Δ t ⌋ , 0 , 89 ) . k(t) = \operatorname{clip}(\lfloor t/\Delta t \rfloor, 0, 89). k ( t ) = clip (⌊ t /Δ t ⌋ , 0 , 89 ) .
More precisely, we consider a target-conditioned Q-table, whose entries are indexed by the current node v v v , the request node z z z , the traffic-phase index k k k , and the outgoing action a a a : Q ( v , z , k , a ) Q(v,z,k,a) Q ( v , z , k , a ) . For the Manhattan graph, this gives a table of shape
∣ V ∣ × ∣ V ∣ × T d i s c × max v ∣ out ( v ) ∣ = 2087 × 2087 × 90 × 4. \lvert V\rvert\times\lvert V\rvert\times T_{\mathrm{disc}} \times\max_{v}\lvert\operatorname{out}(v)\rvert= 2087 \times2087 \times90 \times4. ∣ V ∣ × ∣ V ∣ × T disc × v max ∣ out ( v )∣ = 2087 × 2087 × 90 × 4.
where ∣ out ( v ) ∣ \lvert\operatorname{out}(v)\rvert ∣ out ( v )∣ is the out-degree of node v v v . Invalid padded actions are masked during both action selection and Bellman backups.
The physical edge travel times remain continuous, while the one-second grid is used only to index the Q-table. If T ‾ v , a \overline{T}_{v,a} T v , a is the nominal travel time of edge ( v , a ) (v,a) ( v , a ) and u u u is its congestion perturbation, the realized edge travel time is T ‾ v , a = T v , a ( 1 + u ) \overline{T}_{v,a} = T_{v,a}(1+u) T v , a = T v , a ( 1 + u ) . Once the edge has been traversed, the simulator computes the traffic-signal waiting time from the arrival phase. For an edge with period P v , a P_{v,a} P v , a , offset o v , a o_{v,a} o v , a , and green duration g v , a g_{v,a} g v , a , this phase is
ϕ ( t ) = ( t + T ‾ v , a + o v , a ) m o d P v , a , \phi(t) = (t + \overline{T}_{v,a} + o_{v,a}) \bmod P_{v,a}, ϕ ( t ) = ( t + T v , a + o v , a ) mod P v , a ,
and the corresponding waiting time is
W v , a ( t ) = { 0 , ϕ < g v , a , P v , a − ϕ , ϕ ≥ g v , a . W_{v,a}(t) =
\begin{cases}
0, & \phi< g_{v,a},\\
P_{v,a} - \phi, & \phi\ge g_{v,a}.
\end{cases} W v , a ( t ) = { 0 , P v , a − ϕ , ϕ < g v , a , ϕ ≥ g v , a .
The next physical time is therefore
t ′ = t + T ‾ v , a + W v , a ( t ) , t' = t + \overline{T}_{v,a} + W_{v,a}(t), t ′ = t + T v , a + W v , a ( t ) ,
and to obtain the next Q-table index, we round t ′ t' t ′ to the discrete grid and reduce it modulo the common traffic-signal period:
k ′ = round ( t ′ / Δ t ) m o d T d i s c . k' = \operatorname{round}(t'/\Delta t) \bmod T_{\mathrm{disc}}. k ′ = round ( t ′ /Δ t ) mod T disc .
The next lookup therefore uses the resulting phase index k ′ k' k ′ . The reward is based on the same realized delay T ‾ v , a + W v , a ( t ) \overline{T}_{v,a} + W_{v,a}(t) T v , a + W v , a ( t ) , expressed in minutes.
Task setup. We sample the starting positions from the three fixed depot nodes { 587 , 1226 , 1972 } \{587, 1226, 1972\} { 587 , 1226 , 1972 } , which have the highest betweenness centrality, while each request is represented by a destination node z z z sampled uniformly from the graph. During training, we assign the requests at the beginning of each population episode, by solving an epsilon-greedy optimal transport problem, and keep the assignment fixed for the entire training episode, as required to learn the target-conditioned subproblem. At evaluation, instead, the online-assignment SALT policy recomputes the optimal transport problem from the agents’ current nodes and traffic phases at every routing step, and then takes one greedy Q-learning action. Figure 4 compares this policy with a static nominal shortest-path baseline. The baseline computes a single distance-based optimal transport assignment at t = 0 t = 0 t = 0 and follows nominal shortest paths without accounting for the signal phase or congestion.
Learning algorithm. For the Manhattan experiment, we use tabular Q-learning with learning rate 0.1 and no discount factor. Action exploration is ε \varepsilon ε -greedy, with ε \varepsilon ε decreasing linearly from 1.0 to 0.01 over 90% of the total training steps. The training budget is 5,000,000 population episodes with 20 agents. The maximum shortest-path length in the processed graph is 66 edges, thus during both training and the reported final evaluation, we cap each trajectory at twice this value, or 132 road traversals. To provide a warm start to the learning algorithm, we initialize the Q-table using nominal shortest-path travel-time estimates, hence ignoring congestion noise or traffic-signal phases.
Reward. At each step, the stage reward of an active agent is the negative realized delay in minutes:
r t = − T ‾ v , a + W v , a ( t ) 60 . r_t = -\frac{\overline{T}_{v,a} + W_{v,a}(t)}{60}. r t = − 60 T v , a + W v , a ( t ) .
Inactive agents enter an absorbing state and receive zero reward thereafter. If the 132-traversal cap is reached before the target, the terminal Bellman target also accounts for the remaining distance: it includes − d S P ( v , z ) / 60 -d_{\mathrm{SP}}(v,z)/60 − d SP ( v , z ) /60 , the negative nominal shortest-path travel time from the current node v v v to the target z z z , rather than an arbitrary fixed timeout penalty.
Evaluation and logging. We report results for 8 training seeds (0–7). For each seed and each fleet size N ∈ { 1 , 5 , 10 , 20 , 100 } N \in\{1, 5, 10, 20, 100\} N ∈ { 1 , 5 , 10 , 20 , 100 } , we draw 500 evaluation instances, giving 4000 outcomes per fleet size. Within each evaluation run, SALT-Q and all shortest-path baselines share starts, requests, and the fixed traffic-signal configuration, while the separate rollouts use independent congestion-noise streams. We then report the continuous travel-time improvements over the baseline as 100 ( T b a s e l i n e − T S A L T − Q ) / T b a s e l i n e 100(T_{\mathrm{baseline}} - T_{\mathrm{SALT-Q}})/T_{\mathrm{baseline}} 100 ( T baseline − T SALT − Q ) / T baseline per instance. Figure 4 and Figure 9 show medians and interquartile ranges after pooling outcomes across training seeds. Here, whiskers follow the Tukey 1.5-interquartile-range rule and outliers remain in the underlying samples but are not represented in the plots.
Figure 9. Relative travel-time improvement of SALT-Q against nominal shortest-path routing with a static assignment (left), reassignment every five routing steps (center), or reassignment every ten routing steps (right). The evaluation samples and boxplot conventions match Figure 4 .
We evaluate the performance of SALT over three shortest-path baselines differing in the reassignment frequency they use. Static shortest path computes the optimal transport assignment once at the beginning of the episode and keep the assignment fixed until the end of the episode itself. K K K -reassignment shortest path instead recomputes the optimal transport assignment every K K K routing steps from the vehicles’ current nodes to the still-unserved requests, served requests instead remain locked to the vehicles that already reached them. Between reassignments, routing follows the same nominal shortest paths as the static baseline. In Figure 9 we show the performance of 5-reassignment and 10-reassignment shortest path. As visible, the reassignment improvement is minimal. To measure it directly, define G K ( N ) = 100 ( T ‾ s t a t i c − T ‾ r e a s s i g n , K ) / T ‾ s t a t i c G_K(N) = 100(\overline{T}_{\mathrm{static}}-\overline{T}_{\mathrm{reassign},K})/\overline{T}_{\mathrm{static}} G K ( N ) = 100 ( T static − T reassign , K ) / T static , where bars denote means over the available continuous-time outcomes. For N = ( 5 , 10 , 20 , 100 ) N = (5, 10, 20, 100) N = ( 5 , 10 , 20 , 100 ) , we obtain G 5 = ( − 0.0014 , 0.0008 , 0.0031 , 0.0086 ) % G_5 = (-0.0014, 0.0008, 0.0031, 0.0086)\% G 5 = ( − 0.0014 , 0.0008 , 0.0031 , 0.0086 ) % and G 10 = ( − 0.0480 , − 0.0243 , − 0.0406 , − 0.0045 ) % G_{10} = (-0.0480, -0.0243, -0.0406, -0.0045)\% G 10 = ( − 0.0480 , − 0.0243 , − 0.0406 , − 0.0045 ) % . The marginal differences are extremely small and can be traced back to the stochasticity of the environment. Therefore, we can conclude that for N = 1 N = 1 N = 1 reassignment cannot change the matching, and the small aggregate differences can only be due to different stochastic congestion realizations. Therefore, we can conclude that periodic target reassignment has essentially no practical improvement to nominal shortest-path routing in this experiment, and that the substantial advantage in Figure 4 is not an artifact of comparing online SALT assignment against a static assignment heuristic.
Compute resources
We ran all reported experiments on the Clariden supercomputer at the Swiss National Supercomputing Centre (CSCS). Each Clariden compute node contains four NVIDIA GH200 modules, each with a 72-core Grace CPU and 96 GB of HBM3 memory. An independent rank requested one GH200 and 72 CPU cores.
For the grid-world experiments, we used PyTorch [50 ] with CUDA [51 ] and assigned one independent rank to each module. Within a grid-world condition, the recorded training times summed across methods range from 3.9 to 4.9 hours, with a mean of 4.2 hours, and the 50 final rollouts per method were generated in the same jobs and required little additional time relative to training. The eight-condition scalability sweep used the same allocation and at most 96 additional active GPU-hours under the 12-hour job limit.
For the Manhattan experiments, we used JAX [52 ] with CUDA and the same one-GH200, 72-core allocation for each training seed. We ran four training seeds on each node. Training one policy for 5,000,000 episodes required approximately 77.5 active GPU-hours.