Introduction to Markov Decision Processes
Model, assumptions, and objective
Markov decision processes (MDPs) are used to model sequential decision-making problems. An MDP M \mathcal{M} M is a tuple M = ( S , A , r , P , p 0 ) \mathcal{M} = (\mathcal{S}, \mathcal{A}, r, P, p_0) M = ( S , A , r , P , p 0 ) where:
S \mathcal{S} S and A \mathcal{A} A are the finite sets of states and actions,
r ∈ R S × A × S r \in\mathbb{R}^{\mathcal{S} \times\mathcal{A} \times\mathcal{S}} r ∈ R S × A × S models the instantaneous rewards attached to each ( s , a , s ′ ) (s, a, s') ( s , a , s ′ ) with ( s , a ) ∈ S × A (s, a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A the current state-action pair and s ′ s' s ′ the next state,
P ∈ Δ ( S ) S × A P \in\Delta(\mathcal{S})^{\mathcal{S} \times\mathcal{A}} P ∈ Δ ( S ) S × A , where P ( s , a , s ′ ) P(s, a, s') P ( s , a , s ′ ) models the transition probabilities toward the next state s ′ s' s ′ given a current state-action pair ( s , a ) (s, a) ( s , a ) ,
p 0 ∈ Δ ( S ) p_0 \in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) is the initial distribution over the set of states: p 0 ( s ) p_0(s) p 0 ( s ) is the probability that we start in state s ∈ S s \in\mathcal{S} s ∈ S .
Recall that the simplex Δ ( E ) \Delta(\mathcal{E}) Δ ( E ) over a finite set E \mathcal{E} E is the set of all probability distributions over E \mathcal{E} E and can be identified with { x ∈ R E ∣ ∑ e ∈ E x ( e ) = 1 , x ( e ) ≥ 0 , ∀ e ∈ E } \{x \in\mathbb{R}^{\mathcal{E}} \mid\sum_{e \in\mathcal{E}} x(e) = 1, x(e) \ge0, \ \forall e \in\mathcal{E}\} { x ∈ R E ∣ ∑ e ∈ E x ( e ) = 1 , x ( e ) ≥ 0 , ∀ e ∈ E } . We usually represent the dynamics as in Figure 1.1 . It is worth emphasizing the assumptions listed above:
Figure 1.1. Dynamics for Markov decision processes. The next state is sampled with the probability distribution P ( s , a ) ∈ Δ ( S ) P(s,a) \in\Delta(\mathcal{S}) P ( s , a ) ∈ Δ ( S ) . We choose action a a a in state s s s to balance the instantaneous reward r ( s , a , s ′ ) r(s,a,s') r ( s , a , s ′ ) and the likelihood of obtaining good rewards in the future (from the next state s ′ s' s ′ ).
In these notes we focus on finite sets of states and actions, and discrete time periods. Already in this setting many interesting phenomena occur. We will later need compact action sets (see § 3.2 ).
The dynamics is assumed to be Markovian, in that a current state-action pair ( s t , a t ) (s_t,a_t) ( s t , a t ) at period t t t completely suffices to specify the distribution of the next state s t + 1 s_{t+1} s t + 1 as a random variable with values in S \mathcal{S} S .
For now, we assume that the same set of actions is available at each state. This is without loss of generality, and we do so to lighten notation for now, but we will need MDPs with state-dependent action sets when we study robust MDPs in § 3.2 .
We will tend to emphasize the role of the transition probabilities since this will play a key role for robust MDPs.
Policies. We now explain how we make decisions in an MDP. A policy π \pi π should choose an action at each period t ∈ N t \in\mathbb{N} t ∈ N , based on all the information observed so far. At period t ∈ N t \in\mathbb{N} t ∈ N , we can observe h t = ( s 0 , a 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t ) ∈ ( S × A ) t × S h_t=(s_0,a_0,s_1,a_1,\ldots,s_{t-1},a_{t-1},s_t) \in(\mathcal{S} \times\mathcal{A})^t \times\mathcal{S} h t = ( s 0 , a 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t ) ∈ ( S × A ) t × S , i.e., we can observe the sequence of all states visited and actions chosen, we know the current state, and we need to decide what action to pick next. Thus in all generality a policy is a map from the set of all finite histories H = ⋃ t ∈ N ( S × A ) t × S \mathcal{H}=\bigcup_{t\in\mathbb{N}}(\mathcal{S} \times\mathcal{A})^t \times\mathcal{S} H = ⋃ t ∈ N ( S × A ) t × S to the simplex Δ ( A ) \Delta(\mathcal{A}) Δ ( A ) over the action set A \mathcal{A} A .
We use the following notation for the sets of policies of interest:
Π H \Pi_{\mathrm{H}} Π H is the set of all policies, possibly history-dependent: Π H = { π : H → Δ ( A ) } \Pi_{\mathrm{H}}=\{\pi:\mathcal{H}\to\Delta(\mathcal{A})\} Π H = { π : H → Δ ( A )} .
Π M \Pi_{\mathrm{M}} Π M is the set of Markovian policies, i.e., policies where the action chosen at period t t t may only depend on the current state s t ∈ S s_t \in\mathcal{S} s t ∈ S and period t t t , but not on the state-action pairs ( s 0 , a 0 , … , s t − 1 , a t − 1 ) (s_0,a_0,\ldots,s_{t-1},a_{t-1}) ( s 0 , a 0 , … , s t − 1 , a t − 1 ) from the past.
Π S \Pi_{\mathrm{S}} Π S is the set of stationary policies, i.e., policies where the action chosen at period t t t may only depend on the current state s t ∈ S s_t \in\mathcal{S} s t ∈ S .
Π S D \Pi_{\mathrm{SD}} Π SD is the set of stationary deterministic policies, i.e., policies where the action chosen at period t t t may only depend on the current state s t ∈ S s_t \in\mathcal{S} s t ∈ S and the same action is always chosen at a given state.
Of course we have Π S D ⊂ Π S ⊂ Π M ⊂ Π H \Pi_{\mathrm{SD}} \subset\Pi_{\mathrm{S}} \subset\Pi_{\mathrm{M}} \subset\Pi_{\mathrm{H}} Π SD ⊂ Π S ⊂ Π M ⊂ Π H . Note that we can naturally identify the set Π M \Pi_{\mathrm{M}} Π M of Markovian policies with Δ ( A ) S × N \Delta(\mathcal{A})^{\mathcal{S}\times\mathbb{N}} Δ ( A ) S × N , i.e., with maps returning a randomized action in Δ ( A ) \Delta(\mathcal{A}) Δ ( A ) for each pair ( s , t ) (s,t) ( s , t ) of state-period in S × N \mathcal{S} \times\mathbb{N} S × N . We can identify the set Π S \Pi_{\mathrm{S}} Π S of stationary policies with the compact set Δ ( A ) S \Delta(\mathcal{A})^{\mathcal{S}} Δ ( A ) S (returning a randomized action for each state) and the set Π S D \Pi_{\mathrm{SD}} Π SD of stationary deterministic policies with the finite set A S \mathcal{A}^{\mathcal{S}} A S .
Note that history-dependent policies require (in the worst case) keeping track of everything that happened in the past and may be difficult to store and deploy. Similarly, randomized stationary policies mean that every time we visit the same state s s s , we need to sample from π ( s ) ∈ Δ ( A ) \pi(s) \in\Delta(\mathcal{A}) π ( s ) ∈ Δ ( A ) . This may be difficult to explain in some applications (e.g., in a healthcare application where the set of states is the set of health conditions for our patients and the set of actions is the set of feasible treatments), and in fact, randomization means delegating part of the decision to a generator of random numbers, which again may be problematic (how do you explain this to practitioners?). For all purposes, the set of stationary deterministic policies is the simplest, and its policies are the easiest to implement.
Example 1.1. Consider the MDP instance from Figure 1.2 . The set of states is S = { s 0 , G , B } \mathcal{S} = \{s_0, G, B\} S = { s 0 , G , B } and there are two actions: A = { a 1 , a 2 } \mathcal{A} = \{a_1, a_2\} A = { a 1 , a 2 } . The rewards are 0 0 0 starting from s 0 s_0 s 0 or B B B , and the reward from G G G is always 1 1 1 . The transition probabilities are parametrized by some p ∈ [ 0 , 1 ] p \in[0, 1] p ∈ [ 0 , 1 ] . Note that the only difference between a 1 a_1 a 1 and a 2 a_2 a 2 lies in the transitions out of s 0 s_0 s 0 . For instance, if p = 0 p = 0 p = 0 , choosing action a 1 a_1 a 1 in s 0 s_0 s 0 leads to B B B deterministically and choosing a 2 a_2 a 2 in s 0 s_0 s 0 leads to G G G deterministically. If p = 1 p = 1 p = 1 it is the reverse. The transition probabilities are given by P ( s 0 , a 1 ) = ( 0 , p , 1 − p ) P(s_0, a_1) = (0, p, 1 - p) P ( s 0 , a 1 ) = ( 0 , p , 1 − p ) , P ( s 0 , a 2 ) = ( 0 , 1 − p , p ) P(s_0, a_2) = (0, 1 - p, p) P ( s 0 , a 2 ) = ( 0 , 1 − p , p ) , P ( G , a 1 ) = P ( G , a 2 ) = P ( B , a 1 ) = P ( B , a 2 ) = ( 1 , 0 , 0 ) P(G, a_1) = P(G, a_2) = P(B, a_1) = P(B, a_2) = (1, 0, 0) P ( G , a 1 ) = P ( G , a 2 ) = P ( B , a 1 ) = P ( B , a 2 ) = ( 1 , 0 , 0 ) . The “good” state G G G yields a reward of 1 1 1 , while the “bad” state yields a reward of 0 0 0 . Obviously we want to pick our actions so that we pass through state G G G more often than state B B B .
Figure 1.2. MDP instance for Example 1.1. The transition probabilities and rewards are indicated beside each arc.
Example 1.2. Consider the MDP instance from Figure 1.3 , where there are five states { s 0 , s 1 , s 2 , s 3 , s 4 } \{s_0, s_1, s_2, s_3, s_4\} { s 0 , s 1 , s 2 , s 3 , s 4 } and two actions a 1 a_1 a 1 and a 2 a_2 a 2 . The transition probabilities are parametrized by p , p ′ ∈ [ 0 , 1 ] p, p' \in[0, 1] p , p ′ ∈ [ 0 , 1 ] . Absorbing states are represented with squares. The rewards are only obtained in the absorbing states s 2 , s 3 , s 4 s_2, s_3, s_4 s 2 , s 3 , s 4 . Actions a 1 a_1 a 1 and a 2 a_2 a 2 swap the probabilities towards the next states from s 0 s_0 s 0 and s 1 s_1 s 1 .
Figure 1.3. MDP instance for Example 1.2. The transition probabilities and rewards are indicated beside each arc.
Model (A): we assume that we visit independent copies of the state set at each period (so that each state is visited only once), or
Model (B): we assume that we can visit the same states multiple times. Both models are equivalent, but stationary policies in case (A) correspond to Markovian policies in case (B). In these notes we’ll focus on infinite-horizon models, and our results apply to finite-horizon models with model (A) (up to renormalizing the rewards at each state with the appropriate power of the discount factor).
Dynamics and objective function. Let’s say we choose some policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H and we “let the dynamics run”: with probability p 0 ( s 0 ) ∈ [ 0 , 1 ] p_0(s_0) \in[0,1] p 0 ( s 0 ) ∈ [ 0 , 1 ] we start in state s 0 s_0 s 0 , then we choose some action a 0 ∈ A a_0 \in\mathcal{A} a 0 ∈ A (possibly at random), transition to a next state s 1 s_1 s 1 , pick another action, and so forth: a t ∈ A a_t \in\mathcal{A} a t ∈ A is sampled from the distribution π ( s 0 , a 0 , … , s t − 1 , a t − 1 , s t ) ∈ Δ ( A ) \pi(s_0,a_0,\ldots,s_{t-1},a_{t-1},s_t) \in\Delta(\mathcal{A}) π ( s 0 , a 0 , … , s t − 1 , a t − 1 , s t ) ∈ Δ ( A ) , and s t + 1 ∈ S s_{t+1} \in\mathcal{S} s t + 1 ∈ S is sampled from P ( s t , a t ) ∈ Δ ( S ) P(s_t,a_t) \in\Delta(\mathcal{S}) P ( s t , a t ) ∈ Δ ( S ) . This results in an infinite trajectory of visited state-action pairs ( s t , a t ) t ∈ N (s_t,a_t)_{t\in\mathbb{N}} ( s t , a t ) t ∈ N , an element of ( S × A ) N (\mathcal{S} \times\mathcal{A})^{\mathbb{N}} ( S × A ) N , and given a policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H and transition probabilities P P P , some trajectories are more likely than others. This can be made rigorous with the Ionescu-Tulcea theorem, which states that the pair ( π , P ) (\pi,P) ( π , P ) induces a probability distribution over the set ( S × A ) N (\mathcal{S} \times\mathcal{A})^{\mathbb{N}} ( S × A ) N of infinite trajectories, compatible with the probabilities of any finite trajectories. We use the notation E π , P \mathbb{E}^{\pi,P} E π , P for the expectation with respect to this probability distribution. A trajectory ( s t , a t ) t ∈ N (s_t,a_t)_{t\in\mathbb{N}} ( s t , a t ) t ∈ N induces a stream of payoffs ( r ( s t , a t , s t + 1 ) ) t ∈ N (r(s_t,a_t,s_{t+1}))_{t\in\mathbb{N}} ( r ( s t , a t , s t + 1 ) ) t ∈ N , and given that ( π , P ) (\pi,P) ( π , P ) induces a distribution over the possible ( s t , a t ) t ∈ N (s_t,a_t)_{t\in\mathbb{N}} ( s t , a t ) t ∈ N then ( π , P ) (\pi,P) ( π , P ) induces a distribution over the possible streams of payoffs ( r ( s t , a t , s t + 1 ) ) t ∈ N (r(s_t,a_t,s_{t+1}))_{t\in\mathbb{N}} ( r ( s t , a t , s t + 1 ) ) t ∈ N . We are now left with designing a scalar objective function. In these notes we focus on the following discounted return : for π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H and P ∈ Δ ( S ) S × A P \in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ∈ Δ ( S ) S × A ,
R ( π , P ) = E π , P [ ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) ] (1) R(\pi,P) = \mathbb{E}^{\pi,P}\left[\sum_{t=0}^{+\infty}\gamma^t r(s_t,a_t,s_{t+1})\right] \tag*{(1)} R ( π , P ) = E π , P [ t = 0 ∑ + ∞ γ t r ( s t , a t , s t + 1 ) ] (1)
where γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) is the discount factor and the expectation is over the streams of payoffs associated with (random) trajectories ( s t , a t ) t ∈ N (s_t,a_t)_{t\in\mathbb{N}} ( s t , a t ) t ∈ N . The discount factor captures the diminishing importance of future rewards. In practice, this parameter governs the convergence rate and complexity of algorithms for discounted MDPs: most of them scale with ( 1 − γ ) − 1 (1-\gamma)^{-1} ( 1 − γ ) − 1 , so MDPs become increasingly hard to solve when γ \gamma γ is close to 1 and we want to be more farsighted. The main objective in an MDP is to solve the following maximization problem:
sup π ∈ Π H R ( π , P ) (2) \sup_{\pi\in\Pi_{\mathrm{H}}} R(\pi, P) \tag*{(2)} π ∈ Π H sup R ( π , P ) (2)
A few modeling remarks:
We use the convention that MDPs are maximizing rewards here. Since E [ X ] = − E [ − X ] \mathbb{E}[X] = -\mathbb{E}[-X] E [ X ] = − E [ − X ] we could have used the convention that MDPs minimize cost, and all results presented here remain the same. In particular we will use “minimizing MDPs” to model the policy evaluation problem in § 3.2 .
By linearity of the expectation, scaling by a positive factor and translating the rewards (i.e. r ← α r + β r \leftarrow\alpha r+\beta r ← α r + β with α > 0 \alpha> 0 α > 0 , β ∈ R \beta\in\mathbb{R} β ∈ R ) does not change the set of optimal solutions in (2) .
It is good to keep in mind that ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) \sum_{t=0}^{+\infty} \gamma^t r(s_t, a_t, s_{t+1}) ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) is a random variable, given that the trajectories ( s t , a t ) t ∈ N (s_t, a_t)_{t \in\mathbb{N}} ( s t , a t ) t ∈ N are themselves random.
The following definition will simplify the statements of our theorems.
Definition 1.1.
A policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H is optimal if it is an optimal solution of (2) .
A policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H is uniformly optimal if it is an optimal solution of (2) for all initial distributions p 0 ∈ Δ ( S ) p_0 \in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) at the same time.
When p 0 p_0 p 0 has full support, optimal policies are also uniformly optimal, but in general uniform optimality is a much stronger condition than optimality. While, in principle, we are interested in computing optimal policies (and at this point it may not even be obvious that uniformly optimal policies exist), we will see that most algorithms actually return uniformly optimal policies. For some ϵ > 0 \epsilon> 0 ϵ > 0 we will also use the notion of ϵ \epsilon ϵ -optimal policies, which are policies π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H such that R ( π , P ) ≥ sup π ′ ∈ Π H R ( π ′ , P ) − ϵ R(\pi, P) \ge\sup_{\pi' \in\Pi_{\mathrm{H}}} R(\pi', P) - \epsilon R ( π , P ) ≥ sup π ′ ∈ Π H R ( π ′ , P ) − ϵ , and uniformly ϵ \epsilon ϵ -optimal policies, which are ϵ \epsilon ϵ -optimal for all initial distributions.
The main difficulty in computing optimal policies lies in the multistage aspect of (2) . Indeed when visiting a state s s s we have to balance (a) achieving good instantaneous rewards by choosing actions yielding large values of r ( s , a , s ′ ) r(s, a, s') r ( s , a , s ′ ) , and (b) the probability to transition to a next state s ′ s' s ′ where we may be cornered and unable to derive good rewards from there on. This sort of tradeoff is best expressed in the framework of value functions.
Value functions. The objective function π ↦ R ( π , P ) \pi\mapsto R(\pi,P) π ↦ R ( π , P ) can be decomposed as follows by conditioning on the realization of the first state:
E π , P [ ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) ] = ∑ s ∈ S p 0 ( s ) E π , P [ ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) ∣ s 0 = s ] = ∑ s ∈ S p 0 ( s ) v π , P ( s ) = p 0 ⊤ v π , P \mathbb{E}^{\pi,P}\left[\sum_{t=0}^{+\infty}\gamma^t r(s_t,a_t,s_{t+1})\right]
= \sum_{s\in\mathcal{S}}p_0(s)\mathbb{E}^{\pi,P}\left[\left.\sum_{t=0}^{+\infty}\gamma^t r(s_t,a_t,s_{t+1})\right|s_0=s\right]
= \sum_{s\in\mathcal{S}}p_0(s)v^{\pi,P}(s)
= p_0^\top v^{\pi,P} E π , P [ t = 0 ∑ + ∞ γ t r ( s t , a t , s t + 1 ) ] = s ∈ S ∑ p 0 ( s ) E π , P [ t = 0 ∑ + ∞ γ t r ( s t , a t , s t + 1 ) s 0 = s ] = s ∈ S ∑ p 0 ( s ) v π , P ( s ) = p 0 ⊤ v π , P
with v π , P ∈ R S v^{\pi,P}\in\mathbb{R}^{S} v π , P ∈ R S the value function associated with a policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H and a transition kernel P P P :
v π , P ( s ) = E π , P [ ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) ∣ s 0 = s ] , ∀ s ∈ S . (3) v^{\pi,P}(s)=\mathbb{E}^{\pi,P}\left[\left.\sum_{t=0}^{+\infty}\gamma^t r(s_t,a_t,s_{t+1})\right|s_0=s\right],\qquad\forall\ s\in\mathcal{S}. \tag*{(3)} v π , P ( s ) = E π , P [ t = 0 ∑ + ∞ γ t r ( s t , a t , s t + 1 ) s 0 = s ] , ∀ s ∈ S . (3)
Note that in principle the trajectory laws depend on p 0 p_0 p 0 so that conditioning on s 0 = s s_0=s s 0 = s can be ill-defined if p 0 ( s ) = 0 p_0(s)=0 p 0 ( s ) = 0 ; in the expression above with the conditioning on s 0 = s s_0=s s 0 = s the expectation over the trajectory law is taken with initial state s s s , irrespective of the initial distribution. The scalar v π , P ( s ) v^{\pi,P}(s) v π , P ( s ) represents the “value” that the decision maker would obtain if starting in state s s s (with probability 1) and playing according to π \pi π along the trajectories. For stationary policies π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S this has a closed-form. Indeed, for π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S the pair ( π , P ) (\pi,P) ( π , P ) induces a Markov chain over the finite set S \mathcal{S} S , with transition matrix L π , P ∈ R S × S L^{\pi,P}\in\mathbb{R}^{S\times S} L π , P ∈ R S × S such that
L s s ′ π , P = ∑ a ∈ A π ( s , a ) P ( s , a , s ′ ) , ∀ s , s ′ ∈ S . L^{\pi,P}_{ss'}=\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a,s'),\qquad\forall\ s,s'\in\mathcal{S}. L s s ′ π , P = a ∈ A ∑ π ( s , a ) P ( s , a , s ′ ) , ∀ s , s ′ ∈ S .
Indeed if we are in state s s s , then with probability π ( s , a ) \pi(s,a) π ( s , a ) we pick action a a a and then with probability P ( s , a , s ′ ) P(s,a,s') P ( s , a , s ′ ) we transition to s ′ s' s ′ . Associated with this transition s → s ′ s\to s' s → s ′ is the function r π , P ∈ R S r^{\pi,P}\in\mathbb{R}^{S} r π , P ∈ R S defined as
r π , P ( s ) = ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ r ( s , a ) , (4) r^{\pi,P}(s)=\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a)^\top r(s,a), \tag*{(4)} r π , P ( s ) = a ∈ A ∑ π ( s , a ) P ( s , a ) ⊤ r ( s , a ) , (4)
which captures the expected reward that we obtain at (only) one period in each state, with r ( s , a ) = ( r ( s , a , s ′ ) ) s ′ ∈ S ∈ R S r(s,a)=(r(s,a,s'))_{s'\in\mathcal{S}}\in\mathbb{R}^{S} r ( s , a ) = ( r ( s , a , s ′ ) ) s ′ ∈ S ∈ R S . Finally, recall that the t t t -th power ( L π , P ) t (L^{\pi,P})^t ( L π , P ) t captures the distribution over the state at period t t t . With this notation, we get that
v π , P = ∑ t = 0 + ∞ γ t ( L π , P ) t r π , P = ( I − γ L π , P ) − 1 r π , P . (5) v^{\pi,P}=\sum_{t=0}^{+\infty}\gamma^t(L^{\pi,P})^t r^{\pi,P}=(I-\gamma L^{\pi,P})^{-1}r^{\pi,P}. \tag*{(5)} v π , P = t = 0 ∑ + ∞ γ t ( L π , P ) t r π , P = ( I − γ L π , P ) − 1 r π , P . (5)
The first equality is by definition of value functions as in (3) and interchanging the summation and the expectation (we can because of the discount factor). The second equality is by ( I − A ) − 1 = ∑ t = 0 + ∞ A t (I-A)^{-1}=\sum_{t=0}^{+\infty}A^t ( I − A ) − 1 = ∑ t = 0 + ∞ A t for A A A a matrix with spectral radius σ ( A ) \sigma(A) σ ( A ) strictly less than 1 (again, σ ( L π , P ) = 1 \sigma(L^{\pi,P})=1 σ ( L π , P ) = 1 because L π , P L^{\pi,P} L π , P is a transition matrix, and σ ( γ A ) = γ σ ( A ) \sigma(\gamma A)=\gamma\sigma(A) σ ( γ A ) = γ σ ( A ) for γ > 0 \gamma>0 γ > 0 ). This shows that we can compute v π , P v^{\pi,P} v π , P , at least for stationary policies, by inverting a matrix. This can become expensive for large state sets.
Note that uniform optimality can be interpreted in terms of value functions. In particular, the policy π P ∈ Π H \pi^{P} \in\Pi_H π P ∈ Π H is uniformly optimal if and only if
∀ p 0 ∈ Δ ( S ) , ∀ π ∈ Π H , R ( π , P ) ≤ R ( π P , P ) . \forall p_0 \in\Delta(\mathcal{S}),\ \forall\pi\in\Pi_H,\ R(\pi,P) \le R(\pi^{P},P). ∀ p 0 ∈ Δ ( S ) , ∀ π ∈ Π H , R ( π , P ) ≤ R ( π P , P ) .
Hence, choosing p 0 p_0 p 0 that puts probability 1 on a single state s s s , for each s ∈ S s \in\mathcal{S} s ∈ S , we get that π P \pi^{P} π P is uniformly optimal if and only if its value function dominates componentwise the value function of any other policy:
∀ π ∈ Π H , v π , P ≤ v π P , P . \forall\pi\in\Pi_H,\ v^{\pi,P} \le v^{\pi^{P},P}. ∀ π ∈ Π H , v π , P ≤ v π P , P .
In particular this implies that the value functions of all uniformly optimal policies must coincide. In some lecture notes (and in the literature on stochastic games) the optimality criteria are usually presented as finding a stationary uniformly optimal policy. In robust MDPs we will use optimality criteria based on R ( π , P ) R(\pi,P) R ( π , P ) rather than on the componentwise comparisons of value functions, mostly because we are interested in situations where the transition probabilities are not independent across states (so-called non-rectangular instances) so that componentwise comparison of (worst-case) value functions may be meaningless (or rather, they only provide lower bounds on the worst-case returns). We will be able to recover uniformly optimal policies for robust MDPs only under some specific assumptions (like rectangularity).
Example 1.1 continued. Consider the trajectories in the MDP instance from Figure 1.2 starting in s 0 s_0 s 0 . At even periods we are in state s 0 s_0 s 0 and at odd periods we are in state G G G or B B B ; we pick an action in { a 1 , a 2 } \{a_1,a_2\} { a 1 , a 2 } at each period. A stream of payoffs is therefore equal to a sequence of 0 and 1, where even elements are 0’s, and odd elements are 1 (if we visit G G G at this period) or 0 (if we visit B B B at this period). A stationary policy maps each element of { s 0 , G , B } \{s_0,G,B\} { s 0 , G , B } to a probability distribution over { a 1 , a 2 } \{a_1,a_2\} { a 1 , a 2 } . Note that the choice of actions is irrelevant at G , B G,B G , B (the rewards and transitions are the same for a 1 a_1 a 1 and a 2 a_2 a 2 ) so we can parametrize stationary policies by x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] modeling π ( s 0 , a 1 ) \pi(s_0,a_1) π ( s 0 , a 1 ) (we can then recover π ( s 0 , a 2 ) \pi(s_0,a_2) π ( s 0 , a 2 ) as 1 − π ( s 0 , a 1 ) 1-\pi(s_0,a_1) 1 − π ( s 0 , a 1 ) ). We can compute the value function of a stationary policy as follows: v ( G ) = 1 + γ v ( s 0 ) v(G)=1+\gamma v(s_0) v ( G ) = 1 + γ v ( s 0 ) , v ( B ) = 0 + γ v ( s 0 ) v(B)=0+\gamma v(s_0) v ( B ) = 0 + γ v ( s 0 ) , and
v ( s 0 ) = x ( p ⋅ γ v ( G ) + ( 1 − p ) ⋅ γ v ( B ) ) + ( 1 − x ) ( ( 1 − p ) ⋅ γ v ( G ) + p ⋅ γ v ( B ) ) . v(s_0)=x\bigl(p\cdot\gamma v(G)+(1-p)\cdot\gamma v(B)\bigr)+(1-x)\bigl((1-p)\cdot\gamma v(G)+p\cdot\gamma v(B)\bigr). v ( s 0 ) = x ( p ⋅ γ v ( G ) + ( 1 − p ) ⋅ γ v ( B ) ) + ( 1 − x ) ( ( 1 − p ) ⋅ γ v ( G ) + p ⋅ γ v ( B ) ) .
Replacing v ( G ) , v ( B ) v(G),v(B) v ( G ) , v ( B ) by their expressions in terms of v ( s 0 ) v(s_0) v ( s 0 ) in the equation above and rearranging the terms, we get
v ( s 0 ) = γ x p + ( 1 − p ) ( 1 − x ) 1 − γ 2 v(s_0)=\gamma\frac{xp+(1-p)(1-x)}{1-\gamma^2} v ( s 0 ) = γ 1 − γ 2 x p + ( 1 − p ) ( 1 − x )
and we observe that x p + ( 1 − p ) ( 1 − x ) xp+(1-p)(1-x) x p + ( 1 − p ) ( 1 − x ) is the probability to reach G G G from s 0 s_0 s 0 . Maximizing over x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] yields an optimal policy among stationary policies. Hence if we maximize over x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] we end up choosing x = 0 x=0 x = 0 if p ∈ [ 0 , 1 / 2 ) p \in[0,1/2) p ∈ [ 0 , 1/2 ) , x = 1 x=1 x = 1 if p ∈ ( 1 / 2 , 1 ] p \in(1/2,1] p ∈ ( 1/2 , 1 ] , and at p = 1 / 2 p=1/2 p = 1/2 we are indifferent and we can pick any x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] as an optimal solution.
Example 1.2 continued. To compute value functions one can use backward induction, that is, we first compute the value functions of the terminal states then propagate the value backward to their predecessors. Let π ∈ Π S \pi\in\Pi_S π ∈ Π S be a policy, parametrized by x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] the probability to play a 1 a_1 a 1 in s 0 s_0 s 0 and y ∈ [ 0 , 1 ] y \in[0,1] y ∈ [ 0 , 1 ] the probability to play a 1 a_1 a 1 in s 1 s_1 s 1 . We get v ( s 4 ) = 1 1 − γ v(s_4)=\frac{1}{1-\gamma} v ( s 4 ) = 1 − γ 1 , v ( s 3 ) = − 1 1 − γ v(s_3)=-\frac{1}{1-\gamma} v ( s 3 ) = − 1 − γ 1 , v ( s 2 ) = 1 2 1 1 − γ v(s_2)=\frac{1}{2}\frac{1}{1-\gamma} v ( s 2 ) = 2 1 1 − γ 1 . This yields
v ( s 1 ) = 0 + y γ ( p ′ v ( s 3 ) + ( 1 − p ′ ) v ( s 4 ) ) + ( 1 − y ) γ ( ( 1 − p ′ ) v ( s 3 ) + p v ( s 4 ) ) = γ 1 − γ ( 2 y − 1 ) ( 1 − 2 p ′ ) . v(s_1)=0+y\gamma\left(p'v(s_3)+(1-p')v(s_4)\right)+(1-y)\gamma\left((1-p')v(s_3)+pv(s_4)\right)=\frac{\gamma}{1-\gamma}(2y-1)(1-2p'). v ( s 1 ) = 0 + y γ ( p ′ v ( s 3 ) + ( 1 − p ′ ) v ( s 4 ) ) + ( 1 − y ) γ ( ( 1 − p ′ ) v ( s 3 ) + p v ( s 4 ) ) = 1 − γ γ ( 2 y − 1 ) ( 1 − 2 p ′ ) .
Now v ( s 0 ) = x γ ( p v ( s 1 ) + ( 1 − p ) v ( s 2 ) ) + ( 1 − x ) γ ( ( 1 − p ) v ( s 1 ) + p v ( s 2 ) ) v(s_0)=x\gamma\left(pv(s_1)+(1-p)v(s_2)\right)+(1-x)\gamma\left((1-p)v(s_1)+pv(s_2)\right) v ( s 0 ) = x γ ( p v ( s 1 ) + ( 1 − p ) v ( s 2 ) ) + ( 1 − x ) γ ( ( 1 − p ) v ( s 1 ) + p v ( s 2 ) ) . This yields the following closed-form expression for v ( s 0 ) v(s_0) v ( s 0 ) :
v ( s 0 ) = γ 1 − γ ( γ ( 1 − 2 p ′ ) ( 2 ( 2 p − 1 ) x y − ( 2 p − 1 ) x + 2 ( 1 − p ) y − ( 1 − p ) ) + 1 − 2 p 2 x + p 2 ) . (6) v(s_0)=\frac{\gamma}{1-\gamma}\left(\gamma(1-2p')\left(2(2p-1)xy-(2p-1)x+2(1-p)y-(1-p)\right)+\frac{1-2p}{2}x+\frac{p}{2}\right). \tag*{(6)} v ( s 0 ) = 1 − γ γ ( γ ( 1 − 2 p ′ ) ( 2 ( 2 p − 1 ) x y − ( 2 p − 1 ) x + 2 ( 1 − p ) y − ( 1 − p ) ) + 2 1 − 2 p x + 2 p ) . (6)
A very important fact about the function ( π , P ) ↦ R ( π , P ) (\pi,P)\mapsto R(\pi,P) ( π , P ) ↦ R ( π , P ) is that it is somewhat “ill-behaved” from a classical optimization standpoint. In particular, the function π ↦ R ( π , P ) \pi\mapsto R(\pi,P) π ↦ R ( π , P ) need not be concave (we want to maximize it) and the function P ↦ R ( π , P ) P\mapsto R(\pi,P) P ↦ R ( π , P ) need not be convex (we will want to minimize later on). This is exemplified by the computations of the value functions from Example 1.2, where we see that starting in s 0 s_0 s 0 yields a value function v ( s 0 ) v(s_0) v ( s 0 ) that depends on some products of the probabilities chosen by the policy (terms in x ⋅ y x\cdot y x ⋅ y in Equation (6) ) and some products of the transition probabilities (terms in p ⋅ p ′ p\cdot p' p ⋅ p ′ above). In fact if we had more periods the return R ( π , P ) R(\pi,P) R ( π , P ) would involve higher-order products of the entries of π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S . Thus in general maximizing π ↦ R ( π , P ) \pi\mapsto R(\pi,P) π ↦ R ( π , P ) over stationary policies does not fall in the class of “usual easy problems” that we are used to in convex optimization (let alone optimizing over the entire class of history-dependent policies). The next section shows how to solve MDPs.
State-action frequencies. The previous reformulation relating returns and value functions is the cornerstone of the analysis of robust MDPs. We derive here another very useful equality that relates the return and the state-action frequencies (also called occupancy frequencies). Let π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H . Then
R ( π , P ) = E π , P [ ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) ] = ∑ t = 0 + ∞ γ t E π , P [ r ( s t , a t , s t + 1 ) ] = ∑ t = 0 + ∞ γ t ∑ s , a P ( s , a ) T r ( s , a ) ⋅ P π , P ( ( s t , a t ) = ( s , a ) ) = ∑ s , a P ( s , a ) T r ( s , a ) ∑ t = 0 + ∞ γ t P π , P ( ( s t , a t ) = ( s , a ) ) = ∑ s , a μ π ( s , a ) P ( s , a ) T r ( s , a ) \begin{aligned}
R(\pi,P)&=\mathbb{E}^{\pi,P}\left[\sum_{t=0}^{+\infty}\gamma^t r(s_t,a_t,s_{t+1})\right]\\
&=\sum_{t=0}^{+\infty}\gamma^t\mathbb{E}^{\pi,P}\left[r(s_t,a_t,s_{t+1})\right]\\
&=\sum_{t=0}^{+\infty}\gamma^t\sum_{s,a}P(s,a)^{\mathsf{T}}r(s,a)\cdot\mathbb{P}^{\pi,P}\left((s_t,a_t)=(s,a)\right)\\
&=\sum_{s,a}P(s,a)^{\mathsf{T}}r(s,a)\sum_{t=0}^{+\infty}\gamma^t\mathbb{P}^{\pi,P}\left((s_t,a_t)=(s,a)\right)\\
&=\sum_{s,a}\mu^\pi(s,a)P(s,a)^{\mathsf{T}}r(s,a)
\end{aligned} R ( π , P ) = E π , P [ t = 0 ∑ + ∞ γ t r ( s t , a t , s t + 1 ) ] = t = 0 ∑ + ∞ γ t E π , P [ r ( s t , a t , s t + 1 ) ] = t = 0 ∑ + ∞ γ t s , a ∑ P ( s , a ) T r ( s , a ) ⋅ P π , P ( ( s t , a t ) = ( s , a ) ) = s , a ∑ P ( s , a ) T r ( s , a ) t = 0 ∑ + ∞ γ t P π , P ( ( s t , a t ) = ( s , a ) ) = s , a ∑ μ π ( s , a ) P ( s , a ) T r ( s , a )
where the first equality is by definition, the second equality is to be justified in § 1.6 , and the other equalities are from reformulating and rearranging the terms (note P π , P \mathbb{P}^{\pi,P} P π , P is the probability with respect to the distribution over the infinite trajectories induced by the pair ( π , P ) (\pi, P) ( π , P ) , and where we define μ π ∈ R + S × A \mu^\pi\in\mathbb{R}^{S \times A}_{+} μ π ∈ R + S × A as, for any pair ( s , a ) ∈ S × A (s, a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A ,
μ π ( s , a ) = ∑ t = 0 + ∞ γ t P π , P ( ( s t , a t ) = ( s , a ) ) (7) \mu^\pi(s, a) = \sum_{t=0}^{+\infty} \gamma^t \mathbb{P}^{\pi,P}((s_t, a_t) = (s, a)) \tag*{(7)} μ π ( s , a ) = t = 0 ∑ + ∞ γ t P π , P (( s t , a t ) = ( s , a )) (7)
The scalar μ π ( s , a ) \mu^\pi(s, a) μ π ( s , a ) is the discounted sum of probabilities of visiting the pair ( s , a ) (s, a) ( s , a ) across the entire infinite horizon of play, and μ π \mu^\pi μ π is referred to as the state-action frequencies . Overall we obtained that for π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H we have
R ( π , P ) = ∑ s ∈ S ∑ a ∈ A μ π ( s , a ) P ( s , a ) ⊤ r ( s , a ) (8) R(\pi, P) = \sum_{s \in\mathcal{S}} \sum_{a \in\mathcal{A}} \mu^\pi(s, a) P(s, a)^{\top} r(s, a) \tag*{(8)} R ( π , P ) = s ∈ S ∑ a ∈ A ∑ μ π ( s , a ) P ( s , a ) ⊤ r ( s , a ) (8)
and in fact the above expression is also linear in the rewards (which will play a role in Section 4.1 for the case of uncertain rewards). Note that (1.8) has a natural interpretation: the state-action frequencies μ π ( s , a ) \mu^\pi(s, a) μ π ( s , a ) count in a discounted manner how much time we spend at a state-action pair, and P ( s , a ) ⊤ r ( s , a ) P(s, a)^{\top}r(s, a) P ( s , a ) ⊤ r ( s , a ) is the expected instantaneous reward we get from ( s , a ) (s, a) ( s , a ) . We can also derive (1.8) for stationary policies by noting that R ( π , P ) = p 0 ⊤ v π , P R(\pi, P) = p_0^{\top}v^{\pi,P} R ( π , P ) = p 0 ⊤ v π , P , using (1.5) to get R ( π , P ) = p 0 ⊤ ( I − γ L π , P ) − 1 r π , P = r π , P ⊤ ( I − γ L π , P ⊤ ) − 1 p 0 R(\pi, P) = p_0^{\top}(I - \gamma L^{\pi,P})^{-1}r^{\pi,P} = r^{\pi,P\top}(I - \gamma L^{\pi,P\top})^{-1}p_0 R ( π , P ) = p 0 ⊤ ( I − γ L π , P ) − 1 r π , P = r π , P ⊤ ( I − γ L π , P ⊤ ) − 1 p 0 and then reformulating this last expression by isolating the factor in front of P ( s , a ) ⊤ r ( s , a ) P(s, a)^{\top}r(s, a) P ( s , a ) ⊤ r ( s , a ) as μ π ( s , a ) = π ( s , a ) ( ( I − γ L π , P ⊤ ) − 1 p 0 ) ( s ) \mu^\pi(s, a) = \pi(s, a)\left((I - \gamma L^{\pi,P\top})^{-1}p_0\right)(s) μ π ( s , a ) = π ( s , a ) ( ( I − γ L π , P ⊤ ) − 1 p 0 ) ( s ) for each ( s , a ) (s, a) ( s , a ) . The matrix-vector product ( I − γ L π , P ⊤ ) − 1 p 0 (I - \gamma L^{\pi,P\top})^{-1}p_0 ( I − γ L π , P ⊤ ) − 1 p 0 is precisely the term giving the next flow constraints (1.9). Note that for s ′ ∈ S s' \in\mathcal{S} s ′ ∈ S we have
∑ a ∈ A μ π ( s ′ , a ) = ∑ t = 0 + ∞ γ t P π , P ( ( s t ) = s ′ ) = p 0 ( s ′ ) + ∑ t = 1 + ∞ γ t P π , P ( ( s t ) = s ′ ) = p 0 ( s ′ ) + γ ∑ t = 0 + ∞ γ t P π , P ( ( s t + 1 ) = s ′ ) = p 0 ( s ′ ) + γ ∑ s , a P ( s , a , s ′ ) ∑ t = 0 + ∞ γ t P π , P ( ( s t , a t ) = ( s , a ) ) = p 0 ( s ′ ) + ∑ s ∈ S ∑ a ∈ A γ P ( s , a , s ′ ) μ π ( s , a ) \begin{aligned}
\sum_{a \in\mathcal{A}} \mu^\pi(s', a)
&= \sum_{t=0}^{+\infty} \gamma^t \mathbb{P}^{\pi,P}((s_t) = s') \\
&= p_0(s') + \sum_{t=1}^{+\infty} \gamma^t \mathbb{P}^{\pi,P}((s_t) = s') \\
&= p_0(s') + \gamma\sum_{t=0}^{+\infty} \gamma^t \mathbb{P}^{\pi,P}((s_{t+1}) = s') \\
&= p_0(s') + \gamma\sum_{s,a} P(s,a,s') \sum_{t=0}^{+\infty} \gamma^t \mathbb{P}^{\pi,P}((s_t,a_t) = (s,a)) \\
&= p_0(s') + \sum_{s \in\mathcal{S}} \sum_{a \in\mathcal{A}} \gamma P(s,a,s')\mu^\pi(s,a)
\end{aligned} a ∈ A ∑ μ π ( s ′ , a ) = t = 0 ∑ + ∞ γ t P π , P (( s t ) = s ′ ) = p 0 ( s ′ ) + t = 1 ∑ + ∞ γ t P π , P (( s t ) = s ′ ) = p 0 ( s ′ ) + γ t = 0 ∑ + ∞ γ t P π , P (( s t + 1 ) = s ′ ) = p 0 ( s ′ ) + γ s , a ∑ P ( s , a , s ′ ) t = 0 ∑ + ∞ γ t P π , P (( s t , a t ) = ( s , a )) = p 0 ( s ′ ) + s ∈ S ∑ a ∈ A ∑ γ P ( s , a , s ′ ) μ π ( s , a )
i.e., we have shown that the following natural flow constraints hold:
p 0 ( s ′ ) + ∑ s ∈ S ∑ a ∈ A γ P ( s , a , s ′ ) μ π ( s , a ) = ∑ a ∈ A μ π ( s ′ , a ) , ∀ s ′ ∈ S (9) p_0(s') + \sum_{s \in\mathcal{S}} \sum_{a \in\mathcal{A}} \gamma P(s,a,s')\mu^\pi(s,a) = \sum_{a \in\mathcal{A}} \mu^\pi(s',a), \qquad\forall\ s' \in\mathcal{S} \tag*{(9)} p 0 ( s ′ ) + s ∈ S ∑ a ∈ A ∑ γ P ( s , a , s ′ ) μ π ( s , a ) = a ∈ A ∑ μ π ( s ′ , a ) , ∀ s ′ ∈ S (9)
where the right-hand side is the (discounted) amount of time spent in state s ′ s' s ′ and the left-hand side is the incoming flow at s ′ s' s ′ . Therefore the set Q : = { μ π ∈ R + S × A ∣ π ∈ Π H } \mathcal{Q} := \{\mu^\pi\in\mathbb{R}^{S \times A}_{+} \mid\pi\in\Pi_{\mathrm{H}}\} Q := { μ π ∈ R + S × A ∣ π ∈ Π H } can be written as the following polytope:
Q = { μ ∈ R S × A ∣ μ ≥ 0 , p 0 ( s ′ ) + ∑ s ∈ S ∑ a ∈ A γ P ( s , a , s ′ ) μ ( s , a ) = ∑ a ∈ A μ ( s ′ , a ) , ∀ s ′ ∈ S } (10) \mathcal{Q} = \left\{ \mu\in\mathbb{R}^{S \times A} \mid\mu\ge0,\ p_0(s') + \sum_{s \in\mathcal{S}} \sum_{a \in\mathcal{A}} \gamma P(s,a,s')\mu(s,a) = \sum_{a \in\mathcal{A}} \mu(s',a),\quad\forall\ s' \in\mathcal{S} \right\} \tag*{(10)} Q = { μ ∈ R S × A ∣ μ ≥ 0 , p 0 ( s ′ ) + s ∈ S ∑ a ∈ A ∑ γ P ( s , a , s ′ ) μ ( s , a ) = a ∈ A ∑ μ ( s ′ , a ) , ∀ s ′ ∈ S } (10)
and given μ ∈ Q \mu\in Q μ ∈ Q we can recover a feasible stationary policy π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S as
π ( s , a ) = μ ( s , a ) ∑ b ∈ A μ ( s , b ) \pi(s,a) = \frac{\mu(s,a)}{\sum_{b\in\mathcal{A}}\mu(s,b)} π ( s , a ) = ∑ b ∈ A μ ( s , b ) μ ( s , a )
if ∑ b ∈ A μ ( s , b ) ≠ 0 \sum_{b\in\mathcal{A}}\mu(s,b) \ne0 ∑ b ∈ A μ ( s , b ) = 0 , and π ( s , a ) = 1 / ∣ A ∣ \pi(s,a) = 1/\lvert\mathcal{A}\rvert π ( s , a ) = 1/ ∣ A ∣ otherwise (we won’t visit state s s s anyway so we can play anything there). Conversely, given π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S , (1.9) implies that we can compute μ π \mu^\pi μ π by solving a linear system of equations, which can be done efficiently. In turn, the extreme points of Q Q Q correspond to the state-action frequencies induced by stationary deterministic policies.
Overall we have seen three crucial properties of state-action frequencies:
we can map policies π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H to elements μ π \mu^\pi μ π of the polytope Q Q Q ,
we can map each element μ ∈ Q \mu\in Q μ ∈ Q to a stationary policy π \pi π ,
the return R ( π , P ) R(\pi,P) R ( π , P ) of a policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H is a linear function of the state-action frequency μ π \mu^\pi μ π .
In fact the three points above already suggest that stationary policies will be sufficient for optimality in MDPs, as we will prove next. The state-action frequencies are powerful tools in the MDP toolkit and have a natural linear programming interpretation (see § 1.3.1 ). The state-action frequencies will not play a role for robust MDPs with uncertain transitions. This is because the polytope Q Q Q actually depends on P P P for the expressions of the flow constraints (in a “non-rectangular manner”: every s ′ s' s ′ -th constraint depends on P ( s , a , s ′ ) P(s,a,s') P ( s , a , s ′ ) across all ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A ). Additionally Q Q Q depends on p 0 p_0 p 0 so uniform optimality is harder to prove based on frequencies than based on value functions (which are defined component-wise and independently of p 0 p_0 p 0 ). Still the polytope Q Q Q will play a crucial role for robust MDPs with uncertain rewards since the return is linear in the rewards (see § 4.1 ).
Main results for MDPs
Operators and optimal policies.
The first crucial result relates the value function of a stationary policy to the fixed point of an operator. For π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S , we write T π , P : R S → R S T^{\pi,P} : \mathbb{R}^{\mathcal{S}} \to\mathbb{R}^{\mathcal{S}} T π , P : R S → R S for the operator defined as
T π , P v ( s ) = ∑ a ∈ A π ( s , a ) P ( s , a ) T ( r ( s , a ) + γ v ) , ∀ v ∈ R S , s ∈ S . (11) T^{\pi,P}v(s) = \sum_{a\in\mathcal{A}} \pi(s,a)P(s,a)^{\mathsf{T}}\left(r(s,a) + \gamma v\right), \quad\forall v \in\mathbb{R}^{\mathcal{S}},\ s \in\mathcal{S}. \tag*{(11)} T π , P v ( s ) = a ∈ A ∑ π ( s , a ) P ( s , a ) T ( r ( s , a ) + γ v ) , ∀ v ∈ R S , s ∈ S . (11)
We can interpret T π , P v ( s ) T^{\pi,P}v(s) T π , P v ( s ) as the expected return that one would obtain if starting in state s ∈ S s \in\mathcal{S} s ∈ S , playing according to π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S for the first period, transitioning to a next state s ′ s' s ′ with distribution P ( s , a ) ∈ Δ ( S ) P(s,a) \in\Delta(\mathcal{S}) P ( s , a ) ∈ Δ ( S ) , and quitting the dynamics with the termination value v ( s ′ ) ∈ R v(s') \in\mathbb{R} v ( s ′ ) ∈ R , i.e., we have
T π , P v ( s ) = E a ∼ π ( s ) , s ′ ∼ P ( s , a ) [ r ( s , a , s ′ ) + γ v ( s ′ ) ] . T^{\pi,P}v(s) = \mathbb{E}_{a\sim\pi(s),s'\sim P(s,a)}\left[r(s,a,s') + \gamma v(s')\right]. T π , P v ( s ) = E a ∼ π ( s ) , s ′ ∼ P ( s , a ) [ r ( s , a , s ′ ) + γ v ( s ′ ) ] .
Remark that the s s s -th component of T π , P v T^{\pi,P}v T π , P v only depends on the s s s -th component of π \pi π (i.e., how we choose actions at state s s s ) and on the s s s -th component of P P P (i.e., the transition probabilities out of state s s s ), and not on the other components of π \pi π and P P P related to other states (this will play a crucial role in the later chapters).
From there we get that if we define the sequence v t + 1 : = T π , P v t v^{t+1} := T^{\pi,P}v^t v t + 1 := T π , P v t , v 0 = 0 ∈ R S v^0 = 0 \in\mathbb{R}^{S} v 0 = 0 ∈ R S then v t v^t v t is the total discounted return obtained up to period t − 1 t-1 t − 1 (and terminating with a reward of 0 in the last state):
v t ( s ) = E π , P [ ∑ t ′ = 0 t − 1 γ t ′ r ( s t ′ , a t ′ , s t ′ + 1 ) ∣ s 0 = s ] , ∀ s ∈ S . v^t(s) = \mathbb{E}^{\pi,P}\left[\left.\sum_{t'=0}^{t-1}\gamma^{t'}r(s_{t'},a_{t'},s_{t'+1})\right|s_0=s\right], \quad\forall s \in\mathcal{S}. v t ( s ) = E π , P [ t ′ = 0 ∑ t − 1 γ t ′ r ( s t ′ , a t ′ , s t ′ + 1 ) s 0 = s ] , ∀ s ∈ S .
This suggests that we can interpret v π , P v^{\pi,P} v π , P as the limit of a sequence ( T π , P ) t 0 S (T^{\pi,P})^t0_S ( T π , P ) t 0 S for t → + ∞ t \to+\infty t → + ∞ (with 0 S = ( 0 , … , 0 ) ∈ R S 0_S=(0,\ldots,0)\in\mathbb{R}^{S} 0 S = ( 0 , … , 0 ) ∈ R S ). In fact, if we consider a sequence of policies π 0 , … , π t ∈ Π S \pi^0,\ldots,\pi^t\in\Pi_S π 0 , … , π t ∈ Π S then T π 0 , P ∘ ⋯ ∘ T π t , P 0 S T^{\pi^0,P}\circ\cdots\circ T^{\pi^t,P}0_S T π 0 , P ∘ ⋯ ∘ T π t , P 0 S is the reward obtained after t + 1 t+1 t + 1 periods (recall period indices start at 0) by playing according to policy π 0 \pi^0 π 0 at period 0, then according to policy π 1 \pi^1 π 1 at period 1, etc.
To make this rigorous, we note that T π , P T^{\pi,P} T π , P satisfies the following two key properties, defined for general operators F : R S → R S F:\mathbb{R}^{S}\to\mathbb{R}^{S} F : R S → R S .
Definition 1.2. Let F : R S → R S F:\mathbb{R}^{S}\to\mathbb{R}^{S} F : R S → R S be an operator.
u ≤ v ⇒ F u ≤ F v , ∀ u , v ∈ R S . (12) u \le v \Rightarrow Fu \le Fv,\ \forall u,v\in\mathbb{R}^{S}. \tag*{(12)} u ≤ v ⇒ F u ≤ F v , ∀ u , v ∈ R S . (12)
∃ λ ∈ [ 0 , 1 ) , ∀ u , v ∈ R S , ∥ F u − F v ∥ ∞ ≤ λ ∥ u − v ∥ ∞ . (13) \exists\lambda\in[0,1),\ \forall u,v\in\mathbb{R}^{S},\ \|Fu-Fv\|_{\infty}\le\lambda\|u-v\|_{\infty}. \tag*{(13)} ∃ λ ∈ [ 0 , 1 ) , ∀ u , v ∈ R S , ∥ F u − F v ∥ ∞ ≤ λ ∥ u − v ∥ ∞ . (13)
As highlighted from the calculations of the value functions for our simple examples above, we can compute v π , P v^{\pi,P} v π , P as the fixed point of a monotone contraction. This yields the next proposition.
Proposition 1.1. Let π ∈ Π S \pi\in\Pi_S π ∈ Π S .
The operator T π , P T^{\pi,P} T π , P defined in (11) is a monotone contraction .
The value function v π , P ∈ R S v^{\pi,P}\in\mathbb{R}^{S} v π , P ∈ R S is the unique fixed point of the operator T π , P T^{\pi,P} T π , P .
Proof. 1. Monotonicity follows from P ( s , a ) ∈ R + S P(s,a)\in\mathbb{R}^{S}_{+} P ( s , a ) ∈ R + S for any ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A :
u ≤ v ⇒ P ( s , a ) ⊤ u ≤ P ( s , a ) ⊤ v ⇒ T π , P u ≤ T π , P v . u\le v\Rightarrow P(s,a)^\top u\le P(s,a)^\top v\Rightarrow T^{\pi,P}u\le T^{\pi,P}v. u ≤ v ⇒ P ( s , a ) ⊤ u ≤ P ( s , a ) ⊤ v ⇒ T π , P u ≤ T π , P v .
The contraction property follows from P ( s , a ) ∈ R + S P(s,a)\in\mathbb{R}^{S}_{+} P ( s , a ) ∈ R + S and ∑ s ′ ∈ S P ( s , a , s ′ ) = 1 \sum_{s'\in\mathcal{S}}P(s,a,s')=1 ∑ s ′ ∈ S P ( s , a , s ′ ) = 1 , i.e., from P ( s , a ) ∈ Δ ( S ) P(s,a)\in\Delta(\mathcal{S}) P ( s , a ) ∈ Δ ( S ) . To see this, note that
T π , P u = T π , P ( v + ( u − v ) ) ≤ T π , P v + γ ∥ u − v ∥ ∞ 1 S T^{\pi,P}u=T^{\pi,P}(v+(u-v))\le T^{\pi,P}v+\gamma\|u-v\|_{\infty}1_S T π , P u = T π , P ( v + ( u − v )) ≤ T π , P v + γ ∥ u − v ∥ ∞ 1 S
where the inequality is from the componentwise inequality u − v ≤ ∥ u − v ∥ ∞ 1 S u-v\le\|u-v\|_{\infty}1_S u − v ≤ ∥ u − v ∥ ∞ 1 S . Rearranging the terms we obtain T π , P u − T π , P v ≤ γ ∥ u − v ∥ ∞ 1 S T^{\pi,P}u-T^{\pi,P}v\le\gamma\|u-v\|_{\infty}1_S T π , P u − T π , P v ≤ γ ∥ u − v ∥ ∞ 1 S . Reversing the roles of u u u and v v v yields the contraction property for T π , P T^{\pi,P} T π , P .
We have v π , P = ( I − γ L π , P ) − 1 r π , P ⟺ ( I − γ L π , P ) v π , P = r π , P ⟺ v π , P = r π , P + γ L π , P v π , P ⟺ v π , P = T π , P v π , P v^{\pi,P}=(I-\gamma L^{\pi,P})^{-1}r^{\pi,P}\Longleftrightarrow(I-\gamma L^{\pi,P})v^{\pi,P}=r^{\pi,P}\Longleftrightarrow v^{\pi,P}=r^{\pi,P}+\gamma L^{\pi,P}v^{\pi,P}\Longleftrightarrow v^{\pi,P}=T^{\pi,P}v^{\pi,P} v π , P = ( I − γ L π , P ) − 1 r π , P ⟺ ( I − γ L π , P ) v π , P = r π , P ⟺ v π , P = r π , P + γ L π , P v π , P ⟺ v π , P = T π , P v π , P where the first equivalence is by multiplying on both sides by ( I − γ L π , P ) (I-\gamma L^{\pi,P}) ( I − γ L π , P ) , the second equivalence is by reordering the terms on both sides, and the last equivalence uses the expression of T π , P T^{\pi,P} T π , P .
The second important result for MDPs pertains to the existence of stationary deterministic optimal policies. We will use the following Bellman optimality operator T P : R S → R S T^P:\mathbb{R}^{S}\to\mathbb{R}^{S} T P : R S → R S defined as
T P v ( s ) = max π ∈ Π S T π , P v ( s ) (14) T^P v(s)=\max_{\pi\in\Pi_{\mathcal S}}T^{\pi,P}v(s) \tag*{(14)} T P v ( s ) = π ∈ Π S max T π , P v ( s ) (14)
For s ∈ S s\in\mathcal{S} s ∈ S , note that the s s s -th component of T π , P v T^{\pi,P}v T π , P v only depends on π ( s ) ∈ Δ ( A ) \pi(s)\in\Delta(\mathcal{A}) π ( s ) ∈ Δ ( A ) and ( P ( s , a ) ) a ∈ A ∈ Δ ( S ) A (P(s,a))_{a\in\mathcal{A}}\in\Delta(\mathcal{S})^{\mathcal{A}} ( P ( s , a ) ) a ∈ A ∈ Δ ( S ) A , i.e., on the s s s -th component of P ∈ Δ ( S ) S × A P\in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ∈ Δ ( S ) S × A . Hence we actually have
T P v ( s ) = max π ( s ) ∈ Δ ( A ) ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) = max a ∈ A P ( s , a ) ⊤ ( r ( s , a ) + γ v ) \begin{aligned}
T^P v(s)&=\max_{\pi(s)\in\Delta(\mathcal{A})}\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a)^\top(r(s,a)+\gamma v)\\
&=\max_{a\in\mathcal{A}}P(s,a)^\top(r(s,a)+\gamma v)
\end{aligned} T P v ( s ) = π ( s ) ∈ Δ ( A ) max a ∈ A ∑ π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) = a ∈ A max P ( s , a ) ⊤ ( r ( s , a ) + γ v )
for each v ∈ R S v\in\mathbb{R}^{S} v ∈ R S and s ∈ S s\in\mathcal{S} s ∈ S , where the second equality uses the fact that the maximum of a linear form over a polytope is attained (among other solutions) by at least one extreme point, and we can identify the extreme points of Δ ( A ) \Delta(\mathcal{A}) Δ ( A ) with the deterministic choice of actions. In short this shows that
T P v ( s ) = max π ∈ Π S D T π , P v ( s ) . T^P v(s)=\max_{\pi\in\Pi_{\mathrm{SD}}}T^{\pi,P}v(s). T P v ( s ) = π ∈ Π SD max T π , P v ( s ) .
Since the maximum over a finite set is always attained, we get the following attainability property for the family { T π , P ∣ π ∈ Π S D } \{T^{\pi,P}\mid\pi\in\Pi_{\mathrm{SD}}\} { T π , P ∣ π ∈ Π SD } with respect to T P T^P T P : for any v ∈ R S v\in\mathbb{R}^{S} v ∈ R S , T P v T^P v T P v is actually T π , P v T^{\pi,P}v T π , P v for some stationary deterministic π ∈ Π S D \pi\in\Pi_{\mathrm{SD}} π ∈ Π SD , i.e.,
∀ v ∈ R S , ∃ π ∈ Π S D , T P v = T π , P v . (15) \forall v\in\mathbb{R}^{S},\ \exists\ \pi\in\Pi_{\mathrm{SD}},\ T^P v=T^{\pi,P}v. \tag*{(15)} ∀ v ∈ R S , ∃ π ∈ Π SD , T P v = T π , P v . (15)
As a side remark, note that we should not hope to be able to reverse the quantifiers in the above equations except in some specific cases. We can now state the “fundamental theorem of MDPs”.
Theorem 1.1. Let M = ( S , A , r , P , p 0 ) \mathcal{M}=(\mathcal{S},\mathcal{A},r,P,p_0) M = ( S , A , r , P , p 0 ) be an MDP with finite sets of states and actions and consider computing an optimal policy for the discounted return as in (1.2).
Bellman equation. The Bellman optimality operator defined in (1.14) is a monotone contraction with unique fixed point v P ∈ R S v^P\in\mathbb{R}^{S} v P ∈ R S , satisfying the following Bellman equations in the variable v ∈ R S v\in\mathbb{R}^{S} v ∈ R S :
v ( s ) = max a ∈ A P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ s ∈ S . (16) v(s)=\max_{a\in\mathcal{A}}P(s,a)^\top(r(s,a)+\gamma v),\qquad\forall s\in\mathcal{S}. \tag*{(16)} v ( s ) = a ∈ A max P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ s ∈ S . (16) Uniform optimality. A uniformly optimal policy π P \pi^P π P exists and can be chosen stationary and deterministic, and the value function v π P , P v^{\pi^P,P} v π P , P of a stationary uniformly optimal policy π P \pi^P π P coincides with v P ∈ R S v^P\in\mathbb{R}^{S} v P ∈ R S , the unique fixed point of T P T^P T P .
Optimality conditions
We can compute a stationary uniformly optimal policy by choosing any policy that jointly attains the arg max in T P v P ( s ) T^{P}v^{P}(s) T P v P ( s ) across all s ∈ S s \in\mathcal{S} s ∈ S , i.e., we have:
π P ∈ Π S is uniformly optimal ⟺ T P v P = T π P , P v P ⟺ the support of π P ( s ) ∈ Δ ( A ) is contained in arg max a ∈ A P ( s , a ) T ( r ( s , a ) + γ v P ) , ∀ s ∈ S ⟺ π P ∈ ⋂ s ∈ S arg max π ∈ Π S T π , P v P ( s ) \begin{aligned}
\pi^{P} \in\Pi_{\mathcal{S}}\text{ is uniformly optimal}
&\Longleftrightarrow T^{P}v^{P}=T^{\pi^{P},P}v^{P}\\
&\Longleftrightarrow\text{the support of }\pi^{P}(s)\in\Delta(\mathcal{A})\text{ is contained in}\\
&\qquad\underset{a\in\mathcal{A}}{\operatorname{arg\,max}}\ P(s,a)^{\mathsf{T}}\left(r(s,a)+\gamma v^{P}\right),\ \forall s\in\mathcal{S}\\
&\Longleftrightarrow\pi^{P}\in\bigcap_{s\in\mathcal{S}}\underset{\pi\in\Pi_{\mathcal{S}}}{\operatorname{arg\,max}}\ T^{\pi,P}v^{P}(s)
\end{aligned} π P ∈ Π S is uniformly optimal ⟺ T P v P = T π P , P v P ⟺ the support of π P ( s ) ∈ Δ ( A ) is contained in a ∈ A arg max P ( s , a ) T ( r ( s , a ) + γ v P ) , ∀ s ∈ S ⟺ π P ∈ s ∈ S ⋂ π ∈ Π S arg max T π , P v P ( s )
Proof. We actually prove all statements at the same time. The proof that T P T^{P} T P is a monotone contraction is identical to that of the same properties of T π , P T^{\pi,P} T π , P for π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S and is omitted. Let v P ∈ R S v^{P}\in\mathbb{R}^{\mathcal{S}} v P ∈ R S be its unique fixed point. Let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S be any policy such that T P v P = T π , P v P T^{P}v^{P}=T^{\pi,P}v^{P} T P v P = T π , P v P (the existence of such π \pi π is from the attainability property). Note that T P v P = T π , P v P T^{P}v^{P}=T^{\pi,P}v^{P} T P v P = T π , P v P implies in particular that v P = T π , P v P v^{P}=T^{\pi,P}v^{P} v P = T π , P v P since v P v^{P} v P is the fixed point of T P T^{P} T P . Hence by uniqueness of the fixed point of T π , P T^{\pi,P} T π , P we actually get v P = v π , P v^{P}=v^{\pi,P} v P = v π , P . We now prove that π \pi π is a uniformly optimal policy. The fact that v π , P ≥ v π ′ , P v^{\pi,P}\geq v^{\pi',P} v π , P ≥ v π ′ , P for any stationary π ′ ∈ Π S \pi'\in\Pi_{\mathcal{S}} π ′ ∈ Π S is easy to prove: indeed we have
v π , P = v P = T P v P ≥ T π ′ , P v P v^{\pi,P}=v^{P}=T^{P}v^{P}\geq T^{\pi',P}v^{P} v π , P = v P = T P v P ≥ T π ′ , P v P
which implies, using monotonicity, that v P ≥ ( T π ′ , P ) t v P v^{P}\geq(T^{\pi',P})^{t}v^{P} v P ≥ ( T π ′ , P ) t v P for each t ∈ N t\in\mathbb{N} t ∈ N , and taking the limit as t → + ∞ t\to+\infty t → + ∞ and using the contraction property, that v π , P ≥ v π ′ , P v^{\pi,P}\geq v^{\pi',P} v π , P ≥ v π ′ , P for any π ′ ∈ Π S \pi'\in\Pi_{\mathcal{S}} π ′ ∈ Π S . We must also show that v π , P ≥ v π ′ , P v^{\pi,P}\geq v^{\pi',P} v π , P ≥ v π ′ , P for π ′ ∈ Π H \pi'\in\Pi_{\mathcal{H}} π ′ ∈ Π H , i.e. we must show that π \pi π is better than history-dependent policies.
Let π ′ ∈ Π H \pi'\in\Pi_{\mathcal{H}} π ′ ∈ Π H and h t h_t h t be a history ending at period t t t at s t s_t s t . Consider playing for one period then quitting the game with the value given by v P v^{P} v P . Then from s t s_t s t the policy π ′ \pi' π ′ would obtain
E π ′ , P [ r ( s t , a t , s t + 1 ) + γ v P ( s t + 1 ) ∣ h t ] = ∑ a ∈ A π ′ ( h t , a ) P ( s t , a ) T ( r ( s t , a ) + γ v P ) ≤ max a ∈ A P ( s t , a ) T ( r ( s t , a ) + γ v P ) = T P v P ( s t ) = v P ( s t ) \begin{aligned}
\mathbb{E}^{\pi',P}\left[r(s_t,a_t,s_{t+1})+\gamma v^{P}(s_{t+1})\mid h_t\right]=\sum_{a\in\mathcal{A}}\pi'(h_t,a)P(s_t,a)^{\mathsf{T}}\left(r(s_t,a)+\gamma v^{P}\right) \\
&\leq\max_{a\in\mathcal{A}}P(s_t,a)^{\mathsf{T}}\left(r(s_t,a)+\gamma v^{P}\right)=T^{P}v^{P}(s_t)=v^{P}(s_t)
\end{aligned} E π ′ , P [ r ( s t , a t , s t + 1 ) + γ v P ( s t + 1 ) ∣ h t ] = a ∈ A ∑ π ′ ( h t , a ) P ( s t , a ) T ( r ( s t , a ) + γ v P ) ≤ a ∈ A max P ( s t , a ) T ( r ( s t , a ) + γ v P ) = T P v P ( s t ) = v P ( s t )
where the expectation in the first line is over the chosen action a t a_t a t and the next state s t + 1 s_{t+1} s t + 1 , and where π ′ ( h t , a ) \pi'(h_t,a) π ′ ( h t , a ) is the probability that π ′ \pi' π ′ chooses action a a a for h t h_t h t . Repeating this argument over H + 1 H+1 H + 1 periods we get
E π ′ , P [ ∑ t = 0 H γ t r ( s t , a t , s t + 1 ) + γ H + 1 v P ( s H + 1 ) ∣ s 0 = s ] ≤ v P ( s ) . \mathbb{E}^{\pi',P}\left[\left.\sum_{t=0}^{H}\gamma^{t}r(s_t,a_t,s_{t+1})+\gamma^{H+1}v^{P}(s_{H+1})\right|s_0=s\right]\leq v^{P}(s). E π ′ , P [ t = 0 ∑ H γ t r ( s t , a t , s t + 1 ) + γ H + 1 v P ( s H + 1 ) s 0 = s ] ≤ v P ( s ) .
Since γ H → 0 \gamma^{H}\to0 γ H → 0 and the rewards are bounded, we can conclude that
v π ′ , P ( s ) = E π ′ , P [ ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) ∣ s 0 = s ] ≤ v P ( s ) . v^{\pi',P}(s)=\mathbb{E}^{\pi',P}\left[\left.\sum_{t=0}^{+\infty}\gamma^{t}r(s_t,a_t,s_{t+1})\right|s_0=s\right]\leq v^{P}(s). v π ′ , P ( s ) = E π ′ , P [ t = 0 ∑ + ∞ γ t r ( s t , a t , s t + 1 ) s 0 = s ] ≤ v P ( s ) .
Since v P = v π , P v^{P}=v^{\pi,P} v P = v π , P we conclude that π \pi π is a uniformly optimal policy.
We now prove that any stationary uniformly optimal policy π ˉ ∈ Π S \bar{\pi} \in\Pi_{\mathrm{S}} π ˉ ∈ Π S is such that T P v P = T π ˉ , P v P T^{P}v^{P}=T^{\bar{\pi},P}v^{P} T P v P = T π ˉ , P v P . For π ˉ ∈ Π S \bar{\pi} \in\Pi_{\mathrm{S}} π ˉ ∈ Π S stationary uniformly optimal we must have v π ˉ , P = v P v^{\bar{\pi},P}=v^{P} v π ˉ , P = v P (since v P v^{P} v P is attained by some policies). This implies that T P v P = v P = v π ˉ , P = T π ˉ , P v π ˉ , P = T π ˉ , P v P T^{P}v^{P}=v^{P}=v^{\bar{\pi},P}=T^{\bar{\pi},P}v^{\bar{\pi},P}=T^{\bar{\pi},P}v^{P} T P v P = v P = v π ˉ , P = T π ˉ , P v π ˉ , P = T π ˉ , P v P , i.e., T P v P = T π ˉ , P v P T^{P}v^{P}=T^{\bar{\pi},P}v^{P} T P v P = T π ˉ , P v P , as claimed.
Finally, we can find a stationary deterministic uniformly optimal policy since each component of the Bellman operator (14) optimizes a linear form over the set Π S \Pi_{\mathrm{S}} Π S , and the extreme points of the set of stationary policies Π S \Pi_{\mathrm{S}} Π S coincide with the set of stationary deterministic policies Π S D \Pi_{\mathrm{SD}} Π SD .
From the above theorem we get that the equation v = T P v v=T^{P}v v = T P v in the variable v ∈ R S v\in\mathbb{R}^{S} v ∈ R S has a unique solution which is equal to v P ∈ R S v^{P}\in\mathbb{R}^{S} v P ∈ R S ; this equation is referred to as the “Bellman optimality equations”. There are some misconceptions that we should avoid when thinking about the above theorem:
Non-stationary optimal policies. While an optimal policy may be chosen stationary and deterministic, there may exist stationary randomized optimal policies, and even history-dependent optimal policies. This is because any policy that chooses (randomly) an action in arg max a ∈ A P ( s t , a ) T ( r ( s t , a ) + γ v P ) \mathop{\arg\max}_{a\in A}P(s_t,a)^{\mathsf{T}}(r(s_t,a)+\gamma v^{P}) arg max a ∈ A P ( s t , a ) T ( r ( s t , a ) + γ v P ) at each state s t ∈ S s_t\in S s t ∈ S visited at period t ∈ N t\in\mathbb{N} t ∈ N is optimal across all initial distributions p 0 p_0 p 0 . Hence in general it is improper to talk of “the” solution of (2) (e.g. “the optimal policy ...”), one should talk of “a” solution.
I emphasized the word “visited” above because, obviously, we don’t need to play optimally at states that are never visited! This highlights that while we have defined optimal policies as optimal solutions in (2) (which in principle depends on the choice of p 0 p_0 p 0 ), in practice we will most of the time be interested in computing a uniformly optimal solution π P ∈ Π S D \pi^{P}\in\Pi_{\mathrm{SD}} π P ∈ Π SD that is optimal in a stronger sense (optimal for all p 0 p_0 p 0 at the same time).
Optimality vs. uniform optimality. Note that the second statement is false if we replace “... of a stationary uniformly optimal policy π P \pi^{P} π P ...” by “... of an optimal stationary policy π π \pi^{\pi} π π ...”, i.e., the value function of an optimal policy need not be equal to the fixed point of the Bellman operator!
Indeed an optimal policy that is non-uniformly optimal must choose suboptimal actions in some states, which from the above item means it doesn’t play an action in the arg max \arg\max arg max of the Bellman operator T P T^{P} T P at some states.
A more mathematical reformulation of the last paragraph is as follows: given v P v^{P} v P the fixed point of T P T^{P} T P , any policy π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S such that T π , P v P = T P v P T^{\pi,P}v^{P}=T^{P}v^{P} T π , P v P = T P v P is uniformly optimal. However, there may be optimal (non-uniformly optimal) policies π ′ ∈ Π S \pi'\in\Pi_{\mathrm{S}} π ′ ∈ Π S that do not satisfy T π ′ , P v P = T P v P T^{\pi',P}v^{P}=T^{P}v^{P} T π ′ , P v P = T P v P . This is because
R ( π , P ) = ∑ s ∈ S p 0 ( s ) v π , P ( s ) R(\pi,P)=\sum_{s\in S}p_0(s)v^{\pi,P}(s) R ( π , P ) = s ∈ S ∑ p 0 ( s ) v π , P ( s )
hence if a policy chooses suboptimal actions at a state that is never visited, this does not impact the discounted return R ( π , P ) R(\pi,P) R ( π , P ) . I give an example of this type of issue below (see continued Example 1.2). We also note that the existence of uniformly optimal policies makes the initial distribution p 0 p_0 p 0 irrelevant in the description of the MDP instance. This will not be true in general for robust MDPs, but this will be true for robust MDPs with rectangular models of uncertainty (see Chapter 2 and Chapter 3).
Beyond stationarity. It is worth emphasizing that the value functions of non-stationary (history-dependent or Markovian) policies π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H are well-defined objects. However we can’t define the Bellman operator T π , P T^{\pi,P} T π , P as in (1.11) because (1.11) involves the notation π ( s , a ) \pi(s,a) π ( s , a ) , which is ill-defined for non-stationary policies: by definition for non-stationary policies we need more information than just the current state to pick some actions (for Markovian policies we need additionally the current period, and for history-dependent policies we need the full history up to the current state).
We compute the optimal policies and uniformly optimal policies in the next example.
Example 1.1 continued. We saw that the value function of a policy choosing a 1 a_1 a 1 with probability x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] in state s 0 s_0 s 0 is v ( s 0 ) = γ 2 ( 1 − γ ) ( x p + ( 1 − p ) ( 1 − x ) ) v(s_0) = \frac{\gamma}{2(1-\gamma)}(xp + (1-p)(1-x)) v ( s 0 ) = 2 ( 1 − γ ) γ ( x p + ( 1 − p ) ( 1 − x )) . Note that x p + ( 1 − p ) ( 1 − x ) = ( 2 p − 1 ) ⋅ x + ( 1 − p ) xp + (1-p)(1-x) = (2p-1)\cdot x + (1-p) x p + ( 1 − p ) ( 1 − x ) = ( 2 p − 1 ) ⋅ x + ( 1 − p ) . Maximizing this last expression in x x x allows us to compute optimal policies: if p < 1 / 2 p < 1/2 p < 1/2 then 2 p − 1 < 0 2p-1 < 0 2 p − 1 < 0 and we pick action a 2 a_2 a 2 with probability 1 (i.e., we choose x ⋆ = 0 x^\star= 0 x ⋆ = 0 ), otherwise if p > 1 / 2 p > 1/2 p > 1/2 we pick action a 1 a_1 a 1 (i.e., x ⋆ = 1 x^\star= 1 x ⋆ = 1 ). At p = 1 / 2 p = 1/2 p = 1/2 we have 2 p − 1 = 0 2p-1 = 0 2 p − 1 = 0 , we are indifferent and we always get a payoff proportional to 1 − p = 1 / 2 1-p = 1/2 1 − p = 1/2 . Note that here we also computed a uniformly optimal policy (since from G , B G,B G , B we transition deterministically to s 0 s_0 s 0 ).
Example 1.2 continued. The optimal policies depend on the scalar values p , p ′ ∈ [ 0 , 1 ] p,p' \in[0,1] p , p ′ ∈ [ 0 , 1 ] and it is a bit tedious to enumerate all possibilities. Assume for now that we start in s 0 s_0 s 0 (p 0 ( s 0 ) = 1 p_0(s_0)=1 p 0 ( s 0 ) = 1 and p 0 ( s ) = 0 p_0(s)=0 p 0 ( s ) = 0 for s ∈ { s 1 , s 2 , s 3 , s 4 } s \in\{s_1,s_2,s_3,s_4\} s ∈ { s 1 , s 2 , s 3 , s 4 } ) and that p = 0 p=0 p = 0 , p ′ = 1 / 4 p'=1/4 p ′ = 1/4 . The value functions that we already computed simplify as follows. Choosing action a 1 a_1 a 1 at s 1 s_1 s 1 (y = 1 y=1 y = 1 ) yields a value from s 1 s_1 s 1 of γ 2 ( 1 − γ ) \frac{\gamma}{2(1-\gamma)} 2 ( 1 − γ ) γ while action a 2 a_2 a 2 (y = 0 y=0 y = 0 ) yields a value from s 1 s_1 s 1 of − γ 2 ( 1 − γ ) -\frac{\gamma}{2(1-\gamma)} − 2 ( 1 − γ ) γ . Hence we want to pick action a 1 a_1 a 1 at s 1 s_1 s 1 , i.e., y = 1 y=1 y = 1 . This makes sense just looking at the probabilities: since p ′ = 1 / 4 < 1 / 2 p'=1/4 < 1/2 p ′ = 1/4 < 1/2 then a 1 a_1 a 1 gives us more chance than a 2 a_2 a 2 to end up in the “good” state s 4 s_4 s 4 with reward + 1 +1 + 1 instead of ending up in the “bad” state s 3 s_3 s 3 with reward − 1 -1 − 1 .
Now let us compare the two actions at state s 0 s_0 s 0 . If we choose a 1 a_1 a 1 at s 0 s_0 s 0 we transition deterministically to s 2 s_2 s 2 and we get a return of γ v ( s 2 ) = γ 2 ( 1 − γ ) \gamma v(s_2)=\frac{\gamma}{2(1-\gamma)} γ v ( s 2 ) = 2 ( 1 − γ ) γ . Otherwise if we pick a 2 a_2 a 2 at s 0 s_0 s 0 we transition deterministically to s 1 s_1 s 1 and we get a return of γ v ( s 1 ) = γ 2 2 ( 1 − γ ) \gamma v(s_1)=\frac{\gamma^2}{2(1-\gamma)} γ v ( s 1 ) = 2 ( 1 − γ ) γ 2 . Since γ 2 < γ \gamma^2 < \gamma γ 2 < γ for γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) , we see that we prefer to transition to s 2 s_2 s 2 from s 0 s_0 s 0 , i.e., we pick action a 1 a_1 a 1 in $s_0. This shows that the action that we choose at s 1 s_1 s 1 is irrelevant: any policy that chooses a 1 a_1 a 1 with probability 1 at s 0 s_0 s 0 (i.e. x = 1 x=1 x = 1 ) is optimal, whatever the choice of actions at s 1 s_1 s 1 . However if we want to choose a uniformly optimal policy we must also play optimally at s 1 s_1 s 1 (i.e., choose action a 1 a_1 a 1 at s 1 s_1 s 1 ).
We conclude by highlighting another important difference between the set of stationary uniformly optimal policies and the set of stationary optimal policies: the former set is always convex, while the latter may not be convex. This is because optimal policies need not take optimal actions in parts of the set of states that they don’t visit, whereas uniform optimality imposes componentwise dominance of the value function, i.e., a uniformly optimal policy plays optimally starting from any state. This convexity property is why it is often much simpler to work with uniform optimality than with vanilla optimality.
Lemma 1.1. 1. The set of stationary uniformly optimal policies is always a convex closed set.
The set of stationary optimal policies is always closed but may not be convex.
Proof. 1. From the third statement in Theorem 1.1 we know that the set of stationary uniformly optimal policies is the finite intersection over s ∈ S s \in\mathcal{S} s ∈ S of the sets a r g m a x π ∈ Π S T π , P v P ( s ) \mathop{\rm arg\,max}_{\pi\in\Pi_{\mathrm{S}}} T^{\pi,P}v_P(s) arg max π ∈ Π S T π , P v P ( s ) . Each of these optimization programs optimizes a linear form π ↦ T π , P v P ( s ) \pi\mapsto T^{\pi,P}v_P(s) π ↦ T π , P v P ( s ) over the polytope Π S \Pi_{\mathrm{S}} Π S , hence these sets are convex, and the intersection of these sets is convex and closed hence it is a convex closed set.
The set of stationary optimal policies is closed by continuity of π ↦ R ( π , P ) \pi\mapsto R(\pi,P) π ↦ R ( π , P ) . For the non-convexity of the set of optimal policies, consider the MDP instance depicted in Figure 1.4 . There are five states { s 0 , s 1 , s 2 , G , B } \{s_0,s_1,s_2,G,B\} { s 0 , s 1 , s 2 , G , B } and two actions a 1 , a 2 a_1,a_2 a 1 , a 2 . The states
Figure 1.4. (a) Transitions and rewards for action a 1 a_1 a 1 ; (b) Transitions and rewards for action a 2 a_2 a 2 . The transition probabilities and rewards are indicated beside each arc. Absorbing states are represented by squares.
G , B G,B G , B are absorbing and the MDP starts in state s 0 s_0 s 0 with probability 1. The rewards are 0 everywhere except at G G G where we obtain a reward of +1. The transitions are deterministic as follows: under a 1 a_1 a 1 the transitions are s 0 → s 1 s_0 \to s_1 s 0 → s 1 , s 1 → G s_1 \to G s 1 → G , s 2 → B s_2 \to B s 2 → B , and under a 2 a_2 a 2 the transitions are s 0 → s 2 s_0 \to s_2 s 0 → s 2 , s 1 → B s_1 \to B s 1 → B , s 2 → G s_2 \to G s 2 → G .
A policy is optimal if and only if with probability 1 we end up in state G G G . Let π 1 ∈ Π S \pi^1 \in\Pi_{\mathrm{S}} π 1 ∈ Π S be defined as π 1 ( s 0 , a 1 ) = π 1 ( s 1 , a 1 ) = π 1 ( s 2 , a 1 ) = 1 \pi^1(s_0,a_1)=\pi^1(s_1,a_1)=\pi^1(s_2,a_1)=1 π 1 ( s 0 , a 1 ) = π 1 ( s 1 , a 1 ) = π 1 ( s 2 , a 1 ) = 1 . The path followed under π 1 \pi^1 π 1 is s 0 → s 1 → G s_0 \to s_1 \to G s 0 → s 1 → G and s 2 s_2 s 2 is never visited (even though we take a suboptimal action at s 2 s_2 s 2 ). Hence π 1 \pi^1 π 1 is optimal. Let π 2 ∈ Π S \pi^2 \in\Pi_{\mathrm{S}} π 2 ∈ Π S be defined as π 2 ( s 0 , a 2 ) = π 2 ( s 1 , a 2 ) = π 2 ( s 2 , a 2 ) = 1 \pi^2(s_0,a_2)=\pi^2(s_1,a_2)=\pi^2(s_2,a_2)=1 π 2 ( s 0 , a 2 ) = π 2 ( s 1 , a 2 ) = π 2 ( s 2 , a 2 ) = 1 . The path followed under π 2 \pi^2 π 2 is s 0 → s 2 → G s_0 \to s_2 \to G s 0 → s 2 → G and s 1 s_1 s 1 is never visited (even though we take a suboptimal action at s 1 s_1 s 1 ). However it is clear that 1 2 π 1 + 1 2 π 2 \frac{1}{2}\pi^1+\frac{1}{2}\pi^2 2 1 π 1 + 2 1 π 2 , the policy that plays a 1 a_1 a 1 and a 2 a_2 a 2 with probability 1 / 2 1/2 1/2 at s 0 , s 1 s_0,s_1 s 0 , s 1 and s 2 s_2 s 2 , is strictly suboptimal, because there is a positive probability (here, 1 / 2 1/2 1/2 ) that we end up in state B B B under this policy.
Recall that a function f : R n → R f : \mathbb{R}^{n} \to\mathbb{R} f : R n → R is said to be quasi-convex if its sublevel set { x ∈ R n ∣ f ( x ) ≤ α } \{x \in\mathbb{R}^{n} \mid f(x) \le\alpha\} { x ∈ R n ∣ f ( x ) ≤ α } is convex for each α ∈ R \alpha\in\mathbb{R} α ∈ R (this is equivalent to saying that the maximum of f f f over any segment is attained at an endpoint of this segment). The second statement in Lemma 1.1 implies that π ↦ R ( π , P ) \pi\mapsto R(\pi, P) π ↦ R ( π , P ) is not a quasi-convex function in general.
Corollary 1.1. The function π ↦ R ( π , P ) \pi\mapsto R(\pi, P) π ↦ R ( π , P ) may not be quasi-convex over Π S \Pi_{\mathrm{S}} Π S .
Proof. Take α = − max π ∈ Π S R ( π , P ) \alpha= -\max_{\pi\in\Pi_{\mathrm{S}}} R(\pi, P) α = − max π ∈ Π S R ( π , P ) and consider the MDP from the second statement in Lemma 1.1 but with a minus sign for the rewards. Then the sublevel set associated with α \alpha α for the return in this “flipped” MDP instance is the set of optimal policies, which is not convex, hence the return is not quasi-convex.
Algorithms and complexity.
Theorem 1.1 shows how to compute an optimal policy as soon as we know v P v^{P} v P , the fixed point of T P T^{P} T P . An interesting observation is that computing v P v^{P} v P requires (a priori) solving a non-linear program (computing the fixed point of the non-linear operator T P T^{P} T P ). However the same theorem tells us that once we know v P v^{P} v P we can compute a uniformly optimal policy π ∈ Π S D \pi\in\Pi_{\mathrm{SD}} π ∈ Π SD by choosing, for each s ∈ S s \in S s ∈ S , an action in
arg max a ∈ A P ( s , a ) T ( r ( s , a ) + γ v P ) . \underset{a \in A}{\operatorname{arg\,max}}\; P(s,a)^{\mathsf{T}}\left(r(s,a)+\gamma v^{P}\right). a ∈ A arg max P ( s , a ) T ( r ( s , a ) + γ v P ) .
The classical algorithms for solving MDPs are variants of the following three methods for computing the fixed point of the Bellman optimality operator T P T^{P} T P : value iteration, policy iteration, and linear programming. We do not cover (yet?) policy mirror descent in these notes.
Value iteration. Value iteration is based on the contraction property of T P T^{P} T P . The convergence rate of Algorithm 1 is given next. It is a straightforward corollary of a more general version of Value Iteration, given in Appendix § 8 .
Algorithm 1 Value iteration for nominal MDPs
Initialize v 0 ∈ R S v^{0} \in\mathbb{R}^{S} v 0 ∈ R S .
for t ∈ N t \in\mathbb{N} t ∈ N do
v t + 1 = T P v t v^{t+1}=T^{P}v^{t} v t + 1 = T P v t
end for
Hence rate of Algorithm 1 is given next. It is a straightforward corollary of a more general version of Value Iteration, given in Appendix § 8 .
Proposition 1.2. Let ( v t ) t ∈ N (v^{t})_{t \in\mathbb{N}} ( v t ) t ∈ N be the sequence of vectors generated by Algorithm 1.
The sequence ( v t ) t ∈ N (v^{t})_{t \in\mathbb{N}} ( v t ) t ∈ N converges to v P v^{P} v P at a geometric rate of γ \gamma γ :
∥ v P − v t ∥ ∞ ≤ γ t ∥ v P − v 0 ∥ ∞ \lVert v^{P}-v^{t}\rVert_{\infty} \leq\gamma^{t}\lVert v^{P}-v^{0}\rVert_{\infty} ∥ v P − v t ∥ ∞ ≤ γ t ∥ v P − v 0 ∥ ∞ Let ϵ > 0 \epsilon> 0 ϵ > 0 . Assume that ∥ v P − v t ∥ ∞ ≤ ϵ \lVert v^{P}-v^{t}\rVert_{\infty} \leq\epsilon ∥ v P − v t ∥ ∞ ≤ ϵ for some t ∈ N t \in\mathbb{N} t ∈ N and let π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S such that T π , P v t = T P v t T^{\pi,P}v^{t}=T^{P}v^{t} T π , P v t = T P v t . Then π \pi π is a uniformly 2 ϵ γ ( 1 − γ ) − 1 2\epsilon\gamma(1-\gamma)^{-1} 2 ϵ γ ( 1 − γ ) − 1 -optimal policy: ∥ v P − v π , P ∥ ∞ ≤ 2 ϵ γ ( 1 − γ ) − 1 \lVert v^{P}-v^{\pi,P}\rVert_{\infty} \leq2\epsilon\gamma(1-\gamma)^{-1} ∥ v P − v π , P ∥ ∞ ≤ 2 ϵ γ ( 1 − γ ) − 1 .
Additionally, we can check that ∥ v P − v t ∥ ∞ ≤ ϵ \lVert v^{P}-v^{t}\rVert_{\infty}\le\epsilon ∥ v P − v t ∥ ∞ ≤ ϵ by checking ∥ T P v t − v t ∥ ∞ ≤ ϵ ( 1 − γ ) \lVert T^{P}v^{t}-v^{t}\rVert_{\infty}\le\epsilon(1-\gamma) ∥ T P v t − v t ∥ ∞ ≤ ϵ ( 1 − γ ) , and that last inequality holds after at most ⌊ 1 1 − γ log ( ∥ v 1 − v 0 ∥ ∞ ϵ ( 1 − γ ) ) ⌋ + 1 \left\lfloor\frac{1}{1-\gamma}\log\left(\frac{\lVert v^{1}-v^{0}\rVert_{\infty}}{\epsilon(1-\gamma)}\right)\right\rfloor+1 ⌊ 1 − γ 1 log ( ϵ ( 1 − γ ) ∥ v 1 − v 0 ∥ ∞ ) ⌋ + 1 iterations of Algorithm 1.
Note that the second item of the above proposition provides an implementable stopping criterion for Algorithm 1 to ensure that we obtain a uniformly ϵ \epsilon ϵ -optimal policy.
Policy iteration. Policy iteration is an exact method that terminates with an optimal policy (whereas value iteration needs to be stopped and returns an ϵ \epsilon ϵ -optimal policy, for some ϵ > 0 \epsilon>0 ϵ > 0 that we can control). In policy iteration we generate a sequence of stationary deterministic policies ( π t ) t ∈ N (\pi^{t})_{t\in\mathbb{N}} ( π t ) t ∈ N , where the current policy π t \pi^{t} π t greedily improves upon the previous one π t − 1 \pi^{t-1} π t − 1 . A natural stopping criterion is to stop as soon as we can choose π t = π t − 1 \pi^{t}=\pi^{t-1} π t = π t − 1 . Policy
Algorithm 2 Policy iteration for nominal MDPs
Initialize π 0 ∈ Π S D \pi^{0}\in\Pi_{\mathrm{SD}} π 0 ∈ Π SD , v 0 = v π 0 , P v^{0}=v^{\pi^{0},P} v 0 = v π 0 , P
for t ∈ N t\in\mathbb{N} t ∈ N do
Choose π t + 1 ∈ Π S D \pi^{t+1}\in\Pi_{\mathrm{SD}} π t + 1 ∈ Π SD such that T P v t ( s ) = T π t + 1 , P v t ( s ) T^{P}v^{t}(s)=T^{\pi^{t+1},P}v^{t}(s) T P v t ( s ) = T π t + 1 , P v t ( s ) for all s ∈ S s\in S s ∈ S (and equal to π t \pi^{t} π t if we can)
Choose v t + 1 = v π t + 1 , P v^{t+1}=v^{\pi^{t+1},P} v t + 1 = v π t + 1 , P
if π t + 1 = π t \pi^{t+1}=\pi^{t} π t + 1 = π t then return π t + 1 \pi^{t+1} π t + 1
end if
end for
iteration can be interpreted as a “block simplex” algorithm on the linear programming formulation of MDPs (see next algorithm). The proof of the next proposition is given (in more generality) in Appendix B. The convergence of Algorithm 2 is a bit longer to prove than for Algorithm 1 but uses the same ideas; again we only use the monotonicity, contractivity, and attainability properties of the family of operators { T π , P ∣ π ∈ Π S D } \{T^{\pi,P}\mid\pi\in\Pi_{\mathrm{SD}}\} { T π , P ∣ π ∈ Π SD } . More refined convergence analysis exists, based on analyzing the connection between policy iteration and (block) simplex algorithms for the next linear programming formulation, see Section 1.6.
Proposition 1.3. Let ( π t ) t ∈ N (\pi^{t})_{t\in\mathbb{N}} ( π t ) t ∈ N be the sequence of policies generated by Algorithm 2. Then ∥ v P − v π t , P ∥ ∞ ≤ γ t ∥ v P − v π 0 , P ∥ ∞ \lVert v^{P}-v^{\pi^{t},P}\rVert_{\infty}\le\gamma^{t}\lVert v^{P}-v^{\pi^{0},P}\rVert_{\infty} ∥ v P − v π t , P ∥ ∞ ≤ γ t ∥ v P − v π 0 , P ∥ ∞ .
Linear programming and the contraction lemma. The linear programming formulation of MDPs is based on the following contraction lemma , which shows that the fixed point of a monotone contraction is the unique solution to minimizing a componentwise increasing function over the set of super-solutions.
Lemma 1.2 (Contraction lemma). Let F : R S → R S F:\mathbb{R}^{S}\to\mathbb{R}^{S} F : R S → R S be a monotone contraction with fixed point v P ∈ R S v^{P}\in\mathbb{R}^{S} v P ∈ R S . Then
{ v P } = a r g m i n { ∑ s ∈ S v ( s ) | v ≥ F v , v ∈ R S } . \{v^{P}\}=\mathop{\rm arg\,min}\left\{\sum_{s\in S}v(s)\ \middle|\ v\geq Fv,\ v\in\mathbb{R}^{S}\right\}. { v P } = arg min { s ∈ S ∑ v ( s ) v ≥ F v , v ∈ R S } .
Proof. By definition v P v^{P} v P is a feasible solution of the optimization problem on the right-hand side of the above equation. Additionally for v ∈ R S v \in\mathbb{R}^{S} v ∈ R S feasible we get, using monotonicity, that v ≥ F v ≥ F t v v \ge Fv \ge F^{t}v v ≥ F v ≥ F t v and since lim t → + ∞ F t v = v P \lim_{t\to+\infty}F^{t}v=v^{P} lim t → + ∞ F t v = v P we get that v ≥ v P v \ge v^{P} v ≥ v P .
Applying the contraction lemma to the Bellman operator T P T^{P} T P yields the following proposition.
Proposition 1.4. The optimal value function v P v^{P} v P of an MDP is the unique optimal solution to the following linear program:
min { ∑ s ∈ S v ( s ) | v ( s ) ≥ P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ ( s , a ) ∈ S × A , v ∈ R S } (17) \min\left\{ \sum_{s\in S}v(s)\ \middle|\ v(s) \ge P(s,a)^{\top}(r(s,a)+\gamma v),\ \forall(s,a)\in S\times A,\ v\in\mathbb{R}^{S} \right\} \tag*{(17)} min { s ∈ S ∑ v ( s ) v ( s ) ≥ P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ ( s , a ) ∈ S × A , v ∈ R S } (17)
Proof. Immediate from the contraction lemma and the fact that v ( s ) ≥ max a ∈ A P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ s ∈ S ⟺ v ( s ) ≥ P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ ( s , a ) ∈ S × A v(s) \ge\max_{a\in A}P(s,a)^{\top}(r(s,a)+\gamma v),\ \forall s\in S \Longleftrightarrow v(s) \ge P(s,a)^{\top}(r(s,a)+\gamma v),\ \forall(s,a)\in S\times A v ( s ) ≥ max a ∈ A P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ s ∈ S ⟺ v ( s ) ≥ P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ ( s , a ) ∈ S × A .
Note that we use the objective function v ↦ ∑ s ∈ S v ( s ) v\mapsto\sum_{s\in S}v(s) v ↦ ∑ s ∈ S v ( s ) . It would be more natural to use v ↦ p 0 ⊤ v v\mapsto p_{0}^{\top}v v ↦ p 0 ⊤ v but if some coefficients of p 0 p_{0} p 0 are equal to 0 then there may be other optimal solutions than v P v^{P} v P , from which we wouldn’t necessarily recover a uniformly optimal policy. In fact we may not recover an optimal policy at all if p 0 p_{0} p 0 does not have full support and we just consider an arbitrary optimal solution to min { p 0 ⊤ v ∣ v ≥ T P v } \min\{p_{0}^{\top}v\mid v\ge T^{P}v\} min { p 0 ⊤ v ∣ v ≥ T P v } , as the next example shows.
Example 1.2 continued. Consider Example 1.2 with p = 0 p=0 p = 0 , p ′ = 1 / 4 p'=1/4 p ′ = 1/4 , γ = 1 / 2 \gamma=1/2 γ = 1/2 , p 0 ( s 0 ) = 1 p_{0}(s_{0})=1 p 0 ( s 0 ) = 1 (the same parameters that we used right before Lemma 1.1). The linear program min { p 0 ⊤ v ∣ v ≥ T P v } \min\{p_{0}^{\top}v\mid v\ge T^{P}v\} min { p 0 ⊤ v ∣ v ≥ T P v } becomes
min v ( s 0 ) s.t. v ( s 0 ) ≥ 1 2 v ( s 2 ) v ( s 0 ) ≥ 1 2 v ( s 1 ) v ( s 1 ) ≥ 1 8 v ( s 3 ) + 3 8 v ( s 4 ) v ( s 1 ) ≥ 3 8 v ( s 3 ) + 1 8 v ( s 4 ) v ( s 2 ) ≥ 1 , v ( s 3 ) ≥ − 2 , v ( s 4 ) ≥ 2 , v ∈ R S . (18) \begin{aligned}
\min\quad&v(s_{0})\\
\text{s.t.}\quad&v(s_{0})\ge\frac{1}{2}v(s_{2})\\
&v(s_{0})\ge\frac{1}{2}v(s_{1})\\
&v(s_{1})\ge\frac{1}{8}v(s_{3})+\frac{3}{8}v(s_{4})\\
&v(s_{1})\ge\frac{3}{8}v(s_{3})+\frac{1}{8}v(s_{4})\\
&v(s_{2})\ge1,\ v(s_{3})\ge-2,\ v(s_{4})\ge2,\\
&v\in\mathbb{R}^{S}. \tag*{(18)}
\end{aligned} min s.t. v ( s 0 ) v ( s 0 ) ≥ 2 1 v ( s 2 ) v ( s 0 ) ≥ 2 1 v ( s 1 ) v ( s 1 ) ≥ 8 1 v ( s 3 ) + 8 3 v ( s 4 ) v ( s 1 ) ≥ 8 3 v ( s 3 ) + 8 1 v ( s 4 ) v ( s 2 ) ≥ 1 , v ( s 3 ) ≥ − 2 , v ( s 4 ) ≥ 2 , v ∈ R S . (18)
We have already computed an optimal policy and we know that v P = ( 1 2 , 1 2 , 1 , − 2 , 2 ) v^{P}=(\frac{1}{2},\frac{1}{2},1,-2,2) v P = ( 2 1 , 2 1 , 1 , − 2 , 2 ) with optimal value v P ( s 0 ) = 1 2 v^{P}(s_{0})=\frac{1}{2} v P ( s 0 ) = 2 1 . We already argued that v P v^{P} v P is an optimal solution for the linear program (1.18). However note that for v P v^{P} v P there is a positive slack in the second constraint, involving v ( s 1 ) v(s_{1}) v ( s 1 ) and v ( s 0 ) v(s_{0}) v ( s 0 ) . Hence we can increase v P ( s 1 ) v^{P}(s_{1}) v P ( s 1 ) and still get an optimal solution to (1.18) (the objective only involves v ( s 0 ) v(s_{0}) v ( s 0 ) ). In particular, v ^ = ( 1 2 , 1 , 1 , − 2 , 2 ) \hat{v}=(\frac{1}{2},1,1,-2,2) v ^ = ( 2 1 , 1 , 1 , − 2 , 2 ) is feasible and optimal in (1.18), however, based on this vector, at s 0 s_{0} s 0 action a 1 a_{1} a 1 and a 2 a_{2} a 2 both lead to a continuation value of 1 / 2 1/2 1/2 , so assume that a greedy policy for this vector chooses action a 2 a_{2} a 2 in s 0 s_{0} s 0 ; at s 1 s_{1} s 1 , a 1 a_{1} a 1 yields a continuation value of 1 / 2 1/2 1/2 and a 2 a_{2} a 2 yields a continuation value of − 1 / 2 -1/2 − 1/2 , so we would take a 1 a_{1} a 1 . But the resulting policy has return 1 / 4 1/4 1/4 , which is strictly suboptimal (the optimal objective is 1 / 2 1/2 1/2 ). The issue here comes from the fact that v ^ \hat{v} v ^ is not a fixed point of T P T^{P} T P so that in particular a greedy policy for v ^ \hat{v} v ^ need not have value function v ^ \hat{v} v ^ .
Finally, it is worth mentioning that the dual of (17) (if we use v ↦ p 0 ⊤ v v \mapsto p_{0}^{\top}v v ↦ p 0 ⊤ v in the objective, instead of the sum of the components of v v v ) yields a linear program whose variables are the state-action frequencies μ ∈ Q ⊂ R + S × A \mu\in Q \subset\mathbb{R}_{+}^{S \times A} μ ∈ Q ⊂ R + S × A , defined in (10) :
max { ∑ s ∈ S ∑ a ∈ A μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) ∣ μ ∈ Q } (19) \max\left\{\sum_{s\in S}\sum_{a\in A}\mu(s,a)P(s,a)^{\top}r(s,a)\mid\mu\in Q\right\} \tag*{(19)} max { s ∈ S ∑ a ∈ A ∑ μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) ∣ μ ∈ Q } (19)
So we can obtain two linear programs to solve MDPs: one with variables v ∈ R S v \in\mathbb{R}^{S} v ∈ R S (given in (17) ) and one with variables μ ∈ R + S × A \mu\in\mathbb{R}_{+}^{S \times A} μ ∈ R + S × A . While we can recover optimal policies from any optimal solutions of these linear programs, to the best of our knowledge, there is no linear programming formulation with variables π ∈ Π S \pi\in\Pi_{S} π ∈ Π S directly (which would be the most natural).
Complexity. Both value iteration and policy iteration require evaluating the Bellman operator T P v T^{P}v T P v at some point v ∈ R S v \in\mathbb{R}^{S} v ∈ R S . This can be done by computing, for each s ∈ S s \in S s ∈ S , the maximum over actions in A A A , of values involving the scalar product of vectors in R S \mathbb{R}^{S} R S . This can be done in O ( ∣ S ∣ 2 ∣ A ∣ ) O(|S|^{2}|A|) O ( ∣ S ∣ 2 ∣ A ∣ ) arithmetic operations. Policy Iteration additionally requires computing the value function for some stationary deterministic policies at each iteration, which may be expensive, as it involves inverting a matrix of dimension ∣ S ∣ × ∣ S ∣ |S| \times|S| ∣ S ∣ × ∣ S ∣ . To avoid doing so it is common to resort to modified policy iteration , and instead of computing v π , P v^{\pi,P} v π , P , to just compute a few consecutive images of T π , P T^{\pi,P} T π , P . The linear program for MDPs can be solved by any existing methods for linear programs, such as the simplex algorithm or interior point methods.
Additional properties*
We present here some additional properties of MDPs, which are not necessary for a first reading but provide interesting insights.
Behavioral policies. (A variant of) Kuhn’s theorem asserts that to describe a policy in our sequential decision-making process it is equivalent to (A) assign a (possibly randomized) action to each finite history ( s 0 , a 0 , … , s t − 1 , a t − 1 , s t ) (s_{0},a_{0},\ldots,s_{t-1},a_{t-1},s_{t}) ( s 0 , a 0 , … , s t − 1 , a t − 1 , s t ) for each period t t t (so-called “behavioral form”), or (B) to directly choose a probability distribution over trajectories (compatible with the transition probabilities, e.g., we can’t assign positive probabilities to sets of trajectories that contain transitions s , a → s ′ s,a \to s' s , a → s ′ that are forbidden, i.e. for which P ( s , a , s ′ ) = 0 P(s,a,s')=0 P ( s , a , s ′ ) = 0 ).
For instance consider Example 1.2 starting from s 0 s_{0} s 0 with probability 1. The set of all trajectories is:
( s 0 , a 1 , s 2 ∞ ) , ( s 0 , a 2 , s 2 ∞ ) ( s 0 , a 1 , s 1 , a 1 , s 3 ∞ ) , ( s 0 , a 1 , s 1 , a 2 , s 3 ∞ ) ( s 0 , a 2 , s 1 , a 1 , s 3 ∞ ) , ( s 0 , a 2 , s 1 , a 2 , s 3 ∞ ) ( s 0 , a 1 , s 1 , a 1 , s 4 ∞ ) , ( s 0 , a 1 , s 1 , a 2 , s 4 ∞ ) ( s 0 , a 2 , s 1 , a 1 , s 4 ∞ ) , ( s 0 , a 2 , s 1 , a 2 , s 4 ∞ ) \begin{aligned}
&(s_{0},a_{1},s_{2}^{\infty}),(s_{0},a_{2},s_{2}^{\infty})\\
&(s_{0},a_{1},s_{1},a_{1},s_{3}^{\infty}),(s_{0},a_{1},s_{1},a_{2},s_{3}^{\infty})\\
&(s_{0},a_{2},s_{1},a_{1},s_{3}^{\infty}),(s_{0},a_{2},s_{1},a_{2},s_{3}^{\infty})\\
&(s_{0},a_{1},s_{1},a_{1},s_{4}^{\infty}),(s_{0},a_{1},s_{1},a_{2},s_{4}^{\infty})\\
&(s_{0},a_{2},s_{1},a_{1},s_{4}^{\infty}),(s_{0},a_{2},s_{1},a_{2},s_{4}^{\infty})
\end{aligned} ( s 0 , a 1 , s 2 ∞ ) , ( s 0 , a 2 , s 2 ∞ ) ( s 0 , a 1 , s 1 , a 1 , s 3 ∞ ) , ( s 0 , a 1 , s 1 , a 2 , s 3 ∞ ) ( s 0 , a 2 , s 1 , a 1 , s 3 ∞ ) , ( s 0 , a 2 , s 1 , a 2 , s 3 ∞ ) ( s 0 , a 1 , s 1 , a 1 , s 4 ∞ ) , ( s 0 , a 1 , s 1 , a 2 , s 4 ∞ ) ( s 0 , a 2 , s 1 , a 1 , s 4 ∞ ) , ( s 0 , a 2 , s 1 , a 2 , s 4 ∞ )
where we use s ∞ s^{\infty} s ∞ to indicate that we have reached an absorbing state (hence from there on the state s s s is repeated in the trajectory and the actions are irrelevant). For some choices of p , p ′ ∈ [ 0 , 1 ] p, p' \in[0,1] p , p ′ ∈ [ 0 , 1 ] some of these trajectories must have probability 0. For instance if p = 0 p=0 p = 0 then choosing action a 2 a_2 a 2 cannot lead to s 2 s_2 s 2 from s 0 s_0 s 0 hence ( s 0 , a 2 , s 2 ∞ ) (s_0,a_2,s_2^{\infty}) ( s 0 , a 2 , s 2 ∞ ) must have probability 0. For the choice p = p ′ = 0 p=p'=0 p = p ′ = 0 the set of feasible trajectories is given by
h 1 : = ( s 0 , a 1 , s 2 ∞ ) h 2 : = ( s 0 , a 2 , s 1 , a 1 , s 4 ∞ ) h 3 : = ( s 0 , a 2 , s 1 , a 2 , s 3 ∞ ) \begin{aligned}
h_1 &:= (s_0,a_1,s_2^{\infty})\\
h_2 &:= (s_0,a_2,s_1,a_1,s_4^{\infty})\\
h_3 &:= (s_0,a_2,s_1,a_2,s_3^{\infty})
\end{aligned} h 1 h 2 h 3 := ( s 0 , a 1 , s 2 ∞ ) := ( s 0 , a 2 , s 1 , a 1 , s 4 ∞ ) := ( s 0 , a 2 , s 1 , a 2 , s 3 ∞ )
and specifying a probability distribution over these elements yields a policy. For instance choosing P ( h 1 ) = 1 / 2 \mathbb{P}(h_1)=1/2 P ( h 1 ) = 1/2 , P ( h 2 ) = P ( h 3 ) = 1 / 4 \mathbb{P}(h_2)=\mathbb{P}(h_3)=1/4 P ( h 2 ) = P ( h 3 ) = 1/4 is equivalent to playing a 1 a_1 a 1 with probability 1 / 2 1/2 1/2 at s 0 s_0 s 0 and s 1 s_1 s 1 . Note that we must have 1 − P ( h 1 ) = P ( h 2 ) + P ( h 3 ) 1-\mathbb{P}(h_1)=\mathbb{P}(h_2)+\mathbb{P}(h_3) 1 − P ( h 1 ) = P ( h 2 ) + P ( h 3 ) given we play either a 1 a_1 a 1 or a 2 a_2 a 2 in s 0 s_0 s 0 . Later in these notes, we will touch upon contexts where it is easier to reason in terms of distributions over trajectories than in terms of behavioral policies, typically when the objective function is not continuous, such as for mean-payoff objectives.
Adjacent policies. When a policy π ∈ Π S \pi\in\Pi_S π ∈ Π S is not uniformly optimal, we can always find a policy π ′ ∈ Π S \pi'\in\Pi_S π ′ ∈ Π S that coincides with π \pi π at all states but one state s ˉ ∈ S \bar{s}\in S s ˉ ∈ S , is deterministic at s ˉ \bar{s} s ˉ , and such that v π ′ , P ≥ v π , P v^{\pi',P}\geq v^{\pi,P} v π ′ , P ≥ v π , P with v π ′ , P ( s ˉ ) > v π , P ( s ˉ ) v^{\pi',P}(\bar{s})>v^{\pi,P}(\bar{s}) v π ′ , P ( s ˉ ) > v π , P ( s ˉ ) . Therefore, we can “strictly improve” upon π \pi π by changing the action chosen at only one state; this is the basis of a “lazy” version of policy iteration, which updates the running policy at only one state (instead of potentially changing the running policy entirely as in Algorithm 2). We will prove a stronger (robust) version of this result in Lemma 3.3. It is worth emphasizing that the same statement for return (instead of value functions) is false! That is, if a policy π ∈ Π S \pi\in\Pi_S π ∈ Π S is such that R ( π , P ) < max π ′ ∈ Π S R ( π ′ , P ) R(\pi,P)<\max_{\pi'\in\Pi_S}R(\pi',P) R ( π , P ) < max π ′ ∈ Π S R ( π ′ , P ) , to improve R ( π , P ) R(\pi,P) R ( π , P ) we may need to change the actions in at least two states, as shown in the next example.
Example 1.3. Consider the MDP instance described in Figure 1.5, and assume that we start at s 0 s_0 s 0 and that π \pi π chooses action a 2 a_2 a 2 at s 0 s_0 s 0 and s 1 s_1 s 1 . Then if we change only the action chosen at s 0 s_0 s 0 or only the action chosen at s 1 s_1 s 1 we would still get a return of 0. Here to get a positive return we need to take a 1 a_1 a 1 in both s 0 s_0 s 0 and $s_1.
Choice of rewards. We have introduced MDPs with instantaneous rewards r ∈ R S × A × S r\in\mathbb{R}^{S\times A\times S} r ∈ R S × A × S indexed by the current state-action pair ( s , a ) (s,a) ( s , a ) and the next state s ′ s' s ′ . For nominal MDPs, it is without loss of generality to assume that the rewards only depend on the current state-action pair ( s , a ) (s,a) ( s , a ) and not the next state s ′ s' s ′ , simply by replacing the rewards by their expectation over the next states. There are two ways to prove this fact. The first way is to notice that
T P v ( s ) = max a ∈ A P ( s , a ) T ( r ( s , a ) + γ v ) = max a ∈ A P ( s , a ) T r ( s , a ) + γ P ( s , a ) T v = max a ∈ A r ~ ( s , a ) + γ P ( s , a ) T v \begin{aligned}
T^P v(s) &= \max_{a\in A} P(s,a)^\mathsf{T}(r(s,a)+\gamma v)\\
&= \max_{a\in A} P(s,a)^\mathsf{T}r(s,a)+\gamma P(s,a)^\mathsf{T}v\\
&= \max_{a\in A} \tilde{r}(s,a)+\gamma P(s,a)^\mathsf{T}v
\end{aligned} T P v ( s ) = a ∈ A max P ( s , a ) T ( r ( s , a ) + γ v ) = a ∈ A max P ( s , a ) T r ( s , a ) + γ P ( s , a ) T v = a ∈ A max r ~ ( s , a ) + γ P ( s , a ) T v
where in the last line we introduce r ~ ∈ R S × A \tilde{r}\in\mathbb{R}^{S\times A} r ~ ∈ R S × A defined as r ~ ( s , a ) : = P ( s , a ) T r ( s , a ) \tilde{r}(s,a):=P(s,a)^\mathsf{T}r(s,a) r ~ ( s , a ) := P ( s , a ) T r ( s , a ) . Starting from an MDP with instantaneous reward r ∈ R S × A × S r\in\mathbb{R}^{S\times A\times S} r ∈ R S × A × S , we can replace it by an MDP with the same optimal policies but instantaneous rewards r ~ \tilde{r} r ~ that are independent of the next state.
Figure 1.5. MDP instance for Example 1.3. The transition probabilities and rewards are indicated beside each arc. Absorbing states are represented by squares.
A second proof is by augmenting the set of states with intermediate “fictitious states” ( s , a , s ′ ) ∈ S × A × S (s, a, s') \in S \times A \times S ( s , a , s ′ ) ∈ S × A × S ; rewards are obtained only at the “state” ( s , a , s ′ ) (s, a, s') ( s , a , s ′ ) , so that in this model rewards only depend on the current state (and not even on the action chosen), and transitions are chosen to make this new model consistent with the original MDP; that is, s → ( s , a , s ′ ) s \to(s, a, s') s → ( s , a , s ′ ) with probability P ( s , a , s ′ ) P(s, a, s') P ( s , a , s ′ ) when choosing action a ∈ A a \in A a ∈ A in state s s s . One also needs to replace the discount factor γ \gamma γ by γ \sqrt{\gamma} γ (since rewards are obtained every other period). It is worth noting here that for robust MDPs we cannot make such reductions in our choices of rewards in all generality, as will be made clear in Section 4.9. We also note that this reduction may not be very practical as it introduces many dummy states, which can slow down the algorithms to compute optimal policies.
Main takeaways
We summarize the main takeaways for this chapter below:
Return reformulation. The discounted return can be modeled via two complementary approaches: one approach is based on value functions (3) and one approach is based on state-action frequencies (8) . The first approach leads to studying the fixed point of monotone contraction operations defined componentwise, known as Bellman operators ((11) and (14) ); the second approach makes the return linear in the frequencies.
Optimality equations and computation. An optimal policy can be chosen stationary, deterministic, and uniformly optimal, that is, optimal for all choices of initial distributions; it is characterized by the so-called Bellman equations, related to the fixed point of the Bellman operator (Theorem 1.1). One can then use Value Iteration, Policy Iteration, and linear programming (Algorithm 1, Algorithm 2, (17) ) to compute (approximate) fixed points of the Bellman operators.
Exercises
The following is a list of observations/facts that help understand MDPs beyond what is presented above. They are presented in the same order as the objects introduced above.
Definitions of the discounted return. Recall that we have defined the discounted return as (1.1).
(a) Justify that the limit exists in lim T → + ∞ E π , P [ ∑ t = 0 T γ t r ( s t , a t , s t + 1 ) ] \lim_{T \to+\infty} \mathbb{E}^{\pi,P}\left[\sum_{t=0}^{T} \gamma^{t}r(s_t,a_t,s_{t+1})\right] lim T → + ∞ E π , P [ ∑ t = 0 T γ t r ( s t , a t , s t + 1 ) ] .
(b) Justify that
E π , P [ ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) ] = ∑ t = 0 + ∞ γ t E π , P [ r ( s t , a t , s t + 1 ) ] \mathbb{E}^{\pi,P}\left[\sum_{t=0}^{+\infty}\gamma^{t}r(s_t,a_t,s_{t+1})\right]=\sum_{t=0}^{+\infty}\gamma^{t}\mathbb{E}^{\pi,P}\left[r(s_t,a_t,s_{t+1})\right] E π , P [ t = 0 ∑ + ∞ γ t r ( s t , a t , s t + 1 ) ] = t = 0 ∑ + ∞ γ t E π , P [ r ( s t , a t , s t + 1 ) ]
and explain the different interpretations for the terms on each side of the above equation.
(c) What would change if we introduced state-dependent discount factors γ ( s ) ∈ [ 0 , 1 ) \gamma(s) \in[0,1) γ ( s ) ∈ [ 0 , 1 ) for each state s ∈ S s \in S s ∈ S ?
Supremum is attained. Give the topological arguments to claim that the supremum in (1.2) is always attained (over Π H \Pi_{\mathrm{H}} Π H ) (of course, do not use Theorem 1.1).
Compact heterogeneous action sets. Go through the main results in these notes and justify that Theorem 1.1 remains true when considering compact heterogeneous action sets, i.e., when the action set at each state s ∈ S s \in S s ∈ S is A s A_s A s , a compact action set (not necessarily finite), and that rewards and transition probabilities are continuous in the actions.
Operator properties. Let F : R S → R S F:\mathbb{R}^{S}\to\mathbb{R}^{S} F : R S → R S be an operator.
(a) Relaxed value iteration / Krasnoselskii-Mann. Let F α = α I + ( 1 − α ) F F_\alpha=\alpha I+(1-\alpha)F F α = α I + ( 1 − α ) F for α ∈ R \alpha\in\mathbb{R} α ∈ R . Assume that F F F is a monotone contraction. Show that F α F_\alpha F α is a monotone contraction for α ∈ [ 0 , 1 ) \alpha\in[0,1) α ∈ [ 0 , 1 ) . What happens for α \alpha α outside this interval? Compare the fixed point of F α F_\alpha F α with that of F F F . Can we obtain better convergence rates for Algorithm 1 by considering F α F_\alpha F α instead of F F F ?
(b) Crandall-Tartar theorem. We say that F F F is additively homogeneous if F ( x + λ 1 S ) = F x + λ 1 S , ∀ λ ∈ R , ∀ x ∈ R S F(x+\lambda1_S)=Fx+\lambda1_S,\ \forall\lambda\in\mathbb{R},\ \forall x\in\mathbb{R}^{S} F ( x + λ 1 S ) = F x + λ 1 S , ∀ λ ∈ R , ∀ x ∈ R S , with 1 S = ( 1 , … , 1 ) ∈ R S 1_S=(1,\ldots,1)\in\mathbb{R}^{S} 1 S = ( 1 , … , 1 ) ∈ R S . We say F F F is non-expansive if ∥ F x − F y ∥ ∞ ≤ ∥ x − y ∥ ∞ \lVert Fx-Fy\rVert_\infty\leq\lVert x-y\rVert_\infty ∥ F x − F y ∥ ∞ ≤ ∥ x − y ∥ ∞ . Note F F F non-expansive implies that v ↦ F ( γ v ) v\mapsto F(\gamma v) v ↦ F ( γ v ) is a contraction (hence non-expansiveness is a natural notion to study in our area).
Show the following theorem: assume that F F F is additively homogeneous. Then F F F is monotone if and only if F F F is non-expansive.
Hint: use the following characterization of the ℓ ∞ \ell_\infty ℓ ∞ -norm for x ∈ R S x\in\mathbb{R}^{S} x ∈ R S :
∥ x ∥ ∞ = inf { λ ∈ R ∣ − λ ≤ x ( s ) ≤ λ , ∀ s ∈ S } . \lVert x\rVert_\infty=\inf\{\lambda\in\mathbb{R}\mid-\lambda\leq x(s)\leq\lambda,\ \forall s\in S\}. ∥ x ∥ ∞ = inf { λ ∈ R ∣ − λ ≤ x ( s ) ≤ λ , ∀ s ∈ S } .
About discount factors. Justify that we can interpret the quantity 1 − γ 1-\gamma 1 − γ as a probability of “quitting the system with reward 0 onward” (this is the original model for MDPs back to the 1950s). Can the results in this section be adapted to state-dependent discount factors?
About the contraction lemma.
(a) Let F F F be a monotone contraction with fixed point v P v^{P} v P . Show that v P v^{P} v P is the maximum of the sub-solutions:
{ v P } = arg max { ∑ s ∈ S v ( s ) ∣ v ≤ F v , v ∈ R S } \{v^{P}\} = \operatorname*{arg\,max}\left\{\sum_{s\in S}v(s)\mid v\leq Fv,v\in\mathbb{R}^{S}\right\} { v P } = arg max { s ∈ S ∑ v ( s ) ∣ v ≤ F v , v ∈ R S }
Is this a better formulation than the minimization program already given in the contraction lemma?
Show that min { p 0 T v ∣ v ≥ F v , v ∈ R S } = p 0 T v P \min\{p_{0}^{\mathsf{T}}v\mid v\geq Fv,v\in\mathbb{R}^{S}\}=p_{0}^{\mathsf{T}}v^{P} min { p 0 T v ∣ v ≥ F v , v ∈ R S } = p 0 T v P for any p 0 ∈ Δ ( S ) p_{0}\in\Delta(S) p 0 ∈ Δ ( S ) and that v P v^{P} v P is the unique minimizer if p 0 p_{0} p 0 has full support. What are the other minimizers otherwise?
(b) Let π ∈ Π S \pi\in\Pi_{S} π ∈ Π S . Write the contraction lemma applied to the monotone contraction T π , P T^{\pi,P} T π , P . Does this yield a “fast” algorithm for computing the fixed point v π , P v^{\pi,P} v π , P ?
References
MDPs date back to the late 1940s-1950s where they were used to model operations research problems (typically how to manage inventories over time; this particular example comes back very often in papers from that period). There are so many seminal references from the 1950s to early 1980s that I do not attempt to review all of them here. For historical background on MDPs I refer to Chapter 1 and Chapter 2 in [19 ] , and the “Bibliographic Remarks” at the end of each chapter in [60 ] . For a good modern introduction to MDPs I recommend the lecture notes by Joon Kwon [42 ] , which are available on GitHub. The lecture notes in [38 ] provide a comprehensive overview of MDPs. Bellman’s book [4 ] and Howard’s book [33 ] played an important role and Bellman is credited with coining the term “dynamic programming”. Shapley’s work on stochastic games [73 ] introduced a two-player version of MDPs. [14 ] introduced the linear programming formulation for MDPs with discounted objectives (both the primal and the dual formulations; at the same time [15 , 50 ] introduced a linear programming formulation for irreducible MDPs with a mean-payoff objective). We refer to [83 , 72 ] for a refined complexity analysis of policy iteration. The contraction lemma we give here is from [58 ] but it is already present in [14 ] . The Crandall-Tartar theorem can be found in [12 ] . Kuhn’s theorem is from [41 ] .
Robust MDPs with general uncertainty sets
This chapter introduces the concept of uncertainty set, interpreted as a set of all plausible transition probabilities (e.g., consistent with some historical dataset) and describes the robust MDP problem. The tractability properties of worst-case kernels and optimal policies are then derived for general uncertainty sets.
Main model and assumptions
In many real-world setups, we may face some uncertainty on the “true” value of the transition parameters. This is the case for instance if we only have access to a finite dataset of historical transitions. What should we do in this case? A natural approach would be to compute P ^ \hat{P} P ^ the empirical average of the observed transitions and compute an optimal policy for the MDP with transition probabilities P ^ \hat{P} P ^ . However, we know that the value functions may depend on the entries of P ^ \hat{P} P ^ in a rather complicated manner (see the formula in (5) ), and it is unclear if a policy that performs well for P ^ \hat{P} P ^ would perform well for other transition probabilities, even close to P ^ \hat{P} P ^ .
Hence in our search for a “good” policy we may want to incorporate additional information, for instance we may want to take into account the confidence intervals that one can construct around the empirical average P ^ \hat{P} P ^ , and search for a policy with “good” performance for any realization of the transition probabilities in these confidence intervals. For instance we may want to consider the set of transition probabilities that are ϵ \epsilon ϵ -close to P ^ \hat{P} P ^ :
{ P ∈ Δ ( S ) S × A ∣ ∣ P ( s , a , s ′ ) − P ^ ( s , a , s ′ ) ∣ ≤ ϵ , ∀ ( s , a , s ′ ) ∈ S × A × S } \left\{P \in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} \mid\left|P(s,a,s')-\hat{P}(s,a,s')\right| \le\epsilon,\ \forall\ (s,a,s')\in\mathcal{S}\times\mathcal{A}\times\mathcal{S}\right\} { P ∈ Δ ( S ) S × A ∣ P ( s , a , s ′ ) − P ^ ( s , a , s ′ ) ≤ ϵ , ∀ ( s , a , s ′ ) ∈ S × A × S }
and we could adjust the radius ϵ \epsilon ϵ to each triplet ( s , a , s ′ ) (s,a,s') ( s , a , s ′ ) depending on the number of observations for the transition ( s , a ) → s ′ (s,a)\to s' ( s , a ) → s ′ .
More generally, we may want to construct a set P ⊂ Δ ( S ) S × A \mathcal{P}\subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A of feasible transition probabilities, called the uncertainty set , which captures all the plausible realizations of the transitions (plausible given the dataset we have). The robust MDP problem is then to solve the following optimization problem:
sup π ∈ Π H inf P ∈ P R ( π , P ) (20) \sup_{\pi\in\Pi_H}\inf_{P\in\mathcal{P}}R(\pi,P) \tag*{(20)} π ∈ Π H sup P ∈ P inf R ( π , P ) (20)
An optimal solution of (20) (provided it exists) guarantees the highest worst-case return possible across all realizations of P ∈ P P \in\mathcal{P} P ∈ P ; a uniformly optimal policy is optimal for all initial distributions. This robust model is also related to percentile guarantees (see § 2.5 ). We emphasize that there are many other models that can be used to cope with parameter uncertainty (e.g. multi-objective models, partially observable MDPs, percentile optimization, distributionally robust MDPs, etc.), many of these problems having a deep connection to robust MDPs as we will show in Chapter 4.
For π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H , the policy evaluation problem asks to evaluate the worst-case return of π \pi π across all transition probabilities in P \mathcal{P} P and is defined as follows:
inf P ∈ P R ( π , P ) (21) \inf_{P \in\mathcal{P}} R(\pi,P) \tag*{(21)} P ∈ P inf R ( π , P ) (21)
It is common to interpret (20) as a “game” (loosely defined for now) where, before the dynamics start, we first choose a policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H and an “adversary” chooses the transition probabilities P ∈ P P \in\mathcal{P} P ∈ P ; the adversary is also sometimes called “Nature”. The game is zero-sum, in that we are maximizing the objective and the adversary is minimizing it. Note that there is an asymmetry in the decisions of the first player (choosing π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H , i.e. eventually changing their decision rules at each period, depending on the realization of the current trajectory) and the second player (choosing P ∈ P P \in\mathcal{P} P ∈ P which is independent of time). We will treat the case of time-varying transition probabilities in § 4.4 .
The rest of this chapter is devoted to understanding the fundamental properties (existence and characterization) of the optimal solutions in the robust MDP problem (20) and the policy evaluation problem (21) . In this chapter, we limit the assumptions to what is strictly necessary for stating our results (in particular, many of the existence results for optimal policies don’t even need compactness of P \mathcal{P} P ); in the next chapter, we’ll make additional assumptions to simplify the proofs. The topological results needed below (e.g. Weierstrass’ theorem or Tykhonoff’s theorem) are given and explained in Appendix § 7 .
Remark 2.1. In these notes I mostly focus on the case of uncertain transition probabilities and known rewards. The case of uncertain rewards is somewhat easier (at least for the main results) and is treated in Chapter 4.
Existence of optimal solutions
In this section we show that under mild technical conditions, worst-case kernels and optimal policies exist for the robust MDP problem. Here we’ll always assume that P ≠ ∅ \mathcal{P} \ne\varnothing P = ∅ . We start by studying the continuity properties of the return.
Proposition 2.1. Let P ∈ Δ ( S ) S × A P \in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ∈ Δ ( S ) S × A .
The function ( π , P ) ↦ R ( π , P ) (\pi,P) \mapsto R(\pi,P) ( π , P ) ↦ R ( π , P ) is continuous over Π H × P \Pi_{\mathrm{H}} \times\mathcal{P} Π H × P .
The function π ↦ inf P ∈ P R ( π , P ) \pi\mapsto\inf_{P \in\mathcal{P}} R(\pi,P) π ↦ inf P ∈ P R ( π , P ) is upper semicontinuous over Π H \Pi_{\mathrm{H}} Π H .
Proof. 1. Let r ∞ = max s , a , s ′ ∣ r ( s , a , s ′ ) ∣ r_{\infty}=\max_{s,a,s'}\lvert r(s,a,s')\rvert r ∞ = max s , a , s ′ ∣ r ( s , a , s ′ )∣ . We have r ∞ < + ∞ r_{\infty}<+\infty r ∞ < + ∞ since S \mathcal{S} S and A \mathcal{A} A are finite sets. For H ∈ N H \in\mathbb{N} H ∈ N , consider the truncated return R H ( π , P ) = E π , P [ ∑ t = 0 H − 1 γ t r ( s t , a t , s t + 1 ) ] R_H(\pi,P)=\mathbb{E}^{\pi,P}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t,s_{t+1})\right] R H ( π , P ) = E π , P [ ∑ t = 0 H − 1 γ t r ( s t , a t , s t + 1 ) ] . The function ( π , P ) ↦ R H ( π , P ) (\pi, P) \mapsto R_H(\pi, P) ( π , P ) ↦ R H ( π , P ) is a polynomial in finitely many entries of π ∈ Π H \pi\in\Pi_H π ∈ Π H and P ∈ P P \in\mathcal{P} P ∈ P since it depends only on the distributions induced by π , P \pi, P π , P over the first H + 1 H+1 H + 1 states visited and first H H H actions chosen. Moreover we have
∣ R ( π , P ) − R H ( π , P ) ∣ ≤ r ∞ γ H 1 − γ , ∀ ( π , P ) ∈ Π H × P . \lvert R(\pi, P) - R_H(\pi, P) \rvert\le r_{\infty} \frac{\gamma^H}{1-\gamma}, \quad\forall\,(\pi, P) \in\Pi_H \times\mathcal{P}. ∣ R ( π , P ) − R H ( π , P )∣ ≤ r ∞ 1 − γ γ H , ∀ ( π , P ) ∈ Π H × P .
Hence ( π , P ) ↦ R ( π , P ) (\pi, P) \mapsto R(\pi, P) ( π , P ) ↦ R ( π , P ) is the uniform limit, as H → + ∞ H \to+\infty H → + ∞ , of the continuous functions ( π , P ) ↦ R H ( π , P ) (\pi, P) \mapsto R_H(\pi, P) ( π , P ) ↦ R H ( π , P ) , and therefore it is continuous.
We first show that the policy evaluation problem (21) always has a solution when P \mathcal{P} P is compact, i.e., a worst-case transition kernel always exists.
Proposition 2.2. Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be a compact set and π ∈ Π H \pi\in\Pi_H π ∈ Π H . Then inf P ∈ P R ( π , P ) \inf_{P\in\mathcal{P}} R(\pi, P) inf P ∈ P R ( π , P ) is attained.
Proof. Let π ∈ Π H \pi\in\Pi_H π ∈ Π H . By assumption P \mathcal{P} P is compact, and from the first statement in Proposition 2.1 the function P ↦ R ( π , P ) P \mapsto R(\pi, P) P ↦ R ( π , P ) is continuous. Weierstrass’ theorem states that a continuous function on a (nonempty) compact set attains its minimum, which yields the proposition.
This enables us to talk later of “worst-case transition probabilities” given a fixed policy π ∈ Π H \pi\in\Pi_H π ∈ Π H . It is worth noting that in general there may be several different worst-case transition probabilities, leading to different value functions (see Proposition 2.9 ). Regarding the existence of optimal policies, we can prove the following two positive results. First, we note that an optimal robust policy always exists, i.e. the supremum in (20) is always attained.
Proposition 2.3. Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A . An optimal policy (eventually in Π H \Pi_H Π H ) always exists for the robust MDP problem (20) .
Proof. The set Π H \Pi_H Π H is the product over all finite trajectories h ∈ H h \in\mathcal{H} h ∈ H of the compact set Δ ( A ) \Delta(\mathcal{A}) Δ ( A ) . Note that H \mathcal{H} H is countable since S , A \mathcal{S}, \mathcal{A} S , A are finite sets. From Tykhonoff’s theorem, Π H \Pi_H Π H is a compact set for the product topology (see Remark 2.2). Additionally, from the second statement in Proposition 2.1 we have that π ↦ inf P ∈ P R ( π , P ) \pi\mapsto\inf_{P\in\mathcal{P}} R(\pi, P) π ↦ inf P ∈ P R ( π , P ) is an upper semicontinuous function. Weierstrass’ theorem gives that it attains its supremum over a compact set.
Note that the above proposition does not provide any characterization of optimal policies (e.g., we do not know if they can be chosen stationary) and does not require P \mathcal{P} P to be compact (we optimize over Π H \Pi_H Π H in (20) ). The next proposition shows the existence of a policy that is optimal across all choices of deterministic initial distributions (note the difference with uniform optimality, which requires to be optimal across all choices of initial distributions).
Proposition 2.4. Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A . There exists a policy π ∈ Π H \pi\in\Pi_H π ∈ Π H that is optimal in (20) for all choices of deterministic initial distribution.
Proof. The proof follows by patching together optimal policies for each Dirac distribution. For s ∈ S s \in\mathcal{S} s ∈ S let us consider π ⋆ , s ∈ Π H \pi^{\star,s} \in\Pi_{\mathrm{H}} π ⋆ , s ∈ Π H an optimal policy (possibly history-dependent) when starting in state s s s ; the existence of such a policy follows from Proposition 2.3 . Now define a single policy π ˉ ∈ Π H \bar{\pi} \in\Pi_{\mathrm{H}} π ˉ ∈ Π H that coincides with π ⋆ , s \pi^{\star,s} π ⋆ , s along trajectories starting in s ∈ S s \in\mathcal{S} s ∈ S . By definition π ˉ \bar{\pi} π ˉ is optimal across all deterministic initial distributions.
Properties of optimal solutions
We now analyze the properties of optimal policies for the robust MDP problem (the existence of which we proved in Proposition 2.3 ). In all generality, we first note that we may need non-stationary optimal policies. This is in contrast to the case of nominal MDPs studied in Chapter 1.
Proposition 2.5. For a general uncertainty set P \mathcal{P} P , an optimal policy for the robust MDP problem (2.1) may need to be history-dependent (non-stationary and non-Markovian).
The proof of Proposition 2.5 follows from analyzing the optimal policies in the MDP instance from Example 1.1.
Example 1.1 continued. Consider the MDP instance from Figure 1.2 (starting in s 0 s_0 s 0 ) and consider the uncertainty set of cardinality 2, corresponding to the transition matrices induced by the choice p = 0 p = 0 p = 0 or by the choice p = 1 p = 1 p = 1 . We first compute the payoff of the best stationary policies. A stationary policy π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S is determined by x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] the probability that π \pi π chooses action a 1 a_1 a 1 in state s 0 s_0 s 0 , and we computed the discounted return of such a stationary policy as (for fixed p p p ) 1 1 − γ 2 ( x p + ( 1 − x ) ( 1 − p ) ) \frac{1}{1-\gamma^2}(xp + (1-x)(1-p)) 1 − γ 2 1 ( x p + ( 1 − x ) ( 1 − p )) . Hence, the best stationary policy solves
max x ∈ [ 0 , 1 ] min p ∈ { 0 , 1 } x p + ( 1 − x ) ( 1 − p ) = max x ∈ [ 0 , 1 ] min { x , 1 − x } . \max_{x \in[0,1]} \min_{p \in\{0,1\}} xp + (1-x)(1-p) = \max_{x \in[0,1]} \min\{x,1-x\}. x ∈ [ 0 , 1 ] max p ∈ { 0 , 1 } min x p + ( 1 − x ) ( 1 − p ) = x ∈ [ 0 , 1 ] max min { x , 1 − x } .
From this , we get that the best stationary policy chooses x = 1 / 2 x = 1/2 x = 1/2 , for a discounted return of 1 2 1 1 − γ 2 \frac{1}{2}\frac{1}{1-\gamma^2} 2 1 1 − γ 2 1 . Now consider the following choice of non-stationary policy. We choose a 1 a_1 a 1 with probability 1 / 2 1/2 1/2 at period 0 (to match what the best stationary policy does). Then we get back to state s 0 s_0 s 0 at period 2, and what we do depends on the observed trajectory so far (below we omit the actions chosen at G G G and B B B since they are irrelevant):
If we observe ( s 0 , a 1 , G , s 0 ) (s_0,a_1,G,s_0) ( s 0 , a 1 , G , s 0 ) : we play a 1 a_1 a 1 forever onward.
If we observe ( s 0 , a 1 , B , s 0 ) (s_0,a_1,B,s_0) ( s 0 , a 1 , B , s 0 ) : we play a 2 a_2 a 2 forever onward.
If we observe ( s 0 , a 2 , G , s 0 ) (s_0,a_2,G,s_0) ( s 0 , a 2 , G , s 0 ) : we play a 2 a_2 a 2 forever onward.
If we observe ( s 0 , a 2 , B , s 0 ) (s_0, a_2, B, s_0) ( s 0 , a 2 , B , s 0 ) : we play a 1 a_1 a 1 forever onward.
In short, the decisions outlined above check whether we played optimally during the first period (i.e. whether we reached G G G instead of B B B at period t = 1 t = 1 t = 1 ) and adjust the actions chosen next accordingly. This ensures that we get the same reward as the best stationary policy in the first period (t = 0 t = 0 t = 0 ), and then we’ll always visit the good state G G G after s 0 s_0 s 0 , so we get the maximum reward possible starting from period t = 2 t = 2 t = 2 , i.e. we get a discounted return of γ ( 1 2 + γ 2 1 − γ 2 ) \gamma(\frac{1}{2} + \frac{\gamma^2}{1-\gamma^2}) γ ( 2 1 + 1 − γ 2 γ 2 ) . Therefore this non-stationary policy strictly outperforms the best stationary policy. Note that the worst-case transition probabilities here are attained by both choices of p = 0 p = 0 p = 0 and p = 1 p = 1 p = 1 , since at period t = 0 t = 0 t = 0 the choice of p p p doesn’t matter (we play uniformly) and for later periods we always visit G G G instead of B B B no matter the choice of p p p .
Intuitively, non-stationary policies beat stationary policies in the above example because they can exploit information revealed over time (in this example, we can determine whether the adversary chose p = 0 p = 0 p = 0 or p = 1 p = 1 p = 1 by observing the first transition, i.e., after observing the realized state at period 1). Note that in the example above, we need genuinely history-dependent policies (the optimal decisions at any period t ≥ 1 t \ge1 t ≥ 1 depend on the realized transitions at time t = 0 t = 0 t = 0 so optimal policies are non-Markovian). It is also worth emphasizing that if we are only interested in ϵ \epsilon ϵ -optimal policies, we can play eventually stationary policies, i.e., policies that become stationary for large enough periods.
Proposition 2.6. Let ϵ > 0 \epsilon> 0 ϵ > 0 . We can find an ϵ \epsilon ϵ -optimal policy that is eventually stationary.
Proof. Let ϵ > 0 \epsilon> 0 ϵ > 0 . We can just truncate the return up to period H − 1 H - 1 H − 1 such that the remaining discounted return obtained after the first H H H periods is always smaller than ϵ \epsilon ϵ . We can find such H ∈ N H \in\mathbb{N} H ∈ N because the rewards are bounded and discounted by some γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) . Now consider the policy that plays like an optimal (potentially history-dependent) policy for the first H H H periods, then always play action a 1 a_1 a 1 afterwards (or any other stationary policy). By construction this is an ϵ \epsilon ϵ -optimal policy.
Note that from the above construction, as ϵ \epsilon ϵ goes to 0, the period at which we can start to play stationary while guaranteeing ϵ \epsilon ϵ -optimality increases to + ∞ +\infty + ∞ , which is not very practical. Additionally, there need not exist a uniformly optimal policy.
Proposition 2.7. For a general uncertainty set P \mathcal{P} P , there may not exist any uniformly optimal policy that is optimal for the robust MDP problem (20) , even among history-dependent policies.
The proof is based on an MDP close to Example 1.2.
Example 1.2 continued. *Consider an MDP instance with the same transition probabilities as in Figure 1.3 , and take p = p ′ p = p' p = p ′ , p ∈ [ 0 , 1 ] p \in[0,1] p ∈ [ 0 , 1 ] . For the rewards, choose the following intermediate rewards: for s ′ ∈ { s 1 , s 2 } s' \in\{s_1,s_2\} s ′ ∈ { s 1 , s 2 } , r ( s 0 , a 1 , s 2 ) = 1 r(s_0,a_1,s_2) = 1 r ( s 0 , a 1 , s 2 ) = 1 , r ( s 0 , a 2 , s ′ ) = − 1 r(s_0,a_2,s') = -1 r ( s 0 , a 2 , s ′ ) = − 1 , r ( s 1 , a 2 , s 3 ) = − 1 r(s_1,a_2,s_3) = -1 r ( s 1 , a 2 , s 3 ) = − 1 , r ( s 1 , a 2 , s 4 ) = 1 r(s_1,a_2,s_4) = 1 r ( s 1 , a 2 , s 4 ) = 1 , and all other rewards are 0. We choose γ = 0 \gamma= 0 γ = 0 so that only the action chosen at period t = 0 t = 0 t = 0 impacts the return (and in particular when starting from s 0 s_0 s 0 or s 1 s_1 s 1 the rewards obtained in the absorbing states s 2 , s 3 , s 4 s_2,s_3,s_4 s 2 , s 3 , s 4 are irrelevant). Similarly as in Example 1.2, stationary policies π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S can be parametrized by x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] the probability to play a 1 a_1 a 1 in s 0 s_0 s 0 and y ∈ [ 0 , 1 ] y \in[0,1] y ∈ [ 0 , 1 ] the probability to play a 1 a_1 a 1 in s 1 s_1 s 1 . Because γ = 0 \gamma= 0 γ = 0 , stationary policies and history-dependent policies induce the same set of returns, and we can focus on searching for optimal policies among stationary ones.
Let us compute the set of all optimal policies when starting in s 1 s_1 s 1 with probability 1. Action a 1 a_1 a 1 gives a return of 0 while action a 2 a_2 a 2 gives a return of ( 1 − p ) ⋅ ( − 1 ) + p ⋅ 1 = 2 p − 1 (1-p)\cdot(-1)+p\cdot1=2p-1 ( 1 − p ) ⋅ ( − 1 ) + p ⋅ 1 = 2 p − 1 . Hence the worst-case return of π \pi π is inf p ∈ [ 0 , 1 ] ( 1 − y ) ( 2 p − 1 ) = y − 1 \inf_{p\in[0,1]}(1-y)(2p-1)=y-1 inf p ∈ [ 0 , 1 ] ( 1 − y ) ( 2 p − 1 ) = y − 1 attained at p = 0 p=0 p = 0 so that optimality requires y = 1 y=1 y = 1 . Similarly it is easy to see that starting in s 0 s_0 s 0 the decision maker would choose a 1 a_1 a 1 and the adversary would choose p = 1 p=1 p = 1 .
We now compute the set of all optimal policies when starting in s 0 s_0 s 0 or in s 1 s_1 s 1 with probability 1 / 2 1/2 1/2 . A few lines of algebra show that
R ( π , P ) = 1 2 ( − 1 + x ( 2 − p ) + ( 1 − y ) ( 2 p − 1 ) ) R(\pi,P)=\frac{1}{2}\left(-1+x(2-p)+(1-y)(2p-1)\right) R ( π , P ) = 2 1 ( − 1 + x ( 2 − p ) + ( 1 − y ) ( 2 p − 1 ) )
This expression is affine in p ∈ [ 0 , 1 ] p\in[0,1] p ∈ [ 0 , 1 ] hence the worst-case over [ 0 , 1 ] [0,1] [ 0 , 1 ] is always attained in { 0 , 1 } \{0,1\} { 0 , 1 } . Therefore
inf P ∈ P R ( π , P ) = min { x + y 2 − 1 , x − y 2 } \inf_{P\in\mathcal{P}}R(\pi,P)=\min\left\{x+\frac{y}{2}-1,\frac{x-y}{2}\right\} P ∈ P inf R ( π , P ) = min { x + 2 y − 1 , 2 x − y }
This expression always increases in x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] hence a maximizer must take x = 1 x=1 x = 1 for which the objective becomes 1 2 min { y , 1 − y } \frac{1}{2}\min\{y,1-y\} 2 1 min { y , 1 − y } , maximized at y = 1 / 2 y=1/2 y = 1/2 . Thus the sets of optimal policies when starting in s 1 s_1 s 1 with probability 1 or in { s 0 , s 1 } \{s_0,s_1\} { s 0 , s 1 } uniformly are disjoint, and a uniformly optimal policy cannot exist in this example.
The previous proposition should be contrasted with Proposition 2.4 , which shows that we can find policies that are optimal across all deterministic initial distributions; it is the fact that initial distributions may be randomized that prevents the existence of uniformly optimal policies in all generality.
The key in the above example is that the same parameter p ∈ [ 0 , 1 ] p\in[0,1] p ∈ [ 0 , 1 ] appears in both the transitions out of the two states s 0 s_0 s 0 and s 1 s_1 s 1 . From s 0 s_0 s 0 we want to choose a 1 a_1 a 1 and the adversary wants to choose p = 1 p=1 p = 1 ; for s 1 s_1 s 1 we want to choose a 1 a_1 a 1 and the adversary wants to choose p = 0 p=0 p = 0 . But if we start in s 1 s_1 s 1 or s 0 s_0 s 0 , each with probability 1 / 2 1/2 1/2 , then the adversary has “conflicting” worst-case kernels from these two states (p = 0 p=0 p = 0 from s 1 s_1 s 1 and p = 1 p=1 p = 1 from s 0 s_0 s 0 ), which we can exploit with the choice y = 1 / 2 y=1/2 y = 1/2 . It is perhaps also worth noting that the previous example shows, in passing, that we may need randomized (non-deterministic) policies in general (this is the case when starting in s 0 s_0 s 0 or s 1 s_1 s 1 with probability 1 / 2 1/2 1/2 ).
An additional potential hiccup is that optimal solutions of both (20) and (21) may have irrational entries, even when all the inputs (rewards, uncertainty set, discount factor) have rational (or even integer) entries.
Proposition 2.8. There exists a robust MDP with two states and two actions, binary rewards, such that for a choice of discount factor γ = 1 / 2 \gamma=1/2 γ = 1/2 and P \mathcal{P} P a convex compact polytope with constraints described by integers, every stationary optimal policy has irrational entries.
The proof is based on the following example.
Example 2.1. Consider an MDP instance with states s 0 , s 1 s_0,s_1 s 0 , s 1 and actions a 1 , a 2 a_1,a_2 a 1 , a 2 , represented in Figure 2.1 . For now we’ll assume that an optimal policy can be chosen stationary (this
Figure 2.1. MDP instance for Example 2.1. The transition probabilities and rewards are indicated beside each arc.
will be justified in the next chapter in Theorem 3.2. Let x = π ( s 0 , a 1 ) ∈ [ 0 , 1 ] x = \pi(s_0, a_1) \in[0,1] x = π ( s 0 , a 1 ) ∈ [ 0 , 1 ] and p ∈ [ 0 , 1 ] p \in[0,1] p ∈ [ 0 , 1 ] . Choose γ = 1 / 2 \gamma= 1/2 γ = 1/2 . The value function v x , p ( s 0 ) v^{x,p}(s_0) v x , p ( s 0 ) is
v x , p ( s 0 ) = 2 x p + 2 ( 1 − x ) ( 1 − p ) 1 + x p + ( 1 − x ) ( 1 − p ) . v^{x,p}(s_0) = 2 \frac{xp + 2(1-x)(1-p)}{1 + xp + (1-x)(1-p)}. v x , p ( s 0 ) = 2 1 + x p + ( 1 − x ) ( 1 − p ) x p + 2 ( 1 − x ) ( 1 − p ) .
This is linear-fractional in p p p so the worst-case over p ∈ [ 0 , 1 ] p \in[0,1] p ∈ [ 0 , 1 ] is attained at some p ∈ { 0 , 1 } p \in\{0,1\} p ∈ { 0 , 1 } . Consequently,
max x ∈ [ 0 , 1 ] min p ∈ [ 0 , 1 ] v x , p ( s 0 ) = max x ∈ [ 0 , 1 ] min { 4 ( 1 − x ) 2 − x , 2 x 1 + x } . \max_{x \in[0,1]} \min_{p \in[0,1]} v^{x,p}(s_0) = \max_{x \in[0,1]} \min\left\{\frac{4(1-x)}{2-x}, \frac{2x}{1+x}\right\}. x ∈ [ 0 , 1 ] max p ∈ [ 0 , 1 ] min v x , p ( s 0 ) = x ∈ [ 0 , 1 ] max min { 2 − x 4 ( 1 − x ) , 1 + x 2 x } .
This yields a maximizer x ⋆ ∈ [ 0 , 1 ] x^\star\in[0,1] x ⋆ ∈ [ 0 , 1 ] satisfying 4 ( 1 − x ⋆ ) ( 1 + x ⋆ ) = 2 x ⋆ ( 2 − x ⋆ ) 4(1-x^\star)(1+x^\star) = 2x^\star(2-x^\star) 4 ( 1 − x ⋆ ) ( 1 + x ⋆ ) = 2 x ⋆ ( 2 − x ⋆ ) so that x ⋆ 2 + 2 x ⋆ − 2 = 0 x^{\star2} + 2x^\star- 2 = 0 x ⋆ 2 + 2 x ⋆ − 2 = 0 i.e. x ⋆ = 3 − 1 ∈ ( 0 , 1 ) x^\star= \sqrt{3} - 1 \in(0,1) x ⋆ = 3 − 1 ∈ ( 0 , 1 ) , which has irrational entries.
This situation is in line with the case of Nash equilibria of finite games with more than two players, as shown by Nash in his original 1951 paper, and in contrast with the case of linear programs with rational inputs (where there exist some optimal solutions with rational entries) and MDPs: they can be solved by linear programming, as we saw in the previous chapter, hence we can find optimal policies with rational entries. It suggests that even representing optimal solutions for robust MDPs may be non-trivial. An additional difficulty comes from the fact that, for a fixed policy π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S , worst-case transition probabilities need not be unique; in the subparts of the set of states that are not visited (or whose visits are irrelevant), the adversary may choose any transition probabilities. However value functions are vector-valued functions of the policies and transition probabilities, indexed by the entire set of states. Consequently, worst-case value functions need not be unique (although of course the worst-case return is unique)! This is described in the next proposition.
Proposition 2.9. Let π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S . In general there may be multiple solutions in arg min P ∈ P R ( π , P ) \arg\min_{P \in\mathcal{P}} R(\pi,P) arg min P ∈ P R ( π , P ) , and for P 1 , P 2 ∈ arg min P ∈ P R ( π , P ) P_1, P_2 \in\arg\min_{P \in\mathcal{P}} R(\pi,P) P 1 , P 2 ∈ arg min P ∈ P R ( π , P ) we may have v π , P 1 ≠ v π , P 2 v^{\pi,P_1} \ne v^{\pi,P_2} v π , P 1 = v π , P 2 .
Proof. It is easy to observe this phenomenon for the robust MDP instance from Figure 1.3 , with an uncertainty set described by p = 0 p = 0 p = 0 , p ′ ∈ [ 0 , 1 ] p' \in[0,1] p ′ ∈ [ 0 , 1 ] . Say we start in s 0 s_0 s 0 with probability 1 1 1 and γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) . Take for instance π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S that chooses action a 1 a_1 a 1 with probability 1 1 1 in s 0 s_0 s 0 and with probability 3 / 4 3/4 3/4 in s 1 s_1 s 1 . With this choice of policy every kernel attains the worst-case return: R ( π , P ) = γ 2 ( 1 − γ ) R(\pi,P) = \frac{\gamma}{2(1-\gamma)} R ( π , P ) = 2 ( 1 − γ ) γ for any P ∈ P P \in\mathcal{P} P ∈ P but we get v π , P ( s 1 ) = γ ( 1 − 2 p ′ ) 2 ( 1 − γ ) v^{\pi,P}(s_1) = \frac{\gamma(1-2p')}{2(1-\gamma)} v π , P ( s 1 ) = 2 ( 1 − γ ) γ ( 1 − 2 p ′ ) .
Hence in general it is more proper to talk of the set of worst-case value functions of a policy: { v π , P ∣ P ∈ arg min P ∈ P R ( π , P ) } \{v^{\pi,P} \mid P \in\arg\min_{P \in\mathcal{P}} R(\pi,P)\} { v π , P ∣ P ∈ arg min P ∈ P R ( π , P )} , rather than “the” worst-case value function of a policy (except if we proved either (a) that there is a unique worst-case kernel, or (b) that all worst-case kernels yield the same value function). One may wonder whether there is a worst-case value function of particular interest, for instance, whether we can define a notion of “smallest” (i.e., worst) worst-case value function. We will see in the next chapter that for rectangular uncertainty sets, there is indeed a worst-case value function which is componentwise smaller than any other worst-case value function, and how to compute this particular vector.
We conclude this section by showing that even evaluating the worst-case return of a policy may be a difficult task1 ^{1} 1 .
Proposition 2.10. Let π ∈ Π S D \pi\in\Pi_{\mathrm{SD}} π ∈ Π SD . For a general convex compact uncertainty set P \mathcal{P} P and c ∈ R c \in\mathbb{R} c ∈ R , it may be NP-hard to decide if inf P ∈ P R ( π , P ) ≤ c \inf_{P \in\mathcal{P}} R(\pi,P) \le c inf P ∈ P R ( π , P ) ≤ c .
There exist several proofs of this statement (see references at the end of this chapter). I adapt here a proof by Joel Goh and coauthors, based on the Hamiltonian cycle problem.
Proof of Proposition 2.10. Given a directed graph G = ( V , E ) G = (V,E) G = ( V , E ) with non-empty edge set E ⊂ V × V E \subset V \times V E ⊂ V × V , the Hamiltonian cycle problem asks whether there exists a Hamiltonian cycle in G G G , i.e. a finite sequence of vertices ( s 0 , s 1 , … , s H , s 0 ) (s_0,s_1,\ldots,s_H,s_0) ( s 0 , s 1 , … , s H , s 0 ) where ( s i , s i + 1 ) ∈ E (s_i,s_{i+1}) \in E ( s i , s i + 1 ) ∈ E for i ∈ { 0 , … , H − 1 } i \in\{0,\ldots,H-1\} i ∈ { 0 , … , H − 1 } , ( s H , s 0 ) ∈ E (s_H,s_0) \in E ( s H , s 0 ) ∈ E , such that this cycle visits all states at most once: ∣ { s 0 , s 1 , … , s H } ∣ = ∣ V ∣ |\{s_0,s_1,\ldots,s_H\}| = |V| ∣ { s 0 , s 1 , … , s H } ∣ = ∣ V ∣ .
We can reduce this problem to comparing the optimal objective of a robust policy evaluation problem as follows. Choose S = V \mathcal{S} = V S = V and A = { a 1 } \mathcal{A} = \{a_1\} A = { a 1 } (i.e. there is no action to choose). Let π : S → { a 1 } \pi: \mathcal{S} \to\{a_1\} π : S → { a 1 } be the constant policy. Let us write S = { s 0 , … , s H } \mathcal{S} = \{s_0,\ldots,s_H\} S = { s 0 , … , s H } for H = ∣ V ∣ − 1 H = |V| - 1 H = ∣ V ∣ − 1 . The uncertainty set is the set of bistochastic matrices compatible with the set of edges E E E :
P = { P ∈ R S × S ∣ P ( s , s ′ ) ∈ R + , P 1 S = 1 S , P ⊤ 1 S = 1 S , P ( s , s ′ ) = 0 if ( s , s ′ ) ∉ E } . \mathcal{P} = \{P \in\mathbb{R}^{\mathcal{S} \times\mathcal{S}} \mid P(s,s') \in\mathbb{R}_{+}, P\mathbf{1}_{\mathcal{S}} = \mathbf{1}_{\mathcal{S}}, P^{\top}\mathbf{1}_{\mathcal{S}} = \mathbf{1}_{\mathcal{S}}, P(s,s') = 0 \text{ if } (s,s') \notin E\}. P = { P ∈ R S × S ∣ P ( s , s ′ ) ∈ R + , P 1 S = 1 S , P ⊤ 1 S = 1 S , P ( s , s ′ ) = 0 if ( s , s ′ ) ∈ / E } .
with 1 S = ( 1 , … , 1 ) ∈ R S \mathbf{1}_{\mathcal{S}} = (1,\ldots,1) \in\mathbb{R}^{\mathcal{S}} 1 S = ( 1 , … , 1 ) ∈ R S . Note that we can omit the dimension associated with A \mathcal{A} A since ∣ A ∣ = 1 |\mathcal{A}| = 1 ∣ A ∣ = 1 . Choose the rewards as follows: there is a reward of 1 when we are at state s 0 s_0 s 0 and a reward of 0 everywhere else. Since the adversary solves inf P ∈ P R ( π , P ) \inf_{P \in\mathcal{P}} R(\pi,P) inf P ∈ P R ( π , P ) , the adversary will choose a matrix in P \mathcal{P} P to spend as little time as possible in state s 0 s_0 s 0 (the only state giving us a positive reward). The initial distribution starts deterministically in s 0 s_0 s 0 . The return R ( π , P ) R(\pi,P) R ( π , P ) for P ∈ P P \in\mathcal{P} P ∈ P is
R ( π , P ) = E P [ ∑ t = 0 + ∞ γ t 1 { s t = s 0 } ] . R(\pi,P) = \mathbb{E}^{P}\left[\sum_{t=0}^{+\infty} \gamma^t \mathbf{1}_{\{s_t=s_0\}}\right]. R ( π , P ) = E P [ t = 0 ∑ + ∞ γ t 1 { s t = s 0 } ] .
We first claim that inf P ∈ P R ( π , P ) ≥ 1 1 − γ ∣ V ∣ \inf_{P \in\mathcal{P}} R(\pi,P) \ge\frac{1}{1-\gamma^{|V|}} inf P ∈ P R ( π , P ) ≥ 1 − γ ∣ V ∣ 1 . Indeed for τ \tau τ the random variable modeling the return time to s 0 s_0 s 0 we get that
R ( π , P ) = E P [ ∑ t = 0 + ∞ γ τ t ] = ∑ t = 0 + ∞ E P [ γ τ t ] ≥ ∑ t = 0 + ∞ γ E P [ τ ] t = 1 1 − γ E P [ τ ] . (22) R(\pi,P) = \mathbb{E}^{P}\left[\sum_{t=0}^{+\infty} \gamma^{\tau t}\right] = \sum_{t=0}^{+\infty}\mathbb{E}^{P}\left[\gamma^{\tau t}\right] \ge\sum_{t=0}^{+\infty}\gamma^{\mathbb{E}^{P}[\tau]t} = \frac{1}{1-\gamma^{\mathbb{E}^{P}[\tau]}}. \tag*{(22)} R ( π , P ) = E P [ t = 0 ∑ + ∞ γ τ t ] = t = 0 ∑ + ∞ E P [ γ τ t ] ≥ t = 0 ∑ + ∞ γ E P [ τ ] t = 1 − γ E P [ τ ] 1 . (22)
where the second equality is from the Monotone Convergence Theorem and the inequality is from Jensen’s inequality. Finally we note that we must have E P [ τ ] ≤ ∣ V ∣ \mathbb{E}^{P}[\tau] \leq|V| E P [ τ ] ≤ ∣ V ∣ , because the expected return time E P [ τ ] \mathbb{E}^{P}[\tau] E P [ τ ] is the inverse of the s 0 s_0 s 0 -component in the invariant distribution of the Markov chain induced by P ∈ P P \in\mathcal{P} P ∈ P , and since P P P is bistochastic (i.e. P P P is a stochastic matrix and P ⊤ P^{\top} P ⊤ too), we must have that this invariant distribution is uniform over the closed communicating class containing s 0 s_0 s 0 . Now we claim that there exists a Hamiltonian cycle in G G G if and only if inf P ∈ P R ( π , P ) = 1 1 − γ ∣ V ∣ \inf_{P \in\mathcal{P}} R(\pi, P) = \frac{1}{1-\gamma^{|V|}} inf P ∈ P R ( π , P ) = 1 − γ ∣ V ∣ 1 .
We first prove the easy implication. If there is a Hamiltonian cycle in G G G , then there is a corresponding matrix P ∈ P P \in\mathcal{P} P ∈ P such that the dynamics is deterministic under P P P and follows the Hamiltonian cycle. By definition for this choice of P P P we will pass by s 0 s_0 s 0 every ∣ V ∣ |V| ∣ V ∣ iterations and therefore the discounted return is R ( π , P ) = 1 1 − γ ∣ V ∣ R(\pi, P) = \frac{1}{1-\gamma^{|V|}} R ( π , P ) = 1 − γ ∣ V ∣ 1 .
Now assume that inf P ∈ P R ( π , P ) = 1 1 − γ ∣ V ∣ \inf_{P \in\mathcal{P}} R(\pi, P) = \frac{1}{1-\gamma^{|V|}} inf P ∈ P R ( π , P ) = 1 − γ ∣ V ∣ 1 . By Proposition 2.2, the infimum is attained at some P ⋆ ∈ P P^{\star} \in\mathcal{P} P ⋆ ∈ P . The proof of the lower bound on inf P ∈ P R ( π , P ) \inf_{P \in\mathcal{P}} R(\pi, P) inf P ∈ P R ( π , P ) shows that we must have E P ⋆ [ τ ] = ∣ V ∣ \mathbb{E}^{P^{\star}}[\tau] = |V| E P ⋆ [ τ ] = ∣ V ∣ as well as equality in Jensen’s inequality, which implies the stronger property that τ = ∣ V ∣ \tau= |V| τ = ∣ V ∣ almost surely. This means that starting from s 0 s_0 s 0 we come back to s 0 s_0 s 0 exactly at period ∣ V ∣ |V| ∣ V ∣ . It is not straightforward to construct a Hamiltonian cycle from P ⋆ P^{\star} P ⋆ , because P ⋆ P^{\star} P ⋆ may include randomized transitions (the matrix is bistochastic but the entries need not be 0 or 1, we can have entries strictly between 0 and 1). However a theorem by Birkhoff says that the polytope P \mathcal{P} P admits the permutation matrices (compatible with E E E ) as extreme points, i.e. we can write P ⋆ = ∑ ℓ = 1 m λ ℓ Q ℓ P^{\star} = \sum_{\ell=1}^{m} \lambda_{\ell}Q^{\ell} P ⋆ = ∑ ℓ = 1 m λ ℓ Q ℓ with Q ℓ ∈ P Q^{\ell} \in\mathcal{P} Q ℓ ∈ P a permutation matrix for each ℓ ∈ { 1 , … , m } \ell\in\{1,\ldots,m\} ℓ ∈ { 1 , … , m } : Q s s ′ ℓ = 1 { s = σ ℓ ( s ′ ) } Q^{\ell}_{ss'} = \mathbf{1}_{\{s=\sigma^{\ell}(s')\}} Q s s ′ ℓ = 1 { s = σ ℓ ( s ′ )} for some permutation σ ℓ \sigma^{\ell} σ ℓ of S S S , and λ ℓ ≥ 0 \lambda_{\ell} \geq0 λ ℓ ≥ 0 , ∑ ℓ = 1 m λ ℓ = 1 \sum_{\ell=1}^{m} \lambda_{\ell} = 1 ∑ ℓ = 1 m λ ℓ = 1 . Each permutation matrix decomposes into disjoint directed cycles, and for some ℓ ∈ { 1 , … , m } \ell\in\{1,\ldots,m\} ℓ ∈ { 1 , … , m } consider the cycle of Q ℓ Q^{\ell} Q ℓ containing s 0 s_0 s 0 . If this cycle has length strictly smaller than ∣ V ∣ |V| ∣ V ∣ then we would have that P P ⋆ ( τ < ∣ V ∣ ) > 0 \mathbb{P}_{P^{\star}}(\tau< |V|) > 0 P P ⋆ ( τ < ∣ V ∣ ) > 0 , which is impossible since τ = ∣ V ∣ \tau= |V| τ = ∣ V ∣ almost surely (which exactly means P P ⋆ ( τ = ∣ V ∣ ) = 1 \mathbb{P}_{P^{\star}}(\tau= |V|) = 1 P P ⋆ ( τ = ∣ V ∣ ) = 1 ). Hence for each Q ℓ Q^{\ell} Q ℓ the cycle containing s 0 s_0 s 0 induced by Q ℓ Q^{\ell} Q ℓ has length ∣ V ∣ |V| ∣ V ∣ , i.e., there exists a Hamiltonian cycle.
Therefore we can verify if there exists a Hamiltonian cycle by verifying inf P ∈ P R ( π , P ) ≤ 1 1 − γ ∣ V ∣ \inf_{P \in\mathcal{P}} R(\pi, P) \leq\frac{1}{1-\gamma^{|V|}} inf P ∈ P R ( π , P ) ≤ 1 − γ ∣ V ∣ 1 (in which case we must have inf P ∈ P R ( π , P ) = 1 1 − γ ∣ V ∣ \inf_{P \in\mathcal{P}} R(\pi, P) = \frac{1}{1-\gamma^{|V|}} inf P ∈ P R ( π , P ) = 1 − γ ∣ V ∣ 1 ). This shows that solving the policy evaluation problem is NP-hard.
Main takeaways
Let us summarize the main results in this section. For a general uncertainty set P \mathcal{P} P :
Properties of optimal policies. Optimal policies for (2.1) always exist but may require being non-stationary, even when there are only two possible transition kernels (Proposition 2.3 and Proposition 2.5 ). Approximately optimal policies may be chosen “eventually stationary” but we may need longer and longer horizons to better approximate optimal policies (Proposition 2.6 ). Even for instances described by simple (rational) entries, we may need irrational real values to describe stationary optimal policies (Proposition 2.8 ).
Hardness and multiple worst-case kernels. Evaluating the worst-case return of a stationary policy may be computationally challenging, even when the uncertainty set is a convex compact polytope, but worst-case transition probabilities always exist for compact uncertainty sets (Proposition 2.2 and Proposition 2.10 ). Finally, there may exist multiple worst-case value functions, since worst-case transition kernels need not be unique (Proposition 2.9 ).
Let’s take a step back. Overall it appears that solving the robust MDP problem in general is a difficult task under the various settings considered in this chapter. This is in stark contrast to the numerous favorable tractability properties of the nominal MDP setting studied in Chapter 1. At this point this discrepancy may seem surprising. Indeed, the roles of π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S and P ∈ P P \in\mathcal{P} P ∈ P in the expression of the value function v π , P v^{\pi,P} v π , P as in (5) are similar: v π , P v^{\pi,P} v π , P depends on the transition matrix L π , P ∈ R S × S L^{\pi,P} \in\mathbb{R}^{S \times S} L π , P ∈ R S × S of the Markov chain over S \mathcal{S} S induced by π \pi π , P P P , and on r π , P ∈ R S r^{\pi,P} \in\mathbb{R}^{S} r π , P ∈ R S the expected instantaneous rewards. Both L π , P L^{\pi,P} L π , P and r π , P r^{\pi,P} r π , P are bilinear functions of ( π , P ) (\pi,P) ( π , P ) . So why is optimizing R ( π , P ) R(\pi,P) R ( π , P ) over P ∈ P P \in\mathcal{P} P ∈ P difficult (for fixed π \pi π ), while optimizing R ( π , P ) R(\pi,P) R ( π , P ) over π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S is tractable (for fixed P P P )? The main issue must come from the properties of the domains Π S \Pi_{\mathrm{S}} Π S and P \mathcal{P} P of each variable π \pi π and P P P (given the “symmetry” of ( π , P ) ↦ v π , P (\pi,P) \mapsto v^{\pi,P} ( π , P ) ↦ v π , P in π \pi π and P P P ). What is the fundamental structure of Π S \Pi_{\mathrm{S}} Π S that makes nominal MDPs tractable, and that is lacking in general P \mathcal{P} P ? Answering this question is the focus of the next chapter.
Exercises
Worst-case transitions in the relative interior of the uncertainty sets. Consider the robust MDP instance from Figure 2.2 , with an uncertainty set P \mathcal{P} P parametrized by p ∈ [ 0 , 1 ] p \in[0,1] p ∈ [ 0 , 1 ] . Use this instance to show that (optimal) worst-case transition probabilities may belong to the relative interior of P \mathcal{P} P .
Figure 2.2. MDP instance for § 2.5 . The transition probabilities and rewards are indicated beside each arc.
Percentile guarantees. Assume that P \mathcal{P} P is constructed from some random observations sampled from a fixed true constant transition kernel P 0 P^{0} P 0 , so that P \mathcal{P} P is a random variable.
Let z ⋆ z^\star z ⋆ be the optimal objective value and π P \pi^{\mathcal{P}} π P be an optimal solution in (20) ; note that z ⋆ z^\star z ⋆ , π P \pi^{\mathcal{P}} π P are random variables (they depend on the random variable P \mathcal{P} P ). Show that π P \pi^{\mathcal{P}} π P guarantees its worst-case return over P \mathcal{P} P with a probability at least as large as P ( P 0 ∈ P ) \mathbb{P}(P^0 \in\mathcal{P}) P ( P 0 ∈ P ) , i.e., show that P ( R ( π P , P 0 ) ≥ z ⋆ ) ≥ P ( P 0 ∈ P ) \mathbb{P}(\mathcal{R}(\pi^{\mathcal{P}},P^0) \ge z^\star) \ge\mathbb{P}(P^0 \in\mathcal{P}) P ( R ( π P , P 0 ) ≥ z ⋆ ) ≥ P ( P 0 ∈ P ) , where the probability P \mathbb{P} P is over the random variables π P \pi^{\mathcal{P}} π P , z ⋆ z^\star z ⋆ , P \mathcal{P} P .
Entries of an optimal policy. Show that for any d ∈ N d \in\mathbb{N} d ∈ N , there exist simple robust MDP instances described by rational entries such that the entries of any stationary optimal policy may need to be algebraic numbers of degree d d d .
References
Robust MDPs date as far back as [75 ] . The fact that history-dependent policies may be necessary to solve robust MDPs with general uncertainty sets is proved in [80 ] ; the proof presented above is adapted from an example in [81 ] . The potential non-existence of uniformly optimal policies follows from more general arguments given in [6 ] . The irrationality of the entries of optimal policies/transition probabilities mirrors existing results for concurrent stochastic games (e.g. Example 3.2.1 in [21 ] ) and finite games with more than two players [57 ] . Finally, there are many NP-hardness results for solving robust MDPs or policy evaluation for general uncertainty sets; I use here the elegant proof from [22 ] and [3 ] based on Hamiltonian cycles. I also refer to the thesis of Le Tallec [49 ] which contains a flurry of counterexamples and hardness results. The instance from Figure 2.2 is from [25 ] . A recent survey on robust MDPs can be found in [70 ] .
Robust MDPs with rectangular uncertainty sets
This chapter introduces the s-rectangular and sa-rectangular models of uncertainty and proves several of the favorable tractability properties implied by these rectangularity assumptions (strong duality, optimality of stationary policies, existence of uniformly optimal policies, fixed point optimality equations). It ends with a more advanced section showing that in some sense, rectangularity is the “weakest” assumption one needs to make to ensure these favorable properties. We assume P ≠ ∅ \mathcal{P} \ne\emptyset P = ∅ throughout.
Modeling assumptions
In the previous chapter, we have seen several examples of uncertainty sets leading to difficult optimization problems. For instance, the uncertainty set for Example 1.2 with p = p ′ , p ∈ [ 0 , 1 ] p = p', p \in[0,1] p = p ′ , p ∈ [ 0 , 1 ] is convex but the same parameter p p p appears for the transitions out of the two different states s 0 s_0 s 0 and s 1 s_1 s 1 . The class of s-rectangular uncertainty sets precisely remedies this issue.
Definition 3.1. An uncertainty set P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A is s-rectangular if we can write P = × s ∈ S P s \mathcal{P} = \mathop{\times}_{s\in\mathcal{S}}\mathcal{P}_s P = × s ∈ S P s for some sets P s ⊆ Δ ( S ) A \mathcal{P}_s \subseteq\Delta(\mathcal{S})^\mathcal{A} P s ⊆ Δ ( S ) A for each s ∈ S s \in\mathcal{S} s ∈ S .
Let us parse this definition. If P \mathcal{P} P is s-rectangular then for each s ∈ S s \in\mathcal{S} s ∈ S the set P s ⊆ Δ ( S ) A \mathcal{P}_s \subseteq\Delta(\mathcal{S})^\mathcal{A} P s ⊆ Δ ( S ) A must be equal to the set { ( P ( s , a ) ) a ∈ A ∣ P ∈ P } \{(P(s,a))_{a\in\mathcal{A}} \mid P \in\mathcal{P}\} {( P ( s , a ) ) a ∈ A ∣ P ∈ P } , the set of feasible transitions out of state s s s . The Cartesian product in the definition then means that the transitions out of each state s s s can be chosen (by the adversary) independently of the transitions out of the other states. We can revisit the examples studied so far and inspect their rectangularity or non-rectangularity properties:
Example 1.1 with p ∈ { 0 , 1 } p \in\{0,1\} p ∈ { 0 , 1 } is s-rectangular. Let us index the columns by ( a 1 , a 2 ) (a_1,a_2) ( a 1 , a 2 ) and the rows by ( s 0 , G , B ) (s_0,G,B) ( s 0 , G , B ) . Then P ( G ) = P ( B ) = ( 1 1 0 0 0 0 ) P(G)=P(B)=\begin{pmatrix}1&1\\0&0\\0&0\end{pmatrix} P ( G ) = P ( B ) = 1 0 0 1 0 0 since there is no uncertainty for the transitions out of the states G , B G,B G , B . Additionally for p = 0 p=0 p = 0 the transitions out of s 0 s_0 s 0 are ( 0 0 0 1 1 0 ) \begin{pmatrix}0&0\\0&1\\1&0\end{pmatrix} 0 0 1 0 1 0 , and for p = 1 p=1 p = 1 the transitions out of s 0 s_0 s 0 are ( 0 0 1 0 0 1 ) \begin{pmatrix}0&0\\1&0\\0&1\end{pmatrix} 0 1 0 0 0 1 . Hence
P s 0 = { ( 0 0 0 1 1 0 ) , ( 0 0 1 0 0 1 ) } . \mathcal{P}_{s_0}=\left\{\begin{pmatrix}0 & 0\\0 & 1\\1 & 0\end{pmatrix},\begin{pmatrix}0 & 0\\1 & 0\\0 & 1\end{pmatrix}\right\}. P s 0 = ⎩ ⎨ ⎧ 0 0 1 0 1 0 , 0 1 0 0 0 1 ⎭ ⎬ ⎫ .
Hence P = P s 0 × { P ( G ) } × { P ( B ) } \mathcal{P}=\mathcal{P}_{s_0}\times\{\mathcal{P}(G)\}\times\{\mathcal{P}(B)\} P = P s 0 × { P ( G )} × { P ( B )} is s-rectangular. Note that P \mathcal{P} P is not convex (it is finite).
Example 1.2 with p , p ′ ∈ [ 0 , 1 ] p,p'\in[0,1] p , p ′ ∈ [ 0 , 1 ] is s-rectangular and convex: we can choose the transitions out of state s 0 s_0 s 0 (parameter p p p ) and out of state s 1 s_1 s 1 (parameter p ′ p' p ′ ) independently.
Example 1.2 with p = p ′ p=p' p = p ′ is not s-rectangular: if we choose a value for p p p we fix the transitions out of both s 0 s_0 s 0 and s 1 s_1 s 1 . Still we would get a convex set even for p = p ′ p=p' p = p ′ .
Example 2.1 with p ∈ [ 0 , 1 ] p\in[0,1] p ∈ [ 0 , 1 ] is s-rectangular and convex: there is no uncertainty for the transitions out of the absorbing state s 1 s_1 s 1 , so that p p p only influences the transitions out of s 0 s_0 s 0 . Indexing columns by ( a 1 , a 2 ) (a_1,a_2) ( a 1 , a 2 ) and rows by ( s 0 , s 1 ) (s_0,s_1) ( s 0 , s 1 ) , the set of feasible transitions out of s 0 s_0 s 0 is P s 0 = { ( 1 − p p p 1 − p ) ∣ p ∈ [ 0 , 1 ] } \mathcal{P}_{s_0}=\left\{\begin{pmatrix}1-p & p\\p& 1-p\end{pmatrix}\mathrel{\big|}p\in[0,1]\right\} P s 0 = { ( 1 − p p p 1 − p ) p ∈ [ 0 , 1 ] } .
In applications of robust MDPs, one typically considers sets based on distance from an empirical estimation P ^ ∈ Δ ( S ) S × A \widehat{P}\in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ∈ Δ ( S ) S × A , of the form
{ P ∈ Δ ( S ) S × A ∣ D ( P s , P ^ s ) ≤ ϵ s , ∀ s ∈ S } \left\{P\in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}}\mathrel{\big|}D(P_s,\widehat{P}_s)\le\epsilon_s,\ \forall s\in\mathcal{S}\right\} { P ∈ Δ ( S ) S × A D ( P s , P s ) ≤ ϵ s , ∀ s ∈ S }
with D D D that can be the ℓ ∞ \ell_\infty ℓ ∞ -distance, the ℓ 1 \ell_1 ℓ 1 -distance, the ℓ 2 \ell_2 ℓ 2 -distance, or based on Kullback-Leibler or ϕ \phi ϕ -divergences.
One way to “visually check” the s-rectangularity of a robust MDP instance represented on a graph (as in most examples in these notes) is to check that there is no parameter (such as some p ∈ [ 0 , 1 ] p\in[0,1] p ∈ [ 0 , 1 ] ) involved in the transitions out of two different states, and no constraints across the transition parameters out of different states. A more abstract way to define s-rectangularity is to say that P \mathcal{P} P is s-rectangular if and only if we can mix the entries (indexed by states) of any two kernels in P \mathcal{P} P and still recover a feasible kernel in P \mathcal{P} P (see § 3.8 for a rigorous statement). While we have defined s-rectangularity for uncertainty sets, this property is merely saying that P \mathcal{P} P is a Cartesian product, and we can wonder if other objects studied so far have this property. Crucially, this is indeed the case for the set of policies:
The set of stationary policies Π S \Pi_S Π S can be identified with Δ ( A ) S \Delta(\mathcal{A})^\mathcal{S} Δ ( A ) S , which is a Cartesian product: Δ ( A ) S = × s ∈ S Δ ( A ) \Delta(\mathcal{A})^\mathcal{S}=\times_{s\in\mathcal{S}}\Delta(\mathcal{A}) Δ ( A ) S = × s ∈ S Δ ( A ) .
The set of history-dependent policies Π H \Pi_H Π H is also a Cartesian product of copies of the simplex Δ ( A ) \Delta(\mathcal{A}) Δ ( A ) , taken over all possible finite trajectories in H \mathcal{H} H .
Since s-rectangularity makes the uncertainty set P \mathcal{P} P (which we can interpret as the “decision set” of the adversary choosing a worst-case transition kernel) much like the set of policies for nominal MDPs, we can expect that we can now derive more favorable tractability properties than what we’ve seen in the previous chapter. Regarding assumptions, we will assume compactness since this simplifies the exposition of the results; we only assume convexity where we need it (in particular we don’t need convexity in Section 3.2 but we need it in Section 3.3).
Remark 3.1. Rectangularity is in fact encountered in many other research areas. For instance:
The two-dimensional rectangle { ( x , y ) ∈ R 2 ∣ a ≤ x ≤ b , c ≤ y ≤ d } \{(x,y) \in\mathbb{R}^{2} \mid a \le x \le b, c \le y \le d\} {( x , y ) ∈ R 2 ∣ a ≤ x ≤ b , c ≤ y ≤ d } is rectangular, and more generally n n n -dimensional boxes are rectangular (this is the prototypical example).
The “independent row uncertainty” and “independent column uncertainty” assumptions in the literature on computing joint spectral radii exactly say the set of admissible matrices is a Cartesian product (over rows or columns).
A multivariate random variable with independent components (i.e., its distribution is a product distribution) has a rectangular support.
Policy evaluation
For now we focus on solving the policy evaluation problem (21) for a given stationary policy. Under the s-rectangularity assumption, we will see that this problem can be reformulated as an adversarial MDP , defined next.
Definition 3.2. Let M = ( S , A , r , P , p 0 ) \mathcal{M} = (\mathcal{S}, \mathcal{A}, r, P, p_{0}) M = ( S , A , r , P , p 0 ) be an MDP and let P ⊆ Δ ( S ) S × A \mathcal{P} \subseteq\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊆ Δ ( S ) S × A be a compact s-rectangular uncertainty set. Let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S be a stationary policy. The adversarial MDP is defined as the following MDP instance M ′ = ( S ′ , A ′ , r ′ , P ′ , p 0 ′ ) \mathcal{M}' = (\mathcal{S}', \mathcal{A}', r', P', p'_{0}) M ′ = ( S ′ , A ′ , r ′ , P ′ , p 0 ′ ) :
The set of states S ′ \mathcal{S}' S ′ is equal to S \mathcal{S} S .
The set of actions A s ′ \mathcal{A}'_{s} A s ′ available at state s ∈ S ′ s \in\mathcal{S}' s ∈ S ′ is the compact set P s ⊆ Δ ( S ) A \mathcal{P}_{s} \subseteq\Delta(\mathcal{S})^{\mathcal{A}} P s ⊆ Δ ( S ) A .
The instantaneous reward for choosing action P s ∈ P s P_{s} \in\mathcal{P}_{s} P s ∈ P s at state s ∈ S ′ s \in\mathcal{S}' s ∈ S ′ is r ′ ( s , P s ) = − ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ r ( s , a ) r'(s,P_{s}) = -\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a)^{\top}r(s,a) r ′ ( s , P s ) = − ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ r ( s , a ) , which is independent of the next state. We will write r ′ ( s , a ′ ) 1 S ′ r'(s,a')\mathbf{1}_{\mathcal{S}'} r ′ ( s , a ′ ) 1 S ′ for the vector ( r ′ ( s , a ′ ) , … , r ′ ( s , a ′ ) ) ∈ R S ′ (r'(s,a'),\ldots,r'(s,a')) \in\mathbb{R}^{\mathcal{S}'} ( r ′ ( s , a ′ ) , … , r ′ ( s , a ′ )) ∈ R S ′ , given some “action” a ′ ∈ P s a' \in\mathcal{P}_{s} a ′ ∈ P s of the adversary.
The transition probability toward s ′ ∈ S s' \in\mathcal{S} s ′ ∈ S after choosing action P s ∈ P s P_{s} \in\mathcal{P}_{s} P s ∈ P s at state s ∈ S ′ s \in\mathcal{S}' s ∈ S ′ is ∑ a ∈ A π ( s , a ) P ( s , a , s ′ ) ∈ R \sum_{a\in\mathcal{A}}\pi(s,a)P(s,a,s') \in\mathbb{R} ∑ a ∈ A π ( s , a ) P ( s , a , s ′ ) ∈ R .
The initial distribution p 0 ′ p'_{0} p 0 ′ is equal to p 0 p_{0} p 0 .
In the adversarial MDP, a stationary deterministic policy is a choice of P s ∈ P s P_{s} \in\mathcal{P}_{s} P s ∈ P s for each state s ∈ S s \in\mathcal{S} s ∈ S , i.e., it is a choice of P ∈ P = × s ∈ S P s P \in\mathcal{P} = \times_{s\in\mathcal{S}}\mathcal{P}_{s} P ∈ P = × s ∈ S P s . As we discussed at the end of the previous chapter, the main missing ingredient to make (21) tractable for general P \mathcal{P} P was the (potential) lack of rectangularity. Now that we have assumed P \mathcal{P} P to be s-rectangular, we can indeed interpret the set P \mathcal{P} P as a set of stationary deterministic policies in the adversarial MDP; the set of stationary randomized policies in the adversarial MDP is × s ∈ S Δ ( P s ) \times_{s\in\mathcal{S}}\Delta(\mathcal{P}_{s}) × s ∈ S Δ ( P s ) . We need the minus sign to turn the minimization problem (21) into the adversarial MDP maximization problem (2) . The following theorem is the main result of this section.
Theorem 3.1. Let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S and P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be a compact s-rectangular uncertainty set.
Value function correspondence. Let v P , P ′ ∈ R S v^{P,P'} \in\mathbb{R}^{S} v P , P ′ ∈ R S be the value function of the stationary deterministic adversarial policy P ∈ P P \in\mathcal{P} P ∈ P in the adversarial MDP. Then v P , P ′ = − v π , P v^{P,P'}=-v^{\pi,P} v P , P ′ = − v π , P for v π , P ∈ R S v^{\pi,P}\in\mathbb{R}^{S} v π , P ∈ R S the value function of the policy π ∈ Π S \pi\in\Pi_{S} π ∈ Π S in the nominal MDP with transitions P P P .
Optimal objective correspondence. We have arg min P ∈ P R ( π , P ) = arg max P ∈ P p 0 T v P , P ′ \mathop{\arg\min}_{P\in\mathcal{P}} R(\pi,P)=\mathop{\arg\max}_{P\in\mathcal{P}}p_{0}^{\mathsf{T}}v^{P,P'} arg min P ∈ P R ( π , P ) = arg max P ∈ P p 0 T v P , P ′ .
Proof. 1. By definition, the value function v P , P ′ ∈ R S v^{P,P'}\in\mathbb{R}^{S} v P , P ′ ∈ R S of a stationary deterministic adversarial policy P ∈ P P\in\mathcal{P} P ∈ P for the transition probability P ′ P' P ′ in the adversarial MDP satisfies the following fixed point equation:
v P , P ′ ( s ) = E a ′ = P ( s ) , s ′ ∼ P ′ ( s , a ′ ) [ r ′ ( s , a ′ ) + γ v P , P ′ ( s ′ ) ] = P ′ ( s , a ′ ) T ( r ′ ( s , a ′ ) 1 S ′ + γ v P , P ′ ) = ( ∑ a ∈ A π ( s , a ) P ( s , a ) ) T ( − ∑ a ∈ A π ( s , a ) P ( s , a ) T r ( s , a ) + γ v P , P ′ ( s ′ ) ) s ′ ∈ S = − ∑ a ∈ A π ( s , a ) P ( s , a ) T ( r ( s , a ) + γ ( − v P , P ′ ) ) = − T π , P ( − v P , P ′ ) ( s ) . \begin{aligned}
v^{P,P'}(s)=\mathbb{E}_{a'=P(s),s'\sim P'(s,a')}\left[r'(s,a')+\gamma v^{P,P'}(s')\right] \\
&= P'(s,a')^{\mathsf{T}}\left(r'(s,a')\mathbf{1}_{S'}+\gamma v^{P,P'}\right) \\
&= \left(\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a)\right)^{\mathsf{T}}\left(-\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a)^{\mathsf{T}}r(s,a)+\gamma v^{P,P'}(s')\right)_{s'\in\mathcal{S}} \\
&=-\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a)^{\mathsf{T}}\left(r(s,a)+\gamma(-v^{P,P'})\right) \\
&=-T^{\pi,P}(-v^{P,P'})(s).
\end{aligned} v P , P ′ ( s ) = E a ′ = P ( s ) , s ′ ∼ P ′ ( s , a ′ ) [ r ′ ( s , a ′ ) + γ v P , P ′ ( s ′ ) ] = P ′ ( s , a ′ ) T ( r ′ ( s , a ′ ) 1 S ′ + γ v P , P ′ ) = ( a ∈ A ∑ π ( s , a ) P ( s , a ) ) T ( − a ∈ A ∑ π ( s , a ) P ( s , a ) T r ( s , a ) + γ v P , P ′ ( s ′ ) ) s ′ ∈ S = − a ∈ A ∑ π ( s , a ) P ( s , a ) T ( r ( s , a ) + γ ( − v P , P ′ ) ) = − T π , P ( − v P , P ′ ) ( s ) .
The last equality shows that v P , P ′ v^{P,P'} v P , P ′ is a fixed point of the operator v ↦ − T π , P ( − v ) v\mapsto-T^{\pi,P}(-v) v ↦ − T π , P ( − v ) , with T π , P T^{\pi,P} T π , P defined in (1.11). Hence v P , P ′ = − v π , P v^{P,P'}=-v^{\pi,P} v P , P ′ = − v π , P (since v π , P v^{\pi,P} v π , P is the unique fixed point of the contraction T π , P T^{\pi,P} T π , P ).
We have
min P ∈ P R ( π , P ) = min P ∈ P p 0 T v π , P = − max P ∈ P p 0 T ( − v π , P ) = − max P ∈ P p 0 T v P , P ′ . \min_{P\in\mathcal{P}}R(\pi,P)=\min_{P\in\mathcal{P}}p_{0}^{\mathsf{T}}v^{\pi,P}=-\max_{P\in\mathcal{P}}p_{0}^{\mathsf{T}}(-v^{\pi,P})=-\max_{P\in\mathcal{P}}p_{0}^{\mathsf{T}}v^{P,P'}. P ∈ P min R ( π , P ) = P ∈ P min p 0 T v π , P = − P ∈ P max p 0 T ( − v π , P ) = − P ∈ P max p 0 T v P , P ′ .
where the first equality is by definition of the return in the nominal MDP, the second equality is by min f = − max − f \min f=-\max-f min f = − max − f , and the third equality is from the value function correspondence (first statement in this theorem). Hence the sets of optimal solutions for min P ∈ P R ( π , P ) \min_{P\in\mathcal{P}}R(\pi,P) min P ∈ P R ( π , P ) and max P ∈ P p 0 T v P , P ′ \max_{P\in\mathcal{P}}p_{0}^{\mathsf{T}}v^{P,P'} max P ∈ P p 0 T v P , P ′ coincide.
Note that max P ∈ P p 0 T v P , P ′ \max_{P\in\mathcal{P}}p_{0}^{\mathsf{T}}v^{P,P'} max P ∈ P p 0 T v P , P ′ computes the best stationary deterministic adversarial policies in the adversarial MDP (the transition probabilities are constant over time; defining history-dependent adversaries requires a bit more care and will be done in Chapter 4). Since in the adversarial MDP the action sets are compact and the rewards and transitions are continuous in the actions chosen, we can apply the results from Chapter 1 (see Section 1.5 for the case of nominal MDPs with state-dependent compact action sets) and we know that in the adversarial MDP a uniformly optimal policy P P P exists and can be chosen stationary deterministic (i.e. P ∈ P P\in\mathcal{P} P ∈ P ). We define the adversarial Bellman operator T π : R S → R S T^\pi:\mathbb{R}^{S}\to\mathbb{R}^{S} T π : R S → R S as
T π v ( s ) = min P ∈ P T π , P v ( s ) , ∀ v ∈ R S , s ∈ S . (23) T^\pi v(s)=\min_{P\in\mathcal{P}}T^{\pi,P}v(s),\quad\forall\,v\in\mathbb{R}^{S},\ s\in\mathcal{S}. \tag*{(23)} T π v ( s ) = P ∈ P min T π , P v ( s ) , ∀ v ∈ R S , s ∈ S . (23)
Note that the family { T π , P ∣ P ∈ P } \{T^{\pi,P}\mid P\in\mathcal{P}\} { T π , P ∣ P ∈ P } satisfies the attainability property with respect to T π T^\pi T π , as we state formally next.
Lemma 3.1. Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be a compact s s s -rectangular uncertainty set and let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S . For any v ∈ R S v \in\mathbb{R}^{\mathcal{S}} v ∈ R S , there exists P ∈ P P \in\mathcal{P} P ∈ P such that T π v = T π , P v T^\pi v = T^{\pi,P}v T π v = T π , P v .
Proof. Let s ∈ S s \in\mathcal{S} s ∈ S and v ∈ R S v \in\mathbb{R}^{\mathcal{S}} v ∈ R S . Note that the minimization program in the definition of T π T^\pi T π as in (3.1) only depends on P ( s ) P(s) P ( s ) the s s s -th component of P P P . Since P \mathcal{P} P is s s s -rectangular compact we get that we can solve each of these minimization programs independently and recover a joint solution that is optimal simultaneously across all s ∈ S s \in\mathcal{S} s ∈ S .
In fact one can easily verify that the set { T π , P ∣ P ∈ P } \{T^{\pi,P}\mid P \in\mathcal{P}\} { T π , P ∣ P ∈ P } is a Cartesian product when P \mathcal{P} P is s s s -rectangular; for a non-rectangular set P \mathcal{P} P , this set is still well-defined, as well as T π T^\pi T π as in (3.1), but the attainability property may not hold (see Section 3.5 for more on this topic).
It is worth emphasizing one more time the role of rectangularity. For nominal MDPs, the Bellman operator T π T^\pi T π imposes componentwise maximization over the policies, i.e., it imposes that we can choose actions independently across each state. Rectangularity of the uncertainty set ensures the same property, that is, the adversary can independently minimize the values obtained at each state. Since we have already noted right before § 2.5 how similar the two functions π ↦ v π , P \pi\mapsto v^{\pi,P} π ↦ v π , P and P ↦ v π , P P \mapsto v^{\pi,P} P ↦ v π , P are, the interpretation of the policy evaluation as an adversarial MDP is natural (once we allow for rectangular sets of decisions for the adversary, i.e., once we have s s s -rectangular uncertainty). Table 3.1 summarizes the similarities between nominal MDPs and adversarial MDPs. Note that in general it is much easier to work with value functions than directly with the return function; in some sense, non-rectangular models typically lead to difficult optimization problems because one cannot, in general, rely on value functions (which are defined componentwise, i.e., in a rectangular manner).
Table 3.1. Comparisons of the properties of nominal MDPs, adversarial MDPs, and robust MDPs under an s s s -rectangular uncertainty set P \mathcal{P} P . The operator T π , P : R S → R S T^{\pi,P}:\mathbb{R}^{\mathcal{S}}\to\mathbb{R}^{\mathcal{S}} T π , P : R S → R S is defined as T π , P v ( s ) = ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) T^{\pi,P}v(s)=\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a)^\top(r(s,a)+\gamma v) T π , P v ( s ) = ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , for s ∈ S s\in\mathcal{S} s ∈ S and v ∈ R S v\in\mathbb{R}^{\mathcal{S}} v ∈ R S . The column for adversarial MDPs requires compactness of P \mathcal{P} P while the column for robust MDPs requires compactness and convexity of P \mathcal{P} P . For adversarial MDPs we identify stationary randomized policies (which should be × s ∈ S Δ ( P s ) \times_{s\in\mathcal{S}}\Delta(\mathcal{P}_s) × s ∈ S Δ ( P s ) ) with their effective transition probabilities equal to × s ∈ S conv ( P s ) \times_{s\in\mathcal{S}}\operatorname{conv}(\mathcal{P}_s) × s ∈ S conv ( P s ) .
The following direct corollary is a reformulation of Proposition 1.1 and Theorem 1.1 for the adversarial MDP; note that we do not need any convexity assumption on P \mathcal{P} P for now (indeed in Chapter 1 we worked with A \mathcal{A} A finite and then extended the results to A \mathcal{A} A compact in Section 1.5).
Corollary 3.1. Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be a compact s s s -rectangular uncertainty set and let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S .
Bellman equations. The operator T π T^\pi T π defined in (3.1) is a monotone contraction with unique fixed point v π ∈ R S v^\pi\in\mathbb{R}^{\mathcal{S}} v π ∈ R S , satisfying the following worst-case Bellman equations in the variable v ∈ R S v \in\mathbb{R}^{\mathcal{S}} v ∈ R S :
v ( s ) = min P ∈ P ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ s ∈ S . (24) v(s) = \min_{P \in\mathcal{P}} \sum_{a \in\mathcal{A}} \pi(s,a)P(s,a)^\top(r(s,a)+\gamma v), \qquad\forall\ s \in\mathcal{S}. \tag*{(24)} v ( s ) = P ∈ P min a ∈ A ∑ π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ s ∈ S . (24) Uniform optimality. There exists a uniform worst-case transition probabilities P π ∈ P P^\pi\in\mathcal{P} P π ∈ P , that solves the policy evaluation problem (2.2) simultaneously for all initial distributions p 0 ∈ Δ ( S ) p_0 \in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) , and the value function v π , P π ∈ R S v^{\pi,P^\pi} \in\mathbb{R}^{\mathcal{S}} v π , P π ∈ R S coincides with v π v^\pi v π .
Optimality conditions. We can compute a uniform worst-case transition kernel P π ∈ P P^\pi\in\mathcal{P} P π ∈ P by choosing any transition kernel that jointly attains that arg min \arg\min arg min in T π v π ( s ) T^\pi v^\pi(s) T π v π ( s ) across all s ∈ S s \in\mathcal{S} s ∈ S , i.e., we have that P π P^\pi P π is a uniform worst-case transition kernel if and only if T π v π = T π , P π v π T^\pi v^\pi= T^{\pi,P^\pi}v^\pi T π v π = T π , P π v π .
Note that the first and second statements in Corollary 3.1 show that v π ∈ R S v^\pi\in\mathbb{R}^{\mathcal{S}} v π ∈ R S is the “smallest” worst-case value function of π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S , in that
v π ( s ) = min P ∈ P v π , P ( s ) , ∀ s ∈ S v^\pi(s) = \min_{P\in\mathcal{P}} v^{\pi,P}(s), \quad\forall\ s \in\mathcal{S} v π ( s ) = P ∈ P min v π , P ( s ) , ∀ s ∈ S
and for any p 0 ∈ Δ ( S ) p_0 \in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) we must have
min P ∈ P R ( π , P ) = p 0 T v π . \min_{P\in\mathcal{P}} R(\pi,P) = p_0^\mathsf{T}v^\pi. P ∈ P min R ( π , P ) = p 0 T v π .
Remark 3.2. Proposition 2.9 shows that even for s-rectangular uncertainty sets, there may be multiple worst-case value functions (for a single policy π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S ), since worst-case kernels need not be unique. As such, the vector v π ∈ R S v^\pi\in\mathbb{R}^{\mathcal{S}} v π ∈ R S is in some sense the component-wise smallest such worst-case value function (as shown by the last-but-one equation above).
Computing worst-case kernels. Since we interpret the policy evaluation problem as an adversarial MDP, we use the whole machinery developed in Chapter 1 for computing a uniform worst-case optimal kernel. The value iteration and policy iteration algorithms and their convergence rates are given next.
Algorithm 3 Value iteration for adversarial MDPs
Input: π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S and P \mathcal{P} P compact s-rectangular
Initialize v 0 ∈ R S v^0 \in\mathbb{R}^{\mathcal{S}} v 0 ∈ R S .
for t ∈ N t \in\mathbb{N} t ∈ N do
v t + 1 = T π v t v^{t+1} = T^\pi v^t v t + 1 = T π v t
end for
Proposition 3.1. Let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S and let P \mathcal{P} P be a compact s-rectangular uncertainty set. Let ( v t ) t ∈ N (v^t)_{t\in\mathbb{N}} ( v t ) t ∈ N be the sequence of vectors generated by Algorithm 3.
The sequence ( v t ) t ∈ N (v^t)_{t\in\mathbb{N}} ( v t ) t ∈ N converges to v π v^\pi v π at a geometric rate of γ \gamma γ :
∥ v π − v t ∥ ∞ ≤ γ t ∥ v π − v 0 ∥ ∞ \lVert v^\pi-v^t\rVert_\infty\leq\gamma^t\lVert v^\pi-v^0\rVert_\infty ∥ v π − v t ∥ ∞ ≤ γ t ∥ v π − v 0 ∥ ∞
Assume that ∥ v π − v t ∥ ∞ ≤ ϵ \lVert v^\pi-v^t\rVert_\infty\leq\epsilon ∥ v π − v t ∥ ∞ ≤ ϵ for some t ∈ N t \in\mathbb{N} t ∈ N and let P ∈ P P \in\mathcal{P} P ∈ P such that T π , P v t = T π v t T^{\pi,P}v^t=T^\pi v^t T π , P v t = T π v t . Then P P P is a uniform 2 ϵ γ ( 1 − γ ) − 1 2\epsilon\gamma(1-\gamma)^{-1} 2 ϵ γ ( 1 − γ ) − 1 worst-case transition probabilities for π \pi π :
∥ v π − v π , P ∥ ∞ ≤ 2 ϵ γ ( 1 − γ ) − 1 . \lVert v^\pi-v^{\pi,P}\rVert_\infty\leq2\epsilon\gamma(1-\gamma)^{-1}. ∥ v π − v π , P ∥ ∞ ≤ 2 ϵ γ ( 1 − γ ) − 1 .
Proposition 3.2. Let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S and let P \mathcal{P} P be a compact s-rectangular uncertainty set. Let ( P t ) t ∈ N (P^t)_{t\in\mathbb{N}} ( P t ) t ∈ N be the sequence of policies generated by Algorithm 4. Then ∥ v π − v π , P t ∥ ∞ ≤ γ t ∥ v π − v π , P 0 ∥ ∞ \lVert v^\pi-v^{\pi,P^t}\rVert_\infty\leq\gamma^t\lVert v^\pi-v^{\pi,P^0}\rVert_\infty ∥ v π − v π , P t ∥ ∞ ≤ γ t ∥ v π − v π , P 0 ∥ ∞ .
Similarly as in Chapter 1, the proofs of the above convergence rates follow from the more general versions of the algorithms analyzed in Appendix B. We can still obtain a mathematical programming formulation for the policy evaluation program. However we need to be careful here: the operator T π T^\pi T π is componentwise concave (as the minima over affine forms) whereas the operator T P T^\mathsf{P} T P is componentwise convex (as the maxima over affine forms). Hence to obtain a convex formulation for computing v π v^\pi v π we need to use the alternate version of the contraction lemma mentioned in Section 1.5, which yields the following proposition.
Algorithm 4 Policy iteration for adversarial MDPs
Input: π ∈ Π S \pi\in\Pi_{\mathcal S} π ∈ Π S and P \mathcal P P compact s-rectangular
Initialize P 0 ∈ P P^0 \in\mathcal P P 0 ∈ P , v 0 = v π , P 0 v^0 = v^{\pi,P^0} v 0 = v π , P 0 .
for t ∈ N t \in\mathbb{N} t ∈ N do
Choose P t + 1 ∈ P P^{t+1} \in\mathcal P P t + 1 ∈ P such that T π v t ( s ) = T π , P t + 1 v t ( s ) T^\pi v^t(s) = T^{\pi,P^{t+1}}v^t(s) T π v t ( s ) = T π , P t + 1 v t ( s ) for all s ∈ S s \in\mathcal S s ∈ S
Choose v t + 1 = v π , P t + 1 v^{t+1} = v^{\pi,P^{t+1}} v t + 1 = v π , P t + 1
end for
Proposition 3.3. Let π ∈ Π S \pi\in\Pi_{\mathcal S} π ∈ Π S and let P \mathcal P P be a compact s-rectangular uncertainty set. The smallest worst-case value function v π v^\pi v π is the unique optimal solution to the following convex program:
max { ∑ s ∈ S v ( s ) | v ( s ) ≤ min P ∈ P ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ s ∈ S , v ∈ R S } (25) \max\left\{ \sum_{s \in\mathcal S} v(s) \ \middle| \ v(s) \le\min_{P \in\mathcal P} \sum_{a \in\mathcal A} \pi(s,a)P(s,a)^\top(r(s,a) + \gamma v),\ \forall s \in\mathcal S, v \in\mathbb{R}^{\mathcal S} \right\} \tag*{(25)} max { s ∈ S ∑ v ( s ) v ( s ) ≤ P ∈ P min a ∈ A ∑ π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ s ∈ S , v ∈ R S } (25)
Complexity. The main difference with the complexity given for nominal MDPs in Chapter 1 pertains to how we can evaluate the operator T π T^\pi T π at each iteration of these algorithms. This requires solving an optimization problem with linear objective over the compact set P \mathcal P P . This can be done efficiently for s-rectangular uncertainty sets with additional structure (e.g. based on ℓ ∞ \ell_\infty ℓ ∞ -balls, ℓ p \ell_p ℓ p -balls for p ∈ { 1 , 2 } p \in\{1,2\} p ∈ { 1 , 2 } , based on Kullback-Leibler divergence, etc.), and we point to § 3.10 for some references.
Remark 3.3. In this section on the adversarial MDP we do not need convexity of P \mathcal P P for our results. However the Bellman operator T π T^\pi T π only involves optimizing linear functions over P \mathcal P P , hence we can replace P \mathcal P P by conv ( P ) \operatorname{conv}(\mathcal P) conv ( P ) in the expression of the Bellman operator T π T^\pi T π (and therefore in the definition of the worst-case value function v π v^\pi v π as the fixed point of T π T^\pi T π ). This will not be true in the next section on robust MDPs (i.e., on the maxmin formulation (20) ), as we will see next.
Optimal policies for rectangular robust MDPs
We now get back to solving the robust MDP problem (20) . Similarly to the case of nominal MDPs and adversarial MDPs, robust MDPs can be analyzed via the properties and fixed point of the following robust Bellman operator T : R S → R S T : \mathbb{R}^{\mathcal S} \to\mathbb{R}^{\mathcal S} T : R S → R S , defined as
T v ( s ) = max π ∈ Π S T π v ( s ) = max π ∈ Π S min P ∈ P T π , P v ( s ) , ∀ v ∈ R S , s ∈ S . (26) Tv(s) = \max_{\pi\in\Pi_{\mathcal S}} T^\pi v(s) = \max_{\pi\in\Pi_{\mathcal S}} \min_{P \in\mathcal P} T^{\pi,P}v(s),\ \forall v \in\mathbb{R}^{\mathcal S},\ s \in\mathcal S. \tag*{(26)} T v ( s ) = π ∈ Π S max T π v ( s ) = π ∈ Π S max P ∈ P min T π , P v ( s ) , ∀ v ∈ R S , s ∈ S . (26)
The robust Bellman operator inherits the monotonicity and contraction properties of the Bellman operator T π , P T^{\pi,P} T π , P , as shown in the next proposition.
Proposition 3.4. Let P ⊆ Δ ( S ) S × A \mathcal P \subseteq\Delta(\mathcal S)^{\mathcal S \times\mathcal A} P ⊆ Δ ( S ) S × A be a compact uncertainty set. The Bellman operator T T T defined in (26) is a monotone contraction with unique fixed point v ⋆ ∈ R S v^\star\in\mathbb{R}^{\mathcal S} v ⋆ ∈ R S , satisfying the following robust Bellman equation in the variable v ∈ R S v \in\mathbb{R}^{\mathcal S} v ∈ R S :
v ( s ) = max π ∈ Π S min P ∈ P ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ s ∈ S (27) v(s) = \max_{\pi\in\Pi_{\mathcal S}} \min_{P \in\mathcal P} \sum_{a \in\mathcal A} \pi(s,a)P(s,a)^\top(r(s,a) + \gamma v),\quad\forall s \in\mathcal S \tag*{(27)} v ( s ) = π ∈ Π S max P ∈ P min a ∈ A ∑ π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) , ∀ s ∈ S (27)
Proof. Since T π , P T^{\pi,P} T π , P is monotone we directly get that T T T is monotone. To show that T T T is a contraction, we note that for v , w ∈ R S v,w \in\mathbb{R}^{S} v , w ∈ R S and s ∈ S s \in\mathcal{S} s ∈ S we have
T v ( s ) = max π ∈ Π S min P ∈ P T π , P v ( s ) ≤ max π ∈ Π S min P ∈ P ( T π , P w ( s ) + γ ∥ v − w ∥ ∞ ) ≤ γ ∥ v − w ∥ ∞ + max π ∈ Π S min P ∈ P T π , P w ( s ) = γ ∥ v − w ∥ ∞ + T w ( s ) \begin{aligned}
Tv(s) &= \max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}} T^{\pi,P}v(s) \\
&\le\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}\left(T^{\pi,P}w(s)+\gamma\lVert v-w\rVert_{\infty}\right) \\
&\le\gamma\lVert v-w\rVert_{\infty}+\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}T^{\pi,P}w(s) \\
&=\gamma\lVert v-w\rVert_{\infty}+Tw(s)
\end{aligned} T v ( s ) = π ∈ Π S max P ∈ P min T π , P v ( s ) ≤ π ∈ Π S max P ∈ P min ( T π , P w ( s ) + γ ∥ v − w ∥ ∞ ) ≤ γ ∥ v − w ∥ ∞ + π ∈ Π S max P ∈ P min T π , P w ( s ) = γ ∥ v − w ∥ ∞ + T w ( s )
where the first inequality is from the contraction property of T π , P T^{\pi,P} T π , P for π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S , P ∈ P P \in\mathcal{P} P ∈ P . This shows T v ( s ) − T w ( s ) ≤ γ ∥ v − w ∥ ∞ Tv(s)-Tw(s)\leq\gamma\lVert v-w\rVert_{\infty} T v ( s ) − T w ( s ) ≤ γ ∥ v − w ∥ ∞ . Reversing the roles of v v v and w w w above yields T w ( s ) − T v ( s ) ≤ γ ∥ v − w ∥ ∞ Tw(s)-Tv(s)\leq\gamma\lVert v-w\rVert_{\infty} T w ( s ) − T v ( s ) ≤ γ ∥ v − w ∥ ∞ , so that we have proved ∥ T v − T w ∥ ∞ ≤ γ ∥ v − w ∥ ∞ \lVert Tv-Tw\rVert_{\infty}\leq\gamma\lVert v-w\rVert_{\infty} ∥ T v − T w ∥ ∞ ≤ γ ∥ v − w ∥ ∞ as claimed. Therefore there exists a unique fixed point v ⋆ ∈ R S v^{\star}\in\mathbb{R}^{S} v ⋆ ∈ R S of the operator T T T and this fixed point satisfies the equation (27) .
Note that we do not need s-rectangularity for the previous proposition; this assumption will be needed for interpreting the fixed point v ⋆ v^{\star} v ⋆ as the value function of an optimal policy. Let us first investigate the case where P \mathcal{P} P is s-rectangular but not necessarily convex. We have already seen that an optimal policy for the robust MDP problem (20) may need to be history-dependent.
Proposition 3.5. Let P ⊆ Δ ( S ) S × A \mathcal{P}\subseteq\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊆ Δ ( S ) S × A be a compact s-rectangular uncertainty set. There may not exist any stationary optimal policies, i.e., we may have
sup π ∈ Π H inf P ∈ P R ( π , P ) > max π ∈ Π S min P ∈ P R ( π , P ) . \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{P\in\mathcal{P}}R(\pi,P)>\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}R(\pi,P). π ∈ Π H sup P ∈ P inf R ( π , P ) > π ∈ Π S max P ∈ P min R ( π , P ) .
Proof. This was already stated in Proposition 2.5, where the proof considers an s-rectangular non-convex uncertainty set.
We may still wonder if there is any useful information that we can obtain from the robust Bellman operator T T T and its fixed point v ⋆ v^{\star} v ⋆ . These objects can be used to characterize “the best policy among stationary ones”, as proved next.
Proposition 3.6. Let P ⊆ Δ ( S ) S × A \mathcal{P}\subseteq\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊆ Δ ( S ) S × A be a compact s-rectangular uncertainty set. Let π ⋆ ∈ Π S \pi^{\star}\in\Pi_{\mathrm{S}} π ⋆ ∈ Π S . The two statements are equivalent:
π ⋆ ∈ Π S \pi^{\star}\in\Pi_{\mathrm{S}} π ⋆ ∈ Π S is a best stationary policy across all initial distributions:
∀ p 0 ∈ Δ ( S ) , π ⋆ ∈ arg max π ∈ Π S min P ∈ P R ( π , P ) . (28) \forall\,p_{0}\in\Delta(\mathcal{S}),\ \pi^{\star}\in\arg\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}R(\pi,P). \tag*{(28)} ∀ p 0 ∈ Δ ( S ) , π ⋆ ∈ arg π ∈ Π S max P ∈ P min R ( π , P ) . (28) T π ⋆ v ⋆ = T v ⋆ T^{\pi^{\star}}v^{\star}=Tv^{\star} T π ⋆ v ⋆ = T v ⋆ .
Proof. First, we note that the set ∩ s ∈ S arg max π ∈ Π S T π v ⋆ ( s ) \cap_{s\in\mathcal{S}}\arg\max_{\pi\in\Pi_{\mathrm{S}}}T^{\pi}v^{\star}(s) ∩ s ∈ S arg max π ∈ Π S T π v ⋆ ( s ) coincides with the set of policies π ˉ ∈ Π S \bar{\pi}\in\Pi_{\mathrm{S}} π ˉ ∈ Π S such that T v ⋆ = T π ˉ v ⋆ Tv^{\star}=T^{\bar{\pi}}v^{\star} T v ⋆ = T π ˉ v ⋆ . We show that any policy π ˉ \bar{\pi} π ˉ in ∩ s ∈ S arg max π ∈ Π S T π v ⋆ ( s ) \cap_{s\in\mathcal{S}}\arg\max_{\pi\in\Pi_{\mathrm{S}}}T^{\pi}v^{\star}(s) ∩ s ∈ S arg max π ∈ Π S T π v ⋆ ( s ) is uniformly optimal (among stationary policies); this set is non-empty because for any s ∈ S s\in\mathcal{S} s ∈ S , the scalar T π v ⋆ ( s ) T^{\pi}v^{\star}(s) T π v ⋆ ( s ) only depends on the s s s -th component of π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S , and Π S \Pi_{\mathrm{S}} Π S is a Cartesian product. Since T v ⋆ = v ⋆ T v^\star= v^\star T v ⋆ = v ⋆ by definition of v ⋆ v^\star v ⋆ , we get that T π ˉ v ⋆ = v ⋆ T^{\bar{\pi}}v^\star= v^\star T π ˉ v ⋆ = v ⋆ so that v ⋆ v^\star v ⋆ coincides with v π ˉ ∈ R S v^{\bar{\pi}} \in\mathbb{R}^{S} v π ˉ ∈ R S , the smallest worst-case value function of π ˉ \bar{\pi} π ˉ . Additionally, for any π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S we have v π = T π v π ≤ T v π v^\pi= T^\pi v^\pi\leq T v^\pi v π = T π v π ≤ T v π , where the inequality is by definition of T T T as in (26) . Hence v π ≤ T v π v^\pi\leq T v^\pi v π ≤ T v π , and since T T T is monotone we have v π ≤ T n v π v^\pi\leq T^n v^\pi v π ≤ T n v π for any n ∈ N n \in\mathbb{N} n ∈ N . Since T T T is a contraction, taking n → + ∞ n \to+\infty n → + ∞ yields v π ≤ v ⋆ = v π ˉ v^\pi\leq v^\star= v^{\bar{\pi}} v π ≤ v ⋆ = v π ˉ . Since p 0 ∈ Δ ( S ) p_0 \in\Delta(S) p 0 ∈ Δ ( S ) , we conclude that the set ⋂ s ∈ S arg max π ∈ Π S T π v ⋆ ( s ) \bigcap_{s\in S}\arg\max_{\pi\in\Pi_{\mathrm{S}}}T^\pi v^\star(s) ⋂ s ∈ S arg max π ∈ Π S T π v ⋆ ( s ) contains only policies that are uniformly optimal stationary policies.
Now let π ˉ ∈ Π S \bar{\pi} \in\Pi_{\mathrm{S}} π ˉ ∈ Π S be any stationary uniformly optimal policy. Since we must also have v π ˉ ≥ v π v^{\bar{\pi}} \geq v^\pi v π ˉ ≥ v π for any other policy π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S , we get that v π ˉ = v ⋆ v^{\bar{\pi}} = v^\star v π ˉ = v ⋆ . Therefore T π ˉ v ⋆ = T π ˉ v π ˉ = v π ˉ = v ⋆ = T v ⋆ T^{\bar{\pi}}v^\star= T^{\bar{\pi}}v^{\bar{\pi}} = v^{\bar{\pi}} = v^\star= Tv^\star T π ˉ v ⋆ = T π ˉ v π ˉ = v π ˉ = v ⋆ = T v ⋆ so that π ˉ ∈ ⋂ s ∈ S arg max π ∈ Π S T π v ⋆ ( s ) \bar{\pi} \in\bigcap_{s\in S}\arg\max_{\pi\in\Pi_{\mathrm{S}}}T^\pi v^\star(s) π ˉ ∈ ⋂ s ∈ S arg max π ∈ Π S T π v ⋆ ( s ) , which shows that ⋂ s ∈ S arg max π ∈ Π S T π v ⋆ ( s ) \bigcap_{s\in S}\arg\max_{\pi\in\Pi_{\mathrm{S}}}T^\pi v^\star(s) ⋂ s ∈ S arg max π ∈ Π S T π v ⋆ ( s ) is the set of uniformly optimal stationary policies.
Note that in the proof of the above proposition we use “optimal stationary policies” as solutions to sup π ∈ Π S min P ∈ P R ( π , P ) \sup_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}R(\pi,P) sup π ∈ Π S min P ∈ P R ( π , P ) , whereas in these notes “stationary optimal policies” means stationary solutions to sup π ∈ Π H inf P ∈ P R ( π , P ) \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{P\in\mathcal{P}}R(\pi,P) sup π ∈ Π H inf P ∈ P R ( π , P ) . From Proposition 3.5, stationary optimal policies need not exist for s-rectangular compact sets (without convexity). We show below that they exist as soon as we assume convexity.
Theorem 3.2. Let P ⊆ Δ ( S ) S × A \mathcal{P} \subseteq\Delta(S)^{S\times A} P ⊆ Δ ( S ) S × A be a compact convex s-rectangular uncertainty set. The following statements hold:
Strong duality. The following strong duality result holds:
max π ∈ Π S min P ∈ P R ( π , P ) = min P ∈ P max π ∈ Π S R ( π , P ) . (29) \max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}} R(\pi,P)=\min_{P\in\mathcal{P}}\max_{\pi\in\Pi_{\mathrm{S}}}R(\pi,P). \tag*{(29)} π ∈ Π S max P ∈ P min R ( π , P ) = P ∈ P min π ∈ Π S max R ( π , P ) . (29) Optimality of stationary policies. We have
sup π ∈ Π H inf P ∈ P R ( π , P ) = max π ∈ Π S min P ∈ P R ( π , P ) . \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{P\in\mathcal{P}}R(\pi,P)=\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}R(\pi,P). π ∈ Π H sup P ∈ P inf R ( π , P ) = π ∈ Π S max P ∈ P min R ( π , P ) . Optimality conditions. π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S is a stationary uniformly optimal policy for (20) if and only if T π v ⋆ = T v ⋆ T^\pi v^\star= Tv^\star T π v ⋆ = T v ⋆ .
Additionally, let P ⋆ ∈ P P^\star\in\mathcal{P} P ⋆ ∈ P such that T P ⋆ v ⋆ = T v ⋆ T^{P^\star}v^\star= Tv^\star T P ⋆ v ⋆ = T v ⋆ . Then: P ⋆ P^\star P ⋆ is a uniform worst-case kernel for any stationary uniformly optimal policy π ⋆ ∈ Π S \pi^\star\in\Pi_{\mathrm{S}} π ⋆ ∈ Π S , and any stationary uniformly optimal policy π ⋆ ∈ Π S \pi^\star\in\Pi_{\mathrm{S}} π ⋆ ∈ Π S is actually a nominal uniformly optimal policy in the nominal MDP with transition probabilities P ⋆ P^\star P ⋆ .
We encourage the reader to compare Theorem 3.2 with the main theorem for adversarial MDPs (Corollary 3.1) and the main theorem for nominal MDPs (Theorem 1.1); we refer to Table 3.1 for a comparison of the main objects for these three optimization problems. The remainder of this section is devoted to the proof of Theorem 3.2. Before we do so, it is worth discussing the intuition and implications of this theorem:
Contrary to the nominal case, we may need randomized policies at optimality. We already observed this in Example 2.1, where we computed the unique optimal policy and noticed it was randomized. From optimization and game-theoretic standpoints,
this is not surprising, since the robust MDP problem (20) has a “two-player zero-sum game” flavor and it is a maxmin problem. Randomization can also be understood “geometrically”: from the third statement in Theorem 3.2, a stationary uniformly optimal policy solves the maxmin problem max π ∈ Π S min P ∈ P T π , P v ⋆ ( s ) \max_{\pi\in\Pi_{\mathrm{S}}\!}\min_{P\in\mathcal{P}}T^{\pi,P}v^\star(s) max π ∈ Π S min P ∈ P T π , P v ⋆ ( s ) for each s ∈ S s\in\mathcal{S} s ∈ S , with ( π , P ) ↦ T π , P v ⋆ ( s ) (\pi,P)\mapsto T^{\pi,P}v^\star(s) ( π , P ) ↦ T π , P v ⋆ ( s ) bilinear. If we dualize the inner minimization we obtain that uniformly optimal solutions are the optimal solutions to a (larger) maximization problem over π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S and some additional constraints (coming from the dualization of the inner minimization over P ∈ P P\in\mathcal{P} P ∈ P ), and these additional constraints may prevent us from choosing deterministic policies (which correspond to the extreme points of Π S \Pi_{\mathrm{S}} Π S ).
Additionally, for ( π ⋆ , P ⋆ ) (\pi^\star,P^\star) ( π ⋆ , P ⋆ ) as in the last statement of Theorem 3.2, we get that P ⋆ P^\star P ⋆ is a worst-case transition kernel for π ⋆ \pi^\star π ⋆ , and that π ⋆ \pi^\star π ⋆ is a nominal optimal policy in the MDP with transitions P ⋆ P^\star P ⋆ . Therefore in general P ⋆ P^\star P ⋆ must induce a particular MDP instance where an optimal policy can be chosen randomized, whereas we saw in Chapter 1 that a nominal optimal policy can always be chosen deterministic; we refer to Proposition 3.7 for more detail on this.
We didn’t need the convexity assumption on P \mathcal{P} P in § 3.2 : this is because we reduced the policy evaluation problem to an MDP, and MDPs are tractable even when the set of actions is not convex (e.g. we derived the results from Chapter 1 with finite action sets). In contrast, we crucially need the convexity assumption for P \mathcal{P} P in Theorem 3.2. This is because we will rely on minimax arguments and saddle-point properties that require convex domains. For sa-rectangular sets (see next section) convexity is not needed because we don’t rely on minimax theorems.
Keep in mind that optimal policies may require irrational entries, as in Example 2.1, even for very simple convex compact s-rectangular uncertainty sets. In practice the algorithms we’ll run only compute approximately optimal policies (see § 3.3.3 ) so this is not an issue.
If one works with non-closed, convex s-rectangular uncertainty sets P \mathcal{P} P then the same techniques as we used for proving Theorem 3.2 would still show that optimal policies can be chosen stationary (although we wouldn’t guarantee the existence of a worst-case kernel anymore). This situation is akin to the case of robust MDPs with mean-payoff objective, see § 5.1 .
It is also worth discussing the interpretation of strong duality from a few different angles:
From an optimization standpoint, it is important to emphasize that in principle we do not expect such a maxmin duality result to hold: while P \mathcal{P} P and Π S \Pi_{\mathrm{S}} Π S are convex compact sets, the function ( π , P ) ↦ R ( π , P ) (\pi,P)\mapsto R(\pi,P) ( π , P ) ↦ R ( π , P ) need not be concave-convex (nor quasi-concave quasi-convex for that matter, see § 3.8 ), so the usual von Neumann or Sion’s minimax theorems do not apply readily.
From a game-theoretic standpoint, the left-hand side is the best worst-case return over stationary policies, i.e., the left-hand side is the largest scalar value that the decision maker can be guaranteed to obtain if the decision maker first chooses π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S and then the adversary chooses P ∈ P P\in\mathcal{P} P ∈ P . In contrast, the right-hand side is the worst-case best return, i.e., if the adversary first chooses P ∈ P P\in\mathcal{P} P ∈ P and then the decision maker chooses π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S , the right-hand side is the smallest scalar value that the adversary can “pay” to the decision maker. While it is clear that the right-hand side (“worst-case best return”) is always greater than the left-hand side (“best worst-case return”), it is by no means obvious that these two quantities are equal.
It is also crucial to reckon that (3.7) is equivalent to the existence of a “Nash equilibrium”, i.e., to a pair ( π ⋆ , P ⋆ ) ∈ Π S × P (\pi^\star,P^\star)\in\Pi_{\mathrm{S}}\times\mathcal{P} ( π ⋆ , P ⋆ ) ∈ Π S × P that are “best responses” to each other (I use quotation marks because the concept of best response is for games and I did not rigorously define the game here). We make this formal in the next proposition.
Proposition 3.7. Let P ⊆ Δ ( S ) S × A \mathcal{P}\subseteq\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊆ Δ ( S ) S × A be a compact set.
The following statements are equivalent:
(a) Strong duality holds: max π ∈ Π S min P ∈ P R ( π , P ) = min P ∈ P max π ∈ Π S R ( π , P ) \max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}R(\pi,P)=\min_{P\in\mathcal{P}}\max_{\pi\in\Pi_{\mathrm{S}}}R(\pi,P) max π ∈ Π S min P ∈ P R ( π , P ) = min P ∈ P max π ∈ Π S R ( π , P ) .
(b) There exists a pair ( π ⋆ , P ⋆ ) ∈ Π S × P (\pi^\star,P^\star)\in\Pi_{\mathrm{S}}\times\mathcal{P} ( π ⋆ , P ⋆ ) ∈ Π S × P such that
π ⋆ ∈ arg max π ∈ Π S R ( π , P ⋆ ) P ⋆ ∈ arg min P ∈ P R ( π ⋆ , P ) (30) \begin{aligned}
\pi^\star&\in\operatorname*{arg\,max}_{\pi\in\Pi_{\mathrm{S}}} R(\pi,P^\star) \\
P^\star&\in\operatorname*{arg\,min}_{P\in\mathcal{P}} R(\pi^\star,P) \tag*{(30)}
\end{aligned} π ⋆ P ⋆ ∈ π ∈ Π S arg max R ( π , P ⋆ ) ∈ P ∈ P arg min R ( π ⋆ , P ) (30)
If strong duality holds, the pairs ( π ⋆ , P ⋆ ) ∈ Π S × P (\pi^\star, P^\star) \in\Pi_{\mathrm{S}} \times\mathcal{P} ( π ⋆ , P ⋆ ) ∈ Π S × P satisfying (30) are exactly the pairs ( π ⋆ , P ⋆ ) ∈ Π S × P (\pi^\star, P^\star) \in\Pi_{\mathrm{S}} \times\mathcal{P} ( π ⋆ , P ⋆ ) ∈ Π S × P satisfying
π ⋆ ∈ arg max π ∈ Π S min P ∈ P R ( π , P ) \pi^\star\in\underset{\pi\in\Pi_{\mathrm{S}}}{\arg\max}\ \underset{P \in\mathcal{P}}{\min}\ R(\pi,P) π ⋆ ∈ π ∈ Π S arg max P ∈ P min R ( π , P )
P ⋆ ∈ arg min P ∈ P max π ∈ Π S R ( π , P ) . (31) P^\star\in\underset{P \in\mathcal{P}}{\arg\min}\ \underset{\pi\in\Pi_{\mathrm{S}}}{\max}\ R(\pi,P). \tag*{(31)} P ⋆ ∈ P ∈ P arg min π ∈ Π S max R ( π , P ) . (31)
Proof. 1. We first prove the first implication. Assume that strong duality holds and let ( π ⋆ , P ⋆ ) (\pi^\star, P^\star) ( π ⋆ , P ⋆ ) be as in (31) ; we can always find such a pair of policy-kernel because Π S , P \Pi_{\mathrm{S}}, \mathcal{P} Π S , P are compact and ( π , P ) ↦ R ( π , P ) (\pi, P) \mapsto R(\pi,P) ( π , P ) ↦ R ( π , P ) is continuous.
Then we have
max π ∈ Π S min P ∈ P R ( π , P ) = min P ∈ P R ( π ⋆ , P ) ≤ R ( π ⋆ , P ⋆ ) ≤ max π ∈ Π S R ( π , P ⋆ ) = min P ∈ P max π ∈ Π S R ( π , P ) \underset{\pi\in\Pi_{\mathrm{S}}}{\max}\ \underset{P \in\mathcal{P}}{\min}\ R(\pi,P) = \underset{P \in\mathcal{P}}{\min}\ R(\pi^\star,P) \le R(\pi^\star,P^\star) \le\underset{\pi\in\Pi_{\mathrm{S}}}{\max}\ R(\pi,P^\star) = \underset{P \in\mathcal{P}}{\min}\ \underset{\pi\in\Pi_{\mathrm{S}}}{\max}\ R(\pi,P) π ∈ Π S max P ∈ P min R ( π , P ) = P ∈ P min R ( π ⋆ , P ) ≤ R ( π ⋆ , P ⋆ ) ≤ π ∈ Π S max R ( π , P ⋆ ) = P ∈ P min π ∈ Π S max R ( π , P )
Since we assumed strong duality holds then all terms above are equal so that
R ( π ⋆ , P ⋆ ) = max π ∈ Π S R ( π , P ⋆ ) = min P ∈ P R ( π ⋆ , P ) R(\pi^\star,P^\star) = \underset{\pi\in\Pi_{\mathrm{S}}}{\max}\ R(\pi,P^\star) = \underset{P \in\mathcal{P}}{\min}\ R(\pi^\star,P) R ( π ⋆ , P ⋆ ) = π ∈ Π S max R ( π , P ⋆ ) = P ∈ P min R ( π ⋆ , P )
and we have proved (30) .
Let us now assume the existence of ( π ⋆ , P ⋆ ) (\pi^\star, P^\star) ( π ⋆ , P ⋆ ) satisfying (30) . Then
max π ∈ Π S min P ∈ P R ( π , P ) ≥ min P ∈ P R ( π ⋆ , P ) = R ( π ⋆ , P ⋆ ) = max π ∈ Π S R ( π , P ⋆ ) ≥ min P ∈ P max π ∈ Π S R ( π , P ) \underset{\pi\in\Pi_{\mathrm{S}}}{\max}\ \underset{P \in\mathcal{P}}{\min}\ R(\pi,P) \ge\underset{P \in\mathcal{P}}{\min}\ R(\pi^\star,P) = R(\pi^\star,P^\star) = \underset{\pi\in\Pi_{\mathrm{S}}}{\max}\ R(\pi,P^\star) \ge\underset{P \in\mathcal{P}}{\min}\ \underset{\pi\in\Pi_{\mathrm{S}}}{\max}\ R(\pi,P) π ∈ Π S max P ∈ P min R ( π , P ) ≥ P ∈ P min R ( π ⋆ , P ) = R ( π ⋆ , P ⋆ ) = π ∈ Π S max R ( π , P ⋆ ) ≥ P ∈ P min π ∈ Π S max R ( π , P )
and from weak duality we get that all terms above are equal, which shows that strong duality holds.
This follows from the analysis of the first statement of this proposition.
Note that the second statement in the above proposition shows the property that the set of pairs ( π ⋆ , P ⋆ ) ∈ Π S × P (\pi^\star, P^\star) \in\Pi_{\mathrm{S}} \times\mathcal{P} ( π ⋆ , P ⋆ ) ∈ Π S × P that satisfy (30) is actually a Cartesian product; this is surprising since (30) states that π ⋆ , P ⋆ \pi^\star, P^\star π ⋆ , P ⋆ are best responses to each other.
Finally, note that arg min P ∈ P R ( π ⋆ , P ) \arg\min_{P \in\mathcal{P}} R(\pi^\star,P) arg min P ∈ P R ( π ⋆ , P ) , the set of worst-case kernels for π ⋆ \pi^\star π ⋆ a solution to the maxmin problem, contains at least one P ⋆ ∈ P P^\star\in\mathcal{P} P ⋆ ∈ P such that P ⋆ ∈ arg min P ∈ P max π ∈ Π S R ( π , P ) P^\star\in\arg\min_{P \in\mathcal{P}} \max_{\pi\in\Pi_{\mathrm{S}}} R(\pi,P) P ⋆ ∈ arg min P ∈ P max π ∈ Π S R ( π , P ) , but this need not be the case for all elements in arg min P ∈ P R ( π ⋆ , P ) \arg\min_{P \in\mathcal{P}} R(\pi^\star,P) arg min P ∈ P R ( π ⋆ , P ) . This situation is analogous for Nash equilibria in finite zero-sum games: the set of best-responses to a Nash equilibrium contains a Nash equilibrium (by definition) but may contain other strategy profiles too.
Proof of strong duality
We give two different proofs of strong duality. The first proof is based on ideas from robust optimization. The second proof is based on Kakutani’s fixed point theorem. A third proof based on Brouwer’s fixed point theorem is suggested in § 3.8 .
First proof of strong duality (based on robust optimization). We will make repeated use of the contraction lemma (Lemma 1.2).
max π ∈ Π S min P ∈ P R ( π , P ) = max π ∈ Π S min P ∈ P p 0 T v π , P = max π ∈ Π S min P ∈ P { p 0 T v ∣ v ( s ) ≥ T π , P v ( s ) ∀ s ∈ S , v ∈ R S } = max π ∈ Π S min { p 0 T v ∣ v ( s ) ≥ min P ∈ P T π , P v ( s ) ∀ s ∈ S , v ∈ R S } = min { p 0 T v ∣ v ( s ) ≥ max π ∈ Π S min P ∈ P T π , P v ( s ) ∀ s ∈ S , v ∈ R S } = min { p 0 T v ∣ v ( s ) ≥ min P ∈ P max π ∈ Π S T π , P v ( s ) ∀ s ∈ S , v ∈ R S } = min P ∈ P min { p 0 T v ∣ v ( s ) ≥ max π ∈ Π S T π , P v ( s ) ∀ s ∈ S , v ∈ R S } = min P ∈ P max π ∈ Π S p 0 T v π , P = min P ∈ P max π ∈ Π S R ( π , P ) \begin{aligned}
\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}R(\pi,P)
&=\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}p_0^{\mathsf{T}}v^{\pi,P}\\
&=\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}\left\{p_0^{\mathsf{T}}v\mid v(s)\geq T^{\pi,P}v(s)\ \forall\ s\in\mathcal{S},v\in\mathbb{R}^{\mathcal{S}}\right\}\\
&=\max_{\pi\in\Pi_{\mathrm{S}}}\min\left\{p_0^{\mathsf{T}}v\mid v(s)\geq\min_{P\in\mathcal{P}}T^{\pi,P}v(s)\ \forall\ s\in\mathcal{S},v\in\mathbb{R}^{\mathcal{S}}\right\}\\
&=\min\left\{p_0^{\mathsf{T}}v\mid v(s)\geq\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}T^{\pi,P}v(s)\ \forall\ s\in\mathcal{S},v\in\mathbb{R}^{\mathcal{S}}\right\}\\
&=\min\left\{p_0^{\mathsf{T}}v\mid v(s)\geq\min_{P\in\mathcal{P}}\max_{\pi\in\Pi_{\mathrm{S}}}T^{\pi,P}v(s)\ \forall\ s\in\mathcal{S},v\in\mathbb{R}^{\mathcal{S}}\right\}\\
&=\min_{P\in\mathcal{P}}\min\left\{p_0^{\mathsf{T}}v\mid v(s)\geq\max_{\pi\in\Pi_{\mathrm{S}}}T^{\pi,P}v(s)\ \forall\ s\in\mathcal{S},v\in\mathbb{R}^{\mathcal{S}}\right\}\\
&=\min_{P\in\mathcal{P}}\max_{\pi\in\Pi_{\mathrm{S}}}p_0^{\mathsf{T}}v^{\pi,P}\\
&=\min_{P\in\mathcal{P}}\max_{\pi\in\Pi_{\mathrm{S}}}R(\pi,P)
\end{aligned} π ∈ Π S max P ∈ P min R ( π , P ) = π ∈ Π S max P ∈ P min p 0 T v π , P = π ∈ Π S max P ∈ P min { p 0 T v ∣ v ( s ) ≥ T π , P v ( s ) ∀ s ∈ S , v ∈ R S } = π ∈ Π S max min { p 0 T v ∣ v ( s ) ≥ P ∈ P min T π , P v ( s ) ∀ s ∈ S , v ∈ R S } = min { p 0 T v ∣ v ( s ) ≥ π ∈ Π S max P ∈ P min T π , P v ( s ) ∀ s ∈ S , v ∈ R S } = min { p 0 T v ∣ v ( s ) ≥ P ∈ P min π ∈ Π S max T π , P v ( s ) ∀ s ∈ S , v ∈ R S } = P ∈ P min min { p 0 T v ∣ v ( s ) ≥ π ∈ Π S max T π , P v ( s ) ∀ s ∈ S , v ∈ R S } = P ∈ P min π ∈ Π S max p 0 T v π , P = P ∈ P min π ∈ Π S max R ( π , P )
where the first equality is by definition of the return R ( π , P ) R(\pi,P) R ( π , P ) and the value function v π , P ∈ R S v^{\pi,P}\in\mathbb{R}^{\mathcal{S}} v π , P ∈ R S , the second equality is by the contraction lemma applied to the fixed point v π , P v^{\pi,P} v π , P of the monotone contraction T π , P T^{\pi,P} T π , P , the third and fourth equalities will be justified next, the fifth equality is from Sion’s minimax theorem applied to the concave-convex (bilinear) functions ( π , P ) ↦ T π , P v ( s ) (\pi,P)\mapsto T^{\pi,P}v(s) ( π , P ) ↦ T π , P v ( s ) for each s ∈ S s\in\mathcal{S} s ∈ S and fixed v ∈ R S v\in\mathbb{R}^{\mathcal{S}} v ∈ R S (see Appendix § 7 ), the sixth and seventh equalities will be justified next, and the last two equalities are again by the contraction lemma and the definition of the return and value functions.
We are left with justifying the third, fourth, sixth, and seventh equalities.
The third and the sixth proceed from the same argument, which is that P \mathcal{P} P is s s s -rectangular so that the s s s -th constraint v ( s ) ≥ T π , P v ( s ) v(s)\geq T^{\pi,P}v(s) v ( s ) ≥ T π , P v ( s ) only depends on the s s s -th component of P P P , and a minimization program over two variables (here P ∈ P P\in\mathcal{P} P ∈ P and v ∈ R S v\in\mathbb{R}^{\mathcal{S}} v ∈ R S ) can be written as two consecutive minimization programs.
The fourth and seventh equalities proceed from the following facts: (a) the rectangularity of the set Π S = × s ∈ S Δ ( A ) \Pi_{\mathrm{S}}=\mathop{\times}_{s\in\mathcal{S}}\Delta(\mathcal{A}) Π S = × s ∈ S Δ ( A ) (b) the s s s -th constraint v ( s ) ≥ min P ∈ P T π , P v ( s ) v(s)\geq\min_{P\in\mathcal{P}}T^{\pi,P}v(s) v ( s ) ≥ min P ∈ P T π , P v ( s ) only depends on the s s s -th component of π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S (c) the outer maximum over π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S is always attained in the fourth and seventh equalities, i.e., there exists π ′ ∈ Π S \pi'\in\Pi_{\mathrm{S}} π ′ ∈ Π S such that
max π ∈ Π S min { p 0 T v ∣ v ( s ) ≥ T π , P v ( s ) ∀ s ∈ S , v ∈ R S } \max_{\pi\in\Pi_{\mathrm{S}}}\min\left\{p_0^{\mathsf{T}}v\mid v(s)\geq T^{\pi,P}v(s)\ \forall\ s\in\mathcal{S},v\in\mathbb{R}^{\mathcal{S}}\right\} π ∈ Π S max min { p 0 T v ∣ v ( s ) ≥ T π , P v ( s ) ∀ s ∈ S , v ∈ R S }
is equal to
min { p 0 T v ∣ v ( s ) ≥ T π ′ , P v ( s ) ∀ s ∈ S , v ∈ R S } \min\left\{p_0^{\mathsf{T}}v\mid v(s)\geq T^{\pi',P}v(s)\ \forall\ s\in\mathcal{S},v\in\mathbb{R}^{\mathcal{S}}\right\} min { p 0 T v ∣ v ( s ) ≥ T π ′ , P v ( s ) ∀ s ∈ S , v ∈ R S }
and there exists π ′ ′ ∈ Π S \pi''\in\Pi_{\mathrm{S}} π ′′ ∈ Π S such that
max π ∈ Π S min { p 0 T v ∣ v ( s ) ≥ min P ∈ P T π , P v ( s ) ∀ s ∈ S , v ∈ R S } \max_{\pi\in\Pi_{\mathrm{S}}}\min\left\{p_0^{\mathsf{T}}v\mid v(s)\geq\min_{P\in\mathcal{P}}T^{\pi,P}v(s)\ \forall\ s\in\mathcal{S},v\in\mathbb{R}^{\mathcal{S}}\right\} π ∈ Π S max min { p 0 T v ∣ v ( s ) ≥ P ∈ P min T π , P v ( s ) ∀ s ∈ S , v ∈ R S }
is equal to
min { p 0 T v ∣ v ( s ) ≥ min P ∈ P T π ′ ′ , P v ( s ) ∀ s ∈ S , v ∈ R S } . \min\left\{p_0^{\mathsf{T}}v\mid v(s)\geq\min_{P\in\mathcal{P}}T^{\pi'',P}v(s)\ \forall\ s\in\mathcal{S},v\in\mathbb{R}^{\mathcal{S}}\right\}. min { p 0 T v ∣ v ( s ) ≥ P ∈ P min T π ′′ , P v ( s ) ∀ s ∈ S , v ∈ R S } .
Notice the similarity between these last two facts and the attainability property; this concludes the proof.
In the proof above, we emphasize that we use Sion’s minimax theorem to prove strong duality as in (29) , except that we can’t apply it directly to ( π , P ) ↦ R ( π , P ) (\pi, P) \mapsto R(\pi, P) ( π , P ) ↦ R ( π , P ) but we can apply it to ( π , P ) ↦ T π , P v ( s ) (\pi, P) \mapsto T^{\pi,P}v(s) ( π , P ) ↦ T π , P v ( s ) for s ∈ S s \in\mathcal{S} s ∈ S and a fixed v ∈ R S v \in\mathbb{R}^{\mathcal{S}} v ∈ R S .
The second proof is an extension of the original proof for the existence of Nash equilibria in finite games to the case of robust MDPs. We write 2 X 2^{\mathcal{X}} 2 X for the power set of a set X \mathcal{X} X , i.e., the set of all subsets of X \mathcal{X} X . We need the notion of closed graph for set-valued functions, defined next.
Definition 3.3. Let X ⊂ R n \mathcal{X} \subset\mathbb{R}^{n} X ⊂ R n be a compact set and F : X → 2 X F : \mathcal{X} \to2^{\mathcal{X}} F : X → 2 X . We say that F F F has a closed graph if for any sequences ( x n ) n ∈ N (x_n)_{n\in\mathbb{N}} ( x n ) n ∈ N and ( y n ) n ∈ N (y_n)_{n\in\mathbb{N}} ( y n ) n ∈ N such that ∀ n ∈ N , y n ∈ F ( x n ) \forall n \in\mathbb{N}, y_n \in F(x_n) ∀ n ∈ N , y n ∈ F ( x n ) and lim n → + ∞ x n = x ∈ X \lim_{n\to+\infty}x_n = x \in\mathcal{X} lim n → + ∞ x n = x ∈ X and lim n → + ∞ y n = y \lim_{n\to+\infty}y_n = y lim n → + ∞ y n = y , we have y ∈ F ( x ) y \in F(x) y ∈ F ( x ) .
We can now state Kakutani’s fixed point theorem.
Theorem 3.3 (Kakutani’s fixed point theorem). Let X ⊂ R n \mathcal{X} \subset\mathbb{R}^{n} X ⊂ R n be a nonempty compact convex set. Let F : X → 2 X F : \mathcal{X} \to2^{\mathcal{X}} F : X → 2 X be a function with a closed graph and assume that for any x ∈ X x \in\mathcal{X} x ∈ X , F ( x ) F(x) F ( x ) is a nonempty closed convex set. Then there exists x ∈ X x \in\mathcal{X} x ∈ X such that x ∈ F ( x ) x \in F(x) x ∈ F ( x ) .
Second proof of strong duality (based on Kakutani). We want to use Kakutani’s fixed point theorem to show the existence of a pair ( π ⋆ , P ⋆ ) ∈ Π S × P (\pi^\star, P^\star) \in\Pi_{\mathcal{S}} \times\mathcal{P} ( π ⋆ , P ⋆ ) ∈ Π S × P such that π ⋆ \pi^\star π ⋆ is an optimal policy for P ⋆ P^\star P ⋆ and P ⋆ P^\star P ⋆ is a worst-case transition kernel for π ⋆ \pi^\star π ⋆ :
π ⋆ ∈ arg max π ∈ Π S R ( π , P ⋆ ) , P ⋆ ∈ arg min P ∈ P R ( π ⋆ , P ) . \pi^\star\in\arg\max_{\pi\in\Pi_{\mathcal{S}}} R(\pi,P^\star), \quad P^\star\in\arg\min_{P\in\mathcal{P}} R(\pi^\star,P). π ⋆ ∈ arg π ∈ Π S max R ( π , P ⋆ ) , P ⋆ ∈ arg P ∈ P min R ( π ⋆ , P ) .
In fact we will show a stronger result: we can pick such π ⋆ , P ⋆ \pi^\star, P^\star π ⋆ , P ⋆ as uniformly optimal policies and worst-case transition probabilities. For ( π , P ) ∈ Π S × P (\pi, P) \in\Pi_{\mathcal{S}} \times\mathcal{P} ( π , P ) ∈ Π S × P we let Π S ⋆ ( P ) \Pi^\star_{\mathcal{S}}(P) Π S ⋆ ( P ) be the set of uniformly optimal stationary policies when the transition kernel is P P P , and P ⋆ ( π ) \mathcal{P}^\star(\pi) P ⋆ ( π ) be the set of uniform worst-case transition probabilities for the policy π \pi π . Note that both sets are the sets of uniformly optimal policies for some MDP instances with finitely many states, compact action sets, and continuous rewards and transitions (for Π S ⋆ ( P ) \Pi^\star_{\mathcal{S}}(P) Π S ⋆ ( P ) this is direct, and for P ⋆ ( π ) \mathcal{P}^\star(\pi) P ⋆ ( π ) this is from the interpretation of the policy evaluation problem as an adversarial MDP). From Lemma 1.1 we get that both Π S ⋆ ( P ) \Pi^\star_{\mathcal{S}}(P) Π S ⋆ ( P ) and P ⋆ ( π ) \mathcal{P}^\star(\pi) P ⋆ ( π ) are convex, closed, and always non-empty.
Let F : Π S × P → 2 Π S × P F : \Pi_{\mathcal{S}} \times\mathcal{P} \to2^{\Pi_{\mathcal{S}}\times\mathcal{P}} F : Π S × P → 2 Π S × P be defined as F ( π ˉ , P ˉ ) = Π S ⋆ ( P ˉ ) × P ⋆ ( π ˉ ) F(\bar{\pi}, \bar{P}) = \Pi^\star_{\mathcal{S}}(\bar{P}) \times\mathcal{P}^\star(\bar{\pi}) F ( π ˉ , P ˉ ) = Π S ⋆ ( P ˉ ) × P ⋆ ( π ˉ ) . To apply Theorem 3.3, it remains to show that F F F has a closed graph. Let ( π n , P n ) ∈ Π S × P (\pi_n, P_n) \in\Pi_{\mathcal{S}} \times\mathcal{P} ( π n , P n ) ∈ Π S × P and ( π ^ n , P ^ n ) ∈ Π S ⋆ ( P n ) × P ⋆ ( π n ) (\hat{\pi}_n, \hat{P}_n) \in\Pi^\star_{\mathcal{S}}(P_n) \times\mathcal{P}^\star(\pi_n) ( π ^ n , P ^ n ) ∈ Π S ⋆ ( P n ) × P ⋆ ( π n ) for n ∈ N n \in\mathbb{N} n ∈ N . Let ( π , P ) = lim n → + ∞ ( π n , P n ) (\pi, P) = \lim_{n\to+\infty}(\pi_n, P_n) ( π , P ) = lim n → + ∞ ( π n , P n ) and ( π ^ , P ^ ) = lim n → + ∞ ( π ^ n , P ^ n ) (\hat{\pi}, \hat{P}) = \lim_{n\to+\infty}(\hat{\pi}_n, \hat{P}_n) ( π ^ , P ^ ) = lim n → + ∞ ( π ^ n , P ^ n ) (the limits are assumed to exist). We want to show that ( π ^ , P ^ ) ∈ Π S ⋆ ( P ) × P ⋆ ( π ) (\hat{\pi}, \hat{P}) \in\Pi^\star_{\mathcal{S}}(P) \times\mathcal{P}^\star(\pi) ( π ^ , P ^ ) ∈ Π S ⋆ ( P ) × P ⋆ ( π ) . From ( π ^ n , P ^ n ) ∈ Π S ⋆ ( P n ) × P ⋆ ( π n ) (\hat{\pi}_n, \hat{P}_n) \in\Pi^\star_{\mathcal{S}}(P_n) \times\mathcal{P}^\star(\pi_n) ( π ^ n , P ^ n ) ∈ Π S ⋆ ( P n ) × P ⋆ ( π n ) , we know that for each n ∈ N n \in\mathbb{N} n ∈ N we have
T π ^ n , P ^ n v π ^ n , P ^ n ( s ) ≥ T π ′ , P n v π ^ n , P ^ n ( s ) , ∀ s ∈ S , ∀ π ′ ∈ Π S , T^{\hat{\pi}_n,\hat{P}_n}v^{\hat{\pi}_n,\hat{P}_n}(s) \ge T^{\pi',P_n}v^{\hat{\pi}_n,\hat{P}_n}(s), \ \forall s \in\mathcal{S},\ \forall\pi' \in\Pi_{\mathcal{S}}, T π ^ n , P ^ n v π ^ n , P ^ n ( s ) ≥ T π ′ , P n v π ^ n , P ^ n ( s ) , ∀ s ∈ S , ∀ π ′ ∈ Π S ,
T π n , P ′ v π n , P ^ n ( s ) ≥ T π n , P ^ n v π n , P ^ n ( s ) , ∀ s ∈ S , ∀ P ′ ∈ P . T^{\pi_n,P'}v^{\pi_n,\hat{P}_n}(s) \ge T^{\pi_n,\hat{P}_n}v^{\pi_n,\hat{P}_n}(s), \ \forall s \in\mathcal{S},\ \forall P' \in\mathcal{P}. T π n , P ′ v π n , P ^ n ( s ) ≥ T π n , P ^ n v π n , P ^ n ( s ) , ∀ s ∈ S , ∀ P ′ ∈ P .
Since ( π , P ) ↦ T π , P v π , P ( s ) (\pi, P) \mapsto T^{\pi,P}v^{\pi,P}(s) ( π , P ) ↦ T π , P v π , P ( s ) is a continuous function for each s ∈ S s \in\mathcal{S} s ∈ S , taking the limit as n → + ∞ n \to+\infty n → + ∞ in the two inequalities above gives
T π ^ , P v π ^ , P ( s ) ≥ T π ′ , P v π ^ , P ( s ) , ∀ s ∈ S , ∀ π ′ ∈ Π S , T π , P ^ v π , P ^ ( s ) ≥ T π , P ^ ′ v π , P ^ ( s ) , ∀ s ∈ S , ∀ P ′ ∈ P . \begin{aligned}
T^{\hat{\pi},P}v^{\hat{\pi},P}(s) &\geq T^{\pi',P}v^{\hat{\pi},P}(s), &&\forall\ s \in\mathcal{S},\ \forall\ \pi' \in\Pi_{\mathcal{S}},\\
T^{\pi,\hat{P}}v^{\pi,\hat{P}}(s) &\geq T^{\pi,\hat{P}'}v^{\pi,\hat{P}}(s), &&\forall\ s \in\mathcal{S},\ \forall\ P' \in\mathcal{P}.
\end{aligned} T π ^ , P v π ^ , P ( s ) T π , P ^ v π , P ^ ( s ) ≥ T π ′ , P v π ^ , P ( s ) , ≥ T π , P ^ ′ v π , P ^ ( s ) , ∀ s ∈ S , ∀ π ′ ∈ Π S , ∀ s ∈ S , ∀ P ′ ∈ P .
which shows that π ^ ∈ Π S ⋆ ( P ) \hat{\pi} \in\Pi_{\mathcal{S}}^{\star}(P) π ^ ∈ Π S ⋆ ( P ) and P ^ ∈ P ⋆ ( π ) \hat{P} \in\mathcal{P}^{\star}(\pi) P ^ ∈ P ⋆ ( π ) , so that F F F has a closed graph. We can therefore apply Theorem 3.3 to obtain the existence of a pair ( π ⋆ , P ⋆ ) ∈ Π S × P (\pi^{\star}, P^{\star}) \in\Pi_{\mathcal{S}} \times\mathcal{P} ( π ⋆ , P ⋆ ) ∈ Π S × P such that π ⋆ ∈ Π S ⋆ ( P ⋆ ) , P ⋆ ∈ P ⋆ ( π ⋆ ) \pi^{\star} \in\Pi_{\mathcal{S}}^{\star}(P^{\star}), P^{\star} \in\mathcal{P}^{\star}(\pi^{\star}) π ⋆ ∈ Π S ⋆ ( P ⋆ ) , P ⋆ ∈ P ⋆ ( π ⋆ ) , from which we conclude that strong duality holds following Proposition 3.7.
Remark 3.4. Recall that from Chapter 1 we know that given a nominal MDP, the set of optimal policies may not be convex (see Lemma 1.1). Hence we cannot apply Kakutani’s fixed point theorem to the correspondence that maps ( π ˉ , P ˉ ) ∈ Π S × P (\bar{\pi}, \bar{P}) \in\Pi_{\mathcal{S}} \times\mathcal{P} ( π ˉ , P ˉ ) ∈ Π S × P to the Cartesian product of the set of optimal policies for P ˉ \bar{P} P ˉ and the set of worst-case transition probabilities for π ˉ \bar{\pi} π ˉ , i.e., to ( π ˉ , P ˉ ) ↦ arg max π ∈ Π S R ( π , P ˉ ) × arg min P ∈ P R ( π ˉ , P ) (\bar{\pi}, \bar{P}) \mapsto\arg\max_{\pi\in\Pi_{\mathcal{S}}} R(\pi,\bar{P}) \times\arg\min_{P \in\mathcal{P}} R(\bar{\pi},P) ( π ˉ , P ˉ ) ↦ arg max π ∈ Π S R ( π , P ˉ ) × arg min P ∈ P R ( π ˉ , P ) . This is why we considered the sets of uniformly optimal policies and worst-case transition probabilities, which are convex sets from Lemma 1.1.
Proof of uniform optimality and optimality conditions
We first prove that stationary optimal (not necessarily uniformly optimal) policies exist. We have
sup π ∈ Π H inf P ∈ P R ( π , P ) ≤ inf P ∈ P sup π ∈ Π H R ( π , P ) ≤ inf P ∈ P max π ∈ Π S R ( π , P ) = max π ∈ Π S min P ∈ P R ( π , P ) ≤ sup π ∈ Π H inf P ∈ P R ( π , P ) . \begin{aligned}
\sup_{\pi\in\Pi_{\mathcal{H}}}\inf_{P \in\mathcal{P}} R(\pi,P)
&\leq\inf_{P \in\mathcal{P}}\sup_{\pi\in\Pi_{\mathcal{H}}} R(\pi,P)\\
&\leq\inf_{P \in\mathcal{P}}\max_{\pi\in\Pi_{\mathcal{S}}} R(\pi,P)\\
&= \max_{\pi\in\Pi_{\mathcal{S}}}\min_{P \in\mathcal{P}} R(\pi,P)\\
&\leq\sup_{\pi\in\Pi_{\mathcal{H}}}\inf_{P \in\mathcal{P}} R(\pi,P).
\end{aligned} π ∈ Π H sup P ∈ P inf R ( π , P ) ≤ P ∈ P inf π ∈ Π H sup R ( π , P ) ≤ P ∈ P inf π ∈ Π S max R ( π , P ) = π ∈ Π S max P ∈ P min R ( π , P ) ≤ π ∈ Π H sup P ∈ P inf R ( π , P ) .
where the first inequality is always true from weak duality, the second inequality comes from the inner problem being a nominal MDP and therefore sup π ∈ Π H R ( π , P ) = max π ∈ Π S R ( π , P ) \sup_{\pi\in\Pi_{\mathcal{H}}} R(\pi,P) = \max_{\pi\in\Pi_{\mathcal{S}}} R(\pi,P) sup π ∈ Π H R ( π , P ) = max π ∈ Π S R ( π , P ) for any fixed P ∈ P P \in\mathcal{P} P ∈ P , the equality comes from the strong duality result, and the last inequality comes from Π S ⊂ Π H \Pi_{\mathcal{S}} \subset\Pi_{\mathcal{H}} Π S ⊂ Π H . We conclude that all terms in the equations above are equal, hence sup π ∈ Π H inf P ∈ P R ( π , P ) = max π ∈ Π S inf P ∈ P R ( π , P ) \sup_{\pi\in\Pi_{\mathcal{H}}}\inf_{P \in\mathcal{P}} R(\pi,P) = \max_{\pi\in\Pi_{\mathcal{S}}}\inf_{P \in\mathcal{P}} R(\pi,P) sup π ∈ Π H inf P ∈ P R ( π , P ) = max π ∈ Π S inf P ∈ P R ( π , P ) so that there exists a stationary optimal policy. Proposition 3.6 shows that the problem max π ∈ Π S min P ∈ P R ( π , P ) \max_{\pi\in\Pi_{\mathcal{S}}}\min_{P \in\mathcal{P}} R(\pi,P) max π ∈ Π S min P ∈ P R ( π , P ) admits stationary uniformly optimal policies that are characterized as solutions to T π v ⋆ = T v ⋆ T^{\pi}v^{\star} = Tv^{\star} T π v ⋆ = T v ⋆ (as an equation in π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S ). The statement about P ⋆ ∈ P P^{\star} \in\mathcal{P} P ⋆ ∈ P such that T P ⋆ v ⋆ = T v ⋆ T^{P^{\star}}v^{\star} = Tv^{\star} T P ⋆ v ⋆ = T v ⋆ follows analogously by (30) in Proposition 3.7.
Computing optimal policies
Theorem 3.2 shows that we can compute optimal policies for (20) by computing the fixed point v ⋆ ∈ R S v^{\star} \in\mathbb{R}^{\mathcal{S}} v ⋆ ∈ R S of the robust Bellman operator T T T defined in (26) . For nominal MDPs, we saw in § 1.3.1 that this can be done efficiently with value iteration, policy iteration, and linear programming. We describe the corresponding algorithms for robust MDPs next.
Iterative algorithms. The proofs of convergence of value iteration and policy iteration only rely on the fact that we want to compute the fixed point of an optimality operator that is attainable by a set of monotone contractions. In particular, the family { T π ∣ π ∈ Π S } \{T^\pi\mid\pi\in\Pi_{\mathcal{S}}\} { T π ∣ π ∈ Π S } satisfies the attainability property with respect to T T T :
Lemma 3.2. Let P \mathcal{P} P be a compact uncertainty set. For any v ∈ R S v \in\mathbb{R}^{\mathcal{S}} v ∈ R S , there exists π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S such that T v = T π v Tv = T^\pi v T v = T π v .
Proof. The proof follows directly from the compactness of Π S \Pi_{\mathcal{S}} Π S , the fact that the s s s -th component of T π T^\pi T π only depends on π ( s ) \pi(s) π ( s ) , the fact that the s s s -th component of T T T optimizes over π ( s ) ∈ Δ ( A ) \pi(s) \in\Delta(\mathcal{A}) π ( s ) ∈ Δ ( A ) , and the fact that Π S \Pi_{\mathcal{S}} Π S is a rectangular set: Π S = × s ∈ S Δ ( A ) \Pi_{\mathcal{S}} = \times_{s \in\mathcal{S}}\Delta(\mathcal{A}) Π S = × s ∈ S Δ ( A ) .
Similarly to the case of nominal MDPs and adversarial MDPs, the convergence of value iteration and policy iteration for robust MDPs (Algorithm 5 and Algorithm 6) follows from the general convergence theorems given in Appendix § 8 .
Algorithm 5 Value iteration for robust MDPs
Initialize v 0 ∈ R S v^0 \in\mathbb{R}^{\mathcal{S}} v 0 ∈ R S .
for t ∈ N t \in\mathbb{N} t ∈ N do
v t + 1 = T v t v^{t+1} = Tv^t v t + 1 = T v t
end for
Algorithm 6 Policy iteration for robust MDPs
Initialize π 0 ∈ Π S \pi^0 \in\Pi_{\mathcal{S}} π 0 ∈ Π S , v 0 = v π 0 v^0 = v^{\pi^0} v 0 = v π 0 .
for t ∈ N t \in\mathbb{N} t ∈ N do
Choose π t + 1 ∈ Π S \pi^{t+1} \in\Pi_{\mathcal{S}} π t + 1 ∈ Π S such that T v t ( s ) = T π t + 1 v t ( s ) Tv^t(s) = T^{\pi^{t+1}}v^t(s) T v t ( s ) = T π t + 1 v t ( s ) for all s ∈ S s \in\mathcal{S} s ∈ S (and equal to π t \pi^t π t if we can).
Choose v t + 1 = v π t + 1 v^{t+1} = v^{\pi^{t+1}} v t + 1 = v π t + 1 .
if π t + 1 = π t \pi^{t+1} = \pi^t π t + 1 = π t then return π t + 1 \pi^{t+1} π t + 1 .
end if
end for
Proposition 3.8. Let P \mathcal{P} P be a compact convex s s s -rectangular uncertainty set. Let ( v t ) t ∈ N (v^t)_{t \in\mathbb{N}} ( v t ) t ∈ N be the sequence of vectors generated by Algorithm 5.
The sequence ( v t ) t ∈ N (v^t)_{t \in\mathbb{N}} ( v t ) t ∈ N converges to v ⋆ v^\star v ⋆ at a geometric rate of γ \gamma γ :
∥ v ⋆ − v t ∥ ∞ ≤ γ t ∥ v ⋆ − v 0 ∥ ∞ \lVert v^\star- v^t \rVert_\infty\leq\gamma^t \lVert v^\star- v^0 \rVert_\infty ∥ v ⋆ − v t ∥ ∞ ≤ γ t ∥ v ⋆ − v 0 ∥ ∞ Let ϵ > 0 \epsilon> 0 ϵ > 0 . Assume that ∥ v ⋆ − v t ∥ ∞ ≤ ϵ \lVert v^\star- v^t \rVert_\infty\leq\epsilon ∥ v ⋆ − v t ∥ ∞ ≤ ϵ for some t ∈ N t \in\mathbb{N} t ∈ N and let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S such that T π v t = T v t T^\pi v^t = Tv^t T π v t = T v t . Then π \pi π is a uniform 2 ϵ γ ( 1 − γ ) − 1 2\epsilon\gamma(1-\gamma)^{-1} 2 ϵ γ ( 1 − γ ) − 1 -optimal policy: ∥ v ⋆ − v π ∥ ∞ ≤ 2 ϵ γ ( 1 − γ ) − 1 \lVert v^\star- v^\pi\rVert_\infty\leq2\epsilon\gamma(1-\gamma)^{-1} ∥ v ⋆ − v π ∥ ∞ ≤ 2 ϵ γ ( 1 − γ ) − 1 .
Additionally, we can check that ∥ v ⋆ − v t ∥ ∞ ≤ ϵ \lVert v^\star-v^t\rVert_\infty\le\epsilon ∥ v ⋆ − v t ∥ ∞ ≤ ϵ by checking ∥ T v t − v t ∥ ∞ ≤ ϵ ( 1 − γ ) \lVert Tv^t-v^t\rVert_\infty\le\epsilon(1-\gamma) ∥ T v t − v t ∥ ∞ ≤ ϵ ( 1 − γ ) , and that last inequality holds after at most ⌊ 1 1 − γ log ( ∥ v μ − v 0 ∥ ∞ ϵ ( 1 − γ ) ) ⌋ + 1 \left\lfloor\frac{1}{1-\gamma}\log\left(\frac{\lVert v^\mu-v^0\rVert_\infty}{\epsilon(1-\gamma)}\right)\right\rfloor+1 ⌊ 1 − γ 1 log ( ϵ ( 1 − γ ) ∥ v μ − v 0 ∥ ∞ ) ⌋ + 1 iterations of Algorithm 5.
Proposition 3.9. Let P \mathcal{P} P be a compact convex s s s -rectangular uncertainty set. Let ( π t ) t ∈ N (\pi^t)_{t\in\mathbb{N}} ( π t ) t ∈ N be the sequence of policies generated by Algorithm 6. Then ∥ v ⋆ − v π t ∥ ∞ ≤ γ t ∥ v ⋆ − v π 0 ∥ ∞ \lVert v^\star-v^{\pi^t}\rVert_\infty\le\gamma^t\lVert v^\star-v^{\pi^0}\rVert_\infty ∥ v ⋆ − v π t ∥ ∞ ≤ γ t ∥ v ⋆ − v π 0 ∥ ∞ .
A more refined convergence analysis can be given for policy iteration (Algorithm 6), see the references in § 3.10 . Note that Algorithm 5 and Algorithm 6 require estimating the robust Bellman operator T T T , which is a maxmin optimization problem with bilinear objective in ( π , P ) (\pi,P) ( π , P ) over the compact convex sets Π S \Pi_S Π S and P \mathcal{P} P , as can be seen from its definition {(xref}(3.4). This family of optimization problems can be solved efficiently via a variety of methods. Without exploiting any further the structure of the problem, saddle-point optimization problems can be solved via first-order methods (mirror descent, mirror prox, extragradient algorithm, etc.). Over the years there have been several papers providing more efficient algorithms in the case where P \mathcal{P} P has some additional structure, e.g., if P \mathcal{P} P is based on some ℓ p \ell_p ℓ p -distance from some nominal kernel, see § 3.10 for the references there. Note that Algorithm 6 also requires computing a worst-case value function of the current policy at every iteration, which is equivalent to solving an MDP (following Corollary 3.1); thus we note that Algorithm 6 is a priori much more computationally intensive than Algorithm 5.
Convex programming formulation. The linear programming formulation for MDPs obtained in Proposition 1.4 was obtained in two steps. In the first step we used the contraction lemma (Lemma 1.2) to obtain a mathematical program whose set of optimal solutions was exactly the optimal value function. In the second step we used the componentwise convexity of the nominal Bellman operator to conclude that this mathematical program was a convex (linear) program.
The first step still applies for robust MDPs: since T T T is a monotone contraction, we still have that
{ v ⋆ } = a r g m i n { ∑ s ∈ S v ( s ) ∣ v ≥ T v } = a r g m a x { ∑ s ∈ S v ( s ) ∣ v ≤ T v } \{v^\star\}=\mathop{\rm arg\,min}\left\{\sum_{s\in S}v(s)\mid v\ge Tv\right\}=\mathop{\rm arg\,max}\left\{\sum_{s\in S}v(s)\mid v\le Tv\right\} { v ⋆ } = arg min { s ∈ S ∑ v ( s ) ∣ v ≥ T v } = arg max { s ∈ S ∑ v ( s ) ∣ v ≤ T v }
where the first equality is the contraction lemma and the second optimization program is given in Section 1.5. However T T T need not be componentwise convex anymore, nor componentwise concave, because of the maxmin formulation in {(xref}(3.4), so we cannot conclude that the mathematical programs above are convex optimization problems. Some convex formulations can compute ϵ \epsilon ϵ -optimal policies but this requires more work, see § 3.10 .
A closer look at non-convex s-rectangular uncertainty sets*
We get back to non-convex s s s -rectangular sets. In this case we can still try to compute a best policy among stationary ones, i.e., to solve
max π ∈ Π S min P ∈ P R ( π , P ) . \max_{\pi\in\Pi_S}\min_{P\in\mathcal{P}}R(\pi,P). π ∈ Π S max P ∈ P min R ( π , P ) .
From Proposition 3.6 this can be done by (a) computing v ⋆ ∈ R S v^\star\in\mathbb{R}^{S} v ⋆ ∈ R S the fixed point of the robust Bellman operator T T T , and (b) computing π ⋆ \pi^\star π ⋆ such that T π ⋆ v ⋆ = T v ⋆ T^{\pi^\star}v^\star= Tv^\star T π ⋆ v ⋆ = T v ⋆ . As long as we can evaluate T v Tv T v for any v ∈ R S v \in\mathbb{R}^{S} v ∈ R S , we can run value iteration or policy iteration to compute v ⋆ v^\star v ⋆ , similarly as in § 3.3.3 . Whether or not we can evaluate T v Tv T v depends on the tractability properties of max π ∈ Π S min P ∈ P T π , P v ( s ) \max_{\pi\in\Pi_{S}}\min_{P \in\mathcal{P}} T^{\pi,P}v(s) max π ∈ Π S min P ∈ P T π , P v ( s ) . Even for P \mathcal{P} P non-convex we get that
max π ∈ Π S min P ∈ P T π , P v ( s ) = max π ∈ Π S min P ∈ conv ( P ) T π , P v ( s ) = min P ∈ conv ( P ) max π ∈ Π S T π , P v ( s ) \max_{\pi\in\Pi_{S}}\min_{P \in\mathcal{P}} T^{\pi,P}v(s) = \max_{\pi\in\Pi_{S}}\min_{P \in\operatorname{conv}(\mathcal{P})} T^{\pi,P}v(s) = \min_{P \in\operatorname{conv}(\mathcal{P})}\max_{\pi\in\Pi_{S}} T^{\pi,P}v(s) π ∈ Π S max P ∈ P min T π , P v ( s ) = π ∈ Π S max P ∈ conv ( P ) min T π , P v ( s ) = P ∈ conv ( P ) min π ∈ Π S max T π , P v ( s )
where the first equality is because the inner minimization has a linear objective over P \mathcal{P} P , while the second equality is from Sion’s minimax theorem. The first equality implies that the tractability properties of T v Tv T v are the same for P \mathcal{P} P , s-rectangular and potentially non-convex, and conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) , which is s-rectangular and convex!
We can now take a second look at strong duality for rectangular models. For compact s-rectangular uncertainty sets P \mathcal{P} P we have
max π ∈ Π S min P ∈ P R ( π , P ) = max π ∈ Π S min P ∈ conv ( P ) R ( π , P ) = min P ∈ conv ( P ) max π ∈ Π S R ( π , P ) ≤ min P ∈ P max π ∈ Π S R ( π , P ) . \begin{aligned}
\max_{\pi\in\Pi_{S}}\min_{P \in\mathcal{P}} R(\pi,P) &= \max_{\pi\in\Pi_{S}}\min_{P \in\operatorname{conv}(\mathcal{P})} R(\pi,P) \\
&= \min_{P \in\operatorname{conv}(\mathcal{P})}\max_{\pi\in\Pi_{S}} R(\pi,P) \\
&\le\min_{P \in\mathcal{P}}\max_{\pi\in\Pi_{S}} R(\pi,P).
\end{aligned} π ∈ Π S max P ∈ P min R ( π , P ) = π ∈ Π S max P ∈ conv ( P ) min R ( π , P ) = P ∈ conv ( P ) min π ∈ Π S max R ( π , P ) ≤ P ∈ P min π ∈ Π S max R ( π , P ) .
where the first equality is because the inner minimization is the adversarial MDP, so we can replace stationary deterministic policies of the adversary (the set P \mathcal{P} P ) by stationary randomized policies of the adversary (which yields conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) ), the second equality is from strong duality for s-rectangular convex compact sets, and the inequality is from P ⊆ conv ( P ) \mathcal{P} \subseteq\operatorname{conv}(\mathcal{P}) P ⊆ conv ( P ) . This shows that strong duality holds (for potentially non-convex s-rectangular compact sets) if and only if min P ∈ conv ( P ) max π ∈ Π S R ( π , P ) = min P ∈ P max π ∈ Π S R ( π , P ) \min_{P \in\operatorname{conv}(\mathcal{P})}\max_{\pi\in\Pi_{S}} R(\pi,P) = \min_{P \in\mathcal{P}}\max_{\pi\in\Pi_{S}} R(\pi,P) min P ∈ conv ( P ) max π ∈ Π S R ( π , P ) = min P ∈ P max π ∈ Π S R ( π , P ) . Of course, if P \mathcal{P} P is convex this last equality is true, but we’ll see in § 3.4 this last equality also holds for sa-rectangular compact uncertainty sets.
Additional properties of optimal solutions*
Worst-case transition probabilities. For s-rectangular compact convex uncertainty sets, we have seen that a uniform worst-case transition kernel P π P^\pi P π of a stationary policy π ∈ Π S \pi\in\Pi_{S} π ∈ Π S can be computed as the solution of an MDP with action set P s \mathcal{P}_{s} P s at each state s ∈ S s \in\mathcal{S} s ∈ S . In particular, this implies that we can always compute the worst-case probabilities among the set ext ( P ) \operatorname{ext}(\mathcal{P}) ext ( P ) of extreme points of P \mathcal{P} P . However, if we are interested in finding uniformly optimal worst-case transition probabilities, from Theorem 3.2 we want to find P ⋆ ∈ P P^\star\in\mathcal{P} P ⋆ ∈ P such that T P ⋆ v ⋆ = T v ⋆ T^{P^\star}v^\star= Tv^\star T P ⋆ v ⋆ = T v ⋆ , i.e., we want P ⋆ P^\star P ⋆ to be a solution of the following optimization problem:
min P ∈ P max π ∈ Π S ∑ a ∈ A π ( s , a ) P ( s , a ) T ( r ( s , a ) + γ v ⋆ ) . \min_{P \in\mathcal{P}}\max_{\pi\in\Pi_{S}}\sum_{a \in\mathcal{A}}\pi(s,a)P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v^\star). P ∈ P min π ∈ Π S max a ∈ A ∑ π ( s , a ) P ( s , a ) T ( r ( s , a ) + γ v ⋆ ) .
In general the minimum of the above optimization problem need not be attained at an extreme point of P \mathcal{P} P (because we’ll dualize the inner maximization program), so if we’re interested in finding uniformly optimal worst-case transition probabilities we may have to consider non-extreme points of P \mathcal{P} P .
Adjacent improving policies. Recall two policies are adjacent if they differ at exactly one state (we defined this notion in Chapter 1). The next lemma generalizes the statement in Section 1.3.
Lemma 3.3. Let P \mathcal{P} P be compact convex s s s -rectangular. Let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S be non-uniformly optimal. Then there exists π ′ ∈ Π S \pi' \in\Pi_{\mathcal{S}} π ′ ∈ Π S , that differs from π \pi π in only one state, such that v π ′ ≥ v π v^{\pi'} \geq v^\pi v π ′ ≥ v π , with strict inequality in at least one component.
Proof. Note that we always have v π = T π v π ≤ T v π v^\pi= T^\pi v^\pi\leq T v^\pi v π = T π v π ≤ T v π , where the equality follows from the definition of v π ∈ R S v^\pi\in\mathbb{R}^{\mathcal{S}} v π ∈ R S as the fixed point of T π T^\pi T π and the componentwise inequality follows from the definition of T T T . Since π \pi π is not uniformly optimal, we must have that there exists a state s ˉ ∈ S \bar{s} \in\mathcal{S} s ˉ ∈ S such that v π ( s ˉ ) < T v π ( s ˉ ) v^\pi(\bar{s}) < T v^\pi(\bar{s}) v π ( s ˉ ) < T v π ( s ˉ ) . Choose π ′ \pi' π ′ the policy that coincides with π \pi π everywhere except at s ˉ \bar{s} s ˉ , where π ′ \pi' π ′ chooses a randomized action attaining the maximum in T v π ( s ˉ ) T v^\pi(\bar{s}) T v π ( s ˉ ) . Then we must have T π ′ v π ′ ≥ v π ′ T^{\pi'}v^{\pi'} \geq v^{\pi'} T π ′ v π ′ ≥ v π ′ , with strict inequality at s ˉ \bar{s} s ˉ . Monotonicity and contraction of T π ′ T^{\pi'} T π ′ imply that v π ′ ≥ v π v^{\pi'} \geq v^\pi v π ′ ≥ v π . Additionally v π ′ ( s ˉ ) = T π ′ v π ′ ( s ˉ ) ≥ T π ′ v π ( s ˉ ) > v π ( s ˉ ) v^{\pi'}(\bar{s}) = T^{\pi'}v^{\pi'}(\bar{s}) \geq T^{\pi'}v^\pi(\bar{s}) > v^\pi(\bar{s}) v π ′ ( s ˉ ) = T π ′ v π ′ ( s ˉ ) ≥ T π ′ v π ( s ˉ ) > v π ( s ˉ ) , where the equality is from the fixed point equation for v π ′ v^{\pi'} v π ′ , the inequality is from v π ′ ≥ v π v^{\pi'} \geq v^\pi v π ′ ≥ v π , and the strict inequality is from the choice of s ˉ \bar{s} s ˉ and π ′ \pi' π ′ .
Example 1.3 shows that the same statement is false if we replace worst-case value functions by worst-case returns.
Support of a randomized policy. We have seen in Theorem 3.2 that optimal policies can be chosen in Π S \Pi_{\mathcal{S}} Π S , so that we may need randomization between several actions to achieve optimality in s s s -rectangular robust MDPs. Recall that from a mathematical programming standpoint, randomization can be traced to the dualization of the inner minimum over P \mathcal{P} P in the maxmin expression of the robust Bellman operator T T T . One may wonder between how many actions must we randomize; we provide a partial answer in the next proposition and relate the size of the randomization support with the affine dimension of each P s \mathcal{P}_s P s .
Proposition 3.10. Let P \mathcal{P} P be compact convex s s s -rectangular. Let s ∈ S s \in\mathcal{S} s ∈ S and v ∈ R S v \in\mathbb{R}^{\mathcal{S}} v ∈ R S . Let dim a f f ( P s ) ∈ N \operatorname{dim}_{\mathrm{aff}}(\mathcal{P}_s) \in\mathbb{N} dim aff ( P s ) ∈ N be the affine dimension of P s \mathcal{P}_s P s , i.e., the dimension of the affine hull of P s ⊂ Δ ( S ) A \mathcal{P}_s \subset\Delta(\mathcal{S})^{\mathcal{A}} P s ⊂ Δ ( S ) A as an affine subspace of R ∣ S ∣ × ∣ A ∣ \mathbb{R}^{|\mathcal{S}|\times|\mathcal{A}|} R ∣ S ∣ × ∣ A ∣ . For any v ∈ R S v \in\mathbb{R}^{\mathcal{S}} v ∈ R S and s ∈ S s \in\mathcal{S} s ∈ S there exists a maximizer of
max π ( s ) ∈ Δ ( A ) min P ( s ) ∈ P s ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) \max_{\pi(s) \in\Delta(\mathcal{A})}\ \min_{P(s) \in\mathcal{P}_s}\ \sum_{a \in\mathcal{A}} \pi(s,a)P(s,a)^\top(r(s,a)+\gamma v) π ( s ) ∈ Δ ( A ) max P ( s ) ∈ P s min a ∈ A ∑ π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ) with support of cardinality at most min { ∣ A ∣ , dim a f f ( P s ) + 1 } \min\{|\mathcal{A}|,\operatorname{dim}_{\mathrm{aff}}(\mathcal{P}_s)+1\} min { ∣ A ∣ , dim aff ( P s ) + 1 } .
Proof. Here we overload notation to write the maxmin problem from above as
max π ( s ) ∈ Δ ( A ) min P ( s ) ∈ P s T π ( s ) , P ( s ) v ( s ) \max_{\pi(s) \in\Delta(\mathcal{A})}\ \min_{P(s) \in\mathcal{P}_s} T^{\pi(s),P(s)}v(s) π ( s ) ∈ Δ ( A ) max P ( s ) ∈ P s min T π ( s ) , P ( s ) v ( s )
(in principle writing “T π , P T^{\pi,P} T π , P ” requires defining a full policy π \pi π and kernel P P P , but the s s s -th component of T π , P T^{\pi,P} T π , P only depends on the s s s -th component of π \pi π and P P P ). By strong duality of minmax problems with convex compact domains and bilinear objectives we have max π ( s ) ∈ Δ ( A ) min P ( s ) ∈ P s T π ( s ) , P ( s ) v ( s ) = min P ( s ) ∈ P s max π ( s ) ∈ Δ ( A ) T π ( s ) , P ( s ) v ( s ) \max_{\pi(s)\in\Delta(\mathcal{A})}\min_{P(s)\in\mathcal{P}_s} T^{\pi(s),P(s)}v(s)=\min_{P(s)\in\mathcal{P}_s}\max_{\pi(s)\in\Delta(\mathcal{A})} T^{\pi(s),P(s)}v(s) max π ( s ) ∈ Δ ( A ) min P ( s ) ∈ P s T π ( s ) , P ( s ) v ( s ) = min P ( s ) ∈ P s max π ( s ) ∈ Δ ( A ) T π ( s ) , P ( s ) v ( s ) . Let P ⋆ ( s ) ∈ P s P^{\star}(s)\in\mathcal{P}_s P ⋆ ( s ) ∈ P s in arg min P ( s ) ∈ P s max π ( s ) ∈ Δ ( A ) T π ( s ) , P ( s ) v ( s ) \operatorname*{arg\,min}_{P(s)\in\mathcal{P}_s}\max_{\pi(s)\in\Delta(\mathcal{A})}T^{\pi(s),P(s)}v(s) arg min P ( s ) ∈ P s max π ( s ) ∈ Δ ( A ) T π ( s ) , P ( s ) v ( s ) . The minimax theorem for
min P ( s ) ∈ P s max π ( s ) ∈ Δ ( A ) T π ( s ) , P ( s ) v ( s ) \min_{P(s)\in\mathcal{P}_s}\max_{\pi(s)\in\Delta(\mathcal{A})}T^{\pi(s),P(s)}v(s) P ( s ) ∈ P s min π ( s ) ∈ Δ ( A ) max T π ( s ) , P ( s ) v ( s )
gives the existence of some λ ∈ Δ ( A ) \lambda\in\Delta(\mathcal{A}) λ ∈ Δ ( A ) such that λ ( a ) = 0 \lambda(a)=0 λ ( a ) = 0 if a ∉ arg max a ′ ∈ A T a ′ , P ⋆ ( s ) v ( s ) a\notin\operatorname*{arg\,max}_{a'\in\mathcal{A}}T^{a',P^{\star}(s)}v(s) a ∈ / arg max a ′ ∈ A T a ′ , P ⋆ ( s ) v ( s ) (noting T a ′ , P ⋆ ( s ) v ( s ) = P ⋆ ( s , a ′ ) ⊤ ( r ( s , a ′ ) + γ v ) ) T^{a',P^{\star}(s)}v(s)=P^{\star}(s,a')^{\top}(r(s,a')+\gamma v)) T a ′ , P ⋆ ( s ) v ( s ) = P ⋆ ( s , a ′ ) ⊤ ( r ( s , a ′ ) + γ v )) and
∑ a ∈ A λ ( a ) ( T a , P ( s ) v ( s ) − T a , P ⋆ ( s ) v ( s ) ) ≥ 0 , ∀ P ( s ) ∈ P s . \sum_{a\in\mathcal{A}}\lambda(a)\left(T^{a,P(s)}v(s)-T^{a,P^{\star}(s)}v(s)\right)\ge0,\ \forall P(s)\in\mathcal{P}_s. a ∈ A ∑ λ ( a ) ( T a , P ( s ) v ( s ) − T a , P ⋆ ( s ) v ( s ) ) ≥ 0 , ∀ P ( s ) ∈ P s .
Now note that T a , P ( s ) v ( s ) − T a , P ⋆ ( s ) v ( s ) = T a , P ( s ) − P ⋆ ( s ) v ( s ) T^{a,P(s)}v(s)-T^{a,P^{\star}(s)}v(s)=T^{a,P(s)-P^{\star}(s)}v(s) T a , P ( s ) v ( s ) − T a , P ⋆ ( s ) v ( s ) = T a , P ( s ) − P ⋆ ( s ) v ( s ) because the same action a a a is chosen at state s s s in both operators. Hence we can interpret the left-hand side of the above inequality as a convex combination of elements of dimensions at most dimaff ( P s ) \operatorname{dimaff}(\mathcal{P}_s) dimaff ( P s ) . From Carathéodory’s theorem we can choose such λ ∈ Δ ( A ) \lambda\in\Delta(\mathcal{A}) λ ∈ Δ ( A ) with at most dimaff ( P s ) + 1 \operatorname{dimaff}(\mathcal{P}_s)+1 dimaff ( P s ) + 1 nonzero entries.
Now we choose π ⋆ ( s ) = λ \pi^{\star}(s)=\lambda π ⋆ ( s ) = λ . The first-order condition gives that for any P ( s ) ∈ P s P(s)\in\mathcal{P}_s P ( s ) ∈ P s we have
∑ a ∈ A π ⋆ ( s , a ) T a , P ( s ) v ( s ) ≥ ∑ a ∈ A π ⋆ ( s , a ) T a , P ⋆ ( s ) v ( s ) = max a ∈ A T a , P ⋆ ( s ) v ( s ) \sum_{a\in\mathcal{A}}\pi^{\star}(s,a)T^{a,P(s)}v(s)\ge\sum_{a\in\mathcal{A}}\pi^{\star}(s,a)T^{a,P^{\star}(s)}v(s)=\max_{a\in\mathcal{A}}T^{a,P^{\star}(s)}v(s) a ∈ A ∑ π ⋆ ( s , a ) T a , P ( s ) v ( s ) ≥ a ∈ A ∑ π ⋆ ( s , a ) T a , P ⋆ ( s ) v ( s ) = a ∈ A max T a , P ⋆ ( s ) v ( s )
where the equality is because π ⋆ ( s ) \pi^{\star}(s) π ⋆ ( s ) is supported on the active actions at P ⋆ ( s ) P^{\star}(s) P ⋆ ( s ) . Conversely for any π ( s ) ∈ Δ ( A ) \pi(s)\in\Delta(\mathcal{A}) π ( s ) ∈ Δ ( A ) we have
min P ( s ) ∈ P s ∑ a ∈ A π ( s , a ) T a , P ( s ) v ( s ) ≤ ∑ a ∈ A π ( s , a ) T a , P ⋆ ( s ) v ( s ) ≤ max a ∈ A T a , P ⋆ ( s ) v ( s ) \min_{P(s)\in\mathcal{P}_s}\sum_{a\in\mathcal{A}}\pi(s,a)T^{a,P(s)}v(s)\le\sum_{a\in\mathcal{A}}\pi(s,a)T^{a,P^{\star}(s)}v(s)\le\max_{a\in\mathcal{A}}T^{a,P^{\star}(s)}v(s) P ( s ) ∈ P s min a ∈ A ∑ π ( s , a ) T a , P ( s ) v ( s ) ≤ a ∈ A ∑ π ( s , a ) T a , P ⋆ ( s ) v ( s ) ≤ a ∈ A max T a , P ⋆ ( s ) v ( s )
hence we conclude that if we construct π ⋆ \pi^{\star} π ⋆ this way for v ⋆ v^{\star} v ⋆ the fixed point of T T T and for each state s ∈ S s\in\mathcal{S} s ∈ S we get an optimal policy, with support at most min { ∣ A ∣ , dimaff ( P s ) + 1 } \min\{|\mathcal{A}|,\operatorname{dimaff}(\mathcal{P}_s)+1\} min { ∣ A ∣ , dimaff ( P s ) + 1 } .
The case of sa-rectangular models
The previous sections have highlighted the good tractability and game-theoretic properties of s-rectangular models. The following sa-rectangular model imposes some additional structural assumption that the adversary can choose their decisions (worst-case transition probabilities) independently across state-action pairs.
Definition 3.4. An uncertainty set P ⊂ Δ ( S ) S × A \mathcal{P}\subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A is sa-rectangular if we can write P = × ( s , a ) ∈ S × A P s a \mathcal{P}=\mathop{\times}_{(s,a)\in\mathcal{S}\times\mathcal{A}}\mathcal{P}_{sa} P = × ( s , a ) ∈ S × A P s a for some sets P s a ⊆ Δ ( S ) \mathcal{P}_{sa}\subseteq\Delta(\mathcal{S}) P s a ⊆ Δ ( S ) for each ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A .
A typical example of sa-rectangular uncertainty set is based on P s a ⊆ Δ ( S ) \mathcal{P}_{sa}\subseteq\Delta(\mathcal{S}) P s a ⊆ Δ ( S ) as
P s a = { p ∈ Δ ( S ) ∣ d ( p , P ^ ( s , a ) ) ≤ ϵ s a } \mathcal{P}_{sa}=\{p\in\Delta(\mathcal{S})\mid d(p,\widehat{P}(s,a))\le\epsilon_{sa}\} P s a = { p ∈ Δ ( S ) ∣ d ( p , P ( s , a )) ≤ ϵ s a }
for some reference transitions P ^ ∈ Δ ( S ) S × A \widehat{P}\in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ∈ Δ ( S ) S × A . We can check “visually” the sa-rectangularity assumption (for robust MDP instances represented as in the examples from Chapter 1 and Chapter 2) if the parameters present in each figure representing the transition for each state-action pair are independent. Note that sa-rectangular sets are s-rectangular, hence all the favorable properties (strong duality, existence of stationary optimal policies) outlined in the previous section apply (under the compactness and convexity assumptions). However, we will show below that for sa-rectangular uncertainty sets P \mathcal{P} P we do not need P \mathcal{P} P to be convex to conclude as in Theorem 3.2.
Additional properties. The sa-rectangularity assumption provides some structural results on the Bellman operators. Let π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S , then
T π v ( s ) = min P ∈ P T π , P v ( s ) = min ( P ( s , a ) ) s , a ∈ × ( s , a ) P s a ∑ a ∈ A π ( s , a ) P ( s , a ) T ( r ( s , a ) + γ v ) = ∑ a ∈ A π ( s , a ) min P ( s , a ) ∈ P s a P ( s , a ) T ( r ( s , a ) + γ v ) . \begin{aligned}
T^{\pi}v(s)=\min_{P\in\mathcal{P}}T^{\pi,P}v(s)=\min_{(P(s,a))_{s,a}\in\mathop{\times}_{(s,a)}\mathcal{P}_{sa}}\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v) \\
&=\sum_{a\in\mathcal{A}}\pi(s,a)\min_{P(s,a)\in\mathcal{P}_{sa}}P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v).
\end{aligned} T π v ( s ) = P ∈ P min T π , P v ( s ) = ( P ( s , a ) ) s , a ∈ × ( s , a ) P s a min a ∈ A ∑ π ( s , a ) P ( s , a ) T ( r ( s , a ) + γ v ) = a ∈ A ∑ π ( s , a ) P ( s , a ) ∈ P s a min P ( s , a ) T ( r ( s , a ) + γ v ) .
that is, we can interchange the summation over a ∈ A a \in\mathcal{A} a ∈ A and the minimization program over P ∈ P P \in\mathcal{P} P ∈ P . The first equality above holds by definition of T π T^{\pi} T π , the second equality just uses the definition of T π , P T^{\pi,P} T π , P , and the third equality is because each term in the summation over a ∈ A a \in\mathcal{A} a ∈ A only depends on P ( s , a ) P(s,a) P ( s , a ) and P \mathcal{P} P is a Cartesian product over ( s , a ) (s,a) ( s , a ) (by the sa-rectangularity assumption). In particular, this implies that
T v ( s ) = max π ∈ Π S min P ∈ P T π , P v ( s ) = max π ∈ Π S ∑ a ∈ A π ( s , a ) min P ( s , a ) ∈ P s a P ( s , a ) T ( r ( s , a ) + γ v ) = max a ∈ A min P ( s , a ) ∈ P s a P ( s , a ) T ( r ( s , a ) + γ v ) . (32) \begin{aligned}
Tv(s)=\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}T^{\pi,P}v(s)=\max_{\pi\in\Pi_{\mathrm{S}}}\sum_{a\in\mathcal{A}}\pi(s,a)\min_{P(s,a)\in\mathcal{P}_{sa}}P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v) \\
&=\max_{a\in\mathcal{A}}\min_{P(s,a)\in\mathcal{P}_{sa}}P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v). \tag*{(32)}
\end{aligned} T v ( s ) = π ∈ Π S max P ∈ P min T π , P v ( s ) = π ∈ Π S max a ∈ A ∑ π ( s , a ) P ( s , a ) ∈ P s a min P ( s , a ) T ( r ( s , a ) + γ v ) = a ∈ A max P ( s , a ) ∈ P s a min P ( s , a ) T ( r ( s , a ) + γ v ) . (32)
i.e., to evaluate T v ( s ) Tv(s) T v ( s ) we do not need to solve a “true” saddle-point problem but we only need to compute the maximum of ∣ A ∣ |\mathcal{A}| ∣ A ∣ real numbers given by min P ( s , a ) ∈ P s a P ( s , a ) T ( r ( s , a ) + γ v ) \min_{P(s,a)\in\mathcal{P}_{sa}}P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v) min P ( s , a ) ∈ P s a P ( s , a ) T ( r ( s , a ) + γ v ) for each a ∈ A a \in\mathcal{A} a ∈ A . In passing, we’ve obtained that
T v ( s ) = max π ∈ Π S D min P ∈ P T π , P v ( s ) (33) Tv(s)=\max_{\pi\in\Pi_{\mathrm{SD}}}\min_{P\in\mathcal{P}}T^{\pi,P}v(s) \tag*{(33)} T v ( s ) = π ∈ Π SD max P ∈ P min T π , P v ( s ) (33)
for sa-rectangular compact uncertainty sets (this will be useful in the next section). Since Theorem 3.2 gives that the set of stationary uniformly optimal policies is the set of policies π ˉ \bar{\pi} π ˉ such that T π ˉ v ⋆ = T v ⋆ T^{\bar{\pi}}v^{\star}=Tv^{\star} T π ˉ v ⋆ = T v ⋆ , the next lemma follows directly from (3.11).
Lemma 3.4. Let P ⊂ Δ ( S ) S × A \mathcal{P}\subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be a convex compact sa-rectangular uncertainty set. Then there exists a stationary deterministic uniformly optimal policy , which can be found as any π ∈ Π S D \pi\in\Pi_{\mathrm{SD}} π ∈ Π SD such that T π v ⋆ = T v ⋆ T^{\pi}v^{\star}=Tv^{\star} T π v ⋆ = T v ⋆ .
We give above the results for sa-rectangular compact convex uncertainty sets, because we required convexity for s-rectangular sets. However the convexity assumption is unnecessary for sa-rectangular robust MDPs, as we show next.
Lemma 3.5. Let P \mathcal{P} P be sa-rectangular compact. Let v ∈ R S v\in\mathbb{R}^{\mathcal{S}} v ∈ R S and s ∈ S s\in\mathcal{S} s ∈ S . Then
T v ( s ) = max π ∈ Π S D min P ∈ P T π , P v ( s ) = min P ∈ P max π ∈ Π S D T π , P v ( s ) . (34) Tv(s)=\max_{\pi\in\Pi_{\mathrm{SD}}}\min_{P\in\mathcal{P}}T^{\pi,P}v(s)=\min_{P\in\mathcal{P}}\max_{\pi\in\Pi_{\mathrm{SD}}}T^{\pi,P}v(s). \tag*{(34)} T v ( s ) = π ∈ Π SD max P ∈ P min T π , P v ( s ) = P ∈ P min π ∈ Π SD max T π , P v ( s ) . (34) Proof. From (3.10) we know that
max π ∈ Π S D min P ∈ P T π , P v ( s ) = max a ∈ A min P ( s , a ) ∈ P s a P ( s , a ) T ( r ( s , a ) + γ v ) . \max_{\pi\in\Pi_{\mathrm{SD}}}\min_{P\in\mathcal{P}}T^{\pi,P}v(s)=\max_{a\in\mathcal{A}}\min_{P(s,a)\in\mathcal{P}_{sa}}P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v). π ∈ Π SD max P ∈ P min T π , P v ( s ) = a ∈ A max P ( s , a ) ∈ P s a min P ( s , a ) T ( r ( s , a ) + γ v ) . From weak duality we always have
max a ∈ A min P ( s , a ) ∈ P s a P ( s , a ) T ( r ( s , a ) + γ v ) ≤ min P ∈ P max a ∈ A P ( s , a ) T ( r ( s , a ) + γ v ) . \max_{a\in\mathcal{A}}\min_{P(s,a)\in\mathcal{P}_{sa}}P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v)\leq\min_{P\in\mathcal{P}}\max_{a\in\mathcal{A}}P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v). a ∈ A max P ( s , a ) ∈ P s a min P ( s , a ) T ( r ( s , a ) + γ v ) ≤ P ∈ P min a ∈ A max P ( s , a ) T ( r ( s , a ) + γ v ) .
To show the equality we just need to find P ~ ∈ P \widetilde{P} \in\mathcal{P} P ∈ P such that, for all s ∈ S s \in\mathcal{S} s ∈ S , we have
max a ∈ A P ~ ( s , a ) ⊤ ( r ( s , a ) + γ v ) = max a ∈ A min P ( s , a ) ∈ P s a P ( s , a ) ⊤ ( r ( s , a ) + γ v ) . \max_{a \in\mathcal{A}} \widetilde{P}(s,a)^{\top}(r(s,a)+\gamma v)=\max_{a \in\mathcal{A}}\min_{P(s,a)\in\mathcal{P}_{sa}}P(s,a)^{\top}(r(s,a)+\gamma v). a ∈ A max P ( s , a ) ⊤ ( r ( s , a ) + γ v ) = a ∈ A max P ( s , a ) ∈ P s a min P ( s , a ) ⊤ ( r ( s , a ) + γ v ) .
Select P ~ ∈ P \widetilde{P} \in\mathcal{P} P ∈ P such that, for each ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A we have
P ~ ( s , a ) ∈ arg min P ( s , a ) ∈ P s a P ( s , a ) ⊤ ( r ( s , a ) + γ v ) . (35) \widetilde{P}(s,a) \in\arg\min_{P(s,a)\in\mathcal{P}_{sa}}P(s,a)^{\top}(r(s,a)+\gamma v). \tag*{(35)} P ( s , a ) ∈ arg P ( s , a ) ∈ P s a min P ( s , a ) ⊤ ( r ( s , a ) + γ v ) . (35)
Since P \mathcal{P} P is sa-rectangular we get that P ~ ∈ P \widetilde{P} \in\mathcal{P} P ∈ P and then by construction we have
max a ∈ A P ~ ( s , a ) ⊤ ( r ( s , a ) + γ v ) = max a ∈ A min P ( s , a ) ∈ P s a P ( s , a ) ⊤ ( r ( s , a ) + γ v ) \max_{a \in\mathcal{A}}\widetilde{P}(s,a)^{\top}(r(s,a)+\gamma v)=\max_{a \in\mathcal{A}}\min_{P(s,a)\in\mathcal{P}_{sa}}P(s,a)^{\top}(r(s,a)+\gamma v) a ∈ A max P ( s , a ) ⊤ ( r ( s , a ) + γ v ) = a ∈ A max P ( s , a ) ∈ P s a min P ( s , a ) ⊤ ( r ( s , a ) + γ v )
which shows that (34) holds. □
The proof of Lemma 3.5 may seem “magic” at first glance, and it may seem surprising that we can get a strong duality result without the convexity assumption. Here is a game-theoretic interpretation of the proof: since the adversary can independently minimize each optimization problem over all pairs ( s , a ) (s,a) ( s , a ) , the adversary does not need to “adapt” to the policy chosen by the decision maker. The fact that this “worst-case transition kernel” (as defined in (35) ) can be chosen independently of the chosen policy ensures that the value of the game is the same when the adversary plays first (i.e., when solving the minmax) and when the adversary plays second (i.e., when solving the maxmin). Still, note that the solution P ~ \widetilde{P} P defined in (35) depends on the vector v ∈ R S v \in\mathbb{R}^{S} v ∈ R S , so we are not claiming that there exists some transition kernel that attains the worst-case return simultaneously across all policies. Lemma 3.5 easily implies that strong duality holds for P \mathcal{P} P a compact sa-rectangular uncertainty set.
Proposition 3.11. Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be compact sa-rectangular. We have
max π ∈ Π S D min P ∈ P R ( π , P ) = min P ∈ P max π ∈ Π S D R ( π , P ) . (36) \max_{\pi\in\Pi_{\mathrm{SD}}}\min_{P\in\mathcal{P}}R(\pi,P)=\min_{P\in\mathcal{P}}\max_{\pi\in\Pi_{\mathrm{SD}}}R(\pi,P). \tag*{(36)} π ∈ Π SD max P ∈ P min R ( π , P ) = P ∈ P min π ∈ Π SD max R ( π , P ) . (36) Proof. Note that min P ∈ conv ( P ) max π ∈ Π S R ( π , P ) \min_{P\in\operatorname{conv}(\mathcal{P})}\max_{\pi\in\Pi_{\mathrm{S}}}R(\pi,P) min P ∈ conv ( P ) max π ∈ Π S R ( π , P ) admits an optimal solution at some P ∈ conv ( P ) P \in\operatorname{conv}(\mathcal{P}) P ∈ conv ( P ) that satisfies, for all s ∈ S s \in\mathcal{S} s ∈ S , P ∈ arg min P ∈ conv ( P ) max π ∈ Π S T π , P v ⋆ ( s ) P \in\arg\min_{P\in\operatorname{conv}(\mathcal{P})}\max_{\pi\in\Pi_{\mathrm{S}}}T^{\pi,P}v^{\star}(s) P ∈ arg min P ∈ conv ( P ) max π ∈ Π S T π , P v ⋆ ( s ) . But for P \mathcal{P} P compact sa-rectangular we have
min P ∈ conv ( P ) max π ∈ Π S T π , P v ⋆ ( s ) ≤ min P ∈ P max π ∈ Π S T π , P v ⋆ ( s ) = min P ∈ P max π ∈ Π S D T π , P v ⋆ ( s ) = max π ∈ Π S D min P ∈ P T π , P v ⋆ ( s ) = max π ∈ Π S D min P ∈ conv ( P ) T π , P v ⋆ ( s ) ≤ max π ∈ Π S min P ∈ conv ( P ) T π , P v ⋆ ( s ) \begin{aligned}
\min_{P\in\operatorname{conv}(\mathcal{P})}\max_{\pi\in\Pi_{\mathrm{S}}}T^{\pi,P}v^{\star}(s)
&\le\min_{P\in\mathcal{P}}\max_{\pi\in\Pi_{\mathrm{S}}}T^{\pi,P}v^{\star}(s)\\
&= \min_{P\in\mathcal{P}}\max_{\pi\in\Pi_{\mathrm{SD}}}T^{\pi,P}v^{\star}(s)\\
&= \max_{\pi\in\Pi_{\mathrm{SD}}}\min_{P\in\mathcal{P}}T^{\pi,P}v^{\star}(s)\\
&= \max_{\pi\in\Pi_{\mathrm{SD}}}\min_{P\in\operatorname{conv}(\mathcal{P})}T^{\pi,P}v^{\star}(s)\\
&\le\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\operatorname{conv}(\mathcal{P})}T^{\pi,P}v^{\star}(s)
\end{aligned} P ∈ conv ( P ) min π ∈ Π S max T π , P v ⋆ ( s ) ≤ P ∈ P min π ∈ Π S max T π , P v ⋆ ( s ) = P ∈ P min π ∈ Π SD max T π , P v ⋆ ( s ) = π ∈ Π SD max P ∈ P min T π , P v ⋆ ( s ) = π ∈ Π SD max P ∈ conv ( P ) min T π , P v ⋆ ( s ) ≤ π ∈ Π S max P ∈ conv ( P ) min T π , P v ⋆ ( s ) where the inequality is from P ⊂ conv ( P ) \mathcal{P} \subset\operatorname{conv}(\mathcal{P}) P ⊂ conv ( P ) , the first equality is from the inner maximization problem having a linear objective for each fixed P ∈ P P \in\mathcal{P} P ∈ P , the second equality is from (34) , the
third equality is from the inner minimization being linear in P P P , and the last inequality is from Π S D ⊂ Π S \Pi_{\mathrm{SD}} \subset\Pi_{\mathrm{S}} Π SD ⊂ Π S . So all terms above are equal and min P ∈ conv ( P ) max π ∈ Π S R ( π , P ) = min P ∈ P max π ∈ Π S R ( π , P ) \min_{P \in\operatorname{conv}(\mathcal{P})}\max_{\pi\in\Pi_{\mathrm{S}}} R(\pi,P) = \min_{P \in\mathcal{P}}\max_{\pi\in\Pi_{\mathrm{S}}} R(\pi,P) min P ∈ conv ( P ) max π ∈ Π S R ( π , P ) = min P ∈ P max π ∈ Π S R ( π , P ) holds for sa-rectangular compact uncertainty sets, without requiring the convexity assumption.
Therefore all the results from Theorem 3.2 and Lemma 3.4 are still true for sa-rectangular compact (not necessarily convex) uncertainty sets.
Algorithms for sa-rectangular robust MDPs
Iterative algorithms. The value iteration and policy iteration for s-rectangular robust MDPs (Algorithm 5 and Algorithm 6) still apply for sa-rectangular models, the main difference being that it is usually more efficient to evaluate the operator T T T in the case of sa-rectangular models, as explained after (32) . There have been many papers working on giving fast algorithms for evaluating T v Tv T v and T π v T^{\pi}v T π v for sa-rectangular models with specific structural assumptions, and we refer to § 3.10 for more details.
Convex formulation. In the case of sa-rectangular models, under the assumption that each P s a \mathcal{P}_{sa} P s a is a polytope, we can obtain a convex formulation for computing an optimal policy relatively easily. The contraction lemma (Lemma 1.2, as extended in § 1.6 ) states that
{ v ⋆ } = arg max { ∑ s ∈ S v ( s ) ∣ v ≤ T v , v ∈ R S } . \{v^\star\} = \operatorname*{arg\,max}\left\{\sum_{s \in\mathcal{S}} v(s) \mid v \leq Tv,\ v \in\mathbb{R}^{S}\right\}. { v ⋆ } = arg max { s ∈ S ∑ v ( s ) ∣ v ≤ T v , v ∈ R S } .
Let V : = { v ∈ R S ∣ v ≤ T v } \mathcal{V} := \{v \in\mathbb{R}^{S} \mid v \leq Tv\} V := { v ∈ R S ∣ v ≤ T v } . Since T v ( s ) = max π ∈ Π S D min P ∈ P T π , P v ( s ) Tv(s) = \max_{\pi\in\Pi_{\mathrm{SD}}}\min_{P \in\mathcal{P}} T^{\pi,P}v(s) T v ( s ) = max π ∈ Π SD min P ∈ P T π , P v ( s ) we get that V = ⋃ π ∈ Π S D V π \mathcal{V} = \bigcup_{\pi\in\Pi_{\mathrm{SD}}}\mathcal{V}^{\pi} V = ⋃ π ∈ Π SD V π , with V π : = { v ∈ R S ∣ v ≤ T π v } \mathcal{V}^{\pi} := \{v \in\mathbb{R}^{S} \mid v \leq T^{\pi}v\} V π := { v ∈ R S ∣ v ≤ T π v } . For V ‾ π = V π ∩ [ − M , M ] S \overline{\mathcal{V}}^{\pi} = \mathcal{V}^{\pi} \cap[-M,M]^{S} V π = V π ∩ [ − M , M ] S with M = ( 1 − γ ) − 1 max s , a , s ′ ∣ r ( s , a , s ′ ) ∣ M = (1-\gamma)^{-1}\max_{s,a,s'}|r(s,a,s')| M = ( 1 − γ ) − 1 max s , a , s ′ ∣ r ( s , a , s ′ ) ∣ we still get v π ∈ V ‾ π v^{\pi} \in\overline{\mathcal{V}}^{\pi} v π ∈ V π for each π ∈ Π S D \pi\in\Pi_{\mathrm{SD}} π ∈ Π SD so that
{ v ⋆ } = arg max { ∑ s ∈ S v ( s ) ∣ v ∈ conv ( ⋃ π ∈ Π S D V ‾ π ) } . \{v^\star\} = \operatorname*{arg\,max}\left\{\sum_{s \in\mathcal{S}}v(s) \mid v \in\operatorname{conv}\left(\bigcup_{\pi\in\Pi_{\mathrm{SD}}}\overline{\mathcal{V}}^{\pi}\right)\right\}. { v ⋆ } = arg max { s ∈ S ∑ v ( s ) ∣ v ∈ conv ( π ∈ Π SD ⋃ V π ) } .
since optimizing a linear function over a set or over its convex hull yields the same value. Now a theorem from Balas (see § 3.10 ) provides the following polytope reformulation for the convex hull of a union of polytopes: for Q i = { x ∣ A i x ≤ b i } Q_i = \{x \mid A_i x \leq b_i\} Q i = { x ∣ A i x ≤ b i } with i = 1 , … , N i = 1,\ldots,N i = 1 , … , N we have that
conv ( ⋃ i = 1 N Q i ) = { z ∣ z = ∑ i = 1 N z i , A i z i ≤ λ i b i , ∀ i , λ ∈ Δ ( { 1 , … , N } ) } . \operatorname{conv}\left(\bigcup_{i=1}^{N}Q_i\right) = \left\{z \mid z = \sum_{i=1}^{N}z^i,\ A_i z^i \leq\lambda_i b_i,\ \forall i,\ \lambda\in\Delta(\{1,\ldots,N\})\right\}. conv ( i = 1 ⋃ N Q i ) = { z ∣ z = i = 1 ∑ N z i , A i z i ≤ λ i b i , ∀ i , λ ∈ Δ ({ 1 , … , N }) } .
Applying this formula for \operatorname{conv}\left(\bigcup_{\pi\in\Pi_{\mathrm{SD}}\overline{\mathcal{V}}^{\pi}\right)} when each P s a \mathcal{P}_{sa} P s a is a polytope yields a linear programming formulation for computing v ⋆ v^\star v ⋆ :
If P s a \mathcal{P}_{sa} P s a is represented as P s a = conv ( P 1 ( s , a ) , … , P k s a ( s , a ) ) \mathcal{P}_{sa}=\operatorname{conv}\left(P^{1}(s,a),\ldots,P^{k_{sa}}(s,a)\right) P s a = conv ( P 1 ( s , a ) , … , P k s a ( s , a ) ) for k s a ∈ N k_{sa}\in\mathbb{N} k s a ∈ N for each state-action pair ( s , a ) (s,a) ( s , a ) we get the following optimization program:
max ∑ s ∈ S v ( s ) v = ∑ π ∈ Π S D v π v π ( s ) ≤ λ π P k ( s , π ( s ) ) T r ( s , π ( s ) ) + γ P k ( s , π ( s ) ) T v π , ∀ π ∈ Π S D , s ∈ S , k ∈ { 1 , … , k s , π ( s ) } , − M λ π ≤ v π ( s ) ≤ M λ π , ∀ π ∈ Π S D , s ∈ S , λ ∈ Δ ( Π S D ) , v π ∈ R S , ∀ π ∈ Π S D . \begin{aligned}
\max\quad&\sum_{s\in\mathcal{S}}v(s)\\
v&=\sum_{\pi\in\Pi_{\mathrm{SD}}}v^{\pi}\\
v^{\pi}(s)&\leq\lambda_{\pi}P^{k}(s,\pi(s))^{\mathsf{T}}r(s,\pi(s))+\gamma P^{k}(s,\pi(s))^{\mathsf{T}}v^{\pi},\quad\forall\ \pi\in\Pi_{\mathrm{SD}},\ s\in\mathcal{S},\ k\in\{1,\ldots,k_{s,\pi(s)}\},\\
-M\lambda_{\pi}&\leq v^{\pi}(s)\leq M\lambda_{\pi},\quad\forall\ \pi\in\Pi_{\mathrm{SD}},\ s\in\mathcal{S},\\
\lambda&\in\Delta(\Pi_{\mathrm{SD}}),\ v^{\pi}\in\mathbb{R}^{S},\quad\forall\ \pi\in\Pi_{\mathrm{SD}}.
\end{aligned} max v v π ( s ) − M λ π λ s ∈ S ∑ v ( s ) = π ∈ Π SD ∑ v π ≤ λ π P k ( s , π ( s ) ) T r ( s , π ( s )) + γ P k ( s , π ( s ) ) T v π , ∀ π ∈ Π SD , s ∈ S , k ∈ { 1 , … , k s , π ( s ) } , ≤ v π ( s ) ≤ M λ π , ∀ π ∈ Π SD , s ∈ S , ∈ Δ ( Π SD ) , v π ∈ R S , ∀ π ∈ Π SD .
Note that in the optimization program above v π v^{\pi} v π is an auxiliary variable and need not be equal to a worst-case value function of π ∈ Π S D \pi\in\Pi_{\mathrm{SD}} π ∈ Π SD .
If P s a \mathcal{P}_{sa} P s a is represented as P s a = { p ∣ A s a T p = b s a , p ≥ 0 } \mathcal{P}_{sa}=\{p\mid A_{sa}^{\mathsf{T}}p=b_{sa},p\geq0\} P s a = { p ∣ A s a T p = b s a , p ≥ 0 } then from strong duality we’d get min { c s a T p ∣ A s a T p = b s a , p ≥ 0 } = max { b s a T y ∣ A s a y ≤ c s a } \min\{c_{sa}^{\mathsf{T}}p\mid A_{sa}^{\mathsf{T}}p=b_{sa},p\geq0\}=\max\{b_{sa}^{\mathsf{T}}y\mid A_{sa}y\leq c_{sa}\} min { c s a T p ∣ A s a T p = b s a , p ≥ 0 } = max { b s a T y ∣ A s a y ≤ c s a } hence we would arrive at the following linear program with unique optimal solution v ∗ v^{*} v ∗ (for the v v v -component of the solutions):
max ∑ s ∈ S v ( s ) v = ∑ π ∈ Π S D v π A s , π ( s ) y s π ≤ λ π r ( s , π ( s ) ) + γ v π , ∀ π ∈ Π S D , s ∈ S , v π ( s ) ≤ b s π ( s ) T y s π , ∀ π ∈ Π S D , s ∈ S , − M λ π ≤ v π ( s ) ≤ M λ π , ∀ π ∈ Π S D , s ∈ S , λ ∈ Δ ( Π S D ) , v π ∈ R S , y s π ∈ R m , ∀ π ∈ Π S D , s ∈ S . \begin{aligned}
\max\quad&\sum_{s\in\mathcal{S}}v(s)\\
v&=\sum_{\pi\in\Pi_{\mathrm{SD}}}v^{\pi}\\
A_{s,\pi(s)}y_{s}^{\pi}&\leq\lambda_{\pi}r(s,\pi(s))+\gamma v^{\pi},\quad\forall\ \pi\in\Pi_{\mathrm{SD}},\ s\in\mathcal{S},\\
v^{\pi}(s)&\leq b_{s\pi(s)}^{\mathsf{T}}y_{s}^{\pi},\quad\forall\ \pi\in\Pi_{\mathrm{SD}},\ s\in\mathcal{S},\\
-M\lambda_{\pi}&\leq v^{\pi}(s)\leq M\lambda_{\pi},\quad\forall\ \pi\in\Pi_{\mathrm{SD}},\ s\in\mathcal{S},\\
\lambda&\in\Delta(\Pi_{\mathrm{SD}}),\ v^{\pi}\in\mathbb{R}^{S},\ y_{s}^{\pi}\in\mathbb{R}^{m},\quad\forall\ \pi\in\Pi_{\mathrm{SD}},\ s\in\mathcal{S}.
\end{aligned} max v A s , π ( s ) y s π v π ( s ) − M λ π λ s ∈ S ∑ v ( s ) = π ∈ Π SD ∑ v π ≤ λ π r ( s , π ( s )) + γ v π , ∀ π ∈ Π SD , s ∈ S , ≤ b s π ( s ) T y s π , ∀ π ∈ Π SD , s ∈ S , ≤ v π ( s ) ≤ M λ π , ∀ π ∈ Π SD , s ∈ S , ∈ Δ ( Π SD ) , v π ∈ R S , y s π ∈ R m , ∀ π ∈ Π SD , s ∈ S .
Note that in both cases since ∣ Π S D ∣ = ∣ A ∣ ∣ S ∣ |\Pi_{\mathrm{SD}}|=|\mathcal{A}|^{|\mathcal{S}|} ∣ Π SD ∣ = ∣ A ∣ ∣ S ∣ the number of variables in the linear programming formulation is exponential in the number of states, so these reformulations are most likely impractical.
Minimality of the rectangularity assumption*
The above sections have highlighted several of the favorable properties of rectangular (s-rectangular and sa-rectangular) uncertainty sets. It is natural to ask if these properties would still hold for other non-rectangular models of uncertainty. To formalize this question, we note that ideally we would like our model of uncertainty to satisfy the desirable properties (e.g., existence of stationary optimal policies, or existence of a robust Bellman equation for the optimal value functions) “across all realistic situations we may face”, i.e., for any instances we may have to solve. Hence in this section we characterize the class of uncertainty sets that have good properties across all choices of the instantaneous rewards r ∈ R S × A × S r\in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} r ∈ R S × A × S , initial distribution p 0 ∈ Δ ( S ) p_{0}\in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) , and discount factor γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) . In Section 3.5.1 we characterize the class of uncertainty sets for which the robust MDP problem (2.1) can be solved via a Bellman equation (also known as a dynamic programming principle ), and in Section 3.5.2 we characterize the class of uncertainty sets for which the robust MDP problem (2.1) always admits a stationary uniformly optimal policy. We will see that each of these properties separately characterizes the rectangularity assumptions.
Minimality for the dynamic programming principle*
Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be a compact uncertainty set. Note that we can always define the robust Bellman operator T T T as in (3.4) (even when P \mathcal{P} P is not s s s -rectangular), and T T T is always a contraction, so that its fixed point v ⋆ ∈ R S v^\star\in\mathbb{R}^{\mathcal{S}} v ⋆ ∈ R S always exists. The main objective of this section is to characterize the set P \mathcal{P} P such that v ⋆ v^\star v ⋆ always encodes the optimal return of the robust MDP problem, e.g., for which the following equality holds across all r ∈ R S × A × S r \in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} r ∈ R S × A × S , all p 0 ∈ Δ ( S ) p_0 \in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) and all γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) :
sup π ∈ Π H min P ∈ P R ( π , P ) = p 0 ⊤ v ⋆ (37) \sup_{\pi\in\Pi_H}\min_{P \in\mathcal{P}} R(\pi,P) = p_0^\top v^\star \tag*{(37)} π ∈ Π H sup P ∈ P min R ( π , P ) = p 0 ⊤ v ⋆ (37)
Equality (37) is referred to as a dynamic programming principle , as it relates the optimal objective of the robust MDP problem to the fixed point of the robust Bellman operator T T T . The next theorem uniquely characterizes the uncertainty set satisfying the dynamic programming principle across all instances. We give a slightly more general statement that allows us to characterize this property beyond convex sets.
Theorem 3.4. Let P \mathcal{P} P be a compact uncertainty set. The following statements are equivalent:
Dynamic programming principle. For any r ∈ R S × A × S r \in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} r ∈ R S × A × S , any p 0 ∈ Δ ( S ) p_0 \in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) and any γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) , we have that
sup π ∈ Π H min P ∈ conv ( P ) R ( π , P ) = p 0 ⊤ v ⋆ , \sup_{\pi\in\Pi_H}\min_{P \in\operatorname{conv}(\mathcal{P})} R(\pi,P) = p_0^\top v^\star, π ∈ Π H sup P ∈ conv ( P ) min R ( π , P ) = p 0 ⊤ v ⋆ , with v ⋆ ∈ R S v^\star\in\mathbb{R}^{\mathcal{S}} v ⋆ ∈ R S the unique fixed point of the robust Bellman operator as defined in (3.4).
Simultaneous solvability property. The following simultaneous solvability property (SSP) holds:
⋂ s ∈ S arg min P ∈ P ∑ a ∈ A P ( s , a ) ⊤ U ( s , a ) ≠ ∅ , ∀ U ∈ R S × A × S . (38) \bigcap_{s \in\mathcal{S}}\operatorname*{arg\,min}_{P \in\mathcal{P}}\sum_{a \in\mathcal{A}}P(s,a)^\top U(s,a) \ne\varnothing,\qquad\forall U \in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}}. \tag*{(38)} s ∈ S ⋂ P ∈ P arg min a ∈ A ∑ P ( s , a ) ⊤ U ( s , a ) = ∅ , ∀ U ∈ R S × A × S . (38) s s s -rectangularity. conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is s s s -rectangular.
Proof of Theorem 3.4. Dynamic programming implies simultaneous solvability. Let U ∈ R S × A × S U \in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} U ∈ R S × A × S and s ∈ S s \in\mathcal{S} s ∈ S . Let f s : Δ ( A ) → R f_s:\Delta(\mathcal{A}) \to\mathbb{R} f s : Δ ( A ) → R be defined as, for x ∈ Δ ( A ) x \in\Delta(\mathcal{A}) x ∈ Δ ( A ) ,
f s ( x ) = min P ∈ P ∑ a ∈ A x ( a ) P ( s , a ) ⊤ U ( s , a ) . f_s(x) = \min_{P \in\mathcal{P}}\sum_{a \in\mathcal{A}}x(a)P(s,a)^\top U(s,a). f s ( x ) = P ∈ P min a ∈ A ∑ x ( a ) P ( s , a ) ⊤ U ( s , a ) .
Note that f s f_s f s is continuous and concave (as the pointwise minimum of some linear functions). Let x ˉ ∈ Δ ( A ) \bar{x} \in\Delta(\mathcal{A}) x ˉ ∈ Δ ( A ) be the uniform distribution over A \mathcal{A} A . The supporting hyperplane theorem (see Appendix § 7 ) gives that there exists a supergradient g s ∈ R A g_s \in\mathbb{R}^{\mathcal{A}} g s ∈ R A such that
f s ( x ) ≤ f s ( x ˉ ) + g s ⊤ ( x − x ˉ ) , ∀ x ∈ Δ ( A ) . (39) f_s(x) \le f_s(\bar{x}) + g_s^\top(x-\bar{x}),\qquad\forall x \in\Delta(\mathcal{A}). \tag*{(39)} f s ( x ) ≤ f s ( x ˉ ) + g s ⊤ ( x − x ˉ ) , ∀ x ∈ Δ ( A ) . (39)
Choose γ = 0 \gamma= 0 γ = 0 and define the rewards as r ( s , a , s ′ ) = U ( s , a , s ′ ) − g s ( a ) r(s,a,s') = U(s,a,s') - g_s(a) r ( s , a , s ′ ) = U ( s , a , s ′ ) − g s ( a ) , so that
min P ∈ P ∑ a ∈ A x ( a ) P ( s , a ) T r ( s , a ) = f s ( x ) − g s T x . \min_{P\in\mathcal{P}}\sum_{a\in\mathcal{A}} x(a)P(s,a)^{\mathsf{T}}r(s,a) = f_s(x) - g_s^{\mathsf{T}}x. P ∈ P min a ∈ A ∑ x ( a ) P ( s , a ) T r ( s , a ) = f s ( x ) − g s T x .
From (39) we get that x ˉ \bar{x} x ˉ is an optimal policy so that v ∗ ( s ) = f s ( x ˉ ) − g s T x ˉ v^*(s) = f_s(\bar{x}) - g_s^{\mathsf{T}}\bar{x} v ∗ ( s ) = f s ( x ˉ ) − g s T x ˉ . For p 0 ∈ Δ ( S ) p_0 \in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) the uniform distribution over S \mathcal{S} S , the dynamic programming principle gives
sup π ∈ Π H min P ∈ conv ( P ) R ( π , P ) = p 0 T v ∗ , \sup_{\pi\in\Pi_H}\min_{P\in\operatorname{conv}(\mathcal{P})} R(\pi,P) = p_0^{\mathsf{T}}v^*, π ∈ Π H sup P ∈ conv ( P ) min R ( π , P ) = p 0 T v ∗ ,
and we conclude that the policy π ˉ \bar{\pi} π ˉ choosing the uniform x ˉ \bar{x} x ˉ at every state is optimal, since we then get
p 0 T v ∗ = ∑ s ∈ S p 0 ( s ) min P ∈ P π ˉ T , P ( s ) ≤ min P ∈ conv ( P ) R ( π ˉ , P ) ≤ sup π ∈ Π H min P ∈ conv ( P ) R ( π , P ) = p 0 T v ∗ . p_0^{\mathsf{T}}v^* = \sum_{s\in\mathcal{S}}p_0(s)\min_{P\in\mathcal{P}} \bar{\pi}^{\mathsf{T},P}(s) \leq\min_{P\in\operatorname{conv}(\mathcal{P})}R(\bar{\pi},P) \leq\sup_{\pi\in\Pi_H}\min_{P\in\operatorname{conv}(\mathcal{P})}R(\pi,P) = p_0^{\mathsf{T}}v^*. p 0 T v ∗ = s ∈ S ∑ p 0 ( s ) P ∈ P min π ˉ T , P ( s ) ≤ P ∈ conv ( P ) min R ( π ˉ , P ) ≤ π ∈ Π H sup P ∈ conv ( P ) min R ( π , P ) = p 0 T v ∗ .
where the last equality is from the dynamic programming principle. Note that with γ = 0 \gamma= 0 γ = 0 we have that P ↦ R ( π ˉ , P ) P \mapsto R(\bar{\pi},P) P ↦ R ( π ˉ , P ) is linear so that min P ∈ P R ( π ˉ , P ) = min P ∈ conv ( P ) R ( π ˉ , P ) \min_{P\in\mathcal{P}}R(\bar{\pi},P) = \min_{P\in\operatorname{conv}(\mathcal{P})}R(\bar{\pi},P) min P ∈ P R ( π ˉ , P ) = min P ∈ conv ( P ) R ( π ˉ , P ) . Hence for P ˉ ∈ arg min P ∈ P R ( π ˉ , P ) \bar{P} \in\arg\min_{P\in\mathcal{P}}R(\bar{\pi},P) P ˉ ∈ arg min P ∈ P R ( π ˉ , P ) a worst-case transition kernel for π ˉ \bar{\pi} π ˉ we get that
0 = ∑ s ∈ S p 0 ( s ) ( ∑ a ∈ A π ˉ ( s , a ) P ˉ ( s , a ) T U ( s , a ) − f s ( x ˉ ) ) . 0 = \sum_{s\in\mathcal{S}}p_0(s)\left(\sum_{a\in\mathcal{A}}\bar{\pi}(s,a)\bar{P}(s,a)^{\mathsf{T}}U(s,a)-f_s(\bar{x})\right). 0 = s ∈ S ∑ p 0 ( s ) ( a ∈ A ∑ π ˉ ( s , a ) P ˉ ( s , a ) T U ( s , a ) − f s ( x ˉ ) ) .
Each term of the sum is nonnegative, hence all terms are 0. Since π ˉ ( s , a ) = 1 / ∣ A ∣ \bar{\pi}(s,a)=1/|\mathcal{A}| π ˉ ( s , a ) = 1/∣ A ∣ is independent of s s s , we get that P ˉ ∈ ⋂ s ∈ S arg min P ∈ P ∑ a ∈ A P ( s , a ) T U ( s , a ) \bar{P}\in\bigcap_{s\in\mathcal{S}}\arg\min_{P\in\mathcal{P}}\sum_{a\in\mathcal{A}}P(s,a)^{\mathsf{T}}U(s,a) P ˉ ∈ ⋂ s ∈ S arg min P ∈ P ∑ a ∈ A P ( s , a ) T U ( s , a ) , so the simultaneous solvability property (38) holds.
Simultaneous solvability implies s s s -rectangularity of the convex hull. To show that conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is s s s -rectangular, we need to show that it is equal to the Cartesian product over its s s s -th marginals. Let P s c = { P ( s ) ∣ P ∈ conv ( P ) } \mathcal{P}_s^c = \{P(s)\mid P\in\operatorname{conv}(\mathcal{P})\} P s c = { P ( s ) ∣ P ∈ conv ( P )} be the set of s s s -th marginals of conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) (we use a superscript here to distinguish this set from the s s s -th marginal of P \mathcal{P} P defined for an s s s -rectangular set in Definition 3.1). We want to show that conv ( P ) = × s ∈ S P s c \operatorname{conv}(\mathcal{P}) = \times_{s\in\mathcal{S}}\mathcal{P}_s^c conv ( P ) = × s ∈ S P s c . Of course we always have conv ( P ) ⊂ × s ∈ S P s c \operatorname{conv}(\mathcal{P}) \subset\times_{s\in\mathcal{S}}\mathcal{P}_s^c conv ( P ) ⊂ × s ∈ S P s c so we need to prove the reverse inclusion.
We first note that the simultaneous solvability property implies that, for each U ∈ R S × A × S U\in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} U ∈ R S × A × S , we have
min P ∈ conv ( P ) ∑ s ∈ S ∑ a ∈ A P ( s , a ) T U ( s , a ) = min P ∈ × s ∈ S P s c ∑ s ∈ S ∑ a ∈ A P ( s , a ) T U ( s , a ) . (40) \min_{P\in\operatorname{conv}(\mathcal{P})}\sum_{s\in\mathcal{S}}\sum_{a\in\mathcal{A}}P(s,a)^{\mathsf{T}}U(s,a)
=
\min_{P\in\times_{s\in\mathcal{S}}\mathcal{P}_s^c}\sum_{s\in\mathcal{S}}\sum_{a\in\mathcal{A}}P(s,a)^{\mathsf{T}}U(s,a). \tag*{(40)} P ∈ conv ( P ) min s ∈ S ∑ a ∈ A ∑ P ( s , a ) T U ( s , a ) = P ∈ × s ∈ S P s c min s ∈ S ∑ a ∈ A ∑ P ( s , a ) T U ( s , a ) . (40)
This is because (a) we are optimizing linear forms so we can replace minimization over P \mathcal{P} P by minimization over conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) (b) the simultaneous solvability property states that an optimal solution of the left-hand side above can be chosen independently for each state s ∈ S s\in\mathcal{S} s ∈ S .
Now for the sake of contradiction, suppose that conv ( P ) ≠ × s ∈ S P s c \operatorname{conv}(\mathcal{P})\neq\times_{s\in\mathcal{S}}\mathcal{P}_s^c conv ( P ) = × s ∈ S P s c . Then we can find P ′ ∈ × s ∈ S P s c P'\in\times_{s\in\mathcal{S}}\mathcal{P}_s^c P ′ ∈ × s ∈ S P s c with P ′ ∉ conv ( P ) P'\notin\operatorname{conv}(\mathcal{P}) P ′ ∈ / conv ( P ) . Let P ′ ′ ∈ conv ( P ) P''\in\operatorname{conv}(\mathcal{P}) P ′′ ∈ conv ( P ) be the orthogonal projection of P ′ P' P ′ onto conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) . The first-order optimality condition for the orthogonal projection (see Appendix § 7 ) implies that ⟨ P ′ ′ − P ′ , P ⟩ ≥ ⟨ P ′ ′ − P ′ , P ′ ′ ⟩ \langle P''-P',P\rangle\geq\langle P''-P',P''\rangle ⟨ P ′′ − P ′ , P ⟩ ≥ ⟨ P ′′ − P ′ , P ′′ ⟩ for any P ∈ conv ( P ) P\in\operatorname{conv}(\mathcal{P}) P ∈ conv ( P ) . However we get
⟨ P ′ ′ − P ′ , P ′ ⟩ = ⟨ P ′ ′ − P ′ , P ′ ′ ⟩ − ∥ P ′ − P ′ ′ ∥ 2 2 < ⟨ P ′ ′ − P ′ , P ′ ′ ⟩ . \langle P''-P',P'\rangle= \langle P''-P',P''\rangle-\lVert P'-P''\rVert_2^2 < \langle P''-P',P''\rangle. ⟨ P ′′ − P ′ , P ′ ⟩ = ⟨ P ′′ − P ′ , P ′′ ⟩ − ∥ P ′ − P ′′ ∥ 2 2 < ⟨ P ′′ − P ′ , P ′′ ⟩ .
Therefore the linear form P ↦ ⟨ P ′ ′ − P ′ , P ⟩ P\mapsto\langle P''-P',P\rangle P ↦ ⟨ P ′′ − P ′ , P ⟩ has a strictly smaller minimum over × s ∈ S P s c \times_{s\in\mathcal{S}}\mathcal{P}_s^c × s ∈ S P s c than over conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) , which contradicts (40) . Hence we must have × s ∈ S P s c = conv ( P ) \times_{s\in\mathcal{S}}\mathcal{P}_s^c=\operatorname{conv}(\mathcal{P}) × s ∈ S P s c = conv ( P ) , and conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is s s s -rectangular.
s-rectangularity implies dynamic programming principle. This is the result from Theorem 3.2 applied to conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) .
Note that the first implication only requires compactness; convexity of conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is necessary in the second and third implications. Recall that T T T , as defined in (26) , takes componentwise minimization over P ∈ P P \in\mathcal{P} P ∈ P for each state s ∈ S s \in\mathcal{S} s ∈ S . Hence the operator T T T only “sees” an s s s -rectangular extension of P \mathcal{P} P , and the reader should not be surprised that the dynamic programming principle implies that conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is s s s -rectangular. The same lines of argument yield the following equivalent results for s a sa s a -rectangular uncertainty sets, except that the Bellman operator now has more structure. In particular, let T ^ : R S → R S \hat{T}: \mathbb{R}^{S} \to\mathbb{R}^{S} T ^ : R S → R S be defined as
T ^ v ( s ) = max π ∈ Π S D min P ∈ P T π , P v ( s ) , ∀ v ∈ R S , s ∈ S . (41) \hat{T}v(s)=\max_{\pi\in\Pi_{\mathrm{SD}}}\min_{P\in\mathcal{P}}T^{\pi,P}v(s),\quad\forall v\in\mathbb{R}^{S},\ s\in\mathcal{S}. \tag*{(41)} T ^ v ( s ) = π ∈ Π SD max P ∈ P min T π , P v ( s ) , ∀ v ∈ R S , s ∈ S . (41)
Note that we do not need any rectangularity assumption on P \mathcal{P} P to define T ^ \hat{T} T ^ . However, since T ^ \hat{T} T ^ is defined componentwise and the outer maximization is over Π S D \Pi_{\mathrm{SD}} Π SD , we get that T ^ \hat{T} T ^ only depends on the ( s , a ) (s,a) ( s , a ) -marginals of P \mathcal{P} P . Hence in some sense T ^ \hat{T} T ^ only depends on a s a sa s a -rectangular extension of P \mathcal{P} P .
Theorem 3.5. Let P \mathcal{P} P be a compact uncertainty set. The following statements are equivalent:
Dynamic programming principle. For any r ∈ R S × A × S r \in\mathbb{R}^{S\times A\times S} r ∈ R S × A × S , any p 0 ∈ Δ ( S ) p_0 \in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) and any γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) , we have that sup π ∈ Π H min P ∈ P R ( π , P ) = p 0 ⊤ v ^ ⋆ \sup_{\pi\in\Pi_{\mathrm{H}}}\min_{P\in\mathcal{P}}R(\pi,P)=p_0^{\top}\hat{v}^{\star} sup π ∈ Π H min P ∈ P R ( π , P ) = p 0 ⊤ v ^ ⋆ , with v ^ ⋆ ∈ R S \hat{v}^{\star}\in\mathbb{R}^{S} v ^ ⋆ ∈ R S the unique fixed point of the robust Bellman operator T ^ \hat{T} T ^ as defined in (41) .
Simultaneous solvability property. The following simultaneous solvability property (SSP) holds:
⋂ ( s , a ) ∈ S × A arg min P ∈ P P ( s , a ) ⊤ U ( s , a ) ≠ ∅ , ∀ U ∈ R S × A × S . (42) \bigcap_{(s,a)\in\mathcal{S}\times\mathcal{A}}\arg\min_{P\in\mathcal{P}}P(s,a)^{\top}U(s,a)\ne\varnothing,\quad\forall U\in\mathbb{R}^{S\times A\times S}. \tag*{(42)} ( s , a ) ∈ S × A ⋂ arg P ∈ P min P ( s , a ) ⊤ U ( s , a ) = ∅ , ∀ U ∈ R S × A × S . (42) s a sa s a -rectangularity. conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is s a sa s a -rectangular.
Proof. Dynamic programming implies simultaneous solvability. Let U ∈ R S × A × S U \in\mathbb{R}^{S\times A\times S} U ∈ R S × A × S and use γ = 0 \gamma=0 γ = 0 . Define r ( s , a , s ′ ) = U ( s , a , s ′ ) − min P ∈ P P ( s , a ) ⊤ U ( s , a ) r(s,a,s^{\prime})=U(s,a,s^{\prime})-\min_{P\in\mathcal{P}}P(s,a)^{\top}U(s,a) r ( s , a , s ′ ) = U ( s , a , s ′ ) − min P ∈ P P ( s , a ) ⊤ U ( s , a ) so that min P ∈ P P ( s , a ) ⊤ r ( s , a ) = 0 \min_{P\in\mathcal{P}}P(s,a)^{\top}r(s,a)=0 min P ∈ P P ( s , a ) ⊤ r ( s , a ) = 0 . Note that this implies that v ^ ⋆ ( s ) = 0 \hat{v}^{\star}(s)=0 v ^ ⋆ ( s ) = 0 for each s ∈ S s\in\mathcal{S} s ∈ S (the worst-case value obtained from each state is 0). Take p 0 p_0 p 0 the uniform initial distribution and π ˉ \bar{\pi} π ˉ the uniform policy, so that
R ( π ˉ , P ) = ∑ s , a p 0 ( s ) π ˉ ( s , a ) P ( s , a ) ⊤ r ( s , a ) ≥ 0 R(\bar{\pi},P)=\sum_{s,a}p_0(s)\bar{\pi}(s,a)P(s,a)^{\top}r(s,a)\ge0 R ( π ˉ , P ) = s , a ∑ p 0 ( s ) π ˉ ( s , a ) P ( s , a ) ⊤ r ( s , a ) ≥ 0
and min P ∈ P R ( π ˉ , P ) = 0 \min_{P\in\mathcal{P}}R(\bar{\pi},P)=0 min P ∈ P R ( π ˉ , P ) = 0 by construction of the rewards and v ^ ⋆ = 0 S \hat{v}^{\star}=0_{\mathcal{S}} v ^ ⋆ = 0 S , since p 0 ⊤ v ^ ⋆ = 0 p_0^{\top}\hat{v}^{\star}=0 p 0 ⊤ v ^ ⋆ = 0 and p 0 ⊤ v ^ ⋆ = sup π ∈ Π H min P ∈ P R ( π , P ) p_0^{\top}\hat{v}^{\star}=\sup_{\pi\in\Pi_{\mathrm{H}}}\min_{P\in\mathcal{P}}R(\pi,P) p 0 ⊤ v ^ ⋆ = sup π ∈ Π H min P ∈ P R ( π , P ) from the dynamic programming principle. For P ˉ ∈ arg min P ∈ P R ( π ˉ , P ) \bar{P}\in\arg\min_{P\in\mathcal{P}}R(\bar{\pi},P) P ˉ ∈ arg min P ∈ P R ( π ˉ , P ) we must have that for all state-action pair ( s , a ) (s,a) ( s , a ) we have P ˉ ∈ arg min P ∈ P P ( s , a ) ⊤ r ( s , a ) = arg min P ∈ P P ( s , a ) ⊤ U ( s , a ) \bar{P}\in\arg\min_{P\in\mathcal{P}}P(s,a)^{\top}r(s,a)=\arg\min_{P\in\mathcal{P}}P(s,a)^{\top}U(s,a) P ˉ ∈ arg min P ∈ P P ( s , a ) ⊤ r ( s , a ) = arg min P ∈ P P ( s , a ) ⊤ U ( s , a ) , which shows that the simultaneous solvability property holds.
Simultaneous solvability implies s a sa s a -rectangularity of the convex hull. This proof is similar to the second part of the proof of Theorem 3.4.
sa-rectangularity implies dynamic programming principle. We first show that (3.20) holds. Indeed for U ∈ R S × A × S U \in\mathbb{R}^{S\times A\times S} U ∈ R S × A × S we have
min P ∈ P ∑ s , a P ( s , a ) ⊤ U ( s , a ) = min P ∈ conv ( P ) ∑ s , a P ( s , a ) ⊤ U ( s , a ) = ∑ s , a min P ∈ conv ( P ) P ( s , a ) ⊤ U ( s , a ) = ∑ s , a min P ∈ P P ( s , a ) ⊤ U ( s , a ) \begin{aligned}
\min_{P\in\mathcal{P}}\sum_{s,a}P(s,a)^\top U(s,a)=\min_{P\in\operatorname{conv}(\mathcal{P})}\sum_{s,a}P(s,a)^\top U(s,a) \\
&=\sum_{s,a}\min_{P\in\operatorname{conv}(\mathcal{P})}P(s,a)^\top U(s,a) \\
&=\sum_{s,a}\min_{P\in\mathcal{P}}P(s,a)^\top U(s,a)
\end{aligned} P ∈ P min s , a ∑ P ( s , a ) ⊤ U ( s , a ) = P ∈ conv ( P ) min s , a ∑ P ( s , a ) ⊤ U ( s , a ) = s , a ∑ P ∈ conv ( P ) min P ( s , a ) ⊤ U ( s , a ) = s , a ∑ P ∈ P min P ( s , a ) ⊤ U ( s , a )
where the first and third equalities are because the objective function is linear, and the second equality is because conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is s a sa s a -rectangular by assumption. The rest of the proof follows from considering (3.20) applied to U ( s , a ) = r ( s , a ) + γ v ^ ⋆ U(s,a)=r(s,a)+\gamma\hat{v}^{\star} U ( s , a ) = r ( s , a ) + γ v ^ ⋆ . In this case we can find P ⋆ ∈ P P^{\star}\in\mathcal{P} P ⋆ ∈ P such that
P ⋆ ∈ ⋂ ( s , a ) ∈ S × A arg min P ∈ P P ( s , a ) ⊤ U ( s , a ) P^{\star}\in\bigcap_{(s,a)\in\mathcal{S}\times\mathcal{A}}\arg\min_{P\in\mathcal{P}}P(s,a)^\top U(s,a) P ⋆ ∈ ( s , a ) ∈ S × A ⋂ arg P ∈ P min P ( s , a ) ⊤ U ( s , a )
and π ⋆ ∈ Π S D \pi^{\star}\in\Pi_{\mathrm{SD}} π ⋆ ∈ Π SD such that
π ⋆ ( s ) ∈ arg max a ∈ A min P ∈ P P ( s , a ) ⊤ ( r ( s , a ) + γ v ^ ⋆ ) = arg max a ∈ A P ⋆ ( s , a ) ⊤ ( r ( s , a ) + γ v ^ ⋆ ) , ∀ s ∈ S . \pi^{\star}(s)\in\arg\max_{a\in\mathcal{A}}\min_{P\in\mathcal{P}}P(s,a)^\top(r(s,a)+\gamma\hat{v}^{\star})=\arg\max_{a\in\mathcal{A}}P^{\star}(s,a)^\top(r(s,a)+\gamma\hat{v}^{\star}),\quad\forall s\in\mathcal{S}. π ⋆ ( s ) ∈ arg a ∈ A max P ∈ P min P ( s , a ) ⊤ ( r ( s , a ) + γ v ^ ⋆ ) = arg a ∈ A max P ⋆ ( s , a ) ⊤ ( r ( s , a ) + γ v ^ ⋆ ) , ∀ s ∈ S .
Then ( π ⋆ , P ⋆ ) (\pi^{\star},P^{\star}) ( π ⋆ , P ⋆ ) is a saddle point: by construction P ⋆ P^{\star} P ⋆ is a worst-case kernel for π ⋆ \pi^{\star} π ⋆ , while π ⋆ \pi^{\star} π ⋆ is an optimal policy for P ⋆ P^{\star} P ⋆ , and v ^ ⋆ = v π ⋆ , P ⋆ \hat{v}^{\star}=v^{\pi^{\star},P^{\star}} v ^ ⋆ = v π ⋆ , P ⋆ . Therefore sup π ∈ Π H min P ∈ P R ( π , P ) = p 0 ⊤ v ^ ⋆ \sup_{\pi\in\Pi_{\mathrm{H}}}\min_{P\in\mathcal{P}}R(\pi,P)=p_{0}^{\top}\hat{v}^{\star} sup π ∈ Π H min P ∈ P R ( π , P ) = p 0 ⊤ v ^ ⋆ .
Discussion
The simultaneous solvability properties (3.16) and (3.20) say that we can optimize linear forms independently across the different dimensions of P ⊂ Δ ( S ) S × A \mathcal{P}\subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A . Hence, the SSPs are inherently statements about the set of extreme points of P \mathcal{P} P ; Theorem 3.4 and Theorem 3.5 characterize the convex hull of P \mathcal{P} P and not necessarily P \mathcal{P} P . In particular if we don’t impose the convexity assumption we can have non-rectangular sets satisfying the SSPs, simply by considering finite s-rectangular sets and adding one point in the middle, as shown in the next example.
Example 3.1. Take S = { s 0 , s 1 } \mathcal{S}=\{s_{0},s_{1}\} S = { s 0 , s 1 } and assume that there is only one action a 1 a_{1} a 1 . We can identify Δ ( S ) S × A \Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} Δ ( S ) S × A with { ( ( p , 1 − p ) , ( q , 1 − q ) ) ∣ p , q ∈ [ 0 , 1 ] } \{((p,1-p),(q,1-q))\mid p,q\in[0,1]\} {(( p , 1 − p ) , ( q , 1 − q )) ∣ p , q ∈ [ 0 , 1 ]} where p p p is the probability to transition to s 0 s_{0} s 0 out of s 0 s_{0} s 0 and q q q is the probability to transition to s 0 s_{0} s 0 out of s 1 s_{1} s 1 . We can identify P ∈ Δ ( S ) S × A P\in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ∈ Δ ( S ) S × A with the pair ( p , q ) ∈ [ 0 , 1 ] 2 (p,q)\in[0,1]^{2} ( p , q ) ∈ [ 0 , 1 ] 2 . For instance P = ( 1 , 1 ) P=(1,1) P = ( 1 , 1 ) corresponds to s 0 s_{0} s 0 absorbing and s 1 → s 0 s_{1}\to s_{0} s 1 → s 0 deterministically. Consider the following finite uncertainty set: P 0 = { ( 0 , 0 ) , ( 0 , 1 ) , ( 1 , 0 ) , ( 1 , 1 ) } \mathcal{P}_{0}=\{(0,0),(0,1),(1,0),(1,1)\} P 0 = {( 0 , 0 ) , ( 0 , 1 ) , ( 1 , 0 ) , ( 1 , 1 )} . We can see P 0 \mathcal{P}_{0} P 0 as the set of four extreme points of the square in [ 0 , 1 ] 2 [0,1]^{2} [ 0 , 1 ] 2 centered at ( 1 / 2 , 1 / 2 ) (1/2,1/2) ( 1/2 , 1/2 ) , and P 0 \mathcal{P}_{0} P 0 is compact s-rectangular (not convex since it is finite) and satisfies (3.16). Now add the center and consider P = { ( 0 , 0 ) , ( 0 , 1 ) , ( 1 , 0 ) , ( 1 , 1 ) , ( 1 / 2 , 1 / 2 ) } \mathcal{P}=\{(0,0),(0,1),(1,0),(1,1),(1/2,1/2)\} P = {( 0 , 0 ) , ( 0 , 1 ) , ( 1 , 0 ) , ( 1 , 1 ) , ( 1/2 , 1/2 )} . Note that P \mathcal{P} P is still compact and not convex, but it is not s-rectangular anymore because of the point ( 1 / 2 , 1 / 2 ) (1/2,1/2) ( 1/2 , 1/2 ) , e.g., ( 1 / 2 , 1 / 2 ) ∈ P (1/2,1/2)\in\mathcal{P} ( 1/2 , 1/2 ) ∈ P and ( 0 , 0 ) ∈ P (0,0)\in\mathcal{P} ( 0 , 0 ) ∈ P but ( 0 , 1 / 2 ) ∉ P (0,1/2)\notin\mathcal{P} ( 0 , 1/2 ) ∈ / P . However the SSP (3.16) still holds for P \mathcal{P} P , because the point ( 1 / 2 , 1 / 2 ) (1/2,1/2) ( 1/2 , 1/2 ) is irrelevant from an optimization standpoint - it is never the unique minimizer of a linear form over P \mathcal{P} P , and P 0 = P ∖ { ( 1 / 2 , 1 / 2 ) } \mathcal{P}_{0}=\mathcal{P}\setminus\{(1/2,1/2)\} P 0 = P ∖ {( 1/2 , 1/2 )} is s-rectangular.
For s-rectangular convex compact uncertainty sets, the existence of a stationary uniformly optimal policy is stated and proved in Theorem 3.2; this is of course very convenient from an implementation standpoint (we do not need to know where we start to play optimally). We show that an even weaker property (just the existence of uniformly optimal policies) completely characterizes s-rectangularity among compact convex uncertainty sets in the next theorem.
Theorem 3.6. Let A , S \mathcal{A},\mathcal{S} A , S be the sets of actions and states with ∣ S ∣ ≥ 2 |\mathcal{S}| \ge2 ∣ S ∣ ≥ 2 , ∣ A ∣ ≥ 2 |\mathcal{A}| \ge2 ∣ A ∣ ≥ 2 . Let P \mathcal{P} P be a compact convex uncertainty set. The following statements are equivalent:
For any choice of r ∈ R S × A × S r \in\mathbb{R}^{\mathcal{S} \times\mathcal{A} \times\mathcal{S}} r ∈ R S × A × S , γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) , there exists a uniformly optimal policy π ⋆ \pi^\star π ⋆ :
∀ r ∈ R S × A × S , γ ∈ [ 0 , 1 ) , ∃ π ⋆ ∈ Π H , ∀ p 0 ∈ Δ ( S ) , min P ∈ P R ( π ⋆ , P ) = sup π ∈ Π H min P ∈ P R ( π , P ) . \forall r \in\mathbb{R}^{\mathcal{S} \times\mathcal{A} \times\mathcal{S}},\ \gamma\in[0,1),\ \exists\pi^\star\in\Pi_{\mathrm{H}},\ \forall p_0 \in\Delta(\mathcal{S}),\quad\min_{P \in\mathcal{P}} R(\pi^\star,P) = \sup_{\pi\in\Pi_{\mathrm{H}}} \min_{P \in\mathcal{P}} R(\pi,P). ∀ r ∈ R S × A × S , γ ∈ [ 0 , 1 ) , ∃ π ⋆ ∈ Π H , ∀ p 0 ∈ Δ ( S ) , P ∈ P min R ( π ⋆ , P ) = π ∈ Π H sup P ∈ P min R ( π , P ) . P \mathcal{P} P is s-rectangular.
Proof. Recall that we have proved that the second statement implies the first one (in Theorem 3.2), so we are left to prove that the first statement implies the second one. Let ∣ A ∣ ≥ 2 |\mathcal{A}| \ge2 ∣ A ∣ ≥ 2 , ∣ S ∣ ≥ 2 |\mathcal{S}| \ge2 ∣ S ∣ ≥ 2 , and P \mathcal{P} P be compact convex. We proceed by contradiction. Assume that the first statement holds but that P \mathcal{P} P is not s-rectangular. Choose γ = 0 \gamma= 0 γ = 0 , so that the return of any history-dependent policy only depends on its chosen first action, so that optimal policies can be chosen stationary anyway. Here we write P ( s ) ⊂ Δ ( S ) A \mathcal{P}(s) \subset\Delta(\mathcal{S})^{\mathcal{A}} P ( s ) ⊂ Δ ( S ) A for the s-th marginal of P \mathcal{P} P : P ( s ) : = { P ( s ) ∣ ∃ P ′ ∈ P , P ′ ( s ) = P ( s ) } \mathcal{P}(s) := \{P(s) \mid\exists P' \in\mathcal{P}, P'(s) = P(s)\} P ( s ) := { P ( s ) ∣ ∃ P ′ ∈ P , P ′ ( s ) = P ( s )} . Following Theorem 3.4, we can choose U 0 ∈ R S × A × S U^0 \in\mathbb{R}^{\mathcal{S} \times\mathcal{A} \times\mathcal{S}} U 0 ∈ R S × A × S such that, for Rec s ( P ) = × s ∈ S P ( s ) \operatorname{Rec}_{s}(\mathcal{P}) = \times_{s \in\mathcal{S}} \mathcal{P}(s) Rec s ( P ) = × s ∈ S P ( s ) the s-rectangular extension of P \mathcal{P} P (see § 4.4 for more detail) we have
min P ∈ P ⟨ U 0 , P ⟩ = min P ∈ P ∑ s ∈ S ⟨ P ( s ) , U 0 ( s ) ⟩ > min P ∈ Rec s ( P ) ⟨ U 0 , P ⟩ = ∑ s ∈ S min P ∈ P ⟨ P ( s ) , U 0 ( s ) ⟩ . (43) \min_{P \in\mathcal{P}} \langle U^0,P\rangle= \min_{P \in\mathcal{P}} \sum_{s \in\mathcal{S}} \langle P(s),U^0(s)\rangle> \min_{P \in\operatorname{Rec}_{s}(\mathcal{P})} \langle U^0,P\rangle= \sum_{s \in\mathcal{S}} \min_{P \in\mathcal{P}} \langle P(s),U^0(s)\rangle. \tag*{(43)} P ∈ P min ⟨ U 0 , P ⟩ = P ∈ P min s ∈ S ∑ ⟨ P ( s ) , U 0 ( s )⟩ > P ∈ Rec s ( P ) min ⟨ U 0 , P ⟩ = s ∈ S ∑ P ∈ P min ⟨ P ( s ) , U 0 ( s )⟩ . (43)
Note that we can perturb U 0 U^0 U 0 to a certain U ∈ R S × A × S U \in\mathbb{R}^{\mathcal{S} \times\mathcal{A} \times\mathcal{S}} U ∈ R S × A × S to both preserve the strict inequality and ensure uniqueness of minimizers on both sides. We can then scale and add a positive constant to every component of U U U to ensure 1 ≤ U ≤ 2 1 \le U \le2 1 ≤ U ≤ 2 (since P ( s , a ) ∈ Δ ( S ) P(s,a) \in\Delta(\mathcal{S}) P ( s , a ) ∈ Δ ( S ) ). Let P ⋆ P^\star P ⋆ be in arg min P ∈ P ⟨ U , P ⟩ \arg\min_{P \in\mathcal{P}} \langle U,P\rangle arg min P ∈ P ⟨ U , P ⟩ . There exists a state s ˉ ∈ S \bar{s} \in\mathcal{S} s ˉ ∈ S such that
⟨ P ⋆ ( s ˉ ) , U ( s ˉ ) ⟩ > min P ∈ P ⟨ P ( s ˉ ) , U ( s ˉ ) ⟩ \langle P^\star(\bar{s}),U(\bar{s})\rangle> \min_{P \in\mathcal{P}} \langle P(\bar{s}),U(\bar{s})\rangle ⟨ P ⋆ ( s ˉ ) , U ( s ˉ )⟩ > P ∈ P min ⟨ P ( s ˉ ) , U ( s ˉ )⟩
and by construction of U U U the objective on the right-hand side, involving only the s-th component of P P P , is minimized at a unique component P ′ ( s ˉ ) P'(\bar{s}) P ′ ( s ˉ ) . Note that both vectors ( P ⋆ ( s ˉ , a ) ⊤ U ( s ˉ , a ) ) a ∈ A (P^\star(\bar{s},a)^\top U(\bar{s},a))_{a \in\mathcal{A}} ( P ⋆ ( s ˉ , a ) ⊤ U ( s ˉ , a ) ) a ∈ A and ( P ′ ( s ˉ , a ) ⊤ U ( s ˉ , a ) ) a ∈ A (P'(\bar{s},a)^\top U(\bar{s},a))_{a \in\mathcal{A}} ( P ′ ( s ˉ , a ) ⊤ U ( s ˉ , a ) ) a ∈ A are strictly positive (by U ≥ 1 U \ge1 U ≥ 1 and P ( s , a ) ∈ Δ ( S ) P(s,a) \in\Delta(\mathcal{S}) P ( s , a ) ∈ Δ ( S ) ) and the first vector has a strictly larger sum than the second. If these vectors are proportional, we can modify the vector U U U to ensure that the new corresponding vectors are non-proportional by adding 1 1 1 to every component of U ( s ˉ , a 1 ) ∈ R S U(\bar{s},a_1) \in\mathbb{R}^{\mathcal{S}} U ( s ˉ , a 1 ) ∈ R S for some particular a 1 ∈ A a_1 \in\mathcal{A} a 1 ∈ A ; this again does not change the strict gap in (3.21), and we now have 1 ≤ U ( s , a , s ′ ) ≤ 3 1 \le U(s,a,s') \le3 1 ≤ U ( s , a , s ′ ) ≤ 3 for any s , a , s ′ s,a,s' s , a , s ′ . We can now construct a uniformly optimal policy for a particular choice of rewards (since we assumed the first statement). We will do so using Kakutani’s fixed point theorem (Theorem 3.3). Consider the correspondence Π S → 2 Π S \Pi_{\mathrm{S}} \to2^{\Pi_{\mathrm{S}}} Π S → 2 Π S that maps each π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S to the set of stationary uniformly optimal policies in the robust MDP with uncertainty set P \mathcal{P} P and with instantaneous rewards defined as r ( s , a , s ′ ) = U ( s , a , s ′ ) / max { π ( s , a ) , ( 6 ∣ A ∣ ) − 1 } r(s,a,s') = U(s,a,s')/\max\{\pi(s,a),(6|\mathcal{A}|)^{-1}\} r ( s , a , s ′ ) = U ( s , a , s ′ ) / max { π ( s , a ) , ( 6∣ A ∣ ) − 1 } (the maximum term in the denominator is here to allow for policies without full support). This correspondence satisfies the assumptions for Theorem 3.3 (recall that γ = 0 \gamma= 0 γ = 0 so that the return is a linear function of the policies and the worst-case return is continuous and concave in the policy). Hence we get that there exists π ⋆ ∈ Π S \pi^\star\in\Pi_{\mathrm{S}} π ⋆ ∈ Π S that is a fixed point of this correspondence. We want to show that π ⋆ ( s , a ) ≥ ( 6 ∣ A ∣ ) − 1 \pi^\star(s,a) \ge(6|\mathcal{A}|)^{-1} π ⋆ ( s , a ) ≥ ( 6∣ A ∣ ) − 1 for each ( s , a ) (s,a) ( s , a ) so that π ⋆ \pi^\star π ⋆ has full support. Fix ( s , a ) (s,a) ( s , a ) and let π ^ \hat{\pi} π ^ that chooses action a a a with probability 1 in state s s s . We have
1 max { π ⋆ ( s , a ) , ( 6 ∣ A ∣ ) − 1 } ≤ min P ∈ P P ( s , a ) T U ( s , a ) max { π ⋆ ( s , a ) , ( 6 ∣ A ∣ ) − 1 } = min P ∈ P v π ^ , P ( s ) ≤ min P ∈ P v π ⋆ , P ( s ) ≤ 3 ∑ a ′ ∈ A π ⋆ ( s , a ′ ) max { π ⋆ ( s , a ′ ) , ( 6 ∣ A ∣ ) − 1 } ≤ 3 ∣ A ∣ . \begin{aligned}
\frac{1}{\max\{\pi^\star(s,a),(6|\mathcal{A}|)^{-1}\}} &\le\min_{P\in\mathcal{P}}\frac{P(s,a)^{\mathsf{T}}U(s,a)}{\max\{\pi^\star(s,a),(6|\mathcal{A}|)^{-1}\}}\\
&= \min_{P\in\mathcal{P}}v^{\hat{\pi},P}(s)\\
&\le\min_{P\in\mathcal{P}}v^{\pi^\star,P}(s)\\
&\le3\sum_{a'\in\mathcal{A}}\frac{\pi^\star(s,a')}{\max\{\pi^\star(s,a'),(6|\mathcal{A}|)^{-1}\}}\\
&\le3|\mathcal{A}|.
\end{aligned} max { π ⋆ ( s , a ) , ( 6∣ A ∣ ) − 1 } 1 ≤ P ∈ P min max { π ⋆ ( s , a ) , ( 6∣ A ∣ ) − 1 } P ( s , a ) T U ( s , a ) = P ∈ P min v π ^ , P ( s ) ≤ P ∈ P min v π ⋆ , P ( s ) ≤ 3 a ′ ∈ A ∑ max { π ⋆ ( s , a ′ ) , ( 6∣ A ∣ ) − 1 } π ⋆ ( s , a ′ ) ≤ 3∣ A ∣.
where the first inequality is from U ( s , a , s ′ ) ≥ 1 U(s,a,s') \ge1 U ( s , a , s ′ ) ≥ 1 , P ( s , a ) ∈ Δ ( S ) P(s,a) \in\Delta(\mathcal{S}) P ( s , a ) ∈ Δ ( S ) , the equality is by definition of the instantaneous rewards and π ^ \hat{\pi} π ^ , the second inequality is from the uniform optimality of π ⋆ \pi^\star π ⋆ , the third inequality is from U ( s , a , s ′ ) ≤ 3 U(s,a,s') \le3 U ( s , a , s ′ ) ≤ 3 , and the fourth inequality is always true. Hence we must have that π ⋆ ( s , a ) ≥ ( 3 ∣ A ∣ ) − 1 > ( 6 ∣ A ∣ ) − 1 \pi^\star(s,a) \ge(3|\mathcal{A}|)^{-1} > (6|\mathcal{A}|)^{-1} π ⋆ ( s , a ) ≥ ( 3∣ A ∣ ) − 1 > ( 6∣ A ∣ ) − 1 , π ⋆ \pi^\star π ⋆ has full support and r ( s , a , s ′ ) = U ( s , a , s ′ ) / π ⋆ ( s , a ) r(s,a,s') = U(s,a,s')/\pi^\star(s,a) r ( s , a , s ′ ) = U ( s , a , s ′ ) / π ⋆ ( s , a ) ; in particular we get a simple expression for v π ⋆ , P ( s ) v^{\pi^\star,P}(s) v π ⋆ , P ( s ) as
v π ⋆ , P ( s ) = ∑ a ∈ A P ( s , a ) T U ( s , a ) . v^{\pi^\star,P}(s)=\sum_{a\in\mathcal{A}}P(s,a)^{\mathsf{T}}U(s,a). v π ⋆ , P ( s ) = a ∈ A ∑ P ( s , a ) T U ( s , a ) .
We can now arrive at a contradiction. First, let p 0 p_0 p 0 be the uniform initial distribution so that R ( π ⋆ , P ) = 1 ∣ S ∣ ⟨ P , U ⟩ R(\pi^\star,P)=\frac{1}{|\mathcal{S}|}\langle P,U\rangle R ( π ⋆ , P ) = ∣ S ∣ 1 ⟨ P , U ⟩ . From (43) the unique worst-case kernel is P ⋆ P^\star P ⋆ , and from the analysis in Proposition 3.7 we must have that π ⋆ \pi^\star π ⋆ maximizes the nominal return against P ⋆ P^\star P ⋆ ; in particular π ⋆ ( s ˉ ) ∈ Δ ( A ) \pi^\star(\bar{s}) \in\Delta(\mathcal{A}) π ⋆ ( s ˉ ) ∈ Δ ( A ) maximizes the return against P ⋆ ( s ˉ ) P^\star(\bar{s}) P ⋆ ( s ˉ ) , i.e.,
π ⋆ ( s ˉ ) ∈ arg max π ( s ˉ ) ∈ Δ ( A ) ∑ a ∈ A π ( s ˉ , a ) P ⋆ ( s ˉ , a ) T r ( s ˉ , a ) . \pi^\star(\bar{s}) \in\underset{\pi(\bar{s})\in\Delta(\mathcal{A})}{\arg\max}\ \sum_{a\in\mathcal{A}}\pi(\bar{s},a)P^\star(\bar{s},a)^{\mathsf{T}}r(\bar{s},a). π ⋆ ( s ˉ ) ∈ π ( s ˉ ) ∈ Δ ( A ) arg max a ∈ A ∑ π ( s ˉ , a ) P ⋆ ( s ˉ , a ) T r ( s ˉ , a ) .
Second, take p 0 p_0 p 0 starting in s ˉ \bar{s} s ˉ with probability 1. Then P ′ ( s ˉ ) ∈ Δ ( S ) A P'(\bar{s}) \in\Delta(\mathcal{S})^{\mathcal{A}} P ′ ( s ˉ ) ∈ Δ ( S ) A is the unique s ˉ \bar{s} s ˉ -component of a worst-case kernel (by construction of U U U ). Therefore π ⋆ ( s ˉ ) \pi^\star(\bar{s}) π ⋆ ( s ˉ ) also maximizes the return against P ′ ( s ˉ ) P'(\bar{s}) P ′ ( s ˉ ) . Now consider the optimality of π ⋆ ( s ˉ ) \pi^\star(\bar{s}) π ⋆ ( s ˉ ) in
max π ( s ˉ ) ∈ Δ ( A ) min P ( s ˉ ) ∈ P ( s ˉ ) ∑ a ∈ A π ( s ˉ , a ) P ( s ˉ , a ) T r ( s ˉ , a ) (44) \max_{\pi(\bar{s})\in\Delta(\mathcal{A})}\ \min_{P(\bar{s})\in\mathcal{P}(\bar{s})}\sum_{a\in\mathcal{A}}\pi(\bar{s},a)P(\bar{s},a)^{\mathsf{T}}r(\bar{s},a) \tag*{(44)} π ( s ˉ ) ∈ Δ ( A ) max P ( s ˉ ) ∈ P ( s ˉ ) min a ∈ A ∑ π ( s ˉ , a ) P ( s ˉ , a ) T r ( s ˉ , a ) (44)
For each P ˉ ( s ˉ ) ∈ { P ′ ( s ˉ ) , P ⋆ ( s ˉ ) } \bar{P}(\bar{s}) \in\{P'(\bar{s}),P^\star(\bar{s})\} P ˉ ( s ˉ ) ∈ { P ′ ( s ˉ ) , P ⋆ ( s ˉ )} , the nominal optimality of π ⋆ ( s ˉ ) \pi^\star(\bar{s}) π ⋆ ( s ˉ ) above for the corresponding initial distribution implies the complementary-slackness condition:
0 = ∑ a ∈ A π ⋆ ( s ˉ , a ) ( max a ′ ∈ A P ˉ ( s ˉ , a ′ ) T r ( s ˉ , a ′ ) − P ˉ ( s ˉ , a ) T r ( s ˉ , a ) ) . 0=\sum_{a\in\mathcal{A}}\pi^\star(\bar{s},a)\left(\max_{a'\in\mathcal{A}}\bar{P}(\bar{s},a')^{\mathsf{T}}r(\bar{s},a')-\bar{P}(\bar{s},a)^{\mathsf{T}}r(\bar{s},a)\right). 0 = a ∈ A ∑ π ⋆ ( s ˉ , a ) ( a ′ ∈ A max P ˉ ( s ˉ , a ′ ) T r ( s ˉ , a ′ ) − P ˉ ( s ˉ , a ) T r ( s ˉ , a ) ) .
Since π ⋆ \pi^\star π ⋆ has full support we must then have
max a ′ ∈ A P ‾ ( s ‾ , a ′ ) ⊤ r ( s ‾ , a ′ ) = P ‾ ( s ‾ , a ) ⊤ r ( s ‾ , a ) , ∀ a ∈ A . \max_{a'\in\mathcal{A}}\overline{P}(\overline{s},a')^\top r(\overline{s},a')=\overline{P}(\overline{s},a)^\top r(\overline{s},a),\ \forall a\in\mathcal{A}. a ′ ∈ A max P ( s , a ′ ) ⊤ r ( s , a ′ ) = P ( s , a ) ⊤ r ( s , a ) , ∀ a ∈ A .
In particular, the scalar c : = P ‾ ( s ‾ , a ) ⊤ r ( s ‾ , a ) c:=\overline{P}(\overline{s},a)^\top r(\overline{s},a) c := P ( s , a ) ⊤ r ( s , a ) is independent of a ∈ A a\in\mathcal{A} a ∈ A . Using the definition of the instantaneous rewards we know that P ‾ ( s ‾ , a ) ⊤ U ( s ‾ , a ) = π ⋆ ( s ‾ , a ) P ‾ ( s ‾ , a ) ⊤ r ( s ‾ , a ) \overline{P}(\overline{s},a)^\top U(\overline{s},a)=\pi^\star(\overline{s},a)\overline{P}(\overline{s},a)^\top r(\overline{s},a) P ( s , a ) ⊤ U ( s , a ) = π ⋆ ( s , a ) P ( s , a ) ⊤ r ( s , a ) . We can determine the constant c c c as follows: since π ⋆ ( s ‾ ) ∈ Δ ( A ) \pi^\star(\overline{s})\in\Delta(\mathcal{A}) π ⋆ ( s ) ∈ Δ ( A ) we have
c = ∑ a ∈ A π ⋆ ( s ‾ , a ) c = ∑ a ∈ A π ⋆ ( s ‾ , a ) P ‾ ( s ‾ , a ) ⊤ r ( s ‾ , a ) = ∑ a ∈ A P ‾ ( s ‾ , a ) ⊤ U ( s ‾ , a ) . c=\sum_{a\in\mathcal{A}}\pi^\star(\overline{s},a)c=\sum_{a\in\mathcal{A}}\pi^\star(\overline{s},a)\overline{P}(\overline{s},a)^\top r(\overline{s},a)=\sum_{a\in\mathcal{A}}\overline{P}(\overline{s},a)^\top U(\overline{s},a). c = a ∈ A ∑ π ⋆ ( s , a ) c = a ∈ A ∑ π ⋆ ( s , a ) P ( s , a ) ⊤ r ( s , a ) = a ∈ A ∑ P ( s , a ) ⊤ U ( s , a ) .
Applying the algebra above to P ‾ ∈ { P ′ , P ⋆ } \overline{P}\in\{P',P^\star\} P ∈ { P ′ , P ⋆ } we conclude that
P ⋆ ( s ‾ , a ) ⊤ U ( s ‾ , a ) = π ⋆ ( s ‾ , a ) c ⋆ P ′ ( s ‾ , a ) ⊤ U ( s ‾ , a ) = π ⋆ ( s ‾ , a ) c ′ (45) \begin{aligned}
P^\star(\overline{s},a)^\top U(\overline{s},a)&=\pi^\star(\overline{s},a)c^\star\\
P'(\overline{s},a)^\top U(\overline{s},a)&=\pi^\star(\overline{s},a)c' \tag*{(45)}
\end{aligned} P ⋆ ( s , a ) ⊤ U ( s , a ) P ′ ( s , a ) ⊤ U ( s , a ) = π ⋆ ( s , a ) c ⋆ = π ⋆ ( s , a ) c ′ (45)
with c ⋆ : = ∑ a ∈ A P ⋆ ( s ‾ , a ) ⊤ U ( s ‾ , a ) , c ′ : = ∑ a ∈ A P ′ ( s ‾ , a ) ⊤ U ( s ‾ , a ) c^\star:=\sum_{a\in\mathcal{A}}P^\star(\overline{s},a)^\top U(\overline{s},a),c':=\sum_{a\in\mathcal{A}}P'(\overline{s},a)^\top U(\overline{s},a) c ⋆ := ∑ a ∈ A P ⋆ ( s , a ) ⊤ U ( s , a ) , c ′ := ∑ a ∈ A P ′ ( s , a ) ⊤ U ( s , a ) . But (45) means that the vectors ( P ⋆ ( s ‾ , a ) ⊤ U ( s ‾ , a ) ) a ∈ A (P^\star(\overline{s},a)^\top U(\overline{s},a))_{a\in\mathcal{A}} ( P ⋆ ( s , a ) ⊤ U ( s , a ) ) a ∈ A and ( P ′ ( s ‾ , a ) ⊤ U ( s ‾ , a ) ) a ∈ A (P'(\overline{s},a)^\top U(\overline{s},a))_{a\in\mathcal{A}} ( P ′ ( s , a ) ⊤ U ( s , a ) ) a ∈ A are both proportional to π ⋆ ( s ‾ ) \pi^\star(\overline{s}) π ⋆ ( s ) , which is a contradiction.
Recall that sa-rectangularity buys two additional properties, compared to s-rectangularity: (a) a stationary uniformly optimal policy can be chosen deterministic , (b) we do not need the convexity assumption. The next theorem shows a converse statement as Theorem 3.6 but for sa-rectangular uncertainty sets. We write Π H D \Pi_{\mathrm{HD}} Π HD for the set of deterministic history-dependent policies.
Theorem 3.7. Let A , S \mathcal{A},\mathcal{S} A , S be the sets of actions and states with ∣ S ∣ ≥ 2 , ∣ A ∣ ≥ 2 |\mathcal{S}|\ge2,|\mathcal{A}|\ge2 ∣ S ∣ ≥ 2 , ∣ A ∣ ≥ 2 . Let P \mathcal{P} P be a compact uncertainty set. The following statements are equivalent:
For any choice of r ∈ R S × A × S r\in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} r ∈ R S × A × S and γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) , there exists a uniformly optimal deterministic policy π ⋆ \pi^\star π ⋆ :
∀ r ∈ R S × A × S , γ ∈ [ 0 , 1 ) , ∃ π ⋆ ∈ Π H D , ∀ p 0 ∈ Δ ( S ) , min P ∈ P R ( π ⋆ , P ) = sup π ∈ Π H min P ∈ P R ( π , P ) . \forall r\in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}},\ \gamma\in[0,1),\ \exists\pi^\star\in\Pi_{\mathrm{HD}},\ \forall p_0\in\Delta(\mathcal{S}),\ \min_{P\in\mathcal{P}}R(\pi^\star,P)=\sup_{\pi\in\Pi_{\mathrm{H}}}\min_{P\in\mathcal{P}}R(\pi,P). ∀ r ∈ R S × A × S , γ ∈ [ 0 , 1 ) , ∃ π ⋆ ∈ Π HD , ∀ p 0 ∈ Δ ( S ) , P ∈ P min R ( π ⋆ , P ) = π ∈ Π H sup P ∈ P min R ( π , P ) . conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is sa-rectangular.
Proof. The fact that the second statement implies the first one was proved in Theorem 3.5, so we just need to prove that the first statement implies the second one. Now assume that the first statement is true. Let γ = 0 \gamma=0 γ = 0 and π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S , we have
R ( π , P ) = ∑ s ∈ S p 0 ( s ) ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ r ( s , a ) R(\pi,\mathcal{P})=\sum_{s\in\mathcal{S}}p_0(s)\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a)^\top r(s,a) R ( π , P ) = s ∈ S ∑ p 0 ( s ) a ∈ A ∑ π ( s , a ) P ( s , a ) ⊤ r ( s , a )
so that min P ∈ P R ( π , P ) = min P ∈ conv ( P ) R ( π , P ) \min_{P\in\mathcal{P}}R(\pi,P)=\min_{P\in\operatorname{conv}(\mathcal{P})}R(\pi,P) min P ∈ P R ( π , P ) = min P ∈ conv ( P ) R ( π , P ) . The necessity proof of Theorem 3.6 only uses γ = 0 \gamma=0 γ = 0 , hence from Theorem 3.6 we must have that conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is s-rectangular. It remains to show that it is sa-rectangular. For the
::: sake of contradiction, assume that conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is not sa-rectangular. From the Simultaneous Solvability Property (42) there exists U ∈ R S × A × S U \in\mathbb{R}^{S \times A \times S} U ∈ R S × A × S such that
⋂ ( s , a ) ∈ S × A arg min P ∈ conv ( P ) P ( s , a ) T U ( s , a ) = ∅ . (46) \bigcap_{(s,a)\in S\times A}\arg\min_{P\in\operatorname{conv}(\mathcal{P})} P(s,a)^{\mathsf{T}}U(s,a)=\varnothing. \tag*{(46)} ( s , a ) ∈ S × A ⋂ arg P ∈ conv ( P ) min P ( s , a ) T U ( s , a ) = ∅ . (46)
Note that we can assume that
min P ∈ conv ( P ) P ( s , a ) T U ( s , a ) = 0 , ∀ ( s , a ) ∈ S × A . (47) \min_{P\in\operatorname{conv}(\mathcal{P})} P(s,a)^{\mathsf{T}}U(s,a)=0,\ \forall(s,a)\in S\times A. \tag*{(47)} P ∈ conv ( P ) min P ( s , a ) T U ( s , a ) = 0 , ∀ ( s , a ) ∈ S × A . (47)
This follows from P ( s , a ) ∈ Δ ( S ) P(s,a)\in\Delta(S) P ( s , a ) ∈ Δ ( S ) so we can subtract any sa-dependent constant from the sa-component of U U U and this yields the same set arg min P ∈ conv ( P ) P ( s , a ) T U ( s , a ) \arg\min_{P\in\operatorname{conv}(\mathcal{P})}P(s,a)^{\mathsf{T}}U(s,a) arg min P ∈ conv ( P ) P ( s , a ) T U ( s , a ) . We now show that because conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) is s-rectangular (despite not being sa-rectangular), there exists a state s 0 ∈ S s_0\in S s 0 ∈ S such that ⋂ a ∈ A arg min P ∈ conv ( P ) P ( s 0 , a ) T U ( s 0 , a ) = ∅ \bigcap_{a\in A}\arg\min_{P\in\operatorname{conv}(\mathcal{P})}P(s_0,a)^{\mathsf{T}}U(s_0,a)=\varnothing ⋂ a ∈ A arg min P ∈ conv ( P ) P ( s 0 , a ) T U ( s 0 , a ) = ∅ . It also satisfies the Simultaneous Solvability Property (38) for s-rectangular sets:
∀ U ′ ∈ R S × A × S , ⋂ s ∈ S arg min P ∈ conv ( P ) ∑ a ∈ A P ( s , a ) T U ′ ( s , a ) ≠ ∅ , \forall U' \in\mathbb{R}^{S\times A\times S},\ \bigcap_{s\in S}\arg\min_{P\in\operatorname{conv}(\mathcal{P})}\sum_{a\in A}P(s,a)^{\mathsf{T}}U'(s,a)\ne\varnothing, ∀ U ′ ∈ R S × A × S , s ∈ S ⋂ arg P ∈ conv ( P ) min a ∈ A ∑ P ( s , a ) T U ′ ( s , a ) = ∅ ,
so there exists P ′ ∈ conv ( P ) P'\in\operatorname{conv}(\mathcal{P}) P ′ ∈ conv ( P ) such that for any s ∈ S s\in S s ∈ S , we have
∑ a ∈ A P ′ ( s , a ) T U ( s , a ) = min P ∈ conv ( P ) ∑ a ∈ A P ( s , a ) T U ( s , a ) . \sum_{a\in A}P'(s,a)^{\mathsf{T}}U(s,a)=\min_{P\in\operatorname{conv}(\mathcal{P})}\sum_{a\in A}P(s,a)^{\mathsf{T}}U(s,a). a ∈ A ∑ P ′ ( s , a ) T U ( s , a ) = P ∈ conv ( P ) min a ∈ A ∑ P ( s , a ) T U ( s , a ) .
If we had that ⋂ a ∈ A arg min P ∈ conv ( P ) P ( s , a ) T U ( s , a ) ≠ ∅ \bigcap_{a\in A}\arg\min_{P\in\operatorname{conv}(\mathcal{P})}P(s,a)^{\mathsf{T}}U(s,a)\ne\varnothing ⋂ a ∈ A arg min P ∈ conv ( P ) P ( s , a ) T U ( s , a ) = ∅ for each s ∈ S s\in S s ∈ S , then
min P ∈ conv ( P ) ∑ a ∈ A P ( s , a ) T U ( s , a ) = 0 \min_{P\in\operatorname{conv}(\mathcal{P})}\sum_{a\in A}P(s,a)^{\mathsf{T}}U(s,a)=0 P ∈ conv ( P ) min a ∈ A ∑ P ( s , a ) T U ( s , a ) = 0
for each s ∈ S s\in S s ∈ S and P ′ ∈ ⋂ ( s , a ) ∈ S × A arg min P ∈ conv ( P ) P ( s , a ) T U ( s , a ) P'\in\bigcap_{(s,a)\in S\times A}\arg\min_{P\in\operatorname{conv}(\mathcal{P})}P(s,a)^{\mathsf{T}}U(s,a) P ′ ∈ ⋂ ( s , a ) ∈ S × A arg min P ∈ conv ( P ) P ( s , a ) T U ( s , a ) , which contradicts (46) . Hence, there exists a state s 0 ∈ S s_0\in S s 0 ∈ S such that ⋂ a ∈ A arg min P ∈ conv ( P ) P ( s 0 , a ) T U ( s 0 , a ) = ∅ \bigcap_{a\in A}\arg\min_{P\in\operatorname{conv}(\mathcal{P})}P(s_0,a)^{\mathsf{T}}U(s_0,a)=\varnothing ⋂ a ∈ A arg min P ∈ conv ( P ) P ( s 0 , a ) T U ( s 0 , a ) = ∅ . Therefore, because all the ( P ( s 0 , a ) T U ( s 0 , a ) ) a ∈ A \left(P(s_0,a)^{\mathsf{T}}U(s_0,a)\right)_{a\in A} ( P ( s 0 , a ) T U ( s 0 , a ) ) a ∈ A are nonnegative:
inf P ∈ conv ( P ) ∑ a ∈ A P ( s 0 , a ) T U ( s 0 , a ) > 0. \inf_{P\in\operatorname{conv}(\mathcal{P})}\sum_{a\in A}P(s_0,a)^{\mathsf{T}}U(s_0,a)>0. P ∈ conv ( P ) inf a ∈ A ∑ P ( s 0 , a ) T U ( s 0 , a ) > 0.
Consider the following reward function: r ( s 0 , a , s ′ ) = U ( s 0 , a , s ′ ) r(s_0,a,s')=U(s_0,a,s') r ( s 0 , a , s ′ ) = U ( s 0 , a , s ′ ) for any ( a , s ′ ) (a,s') ( a , s ′ ) in A × S A\times S A × S , and r ( s , a , s ′ ) = 0 r(s,a,s')=0 r ( s , a , s ′ ) = 0 otherwise. Now consider p 0 p_0 p 0 the initial distribution that puts all mass at s 0 s_0 s 0 . The objective function of a policy π ∈ Π S \pi\in\Pi_S π ∈ Π S is
inf P ∈ conv ( P ) ∑ a ∈ A π ( s 0 , a ) ∑ s ′ ∈ S P ( s 0 , a , s ′ ) r ( s 0 , a , s ′ ) . \inf_{P\in\operatorname{conv}(\mathcal{P})}\sum_{a\in A}\pi(s_0,a)\sum_{s'\in S}P(s_0,a,s')r(s_0,a,s'). P ∈ conv ( P ) inf a ∈ A ∑ π ( s 0 , a ) s ′ ∈ S ∑ P ( s 0 , a , s ′ ) r ( s 0 , a , s ′ ) .
Any deterministic policy obtains a worst-case objective of 0, following (47) . Consider a policy that uniformly randomizes over actions in A A A , i.e., that chooses each action a ∈ A a\in A a ∈ A with probability 1 / ∣ A ∣ 1/|A| 1/∣ A ∣ . Then its objective function is
1 ∣ A ∣ inf P ∈ conv ( P ) ∑ a ∈ A P ( s 0 , a ) T U ( s 0 , a ) > 0. \frac{1}{|A|}\inf_{P\in\operatorname{conv}(\mathcal{P})}\sum_{a\in A}P(s_0,a)^{\mathsf{T}}U(s_0,a)>0. ∣ A ∣ 1 P ∈ conv ( P ) inf a ∈ A ∑ P ( s 0 , a ) T U ( s 0 , a ) > 0.
This shows that in this example, a randomized policy may achieve positive return while all deterministic policies achieve a return equal to 0, which concludes the proof by contradiction.
Discussion. The theorems in this section show that the existence of uniformly optimal policies for all reward parameters is equivalent to rectangularity. The fact that we consider all possible values for r ∈ R S × A × S r \in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} r ∈ R S × A × S is essential in our proof of Theorem 3.6 and Theorem 3.7, and means that we want the existence of uniformly optimal policies for all MDP instances we may encounter. However, one may be interested in a more refined characterization where the set of feasible rewards is a strict subset of R S × A × S \mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} R S × A × S , i.e., where we only consider a particular subset of instances. The case of rewards independent of the next states (i.e., restricting instantaneous rewards to belong to { r ∈ R S × A × S ∣ r ( s , a , s ′ ) = r ( s , a , s ′ ′ ) , ∀ s , s ′ , s ′ ′ ∈ S , a ∈ A } \{r \in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} \mid r(s,a,s') = r(s,a,s''), \forall\ s,s',s'' \in\mathcal{S}, a \in\mathcal{A}\} { r ∈ R S × A × S ∣ r ( s , a , s ′ ) = r ( s , a , s ′′ ) , ∀ s , s ′ , s ′′ ∈ S , a ∈ A } ) is of particular interest and several non-rectangular models of uncertainty retain favorable properties for this class of rewards, see Section 4.9.
Main takeaways
Before giving the exercises in the next section we review the main results of this chapter below:
Policy evaluation (s-rec.). Under the s-rectangularity assumption, the policy evaluation problem can be reformulated as an adversarial MDP , where P \mathcal{P} P is interpreted as a set of (stationary deterministic) policies (Theorem 3.1), hence we can compute the worst-case return of a stationary policy via the fixed point machinery from Chapter 1 for nominal MDPs (Corollary 3.1).
Policy optimization (s-rec.). Under the s-rectangularity assumption, optimal policies may still need to be history-dependent (Proposition 3.5). However if we additionally assume convexity and compactness of the s-rectangular uncertainty set, we can find stationary uniformly optimal policies, and they are characterized by a robust Bellman equation (Theorem 3.2). This can be proved via strong duality in convex optimization or via the existence of a Nash equilibrium (based on Kakutani’s fixed point theorem for correspondences). Optimal policies can then be computed via robust variants of value iteration and policy iteration (Algorithm 5-Algorithm 6), but it is less direct to obtain convex formulations. If the uncertainty set is s-rectangular but not convex, we can still compute the best stationary policies.
The case of sa-rectangular models. Under the sa-rectangularity assumption, we don’t need the convexity assumption anymore and uniformly optimal policies can be chosen stationary, because strong duality holds (Proposition 3.11 ). Additionally optimal policies can be chosen deterministic. We describe algorithms based on value iteration, policy iteration and convex programming in § 3.4.1 .
(*) Necessary and sufficient conditions for tractability. In some sense made rigorous in Section 3.5, rectangularity is the minimal assumption enabling dynamic programming algorithms to compute optimal policies (Theorem 3.4-Theorem 3.5), and enabling the existence of stationary uniformly optimal policies (Theorem 3.6-Theorem 3.7).
Exercises
Alternative definitions of rectangularity.
Prove that P \mathcal{P} P is s-rectangular (as defined in Definition 3.1) if and only if for any two P 1 , P 2 ∈ P \mathcal{P}^{1},\mathcal{P}^{2} \in\mathcal{P} P 1 , P 2 ∈ P and any function f : S → { 1 , 2 } f : \mathcal{S} \to\{1,2\} f : S → { 1 , 2 } , the transition kernel P 3 = ( P f ( s ) ( s ) ) s ∈ S \mathcal{P}^{3} = (\mathcal{P}^{f(s)}(s))_{s\in\mathcal{S}} P 3 = ( P f ( s ) ( s ) ) s ∈ S belongs to P \mathcal{P} P .
Prove that P \mathcal{P} P is s-rectangular if and only if it is equal to the smallest (in the sense of the inclusion) s-rectangular uncertainty set containing P \mathcal{P} P . Can you create an analogous definition with a notion of “largest s-rectangular uncertainty set contained in P \mathcal{P} P ”?
Evaluating the return of general policies. Section 3.2 focuses on solving the policy evaluation problem for a stationary policy π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S . If you are interested in the policy evaluation problem for a history-dependent policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H , you can think for yourself how we can even evaluate the return R ( π , P ) R(\pi,\mathcal{P}) R ( π , P ) for a fixed P ∈ P \mathcal{P} \in\mathcal{P} P ∈ P .
Properties of the return. Show that π ↦ R ( π , P ) \pi\mapsto R(\pi,\mathcal{P}) π ↦ R ( π , P ) need not be quasi-concave and that P ↦ R ( π , P ) \mathcal{P} \mapsto R(\pi,\mathcal{P}) P ↦ R ( π , P ) need not be quasi-convex.
Hint: use the properties of the sets of optimal policies of nominal MDPs and adversarial MDPs.
Strong duality from Brouwer. Brouwer’s fixed point theorem states that a continuous map from a nonempty convex compact set of a finite-dimensional Euclidean space to itself admits a fixed point. Use this theorem to prove strong duality for compact convex s-rectangular uncertainty sets.
Hint: define the corresponding “Nash improvement function”.
Convexification of policy evaluation.
(a) Let n ∈ N n \in\mathbb{N} n ∈ N and E = { 1 , … , m } \mathcal{E} = \{1,\ldots,m\} E = { 1 , … , m } be a finite set of indices. Consider any sets C 1 , … , C m ⊆ R n \mathcal{C}_{1},\ldots,\mathcal{C}_{m} \subseteq\mathbb{R}^{n} C 1 , … , C m ⊆ R n . Show that the convex hull operation and the Cartesian product commute:
conv ( × i ∈ E C i ) = × i ∈ E conv ( C i ) . \operatorname{conv}\left(\mathop{\times}_{i\in\mathcal{E}}\mathcal{C}_{i}\right) = \mathop{\times}_{i\in\mathcal{E}}\operatorname{conv}(\mathcal{C}_{i}). conv ( × i ∈ E C i ) = × i ∈ E conv ( C i ) .
(b) Let P \mathcal{P} P be s-rectangular compact and let π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S .
i. Show that the worst-case Bellman operator T π T^{\pi} T π remains the same if we replace P \mathcal{P} P by conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) .
ii. Conclude that min P ∈ P R ( π , P ) = min P ∈ conv ( P ) R ( π , P ) \min_{\mathcal{P}\in\mathcal{P}} R(\pi,\mathcal{P}) = \min_{\mathcal{P}\in\operatorname{conv}(\mathcal{P})} R(\pi,\mathcal{P}) min P ∈ P R ( π , P ) = min P ∈ conv ( P ) R ( π , P ) for π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S , i.e., why we can replace P \mathcal{P} P by conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) in the policy evaluation problem.
iii. Explain why we cannot (in general) replace P \mathcal{P} P by conv ( P ) \operatorname{conv}(\mathcal{P}) conv ( P ) in the robust MDP problem sup π ∈ Π H inf P ∈ P R ( π , P ) \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\mathcal{P}\in\mathcal{P}} R(\pi,\mathcal{P}) sup π ∈ Π H inf P ∈ P R ( π , P ) .
Optimistic MDPs. The goal of this exercise is to leverage the analysis and tools of this chapter to easily derive the properties for the following optimistic MDP problem:
sup π ∈ Π H sup P ∈ P R ( π , P ) (48) \sup_{\pi\in\Pi_{\mathrm{H}}}\sup_{\mathcal{P}\in\mathcal{P}} R(\pi,\mathcal{P}) \tag*{(48)} π ∈ Π H sup P ∈ P sup R ( π , P ) (48)
given some uncertainty set P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A . (a) Show that for any P ≠ ∅ \mathcal{P} \ne\varnothing P = ∅ (not necessarily s-rectangular, nor convex, nor compact), an optimal policy in (3.26) exists and can be chosen stationary and deterministic. Hint: use the fact that two suprema commute. For the rest of the exercise we assume that P \mathcal{P} P is s-rectangular compact.
(b) Explain how we can solve, for some π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S , the Optimistic Policy Evaluation problem sup P ∈ P R ( π , P ) \sup_{P \in\mathcal{P}} R(\pi,P) sup P ∈ P R ( π , P ) .
(c) Prove that we can interpret (3.26) as a nominal MDP with state set S \mathcal{S} S and action set A × P s \mathcal{A} \times\mathcal{P}_{s} A × P s available at each state s ∈ S s \in\mathcal{S} s ∈ S . Write the instantaneous rewards and transition probabilities in this nominal MDP with augmented sets of actions.
(d) Give an optimality equation for the uniformly optimal stationary policies in (3.26). Hint: use the “optimistic Bellman operator” v ↦ ( max a ∈ A max P ∈ P P ( s , a ) T ( r ( s , a ) + γ v ) ) s ∈ S v \mapsto\left(\max_{a \in\mathcal{A}}\max_{P \in\mathcal{P}} P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v)\right)_{s \in\mathcal{S}} v ↦ ( max a ∈ A max P ∈ P P ( s , a ) T ( r ( s , a ) + γ v ) ) s ∈ S .
(e) Adapt the algorithms from Section 3.3.3 to solve (3.26). Can you easily obtain a convex programming formulation for solving (3.26)?
A non-attainable set of operators. Let P \mathcal{P} P be compact convex non-s-rectangular and let T T T be defined as in (3.4). Show that we can choose some rewards such that { T P ∣ P ∈ P } \{T^{P} \mid P \in\mathcal{P}\} { T P ∣ P ∈ P } does not satisfy the attainability property for T T T . Hint: use γ = 0 \gamma= 0 γ = 0 and v = ( 0 , … , 0 ) ∈ R S v = (0,\ldots,0) \in\mathbb{R}^{\mathcal{S}} v = ( 0 , … , 0 ) ∈ R S .
Convex formulation of s-rectangular robust MDPs. The goal is to outline the steps to obtain a convex formulation for approximating the optimal value function. Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S} \times\mathcal{A}} P ⊂ Δ ( S ) S × A be a compact convex s-rectangular uncertainty set. Let π ˉ ∈ Π S \bar{\pi} \in\Pi_{\mathcal{S}} π ˉ ∈ Π S be the uniform policy.
(a) Let s ∈ S s \in\mathcal{S} s ∈ S . Explain why v ↦ T v ( s ) v \mapsto Tv(s) v ↦ T v ( s ) may not be convex nor concave.
(b) Let b > 0 b > 0 b > 0 and consider the following operator T ~ : R S → R S \widetilde{T} : \mathbb{R}^{\mathcal{S}} \to\mathbb{R}^{\mathcal{S}} T : R S → R S defined as
T ~ v ( s ) = max π ∈ Π S min P ∈ P T π , P v ( s ) − b − 1 K L ( π ( s ) , π ˉ ( s ) ) (49) \widetilde{T}v(s)=\max_{\pi\in\Pi_{\mathcal{S}}}\min_{P\in\mathcal{P}}T^{\pi,P}v(s)-b^{-1}\mathrm{KL}(\pi(s),\bar{\pi}(s)) \tag*{(49)} T v ( s ) = π ∈ Π S max P ∈ P min T π , P v ( s ) − b − 1 KL ( π ( s ) , π ˉ ( s )) (49)
with the Kullback-Leibler divergence defined as K L ( x , π ˉ ( s ) ) = ∑ a ∈ A x ( a ) log ( x ( a ) / π ˉ ( s , a ) ) \mathrm{KL}(x,\bar{\pi}(s))=\sum_{a\in\mathcal{A}}x(a)\log(x(a)/\bar{\pi}(s,a)) KL ( x , π ˉ ( s )) = ∑ a ∈ A x ( a ) log ( x ( a ) / π ˉ ( s , a )) for x ∈ Δ ( A ) x\in\Delta(\mathcal{A}) x ∈ Δ ( A ) , with 0 log ( 0 ) = 0 0\log(0)=0 0 log ( 0 ) = 0 by convention.
i. Show that T ~ \widetilde{T} T is a monotone contraction. How does T v Tv T v compare to T ~ v \widetilde{T}v T v for v ∈ R S v\in\mathbb{R}^{\mathcal{S}} v ∈ R S ?
ii. Let v ~ ⋆ ∈ R S \widetilde{v}^{\star}\in\mathbb{R}^{\mathcal{S}} v ⋆ ∈ R S be the unique fixed point of T ~ \widetilde{T} T . Show that
v ~ ⋆ ( s ) ≤ v ⋆ ( s ) ≤ v ~ ⋆ ( s ) + b − 1 ( 1 − γ ) − 1 log ( ∣ A ∣ ) , ∀ s ∈ S . \widetilde{v}^{\star}(s)\leq v^{\star}(s)\leq\widetilde{v}^{\star}(s)+b^{-1}(1-\gamma)^{-1}\log(|\mathcal{A}|),\qquad\forall\ s\in\mathcal{S}. v ⋆ ( s ) ≤ v ⋆ ( s ) ≤ v ⋆ ( s ) + b − 1 ( 1 − γ ) − 1 log ( ∣ A ∣ ) , ∀ s ∈ S .
iii. Use the Donsker-Varadhan variational formula to show that
T ~ v ( s ) = b − 1 log ( min P ( s ) ∈ P s 1 ∣ A ∣ ∑ a ∈ A exp ( b ⋅ P ( s , a ) T ( r ( s , a ) + γ v ) ) ) , ∀ s ∈ S . \widetilde{T}v(s)=b^{-1}\log\left(\min_{P(s)\in\mathcal{P}_{s}}\frac{1}{|\mathcal{A}|}\sum_{a\in\mathcal{A}}\exp\left(b\cdot P(s,a)^{\mathsf{T}}(r(s,a)+\gamma v)\right)\right),\qquad\forall\ s\in\mathcal{S}. T v ( s ) = b − 1 log ( P ( s ) ∈ P s min ∣ A ∣ 1 a ∈ A ∑ exp ( b ⋅ P ( s , a ) T ( r ( s , a ) + γ v ) ) ) , ∀ s ∈ S .
(c) Let log b \log_b log b and exp b \exp_b exp b be defined as:
log b ( z ) = log ( z ) / b , exp b ( z ′ ) = exp ( b ⋅ z ′ ) , ∀ z > 0 , ∀ z ′ ∈ R . \log_b(z) = \log(z)/b,\qquad\exp_b(z') = \exp(b \cdot z'),\qquad\forall z > 0,\ \forall z' \in\mathbb{R}. log b ( z ) = log ( z ) / b , exp b ( z ′ ) = exp ( b ⋅ z ′ ) , ∀ z > 0 , ∀ z ′ ∈ R .
Show that, coordinate-wise, we have
T ~ v = log b ∘ t ~ ∘ exp b v , \widetilde{T}v = \log_b \circ\widetilde{t} \circ\exp_b v, T v = log b ∘ t ∘ exp b v ,
with the operator t ~ : ( R + ∗ ) S → ( R + ∗ ) S \widetilde{t} : (\mathbb{R}^{*}_{+})^{\mathcal{S}} \to(\mathbb{R}^{*}_{+})^{\mathcal{S}} t : ( R + ∗ ) S → ( R + ∗ ) S defined for each state s ∈ S s \in\mathcal{S} s ∈ S and x ∈ ( R + ∗ ) S x \in(\mathbb{R}^{*}_{+})^{\mathcal{S}} x ∈ ( R + ∗ ) S as
t ~ x ( s ) = min P ( s ) ∈ P s ∑ a ∈ A 1 ∣ A ∣ exp b ( P ( s , a ) T r ( s , a ) ) ∏ s ′ ∈ S x ( s ′ ) γ P ( s , a , s ′ ) . (50) \widetilde{t}x(s) = \min_{P(s)\in\mathcal{P}_s}\sum_{a\in\mathcal{A}}\frac{1}{|\mathcal{A}|}\exp_b\left(P(s,a)^{\mathsf{T}}r(s,a)\right)\prod_{s'\in\mathcal{S}}x(s')^{\gamma P(s,a,s')}. \tag*{(50)} t x ( s ) = P ( s ) ∈ P s min a ∈ A ∑ ∣ A ∣ 1 exp b ( P ( s , a ) T r ( s , a ) ) s ′ ∈ S ∏ x ( s ′ ) γ P ( s , a , s ′ ) . (50)
Prove that t ~ \widetilde{t} t is a componentwise concave function.
(d) Combine the above steps to obtain a convex formulation for computing an ϵ \epsilon ϵ -approximation of v ∗ v^* v ∗ . Hint: use the contraction lemma on T ~ \widetilde{T} T to compute v ~ ∗ \widetilde{v}^* v ∗ . Use t ~ \widetilde{t} t to make a change of variable based on exp b \exp_b exp b ; conclude for b b b large enough.
(e) (*) The approach above can be seen as using an entropic regularizer then doing a corresponding change of variables. Could the same approach work with a different choice of regularizer, for instance the squared-ℓ 2 \ell_2 ℓ 2 norm, if we could adapt the change of variables accordingly?
Complexity of checking the product structure
(a) Assume that you are given a polytope as Q = { x ∈ R n ∣ A x ≤ b } Q = \{x \in\mathbb{R}^n \mid Ax \le b\} Q = { x ∈ R n ∣ A x ≤ b } for some b ∈ R m b \in\mathbb{R}^m b ∈ R m , A ∈ R m × n A \in\mathbb{R}^{m\times n} A ∈ R m × n . How can you check that Q Q Q is a Cartesian product over i ∈ { 1 , … , n } i \in\{1,\ldots,n\} i ∈ { 1 , … , n } ?
(b) Assume that you are given a polytope as Q = conv ( x 1 , … , x p ) Q = \operatorname{conv}(x_1,\ldots,x_p) Q = conv ( x 1 , … , x p ) for some finite set { x 1 , … , x p } ⊂ R n \{x_1,\ldots,x_p\} \subset\mathbb{R}^n { x 1 , … , x p } ⊂ R n . How can you check that Q Q Q is a Cartesian product over i ∈ { 1 , … , n } i \in\{1,\ldots,n\} i ∈ { 1 , … , n } ?
References
The first paper on robust MDPs is [75 ] where the authors consider what is now called sa-rectangular uncertainty sets and note the similarity with Shapley’s stochastic games [73 ] (in fact sa-rectangular robust MDPs are closer to perfect-information stochastic games than to Shapley’s concurrent stochastic games where players choose their decisions at the same time, see Chapter 4). [28 ] use such an sa-rectangular model (for “interval MDPs” where the uncertainty set is such that each transition probability P ( s , a , s ′ ) P(s,a,s') P ( s , a , s ′ ) belongs to a prescribed interval). The model now called “sa-rectangular” comes from [58 , 36 ] and the name “s-rectangular” comes from [80 ] ([49 ] uses the term “state-rectangular”).
The first proof of strong duality for convex compact s-rectangular models presented above is from [58 , 80 ] and the second proof (based on Kakutani’s fixed point theorem) and third proof (based on Brouwer’s fixed point theorem, in § 3.8 ) are new (as far as I know); [49 ] proves strong duality for s-rectangular models by invoking the seminal results from Shapley’s paper [73 ] . The connection between Nash equilibria and strong duality in two-player zero-sum games is standard.
For the adversarial MDP in the s-rectangular case I follow [34 ] . This idea is present in both [58 , 36 ] for the sa-rectangular case (this is implied by the connection with perfect-information stochastic games given by these authors), and it is present in [22 , 27 ] for other models of uncertainty (r-rectangular under some additional assumptions on the rewards); it is worth noting that in stochastic games the idea that the decision problem faced by one player (fixing the opponent’s strategy) is an MDP problem appeared from the start [73 ] .
Policy iteration as in Algorithm 6 is akin to the Strategy Iteration proposed by Hoffman and Karp [30 ] and [61 ] for stochastic games. A convergence rate of Strategy Iteration is given in [32 ] (see [1 ] for an improved convergence rate) for perfect-information stochastic games, a setup close to sa-rectangular robust MDPs.
There are many papers that give efficient algorithms for computing the robust Bellman operator T T T and the worst-case Bellman operator T π T^{\pi} T π . [58 , 36 ] give algorithms for T π T^{\pi} T π for sa-rectangular uncertainty sets based on ℓ 2 \ell_2 ℓ 2 -balls, ℓ 1 \ell_1 ℓ 1 -balls, ℓ ∞ \ell_{\infty} ℓ ∞ -balls, entropy, finite models, and the Kullback-Leibler divergence. [28 ] focuses on sa-rectangular interval models (similar to models based on ℓ ∞ \ell_{\infty} ℓ ∞ -balls). [34 , 35 , 7 ] give algorithms for fast Bellman updates for s-rectangular robust MDPs based on ℓ 1 \ell_1 ℓ 1 -balls, ℓ ∞ \ell_{\infty} ℓ ∞ -balls, and ϕ \phi ϕ -divergences. A convex formulation for approximating the optimal value functions of s-rectangular robust MDPs is given in [23 ] . The convex formulation based on disjunctive programming for sa-rectangular uncertainty sets is from some unpublished research notes with Marek Petrik; the theorem from Balas for disjunctive programming can be found as Theorem 5.1 in [11 ] . The reader interested in the case of non-convex uncertainty sets can find some discussion in [81 ] .
The connection between optimistic MDPs and robust MDPs (related to the exercise outlined in § 3.8 ) is well known, for instance [28 , 78 ] deal with both optimistic and robust (pessimistic) MDPs at the same time while [36 ] argue (in their Remark 2.1) that their results for robust MDPs imply the corresponding results for optimistic MDPs. What we show in the exercise in § 3.8 is slightly stronger (in particular stationary deterministic optimal policies always exist for optimistic MDPs, in contrast to robust MDPs). The idea to interpret optimistic MDPs (with rectangular uncertainty sets) as nominal MDPs with augmented sets of actions was suggested to me by Raimundo Saona in research discussion and is also not new, see e.g. section 3.1.1 in the UCRL2 paper [37 ] .
The minimality of rectangularity properties is studied in [25 ] (for the dynamic programming principle for the policy evaluation problem) and [6 ] (for the existence of uniformly optimal policies); the reader interested in these types of necessary and sufficient conditions for tractability of robust MDPs may also be interested in [5 ] (which also highlights the attainability property) and [48 ] .
Chapter 4
Connection with other models
This chapter discusses the connection of the robust MDP model studied in these notes with several other models for sequential decision-making in uncertain environments:
In § 4.1 and § 4.2 we analyze robust MDPs where the uncertain rewards are unknown, both without and with uncertainty in the transition probabilities.
In § 4.3 we analyze the connection between multi-objective MDPs, Pareto optimality and robust MDPs.
In § 4.4 we study three different variants of robust MDPs where the transition probabilities may change over time and in § 5.1 we highlight the connection with stochastic games.
In § 5.2 we show that distributionally robust MDPs reduce to robust MDPs (despite being a generalization) and illustrate this with ambiguity sets based on Wasserstein distances.
In § 5.3 we study three classes of non-rectangular uncertainty sets leading to favorable tractability properties (under some additional assumptions on the model).
In § 5.5 we analyze the reformulations of risk-averse MDPs as robust MDPs for some particular classes of risk measures (coherent risk measures).
In Section 4.9 we discuss the natural assumptions that we can make about the reward models and analyze which ones are without loss of generality.
A “visual” summary of the main takeaways is given in § 5.7 . Throughout we assume P ≠ ∅ \mathcal{P} \ne\varnothing P = ∅ .
Robust MDPs with uncertain rewards
In the previous chapters we assumed that the reward parameters are known to the decision maker and that only the transition probabilities are uncertain. Here we first consider the corresponding setting, where the transition probabilities are known to the decision maker but all we know about the rewards is that they belong to a set R ⊂ R S × A × S \mathcal{R} \subset\mathbb{R}^{S \times A \times S} R ⊂ R S × A × S . Since we keep P P P fixed here and vary r r r , we write R ( π , r ) R(\pi,r) R ( π , r ) and v π , r v^{\pi,r} v π , r for the return and value function of a policy π ∈ Π H \pi\in\Pi_H π ∈ Π H given a realization of r ∈ R r \in\mathcal{R} r ∈ R . The goal in this section is to analyze the following optimization problem:
sup π ∈ Π H inf r ∈ R R ( π , r ) (51) \sup_{\pi\in\Pi_H} \inf_{r \in\mathcal{R}} R(\pi,r) \tag*{(51)} π ∈ Π H sup r ∈ R inf R ( π , r ) (51)
We have seen in § 1.1 that we can express the return R ( π , r ) R(\pi,r) R ( π , r ) of a policy π ∈ Π H \pi\in\Pi_H π ∈ Π H as
R ( π , r ) = ∑ s ∈ S ∑ a ∈ A μ π ( s , a ) P ( s , a ) ⊤ r ( s , a ) (52) R(\pi,r) = \sum_{s \in S} \sum_{a \in A} \mu^\pi(s,a)P(s,a)^\top r(s,a) \tag*{(52)} R ( π , r ) = s ∈ S ∑ a ∈ A ∑ μ π ( s , a ) P ( s , a ) ⊤ r ( s , a ) (52)
where μ π ∈ R S × A \mu^\pi\in\mathbb{R}^{S \times A} μ π ∈ R S × A are the state-action frequencies defined in (7) . An amazing property of the above expression for the return is that it is linear in the reward function r r r , which will prove central in all the arguments in this section.
Policy evaluation. We start by considering the problem of solving inf r ∈ R R ( π , r ) \inf_{r \in\mathcal{R}} R(\pi,r) inf r ∈ R R ( π , r ) for some fixed π ∈ Π S \pi\in\Pi_S π ∈ Π S . From (52) , we get that, for c π ∈ R S × A × S c^\pi\in\mathbb{R}^{S \times A \times S} c π ∈ R S × A × S defined as c π ( s , a , s ′ ) = μ π ( s , a ) P ( s , a , s ′ ) c^\pi(s,a,s') = \mu^\pi(s,a)P(s,a,s') c π ( s , a , s ′ ) = μ π ( s , a ) P ( s , a , s ′ ) , this optimization problem can be reformulated as
inf { c π ⊤ r ∣ r ∈ R } (53) \inf\{c^\pi{}^\top r \mid r \in\mathcal{R}\} \tag*{(53)} inf { c π ⊤ r ∣ r ∈ R } (53)
where we still use the scalar product notation ( ⊤ ) (^\top) ( ⊤ ) for objects in R S × A × S \mathbb{R}^{S \times A \times S} R S × A × S :
c π ⊤ r = ∑ s , a , s ′ c π ( s , a , s ′ ) r ( s , a , s ′ ) . c^\pi{}^\top r = \sum_{s,a,s'} c^\pi(s,a,s')r(s,a,s'). c π ⊤ r = s , a , s ′ ∑ c π ( s , a , s ′ ) r ( s , a , s ′ ) .
The mathematical program (53) minimizes a linear form over the set R \mathcal{R} R ; whether this is doable efficiently (e.g., in polynomial time in the input size of the problem) depends on the properties of the set R ⊂ R S × A × S \mathcal{R} \subset\mathbb{R}^{S \times A \times S} R ⊂ R S × A × S , but for instance efficient algorithms exist if R \mathcal{R} R is a polyhedron. Of course to solve (53) we need to compute c π c^\pi c π and therefore we need to compute μ π \mu^\pi μ π first, but this can be done efficiently by solving a linear system of equations, as argued at the end of § 1.1 .
Properties of optimal policies. We first note that optimal policies in (51) can always be chosen stationary for a compact uncertainty set R \mathcal{R} R , with a proof much simpler than for the case of uncertain transitions.
Proposition 4.1. Let R ⊂ R S × A × S \mathcal{R} \subset\mathbb{R}^{S \times A \times S} R ⊂ R S × A × S be a compact set. Then there exists a stationary optimal policy:
sup π ∈ Π H min r ∈ R R ( π , r ) = max π ∈ Π S min r ∈ R R ( π , r ) . \sup_{\pi\in\Pi_H} \min_{r \in\mathcal{R}} R(\pi,r) = \max_{\pi\in\Pi_S} \min_{r \in\mathcal{R}} R(\pi,r). π ∈ Π H sup r ∈ R min R ( π , r ) = π ∈ Π S max r ∈ R min R ( π , r ) .
Proof. This follows from
sup π ∈ Π H min r ∈ R R ( π , r ) = sup π ∈ Π H min r ∈ R ∑ s , a μ π ( s , a ) P ( s , a ) T r ( s , a ) = sup μ ∈ Q min r ∈ R ∑ s , a μ ( s , a ) P ( s , a ) T r ( s , a ) = sup π ∈ Π S min r ∈ R ∑ s , a μ π ( s , a ) P ( s , a ) T r ( s , a ) = max π ∈ Π S min r ∈ R R ( π , r ) \begin{aligned}
\sup_{\pi\in\Pi_{\mathrm H}}\min_{r\in\mathcal{R}}R(\pi,r)
&=\sup_{\pi\in\Pi_{\mathrm H}}\min_{r\in\mathcal{R}}\sum_{s,a}\mu^\pi(s,a)P(s,a)^{\mathsf{T}}r(s,a)\\
&=\sup_{\mu\in\mathcal{Q}}\min_{r\in\mathcal{R}}\sum_{s,a}\mu(s,a)P(s,a)^{\mathsf{T}}r(s,a)\\
&=\sup_{\pi\in\Pi_{\mathrm S}}\min_{r\in\mathcal{R}}\sum_{s,a}\mu^\pi(s,a)P(s,a)^{\mathsf{T}}r(s,a)\\
&=\max_{\pi\in\Pi_{\mathrm S}}\min_{r\in\mathcal{R}}R(\pi,r)
\end{aligned} π ∈ Π H sup r ∈ R min R ( π , r ) = π ∈ Π H sup r ∈ R min s , a ∑ μ π ( s , a ) P ( s , a ) T r ( s , a ) = μ ∈ Q sup r ∈ R min s , a ∑ μ ( s , a ) P ( s , a ) T r ( s , a ) = π ∈ Π S sup r ∈ R min s , a ∑ μ π ( s , a ) P ( s , a ) T r ( s , a ) = π ∈ Π S max r ∈ R min R ( π , r )
where the first equality is via the expression of the return R ( π , r ) R(\pi,r) R ( π , r ) as a function of the state-action frequencies as shown in § 1.1 , and the second and third equalities are because the polytope Q \mathcal{Q} Q of the state-action frequencies induced by all π ∈ Π H \pi\in\Pi_{\mathrm H} π ∈ Π H coincides with the state-action frequencies induced by all π ∈ Π S \pi\in\Pi_{\mathrm S} π ∈ Π S .
Note that we may need randomization at optimality, as shown in the next simple example.
Example 4.1. Consider an MDP instance with one state s 0 s_0 s 0 and two actions a 1 , a 2 a_1,a_2 a 1 , a 2 . The rewards are either r 1 r_1 r 1 with r 1 ( s 0 , a 1 , s 0 ) = 1 , r 1 ( s 0 , a 2 , s 0 ) = 0 r_1(s_0,a_1,s_0)=1,r_1(s_0,a_2,s_0)=0 r 1 ( s 0 , a 1 , s 0 ) = 1 , r 1 ( s 0 , a 2 , s 0 ) = 0 or r 2 r_2 r 2 with r 2 ( s 0 , a 1 , s 0 ) = 0 , r 2 ( s 0 , a 2 , s 0 ) = 1 r_2(s_0,a_1,s_0)=0,r_2(s_0,a_2,s_0)=1 r 2 ( s 0 , a 1 , s 0 ) = 0 , r 2 ( s 0 , a 2 , s 0 ) = 1 . We can parametrize a stationary policy by x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] the probability to play a 1 a_1 a 1 in state s 0 s_0 s 0 , which gives a worst-case return of v x ( s 0 ) = 1 1 − γ min { x , 1 − x } v^x(s_0)=\frac{1}{1-\gamma}\min\{x,1-x\} v x ( s 0 ) = 1 − γ 1 min { x , 1 − x } . Hence an optimal policy chooses the randomized decision x ⋆ = 1 / 2 x^\star=1/2 x ⋆ = 1/2 .
It is also worth emphasizing that without a rectangularity assumption for R \mathcal{R} R (of the kind R = × s ∈ S R s \mathcal{R}=\times_{s\in\mathcal{S}}\mathcal{R}_s R = × s ∈ S R s with R s ⊂ R A × S \mathcal{R}_s\subset\mathbb{R}^{A\times S} R s ⊂ R A × S ), there may not exist uniformly optimal policies in general, i.e., an optimal policy may need to depend on where we start.
Example 4.2. Consider the MDP instance with two states s 0 , s 1 s_0,s_1 s 0 , s 1 , two actions a 1 , a 2 a_1,a_2 a 1 , a 2 and take γ = 0 \gamma=0 γ = 0 . The rewards are independent of the next state and are given by r t ( s 0 , a 1 ) = 1 − t , r t ( s 0 , a 2 ) = − 1 , r t ( s 1 , a 1 ) = 0 , r t ( s 1 , a 2 ) = 2 t − 1 r_t(s_0,a_1)=1-t,r_t(s_0,a_2)=-1,r_t(s_1,a_1)=0,r_t(s_1,a_2)=2t-1 r t ( s 0 , a 1 ) = 1 − t , r t ( s 0 , a 2 ) = − 1 , r t ( s 1 , a 1 ) = 0 , r t ( s 1 , a 2 ) = 2 t − 1 , parametrized by t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] . In s 0 s_0 s 0 the adversary wants to choose t t t large while in s 1 s_1 s 1 the adversary wants to choose t t t small; this will prevent the existence of a uniformly optimal policy here. Note that we do not specify the transition probabilities because they are irrelevant (γ = 0 \gamma=0 γ = 0 and the rewards are independent of the next states). Let us write x = π ( s 0 , a 1 ) , y = π ( s 1 , a 1 ) x=\pi(s_0,a_1),y=\pi(s_1,a_1) x = π ( s 0 , a 1 ) , y = π ( s 1 , a 1 ) with x , y ∈ [ 0 , 1 ] x,y\in[0,1] x , y ∈ [ 0 , 1 ] . If we start in s 1 s_1 s 1 with probability 1 then the worst-case return is min t ∈ [ 0 , 1 ] ( 1 − y ) ( 2 t − 1 ) = y − 1 \min_{t\in[0,1]}(1-y)(2t-1)=y-1 min t ∈ [ 0 , 1 ] ( 1 − y ) ( 2 t − 1 ) = y − 1 attained at t = 0 t=0 t = 0 and maximized at y = 1 y=1 y = 1 . If we start in s 0 s_0 s 0 and s 1 s_1 s 1 each with probability 1 / 2 1/2 1/2 then the worst-case return is min { x + y 2 − 1 , x − y 2 } \min\{x+\frac{y}{2}-1,\frac{x-y}{2}\} min { x + 2 y − 1 , 2 x − y } which is maximized at x = 1 x=1 x = 1 and y = 1 / 2 y=1/2 y = 1/2 . Hence there are no optimal policies in common for p 0 p_0 p 0 starting in s 1 s_1 s 1 and p 0 p_0 p 0 the uniform distribution over { s 0 , s 1 } \{s_0,s_1\} { s 0 , s 1 } , and therefore there is no uniformly optimal policy.
When the set R \mathcal{R} R is additionally convex compact, we obtain a strong duality result akin to that of Chapter 3; again the proof is much simpler than the corresponding results for uncertain transitions because of the bilinearity of R ( π , r ) R(\pi,r) R ( π , r ) in μ π \mu^\pi μ π and r r r .
Proposition 4.2. Let R ⊂ R S × A × S \mathcal{R}\subset\mathbb{R}^{S\times A\times S} R ⊂ R S × A × S be a convex compact set. Then the following strong duality result holds:
max π ∈ Π S min r ∈ R R ( π , r ) = min r ∈ R max π ∈ Π S R ( π , r ) (54) \max_{\pi\in\Pi_{\mathrm S}}\min_{r\in\mathcal{R}}R(\pi,r)
=
\min_{r\in\mathcal{R}}\max_{\pi\in\Pi_{\mathrm S}}R(\pi,r) \tag*{(54)} π ∈ Π S max r ∈ R min R ( π , r ) = r ∈ R min π ∈ Π S max R ( π , r ) (54)
Proof. We have
max π ∈ Π S min r ∈ R R ( π , r ) = max μ ∈ Q min r ∈ R ∑ ( s , a ) μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) = min r ∈ R max μ ∈ Q ∑ ( s , a ) μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) = min r ∈ R max π ∈ Π S R ( π , r ) \begin{aligned}
\max_{\pi\in\Pi_S}\min_{r\in\mathcal{R}} R(\pi,r)
= \max_{\mu\in\mathcal{Q}}\min_{r\in\mathcal{R}} \sum_{(s,a)} \mu(s,a)P(s,a)^\top r(s,a) \\
&= \min_{r\in\mathcal{R}}\max_{\mu\in\mathcal{Q}} \sum_{(s,a)} \mu(s,a)P(s,a)^\top r(s,a) \\
&= \min_{r\in\mathcal{R}}\max_{\pi\in\Pi_S} R(\pi,r)
\end{aligned} π ∈ Π S max r ∈ R min R ( π , r ) = μ ∈ Q max r ∈ R min ( s , a ) ∑ μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) = r ∈ R min μ ∈ Q max ( s , a ) ∑ μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) = r ∈ R min π ∈ Π S max R ( π , r )
where the first equality is via the expression of the return R ( π , r ) R(\pi,r) R ( π , r ) as a function of the state-action frequency, the second equality is via strong duality for maxmin optimization problems with bilinear objectives and convex compact domains, and the third equality uses again the expression of R ( π , r ) R(\pi,r) R ( π , r ) .
Computation of optimal policies. To compute an optimal policy we want to solve max π ∈ Π S min r ∈ R R ( π , r ) \max_{\pi\in\Pi_S}\min_{r\in\mathcal{R}} R(\pi,r) max π ∈ Π S min r ∈ R R ( π , r ) . As we saw in the proof of Proposition 4.1, this can be reformulated as
max μ ∈ Q min r ∈ R ∑ ( s , a ) μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) . (55) \max_{\mu\in\mathcal{Q}}\min_{r\in\mathcal{R}} \sum_{(s,a)} \mu(s,a)P(s,a)^\top r(s,a). \tag*{(55)} μ ∈ Q max r ∈ R min ( s , a ) ∑ μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) . (55)
This last optimization problem has a bilinear objective and its tractability property will depend on the set R \mathcal{R} R . In general we can write
max μ ∈ Q min r ∈ R ∑ ( s , a ) μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) = max t \max_{\mu\in\mathcal{Q}}\min_{r\in\mathcal{R}} \sum_{(s,a)} \mu(s,a)P(s,a)^\top r(s,a) = \max t μ ∈ Q max r ∈ R min ( s , a ) ∑ μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) = max t
t ≤ min r ∈ R ∑ ( s , a ) μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) , t \le\min_{r\in\mathcal{R}} \sum_{(s,a)} \mu(s,a)P(s,a)^\top r(s,a), t ≤ r ∈ R min ( s , a ) ∑ μ ( s , a ) P ( s , a ) ⊤ r ( s , a ) ,
t ∈ R , μ ∈ Q . t\in\mathbb{R},\ \mu\in\mathcal{Q}. t ∈ R , μ ∈ Q .
The tractability property of the right-hand side can be described in some simple settings:
When R \mathcal{R} R consists of a finite set of scenarios: R = { r 1 , … , r K } \mathcal{R}=\{r_1,\ldots,r_K\} R = { r 1 , … , r K } for some K ∈ N K\in\mathbb{N} K ∈ N and r i ∈ R S × A × S r_i\in\mathbb{R}^{S\times A\times S} r i ∈ R S × A × S for i ∈ { 1 , … , K } i\in\{1,\ldots,K\} i ∈ { 1 , … , K } we get a linear program:
t ≤ ∑ ( s , a ) μ ( s , a ) P ( s , a ) ⊤ r i ( s , a ) , ∀ i ∈ { 1 , … , K } , t \le\sum_{(s,a)} \mu(s,a)P(s,a)^\top r_i(s,a),\ \forall i\in\{1,\ldots,K\}, t ≤ ( s , a ) ∑ μ ( s , a ) P ( s , a ) ⊤ r i ( s , a ) , ∀ i ∈ { 1 , … , K } ,
t ∈ R , μ ∈ Q . t\in\mathbb{R},\ \mu\in\mathcal{Q}. t ∈ R , μ ∈ Q .
When R \mathcal{R} R is a polytope, we can use strong duality to turn the minimization problem over r ∈ R r\in\mathcal{R} r ∈ R in the constraints into a maximization problem, from which we can get again a linear program.
Comparison with robust MDPs with uncertain transitions. Robust MDPs with uncertain rewards appear to have much more favorable properties than robust MDPs with uncertain transitions, in that we need fewer assumptions on the set R \mathcal{R} R (no rectangularity assumption is necessary to derive the results in this section) to get tractable problems (i.e., with stationary optimal solutions that we can compute) than on P \mathcal{P} P . This discrepancy comes from the linearity of the return r ↦ R ( π , r ) r \mapsto R(\pi,r) r ↦ R ( π , r ) in r r r , whereas in general P ↦ R ( π , P ) \mathcal{P} \mapsto R(\pi,\mathcal{P}) P ↦ R ( π , P ) is not a linear function of the transition probabilities (since we need to invert a matrix in (5) ).
What is observed? (part 1) It is worth taking a second look at Example 4.1. In this example it is a bit puzzling that non-stationary policies can’t outperform stationary policies (as proved in Proposition 4.1). In particular, if we could observe the instantaneous rewards obtained at each period, then we could determine whether the realization of r r r chosen by the adversary is r 1 r_1 r 1 or r 2 r_2 r 2 (simply by playing a 1 a_1 a 1 with probability 1 / 2 1/2 1/2 during the first period and checking whether we obtained a reward of 0 or 1), and then play accordingly onward (that is, if the realized reward is 1 we keep playing the same action as in period 0, otherwise the realized reward is 0 and we should keep playing the other action). This yields a return of 1 2 + γ 1 − γ \frac{1}{2}+\frac{\gamma}{1-\gamma} 2 1 + 1 − γ γ , beating the stationary optimum when γ > 0 \gamma> 0 γ > 0 . The subtlety lies in the definition of history-dependent policies. Recall that history-dependent policies Π H \Pi_{\mathrm{H}} Π H are defined in Section 1.1 as maps from all finite histories ( s 0 , a 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t ) (s_0,a_0,s_1,a_1,\ldots,s_{t-1},a_{t-1},s_t) ( s 0 , a 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t ) to mixed actions in Δ ( A ) \Delta(\mathcal{A}) Δ ( A ) ; in particular, the realized rewards r ( s τ , a τ , s τ + 1 ) r(s_\tau,a_\tau,s_{\tau+1}) r ( s τ , a τ , s τ + 1 ) for τ ≤ t − 1 \tau\le t-1 τ ≤ t − 1 are not part of the histories, and so in principle history-dependent policies are not allowed to observe the rewards gathered so far at each period and adjust their decisions based on them. This is somewhat similar to robust MDPs with uncertain transitions, where we did not assume that the decision maker can observe the value of P ( s τ , a τ , s τ + 1 ) P(s_\tau,a_\tau,s_{\tau+1}) P ( s τ , a τ , s τ + 1 ) for τ ≤ t − 1 \tau\le t-1 τ ≤ t − 1 along the trajectories ( s 0 , a 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t ) (s_0,a_0,s_1,a_1,\ldots,s_{t-1},a_{t-1},s_t) ( s 0 , a 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t ) .
We can still obtain some results for the case where the decision maker is allowed to observe the instantaneous rewards at each period (in addition to the sequence of state-action pairs visited along the trajectory); let’s call this class of policies reward-augmented history-dependent policies Π H r a \Pi_{\mathrm{H}}^{\mathrm{ra}} Π H ra . Indeed, in this model the set of stationary policies Π S \Pi_{\mathrm{S}} Π S , which only depend on the current state, is still identifiable with Π S = × s ∈ S Δ ( A ) \Pi_{\mathrm{S}} = \times_{s\in\mathcal{S}}\Delta(\mathcal{A}) Π S = × s ∈ S Δ ( A ) , i.e., the sets of stationary policies coincide in the classical setup (with history-dependent policies as defined in Chapter 1) and in the setup where the decision maker can observe the realized rewards. Therefore, for R \mathcal{R} R convex compact, strong duality as proved in Proposition 4.2 would still hold, and we have seen at the beginning of Section 3.3.2 that strong duality implies the existence of stationary optimal policies. Hence we conclude that for R \mathcal{R} R convex compact, an optimal solution to sup π ∈ Π H r a inf r ∈ R R ( π , r ) \sup_{\pi\in\Pi_{\mathrm{H}}^{\mathrm{ra}}}\inf_{r\in\mathcal{R}} R(\pi,r) sup π ∈ Π H ra inf r ∈ R R ( π , r ) , with Π H r a \Pi_{\mathrm{H}}^{\mathrm{ra}} Π H ra the set of reward-augmented history-dependent policies, can be chosen stationary:
sup π ∈ Π H r a inf r ∈ R R ( π , r ) = max π ∈ Π S min r ∈ R R ( π , r ) \sup_{\pi\in\Pi_{\mathrm{H}}^{\mathrm{ra}}}\inf_{r\in\mathcal{R}} R(\pi,r)=\max_{\pi\in\Pi_{\mathrm{S}}}\min_{r\in\mathcal{R}} R(\pi,r) π ∈ Π H ra sup r ∈ R inf R ( π , r ) = π ∈ Π S max r ∈ R min R ( π , r )
and a stationary optimal policy can be computed via the methods described right above. If R \mathcal{R} R is not convex, then we may need non-stationary reward-augmented policies, as argued in the previous paragraph and exemplified in Example 4.1.
Robust MDPs with uncertain rewards and uncertain transitions
In this section, we treat the case where both the rewards r ∈ R S × A × S r \in\mathbb{R}^{S \times A \times S} r ∈ R S × A × S and the transitions P ∈ Δ ( S ) S × A P \in\Delta(S)^{S \times A} P ∈ Δ ( S ) S × A are uncertain. In this case, we are given an uncertainty set U ⊂ R S × A × S × Δ ( S ) S × A \mathcal{U} \subset\mathbb{R}^{S \times A \times S} \times\Delta(S)^{S \times A} U ⊂ R S × A × S × Δ ( S ) S × A , which models the set of realizable pairs of rewards-transitions ( r , P ) ∈ R S × A × S × Δ ( S ) S × A (r,P) \in\mathbb{R}^{S \times A \times S} \times\Delta(S)^{S \times A} ( r , P ) ∈ R S × A × S × Δ ( S ) S × A . Note that in principle there may be joint constraints between r r r and P P P in U \mathcal{U} U , i.e., we do not require U \mathcal{U} U to be a Cartesian product R × P \mathcal{R} \times\mathcal{P} R × P with R ⊂ R S × A × S \mathcal{R} \subset\mathbb{R}^{S \times A \times S} R ⊂ R S × A × S and P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(S)^{S \times A} P ⊂ Δ ( S ) S × A .
We extend the return to R ( π , r , P ) R(\pi,r,P) R ( π , r , P ) with ( r , P ) ∈ U (r,P) \in\mathcal{U} ( r , P ) ∈ U , and the goal in this section is to solve
sup π ∈ Π H inf ( r , P ) ∈ U R ( π , r , P ) (56) \sup_{\pi\in\Pi_{\mathrm{H}}} \inf_{(r,P) \in\mathcal{U}} R(\pi,r,P) \tag*{(56)} π ∈ Π H sup ( r , P ) ∈ U inf R ( π , r , P ) (56)
The first thing to notice is that given ( r , P ) ∈ U (r,P) \in\mathcal{U} ( r , P ) ∈ U we can replace r ∈ R S × A × S r \in\mathbb{R}^{S \times A \times S} r ∈ R S × A × S by r ^ P , r ∈ R S × A × S \hat{r}_{P,r} \in\mathbb{R}^{S \times A \times S} r ^ P , r ∈ R S × A × S the expected one-step reward as follows.
Lemma 4.1. Let π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H , ( r , P ) ∈ U (r,P) \in\mathcal{U} ( r , P ) ∈ U . Then R ( π , r , P ) = R ( π , r ^ P , r , P ) R(\pi,r,P) = R(\pi,\hat{r}_{P,r},P) R ( π , r , P ) = R ( π , r ^ P , r , P ) with r ^ P , r ∈ R S × A × S \hat{r}_{P,r} \in\mathbb{R}^{S \times A \times S} r ^ P , r ∈ R S × A × S defined as
r ^ P , r ( s , a , s ′ ) = P ( s , a ) T r ( s , a ) , ∀ s , a , s ′ . \hat{r}_{P,r}(s,a,s') = P(s,a)^{\mathsf{T}}r(s,a), \ \forall\ s,a,s'. r ^ P , r ( s , a , s ′ ) = P ( s , a ) T r ( s , a ) , ∀ s , a , s ′ .
Proof. We already did such a calculation in Section 1.1 when dealing with state-action frequencies. By definition
R ( π , r , P ) = E π , P [ ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) ] = ∑ t = 0 + ∞ γ t E π , P [ r ( s t , a t , s t + 1 ) ] . R(\pi,r,P) = \mathbb{E}^{\pi,P}\left[\sum_{t=0}^{+\infty}\gamma^t r(s_t,a_t,s_{t+1})\right] = \sum_{t=0}^{+\infty}\gamma^t\mathbb{E}^{\pi,P}[r(s_t,a_t,s_{t+1})]. R ( π , r , P ) = E π , P [ t = 0 ∑ + ∞ γ t r ( s t , a t , s t + 1 ) ] = t = 0 ∑ + ∞ γ t E π , P [ r ( s t , a t , s t + 1 )] .
and E π , P [ r ( s t , a t , s t + 1 ) ] = E π , P [ P ( s t , a t ) T r ( s t , a t ) ] = E π , P [ r ^ ( s t , a t , s t + 1 ) ] \mathbb{E}^{\pi,P}[r(s_t,a_t,s_{t+1})] = \mathbb{E}^{\pi,P}\left[P(s_t,a_t)^{\mathsf{T}}r(s_t,a_t)\right] = \mathbb{E}^{\pi,P}[\hat{r}(s_t,a_t,s_{t+1})] E π , P [ r ( s t , a t , s t + 1 )] = E π , P [ P ( s t , a t ) T r ( s t , a t ) ] = E π , P [ r ^ ( s t , a t , s t + 1 )] .
Note r ^ ( s , a , s ′ ) \hat{r}(s,a,s') r ^ ( s , a , s ′ ) is independent of the next state s ′ s' s ′ , so we could have identified r ^ \hat{r} r ^ with an element of R S × A \mathbb{R}^{S \times A} R S × A instead of R S × A × S \mathbb{R}^{S \times A \times S} R S × A × S . We still write r ^ ∈ R S × A × S \hat{r} \in\mathbb{R}^{S \times A \times S} r ^ ∈ R S × A × S for consistency (given it is an input of the return function) but we will allow ourselves to write r ^ ( s , a ) \hat{r}(s,a) r ^ ( s , a ) instead of r ^ ( s , a , s ′ ) \hat{r}(s,a,s') r ^ ( s , a , s ′ ) later in this section.
Given an uncertainty set U ⊂ R S × A × S × Δ ( S ) S × A \mathcal{U} \subset\mathbb{R}^{S \times A \times S} \times\Delta(S)^{S \times A} U ⊂ R S × A × S × Δ ( S ) S × A we can therefore construct an effective uncertainty set U f \mathcal{U}^{f} U f as follows:
U f = { ( r ^ , P ) ∣ ∃ ( r , P ) ∈ U , r ^ ( s , a , s ′ ) = P ( s , a ) T r ( s , a ) , ∀ s , a , s ′ } . (57) \mathcal{U}^{f} = \{(\hat{r},P) \mid\exists(r,P) \in\mathcal{U},\ \hat{r}(s,a,s') = P(s,a)^{\mathsf{T}}r(s,a),\ \forall\ s,a,s'\}. \tag*{(57)} U f = {( r ^ , P ) ∣ ∃ ( r , P ) ∈ U , r ^ ( s , a , s ′ ) = P ( s , a ) T r ( s , a ) , ∀ s , a , s ′ } . (57)
Note that the first component of U f \mathcal{U}^{f} U f has a bilinear expression in the original component ( r , P ) (r,P) ( r , P ) in U \mathcal{U} U . What properties of U \mathcal{U} U may the effective uncertainty set U f \mathcal{U}^{f} U f inherit?
Clearly if U \mathcal{U} U is compact then U f \mathcal{U}^{f} U f is compact.
If U \mathcal{U} U is s-rectangular: U = × s ∈ S U s \mathcal{U} = \times_{s \in S}\mathcal{U}_s U = × s ∈ S U s with U s ⊂ { ( r ( s ) , P ( s ) ) ∣ r ( s ) ∈ R A × S , P ( s ) ∈ Δ ( S ) A } \mathcal{U}_s \subset\{(r(s),P(s)) \mid r(s) \in\mathbb{R}^{A \times S}, P(s) \in\Delta(S)^A\} U s ⊂ {( r ( s ) , P ( s )) ∣ r ( s ) ∈ R A × S , P ( s ) ∈ Δ ( S ) A } then U f \mathcal{U}^{f} U f is s-rectangular: U f = × s ∈ S U s f \mathcal{U}^{f} = \times_{s \in S}\mathcal{U}^{f}_s U f = × s ∈ S U s f with U s f = { ( r ^ ( s ) , P ( s ) ) ∣ ∃ ( r ( s ) , P ( s ) ) ∈ U s , r ^ ( s , a , s ′ ) = P ( s , a ) T r ( s , a ) , ∀ a , s ′ } \mathcal{U}^{f}_s = \{(\hat{r}(s),P(s)) \mid\exists(r(s),P(s)) \in\mathcal{U}_s,\ \hat{r}(s,a,s') = P(s,a)^{\mathsf{T}}r(s,a),\ \forall\ a,s'\} U s f = {( r ^ ( s ) , P ( s )) ∣ ∃ ( r ( s ) , P ( s )) ∈ U s , r ^ ( s , a , s ′ ) = P ( s , a ) T r ( s , a ) , ∀ a , s ′ } .
The main issue is regarding convexity. If U \mathcal{U} U is convex, it may be the case that U f \mathcal{U}^{f} U f is not convex, because of the bilinearity of r ^ \hat{r} r ^ , the reward-component of U f \mathcal{U}^{f} U f , in ( r , P ) ∈ U (r, P) \in\mathcal{U} ( r , P ) ∈ U . We give an example where U \mathcal{U} U is convex but U f \mathcal{U}^{f} U f is not convex below.
Example 4.3. Assume we are given a state-action pair ( s 0 , a 0 ) (s_0, a_0) ( s 0 , a 0 ) and the next possible states are s 0 , s 1 s_0, s_1 s 0 , s 1 . Let ( r 0 , P 0 ) ∈ U (r^0, P^0) \in\mathcal{U} ( r 0 , P 0 ) ∈ U such that r 0 ( s 0 , a 0 , s 0 ) = r 0 ( s 0 , a 0 , s 1 ) = 0 r^0(s_0, a_0, s_0) = r^0(s_0, a_0, s_1) = 0 r 0 ( s 0 , a 0 , s 0 ) = r 0 ( s 0 , a 0 , s 1 ) = 0 , P 0 ( s 0 , a 0 , s 0 ) = 1 P^0(s_0, a_0, s_0) = 1 P 0 ( s 0 , a 0 , s 0 ) = 1 , and let ( r 1 , P 1 ) ∈ U (r^1, P^1) \in\mathcal{U} ( r 1 , P 1 ) ∈ U such that r 1 ( s 0 , a 0 , s 0 ) = 0 , r 1 ( s 0 , a 0 , s 1 ) = 1 , P 1 ( s 0 , a 0 , s 1 ) = 1 r^1(s_0, a_0, s_0) = 0, r^1(s_0, a_0, s_1) = 1, P^1(s_0, a_0, s_1) = 1 r 1 ( s 0 , a 0 , s 0 ) = 0 , r 1 ( s 0 , a 0 , s 1 ) = 1 , P 1 ( s 0 , a 0 , s 1 ) = 1 . Let λ ∈ [ 0 , 1 ] \lambda\in[0, 1] λ ∈ [ 0 , 1 ] and ( r λ , P λ ) = ( 1 − λ ) ( r 0 , P 0 ) + λ ( r 1 , P 1 ) (r^\lambda, P^\lambda) = (1-\lambda)(r^0, P^0) + \lambda(r^1, P^1) ( r λ , P λ ) = ( 1 − λ ) ( r 0 , P 0 ) + λ ( r 1 , P 1 ) . The expected instantaneous reward r ^ λ ( s 0 , a 0 , s ′ ) \hat{r}^\lambda(s_0, a_0, s') r ^ λ ( s 0 , a 0 , s ′ ) is r ^ λ ( s 0 , a 0 , s ′ ) = P λ ( s 0 , a 0 ) T r λ ( s 0 , a 0 ) = λ 2 \hat{r}^\lambda(s_0, a_0, s') = P^\lambda(s_0, a_0)^\mathsf{T}r^\lambda(s_0, a_0) = \lambda^2 r ^ λ ( s 0 , a 0 , s ′ ) = P λ ( s 0 , a 0 ) T r λ ( s 0 , a 0 ) = λ 2 . Hence the ( s 0 , a 0 ) (s_0, a_0) ( s 0 , a 0 ) -th component of U f \mathcal{U}^{f} U f is { ( λ 2 , λ ) ∣ λ ∈ [ 0 , 1 ] } \{(\lambda^2, \lambda) \mid\lambda\in[0, 1]\} {( λ 2 , λ ) ∣ λ ∈ [ 0 , 1 ]} , which is not a convex set (e.g. the midpoint between ( 0 , 0 ) (0, 0) ( 0 , 0 ) and ( 1 , 1 ) (1, 1) ( 1 , 1 ) is not feasible).
Note that if U f \mathcal{U}^{f} U f is convex compact and s-rectangular, everything follows as in Chapter 3. For ( r ^ , P ) ∈ U f (\hat{r}, P) \in\mathcal{U}^{f} ( r ^ , P ) ∈ U f , π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S we define the operators T f π , r ^ , P , T f π , T f : R S → R S T_f^{\pi,\hat{r},P}, T_f^\pi, T_f : \mathbb{R}^{\mathcal{S}} \to\mathbb{R}^{\mathcal{S}} T f π , r ^ , P , T f π , T f : R S → R S as usual:
T f π , r ^ , P v ( s ) = ∑ a ∈ A π ( s , a ) ( r ^ ( s , a ) + γ P ( s , a ) T v ) T_f^{\pi,\hat{r},P}v(s) = \sum_{a\in\mathcal{A}} \pi(s,a)\left(\hat{r}(s,a) + \gamma P(s,a)^\mathsf{T}v\right) T f π , r ^ , P v ( s ) = a ∈ A ∑ π ( s , a ) ( r ^ ( s , a ) + γ P ( s , a ) T v )
T f π v ( s ) = min ( r ^ , P ) ∈ U f T f π , r ^ , P v ( s ) T_f^\pi v(s) = \min_{(\hat{r},P)\in\mathcal{U}^{f}} T_f^{\pi,\hat{r},P}v(s) T f π v ( s ) = ( r ^ , P ) ∈ U f min T f π , r ^ , P v ( s )
T f v ( s ) = max π ∈ Π S min ( r ^ , P ) ∈ U f T f π , r ^ , P v ( s ) T_f v(s) = \max_{\pi\in\Pi_{\mathcal{S}}}\min_{(\hat{r},P)\in\mathcal{U}^{f}} T_f^{\pi,\hat{r},P}v(s) T f v ( s ) = π ∈ Π S max ( r ^ , P ) ∈ U f min T f π , r ^ , P v ( s )
We of course get that T f π , r ^ , P , T f π , T f T_f^{\pi,\hat{r},P}, T_f^\pi, T_f T f π , r ^ , P , T f π , T f are monotone contractions, and we write v ⋆ ∈ R S v^\star\in\mathbb{R}^{\mathcal{S}} v ⋆ ∈ R S for the fixed point of T f T_f T f .
Theorem 4.1. Let U ⊂ R S × A × S × Δ ( S ) S × A \mathcal{U} \subset\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} \times\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} U ⊂ R S × A × S × Δ ( S ) S × A be an uncertainty set for rewards-transition pairs. Assume that U f \mathcal{U}^{f} U f is compact, convex and s-rectangular, with U f \mathcal{U}^{f} U f defined as in (57) . The following statements hold:
Strong duality. The following strong duality holds:
max π ∈ Π S min ( r ^ , P ) ∈ U f R ( π , r ^ , P ) = min ( r ^ , P ) ∈ U f max π ∈ Π S R ( π , r ^ , P ) . (58) \max_{\pi\in\Pi_{\mathcal{S}}}\min_{(\hat{r},P)\in\mathcal{U}^{f}} R(\pi,\hat{r},P) = \min_{(\hat{r},P)\in\mathcal{U}^{f}}\max_{\pi\in\Pi_{\mathcal{S}}} R(\pi,\hat{r},P). \tag*{(58)} π ∈ Π S max ( r ^ , P ) ∈ U f min R ( π , r ^ , P ) = ( r ^ , P ) ∈ U f min π ∈ Π S max R ( π , r ^ , P ) . (58) Optimality of stationary policies. We have
sup π ∈ Π H min ( r ^ , P ) ∈ U f R ( π , r ^ , P ) = max π ∈ Π S min ( r ^ , P ) ∈ U f R ( π , r ^ , P ) . \sup_{\pi\in\Pi_{\mathcal{H}}}\min_{(\hat{r},P)\in\mathcal{U}^{f}} R(\pi,\hat{r},P) = \max_{\pi\in\Pi_{\mathcal{S}}}\min_{(\hat{r},P)\in\mathcal{U}^{f}} R(\pi,\hat{r},P). π ∈ Π H sup ( r ^ , P ) ∈ U f min R ( π , r ^ , P ) = π ∈ Π S max ( r ^ , P ) ∈ U f min R ( π , r ^ , P ) . Optimality conditions. π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S is a stationary uniformly optimal policy for (56) if and only if T f π v ⋆ = T f v ⋆ T_f^\pi v^\star= T_fv^\star T f π v ⋆ = T f v ⋆
The proof follows exactly the same lines as the proof of Theorem 3.2; we can still apply Sion’s minimax theorem to show that in the definition of T f T_f T f we can turn the max min into a min max, and this is where we require convexity of U f \mathcal{U}^{f} U f . It is worth emphasizing that we do require s-rectangularity of R \mathcal{R} R (the reward-component induced by U f \mathcal{U}^{f} U f ) for Theorem 4.1, whereas in Section 4.1, where P P P was fixed and the rewards were unknown, we required R \mathcal{R} R to be convex and compact. This comes from the difference in solution methods between robust MDPs with uncertain rewards (which we solved in § 4.1 based on state-action frequencies without requiring rectangularity) and robust MDPs with uncertain transitions (which we solved in Chapter § 1.3 based on value functions, which are naturally componentwise objects and therefore require rectangularity to define proper worst-case value functions). To reconcile the two models and solve robust MDPs with uncertain rewards and uncertain transitions, we need stronger assumptions and therefore we assume s-rectangularity of U \mathcal{U} U .
When U \mathcal{U} U is a Cartesian product of sa-rectangular sets, we do not need convexity (similar to the case of sa-rectangular uncertainty set in § 3.4 ), that is, if U = R × P \mathcal{U}=\mathcal{R}\times\mathcal{P} U = R × P with R = × s , a R s a , P = × s , a P s a \mathcal{R}=\times_{s,a}\mathcal{R}_{sa},\mathcal{P}=\times_{s,a}\mathcal{P}_{sa} R = × s , a R s a , P = × s , a P s a with R s a ⊂ R S , P s a ⊂ Δ ( S ) \mathcal{R}_{sa}\subset\mathbb{R}^{S},\mathcal{P}_{sa}\subset\Delta(S) R s a ⊂ R S , P s a ⊂ Δ ( S ) compact sets then the conclusions of Theorem 4.1 hold and additionally we get that an optimal policy can be chosen stationary and deterministic.
It remains to answer a central question: under what conditions is the effective uncertainty set U f \mathcal{U}^{f} U f convex?
When is U f \mathcal{U}^{f} U f convex? We have observed above that s-rectangularity and compactness of U \mathcal{U} U naturally imply s-rectangularity and compactness of U f \mathcal{U}^{f} U f . However obtaining the convexity of U f \mathcal{U}^{f} U f is trickier. An interesting case is when the rewards are independent of the next state: If indeed we have that for any ( r , P ) ∈ U (r,P)\in\mathcal{U} ( r , P ) ∈ U then r ( s , a , s ′ ) = r ( s , a , s ′ ′ ) r(s,a,s')=r(s,a,s'') r ( s , a , s ′ ) = r ( s , a , s ′′ ) , ∀ s , s ′ , s ′ ′ ∈ S , a ∈ A \forall\ s,s',s''\in S,a\in A ∀ s , s ′ , s ′′ ∈ S , a ∈ A , we then get that P ( s , a ) ⊤ r ( s , a ) = r ( s , a , s ′ ) P(s,a)^{\top}r(s,a)=r(s,a,s') P ( s , a ) ⊤ r ( s , a ) = r ( s , a , s ′ ) (where we can choose any s ′ s' s ′ in the expression above), so that U f = U \mathcal{U}^{f}=\mathcal{U} U f = U , so that U f \mathcal{U}^{f} U f is convex when U \mathcal{U} U is convex. To the best of our knowledge, most papers that focus on uncertain rewards and transition probabilities make this assumption, yet this is not without loss of generality, as we have seen repeatedly in these notes. We conclude this section with the following hardness result, which shows that policy evaluation is difficult when both ( r , P ) (r,P) ( r , P ) are uncertain.
Proposition 4.3. Let π ∈ Π S D \pi\in\Pi_{\mathrm{SD}} π ∈ Π SD . For a general convex compact s-rectangular polytope U ⊂ R S × A × S × Δ ( S ) S × A \mathcal{U}\subset\mathbb{R}^{S\times A\times S}\times\Delta(S)^{S\times A} U ⊂ R S × A × S × Δ ( S ) S × A , it is in general N P NP N P -hard to decide inf ( r , P ) ∈ U R ( π , r , P ) ≤ c \inf_{(r,P)\in\mathcal{U}}R(\pi,r,P)\le c inf ( r , P ) ∈ U R ( π , r , P ) ≤ c .
Proof. The proof is based on a reduction from the Clique problem. Consider an undirected graph G = ( V , E ) G=(V,E) G = ( V , E ) with V = { 1 , … , n } V=\{1,\ldots,n\} V = { 1 , … , n } and A ∈ { 0 , 1 } n × n A\in\{0,1\}^{n\times n} A ∈ { 0 , 1 } n × n its adjacency matrix. A clique is a subset of vertices all adjacent to each other. The size of a clique is its number of vertices. For some k ∈ N k\in\mathbb{N} k ∈ N , the Clique problem asks to decide whether there is a clique of size at least k k k or not.
We construct a reduction from the Clique problem to deciding inf ( r , P ) ∈ U R ( π , r , P ) ≤ c \inf_{(r,P)\in\mathcal{U}}R(\pi,r,P)\le c inf ( r , P ) ∈ U R ( π , r , P ) ≤ c as follows. We consider an MDP instance where S = V ∪ { s 0 } S=V\cup\{s_{0}\} S = V ∪ { s 0 } and there is one action a a a (irrelevant here). The initial state is s 0 s_{0} s 0 , every state j ∈ V j\in V j ∈ V is absorbing with reward 0 0 0 . The only uncertain probabilities and rewards are out of state s 0 s_{0} s 0 , and the set of feasible ( r ( s 0 ) , P ( s 0 ) ) ∈ R A × S × Δ ( S ) A (r(s_{0}),P(s_{0}))\in\mathbb{R}^{A\times S}\times\Delta(S)^{A} ( r ( s 0 ) , P ( s 0 )) ∈ R A × S × Δ ( S ) A is given by
U s 0 = { ( r ( s 0 ) , P ( s 0 ) ) ∣ x ∈ Δ ( V ) , ( P ( s 0 , a , j ) ) j ∈ V = x , ( r ( s 0 , a , j ) ) j ∈ V = − A x , r ( s 0 , a , s 0 ) = 0 } . \mathcal{U}_{s_{0}}=\{(r(s_{0}),P(s_{0}))\mid x\in\Delta(V),\ (P(s_{0},a,j))_{j\in V}=x,\ (r(s_{0},a,j))_{j\in V}=-Ax,\ r(s_{0},a,s_{0})=0\}. U s 0 = {( r ( s 0 ) , P ( s 0 )) ∣ x ∈ Δ ( V ) , ( P ( s 0 , a , j ) ) j ∈ V = x , ( r ( s 0 , a , j ) ) j ∈ V = − A x , r ( s 0 , a , s 0 ) = 0 } .
Clearly U \mathcal{U} U is s-rectangular since the only state with some uncertainty is s 0 s_{0} s 0 . Moreover U s 0 \mathcal{U}_{s_{0}} U s 0 is an affine image of the polytope Δ ( V ) \Delta(V) Δ ( V ) hence it is compact, convex and it is a polytope. We can take γ = 0 \gamma=0 γ = 0 (it does not play any role since the MDP goes to an absorbing state right after period t = 0 t=0 t = 0 ). The return R ( π , r , P ) R(\pi,r,P) R ( π , r , P ) is equal to R ( π , r , P ) = − x ⊤ A x R(\pi,r,P)=-x^{\top}Ax R ( π , r , P ) = − x ⊤ A x for some x ∈ Δ ( V ) x\in\Delta(V) x ∈ Δ ( V ) . Now Motzkin-Straus theorem states that given a graph G G G with adjacency matrix A ∈ { 0 , 1 } n × n A \in\{0,1\}^{n \times n} A ∈ { 0 , 1 } n × n then max x ∈ Δ ( V ) x T A x = 1 − 1 ω ( G ) \max_{x \in\Delta(V)} x^{\mathsf{T}} A x = 1 - \frac{1}{\omega(G)} max x ∈ Δ ( V ) x T A x = 1 − ω ( G ) 1 with ω ( G ) \omega(G) ω ( G ) the clique number of G G G , i.e., the maximum size of a clique in G G G . Hence
inf ( r , P ) ∈ U R ( π , r , P ) = − 1 + 1 ω ( G ) \inf_{(r,P)\in\mathcal{U}} R(\pi,r,P)=-1+\frac{1}{\omega(G)} ( r , P ) ∈ U inf R ( π , r , P ) = − 1 + ω ( G ) 1
and if we could compare inf ( r , P ) ∈ U R ( π , r , P ) \inf_{(r,P)\in\mathcal{U}} R(\pi,r,P) inf ( r , P ) ∈ U R ( π , r , P ) to − 1 + 1 / k -1+1/k − 1 + 1/ k we could compare ω ( G ) \omega(G) ω ( G ) to some k k k , which concludes our reduction.
Note that in the reduction for the proof of the above proposition we get that the coordinate s 0 s_0 s 0 of U f \mathcal{U}^f U f can be identified with { ( − x T A x , x ) ∣ x ∈ Δ ( V ) } \{(-x^{\mathsf{T}}Ax,x)\mid x\in\Delta(V)\} {( − x T A x , x ) ∣ x ∈ Δ ( V )} which need not be convex, similarly as in Example 4.3. This hardness result shows that non-convexity of U f \mathcal{U}^f U f can be the main difficulty when studying robust MDPs with uncertain rewards and uncertain transitions, and general reward models.
Multi-objective MDPs and Pareto optimality
This section focuses on multi-objective MDPs , where we are given several different MDP models M 1 , … , M N \mathcal{M}_1,\ldots,\mathcal{M}_N M 1 , … , M N with common state and action sets but (potentially) different instantaneous rewards and transition probabilities M i = ( S , A , r i , P i , p 0 ) \mathcal{M}_i=(\mathcal{S},\mathcal{A},r^i,P^i,p_0) M i = ( S , A , r i , P i , p 0 ) with r i ∈ R S × A × S r^i\in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} r i ∈ R S × A × S , P i ∈ Δ ( S ) S × A P^i\in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P i ∈ Δ ( S ) S × A for i ∈ { 1 , … , N } i\in\{1,\ldots,N\} i ∈ { 1 , … , N } . We have some initial distribution p 0 p_0 p 0 , common for each i ∈ { 1 , … , N } i\in\{1,\ldots,N\} i ∈ { 1 , … , N } . Since the sets of states and actions are common across all MDP models here, it makes sense to talk about a policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H ; however, it is unclear which objective we should pursue now. In particular a policy may perform well for some MDP model M i \mathcal{M}_i M i but poorly under some other model M j \mathcal{M}_j M j , for some j ≠ i j\ne i j = i . We will write R ( π , M i ) ∈ R R(\pi,\mathcal{M}_i)\in\mathbb{R} R ( π , M i ) ∈ R for the discounted return of π \pi π in the i i i -th MDP model:
R ( π , M i ) = E π , P i [ ∑ t = 0 + ∞ γ t r i ( s t , a t , s t + 1 ) ] . R(\pi,\mathcal{M}_i)=\mathbb{E}^{\pi,P^i}\left[\sum_{t=0}^{+\infty}\gamma^t r^i(s_t,a_t,s_{t+1})\right]. R ( π , M i ) = E π , P i [ t = 0 ∑ + ∞ γ t r i ( s t , a t , s t + 1 ) ] .
We will write R ( π ) ∈ R N R(\pi)\in\mathbb{R}^N R ( π ) ∈ R N for the vector of returns of a policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H in each model M i \mathcal{M}_i M i :
R ( π ) = ( R ( π , M i ) ) i ≤ N . R(\pi)=\left(R(\pi,\mathcal{M}_i)\right)_{i\le N}. R ( π ) = ( R ( π , M i ) ) i ≤ N .
Naturally one arrives at the following definition of Pareto optimality .
Definition 4.1 (Pareto domination and Pareto optimality). Let π , π ′ ∈ Π H \pi,\pi'\in\Pi_{\mathrm{H}} π , π ′ ∈ Π H and let Π ⊂ Π H \Pi\subset\Pi_{\mathrm{H}} Π ⊂ Π H be a class of policies (e.g., Π S \Pi_{\mathrm{S}} Π S or Π M \Pi_{\mathrm{M}} Π M ).
We say that π \pi π Pareto dominates π ′ \pi' π ′ if R ( π ) ≥ R ( π ′ ) R(\pi)\ge R(\pi') R ( π ) ≥ R ( π ′ ) (componentwise) and R ( π , M ) > R ( π ′ , M ) R(\pi,\mathcal{M})>R(\pi',\mathcal{M}) R ( π , M ) > R ( π ′ , M ) for at least one model M ∈ { M 1 , … , M N } \mathcal{M}\in\{\mathcal{M}_1,\ldots,\mathcal{M}_N\} M ∈ { M 1 , … , M N } .
We say that π \pi π strictly Pareto dominates π ′ \pi' π ′ if R ( π , M ) > R ( π ′ , M ) R(\pi,\mathcal{M})>R(\pi',\mathcal{M}) R ( π , M ) > R ( π ′ , M ) for any MDP model M ∈ { M 1 , … , M N } \mathcal{M}\in\{\mathcal{M}_1,\ldots,\mathcal{M}_N\} M ∈ { M 1 , … , M N } .
When π ∈ Π \pi\in\Pi π ∈ Π , we say that π \pi π is Pareto optimal in Π \Pi Π if no policy in Π \Pi Π Pareto dominates it, and weakly Pareto optimal if no policy in Π \Pi Π strictly Pareto dominates it.
With a slight abuse of terminology we will also say that “R R R Pareto dominates R ′ R' R ′ ” for R , R ′ ∈ R N R, R' \in\mathbb{R}^{N} R , R ′ ∈ R N when R ≥ R ′ R \geq R' R ≥ R ′ componentwise and R ≠ R ′ R \ne R' R = R ′ . We also consider two sets of interest induced by a particular subclass of policies (think of the set of Markovian, or stationary, or stationary deterministic policies).
Definition 4.2 (Pareto front and achievable set). Let Π ⊂ Π H \Pi\subset\Pi_{\mathrm{H}} Π ⊂ Π H be a class of policies.
The achievable set R ( Π ) R(\Pi) R ( Π ) is the set of vectors R ( π ) ∈ R N R(\pi) \in\mathbb{R}^{N} R ( π ) ∈ R N induced by some π ∈ Π \pi\in\Pi π ∈ Π .
The Pareto front F ( Π ) \mathcal{F}(\Pi) F ( Π ) is the set of all vectors R ( π ) ∈ R N R(\pi) \in\mathbb{R}^{N} R ( π ) ∈ R N induced by policies π ∈ Π \pi\in\Pi π ∈ Π that are Pareto optimal in Π \Pi Π .
Note that we always have, by definition, R ( Π ) ⊆ R ( Π H ) R(\Pi) \subseteq R(\Pi_{\mathrm{H}}) R ( Π ) ⊆ R ( Π H ) when Π ⊂ Π H \Pi\subset\Pi_{\mathrm{H}} Π ⊂ Π H . One of the main questions in the literature on multi-objective MDPs is to compute Pareto optimal policies, and even better, when it is feasible, to enumerate the set of all Pareto optimal policies. Of course the definitions above are very general and can be extended to the case of MDPs with finite horizon, to MDPs with average (mean-payoff) return (see § 5.1 ), or to MDPs with risk-averse returns (see § 5.5 ). Here, we focus on the discounted return, as this simplifies the exposition of the connection to the robust MDP model given above. We analyze a simple example below.
Example 4.4. Consider the MDP instance from Figure 2.1 with two MDP models that differ only in their transitions. We choose γ = 1 / 2 \gamma= 1/2 γ = 1/2 , we start in s 0 s_{0} s 0 with probability 1 1 1 , and p = 1 p = 1 p = 1 in M 1 \mathcal{M}_{1} M 1 and p = 0 p = 0 p = 0 in M 2 \mathcal{M}_{2} M 2 (i.e., the effects of the actions a 1 a_{1} a 1 and a 2 a_{2} a 2 are reversed in M 1 \mathcal{M}_{1} M 1 and M 2 \mathcal{M}_{2} M 2 ). We only change the reward r ( s 0 , a 2 , s 1 ) r(s_{0}, a_{2}, s_{1}) r ( s 0 , a 2 , s 1 ) from 1 1 1 (in Figure 2.1 ) to r ( s 0 , a 2 , s 1 ) = 3 r(s_{0}, a_{2}, s_{1}) = 3 r ( s 0 , a 2 , s 1 ) = 3 . Let π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S be a stationary policy and x = π ( s 0 , a 1 ) x = \pi(s_{0}, a_{1}) x = π ( s 0 , a 1 ) . We write v π i ( s 0 ) v_{\pi}^{i}(s_{0}) v π i ( s 0 ) for the value function derived by policy π \pi π starting in s 0 s_{0} s 0 in the MDP model M i \mathcal{M}_{i} M i . The Bellman equations give
v π 1 ( s 0 ) = x + 1 2 ( 1 − x ) v π 1 ( s 0 ) , v π 2 ( s 0 ) = 4 ( 1 − x ) + 1 2 x v π 2 ( s 0 ) . v_{\pi}^{1}(s_{0}) = x + \frac{1}{2}(1-x)v_{\pi}^{1}(s_{0}), \qquad v_{\pi}^{2}(s_{0}) = 4(1-x) + \frac{1}{2}xv_{\pi}^{2}(s_{0}). v π 1 ( s 0 ) = x + 2 1 ( 1 − x ) v π 1 ( s 0 ) , v π 2 ( s 0 ) = 4 ( 1 − x ) + 2 1 x v π 2 ( s 0 ) . Therefore in this example
R ( Π S ) = { ( 2 x 1 + x , 8 ( 1 − x ) 2 − x ) ∣ x ∈ [ 0 , 1 ] } . R(\Pi_{\mathrm{S}}) = \left\{\left(\frac{2x}{1+x}, \frac{8(1-x)}{2-x}\right) \mid x \in[0,1]\right\}. R ( Π S ) = { ( 1 + x 2 x , 2 − x 8 ( 1 − x ) ) ∣ x ∈ [ 0 , 1 ] } . *Note that the coordinates are respectively increasing and decreasing, hence R ( Π S ) = F ( Π S ) R(\Pi_{\mathrm{S}}) = \mathcal{F}(\Pi_{\mathrm{S}}) R ( Π S ) = F ( Π S ) . It is easy to check that this set is not convex, see Figure 4.1 .
We now focus on describing R ( Π H ) R(\Pi_{\mathrm{H}}) R ( Π H ) . Given π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H we write x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] for the probability that π \pi π chooses a 1 a_{1} a 1 at period t = 0 t = 0 t = 0 , and we now bound the return R ( π , M ) R(\pi,\mathcal{M}) R ( π , M ) for M ∈ { M 1 , M 2 } \mathcal{M} \in\{\mathcal{M}_{1},\mathcal{M}_{2}\} M ∈ { M 1 , M 2 } . In the MDP model M 1 \mathcal{M}_{1} M 1 , with probability x x x we obtain a return of 1 1 1 (i.e. choosing a 1 a_{1} a 1 at period 0 0 0 ) and with probability 1 − x 1-x 1 − x we stay in s 0 s_{0} s 0 , and from then on we can obtain at most a return of 1 / 2 1/2 1/2 (by choosing a 1 a_{1} a 1 ). Hence x ≤ R ( π , M 1 ) ≤ x + 1 − x 2 = 1 + x 2 x \leq R(\pi,\mathcal{M}_{1}) \leq x + \frac{1-x}{2} = \frac{1+x}{2} x ≤ R ( π , M 1 ) ≤ x + 2 1 − x = 2 1 + x . Similarly, in the MDP model M 2 \mathcal{M}_{2} M 2 , if we choose a 2 a_{2} a 2 at period 0 0 0 then the return is 4 4 4 , otherwise we stay in s 0 s_{0} s 0 and from there we can obtain at most 2 2 2 , by choosing a 2 a_{2} a 2 (at period t = 1 t = 1 t = 1 ). Hence 4 ( 1 − x ) ≤ R ( π , M 2 ) ≤ 4 ( 1 − x ) + 2 x = 4 − 2 x 4(1-x) \leq R(\pi,\mathcal{M}_{2}) \leq4(1-x) + 2x = 4 - 2x 4 ( 1 − x ) ≤ R ( π , M 2 ) ≤ 4 ( 1 − x ) + 2 x = 4 − 2 x . Hence ( R ( π , M 1 ) , R ( π , M 2 ) ) ∈ [ x , ( 1 + x ) / 2 ] × [ 4 ( 1 − x ) , 4 − 2 x ] (R(\pi,\mathcal{M}_{1}), R(\pi,\mathcal{M}_{2})) \in[x,(1+x)/2] \times[4(1-x),4-2x] ( R ( π , M 1 ) , R ( π , M 2 )) ∈ [ x , ( 1 + x ) /2 ] × [ 4 ( 1 − x ) , 4 − 2 x ] for x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] , and every point in the rectangle in right-hand side is attainable, by adjusting the probability to play a 1 a_{1} a 1 or a 2 a_{2} a 2 at period t = 1 t = 1 t = 1 , respectively after choosing a 2 a_{2} a 2 or a 1 a_{1} a 1 at period t = 0 t = 0 t = 0 . Therefore R ( Π H ) = ⋃ x ∈ [ 0 , 1 ] [ x , ( 1 + x ) / 2 ] × [ 4 ( 1 − x ) , 4 − 2 x ] R(\Pi_{\mathrm{H}}) = \bigcup_{x \in[0,1]} [x,(1+x)/2] \times[4(1-x),4-2x] R ( Π H ) = ⋃ x ∈ [ 0 , 1 ] [ x , ( 1 + x ) /2 ] × [ 4 ( 1 − x ) , 4 − 2 x ] . What does this look like? This is a quadrilateral with vertices ( 0 , 4 ) (0,4) ( 0 , 4 ) , ( 1 / 2 , 4 ) (1/2,4) ( 1/2 , 4 ) , ( 1 , 2 ) (1,2) ( 1 , 2 ) and ( 1 , 0 ) (1,0) ( 1 , 0 ) , corresponding to the returns achievable by the four policies “always play a 2 a_{2} a 2 ”, “play a 2 a_{2} a 2 then a 1 a_{1} a 1 ”, “play a 1 a_{1} a 1 then a 2 a_{2} a 2 ”, “always play a 1 a_{1} a 1 ”, see Figure 4.1 . The Pareto front is the upper-right boundary of R ( Π H ) R(\Pi_{\mathrm{H}}) R ( Π H ) , corresponding to F ( Π H ) = { ( ( 1 + x ) / 2 , 4 − 2 x ) ∣ x ∈ [ 0 , 1 ] } \mathcal{F}(\Pi_{\mathrm{H}}) = \{((1+x)/2,4-2x) \mid x \in[0,1]\} F ( Π H ) = {(( 1 + x ) /2 , 4 − 2 x ) ∣ x ∈ [ 0 , 1 ]} .
Figure 4.1. Illustration of the achievable sets R ( Π S ) R(\Pi_{\mathrm{S}}) R ( Π S ) , R ( Π H ) R(\Pi_{\mathrm{H}}) R ( Π H ) and the Pareto fronts F ( Π S ) \mathcal{F}(\Pi_{\mathrm{S}}) F ( Π S ) , F ( Π H ) \mathcal{F}(\Pi_{\mathrm{H}}) F ( Π H ) (Figure 4.1a), and the optimal solution of the scalarized problem with the infimum scalarization function as in (61) : Π ⋆ = arg max π ∈ Π H min { R ( π , M 1 ) , R ( π , M 2 ) } \Pi^{\star} = \arg\max_{\pi\in\Pi_{\mathrm{H}}} \min\{R(\pi,\mathcal{M}_1), R(\pi,\mathcal{M}_2)\} Π ⋆ = arg max π ∈ Π H min { R ( π , M 1 ) , R ( π , M 2 )} . (Figure 4.1b).
We next investigate the properties of the sets R ( Π ) R(\Pi) R ( Π ) and F ( Π ) \mathcal{F}(\Pi) F ( Π ) under various classes of policies Π ⊂ Π H \Pi\subset\Pi_{\mathrm{H}} Π ⊂ Π H and various assumptions on the unknown transitions and rewards. This will play an important role in the tractability (or intractability) of computing Pareto optimal policies, as we will see later in this section.
Theorem 4.2. Consider an instance M 1 , … , M N \mathcal{M}_1,\ldots,\mathcal{M}_N M 1 , … , M N of multi-model MDPs.
Assume that the transition probabilities are the same across the MDP models M 1 , … , M N \mathcal{M}_1,\ldots,\mathcal{M}_N M 1 , … , M N . Then:
(a) The set R ( Π H ) R(\Pi_{\mathrm{H}}) R ( Π H ) is a polytope, i.e., it is the convex hull of finitely many points:
R ( Π H ) = conv ( { ( R ( π , M i ) ) i = 1 , … , N ∣ π ∈ Π S D } ) R(\Pi_{\mathrm{H}}) = \operatorname{conv}\left(\left\{\left(R(\pi,\mathcal{M}_i)\right)_{i=1,\ldots,N} \mid\pi\in\Pi_{\mathrm{SD}}\right\}\right) R ( Π H ) = conv ( { ( R ( π , M i ) ) i = 1 , … , N ∣ π ∈ Π SD } ) (b) R ( Π H ) = R ( Π S ) R(\Pi_{\mathrm{H}}) = R(\Pi_{\mathrm{S}}) R ( Π H ) = R ( Π S ) .
(c) F ( Π H ) = F ( Π S ) \mathcal{F}(\Pi_{\mathrm{H}}) = \mathcal{F}(\Pi_{\mathrm{S}}) F ( Π H ) = F ( Π S ) .
Assume that the transition probabilities may differ across the MDP models M 1 , … , M N \mathcal{M}_1,\ldots,\mathcal{M}_N M 1 , … , M N . Then:
(a) The set R ( Π H ) R(\Pi_{\mathrm{H}}) R ( Π H ) is always compact and convex.
(b) We may have that R ( Π S ) R(\Pi_{\mathrm{S}}) R ( Π S ) is not convex.
(c) We may have that F ( Π H ) ≠ F ( Π S ) \mathcal{F}(\Pi_{\mathrm{H}}) \ne\mathcal{F}(\Pi_{\mathrm{S}}) F ( Π H ) = F ( Π S ) .
Proof. 1. We prove the three statements 1(a), 1(b), and 1(c) at the same time. Recall that by linearity of the expectation, the return is a linear function of the instantaneous rewards. Additionally, the state-action frequencies across all different models M 1 , … , M N \mathcal{M}_1,\ldots,\mathcal{M}_N M 1 , … , M N can be described by a single variable μ \mu μ given that the transition probabilities are the same across all MDP models (we write this common kernel P P P here). This means in particular that
R ( Π H ) = { ( R ( π , M i ) ) i = 1 , … , N ∣ π ∈ Π H } = { ( R ( π , M i ) ) i = 1 , … , N ∣ π ∈ Π S } = { ( ∑ s , a μ ( s , a ) P ( s , a ) T r i ( s , a ) ) i = 1 , … , N ∣ μ ∈ Q } = conv { ( R ( π , M i ) ) i = 1 , … , N ∣ π ∈ Π S D } . \begin{aligned}
R(\Pi_{\mathrm{H}})=\{(R(\pi,\mathcal{M}_i))_{i=1,\ldots,N}\mid\pi\in\Pi_{\mathrm{H}}\}=\{(R(\pi,\mathcal{M}_i))_{i=1,\ldots,N}\mid\pi\in\Pi_{\mathrm{S}}\} \\
&=\left\{\left(\sum_{s,a}\mu(s,a)P(s,a)^{\mathsf{T}}r^i(s,a)\right)_{i=1,\ldots,N}\mid\mu\in\mathcal{Q}\right\} \\
&=\operatorname{conv}\{(R(\pi,\mathcal{M}_i))_{i=1,\ldots,N}\mid\pi\in\Pi_{\mathrm{SD}}\}.
\end{aligned} R ( Π H ) = {( R ( π , M i ) ) i = 1 , … , N ∣ π ∈ Π H } = {( R ( π , M i ) ) i = 1 , … , N ∣ π ∈ Π S } = ⎩ ⎨ ⎧ ( s , a ∑ μ ( s , a ) P ( s , a ) T r i ( s , a ) ) i = 1 , … , N ∣ μ ∈ Q ⎭ ⎬ ⎫ = conv {( R ( π , M i ) ) i = 1 , … , N ∣ π ∈ Π SD } .
where we recall that Q \mathcal{Q} Q is the set of state-action frequencies defined in (10) , whose extreme points are the state-action frequencies induced by stationary deterministic policies. Hence R ( Π H ) = R ( Π S ) R(\Pi_{\mathrm{H}})=R(\Pi_{\mathrm{S}}) R ( Π H ) = R ( Π S ) is a polytope (it is the convex hull of finitely many points in R N \mathbb{R}^{N} R N ) and every attainable return vector in R ( Π H ) R(\Pi_{\mathrm{H}}) R ( Π H ) can be achieved by some stationary policy π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S . From R ( Π H ) = R ( Π S ) R(\Pi_{\mathrm{H}})=R(\Pi_{\mathrm{S}}) R ( Π H ) = R ( Π S ) we directly get F ( Π H ) = F ( Π S ) \mathcal{F}(\Pi_{\mathrm{H}})=\mathcal{F}(\Pi_{\mathrm{S}}) F ( Π H ) = F ( Π S ) .
(a) We have seen in Proposition 2.1 that π ↦ R ( π , M ) \pi\mapsto R(\pi,\mathcal{M}) π ↦ R ( π , M ) is continuous for each model M ∈ { M 1 , … , M N } \mathcal{M}\in\{\mathcal{M}_1,\ldots,\mathcal{M}_N\} M ∈ { M 1 , … , M N } , and in Remark 2.2 that Π H \Pi_{\mathrm{H}} Π H is compact (for the product topology). Hence the image R ( Π H ) R(\Pi_{\mathrm{H}}) R ( Π H ) is compact. We now show that it is a convex set. Let π , π ′ ∈ Π H \pi,\pi'\in\Pi_{\mathrm{H}} π , π ′ ∈ Π H , λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] . We will construct a policy π ′ ′ ∈ Π H \pi''\in\Pi_{\mathrm{H}} π ′′ ∈ Π H such that R ( π ′ ′ , M ) = λ R ( π , M ) + ( 1 − λ ) R ( π ′ , M ) R(\pi'',\mathcal{M})=\lambda R(\pi,\mathcal{M})+(1-\lambda)R(\pi',\mathcal{M}) R ( π ′′ , M ) = λ R ( π , M ) + ( 1 − λ ) R ( π ′ , M ) simultaneously for all MDP models M \mathcal{M} M . To do so, note that we just need to construct π ′ ′ \pi'' π ′′ such that, for any t ∈ N t\in\mathbb{N} t ∈ N and any finite history h t = ( s 0 , a 0 , … , s t − 1 , a t − 1 , s t ) h_t=(s_0,a_0,\ldots,s_{t-1},a_{t-1},s_t) h t = ( s 0 , a 0 , … , s t − 1 , a t − 1 , s t ) , we have
∏ t ′ < t π ′ ′ ( h t ′ ) ( a t ′ ) = λ ∏ t ′ < t π ( h t ′ ) ( a t ′ ) + ( 1 − λ ) ∏ t ′ < t π ′ ( h t ′ ) ( a t ′ ) . \prod_{t'<t}\pi''(h_{t'})(a_{t'})=\lambda\prod_{t'<t}\pi(h_{t'})(a_{t'})+(1-\lambda)\prod_{t'<t}\pi'(h_{t'})(a_{t'}). t ′ < t ∏ π ′′ ( h t ′ ) ( a t ′ ) = λ t ′ < t ∏ π ( h t ′ ) ( a t ′ ) + ( 1 − λ ) t ′ < t ∏ π ′ ( h t ′ ) ( a t ′ ) .
This can be done by choosing π ′ ′ ( h t ) ( a ) \pi''(h_t)(a) π ′′ ( h t ) ( a ) proportional to λ ( ∏ t ′ < t π ( h t ′ ) ( a t ′ ) ) π ( h t ) ( a ) + ( 1 − λ ) ( ∏ t ′ < t π ′ ( h t ′ ) ( a t ′ ) ) π ′ ( h t ) ( a ) \lambda\left(\prod_{t'<t}\pi(h_{t'})(a_{t'})\right)\pi(h_t)(a)+(1-\lambda)\left(\prod_{t'<t}\pi'(h_{t'})(a_{t'})\right)\pi'(h_t)(a) λ ( ∏ t ′ < t π ( h t ′ ) ( a t ′ ) ) π ( h t ) ( a ) + ( 1 − λ ) ( ∏ t ′ < t π ′ ( h t ′ ) ( a t ′ ) ) π ′ ( h t ) ( a ) for each h t h_t h t and candidate action a ∈ A a\in\mathcal{A} a ∈ A when λ ( ∏ t ′ < t π ( h t ′ ) ( a t ′ ) ) + ( 1 − λ ) ( ∏ t ′ < t π ′ ( h t ′ ) ( a t ′ ) ) \lambda\left(\prod_{t'<t}\pi(h_{t'})(a_{t'})\right)+(1-\lambda)\left(\prod_{t'<t}\pi'(h_{t'})(a_{t'})\right) λ ( ∏ t ′ < t π ( h t ′ ) ( a t ′ ) ) + ( 1 − λ ) ( ∏ t ′ < t π ′ ( h t ′ ) ( a t ′ ) ) is not null, otherwise π ′ ′ ( h t ) ( a ) = 1 / ∣ A ∣ \pi''(h_t)(a)=1/|\mathcal{A}| π ′′ ( h t ) ( a ) = 1/∣ A ∣ for each action a a a . By induction, the distribution of the trajectories under π ′ ′ \pi'' π ′′ is indeed the mixture of the laws under π \pi π and π ′ \pi' π ′ , hence we get that R ( π ′ ′ , M ) = λ R ( π , M ) + ( 1 − λ ) R ( π ′ , M ) R(\pi'',\mathcal{M})=\lambda R(\pi,\mathcal{M})+(1-\lambda)R(\pi',\mathcal{M}) R ( π ′′ , M ) = λ R ( π , M ) + ( 1 − λ ) R ( π ′ , M ) .
(b) See Figure 4.1 and the analysis in Example 4.4.
(c) See Figure 4.1 and the analysis in Example 4.4.
Theorem 4.2 gives a sufficient condition for F ( Π S ) = F ( Π H ) \mathcal{F}(\Pi_{\mathrm{S}})=\mathcal{F}(\Pi_{\mathrm{H}}) F ( Π S ) = F ( Π H ) , i.e., for policies in Π S \Pi_{\mathrm{S}} Π S to suffice to attain every point of the Pareto front F ( Π H ) \mathcal{F}(\Pi_{\mathrm{H}}) F ( Π H ) . In particular, this is greatly dependent on the assumptions on the unknown transitions and rewards: the case of a common transition kernel has much more favorable properties than the case where the kernels differ across models. We next investigate the computation of Pareto optimal policies based on so-called “scalarization functions”.
Scalarization functions. A natural approach to deal with multi-objective optimization (beyond MDPs) is to introduce a scalarization function , which is simply a map f : R N → R f : \mathbb{R}^{N} \to\mathbb{R} f : R N → R . We then seek to find a policy that optimizes f ( R ( π ) ) f(R(\pi)) f ( R ( π )) over a chosen policy class Π ⊂ Π H \Pi\subset\Pi_{\mathrm{H}} Π ⊂ Π H , i.e., we want to solve
sup π ∈ Π f ( R ( π ) ) . (59) \sup_{\pi\in\Pi} f(R(\pi)). \tag*{(59)} π ∈ Π sup f ( R ( π )) . (59)
This model is of course very general:
When f ( R ) = R i f(R) = R_i f ( R ) = R i for R ∈ R N R \in\mathbb{R}^{N} R ∈ R N then (59) recovers the nominal MDP model from Chapter 1. More generally, when the scalarization function is linear: f ( R ) = ∑ i = 1 N w i R i f(R) = \sum_{i=1}^{N} w_iR_i f ( R ) = ∑ i = 1 N w i R i for some w ∈ Δ ( { 1 , … , N } ) w \in\Delta(\{1, \ldots, N\}) w ∈ Δ ({ 1 , … , N }) , then we obtain the so-called multi-model MDPs . Note that the class of functions f : R ↦ w ⊤ R f : R \mapsto w^{\top}R f : R ↦ w ⊤ R for some w ∈ R + N w \in\mathbb{R}^{N}_{+} w ∈ R + N yields the same optimizers of (59) as when w ∈ Δ ( { 1 , … , N } ) w \in\Delta(\{1, \ldots, N\}) w ∈ Δ ({ 1 , … , N }) (since we can just renormalize the weight vector).
When f ( R ) = min i = 1 , … , N R i f(R) = \min_{i=1,\ldots,N} R_i f ( R ) = min i = 1 , … , N R i for R ∈ R N R \in\mathbb{R}^{N} R ∈ R N then (59) recovers the robust MDP model with uncertain rewards and transitions from Section 4.2, when there are finitely many unknown scenarios.
We now discuss two important questions:
Question 1: Given a policy class Π \Pi Π , when does solving (59) over Π \Pi Π yield Pareto optimal policies in Π \Pi Π ? Can every Pareto optimal policy in Π \Pi Π be computed this way for an appropriate scalarization function?
Question 2: Given a policy class Π \Pi Π , are there efficient algorithms for solving (59) (so that we can leverage the answer to the previous question to compute Pareto optimal policies)?
The answers to the above questions depend on (A) the geometry of R ( Π ) R(\Pi) R ( Π ) , which in turn depends on the assumptions made on the rewards and transition probabilities in the models M 1 , … , M N \mathcal{M}_1,\ldots,\mathcal{M}_N M 1 , … , M N , and (B) the choice of the scalarization function f f f . We will study each set of assumptions on the uncertain models separately, and we will focus on linear scalarizations (since these are by far the most popular scalarization functions for multi-objective MDPs) and on worst-case scalarization (which recovers robust MDPs). We start with linear scalarization.
Linear scalarization and multi-model MDPs
Here we focus on the case where we are given some w ∈ Δ ( { 1 , … , N } ) w \in\Delta(\{1,\ldots,N\}) w ∈ Δ ({ 1 , … , N }) and where f : R ↦ w ⊤ R f : R \mapsto w^{\top}R f : R ↦ w ⊤ R , so that (59) becomes, for a chosen class of policies Π ⊂ Π H \Pi\subset\Pi_{\mathrm{H}} Π ⊂ Π H , the following optimization problem:
sup π ∈ Π ∑ i = 1 N w i R ( π , M i ) (60) \sup_{\pi\in\Pi} \sum_{i=1}^{N} w_i R(\pi,\mathcal{M}_i) \tag*{(60)} π ∈ Π sup i = 1 ∑ N w i R ( π , M i ) (60)
We next give a general response to the two questions outlined above, for the case of linear scalarization. We start with the first question (when linear scalarization recovers Pareto optimal solutions).
Theorem 4.3. Let Π ⊆ Π H \Pi\subseteq\Pi_{\mathrm{H}} Π ⊆ Π H be a class of policies (e.g. Π ∈ { Π S , Π M , Π H } \Pi\in\{\Pi_{\mathrm{S}},\Pi_{\mathrm{M}},\Pi_{\mathrm{H}}\} Π ∈ { Π S , Π M , Π H } ). The following statements hold:
Every optimal solution to (60) for some w ∈ Δ ( { 1 , … , N } ) w \in\Delta(\{1,\ldots,N\}) w ∈ Δ ({ 1 , … , N }) is weakly Pareto optimal in Π \Pi Π .
If w i > 0 w_i > 0 w i > 0 for every i i i , every optimal solution to (60) is Pareto optimal in Π \Pi Π .
If R ( Π ) R(\Pi) R ( Π ) is a convex set, then every weakly Pareto optimal policy in Π \Pi Π is an optimal solution of the scalarized problem for some w ∈ Δ ( { 1 , … , N } ) w \in\Delta(\{1,\ldots,N\}) w ∈ Δ ({ 1 , … , N }) and f : R ↦ w T R f:\mathbb{R}\mapsto w^{\mathsf{T}}R f : R ↦ w T R .
If R ( Π ) R(\Pi) R ( Π ) is a polytope, then every Pareto optimal policy in Π \Pi Π is an optimal solution of (60) for some w ∈ Δ ( { 1 , … , N } ) w \in\Delta(\{1,\ldots,N\}) w ∈ Δ ({ 1 , … , N }) with w i > 0 w_i > 0 w i > 0 for every i i i , and f : R ↦ w T R f:\mathbb{R}\mapsto w^{\mathsf{T}}R f : R ↦ w T R .
Proof. 1. This first statement simply follows from the non-negativity of the weight vector w ∈ Δ ( { 1 , … , N } ) w \in\Delta(\{1,\ldots,N\}) w ∈ Δ ({ 1 , … , N }) . Indeed if π \pi π is optimal in (60) then if there is a policy in Π \Pi Π that strictly Pareto dominates π \pi π it would yield a strictly better objective value in (60) . Hence π \pi π optimal in (60) implies that π \pi π is weakly Pareto optimal.
In the case where w i > 0 w_i > 0 w i > 0 for every model i ∈ { 1 , … , N } i \in\{1,\ldots,N\} i ∈ { 1 , … , N } then the same argument as for the first statement shows that π \pi π optimal in (60) implies that π \pi π is Pareto optimal.
This is an implication of the hyperplane separation theorem (see Appendix § 7 ). Let R ∈ R ( Π ) R \in R(\Pi) R ∈ R ( Π ) be weakly Pareto optimal. Then R ( Π ) − R R(\Pi)-R R ( Π ) − R and ( 0 , + ∞ ) N (0,+\infty)^N ( 0 , + ∞ ) N are two disjoint convex sets. Therefore we can find a direction w ∈ R + N w \in\mathbb{R}_{+}^{N} w ∈ R + N , w ≠ ( 0 , … , 0 ) w \ne(0,\ldots,0) w = ( 0 , … , 0 ) such that w T ( R ′ − R ) ≤ 0 w^{\mathsf{T}}(R^{\prime}-R) \le0 w T ( R ′ − R ) ≤ 0 , ∀ R ′ ∈ R ( Π ) \forall R^{\prime} \in R(\Pi) ∀ R ′ ∈ R ( Π ) . Renormalizing w w w to an element in the simplex Δ ( { 1 , … , N } ) \Delta(\{1,\ldots,N\}) Δ ({ 1 , … , N }) yields the conclusion.
This is an application of duality in linear programming. Let R R R be the vector of returns of a Pareto optimal policy. Then R R R is the only feasible vector in the optimization program max { ∑ i = 1 N R i ′ ∣ R ′ ∈ R ( Π ) , R ′ ≥ R } \max\left\{\sum_{i=1}^{N}R_i^{\prime}\mid R^{\prime}\in R(\Pi),R^{\prime}\ge R\right\} max { ∑ i = 1 N R i ′ ∣ R ′ ∈ R ( Π ) , R ′ ≥ R } . From strong duality we get some nonnegative Lagrange multipliers λ 1 , … , λ N ≥ 0 \lambda_1,\ldots,\lambda_N\ge0 λ 1 , … , λ N ≥ 0 such that R ∈ arg max { ∑ i = 1 N ( 1 + λ i ) R i ′ ∣ R ′ ∈ R ( Π ) } R\in\operatorname*{arg\,max}\left\{\sum_{i=1}^{N}(1+\lambda_i)R_i^{\prime}\mid R^{\prime}\in R(\Pi)\right\} R ∈ arg max { ∑ i = 1 N ( 1 + λ i ) R i ′ ∣ R ′ ∈ R ( Π ) } . This yields the optimization problem sup π ∈ Π { ∑ i = 1 N ( 1 + λ i ) R ( π , M i ) ∣ π ∈ Π } \sup_{\pi\in\Pi}\left\{\sum_{i=1}^{N}(1+\lambda_i)R(\pi,\mathcal{M}_i)\mid\pi\in\Pi\right\} sup π ∈ Π { ∑ i = 1 N ( 1 + λ i ) R ( π , M i ) ∣ π ∈ Π } of the form (60) with positive coefficients.
:::
Combining Theorem 4.3 with Theorem 4.2, we get the following implications:
When the transition probabilities are common across all models, then we can recover every Pareto optimal policy in Π H \Pi_{\mathrm{H}} Π H as a solution of (60) for some positive weight vector w w w , and since R ( Π S ) = R ( Π H ) R(\Pi_{\mathrm{S}})=R(\Pi_{\mathrm{H}}) R ( Π S ) = R ( Π H ) in this case, stationary (randomized) policies suffice to attain every point of the Pareto frontier F ( Π H ) \mathcal{F}(\Pi_{\mathrm{H}}) F ( Π H ) .
When the transition probabilities may be different across the models, then we can recover every weakly Pareto optimal policy in Π H \Pi_{\mathrm{H}} Π H as a solution of (60) for some nonnegative weight vector w w w . We may need history-dependent policies to reach some Pareto optimal vector in F ( Π H ) \mathcal{F}(\Pi_{\mathrm{H}}) F ( Π H ) .
We now describe the complexity properties of the optimization program in (60) .
Tractability of (4.10): the case of uncertain rewards. We assume here that the transition probabilities are the same across all MDP models M 1 , … , M N \mathcal{M}_1,\ldots,\mathcal{M}_N M 1 , … , M N and are equal to some P ∈ Δ ( S ) S × A P \in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ∈ Δ ( S ) S × A . In this case we can reformulate R ( π ) R(\pi) R ( π ) given some π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H . Indeed, by linearity of the expectation, the return is a linear function of the rewards. Hence we get, for some f : R ↦ w ⊤ R f : \mathbb{R} \mapsto w^{\top}R f : R ↦ w ⊤ R with w ∈ R + N w \in\mathbb{R}^{N}_{+} w ∈ R + N , that
f ( R ( π ) ) = ∑ i = 1 N w i R ( π , M i ) = ∑ i = 1 N w i E π , P [ ∑ t = 0 + ∞ γ t r i ( s t , a t , s t + 1 ) ] = E π , P [ ∑ t = 0 + ∞ γ t ( ∑ i = 1 N w i r i ( s t , a t , s t + 1 ) ) ] = R ( π , M ‾ ) \begin{aligned}
f(R(\pi)) &= \sum_{i=1}^{N} w_i R(\pi,\mathcal{M}_i) = \sum_{i=1}^{N} w_i \mathbb{E}^{\pi,P}\left[\sum_{t=0}^{+\infty} \gamma^{t}r^{i}(s_t,a_t,s_{t+1})\right] \\
&= \mathbb{E}^{\pi,P}\left[\sum_{t=0}^{+\infty} \gamma^{t}\left(\sum_{i=1}^{N}w_i r^{i}(s_t,a_t,s_{t+1})\right)\right] \\
&= R(\pi,\overline{\mathcal{M}})
\end{aligned} f ( R ( π )) = i = 1 ∑ N w i R ( π , M i ) = i = 1 ∑ N w i E π , P [ t = 0 ∑ + ∞ γ t r i ( s t , a t , s t + 1 ) ] = E π , P [ t = 0 ∑ + ∞ γ t ( i = 1 ∑ N w i r i ( s t , a t , s t + 1 ) ) ] = R ( π , M )
for M ‾ \overline{\mathcal{M}} M the MDP model ( S , A , ∑ i = 1 N w i r i , P , p 0 ) (\mathcal{S},\mathcal{A},\sum_{i=1}^{N}w_i r^i,P,p_0) ( S , A , ∑ i = 1 N w i r i , P , p 0 ) . Therefore in this case (4.10) boils down to a nominal MDP model as studied in Chapter 1, where the instantaneous rewards are the weighted sum of the instantaneous rewards of each model, so we can solve (4.10) efficiently (see Section 1.2.2 for the description of the algorithms studied in these notes) for Π ∈ { Π H , Π M , Π S } \Pi\in\{\Pi_{\mathrm{H}},\Pi_{\mathrm{M}},\Pi_{\mathrm{S}}\} Π ∈ { Π H , Π M , Π S } (and more generally any class of policies containing Π S D \Pi_{\mathrm{SD}} Π SD ).
Tractability of (4.10): the case of uncertain transitions. When the transition probabilities are also uncertain and may differ across MDP models, things can become very complicated. For instance, it is known that solving the following optimization problem is NP-hard:
sup π ∈ Π H 1 2 R ( π , M 1 ) + 1 2 R ( π , M 2 ) \sup_{\pi\in\Pi_{\mathrm{H}}} \frac{1}{2}R(\pi,\mathcal{M}_1)+\frac{1}{2}R(\pi,\mathcal{M}_2) π ∈ Π H sup 2 1 R ( π , M 1 ) + 2 1 R ( π , M 2 )
for the discounted return considered in this section, but this remains the case also for MDPs with finite horizon. Hence solving (4.10) may not be feasible when the transition probabilities are not the same across all models.
Worst-case scalarization and robust MDPs
This is the case most akin to robust MDPs as we studied here, and we consider solving the following optimization problem to compute Pareto optimal policies:
sup π ∈ Π min i = 1 , … , N R ( π , M i ) (61) \sup_{\pi\in\Pi}\min_{i=1,\ldots,N}R(\pi,\mathcal{M}_i) \tag*{(61)} π ∈ Π sup i = 1 , … , N min R ( π , M i ) (61)
We write Π ⋆ = arg max π ∈ Π min i = 1 , … , N R ( π , M i ) \Pi^\star=\arg\max_{\pi\in\Pi}\min_{i=1,\ldots,N}R(\pi,\mathcal{M}_i) Π ⋆ = arg max π ∈ Π min i = 1 , … , N R ( π , M i ) . The motivation for solving the above mathematical program also sometimes comes from fairness considerations (we want to perform well across all scenarios and not just the most important ones). The following theorem characterizes the Pareto optimality properties of optimal solutions in (4.11).
Theorem 4.4. Let Π ⊂ Π H \Pi\subset\Pi_{\mathrm{H}} Π ⊂ Π H be a policy class such that R ( Π ) R(\Pi) R ( Π ) is compact . The following statements hold:
Every optimal solution of (4.11) is weakly Pareto optimal in Π \Pi Π .
There exists at least one optimal solution of (4.11) that is Pareto optimal in Π \Pi Π .
Not every optimal solution of (4.11) is necessarily Pareto optimal in Π \Pi Π .
Not every Pareto optimal policy in Π \Pi Π can necessarily be recovered as a solution of (4.11).
Proof. 1. This is a direct reformulation of strict Pareto domination (and therefore weak Pareto optimality): if π ∈ Π \pi\in\Pi π ∈ Π is an optimal solution to (4.11) then if π ′ ∈ Π \pi' \in\Pi π ′ ∈ Π strictly Pareto dominates π \pi π we would have min i R ( π ′ , M i ) > min i R ( π , M i ) \min_i R(\pi',\mathcal{M}_i) > \min_i R(\pi,\mathcal{M}_i) min i R ( π ′ , M i ) > min i R ( π , M i ) , which contradicts the optimality of π \pi π in (4.11).
Consider Π ⋆ = arg max π ∈ Π min i R ( π , M i ) \Pi^\star= \arg\max_{\pi\in\Pi}\min_i R(\pi,\mathcal{M}_i) Π ⋆ = arg max π ∈ Π min i R ( π , M i ) the set of optimal solutions in (4.11). The set R ( Π ⋆ ) R(\Pi^\star) R ( Π ⋆ ) is compact by compactness of R ( Π ) R(\Pi) R ( Π ) and continuity of R ↦ min i R i R \mapsto\min_i R_i R ↦ min i R i . Now consider the subset Π s u m ⋆ = arg max π ∈ Π ⋆ ∑ i = 1 N R ( π , M i ) \Pi^\star_{\mathrm{sum}} = \arg\max_{\pi\in\Pi^\star}\sum_{i=1}^{N} R(\pi,\mathcal{M}_i) Π sum ⋆ = arg max π ∈ Π ⋆ ∑ i = 1 N R ( π , M i ) , which corresponds to the set of optimal solutions in (4.11) that also maximize the sum. Then policies in Π s u m ⋆ \Pi^\star_{\mathrm{sum}} Π sum ⋆ are both Pareto optimal and, by construction, optimal solutions in (4.11).
See Figure 4.1 and the analysis in Example 4.4 and below.
See Figure 4.1 and the analysis in Example 4.4 and below. □ \square □
Example 4.4 continued. We continue with the multi-objective MDP instance from Example 4.4, and we compute the set Π ⋆ = arg max π ∈ Π H min { R ( π , M 1 ) , R ( π , M 2 ) } \Pi^\star= \arg\max_{\pi\in\Pi_{\mathrm{H}}}\min\{R(\pi,\mathcal{M}_1), R(\pi,\mathcal{M}_2)\} Π ⋆ = arg max π ∈ Π H min { R ( π , M 1 ) , R ( π , M 2 )} . We have shown in Example 4.4 that R ( π , M 1 ) ≤ 1 R(\pi,\mathcal{M}_1) \le1 R ( π , M 1 ) ≤ 1 for any π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H , and we have seen that we can achieve ( 1 , 2 ) (1, 2) ( 1 , 2 ) with the policy “play a 1 a_1 a 1 then a 2 a_2 a 2 ”, hence max π ∈ Π H min { R ( π , M 1 ) , R ( π , M 2 ) } = 1 \max_{\pi\in\Pi_{\mathrm{H}}}\min\{R(\pi,\mathcal{M}_1), R(\pi,\mathcal{M}_2)\} = 1 max π ∈ Π H min { R ( π , M 1 ) , R ( π , M 2 )} = 1 . The policies achieving this maximum must be such that R ( π , M 1 ) = 1 R(\pi,\mathcal{M}_1) = 1 R ( π , M 1 ) = 1 (since this is the best one can achieve in M 1 \mathcal{M}_1 M 1 ). Our characterization of R ( Π H ) R(\Pi_{\mathrm{H}}) R ( Π H ) from Example 4.4 gives that for such policies we must have R ( π , M 2 ) ∈ [ 0 , 2 ] R(\pi,\mathcal{M}_2) \in[0,2] R ( π , M 2 ) ∈ [ 0 , 2 ] , and for a policy to be optimal in (4.11) we must also have R ( π , M 2 ) ∈ [ 1 , 2 ] R(\pi,\mathcal{M}_2) \in[1,2] R ( π , M 2 ) ∈ [ 1 , 2 ] . Hence R ( Π ⋆ ) = { 1 } × [ 1 , 2 ] R(\Pi^\star) = \{1\} \times[1,2] R ( Π ⋆ ) = { 1 } × [ 1 , 2 ] , attained by policies choosing a 1 a_1 a 1 at period 0, then if still at s 0 s_0 s 0 at period 1 choosing a 2 a_2 a 2 with some probability y ∈ [ 0 , 1 ] y \in[0,1] y ∈ [ 0 , 1 ] , otherwise choosing a 2 a_2 a 2 at period 2 (if still in s 0 s_0 s 0 at period 2). This yields R ( π ) = ( 1 , 1 + y ) R(\pi) = (1,1+y) R ( π ) = ( 1 , 1 + y ) for y ∈ [ 0 , 1 ] y \in[0,1] y ∈ [ 0 , 1 ] . Note that ( 1 , 1 ) ∈ R ( Π ⋆ ) (1,1) \in R(\Pi^\star) ( 1 , 1 ) ∈ R ( Π ⋆ ) but this vector is Pareto dominated by ( 1 , 2 ) (1,2) ( 1 , 2 ) , therefore optimality in (4.11) does not imply Pareto optimality; additionally, ( 1 / 2 , 4 ) ∈ F ( Π H ) (1/2,4) \in\mathcal{F}(\Pi_{\mathrm{H}}) ( 1/2 , 4 ) ∈ F ( Π H ) but is not attained by an optimal solution in (4.11). This provides the proofs for the third and fourth statements of Theorem 4.4.
Tractability properties of (4.11): the case of uncertain rewards. In this case with Π = Π H \Pi= \Pi_{\mathrm{H}} Π = Π H we are back to the case of robust MDPs with uncertain rewards as studied in § 4.1 . In particular for R = { r 1 , … , r N } \mathcal{R} = \{r^1,\ldots,r^N\} R = { r 1 , … , r N } then (4.11) is equivalent to (4.1), i.e., to sup π ∈ Π H min r ∈ R R ( π , r ) \sup_{\pi\in\Pi_{\mathrm{H}}}\min_{r\in\mathcal{R}} R(\pi,r) sup π ∈ Π H min r ∈ R R ( π , r ) ; we refer to § 4.1 for the favorable properties of this optimization problem (among other things, we can solve this problem via linear programming).
Tractability properties of (4.11): the case of uncertain transitions. Here for Π = Π H \Pi= \Pi_{\mathrm{H}} Π = Π H the properties of (4.11) are similar to those of (4.10); it is known (see references at the end of this chapter) that solving
sup π ∈ Π H min { R ( π , P 1 ) , R ( π , P 2 ) } \sup_{\pi\in\Pi_{\mathrm{H}}} \min\{R(\pi,P^{1}), R(\pi,P^{2})\} π ∈ Π H sup min { R ( π , P 1 ) , R ( π , P 2 )}
is NP-hard in all generality. When the set P = { P 1 , … , P N } \mathcal{P} = \{P^{1},\ldots,P^{N}\} P = { P 1 , … , P N } is s-rectangular (not necessarily convex) and the rewards are common across the models, then Section 3.3.4 provides algorithms for finding a best stationary policy (not necessarily optimal among history-dependent policies). If P \mathcal{P} P is sa-rectangular then the results in Section 3.4 apply directly (we don’t need convexity for solving sa-rectangular robust MDPs). If the rewards also differ across models, then we refer to Section 4.2.
Connection with linear scalarizations. We conclude this section with a short comparison between linear scalarization and worst-case scalarization. Indeed we always have
sup π ∈ Π H min i = 1 , … , N R ( π , M i ) = sup π ∈ Π H min w ∈ Δ ( { 1 , … , N } ) ∑ i = 1 N w i R ( π , M i ) \sup_{\pi\in\Pi_{\mathrm{H}}}\min_{i=1,\ldots,N} R(\pi,\mathcal{M}_{i}) = \sup_{\pi\in\Pi_{\mathrm{H}}}\min_{w \in\Delta(\{1,\ldots,N\})}\sum_{i=1}^{N}w_{i}R(\pi,\mathcal{M}_{i}) π ∈ Π H sup i = 1 , … , N min R ( π , M i ) = π ∈ Π H sup w ∈ Δ ({ 1 , … , N }) min i = 1 ∑ N w i R ( π , M i )
so we can interpret the worst-case scalarization as the infimum over all linear scalarizations. This explains why it has “weaker” properties (in terms of recovering Pareto optimal policies) than linear scalarization (as evident from comparing Theorem 4.3 with Theorem 4.4).
Robust MDPs with time-varying transition probabilities
In Chapter 2 and Chapter 3 we assumed that the transition probabilities were chosen by the adversary once and for all before the decision maker starts choosing some actions. However we can imagine some setups where the transition probabilities are non-stationary and may change over time. We then need to choose a model for how the transition kernel can evolve.
Three models of dynamic transition uncertainties
Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be any uncertainty set. We will assume that at every period t t t , the transition kernel is some P t ∈ P P^{t} \in\mathcal{P} P t ∈ P chosen by the adversary, that is, the transition probabilities may change but always belong to the same set (this simplifies greatly the arguments in this section). We now need to rigorously describe how the transition P t P^{t} P t for period t t t is chosen by the adversary. Similarly to when we defined the classes of stationary, Markovian, and history-dependent policies for the decision maker in Chapter 1, we need to describe what kind of information the adversary can observe when choosing the transition probabilities P t P^{t} P t at each time period. Of course, at each period t t t the transition probabilities P t P^{t} P t need to be determined before the next state s t + 1 s_{t+1} s t + 1 is realized. This leaves basically three levels of information that the adversary could use:
Either P t P^t P t is chosen after observing the current ( s t , a t ) (s_t,a_t) ( s t , a t ) state-action realization,
Or P t P^t P t is chosen after observing the current state s t s_t s t but before the decision maker chooses their action a t a_t a t ,
Or P t P^t P t is chosen before observing the current state s t s_t s t .
These three cases have deep connections with sa-rectangular, s-rectangular, and non-rectangular models, as we investigate next.
What is observed? (part 2) We will use the notation
G t − 1 = ( S × A ) t , t ∈ N \mathcal{G}_{t-1}=(\mathcal{S}\times\mathcal{A})^t,\qquad t\in\mathbb{N} G t − 1 = ( S × A ) t , t ∈ N
to denote the set of all finite histories ( s 0 , a 0 , … , s t − 1 , a t − 1 ) (s_0,a_0,\ldots,s_{t-1},a_{t-1}) ( s 0 , a 0 , … , s t − 1 , a t − 1 ) up to time t − 1 t-1 t − 1 , completed with the last chosen action (recall that by definition from Chapter 1 histories do not include the last chosen action). We write g t g_t g t for a generic element in G t − 1 \mathcal{G}_{t-1} G t − 1 . For each t ∈ N t\in\mathbb{N} t ∈ N , a decision rule of the adversary at period t t t maps a realization ( g t , s t , a t ) ∈ ( S × A ) t + 1 (g_t,s_t,a_t)\in(\mathcal{S}\times\mathcal{A})^{t+1} ( g t , s t , a t ) ∈ ( S × A ) t + 1 to some transition probabilities in Δ ( S ) \Delta(\mathcal{S}) Δ ( S ) , representing the distribution of the next state s t + 1 ∈ S s_{t+1}\in\mathcal{S} s t + 1 ∈ S given s t , a t s_t,a_t s t , a t . A history-dependent policy κ \kappa κ of the adversary is given by a sequence of decision rules ( κ t ) t ∈ N (\kappa_t)_{t\in\mathbb{N}} ( κ t ) t ∈ N with κ t : ( S × A ) t + 1 → Δ ( S ) \kappa_t:(\mathcal{S}\times\mathcal{A})^{t+1}\to\Delta(\mathcal{S}) κ t : ( S × A ) t + 1 → Δ ( S ) for each t ∈ N t\in\mathbb{N} t ∈ N .
We will also need the following notation. For a given uncertainty set P ⊂ Δ ( S ) S × A \mathcal{P}\subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A , we consider its sets of s a sa s a -marginals P s , a : = { P ( s , a ) ∣ P ∈ P } ⊂ Δ ( S ) \mathcal{P}_{s,a}:=\{P(s,a)\mid P\in\mathcal{P}\}\subset\Delta(\mathcal{S}) P s , a := { P ( s , a ) ∣ P ∈ P } ⊂ Δ ( S ) for each pair ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A , and its sets of s s s -marginals P s : = { P ( s ) ∣ P ∈ P } ⊂ Δ ( S ) A \mathcal{P}_s:=\{P(s)\mid P\in\mathcal{P}\}\subset\Delta(\mathcal{S})^{\mathcal{A}} P s := { P ( s ) ∣ P ∈ P } ⊂ Δ ( S ) A for each s ∈ S s\in\mathcal{S} s ∈ S . We consider the following models of time-varying transition probabilities:
Model 1 (“post-state, post-action”): P H S A \mathcal{P}^{\mathrm{SA}}_{\mathrm{H}} P H SA . The set P H S A \mathcal{P}^{\mathrm{SA}}_{\mathrm{H}} P H SA coincides with the set of adversary policies that choose their decision rules consistent with P \mathcal{P} P , after observing the current state-action pair ( s t , a t ) (s_t,a_t) ( s t , a t ) :
P H S A = { ( κ t ) t ∈ N ∣ κ t ( g t , s t , a t ) ∈ P s t , a t , ∀ t ∈ N , g t ∈ G t − 1 , s t ∈ S , a t ∈ A } . (62) \mathcal{P}^{\mathrm{SA}}_{\mathrm{H}}=\left\{(\kappa_t)_{t\in\mathbb{N}}\mid\kappa_t(g_t,s_t,a_t)\in\mathcal{P}_{s_t,a_t},\ \forall t\in\mathbb{N},\ g_t\in\mathcal{G}_{t-1},\ s_t\in\mathcal{S},\ a_t\in\mathcal{A}\right\}. \tag*{(62)} P H SA = { ( κ t ) t ∈ N ∣ κ t ( g t , s t , a t ) ∈ P s t , a t , ∀ t ∈ N , g t ∈ G t − 1 , s t ∈ S , a t ∈ A } . (62)
Model 2 (“post state, pre-action”): P H S \mathcal{P}^{\mathrm{S}}_{\mathrm{H}} P H S . The set P H S \mathcal{P}^{\mathrm{S}}_{\mathrm{H}} P H S coincides with the set of adversary policies that choose their decision rules consistent with P \mathcal{P} P , after observing the current state s t s_t s t :
P H S = { ( κ t ) t ∈ N ∣ ( κ t ( g t , s t , a ) ) a ∈ A ∈ P s t , ∀ t ∈ N , g t ∈ G t − 1 , s t ∈ S } . (63) \mathcal{P}^{\mathrm{S}}_{\mathrm{H}}=\left\{(\kappa_t)_{t\in\mathbb{N}}\mid(\kappa_t(g_t,s_t,a))_{a\in\mathcal{A}}\in\mathcal{P}_{s_t},\ \forall t\in\mathbb{N},\ g_t\in\mathcal{G}_{t-1},\ s_t\in\mathcal{S}\right\}. \tag*{(63)} P H S = { ( κ t ) t ∈ N ∣ ( κ t ( g t , s t , a ) ) a ∈ A ∈ P s t , ∀ t ∈ N , g t ∈ G t − 1 , s t ∈ S } . (63)
P H G = { ( κ t ) t ∈ N ∣ ( κ t ( g t , s , a ) ) s , a ∈ P , ∀ t ∈ N , g t ∈ G t − 1 } . (64) \mathcal{P}^{\mathrm{G}}_{\mathrm{H}}=\left\{(\kappa_t)_{t\in\mathbb{N}}\mid(\kappa_t(g_t,s,a))_{s,a}\in\mathcal{P},\ \forall t\in\mathbb{N},\ g_t\in\mathcal{G}_{t-1}\right\}. \tag*{(64)} P H G = { ( κ t ) t ∈ N ∣ ( κ t ( g t , s , a ) ) s , a ∈ P , ∀ t ∈ N , g t ∈ G t − 1 } . (64)
Given that Model i + 1 i+1 i + 1 imposes some additional constraints compared to Model i i i above (for i ∈ { 1 , 2 } i\in\{1,2\} i ∈ { 1 , 2 } ), we easily get that P H G ⊆ P H S ⊆ P H S A \mathcal{P}^{\mathrm{G}}_{\mathrm{H}}\subseteq\mathcal{P}^{\mathrm{S}}_{\mathrm{H}}\subseteq\mathcal{P}^{\mathrm{SA}}_{\mathrm{H}} P H G ⊆ P H S ⊆ P H SA .
The first important thing to notice is that in the definitions of Model 1 and Model 2, the uncertainty set P \mathcal{P} P only appears via its marginals (P s , a \mathcal{P}_{s,a} P s , a for each state-action pair ( s , a ) (s,a) ( s , a ) for Model 1 and P s \mathcal{P}_s P s for each state s s s in Model 2). This suggests that only certain “rectangularizations” of P \mathcal{P} P matter in Model 1 and Model 2. We make this notion rigorous next.
Definition 4.3. Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be an uncertainty set.
Rec s ( P ) = × s ∈ S { P ( s ) ∣ P ∈ P } . \operatorname{Rec}_{s}(\mathcal{P}) = \mathop{\times}_{s\in\mathcal{S}}\{P(s) \mid P \in\mathcal{P}\}. Rec s ( P ) = × s ∈ S { P ( s ) ∣ P ∈ P } . Rec s a ( P ) = × ( s , a ) ∈ S × A { P ( s , a ) ∣ P ∈ P } . \operatorname{Rec}_{sa}(\mathcal{P}) = \mathop{\times}_{(s,a)\in\mathcal{S}\times\mathcal{A}}\{P(s,a) \mid P \in\mathcal{P}\}. Rec s a ( P ) = × ( s , a ) ∈ S × A { P ( s , a ) ∣ P ∈ P } .
Equivalently, the s-rectangular extension Rec s ( P ) \operatorname{Rec}_{s}(\mathcal{P}) Rec s ( P ) is the smallest s-rectangular set containing P \mathcal{P} P , and the sa-rectangular extension Rec s a ( P ) \operatorname{Rec}_{sa}(\mathcal{P}) Rec s a ( P ) is the smallest sa-rectangular set containing P \mathcal{P} P . Note that obviously the s-rectangular extension of an s-rectangular set is itself, and the sa-rectangular extension of an sa-rectangular set is itself. The extensions play a crucial role for studying time-varying transitions, as shown in the next lemma.
Lemma 4.2. Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be an uncertainty set.
We have P H S A = Rec s a ( P ) H S A \mathcal{P}_{\mathrm{H}}^{\mathrm{SA}} = \operatorname{Rec}_{sa}(\mathcal{P})_{\mathrm{H}}^{\mathrm{SA}} P H SA = Rec s a ( P ) H SA and P H S = Rec s ( P ) H S \mathcal{P}_{\mathrm{H}}^{\mathrm{S}} = \operatorname{Rec}_{s}(\mathcal{P})_{\mathrm{H}}^{\mathrm{S}} P H S = Rec s ( P ) H S .
If additionally P \mathcal{P} P is s-rectangular, then P H S = P H G \mathcal{P}_{\mathrm{H}}^{\mathrm{S}} = \mathcal{P}_{\mathrm{H}}^{\mathrm{G}} P H S = P H G .
If additionally P \mathcal{P} P is sa-rectangular, then P H S A = P H S = P H G \mathcal{P}_{\mathrm{H}}^{\mathrm{SA}} = \mathcal{P}_{\mathrm{H}}^{\mathrm{S}} = \mathcal{P}_{\mathrm{H}}^{\mathrm{G}} P H SA = P H S = P H G .
Proof. 1. This follows from the observation that the s s s -th marginal of Rec s ( P ) \operatorname{Rec}_{s}(\mathcal{P}) Rec s ( P ) is the same as the s s s -th marginal of P \mathcal{P} P , by construction. Since P H S \mathcal{P}_{\mathrm{H}}^{\mathrm{S}} P H S only depends on ( P s ) s ∈ S (P_s)_{s\in\mathcal{S}} ( P s ) s ∈ S we must have that P H S = Rec s ( P ) H S \mathcal{P}_{\mathrm{H}}^{\mathrm{S}} = \operatorname{Rec}_{s}(\mathcal{P})_{\mathrm{H}}^{\mathrm{S}} P H S = Rec s ( P ) H S . The proof for the set P H S A \mathcal{P}_{\mathrm{H}}^{\mathrm{SA}} P H SA is similar.
Let t ∈ N t \in\mathbb{N} t ∈ N , g t ∈ G t − 1 g_t \in\mathcal{G}_{t-1} g t ∈ G t − 1 . For P \mathcal{P} P s-rectangular we get that the constraint “( κ t ( g t , s , a ) ) s , a ∈ P (\kappa_t(g_t,s,a))_{s,a} \in\mathcal{P} ( κ t ( g t , s , a ) ) s , a ∈ P ” in the definition of P H G \mathcal{P}_{\mathrm{H}}^{\mathrm{G}} P H G as in (64) is equivalent to “( κ t ( g t , s , a ) ) a ∈ P s , ∀ s ∈ S (\kappa_t(g_t,s,a))_a \in\mathcal{P}_s, \ \forall\ s \in\mathcal{S} ( κ t ( g t , s , a ) ) a ∈ P s , ∀ s ∈ S ”, which is the constraint appearing in the definition of P H S \mathcal{P}_{\mathrm{H}}^{\mathrm{S}} P H S as in (63) .
This proof is similar to that of the statement right above.
Therefore in the models of time-varying adversaries described by Model 1 and Model 2, it is sufficient to consider sa-rectangular and s-rectangular uncertainty sets. Additionally, for any of the three cases above, we can then ask that the decisions over time are always the same, in which case we say that the adversary policy is stationary and we identify the set of stationary policies with Rec s a ( P ) \operatorname{Rec}_{sa}(\mathcal{P}) Rec s a ( P ) , Rec s ( P ) \operatorname{Rec}_{s}(\mathcal{P}) Rec s ( P ) and P \mathcal{P} P respectively.
Remark 4.1. It is tempting to also define Markovian adversaries as time-varying transition probabilities that only depend on the current states, since we also defined this notion for the decision maker; e.g. for Model 3 we could identify this set as P N \mathcal{P}^{\mathbb{N}} P N , see § 5.8 for some references on these models.
Main results for Model 1 and Model 2
Given a policy of the adversary κ ∈ P H S A \kappa\in\mathcal{P}^{\mathrm{SA}}_{\mathrm{H}} κ ∈ P H SA (which contains the other two classes of history-dependent adversary policies) and a policy π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H of the decision maker, we still get a distribution over the set of infinite trajectories, and therefore we can extend the function P ↦ R ( π , P ) P \mapsto R(\pi,P) P ↦ R ( π , P ) defined on P \mathcal{P} P to κ ↦ R ( π , κ ) \kappa\mapsto R(\pi,\kappa) κ ↦ R ( π , κ ) defined on P H S A \mathcal{P}^{\mathrm{SA}}_{\mathrm{H}} P H SA naturally as
R ( π , κ ) = E π , κ [ ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) ] . R(\pi,\kappa)=\mathbb{E}^{\pi,\kappa}\left[\sum_{t=0}^{+\infty}\gamma^t r(s_t,a_t,s_{t+1})\right]. R ( π , κ ) = E π , κ [ t = 0 ∑ + ∞ γ t r ( s t , a t , s t + 1 ) ] .
The main goal in this section is to understand the properties of the policy evaluation problems
inf κ ∈ P H i R ( π , κ ) \inf_{\kappa\in\mathcal{P}^{i}_{\mathrm{H}}} R(\pi,\kappa) κ ∈ P H i inf R ( π , κ )
and robust MDP problem
sup π ∈ Π H inf κ ∈ P H i R ( π , κ ) \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\kappa\in\mathcal{P}^{i}_{\mathrm{H}}} R(\pi,\kappa) π ∈ Π H sup κ ∈ P H i inf R ( π , κ )
for i ∈ { S A , S , G } i \in\{\mathrm{SA},\mathrm{S},\mathrm{G}\} i ∈ { SA , S , G } . We choose to organize the main results by model of time-varying transitions.
The case of Model 1. We start with Model 1. We focus on the properties of the policy evaluation problem
inf κ ∈ P H S A R ( π , κ ) \inf_{\kappa\in\mathcal{P}^{\mathrm{SA}}_{\mathrm{H}}} R(\pi,\kappa) κ ∈ P H SA inf R ( π , κ )
and robust MDP problem
sup π ∈ Π H inf κ ∈ P H S A R ( π , κ ) . (65) \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\kappa\in\mathcal{P}^{\mathrm{SA}}_{\mathrm{H}}} R(\pi,\kappa). \tag*{(65)} π ∈ Π H sup κ ∈ P H SA inf R ( π , κ ) . (65)
Theorem 4.5 (Robust MDPs with time-varying transitions, Model 1). Let P ⊂ Δ ( S ) S × A \mathcal{P}\subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be compact. The following statements hold.
Policy evaluation. Let π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S . Then the worst-case return of π \pi π against κ ∈ P H S A \kappa\in\mathcal{P}^{\mathrm{SA}}_{\mathrm{H}} κ ∈ P H SA is attained at a stationary transition kernel in the s a sa s a -rectangular extension of P \mathcal{P} P :
inf κ ∈ P H S A R ( π , κ ) = min P ∈ Rec s a ( P ) R ( π , P ) . \inf_{\kappa\in\mathcal{P}^{\mathrm{SA}}_{\mathrm{H}}}R(\pi,\kappa)=\min_{P\in\operatorname{Rec}_{sa}(\mathcal{P})}R(\pi,P). κ ∈ P H SA inf R ( π , κ ) = P ∈ Rec s a ( P ) min R ( π , P ) . Optimal policies. An optimal policy for the optimization problem (4.15) can always be chosen stationary:
sup π ∈ Π H inf κ ∈ P H S A R ( π , κ ) = sup π ∈ Π S inf κ ∈ P H S A R ( π , κ ) . \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\kappa\in\mathcal{P}^{\mathrm{SA}}_{\mathrm{H}}}R(\pi,\kappa)=\sup_{\pi\in\Pi_{\mathrm{S}}}\inf_{\kappa\in\mathcal{P}^{\mathrm{SA}}_{\mathrm{H}}}R(\pi,\kappa). π ∈ Π H sup κ ∈ P H SA inf R ( π , κ ) = π ∈ Π S sup κ ∈ P H SA inf R ( π , κ ) . In fact, a stationary uniformly optimal policy always exists for (4.15) and it can be chosen to be also uniformly optimal for the robust MDP problem over the stationary s a sa s a -rectangular extension of P \mathcal{P} P , i.e., sup π ∈ Π H inf κ ∈ P H S A R ( π , κ ) = max π ∈ Π S min P ∈ Rec s a ( P ) R ( π , P ) \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\kappa\in\mathcal{P}^{\mathrm{SA}}_{\mathrm{H}}}R(\pi,\kappa)=\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\operatorname{Rec}_{sa}(\mathcal{P})}R(\pi,P) sup π ∈ Π H inf κ ∈ P H SA R ( π , κ ) = max π ∈ Π S min P ∈ Rec s a ( P ) R ( π , P ) .
The above theorem is an immediate consequence of Lemma 4.2 and the following proposition, specialized for sa-rectangular uncertainty sets (note that we don’t need convexity here, similarly as in Section 3.4).
Proposition 4.4 (sa-rectangular time-varying transitions). Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be sa-rectangular compact. The following statements hold.
Policy evaluation. Let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S . Then the worst-case return of π \pi π against time-varying transitions in P H S A \mathcal{P}_{\mathrm{H}}^{\mathrm{SA}} P H SA is attained at a stationary transition kernel in the sa-rectangular extension of P \mathcal{P} P :
inf κ ∈ P H S A R ( π , κ ) = min P ∈ P R ( π , P ) . \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{SA}}} R(\pi,\kappa) = \min_{P \in\mathcal{P}} R(\pi,P). κ ∈ P H SA inf R ( π , κ ) = P ∈ P min R ( π , P ) . Optimal policies. An optimal policy for the optimization problem (4.15) can always be chosen stationary:
sup π ∈ Π H inf κ ∈ P H S A R ( π , κ ) = max π ∈ Π S inf κ ∈ P H S A R ( π , κ ) \sup_{\pi\in\Pi_{\mathrm{H}}} \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{SA}}} R(\pi,\kappa) = \max_{\pi\in\Pi_{\mathcal{S}}} \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{SA}}} R(\pi,\kappa) π ∈ Π H sup κ ∈ P H SA inf R ( π , κ ) = π ∈ Π S max κ ∈ P H SA inf R ( π , κ ) In fact, a stationary uniformly optimal policy always exists for (4.15) and it can be chosen to be also uniformly optimal for the robust MDP problem over the stationary transition probabilities, i.e., sup π ∈ Π H inf κ ∈ P H S A R ( π , κ ) = max π ∈ Π S min P ∈ P R ( π , P ) \sup_{\pi\in\Pi_{\mathrm{H}}} \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{SA}}} R(\pi,\kappa) = \max_{\pi\in\Pi_{\mathcal{S}}} \min_{P \in\mathcal{P}} R(\pi,P) sup π ∈ Π H inf κ ∈ P H SA R ( π , κ ) = max π ∈ Π S min P ∈ P R ( π , P ) .
Proof. 1. We can interpret inf κ ∈ P H S A R ( π , κ ) \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{SA}}} R(\pi,\kappa) inf κ ∈ P H SA R ( π , κ ) as an adversarial MDP where the set of states is S × A \mathcal{S} \times\mathcal{A} S × A and the compact set of actions available at state ( s , a ) (s,a) ( s , a ) is P s a \mathcal{P}_{sa} P s a . The instantaneous reward for choosing action P ( s , a ) ∈ P s a P(s,a) \in\mathcal{P}_{sa} P ( s , a ) ∈ P s a at ( s , a ) (s,a) ( s , a ) is P ( s , a ) ⊤ r ( s , a ) P(s,a)^{\top}r(s,a) P ( s , a ) ⊤ r ( s , a ) (independent of the next state ( s ′ , a ′ ) (s',a') ( s ′ , a ′ ) ) and the transition probabilities from ( s , a ) (s,a) ( s , a ) to a next state ( s ′ , a ′ ) (s',a') ( s ′ , a ′ ) is P ( s , a , s ′ ) π ( s ′ , a ′ ) ∈ [ 0 , 1 ] P(s,a,s')\pi(s',a') \in[0,1] P ( s , a , s ′ ) π ( s ′ , a ′ ) ∈ [ 0 , 1 ] (at this point the reader should notice the similarity with the adversarial MDP defined in Section 3.2 for s-rectangular model). We conclude that a stationary deterministic optimal adversarial policy exists, i.e., inf κ ∈ P H S A R ( π , κ ) = min P ∈ P R ( π , P ) \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{SA}}} R(\pi,\kappa) = \min_{P \in\mathcal{P}} R(\pi,P) inf κ ∈ P H SA R ( π , κ ) = min P ∈ P R ( π , P ) .
The proof is very close to the proof for the existence of stationary optimal policies for stationary kernels (beginning of Section 3.3.2). We have
sup π ∈ Π H inf P ∈ P R ( π , P ) ≤ inf P ∈ P sup π ∈ Π H R ( π , P ) ≤ inf P ∈ P max π ∈ Π S R ( π , P ) = max π ∈ Π S min P ∈ P R ( π , P ) = max π ∈ Π S inf κ ∈ P H S A R ( π , κ ) ≤ sup π ∈ Π H inf κ ∈ P H S A R ( π , κ ) \begin{aligned}
\sup_{\pi\in\Pi_{\mathrm{H}}} \inf_{P \in\mathcal{P}} R(\pi,P) \le\inf_{P \in\mathcal{P}} \sup_{\pi\in\Pi_{\mathrm{H}}} R(\pi,P) \\
&\le\inf_{P \in\mathcal{P}} \max_{\pi\in\Pi_{\mathcal{S}}} R(\pi,P) \\
&= \max_{\pi\in\Pi_{\mathcal{S}}} \min_{P \in\mathcal{P}} R(\pi,P) \\
&= \max_{\pi\in\Pi_{\mathcal{S}}} \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{SA}}} R(\pi,\kappa) \\
&\le\sup_{\pi\in\Pi_{\mathrm{H}}} \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{SA}}} R(\pi,\kappa)
\end{aligned} π ∈ Π H sup P ∈ P inf R ( π , P ) ≤ P ∈ P inf π ∈ Π H sup R ( π , P ) ≤ P ∈ P inf π ∈ Π S max R ( π , P ) = π ∈ Π S max P ∈ P min R ( π , P ) = π ∈ Π S max κ ∈ P H SA inf R ( π , κ ) ≤ π ∈ Π H sup κ ∈ P H SA inf R ( π , κ )
where the first inequality is weak duality, the second inequality is because the inner maximization is an MDP, the first equality is strong duality for sa-rectangular compact uncertainty sets as in Section 3.4, the second equality is via the first statement of this proposition, and the third is by Π S ⊂ Π H \Pi_{\mathrm{S}} \subset\Pi_{\mathrm{H}} Π S ⊂ Π H . Since the last term above is always smaller than the first term above (identifying P \mathcal{P} P with constant maps in P H S A \mathcal{P}_{\mathrm{H}}^{\mathrm{SA}} P H SA ), we get that all terms above are equal, so that a stationary optimal policy exists for (4.15) and can be computed as a solution to max π ∈ Π S inf P ∈ P R ( π , P ) \max_{\pi\in\Pi_{\mathrm{S}}} \inf_{P \in\mathcal{P}} R(\pi, P) max π ∈ Π S inf P ∈ P R ( π , P ) . This additionally shows that a stationary uniformly optimal policy exists (again via the results for sa-rectangular robust MDPs from Section 3.4).
The case of Model 2
We now focus on Model 2. We focus on the properties of the policy evaluation problem
inf κ ∈ P H S R ( π , κ ) \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{S}}} R(\pi,\kappa) κ ∈ P H S inf R ( π , κ )
and robust MDP problem
sup π ∈ Π H inf κ ∈ P H S R ( π , κ ) . (66) \sup_{\pi\in\Pi_{\mathrm{H}}} \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{S}}} R(\pi,\kappa). \tag*{(66)} π ∈ Π H sup κ ∈ P H S inf R ( π , κ ) . (66)
The analogous statements for robust MDPs with time-varying transition probabilities, as in Model 2, are given next.
Theorem 4.6 (Robust MDPs with time-varying transitions, Model 2). Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S} \times\mathcal{A}} P ⊂ Δ ( S ) S × A be compact. The following statements hold.
Policy evaluation. Let π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S . Then the worst-case return of π \pi π against time-varying transitions in P H S \mathcal{P}_{\mathrm{H}}^{\mathrm{S}} P H S is attained at a stationary transition kernel in the s s s -rectangular extension of P \mathcal{P} P :
inf κ ∈ P H S R ( π , κ ) = min P ∈ Rec S ( P ) R ( π , P ) . \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{S}}} R(\pi,\kappa) = \min_{P \in\operatorname{Rec}_{\mathrm{S}}(\mathcal{P})} R(\pi,P). κ ∈ P H S inf R ( π , κ ) = P ∈ Rec S ( P ) min R ( π , P ) . Optimal policies. An optimal policy for the optimization problem (4.16) can always be chosen stationary:
sup π ∈ Π H inf κ ∈ P H S R ( π , κ ) = max π ∈ Π S inf κ ∈ P H S R ( π , κ ) \sup_{\pi\in\Pi_{\mathrm{H}}} \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{S}}} R(\pi,\kappa) = \max_{\pi\in\Pi_{\mathrm{S}}} \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{S}}} R(\pi,\kappa) π ∈ Π H sup κ ∈ P H S inf R ( π , κ ) = π ∈ Π S max κ ∈ P H S inf R ( π , κ ) In fact, a stationary uniformly optimal policy always exists for (4.16), and it can be chosen as a best policy among stationary policies for the robust MDP problem over the stationary s s s -rectangular extension of P \mathcal{P} P , i.e.,
sup π ∈ Π H inf κ ∈ P H S R ( π , κ ) = max π ∈ Π S min P ∈ Rec S ( P ) R ( π , P ) . \sup_{\pi\in\Pi_{\mathrm{H}}} \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{S}}} R(\pi,\kappa) = \max_{\pi\in\Pi_{\mathrm{S}}} \min_{P \in\operatorname{Rec}_{\mathrm{S}}(\mathcal{P})} R(\pi,P). π ∈ Π H sup κ ∈ P H S inf R ( π , κ ) = π ∈ Π S max P ∈ Rec S ( P ) min R ( π , P ) . If additionally Rec S ( P ) \operatorname{Rec}_{\mathrm{S}}(\mathcal{P}) Rec S ( P ) is convex, we additionally get that
max π ∈ Π S min P ∈ Rec S ( P ) R ( π , P ) = sup π ∈ Π H min P ∈ Rec S ( P ) R ( π , P ) \max_{\pi\in\Pi_{\mathrm{S}}} \min_{P \in\operatorname{Rec}_{\mathrm{S}}(\mathcal{P})} R(\pi,P) = \sup_{\pi\in\Pi_{\mathrm{H}}} \min_{P \in\operatorname{Rec}_{\mathrm{S}}(\mathcal{P})} R(\pi,P) π ∈ Π S max P ∈ Rec S ( P ) min R ( π , P ) = π ∈ Π H sup P ∈ Rec S ( P ) min R ( π , P ) and we can conclude that a stationary uniformly optimal policy for (4.16) can be chosen as a stationary uniformly optimal policy for the robust MDP problem with stationary transitions over Rec S ( P ) \operatorname{Rec}_{\mathrm{S}}(\mathcal{P}) Rec S ( P ) .
Proof. 1. Lemma 4.2 shows that inf κ ∈ P H S R ( π , κ ) = inf κ ∈ Rec S ( P ) R ( π , κ ) \inf_{\kappa\in\mathcal{P}^{S}_{H}} R(\pi,\kappa)=\inf_{\kappa\in\operatorname{Rec}_{S}(\mathcal{P})} R(\pi,\kappa) inf κ ∈ P H S R ( π , κ ) = inf κ ∈ Rec S ( P ) R ( π , κ ) and we can interpret the left-hand side as an adversarial MDP (see § 3.2 ) with a finite set of states, compact action sets, continuous rewards and transitions. Therefore inf κ ∈ Rec S ( P ) R ( π , κ ) = inf P ∈ Rec S ( P ) R ( π , P ) \inf_{\kappa\in\operatorname{Rec}_{S}(\mathcal{P})} R(\pi,\kappa)=\inf_{P\in\operatorname{Rec}_{S}(\mathcal{P})} R(\pi,P) inf κ ∈ Rec S ( P ) R ( π , κ ) = inf P ∈ Rec S ( P ) R ( π , P ) .
The proof for the existence of stationary optimal policies is direct. Define T T T the robust Bellman operator as in (26) . Since T T T is defined componentwise and each s s s -component only depends on the s s s -th marginal of P \mathcal{P} P , we get that the set P \mathcal{P} P and the set Rec S ( P ) \operatorname{Rec}_{S}(\mathcal{P}) Rec S ( P ) induce the same robust Bellman operator. Let v ⋆ v^{\star} v ⋆ be its fixed point, and π ⋆ ∈ Π S \pi^{\star}\in\Pi_{S} π ⋆ ∈ Π S such that T v ⋆ = T π ⋆ v ⋆ Tv^{\star}=T^{\pi^{\star}}v^{\star} T v ⋆ = T π ⋆ v ⋆ . For any κ ∈ P H S \kappa\in\mathcal{P}^{S}_{H} κ ∈ P H S , t ∈ N t\in\mathbb{N} t ∈ N and finite history h t ∈ ( S × A ) t × S h_{t}\in(\mathcal{S}\times\mathcal{A})^{t}\times\mathcal{S} h t ∈ ( S × A ) t × S , we get that E π ⋆ , κ [ r ( s t , a t , s t + 1 ) + γ v ⋆ ( s t + 1 ) ∣ h t ] ≥ T v ⋆ ( s t ) = v ⋆ ( s t ) \mathbb{E}^{\pi^{\star},\kappa}[r(s_{t},a_{t},s_{t+1})+\gamma v^{\star}(s_{t+1})\mid h_{t}]\geq Tv^{\star}(s_{t})=v^{\star}(s_{t}) E π ⋆ , κ [ r ( s t , a t , s t + 1 ) + γ v ⋆ ( s t + 1 ) ∣ h t ] ≥ T v ⋆ ( s t ) = v ⋆ ( s t ) , so by induction and taking the limit (similarly to the proof of Theorem 1.1), we get that R ( π ⋆ , κ ) ≥ p 0 T v ⋆ R(\pi^{\star},\kappa)\geq p_{0}^{\mathsf{T}}v^{\star} R ( π ⋆ , κ ) ≥ p 0 T v ⋆ , i.e. we have shown inf κ ∈ P H S R ( π ⋆ , κ ) ≥ p 0 T v ⋆ \inf_{\kappa\in\mathcal{P}^{S}_{H}}R(\pi^{\star},\kappa)\geq p_{0}^{\mathsf{T}}v^{\star} inf κ ∈ P H S R ( π ⋆ , κ ) ≥ p 0 T v ⋆ . Conversely, for π ∈ Π S \pi\in\Pi_{S} π ∈ Π S , the same argument combined with v ⋆ = T v ⋆ = T π ⋆ v ⋆ ≥ T π v ⋆ v^{\star}=Tv^{\star}=T^{\pi^{\star}}v^{\star}\geq T^{\pi}v^{\star} v ⋆ = T v ⋆ = T π ⋆ v ⋆ ≥ T π v ⋆ yields inf κ ∈ P H S R ( π , κ ) ≤ p 0 T v ⋆ \inf_{\kappa\in\mathcal{P}^{S}_{H}}R(\pi,\kappa)\leq p_{0}^{\mathsf{T}}v^{\star} inf κ ∈ P H S R ( π , κ ) ≤ p 0 T v ⋆ . Therefore π ⋆ ∈ Π S \pi^{\star}\in\Pi_{S} π ⋆ ∈ Π S is an optimal policy for (4.16), and it has been constructed independently of p 0 p_{0} p 0 hence it is uniformly optimal for (4.16). The fact that π ⋆ \pi^{\star} π ⋆ is additionally optimal in max π ∈ Π S min P ∈ Rec S ( P ) R ( π , P ) \max_{\pi\in\Pi_{S}}\min_{P\in\operatorname{Rec}_{S}(\mathcal{P})}R(\pi,P) max π ∈ Π S min P ∈ Rec S ( P ) R ( π , P ) has been shown in § 3.3 and follows easily from T v ⋆ = T π ⋆ v ⋆ Tv^{\star}=T^{\pi^{\star}}v^{\star} T v ⋆ = T π ⋆ v ⋆ .
Finally, we know that for s-rectangular convex compact set an optimal policy for (2.1) can be chosen stationary, which concludes the proof. Note that convexity of P \mathcal{P} P implies convexity of Rec S ( P ) \operatorname{Rec}_{S}(\mathcal{P}) Rec S ( P ) .
We emphasize that without the convexity assumption on P \mathcal{P} P , optimal policies for (4.16) against time-varying transitions and optimal policies for (2.1) against stationary transitions in Rec S ( P ) \operatorname{Rec}_{S}(\mathcal{P}) Rec S ( P ) may differ (indeed the latter set of optimal policies may require history-dependent policies, whereas the former set of optimal policies always contains a stationary policy). The next example illustrates a decision problem with P H S \mathcal{P}^{S}_{H} P H S .
Example 4.5. Consider the robust MDP instance from Example 1.1, continued in Section 2.3 (right after Proposition 2.5). There P = { P 1 , P 2 } \mathcal{P}=\{P^{1},P^{2}\} P = { P 1 , P 2 } is s-rectangular but not convex. We saw that in this instance sup π ∈ Π H min P ∈ P R ( π , P ) \sup_{\pi\in\Pi_{H}}\min_{P\in\mathcal{P}}R(\pi,P) sup π ∈ Π H min P ∈ P R ( π , P ) is in general solved by non-stationary policies, which can exploit the information revealed after period t = 0 t=0 t = 0 to correctly guess the transition probabilities (P 1 P^{1} P 1 or P 2 P^{2} P 2 ) chosen by the adversary. In contrast Theorem 4.6 tells us that an optimal policy can be chosen stationary in (4.16). We analyze this specific instance in more detail. Rewards are only obtained at odd periods. For π ∈ Π H \pi\in\Pi_{H} π ∈ Π H and t ∈ N t\in\mathbb{N} t ∈ N let x t ∈ [ 0 , 1 ] x_{t}\in[0,1] x t ∈ [ 0 , 1 ] be the probability that the decision maker chooses action a 1 a_{1} a 1 at period 2 t 2t 2 t and for κ ∈ P H S \kappa\in\mathcal{P}^{S}_{H} κ ∈ P H S let p t ∈ { 0 , 1 } p_{t}\in\{0,1\} p t ∈ { 0 , 1 } be the transition probabilities chosen by the adversary for period 2 t 2t 2 t ; in principle,* x t x_t x t and p t p_t p t depend on h 2 t h_{2t} h 2 t the history up to period 2 t 2t 2 t . Then
R ( π , κ ) = E π , κ [ ∑ t = 0 + ∞ γ t r ( s t , a t , s t + 1 ) ] = ∑ t = 0 + ∞ γ 2 t + 1 E π , κ [ r ( s 2 t + 1 , a 2 t + 1 , s 2 t + 2 ) ] = ∑ t = 0 + ∞ γ 2 t + 1 E π , κ [ x t p t + ( 1 − x t ) ( 1 − p t ) ] . \begin{aligned}
R(\pi,\kappa) &= \mathbb{E}^{\pi,\kappa}\left[\sum_{t=0}^{+\infty}\gamma^t r(s_t,a_t,s_{t+1})\right] \\
&= \sum_{t=0}^{+\infty}\gamma^{2t+1}\mathbb{E}^{\pi,\kappa}\left[r(s_{2t+1},a_{2t+1},s_{2t+2})\right] \\
&= \sum_{t=0}^{+\infty}\gamma^{2t+1}\mathbb{E}^{\pi,\kappa}\left[x_t p_t+(1-x_t)(1-p_t)\right].
\end{aligned} R ( π , κ ) = E π , κ [ t = 0 ∑ + ∞ γ t r ( s t , a t , s t + 1 ) ] = t = 0 ∑ + ∞ γ 2 t + 1 E π , κ [ r ( s 2 t + 1 , a 2 t + 1 , s 2 t + 2 ) ] = t = 0 ∑ + ∞ γ 2 t + 1 E π , κ [ x t p t + ( 1 − x t ) ( 1 − p t ) ] .
Minimizing with respect to κ \kappa κ we get that min κ ∈ P H S R ( π , κ ) \min_{\kappa\in\mathcal{P}^{\mathrm{S}}_{\mathrm{H}}}R(\pi,\kappa) min κ ∈ P H S R ( π , κ ) , where in principle the adversary can choose p t = 0 p_t=0 p t = 0 at some period 2 t 2t 2 t and p t ′ = 1 p_{t'}=1 p t ′ = 1 at some other period 2 t ′ 2t' 2 t ′ , is maximized with x t = 1 / 2 x_t=1/2 x t = 1/2 for all t ∈ N t\in\mathbb{N} t ∈ N so that an optimal policy can be chosen stationary in this example.
Note that if optimal policies for (20) against stationary transitions can be chosen stationary, then in fact these stationary optimal policies coincide with stationary optimal policies for (66) against time-varying transitions. An intermediate conclusion regarding Model 1 and Model 2 is as follows:
For both Model 1 and Model 2, all uncertainty sets are equivalent to their rectangular counterparts, and optimal policies can be chosen stationary (there is no value in learning with history-dependent policies since the transition probabilities may be changing with time), which then yields stationary transition probabilities as worst-cases;
For Model 1 we additionally get that optimal policies for time-varying transitions coincide with optimal policies for stationary transitions over their sa-rectangular extension (as defined in (20) ), whereas for Model 2 the corresponding statement requires a convexity assumption, a contrast that we already encountered in Chapter 3.
Perhaps surprisingly, this shows that with time-varying transition probabilities as Model 1 or Model 2, we actually obtain more favorable properties than for the original robust MDP problem with stationary transitions as in (20) !
The case of Model 3 and connection with POMDPs
We are left with considering time-varying transitions as in Model 3. The conclusions differ depending on the rectangularity properties of P \mathcal{P} P :
When P \mathcal{P} P is sa-rectangular compact, from Lemma 4.2 we know that P H G = P H S = P H S A \mathcal{P}^{\mathrm{G}}_{\mathrm{H}}=\mathcal{P}^{\mathrm{S}}_{\mathrm{H}}=\mathcal{P}^{\mathrm{SA}}_{\mathrm{H}} P H G = P H S = P H SA hence the conclusions of Theorem 4.5 apply verbatim (we just need to replace P H S A \mathcal{P}^{\mathrm{SA}}_{\mathrm{H}} P H SA by P H G \mathcal{P}^{\mathrm{G}}_{\mathrm{H}} P H G in the statements).
When P \mathcal{P} P is s-rectangular compact, from Lemma 4.2 we know that P H G = P H S \mathcal{P}^{\mathrm{G}}_{\mathrm{H}}=\mathcal{P}^{\mathrm{S}}_{\mathrm{H}} P H G = P H S so that the conclusions from Theorem 4.6 apply verbatim (we just need to replace P H S \mathcal{P}^{\mathrm{S}}_{\mathrm{H}} P H S by P H G \mathcal{P}^{\mathrm{G}}_{\mathrm{H}} P H G in the statements).
Interesting properties occur when P \mathcal{P} P is not s-rectangular. We start with policy evaluation. We first show that for non-rectangular uncertainty sets the worst-case return of a stationary policy may be achieved at a non-stationary adversary, i.e. we may have inf κ ∈ P H G R ( π , κ ) < min P ∈ P R ( π , P ) \inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{G}} R(\pi,\kappa) < \min_{P \in\mathcal{P}} R(\pi,P) inf κ ∈ P H G R ( π , κ ) < min P ∈ P R ( π , P ) .
Example 4.6. Consider an MDP instance with two states s 0 , s 1 s_0, s_1 s 0 , s 1 and a single action a 1 a_1 a 1 , with γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) . Assume that we start in s 0 s_0 s 0 with probability 1. The instantaneous rewards are r ( s , a , s ′ ) = 1 r(s,a,s') = 1 r ( s , a , s ′ ) = 1 if s = s ′ s=s' s = s ′ and 0 otherwise, i.e., we obtain a reward of 1 if we don’t change states. Consider the uncertainty set P = conv ( { P 1 , P 2 } ) \mathcal{P} = \operatorname{conv}(\{P^1,P^2\}) P = conv ({ P 1 , P 2 }) where P 1 P^1 P 1 sends both s 0 , s 1 s_0,s_1 s 0 , s 1 to s 0 s_0 s 0 (we can identify P 1 P^1 P 1 with the transition matrix ( 1 0 1 0 ) \begin{pmatrix} 1 & 0 \\ 1 & 0 \end{pmatrix} ( 1 1 0 0 ) ) while P 2 P^2 P 2 sends both s 0 , s 1 s_0,s_1 s 0 , s 1 to s 1 s_1 s 1 (we can identify P 2 P^2 P 2 with the transition matrix ( 0 1 0 1 ) \begin{pmatrix} 0 & 1 \\ 0 & 1 \end{pmatrix} ( 0 0 1 1 ) ). Note that P \mathcal{P} P is not s-rectangular but it is convex. Then the adversary can choose the following feasible policy in P H G \mathcal{P}_{\mathrm{H}}^{G} P H G : choose P 2 P^2 P 2 at even periods and P 1 P^1 P 1 at odd periods. This way we ensure that the sequence of visited states alternates between s 0 s_0 s 0 and s 1 s_1 s 1 , so that the return is 0. In contrast if the adversary always chooses the same transition probabilities in P \mathcal{P} P then the return is positive. Hence in this example the worst-case return of the policy { s 0 , s 1 } → { a 1 } \{s_0,s_1\} \to\{a_1\} { s 0 , s 1 } → { a 1 } over κ ∈ P H G \kappa\in\mathcal{P}_{\mathrm{H}}^{G} κ ∈ P H G is 0 and attained at a non-stationary adversary policy.
From the adversary’s standpoint, the optimization problem resembles a Partially Observable MDP (POMDP), which is a class of sequential decision-making problems where the decision maker does not observe the current state s t s_t s t fully but only a “masked” version (the so-called “observation” o t o_t o t ). The decision maker then forms a certain belief b t ∈ Δ ( S ) b_t \in\Delta(\mathcal{S}) b t ∈ Δ ( S ) of what is the current state at period t t t and can act accordingly. Here we are in the case where the information is delayed, i.e., the state s t s_t s t is revealed to the adversary only right before period t + 1 t+1 t + 1 (after the decision maker has chosen a t a_t a t and after the adversary has chosen P t P_t P t ). The next proposition shows how to apply classical results on POMDPs to our setup. We denote C ( Δ ( S ) , R ) C(\Delta(\mathcal{S}),\mathbb{R}) C ( Δ ( S ) , R ) the set of continuous maps from Δ ( S ) → R \Delta(\mathcal{S}) \to\mathbb{R} Δ ( S ) → R . For π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S and P ∈ P P \in\mathcal{P} P ∈ P we write B π , P : C ( Δ ( S ) , R ) → C ( Δ ( S ) , R ) B^{\pi,P}: C(\Delta(\mathcal{S}),\mathbb{R}) \to C(\Delta(\mathcal{S}),\mathbb{R}) B π , P : C ( Δ ( S ) , R ) → C ( Δ ( S ) , R ) the operator defined as, for w ∈ C ( Δ ( S ) , R ) w \in C(\Delta(\mathcal{S}),\mathbb{R}) w ∈ C ( Δ ( S ) , R ) ,
( B π , P w ) ( b ) = ∑ s ∈ S b ( s ) ∑ a ∈ A π ( s , a ) ( P ( s , a ) ⊤ r ( s , a ) + γ w ( P ( s , a ) ) ) , ∀ b ∈ Δ ( S ) . \left(B^{\pi,P}w\right)(b) = \sum_{s \in\mathcal{S}} b(s) \sum_{a \in\mathcal{A}} \pi(s,a)\left(P(s,a)^{\top}r(s,a) + \gamma w(P(s,a))\right), \ \forall b \in\Delta(\mathcal{S}). ( B π , P w ) ( b ) = s ∈ S ∑ b ( s ) a ∈ A ∑ π ( s , a ) ( P ( s , a ) ⊤ r ( s , a ) + γ w ( P ( s , a )) ) , ∀ b ∈ Δ ( S ) .
In the equation above we interpret b ∈ Δ ( S ) b \in\Delta(\mathcal{S}) b ∈ Δ ( S ) as a certain belief of what will be the current state at the period of decision. Hence the right-hand side is the sum of the reward obtained in one period with a continuation value of w ∈ C ( Δ ( S ) , R ) w \in C(\Delta(\mathcal{S}),\mathbb{R}) w ∈ C ( Δ ( S ) , R ) . Hence the operator B π , P B^{\pi,P} B π , P plays the same role as the operator T π , P T^{\pi,P} T π , P defined in (1.11) in Chapter 1, and Δ ( S ) \Delta(\mathcal{S}) Δ ( S ) plays the role of S \mathcal{S} S (since the adversary can only form beliefs about the current state). Note that we use P ( s , a ) P(s,a) P ( s , a ) as the argument for the next belief because P ( s , a ) P(s,a) P ( s , a ) is the distribution of s t + 1 s_{t+1} s t + 1 given a current state-action pair ( s , a ) (s,a) ( s , a ) , and the adversary, when choosing P t + 1 P^{t+1} P t + 1 for period t + 1 t+1 t + 1 , has observed s t s_t s t and a t a_t a t (following the definition of Model 3).
Proposition 4.5. Let P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be a compact uncertainty set. Let π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S . Let B π : C ( Δ ( S ) , R ) → C ( Δ ( S ) , R ) B^\pi: C(\Delta(\mathcal{S}),\mathbb{R}) \to C(\Delta(\mathcal{S}),\mathbb{R}) B π : C ( Δ ( S ) , R ) → C ( Δ ( S ) , R ) be the operator defined as, for w ∈ C ( Δ ( S ) , R ) w \in C(\Delta(\mathcal{S}),\mathbb{R}) w ∈ C ( Δ ( S ) , R ) ,
( B π w ) ( b ) = min P ∈ P ( B π , P w ) ( b ) , ∀ b ∈ Δ ( S ) . \left(B^\pi w\right)(b) = \min_{P \in\mathcal{P}} \left(B^{\pi,P}w\right)(b), \ \forall b \in\Delta(\mathcal{S}). ( B π w ) ( b ) = P ∈ P min ( B π , P w ) ( b ) , ∀ b ∈ Δ ( S ) .
The following statements hold:
The operator B π B^\pi B π is a monotone contraction for ∥ ⋅ ∥ ∞ \|\cdot\|_\infty ∥ ⋅ ∥ ∞ on C ( Δ ( S ) , R ) C(\Delta(S),\mathbb{R}) C ( Δ ( S ) , R ) , with unique fixed point w π ∈ C ( Δ ( S ) , R ) w^\pi\in C(\Delta(S),\mathbb{R}) w π ∈ C ( Δ ( S ) , R ) .
We have
inf κ ∈ P H G R ( π , κ ) = w π ( p 0 ) \inf_{\kappa\in\mathcal{P}^{G}_{H}} R(\pi,\kappa)=w^\pi(p_0) κ ∈ P H G inf R ( π , κ ) = w π ( p 0 )
and an optimal policy for the adversary can be chosen stationary in the belief b t : = P t − 1 π ( s t − 1 , a t − 1 ) ∈ Δ ( S ) b_t:=P^{\pi}_{t-1}(s_{t-1},a_{t-1})\in\Delta(S) b t := P t − 1 π ( s t − 1 , a t − 1 ) ∈ Δ ( S ) at each t ≥ 1 t\geq1 t ≥ 1 and b 0 = p 0 b_0=p_0 b 0 = p 0 .
Proof. 1. The proof that B π B^\pi B π is a monotone contraction follows the same lines as for these properties for T π T^\pi T π . The fact that B π w B^\pi w B π w is always a continuous function (given w ∈ C ( Δ ( S ) , R ) w\in C(\Delta(S),\mathbb{R}) w ∈ C ( Δ ( S ) , R ) ) follows from the continuity of ( b , P ) ↦ ( B π , P w ) ( b ) (b,P)\mapsto(B^{\pi,P}w)(b) ( b , P ) ↦ ( B π , P w ) ( b ) . Now C ( Δ ( S ) , R ) C(\Delta(S),\mathbb{R}) C ( Δ ( S ) , R ) is complete for ∥ ⋅ ∥ ∞ \|\cdot\|_\infty ∥ ⋅ ∥ ∞ hence Banach’s fixed point theorem gives the existence and uniqueness of the fixed point w π ∈ C ( Δ ( S ) , R ) w^\pi\in C(\Delta(S),\mathbb{R}) w π ∈ C ( Δ ( S ) , R ) .
From the fixed point equation on w π w^\pi w π and the definition of B π B^\pi B π we get that w π ( b t ) ≤ ( B π , P t w π ) ( b t ) w^\pi(b_t)\leq(B^{\pi,P^t}w^\pi)(b_t) w π ( b t ) ≤ ( B π , P t w π ) ( b t ) for any admissible P t ∈ P P^t\in\mathcal{P} P t ∈ P . That is, for any g t ∈ G t − 1 g_t\in\mathcal{G}_{t-1} g t ∈ G t − 1 and κ ∈ P H G \kappa\in\mathcal{P}^{G}_{H} κ ∈ P H G we get
w π ( b t ) ≤ E π , κ [ r ( s t , a t , s t + 1 ) + γ w π ( b t + 1 ) ∣ g t ] . w^\pi(b_t)\leq\mathbb{E}^{\pi,\kappa}\left[r(s_t,a_t,s_{t+1})+\gamma w^\pi(b_{t+1})\mid g_t\right]. w π ( b t ) ≤ E π , κ [ r ( s t , a t , s t + 1 ) + γ w π ( b t + 1 ) ∣ g t ] .
By induction we get
w π ( p 0 ) ≤ E π , κ [ ∑ t ′ = 0 t γ t ′ r ( s t ′ , a t ′ , s t ′ + 1 ) + γ t + 1 w π ( b t + 1 ) ] . w^\pi(p_0)\leq\mathbb{E}^{\pi,\kappa}\left[\sum_{t'=0}^{t}\gamma^{t'}r(s_{t'},a_{t'},s_{t'+1})+\gamma^{t+1}w^\pi(b_{t+1})\right]. w π ( p 0 ) ≤ E π , κ [ t ′ = 0 ∑ t γ t ′ r ( s t ′ , a t ′ , s t ′ + 1 ) + γ t + 1 w π ( b t + 1 ) ] .
Since w π w^\pi w π is a bounded function, taking the limit as t → + ∞ t\to+\infty t → + ∞ yields that for any κ ∈ P H G \kappa\in\mathcal{P}^{G}_{H} κ ∈ P H G we have w π ( p 0 ) ≤ R ( π , κ ) w^\pi(p_0)\leq R(\pi,\kappa) w π ( p 0 ) ≤ R ( π , κ ) , hence w π ( p 0 ) ≤ inf κ ∈ P H G R ( π , κ ) w^\pi(p_0)\leq\inf_{\kappa\in\mathcal{P}^{G}_{H}}R(\pi,\kappa) w π ( p 0 ) ≤ inf κ ∈ P H G R ( π , κ ) .
To show the converse inequality we just choose P ⋆ ( b ) ∈ arg min P ∈ P ( B π , P w π ) ( b ) P^\star(b)\in\arg\min_{P\in\mathcal{P}}(B^{\pi,P}w^\pi)(b) P ⋆ ( b ) ∈ arg min P ∈ P ( B π , P w π ) ( b ) for any belief b ∈ Δ ( S ) b\in\Delta(S) b ∈ Δ ( S ) and construct κ ⋆ ∈ P H G \kappa^\star\in\mathcal{P}^{G}_{H} κ ⋆ ∈ P H G accordingly such that for any g t = ( s 0 , a 0 , … , s t − 1 , a t − 1 ) g_t=(s_0,a_0,\ldots,s_{t-1},a_{t-1}) g t = ( s 0 , a 0 , … , s t − 1 , a t − 1 ) we have, ( κ t ⋆ ( g t , s , a ) ) s , a = P ⋆ ( b t ) ∈ P (\kappa^\star_t(g_t,s,a))_{s,a}=P^\star(b_t)\in\mathcal{P} ( κ t ⋆ ( g t , s , a ) ) s , a = P ⋆ ( b t ) ∈ P , which concludes the proof. □ \square □
The next example continues the analysis of Example 4.6 in light of the previous proposition.
Example 4.6 continued. Consider the setting of Example 4.6. Since we start at s 0 s_0 s 0 with probability 1 the belief for period 0 is b 0 = ( 1 , 0 ) b_0=(1,0) b 0 = ( 1 , 0 ) , and for b = ( b ( s 0 ) , b ( s 1 ) ) ∈ Δ ( S ) b=(b(s_0),b(s_1))\in\Delta(S) b = ( b ( s 0 ) , b ( s 1 )) ∈ Δ ( S ) the belief of where the current state is, we have
( B w ) ( b ) = min λ ∈ [ 0 , 1 ] b ( s 0 ) λ + b ( s 1 ) ( 1 − λ ) + γ w ( ( λ , 1 − λ ) ) . (Bw)(b)=\min_{\lambda\in[0,1]} b(s_0)\lambda+b(s_1)(1-\lambda)+\gamma w((\lambda,1-\lambda)). ( B w ) ( b ) = λ ∈ [ 0 , 1 ] min b ( s 0 ) λ + b ( s 1 ) ( 1 − λ ) + γ w (( λ , 1 − λ )) .
At the fixed point we have that for b = ( 1 , 0 ) b=(1,0) b = ( 1 , 0 ) the optimal λ \lambda λ above is λ = 0 \lambda=0 λ = 0 and the next belief is ( 0 , 1 ) (0,1) ( 0 , 1 ) ; for b = ( 0 , 1 ) b=(0,1) b = ( 0 , 1 ) we are in the reverse situation and the optimal λ \lambda λ is 1 so that the next belief is ( 1 , 0 ) (1,0) ( 1 , 0 ) , that is, w ( b ) = min { b ( s 0 ) , b ( s 1 ) } w(b)=\min\{b(s_0),b(s_1)\} w ( b ) = min { b ( s 0 ) , b ( s 1 )} . This is exactly the alternating adversary policy that we highlighted in Example 4.6, where P 2 P^2 P 2 is chosen at even periods and P 1 P^1 P 1 is chosen at odd periods.
Again, for non-rectangular robust MDPs we are left with a simpler picture when the transitions are time-varying (since we have a connection with the well-studied field of POMDPs) than for stationary transitions. For the sake of comparison, note that past works by Littman and coauthors have shown that finding good stationary policies in POMDPs is hard (see reference section below). We conclude with an example showing that an optimal policy for the decision maker need not be stationary in this setup.
Example 4.7. There are four states s 0 , s 1 , G , B s_0,s_1,G,B s 0 , s 1 , G , B where G , B G,B G , B are absorbing, and two actions a 1 , a 2 a_1,a_2 a 1 , a 2 . The uncertainty set P \mathcal{P} P contains two transition kernels P 1 , P 2 \mathsf{P}^1,\mathsf{P}^2 P 1 , P 2 defined as follows. Under P 1 \mathsf{P}^1 P 1 , action a 1 a_1 a 1 yields the deterministic transitions s 0 → G s_0 \to G s 0 → G and s 1 → B s_1 \to B s 1 → B , while action a 2 a_2 a 2 yields the transitions s 0 → B , s 1 → B s_0 \to B,s_1 \to B s 0 → B , s 1 → B . Under P 2 \mathsf{P}^2 P 2 , action a 1 a_1 a 1 yields the deterministic transitions s 0 → B s_0 \to B s 0 → B and s 1 → s 0 s_1 \to s_0 s 1 → s 0 , while action a 2 a_2 a 2 yields the transitions s 0 → G , s 1 → s 0 s_0 \to G,s_1 \to s_0 s 0 → G , s 1 → s 0 . The rewards are 0 0 0 everywhere except r ( s 0 , a , G ) = 1 r(s_0,a,G)=1 r ( s 0 , a , G ) = 1 for a ∈ { a 1 , a 2 } a\in\{a_1,a_2\} a ∈ { a 1 , a 2 } . Thus the decision maker wants to perform the transition s 0 → G s_0 \to G s 0 → G , while the adversary tries to guess their action and react accordingly (i.e. if the decision maker chooses a 1 a_1 a 1 in s 0 s_0 s 0 the adversary wants to choose P 2 \mathsf{P}^2 P 2 ). The initial distribution is uniform over { s 0 , s 1 } \{s_0,s_1\} { s 0 , s 1 } . Take γ = 1 / 2 \gamma=1/2 γ = 1/2 to simplify our calculations. Let π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H . Assume that s 0 s_0 s 0 is reached at period t = 1 t=1 t = 1 . The decision maker faces the typical “max x min p x p + ( 1 − x ) ( 1 − p ) \max_x\min_p xp+(1-x)(1-p) max x min p x p + ( 1 − x ) ( 1 − p ) ” situation we have encountered in many previous examples (e.g. right after Proposition 2.5 ), with x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] the probability that the decision maker plays action a 1 a_1 a 1 and p ∈ { 0 , 1 } p\in\{0,1\} p ∈ { 0 , 1 } modeling the choice of the adversary (p = 1 p=1 p = 1 for choosing P 1 \mathsf{P}^1 P 1 , p = 0 p=0 p = 0 otherwise). Hence the best worst-case continuation value from s 0 s_0 s 0 at period t = 1 t=1 t = 1 is 1 / 2 1/2 1/2 , attained by the uniform randomization x ⋆ = 1 / 2 x^\star=1/2 x ⋆ = 1/2 . Now let z ∈ [ 0 , 1 ] z\in[0,1] z ∈ [ 0 , 1 ] the probability that the decision maker chooses a 1 a_1 a 1 at period t = 0 t=0 t = 0 when starting in s 0 s_0 s 0 . Then R ( π , P 1 ) = z ⋅ 1 2 R(\pi,\mathsf{P}^1)=z\cdot\frac{1}{2} R ( π , P 1 ) = z ⋅ 2 1 (with the factor 1 / 2 1/2 1/2 coming from the initial distribution), and R ( π , P 2 ) = 1 − z 2 + 1 2 ⋅ 1 2 ⋅ 1 2 = 1 2 z + 1 8 R(\pi,\mathsf{P}^2)=\frac{1-z}{2}+\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{1}{2}=\frac{1}{2}z+\frac{1}{8} R ( π , P 2 ) = 2 1 − z + 2 1 ⋅ 2 1 ⋅ 2 1 = 2 1 z + 8 1 . Therefore,
sup π ∈ Π H inf κ ∈ P H G R ( π , κ ) = max z ∈ [ 0 , 1 ] min { z 2 , 1 − z 2 + 1 8 } = 5 16 . \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{G}}}R(\pi,\kappa)=\max_{z\in[0,1]}\min\left\{\frac{z}{2},\frac{1-z}{2}+\frac{1}{8}\right\}=\frac{5}{16}. π ∈ Π H sup κ ∈ P H G inf R ( π , κ ) = z ∈ [ 0 , 1 ] max min { 2 z , 2 1 − z + 8 1 } = 16 5 . which is attained at z ⋆ = 5 / 8 z^\star=5/8 z ⋆ = 5/8 , that is, at the Markovian policy choosing a 1 a_1 a 1 with probability 5 / 8 5/8 5/8 at period t = 0 t=0 t = 0 then a 1 a_1 a 1 with probability 1 / 2 1/2 1/2 for t ≥ 1 t\ge1 t ≥ 1 . In contrast, a stationary policy π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S choosing always a 1 a_1 a 1 with probability x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] obtains a worst-case return of
max π ∈ Π S inf κ ∈ P H G R ( π , κ ) = max z ∈ [ 0 , 1 ] min { z 2 , 1 − z 2 + 1 4 min { z , 1 − z } } . \max_{\pi\in\Pi_{\mathrm{S}}}\inf_{\kappa\in\mathcal{P}_{\mathrm{H}}^{\mathrm{G}}}R(\pi,\kappa)=\max_{z\in[0,1]}\min\left\{\frac{z}{2},\frac{1-z}{2}+\frac{1}{4}\min\{z,1-z\}\right\}. π ∈ Π S max κ ∈ P H G inf R ( π , κ ) = z ∈ [ 0 , 1 ] max min { 2 z , 2 1 − z + 4 1 min { z , 1 − z } } . One can check that the maximum of this quantity over z ∈ [ 0 , 1 ] z\in[0,1] z ∈ [ 0 , 1 ] is 3 10 \frac{3}{10} 10 3 attained at z = 3 / 5 z=3/5 z = 3/5 . Since 3 / 10 < 5 / 16 3/10<5/16 3/10 < 5/16 we get that stationary policies are strictly suboptimal in this example.
The previous example has a particular structure, in that the transitions are deterministic: for all P ∈ P P\in\mathcal{P} P ∈ P , P ( s , a , s ′ ) ∈ { 0 , 1 } , ∀ ( s , a , s ′ ) ∈ S × A × S P(s,a,s')\in\{0,1\},\ \forall(s,a,s')\in\mathcal{S}\times\mathcal{A}\times\mathcal{S} P ( s , a , s ′ ) ∈ { 0 , 1 } , ∀ ( s , a , s ′ ) ∈ S × A × S . In this case the adversary can correctly guess the current state from period 1 onward; only the state at period t = 0 t=0 t = 0 , sampled from p 0 ∈ Δ ( S ) p_0\in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) , is uncertain. Thus from period 1 onward we are in a similar setup as Model 2 where the adversary can observe the current state and the decision maker can play stationary from period t = 1 t=1 t = 1 onward. This yields a value function v ~ ⋆ ∈ R S \tilde v^\star\in\mathbb{R}^{\mathcal{S}} v ~ ⋆ ∈ R S for the rewards obtained starting from period 1, and then we can play at period 0 by solving
max π ∈ Π S min P ∈ P ∑ s ∈ S p 0 ( s ) ∗ ∑ a ∈ A π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ~ ⋆ ) \max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}\sum_{s\in\mathcal{S}}p_0(s)\ast\sum_{a\in\mathcal{A}}\pi(s,a)P(s,a)^\top\left(r(s,a)+\gamma\tilde v^\star\right) π ∈ Π S max P ∈ P min s ∈ S ∑ p 0 ( s ) ∗ a ∈ A ∑ π ( s , a ) P ( s , a ) ⊤ ( r ( s , a ) + γ v ~ ⋆ )
which maximizes the sum of the instantaneous reward obtained at period t = 0 t = 0 t = 0 and then the expected continuation values starting from t = 1 t = 1 t = 1 , where we know we can play stationary and obtain v ~ ⋆ ∈ R S \tilde{v}^{\star} \in\mathbb{R}^{S} v ~ ⋆ ∈ R S as continuation values.
Stochastic games
Stochastic games (SGs) were introduced by Shapley in 1953. We focus here on two-player zero-sum stochastic games, where players have opposite interests and choose some actions at each period, which influence the dynamics of the observed states, and players optimize for their gains.
Stochastic game model. Let us introduce this model more formally. In SGs it is common to distinguish between an arena , which is the setup in which players make their decisions, and the objective that the players optimize. A stochastic game consists of an arena and an objective. An arena of a two-player zero-sum stochastic game Γ \Gamma Γ is determined by a tuple ( Ω , I , J , g , q ) (\Omega,\mathcal{I},\mathcal{J},g,q) ( Ω , I , J , g , q ) with Ω \Omega Ω a finite set of states with cardinality d d d , I \mathcal{I} I the set of actions of the first player, J \mathcal{J} J the set of actions of the second player, g : I × J × Ω → R g : \mathcal{I} \times\mathcal{J} \times\Omega\to\mathbb{R} g : I × J × Ω → R is the instantaneous reward associated with each state ω ∈ Ω \omega\in\Omega ω ∈ Ω and pair of actions ( i , j ) ∈ I × J (i,j) \in\mathcal{I} \times\mathcal{J} ( i , j ) ∈ I × J , and q : I × J × Ω → Δ ( Ω ) q : \mathcal{I} \times\mathcal{J} \times\Omega\to\Delta(\Omega) q : I × J × Ω → Δ ( Ω ) gives the transition probabilities. Sometimes this kind of SG is called concurrent because players choose their actions simultaneously. For the sake of conciseness, we will refer to two-player zero-sum stochastic games simply as stochastic games (SGs). We will assume that the sets of actions I , J \mathcal{I},\mathcal{J} I , J are compact and that the payoff and transition functions are continuous.
The game proceeds as follows. There is an initial state ω 0 \omega_0 ω 0 known to both players at period t = 0 t = 0 t = 0 . Both players simultaneously choose their actions ( x 0 , y 0 ) (x_0,y_0) ( x 0 , y 0 ) in I × J \mathcal{I} \times\mathcal{J} I × J , the payoff is g ( x 0 , y 0 , ω 0 ) g(x_0,y_0,\omega_0) g ( x 0 , y 0 , ω 0 ) , and the system transitions to ω 1 ∈ Ω \omega_1 \in\Omega ω 1 ∈ Ω with probability q ( ω 1 ∣ x 0 , y 0 , ω 0 ) q(\omega_1 \mid x_0,y_0,\omega_0) q ( ω 1 ∣ x 0 , y 0 , ω 0 ) at the next period. This sequence of events is repeated infinitely, and we note H t = ( Ω × I × J ) t × Ω H_t = (\Omega\times\mathcal{I} \times\mathcal{J})^t \times\Omega H t = ( Ω × I × J ) t × Ω the set of histories of length t ∈ N t \in\mathbb{N} t ∈ N , H = ⋃ t ∈ N H t H = \bigcup_{t\in\mathbb{N}} H_t H = ⋃ t ∈ N H t the set of all finite histories, and H ∞ = ( Ω × I × J ) N H_\infty= (\Omega\times\mathcal{I} \times\mathcal{J})^{\mathbb{N}} H ∞ = ( Ω × I × J ) N the set of infinite histories. A strategy for player 1 (resp. for player 2) is a map σ : H → Δ ( I ) \sigma: H \to\Delta(\mathcal{I}) σ : H → Δ ( I ) (resp. τ : H → Δ ( J ) \tau: H \to\Delta(\mathcal{J}) τ : H → Δ ( J ) ), and we write Π H i \Pi^i_H Π H i for the set of all history-dependent, randomized strategies of player i i i , i.e., the set Δ ( I ) H \Delta(\mathcal{I})^H Δ ( I ) H for the first player and the set Δ ( J ) H \Delta(\mathcal{J})^H Δ ( J ) H for the second player. We also write Π S i \Pi^i_S Π S i for the set of stationary strategies of player i i i , which we can identify with the sets Δ ( I ) Ω \Delta(\mathcal{I})^\Omega Δ ( I ) Ω and Δ ( J ) Ω \Delta(\mathcal{J})^\Omega Δ ( J ) Ω , and Π S D i \Pi^i_{SD} Π S D i for the set of stationary deterministic strategies, which we can identify with the sets I Ω \mathcal{I}^\Omega I Ω and J Ω \mathcal{J}^\Omega J Ω . Note that history-dependent , stationary , and deterministic strategies are usually called behavioral , positional , and pure in the SG literature.
A pair of strategies ( σ , τ ) ∈ Π H 1 × Π H 2 (\sigma,\tau) \in\Pi^1_H \times\Pi^2_H ( σ , τ ) ∈ Π H 1 × Π H 2 induces a distribution over the set of infinite histories. There are several ways to aggregate the random infinite stream of payoffs ( g ( x t , y t , ω t ) ) t ≥ 0 (g(x_t,y_t,\omega_t))_{t\geq0} ( g ( x t , y t , ω t ) ) t ≥ 0 induced by two strategies ( σ , τ ) ∈ Π H 1 × Π H 2 (\sigma,\tau) \in\Pi^1_H \times\Pi^2_H ( σ , τ ) ∈ Π H 1 × Π H 2 . The mean-payoff starting from state ω ∈ Ω \omega\in\Omega ω ∈ Ω is
γ ( σ , τ , ω ) : = E σ , τ [ lim sup T → + ∞ 1 T + 1 ∑ t = 0 T g ( x t , y t , ω t ) | ω 0 = ω ] (67) \gamma(\sigma,\tau,\omega) := \mathbb{E}^{\sigma,\tau}\left[\limsup_{T\to+\infty}\frac{1}{T+1}\sum_{t=0}^{T}g(x_t,y_t,\omega_t)\,\middle|\,\omega_0=\omega\right] \tag*{(67)} γ ( σ , τ , ω ) := E σ , τ [ T → + ∞ lim sup T + 1 1 t = 0 ∑ T g ( x t , y t , ω t ) ω 0 = ω ] (67)
and the discounted payoff is, for λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) ,
γ λ ( σ , τ , ω ) : = λ E σ , τ [ ∑ t = 0 + ∞ ( 1 − λ ) t g ( x t , y t , ω t ) | ω 0 = ω ] . (68) \gamma_\lambda(\sigma,\tau,\omega) := \lambda\mathbb{E}^{\sigma,\tau}\left[\sum_{t=0}^{+\infty}(1-\lambda)^t g(x_t,y_t,\omega_t)\,\middle|\,\omega_0=\omega\right]. \tag*{(68)} γ λ ( σ , τ , ω ) := λ E σ , τ [ t = 0 ∑ + ∞ ( 1 − λ ) t g ( x t , y t , ω t ) ω 0 = ω ] . (68)
Of course robust MDPs greatly resemble this model despite different terminologies and notation (e.g., the terms “policies” and “history-dependent/stationary” correspond to “strategies”, “behavioral/positional”, the Shapley operator in SGs corresponds to the robust Bellman operator T T T , etc.), and this has been observed repeatedly in the past, as early as in the first paper on robust MDPs by Satia and Lave in 1973. We describe this connection for s-rectangular and sa-rectangular robust MDPs below.
Perfect-information SGs and sa-rectangular robust MDPs
Consider an sa-rectangular compact uncertainty set P = × s , a P s a \mathcal{P} = \times_{s,a}\mathcal{P}_{sa} P = × s , a P s a , with state set S \mathcal{S} S , action set A \mathcal{A} A , rewards r ∈ R S × A × S r \in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} r ∈ R S × A × S . Then we can model the corresponding robust MDP problem (20) as a perfect-information stochastic game, which is a particular class of SGs where each state is controlled by exactly one player. In fact we will show that we can reformulate (20) as a turn-based SG, where players play in turn: the first player chooses actions at even periods and the second player chooses actions at odd periods.
Consider the following game instance. The first player is the decision maker and the second player is the adversary. The set of states is augmented to Ω = S ∪ S × A \Omega= \mathcal{S} \cup\mathcal{S} \times\mathcal{A} Ω = S ∪ S × A . States ω ∈ S \omega\in\mathcal{S} ω ∈ S are visited at even periods (where the decision maker plays) and states ω ∈ S × A \omega\in\mathcal{S} \times\mathcal{A} ω ∈ S × A are visited at odd periods (where the adversary plays). In each state s ∈ S s \in\mathcal{S} s ∈ S the available actions for the decision maker are I = A \mathcal{I} = \mathcal{A} I = A . If action a ∈ A a \in\mathcal{A} a ∈ A is chosen the dynamics transitions deterministically to the next state ω ′ = ( s , a ) \omega' = (s,a) ω ′ = ( s , a ) and the reward for this transition is 0. At a state ω ′ = ( s , a ) \omega' = (s,a) ω ′ = ( s , a ) the set of actions for the adversary is P s a \mathcal{P}_{sa} P s a (note that we can allow for state-dependent action sets in SGs, it only complicates a bit the notation). If an “action” P ( s , a ) ∈ P s a P(s,a) \in\mathcal{P}_{sa} P ( s , a ) ∈ P s a is chosen, the probability to transition to the next state s ′ s' s ′ is P ( s , a , s ′ ) P(s,a,s') P ( s , a , s ′ ) and the instantaneous reward is P ( s , a ) ⊤ r ( s , a ) P(s,a)^\top r(s,a) P ( s , a ) ⊤ r ( s , a ) (which only depends on the current state ( s , a ) (s,a) ( s , a ) and action P ( s , a ) P(s,a) P ( s , a ) but not on the next state s ′ s' s ′ ). The dynamics for this SG is represented in Figure 4.2 . To ensure that the discounted payoff for the SG is the discounted return for the robust MDP, we need to choose a discount factor γ \gamma γ such that 1 − λ = γ 1-\lambda= \sqrt{\gamma} 1 − λ = γ (and divide the reward at the intermediate states by ( 1 − γ ) γ (1-\sqrt{\gamma})\sqrt{\gamma} ( 1 − γ ) γ ; this is not represented in Figure 4.2 for the sake of readability) in this turn-based SG, because rewards are obtained only at odd periods (instead of every period in the original robust MDP instance).
Figure 4.2. An sa-rectangular robust MDP as a perfect-information (turn-based) game. The decision-maker controls the states s ∈ S s \in\mathcal{S} s ∈ S and chooses an action a ∈ A a \in\mathcal{A} a ∈ A . The adversary controls the states ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A and chooses transition probabilities P ( s , a ) ∈ P s a P(s,a) \in\mathcal{P}_{sa} P ( s , a ) ∈ P s a . The pairs above the arcs represent the transition probabilities and instantaneous rewards.
There are two important caveats to the above constructions, pertaining to the sets of policies of each player:
In SGs each player should be allowed to play mixed strategies, i.e., to randomize over their sets of actions. For the decision maker this is akin to the situation in MDPs and robust MDPs where they can choose a randomized decision rule in Δ ( A ) \Delta(\mathcal{A}) Δ ( A ) at each period, instead of a fixed a ∈ A a \in\mathcal{A} a ∈ A . For the adversary, in principle at each odd period we need to allow for a choice of randomized decision rule in Δ ( P s a ) \Delta(\mathcal{P}_{sa}) Δ ( P s a ) (interpreted as the set of probability distributions over P s a \mathcal{P}_{sa} P s a ) instead of a choice of deterministic action in P s a \mathcal{P}_{sa} P s a .
In SGs each player should be allowed to play history-dependent strategies, whereas in classical robust MDPs as in (2.1) the decision maker can choose history-dependent policies while the adversary must choose the transition probabilities once and for all. However even the definition of what constitutes a “history” is different in SGs and in robust MDPs. Robust MDPs build upon nominal MDPs and define the set of all finite histories from the decision maker’s perspective:
H = ⋃ t ∈ N H t , \mathcal{H} = \bigcup_{t \in\mathbb{N}} \mathcal{H}_t, H = t ∈ N ⋃ H t ,
H t = ( S × A ) t × S \mathcal{H}_t = (\mathcal{S} \times\mathcal{A})^t \times\mathcal{S} H t = ( S × A ) t × S
where H t \mathcal{H}_t H t is the set of finite histories ( s 0 , a 0 , … , s t − 1 , a t − 1 , s t ) (s_0, a_0, \ldots, s_{t-1}, a_{t-1}, s_t) ( s 0 , a 0 , … , s t − 1 , a t − 1 , s t ) defined in § 1.1 : a history includes the state and the action sampled by the decision maker (first player) at each period. However in SGs a history must also include the past decisions of the second player! The first item above explains that the second player should choose a randomized decision rule in Δ ( P s a ) \Delta(\mathcal{P}_{sa}) Δ ( P s a ) at each period, and sample a realization P ( s , a ) ∈ P s a P(s,a) \in\mathcal{P}_{sa} P ( s , a ) ∈ P s a (according to this distribution). Hence in principle a finite history for the first player in the SG representation of this sa-rectangular robust MDP is some sequence
( s 0 , a 0 , ( s 0 , a 0 ) , P 0 ( s 0 , a 0 ) , … , s t − 1 , a t − 1 , ( s t − 1 , a t − 1 ) , P t − 1 ( s t − 1 , a t − 1 ) , s t ) (s_0, a_0, (s_0, a_0), P^0(s_0, a_0), \ldots, s_{t-1}, a_{t-1}, (s_{t-1}, a_{t-1}), P^{t-1}(s_{t-1}, a_{t-1}), s_t) ( s 0 , a 0 , ( s 0 , a 0 ) , P 0 ( s 0 , a 0 ) , … , s t − 1 , a t − 1 , ( s t − 1 , a t − 1 ) , P t − 1 ( s t − 1 , a t − 1 ) , s t )
for some t ∈ N t \in\mathbb{N} t ∈ N and P 0 ( s 0 , a 0 ) ∈ P s 0 a 0 , … , P t − 1 ( s t − 1 , a t − 1 ) ∈ P s t − 1 a t − 1 P^0(s_0, a_0) \in\mathcal{P}_{s_0a_0}, \ldots, P^{t-1}(s_{t-1}, a_{t-1}) \in\mathcal{P}_{s_{t-1}a_{t-1}} P 0 ( s 0 , a 0 ) ∈ P s 0 a 0 , … , P t − 1 ( s t − 1 , a t − 1 ) ∈ P s t − 1 a t − 1 (note that we include the “states” ( s 0 , a 0 ) , … , ( s t − 1 , a t − 1 ) (s_0, a_0), \ldots, (s_{t-1}, a_{t-1}) ( s 0 , a 0 ) , … , ( s t − 1 , a t − 1 ) in this history since these are states visited in the SG, where the adversary makes their decisions) and for the second player it is some sequence
( s 0 , a 0 , ( s 0 , a 0 ) , P 0 ( s 0 , a 0 ) , … , s t − 1 , a t − 1 , ( s t − 1 , a t − 1 ) , P t − 1 ( s t − 1 , a t − 1 ) , s t , a t , ( s t , a t ) ) . (s_0, a_0, (s_0, a_0), P^0(s_0, a_0), \ldots, s_{t-1}, a_{t-1}, (s_{t-1}, a_{t-1}), P^{t-1}(s_{t-1}, a_{t-1}), s_t, a_t, (s_t, a_t)). ( s 0 , a 0 , ( s 0 , a 0 ) , P 0 ( s 0 , a 0 ) , … , s t − 1 , a t − 1 , ( s t − 1 , a t − 1 ) , P t − 1 ( s t − 1 , a t − 1 ) , s t , a t , ( s t , a t )) .
This departs from the set of histories (see § 1.1 ) and time-varying transitions (see § 4.4 ) in robust MDPs, since in SGs both players have much more information (they can observe the realizations of the actions of the second player along each trajectory).
Fortunately, there are some special cases where the two caveats above do not matter for the interpretation of robust MDPs with sa-rectangular uncertainty sets as turn-based SGs. These are the cases of the objective functions for which we have that a pair of optimal strategies in the SG can be chosen stationary (only dependent on the current state) and deterministic. For instance the literature on SGs has shown that this is true in:
Case 1: perfect-information SGs with the discounted objective when both players have compact action sets. For the sa-rectangular robust MDP it corresponds to compactness of A \mathcal{A} A and compactness of P s a \mathcal{P}_{sa} P s a , which are usual assumptions.
Case 2: perfect-information SGs with the mean-payoff objective when both players have finite action sets. For the sa-rectangular robust MDP it corresponds to finiteness of A \mathcal{A} A and finiteness of P s a \mathcal{P}_{sa} P s a , and the average return R a v g ( π , P ) R_{\mathrm{avg}}(\pi,P) R avg ( π , P ) is defined as (compare with (67) )
R a v g ( π , P ) = E π , P [ lim sup T → + ∞ 1 T + 1 ∑ t = 0 T r ( s t , a t , s t + 1 ) ] R_{\mathrm{avg}}(\pi,P)=\mathbb{E}^{\pi,P}\left[\limsup_{T\to+\infty}\frac{1}{T+1}\sum_{t=0}^{T}r(s_t,a_t,s_{t+1})\right] R avg ( π , P ) = E π , P [ T → + ∞ lim sup T + 1 1 t = 0 ∑ T r ( s t , a t , s t + 1 ) ]
Stationarity of the first player’s strategy means that their decision rules only depend on the current state s t s_t s t , and stationarity of the second player’s strategy means that their decision rules depend on the current pair ( s t , a t ) (s_t,a_t) ( s t , a t ) , and not on the past histories. Hence for Case 1 and Case 2 above it is irrelevant that: (A) turn-based SGs allow for history-dependent adversaries, since an optimal strategy for the adversary in the turn-based SG formulation can be chosen stationary anyway, (B) the sets of histories for SGs and for robust MDPs differ, since optimal policies are stationary and only depend on the last states, where histories of MDPs and histories of SGs coincide. In particular, the adversary and the decision maker do not need to see the realizations of the past chosen transition probabilities to play optimally. Note Case 1 above corresponds to the setting we have studied in § 3.4 , and to some extent the results we obtained there are relatively direct corollaries of existing results for stochastic games. Case 2 above yields the following proposition.
Proposition 4.6. Consider an sa-rectangular robust MDP with a finite set of transition probabilities. Then an optimal policy for the average return objective can be chosen stationary and deterministic:
sup π ∈ Π H inf P ∈ P R a v g ( π , P ) = max π ∈ Π S D min P ∈ P R a v g ( π , P ) . \sup_{\pi\in\Pi_H}\inf_{P\in\mathcal{P}}R_{\mathrm{avg}}(\pi,P)=\max_{\pi\in\Pi_{SD}}\min_{P\in\mathcal{P}}R_{\mathrm{avg}}(\pi,P). π ∈ Π H sup P ∈ P inf R avg ( π , P ) = π ∈ Π S D max P ∈ P min R avg ( π , P ) .
It is worth noting that even though Case 2 above requires finiteness of P s a \mathcal{P}_{sa} P s a , it is not hard to extend the results to the case of compact P s a \mathcal{P}_{sa} P s a , although in this case worst-case kernels need not exist anymore (see references at the end of this section).
Concurrent SGs and s-rectangular robust MDPs
Consider an s-rectangular convex compact uncertainty set P \mathcal{P} P , with state set S \mathcal{S} S , action set A \mathcal{A} A , rewards r ∈ R S × A × S r\in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} r ∈ R S × A × S . We can reformulate the corresponding robust MDP problem as a concurrent stochastic game, where both players choose their actions at the same time, as follows. The first player is the decision maker and the second player is the adversary. The set of states for the SG is now Ω = S \Omega=\mathcal{S} Ω = S . In each state s ∈ S s\in\mathcal{S} s ∈ S , the decision maker samples an action a ∈ A a \in\mathcal{A} a ∈ A from some (possibly randomized) decision rule in Δ ( A ) \Delta(\mathcal{A}) Δ ( A ) , while the adversary chooses an action P ( s ) ∈ P s P(s) \in\mathcal{P}_s P ( s ) ∈ P s . Since P s \mathcal{P}_s P s is convex we can interpret choosing an element P ( s ) ∈ P s P(s) \in\mathcal{P}_s P ( s ) ∈ P s as choosing the realization of a randomized decision over Δ ( P s ) \Delta(\mathcal{P}_s) Δ ( P s ) from some (possibly randomized) decision rule in Δ ( P s ) \Delta(\mathcal{P}_s) Δ ( P s ) .
The instantaneous payoff is P ( s , a ) ⊤ r ( s , a ) P(s,a)^{\top}r(s,a) P ( s , a ) ⊤ r ( s , a ) and the transition probability to the next state is P ( s , a , s ′ ) P(s,a,s') P ( s , a , s ′ ) . We can now use a discount factor of γ \gamma γ (the same as for the robust MDP instance). Note that the second caveat still applies here: a finite history for this SG consists of some sequence
( s 0 , ( a 0 , P 0 ( s 0 ) ) , s 1 , ( a 1 , P 1 ( s 1 ) ) , … , s t − 1 , ( a t − 1 , P t − 1 ( s t − 1 ) ) , s t ) (s_0,(a_0,P^0(s_0)),s_1,(a_1,P^1(s_1)),\ldots,s_{t-1},(a_{t-1},P^{t-1}(s_{t-1})),s_t) ( s 0 , ( a 0 , P 0 ( s 0 )) , s 1 , ( a 1 , P 1 ( s 1 )) , … , s t − 1 , ( a t − 1 , P t − 1 ( s t − 1 )) , s t )
for some t ∈ N t \in\mathbb{N} t ∈ N and P 0 , … , P t − 1 ∈ P P^0,\ldots,P^{t-1} \in\mathcal{P} P 0 , … , P t − 1 ∈ P . However, for concurrent SGs with discounted return and a finite set of actions for the first player and a convex compact set of actions for the second player, it is known that stationary optimal strategies exist for both players. In principle this yields a mixed strategy μ ∈ × s ∈ S Δ ( P s ) \mu\in\times_{s\in S}\Delta(\mathcal{P}_s) μ ∈ × s ∈ S Δ ( P s ) for the adversary, but since μ \mu μ only plays a role via its expectation (the robust Bellman operator is linear in P P P ), convexity of P \mathcal{P} P enables us to replace the mixed strategy μ \mu μ by its expected kernel P ˉ = E P ∼ μ [ P ] \bar{P} = \mathbb{E}_{P\sim\mu}[P] P ˉ = E P ∼ μ [ P ] , interpreted as a stationary pure strategy for the adversary. Hence again the distinction between the histories for SGs and histories for robust MDPs does not matter in this setting, and again the adversary and the decision maker do not need to see the realizations of the past chosen transition probabilities to play optimally.
It is worth emphasizing that the rectangularity assumption, which plays a central role for robust MDPs, corresponds in SGs to the fact that each player can choose their actions independently at each state.
Applications. Stochastic games date back to the 1950s, while the vast majority of the robust MDP literature has been published after the 2000s. Hence, the connection between robust MDPs and SGs is particularly useful when it enables us to claim new results for robust MDPs as more or less “direct corollaries” of existing results for SGs (using the reformulations of robust MDPs as SGs with particular properties), or at least to use for robust MDPs some techniques developed for SGs. We did so above in Proposition 4.6. Some setups where results for SGs are more advanced than for robust MDPs include the mean-payoff objective (defined above for SGs), which corresponds in robust MDPs to the average return R a v g ( π , P ) R_{\mathrm{avg}}(\pi,P) R avg ( π , P ) defined above. This kind of return has been studied by the SG community since the late 1950s, while this question has only recently been studied for robust MDPs. The same is true, for instance, for the Blackwell optimality criterion (close to the uniform value in SGs), SGs with incomplete information, and SGs that are not zero-sum.
We conclude by emphasizing that sometimes it is not direct to “translate” results for SGs to results for robust MDPs. For instance, if one is interested in the average return R a v g ( π , P ) R_{\mathrm{avg}}(\pi,P) R avg ( π , P ) for s-rectangular robust MDPs, naturally one would want to leverage existing results for SGs with mean-payoff objective. Perhaps the strongest result in this direction is Mertens-Neyman theorem, which states that concurrent SGs with finite sets of states and actions have a value among history-dependent strategies , that is, we have for each state ω ∈ Ω \omega\in\Omega ω ∈ Ω , that
sup σ ∈ Π H 1 inf τ ∈ Π H 2 γ ( σ , τ , ω ) = inf τ ∈ Π H 2 sup σ ∈ Π H 1 γ ( σ , τ , ω ) . \sup_{\sigma\in\Pi_H^1}\inf_{\tau\in\Pi_H^2}\gamma(\sigma,\tau,\omega)
=
\inf_{\tau\in\Pi_H^2}\sup_{\sigma\in\Pi_H^1}\gamma(\sigma,\tau,\omega). σ ∈ Π H 1 sup τ ∈ Π H 2 inf γ ( σ , τ , ω ) = τ ∈ Π H 2 inf σ ∈ Π H 1 sup γ ( σ , τ , ω ) .
Hence the sup and inf need not be attained and in general it is required that the players can choose history-dependent (non-stationary, non-Markovian) strategies. Hence this result does not translate directly to studying sup π ∈ Π H inf P ∈ P R a v g ( π , P ) \sup_{\pi\in\Pi_{\mathrm{H}}} \inf_{P \in\mathcal{P}} R_{\mathrm{avg}}(\pi, P) sup π ∈ Π H inf P ∈ P R avg ( π , P ) because now optimal strategies in SGs make full use of the histories (in particular, using the other players’ past decisions) whereas this is not allowed for the adversary in robust MDPs.
Distributionally robust MDPs
Distributionally robust MDPs (DRMDPs) assume that the transition probabilities are random and follow some uncertain distribution ν ∈ V \nu\in\mathcal{V} ν ∈ V , where V \mathcal{V} V is a set of distributions over Δ ( S ) S × A \Delta(\mathcal{S})^{\mathcal{S} \times\mathcal{A}} Δ ( S ) S × A , the set of all possible transition probabilities. The set V \mathcal{V} V is sometimes called the ambiguity set , not to be confused with the uncertainty set P \mathcal{P} P (note that these two sets don’t live in the same space: P \mathcal{P} P is a set of transition probabilities whereas V \mathcal{V} V is a set of probability measures over transition probabilities). There are two main assumptions in the DRMDPs literature:
Product distributions. First it is assumed that V \mathcal{V} V is a Cartesian product: V = × s ∈ S V s \mathcal{V} = \times_{s \in\mathcal{S}} \mathcal{V}_s V = × s ∈ S V s , where V s \mathcal{V}_s V s is a set of distributions over Δ ( S ) A \Delta(\mathcal{S})^{\mathcal{A}} Δ ( S ) A . Therefore the measure ν \nu ν for the random transitions P ∈ Δ ( S ) S × A P \in\Delta(\mathcal{S})^{\mathcal{S} \times\mathcal{A}} P ∈ Δ ( S ) S × A is a product measure over s ∈ S s \in\mathcal{S} s ∈ S . Each ν s ∈ V s \nu_s \in\mathcal{V}_s ν s ∈ V s represents a possible distribution for the s s s -th component P ( s ) P(s) P ( s ) of the transition probabilities. Typically we will assume that the set of expectations P s = { E P ( s ) ∼ ν s [ P ( s ) ] ∣ ν s ∈ V s } \mathcal{P}_s = \{\mathbb{E}_{P(s) \sim\nu_s}[P(s)] \mid\nu_s \in\mathcal{V}_s\} P s = { E P ( s ) ∼ ν s [ P ( s )] ∣ ν s ∈ V s } is convex and compact.
Resampled kernels. Second, the transition probabilities are resampled independently at every iteration from some ν t ∈ V \nu^t \in\mathcal{V} ν t ∈ V (chosen before the dynamics start), and the realizations are not observed by the decision maker.
Using the terminology from the seminal paper by Xu and Mannor we can then consider two classes of time-varying random transition probabilities:
V S : = { ( ν ) t ∈ N ∣ ν ∈ V } . \mathcal{V}_{\mathrm{S}} := \{(\nu)_{t \in\mathbb{N}} \mid\nu\in\mathcal{V}\}. V S := {( ν ) t ∈ N ∣ ν ∈ V } .
V N S : = { ( ν t ) t ∈ N ∣ ν t ∈ V , ∀ t ∈ N } = × t ∈ N V . \mathcal{V}_{\mathrm{NS}} := \{(\nu^t)_{t \in\mathbb{N}} \mid\nu^t \in\mathcal{V}, \forall t \in\mathbb{N}\} = \times_{t \in\mathbb{N}} \mathcal{V}. V NS := {( ν t ) t ∈ N ∣ ν t ∈ V , ∀ t ∈ N } = × t ∈ N V .
Note the similarity with the definition of Model 2 from § 4.4 . We could define more generally history-dependent policies for the adversary but we’ll see that all these models collapse to stationary adversaries anyway. Note that V S ⊂ V N S \mathcal{V}_{\mathrm{S}} \subset\mathcal{V}_{\mathrm{NS}} V S ⊂ V NS .
for all. In this case we want to solve sup π ∈ Π H inf ν ∈ V E P ∼ ν [ R ( π , P ) ] \sup_{\pi\in\Pi_H}\inf_{\nu\in\mathcal{V}}\mathbb{E}_{P\sim\nu}[R(\pi,P)] sup π ∈ Π H inf ν ∈ V E P ∼ ν [ R ( π , P )] . The special case where V \mathcal{V} V is the singleton { ν } \{\nu\} { ν } yields the optimization problem sup π ∈ Π H E P ∼ ν [ R ( π , P ) ] \sup_{\pi\in\Pi_H}\mathbb{E}_{P\sim\nu}[R(\pi,P)] sup π ∈ Π H E P ∼ ν [ R ( π , P )] which is known as multi-model MDPs and is in general intractable (even when ν \nu ν has a support of cardinality two), see Section 4.3 and the reference at the end of this chapter.
Recall that we have defined the return R ( π , P ) R(\pi,P) R ( π , P ) with a fixed kernel P ∈ Δ ( S ) S × A P\in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ∈ Δ ( S ) S × A . Given a sequence ( P t ) t ∈ N ∈ ( Δ ( S ) S × A ) N (P^t)_{t\in\mathbb{N}}\in(\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}})^{\mathbb{N}} ( P t ) t ∈ N ∈ ( Δ ( S ) S × A ) N we can extend the return function to R ( π , ( P t ) t ) R(\pi,(P^t)_t) R ( π , ( P t ) t ) since the pair ( π , ( P t ) t ) (\pi,(P^t)_t) ( π , ( P t ) t ) still induces a distribution over the set of trajectories. Therefore for μ ∈ V N S \mu\in\mathcal{V}_{\mathbb{N}\mathcal{S}} μ ∈ V N S we can define
R ( π , μ ) : = E ( P t ) t ∼ μ [ R ( π , ( P t ) t ) ] . R(\pi,\mu):=\mathbb{E}_{(P^t)_{t\sim\mu}}\left[R\left(\pi,(P^t)_t\right)\right]. R ( π , μ ) := E ( P t ) t ∼ μ [ R ( π , ( P t ) t ) ] .
The objective in distributionally robust MDPs is to solve
sup π ∈ Π H inf μ ∈ V S R ( π , μ ) (69) \sup_{\pi\in\Pi_H}\inf_{\mu\in\mathcal{V}_{\mathcal{S}}}R(\pi,\mu) \tag*{(69)} π ∈ Π H sup μ ∈ V S inf R ( π , μ ) (69)
and the same problem with worst-cases against time-varying distributions:
sup π ∈ Π H inf μ ∈ V N S R ( π , μ ) (70) \sup_{\pi\in\Pi_H}\inf_{\mu\in\mathcal{V}_{\mathbb{N}\mathcal{S}}}R(\pi,\mu) \tag*{(70)} π ∈ Π H sup μ ∈ V N S inf R ( π , μ ) (70)
We will see that both (4.19) and (4.20) are equivalent to robust MDPs, in a sense made precise below. We first show that given some μ ∈ V N S \mu\in\mathcal{V}_{\mathbb{N}\mathcal{S}} μ ∈ V N S , the return R ( π , μ ) R(\pi,\mu) R ( π , μ ) only depends on the expected transition probabilities at each period, and not on their entire distribution.
Lemma 4.3. Let π ∈ Π H \pi\in\Pi_H π ∈ Π H and μ = ( ν t ) t ∈ N ∈ V N S \mu=(\nu^t)_{t\in\mathbb{N}}\in\mathcal{V}_{\mathbb{N}\mathcal{S}} μ = ( ν t ) t ∈ N ∈ V N S . Let P ˉ t = E P t ∼ ν t [ P t ] \bar{P}^t=\mathbb{E}_{P_t\sim\nu^t}[P^t] P ˉ t = E P t ∼ ν t [ P t ] . Then
R ( π , μ ) = R ( π , ( P ˉ t ) t ∈ N ) . R(\pi,\mu)=R\left(\pi,(\bar{P}^t)_{t\in\mathbb{N}}\right). R ( π , μ ) = R ( π , ( P ˉ t ) t ∈ N ) .
Proof. Recall that R ( π , μ ) R(\pi,\mu) R ( π , μ ) is a function of P π , μ P^{\pi,\mu} P π , μ , the distribution over infinite histories induced by the pair ( π , μ ) (\pi,\mu) ( π , μ ) . Induction on the period t t t shows that ( P ˉ t ) t ∈ N (\bar{P}^t)_{t\in\mathbb{N}} ( P ˉ t ) t ∈ N and μ \mu μ induce the same distribution over all finite histories h t h_t h t . Hence, they induce the same distribution over all infinite histories, and R ( π , μ ) = R ( π , ( P ˉ t ) t ∈ N ) R(\pi,\mu)=R\left(\pi,(\bar{P}^t)_{t\in\mathbb{N}}\right) R ( π , μ ) = R ( π , ( P ˉ t ) t ∈ N ) .
In particular, when μ ∈ V S \mu\in\mathcal{V}_{\mathcal{S}} μ ∈ V S we have μ = ( ν ) t ∈ N \mu=(\nu)_{t\in\mathbb{N}} μ = ( ν ) t ∈ N for some ν ∈ V \nu\in\mathcal{V} ν ∈ V and R ( π , μ ) = R ( π , P ˉ ) R(\pi,\mu)=R(\pi,\bar{P}) R ( π , μ ) = R ( π , P ˉ ) with P ˉ = E P ∼ ν [ P ] \bar{P}=\mathbb{E}_{P\sim\nu}[P] P ˉ = E P ∼ ν [ P ] . From the above lemma we easily see that distributionally robust MDPs can model robust MDPs as a special case: for P ⊂ Δ ( S ) S × A \mathcal{P}\subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A a given s-rectangular uncertainty set, let V = × s ∈ S { δ { P ( s ) } ∣ P ∈ P } \mathcal{V}=\times_{s\in\mathcal{S}}\{\delta_{\{P(s)\}}\mid P\in\mathcal{P}\} V = × s ∈ S { δ { P ( s )} ∣ P ∈ P } with δ { P ( s ) } \delta_{\{P(s)\}} δ { P ( s )} the Dirac distribution which puts mass 1 at P ( s ) ∈ P s P(s)\in\mathcal{P}_s P ( s ) ∈ P s . Then P ˉ = P \bar{P}=\mathcal{P} P ˉ = P and from Lemma 4.3 we have inf μ ∈ V S R ( π , μ ) = inf P ∈ P R ( π , P ) \inf_{\mu\in\mathcal{V}_{\mathcal{S}}}R(\pi,\mu)=\inf_{P\in\mathcal{P}}R(\pi,P) inf μ ∈ V S R ( π , μ ) = inf P ∈ P R ( π , P ) . A somewhat surprising property is that the reverse is also true: distributionally robust MDPs will reduce to robust MDPs (this is because the distribution μ ∈ V N S \mu\in\mathcal{V}_{\mathbb{N}\mathcal{S}} μ ∈ V N S only plays a role via its mean). We show that this is indeed the case in the next theorem, which is the fundamental result for distributionally robust MDPs.
Theorem 4.7. Let V = × s ∈ S V s \mathcal{V}=\times_{s\in\mathcal{S}}\mathcal{V}_s V = × s ∈ S V s be a set of product distributions over Δ ( S ) S × A \Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} Δ ( S ) S × A . Let P ˉ = { E P ∼ ν [ P ] ∣ ν ∈ V } \bar{\mathcal{P}}=\{\mathbb{E}_{P\sim\nu}[P]\mid\nu\in\mathcal{V}\} P ˉ = { E P ∼ ν [ P ] ∣ ν ∈ V } be the set of expected transition probabilities.
The set P ˉ \bar{\mathcal{P}} P ˉ is always s-rectangular.
Assume that P ‾ \overline{\mathcal{P}} P is compact and convex. Then
sup π ∈ Π H inf μ ∈ V S R ( π , μ ) = sup π ∈ Π H inf μ ∈ V N S R ( π , μ ) = sup π ∈ Π H inf P ‾ ∈ P ‾ R ( π , P ‾ ) = max π ∈ Π S min P ‾ ∈ P ‾ R ( π , P ‾ ) \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\mu\in\mathcal{V}_{\mathrm{S}}} R(\pi,\mu)
= \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\mu\in\mathcal{V}_{\mathrm{NS}}} R(\pi,\mu)
= \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\overline{P} \in\overline{\mathcal{P}}} R(\pi,\overline{P})
= \max_{\pi\in\Pi_{\mathrm{S}}}\min_{\overline{P} \in\overline{\mathcal{P}}} R(\pi,\overline{P}) π ∈ Π H sup μ ∈ V S inf R ( π , μ ) = π ∈ Π H sup μ ∈ V NS inf R ( π , μ ) = π ∈ Π H sup P ∈ P inf R ( π , P ) = π ∈ Π S max P ∈ P min R ( π , P )
and max π ∈ Π S min P ‾ ∈ P ‾ R ( π , P ‾ ) \max_{\pi\in\Pi_{\mathrm{S}}}\min_{\overline{P} \in\overline{\mathcal{P}}} R(\pi,\overline{P}) max π ∈ Π S min P ∈ P R ( π , P ) is a robust MDP problem with s s s -rectangular convex compact uncertainty set P ‾ \overline{\mathcal{P}} P , which admits a stationary uniformly optimal policy.
Proof. 1. The s s s -rectangularity of P ‾ \overline{\mathcal{P}} P is a direct consequence of the fact that each ν ∈ V \nu\in\mathcal{V} ν ∈ V is a product distribution with marginals that can be chosen independently in V s \mathcal{V}_{s} V s for each s ∈ S s \in\mathcal{S} s ∈ S , so that each s s s -th component P s P_{s} P s of the transition probabilities can be sampled independently from some ν s ∈ V s \nu_{s} \in\mathcal{V}_{s} ν s ∈ V s .
The proof can be done from scratch (using truncated returns as has been done several times in the rest of these notes) but it is faster to reuse the main results from § 4.4 . We start by analyzing sup π ∈ Π H inf μ ∈ V N S R ( π , μ ) \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\mu\in\mathcal{V}_{\mathrm{NS}}} R(\pi,\mu) sup π ∈ Π H inf μ ∈ V NS R ( π , μ ) . From Lemma 4.3 we can see this as a special case of robust MDPs with time-varying transition probabilities, where we have that at each period the transition probabilities must belong to P ‾ \overline{\mathcal{P}} P and are chosen dependent on time but not on the current state. Recall that Model 3 in § 4.4 allows for transition probabilities at period t t t dependent on ( s 0 , a 0 , … , s t − 1 , a t − 1 ) (s_{0},a_{0},\ldots,s_{t-1},a_{t-1}) ( s 0 , a 0 , … , s t − 1 , a t − 1 ) . Hence inf μ ∈ V N S R ( π , μ ) \inf_{\mu\in\mathcal{V}_{\mathrm{NS}}} R(\pi,\mu) inf μ ∈ V NS R ( π , μ ) can be interpreted as optimizing over a class of adversary policies P ‾ M G \overline{\mathcal{P}}_{\mathrm{M}}^{\mathrm{G}} P M G that is a subclass of P ‾ H G \overline{\mathcal{P}}_{\mathrm{H}}^{\mathrm{G}} P H G (we use the subscript “M” because it corresponds to a weaker class than Markovian adversary policies). Since P ‾ \overline{\mathcal{P}} P is convex compact and s s s -rectangular (by assumption), we get that P ‾ H G = P ‾ S G \overline{\mathcal{P}}_{\mathrm{H}}^{\mathrm{G}} = \overline{\mathcal{P}}_{\mathrm{S}}^{\mathrm{G}} P H G = P S G (from Lemma 4.2), and from Theorem 4.6 we get that stationary policies are optimal against adversary policies in P ‾ H G \overline{\mathcal{P}}_{\mathrm{H}}^{\mathrm{G}} P H G and, in particular, are optimal against adversary policies in P ‾ S G \overline{\mathcal{P}}_{\mathrm{S}}^{\mathrm{G}} P S G . This shows
sup π ∈ Π H inf μ ∈ V N S R ( π , μ ) = max π ∈ Π S min P ‾ ∈ P ‾ R ( π , P ‾ ) . \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\mu\in\mathcal{V}_{\mathrm{NS}}} R(\pi,\mu)
= \max_{\pi\in\Pi_{\mathrm{S}}}\min_{\overline{P} \in\overline{\mathcal{P}}} R(\pi,\overline{P}). π ∈ Π H sup μ ∈ V NS inf R ( π , μ ) = π ∈ Π S max P ∈ P min R ( π , P ) .
In particular this shows that the worst-case over V N S \mathcal{V}_{\mathrm{NS}} V NS is attained at a stationary distribution in V S \mathcal{V}_{\mathrm{S}} V S when π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S , so that sup π ∈ Π H inf μ ∈ V N S R ( π , μ ) = sup π ∈ Π H inf μ ∈ V S R ( π , μ ) \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\mu\in\mathcal{V}_{\mathrm{NS}}} R(\pi,\mu) = \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{\mu\in\mathcal{V}_{\mathrm{S}}} R(\pi,\mu) sup π ∈ Π H inf μ ∈ V NS R ( π , μ ) = sup π ∈ Π H inf μ ∈ V S R ( π , μ ) . The rest of the theorem follows from Theorem 3.2.
We proved that distributionally robust MDPs, where transition probabilities are sampled at each period, are actually equivalent to robust MDPs, where the transitions are stationary! An optimal policy for a distributionally robust MDP can therefore be computed by the algorithms presented in § 3.3.3 . As evident from the proof of Theorem 4.7, this equivalence between distributionally robust MDPs and robust MDPs remains true even if we allow for more complicated adversary policies (e.g. history-dependent policies that can observe the current state but not the current action chosen by the decision maker), because the analysis we did for Model 2 in § 4.4 also works for history-dependent adversaries.
Example: ambiguity sets based on Wasserstein distances. Wasserstein distances have become popular in distributionally robust optimization (beyond distributionally robust MDPs), and we illustrate the reductions outlined above in this case. To simplify the notation, we use D \mathcal{D} D for the set of all possible transition probabilities out of a state: D = Δ ( S ) A \mathcal{D} = \Delta(\mathcal{S})^{\mathcal{A}} D = Δ ( S ) A . Assume we are given a metric d d d induced by a norm over D \mathcal{D} D and an integer p ∈ N ∖ { 0 } p \in\mathbb{N} \setminus\{0\} p ∈ N ∖ { 0 } (the “type”). The Wasserstein distance W p ( ν , ν ′ ) W_p(\nu,\nu') W p ( ν , ν ′ ) between two distributions ν , ν ′ \nu,\nu' ν , ν ′ over D \mathcal{D} D is
W p ( ν , ν ′ ) = min κ ∈ Δ ( D × D ) Π 1 κ = ν , Π 2 κ = ν ′ ( E ( P , P ′ ) ∼ κ [ d ( P , P ′ ) p ] ) 1 / p W_p(\nu,\nu') = \min_{\substack{\kappa\in\Delta(\mathcal{D} \times\mathcal{D}) \\ \Pi_1\kappa= \nu,\ \Pi_2\kappa= \nu'}} \left(\mathbb{E}_{(P,P') \sim\kappa}\left[d(P,P')^p\right]\right)^{1/p} W p ( ν , ν ′ ) = κ ∈ Δ ( D × D ) Π 1 κ = ν , Π 2 κ = ν ′ min ( E ( P , P ′ ) ∼ κ [ d ( P , P ′ ) p ] ) 1/ p
where Π 1 κ \Pi_1\kappa Π 1 κ and Π 2 κ \Pi_2\kappa Π 2 κ are the first and second marginals for a coupling κ \kappa κ on D × D \mathcal{D} \times\mathcal{D} D × D . We can also allow p = + ∞ p = +\infty p = + ∞ in which case we define W ∞ ( ν , ν ′ ) W_\infty(\nu,\nu') W ∞ ( ν , ν ′ ) by
W ∞ ( ν , ν ′ ) = min κ ∈ Δ ( D × D ) Π 1 κ = ν , Π 2 κ = ν ′ κ -ess.sup ( d ) W_\infty(\nu,\nu') = \min_{\substack{\kappa\in\Delta(\mathcal{D} \times\mathcal{D}) \\ \Pi_1\kappa= \nu,\ \Pi_2\kappa= \nu'}} \kappa\text{-ess.sup}(d) W ∞ ( ν , ν ′ ) = κ ∈ Δ ( D × D ) Π 1 κ = ν , Π 2 κ = ν ′ min κ -ess.sup ( d )
with κ \kappa κ -ess.sup( d ) (d) ( d ) the essential supremum of d d d under κ \kappa κ , equal to inf { c ∈ R ∣ κ ( { ( x , y ) ∣ d ( x , y ) > c } ) = 0 } \inf\{c \in\mathbb{R} \mid\kappa(\{(x,y) \mid d(x,y) > c\}) = 0\} inf { c ∈ R ∣ κ ({( x , y ) ∣ d ( x , y ) > c }) = 0 } . Suppose that we have a nominal estimate ν ˉ ( s ) ∈ Δ ( D ) \bar{\nu}(s) \in\Delta(\mathcal{D}) ν ˉ ( s ) ∈ Δ ( D ) for each s ∈ S s \in\mathcal{S} s ∈ S , and we are interested in ambiguity sets in the form of balls for the Wasserstein distance: for p ∈ N ∪ { + ∞ } p \in\mathbb{N} \cup\{+\infty\} p ∈ N ∪ { + ∞ } , define
V = × s ∈ S V s , V s = { ν ∈ Δ ( D ) ∣ W p ( ν , ν ˉ ( s ) ) ≤ α } \mathcal{V} = \mathop{\times}_{s \in\mathcal{S}}\mathcal{V}_s,\quad\mathcal{V}_s = \{\nu\in\Delta(\mathcal{D}) \mid W_p(\nu,\bar{\nu}(s)) \le\alpha\} V = × s ∈ S V s , V s = { ν ∈ Δ ( D ) ∣ W p ( ν , ν ˉ ( s )) ≤ α }
with ν ˉ = ⨂ s ∈ S ν ˉ ( s ) \bar{\nu} = \bigotimes_{s \in\mathcal{S}}\bar{\nu}(s) ν ˉ = ⨂ s ∈ S ν ˉ ( s ) (a product distribution). Assume that ν ˉ \bar{\nu} ν ˉ is of the form ν ˉ ( s ) = 1 N ∑ i = 1 N δ { P ˉ i ( s ) } \bar{\nu}(s) = \frac{1}{N}\sum_{i=1}^{N}\delta_{\{\bar{P}_i(s)\}} ν ˉ ( s ) = N 1 ∑ i = 1 N δ { P ˉ i ( s )} for some P ˉ i ( s ) ∈ D \bar{P}_i(s) \in\mathcal{D} P ˉ i ( s ) ∈ D . Then the set of expected kernels P \mathcal{P} P induced by V \mathcal{V} V is P p = × s ∈ S P p ( s ) \mathcal{P}_p = \mathop{\times}_{s \in\mathcal{S}}\mathcal{P}_p(s) P p = × s ∈ S P p ( s ) , with, when p ∈ N ∖ { 0 } p \in\mathbb{N} \setminus\{0\} p ∈ N ∖ { 0 } ,
P p ( s ) = { 1 N ∑ i = 1 N P i ( s ) | 1 N ∑ i = 1 N d ( P i ( s ) , P ˉ i ( s ) ) p ≤ α p , P i ( s ) ∈ D , ∀ i ∈ { 1 , … , N } } \mathcal{P}_p(s) = \left\{\frac{1}{N}\sum_{i=1}^{N}P_i(s)\ \middle|\ \frac{1}{N}\sum_{i=1}^{N}d\bigl(P_i(s),\bar{P}_i(s)\bigr)^p \le\alpha^p,\ P_i(s) \in\mathcal{D},\ \forall i \in\{1,\ldots,N\}\right\} P p ( s ) = { N 1 i = 1 ∑ N P i ( s ) N 1 i = 1 ∑ N d ( P i ( s ) , P ˉ i ( s ) ) p ≤ α p , P i ( s ) ∈ D , ∀ i ∈ { 1 , … , N } }
and with p = + ∞ p = +\infty p = + ∞ we have that P ∞ ( s ) \mathcal{P}_\infty(s) P ∞ ( s ) is defined by
P ∞ ( s ) = { 1 N ∑ i = 1 N P i ( s ) | d ( P i ( s ) , P ˉ i ( s ) ) ≤ α , P i ( s ) ∈ D , ∀ i ∈ { 1 , … , N } } . \mathcal{P}_\infty(s) = \left\{\frac{1}{N}\sum_{i=1}^{N}P_i(s)\ \middle|\ d\bigl(P_i(s),\bar{P}_i(s)\bigr) \le\alpha,\ P_i(s) \in\mathcal{D},\ \forall i \in\{1,\ldots,N\}\right\}. P ∞ ( s ) = { N 1 i = 1 ∑ N P i ( s ) d ( P i ( s ) , P ˉ i ( s ) ) ≤ α , P i ( s ) ∈ D , ∀ i ∈ { 1 , … , N } } .
Special classes of non-rectangular uncertainty sets
We have seen in Chapter 2 that for general uncertainty sets, several optimization problems related to robust MDPs can be difficult, e.g., computing the worst-case transition probabilities for some policy may be NP-hard (Proposition 2.10 ) and we may need history-dependent policies to play optimally. In fact Section 3.5 shows that rectangular models of uncertainty are “necessary” (in a certain sense made precise there) to obtain favorable properties, such as the validity of Bellman (dynamic programming) equations and the existence of uniformly optimal policies. In this section we will focus on some special cases of non-rectangular uncertainty sets that retain these favorable properties under some additional assumptions.
r-rectangularity and generalizations
We start with the definition of r-rectangular uncertainty sets.
Definition 4.4. We say that P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A is r-rectangular if there exist r ∈ N r \in\mathbb{N} r ∈ N , some sets W 1 , … , W r ⊂ Δ ( S ) \mathcal{W}^{1},\ldots,\mathcal{W}^{r} \subset\Delta(\mathcal{S}) W 1 , … , W r ⊂ Δ ( S ) , and some coefficients u s a ∈ Δ ( [ r ] ) u_{sa} \in\Delta([r]) u s a ∈ Δ ([ r ]) for each state-action pair ( s , a ) ∈ S × A (s,a) \in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A such that
P = { P ∣ P ( s , a ) = ∑ i ∈ [ r ] u s a ( i ) w i , w i ∈ W i , ∀ i ∈ [ r ] , ∀ ( s , a ) ∈ S × A } (71) \mathcal{P}=\left\{P \mid P(s,a)=\sum_{i\in[r]}u_{sa}(i)w^{i},w^{i}\in\mathcal{W}^{i},\ \forall i\in[r],\ \forall(s,a)\in\mathcal{S}\times\mathcal{A}\right\} \tag*{(71)} P = ⎩ ⎨ ⎧ P ∣ P ( s , a ) = i ∈ [ r ] ∑ u s a ( i ) w i , w i ∈ W i , ∀ i ∈ [ r ] , ∀ ( s , a ) ∈ S × A ⎭ ⎬ ⎫ (71)
The vectors w 1 , … , w r w^{1},\ldots,w^{r} w 1 , … , w r are called the factors ; note that the coefficients (u’s) are fixed and known, but the factors can vary. If the sets W 1 , … , W r \mathcal{W}^{1},\ldots,\mathcal{W}^{r} W 1 , … , W r are compact then P \mathcal{P} P is compact; vice versa if P \mathcal{P} P is compact we can replace W 1 , … , W r \mathcal{W}^{1},\ldots,\mathcal{W}^{r} W 1 , … , W r by their closures. Hence when we assume that P \mathcal{P} P is r-rectangular and compact we will assume that W 1 , … , W r \mathcal{W}^{1},\ldots,\mathcal{W}^{r} W 1 , … , W r are compact. There are a few interpretations for the r-rectangularity assumption:
Equation (71) states that for each state-action pair ( s , a ) (s,a) ( s , a ) , the transition probabilities P ( s , a ) ∈ Δ ( S ) P(s,a)\in\Delta(\mathcal{S}) P ( s , a ) ∈ Δ ( S ) toward the next state can be factorized as first sampling some i ∈ [ r ] i\in[r] i ∈ [ r ] following the distribution u s a ∈ Δ ( [ r ] ) u_{sa}\in\Delta([r]) u s a ∈ Δ ([ r ]) and then sampling the next state according to the factor distribution w i ∈ Δ ( S ) w^{i}\in\Delta(\mathcal{S}) w i ∈ Δ ( S ) .
We can rewrite P ( s , a ) = ∑ i ∈ [ r ] u s a ( i ) w i P(s,a)=\sum_{i\in[r]}u_{sa}(i)w^{i} P ( s , a ) = ∑ i ∈ [ r ] u s a ( i ) w i as P ( s , a ) = W u s a P(s,a)=Wu_{sa} P ( s , a ) = W u s a with W ∈ R S × [ r ] W\in\mathbb{R}^{\mathcal{S}\times[r]} W ∈ R S × [ r ] the matrix with columns w 1 , … , w r ∈ Δ ( S ) w^{1},\ldots,w^{r}\in\Delta(\mathcal{S}) w 1 , … , w r ∈ Δ ( S ) , hence P ( s ) = W u s P(s)=Wu_{s} P ( s ) = W u s with P ( s ) ∈ Δ ( S ) A P(s)\in\Delta(\mathcal{S})^{\mathcal{A}} P ( s ) ∈ Δ ( S ) A and u s ∈ ( Δ ( [ r ] ) ) A u_{s}\in(\Delta([r]))^{\mathcal{A}} u s ∈ ( Δ ([ r ]) ) A . Hence we can interpret r-rectangularity as some form of non-negative matrix factorization, which is the optimization problem of factoring a nonnegative matrix A A A as a product of non-negative matrices B , C B,C B , C : A = B C A=BC A = B C ; in our case one matrix is kept fixed (the coefficients u u u ) while the other can vary (the factor matrix W W W ).
Relations with rectangular models
It should be clear that in general r-rectangular uncertainty sets are neither s-rectangular nor sa-rectangular. For instance if r = 1 r=1 r = 1 then we have a “rank one” uncertainty set where for every state-action pair ( s , a ) (s,a) ( s , a ) we have P ( s , a ) = w 1 P(s,a)=w^{1} P ( s , a ) = w 1 , w 1 ∈ W 1 w^{1}\in\mathcal{W}^{1} w 1 ∈ W 1 . Clearly this set is neither s-rectangular nor sa-rectangular (unless W 1 \mathcal{W}^{1} W 1 is a singleton), i.e. we have the following lemma.
Lemma 4.4. There exists P ⊂ Δ ( S ) S × A \mathcal{P}\subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A that is r-rectangular but neither s-rectangular nor sa-rectangular.
However we always have that ( w 1 , … , w r ) (w^{1},\ldots,w^{r}) ( w 1 , … , w r ) belongs to the Cartesian product set W 1 × ⋯ × W r \mathcal{W}^{1}\times\cdots\times\mathcal{W}^{r} W 1 × ⋯ × W r , even though P \mathcal{P} P itself may not be a Cartesian product set under the r-rectangularity assumption. Hence we can interpret r-rectangularity as a form of “underlying” rectangularity. In particular r-rectangular models contain sa-rectangular models as a “limiting case” (r = ∣ S ∣ × ∣ A ∣ r=|\mathcal{S}|\times|\mathcal{A}| r = ∣ S ∣ × ∣ A ∣ ), as we show next.
Lemma 4.5. Let P ⊂ Δ ( S ) S × A \mathcal{P}\subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be sa-rectangular. Then P \mathcal{P} P is r-rectangular and we can choose r = ∣ S ∣ × ∣ A ∣ r=|\mathcal{S}|\times|\mathcal{A}| r = ∣ S ∣ × ∣ A ∣ .
Proof. Let P = × s , a P s a \mathcal{P} = \times_{s,a}\mathcal{P}_{sa} P = × s , a P s a . Then we can just use r = ∣ S ∣ × ∣ A ∣ r = |\mathcal{S}| \times|\mathcal{A}| r = ∣ S ∣ × ∣ A ∣ and “enumerate”. We have a bijection ϕ : S × A → [ r ] \phi: \mathcal{S} \times\mathcal{A} \to[r] ϕ : S × A → [ r ] . Then we can write P \mathcal{P} P as an r-rectangular uncertainty set by constructing the coefficients u s a u_{sa} u s a as u s a ( i ) = 1 u_{sa}(i) = 1 u s a ( i ) = 1 if ϕ ( ( s , a ) ) = i \phi((s,a)) = i ϕ (( s , a )) = i and u s a ( i ) = 0 u_{sa}(i) = 0 u s a ( i ) = 0 otherwise, and w i = P ( ϕ − 1 ( i ) ) , W i = P ϕ − 1 ( i ) w^i = P(\phi^{-1}(i)), \mathcal{W}^i = \mathcal{P}_{\phi^{-1}(i)} w i = P ( ϕ − 1 ( i )) , W i = P ϕ − 1 ( i ) .
The fact that there exist s-rectangular uncertainty sets that are not r-rectangular will be shown after Corollary 4.1. It is worth emphasizing that sa-rectangularity and s-rectangularity are geometric properties (being equal to its s-rectangular or sa-rectangular extensions), while r-rectangularity is an “existence property” (see Definition 4.4), which may be harder to disprove. So the easiest way to prove that a set is not r-rectangular is to prove that it does not have some properties implied by r-rectangularity.
r-rectangular models are intractable in general. Section 3.5 tells us that we should not expect r-rectangular models to have favorable tractability properties in all generality. Indeed we first show that we may need history-dependent policies even in rank one uncertainty sets.
Proposition 4.7. There exists a compact convex r-rectangular uncertainty set P \mathcal{P} P with ∣ S ∣ = ∣ A ∣ = 2 |\mathcal{S}| = |\mathcal{A}| = 2 ∣ S ∣ = ∣ A ∣ = 2 and rank r = 1 r = 1 r = 1 , for which for some choice of rewards r ∈ R S × A × S r \in\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} r ∈ R S × A × S and initial distribution p 0 ∈ Δ ( S ) p_0 \in\Delta(\mathcal{S}) p 0 ∈ Δ ( S ) , an optimal policy must be non-stationary for any choice of γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) .
Proof. We consider an r-rectangular MDP with two states S = { s 0 , s 1 } \mathcal{S} = \{s_0,s_1\} S = { s 0 , s 1 } and two actions { a 1 , a 2 } \{a_1,a_2\} { a 1 , a 2 } . With a rank one r-rectangular robust MDP we get that starting from period t ≥ 1 t \ge1 t ≥ 1 the next state is sampled following w = ( p , 1 − p ) ∈ Δ ( S ) w = (p,1-p) \in\Delta(\mathcal{S}) w = ( p , 1 − p ) ∈ Δ ( S ) , with p ∈ [ 0 , 1 ] p \in[0,1] p ∈ [ 0 , 1 ] the probability that the next state is s 0 s_0 s 0 ; the state at period t = 0 t = 0 t = 0 is sampled from p 0 p_0 p 0 and we take here p 0 = ( 1 , 0 ) p_0 = (1,0) p 0 = ( 1 , 0 ) . This is represented in Figure 4.3 . We choose the rewards as follows:
Figure 4.3. A rank one r-rectangular robust MDP.
r ( s 0 , a 1 , s 0 ) = 0 , r ( s 0 , a 1 , s 1 ) = − 1 r ( s 0 , a 2 , s 0 ) = − 1 , r ( s 0 , a 2 , s 1 ) = 1 r ( s 1 , a 1 , s 0 ) = − 1 , r ( s 1 , a 1 , s 1 ) = 0 r ( s 1 , a 2 , s 0 ) = 1 , r ( s 1 , a 2 , s 1 ) = − 1. \begin{aligned}
r(s_0,a_1,s_0)&=0,\quad r(s_0,a_1,s_1)=-1\\
r(s_0,a_2,s_0)&=-1,\quad r(s_0,a_2,s_1)=1\\
r(s_1,a_1,s_0)&=-1,\quad r(s_1,a_1,s_1)=0\\
r(s_1,a_2,s_0)&=1,\quad r(s_1,a_2,s_1)=-1.
\end{aligned} r ( s 0 , a 1 , s 0 ) r ( s 0 , a 2 , s 0 ) r ( s 1 , a 1 , s 0 ) r ( s 1 , a 2 , s 0 ) = 0 , r ( s 0 , a 1 , s 1 ) = − 1 = − 1 , r ( s 0 , a 2 , s 1 ) = 1 = − 1 , r ( s 1 , a 1 , s 1 ) = 0 = 1 , r ( s 1 , a 2 , s 1 ) = − 1.
The decision maker is facing the following tradeoff. Action a 1 a_1 a 1 is the best action to choose in either state if we think that we are going to remain in that same state at the next period; otherwise we better choose a 2 a_2 a 2 . The adversary chooses the probability p ∈ [ 0 , 1 ] p \in[0,1] p ∈ [ 0 , 1 ] that the next state is s 0 s_0 s 0 (otherwise it is s 1 s_1 s 1 with probability 1 − p 1-p 1 − p ). Direct calculations give that for π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S we have
( 1 − γ ) R ( π , P ) = ( 1 − γ + γ p ) ( 1 − 2 p + ( 3 p − 2 ) π ( s 0 , a 1 ) ) + γ ( 1 − p ) ( 2 p − 1 + ( 1 − 3 p ) π ( s 1 , a 1 ) ) (1-\gamma)R(\pi,P)=(1-\gamma+\gamma p)\left(1-2p+(3p-2)\pi(s_0,a_1)\right)+\gamma(1-p)\left(2p-1+(1-3p)\pi(s_1,a_1)\right) ( 1 − γ ) R ( π , P ) = ( 1 − γ + γ p ) ( 1 − 2 p + ( 3 p − 2 ) π ( s 0 , a 1 ) ) + γ ( 1 − p ) ( 2 p − 1 + ( 1 − 3 p ) π ( s 1 , a 1 ) )
and therefore the optimal policy among stationary ones is to choose π ( s 0 , a 1 ) = π ( s 1 , a 1 ) = 2 / 3 \pi(s_0,a_1)=\pi(s_1,a_1)=2/3 π ( s 0 , a 1 ) = π ( s 1 , a 1 ) = 2/3 with a worst-case return of − 1 3 − 1 1 − γ -\frac{1}{3}-\frac{1}{1-\gamma} − 3 1 − 1 − γ 1 .
Now consider the following history-dependent policy π ′ \pi' π ′ : in the first two periods we match the best stationary policies: at t = 0 t=0 t = 0 play π ( s 0 , a 1 ) = 2 / 3 \pi(s_0,a_1)=2/3 π ( s 0 , a 1 ) = 2/3 , at t = 1 t=1 t = 1 play the optimal stationary policy again, then for t ≥ 2 t\geq2 t ≥ 2 : if the state visited at period 1 is equal to s 0 s_0 s 0 then play forever π ( s 0 , a 1 ) = 1 \pi(s_0,a_1)=1 π ( s 0 , a 1 ) = 1 , π ( s 1 , a 2 ) = 1 \pi(s_1,a_2)=1 π ( s 1 , a 2 ) = 1 (that is, we bet that p ≥ 1 / 2 p\geq1/2 p ≥ 1/2 , and that we will often “stay” if in s 0 s_0 s 0 and “move away” if we’re in s 1 s_1 s 1 ) otherwise play forever π ( s 0 , a 2 ) = 1 , π ( s 1 , a 1 ) = 1 \pi(s_0,a_2)=1,\pi(s_1,a_1)=1 π ( s 0 , a 2 ) = 1 , π ( s 1 , a 1 ) = 1 (that is, we bet that p ≤ 1 / 2 p\leq1/2 p ≤ 1/2 , and that we will often “move away” if in s 0 s_0 s 0 and “stay” if in s 1 s_1 s 1 ).
For this history-dependent policy, we have a return of R ( π ′ , P ) = − 1 3 − γ 3 − γ 2 p ( 1 − p ) 1 − γ R(\pi',P)=-\frac{1}{3}-\frac{\gamma}{3}-\frac{\gamma^2p(1-p)}{1-\gamma} R ( π ′ , P ) = − 3 1 − 3 γ − 1 − γ γ 2 p ( 1 − p ) , with worst-case attained at p = 1 / 2 p=1/2 p = 1/2 and a worst-case return of − 1 3 − γ 3 − γ 2 4 ( 1 − γ ) -\frac{1}{3}-\frac{\gamma}{3}-\frac{\gamma^2}{4(1-\gamma)} − 3 1 − 3 γ − 4 ( 1 − γ ) γ 2 , which is strictly larger than the best worst-case return for a stationary policy computed above. This shows that non-stationary policies strictly outperform stationary ones in this small example.
We next prove that solving the policy evaluation problem is NP-hard for r-rectangular uncertainty sets, even when r = 1 r=1 r = 1 .
Proposition 4.8. Let π ∈ Π S D \pi\in\Pi_{\mathrm{SD}} π ∈ Π SD . For a general convex compact r r r -rectangular polytope P ⊂ R S × A × S \mathcal{P} \subset\mathbb{R}^{\mathcal{S}\times\mathcal{A}\times\mathcal{S}} P ⊂ R S × A × S with r = 1 r=1 r = 1 , it is in general NP-hard to decide inf P ∈ P R ( π , P ) ≤ c \inf_{P\in\mathcal{P}}R(\pi,P)\leq c inf P ∈ P R ( π , P ) ≤ c .
Proof. The proof is similar to the hardness result from Proposition 4.3 and uses a reduction from the Clique problem. Let G = ( V , E ) G=(V,E) G = ( V , E ) be an instance of a Clique decision problem with V = { 1 , … , n } V=\{1,\ldots,n\} V = { 1 , … , n } and A ∈ { 0 , 1 } n × n A\in\{0,1\}^{n\times n} A ∈ { 0 , 1 } n × n the adjacency matrix. We take S = { s 0 } ∪ V \mathcal{S}=\{s_0\}\cup V S = { s 0 } ∪ V , A = { a 0 } \mathcal{A}=\{a_0\} A = { a 0 } , p 0 = δ s 0 p_0=\delta_{s_0} p 0 = δ s 0 , γ = 1 / 2 \gamma=1/2 γ = 1/2 (we could take any γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) here). The uncertainty set is parametrized by p ∈ Δ ( V ) p\in\Delta(V) p ∈ Δ ( V ) and is defined as
P = { P ∣ ∃ p ∈ Δ ( V ) , P ( s , a 0 , s 0 ) = 0 , ( P ( s , a 0 , s ′ ) ) s ′ ∈ V = p , ∀ s ∈ S } . \mathcal{P}=\left\{P\mid\exists p\in\Delta(V),P(s,a_0,s_0)=0,\left(P(s,a_0,s')\right)_{s'\in V}=p,\ \forall s\in\mathcal{S}\right\}. P = { P ∣ ∃ p ∈ Δ ( V ) , P ( s , a 0 , s 0 ) = 0 , ( P ( s , a 0 , s ′ ) ) s ′ ∈ V = p , ∀ s ∈ S } .
Hence P \mathcal{P} P is r-rectangular of rank r = 1 r=1 r = 1 and it is a compact convex polytope. We choose the rewards as r ( s , a 0 , s ′ ) = 1 − A s s ′ r(s,a_0,s')=1-A_{ss'} r ( s , a 0 , s ′ ) = 1 − A s s ′ for s , s ′ ∈ V s,s'\in V s , s ′ ∈ V and 0 everywhere else. With these definitions the expected one-step instantaneous reward is ∑ s , s ′ ∈ V p s p s ′ ( 1 − A s s ′ ) = 1 − p T A p \sum_{s,s'\in V}p_sp_{s'}(1-A_{ss'})=1-p^{\mathsf T}Ap ∑ s , s ′ ∈ V p s p s ′ ( 1 − A s s ′ ) = 1 − p T A p from period 1 onward, when the transition probabilities are parametrized by p ∈ Δ ( V ) p\in\Delta(V) p ∈ Δ ( V ) . Hence R ( π , P ) = ∑ t = 1 + ∞ 2 − t ( 1 − p T A p ) = 1 − p T A p R(\pi,P)=\sum_{t=1}^{+\infty}2^{-t}(1-p^{\mathsf T}Ap)=1-p^{\mathsf T}Ap R ( π , P ) = ∑ t = 1 + ∞ 2 − t ( 1 − p T A p ) = 1 − p T A p . Motzkin-Straus theorem gives again that max p ∈ Δ ( V ) p T A p = 1 − ω ( G ) − 1 \max_{p\in\Delta(V)}p^{\mathsf T}Ap=1-\omega(G)^{-1} max p ∈ Δ ( V ) p T A p = 1 − ω ( G ) − 1 with ω ( G ) \omega(G) ω ( G ) the clique number of G G G . So again if we could decide min P ∈ P R ( π , P ) ≤ c \min_{P\in\mathcal{P}}R(\pi,P)\leq c min P ∈ P R ( π , P ) ≤ c with c = 1 / k c=1/k c = 1/ k for k ∈ N k\in\mathbb{N} k ∈ N we could decide ω ( G ) ≥ k \omega(G)\geq k ω ( G ) ≥ k , which concludes the reduction.
Tractability under some additional assumptions. The previous propositions suggest that we need some additional assumptions on the structure of the MDP instance to obtain favorable properties for r-rectangular uncertainty sets. Indeed, the original papers on robust MDPs with r-rectangular uncertainty sets claim tractability of the policy evaluation problem and of computing an optimal policy. This holds when we use the assumption that the rewards are independent of the next state, that is, that r ( s , a , s ′ ) = r ( s , a , s ′ ′ ) r(s,a,s') = r(s,a,s'') r ( s , a , s ′ ) = r ( s , a , s ′′ ) for all s , s ′ , s ′ ′ ∈ S s,s',s'' \in\mathcal{S} s , s ′ , s ′′ ∈ S and all a ∈ A a \in\mathcal{A} a ∈ A . In fact in this case, we can show that r-rectangular uncertainty sets are equivalent to sa-rectangular uncertainty sets, in two different ways:
We can augment the set of states to show that an r-rectangular model of uncertainty on the original robust MDP instance yields an sa-rectangular uncertainty set on the augmented robust MDP;
Even without state-augmentation, we can show that an r-rectangular uncertainty set P \mathcal{P} P yields the same Bellman operator as its sa-rectangular extension Rec s a ( P ) \operatorname{Rec}_{sa}(\mathcal{P}) Rec s a ( P ) (defined in § 4.4 ).
We make these two statements precise next. We start with the first statement below.
Proposition 4.9. Consider a robust MDP instance ( S , A , P , r , γ ) (\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma) ( S , A , P , r , γ ) with P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A r-rectangular compact. Assume that the rewards are independent of the next state and write r ( s , a ) ∈ R r(s,a) \in\mathbb{R} r ( s , a ) ∈ R for the rewards at the state-action pair ( s , a ) (s,a) ( s , a ) . Consider the following robust MDP instance ( S ^ , A ^ , P ^ , r ^ , γ ^ ) (\hat{\mathcal{S}},\hat{\mathcal{A}},\hat{\mathcal{P}},\hat{r},\hat{\gamma}) ( S ^ , A ^ , P ^ , r ^ , γ ^ ) :
The set of augmented states is S ^ = S ∪ [ r ] \hat{\mathcal{S}} = \mathcal{S} \cup[r] S ^ = S ∪ [ r ]
The set of actions is A \mathcal{A} A at s ^ ∈ S ^ \hat{s} \in\hat{\mathcal{S}} s ^ ∈ S ^ such that s ^ ∈ S \hat{s} \in\mathcal{S} s ^ ∈ S , and { 1 } \{1\} { 1 } for s ^ ∈ [ r ] \hat{s} \in[r] s ^ ∈ [ r ] (that is the decision maker has no choice at states in [ r ] [r] [ r ] ). We can identify the set of stationary policies Π ^ S \hat{\Pi}_{\mathcal{S}} Π ^ S in this augmented robust MDP instance with { ϕ ( π ) ∣ π ∈ Π S } \{\phi(\pi) \mid\pi\in\Pi_{\mathcal{S}}\} { ϕ ( π ) ∣ π ∈ Π S } for ϕ : Π S → R ( ∣ S ∣ + r ) × ( ∣ A ∣ + 1 ) \phi: \Pi_{\mathcal{S}} \to\mathbb{R}^{(|\mathcal{S}|+r)\times(|\mathcal{A}|+1)} ϕ : Π S → R ( ∣ S ∣ + r ) × ( ∣ A ∣ + 1 ) defined as
ϕ ( π ) = ( π 0 S 0 [ r ] × A 1 [ r ] ) , ∀ π ∈ Π S . \phi(\pi) = \begin{pmatrix} \pi& 0_{\mathcal{S}} \\ 0_{[r]\times\mathcal{A}} & 1_{[r]} \end{pmatrix}, \quad\forall\pi\in\Pi_{\mathcal{S}}. ϕ ( π ) = ( π 0 [ r ] × A 0 S 1 [ r ] ) , ∀ π ∈ Π S . For ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A we choose r ^ ( s , a ) = r ( s , a ) \hat{r}(s,a) = r(s,a) r ^ ( s , a ) = r ( s , a ) and r ^ ( i , 1 ) = 0 \hat{r}(i,1) = 0 r ^ ( i , 1 ) = 0 for i ∈ [ r ] i \in[r] i ∈ [ r ] .
For ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A we define P ^ s a = { ( 0 S , u s a ) } \hat{\mathcal{P}}_{sa} = \{(0_{\mathcal{S}},u_{sa})\} P ^ s a = {( 0 S , u s a )} and for i ∈ [ r ] i \in[r] i ∈ [ r ] we define P ^ i 1 = { ( w i , 0 [ r ] ) ∣ w i ∈ W i } \hat{\mathcal{P}}_{i1} = \{(w^i,0_{[r]}) \mid w^i \in W^i\} P ^ i 1 = {( w i , 0 [ r ] ) ∣ w i ∈ W i } .
The initial distribution p ^ 0 \hat{p}_0 p ^ 0 assigns probability p 0 ( s ) p_0(s) p 0 ( s ) on state s ∈ S s \in\mathcal{S} s ∈ S .
The discount factor γ ^ \hat{\gamma} γ ^ is γ ^ = γ \hat{\gamma} = \sqrt{\gamma} γ ^ = γ .
Then P ^ \hat{\mathcal{P}} P ^ is sa-rectangular and ϕ \phi ϕ is a bijection between stationary policies in the original robust MDP instance and in the lifted robust MDP instance.
Proof. The proof is a direct reformulation and is illustrated in Figure 4.4 .
Figure 4.4. Illustration for the state-augmentation reformulation from Proposition 4.9.
This proposition shows that r-rectangularity of P \mathcal{P} P is just sa-rectangularity on an augmented state space; that is, the non-rectangularity of P \mathcal{P} P on the original set of states comes from the “projection” of some components from the augmented set of states. So already we can see that the favorable tractability properties of sa-rectangularity will hold for r-rectangular models (under a compactness assumption so that we can apply the results from § 3.4 ), that is, we directly obtain the existence of stationary uniformly optimal policies for P \mathcal{P} P r-rectangular compact uncertainty sets. However we can prove even stronger results: r-rectangular uncertainty sets are equivalent to sa-rectangular uncertainty sets. This is the purpose of the next proposition.
Proposition 4.10. Let P \mathcal{P} P be a compact r-rectangular uncertainty set and assume that the rewards are independent of the next state. Then
∀ v ∈ R S , ∃ P ′ ∈ P , ∀ π ∈ Π S , ∀ s ∈ S , T π , P ′ v ( s ) = min P ∈ Rec s a ( P ) T π , P v ( s ) . \forall v \in\mathbb{R}^{\mathcal{S}},\ \exists P' \in\mathcal{P},\ \forall\pi\in\Pi_{\mathcal{S}},\ \forall s \in\mathcal{S},\ T^{\pi,P'}v(s)=\min_{P\in\operatorname{Rec}_{sa}(\mathcal{P})}T^{\pi,P}v(s). ∀ v ∈ R S , ∃ P ′ ∈ P , ∀ π ∈ Π S , ∀ s ∈ S , T π , P ′ v ( s ) = P ∈ Rec s a ( P ) min T π , P v ( s ) . where Rec s a ( P ) \operatorname{Rec}_{sa}(\mathcal{P}) Rec s a ( P ) is the sa-rectangular extension of P \mathcal{P} P defined in § 4.4 . In particular this implies
∀ v ∈ R S , ∀ π ∈ Π S , ∀ s ∈ S , min P ∈ P T π , P v ( s ) = min P ∈ Rec s a ( P ) T π , P v ( s ) . \forall v \in\mathbb{R}^{\mathcal{S}},\ \forall\pi\in\Pi_{\mathcal{S}},\ \forall s \in\mathcal{S},\ \min_{P\in\mathcal{P}}T^{\pi,P}v(s)=\min_{P\in\operatorname{Rec}_{sa}(\mathcal{P})}T^{\pi,P}v(s). ∀ v ∈ R S , ∀ π ∈ Π S , ∀ s ∈ S , P ∈ P min T π , P v ( s ) = P ∈ Rec s a ( P ) min T π , P v ( s ) .
Proof. We have by definition of Rec s a ( P ) \operatorname{Rec}_{sa}(\mathcal{P}) Rec s a ( P ) that
min P ∈ Rec s a ( P ) T π , P v ( s ) = ∑ a ∈ A π ( s , a ) r ( s , a ) + γ ∑ a ∈ A π ( s , a ) min P ∈ P P ( s , a ) ⊤ v = ∑ a ∈ A π ( s , a ) r ( s , a ) + γ ∑ a ∈ A π ( s , a ) ∑ i ∈ [ r ] u s a ( i ) min w i ∈ W i ( w i ) ⊤ v . \begin{aligned}
\min_{P\in\operatorname{Rec}_{sa}(\mathcal{P})}T^{\pi,P}v(s)
&=\sum_{a\in\mathcal{A}}\pi(s,a)r(s,a)+\gamma\sum_{a\in\mathcal{A}}\pi(s,a)\min_{P\in\mathcal{P}}P(s,a)^{\top}v\\
&=\sum_{a\in\mathcal{A}}\pi(s,a)r(s,a)+\gamma\sum_{a\in\mathcal{A}}\pi(s,a)\sum_{i\in[r]}u_{sa}(i)\min_{w^i\in W_i}(w^i)^{\top}v.
\end{aligned} P ∈ Rec s a ( P ) min T π , P v ( s ) = a ∈ A ∑ π ( s , a ) r ( s , a ) + γ a ∈ A ∑ π ( s , a ) P ∈ P min P ( s , a ) ⊤ v = a ∈ A ∑ π ( s , a ) r ( s , a ) + γ a ∈ A ∑ π ( s , a ) i ∈ [ r ] ∑ u s a ( i ) w i ∈ W i min ( w i ) ⊤ v .
By choosing factors in arg min w i ∈ W i ( w i ) ⊤ v \operatorname*{arg\,min}_{w^i\in W_i}(w^i)^{\top}v arg min w i ∈ W i ( w i ) ⊤ v for each i ∈ [ r ] i\in[r] i ∈ [ r ] we recover P ′ P' P ′ feasible in P \mathcal{P} P , independent of π \pi π , s s s , such that T π , P ′ v ( s ) = min P ∈ Rec s a ( P ) T π , P v ( s ) T^{\pi,P'}v(s)=\min_{P\in\operatorname{Rec}_{sa}(\mathcal{P})}T^{\pi,P}v(s) T π , P ′ v ( s ) = min P ∈ Rec s a ( P ) T π , P v ( s ) , ∀ s ∈ S \forall s\in\mathcal{S} ∀ s ∈ S as claimed. The second implication is because P ⊂ Rec s a ( P ) \mathcal{P}\subset\operatorname{Rec}_{sa}(\mathcal{P}) P ⊂ Rec s a ( P ) . □ \square □
A rephrasing of the above proposition is that the set of operators { T π , P ∣ P ∈ P } \{T^{\pi,P}\mid P\in\mathcal{P}\} { T π , P ∣ P ∈ P } satisfies the attainability property with the contraction v ↦ ( min P ∈ Rec s a ( P ) T π , P v ( s ) ) s ∈ S v\mapsto(\min_{P\in\operatorname{Rec}_{sa}(\mathcal{P})}T^{\pi,P}v(s))_{s\in\mathcal{S}} v ↦ ( min P ∈ Rec s a ( P ) T π , P v ( s ) ) s ∈ S . Note that we are not claiming that P \mathcal{P} P and Rec s a ( P ) \operatorname{Rec}_{sa}(\mathcal{P}) Rec s a ( P ) coincide (in particular P \mathcal{P} P is only r-rectangular so it may not be sa-rectangular), but rather that minimizing P ↦ T π , P v ( s ) P\mapsto T^{\pi,P}v(s) P ↦ T π , P v ( s ) over P \mathcal{P} P and Rec s a ( P ) \operatorname{Rec}_{sa}(\mathcal{P}) Rec s a ( P ) yields the same optimal objectives. The above proposition is enough to show that P \mathcal{P} P can be replaced by its sa-rectangular extension in the following sense.
Proposition 4.11. Let P \mathcal{P} P be a compact r r r -rectangular uncertainty set and assume that the rewards are independent of the next state. Let π ∈ Π S \pi\in\Pi_{\mathrm{S}} π ∈ Π S . Then
min P ∈ P R ( π , P ) = min P ∈ Rec s a ( P ) R ( π , P ) . \min_{P\in\mathcal{P}} R(\pi,P)=\min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}R(\pi,P). P ∈ P min R ( π , P ) = P ∈ Rec sa ( P ) min R ( π , P ) . Proof. We have
min P ∈ Rec s a ( P ) R ( π , P ) = p 0 T v ^ π \min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}R(\pi,P)=p_0^{\mathsf{T}}\hat{v}^{\pi} P ∈ Rec sa ( P ) min R ( π , P ) = p 0 T v ^ π with v ^ π \hat{v}^{\pi} v ^ π the unique fixed point of
v ↦ ( min P ∈ Rec s a ( P ) T π , P v ( s ) ) s ∈ S . v\mapsto\left(\min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}T^{\pi,P}v(s)\right)_{s\in\mathcal{S}}. v ↦ ( P ∈ Rec sa ( P ) min T π , P v ( s ) ) s ∈ S . But we know that { T π , P ∣ P ∈ P } \{T^{\pi,P}\mid P\in\mathcal{P}\} { T π , P ∣ P ∈ P } satisfies the attainability property with this contraction. Hence v ^ π = v π , P π \hat{v}^{\pi}=v^{\pi,P^{\pi}} v ^ π = v π , P π for some feasible P π ∈ P P^{\pi}\in\mathcal{P} P π ∈ P . Hence min P ∈ Rec s a ( P ) R ( π , P ) = R ( π , P π ) ≥ min P ∈ P R ( π , P ) \min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}R(\pi,P)=R(\pi,P^{\pi})\geq\min_{P\in\mathcal{P}}R(\pi,P) min P ∈ Rec sa ( P ) R ( π , P ) = R ( π , P π ) ≥ min P ∈ P R ( π , P ) . Since the reverse inequality is always true (given P ⊂ Rec s a ( P ) \mathcal{P}\subset\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P}) P ⊂ Rec sa ( P ) ) we get that min P ∈ P R ( π , P ) = min P ∈ Rec s a ( P ) R ( π , P ) \min_{P\in\mathcal{P}}R(\pi,P)=\min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}R(\pi,P) min P ∈ P R ( π , P ) = min P ∈ Rec sa ( P ) R ( π , P ) .
Combining the arguments above proves the following corollary.
Corollary 4.1. Let P \mathcal{P} P be an r r r -rectangular compact uncertainty set and assume that the rewards are independent of the next state. Then the following properties hold:
We have
sup π ∈ Π H inf P ∈ P R ( π , P ) = max π ∈ Π S min P ∈ P R ( π , P ) = max π ∈ Π S min P ∈ Rec s a ( P ) R ( π , P ) . \sup_{\pi\in\Pi_{\mathrm{H}}}\inf_{P\in\mathcal{P}}R(\pi,P)=\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\mathcal{P}}R(\pi,P)=\max_{\pi\in\Pi_{\mathrm{S}}}\min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}R(\pi,P). π ∈ Π H sup P ∈ P inf R ( π , P ) = π ∈ Π S max P ∈ P min R ( π , P ) = π ∈ Π S max P ∈ Rec sa ( P ) min R ( π , P ) . Optimal policies. The set of stationary deterministic uniformly optimal policies for P \mathcal{P} P coincides with the set of stationary deterministic uniformly optimal policies for Rec s a ( P ) \operatorname{Rec}_{\mathrm{sa}}(\mathcal{P}) Rec sa ( P ) . In particular the set of stationary deterministic uniformly optimal policies for P \mathcal{P} P is non-empty and can be computed with the algorithms outlined in § 3.4 .
Proof. 1. Let π ⋆ \pi^{\star} π ⋆ be a stationary deterministic uniformly optimal policy for Rec s a ( P ) \operatorname{Rec}_{\mathrm{sa}}(\mathcal{P}) Rec sa ( P ) , and P ⋆ P^{\star} P ⋆ such that P ⋆ ∈ arg min P ∈ Rec s a ( P ) R ( π ⋆ , P ) P^{\star}\in\arg\min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}R(\pi^{\star},P) P ⋆ ∈ arg min P ∈ Rec sa ( P ) R ( π ⋆ , P ) , π ⋆ ∈ arg max π ∈ Π S D R ( π , P ⋆ ) \pi^{\star}\in\arg\max_{\pi\in\Pi_{\mathrm{SD}}}R(\pi,P^{\star}) π ⋆ ∈ arg max π ∈ Π SD R ( π , P ⋆ ) .
We know that R ( π , P ⋆ ) ≤ R ( π ⋆ , P ⋆ ) R(\pi,P^{\star})\leq R(\pi^{\star},P^{\star}) R ( π , P ⋆ ) ≤ R ( π ⋆ , P ⋆ ) for any π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H . From Proposition 4.10 we can actually choose P ⋆ ∈ P P^{\star}\in\mathcal{P} P ⋆ ∈ P , hence sup π ∈ Π H min P ∈ P R ( π , P ) ≤ R ( π ⋆ , P ⋆ ) \sup_{\pi\in\Pi_{\mathrm{H}}}\min_{P\in\mathcal{P}}R(\pi,P)\leq R(\pi^{\star},P^{\star}) sup π ∈ Π H min P ∈ P R ( π , P ) ≤ R ( π ⋆ , P ⋆ ) . But R ( π ⋆ , P ⋆ ) = max π ∈ Π S D min P ∈ Rec s a ( P ) R ( π , P ) = sup π ∈ Π H min P ∈ Rec s a ( P ) R ( π , P ) R(\pi^{\star},P^{\star})=\max_{\pi\in\Pi_{\mathrm{SD}}}\min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}R(\pi,P)=\sup_{\pi\in\Pi_{\mathrm{H}}}\min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}R(\pi,P) R ( π ⋆ , P ⋆ ) = max π ∈ Π SD min P ∈ Rec sa ( P ) R ( π , P ) = sup π ∈ Π H min P ∈ Rec sa ( P ) R ( π , P ) , hence we have proved
sup π ∈ Π H min P ∈ P R ( π , P ) ≤ max π ∈ Π S D min P ∈ Rec s a ( P ) R ( π , P ) = sup π ∈ Π H min P ∈ Rec s a ( P ) R ( π , P ) . \sup_{\pi\in\Pi_{\mathrm{H}}}\min_{P\in\mathcal{P}}R(\pi,P)\leq\max_{\pi\in\Pi_{\mathrm{SD}}}\min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}R(\pi,P)=\sup_{\pi\in\Pi_{\mathrm{H}}}\min_{P\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}R(\pi,P). π ∈ Π H sup P ∈ P min R ( π , P ) ≤ π ∈ Π SD max P ∈ Rec sa ( P ) min R ( π , P ) = π ∈ Π H sup P ∈ Rec sa ( P ) min R ( π , P ) . Since P ⊂ Rec s a ( P ) \mathcal{P}\subset\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P}) P ⊂ Rec sa ( P ) we must have that all terms above are equal, which concludes the proof for the first statement.
Proposition 4.11 shows that every stationary policy has the same worst-case value over P \mathcal{P} P and over Rec s a ( P ) \operatorname{Rec}_{\mathrm{sa}}(\mathcal{P}) Rec sa ( P ) , and that the two maxmin problems have the same optimal value. Hence a stationary deterministic policy is uniformly optimal for P \mathcal{P} P if and only if it is uniformly optimal for Rec s a ( P ) \operatorname{Rec}_{\mathrm{sa}}(\mathcal{P}) Rec sa ( P ) .
The fact that optimal policies for r r r -rectangular compact uncertainty sets can be chosen deterministic implies that some s s s -rectangular sets are not r r r -rectangular (for instance we saw in the previous chapter that this is the case for Example 1.1 with p ∈ [ 0 , 1 ] p\in[0,1] p ∈ [ 0 , 1 ] ).
Generalizations. Proposition 4.11 has highlighted an interesting property: some uncertainty sets P \mathcal{P} P are such that the induced set of operators { T π , P ∣ P ∈ P } \{T^{\pi,\mathcal{P}} \mid\mathcal{P} \in\mathcal{P}\} { T π , P ∣ P ∈ P } has the attainability property for v ↦ ( min P ∈ Rec s a ( P ) T π , P v ( s ) ) s ∈ S v \mapsto(\min_{\mathcal{P}\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})} T^{\pi,\mathcal{P}}v(s))_{s\in\mathcal{S}} v ↦ ( min P ∈ Rec sa ( P ) T π , P v ( s ) ) s ∈ S (in particular this is true for r-rectangular uncertainty sets when the rewards are independent of the next state). Corollary 4.1 shows that this attainability property yields that P \mathcal{P} P inherits the good properties of Rec s a ( P ) \operatorname{Rec}_{\mathrm{sa}}(\mathcal{P}) Rec sa ( P ) in terms of optimal policies and tractability (we don’t use at all r-rectangularity for Corollary 4.1, only the attainability outlined above). This yields the following two interesting questions:
First question: are there other classes of uncertainty sets P \mathcal{P} P such that, when the rewards are independent of the next state, we have
∀ v ∈ R S , ∃ P ′ ∈ P , ∀ π ∈ Π S , ∀ s ∈ S , T π , P ′ v ( s ) = min P ∈ Rec s a ( P ) T π , P v ( s ) . \forall v \in\mathbb{R}^{\mathcal{S}},\ \exists\mathcal{P}' \in\mathcal{P},\ \forall\pi\in\Pi_{\mathcal{S}},\ \forall s \in\mathcal{S},\ T^{\pi,\mathcal{P}'}v(s)=\min_{\mathcal{P}\in\operatorname{Rec}_{\mathrm{sa}}(\mathcal{P})}T^{\pi,\mathcal{P}}v(s). ∀ v ∈ R S , ∃ P ′ ∈ P , ∀ π ∈ Π S , ∀ s ∈ S , T π , P ′ v ( s ) = P ∈ Rec sa ( P ) min T π , P v ( s ) .
Second question: how about the same question with s-rectangular extensions, that is, are there other classes of uncertainty sets P \mathcal{P} P such that when rewards are independent of the next state, we have
∀ v ∈ R S , ∀ π ∈ Π S , ∃ P ′ ∈ P , ∀ s ∈ S , T π , P ′ v ( s ) = min P ∈ Rec s ( P ) T π , P v ( s ) . \forall v \in\mathbb{R}^{\mathcal{S}},\ \forall\pi\in\Pi_{\mathcal{S}},\ \exists\mathcal{P}' \in\mathcal{P},\ \forall s \in\mathcal{S},\ T^{\pi,\mathcal{P}'}v(s)=\min_{\mathcal{P}\in\operatorname{Rec}_{\mathrm{s}}(\mathcal{P})}T^{\pi,\mathcal{P}}v(s). ∀ v ∈ R S , ∀ π ∈ Π S , ∃ P ′ ∈ P , ∀ s ∈ S , T π , P ′ v ( s ) = P ∈ Rec s ( P ) min T π , P v ( s ) .
Note the slightly weaker formulation for the second question compared to the first one (I invert the positions of the quantifiers for the kernel P ′ ∈ P \mathcal{P}' \in\mathcal{P} P ′ ∈ P and the policy π ∈ Π S \pi\in\Pi_{\mathcal{S}} π ∈ Π S ); this is because the second question involves the s-rectangular extension of P \mathcal{P} P , so that the choice of policy may matter for this formulation, whereas the sa-rectangular extension is state-action wise. The answers to both questions are positive, and several classes of non-rectangular uncertainty sets have been proposed that satisfy these attainability properties. I give here two very simple examples (left as an exercise):
Consider P ⊂ Δ ( S ) S × A \mathcal{P} \subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A such that
P = { ( ∑ i = 1 r u s a ( i ) w i ) s a | ( w 1 , … , w r ) ∈ W , ( u s a ) s a ∈ U } \mathcal{P}=\left\{\left(\sum_{i=1}^{r}u_{\mathrm{sa}}(i)w^{i}\right)_{\mathrm{sa}}\middle|(w^{1},\ldots,w^{r})\in\mathcal{W},\ (u_{\mathrm{sa}})_{\mathrm{sa}}\in\mathcal{U}\right\} P = { ( i = 1 ∑ r u sa ( i ) w i ) sa ( w 1 , … , w r ) ∈ W , ( u sa ) sa ∈ U }
W = × i ∈ [ r ] W i , W i ⊂ Δ ( S ) , ∀ i ∈ [ r ] , \mathcal{W}=\times_{i\in[r]}\mathcal{W}^{i},\ \mathcal{W}^{i}\subset\Delta(\mathcal{S}),\ \forall i\in[r], W = × i ∈ [ r ] W i , W i ⊂ Δ ( S ) , ∀ i ∈ [ r ] ,
U = × ( s , a ) ∈ S × A U s a , U s a ⊂ Δ ( [ r ] ) , ∀ ( s , a ) ∈ S × A . (72) \mathcal{U}=\times_{(s,a)\in\mathcal{S}\times\mathcal{A}}\mathcal{U}^{\mathrm{sa}},\ \mathcal{U}^{\mathrm{sa}}\subset\Delta([r]),\ \forall(s,a)\in\mathcal{S}\times\mathcal{A}. \tag*{(72)} U = × ( s , a ) ∈ S × A U sa , U sa ⊂ Δ ([ r ]) , ∀ ( s , a ) ∈ S × A . (72)
Then P \mathcal{P} P provides a positive answer to the first question (when W i \mathcal{W}^{i} W i and U s a \mathcal{U}^{\mathrm{sa}} U sa are compact sets for i ∈ [ r ] i\in[r] i ∈ [ r ] and ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A ).
Let P ⊂ Δ ( S ) S × A \mathcal{P}\subset\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ⊂ Δ ( S ) S × A be an uncertainty set and assume that S \mathcal{S} S can be partitioned into S 1 \mathcal{S}_{1} S 1 and S 2 \mathcal{S}_{2} S 2 such that the transitions out of the states in S 1 \mathcal{S}_{1} S 1 are s-rectangular while the transitions out of S 2 \mathcal{S}_{2} S 2 are r-rectangular:
P = P 1 × P 2 , P 1 ⊂ Δ ( S ) S 1 × A , P 2 ⊂ Δ ( S ) S 2 × A , \mathcal{P}=\mathcal{P}_{1}\times\mathcal{P}_{2},\ \mathcal{P}_{1}\subset\Delta(\mathcal{S})^{\mathcal{S}_{1}\times\mathcal{A}},\ \mathcal{P}_{2}\subset\Delta(\mathcal{S})^{\mathcal{S}_{2}\times\mathcal{A}}, P = P 1 × P 2 , P 1 ⊂ Δ ( S ) S 1 × A , P 2 ⊂ Δ ( S ) S 2 × A ,
P 1 = × s ∈ S 1 P s , P s ⊂ Δ ( S ) A , \mathcal{P}_{1}=\times_{s\in\mathcal{S}_{1}}\mathcal{P}_{s},\ \mathcal{P}_{s}\subset\Delta(\mathcal{S})^{\mathcal{A}}, P 1 = × s ∈ S 1 P s , P s ⊂ Δ ( S ) A ,
P 2 = { ( ∑ i = 1 r u s a ( i ) w i ) s ∈ S 2 , a ∈ A | ( w 1 , … , w r ) ∈ W 1 × ⋯ × W r } . (73) \mathcal{P}_{2}=\left\{\left(\sum_{i=1}^{r}u_{\mathrm{sa}}(i)w^{i}\right)_{s\in\mathcal{S}_{2},a\in\mathcal{A}}\middle|(w^{1},\ldots,w^{r})\in\mathcal{W}^{1}\times\cdots\times\mathcal{W}^{r}\right\}. \tag*{(73)} P 2 = ⎩ ⎨ ⎧ ( i = 1 ∑ r u sa ( i ) w i ) s ∈ S 2 , a ∈ A ( w 1 , … , w r ) ∈ W 1 × ⋯ × W r ⎭ ⎬ ⎫ . (73)
Then P \mathcal{P} P gives a positive answer to the second question (when P 1 \mathcal{P}_1 P 1 and P 2 \mathcal{P}_2 P 2 are compact sets).
It is not hard to come up with other classes of uncertainty sets that give a positive answer to the first or second question above (see references of this chapter). However in some sense (more precisely: regarding policy evaluation), all of these uncertainty sets are “equivalent” to their sa-rectangular extensions (for those with a positive answer to the first question above) or their s-rectangular extensions (for those with a positive answer to the second question above), so perhaps it is easier to work with s-rectangular or sa-rectangular uncertainty sets directly.
Lightning-Does-not-Strike-Twice uncertainty set
Another popular model is the so-called Lightning-Does-not-Strike-Twice (LDST) model of uncertainty, and its generalization, k k k -rectangularity. In the (adaptive) LDST model we are given a nominal kernel P ^ ∈ Δ ( S ) S × A \widehat{P} \in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}} P ∈ Δ ( S ) S × A , and some s-rectangular uncertainty set P = × s P s \mathcal{P} = \times_{s}\mathcal{P}_{s} P = × s P s , assumed to be compact here, such that P ^ ∈ P \widehat{P} \in\mathcal{P} P ∈ P . The transition probabilities are allowed to vary over time and the dynamics unfold as follows: at period t t t , the decision maker observes the current state s t s_t s t and all the past choices P t ′ ( s t ′ ) ∈ Δ ( S ) A P^{t'}(s_{t'}) \in\Delta(\mathcal{S})^{\mathcal{A}} P t ′ ( s t ′ ) ∈ Δ ( S ) A chosen by the adversary at the past periods t ′ < t t' < t t ′ < t out of the past states s 0 , … , s t − 1 s_0, \ldots, s_{t-1} s 0 , … , s t − 1 . Based on this information, the decision maker chooses a t a_t a t , the adversary observes s t , a t s_t,a_t s t , a t and chooses the transition probabilities P t ( s t ) P^t(s_t) P t ( s t ) out of s t s_t s t for period t t t . The next state s t + 1 ∈ S s_{t+1} \in\mathcal{S} s t + 1 ∈ S is sampled from P t ( s t , a t ) ∈ Δ ( S ) P^t(s_t,a_t) \in\Delta(\mathcal{S}) P t ( s t , a t ) ∈ Δ ( S ) . The constraint for the adversary is that along each realized infinite trajectory, the transition probabilities can deviate at most k k k times from the nominal kernel P ^ \widehat{P} P , where the integer k ∈ N k \in\mathbb{N} k ∈ N is given and known to the decision maker and the adversary. Note the difference with the previous robust MDP model considered in these notes: in the LDST model, at any period t ∈ N t \in\mathbb{N} t ∈ N the decision maker is supposed to be able to know the cardinality of { t ′ ∣ P t ′ ( s t ′ ) ≠ P ^ ( s t ′ ) , t ′ < t } \{t' \mid P^{t'}(s_{t'}) \ne\widehat{P}(s_{t'}), t' < t\} { t ′ ∣ P t ′ ( s t ′ ) = P ( s t ′ ) , t ′ < t } , i.e., the decision maker knows how many times the kernel has deviated from P ^ \widehat{P} P and can calculate how much of its “budget of deviation” the adversary has used until period t t t . We emphasize that this is a significant departure from the robust MDP model studied in the rest of these notes, where the decision maker makes their decision at period t ∈ N t \in\mathbb{N} t ∈ N based on ( s 0 , a 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t ) (s_0,a_0,s_1,a_1,\ldots,s_{t-1},a_{t-1},s_t) ( s 0 , a 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t ) the current history up to period t t t , whereas in the LDST model the decision maker makes its decision at period t t t based on ( s 0 , a 0 , k 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t , k t ) (s_0,a_0,k_0,s_1,a_1,\ldots,s_{t-1},a_{t-1},s_t,k_t) ( s 0 , a 0 , k 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t , k t ) where k t = ∑ t ′ = 0 t − 1 1 { P t ′ ( s t ′ ) ≠ P ^ ( s t ′ ) } k_t = \sum_{t'=0}^{t-1}\mathbf{1}_{\{P^{t'}(s_{t'})\ne\widehat{P}(s_{t'})\}} k t = ∑ t ′ = 0 t − 1 1 { P t ′ ( s t ′ ) = P ( s t ′ )} , and the adversary bases its decision on ( s 0 , a 0 , k 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t , k t , a t ) (s_0,a_0,k_0,s_1,a_1,\ldots,s_{t-1},a_{t-1},s_t,k_t,a_t) ( s 0 , a 0 , k 0 , s 1 , a 1 , … , s t − 1 , a t − 1 , s t , k t , a t ) . In particular one may wonder how the decision maker, which in principle only observing the sequence of states, could infer that the transitions ( s t , a t ) → s t + 1 (s_t,a_t) \to s_{t+1} ( s t , a t ) → s t + 1 happened based on some particular distribution P ( s t , a t ) ∈ Δ ( S ) P(s_t,a_t) \in\Delta(\mathcal{S}) P ( s t , a t ) ∈ Δ ( S ) over the next states. That could be the case if s t + 1 s_{t+1} s t + 1 does not belong to the support of P ^ ( s t , a t ) \widehat{P}(s_t,a_t) P ( s t , a t ) . Here it is interesting to compare the LDST model for the sets of information available to the decision maker and adversary, and the connection we made between sa-rectangular robust MDPs and perfect-information stochastic games in § 5.1 . In particular, it is straightforward to reformulate robust MDPs with the LDST model of uncertainty as a perfect-information stochastic game as follows:
The set of states of the perfect-information stochastic game is S × { 0 , … , k } ∪ ( S × { 0 , … , k } × A ) \mathcal{S} \times\{0,\ldots,k\} \cup(\mathcal{S} \times\{0,\ldots,k\} \times\mathcal{A}) S × { 0 , … , k } ∪ ( S × { 0 , … , k } × A ) , which we can represent visually as two stacks: a first stack of $k+1
copies of S \mathcal{S} S and a second stack of k + 1 k+1 k + 1 copies of S × A \mathcal{S}\times\mathcal{A} S × A . The decision maker controls the states in S × { 0 , … , k } \mathcal{S}\times\{0,\ldots,k\} S × { 0 , … , k } and the adversary controls the states in S × { 0 , … , k } × A \mathcal{S}\times\{0,\ldots,k\}\times\mathcal{A} S × { 0 , … , k } × A .
Let ( s , k ′ ) ∈ S × { 0 , … , k − 1 } (s,k')\in\mathcal{S}\times\{0,\ldots,k-1\} ( s , k ′ ) ∈ S × { 0 , … , k − 1 } . When the decision maker chooses action a ∈ A a\in\mathcal{A} a ∈ A in the state ( s , k ′ ) (s,k') ( s , k ′ ) the transition is deterministic to ( s , k ′ , a ) (s,k',a) ( s , k ′ , a ) . When in ( s , k ′ , a ) (s,k',a) ( s , k ′ , a ) the adversary chooses the current transition probabilities P ( s , a ) ∈ P s a P(s,a)\in\mathcal{P}_{sa} P ( s , a ) ∈ P s a . If P ^ ( s , a ) ≠ P ( s , a ) \hat{P}(s,a)\ne P(s,a) P ^ ( s , a ) = P ( s , a ) then the transition is to ( s ′ , k ′ + 1 ) (s',k'+1) ( s ′ , k ′ + 1 ) with probability P ( s , a , s ′ ) P(s,a,s') P ( s , a , s ′ ) , otherwise the transition is to ( s ′ , k ′ ) (s',k') ( s ′ , k ′ ) with probability P ( s , a , s ′ ) P(s,a,s') P ( s , a , s ′ ) . In both cases, the rewards are obtained at the transition ( s , k ′ , a ) (s,k',a) ( s , k ′ , a ) to ( s ′ , k ′ ′ ) (s',k'') ( s ′ , k ′′ ) with k ′ ′ ∈ { k ′ , k ′ + 1 } k''\in\{k',k'+1\} k ′′ ∈ { k ′ , k ′ + 1 } (its value depending on whether P ^ \hat{P} P ^ was chosen at period t t t ) and are equal to P ( s , a ) ⊤ r ( s , a ) ∈ R P(s,a)^\top r(s,a)\in\mathbb{R} P ( s , a ) ⊤ r ( s , a ) ∈ R .
For the states in the form ( s , k ) (s,k) ( s , k ) the situation is the same for the decision maker but the adversary is constrained to choosing P ^ ( s , a ) \hat{P}(s,a) P ^ ( s , a ) , i.e., the MDP becomes “stuck” in the last level of the stacked copies of S \mathcal{S} S and S × A \mathcal{S}\times\mathcal{A} S × A (since the budget of deviations is exhausted).
The resulting perfect-information SG is represented in Figure 4.5 . For the sake of conciseness, for a given v ∈ R S × { 0 , … , k } v\in\mathbb{R}^{\mathcal{S}\times\{0,\ldots,k\}} v ∈ R S × { 0 , … , k } we write v ( : , k ′ ) ∈ R S v(:,k')\in\mathbb{R}^{\mathcal{S}} v ( : , k ′ ) ∈ R S for ( v ( s , k ′ ) ) s ∈ S (v(s,k'))_{s\in\mathcal{S}} ( v ( s , k ′ ) ) s ∈ S , and similarly we write v ( : , k ′ , a ) ∈ R S v(:,k',a)\in\mathbb{R}^{\mathcal{S}} v ( : , k ′ , a ) ∈ R S for ( v ( s , k ′ , a ) ) s ∈ S (v(s,k',a))_{s\in\mathcal{S}} ( v ( s , k ′ , a ) ) s ∈ S given v ∈ R S × { 0 , … , k } × A v\in\mathbb{R}^{\mathcal{S}\times\{0,\ldots,k\}\times\mathcal{A}} v ∈ R S × { 0 , … , k } × A . With this notation, the optimality (fixed point) equations for ( v 1 , v 2 ) ∈ R S × { 0 , … , k } × R S × { 0 , … , k } × A (v_1,v_2)\in\mathbb{R}^{\mathcal{S}\times\{0,\ldots,k\}}\times\mathbb{R}^{\mathcal{S}\times\{0,\ldots,k\}\times\mathcal{A}} ( v 1 , v 2 ) ∈ R S × { 0 , … , k } × R S × { 0 , … , k } × A are
Figure 4.5. The perfect-information stochastic game reformulation of robust MDPs with Lightning-Does-not-Strike-Twice (LDST) model of uncertainty.
v 1 ( s , k ′ ) = max a ∈ A γ ⋅ v 2 ( s , k ′ , a ) , ∀ s ∈ S , k ′ ∈ { 0 , … , k } , v_1(s,k')=\max_{a\in\mathcal{A}}\sqrt{\gamma}\cdot v_2(s,k',a),\quad\forall\ s\in\mathcal{S},\ k'\in\{0,\ldots,k\}, v 1 ( s , k ′ ) = a ∈ A max γ ⋅ v 2 ( s , k ′ , a ) , ∀ s ∈ S , k ′ ∈ { 0 , … , k } ,
v 2 ( s , k ′ , a ) = min { P ^ ( s , a ) ⊤ ( r ( s , a ) + γ v 1 ( : , k ′ ) ) , min P ( s , a ) ∈ P s a P ( s , a ) ⊤ ( r ( s , a ) + γ v 1 ( : , k ′ + 1 ) ) } , ∀ ( s , a ) ∈ S × A , k ′ < k , \begin{aligned}
v_2(s,k',a)=\min\bigg\{&\hat{P}(s,a)^\top\left(r(s,a)+\sqrt{\gamma}v_1(:,k')\right),\\
&\min_{P(s,a)\in\mathcal{P}_{sa}}P(s,a)^\top\left(r(s,a)+\sqrt{\gamma}v_1(:,k'+1)\right)\bigg\},\quad\forall\ (s,a)\in\mathcal{S}\times\mathcal{A},\ k'<k,
\end{aligned} v 2 ( s , k ′ , a ) = min { P ^ ( s , a ) ⊤ ( r ( s , a ) + γ v 1 ( : , k ′ ) ) , P ( s , a ) ∈ P s a min P ( s , a ) ⊤ ( r ( s , a ) + γ v 1 ( : , k ′ + 1 ) ) } , ∀ ( s , a ) ∈ S × A , k ′ < k ,
v 2 ( s , k , a ) = P ^ ( s , a ) ⊤ ( r ( s , a ) + γ v 1 ( : , k ) ) , ∀ ( s , a ) ∈ S × A v_2(s,k,a)=\hat{P}(s,a)^\top\left(r(s,a)+\sqrt{\gamma}v_1(:,k)\right),\quad\forall\ (s,a)\in\mathcal{S}\times\mathcal{A} v 2 ( s , k , a ) = P ^ ( s , a ) ⊤ ( r ( s , a ) + γ v 1 ( : , k ) ) , ∀ ( s , a ) ∈ S × A
The third line above is the case where the adversary has exhausted all the budget, in which case k ′ = k k'=k k ′ = k and the adversary must choose P ^ \hat{P} P ^ . The second line above is the case where already k ′ < k k'<k k ′ < k deviations have occurred in the past and the adversary can choose to “consume” one more unit of deviation (choose some P ( s , a ) P(s,a) P ( s , a ) , eventually with P ( s , a ) ≠ P ^ ( s , a ) P(s,a)\ne\hat{P}(s,a) P ( s , a ) = P ^ ( s , a ) ) or not (stick to the nominal kernel). Note that a pair ( v 1 ⋆ , v 2 ⋆ ) (v_1^\star,v_2^\star) ( v 1 ⋆ , v 2 ⋆ ) of solutions to the fixed point equations above exists because the relevant operators are contractions, and in fact on the augmented state set we have a problem similar to Model 1 from § 4.4 , so that we can invoke Theorem 4.5 and Proposition 4.4 to conclude that there exists a uniformly optimal stationary policy (stationary in the augmented states). Note that this does not require convexity of P \mathcal{P} P in the first place, only compactness to invoke results for sa-rectangular robust MDPs. This is because the decisions of the adversary happen after observing the action a t a_t a t picked by the decision maker. In particular, given a pair ( v 1 ⋆ , v 2 ⋆ ) (v_1^\star,v_2^\star) ( v 1 ⋆ , v 2 ⋆ ) of solutions to the above equations we get that an optimal policy for the decision maker can be chosen as a function of the current state s s s and the current number of “kernel deviations” k ′ k' k ′ in the arg max in the first equation above, and similarly for the adversary. It is worth emphasizing that the resulting policy of the decision maker is not stationary in the sense defined in Chapter 1, since it may in general depend on k ′ k' k ′ , so that when visiting the same state s s s over time an optimal decision maker may choose different decision rules. However, an optimal policy is stationary in the augmented state ( s , k ′ ) (s,k') ( s , k ′ ) , so in some sense in the LDST model we have a “summary statistic” that we can use to make our decisions: even though an optimal policy may be non-stationary over S \mathcal{S} S we still need only a small amount of information to make our decisions.
Remark 4.3. In principle, we should divide the rewards in the stochastic game (SG) by γ \sqrt{\gamma} γ so that the return in the LDST robust MDP and the SG are the same; in the construction above there is an additional γ \sqrt{\gamma} γ multiplicative factor for the return in the SG compared to that of the robust MDP; of course this does not change the set of optimal policies, so we avoid this rescaling above for the sake of readability.
Remark 4.4. The dynamic programming equations count the number of deviations of the vector P ( s , a ) ≠ P ^ ( s , a ) P(s,a) \ne\hat{P}(s,a) P ( s , a ) = P ^ ( s , a ) instead of the “block” P ( s ) ≠ P ^ ( s ) P(s) \ne\hat{P}(s) P ( s ) = P ^ ( s ) . However for LDST the adversary chooses P ( s t ) P(s_t) P ( s t ) after observing a t a_t a t so these two models are the same.
k-rectangular uncertainty set
The model of k k k -rectangularity is a natural extension of the LDST model of uncertainty for the case of finite horizon robust MDPs. Assume that there are H H H periods of decisions { 0 , … , H − 1 } \{0,\ldots,H-1\} { 0 , … , H − 1 } and the sets of states S 0 , … , S H \mathcal{S}_0,\ldots,\mathcal{S}_H S 0 , … , S H visited at each period are all pairwise disjoint (i.e., a state is visited at most once along a trajectory); the set S H \mathcal{S}_H S H is the set of terminal states. We write S = S 0 ∪ … ∪ S H \mathcal{S}=\mathcal{S}_0\cup\ldots\cup\mathcal{S}_H S = S 0 ∪ … ∪ S H for the set of states. We are given an uncertainty set P \mathcal{P} P for the transition probabilities across states. The dynamics unfold as follows: at each period t t t , in the current state s t ∈ S t s_t\in\mathcal{S}_t s t ∈ S t ,
the decision maker will pick an action a t a_t a t and the adversary will pick a block P ( s t ) ∈ Δ ( S ) A P(s_t)\in\Delta(\mathcal{S})^{\mathcal{A}} P ( s t ) ∈ Δ ( S ) A for the transitions out of s t s_t s t .
The decision maker can observe the trajectory ( s 0 , a 0 , … , s t ) (s_0,a_0,\ldots,s_t) ( s 0 , a 0 , … , s t ) up to period t t t as well as the past choices P ( s 0 ) , … , P ( s t − 1 ) ∈ Δ ( S ) A P(s_0),\ldots,P(s_{t-1})\in\Delta(\mathcal{S})^{\mathcal{A}} P ( s 0 ) , … , P ( s t − 1 ) ∈ Δ ( S ) A of the adversary.
The decision maker first chooses their action a t a_t a t based on this information, then the adversary observes a t a_t a t and chooses P t ( s t ) ∈ Δ ( S ) A P^t(s_t) \in\Delta(\mathcal{S})^\mathcal{A} P t ( s t ) ∈ Δ ( S ) A . The next state s t + 1 s_{t+1} s t + 1 is sampled from P ( s t , a t ) ∈ Δ ( S ) P(s_t,a_t) \in\Delta(\mathcal{S}) P ( s t , a t ) ∈ Δ ( S ) and P t ( s t ) P^t(s_t) P t ( s t ) is revealed to the decision maker.
The objective here is the finite-horizon (total) return
R ( π , P ) = E π , P [ ∑ t = 0 H − 1 r ( s t , a t , s t + 1 ) ] R(\pi,P)=\mathbb{E}^{\pi,P}\left[\sum_{t=0}^{H-1}r(s_t,a_t,s_{t+1})\right] R ( π , P ) = E π , P [ t = 0 ∑ H − 1 r ( s t , a t , s t + 1 ) ]
Note that this is close to Model 1 in § 4.4 except that the decision maker can observe the realizations of the past transition probabilities (instead of the realized next state). Note that we can write P ( s t ) P(s_t) P ( s t ) instead of P t ( s t ) P^t(s_t) P t ( s t ) because each state is visited at most once. There are two main departures from the robust MDP model studied in these notes (apart from the finite-horizon assumption, which we can always model as an infinite-horizon problem anyway by adding some terminal absorbing states with rewards 0):
Along each realized trajectory, the decision maker observes the realizations of the successive adversary choices of the entire P ( s t ) ∈ Δ ( S ) A P(s_t) \in\Delta(\mathcal{S})^\mathcal{A} P ( s t ) ∈ Δ ( S ) A at each period t t t after taking their action a t a_t a t , instead of solely the realization of s t + 1 s_{t+1} s t + 1 as in the robust MDP problem studied so far.
Given the first condition, it makes sense to assume that we visit each state-action pair at most once (this is the assumption that S 0 , … , S H \mathcal{S}_0,\ldots,\mathcal{S}_H S 0 , … , S H are pairwise disjoint; otherwise the adversary would be constrained to take the same choice again, to keep compatibility with a kernel in P \mathcal{P} P ). This first condition naturally leads to the definition of the projection of the uncertainty set P \mathcal{P} P onto a subset of states. Given S ′ ⊂ S \mathcal{S}' \subset\mathcal{S} S ′ ⊂ S and P ∈ P P \in\mathcal{P} P ∈ P we write P S ′ = ( P ( s ) ) s ∈ S ′ P_{\mathcal{S}'}=(P(s))_{s\in\mathcal{S}'} P S ′ = ( P ( s ) ) s ∈ S ′ for the transition probabilities out of states in S ′ \mathcal{S}' S ′ . The projection P S ′ ⊂ ( Δ ( S ) ) S ′ × A \mathcal{P}_{\mathcal{S}'} \subset(\Delta(\mathcal{S}))^{\mathcal{S}'\times\mathcal{A}} P S ′ ⊂ ( Δ ( S ) ) S ′ × A of P \mathcal{P} P onto S ′ \mathcal{S}' S ′ is the set of feasible transition probabilities out of S ′ \mathcal{S}' S ′ , generated by transition probabilities in P \mathcal{P} P :
P S ′ = { P S ′ ∣ ∃ P S ∖ S ′ such that ( P S ′ , P S ∖ S ′ ) ∈ P } . \mathcal{P}_{\mathcal{S}'}=\{P_{\mathcal{S}'}\mid\exists P_{\mathcal{S}\setminus\mathcal{S}'}\text{ such that }(P_{\mathcal{S}'},P_{\mathcal{S}\setminus\mathcal{S}'})\in\mathcal{P}\}. P S ′ = { P S ′ ∣ ∃ P S ∖ S ′ such that ( P S ′ , P S ∖ S ′ ) ∈ P } .
Now in the model of time-varying adversary for k k k -rectangular sets, we are interested in counting, given the observed realizations of the transition probabilities out of states in some S 0 ⊂ S \mathcal{S}_0 \subset\mathcal{S} S 0 ⊂ S (think of all visited states so far in the MDP up to the current period), the number of feasible realizations for the transition probabilities out of states in some other S ⋆ ⊂ S ∖ S 0 \mathcal{S}^\star\subset\mathcal{S}\setminus\mathcal{S}_0 S ⋆ ⊂ S ∖ S 0 (think of the future states visited during the later dynamics). This leads to the definition of conditional projection onto a set of states, given a realization of the transition probabilities outside of this set of states. In particular, let P S 0 ∈ P S 0 P_{\mathcal{S}_0}\in\mathcal{P}_{\mathcal{S}_0} P S 0 ∈ P S 0 be a realization of the parameters in S 0 \mathcal{S}_0 S 0 . Then the conditional projection of P \mathcal{P} P onto S ⋆ \mathcal{S}^\star S ⋆ given P S 0 P_{\mathcal{S}_0} P S 0 is
P S ⋆ ( P S 0 ) = { P S ⋆ ∣ ∃ P S ∖ ( S 0 ∪ S ⋆ ) , ( P S 0 , P S ⋆ , P S ∖ ( S 0 ∪ S ⋆ ) ) ∈ P } . \mathcal{P}_{\mathcal{S}^\star}(P_{\mathcal{S}_0})=\{P_{\mathcal{S}^\star}\mid\exists P_{\mathcal{S}\setminus(\mathcal{S}_0\cup\mathcal{S}^\star)},(P_{\mathcal{S}_0},P_{\mathcal{S}^\star},P_{\mathcal{S}\setminus(\mathcal{S}_0\cup\mathcal{S}^\star)})\in\mathcal{P}\}. P S ⋆ ( P S 0 ) = { P S ⋆ ∣ ∃ P S ∖ ( S 0 ∪ S ⋆ ) , ( P S 0 , P S ⋆ , P S ∖ ( S 0 ∪ S ⋆ ) ) ∈ P } .
This is the set of feasible (future) realizations for the transitions out of S ⋆ \mathcal{S}^\star S ⋆ given the observed (past) realizations P S 0 P_{\mathcal{S}_0} P S 0 out of S 0 \mathcal{S}_0 S 0 . A natural “consistency identity”, which plays a crucial role in the later state-augmentation technique for solving k-rectangular robust MDPs, is that given S 1 , S 2 , S 3 ⊂ S S_1,S_2,S_3 \subset\mathcal{S} S 1 , S 2 , S 3 ⊂ S pairwise disjoint and P 1 ∈ P S 1 , P 2 ∈ P S 2 P_1 \in\mathcal{P}_{S_1}, P_2 \in\mathcal{P}_{S_2} P 1 ∈ P S 1 , P 2 ∈ P S 2 jointly feasible, we have
P S 3 ( ( P 1 , P 2 ) ) = ( P S 2 ∪ S 3 ( P 1 ) ) S 3 ( P 2 ) (74) \mathcal{P}_{S_3}((P_1,P_2)) = \left(\mathcal{P}_{S_2\cup S_3}(P_1)\right)_{S_3}(P_2) \tag*{(74)} P S 3 (( P 1 , P 2 )) = ( P S 2 ∪ S 3 ( P 1 ) ) S 3 ( P 2 ) (74)
The left-hand side P S 3 ( ( P 1 , P 2 ) ) \mathcal{P}_{S_3}((P_1,P_2)) P S 3 (( P 1 , P 2 )) is just the transition probabilities out of S 3 S_3 S 3 compatible with the realization P 1 , P 2 P_1,P_2 P 1 , P 2 out of S 1 S_1 S 1 and S 2 S_2 S 2 , while the right-hand side states that we decompose this set as first the conditional projection of P \mathcal{P} P onto S 2 ∪ S 3 S_2 \cup S_3 S 2 ∪ S 3 given P 1 P_1 P 1 , which we project onto S 3 S_3 S 3 given P 2 P_2 P 2 afterwards. We can already see here some form of dynamic programming principle: as information (realization of the current transition probabilities) is revealed over time, we can decompose the set of transition probabilities out of the remaining non-visited states sequentially.
We are now ready to define the k-rectangularity assumption. Given S ⋆ ⊂ S S^\star\subset\mathcal{S} S ⋆ ⊂ S (representing the set of states that we have not visited yet), we define the following collection of sets:
E S ⋆ = { P S ⋆ ( P S 0 ) ∣ S 0 ⊂ S ∖ S ⋆ , P S 0 ∈ P S 0 } \mathcal{E}_{S^\star} = \{\mathcal{P}_{S^\star}(P_{S_0}) \mid S_0 \subset\mathcal{S} \setminus S^\star, P_{S_0} \in\mathcal{P}_{S_0}\} E S ⋆ = { P S ⋆ ( P S 0 ) ∣ S 0 ⊂ S ∖ S ⋆ , P S 0 ∈ P S 0 }
The set E S ⋆ \mathcal{E}_{S^\star} E S ⋆ contains all the conditional projections of P \mathcal{P} P induced by the realizations of some transition probabilities outside of S ⋆ S^\star S ⋆ .
Definition 4.5. An uncertainty set P \mathcal{P} P is k-rectangular if ∣ E S ⋆ ∣ ≤ k |\mathcal{E}_{S^\star}| \le k ∣ E S ⋆ ∣ ≤ k for every S ⋆ ⊂ S S^\star\subset\mathcal{S} S ⋆ ⊂ S .
This means that for any subset of states S ⋆ ⊂ S S^\star\subset\mathcal{S} S ⋆ ⊂ S for which we do not know the realizations of the transition probabilities, the realizations of the transitions on any combination of states outside of S ⋆ S^\star S ⋆ yield at most k k k distinct sets of transitions out of S ⋆ S^\star S ⋆ . Note that here for each realization of transitions outside of S ⋆ S^\star S ⋆ we may have an infinite set of possible kernels out of S ⋆ S^\star S ⋆ ; it is the number of such possible sets that is finite, not the number of possible kernels. It is worth investigating some examples.
Example 4.8 (s-rectangular uncertainty sets). Assume that P \mathcal{P} P is s-rectangular. For S 0 , S ⋆ S_0,S^\star S 0 , S ⋆ disjoint we have P S ⋆ ( P S 0 ) = × s ∈ S ⋆ P s \mathcal{P}_{S^\star}(P_{S_0}) = \times_{s\in S^\star}\mathcal{P}_s P S ⋆ ( P S 0 ) = × s ∈ S ⋆ P s (independently of the realization of P S 0 ∈ P S 0 P_{S_0} \in\mathcal{P}_{S_0} P S 0 ∈ P S 0 ). Hence ∣ E S ⋆ ∣ = 1 |\mathcal{E}_{S^\star}| = 1 ∣ E S ⋆ ∣ = 1 and s-rectangular uncertainty sets are k-rectangular with k = 1 k=1 k = 1 .
Example 4.9 (LDST uncertainty sets). Consider an LDST uncertainty set where we have at most K ∈ N K \in\mathbb{N} K ∈ N deviations along any trajectory. Then given some realizations of the transition probabilities at S 0 S_0 S 0 , we can compute K 0 K_0 K 0 the number of deviations from the nominal P ^ \hat{P} P ^ for the states in S 0 S_0 S 0 . The projection is
P S ⋆ ( P S 0 ) = { P S ⋆ | ∑ s ∈ S ⋆ 1 { P ( s ) ≠ P ^ ( s ) } ≤ K − K 0 , P ( s ) ∈ P s , ∀ s ∈ S ⋆ } \mathcal{P}_{S^\star}(P_{S_0}) = \left\{P_{S^\star} \,\middle|\, \sum_{s\in S^\star} \mathbf{1}_{\{P(s)\ne\hat{P}(s)\}} \le K-K_0,\ P(s)\in\mathcal{P}_s,\ \forall s\in S^\star\right\} P S ⋆ ( P S 0 ) = { P S ⋆ s ∈ S ⋆ ∑ 1 { P ( s ) = P ^ ( s )} ≤ K − K 0 , P ( s ) ∈ P s , ∀ s ∈ S ⋆ } hence there are at most K + 1 K+1 K + 1 possible realizations of the projections. Here LDST uncertainty sets are k-rectangular with k = K + 1 k=K+1 k = K + 1 .
There is an interesting algebra for the set of k-rectangular sets for finite values of k k k , which is stable by union, intersection, Cartesian product, and Minkowski averaging; we refer to Section 4 in the original paper on k-rectangularity (see § 5.8 ) for more details on this.
State-augmentation technique. We now describe how we can compute an optimal policy for the decision maker for a k-rectangular uncertainty set. Assume that we are at a given period t ∈ N t \in\mathbb{N} t ∈ N and that we have visited the states s 0 , … , s t − 1 s_0,\ldots,s_{t-1} s 0 , … , s t − 1 so far, and that the decision maker knows the corresponding realizations of the transition probabilities P ( s 0 ) , … , P ( s t − 1 ) P(s_0),\ldots,P(s_{t-1}) P ( s 0 ) , … , P ( s t − 1 ) . The set of all possible future states is S t : H : = S t ∪ … ∪ S H \mathcal{S}_{t:H} := \mathcal{S}_t \cup\ldots\cup\mathcal{S}_H S t : H := S t ∪ … ∪ S H , and the uncertainty set of all feasible future transition probabilities compatible with what we observed so far is P S t : H ( P s 0 , … , P s t − 1 ) \mathcal{P}_{\mathcal{S}_{t:H}}(P_{s_0},\ldots,P_{s_{t-1}}) P S t : H ( P s 0 , … , P s t − 1 ) . By definition P S t : H ( P s 0 , … , P s t − 1 ) ∈ E S t : H \mathcal{P}_{\mathcal{S}_{t:H}}(P_{s_0},\ldots,P_{s_{t-1}}) \in\mathcal{E}_{\mathcal{S}_{t:H}} P S t : H ( P s 0 , … , P s t − 1 ) ∈ E S t : H and from the k-rectangularity assumption we know that ∣ E S t : H ∣ ≤ k |\mathcal{E}_{\mathcal{S}_{t:H}}| \leq k ∣ E S t : H ∣ ≤ k . We will write x t x_t x t for a general element of E S t : H \mathcal{E}_{\mathcal{S}_{t:H}} E S t : H . Once the decision maker has chosen a t a_t a t and the adversary has chosen P s t ∈ P S t : H ( P s 0 , … , P s t − 1 ) { s t } P_{s_t} \in\mathcal{P}_{\mathcal{S}_{t:H}}(P_{s_0},\ldots,P_{s_{t-1}})\{s_t\} P s t ∈ P S t : H ( P s 0 , … , P s t − 1 ) { s t } , the set of feasible future transition probabilities compatible with what we observed so far becomes ( P S t : H ( P s 0 , … , P s t − 1 ) ) S t + 1 : H ( P ( s t ) ) (\mathcal{P}_{\mathcal{S}_{t:H}}(P_{s_0},\ldots,P_{s_{t-1}}))_{\mathcal{S}_{t+1:H}}(P(s_t)) ( P S t : H ( P s 0 , … , P s t − 1 ) ) S t + 1 : H ( P ( s t )) , which, given (4.24), is equal to P S t + 1 : H ( P s 0 , … , P s t ) \mathcal{P}_{\mathcal{S}_{t+1:H}}(P_{s_0},\ldots,P_{s_t}) P S t + 1 : H ( P s 0 , … , P s t ) . Therefore, once x t x_t x t is known, the distribution of the next state only depends on the current state-action ( s t , a t ) (s_t,a_t) ( s t , a t ) , the current x t x_t x t , and the choice of transition probabilities at period t t t ; we do not need to know the rest of the history up to period t t t . This means that we can reformulate the robust MDP problem over a compact k-rectangular uncertainty set with state set S \mathcal{S} S as a robust MDP problem with a compact s a sa s a -rectangular uncertainty set over the augmented set of states S ^ = S ^ 0 ∪ … ∪ S ^ H \widehat{\mathcal{S}}=\widehat{\mathcal{S}}_0\cup\ldots\cup\widehat{\mathcal{S}}_H S = S 0 ∪ … ∪ S H with S ^ t = S t × E S t : H \widehat{\mathcal{S}}_t=\mathcal{S}_t\times\mathcal{E}_{\mathcal{S}_{t:H}} S t = S t × E S t : H for each period t t t , with time-varying adversaries, for which we can apply the dynamic programming equations and reductions outlined in Section 4.4 for P H S A P_H^{\mathrm{SA}} P H SA (which can be adapted to deal with finite-horizon problems by adding absorbing states with rewards 0 and adjusting the instantaneous rewards at each period to account for discounting). The k-rectangularity assumption implies that the augmented set of states has finite cardinality, since ∣ S ^ t ∣ = ∣ S t ∣ × ∣ E S t : H ∣ ≤ k ∣ S t ∣ |\widehat{\mathcal{S}}_t|=|\mathcal{S}_t|\times|\mathcal{E}_{\mathcal{S}_{t:H}}|\leq k|\mathcal{S}_t| ∣ S t ∣ = ∣ S t ∣ × ∣ E S t : H ∣ ≤ k ∣ S t ∣ so that S ^ ≤ k ∣ S ∣ \widehat{\mathcal{S}}\leq k|\mathcal{S}| S ≤ k ∣ S ∣ . More precisely, the lifted set of states at period t t t is S t × E S t : H \mathcal{S}_t\times\mathcal{E}_{\mathcal{S}_{t:H}} S t × E S t : H , the set of actions available at this augmented state is A \mathcal{A} A , the transition probability ( s , x ) → ( s ′ , x ′ ) (s,x)\to(s',x') ( s , x ) → ( s ′ , x ′ ) when choosing action a a a at period t t t is P ( s , a , s ′ ) 1 { x ′ = x S t + 1 : H ( P ( s ) ) } P(s,a,s')\mathbf{1}_{\{x'=x_{\mathcal{S}_{t+1:H}}(P(s))\}} P ( s , a , s ′ ) 1 { x ′ = x S t + 1 : H ( P ( s ))} and the lifted instantaneous rewards are r ( ( s , x ) , a , ( s ′ , x ′ ) ) = r ( s , a , s ′ ) r((s,x),a,(s',x'))=r(s,a,s') r (( s , x ) , a , ( s ′ , x ′ )) = r ( s , a , s ′ ) . Since the adversary chooses their decisions after observing the action chosen by the decision maker, the dynamic programming equation for this model is, for t ∈ { 0 , … , H − 1 } t\in\{0,\ldots,H-1\} t ∈ { 0 , … , H − 1 } and, for s ∈ S t s\in\mathcal{S}_t s ∈ S t and x ∈ E S t : H x\in\mathcal{E}_{\mathcal{S}_{t:H}} x ∈ E S t : H ,
v ( s , x ) = max a ∈ A inf P ( s ) ∈ x { s } ∑ s ′ ∈ S t + 1 P ( s , a , s ′ ) ( r ( s , a , s ′ ) + v ( s ′ , x S t + 1 : H ( P ( s ) ) ) ) v(s,x)=\max_{a\in\mathcal{A}}\inf_{P(s)\in x_{\{s\}}}\sum_{s'\in\mathcal{S}_{t+1}}P(s,a,s')\left(r(s,a,s')+v\left(s',x_{\mathcal{S}_{t+1:H}}(P(s))\right)\right) v ( s , x ) = a ∈ A max P ( s ) ∈ x { s } inf s ′ ∈ S t + 1 ∑ P ( s , a , s ′ ) ( r ( s , a , s ′ ) + v ( s ′ , x S t + 1 : H ( P ( s )) ) )
and v ( s , x ) = 0 , ∀ s ∈ S H v(s,x)=0,\ \forall s\in\mathcal{S}_H v ( s , x ) = 0 , ∀ s ∈ S H . For instance in the LDST problem we have seen that x t x_t x t can be mapped to the number of past deviations from the nominal kernels.
Remarks on k-rectangular models. It is important to recall again that the information model of k-rectangularity assumes that the decision maker can observe the realizations of the transition probabilities along the trajectories, which is radically different from the other models of robust MDPs studied in these notes. In particular if the decision maker only observes the current state s t s_t s t then they can’t build the variable x t x_t x t (label of the current conditional projection of the feasible remaining kernels) and cannot play optimally. This is an important limitation of the k-rectangular model of uncertainty, along with the fact that some uncertainty sets are k-rectangular only for very large k k k (e.g. a union of J ∈ N J\in\mathbb{N} J ∈ N s-rectangular sets is k-rectangular with k k k at most 2 J − 1 2^J-1 2 J − 1 which is very large), so that the resulting number of augmented states may be very large. Additionally, it is unclear how one can extend this modeling assumption to the case of infinite-horizon problems, where we cannot assume that each state is visited once (otherwise we have an infinite number of states, so the methods developed in these notes do not apply).
Risk-averse MDPs
The study of risk-averse MDPs is a large and active field of research; as such we are by no means trying to be exhaustive in this section and our only goal is to highlight the well-known connection between risk-averse MDPs and robust MDPs. We will use a tilde symbol to denote random variables in this section.
Preliminaries on risk measures. We introduce here the definition of risk measures and coherent risk measures. Risk measures are real functions of random variables, which we can interpret as assigning a scalar “score” to a random variable.
Definition 4.6. Let ( Ω , F , P ) (\Omega, \mathcal{F}, \mathbb{P}) ( Ω , F , P ) be a probability space. We write L ( Ω , F , P ; R ) \mathcal{L}(\Omega, \mathcal{F}, \mathbb{P}; \mathbb{R}) L ( Ω , F , P ; R ) for the set of essentially bounded random variables, defined as the set of F \mathcal{F} F -measurable functions Ω → R \Omega\to\mathbb{R} Ω → R that are bounded outside a set of measure 0 (for P \mathbb{P} P ). A risk measure is a map from L ( Ω , F , P ; R ) \mathcal{L}(\Omega, \mathcal{F}, \mathbb{P}; \mathbb{R}) L ( Ω , F , P ; R ) to R \mathbb{R} R .
The reader should already be familiar with the most classical examples such as the expectation: X ~ ↦ E P [ X ~ ] \widetilde{X} \mapsto\mathbb{E}_{\mathbb{P}}\left[\widetilde{X}\right] X ↦ E P [ X ] with E P [ X ~ ] = ∫ ω ∈ Ω X ~ ( ω ) d P ( ω ) \mathbb{E}_{\mathbb{P}}\left[\widetilde{X}\right] = \int_{\omega\in\Omega} \widetilde{X}(\omega)\mathrm{d}\mathbb{P}(\omega) E P [ X ] = ∫ ω ∈ Ω X ( ω ) d P ( ω ) , the variance X ~ ↦ E P [ ( X ~ − E P ( X ~ ) ) 2 ] \widetilde{X} \mapsto\mathbb{E}_{\mathbb{P}}\left[\left(\widetilde{X} - \mathbb{E}_{\mathbb{P}}\left(\widetilde{X}\right)\right)^2\right] X ↦ E P [ ( X − E P ( X ) ) 2 ] , the Value-at-Risk, the Conditional-Value-at-Risk, etc. A risk measure ρ \rho ρ is said to be:
monotone if for any random variables X ~ , Y ~ \widetilde{X},\widetilde{Y} X , Y , X ~ ≤ Y ~ \widetilde{X} \le\widetilde{Y} X ≤ Y a.s. implies ρ ( X ~ ) ≤ ρ ( Y ~ ) \rho(\widetilde{X}) \le\rho(\widetilde{Y}) ρ ( X ) ≤ ρ ( Y ) ;
translation-invariant if ρ ( X ~ + c ) = ρ ( X ~ ) + c \rho(\widetilde{X}+c) = \rho(\widetilde{X})+c ρ ( X + c ) = ρ ( X ) + c for any c ∈ R c \in\mathbb{R} c ∈ R and for any random variable X ~ \widetilde{X} X ;
concave if it is monotone, translation invariant, and additionally for any random variables X ~ , Y ~ \widetilde{X},\widetilde{Y} X , Y and scalar λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] we have ρ ( λ X ~ + ( 1 − λ ) Y ~ ) ≥ λ ρ ( X ~ ) + ( 1 − λ ) ρ ( Y ~ ) \rho(\lambda\widetilde{X} + (1-\lambda)\widetilde{Y}) \ge\lambda\rho(\widetilde{X}) + (1-\lambda)\rho(\widetilde{Y}) ρ ( λ X + ( 1 − λ ) Y ) ≥ λ ρ ( X ) + ( 1 − λ ) ρ ( Y ) ;
coherent if it is monotone, translation-invariant, concave, and additionally it is positively homogeneous: ρ ( α X ~ ) = α ρ ( X ~ ) \rho(\alpha\widetilde{X}) = \alpha\rho(\widetilde{X}) ρ ( α X ) = α ρ ( X ) for any α ≥ 0 \alpha\ge0 α ≥ 0 and any random variable X ~ \widetilde{X} X .
The reader can check that the expectation is coherent but not the variance; the essential infimum essinf ( X ~ ) = sup { b ∈ R ∣ P ( X ~ < b ) = 0 } \operatorname{essinf}(\widetilde{X}) = \sup\{b \in\mathbb{R} \mid\mathbb{P}(\widetilde{X} < b) = 0\} essinf ( X ) = sup { b ∈ R ∣ P ( X < b ) = 0 } is coherent. It is worth emphasizing that we are focusing on random variables that model the return (that we want to maximize), whereas a large part of the literature on risk measures focuses on random variables modeling losses. So, for instance, we focus on concave risk measures instead of convex risk measures. Similarly, our definition of Conditional-Value-at-Risk will focus on the smallest (bottom) α \alpha α -fraction of the return instead of the largest (top) α \alpha α -fraction of the return (we would use the later if we were focusing on losses or costs, i.e., on quantities we want to minimize).
The following theorem, known as the dual representation of coherent and concave risk measures , provides the fundamental tools toward connecting risk-averse MDPs and robust MDPs.
Theorem 4.8. Let ( Ω , F , P ) (\Omega, \mathcal{F}, \mathbb{P}) ( Ω , F , P ) be a probability space with Ω \Omega Ω finite: Ω = { 1 , … , m } \Omega= \{1,\ldots,m\} Ω = { 1 , … , m } , F = 2 Ω \mathcal{F} = 2^{\Omega} F = 2 Ω , and let ρ \rho ρ be a risk measure.
If ρ \rho ρ is a concave risk measure, there exists α : Δ ( Ω ) → R ∪ { + ∞ } \alpha: \Delta(\Omega) \to\mathbb{R} \cup\{+\infty\} α : Δ ( Ω ) → R ∪ { + ∞ } such that
ρ ( X ~ ) = min P ′ ∈ Δ ( Ω ) E P ′ [ X ~ ] + α ( P ′ ) (75) \rho(\widetilde{X}) = \min_{\mathbb{P}' \in\Delta(\Omega)} \mathbb{E}_{\mathbb{P}'}\left[\widetilde{X}\right] + \alpha(\mathbb{P}') \tag*{(75)} ρ ( X ) = P ′ ∈ Δ ( Ω ) min E P ′ [ X ] + α ( P ′ ) (75) If ρ \rho ρ is a coherent risk measure, there exists a nonempty compact convex subset Q ρ ⊆ Δ ( Ω ) Q^{\rho} \subseteq\Delta(\Omega) Q ρ ⊆ Δ ( Ω ) of measures over Ω \Omega Ω , called the risk envelope, such that
ρ ( X ~ ) = min P ′ ∈ Q ρ E P ′ [ X ~ ] (76) \rho(\widetilde{X}) = \min_{\mathbb{P}' \in Q^{\rho}} \mathbb{E}_{\mathbb{P}'}\left[\widetilde{X}\right] \tag*{(76)} ρ ( X ) = P ′ ∈ Q ρ min E P ′ [ X ] (76)
For two distributions P , P ′ \mathbb{P}, \mathbb{P}' P , P ′ in Δ ( Ω ) \Delta(\Omega) Δ ( Ω ) we note P ′ ≪ P \mathbb{P}' \ll\mathbb{P} P ′ ≪ P if P ′ \mathbb{P}' P ′ is absolutely continuous with respect to P \mathbb{P} P , i.e., if for any ω ∈ Ω \omega\in\Omega ω ∈ Ω we have P ( ω ) = 0 ⇒ P ′ ( ω ) = 0 \mathbb{P}(\omega) = 0 \Rightarrow\mathbb{P}'(\omega) = 0 P ( ω ) = 0 ⇒ P ′ ( ω ) = 0 . It is worth studying in more detail the dual representations of some classical risk measures:
For a given λ ∈ ( 0 , 1 ] \lambda\in(0,1] λ ∈ ( 0 , 1 ] , CVaR λ ( X ~ ) \operatorname{CVaR}_{\lambda}(\widetilde{X}) CVaR λ ( X ) is defined as the expectation of the worst λ \lambda λ -fraction of the random variable. The risk envelope theorem gives
CVaR λ ( X ~ ) = inf { E P ′ [ X ~ ] ∣ P ′ ∈ Δ ( Ω ) , 0 ≤ P ′ ( ω ) ≤ λ − 1 P ( ω ) , ∀ ω ∈ Ω } . \operatorname{CVaR}_{\lambda}(\widetilde{X}) = \inf\left\{\mathbb{E}_{\mathbb{P}'}\left[\widetilde{X}\right] \mid\mathbb{P}' \in\Delta(\Omega),\ 0 \leq\mathbb{P}'(\omega) \leq\lambda^{-1}\mathbb{P}(\omega),\ \forall\ \omega\in\Omega\right\}. CVaR λ ( X ) = inf { E P ′ [ X ] ∣ P ′ ∈ Δ ( Ω ) , 0 ≤ P ′ ( ω ) ≤ λ − 1 P ( ω ) , ∀ ω ∈ Ω } .
For a given λ > 0 \lambda> 0 λ > 0 , the entropic risk measure (ERM) is defined as ERM λ ( X ~ ) = − λ − 1 log ( E P [ exp ( − λ X ~ ) ] ) \operatorname{ERM}_{\lambda}(\widetilde{X}) = -\lambda^{-1}\log\left(\mathbb{E}_{\mathbb{P}}\left[\exp(-\lambda\widetilde{X})\right]\right) ERM λ ( X ) = − λ − 1 log ( E P [ exp ( − λ X ) ] ) . We already encountered in the exercises from § 3.8 the Donsker–Varadhan variational formula:
ERM λ ( X ~ ) = inf { E P ′ [ X ~ ] + λ − 1 KL ( P ′ , P ) ∣ P ′ ∈ Δ ( Ω ) , P ′ ≪ P } . \operatorname{ERM}_{\lambda}(\widetilde{X}) = \inf\left\{\mathbb{E}_{\mathbb{P}'}\left[\widetilde{X}\right] + \lambda^{-1}\operatorname{KL}(\mathbb{P}',\mathbb{P}) \mid\mathbb{P}' \in\Delta(\Omega),\ \mathbb{P}' \ll\mathbb{P}\right\}. ERM λ ( X ) = inf { E P ′ [ X ] + λ − 1 KL ( P ′ , P ) ∣ P ′ ∈ Δ ( Ω ) , P ′ ≪ P } .
with the Kullback-Leibler divergence defined as
KL ( P ′ , P ) = ∑ ω ∈ Ω P ′ ( ω ) log ( P ′ ( ω ) P ( ω ) ) \operatorname{KL}(\mathbb{P}',\mathbb{P}) = \sum_{\omega\in\Omega} \mathbb{P}'(\omega)\log\left(\frac{\mathbb{P}'(\omega)}{\mathbb{P}(\omega)}\right) KL ( P ′ , P ) = ω ∈ Ω ∑ P ′ ( ω ) log ( P ( ω ) P ′ ( ω ) )
under the convention that 0 log ( 0 ) = 0 0\log(0)=0 0 log ( 0 ) = 0 .
For λ ∈ ( 0 , 1 ] \lambda\in(0,1] λ ∈ ( 0 , 1 ] , we define the entropic value at risk (EVaR) as EVaR λ ( X ~ ) = sup θ > 0 { ERM θ ( X ~ ) + θ − 1 log ( λ ) } \operatorname{EVaR}_{\lambda}(\widetilde{X}) = \sup_{\theta> 0}\{\operatorname{ERM}_{\theta}(\widetilde{X}) + \theta^{-1}\log(\lambda)\} EVaR λ ( X ) = sup θ > 0 { ERM θ ( X ) + θ − 1 log ( λ )} . The dual representation of EVaR is
EVaR λ ( X ~ ) = inf { E P ′ [ X ~ ] ∣ P ′ ∈ Δ ( Ω ) , P ′ ≪ P , KL ( P ′ , P ) ≤ − log ( λ ) } . \operatorname{EVaR}_{\lambda}(\widetilde{X}) = \inf\left\{\mathbb{E}_{\mathbb{P}'}\left[\widetilde{X}\right] \mid\mathbb{P}' \in\Delta(\Omega),\ \mathbb{P}' \ll\mathbb{P},\ \operatorname{KL}(\mathbb{P}',\mathbb{P}) \leq-\log(\lambda)\right\}. EVaR λ ( X ) = inf { E P ′ [ X ] ∣ P ′ ∈ Δ ( Ω ) , P ′ ≪ P , KL ( P ′ , P ) ≤ − log ( λ ) } .
At this point (4.26) already suggests some connections with robust MDPs, where Q ρ Q^{\rho} Q ρ could play the role of an uncertainty set. Indeed the worst-case Bellman operator can be understood as a worst-case expectation for the continuation values (see § 3.2 ). However, Q ρ Q^{\rho} Q ρ is not a set of transition probabilities (we didn’t define any notions of dynamic processes anyway so far in this section), it is a set of measures over the universe Ω \Omega Ω . Later in this section we will see that we can convert Q ρ Q^{\rho} Q ρ into an uncertainty set in some specific cases. We also note that (4.25) suggests an alternative model to robust MDPs, where instead of having hard constraints on the transition probabilities the adversary instead incurs a “cost” α ( P ′ ) \alpha(\mathbb{P}') α ( P ′ ) when choosing P ′ \mathbb{P}' P ′ . This model is sometimes called “soft-robust MDPs” (although there are multiple meanings of this term, which is also used to denote multi-model MDPs).
Finite-horizon robust MDPs: static risk measure. Consider a finite-horizon MDP where the dynamics end at period H H H and the discount factor is γ = 1 \gamma= 1 γ = 1 . The set of states is S \mathcal{S} S . We write Π M D \Pi_{\mathrm{MD}} Π MD for the set of Markovian, deterministic policies. Consider a fixed Markovian deterministic policy π = ( π 0 , … , π H − 1 ) ∈ Π M D \pi= (\pi_0, \ldots, \pi_{H-1}) \in\Pi_{\mathrm{MD}} π = ( π 0 , … , π H − 1 ) ∈ Π MD and a deterministic initial distribution p 0 p_0 p 0 (both of the assumptions can be removed but they simplify the later calculations). Assume that the nominal transition kernel is P 0 ∈ Δ ( S ) S × A P^0 \in\Delta(\mathcal{S})^{\mathcal{S} \times\mathcal{A}} P 0 ∈ Δ ( S ) S × A . Given π \pi π , P 0 P^0 P 0 , if we are in state s t s_t s t at period t t t the next state s ~ t + 1 \widetilde{s}_{t+1} s t + 1 is random with a probability distribution P 0 ( s t , π t ( s t ) ) P^0(s_t, \pi_t(s_t)) P 0 ( s t , π t ( s t )) . Hence the total return R ~ ( π , P 0 ) \widetilde{R}(\pi, P^0) R ( π , P 0 ) is a random variable, with
R ~ ( π , P 0 ) = ∑ t = 0 H − 1 r ( s ~ t , π t ( s ~ t ) , s ~ t + 1 ) (77) \widetilde{R}(\pi, P^0) = \sum_{t=0}^{H-1} r(\widetilde{s}_t, \pi_t(\widetilde{s}_t), \widetilde{s}_{t+1}) \tag*{(77)} R ( π , P 0 ) = t = 0 ∑ H − 1 r ( s t , π t ( s t ) , s t + 1 ) (77)
where ( s ~ t ) t ≤ H (\widetilde{s}_t)_{t \le H} ( s t ) t ≤ H is a random sequence of states. This random variable is defined for the probability space with universe Ω = S H + 1 \Omega= \mathcal{S}^{H+1} Ω = S H + 1 corresponding to all possible realizations of the random sequence of states visited at period t = 0 , t = 1 , … , t = H t = 0, t = 1, \ldots, t = H t = 0 , t = 1 , … , t = H , with the natural filtration
F t = σ ( s ~ 0 , … , s ~ t ) ⊆ 2 Ω \mathcal{F}_t = \sigma(\widetilde{s}_0, \ldots, \widetilde{s}_t) \subseteq2^\Omega F t = σ ( s 0 , … , s t ) ⊆ 2 Ω
for t = 0 , … , H t = 0, \ldots, H t = 0 , … , H , and the nominal probability measure P π , P 0 \mathbb{P}^{\pi,P^0} P π , P 0 on Ω \Omega Ω is
P π , P 0 ( s 0 , … , s H ) = p 0 ( s 0 ) ∏ t = 0 H − 1 P 0 ( s t , π t ( s t ) , s t + 1 ) . \mathbb{P}^{\pi,P^0}(s_0, \ldots, s_H) = p_0(s_0) \prod_{t=0}^{H-1} P^0(s_t, \pi_t(s_t), s_{t+1}). P π , P 0 ( s 0 , … , s H ) = p 0 ( s 0 ) t = 0 ∏ H − 1 P 0 ( s t , π t ( s t ) , s t + 1 ) .
In risk-averse MDPs we are interested in evaluating ρ ( R ~ ( π , P 0 ) ) \rho(\widetilde{R}(\pi, P^0)) ρ ( R ( π , P 0 )) for some risk measure ρ \rho ρ . Note that for ρ = E P π , P 0 \rho= \mathbb{E}_{\mathbb{P}^{\pi,P^0}} ρ = E P π , P 0 this is just a nominal MDP where the objective function E P π , P 0 [ R ~ ( π , P 0 ) ] \mathbb{E}_{\mathbb{P}^{\pi,P^0}}[\widetilde{R}(\pi, P^0)] E P π , P 0 [ R ( π , P 0 )] coincides with our original definition of the expected return as in Section 1.1. We also want to optimize π ↦ ρ ( R ~ ( π , P 0 ) ) \pi\mapsto\rho(\widetilde{R}(\pi, P^0)) π ↦ ρ ( R ( π , P 0 )) but this requires more formal definitions (since we need to define R ~ ( π , P 0 ) \widetilde{R}(\pi, P^0) R ( π , P 0 ) and the probability space for history-dependent randomized policies) that are not relevant for the connection with robust MDPs. Given a coherent risk measure on ( Ω , F , P π , P 0 ) (\Omega, \mathcal{F}, \mathbb{P}^{\pi,P^0}) ( Ω , F , P π , P 0 ) , the risk envelope Q ρ \mathcal{Q}^{\rho} Q ρ is a set of distributions over Ω \Omega Ω (which we can identify with the set of trajectories), and ρ ( R ~ ( π , P 0 ) ) \rho(\widetilde{R}(\pi, P^0)) ρ ( R ( π , P 0 )) is such that
ρ ( R ~ ( π , P 0 ) ) = inf P ′ ∈ Q ρ E P ′ [ ∑ t = 0 H − 1 r ( s ~ t , π t ( s ~ t ) , s ~ t + 1 ) ] . (78) \rho(\widetilde{R}(\pi, P^0)) = \inf_{\mathbb{P}' \in\mathcal{Q}^{\rho}} \mathbb{E}_{\mathbb{P}'}\left[\sum_{t=0}^{H-1} r(\widetilde{s}_t, \pi_t(\widetilde{s}_t), \widetilde{s}_{t+1})\right]. \tag*{(78)} ρ ( R ( π , P 0 )) = P ′ ∈ Q ρ inf E P ′ [ t = 0 ∑ H − 1 r ( s t , π t ( s t ) , s t + 1 ) ] . (78)
Note that Q ρ \mathcal{Q}^{\rho} Q ρ is a set of probability measures over trajectories, so we cannot interpret the right-hand side in the above equation as a policy evaluation problem in a robust MDP. To connect risk-averse MDPs and robust MDPs we want to reformulate the right-hand side of the above equation as a minimization over transition probabilities (possibly history-dependent). This can be done with the chain rule, as explained in the next proposition.
Proposition 4.12. 1. Let P ′ \mathbb{P}' P ′ be a probability law on Ω = S H + 1 \Omega= \mathcal{S}^{H+1} Ω = S H + 1 with initial marginal p 0 p_0 p 0 . For any h t = ( s 0 , … , s t ) ∈ S t + 1 h_t = (s_0, \ldots, s_t) \in\mathcal{S}^{t+1} h t = ( s 0 , … , s t ) ∈ S t + 1 with P ′ ( h t ) > 0 \mathbb{P}'(h_t) > 0 P ′ ( h t ) > 0 (in particular we must have s 0 s_0 s 0 equal to the state chosen deterministically by the initial distribution p 0 p_0 p 0 ), define P ( h t ) ∈ Δ ( S ) P(h_t) \in\Delta(\mathcal{S}) P ( h t ) ∈ Δ ( S ) as P ( h t , s ′ ) = P ′ ( h t , s ′ ) / P ′ ( h t ) P(h_t, s') = \mathbb{P}'(h_t, s')/\mathbb{P}'(h_t) P ( h t , s ′ ) = P ′ ( h t , s ′ ) / P ′ ( h t ) for any s ′ ∈ S s' \in\mathcal{S} s ′ ∈ S . Then for any ( s 0 , … , s H ) ∈ Ω (s_0, \ldots, s_H) \in\Omega ( s 0 , … , s H ) ∈ Ω we have
P ′ ( s 0 , … , s H ) = p 0 ( s 0 ) ∏ t = 0 H − 1 P ( h t , s t + 1 ) . (79) \mathbb{P}'(s_0, \ldots, s_H) = p_0(s_0) \prod_{t=0}^{H-1} P(h_t, s_{t+1}). \tag*{(79)} P ′ ( s 0 , … , s H ) = p 0 ( s 0 ) t = 0 ∏ H − 1 P ( h t , s t + 1 ) . (79)
Conversely, for any collection of P ( h t ) ∈ Δ ( S ) P(h_t) \in\Delta(\mathcal{S}) P ( h t ) ∈ Δ ( S ) defined for each h t ∈ S t + 1 h_t \in\mathcal{S}^{t+1} h t ∈ S t + 1 for each t < H t < H t < H , we can define a probability measure P ′ \mathbb{P}^{\prime} P ′ on Ω = S H + 1 \Omega= \mathcal{S}^{H+1} Ω = S H + 1 by the formula (79) .
Proof. Both statements are just plain rewriting of the definitions of P ( h t ) P(h_t) P ( h t ) as P ( h t , s ′ ) = P ′ ( h t , s ′ ) / P ′ ( h t ) P(h_t,s^{\prime}) = \mathbb{P}^{\prime}(h_t,s^{\prime})/\mathbb{P}^{\prime}(h_t) P ( h t , s ′ ) = P ′ ( h t , s ′ ) / P ′ ( h t ) .
Note that in general we need history-dependent transition probabilities, i.e., P ( h t ) ∈ Δ ( S ) P(h_t) \in\Delta(\mathcal{S}) P ( h t ) ∈ Δ ( S ) for the distribution over the state at period t + 1 t+1 t + 1 after observing history h t = ( s 0 , … , s t ) h_t=(s_0,\ldots,s_t) h t = ( s 0 , … , s t ) , and not necessarily stationary transition probabilities. We show how this works for CVaR and EVaR below:
For CVaR, using the dual representation theorem (Theorem 4.8) to obtain (78) and then Proposition 4.12 to convert a minimization problem over measures in Δ ( S H + 1 ) \Delta(\mathcal{S}^{H+1}) Δ ( S H + 1 ) into transition probabilities, we get that CVaR λ ( R ~ ( π , P 0 ) ) \operatorname{CVaR}_{\lambda}(\widetilde{R}(\pi,P^0)) CVaR λ ( R ( π , P 0 )) is equal to
inf E π , P [ ∑ t = 0 H − 1 r ( s ~ t , π t ( s ~ t ) , s ~ t + 1 ) ] \inf\mathbb{E}^{\pi,P}\left[\sum_{t=0}^{H-1}r(\widetilde{s}_t,\pi_t(\widetilde{s}_t),\widetilde{s}_{t+1})\right] inf E π , P [ t = 0 ∑ H − 1 r ( s t , π t ( s t ) , s t + 1 ) ]
∏ t = 0 H − 1 P ( s 0 , … , s t , s t + 1 ) P 0 ( s t , π t ( s t ) , s t + 1 ) ≤ λ − 1 , ∀ ( s t ) t ≤ H ∈ S H + 1 , \prod_{t=0}^{H-1}\frac{P(s_0,\ldots,s_t,s_{t+1})}{P^0(s_t,\pi_t(s_t),s_{t+1})}\leq\lambda^{-1},\ \forall\ (s_t)_{t\leq H}\in\mathcal{S}^{H+1}, t = 0 ∏ H − 1 P 0 ( s t , π t ( s t ) , s t + 1 ) P ( s 0 , … , s t , s t + 1 ) ≤ λ − 1 , ∀ ( s t ) t ≤ H ∈ S H + 1 ,
P ( s 0 , … , s t ) ∈ Δ ( S ) , P ( s 0 , … , s t ) ≪ P 0 ( s t , π t ( s t ) ) , ∀ ( s t ) t ≤ H ∈ S H + 1 , t ≤ H P(s_0,\ldots,s_t)\in\Delta(\mathcal{S}),\ P(s_0,\ldots,s_t)\ll P^0(s_t,\pi_t(s_t)),\ \forall\ (s_t)_{t\leq H}\in\mathcal{S}^{H+1},\ t\leq H P ( s 0 , … , s t ) ∈ Δ ( S ) , P ( s 0 , … , s t ) ≪ P 0 ( s t , π t ( s t )) , ∀ ( s t ) t ≤ H ∈ S H + 1 , t ≤ H
where E π , P \mathbb{E}^{\pi,P} E π , P is the expectation with respect to the probability measure over S H + 1 \mathcal{S}^{H+1} S H + 1 defined by π ∈ Π M D \pi\in\Pi_{\mathrm{MD}} π ∈ Π MD and P P P satisfying the constraints in the above optimization program. This is a policy evaluation problem (in the finite-horizon setting) with an uncertainty set characterized by the set of constraints in the optimization problem above. The resulting uncertainty set has history-dependent transition probabilities and is non-rectangular given we have constraints across the transition probabilities out of different states (linked by the product ∏ t = 0 H − 1 \prod_{t=0}^{H-1} ∏ t = 0 H − 1 ) and because the constraints in this uncertainty set are trajectory-wise (instead of state-wise). Note that we already encountered robust MDPs with time-varying transition probabilities in § 4.4 , but there we assumed that the transition probabilities chosen at period t t t out of s t s_t s t were always constrained to belong to the same set P s t \mathcal{P}_{s_t} P s t , we have constraints over the transition probabilities at different periods and when visiting the same state over time the sets of constraints for the transition probabilities out of this state change, hence the results from § 4.4 do not apply directly.
For EVaR, using the dual representation theorem (Theorem 4.8) we get that EVaR λ ( R ~ ( π , P 0 ) ) \operatorname{EVaR}_{\lambda}(\widetilde{R}(\pi,P^0)) EVaR λ ( R ( π , P 0 )) is equal to
inf E π , P [ ∑ t = 0 H − 1 r ( s ~ t , π t ( s ~ t ) , s ~ t + 1 ) ] \inf\mathbb{E}^{\pi,P}\left[\sum_{t=0}^{H-1}r(\widetilde{s}_t,\pi_t(\widetilde{s}_t),\widetilde{s}_{t+1})\right] inf E π , P [ t = 0 ∑ H − 1 r ( s t , π t ( s t ) , s t + 1 ) ]
∑ t = 0 H − 1 E π , P [ KL ( P ( s ~ 0 , … , s ~ t ) , P 0 ( s ~ t , π t ( s ~ t ) ) ) ] ≤ − log ( λ ) , \sum_{t=0}^{H-1}\mathbb{E}^{\pi,P}\left[\operatorname{KL}\left(P(\widetilde{s}_0,\ldots,\widetilde{s}_t),P^0(\widetilde{s}_t,\pi_t(\widetilde{s}_t))\right)\right]\leq-\log(\lambda), t = 0 ∑ H − 1 E π , P [ KL ( P ( s 0 , … , s t ) , P 0 ( s t , π t ( s t )) ) ] ≤ − log ( λ ) ,
P ( s 0 , … , s t ) ∈ Δ ( S ) , P ( s 0 , … , s t ) ≪ P 0 ( s t , π t ( s t ) ) , ∀ ( s t ) t ≤ H ∈ S H + 1 , t ≤ H P(s_0,\ldots,s_t)\in\Delta(\mathcal{S}),\ P(s_0,\ldots,s_t)\ll P^0(s_t,\pi_t(s_t)),\ \forall\ (s_t)_{t\leq H}\in\mathcal{S}^{H+1},\ t\leq H P ( s 0 , … , s t ) ∈ Δ ( S ) , P ( s 0 , … , s t ) ≪ P 0 ( s t , π t ( s t )) , ∀ ( s t ) t ≤ H ∈ S H + 1 , t ≤ H
This again yields an uncertainty set with history-dependent, non-rectangular transition probabilities.
So we have seen that in general risk-averse MDPs with a coherent risk measure will induce a particular type of robust MDPs where the transition probabilities are allowed to be history-dependent and constrained in a “nasty” way: there may be constraints across different periods and constraints across the transitions out of different states, which typically yields difficult optimization problems, as we have seen in the previous chapters. For a concave risk measure ρ \rho ρ we obtain an optimization problem sometimes called soft-robust MDPs , where there is a “soft” penalization term α ( P ) ∈ R ∪ { + ∞ } \alpha(\mathbb{P}) \in\mathbb{R} \cup\{+\infty\} α ( P ) ∈ R ∪ { + ∞ } in the objective:
ρ ( R ~ ( π , P 0 ) ) = inf P ′ ∈ Δ ( Ω ) E P ′ [ ∑ t = 0 H − 1 r ( s ~ t , π t ( s ~ t ) , s ~ t + 1 ) ] + α ( P ′ ) . \rho(\widetilde{R}(\pi,\mathbb{P}^{0}))=\inf_{\mathbb{P}^{\prime}\in\Delta(\Omega)}\mathbb{E}_{\mathbb{P}^{\prime}}\left[\sum_{t=0}^{H-1}r(\widetilde{s}_{t},\pi_{t}(\widetilde{s}_{t}),\widetilde{s}_{t+1})\right]+\alpha(\mathbb{P}^{\prime}). ρ ( R ( π , P 0 )) = P ′ ∈ Δ ( Ω ) inf E P ′ [ t = 0 ∑ H − 1 r ( s t , π t ( s t ) , s t + 1 ) ] + α ( P ′ ) .
In principle the term α ( P ′ ) \alpha(\mathbb{P}^{\prime}) α ( P ′ ) need not decompose period-wise, so that evaluating ρ ( R ~ ( π , P 0 ) ) \rho(\widetilde{R}(\pi,\mathbb{P}^{0})) ρ ( R ( π , P 0 )) may be difficult, let alone optimizing it over π \pi π ; ERM is one particular example where the penalty term α \alpha α decomposes across periods (it is a KL divergence). Note that hard constraints can be encoded this way by taking α ( P ′ ) = + ∞ \alpha(\mathbb{P}^{\prime})=+\infty α ( P ′ ) = + ∞ for some forbidden measures P ′ \mathbb{P}^{\prime} P ′ outside of some prescribed set Q ρ \mathcal{Q}^{\rho} Q ρ .
Finite-horizon robust MDPs: conditional and Markov risk measures. We have seen in the examples above that coherent risk measures can be related to robust MDPs with non-rectangular uncertainty sets, which are in general difficult to optimize over. A natural idea to recover robust MDPs with rectangular uncertainty sets from risk-averse MDPs is the following: instead of applying the risk measure ρ \rho ρ to the return of the entire trajectory, we apply ρ \rho ρ inductively, e.g., informally we consider the following objective (for s 0 ∈ S s_{0} \in\mathcal{S} s 0 ∈ S the state chosen deterministically by p 0 p_{0} p 0 ):
ρ ( r ( s 0 , a ~ 0 , s ~ 1 ) + ρ ( r ( s ~ 1 , a ~ 1 , s ~ 2 ) + ρ ( … + ρ ( r ( s ~ H − 1 , a ~ H − 1 , s ~ H ) ) ) ) ) (80) \rho\left(r(s_{0},\widetilde{a}_{0},\widetilde{s}_{1})+\rho\left(r(\widetilde{s}_{1},\widetilde{a}_{1},\widetilde{s}_{2})+\rho(\ldots+\rho(r(\widetilde{s}_{H-1},\widetilde{a}_{H-1},\widetilde{s}_{H})))\right)\right) \tag*{(80)} ρ ( r ( s 0 , a 0 , s 1 ) + ρ ( r ( s 1 , a 1 , s 2 ) + ρ ( … + ρ ( r ( s H − 1 , a H − 1 , s H ))) ) ) (80)
This construction is known as nested conditional risk measures (this choice of terminology comes from the fact that the risk measure ρ \rho ρ is applied at every period as information about the current state is revealed, rather than to the entire trajectory). More formally, in the inner evaluation above each risk measure is actually a conditional risk measure ρ ( ⋅ ∣ F t ) : L ( Ω , F t + 1 , P π , P 0 ; R ) → L ( Ω , F t , P π , P 0 ; R ) \rho(\cdot\mid\mathcal{F}_{t}):\mathcal{L}(\Omega,\mathcal{F}_{t+1},\mathbb{P}^{\pi,\mathbb{P}^{0}};\mathbb{R})\to\mathcal{L}(\Omega,\mathcal{F}_{t},\mathbb{P}^{\pi,\mathbb{P}^{0}};\mathbb{R}) ρ ( ⋅ ∣ F t ) : L ( Ω , F t + 1 , P π , P 0 ; R ) → L ( Ω , F t , P π , P 0 ; R ) in the sense of Ruszczyński–Shapiro (see references at the end of this chapter): for an F t + 1 \mathcal{F}_{t+1} F t + 1 -measurable random variable X ~ \widetilde{X} X , we get that ρ ( X ~ ∣ F t ) \rho(\widetilde{X}\mid\mathcal{F}_{t}) ρ ( X ∣ F t ) is an F t \mathcal{F}_{t} F t -measurable random variable, i.e. its universe is the set of finite histories up to period t t t . Hence in (4.30) the outer ρ \rho ρ is actually ρ ( ⋅ ∣ F 0 ) \rho(\cdot\mid\mathcal{F}_{0}) ρ ( ⋅ ∣ F 0 ) , the first inner ρ \rho ρ is actually ρ ( ⋅ ∣ F 1 ) \rho(\cdot\mid\mathcal{F}_{1}) ρ ( ⋅ ∣ F 1 ) , etc. To construct a family of coherent conditional risk measures in our context, following the dual representation theorem (Theorem 4.8) we can simply choose, for each period t t t and history h t = ( s 0 , a 0 , … , s t ) h_{t}=(s_{0},a_{0},\ldots,s_{t}) h t = ( s 0 , a 0 , … , s t ) a history-dependent risk envelope
Q ρ ( h t ) ⊆ { p ∈ Δ ( S ) ∣ p ≪ P 0 ( s t , π t ( s t ) ) } \mathcal{Q}^{\rho}(h_{t})\subseteq\{p\in\Delta(\mathcal{S})\mid p\ll\mathbb{P}^{0}(s_{t},\pi_{t}(s_{t}))\} Q ρ ( h t ) ⊆ { p ∈ Δ ( S ) ∣ p ≪ P 0 ( s t , π t ( s t ))}
and then define, for a given F t + 1 \mathcal{F}_{t+1} F t + 1 -measurable random variable X ~ \widetilde{X} X ,
( ρ ( X ~ ∣ F t ) ) ( h t ) : = inf { E s ~ ∼ p ( h t ) [ X ~ ( h t , s ~ ) ] ∣ p ( h t ) ∈ Q ρ ( h t ) } . (81) \left(\rho(\widetilde{X}\mid\mathcal{F}_{t})\right)(h_{t}):=\inf\left\{\mathbb{E}_{\widetilde{s}\sim p(h_{t})}\left[\widetilde{X}(h_{t},\widetilde{s})\right]\mid p(h_{t})\in\mathcal{Q}^{\rho}(h_{t})\right\}. \tag*{(81)} ( ρ ( X ∣ F t ) ) ( h t ) := inf { E s ∼ p ( h t ) [ X ( h t , s ) ] ∣ p ( h t ) ∈ Q ρ ( h t ) } . (81)
With this construction, the objective (80) is well-defined, and the monotonicity of the conditional risk measures ensures that time-consistency , i.e., dynamic programming, holds. We can additionally assume that the risk measures satisfy the following Markov property, that the risk envelopes Q ρ ( h t ) \mathcal{Q}^{\rho}(h_t) Q ρ ( h t ) (with h t = ( s 0 , a 0 , … , s t ) h_t=(s_0,a_0,\ldots,s_t) h t = ( s 0 , a 0 , … , s t ) ) only depend on the last state s t s_t s t and the deterministic choice of action π t ( s t ) \pi_t(s_t) π t ( s t ) , i.e., that there exists some P s t π t ( s t ) ρ ⊆ Δ ( S ) \mathcal{P}^{\rho}_{s_t\pi_t(s_t)}\subseteq\Delta(\mathcal{S}) P s t π t ( s t ) ρ ⊆ Δ ( S ) (say compact) such that Q ρ ( h t ) = P s t π t ( s t ) ρ \mathcal{Q}^{\rho}(h_t)=\mathcal{P}^{\rho}_{s_t\pi_t(s_t)} Q ρ ( h t ) = P s t π t ( s t ) ρ . For instance, we get the following constraints on the corresponding uncertainty P ρ \mathcal{P}^{\rho} P ρ for conditional CVaR and conditional EVaR:
P s a C V a R = { p ∈ Δ ( S ) ∣ 0 ≤ p ( s ′ ) ≤ λ − 1 P 0 ( s , a , s ′ ) , ∀ s ′ ∈ S } \mathcal{P}^{\mathrm{CVaR}}_{sa}=\left\{p\in\Delta(\mathcal{S})\mid0\leq p(s')\leq\lambda^{-1}P^0(s,a,s'),\ \forall s'\in\mathcal{S}\right\} P s a CVaR = { p ∈ Δ ( S ) ∣ 0 ≤ p ( s ′ ) ≤ λ − 1 P 0 ( s , a , s ′ ) , ∀ s ′ ∈ S }
P s a E V a R = { p ∈ Δ ( S ) ∣ KL ( p , P 0 ( s , a ) ) ≤ − log ( λ ) } \mathcal{P}^{\mathrm{EVaR}}_{sa}=\left\{p\in\Delta(\mathcal{S})\mid\operatorname{KL}(p,P^0(s,a))\leq-\log(\lambda)\right\} P s a EVaR = { p ∈ Δ ( S ) ∣ KL ( p , P 0 ( s , a )) ≤ − log ( λ ) }
Applying this in (81) , the reader should now recognize something very close to an sa-rectangular robust MDP formulation. Indeed, we now get that
( ρ ( X ~ ∣ F t ) ( h t ) : = min { E s ~ ∼ p [ X ~ ( h t , s ~ ) ] ∣ p ∈ P s t π t ( s t ) ρ } ) . \left(\rho(\widetilde{X}\mid\mathcal{F}_t)(h_t):=\min\left\{\mathbb{E}_{\widetilde{s}\sim p}\left[\widetilde{X}(h_t,\widetilde{s})\right]\mid p\in\mathcal{P}^{\rho}_{s_t\pi_t(s_t)}\right\}\right). ( ρ ( X ∣ F t ) ( h t ) := min { E s ∼ p [ X ( h t , s ) ] ∣ p ∈ P s t π t ( s t ) ρ } ) .
In particular for v ∈ R S v\in\mathbb{R}^{S} v ∈ R S we have
ρ ( r ( s ~ t , π t ( s ~ t ) , s ~ t + 1 ) + v ( s ~ t + 1 ) ∣ F t ) = T t π , ρ v ( s ~ t ) \rho\left(r(\widetilde{s}_t,\pi_t(\widetilde{s}_t),\widetilde{s}_{t+1})+v(\widetilde{s}_{t+1})\mid\mathcal{F}_t\right)=T_t^{\pi,\rho}v(\widetilde{s}_t) ρ ( r ( s t , π t ( s t ) , s t + 1 ) + v ( s t + 1 ) ∣ F t ) = T t π , ρ v ( s t )
for the operator T t π , ρ : R S → R S T_t^{\pi,\rho}:\mathbb{R}^{S}\to\mathbb{R}^{S} T t π , ρ : R S → R S as
T t π , ρ v ( s ) : = min p ∈ P s π t ( s ) ρ p T ( r ( s , π t ( s ) ) + v ) , ∀ v ∈ R S , s ∈ S . T_t^{\pi,\rho}v(s):=\min_{p\in\mathcal{P}^{\rho}_{s\pi_t(s)}}p^{\mathsf{T}}\left(r(s,\pi_t(s))+v\right),\ \forall v\in\mathbb{R}^{S},\ s\in\mathcal{S}. T t π , ρ v ( s ) := p ∈ P s π t ( s ) ρ min p T ( r ( s , π t ( s )) + v ) , ∀ v ∈ R S , s ∈ S .
With these definitions, we finally obtain
( T 0 π , ρ ∘ T 1 π , ρ ∘ ⋯ ∘ T H − 1 π , ρ ) 0 S ( s 0 ) = ρ ( r ( s 0 , a ~ 0 , s ~ 1 ) + ρ ( r ( s ~ 1 , a ~ 1 , s ~ 2 ) + ρ ( ⋯ + ρ ( r ( s ~ H − 1 , a ~ H − 1 , s ~ H ) ) ) ) ) \left(T_0^{\pi,\rho}\circ T_1^{\pi,\rho}\circ\cdots\circ T_{H-1}^{\pi,\rho}\right)0_{\mathcal{S}}(s_0)=\rho\left(r(s_0,\widetilde{a}_0,\widetilde{s}_1)+\rho\left(r(\widetilde{s}_1,\widetilde{a}_1,\widetilde{s}_2)+\rho\left(\cdots+\rho\left(r(\widetilde{s}_{H-1},\widetilde{a}_{H-1},\widetilde{s}_H)\right)\right)\right)\right) ( T 0 π , ρ ∘ T 1 π , ρ ∘ ⋯ ∘ T H − 1 π , ρ ) 0 S ( s 0 ) = ρ ( r ( s 0 , a 0 , s 1 ) + ρ ( r ( s 1 , a 1 , s 2 ) + ρ ( ⋯ + ρ ( r ( s H − 1 , a H − 1 , s H ) ) ) ) )
with 0 S = ( 0 , … , 0 ) ∈ R S 0_{\mathcal{S}}=(0,\ldots,0)\in\mathbb{R}^{S} 0 S = ( 0 , … , 0 ) ∈ R S ; this can be compared with the same kind of expressions we obtained for nominal objectives right before Definition 1.2. In particular, the left-hand side above can be interpreted as the (total, undiscounted) return of a policy π ∈ Π M D \pi\in\Pi_{\mathrm{MD}} π ∈ Π MD obtained after H H H periods of decisions in a finite-horizon robust MDP where the transitions are chosen adversarially at each period in an sa-rectangular uncertainty set; this is akin to Model 1 from § 4.4 , and we know that, in the case of finite horizon, an optimal policy for this model can be chosen Markovian deterministic (see Theorem 4.5).
One of the main issues of the above approach is that the nested objective (80) need not be law-invariant, in that the returned scalar value may depend on the timing at which the rewards are obtained (and not solely on the distribution of the random return). In particular (80) may differ from ρ ( ∑ t = 0 H − 1 r ( s ~ t , a ~ t , s ~ t + 1 ) ) \rho\left(\sum_{t=0}^{H-1}r(\widetilde{s}_t,\widetilde{a}_t,\widetilde{s}_{t+1})\right) ρ ( ∑ t = 0 H − 1 r ( s t , a t , s t + 1 ) ) . It is known that, under fairly general assumptions (see references at the end of this chapter), the only conditional Markov law-invariant risk measure is the entropic risk measure (extended to λ = + ∞ \lambda=+\infty λ = + ∞ and λ = 0 \lambda=0 λ = 0 by continuity), and if additionally we assume that the corresponding law-invariant risk measure must be coherent, only the expectation and the essential infimum satisfy this time-consistency property. So while conditional risk measures “restore” the dynamic programming property for risk-averse MDPs, in some sense the fact that almost all of the conditional risk measures are not law-invariant induces some strange estimation properties for the return.
Infinite-horizon problems. We now explain how to extend these formulations for the case of infinite-horizon problems. We assume γ < 1 \gamma< 1 γ < 1 and we still have π ∈ Π M D \pi\in\Pi_{\mathrm{MD}} π ∈ Π MD . In this case the random return is R ~ ( π , P 0 ) = ∑ t = 0 ∞ γ t r ( s ~ t , π t ( s ~ t ) , s ~ t + 1 ) \widetilde{R}(\pi, P^{0}) = \sum_{t=0}^{\infty} \gamma^{t} r(\widetilde{s}_{t}, \pi_{t}(\widetilde{s}_{t}), \widetilde{s}_{t+1}) R ( π , P 0 ) = ∑ t = 0 ∞ γ t r ( s t , π t ( s t ) , s t + 1 ) and the universe is the set of all infinite trajectories S N \mathcal{S}^{\mathbb{N}} S N , which is of course not finite anymore. Static coherent risk measures still admit some dual representations (if we additionally assume the Fatou property) but the resulting uncertainty set reformulations are harder to describe:
For CVaR we get the following uncertainty set of history-dependent transition probabilities:
P H C V a R = { P ∣ P ( s 0 , … , s t ) ∈ Δ ( S ) , P ( s 0 , … , s t ) ≪ P 0 ( s t , π t ( s t ) ) , ∀ ( s 0 , … , s t ) ∈ S t + 1 , ∀ t ∈ N , ∏ t = 0 H − 1 P ( s 0 , … , s t , s t + 1 ) P 0 ( s t , π t ( s t ) , s t + 1 ) ≤ λ − 1 , ∀ H ≥ 1 , ∀ h H ∈ S H + 1 } . \mathcal{P}_{\mathrm{H}}^{\mathrm{CVaR}} = \left\{ P \mid P(s_{0}, \ldots, s_{t}) \in\Delta(\mathcal{S}),\ P(s_{0}, \ldots, s_{t}) \ll P^{0}(s_{t}, \pi_{t}(s_{t})),\ \forall(s_{0}, \ldots, s_{t}) \in\mathcal{S}^{t+1},\ \forall t \in\mathbb{N},
\prod_{t=0}^{H-1} \frac{P(s_{0}, \ldots, s_{t}, s_{t+1})}{P^{0}(s_{t}, \pi_{t}(s_{t}), s_{t+1})} \leq\lambda^{-1},\ \forall H \geq1,\ \forall h_{H} \in\mathcal{S}^{H+1} \right\}. P H CVaR = { P ∣ P ( s 0 , … , s t ) ∈ Δ ( S ) , P ( s 0 , … , s t ) ≪ P 0 ( s t , π t ( s t )) , ∀ ( s 0 , … , s t ) ∈ S t + 1 , ∀ t ∈ N , t = 0 ∏ H − 1 P 0 ( s t , π t ( s t ) , s t + 1 ) P ( s 0 , … , s t , s t + 1 ) ≤ λ − 1 , ∀ H ≥ 1 , ∀ h H ∈ S H + 1 } .
For EVaR we obtain
P H E V a R = { P ∣ P ( s 0 , … , s t ) ∈ Δ ( S ) , P ( s 0 , … , s t ) ≪ P 0 ( s t , π t ( s t ) ) , ∀ ( s 0 , … , s t ) ∈ S t + 1 , ∀ t ∈ N , ∑ t = 0 + ∞ E π , P [ KL ( P ( s ~ 0 , … , s ~ t ) , P 0 ( s ~ t , π t ( s ~ t ) ) ) ] ≤ − log ( λ ) } . \mathcal{P}_{\mathrm{H}}^{\mathrm{EVaR}} = \left\{ P \mid P(s_{0}, \ldots, s_{t}) \in\Delta(\mathcal{S}),\ P(s_{0}, \ldots, s_{t}) \ll P^{0}(s_{t}, \pi_{t}(s_{t})),\ \forall(s_{0}, \ldots, s_{t}) \in\mathcal{S}^{t+1},\ \forall t \in\mathbb{N},
\sum_{t=0}^{+\infty} \mathbb{E}^{\pi,P}\left[\operatorname{KL}\left(P(\widetilde{s}_{0}, \ldots, \widetilde{s}_{t}), P^{0}(\widetilde{s}_{t}, \pi_{t}(\widetilde{s}_{t}))\right)\right] \leq-\log(\lambda) \right\}. P H EVaR = { P ∣ P ( s 0 , … , s t ) ∈ Δ ( S ) , P ( s 0 , … , s t ) ≪ P 0 ( s t , π t ( s t )) , ∀ ( s 0 , … , s t ) ∈ S t + 1 , ∀ t ∈ N , t = 0 ∑ + ∞ E π , P [ KL ( P ( s 0 , … , s t ) , P 0 ( s t , π t ( s t )) ) ] ≤ − log ( λ ) } .
Hence for static CVaR and static EVaR we obtain reformulations of risk-averse MDPs as robust MDPs with history-dependent transition probabilities that have constraints across both periods and states; arguably this should constitute difficult optimization problems. In contrast in the case of conditional Markov risk measures we obtain a very simple reformulation: for π ∈ Π S D \pi\in\Pi_{\mathrm{SD}} π ∈ Π SD the corresponding Bellman operator becomes T π , ρ : R S → R S T^{\pi,\rho} : \mathbb{R}^{\mathcal{S}} \to\mathbb{R}^{\mathcal{S}} T π , ρ : R S → R S defined as
T π , ρ v ( s ) = ρ ( r ( s , π ( s ) , s ~ ′ ) + γ v ( s ~ ′ ) ) T^{\pi,\rho}v(s) = \rho(r(s,\pi(s),\widetilde{s}^{\prime}) + \gamma v(\widetilde{s}^{\prime})) T π , ρ v ( s ) = ρ ( r ( s , π ( s ) , s ′ ) + γ v ( s ′ ))
where s ~ ′ ∼ P 0 ( s , π ( s ) ) \widetilde{s}^{\prime} \sim P^{0}(s,\pi(s)) s ′ ∼ P 0 ( s , π ( s )) ; note that in the equations above we slightly abuse notation since we note that the random variable s ~ ′ \widetilde{s}^{\prime} s ′ has different laws P 0 ( s , a ) ∈ Δ ( S ) P^{0}(s,a) \in\Delta(\mathcal{S}) P 0 ( s , a ) ∈ Δ ( S ) for different state-action pairs s , a s,a s , a , hence in principle we should define one risk measure for each state-action pair; this is not necessary if we use law-invariant risk measures. The operator T π , ρ T^{\pi,\rho} T π , ρ has several important properties:
T π , ρ T^{\pi,\rho} T π , ρ is a monotone contraction on R S \mathbb{R}^{\mathcal{S}} R S as long as ρ \rho ρ is monotone and translation-invariant (this is related to Crandall-Tartar theorem as in § 1.6 ), hence it admits a unique fixed point v π , ρ ∈ R S v^{\pi,\rho} \in\mathbb{R}^{\mathcal{S}} v π , ρ ∈ R S .
T π , ρ v ( s ) = min P ( s , π ( s ) ) ∈ P π ( s ) P ( s , π ( s ) ) T ( r ( s , π ( s ) ) + γ v ) T^{\pi,\rho}v(s) = \min_{P(s,\pi(s)) \in\mathcal{P}_{\pi(s)}} P(s,\pi(s))^{\mathsf{T}}(r(s,\pi(s)) + \gamma v) T π , ρ v ( s ) = P ( s , π ( s )) ∈ P π ( s ) min P ( s , π ( s ) ) T ( r ( s , π ( s )) + γ v )
for some nonempty convex compact set P π ( s ) \mathcal{P}_{\pi(s)} P π ( s ) , hence we again obtain that risk-averse MDPs with Markovian coherent risk measures are equivalent to sa-rectangular robust MDPs.
As a short conclusion we note that all the analysis above can be made for history-dependent randomized policies π ∈ Π H \pi\in\Pi_{\mathrm{H}} π ∈ Π H and randomized initial distribution p 0 ∈ Δ ( S ) p_0 \in\Delta(S) p 0 ∈ Δ ( S ) (instead of π ∈ Π M D \pi\in\Pi_{\mathrm{MD}} π ∈ Π MD and deterministic initial distributions). The main changes are that the trajectory law of the next state also depends on the random variables a ~ t \widetilde{a}_t a t for the actions chosen following π \pi π and depends on p 0 p_0 p 0 , and therefore the corresponding uncertainty sets become policy-dependent, initial-distribution dependent, and the action probabilities also change, which departs even more from our model of robust MDPs with unknown stationary transition probabilities. Also we only focused on the “policy evaluation problem” for risk-averse MDPs (i.e., estimating ρ ( R ~ ( π , P 0 ) ) \rho(\widetilde{R}(\pi,P^0)) ρ ( R ( π , P 0 )) ) and not on the policy optimization problem (i.e., optimizing π ↦ ρ ( R ~ ( π , P 0 ) ) \pi\mapsto\rho(\widetilde{R}(\pi,P^0)) π ↦ ρ ( R ( π , P 0 )) ); we point the reader to § 5.8 for references on these topics.
Assumptions on the instantaneous rewards*
I suggest reading Section 1.3 and § 4.2 before diving into this topic. We have encountered a few cases where things are considerably harder for general rewards than for rewards that are independent of the next state:
In Section 3.5, the minimality of s-rectangularity and sa-rectangularity for dynamic programming and the existence of uniformly optimal policies crucially relies on general reward models. If we instead restrict ourselves to next state-independent rewards then the so-called r-rectangular model of uncertainty and its generalization also satisfy these desirable properties (see § 5.3.1 ). In turn, r-rectangular models are NP-hard to solve for general models of rewards (Proposition 4.8).
For uncertain rewards and uncertain transitions, general rewards make the policy evaluation problem NP-hard even for convex compact s-rectangular uncertainty sets (Proposition 4.3) while under the same assumptions on the uncertainty set, next-state independent rewards make both policy evaluation and computing an optimal policy tractable (Theorem 4.1).
This may seem a bit puzzling, since we saw in Section 1.3 that for nominal MDPs we could have assumed that the rewards were next-state independent, without loss of generality. This fact was proved with two different techniques: state-augmentation (preserving the expression of the discounted return) or replacing rewards by expected rewards (and noticing the Bellman operator T P T^P T P didn’t change). We explain here in more detail when these techniques may work or fail:
Assume first that the rewards are fixed.
For P \mathcal{P} P s-rectangular convex the state-augmentation technique yields an s-rectangular convex uncertainty set over the augmented state set, where rewards are dependent only on the current state, leading to a tractable problem (it falls under the umbrella of models studied in Chapter 3). The “expected reward” technique yields a robust MDP with s-rectangular uncertainty set U f \mathcal{U}^f U f over the uncertain rewards and uncertain transitions (expected rewards are uncertain because they involve P ∈ P P \in\mathcal{P} P ∈ P ), and U f \mathcal{U}^f U f is convex; this falls under the umbrella of tractable problems described in § 4.2 .
Therefore when the rewards are fixed and P \mathcal{P} P is s-rectangular convex we can safely assume that the rewards only depend on the current state.
For the “expected reward” technique we’ve seen that if P \mathcal{P} P and R \mathcal{R} R are s-rectangular convex the corresponding U f \mathcal{U}^{f} U f is s-rectangular but it may not be convex, so this method may fail in this case (it may not lead to tractable reformulations). Note that from Section 4.2 this method works when P \mathcal{P} P is sa-rectangular compact and R \mathcal{R} R is sa-rectangular compact (since we don’t need convexity of the resulting U f \mathcal{U}^{f} U f in this case).
In contrast the state-augmentation may fail even for sa-rectangular sets P \mathcal{P} P and R \mathcal{R} R . This is because the rewards out of the different fictitious states ( s , a , s ′ ) (s,a,s') ( s , a , s ′ ) can be “correlated” in R s a \mathcal{R}_{sa} R s a so that the corresponding set of uncertain rewards in the augmented state robust MDPs is not sa-rectangular. For this technique to work a sufficient condition is R = × s , a , s ′ R s a s ′ \mathcal{R} = \times_{s,a,s'}\mathcal{R}_{sas'} R = × s , a , s ′ R s a s ′ with R s a s ′ ⊂ R \mathcal{R}_{sas'} \subset\mathbb{R} R s a s ′ ⊂ R , which appears to be a very strong assumption (the adversary can choose the rewards independently across all ( s , a , s ′ ) (s,a,s') ( s , a , s ′ ) ).
This is summarized in Table 4.1 .
Table 4.1. Tractability of the expected-reward and state-augmentation reformulations under different rectangularity assumptions. A checkmark (✓) means that the resulting reformulation leads to a convex compact s-rectangular robust MDP or a compact sa-rectangular robust MDP, and a cross (×) means the contrary (the reformulation need not lead to a convex compact s-rectangular robust MDP). This table assumes U = R × P \mathcal{U} = \mathcal{R} \times\mathcal{P} U = R × P (using the notation of Section 4.2), with R , P \mathcal{R},\mathcal{P} R , P compact; P \mathcal{P} P must be convex for the first column, whereas P \mathcal{P} P need not be convex for the second column.
Main takeaways
This chapter has highlighted the connections between robust MDPs and several models of sequential decision-making under uncertainty. We summarize the main reductions in Figure 4.6 and below:
Figure 4.6. Main reductions proved in this chapter. An arrow indicates that an optimization problem reduces to another one; next to an arrow is either an assumption under which this reduction works (e.g., “P \mathcal{P} P cvx” for P \mathcal{P} P a convex set) or a label of the kind “P ← Rec s a ( P ) \mathcal{P} \leftarrow\operatorname{Rec}_{sa}(\mathcal{P}) P ← Rec s a ( P ) ”, which means that this reduction always works by replacing P \mathcal{P} P by Rec s a ( P ) \operatorname{Rec}_{sa}(\mathcal{P}) Rec s a ( P ) in the decision problem at the end of the arrow (and similarly for “P ← Rec s ( P ) \mathcal{P} \leftarrow\operatorname{Rec}_{s}(\mathcal{P}) P ← Rec s ( P ) ” and “P ← P ‾ \mathcal{P} \leftarrow\overline{\mathcal{P}} P ← P ”). The s-rectangular and sa-rectangular extensions Rec s ( P ) \operatorname{Rec}_{s}(\mathcal{P}) Rec s ( P ) and Rec s a ( P ) \operatorname{Rec}_{sa}(\mathcal{P}) Rec s a ( P ) have been defined in § 4.4 . The set P ‾ \overline{\mathcal{P}} P is the set of expected kernels induced by a set of measures V ⊂ Δ ( Δ ( S ) S × A ) \mathcal{V} \subset\Delta(\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}}) V ⊂ Δ ( Δ ( S ) S × A ) as defined in § 5.2 . The reduction from risk-averse MDPs to robust MDPs for nested coherent risk measures has only been sketched in § 5.5 and is added here for the sake of completeness.
Section 4.1: in the case where the transition probabilities are fixed and the rewards are unknown, optimal policies can be chosen stationary robust MDPs can be solved via mathematical programming, even when the uncertainty set (for the rewards) is non-rectangular. This approach is based on the fact that the return is linear in the state-action frequencies and in the instantaneous rewards.
Section 4.2 : when both the rewards and the transitions are unknown, things are much more complicated. In particular the Bellman operators now involve a term P ( s , a ) ⊤ r ( s , a ) P(s,a)^{\top}r(s,a) P ( s , a ) ⊤ r ( s , a ) which is bilinear in the variables P P P and r r r . In general this leads to difficult (NP-hard) optimization problems even when the uncertainty sets for P P P and r r r are s-rectangular, except when the “effective uncertainty set” (see (4.7)) is convex (and s-rectangular).
Section 4.3: for multi-objective MDPs, the infimum as a scalarization function yields robust MDPs, which can be interpreted as “worst-case” multi-model MDPs (with linear scalarization function). We can recover Pareto optimal policies by solving robust MDPs, but in many cases it is NP-hard to do so (except when only the rewards are uncertain and the transitions are common across models).
Section 4.5: for the discounted return and the mean-payoff, sa-rectangular robust MDPs can be formulated as perfect-information stochastic games (SGs). For the discounted return, s-rectangular robust MDPs can be formulated as concurrent stochastic games. The “information” available to the decision maker (in robust MDPs) and the players (in stochastic games) differ: in SGs, histories include the past decisions chosen by all players, whereas in robust MDPs we cannot observe the values of the past transition probabilities. This difference is irrelevant when stationary optimal strategies exist in SGs, as is the case for the objectives described above.
Section 4.6: distributionally robust MDPs is a natural extension to robust MDPs that allow for distributional information over the set of feasible transition probabilities. Perhaps surprisingly, distributionally robust MDPs (where the transition probabilities are sampled independently across each period) can be reformulated as robust MDPs with stationary transition probabilities.
Section 4.7: under an additional assumption that the instantaneous rewards are independent of the next state, several non-rectangular uncertainty sets (including r-rectangular and some extensions of this model) can be solved by dynamic programming over the original set of states and in fact can be replaced by their sa-rectangular extension. The model of Lightning-Does-not-Strike-Twice and its generalization of k-rectangular uncertainty sets require a state augmentation to recover dynamic programming equations.
Section 4.8: risk-averse MDPs with coherent risk measures can be reformulated as robust MDPs with non-rectangular uncertainty; to recover rectangular uncertainty sets one can resort to nested coherent risk measures. These results are only derived for the policy evaluation problem (i.e., for a fixed policy of the decision maker) to simplify the exposition.
References
For § 4.1 I follow the model description in [51 ] (robust MDPs with uncertain rewards), § 4.2 extends this model to joint uncertainty in rewards and transitions. The optimality of stationary policies for the case of uncertain rewards and fixed transitions is in [GDK+24], and interestingly uncertainty in rewards is related to state entropy regularization [2 ] . It is worth mentioning that some seminal robust MDP papers allow (such as [82 ] ) both r r r and P P P to be unknown but do assume that the rewards do not depend on the next state. To the best of my knowledge, the hardness result for the case of convex rectangular robust MDPs with uncertain rewards and transitions (Proposition 4.3) is new; it is worth mentioning a closely related result (Proposition 10 in [39 ] ) for the case of finite uncertainty sets.
For § 4.3 , I refer to [69 ] for a survey, to [RRS15] for a very complete paper studying several different objectives (mean-payoff, limsup and liminf of the payoff, discounted returns, etc), and to [64 , 44 , 16 ] for some more recent works. The vast majority of the papers on multi-objective MDPs (including the ones cited in the previous sentence) focus on the case where the transitions are common across all models, with the notable exception of [71 , 29 ] who focus on interval MDPs. The relation between the convexity properties of the achievable set R ( Π ) R(\Pi) R ( Π ) and the possibility to recover Pareto optimal policies from linear scalarization is well-known, see e.g. [18 , 31 ] . The case of multi-model MDPs , who study the scalarized problem (4.10) has been studied under different names over the years: [17 , 43 ] call it soft-robust MDPs, [9 , 8 ] call it concurrent MDPs, [67 ] call it multiple-environment MDPs. There exist many papers showing that the scalarized problem (with a linear function) is difficult, see e.g. [49 ] , [8 ] , as well as [45 ] (for the case of the scalarized problem with the infimum function).
For § 4.4 (time-varying transitions) I follow Section 2 in [81 ] for the description of the sets of time-varying uncertainty, which provides a rigorous analysis of the several possible corner cases that occur when inspecting various restrictions on the sets of policies (e.g., the decision maker can only use deterministic policies potentially history-dependent) and uncertainty sets (s-rectangular, sa-rectangular, non-rectangular, nonconvex ...). [48 ] also focuses on the relation between time-varying transition probabilities, rectangularity, and so-called “game formulations” of robust sequential decision problems. The connection between the policy evaluation problem for time-varying non-rectangular uncertainty sets and POMDPs is from conversations with Marek Petrik and with Stéphane Gaubert and Marianne Akian; I believe that it is known more widely but I am not aware of a more formal argument. The two papers [46 , 79 ] show hardness results for stationary decision makers in POMDPs.
For § 5.1 (connection with stochastic games) I refer to [59 , 56 ] for classical textbooks on SGs and to [47 , 62 ] for shorter recent surveys. The connection between stochastic games and robust MDPs can be found in several places, including [75 ] , section 5 of [36 ] , the last paragraph of page 7 of [82 ] , the last paragraph of Section 2 in [80 ] , and the introduction of [23 ] . Proposition 4.6 is in [24 ] . The Mertens-Neyman theorem is proved in [55 ] .
For § 5.2 (distributionally robust MDPs) I adapted [82 ] to the case of uncertain transitions ([82 ] focuses on uncertain rewards and uncertain transitions, but assumes that rewards are independent of the next state, which is not without loss of generality, see § 4.2 ). I refer to [49 , 9 ] for hardness results for multi-model MDPs with discounted return.
For § 5.3 (non-rectangular uncertainty sets), I follow [25 ] for the analysis of r-rectangular models. The model of r-rectangularity was introduced in [22 ] for policy evaluation, and the analysis of optimal policies was done in [27 ] ; the equivalence with sa-rectangularity over an extended state set was suggested to the author by Erick Delage in 2019. For the connections between non-rectangular models and their rectangular extensions when rewards are independent of the next state I follow [25 ] (see also [48 ] for the finite-horizon case and connection with multistage stochastic optimization). The LightningDoes-not-Strike-Twice model is from [53 ] and the k-rectangular model of uncertainty is by the same authors in [54 ] .
For § 5.5 (connection with risk-averse MDPs) I follow [13 ] for the formulation of CVaR risk-averse MDPs as a robust MDP with non-rectangular uncertainty sets; recent such formulations can be found for instance in [76 , 77 ] . Conditional risk measures are introduced in [66 ] . The reader interested in Markovian risk measures for MDPs can refer to [68 ] ; the main results about time-consistent, law-invariant risk measures are in [40 , 74 , 52 ] . An excellent textbook on risk measures is [20 ] .
Appendices
Background in topology and convex analysis
Compactness and continuity. The following are used in Chapter 2. We want to study the compactness of Π H \Pi_{\mathcal{H}} Π H . Note that Π H \Pi_{\mathcal{H}} Π H , the set of all history-dependent policies, is contained in an infinite-dimensional space since Π H = × h ∈ H Δ ( A ) \Pi_{\mathcal{H}} = \times_{h \in\mathcal{H}} \Delta(\mathcal{A}) Π H = × h ∈ H Δ ( A ) with H = ⋃ t ∈ N ( S × A ) t × S \mathcal{H} = \bigcup_{t \in\mathbb{N}} (\mathcal{S} \times\mathcal{A})^{t} \times\mathcal{S} H = ⋃ t ∈ N ( S × A ) t × S . Therefore we need to be careful when describing its “compactness” (a notion that in all generality depends on the topology chosen over Π H \Pi_{\mathcal{H}} Π H ). We use the product topology: convergence in this topology means convergence of the action distributions at each fixed finite history. The product topology has the following important property.
Theorem A.1 (Tykhonoff’s theorem). The product of any collection of compact topological spaces is compact for the product topology.
We recall the definition of upper semicontinuity here.
Definition A.1 (Upper semicontinuity). Let f : X → R ∪ { + ∞ , − ∞ } f : \mathcal{X} \to\mathbb{R} \cup\{+\infty, -\infty\} f : X → R ∪ { + ∞ , − ∞ } . The function f f f is upper semicontinuous on X \mathcal{X} X if for every y ∈ R y \in\mathbb{R} y ∈ R the set { x ∈ X ∣ f ( x ) < y } \{x \in\mathcal{X} \mid f(x) < y\} { x ∈ X ∣ f ( x ) < y } is open in X \mathcal{X} X .
In the definition above X \mathcal{X} X is a topological space, so the upper semicontinuity property of f f f depends on the topology chosen on X \mathcal{X} X . The product topology makes Π H \Pi_{\mathcal{H}} Π H compact (from Tykhonoff’s theorem above), and then we can use Weierstrass’ theorem in the following general version.
Theorem A.2 (Weierstrass theorem). Let X \mathcal{X} X be a nonempty compact topological space and f : X → R ∪ { + ∞ , − ∞ } f : \mathcal{X} \to\mathbb{R} \cup\{+\infty, -\infty\} f : X → R ∪ { + ∞ , − ∞ } be upper semicontinuous . Then f f f attains its supremum over X \mathcal{X} X .
Minimax theorems. In Chapter 3 and Chapter 4 we use strong duality in minimax problems a few times. The first such result was obtained by von Neumann in 1928 for continuous quasi-convex quasi-concave functions over simplexes (although people generally refer to this result only for bilinear objectives); for our purpose we use the following version of Sion’s theorem , which allows for more general domains. I only give the version we need (i.e., the domains belong to Euclidean spaces and the objective is continuous in each variable separately), but Sion’s theorem applies more broadly.
Theorem A.3 (Sion’s theorem (for our purpose)). Let X , Y \mathcal{X},\mathcal{Y} X , Y be nonempty convex compact subsets of Euclidean spaces and f : X × Y → R f:\mathcal{X}\times\mathcal{Y}\to\mathbb{R} f : X × Y → R . Assume that:
For any y ∈ Y y\in\mathcal{Y} y ∈ Y , x ↦ f ( x , y ) x\mapsto f(x,y) x ↦ f ( x , y ) is continuous and quasi-concave on X \mathcal{X} X ,
For any x ∈ X x\in\mathcal{X} x ∈ X , y ↦ f ( x , y ) y\mapsto f(x,y) y ↦ f ( x , y ) is continuous and quasi-convex on Y \mathcal{Y} Y .
Then we have that sup x ∈ X inf y ∈ Y f ( x , y ) = inf y ∈ Y sup x ∈ X f ( x , y ) \sup_{x\in\mathcal{X}}\inf_{y\in\mathcal{Y}}f(x,y)=\inf_{y\in\mathcal{Y}}\sup_{x\in\mathcal{X}}f(x,y) sup x ∈ X inf y ∈ Y f ( x , y ) = inf y ∈ Y sup x ∈ X f ( x , y ) and all the suprema and infima are attained.
Convex geometry and analysis. In Chapter 3 and Chapter 4 we use the following results from convex analysis and convex geometry. We start with Banach’s fixed point theorem.
Theorem A.4 (Banach’s fixed point theorem). Let X \mathcal{X} X be a complete metric space. A contraction T : X → X T:\mathcal{X}\to\mathcal{X} T : X → X has a unique fixed point v T ∈ X v_T\in\mathcal{X} v T ∈ X , and for any v 0 ∈ X v_0\in\mathcal{X} v 0 ∈ X , lim t → + ∞ T t v 0 \lim_{t\to+\infty}T^t v_0 lim t → + ∞ T t v 0 exists and lim t → + ∞ T t v 0 = v T \lim_{t\to+\infty}T^t v_0=v_T lim t → + ∞ T t v 0 = v T .
Theorem A.5 (Supporting hyperplane theorem for concave functions). Let X \mathcal{X} X be a nonempty convex compact subset of R n \mathbb{R}^n R n and let f : X → R f:\mathcal{X}\to\mathbb{R} f : X → R be a continuous concave function. For every x 0 x_0 x 0 in the relative interior of X \mathcal{X} X there exists g ∈ R n g\in\mathbb{R}^n g ∈ R n such that f ( x ) ≤ f ( x 0 ) + g ⊤ ( x − x 0 ) f(x)\leq f(x_0)+g^\top(x-x_0) f ( x ) ≤ f ( x 0 ) + g ⊤ ( x − x 0 ) , ∀ x ∈ X \forall x\in\mathcal{X} ∀ x ∈ X .
The vector g g g in the above theorem is a supergradient of f f f at x 0 x_0 x 0 ; we refer to [63 ] , Theorem 23.4 (which is stated in the more familiar case of subgradients of proper continuous convex functions).
The next theorem is known as the separating hyperplane theorem ; we give it here its statement for Euclidean space. We refer to Section 11 in [63 ] for more general statements and to Section 2.5 in [10 ] for the statement below.
Theorem A.6. Let X , Y ⊂ R n \mathcal{X},\mathcal{Y}\subset\mathbb{R}^n X , Y ⊂ R n be two nonempty disjoint convex sets. Then there exists w ∈ R n w\in\mathbb{R}^n w ∈ R n , w ≠ 0 w\neq0 w = 0 and α ∈ R \alpha\in\mathbb{R} α ∈ R such that w ⊤ x ≤ α ≤ w ⊤ y w^\top x\leq\alpha\leq w^\top y w ⊤ x ≤ α ≤ w ⊤ y , ∀ ( x , y ) ∈ X × Y \forall(x,y)\in\mathcal{X}\times\mathcal{Y} ∀ ( x , y ) ∈ X × Y .
The next result provides the first-order optimality conditions for orthogonal projections (here it’s useful to draw a picture, for instance of the orthogonal projection onto the unit disk in R 2 \mathbb{R}^2 R 2 ), see pages 270-271 in [63 ] .
Theorem A.7. Let X \mathcal{X} X be a nonempty convex closed subset of R n \mathbb{R}^n R n and let x ∈ R n x\in\mathbb{R}^n x ∈ R n . The orthogonal projection x ˉ \bar{x} x ˉ of x x x onto X \mathcal{X} X is the unique minimizer of y ↦ ∥ x − y ∥ 2 y\mapsto\lVert x-y\rVert_2 y ↦ ∥ x − y ∥ 2 over y ∈ X y\in\mathcal{X} y ∈ X and is characterized by x ˉ ∈ X \bar{x}\in\mathcal{X} x ˉ ∈ X and
⟨ x ˉ − x , y − x ˉ ⟩ ≥ 0 , ∀ y ∈ X . \langle\bar{x}-x,y-\bar{x}\rangle\geq0,\ \forall y\in\mathcal{X}. ⟨ x ˉ − x , y − x ˉ ⟩ ≥ 0 , ∀ y ∈ X .
Algorithms for fixed point computations
In this appendix we present two general versions of value iteration and policy iteration and prove their convergence guarantees. Doing so enables us to analyze simultaneously the convergence rates of value iteration and policy iteration in the three distinct setups of interest in these notes: (a) nominal MDPs (Chapter 1), (b) adversarial MDPs under s-rectangularity (§ 3.2 ), (c) robust MDPs under s-rectangularity (§ 3.3 ).
We will use the following notation: we study families of monotone contractions { T θ ∣ θ ∈ Θ } \{T^\theta\mid\theta\in\Theta\} { T θ ∣ θ ∈ Θ } indexed by a compact set Θ \Theta Θ , with T θ : R S → R S T^\theta: \mathbb{R}^{S} \to\mathbb{R}^{S} T θ : R S → R S (for S S S a finite set). The rate of contraction for ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∞ is γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) for each contraction T θ T^\theta T θ , and we write v θ ∈ R S v^\theta\in\mathbb{R}^{S} v θ ∈ R S for its fixed point. We will consider the “componentwise supremum” of the family, defined as the operator T ‾ : R S → R S \overline{T} : \mathbb{R}^{S} \to\mathbb{R}^{S} T : R S → R S with
T ‾ v ( s ) = sup θ ∈ Θ T θ v ( s ) , ∀ v ∈ R S , s ∈ S . \overline{T}v(s) = \sup_{\theta\in\Theta} T^\theta v(s), \ \forall v \in\mathbb{R}^{S}, s \in S. T v ( s ) = θ ∈ Θ sup T θ v ( s ) , ∀ v ∈ R S , s ∈ S .
We assume that this supremum is attained on each component here (this is true for our applications for MDPs and robust MDPs). We know that T ‾ \overline{T} T is a monotone contraction (the arguments follow similarly as in the proof of Proposition 1.1), and we define v ‾ ∈ R S \overline{v} \in\mathbb{R}^{S} v ∈ R S as its unique fixed point. Finally, the attainability property is defined as follows.
Definition B.1. The family { T θ ∣ θ ∈ Θ } \{T^\theta\mid\theta\in\Theta\} { T θ ∣ θ ∈ Θ } satisfies the attainability property with respect to the contraction T ‾ \overline{T} T if for any v ∈ R S v \in\mathbb{R}^{S} v ∈ R S , there exists θ ‾ ∈ Θ \overline{\theta} \in\Theta θ ∈ Θ such that T ‾ v = T θ ‾ v \overline{T}v = T^{\overline{\theta}}v T v = T θ v .
The value iteration and policy iteration algorithms are two methods to compute v ‾ \overline{v} v for families of monotone contractions satisfying the attainability property.
Value iteration. Here we rederive the convergence rate of Picard’s iteration.
Proposition B.1. Let ( v t ) t ∈ N (v^t)_{t\in\mathbb{N}} ( v t ) t ∈ N be the sequence of vectors generated by Algorithm 7.
The sequence ( v t ) t ∈ N (v^t)_{t\in\mathbb{N}} ( v t ) t ∈ N converges to v ‾ \overline{v} v at a geometric rate of γ \gamma γ :
∥ v ‾ − v t ∥ ∞ ≤ γ t ∥ v ‾ − v 0 ∥ ∞ \lVert\overline{v} - v^t \rVert_\infty\leq\gamma^t \lVert\overline{v} - v^0 \rVert_\infty ∥ v − v t ∥ ∞ ≤ γ t ∥ v − v 0 ∥ ∞
Algorithm 7 Value iteration (general algorithm)
Input: A compact set Θ \Theta Θ , a family of monotone contractions { T θ ∣ θ ∈ Θ } \{T^\theta\mid\theta\in\Theta\} { T θ ∣ θ ∈ Θ } with constant γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) , satisfying the attainability property with respect to T ˉ \bar{T} T ˉ
Initialize v 0 ∈ R S v^0 \in\mathbb{R}^S v 0 ∈ R S .
for t ∈ N t \in\mathbb{N} t ∈ N do
v t + 1 = T ˉ v t v^{t+1}=\bar{T}v^t v t + 1 = T ˉ v t
end for
Let ϵ > 0 \epsilon> 0 ϵ > 0 . Assume that ∥ v ˉ − v t ∥ ∞ ≤ ϵ \|\bar{v}-v^t\|_\infty\leq\epsilon ∥ v ˉ − v t ∥ ∞ ≤ ϵ for some t ∈ N t \in\mathbb{N} t ∈ N and let θ ∈ Θ \theta\in\Theta θ ∈ Θ such that T θ v t = T ˉ v t T^\theta v^t=\bar{T}v^t T θ v t = T ˉ v t . Then the fixed point of T θ T^\theta T θ satisfies ∥ v ˉ − v θ ∥ ∞ ≤ 2 ϵ γ ( 1 − γ ) − 1 \|\bar{v}-v^\theta\|_\infty\leq2\epsilon\gamma(1-\gamma)^{-1} ∥ v ˉ − v θ ∥ ∞ ≤ 2 ϵ γ ( 1 − γ ) − 1 . Additionally, we can check that ∥ v ˉ − v t ∥ ∞ ≤ ϵ \|\bar{v}-v^t\|_\infty\leq\epsilon ∥ v ˉ − v t ∥ ∞ ≤ ϵ by checking ∥ T ˉ v t − v t ∥ ∞ ≤ ϵ ( 1 − γ ) \|\bar{T}v^t-v^t\|_\infty\leq\epsilon(1-\gamma) ∥ T ˉ v t − v t ∥ ∞ ≤ ϵ ( 1 − γ ) , and that last inequality holds after at most ⌊ 1 1 − γ log ( ∥ v 1 − v 0 ∥ ∞ ϵ ( 1 − γ ) ) ⌋ + 1 \left\lfloor\frac{1}{1-\gamma}\log\left(\frac{\|v^1-v^0\|_\infty}{\epsilon(1-\gamma)}\right)\right\rfloor+1 ⌊ 1 − γ 1 log ( ϵ ( 1 − γ ) ∥ v 1 − v 0 ∥ ∞ ) ⌋ + 1 iterations of Algorithm 7.
Proof. 1. We have
∥ v ˉ − v t + 1 ∥ ∞ = ∥ T ˉ v ˉ − T ˉ v t ∥ ∞ ≤ γ ∥ v ˉ − v t ∥ ∞ \|\bar{v}-v^{t+1}\|_\infty=\|\bar{T}\bar{v}-\bar{T}v^t\|_\infty\leq\gamma\|\bar{v}-v^t\|_\infty ∥ v ˉ − v t + 1 ∥ ∞ = ∥ T ˉ v ˉ − T ˉ v t ∥ ∞ ≤ γ ∥ v ˉ − v t ∥ ∞
where the inequality uses the contraction property of T ˉ \bar{T} T ˉ . We get the desired inequality by induction.
For θ \theta θ constructed as in the statement of the proposition, we have
∥ v θ − v ˉ ∥ ∞ = ∥ v θ − T θ v t + T θ v t − v ˉ ∥ ∞ ≤ ∥ v θ − T θ v t ∥ ∞ + ∥ T θ v t − v ˉ ∥ ∞ = ∥ T θ v θ − T θ v t ∥ ∞ + ∥ T ˉ v t − T ˉ v ˉ ∥ ∞ ≤ γ ∥ v θ − v t ∥ ∞ + γ ∥ v t − v ˉ ∥ ∞ ≤ γ ∥ v θ − v ˉ ∥ ∞ + 2 γ ∥ v t − v ˉ ∥ ∞ \begin{aligned}
\|v^\theta-\bar{v}\|_\infty&=\|v^\theta-T^\theta v^t+T^\theta v^t-\bar{v}\|_\infty\\
&\leq\|v^\theta-T^\theta v^t\|_\infty+\|T^\theta v^t-\bar{v}\|_\infty\\
&=\|T^\theta v^\theta-T^\theta v^t\|_\infty+\|\bar{T}v^t-\bar{T}\bar{v}\|_\infty\\
&\leq\gamma\|v^\theta-v^t\|_\infty+\gamma\|v^t-\bar{v}\|_\infty\\
&\leq\gamma\|v^\theta-\bar{v}\|_\infty+2\gamma\|v^t-\bar{v}\|_\infty
\end{aligned} ∥ v θ − v ˉ ∥ ∞ = ∥ v θ − T θ v t + T θ v t − v ˉ ∥ ∞ ≤ ∥ v θ − T θ v t ∥ ∞ + ∥ T θ v t − v ˉ ∥ ∞ = ∥ T θ v θ − T θ v t ∥ ∞ + ∥ T ˉ v t − T ˉ v ˉ ∥ ∞ ≤ γ ∥ v θ − v t ∥ ∞ + γ ∥ v t − v ˉ ∥ ∞ ≤ γ ∥ v θ − v ˉ ∥ ∞ + 2 γ ∥ v t − v ˉ ∥ ∞
and rearranging the terms yields the first desired inequality.
Now note that
∥ v ˉ − v t ∥ ∞ = ∥ v ˉ − T ˉ v t + T ˉ v t − v t ∥ ∞ ≤ γ ∥ v ˉ − v t ∥ ∞ + ∥ T ˉ v t − v t ∥ ∞ \|\bar{v}-v^t\|_\infty=\|\bar{v}-\bar{T}v^t+\bar{T}v^t-v^t\|_\infty\leq\gamma\|\bar{v}-v^t\|_\infty+\|\bar{T}v^t-v^t\|_\infty ∥ v ˉ − v t ∥ ∞ = ∥ v ˉ − T ˉ v t + T ˉ v t − v t ∥ ∞ ≤ γ ∥ v ˉ − v t ∥ ∞ + ∥ T ˉ v t − v t ∥ ∞
so that rearranging we obtained
∥ v ˉ − v t ∥ ∞ ≤ ∥ T ˉ v t − v t ∥ ∞ 1 − γ . \|\bar{v}-v^t\|_\infty\leq\frac{\|\bar{T}v^t-v^t\|_\infty}{1-\gamma}. ∥ v ˉ − v t ∥ ∞ ≤ 1 − γ ∥ T ˉ v t − v t ∥ ∞ .
Now ∥ T ˉ v t − v t ∥ ∞ = ∥ v t + 1 − v t ∥ ∞ ≤ γ ∥ v t − v t − 1 ∥ ∞ \|\bar{T}v^t-v^t\|_\infty=\|v^{t+1}-v^t\|_\infty\leq\gamma\|v^t-v^{t-1}\|_\infty ∥ T ˉ v t − v t ∥ ∞ = ∥ v t + 1 − v t ∥ ∞ ≤ γ ∥ v t − v t − 1 ∥ ∞ hence ∥ v t + 1 − v t ∥ ∞ ≤ γ t ∥ v 1 − v 0 ∥ ∞ \|v^{t+1}-v^t\|_\infty\leq\gamma^t\|v^1-v^0\|_\infty ∥ v t + 1 − v t ∥ ∞ ≤ γ t ∥ v 1 − v 0 ∥ ∞ . To ensure ∥ v ˉ − v t ∥ ∞ ≤ ϵ \|\bar{v}-v^t\|_\infty\leq\epsilon ∥ v ˉ − v t ∥ ∞ ≤ ϵ it is sufficient that γ t ∥ v 1 − v 0 ∥ ∞ ≤ ϵ ( 1 − γ ) \gamma^t\|v^1-v^0\|_\infty\leq\epsilon(1-\gamma) γ t ∥ v 1 − v 0 ∥ ∞ ≤ ϵ ( 1 − γ ) , for which we can take t ˉ = ⌊ 1 1 − γ log ( ∥ v 1 − v 0 ∥ ∞ ϵ ( 1 − γ ) ) ⌋ + 1 \bar{t}=\left\lfloor\frac{1}{1-\gamma}\log\left(\frac{\|v^1-v^0\|_\infty}{\epsilon(1-\gamma)}\right)\right\rfloor+1 t ˉ = ⌊ 1 − γ 1 log ( ϵ ( 1 − γ ) ∥ v 1 − v 0 ∥ ∞ ) ⌋ + 1 .
Algorithm 8 Policy iteration (general algorithm)
Input: A compact set Θ \Theta Θ , a family of monotone contractions { T θ ∣ θ ∈ Θ } \{T^\theta\mid\theta\in\Theta\} { T θ ∣ θ ∈ Θ } with constant γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) , satisfying the attainability property with respect to T ‾ \overline{T} T
Initialize θ 0 ∈ Θ \theta^0 \in\Theta θ 0 ∈ Θ , v 0 v^0 v 0 the fixed point of T θ 0 T^{\theta^0} T θ 0
for t ∈ N t \in\mathbb{N} t ∈ N do
Choose θ t + 1 ∈ Θ \theta^{t+1} \in\Theta θ t + 1 ∈ Θ such that max θ ∈ Θ T θ v t ( s ) = T θ t + 1 v t ( s ) \max_{\theta\in\Theta} T^\theta v^t(s) = T^{\theta^{t+1}}v^t(s) max θ ∈ Θ T θ v t ( s ) = T θ t + 1 v t ( s ) for all s ∈ S s \in\mathcal{S} s ∈ S (and equal to θ t \theta^t θ t if we can)
Choose v t + 1 ∈ R S v^{t+1} \in\mathbb{R}^{\mathcal{S}} v t + 1 ∈ R S as the fixed point of T θ t + 1 T^{\theta^{t+1}} T θ t + 1
end for
Policy iteration. The convergence of Algorithm 8 is a little longer to prove than for Algorithm 7 but uses the same ideas; again we only need the monotonicity, contractivity, and attainability properties.
Proposition B.2. Let ( θ t ) t ∈ N (\theta^t)_{t\in\mathbb{N}} ( θ t ) t ∈ N be the sequence generated by Algorithm 8. Then ∥ v ˉ − v θ t ∥ ∞ ≤ γ t ∥ v ˉ − v θ 0 ∥ ∞ \lVert\bar{v}-v^{\theta^t}\rVert_\infty\leq\gamma^t\lVert\bar{v}-v^{\theta^0}\rVert_\infty ∥ v ˉ − v θ t ∥ ∞ ≤ γ t ∥ v ˉ − v θ 0 ∥ ∞ .
Proof. Let t ∈ N t \in\mathbb{N} t ∈ N . Note that v θ t ≤ v θ t + 1 v^{\theta^t} \leq v^{\theta^{t+1}} v θ t ≤ v θ t + 1 . Indeed
T θ t + 1 v θ t = T ‾ v θ t ≥ T θ t v θ t = v θ t . T^{\theta^{t+1}}v^{\theta^t}=\overline{T}v^{\theta^t}\geq T^{\theta^t}v^{\theta^t}=v^{\theta^t}. T θ t + 1 v θ t = T v θ t ≥ T θ t v θ t = v θ t .
Using then the monotonicity and contraction properties of the operator T θ t + 1 T^{\theta^{t+1}} T θ t + 1 , we can apply T θ t + 1 T^{\theta^{t+1}} T θ t + 1 on both sides of the inequality T θ t + 1 v θ t ≥ v θ t T^{\theta^{t+1}}v^{\theta^t}\geq v^{\theta^t} T θ t + 1 v θ t ≥ v θ t . This yields v θ t ≤ v θ t + 1 v^{\theta^t}\leq v^{\theta^{t+1}} v θ t ≤ v θ t + 1 .
We now prove the desired convergence rate. We first note that
v ˉ − v θ t + 1 = v ˉ − T θ t + 1 v θ t + 1 = v ˉ − T θ t + 1 v θ t + T θ t + 1 v θ t − T θ t + 1 v θ t + 1 . \bar{v}-v^{\theta^{t+1}}=\bar{v}-T^{\theta^{t+1}}v^{\theta^{t+1}}=\bar{v}-T^{\theta^{t+1}}v^{\theta^t}+T^{\theta^{t+1}}v^{\theta^t}-T^{\theta^{t+1}}v^{\theta^{t+1}}. v ˉ − v θ t + 1 = v ˉ − T θ t + 1 v θ t + 1 = v ˉ − T θ t + 1 v θ t + T θ t + 1 v θ t − T θ t + 1 v θ t + 1 .
Hence we conclude that 0 ≤ v ˉ − v θ t + 1 ≤ v ˉ − T θ t + 1 v θ t 0\leq\bar{v}-v^{\theta^{t+1}}\leq\bar{v}-T^{\theta^{t+1}}v^{\theta^t} 0 ≤ v ˉ − v θ t + 1 ≤ v ˉ − T θ t + 1 v θ t since T θ t + 1 v θ t ≤ T θ t + 1 v θ t + 1 T^{\theta^{t+1}}v^{\theta^t}\leq T^{\theta^{t+1}}v^{\theta^{t+1}} T θ t + 1 v θ t ≤ T θ t + 1 v θ t + 1 (from monotonicity of T θ t + 1 T^{\theta^{t+1}} T θ t + 1 and v θ t ≤ v θ t + 1 v^{\theta^t}\leq v^{\theta^{t+1}} v θ t ≤ v θ t + 1 ). Now we get
∥ v ˉ − T θ t + 1 v θ t ∥ ∞ = ∥ T ‾ v ˉ − T ‾ v θ t ∥ ∞ ≤ γ ∥ v ˉ − v θ t ∥ ∞ . \lVert\bar{v}-T^{\theta^{t+1}}v^{\theta^t}\rVert_\infty=\lVert\overline{T}\bar{v}-\overline{T}v^{\theta^t}\rVert_\infty\leq\gamma\lVert\bar{v}-v^{\theta^t}\rVert_\infty. ∥ v ˉ − T θ t + 1 v θ t ∥ ∞ = ∥ T v ˉ − T v θ t ∥ ∞ ≤ γ ∥ v ˉ − v θ t ∥ ∞ .
By induction ∥ v ˉ − T θ t + 1 v θ t ∥ ∞ ≤ γ t + 1 ∥ v ˉ − v θ 0 ∥ ∞ \lVert\bar{v}-T^{\theta^{t+1}}v^{\theta^t}\rVert_\infty\leq\gamma^{t+1}\lVert\bar{v}-v^{\theta^0}\rVert_\infty ∥ v ˉ − T θ t + 1 v θ t ∥ ∞ ≤ γ t + 1 ∥ v ˉ − v θ 0 ∥ ∞ and since 0 ≤ v ˉ − v θ t + 1 ≤ v ˉ − T θ t + 1 v θ t 0\leq\bar{v}-v^{\theta^{t+1}}\leq\bar{v}-T^{\theta^{t+1}}v^{\theta^t} 0 ≤ v ˉ − v θ t + 1 ≤ v ˉ − T θ t + 1 v θ t we get the desired inequality on ∥ v ˉ − v θ t + 1 ∥ ∞ \lVert\bar{v}-v^{\theta^{t+1}}\rVert_\infty ∥ v ˉ − v θ t + 1 ∥ ∞ .
Bibliography