Introduction
Stochastic games are a canonical model of quantitative adversarial decision making under uncertainty [31 ] . In the classical finite discounted setting, the current configuration is a single state of an arena. The players choose actions, the arena makes a stochastic transition, and the resulting rewards are accumulated in expectation. Such games have a value and admit optimal stationary strategies. Their values are characterised as the unique fixed point of a contractive Shapley operator, giving rise to value- and policy-iteration algorithms. Underlying this theory is the affine manner in which strategic randomisation and stochastic transitions enter the expected payoff.
Probability distributions as configurations. In many systems, however, the current configuration is naturally represented by a distribution rather than a single state. Consider a distributed computing system in which a fixed population of jobs moves among a collection of servers. The system operator controls routing at some servers, while an adversary controls compromised servers. The operator seeks to spread the workload across the system, whereas the adversary seeks to concentrate it at a few servers and create bottlenecks. The resulting configuration is described by a distribution μ \mu μ over the servers, where μ ( s ) \mu(s) μ ( s ) is the fraction of jobs at server s s s . Its Shannon entropy, H ( μ ) = − ∑ s μ ( s ) log 2 μ ( s ) \mathbf{H}(\mu)=-\sum_s\mu(s)\log_2\mu(s) H ( μ ) = − ∑ s μ ( s ) log 2 μ ( s ) , measures workload dispersion: high entropy corresponds to a broadly distributed load, while low entropy indicates concentration. Analogous configurations arise in transportation networks, where mass represents vehicles distributed across locations, and in epidemiological and ecological models, where it represents populations distributed across regions or compartments.
A complementary interpretation arises in security, surveillance, and active sensing, where the distribution represents uncertainty about a hidden system state. In quantitative opacity and information-flow analysis, for example, a defender may seek to keep an observer’s belief dispersed across several possible states, while the observer seeks to concentrate it sufficiently to infer a secret [20 , 25 , 15 , 33 ] . Thus, across these settings, opposing objectives may favour either dispersion or concentration.
Motivated by these examples, we introduce stochastic games with distributional entropy objectives , two-player zero-sum games played over a finite arena whose configurations are probability distributions over its states. Shannon entropy of the distribution serves as the stage reward, and the payoff is its infinite discounted sum. We discount because this makes the Bellman operators contractions and the values well defined for every continuous stage reward; it also weights the early rounds more, so the objective is about how dispersed the distribution becomes soon. A mean-payoff variant is a natural alternative that we leave open (§ 8 ). The entropy-minimising player Min seeks to concentrate the distribution, while the entropy-maximising player Max seeks to disperse it.
Three roles for probability. Once distributions themselves become game configurations, probability plays three conceptually different roles: the current distribution may represent either population mass or uncertainty over states, the players may randomise their decisions, and the transitions of the arena may be stochastic. This raises a modelling question that does not arise for state-based games: when forming the next distribution, which sources of randomness describe fractional flow , and which describe an event that is actually realised ?
One possible interpretation is that all probabilities describe fractional flow. If a routing decision selects two routes with probabilities 0.3 and 0.7, then the corresponding fractions of the population follow those routes. A stochastic transition with probabilities 0.2 and 0.8 similarly divides the population in those proportions. Once the players’ mixed decisions are fixed, strategic and transition randomness therefore produce a single successor distribution. We call this averaged-distribution semantics .
A second possibility is that strategic randomisation selects a control regime rather than dividing the population. At the beginning of a time interval, for example, each player may randomly select one complete routing plan. The selected plans are realised, but the population subsequently moves according to their transition rates. Thus, action profiles are sampled, while stochastic transitions continue to represent fractional population flow. We call this sampled-profile semantics .
Finally, transition probabilities may themselves describe uncertainty about a common environmental condition. A transition probability may, for example, specify whether a communication link is available during the interval. The link is either available or unavailable, and all jobs at a given source server experience the same realised outcome. In this interpretation, both the control regime and the stochastic environment are realised before entropy is evaluated on the resulting distribution. We call this sampled-transition semantics .
These interpretations are indistinguishable when rewards are affine in the distribution, since affine functions commute with averaging. Entropy behaves differently: it is strictly concave, so mixing distinct distributions produces greater entropy than the corresponding average of their entropies. Formally, if μ ~ \tilde{\mu} μ ~ is a random successor distribution, Jensen’s inequality gives H ( E [ μ ~ ] ) ≥ E [ H ( μ ~ ) ] H(\mathbb{E}[\tilde{\mu}]) \ge\mathbb{E}[H(\tilde{\mu})] H ( E [ μ ~ ]) ≥ E [ H ( μ ~ )] , with strict inequality whenever μ ~ \tilde{\mu} μ ~ is not almost surely constant. Thus, averaging random outcomes before evaluating entropy need not agree with first realising those outcomes and then averaging their entropies. The three semantics accordingly capture different interpretations of probability: distributional mass or belief, randomised decisions, and uncertainty over realised worlds.
This leads to the central theoretical question of the paper: which parts of the classical theory of discounted stochastic games survive when configurations are probability distributions and rewards are nonlinear functions of those distributions? Our results show that the answer depends sharply on how randomness is interpreted.
Determinacy: averaging versus sampling. Under averaged-distribution semantics, the continuation value is applied to the distribution produced jointly by the players’ mixed choices. Because this dependence is nonlinear, the classical minimax argument no longer applies. More strongly, determinacy itself can fail, even in a four-state arena with deterministic transitions. In the game shown in Figure 1, the initial distribution places equal mass on s min s_{\min} s m i n controlled by Min and s max s_{\max} s m a x controlled by Max. Each player routes its mass to one of two absorbing states. If x x x and y y y are the probabilities with which Min and Max, respectively, choose the second absorbing state, then the normalised discounted payoff is
G γ ( x , y ) = ( 1 − γ ) + γ h ( x + y 2 ) G_\gamma(x,y)=(1-\gamma)+\gamma h\left(\frac{x+y}{2}\right) G γ ( x , y ) = ( 1 − γ ) + γ h ( 2 x + y )
where h h h is binary entropy, h ( z ) = − z log 2 z − ( 1 − z ) log 2 ( 1 − z ) h(z)=-z\log_2z-(1-z)\log_2(1-z) h ( z ) = − z log 2 z − ( 1 − z ) log 2 ( 1 − z ) . The initial distribution is uniform on { s min , s max } \{s_{\min},s_{\max}\} { s m i n , s m a x } , so the first stage has entropy 1 1 1 , and its discounted weight gives the first term ( 1 − γ ) ⋅ 1 (1-\gamma)\cdot1 ( 1 − γ ) ⋅ 1 . From the second stage on, the mass is ( 1 − x + y 2 , x + y 2 ) \left(1-\frac{x+y}{2},\frac{x+y}{2}\right) ( 1 − 2 x + y , 2 x + y ) on the two absorbing states and stays fixed there, so every later stage has entropy h ( x + y 2 ) h\left(\frac{x+y}{2}\right) h ( 2 x + y ) , and together they contribute the second term γ h ( x + y 2 ) \gamma h\left(\frac{x+y}{2}\right) γ h ( 2 x + y ) . The two orders of optimisation give
inf x ∈ [ 0 , 1 ] sup y ∈ [ 0 , 1 ] G γ ( x , y ) = 1 > ( 1 − γ ) + γ h ( 1 4 ) = sup y ∈ [ 0 , 1 ] inf x ∈ [ 0 , 1 ] G γ ( x , y ) . \inf_{x\in[0,1]}\sup_{y\in[0,1]}G_\gamma(x,y)=1>(1-\gamma)+\gamma h\left(\frac{1}{4}\right)=\sup_{y\in[0,1]}\inf_{x\in[0,1]}G_\gamma(x,y). x ∈ [ 0 , 1 ] inf y ∈ [ 0 , 1 ] sup G γ ( x , y ) = 1 > ( 1 − γ ) + γ h ( 4 1 ) = y ∈ [ 0 , 1 ] sup x ∈ [ 0 , 1 ] inf G γ ( x , y ) .
Thus, the upper and lower values differ: averaging the players’ mixed choices before applying entropy can destroy determinacy. Averaging can destroy determinacy without destroying Bellman structure: the two ordered Bellman operators remain contractions, separately characterising the upper and lower values and yielding convergent finite-horizon approximations.
The situation changes under the two sampled semantics. Once a pair of pure action profiles has been selected, it determines a scalar continuation payoff: the value at the resulting successor distribution under sampled-profile semantics, and the expected value over realised transitions under sampled-transition semantics. Randomisation over pure-profile pairs therefore produces a payoff that is bilinear in the players’ mixed strategies. The one-step interaction becomes a finite zero-sum matrix game, allowing minimax to be applied at every distribution. We prove that both sampled semantics are determined and that both players have optimal stationary randomised strategies, even against opponents whose strategies depend on the complete history of observed distributions.
We compare the values induced by the three interpretations. Even the lower value under averaged-distribution semantics dominates the value under either sampled semantics
(Theorem 5). The sampled-profile and sampled-transition values, however, are incomparable (Example 39 ). With a single entropy-minimising player, averaged-distribution and sampled-profile semantics coincide, and their common value dominates that under sampled-transition semantics (Theorem 5).
Algorithms and complexity. The Bellman characterisations yield convergent finite-horizon approximations. Under averaged-distribution semantics, separate ordered recursions approximate the upper and lower values; under either sampled semantics, each Bellman update is a finite zero-sum matrix game.
For every fixed rational discount and positive rational promise gap, the averaged-distribution (upper and lower value) gap problems and both sampled gap problems lie in PSPACE. Averaged-distribution semantics uses a rational grid for mixed actions; sampled-profile semantics uses sparse near-optimal mixtures over pure profiles; and sampled-transition semantics additionally enumerates transition realisations sequentially. Depth-first evaluation keeps the space polynomial.
For a single entropy-minimising player, the averaged-distribution and sampled-profile problems improve to NP, since pure profiles suffice over the truncated horizon; sampled-transition remains in PSPACE. Conversely, entropy optimisation is NP-hard under all three semantics, already for deterministic one-player games, where pure memoryless strategies are optimal, and remains NP-hard to approximate within a suitable fixed additive constant. This contrasts with classical discounted MDPs, which are solvable in polynomial time [29 ] .
Table 1 summarises the upper bounds. The averaged-distribution and sampled-profile one-player cases are tight; the remaining cases lie between NP-hardness and PSPACE. The lower bounds follow immediately from the one-player construction of Theorem 13.
Table 1. Upper bounds for the discounted gap problem for rational γ \gamma γ and ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 ; the general-game averaged-distribution entry holds for both V + V^+ V + and V − . V^-. V − .
Related Work. Opacity enforcement in discrete-event systems modifies system dynamics to prevent an observer from inferring a secret predicate [20 , 25 , 37 ] , while quantitative information flow studies how much uncertainty about a secret is retained or revealed through observations [15 , 23 , 33 , 8 ] . Moving-target defence can similarly be viewed as maintaining uncertainty against an entropy-minimising observer. Our setting instead treats this uncertainty as the state of a sequential two-player game in which both players influence the entropy of the evolving distribution. Entropy-based objectives have also been studied in single-agent exploration and control [17 ] , whereas we consider a zero-sum interaction between entropy minimisation and maximisation. Cyber-physical security games likewise model adversarial interaction [19 ] , but typically use state- or action-based rewards rather than Shannon entropy of the full state distribution. Entropy has appeared in stochastic games through entropic-risk objectives [11 ] , where an exponential transformation is applied to accumulated conventional reward.
Games called entropy games in [10 , 2 , 3 ] have a payoff given by the growth rate of the number of behaviours, equivalently of matrix products (in the operator view of [2 , 3 ] , a mean-payoff game with relative-entropy payments over action simplices); there is no state distribution, and the reward is not the Shannon entropy of one. Maximising the entropy (rate) of the path distribution of Markov chains, interval Markov chains and MDPs [13 , 14 , 30 ] concerns a single controller and the entropy of the induced distribution on paths, whereas we study a two-player zero-sum game on the entropy of the state distribution at each step. MDPs and Markov chains have been analysed as transformers of probability distributions, with properties given by linear predicates on the distribution, temporal and monadic logics of probability over such predicates, and safety of the distribution inside a polytope [24 , 1 , 12 , 4 ] , but not with a nonlinear quantitative payoff on the distribution. A related distinction between chance (sampled) and mass (averaged) readings of probabilities is studied by Tsai et al. [35 ] for a single agent and threshold reachability, with configurations read as mass. Belief-dependent rewards, including entropy of the belief, appear in ρ \rho ρ -POMDPs [9 ] ; there a single agent controls the belief, while here two players with opposed aims control the distribution, and the averaged and sampled semantics distinguish how randomisation reaches it.
Finally, [5 ] study entropy objectives for a single controller in a standard MDP: they minimise the worst-case (over time, after a warm-up) entropy of the state distribution induced by a policy, allow history-dependent policies, and show that memoryless policies do not suffice for their objective. For memoryless policies their update is our one-player averaged-distribution update, but they have no analogue of the sampled semantics, no adversary, and a different objective (worst case over time rather than a discounted sum). In the one-player case both hardness proofs use a one-step deterministic gadget; theirs is a randomized reduction from bounded-occurrence MAX-3SAT [34 ] , so it needs N P ≠ R P \mathrm{NP} \ne\mathrm{RP} NP = RP , whereas our Theorem 13 is a deterministic reduction from the gap of [28 ] and needs only P ≠ N P \mathrm{P} \ne\mathrm{NP} P = NP , so it also yields their hardness result under the weaker assumption.
Contributions. Our main contributions are:
Model and semantics. We introduce stochastic games with distributional entropy objectives, in which the stage reward is the Shannon entropy of the state distribution and two players with opposed aims control that distribution. We formalise three semantics (averaged distribution, sampled profile, sampled transition), which differ in how randomisation reaches the distribution, and show that they give different values: we establish dominance and incomparability relations among them.
Determinacy and Bellman characterisations. We give contractive Bellman characterisations of the upper and lower averaged values and the two sampled values. Averaged-distribution semantics may be nondetermined even on a four-state deterministic arena, whereas both sampled semantics are determined and admit optimal stationary randomised strategies.
Upper bounds. For fixed rational discount and positive rational promise gap, the averaged-distribution (upper and lower value) gap problems and both sampled gap problems lie in PSPACE; for a single entropy-minimising player, the averaged and sampled-profile problems lie in NP.
Hardness. Under all three semantics, for every discount there is a constant ϵ 0 > 0 \epsilon_0 > 0 ϵ 0 > 0 such that the gap problem is NP-hard already for deterministic one-player games, where pure memoryless strategies are optimal. The reduction is deterministic, so the hardness holds unless P = N P \mathrm{P} = \mathrm{NP} P = NP .
Problem Definition
For a finite set X X X , let D ( X ) : = { μ : X → [ 0 , 1 ] ∣ ∑ x ∈ X μ ( x ) = 1 } \mathcal{D}(X) := \{\mu: X \to[0,1] \mid\sum_{x\in X}\mu(x) = 1\} D ( X ) := { μ : X → [ 0 , 1 ] ∣ ∑ x ∈ X μ ( x ) = 1 } denote the set of probability distributions over X X X . For μ ∈ D ( X ) \mu\in\mathcal{D}(X) μ ∈ D ( X ) , let supp ( μ ) : = { x : μ ( x ) > 0 } \operatorname{supp}(\mu) := \{x : \mu(x) > 0\} supp ( μ ) := { x : μ ( x ) > 0 } , and write δ x \delta_x δ x for the Dirac distribution concentrated at x x x . For μ ∈ D ( X ) \mu\in\mathcal{D}(X) μ ∈ D ( X ) and g : X → R g : X \to\mathbb{R} g : X → R , the expectation of g g g under μ \mu μ is E x ∼ μ [ g ( x ) ] : = ∑ x μ ( x ) g ( x ) \mathbb{E}_{x\sim\mu}[g(x)] := \sum_x \mu(x)g(x) E x ∼ μ [ g ( x )] := ∑ x μ ( x ) g ( x ) . For vector-valued g g g , expectation is taken componentwise.
Entropy. The Shannon entropy of μ ∈ D ( X ) \mu\in\mathcal{D}(X) μ ∈ D ( X ) is
H ( μ ) : = − ∑ x ∈ supp ( μ ) μ ( x ) log 2 μ ( x ) , 0 log 2 0 : = 0. \mathrm{H}(\mu) := - \sum_{x\in\operatorname{supp}(\mu)} \mu(x)\log_2\mu(x), \qquad0\log_2 0 := 0. H ( μ ) := − x ∈ supp ( μ ) ∑ μ ( x ) log 2 μ ( x ) , 0 log 2 0 := 0.
For ∣ X ∣ = n |X| = n ∣ X ∣ = n , 0 ≤ H ( μ ) ≤ log 2 n 0 \le\mathrm{H}(\mu) \le\log_2 n 0 ≤ H ( μ ) ≤ log 2 n ; H \mathrm{H} H is strictly concave and continuous on D ( X ) \mathcal{D}(X) D ( X ) . For X = { 0 , 1 } X = \{0,1\} X = { 0 , 1 } , write h ( z ) : = − z log 2 z − ( 1 − z ) log 2 ( 1 − z ) h(z) := -z\log_2 z - (1-z)\log_2(1-z) h ( z ) := − z log 2 z − ( 1 − z ) log 2 ( 1 − z ) ; its restriction to [ 0 , 1 2 ] [0,\frac{1}{2}] [ 0 , 2 1 ] is a strictly increasing bijection onto [ 0 , 1 ] [0,1] [ 0 , 1 ] . See Proposition 16 , Appendix § 11 .§ 1 for these standard facts.
Our game setting. A stochastic game with distributional entropy objectives is a finite turn-based stochastic arena E = ( S , S min , S max , A , Δ , μ 0 ) \mathcal{E} = (S,S_{\min},S_{\max},A,\Delta,\mu_0) E = ( S , S m i n , S m a x , A , Δ , μ 0 ) , where S = S min ⊎ S max S = S_{\min} \uplus S_{\max} S = S m i n ⊎ S m a x , each s ∈ S s \in S s ∈ S has enabled actions A ( s ) ⊆ A A(s) \subseteq A A ( s ) ⊆ A , Δ : S × A → D ( S ) \Delta: S \times A \to\mathcal{D}(S) Δ : S × A → D ( S ) , and initial distribution μ 0 ∈ D ( S ) \mu_0 \in\mathcal{D}(S) μ 0 ∈ D ( S ) . The game state is a distribution μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) , so mass may lie simultaneously on states controlled by both players.
A pure profile a ∈ P min a \in\mathcal{P}_{\min} a ∈ P m i n assigns an enabled action a ( s ) a(s) a ( s ) to every s ∈ S min s \in S_{\min} s ∈ S m i n (likewise b ∈ P max b \in\mathcal{P}_{\max} b ∈ P m a x ). Writing ρ a , b ( s ) = a ( s ) \rho_{a,b}(s) = a(s) ρ a , b ( s ) = a ( s ) on S min S_{\min} S m i n and b ( s ) b(s) b ( s ) on S max S_{\max} S m a x , define
Φ a , b ( μ ) ( s ′ ) : = ∑ s ∈ S μ ( s ) Δ ( s , ρ a , b ( s ) , s ′ ) . \Phi_{a,b}(\mu)(s') := \sum_{s\in S} \mu(s)\Delta(s,\rho_{a,b}(s),s'). Φ a , b ( μ ) ( s ′ ) := s ∈ S ∑ μ ( s ) Δ ( s , ρ a , b ( s ) , s ′ ) .
A statewise mixed action for Min is x : S min → D ( A ) x : S_{\min} \to\mathcal{D}(A) x : S m i n → D ( A ) with x ( s ) x(s) x ( s ) supported on A ( s ) A(s) A ( s ) ; write X \mathcal{X} X for all such x x x , and analogously Y \mathcal{Y} Y for Max. With π x , y ( s ) = x ( s ) \pi_{x,y}(s) = x(s) π x , y ( s ) = x ( s ) on S min S_{\min} S m i n and y ( s ) y(s) y ( s ) on S max S_{\max} S m a x ,
Υ x , y ( μ ) ( s ′ ) : = ∑ s ∈ S μ ( s ) ∑ α ∈ A ( s ) π x , y ( s ) ( α ) Δ ( s , α , s ′ ) . \Upsilon_{x,y}(\mu)(s') := \sum_{s\in S}\mu(s)\sum_{\alpha\in A(s)}\pi_{x,y}(s)(\alpha)\Delta(s,\alpha,s'). Υ x , y ( μ ) ( s ′ ) := s ∈ S ∑ μ ( s ) α ∈ A ( s ) ∑ π x , y ( s ) ( α ) Δ ( s , α , s ′ ) .
Thus Φ a , b = Υ x , y \Phi_{a,b} = \Upsilon_{x,y} Φ a , b = Υ x , y for x ( s ) = δ a ( s ) x(s) = \delta_{a(s)} x ( s ) = δ a ( s ) and y ( s ) = δ b ( s ) y(s) = \delta_{b(s)} y ( s ) = δ b ( s ) . Mixed actions induce product profile laws x ^ ( a ) : = ∏ s ∈ S min x ( s ) ( a ( s ) ) \widehat{x}(a) := \prod_{s\in S_{\min}}x(s)(a(s)) x ( a ) := ∏ s ∈ S m i n x ( s ) ( a ( s )) and analogously y ^ \widehat{y} y .
Three semantics. In words, the semantics differ in which randomness is averaged into the distribution and which is sampled before entropy is evaluated:
Table 2.
A fourth combination, which we do not consider, averages the players’ mixed actions but samples transitions: it splits the mass at each state across actions and lets each part move to its own sampled successor. Throughout, the game starts from the distribution μ 0 \mu_0 μ 0 itself, with mass μ 0 ( s ) \mu_0(s) μ 0 ( s ) at each state s s s ; starting from a single state drawn from μ 0 \mu_0 μ 0 is the case of a Dirac start. Strategies observe the sequence of distributions. Under av nothing else is revealed; under sp the sampled profile is also revealed through the next distribution; under st the realised successors are revealed as well.
A history is h t = ( μ 0 , … , μ t ) ∈ D ( S ) t + 1 h_t = (\mu_0,\ldots,\mu_t) \in\mathcal{D}(S)^{t+1} h t = ( μ 0 , … , μ t ) ∈ D ( S ) t + 1 . Under averaged-distribution semantics (av), strategies are Borel maps from histories to X \mathcal{X} X and Y \mathcal{Y} Y denoted by π a v , t min , π a v , t max \pi^{\min}_{\mathrm{av},t}, \pi^{\max}_{\mathrm{av},t} π av , t m i n , π av , t m a x ; under sampled-profile and sampled-transition semantics (⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } ), they map histories to D ( P min ) \mathcal{D}(\mathcal{P}_{\min}) D ( P m i n ) and D ( P max ) \mathcal{D}(\mathcal{P}_{\max}) D ( P m a x ) denoted by π ⋆ , t min , π ⋆ , t max \pi_{\star,t}^{\min}, \pi_{\star,t}^{\max} π ⋆ , t m i n , π ⋆ , t m a x . Write Π min ⋆ , Π max ⋆ \Pi_{\min}^{\star}, \Pi_{\max}^{\star} Π m i n ⋆ , Π m a x ⋆ for the resulting strategy spaces. For Ξ : = S S \Xi:= \mathcal{S}^{S} Ξ := S S and ξ ∈ Ξ \xi\in\Xi ξ ∈ Ξ , define
P a , b ( ξ ) : = ∏ s ∈ S Δ ( s , ρ a , b ( s ) , ξ ( s ) ) , Ψ ξ ( μ ) ( s ′ ) : = ∑ s : ξ ( s ) = s ′ μ ( s ) . \mathcal{P}_{a,b}(\xi) := \prod_{s \in S} \Delta(s,\rho_{a,b}(s),\xi(s)), \qquad\Psi_{\xi}(\mu)(s') := \sum_{s:\xi(s)=s'} \mu(s). P a , b ( ξ ) := s ∈ S ∏ Δ ( s , ρ a , b ( s ) , ξ ( s )) , Ψ ξ ( μ ) ( s ′ ) := s : ξ ( s ) = s ′ ∑ μ ( s ) .
Given a history h t h_t h t , let x t = π a v , t min ( h t ) , y t = π a v , t max ( h t ) x_t=\pi_{\mathrm{av},t}^{\min}(h_t), y_t=\pi_{\mathrm{av},t}^{\max}(h_t) x t = π av , t m i n ( h t ) , y t = π av , t m a x ( h t ) under av and σ t = π ⋆ , t min ( h t ) , τ t = π ⋆ , t max ( h t ) \sigma_t=\pi_{\star,t}^{\min}(h_t),\tau_t=\pi_{\star,t}^{\max}(h_t) σ t = π ⋆ , t m i n ( h t ) , τ t = π ⋆ , t m a x ( h t ) under ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } , the three distribution updates are
av (averaged distribution): μ t + 1 = Υ x t , y t ( μ t ) \mu_{t+1}=\Upsilon_{x_t,y_t}(\mu_t) μ t + 1 = Υ x t , y t ( μ t ) ,
sp (sampled profile): μ t + 1 = Φ a t , b t ( μ t ) \mu_{t+1}=\Phi_{a_t,b_t}(\mu_t) μ t + 1 = Φ a t , b t ( μ t ) , \quad ( a t , b t ) ∼ σ t × τ t (a_t,b_t)\sim\sigma_t\times\tau_t ( a t , b t ) ∼ σ t × τ t ,
st (sampled transition): μ t + 1 = Ψ ξ t ( μ t ) \mu_{t+1}=\Psi_{\xi_t}(\mu_t) μ t + 1 = Ψ ξ t ( μ t ) , \quad ξ t ∼ P a t , b t \xi_t\sim\mathcal{P}_{a_t,b_t} ξ t ∼ P a t , b t conditional on ( a t , b t ) (a_t,b_t) ( a t , b t ) .
Thus av averages both strategic and transition randomness, sp samples the profile but averages transitions, and st samples both. Moreover, E ξ ∼ P a , b [ Ψ ξ ( μ ) ] = Φ a , b ( μ ) \mathbb{E}_{\xi\sim\mathcal{P}_{a,b}}[\Psi_{\xi}(\mu)]=\Phi_{a,b}(\mu) E ξ ∼ P a , b [ Ψ ξ ( μ )] = Φ a , b ( μ ) .
A strategy is stationary if it depends only on the current distribution. It is pure under av if its mixed actions are point-valued, and under sp, st if it selects a single pure profile.
These updates induce Borel kernels on D ( S ) \mathcal{D}(S) D ( S ) and hence, by the Ionescu-Tulcea theorem [21 ] , unique path measures and expectations P μ ⋆ , π min , π max \mathbb{P}_{\mu}^{\star,\pi_{\min},\pi_{\max}} P μ ⋆ , π m i n , π m a x and E μ ⋆ , π min , π max \mathbb{E}_{\mu}^{\star,\pi_{\min},\pi_{\max}} E μ ⋆ , π m i n , π m a x . Measurability details are deferred to Appendix B. For fixed x , y x,y x , y , couple ( A , B ) ∼ x ~ × y ~ (A,B)\sim\tilde{x}\times\tilde{y} ( A , B ) ∼ x ~ × y ~ and ξ ∼ P A , B \xi\sim\mathcal{P}_{A,B} ξ ∼ P A , B conditionally. Then
Υ x , y ( μ ) = E [ Φ A , B ( μ ) ] = E [ Ψ ξ ( μ ) ] . \Upsilon_{x,y}(\mu)=\mathbb{E}[\Phi_{A,B}(\mu)]=\mathbb{E}[\Psi_{\xi}(\mu)]. Υ x , y ( μ ) = E [ Φ A , B ( μ )] = E [ Ψ ξ ( μ )] .
and Jensen’s inequality gives
H ( Υ x , y ( μ ) ) ≥ E [ H ( Φ A , B ( μ ) ) ] ≥ E [ H ( Ψ ξ ( μ ) ) ] . \mathrm{H}(\Upsilon_{x,y}(\mu))\geq\mathbb{E}[\mathrm{H}(\Phi_{A,B}(\mu))]\geq\mathbb{E}[\mathrm{H}(\Psi_{\xi}(\mu))]. H ( Υ x , y ( μ )) ≥ E [ H ( Φ A , B ( μ ))] ≥ E [ H ( Ψ ξ ( μ ))] .
In words, av evaluates entropy on the averaged successor, sp on a successor that already fixes the pure profile, and st on one that also fixes each state’s realised successor. By Jensen, for fixed mixed actions each sampling step can only lower the expected one-step entropy; this does not order the values, since V s p V_{\mathrm{sp}} V sp and V s t V_{\mathrm{st}} V st are incomparable (Example 39).
This is the one-step source of the differing semantics (Lemma 17 and Corollary 22); full proofs are in Appendix B. The space D ( S ) \mathcal{D}(S) D ( S ) is compact and convex, and all three successor maps are affine and continuous (Proposition 23).
Determinacy and Decision Problems. Fix γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) . For a path ω = ( μ t ) t ≥ 0 \omega=(\mu_t)_{t\geq0} ω = ( μ t ) t ≥ 0 , define
Pay γ ( ω ) : = ( 1 − γ ) ∑ t = 0 ∞ γ t H ( μ t ) . (1) \operatorname{Pay}_{\gamma}(\omega):=(1-\gamma)\sum_{t=0}^{\infty}\gamma^t\mathrm{H}(\mu_t). \tag*{(1)} Pay γ ( ω ) := ( 1 − γ ) t = 0 ∑ ∞ γ t H ( μ t ) . (1)
Since 0 ≤ H ≤ log 2 ∣ S ∣ 0\leq\mathrm{H}\leq\log_2|S| 0 ≤ H ≤ log 2 ∣ S ∣ , this payoff is bounded. For ⋆ ∈ { a v , s p , s t } \star\in\{\mathrm{av},\mathrm{sp},\mathrm{st}\} ⋆ ∈ { av , sp , st } , define
Obj ⋆ ( π min , π max ; μ ) : = E μ ⋆ , π min , π max [ Pay γ ] . (2) \operatorname{Obj}_{\star}(\pi_{\min},\pi_{\max};\mu):=\mathbb{E}_{\mu}^{\star,\pi_{\min},\pi_{\max}}[\operatorname{Pay}_{\gamma}]. \tag*{(2)} Obj ⋆ ( π m i n , π m a x ; μ ) := E μ ⋆ , π m i n , π m a x [ Pay γ ] . (2)
By Tonelli’s theorem [16 Theorem 2.37] , this equals ( 1 − γ ) ∑ t ≥ 0 γ t E μ ⋆ , π min , π max [ H ( μ t ) ] (1-\gamma)\sum_{t\geq0}\gamma^t\mathbb{E}_{\mu}^{\star,\pi_{\min},\pi_{\max}}[\mathrm{H}(\mu_t)] ( 1 − γ ) ∑ t ≥ 0 γ t E μ ⋆ , π m i n , π m a x [ H ( μ t )] . Define the upper and lower values
V ⋆ + ( μ ) : = inf π min sup π max Obj ⋆ ( π min , π max ; μ ) , V ⋆ − ( μ ) : = sup π max inf π min Obj ⋆ ( π min , π max ; μ ) . V_{\star}^{+}(\mu):=\inf_{\pi_{\min}}\sup_{\pi_{\max}}\operatorname{Obj}_{\star}(\pi_{\min},\pi_{\max};\mu), \qquad V_{\star}^{-}(\mu):=\sup_{\pi_{\max}}\inf_{\pi_{\min}}\operatorname{Obj}_{\star}(\pi_{\min},\pi_{\max};\mu). V ⋆ + ( μ ) := π m i n inf π m a x sup Obj ⋆ ( π m i n , π m a x ; μ ) , V ⋆ − ( μ ) := π m a x sup π m i n inf Obj ⋆ ( π m i n , π m a x ; μ ) .
By the minimax inequality, we have V ⋆ − ( μ ) ≤ V ⋆ + ( μ ) V_{\star}^{-}(\mu)\leq V_{\star}^{+}(\mu) V ⋆ − ( μ ) ≤ V ⋆ + ( μ ) .
Problem: Determinacy and Decision Problems
Fix ⋆ ∈ { a v , s p , s t } \star\in\{\mathrm{av},\mathrm{sp},\mathrm{st}\} ⋆ ∈ { av , sp , st } and rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) . The game is determined from μ \mu μ under ⋆ \star ⋆ if V ⋆ − ( μ ) = V ⋆ + ( μ ) V_{\star}^{-}(\mu)=V_{\star}^{+}(\mu) V ⋆ − ( μ ) = V ⋆ + ( μ ) . An instance consists of a stochastic game with distributional entropy objectives, having rational transitions and initial distribution μ 0 \mu_0 μ 0 , together with a rational threshold v v v . For rational ε 0 > 0 \varepsilon_0>0 ε 0 > 0 , GapVal ⋆ , ε 0 ⋄ \operatorname{GapVal}_{\star,\varepsilon_0}^{\diamond} GapVal ⋆ , ε 0 ⋄ , ⋄ ∈ { + , − } \diamond\in\{+,-\} ⋄ ∈ { + , − } , is the promise problem of distinguishing
V ⋆ ⋄ ( μ 0 ) ≤ v − ε 0 from V ⋆ ⋄ ( μ 0 ) ≥ v + ε 0 . V_{\star}^{\diamond}(\mu_0)\le v-\varepsilon_0
\qquad\text{from}\qquad
V_{\star}^{\diamond}(\mu_0)\ge v+\varepsilon_0. V ⋆ ⋄ ( μ 0 ) ≤ v − ε 0 from V ⋆ ⋄ ( μ 0 ) ≥ v + ε 0 .
The exact problem ValDec ⋆ ⋄ \operatorname{ValDec}_{\star}^{\diamond} ValDec ⋆ ⋄ asks whether V ⋆ ⋄ ( μ 0 ) ≤ v V_{\star}^{\diamond}(\mu_0)\le v V ⋆ ⋄ ( μ 0 ) ≤ v . When ⋆ \star ⋆ is determined we omit ⋄ \diamond ⋄ ; unqualified ValDec \operatorname{ValDec} ValDec and GapVal ε 0 \operatorname{GapVal}_{\varepsilon_0} GapVal ε 0 denote the upper-value problems.
A promise gap instance guarantees that the value is not within ε 0 \varepsilon_0 ε 0 of the threshold v v v ; an algorithm must answer correctly only on such instances, which lets it absorb truncation, discretisation and numerical errors below ε 0 \varepsilon_0 ε 0 . The “discounted gap problem” of Table 1 is GapVal ⋆ , ε 0 ⋄ \operatorname{GapVal}_{\star,\varepsilon_0}^{\diamond} GapVal ⋆ , ε 0 ⋄ for a fixed rational discount γ \gamma γ and gap ε 0 \varepsilon_0 ε 0 ; the averaged entry covers both ⋄ ∈ { + , − } \diamond\in\{+,-\} ⋄ ∈ { + , − } .
Why discounting. The payoff ( 1 − γ ) ∑ t γ t H ( μ t ) (1-\gamma)\sum_t\gamma^t\mathrm{H}(\mu_t) ( 1 − γ ) ∑ t γ t H ( μ t ) is the expected entropy at a random stopping time T T T with Pr [ T = t ] = ( 1 − γ ) γ t \Pr[T=t]=(1-\gamma)\gamma^t Pr [ T = t ] = ( 1 − γ ) γ t , that is, the game ends after each round with probability 1 − γ 1-\gamma 1 − γ . It models sessions of unknown, bounded length, where dispersing or concentrating mass matters while the session lasts, and it makes all Bellman operators γ \gamma γ -contractions. A mean-payoff objective is natural when long-run behaviour matters, but the geometric-tail arguments used here do not apply to it, and we leave it open (§ 8 ).
Bellman Characterisations
We first give a Bellman characterisation of the four relevant value functions. Let C ( D ( S ) ) C(\mathcal{D}(S)) C ( D ( S )) denote the Banach space of continuous real-valued functions on D ( S ) \mathcal{D}(S) D ( S ) , equipped with the supremum norm ∥ f ∥ ∞ : = sup μ ∈ D ( S ) ∣ f ( μ ) ∣ \lVert f\rVert_\infty:=\sup_{\mu\in\mathcal{D}(S)}\lvert f(\mu)\rvert ∥ f ∥ ∞ := sup μ ∈ D ( S ) ∣ f ( μ )∣ .
Theorem 1 (Bellman characterisations). For f ∈ C ( D ( S ) ) f\in C(\mathcal{D}(S)) f ∈ C ( D ( S )) , define the averaged Bellman operators by
( T a v + f ) ( μ ) : = ( 1 − γ ) H ( μ ) + γ min x ∈ X max y ∈ Y f ( Υ x , y ( μ ) ) , (3) (T_{\mathrm{av}}^{+}f)(\mu):=(1-\gamma)\mathrm{H}(\mu)+\gamma\min_{x\in\mathcal{X}}\max_{y\in\mathcal{Y}}f(\Upsilon_{x,y}(\mu)), \tag*{(3)} ( T av + f ) ( μ ) := ( 1 − γ ) H ( μ ) + γ x ∈ X min y ∈ Y max f ( Υ x , y ( μ )) , (3) ( T a v − f ) ( μ ) : = ( 1 − γ ) H ( μ ) + γ max y ∈ Y min x ∈ X f ( Υ x , y ( μ ) ) . (4) (T_{\mathrm{av}}^{-}f)(\mu):=(1-\gamma)\mathrm{H}(\mu)+\gamma\max_{y\in\mathcal{Y}}\min_{x\in\mathcal{X}}f(\Upsilon_{x,y}(\mu)). \tag*{(4)} ( T av − f ) ( μ ) := ( 1 − γ ) H ( μ ) + γ y ∈ Y max x ∈ X min f ( Υ x , y ( μ )) . (4) For ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } , define
( T ⋆ f ) ( μ ) : = ( 1 − γ ) H ( μ ) + γ val ( M ⋆ f ( μ ) ) , (5) (T_{\star}f)(\mu):=(1-\gamma)\mathrm{H}(\mu)+\gamma\operatorname{val}(M_{\star}^{f}(\mu)), \tag*{(5)} ( T ⋆ f ) ( μ ) := ( 1 − γ ) H ( μ ) + γ val ( M ⋆ f ( μ )) , (5) where M ⋆ f ( μ ) M_{\star}^{f}(\mu) M ⋆ f ( μ ) is the finite matrix indexed by a ∈ P min a\in\mathcal{P}_{\min} a ∈ P m i n and b ∈ P max b\in\mathcal{P}_{\max} b ∈ P m a x , with entries
M s p f ( μ ) a , b : = f ( Φ a , b ( μ ) ) , (6) M_{\mathrm{sp}}^{f}(\mu)_{a,b}:=f(\Phi_{a,b}(\mu)), \tag*{(6)} M sp f ( μ ) a , b := f ( Φ a , b ( μ )) , (6) M s t f ( μ ) a , b : = ∑ ξ ∈ Ξ P a , b ( ξ ) f ( Ψ ξ ( μ ) ) . (7) M_{\mathrm{st}}^{f}(\mu)_{a,b}:=\sum_{\xi\in\Xi}P_{a,b}(\xi)f(\Psi_{\xi}(\mu)). \tag*{(7)} M st f ( μ ) a , b := ξ ∈ Ξ ∑ P a , b ( ξ ) f ( Ψ ξ ( μ )) . (7) Here val \operatorname{val} val is the value of the zero-sum matrix game in which Min \operatorname{Min} Min chooses rows and Max \operatorname{Max} Max chooses columns. Each of these operators maps C ( D ( S ) ) C(\mathcal{D}(S)) C ( D ( S )) into itself and is a γ \gamma γ -contraction:
∥ T f − T g ∥ ∞ ≤ γ ∥ f − g ∥ ∞ . \lVert Tf-Tg\rVert_\infty\le\gamma\lVert f-g\rVert_\infty. ∥ T f − T g ∥ ∞ ≤ γ ∥ f − g ∥ ∞ .
The unique fixed points of T a v + T_{\mathrm{av}}^{+} T av + and T a v − T_{\mathrm{av}}^{-} T av − are respectively V a v + V_{\mathrm{av}}^{+} V av + and V a v − V_{\mathrm{av}}^{-} V av − . For ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } , the unique fixed point of T ⋆ T_{\star} T ⋆ equals both V ⋆ − V_{\star}^{-} V ⋆ − and V ⋆ + V_{\star}^{+} V ⋆ + ; we write V ⋆ V_{\star} V ⋆ for this common value.
For each operator T T T and corresponding value V V V , let W 0 ≡ 0 W_0 \equiv0 W 0 ≡ 0 and W t + 1 = T W t W_{t+1}=TW_t W t + 1 = T W t . Then W t W_t W t is the corresponding t t t -stage value, with rewards at times 0 , … , t − 1 0,\ldots,t-1 0 , … , t − 1 , and
0 ≤ V ( μ ) − W t ( μ ) ≤ γ t log 2 ∣ S ∣ for every μ ∈ D ( S ) . (8) 0 \le V(\mu)-W_t(\mu) \le\gamma^t \log_2 |S| \qquad\text{for every } \mu\in\mathcal{D}(S). \tag*{(8)} 0 ≤ V ( μ ) − W t ( μ ) ≤ γ t log 2 ∣ S ∣ for every μ ∈ D ( S ) . (8)
The proof combines standard contraction and finite-horizon arguments; full details are given in Appendix § 13 . These contraction and finite-horizon arguments use only continuity of the stage reward on D ( S ) \mathcal{D}(S) D ( S ) , so they carry over to any continuous reward function of the distribution in place of H \mathbf{H} H .
The theorem immediately yields a value-iteration scheme. Starting from W 0 ≡ 0 W_0 \equiv0 W 0 ≡ 0 , repeatedly apply the appropriate Bellman operator. Once γ t log 2 ∣ S ∣ ≤ ε \gamma^t \log_2 |S| \le\varepsilon γ t log 2 ∣ S ∣ ≤ ε , the iterate W t W_t W t approximates the corresponding value uniformly within ε \varepsilon ε . The remaining algorithmic difficulty is therefore the effective evaluation of a Bellman step. In § 5 , we turn these recursions into finite computations and bound their space requirements.
Determinacy Across the Three Semantics
The three semantics behave differently with respect to determinacy. Under averaging, the continuation value is applied to a successor distribution formed by the players’ mixed choices, producing a nonlinear one-step interaction. Under either sampled semantics, each pure profile induces a scalar continuation payoff, so randomisation yields a zero-sum matrix game.
Theorem 2 (Nondeterminacy under averaging). Stochastic games with distributional entropy objectives under averaged-distribution semantics are not determined.
The four-state game of Figure 1 witnesses the result; the full argument appears in Appendix § 14 .
Remark 3 (Strategy classes and mixed extensions). Under averaged-distribution semantics each player chooses a mixed action at every state independently, and only the resulting distribution is observed. Sampled-profile strategies are mixtures over pure profiles, so they may correlate actions across states. Nondeterminacy is a statement about these averaged strategies. If Min may also randomise over its averaged strategies by a pre-game coin (a mixed extension), the payoff becomes bilinear in the two mixtures. In the game of Figure 1, a fair coin for Min over x ∈ { 0 , 1 } x \in\{0,1\} x ∈ { 0 , 1 } gives expected payoff ( 1 − γ ) + γ ⋅ 1 2 [ h ( y / 2 ) + h ( ( 1 − y ) / 2 ) ] (1-\gamma)+\gamma\cdot\frac{1}{2}[h(y/2)+h((1-y)/2)] ( 1 − γ ) + γ ⋅ 2 1 [ h ( y /2 ) + h (( 1 − y ) /2 )] , which is at most ( 1 − γ ) + γ h ( 1 / 4 ) (1-\gamma)+\gamma h(1/4) ( 1 − γ ) + γ h ( 1/4 ) for every y y y , so the mixed extension is determined there, with value V a v − V_{\mathrm{av}}^{-} V av − . This is consistent with Theorem 2 , which concerns averaged strategies: the coin is itself sampled randomness, so the mixed extension is no longer the averaged-distribution reading, in which the only randomness is the splitting of mass. This matches the determinacy of the sampled semantics (Theorem 4): once randomisation is sampled, not averaged, the minimax argument applies. A natural attempt to restore determinacy by adding a dual variable for the entropy also fails; see Remark 24.
For sampled-profile and sampled-transition semantics, fixing a pure profile reduces the one-step continuation payoff to a scalar. Thus the players’ mixed choices enter bilinearly, and von Neumann’s minimax theorem applies at each finite horizon.
Theorem 4 (Sampled-profile and sampled-transition determinacy). For every ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } , V ⋆ − = V ⋆ + V_{\star}^{-}=V_{\star}^{+} V ⋆ − = V ⋆ + . Moreover, both players have optimal stationary randomised strategies, which remain optimal against arbitrary history-dependent randomised strategies of the opponent.
Proof intuition (full proof in Appendix § 14 ). By Theorem 1 , the two sampled semantics are determined. It remains to establish stationary optimal strategies. Using V ⋆ V_{\star} V ⋆ as the continuation value gives a finite matrix game at every distribution. Measurable selection yields optimal mixed choices as functions of the current distribution, and their one-step guarantees telescope over time, proving optimality against arbitrary history-dependent opponents.
Theorem 5 (Comparison of the three semantics). The averaged values V a v + V_{\mathrm{av}}^{+} V av + and V a v − V_{\mathrm{av}}^{-} V av − are concave and, for every μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) , V a v + ( μ ) ≥ V a v − ( μ ) ≥ max { V s p ( μ ) , V s t ( μ ) } V_{\mathrm{av}}^{+}(\mu) \ge V_{\mathrm{av}}^{-}(\mu) \ge\max\{V_{\mathrm{sp}}(\mu),V_{\mathrm{st}}(\mu)\} V av + ( μ ) ≥ V av − ( μ ) ≥ max { V sp ( μ ) , V st ( μ )} . If S max = ∅ S_{\max} = \varnothing S m a x = ∅ , then V a v = V s p ≥ V s t V_{\mathrm{av}} = V_{\mathrm{sp}} \ge V_{\mathrm{st}} V av = V sp ≥ V st . In general, V s p V_{\mathrm{sp}} V sp and V s t V_{\mathrm{st}} V st are incomparable: either strict inequality can occur.
The dominance follows from concavity and Jensen’s inequality applied to the finite-horizon Bellman recursions; the two sampled values are separated by explicit examples. Full proofs appear in Appendix § 17 .
Gap Value Upper Bounds
We show that the gap problem from § 2 lies in PSPACE under all three semantics. Each proof truncates the infinite horizon and evaluates the resulting finite recursion in polynomial space. The main difference is how the one-step choices and successor values are handled finitely. Under averaged-distribution semantics, each player chooses a mixed action from a continuous simplex, and the resulting entropy values may be irrational. We therefore need both a finite discretisation of the action space with controlled error and an efficient finite-precision evaluation of entropy. The following lemma provides these two ingredients.
Lemma 6 (Entropy tools). Let n = ∣ S ∣ n = |S| n = ∣ S ∣ , and let μ , ν ∈ D ( S ) \mu,\nu\in\mathcal{D}(S) μ , ν ∈ D ( S ) , with d T V ( μ , ν ) = 1 2 ∥ μ − ν ∥ 1 d_{\mathrm{TV}}(\mu,\nu) = \frac{1}{2}\lVert\mu-\nu\rVert_{1} d TV ( μ , ν ) = 2 1 ∥ μ − ν ∥ 1 .
Continuity. Let ω n ( r ) = r log 2 ( n − 1 ) + h ( r ) \omega_n(r) = r\log_{2}(n-1) + h(r) ω n ( r ) = r log 2 ( n − 1 ) + h ( r ) , where h ( r ) = − r log 2 r − ( 1 − r ) log 2 ( 1 − r ) h(r) = -r\log_{2}r - (1-r)\log_{2}(1-r) h ( r ) = − r log 2 r − ( 1 − r ) log 2 ( 1 − r ) denotes binary entropy. For 0 ≤ r ≤ 1 − 1 / n 0 \le r \le1 - 1/n 0 ≤ r ≤ 1 − 1/ n , ω n \omega_n ω n is nondecreasing, and d T V ( μ , ν ) ≤ r d_{\mathrm{TV}}(\mu,\nu) \le r d TV ( μ , ν ) ≤ r implies
∣ H ( μ ) − H ( ν ) ∣ ≤ ω n ( r ) . \lvert\mathrm{H}(\mu)-\mathrm{H}(\nu)\rvert\le\omega_n(r). ∣ H ( μ ) − H ( ν )∣ ≤ ω n ( r ) . Moreover, h ( r ) ≤ 2 r h(r) \le2\sqrt{r} h ( r ) ≤ 2 r for 0 ≤ r ≤ 1 / 2 0 \le r \le1/2 0 ≤ r ≤ 1/2 .
Effective evaluation. Let L : = ⌈ log 2 n ⌉ L := \lceil\log_{2} n\rceil L := ⌈ log 2 n ⌉ . For rational μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) and integer p ≥ 1 p \ge1 p ≥ 1 , a deterministic algorithm computes a dyadic number H ^ p ( μ ) ∈ [ 0 , L ] \widehat{\mathrm{H}}_{p}(\mu) \in[0,L] H p ( μ ) ∈ [ 0 , L ] , with denominator 2 p 2^{p} 2 p , s.t.
∣ H ^ p ( μ ) − H ( μ ) ∣ ≤ 2 − p , \lvert\widehat{\mathrm{H}}_{p}(\mu)-\mathrm{H}(\mu)\rvert\le2^{-p}, ∣ H p ( μ ) − H ( μ )∣ ≤ 2 − p , in time polynomial in size ( μ ) + p \operatorname{size}(\mu) + p size ( μ ) + p , where size ( μ ) \operatorname{size}(\mu) size ( μ ) is the total bit length of the coordinates of μ \mu μ written as reduced fractions.
Part 1 follows from an optimal coupling argument and the chain rule for entropy. Part 2 uses a rapidly convergent series expansion for the logarithm after rescaling each nonzero coordinate to a bounded range. The full proof appears in Appendix § 15 .§ 1 .
The algorithms below and the hardness result use two properties specific to entropy, namely its modulus of continuity and effective evaluation (Lemma 6).
Theorem 7 (Gap bound under averaged-distribution semantics). Fix rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 . Then, for both the upper and the lower value, GapVal a v , ε 0 ⋄ ∈ P S P A C E \operatorname{GapVal}_{\mathrm{av},\varepsilon_0}^{\diamond} \in\mathrm{PSPACE} GapVal av , ε 0 ⋄ ∈ PSPACE for ⋄ ∈ { + , − } \diamond\in\{+,-\} ⋄ ∈ { + , − } .
Proof intuition (full proof in Appendix § 15 .§ 1 ). Since entropy is bounded by log 2 n \log_{2} n log 2 n , truncating the discounted objective after polynomially many rounds incurs only a small tail error. The resulting finite-horizon value satisfies a one-step recursion over the current distribution. The remaining difficulty is that each player chooses from a continuous simplex. Using Lemma 6, we replace these mixed-action spaces by sufficiently fine rational grids, while approximating entropy to polynomially many bits of precision. The continuity bound controls the discretisation error, and effective entropy evaluation controls the numerical error.
The resulting finite recursion is evaluated by an alternating Turing machine: one player’s grid choice is guessed existentially, the other’s universally, and the machine recurses on the successor distribution. The horizon and all required precisions are polynomially bounded, so the computation runs in alternating polynomial time and hence in PSPACE. The total truncation, discretisation, and numerical error is kept below the promised gap.
Under sampled-profile and sampled-transition semantics, no discretisation of continuous mixed actions is needed. Instead, each one-step game is indexed by pure profiles, whose number may be exponential. The following lemma shows that near-optimal mixed strategies nevertheless admit polynomial-size sparse supports.
Lemma 8 (Sparse minimax certificates). Let M ∈ [ 0 , U ] R × C M \in[0,U]^{R \times C} M ∈ [ 0 , U ] R × C be a finite zero-sum game, with the row player minimising. Write [ R ] = { 1 , … , R } [R]=\{1,\ldots,R\} [ R ] = { 1 , … , R } and [ C ] = { 1 , … , C } [C]=\{1,\ldots,C\} [ C ] = { 1 , … , C } for the rows and columns, and val ( M ) = min p max b ∈ [ C ] ∑ a p a M ( a , b ) = max q min a ∈ [ R ] ∑ b q b M ( a , b ) \operatorname{val}(M)=\min_{p}\max_{b\in[C]}\sum_{a}p_aM(a,b)=\max_{q}\min_{a\in[R]}\sum_b q_bM(a,b) val ( M ) = min p max b ∈ [ C ] ∑ a p a M ( a , b ) = max q min a ∈ [ R ] ∑ b q b M ( a , b ) over probability vectors p , q p,q p , q on rows and columns. A distribution supported on columns b 1 , … , b k b_1,\ldots,b_k b 1 , … , b k is given by weights q 1 , … , q k ≥ 0 q_1,\ldots,q_k\ge0 q 1 , … , q k ≥ 0 with ∑ i q i = 1 \sum_i q_i=1 ∑ i q i = 1 , and likewise for rows with weights p i p_i p i .
(i) (Soundness.) If a distribution supported on columns b 1 , … , b k b_1,\ldots,b_k b 1 , … , b k satisfies ∑ i q i M ( a , b i ) ≥ v \sum_i q_iM(a,b_i)\ge v ∑ i q i M ( a , b i ) ≥ v for every a ∈ [ R ] a\in[R] a ∈ [ R ] , then val ( M ) ≥ v \operatorname{val}(M)\ge v val ( M ) ≥ v . Dually, if a distribution supported on rows a 1 , … , a k a_1,\ldots,a_k a 1 , … , a k satisfies ∑ i p i M ( a i , b ) ≤ v \sum_i p_iM(a_i,b)\le v ∑ i p i M ( a i , b ) ≤ v for every b ∈ [ C ] b\in[C] b ∈ [ C ] , then val ( M ) ≤ v \operatorname{val}(M)\le v val ( M ) ≤ v .
(ii) (Existence.) If val ( M ) ≥ v + ϵ \operatorname{val}(M)\ge v+\epsilon val ( M ) ≥ v + ϵ , then for every k ≥ max { 1 , ⌈ 2 U 2 ln ( 2 R ) / ϵ 2 ⌉ } k\ge\max\{1,\lceil2U^2\ln(2R)/\epsilon^2\rceil\} k ≥ max { 1 , ⌈ 2 U 2 ln ( 2 R ) / ϵ 2 ⌉} there exist columns b 1 , … , b k b_1,\ldots,b_k b 1 , … , b k such that 1 k ∑ i M ( a , b i ) ≥ v \frac{1}{k}\sum_i M(a,b_i)\ge v k 1 ∑ i M ( a , b i ) ≥ v for every a ∈ [ R ] a\in[R] a ∈ [ R ] . Dually, if val ( M ) ≤ v − ϵ \operatorname{val}(M)\le v-\epsilon val ( M ) ≤ v − ϵ , then for every k ≥ max { 1 , ⌈ 2 U 2 ln ( 2 C ) / ϵ 2 ⌉ } k\ge\max\{1,\lceil2U^2\ln(2C)/\epsilon^2\rceil\} k ≥ max { 1 , ⌈ 2 U 2 ln ( 2 C ) / ϵ 2 ⌉} there exist rows a 1 , … , a k a_1,\ldots,a_k a 1 , … , a k such that 1 k ∑ i M ( a i , b ) ≤ v \frac{1}{k}\sum_i M(a_i,b)\le v k 1 ∑ i M ( a i , b ) ≤ v for every b ∈ [ C ] b\in[C] b ∈ [ C ] . Profiles may be repeated and every weight can be chosen equal to 1 / k 1/k 1/ k . The assertion holds for every integer k k k at least the displayed bound.
Part (ii) is due to Lipton and Young [26 ] and, independently, Althöfer [7 ] ; we use it to sparsify one player’s mixed strategies in each Bellman step.
Part (i) is immediate from the definition of val ( M ) \operatorname{val}(M) val ( M ) . For Part (ii), sample k k k pure strategies independently from an optimal mixed strategy. Hoeffding’s inequality [18 Theorem 2] , followed by a union bound over all responses, shows that some empirical uniform mixture satisfies the required inequalities.
Theorem 9 (Polynomial-space upper bound under sampled-profile semantics). Fix rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 . Then G a p V a l s p , ϵ 0 ∈ P S P A C E \mathrm{GapVal}_{\mathrm{sp},\epsilon_0}\in\mathrm{PSPACE} GapVal sp , ϵ 0 ∈ PSPACE .
Proof intuition (full proof in Appendix § 15 .§ 2 ). As under averaged-distribution semantics, we truncate the horizon and evaluate the resulting finite recursion. The difference is that each one-step game is now a finite matrix game over pure profiles, but may have exponentially many rows and columns.
By Lemma 8, near-optimal mixed strategies can be replaced by uniform mixtures over only polynomially many pure profiles. Since each pure profile has a polynomial-size description, these sparse mixtures can be guessed and checked in polynomial space. Evaluating the recursion depth-first and recomputing subproblems as needed keeps only a polynomial-size recursion stack. With the truncation, sparsification, and finite-precision errors chosen below the promise gap, this yields a PSPACE algorithm.
Theorem 10 (Polynomial-space upper bound under sampled-transition semantics). For fixed rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 , GapVal s t , ε 0 ∈ P S P A C E \operatorname{GapVal}_{\mathrm{st},\varepsilon_0} \in\mathrm{PSPACE} GapVal st , ε 0 ∈ PSPACE .
The proof, given in Appendix § 15 .§ 3 , follows Theorem 9 with one additional step. For a fixed pure-profile pair, sampled-transition semantics may produce exponentially many successor distributions, so each matrix entry is a weighted average over these realisations rather than a single recursive value. Enumerating the realisations one at a time and maintaining a running accumulator uses only polynomial space. The remaining truncation, sparsification, and error arguments are unchanged. Together, Theorems 7, 9, and 10 place the gap-value problem in PSPACE under all three semantics.
Dependence on γ \gamma γ and ε 0 \varepsilon_0 ε 0 . We treat γ \gamma γ and ε 0 \varepsilon_0 ε 0 as fixed. Tracing the proofs, the horizon is T = O ( log ( L / e ) / log ( 1 / γ ) ) T = O(\log(L/e)/\log(1/\gamma)) T = O ( log ( L / e ) / log ( 1/ γ )) , the grid denominator is polynomial in n n n , A A A , 1 / e 1/e 1/ e and 1 / ( 1 − γ ) 1/(1-\gamma) 1/ ( 1 − γ ) , and the sample size is k = O ( n ( log n ) 2 log ( 2 A ) / ( ( 1 − γ ) 2 e 2 ) ) k = O(n(\log n)^2\log(2A)/((1-\gamma)^2e^2)) k = O ( n ( log n ) 2 log ( 2 A ) / (( 1 − γ ) 2 e 2 )) . The algorithms therefore run in space polynomial in the input length, 1 / ( 1 − γ ) 1/(1-\gamma) 1/ ( 1 − γ ) and 1 / ε 0 1/\varepsilon_0 1/ ε 0 ; if γ \gamma γ and ε 0 \varepsilon_0 ε 0 are given in binary, the bounds are exponential in their bit lengths.
The complexity gap. Theorems 7, 9 and 10 give PSPACE upper bounds, while Theorem 13 gives NP-hardness already for one player. Whether the two-player gap problems lie in a lower class is open. Our alternating machine uses T = Θ ( log n ) T = \Theta(\log n) T = Θ ( log n ) rounds with polynomially long guesses, so it does not place the problem in a fixed level of the polynomial hierarchy.
The One-Player Case. Suppose S max = ∅ S_{\max} = \varnothing S m a x = ∅ . Then only Min chooses actions, and the one-step optimisation under every semantics is over Min’s mixed actions. The following lemma shows that pure profiles always suffice.
Lemma 11 (Pure strategies). Suppose S max = ∅ S_{\max} = \varnothing S m a x = ∅ . Define W 0 ≡ 0 W_0 \equiv0 W 0 ≡ 0 and W t + 1 = T ⋆ W t W_{t+1} = \mathcal{T}_{\star}W_t W t + 1 = T ⋆ W t , where T ⋆ \mathcal{T}_{\star} T ⋆ is the one-step Bellman operator under semantics ⋆ \star ⋆ . Then for every ⋆ ∈ { a v , s p , s t } \star\in\{\mathrm{av},\mathrm{sp},\mathrm{st}\} ⋆ ∈ { av , sp , st } , the minimum defining W t + 1 W_{t+1} W t + 1 is attained by a pure profile. In particular,
W t + 1 ( μ ) = ( 1 − γ ) H ( μ ) + γ min a ∈ P min C t ⋆ ( μ , a ) , where W_{t+1}(\mu) = (1-\gamma)H(\mu) + \gamma\min_{a \in\mathcal{P}_{\min}} C_t^\star(\mu,a), \qquad\text{where} W t + 1 ( μ ) = ( 1 − γ ) H ( μ ) + γ a ∈ P m i n min C t ⋆ ( μ , a ) , where C t a v ( μ , a ) = C t s p ( μ , a ) = W t ( Φ a ( μ ) ) and C t s t ( μ , a ) = ∑ ξ ∈ Ξ P a ( ξ ) W t ( Ψ ξ ( μ ) ) . C_t^{\mathrm{av}}(\mu,a) = C_t^{\mathrm{sp}}(\mu,a) = W_t(\Phi_a(\mu)) \quad\text{and} \quad C_t^{\mathrm{st}}(\mu,a) = \sum_{\xi\in\Xi} P_a(\xi) W_t(\Psi_\xi(\mu)). C t av ( μ , a ) = C t sp ( μ , a ) = W t ( Φ a ( μ )) and C t st ( μ , a ) = ξ ∈ Ξ ∑ P a ( ξ ) W t ( Ψ ξ ( μ )) .
For sp and st, the claim is immediate because the one-step objective is linear in the mixed action and therefore attains its minimum at a vertex of the simplex. For av, induction shows that each W t W_t W t is concave; Jensen’s inequality then implies that no mixed action can improve on the best pure profile. The full proof appears in Appendix § 16 .
Lemma 12 (One-player upper bounds). For fixed rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 , if S max = ∅ S_{\max} = \varnothing S m a x = ∅ , then GapVal a v , ε 0 ∈ N P \operatorname{GapVal}_{\mathrm{av},\varepsilon_0} \in\mathrm{NP} GapVal av , ε 0 ∈ NP , GapVal s p , ε 0 ∈ N P \operatorname{GapVal}_{\mathrm{sp},\varepsilon_0} \in\mathrm{NP} GapVal sp , ε 0 ∈ NP , and GapVal s t , ε 0 ∈ P S P A C E \operatorname{GapVal}_{\mathrm{st},\varepsilon_0} \in\mathrm{PSPACE} GapVal st , ε 0 ∈ PSPACE .
Proof intuition (full proof in Appendix § 16 .§ 1 ). By Lemma 11, finite-horizon optimal play may be restricted to pure profiles. Under av and sp, a polynomial-length sequence of pure profiles therefore serves as a certificate: the verifier reconstructs the induced distributions, approximates their entropies, and checks the threshold, giving NP.
Under st, fixing a pure profile still leaves stochastic transition realisations, so such a deterministic certificate does not suffice. Pure profiles nevertheless eliminate the sparsification step from the general PSPACE algorithm, yielding the stated PSPACE upper bound.
Gap Value Lower Bounds
We complement the upper bounds with a matching hardness result: even for deterministic one-player games, approximating the gap value within a fixed additive constant is NP-hard.
Theorem 13 (One-player hardness). For every rational discount γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) there exists a rational constant ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 , depending only on γ \gamma γ , such that, for every ⋆ ∈ { a v , s p , s t } \star\in\{\mathrm{av},\mathrm{sp},\mathrm{st}\} ⋆ ∈ { av , sp , st } , GapVal ⋆ , ε 0 \operatorname{GapVal}_{\star,\varepsilon_0} GapVal ⋆ , ε 0 is NP-hard, already for deterministic one-player games. In the games constructed, pure memoryless strategies are optimal (Lemma 11), so the hardness does not rely on restricting the strategy class.
Proof intuition (full proof in Appendix § 16 .§ 2 ). The reduction starts from a gap version of Set Cover. It creates one state for each universe element and one absorbing state for each candidate set. The initial distribution is uniform over the element states, and the sole player routes each element to a set containing it.
Because the game is deterministic and has only one player, all three semantics coincide. After one round the process is absorbing, so the objective differs from a constant only through the entropy of the induced distribution over set states. Using fewer sets concentrates probability mass and lowers entropy; using more sets spreads the mass and raises it. A fixed gap in the minimum cover size is therefore converted into a fixed additive gap in achievable entropy. The required Set-Cover gap follows from Petrank’s hardness result for bounded-occurrence three-dimensional matching [28 Theorem 4.4] . Choosing the threshold between the two resulting entropy ranges yields the claimed NP-hardness for all three semantics.
Exact-Threshold Complexity
The upper-bound algorithms use the promise margin to absorb errors from horizon truncation, entropy evaluation, and, where needed, discretisation or sparsification. They therefore do not yield exact-threshold decision procedures.
threshold v v v is a rational multiple of log 2 2 \log_2 2 log 2 2 , so V π ( μ 0 ) − v V^\pi(\mu_0)-v V π ( μ 0 ) − v is such a combination too. It is zero iff all its prime coefficients vanish (unique factorisation); otherwise its sign is found by numerical approximation to increasing precision, which terminates because the value is nonzero. There are finitely many policies, so exact-threshold comparison is decidable unconditionally in this case; we do not bound the running time. The argument does not extend to two players, where the optimal strategies depend on the values and the entropy constants enter polynomial matrix-game equations.
Theorem 15 (Exact-value hardness). For every ⋆ ∈ { a v , s p , s t } \star\in\{\mathrm{av},\mathrm{sp},\mathrm{st}\} ⋆ ∈ { av , sp , st } and ⋄ ∈ { + , − } \diamond\in\{+,-\} ⋄ ∈ { + , − } , ValDec ⋆ ⋄ \operatorname{ValDec}^{\diamond}_{\star} ValDec ⋆ ⋄ is NP-hard, already for deterministic single-player games. In the games constructed, pure memoryless strategies are optimal (Lemma 11).
Proof. The reduction of Theorem 13 produces a promise gap of width 2 ε 0 2\varepsilon_0 2 ε 0 around a midpoint M M M , which is generally irrational. Choose a rational v ′ v' v ′ with ∣ v ′ − M ∣ < ε 0 / 2 |v'-M| < \varepsilon_0/2 ∣ v ′ − M ∣ < ε 0 /2 , computable in polynomial time by the logarithm series used for Lemma 6. Yes-instances have value at most M − ε 0 < v ′ M-\varepsilon_0 < v' M − ε 0 < v ′ and no-instances at least M + ε 0 > v ′ M+\varepsilon_0 > v' M + ε 0 > v ′ . Hence the exact comparison V ⋆ ⋄ ( μ 0 ) ≤ v ′ V^\diamond_{\star}(\mu_0) \leq v' V ⋆ ⋄ ( μ 0 ) ≤ v ′ decides the same NP-hard language; with a single player the upper and lower values coincide, so this holds for both ⋄ ∈ { + , − } \diamond\in\{+,-\} ⋄ ∈ { + , − } . The construction is deterministic and single-player. By Lemma 11, pure profiles attain the same value as arbitrary history-dependent strategies, as shown at the start of Appendix § 16 .§ 2 ; in the constructed games all mass is absorbed after one round, so a single pure profile suffices at every step. Lemma 31 then computes the value of a pure profile, independently of the semantics.
Conclusion
We studied two-player zero-sum stochastic games whose stage reward is the Shannon entropy of the state distribution. Because entropy is nonlinear, the way randomisation enters the distribution matters: the three interpretations agree in expectation for affine rewards, but give genuinely different games here. Averaging before evaluating entropy can destroy determinacy, even on a four-state deterministic arena, whereas both sampled semantics are determined and admit optimal stationary randomised strategies. The values are only partly ordered: even the lower averaged-distribution value dominates both sampled values, while sampled-profile and sampled-transition values are incomparable in general. With a single entropy minimiser, averaged-distribution and sampled-profile semantics coincide and dominate sampled-transition semantics.
For fixed rational discount and positive gap, the gap-value problem lies in PSPACE under all three semantics, and in NP for a single entropy minimiser under averaged-distribution and sampled-profile semantics. All three semantics are NP-hard already on deterministic one-player games, so the NP bounds are tight.
Several questions remain open: the exact complexity of the remaining PSPACE cases, and exact-threshold comparison, which we can decide unconditionally only for a single player under sampled-transition semantics (conditionally for two players under that semantics, and not at all for the others). Extending the theory beyond fixed-discount objectives, in particular to mean payoff, is another natural direction, since the geometric-tail arguments used here no longer apply. The fourth combination of the readings, averaging the players’ mixed actions but sampling transitions, is also left open. More broadly, these questions ask how much of classical stochastic-game theory survives when the payoff is a nonlinear function of the evolving state distribution.
References
Guide to the appendix
Each row lists a section or subsection of the appendix, what it contains, and the part of the main text it supports.
Table 1 (PDF p. 17).
Extended Preliminaries
Finite distributions and entropy
For a finite set X X X , let
D ( X ) : = { μ : X → [ 0 , 1 ] ∣ ∑ x ∈ X μ ( x ) = 1 } . \mathcal{D}(X) := \left\{ \mu: X \to[0,1] \mathrel{\Big|} \sum_{x\in X}\mu(x)=1 \right\}. D ( X ) := { μ : X → [ 0 , 1 ] x ∈ X ∑ μ ( x ) = 1 } .
denote the set of probability distributions over X X X . The support of μ ∈ D ( X ) \mu\in\mathcal{D}(X) μ ∈ D ( X ) is supp ( μ ) : = { x ∈ X : μ ( x ) > 0 } \operatorname{supp}(\mu) := \{x \in X : \mu(x) > 0\} supp ( μ ) := { x ∈ X : μ ( x ) > 0 } , and the Dirac distribution concentrated at x ∈ X x \in X x ∈ X is denoted by δ x \delta_x δ x . For a function g : X → R g : X \to\mathbb{R} g : X → R , we write
E x ∼ μ [ g ( x ) ] : = ∑ x ∈ X μ ( x ) g ( x ) . \mathbb{E}_{x\sim\mu}[g(x)] := \sum_{x\in X}\mu(x)g(x). E x ∼ μ [ g ( x )] := x ∈ X ∑ μ ( x ) g ( x ) .
The same notation is used for vector-valued g g g , with expectation taken coordinatewise.
Shannon entropy. For a finite set X X X , the Shannon entropy of a distribution μ ∈ D ( X ) \mu\in\mathcal{D}(X) μ ∈ D ( X ) is
H ( μ ) : = − ∑ x ∈ supp ( μ ) μ ( x ) log 2 μ ( x ) , \mathbf{H}(\mu) := -\sum_{x\in\operatorname{supp}(\mu)}\mu(x)\log_2\mu(x), H ( μ ) := − x ∈ supp ( μ ) ∑ μ ( x ) log 2 μ ( x ) ,
with the convention 0 log 2 0 = 0 0\log_2 0 = 0 0 log 2 0 = 0 .
Norms and total variation distance. For a real-valued function f f f on D ( S ) \mathcal{D}(S) D ( S ) , write ∥ f ∥ ∞ : = sup μ ∈ D ( S ) ∣ f ( μ ) ∣ \lVert f\rVert_\infty:= \sup_{\mu\in\mathcal{D}(S)}|f(\mu)| ∥ f ∥ ∞ := sup μ ∈ D ( S ) ∣ f ( μ ) ∣ for its supremum norm. For a vector μ ∈ R S \mu\in\mathbb{R}^{S} μ ∈ R S , such as a signed difference of two distributions, write ∥ μ ∥ 1 : = ∑ s ∈ S ∣ μ ( s ) ∣ \lVert\mu\rVert_1 := \sum_{s\in S}|\mu(s)| ∥ μ ∥ 1 := ∑ s ∈ S ∣ μ ( s ) ∣ for its ℓ 1 \ell_1 ℓ 1 norm. For μ , ν ∈ D ( S ) \mu,\nu\in\mathcal{D}(S) μ , ν ∈ D ( S ) , the total variation distance is d T V ( μ , ν ) : = 1 2 ∥ μ − ν ∥ 1 d_{\mathrm{TV}}(\mu,\nu) := \frac{1}{2}\lVert\mu-\nu\rVert_1 d TV ( μ , ν ) := 2 1 ∥ μ − ν ∥ 1 .
Proposition 16. We will make repeated use of the following basic properties of Shannon entropy. Let X X X be a finite set with ∣ X ∣ = n |X|=n ∣ X ∣ = n , and let μ , ν ∈ D ( X ) \mu,\nu\in\mathcal{D}(X) μ , ν ∈ D ( X ) .
(i) Bounds and extremizers. 0 ≤ H ( μ ) ≤ log 2 n 0 \le\mathbf{H}(\mu) \le\log_2 n 0 ≤ H ( μ ) ≤ log 2 n . Moreover, H ( μ ) = 0 \mathbf{H}(\mu)=0 H ( μ ) = 0 if and only if μ = δ x \mu=\delta_x μ = δ x for some x ∈ X x\in X x ∈ X , and H ( μ ) = log 2 n \mathbf{H}(\mu)=\log_2 n H ( μ ) = log 2 n if and only if μ \mu μ is uniform on X X X .
(ii) Strict concavity. H \mathbf{H} H is strictly concave: for every λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] ,
H ( λ μ + ( 1 − λ ) ν ) ≥ λ H ( μ ) + ( 1 − λ ) H ( ν ) , \mathbf{H}(\lambda\mu+(1-\lambda)\nu)\geq\lambda\mathbf{H}(\mu)+(1-\lambda)\mathbf{H}(\nu), H ( λ μ + ( 1 − λ ) ν ) ≥ λ H ( μ ) + ( 1 − λ ) H ( ν ) , with strict inequality whenever μ ≠ ν \mu\neq\nu μ = ν and λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) .
(iii) Continuity. H \mathbf{H} H is continuous on the compact set D ( X ) \mathcal{D}(X) D ( X ) and is therefore bounded and uniformly continuous.
(iv) Binary entropy. For a binary state space X = { 0 , 1 } X=\{0,1\} X = { 0 , 1 } , every distribution has the form ( z , 1 − z ) (z,1-z) ( z , 1 − z ) for some z ∈ [ 0 , 1 ] z\in[0,1] z ∈ [ 0 , 1 ] . We write its entropy as h ( z ) : = − z log 2 z − ( 1 − z ) log 2 ( 1 − z ) h(z):=-z\log_2z-(1-z)\log_2(1-z) h ( z ) := − z log 2 z − ( 1 − z ) log 2 ( 1 − z ) . The function h h h is symmetric about 1 / 2 1/2 1/2 and restricts to a strictly increasing bijection from [ 0 , 1 2 ] [0,\frac{1}{2}] [ 0 , 2 1 ] onto [ 0 , 1 ] [0,1] [ 0 , 1 ] .
Figure 1 (PDF p. 19).
Proof. These are standard facts about Shannon entropy. Write f ( p ) : = − p log 2 p f(p):=-p\log_{2}p f ( p ) := − p log 2 p for p ∈ ( 0 , 1 ] p\in(0,1] p ∈ ( 0 , 1 ] and f ( 0 ) : = 0 f(0):=0 f ( 0 ) := 0 , so that H ( μ ) = ∑ x ∈ X f ( μ ( x ) ) \mathbf{H}(\mu)=\sum_{x\in X}f(\mu(x)) H ( μ ) = ∑ x ∈ X f ( μ ( x )) ; f f f is continuous on [ 0 , 1 ] [0,1] [ 0 , 1 ] since lim p → 0 + p log 2 p = 0 \lim_{p\to0+}p\log_{2}p=0 lim p → 0 + p log 2 p = 0 .
Part (i). Each term f ( μ ( x ) ) f(\mu(x)) f ( μ ( x )) is nonnegative, and f ( p ) = 0 f(p)=0 f ( p ) = 0 iff p ∈ { 0 , 1 } p\in\{0,1\} p ∈ { 0 , 1 } , so H ( μ ) = 0 \mathbf{H}(\mu)=0 H ( μ ) = 0 iff μ = δ x \mu=\delta_x μ = δ x for some x x x . For the upper bound let S : = supp ( μ ) S:=\operatorname{supp}(\mu) S := supp ( μ ) , m : = ∣ S ∣ ≤ n m:=|S|\le n m := ∣ S ∣ ≤ n , and let Y : = 1 / μ ( x ) Y:=1/\mu(x) Y := 1/ μ ( x ) on the event { x } \{x\} { x } , x ∈ S x\in S x ∈ S . Then E μ [ Y ] = m \mathbb{E}_{\mu}[Y]=m E μ [ Y ] = m and E μ [ log 2 Y ] = H ( μ ) \mathbb{E}_{\mu}[\log_{2}Y]=\mathbf{H}(\mu) E μ [ log 2 Y ] = H ( μ ) , so Jensen’s inequality for the strictly concave log 2 \log_{2} log 2 gives H ( μ ) ≤ log 2 E μ [ Y ] = log 2 m ≤ log 2 n \mathbf{H}(\mu)\le\log_{2}\mathbb{E}_{\mu}[Y]=\log_{2}m\le\log_{2}n H ( μ ) ≤ log 2 E μ [ Y ] = log 2 m ≤ log 2 n . Equality in Jensen holds iff Y Y Y is μ \mu μ -a.s. constant, i.e. μ \mu μ is uniform on S S S , and m = n m=n m = n forces S = X S=X S = X , so H ( μ ) = log 2 n \mathbf{H}(\mu)=\log_{2}n H ( μ ) = log 2 n iff μ \mu μ is uniform on X X X .
Part (ii). On ( 0 , 1 ) (0,1) ( 0 , 1 ) , f ′ ′ ( p ) = − 1 / ( p ln 2 ) < 0 f^{\prime\prime}(p)=-1/(p\ln2)<0 f ′′ ( p ) = − 1/ ( p ln 2 ) < 0 , so f f f is strictly concave there. Since f f f is continuous on [ 0 , 1 ] [0,1] [ 0 , 1 ] , it is concave on [ 0 , 1 ] [0,1] [ 0 , 1 ] . It is strictly concave on [ 0 , 1 ] [0,1] [ 0 , 1 ] : if f f f met the chord over some segment [ p , q ] [p,q] [ p , q ] at an interior point, concavity would force f f f to be affine on [ p , q ] [p,q] [ p , q ] , which is impossible because f f f is strictly concave on ( p , q ) ⊂ ( 0 , 1 ) (p,q)\subset(0,1) ( p , q ) ⊂ ( 0 , 1 ) . Applying this at μ ( x ) , ν ( x ) \mu(x),\nu(x) μ ( x ) , ν ( x ) for each x x x and summing gives concavity of H \mathbf{H} H . If μ ≠ ν \mu\ne\nu μ = ν and λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) , the term at some x 0 x_0 x 0 with μ ( x 0 ) ≠ ν ( x 0 ) \mu(x_0)\ne\nu(x_0) μ ( x 0 ) = ν ( x 0 ) is strict, hence so is the sum.
Part (iii). H \mathbf{H} H is a finite sum of the continuous maps μ ↦ f ( μ ( x ) ) \mu\mapsto f(\mu(x)) μ ↦ f ( μ ( x )) , and D ( X ) \mathcal{D}(X) D ( X ) is closed and bounded in R X \mathbb{R}^{X} R X , hence compact; so H \mathbf{H} H is bounded and uniformly continuous (extreme value and Heine–Cantor theorems).
Part (iv). h ′ ( z ) = log 2 1 − z z > 0 h^{\prime}(z)=\log_{2}\frac{1-z}{z}>0 h ′ ( z ) = log 2 z 1 − z > 0 on ( 0 , 1 2 ) (0,\frac{1}{2}) ( 0 , 2 1 ) , so h h h is strictly increasing on [ 0 , 1 2 ] [0,\frac{1}{2}] [ 0 , 2 1 ] ; as h ( 0 ) = 0 h(0)=0 h ( 0 ) = 0 and h ( 1 2 ) = 1 h(\frac{1}{2})=1 h ( 2 1 ) = 1 , continuity gives the bijection.
Stochastic Games with Distributional Entropy Objectives
First, we give an illustration that describes the three semantics in Figure 2 .
Figure 2. The three semantics as pipelines from μ t \mu_t μ t to μ t + 1 \mu_{t+1} μ t + 1 . Orange marks the profile draw ( a , b ) ∼ σ t × τ t (a,b)\sim\sigma_t\times\tau_t ( a , b ) ∼ σ t × τ t , teal the transition draw ξ ∼ P a , b \xi\sim P_{a,b} ξ ∼ P a , b , black the deterministic maps. (a) Averaged: no random draw. (b) Sampled profile: one draw, then Φ a i , b i \Phi_{a_i,b_i} Φ a i , b i . (c) Sampled transition: a profile draw and then a transition draw, then Ψ ξ i j \Psi_{\xi_{ij}} Ψ ξ ij ; the numbers m m m , l l l of realizable ξ \xi ξ may differ between profiles.
Distribution Updates across the 3 semantics
Four maps describe how one round of play updates the current distribution, depending on how much randomness has already been resolved at the point the update is taken. Φ a , b \Phi_{a,b} Φ a , b ((9) ) gives the successor distribution once a pure profile ( a , b ) (a,b) ( a , b ) is fixed and the arena’s own transition probabilities are averaged in directly. Υ x , y \Upsilon_{x,y} Υ x , y ((10) ) does the same starting from a statewise-mixed profile ( x , y ) (x,y) ( x , y ) , averaging over the players’ randomised choices as well. P a , b P_{a,b} P a , b ((11) ) is instead the probability law of a fully realised transition outcome ξ \xi ξ , given a fixed pure profile ( a , b ) (a,b) ( a , b ) ; and Ψ ξ \Psi_{\xi} Ψ ξ ((12) ) gives the successor distribution once that particular realisation ξ \xi ξ has actually occurred.
Φ a , b ( μ ) ( s ′ ) : = ∑ s ∈ S μ ( s ) Δ ( s , ρ a , b ( s ) , s ′ ) . (9) \Phi_{a,b}(\mu)(s') := \sum_{s\in S}\mu(s)\,\Delta(s,\rho_{a,b}(s),s'). \tag*{(9)} Φ a , b ( μ ) ( s ′ ) := s ∈ S ∑ μ ( s ) Δ ( s , ρ a , b ( s ) , s ′ ) . (9)
Υ x , y ( μ ) ( s ′ ) : = ∑ s ∈ S μ ( s ) ∑ α ∈ A ( s ) π x , y ( s ) ( α ) Δ ( s , α , s ′ ) . (10) \Upsilon_{x,y}(\mu)(s') := \sum_{s\in S}\mu(s)\sum_{\alpha\in A(s)}\pi_{x,y}(s)(\alpha)\,\Delta(s,\alpha,s'). \tag*{(10)} Υ x , y ( μ ) ( s ′ ) := s ∈ S ∑ μ ( s ) α ∈ A ( s ) ∑ π x , y ( s ) ( α ) Δ ( s , α , s ′ ) . (10)
P a , b ( ξ ) : = ∏ s ∈ S Δ ( s , ρ a , b ( s ) , ξ ( s ) ) . (11) P_{a,b}(\xi):=\prod_{s\in S}\Delta(s,\rho_{a,b}(s),\xi(s)). \tag*{(11)} P a , b ( ξ ) := s ∈ S ∏ Δ ( s , ρ a , b ( s ) , ξ ( s )) . (11)
Ψ ξ ( μ ) ( s ′ ) : = ∑ s ∈ S ξ ( s ) = s ′ μ ( s ) . (12) \Psi_{\xi}(\mu)(s'):=\sum_{\substack{s\in S\\ \xi(s)=s'}}\mu(s). \tag*{(12)} Ψ ξ ( μ ) ( s ′ ) := s ∈ S ξ ( s ) = s ′ ∑ μ ( s ) . (12)
A history is h t = ( μ 0 , … , μ t ) ∈ D ( S ) t + 1 h_t=(\mu_0,\ldots,\mu_t)\in\mathcal{D}(S)^{t+1} h t = ( μ 0 , … , μ t ) ∈ D ( S ) t + 1 . Let Hist t \operatorname{Hist}_t Hist t denote the set of histories.
Only representative branches are drawn, with ⋮ \vdots ⋮ for the rest.
A strategy under averaged-distribution semantics (denoted av ) is a sequence of Borel-measurable maps
π min , t a v : H i s t t → X , π max , t a v : H i s t t → Y . \pi^{\mathrm{av}}_{\min,t}:\mathrm{Hist}_t\to\mathcal{X},\qquad\pi^{\mathrm{av}}_{\max,t}:\mathrm{Hist}_t\to\mathcal{Y}. π m i n , t av : Hist t → X , π m a x , t av : Hist t → Y .
Under the average distribution semantics, let x t = π min , t a v ( h t ) x_t=\pi^{\mathrm{av}}_{\min,t}(h_t) x t = π m i n , t av ( h t ) and y t = π max , t a v ( h t ) y_t=\pi^{\mathrm{av}}_{\max,t}(h_t) y t = π m a x , t av ( h t ) , where h t h_t h t is a history. The next distribution is the deterministic update
μ t + 1 = Υ x t , y t ( μ t ) . (13) \mu_{t+1}=\Upsilon_{x_t,y_t}(\mu_t). \tag*{(13)} μ t + 1 = Υ x t , y t ( μ t ) . (13)
For ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } , where sp , st respectively denote the sampled-profile and the sampled-transition semantics, a strategy under semantics ⋆ \star ⋆ is a sequence of Borel-measurable maps
π min , t ⋆ : H i s t t → D ( P min ) , π max , t ⋆ : H i s t t → D ( P max ) . \pi^\star_{\min,t}:\mathrm{Hist}_t\to\mathcal{D}(\mathcal{P}_{\min}),\qquad\pi^\star_{\max,t}:\mathrm{Hist}_t\to\mathcal{D}(\mathcal{P}_{\max}). π m i n , t ⋆ : Hist t → D ( P m i n ) , π m a x , t ⋆ : Hist t → D ( P m a x ) .
Under sampled-profile semantics, at history h t h_t h t , put σ t = π min , t s p ( h t ) \sigma_t=\pi^{\mathrm{sp}}_{\min,t}(h_t) σ t = π m i n , t sp ( h t ) and τ t = π max , t s p ( h t ) \tau_t=\pi^{\mathrm{sp}}_{\max,t}(h_t) τ t = π m a x , t sp ( h t ) . Conditional on h t h_t h t , independently sample a t ∼ σ t a_t\sim\sigma_t a t ∼ σ t and b t ∼ τ t b_t\sim\tau_t b t ∼ τ t , that is, a t ∈ P min a_t\in\mathcal{P}_{\min} a t ∈ P m i n and b t ∈ P max b_t\in\mathcal{P}_{\max} b t ∈ P m a x are random variables with Pr [ a t = a ] = σ t ( a ) \Pr[a_t=a]=\sigma_t(a) Pr [ a t = a ] = σ t ( a ) and Pr [ b t = b ] = τ t ( b ) \Pr[b_t=b]=\tau_t(b) Pr [ b t = b ] = τ t ( b ) for every a , b a,b a , b , and Pr [ a t = a , b t = b ] = σ t ( a ) τ t ( b ) \Pr[a_t=a,b_t=b]=\sigma_t(a)\tau_t(b) Pr [ a t = a , b t = b ] = σ t ( a ) τ t ( b ) , and set
μ t + 1 = Φ a t , b t ( μ t ) . (14) \mu_{t+1}=\Phi_{a_t,b_t}(\mu_t). \tag*{(14)} μ t + 1 = Φ a t , b t ( μ t ) . (14)
Under sampled-transition semantics, at history h t h_t h t , put σ t = π min , t s t ( h t ) \sigma_t=\pi^{\mathrm{st}}_{\min,t}(h_t) σ t = π m i n , t st ( h t ) and τ t = π max , t s t ( h t ) \tau_t=\pi^{\mathrm{st}}_{\max,t}(h_t) τ t = π m a x , t st ( h t ) . Conditional on h t h_t h t , first draw a t ∼ σ t a_t\sim\sigma_t a t ∼ σ t and b t ∼ τ t b_t\sim\tau_t b t ∼ τ t independently, then draw ξ t ∼ P a t , b t \xi_t\sim P_{a_t,b_t} ξ t ∼ P a t , b t conditionally on ( a t , b t ) (a_t,b_t) ( a t , b t ) , and set
μ t + 1 = Ψ ξ t ( μ t ) . (15) \mu_{t+1}=\Psi_{\xi_t}(\mu_t). \tag*{(15)} μ t + 1 = Ψ ξ t ( μ t ) . (15)
Canonical path laws
The kernels below instantiate the background from § 2 . All three semantics can be placed on the same measurable path space. Let
Ω : = D ( S ) N 0 , F : = B ( D ( S ) ) ⊗ N 0 , \Omega:= \mathcal{D}(S)^{\mathbb{N}_0}, \qquad\mathcal{F}:=\mathcal{B}(\mathcal{D}(S))^{\otimes\mathbb{N}_0}, Ω := D ( S ) N 0 , F := B ( D ( S ) ) ⊗ N 0 ,
where F \mathcal{F} F is the product σ \sigma σ -algebra on Ω \Omega Ω : the smallest σ \sigma σ -algebra on D ( S ) N 0 \mathcal{D}(S)^{\mathbb{N}_0} D ( S ) N 0 making every coordinate map ω ↦ ω t \omega\mapsto\omega_t ω ↦ ω t measurable, equivalently the σ \sigma σ -algebra generated by the cylinder sets { ω ∈ Ω : ω 0 ∈ C 0 , … , ω n ∈ C n } \{\omega\in\Omega:\omega_0\in C_0,\ldots,\omega_n\in C_n\} { ω ∈ Ω : ω 0 ∈ C 0 , … , ω n ∈ C n } for Borel sets C 0 , … , C n ⊆ D ( S ) C_0,\ldots,C_n\subseteq\mathcal{D}(S) C 0 , … , C n ⊆ D ( S ) and $n\ge 0, i.e. events that depend on only finitely many coordinates of the path. Let μ t : Ω → D ( S ) \mu_t:\Omega\to\mathcal{D}(S) μ t : Ω → D ( S ) be the coordinate map μ t ( ω ) = ω t \mu_t(\omega)=\omega_t μ t ( ω ) = ω t . Write F t \mathcal{F}_t F t for the canonical filtration generated by the coordinate process μ 0 , … , μ t \mu_0,\ldots,\mu_t μ 0 , … , μ t alone, namely the smallest σ \sigma σ -algebra with respect to which each of μ 0 , … , μ t \mu_0,\ldots,\mu_t μ 0 , … , μ t is measurable (this is the standard probabilistic notion of a σ \sigma σ -algebra, unrelated to the strategy-sampling distribution σ \sigma σ used elsewhere in the paper). This makes ( μ t ) t ≥ 0 (\mu_t)_{t\geq0} ( μ t ) t ≥ 0 adapted: at each time t t t , whether μ t \mu_t μ t has landed in any given Borel set is already decided by the information in F t \mathcal{F}_t F t , i.e. by the path so far up to and including round t t t ; nothing about later rounds is needed to know it, and nothing about it depends on information from the future.
For a Borel set C ⊆ D ( S ) C\subseteq\mathcal{D}(S) C ⊆ D ( S ) , write 1 C ( z ) = 1 \mathbf{1}_C(z)=1 1 C ( z ) = 1 if z ∈ C z\in C z ∈ C and 1 C ( z ) = 0 \mathbf{1}_C(z)=0 1 C ( z ) = 0 otherwise, and define the one-step kernels induced by a strategy pair as follows:
Q t a v ( C ∣ h t ) : = 1 C ( Υ π min , t a v ( h t ) , π max , t a v ( h t ) ( μ t ) ) , (16) Q_t^{\mathrm{av}}(C\mid h_t):=\mathbf{1}_C\left(\Upsilon_{\pi_{\min,t}^{\mathrm{av}}(h_t),\pi_{\max,t}^{\mathrm{av}}(h_t)}(\mu_t)\right), \tag*{(16)} Q t av ( C ∣ h t ) := 1 C ( Υ π m i n , t av ( h t ) , π m a x , t av ( h t ) ( μ t ) ) , (16)
Q t s p ( C ∣ h t ) : = ∑ a ∈ P min ∑ b ∈ P max σ t ( a ) τ t ( b ) 1 C ( Φ a , b ( μ t ) ) , (17) Q_t^{\mathrm{sp}}(C\mid h_t):=\sum_{a\in P_{\min}}\sum_{b\in P_{\max}}\sigma_t(a)\tau_t(b)\mathbf{1}_C\left(\Phi_{a,b}(\mu_t)\right), \tag*{(17)} Q t sp ( C ∣ h t ) := a ∈ P m i n ∑ b ∈ P m a x ∑ σ t ( a ) τ t ( b ) 1 C ( Φ a , b ( μ t ) ) , (17)
Q t s t ( C ∣ h t ) : = ∑ a ∈ P min ∑ b ∈ P max σ t ( a ) τ t ( b ) ∑ ξ ∈ Ξ P a , b ( ξ ) 1 C ( Ψ ξ ( μ t ) ) . (18) Q_t^{\mathrm{st}}(C\mid h_t):=\sum_{a\in P_{\min}}\sum_{b\in P_{\max}}\sigma_t(a)\tau_t(b)\sum_{\xi\in\Xi}P_{a,b}(\xi)\mathbf{1}_C\left(\Psi_\xi(\mu_t)\right). \tag*{(18)} Q t st ( C ∣ h t ) := a ∈ P m i n ∑ b ∈ P m a x ∑ σ t ( a ) τ t ( b ) ξ ∈ Ξ ∑ P a , b ( ξ ) 1 C ( Ψ ξ ( μ t ) ) . (18)
In words: (16) says that, since av-semantics has no randomness, the probability of C C C is either 1 or 0, according to whether the single deterministic successor Υ x t , y t ( μ t ) \Upsilon_{x_t,y_t}(\mu_t) Υ x t , y t ( μ t ) happens to lie in C C C , the Dirac mass at that one point. (17) says that the probability of C C C is obtained by adding up σ t ( a ) τ t ( b ) \sigma_t(a)\tau_t(b) σ t ( a ) τ t ( b ) , the probability of drawing the pure profile ( a , b ) (a,b) ( a , b ) , over exactly those profiles whose resulting successor Φ a , b ( μ t ) \Phi_{a,b}(\mu_t) Φ a , b ( μ t ) happens to land in C C C . (18) repeats this one layer deeper: for every profile ( a , b ) (a,b) ( a , b ) (weighted by σ t ( a ) τ t ( b ) \sigma_t(a)\tau_t(b) σ t ( a ) τ t ( b ) ) and every realized transition ξ \xi ξ (weighted, conditional on that profile, by P a , b ( ξ ) P_{a,b}(\xi) P a , b ( ξ ) ), add up the combined weight over exactly those pairs whose resulting successor Ψ ξ ( μ t ) \Psi_\xi(\mu_t) Ψ ξ ( μ t ) lands in C C C . In (17) and (18), σ t \sigma_t σ t and τ t \tau_t τ t are the distributions prescribed at h t h_t h t by the strategies for the indicated semantics. These are Borel probability kernels: the successor maps are continuous, the strategies are Borel measurable, and all displayed sums are finite. This is the precise sense in which the first semantics has a deterministic distribution trajectory and the two sampled semantics produce stochastic processes. For each semantics ⋆ ∈ { a v , s p , s t } \star\in\{\mathrm{av},\mathrm{sp},\mathrm{st}\} ⋆ ∈ { av , sp , st } , initial distribution μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) , and strategy pair ( π min , π max ) ∈ Π min ⋆ × Π max ⋆ (\pi_{\min},\pi_{\max})\in\Pi_{\min}^{\star}\times\Pi_{\max}^{\star} ( π m i n , π m a x ) ∈ Π m i n ⋆ × Π m a x ⋆ , the Ionescu-Tulcea theorem therefore gives a unique probability measure
P μ ⋆ , π min , π max on ( Ω , F ) \mathbb{P}_{\mu}^{\star,\pi_{\min},\pi_{\max}}\quad\text{on }(\Omega,\mathcal{F}) P μ ⋆ , π m i n , π m a x on ( Ω , F )
such that μ 0 = μ \mu_0=\mu μ 0 = μ almost surely and, for every $t\geq 0,
P μ ⋆ , π min , π max ( μ t + 1 ∈ C ∣ F t ) = Q t ⋆ ( C ∣ μ 0 , … , μ t ) . (19) \mathbb{P}_{\mu}^{\star,\pi_{\min},\pi_{\max}}\left(\mu_{t+1}\in C\mid\mathcal{F}_t\right)=Q_t^\star(C\mid\mu_0,\ldots,\mu_t). \tag*{(19)} P μ ⋆ , π m i n , π m a x ( μ t + 1 ∈ C ∣ F t ) = Q t ⋆ ( C ∣ μ 0 , … , μ t ) . (19)
We write E μ ⋆ , π min , π max \mathbb{E}_{\mu}^{\star,\pi_{\min},\pi_{\max}} E μ ⋆ , π m i n , π m a x for expectation with respect to this measure: for a F \mathcal{F} F -measurable function Y : Ω → R Y:\Omega\to\mathbb{R} Y : Ω → R (bounded, or nonnegative so the integral is well-defined in [ 0 , ∞ ] [0,\infty] [ 0 , ∞ ] ),
E μ ⋆ , π min , π max [ Y ] : = ∫ Ω Y ( ω ) P μ ⋆ , π min , π max ( d ω ) . \mathbb{E}_{\mu}^{\star,\pi_{\min},\pi_{\max}}[Y] := \int_{\Omega} Y(\omega)\,P_{\mu}^{\star,\pi_{\min},\pi_{\max}}(\mathrm{d}\omega). E μ ⋆ , π m i n , π m a x [ Y ] := ∫ Ω Y ( ω ) P μ ⋆ , π m i n , π m a x ( d ω ) .
Under averaged-distribution semantics every kernel is Dirac, so this path measure is concentrated on the unique trajectory generated by (13) . Under the two sampled semantics, ( μ t ) t ≥ 0 (\mu_t)_{t\geq0} ( μ t ) t ≥ 0 is generally a non-deterministic stochastic process, with laws distinguished precisely by (17) and (18) .
A running example
We illustrate the three semantics on a three-state arena. State A A A belongs to Min and has actions α , β \alpha,\beta α , β leading deterministically to B B B and to C C C respectively. State B B B belongs to Max: under α \alpha α it moves to A A A with probability 3 4 \frac{3}{4} 4 3 and stays at B B B with probability 1 4 \frac{1}{4} 4 1 , under β \beta β it moves to A A A with probability 1 4 \frac{1}{4} 4 1 and stays with probability 3 4 \frac{3}{4} 4 3 . State C C C is absorbing. Start from μ 0 = ( 0 , 1 2 , 1 2 ) \mu_0=(0,\frac{1}{2},\frac{1}{2}) μ 0 = ( 0 , 2 1 , 2 1 ) (coordinates A , B , C A,B,C A , B , C ), so H ( μ 0 ) = 1 \mathbf{H}(\mu_0)=1 H ( μ 0 ) = 1 , and let Max choose α \alpha α at B B B with probability y = 1 4 y=\frac{1}{4} y = 4 1 (the actions at A A A and C C C do not affect round 1).
Averaged distribution.* The mixed action at B B B gives the transition row 1 4 ( 3 4 , 1 4 ) + 3 4 ( 1 4 , 3 4 ) = ( 3 8 , 5 8 ) \frac{1}{4}(\frac{3}{4},\frac{1}{4})+\frac{3}{4}(\frac{1}{4},\frac{3}{4})=(\frac{3}{8},\frac{5}{8}) 4 1 ( 4 3 , 4 1 ) + 4 3 ( 4 1 , 4 3 ) = ( 8 3 , 8 5 ) on ( A , B ) (A,B) ( A , B ) , so μ 1 = ( 3 16 , 5 16 , 1 2 ) \mu_1=(\frac{3}{16},\frac{5}{16},\frac{1}{2}) μ 1 = ( 16 3 , 16 5 , 2 1 ) with H ( μ 1 ) ≈ 1.4772 \mathbf{H}(\mu_1)\approx1.4772 H ( μ 1 ) ≈ 1.4772 .
Sampled profile.* The pure action at B B B is sampled. With probability 1 4 \frac{1}{4} 4 1 it is α \alpha α and μ 1 = ( 3 8 , 1 8 , 1 2 ) \mu_1=(\frac{3}{8},\frac{1}{8},\frac{1}{2}) μ 1 = ( 8 3 , 8 1 , 2 1 ) , with probability 3 4 \frac{3}{4} 4 3 it is β \beta β and μ 1 = ( 1 8 , 3 8 , 1 2 ) \mu_1=(\frac{1}{8},\frac{3}{8},\frac{1}{2}) μ 1 = ( 8 1 , 8 3 , 2 1 ) . Both have entropy ≈ 1.4056 \approx1.4056 ≈ 1.4056 , so H ( μ 1 ) ≈ 1.4056 \mathbf{H}(\mu_1)\approx1.4056 H ( μ 1 ) ≈ 1.4056 almost surely.
Sampled transition.* The successor of B B B is sampled as well: ξ ( B ) = A \xi(B)=A ξ ( B ) = A with probability 1 4 ⋅ 3 4 + 3 4 ⋅ 1 4 = 3 8 \frac{1}{4}\cdot\frac{3}{4}+\frac{3}{4}\cdot\frac{1}{4}=\frac{3}{8} 4 1 ⋅ 4 3 + 4 3 ⋅ 4 1 = 8 3 , and ξ ( B ) = B \xi(B)=B ξ ( B ) = B otherwise. Then μ 1 = ( 1 2 , 0 , 1 2 ) \mu_1=(\frac{1}{2},0,\frac{1}{2}) μ 1 = ( 2 1 , 0 , 2 1 ) or ( 0 , 1 2 , 1 2 ) (0,\frac{1}{2},\frac{1}{2}) ( 0 , 2 1 , 2 1 ) , and H ( μ 1 ) = 1 \mathbf{H}(\mu_1)=1 H ( μ 1 ) = 1 almost surely.
In all three cases the mean of μ 1 \mu_1 μ 1 is ( 3 16 , 5 16 , 1 2 ) (\frac{3}{16},\frac{5}{16},\frac{1}{2}) ( 16 3 , 16 5 , 2 1 ) , as Lemma 17 predicts, but the expected entropies are ordered
1.4772 > 1.4056 > 1.0000 , 1.4772 > 1.4056 > 1.0000, 1.4772 > 1.4056 > 1.0000 ,
which is the one-step inequality of Corollary 22. The update Φ a , b \Phi_{a,b} Φ a , b keeps the row of B B B as fractional mass, while Ψ ξ \Psi_{\xi} Ψ ξ realises one successor and produces only a two-point distribution. That corollary compares the three updates from a common current distribution and does not by itself assert an ordering along whole trajectories.
Relating the three semantics
Every x ∈ X x\in\mathcal{X} x ∈ X and y ∈ Y y\in\mathcal{Y} y ∈ Y induce product laws on pure profiles a , b a,b a , b :
x ^ ( a ) : = ∏ s ∈ S min x ( s ) ( a ( s ) ) , y ^ ( b ) : = ∏ s ∈ S max y ( s ) ( b ( s ) ) . (20) \widehat{x}(a):=\prod_{s\in S_{\min}}x(s)(a(s)),\qquad\widehat{y}(b):=\prod_{s\in S_{\max}}y(s)(b(s)). \tag*{(20)} x ( a ) := s ∈ S m i n ∏ x ( s ) ( a ( s )) , y ( b ) := s ∈ S m a x ∏ y ( s ) ( b ( s )) . (20)
Here and below an empty product equals 1 1 1 : this is the standard convention that a product with no factors equals the multiplicative identity, exactly as an empty sum equals 0 0 0 . It is needed precisely when a player controls no states at all, i.e. S min = ∅ S_{\min}=\varnothing S m i n = ∅ or S max = ∅ S_{\max}=\varnothing S m a x = ∅ : then P min \mathcal{P}_{\min} P m i n (respectively P max \mathcal{P}_{\max} P m a x ) contains only the single, vacuous profile assigning no actions, and the product in (20) ranges over this empty index set. The convention gives x ^ ( a ) = 1 \widehat{x}(a)=1 x ( a ) = 1 for that unique profile, correctly making x ^ \widehat{x} x the point mass required for it to be a valid element of D ( P min ) \mathcal{D}(\mathcal{P}_{\min}) D ( P m i n ) .
To see that (20) defines an element of D ( P min ) \mathcal{D}(\mathcal{P}_{\min}) D ( P m i n ) in general: each factor x ( s ) ( a ( s ) ) x(s)(a(s)) x ( s ) ( a ( s )) is non-negative, so x ^ ( a ) ≥ 0 \hat{x}(a) \ge0 x ^ ( a ) ≥ 0 for every profile a a a ; and since P min = ∏ s ∈ S min A ( s ) \mathcal{P}_{\min} = \prod_{s\in S_{\min}} A(s) P m i n = ∏ s ∈ S m i n A ( s ) is exactly the Cartesian product of the per-state action sets, summing over all profiles distributes over the product (using that S min S_{\min} S m i n is finite):
∑ a ∈ P min x ^ ( a ) = ∑ a ∈ P min ∏ s ∈ S min x ( s ) ( a ( s ) ) = ∏ s ∈ S min ∑ α ∈ A ( s ) x ( s ) ( α ) = ∏ s ∈ S min 1 = 1. \sum_{a\in\mathcal{P}_{\min}} \hat{x}(a) = \sum_{a\in\mathcal{P}_{\min}} \prod_{s\in S_{\min}} x(s)(a(s)) = \prod_{s\in S_{\min}} \sum_{\alpha\in A(s)} x(s)(\alpha) = \prod_{s\in S_{\min}} 1 = 1. a ∈ P m i n ∑ x ^ ( a ) = a ∈ P m i n ∑ s ∈ S m i n ∏ x ( s ) ( a ( s )) = s ∈ S m i n ∏ α ∈ A ( s ) ∑ x ( s ) ( α ) = s ∈ S m i n ∏ 1 = 1.
So x ^ \hat{x} x ^ has non-negative entries summing to 1, i.e. is a genuine element of D ( P min ) \mathcal{D}(\mathcal{P}_{\min}) D ( P m i n ) ; the argument for y ^ \hat{y} y ^ is identical.
This construction produces only product distributions : by (20) the choice prescribed at each state is independent of the choice at every other state. A general μ ∈ D ( P min ) \mu\in\mathcal{D}(\mathcal{P}_{\min}) μ ∈ D ( P m i n ) need not factor this way. For instance, with two states s 1 , s 2 ∈ S min s_1, s_2 \in S_{\min} s 1 , s 2 ∈ S m i n each offering actions { 0 , 1 } \{0,1\} { 0 , 1 } , let μ \mu μ place mass 1 2 \frac{1}{2} 2 1 on the profile ( 0 , 0 ) (0,0) ( 0 , 0 ) and mass 1 2 \frac{1}{2} 2 1 on ( 1 , 1 ) (1,1) ( 1 , 1 ) . This μ \mu μ has the same single-state marginals as the product x ^ \hat{x} x ^ with x ( s 1 ) = x ( s 2 ) x(s_1)=x(s_2) x ( s 1 ) = x ( s 2 ) the fair coin on { 0 , 1 } \{0,1\} { 0 , 1 } , yet μ ( 0 , 1 ) = 0 \mu(0,1)=0 μ ( 0 , 1 ) = 0 while x ^ ( 0 , 1 ) = 1 4 \hat{x}(0,1)=\frac{1}{4} x ^ ( 0 , 1 ) = 4 1 : under μ \mu μ the two states are perfectly correlated, under x ^ \hat{x} x ^ they are independent. So x ↦ x ^ x \mapsto\hat{x} x ↦ x ^ picks out exactly the product-form (uncorrelated) distributions inside the strictly larger space D ( P min ) \mathcal{D}(\mathcal{P}_{\min}) D ( P m i n ) , which also contains distributions like μ \mu μ that correlate the actions at different states.
Thus x ^ ∈ D ( P min ) \hat{x} \in\mathcal{D}(\mathcal{P}_{\min}) x ^ ∈ D ( P m i n ) and y ^ ∈ D ( P max ) \hat{y} \in\mathcal{D}(\mathcal{P}_{\max}) y ^ ∈ D ( P m a x ) . These are product distributions across controlled states; by contrast, an arbitrary element of D ( P i ) \mathcal{D}(\mathcal{P}_i) D ( P i ) may correlate the actions prescribed at different states.
For the remainder of this subsection, let A and B be random variables taking values in P min \mathcal{P}_{\min} P m i n and P max \mathcal{P}_{\max} P m a x respectively, with joint law x ^ × y ^ \hat{x}\times\hat{y} x ^ × y ^ (i.e. A has law x ^ \hat{x} x ^ , B has law y ^ \hat{y} y ^ , and A, B are independent), and let ξ \xi ξ be a random variable taking values in Ξ \Xi Ξ such that, conditional on ( A , B ) = ( a , b ) (A,B)=(a,b) ( A , B ) = ( a , b ) , ξ \xi ξ has law P a , b P_{a,b} P a , b . Indeed, by the chain rule of probability,
Pr [ A = a , B = b , ξ = ξ ] = Pr [ A = a , B = b ] ⋅ Pr [ ξ = ξ ∣ A = a , B = b ] . \Pr[A=a,B=b,\xi=\xi] = \Pr[A=a,B=b]\cdot\Pr[\xi=\xi\mid A=a,B=b]. Pr [ A = a , B = b , ξ = ξ ] = Pr [ A = a , B = b ] ⋅ Pr [ ξ = ξ ∣ A = a , B = b ] .
The first factor is x ^ ( a ) y ^ ( b ) \hat{x}(a)\hat{y}(b) x ^ ( a ) y ^ ( b ) , since ( A , B ) ∼ x ^ × y ^ (A,B)\sim\hat{x}\times\hat{y} ( A , B ) ∼ x ^ × y ^ ; the second factor is P a , b ( ξ ) P_{a,b}(\xi) P a , b ( ξ ) , by definition of “ξ \xi ξ has law P a , b P_{a,b} P a , b conditional on ( A , B ) = ( a , b ) (A,B)=(a,b) ( A , B ) = ( a , b ) .” Hence the joint law is
Pr [ A = a , B = b , ξ = ξ ] = x ^ ( a ) y ^ ( b ) P a , b ( ξ ) . (21) \Pr[A=a,B=b,\xi=\xi] = \hat{x}(a)\hat{y}(b)P_{a,b}(\xi). \tag*{(21)} Pr [ A = a , B = b , ξ = ξ ] = x ^ ( a ) y ^ ( b ) P a , b ( ξ ) . (21)
All expectations below are finite sums under this law.
Lemma 17 (Mean successor under product profile sampling). For every μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) , x ∈ X x \in\mathcal{X} x ∈ X , and y ∈ Y y \in\mathcal{Y} y ∈ Y ,
Υ x , y ( μ ) = E [ Φ A , B ( μ ) ] . \Upsilon_{x,y}(\mu)=\mathbb{E}\left[\Phi_{A,B}(\mu)\right]. Υ x , y ( μ ) = E [ Φ A , B ( μ ) ] . The equality is coordinatewise in R S \mathbb{R}^{S} R S .
Proof. Under x ^ \hat{x} x ^ , the marginal law of A ( s ) A(s) A ( s ) is x ( s ) x(s) x ( s ) . Indeed, summing the product measure x ^ \hat{x} x ^ over every profile a a a agreeing with α \alpha α at s s s ,
Pr [ A ( s ) = α ] = ∑ a ∈ P min a ( s ) = α x ^ ( a ) = ∑ a ∈ P min a ( s ) = α x ( s ) ( a ( s ) ) ∏ r ∈ S min ∖ { s } x ( r ) ( a ( r ) ) . \Pr[A(s)=\alpha] = \sum_{\substack{a\in\mathcal{P}_{\min}\\a(s)=\alpha}} \hat{x}(a) = \sum_{\substack{a\in\mathcal{P}_{\min}\\a(s)=\alpha}} x(s)(a(s)) \prod_{r\in S_{\min}\setminus\{s\}} x(r)(a(r)). Pr [ A ( s ) = α ] = a ∈ P m i n a ( s ) = α ∑ x ^ ( a ) = a ∈ P m i n a ( s ) = α ∑ x ( s ) ( a ( s )) r ∈ S m i n ∖ { s } ∏ x ( r ) ( a ( r )) .
On this restricted sum a ( s ) a(s) a ( s ) is pinned to α \alpha α , so the factor x ( s ) ( a ( s ) ) x(s)(a(s)) x ( s ) ( a ( s )) is the constant x ( s ) ( α ) x(s)(\alpha) x ( s ) ( α ) and pulls out; what remains is a sum over every unconstrained choice of a ( r ) a(r) a ( r ) at the other Min-states r ≠ s r\ne s r = s , which factors coordinatewise exactly as in the proof of (11) :
Pr [ A ( s ) = α ] = x ( s ) ( α ) ∏ r ∈ S min ∖ { s } ( ∑ β ∈ A ( r ) x ( r ) ( β ) ) = x ( s ) ( α ) , (22) \Pr[A(s)=\alpha] = x(s)(\alpha)\prod_{r\in S_{\min}\setminus\{s\}}\left(\sum_{\beta\in A(r)}x(r)(\beta)\right)=x(s)(\alpha), \tag*{(22)} Pr [ A ( s ) = α ] = x ( s ) ( α ) r ∈ S m i n ∖ { s } ∏ β ∈ A ( r ) ∑ x ( r ) ( β ) = x ( s ) ( α ) , (22)
since each inner sum is 1 because x ( r ) ∈ D ( A ( r ) ) x(r) \in\mathcal{D}(A(r)) x ( r ) ∈ D ( A ( r )) . The analogous marginal identity holds for B ( s ) B(s) B ( s ) and y ( s ) y(s) y ( s ) . Crucially, ρ A , B ( s ) \rho_{A,B}(s) ρ A , B ( s ) , meaning the deterministic function ρ a , b ( s ) \rho_{a,b}(s) ρ a , b ( s ) evaluated at the random profile pair ( a , b ) = ( A , B ) (a,b)=(A,B) ( a , b ) = ( A , B ) , depends only on the single coordinate of the profile at s s s : by definition ρ a , b ( s ) = a ( s ) \rho_{a,b}(s)=a(s) ρ a , b ( s ) = a ( s ) if s ∈ S min s \in S_{\min} s ∈ S m i n and ρ a , b ( s ) = b ( s ) \rho_{a,b}(s)=b(s) ρ a , b ( s ) = b ( s ) if s ∈ S max s \in S_{\max} s ∈ S m a x , so substituting ( a , b ) = ( A , B ) (a,b)=(A,B) ( a , b ) = ( A , B ) gives ρ A , B ( s ) = A ( s ) \rho_{A,B}(s)=A(s) ρ A , B ( s ) = A ( s ) if s ∈ S min s \in S_{\min} s ∈ S m i n and ρ A , B ( s ) = B ( s ) \rho_{A,B}(s)=B(s) ρ A , B ( s ) = B ( s ) if s ∈ S max s \in S_{\max} s ∈ S m a x ; hence Δ ( s , ρ A , B ( s ) , s ′ ) \Delta(s,\rho_{A,B}(s),s') Δ ( s , ρ A , B ( s ) , s ′ ) is a function of A ( s ) A(s) A ( s ) alone (respectively B ( s ) B(s) B ( s ) alone), not of the full profile. Its expectation is therefore determined entirely by the marginal law just computed. Explicitly, if s ∈ S min s \in S_{\min} s ∈ S m i n ,
E [ Δ ( s , ρ A , B ( s ) , s ′ ) ] = E [ Δ ( s , A ( s ) , s ′ ) ] = ∑ α ∈ A ( s ) Pr [ A ( s ) = α ] Δ ( s , α , s ′ ) = ∑ α ∈ A ( s ) x ( s ) ( α ) Δ ( s , α , s ′ ) . \mathbb{E}[\Delta(s,\rho_{A,B}(s),s')] = \mathbb{E}[\Delta(s,A(s),s')] = \sum_{\alpha\in A(s)} \Pr[A(s)=\alpha]\Delta(s,\alpha,s') = \sum_{\alpha\in A(s)} x(s)(\alpha)\Delta(s,\alpha,s'). E [ Δ ( s , ρ A , B ( s ) , s ′ )] = E [ Δ ( s , A ( s ) , s ′ )] = α ∈ A ( s ) ∑ Pr [ A ( s ) = α ] Δ ( s , α , s ′ ) = α ∈ A ( s ) ∑ x ( s ) ( α ) Δ ( s , α , s ′ ) .
and symmetrically with B ( s ) B(s) B ( s ) and y ( s ) y(s) y ( s ) if s ∈ S max s \in S_{\max} s ∈ S m a x . Since π x , y ( s ) = x ( s ) \pi_{x,y}(s)=x(s) π x , y ( s ) = x ( s ) for s ∈ S min s \in S_{\min} s ∈ S m i n and π x , y ( s ) = y ( s ) \pi_{x,y}(s)=y(s) π x , y ( s ) = y ( s ) for s ∈ S max s \in S_{\max} s ∈ S m a x , both cases read uniformly as
E [ Δ ( s , ρ A , B ( s ) , s ′ ) ] = ∑ α ∈ A ( s ) π x , y ( s ) ( α ) Δ ( s , α , s ′ ) . \mathbb{E}[\Delta(s,\rho_{A,B}(s),s')] = \sum_{\alpha\in A(s)} \pi_{x,y}(s)(\alpha)\Delta(s,\alpha,s'). E [ Δ ( s , ρ A , B ( s ) , s ′ )] = α ∈ A ( s ) ∑ π x , y ( s ) ( α ) Δ ( s , α , s ′ ) .
Hence, using finiteness to exchange sums and expectations,
E [ Φ A , B ( μ ) ( s ′ ) ] = ∑ s ∈ S μ ( s ) E [ Δ ( s , ρ A , B ( s ) , s ′ ) ] = ∑ s ∈ S μ ( s ) ∑ α ∈ A ( s ) π x , y ( s ) ( α ) Δ ( s , α , s ′ ) = Υ x , y ( μ ) ( s ′ ) , \begin{aligned}
\mathbb{E}[\Phi_{A,B}(\mu)(s')] &= \sum_{s\in S}\mu(s)\mathbb{E}[\Delta(s,\rho_{A,B}(s),s')] \\
&= \sum_{s\in S}\mu(s)\sum_{\alpha\in A(s)}\pi_{x,y}(s)(\alpha)\Delta(s,\alpha,s') \\
&= \Upsilon_{x,y}(\mu)(s'),
\end{aligned} E [ Φ A , B ( μ ) ( s ′ )] = s ∈ S ∑ μ ( s ) E [ Δ ( s , ρ A , B ( s ) , s ′ )] = s ∈ S ∑ μ ( s ) α ∈ A ( s ) ∑ π x , y ( s ) ( α ) Δ ( s , α , s ′ ) = Υ x , y ( μ ) ( s ′ ) ,
where the last equality is (10). This holds for every coordinate s ′ s' s ′ .
Lemma 18 (Averaged distribution versus sampled profile). For every μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) , x ∈ X x \in\mathcal{X} x ∈ X , and y ∈ Y y \in\mathcal{Y} y ∈ Y ,
H ( Υ x , y ( μ ) ) ≥ E [ H ( Φ A , B ( μ ) ) ] . \mathbf{H}(\Upsilon_{x,y}(\mu)) \ge\mathbb{E}[\mathbf{H}(\Phi_{A,B}(\mu))]. H ( Υ x , y ( μ )) ≥ E [ H ( Φ A , B ( μ ))] . Proof. By Lemma 17, Υ x , y ( μ ) = E [ Φ A , B ( μ ) ] \Upsilon_{x,y}(\mu)=\mathbb{E}[\Phi_{A,B}(\mu)] Υ x , y ( μ ) = E [ Φ A , B ( μ )] , so the left-hand side is H \mathbf{H} H evaluated at the mean of the D ( S ) \mathcal{D}(S) D ( S ) -valued random variable Φ A , B ( μ ) \Phi_{A,B}(\mu) Φ A , B ( μ ) . Since P min \mathcal{P}_{\min} P m i n and P max \mathcal{P}_{\max} P m a x are finite and A , B A,B A , B are independent with laws x ~ , y ~ \tilde{x},\tilde{y} x ~ , y ~ , the random variable Φ A , B ( μ ) \Phi_{A,B}(\mu) Φ A , B ( μ ) takes only the finitely many values Φ a , b ( μ ) \Phi_{a,b}(\mu) Φ a , b ( μ ) , ( a , b ) ∈ P min × P max (a,b) \in\mathcal{P}_{\min} \times\mathcal{P}_{\max} ( a , b ) ∈ P m i n × P m a x , with Pr [ A = a , B = b ] = x ~ ( a ) y ~ ( b ) \Pr[A=a,B=b]=\tilde{x}(a)\tilde{y}(b) Pr [ A = a , B = b ] = x ~ ( a ) y ~ ( b ) . Hence its mean is the explicit finite convex combination
Υ x , y ( μ ) = E [ Φ A , B ( μ ) ] = ∑ ( a , b ) ∈ P min × P max x ~ ( a ) y ~ ( b ) Φ a , b ( μ ) , \Upsilon_{x,y}(\mu)=\mathbb{E}[\Phi_{A,B}(\mu)]=\sum_{(a,b)\in\mathcal{P}_{\min}\times\mathcal{P}_{\max}}\tilde{x}(a)\tilde{y}(b)\Phi_{a,b}(\mu), Υ x , y ( μ ) = E [ Φ A , B ( μ )] = ( a , b ) ∈ P m i n × P m a x ∑ x ~ ( a ) y ~ ( b ) Φ a , b ( μ ) , a weighted average, with nonnegative weights summing to 1, of the points Φ a , b ( μ ) ∈ D ( S ) \Phi_{a,b}(\mu) \in\mathcal{D}(S) Φ a , b ( μ ) ∈ D ( S ) . By Proposition 16(ii), H \mathbf{H} H is concave on D ( S ) \mathcal{D}(S) D ( S ) ; concavity for a two-point combination extends by induction to any finite convex combination, giving
H ( ∑ i p i z i ) ≥ ∑ i p i H ( z i ) \mathbf{H}\left(\sum_i p_i z_i\right) \ge\sum_i p_i\mathbf{H}(z_i) H ( i ∑ p i z i ) ≥ i ∑ p i H ( z i ) for any weights p i ≥ 0 p_i \ge0 p i ≥ 0 with ∑ i p i = 1 \sum_i p_i=1 ∑ i p i = 1 and points z i ∈ D ( S ) z_i \in\mathcal{D}(S) z i ∈ D ( S ) . Applying this with p ( a , b ) = x ~ ( a ) y ~ ( b ) p_{(a,b)}=\tilde{x}(a)\tilde{y}(b) p ( a , b ) = x ~ ( a ) y ~ ( b ) and z ( a , b ) = Φ a , b ( μ ) z_{(a,b)}=\Phi_{a,b}(\mu) z ( a , b ) = Φ a , b ( μ ) yields
H ( Υ x , y ( μ ) ) = H ( ∑ ( a , b ) x ~ ( a ) y ~ ( b ) Φ a , b ( μ ) ) ≥ ∑ ( a , b ) x ~ ( a ) y ~ ( b ) H ( Φ a , b ( μ ) ) = E [ H ( Φ A , B ( μ ) ) ] , \mathbf{H}(\Upsilon_{x,y}(\mu))=\mathbf{H}\left(\sum_{(a,b)}\tilde{x}(a)\tilde{y}(b)\Phi_{a,b}(\mu)\right)\ge\sum_{(a,b)}\tilde{x}(a)\tilde{y}(b)\mathbf{H}(\Phi_{a,b}(\mu))=\mathbb{E}[\mathbf{H}(\Phi_{A,B}(\mu))], H ( Υ x , y ( μ )) = H ( a , b ) ∑ x ~ ( a ) y ~ ( b ) Φ a , b ( μ ) ≥ ( a , b ) ∑ x ~ ( a ) y ~ ( b ) H ( Φ a , b ( μ )) = E [ H ( Φ A , B ( μ ))] , the last equality again being the definition of expectation of the finite-valued real random variable H ( Φ A , B ( μ ) ) \mathbf{H}(\Phi_{A,B}(\mu)) H ( Φ A , B ( μ )) . This is the claimed inequality. ◀
Lemma 19 (Connecting sampled profile and sampled transition). For every fixed pure-profile pair ( a , b ) (a,b) ( a , b ) ,
E ξ ∼ P a , b [ Ψ ξ ( μ ) ] = Φ a , b ( μ ) . (23) \mathbb{E}_{\xi\sim P_{a,b}}[\Psi_{\xi}(\mu)] = \Phi_{a,b}(\mu). \tag*{(23)} E ξ ∼ P a , b [ Ψ ξ ( μ )] = Φ a , b ( μ ) . (23) Proof. Indeed, by definition the expectation of the finitely-valued D ( S ) \mathcal{D}(S) D ( S ) -valued random variable Ψ ξ ( μ ) \Psi_{\xi}(\mu) Ψ ξ ( μ ) is the probability-weighted sum over its possible values,
E ξ ∼ P a , b [ Ψ ξ ( μ ) ] = ∑ ξ ∈ Ξ P a , b ( ξ ) Ψ ξ ( μ ) , \mathbb{E}_{\xi\sim P_{a,b}}[\Psi_{\xi}(\mu)] = \sum_{\xi\in\Xi} P_{a,b}(\xi)\Psi_{\xi}(\mu), E ξ ∼ P a , b [ Ψ ξ ( μ )] = ξ ∈ Ξ ∑ P a , b ( ξ ) Ψ ξ ( μ ) , a vector equation in R S \mathbb{R}^{S} R S . Taking the s ′ s' s ′ coordinate of both sides and substituting (12) ,
E ξ ∼ P a , b [ Ψ ξ ( μ ) ] ( s ′ ) = ∑ ξ ∈ Ξ P a , b ( ξ ) ∑ s : ξ ( s ) = s ′ μ ( s ) = ∑ s ∈ S μ ( s ) ∑ ξ ∈ Ξ ξ ( s ) = s ′ P a , b ( ξ ) = ∑ s ∈ S μ ( s ) Pr ξ ∼ P a , b [ ξ ( s ) = s ′ ] , \mathbb{E}_{\xi\sim P_{a,b}}[\Psi_{\xi}(\mu)](s') = \sum_{\xi\in\Xi}P_{a,b}(\xi)\sum_{s:\xi(s)=s'}\mu(s) = \sum_{s\in S}\mu(s)\sum_{\substack{\xi\in\Xi\\\xi(s)=s'}}P_{a,b}(\xi) = \sum_{s\in S}\mu(s)\Pr_{\xi\sim P_{a,b}}[\xi(s)=s'], E ξ ∼ P a , b [ Ψ ξ ( μ )] ( s ′ ) = ξ ∈ Ξ ∑ P a , b ( ξ ) s : ξ ( s ) = s ′ ∑ μ ( s ) = s ∈ S ∑ μ ( s ) ξ ∈ Ξ ξ ( s ) = s ′ ∑ P a , b ( ξ ) = s ∈ S ∑ μ ( s ) ξ ∼ P a , b Pr [ ξ ( s ) = s ′ ] , where the sums over ξ \xi ξ and s s s were exchanged (both finite), and the inner sum was recognized as Pr ξ ∼ P a , b [ ξ ( s ) = s ′ ] \Pr_{\xi\sim P_{a,b}}[\xi(s)=s'] Pr ξ ∼ P a , b [ ξ ( s ) = s ′ ] by definition of probability. To evaluate Pr ξ ∼ P a , b [ ξ ( s ) = s ′ ] \Pr_{\xi\sim P_{a,b}}[\xi(s)=s'] Pr ξ ∼ P a , b [ ξ ( s ) = s ′ ] , marginalize the product measure P a , b P_{a,b} P a , b over every coordinate but s s s :
Pr ξ ∼ P a , b [ ξ ( s ) = s ′ ] = ∑ ξ ∈ Ξ ξ ( s ) = s ′ P a , b ( ξ ) = ∑ ξ ∈ Ξ ξ ( s ) = s ′ Δ ( s , ρ a , b ( s ) , ξ ( s ) ) ∏ s ′ ′ ≠ s Δ ( s ′ ′ , ρ a , b ( s ′ ′ ) , ξ ( s ′ ′ ) ) . \Pr_{\xi\sim P_{a,b}}[\xi(s)=s'] = \sum_{\substack{\xi\in\Xi\\\xi(s)=s'}}P_{a,b}(\xi) = \sum_{\substack{\xi\in\Xi\\\xi(s)=s'}}\Delta(s,\rho_{a,b}(s),\xi(s))\prod_{s''\ne s}\Delta(s'',\rho_{a,b}(s''),\xi(s'')). ξ ∼ P a , b Pr [ ξ ( s ) = s ′ ] = ξ ∈ Ξ ξ ( s ) = s ′ ∑ P a , b ( ξ ) = ξ ∈ Ξ ξ ( s ) = s ′ ∑ Δ ( s , ρ a , b ( s ) , ξ ( s )) s ′′ = s ∏ Δ ( s ′′ , ρ a , b ( s ′′ ) , ξ ( s ′′ )) . On this restricted sum ξ ( s ) \xi(s) ξ ( s ) is pinned to s ′ s' s ′ , so the factor Δ ( s , ρ a , b ( s ) , ξ ( s ) ) \Delta(s,\rho_{a,b}(s),\xi(s)) Δ ( s , ρ a , b ( s ) , ξ ( s )) is the constant Δ ( s , ρ a , b ( s ) , s ′ ) \Delta(s,\rho_{a,b}(s),s') Δ ( s , ρ a , b ( s ) , s ′ ) and pulls out; what remains is a sum over every unconstrained choice of ξ ( s ′ ′ ) \xi(s'') ξ ( s ′′ ) at the other states s ′ ′ ≠ s s''\ne s s ′′ = s , which again factors as in the proof of (11) :
Pr ξ ∼ P a , b [ ξ ( s ) = s ′ ] = Δ ( s , ρ a , b ( s ) , s ′ ) ⋅ ∏ s ′ ≠ s ( ∑ s ′ ′ ′ ∈ S Δ ( s ′ , ρ a , b ( s ′ ) , s ′ ′ ′ ) ) = Δ ( s , ρ a , b ( s ) , s ′ ) ⋅ 1 , \Pr_{\xi\sim P_{a,b}}[\xi(s)=s'] = \Delta(s,\rho_{a,b}(s),s')\cdot\prod_{s'\ne s}\left(\sum_{s'''\in S}\Delta(s',\rho_{a,b}(s'),s''')\right) = \Delta(s,\rho_{a,b}(s),s')\cdot1, ξ ∼ P a , b Pr [ ξ ( s ) = s ′ ] = Δ ( s , ρ a , b ( s ) , s ′ ) ⋅ s ′ = s ∏ ( s ′′′ ∈ S ∑ Δ ( s ′ , ρ a , b ( s ′ ) , s ′′′ ) ) = Δ ( s , ρ a , b ( s ) , s ′ ) ⋅ 1 , since each inner sum equals 1 because Δ ( s ′ ′ , ρ a , b ( s ′ ′ ) , ⋅ ) ∈ D ( S ) \Delta(s'',\rho_{a,b}(s''),\cdot)\in\mathcal{D}(S) Δ ( s ′′ , ρ a , b ( s ′′ ) , ⋅ ) ∈ D ( S ) . Hence ξ ( s ) \xi(s) ξ ( s ) has marginal law Δ ( s , ρ a , b ( s ) , ⋅ ) \Delta(s,\rho_{a,b}(s),\cdot) Δ ( s , ρ a , b ( s ) , ⋅ ) , i.e. Pr ξ ∼ P a , b [ ξ ( s ) = s ′ ] = Δ ( s , ρ a , b ( s ) , s ′ ) \Pr_{\xi\sim P_{a,b}}[\xi(s)=s']=\Delta(s,\rho_{a,b}(s),s') Pr ξ ∼ P a , b [ ξ ( s ) = s ′ ] = Δ ( s , ρ a , b ( s ) , s ′ ) , and so the s ′ s' s ′ coordinate of the left-hand side is
∑ s ∈ S μ ( s ) Pr ξ ∼ P a , b [ ξ ( s ) = s ′ ] = ∑ s ∈ S μ ( s ) Δ ( s , ρ a , b ( s ) , s ′ ) , \sum_{s\in S}\mu(s)\Pr_{\xi\sim P_{a,b}}[\xi(s)=s'] = \sum_{s\in S}\mu(s)\Delta(s,\rho_{a,b}(s),s'), s ∈ S ∑ μ ( s ) ξ ∼ P a , b Pr [ ξ ( s ) = s ′ ] = s ∈ S ∑ μ ( s ) Δ ( s , ρ a , b ( s ) , s ′ ) , which is exactly the s ′ s' s ′ coordinate of Φ a , b ( μ ) \Phi_{a,b}(\mu) Φ a , b ( μ ) by (9) . Thus sampled-profile semantics uses the conditional mean of the sampled-transition successor, whereas sampled-transition semantics retains the individual realized successor before the nonlinear continuation value is evaluated. See also Figure 2 (c).
Lemma 20 (Sampled profile versus sampled transition). For every μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) and fixed ( a , b ) ∈ P min × P max (a,b)\in P_{\min}\times P_{\max} ( a , b ) ∈ P m i n × P m a x ,
H ( Φ a , b ( μ ) ) ≥ E ξ ∼ P a , b [ H ( Ψ ξ ( μ ) ) ] . (24) H(\Phi_{a,b}(\mu))\ge\mathbb{E}_{\xi\sim P_{a,b}}[H(\Psi_{\xi}(\mu))]. \tag*{(24)} H ( Φ a , b ( μ )) ≥ E ξ ∼ P a , b [ H ( Ψ ξ ( μ ))] . (24) Consequently,
E [ H ( Φ A , B ( μ ) ) ] ≥ E [ H ( Ψ ξ ( μ ) ) ] . \mathbb{E}[H(\Phi_{A,B}(\mu))]\ge\mathbb{E}[H(\Psi_{\xi}(\mu))]. E [ H ( Φ A , B ( μ ))] ≥ E [ H ( Ψ ξ ( μ ))] . Proof. Fix ( a , b ) (a,b) ( a , b ) . By (22), Φ a , b ( μ ) = E ξ ∼ P a , b [ Ψ ξ ( μ ) ] \Phi_{a,b}(\mu)=\mathbb{E}_{\xi\sim P_{a,b}}[\Psi_{\xi}(\mu)] Φ a , b ( μ ) = E ξ ∼ P a , b [ Ψ ξ ( μ )] , so the left-hand side of (23) is H H H evaluated at the mean of the D ( S ) \mathcal{D}(S) D ( S ) -valued random variable Ψ ξ ( μ ) \Psi_{\xi}(\mu) Ψ ξ ( μ ) , ξ ∼ P a , b \xi\sim P_{a,b} ξ ∼ P a , b . Since Ξ \Xi Ξ is finite, this mean is the explicit finite convex combination
Φ a , b ( μ ) = E ξ ∼ P a , b [ Ψ ξ ( μ ) ] = ∑ ξ ∈ Ξ P a , b ( ξ ) Ψ ξ ( μ ) , \Phi_{a,b}(\mu)=\mathbb{E}_{\xi\sim P_{a,b}}[\Psi_{\xi}(\mu)]=\sum_{\xi\in\Xi}P_{a,b}(\xi)\Psi_{\xi}(\mu), Φ a , b ( μ ) = E ξ ∼ P a , b [ Ψ ξ ( μ )] = ξ ∈ Ξ ∑ P a , b ( ξ ) Ψ ξ ( μ ) ,
a weighted average, with nonnegative weights summing to 1, of the points Ψ ξ ( μ ) ∈ D ( S ) \Psi_{\xi}(\mu) \in\mathcal{D}(S) Ψ ξ ( μ ) ∈ D ( S ) . By Proposition 16 (ii) and the same finite-concavity inequality used in Lemma 18 ,
H ( Φ a , b ( μ ) ) = H ( ∑ ξ ∈ Ξ P a , b ( ξ ) Ψ ξ ( μ ) ) ≥ ∑ ξ ∈ Ξ P a , b ( ξ ) H ( Ψ ξ ( μ ) ) = E ξ ∼ P a , b [ H ( Ψ ξ ( μ ) ) ] , (25) \mathbf{H}(\Phi_{a,b}(\mu)) = \mathbf{H}\left(\sum_{\xi\in\Xi} P_{a,b}(\xi)\Psi_{\xi}(\mu)\right) \geq\sum_{\xi\in\Xi} P_{a,b}(\xi)\mathbf{H}(\Psi_{\xi}(\mu)) = \mathbb{E}_{\xi\sim P_{a,b}}[\mathbf{H}(\Psi_{\xi}(\mu))], \tag*{(25)} H ( Φ a , b ( μ )) = H ξ ∈ Ξ ∑ P a , b ( ξ ) Ψ ξ ( μ ) ≥ ξ ∈ Ξ ∑ P a , b ( ξ ) H ( Ψ ξ ( μ )) = E ξ ∼ P a , b [ H ( Ψ ξ ( μ ))] , (25)
which is (23). For the consequence, multiply (23) by x ^ ( a ) y ^ ( b ) ≥ 0 \hat{x}(a)\hat{y}(b) \geq0 x ^ ( a ) y ^ ( b ) ≥ 0 and sum over every ( a , b ) ∈ P min × P max (a,b) \in\mathcal{P}_{\min} \times\mathcal{P}_{\max} ( a , b ) ∈ P m i n × P m a x :
∑ ( a , b ) x ^ ( a ) y ^ ( b ) H ( Φ a , b ( μ ) ) ≥ ∑ ( a , b ) x ^ ( a ) y ^ ( b ) ∑ ξ ∈ Ξ P a , b ( ξ ) H ( Ψ ξ ( μ ) ) . \sum_{(a,b)} \hat{x}(a)\hat{y}(b)\mathbf{H}(\Phi_{a,b}(\mu)) \geq\sum_{(a,b)} \hat{x}(a)\hat{y}(b)\sum_{\xi\in\Xi}P_{a,b}(\xi)\mathbf{H}(\Psi_{\xi}(\mu)). ( a , b ) ∑ x ^ ( a ) y ^ ( b ) H ( Φ a , b ( μ )) ≥ ( a , b ) ∑ x ^ ( a ) y ^ ( b ) ξ ∈ Ξ ∑ P a , b ( ξ ) H ( Ψ ξ ( μ )) .
The left-hand side is exactly E [ H ( Φ A , B ( μ ) ) ] \mathbb{E}[\mathbf{H}(\Phi_{A,B}(\mu))] E [ H ( Φ A , B ( μ ))] , by the same definition of expectation of a finite-valued real random variable used in Lemma 18 . For the right-hand side, distribute the outer sum and swap the (finite) order of summation over ( a , b ) (a,b) ( a , b ) and ξ \xi ξ :
∑ ( a , b ) ∑ ξ ∈ Ξ [ x ^ ( a ) y ^ ( b ) P a , b ( ξ ) ] H ( Ψ ξ ( μ ) ) = ∑ ξ ∈ Ξ ( ∑ ( a , b ) x ^ ( a ) y ^ ( b ) P a , b ( ξ ) ) H ( Ψ ξ ( μ ) ) . \sum_{(a,b)}\sum_{\xi\in\Xi}[\hat{x}(a)\hat{y}(b)P_{a,b}(\xi)]\mathbf{H}(\Psi_{\xi}(\mu)) = \sum_{\xi\in\Xi}\left(\sum_{(a,b)}\hat{x}(a)\hat{y}(b)P_{a,b}(\xi)\right)\mathbf{H}(\Psi_{\xi}(\mu)). ( a , b ) ∑ ξ ∈ Ξ ∑ [ x ^ ( a ) y ^ ( b ) P a , b ( ξ )] H ( Ψ ξ ( μ )) = ξ ∈ Ξ ∑ ( a , b ) ∑ x ^ ( a ) y ^ ( b ) P a , b ( ξ ) H ( Ψ ξ ( μ )) .
By the joint law (21) , Pr [ A = a , B = b , ξ = ξ ] = x ^ ( a ) y ^ ( b ) P a , b ( ξ ) \Pr[A=a,B=b,\xi=\xi]=\hat{x}(a)\hat{y}(b)P_{a,b}(\xi) Pr [ A = a , B = b , ξ = ξ ] = x ^ ( a ) y ^ ( b ) P a , b ( ξ ) (first the profile is drawn, then the transition is drawn conditionally on it), so summing this joint probability over all ( a , b ) (a,b) ( a , b ) for fixed ξ \xi ξ gives exactly the marginal law of the fully randomized transition ξ \xi ξ :
∑ ( a , b ) x ^ ( a ) y ^ ( b ) P a , b ( ξ ) = Pr [ ξ = ξ ] . \sum_{(a,b)}\hat{x}(a)\hat{y}(b)P_{a,b}(\xi)=\Pr[\xi=\xi]. ( a , b ) ∑ x ^ ( a ) y ^ ( b ) P a , b ( ξ ) = Pr [ ξ = ξ ] .
Substituting, the right-hand side becomes
∑ ξ ∈ Ξ Pr [ ξ = ξ ] H ( Ψ ξ ( μ ) ) = E [ H ( Ψ ξ ( μ ) ) ] , \sum_{\xi\in\Xi}\Pr[\xi=\xi]\mathbf{H}(\Psi_{\xi}(\mu))=\mathbb{E}[\mathbf{H}(\Psi_{\xi}(\mu))], ξ ∈ Ξ ∑ Pr [ ξ = ξ ] H ( Ψ ξ ( μ )) = E [ H ( Ψ ξ ( μ ))] ,
again by definition of expectation. Combining the two displays gives
E [ H ( Φ A , B ( μ ) ) ] ≥ E [ H ( Ψ ξ ( μ ) ) ] . \mathbb{E}[\mathbf{H}(\Phi_{A,B}(\mu))]\geq\mathbb{E}[\mathbf{H}(\Psi_{\xi}(\mu))]. E [ H ( Φ A , B ( μ ))] ≥ E [ H ( Ψ ξ ( μ ))] .
as claimed. Equivalently, this whole computation is an instance of the tower property (law of total expectation), applied to the real-valued random variable Z : = H ( Ψ ξ ( μ ) ) Z:=\mathbf{H}(\Psi_{\xi}(\mu)) Z := H ( Ψ ξ ( μ )) :
E [ Z ] = E [ E [ Z ∣ A , B ] ] . \mathbb{E}[Z]=\mathbb{E}[\mathbb{E}[Z\mid A,B]]. E [ Z ] = E [ E [ Z ∣ A , B ]] .
Here the inner conditional expectation, evaluated at ( A , B ) = ( a , b ) (A,B)=(a,b) ( A , B ) = ( a , b ) , is exactly E ξ ∼ P a , b [ H ( Ψ ξ ( μ ) ) ] \mathbb{E}_{\xi\sim P_{a,b}}[\mathbf{H}(\Psi_{\xi}(\mu))] E ξ ∼ P a , b [ H ( Ψ ξ ( μ ))] , since ξ \xi ξ has law P a , b P_{a,b} P a , b conditional on that event; averaging this conditional expectation against the law x ^ × y ^ \hat{x}\times\hat{y} x ^ × y ^ of ( A , B ) (A,B) ( A , B ) recovers the unconditional expectation E [ Z ] = E [ H ( Ψ ξ ( μ ) ) ] \mathbb{E}[Z]=\mathbb{E}[\mathbf{H}(\Psi_{\xi}(\mu))] E [ Z ] = E [ H ( Ψ ξ ( μ ))] , which is exactly the computation carried out above by multiplying by x ^ ( a ) y ^ ( b ) \hat{x}(a)\hat{y}(b) x ^ ( a ) y ^ ( b ) , summing, and recognizing the result as the marginal law of ξ \xi ξ .
Lemma 21 (Averaged distribution as the mean realized successor). For every μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) , x ∈ X x \in\mathcal{X} x ∈ X , and y ∈ Y y \in\mathcal{Y} y ∈ Y ,
Υ x , y ( μ ) = E [ Ψ ξ ( μ ) ] . (26) \Upsilon_{x,y}(\mu)=\mathbb{E}[\Psi_{\xi}(\mu)]. \tag*{(26)} Υ x , y ( μ ) = E [ Ψ ξ ( μ )] . (26) Consequently,
H ( Υ x , y ( μ ) ) ≥ E [ H ( Ψ ξ ( μ ) ) ] . \mathbf{H}(\Upsilon_{x,y}(\mu))\geq\mathbb{E}[\mathbf{H}(\Psi_{\xi}(\mu))]. H ( Υ x , y ( μ )) ≥ E [ H ( Ψ ξ ( μ ))] .
Proof. We first establish (26) by applying the tower property to the D ( S ) \mathcal{D}(S) D ( S ) -valued random variable Ψ ξ ( μ ) \Psi_{\xi}(\mu) Ψ ξ ( μ ) , coordinatewise: for each s ′ ∈ S s' \in S s ′ ∈ S , Ψ ξ ( μ ) ( s ′ ) \Psi_{\xi}(\mu)(s') Ψ ξ ( μ ) ( s ′ ) is a real-valued random variable, so the ordinary (scalar) tower property gives
E [ Ψ ξ ( μ ) ( s ′ ) ] = E [ E [ Ψ ξ ( μ ) ( s ′ ) ∣ A , B ] ] . \mathbb{E}\left[\Psi_{\xi}(\mu)(s')\right] = \mathbb{E}\left[\mathbb{E}\left[\Psi_{\xi}(\mu)(s') \mid A,B\right]\right]. E [ Ψ ξ ( μ ) ( s ′ ) ] = E [ E [ Ψ ξ ( μ ) ( s ′ ) ∣ A , B ] ] .
and since this holds for every coordinate s ′ s' s ′ , the corresponding vector identity holds:
E [ Ψ ξ ( μ ) ] = E [ E [ Ψ ξ ( μ ) ∣ A , B ] ] . \mathbb{E}\left[\Psi_{\xi}(\mu)\right] = \mathbb{E}\left[\mathbb{E}\left[\Psi_{\xi}(\mu) \mid A,B\right]\right]. E [ Ψ ξ ( μ ) ] = E [ E [ Ψ ξ ( μ ) ∣ A , B ] ] .
Evaluate the inner conditional expectation at ( A , B ) = ( a , b ) (A,B)=(a,b) ( A , B ) = ( a , b ) : since ξ \xi ξ has law P a , b P_{a,b} P a , b conditional on this event,
E [ Ψ ξ ( μ ) ∣ A = a , B = b ] = E ξ ∼ P a , b [ Ψ ξ ( μ ) ] = Φ a , b ( μ ) , \mathbb{E}\left[\Psi_{\xi}(\mu) \mid A=a,B=b\right] = \mathbb{E}_{\xi\sim P_{a,b}}\left[\Psi_{\xi}(\mu)\right] = \Phi_{a,b}(\mu), E [ Ψ ξ ( μ ) ∣ A = a , B = b ] = E ξ ∼ P a , b [ Ψ ξ ( μ ) ] = Φ a , b ( μ ) ,
the last equality being (22). Thus the inner conditional expectation, as a function of ( A , B ) (A,B) ( A , B ) , is exactly the random variable Φ A , B ( μ ) \Phi_{A,B}(\mu) Φ A , B ( μ ) , and so
E [ Ψ ξ ( μ ) ] = E [ Φ A , B ( μ ) ] = Υ x , y ( μ ) , \mathbb{E}\left[\Psi_{\xi}(\mu)\right] = \mathbb{E}\left[\Phi_{A,B}(\mu)\right] = \Upsilon_{x,y}(\mu), E [ Ψ ξ ( μ ) ] = E [ Φ A , B ( μ ) ] = Υ x , y ( μ ) ,
the last equality being Lemma 17 . This proves (26) . For the entropy inequality, note Ξ \Xi Ξ is finite, so Ψ ξ ( μ ) \Psi_{\xi}(\mu) Ψ ξ ( μ ) takes only finitely many values, and its mean is the explicit finite convex combination
E [ Ψ ξ ( μ ) ] = ∑ ξ ∈ Ξ Pr [ ξ = ξ ] Ψ ξ ( μ ) , \mathbb{E}\left[\Psi_{\xi}(\mu)\right] = \sum_{\xi\in\Xi}\Pr[\xi=\xi]\Psi_{\xi}(\mu), E [ Ψ ξ ( μ ) ] = ξ ∈ Ξ ∑ Pr [ ξ = ξ ] Ψ ξ ( μ ) ,
a weighted average, with nonnegative weights summing to 1, of the points Ψ ξ ( μ ) ∈ D ( S ) \Psi_{\xi}(\mu)\in\mathcal{D}(S) Ψ ξ ( μ ) ∈ D ( S ) . By Proposition 16 (ii) and the same finite-concavity inequality used in Lemmas 18 and 20 ,
H ( E [ Ψ ξ ( μ ) ] ) = H ( ∑ ξ ∈ Ξ Pr [ ξ = ξ ] Ψ ξ ( μ ) ) ≥ ∑ ξ ∈ Ξ Pr [ ξ = ξ ] H ( Ψ ξ ( μ ) ) = E [ H ( Ψ ξ ( μ ) ) ] . H\left(\mathbb{E}\left[\Psi_{\xi}(\mu)\right]\right) = H\left(\sum_{\xi\in\Xi}\Pr[\xi=\xi]\Psi_{\xi}(\mu)\right) \ge\sum_{\xi\in\Xi}\Pr[\xi=\xi]H\left(\Psi_{\xi}(\mu)\right) = \mathbb{E}\left[H\left(\Psi_{\xi}(\mu)\right)\right]. H ( E [ Ψ ξ ( μ ) ] ) = H ξ ∈ Ξ ∑ Pr [ ξ = ξ ] Ψ ξ ( μ ) ≥ ξ ∈ Ξ ∑ Pr [ ξ = ξ ] H ( Ψ ξ ( μ ) ) = E [ H ( Ψ ξ ( μ ) ) ] .
Combining this with (26) gives
H ( Υ x , y ( μ ) ) = H ( E [ Ψ ξ ( μ ) ] ) ≥ E [ H ( Ψ ξ ( μ ) ) ] , H\left(\Upsilon_{x,y}(\mu)\right) = H\left(\mathbb{E}\left[\Psi_{\xi}(\mu)\right]\right) \ge\mathbb{E}\left[H\left(\Psi_{\xi}(\mu)\right)\right], H ( Υ x , y ( μ ) ) = H ( E [ Ψ ξ ( μ ) ] ) ≥ E [ H ( Ψ ξ ( μ ) ) ] ,
as claimed.
Corollary 22 (One-step entropy ordering). For every μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) , x ∈ X x\in\mathcal{X} x ∈ X , and y ∈ Y y\in\mathcal{Y} y ∈ Y , under the joint law (21) ,
H ( Υ x , y ( μ ) ) ≥ E [ H ( Φ A , B ( μ ) ) ] ≥ E [ H ( Ψ ξ ( μ ) ) ] . H\left(\Upsilon_{x,y}(\mu)\right) \ge\mathbb{E}\left[H\left(\Phi_{A,B}(\mu)\right)\right] \ge\mathbb{E}\left[H\left(\Psi_{\xi}(\mu)\right)\right]. H ( Υ x , y ( μ ) ) ≥ E [ H ( Φ A , B ( μ ) ) ] ≥ E [ H ( Ψ ξ ( μ ) ) ] .
Proof. Combine Lemmas 18 and 20 . The direct comparison of the first and third terms is also Lemma 21 .
Proposition 23 (Compactness and continuity of the distribution dynamics). The simplex D ( S ) \mathcal{D}(S) D ( S ) is compact and convex. Moreover:
(i) for every statewise mixed-action pair ( x , y ) (x,y) ( x , y ) , μ ↦ Υ x , y ( μ ) \mu\mapsto\Upsilon_{x,y}(\mu) μ ↦ Υ x , y ( μ ) is affine and continuous, and ( μ , x , y ) ↦ Υ x , y ( μ ) (\mu,x,y)\mapsto\Upsilon_{x,y}(\mu) ( μ , x , y ) ↦ Υ x , y ( μ ) is jointly continuous;
(ii) for every pure-profile pair ( a , b ) (a,b) ( a , b ) , Φ a , b : D ( S ) → D ( S ) \Phi_{a,b}:\mathcal{D}(S)\to\mathcal{D}(S) Φ a , b : D ( S ) → D ( S ) is affine and continuous;
(iii) for every transition realization ξ \xi ξ , Ψ ξ : D ( S ) → D ( S ) \Psi_{\xi}:\mathcal{D}(S)\to\mathcal{D}(S) Ψ ξ : D ( S ) → D ( S ) is affine and continuous.
Proof. D ( S ) = { μ ∈ R S : μ ( s ) ≥ 0 for all s ∈ S , ∑ s ∈ S μ ( s ) = 1 } \mathcal{D}(S)=\{\mu\in\mathbb{R}^{S}:\mu(s)\geq0\text{ for all }s\in S,\ \sum_{s\in S}\mu(s)=1\} D ( S ) = { μ ∈ R S : μ ( s ) ≥ 0 for all s ∈ S , ∑ s ∈ S μ ( s ) = 1 } is the intersection of finitely many closed half-spaces (one for each nonnegativity constraint μ ( s ) ≥ 0 \mu(s)\geq0 μ ( s ) ≥ 0 ) with a single hyperplane (the constraint ∑ s μ ( s ) = 1 \sum_s\mu(s)=1 ∑ s μ ( s ) = 1 ); an intersection of convex sets is convex, so D ( S ) \mathcal{D}(S) D ( S ) is convex. It is also closed (an intersection of closed sets) and bounded (each coordinate lies in [ 0 , 1 ] [0,1] [ 0 , 1 ] ), hence compact in the finite-dimensional space R S \mathbb{R}^{S} R S by Heine–Borel.
(i) Fix ( x , y ) (x,y) ( x , y ) . By (10) , the s ′ s' s ′ coordinate of Υ x , y ( μ ) \Upsilon_{x,y}(\mu) Υ x , y ( μ ) is
Υ x , y ( μ ) ( s ′ ) = ∑ s ∈ S μ ( s ) ∑ α ∈ A ( s ) π x , y ( s ) ( α ) Δ ( s , α , s ′ ) . \Upsilon_{x,y}(\mu)(s')=\sum_{s\in S}\mu(s)\sum_{\alpha\in A(s)}\pi_{x,y}(s)(\alpha)\Delta(s,\alpha,s'). Υ x , y ( μ ) ( s ′ ) = s ∈ S ∑ μ ( s ) α ∈ A ( s ) ∑ π x , y ( s ) ( α ) Δ ( s , α , s ′ ) .
For fixed ( x , y ) (x,y) ( x , y ) , each coefficient c s ( x , y , s ′ ) c_s(x,y,s') c s ( x , y , s ′ ) does not depend on μ \mu μ , so Υ x , y ( μ ) ( s ′ ) = ∑ s c s ( x , y , s ′ ) μ ( s ) \Upsilon_{x,y}(\mu)(s')=\sum_s c_s(x,y,s')\mu(s) Υ x , y ( μ ) ( s ′ ) = ∑ s c s ( x , y , s ′ ) μ ( s ) is a linear (hence affine) function of μ \mu μ ; this holds for every coordinate s ′ s' s ′ , so μ ↦ Υ x , y ( μ ) \mu\mapsto\Upsilon_{x,y}(\mu) μ ↦ Υ x , y ( μ ) is linear, and every linear map between finite-dimensional real vector spaces is continuous. For joint continuity in ( μ , x , y ) (\mu,x,y) ( μ , x , y ) : expanding c s ( x , y , s ′ ) = ∑ α π x , y ( s ) ( α ) Δ ( s , α , s ′ ) c_s(x,y,s')=\sum_\alpha\pi_{x,y}(s)(\alpha)\Delta(s,\alpha,s') c s ( x , y , s ′ ) = ∑ α π x , y ( s ) ( α ) Δ ( s , α , s ′ ) , each term μ ( s ) π x , y ( s ) ( α ) Δ ( s , α , s ′ ) \mu(s)\pi_{x,y}(s)(\alpha)\Delta(s,\alpha,s') μ ( s ) π x , y ( s ) ( α ) Δ ( s , α , s ′ ) is a product of the coordinate μ ( s ) \mu(s) μ ( s ) , the coordinate π x , y ( s ) ( α ) \pi_{x,y}(s)(\alpha) π x , y ( s ) ( α ) (equal to x ( s ) ( α ) x(s)(\alpha) x ( s ) ( α ) or y ( s ) ( α ) y(s)(\alpha) y ( s ) ( α ) , a coordinate of x x x or y y y ), and the constant Δ ( s , α , s ′ ) \Delta(s,\alpha,s') Δ ( s , α , s ′ ) ; a finite sum of such products is a polynomial (degree 2) in the finitely many real coordinates of ( μ , x , y ) (\mu,x,y) ( μ , x , y ) , and every polynomial in finitely many real variables is continuous, since addition and multiplication are continuous on R \mathbb{R} R . Hence each coordinate of ( μ , x , y ) ↦ Υ x , y ( μ ) (\mu,x,y)\mapsto\Upsilon_{x,y}(\mu) ( μ , x , y ) ↦ Υ x , y ( μ ) is jointly continuous, and therefore so is the map into D ( S ) ⊂ R S \mathcal{D}(S)\subset\mathbb{R}^{S} D ( S ) ⊂ R S .
(ii) Fix ( a , b ) (a,b) ( a , b ) . By (9) , the s ′ s' s ′ coordinate of Φ a , b ( μ ) \Phi_{a,b}(\mu) Φ a , b ( μ ) is
Φ a , b ( μ ) ( s ′ ) = ∑ s ∈ S μ ( s ) Δ ( s , ρ a , b ( s ) , s ′ ) . \Phi_{a,b}(\mu)(s')=\sum_{s\in S}\mu(s)\Delta(s,\rho_{a,b}(s),s'). Φ a , b ( μ ) ( s ′ ) = s ∈ S ∑ μ ( s ) Δ ( s , ρ a , b ( s ) , s ′ ) .
where the coefficient Δ ( s , ρ a , b ( s ) , s ′ ) \Delta(s,\rho_{a,b}(s),s') Δ ( s , ρ a , b ( s ) , s ′ ) is a fixed constant (it depends only on the fixed ( a , b ) (a,b) ( a , b ) and the arena, not on μ \mu μ ). So, as in (i), each coordinate is a linear function of μ \mu μ , hence Φ a , b \Phi_{a,b} Φ a , b is linear (affine) and, being linear on a finite-dimensional space, continuous.
(iii) Fix ξ \xi ξ . By (12) , the s ′ s' s ′ coordinate of Ψ ξ ( μ ) \Psi_{\xi}(\mu) Ψ ξ ( μ ) is
Ψ ξ ( μ ) ( s ′ ) = ∑ s : ξ ( s ) = s ′ μ ( s ) = ∑ s ∈ S 1 { ξ ( s ) = s ′ } μ ( s ) , \Psi_{\xi}(\mu)(s')=\sum_{s:\xi(s)=s'}\mu(s)=\sum_{s\in S}\mathbf{1}_{\{\xi(s)=s'\}}\mu(s), Ψ ξ ( μ ) ( s ′ ) = s : ξ ( s ) = s ′ ∑ μ ( s ) = s ∈ S ∑ 1 { ξ ( s ) = s ′ } μ ( s ) ,
using the indicator notation of (16) , here applied to the condition ξ ( s ) = s ′ \xi(s)=s' ξ ( s ) = s ′ rather than to a named set: 1 { ξ ( s ) = s ′ } = 1 \mathbf{1}_{\{\xi(s)=s'\}}=1 1 { ξ ( s ) = s ′ } = 1 if ξ ( s ) = s ′ \xi(s)=s' ξ ( s ) = s ′ and 0 0 0 otherwise. Each coefficient 1 { ξ ( s ) = s ′ } ∈ { 0 , 1 } \mathbf{1}_{\{\xi(s)=s'\}}\in\{0,1\} 1 { ξ ( s ) = s ′ } ∈ { 0 , 1 } is fixed once ξ \xi ξ is fixed and does not depend on μ \mu μ . Again each coordinate is a linear function of μ \mu μ , so Ψ ξ \Psi_{\xi} Ψ ξ is linear (affine) and continuous by the same finite-dimensional argument. ◀
Bellman characterisations
Theorem 1 (Bellman characterisations). For f ∈ C ( D ( S ) ) f\in C(\mathcal{D}(S)) f ∈ C ( D ( S )) , define the averaged Bellman operators by
( T a v + f ) ( μ ) : = ( 1 − γ ) H ( μ ) + γ min x ∈ X max y ∈ Y f ( Υ x , y ( μ ) ) , (27) (\mathcal{T}_{\mathrm{av}}^{+}f)(\mu):=(1-\gamma)\mathrm{H}(\mu)+\gamma\min_{x\in X}\max_{y\in Y}f(\Upsilon_{x,y}(\mu)), \tag*{(27)} ( T av + f ) ( μ ) := ( 1 − γ ) H ( μ ) + γ x ∈ X min y ∈ Y max f ( Υ x , y ( μ )) , (27) ( T a v f ) ( μ ) : = ( 1 − γ ) H ( μ ) + γ max y ∈ Y min x ∈ X f ( Υ x , y ( μ ) ) . (28) (\mathcal{T}_{\mathrm{av}}f)(\mu):=(1-\gamma)\mathrm{H}(\mu)+\gamma\max_{y\in Y}\min_{x\in X}f(\Upsilon_{x,y}(\mu)). \tag*{(28)} ( T av f ) ( μ ) := ( 1 − γ ) H ( μ ) + γ y ∈ Y max x ∈ X min f ( Υ x , y ( μ )) . (28) For ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } , define
( T ⋆ f ) ( μ ) : = ( 1 − γ ) H ( μ ) + γ val ( M f ⋆ ( μ ) ) , (29) (\mathcal{T}_{\star}f)(\mu):=(1-\gamma)\mathrm{H}(\mu)+\gamma\operatorname{val}(M_f^{\star}(\mu)), \tag*{(29)} ( T ⋆ f ) ( μ ) := ( 1 − γ ) H ( μ ) + γ val ( M f ⋆ ( μ )) , (29)
where M f ⋆ ( μ ) M_f^\star(\mu) M f ⋆ ( μ ) is the finite matrix indexed by a ∈ P min a \in\mathcal{P}_{\min} a ∈ P m i n and b ∈ P max b \in\mathcal{P}_{\max} b ∈ P m a x , with entries
M f s p ( μ ) a , b : = f ( Φ a , b ( μ ) ) , (30) M_f^{\mathrm{sp}}(\mu)_{a,b} := f(\Phi_{a,b}(\mu)), \tag*{(30)} M f sp ( μ ) a , b := f ( Φ a , b ( μ )) , (30)
M f s t ( μ ) a , b : = ∑ ξ ∈ Ξ P a , b ( ξ ) f ( Ψ ξ ( μ ) ) . (31) M_f^{\mathrm{st}}(\mu)_{a,b} := \sum_{\xi\in\Xi} P_{a,b}(\xi) f(\Psi_\xi(\mu)). \tag*{(31)} M f st ( μ ) a , b := ξ ∈ Ξ ∑ P a , b ( ξ ) f ( Ψ ξ ( μ )) . (31)
Here val \operatorname{val} val is the value of the zero-sum matrix game in which Min chooses rows and Max chooses columns. Each of these operators maps C ( D ( S ) ) C(\mathcal{D}(S)) C ( D ( S )) into itself and is a γ \gamma γ -contraction:
∥ T f − T g ∥ ∞ ≤ γ ∥ f − g ∥ ∞ . \|Tf-Tg\|_\infty\leq\gamma\|f-g\|_\infty. ∥ T f − T g ∥ ∞ ≤ γ ∥ f − g ∥ ∞ .
The unique fixed points of T a v + T_{\mathrm{av}}^+ T av + and T a v − T_{\mathrm{av}}^- T av − are respectively V a v + V_{\mathrm{av}}^+ V av + and V a v − V_{\mathrm{av}}^- V av − . For ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } , the unique fixed point of T ⋆ T_\star T ⋆ equals both V ⋆ − V_\star^- V ⋆ − and V ⋆ + V_\star^+ V ⋆ + ; we write V ⋆ V_\star V ⋆ for this common value.
For each operator T T T and corresponding value V V V , let W 0 ≡ 0 W_0 \equiv0 W 0 ≡ 0 and W t + 1 = T W t W_{t+1}=TW_t W t + 1 = T W t . Then W t W_t W t is the corresponding t t t -stage value, with rewards at times 0 , … , t − 1 0,\ldots,t-1 0 , … , t − 1 , and
0 ≤ V ( μ ) − W t ( μ ) ≤ γ t log 2 ∣ S ∣ for every μ ∈ D ( S ) . (32) 0 \leq V(\mu)-W_t(\mu) \leq\gamma^t\log_2 |S| \qquad\text{for every } \mu\in\mathcal{D}(S). \tag*{(32)} 0 ≤ V ( μ ) − W t ( μ ) ≤ γ t log 2 ∣ S ∣ for every μ ∈ D ( S ) . (32)
Proof. Write K : = D ( S ) K := \mathcal{D}(S) K := D ( S ) and L : = log 2 ∣ S ∣ L := \log_2 |S| L := log 2 ∣ S ∣ . Since K K K is compact, C ( K ) C(K) C ( K ) equipped with the supremum norm is a Banach space. We establish continuity, contraction, the finite-horizon recursions, and finally the infinite-horizon identification.
Continuity and attainment. Fix f ∈ C ( K ) f \in C(K) f ∈ C ( K ) . Since D ( S ) \mathcal{D}(S) D ( S ) , X \mathcal{X} X , Y \mathcal{Y} Y are each compact (Proposition 23 for D ( S ) \mathcal{D}(S) D ( S ) ; X \mathcal{X} X , Y \mathcal{Y} Y are finite products of simplices) and ( μ , x , y ) ↦ f ( Υ x , y ( μ ) ) (\mu,x,y) \mapsto f(\Upsilon_{x,y}(\mu)) ( μ , x , y ) ↦ f ( Υ x , y ( μ )) is continuous (Proposition 23 again, composed with f f f ), this map is uniformly continuous on the compact set K × X × Y K \times\mathcal{X} \times\mathcal{Y} K × X × Y .
Fix η > 0 \eta> 0 η > 0 . Uniform continuity gives ρ > 0 \rho> 0 ρ > 0 such that d ( μ , μ ′ ) < ρ d(\mu,\mu') < \rho d ( μ , μ ′ ) < ρ implies ∣ f ( Υ x , y ( μ ) ) − f ( Υ x , y ( μ ′ ) ) ∣ < η |f(\Upsilon_{x,y}(\mu))-f(\Upsilon_{x,y}(\mu'))| < \eta ∣ f ( Υ x , y ( μ )) − f ( Υ x , y ( μ ′ )) ∣ < η for every x ∈ X , y ∈ Y x \in\mathcal{X}, y \in\mathcal{Y} x ∈ X , y ∈ Y simultaneously. For such μ , μ ′ \mu,\mu' μ , μ ′ , writing g ( x , y ) : = f ( Υ x , y ( μ ) ) g(x,y) := f(\Upsilon_{x,y}(\mu)) g ( x , y ) := f ( Υ x , y ( μ )) and g ′ ( x , y ) : = f ( Υ x , y ( μ ′ ) ) g'(x,y) := f(\Upsilon_{x,y}(\mu')) g ′ ( x , y ) := f ( Υ x , y ( μ ′ )) , g ( x , y ) ≤ g ′ ( x , y ) + η g(x,y) \leq g'(x,y) + \eta g ( x , y ) ≤ g ′ ( x , y ) + η for every x , y x,y x , y ; taking max y \max_y max y of both sides and then min x \min_x min x gives min x max y g ( x , y ) ≤ min x max y g ′ ( x , y ) + η \min_x \max_y g(x,y) \leq\min_x \max_y g'(x,y) + \eta min x max y g ( x , y ) ≤ min x max y g ′ ( x , y ) + η , and the symmetric argument with g , g ′ g,g' g , g ′ exchanged gives the reverse inequality, so
∣ min x ∈ X max y ∈ Y f ( Υ x , y ( μ ) ) − min x ∈ X max y ∈ Y f ( Υ x , y ( μ ′ ) ) ∣ ≤ η , \left|\min_{x \in\mathcal{X}}\max_{y \in\mathcal{Y}} f(\Upsilon_{x,y}(\mu))-\min_{x \in\mathcal{X}}\max_{y \in\mathcal{Y}} f(\Upsilon_{x,y}(\mu'))\right| \leq\eta, x ∈ X min y ∈ Y max f ( Υ x , y ( μ )) − x ∈ X min y ∈ Y max f ( Υ x , y ( μ ′ )) ≤ η ,
and the identical argument with min and max exchanged bounds the other ordered extremum by η \eta η as well. Since η \eta η was arbitrary, both μ ↦ min x max y f ( Υ x , y ( μ ) ) \mu\mapsto\min_x \max_y f(\Upsilon_{x,y}(\mu)) μ ↦ min x max y f ( Υ x , y ( μ )) and μ ↦ max y min x f ( Υ x , y ( μ ) ) \mu\mapsto\max_y \min_x f(\Upsilon_{x,y}(\mu)) μ ↦ max y min x f ( Υ x , y ( μ )) are (uniformly) continuous. The same bound, applied with μ \mu μ fixed and only x x x (or only y y y ) varying, shows x ↦ max y ∈ Y f ( Υ x , y ( μ ) ) x \mapsto\max_{y \in\mathcal{Y}} f(\Upsilon_{x,y}(\mu)) x ↦ max y ∈ Y f ( Υ x , y ( μ )) is continuous on the compact set X \mathcal{X} X for fixed μ \mu μ , so by the extreme value theorem its minimum over X \mathcal{X} X is attained; the dual argument attains the maximum, then the minimum, for the other ordered extremum. Since H H H is continuous, both averaged operators preserve C ( K ) C(K) C ( K ) .
For either sampled semantics, every entry of M f ⋆ ( μ ) M_f^\star(\mu) M f ⋆ ( μ ) is continuous in μ \mu μ , since Φ a , b \Phi_{a,b} Φ a , b and Ψ ξ \Psi_\xi Ψ ξ are continuous and the sum over ξ ∈ Ξ \xi\in\Xi ξ ∈ Ξ is finite. For finite matrices M , N M,N M , N of the same dimensions,
∣ val ( M ) − val ( N ) ∣ ≤ max a , b ∣ M a , b − N a , b ∣ . (33) |\operatorname{val}(M)-\operatorname{val}(N)| \leq\max_{a,b}|M_{a,b}-N_{a,b}|. \tag*{(33)} ∣ val ( M ) − val ( N ) ∣ ≤ a , b max ∣ M a , b − N a , b ∣. (33)
Indeed, if the right-hand side is δ \delta δ , every mixed-strategy payoff changes by at most δ \delta δ , and taking the minimax value preserves this bound. Hence val \operatorname{val} val is continuous in the matrix entries, and T ⋆ T_\star T ⋆ also preserves C ( K ) C(K) C ( K ) .
We also require measurable optimal choices. The measurable maximum theorem [6 Theorem 18.19] gives Borel selectors for the outer extrema of the averaged operators. For the sampled operators, applying the same theorem to
( μ , σ ) ⟼ max τ ∈ D ( P max ) σ T M f ∗ ( μ ) τ (\mu,\sigma)\longmapsto\max_{\tau\in\mathcal{D}(\mathcal{P}_{\max})}\sigma^{\mathsf{T}}M_f^*(\mu)\tau ( μ , σ ) ⟼ τ ∈ D ( P m a x ) max σ T M f ∗ ( μ ) τ
and
( μ , τ ) ⟼ min σ ∈ D ( P min ) σ T M f ∗ ( μ ) τ (\mu,\tau)\longmapsto\min_{\sigma\in\mathcal{D}(\mathcal{P}_{\min})}\sigma^{\mathsf{T}}M_f^*(\mu)\tau ( μ , τ ) ⟼ σ ∈ D ( P m i n ) min σ T M f ∗ ( μ ) τ
gives Borel selectors of optimal mixed rows and columns.
Contraction. Let f , g ∈ C ( K ) f,g\in C(K) f , g ∈ C ( K ) and put δ = ∥ f − g ∥ ∞ \delta=\lVert f-g\rVert_\infty δ = ∥ f − g ∥ ∞ . For every μ , x , y \mu,x,y μ , x , y ,
∣ f ( Υ x , y ( μ ) ) − g ( Υ x , y ( μ ) ) ∣ ≤ δ . \left|f(\Upsilon_{x,y}(\mu))-g(\Upsilon_{x,y}(\mu))\right|\leq\delta. ∣ f ( Υ x , y ( μ )) − g ( Υ x , y ( μ )) ∣ ≤ δ .
Minimisation and maximisation preserve a uniform difference bound, and the current entropy terms cancel. Hence
∥ T a v ± f − T a v ± g ∥ ∞ ≤ γ δ . \lVert\mathcal{T}^{\pm}_{\mathrm{av}}f-\mathcal{T}^{\pm}_{\mathrm{av}}g\rVert_\infty\leq\gamma\delta. ∥ T av ± f − T av ± g ∥ ∞ ≤ γ δ .
For sampled-profile semantics, corresponding matrix entries differ by at most δ \delta δ . Under sampled-transition semantics,
∣ M f s t ( μ ) a , b − M g s t ( μ ) a , b ∣ ≤ ∑ ξ ∈ Ξ P a , b ( ξ ) ∣ f ( Ψ ξ ( μ ) ) − g ( Ψ ξ ( μ ) ) ∣ ≤ δ . \begin{aligned}
\left|M_f^{\mathrm{st}}(\mu)_{a,b}-M_g^{\mathrm{st}}(\mu)_{a,b}\right|
\leq\sum_{\xi\in\Xi}P_{a,b}(\xi)\left|f(\Psi_\xi(\mu))-g(\Psi_\xi(\mu))\right| \\
&\leq\delta.
\end{aligned} M f st ( μ ) a , b − M g st ( μ ) a , b ≤ ξ ∈ Ξ ∑ P a , b ( ξ ) ∣ f ( Ψ ξ ( μ )) − g ( Ψ ξ ( μ )) ∣ ≤ δ .
Together with (33) , this gives
∥ T ⋆ f − T ⋆ g ∥ ∞ ≤ γ δ . \lVert\mathcal{T}_\star f-\mathcal{T}_\star g\rVert_\infty\leq\gamma\delta. ∥ T ⋆ f − T ⋆ g ∥ ∞ ≤ γ δ .
Thus every operator is a γ \gamma γ -contraction. Since 0 < γ < 1 0<\gamma<1 0 < γ < 1 , Banach’s fixed-point theorem gives each operator a unique fixed point, and its iterates from zero converge uniformly to that fixed point.
Finite-horizon identification. For t ≥ 0 t\geq0 t ≥ 0 , let
Pay γ , t : = ( 1 − γ ) ∑ j = 0 t − 1 γ j H ( μ j ) , \operatorname{Pay}_{\gamma,t}:=(1-\gamma)\sum_{j=0}^{t-1}\gamma^j\mathrm{H}(\mu_j), Pay γ , t := ( 1 − γ ) j = 0 ∑ t − 1 γ j H ( μ j ) ,
with the empty sum equal to zero, and let V ⋆ , t + V^+_{\star,t} V ⋆ , t + and V ⋆ , t − V^-_{\star,t} V ⋆ , t − be its upper and lower values over the same history-dependent strategy spaces as in the infinite-horizon game. For averaged-distribution semantics, backward induction gives
V a v , 0 + = V a v , 0 − = 0 , (34) V^+_{\mathrm{av},0}=V^-_{\mathrm{av},0}=0, \tag*{(34)} V av , 0 + = V av , 0 − = 0 , (34)
V a v , t + 1 + ( μ ) = ( 1 − γ ) H ( μ ) + γ min x max y V a v , t + ( Υ x , y ( μ ) ) , (35) V^+_{\mathrm{av},t+1}(\mu)=(1-\gamma)\mathrm{H}(\mu)+\gamma\min_x\max_y V^+_{\mathrm{av},t}(\Upsilon_{x,y}(\mu)), \tag*{(35)} V av , t + 1 + ( μ ) = ( 1 − γ ) H ( μ ) + γ x min y max V av , t + ( Υ x , y ( μ )) , (35)
V a v , t + 1 − ( μ ) = ( 1 − γ ) H ( μ ) + γ max y min x V a v , t − ( Υ x , y ( μ ) ) . (36) V^-_{\mathrm{av},t+1}(\mu)=(1-\gamma)\mathrm{H}(\mu)+\gamma\max_y\min_x V^-_{\mathrm{av},t}(\Upsilon_{x,y}(\mu)). \tag*{(36)} V av , t + 1 − ( μ ) = ( 1 − γ ) H ( μ ) + γ y max x min V av , t − ( Υ x , y ( μ )) . (36)
We recall the argument for completeness. For the upper value, a Borel minimiser of the right-hand side of (35) , followed after the successor distribution by an optimal t t t -stage policy, gives the “≤ \leq ≤ ” inequality. Conversely, fix any ( t + 1 ) (t+1) ( t + 1 ) -stage strategy of Min and let x x x be its first action. Against any first-round response y y y , with ν = Υ x , y ( μ ) \nu=\Upsilon_{x,y}(\mu) ν = Υ x , y ( μ ) , its continuation has worst-case t t t -stage payoff at least V a v , t + ( ν ) V^+_{\mathrm{av},t}(\nu) V av , t + ( ν ) . Max can therefore choose a continuation arbitrarily close to this worst case. Maximising over y y y and then taking the infimum over Min gives the reverse inequality.
The lower recursion is dual. A Borel maximiser of the right-hand side of (36) , followed by an optimal t t t -stage policy, gives the “≥ \ge ≥ ” inequality. Conversely, fix a ( t + 1 ) (t+1) ( t + 1 ) -stage strategy of Max and let y y y be its first action. For each response x x x , with ν = Υ x , y ( μ ) \nu=\Upsilon_{x,y}(\mu) ν = Υ x , y ( μ ) , Min can choose a continuation whose payoff against the fixed continuation of Max is arbitrarily close from above to V a v , t − ( ν ) V^{-}_{\mathrm{av},t}(\nu) V av , t − ( ν ) , since
inf π min Obj γ , t ( π min , π max ; ν ) ≤ V a v , t − ( ν ) \inf_{\pi_{\min}}\operatorname{Obj}_{\gamma,t}(\pi_{\min},\pi_{\max};\nu)\le V^{-}_{\mathrm{av},t}(\nu) π m i n inf Obj γ , t ( π m i n , π m a x ; ν ) ≤ V av , t − ( ν )
for every fixed π max \pi_{\max} π m a x . Minimising over x x x and then taking the supremum over Max gives (36) . The measurable selectors above supply the required Markov policies. Thus
V a v , t + = ( T a v + ) t 0 , V a v , t − = ( T a v − ) t 0. V^{+}_{\mathrm{av},t}=(\mathcal{T}^{+}_{\mathrm{av}})^t0,\qquad V^{-}_{\mathrm{av},t}=(\mathcal{T}^{-}_{\mathrm{av}})^t0. V av , t + = ( T av + ) t 0 , V av , t − = ( T av − ) t 0.
Now fix ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } . We prove inductively that
V ⋆ , t − = V ⋆ , t + = W t , W 0 = 0 , W t + 1 = T ⋆ W t . V^{-}_{\star,t}=V^{+}_{\star,t}=W_t,\qquad W_0=0,\qquad W_{t+1}=\mathcal{T}_{\star}W_t. V ⋆ , t − = V ⋆ , t + = W t , W 0 = 0 , W t + 1 = T ⋆ W t .
For t + 1 t+1 t + 1 stages, committing in the first round to a pure-profile pair ( a , b ) (a,b) ( a , b ) gives continuation value
W t ( Φ a , b ( μ ) ) W_t(\Phi_{a,b}(\mu)) W t ( Φ a , b ( μ ))
under sampled-profile semantics, and
∑ ξ ∈ Ξ P a , b ( ξ ) W t ( Ψ ξ ( μ ) ) \sum_{\xi\in\Xi}P_{a,b}(\xi)W_t(\Psi_{\xi}(\mu)) ξ ∈ Ξ ∑ P a , b ( ξ ) W t ( Ψ ξ ( μ ))
under sampled-transition semantics. These are precisely the entries of M W t ⋆ ( μ ) M^{\star}_{W_t}(\mu) M W t ⋆ ( μ ) . The first-round interaction is therefore the finite zero-sum matrix game M W t ⋆ ( μ ) M^{\star}_{W_t}(\mu) M W t ⋆ ( μ ) . Finite minimax gives a common value and optimal mixed rows and columns; the selectors above make these choices Borel in μ \mu μ . After the successor distribution is observed, each player follows its inductively supplied optimal t t t -stage policy. Hence, against arbitrary history-dependent opponents,
V ⋆ , t + 1 + ( μ ) ≤ ( T ⋆ W t ) ( μ ) ≤ V ⋆ , t + 1 − ( μ ) . V^{+}_{\star,t+1}(\mu)\le(\mathcal{T}_{\star}W_t)(\mu)\le V^{-}_{\star,t+1}(\mu). V ⋆ , t + 1 + ( μ ) ≤ ( T ⋆ W t ) ( μ ) ≤ V ⋆ , t + 1 − ( μ ) .
Together with the minimax inequality V ⋆ , t + 1 − ≤ V ⋆ , t + 1 + V^{-}_{\star,t+1}\le V^{+}_{\star,t+1} V ⋆ , t + 1 − ≤ V ⋆ , t + 1 + , this proves equality and completes the induction.
Infinite-horizon identification and error bound. Since 0 ≤ H ( μ j ) ≤ L 0\le H(\mu_j)\le L 0 ≤ H ( μ j ) ≤ L , every path satisfies
0 ≤ Pay γ − Pay γ , t ≤ ( 1 − γ ) ∑ j = t ∞ γ j L = γ t L . 0\le\operatorname{Pay}_{\gamma}-\operatorname{Pay}_{\gamma,t}\le(1-\gamma)\sum_{j=t}^{\infty}\gamma^jL=\gamma^tL. 0 ≤ Pay γ − Pay γ , t ≤ ( 1 − γ ) j = t ∑ ∞ γ j L = γ t L .
The bound is uniform over initial distributions and strategy pairs. Taking expectations and then either ordered optimisation gives
0 ≤ V + ( μ ) − V ⋆ , t + ( μ ) ≤ γ t L . 0\le V^{+}(\mu)-V^{+}_{\star,t}(\mu)\le\gamma^tL. 0 ≤ V + ( μ ) − V ⋆ , t + ( μ ) ≤ γ t L .
For averaged semantics, the finite-horizon identities therefore show that the iterates of T a v ± \mathcal{T}^{\pm}_{\mathrm{av}} T av ± converge uniformly to V a v ± V^{\pm}_{\mathrm{av}} V av ± . Hence these values are the respective unique fixed points. For sampled semantics, both finite-horizon values equal W t W_t W t , so
W t ( μ ) ≤ V ⋆ − ( μ ) ≤ V ⋆ + ( μ ) ≤ W t ( μ ) + γ t L . W_t(\mu)\le V^{-}_{\star}(\mu)\le V^{+}_{\star}(\mu)\le W_t(\mu)+\gamma^tL. W t ( μ ) ≤ V ⋆ − ( μ ) ≤ V ⋆ + ( μ ) ≤ W t ( μ ) + γ t L .
Letting t → ∞ t\to\infty t → ∞ proves V ⋆ − = V ⋆ + V^{-}_{\star}=V^{+}_{\star} V ⋆ − = V ⋆ + and identifies the common value with the unique fixed point of T ⋆ \mathcal{T}_{\star} T ⋆ . The same inequalities give
0 ≤ V ( μ ) − W t ( μ ) ≤ γ t log 2 ∣ S ∣ , 0\le V(\mu)-W_t(\mu)\le\gamma^t\log_2|S|, 0 ≤ V ( μ ) − W t ( μ ) ≤ γ t log 2 ∣ S ∣ ,
as claimed. ◀
Determinacy Across the Three Semantics (with details)
Averaged-distribution semantics
Theorem 2. (Nondeterminacy under averaging). Stochastic games with distributional entropy objectives under averaged-distribution semantics are not determined.
For p ∈ [ 0 , 1 ] p \in[0,1] p ∈ [ 0 , 1 ] , write ( 1 − p , p ) (1-p,p) ( 1 − p , p ) for the distribution assigning mass 1 − p 1-p 1 − p to one outcome and mass p p p to the other, and define the binary entropy function (proposition 16 (iv))
h ( p ) : = H ( 1 − p , p ) = − p log 2 p − ( 1 − p ) log 2 ( 1 − p ) , h(p) := \mathbf{H}(1-p,p) = -p\log_{2}p-(1-p)\log_{2}(1-p), h ( p ) := H ( 1 − p , p ) = − p log 2 p − ( 1 − p ) log 2 ( 1 − p ) ,
Proof. Consider the four-state game of Figure 1. Write e = s min e=s_{\min} e = s m i n and a = s max a=s_{\max} a = s m a x . Initially μ 0 ( e ) = μ 0 ( a ) = 1 2 \mu_{0}(e)=\mu_{0}(a)=\frac{1}{2} μ 0 ( e ) = μ 0 ( a ) = 2 1 , while q 0 q_{0} q 0 and q 1 q_{1} q 1 have no mass. At both e e e and a a a , action i ∈ { 0 , 1 } i \in\{0,1\} i ∈ { 0 , 1 } moves deterministically to q i q_{i} q i ; the two q q q -states have only an absorbing self-loop. Let x x x be the probability with which Min chooses action 1 at e e e , and let y y y be the corresponding probability for Max at a a a . Only e e e and a a a contribute to the first update. Consequently,
μ 1 ( q 1 ) = μ 0 ( e ) x + μ 0 ( a ) y = x + y 2 , μ 1 ( q 0 ) = 1 − x + y 2 . \mu_{1}(q_{1})=\mu_{0}(e)x+\mu_{0}(a)y=\frac{x+y}{2}, \qquad\mu_{1}(q_{0})=1-\frac{x+y}{2}. μ 1 ( q 1 ) = μ 0 ( e ) x + μ 0 ( a ) y = 2 x + y , μ 1 ( q 0 ) = 1 − 2 x + y .
The two strategic probabilities occur in separate summands: no product x y xy x y is formed. Since q 0 q_{0} q 0 and q 1 q_{1} q 1 are absorbing, μ t = μ 1 \mu_{t}=\mu_{1} μ t = μ 1 for all t ≥ 1 t \ge1 t ≥ 1 . Moreover, H ( μ 0 ) = 1 \mathbf{H}(\mu_{0})=1 H ( μ 0 ) = 1 , because μ 0 \mu_{0} μ 0 is uniform on two states, and H ( μ 1 ) = h ( ( x + y ) / 2 ) \mathbf{H}(\mu_{1})=h((x+y)/2) H ( μ 1 ) = h (( x + y ) /2 ) since μ 1 \mu_{1} μ 1 is supported on { q 0 , q 1 } \{q_{0},q_{1}\} { q 0 , q 1 } with mass ( x + y ) / 2 (x+y)/2 ( x + y ) /2 on q 1 q_{1} q 1 . The entire discounted payoff is therefore
Obj γ a v ( x , y ) = ( 1 − γ ) H ( μ 0 ) + ( 1 − γ ) h ( x + y 2 ) ∑ t = 1 ∞ γ t = ( 1 − γ ) + γ h ( x + y 2 ) . \operatorname{Obj}_{\gamma}^{\mathrm{av}}(x,y)=(1-\gamma)\mathbf{H}(\mu_{0})+(1-\gamma)h\left(\frac{x+y}{2}\right)\sum_{t=1}^{\infty}\gamma^{t}=(1-\gamma)+\gamma h\left(\frac{x+y}{2}\right). Obj γ av ( x , y ) = ( 1 − γ ) H ( μ 0 ) + ( 1 − γ ) h ( 2 x + y ) t = 1 ∑ ∞ γ t = ( 1 − γ ) + γ h ( 2 x + y ) .
For calculating V a v + ( μ 0 ) V_{\mathrm{av}}^{+}(\mu_{0}) V av + ( μ 0 ) , fix any x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] . The response y = 1 − x y=1-x y = 1 − x makes the successor distribution uniform on { q 0 , q 1 } \{q_{0},q_{1}\} { q 0 , q 1 } . Since h ≤ 1 h \le1 h ≤ 1 everywhere,
sup y ∈ [ 0 , 1 ] h ( x + y 2 ) = 1 for every x , \sup_{y\in[0,1]}h\left(\frac{x+y}{2}\right)=1 \quad\text{for every }x, y ∈ [ 0 , 1 ] sup h ( 2 x + y ) = 1 for every x ,
hence
sup y ∈ [ 0 , 1 ] Obj γ a v ( x , y ) = ( 1 − γ ) + γ ⋅ 1 = 1 for every x , \sup_{y\in[0,1]}\operatorname{Obj}_{\gamma}^{\mathrm{av}}(x,y)=(1-\gamma)+\gamma\cdot1=1 \quad\text{for every }x, y ∈ [ 0 , 1 ] sup Obj γ av ( x , y ) = ( 1 − γ ) + γ ⋅ 1 = 1 for every x ,
and therefore
V a v + ( μ 0 ) = inf x ∈ [ 0 , 1 ] 1 = 1. V_{\mathrm{av}}^{+}(\mu_{0})=\inf_{x\in[0,1]}1=1. V av + ( μ 0 ) = x ∈ [ 0 , 1 ] inf 1 = 1.
For calculating V a v − ( μ 0 ) V_{\mathrm{av}}^{-}(\mu_{0}) V av − ( μ 0 ) value, fix y y y . As x x x ranges over [ 0 , 1 ] [0,1] [ 0 , 1 ] , the argument of h h h ranges over [ y / 2 , ( 1 + y ) / 2 ] [y/2,(1+y)/2] [ y /2 , ( 1 + y ) /2 ] . The map x ↦ h ( ( x + y ) / 2 ) x\mapsto h((x+y)/2) x ↦ h (( x + y ) /2 ) is concave (a concave function composed with an affine map of x x x ), and a concave function on a compact interval lies on or above the chord joining its endpoint values, so every interior point dominates the smaller of the two endpoints; hence its minimum over [ 0 , 1 ] [0,1] [ 0 , 1 ] occurs at x = 0 x=0 x = 0 or x = 1 x=1 x = 1 , giving
inf x ∈ [ 0 , 1 ] h ( x + y 2 ) = min { h ( y 2 ) , h ( 1 + y 2 ) } = min { h ( y 2 ) , h ( 1 − y 2 ) } . \inf_{x\in[0,1]}h\left(\frac{x+y}{2}\right)=\min\left\{h\left(\frac{y}{2}\right),h\left(\frac{1+y}{2}\right)\right\}=\min\left\{h\left(\frac{y}{2}\right),h\left(\frac{1-y}{2}\right)\right\}. x ∈ [ 0 , 1 ] inf h ( 2 x + y ) = min { h ( 2 y ) , h ( 2 1 + y ) } = min { h ( 2 y ) , h ( 2 1 − y ) } .
after using symmetry h ( 1 − p ) = h ( p ) h(1-p)=h(p) h ( 1 − p ) = h ( p ) to rewrite h ( ( 1 + y ) / 2 ) = h ( ( 1 − y ) / 2 ) h((1+y)/2)=h((1-y)/2) h (( 1 + y ) /2 ) = h (( 1 − y ) /2 ) . Both y / 2 y/2 y /2 and ( 1 − y ) / 2 (1-y)/2 ( 1 − y ) /2 lie in [ 0 , 1 2 ] [0,\frac{1}{2}] [ 0 , 2 1 ] , where h h h is increasing, so the smaller argument gives the smaller value:
inf x ∈ [ 0 , 1 ] h ( x + y 2 ) = h ( min { y , 1 − y } 2 ) . \inf_{x\in[0,1]}h\left(\frac{x+y}{2}\right)=h\left(\frac{\min\{y,1-y\}}{2}\right). x ∈ [ 0 , 1 ] inf h ( 2 x + y ) = h ( 2 min { y , 1 − y } ) .
The quantity min { y , 1 − y } \min\{y,1-y\} min { y , 1 − y } equals y y y (increasing in y y y ) for y ≤ 1 2 y \le\frac{1}{2} y ≤ 2 1 and 1 − y 1-y 1 − y (decreasing in y y y ) for y ≥ 1 2 y \ge\frac{1}{2} y ≥ 2 1 , so it is maximised at y = 1 2 y=\frac{1}{2} y = 2 1 , where it equals 1 2 \frac{1}{2} 2 1 . Since h h h is strictly increasing on [ 0 , 1 2 ] ⊇ [ 0 , 1 4 ] [0,\frac{1}{2}] \supseteq[0,\frac{1}{4}] [ 0 , 2 1 ] ⊇ [ 0 , 4 1 ] and min { y , 1 − y } / 2 \min\{y,1-y\}/2 min { y , 1 − y } /2 ranges over [ 0 , 1 4 ] [0,\frac{1}{4}] [ 0 , 4 1 ] , composing with h h h preserves this maximiser: the expression above is maximised uniquely at y = 1 2 y=\frac{1}{2} y = 2 1 , with value h ( 1 / 4 ) h(1/4) h ( 1/4 ) . Hence
V a v − ( μ 0 ) = sup y ∈ [ 0 , 1 ] ( ( 1 − γ ) + γ inf x ∈ [ 0 , 1 ] h ( x + y 2 ) ) = ( 1 − γ ) + γ h ( 1 4 ) . V_{\mathrm{av}}^{-}(\mu_0)=\sup_{y\in[0,1]}\left((1-\gamma)+\gamma\inf_{x\in[0,1]}h\left(\frac{x+y}{2}\right)\right)=(1-\gamma)+\gamma h\left(\frac{1}{4}\right). V av − ( μ 0 ) = y ∈ [ 0 , 1 ] sup ( ( 1 − γ ) + γ x ∈ [ 0 , 1 ] inf h ( 2 x + y ) ) = ( 1 − γ ) + γ h ( 4 1 ) .
Finally,
h ( 1 4 ) = 2 − 3 4 log 2 3 ≈ 0.8113 , h\left(\frac{1}{4}\right)=2-\frac{3}{4}\log_2 3\approx0.8113, h ( 4 1 ) = 2 − 4 3 log 2 3 ≈ 0.8113 ,
so
V a v + ( μ 0 ) − V a v − ( μ 0 ) = 1 − ( ( 1 − γ ) + γ h ( 1 4 ) ) V_{\mathrm{av}}^{+}(\mu_0)-V_{\mathrm{av}}^{-}(\mu_0)=1-\left((1-\gamma)+\gamma h\left(\frac{1}{4}\right)\right) V av + ( μ 0 ) − V av − ( μ 0 ) = 1 − ( ( 1 − γ ) + γ h ( 4 1 ) )
V a v + ( μ 0 ) − V a v − ( μ 0 ) = γ ( 1 − h ( 1 4 ) ) ≈ 0.1887 γ > 0. V_{\mathrm{av}}^{+}(\mu_0)-V_{\mathrm{av}}^{-}(\mu_0)=\gamma\left(1-h\left(\frac{1}{4}\right)\right)\approx0.1887\gamma>0. V av + ( μ 0 ) − V av − ( μ 0 ) = γ ( 1 − h ( 4 1 ) ) ≈ 0.1887 γ > 0.
Because q 0 , q 1 q_0,q_1 q 0 , q 1 absorb all mass after round 0, only the round-0 actions at e e e and a a a affect H ( μ t ) \mathbf{H}(\mu_t) H ( μ t ) for any t ≥ 1 t\ge1 t ≥ 1 ; every Borel strategy in Π a v min \Pi_{\mathrm{av}}^{\min} Π av m i n or Π a v max \Pi_{\mathrm{av}}^{\max} Π av m a x therefore induces the same value of Obj γ a v \operatorname{Obj}_{\gamma}^{\mathrm{av}} Obj γ av as some constant pair ( x , y ) ∈ [ 0 , 1 ] 2 (x,y)\in[0,1]^2 ( x , y ) ∈ [ 0 , 1 ] 2 specifying that single round-0 mixed action. Consequently V a v + ( μ 0 ) = inf σ sup τ Obj γ a v ( σ , τ ; μ 0 ) V_{\mathrm{av}}^{+}(\mu_0)=\inf_{\sigma}\sup_{\tau}\operatorname{Obj}_{\gamma}^{\mathrm{av}}(\sigma,\tau;\mu_0) V av + ( μ 0 ) = inf σ sup τ Obj γ av ( σ , τ ; μ 0 ) , the infimum and supremum taken over the full Borel strategy spaces, equals inf x ∈ [ 0 , 1 ] sup y ∈ [ 0 , 1 ] Obj γ a v ( x , y ) \inf_{x\in[0,1]}\sup_{y\in[0,1]}\operatorname{Obj}_{\gamma}^{\mathrm{av}}(x,y) inf x ∈ [ 0 , 1 ] sup y ∈ [ 0 , 1 ] Obj γ av ( x , y ) , the quantity computed above; likewise V a v − ( μ 0 ) V_{\mathrm{av}}^{-}(\mu_0) V av − ( μ 0 ) equals sup y ∈ [ 0 , 1 ] inf x ∈ [ 0 , 1 ] Obj γ a v ( x , y ) \sup_{y\in[0,1]}\inf_{x\in[0,1]}\operatorname{Obj}_{\gamma}^{\mathrm{av}}(x,y) sup y ∈ [ 0 , 1 ] inf x ∈ [ 0 , 1 ] Obj γ av ( x , y ) . Each of these is an equality between a full-strategy-space value and the constant-pair value of the same polarity (V + V^{+} V + with V + V^{+} V + , V − V^{-} V − with V − V^{-} V − ); it is not a claim that V a v + = V a v − V_{\mathrm{av}}^{+}=V_{\mathrm{av}}^{-} V av + = V av − , which the computation above already shows is false. There is no history to react to, so history-dependent strategies cannot close either gap.
Remark 24 (A variational reformulation). Since H ( μ ) = min q ∑ s μ ( s ) log 2 1 q ( s ) H(\mu)=\min_q\sum_s\mu(s)\log_2\frac{1}{q(s)} H ( μ ) = min q ∑ s μ ( s ) log 2 q ( s ) 1 , with the minimum at q = μ q=\mu q = μ , one may try to replace distributions by states and let an extra minimiser choice q q q play the dual variable. This does not give an equivalent game on S S S . If q q q is fixed together with Min’s action, so that it cannot depend on Max’s choice in the same round, then in the game of Figure 1 the upper value is 1 1 1 and the lower value is ( 1 − γ ) + γ h ( 1 / 4 ) (1-\gamma)+\gamma h(1/4) ( 1 − γ ) + γ h ( 1/4 ) , so the values still differ. If q q q may depend on the distribution it evaluates, which is determined by both players’ choices in the round, the game is the one we study, on distributions.
Figure 1. Nondeterminacy under averaged-distribution semantics.
Sampled-profile and sampled-transition semantics
The two sampled semantics have different one-step dynamics, but they share the feature that makes determinacy possible: once the current pure profiles have been fixed, all remaining one-step randomness produces a single scalar expected continuation payoff. This scalar can be placed in a finite payoff matrix, with one row for each pure profile of Min and one column for each pure profile of Max. To make this precise, fix ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } and let K ⋆ ( ⋅ ∣ μ , a , b ) K_{\star}(\mathord{\cdot}\mid\mu,a,b) K ⋆ ( ⋅ ∣ μ , a , b ) denote the law of the next distribution when the current distribution is μ \mu μ and the pure-profile pair ( a , b ) (a,b) ( a , b ) has been selected.
K s p ( ⋅ ∣ μ , a , b ) = δ Φ a , b ( μ ) . K_{\mathrm{sp}}(\mathord{\cdot}\mid\mu,a,b)=\delta_{\Phi_{a,b}(\mu)}. K sp ( ⋅ ∣ μ , a , b ) = δ Φ a , b ( μ ) .
K s t ( ⋅ ∣ μ , a , b ) = ∑ ξ ∈ Ξ P a , b ( ξ ) δ Ψ ξ ( μ ) . K_{\mathrm{st}}(\mathord{\cdot}\mid\mu,a,b)=\sum_{\xi\in\Xi}P_{a,b}(\xi)\,\delta_{\Psi_{\xi}(\mu)}. K st ( ⋅ ∣ μ , a , b ) = ξ ∈ Ξ ∑ P a , b ( ξ ) δ Ψ ξ ( μ ) .
We briefly describe the matrix-game background used below. A finite zero-sum matrix game is a one-round game specified by a real matrix M = ( M a , b ) a ∈ P min , b ∈ P max M=(M_{a,b})_{a\in P_{\min},b\in P_{\max}} M = ( M a , b ) a ∈ P m i n , b ∈ P m a x . The row player Min chooses a row and seeks to minimise the entry, while the column player Max chooses a column and seeks to maximise it. The players may randomise: Min chooses σ ∈ D ( P min ) \sigma\in\mathcal{D}(P_{\min}) σ ∈ D ( P m i n ) and Max chooses τ ∈ D ( P max ) \tau\in\mathcal{D}(P_{\max}) τ ∈ D ( P m a x ) , independently. Their expected payoff is
G M ( σ , τ ) : = ∑ a , b σ ( a ) τ ( b ) M a , b . G_M(\sigma,\tau):=\sum_{a,b}\sigma(a)\tau(b)M_{a,b}. G M ( σ , τ ) := a , b ∑ σ ( a ) τ ( b ) M a , b .
This expression is bilinear: it is linear in σ \sigma σ when τ \tau τ is fixed and linear in τ \tau τ when σ \sigma σ is fixed. The finite minimax theorem, also called von Neumann’s minimax theorem [36 ] , states that every finite zero-sum matrix game has a common mixed-strategy value:
min σ ∈ D ( P min ) max τ ∈ D ( P max ) G M ( σ , τ ) = max τ ∈ D ( P max ) min σ ∈ D ( P min ) G M ( σ , τ ) . (37) \min_{\sigma\in\mathcal{D}(P_{\min})}\max_{\tau\in\mathcal{D}(P_{\max})}G_M(\sigma,\tau)
=
\max_{\tau\in\mathcal{D}(P_{\max})}\min_{\sigma\in\mathcal{D}(P_{\min})}G_M(\sigma,\tau). \tag*{(37)} σ ∈ D ( P m i n ) min τ ∈ D ( P m a x ) max G M ( σ , τ ) = τ ∈ D ( P m a x ) max σ ∈ D ( P m i n ) min G M ( σ , τ ) . (37)
We denote their common number by val ( M ) \operatorname{val}(M) val ( M ) . Since the mixed-strategy simplices are compact and G M G_M G M is continuous, the minimum and maximum are attained. Thus there are optimal mixed strategies σ ∗ \sigma^* σ ∗ and τ ∗ \tau^* τ ∗ satisfying the saddle inequalities
G M ( σ ∗ , τ ) ≤ val ( M ) ≤ G M ( σ , τ ∗ ) for every σ , τ . (38) G_M(\sigma^*,\tau)\leq\operatorname{val}(M)\leq G_M(\sigma,\tau^*)\quad\text{for every }\sigma,\tau. \tag*{(38)} G M ( σ ∗ , τ ) ≤ val ( M ) ≤ G M ( σ , τ ∗ ) for every σ , τ . (38)
This theorem applies naturally to the sampled semantics. If F : D ( S ) → R F:\mathcal{D}(S)\to\mathbb{R} F : D ( S ) → R is a continuation payoff, then the entry associated with ( a , b ) (a,b) ( a , b ) is the scalar
∫ D ( S ) F ( ν ) K ⋆ ( d ν ∣ μ , a , b ) . \int_{\mathcal{D}(S)}F(\nu)\,K_\star(\mathrm{d}\nu\mid\mu,a,b). ∫ D ( S ) F ( ν ) K ⋆ ( d ν ∣ μ , a , b ) .
Randomising over the current pure profiles averages these already-evaluated scalar entries and therefore produces exactly the bilinear payoff G M G_M G M .
Theorem 4 (Sampled-profile and sampled-transition determinacy). For every ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } , V ⋆ − = V ⋆ + V_\star^-=V_\star^+ V ⋆ − = V ⋆ + . Moreover, both players have optimal stationary randomised strategies, which remain optimal against arbitrary history-dependent randomised strategies of the opponent.
Proof. Let E = ( S , S min , S max , A , Δ , μ 0 ) \mathcal{E}=(S,S_{\min},S_{\max},A,\Delta,\mu_0) E = ( S , S m i n , S m a x , A , Δ , μ 0 ) be an entropy game, and let ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } . Put L : = log 2 ∣ S ∣ L:=\log_2|S| L := log 2 ∣ S ∣ .
Theorem 4 makes two claims: (1) V ⋆ − = V ⋆ + V_\star^-=V_\star^+ V ⋆ − = V ⋆ + , and (2) both players have optimal stationary strategies that stay optimal against arbitrary history-dependent opponents. The proof has three steps. Step 1 collects what the proof of Theorem 1 (Section C) already establishes, which gives claim (1) and a one-step identity for the true value V ⋆ V_\star V ⋆ . Step 2 builds Borel measurable stationary saddle strategies from that identity, and Step 3 shows that they are optimal against the whole history-dependent strategy spaces, which is claim (2).
Step 1: determinacy and the one-step identity. For the kernel K ⋆ K_\star K ⋆ above and a continuous f : D ( S ) → R f:\mathcal{D}(S)\to\mathbb{R} f : D ( S ) → R , the entry ∫ D ( S ) f ( ν ) K ⋆ ( d ν ∣ μ , a , b ) \int_{\mathcal{D}(S)}f(\nu)\,K_\star(\mathrm{d}\nu\mid\mu,a,b) ∫ D ( S ) f ( ν ) K ⋆ ( d ν ∣ μ , a , b ) equals f ( Φ a , b ( μ ) ) f(\Phi_{a,b}(\mu)) f ( Φ a , b ( μ )) for ⋆ = s p \star=\mathrm{sp} ⋆ = sp and ∑ ξ ∈ Ξ P a , b ( ξ ) f ( Ψ ξ ( μ ) ) \sum_{\xi\in\Xi}P_{a,b}(\xi)f(\Psi_\xi(\mu)) ∑ ξ ∈ Ξ P a , b ( ξ ) f ( Ψ ξ ( μ )) for ⋆ = s t \star=\mathrm{st} ⋆ = st . These are exactly the entries of the matrix M ⋆ f ( μ ) M_\star^f(\mu) M ⋆ f ( μ ) of Theorem 1 , so T ⋆ f = ( 1 − γ ) H + γ val ( M ⋆ f ) \mathcal{T}_\star f=(1-\gamma)\mathbf{H}+\gamma\operatorname{val}(M_\star^f) T ⋆ f = ( 1 − γ ) H + γ val ( M ⋆ f ) . The proof of Theorem 1 (Section C) shows, for L : = log 2 ∣ S ∣ L:=\log_2|S| L := log 2 ∣ S ∣ , that the n n n -stage upper and lower values coincide with W n ∗ = ( T ⋆ ) n 0 W_n^*=(\mathcal{T}_\star)^n0 W n ∗ = ( T ⋆ ) n 0 , which is continuous, with Borel Markov optimal strategies at every horizon, and that
∥ V ⋆ − W n ⋆ ∥ ∞ ≤ γ n L , V ⋆ − = V ⋆ + = : V ⋆ . (39) \lVert V_\star-W_n^\star\rVert_\infty\leq\gamma^nL,\qquad V_\star^-=V_\star^+=:V_\star. \tag*{(39)} ∥ V ⋆ − W n ⋆ ∥ ∞ ≤ γ n L , V ⋆ − = V ⋆ + =: V ⋆ . (39)
where ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_{\infty} ∥ ⋅ ∥ ∞ is as in Section A.1. This proves claim (1). It also shows that V ⋆ V_{\star} V ⋆ is continuous (as a uniform limit of continuous functions), that 0 ≤ V ⋆ ≤ L 0 \le V_{\star} \le L 0 ≤ V ⋆ ≤ L , and that V ⋆ V_{\star} V ⋆ is the unique fixed point of T ⋆ \mathcal{T}_{\star} T ⋆ . Define the limiting continuation matrix
M ⋆ ( μ ) a , b : = ∫ D ( S ) V ⋆ ( ν ) K ⋆ ( d ν ∣ μ , a , b ) , (40) M^{\star}(\mu)_{a,b} := \int_{\mathcal{D}(S)} V_{\star}(\nu)K_{\star}(\mathrm{d}\nu\mid\mu,a,b), \tag*{(40)} M ⋆ ( μ ) a , b := ∫ D ( S ) V ⋆ ( ν ) K ⋆ ( d ν ∣ μ , a , b ) , (40)
that is, M ⋆ = M V ⋆ ⋆ M^{\star}=M^{\star}_{V_{\star}} M ⋆ = M V ⋆ ⋆ , with entries V s p ( Φ a , b ( μ ) ) V_{\mathrm{sp}}(\Phi_{a,b}(\mu)) V sp ( Φ a , b ( μ )) under sampled-profile semantics and ∑ ξ ∈ Ξ P a , b ( ξ ) V s t ( Ψ ξ ( μ ) ) \sum_{\xi\in\Xi}P_{a,b}(\xi)V_{\mathrm{st}}(\Psi_{\xi}(\mu)) ∑ ξ ∈ Ξ P a , b ( ξ ) V st ( Ψ ξ ( μ )) under sampled-transition semantics. The fixed-point equation V ⋆ = T ⋆ V ⋆ V_{\star}=\mathcal{T}_{\star}V_{\star} V ⋆ = T ⋆ V ⋆ then reads
V ⋆ ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M ⋆ ( μ ) ) . (41) V_{\star}(\mu)=(1-\gamma)\mathrm{H}(\mu)+\gamma\operatorname{val}(M^{\star}(\mu)). \tag*{(41)} V ⋆ ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M ⋆ ( μ )) . (41)
Step 2: measurable stationary saddle strategies. For each μ \mu μ , let Σ min ⋆ ( μ ) \Sigma^{\star}_{\min}(\mu) Σ m i n ⋆ ( μ ) and Σ max ⋆ ( μ ) \Sigma^{\star}_{\max}(\mu) Σ m a x ⋆ ( μ ) be the sets of optimal mixed rows and columns in M ⋆ ( μ ) M^{\star}(\mu) M ⋆ ( μ ) . The entries of this matrix depend continuously on μ \mu μ : this follows from continuity of V ⋆ V_{\star} V ⋆ and of the successor maps, as in the proof of Theorem 1 . Hence the optimal-strategy sets are nonempty and compact by the finite minimax theorem and vary measurably with μ \mu μ . The same measurable maximum theorem used in Section C therefore provides Borel selectors
σ ⋆ ( μ ) ∈ Σ min ⋆ ( μ ) , τ ⋆ ( μ ) ∈ Σ max ⋆ ( μ ) . \sigma^{\star}(\mu)\in\Sigma^{\star}_{\min}(\mu),\qquad\tau^{\star}(\mu)\in\Sigma^{\star}_{\max}(\mu). σ ⋆ ( μ ) ∈ Σ m i n ⋆ ( μ ) , τ ⋆ ( μ ) ∈ Σ m a x ⋆ ( μ ) .
Because these selectors depend only on the current distribution, using them at every round defines stationary randomised strategies of the type specified in § 2 . Measurability is what ensures that the corresponding one-step kernels and canonical path laws remain well defined. Optimality in the one-step matrix game and (33) imply, for every μ \mu μ and every mixed profile τ \tau τ of Max,
V ⋆ ( μ ) ≥ ( 1 − γ ) H ( μ ) + γ ∑ a , b σ ⋆ ( μ ) ( a ) τ ( b ) M ⋆ ( μ ) a , b , (42) V_{\star}(\mu)\ge(1-\gamma)\mathrm{H}(\mu)+\gamma\sum_{a,b}\sigma^{\star}(\mu)(a)\tau(b)M^{\star}(\mu)_{a,b}, \tag*{(42)} V ⋆ ( μ ) ≥ ( 1 − γ ) H ( μ ) + γ a , b ∑ σ ⋆ ( μ ) ( a ) τ ( b ) M ⋆ ( μ ) a , b , (42)
and, for every mixed profile σ \sigma σ of Min,
V ⋆ ( μ ) ≤ ( 1 − γ ) H ( μ ) + γ ∑ a , b σ ( a ) τ ⋆ ( μ ) ( b ) M ⋆ ( μ ) a , b . (43) V_{\star}(\mu)\le(1-\gamma)\mathrm{H}(\mu)+\gamma\sum_{a,b}\sigma(a)\tau^{\star}(\mu)(b)M^{\star}(\mu)_{a,b}. \tag*{(43)} V ⋆ ( μ ) ≤ ( 1 − γ ) H ( μ ) + γ a , b ∑ σ ( a ) τ ⋆ ( μ ) ( b ) M ⋆ ( μ ) a , b . (43)
Indeed, the first inequality is obtained from G M ⋆ ( μ ) ( σ ⋆ ( μ ) , τ ) ≤ val ( M ⋆ ( μ ) ) G_{M^{\star}(\mu)}(\sigma^{\star}(\mu),\tau)\le\operatorname{val}(M^{\star}(\mu)) G M ⋆ ( μ ) ( σ ⋆ ( μ ) , τ ) ≤ val ( M ⋆ ( μ )) , and the second from the other saddle inequality in (38) .
Step 3: verification against history-dependent opponents. Let the minimiser use σ ⋆ \sigma^{\star} σ ⋆ against an arbitrary history-dependent randomised strategy of the maximiser. Conditional on F t \mathcal{F}_{t} F t , the observed history ( μ 0 , … , μ t ) (\mu_{0},\ldots,\mu_{t}) ( μ 0 , … , μ t ) and therefore the current distribution μ t \mu_{t} μ t are fixed, while the opponent’s current prescription is some F t \mathcal{F}_{t} F t -measurable mixed profile τ t \tau_{t} τ t . By the path-law identity (19) , conditional on F t \mathcal{F}_{t} F t the law of μ t + 1 \mu_{t+1} μ t + 1 is the mixture ∑ a , b σ ⋆ ( μ t ) ( a ) τ t ( b ) K ⋆ ( ⋅ ∣ μ t , a , b ) \sum_{a,b}\sigma^{\star}(\mu_{t})(a)\tau_{t}(b)K_{\star}(\cdot\mid\mu_{t},a,b) ∑ a , b σ ⋆ ( μ t ) ( a ) τ t ( b ) K ⋆ ( ⋅ ∣ μ t , a , b ) of one-step kernels, weighted by the current prescriptions σ ⋆ ( μ t ) \sigma^{\star}(\mu_{t}) σ ⋆ ( μ t ) and τ t \tau_{t} τ t . Taking the conditional expectation of V ⋆ V_{\star} V ⋆ against this law, and swapping the finite sum with the integral, gives
E [ V ⋆ ( μ t + 1 ) ∣ F t ] = ∑ a , b σ ⋆ ( μ t ) ( a ) τ t ( b ) ∫ D ( S ) V ⋆ ( ν ) K ⋆ ( d ν ∣ μ t , a , b ) . \begin{aligned}
\mathbb{E}[V_{\star}(\mu_{t+1})\mid\mathcal{F}_{t}]
&=\sum_{a,b}\sigma^{\star}(\mu_{t})(a)\tau_{t}(b)
\int_{\mathcal{D}(S)}V_{\star}(\nu)K_{\star}(\mathrm{d}\nu\mid\mu_{t},a,b).
\end{aligned} E [ V ⋆ ( μ t + 1 ) ∣ F t ] = a , b ∑ σ ⋆ ( μ t ) ( a ) τ t ( b ) ∫ D ( S ) V ⋆ ( ν ) K ⋆ ( d ν ∣ μ t , a , b ) .
Each inner integral is exactly M ⋆ ( μ t ) a , b M^{\star}(\mu_t)_{a,b} M ⋆ ( μ t ) a , b by its defining equation (40) , so
E [ V ⋆ ( μ t + 1 ) ∣ F t ] = ∑ a , b σ ⋆ ( μ t ) ( a ) τ t ( b ) M ⋆ ( μ t ) a , b . \mathbb{E}[V_{\star}(\mu_{t+1}) \mid\mathcal{F}_t]
= \sum_{a,b}\sigma^{\star}(\mu_t)(a)\tau_t(b)M^{\star}(\mu_t)_{a,b}. E [ V ⋆ ( μ t + 1 ) ∣ F t ] = a , b ∑ σ ⋆ ( μ t ) ( a ) τ t ( b ) M ⋆ ( μ t ) a , b .
Substituting this equality into (42) gives, almost surely,
V ⋆ ( μ t ) ≥ ( 1 − γ ) H ( μ t ) + γ E [ V ⋆ ( μ t + 1 ) ∣ F t ] . V_{\star}(\mu_t) \ge(1-\gamma)\mathrm{H}(\mu_t)+\gamma\mathbb{E}[V_{\star}(\mu_{t+1})\mid\mathcal{F}_t]. V ⋆ ( μ t ) ≥ ( 1 − γ ) H ( μ t ) + γ E [ V ⋆ ( μ t + 1 ) ∣ F t ] .
Multiply by γ t \gamma^t γ t , take expectations, and use the tower property E [ E [ X ∣ F t ] ] = E [ X ] \mathbb{E}[\mathbb{E}[X\mid\mathcal{F}_t]]=\mathbb{E}[X] E [ E [ X ∣ F t ]] = E [ X ] . Summing for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 cancels every intermediate continuation term: the term γ t E [ V ⋆ ( μ t ) ] \gamma^t\mathbb{E}[V_{\star}(\mu_t)] γ t E [ V ⋆ ( μ t )] on the left at round t t t cancels the corresponding term produced on the right at round t − 1 t-1 t − 1 . Only the initial and terminal terms remain, giving
( 1 − γ ) ∑ t = 0 T − 1 γ t E [ H ( μ t ) ] ≤ V ⋆ ( μ 0 ) − γ T E [ V ⋆ ( μ T ) ] ≤ V ⋆ ( μ 0 ) . (1-\gamma)\sum_{t=0}^{T-1}\gamma^t\mathbb{E}[\mathrm{H}(\mu_t)]\le V_{\star}(\mu_0)-\gamma^T\mathbb{E}[V_{\star}(\mu_T)]\le V_{\star}(\mu_0). ( 1 − γ ) t = 0 ∑ T − 1 γ t E [ H ( μ t )] ≤ V ⋆ ( μ 0 ) − γ T E [ V ⋆ ( μ T )] ≤ V ⋆ ( μ 0 ) .
Recall from § 2 that
Obj γ ⋆ ( π min , π max ; μ ) = ( 1 − γ ) ∑ t = 0 ∞ γ t E μ ⋆ , π min , π max [ H ( μ t ) ] . (44) \operatorname{Obj}^{\star}_{\gamma}(\pi_{\min},\pi_{\max};\mu)=(1-\gamma)\sum_{t=0}^{\infty}\gamma^t\mathbb{E}^{\star,\pi_{\min},\pi_{\max}}_{\mu}[\mathrm{H}(\mu_t)]. \tag*{(44)} Obj γ ⋆ ( π m i n , π m a x ; μ ) = ( 1 − γ ) t = 0 ∑ ∞ γ t E μ ⋆ , π m i n , π m a x [ H ( μ t )] . (44)
Letting T → ∞ T\to\infty T → ∞ and using Equation (44) yields
Obj γ ⋆ ( σ ⋆ , π max ; μ 0 ) ≤ V ⋆ ( μ 0 ) for every π max . \operatorname{Obj}^{\star}_{\gamma}(\sigma^{\star},\pi_{\max};\mu_0)\le V_{\star}(\mu_0)\quad\text{for every }\pi_{\max}. Obj γ ⋆ ( σ ⋆ , π m a x ; μ 0 ) ≤ V ⋆ ( μ 0 ) for every π m a x .
For completeness, let the maximiser use τ ⋆ \tau^{\star} τ ⋆ against an arbitrary history-dependent minimising strategy. Conditional on F t \mathcal{F}_t F t , let σ t \sigma_t σ t be the minimiser’s current mixed profile. Then
E [ V ⋆ ( μ t + 1 ) ∣ F t ] = ∑ a , b σ t ( a ) τ ⋆ ( μ t ) ( b ) M ⋆ ( μ t ) a , b . \mathbb{E}[V_{\star}(\mu_{t+1})\mid\mathcal{F}_t]=\sum_{a,b}\sigma_t(a)\tau^{\star}(\mu_t)(b)M^{\star}(\mu_t)_{a,b}. E [ V ⋆ ( μ t + 1 ) ∣ F t ] = a , b ∑ σ t ( a ) τ ⋆ ( μ t ) ( b ) M ⋆ ( μ t ) a , b .
Substitution into (43) , followed by the same discounting, expectation, and telescoping calculation, yields
( 1 − γ ) ∑ t = 0 T − 1 γ t E [ H ( μ t ) ] ≥ V ⋆ ( μ 0 ) − γ T E [ V ⋆ ( μ T ) ] . (1-\gamma)\sum_{t=0}^{T-1}\gamma^t\mathbb{E}[\mathrm{H}(\mu_t)]\ge V_{\star}(\mu_0)-\gamma^T\mathbb{E}[V_{\star}(\mu_T)]. ( 1 − γ ) t = 0 ∑ T − 1 γ t E [ H ( μ t )] ≥ V ⋆ ( μ 0 ) − γ T E [ V ⋆ ( μ T )] .
Since 0 ≤ V ⋆ ≤ L 0\le V_{\star}\le L 0 ≤ V ⋆ ≤ L ,
0 ≤ γ T E [ V ⋆ ( μ T ) ] ≤ γ T L ⟶ 0. 0\le\gamma^T\mathbb{E}[V_{\star}(\mu_T)]\le\gamma^T L\longrightarrow0. 0 ≤ γ T E [ V ⋆ ( μ T )] ≤ γ T L ⟶ 0.
Thus Equation (44) gives
Obj γ ⋆ ( π min , τ ⋆ ; μ 0 ) ≥ V ⋆ ( μ 0 ) for every π min . \operatorname{Obj}^{\star}_{\gamma}(\pi_{\min},\tau^{\star};\mu_0)\ge V_{\star}(\mu_0)\quad\text{for every }\pi_{\min}. Obj γ ⋆ ( π m i n , τ ⋆ ; μ 0 ) ≥ V ⋆ ( μ 0 ) for every π m i n .
Thus σ ⋆ \sigma^{\star} σ ⋆ guarantees at most V ⋆ V_{\star} V ⋆ and τ ⋆ \tau^{\star} τ ⋆ guarantees at least V ⋆ V_{\star} V ⋆ . They are therefore optimal against the full history-dependent strategy spaces. The argument used only the common kernel description: substituting K s p K_{\mathrm{sp}} K sp proves the sp case of Theorem 4, and substituting K s t K_{\mathrm{st}} K st proves the st case.
Gap Value Upper Bounds
We study the gap problem from § 2 , grouping the results by semantics. Throughout this section, the discount γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and gap ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 are fixed rational constants. The input consists of the explicitly listed arena, rational transition entries Δ ( s , a , s ′ ) \Delta(s,a,s') Δ ( s , a , s ′ ) , rational initial distribution, and rational threshold v v v , all encoded in binary. Write ℓ \ell ℓ for the total input length, n = ∣ S ∣ n = |S| n = ∣ S ∣ , A = max s ∣ A ( s ) ∣ A = \max_{s} |A(s)| A = max s ∣ A ( s ) ∣ , and e = min { ε 0 , 1 } e = \min\{\varepsilon_0,1\} e = min { ε 0 , 1 } . For n ≥ 2 n \ge2 n ≥ 2 put L = ⌈ log 2 n ⌉ L = \lceil\log_2 n\rceil L = ⌈ log 2 n ⌉ ; one-state games have value zero.
Averaged-distribution semantics
The upper and lower values can differ under this semantics, so we explicitly study GapVal a v , ε 0 + \operatorname{GapVal}^{+}_{\mathrm{av},\varepsilon_0} GapVal av , ε 0 + without assuming determinacy.
Entropy is a nonlinear, transcendental stage reward: − x log 2 x -x\log_2 x − x log 2 x has no closed rational form except at x ∈ { 0 , 1 } x \in\{0,1\} x ∈ { 0 , 1 } . Lemma 6 packages two tools needed to tame it, each used later for a different purpose. Part 1 (continuity) bounds how much entropy can change between nearby distributions. Part 2 (effective evaluation) computes H ( μ ) \mathbf{H}(\mu) H ( μ ) to any desired number of bits in time polynomial in that number; this is what lets Theorem 7 below evaluate entropy exactly enough to decide the gap problem, without ever needing its closed, transcendental form.
Lemma 6 (Entropy tools). Let n = ∣ S ∣ n = |S| n = ∣ S ∣ , and let μ , ν ∈ D ( S ) \mu,\nu\in\mathcal{D}(S) μ , ν ∈ D ( S ) , with d T V ( μ , ν ) = 1 2 ∥ μ − ν ∥ 1 d_{\mathrm{TV}}(\mu,\nu) = \frac{1}{2}\lVert\mu-\nu\rVert_1 d TV ( μ , ν ) = 2 1 ∥ μ − ν ∥ 1 .
Continuity. Let ω n ( r ) = r log 2 ( n − 1 ) + h ( r ) \omega_n(r) = r\log_2(n-1) + h(r) ω n ( r ) = r log 2 ( n − 1 ) + h ( r ) , where h ( r ) = − r log 2 r − ( 1 − r ) log 2 ( 1 − r ) h(r) = -r\log_2 r - (1-r)\log_2(1-r) h ( r ) = − r log 2 r − ( 1 − r ) log 2 ( 1 − r ) denotes binary entropy. For 0 ≤ r ≤ 1 − 1 n 0 \le r \le1-\frac{1}{n} 0 ≤ r ≤ 1 − n 1 , ω n \omega_n ω n is nondecreasing, and d T V ( μ , ν ) ≤ r d_{\mathrm{TV}}(\mu,\nu) \le r d TV ( μ , ν ) ≤ r implies
∣ H ( μ ) − H ( ν ) ∣ ≤ ω n ( r ) . |\mathbf{H}(\mu)-\mathbf{H}(\nu)| \le\omega_n(r). ∣ H ( μ ) − H ( ν ) ∣ ≤ ω n ( r ) .
Moreover, h ( r ) ≤ 2 r h(r) \le2\sqrt{r} h ( r ) ≤ 2 r for 0 ≤ r ≤ 1 / 2 0 \le r \le1/2 0 ≤ r ≤ 1/2 .
Effective evaluation. Let L : = ⌈ log 2 n ⌉ L := \lceil\log_2 n\rceil L := ⌈ log 2 n ⌉ . For rational μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) and integer p ≥ 1 p \ge1 p ≥ 1 , a deterministic algorithm computes a dyadic number H ^ p ( μ ) ∈ [ 0 , L ] \widehat{\mathbf{H}}_p(\mu) \in[0,L] H p ( μ ) ∈ [ 0 , L ] , with denominator 2 p 2^p 2 p , s.t.
∣ H ^ p ( μ ) − H ( μ ) ∣ ≤ 2 − p , |\widehat{\mathbf{H}}_p(\mu)-\mathbf{H}(\mu)| \le2^{-p}, ∣ H p ( μ ) − H ( μ ) ∣ ≤ 2 − p ,
in time polynomial in size ( μ ) + p \operatorname{size}(\mu)+p size ( μ ) + p , where size ( μ ) \operatorname{size}(\mu) size ( μ ) is the total bit length of the coordinates of μ \mu μ written as reduced fractions.
:::
Proof. Part 1: Continuity. Couple X ∼ μ X \sim\mu X ∼ μ and Y ∼ ν Y \sim\nu Y ∼ ν on a common probability space so as to agree as often as possible, as follows. Since min ( μ ( s ) , ν ( s ) ) = 1 2 ( μ ( s ) + ν ( s ) − ∣ μ ( s ) − ν ( s ) ∣ ) \min(\mu(s),\nu(s)) = \frac{1}{2}(\mu(s)+\nu(s)-|\mu(s)-\nu(s)|) min ( μ ( s ) , ν ( s )) = 2 1 ( μ ( s ) + ν ( s ) − ∣ μ ( s ) − ν ( s ) ∣ ) for each s s s , summing over s ∈ S s \in S s ∈ S gives
∑ s ∈ S min ( μ ( s ) , ν ( s ) ) = 1 2 ( 1 + 1 ) − 1 2 ∑ s ∈ S ∣ μ ( s ) − ν ( s ) ∣ = 1 − d T V ( μ , ν ) , \sum_{s\in S}\min(\mu(s),\nu(s))=\frac{1}{2}(1+1)-\frac{1}{2}\sum_{s\in S}|\mu(s)-\nu(s)|=1-d_{\mathrm{TV}}(\mu,\nu), s ∈ S ∑ min ( μ ( s ) , ν ( s )) = 2 1 ( 1 + 1 ) − 2 1 s ∈ S ∑ ∣ μ ( s ) − ν ( s ) ∣ = 1 − d TV ( μ , ν ) ,
using ∑ s μ ( s ) = ∑ s ν ( s ) = 1 \sum_s\mu(s)=\sum_s\nu(s)=1 ∑ s μ ( s ) = ∑ s ν ( s ) = 1 . With probability 1 − d T V ( μ , ν ) 1-d_{\mathrm{TV}}(\mu,\nu) 1 − d TV ( μ , ν ) , draw a single state s s s with probability min ( μ ( s ) , ν ( s ) ) / ( 1 − d T V ( μ , ν ) ) \min(\mu(s),\nu(s))/(1-d_{\mathrm{TV}}(\mu,\nu)) min ( μ ( s ) , ν ( s )) / ( 1 − d TV ( μ , ν )) and set X = Y = s X=Y=s X = Y = s ; otherwise, with the complementary probability d T V ( μ , ν ) d_{\mathrm{TV}}(\mu,\nu) d TV ( μ , ν ) , draw X X X and Y Y Y independently from the normalized leftover distributions ( μ ( s ) − min ( μ ( s ) , ν ( s ) ) ) / d T V ( μ , ν ) (\mu(s)-\min(\mu(s),\nu(s)))/d_{\mathrm{TV}}(\mu,\nu) ( μ ( s ) − min ( μ ( s ) , ν ( s ))) / d TV ( μ , ν ) and ( ν ( s ) − min ( μ ( s ) , ν ( s ) ) ) / d T V ( μ , ν ) (\nu(s)-\min(\mu(s),\nu(s)))/d_{\mathrm{TV}}(\mu,\nu) ( ν ( s ) − min ( μ ( s ) , ν ( s ))) / d TV ( μ , ν ) respectively. These two leftover distributions have disjoint supports, wherever μ ( s ) > ν ( s ) \mu(s)>\nu(s) μ ( s ) > ν ( s ) the second is 0 0 0 at s s s , and vice versa, so X ≠ Y X\ne Y X = Y almost surely in this second branch. Under this construction X X X has marginal law μ \mu μ and Y Y Y has marginal law ν \nu ν (each coordinate recovers μ ( s ) \mu(s) μ ( s ) , respectively ν ( s ) \nu(s) ν ( s ) , as the weighted sum of its shared and leftover parts), and X = Y X=Y X = Y occurs exactly in the first branch, so q = Pr [ X ≠ Y ] = d T V ( μ , ν ) q=\Pr[X\ne Y]=d_{\mathrm{TV}}(\mu,\nu) q = Pr [ X = Y ] = d TV ( μ , ν ) . Let E = 1 { X ≠ Y } E = 1_{\{X \ne Y\}} E = 1 { X = Y } be the indicator of this coupling’s disagreement event, a single Bernoulli(q q q ) random variable. Extending H ( ⋅ ) \mathbf{H}(\cdot) H ( ⋅ ) to random variables in the usual way, so that H ( X ∣ Y ) \mathbf{H}(X \mid Y) H ( X ∣ Y ) denotes conditional entropy: since E E E is a deterministic function of ( X , Y ) (X,Y) ( X , Y ) ,
H ( X , E ∣ Y ) = H ( X ∣ Y ) + H ( E ∣ X , Y ) = H ( X ∣ Y ) , \mathbf{H}(X,E \mid Y) = \mathbf{H}(X \mid Y) + \mathbf{H}(E \mid X,Y) = \mathbf{H}(X \mid Y), H ( X , E ∣ Y ) = H ( X ∣ Y ) + H ( E ∣ X , Y ) = H ( X ∣ Y ) ,
while the chain rule expands the same quantity as H ( X , E ∣ Y ) = H ( E ∣ Y ) + H ( X ∣ Y , E ) \mathbf{H}(X,E \mid Y) = \mathbf{H}(E \mid Y) + \mathbf{H}(X \mid Y,E) H ( X , E ∣ Y ) = H ( E ∣ Y ) + H ( X ∣ Y , E ) . Equating the two and using H ( E ∣ Y ) ≤ H ( E ) \mathbf{H}(E \mid Y) \le\mathbf{H}(E) H ( E ∣ Y ) ≤ H ( E ) (conditioning cannot increase entropy) gives
H ( X ∣ Y ) ≤ H ( E ) + H ( X ∣ Y , E ) . \mathbf{H}(X \mid Y) \le\mathbf{H}(E) + \mathbf{H}(X \mid Y,E). H ( X ∣ Y ) ≤ H ( E ) + H ( X ∣ Y , E ) .
Split the remaining term on the value of E E E : if E = 0 E = 0 E = 0 then X = Y X = Y X = Y , so X X X is determined once Y Y Y and E = 0 E = 0 E = 0 are known, contributing 0; if E = 1 E = 1 E = 1 then X X X is confined to the n − 1 n-1 n − 1 states other than Y Y Y ’s value, so it contributes at most log 2 ( n − 1 ) \log_2(n-1) log 2 ( n − 1 ) (the max-entropy bound on a support of size n − 1 n-1 n − 1 ). Averaging over E E E with weights 1 − q , q 1-q,q 1 − q , q gives H ( X ∣ Y , E ) ≤ q log 2 ( n − 1 ) \mathbf{H}(X \mid Y,E) \le q \log_2(n-1) H ( X ∣ Y , E ) ≤ q log 2 ( n − 1 ) , and since E E E is Bernoulli(q q q ), H ( E ) = h ( q ) \mathbf{H}(E) = h(q) H ( E ) = h ( q ) . Together,
H ( X ∣ Y ) ≤ h ( q ) + q log 2 ( n − 1 ) = ω n ( q ) . \mathbf{H}(X \mid Y) \le h(q) + q \log_2(n-1) = \omega_n(q). H ( X ∣ Y ) ≤ h ( q ) + q log 2 ( n − 1 ) = ω n ( q ) .
Thus H ( X ) − H ( Y ) ≤ H ( X ∣ Y ) ≤ ω n ( q ) \mathbf{H}(X) - \mathbf{H}(Y) \le\mathbf{H}(X \mid Y) \le\omega_n(q) H ( X ) − H ( Y ) ≤ H ( X ∣ Y ) ≤ ω n ( q ) ; interchanging X , Y X,Y X , Y proves the absolute-value bound. On ( 0 , 1 − 1 / n ) (0,1-1/n) ( 0 , 1 − 1/ n ) , ω n ′ ( r ) = log 2 ( ( n − 1 ) ( 1 − r ) / r ) ≥ 0 \omega_n'(r) = \log_2((n-1)(1-r)/r) \ge0 ω n ′ ( r ) = log 2 (( n − 1 ) ( 1 − r ) / r ) ≥ 0 , so q ≤ r q \le r q ≤ r gives the stated modulus. The elementary binary-entropy bound h ( r ) ≤ 2 r ( 1 − r ) ≤ 2 r h(r) \le2\sqrt{r(1-r)} \le2\sqrt{r} h ( r ) ≤ 2 r ( 1 − r ) ≤ 2 r is a standard bound; for instance, h ( r ) ≤ r log 2 ( e / r ) h(r) \le r\log_2(e/r) h ( r ) ≤ r log 2 ( e / r ) (as ( 1 − r ) ln 1 1 − r ≤ r (1-r)\ln\frac{1}{1-r} \le r ( 1 − r ) ln 1 − r 1 ≤ r ) and r log 2 ( e / r ) ≤ 2 r r\log_2(e/r) \le2\sqrt{r} r log 2 ( e / r ) ≤ 2 r on ( 0 , 1 ] (0,1] ( 0 , 1 ] .
Computing H ( μ ) \mathbf{H}(\mu) H ( μ ) exactly is not an option: for most rational μ \mu μ the value involves log 2 \log_2 log 2 of a rational number, which is irrational. Any algorithm that later needs to compare H ( μ ) \mathbf{H}(\mu) H ( μ ) against a threshold, or feed it into a larger computation, can only work with rationals, so what is actually needed is a way to compute H ( μ ) \mathbf{H}(\mu) H ( μ ) to any desired number of bits, with an explicit error bound, in time that stays polynomial as that number of bits grows. The natural tool is a series expansion for log 2 \log_2 log 2 , but the standard one converges too slowly (or not at all) unless its argument is small, which is what step 2 below arranges. Part 2 : Effective dyadic evaluation . On input ( μ , p ) (\mu,p) ( μ , p ) , compute H ^ p ( μ ) \widehat{\mathbf{H}}_p(\mu) H p ( μ ) as follows.
Zero coordinates. For each s s s with μ ( s ) = 0 \mu(s)=0 μ ( s ) = 0 , contribute 0 (by convention 0 log 2 0 = 0 0\log_2 0=0 0 log 2 0 = 0 ).
Per-coordinate logarithm. For each s s s with q = μ ( s ) > 0 q=\mu(s)>0 q = μ ( s ) > 0 , write q = 2 − j u q=2^{-j}u q = 2 − j u with 1 ≤ u < 2 1 \le u < 2 1 ≤ u < 2 , where j j j ’s size is bounded by the encoding of q q q . Compute
ln u = 2 ∑ i = 0 m z 2 i + 1 2 i + 1 + R m , z = u − 1 u + 1 ∈ [ 0 , 1 / 3 ] , ∣ R m ∣ ≤ 2 z 2 m + 3 ( 2 m + 3 ) ( 1 − z 2 ) , \ln u = 2\sum_{i=0}^{m}\frac{z^{2i+1}}{2i+1} + R_m,\qquad z=\frac{u-1}{u+1}\in[0,1/3],\qquad|R_m|\le\frac{2z^{2m+3}}{(2m+3)(1-z^2)}, ln u = 2 i = 0 ∑ m 2 i + 1 z 2 i + 1 + R m , z = u + 1 u − 1 ∈ [ 0 , 1/3 ] , ∣ R m ∣ ≤ ( 2 m + 3 ) ( 1 − z 2 ) 2 z 2 m + 3 ,
with m = O ( p + size ( q ) + log n ) m=O(p+\operatorname{size}(q)+\log n) m = O ( p + size ( q ) + log n ) terms; the same expansion at u = 2 u=2 u = 2 gives ln 2 \ln2 ln 2 , bounded away from zero. Since log 2 q = − j + ln u / ln 2 \log_2 q=-j+\ln u/\ln2 log 2 q = − j + ln u / ln 2 , this yields − q log 2 q -q\log_2 q − q log 2 q to within 2 − p − 1 / n 2^{-p-1}/n 2 − p − 1 / n , using only polynomially many rational operations on rationals of polynomial bit length.
Summation. Add the (at most n n n ) per-coordinate approximations; the total evaluation error is at most n ⋅ 2 − p − 1 / n = 2 − p − 1 n\cdot2^{-p-1}/n=2^{-p-1} n ⋅ 2 − p − 1 / n = 2 − p − 1 .
Rounding. Round the sum to the nearest multiple of 2 − p 2^{-p} 2 − p , adding at most 2 − p − 1 2^{-p-1} 2 − p − 1 more error.
Clipping. Clip the result to [ 0 , L ] [0,L] [ 0 , L ] ; since H ( μ ) ∈ [ 0 , L ] \mathbf{H}(\mu)\in[0,L] H ( μ ) ∈ [ 0 , L ] , this cannot increase the distance to H ( μ ) \mathbf{H}(\mu) H ( μ ) .
Steps 3–4 give total error at most 2 − p − 1 + 2 − p − 1 = 2 − p 2^{-p-1}+2^{-p-1}=2^{-p} 2 − p − 1 + 2 − p − 1 = 2 − p , and step 5 does not increase it, so the output satisfies the claimed bound. Every step uses only polynomially many rational operations on rationals of polynomial bit length, so the whole procedure runs in time polynomial in size ( μ ) + p \operatorname{size}(\mu)+p size ( μ ) + p . This proves the claimed algorithm.
Theorem 7 (Gap bound under averaged-distribution semantics). Fix rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 . Then, for both the upper and the lower value, GapVal a v , ε 0 ⋄ ∈ P S P A C E \operatorname{GapVal}^{\diamond}_{\mathrm{av},\varepsilon_0} \in\mathrm{PSPACE} GapVal av , ε 0 ⋄ ∈ PSPACE for ⋄ ∈ { + , − } \diamond\in\{+,-\} ⋄ ∈ { + , − } .
Proof. Let ℓ \ell ℓ denote the total input encoding length (the arena, the rational transition probabilities, the initial distribution, and the threshold v v v , all written in binary), and recall n = ∣ S ∣ n = |S| n = ∣ S ∣ , A = max s ∣ A ( s ) ∣ A = \max_{s} |A(s)| A = max s ∣ A ( s ) ∣ , and e = min { ε 0 , 1 } e = \min\{\varepsilon_0,1\} e = min { ε 0 , 1 } . Put L = ⌈ log 2 n ⌉ L = \lceil\log_2 n\rceil L = ⌈ log 2 n ⌉ , the maximum possible entropy on n n n states. If n = 1 n = 1 n = 1 , entropy is identically zero on the single state, so V a v + ( μ 0 ) = 0 V^{+}_{\mathrm{av}}(\mu_0) = 0 V av + ( μ 0 ) = 0 regardless of either player’s strategy; comparing 0 0 0 against v v v decides the promise directly, with no further construction needed. Henceforth assume n ≥ 2 n \ge2 n ≥ 2 . The proof has two parts. Steps 1-2 reduce the infinite-horizon value to a finite, memoryless recursion on the current distribution. Step 3 then gives a PSPACE algorithm deciding the promise gap directly, by evaluating that recursion on a finite grid of actions via an alternating Turing machine.
Step 1: uniform horizon truncation. The value V a v + V^{+}_{\mathrm{av}} V av + is defined by summing over infinitely many rounds, but any algorithm can only refer to finitely many of them. So before anything else, the infinite horizon must be cut down to some finite T T T , with a guarantee that the discarded tail is too small to affect the promise-gap decision. For every strategy pair the omitted tail satisfies
0 ≤ ( 1 − γ ) ∑ t = T ∞ γ t H ( μ t ) ≤ γ T log 2 n ≤ γ T L . 0 \le(1-\gamma)\sum_{t=T}^{\infty}\gamma^t\mathrm{H}(\mu_t) \le\gamma^T\log_2 n \le\gamma^T L. 0 ≤ ( 1 − γ ) t = T ∑ ∞ γ t H ( μ t ) ≤ γ T log 2 n ≤ γ T L .
Choose the least positive integer T T T with γ T L ≤ e / 4 \gamma^T L \le e/4 γ T L ≤ e /4 . It can be found by exact rational comparisons, and T = O ( 1 + log ( n / e ) ) T = O(1+\log(n/e)) T = O ( 1 + log ( n / e )) . Both γ \gamma γ and e = min { ε 0 , 1 } e = \min\{\varepsilon_0,1\} e = min { ε 0 , 1 } are themselves fixed constants, not part of the varying instance; the 1 / e 1/e 1/ e dependence inside the logarithm is displayed explicitly to show how T T T scales for other fixed choices of the gap ε 0 \varepsilon_0 ε 0 . There are T T T rewards, at times 0 , … , T − 1 0,\ldots,T-1 0 , … , T − 1 , but only T − 1 T-1 T − 1 relevant action pairs ( x t , y t ) (x_t,y_t) ( x t , y t ) (one per transition μ t → μ t + 1 \mu_t \to\mu_{t+1} μ t → μ t + 1 ), since the last rewarded distribution μ T − 1 \mu_{T-1} μ T − 1 needs no outgoing action.
Step 2: finite-horizon values and public histories. For r > 0 r > 0 r > 0 let U r ( μ ) U_r(\mu) U r ( μ ) be the upper value, over strategies whose action at time t t t may depend on the whole observed distribution history ( μ 0 , … , μ t ) (\mu_0,\ldots,\mu_t) ( μ 0 , … , μ t ) , of the truncated payoff ( 1 − γ ) ∑ t = 0 r − 1 γ t H ( μ t ) (1-\gamma)\sum_{t=0}^{r-1}\gamma^t\mathrm{H}(\mu_t) ( 1 − γ ) ∑ t = 0 r − 1 γ t H ( μ t ) started at μ \mu μ . This is the quantity V a v , r + V^{+}_{\mathrm{av},r} V av , r + of the proof of Theorem 1 (Section C), where the following facts are proved, with the same history-dependent strategy spaces. Each U r U_r U r is continuous, the minimiser has a Borel Markov policy attaining it from every distribution, and
U 0 = 0 , U r + 1 ( μ ) = ( 1 − γ ) H ( μ ) + γ min x ∈ X max y ∈ Y U r ( x , y , μ ) , (45) U_0 = 0,\qquad U_{r+1}(\mu) = (1-\gamma)\mathrm{H}(\mu) + \gamma\min_{x\in\mathcal{X}}\max_{y\in\mathcal{Y}} U_r(x,y,\mu), \tag*{(45)} U 0 = 0 , U r + 1 ( μ ) = ( 1 − γ ) H ( μ ) + γ x ∈ X min y ∈ Y max U r ( x , y , μ ) , (45)
that is, U r = ( T + ) r 0 U_r = (\mathcal{T}^{+})^r0 U r = ( T + ) r 0 . The same proof bounds the truncation error uniformly in the strategy pair: the omitted tail of the payoff is at most γ T L ′ \gamma^T L' γ T L ′ with L ′ = log 2 ∣ S ∣ ≤ L L' = \log_2 |S| \le L L ′ = log 2 ∣ S ∣ ≤ L . Taking either ordered optimisation of this pointwise bound, and using γ T L ≤ e / 4 \gamma^T L \le e/4 γ T L ≤ e /4 from Step 1, gives
0 ≤ V a v + ( μ ) − U T ( μ ) ≤ e / 4. (46) 0 \le V^{+}_{\mathrm{av}}(\mu) - U_T(\mu) \le e/4. \tag*{(46)} 0 ≤ V av + ( μ ) − U T ( μ ) ≤ e /4. (46)
Step 3 below uses only (37), continuity of U r U_r U r , and (38).
Step 3: a finite-grid PSPACE algorithm. Step 2 characterised U T ( μ ) U_T(\mu) U T ( μ ) via a recursion over x ∈ X , y ∈ Y x \in\mathcal{X}, y \in\mathcal{Y} x ∈ X , y ∈ Y , continuous, infinite sets of mixed actions. A Turing machine cannot search over a continuum, so before this recursion can become an algorithm, the action spaces must be replaced by a finite grid, and the entropy reward must be evaluated to finite precision rather than exactly. Both replacements introduce error, which has to be tracked and kept within budget alongside Step 1’s truncation error. The construction below does this in three stages: first, bound how much rounding an action to the nearest grid point can perturb the resulting distribution and, in turn, the value U r U_r U r ; second, choose the grid fine enough that this perturbation costs at most ϵ / 8 \epsilon/8 ϵ /8 of the total budget; third, build an alternating Turing machine that evaluates the grid-restricted recursion exactly, guessing the minimiser’s action existentially and the maximiser’s universally at each round, and confirm the whole computation fits in polynomial space via AP = PSPACE.
Rounding actions to a grid. This step produces the basic tool everything else relies on: a way to replace an arbitrary mixed action by one from a finite grid, with an explicit, quantified bound on how much the resulting distribution can change as a result. Let X Q , Y Q \mathcal{X}_Q, \mathcal{Y}_Q X Q , Y Q be the products of statewise action simplices with coordinates in { 0 , 1 / Q , … , 1 } \{0,1/Q,\ldots,1\} { 0 , 1/ Q , … , 1 } : grids of mixed actions with denominator Q Q Q . Any mixed action x ∈ X x \in\mathcal{X} x ∈ X can be rounded to a grid point x Q x^Q x Q as follows: at each state s ∈ S min s \in S_{\min} s ∈ S m i n with k = ∣ A ( s ) ∣ ≤ A k = |A(s)| \le A k = ∣ A ( s ) ∣ ≤ A actions, let q a = Q × x ( s ) ( a ) q_a = Q \times x(s)(a) q a = Q × x ( s ) ( a ) for each a ∈ A ( s ) a \in A(s) a ∈ A ( s ) , so ∑ a q a = Q \sum_a q_a = Q ∑ a q a = Q . Set x Q ( s ) ( a ) = ⌊ q a ⌋ / Q x^Q(s)(a) = \lfloor q_a\rfloor/Q x Q ( s ) ( a ) = ⌊ q a ⌋ / Q initially; since the fractional parts q a − ⌊ q a ⌋ q_a - \lfloor q_a\rfloor q a − ⌊ q a ⌋ sum to an integer R ∈ [ 0 , k − 1 ] R \in[0,k-1] R ∈ [ 0 , k − 1 ] , add 1 / Q 1/Q 1/ Q to the R R R coordinates with the largest fractional parts, restoring the total to 1 1 1 . Each single coordinate changes by less than 1 / Q 1/Q 1/ Q under this procedure (rounding down or up moves q a / Q q_a/Q q a / Q by less than 1 / Q 1/Q 1/ Q ), so summing over at most A A A coordinates,
∥ x ( s ) − x Q ( s ) ∥ 1 = ∑ a ∈ A ( s ) ∣ x ( s ) ( a ) − x Q ( s ) ( a ) ∣ ≤ ∣ A ( s ) ∣ Q ≤ A Q . \lVert x(s)-x^Q(s)\rVert_1=\sum_{a\in A(s)}|x(s)(a)-x^Q(s)(a)|\le\frac{|A(s)|}{Q}\le\frac{A}{Q}. ∥ x ( s ) − x Q ( s ) ∥ 1 = a ∈ A ( s ) ∑ ∣ x ( s ) ( a ) − x Q ( s ) ( a ) ∣ ≤ Q ∣ A ( s ) ∣ ≤ Q A .
To convert this into a bound on the successor distribution, write out the difference coordinatewise using
Γ x , y ( μ ) ( s ′ ) = ∑ s ∈ S min μ ( s ) ∑ a x ( s ) ( a ) Δ ( s , a , s ′ ) + ∑ s ∈ S max μ ( s ) ∑ a y ( s ) ( a ) Δ ( s , a , s ′ ) \Gamma_{x,y}(\mu)(s')=\sum_{s\in S_{\min}}\mu(s)\sum_a x(s)(a)\Delta(s,a,s')+\sum_{s\in S_{\max}}\mu(s)\sum_a y(s)(a)\Delta(s,a,s') Γ x , y ( μ ) ( s ′ ) = s ∈ S m i n ∑ μ ( s ) a ∑ x ( s ) ( a ) Δ ( s , a , s ′ ) + s ∈ S m a x ∑ μ ( s ) a ∑ y ( s ) ( a ) Δ ( s , a , s ′ )
(only the Min-controlled part depends on x x x , so the Max-controlled part cancels):
Γ x , y ( μ ) ( s ′ ) − Γ x Q , y ( μ ) ( s ′ ) = ∑ s ∈ S min μ ( s ) ∑ a [ x ( s ) ( a ) − x Q ( s ) ( a ) ] Δ ( s , a , s ′ ) . \Gamma_{x,y}(\mu)(s')-\Gamma_{x^Q,y}(\mu)(s')=\sum_{s\in S_{\min}}\mu(s)\sum_a\left[x(s)(a)-x^Q(s)(a)\right]\Delta(s,a,s'). Γ x , y ( μ ) ( s ′ ) − Γ x Q , y ( μ ) ( s ′ ) = s ∈ S m i n ∑ μ ( s ) a ∑ [ x ( s ) ( a ) − x Q ( s ) ( a ) ] Δ ( s , a , s ′ ) .
Summing absolute values over s ′ s' s ′ and using ∑ s ′ Δ ( s , a , s ′ ) = 1 \sum_{s'}\Delta(s,a,s')=1 ∑ s ′ Δ ( s , a , s ′ ) = 1 ,
∥ Γ x , y ( μ ) − Γ x Q , y ( μ ) ∥ 1 ≤ ∑ s ∈ S min μ ( s ) ∑ a ∣ x ( s ) ( a ) − x Q ( s ) ( a ) ∣ = ∑ s ∈ S min μ ( s ) ∥ x ( s ) − x Q ( s ) ∥ 1 . \lVert\Gamma_{x,y}(\mu)-\Gamma_{x^Q,y}(\mu)\rVert_1\le\sum_{s\in S_{\min}}\mu(s)\sum_a|x(s)(a)-x^Q(s)(a)|=\sum_{s\in S_{\min}}\mu(s)\lVert x(s)-x^Q(s)\rVert_1. ∥ Γ x , y ( μ ) − Γ x Q , y ( μ ) ∥ 1 ≤ s ∈ S m i n ∑ μ ( s ) a ∑ ∣ x ( s ) ( a ) − x Q ( s ) ( a ) ∣ = s ∈ S m i n ∑ μ ( s ) ∥ x ( s ) − x Q ( s ) ∥ 1 .
Dividing by 2 and using the coordinate bound above, with ∑ s ∈ S min μ ( s ) ≤ 1 \sum_{s\in S_{\min}}\mu(s)\le1 ∑ s ∈ S m i n μ ( s ) ≤ 1 ,
d T V ( Γ x , y ( μ ) , Γ x Q , y ( μ ) ) ≤ 1 2 ∑ s ∈ S min μ ( s ) ∥ x ( s ) − x Q ( s ) ∥ 1 ≤ A 2 Q . (47) d_{\mathrm{TV}}(\Gamma_{x,y}(\mu),\Gamma_{x^Q,y}(\mu))\le\frac{1}{2}\sum_{s\in S_{\min}}\mu(s)\lVert x(s)-x^Q(s)\rVert_1\le\frac{A}{2Q}. \tag*{(47)} d TV ( Γ x , y ( μ ) , Γ x Q , y ( μ )) ≤ 2 1 s ∈ S m i n ∑ μ ( s ) ∥ x ( s ) − x Q ( s ) ∥ 1 ≤ 2 Q A . (47)
The same holds for rounding y y y .
From rounding error to value error. The previous step bounded how much rounding perturbs a distribution; this step upgrades that into a bound on how much the value U r U_r U r itself can shift as a result, and uses it to compare the exact one-step value against its grid-restricted counterpart.
For fixed x , y x,y x , y , the successor map is nonexpansive in total variation: writing Γ x , y ( μ ) − Γ x , y ( ν ) = ( μ − ν ) K \Gamma_{x,y}(\mu)-\Gamma_{x,y}(\nu)=(\mu-\nu)K Γ x , y ( μ ) − Γ x , y ( ν ) = ( μ − ν ) K for the stochastic matrix K ( s , s ′ ) = ∑ a π x , y ( s ) ( a ) Δ ( s , a , s ′ ) K(s,s')=\sum_a\pi_{x,y}(s)(a)\Delta(s,a,s') K ( s , s ′ ) = ∑ a π x , y ( s ) ( a ) Δ ( s , a , s ′ ) whose rows sum to 1 1 1 ,
∥ Γ x , y ( μ ) − Γ x , y ( ν ) ∥ 1 = ∑ s ′ ∣ ∑ s ( μ ( s ) − ν ( s ) ) K ( s , s ′ ) ∣ ≤ ∑ s ∣ μ ( s ) − ν ( s ) ∣ ∑ s ′ K ( s , s ′ ) = ∥ μ − ν ∥ 1 , \lVert\Gamma_{x,y}(\mu)-\Gamma_{x,y}(\nu)\rVert_1=\sum_{s'}\left|\sum_s(\mu(s)-\nu(s))K(s,s')\right|\le\sum_s|\mu(s)-\nu(s)|\sum_{s'}K(s,s')=\lVert\mu-\nu\rVert_1, ∥ Γ x , y ( μ ) − Γ x , y ( ν ) ∥ 1 = s ′ ∑ s ∑ ( μ ( s ) − ν ( s )) K ( s , s ′ ) ≤ s ∑ ∣ μ ( s ) − ν ( s ) ∣ s ′ ∑ K ( s , s ′ ) = ∥ μ − ν ∥ 1 ,
so d T V ( Υ x , y ( μ ) , Υ x , y ( ν ) ) ≤ d T V ( μ , ν ) d_{\mathrm{TV}}(\Upsilon_{x,y}(\mu),\Upsilon_{x,y}(\nu)) \le d_{\mathrm{TV}}(\mu,\nu) d TV ( Υ x , y ( μ ) , Υ x , y ( ν )) ≤ d TV ( μ , ν ) : applying the same actions to two distributions cannot push their successors further apart than the originals were. We prove, for 0 ≤ r 0 ≤ 1 − 1 / n 0 \le r_0 \le1 - 1/n 0 ≤ r 0 ≤ 1 − 1/ n and every r ≥ 0 r \ge0 r ≥ 0 ,
d T V ( μ , ν ) ≤ r 0 ⟹ ∣ U r ( μ ) − U r ( ν ) ∣ ≤ ω n ( r 0 ) . (48) d_{\mathrm{TV}}(\mu,\nu) \le r_0 \Longrightarrow\lvert U_r(\mu)-U_r(\nu)\rvert\le\omega_n(r_0). \tag*{(48)} d TV ( μ , ν ) ≤ r 0 ⟹ ∣ U r ( μ ) − U r ( ν )∣ ≤ ω n ( r 0 ) . (48)
For r = 0 r=0 r = 0 , U 0 ≡ 0 U_0 \equiv0 U 0 ≡ 0 , so ∣ U 0 ( μ ) − U 0 ( ν ) ∣ = 0 ≤ ω n ( r 0 ) \lvert U_0(\mu)-U_0(\nu)\rvert=0 \le\omega_n(r_0) ∣ U 0 ( μ ) − U 0 ( ν )∣ = 0 ≤ ω n ( r 0 ) , since ω n ≥ 0 \omega_n \ge0 ω n ≥ 0 ; this is the base case. Assume (48) for r r r and suppose d T V ( μ , ν ) ≤ r 0 d_{\mathrm{TV}}(\mu,\nu) \le r_0 d TV ( μ , ν ) ≤ r 0 . By Part 1 of Lemma 6, the entropy term contributes ( 1 − γ ) ∣ H ( μ ) − H ( ν ) ∣ ≤ ( 1 − γ ) ω n ( r 0 ) (1-\gamma)\lvert H(\mu)-H(\nu)\rvert\le(1-\gamma)\omega_n(r_0) ( 1 − γ ) ∣ H ( μ ) − H ( ν )∣ ≤ ( 1 − γ ) ω n ( r 0 ) . For the continuation term, nonexpansiveness gives d T V ( Υ x , y ( μ ) , Υ x , y ( ν ) ) ≤ r 0 d_{\mathrm{TV}}(\Upsilon_{x,y}(\mu),\Upsilon_{x,y}(\nu)) \le r_0 d TV ( Υ x , y ( μ ) , Υ x , y ( ν )) ≤ r 0 for every fixed ( x , y ) (x,y) ( x , y ) , so the inductive hypothesis applies to that pair of successors: ∣ U r ( Υ x , y ( μ ) ) − U r ( Υ x , y ( ν ) ) ∣ ≤ ω n ( r 0 ) \lvert U_r(\Upsilon_{x,y}(\mu))-U_r(\Upsilon_{x,y}(\nu))\rvert\le\omega_n(r_0) ∣ U r ( Υ x , y ( μ )) − U r ( Υ x , y ( ν ))∣ ≤ ω n ( r 0 ) for every x , y x,y x , y . Adding a constant bound to a function does not change which point maximizes or minimizes it, only shifts the extremal value by that constant, so taking max y \max_y max y and then min x \min_x min x of both sides (and the symmetric argument swapping μ , ν \mu,\nu μ , ν ) gives
∣ min x max y U r ( Υ x , y ( μ ) ) − min x max y U r ( Υ x , y ( ν ) ) ∣ ≤ ω n ( r 0 ) . \left\lvert\min_x \max_y U_r(\Upsilon_{x,y}(\mu))-\min_x \max_y U_r(\Upsilon_{x,y}(\nu))\right\rvert\le\omega_n(r_0). x min y max U r ( Υ x , y ( μ )) − x min y max U r ( Υ x , y ( ν )) ≤ ω n ( r 0 ) .
Combining the two terms of (45) ,
∣ U r + 1 ( μ ) − U r + 1 ( ν ) ∣ ≤ ( 1 − γ ) ω n ( r 0 ) + γ ω n ( r 0 ) = ω n ( r 0 ) , \lvert U_{r+1}(\mu)-U_{r+1}(\nu)\rvert\le(1-\gamma)\omega_n(r_0)+\gamma\omega_n(r_0)=\omega_n(r_0), ∣ U r + 1 ( μ ) − U r + 1 ( ν )∣ ≤ ( 1 − γ ) ω n ( r 0 ) + γ ω n ( r 0 ) = ω n ( r 0 ) ,
completing the induction. This is what lets grid-rounding error be converted into value error. Fix μ \mu μ and write F ( x , y ) = U r ( Υ x , y ( μ ) ) F(x,y)=U_r(\Upsilon_{x,y}(\mu)) F ( x , y ) = U r ( Υ x , y ( μ )) for the one-step continuation value as a function of the two players’ actions. Let M = min x max y F M=\min_x\max_y F M = min x max y F be the exact value, M X = min x ∈ X Q max y F M_X=\min_{x\in\mathcal{X}_Q}\max_y F M X = min x ∈ X Q max y F the value once the minimiser alone is restricted to the grid, and M Q = min x ∈ X Q max y ∈ Y Q F M_Q=\min_{x\in\mathcal{X}_Q}\max_{y\in\mathcal{Y}_Q}F M Q = min x ∈ X Q max y ∈ Y Q F the value once both players are restricted. Since X Q ⊆ X \mathcal{X}_Q \subseteq\mathcal{X} X Q ⊆ X , minimising over the smaller set X Q \mathcal{X}_Q X Q cannot decrease the minimum, so M ≤ M X M \le M_X M ≤ M X . For the other direction, let x ∗ x^* x ∗ achieve M = max y F ( x ∗ , y ) M=\max_y F(x^*,y) M = max y F ( x ∗ , y ) and round it to x Q ∗ ∈ X Q x_Q^*\in\mathcal{X}_Q x Q ∗ ∈ X Q ; by (47) with η = A / ( 2 Q ) ≤ 1 − 1 / n \eta=A/(2Q)\le1-1/n η = A / ( 2 Q ) ≤ 1 − 1/ n and (48) , F ( x Q ∗ , y ) ≤ F ( x ∗ , y ) + ω n ( η ) F(x_Q^*,y)\le F(x^*,y)+\omega_n(\eta) F ( x Q ∗ , y ) ≤ F ( x ∗ , y ) + ω n ( η ) for every y y y , so max y F ( x Q ∗ , y ) ≤ M + ω n ( η ) \max_y F(x_Q^*,y)\le M+\omega_n(\eta) max y F ( x Q ∗ , y ) ≤ M + ω n ( η ) ; since x Q ∗ ∈ X Q x_Q^*\in\mathcal{X}_Q x Q ∗ ∈ X Q , M X ≤ max y F ( x Q ∗ , y ) ≤ M + ω n ( η ) M_X\le\max_y F(x_Q^*,y)\le M+\omega_n(\eta) M X ≤ max y F ( x Q ∗ , y ) ≤ M + ω n ( η ) . Together,
M ≤ M X ≤ M + ω n ( η ) . M \le M_X \le M+\omega_n(\eta). M ≤ M X ≤ M + ω n ( η ) .
The same argument, applied to rounding the maximiser’s action instead (restricting Y \mathcal{Y} Y to Y Q \mathcal{Y}_Q Y Q can only decrease a maximum, giving M Q ≤ M X M_Q\le M_X M Q ≤ M X ; rounding the maximiser’s best response at the minimiser’s grid-optimal x x x gives the other direction), gives
M Q ≤ M X ≤ M Q + ω n ( η ) . M_Q \le M_X \le M_Q+\omega_n(\eta). M Q ≤ M X ≤ M Q + ω n ( η ) .
By the triangle inequality, ∣ M − M Q ∣ ≤ ∣ M − M X ∣ + ∣ M X − M Q ∣ ≤ ω n ( η ) + ω n ( η ) = 2 ω n ( η ) \lvert M-M_Q\rvert\le\lvert M-M_X\rvert+\lvert M_X-M_Q\rvert\le\omega_n(\eta)+\omega_n(\eta)=2\omega_n(\eta) ∣ M − M Q ∣ ≤ ∣ M − M X ∣ + ∣ M X − M Q ∣ ≤ ω n ( η ) + ω n ( η ) = 2 ω n ( η ) : restricting both players to the grid changes the one-step value by at most 2 ω n ( η ) 2\omega_n(\eta) 2 ω n ( η ) .
Choosing the grid. The previous step bounded the value error introduced by the grid in a single round; this step accumulates that error across all T T T rounds of the recursion and picks concrete constants so the total stays within the e / 8 e/8 e /8 budget. Define U r Q U_r^Q U r Q by the same recursion, with both action sets restricted to the grid throughout. Since each round’s discrepancy from the exact value is at most 2 ω n ( η ) 2\omega_n(\eta) 2 ω n ( η ) , and the recursion discounts the previous round’s error by γ \gamma γ , the total discrepancy after r r r rounds, D r = ∥ U r − U r Q ∥ ∞ D_r=\lVert U_r-U_r^Q\rVert_\infty D r = ∥ U r − U r Q ∥ ∞ , satisfies D r + 1 ≤ γ D r + 2 γ ω n ( η ) D_{r+1}\le\gamma D_r+2\gamma\omega_n(\eta) D r + 1 ≤ γ D r + 2 γ ω n ( η ) and D 0 = 0 D_0=0 D 0 = 0 . Unrolling this recursion, D r ≤ 2 γ ω n ( η ) ( 1 + γ + ⋯ + γ r − 1 ) = 2 γ ω n ( η ) ⋅ 1 − γ r 1 − γ ≤ 2 γ 1 − γ ω n ( η ) D_r\le2\gamma\omega_n(\eta)(1+\gamma+\cdots+\gamma^{r-1})=2\gamma\omega_n(\eta)\cdot\frac{1-\gamma^r}{1-\gamma}\le\frac{2\gamma}{1-\gamma}\omega_n(\eta) D r ≤ 2 γ ω n ( η ) ( 1 + γ + ⋯ + γ r − 1 ) = 2 γ ω n ( η ) ⋅ 1 − γ 1 − γ r ≤ 1 − γ 2 γ ω n ( η ) . Choose the rational constants and integer
c = e ( 1 − γ ) 16 γ , r ∗ = min { 1 / 2 , c / ( 2 n ) , c 2 / 16 } , Q = ⌈ A 2 r ∗ ⌉ . c=\frac{e(1-\gamma)}{16\gamma},\qquad r_*=\min\{1/2,c/(2n),c^2/16\},\qquad Q=\left\lceil\frac{A}{2r_*}\right\rceil. c = 16 γ e ( 1 − γ ) , r ∗ = min { 1/2 , c / ( 2 n ) , c 2 /16 } , Q = ⌈ 2 r ∗ A ⌉ .
This choice is engineered so that η = A / ( 2 Q ) ≤ r ∗ ≤ 1 − 1 / n \eta=A/(2Q)\le r_\ast\le1-1/n η = A / ( 2 Q ) ≤ r ∗ ≤ 1 − 1/ n (the modulus bound applies), and, since ω n \omega_n ω n is nondecreasing on this range and ω n ( r ) ≤ n r + 2 r \omega_n(r)\le nr+2\sqrt{r} ω n ( r ) ≤ n r + 2 r (using log 2 ( n − 1 ) ≤ n \log_2(n-1)\le n log 2 ( n − 1 ) ≤ n and h ( r ) ≤ 2 r h(r)\le2\sqrt{r} h ( r ) ≤ 2 r from Lemma 6), ω n ( η ) ≤ ω n ( r ∗ ) ≤ n r ∗ + 2 r ∗ ≤ c 2 + c 2 = c \omega_n(\eta)\le\omega_n(r_\ast)\le nr_\ast+2\sqrt{r_\ast}\le\frac{c}{2}+\frac{c}{2}=c ω n ( η ) ≤ ω n ( r ∗ ) ≤ n r ∗ + 2 r ∗ ≤ 2 c + 2 c = c , using r ∗ ≤ c / ( 2 n ) r_\ast\le c/(2n) r ∗ ≤ c / ( 2 n ) for the first term and r ∗ ≤ c 2 / 16 r_\ast\le c^2/16 r ∗ ≤ c 2 /16 for the second. Substituting ω n ( η ) ≤ c \omega_n(\eta)\le c ω n ( η ) ≤ c into the unrolled bound above,
∥ U T − U Q ∥ ∞ ≤ 2 γ 1 − γ ω n ( η ) ≤ 2 γ 1 − γ ⋅ e ( 1 − γ ) 16 γ = e / 8. (49) \lVert U_T-U^Q\rVert_\infty\le\frac{2\gamma}{1-\gamma}\omega_n(\eta)\le\frac{2\gamma}{1-\gamma}\cdot\frac{e(1-\gamma)}{16\gamma}=e/8. \tag*{(49)} ∥ U T − U Q ∥ ∞ ≤ 1 − γ 2 γ ω n ( η ) ≤ 1 − γ 2 γ ⋅ 16 γ e ( 1 − γ ) = e /8. (49)
Since γ , e \gamma,e γ , e are fixed, c c c is a fixed positive constant not depending on n n n or A A A . If c / ( 2 n ) c/(2n) c / ( 2 n ) is the smallest of the three terms in r ∗ = min { 1 / 2 , c / ( 2 n ) , c 2 / 16 } r_\ast=\min\{1/2,c/(2n),c^2/16\} r ∗ = min { 1/2 , c / ( 2 n ) , c 2 /16 } , then r ∗ = c / ( 2 n ) r_\ast=c/(2n) r ∗ = c / ( 2 n ) ; otherwise r ∗ r_\ast r ∗ equals one of the two constants $1/2,c^2/16$, which forces n ≤ c / ( 2 r ∗ ) = O ( 1 ) n\le c/(2r_\ast)=O(1) n ≤ c / ( 2 r ∗ ) = O ( 1 ) for fixed $c. In either case r ∗ = Ω ( 1 / n ) r_\ast=\Omega(1/n) r ∗ = Ω ( 1/ n ) for fixed γ , e \gamma,e γ , e , so Q = ⌈ A / ( 2 r ∗ ) ⌉ = O ( n A ) Q=\lceil A/(2r_\ast)\rceil=O(nA) Q = ⌈ A / ( 2 r ∗ )⌉ = O ( n A ) : discretising the action spaces to a grid of denominator Q Q Q costs at most e / 8 e/8 e /8 of the total error budget.
The alternating machine. With the action spaces now finite and the rounding error accounted for, this step builds the actual algorithm: a machine that evaluates the grid-restricted recursion round by round, guessing each player’s action, and checks its acceptance condition decides the promise gap correctly. With grid actions in hand, the recursion can finally be evaluated directly. Choose the least integer p ≥ 1 p\ge1 p ≥ 1 with 2 − p ≤ e / 8 2^{-p}\le e/8 2 − p ≤ e /8 , so p = O ( log ( 1 / e ) ) p=O(\log(1/e)) p = O ( log ( 1/ e )) bits suffice, and entropy can be evaluated to error 2 − p 2^{-p} 2 − p via H ^ p \widehat{H}_p H p (Part 2 of Lemma 6).
Why min becomes ∃ \exists ∃ and max becomes ∀ \forall ∀ . For any threshold v v v and any function g g g on a nonempty finite set, min x g ( x ) ≤ v \min_x g(x)\le v min x g ( x ) ≤ v iff some x x x has g ( x ) ≤ v g(x)\le v g ( x ) ≤ v , i.e. min x g ( x ) ≤ v ⟺ ∃ x : g ( x ) ≤ v \min_x g(x)\le v\Longleftrightarrow\exists x:g(x)\le v min x g ( x ) ≤ v ⟺ ∃ x : g ( x ) ≤ v ; dually, max y h ( y ) ≤ v \max_y h(y)\le v max y h ( y ) ≤ v iff every y y y has h ( y ) ≤ v h(y)\le v h ( y ) ≤ v , i.e. max y h ( y ) ≤ v ⟺ ∀ y : h ( y ) ≤ v \max_y h(y)\le v\Longleftrightarrow\forall y:h(y)\le v max y h ( y ) ≤ v ⟺ ∀ y : h ( y ) ≤ v . These hold against the same fixed v v v on both sides, so they compose under nesting: for g ( x ) = max y f ( x , y ) g(x)=\max_y f(x,y) g ( x ) = max y f ( x , y ) , min x max y f ( x , y ) ≤ v ⟺ ∃ x [ g ( x ) ≤ v ] ⟺ ∃ x [ max y f ( x , y ) ≤ v ] ⟺ ∃ x ∀ y : f ( x , y ) ≤ v \min_x\max_y f(x,y)\le v\Longleftrightarrow\exists x\,[g(x)\le v]\Longleftrightarrow\exists x\,[\max_y f(x,y)\le v]\Longleftrightarrow\exists x\forall y:f(x,y)\le v min x max y f ( x , y ) ≤ v ⟺ ∃ x [ g ( x ) ≤ v ] ⟺ ∃ x [ max y f ( x , y ) ≤ v ] ⟺ ∃ x ∀ y : f ( x , y ) ≤ v . This composition argument used only that both sides compare against the same fixed threshold v v v , never the specific form of the payoff, so it applies unchanged with any function of μ t \mu_t μ t in place of f f f .
Let z z z denote the quantity obtained from the same recursion as U R Q U^Q_R U R Q (grid-restricted actions, as defined above) but with H H H replaced throughout by the dyadic H ^ p \widehat{H}_p H p from Lemma 6. Iterating the substitution above down all T − 1 T-1 T − 1 rounds gives
z ≤ v ⟺ ∃ x 0 ∀ y 0 ⋯ ∃ x T − 2 ∀ y T − 2 : ( 1 − γ ) ∑ t = 0 T − 1 γ t H ^ p ( μ t ) ≤ v , z\le v\Longleftrightarrow\exists x_0\forall y_0\cdots\exists x_{T-2}\forall y_{T-2}:(1-\gamma)\sum_{t=0}^{T-1}\gamma^t\widehat{H}_p(\mu_t)\le v, z ≤ v ⟺ ∃ x 0 ∀ y 0 ⋯ ∃ x T − 2 ∀ y T − 2 : ( 1 − γ ) t = 0 ∑ T − 1 γ t H p ( μ t ) ≤ v ,
with μ 0 = μ 0 \mu_0=\mu_0 μ 0 = μ 0 and μ t + 1 = Υ x t , y t ( μ t ) \mu_{t+1}=\Upsilon_{x_t,y_t}(\mu_t) μ t + 1 = Υ x t , y t ( μ t ) : the entire nested min-max recursion, evaluated once, is equivalent to a single alternating quantifier prefix over the one payoff inequality at the end. Since an alternating machine can evaluate and compare H ^ p \widehat{H}_p H p exactly, this is exactly the shape such a machine decides natively: an existential state accepts iff some child does (matching ∃ \exists ∃ /min), a universal state accepts iff every child does (matching ∀ \forall ∀ /max).
What happens at each round. The machine carries the current distribution μ t \mu_t μ t , the round index t t t , and the running weighted sum ( 1 − γ ) ∑ t ′ = 0 t γ t ′ H ^ p ( μ t ′ ) (1-\gamma)\sum_{t'=0}^{t}\gamma^{t'}\widehat{H}_p(\mu_{t'}) ( 1 − γ ) ∑ t ′ = 0 t γ t ′ H p ( μ t ′ ) on its tape. At round t = 0 , … , T − 2 t=0,\ldots,T-2 t = 0 , … , T − 2 it: (1) computes the current reward H ^ p ( μ t ) \widehat{H}_p(\mu_t) H p ( μ t ) and adds γ t H ^ p ( μ t ) \gamma^t\widehat{H}_p(\mu_t) γ t H p ( μ t ) , weighted by ( 1 − γ ) (1-\gamma) ( 1 − γ ) , to the running sum; (2) enters an existential state and guesses the numerators of x t ∈ X Q x_t\in\mathcal{X}_Q x t ∈ X Q , one per ( s , α ) (s,\alpha) ( s , α ) with s ∈ S min s\in S_{\min} s ∈ S m i n , α ∈ A ( s ) \alpha\in A(s) α ∈ A ( s ) , each using ⌈ log 2 ( Q + 1 ) ⌉ \lceil\log_2(Q+1)\rceil ⌈ log 2 ( Q + 1 )⌉ bits; (3) validates them: each numerator must lie in [ 0 , Q ] [0,Q] [ 0 , Q ] , and at each state s ∈ S min s\in S_{\min} s ∈ S m i n the numerators over α ∈ A ( s ) \alpha\in A(s) α ∈ A ( s ) must sum to exactly Q Q Q (encoding a valid grid probability vector); an invalid guess makes this branch reject immediately, since a malformed x t x_t x t is not a legitimate witness for the ∃ \exists ∃ and must not be able to manufacture acceptance; (4) on a valid guess, enters a universal state and guesses the numerators of y t ∈ Y Q y_t\in\mathcal{Y}_Q y t ∈ Y Q the same way, this time over S max S_{\max} S m a x ; (5) validates them identically, except an invalid guess now makes this branch accept immediately, since a malformed y t y_t y t is not a legitimate move for the ∀ \forall ∀ and must not be able to manufacture rejection; (6) on a valid guess, computes the exact rational successor μ t + 1 = Υ x t , y t ( μ t ) \mu_{t+1}=\Upsilon_{x_t,y_t}(\mu_t) μ t + 1 = Υ x t , y t ( μ t ) and recurses to round t + 1 t+1 t + 1 . At μ T − 1 \mu_{T-1} μ T − 1 there is no outgoing action, matching Step 1’s observation that μ T − 1 \mu_{T-1} μ T − 1 needs no outgoing action; the machine adds the final term ( 1 − γ ) γ T − 1 H ^ p ( μ T − 1 ) (1-\gamma)\gamma^{T-1}\widehat{H}_p(\mu_{T-1}) ( 1 − γ ) γ T − 1 H p ( μ T − 1 ) to the running sum, which is then complete over all T T T reward terms, and the branch accepts exactly when
( 1 − γ ) ∑ t = 0 T − 1 γ t H ^ p ( μ t ) ≤ v . (1-\gamma)\sum_{t=0}^{T-1}\gamma^t\widehat{H}_p(\mu_t)\le v. ( 1 − γ ) t = 0 ∑ T − 1 γ t H p ( μ t ) ≤ v .
Recall that z z z is the value of this numerically evaluated grid game, so the machine accepts exactly when z ≤ v z\le v z ≤ v , by the equivalence displayed above. By construction z z z differs from U T Q ( μ 0 ) U_T^Q(\mu_0) U T Q ( μ 0 ) by at most 2 − p ≤ ϵ / 8 2^{-p}\le\epsilon/8 2 − p ≤ ϵ /8 , the entropy precision fixed by the choice of p p p above. By (49) , U T Q ( μ 0 ) U_T^Q(\mu_0) U T Q ( μ 0 ) differs from U T ( μ 0 ) U_T(\mu_0) U T ( μ 0 ) by at most ϵ / 8 \epsilon/8 ϵ /8 . By (46) , U T ( μ 0 ) U_T(\mu_0) U T ( μ 0 ) differs from V a v + ( μ 0 ) V_{\mathrm{av}}^+(\mu_0) V av + ( μ 0 ) by at most ϵ / 4 \epsilon/4 ϵ /4 . Hence
∣ z − V a v + ( μ 0 ) ∣ ≤ ∣ z − U T Q ( μ 0 ) ∣ + ∣ U T Q ( μ 0 ) − U T ( μ 0 ) ∣ + ∣ U T ( μ 0 ) − V a v + ( μ 0 ) ∣ ≤ ϵ / 8 + ϵ / 8 + ϵ / 4 = ϵ / 2 < ϵ 0 . \left|z-V_{\mathrm{av}}^+(\mu_0)\right|\le\left|z-U_T^Q(\mu_0)\right|+\left|U_T^Q(\mu_0)-U_T(\mu_0)\right|+\left|U_T(\mu_0)-V_{\mathrm{av}}^+(\mu_0)\right|\le\epsilon/8+\epsilon/8+\epsilon/4=\epsilon/2<\epsilon_0. z − V av + ( μ 0 ) ≤ z − U T Q ( μ 0 ) + U T Q ( μ 0 ) − U T ( μ 0 ) + U T ( μ 0 ) − V av + ( μ 0 ) ≤ ϵ /8 + ϵ /8 + ϵ /4 = ϵ /2 < ϵ 0 .
If V a v + ( μ 0 ) ≤ v − ϵ 0 V_{\mathrm{av}}^+(\mu_0)\le v-\epsilon_0 V av + ( μ 0 ) ≤ v − ϵ 0 , then z ≤ V a v + ( μ 0 ) + ϵ / 2 ≤ v − ϵ 0 + ϵ / 2 < v z\le V_{\mathrm{av}}^+(\mu_0)+\epsilon/2\le v-\epsilon_0+\epsilon/2<v z ≤ V av + ( μ 0 ) + ϵ /2 ≤ v − ϵ 0 + ϵ /2 < v , so the machine accepts. If V a v + ( μ 0 ) ≥ v + ϵ 0 V_{\mathrm{av}}^+(\mu_0)\ge v+\epsilon_0 V av + ( μ 0 ) ≥ v + ϵ 0 , then z ≥ V a v + ( μ 0 ) − ϵ / 2 ≥ v + ϵ 0 − ϵ / 2 > v z\ge V_{\mathrm{av}}^+(\mu_0)-\epsilon/2\ge v+\epsilon_0-\epsilon/2>v z ≥ V av + ( μ 0 ) − ϵ /2 ≥ v + ϵ 0 − ϵ /2 > v , so the machine rejects. This proves promise correctness: the machine accepts every yes-instance and rejects every no-instance.
Space bound. It remains to check the machine runs in polynomial time along every branch, not just that its numbers are individually accurate; this is what AP = PSPACE actually requires.
Let D D D be the product of the denominators of all input transition probabilities, and D 0 D_0 D 0 the product of the denominators of the initial distribution’s coordinates. Each individual denominator is a substring of the ℓ \ell ℓ -bit input encoding, so the sum of their bit lengths is at most ℓ \ell ℓ ; since the bit length of a product of positive integers is at most the sum of the bit lengths of the factors, both D D D and D 0 D_0 D 0 have O ( ℓ ) O(\ell) O ( ℓ ) bits.
Fix a branch, determined by the guessed grid actions ( x t , y t ) (x_t,y_t) ( x t , y t ) at each round t t t . Each numerator of x t , y t x_t,y_t x t , y t has denominator Q Q Q by construction, and each Δ ( s , α , s ′ ) \Delta(s,\alpha,s') Δ ( s , α , s ′ ) has denominator dividing D D D ; by (10) , Υ x t , y t ( μ ) ( s ′ ) = ∑ s μ ( s ) ∑ α π x t , y t ( s ) ( α ) Δ ( s , α , s ′ ) \Upsilon_{x_t,y_t}(\mu)(s')=\sum_s\mu(s)\sum_\alpha\pi_{x_t,y_t}(s)(\alpha)\Delta(s,\alpha,s') Υ x t , y t ( μ ) ( s ′ ) = ∑ s μ ( s ) ∑ α π x t , y t ( s ) ( α ) Δ ( s , α , s ′ ) is a sum of products of a denominator-Q Q Q term and a denominator-D D D term, so the transition matrix for this fixed pair has common denominator dividing Q D QD Q D .
This allows an induction on t t t : the base case μ 0 = μ 0 \mu_0=\mu_0 μ 0 = μ 0 has common denominator dividing D 0 D_0 D 0 . For the inductive step, if every coordinate of μ t \mu_t μ t has the form m s / ( D 0 ( Q D ) t ) m_s/(D_0(QD)^t) m s / ( D 0 ( Q D ) t ) for an integer m s m_s m s , and ∑ α π x t , y t ( s ) ( α ) Δ ( s , α , s ′ ) = q s , s ′ / ( Q D ) \sum_\alpha\pi_{x_t,y_t}(s)(\alpha)\Delta(s,\alpha,s')=q_{s,s'}/(QD) ∑ α π x t , y t ( s ) ( α ) Δ ( s , α , s ′ ) = q s , s ′ / ( Q D ) for an integer q s , s ′ q_{s,s'} q s , s ′ , then
μ t + 1 ( s ′ ) = ∑ s ∈ S m s D 0 ( Q D ) t ⋅ q s , s ′ Q D = ∑ s m s q s , s ′ D 0 ( Q D ) t + 1 , \mu_{t+1}(s')=\sum_{s\in S}\frac{m_s}{D_0(QD)^t}\cdot\frac{q_{s,s'}}{QD}=\frac{\sum_s m_sq_{s,s'}}{D_0(QD)^{t+1}}, μ t + 1 ( s ′ ) = s ∈ S ∑ D 0 ( Q D ) t m s ⋅ Q D q s , s ′ = D 0 ( Q D ) t + 1 ∑ s m s q s , s ′ ,
an integer numerator over D 0 ( Q D ) t + 1 D_0(QD)^{t+1} D 0 ( Q D ) t + 1 . So μ t \mu_t μ t has common denominator dividing D 0 ( Q D ) t D_0(QD)^t D 0 ( Q D ) t and each of its n n n coordinates uses O ( ℓ + t ( ℓ + log Q ) ) O(\ell+t(\ell+\log Q)) O ( ℓ + t ( ℓ + log Q )) bits. Exact updates are therefore additions and multiplications of integers of this bit length, which elementary arithmetic performs in time polynomial in ℓ + T ( ℓ + log Q ) \ell+T(\ell+\log Q) ℓ + T ( ℓ + log Q ) , itself polynomial in ℓ \ell ℓ since T , Q T,Q T , Q are polynomial in ℓ \ell ℓ .
The discounted dyadic sum also has polynomial bit length: if g g g is the fixed denominator of γ \gamma γ , then ( 1 − γ ) (1-\gamma) ( 1 − γ ) and γ t \gamma^t γ t each have denominator dividing g t + 1 g^{t+1} g t + 1 , and H ^ p ( μ t ) \widehat{H}_p(\mu_t) H p ( μ t ) has denominator 2 p 2^p 2 p by construction, so each summand ( 1 − γ ) γ t H ^ p ( μ t ) (1-\gamma)\gamma^t\widehat{H}_p(\mu_t) ( 1 − γ ) γ t H p ( μ t ) has denominator dividing g t + 2 2 p g^{t+2}2^p g t + 2 2 p , hence dividing g T 2 p g^{T}2^{p} g T 2 p for every t ≤ T − 1 t \le T-1 t ≤ T − 1 ; the sum of the T T T summands therefore has denominator dividing g T 2 p g^{T}2^{p} g T 2 p , a polynomial-bit rational since T , p T,p T , p are polynomial in ℓ \ell ℓ .
There are only T − 1 T-1 T − 1 action pairs along any branch. At each, validating the guessed numerators (checking they lie in [ 0 , Q ] [0,Q] [ 0 , Q ] and sum to Q Q Q at every controlled state) is a comparison of integers already bounded above, and computing H ^ p ( μ t ) \widehat{H}_{p}(\mu_t) H p ( μ t ) takes time polynomial in size( μ t ) + p (\mu_t)+p ( μ t ) + p by Lemma 6, itself polynomial in ℓ \ell ℓ since μ t \mu_t μ t ’s bit length and p p p both are. Summing these T − 1 T-1 T − 1 polynomial-time steps gives a polynomial total time along the branch. This is an alternating polynomial-time computation, hence is in PSPACE by AP = PSPACE. Equivalently, since the branch depth T − 1 T-1 T − 1 and every configuration’s bit length are both polynomial in ℓ \ell ℓ , a deterministic depth-first traversal of its polynomial-depth binary computation tree uses a polynomial stack of polynomial-size configurations.
The lower value. For V a v − V_{\mathrm{av}}^{-} V av − the same proof applies with the order of the extrema swapped. Let U r − = ( T a v − ) r 0 U_r^{-}=(T_{\mathrm{av}}^{-})^{r}0 U r − = ( T av − ) r 0 , so that U r + 1 − ( μ ) = ( 1 − γ ) H ( μ ) + γ max y min x U r − ( Υ x , y ( μ ) ) U_{r+1}^{-}(\mu)=(1-\gamma)H(\mu)+\gamma\max_y\min_x U_r^{-}(\Upsilon_{x,y}(\mu)) U r + 1 − ( μ ) = ( 1 − γ ) H ( μ ) + γ max y min x U r − ( Υ x , y ( μ )) . By the finite-horizon identification and error bound in the proof of Theorem 1, 0 ≤ V a v − − U T − ≤ γ T L ≤ ϵ / 4 0\le V_{\mathrm{av}}^{-}-U_T^{-}\le\gamma^T L\le\epsilon/4 0 ≤ V av − − U T − ≤ γ T L ≤ ϵ /4 . The modulus (40) holds for U r − U_r^{-} U r − with the same proof, since adding a uniform bound to the argument shifts max y min x \max_y\min_x max y min x by at most that bound. For the grid, put F ( x , y ) = U r − ( Υ x , y ( μ ) ) F(x,y)=U_r^{-}(\Upsilon_{x,y}(\mu)) F ( x , y ) = U r − ( Υ x , y ( μ )) and M − = max y min x F M^{-}=\max_y\min_x F M − = max y min x F . Restricting y y y to Y Q \mathcal{Y}_Q Y Q gives M Y − ≤ M − M_Y^{-}\le M^{-} M Y − ≤ M − , and rounding a maximiser y ∗ y^* y ∗ to y Q ∗ y_Q^* y Q ∗ changes min x F ( x , ⋅ ) \min_x F(x,\cdot) min x F ( x , ⋅ ) by at most ω n ( η ) \omega_n(\eta) ω n ( η ) , by (39) applied for every x x x and by (40); so M − ≤ M Y − + ω n ( η ) M^{-}\le M_Y^{-}+\omega_n(\eta) M − ≤ M Y − + ω n ( η ) . Restricting the inner minimum to X Q \mathcal{X}_Q X Q can only increase it, and for every y y y rounding a best response raises it by at most ω n ( η ) \omega_n(\eta) ω n ( η ) . Hence the value changes by at most 2 ω n ( η ) 2\omega_n(\eta) 2 ω n ( η ) per round, and (41) holds for U T − U_T^{-} U T − with the same c c c and Q Q Q . Since max y g ( y ) ≤ v ⟺ ∀ y [ g ( y ) ≤ v ] \max_y g(y)\le v\Longleftrightarrow\forall y\,[g(y)\le v] max y g ( y ) ≤ v ⟺ ∀ y [ g ( y ) ≤ v ] and min x h ( x ) ≤ v ⟺ ∃ x [ h ( x ) ≤ v ] \min_x h(x)\le v\Longleftrightarrow\exists x\,[h(x)\le v] min x h ( x ) ≤ v ⟺ ∃ x [ h ( x ) ≤ v ] , the machine uses the prefix ∀ y 0 ∃ x 0 ⋯ ∀ y T − 2 ∃ x T − 2 \forall y_0\exists x_0\cdots\forall y_{T-2}\exists x_{T-2} ∀ y 0 ∃ x 0 ⋯ ∀ y T − 2 ∃ x T − 2 : at each round it guesses y t y_t y t universally and then x t x_t x t existentially. A malformed universal guess accepts and a malformed existential guess rejects. Correctness (∣ z − V a v − ( μ 0 ) ∣ ≤ ϵ / 2 < ϵ 0 |z-V_{\mathrm{av}}^{-}(\mu_0)|\le\epsilon/2<\epsilon_0 ∣ z − V av − ( μ 0 ) ∣ ≤ ϵ /2 < ϵ 0 ) and the polynomial bounds are unchanged. ◀
Sampled-profile semantics
By Theorem 4, the sampled-profile upper and lower values coincide, so the polarity may be omitted from GapVal s p , ϵ 0 \operatorname{GapVal}_{\mathrm{sp},\epsilon_0} GapVal sp , ϵ 0 and V s p V_{\mathrm{sp}} V sp .
Theorem 9 (Polynomial-space upper bound under sampled-profile semantics). Fix rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 . Then GapVal s p , ϵ 0 ∈ P S P A C E \operatorname{GapVal}_{\mathrm{sp},\epsilon_0}\in\mathrm{PSPACE} GapVal sp , ϵ 0 ∈ PSPACE .
Under sampled profile semantics, a sampled profile mixture has many successor distributions, and its value is the average of their continuation values, not the value of their mean, so it cannot be evaluated the way it is under averaged-distribution semantics. We retain these different successors but evaluate them one at a time.
Deciding GapVal s p , ϵ 0 \operatorname{GapVal}_{\mathrm{sp},\epsilon_0} GapVal sp , ϵ 0 comes down to computing V s p ( μ 0 ) V_{\mathrm{sp}}(\mu_0) V sp ( μ 0 ) to enough accuracy to compare it with v v v . In Appendix E.2.1, we show that V s p V_{\mathrm{sp}} V sp satisfies a recursion in which each step solves a matrix game val ( M ) \operatorname{val}(M) val ( M ) over the pure profile sets P min , P max \mathcal{P}_{\min},\mathcal{P}_{\max} P m i n , P m a x , and these sets can have up to A n A^n A n elements: writing down even a single mixed strategy over them, or a single profile as an explicit vector, would already cost exponential space. We describe below the steps that help overcome this:
Appendix E.2.2 removes the need for a full mixed strategy ranging over all of P min \mathcal{P}_{\min} P m i n or P max \mathcal{P}_{\max} P m a x : it shows a near-optimal choice can always be taken to be a uniform mixture of only polynomially many pure profiles, so solving val ( M ) \operatorname{val}(M) val ( M ) approximately becomes a search over polynomially many candidates instead of an exponential simplex.
(ii) Appendix E.2.3 gives each pure profile an O ( n log ( 2 A ) ) O(n \log(2A)) O ( n log ( 2 A )) -bit description, so listing the polynomially many profiles from (i) still costs only polynomially many bits, not exponentially many.
(iii) Appendix E.2.4 puts the two together: it evaluates the finite-horizon recursion for V s p V_{\mathrm{sp}} V sp depth first, at each step searching only over the polynomially many candidate mixtures from (i), each written down using the compact profile encoding from (ii). It proves a uniform bound on the rounding and truncation error this introduces, and it bounds the space of the whole computation by showing that only the current path of recursive calls needs to be held in memory at once, not the entire recursion tree. The result is a deterministic polynomial-space procedure that decides G a p V a l s p , ε 0 \mathrm{GapVal}_{\mathrm{sp},\varepsilon_0} GapVal sp , ε 0 .
Background
Recall that n = ∣ S ∣ n = |S| n = ∣ S ∣ and A = max s ∈ S ∣ A ( s ) ∣ A = \max_{s \in S} |A(s)| A = max s ∈ S ∣ A ( s ) ∣ , and fix a rational discount γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) . Under sampled-profile semantics, Theorem 4 shows that the upper and lower values coincide; write V s p : D ( S ) → [ 0 , log 2 n ] V_{\mathrm{sp}} : \mathcal{D}(S) \to[0,\log_2 n] V sp : D ( S ) → [ 0 , log 2 n ] for this common value (Step 2 of that proof). Its range follows because V s p V_{\mathrm{sp}} V sp is a discounted convex combination of entropy values, each lying in [ 0 , log 2 n ] [0,\log_2 n] [ 0 , log 2 n ] by Proposition 16(i). Step 1 of the same proof derives the one-step identity (41) for V s p V_{\mathrm{sp}} V sp , which is exactly the statement that V s p V_{\mathrm{sp}} V sp is a fixed point of the operator T s p \mathcal{T}_{\mathrm{sp}} T sp defined below by that same formula; since T s p \mathcal{T}_{\mathrm{sp}} T sp is a γ \gamma γ -contraction, V s p V_{\mathrm{sp}} V sp is in fact its unique fixed point, its Bellman operator :
( T s p V ) ( μ ) = ( 1 − γ ) H ( μ ) + γ ⋅ val ( M V ( μ ) ) , M V ( μ ) a , b : = V ( Φ a , b ( μ ) ) . (50) (\mathcal{T}_{\mathrm{sp}}V)(\mu) = (1-\gamma)H(\mu) + \gamma\cdot\operatorname{val}(M_V(\mu)), \qquad M_V(\mu)_{a,b} := V(\Phi_{a,b}(\mu)). \tag*{(50)} ( T sp V ) ( μ ) = ( 1 − γ ) H ( μ ) + γ ⋅ val ( M V ( μ )) , M V ( μ ) a , b := V ( Φ a , b ( μ )) . (50)
where, for a matrix M ∈ R P min × P max M \in\mathbb{R}^{\mathcal{P}_{\min} \times\mathcal{P}_{\max}} M ∈ R P m i n × P m a x , val ( M ) : = max τ ∈ D ( P max ) min σ ∈ D ( P min ) σ T M τ \operatorname{val}(M) := \max_{\tau\in\mathcal{D}(\mathcal{P}_{\max})} \min_{\sigma\in\mathcal{D}(\mathcal{P}_{\min})} \sigma^{\mathsf{T}}M\tau val ( M ) := max τ ∈ D ( P m a x ) min σ ∈ D ( P m i n ) σ T M τ is the value of the associated zero-sum matrix game (Min minimising, Max maximising), well defined by von Neumann’s minimax theorem [36 , 32 ] since P min , P max \mathcal{P}_{\min},\mathcal{P}_{\max} P m i n , P m a x are finite. Here σ \sigma σ and τ \tau τ play the same role as the sampling strategies σ t , τ t \sigma_t,\tau_t σ t , τ t from the definition of sampled-profile semantics (see Appendix § 12 .§ 1 ). σ T M τ = ∑ a , b σ ( a ) τ ( b ) M ( a , b ) = E a ∼ σ , b ∼ τ [ M ( a , b ) ] \sigma^{\mathsf{T}}M\tau= \sum_{a,b}\sigma(a)\tau(b)M(a,b) = \mathbb{E}_{a \sim\sigma,\,b \sim\tau}[M(a,b)] σ T M τ = ∑ a , b σ ( a ) τ ( b ) M ( a , b ) = E a ∼ σ , b ∼ τ [ M ( a , b )] is exactly the expected continuation value under the same sampling distribution σ × τ \sigma\times\tau σ × τ from which the round’s profile ( a t , b t ) (a_t,b_t) ( a t , b t ) is drawn and then executed (Equation (14) ); val ( M ) \operatorname{val}(M) val ( M ) asks what expected value each player can guarantee by choosing that sampling distribution optimally, before the profile is actually sampled.
This T s p \mathcal{T}_{\mathrm{sp}} T sp specialises, at ⋆ = s p \star= \mathrm{sp} ⋆ = sp , the general one-step identity (41) from Step 1 of Theorem 4’s proof,
V ⋆ ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M ⋆ ( μ ) ) , V_{\star}(\mu) = (1-\gamma)H(\mu) + \gamma\operatorname{val}(M^{\star}(\mu)), V ⋆ ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M ⋆ ( μ )) ,
where M s p ( μ ) a , b = V s p ( Φ a , b ( μ ) ) M^{\mathrm{sp}}(\mu)_{a,b} = V_{\mathrm{sp}}(\Phi_{a,b}(\mu)) M sp ( μ ) a , b = V sp ( Φ a , b ( μ )) is the sampled-profile case of (40) there. This is the same formula, evaluated at the same function, as M V ( μ ) a , b : = V ( Φ a , b ( μ ) ) M_V(\mu)_{a,b} := V(\Phi_{a,b}(\mu)) M V ( μ ) a , b := V ( Φ a , b ( μ )) above with V = V s p V = V_{\mathrm{sp}} V = V sp substituted, so
M V s p ( μ ) = M s p ( μ ) for every μ ∈ D ( S ) . (51) M_{V_{\mathrm{sp}}}(\mu) = M^{\mathrm{sp}}(\mu) \qquad\text{for every } \mu\in\mathcal{D}(S). \tag*{(51)} M V sp ( μ ) = M sp ( μ ) for every μ ∈ D ( S ) . (51)
Substituting V = V s p V = V_{\mathrm{sp}} V = V sp into the definition of T s p \mathcal{T}_{\mathrm{sp}} T sp and applying (51) gives, for every μ \mu μ ,
( T s p V s p ) ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M V s p ( μ ) ) = (43) ( 1 − γ ) H ( μ ) + γ val ( M s p ( μ ) ) , (\mathcal{T}_{\mathrm{sp}}V_{\mathrm{sp}})(\mu) = (1-\gamma)H(\mu) + \gamma\operatorname{val}(M_{V_{\mathrm{sp}}}(\mu)) \mathrel{\overset{\text{(43)}}{=}} (1-\gamma)H(\mu) + \gamma\operatorname{val}(M^{\mathrm{sp}}(\mu)), ( T sp V sp ) ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M V sp ( μ )) = (43) ( 1 − γ ) H ( μ ) + γ val ( M sp ( μ )) ,
and the right-hand side here is exactly the right-hand side of (41) at ⋆ = s p \star= \mathrm{sp} ⋆ = sp , hence equals V s p ( μ ) V_{\mathrm{sp}}(\mu) V sp ( μ ) . So ( T s p V s p ) ( μ ) = V s p ( μ ) (\mathcal{T}_{\mathrm{sp}}V_{\mathrm{sp}})(\mu) = V_{\mathrm{sp}}(\mu) ( T sp V sp ) ( μ ) = V sp ( μ ) for every μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) , i.e. V s p = T s p V s p V_{\mathrm{sp}} = \mathcal{T}_{\mathrm{sp}}V_{\mathrm{sp}} V sp = T sp V sp : V s p V_{\mathrm{sp}} V sp is a fixed point of T s p \mathcal{T}_{\mathrm{sp}} T sp ; since T s p \mathcal{T}_{\mathrm{sp}} T sp is a γ \gamma γ -contraction (γ < 1 \gamma< 1 γ < 1 ) in sup norm on the complete space of bounded functions D ( S ) → R \mathcal{D}(S) \to\mathbb{R} D ( S ) → R , this fixed point is automatically unique.
Recall val ( M ) = max τ min σ σ T M τ \operatorname{val}(M)=\max_{\tau}\min_{\sigma}\sigma^{\mathsf{T}}M\tau val ( M ) = max τ min σ σ T M τ from its definition above; the minimax theorem invoked there allows opposite order of computation as well, val ( M ) = min σ max τ σ T M τ \operatorname{val}(M)=\min_{\sigma}\max_{\tau}\sigma^{\mathsf{T}}M\tau val ( M ) = min σ max τ σ T M τ . In each order, the inner optimisation is over a variable that enters the objective only linearly, so it is attained at a vertex of its simplex, i.e. at a pure strategy: fixing τ \tau τ in the first order, σ T M τ = ∑ a σ ( a ) ( M τ ) a \sigma^{\mathsf{T}}M\tau=\sum_a\sigma(a)(M\tau)_a σ T M τ = ∑ a σ ( a ) ( M τ ) a is linear in σ \sigma σ , so min σ σ T M τ = min a ( M τ ) a \min_{\sigma}\sigma^{\mathsf{T}}M\tau=\min_a(M\tau)_a min σ σ T M τ = min a ( M τ ) a ; fixing σ \sigma σ in the second order, σ T M τ = ∑ b ( σ T M ) b τ ( b ) \sigma^{\mathsf{T}}M\tau=\sum_b(\sigma^{\mathsf{T}}M)_b\tau(b) σ T M τ = ∑ b ( σ T M ) b τ ( b ) is linear in τ \tau τ , so max τ σ T M τ = max b ( σ T M ) b \max_{\tau}\sigma^{\mathsf{T}}M\tau=\max_b(\sigma^{\mathsf{T}}M)_b max τ σ T M τ = max b ( σ T M ) b . This yields the two equivalent one-sided forms of val \operatorname{val} val : only one player’s strategy remains mixed, the other having collapsed to a pure-strategy optimum
val ( M ) = max τ ∈ D ( P max ) min a ∈ P min ( M τ ) a = min σ ∈ D ( P min ) max b ∈ P max ( σ T M ) b . (52) \operatorname{val}(M)=\max_{\tau\in\mathcal{D}(\mathcal{P}_{\max})}\min_{a\in\mathcal{P}_{\min}}(M\tau)_a=\min_{\sigma\in\mathcal{D}(\mathcal{P}_{\min})}\max_{b\in\mathcal{P}_{\max}}(\sigma^{\mathsf{T}}M)_b. \tag*{(52)} val ( M ) = τ ∈ D ( P m a x ) max a ∈ P m i n min ( M τ ) a = σ ∈ D ( P m i n ) min b ∈ P m a x max ( σ T M ) b . (52)
Having recalled this background, we now turn to the problem of interest, GapVal s p , ε 0 \operatorname{GapVal}_{\mathrm{sp},\varepsilon_0} GapVal sp , ε 0 .
A Sparse Certificate Lemma for Zero-Sum Matrix Games
Direct evaluation of val ( M ) \operatorname{val}(M) val ( M ) via (52) requires quantifying over a full mixed strategy in D ( P min ) \mathcal{D}(\mathcal{P}_{\min}) D ( P m i n ) or D ( P max ) \mathcal{D}(\mathcal{P}_{\max}) D ( P m a x ) : simplices whose vertex count can be as large as A n A^n A n (Appendix § 15 .2.1). So representing even a single such mixed strategy would already cost exponentially many variables, incompatible with the space bound targeted here. Lemma 8(ii) removes this obstacle: for any target accuracy ε > 0 \varepsilon>0 ε > 0 , a mixed strategy that is optimal to within ε \varepsilon ε can always be taken sparse , a uniform mixture of only k = poly ( n , log A , 1 / ε ) k=\operatorname{poly}(n,\log A,1/\varepsilon) k = poly ( n , log A , 1/ ε ) pure strategies, which is what lets Appendix § 15 .2.4 below quantify over a near-optimal strategy compactly.
Only one player’s strategy is ever sparsified in a single application of Lemma 8: as the one-sided forms (52) show, whichever player’s strategy is treated as the outer optimization there is the one that may need full, possibly exponential, support, while the other player’s best response, the inner optimization, is attained exactly at a single pure strategy with no approximation at all. Appendix § 15 .2.4 below always sparsifies Min’s strategy this way, so Max’s response is quantified exactly over the whole of P max \mathcal{P}_{\max} P m a x , via the compact encoding of Appendix § 15 .2.3, rather than approximated.
Part (ii) of Lemma 8 (existence) is due to Lipton and Young [26 ] and, independently, Althöfer [7 ] : a sparse ε \varepsilon ε -optimal strategy of the stated support size always exists. Part (i) (exact soundness ), by contrast, is not due to either source; it is the elementary observation that any sparse strategy, optimal or not, already certifies a matching one-sided bound on val ( M ) \operatorname{val}(M) val ( M ) directly from (52) . Both parts are stated together because the proof of Theorem 9 uses both directions.
Lemma 8 (Sparse minimax certificates). Let M ∈ [ 0 , U ] R × C M\in[0,U]^{R\times C} M ∈ [ 0 , U ] R × C be a finite zero-sum game, with the row player minimising. Write [ R ] = { 1 , … , R } [R]=\{1,\ldots,R\} [ R ] = { 1 , … , R } and [ C ] = { 1 , … , C } [C]=\{1,\ldots,C\} [ C ] = { 1 , … , C } for the rows and columns, and val ( M ) = min p max b ∈ [ C ] ∑ a p a M ( a , b ) = max q min a ∈ [ R ] ∑ b q b M ( a , b ) \operatorname{val}(M)=\min_p\max_{b\in[C]}\sum_a p_aM(a,b)=\max_q\min_{a\in[R]}\sum_bq_bM(a,b) val ( M ) = min p max b ∈ [ C ] ∑ a p a M ( a , b ) = max q min a ∈ [ R ] ∑ b q b M ( a , b ) over probability vectors p , q p,q p , q on rows and columns. A distribution supported on columns b 1 , … , b k b_1,\ldots,b_k b 1 , … , b k is given by weights q 1 , … , q k ≥ 0 q_1,\ldots,q_k\geq0 q 1 , … , q k ≥ 0 with ∑ i q i = 1 \sum_iq_i=1 ∑ i q i = 1 , and likewise for rows with weights p i p_i p i .
(i) (Soundness.) If a distribution supported on columns b 1 , … , b k b_1,\ldots,b_k b 1 , … , b k satisfies ∑ i q i M ( a , b i ) ≥ v \sum_iq_iM(a,b_i)\geq v ∑ i q i M ( a , b i ) ≥ v for every a ∈ [ R ] a\in[R] a ∈ [ R ] , then val ( M ) ≥ v \operatorname{val}(M)\geq v val ( M ) ≥ v . Dually, if a distribution supported on rows a 1 , … , a k a_1,\ldots,a_k a 1 , … , a k satisfies ∑ i p i M ( a i , b ) ≤ v \sum_ip_iM(a_i,b)\leq v ∑ i p i M ( a i , b ) ≤ v for every b ∈ [ C ] b\in[C] b ∈ [ C ] , then val ( M ) ≤ v \operatorname{val}(M)\leq v val ( M ) ≤ v .
(ii) (Existence.) If val ( M ) ≥ v + ε \operatorname{val}(M)\geq v+\varepsilon val ( M ) ≥ v + ε , then for every k ≥ max { 1 , ⌈ 2 U 2 ln ( 2 R ) / ε 2 ⌉ } k\geq\max\{1,\lceil2U^2\ln(2R)/\varepsilon^2\rceil\} k ≥ max { 1 , ⌈ 2 U 2 ln ( 2 R ) / ε 2 ⌉} there exist columns b 1 , … , b k b_1,\ldots,b_k b 1 , … , b k such that 1 k ∑ i M ( a , b i ) ≥ v \frac{1}{k}\sum_iM(a,b_i)\geq v k 1 ∑ i M ( a , b i ) ≥ v for every a ∈ [ R ] a\in[R] a ∈ [ R ] . Dually, if val ( M ) ≤ v − ε \operatorname{val}(M)\leq v-\varepsilon val ( M ) ≤ v − ε , then for every k ≥ max { 1 , ⌈ 2 U 2 ln ( 2 C ) / ε 2 ⌉ } k\geq\max\{1,\lceil2U^2\ln(2C)/\varepsilon^2\rceil\} k ≥ max { 1 , ⌈ 2 U 2 ln ( 2 C ) / ε 2 ⌉} there exist rows a 1 , … , a k a_1,\ldots,a_k a 1 , … , a k such that 1 k ∑ i M ( a i , b ) ≤ v \frac{1}{k}\sum_iM(a_i,b)\leq v k 1 ∑ i M ( a i , b ) ≤ v for every b ∈ [ C ] b\in[C] b ∈ [ C ] . Profiles may be repeated and every weight can be chosen equal to 1 / k 1/k 1/ k . The assertion holds for every integer k k k at least the displayed bound.
Proof. (i) Set τ : = ∑ i = 1 k w i δ b i \tau:= \sum_{i=1}^{k} w_i\delta_{b_i} τ := ∑ i = 1 k w i δ b i , where δ b i \delta_{b_i} δ b i denotes the point mass at b i ∈ [ C ] b_i \in[C] b i ∈ [ C ] . Since w i ≥ 0 w_i \ge0 w i ≥ 0 for every i i i and ∑ i w i = 1 \sum_i w_i = 1 ∑ i w i = 1 , τ ∈ D ( [ C ] ) \tau\in\mathcal{D}([C]) τ ∈ D ([ C ]) : it assigns each b ∈ [ C ] b \in[C] b ∈ [ C ] the nonnegative weight τ ( b ) = ∑ i : b i = b w i \tau(b) = \sum_{i:b_i=b} w_i τ ( b ) = ∑ i : b i = b w i , and these weights sum to 1; the b i b_i b i need not be distinct, and if some value of b b b never occurs among b 1 , … , b k b_1,\ldots,b_k b 1 , … , b k it simply gets τ ( b ) = 0 \tau(b)=0 τ ( b ) = 0 . For any a ∈ [ R ] a \in[R] a ∈ [ R ] ,
( M τ ) a = ∑ b ∈ [ C ] M ( a , b ) τ ( b ) = ∑ i = 1 k M ( a , b i ) w i . (M\tau)_a = \sum_{b\in[C]} M(a,b)\,\tau(b) = \sum_{i=1}^{k} M(a,b_i)\,w_i. ( M τ ) a = b ∈ [ C ] ∑ M ( a , b ) τ ( b ) = i = 1 ∑ k M ( a , b i ) w i .
where the second equality regroups the sum over b b b into the (possibly repeated) list b 1 , … , b k b_1,\ldots,b_k b 1 , … , b k , each contributing its weight w i w_i w i , the quantity ∑ i = 1 k w i M ( a , b i ) \sum_{i=1}^{k} w_i M(a,b_i) ∑ i = 1 k w i M ( a , b i ) appearing in the hypothesis. By hypothesis this is ≥ v \ge v ≥ v for every a ∈ [ R ] a \in[R] a ∈ [ R ] , so
min a ∈ [ R ] ( M τ ) a ≥ v . \min_{a\in[R]}(M\tau)_a \ge v. a ∈ [ R ] min ( M τ ) a ≥ v .
Now, τ \tau τ is one particular element of D ( [ C ] ) \mathcal{D}([C]) D ([ C ]) , not necessarily the maximizer. So it can only be a feasible competitor in the outer maximization over τ \tau τ in the first expression of (52) : taking the maximum over all of D ( [ C ] ) \mathcal{D}([C]) D ([ C ]) can only match or exceed the value at this particular τ \tau τ ,
val ( M ) = ( 44 ) max τ ′ ∈ D ( [ C ] ) min a ∈ [ R ] ( M τ ′ ) a ≥ min a ∈ [ R ] ( M τ ) a ≥ v . \operatorname{val}(M) \mathrel{\overset{(44)}{=}} \max_{\tau'\in\mathcal{D}([C])}\min_{a\in[R]}(M\tau')_a \ge\min_{a\in[R]}(M\tau)_a \ge v. val ( M ) = ( 44 ) τ ′ ∈ D ([ C ]) max a ∈ [ R ] min ( M τ ′ ) a ≥ a ∈ [ R ] min ( M τ ) a ≥ v .
This proves the first half of (i). For the symmetric half, set σ : = ∑ i = 1 k u i δ a i ∈ D ( [ R ] ) \sigma:= \sum_{i=1}^{k} u_i\delta_{a_i} \in\mathcal{D}([R]) σ := ∑ i = 1 k u i δ a i ∈ D ([ R ]) , again a valid mixed strategy since u i ≥ 0 u_i \ge0 u i ≥ 0 and ∑ i u i = 1 \sum_i u_i = 1 ∑ i u i = 1 . For any b ∈ [ C ] b \in[C] b ∈ [ C ] ,
( σ T M ) b = ∑ a ∈ [ R ] σ ( a ) M ( a , b ) = ∑ i = 1 k u i M ( a i , b ) ≤ v (\sigma^{\mathsf T}M)_b = \sum_{a\in[R]} \sigma(a)M(a,b) = \sum_{i=1}^{k} u_iM(a_i,b) \le v ( σ T M ) b = a ∈ [ R ] ∑ σ ( a ) M ( a , b ) = i = 1 ∑ k u i M ( a i , b ) ≤ v
by hypothesis, for every b ∈ [ C ] b \in[C] b ∈ [ C ] , so max b ∈ [ C ] ( σ T M ) b ≤ v \max_{b\in[C]}(\sigma^{\mathsf T}M)_b \le v max b ∈ [ C ] ( σ T M ) b ≤ v . This time σ \sigma σ is a feasible competitor in the outer minimisation over σ \sigma σ in the second expression of (52) , so taking the minimum over all of D ( [ R ] ) \mathcal{D}([R]) D ([ R ]) can only match or fall below the value at this particular σ \sigma σ ,
val ( M ) = ( 44 ) min σ ′ ∈ D ( [ R ] ) max b ∈ [ C ] ( σ ′ T M ) b ≤ max b ∈ [ C ] ( σ T M ) b ≤ v , \operatorname{val}(M) \mathrel{\overset{(44)}{=}} \min_{\sigma'\in\mathcal{D}([R])}\max_{b\in[C]}(\sigma'^{\mathsf T}M)_b \le\max_{b\in[C]}(\sigma^{\mathsf T}M)_b \le v, val ( M ) = ( 44 ) σ ′ ∈ D ([ R ]) min b ∈ [ C ] max ( σ ′ T M ) b ≤ b ∈ [ C ] max ( σ T M ) b ≤ v ,
proving the second half of (i).
(ii) Let τ ∗ ∈ D ( [ C ] ) \tau^* \in\mathcal{D}([C]) τ ∗ ∈ D ([ C ]) attain min a ( M τ ∗ ) a = val ( M ) ≥ v + ϵ \min_a(M\tau^*)_a = \operatorname{val}(M) \ge v+\epsilon min a ( M τ ∗ ) a = val ( M ) ≥ v + ϵ (such a τ ∗ \tau^* τ ∗ exists by compactness of D ( [ C ] ) \mathcal{D}([C]) D ([ C ]) and continuity of τ ↦ min a ( M τ ) a \tau\mapsto\min_a(M\tau)_a τ ↦ min a ( M τ ) a ; the inner minimum over a a a is attained because [ R ] [R] [ R ] is finite). Therefore, ( M τ ∗ ) a ≥ min a ( M τ ∗ ) a ≥ v + ϵ (M\tau^*)_a \ge\min_a(M\tau^*)_a \ge v+\epsilon ( M τ ∗ ) a ≥ min a ( M τ ∗ ) a ≥ v + ϵ .
If U = 0 U=0 U = 0 , then M ∈ [ 0 , 0 ] R × C M\in[0,0]^{R\times C} M ∈ [ 0 , 0 ] R × C , i.e. M ( a , b ) = 0 M(a,b)=0 M ( a , b ) = 0 for every a ∈ [ R ] , b ∈ [ C ] a\in[R],b\in[C] a ∈ [ R ] , b ∈ [ C ] ; consequently ( M τ ) a = ∑ b M ( a , b ) τ ( b ) = 0 (M\tau)_a=\sum_b M(a,b)\tau(b)=0 ( M τ ) a = ∑ b M ( a , b ) τ ( b ) = 0 for every τ ∈ D ( [ C ] ) \tau\in\mathcal{D}([C]) τ ∈ D ([ C ]) and every a ∈ [ R ] a\in[R] a ∈ [ R ] , so val ( M ) = max τ min a ( M τ ) a = 0 \operatorname{val}(M)=\max_\tau\min_a(M\tau)_a=0 val ( M ) = max τ min a ( M τ ) a = 0 . The hypothesis val ( M ) ≥ v + ϵ \operatorname{val}(M)\ge v+\epsilon val ( M ) ≥ v + ϵ therefore forces $0\ge v+\epsilon$, i.e. v ≤ − ϵ < 0 v\le-\epsilon<0 v ≤ − ϵ < 0 . Taking k = 1 k=1 k = 1 (matching the general formula for k k k , which evaluates to max { 1 , ⌈ 0 ⌉ } = 1 \max\{1,\lceil0\rceil\}=1 max { 1 , ⌈ 0 ⌉} = 1 when U = 0 U=0 U = 0 ), any b 1 ∈ [ C ] b_1\in[C] b 1 ∈ [ C ] , and $w_1=1: since M ≡ 0 M\equiv0 M ≡ 0 ,
∑ i = 1 1 w i M ( a , b 1 ) = M ( a , b 1 ) = 0 \sum_{i=1}^{1} w_iM(a,b_1)=M(a,b_1)=0 i = 1 ∑ 1 w i M ( a , b 1 ) = M ( a , b 1 ) = 0
for every a ∈ [ R ] a\in[R] a ∈ [ R ] , and 0 ≥ v 0\ge v 0 ≥ v holds because v < 0 v<0 v < 0 . So this choice already satisfies the required inequality for every a a a , and no probabilistic argument is needed to get it; since M = 0 M=0 M = 0 , the sampled value M ( a , b i ) M(a,b_i) M ( a , b i ) is not actually random at all, whichever b i b_i b i is drawn, it is the constant 0.
Assume U > 0 U>0 U > 0 . We prove existence by the probabilistic method: a random empirical law succeeds with positive probability, so a fixed successful tuple exists. The deterministic algorithm can enumerate all such tuples; it does not need to sample from, or already know, an optimal law. Draw b 1 , … , b k ∈ [ C ] b_1,\ldots,b_k \in[C] b 1 , … , b k ∈ [ C ] independently, each according to τ ∗ \tau^* τ ∗ : that is Pr [ b i = b ] = τ ∗ ( b ) \Pr[b_i=b]=\tau^*(b) Pr [ b i = b ] = τ ∗ ( b ) for every b ∈ [ C ] b \in[C] b ∈ [ C ] , and the k k k draws i = 1 , … , k i=1,\ldots,k i = 1 , … , k do not affect one another, and set w i = 1 / k w_i=1/k w i = 1/ k for every i i i , so
τ ^ : = 1 k ∑ i = 1 k δ b i ∈ D ( [ C ] ) \hat{\tau}:=\frac{1}{k}\sum_{i=1}^{k}\delta_{b_i}\in\mathcal{D}([C]) τ ^ := k 1 i = 1 ∑ k δ b i ∈ D ([ C ])
is the empirical distribution of the sample. This is the same kind of point-mass mixture as the τ \tau τ built in the proof of (i) above, except that here the b 1 , … , b k b_1,\ldots,b_k b 1 , … , b k are random: the outcome of the k k k draws, so τ ^ \hat{\tau} τ ^ , and consequently ( M τ ^ ) a (M\hat{\tau})_a ( M τ ^ ) a below, is a random object rather than a fixed one. Fix a ∈ [ R ] a\in[R] a ∈ [ R ] . The variable ( M τ ^ ) a = 1 k ∑ i M ( a , b i ) (M\hat{\tau})_a=\frac{1}{k}\sum_i M(a,b_i) ( M τ ^ ) a = k 1 ∑ i M ( a , b i ) is an average of k k k independent random variables, each taking values in [ 0 , U ] [0,U] [ 0 , U ] . Each of these variables M ( a , b i ) M(a,b_i) M ( a , b i ) has the same mean, namely
E [ M ( a , b i ) ] = ∑ b ∈ [ C ] τ ∗ ( b ) M ( a , b ) = ( M τ ∗ ) a for every i , \mathbb{E}[M(a,b_i)]=\sum_{b\in[C]}\tau^*(b)M(a,b)=(M\tau^*)_a\qquad\text{for every }i, E [ M ( a , b i )] = b ∈ [ C ] ∑ τ ∗ ( b ) M ( a , b ) = ( M τ ∗ ) a for every i ,
because every b i b_i b i is drawn from the same distribution τ ∗ \tau^* τ ∗ ; averaging k k k copies of this same mean therefore gives
E [ ( M τ ^ ) a ] = 1 k ∑ i = 1 k E [ M ( a , b i ) ] = ( M τ ∗ ) a ≥ v + ϵ , \mathbb{E}[(M\hat{\tau})_a]=\frac{1}{k}\sum_{i=1}^{k}\mathbb{E}[M(a,b_i)]=(M\tau^*)_a\ge v+\epsilon, E [( M τ ^ ) a ] = k 1 i = 1 ∑ k E [ M ( a , b i )] = ( M τ ∗ ) a ≥ v + ϵ ,
i.e. ( M τ ^ ) a (M\hat{\tau})_a ( M τ ^ ) a has common mean ( M τ ∗ ) a (M\tau^*)_a ( M τ ∗ ) a . Hoeffding’s inequality [Theorem 2,[18]], in its standard form, bounds a sum S = ∑ i = 1 k X i S=\sum_{i=1}^{k}X_i S = ∑ i = 1 k X i of independent random variables X i ∈ [ α i , β i ] X_i\in[\alpha_i,\beta_i] X i ∈ [ α i , β i ] by Pr [ S − E ( S ) ≥ t ] ≤ exp ( − 2 t 2 ∑ i ( β i − α i ) 2 ) \Pr[S-\mathbb{E}(S)\ge t]\le\exp\left(-\frac{2t^2}{\sum_i(\beta_i-\alpha_i)^2}\right) Pr [ S − E ( S ) ≥ t ] ≤ exp ( − ∑ i ( β i − α i ) 2 2 t 2 ) for every t ≥ 0 t\ge0 t ≥ 0 : this bound holds simultaneously for every threshold t t t , so any particular value of t t t may be substituted into it freely; which value is useful is dictated only by what we actually want to bound. Applying this with X i = M ( a , b i ) ∈ [ 0 , U ] X_i=M(a,b_i)\in[0,U] X i = M ( a , b i ) ∈ [ 0 , U ] for every i i i , so ∑ i ( β i − α i ) 2 = k U 2 \sum_i(\beta_i-\alpha_i)^2=kU^2 ∑ i ( β i − α i ) 2 = k U 2 : what we want is a bound on the probability that the average ( M τ ^ ) a = S / k (M\hat{\tau})_a=S/k ( M τ ^ ) a = S / k deviates from its mean by at least ϵ \epsilon ϵ , i.e. on Pr [ S / k − E [ S / k ] ≥ ϵ ] \Pr[S/k-\mathbb{E}[S/k]\ge\epsilon] Pr [ S / k − E [ S / k ] ≥ ϵ ] . Since k > 0 k>0 k > 0 , multiplying this inequality through by k k k does not change which outcomes satisfy it, so
{ S k − E ( S ) k ≥ ϵ } = { S − E ( S ) ≥ k ϵ } \left\{\frac{S}{k}-\frac{\mathbb{E}(S)}{k}\ge\epsilon\right\}=\{S-\mathbb{E}(S)\ge k\epsilon\} { k S − k E ( S ) ≥ ϵ } = { S − E ( S ) ≥ k ϵ }
Hence Pr [ S / k − E [ S / k ] ≥ ϵ ] = Pr [ S − E ( S ) ≥ k ϵ ] \Pr[S/k-\mathbb{E}[S/k]\ge\epsilon]=\Pr[S-\mathbb{E}(S)\ge k\epsilon] Pr [ S / k − E [ S / k ] ≥ ϵ ] = Pr [ S − E ( S ) ≥ k ϵ ] . So we can choose t = k ϵ t=k\epsilon t = k ϵ to substitute: doing so in the bound above gives the upper-tail bound
Pr [ ( M τ ^ ) a − ( M τ ∗ ) a ≥ ϵ ] ≤ exp ( − 2 k ϵ 2 U 2 ) \Pr[(M\hat{\tau})_a-(M\tau^*)_a\ge\epsilon]\le\exp\left(-\frac{2k\epsilon^2}{U^2}\right) Pr [( M τ ^ ) a − ( M τ ∗ ) a ≥ ϵ ] ≤ exp ( − U 2 2 k ϵ 2 )
The mirror-image lower-tail statement, obtained identically by applying the same bound to − X i ∈ [ − U , 0 ] -X_i\in[-U,0] − X i ∈ [ − U , 0 ] (the same range width U U U ):
Pr [ ( M τ ^ ) a < ( M τ ∗ ) a − ϵ ] ≤ exp ( − 2 k ϵ 2 U 2 ) \Pr[(M\hat{\tau})_a<(M\tau^*)_a-\epsilon]\le\exp\left(-\frac{2k\epsilon^2}{U^2}\right) Pr [( M τ ^ ) a < ( M τ ∗ ) a − ϵ ] ≤ exp ( − U 2 2 k ϵ 2 )
Since ( M τ ∗ ) a ≥ v + ϵ (M\tau^*)_a\ge v+\epsilon ( M τ ∗ ) a ≥ v + ϵ , i.e. ( M τ ∗ ) a − ϵ ≥ v (M\tau^*)_a-\epsilon\ge v ( M τ ∗ ) a − ϵ ≥ v , the event ( M τ ^ ) a < v (M\hat{\tau})_a<v ( M τ ^ ) a < v forces ( M τ ^ ) a < ( M τ ∗ ) a − ϵ (M\hat{\tau})_a<(M\tau^*)_a-\epsilon ( M τ ^ ) a < ( M τ ∗ ) a − ϵ . So { ( M τ ^ ) a < v } ⊆ { ( M τ ^ ) a < ( M τ ∗ ) a − ϵ } \{(M\hat{\tau})_a<v\}\subseteq\{(M\hat{\tau})_a<(M\tau^*)_a-\epsilon\} {( M τ ^ ) a < v } ⊆ {( M τ ^ ) a < ( M τ ∗ ) a − ϵ } , and hence, by the Hoeffding bound above,
Pr [ ( M τ ^ ) a < v ] ≤ Pr [ ( M τ ^ ) a < ( M τ ∗ ) a − ϵ ] ≤ exp ( − 2 k ϵ 2 U 2 ) . \Pr[(M\hat{\tau})_a<v]\le\Pr[(M\hat{\tau})_a<(M\tau^*)_a-\epsilon]\le\exp\left(-\frac{2k\epsilon^2}{U^2}\right). Pr [( M τ ^ ) a < v ] ≤ Pr [( M τ ^ ) a < ( M τ ∗ ) a − ϵ ] ≤ exp ( − U 2 2 k ϵ 2 ) .
Taking a union bound over the R R R values of a a a ,
Pr [ ∃ a : ( M τ ^ ) a < v ] ≤ ∑ a ∈ [ R ] Pr [ ( M τ ^ ) a < v ] ≤ R exp ( − 2 k ϵ 2 U 2 ) < 1 \Pr[\exists a : (M\hat{\tau})_a < v] \le\sum_{a\in[R]} \Pr[(M\hat{\tau})_a < v] \le R\exp\left(-\frac{2k\epsilon^2}{U^2}\right) < 1 Pr [ ∃ a : ( M τ ^ ) a < v ] ≤ a ∈ [ R ] ∑ Pr [( M τ ^ ) a < v ] ≤ R exp ( − U 2 2 k ϵ 2 ) < 1
for the stated k k k : since U > 0 U>0 U > 0 here, k = max { 1 , ⌈ 2 U 2 ln ( 2 R ) / ϵ 2 ⌉ } = ⌈ 2 U 2 ln ( 2 R ) / ϵ 2 ⌉ k=\max\{1,\lceil2U^2\ln(2R)/\epsilon^2\rceil\}=\lceil2U^2\ln(2R)/\epsilon^2\rceil k = max { 1 , ⌈ 2 U 2 ln ( 2 R ) / ϵ 2 ⌉} = ⌈ 2 U 2 ln ( 2 R ) / ϵ 2 ⌉ (the maximum with 1 is automatic, since 2 U 2 ln ( 2 R ) / ϵ 2 > 0 2U^2\ln(2R)/\epsilon^2>0 2 U 2 ln ( 2 R ) / ϵ 2 > 0 and a ceiling of a positive number is always ≥ 1 \ge1 ≥ 1 ); a ceiling is always at least the number itself, so k ≥ 2 U 2 ln ( 2 R ) / ϵ 2 k\ge2U^2\ln(2R)/\epsilon^2 k ≥ 2 U 2 ln ( 2 R ) / ϵ 2 , i.e. 2 k ϵ 2 / U 2 ≥ 4 ln ( 2 R ) 2k\epsilon^2/U^2\ge4\ln(2R) 2 k ϵ 2 / U 2 ≥ 4 ln ( 2 R ) ; since exp ( ⋅ ) \exp(\cdot) exp ( ⋅ ) is increasing, negating both sides gives exp ( − 2 k ϵ 2 / U 2 ) ≤ exp ( − 4 ln ( 2 R ) ) \exp(-2k\epsilon^2/U^2)\le\exp(-4\ln(2R)) exp ( − 2 k ϵ 2 / U 2 ) ≤ exp ( − 4 ln ( 2 R )) , and multiplying by the positive constant R R R ,
R exp ( − 2 k ϵ 2 U 2 ) ≤ R exp ( − 4 ln ( 2 R ) ) . R\exp\left(-\frac{2k\epsilon^2}{U^2}\right)\le R\exp(-4\ln(2R)). R exp ( − U 2 2 k ϵ 2 ) ≤ R exp ( − 4 ln ( 2 R )) .
This right-hand side simplifies: since exp ( − 4 ln ( 2 R ) ) = exp ( ln ( ( 2 R ) − 4 ) ) = ( 2 R ) − 4 = 1 / ( 16 R 4 ) \exp(-4\ln(2R))=\exp(\ln((2R)^{-4}))=(2R)^{-4}=1/(16R^4) exp ( − 4 ln ( 2 R )) = exp ( ln (( 2 R ) − 4 )) = ( 2 R ) − 4 = 1/ ( 16 R 4 ) ,
R exp ( − 4 ln ( 2 R ) ) = R 16 R 4 = 1 16 R 3 ≤ 1 16 < 1 , R\exp(-4\ln(2R))=\frac{R}{16R^4}=\frac{1}{16R^3}\le\frac{1}{16}<1, R exp ( − 4 ln ( 2 R )) = 16 R 4 R = 16 R 3 1 ≤ 16 1 < 1 ,
using R ≥ 1 R\ge1 R ≥ 1 (a game always has at least one pure strategy) for the middle inequality.
Hence with strictly positive probability the sampled ( b 1 , … , b k ) (b_1,\ldots,b_k) ( b 1 , … , b k ) satisfies ( M τ ^ ) a ≥ v (M\hat{\tau})_a\ge v ( M τ ^ ) a ≥ v for every a a a simultaneously, so such a tuple exists. Since w i = 1 / k w_i=1/k w i = 1/ k for every i i i , this tuple is exactly a choice of b 1 , … , b k ∈ [ C ] b_1,\ldots,b_k\in[C] b 1 , … , b k ∈ [ C ] and weights w 1 , … , w k ≥ 0 w_1,\ldots,w_k\ge0 w 1 , … , w k ≥ 0 summing to 1 with ∑ i w i M ( a , b i ) ≥ v \sum_i w_iM(a,b_i)\ge v ∑ i w i M ( a , b i ) ≥ v for every a ∈ [ R ] a\in[R] a ∈ [ R ] , which is what was asserted in part (ii)’s first case. The symmetric statement is identical, sampling from an optimal σ ∗ \sigma^* σ ∗ and taking a union bound over the C C C values of b b b . The probability bound decreases with k k k , so the proof also covers every larger integer k k k .
Applied here with U = log 2 n U=\log_2 n U = log 2 n (every value arising below lies in [ 0 , log 2 n ] [0,\log_2 n] [ 0 , log 2 n ] , Proposition 16 (i)) and R , C ≤ A n R,C\le A^n R , C ≤ A n , so ln ( 2 R ) , ln ( 2 C ) ≤ n ln ( 2 A ) \ln(2R),\ln(2C)\le n\ln(2A) ln ( 2 R ) , ln ( 2 C ) ≤ n ln ( 2 A ) , Lemma 8 gives ϵ \epsilon ϵ -optimal sparse strategies of support size
k ( ϵ ) = O ( 1 + n ( log n ) 2 log ( 2 A ) ϵ 2 ) , (53) k(\epsilon)=O\left(1+\frac{n(\log n)^2\log(2A)}{\epsilon^2}\right), \tag*{(53)} k ( ϵ ) = O ( 1 + ϵ 2 n ( log n ) 2 log ( 2 A ) ) , (53)
polynomial in n , log ( 2 A ) , 1 / ϵ n,\log(2A),1/\epsilon n , log ( 2 A ) , 1/ ϵ despite R , C R,C R , C being exponential in n n n .
Compact Encoding of a Pure Profile
A pure profile a ∈ P min a\in\mathcal{P}_{\min} a ∈ P m i n has an O ( n log ( 2 A ) ) O(n\log(2A)) O ( n log ( 2 A )) -bit description: fix an ordering of each enabled action set and store one action index per controlled state. Because this description is short, profiles can be enumerated one at a time, in this order, without ever writing out or storing the full set P min \mathcal{P}_{\min} P m i n . If we have no state sets, then we have one empty profile. A tuple of k k k profiles takes O ( k n log ( 2 A ) ) O(kn\log(2A)) O ( k n log ( 2 A )) bits, regardless of repetitions.
The Polynomial-Space Bound
We separate the proof into six steps, each closing one gap between the abstract value V s p V_{\mathrm{sp}} V sp and an actual computation. Step 1 pins down what a finite-horizon version of V s p V_{\mathrm{sp}} V sp means, fixes a specific horizon length T T T , and bounds the error from stopping after these T T T rounds; every later step works within this fixed horizon. Step 2 replaces the exact matrix game at each round, which may need a mixed profile with exponential support to solve exactly, by a finite empirical choice, using Lemma 8. Step 3 turns this into arithmetic a machine can actually execute: exact rational operations combined with rounding to a fixed number of bits. Step 4 tracks how much error the roundings and empirical approximations of Steps 2-3 can accumulate over the whole recursion, proving a single uniform bound. Step 5 uses that bound to show that comparing the final computed value against v v v correctly decides the promise gap. Step 6 checks that the whole computation, not just its final answer, fits in polynomial space, by bounding the bit length of every intermediate quantity and the number of recursive frames active at once.
Proof of Theorem 9. Let n = ∣ S ∣ n = |S| n = ∣ S ∣ , A = max s ∈ S ∣ A ( s ) ∣ A = \max_{s\in S} |A(s)| A = max s ∈ S ∣ A ( s ) ∣ , e = min { ε 0 , 1 } e = \min\{\varepsilon_0,1\} e = min { ε 0 , 1 } , and, for n ≥ 2 n \ge2 n ≥ 2 , L = ⌈ log 2 n ⌉ L = \lceil\log_2 n\rceil L = ⌈ log 2 n ⌉ ; let ℓ \ell ℓ be the total bit length of the binary encoding of the arena, its rational transition probabilities Δ ( s , a , s ′ ) \Delta(s,a,s') Δ ( s , a , s ′ ) , the rational initial distribution μ 0 \mu_0 μ 0 , and the rational threshold v v v . For n = 1 n = 1 n = 1 compare the identically zero value with v v v . Assume n ≥ 2 n \ge2 n ≥ 2 below.
Step 1: finite horizons and their semantic meaning. The value V s p V_{\mathrm{sp}} V sp is defined over infinitely many rounds, so before any computation can begin, the infinite horizon has to be replaced by a finite one, together with a provable bound on how much accuracy this costs. This step does exactly that: it defines the finite-horizon value W r s p W^{\mathrm{sp}}_r W r sp , bounds how far it can drift from the true value V s p V_{\mathrm{sp}} V sp as a function of the number of rounds kept, and then fixes a specific horizon T T T small enough to keep this drift within the overall error budget.
For each integer r ≥ 0 r \ge0 r ≥ 0 , let W r s p ( μ ) W^{\mathrm{sp}}_r(\mu) W r sp ( μ ) be the value of the r r r -round truncated game started at μ \mu μ , in which only the first r r r rounds earn a reward: Theorem 4 shows this value is well defined (the upper and lower values, over strategies depending on the full observed distribution history, coincide) and (see also the finite-horizon identification in the proof of Theorem 1 , Appendix C) satisfies
W 0 s p ( μ ) = 0 , W^{\mathrm{sp}}_0(\mu) = 0, W 0 sp ( μ ) = 0 ,
W r s p ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( [ W r − 1 s p ( Φ a , b ( μ ) ) ] a , b ) . (54) W^{\mathrm{sp}}_r(\mu) = (1-\gamma)\mathrm{H}(\mu) + \gamma\operatorname{val}\left(\left[W^{\mathrm{sp}}_{r-1}\left(\Phi_{a,b}(\mu)\right)\right]_{a,b}\right). \tag*{(54)} W r sp ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( [ W r − 1 sp ( Φ a , b ( μ ) ) ] a , b ) . (54)
Thus W 1 s p = ( 1 − γ ) H W^{\mathrm{sp}}_1 = (1-\gamma)\mathrm{H} W 1 sp = ( 1 − γ ) H needs no action, and r ≥ 2 r \ge2 r ≥ 2 uses a continuation with r − 1 r-1 r − 1 remaining rewards. The index r r r counts down: it is the number of reward-earning rounds still to come, not the number already played, so W T s p W^{\mathrm{sp}}_T W T sp , evaluated at the start of play, is the value of the full T T T -round truncated game, while W 0 s p ≡ 0 W^{\mathrm{sp}}_0 \equiv0 W 0 sp ≡ 0 is reached once no rounds remain. This distinction matters directly for Step 3 below: the recursive algorithm there is written in terms of this remaining horizon r r r , decreasing from T T T down to the base case r = 0 r = 0 r = 0 at which no further call is made, so the stack depth of active calls in Step 6’s space bound equals the remaining-horizon count at the point of recursion, not elapsed real time. Indexing by elapsed time instead would need a separate counter to detect when to stop.
The Borel Markov saddle policies that Theorem 4 supplies for each W r s p W^{\mathrm{sp}}_r W r sp are optimal against all distribution-history opponents, not just Markov ones. Hence (54) is not a recursion in which previous sampled profiles become observable: its value depends only on the current distribution and the remaining horizon.
Monotonicity and stability of val. To bound the truncation error, we first record an elementary stability property of val \operatorname{val} val : for matrices M , N M,N M , N of the same dimensions, ∣ val ( M ) − val ( N ) ∣ ≤ max a , b ∣ M a , b − N a , b ∣ |\operatorname{val}(M)-\operatorname{val}(N)| \le\max_{a,b}|M_{a,b}-N_{a,b}| ∣ val ( M ) − val ( N ) ∣ ≤ max a , b ∣ M a , b − N a , b ∣ , and val \operatorname{val} val is monotone: M ≥ N M \ge N M ≥ N entrywise implies val ( M ) ≥ val ( N ) \operatorname{val}(M) \ge\operatorname{val}(N) val ( M ) ≥ val ( N ) . Both follow directly from the first form of (52) , val ( M ) = max τ min a ( M τ ) a \operatorname{val}(M) = \max_{\tau}\min_a(M\tau)_a val ( M ) = max τ min a ( M τ ) a : for monotonicity, if M ≥ N M \ge N M ≥ N entrywise then ( M τ ) a ≥ ( N τ ) a (M\tau)_a \ge(N\tau)_a ( M τ ) a ≥ ( N τ ) a for every a , τ a,\tau a , τ , so min a ( M τ ) a ≥ min a ( N τ ) a \min_a(M\tau)_a \ge\min_a(N\tau)_a min a ( M τ ) a ≥ min a ( N τ ) a for every τ \tau τ , and taking the max over τ \tau τ on both sides preserves the inequality; for the stability bound, write Δ = max a , b ∣ M a , b − N a , b ∣ \Delta= \max_{a,b}|M_{a,b}-N_{a,b}| Δ = max a , b ∣ M a , b − N a , b ∣ , and let 1 \mathbf{1} 1 denote the all-ones matrix of the same dimensions as M , N M,N M , N , so that Δ 1 \Delta\mathbf{1} Δ 1 is the matrix with every entry equal to $\Delta. Then N − Δ 1 ≤ M ≤ N + Δ 1 N-\Delta1 \le M \le N+\Delta1 N − Δ1 ≤ M ≤ N + Δ1 entrywise, and monotonicity applied to each side, together with val ( N ± Δ 1 ) = val ( N ) ± Δ \operatorname{val}(N\pm\Delta1)=\operatorname{val}(N)\pm\Delta val ( N ± Δ1 ) = val ( N ) ± Δ (adding a constant to every entry adds the same constant to every ( M τ ) a (M_\tau)_a ( M τ ) a and hence to val \operatorname{val} val ), gives val ( N ) − Δ ≤ val ( M ) ≤ val ( N ) + Δ \operatorname{val}(N)-\Delta\le\operatorname{val}(M)\le\operatorname{val}(N)+\Delta val ( N ) − Δ ≤ val ( M ) ≤ val ( N ) + Δ .
Bounding the truncation error. By the paragraphs “Finite-horizon identification” and “Infinite-horizon identification and error bound” of the proof of Theorem 1 (Section C), W r s p = ( T s p ) r 0 W_r^{\mathrm{sp}}=(T_{\mathrm{sp}})^r0 W r sp = ( T sp ) r 0 and 0 ≤ V s p − W r s p ≤ γ r log 2 ∣ S ∣ ≤ γ r L 0\le V_{\mathrm{sp}}-W_r^{\mathrm{sp}}\le\gamma^r\log_2|S|\le\gamma^rL 0 ≤ V sp − W r sp ≤ γ r log 2 ∣ S ∣ ≤ γ r L for every r ≥ 0 r\ge0 r ≥ 0 (equivalently, V s p = T s p V s p V_{\mathrm{sp}}=T_{\mathrm{sp}}V_{\mathrm{sp}} V sp = T sp V sp , the cancellation of the entropy terms, monotonicity and the stability bound above give 0 ≤ Δ r ≤ γ sup Δ r − 1 0\le\Delta_r\le\gamma\sup\Delta_{r-1} 0 ≤ Δ r ≤ γ sup Δ r − 1 for Δ r = V s p − W r s p \Delta_r=V_{\mathrm{sp}}-W_r^{\mathrm{sp}} Δ r = V sp − W r sp , with Δ 0 = V s p ∈ [ 0 , L ] \Delta_0=V_{\mathrm{sp}}\in[0,L] Δ 0 = V sp ∈ [ 0 , L ] ). In particular, for every μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) ,
0 ≤ V s p ( μ ) − W T s p ( μ ) ≤ γ T L . (55) 0\le V_{\mathrm{sp}}(\mu)-W_T^{\mathrm{sp}}(\mu)\le\gamma^T L. \tag*{(55)} 0 ≤ V sp ( μ ) − W T sp ( μ ) ≤ γ T L . (55)
The horizon T T T . Since γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and e ≤ 1 e\le1 e ≤ 1 are fixed and L ≥ 1 L\ge1 L ≥ 1 (as n ≥ 2 n\ge2 n ≥ 2 ), the least positive integer T T T with γ T L ≤ e / 4 \gamma^T L\le e/4 γ T L ≤ e /4 exists and can be found by exact rational comparisons for T = 1 , 2 , … T=1,2,\ldots T = 1 , 2 , … . With c γ : = 1 / log ( 1 / γ ) c_\gamma:=1/\log(1/\gamma) c γ := 1/ log ( 1/ γ ) , the condition is equivalent to T ≥ c γ log ( 4 L / e ) T\ge c_\gamma\log(4L/e) T ≥ c γ log ( 4 L / e ) , so
T = ⌈ c γ log ( 4 L / e ) ⌉ ≤ c γ log ( 4 L / e ) + 1 = O ( 1 + log ( L / e ) ) . T=\lceil c_\gamma\log(4L/e)\rceil\le c_\gamma\log(4L/e)+1=O(1+\log(L/e)). T = ⌈ c γ log ( 4 L / e )⌉ ≤ c γ log ( 4 L / e ) + 1 = O ( 1 + log ( L / e )) .
Range of the finite-horizon values. For any matrix M M M with entries in [ 0 , U ] [0,U] [ 0 , U ] we have val ( M ) ∈ [ 0 , U ] \operatorname{val}(M)\in[0,U] val ( M ) ∈ [ 0 , U ] : by the first form of (52) , val ( M ) = max τ min a ( M τ ) a \operatorname{val}(M)=\max_\tau\min_a(M_\tau)_a val ( M ) = max τ min a ( M τ ) a , and each ( M τ ) a (M_\tau)_a ( M τ ) a is a convex combination of entries in [ 0 , U ] [0,U] [ 0 , U ] . Hence, by induction on r r r using (54) and H ∈ [ 0 , log 2 n ] H\in[0,\log_2 n] H ∈ [ 0 , log 2 n ] (Proposition 16 (i)), W r s p ∈ [ 0 , log 2 n ] W_r^{\mathrm{sp}}\in[0,\log_2 n] W r sp ∈ [ 0 , log 2 n ] for every r r r (this is also immediate from W r s p W_r^{\mathrm{sp}} W r sp being a value of a payoff in [ 0 , log 2 n ] [0,\log_2 n] [ 0 , log 2 n ] ). Consequently, for every r ≥ 1 r\ge1 r ≥ 1 and μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) the matrix [ W r − 1 s p ( Φ a , b ( μ ) ) ] a , b [W_{r-1}^{\mathrm{sp}}(\Phi_{a,b}(\mu))]_{a,b} [ W r − 1 sp ( Φ a , b ( μ )) ] a , b has all entries in [ 0 , log 2 n ] ⊆ [ 0 , L ] [0,\log_2 n]\subseteq[0,L] [ 0 , log 2 n ] ⊆ [ 0 , L ] , which is exactly the boundedness hypothesis of Lemma 8 (in its part (ii) the bound controls the deviation of the empirical average from its mean).
Step 2: a finite empirical replacement for the matrix value
Step 1 reduced V s p ( μ 0 ) V_{\mathrm{sp}}(\mu_0) V sp ( μ 0 ) to evaluating W T s p W_T^{\mathrm{sp}} W T sp via a finite recursion, but each round of that recursion still asks for val ( M ) \operatorname{val}(M) val ( M ) of a matrix indexed by P min × P max \mathcal{P}_{\min}\times\mathcal{P}_{\max} P m i n × P m a x , sets that can have up to A n A^n A n rows and columns: computing val ( M ) \operatorname{val}(M) val ( M ) exactly would mean optimising over a mixed strategy with possibly exponential support. This step replaces that exact computation with a finite empirical one, s k ( M ) s_k(M) s k ( M ) , that searches only over k k k -tuples of pure profiles; Lemma 8 guarantees this gets within a controlled error α \alpha α of the true value and shows this replacement is itself stable: nearby matrices produce nearby empirical values, which is exactly what lets Step 4 track error through the whole recursion later.
Put α = ( 1 − γ ) e / 8 \alpha=(1-\gamma)e/8 α = ( 1 − γ ) e /8 and define integers
J A = 1 + n ⌈ log 2 ( max { 2 , A } ) ⌉ , k = max { 2 , ⌈ 2 L 2 J A α 2 ⌉ } . (56) J_A=1+n\lceil\log_2(\max\{2,A\})\rceil,\qquad
k=\max\left\{2,\left\lceil\frac{2L^2J_A}{\alpha^2}\right\rceil\right\}. \tag*{(56)} J A = 1 + n ⌈ log 2 ( max { 2 , A })⌉ , k = max { 2 , ⌈ α 2 2 L 2 J A ⌉ } . (56)
We show that k k k meets the row-mixture bound in Lemma 8; it is polynomial in n , log ( 2 A ) , 1 / e n,\log(2A),1/e n , log ( 2 A ) , 1/ e for fixed γ \gamma γ , and its construction needs only rational arithmetic and integer binary lengths. Since ∣ P max ∣ ≤ A n |\mathcal{P}_{\max}|\le A^n ∣ P m a x ∣ ≤ A n (Appendix § 15 .2.1), ln ( 2 ∣ P max ∣ ) ≤ ln ( 2 A n ) = ln 2 + n ln A \ln(2|\mathcal{P}_{\max}|)\le\ln(2A^n)=\ln2+n\ln A ln ( 2∣ P m a x ∣ ) ≤ ln ( 2 A n ) = ln 2 + n ln A . If A ≥ 2 A\ge2 A ≥ 2 , then ⌈ log 2 A ⌉ ≥ ln A / ln 2 \lceil\log_2 A\rceil\ge\ln A/\ln2 ⌈ log 2 A ⌉ ≥ ln A / ln 2 , so n ⌈ log 2 A ⌉ > n ln A n\lceil\log_2 A\rceil>n\ln A n ⌈ log 2 A ⌉ > n ln A (as 1 / ln 2 > 1 1/\ln2>1 1/ ln 2 > 1 ), giving J A = 1 + n ⌈ log 2 A ⌉ > 1 + n ln A > ln 2 + n ln A J_A=1+n\lceil\log_2 A\rceil>1+n\ln A>\ln2+n\ln A J A = 1 + n ⌈ log 2 A ⌉ > 1 + n ln A > ln 2 + n ln A ; if A = 1 A=1 A = 1 , then max { 2 , A } = 2 \max\{2,A\}=2 max { 2 , A } = 2 and J A = 1 + n > ln 2 = ln ( 2 ⋅ 1 n ) J_A=1+n>\ln2=\ln(2\cdot1^n) J A = 1 + n > ln 2 = ln ( 2 ⋅ 1 n ) trivially. In either case J A ≥ ln ( 2 ∣ P max ∣ ) J_A\ge\ln(2|\mathcal{P}_{\max}|) J A ≥ ln ( 2∣ P m a x ∣ ) , so 2 L 2 J A / α 2 ≥ 2 L 2 ln ( 2 ∣ P max ∣ ) / α 2 2L^2J_A/\alpha^2\ge2L^2\ln(2|\mathcal{P}_{\max}|)/\alpha^2 2 L 2 J A / α 2 ≥ 2 L 2 ln ( 2∣ P m a x ∣ ) / α 2 , and k k k , being at least ⌈ 2 L 2 J A / α 2 ⌉ \lceil2L^2J_A/\alpha^2\rceil ⌈ 2 L 2 J A / α 2 ⌉ , is at least the row-mixture bound max { 1 , ⌈ 2 L 2 ln ( 2 ∣ P max ∣ ) / α 2 ⌉ } \max\{1,\lceil2L^2\ln(2|\mathcal{P}_{\max}|)/\alpha^2\rceil\} max { 1 , ⌈ 2 L 2 ln ( 2∣ P m a x ∣ ) / α 2 ⌉} that the row-player case of Lemma 8(ii) requires (with U = L U=L U = L , C = ∣ P max ∣ C=|\mathcal{P}_{\max}| C = ∣ P m a x ∣ , ϵ = α \epsilon=\alpha ϵ = α ).
For the polynomial bound: L = ⌈ log 2 n ⌉ = O ( log n ) L=\lceil\log_2n\rceil=O(\log n) L = ⌈ log 2 n ⌉ = O ( log n ) , and J A = 1 + n ⌈ log 2 ( max { 2 , A } ) ⌉ = O ( n log ( 2 A ) ) J_A=1+n\lceil\log_2(\max\{2,A\})\rceil=O(n\log(2A)) J A = 1 + n ⌈ log 2 ( max { 2 , A })⌉ = O ( n log ( 2 A )) . Since γ \gamma γ is fixed and e = min { ϵ 0 , 1 } e=\min\{\epsilon_0,1\} e = min { ϵ 0 , 1 } , α = ( 1 − γ ) e / 8 = Θ ( e ) \alpha=(1-\gamma)e/8=\Theta(e) α = ( 1 − γ ) e /8 = Θ ( e ) , so
k = max { 2 , ⌈ 2 L 2 J A α 2 ⌉ } = O ( ( log n ) 2 ⋅ n log ( 2 A ) e 2 ) , (57) k=\max\left\{2,\left\lceil\frac{2L^2J_A}{\alpha^2}\right\rceil\right\}
=O\left(\frac{(\log n)^2\cdot n\log(2A)}{e^2}\right), \tag*{(57)} k = max { 2 , ⌈ α 2 2 L 2 J A ⌉ } = O ( e 2 ( log n ) 2 ⋅ n log ( 2 A ) ) , (57)
polynomial in n n n , log ( 2 A ) \log(2A) log ( 2 A ) , 1 / e 1/e 1/ e . Both γ \gamma γ and e e e are themselves fixed rational constants of the promise problem by Appendix § 15 ; the 1 / e 1/e 1/ e dependence is displayed explicitly to show how k k k scales for other fixed choices of the gap ε 0 \varepsilon_0 ε 0 .
For any matrix M M M with these profile indices, define the finite optimisation over exactly the uniform-weight empirical laws that Lemma 8(ii) shows suffice (every weight equal to 1 / k 1/k 1/ k , profiles allowed to repeat):
s k ( M ) = min ( a 1 , … , a k ) ∈ P min k max b ∈ P max 1 k ∑ i = 1 k M a i , b . s_k(M)=\min_{(a_1,\ldots,a_k)\in\mathcal{P}_{\min}^k}\max_{b\in\mathcal{P}_{\max}}\frac{1}{k}\sum_{i=1}^{k}M_{a_i,b}. s k ( M ) = ( a 1 , … , a k ) ∈ P m i n k min b ∈ P m a x max k 1 i = 1 ∑ k M a i , b .
The tuple represents a legitimate mixed strategy for Min, σ = k − 1 ∑ i δ a i ∈ D ( P min ) \sigma=k^{-1}\sum_i\delta_{a_i}\in\mathcal{D}(\mathcal{P}_{\min}) σ = k − 1 ∑ i δ a i ∈ D ( P m i n ) : the uniform distribution over the (possibly repeated) profiles a 1 , … , a k a_1,\ldots,a_k a 1 , … , a k . Call any mixed strategy of this form, arising as the uniform distribution of some size-k k k sample of pure profiles, empirical . Because σ \sigma σ ranges over whole profiles rather than independent per-state choices, it may correlate actions across a player’s different controlled states, exactly as the sampled-profile semantics permits.
For entries in [ 0 , L ] [0,L] [ 0 , L ] Lemma 8 gives
val ( M ) ≤ s k ( M ) ≤ val ( M ) + α . (58) \operatorname{val}(M)\le s_k(M)\le\operatorname{val}(M)+\alpha. \tag*{(58)} val ( M ) ≤ s k ( M ) ≤ val ( M ) + α . (58)
The first inequality holds because empirical mixed strategies are a subset of all of D ( P min ) \mathcal{D}(\mathcal{P}_{\min}) D ( P m i n ) . For σ = k − 1 ∑ i δ a i \sigma=k^{-1}\sum_i\delta_{a_i} σ = k − 1 ∑ i δ a i as above, ( σ T M ) b = ∑ a ∈ P min σ ( a ) M ( a , b ) = 1 k ∑ i = 1 k M ( a i , b ) (\sigma^{\mathsf T}M)_b=\sum_{a\in\mathcal{P}_{\min}}\sigma(a)M(a,b)=\frac{1}{k}\sum_{i=1}^{k}M(a_i,b) ( σ T M ) b = ∑ a ∈ P m i n σ ( a ) M ( a , b ) = k 1 ∑ i = 1 k M ( a i , b ) for every b b b (the same regrouping used in the proof of Lemma 8(i)), so
s k ( M ) = min ( a 1 , … , a k ) ∈ P min k max b 1 k ∑ i = 1 k M ( a i , b ) = min ( a 1 , … , a k ) ∈ P min k max b ( σ T M ) b = min σ e m p i r i c a l max b ( σ T M ) b . \begin{aligned}
s_k(M)&=\min_{(a_1,\ldots,a_k)\in\mathcal{P}_{\min}^k}\max_b\frac{1}{k}\sum_{i=1}^{k}M(a_i,b)\\
&=\min_{(a_1,\ldots,a_k)\in\mathcal{P}_{\min}^k}\max_b(\sigma^{\mathsf T}M)_b
=\min_{\sigma\ \mathrm{empirical}}\max_b(\sigma^{\mathsf T}M)_b.
\end{aligned} s k ( M ) = ( a 1 , … , a k ) ∈ P m i n k min b max k 1 i = 1 ∑ k M ( a i , b ) = ( a 1 , … , a k ) ∈ P m i n k min b max ( σ T M ) b = σ empirical min b max ( σ T M ) b .
where the last minimisation ranges only over the empirical σ \sigma σ ’s of this uniform form. Restricting the minimisation in the second form of (52) to this subset of D ( P min ) \mathcal{D}(\mathcal{P}_{\min}) D ( P m i n ) can only raise its value (minimising over fewer candidates gives a result at least as large), giving s k ( M ) ≥ val ( M ) s_k(M)\ge\operatorname{val}(M) s k ( M ) ≥ val ( M ) .
For the second inequality, apply Lemma 8(ii) (the row-player case) to M M M with ϵ = α \epsilon=\alpha ϵ = α and v = val ( M ) + α v=\operatorname{val}(M)+\alpha v = val ( M ) + α : the hypothesis val ( M ) ≤ v − ϵ \operatorname{val}(M)\le v-\epsilon val ( M ) ≤ v − ϵ holds with equality, so the lemma supplies a 1 , … , a k ∈ P min a_1,\ldots,a_k\in\mathcal{P}_{\min} a 1 , … , a k ∈ P m i n (repeats allowed) with equal weights 1 / k 1/k 1/ k such that 1 k ∑ i M ( a i , b ) ≤ v = val ( M ) + α \frac{1}{k}\sum_i M(a_i,b)\le v=\operatorname{val}(M)+\alpha k 1 ∑ i M ( a i , b ) ≤ v = val ( M ) + α for every b ∈ P max b\in\mathcal{P}_{\max} b ∈ P m a x simultaneously, provided k k k is at least the lemma’s general bound ⌈ 2 U 2 ln ( 2 ∣ P max ∣ ) / α 2 ⌉ \lceil2U^2\ln(2|\mathcal{P}_{\max}|)/\alpha^2\rceil ⌈ 2 U 2 ln ( 2∣ P m a x ∣ ) / α 2 ⌉ specialized to U = L U=L U = L , which is valid since M M M has entries in [ 0 , L ] [0,L] [ 0 , L ] as shown in Step 1: this specialization gives k ≥ ⌈ 2 L 2 ln ( 2 ∣ P max ∣ ) / α 2 ⌉ k\ge\lceil2L^2\ln(2|\mathcal{P}_{\max}|)/\alpha^2\rceil k ≥ ⌈ 2 L 2 ln ( 2∣ P m a x ∣ ) / α 2 ⌉ . Since J A ≥ ln ( 2 ∣ P max ∣ ) J_A\ge\ln(2|\mathcal{P}_{\max}|) J A ≥ ln ( 2∣ P m a x ∣ ) , the k k k fixed in (56) already meets this, and the lemma’s closing remark extends the guarantee to this (possibly larger) k k k . This tuple is one feasible choice in the minimum defining s k ( M ) s_k(M) s k ( M ) , so s k ( M ) ≤ max b 1 k ∑ i M ( a i , b ) ≤ val ( M ) + α s_k(M)\le\max_b\frac{1}{k}\sum_iM(a_i,b)\le\operatorname{val}(M)+\alpha s k ( M ) ≤ max b k 1 ∑ i M ( a i , b ) ≤ val ( M ) + α .
For any two matrices of the same dimensions, averaging, finite maxima, and finite minima also give
∣ s k ( M ) − s k ( N ) ∣ ≤ max a , b ∣ M a , b − N a , b ∣ . (59) \left|s_k(M)-s_k(N)\right|\le\max_{a,b}|M_{a,b}-N_{a,b}|. \tag*{(59)} ∣ s k ( M ) − s k ( N ) ∣ ≤ a , b max ∣ M a , b − N a , b ∣. (59)
To verify this, write Δ = max a , b ∣ M a , b − N a , b ∣ \Delta=\max_{a,b}|M_{a,b}-N_{a,b}| Δ = max a , b ∣ M a , b − N a , b ∣ , so N − Δ 1 ≤ M ≤ N + Δ 1 N-\Delta\mathbf{1}\le M\le N+\Delta\mathbf{1} N − Δ 1 ≤ M ≤ N + Δ 1 entrywise (with 1 \mathbf{1} 1 the all-ones matrix of the same dimensions). Two elementary facts about s k s_k s k , proved the same way as for val above: it is monotone (M ≥ N M\ge N M ≥ N entrywise implies s k ( M ) ≥ s k ( N ) s_k(M)\ge s_k(N) s k ( M ) ≥ s k ( N ) ), since for every fixed tuple and column, 1 k ∑ i M a i , b ≥ 1 k ∑ i N a i , b \frac{1}{k}\sum_iM_{a_i,b}\ge\frac{1}{k}\sum_iN_{a_i,b} k 1 ∑ i M a i , b ≥ k 1 ∑ i N a i , b , so the max over columns and then the min over tuples both preserve the inequality, and adding a constant to every entry shifts s k s_k s k by that same constant (s k ( N ± Δ 1 ) = s k ( N ) ± Δ s_k(N\pm\Delta\mathbf{1})=s_k(N)\pm\Delta s k ( N ± Δ 1 ) = s k ( N ) ± Δ , since shifting every entry by $\Delta shifts every average 1 k ∑ i M a i , b \frac{1}{k}\sum_i M_{a_i,b} k 1 ∑ i M a i , b , hence its max over b b b , hence the min over tuples, by exactly Δ \Delta Δ ). Applying monotonicity to each side of N − Δ 1 ≤ M ≤ N + Δ 1 N-\Delta\mathbf{1}\le M\le N+\Delta\mathbf{1} N − Δ 1 ≤ M ≤ N + Δ 1 , together with these shift identities, gives s k ( N ) − Δ ≤ s k ( M ) ≤ s k ( N ) + Δ s_k(N)-\Delta\le s_k(M)\le s_k(N)+\Delta s k ( N ) − Δ ≤ s k ( M ) ≤ s k ( N ) + Δ , i.e.
∣ s k ( M ) − s k ( N ) ∣ ≤ Δ = max a , b ∣ M a , b − N a , b ∣ . \left|s_k(M)-s_k(N)\right|\le\Delta=\max_{a,b}\left|M_{a,b}-N_{a,b}\right|. ∣ s k ( M ) − s k ( N ) ∣ ≤ Δ = a , b max ∣ M a , b − N a , b ∣ .
This elementary stability property will carry child evaluation errors through the recursion.
Step 3: finite-precision recursive evaluation. Steps 1-2 replaced the infinite horizon with a finite one, and the exact matrix game at each round with a finite empirical search over s k s_k s k , but both are still stated in terms of exact real numbers: the entropy reward H ( μ ) H(\mu) H ( μ ) is transcendental, and s k ( M ) s_k(M) s k ( M ) as defined is an exact minimum and maximum of real-valued sums.
We now build a numerical recursion W ^ r s p \widehat{W}^{\mathrm{sp}}_r W r sp , evaluated entirely in rational arithmetic, that will approximate W r s p W^{\mathrm{sp}}_r W r sp closely enough for Step 5’s promise-correctness decision to be correct. To keep every intermediate value at a bounded, fixed number of bits, which Step 6 needs for its space bound, every quantity this recursion produces is rounded to the nearest multiple of a fixed precision d = 2 − p d=2^{-p} d = 2 − p , rather than kept exact. Choose the least integer p ≥ 1 p\ge1 p ≥ 1 such that d = 2 − p ≤ ( 1 − γ ) ε / 16 d=2^{-p}\le(1-\gamma)\varepsilon/16 d = 2 − p ≤ ( 1 − γ ) ε /16 ; this specific threshold is not arbitrary, it is exactly what makes the total rounding error come out within budget once Step 4 proves the error bound. With p p p fixed this way, every recursive call of W ^ r s p \widehat{W}^{\mathrm{sp}}_r W r sp will return a value that is a multiple of d d d lying in [ 0 , L ] [0,L] [ 0 , L ] , and the following rounding operation is what enforces this at each step.
Define the rational rounding operation
R p ( z ) = 2 − p min { L 2 p , max { 0 , ⌊ 2 p z + 1 2 ⌋ } } . R_p(z)=2^{-p}\min\left\{L2^p,\max\left\{0,\left\lfloor2^p z+\frac{1}{2}\right\rfloor\right\}\right\}. R p ( z ) = 2 − p min { L 2 p , max { 0 , ⌊ 2 p z + 2 1 ⌋ } } .
For every w ∈ [ 0 , L ] w\in[0,L] w ∈ [ 0 , L ] it satisfies the clipping inequality
∣ R p ( z ) − w ∣ ≤ ∣ z − w ∣ + d : \left|R_p(z)-w\right|\le|z-w|+d: ∣ R p ( z ) − w ∣ ≤ ∣ z − w ∣ + d :
rounding changes a number by at most d / 2 d/2 d /2 , and clipping to [ 0 , L ] [0,L] [ 0 , L ] cannot increase its distance from w w w . All comparisons and rounding operations here are on rationals.
The current entropy H ( μ ) H(\mu) H ( μ ) itself need not be computed exactly: by Lemma 6 (effective dyadic evaluation, Part 2), for every integer p ≥ 1 p\ge1 p ≥ 1 there is a rational H ^ p ( μ ) ∈ [ 0 , L ] \widehat{H}_p(\mu)\in[0,L] H p ( μ ) ∈ [ 0 , L ] with ∣ H ^ p ( μ ) − H ( μ ) ∣ ≤ 2 − p = d \left|\widehat{H}_p(\mu)-H(\mu)\right|\le2^{-p}=d H p ( μ ) − H ( μ ) ≤ 2 − p = d , computable in time polynomial in size( μ ) + p (\mu)+p ( μ ) + p . We use this H ^ p \widehat{H}_p H p below in place of H H H .
For rational μ \mu μ , define W ^ 0 s p ( μ ) = 0 \widehat{W}^{\mathrm{sp}}_0(\mu)=0 W 0 sp ( μ ) = 0 and
W ^ r s p ( μ ) = R p ( ( 1 − γ ) H ^ p ( μ ) + γ min a ⃗ ∈ P min k max b ∈ P max 1 k ∑ i = 1 k W ^ r − 1 s p ( Φ a i , b ( μ ) ) ) . (60) \widehat{W}^{\mathrm{sp}}_r(\mu)=R_p\left((1-\gamma)\widehat{H}_p(\mu)+\gamma\min_{\vec{a}\in\mathcal{P}_{\min}^k}\max_{b\in\mathcal{P}_{\max}}\frac{1}{k}\sum_{i=1}^{k}\widehat{W}^{\mathrm{sp}}_{r-1}\left(\Phi_{a_i,b}(\mu)\right)\right). \tag*{(60)} W r sp ( μ ) = R p ( ( 1 − γ ) H p ( μ ) + γ a ∈ P m i n k min b ∈ P m a x max k 1 i = 1 ∑ k W r − 1 sp ( Φ a i , b ( μ ) ) ) . (60)
This describes exactly how the nested min a ⃗ max b 1 k ∑ i \min_{\vec{a}}\max_b\frac{1}{k}\sum_i min a max b k 1 ∑ i of (60) is evaluated procedurally.
The r = 1 r=1 r = 1 shortcut. Every child call is W ^ 0 s p ( ⋅ ) ≡ 0 \widehat{W}^{\mathrm{sp}}_0(\cdot)\equiv0 W 0 sp ( ⋅ ) ≡ 0 regardless of its argument, so the whole min max ∑ \min\max\sum min max ∑ term is 0 0 0 without needing to try any tuple or column at all, and W ^ 1 s p ( μ ) = R p ( ( 1 − γ ) H ^ p ( μ ) ) \widehat{W}^{\mathrm{sp}}_1(\mu)=R_p((1-\gamma)\widehat{H}_p(\mu)) W 1 sp ( μ ) = R p (( 1 − γ ) H p ( μ )) directly.
Outer loop, r ≥ 2 r\ge2 r ≥ 2 . Try every k k k -tuple a ⃗ = ( a 1 , … , a k ) ∈ P min k \vec{a}=(a_1,\ldots,a_k)\in\mathcal{P}_{\min}^k a = ( a 1 , … , a k ) ∈ P m i n k in turn, enumerating each of the k k k coordinates one profile at a time as in Appendix § 15 .2.3.
Inner loop. For each tuple, try every b ∈ P max b\in\mathcal{P}_{\max} b ∈ P m a x in turn, enumerated the same way.
Per-pair computation. For each fixed pair ( a ⃗ , b ) (\vec{a}, b) ( a , b ) , compute the k k k child values W ^ r − 1 s p ( Φ a 1 , b ( μ ) ) , … , W ^ r − 1 s p ( Φ a k , b ( μ ) ) \widehat{W}^{\mathrm{sp}}_{r-1}(\Phi_{a_1,b}(\mu)), \ldots, \widehat{W}^{\mathrm{sp}}_{r-1}(\Phi_{a_k,b}(\mu)) W r − 1 sp ( Φ a 1 , b ( μ )) , … , W r − 1 sp ( Φ a k , b ( μ )) one at a time, via a fresh recursive call each, adding each to a running sum as it is produced, so only one child value is ever held in memory at once, never all k k k simultaneously. The completed sum gives 1 k ∑ i W ^ r − 1 s p ( Φ a i , b ( μ ) ) \frac{1}{k}\sum_i \widehat{W}^{\mathrm{sp}}_{r-1}(\Phi_{a_i,b}(\mu)) k 1 ∑ i W r − 1 sp ( Φ a i , b ( μ )) for this ( a ⃗ , b ) (\vec{a}, b) ( a , b ) .
Max/min bookkeeping. As each column b b b finishes, compare its value against a running maximum kept for the current tuple, updating it if larger. Once every b b b has been tried, this maximum is the value of the inner max b \max_b max b for the current a ⃗ \vec{a} a ; compare it against a running minimum kept across tuples, updating it if smaller.
No caching. If the same subproblem ( r − 1 , Φ a i , b ( μ ) ) (r-1,\Phi_{a_i,b}(\mu)) ( r − 1 , Φ a i , b ( μ )) arises again, whether for a different i i i , a different b b b , or a different tuple entirely, simply recompute it from scratch by another recursive call rather than looking it up in a table. This is what keeps the space bound in Step 6 polynomial despite the number of distinct subproblems being potentially exponential: memory only ever holds the current path of nested calls, never a record of every subproblem visited.
Termination. P min k \mathcal{P}^{k}_{\min} P m i n k and P max \mathcal{P}_{\max} P m a x are both finite, every loop over them is bounded, and every recursive call is made at strictly smaller r r r , so the recursion bottoms out after at most T T T levels.
This computes a value depending only on ( r , μ ) (r,\mu) ( r , μ ) , in line with the Markov-sufficiency recorded in Step 1 (proved in Theorem 4). Its result is the same every time the same subproblem ( r , μ ) (r,\mu) ( r , μ ) is requested, which is exactly what makes recomputing it, rather than caching it, harmless for correctness.
Step 4: a uniform two-sided error invariant. Step 3 built a numerical recursion W ^ r s p \widehat{W}^{\mathrm{sp}}_r W r sp that a machine can actually execute, but executing it is not enough on its own: every round introduces two new sources of error, from replacing val \operatorname{val} val by s k s_k s k (up to α \alpha α ) and from rounding to precision d d d , and nothing so far bounds how these errors accumulate over T T T rounds of recursion. This step proves they do not compound uncontrollably: it establishes a single, uniform bound on ∣ W ^ r s p ( μ ) − W r s p ( μ ) ∣ |\widehat{W}^{\mathrm{sp}}_r(\mu)-W^{\mathrm{sp}}_r(\mu)| ∣ W r sp ( μ ) − W r sp ( μ ) ∣ that holds for every round and every distribution, which is exactly what Step 5 needs to know the final computed value is close enough to the true value to decide the promise correctly.
We prove for every rational μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) and r ≥ 0 r \ge0 r ≥ 0 that
∣ W ^ r s p ( μ ) − W r s p ( μ ) ∣ ≤ B r , B 0 = 0 , B r = γ B r − 1 + γ α + 2 d . (61) |\widehat{W}^{\mathrm{sp}}_r(\mu)-W^{\mathrm{sp}}_r(\mu)| \le B_r,\qquad B_0=0,\qquad B_r=\gamma B_{r-1}+\gamma\alpha+2d. \tag*{(61)} ∣ W r sp ( μ ) − W r sp ( μ ) ∣ ≤ B r , B 0 = 0 , B r = γ B r − 1 + γ α + 2 d . (61)
The base case is exact. For the inductive step let M a , b = W r − 1 s p ( Φ a , b ( μ ) ) M_{a,b}=W^{\mathrm{sp}}_{r-1}(\Phi_{a,b}(\mu)) M a , b = W r − 1 sp ( Φ a , b ( μ )) and M ^ a , b = W ^ r − 1 s p ( Φ a , b ( μ ) ) \widehat{M}_{a,b}=\widehat{W}^{\mathrm{sp}}_{r-1}(\Phi_{a,b}(\mu)) M a , b = W r − 1 sp ( Φ a , b ( μ )) . Successors are rational, so the induction hypothesis bounds each entrywise difference by B r − 1 B_{r-1} B r − 1 : max a , b ∣ M ^ a , b − M a , b ∣ = max a , b ∣ W ^ r − 1 s p ( Φ a , b ( μ ) ) − W r − 1 s p ( Φ a , b ( μ ) ) ∣ ≤ B r − 1 \max_{a,b}|\widehat{M}_{a,b}-M_{a,b}|=\max_{a,b}|\widehat{W}^{\mathrm{sp}}_{r-1}(\Phi_{a,b}(\mu))-W^{\mathrm{sp}}_{r-1}(\Phi_{a,b}(\mu))|\le B_{r-1} max a , b ∣ M a , b − M a , b ∣ = max a , b ∣ W r − 1 sp ( Φ a , b ( μ )) − W r − 1 sp ( Φ a , b ( μ )) ∣ ≤ B r − 1 . By (59) ,
∣ s k ( M ^ ) − s k ( M ) ∣ ≤ max a , b ∣ M ^ a , b − M a , b ∣ ≤ B r − 1 , i.e. s k ( M ) − B r − 1 ≤ s k ( M ^ ) ≤ s k ( M ) + B r − 1 . |s_k(\widehat{M})-s_k(M)|\le\max_{a,b}|\widehat{M}_{a,b}-M_{a,b}|\le B_{r-1},\qquad\text{i.e.}\qquad s_k(M)-B_{r-1}\le s_k(\widehat{M})\le s_k(M)+B_{r-1}. ∣ s k ( M ) − s k ( M ) ∣ ≤ a , b max ∣ M a , b − M a , b ∣ ≤ B r − 1 , i.e. s k ( M ) − B r − 1 ≤ s k ( M ) ≤ s k ( M ) + B r − 1 .
By (58) , val ( M ) ≤ s k ( M ) ≤ val ( M ) + α \operatorname{val}(M)\le s_k(M)\le\operatorname{val}(M)+\alpha val ( M ) ≤ s k ( M ) ≤ val ( M ) + α . Combining the two: for the upper bound, s k ( M ^ ) ≤ s k ( M ) + B r − 1 ≤ val ( M ) + α + B r − 1 s_k(\widehat{M})\le s_k(M)+B_{r-1}\le\operatorname{val}(M)+\alpha+B_{r-1} s k ( M ) ≤ s k ( M ) + B r − 1 ≤ val ( M ) + α + B r − 1 ; for the lower bound, s k ( M ^ ) ≥ s k ( M ) − B r − 1 ≥ val ( M ) − B r − 1 s_k(\widehat{M})\ge s_k(M)-B_{r-1}\ge\operatorname{val}(M)-B_{r-1} s k ( M ) ≥ s k ( M ) − B r − 1 ≥ val ( M ) − B r − 1 (using s k ( M ) ≥ val ( M ) s_k(M)\ge\operatorname{val}(M) s k ( M ) ≥ val ( M ) ). Together,
val ( M ) − B r − 1 ≤ s k ( M ^ ) ≤ val ( M ) + α + B r − 1 . \operatorname{val}(M)-B_{r-1}\le s_k(\widehat{M})\le\operatorname{val}(M)+\alpha+B_{r-1}. val ( M ) − B r − 1 ≤ s k ( M ) ≤ val ( M ) + α + B r − 1 .
This supplies both directions of the comparison: the algorithm built in Step 3 cannot underestimate a continuation by more than B r − 1 B_{r-1} B r − 1 , or overestimate it by more than B r − 1 + α B_{r-1}+\alpha B r − 1 + α . Write z = ( 1 − γ ) H ^ p ( μ ) + γ s k ( M ^ ) z=(1-\gamma)\widehat{H}_{p}(\mu)+\gamma s_k(\widehat{M}) z = ( 1 − γ ) H p ( μ ) + γ s k ( M ) for the argument passed to the rounding operation, and recall W r s p ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M ) W^{\mathrm{sp}}_r(\mu)=(1-\gamma)H(\mu)+\gamma\operatorname{val}(M) W r sp ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M ) . Subtracting,
z − W r s p ( μ ) = ( 1 − γ ) [ H ^ p ( μ ) − H ( μ ) ] + γ [ s k ( M ^ ) − val ( M ) ] . z-W^{\mathrm{sp}}_r(\mu)=(1-\gamma)\left[\widehat{H}_{p}(\mu)-H(\mu)\right]+\gamma\left[s_k(\widehat{M})-\operatorname{val}(M)\right]. z − W r sp ( μ ) = ( 1 − γ ) [ H p ( μ ) − H ( μ ) ] + γ [ s k ( M ) − val ( M ) ] .
The first bracket is at most d d d in absolute value, since H ^ p \widehat{H}_p H p approximates H \mathbf{H} H to error 2 − p = d 2^{-p}=d 2 − p = d (Lemma 6, Part 2, as used in Step 3); the second bracket lies in [ − B r − 1 , α + B r − 1 ] [-B_{r-1},\alpha+B_{r-1}] [ − B r − 1 , α + B r − 1 ] by the two-sided bound just proved. Since W r s p ( μ ) ∈ [ 0 , log 2 n ] ⊆ [ 0 , L ] W^{\mathrm{sp}}_r(\mu)\in[0,\log_2 n]\subseteq[0,L] W r sp ( μ ) ∈ [ 0 , log 2 n ] ⊆ [ 0 , L ] (Step 1), the clipping inequality applies with w = W r s p ( μ ) w=W^{\mathrm{sp}}_r(\mu) w = W r sp ( μ ) , giving
∣ W ^ r s p ( μ ) − W r s p ( μ ) ∣ = ∣ R p ( z ) − W r s p ( μ ) ∣ ≤ ∣ z − W r s p ( μ ) ∣ + d . \left|\widehat{W}^{\mathrm{sp}}_r(\mu)-W^{\mathrm{sp}}_r(\mu)\right|=\left|R_p(z)-W^{\mathrm{sp}}_r(\mu)\right|\leq\left|z-W^{\mathrm{sp}}_r(\mu)\right|+d. W r sp ( μ ) − W r sp ( μ ) = ∣ R p ( z ) − W r sp ( μ ) ∣ ≤ ∣ z − W r sp ( μ ) ∣ + d .
Bounding ∣ z − W r s p ( μ ) ∣ |z-W^{\mathrm{sp}}_r(\mu)| ∣ z − W r sp ( μ ) ∣ by the worse of the two directions, ( 1 − γ ) d + γ ( B r − 1 + α ) (1-\gamma)d+\gamma(B_{r-1}+\alpha) ( 1 − γ ) d + γ ( B r − 1 + α ) , and adding the extra d d d from the clipping inequality gives
∣ W ^ r s p ( μ ) − W r s p ( μ ) ∣ ≤ γ B r − 1 + γ α + ( 1 − γ ) d + d ≤ γ B r − 1 + γ α + 2 d = B r , \left|\widehat{W}^{\mathrm{sp}}_r(\mu)-W^{\mathrm{sp}}_r(\mu)\right|\leq\gamma B_{r-1}+\gamma\alpha+(1-\gamma)d+d\leq\gamma B_{r-1}+\gamma\alpha+2d=B_r, W r sp ( μ ) − W r sp ( μ ) ≤ γ B r − 1 + γ α + ( 1 − γ ) d + d ≤ γ B r − 1 + γ α + 2 d = B r ,
proving the induction.
The recurrence B 0 = 0 B_0=0 B 0 = 0 , B r = γ B r − 1 + γ α + 2 d B_r=\gamma B_{r-1}+\gamma\alpha+2d B r = γ B r − 1 + γ α + 2 d has the explicit solution
B r = ( γ α + 2 d ) ∑ j = 0 r − 1 γ j . B_r=(\gamma\alpha+2d)\sum_{j=0}^{r-1}\gamma^j. B r = ( γ α + 2 d ) j = 0 ∑ r − 1 γ j .
verified by induction: at r = 0 r=0 r = 0 the sum is empty, giving B 0 = 0 B_0=0 B 0 = 0 ; if it holds at r − 1 r-1 r − 1 , then B r = γ B r − 1 + γ α + 2 d = γ ( γ α + 2 d ) ∑ j = 0 r − 2 γ j + ( γ α + 2 d ) = ( γ α + 2 d ) [ ∑ j = 1 r − 1 γ j + 1 ] = ( γ α + 2 d ) ∑ j = 0 r − 1 γ j B_r=\gamma B_{r-1}+\gamma\alpha+2d=\gamma(\gamma\alpha+2d)\sum_{j=0}^{r-2}\gamma^j+(\gamma\alpha+2d)=(\gamma\alpha+2d)\left[\sum_{j=1}^{r-1}\gamma^j+1\right]=(\gamma\alpha+2d)\sum_{j=0}^{r-1}\gamma^j B r = γ B r − 1 + γ α + 2 d = γ ( γ α + 2 d ) ∑ j = 0 r − 2 γ j + ( γ α + 2 d ) = ( γ α + 2 d ) [ ∑ j = 1 r − 1 γ j + 1 ] = ( γ α + 2 d ) ∑ j = 0 r − 1 γ j . Since all terms of the infinite geometric series are positive, truncating it at r − 1 r-1 r − 1 terms only decreases it, so
B r ≤ ( γ α + 2 d ) ∑ j = 0 ∞ γ j = γ α + 2 d 1 − γ . B_r\leq(\gamma\alpha+2d)\sum_{j=0}^{\infty}\gamma^j=\frac{\gamma\alpha+2d}{1-\gamma}. B r ≤ ( γ α + 2 d ) j = 0 ∑ ∞ γ j = 1 − γ γ α + 2 d .
Recall α = ( 1 − γ ) e / 8 \alpha=(1-\gamma)e/8 α = ( 1 − γ ) e /8 from Step 2 and d ≤ ( 1 − γ ) e / 16 d\leq(1-\gamma)e/16 d ≤ ( 1 − γ ) e /16 from the choice of p p p in Step 3. Substituting,
γ α + 2 d 1 − γ ≤ ( 1 − γ ) e 8 + 2 ( 1 − γ ) e 16 1 − γ = ( 1 − γ ) e 8 ( γ + 1 ) 1 − γ = ( 1 + γ ) e 8 . \frac{\gamma\alpha+2d}{1-\gamma}\leq\frac{\frac{(1-\gamma)e}{8}+2\frac{(1-\gamma)e}{16}}{1-\gamma}=\frac{\frac{(1-\gamma)e}{8}(\gamma+1)}{1-\gamma}=\frac{(1+\gamma)e}{8}. 1 − γ γ α + 2 d ≤ 1 − γ 8 ( 1 − γ ) e + 2 16 ( 1 − γ ) e = 1 − γ 8 ( 1 − γ ) e ( γ + 1 ) = 8 ( 1 + γ ) e .
Since γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) , 1 + γ < 2 1+\gamma<2 1 + γ < 2 , so ( 1 + γ ) e 8 < 2 e 8 = e / 4 \frac{(1+\gamma)e}{8}<\frac{2e}{8}=e/4 8 ( 1 + γ ) e < 8 2 e = e /4 . Hence B r ≤ e / 4 B_r\leq e/4 B r ≤ e /4 for every r r r , in particular r = T r=T r = T .
The bound is uniform in the initial distribution and the enumerated profiles: nowhere in its derivation did μ \mu μ , ν \nu ν , or the specific tuples chosen enter, only r , α , d , γ r,\alpha,d,\gamma r , α , d , γ . It also does not grow with k k k .
Step 5: Correctness of the decision procedure. Steps 1 and 4 each bounded one source of error: Step 1 bounded the cost of truncating to T T T rounds, and Step 4 bounded the cost of the sparsification and rounding used to actually compute a numerical answer. Neither step, on its own, says anything about whether comparing that answer to v v v decides the problem correctly. This step combines the two error bounds into a single total, and uses the promise gap between yes-instances and no-instances to show that the resulting comparison is always correct on promised inputs.
Compute z = W ^ T s p ( μ 0 ) z=\widehat{W}^{\mathrm{sp}}_T(\mu_0) z = W T sp ( μ 0 ) and answer Yes exactly when z ≤ v z\leq v z ≤ v . By Step 4 and (55) ,
∣ z − V s p ( μ 0 ) ∣ ≤ ∣ z − W T s p ( μ 0 ) ∣ + ∣ W T s p ( μ 0 ) − V s p ( μ 0 ) ∣ ≤ e / 4 + e / 4 = e / 2. |z-V_{\mathrm{sp}}(\mu_0)|\leq|z-W^{\mathrm{sp}}_T(\mu_0)|+|W^{\mathrm{sp}}_T(\mu_0)-V_{\mathrm{sp}}(\mu_0)|\leq e/4+e/4=e/2. ∣ z − V sp ( μ 0 ) ∣ ≤ ∣ z − W T sp ( μ 0 ) ∣ + ∣ W T sp ( μ 0 ) − V sp ( μ 0 ) ∣ ≤ e /4 + e /4 = e /2.
Since e = min { ε 0 , 1 } ≤ ε 0 e=\min\{\varepsilon_0,1\}\le\varepsilon_0 e = min { ε 0 , 1 } ≤ ε 0 and ε 0 > 0 \varepsilon_0>0 ε 0 > 0 , e / 2 ≤ ε 0 / 2 < ε 0 e/2\le\varepsilon_0/2<\varepsilon_0 e /2 ≤ ε 0 /2 < ε 0 , so ∣ z − V s p ( μ 0 ) ∣ < ε 0 \lvert z-V_{\mathrm{sp}}(\mu_0)\rvert<\varepsilon_0 ∣ z − V sp ( μ 0 )∣ < ε 0 . If V s p ( μ 0 ) ≤ v − ε 0 V_{\mathrm{sp}}(\mu_0)\le v-\varepsilon_0 V sp ( μ 0 ) ≤ v − ε 0 , then z ≤ V s p ( μ 0 ) + e / 2 ≤ v − ε 0 + e / 2 < v z\le V_{\mathrm{sp}}(\mu_0)+e/2\le v-\varepsilon_0+e/2<v z ≤ V sp ( μ 0 ) + e /2 ≤ v − ε 0 + e /2 < v . If V s p ( μ 0 ) ≥ v + ε 0 V_{\mathrm{sp}}(\mu_0)\ge v+\varepsilon_0 V sp ( μ 0 ) ≥ v + ε 0 , then z ≥ V s p ( μ 0 ) − e / 2 ≥ v + ε 0 − e / 2 > v z\ge V_{\mathrm{sp}}(\mu_0)-e/2\ge v+\varepsilon_0-e/2>v z ≥ V sp ( μ 0 ) − e /2 ≥ v + ε 0 − e /2 > v . So the algorithm answers Yes on every yes-instance and No on every no-instance of the promise. The comparison z ≤ v z\le v z ≤ v is between rationals and does not require deciding equality of a transcendental entropy expression.
Step 6: exact bit lengths and active workspace. Step 5 shows that comparing z = W ^ T s p ( μ 0 ) z=\widehat{W}^{\mathrm{sp}}_T(\mu_0) z = W T sp ( μ 0 ) to v v v decides the promise problem correctly, but correctness alone does not place the problem in PSPACE: it says nothing about the resources used to compute z z z . The algorithm of Step 3 recomputes children when needed and stores only a single active path, rather than caching every subproblem, which avoids holding an exponential number of values at once; but this design choice does not by itself bound anything. Each active frame still holds rational numbers whose bit length was never pinned down, and the active path could in principle hold as many frames as there are levels of recursion, a quantity not yet shown to be small. This step closes both gaps: it bounds the exact bit length of every rational number the algorithm ever writes down, and it bounds the number of frames simultaneously present on the active path, showing that their product is polynomial in ℓ \ell ℓ and thereby completing the proof of Theorem 9.
Bit length of the distribution. Let D D D be the product of the denominators of all input transition probabilities, and D 0 D_0 D 0 the product of the denominators of the initial distribution’s coordinates. Each individual denominator is a substring of the ℓ \ell ℓ -bit input encoding, so the sum of their bit lengths is at most ℓ \ell ℓ . For positive integers a , b a,b a , b with bit lengths p , q p,q p , q , a b < 2 p ⋅ 2 q = 2 p + q ab<2^p\cdot2^q=2^{p+q} ab < 2 p ⋅ 2 q = 2 p + q , so the bit length of a product of positive integers is at most the sum of the bit lengths of the factors; hence both D D D and D 0 D_0 D 0 have O ( ℓ ) O(\ell) O ( ℓ ) bits. Fix a pure profile pair ( a , b ) (a,b) ( a , b ) . Each entry Δ ( s , ρ a , b ( s ) , s ′ ) \Delta(s,\rho_{a,b}(s),s') Δ ( s , ρ a , b ( s ) , s ′ ) of the corresponding transition matrix is one of the input’s own rational transition probabilities, so its denominator divides D D D by definition of D D D ; since the matrix has only finitely many entries, all with denominator dividing D D D , the whole matrix has common denominator dividing D D D . This allows an induction on t t t : the base case μ = μ 0 \mu=\mu_0 μ = μ 0 has common denominator dividing D 0 D_0 D 0 by definition of D 0 D_0 D 0 . For the inductive step, suppose every coordinate of μ \mu μ has the form m s / ( D 0 D t ) m_s/(D_0D^t) m s / ( D 0 D t ) for an integer m s m_s m s , and write Δ ( s , ρ a , b ( s ) , s ′ ) = q s , s ′ / D \Delta(s,\rho_{a,b}(s),s')=q_{s,s'}/D Δ ( s , ρ a , b ( s ) , s ′ ) = q s , s ′ / D for an integer q s , s ′ q_{s,s'} q s , s ′ . By (9) ,
Φ a , b ( μ ) ( s ′ ) = ∑ s ∈ S m s D 0 D t ⋅ q s , s ′ D = ∑ s m s q s , s ′ D 0 D t + 1 , \Phi_{a,b}(\mu)(s')=\sum_{s\in S}\frac{m_s}{D_0D^t}\cdot\frac{q_{s,s'}}{D}=\frac{\sum_s m_sq_{s,s'}}{D_0D^{t+1}}, Φ a , b ( μ ) ( s ′ ) = s ∈ S ∑ D 0 D t m s ⋅ D q s , s ′ = D 0 D t + 1 ∑ s m s q s , s ′ ,
an integer numerator over D 0 D t + 1 D_0D^{t+1} D 0 D t + 1 . So every coordinate of Φ a , b ( μ ) \Phi_{a,b}(\mu) Φ a , b ( μ ) again has denominator dividing D 0 D t + 1 D_0D^{t+1} D 0 D t + 1 , and after t t t updates a distribution has common denominator dividing D 0 D t D_0D^t D 0 D t . Its n n n coordinates therefore occupy O ( n ( t + 1 ) ℓ ) O(n(t+1)\ell) O ( n ( t + 1 ) ℓ ) bits. This is a specific denominator invariant; a polynomial number of arbitrary rational operations alone would not suffice to prove such a bound.
Bit length of the recursion’s values. Bounding the distribution’s denominator does not account for every rational number the algorithm manipulates: it also computes and combines finite-precision values along the recursion (60) . As fixed in Step 3, every recursive call returns a multiple of d = 2 − p d=2^{-p} d = 2 − p lying in [ 0 , L ] [0,L] [ 0 , L ] ; write the i i i -th child’s return value as j i / 2 p j_i/2^p j i / 2 p for an integer j i j_i j i with 0 ≤ j i ≤ L 2 p 0\le j_i\le L2^p 0 ≤ j i ≤ L 2 p . For a fixed tuple and response b b b , store the integer ∑ i j i \sum_i j_i ∑ i j i , between zero and k L 2 p kL2^p k L 2 p , then divide by the common denominator k 2 p k2^p k 2 p . This takes O ( p + log k + log ( L + 1 ) ) O(p+\log k+\log(L+1)) O ( p + log k + log ( L + 1 )) bits. Minima and maxima merely select one such rational, without introducing new denominators. The resulting average, with denominator dividing k 2 p k2^p k 2 p , is then combined with ( 1 − γ ) H ^ p ( μ ) (1-\gamma)\widehat{H}_p(\mu) ( 1 − γ ) H p ( μ ) , which has denominator 2 p 2^p 2 p since H ^ p ( μ ) \widehat{H}_p(\mu) H p ( μ ) does by construction, and scaled by the fixed rational discount γ \gamma γ ; this multiplies denominators together at most once, introducing only γ \gamma γ ’s own fixed denominator on top of k 2 p k2^p k 2 p , so the pre-rounding value still has polynomially many bits. Applying R p R_p R p then resets the return denominator to exactly 2 p 2^p 2 p . Unlike the distribution’s denominator D 0 D t D_0D^t D 0 D t from above, which grows with depth, the value denominator therefore does not compound across recursive calls.
The active workspace. An active frame stores the current distribution, remaining horizon, the k k k -tuple a ⃗ = ( a 1 , … , a k ) ∈ P min k \vec{a}=(a_1,\ldots,a_k)\in\mathcal{P}_{\min}^k a = ( a 1 , … , a k ) ∈ P m i n k currently being enumerated (O ( k n log ( 2 A ) ) O(kn\log(2A)) O ( k n log ( 2 A )) bits, one O ( n log ( 2 A ) ) O(n\log(2A)) O ( n log ( 2 A )) -bit pure profile per coordinate), the current column b ∈ P max b\in\mathcal{P}_{\max} b ∈ P m a x (O ( n log ( 2 A ) ) O(n\log(2A)) O ( n log ( 2 A )) bits), child index, running sum, and running extrema. Rational updates, namely computing Φ a ⃗ , b ( μ ) \Phi_{\vec{a},b}(\mu) Φ a , b ( μ ) , accumulating the running sum, and combining the entropy and discount terms above, are additions, multiplications, and comparisons of integers no longer than the bit lengths already bounded; elementary arithmetic on m m m -bit integers uses O ( m ) O(m) O ( m ) auxiliary space, so these updates use space polynomial in the lengths just established. Lemma 6 bounds only the time to compute H ^ p ( μ ) \widehat{H}_p(\mu) H p ( μ ) , which is polynomial in size ( μ ) + p \operatorname{size}(\mu)+p size ( μ ) + p ; since a deterministic algorithm can write to at most one new memory cell per step, a computation running in time t t t uses at most O ( t ) O(t) O ( t ) space, so computing H ^ p \widehat{H}_p H p also uses only polynomial auxiliary space.
A quantity polynomial in lengths that are themselves polynomial in ℓ \ell ℓ is again polynomial in ℓ \ell ℓ . Summing the bounds above, a single frame occupies O ( n ( T + 1 ) ℓ + k n log ( 2 A ) + p + log k + log ( L + 1 ) ) O(n(T+1)\ell+kn\log(2A)+p+\log k+\log(L+1)) O ( n ( T + 1 ) ℓ + k n log ( 2 A ) + p + log k + log ( L + 1 )) bits: the distribution contributes at most O ( n ( T + 1 ) ℓ ) O(n(T+1)\ell) O ( n ( T + 1 ) ℓ ) bits since its depth is at most T T T , the tuple and column contribute O ( k n log ( 2 A ) ) O(kn\log(2A)) O ( k n log ( 2 A )) and O ( n log ( 2 A ) ) O(n\log(2A)) O ( n log ( 2 A )) bits, and the running sum, extrema, and combined value contribute O ( p + log k + log ( L + 1 ) ) O(p+\log k+\log(L+1)) O ( p + log k + log ( L + 1 )) bits.
Having bounded the space used by a single frame, it remains to bound how many frames coexist. On a recursive call the parent frame is retained, but the completed child’s frame is discarded before the next child is processed. Since the recursion depth strictly decreases from r = T r=T r = T at the root to r = 0 r=0 r = 0 at a base case (Step 3), and only the frames on the current path are retained, at most T T T nontrivial frames coexist at any time. The horizon T T T was shown polynomial in ℓ \ell ℓ in Step 1, the sample size k k k in Step 2, and the precision p p p is fixed in Step 3 independently of the instance. Total space is therefore at most T T T times a single frame’s polynomial bit length, a product of two quantities polynomial in ℓ \ell ℓ , hence itself polynomial in ℓ \ell ℓ . The number of tuples and recomputed subproblems may be exponential; this affects time, not the workspace bound.
Sampled-transition semantics
The value is unambiguous by Theorem 4, so the polarity may be omitted from GapVal s t , ε 0 \operatorname{GapVal}_{\mathrm{st},\varepsilon_0} GapVal st , ε 0 and V s t V_{\mathrm{st}} V st . Unlike sampled-profile semantics, a fixed pure-profile pair no longer determines a single successor distribution, since the arena’s own transitions may realise any of up to n n n^n n n ways. We enumerate every realisation, but sum their contributions into a single accumulator sequentially, rather than storing their recursive subtrees, so the number of realisations affects time, not space.
Theorem 10 (Polynomial-space upper bound under sampled-transition semantics). For fixed rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and ε 0 > 0 \varepsilon_0>0 ε 0 > 0 , GapVal s t , ε 0 ∈ P S P A C E \operatorname{GapVal}_{\mathrm{st},\varepsilon_0}\in\mathrm{PSPACE} GapVal st , ε 0 ∈ PSPACE .
Proof. Let n = ∣ S ∣ n=|S| n = ∣ S ∣ , A = max s ∈ S ∣ A ( s ) ∣ A=\max_{s\in S}|A(s)| A = max s ∈ S ∣ A ( s ) ∣ , ϵ = min { ε 0 , 1 } \epsilon=\min\{\varepsilon_0,1\} ϵ = min { ε 0 , 1 } , and, for n ≥ 2 n\ge2 n ≥ 2 , L = ⌈ log 2 n ⌉ L=\lceil\log_2 n\rceil L = ⌈ log 2 n ⌉ ; let ℓ \ell ℓ be the total bit length of the binary encoding of the arena, its rational transition probabilities Δ ( s , a , s ′ ) \Delta(s,a,s') Δ ( s , a , s ′ ) , the rational initial distribution μ 0 \mu_0 μ 0 , and the rational threshold v v v . For n = 1 n=1 n = 1 compare the identically zero value with v v v . Assume n ≥ 2 n\ge2 n ≥ 2 below.
Recall from § 2 the sampled-transition objects: Ξ = S S \Xi=S^S Ξ = S S , so a realisation ξ ∈ Ξ \xi\in\Xi ξ ∈ Ξ assigns every state a successor state and ∣ Ξ ∣ ≤ n n |\Xi|\le n^n ∣Ξ∣ ≤ n n ; the transition-realisation law P a , b ( ξ ) = P_{a,b}(\xi)= P a , b ( ξ ) =
::: ∏ s ∈ S Δ ( s , ρ a , b ( s ) , ξ ( s ) ) \prod_{s\in S}\Delta(s,\rho_{a,b}(s),\xi(s)) ∏ s ∈ S Δ ( s , ρ a , b ( s ) , ξ ( s )) of (11) , where ρ a , b \rho_{a,b} ρ a , b is as in (9) ; and the realised-successor map Ψ ξ ( μ ) ( s ′ ) = ∑ s : ξ ( s ) = s ′ μ ( s ) \Psi_{\xi}(\mu)(s')=\sum_{s:\xi(s)=s'}\mu(s) Ψ ξ ( μ ) ( s ′ ) = ∑ s : ξ ( s ) = s ′ μ ( s ) of (12) .
By Theorem 4, the value V s t : D ( S ) → [ 0 , log 2 n ] V_{\mathrm{st}}:\mathcal{D}(S)\to[0,\log_{2}n] V st : D ( S ) → [ 0 , log 2 n ] is the unique fixed point of the γ \gamma γ -contraction T s t \mathcal{T}_{\mathrm{st}} T st , defined by
( T s t V ) ( μ ) = ( 1 − γ ) H ( μ ) + γ ⋅ val ( M V ( μ ) ) , M V ( μ ) a , b : = ∑ ξ ∈ Ξ P a , b ( ξ ) V ( Ψ ξ ( μ ) ) . (62) (\mathcal{T}_{\mathrm{st}}V)(\mu)=(1-\gamma)\mathbf{H}(\mu)+\gamma\cdot\operatorname{val}(M_{V}(\mu)),\qquad M_{V}(\mu)_{a,b}:=\sum_{\xi\in\Xi}P_{a,b}(\xi)V(\Psi_{\xi}(\mu)). \tag*{(62)} ( T st V ) ( μ ) = ( 1 − γ ) H ( μ ) + γ ⋅ val ( M V ( μ )) , M V ( μ ) a , b := ξ ∈ Ξ ∑ P a , b ( ξ ) V ( Ψ ξ ( μ )) . (62)
where val \operatorname{val} val is as in (52) . Unlike sampled-profile semantics, once a , b a,b a , b are fixed there is no further strategic choice inside M V ( μ ) a , b M_{V}(\mu)_{a,b} M V ( μ ) a , b : it is a fixed expectation over up to n n n^{n} n n realisations. Lemma 8 reduces the number of profiles in the outer mixture but does not change this expectation, so we keep the entire realisation sum and evaluate its terms sequentially rather than approximate it. Two facts make this possible in polynomial space: realised successors only sum existing coordinates of μ \mu μ , no new mass is created, and every approximate child value shares the same dyadic denominator, so the realisation sum never multiplies together the denominators of different realisations.
Step 1: the exact finite-horizon recursion. Choose the same positive T T T as in Appendix § 15 .2.4, the least with γ T L ≤ e / 4 \gamma^{T}L\le e/4 γ T L ≤ e /4 . For r r r remaining rewards, Theorem 4 establishes
W 0 s t = 0 , W^{\mathrm{st}}_{0}=0, W 0 st = 0 ,
W r s t ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( ( ∑ ξ ∈ Ξ P a , b ( ξ ) W r − 1 s t ( Ψ ξ ( μ ) ) ) a , b ) . (63) W^{\mathrm{st}}_{r}(\mu)=(1-\gamma)\mathbf{H}(\mu)+\gamma\operatorname{val}\left(\left(\sum_{\xi\in\Xi}P_{a,b}(\xi)W^{\mathrm{st}}_{r-1}(\Psi_{\xi}(\mu))\right)_{a,b}\right). \tag*{(63)} W r st ( μ ) = ( 1 − γ ) H ( μ ) + γ val ξ ∈ Ξ ∑ P a , b ( ξ ) W r − 1 st ( Ψ ξ ( μ )) a , b . (63)
In particular, W 1 s t = ( 1 − γ ) H W^{\mathrm{st}}_{1}=(1-\gamma)\mathbf{H} W 1 st = ( 1 − γ ) H contains no outgoing transition. Theorem 4 gives Borel Markov saddle policies for these values against all distribution-history opponents, so (63) is not a game in which earlier profiles or realisations become observable: its value depends only on the current distribution and remaining horizon, and evaluating the same subproblem twice does not license different information for either player.
The finite-horizon identification W r s t = ( T s t ) r 0 W^{\mathrm{st}}_{r}=(\mathcal{T}_{\mathrm{st}})^{r}0 W r st = ( T st ) r 0 is also proved in the proof of Theorem 1 (Appendix § 13 , paragraph “Finite-horizon identification”). Every entry ∑ ξ P a , b ( ξ ) W r − 1 s t ( Ψ ξ ( μ ) ) \sum_{\xi}P_{a,b}(\xi)W^{\mathrm{st}}_{r-1}(\Psi_{\xi}(\mu)) ∑ ξ P a , b ( ξ ) W r − 1 st ( Ψ ξ ( μ )) of the continuation matrix is a convex combination of values in [ 0 , log 2 n ] [0,\log_{2}n] [ 0 , log 2 n ] (the P a , b ( ξ ) P_{a,b}(\xi) P a , b ( ξ ) are nonnegative and sum to 1), so the entries lie in [ 0 , log 2 n ] ⊆ [ 0 , L ] [0,\log_{2}n]\subseteq[0,L] [ 0 , log 2 n ] ⊆ [ 0 , L ] by induction on r r r and the range fact val ( M ) ∈ [ 0 , U ] \operatorname{val}(M)\in[0,U] val ( M ) ∈ [ 0 , U ] for M ∈ [ 0 , U ] R × C M\in[0,U]^{R\times C} M ∈ [ 0 , U ] R × C of Appendix § 15 .2.4 (Step 1). As in the sampled-profile case, this is the boundedness hypothesis of Lemma 8. The truncation bound
0 ≤ V s t ( μ ) − W T s t ( μ ) ≤ γ T L ≤ e / 4 for every μ ∈ D ( S ) (64) 0\le V_{\mathrm{st}}(\mu)-W^{\mathrm{st}}_{T}(\mu)\le\gamma^{T}L\le e/4\qquad\text{for every }\mu\in\mathcal{D}(S) \tag*{(64)} 0 ≤ V st ( μ ) − W T st ( μ ) ≤ γ T L ≤ e /4 for every μ ∈ D ( S ) (64)
is the error bound 0 ≤ V s t − W r s t ≤ γ r log 2 ∣ S ∣ ≤ γ r L 0\le V_{\mathrm{st}}-W^{\mathrm{st}}_{r}\le\gamma^{r}\log_{2}|S|\le\gamma^{r}L 0 ≤ V st − W r st ≤ γ r log 2 ∣ S ∣ ≤ γ r L of that same proof at r = T r=T r = T (the argument is the one given for s p \mathrm{sp} sp in Appendix § 15 .2.4, Step 1, which uses only monotonicity and stability of val \operatorname{val} val and the shape of the recursion).
Step 2: sparse profiles and exact weighted child sums. Take α , k \alpha,k α , k from Appendix § 15 .2.4’s Step 2 and p , d , R p p,d,R_{p} p , d , R p from its Step 3. Set W ^ 0 s t = 0 \widehat{W}^{\mathrm{st}}_{0}=0 W 0 st = 0 . For rational μ \mu μ , first define the numerically evaluated entry
C ^ r − 1 ( μ ) a , b = ∑ ξ ∈ Ξ P a , b ( ξ ) W ^ r − 1 s t ( Ψ ξ ( μ ) ) , (65) \widehat{C}_{r-1}(\mu)_{a,b}=\sum_{\xi\in\Xi}P_{a,b}(\xi)\widehat{W}^{\mathrm{st}}_{r-1}(\Psi_{\xi}(\mu)), \tag*{(65)} C r − 1 ( μ ) a , b = ξ ∈ Ξ ∑ P a , b ( ξ ) W r − 1 st ( Ψ ξ ( μ )) , (65)
and then the returned value
W ^ r s t ( μ ) = R p ( ( 1 − γ ) H ^ p ( μ ) + γ min a ∈ P min k max b ∈ P max 1 k ∑ i C ^ r − 1 ( μ ) a i , b ) . (66) \widehat{W}^{\mathrm{st}}_{r}(\mu)=R_{p}\left((1-\gamma)\widehat{H}_{p}(\mu)+\gamma\min_{a\in\mathcal{P}^{k}_{\min}}\max_{b\in\mathcal{P}_{\max}}\frac{1}{k}\sum_{i}\widehat{C}_{r-1}(\mu)_{a_{i},b}\right). \tag*{(66)} W r st ( μ ) = R p ( ( 1 − γ ) H p ( μ ) + γ a ∈ P m i n k min b ∈ P m a x max k 1 i ∑ C r − 1 ( μ ) a i , b ) . (66)
For r = 1 r = 1 r = 1 the continuation is zero and all action and realisation loops are omitted. Otherwise enumerate tuples and responses as in Appendix § 15 .2.4’s Step 3. For each a i , b a_i,b a i , b , enumerate all maps ξ : S → S \xi:S \to S ξ : S → S , compute the exact product P a i , b ( ξ ) P_{a_i,b}(\xi) P a i , b ( ξ ) , recursively evaluate the one child Ψ ξ ( μ ) \Psi_\xi(\mu) Ψ ξ ( μ ) , add its weighted value to a running sum, and discard that child’s workspace once its contribution is added. Zero-weight realisations may be skipped. Sum the entries for all i i i , then update the maximum for the current tuple and the outer minimum.
The order min a ⃗ max b ∑ i ∑ ξ \min_{\vec a}\max_b\sum_i\sum_\xi min a max b ∑ i ∑ ξ is retained throughout: there is no maximisation over realisations and no existential choice of representative samples, since conditional on the profiles, their probabilities are exactly the arena’s prescribed law. Each call depends only on its rational distribution and remaining horizon, not on the private profile or realisation history leading to it, matching Step 1’s Markov-sufficiency point.
Step 3: a uniform two-sided error invariant. We prove for every rational μ \mu μ and r ≥ 0 r \ge0 r ≥ 0 that
∣ W ^ r s t ( μ ) − W r s t ( μ ) ∣ ≤ B r , B 0 = 0 , B r = γ B r − 1 + γ α + 2 d , \left|\widehat{W}^{\mathrm{st}}_r(\mu)-W^{\mathrm{st}}_r(\mu)\right| \le B_r,\qquad B_0=0,\qquad B_r=\gamma B_{r-1}+\gamma\alpha+2d, W r st ( μ ) − W r st ( μ ) ≤ B r , B 0 = 0 , B r = γ B r − 1 + γ α + 2 d ,
the same recurrence, and hence the same closed form B r ≤ ( γ α + 2 d ) / ( 1 − γ ) ≤ e / 4 B_r \le(\gamma\alpha+2d)/(1-\gamma) \le e/4 B r ≤ ( γ α + 2 d ) / ( 1 − γ ) ≤ e /4 , as Appendix § 15 .2.4’s Step 4. The base case is exact. Assume the bound for r − 1 r-1 r − 1 , and let C r − 1 ( μ ) a , b = ∑ ξ P a , b ( ξ ) W r − 1 s t ( Ψ ξ ( μ ) ) C_{r-1}(\mu)_{a,b}=\sum_\xi P_{a,b}(\xi)W^{\mathrm{st}}_{r-1}(\Psi_\xi(\mu)) C r − 1 ( μ ) a , b = ∑ ξ P a , b ( ξ ) W r − 1 st ( Ψ ξ ( μ )) denote the exact (unrounded) counterpart of (65) . For every fixed pair a , b a,b a , b ,
∣ C ^ r − 1 ( μ ) a , b − C r − 1 ( μ ) a , b ∣ ≤ ∑ ξ P a , b ( ξ ) ∣ W ^ r − 1 s t ( Ψ ξ ( μ ) ) − W r − 1 s t ( Ψ ξ ( μ ) ) ∣ ≤ ∑ ξ P a , b ( ξ ) B r − 1 = B r − 1 , \begin{aligned}
\left|\widehat{C}_{r-1}(\mu)_{a,b}-C_{r-1}(\mu)_{a,b}\right|
\le\sum_\xi P_{a,b}(\xi)\left|\widehat{W}^{\mathrm{st}}_{r-1}(\Psi_\xi(\mu))-W^{\mathrm{st}}_{r-1}(\Psi_\xi(\mu))\right| \\
&\le\sum_\xi P_{a,b}(\xi)B_{r-1}=B_{r-1},
\end{aligned} C r − 1 ( μ ) a , b − C r − 1 ( μ ) a , b ≤ ξ ∑ P a , b ( ξ ) W r − 1 st ( Ψ ξ ( μ )) − W r − 1 st ( Ψ ξ ( μ )) ≤ ξ ∑ P a , b ( ξ ) B r − 1 = B r − 1 ,
using the induction hypothesis on each term and ∑ ξ P a , b ( ξ ) = 1 \sum_\xi P_{a,b}(\xi)=1 ∑ ξ P a , b ( ξ ) = 1 . This is the step that prevents the error from being multiplied by n n n^n n n : the coefficients P a , b ( ξ ) P_{a,b}(\xi) P a , b ( ξ ) form a probability distribution, so averaging over realisations cannot amplify a uniform per-realisation error. The sum in (65) is exact given the dyadic child values W ^ r − 1 s t ( Ψ ξ ( μ ) ) \widehat{W}^{\mathrm{st}}_{r-1}(\Psi_\xi(\mu)) W r − 1 st ( Ψ ξ ( μ )) , so no further rounding error is introduced by the realisation sum itself.
The true matrix [ C r − 1 ( μ ) a , b ] a , b [C_{r-1}(\mu)_{a,b}]_{a,b} [ C r − 1 ( μ ) a , b ] a , b has entries in [ 0 , L ] [0,L] [ 0 , L ] by Step 1’s boundedness argument, applied one level down. Applying (58) and (59) to this matrix and its approximation [ C ^ r − 1 ( μ ) a , b ] a , b [\widehat{C}_{r-1}(\mu)_{a,b}]_{a,b} [ C r − 1 ( μ ) a , b ] a , b gives
val ( C r − 1 ) − B r − 1 ≤ s k ( C ^ r − 1 ) ≤ val ( C r − 1 ) + α + B r − 1 . \operatorname{val}(C_{r-1})-B_{r-1}\le s_k(\widehat{C}_{r-1})\le\operatorname{val}(C_{r-1})+\alpha+B_{r-1}. val ( C r − 1 ) − B r − 1 ≤ s k ( C r − 1 ) ≤ val ( C r − 1 ) + α + B r − 1 .
Adding the current entropy introduces at most ( 1 − γ ) d (1-\gamma)d ( 1 − γ ) d error, and the final rounding and clipping of the complete backup costs at most d d d against a target in [ 0 , L ] [0,L] [ 0 , L ] , by the clipping inequality (Appendix § 15 .2.4, Step 3). This mirrors Appendix § 15 .2.4’s Step 4 exactly: the one-step backup error is decomposed into an entropy term, a matrix-value term, and a final rounding/clipping term, then combined via the triangle inequality,
∣ W ^ r s t ( μ ) − W r s t ( μ ) ∣ ≤ d ⏟ final clipping + ( 1 − γ ) d + γ ∣ s k ( C ^ r − 1 ( μ ) ) − val ( C r − 1 ( μ ) ) ∣ ⏟ matrix-value term ≤ d + ( 1 − γ ) d + γ ( α + B r − 1 ) = γ B r − 1 + γ α + ( 2 − γ ) d ≤ γ B r − 1 + γ α + 2 d = B r , \begin{aligned}
\left|\widehat{W}^{\mathrm{st}}_r(\mu)-W^{\mathrm{st}}_r(\mu)\right|
\le\underbrace{d}_{\text{final clipping}}+(1-\gamma)d+\underbrace{\gamma\left|s_k(\widehat{C}_{r-1}(\mu))-\operatorname{val}(C_{r-1}(\mu))\right|}_{\text{matrix-value term}} \\
&\le d+(1-\gamma)d+\gamma(\alpha+B_{r-1}) \\
&=\gamma B_{r-1}+\gamma\alpha+(2-\gamma)d \\
&\le\gamma B_{r-1}+\gamma\alpha+2d=B_r,
\end{aligned} W r st ( μ ) − W r st ( μ ) ≤ final clipping d + ( 1 − γ ) d + matrix-value term γ s k ( C r − 1 ( μ )) − val ( C r − 1 ( μ )) ≤ d + ( 1 − γ ) d + γ ( α + B r − 1 ) = γ B r − 1 + γ α + ( 2 − γ ) d ≤ γ B r − 1 + γ α + 2 d = B r ,
using ( 2 − γ ) d ≤ 2 d (2-\gamma)d\le2d ( 2 − γ ) d ≤ 2 d since γ > 0 \gamma>0 γ > 0 ; the only difference from Appendix § 15 .2.4’s Step 4 is that the matrix here is C r − 1 ( μ ) C_{r-1}(\mu) C r − 1 ( μ ) , already averaged over realizations by Steps 1–2, in place of sp’s un-averaged M M M . This proves the induction.
Step 4: correctness of the decision procedure. Compute z = W ^ T s t ( μ 0 ) z = \widehat{W}^{\mathrm{st}}_{T}(\mu_0) z = W T st ( μ 0 ) and answer Yes exactly when z ≤ v z \leq v z ≤ v . By Step 3 and (64) ,
∣ z − V s t ( μ 0 ) ∣ ≤ ∣ z − W ^ T s t ( μ 0 ) ∣ + ∣ W ^ T s t ( μ 0 ) − V s t ( μ 0 ) ∣ ≤ ϵ / 4 + ϵ / 4 = ϵ / 2 < ϵ 0 . \lvert z - V_{\mathrm{st}}(\mu_0)\rvert\leq\lvert z - \widehat{W}^{\mathrm{st}}_{T}(\mu_0)\rvert+ \lvert\widehat{W}^{\mathrm{st}}_{T}(\mu_0) - V_{\mathrm{st}}(\mu_0)\rvert\leq\epsilon/4 + \epsilon/4 = \epsilon/2 < \epsilon_0. ∣ z − V st ( μ 0 )∣ ≤ ∣ z − W T st ( μ 0 )∣ + ∣ W T st ( μ 0 ) − V st ( μ 0 )∣ ≤ ϵ /4 + ϵ /4 = ϵ /2 < ϵ 0 .
If V s t ( μ 0 ) ≤ v − ϵ 0 V_{\mathrm{st}}(\mu_0) \leq v - \epsilon_0 V st ( μ 0 ) ≤ v − ϵ 0 , then z ≤ V s t ( μ 0 ) + ϵ / 2 < v − ϵ 0 + ϵ / 2 < v z \leq V_{\mathrm{st}}(\mu_0) + \epsilon/2 < v - \epsilon_0 + \epsilon/2 < v z ≤ V st ( μ 0 ) + ϵ /2 < v − ϵ 0 + ϵ /2 < v . If V s t ( μ 0 ) ≥ v + ϵ 0 V_{\mathrm{st}}(\mu_0) \geq v + \epsilon_0 V st ( μ 0 ) ≥ v + ϵ 0 , then z ≥ V s t ( μ 0 ) − ϵ / 2 ≥ v + ϵ 0 − ϵ / 2 > v z \geq V_{\mathrm{st}}(\mu_0) - \epsilon/2 \geq v + \epsilon_0 - \epsilon/2 > v z ≥ V st ( μ 0 ) − ϵ /2 ≥ v + ϵ 0 − ϵ /2 > v . So the algorithm answers Yes on every yes-instance and No on every no-instance of the promise.
Step 5: exact bit lengths and active workspace. Bit length of the distribution. Let D 0 D_0 D 0 be the product of the denominators in the initial distribution, and write μ 0 ( s ) = m s / D 0 \mu_0(s) = m_s/D_0 μ 0 ( s ) = m s / D 0 for integers m s ≥ 0 m_s \geq0 m s ≥ 0 summing to D 0 D_0 D 0 . Every realised update only reassigns existing mass: by Ψ ξ ( μ ) ( s ′ ) = ∑ s : ξ ( s ) = s ′ μ ( s ) \Psi_{\xi}(\mu)(s') = \sum_{s:\xi(s)=s'} \mu(s) Ψ ξ ( μ ) ( s ′ ) = ∑ s : ξ ( s ) = s ′ μ ( s ) , no transition probability is multiplied into a distribution’s coordinates at all, mass is only summed. Consequently, by induction on the recursion depth, every distribution ever passed to a recursive call has the form ( m s ′ / D 0 ) s (m'_s/D_0)_s ( m s ′ / D 0 ) s for integers 0 ≤ m s ′ ≤ D 0 0 \leq m'_s \leq D_0 0 ≤ m s ′ ≤ D 0 summing to D 0 D_0 D 0 : the base case μ 0 \mu_0 μ 0 has this form by definition, and if μ \mu μ does, then each coordinate of Ψ ξ ( μ ) \Psi_{\xi}(\mu) Ψ ξ ( μ ) is a sum of some of the m s m_s m s , still an integer between 0 0 0 and D 0 D_0 D 0 , over the same denominator D 0 D_0 D 0 . Unlike sampled-profile semantics, this denominator does not grow with depth. It takes O ( n ℓ ) O(n\ell) O ( n ℓ ) bits to store such a distribution at any depth, since D 0 D_0 D 0 has O ( ℓ ) O(\ell) O ( ℓ ) bits (it is a product of denominators that are substrings of the ℓ \ell ℓ -bit input) and there are n n n coordinates.
Bit length of the recursion’s values. Let D = ∏ s ∈ S ∏ a ∈ A ( s ) ∏ s ′ ∈ S den ( Δ ( s , a , s ′ ) ) D = \prod_{s\in S}\prod_{a\in A(s)}\prod_{s'\in S}\operatorname{den}(\Delta(s,a,s')) D = ∏ s ∈ S ∏ a ∈ A ( s ) ∏ s ′ ∈ S den ( Δ ( s , a , s ′ )) , where den \operatorname{den} den denotes a positive input denominator. Each individual denominator is a substring of the ℓ \ell ℓ -bit input, so the sum of their bit lengths is at most ℓ \ell ℓ , and hence D D D has O ( ℓ ) O(\ell) O ( ℓ ) bits by the same product-bit-length fact used in Appendix E.2.4, Step 6. A product P a , b ( ξ ) P_{a,b}(\xi) P a , b ( ξ ) selects one transition entry for each source state, so its unreduced denominator is a subproduct of the factors defining D D D ; therefore w a , b , ξ = D × P a , b ( ξ ) w_{a,b,\xi} = D \times P_{a,b}(\xi) w a , b , ξ = D × P a , b ( ξ ) is a nonnegative integer, computable from the individual factors’ numerators and denominators by elementary integer arithmetic, in time and space polynomial in the bit lengths already bounded, and ∑ ξ ∈ Ξ w a , b , ξ = D \sum_{\xi\in\Xi} w_{a,b,\xi} = D ∑ ξ ∈ Ξ w a , b , ξ = D . If a child returns j ξ / 2 p j_\xi/2^p j ξ / 2 p , then
C ^ r − 1 ( μ ) a , b = ∑ ξ w a , b , ξ j ξ D × 2 p , 0 ≤ ∑ ξ w a , b , ξ j ξ ≤ D × L × 2 p , \widehat{C}_{r-1}(\mu)_{a,b} = \frac{\sum_{\xi} w_{a,b,\xi}j_\xi}{D \times2^p}, \qquad0 \leq\sum_{\xi} w_{a,b,\xi}j_\xi\leq D \times L \times2^p, C r − 1 ( μ ) a , b = D × 2 p ∑ ξ w a , b , ξ j ξ , 0 ≤ ξ ∑ w a , b , ξ j ξ ≤ D × L × 2 p ,
since every summand is nonnegative and the total weight ∑ ξ w a , b , ξ = D \sum_{\xi} w_{a,b,\xi} = D ∑ ξ w a , b , ξ = D times the largest possible child value L 2 p L2^p L 2 p bounds the sum. All partial sums obey the same bound, so the whole realisation loop needs an integer accumulator with O ( ℓ + p + log ( L + 1 ) ) O(\ell+p+\log(L+1)) O ( ℓ + p + log ( L + 1 )) bits: summing exponentially many terms therefore costs time, one addition per realisation, but not space, since the accumulator’s bit length never depends on how many realisations have been added so far. The empirical average over k k k rows has common denominator k D 2 p kD2^p k D 2 p and numerator at most k D L 2 p kDL2^p k D L 2 p , adding only O ( log k ) O(\log k) O ( log k ) bits on top. Computing a new entry reuses this accumulator; it does not multiply together the denominators of different realisations or different rows. Maxima and minima select existing rational values without introducing new denominators, and the final rounding resets the returned value to denominator 2 p 2^p 2 p , exactly as in Appendix E.2.4, Step 6.
The active workspace. A realisation ξ : S → S \xi:S\to S ξ : S → S is enumerated as an n n n -digit base-n n n counter, one digit per state recording its image, using O ( n log n ) O(n\log n) O ( n log n ) bits. An active frame stores the current distribution (O ( n ℓ ) O(n\ell) O ( n ℓ ) bits, above), the current k k k -tuple and column (O ( k n log ( 2 A ) ) O(kn\log(2A)) O ( k n log ( 2 A )) and O ( n log ( 2 A ) ) O(n\log(2A)) O ( n log ( 2 A )) bits, as in Appendix E.2.4, Step 6), the current realisation counter (O ( n log n ) O(n\log n) O ( n log n ) bits), and the bounded accumulators and extrema (O ( ℓ + p + log ( L + 1 ) + log k ) O(\ell+p+\log(L+1)+\log k) O ( ℓ + p + log ( L + 1 ) + log k ) bits, from above). All local arithmetic, including the entropy routine, uses only polynomial auxiliary space in these lengths, by the same two-part argument as Appendix § 15 .2.4, Step 6: elementary arithmetic on integers of already-bounded bit length uses space polynomial in that length, and Lemma 6 bounds only H ^ p \widehat{H}_p H p ’s time, but since a deterministic algorithm can write to at most one new memory cell per step, a computation running in time t t t uses at most O ( t ) O(t) O ( t ) space, so the entropy routine also uses only polynomial auxiliary space. On a recursive call the parent frame is retained, but each realisation’s child frame is discarded once its contribution is added to the running sum, and each completed child in the tuple-and-column enumeration is discarded before the next is processed. Since the recursion depth strictly decreases from r = T r = T r = T at the root to r = 0 r = 0 r = 0 at a base case, and only the frames on the current path are retained, at most T T T nontrivial frames coexist at any time. The horizon T T T was shown polynomial in ℓ \ell ℓ in Appendix § 15 .2.4’s Step 1, the sample size k k k in its Step 2, and the precision p p p is fixed in its Step 3 independently of the instance. Total space is therefore at most T T T times a single frame’s polynomial bit length, a product of two quantities polynomial in ℓ \ell ℓ , hence itself polynomial in ℓ \ell ℓ . The number of realisations, tuples, and recomputed subproblems may be exponential; this affects time, not the workspace bound. ◀
The One Player Case
This section focuses on the single-decision-maker case, S max = ∅ S_{\max} = \varnothing S m a x = ∅ . We begin with Lemma 11, used for the upper-bound results: when there is a single decision-maker, optimising over the full mixed (or, for the averaged semantics, continuous statewise-mixed) strategy space is equivalent to optimising over pure profiles alone.
Fix f ∈ C ( D ( S ) ) f \in C(\mathcal{D}(S)) f ∈ C ( D ( S )) , the space of continuous real-valued functions on D ( S ) \mathcal{D}(S) D ( S ) . For ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } define
M f s p ( μ ) a , b = f ( Φ a , b ( μ ) ) , M f s t ( μ ) a , b = ∑ ξ P a , b ( ξ ) f ( Ψ ξ ( μ ) ) , M^{\mathrm{sp}}_f(\mu)_{a,b} = f(\Phi_{a,b}(\mu)), \qquad M^{\mathrm{st}}_f(\mu)_{a,b} = \sum_{\xi} P_{a,b}(\xi)f(\Psi_{\xi}(\mu)), M f sp ( μ ) a , b = f ( Φ a , b ( μ )) , M f st ( μ ) a , b = ξ ∑ P a , b ( ξ ) f ( Ψ ξ ( μ )) ,
( T ⋆ f ) ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M f ⋆ ( μ ) ) , (T_{\star}f)(\mu) = (1-\gamma)H(\mu) + \gamma\,\operatorname{val}(M^{\star}_f(\mu)), ( T ⋆ f ) ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M f ⋆ ( μ )) ,
where val ( ⋅ ) \operatorname{val}(\cdot) val ( ⋅ ) is the common minimax value of a finite zero-sum matrix game, as given by the finite minimax theorem in (37) ; the proof below also uses the equivalent one-sided characterisation of (52) , which lets either player’s opponent be restricted to a pure best response without changing the value. For the averaged semantics define separately
( T a v + f ) ( μ ) = ( 1 − γ ) H ( μ ) + γ min x ∈ X max y ∈ Y f ( Υ x , y ( μ ) ) , (T_{\mathrm{av}}^{+}f)(\mu) = (1-\gamma)H(\mu) + \gamma\min_{x \in\mathcal{X}}\max_{y \in\mathcal{Y}} f(\Upsilon_{x,y}(\mu)), ( T av + f ) ( μ ) = ( 1 − γ ) H ( μ ) + γ x ∈ X min y ∈ Y max f ( Υ x , y ( μ )) ,
( T a v − f ) ( μ ) = ( 1 − γ ) H ( μ ) + γ max y ∈ Y min x ∈ X f ( Υ x , y ( μ ) ) . (T_{\mathrm{av}}^{-}f)(\mu) = (1-\gamma)H(\mu) + \gamma\max_{y \in\mathcal{Y}}\min_{x \in\mathcal{X}} f(\Upsilon_{x,y}(\mu)). ( T av − f ) ( μ ) = ( 1 − γ ) H ( μ ) + γ y ∈ Y max x ∈ X min f ( Υ x , y ( μ )) .
When S max = ∅ S_{\max} = \varnothing S m a x = ∅ , P max \mathcal{P}_{\max} P m a x contains only the empty (trivial) profile, which we call b 0 b_0 b 0 , and Y \mathcal{Y} Y is the single point { y 0 } \{y_0\} { y 0 } ; write Φ a = Φ a , b 0 \Phi_a = \Phi_{a,b_0} Φ a = Φ a , b 0 , P a ( ξ ) : = P a , b 0 ( ξ ) P_a(\xi) := P_{a,b_0}(\xi) P a ( ξ ) := P a , b 0 ( ξ ) , and Υ x : = Υ x , y 0 \Upsilon_x := \Upsilon_{x,y_0} Υ x := Υ x , y 0 for a ∈ P min a \in\mathcal{P}_{\min} a ∈ P m i n , x ∈ X x \in\mathcal{X} x ∈ X . Consequently max y ∈ Y \max_{y \in\mathcal{Y}} max y ∈ Y is vacuous, so T a v + = T a v − T_{\mathrm{av}}^{+} = T_{\mathrm{av}}^{-} T av + = T av − ; write T a v T_{\mathrm{av}} T av for their common value,
( T a v f ) ( μ ) = ( 1 − γ ) H ( μ ) + γ min x ∈ X f ( Υ x ( μ ) ) . (67) (T_{\mathrm{av}}f)(\mu) = (1-\gamma)H(\mu) + \gamma\min_{x \in\mathcal{X}} f(\Upsilon_x(\mu)). \tag*{(67)} ( T av f ) ( μ ) = ( 1 − γ ) H ( μ ) + γ x ∈ X min f ( Υ x ( μ )) . (67)
Lemma 11 (Pure strategies). Suppose S max = ∅ S_{\max} = \varnothing S m a x = ∅ . Define W 0 ≡ 0 W_0 \equiv0 W 0 ≡ 0 and W t + 1 = T ⋆ W t W_{t+1} = T_{\star}W_t W t + 1 = T ⋆ W t , where T ⋆ T_{\star} T ⋆ is the one-step Bellman operator under semantics ⋆ \star ⋆ . Then for every ⋆ ∈ { a v , s p , s t } \star\in\{\mathrm{av},\mathrm{sp},\mathrm{st}\} ⋆ ∈ { av , sp , st } , the minimum defining W t + 1 W_{t+1} W t + 1 is attained by a pure profile. In particular,
W t + 1 ( μ ) = ( 1 − γ ) H ( μ ) + γ min a ∈ P min C t ⋆ ( μ , a ) , where W_{t+1}(\mu) = (1-\gamma)H(\mu) + \gamma\min_{a \in\mathcal{P}_{\min}} C_t^{\star}(\mu,a), \qquad\text{where} W t + 1 ( μ ) = ( 1 − γ ) H ( μ ) + γ a ∈ P m i n min C t ⋆ ( μ , a ) , where
C t a v ( μ , a ) = C t s p ( μ , a ) = W t ( Φ a ( μ ) ) and C t s t ( μ , a ) = ∑ ξ ∈ Ξ P a ( ξ ) W t ( Ψ ξ ( μ ) ) . C_t^{\mathrm{av}}(\mu,a)=C_t^{\mathrm{sp}}(\mu,a)=W_t(\Phi_a(\mu)) \text{ and } C_t^{\mathrm{st}}(\mu,a)=\sum_{\xi\in\Xi}P_a(\xi)W_t(\Psi_\xi(\mu)). C t av ( μ , a ) = C t sp ( μ , a ) = W t ( Φ a ( μ )) and C t st ( μ , a ) = ξ ∈ Ξ ∑ P a ( ξ ) W t ( Ψ ξ ( μ )) .
Proof. Sampled semantics. Since W t + 1 = T ⋆ W t W_{t+1}=\mathcal{T}_\star W_t W t + 1 = T ⋆ W t , unfolding the definition of T ⋆ \mathcal{T}_\star T ⋆ gives W t + 1 ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M W t ⋆ ( μ ) ) W_{t+1}(\mu)=(1-\gamma)\mathbf{H}(\mu)+\gamma\operatorname{val}(M^\star_{W_t}(\mu)) W t + 1 ( μ ) = ( 1 − γ ) H ( μ ) + γ val ( M W t ⋆ ( μ )) , where M f ⋆ ( μ ) M^\star_f(\mu) M f ⋆ ( μ ) denotes the matrix M f ⋆ ( μ ) M^\star_f(\mu) M f ⋆ ( μ ) defined above, evaluated at f = W t f=W_t f = W t . With S max = ∅ S_{\max}=\varnothing S m a x = ∅ , P max = { b 0 } P_{\max}=\{b_0\} P m a x = { b 0 } , so this matrix has a single column; writing M a : = M W t ⋆ ( μ ) a , b 0 M_a:=M^\star_{W_t}(\mu)_{a,b_0} M a := M W t ⋆ ( μ ) a , b 0 , the definition of M f ⋆ M^\star_f M f ⋆ gives M a = W t ( Φ a ( μ ) ) = C t s p ( μ , a ) M_a=W_t(\Phi_a(\mu))=C_t^{\mathrm{sp}}(\mu,a) M a = W t ( Φ a ( μ )) = C t sp ( μ , a ) for ⋆ = s p \star=\mathrm{sp} ⋆ = sp and M a = ∑ ξ ∈ Ξ P a ( ξ ) W t ( Ψ ξ ( μ ) ) = C t s t ( μ , a ) M_a=\sum_{\xi\in\Xi}P_a(\xi)W_t(\Psi_\xi(\mu))=C_t^{\mathrm{st}}(\mu,a) M a = ∑ ξ ∈ Ξ P a ( ξ ) W t ( Ψ ξ ( μ )) = C t st ( μ , a ) for ⋆ = s t \star=\mathrm{st} ⋆ = st . By (52) , val ( M W t ⋆ ( μ ) ) = min σ ∈ D ( P min ) σ T M W t ⋆ \operatorname{val}(M^\star_{W_t}(\mu))=\min_{\sigma\in\mathcal{D}(P_{\min})}\sigma^{\mathsf{T}}M^\star_{W_t} val ( M W t ⋆ ( μ )) = min σ ∈ D ( P m i n ) σ T M W t ⋆ , a linear functional of σ \sigma σ , minimised at a vertex of D ( P min ) \mathcal{D}(P_{\min}) D ( P m i n ) , i.e. at a pure a a a , for both ⋆ = s p \star=\mathrm{sp} ⋆ = sp and ⋆ = s t \star=\mathrm{st} ⋆ = st .
Averaged-distribution semantics. We show by induction on t t t that W t W_t W t is concave on D ( S ) \mathcal{D}(S) D ( S ) . The base case W 0 ≡ 0 W_0\equiv0 W 0 ≡ 0 is trivially concave. For the inductive step, suppose W t W_t W t is concave. For fixed x ∈ X x\in\mathcal{X} x ∈ X , μ ↦ W t ( Υ x ( μ ) ) \mu\mapsto W_t(\Upsilon_x(\mu)) μ ↦ W t ( Υ x ( μ )) is concave, being W t W_t W t composed with the map μ ↦ Υ x ( μ ) \mu\mapsto\Upsilon_x(\mu) μ ↦ Υ x ( μ ) , which is linear in μ \mu μ for fixed x x x by (10) . The pointwise infimum of any family of concave functions is concave (dual to the fact that a pointwise supremum of convex functions is convex), so μ ↦ inf x ∈ X W t ( Υ x ( μ ) ) \mu\mapsto\inf_{x\in\mathcal{X}}W_t(\Upsilon_x(\mu)) μ ↦ inf x ∈ X W t ( Υ x ( μ )) is concave, and hence so is W t + 1 = ( 1 − γ ) H + γ inf x W t ( Υ x ( ⋅ ) ) W_{t+1}=(1-\gamma)\mathbf{H}+\gamma\inf_x W_t(\Upsilon_x(\cdot)) W t + 1 = ( 1 − γ ) H + γ inf x W t ( Υ x ( ⋅ )) , being a sum of two concave functions; this closes the induction. Now fix μ \mu μ and consider x ↦ W t ( Υ x ( μ ) ) x\mapsto W_t(\Upsilon_x(\mu)) x ↦ W t ( Υ x ( μ )) : Υ x ( μ ) \Upsilon_x(\mu) Υ x ( μ ) is linear in x x x for fixed μ \mu μ (again by (10) ), so this is a concave function of x x x , since W t W_t W t is concave.
Every point x ∈ X x\in\mathcal{X} x ∈ X is exactly the convex combination of its vertices given by its induced profile law x ^ \widehat{x} x ((20) ): x = ∑ a x ^ ( a ) a x=\sum_a\widehat{x}(a)a x = ∑ a x ( a ) a , where x ^ ( a ) = ∏ s x ( s ) ( a ( s ) ) \widehat{x}(a)=\prod_s x(s)(a(s)) x ( a ) = ∏ s x ( s ) ( a ( s )) is the probability of the pure profile a a a under independently sampling each state’s action from x x x . Linearity of Υ x ( μ ) \Upsilon_x(\mu) Υ x ( μ ) in x x x ((10) ) gives
Υ x ( μ ) = ∑ a x ^ ( a ) Υ a ( μ ) = ∑ a x ^ ( a ) Φ a ( μ ) , \Upsilon_x(\mu)=\sum_a\widehat{x}(a)\Upsilon_a(\mu)=\sum_a\widehat{x}(a)\Phi_a(\mu), Υ x ( μ ) = a ∑ x ( a ) Υ a ( μ ) = a ∑ x ( a ) Φ a ( μ ) ,
using Υ a = Φ a \Upsilon_a=\Phi_a Υ a = Φ a as above. Since W t W_t W t is concave, Jensen’s inequality gives
W t ( Υ x ( μ ) ) = W t ( ∑ a x ^ ( a ) Φ a ( μ ) ) ≥ ∑ a x ^ ( a ) W t ( Φ a ( μ ) ) . W_t(\Upsilon_x(\mu))=W_t\left(\sum_a\widehat{x}(a)\Phi_a(\mu)\right)\geq\sum_a\widehat{x}(a)W_t(\Phi_a(\mu)). W t ( Υ x ( μ )) = W t ( a ∑ x ( a ) Φ a ( μ ) ) ≥ a ∑ x ( a ) W t ( Φ a ( μ )) .
Since the x ^ ( a ) \widehat{x}(a) x ( a ) are nonnegative and sum to 1 1 1 , this weighted average of the values W t ( Φ a ( μ ) ) W_t(\Phi_a(\mu)) W t ( Φ a ( μ )) is at least their smallest term:
∑ a x ^ ( a ) W t ( Φ a ( μ ) ) ≥ min a W t ( Φ a ( μ ) ) . \sum_a\widehat{x}(a)W_t(\Phi_a(\mu))\geq\min_a W_t(\Phi_a(\mu)). a ∑ x ( a ) W t ( Φ a ( μ )) ≥ a min W t ( Φ a ( μ )) .
Combining the two,
W t ( Υ x ( μ ) ) ≥ ∑ a x ^ ( a ) W t ( Φ a ( μ ) ) ≥ min a W t ( Φ a ( μ ) ) . W_t(\Upsilon_x(\mu))\geq\sum_a\widehat{x}(a)W_t(\Phi_a(\mu))\geq\min_a W_t(\Phi_a(\mu)). W t ( Υ x ( μ )) ≥ a ∑ x ( a ) W t ( Φ a ( μ )) ≥ a min W t ( Φ a ( μ )) .
Its vertices are exactly the pure profiles; since each pure profile a a a , viewed as the degenerate point x = a ∈ X x=a\in\mathcal{X} x = a ∈ X , satisfies Υ a = Φ a \Upsilon_a=\Phi_a Υ a = Φ a (as above), the minimum over all of X \mathcal{X} X is at most its value at this particular point: min x ∈ X W t ( Υ x ( μ ) ) ≤ W t ( Υ a ( μ ) ) = W t ( Φ a ( μ ) ) \min_{x\in\mathcal{X}}W_t(\Upsilon_x(\mu))\leq W_t(\Upsilon_a(\mu))=W_t(\Phi_a(\mu)) min x ∈ X W t ( Υ x ( μ )) ≤ W t ( Υ a ( μ )) = W t ( Φ a ( μ )) for every pure a a a . Taking the minimum over a a a on the right-hand side gives min x ∈ X W t ( Υ x ( μ ) ) ≤ min a W t ( Φ a ( μ ) ) \min_{x\in\mathcal{X}}W_t(\Upsilon_x(\mu))\leq\min_a W_t(\Phi_a(\mu)) min x ∈ X W t ( Υ x ( μ )) ≤ min a W t ( Φ a ( μ )) .
The Jensen inequality above, W t ( Υ x ( μ ) ) ≥ min a W t ( Φ a ( μ ) ) W_t(\Upsilon_x(\mu))\geq\min_a W_t(\Phi_a(\mu)) W t ( Υ x ( μ )) ≥ min a W t ( Φ a ( μ )) , holds for every x ∈ X x\in\mathcal{X} x ∈ X , and its right-hand side does not depend on x x x ; taking the minimum over x ∈ X x\in\mathcal{X} x ∈ X on both sides therefore gives min x ∈ X W t ( Υ x ( μ ) ) ≥ min a W t ( Φ a ( μ ) ) \min_{x\in\mathcal{X}}W_t(\Upsilon_x(\mu))\geq\min_a W_t(\Phi_a(\mu)) min x ∈ X W t ( Υ x ( μ )) ≥ min a W t ( Φ a ( μ )) . Combined with the reverse inequality just shown, min x ∈ X W t ( Υ x ( μ ) ) = min a W t ( Φ a ( μ ) ) \min_{x\in\mathcal{X}}W_t(\Upsilon_x(\mu))=\min_a W_t(\Phi_a(\mu)) min x ∈ X W t ( Υ x ( μ )) = min a W t ( Φ a ( μ )) , so the infimum over all of X \mathcal{X} X is attained at a pure profile.
Since P min \mathcal{P}_{\min} P m i n is finite, the minimum in each step is attained; picking a minimising a a a at each step of the recursion yields, from any starting distribution, a witnessing sequence of pure actions achieving W t + 1 W_{t+1} W t + 1 .
Corollary 25 (One-player averaged and sampled-profile values coincide). If S max = ∅ S_{\max} = \varnothing S m a x = ∅ , then V a v = V s p V_{\mathrm{av}} = V_{\mathrm{sp}} V av = V sp .
Proof. Both value functions arise as limits of finite-horizon backups from the same base case W 0 ≡ 0 W_0 \equiv0 W 0 ≡ 0 :
W t + 1 a v = ( 1 − γ ) H + γ min x ∈ X W t a v ( Υ x ( ⋅ ) ) W^{\mathrm{av}}_{t+1} = (1-\gamma)H+\gamma\min_{x\in X} W^{\mathrm{av}}_t(\Upsilon_x(\cdot)) W t + 1 av = ( 1 − γ ) H + γ x ∈ X min W t av ( Υ x ( ⋅ )) and
W t + 1 s p = ( 1 − γ ) H + γ min a ∈ P min W t s p ( Φ a ( ⋅ ) ) . W^{\mathrm{sp}}_{t+1} = (1-\gamma)H+\gamma\min_{a\in\mathcal{P}_{\min}} W^{\mathrm{sp}}_t(\Phi_a(\cdot)). W t + 1 sp = ( 1 − γ ) H + γ a ∈ P m i n min W t sp ( Φ a ( ⋅ )) . The proof of Lemma 11 shows, at every step of this recursion, that min x ∈ X W t ( Υ x ( μ ) ) = min a ∈ P min W t ( Φ a ( μ ) ) \min_{x\in X} W_t(\Upsilon_x(\mu)) = \min_{a\in\mathcal{P}_{\min}} W_t(\Phi_a(\mu)) min x ∈ X W t ( Υ x ( μ )) = min a ∈ P m i n W t ( Φ a ( μ )) whenever W t W_t W t is concave, and that concavity is preserved from step to step. So if W t a v = W t s p W^{\mathrm{av}}_t = W^{\mathrm{sp}}_t W t av = W t sp , the two recursions apply the same minimisation to the same function and produce the same W t + 1 W_{t+1} W t + 1 ; since W 0 a v = W 0 s p ≡ 0 W^{\mathrm{av}}_0 = W^{\mathrm{sp}}_0 \equiv0 W 0 av = W 0 sp ≡ 0 , induction on t t t gives W t a v = W t s p W^{\mathrm{av}}_t = W^{\mathrm{sp}}_t W t av = W t sp for every t ≥ 0 t \ge0 t ≥ 0 . Both sequences converge uniformly, at the same geometric rate, to their respective infinite-horizon values as t → ∞ t \to\infty t → ∞ ((46) for av, (39) for sp), and a sequence has at most one limit, so V a v = lim t W t a v = lim t W t s p = V s p V_{\mathrm{av}} = \lim_t W^{\mathrm{av}}_t = \lim_t W^{\mathrm{sp}}_t = V_{\mathrm{sp}} V av = lim t W t av = lim t W t sp = V sp .
Upper Bounds for One Player
Since S max = ∅ S_{\max} = \varnothing S m a x = ∅ , P max \mathcal{P}_{\max} P m a x contains only the empty profile b 0 b_0 b 0 , so Π max a v \Pi^{\mathrm{av}}_{\max} Π m a x av contains only the single strategy τ 0 \tau_0 τ 0 that plays b 0 b_0 b 0 at every stage, regardless of history. Consequently sup π max ∈ Π max a v \sup_{\pi_{\max}\in\Pi^{\mathrm{av}}_{\max}} sup π m a x ∈ Π m a x av in the definitions of both V a v + V^+_{\mathrm{av}} V av + and V a v − V^-_{\mathrm{av}} V av − ranges over the one strategy τ 0 \tau_0 τ 0 and may be dropped, so
V a v + ( μ ) = inf π min ∈ Π min a v Obj γ a v ( π min , τ 0 ; μ ) = V a v − ( μ ) V^+_{\mathrm{av}}(\mu) = \inf_{\pi_{\min}\in\Pi^{\mathrm{av}}_{\min}} \operatorname{Obj}^{\mathrm{av}}_\gamma(\pi_{\min},\tau_0;\mu) = V^-_{\mathrm{av}}(\mu) V av + ( μ ) = π m i n ∈ Π m i n av inf Obj γ av ( π m i n , τ 0 ; μ ) = V av − ( μ )
for every μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) . As with sp and st (Theorem 4), we therefore omit the polarity in the one-player case and write V a v V_{\mathrm{av}} V av , GapVal a v , ε 0 \operatorname{GapVal}_{\mathrm{av},\varepsilon_0} GapVal av , ε 0 .
Lemma 26 (One-player averaged gap value lies in NP). For fixed rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 , if S max = ∅ S_{\max} = \varnothing S m a x = ∅ then GapVal a v , ε 0 ∈ N P \operatorname{GapVal}_{\mathrm{av},\varepsilon_0} \in\mathrm{NP} GapVal av , ε 0 ∈ NP .
Proof. Recall that a promise problem lies in NP if there is a polynomial-length certificate and a polynomial-time verifier such that every yes-instance has some certificate the verifier accepts (completeness) and every no-instance has every certificate rejected by the verifier (soundness); we exhibit exactly these four ingredients.
Since S max = ∅ S_{\max} = \varnothing S m a x = ∅ , every state lies in S min S_{\min} S m i n , and by Lemma 11, a finite-horizon optimum is attained by pure profiles, so every trajectory considered here is pure. So by (9) ,(10) the successor map Υ a , b = Φ a , b \Upsilon_{a,b} = \Phi_{a,b} Υ a , b = Φ a , b does not depend on b b b ; write
Φ a ( μ ) ( s ′ ) : = ∑ s ∈ S μ ( s ) Δ ( s , a ( s ) , s ′ ) \Phi_a(\mu)(s') := \sum_{s\in S} \mu(s)\Delta(s,a(s),s') Φ a ( μ ) ( s ′ ) := s ∈ S ∑ μ ( s ) Δ ( s , a ( s ) , s ′ ) for this common value.
Let e = min { ε 0 , 1 } e = \min\{\varepsilon_0,1\} e = min { ε 0 , 1 } and L = ⌈ log 2 n ⌉ L = \lceil\log_2 n\rceil L = ⌈ log 2 n ⌉ , and let T T T be as in the proof of Theorem 7 (Step 1): the least positive integer with γ T L ≤ e / 4 \gamma^T L \le e/4 γ T L ≤ e /4 . Thus U T ( μ 0 ) = min a ⃗ U T a ⃗ ( μ 0 ) U_T(\mu_0) = \min_{\vec{a}} U^{\vec{a}}_T(\mu_0) U T ( μ 0 ) = min a U T a ( μ 0 ) , where a ⃗ = ( a 0 , … , a T − 2 ) \vec{a} = (a_0,\ldots,a_{T-2}) a = ( a 0 , … , a T − 2 ) ranges over sequences of pure profiles, μ 0 = μ 0 \mu_0 = \mu_0 μ 0 = μ 0 , μ t + 1 = Φ a t ( μ t ) \mu_{t+1} = \Phi_{a_t}(\mu_t) μ t + 1 = Φ a t ( μ t ) , and
U T a ⃗ ( μ 0 ) = ( 1 − γ ) ∑ t = 0 T − 1 γ t H ( μ t ) . U^{\vec{a}}_T(\mu_0) = (1-\gamma)\sum_{t=0}^{T-1}\gamma^t H(\mu_t). U T a ( μ 0 ) = ( 1 − γ ) t = 0 ∑ T − 1 γ t H ( μ t ) . The certificate is such a sequence a ⃗ \vec{a} a , with O ( T n log ( 2 A ) ) O(Tn\log(2A)) O ( T n log ( 2 A )) bits by Appendix E.2.3.
Choose the least p ≥ 1 p \ge1 p ≥ 1 with 2 − p ≤ e / 8 2^{-p} \le e/8 2 − p ≤ e /8 . The verifier computes
z a ⃗ = ( 1 − γ ) ∑ t = 0 T − 1 γ t H ^ p ( μ t ) z^{\vec{a}} = (1-\gamma)\sum_{t=0}^{T-1}\gamma^t \widehat{H}_p(\mu_t) z a = ( 1 − γ ) t = 0 ∑ T − 1 γ t H p ( μ t ) along the guessed a ⃗ \vec{a} a and accepts iff z a ⃗ ≤ v z^{\vec{a}} \le v z a ≤ v . Since ( 1 − γ ) ∑ t = 0 T − 1 γ t ≤ 1 (1-\gamma)\sum_{t=0}^{T-1}\gamma^t \le1 ( 1 − γ ) ∑ t = 0 T − 1 γ t ≤ 1 and each term satisfies
∣ H ^ p ( μ t ) − H ( μ t ) ∣ ≤ 2 − p , \left|\widehat{H}_p(\mu_t)-H(\mu_t)\right| \le2^{-p}, H p ( μ t ) − H ( μ t ) ≤ 2 − p , ∣ z a ⃗ − U T a ⃗ ( μ 0 ) ∣ ≤ 2 − p ≤ e / 8 \left|z^{\vec{a}}-U^{\vec{a}}_T(\mu_0)\right| \le2^{-p} \le e/8 z a − U T a ( μ 0 ) ≤ 2 − p ≤ e /8 for every a ⃗ \vec{a} a . By (46) , 0 ≤ V a v + ( μ 0 ) − U T ( μ 0 ) ≤ e / 4 0 \le V^+_{\mathrm{av}}(\mu_0)-U_T(\mu_0) \le e/4 0 ≤ V av + ( μ 0 ) − U T ( μ 0 ) ≤ e /4 ; since V a v + = V a v V^+_{\mathrm{av}} = V_{\mathrm{av}} V av + = V av was shown above, this reads 0 ≤ V a v ( μ 0 ) − U T ( μ 0 ) ≤ e / 4 0 \le V_{\mathrm{av}}(\mu_0)-U_T(\mu_0) \le e/4 0 ≤ V av ( μ 0 ) − U T ( μ 0 ) ≤ e /4 , so for the optimal a ⃗ ∗ \vec{a}^{*} a ∗ ,
∣ z a ⃗ ∗ − V a v ( μ 0 ) ∣ ≤ ∣ z a ⃗ ∗ − U T ( μ 0 ) ∣ + ∣ U T ( μ 0 ) − V a v ( μ 0 ) ∣ ≤ e / 8 + e / 4 = 3 e / 8 , \left|z^{\vec{a}^{*}}-V_{\mathrm{av}}(\mu_0)\right| \le\left|z^{\vec{a}^{*}}-U_T(\mu_0)\right|+\left|U_T(\mu_0)-V_{\mathrm{av}}(\mu_0)\right| \le e/8+e/4 = 3e/8, z a ∗ − V av ( μ 0 ) ≤ z a ∗ − U T ( μ 0 ) + ∣ U T ( μ 0 ) − V av ( μ 0 ) ∣ ≤ e /8 + e /4 = 3 e /8 ,
and for every a ⃗ \vec{a} a , since U T a ⃗ ( μ 0 ) ≥ U T ( μ 0 ) U^{\vec{a}}_{T}(\mu_0) \ge U_T(\mu_0) U T a ( μ 0 ) ≥ U T ( μ 0 ) ,
z a ⃗ ≥ U T a ⃗ ( μ 0 ) − e / 8 ≥ U T ( μ 0 ) − e / 8 ≥ V a v ( μ 0 ) − e / 4 − e / 8 = V a v ( μ 0 ) − 3 e / 8 z^{\vec{a}} \ge U^{\vec{a}}_{T}(\mu_0) - e/8 \ge U_T(\mu_0) - e/8 \ge V_{\mathrm{av}}(\mu_0) - e/4 - e/8 = V_{\mathrm{av}}(\mu_0) - 3e/8 z a ≥ U T a ( μ 0 ) − e /8 ≥ U T ( μ 0 ) − e /8 ≥ V av ( μ 0 ) − e /4 − e /8 = V av ( μ 0 ) − 3 e /8 .
Since e ≤ ε 0 e \le\varepsilon_0 e ≤ ε 0 , 3 e / 8 < ε 0 3e/8 < \varepsilon_0 3 e /8 < ε 0 . If V a v ( μ 0 ) ≤ v − ε 0 V_{\mathrm{av}}(\mu_0) \le v-\varepsilon_0 V av ( μ 0 ) ≤ v − ε 0 , the certificate a ⃗ ∗ \vec{a}^{*} a ∗ gives z a ⃗ ∗ ≤ v − ε 0 + 3 e / 8 < v z^{\vec{a}^{*}} \le v-\varepsilon_0+3e/8 < v z a ∗ ≤ v − ε 0 + 3 e /8 < v , so it is accepted: completeness holds. If V a v ( μ 0 ) ≥ v + ε 0 V_{\mathrm{av}}(\mu_0) \ge v+\varepsilon_0 V av ( μ 0 ) ≥ v + ε 0 , every certificate a ⃗ \vec{a} a gives z a ⃗ ≥ v + ε 0 − 3 e / 8 > v z^{\vec{a}} \ge v+\varepsilon_0-3e/8 > v z a ≥ v + ε 0 − 3 e /8 > v , so every certificate is rejected: soundness holds. Computing each μ t \mu_t μ t takes polynomial time and computing each H ^ p ( μ t ) \widehat{H}_{p}(\mu_t) H p ( μ t ) takes polynomial time by Lemma 6, so verification is polynomial-time. Together with the certificate’s polynomial length, this establishes all four ingredients recalled above, so GapVal a v , ε 0 ∈ N P \operatorname{GapVal}_{\mathrm{av},\varepsilon_0} \in\mathrm{NP} GapVal av , ε 0 ∈ NP .
Lemma 27 (One-player sampled-profile gap value lies in NP). For fixed rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 , if S max = ∅ S_{\max} = \varnothing S m a x = ∅ then GapVal s p , ε 0 ∈ N P \operatorname{GapVal}_{\mathrm{sp},\varepsilon_0} \in\mathrm{NP} GapVal sp , ε 0 ∈ NP .
Proof. With one player the matrix has one column, so val ( M ) = min σ σ T M \operatorname{val}(M) = \min_{\sigma} \sigma^{\mathsf{T}}M val ( M ) = min σ σ T M minimises a linear functional of σ \sigma σ over the probability simplex on P min P_{\min} P m i n . A linear functional over a simplex attains its minimum at an extreme point, and the simplex’s extreme points are exactly the point masses on pure profiles, so the minimum is attained at some pure profile a a a . Consequently a finite-horizon optimum follows a deterministic sequence of T − 1 T-1 T − 1 pure profiles.
Since S max = ∅ S_{\max} = \varnothing S m a x = ∅ , every state lies in S min S_{\min} S m i n , so by (9) the successor map Φ a , b \Phi_{a,b} Φ a , b does not depend on b b b ; write Φ a ( μ ) ( s ′ ) : = ∑ s ∈ S μ ( s ) Δ ( s , a ( s ) , s ′ ) \Phi_a(\mu)(s') := \sum_{s \in S} \mu(s)\Delta(s,a(s),s') Φ a ( μ ) ( s ′ ) := ∑ s ∈ S μ ( s ) Δ ( s , a ( s ) , s ′ ) for this common value, exactly as in Lemma 26. Under sampled-profile semantics a pure profile has nothing to sample: with no genuinely mixed action to aggregate, the distribution is updated by the same map Φ a \Phi_a Φ a used under averaged-distribution semantics, so a fixed pure-profile trajectory’s distribution sequence and entropy values are identical under either semantics. The polynomial-time verifier and the promise margins are therefore exactly those of Lemma 26. This proves NP membership.
Lemma 28 (One-player sampled-transition gap value). For fixed rational γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 , if S max = ∅ S_{\max} = \varnothing S m a x = ∅ then GapVal s t , ε 0 ∈ P S P A C E \operatorname{GapVal}_{\mathrm{st},\varepsilon_0} \in\mathrm{PSPACE} GapVal st , ε 0 ∈ PSPACE .
Proof. With S max = ∅ S_{\max} = \varnothing S m a x = ∅ , every continuation matrix M = [ ∑ ξ P a , b ( ξ ) W r − 1 s t ( Ψ ξ ( μ ) ) ] a , b M = [\sum_{\xi} P_{a,b}(\xi)W^{\mathrm{st}}_{r-1}(\Psi_{\xi}(\mu))]_{a,b} M = [ ∑ ξ P a , b ( ξ ) W r − 1 st ( Ψ ξ ( μ )) ] a , b has a single column; write M a M_a M a for its entry at the trivial column, matching Lemma 11’s notation. Then val ( M ) = min σ σ T M \operatorname{val}(M) = \min_{\sigma} \sigma^{\mathsf{T}}M val ( M ) = min σ σ T M minimises a linear functional of σ \sigma σ over the probability simplex on P min P_{\min} P m i n ; this minimum is attained at an extreme point of the simplex, that is, at some pure profile a ∗ a^* a ∗ , giving val ( M ) = M a ∗ \operatorname{val}(M) = M_{a^*} val ( M ) = M a ∗ . Consequently, for the empirical value s k ( M ) = min ( a 1 , … , a k ) 1 k ∑ i M a i s_k(M) = \min_{(a_1,\ldots,a_k)} \frac{1}{k}\sum_i M_{a_i} s k ( M ) = min ( a 1 , … , a k ) k 1 ∑ i M a i of Appendix E.2.4’s Step 2, setting every a i = a ∗ a_i = a^* a i = a ∗ already attains s k ( M ) = M a ∗ = val ( M ) s_k(M) = M_{a^*} = \operatorname{val}(M) s k ( M ) = M a ∗ = val ( M ) exactly, for every k ≥ 1 k \ge1 k ≥ 1 : no averaging over several profiles is needed, and no sparsification error remains. In particular α = 0 \alpha= 0 α = 0 and k = 1 k = 1 k = 1 suffice in Appendix E.2.4’s Step 2, and the response loop over P max P_{\max} P m a x in its Steps 2 and 3 is omitted, since P max P_{\max} P m a x has only the single trivial column.
Keep the exact sum over all realisations (Step 2 above) and the fixed-precision recursion (66) unchanged. With α = 0 \alpha= 0 α = 0 , the error recurrence of Step 3 becomes B r = γ B r − 1 + 2 d B_r = \gamma B_{r-1} + 2d B r = γ B r − 1 + 2 d , so its general closed form B r ≤ ( γ α + 2 d ) / ( 1 − γ ) B_r \le(\gamma\alpha+ 2d)/(1-\gamma) B r ≤ ( γ α + 2 d ) / ( 1 − γ ) specializes to B r ≤ 2 d / ( 1 − γ ) ≤ e / 4 B_r \le2d/(1-\gamma) \le e/4 B r ≤ 2 d / ( 1 − γ ) ≤ e /4 , so (64) and Step 4’s argument go through unchanged and prove promise correctness. The bit-length argument of Step 5 does not depend on the specific value of k k k , beyond k k k itself being polynomial in ℓ \ell ℓ , which k = 1 k=1 k = 1 trivially is, so the same space bound applies. Unlike the one-player sp case (Lemma 27), a pure choice here does not produce a deterministic successor, since the arena’s own transitions still realise stochastically; so this argument does not yield a polynomial-length trajectory certificate or an NP upper bound, only the polynomial-space bound above.
Lower Bounds
We next give the lower bound once, in a form that can be cited from all three semantic subsections. The argument runs in three steps. We start from a problem already known to be NP-hard: distinguishing, for a family of triples over some universe, an instance that can be partitioned into disjoint triples from one where every such partition falls well short of covering the universe [28 ] . We first convert this into an equally hard question about Set-Cover: distinguishing an instance with a small, exact cover from one where every cover needs substantially more sets (Lemma 34 ). We then feed that Set-Cover instance into the gadget described next. It is built so that its one-step entropy H ( μ 1 ) \mathrm{H}(\mu_1) H ( μ 1 ) comes out small exactly when the chosen cover is small, and larger otherwise. Because every state the gadget reaches after one step stays fixed forever after, this one-step gap passes through unchanged to the infinite-horizon objective.
We restrict this reduction to deterministic arenas for two reasons. First, on a deterministic arena, a pure profile leaves nothing for sampled-profile semantics to sample and nothing for sampled-transition semantics to realise, so all three semantics agree exactly on the resulting trajectory; this is what lets a single construction transfer hardness to all three semantics at once, rather than requiring three separate reductions. Second, since every deterministic game is also a (degenerate) general entropy stochastic game, hardness proved on this restricted family is at least as strong as hardness proved on the general one; restricting to it only makes the construction simpler.
The rest of this subsection assembles the theorem from four facts, each proved once and then combined. Appendix § 16 .2.1 defines the game construction, which turns any Set-Cover instance into a single-player entropy game with one absorbing state per available set. Appendix § 16 .2.2 proves, in Lemma 31, that on this game the value under all three semantics is the same single number: the entropy of the distribution reached after one step, so a gap in that one number is automatically a gap under every semantics at once. Appendix § 16 .2.3 proves, in Lemma 33 , a precise two-sided bound on that one-step entropy in terms of how many sets the routing uses, so a small cover gives low entropy and a cover forced to use more sets gives provably higher entropy. Appendix § 16 .2.4 proves, in Lemma 34 , that distinguishing a small-cover Set-Cover instance from a large-cover one is itself NP-hard, by reduction from Petrank’s perfect-completeness gap for bounded-occurrence three-dimensional matching [28 ] : any sufficiently small cover can be converted, by repeatedly discarding a triple that overlaps another, into an almost-as-small disjoint matching, so a cover much smaller than the claimed gap would yield a matching larger than Petrank’s promise allows. Appendix § 16 .2.5 then combines all three: feeding the hard instances from Lemma 34 through the gadget of Appendix § 16 .2.1, and reading off the resulting entropy gap via Lemmas 31 and 33 , turns the NP-hardness of telling the two Set-Cover cases apart into the NP-hardness of telling the two entropy values apart, which is the theorem.
The Set-Cover gadget
Definition 29 (Deterministic entropy game). A game is deterministic if for every state-action pair s , a s,a s , a , Δ ( s , a ) = δ s ′ \Delta(s,a) = \delta_{s'} Δ ( s , a ) = δ s ′ for some s ′ s' s ′ : every action, at every state, leads with certainty to a single successor state.
Set-Cover problem: Given a finite set, the universe U U U , and a collection of subsets S 1 , … , S M ⊆ U S_1,\ldots,S_M \subseteq U S 1 , … , S M ⊆ U , the goal is to choose as few of the S j S_j S j as possible whose union is all of U U U (a cover); deciding whether a cover of a given size exists is a classical NP-hard problem [22 ] .
Construction 30 (Set-Cover gadget). Given a universe U U U , ∣ U ∣ = m |U| = m ∣ U ∣ = m , and sets S 1 , … , S M ⊆ U S_1,\ldots,S_M \subseteq U S 1 , … , S M ⊆ U each of size at most B B B such that ⋃ j = 1 M S j = U \bigcup_{j=1}^{M} S_j = U ⋃ j = 1 M S j = U . Build the deterministic, single-player entropy game E \mathcal{E} E with S max = ∅ S_{\max} = \varnothing S m a x = ∅ , S min = { e u : u ∈ U } ∪ { f j : j ≤ M } S_{\min} = \{e_u : u \in U\} \cup\{f_j : j \le M\} S m i n = { e u : u ∈ U } ∪ { f j : j ≤ M } , μ 0 \mu_0 μ 0 uniform on { e u } u ∈ U \{e_u\}_{u \in U} { e u } u ∈ U (so μ 0 ( e u ) = 1 / m \mu_0(e_u) = 1/m μ 0 ( e u ) = 1/ m ), and, for each u u u , one action per set S j ∋ u S_j \ni u S j ∋ u at e u e_u e u , deterministically leading to f j f_j f j ; each f j f_j f j is absorbing (its unique available action self-loops). Min (the only player) chooses, possibly with mixed strategies, an action at each e u e_u e u .
Figure 3 illustrates the construction.
Figure 3. The Set-Cover gadget (Construction 30) for the small instance U = { 1 , 2 , 3 , 4 } U = \{1,2,3,4\} U = { 1 , 2 , 3 , 4 } , S 1 = { 1 , 2 } S_1 = \{1,2\} S 1 = { 1 , 2 } , S 2 = { 2 , 3 } S_2 = \{2,3\} S 2 = { 2 , 3 } , S 3 = { 3 , 4 } S_3 = \{3,4\} S 3 = { 3 , 4 } . Left: one state e u e_u e u per universe element u ∈ U u \in U u ∈ U , together carrying the initial distribution μ 0 \mu_0 μ 0 , uniform over all m = ∣ U ∣ m = |U| m = ∣ U ∣ of them. Right: one state f j f_j f j per set S j S_j S j , each absorbing (self-loop). Min’s only choice, at each e u e_u e u , is which containing set’s state to move to: state e 2 e_2 e 2 , for instance, has two available actions, one to f 1 f_1 f 1 (since 2 ∈ S 1 2 \in S_1 2 ∈ S 1 ) and one to f 2 f_2 f 2 (since 2 ∈ S 2 2 \in S_2 2 ∈ S 2 ), while e 1 e_1 e 1 and e 4 e_4 e 4 each have only one, being covered by a single set. Concentrating many elements’ mass onto few f j f_j f j states lowers the entropy of μ 1 \mu_1 μ 1 ; spreading it across many raises it, exactly like choosing a small versus a wasteful cover.
Collapsing to one-step entropy
Since S max = ∅ S_{\max} = \varnothing S m a x = ∅ , Max has only the trivial strategy τ 0 \tau_0 τ 0 ; write Obj γ ⋆ ( π min ; μ ) : = Obj γ ⋆ ( π min , τ 0 ; μ ) \operatorname{Obj}_{\gamma}^{\star}(\pi_{\min};\mu) := \operatorname{Obj}_{\gamma}^{\star}(\pi_{\min},\tau_0;\mu) Obj γ ⋆ ( π m i n ; μ ) := Obj γ ⋆ ( π m i n , τ 0 ; μ ) for the resulting one-argument form. With π max \pi_{\max} π m a x trivial, the upper and lower values inf π min sup π max \inf_{\pi_{\min}}\sup_{\pi_{\max}} inf π m i n sup π m a x and sup π max inf π min \sup_{\pi_{\max}}\inf_{\pi_{\min}} sup π m a x inf π m i n both collapse to inf π min \inf_{\pi_{\min}} inf π m i n alone, so inf π min Obj γ ⋆ ( π min ; μ 0 ) \inf_{\pi_{\min}}\operatorname{Obj}_{\gamma}^{\star}(\pi_{\min};\mu_0) inf π m i n Obj γ ⋆ ( π m i n ; μ 0 ) is exactly the one-player value already in use: the polarity-free V ⋆ ( μ 0 ) V_{\star}(\mu_0) V ⋆ ( μ 0 ) , for every ⋆ ∈ { a v , s p , s t } \star\in\{\mathrm{av},\mathrm{sp},\mathrm{st}\} ⋆ ∈ { av , sp , st } . For av the upper and lower values coincide in this one-player case by the argument given just before Lemma 26 ; for sp and st they already coincide in general, with or without a lone decision-maker (Theorem 4). This is the value used in Lemmas 26 –28. By Lemma 11, restricting Min to pure profiles already suffices to compute this value, and shows the infimum is in fact attained, under every semantics, so the next lemma is stated directly for a pure profile.
Lemma 31. For E \mathcal{E} E as in Construction 30, any semantics ⋆ ∈ { a v , s p , s t } \star\in\{\mathrm{av},\mathrm{sp},\mathrm{st}\} ⋆ ∈ { av , sp , st } , and any pure profile a a a of Min (one action per state e u e_u e u ),
Obj γ ⋆ : = Obj γ ⋆ ( a ; μ 0 ) = ( 1 − γ ) log 2 m + γ ⋅ H ( μ 1 ) . \operatorname{Obj}_{\gamma}^{\star} := \operatorname{Obj}_{\gamma}^{\star}(a;\mu_0) = (1-\gamma)\log_{2}m + \gamma\cdot\mathbf{H}(\mu_1). Obj γ ⋆ := Obj γ ⋆ ( a ; μ 0 ) = ( 1 − γ ) log 2 m + γ ⋅ H ( μ 1 ) . independent of ⋆ \star ⋆ .
Proof. The entropy trajectory takes only two values. The entropy trajectory only ever takes two values: log 2 m \log_2 m log 2 m at time 0, then H ( μ 1 ) H(\mu_1) H ( μ 1 ) forever after, since every f j f_j f j is absorbing. So the infinite discounted sum defining Obj γ \operatorname{Obj}_\gamma Obj γ collapses to a weighted average of just these two fixed numbers, not a sum over a changing trajectory.
Computing H ( μ 0 ) H(\mu_0) H ( μ 0 ) . μ 0 \mu_0 μ 0 is uniform on the m m m states { e u } u ∈ U \{e_u\}_{u\in U} { e u } u ∈ U by construction, and this does not depend on Min’s strategy at all. Min only has actions available at the e u e_u e u states, and choosing one of them determines the transition into μ 1 \mu_1 μ 1 , not μ 0 \mu_0 μ 0 itself; μ 0 \mu_0 μ 0 is fixed data of the instance, not something any strategy can influence. So, by Proposition 16 (i), the uniform distribution on m m m states has entropy exactly log 2 m \log_2 m log 2 m , giving H ( μ 0 ) = log 2 m H(\mu_0) = \log_2 m H ( μ 0 ) = log 2 m regardless of the strategy played.
Computing H ( μ t ) H(\mu_t) H ( μ t ) for t ≥ 1 t \ge1 t ≥ 1 . Every state f j f_j f j is absorbing, meaning its only available action self-loops back to f j f_j f j with certainty (Construction 30). Once the first round of play moves all the mass from the e u e_u e u states onto the f j f_j f j states, producing μ 1 \mu_1 μ 1 , every state in the support of μ 1 \mu_1 μ 1 is absorbing, so no later round of play can move any mass anywhere else: μ t = μ 1 \mu_t = \mu_1 μ t = μ 1 for every t ≥ 1 t \ge1 t ≥ 1 . The entropy trajectory is therefore exactly log 2 m , H ( μ 1 ) , H ( μ 1 ) , … \log_2 m, H(\mu_1), H(\mu_1), \ldots log 2 m , H ( μ 1 ) , H ( μ 1 ) , … , constant from time 1 onward.
Why the trajectory is the same under all three semantics. Sampled-profile semantics differs from averaged-distribution semantics only in how a genuinely mixed action is aggregated, and sampled-transition semantics differs from both only in how a genuinely stochastic transition is realised. For the pure profile a a a used here, the corresponding degenerate statewise mixed action has x ( s ) = δ a ( s ) x(s) = \delta_{a(s)} x ( s ) = δ a ( s ) for every s ∈ S min s \in S_{\min} s ∈ S m i n , so its induced profile law (Equation (20) ) is x ^ ( a ′ ) = ∏ s ∈ S min δ a ( s ) ( a ′ ( s ) ) \hat{x}(a') = \prod_{s\in S_{\min}} \delta_{a(s)}(a'(s)) x ^ ( a ′ ) = ∏ s ∈ S m i n δ a ( s ) ( a ′ ( s )) , which equals 1 if a ′ = a a' = a a ′ = a and 0 otherwise; that is, x ^ = δ a \hat{x} = \delta_a x ^ = δ a , the point mass on a a a . Sampling a profile from x ^ \hat{x} x ^ , as sampled-profile semantics does, therefore always returns a a a itself: there is nothing to sample, exactly as used in Lemma 27 . Likewise, by Definition 29 every Δ ( s , a ) \Delta(s,a) Δ ( s , a ) is already a point mass, so sampled-transition semantics has no randomness to realise either. Hence μ 1 \mu_1 μ 1 , and every subsequent μ t = μ 1 \mu_t = \mu_1 μ t = μ 1 for t ≥ 1 t \ge1 t ≥ 1 , is identical regardless of ⋆ \star ⋆ .
Assembling Obj γ \operatorname{Obj}_\gamma Obj γ . Substituting this trajectory into the definition of Obj γ \operatorname{Obj}_\gamma Obj γ and splitting off the t = 0 t = 0 t = 0 term,
Obj γ = ( 1 − γ ) ∑ t ≥ 0 γ t H ( μ t ) = ( 1 − γ ) H ( μ 0 ) + ( 1 − γ ) H ( μ 1 ) ∑ t ≥ 1 γ t = ( 1 − γ ) log 2 m + γ H ( μ 1 ) , \begin{aligned}
\operatorname{Obj}_\gamma= (1-\gamma)\sum_{t\ge0}\gamma^t H(\mu_t) = (1-\gamma)H(\mu_0) + (1-\gamma)H(\mu_1)\sum_{t\ge1}\gamma^t \\
&= (1-\gamma)\log_2 m + \gamma H(\mu_1),
\end{aligned} Obj γ = ( 1 − γ ) t ≥ 0 ∑ γ t H ( μ t ) = ( 1 − γ ) H ( μ 0 ) + ( 1 − γ ) H ( μ 1 ) t ≥ 1 ∑ γ t = ( 1 − γ ) log 2 m + γ H ( μ 1 ) ,
using ∑ t ≥ 1 γ t = γ ∑ t ≥ 0 γ t = γ 1 − γ \sum_{t\ge1}\gamma^t = \gamma\sum_{t\ge0}\gamma^t = \frac{\gamma}{1-\gamma} ∑ t ≥ 1 γ t = γ ∑ t ≥ 0 γ t = 1 − γ γ , the standard geometric series shifted by one term.
Remark 32 (Reducing Obj γ \operatorname{Obj}_\gamma Obj γ -hardness to H ( μ 1 ) H(\mu_1) H ( μ 1 ) -hardness). Since γ > 0 \gamma> 0 γ > 0 and log 2 m \log_2 m log 2 m are fixed, Obj γ = ( 1 − γ ) log 2 m + γ H ( μ 1 ) \operatorname{Obj}_\gamma= (1-\gamma)\log_2 m + \gamma H(\mu_1) Obj γ = ( 1 − γ ) log 2 m + γ H ( μ 1 ) is an increasing affine function of H ( μ 1 ) H(\mu_1) H ( μ 1 ) . A routing minimising H ( μ 1 ) H(\mu_1) H ( μ 1 ) also minimises Obj γ \operatorname{Obj}_\gamma Obj γ , and an ε \varepsilon ε -gap between two values of H ( μ 1 ) H(\mu_1) H ( μ 1 ) becomes a γ ε \gamma\varepsilon γ ε -gap between the corresponding values of Obj γ \operatorname{Obj}_\gamma Obj γ . We therefore analyse min H ( μ 1 ) \min H(\mu_1) min H ( μ 1 ) once and read off Obj γ \operatorname{Obj}_\gamma Obj γ at the end.
Entropy as a function of the routing
The next lemma shows how H ( μ 1 ) H(\mu_1) H ( μ 1 ) depends on the routing, in terms of how many sets are used and how evenly the universe’s elements are spread across them.
Lemma 33 (Entropy of a routing). Write μ 1 ( f j ) = c j / m \mu_1(f_j) = c_j/m μ 1 ( f j ) = c j / m for the (integer) load c j ∈ { 0 , … , B } c_j \in\{0,\ldots,B\} c j ∈ { 0 , … , B } routed to S j S_j S j (so ∑ j c j = m \sum_j c_j = m ∑ j c j = m ), and s = ∣ { j : c j > 0 } ∣ s = |\{j : c_j > 0\}| s = ∣ { j : c j > 0 } ∣ for the number of sets used, which is necessarily a set cover of U U U . Then, for this integer s s s , $E^*(s) - \frac{B\log_2 B}{m} \le
H ( μ 1 ) ≤ log 2 s \mathbf{H}(\mu_1) \le\log_2 s H ( μ 1 ) ≤ log 2 s where E ∗ ( s ) : = log 2 m − ( m − s ) B log 2 B m ( B − 1 ) E^*(s) := \log_2 m - \frac{(m-s)B\log_2 B}{m(B-1)} E ∗ ( s ) := log 2 m − m ( B − 1 ) ( m − s ) B l o g 2 B and E ∗ E^* E ∗ is strictly increasing in s s s . In general, H ( μ 1 ) ≥ log 2 ( m / B ) \mathbf{H}(\mu_1) \ge\log_2(m/B) H ( μ 1 ) ≥ log 2 ( m / B ) with equality iff B B B divides m m m and s = m / B s=m/B s = m / B , then H ( μ 1 ) = E ∗ ( m / B ) = log 2 ( m / B ) \mathbf{H}(\mu_1)=E^*(m/B)=\log_2(m/B) H ( μ 1 ) = E ∗ ( m / B ) = log 2 ( m / B ) .
Proof. Intuition. The load cap B B B stops any single f j f_j f j from absorbing more than a B / m B/m B / m share of the mass, so using fewer sets forces more mass onto each of the remaining ones; entropy is minimised exactly when this forcing is as extreme as possible, every used set loaded to its cap B B B (uniform on m / B m/B m / B states) when that many sets suffice, and it can only rise if more sets than necessary are used, since that spreads the same total mass thinner.
H ( μ 1 ) = − ∑ j : c j > 0 c j m log 2 c j m = ∑ j : c j > 0 c j m [ log 2 m − log 2 c j ] = log 2 m − 1 m ∑ j : c j > 0 c j log 2 c j \mathbf{H}(\mu_1)=-\sum_{j:c_j>0}\frac{c_j}{m}\log_2\frac{c_j}{m}=\sum_{j:c_j>0}\frac{c_j}{m}\left[\log_2m-\log_2c_j\right]=\log_2m-\frac{1}{m}\sum_{j:c_j>0}c_j\log_2c_j H ( μ 1 ) = − j : c j > 0 ∑ m c j log 2 m c j = j : c j > 0 ∑ m c j [ log 2 m − log 2 c j ] = log 2 m − m 1 j : c j > 0 ∑ c j log 2 c j
Minimizing H ( μ 1 ) \mathbf{H}(\mu_1) H ( μ 1 ) at fixed support size s s s is equivalent to maximising F ( c ) : = ∑ j c j log 2 c j F(c):=\sum_j c_j\log_2c_j F ( c ) := ∑ j c j log 2 c j over the polytope P s = { c ∈ [ 1 , B ] s : ∑ j c j = m } P_s=\{c\in[1,B]^s:\sum_jc_j=m\} P s = { c ∈ [ 1 , B ] s : ∑ j c j = m } , the possible loads when exactly s s s sets are used (we first relax integrality; the correction for integer loads is bounded below).
Upper bound. By concavity of H \mathbf{H} H restricted to distributions supported on a fixed set of size s s s (Proposition 16 (i)), H ( μ 1 ) ≤ log 2 s \mathbf{H}(\mu_1)\le\log_2s H ( μ 1 ) ≤ log 2 s always, with equality iff μ 1 \mu_1 μ 1 is uniform on its support.
Basic lower bound and the equality case. F F F is a sum of the convex function x ↦ x log 2 x x\mapsto x\log_2x x ↦ x log 2 x applied coordinatewise, and every c j ≤ B c_j\le B c j ≤ B caps every atom’s mass at ∥ μ 1 ∥ ∞ ≤ B / m \|\mu_1\|_\infty\le B/m ∥ μ 1 ∥ ∞ ≤ B / m ; the elementary bound H ( ν ) ≥ − log 2 ∥ ν ∥ ∞ \mathbf{H}(\nu)\ge-\log_2\|\nu\|_\infty H ( ν ) ≥ − log 2 ∥ ν ∥ ∞ (equality iff ν \nu ν is uniform on its support) then gives H ( μ 1 ) ≥ log 2 ( m / B ) \mathbf{H}(\mu_1)\ge\log_2(m/B) H ( μ 1 ) ≥ log 2 ( m / B ) for every routing , regardless of s s s . Equality forces ∥ μ 1 ∥ ∞ = B / m \|\mu_1\|_\infty=B/m ∥ μ 1 ∥ ∞ = B / m exactly, i.e. every used set loaded to exactly B B B ; since loads sum to m m m , this is only possible when s = m / B s=m/B s = m / B (if B B B divides m m m ; otherwise the bound is strict), a balanced exact cover .
Refined bound via the vertex structure of P s P_s P s . Fix s s s and consider max c ∈ P s F ( c ) \max_{c\in P_s}F(c) max c ∈ P s F ( c ) . Since F F F is convex and P s P_s P s is a box intersected with one hyperplane, the maximum is attained at a vertex, and every vertex of such a polytope has at most one coordinate strictly inside ( 1 , B ) (1,B) ( 1 , B ) , the rest sitting at a bound; otherwise two interior coordinates could be perturbed by ± ε \pm\varepsilon ± ε while preserving feasibility, contradicting extremality. Writing q = ⌊ m − s B − 1 ⌋ q=\left\lfloor\frac{m-s}{B-1}\right\rfloor q = ⌊ B − 1 m − s ⌋ for the number of coordinates pinned to B B B (each set starts from a baseline load of 1 1 1 and absorbs excess mass m − s m-s m − s in units of B − 1 B-1 B − 1 until exhausted), ρ = 1 + ( m − s ) − q ( B − 1 ) ∈ [ 1 , B ] \rho=1+(m-s)-q(B-1)\in[1,B] ρ = 1 + ( m − s ) − q ( B − 1 ) ∈ [ 1 , B ] for the one fractional coordinate, and the remaining s − 1 − q s-1-q s − 1 − q coordinates at 1 1 1 , the true (integer-load) optimum satisfies
F ∗ ( s ) = q B log 2 B + ρ log 2 ρ , H ( μ 1 ) ≥ log 2 m − 1 m F ∗ ( s ) , F^*(s)=qB\log_2B+\rho\log_2\rho,\qquad\mathbf{H}(\mu_1)\ge\log_2m-\frac{1}{m}F^*(s), F ∗ ( s ) = q B log 2 B + ρ log 2 ρ , H ( μ 1 ) ≥ log 2 m − m 1 F ∗ ( s ) ,
exactly, since the integer feasible region is a subset of the relaxed polytope P s P_s P s .
Since ρ ∈ [ 1 , B ] \rho\in[1,B] ρ ∈ [ 1 , B ] , ρ log 2 ρ ≤ B log 2 B \rho\log_2\rho\le B\log_2B ρ log 2 ρ ≤ B log 2 B . Then,
F ∗ ( s ) ≤ q B log 2 B + B log 2 B = ( q + 1 ) B log 2 B ≤ ( m − s B − 1 + 1 ) B log 2 B F^*(s)\le qB\log_2B+B\log_2B=(q+1)B\log_2B\le\left(\frac{m-s}{B-1}+1\right)B\log_2B F ∗ ( s ) ≤ q B log 2 B + B log 2 B = ( q + 1 ) B log 2 B ≤ ( B − 1 m − s + 1 ) B log 2 B
Hence,
H ( μ 1 ) ≥ log 2 m − 1 m ( ( m − s B − 1 + 1 ) B log 2 B ) = E ∗ ( s ) − B log 2 B m \mathbf{H}(\mu_1)\ge\log_2m-\frac{1}{m}\left(\left(\frac{m-s}{B-1}+1\right)B\log_2B\right)=E^*(s)-\frac{B\log_2B}{m} H ( μ 1 ) ≥ log 2 m − m 1 ( ( B − 1 m − s + 1 ) B log 2 B ) = E ∗ ( s ) − m B log 2 B
where E ∗ ( s ) = log 2 m − ( m − s ) B log 2 B m ( B − 1 ) E^*(s)=\log_2m-\frac{(m-s)B\log_2B}{m(B-1)} E ∗ ( s ) = log 2 m − m ( B − 1 ) ( m − s ) B l o g 2 B .
Hence, E ∗ E^* E ∗ is increasing in s s s , matching the monotonicity claimed in the lemma. (For B = 2 B=2 B = 2 the correction vanishes identically, since B − 1 = 1 B-1=1 B − 1 = 1 divides every integer exactly; the slack term is only needed for B ≥ 3 B\ge3 B ≥ 3 .)
Consistency check. At s = m / B s=m/B s = m / B , m − s = m ( B − 1 ) / B m-s=m(B-1)/B m − s = m ( B − 1 ) / B , so
( m − s ) B log 2 B m ( B − 1 ) = log 2 B \frac{(m-s)B\log_2 B}{m(B-1)}=\log_2 B m ( B − 1 ) ( m − s ) B log 2 B = log 2 B
exactly and E ∗ ( m / B ) = log 2 m − log 2 B = log 2 ( m / B ) E^*(m/B)=\log_2 m-\log_2 B=\log_2(m/B) E ∗ ( m / B ) = log 2 m − log 2 B = log 2 ( m / B ) , recovering the basic lower bound above exactly, with no rounding slack, as it must, since s = m / B s=m/B s = m / B with all loads at B B B is already an integer vertex (q = s q=s q = s ).
Worked example. Take m = 6 m=6 m = 6 , B = 2 B=2 B = 2 . A perfect-matching cover { S 1 , S 2 , S 3 } = { { 1 , 2 } , { 3 , 4 } , { 5 , 6 } } \{S_1,S_2,S_3\}=\{\{1,2\},\{3,4\},\{5,6\}\} { S 1 , S 2 , S 3 } = {{ 1 , 2 } , { 3 , 4 } , { 5 , 6 }} achieves s = m / B = 3 s=m/B=3 s = m / B = 3 with every load exactly 2, so μ 1 \mu_1 μ 1 is uniform on 3 atoms and H ( μ 1 ) = log 2 3 ≈ 1.585 H(\mu_1)=\log_2 3\approx1.585 H ( μ 1 ) = log 2 3 ≈ 1.585 , matching E ∗ ( 3 ) = log 2 6 − log 2 2 = log 2 3 E^*(3)=\log_2 6-\log_2 2=\log_2 3 E ∗ ( 3 ) = log 2 6 − log 2 2 = log 2 3 exactly. If instead the available sets force s = 4 s=4 s = 4 sets to be used, the entropy-minimising loads are ( 2 , 2 , 1 , 1 ) (2,2,1,1) ( 2 , 2 , 1 , 1 ) (the vertex above, with q = ⌊ ( 6 − 4 ) / 1 ⌋ = 2 q=\lfloor(6-4)/1\rfloor=2 q = ⌊( 6 − 4 ) /1 ⌋ = 2 sets at B = 2 B=2 B = 2 and one fractional/remaining coordinate each at 1), giving
H ( μ 1 ) = − 2 ⋅ 2 6 log 2 2 6 − 2 ⋅ 1 6 log 2 1 6 ≈ 1.918 , H(\mu_1)=-2\cdot\frac{2}{6}\log_2\frac{2}{6}-2\cdot\frac{1}{6}\log_2\frac{1}{6}\approx1.918, H ( μ 1 ) = − 2 ⋅ 6 2 log 2 6 2 − 2 ⋅ 6 1 log 2 6 1 ≈ 1.918 ,
matching E ∗ ( 4 ) = log 2 6 − ( 6 − 4 ) ⋅ 2 log 2 2 6 ⋅ 1 = log 2 6 − 2 3 ≈ 1.918 E^*(4)=\log_2 6-\frac{(6-4)\cdot2\log_2 2}{6\cdot1}=\log_2 6-\frac{2}{3}\approx1.918 E ∗ ( 4 ) = log 2 6 − 6 ⋅ 1 ( 6 − 4 ) ⋅ 2 l o g 2 2 = log 2 6 − 3 2 ≈ 1.918 ; the two match exactly here, rather than merely up to the B log 2 B / m B\log_2 B/m B log 2 B / m slack, because B = 2 B=2 B = 2 , for which that correction vanishes identically, as noted above. This need not happen for B ≥ 3 B\ge3 B ≥ 3 : the general conclusion is not that H ( μ 1 ) H(\mu_1) H ( μ 1 ) equals E ∗ ( s ) E^*(s) E ∗ ( s ) , but that both the achieved minimum entropy and its lower bound E ∗ ( s ) − B log 2 B / m E^*(s)-B\log_2 B/m E ∗ ( s ) − B log 2 B / m increase with s s s , since E ∗ E^* E ∗ is increasing in s s s and the subtracted term B log 2 B / m B\log_2 B/m B log 2 B / m does not depend on s s s . Thus being forced from a minimum 3-set cover up to a 4-set cover provably costs at least E ∗ ( 4 ) − E ∗ ( 3 ) ≈ 0.333 E^*(4)-E^*(3)\approx0.333 E ∗ ( 4 ) − E ∗ ( 3 ) ≈ 0.333 bits of entropy in this lower bound, for every B B B , exactly the mechanism Theorem 13 exploits: an inapproximability gap in cover size becomes, via this lemma, a fixed additive gap in entropy. ◀
A perfect-completeness Set-Cover gap
Generic approximation hardness for Set-Cover would not suffice for the theorem below: the entropy calculation needs a yes-case that partitions the universe into maximum-size sets, not merely an approximately-optimal cover.
Lemma 34 (A bounded-Set-Cover gap with perfect completeness). There is a fixed rational c > 0 c>0 c > 0 with the following property. Consider an explicitly given set system consisting of a nonempty universe U U U of size m m m , with 3 ∣ m 3\mid m 3 ∣ m , and sets S 1 , … , S M ⊆ U S_1,\ldots,S_M\subseteq U S 1 , … , S M ⊆ U , each of size at most B = 3 B=3 B = 3 , with ⋃ j S j = U \bigcup_j S_j=U ⋃ j S j = U (so some cover of U U U exists). It is NP-hard to distinguish:
(i) the case where U U U can be partitioned into m / 3 m/3 m /3 of the given sets, each of size exactly 3;
(ii) the case where every cover of U U U by the given sets uses at least ( 1 + c ) m / 3 (1+c)m/3 ( 1 + c ) m /3 of them.
Proof. Use the perfect-completeness gap for bounded-occurrence three-dimensional matching [28 Theorem 4.4] . In the form needed here, there is a fixed rational η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) , independent of the instance, such that it is NP-hard to distinguish a triple system with a perfect matching (a partition of the universe into disjoint triples) from one whose every disjoint matching covers at most a ( 1 − η ) (1-\eta) ( 1 − η ) fraction of the universe. The cited hard-gap statement counts elements occurring exactly once in a chosen subfamily; a disjoint matching is a special case, so its soundness implies this version. Taking three disjoint copies if necessary makes the universe size m = 3 k m=3k m = 3 k without changing either ratio; write k k k for this quotient (the number of triples in a perfect matching, when one exists).
Regard the triples as available sets and add every singleton. This ensures that the union is the whole universe and preserves the bound B = 3 B=3 B = 3 . A perfect matching supplies the yes-case partition, and no cover can use fewer than k k k sets because each set has size at most three.
For soundness, consider any cover consisting of s 3 s_3 s 3 triples and s 1 s_1 s 1 singletons, with s = s 3 + s 1 s=s_3+s_1 s = s 3 + s 1 . If U 3 U_3 U 3 is the union of its triples, every element of the universe outside U 3 U_3 U 3 must be covered by some singleton (triples and singletons are the only available sets), so at most s 1 s_1 s 1 elements lie outside U 3 U_3 U 3 : ∣ U 3 ∣ ≥ m − s 1 |U_3| \ge m - s_1 ∣ U 3 ∣ ≥ m − s 1 . Their excess incidence count is R : = 3 s 3 − ∣ U 3 ∣ R := 3s_3 - |U_3| R := 3 s 3 − ∣ U 3 ∣ (the total multiplicity, 3 s 3 3s_3 3 s 3 , of elements across the s 3 s_3 s 3 triples, minus the number of distinct elements they cover); since ∣ U 3 ∣ ≥ m − s 1 |U_3| \ge m - s_1 ∣ U 3 ∣ ≥ m − s 1 , R ≤ 3 s 3 + s 1 − m R \le3s_3 + s_1 - m R ≤ 3 s 3 + s 1 − m .
Repeatedly delete a triple that intersects another remaining triple, until the remaining triples are pairwise disjoint. Each deletion decreases R R R , recomputed on the remaining triples, by at least one: the deleted triple shares at least one element with another remaining triple, so at most two of its three elements are covered by no other remaining triple, and removing it can only decrease the union of the remaining triples by that many (∣ U 3 ′ ∣ ≥ ∣ U 3 ∣ − 2 |U'_3| \ge|U_3| - 2 ∣ U 3 ′ ∣ ≥ ∣ U 3 ∣ − 2 ), while the triple count drops by exactly one (s 3 ′ = s 3 − 1 s'_3 = s_3 - 1 s 3 ′ = s 3 − 1 ); a short computation gives R ′ = 3 s 3 ′ − ∣ U 3 ′ ∣ ≤ 3 s 3 − 3 − ∣ U 3 ∣ + 2 = R − 1 R' = 3s'_3 - |U'_3| \le3s_3 - 3 - |U_3| + 2 = R - 1 R ′ = 3 s 3 ′ − ∣ U 3 ′ ∣ ≤ 3 s 3 − 3 − ∣ U 3 ∣ + 2 = R − 1 . Since R ≥ 0 R \ge0 R ≥ 0 throughout and strictly decreases at each step, at most R R R deletions occur before the remaining triples are disjoint, and at least s 3 − R s_3 - R s 3 − R triples remain. Using R ≤ 3 s 3 + s 1 − m R \le3s_3 + s_1 - m R ≤ 3 s 3 + s 1 − m ,
s 3 − R ≥ m − 2 s 3 − s 1 ≥ m − 2 ( s 3 + s 1 ) = m − 2 s = 3 k − 2 s . s_3 - R \ge m - 2s_3 - s_1 \ge m - 2(s_3 + s_1) = m - 2s = 3k - 2s. s 3 − R ≥ m − 2 s 3 − s 1 ≥ m − 2 ( s 3 + s 1 ) = m − 2 s = 3 k − 2 s .
The remaining triples form a disjoint matching drawn from the original triple system, of size at least 3 k − 2 s 3k - 2s 3 k − 2 s . In a no-instance, every such matching has size at most ( 1 − η ) k (1 - \eta)k ( 1 − η ) k (a matching of j j j triples covers 3 j 3j 3 j elements, and 3 j ≤ ( 1 − η ) ⋅ 3 k 3j \le(1 - \eta) \cdot3k 3 j ≤ ( 1 − η ) ⋅ 3 k gives j ≤ ( 1 − η ) k j \le(1 - \eta)k j ≤ ( 1 − η ) k ). Therefore 3 k − 2 s ≤ ( 1 − η ) k 3k - 2s \le(1 - \eta)k 3 k − 2 s ≤ ( 1 − η ) k , i.e. s ≥ ( 1 + η / 2 ) k s \ge(1 + \eta/2)k s ≥ ( 1 + η /2 ) k . Setting c : = η / 2 > 0 c := \eta/2 > 0 c := η /2 > 0 , a fixed rational independent of the instance, gives s ≥ ( 1 + c ) k = ( 1 + c ) m / 3 s \ge(1 + c)k = (1 + c)m/3 s ≥ ( 1 + c ) k = ( 1 + c ) m /3 , proving soundness. The transformation, including the added singletons, is computable in time polynomial in m m m , proving the lemma. ◀
The hardness reduction
By the same reasoning used to prove Lemma 31, but without restricting to a pure profile: since μ 0 \mu_0 μ 0 is fixed data of the instance, not a choice made by any strategy, and since every state in the support of the resulting μ 1 \mu_1 μ 1 is absorbing, the entropy trajectory under any π min ∈ Π min ⋆ \pi_{\min} \in\Pi^\star_{\min} π m i n ∈ Π m i n ⋆ , of any memory or randomization, still takes only two values, log 2 m \log_2 m log 2 m at time 0 and H ( μ 1 ) \mathbf{H}(\mu_1) H ( μ 1 ) forever after, so
Obj γ ⋆ ( π min ; μ 0 ) = ( 1 − γ ) log 2 m + γ E [ H ( μ 1 ) ] . \operatorname{Obj}^{\star}_{\gamma}(\pi_{\min};\mu_0) = (1-\gamma)\log_2 m + \gamma\mathbb{E}[\mathbf{H}(\mu_1)]. Obj γ ⋆ ( π m i n ; μ 0 ) = ( 1 − γ ) log 2 m + γ E [ H ( μ 1 )] .
In particular the objective depends on π min \pi_{\min} π m i n only through this one expectation, not on anything else the strategy does.
Moreover μ 0 \mu_0 μ 0 is fixed before play begins, so there is no history prior to Min’s first move: a history-dependent strategy’s action at time 0 is just some (possibly randomized) choice of action at each e u e_u e u , exactly as available to a memoryless strategy, and nothing said above used purity or absence of memory to reach this point. By Lemma 11, applied at μ 0 \mu_0 μ 0 , this one-shot minimization over randomized choices is attained, under every semantics, by a pure profile a a a , i.e. a fixed assignment of one action to each e u e_u e u with no time or history dependence at all, which is precisely a memoryless strategy. Hence
inf π min ∈ Π min ⋆ Obj γ ⋆ ( π min ; μ 0 ) = min a Obj γ ⋆ ( a ; μ 0 ) . \inf_{\pi_{\min}\in\Pi^\star_{\min}} \operatorname{Obj}^{\star}_{\gamma}(\pi_{\min};\mu_0) = \min_a \operatorname{Obj}^{\star}_{\gamma}(a;\mu_0). π m i n ∈ Π m i n ⋆ inf Obj γ ⋆ ( π m i n ; μ 0 ) = a min Obj γ ⋆ ( a ; μ 0 ) .
the minimum on the right taken over pure profiles a a a , which is exactly the quantity Lemma 31 computes. The hardness result below, being a statement about this infimum, therefore already applies with Min restricted to memoryless strategies.
Theorem 13 (One-player hardness). For every rational discount γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) there exists a rational constant ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 , depending only on γ \gamma γ , such that, for every ⋆ ∈ { a v , s p , s t } \star\in\{\mathrm{av},\mathrm{sp},\mathrm{st}\} ⋆ ∈ { av , sp , st } , GapVal ⋆ , ε 0 \operatorname{GapVal}_{\star,\varepsilon_0} GapVal ⋆ , ε 0 is NP-hard, already for deterministic one-player games. In the games constructed, pure memoryless strategies are optimal (Lemma 11), so the hardness does not rely on restricting the strategy class.
Proof. Intuition. Lemma 33 says that using more sets than the minimum cover costs entropy: spreading the same mass over extra sets can only raise H ( μ 1 ) \mathbf{H}(\mu_1) H ( μ 1 ) . So if a Set-Cover instance is hard to approximate, meaning its minimum cover jumps by a constant factor when no small cover exists, that same jump forces a gap onto min H ( μ 1 ) \min\mathbf{H}(\mu_1) min H ( μ 1 ) . E.g. on 6 elements with B = 2 B = 2 B = 2 : a minimum 3-set cover gives H ≈ 1.585 \mathbf{H} \approx1.585 H ≈ 1.585 , while being forced to 4 sets costs ≈ 0.333 \approx0.333 ≈ 0.333 more bits.
Reducing to deterministic routings. For a randomised strategy, the minimum cover strategies is attained at a deterministic (pure) routing under every semantics, by Lemma 11; we therefore analyse deterministic routings, each of which sends every u u u to exactly one S j ∋ u S_j \ni u S j ∋ u . By Lemma 31 it suffices to prove both claims for H ( μ 1 ) \mathbf{H}(\mu_1) H ( μ 1 ) in Construction 30; the discounted case then follows by the affine substitution noted in the main text (which maps a fixed additive gap ε 0 \varepsilon_0 ε 0 in H ( μ 1 ) \mathbf{H}(\mu_1) H ( μ 1 ) to a fixed additive gap γ ε 0 \gamma\varepsilon_0 γ ε 0 in Obj γ \operatorname{Obj}_\gamma Obj γ , and an exact threshold to an exact threshold). Construction 30 is computable in time polynomial in m + M m + M m + M .
Turning the Set-Cover gap into an entropy gap. Fix B = 3 B = 3 B = 3 . The theorem only requires some fixed set-size bound with a fixed gap, not a bound holding for every B B B , so it suffices to exhibit one, and B = 3 B = 3 B = 3 is what Lemma 34 supplies: a fixed rational c > 0 c > 0 c > 0 such that it is NP-hard to distinguish a set system on a universe of size m m m (all sets of size at most 3) that admits an exact partition into m / 3 m/3 m /3 triples from one where every set cover needs at least ( 1 + c ) m / 3 (1+c)m/3 ( 1 + c ) m /3 sets. This constant c c c comes from Lemma 34 ’s reduction from bounded-occurrence three-dimensional matching [28 Theorem 4.4] , not from the entropy game; all we use here is that it is fixed, positive, and independent of m m m .
In the first case, routing every u u u through a set of a minimum cover of size m / B m/B m / B gives every load exactly B B B , so min H ( μ 1 ) = log 2 ( m / B ) \min\mathbf{H}(\mu_1) = \log_2(m/B) min H ( μ 1 ) = log 2 ( m / B ) by the equality case of Lemma 33 . In the second, every routing’s support size s s s satisfies s ≥ ( 1 + c ) m / B s \ge(1+c)m/B s ≥ ( 1 + c ) m / B (any routing’s support is a cover), so by Lemma 33 ,
H ( μ 1 ) ≥ E ∗ ( ( 1 + c ) m / B ) − B log 2 B m = log 2 ( m / B ) + c log 2 B B − 1 − B log 2 B m . \mathbf{H}(\mu_1) \ge\mathbb{E}^{*}\bigl((1+c)m/B\bigr) - \frac{B\log_2 B}{m}
= \log_2(m/B) + \frac{c\log_2 B}{B-1} - \frac{B\log_2 B}{m}. H ( μ 1 ) ≥ E ∗ ( ( 1 + c ) m / B ) − m B log 2 B = log 2 ( m / B ) + B − 1 c log 2 B − m B log 2 B .
For m m m larger than a constant depending only on B , c B,c B , c (m ≥ 2 B ( B − 1 ) c m \ge\frac{2B(B-1)}{c} m ≥ c 2 B ( B − 1 ) ), the correction term O ( log 2 B / m ) O(\log_2 B/m) O ( log 2 B / m ) term is below half of the fixed constant c log 2 B B − 1 \frac{c\log_2 B}{B-1} B − 1 c l o g 2 B , giving H ( μ 1 ) ≥ log 2 ( m / B ) + c log 2 B 4 ( B − 1 ) + ε 0 \mathbf{H}(\mu_1) \ge\log_2(m/B) + \frac{c\log_2 B}{4(B-1)} + \varepsilon_0 H ( μ 1 ) ≥ log 2 ( m / B ) + 4 ( B − 1 ) c l o g 2 B + ε 0 for ε 0 : = c log 2 B 4 ( B − 1 ) > 0 \varepsilon_0 := \frac{c\log_2 B}{4(B-1)} > 0 ε 0 := 4 ( B − 1 ) c l o g 2 B > 0 , a constant independent of the instance. Taking v = log 2 ( m / B ) + c log 2 B 4 ( B − 1 ) v = \log_2(m/B) + \frac{c\log_2 B}{4(B-1)} v = log 2 ( m / B ) + 4 ( B − 1 ) c l o g 2 B , this is exactly a valid ε 0 \varepsilon_0 ε 0 -gap promise separating the two cases: min H ( μ 1 ) = log 2 ( m / B ) ≤ v − ε 0 \min\mathbf{H}(\mu_1) = \log_2(m/B) \le v - \varepsilon_0 min H ( μ 1 ) = log 2 ( m / B ) ≤ v − ε 0 in the first case and min H ( μ 1 ) ≥ v + ε 0 \min\mathbf{H}(\mu_1) \ge v + \varepsilon_0 min H ( μ 1 ) ≥ v + ε 0 in the second. This gives an NP-hard many-one reduction for min H ( μ 1 ) \min\mathbf{H}(\mu_1) min H ( μ 1 ) , at threshold v = log 2 ( m / B ) + c log 2 B 4 ( B − 1 ) v = \log_2(m/B) + \frac{c\log_2 B}{4(B-1)} v = log 2 ( m / B ) + 4 ( B − 1 ) c l o g 2 B and gap ε 0 = c log 2 B 4 ( B − 1 ) \varepsilon_0 = \frac{c\log_2 B}{4(B-1)} ε 0 = 4 ( B − 1 ) c l o g 2 B , and it transfers to Obj γ \operatorname{Obj}_\gamma Obj γ by the substitution of Remark 32.
Extending to Obj γ \operatorname{Obj}_\gamma Obj γ : For a fixed c > 0 c > 0 c > 0 , taking v ∗ = ( 1 − γ ) log 2 m + γ ⋅ ( log 2 ( m / B ) + c log 2 B 4 ( B − 1 ) ) v^* = (1-\gamma)\log_2 m + \gamma\cdot\left(\log_2(m/B) + \frac{c\log_2 B}{4(B-1)}\right) v ∗ = ( 1 − γ ) log 2 m + γ ⋅ ( log 2 ( m / B ) + 4 ( B − 1 ) c l o g 2 B ) and ε ∗ = γ ⋅ c log 2 B 4 ( B − 1 ) \varepsilon^* = \gamma\cdot\frac{c\log_2 B}{4(B-1)} ε ∗ = γ ⋅ 4 ( B − 1 ) c l o g 2 B separates the two cases for Obj γ \operatorname{Obj}_\gamma Obj γ .
Rational threshold and gap. The problem GapVal ⋆ , ε 0 \operatorname{GapVal}_{\star,\varepsilon_0} GapVal ⋆ , ε 0 asks for a rational threshold and a rational gap, whereas v ∗ v^* v ∗ and ε ∗ \varepsilon^* ε ∗ are irrational in general (they involve log 2 B \log_2 B log 2 B and log 2 ( m / B ) \log_2(m/B) log 2 ( m / B ) ).
Fix a rational ε 1 \varepsilon_1 ε 1 with 0 < ε 1 < ε ∗ / 2 0 < \varepsilon_1 < \varepsilon^*/2 0 < ε 1 < ε ∗ /2 ; it depends only on γ \gamma γ and c c c , not on the instance. Using the logarithm series from the proof of Lemma 6, compute in time polynomial in m m m a rational v ′ v' v ′ with ∣ v ′ − v ∗ ∣ < ε ∗ / 2 \lvert v' - v^* \rvert< \varepsilon^*/2 ∣ v ′ − v ∗ ∣ < ε ∗ /2 . In the first case the value is at most v ∗ − ε ∗ < v ′ − ε 1 v^* - \varepsilon^* < v' - \varepsilon_1 v ∗ − ε ∗ < v ′ − ε 1 , and in the second it is at least v ∗ + ε ∗ > v ′ + ε 1 v^* + \varepsilon^* > v' + \varepsilon_1 v ∗ + ε ∗ > v ′ + ε 1 . Hence ( v ′ , ε 1 ) (v',\varepsilon_1) ( v ′ , ε 1 ) is a rational threshold and gap that separate the two cases, and GapVal ⋆ , ε 1 \operatorname{GapVal}_{\star,\varepsilon_1} GapVal ⋆ , ε 1 is NP-hard for every ⋆ \star ⋆ . Instances with m m m below the constant 2 B ( B − 1 ) / c 2B(B-1)/c 2 B ( B − 1 ) / c are solved by brute force, so they do not affect hardness.
Exact NP-hardness (Theorem 15). The exact problem needs a rational threshold, whereas log 2 ( m / B ) \log_2(m/B) log 2 ( m / B ) is generally irrational, so we do not use it as the threshold. Let M : = log 2 ( m / B ) + ε 0 M := \log_2(m/B) + \varepsilon_0 M := log 2 ( m / B ) + ε 0 be the midpoint of the promise gap above, so that yes-instances have min H ( μ 1 ) ≤ M − ε 0 \min\mathbf{H}(\mu_1) \le M - \varepsilon_0 min H ( μ 1 ) ≤ M − ε 0 and no-instances have min H ( μ 1 ) ≥ M + ε 0 \min\mathbf{H}(\mu_1) \ge M + \varepsilon_0 min H ( μ 1 ) ≥ M + ε 0 . Using the rapidly convergent series for the logarithm from the proof of Lemma 6, compute in time polynomial in m m m a rational v ′ v' v ′ with ∣ v ′ − M ∣ < ε 0 / 2 \lvert v' - M \rvert< \varepsilon_0/2 ∣ v ′ − M ∣ < ε 0 /2 (here ε 0 \varepsilon_0 ε 0 is a fixed rational constant, or any rational constant below the fixed gap). Then M − ε 0 < v ′ < M + ε 0 M - \varepsilon_0 < v' < M + \varepsilon_0 M − ε 0 < v ′ < M + ε 0 , hence min H ( μ 1 ) ≤ v ′ \min\mathbf{H}(\mu_1) \le v' min H ( μ 1 ) ≤ v ′ on yes-instances and min H ( μ 1 ) > v ′ \min\mathbf{H}(\mu_1) > v' min H ( μ 1 ) > v ′ on no-instances. Thus the exact comparison “min H ( μ 1 ) ≤ v ′ \min\mathbf{H}(\mu_1) \le v' min H ( μ 1 ) ≤ v ′ ” decides the same NP-hard cover-size question, with the rational threshold v ′ v' v ′ . For Obj γ \operatorname{Obj}_\gamma Obj γ the same argument is applied to the midpoint ( 1 − γ ) log 2 m + γ M (1-\gamma)\log_2 m + \gamma M ( 1 − γ ) log 2 m + γ M with tolerance γ ε 0 / 2 \gamma\varepsilon_0/2 γ ε 0 /2 . The gap statement is thus the stronger of the two, and exact hardness follows from it. ◀
Comparing the semantics
This appendix proves Theorem 5 and gives the examples showing that V s p V_{\mathrm{sp}} V sp and V s t V_{\mathrm{st}} V st are incomparable. Below, “pure” strategy has the meaning of Section 2: it selects one pure profile.
Remark 35 (Deterministic transitions). If every enabled transition is deterministic, then for every profile pair ( a , b ) (a,b) ( a , b ) the realisation ξ \xi ξ is determined, so Ψ ξ = Φ a , b \Psi_\xi= \Phi_{a,b} Ψ ξ = Φ a , b and the sampled-profile and sampled-transition updates coincide for every profile law; hence V s p = V s t V_{\mathrm{sp}} = V_{\mathrm{st}} V sp = V st in such arenas. Deterministic transitions alone do not give equality with the averaged values.
Remark 36 (Pure strategies). For the same pair of pure Borel distribution-history strategies, the averaged-distribution and sampled-profile path laws, and hence their discounted payoffs, coincide, even with stochastic arena transitions. Indeed, a pure strategy selects one pure profile at each history. Its averaged prescription is the family of Dirac actions at that profile, and its sampled prescription is the Dirac law on it. At any common history, with selected pair ( a , b ) (a,b) ( a , b ) , both next-state laws equal δ Φ a , b ( μ ) \delta_{\Phi_{a,b}(\mu)} δ Φ a , b ( μ ) , because Υ a , b = Φ a , b \Upsilon_{a,b} = \Phi_{a,b} Υ a , b = Φ a , b . Starting from the same distribution, induction shows that every finite prefix agrees, so all later selected profiles agree as well, even if the strategies depend on the whole distribution history. With deterministic transitions as well, the sampled-transition law agrees too (previous remark), so all three path laws and payoffs coincide for pure strategies. These are equalities of induced laws specified strategies, not an identification of all admissible averaged and sampled strategies: in Theorem 2 the game is deterministic, but the averaged upper value is one, the averaged lower value is ( 1 − γ ) + γ h ( 1 / 4 ) (1-\gamma) + \gamma h(1/4) ( 1 − γ ) + γ h ( 1/4 ) , and the two sampled values equal ( 1 − γ ) + γ / 2 (1-\gamma) + \gamma/2 ( 1 − γ ) + γ /2 .
Open-loop comparison of the three semantics
Theorem 37 (Open-loop entropy ordering). Let ( σ t , τ t ) t ≥ 0 (\sigma_t,\tau_t)_{t\ge0} ( σ t , τ t ) t ≥ 0 be prescribed profile laws depending on time only, not on observed distributions. Under averaging use their statewise marginals x t , y t x_t,y_t x t , y t ; under both sampled semantics use the laws σ t , τ t \sigma_t,\tau_t σ t , τ t themselves, with fresh independent draws each round. Write μ t a v \mu_t^{\mathrm{av}} μ t av for the deterministic averaged trajectory. For every
t ≥ 0 , t \ge0, t ≥ 0 ,
H ( μ t a v ) ≥ E s p [ H ( μ t ) ] ≥ E s t [ H ( μ t ) ] . \mathrm{H}(\mu_t^{\mathrm{av}}) \ge\mathbb{E}^{\mathrm{sp}}[\mathrm{H}(\mu_t)] \ge\mathbb{E}^{\mathrm{st}}[\mathrm{H}(\mu_t)]. H ( μ t av ) ≥ E sp [ H ( μ t )] ≥ E st [ H ( μ t )] .
Consequently the same weak ordering holds for their normalised discounted payoffs. No independence between a player’s different state coordinates is required. The inequalities need not be strict.
Proof. Throughout, treat a distribution μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) as a row vector in R 1 × S \mathbb{R}^{1 \times S} R 1 × S , so that a stochastic matrix acts on it by right multiplication. For each j ≥ 0 j \ge0 j ≥ 0 , draw a profile pair ( A j , B j ) (A_j,B_j) ( A j , B j ) with law σ j × τ j \sigma_j \times\tau_j σ j × τ j , independently across j j j ; conditional on all of ( A j , B j ) j ≥ 0 (A_j,B_j)_{j \ge0} ( A j , B j ) j ≥ 0 , draw realisations ξ j ∼ P A j , B j \xi_j \sim P_{A_j,B_j} ξ j ∼ P A j , B j , independently across j j j . For a fixed time t t t only finitely many of these draws are ever used. Define two processes from this single common experiment:
μ 0 s p = μ 0 s t = μ 0 , \mu_0^{\mathrm{sp}}=\mu_0^{\mathrm{st}}=\mu_0, μ 0 sp = μ 0 st = μ 0 ,
and, for $j \ge 0,
μ j + 1 s p = Φ A j , B j ( μ j s p ) , μ j + 1 s t = Ψ ξ j ( μ j s t ) . \mu_{j+1}^{\mathrm{sp}}=\Phi_{A_j,B_j}(\mu_j^{\mathrm{sp}}), \qquad\mu_{j+1}^{\mathrm{st}}=\Psi_{\xi_j}(\mu_j^{\mathrm{st}}). μ j + 1 sp = Φ A j , B j ( μ j sp ) , μ j + 1 st = Ψ ξ j ( μ j st ) .
Since both use the same profile draws ( A j , B j ) j ≥ 0 (A_j,B_j)_{j \ge0} ( A j , B j ) j ≥ 0 , and since σ j , τ j \sigma_j,\tau_j σ j , τ j are open-loop (do not depend on the observed distribution), each has exactly the marginal law of the corresponding semantics’ distribution process: this is immediate from (14) and (15) , applied round by round. So it suffices to prove the stated inequalities for μ t s p , μ t s t \mu_t^{\mathrm{sp}},\mu_t^{\mathrm{st}} μ t sp , μ t st as just constructed.
Let K a , b K_{a,b} K a , b be the stochastic matrix with entries ( K a , b ) s , s ′ = Δ ( s , ρ a , b ( s ) , s ′ ) (K_{a,b})_{s,s'}=\Delta(s,\rho_{a,b}(s),s') ( K a , b ) s , s ′ = Δ ( s , ρ a , b ( s ) , s ′ ) , and let R ξ R_{\xi} R ξ be the { 0 , 1 } \{0,1\} { 0 , 1 } -valued matrix with ( R ξ ) s , s ′ = 1 { ξ ( s ) = s ′ } (R_{\xi})_{s,s'}=\mathbf{1}_{\{\xi(s)=s'\}} ( R ξ ) s , s ′ = 1 { ξ ( s ) = s ′ } . By (9) and (12) , Φ a , b ( μ ) = μ K a , b \Phi_{a,b}(\mu)=\mu K_{a,b} Φ a , b ( μ ) = μ K a , b and Ψ ξ ( μ ) = μ R ξ \Psi_{\xi}(\mu)=\mu R_{\xi} Ψ ξ ( μ ) = μ R ξ for every μ \mu μ , so
μ t s p = μ 0 K A 0 , B 0 ⋯ K A t − 1 , B t − 1 , μ t s t = μ 0 R ξ 0 ⋯ R ξ t − 1 , (68) \mu_t^{\mathrm{sp}}=\mu_0K_{A_0,B_0}\cdots K_{A_{t-1},B_{t-1}}, \qquad\mu_t^{\mathrm{st}}=\mu_0R_{\xi_0}\cdots R_{\xi_{t-1}}, \tag*{(68)} μ t sp = μ 0 K A 0 , B 0 ⋯ K A t − 1 , B t − 1 , μ t st = μ 0 R ξ 0 ⋯ R ξ t − 1 , (68)
both exactly, as products of random matrices. Since the coordinates of ξ \xi ξ are independent given ( A , B ) (A,B) ( A , B ) with ξ ( s ) ∼ Δ ( s , ρ A , B ( s ) , ⋅ ) \xi(s)\sim\Delta(s,\rho_{A,B}(s),\cdot) ξ ( s ) ∼ Δ ( s , ρ A , B ( s ) , ⋅ ) , E [ R ξ j ∣ A j = a , B j = b ] = K a , b \mathbb{E}[R_{\xi_j}\mid A_j=a,B_j=b]=K_{a,b} E [ R ξ j ∣ A j = a , B j = b ] = K a , b for every j , a , b j,a,b j , a , b .
We first show, by induction on t t t , that E [ μ t s t ∣ A t ] = μ 0 K A 0 , B 0 ⋯ K A t − 1 , B t − 1 \mathbb{E}[\mu_t^{\mathrm{st}}\mid\mathcal{A}_t]=\mu_0K_{A_0,B_0}\cdots K_{A_{t-1},B_{t-1}} E [ μ t st ∣ A t ] = μ 0 K A 0 , B 0 ⋯ K A t − 1 , B t − 1 , where A t = ( A 0 , B 0 , … , A t − 1 , B t − 1 ) \mathcal{A}_t=(A_0,B_0,\ldots,A_{t-1},B_{t-1}) A t = ( A 0 , B 0 , … , A t − 1 , B t − 1 ) . The base case t = 0 t=0 t = 0 is the equality μ 0 = μ 0 \mu_0=\mu_0 μ 0 = μ 0 . For the step, write μ t + 1 s t = μ t s t R ξ t \mu_{t+1}^{\mathrm{st}}=\mu_t^{\mathrm{st}}R_{\xi_t} μ t + 1 st = μ t st R ξ t . The profile pairs are independent across time, so ( A t , B t ) (A_t,B_t) ( A t , B t ) is independent of A t \mathcal{A}_t A t ; and, conditional on all profiles, ξ t \xi_t ξ t is independent of ξ 0 , … , ξ t − 1 \xi_0,\ldots,\xi_{t-1} ξ 0 , … , ξ t − 1 , hence of μ t s t \mu_t^{\mathrm{st}} μ t st , which is a function of ξ 0 , … , ξ t − 1 \xi_0,\ldots,\xi_{t-1} ξ 0 , … , ξ t − 1 and A t \mathcal{A}_t A t alone. So, conditional on A t + 1 = ( A t , A t , B t ) \mathcal{A}_{t+1}=(\mathcal{A}_t,A_t,B_t) A t + 1 = ( A t , A t , B t ) , the row vector μ t s t \mu_t^{\mathrm{st}} μ t st and the matrix R ξ t R_{\xi_t} R ξ t are independent, and their product’s conditional mean factors entrywise:
E [ μ t s t R ξ t ∣ A t + 1 ] = E [ μ t s t ∣ A t + 1 ] ⋅ E [ R ξ t ∣ A t + 1 ] . \mathbb{E}[\mu_t^{\mathrm{st}}R_{\xi_t}\mid\mathcal{A}_{t+1}]
=
\mathbb{E}[\mu_t^{\mathrm{st}}\mid\mathcal{A}_{t+1}]
\cdot
\mathbb{E}[R_{\xi_t}\mid\mathcal{A}_{t+1}]. E [ μ t st R ξ t ∣ A t + 1 ] = E [ μ t st ∣ A t + 1 ] ⋅ E [ R ξ t ∣ A t + 1 ] .
The first factor equals E [ μ t s t ∣ A t ] \mathbb{E}[\mu_t^{\mathrm{st}}\mid\mathcal{A}_t] E [ μ t st ∣ A t ] , since conditioning further on the independent pair ( A t , B t ) (A_t,B_t) ( A t , B t ) does not change it, which is μ 0 K A 0 , B 0 ⋯ K A t − 1 , B t − 1 \mu_0K_{A_0,B_0}\cdots K_{A_{t-1},B_{t-1}} μ 0 K A 0 , B 0 ⋯ K A t − 1 , B t − 1 by the induction hypothesis. The second factor is E [ R ξ t ∣ A t , B t ] = K A t , B t \mathbb{E}[R_{\xi_t}\mid A_t,B_t]=K_{A_t,B_t} E [ R ξ t ∣ A t , B t ] = K A t , B t , since ξ t \xi_t ξ t ’s law depends only on ( A t , B t ) (A_t,B_t) ( A t , B t ) . Multiplying proves the inductive step. Combined with (68) , this gives
E [ μ t s t ∣ A t ] = μ 0 K A 0 , B 0 ⋯ K A t − 1 , B t − 1 = μ t s p . (69) \mathbb{E}[\mu_t^{\mathrm{st}}\mid\mathcal{A}_t]
=
\mu_0K_{A_0,B_0}\cdots K_{A_{t-1},B_{t-1}}
=
\mu_t^{\mathrm{sp}}. \tag*{(69)} E [ μ t st ∣ A t ] = μ 0 K A 0 , B 0 ⋯ K A t − 1 , B t − 1 = μ t sp . (69)
Next, define the marginal action at s ∈ S min s\in S_{\min} s ∈ S m i n by x j ( s ) ( α ) = ∑ a : a ( s ) = α σ j ( a ) x_j(s)(\alpha)=\sum_{a:a(s)=\alpha}\sigma_j(a) x j ( s ) ( α ) = ∑ a : a ( s ) = α σ j ( a ) , and y j y_j y j analogously from τ j \tau_j τ j at s ∈ S max s\in S_{\max} s ∈ S m a x , and set K ‾ j : = ∑ a , b σ j ( a ) τ j ( b ) K a , b \overline{K}_j:=\sum_{a,b}\sigma_j(a)\tau_j(b)K_{a,b} K j := ∑ a , b σ j ( a ) τ j ( b ) K a , b . At a Min state s s s , every K a , b K_{a,b} K a , b ’s row depends on a , b a,b a , b only through a ( s ) a(s) a ( s ) , so ( K ‾ j ) s , ⋅ = ∑ a σ j ( a ) Δ ( s , a ( s ) , ⋅ ) = ∑ α x j ( s ) ( α ) Δ ( s , α , ⋅ ) (\overline{K}_j)_{s,\cdot}=\sum_a\sigma_j(a)\Delta(s,a(s),\cdot)=\sum_\alpha x_j(s)(\alpha)\Delta(s,\alpha,\cdot) ( K j ) s , ⋅ = ∑ a σ j ( a ) Δ ( s , a ( s ) , ⋅ ) = ∑ α x j ( s ) ( α ) Δ ( s , α , ⋅ ) , and symmetrically at a Max state; thus K ‾ j \overline{K}_j K j exactly equals the matrix of Υ x j , y j \Upsilon_{x_j,y_j} Υ x j , y j , using only the one-state marginals x j ( s ) , y j ( s ) x_j(s),y_j(s) x j ( s ) , y j ( s ) , not any correlation σ j , τ j \sigma_j,\tau_j σ j , τ j may impose across different states. Consequently μ j + 1 a v = Υ x j , y j ( μ j a v ) = μ j a v K ‾ j \mu_{j+1}^{\mathrm{av}}=\Upsilon_{x_j,y_j}(\mu_j^{\mathrm{av}})=\mu_j^{\mathrm{av}}\overline{K}_j μ j + 1 av = Υ x j , y j ( μ j av ) = μ j av K j , so by induction on j j j , μ t a v = μ 0 K ‾ 0 ⋯ K ‾ t − 1 \mu_t^{\mathrm{av}}=\mu_0\overline{K}_0\cdots\overline{K}_{t-1} μ t av = μ 0 K 0 ⋯ K t − 1 exactly. We now show, again by induction on t t t , that E [ μ t s p ] = μ 0 K ‾ 0 ⋯ K ‾ t − 1 \mathbb{E}[\mu_t^{\mathrm{sp}}] = \mu_0\overline{K}_0\cdots\overline{K}_{t-1} E [ μ t sp ] = μ 0 K 0 ⋯ K t − 1 . The base case is trivial. For the step, μ t + 1 s p = μ t s p K A t , B t \mu_{t+1}^{\mathrm{sp}}=\mu_t^{\mathrm{sp}}K_{A_t,B_t} μ t + 1 sp = μ t sp K A t , B t , with μ t s p \mu_t^{\mathrm{sp}} μ t sp a function of A t A_t A t independent of ( A t , B t ) (A_t,B_t) ( A t , B t ) , so
E [ μ t + 1 s p ] = E [ μ t s p ] ⋅ E [ K A t , B t ] = ( μ 0 K ‾ 0 ⋯ K ‾ t − 1 ) K ‾ t , \mathbb{E}[\mu_{t+1}^{\mathrm{sp}}]=\mathbb{E}[\mu_t^{\mathrm{sp}}]\cdot\mathbb{E}[K_{A_t,B_t}]=(\mu_0\overline{K}_0\cdots\overline{K}_{t-1})\overline{K}_t, E [ μ t + 1 sp ] = E [ μ t sp ] ⋅ E [ K A t , B t ] = ( μ 0 K 0 ⋯ K t − 1 ) K t ,
using E [ K A t , B t ] = ∑ a , b σ t ( a ) τ t ( b ) K a , b = K ‾ t \mathbb{E}[K_{A_t,B_t}]=\sum_{a,b}\sigma_t(a)\tau_t(b)K_{a,b}=\overline{K}_t E [ K A t , B t ] = ∑ a , b σ t ( a ) τ t ( b ) K a , b = K t , which proves the step. Combined with μ t a v = μ 0 K ‾ 0 ⋯ K ‾ t − 1 \mu_t^{\mathrm{av}}=\mu_0\overline{K}_0\cdots\overline{K}_{t-1} μ t av = μ 0 K 0 ⋯ K t − 1 above, this gives
E [ μ t s p ] = μ 0 K ‾ 0 ⋯ K ‾ t − 1 = μ t a v . (70) \mathbb{E}[\mu_t^{\mathrm{sp}}]=\mu_0\overline{K}_0\cdots\overline{K}_{t-1}=\mu_t^{\mathrm{av}}. \tag*{(70)} E [ μ t sp ] = μ 0 K 0 ⋯ K t − 1 = μ t av . (70)
Both (69) and (70) are exact identities between (conditional) means of matrix products; being linear in the underlying distribution, they use no concavity and are not an instance of replacing the expectation of a nonlinear quantity by its value at the mean. Concavity enters only now. Applying H \mathbf{H} H ’s concavity to (69) , in the form H ( E [ μ t s t ∣ A t ] ) ≥ E [ H ( μ t s t ) ∣ A t ] \mathbf{H}(\mathbb{E}[\mu_t^{\mathrm{st}}\mid A_t])\geq\mathbb{E}[\mathbf{H}(\mu_t^{\mathrm{st}})\mid A_t] H ( E [ μ t st ∣ A t ]) ≥ E [ H ( μ t st ) ∣ A t ] , and taking expectations over A t A_t A t , gives H ( μ t s p ) ≥ E [ H ( μ t s t ) ∣ A t ] \mathbf{H}(\mu_t^{\mathrm{sp}})\geq\mathbb{E}[\mathbf{H}(\mu_t^{\mathrm{st}})\mid A_t] H ( μ t sp ) ≥ E [ H ( μ t st ) ∣ A t ] pointwise in A t A_t A t (the left side is a function of A t A_t A t alone by (68) ), so taking a further expectation gives E [ H ( μ t s p ) ] ≥ E [ H ( μ t s t ) ] \mathbb{E}[\mathbf{H}(\mu_t^{\mathrm{sp}})]\geq\mathbb{E}[\mathbf{H}(\mu_t^{\mathrm{st}})] E [ H ( μ t sp )] ≥ E [ H ( μ t st )] , the sp versus st inequality. Applying concavity to (70) directly gives H ( μ t a v ) = H ( E [ μ t s p ] ) ≥ E [ H ( μ t s p ) ] \mathbf{H}(\mu_t^{\mathrm{av}})=\mathbf{H}(\mathbb{E}[\mu_t^{\mathrm{sp}}])\geq\mathbb{E}[\mathbf{H}(\mu_t^{\mathrm{sp}})] H ( μ t av ) = H ( E [ μ t sp ]) ≥ E [ H ( μ t sp )] , the av versus sp inequality. Multiplying both by the nonnegative weights ( 1 − γ ) γ t (1-\gamma)\gamma^t ( 1 − γ ) γ t and summing over t t t proves the discounted comparison; boundedness of entropy by log 2 n \log_2 n log 2 n justifies passing from finite partial sums to the infinite discounted sum.
Strict concavity of H \mathbf{H} H (Proposition 16 ) further identifies equality at a given t t t : av versus sp is an equality exactly when μ t s p \mu_t^{\mathrm{sp}} μ t sp is almost surely constant, and sp versus st is an equality exactly when μ t s t \mu_t^{\mathrm{st}} μ t st is conditionally almost surely constant given A t A_t A t . Neither holds in general, so strictness is not universal even in nontrivial arenas. This argument is stated only for open-loop prescriptions because it depends on ξ t \xi_t ξ t being independent of μ t s t \mu_t^{\mathrm{st}} μ t st given A t + 1 A_{t+1} A t + 1 : under feedback, σ t , τ t \sigma_t,\tau_t σ t , τ t may depend on the observed μ t s p \mu_t^{\mathrm{sp}} μ t sp or μ t s t \mu_t^{\mathrm{st}} μ t st , which in turn depends on earlier realisations, so conditioning on a later profile can reveal information about an earlier one, and the independence used above need not hold.
Concavity of the averaged values and dominance
For a continuous concave f : D ( S ) → R f:D(S)\to\mathbb{R} f : D ( S ) → R , write
g f + ( μ ) = min x max y f ( Υ x , y ( μ ) ) , g f − ( μ ) = max y min x f ( Υ x , y ( μ ) ) . g_f^+(\mu)=\min_x\max_y f(\Upsilon_{x,y}(\mu)),\qquad g_f^-(\mu)=\max_y\min_x f(\Upsilon_{x,y}(\mu)). g f + ( μ ) = x min y max f ( Υ x , y ( μ )) , g f − ( μ ) = y max x min f ( Υ x , y ( μ )) .
All extrema are attained by compactness and continuity.
Lemma 38 (Mass-weighted mixing). Both g f + g_f^+ g f + and g f − g_f^- g f − are continuous and concave. For fixed μ , y \mu,y μ , y , the minimum over x x x in g f − g_f^- g f − is attained at a pure profile. The minimum over x x x in g f + g_f^+ g f + is also attained at a pure profile.
Proof. Let μ = λ μ 1 + ( 1 − λ ) μ 2 \mu=\lambda\mu_1+(1-\lambda)\mu_2 μ = λ μ 1 + ( 1 − λ ) μ 2 with 0 < λ < 1 0<\lambda<1 0 < λ < 1 . For any two maximiser prescriptions y 1 , y 2 ∈ Y y_1,y_2\in\mathcal{Y} y 1 , y 2 ∈ Y , define, at every Max-controlled state s s s with μ ( s ) > 0 \mu(s)>0 μ ( s ) > 0 ,
y ( s ) = λ μ 1 ( s ) y 1 ( s ) + ( 1 − λ ) μ 2 ( s ) y 2 ( s ) μ ( s ) . y(s)=\frac{\lambda\mu_1(s)y_1(s)+(1-\lambda)\mu_2(s)y_2(s)}{\mu(s)}. y ( s ) = μ ( s ) λ μ 1 ( s ) y 1 ( s ) + ( 1 − λ ) μ 2 ( s ) y 2 ( s ) . This is a nonnegative combination of two probability distributions on the enabled actions at s s s , with total weight ( λ μ 1 ( s ) + ( 1 − λ ) μ 2 ( s ) ) / μ ( s ) = 1 (\lambda\mu_1(s)+(1-\lambda)\mu_2(s))/\mu(s)=1 ( λ μ 1 ( s ) + ( 1 − λ ) μ 2 ( s )) / μ ( s ) = 1 , hence itself a probability distribution on those actions. At a state of zero mass, both μ 1 ( s ) \mu_1(s) μ 1 ( s ) and μ 2 ( s ) \mu_2(s) μ 2 ( s ) vanish; choose any enabled Dirac action there. This defines y ∈ Y y\in\mathcal{Y} y ∈ Y , a single legitimate prescription, not a strategy that observes which of μ 1 , μ 2 \mu_1,\mu_2 μ 1 , μ 2 has occurred. It satisfies, for every x ∈ X x\in\mathcal{X} x ∈ X ,
Υ x , y ( μ ) = λ Υ x , y 1 ( μ 1 ) + ( 1 − λ ) Υ x , y 2 ( μ 2 ) . (71) \Upsilon_{x,y}(\mu)=\lambda\Upsilon_{x,y_1}(\mu_1)+(1-\lambda)\Upsilon_{x,y_2}(\mu_2). \tag*{(71)} Υ x , y ( μ ) = λ Υ x , y 1 ( μ 1 ) + ( 1 − λ ) Υ x , y 2 ( μ 2 ) . (71)
At Min states this holds because the same x x x appears in both terms; at Max states it holds by construction of y y y : both sides equal, at s ′ ∈ S s' \in S s ′ ∈ S , ∑ α ( λ μ 1 ( s ) y 1 ( s ) + ( 1 − λ ) μ 2 ( s ) y 2 ( s ) ) ( α ) Δ ( s , α , s ′ ) \sum_{\alpha}(\lambda\mu_1(s)y_1(s)+(1-\lambda)\mu_2(s)y_2(s))(\alpha)\Delta(s,\alpha,s') ∑ α ( λ μ 1 ( s ) y 1 ( s ) + ( 1 − λ ) μ 2 ( s ) y 2 ( s )) ( α ) Δ ( s , α , s ′ ) summed over α \alpha α , by (10) and the definition of y y y .
*Concavity of g f − g_f^- g f − . *For i = 1 , 2 i=1,2 i = 1 , 2 , let y i ∈ Y y_i \in\mathcal{Y} y i ∈ Y attain the outer maximum in g f − ( μ i ) = max y ′ min x f ( Υ x , y ′ ( μ i ) ) g_f^-(\mu_i)=\max_{y'}\min_x f(\Upsilon_{x,y'}(\mu_i)) g f − ( μ i ) = max y ′ min x f ( Υ x , y ′ ( μ i )) , i.e. min x f ( Υ x , y i ( μ i ) ) = g f − ( μ i ) \min_x f(\Upsilon_{x,y_i}(\mu_i))=g_f^-(\mu_i) min x f ( Υ x , y i ( μ i )) = g f − ( μ i ) , and build y y y as above. Concavity of f f f and (71) give, for every x ∈ X x \in\mathcal{X} x ∈ X ,
f ( Υ x , y ( μ ) ) ≥ λ f ( Υ x , y 1 ( μ 1 ) ) + ( 1 − λ ) f ( Υ x , y 2 ( μ 2 ) ) ≥ λ min x ′ f ( Υ x ′ , y 1 ( μ 1 ) ) + ( 1 − λ ) min x ′ f ( Υ x ′ , y 2 ( μ 2 ) ) = λ g f − ( μ 1 ) + ( 1 − λ ) g f − ( μ 2 ) . \begin{aligned}
f(\Upsilon_{x,y}(\mu)) &\ge\lambda f(\Upsilon_{x,y_1}(\mu_1))+(1-\lambda)f(\Upsilon_{x,y_2}(\mu_2))\\
&\ge\lambda\min_{x'} f(\Upsilon_{x',y_1}(\mu_1))+(1-\lambda)\min_{x'} f(\Upsilon_{x',y_2}(\mu_2))\\
&=\lambda g_f^-(\mu_1)+(1-\lambda)g_f^-(\mu_2).
\end{aligned} f ( Υ x , y ( μ )) ≥ λ f ( Υ x , y 1 ( μ 1 )) + ( 1 − λ ) f ( Υ x , y 2 ( μ 2 )) ≥ λ x ′ min f ( Υ x ′ , y 1 ( μ 1 )) + ( 1 − λ ) x ′ min f ( Υ x ′ , y 2 ( μ 2 )) = λ g f − ( μ 1 ) + ( 1 − λ ) g f − ( μ 2 ) .
Taking the minimum over x ∈ X x \in\mathcal{X} x ∈ X on the left preserves this lower bound, giving min x f ( Υ x , y ( μ ) ) ≥ λ g f − ( μ 1 ) + ( 1 − λ ) g f − ( μ 2 ) \min_x f(\Upsilon_{x,y}(\mu)) \ge\lambda g_f^-(\mu_1)+(1-\lambda)g_f^-(\mu_2) min x f ( Υ x , y ( μ )) ≥ λ g f − ( μ 1 ) + ( 1 − λ ) g f − ( μ 2 ) . Since y y y is one particular feasible choice in Y \mathcal{Y} Y ,
g f − ( μ ) = max y ′ min x f ( Υ x , y ′ ( μ ) ) ≥ min x f ( Υ x , y ( μ ) ) , g_f^-(\mu)=\max_{y'}\min_x f(\Upsilon_{x,y'}(\mu))\ge\min_x f(\Upsilon_{x,y}(\mu)), g f − ( μ ) = y ′ max x min f ( Υ x , y ′ ( μ )) ≥ x min f ( Υ x , y ( μ )) ,
so g f − ( μ ) ≥ λ g f − ( μ 1 ) + ( 1 − λ ) g f − ( μ 2 ) g_f^-(\mu)\ge\lambda g_f^-(\mu_1)+(1-\lambda)g_f^-(\mu_2) g f − ( μ ) ≥ λ g f − ( μ 1 ) + ( 1 − λ ) g f − ( μ 2 ) , proving concavity of g f − g_f^- g f − .
*Concavity of g f + g_f^+ g f + . *Fix x ∈ X x \in\mathcal{X} x ∈ X . For i = 1 , 2 i=1,2 i = 1 , 2 , let y i ∈ Y y_i \in\mathcal{Y} y i ∈ Y attain max y ′ f ( Υ x , y ′ ( μ i ) ) \max_{y'}f(\Upsilon_{x,y'}(\mu_i)) max y ′ f ( Υ x , y ′ ( μ i )) and build y y y as above; the same two displayed inequalities, with min x ′ \min_{x'} min x ′ everywhere replaced by the fixed x x x , show that μ ↦ max y ′ f ( Υ x , y ′ ( μ ) ) \mu\mapsto\max_{y'}f(\Upsilon_{x,y'}(\mu)) μ ↦ max y ′ f ( Υ x , y ′ ( μ )) is concave, for this fixed x x x . A pointwise infimum of concave functions is concave: if h x ( λ μ 1 + ( 1 − λ ) μ 2 ) ≥ λ h x ( μ 1 ) + ( 1 − λ ) h x ( μ 2 ) h_x(\lambda\mu_1+(1-\lambda)\mu_2)\ge\lambda h_x(\mu_1)+(1-\lambda)h_x(\mu_2) h x ( λ μ 1 + ( 1 − λ ) μ 2 ) ≥ λ h x ( μ 1 ) + ( 1 − λ ) h x ( μ 2 ) holds for every x x x in an index set, then taking inf x \inf_x inf x of both sides preserves the inequality with h x h_x h x replaced by inf x h x \inf_x h_x inf x h x on both occurrences. Applying this to h x ( μ ) = max y ′ f ( Υ x , y ′ ( μ ) ) h_x(\mu)=\max_{y'}f(\Upsilon_{x,y'}(\mu)) h x ( μ ) = max y ′ f ( Υ x , y ′ ( μ )) shows g f + = inf x h x g_f^+=\inf_x h_x g f + = inf x h x is concave.
*Continuity. *f f f is continuous on the compact set D ( S ) \mathcal{D}(S) D ( S ) , hence uniformly continuous, and ( μ , x , y ) ↦ Υ x , y ( μ ) (\mu,x,y)\mapsto\Upsilon_{x,y}(\mu) ( μ , x , y ) ↦ Υ x , y ( μ ) is jointly continuous (Proposition 23), so ( μ , x , y ) ↦ f ( Υ x , y ( μ ) ) (\mu,x,y)\mapsto f(\Upsilon_{x,y}(\mu)) ( μ , x , y ) ↦ f ( Υ x , y ( μ )) is continuous on the compact set D ( S ) × X × Y \mathcal{D}(S)\times\mathcal{X}\times\mathcal{Y} D ( S ) × X × Y , hence itself uniformly continuous there. Fix η > 0 \eta>0 η > 0 and let ρ > 0 \rho>0 ρ > 0 be such that d ( μ , μ ′ ) < ρ d(\mu,\mu')<\rho d ( μ , μ ′ ) < ρ implies ∣ f ( Υ x , y ( μ ) ) − f ( Υ x , y ( μ ′ ) ) ∣ < η |f(\Upsilon_{x,y}(\mu))-f(\Upsilon_{x,y}(\mu'))|<\eta ∣ f ( Υ x , y ( μ )) − f ( Υ x , y ( μ ′ )) ∣ < η for every x ∈ X , y ∈ Y x\in\mathcal{X},y\in\mathcal{Y} x ∈ X , y ∈ Y . For such μ , μ ′ \mu,\mu' μ , μ ′ , every term inside the outer extrema defining g f + ( μ ) g_f^+(\mu) g f + ( μ ) and g f + ( μ ′ ) g_f^+(\mu') g f + ( μ ′ ) changes by less than η \eta η , so, exactly as in (33) , the resulting min max values satisfy ∣ g f + ( μ ) − g f + ( μ ′ ) ∣ ≤ η |g_f^+(\mu)-g_f^+(\mu')|\le\eta ∣ g f + ( μ ) − g f + ( μ ′ ) ∣ ≤ η ; the same bound holds for g f − g_f^- g f − . Since η \eta η was arbitrary, both g f + g_f^+ g f + and g f − g_f^- g f − are (uniformly) continuous on D ( S ) \mathcal{D}(S) D ( S ) .
*Vertex attainment. *For fixed μ , y \mu,y μ , y , the map x ↦ f ( Υ x , y ( μ ) ) x\mapsto f(\Upsilon_{x,y}(\mu)) x ↦ f ( Υ x , y ( μ )) is concave on the finite product of simplices X \mathcal{X} X , so its minimum is attained at an extreme point of X \mathcal{X} X , i.e. at a pure profile: this proves the claim for g f − g_f^- g f − , where y y y is already fixed before the minimum over x x x is taken. For g f + g_f^+ g f + , the map ( x , y ) ↦ f ( Υ x , y ( μ ) ) (x,y)\mapsto f(\Upsilon_{x,y}(\mu)) ( x , y ) ↦ f ( Υ x , y ( μ )) is jointly concave, since Υ x , y ( μ ) \Upsilon_{x,y}(\mu) Υ x , y ( μ ) is affine in the joint action ( x , y ) (x,y) ( x , y ) and f f f is concave. Partial maximisation of a jointly concave function over the convex set Y \mathcal{Y} Y preserves concavity in the remaining variable, so x ↦ max y f ( Υ x , y ( μ ) ) x\mapsto\max_y f(\Upsilon_{x,y}(\mu)) x ↦ max y f ( Υ x , y ( μ )) is concave on X \mathcal{X} X . Its minimum over x ∈ X x\in\mathcal{X} x ∈ X is therefore again attained at a vertex, i.e. a pure profile. Neither argument requires, or asserts, that Max has a pure optimiser: only Min’s minimisation is claimed to be vertex-attaining.
Theorem 5 (Comparison of the three semantics). The averaged values V a v + V_{\mathrm{av}}^+ V av + and V a v − V_{\mathrm{av}}^- V av − are concave and, for every μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) , V a v + ( μ ) ≥ V a v − ( μ ) ≥ max { V s p ( μ ) , V s t ( μ ) } V_{\mathrm{av}}^+(\mu)\ge V_{\mathrm{av}}^-(\mu)\ge\max\{V_{\mathrm{sp}}(\mu),V_{\mathrm{st}}(\mu)\} V av + ( μ ) ≥ V av − ( μ ) ≥ max { V sp ( μ ) , V st ( μ )} . If S max = ∅ S_{\max}=\varnothing S m a x = ∅ , then V a v = V s p ≥ V s t V_{\mathrm{av}}=V_{\mathrm{sp}}\ge V_{\mathrm{st}} V av = V sp ≥ V st . In general, V s p V_{\mathrm{sp}} V sp and V s t V_{\mathrm{st}} V st are incomparable: either strict inequality can occur.
Proof. Write U r + , U r − U_r^+,U_r^- U r + , U r − for the two r r r -stage averaged values, and W r ⋆ W_r^\star W r ⋆ for the common sampled values. By the finite-horizon identification in the proof of Theorem 1 (Section C), U a v , r ± = ( T a v ± ) r 0 U_{\mathrm{av},r}^{\pm}=(T_{\mathrm{av}}^{\pm})^r0 U av , r ± = ( T av ± ) r 0 are continuous and satisfy the averaged recursions below, with 0 ≤ V a v + − 0\le V_{\mathrm{av}}^+- 0 ≤ V av + − U r ± ≤ γ r log 2 n U_r^{\pm} \leq\gamma^r \log_2 n U r ± ≤ γ r log 2 n . The finite-horizon recursion for W r ⋆ W_r^{\star} W r ⋆ was proved in Theorem 4. Both start at zero, U 0 ± ≡ W 0 ⋆ ≡ 0 U_0^{\pm} \equiv W_0^{\star} \equiv0 U 0 ± ≡ W 0 ⋆ ≡ 0 , and, using the notation g f g_f g f of Lemma 38 , the averaged recursions read U r + 1 ± = ( 1 − γ ) H + γ g U r ± ± U_{r+1}^{\pm}=(1-\gamma)\mathbf{H}+\gamma g_{U_r^{\pm}}^{\pm} U r + 1 ± = ( 1 − γ ) H + γ g U r ± ± . We first show by induction on r r r that both U r ± U_r^{\pm} U r ± are continuous and concave. The base case U 0 ± ≡ 0 U_0^{\pm}\equiv0 U 0 ± ≡ 0 is trivial. For the inductive step, if U r ± U_r^{\pm} U r ± is continuous and concave, Lemma 38 (applied with f = U r ± f=U_r^{\pm} f = U r ± ) makes g U r ± ± g_{U_r^{\pm}}^{\pm} g U r ± ± continuous and concave; since H \mathbf{H} H is itself continuous and concave (Proposition 16 ), so is the sum U r + 1 ± = ( 1 − γ ) H + γ g U r ± ± U_{r+1}^{\pm}=(1-\gamma)\mathbf{H}+\gamma g_{U_r^{\pm}}^{\pm} U r + 1 ± = ( 1 − γ ) H + γ g U r ± ± .
We prove simultaneously that U r − ≥ W r s p U_r^{-}\geq W_r^{\mathrm{sp}} U r − ≥ W r sp and U r − ≥ W r s t U_r^{-}\geq W_r^{\mathrm{st}} U r − ≥ W r st , by induction on r r r ; the base case r = 0 r=0 r = 0 is the equality 0 = 0 0=0 0 = 0 . Fix μ ∈ D ( S ) \mu\in\mathcal{D}(S) μ ∈ D ( S ) and a semantics ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } , and let τ ∈ D ( P max ) \tau\in\mathcal{D}(\mathcal{P}_{\max}) τ ∈ D ( P m a x ) be an optimal column strategy in the matrix M W r ⋆ ⋆ ( μ ) M_{W_r^{\star}}^{\star}(\mu) M W r ⋆ ⋆ ( μ ) , i.e. one attaining the outer maximum in the one-sided characterisation (52) of val ( M W r ⋆ ⋆ ( μ ) ) \operatorname{val}(M_{W_r^{\star}}^{\star}(\mu)) val ( M W r ⋆ ⋆ ( μ )) . Define y ˉ ∈ Y \bar{y}\in\mathcal{Y} y ˉ ∈ Y by taking, at each Max-controlled state s s s , the marginal of τ \tau τ at s s s : y ˉ ( s ) ( β ) : = ∑ b : b ( s ) = β τ ( b ) \bar{y}(s)(\beta):=\sum_{b:b(s)=\beta}\tau(b) y ˉ ( s ) ( β ) := ∑ b : b ( s ) = β τ ( b ) . For every pure row a ∈ P min a\in\mathcal{P}_{\min} a ∈ P m i n , the mean-successor identities give
Υ a , y ˉ ( μ ) = ∑ b τ ( b ) Φ a , b ( μ ) = ∑ b , ξ τ ( b ) P a , b ( ξ ) Ψ ξ ( μ ) . \Upsilon_{a,\bar{y}}(\mu)=\sum_b\tau(b)\Phi_{a,b}(\mu)=\sum_{b,\xi}\tau(b)P_{a,b}(\xi)\Psi_{\xi}(\mu). Υ a , y ˉ ( μ ) = b ∑ τ ( b ) Φ a , b ( μ ) = b , ξ ∑ τ ( b ) P a , b ( ξ ) Ψ ξ ( μ ) .
The first equality needs only the one-coordinate marginals of τ \tau τ , not a product-law assumption. Put f = U r − f=U_r^{-} f = U r − , continuous and concave by the previous paragraph. Recall the kernel K ⋆ ( ⋅ ∣ μ , a , b ) K_{\star}(\cdot\mid\mu,a,b) K ⋆ ( ⋅ ∣ μ , a , b ) of Appendix § 14 , the law of the next distribution given the current one and a pure-profile pair, so that Υ a , y ˉ ( μ ) = ∑ b τ ( b ) ∫ D ( S ) ν K ⋆ ( d ν ∣ μ , a , b ) \Upsilon_{a,\bar{y}}(\mu)=\sum_b\tau(b)\int_{\mathcal{D}(S)}\nu K_{\star}(\mathrm{d}\nu\mid\mu,a,b) Υ a , y ˉ ( μ ) = ∑ b τ ( b ) ∫ D ( S ) ν K ⋆ ( d ν ∣ μ , a , b ) ; and write
M W r ⋆ ⋆ ( μ ) a , b : = ∫ D ( S ) W r ⋆ ( ν ) K ⋆ ( d ν ∣ μ , a , b ) M_{W_r^{\star}}^{\star}(\mu)_{a,b}:=\int_{\mathcal{D}(S)}W_r^{\star}(\nu)K_{\star}(\mathrm{d}\nu\mid\mu,a,b) M W r ⋆ ⋆ ( μ ) a , b := ∫ D ( S ) W r ⋆ ( ν ) K ⋆ ( d ν ∣ μ , a , b )
for the matrix of r r r -stage continuation values at μ \mu μ , matching the notation M f ⋆ M_f^{\star} M f ⋆ of Appendix § 16 at f = W r ⋆ f=W_r^{\star} f = W r ⋆ . Jensen’s inequality, applied to the concave f f f over the mixture displayed above, and then the induction hypothesis f ≥ W r ⋆ f\geq W_r^{\star} f ≥ W r ⋆ , give for every a ∈ P min a\in\mathcal{P}_{\min} a ∈ P m i n
f ( Υ a , y ˉ ( μ ) ) ≥ ∑ b τ ( b ) ∫ f ( ν ) K ⋆ ( d ν ∣ μ , a , b ) ≥ ∑ b τ ( b ) ∫ W r ⋆ ( ν ) K ⋆ ( d ν ∣ μ , a , b ) = ∑ b τ ( b ) M W r ⋆ ⋆ ( μ ) a , b . (72) \begin{aligned}
f(\Upsilon_{a,\bar{y}}(\mu))\geq\sum_b\tau(b)\int f(\nu)K_{\star}(\mathrm{d}\nu\mid\mu,a,b) \\
&\geq\sum_b\tau(b)\int W_r^{\star}(\nu)K_{\star}(\mathrm{d}\nu\mid\mu,a,b)=\sum_b\tau(b)M_{W_r^{\star}}^{\star}(\mu)_{a,b}. \tag*{(72)}
\end{aligned} f ( Υ a , y ˉ ( μ )) ≥ b ∑ τ ( b ) ∫ f ( ν ) K ⋆ ( d ν ∣ μ , a , b ) ≥ b ∑ τ ( b ) ∫ W r ⋆ ( ν ) K ⋆ ( d ν ∣ μ , a , b ) = b ∑ τ ( b ) M W r ⋆ ⋆ ( μ ) a , b . (72)
By Lemma 38 , the minimum of x ↦ f ( Υ x , y ˉ ( μ ) ) x\mapsto f(\Upsilon_{x,\bar{y}}(\mu)) x ↦ f ( Υ x , y ˉ ( μ )) over all x ∈ X x\in\mathcal{X} x ∈ X is already attained at a pure row, so min a ∈ P min f ( Υ a , y ˉ ( μ ) ) = min x ∈ X f ( Υ x , y ˉ ( μ ) ) \min_{a\in\mathcal{P}_{\min}}f(\Upsilon_{a,\bar{y}}(\mu))=\min_{x\in\mathcal{X}}f(\Upsilon_{x,\bar{y}}(\mu)) min a ∈ P m i n f ( Υ a , y ˉ ( μ )) = min x ∈ X f ( Υ x , y ˉ ( μ )) . Since y ˉ \bar{y} y ˉ is one particular feasible element of Y \mathcal{Y} Y ,
g f − ( μ ) = max y ′ ∈ Y min x ∈ X f ( Υ x , y ′ ( μ ) ) ≥ min x ∈ X f ( Υ x , y ˉ ( μ ) ) = min a ∈ P min f ( Υ a , y ˉ ( μ ) ) . g_f^{-}(\mu)=\max_{y'\in\mathcal{Y}}\min_{x\in\mathcal{X}}f(\Upsilon_{x,y'}(\mu))\geq\min_{x\in\mathcal{X}}f(\Upsilon_{x,\bar{y}}(\mu))=\min_{a\in\mathcal{P}_{\min}}f(\Upsilon_{a,\bar{y}}(\mu)). g f − ( μ ) = y ′ ∈ Y max x ∈ X min f ( Υ x , y ′ ( μ )) ≥ x ∈ X min f ( Υ x , y ˉ ( μ )) = a ∈ P m i n min f ( Υ a , y ˉ ( μ )) .
Taking the minimum over a a a in (72) and using that τ \tau τ was chosen optimal, so that
min a ∑ b τ ( b ) M W r ⋆ ⋆ ( μ ) a , b = val ( M W r ⋆ ⋆ ( μ ) ) \min_a\sum_b\tau(b)M_{W_r^{\star}}^{\star}(\mu)_{a,b}=\operatorname{val}(M_{W_r^{\star}}^{\star}(\mu)) a min b ∑ τ ( b ) M W r ⋆ ⋆ ( μ ) a , b = val ( M W r ⋆ ⋆ ( μ ))
exactly by (52) , chains with the previous display to give
g f − ( μ ) ≥ min a ∈ P min f ( Υ a , y ˉ ( μ ) ) ≥ val ( M W r ⋆ ⋆ ( μ ) ) . g_f^{-}(\mu)\geq\min_{a\in\mathcal{P}_{\min}}f(\Upsilon_{a,\bar{y}}(\mu))\geq\operatorname{val}(M_{W_r^{\star}}^{\star}(\mu)). g f − ( μ ) ≥ a ∈ P m i n min f ( Υ a , y ˉ ( μ )) ≥ val ( M W r ⋆ ⋆ ( μ )) .
Multiplying by γ \gamma γ and adding the common current reward ( 1 − γ ) H ( μ ) (1-\gamma)\mathbf{H}(\mu) ( 1 − γ ) H ( μ ) to both sides turns the left side into U r + 1 − ( μ ) U_{r+1}^{-}(\mu) U r + 1 − ( μ ) , by definition of the averaged recursion with f = U r − f=U_r^{-} f = U r − , and the right side into W r + 1 ⋆ ( μ ) W_{r+1}^{\star}(\mu) W r + 1 ⋆ ( μ ) , by the sampled Bellman recursion of Theorem 4; this proves U r + 1 − ≥ W r + 1 ⋆ U_{r+1}^{-}\geq W_{r+1}^{\star} U r + 1 − ≥ W r + 1 ⋆ , completing the induction.
Uniform discounted tail estimates, Theorem 1 (proof in Section C) for U r + U_r^+ U r + , U r − U_r^- U r − and (39) for W r ⋆ W_r^\star W r ⋆ , bound ∥ V a v ± − U r ± ∥ ∞ \lVert V_{\mathrm{av}}^\pm-U_r^\pm\rVert_\infty ∥ V av ± − U r ± ∥ ∞ and ∥ V ⋆ − W r ⋆ ∥ ∞ \lVert V_\star-W_r^\star\rVert_\infty ∥ V ⋆ − W r ⋆ ∥ ∞ by γ r log 2 n \gamma^r\log^2 n γ r log 2 n , so both families converge uniformly, as r → ∞ r\to\infty r → ∞ , to their respective infinite-horizon values. Letting r → ∞ r\to\infty r → ∞ in U r − ≥ W r ⋆ U_r^-\ge W_r^\star U r − ≥ W r ⋆ proves V a v − ≥ V ⋆ V_{\mathrm{av}}^-\ge V_\star V av − ≥ V ⋆ for ⋆ ∈ { s p , s t } \star\in\{\mathrm{sp},\mathrm{st}\} ⋆ ∈ { sp , st } , the second inequality of the dominance chain; a uniform limit of continuous concave functions is again continuous and concave, so V a v + V_{\mathrm{av}}^+ V av + and V a v − V_{\mathrm{av}}^- V av − retain these properties. The first inequality of the dominance chain, V a v + ≥ V a v − V_{\mathrm{av}}^+\ge V_{\mathrm{av}}^- V av + ≥ V av − , is the elementary strategic minimax inequality already noted in § 2 .
When only Min has choices, the pure-profile reduction in Lemma 11 makes the av and sp finite-horizon recursions identical at every stage, so their limits agree: V a v = V s p V_{\mathrm{av}}=V_{\mathrm{sp}} V av = V sp . The dominance just proved, specialised to ⋆ = s t \star=\mathrm{st} ⋆ = st , then gives V s p = V a v ≥ V s t V_{\mathrm{sp}}=V_{\mathrm{av}}\ge V_{\mathrm{st}} V sp = V av ≥ V st . The example below establishes that no general ordering between sp and st can be added for games with an active maximiser.
Example 39 (Feedback reverses the comparison of sp and st). Neither inequality between V s p V_{\mathrm{sp}} V sp and V s t V_{\mathrm{st}} V st holds in general. The following arena, shown in Figure 4 , exhibits a strict reversal in one direction, and a second, simpler arena, shown in Figure 5 , gives the reversal in the other direction.
Nine states are arranged in three layers. The top layer u 1 , u 2 , u 3 , r u_1,u_2,u_3,r u 1 , u 2 , u 3 , r carries the initial mass, one quarter at each state, and every transition out of this layer is forced, meaning the state has only one enabled action, so it contributes no strategic choice for either player. States u 1 , u 2 , u 3 u_1,u_2,u_3 u 1 , u 2 , u 3 route deterministically to A , B , C A,B,C A , B , C respectively, and r r r splits its mass evenly
Figure 4. The first arena of Example 39, with initial distribution μ 0 = 1 4 ( δ u 1 + δ u 2 + δ u 3 + δ r ) \mu_0 = \frac{1}{4}(\delta_{u_1}+\delta_{u_2}+\delta_{u_3}+\delta_r) μ 0 = 4 1 ( δ u 1 + δ u 2 + δ u 3 + δ r ) , mass 1 / 4 1/4 1/4 on each of u 1 , u 2 , u 3 , r u_1,u_2,u_3,r u 1 , u 2 , u 3 , r and mass 0 elsewhere. A state is forced if it has exactly one enabled action, so neither player makes any choice there; a forced transition need not be deterministic, as witnessed by r r r , whose single enabled action nonetheless splits its mass evenly between A A A and B B B . Circles u 1 , u 2 , u 3 , r u_1,u_2,u_3,r u 1 , u 2 , u 3 , r are forced (drawn in black): u 1 , u 2 , u 3 u_1,u_2,u_3 u 1 , u 2 , u 3 route their mass deterministically to A , B , C A,B,C A , B , C respectively, and r r r splits its mass with probability 1 / 2 1/2 1/2 each to A A A and B B B . Rectangles A , B , C A,B,C A , B , C are Max-states (blue), each with two enabled actions, one routing deterministically to q 0 q_0 q 0 and the other to q 1 q_1 q 1 ; this is the only place in the arena where either player exercises a randomized choice. The double circles q 0 , q 1 q_0,q_1 q 0 , q 1 are absorbing: both of their actions self-loop, so once mass reaches either one it stays there for the rest of the run.
Figure 5. The second arena of Example 39, giving the reverse strict inequality V s p > V s t V_{\mathrm{sp}} > V_{\mathrm{st}} V sp > V st . The initial distribution is Dirac, μ 0 = δ s 0 \mu_0 = \delta_{s_0} μ 0 = δ s 0 . In the minimal version, s 0 s_0 s 0 has a single enabled action, a forced fair coin flip to q 0 q_0 q 0 or q 1 q_1 q 1 (solid black edges), so s 0 s_0 s 0 , like q 0 q_0 q 0 , q 1 q_1 q 1 , is forced or absorbing and neither player makes any choice anywhere in this arena; the owner of s 0 s_0 s 0 may then be assigned arbitrarily, since there is nothing to optimise there.
between A A A and B B B . The middle layer A , B , C A,B,C A , B , C belongs to the maximiser, and each of these three states has exactly two enabled actions, one routing all of its mass deterministically to the absorbing state q 0 q_0 q 0 and the other to the absorbing state q 1 q_1 q 1 ; this is the only place in the arena where a player makes a randomized choice. The bottom layer q 0 , q 1 q_0,q_1 q 0 , q 1 is absorbing under both actions, so once the mass reaches there the entropy of the distribution is frozen for the rest of the run. States with a single enabled action may be assigned to either player without affecting any value, since there is nothing to optimise there; call them Min-owned for concreteness, so that Min’s only role in this example is to watch the forced transitions happen.
For γ = 9 / 10 \gamma= 9/10 γ = 9/10 , put a = H ( 3 / 8 , 3 / 8 , 1 / 4 ) a = H(3/8,3/8,1/4) a = H ( 3/8 , 3/8 , 1/4 ) and b = h ( 3 / 8 ) b = h(3/8) b = h ( 3/8 ) . Then
V s p = 2 ( 1 − γ ) + γ ( 1 − γ ) a + γ 2 b , V s t = 2 ( 1 − γ ) + 3 2 γ ( 1 − γ ) + γ 2 V_{\mathrm{sp}} = 2(1-\gamma) + \gamma(1-\gamma)a + \gamma^2 b,\qquad V_{\mathrm{st}} = 2(1-\gamma) + \frac{3}{2}\gamma(1-\gamma) + \gamma^2 V sp = 2 ( 1 − γ ) + γ ( 1 − γ ) a + γ 2 b , V st = 2 ( 1 − γ ) + 2 3 γ ( 1 − γ ) + γ 2
Numerically V s p ≈ 1.1136066 V_{\mathrm{sp}} \approx1.1136066 V sp ≈ 1.1136066 and V s t = 1.145 V_{\mathrm{st}} = 1.145 V st = 1.145 , respectively. So, V s t > V s p V_{\mathrm{st}} > V_{\mathrm{sp}} V st > V sp .
In the opposite direction, the arena of Figure 5 , a forced fair transition from a Dirac initial state to two absorbing sinks, gives V s p = ( 1 − γ ) ⋅ 0 + γ ⋅ 1 = γ > V s t = 0 V_{\mathrm{sp}} = (1-\gamma)\cdot0 + \gamma\cdot1 = \gamma> V_{\mathrm{st}} = 0 V sp = ( 1 − γ ) ⋅ 0 + γ ⋅ 1 = γ > V st = 0 . So neither V s p ≥ V s t V_{\mathrm{sp}} \ge V_{\mathrm{st}} V sp ≥ V st nor V s p ≤ V s t V_{\mathrm{sp}} \le V_{\mathrm{st}} V sp ≤ V st holds in general.
Verification. The nine states of Figure 4 are u 1 , u 2 , u 3 , r , A , B , C , q 0 , q 1 u_1,u_2,u_3,r,A,B,C,q_0,q_1 u 1 , u 2 , u 3 , r , A , B , C , q 0 , q 1 . The first four have the forced transitions stated above, A , B , C A,B,C A , B , C belong to Max and each has an action to each q i q_i q i , and the two q i q_i q i are absorbing. All states with forced actions may belong to Min. The initial entropy is two. Under sp, the first successor is deterministically ( 3 / 8 , 3 / 8 , 1 / 4 ) (3/8,3/8,1/4) ( 3/8 , 3/8 , 1/4 ) on ( A , B , C ) (A,B,C) ( A , B , C ) and has entropy a = 11 / 4 − ( 3 / 4 ) log 2 3 a = 11/4 - (3/4)\log_2 3 a = 11/4 − ( 3/4 ) log 2 3 . A pure profile partitions these three masses between two bins. The possible mass at q 1 q_1 q 1 is
0 , 1 / 4 , 3 / 8 , 5 / 8 , 3 / 4 , 1. 0,\quad1/4,\quad3/8,\quad5/8,\quad3/4,\quad1. 0 , 1/4 , 3/8 , 5/8 , 3/4 , 1.
Its largest binary entropy is b = h ( 3 / 8 ) b = h(3/8) b = h ( 3/8 ) . A sampled mixture averages these scalar entropies and cannot improve on their maximum. Once the partition is chosen it is absorbing, proving the displayed sp value. Under st, the first successor is either ( 1 / 2 , 1 / 4 , 1 / 4 ) (1/2,1/4,1/4) ( 1/2 , 1/4 , 1/4 ) or ( 1 / 4 , 1 / 2 , 1 / 4 ) (1/4,1/2,1/4) ( 1/4 , 1/2 , 1/4 ) , each with probability 1 / 2 1/2 1/2 . Its entropy is 3 / 2 3/2 3/2 . The maximiser observes which coordinate has mass 1 / 2 1/2 1/2 and routes that coordinate alone to one sink, the other two to the other sink. The terminal distribution is uniform on two states, attaining its maximal possible entropy one. This proves the st value.
In both models the same pure stationary feedback rule suffices: among subsets of { A , B , C } \{A,B,C\} { A , B , C } choose one whose current mass is nearest to 1 / 2 1/2 1/2 , with a fixed lexicographic tie rule, and route it to q 1 q_1 q 1 ; this rule achieves the optimal value under each semantics separately (they are not the same number, only the same policy). This is a Borel rule, since it compares finitely many continuous functions. Thus the reversal is not an artefact of comparing unrelated policy classes.
Under av Max may split every current atom equally between the sinks, so its value is 2 ( 1 − γ ) + γ ( 1 − γ ) a + γ 2 ≈ 1.1505150 2(1-\gamma)+\gamma(1-\gamma)a+\gamma^{2}\approx1.1505150 2 ( 1 − γ ) + γ ( 1 − γ ) a + γ 2 ≈ 1.1505150 , consistent with the dominance chain.
For the reverse comparison, take the three states s 0 s_0 s 0 , q 0 q_0 q 0 , q 1 q_1 q 1 of Figure 5 : a Dirac initial state makes a fair transition to two distinct absorbing sinks, and, as with the forced states above, the owner of s 0 s_0 s 0 may be assigned arbitrarily since it has only one action in this minimal version. av and sp then have entropy one from time one onward, whereas st follows a Dirac distribution forever. Thus their values are d d d , d d d , 0 0 0 , for any d ∈ ( 0 , 1 ) d\in(0,1) d ∈ ( 0 , 1 ) .