Introduction
Large language model (LLM)-based agents have achieved strong results in multi-step reasoning [44 ] , code generation [33 ] , and tool use [28 ] . Organizing several such agents into a multi-agent system extends these capabilities further, as agents that debate [8 ] , role-play [20 ] , or converse [42 ] often outperform each of their members on complex tasks. This benefit, however, depends on how the collaboration is organized: which agent acts next, which artifacts it may read, and when the process terminates [46 ] . What an agent reads also determines how much it influences the others, and hence what a population of interacting agents comes to believe collectively [21 ] .
Early multi-agent systems specify this organization manually through fixed role assignments and communication pipelines [12 , 42 ] , which limits their transfer across tasks and domains. Subsequent work automates parts of the design: GPTSwarm [52 ] and G-Designer [47 ] optimize communication topologies, MasRouter [45 ] learns to route queries among agents, and ADAS [13 ] , AFlow [49 ] , and MAS [48 ] search over entire workflows. Nevertheless, these methods largely fix the organization before execution and cannot adapt it to the intermediate results produced during a task. More recently, orchestration has been cast as a sequential decision problem, in which a central orchestrator observes the evolving state of a task and is trained by reinforcement learning (RL) to decide how to proceed at every step [7 , 25 ] . The same principle has been extended to training the coordinated agents jointly with the orchestrator [22 , 50 ] , producing systems that surpass the strongest individual agents at reduced inference cost [7 , 25 ] .
Although this paradigm has shown clear empirical gains, its training objectives are inherited directly from single-agent policy optimization [30 , 31 ] , which credits each decision with its own advantage while treating all other decisions in the episode as fixed. This approximation is justified by trust-region arguments that bound its error through the per-step policy divergence [15 , 29 ] , and multi-agent extensions control the error in the same per-step manner [18 , 51 ] . Orchestration violates the premise of these arguments in two ways. First, the decisions within an episode depend on one another, since the effect of each decision on the outcome changes with the decisions made before it. A joint change to several decisions can therefore improve performance even when every single change degrades it, and an objective that adds up per-decision credits cannot detect such joint improvements. Second, an orchestrator makes many decisions in one episode. Even if each decision changes only slightly, the errors add up across decisions, so a per-step trust region no longer keeps the approximation accurate over the length of a whole episode.
To address these challenges, we develop an interaction theory of policy improvement for orchestration and propose IAPO (Interaction-Aware Policy Optimization). We show that the performance difference between two policies is a finite sum of interaction terms among decisions. Each term can be estimated from trajectories without a critic, and the existing objective is the first-order term. This objective has the correct gradient but misses all interactions between organizational decisions, such as which worker to call and which outputs it may read, so any update rule that treats these decisions separately can lose half of the possible gain. We further bound each interaction term by how often the decisions involved are made, which implies that a trust region should limit the policy change weighted by visit counts. Based on this theory, IAPO optimizes the first- and second-order terms in one pass over each trajectory and accepts an update only if a high-confidence lower bound on its gain is positive. We prove that every accepted update improves performance with high probability. On controlled orchestration tasks, IAPO reaches the best possible gain on all three tasks, while the existing objective fails on two, and IAPO never accepts a harmful update. On long-input versions of five benchmarks, against fifteen published multi-agent and workflow-search systems on the same 7B executor, IAPO’s deployed plans reach the strongest system’s accuracy with fewer tokens in every environment and, at that system’s token end_mid=1 budget, exceed it significantly on GSM8K, MATH and DROP under at least one cost weight; removing the interaction term costs 7 to 17 points of accuracy on code.
Exact Interaction Theory: We derive an exact, finite expansion of the performance difference between two orchestration policies, which identifies existing objectives as its single-decision terms and characterizes the omitted interactions through their curvature, visit-count bounds, and sample complexity.
Theoretical and Empirical Validation: We prove that ignoring interactions can forfeit half of the attainable gain and that IAPO accepts only improving updates with high probability, and we confirm on controlled tasks that IAPO recovers gains missed by existing objectives without accepting harmful updates, and on five LLM orchestration benchmarks that it matches or exceeds the strongest of fifteen baselines at lower cost.
Problem Formulation
In this section, we formulate multi-agent orchestration as an episodic MDP and define our optimization objective.
Multi-Agent Orchestration
Let Q \mathcal{Q} Q be a finite set of tasks and Y \mathcal{Y} Y a finite set of outputs, such as text responses. A task q ∈ Q q \in\mathcal{Q} q ∈ Q is drawn from a distribution D \mathcal{D} D and solved by the agents N = { 0 , 1 , … , M } \mathcal{N} = \{0, 1, \ldots, M\} N = { 0 , 1 , … , M } , where agent 0 is the orchestrator and agents j ∈ [ M ] j \in[M] j ∈ [ M ] are workers, such as LLMs, tools, and verifiers. The orchestrator solves the task through a sequence of worker calls. Let y 1 , … , y k ∈ Y y_1, \ldots, y_k \in\mathcal{Y} y 1 , … , y k ∈ Y be the outputs of the first k k k calls. In the ( k + 1 ) (k+1) ( k + 1 ) -th call, the orchestrator chooses a worker j ∈ [ M ] j \in[M] j ∈ [ M ] and an input set E ⊆ { y 1 , … , y k } \mathcal{E} \subseteq\{y_1, \ldots, y_k\} E ⊆ { y 1 , … , y k } of earlier outputs, and worker j j j reads task q q q and E \mathcal{E} E and returns a new output y k + 1 ∈ Y y_{k+1} \in\mathcal{Y} y k + 1 ∈ Y . Each call to worker j j j costs c j ≥ 0 c_j \ge0 c j ≥ 0 , and at most K K K calls are allowed. Instead of calling a worker, the orchestrator may stop, and the task and the outputs collected so far are scored by a task reward R ∈ [ 0 , 1 ] R \in[0,1] R ∈ [ 0 , 1 ] .
Orchestration MDP
We model multi-agent orchestration as an episodic MDP M ( S , A , H , P , r , d 0 ) \mathcal{M}(S, A, H, P, r, d_0) M ( S , A , H , P , r , d 0 ) , where S S S is the state space, A A A is the action space, H H H is the number of steps in each episode, P = { P t } t = 1 H P = \{P_t\}_{t=1}^{H} P = { P t } t = 1 H with P t ( ⋅ ∣ s , a ) P_t(\cdot\mid s, a) P t ( ⋅ ∣ s , a ) the distribution of the next state when action a a a is taken at state s s s in step t t t , r = { r t } t = 1 H r = \{r_t\}_{t=1}^{H} r = { r t } t = 1 H with r t : S × A → R r_t : S \times A \to\mathbb{R} r t : S × A → R the reward function at step t t t , and d 0 = D d_0 = \mathcal{D} d 0 = D the initial state distribution.
Episode dynamics. An episode starts from the task, s 1 = q ∼ D s_1 = q \sim\mathcal{D} s 1 = q ∼ D , and each agent i ∈ N i \in\mathcal{N} i ∈ N selects actions with its own policy π i \pi^i π i . After k k k calls, the interaction history h k = ( ( j 1 , E 1 , y 1 ) , … , ( j k , E k , y k ) ) h_k = ((j_1, \mathcal{E}_1, y_1), \ldots, (j_k, \mathcal{E}_k, y_k)) h k = (( j 1 , E 1 , y 1 ) , … , ( j k , E k , y k )) records each call as a triple: in the ℓ \ell ℓ -th call, worker j ℓ ∈ [ M ] j_\ell\in[M] j ℓ ∈ [ M ] receives the set E ℓ ⊆ Y ℓ − 1 \mathcal{E}_\ell\subseteq\mathcal{Y}_{\ell-1} E ℓ ⊆ Y ℓ − 1 of earlier outputs and returns y ℓ ∈ Y y_\ell\in\mathcal{Y} y ℓ ∈ Y , where Y k = { y 1 , … , y k } \mathcal{Y}_k = \{y_1, \ldots, y_k\} Y k = { y 1 , … , y k } . Each call takes two steps.
Orchestrator step. At state s t = ( q , h k ) s_t = (q, h_k) s t = ( q , h k ) , the orchestrator acts, i t = 0 i_t = 0 i t = 0 . It decides whether to continue, and if so, which worker j ∈ [ M ] j \in[M] j ∈ [ M ] to call and which outputs E ⊆ Y k \mathcal{E} \subseteq\mathcal{Y}_k E ⊆ Y k of earlier workers to give it. Formally, it samples a t ∼ π 0 ( ⋅ ∣ s t ) a_t \sim\pi^0(\cdot\mid s_t) a t ∼ π 0 ( ⋅ ∣ s t ) from:
A ( s t ) = { ( j , E ) : j ∈ [ M ] , E ⊆ Y k } ∪ { s t o p } . \mathcal{A}(s_t) = \{(j,\mathcal{E}) : j \in[M], \mathcal{E} \subseteq\mathcal{Y}_k\} \cup\{\mathrm{stop}\}. A ( s t ) = {( j , E ) : j ∈ [ M ] , E ⊆ Y k } ∪ { stop } .
A call ( j , E ) (j,\mathcal{E}) ( j , E ) receives reward r t = − λ c j r_t = -\lambda c_j r t = − λ c j with cost coefficient λ ≥ 0 \lambda\ge0 λ ≥ 0 and moves the state to s t + 1 = ( q , h k , j , E ) s_{t+1} = (q, h_k, j, \mathcal{E}) s t + 1 = ( q , h k , j , E ) , and stop receives the task reward r t = R ( q , h k ) r_t = R(q, h_k) r t = R ( q , h k ) and ends the episode.
Worker step. Worker j j j then receives the call from the orchestrator at state s t + 1 = ( q , h k , j , E ) s_{t+1} = (q, h_k, j, \mathcal{E}) s t + 1 = ( q , h k , j , E ) , so i t + 1 = j i_{t+1} = j i t + 1 = j . It reads the task q q q and the outputs in E \mathcal{E} E and produces a new output. Formally, it samples a t + 1 = y ∼ π j ( ⋅ ∣ q , E ) a_{t+1} = y \sim\pi^j(\cdot\mid q,\mathcal{E}) a t + 1 = y ∼ π j ( ⋅ ∣ q , E ) from Y \mathcal{Y} Y and receives r t + 1 = 0 r_{t+1} = 0 r t + 1 = 0 . The output is added to the history, h k + 1 = ( h k , ( j , E , y ) ) h_{k+1} = (h_k, (j,\mathcal{E},y)) h k + 1 = ( h k , ( j , E , y )) , and control returns to the orchestrator at state s t + 2 = ( q , h k + 1 ) s_{t+2} = (q, h_{k+1}) s t + 2 = ( q , h k + 1 ) .
The joint policy is therefore π t ( a ∣ s t ) = π i t ( a ∣ s t ) \pi_t(a \mid s_t) = \pi^{i_t}(a \mid s_t) π t ( a ∣ s t ) = π i t ( a ∣ s t ) , and every trajectory ( s 1 , a 1 , … , s H , a H ) (s_1, a_1, \ldots, s_H, a_H) ( s 1 , a 1 , … , s H , a H ) is generated by π \pi π .
Policies and values. A policy π = { π t } t = 1 H \pi= \{\pi_t\}_{t=1}^{H} π = { π t } t = 1 H maps each state s s s to a distribution π t ( ⋅ ∣ s ) \pi_t(\cdot\mid s) π t ( ⋅ ∣ s ) over A ( s ) \mathcal{A}(s) A ( s ) . The value and action-value functions of π \pi π are:
V t π ( s ) = E π [ ∑ t ′ = t H r t ′ ( s t ′ , a t ′ ) | s t = s ] , V_t^\pi(s) = \mathbb{E}_\pi\left[\sum_{t'=t}^{H} r_{t'}(s_{t'}, a_{t'}) \,\middle|\, s_t = s\right], V t π ( s ) = E π [ t ′ = t ∑ H r t ′ ( s t ′ , a t ′ ) s t = s ] ,
Q t π ( s , a ) = E π [ ∑ t ′ = t H r t ′ ( s t ′ , a t ′ ) | s t = s , a t = a ] , Q_t^\pi(s,a) = \mathbb{E}_\pi\left[\sum_{t'=t}^{H} r_{t'}(s_{t'}, a_{t'}) \,\middle|\, s_t = s, a_t = a\right], Q t π ( s , a ) = E π [ t ′ = t ∑ H r t ′ ( s t ′ , a t ′ ) s t = s , a t = a ] ,
and the advantage function is A t π = Q t π − V t π A_t^\pi= Q_t^\pi- V_t^\pi A t π = Q t π − V t π . The return-to-go of a trajectory from step t t t is G t = ∑ t ′ = t H r t ′ ( s t ′ , a t ′ ) G_t = \sum_{t'=t}^{H} r_{t'}(s_{t'}, a_{t'}) G t = ∑ t ′ = t H r t ′ ( s t ′ , a t ′ ) .
Learning Objective
Our goal is to solve each task with high reward at low cost. We keep the worker policies π 1 , … , π M \pi^1,\ldots,\pi^M π 1 , … , π M fixed and optimize the orchestrator policy π 0 \pi^0 π 0 to maximize the expected task reward minus the cost of calls:
max π 0 J ( π ) : = E π [ R ( q , h ) − λ ∑ ( j , E , y ) ∈ h c j ] . (1) \max_{\pi^0} J(\pi) := \mathbb{E}_\pi\left[R(q,h) - \lambda\sum_{(j,E,y)\in h} c_j\right]. \tag*{(1)} π 0 max J ( π ) := E π R ( q , h ) − λ ( j , E , y ) ∈ h ∑ c j . (1)
where h h h is the interaction history when the episode ends. By the rewards defined in § 2.2 , J ( π ) = E π [ ∑ t = 1 H r t ( s t , a t ) ] J(\pi) = \mathbb{E}_\pi[\sum_{t=1}^{H} r_t(s_t,a_t)] J ( π ) = E π [ ∑ t = 1 H r t ( s t , a t )] is the expected return of the MDP, so (1) is a standard RL problem in π 0 \pi^0 π 0 . Our analysis also covers the case where some workers are trained.
IAPO
To improve the reward–cost objective (1), IAPO constructs policy updates from an exact interaction expansion (§ 3.1 ), controls their approximation error (§ 3.2 ), and certifies their performance using independent data (§ 3.3 ). Figure 1 gives an overview.
Figure 1. Overview of IAPO. (a) Orchestration decisions depend on one another: a new earlier worker helps only if a later call may read its output, so neither change helps alone. (b) IAPO credits each orchestrator step and every pair of steps in one pass over each training trajectory, and fits one bounded update per step size. (c) With the candidates fixed, IAPO computes two lower bounds on the gain from fresh trajectories, keeps the larger one, and updates the policy only if it is positive.
An Interaction View of Policy Improvement
An orchestrator decision affects subsequent worker outputs, actions, and call costs. To analyze the resulting interactions, let π \pi π be the reference policy and π ′ \pi' π ′ a candidate that updates only the orchestrator. We impose the following condition to express J ( π ′ ) − J ( π ) J(\pi') - J(\pi) J ( π ′ ) − J ( π ) using trajectories from π \pi π .
Assumption 3.1 (Overlap). There exists W ≥ 1 W \ge1 W ≥ 1 such that π t ′ ( a ∣ s ) ≤ W π t ( a ∣ s ) \pi'_t(a \mid s) \le W\pi_t(a \mid s) π t ′ ( a ∣ s ) ≤ W π t ( a ∣ s ) for every t ∈ [ H ] t \in[H] t ∈ [ H ] , every state s s s reachable under π \pi π , and every a ∈ A ( s ) a \in\mathcal{A}(s) a ∈ A ( s ) .
For a trajectory sampled from π \pi π , define the likelihood ratio ρ t = π t ′ ( a t ∣ s t ) / π t ( a t ∣ s t ) \rho_t = \pi'_t(a_t \mid s_t)/\pi_t(a_t \mid s_t) ρ t = π t ′ ( a t ∣ s t ) / π t ( a t ∣ s t ) . Since worker policies and transitions are fixed, their ratios are 1, and the likelihood ratio of a trajectory prefix through t t t is ∏ t ′ = 1 t ρ t ′ \prod_{t'=1}^{t}\rho_{t'} ∏ t ′ = 1 t ρ t ′ . Importance sampling gives the exact improvement in J J J :
J ( π ′ ) − J ( π ) = E π [ ∑ t = 1 H ( ∏ t ′ = 1 t ρ t ′ − 1 ) r t ( s t , a t ) ] . (2) J(\pi') - J(\pi) = \mathbb{E}_\pi\left[\sum_{t=1}^{H}\left(\prod_{t'=1}^{t}\rho_{t'} - 1\right)r_t(s_t,a_t)\right]. \tag*{(2)} J ( π ′ ) − J ( π ) = E π [ t = 1 ∑ H ( t ′ = 1 ∏ t ρ t ′ − 1 ) r t ( s t , a t ) ] . (2)
Expanding the likelihood-ratio product in (2) , with all summation indices ranging over [ t ] [t] [ t ] , gives:
∏ t ′ = 1 t ρ t ′ − 1 = ∑ t 1 ( ρ t 1 − 1 ) + ∑ t 1 < t 2 ( ρ t 1 − 1 ) ( ρ t 2 − 1 ) + ∑ t 1 < t 2 < t 3 ( ρ t 1 − 1 ) ( ρ t 2 − 1 ) ( ρ t 3 − 1 ) + ∑ m = 4 t ∑ t 1 < ⋯ < t m ( ρ t 1 − 1 ) ⋯ ( ρ t m − 1 ) . \begin{aligned}
\prod_{t'=1}^{t}\rho_{t'} - 1
&= \sum_{t_1}(\rho_{t_1}-1) + \sum_{t_1<t_2}(\rho_{t_1}-1)(\rho_{t_2}-1) \\
&\quad+ \sum_{t_1<t_2<t_3}(\rho_{t_1}-1)(\rho_{t_2}-1)(\rho_{t_3}-1) \\
&\quad+ \sum_{m=4}^{t}\sum_{t_1<\cdots<t_m}(\rho_{t_1}-1)\cdots(\rho_{t_m}-1).
\end{aligned} t ′ = 1 ∏ t ρ t ′ − 1 = t 1 ∑ ( ρ t 1 − 1 ) + t 1 < t 2 ∑ ( ρ t 1 − 1 ) ( ρ t 2 − 1 ) + t 1 < t 2 < t 3 ∑ ( ρ t 1 − 1 ) ( ρ t 2 − 1 ) ( ρ t 3 − 1 ) + m = 4 ∑ t t 1 < ⋯ < t m ∑ ( ρ t 1 − 1 ) ⋯ ( ρ t m − 1 ) .
Each factor ρ t − 1 \rho_t-1 ρ t − 1 represents a decision-level policy change, while products capture interactions across decisions. Rearranging (2) by interaction order and replacing returns by advantages yields the exact decomposition below.
Theorem 3.2 (Interaction expansion). Under Assumption 3.1:
J ( π ′ ) − J ( π ) = ∑ m = 1 H J ( m ) (3) J(\pi') - J(\pi) = \sum_{m=1}^{H} J^{(m)} \tag*{(3)} J ( π ′ ) − J ( π ) = m = 1 ∑ H J ( m ) (3) where the order-m m m interaction is:
J ( m ) = E π [ ∑ 1 ≤ t 1 < ⋯ < t m ≤ H ( ρ t 1 − 1 ) ⋯ ( ρ t m − 1 ) × A t m π ( s t m , a t m ) ] . (4) J^{(m)} = \mathbb{E}_{\pi}\left[\sum_{1\le t_1<\cdots<t_m\le H}(\rho_{t_1}-1)\cdots(\rho_{t_m}-1) \times A_{t_m}^{\pi}(s_{t_m},a_{t_m})\right]. \tag*{(4)} J ( m ) = E π [ 1 ≤ t 1 < ⋯ < t m ≤ H ∑ ( ρ t 1 − 1 ) ⋯ ( ρ t m − 1 ) × A t m π ( s t m , a t m ) ] . (4) Moreover, each J ( m ) J^{(m)} J ( m ) remains unchanged if A t m π A_{t_m}^{\pi} A t m π is replaced by G t m − b ( s t m ) G_{t_m}-b(s_{t_m}) G t m − b ( s t m ) for any bounded function b b b .
The first-order term J ( 1 ) J^{(1)} J ( 1 ) recovers the advantage-weighted objective underlying PPO-style optimization ([30 ] ), and higher orders capture the effects of changing several decisions along the same trajectory.
Frozen workers contribute no likelihood-ratio changes, but repeated orchestrator decisions still interact. Their contribution appears in policy curvature.
Corollary 3.3 (Interaction-induced curvature). Let π θ \pi_\theta π θ be a twice continuously differentiable orchestrator policy, with π θ 0 = π \pi_{\theta^0}=\pi π θ 0 = π having full support and all worker policies fixed. Let J ( m ) ( θ ) J^{(m)}(\theta) J ( m ) ( θ ) denote the interaction terms for π ′ = π θ \pi'=\pi_\theta π ′ = π θ . At θ = θ 0 \theta=\theta^0 θ = θ 0 :
∇ θ J ( π θ ) = ∇ θ J ( 1 ) ( θ ) , \nabla_\theta J(\pi_\theta)=\nabla_\theta J^{(1)}(\theta), ∇ θ J ( π θ ) = ∇ θ J ( 1 ) ( θ ) , ∇ θ 2 J ( π θ ) − ∇ θ 2 J ( 1 ) ( θ ) = ∇ θ 2 J ( 2 ) ( θ ) . \nabla_\theta^2J(\pi_\theta)-\nabla_\theta^2J^{(1)}(\theta)=\nabla_\theta^2J^{(2)}(\theta). ∇ θ 2 J ( π θ ) − ∇ θ 2 J ( 1 ) ( θ ) = ∇ θ 2 J ( 2 ) ( θ ) .
Thus, the first-order objective has the correct local gradient but misses interaction-induced curvature that can affect the quality of finite policy updates.
Interaction-Aware Policy Optimization
Second-order truncation. To obtain a tractable update, we retain the first two orders and bound the remainder. With only the orchestrator updated, define:
β = 2 ( K + 1 ) max s TV ( π θ 0 ( ⋅ ∣ s ) , π θ 0 ( ⋅ ∣ s ) ) , \beta=2(K+1)\max_s\operatorname{TV}\left(\pi^{\theta^0}(\cdot\mid s),\pi^{\theta^0}(\cdot\mid s)\right), β = 2 ( K + 1 ) s max TV ( π θ 0 ( ⋅ ∣ s ) , π θ 0 ( ⋅ ∣ s ) ) ,
which upper-bounds the cumulative magnitude of policy changes across repeated orchestrator decisions. Let G ˉ = 1 + λ K max j c j \bar{G}=1+\lambda K\max_j c_j G ˉ = 1 + λ K max j c j bound the total absolute reward of a trajectory.
Theorem 3.4 (Visit-count bound). Under Assumption 3.1, with workers fixed, for every integer r ≥ 1 r\ge1 r ≥ 1 :
∣ J ( π ′ ) − J ( π ) − ∑ m = 1 r J ( m ) ∣ ≤ G ˉ ∑ m > r β m m ! . (5) \left|J(\pi')-J(\pi)-\sum_{m=1}^{r}J^{(m)}\right|\le\bar{G}\sum_{m>r}\frac{\beta^m}{m!}. \tag*{(5)} J ( π ′ ) − J ( π ) − m = 1 ∑ r J ( m ) ≤ G ˉ m > r ∑ m ! β m . (5)
Taking r = 2 r=2 r = 2 yields:
J ( π ′ ) − J ( π ) ≥ J ( 1 ) + J ( 2 ) − G ˉ ( e β − 1 − β − β 2 2 ) . (6) J(\pi')-J(\pi)\ge J^{(1)}+J^{(2)}-\bar{G}\left(e^\beta-1-\beta-\frac{\beta^2}{2}\right). \tag*{(6)} J ( π ′ ) − J ( π ) ≥ J ( 1 ) + J ( 2 ) − G ˉ ( e β − 1 − β − 2 β 2 ) . (6)
First-order truncation incurs O ( G ˉ β 2 ) O(\bar{G}\beta^2) O ( G ˉ β 2 ) error, and second-order truncation reduces this to O ( G ˉ β 3 ) O(\bar{G}\beta^3) O ( G ˉ β 3 ) . We therefore maximize J ( 1 ) + J ( 2 ) J^{(1)}+J^{(2)} J ( 1 ) + J ( 2 ) while controlling β \beta β , following trust-region policy improvement ([29 ] ).
Bounded policy update. Clipping the objective does not bound the policy itself. We instead enforce the visit-count constraint by construction.
For a learned score f θ ( s , a ) ∈ [ − 1 , 1 ] f_\theta(s,a)\in[-1,1] f θ ( s , a ) ∈ [ − 1 , 1 ] , define its reference-policy-centered form:
u θ ( s , a ) = 1 2 ( f θ ( s , a ) − ∑ a ′ π ( a ′ ∣ s ) f θ ( s , a ′ ) ) . u_\theta(s,a)=\frac{1}{2}\left(f_\theta(s,a)-\sum_{a'}\pi(a'\mid s)f_\theta(s,a')\right). u θ ( s , a ) = 2 1 ( f θ ( s , a ) − a ′ ∑ π ( a ′ ∣ s ) f θ ( s , a ′ ) ) .
Algorithm 1 Interaction-Aware Policy Optimization
Require: Initial policy $\pi$, score family $f_\theta$, grid $\Lambda$, batch sizes $N$, $n \ge 2$, confidence level $\delta$
1: for $\kappa = 1, 2, \ldots$ do
2: Collect $N$ training trajectories $\mathcal{T}$ under $\pi$, and add those of earlier rounds reweighted to $\pi$
3: $b \leftarrow$ mean of $G_1$ over the $N$ new trajectories
4: for each $\eta \in \Lambda$ do
5: Fit $\theta_\eta$ by approximately maximizing the empirical mean of $Z_2$ with baseline $b$ on $\mathcal{T}$
6: $\pi'_\eta \leftarrow \pi'_{\theta_\eta,\eta}$
7: end for
8: Collect $n$ fresh validation trajectories $\mathcal{V}$ under $\pi$
9: $\delta_\kappa \leftarrow \delta/2^\kappa$
10: for each $\eta \in \Lambda$ do
11: Compute $L_\eta$ on $\mathcal{V}$ using {xref}`(13)`, with the fixed $b$, at level $\delta_\kappa$
12: end for
13: $\eta^\star \leftarrow \arg\max_{\eta \in \Lambda} L_\eta$
14: if $L_{\eta^\star} \le 0$ then
15: return $\pi$
16: end if
17: $\pi \leftarrow \pi'_{\eta^\star}$
18: end for
This gives ∑ a π ( a ∣ s ) u θ ( s , a ) = 0 \sum_a \pi(a \mid s)u_\theta(s,a)=0 ∑ a π ( a ∣ s ) u θ ( s , a ) = 0 and ∣ u θ ( s , a ) ∣ ≤ 1 \lvert u_\theta(s,a)\rvert\le1 ∣ u θ ( s , a )∣ ≤ 1 . For η ∈ [ 0 , 1 ] \eta\in[0,1] η ∈ [ 0 , 1 ] , let:
π θ , η ′ ( a ∣ s ) = π ( a ∣ s ) ( 1 + η u θ ( s , a ) ) . (7) \pi'_{\theta,\eta}(a \mid s)=\pi(a \mid s)(1+\eta u_\theta(s,a)). \tag*{(7)} π θ , η ′ ( a ∣ s ) = π ( a ∣ s ) ( 1 + η u θ ( s , a )) . (7)
Only the orchestrator policy is updated, and every worker policy remains fixed.
Proposition 3.5 (Bounded ratios). For any θ \theta θ and η ∈ [ 0 , 1 ] \eta\in[0,1] η ∈ [ 0 , 1 ] , (7) defines a valid action distribution at every orchestrator state s s s , satisfying π θ , η ′ ( a ∣ s ) ≥ π ( a ∣ s ) \pi'_{\theta,\eta}(a \mid s) \ge\pi(a \mid s) π θ , η ′ ( a ∣ s ) ≥ π ( a ∣ s ) . Moreover, on the support of π \pi π , its likelihood ratio satisfies ∣ ρ t − 1 ∣ ≤ η \lvert\rho_t-1\rvert\le\eta ∣ ρ t − 1 ∣ ≤ η , and its total variation distance from π ( ⋅ ∣ s ) \pi(\cdot\mid s) π ( ⋅ ∣ s ) is at most η / 4 \eta/4 η /4 .
Thus, overlap holds with W = 1 + η W=1+\eta W = 1 + η and, with at most K + 1 K+1 K + 1 orchestrator decisions, β ≤ ( K + 1 ) η / 2 \beta\le(K+1)\eta/2 β ≤ ( K + 1 ) η /2 .
Interaction-aware objective. Let u t = u θ ( s t , a t ) u_t=u_\theta(s_t,a_t) u t = u θ ( s t , a t ) at orchestrator steps and 0 0 0 at worker steps. Since ρ t − 1 = η u t \rho_t-1=\eta u_t ρ t − 1 = η u t , Theorem 3.2 gives:
J ( 1 ) + J ( 2 ) = η μ 1 ( θ ) + η 2 μ 2 ( θ ) , (8) J^{(1)}+J^{(2)}=\eta\mu_1(\theta)+\eta^2\mu_2(\theta), \tag*{(8)} J ( 1 ) + J ( 2 ) = η μ 1 ( θ ) + η 2 μ 2 ( θ ) , (8)
where:
μ 1 ( θ ) = E π [ ∑ t u t G t ] , μ 2 ( θ ) = E π [ ∑ t ′ < t u t ′ u t G t ] . \mu_1(\theta)=\mathbb{E}_\pi\left[\sum_t u_tG_t\right],\qquad\mu_2(\theta)=\mathbb{E}_\pi\left[\sum_{t'<t}u_{t'}u_tG_t\right]. μ 1 ( θ ) = E π [ t ∑ u t G t ] , μ 2 ( θ ) = E π [ t ′ < t ∑ u t ′ u t G t ] .
Both terms can be estimated from reference-policy trajectories. We subtract a scalar baseline b b b , estimated from training data and held fixed during validation. By Theorem 3.2, it does not alter the expectation. Grouping pairwise terms by their later decision gives:
Z 2 = ∑ t = 1 H ( ρ t − 1 ) ( 1 + ∑ t ′ < t ( ρ t ′ − 1 ) ) ( G t − b ) . (9) Z_2=\sum_{t=1}^{H}(\rho_t-1)\left(1+\sum_{t'<t}(\rho_{t'}-1)\right)(G_t-b). \tag*{(9)} Z 2 = t = 1 ∑ H ( ρ t − 1 ) ( 1 + t ′ < t ∑ ( ρ t ′ − 1 ) ) ( G t − b ) . (9)
For fixed b b b , E π [ Z 2 ] = J ( 1 ) + J ( 2 ) \mathbb{E}_\pi[Z_2]=J^{(1)}+J^{(2)} E π [ Z 2 ] = J ( 1 ) + J ( 2 ) . In particular, the return-to-go G t G_t G t includes the terminal task reward and all subsequent worker-call costs through the MDP rewards in § 2.2 . Thus, optimizing Z 2 Z_2 Z 2 approximates improvement in the original reward–cost objective (1), not a separate task-accuracy objective. A prefix sum computes Z 2 Z_2 Z 2 and its gradients in O ( H ) O(H) O ( H ) trajectory operations (Algorithm 2 in Appendix § 9 .§ 3 ).
The bounded update admits the explicit truncation bound:
ϵ ( η ) = G ‾ ∑ m = 3 K + 1 ( K + 1 m ) ( η 2 ) m , (10) \epsilon(\eta)=\overline{G}\sum_{m=3}^{K+1}\binom{K+1}{m}\left(\frac{\eta}{2}\right)^m, \tag*{(10)} ϵ ( η ) = G m = 3 ∑ K + 1 ( m K + 1 ) ( 2 η ) m , (10)
so that:
J ( π θ , η ′ ) − J ( π ) ≥ J ( 1 ) + J ( 2 ) − ϵ ( η ) . J(\pi'_{\theta,\eta})-J(\pi)\ge J^{(1)}+J^{(2)}-\epsilon(\eta). J ( π θ , η ′ ) − J ( π ) ≥ J ( 1 ) + J ( 2 ) − ϵ ( η ) .
For fixed η \eta η , maximizing the second-order objective therefore maximizes a valid lower bound on policy improvement.
Theorem 3.6 (Existence of improving steps). Fix θ \theta θ and write μ 1 = μ 1 ( θ ) \mu_1 = \mu_1(\theta) μ 1 = μ 1 ( θ ) and μ 2 = μ 2 ( θ ) \mu_2 = \mu_2(\theta) μ 2 = μ 2 ( θ ) .
(i) First-order gain. If μ 1 > 0 \mu_1 > 0 μ 1 > 0 , then any η ∈ ( 0 , 1 ] \eta\in(0, 1] η ∈ ( 0 , 1 ] with η ( ∣ μ 2 ∣ + ϵ ( 1 ) ) ≤ μ 1 / 2 \eta(|\mu_2| + \epsilon(1)) \le\mu_1/2 η ( ∣ μ 2 ∣ + ϵ ( 1 )) ≤ μ 1 /2 guarantees:
J ( π θ , η ′ ) − J ( π ) ≥ η μ 1 / 2. J(\pi'_{\theta,\eta}) - J(\pi) \ge\eta\mu_1/2. J ( π θ , η ′ ) − J ( π ) ≥ η μ 1 /2. (ii) Second-order gain. If μ 1 = 0 < μ 2 \mu_1 = 0 < \mu_2 μ 1 = 0 < μ 2 , then any η ∈ ( 0 , 1 ] \eta\in(0, 1] η ∈ ( 0 , 1 ] with ϵ ( 1 ) ≤ μ 2 / 2 \epsilon(1) \le\mu_2/2 ϵ ( 1 ) ≤ μ 2 /2 guarantees:
J ( π θ , η ′ ) − J ( π ) ≥ η 2 μ 2 / 2. J(\pi'_{\theta,\eta}) - J(\pi) \ge\eta^2\mu_2/2. J ( π θ , η ′ ) − J ( π ) ≥ η 2 μ 2 /2. Furthermore, if the score class can represent f θ ( s t , a ) = Q t π ( s t , a ) / G ˉ f_\theta(s_t,a) = Q^\pi_t(s_t,a)/\bar{G} f θ ( s t , a ) = Q t π ( s t , a ) / G ˉ , then:
μ 1 = 1 2 G ˉ E π [ ∑ t : i t = 0 A t π ( s t , a t ) 2 ] . \mu_1 = \frac{1}{2\bar{G}}\mathbb{E}_\pi\left[\sum_{t:i_t=0} A^\pi_t(s_t,a_t)^2\right]. μ 1 = 2 G ˉ 1 E π [ t : i t = 0 ∑ A t π ( s t , a t ) 2 ] . which is positive whenever the advantage is nonzero with positive probability at an orchestrator decision.
Pairwise interactions can yield improvement even when first-order gain vanishes (Appendix § 9 .8). Let Λ ⊂ ( 0 , 1 ] \Lambda\subset(0, 1] Λ ⊂ ( 0 , 1 ] be a finite step-size grid. IAPO optimizes θ \theta θ separately for each η ∈ Λ \eta\in\Lambda η ∈ Λ using empirical Z 2 Z_2 Z 2 , then independently validates the resulting candidates. The fit may also reuse the trajectories of earlier rounds, reweighted to π \pi π by their known likelihood ratios, since the certificate uses only fresh validation trajectories.
Certified Update Selection
Sampling error and higher-order truncation can make an apparently promising update harmful. For each η ∈ Λ \eta\in\Lambda η ∈ Λ , let π η ′ \pi'_\eta π η ′ be the trained candidate. After fixing all candidates, we draw n n n independent validation trajectories under the reference policy π \pi π .
Following high-confidence policy improvement (Thomas et al., [35 ] ), we use Z 2 Z_2 Z 2 with truncation error ϵ ( η ) \epsilon(\eta) ϵ ( η ) and an exact importance-sampling statistic. Using the same fixed baseline b b b , (2) gives:
Z I S = ∑ t = 1 H ( ∏ t ′ ≤ t ρ t ′ − 1 ) r t ( s t , a t ) − b ( ∏ t = 1 H ρ t − 1 ) . (11) Z_{\mathrm{IS}} = \sum_{t=1}^{H}\left(\prod_{t'\le t}\rho_{t'}-1\right)r_t(s_t,a_t)-b\left(\prod_{t=1}^{H}\rho_t-1\right). \tag*{(11)} Z IS = t = 1 ∑ H ( t ′ ≤ t ∏ ρ t ′ − 1 ) r t ( s t , a t ) − b ( t = 1 ∏ H ρ t − 1 ) . (11)
The baseline correction has zero expectation, so E π [ Z I S ] = J ( π η ′ ) − J ( π ) \mathbb{E}_\pi[Z_{\mathrm{IS}}] = J(\pi'_\eta) - J(\pi) E π [ Z IS ] = J ( π η ′ ) − J ( π ) , though cumulative importance weights may increase its variance, which the validation sample size must absorb.
Lemma E.3 bounds both statistics:
B 2 , η = ( G ˉ + ∣ b ∣ ) ( ( K + 1 ) η + ( K + 1 2 ) η 2 ) , B_{2,\eta} = (\bar{G}+|b|)\left((K+1)\eta+\binom{K+1}{2}\eta^2\right), B 2 , η = ( G ˉ + ∣ b ∣ ) ( ( K + 1 ) η + ( 2 K + 1 ) η 2 ) ,
B I S , η = ( G ˉ + ∣ b ∣ ) ( ( 1 + η ) K + 1 − 1 ) . B_{\mathrm{IS},\eta} = (\bar{G}+|b|)\left((1+\eta)^{K+1}-1\right). B IS , η = ( G ˉ + ∣ b ∣ ) ( ( 1 + η ) K + 1 − 1 ) .
Since b b b is fixed before validation, these bounds apply conditionally on training. For either statistic Z Z Z , let Z ˉ \bar{Z} Z ˉ and σ ^ Z \hat{\sigma}_Z σ ^ Z be its validation mean and sample standard deviation, and B Z , η B_{Z,\eta} B Z , η its absolute bound. The empirical Bernstein inequality (Maurer and Pontil, [24 ] ) gives the confidence radius:
ξ η ( Z ) = σ ^ Z 2 log ( 8 ∣ Λ ∣ / δ ) n + 14 B Z , η log ( 8 ∣ Λ ∣ / δ ) 3 ( n − 1 ) . (12) \xi_\eta(Z)=\hat{\sigma}_Z\sqrt{\frac{2\log(8|\Lambda|/\delta)}{n}}+\frac{14B_{Z,\eta}\log(8|\Lambda|/\delta)}{3(n-1)}. \tag*{(12)} ξ η ( Z ) = σ ^ Z n 2 log ( 8∣Λ∣/ δ ) + 3 ( n − 1 ) 14 B Z , η log ( 8∣Λ∣/ δ ) . (12)
We take the larger of the two valid lower confidence bounds:
L η = max { Z ˉ 2 − ξ η ( Z 2 ) − ϵ ( η ) , Z ˉ I S − ξ η ( Z I S ) } . (13) L_\eta=\max\left\{\bar{Z}_2-\xi_\eta(Z_2)-\epsilon(\eta),\ \bar{Z}_{\mathrm{IS}}-\xi_\eta(Z_{\mathrm{IS}})\right\}. \tag*{(13)} L η = max { Z ˉ 2 − ξ η ( Z 2 ) − ϵ ( η ) , Z ˉ IS − ξ η ( Z IS ) } . (13)
The first accounts for truncation error, while the second uses unbiased importance sampling. Their maximum remains a lower confidence bound on true policy improvement.
IAPO selects the largest positive certificate, retaining π \pi π if all certificates are nonpositive. Let π ^ \hat{\pi} π ^ be the selected policy and define:
c η = min { ξ η ( Z 2 ) + ϵ ( η ) , ξ η ( Z I S ) } . c_\eta=\min\left\{\xi_\eta(Z_2)+\epsilon(\eta),\xi_\eta(Z_{\mathrm{IS}})\right\}. c η = min { ξ η ( Z 2 ) + ϵ ( η ) , ξ η ( Z IS ) } .
Theorem 3.7 (Certified selection). With probability at least 1 − δ 1-\delta 1 − δ , the following statements hold simultaneously.
(i) Safe selection. For every η ∈ Λ \eta\in\Lambda η ∈ Λ :
L η ≤ J ( π η ′ ) − J ( π ) . L_\eta\le J(\pi'_\eta)-J(\pi). L η ≤ J ( π η ′ ) − J ( π ) . Consequently:
J ( π ^ ) ≥ J ( π ) . J(\hat{\pi}) \ge J(\pi). J ( π ^ ) ≥ J ( π ) . (ii) Oracle inequality. The selected policy satisfies:
J ( π ^ ) − J ( π ) ≥ max { 0 , max η ∈ Λ [ J ( π η ′ ) − J ( π ) − 2 c η ] } . J(\hat{\pi})-J(\pi) \ge\max\left\{0,\max_{\eta\in\Lambda}\left[J(\pi'_\eta)-J(\pi)-2c_\eta\right]\right\}. J ( π ^ ) − J ( π ) ≥ max { 0 , η ∈ Λ max [ J ( π η ′ ) − J ( π ) − 2 c η ] } . (iii) Detection of improvement. If some η ∈ Λ \eta\in\Lambda η ∈ Λ satisfies J ( π η ′ ) > J ( π ) J(\pi'_\eta)>J(\pi) J ( π η ′ ) > J ( π ) and
n − 1 > 256 B I S , η 2 log ( 8 ∣ Λ ∣ / δ ) [ J ( π η ′ ) − J ( π ) ] 2 , n-1>\frac{256B_{\mathrm{IS},\eta}^{2}\log(8|\Lambda|/\delta)}{\left[J(\pi'_\eta)-J(\pi)\right]^2}, n − 1 > [ J ( π η ′ ) − J ( π ) ] 2 256 B IS , η 2 log ( 8∣Λ∣/ δ ) , then π ^ ≠ π \hat{\pi}\ne\pi π ^ = π .
Across successive rounds κ \kappa κ , assigning level δ / 2 κ \delta/2^\kappa δ / 2 κ ensures joint coverage by a union bound. Algorithm 1 summarizes the full update procedure, and the trajectory-level computation is given in Appendix § 9 .§ 3 . The betting confidence bound of Waudby-Smith and Ramdas [40 ] can replace (12) (Appendix § 9 .7), and it admits a per-problem baseline (Appendix § 14 .§ 1 ). Appendix F extends IAPO to policies far from π \pi π , which gain only through coordinated changes.
Experiments
We evaluate IAPO in two settings. Controlled tasks, whose trajectories can be enumerated exactly, check the theory (§ 4.5 ). Orchestration over LLM workers on five long-input benchmarks compares accuracy and cost with published systems and ablates the interaction term (§ 4.1 to § 4.4 ).
Setup
Environments. We build long-input versions of GSM8K (Cobbe et al., [6 ] ), MATH (Hendrycks et al., [11 ] ), DROP (Dua et al., [9 ] ), MBPP (Austin et al., [1 ] ) and HumanEval (Chen et al., [4 ] ). A problem’s own unit (its statements, passage or function specification) is placed among 30 to 90 units of other problems, so one worker reading everything is not already as good as any team. A document holds 81, 91, 31, 61 and 61 units, and the test splits have 790, 220, 800, 333 and 131 problems (Appendix § 13 ). These are our own constructions, not the standard benchmarks, so absolute accuracies are not comparable with published numbers on the originals; every system is evaluated on the same constructed inputs. Every system receives the same question text, which includes the opening words of the target unit. The baselines also read the whole document; the orchestrated team reaches it only through an evidence finder, whose BM25 retriever the baselines lack, or a generalist that reads all of it. All workers are Qwen2.5-7B-Instruct (Yang et al., [43 ] ) served by vLLM (Kwon et al., [19 ] ) at temperature 1, following the protocol of MaAS (Zhang et al., [48 ] ).
Orchestration task. At each of at most five decisions (four on code) the orchestrator calls one of eight roles (planner, evidence finder, analyst, second answerer, calculator, checker, selector) or stops; code adds MaAS’s test-and-repair operator. It observes an embedding of the task and a few state flags, never a score. The reward is accuracy minus λ \lambda λ times tokens per 32,768. We train with λ = 0.076 \lambda=0.076 λ = 0.076 , the protocol’s value (IAPO), and with λ = 1 \lambda=1 λ = 1 , which trades accuracy for tokens (IAPO-Lite). Our trajectories are capped at 32,768 tokens; the baselines, which read the whole document, run with 131,072 or 393,216 tokens.
Training and certification. IAPO runs one round from the uniform policy. It fits one bounded update for each step in Λ = { 0.25 , 0.5 , 1 } \Lambda=\{0.25,0.5,1\} Λ = { 0.25 , 0.5 , 1 } on 6,400 proposal episodes (200 training problems, 32 repeats each), adds three certified leaps (Appendix F.§ 1 ), and certifies every candidate with betting bounds at total level δ = 0.05 \delta=0.05 δ = 0.05 on 8,000 validation episodes drawn from the remaining training problems. Test problems are never used for adaptation or selection. The certificate covers net return on the validation distribution, so test accuracy is an empirical question. Search and training tokens are excluded from the per-problem costs; one MBPP run uses about 1.7 × 10 8 1.7\times10^8 1.7 × 1 0 8 in total (Appendix § 14 ). Each configuration is trained twice (three times for MATH, IAPO), results are averaged over runs, and all rules were fixed before the runs.
Evaluation. Each test problem is run four times, and we report accuracy and tokens per problem for one episode. Since the baselines spend more tokens, we also evaluate at matched budget : a majority vote (on code, MaAS’s selector over programs, with its tokens counted over the most episodes whose tokens fit the strongest baseline’s. The number of episodes never depends on accuracy. Differences are paired by problem and called significant when z > 2 z>2 z > 2 (Appendix § 14 ).
Baselines. We run fifteen published systems on the same executor and splits, each as released, so none has a retriever. The single-agent group is CoT (Wei et al., [41 ] ), ComplexCoT (Fu et al., [10 ] ) and self-consistency (Wang et al., [38 ] ). The hand-crafted multi-agent group is MultiPersona (Wang et al., [39 ] ), LLM-Debate (Du et al., [8 ] ), LLM-Blender (Jiang et al., [14 ] ), DyLAN (Liu et al., [23 ] ), AgentVerse (Chen et al., [5 ] ) and MacNet (Qian et al., [26 ] ). The automated group is AutoAgents (Chen et al., [3 ] ), GPTSwarm (Zhuge et al., [52 ] ), ADAS (Hu et al., [13 ] ), AFlow (Zhang et al., [49 ] ), MAS-Zero (Ke et al., [17 ] ) and MaAS (Zhang et al., [48 ] ). Not every system runs on every benchmark, and Appendix § 13 lists the setup of each.
Accuracy and Cost
Accuracy at matched compute. At the strongest baseline’s budget (Table 1 ), IAPO leads by 2.3 points on GSM8K, 3.6 on DROP and 9.7 on MBPP (z = 6.3 z=6.3 z = 6.3 , against DyLAN), and is level with the strongest baseline on MATH (+ 0.3 +0.3 + 0.3 ) and HumanEval (+ 0.5 +0.5 + 0.5 ). The GSM8K and DROP differences are significant (Appendix § 15 ). IAPO-Lite, trained with the higher token price, goes further on GSM8K (+ 5.6 +5.6 + 5.6 ), MATH (+ 6.0 +6.0 + 6.0 , z = 2.6 z=2.6 z = 2.6 ) and MBPP (+ 7.2 +7.2 + 7.2 , z = 4.6 z=4.6 z = 4.6 ), and gives up accuracy on DROP (− 4.5 -4.5 − 4.5 ) and HumanEval (− 8.1 -8.1 − 8.1 ) in exchange for fewer tokens.
Table 1. Accuracy (%) with its change from Vanilla, and cost (grey, thousand tokens per problem; one episode for baselines, all voted episodes for IAPO) with Qwen2.5-7B-Instruct on long-input versions of five benchmarks. Bold: best; underline: runner-up. IAPO and IAPO-Lite (trained with λ = 1 \lambda=1 λ = 1 ) vote (on code, select programs) within the best baseline’s tokens per column. Avg. covers GSM8K, MATH, MBPP and HumanEval. Two-seed baselines are averaged; “–”: not run; †better accuracy of the upstream and diagnostic arms.
Cost. One deployed IAPO episode costs 5.1k to 11.5k tokens, against 26.5k to 121.9k for the strongest baselines, that is 2.7 to 24 times fewer (Figure 2 ); IAPO-Lite costs 1.0k to 5.6k. On code a single episode already matches or beats the strongest baseline: 81.1% on MBPP at 5.1k tokens against DyLAN’s 71.2% at 121.9k, and 83.7% on HumanEval at 9.9k against MaAS’s 83.8% at 26.5k. The matched-budget plans spend 20.4k to 50.8k tokens per problem, each within the strongest baseline’s budget.
Figure 2. Accuracy (left) and tokens per episode (right, log scale) of IAPO, IAPO-Lite, the strongest baseline of each benchmark and the other baselines. Accuracy is that of the voted plan within the strongest baseline’s budget; tokens are those of a single episode.
Ablation on the Interaction Term
The first-order variant removes J ( 2 ) J^{(2)} J ( 2 ) from fitting and plan ranking; validation, certification, retriever and roles are the same, so its gap to IAPO comes from the interaction term alone. For IAPO, the interaction term raises accuracy by 17.3 points on MBPP and 7.2 on HumanEval (both z > 2 z>2 z > 2 ), by 1.4 on DROP and 0.6 on GSM8K, and leaves MATH unchanged, while IAPO uses 32% to 58% of the first-order variant’s tokens on every benchmark (Figure 3 ). For IAPO-Lite it raises accuracy by 13.0 points on GSM8K, 12.9 on HumanEval, 2.9 on MBPP and 2.7 on MATH (all z > 2 z>2 z > 2 ) and lowers it by 0.5 on DROP, where its tokens rise to 125%. The paired differences and z z z -values are in Appendix § 15 .
Figure 3. With the interaction term (Z 2 Z_2 Z 2 ) and with first order only (J 1 J_1 J 1 ), for IAPO (left) and IAPO-Lite (right): accuracy with its gain in points (bold: z > 2 z>2 z > 2 ), and tokens per episode with IAPO’s tokens as a share of the first-order variant’s.
What the Policies Deploy
The deployed plans show where the interaction term pays (Appendix § 15 ). Test-and-repair helps only after a program exists, a pair of decisions. IAPO deploys it in all four code runs; the first-order variant never proposes it, fills the slots with a second programmer, and loses 17 points on MBPP and 7 on HumanEval at 3.1 and 1.8 times the tokens. On question answering and mathematics, all seven IAPO runs follow the evidence finder with the analyst, usually with a checker and the second answerer after it, while the first-order variant never calls the analyst and reads the whole document with the second answerer instead. The analyst alone loses points on every benchmark, so a first-order score never selects it, yet following the evidence finder with the analyst gains (Appendix § 15 ).
Controlled Tasks
One probability q q q is shared by N N N decisions with frozen binary workers, and all 2 N 2^N 2 N trajectories are enumerated (Appendix § 12 ). Complementarity is the trap of Proposition E.7 , Redundancy has substitutes, and Evidence threshold has N = 6 N=6 N = 6 .
The first-order score is zero for every step on Complementarity (oracle gain 0.148) and overshoots on Redundancy (0.006 against 0.010), while the second-order score picks the oracle step on both. On Evidence threshold the truncation bound exceeds the second-order gain, so Z 2 Z_2 Z 2 alone deploys a gain of 0.087, and adding Z 1 S Z_{1S} Z 1 S restores 0.356. Across 6,000 validation runs, no candidate with negative gain was accepted.
Assumptions, Conventions, and Notation
Throughout, π \pi π is the current policy, which generates the data, π ′ \pi' π ′ is a candidate policy, E π \mathbb{E}_{\pi} E π and Pr π \Pr_{\pi} Pr π are over trajectories of π \pi π , and all other quantities are as in § 2 and § 3 .
The Overlap Assumption
Assumption 3.1 (Overlap). There exists W ≥ 1 W \ge1 W ≥ 1 such that π t ′ ( a ∣ s ) ≤ W π t ( a ∣ s ) \pi'_t(a \mid s) \le W\pi_t(a \mid s) π t ′ ( a ∣ s ) ≤ W π t ( a ∣ s ) for every t ∈ [ H ] t \in[H] t ∈ [ H ] , every state s s s reachable under π \pi π , and every a ∈ A ( s ) a \in\mathcal{A}(s) a ∈ A ( s ) .
By Lemma B.3, Assumption 3.1 gives Pr π ′ ( τ t ) > 0 ⇒ Pr π ( τ t ) > 0 \Pr_{\pi'}(\tau_t) > 0 \Rightarrow\Pr_{\pi}(\tau_t) > 0 Pr π ′ ( τ t ) > 0 ⇒ Pr π ( τ t ) > 0 , and ρ t ′ ≤ W \rho_t' \le W ρ t ′ ≤ W gives ∏ t ′ ≤ t ρ t ′ ′ ≤ W t \prod_{t' \le t}\rho_{t'}' \le W^t ∏ t ′ ≤ t ρ t ′ ′ ≤ W t . Moreover:
W W W enters only the estimators, through ρ t ∈ [ 0 , W ] \rho_t \in[0,W] ρ t ∈ [ 0 , W ] ; the expansion of Theorem 3.2 is an identity for every π ′ \pi' π ′ , and overlap only writes each term as an expectation under π \pi π (Remark B.8).
The bounded update (7) satisfies Assumption 3.1 with W = 1 + η W = 1 + \eta W = 1 + η (Proposition 3.5); PPO clipping bounds ρ t \rho_t ρ t only on the sampled actions and does not imply Assumption 3.1 for the returned policy.
For estimation the assumption cannot be dropped: π t ′ ( a ∣ s ) > 0 = π t ( a ∣ s ) \pi'_t(a \mid s) > 0 = \pi_t(a \mid s) π t ′ ( a ∣ s ) > 0 = π t ( a ∣ s ) at a reachable s s s implies Pr π ( s t = s , a t = a ) = 0 \Pr_{\pi}(s_t = s, a_t = a) = 0 Pr π ( s t = s , a t = a ) = 0 , so no number of trajectories of π \pi π identifies the reward that π ′ \pi' π ′ collects after ( s , a ) (s,a) ( s , a ) .
Conventions
Finite trajectories. ∣ Q ∣ , ∣ Y ∣ , H < ∞ |\mathcal{Q}|, |\mathcal{Y}|, H < \infty ∣ Q ∣ , ∣ Y ∣ , H < ∞ , so all state, action, and trajectory sets are finite, every expectation below is a finite sum, and sums, expectations, and derivatives commute.
Fixed horizon. An episode that stops before step H H H stays until step H H H in an absorbing state with a single action and zero reward, attributed to no agent in N \mathcal{N} N . There ρ t = 1 \rho_t = 1 ρ t = 1 , so these steps enter no term below and no visit count. With at most K K K calls of two steps each and one stop, H = 2 K + 1 H = 2K + 1 H = 2 K + 1 and the orchestrator acts at most K + 1 K + 1 K + 1 times.
The state determines the history. Let τ t : = ( s 1 , a 1 , … , s t , a t ) \tau_t := (s_1, a_1, \ldots, s_t, a_t) τ t := ( s 1 , a 1 , … , s t , a t ) for t ≥ 1 t \ge1 t ≥ 1 and τ 0 \tau_0 τ 0 be empty. An orchestrator state ( q , h k ) (q, h_k) ( q , h k ) and a worker state ( q , h k , j , E ) (q, h_k, j, \mathcal{E}) ( q , h k , j , E ) record the task, the earlier calls ( j ℓ , E ℓ , y ℓ ) (j_\ell, \mathcal{E}_\ell, y_\ell) ( j ℓ , E ℓ , y ℓ ) , and the current call, which are the earlier actions; the earlier states are functions of these, so s t s_t s t determines τ t − 1 \tau_{t-1} τ t − 1 . The proofs use only this property and hold for every finite-horizon MDP that has it.
Acting agents. At every non-absorbing step, i t = i ( s t ) i_t = i(s_t) i t = i ( s t ) and π t ( a ∣ s ) = π i ( s ) ( a ∣ s ) \pi_t(a \mid s) = \pi^{i(s)}(a \mid s) π t ( a ∣ s ) = π i ( s ) ( a ∣ s ) . A worker policy π i ( ⋅ ∣ q , E ) \pi^i(\cdot\mid q,\mathcal{E}) π i ( ⋅ ∣ q , E ) depends on s s s only through ( q , E ) (q,\mathcal{E}) ( q , E ) , a function of s s s , so we write it as π t ( ⋅ ∣ s ) \pi_t(\cdot\mid s) π t ( ⋅ ∣ s ) .
Reward bounds. Let r ˉ t , G ˉ ≥ 0 \bar{r}_t, \bar{G} \ge0 r ˉ t , G ˉ ≥ 0 satisfy ∣ r t ( s , a ) ∣ ≤ r ˉ t |r_t(s,a)| \le\bar{r}_t ∣ r t ( s , a ) ∣ ≤ r ˉ t for all ( s , a ) (s,a) ( s , a ) and ∑ t = 1 H ∣ r t ( s t , a t ) ∣ ≤ G ˉ \sum_{t=1}^{H} |r_t(s_t,a_t)| \le\bar{G} ∑ t = 1 H ∣ r t ( s t , a t ) ∣ ≤ G ˉ on every trajectory. In the orchestration MDP, a trajectory has at most K K K calls, each with reward − λ c j -\lambda c_j − λ c j , and one stop, with reward R ∈ [ 0 , 1 ] R \in[0,1] R ∈ [ 0 , 1 ] , so:
∑ t = 1 H ∣ r t ( s t , a t ) ∣ ≤ λ ∑ calls c j + R ≤ λ K max j c j + 1. \sum_{t=1}^{H} |r_t(s_t,a_t)| \le\lambda\sum_{\text{calls}} c_j + R \le\lambda K \max_j c_j + 1. t = 1 ∑ H ∣ r t ( s t , a t ) ∣ ≤ λ calls ∑ c j + R ≤ λ K j max c j + 1.
Hence G ˉ = 1 + λ K max j c j \bar{G} = 1 + \lambda K \max_j c_j G ˉ = 1 + λ K max j c j and r ˉ t = max { 1 , λ max j c j } \bar{r}_t = \max\{1,\lambda\max_j c_j\} r ˉ t = max { 1 , λ max j c j } are valid.
Notation
For distributions p , p ′ p,p' p , p ′ on a finite set, TV ( p , p ′ ) : = 1 2 ∑ a ∣ p ( a ) − p ′ ( a ) ∣ \operatorname{TV}(p,p') := \frac{1}{2}\sum_a |p(a)-p'(a)| TV ( p , p ′ ) := 2 1 ∑ a ∣ p ( a ) − p ′ ( a ) ∣ . On a trajectory of π \pi π :
ρ t : = π t ′ ( a t ∣ s t ) π t ( a t ∣ s t ) , G t : = ∑ t ′ = t H r t ′ ( s t ′ , a t ′ ) , G H + 1 : = 0. \rho_t := \frac{\pi'_t(a_t \mid s_t)}{\pi_t(a_t \mid s_t)}, \qquad G_t := \sum_{t'=t}^{H} r_{t'}(s_{t'},a_{t'}), \qquad G_{H+1} := 0. ρ t := π t ( a t ∣ s t ) π t ′ ( a t ∣ s t ) , G t := t ′ = t ∑ H r t ′ ( s t ′ , a t ′ ) , G H + 1 := 0.
The policy change of agent i i i is α i : = max s : i ( s ) = i TV ( π i ( ⋅ ∣ s ) , π i ( ⋅ ∣ s ) ) \alpha_i := \max_{s:i(s)=i} \operatorname{TV}(\pi^i(\cdot\mid s),\pi^i(\cdot\mid s)) α i := max s : i ( s ) = i TV ( π i ( ⋅ ∣ s ) , π i ( ⋅ ∣ s )) . Its visit count is n i , t : = max { ∣ { t ′ ≤ t : i t ′ = i } ∣ } n_{i,t} := \max\{|\{t' \le t : i_{t'} = i\}| \} n i , t := max { ∣ { t ′ ≤ t : i t ′ = i } ∣ } , the maximum over trajectories of π \pi π , and n i : = n i , H n_i := n_{i,H} n i := n i , H ; in the orchestration MDP, n 0 ≤ K + 1 n_0 \le K + 1 n 0 ≤ K + 1 .
For S ⊆ [ H ] S \subseteq[H] S ⊆ [ H ] , the agent-count vector k ( S ) ∈ N N \mathbf{k}(S) \in\mathbb{N}^{\mathcal{N}} k ( S ) ∈ N N has entries k i ( S ) : = ∣ { t ∈ S : i t = i } ∣ k_i(S) := |\{t \in S : i_t = i\}| k i ( S ) := ∣ { t ∈ S : i t = i } ∣ . For k ∈ N N \mathbf{k} \in\mathbb{N}^{\mathcal{N}} k ∈ N N , ∣ k ∣ : = ∑ i k i |\mathbf{k}| := \sum_i k_i ∣ k ∣ := ∑ i k i , and e i \mathbf{e}_i e i is the unit vector of agent i i i . The interaction coefficient of k ≠ 0 \mathbf{k} \ne\mathbf{0} k = 0 is:
J k : = E π [ ∑ S : k ( S ) = k ∏ t ∈ S ( ρ t − 1 ) A t π ( s t , a t ) ] , t S : = max S . (14) J_{\mathbf{k}} := \mathbb{E}_{\pi}\left[\sum_{S:\mathbf{k}(S)=\mathbf{k}} \prod_{t \in S}(\rho_t - 1) A_{t}^{\pi}(s_t,a_t)\right], \qquad t_S := \max S. \tag*{(14)} J k := E π S : k ( S ) = k ∑ t ∈ S ∏ ( ρ t − 1 ) A t π ( s t , a t ) , t S := max S . (14)
Every S S S that contains an absorbing step has ∏ t ∈ S ( ρ t − 1 ) = 0 \prod_{t\in S}(\rho_t-1)=0 ∏ t ∈ S ( ρ t − 1 ) = 0 , and every other S S S has ∣ k ( S ) ∣ = ∣ S ∣ |k(S)|=|S| ∣ k ( S ) ∣ = ∣ S ∣ , so grouping the sets of size m m m by their counts gives:
J ( m ) = ∑ ∣ k ∣ = m J k . (15) J^{(m)}=\sum_{|k|=m}J_k. \tag*{(15)} J ( m ) = ∣ k ∣ = m ∑ J k . (15)
The first-order term J ( 1 ) J^{(1)} J ( 1 ) is the objective of PPO. We split it and the second-order term by the acting agents: L i L_i L i is the part of J ( 1 ) J^{(1)} J ( 1 ) from the steps of agent i i i , and:
I i → i ′ : = E π [ ∑ t ′ < t : i t ′ = i i t = i ′ ( ρ t ′ − 1 ) ( ρ t − 1 ) A t π ( s t , a t ) ] . I_{i\to i'}:=\mathbb{E}_{\pi}\left[\sum_{\substack{t'<t:i_{t'}=i\\i_t=i'}}(\rho_{t'}-1)(\rho_t-1)A_t^\pi(s_t,a_t)\right]. I i → i ′ := E π t ′ < t : i t ′ = i i t = i ′ ∑ ( ρ t ′ − 1 ) ( ρ t − 1 ) A t π ( s t , a t ) .
so that J ( 1 ) = ∑ i L i J^{(1)}=\sum_i L_i J ( 1 ) = ∑ i L i and J ( 2 ) = ∑ i , i ′ I i → i ′ J^{(2)}=\sum_{i,i'}I_{i\to i'} J ( 2 ) = ∑ i , i ′ I i → i ′ .
Proof of Theorem 3.2 (Exact Interaction Expansion)
We prove a general form that also covers mixtures of policies, vanishing coefficients, and the first two orders.
The Trajectory Law and the Value Functions
Lemma B.1 (Trajectory law). For every policy π \pi π and every t ∈ [ H ] t\in[H] t ∈ [ H ] :
Pr π ( τ t ) = d 0 ( s 1 ) ∏ t ′ = 1 t π t ′ ( a t ′ ∣ s t ′ ) ∏ t ′ = 1 t − 1 P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) , (16) \Pr_{\pi}(\tau_t)=d_0(s_1)\prod_{t'=1}^{t}\pi_{t'}(a_{t'}\mid s_{t'})\prod_{t'=1}^{t-1}P_{t'}(s_{t'+1}\mid s_{t'},a_{t'}), \tag*{(16)} π Pr ( τ t ) = d 0 ( s 1 ) t ′ = 1 ∏ t π t ′ ( a t ′ ∣ s t ′ ) t ′ = 1 ∏ t − 1 P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) , (16) and the objective is J ( π ) = ∑ t = 1 H ∑ τ t Pr π ( τ t ) r t ( s t , a t ) J(\pi)=\sum_{t=1}^{H}\sum_{\tau_t}\Pr_{\pi}(\tau_t)r_t(s_t,a_t) J ( π ) = ∑ t = 1 H ∑ τ t Pr π ( τ t ) r t ( s t , a t ) .
Proof. Splitting Pr π ( τ t ) \Pr_{\pi}(\tau_t) Pr π ( τ t ) in time order:
Pr π ( τ t ) = Pr ( s 1 ) ∏ t ′ = 1 t Pr π ( a t ′ ∣ τ t ′ − 1 , s t ′ ) ∏ t ′ = 1 t − 1 Pr ( s t ′ + 1 ∣ τ t ′ ) \Pr_{\pi}(\tau_t)=\Pr(s_1)\prod_{t'=1}^{t}\Pr_{\pi}(a_{t'}\mid\tau_{t'-1},s_{t'})\prod_{t'=1}^{t-1}\Pr(s_{t'+1}\mid\tau_{t'}) π Pr ( τ t ) = Pr ( s 1 ) t ′ = 1 ∏ t π Pr ( a t ′ ∣ τ t ′ − 1 , s t ′ ) t ′ = 1 ∏ t − 1 Pr ( s t ′ + 1 ∣ τ t ′ )
= ( a ) d 0 ( s 1 ) ∏ t ′ = 1 t π t ′ ( a t ′ ∣ s t ′ ) ∏ t ′ = 1 t − 1 P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) , \overset{(a)}{=}d_0(s_1)\prod_{t'=1}^{t}\pi_{t'}(a_{t'}\mid s_{t'})\prod_{t'=1}^{t-1}P_{t'}(s_{t'+1}\mid s_{t'},a_{t'}), = ( a ) d 0 ( s 1 ) t ′ = 1 ∏ t π t ′ ( a t ′ ∣ s t ′ ) t ′ = 1 ∏ t − 1 P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) ,
where (a) uses Pr ( s 1 ) = d 0 ( s 1 ) \Pr(s_1)=d_0(s_1) Pr ( s 1 ) = d 0 ( s 1 ) , Pr π ( a t ′ ∣ τ t ′ − 1 , s t ′ ) = π t ′ ( a t ′ ∣ s t ′ ) \Pr_{\pi}(a_{t'}\mid\tau_{t'-1},s_{t'})=\pi_{t'}(a_{t'}\mid s_{t'}) Pr π ( a t ′ ∣ τ t ′ − 1 , s t ′ ) = π t ′ ( a t ′ ∣ s t ′ ) since s t ′ s_{t'} s t ′ determines τ t ′ − 1 \tau_{t'-1} τ t ′ − 1 , and Pr ( s t ′ + 1 ∣ τ t ′ ) = P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) \Pr(s_{t'+1}\mid\tau_{t'})=P_{t'}(s_{t'+1}\mid s_{t'},a_{t'}) Pr ( s t ′ + 1 ∣ τ t ′ ) = P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) since P t ′ P_{t'} P t ′ depends on neither π \pi π nor τ t ′ − 1 \tau_{t'-1} τ t ′ − 1 . This is (16) . Since r t ( s t , a t ) r_t(s_t,a_t) r t ( s t , a t ) is a function of τ t \tau_t τ t , linearity gives:
J ( π ) = ∑ t = 1 H E π [ r t ( s t , a t ) ] = ∑ t = 1 H ∑ τ t Pr π ( τ t ) r t ( s t , a t ) . J(\pi)=\sum_{t=1}^{H}\mathbb{E}_{\pi}[r_t(s_t,a_t)]=\sum_{t=1}^{H}\sum_{\tau_t}\Pr_{\pi}(\tau_t)r_t(s_t,a_t). J ( π ) = t = 1 ∑ H E π [ r t ( s t , a t )] = t = 1 ∑ H τ t ∑ π Pr ( τ t ) r t ( s t , a t ) .
Lemma B.2 (Value functions). For every step t t t , every prefix τ t \tau_t τ t with Pr π ( τ t ) > 0 \Pr_{\pi}(\tau_t)>0 Pr π ( τ t ) > 0 , and every state s s s :
(a) E π [ G t ∣ τ t ] = Q t π ( s t , a t ) \mathbb{E}_{\pi}[G_t\mid\tau_t]=Q_t^\pi(s_t,a_t) E π [ G t ∣ τ t ] = Q t π ( s t , a t ) ;
(b) V t π ( s ) = ∑ a π t ( a ∣ s ) Q t π ( s , a ) V_t^\pi(s)=\sum_a\pi_t(a\mid s)Q_t^\pi(s,a) V t π ( s ) = ∑ a π t ( a ∣ s ) Q t π ( s , a ) , and hence ∑ a π t ( a ∣ s ) A t π ( s , a ) = 0 \sum_a\pi_t(a\mid s)A_t^\pi(s,a)=0 ∑ a π t ( a ∣ s ) A t π ( s , a ) = 0 ;
(c) ∣ Q t π ( s , a ) ∣ ≤ min { ∑ t ′ = t H r ˉ t ′ , G ˉ } |Q_t^\pi(s,a)|\le\min\left\{\sum_{t'=t}^{H}\bar r_{t'},\bar G\right\} ∣ Q t π ( s , a ) ∣ ≤ min { ∑ t ′ = t H r ˉ t ′ , G ˉ } .
Proof. For a completion τ H \tau_H τ H of τ t \tau_t τ t , dividing (16) at H H H by (16) at t t t :
Pr π ( τ H ) Pr π ( τ t ) = ∏ t ′ = t H − 1 P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) π t ′ + 1 ( a t ′ + 1 ∣ s t ′ + 1 ) , \frac{\Pr_{\pi}(\tau_H)}{\Pr_{\pi}(\tau_t)}
=\prod_{t'=t}^{H-1}P_{t'}(s_{t'+1}\mid s_{t'},a_{t'})\pi_{t'+1}(a_{t'+1}\mid s_{t'+1}), Pr π ( τ t ) Pr π ( τ H ) = t ′ = t ∏ H − 1 P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) π t ′ + 1 ( a t ′ + 1 ∣ s t ′ + 1 ) ,
which depends on τ t \tau_t τ t only through ( s t , a t ) (s_t,a_t) ( s t , a t ) . For (a), G t G_t G t is a function of ( s t , a t , … , s H , a H ) (s_t,a_t,\ldots,s_H,a_H) ( s t , a t , … , s H , a H ) , so its law given τ t \tau_t τ t equals its law given ( s t , a t ) (s_t,a_t) ( s t , a t ) :
E π [ G t ∣ τ t ] = E π [ G t ∣ s t , a t ] = Q t π ( s t , a t ) , \mathbb{E}_{\pi}[G_t\mid\tau_t]=\mathbb{E}_{\pi}[G_t\mid s_t,a_t]=Q_t^\pi(s_t,a_t), E π [ G t ∣ τ t ] = E π [ G t ∣ s t , a t ] = Q t π ( s t , a t ) ,
where the last equality is the definition of Q t π Q_t^\pi Q t π (§ 2.2 ).
For (b), by the tower rule over a t ∼ π t ( ⋅ ∣ s ) a_t\sim\pi_t(\cdot\mid s) a t ∼ π t ( ⋅ ∣ s ) and ∑ a π t ( a ∣ s ) = 1 \sum_a\pi_t(a\mid s)=1 ∑ a π t ( a ∣ s ) = 1 :
V t π ( s ) = E π [ G t ∣ s t = s ] = ∑ a π t ( a ∣ s ) E π [ G t ∣ s t = s , a t = a ] = ∑ a π t ( a ∣ s ) Q t π ( s , a ) , V_t^\pi(s)=\mathbb{E}_{\pi}[G_t\mid s_t=s]=\sum_a\pi_t(a\mid s)\mathbb{E}_{\pi}[G_t\mid s_t=s,a_t=a]=\sum_a\pi_t(a\mid s)Q_t^\pi(s,a), V t π ( s ) = E π [ G t ∣ s t = s ] = a ∑ π t ( a ∣ s ) E π [ G t ∣ s t = s , a t = a ] = a ∑ π t ( a ∣ s ) Q t π ( s , a ) ,
∑ a π t ( a ∣ s ) A t π ( s , a ) = ∑ a π t ( a ∣ s ) Q t π ( s , a ) − V t π ( s ) ∑ a π t ( a ∣ s ) = V t π ( s ) − V t π ( s ) = 0. \sum_a\pi_t(a\mid s)A_t^\pi(s,a)=\sum_a\pi_t(a\mid s)Q_t^\pi(s,a)-V_t^\pi(s)\sum_a\pi_t(a\mid s)=V_t^\pi(s)-V_t^\pi(s)=0. a ∑ π t ( a ∣ s ) A t π ( s , a ) = a ∑ π t ( a ∣ s ) Q t π ( s , a ) − V t π ( s ) a ∑ π t ( a ∣ s ) = V t π ( s ) − V t π ( s ) = 0.
For (c), since Q t π ( s , a ) Q_t^\pi(s,a) Q t π ( s , a ) is a conditional mean of G t G_t G t and ∣ G t ∣ ≤ ∑ t ′ = t H ∣ r t ′ ( s t ′ , a t ′ ) ∣ \lvert G_t\rvert\le\sum_{t'=t}^{H}\lvert r_{t'}(s_{t'},a_{t'})\rvert ∣ G t ∣ ≤ ∑ t ′ = t H ∣ r t ′ ( s t ′ , a t ′ )∣ :
∣ Q t π ( s , a ) ∣ ≤ E π [ ∑ t ′ = t H ∣ r t ′ ( s t ′ , a t ′ ) ∣ ∣ s t = s , a t = a ] ≤ min { ∑ t ′ = t H r ‾ t ′ , G ‾ } . \lvert Q_t^\pi(s,a)\rvert\le\mathbb{E}_{\pi}\left[\left.\sum_{t'=t}^{H}\lvert r_{t'}(s_{t'},a_{t'})\rvert\right|s_t=s,a_t=a\right]\le\min\left\{\sum_{t'=t}^{H}\overline{r}_{t'},\overline{G}\right\}. ∣ Q t π ( s , a )∣ ≤ E π [ t ′ = t ∑ H ∣ r t ′ ( s t ′ , a t ′ )∣ s t = s , a t = a ] ≤ min { t ′ = t ∑ H r t ′ , G } .
Reweighting and Centering
Lemma B.3 (Reweighting). Under Assumption 3.1, for every step t t t and every function f f f of τ t \tau_t τ t :
E π ′ [ f ( τ t ) ] = E π [ ∏ t ′ ≤ t ρ t ′ f ( τ t ) ] . (17) \mathbb{E}_{\pi'}[f(\tau_t)]=\mathbb{E}_{\pi}\left[\prod_{t'\le t}\rho_{t'}f(\tau_t)\right]. \tag*{(17)} E π ′ [ f ( τ t )] = E π [ t ′ ≤ t ∏ ρ t ′ f ( τ t ) ] . (17)
Proof. Let Pr π ′ ( τ t ) > 0 \Pr_{\pi'}(\tau_t)>0 Pr π ′ ( τ t ) > 0 . We show Pr π ( τ t ) > 0 \Pr_{\pi}(\tau_t)>0 Pr π ( τ t ) > 0 by induction over t ′ t' t ′ : d 0 ( s 1 ) > 0 d_0(s_1)>0 d 0 ( s 1 ) > 0 , so s 1 s_1 s 1 is reachable under π \pi π , and if s t ′ s_{t'} s t ′ is reachable under π \pi π , Assumption 3.1 gives:
0 < π t ′ ′ ( a t ′ ∣ s t ′ ) ≤ W π t ′ ( a t ′ ∣ s t ′ ) , 0<\pi_{t'}'(a_{t'}\mid s_{t'})\le W\pi_{t'}(a_{t'}\mid s_{t'}), 0 < π t ′ ′ ( a t ′ ∣ s t ′ ) ≤ W π t ′ ( a t ′ ∣ s t ′ ) ,
so π t ′ ( a t ′ ∣ s t ′ ) > 0 \pi_{t'}(a_{t'}\mid s_{t'})>0 π t ′ ( a t ′ ∣ s t ′ ) > 0 , and P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) > 0 P_{t'}(s_{t'+1}\mid s_{t'},a_{t'})>0 P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) > 0 makes s t ′ + 1 s_{t'+1} s t ′ + 1 reachable under π \pi π . Hence all factors of ((16) ) under π \pi π are positive, and the shared initial and transition factors cancel:
Pr π ′ ( τ t ) Pr π ( τ t ) = d 0 ( s 1 ) ∏ t ′ < t π t ′ ′ ( a t ′ ∣ s t ′ ) ∏ t ′ < t P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) d 0 ( s 1 ) ∏ t ′ < t π t ′ ( a t ′ ∣ s t ′ ) ∏ t ′ < t P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) = ∏ t ′ ≤ t ρ t ′ . \frac{\Pr_{\pi'}(\tau_t)}{\Pr_{\pi}(\tau_t)}
=\frac{d_0(s_1)\prod_{t'<t}\pi_{t'}'(a_{t'}\mid s_{t'})\prod_{t'<t}P_{t'}(s_{t'+1}\mid s_{t'},a_{t'})}{d_0(s_1)\prod_{t'<t}\pi_{t'}(a_{t'}\mid s_{t'})\prod_{t'<t}P_{t'}(s_{t'+1}\mid s_{t'},a_{t'})}
=\prod_{t'\le t}\rho_{t'}. Pr π ( τ t ) Pr π ′ ( τ t ) = d 0 ( s 1 ) ∏ t ′ < t π t ′ ( a t ′ ∣ s t ′ ) ∏ t ′ < t P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) d 0 ( s 1 ) ∏ t ′ < t π t ′ ′ ( a t ′ ∣ s t ′ ) ∏ t ′ < t P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) = t ′ ≤ t ∏ ρ t ′ .
Therefore:
E π ′ [ f ( τ t ) ] = ∑ τ t : Pr π ′ ( τ t ) > 0 Pr π ′ ( τ t ) f ( τ t ) = ∑ τ t : Pr π ′ ( τ t ) > 0 Pr π ( τ t ) ∏ t ′ ≤ t ρ t ′ f ( τ t ) \mathbb{E}_{\pi'}[f(\tau_t)]
=\sum_{\tau_t:\Pr_{\pi'}(\tau_t)>0}\Pr_{\pi'}(\tau_t)f(\tau_t)
=\sum_{\tau_t:\Pr_{\pi'}(\tau_t)>0}\Pr_{\pi}(\tau_t)\prod_{t'\le t}\rho_{t'}f(\tau_t) E π ′ [ f ( τ t )] = τ t : P r π ′ ( τ t ) > 0 ∑ π ′ Pr ( τ t ) f ( τ t ) = τ t : P r π ′ ( τ t ) > 0 ∑ π Pr ( τ t ) t ′ ≤ t ∏ ρ t ′ f ( τ t )
= ( a ) ∑ τ t : Pr π ′ ( τ t ) > 0 Pr π ( τ t ) ∏ t ′ ≤ t ρ t ′ f ( τ t ) = E π [ ∏ t ′ ≤ t ρ t ′ f ( τ t ) ] , \underset{(a)}{=}\sum_{\tau_t:\Pr_{\pi'}(\tau_t)>0}\Pr_{\pi}(\tau_t)\prod_{t'\le t}\rho_{t'}f(\tau_t)
=\mathbb{E}_{\pi}\left[\prod_{t'\le t}\rho_{t'}f(\tau_t)\right], ( a ) = τ t : P r π ′ ( τ t ) > 0 ∑ π Pr ( τ t ) t ′ ≤ t ∏ ρ t ′ f ( τ t ) = E π [ t ′ ≤ t ∏ ρ t ′ f ( τ t ) ] ,
where (a) holds since Pr π ( τ t ) > 0 = Pr π ′ ( τ t ) \Pr_{\pi}(\tau_t)>0=\Pr_{\pi'}(\tau_t) Pr π ( τ t ) > 0 = Pr π ′ ( τ t ) forces a factor π t ′ ′ ( a t ′ ∣ s t ′ ) = 0 \pi_{t'}'(a_{t'}\mid s_{t'})=0 π t ′ ′ ( a t ′ ∣ s t ′ ) = 0 in ((16) ), so ∏ t ′ ≤ t ρ t ′ = 0 \prod_{t'\le t}\rho_{t'}=0 ∏ t ′ ≤ t ρ t ′ = 0 .
Lemma B.4 (Centering). Under Assumption 3.1, for every step t t t :
E π [ ρ t − 1 ∣ τ t − 1 , s t ] = 0 , E π [ ∣ ρ t − 1 ∣ ∣ τ t − 1 , s t ] = 2 TV ( π t ′ ( ⋅ ∣ s t ) , π t ( ⋅ ∣ s t ) ) ≤ 2 α i t . \mathbb{E}_{\pi}[\rho_t-1\mid\tau_{t-1},s_t]=0,\qquad
\mathbb{E}_{\pi}[\lvert\rho_t-1\rvert\mid\tau_{t-1},s_t]=2\operatorname{TV}(\pi_t'(\cdot\mid s_t),\pi_t(\cdot\mid s_t))\le2\alpha_{i_t}. E π [ ρ t − 1 ∣ τ t − 1 , s t ] = 0 , E π [∣ ρ t − 1 ∣ ∣ τ t − 1 , s t ] = 2 TV ( π t ′ ( ⋅ ∣ s t ) , π t ( ⋅ ∣ s t )) ≤ 2 α i t .
with the right side read as 0 at absorbing steps, where ρ t = 1 \rho_t=1 ρ t = 1 .
Proof. Given ( τ t − 1 , s t ) (\tau_{t-1},s_t) ( τ t − 1 , s t ) , a t ∼ π t ( ⋅ ∣ s t ) a_t\sim\pi_t(\cdot\mid s_t) a t ∼ π t ( ⋅ ∣ s t ) . Write p a : = π t ( a ∣ s t ) p_a:=\pi_t(a\mid s_t) p a := π t ( a ∣ s t ) and p a ′ : = π t ′ ( a ∣ s t ) p_a':=\pi_t'(a\mid s_t) p a ′ := π t ′ ( a ∣ s t ) . By Assumption 3.1, p a = 0 ⇒ p a ′ = 0 p_a=0\Rightarrow p_a'=0 p a = 0 ⇒ p a ′ = 0 , so ∑ a : p a > 0 p a = ∑ a : p a > 0 p a ′ = 1 \sum_{a:p_a>0}p_a=\sum_{a:p_a>0}p_a'=1 ∑ a : p a > 0 p a = ∑ a : p a > 0 p a ′ = 1 and:
E π [ ρ t − 1 ∣ τ t − 1 , s t ] = ∑ a : p a > 0 p a ( p a ′ p a − 1 ) = ∑ a : p a > 0 p a ′ − ∑ a : p a > 0 p a = 1 − 1 = 0 , \mathbb{E}_{\pi}[\rho_t-1\mid\tau_{t-1},s_t]
=\sum_{a:p_a>0}p_a\left(\frac{p_a'}{p_a}-1\right)
=\sum_{a:p_a>0}p_a'-\sum_{a:p_a>0}p_a=1-1=0, E π [ ρ t − 1 ∣ τ t − 1 , s t ] = a : p a > 0 ∑ p a ( p a p a ′ − 1 ) = a : p a > 0 ∑ p a ′ − a : p a > 0 ∑ p a = 1 − 1 = 0 ,
E π [ ∣ ρ t − 1 ∣ ∣ τ t − 1 , s t ] = ∑ a : p a > 0 p a ∣ p a ′ p a − 1 ∣ = ( a ) ∑ a ∣ p a ′ − p a ∣ = 2 TV ( p ′ , p ) ≤ ( b ) 2 α i t , \mathbb{E}_{\pi}[\lvert\rho_t-1\rvert\mid\tau_{t-1},s_t]
=\sum_{a:p_a>0}p_a\left\lvert\frac{p_a'}{p_a}-1\right\rvert
\underset{(a)}{=}\sum_a\lvert p_a'-p_a\rvert
=2\operatorname{TV}(p',p)\underset{(b)}{\le}2\alpha_{i_t}, E π [∣ ρ t − 1 ∣ ∣ τ t − 1 , s t ] = a : p a > 0 ∑ p a p a p a ′ − 1 ( a ) = a ∑ ∣ p a ′ − p a ∣ = 2 TV ( p ′ , p ) ( b ) ≤ 2 α i t ,
where (a) uses p a = 0 ⇒ p a ′ = 0 p_a=0\Rightarrow p_a'=0 p a = 0 ⇒ p a ′ = 0 , and (b) is the definition of α i t \alpha_{i_t} α i t , since i ( s t ) = i t i(s_t)=i_t i ( s t ) = i t .
Lemma B.5 (Expanding a product). For real numbers x 1 , … , x t x_1,\ldots,x_t x 1 , … , x t :
∏ t ′ = 1 t ( 1 + x t ′ ) = ∑ S ⊆ [ t ] ∏ t ′ ∈ S x t ′ \prod_{t'=1}^{t}(1+x_{t'})=\sum_{S\subseteq[t]}\prod_{t'\in S}x_{t'} t ′ = 1 ∏ t ( 1 + x t ′ ) = S ⊆ [ t ] ∑ t ′ ∈ S ∏ x t ′ where the empty product equals 1.
Proof. Induction on t t t . For t = 1 t=1 t = 1 , 1 + x 1 = ∏ t ′ ∈ ∅ x t ′ + ∏ t ′ ∈ { 1 } x t ′ 1+x_1=\prod_{t'\in\varnothing}x_{t'}+\prod_{t'\in\{1\}}x_{t'} 1 + x 1 = ∏ t ′ ∈ ∅ x t ′ + ∏ t ′ ∈ { 1 } x t ′ . If the identity holds for t − 1 t-1 t − 1 , multiplying by 1 + x t 1+x_t 1 + x t :
∏ t ′ = 1 t ( 1 + x t ′ ) = ∑ S ⊆ [ t − 1 ] ∏ t ′ ∈ S x t ′ + ∑ S ⊆ [ t − 1 ] x t ∏ t ′ ∈ S x t ′ = ∑ S ⊆ [ t ] t ∉ S ∏ t ′ ∈ S x t ′ + ∑ S ⊆ [ t ] t ∈ S ∏ t ′ ∈ S x t ′ , \prod_{t'=1}^{t}(1+x_{t'})=\sum_{S\subseteq[t-1]}\prod_{t'\in S}x_{t'}+\sum_{S\subseteq[t-1]}x_t\prod_{t'\in S}x_{t'}=\sum_{\substack{S\subseteq[t]\\t\notin S}}\prod_{t'\in S}x_{t'}+\sum_{\substack{S\subseteq[t]\\t\in S}}\prod_{t'\in S}x_{t'}, t ′ = 1 ∏ t ( 1 + x t ′ ) = S ⊆ [ t − 1 ] ∑ t ′ ∈ S ∏ x t ′ + S ⊆ [ t − 1 ] ∑ x t t ′ ∈ S ∏ x t ′ = S ⊆ [ t ] t ∈ / S ∑ t ′ ∈ S ∏ x t ′ + S ⊆ [ t ] t ∈ S ∑ t ′ ∈ S ∏ x t ′ , which is the sum over all S ⊆ [ t ] S\subseteq[t] S ⊆ [ t ] . For example, ( 1 + x 1 ) ( 1 + x 2 ) = 1 + x 1 + x 2 + x 1 x 2 (1+x_1)(1+x_2)=1+x_1+x_2+x_1x_2 ( 1 + x 1 ) ( 1 + x 2 ) = 1 + x 1 + x 2 + x 1 x 2 .
The Expansion
Theorem 3.2 (Interaction expansion). Under Assumption 3.1:
J ( π ′ ) − J ( π ) = ∑ m = 1 H J ( m ) , J(\pi')-J(\pi)=\sum_{m=1}^{H}J^{(m)}, J ( π ′ ) − J ( π ) = m = 1 ∑ H J ( m ) , where the order-m m m interaction is:
J ( m ) = E π [ ∑ 1 ≤ t 1 < ⋯ < t m ≤ H ( ρ t 1 − 1 ) ⋯ ( ρ t m − 1 ) A t m π ( s t m , a t m ) ] (18) J^{(m)}=\mathbb{E}_{\pi}\left[\sum_{1\leq t_1<\cdots<t_m\leq H}(\rho_{t_1}-1)\cdots(\rho_{t_m}-1)A^{\pi}_{t_m}(s_{t_m},a_{t_m})\right] \tag*{(18)} J ( m ) = E π [ 1 ≤ t 1 < ⋯ < t m ≤ H ∑ ( ρ t 1 − 1 ) ⋯ ( ρ t m − 1 ) A t m π ( s t m , a t m ) ] (18) Moreover, each J ( m ) J^{(m)} J ( m ) remains unchanged if A t m π A^{\pi}_{t_m} A t m π is replaced by G t m − b ( s t m ) G_{t_m}-b(s_{t_m}) G t m − b ( s t m ) for any bounded function b b b .
By item (ii) below, J k J_{\mathbf{k}} J k is the coefficient of ∏ i ω i k i \prod_i\omega_i^{k_i} ∏ i ω i k i in a polynomial in the mixing weights of the agents.
Theorem B.6 (Exact interaction expansion, general form). Under Assumption 3.1, the following hold.
(i) J ( π ′ ) − J ( π ) = ∑ k ≠ 0 J k = ∑ m = 1 H J ( m ) J(\pi')-J(\pi)=\sum_{\mathbf{k}\ne0}J_{\mathbf{k}}=\sum_{m=1}^{H}J^{(m)} J ( π ′ ) − J ( π ) = ∑ k = 0 J k = ∑ m = 1 H J ( m ) .
(ii) For ω ∈ [ 0 , 1 ] N \boldsymbol{\omega}\in[0,1]^{N} ω ∈ [ 0 , 1 ] N , let π ω \pi_{\boldsymbol{\omega}} π ω use π i + ω i ( π ′ i − π i ) \pi^i+\omega_i(\pi'^i-\pi^i) π i + ω i ( π ′ i − π i ) for every agent i i i . Then:
J ( π ω ) = J ( π ) + ∑ k ≠ 0 J k ∏ i ∈ N ω i k i . J(\pi_{\boldsymbol{\omega}})=J(\pi)+\sum_{\mathbf{k}\ne0}J_{\mathbf{k}}\prod_{i\in N}\omega_i^{k_i}. J ( π ω ) = J ( π ) + k = 0 ∑ J k i ∈ N ∏ ω i k i . (iii) The advantage in (14) can be replaced by G t S − ψ ( s t S ) G_{t_S}-\psi(s_{t_S}) G t S − ψ ( s t S ) for any bounded function ψ \psi ψ .
(iv) J k = 0 J_{\mathbf{k}}=0 J k = 0 if k i > 0 k_i>0 k i > 0 for some unchanged agent i i i , or if no trajectory of π \pi π contains a set of steps of changed agents with the counts k \mathbf{k} k in question.
(v) J e i = L i J_{\mathbf{e}_i}=L_i J e i = L i , J 2 e i = I i → i J_{2\mathbf{e}_i}=I_{i\to i} J 2 e i = I i → i , and J e i + e i ′ = I i → i ′ + I i ′ → i J_{\mathbf{e}_i+\mathbf{e}_{i'}}=I_{i\to i'}+I_{i'\to i} J e i + e i ′ = I i → i ′ + I i ′ → i for i ≠ i ′ i\ne i' i = i ′ .
Proof. By Lemma B.1 and Lemma B.3 with f = r t ( s t , a t ) f=r_t(s_t,a_t) f = r t ( s t , a t ) , we obtain (2) :
J ( π ′ ) − J ( π ) = ∑ t = 1 H ( E π ′ [ r t ( s t , a t ) ] − E π [ r t ( s t , a t ) ] ) = ∑ t = 1 H E π [ ( ∏ t ′ < t ρ t ′ − 1 ) r t ( s t , a t ) ] (19) J(\pi')-J(\pi)=\sum_{t=1}^{H}\left(\mathbb{E}_{\pi'}[r_t(s_t,a_t)]-\mathbb{E}_{\pi}[r_t(s_t,a_t)]\right)=\sum_{t=1}^{H}\mathbb{E}_{\pi}\left[\left(\prod_{t'<t}\rho_{t'}-1\right)r_t(s_t,a_t)\right] \tag*{(19)} J ( π ′ ) − J ( π ) = t = 1 ∑ H ( E π ′ [ r t ( s t , a t )] − E π [ r t ( s t , a t )] ) = t = 1 ∑ H E π [ ( t ′ < t ∏ ρ t ′ − 1 ) r t ( s t , a t ) ] (19)
By Lemma B.5 with x t ′ = ρ t ′ − 1 x_{t'}=\rho_{t'}-1 x t ′ = ρ t ′ − 1 , where S = ∅ S=\varnothing S = ∅ contributes 1 and cancels the − 1 -1 − 1 :
∏ t ′ < t ρ t ′ − 1 = ∏ t ′ < t ( 1 + ( ρ t ′ − 1 ) ) − 1 = ∑ ∅ ≠ S ⊆ [ t ] ∏ t ′ ∈ S ( ρ t ′ − 1 ) . \prod_{t'<t}\rho_{t'}-1=\prod_{t'<t}(1+(\rho_{t'}-1))-1=\sum_{\varnothing\ne S\subseteq[t]}\prod_{t'\in S}(\rho_{t'}-1). t ′ < t ∏ ρ t ′ − 1 = t ′ < t ∏ ( 1 + ( ρ t ′ − 1 )) − 1 = ∅ = S ⊆ [ t ] ∑ t ′ ∈ S ∏ ( ρ t ′ − 1 ) .
Inserting this into (19) , dropping the sets S S S that contain an absorbing step (their products vanish), and splitting the others by k ( S ) \mathbf{k}(S) k ( S ) gives the reward form , used again in Appendix D:
J ( π ′ ) − J ( π ) = ∑ k ≠ 0 ∑ t = 1 H E π [ ∑ S ⊆ [ t ] : k ( S ) = k ∏ t ′ ∈ S ( ρ t ′ − 1 ) r t ( s t , a t ) ] (20) J(\pi')-J(\pi)=\sum_{\mathbf{k}\ne0}\sum_{t=1}^{H}\mathbb{E}_{\pi}\left[\sum_{\substack{S\subseteq[t]:\,\mathbf{k}(S)=\mathbf{k}}}\prod_{t'\in S}(\rho_{t'}-1)r_t(s_t,a_t)\right] \tag*{(20)} J ( π ′ ) − J ( π ) = k = 0 ∑ t = 1 ∑ H E π S ⊆ [ t ] : k ( S ) = k ∑ t ′ ∈ S ∏ ( ρ t ′ − 1 ) r t ( s t , a t ) (20)
With S S S running over { S : k ( S ) = k } \{S : k(S) = k\} { S : k ( S ) = k } in the inner sums:
J ( π ′ ) − J ( π ) = ∑ k ≠ 0 E π [ ∑ S ∏ t ′ ∈ S ( ρ t ′ − 1 ) ∑ t = t S H r t ( s t , a t ) ] = ∑ k ≠ 0 E π [ ∑ S ∏ t ′ ∈ S ( ρ t ′ − 1 ) G t S ] . (21) J(\pi') - J(\pi) = \sum_{k \ne0} \mathbb{E}_{\pi}\left[\sum_{S} \prod_{t' \in S}(\rho_{t'} - 1)\sum_{t=t_S}^{H} r_t(s_t,a_t)\right] = \sum_{k \ne0} \mathbb{E}_{\pi}\left[\sum_{S} \prod_{t' \in S}(\rho_{t'} - 1)G_{t_S}\right]. \tag*{(21)} J ( π ′ ) − J ( π ) = k = 0 ∑ E π [ S ∑ t ′ ∈ S ∏ ( ρ t ′ − 1 ) t = t S ∑ H r t ( s t , a t ) ] = k = 0 ∑ E π [ S ∑ t ′ ∈ S ∏ ( ρ t ′ − 1 ) G t S ] . (21)
where (a) exchanges the finite sums using { ( t , S ) : S ⊆ [ t ] } = { ( t , S ) : t ≥ t S } \{(t,S) : S \subseteq[t]\} = \{(t,S) : t \ge t_S\} {( t , S ) : S ⊆ [ t ]} = {( t , S ) : t ≥ t S } , and (b) is the definition of G t S G_{t_S} G t S .
Fix S S S , let t = t S t = t_S t = t S , and write w S : = ∏ t ′ ∈ S ( ρ t ′ − 1 ) = w S ′ ( ρ t − 1 ) w_S := \prod_{t' \in S}(\rho_{t'} - 1) = w'_S(\rho_t - 1) w S := ∏ t ′ ∈ S ( ρ t ′ − 1 ) = w S ′ ( ρ t − 1 ) with w S ′ : = ∏ t ′ ∈ S , t ′ < t ( ρ t ′ − 1 ) w'_S := \prod_{t' \in S,t'<t}(\rho_{t'} - 1) w S ′ := ∏ t ′ ∈ S , t ′ < t ( ρ t ′ − 1 ) , so w S w_S w S is a function of τ t \tau_t τ t and w S ′ w'_S w S ′ of τ t − 1 \tau_{t-1} τ t − 1 . For every bounded ψ \psi ψ :
E π [ w S G t ] = ( a ) E π [ w S E π [ G t ∣ τ t ] ] = ( b ) E π [ w S Q t π ( s t , a t ) ] , (22) \mathbb{E}_{\pi}[w_S G_t] \mathrel{\overset{(a)}{=}} \mathbb{E}_{\pi}\left[w_S \mathbb{E}_{\pi}[G_t \mid\tau_t]\right] \mathrel{\overset{(b)}{=}} \mathbb{E}_{\pi}\left[w_S Q_t^\pi(s_t,a_t)\right], \tag*{(22)} E π [ w S G t ] = ( a ) E π [ w S E π [ G t ∣ τ t ] ] = ( b ) E π [ w S Q t π ( s t , a t ) ] , (22)
E π [ w S ψ ( s t ) ] = E π [ w S ′ ψ ( s t ) E π [ ρ t − 1 ∣ τ t − 1 , s t ] ] = ( d ) E π [ w S ′ ψ ( s t ) ⋅ 0 ] = 0 , (23) \mathbb{E}_{\pi}[w_S\psi(s_t)] = \mathbb{E}_{\pi}\left[w'_S\psi(s_t)\mathbb{E}_{\pi}[\rho_t-1\mid\tau_{t-1},s_t]\right] \mathrel{\overset{(d)}{=}} \mathbb{E}_{\pi}[w'_S\psi(s_t)\cdot0]=0, \tag*{(23)} E π [ w S ψ ( s t )] = E π [ w S ′ ψ ( s t ) E π [ ρ t − 1 ∣ τ t − 1 , s t ] ] = ( d ) E π [ w S ′ ψ ( s t ) ⋅ 0 ] = 0 , (23)
where (a) is the tower rule, (b) uses Lemma B.2(a), (c) conditions on ( τ t − 1 , s t ) (\tau_{t-1},s_t) ( τ t − 1 , s t ) , and (d) uses Lemma B.4.
For (i) and (iii), subtracting (22) with ψ = V t π \psi=V_t^\pi ψ = V t π from (21) and using A t π = Q t π − V t π A_t^\pi=Q_t^\pi-V_t^\pi A t π = Q t π − V t π :
E π [ w S G t S ] = E π [ w S A t S π ( s t S , a t S ) ] . \mathbb{E}_{\pi}[w_S G_{t_S}] = \mathbb{E}_{\pi}[w_S A_{t_S}^\pi(s_{t_S},a_{t_S})]. E π [ w S G t S ] = E π [ w S A t S π ( s t S , a t S )] .
In (20), the sum over S S S then gives J k J_k J k in (14), and the sum over k k k gives the first equality in (i); the second is (15). For an arbitrary bounded ψ \psi ψ , (22) gives E π [ w S G t S ] = E π [ w S ( G t S − ψ ( s t S ) ) ] \mathbb{E}_{\pi}[w_SG_{t_S}]=\mathbb{E}_{\pi}[w_S(G_{t_S}-\psi(s_{t_S}))] E π [ w S G t S ] = E π [ w S ( G t S − ψ ( s t S ))] , which is (iii).
For (ii), π i + ω i ( π ′ i − π i ) = ( 1 − ω i ) π i + ω i π ′ i \pi^i+\omega_i(\pi'^i-\pi^i)=(1-\omega_i)\pi^i+\omega_i\pi'^i π i + ω i ( π ′ i − π i ) = ( 1 − ω i ) π i + ω i π ′ i is a convex combination of distributions, and on reachable states:
( 1 − ω i ) π i + ω i π ′ i ≤ ( 1 − ω i ) π i + ω i W π i ≤ W π i , (1-\omega_i)\pi^i+\omega_i\pi'^i\le(1-\omega_i)\pi^i+\omega_iW\pi^i\le W\pi^i, ( 1 − ω i ) π i + ω i π ′ i ≤ ( 1 − ω i ) π i + ω i W π i ≤ W π i ,
so π ω \pi_\omega π ω satisfies Assumption 3.1 with the same W W W , and its ratio at step t ′ t' t ′ is:
π t ′ ( a t ′ ∣ s t ′ ) + ω i t ′ ( π t ′ ′ ( a t ′ ∣ s t ′ ) − π t ′ ( a t ′ ∣ s t ′ ) ) π t ′ ( a t ′ ∣ s t ′ ) = 1 + ω i t ′ ( ρ t ′ − 1 ) . \frac{\pi_{t'}(a_{t'}\mid s_{t'})+\omega_{i_{t'}}(\pi'_{t'}(a_{t'}\mid s_{t'})-\pi_{t'}(a_{t'}\mid s_{t'}))}{\pi_{t'}(a_{t'}\mid s_{t'})}
=1+\omega_{i_{t'}}(\rho_{t'}-1). π t ′ ( a t ′ ∣ s t ′ ) π t ′ ( a t ′ ∣ s t ′ ) + ω i t ′ ( π t ′ ′ ( a t ′ ∣ s t ′ ) − π t ′ ( a t ′ ∣ s t ′ )) = 1 + ω i t ′ ( ρ t ′ − 1 ) .
Apply (i) to π ω \pi_\omega π ω , with coefficients under the same π \pi π and A π A^\pi A π . Since ∣ { t ′ ∈ S : i t ′ = i } ∣ = k i ( S ) |\{t'\in S:i_{t'}=i\}|=k_i(S) ∣ { t ′ ∈ S : i t ′ = i } ∣ = k i ( S ) :
∏ t ′ ∈ S ω i t ′ ( ρ t ′ − 1 ) = ( ∏ t ′ ∈ S ω i t ′ ) ∏ t ′ ∈ S ( ρ t ′ − 1 ) = ( ∏ i ∈ N ω i k i ( S ) ) ∏ t ′ ∈ S ( ρ t ′ − 1 ) . \prod_{t'\in S}\omega_{i_{t'}}(\rho_{t'}-1)=\left(\prod_{t'\in S}\omega_{i_{t'}}\right)\prod_{t'\in S}(\rho_{t'}-1)=\left(\prod_{i\in\mathcal{N}}\omega_i^{k_i(S)}\right)\prod_{t'\in S}(\rho_{t'}-1). t ′ ∈ S ∏ ω i t ′ ( ρ t ′ − 1 ) = ( t ′ ∈ S ∏ ω i t ′ ) t ′ ∈ S ∏ ( ρ t ′ − 1 ) = ( i ∈ N ∏ ω i k i ( S ) ) t ′ ∈ S ∏ ( ρ t ′ − 1 ) .
The factor ∏ i ∈ N ω i k i ( S ) \prod_{i\in\mathcal{N}}\omega_i^{k_i(S)} ∏ i ∈ N ω i k i ( S ) depends on S S S only through k ( S ) k(S) k ( S ) , so it leaves the expectation, and J ( π ω ) − J ( π ) = ∑ k J k ∏ i ω i k i J(\pi_\omega)-J(\pi)=\sum_kJ_k\prod_i\omega_i^{k_i} J ( π ω ) − J ( π ) = ∑ k J k ∏ i ω i k i .
For (iv), π ′ i = π i ⇒ ρ t ′ = 1 \pi'^i=\pi^i\Rightarrow\rho_{t'}=1 π ′ i = π i ⇒ ρ t ′ = 1 for all t ′ t' t ′ with i t ′ = i i_{t'}=i i t ′ = i , so k i ( S ) > 0 ⇒ ∏ t ′ ∈ S ( ρ t ′ − 1 ) = 0 k_i(S)>0\Rightarrow\prod_{t'\in S}(\rho_{t'}-1)=0 k i ( S ) > 0 ⇒ ∏ t ′ ∈ S ( ρ t ′ − 1 ) = 0 . If no trajectory of π \pi π has steps of changed agents with counts k \mathbf{k} k , every S S S with k ( S ) = k k(S)=\mathbf{k} k ( S ) = k contains a step of an unchanged agent, so its product is zero and the term vanishes.
For (v), ∣ k ∣ = 1 |k|=1 ∣ k ∣ = 1 gives S = { t } S=\{t\} S = { t } and k ( S ) = e i ⟺ i t = i k(S)=e_i\Longleftrightarrow i_t=i k ( S ) = e i ⟺ i t = i , so J e i = L i J_{e_i}=L_i J e i = L i . ∣ k ∣ = 2 |k|=2 ∣ k ∣ = 2 gives S = { t ′ , t } S=\{t',t\} S = { t ′ , t } with t ′ < t = t S t'<t=t_S t ′ < t = t S , and the term ( ρ t ′ − 1 ) ( ρ t − 1 ) A t π ( s t , a t ) (\rho_{t'}-1)(\rho_t-1)A_{t}^{\pi}(s_t,a_t) ( ρ t ′ − 1 ) ( ρ t − 1 ) A t π ( s t , a t ) belongs to I i t ′ → i t I_{i_{t'}\to i_t} I i t ′ → i t . Since k ( S ) = 2 e i ⟺ i t ′ = i t = i k(S)=2e_i\Longleftrightarrow i_{t'}=i_t=i k ( S ) = 2 e i ⟺ i t ′ = i t = i and k ( S ) = e i + e i ′ ⟺ ( i t ′ , i t ) ∈ { ( i , i ′ ) , ( i ′ , i ) } k(S)=e_i+e_{i'}\Longleftrightarrow(i_{t'},i_t)\in\{(i,i'),(i',i)\} k ( S ) = e i + e i ′ ⟺ ( i t ′ , i t ) ∈ {( i , i ′ ) , ( i ′ , i )} , we get J 2 e i = I i → i J_{2e_i}=I_{i\to i} J 2 e i = I i → i and J e i + e i ′ = I i → i ′ + I i ′ → i J_{e_i+e_{i'}}=I_{i\to i'}+I_{i'\to i} J e i + e i ′ = I i → i ′ + I i ′ → i .
Proof of Theorem 3.2. Items (i) and (iii) of Theorem B.6.
Remark B.7 (No critic is required). Item (iii) with ψ = 0 \psi=0 ψ = 0 gives:
J ( m ) = E π [ ∑ ∣ S ∣ = m ∏ t ∈ S ( ρ t − 1 ) G t S ] . J^{(m)}=\mathbb{E}_{\pi}\left[\sum_{|S|=m}\prod_{t\in S}(\rho_t-1)G_{t_S}\right]. J ( m ) = E π ∣ S ∣ = m ∑ t ∈ S ∏ ( ρ t − 1 ) G t S .
Any state-dependent ψ \psi ψ keeps every term unbiased and changes only the variance; (14) is the choice ψ = V π \psi=V^\pi ψ = V π .
Remark B.8 (Overlap is needed only for estimation). For any π ′ \pi' π ′ , (19) and (20) still hold with Pr π ( τ t ) ∏ t ′ ∈ S ( ρ t ′ − 1 ) \Pr_\pi(\tau_t)\prod_{t'\in S}(\rho_{t'}-1) Pr π ( τ t ) ∏ t ′ ∈ S ( ρ t ′ − 1 ) , for S ⊆ [ t ] S\subseteq[t] S ⊆ [ t ] , replaced by the signed measure:
d 0 ( s 1 ) ∏ t ′ ≤ t , t ′ ∉ S π t ′ ( a t ′ ∣ s t ′ ) ∏ t ′ ∈ S ( π t ′ ′ ( a t ′ ∣ s t ′ ) − π t ′ ( a t ′ ∣ s t ′ ) ) ∏ t ′ < t P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) . d_0(s_1)\prod_{\substack{t'\le t,\ t'\notin S}}\pi_{t'}(a_{t'}\mid s_{t'})\prod_{t'\in S}\left(\pi'_{t'}(a_{t'}\mid s_{t'})-\pi_{t'}(a_{t'}\mid s_{t'})\right)\prod_{t'<t}P_{t'}(s_{t'+1}\mid s_{t'},a_{t'}). d 0 ( s 1 ) t ′ ≤ t , t ′ ∈ / S ∏ π t ′ ( a t ′ ∣ s t ′ ) t ′ ∈ S ∏ ( π t ′ ′ ( a t ′ ∣ s t ′ ) − π t ′ ( a t ′ ∣ s t ′ ) ) t ′ < t ∏ P t ′ ( s t ′ + 1 ∣ s t ′ , a t ′ ) .
The expansion thus holds for every pair of policies; overlap only makes it an expectation under π \pi π , estimable from data of π \pi π .
Remark B.9 (Relation to Taylor expansion policy optimization). For a single policy in a γ \gamma γ -discounted MDP, Tang et al. [34 ] expand Q π ′ Q^{\pi^\prime} Q π ′ around Q π Q^\pi Q π in powers of π ′ − π \pi^\prime-\pi π ′ − π through the resolvent of the transition operator of π \pi π ; their second-order objective adds to the first-order objective:
γ ( 1 − γ ) 2 E [ ( ρ ( x , a ) − 1 ) ( ρ ( x ′ , a ′ ) − 1 ) Q π ( x ′ , a ′ ) ] \frac{\gamma}{(1-\gamma)^2}\mathbb{E}\left[(\rho(x,a)-1)(\rho(x^\prime,a^\prime)-1)Q^\pi(x^\prime,a^\prime)\right] ( 1 − γ ) 2 γ E [ ( ρ ( x , a ) − 1 ) ( ρ ( x ′ , a ′ ) − 1 ) Q π ( x ′ , a ′ ) ]
with ( x , a ) (x,a) ( x , a ) and ( x ′ , a ′ ) (x^\prime,a^\prime) ( x ′ , a ′ ) drawn from discounted visitation distributions of π \pi π and ( x ′ , a ′ ) (x^\prime,a^\prime) ( x ′ , a ′ ) strictly later, the discounted analog of J ( 2 ) J^{(2)} J ( 2 ) . Theorem B.6 differs as follows: (1) their series is infinite and converges if max x ∑ a ∣ π ′ ( a ∣ x ) − π ( a ∣ x ) ∣ < ( 1 − γ ) / γ \max_x\sum_a|\pi^\prime(a\mid x)-\pi(a\mid x)|<(1-\gamma)/\gamma max x ∑ a ∣ π ′ ( a ∣ x ) − π ( a ∣ x ) ∣ < ( 1 − γ ) / γ , whereas (4) has at most H H H terms and is exact under Assumption 3.1; (2) its terms split by acting agents, giving the pair terms of Appendix § 7 ; (3) the remainder after order r r r is bounded by visit counts (Theorem D.2 ), not by a radius on π ′ − π \pi^\prime-\pi π ′ − π ; (4) the second-order estimate enters a high-confidence certificate (Theorem 3.7).
Proof of Corollary 3.3 and Interactions across Agents
Each J k J_k J k with ∣ k ∣ ≥ 2 |k|\ge2 ∣ k ∣ ≥ 2 averages products of ∣ k ∣ |k| ∣ k ∣ factors ρ t − 1 \rho_t-1 ρ t − 1 , which vanish at π \pi π .
Gradient and Curvature
Theorem C.1 (Gradient and curvature). Let each agent i i i have parameters θ i \theta_i θ i , with π i θ \pi^\theta_i π i θ depending on θ \theta θ only through θ i \theta_i θ i and θ ↦ π θ ( a ∣ s ) \theta\mapsto\pi_\theta(a\mid s) θ ↦ π θ ( a ∣ s ) in C 2 C^2 C 2 . Let π ′ = π θ \pi^\prime=\pi_\theta π ′ = π θ and let π = π θ 0 \pi=\pi_{\theta^0} π = π θ 0 have full support. At θ = θ 0 \theta=\theta^0 θ = θ 0 , ∇ θ i J ( π θ ) = ∇ θ i L i \nabla_{\theta_i}J(\pi_\theta)=\nabla_{\theta_i}L_i ∇ θ i J ( π θ ) = ∇ θ i L i , and for i ≠ i ′ i\ne i^\prime i = i ′ :
∇ θ i θ i ′ 2 J ( π θ ) = ∇ θ i θ i ′ 2 ( I i → i ′ + I i ′ → i ) , ∇ θ i θ i ′ 2 J ( 1 ) = 0. \nabla^2_{\theta_i\theta_{i^\prime}}J(\pi_\theta)=\nabla^2_{\theta_i\theta_{i^\prime}}\left(I_{i\to i^\prime}+I_{i^\prime\to i}\right),\qquad\nabla^2_{\theta_i\theta_{i^\prime}}J^{(1)}=0. ∇ θ i θ i ′ 2 J ( π θ ) = ∇ θ i θ i ′ 2 ( I i → i ′ + I i ′ → i ) , ∇ θ i θ i ′ 2 J ( 1 ) = 0.
Throughout, θ = ( θ i ) i ∈ N \theta=(\theta_i)_{i\in\mathbb{N}} θ = ( θ i ) i ∈ N , θ ↦ π θ ( a ∣ s ) \theta\mapsto\pi_\theta(a\mid s) θ ↦ π θ ( a ∣ s ) is C 2 C^2 C 2 , π = π θ 0 \pi=\pi_{\theta^0} π = π θ 0 has full support, and π ′ = π θ \pi^\prime=\pi_\theta π ′ = π θ . Every coefficient takes the expectation and the advantage under π \pi π , so θ \theta θ enters only through:
ρ t ( θ ) : = π θ ( a t ∣ s t ) π ( a t ∣ s t ) , ∇ θ i ′ ρ t = 0 ( i ′ ≠ i t ) . \rho_t(\theta):=\frac{\pi_\theta(a_t\mid s_t)}{\pi(a_t\mid s_t)},\qquad\nabla_{\theta_{i^\prime}}\rho_t=0\quad(i^\prime\ne i_t). ρ t ( θ ) := π ( a t ∣ s t ) π θ ( a t ∣ s t ) , ∇ θ i ′ ρ t = 0 ( i ′ = i t ) .
We write L i ( θ ) L_i(\theta) L i ( θ ) , I i → i ′ ( θ ) I_{i\to i^\prime}(\theta) I i → i ′ ( θ ) , and J k ( θ ) J_k(\theta) J k ( θ ) for the quantities of Appendix § 5 and (14) with π ′ = π θ \pi^\prime=\pi_\theta π ′ = π θ .
Lemma C.2 (Products of vanishing factors). Let f = ∏ ℓ = 1 q f ℓ f=\prod_{\ell=1}^{q}f_\ell f = ∏ ℓ = 1 q f ℓ with f ℓ ∈ C 2 f_\ell\in C^2 f ℓ ∈ C 2 and f ℓ ( θ 0 ) = 0 f_\ell(\theta^0)=0 f ℓ ( θ 0 ) = 0 for every ℓ \ell ℓ . Then ∇ f ( θ 0 ) = 0 \nabla f(\theta^0)=0 ∇ f ( θ 0 ) = 0 if q ≥ 2 q\ge2 q ≥ 2 , and ∇ 2 f ( θ 0 ) = 0 \nabla^2f(\theta^0)=0 ∇ 2 f ( θ 0 ) = 0 if q ≥ 3 q\ge3 q ≥ 3 .
Proof. By the product rule:
∇ f = ∑ ℓ = 1 q ∇ f ℓ ∏ ℓ ′ ≠ ℓ f ℓ ′ , \nabla f=\sum_{\ell=1}^{q}\nabla f_\ell\prod_{\ell^\prime\ne\ell}f_{\ell^\prime}, ∇ f = ℓ = 1 ∑ q ∇ f ℓ ℓ ′ = ℓ ∏ f ℓ ′ ,
∇ 2 f = ∑ ℓ = 1 q ∇ 2 f ℓ ∏ ℓ ′ ≠ ℓ f ℓ ′ + ∑ ℓ ≠ ℓ ′ ′ ∇ f ℓ ∇ f ℓ ′ ′ T ∏ ℓ ′ ≠ ℓ , ℓ ′ ′ f ℓ ′ . \nabla^2f=\sum_{\ell=1}^{q}\nabla^2f_\ell\prod_{\ell^\prime\ne\ell}f_{\ell^\prime}+\sum_{\ell\ne\ell^{\prime\prime}}\nabla f_\ell\nabla f_{\ell^{\prime\prime}}^{\mathsf T}\prod_{\ell^\prime\ne\ell,\ell^{\prime\prime}}f_{\ell^\prime}. ∇ 2 f = ℓ = 1 ∑ q ∇ 2 f ℓ ℓ ′ = ℓ ∏ f ℓ ′ + ℓ = ℓ ′′ ∑ ∇ f ℓ ∇ f ℓ ′′ T ℓ ′ = ℓ , ℓ ′′ ∏ f ℓ ′ .
For q ≥ 2 q\ge2 q ≥ 2 (resp. q ≥ 3 q\ge3 q ≥ 3 ), every summand of ∇ f \nabla f ∇ f (resp. ∇ 2 f \nabla^2f ∇ 2 f ) contains an undifferentiated factor f ℓ ′ f_{\ell^\prime} f ℓ ′ , and f ℓ ′ ( θ 0 ) = 0 f_{\ell^\prime}(\theta^0)=0 f ℓ ′ ( θ 0 ) = 0 .
Theorem C.3 (Gradient and curvature, general form). Let g t : = ∇ θ i t log π θ ( a t ∣ s t ) g_t:=\nabla_{\theta_{i_t}}\log\pi_\theta(a_t\mid s_t) g t := ∇ θ i t log π θ ( a t ∣ s t ) at θ 0 \theta^0 θ 0 . At θ = θ 0 \theta=\theta^0 θ = θ 0 :
(a) ∇ θ i J ( π θ ) = ∇ θ i L i \nabla_{\theta_i}J(\pi_\theta)=\nabla_{\theta_i}L_i ∇ θ i J ( π θ ) = ∇ θ i L i for every agent i i i ;
(b) ∇ θ i θ i ′ 2 J ( 1 ) = 0 \nabla^2_{\theta_i\theta_{i^\prime}}J^{(1)}=0 ∇ θ i θ i ′ 2 J ( 1 ) = 0 for i ≠ i ′ i\ne i^\prime i = i ′ ;
(c) for i ≠ i ′ i\ne i^\prime i = i ′ :
∇ θ i θ i ′ 2 J ( π θ ) = E π [ ∑ t ′ < t ( 1 { i t ′ = i , i t = i ′ } g t ′ g t T + 1 { i t ′ = i ′ , i t = i } g t g t ′ T ) A t π ( s t , a t ) ] , (24) \nabla^2_{\theta_i\theta_{i^\prime}}J(\pi_\theta)=\mathbb{E}_\pi\left[\sum_{t^\prime<t}\left(\boldsymbol{1}\{i_{t^\prime}=i,i_t=i^\prime\}g_{t^\prime}g_t^{\mathsf T}+\boldsymbol{1}\{i_{t^\prime}=i^\prime,i_t=i\}g_tg_{t^\prime}^{\mathsf T}\right)A_t^\pi(s_t,a_t)\right], \tag*{(24)} ∇ θ i θ i ′ 2 J ( π θ ) = E π [ t ′ < t ∑ ( 1 { i t ′ = i , i t = i ′ } g t ′ g t T + 1 { i t ′ = i ′ , i t = i } g t g t ′ T ) A t π ( s t , a t ) ] , (24) which equals ∇ θ i θ i ′ 2 ( I i → i ′ + I i ′ → i ) \nabla^2_{\theta_i\theta_{i^\prime}}(I_{i\to i^\prime}+I_{i^\prime\to i}) ∇ θ i θ i ′ 2 ( I i → i ′ + I i ′ → i ) ;
(d) for every agent i i i :
∇ θ i θ i 2 J ( π θ ) = E π [ ∑ t : i t = i ( ∇ θ i 2 log π θ ( a t ∣ s t ) + g t g t T ) A t π ( s t , a t ) + ∑ t ′ < t : i t ′ = i t = i ( g t g t T + g t g t ′ T ) A t π ( s t , a t ) ] . \nabla^2_{\theta_i\theta_i}J(\pi_\theta)=\mathbb{E}_\pi\left[\sum_{t:i_t=i}\left(\nabla^2_{\theta_i}\log\pi_\theta(a_t\mid s_t)+g_tg_t^{\mathsf T}\right)A_t^\pi(s_t,a_t)+\sum_{t^\prime<t:i_{t^\prime}=i_t=i}\left(g_tg_t^{\mathsf T}+g_tg_{t^\prime}^{\mathsf T}\right)A_t^\pi(s_t,a_t)\right]. ∇ θ i θ i 2 J ( π θ ) = E π t : i t = i ∑ ( ∇ θ i 2 log π θ ( a t ∣ s t ) + g t g t T ) A t π ( s t , a t ) + t ′ < t : i t ′ = i t = i ∑ ( g t g t T + g t g t ′ T ) A t π ( s t , a t ) .
Proof. Full support gives Assumption 3.1 with W = max t , s , a π θ ( a ∣ s ) / π ( a ∣ s ) < ∞ W=\max_{t,s,a}\pi_{\theta}(a\mid s)/\pi(a\mid s)<\infty W = max t , s , a π θ ( a ∣ s ) / π ( a ∣ s ) < ∞ , so by Theorem B.6(i):
J ( π θ ) − J ( π ) = ∑ k ≠ 0 J k ( θ ) , J k ( θ ) = E π [ ∑ S : k ( S ) = k ∏ t ∈ S ( ρ t ( θ ) − 1 ) A t S π ( s t S , a t S ) ] . J(\pi_{\theta})-J(\pi)=\sum_{\mathbf{k}\ne0}J_{\mathbf{k}}(\theta),\qquad J_{\mathbf{k}}(\theta)=\mathbb{E}_{\pi}\left[\sum_{S:k(S)=\mathbf{k}}\prod_{t\in S}\left(\rho_t(\theta)-1\right)A^{\pi}_{t_S}(s_{t_S},a_{t_S})\right]. J ( π θ ) − J ( π ) = k = 0 ∑ J k ( θ ) , J k ( θ ) = E π S : k ( S ) = k ∑ t ∈ S ∏ ( ρ t ( θ ) − 1 ) A t S π ( s t S , a t S ) .
The expectation is a finite sum with θ \theta θ -free weights and ρ t ( θ 0 ) − 1 = 0 ∀ t \rho_t(\theta^0)-1=0\ \forall t ρ t ( θ 0 ) − 1 = 0 ∀ t , so J k J_{\mathbf{k}} J k is a finite weighted sum of products of ∣ k ∣ |\mathbf{k}| ∣ k ∣ such factors, and Lemma C.2 gives:
∇ J k ( θ 0 ) = 0 if ∣ k ∣ ≥ 2 , ∇ 2 J k ( θ 0 ) = 0 if ∣ k ∣ ≥ 3. (25) \nabla J_{\mathbf{k}}(\theta^0)=0\ \text{ if }|\mathbf{k}|\ge2,\qquad\nabla^2J_{\mathbf{k}}(\theta^0)=0\ \text{ if }|\mathbf{k}|\ge3. \tag*{(25)} ∇ J k ( θ 0 ) = 0 if ∣ k ∣ ≥ 2 , ∇ 2 J k ( θ 0 ) = 0 if ∣ k ∣ ≥ 3. (25)
For (a):
∇ θ i J ( π θ ) = ( a ) ∑ ℓ ∈ N ∇ θ i J e ℓ = ( b ) ∑ ℓ ∈ N ∇ θ i L ℓ = ( c ) ∇ θ i L i , \nabla_{\theta_i}J(\pi_\theta)\mathrel{\mathop=^{(a)}}\sum_{\ell\in\mathcal{N}}\nabla_{\theta_i}J_{e_\ell}\mathrel{\mathop=^{(b)}}\sum_{\ell\in\mathcal{N}}\nabla_{\theta_i}L_\ell\mathrel{\mathop=^{(c)}}\nabla_{\theta_i}L_i, ∇ θ i J ( π θ ) = ( a ) ℓ ∈ N ∑ ∇ θ i J e ℓ = ( b ) ℓ ∈ N ∑ ∇ θ i L ℓ = ( c ) ∇ θ i L i ,
where (a) uses (25) , (b) uses Theorem B.6(v), and (c) uses ∇ θ i L ℓ = 0 \nabla_{\theta_i}L_\ell=0 ∇ θ i L ℓ = 0 for ℓ ≠ i \ell\ne i ℓ = i .
For (b), L ℓ L_\ell L ℓ depends only on θ ℓ \theta_\ell θ ℓ , so
∇ θ i θ i ′ 2 J ( 1 ) = ∑ ℓ ∇ θ i θ i ′ 2 L ℓ = 0 for i ≠ i ′ . \nabla^2_{\theta_i\theta_{i'}}J^{(1)}=\sum_\ell\nabla^2_{\theta_i\theta_{i'}}L_\ell=0\text{ for }i\ne i'. ∇ θ i θ i ′ 2 J ( 1 ) = ℓ ∑ ∇ θ i θ i ′ 2 L ℓ = 0 for i = i ′ .
For (c) and (d), let i t = i i_t=i i t = i . Differentiating ∇ log π θ = ∇ π θ / π θ \nabla\log\pi_\theta=\nabla\pi_\theta/\pi_\theta ∇ log π θ = ∇ π θ / π θ gives:
∇ 2 log π θ = ∇ 2 π θ π θ − ∇ π θ π θ ( ∇ π θ π θ ) T . \nabla^2\log\pi_\theta=\frac{\nabla^2\pi_\theta}{\pi_\theta}-\frac{\nabla\pi_\theta}{\pi_\theta}\left(\frac{\nabla\pi_\theta}{\pi_\theta}\right)^{\mathsf{T}}. ∇ 2 log π θ = π θ ∇ 2 π θ − π θ ∇ π θ ( π θ ∇ π θ ) T .
With π θ 0 = π \pi_{\theta^0}=\pi π θ 0 = π , at θ 0 \theta^0 θ 0 :
∇ θ i ρ t = ∇ θ i π θ ( a t ∣ s t ) π θ ( a t ∣ s t ) ∣ θ 0 = ∇ θ i log π θ ( a t ∣ s t ) ∣ θ 0 = g t , (26) \nabla_{\theta_i}\rho_t=\left.\frac{\nabla_{\theta_i}\pi_\theta(a_t\mid s_t)}{\pi_\theta(a_t\mid s_t)}\right|_{\theta^0}=\left.\nabla_{\theta_i}\log\pi_\theta(a_t\mid s_t)\right|_{\theta^0}=g_t, \tag*{(26)} ∇ θ i ρ t = π θ ( a t ∣ s t ) ∇ θ i π θ ( a t ∣ s t ) θ 0 = ∇ θ i log π θ ( a t ∣ s t ) ∣ θ 0 = g t , (26)
∇ θ i 2 ρ t = ∇ θ i 2 π θ ( a t ∣ s t ) π θ ( a t ∣ s t ) ∣ θ 0 = ∇ θ i 2 log π θ ( a t ∣ s t ) ∣ θ 0 + g t g t T . (27) \nabla^2_{\theta_i}\rho_t=\left.\frac{\nabla^2_{\theta_i}\pi_\theta(a_t\mid s_t)}{\pi_\theta(a_t\mid s_t)}\right|_{\theta^0}=\left.\nabla^2_{\theta_i}\log\pi_\theta(a_t\mid s_t)\right|_{\theta^0}+g_tg_t^{\mathsf{T}}. \tag*{(27)} ∇ θ i 2 ρ t = π θ ( a t ∣ s t ) ∇ θ i 2 π θ ( a t ∣ s t ) θ 0 = ∇ θ i 2 log π θ ( a t ∣ s t ) θ 0 + g t g t T . (27)
For (c), (25) leaves ∣ k ∣ ≤ 2 |\mathbf{k}|\le2 ∣ k ∣ ≤ 2 , and J k J_{\mathbf{k}} J k depends on θ j \theta_j θ j only if k j ≥ 1 k_j\ge1 k j ≥ 1 , so only k = e i + e i ′ \mathbf{k}=e_i+e_{i'} k = e i + e i ′ remains:
∇ θ i θ i ′ 2 J ( π θ ) = ∇ θ i θ i ′ 2 J e i + e i ′ = ( a ) ∇ θ i θ i ′ 2 ( I i → i ′ + I i ′ → i ) , (28) \nabla^2_{\theta_i\theta_{i'}}J(\pi_\theta)=\nabla^2_{\theta_i\theta_{i'}}J_{e_i+e_{i'}}\mathrel{\mathop=^{(a)}}\nabla^2_{\theta_i\theta_{i'}}(I_{i\to i'}+I_{i'\to i}), \tag*{(28)} ∇ θ i θ i ′ 2 J ( π θ ) = ∇ θ i θ i ′ 2 J e i + e i ′ = ( a ) ∇ θ i θ i ′ 2 ( I i → i ′ + I i ′ → i ) , (28)
where (a) uses Theorem B.6(v). By (14) , k ( S ) = e i + e i ′ ⟺ S = { t ′ , t } k(S)=e_i+e_{i'}\Longleftrightarrow S=\{t',t\} k ( S ) = e i + e i ′ ⟺ S = { t ′ , t } with t ′ < t t'<t t ′ < t and ( i t ′ , i t ) ∈ { ( i , i ′ ) , ( i ′ , i ) } (i_{t'},i_t)\in\{(i,i'),(i',i)\} ( i t ′ , i t ) ∈ {( i , i ′ ) , ( i ′ , i )} , so:
J e i + e i ′ ( θ ) = E π [ ∑ t ′ < t ( 1 { i t ′ = i , i t = i ′ } + 1 { i t ′ = i ′ , i t = i } ) ( ρ t ′ ( θ ) − 1 ) ( ρ t ( θ ) − 1 ) A t π ( s t , a t ) ] . J_{e_i+e_{i'}}(\theta)=\mathbb{E}_{\pi}\left[\sum_{t'<t}\left(\mathbf{1}\{i_{t'}=i,i_t=i'\}+\mathbf{1}\{i_{t'}=i',i_t=i\}\right)(\rho_{t'}(\theta)-1)(\rho_t(\theta)-1)A^\pi_t(s_t,a_t)\right]. J e i + e i ′ ( θ ) = E π [ t ′ < t ∑ ( 1 { i t ′ = i , i t = i ′ } + 1 { i t ′ = i ′ , i t = i } ) ( ρ t ′ ( θ ) − 1 ) ( ρ t ( θ ) − 1 ) A t π ( s t , a t ) ] .
On { i t ′ = i , i t = i ′ } \{i_{t'}=i,i_t=i'\} { i t ′ = i , i t = i ′ } , ρ t ′ \rho_{t'} ρ t ′ depends only on θ i \theta_i θ i and ρ t \rho_t ρ t only on θ i ′ \theta_{i'} θ i ′ , so:
∇ θ i ∇ θ i ′ T [ ( ρ t ′ − 1 ) ( ρ t − 1 ) ] ∣ θ 0 = ∇ θ i ρ t ′ ( ∇ θ i ′ ρ t ) T = ( a ) g t ′ g t T , \left.\nabla_{\theta_i}\nabla_{\theta_{i'}}^{\mathsf{T}}\left[(\rho_{t'}-1)(\rho_t-1)\right]\right|_{\theta^0}=\nabla_{\theta_i}\rho_{t'}\left(\nabla_{\theta_{i'}}\rho_t\right)^{\mathsf{T}}\mathrel{\mathop=^{(a)}}g_{t'}g_t^{\mathsf{T}}, ∇ θ i ∇ θ i ′ T [ ( ρ t ′ − 1 ) ( ρ t − 1 ) ] θ 0 = ∇ θ i ρ t ′ ( ∇ θ i ′ ρ t ) T = ( a ) g t ′ g t T ,
where (a) uses (26) ; on { i t ′ = i ′ , i t = i } \{i_{t'}=i',i_t=i\} { i t ′ = i ′ , i t = i } the same computation gives g t g t ′ T g_tg_{t'}^{\mathsf{T}} g t g t ′ T . Differentiating termwise gives (24) .
For (d), (25) removes ∣ k ∣ ≥ 3 |\mathbf{k}|\ge3 ∣ k ∣ ≥ 3 . For k = e i + e ℓ \mathbf{k}=e_i+e_\ell k = e i + e ℓ with ℓ ≠ i \ell\ne i ℓ = i , two derivatives in θ i \theta_i θ i leave the factor ρ − 1 \rho-1 ρ − 1 of agent ℓ \ell ℓ , which is 0 at θ 0 \theta^0 θ 0 . Hence only L i = J e i L_i=J_{e_i} L i = J e i and I i → i = J 2 e i I_{i\to i}=J_{2e_i} I i → i = J 2 e i contribute, with summands ( ρ t − 1 ) A t π (\rho_t-1)A^\pi_t ( ρ t − 1 ) A t π over { t : i t = i } \{t:i_t=i\} { t : i t = i } and ( ρ t ′ − 1 ) ( ρ t − 1 ) A t π (\rho_{t'}-1)(\rho_t-1)A^\pi_t ( ρ t ′ − 1 ) ( ρ t − 1 ) A t π over { t ′ < t : i t ′ = i t = i } \{t'<t:i_{t'}=i_t=i\} { t ′ < t : i t ′ = i t = i } . By the product rule:
∇ θ i 2 [ ( ρ t ′ − 1 ) ( ρ t − 1 ) ] = ∇ θ i 2 ρ t ′ ( ρ t − 1 ) + ∇ θ i ρ t ′ ∇ θ i ρ t T + ∇ θ i ρ t ∇ θ i ρ t ′ T + ( ρ t ′ − 1 ) ∇ θ i 2 ρ t . \nabla^2_{\theta_i}\left[(\rho_{t'}-1)(\rho_t-1)\right]=\nabla^2_{\theta_i}\rho_{t'}(\rho_t-1)+\nabla_{\theta_i}\rho_{t'}\nabla_{\theta_i}\rho_t^{\mathsf{T}}+\nabla_{\theta_i}\rho_t\nabla_{\theta_i}\rho_{t'}^{\mathsf{T}}+(\rho_{t'}-1)\nabla^2_{\theta_i}\rho_t. ∇ θ i 2 [ ( ρ t ′ − 1 ) ( ρ t − 1 ) ] = ∇ θ i 2 ρ t ′ ( ρ t − 1 ) + ∇ θ i ρ t ′ ∇ θ i ρ t T + ∇ θ i ρ t ∇ θ i ρ t ′ T + ( ρ t ′ − 1 ) ∇ θ i 2 ρ t .
At θ 0 \theta^0 θ 0 , ρ t ′ − 1 = ρ t − 1 = 0 \rho_{t'}-1=\rho_t-1=0 ρ t ′ − 1 = ρ t − 1 = 0 , so only the middle terms remain:
∇ θ i θ i 2 J ( π θ ) = E π [ ∑ t : i t = i ∇ θ i 2 ρ t A t π ( s t , a t ) + ∑ t ′ < t : i t ′ = i t = i ( ∇ θ i ρ t ′ ∇ θ i ρ t T + ∇ θ i ρ t ∇ θ i ρ t ′ T ) A t π ( s t , a t ) ] \nabla^2_{\theta_i\theta_i}J(\pi_\theta)=\mathbb{E}_{\pi}\left[\sum_{t:i_t=i}\nabla^2_{\theta_i}\rho_t A^\pi_t(s_t,a_t)+\sum_{t'<t:i_{t'}=i_t=i}\left(\nabla_{\theta_i}\rho_{t'}\nabla_{\theta_i}\rho_t^{\mathsf{T}}+\nabla_{\theta_i}\rho_t\nabla_{\theta_i}\rho_{t'}^{\mathsf{T}}\right)A^\pi_t(s_t,a_t)\right] ∇ θ i θ i 2 J ( π θ ) = E π t : i t = i ∑ ∇ θ i 2 ρ t A t π ( s t , a t ) + t ′ < t : i t ′ = i t = i ∑ ( ∇ θ i ρ t ′ ∇ θ i ρ t T + ∇ θ i ρ t ∇ θ i ρ t ′ T ) A t π ( s t , a t )
= ( a ) E π [ ∑ t : i t = i ( ∇ θ i 2 log π θ ( a t ∣ s t ) + g t g t T ) A t π ( s t , a t ) + ∑ t ′ < t : i t ′ = i t = i ( g t ′ g t T + g t g t ′ T ) A t π ( s t , a t ) ] , \mathrel{\mathop=^{(a)}}\mathbb{E}_{\pi}\left[\sum_{t:i_t=i}\left(\nabla^2_{\theta_i}\log\pi_\theta(a_t\mid s_t)+g_tg_t^{\mathsf{T}}\right)A^\pi_t(s_t,a_t)+\sum_{t'<t:i_{t'}=i_t=i}\left(g_t'g_t^{\mathsf{T}}+g_tg_{t'}^{\mathsf{T}}\right)A^\pi_t(s_t,a_t)\right], = ( a ) E π t : i t = i ∑ ( ∇ θ i 2 log π θ ( a t ∣ s t ) + g t g t T ) A t π ( s t , a t ) + t ′ < t : i t ′ = i t = i ∑ ( g t ′ g t T + g t g t ′ T ) A t π ( s t , a t ) ,
where (a) uses (26) and (27) .
Proof of Theorem C.1. It is Theorem C.3(a), (b), and (c).
Proof of Corollary 3.3 . With all workers fixed, only the orchestrator has parameters, and full support of π \pi π gives Assumption 3.1 for π ′ = π θ \pi' = \pi_\theta π ′ = π θ with W = max s , a π θ ( a ∣ s ) / π ( a ∣ s ) < ∞ W = \max_{s,a} \pi_\theta(a \mid s)/\pi(a \mid s) < \infty W = max s , a π θ ( a ∣ s ) / π ( a ∣ s ) < ∞ . By Theorem B.6 (i):
J ( π θ ) = J ( π ) + ∑ m = 1 H J ( m ) ( θ ) . J(\pi_\theta) = J(\pi) + \sum_{m=1}^{H} J^{(m)}(\theta). J ( π θ ) = J ( π ) + m = 1 ∑ H J ( m ) ( θ ) .
J ( π ) J(\pi) J ( π ) does not depend on θ \theta θ , and each J ( m ) ( θ ) J^{(m)}(\theta) J ( m ) ( θ ) is a finite sum of expectations under π \pi π of products of m m m factors ρ t ( θ ) − 1 \rho_t(\theta)-1 ρ t ( θ ) − 1 times A m π A_m^\pi A m π , which does not depend on θ \theta θ . Every factor vanishes at θ 0 \theta^0 θ 0 , so Lemma C.2 gives:
∇ θ J ( m ) ( θ 0 ) = 0 ( m ≥ 2 ) , ∇ θ 2 J ( m ) ( θ 0 ) = 0 ( m ≥ 3 ) . \nabla_\theta J^{(m)}(\theta^0) = 0 \quad(m \ge2), \qquad\nabla_\theta^2 J^{(m)}(\theta^0) = 0 \quad(m \ge3). ∇ θ J ( m ) ( θ 0 ) = 0 ( m ≥ 2 ) , ∇ θ 2 J ( m ) ( θ 0 ) = 0 ( m ≥ 3 ) .
Differentiating the finite sum at θ 0 \theta^0 θ 0 :
∇ θ J ( π θ ) = ∇ θ J ( 1 ) ( θ ) , ∇ θ 2 J ( π θ ) = ∇ θ 2 J ( 1 ) ( θ ) + ∇ θ 2 J ( 2 ) ( θ ) . \nabla_\theta J(\pi_\theta) = \nabla_\theta J^{(1)}(\theta), \qquad\nabla_\theta^2 J(\pi_\theta) = \nabla_\theta^2 J^{(1)}(\theta) + \nabla_\theta^2 J^{(2)}(\theta). ∇ θ J ( π θ ) = ∇ θ J ( 1 ) ( θ ) , ∇ θ 2 J ( π θ ) = ∇ θ 2 J ( 1 ) ( θ ) + ∇ θ 2 J ( 2 ) ( θ ) .
By Theorem C.3 , for v → 0 v \to0 v → 0 :
J ( π θ 0 + v ) − J ( π ) = ∑ i [ ∇ L i T v i + 1 2 v i T ∇ 2 ( L i + I i → i ) v i ] + ∑ i < i ′ v i T ∇ θ i θ i ′ 2 ( I i → i ′ + I i ′ → i ) v i ′ + o ( ∥ v ∥ 2 ) . J(\pi^{\theta^0+v}) - J(\pi) = \sum_i \left[\nabla L_i^{\mathsf T}v_i + \frac{1}{2}v_i^{\mathsf T}\nabla^2(L_i+I_{i\to i})v_i\right] + \sum_{i<i'} v_i^{\mathsf T}\nabla^2_{\theta_i\theta_{i'}}(I_{i\to i'}+I_{i'\to i})v_{i'} + o(\lVert v\rVert^2). J ( π θ 0 + v ) − J ( π ) = i ∑ [ ∇ L i T v i + 2 1 v i T ∇ 2 ( L i + I i → i ) v i ] + i < i ′ ∑ v i T ∇ θ i θ i ′ 2 ( I i → i ′ + I i ′ → i ) v i ′ + o (∥ v ∥ 2 ) .
The first-order objective J ( 1 ) = ∑ i L i J^{(1)}=\sum_i L_i J ( 1 ) = ∑ i L i keeps ∇ L i T v i \nabla L_i^{\mathsf T}v_i ∇ L i T v i and 1 2 v i T ∇ 2 L i v i \frac{1}{2}v_i^{\mathsf T}\nabla^2L_iv_i 2 1 v i T ∇ 2 L i v i but misses every I i → i I_{i\to i} I i → i and every cross block; with frozen workers (v i = 0 v_i=0 v i = 0 for i ≠ 0 i\ne0 i = 0 ) it misses I 0 → 0 I_{0\to0} I 0 → 0 .
Per-Agent Rules Are Blind
For ω ∈ { 0 , 1 } 2 \omega\in\{0,1\}^2 ω ∈ { 0 , 1 } 2 , let Δ ( ω ) \Delta(\omega) Δ ( ω ) be the gain of the policy in which agent i i i adopts π i \pi^i π i if ω i = 1 \omega_i=1 ω i = 1 and keeps π i \pi^i π i otherwise.
Proposition C.4 (Per-agent rules are blind). There exist two problems, each with two agents that act once, with identical ( L i , α i , n i ) i = 1 , 2 (L_i,\alpha_i,n_i)_{i=1,2} ( L i , α i , n i ) i = 1 , 2 and gains:
Δ ( ω ) = ω 1 L 1 + ω 2 L 2 + ω 1 ω 2 I 1 → 2 , \Delta(\omega) = \omega_1L_1 + \omega_2L_2 + \omega_1\omega_2 I_{1\to2}, Δ ( ω ) = ω 1 L 1 + ω 2 L 2 + ω 1 ω 2 I 1 → 2 , where I 1 → 2 > 0 I_{1\to2}>0 I 1 → 2 > 0 in the first problem and I 1 → 2 < 0 I_{1\to2}<0 I 1 → 2 < 0 in the second. Their best gains are Γ > 0 \Gamma>0 Γ > 0 and 0 0 0 . Every rule ω ^ \hat{\omega} ω ^ that depends only on ( L i , α i , n i ) i = 1 , 2 (L_i,\alpha_i,n_i)_{i=1,2} ( L i , α i , n i ) i = 1 , 2 has expected regret E [ max ω Δ ( ω ) − Δ ( ω ^ ) ] ≥ Γ / 2 \mathbb{E}[\max_\omega\Delta(\omega)-\Delta(\hat{\omega})]\ge\Gamma/2 E [ max ω Δ ( ω ) − Δ ( ω ^ )] ≥ Γ/2 on at least one of the two problems that it cannot tell apart.
Proof. Fix p ∈ ( 0 , 1 ) p\in(0,1) p ∈ ( 0 , 1 ) , α ∈ ( 0 , 1 − p ) \alpha\in(0,1-p) α ∈ ( 0 , 1 − p ) , and ζ , γ > 0 \zeta,\gamma>0 ζ , γ > 0 with γ α > 2 ζ \gamma\alpha>2\zeta γ α > 2 ζ , and set:
Γ : = α ( γ α − 2 ζ ) > 0. \Gamma:= \alpha(\gamma\alpha-2\zeta)>0. Γ := α ( γ α − 2 ζ ) > 0.
Agent 1 draws x 1 ∼ Bern ( ν 1 ) x_1\sim\operatorname{Bern}(\nu_1) x 1 ∼ Bern ( ν 1 ) , then agent 2 draws x 2 ∼ Bern ( ν 2 ) x_2\sim\operatorname{Bern}(\nu_2) x 2 ∼ Bern ( ν 2 ) without seeing x 1 x_1 x 1 , and the terminal reward is:
− ζ ( x 1 + x 2 ) + γ ( x 1 − p ) ( x 2 − p ) in M + , − ζ ( x 1 + x 2 ) − γ ( x 1 − p ) ( x 2 − p ) in M − . -\zeta(x_1+x_2)+\gamma(x_1-p)(x_2-p) \quad\text{in } \mathcal{M}^{+},
\qquad
-\zeta(x_1+x_2)-\gamma(x_1-p)(x_2-p) \quad\text{in } \mathcal{M}^{-}. − ζ ( x 1 + x 2 ) + γ ( x 1 − p ) ( x 2 − p ) in M + , − ζ ( x 1 + x 2 ) − γ ( x 1 − p ) ( x 2 − p ) in M − .
By independence, E [ ( x 1 − p ) ( x 2 − p ) ] = ( ν 1 − p ) ( ν 2 − p ) \mathbb{E}[(x_1-p)(x_2-p)]=(\nu_1-p)(\nu_2-p) E [( x 1 − p ) ( x 2 − p )] = ( ν 1 − p ) ( ν 2 − p ) , so:
J ± ( ν 1 , ν 2 ) = − ζ ( ν 1 + ν 2 ) ± γ ( ν 1 − p ) ( ν 2 − p ) . J^{\pm}(\nu_1,\nu_2)=-\zeta(\nu_1+\nu_2)\pm\gamma(\nu_1-p)(\nu_2-p). J ± ( ν 1 , ν 2 ) = − ζ ( ν 1 + ν 2 ) ± γ ( ν 1 − p ) ( ν 2 − p ) .
The current policy is ( p , p ) (p,p) ( p , p ) , the candidate is ( p + α , p + α ) (p+\alpha,p+\alpha) ( p + α , p + α ) , the two problems of Proposition C.4 are M ± \mathcal{M}^{\pm} M ± with gains Δ ± \Delta^{\pm} Δ ± , and π ω = ( p + ω 1 α , p + ω 2 α ) \pi_\omega=(p+\omega_1\alpha,p+\omega_2\alpha) π ω = ( p + ω 1 α , p + ω 2 α ) switches agent i i i to the candidate iff ω i = 1 \omega_i=1 ω i = 1 , so:
Δ ± ( ω ) = J ± ( p + ω 1 α , p + ω 2 α ) − J ± ( p , p ) = − ζ α ω 1 − ζ α ω 2 ± γ α 2 ω 1 ω 2 . (29) \Delta^{\pm}(\omega)=J^{\pm}(p+\omega_1\alpha,p+\omega_2\alpha)-J^{\pm}(p,p)=-\zeta\alpha\omega_1-\zeta\alpha\omega_2\pm\gamma\alpha^2\omega_1\omega_2. \tag*{(29)} Δ ± ( ω ) = J ± ( p + ω 1 α , p + ω 2 α ) − J ± ( p , p ) = − ζ α ω 1 − ζ α ω 2 ± γ α 2 ω 1 ω 2 . (29)
Each agent acts once, so n 1 = n 2 = 1 n_1=n_2=1 n 1 = n 2 = 1 and α 1 = α 2 = TV ( Bern ( p ) , Bern ( p + α ) ) = 1 2 ( α + α ) = α \alpha_1=\alpha_2=\operatorname{TV}(\operatorname{Bern}(p),\operatorname{Bern}(p+\alpha))=\frac{1}{2}(\alpha+\alpha)=\alpha α 1 = α 2 = TV ( Bern ( p ) , Bern ( p + α )) = 2 1 ( α + α ) = α . For ω = e 1 \omega=e_1 ω = e 1 , Theorem B.6 (ii) keeps only k 2 = 0 k_2=0 k 2 = 0 and (iv) with n 1 = 1 n_1=1 n 1 = 1 keeps only k = e 1 k=e_1 k = e 1 , so L 1 = J e 1 = Δ ± ( e 1 ) = − ζ α L_1=J_{e_1}=\Delta^{\pm}(e_1)=-\zeta\alpha L 1 = J e 1 = Δ ± ( e 1 ) = − ζ α ; likewise L 2 = − ζ α L_2=-\zeta\alpha L 2 = − ζ α . Hence ( L i , α i , n i ) i = 1 , 2 (L_i,\alpha_i,n_i)_{i=1,2} ( L i , α i , n i ) i = 1 , 2 is the same in M + \mathcal{M}^{+} M + and M − \mathcal{M}^{-} M − . For a general ω \omega ω , Theorem B.6 (ii) and (iv) leave only k ∈ { e 1 , e 2 , e 1 + e 2 } k\in\{e_1,e_2,e_1+e_2\} k ∈ { e 1 , e 2 , e 1 + e 2 } , so:
Δ ± ( ω ) = ω 1 L 1 + ω 2 L 2 + ω 1 ω 2 ( I 1 → 2 + I 2 → 1 ) . \Delta^{\pm}(\omega)=\omega_1L_1+\omega_2L_2+\omega_1\omega_2(I_{1\to2}+I_{2\to1}). Δ ± ( ω ) = ω 1 L 1 + ω 2 L 2 + ω 1 ω 2 ( I 1 → 2 + I 2 → 1 ) .
Agent 2 never acts before agent 1, so I 2 → 1 = 0 I_{2\to1}=0 I 2 → 1 = 0 , and (29) gives I 1 → 2 = ± γ α 2 I_{1\to2}=\pm\gamma\alpha^2 I 1 → 2 = ± γ α 2 ; hence Δ ± ( ω ) = J ( 1 ) + J ( 2 ) \Delta^{\pm}(\omega)=J^{(1)}+J^{(2)} Δ ± ( ω ) = J ( 1 ) + J ( 2 ) at π ω \pi_\omega π ω . The profiles 0 0 0 , e 1 \mathbf{e}_1 e 1 , e 2 \mathbf{e}_2 e 2 , ( 1 , 1 ) (1,1) ( 1 , 1 ) have gains 0 0 0 , − ζ α -\zeta\alpha − ζ α , − ζ α -\zeta\alpha − ζ α , − 2 ζ α ± γ α 2 -2\zeta\alpha\pm\gamma\alpha^2 − 2 ζ α ± γ α 2 , and:
Δ + ( 1 , 1 ) = − 2 ζ α + γ α 2 = α ( γ α − 2 ζ ) = Γ > 0 , Δ − ( 1 , 1 ) = − α ( γ α + 2 ζ ) < 0. \Delta^{+}(1,1)=-2\zeta\alpha+\gamma\alpha^2=\alpha(\gamma\alpha-2\zeta)=\Gamma>0,\qquad\Delta^{-}(1,1)=-\alpha(\gamma\alpha+2\zeta)<0. Δ + ( 1 , 1 ) = − 2 ζ α + γ α 2 = α ( γ α − 2 ζ ) = Γ > 0 , Δ − ( 1 , 1 ) = − α ( γ α + 2 ζ ) < 0.
Hence max ω Δ + ( ω ) = Δ + ( 1 , 1 ) = Γ \max_{\omega}\Delta^{+}(\omega)=\Delta^{+}(1,1)=\Gamma max ω Δ + ( ω ) = Δ + ( 1 , 1 ) = Γ and max ω Δ − ( ω ) = Δ − ( 0 ) = 0 \max_{\omega}\Delta^{-}(\omega)=\Delta^{-}(0)=0 max ω Δ − ( ω ) = Δ − ( 0 ) = 0 .
Since ω ^ \hat{\omega} ω ^ receives identical inputs in M ± \mathcal{M}^{\pm} M ± , Pr ( ω ^ = ( 1 , 1 ) ) = φ \Pr(\hat{\omega}=(1,1))=\varphi Pr ( ω ^ = ( 1 , 1 )) = φ in both. In M + \mathcal{M}^{+} M + , Δ + ( ω ) ≤ 0 \Delta^{+}(\omega)\leq0 Δ + ( ω ) ≤ 0 for ω ≠ ( 1 , 1 ) \omega\neq(1,1) ω = ( 1 , 1 ) , so the regret is ≥ Γ \geq\Gamma ≥ Γ on { ω ^ ≠ ( 1 , 1 ) } \{\hat{\omega}\neq(1,1)\} { ω ^ = ( 1 , 1 )} ; in M − \mathcal{M}^{-} M − , ( 1 , 1 ) (1,1) ( 1 , 1 ) has regret α ( γ α + 2 ζ ) ≥ Γ \alpha(\gamma\alpha+2\zeta)\geq\Gamma α ( γ α + 2 ζ ) ≥ Γ . The expected regrets are at least ( 1 − φ ) Γ (1-\varphi)\Gamma ( 1 − φ ) Γ and φ Γ \varphi\Gamma φ Γ , so:
max ± E [ max ω Δ ± ( ω ) − Δ ± ( ω ^ ) ] ≥ max { ( 1 − φ ) Γ , φ Γ } ≥ 1 2 ( ( 1 − φ ) Γ + φ Γ ) = Γ 2 . \max_{\pm}\mathbb{E}\left[\max_{\omega}\Delta^{\pm}(\omega)-\Delta^{\pm}(\hat{\omega})\right]\geq\max\{(1-\varphi)\Gamma,\varphi\Gamma\}\geq\frac{1}{2}\bigl((1-\varphi)\Gamma+\varphi\Gamma\bigr)=\frac{\Gamma}{2}. ± max E [ ω max Δ ± ( ω ) − Δ ± ( ω ^ ) ] ≥ max {( 1 − φ ) Γ , φ Γ } ≥ 2 1 ( ( 1 − φ ) Γ + φ Γ ) = 2 Γ .
Per-agent inputs coincide for a complementary and a conflicting pair, which differ only in the sign of I 1 → 2 I_{1\to2} I 1 → 2 ; the order-two score sees this sign and, being exact here, selects the best profile in both problems.
Proof of Theorem 3.4 (Visit-Count Bound)
A Binomial Identity
For a count vector m = ( m i ) i ∈ N \mathbf{m}=(m_i)_{i\in\mathbb{N}} m = ( m i ) i ∈ N and k ∈ Z N \mathbf{k}\in\mathbb{Z}^{\mathbb{N}} k ∈ Z N , define:
B ( k ; m ) : = ∏ i ∈ N ( m i k i ) ( 2 α i ) k i . B(\mathbf{k};\mathbf{m}):=\prod_{i\in\mathbb{N}}\binom{m_i}{k_i}(2\alpha_i)^{k_i}. B ( k ; m ) := i ∈ N ∏ ( k i m i ) ( 2 α i ) k i .
with B ( k ; m ) : = 0 B(\mathbf{k};\mathbf{m}):=0 B ( k ; m ) := 0 if k i < 0 k_i<0 k i < 0 or k i > m i k_i>m_i k i > m i for some i i i , and 0 0 : = 1 0^0:=1 0 0 := 1 .
Lemma D.1 (Properties of B B B ). (i) If m i ≥ 1 m_i\geq1 m i ≥ 1 , then B ( k ; m ) = B ( k ; m − e i ) + 2 α i B ( k − e i ; m − e i ) B(\mathbf{k};\mathbf{m})=B(\mathbf{k};\mathbf{m}-\mathbf{e}_i)+2\alpha_iB(\mathbf{k}-\mathbf{e}_i;\mathbf{m}-\mathbf{e}_i) B ( k ; m ) = B ( k ; m − e i ) + 2 α i B ( k − e i ; m − e i ) .
(ii) If m ′ ≤ m \mathbf{m}'\leq\mathbf{m} m ′ ≤ m entrywise, then B ( k ; m ′ ) ≤ B ( k ; m ) B(\mathbf{k};\mathbf{m}')\leq B(\mathbf{k};\mathbf{m}) B ( k ; m ′ ) ≤ B ( k ; m ) .
(iii) For every r ≥ 1 r\geq1 r ≥ 1 , ∑ ∣ k ∣ = r B ( k ; m ) ≤ 1 r ! ( 2 ∑ i m i α i ) r \sum_{|\mathbf{k}|=r}B(\mathbf{k};\mathbf{m})\leq\frac{1}{r!}\left(2\sum_i m_i\alpha_i\right)^r ∑ ∣ k ∣ = r B ( k ; m ) ≤ r ! 1 ( 2 ∑ i m i α i ) r .
Proof. (i) Only the factor of agent i i i differs, and Pascal’s rule ( m i k i ) = ( m i − 1 k i ) + ( m i − 1 k i − 1 ) \binom{m_i}{k_i}=\binom{m_i-1}{k_i}+\binom{m_i-1}{k_i-1} ( k i m i ) = ( k i m i − 1 ) + ( k i − 1 m i − 1 ) gives:
( m i k i ) ( 2 α i ) k i = ( m i − 1 k i ) ( 2 α i ) k i + 2 α i ( m i − 1 k i − 1 ) ( 2 α i ) k i − 1 . \binom{m_i}{k_i}(2\alpha_i)^{k_i}=\binom{m_i-1}{k_i}(2\alpha_i)^{k_i}+2\alpha_i\binom{m_i-1}{k_i-1}(2\alpha_i)^{k_i-1}. ( k i m i ) ( 2 α i ) k i = ( k i m i − 1 ) ( 2 α i ) k i + 2 α i ( k i − 1 m i − 1 ) ( 2 α i ) k i − 1 .
Multiply by ∏ i ′ ≠ i ( m i ′ k i ′ ) ( 2 α i ′ ) k i ′ \prod_{i'\neq i}\binom{m_{i'}}{k_{i'}}(2\alpha_{i'})^{k_{i'}} ∏ i ′ = i ( k i ′ m i ′ ) ( 2 α i ′ ) k i ′ ; for k i = 0 k_i=0 k i = 0 the last term is 0 0 0 .
(ii) m ↦ ( m k ) \mathbf{m}\mapsto\binom{\mathbf{m}}{\mathbf{k}} m ↦ ( k m ) is nondecreasing and every factor is ≥ 0 \geq0 ≥ 0 .
(iii) With x i : = 2 m i α i x_i:=2m_i\alpha_i x i := 2 m i α i :
∑ ∣ k ∣ = r B ( k ; m ) ≤ ∑ ∣ k ∣ = r ∏ i m i k i ( 2 α i ) k i k i ! = 1 r ! ∑ ∣ k ∣ = r r ! ∏ i k i ! ∏ i x i k i ≤ 1 r ! ( ∑ i x i ) r . \sum_{|\mathbf{k}|=r}B(\mathbf{k};\mathbf{m})\leq\sum_{|\mathbf{k}|=r}\prod_i\frac{m_i^{k_i}(2\alpha_i)^{k_i}}{k_i!}=\frac{1}{r!}\sum_{|\mathbf{k}|=r}\frac{r!}{\prod_i k_i!}\prod_i x_i^{k_i}\leq\frac{1}{r!}\left(\sum_i x_i\right)^r. ∣ k ∣ = r ∑ B ( k ; m ) ≤ ∣ k ∣ = r ∑ i ∏ k i ! m i k i ( 2 α i ) k i = r ! 1 ∣ k ∣ = r ∑ ∏ i k i ! r ! i ∏ x i k i ≤ r ! 1 ( i ∑ x i ) r .
where (a) uses ( m k ) ≤ m k / k ! \binom{m}{k}\leq m^k/k! ( k m ) ≤ m k / k ! and (b) is the multinomial theorem.
The Coefficient Bound
Theorem 3.4 (Visit-count bound). Under Assumption 3.1, with workers fixed, for every integer r ≥ 1 r\geq1 r ≥ 1 :
∣ J ( π ′ ) − J ( π ) − ∑ m = 1 r J ( m ) ∣ ≤ G ˉ ∑ m > r β m m ! . (30) \left|J(\pi')-J(\pi)-\sum_{m=1}^{r}J^{(m)}\right|\leq\bar{G}\sum_{m>r}\frac{\beta^m}{m!}. \tag*{(30)} J ( π ′ ) − J ( π ) − m = 1 ∑ r J ( m ) ≤ G ˉ m > r ∑ m ! β m . (30)
We prove a stronger, per-coefficient bound.
Theorem D.2 (Visit-count bound, general form). Under Assumption 3.1 , for every k ≠ 0 \mathbf{k} \ne0 k = 0 :
∣ J k ∣ ≤ min { ∑ t = 1 H r ˉ t ∏ i ( n i , t k i ) ( 2 α i ) k i , G ˉ ∏ i ( n i k i ) ( 2 α i ) k i } . (31) |\mathcal{J}_{\mathbf{k}}| \le\min\left\{ \sum_{t=1}^{H} \bar{r}_t \prod_i \binom{n_{i,t}}{k_i}(2\alpha_i)^{k_i},\ \bar{G}\prod_i \binom{n_i}{k_i}(2\alpha_i)^{k_i} \right\}. \tag*{(31)} ∣ J k ∣ ≤ min { t = 1 ∑ H r ˉ t i ∏ ( k i n i , t ) ( 2 α i ) k i , G ˉ i ∏ ( k i n i ) ( 2 α i ) k i } . (31) Consequently, for every r ≥ 1 r \ge1 r ≥ 1 :
∣ J ( π ′ ) − J ( π ) − ∑ m = 1 r J ( m ) ∣ ≤ min { ∑ t r ˉ t ∑ ∣ k ∣ > r B ( k ; ( n i , t ) i ) , G ˉ ∑ ∣ k ∣ > r B ( k ; ( n i ) i ) } ≤ G ˉ ∑ m > r 1 m ! ( 2 ∑ i ∈ N n i α i ) m . (32) \left|\mathcal{J}(\pi')-\mathcal{J}(\pi)-\sum_{m=1}^{r}\mathcal{J}^{(m)}\right| \le\min\left\{ \sum_t \bar{r}_t \sum_{|\mathbf{k}|>r} B(\mathbf{k};(n_{i,t})_i),\ \bar{G}\sum_{|\mathbf{k}|>r} B(\mathbf{k};(n_i)_i) \right\} \le\bar{G}\sum_{m>r}\frac{1}{m!}\left(2\sum_{i\in\mathcal{N}}n_i\alpha_i\right)^m. \tag*{(32)} J ( π ′ ) − J ( π ) − m = 1 ∑ r J ( m ) ≤ min ⎩ ⎨ ⎧ t ∑ r ˉ t ∣ k ∣ > r ∑ B ( k ; ( n i , t ) i ) , G ˉ ∣ k ∣ > r ∑ B ( k ; ( n i ) i ) ⎭ ⎬ ⎫ ≤ G ˉ m > r ∑ m ! 1 ( 2 i ∈ N ∑ n i α i ) m . (32)
Proof. Fix a reward step t t t . For t ′ ∈ [ t ] t' \in[t] t ′ ∈ [ t ] define:
Φ t ′ ( k ) : = E π [ ∑ S ⊆ { t ′ , … , t } : k ( S ) = k ∏ t ′ ′ ∈ S ∣ ρ t ′ ′ − 1 ∣ ∣ τ t ′ − 1 , s t ′ ] , Φ t + 1 ( k ) : = 1 { k = 0 } , \Phi_{t'}(\mathbf{k}) := \mathbb{E}_{\pi}\left[\sum_{\substack{S\subseteq\{t',\ldots,t\}:\\ \mathbf{k}(S)=\mathbf{k}}}\prod_{t''\in S}|\rho_{t''}-1|\mathrel{\Big|}\tau_{t'-1},s_{t'}\right], \qquad\Phi_{t+1}(\mathbf{k}):=\mathbf{1}\{\mathbf{k}=0\}, Φ t ′ ( k ) := E π S ⊆ { t ′ , … , t } : k ( S ) = k ∑ t ′′ ∈ S ∏ ∣ ρ t ′′ − 1∣ τ t ′ − 1 , s t ′ , Φ t + 1 ( k ) := 1 { k = 0 } ,
and let m i ( s t ′ ) m_i(s_{t'}) m i ( s t ′ ) be the maximum of ∣ { t ′ ′ ∈ { t ′ , … , t } : i t ′ ′ = i } ∣ |\{t''\in\{t',\ldots,t\}:i_{t''}=i\}| ∣ { t ′′ ∈ { t ′ , … , t } : i t ′′ = i } ∣ over trajectories of π \pi π through s t ′ s_{t'} s t ′ . If s t ′ s_{t'} s t ′ is absorbing, then ρ t ′ = 1 \rho_{t'}=1 ρ t ′ = 1 , every S ∋ t ′ S\ni t' S ∋ t ′ contributes 0 0 0 , Φ t ′ ( k ) = E π [ Φ t ′ + 1 ( k ) ∣ τ t ′ − 1 , s t ′ ] \Phi_{t'}(\mathbf{k})=\mathbb{E}_{\pi}[\Phi_{t'+1}(\mathbf{k})\mid\tau_{t'-1},s_{t'}] Φ t ′ ( k ) = E π [ Φ t ′ + 1 ( k ) ∣ τ t ′ − 1 , s t ′ ] , and m ( s t ′ ) ≥ m ( s t ′ + 1 ) \mathbf{m}(s_{t'})\ge\mathbf{m}(s_{t'+1}) m ( s t ′ ) ≥ m ( s t ′ + 1 ) , so the claim (35) below passes from t ′ + 1 t'+1 t ′ + 1 to t ′ t' t ′ by Lemma D.1 (ii). Otherwise, let i = i t ′ i=i_{t'} i = i t ′ . Splitting by t ′ ∉ S t'\notin S t ′ ∈ / S (S ⊆ { t ′ + 1 , … , t } S\subseteq\{t'+1,\ldots,t\} S ⊆ { t ′ + 1 , … , t } , same counts) and S = { t ′ } ∪ S ′ S=\{t'\}\cup S' S = { t ′ } ∪ S ′ (S ′ ⊆ { t ′ + 1 , … , t } S'\subseteq\{t'+1,\ldots,t\} S ′ ⊆ { t ′ + 1 , … , t } , k ( S ′ ) = k − e i \mathbf{k}(S')=\mathbf{k}-\mathbf{e}_i k ( S ′ ) = k − e i ):
∑ S ⊆ { t ′ , … , t } : k ( S ) = k ∏ t ′ ′ ∈ S ∣ ρ t ′ ′ − 1 ∣ = ∑ S ′ ⊆ { t ′ + 1 , … , t } : k ( S ′ ) = k ∏ t ′ ′ ∈ S ′ ∣ ρ t ′ ′ − 1 ∣ + ∣ ρ t ′ − 1 ∣ ∑ S ′ ⊆ { t ′ + 1 , … , t } : k ( S ′ ) = k − e i ∏ t ′ ′ ∈ S ′ ∣ ρ t ′ ′ − 1 ∣ . \sum_{\substack{S\subseteq\{t',\ldots,t\}:\\\mathbf{k}(S)=\mathbf{k}}}\prod_{t''\in S}|\rho_{t''}-1|
=
\sum_{\substack{S'\subseteq\{t'+1,\ldots,t\}:\\\mathbf{k}(S')=\mathbf{k}}}\prod_{t''\in S'}|\rho_{t''}-1|
+
|\rho_{t'}-1|
\sum_{\substack{S'\subseteq\{t'+1,\ldots,t\}:\\\mathbf{k}(S')=\mathbf{k}-\mathbf{e}_i}}\prod_{t''\in S'}|\rho_{t''}-1|. S ⊆ { t ′ , … , t } : k ( S ) = k ∑ t ′′ ∈ S ∏ ∣ ρ t ′′ − 1∣ = S ′ ⊆ { t ′ + 1 , … , t } : k ( S ′ ) = k ∑ t ′′ ∈ S ′ ∏ ∣ ρ t ′′ − 1∣ + ∣ ρ t ′ − 1∣ S ′ ⊆ { t ′ + 1 , … , t } : k ( S ′ ) = k − e i ∑ t ′′ ∈ S ′ ∏ ∣ ρ t ′′ − 1∣.
Taking E π [ ⋅ ∣ τ t ′ − 1 , s t ′ ] \mathbb{E}_{\pi}[\cdot\mid\tau_{t'-1},s_{t'}] E π [ ⋅ ∣ τ t ′ − 1 , s t ′ ] , with the tower rule over ( a t ′ , s t ′ + 1 ) (a_{t'},s_{t'+1}) ( a t ′ , s t ′ + 1 ) and ∣ ρ t ′ − 1 ∣ |\rho_{t'}-1| ∣ ρ t ′ − 1∣ a function of ( s t ′ , a t ′ ) (s_{t'},a_{t'}) ( s t ′ , a t ′ ) :
Φ t ′ ( k ) = E π [ Φ t ′ + 1 ( k ) ∣ τ t ′ − 1 , s t ′ ] + E π [ ∣ ρ t ′ − 1 ∣ Φ t ′ + 1 ( k − e i ) ∣ τ t ′ − 1 , s t ′ ] . (33) \Phi_{t'}(\mathbf{k})=\mathbb{E}_{\pi}\left[\Phi_{t'+1}(\mathbf{k})\mid\tau_{t'-1},s_{t'}\right]+\mathbb{E}_{\pi}\left[|\rho_{t'}-1|\Phi_{t'+1}(\mathbf{k}-\mathbf{e}_i)\mid\tau_{t'-1},s_{t'}\right]. \tag*{(33)} Φ t ′ ( k ) = E π [ Φ t ′ + 1 ( k ) ∣ τ t ′ − 1 , s t ′ ] + E π [ ∣ ρ t ′ − 1∣ Φ t ′ + 1 ( k − e i ) ∣ τ t ′ − 1 , s t ′ ] . (33)
Since states determine their history, every trajectory of π \pi π through a next state s t ′ + 1 s_{t'+1} s t ′ + 1 passes through s t ′ s_{t'} s t ′ and has i t ′ = i i_{t'}=i i t ′ = i , so its counts on { t ′ , … , t } \{t',\ldots,t\} { t ′ , … , t } are its counts on { t ′ + 1 , … , t } \{t'+1,\ldots,t\} { t ′ + 1 , … , t } plus e i \mathbf{e}_i e i . Maximizing:
m ( s t ′ ) ≥ m ( s t ′ + 1 ) + e i , in particular m i ( s t ′ ) ≥ 1. (34) \mathbf{m}(s_{t'})\ge\mathbf{m}(s_{t'+1})+\mathbf{e}_i,\qquad\text{in particular }m_i(s_{t'})\ge1. \tag*{(34)} m ( s t ′ ) ≥ m ( s t ′ + 1 ) + e i , in particular m i ( s t ′ ) ≥ 1. (34)
We show by backward induction on t ′ t' t ′ that, for all t ′ t' t ′ and k \mathbf{k} k :
Φ t ′ ( k ) ≤ B ( k ; m ( s t ′ ) ) . (35) \Phi_{t'}(\mathbf{k})\le B(\mathbf{k};\mathbf{m}(s_{t'})). \tag*{(35)} Φ t ′ ( k ) ≤ B ( k ; m ( s t ′ )) . (35)
At t ′ = t + 1 t'=t+1 t ′ = t + 1 , m = 0 \mathbf{m}=0 m = 0 and B ( k ; 0 ) = 1 { k = 0 } = Φ t + 1 ( k ) B(\mathbf{k};0)=\mathbf{1}\{\mathbf{k}=0\}=\Phi_{t+1}(\mathbf{k}) B ( k ; 0 ) = 1 { k = 0 } = Φ t + 1 ( k ) . Given (35) at t ′ + 1 t'+1 t ′ + 1 , (34) and Lemma D.1 (ii) give, for every next state s t ′ + 1 s_{t'+1} s t ′ + 1 of a trajectory of π \pi π through s t ′ s_{t'} s t ′ :
Φ t ′ + 1 ( k ) ≤ B ( k ; m ( s t ′ + 1 ) ) ≤ B ( k ; m ( s t ′ ) − e i ) , Φ t ′ + 1 ( k − e i ) ≤ B ( k − e i ; m ( s t ′ ) − e i ) . \Phi_{t'+1}(\mathbf{k})\le B(\mathbf{k};\mathbf{m}(s_{t'+1}))\le B(\mathbf{k};\mathbf{m}(s_{t'})-\mathbf{e}_i),\qquad
\Phi_{t'+1}(\mathbf{k}-\mathbf{e}_i)\le B(\mathbf{k}-\mathbf{e}_i;\mathbf{m}(s_{t'})-\mathbf{e}_i). Φ t ′ + 1 ( k ) ≤ B ( k ; m ( s t ′ + 1 )) ≤ B ( k ; m ( s t ′ ) − e i ) , Φ t ′ + 1 ( k − e i ) ≤ B ( k − e i ; m ( s t ′ ) − e i ) .
These bounds are free of ( a t ′ , s t ′ + 1 ) (a_{t'},s_{t'+1}) ( a t ′ , s t ′ + 1 ) , so (33) gives:
Φ t ′ ( k ) ≤ B ( k ; m ( s t ′ ) − e i ) + B ( k − e i ; m ( s t ′ ) − e i ) E π [ ∣ ρ t ′ − 1 ∣ ∣ τ t ′ − 1 , s t ′ ] ≤ ( a ) B ( k ; m ( s t ′ ) − e i ) + 2 α i B ( k − e i ; m ( s t ′ ) − e i ) ≤ ( b ) B ( k ; m ( s t ′ ) ) . \begin{aligned}
\Phi_{t'}(\mathbf{k})&\le B(\mathbf{k};\mathbf{m}(s_{t'})-\mathbf{e}_i)+B(\mathbf{k}-\mathbf{e}_i;\mathbf{m}(s_{t'})-\mathbf{e}_i)\mathbb{E}_{\pi}\left[|\rho_{t'}-1|\mid\tau_{t'-1},s_{t'}\right]\\
&\overset{(a)}{\le}B(\mathbf{k};\mathbf{m}(s_{t'})-\mathbf{e}_i)+2\alpha_iB(\mathbf{k}-\mathbf{e}_i;\mathbf{m}(s_{t'})-\mathbf{e}_i)\\
&\overset{(b)}{\le}B(\mathbf{k};\mathbf{m}(s_{t'})).
\end{aligned} Φ t ′ ( k ) ≤ B ( k ; m ( s t ′ ) − e i ) + B ( k − e i ; m ( s t ′ ) − e i ) E π [ ∣ ρ t ′ − 1∣ ∣ τ t ′ − 1 , s t ′ ] ≤ ( a ) B ( k ; m ( s t ′ ) − e i ) + 2 α i B ( k − e i ; m ( s t ′ ) − e i ) ≤ ( b ) B ( k ; m ( s t ′ )) .
where (a) uses Lemma B.4 and (b) is Lemma D.1 (i), valid since m i ( s t ′ ) ≥ 1 m_i(s_{t'})\ge1 m i ( s t ′ ) ≥ 1 by (34) .
By (21) and (22) , which hold for each k \mathbf{k} k separately, the reward form (20) holds termwise:
J k = ∑ t = 1 H E π [ ∑ S ⊆ [ t ] : k ( S ) = k ∏ t ′ ′ ∈ S ( ρ t ′ ′ − 1 ) r t ( s t , a t ) ] . \mathcal{J}_{\mathbf{k}}=\sum_{t=1}^{H}\mathbb{E}_{\pi}\left[\sum_{\substack{S\subseteq[t]:\\\mathbf{k}(S)=\mathbf{k}}}\prod_{t''\in S}(\rho_{t''}-1)r_t(s_t,a_t)\right]. J k = t = 1 ∑ H E π S ⊆ [ t ] : k ( S ) = k ∑ t ′′ ∈ S ∏ ( ρ t ′′ − 1 ) r t ( s t , a t ) .
First bound in (31) :
∣ J k ∣ ≤ ∑ t = 1 H r ˉ t E π [ Φ 1 ( k ) ] ≤ ( a ) ∑ t = 1 H r ˉ t E π [ B ( k ; m ( s 1 ) ) ] ≤ ( b ) ∑ t = 1 H r ˉ t ∏ i ( n i , t k i ) ( 2 α i ) k i , |\mathcal{J}_{\mathbf{k}}|\le\sum_{t=1}^{H}\bar{r}_t\mathbb{E}_{\pi}[\Phi_1(\mathbf{k})]\overset{(a)}{\le}\sum_{t=1}^{H}\bar{r}_t\mathbb{E}_{\pi}[B(\mathbf{k};\mathbf{m}(s_1))]\overset{(b)}{\le}\sum_{t=1}^{H}\bar{r}_t\prod_i\binom{n_{i,t}}{k_i}(2\alpha_i)^{k_i}, ∣ J k ∣ ≤ t = 1 ∑ H r ˉ t E π [ Φ 1 ( k )] ≤ ( a ) t = 1 ∑ H r ˉ t E π [ B ( k ; m ( s 1 ))] ≤ ( b ) t = 1 ∑ H r ˉ t i ∏ ( k i n i , t ) ( 2 α i ) k i ,
where (a) uses ∣ r t ∣ ≤ r ˉ t |r_t|\le\bar{r}_t ∣ r t ∣ ≤ r ˉ t , (b) is (35) at t ′ = 1 t'=1 t ′ = 1 for reward step t t t , and (c) uses m i ( s 1 ) ≤ n i , t m_i(s_1)\le n_{i,t} m i ( s 1 ) ≤ n i , t and Lemma D.1 (ii). Second bound in (31) : let X t : = ∑ S ⊆ [ t ] , k ( S ) = k ∏ t ′ ∈ S ∣ ρ t ′ − 1 ∣ X_t := \sum_{S\subseteq[t],\,k(S)=k} \prod_{t'\in S}|\rho_{t'}-1| X t := ∑ S ⊆ [ t ] , k ( S ) = k ∏ t ′ ∈ S ∣ ρ t ′ − 1∣ . The summands are ≥ 0 \ge0 ≥ 0 and { S ⊆ [ t ] } ⊆ { S ⊆ [ t + 1 ] } \{S\subseteq[t]\}\subseteq\{S\subseteq[t+1]\} { S ⊆ [ t ]} ⊆ { S ⊆ [ t + 1 ]} , so X t ≤ X H X_t\le X_H X t ≤ X H , and the integrand of the reward form satisfies:
∑ t = 1 H ∣ r t ( s t , a t ) ∣ X t ≤ X H ∑ t = 1 H ∣ r t ( s t , a t ) ∣ ≤ G ˉ X H . \sum_{t=1}^{H}|r_t(s_t,a_t)|X_t\le X_H\sum_{t=1}^{H}|r_t(s_t,a_t)|\le\bar{G}X_H. t = 1 ∑ H ∣ r t ( s t , a t ) ∣ X t ≤ X H t = 1 ∑ H ∣ r t ( s t , a t ) ∣ ≤ G ˉ X H .
By this display, (35) at t ′ = 1 t'=1 t ′ = 1 with reward step H H H , and Lemma D.1 (ii):
∣ J k ∣ ≤ G ˉ E π [ X H ] = G ˉ E π [ Φ 1 ( k ) ] ≤ G ˉ ∏ i ( n i k i ) ( 2 α i ) k i . |J_k|\le\bar{G}\mathbb{E}_{\pi}[X_H]=\bar{G}\mathbb{E}_{\pi}[\Phi_1(k)]\le\bar{G}\prod_i\binom{n_i}{k_i}(2\alpha_i)^{k_i}. ∣ J k ∣ ≤ G ˉ E π [ X H ] = G ˉ E π [ Φ 1 ( k )] ≤ G ˉ i ∏ ( k i n i ) ( 2 α i ) k i .
Tail: by Theorem B.6 (i) and (15) , J ( π ′ ) − J ( π ) − ∑ m ≤ r J ( m ) = ∑ ∣ k ∣ > r J k J(\pi')-J(\pi)-\sum_{m\le r}J^{(m)}=\sum_{|k|>r}J_k J ( π ′ ) − J ( π ) − ∑ m ≤ r J ( m ) = ∑ ∣ k ∣ > r J k , and the triangle inequality with (31) gives both terms of the minimum in (32) . For the second term:
G ˉ ∑ ∣ k ∣ > r B ( k ; ( n i ) i ) = G ˉ ∑ m > r ∑ ∣ k ∣ = m B ( k ; ( n i ) i ) ≤ ( a ) G ˉ ∑ m > r 1 m ! ( 2 ∑ i n i α i ) m , \bar{G}\sum_{|k|>r}B(k;(n_i)_i)=\bar{G}\sum_{m>r}\sum_{|k|=m}B(k;(n_i)_i)\overset{(a)}{\le}\bar{G}\sum_{m>r}\frac{1}{m!}\left(2\sum_i n_i\alpha_i\right)^m, G ˉ ∣ k ∣ > r ∑ B ( k ; ( n i ) i ) = G ˉ m > r ∑ ∣ k ∣ = m ∑ B ( k ; ( n i ) i ) ≤ ( a ) G ˉ m > r ∑ m ! 1 ( 2 i ∑ n i α i ) m ,
where (a) is Lemma D.1 (iii) with m = ( n i ) i m=(n_i)_i m = ( n i ) i .
Proof of Theorem 3.4 and of (6). With workers fixed, α j = 0 \alpha_j=0 α j = 0 for j ∈ [ M ] j\in[M] j ∈ [ M ] , so 2 ∑ i n i α i = 2 n 0 α 0 ≤ β 2\sum_i n_i\alpha_i=2n_0\alpha_0\le\beta 2 ∑ i n i α i = 2 n 0 α 0 ≤ β by n 0 ≤ K + 1 n_0\le K+1 n 0 ≤ K + 1 . The last inequality of (32) and the monotonicity of x ↦ ∑ m > r x m / m ! x\mapsto\sum_{m>r}x^m/m! x ↦ ∑ m > r x m / m ! on x ≥ 0 x\ge0 x ≥ 0 give Theorem 3.4. For r = 2 r=2 r = 2 , ∑ m > 2 β m / m ! = e β − 1 − β − β 2 / 2 \sum_{m>2}\beta^m/m!=e^\beta-1-\beta-\beta^2/2 ∑ m > 2 β m / m ! = e β − 1 − β − β 2 /2 , so (5) gives J ( π ′ ) − J ( π ) ≥ J ( 1 ) + J ( 2 ) − G ˉ ( e β − 1 − β − β 2 / 2 ) J(\pi')-J(\pi)\ge J^{(1)}+J^{(2)}-\bar{G}(e^\beta-1-\beta-\beta^2/2) J ( π ′ ) − J ( π ) ≥ J ( 1 ) + J ( 2 ) − G ˉ ( e β − 1 − β − β 2 /2 ) , which is (6) .
Depth of the Expansion
Corollary D.4 (Depth). If every trajectory of π \pi π contains at most r r r steps of changed agents, then J ( m ) = 0 J^{(m)}=0 J ( m ) = 0 for all m > r m>r m > r .
Proof. On any trajectory of π \pi π , at most r r r steps belong to changed agents, so ∣ S ∣ > r |S|>r ∣ S ∣ > r implies that some t ′ ∈ S t'\in S t ′ ∈ S has π i t ′ t ′ = π i t ′ t ′ \pi^{t'}_{i_{t'}}=\pi^{t'}_{i_{t'}} π i t ′ t ′ = π i t ′ t ′ , hence ρ t ′ = 1 \rho_{t'}=1 ρ t ′ = 1 and ∏ t ∈ S ( ρ t − 1 ) = 0 \prod_{t\in S}(\rho_t-1)=0 ∏ t ∈ S ( ρ t − 1 ) = 0 . So every term of J ( m ) J^{(m)} J ( m ) in (4) with m > r m>r m > r vanishes.
By Corollary D.4 , a one-shot router or workflow designer has r = 1 r=1 r = 1 (J ( 1 ) J^{(1)} J ( 1 ) is exact), a fixed pipeline with each trained agent acting once has r = ∣ { trained agents } ∣ r=|\{\text{trained agents}\}| r = ∣ { trained agents } ∣ and only terms across agents, and a multi-turn orchestrator has r = K + 1 r=K+1 r = K + 1 .
Proofs for the Algorithm
Throughout, π θ , η ′ \pi'_{\theta,\eta} π θ , η ′ applies (7) to the orchestrator and keeps π 1 , … , π M \pi^1,\ldots,\pi^M π 1 , … , π M , so ρ t = 1 \rho_t=1 ρ t = 1 whenever i t ≠ 0 i_t\ne0 i t = 0 .
The Bounded Update
Proposition 3.5 (Bounded ratios). For any θ \theta θ and η ∈ [ 0 , 1 ] \eta\in[0,1] η ∈ [ 0 , 1 ] , (7) defines a valid action distribution at every orchestrator state s s s , satisfying π θ , η ′ ( a ∣ s ) ≥ π ( a ∣ s ) 2 \pi'_{\theta,\eta}(a\mid s)\ge\pi(a\mid s)^2 π θ , η ′ ( a ∣ s ) ≥ π ( a ∣ s ) 2 . Moreover, on the support of π \pi π , its likelihood ratio satisfies ∣ ρ t − 1 ∣ ≤ η |\rho_t-1|\le\eta ∣ ρ t − 1∣ ≤ η , and its total variation distance from π ( ⋅ ∣ s ) \pi(\cdot\mid s) π ( ⋅ ∣ s ) is at most η / 4 \eta/4 η /4 .
Proof. Fix an orchestrator state s s s and write:
p a : = π ( a ∣ s ) , f a : = f θ ( s , a ) ∈ [ − 1 , 1 ] , f ˉ : = ∑ a ′ p a ′ f a ′ , u a : = 1 2 ( f a − f ˉ ) . p_a:=\pi(a\mid s),\qquad f_a:=f_\theta(s,a)\in[-1,1],\qquad\bar{f}:=\sum_{a'}p_{a'}f_{a'},\qquad u_a:=\frac{1}{2}(f_a-\bar{f}). p a := π ( a ∣ s ) , f a := f θ ( s , a ) ∈ [ − 1 , 1 ] , f ˉ := a ′ ∑ p a ′ f a ′ , u a := 2 1 ( f a − f ˉ ) .
Then π θ , η ′ ( a ∣ s ) = p a ( 1 + η u a ) \pi'_{\theta,\eta}(a \mid s)=p_a(1+\eta u_a) π θ , η ′ ( a ∣ s ) = p a ( 1 + η u a ) and:
∑ a p a u a = 1 2 ( ∑ a p a f a − f ˉ ∑ a p a ) = 1 2 ( f ˉ − f ˉ ) = 0 , ∑ a π θ , η ′ ( a ∣ s ) = ∑ a p a + η ∑ a p a u a = 1. \sum_a p_a u_a=\frac{1}{2}\left(\sum_a p_a f_a-\bar{f}\sum_a p_a\right)=\frac{1}{2}(\bar{f}-\bar{f})=0,\qquad\sum_a \pi'_{\theta,\eta}(a \mid s)=\sum_a p_a+\eta\sum_a p_a u_a=1. a ∑ p a u a = 2 1 ( a ∑ p a f a − f ˉ a ∑ p a ) = 2 1 ( f ˉ − f ˉ ) = 0 , a ∑ π θ , η ′ ( a ∣ s ) = a ∑ p a + η a ∑ p a u a = 1.
Since f a − f ˉ = ( 1 − p a ) f a − ∑ a ′ ≠ a p a ′ f a ′ f_a-\bar{f}=(1-p_a)f_a-\sum_{a'\ne a}p_{a'}f_{a'} f a − f ˉ = ( 1 − p a ) f a − ∑ a ′ = a p a ′ f a ′ with ∣ ( 1 − p a ) f a ∣ ≤ 1 − p a \lvert(1-p_a)f_a\rvert\le1-p_a ∣( 1 − p a ) f a ∣ ≤ 1 − p a and ∣ ∑ a ′ ≠ a p a ′ f a ′ ∣ ≤ 1 − p a \left\lvert\sum_{a'\ne a}p_{a'}f_{a'}\right\rvert\le1-p_a ∑ a ′ = a p a ′ f a ′ ≤ 1 − p a , and η ≤ 1 \eta\le1 η ≤ 1 :
∣ f a − f ˉ ∣ ≤ 2 ( 1 − p a ) , ∣ u a ∣ ≤ 1 − p a ≤ 1 , π θ , η ′ ( a ∣ s ) = p a ( 1 + η u a ) ≥ p a ( 1 − ( 1 − p a ) ) = p a 2 . \lvert f_a-\bar{f}\rvert\le2(1-p_a),\qquad\lvert u_a\rvert\le1-p_a\le1,\qquad\pi'_{\theta,\eta}(a\mid s)=p_a(1+\eta u_a)\ge p_a(1-(1-p_a))=p_a^2. ∣ f a − f ˉ ∣ ≤ 2 ( 1 − p a ) , ∣ u a ∣ ≤ 1 − p a ≤ 1 , π θ , η ′ ( a ∣ s ) = p a ( 1 + η u a ) ≥ p a ( 1 − ( 1 − p a )) = p a 2 .
Hence π θ , η ′ ( ⋅ ∣ s ) \pi'_{\theta,\eta}(\cdot\mid s) π θ , η ′ ( ⋅ ∣ s ) is a distribution on A ( s ) \mathcal{A}(s) A ( s ) , positive wherever π ( ⋅ ∣ s ) \pi(\cdot\mid s) π ( ⋅ ∣ s ) is. If ( s t , a t ) = ( s , a ) (s_t,a_t)=(s,a) ( s t , a t ) = ( s , a ) with i t = 0 i_t=0 i t = 0 , then ρ t = π θ , η ′ ( a ∣ s ) / p a = 1 + η u a \rho_t=\pi'_{\theta,\eta}(a\mid s)/p_a=1+\eta u_a ρ t = π θ , η ′ ( a ∣ s ) / p a = 1 + η u a , so ∣ ρ t − 1 ∣ = η ∣ u a ∣ ≤ η \lvert\rho_t-1\rvert=\eta\lvert u_a\rvert\le\eta ∣ ρ t − 1 ∣ = η ∣ u a ∣ ≤ η . Finally:
T V = 1 2 ∑ a ∣ p a η u a ∣ = η 2 ∑ a p a ∣ f a − f ˉ ∣ ≤ ( a ) η 4 ( ∑ a p a ( f a − f ˉ ) 2 ) 1 / 2 = η 4 ( ∑ a p a f a 2 − f ˉ 2 ) 1 / 2 ≤ ( b ) η 4 . \begin{aligned}
\mathrm{TV}&=\frac{1}{2}\sum_a\lvert p_a\eta u_a\rvert=\frac{\eta}{2}\sum_a p_a\lvert f_a-\bar{f}\rvert\overset{(a)}{\le}\frac{\eta}{4}\left(\sum_a p_a(f_a-\bar{f})^2\right)^{1/2}\\
&=\frac{\eta}{4}\left(\sum_a p_a f_a^2-\bar{f}^2\right)^{1/2}\overset{(b)}{\le}\frac{\eta}{4}.
\end{aligned} TV = 2 1 a ∑ ∣ p a η u a ∣ = 2 η a ∑ p a ∣ f a − f ˉ ∣ ≤ ( a ) 4 η ( a ∑ p a ( f a − f ˉ ) 2 ) 1/2 = 4 η ( a ∑ p a f a 2 − f ˉ 2 ) 1/2 ≤ ( b ) 4 η .
where (a) is the Cauchy–Schwarz inequality with weights p a p_a p a and (b) uses f a 2 ≤ 1 f_a^2\le1 f a 2 ≤ 1 .
Uniformly over θ \theta θ , every ratio is a known function of ( θ , η ) (\theta,\eta) ( θ , η ) and the logged probabilities of π \pi π , Assumption 3.1 holds with W = 1 + η W=1+\eta W = 1 + η , and α 0 ≤ η / 4 \alpha_0\le\eta/4 α 0 ≤ η /4 , α j = 0 \alpha_j=0 α j = 0 for j ∈ [ M ] j\in[M] j ∈ [ M ] .
The First Two Orders and the Remainder
Lemma E.1 (Quadratic form and remainder). For the bounded update with parameters ( θ , η ) (\theta,\eta) ( θ , η ) , with μ 1 , μ 2 \mu_1,\mu_2 μ 1 , μ 2 as in (8) :
J ( 1 ) = η μ 1 ( θ ) , J ( 2 ) = η 2 μ 2 ( θ ) , ∣ J ( π θ , η ′ ) − J ( π ) − η μ 1 ( θ ) − η 2 μ 2 ( θ ) ∣ ≤ ϵ ~ ( η ) ≤ ϵ ( η ) , J^{(1)}=\eta\mu_1(\theta),\qquad J^{(2)}=\eta^2\mu_2(\theta),\qquad\left\lvert J(\pi'_{\theta,\eta})-J(\pi)-\eta\mu_1(\theta)-\eta^2\mu_2(\theta)\right\rvert\le\tilde{\epsilon}(\eta)\le\epsilon(\eta), J ( 1 ) = η μ 1 ( θ ) , J ( 2 ) = η 2 μ 2 ( θ ) , J ( π θ , η ′ ) − J ( π ) − η μ 1 ( θ ) − η 2 μ 2 ( θ ) ≤ ϵ ~ ( η ) ≤ ϵ ( η ) , where ϵ ( η ) \epsilon(\eta) ϵ ( η ) is (10) and:
ϵ ~ ( η ) : = min { ∑ t = 1 H r ˉ t ∑ m ≥ 3 ( n 0 , t m ) ( η 2 ) m , G ˉ ∑ m ≥ 3 ( K + 1 m ) ( η 2 ) m } . \tilde{\epsilon}(\eta):=\min\left\{\sum_{t=1}^{H}\bar{r}_t\sum_{m\ge3}\binom{n_{0,t}}{m}\left(\frac{\eta}{2}\right)^m,\ \bar{G}\sum_{m\ge3}\binom{K+1}{m}\left(\frac{\eta}{2}\right)^m\right\}. ϵ ~ ( η ) := min { t = 1 ∑ H r ˉ t m ≥ 3 ∑ ( m n 0 , t ) ( 2 η ) m , G ˉ m ≥ 3 ∑ ( m K + 1 ) ( 2 η ) m } . Moreover , ϵ ( η ) ≤ G ˉ ( e x − 1 − x − x 2 / 2 ) \epsilon(\eta)\le\bar{G}(e^x-1-x-x^2/2) ϵ ( η ) ≤ G ˉ ( e x − 1 − x − x 2 /2 ) with x = ( K + 1 ) η / 2 x=(K+1)\eta/2 x = ( K + 1 ) η /2 .
Proof. For all t t t , ρ t − 1 = η u t \rho_t-1=\eta u_t ρ t − 1 = η u t , with u t = u θ ( s t , a t ) u_t=u_\theta(s_t,a_t) u t = u θ ( s t , a t ) if i t = 0 i_t=0 i t = 0 and ρ t − 1 = u t = 0 \rho_t-1=u_t=0 ρ t − 1 = u t = 0 if i t ≠ 0 i_t\ne0 i t = 0 . Theorem B.6(iii) with ψ = 0 \psi=0 ψ = 0 (Remark B.7) for ∣ S ∣ ∈ { 1 , 2 } \lvert S\rvert\in\{1,2\} ∣ S ∣ ∈ { 1 , 2 } , with S = { t ′ , t } S=\{t',t\} S = { t ′ , t } , t ′ < t t'<t t ′ < t , t S = t t_S=t t S = t when ∣ S ∣ = 2 \lvert S\rvert=2 ∣ S ∣ = 2 , gives:
J ( 1 ) = E π [ ∑ t = 1 H ( ρ t − 1 ) G t ] = η E π [ ∑ t = 1 H u t G t ] = η μ 1 ( θ ) , J ( 2 ) = E π [ ∑ t ′ < t ( ρ t ′ − 1 ) ( ρ t − 1 ) G t ] = η 2 E π [ ∑ t ′ < t u t ′ u t G t ] = η 2 μ 2 ( θ ) . (36) \begin{aligned}
J^{(1)}&=\mathbb{E}_{\pi}\left[\sum_{t=1}^{H}(\rho_t-1)G_t\right]=\eta\mathbb{E}_{\pi}\left[\sum_{t=1}^{H}u_tG_t\right]=\eta\mu_1(\theta),\\
J^{(2)}&=\mathbb{E}_{\pi}\left[\sum_{t'<t}(\rho_{t'}-1)(\rho_t-1)G_t\right]=\eta^2\mathbb{E}_{\pi}\left[\sum_{t'<t}u_{t'}u_tG_t\right]=\eta^2\mu_2(\theta). \tag*{(36)}
\end{aligned} J ( 1 ) J ( 2 ) = E π [ t = 1 ∑ H ( ρ t − 1 ) G t ] = η E π [ t = 1 ∑ H u t G t ] = η μ 1 ( θ ) , = E π [ t ′ < t ∑ ( ρ t ′ − 1 ) ( ρ t − 1 ) G t ] = η 2 E π [ t ′ < t ∑ u t ′ u t G t ] = η 2 μ 2 ( θ ) . (36)
By Proposition 3.5, α 0 ≤ η / 4 \alpha_0\le\eta/4 α 0 ≤ η /4 and α j = 0 \alpha_j=0 α j = 0 for j ∈ [ M ] j\in[M] j ∈ [ M ] , so B ( k ; ⋅ ) = 0 B(k;\cdot)=0 B ( k ; ⋅ ) = 0 unless k = m e 0 k=m e_0 k = m e 0 , with orchestrator visit counts n 0 , t n_{0,t} n 0 , t and n 0 ≤ K + 1 n_0\le K+1 n 0 ≤ K + 1 . Hence:
∣ J ( π θ , η ′ ) − J ( π ) − J ( 1 ) − J ( 2 ) ∣ ≤ ( a ) min { ∑ t r ˉ t ∑ m ≥ 3 ( n 0 , t m ) ( η 2 ) m , G ˉ ∑ m ≥ 3 ( n 0 m ) ( η 2 ) m } ≤ ( b ) ϵ ~ ( η ) , \left\lvert J(\pi'_{\theta,\eta})-J(\pi)-J^{(1)}-J^{(2)}\right\rvert\overset{(a)}{\le}\min\left\{\sum_t\bar{r}_t\sum_{m\ge3}\binom{n_{0,t}}{m}\left(\frac{\eta}{2}\right)^m,\ \bar{G}\sum_{m\ge3}\binom{n_0}{m}\left(\frac{\eta}{2}\right)^m\right\}
\overset{(b)}{\le}\tilde{\epsilon}(\eta), J ( π θ , η ′ ) − J ( π ) − J ( 1 ) − J ( 2 ) ≤ ( a ) min { t ∑ r ˉ t m ≥ 3 ∑ ( m n 0 , t ) ( 2 η ) m , G ˉ m ≥ 3 ∑ ( m n 0 ) ( 2 η ) m } ≤ ( b ) ϵ ~ ( η ) ,
where (a) is Theorem D.2 with r = 2 r=2 r = 2 and 2 α 0 ≤ η / 2 2\alpha_0\le\eta/2 2 α 0 ≤ η /2 , and (b) uses n 0 ≤ K + 1 n_0\le K+1 n 0 ≤ K + 1 and Lemma D.1(ii). The second entry of ϵ ~ ( η ) \tilde{\epsilon}(\eta) ϵ ~ ( η ) equals ϵ ( η ) \epsilon(\eta) ϵ ( η ) , as ( K + 1 m ) = 0 \binom{K+1}{m}=0 ( m K + 1 ) = 0 for m > K + 1 m>K+1 m > K + 1 . By ( K + 1 m ) ≤ ( K + 1 ) m / m ! \binom{K+1}{m}\le(K+1)^m/m! ( m K + 1 ) ≤ ( K + 1 ) m / m ! :
ϵ ( η ) ≤ G ˉ ∑ m ≥ 3 x m m ! = G ˉ ( e x − 1 − x − x 2 2 ) . \epsilon(\eta)\le\bar{G}\sum_{m\ge3}\frac{x^m}{m!}=\bar{G}\left(e^x-1-x-\frac{x^2}{2}\right). ϵ ( η ) ≤ G ˉ m ≥ 3 ∑ m ! x m = G ˉ ( e x − 1 − x − 2 x 2 ) .
Since the proofs below use only the inequality of Lemma E.1, ϵ ~ ( η ) \tilde{\epsilon}(\eta) ϵ ~ ( η ) can replace ϵ ( η ) \epsilon(\eta) ϵ ( η ) in (13) with every guarantee intact.
Algorithm 2 One pass over a trajectory
Require: trajectory $(s_1, a_1, \ldots, s_H, a_H)$, parameters $\theta$, step size $\eta$, baseline $b$
1: $G_{H+1} \gets 0$; $G_t \gets r_t(s_t, a_t) + G_{t+1}$ for $t = H, \ldots, 1$
2: $U, \nabla U, Z_2, \nabla Z_2, Z_{\mathrm{IS}} \gets 0$; $w \gets 1$
3: for $t = 1, \ldots, H$ do
4: if $i_t = 0$ then
5: $\rho \gets 1 + \eta u_\theta(s_t, a_t)$; $\nabla \rho \gets \eta \nabla_\theta u_\theta(s_t, a_t)$
6: else
7: $\rho \gets 1$; $\nabla \rho \gets 0$
8: end if
9: $Z_2 \gets Z_2 + (\rho - 1)(1 + U)(G_t - b)$
10: $\nabla Z_2 \gets \nabla Z_2 + ((1 + U)\nabla \rho + (\rho - 1)\nabla U)(G_t - b)$
11: $U \gets U + (\rho - 1)$; $\nabla U \gets \nabla U + \nabla \rho$
12: $w \gets w\rho$; $Z_{\mathrm{IS}} \gets Z_{\mathrm{IS}} + (w - 1)r_t(s_t, a_t)$
13: end for
14: $Z_{\mathrm{IS}} \gets Z_{\mathrm{IS}} - b(w - 1)$
15: return $Z_2, \nabla Z_2, Z_{\mathrm{IS}}$
One-Pass Computation
Lemma E.2 verifies Algorithm 2.
Lemma E.2 (One-pass statistic and gradient). Let U 1 = 0 U_1 = 0 U 1 = 0 and U t + 1 = U t + ( ρ t − 1 ) U_{t+1} = U_t + (\rho_t - 1) U t + 1 = U t + ( ρ t − 1 ) . Then:
(a) Z 2 = ∑ t ( ρ t − 1 ) ( G t − b ) + ∑ t ′ < t ( ρ t ′ − 1 ) ( ρ t − 1 ) ( G t − b ) Z_2 = \sum_t(\rho_t - 1)(G_t - b) + \sum_{t'<t}(\rho_{t'} - 1)(\rho_t - 1)(G_t - b) Z 2 = ∑ t ( ρ t − 1 ) ( G t − b ) + ∑ t ′ < t ( ρ t ′ − 1 ) ( ρ t − 1 ) ( G t − b ) , and E π [ Z 2 ] = η μ 1 ( θ ) + η 2 μ 2 ( θ ) \mathbb{E}_\pi[Z_2] = \eta\mu_1(\theta) + \eta^2\mu_2(\theta) E π [ Z 2 ] = η μ 1 ( θ ) + η 2 μ 2 ( θ ) for every fixed b b b ;
(b) ∇ θ Z 2 = ∑ t [ ( 1 + U t ) ∇ θ ρ t + ( ρ t − 1 ) ∇ θ U t ] ( G t − b ) \nabla_\theta Z_2 = \sum_t[(1 + U_t)\nabla_\theta\rho_t + (\rho_t - 1)\nabla_\theta U_t](G_t - b) ∇ θ Z 2 = ∑ t [( 1 + U t ) ∇ θ ρ t + ( ρ t − 1 ) ∇ θ U t ] ( G t − b ) with ∇ θ U t + 1 = ∇ θ U t + ∇ θ ρ t \nabla_\theta U_{t+1} = \nabla_\theta U_t + \nabla_\theta\rho_t ∇ θ U t + 1 = ∇ θ U t + ∇ θ ρ t , and ∇ θ ρ t = η 2 [ ∇ θ f θ ( s t , a t ) − ∑ a ′ π ( a ′ ∣ s t ) ∇ θ f θ ( s t , a ′ ) ] \nabla_\theta\rho_t = \frac{\eta}{2}[\nabla_\theta f_\theta(s_t, a_t) - \sum_{a'}\pi(a' \mid s_t)\nabla_\theta f_\theta(s_t, a')] ∇ θ ρ t = 2 η [ ∇ θ f θ ( s t , a t ) − ∑ a ′ π ( a ′ ∣ s t ) ∇ θ f θ ( s t , a ′ )] at orchestrator steps and ∇ θ ρ t = 0 \nabla_\theta\rho_t = 0 ∇ θ ρ t = 0 at worker steps;
(c) Algorithm 2 returns Z 2 Z_2 Z 2 , ∇ θ Z 2 \nabla_\theta Z_2 ∇ θ Z 2 , and Z I S Z_{\mathrm{IS}} Z IS , in time linear in H H H .
Proof. By induction, U t = ∑ t ′ < t ( ρ t ′ − 1 ) U_t = \sum_{t'<t}(\rho_{t'} - 1) U t = ∑ t ′ < t ( ρ t ′ − 1 ) , since U 1 = 0 U_1 = 0 U 1 = 0 and U t + 1 = U t + ( ρ t − 1 ) = ∑ t ′ ≤ t ( ρ t ′ − 1 ) U_{t+1} = U_t + (\rho_t - 1) = \sum_{t'\leq t}(\rho_{t'} - 1) U t + 1 = U t + ( ρ t − 1 ) = ∑ t ′ ≤ t ( ρ t ′ − 1 ) .
(a) By this identity and (9) :
Z 2 = ∑ t ( ρ t − 1 ) ( 1 + U t ) ( G t − b ) = ∑ t ( ρ t − 1 ) ( G t − b ) + ∑ t ∑ t ′ < t ( ρ t ′ − 1 ) ( ρ t − 1 ) ( G t − b ) . Z_2 = \sum_t(\rho_t - 1)(1 + U_t)(G_t - b) = \sum_t(\rho_t - 1)(G_t - b) + \sum_t\sum_{t'<t}(\rho_{t'} - 1)(\rho_t - 1)(G_t - b). Z 2 = t ∑ ( ρ t − 1 ) ( 1 + U t ) ( G t − b ) = t ∑ ( ρ t − 1 ) ( G t − b ) + t ∑ t ′ < t ∑ ( ρ t ′ − 1 ) ( ρ t − 1 ) ( G t − b ) .
Since 1 + U t 1 + U_t 1 + U t is a function of τ t − 1 \tau_{t-1} τ t − 1 , Lemma B.4 gives:
E π [ ( ρ t − 1 ) ( 1 + U t ) b ] = b E π [ ( 1 + U t ) E π [ ρ t − 1 ∣ τ t − 1 , s t ] ] = 0 , \mathbb{E}_\pi[(\rho_t - 1)(1 + U_t)b] = b\mathbb{E}_\pi[(1 + U_t)\mathbb{E}_\pi[\rho_t - 1 \mid\tau_{t-1}, s_t]] = 0, E π [( ρ t − 1 ) ( 1 + U t ) b ] = b E π [( 1 + U t ) E π [ ρ t − 1 ∣ τ t − 1 , s t ]] = 0 ,
so E π [ Z 2 ] = J ( 1 ) + J ( 2 ) = η μ 1 + η 2 μ 2 \mathbb{E}_\pi[Z_2] = J^{(1)} + J^{(2)} = \eta\mu_1 + \eta^2\mu_2 E π [ Z 2 ] = J ( 1 ) + J ( 2 ) = η μ 1 + η 2 μ 2 by (36) . For { t : i t = 0 } = { 1 , 3 , 5 } \{t : i_t = 0\} = \{1, 3, 5\} { t : i t = 0 } = { 1 , 3 , 5 } and ρ t − 1 = η u t \rho_t - 1 = \eta u_t ρ t − 1 = η u t :
Z 2 = η ( u 1 G 1 + u 3 G 3 + u 5 G 5 ) + η 2 ( u 1 u 3 G 3 + u 1 u 5 G 5 + u 3 u 5 G 5 ) Z_2 = \eta(u_1G_1 + u_3G_3 + u_5G_5) + \eta^2(u_1u_3G_3 + u_1u_5G_5 + u_3u_5G_5) Z 2 = η ( u 1 G 1 + u 3 G 3 + u 5 G 5 ) + η 2 ( u 1 u 3 G 3 + u 1 u 5 G 5 + u 3 u 5 G 5 )
for b = 0 b = 0 b = 0 , which excludes the third-order term η 3 u 1 u 3 u 5 G 5 \eta^3u_1u_3u_5G_5 η 3 u 1 u 3 u 5 G 5 .
(b) Only the ratios depend on θ \theta θ , so by the product rule:
∇ θ Z 2 = ∑ t [ ( 1 + U t ) ∇ θ ρ t + ( ρ t − 1 ) ∇ θ U t ] ( G t − b ) , ∇ θ U t = ∑ t ′ < t ∇ θ ρ t ′ , (37) \nabla_\theta Z_2 = \sum_t[(1 + U_t)\nabla_\theta\rho_t + (\rho_t - 1)\nabla_\theta U_t](G_t - b), \qquad\nabla_\theta U_t = \sum_{t'<t}\nabla_\theta\rho_{t'}, \tag*{(37)} ∇ θ Z 2 = t ∑ [( 1 + U t ) ∇ θ ρ t + ( ρ t − 1 ) ∇ θ U t ] ( G t − b ) , ∇ θ U t = t ′ < t ∑ ∇ θ ρ t ′ , (37)
whence ∇ θ U t + 1 = ∇ θ U t + ∇ θ ρ t \nabla_\theta U_{t+1} = \nabla_\theta U_t + \nabla_\theta\rho_t ∇ θ U t + 1 = ∇ θ U t + ∇ θ ρ t . If i t = 0 i_t = 0 i t = 0 , then ρ t = 1 + η u θ ( s t , a t ) \rho_t = 1 + \eta u_\theta(s_t, a_t) ρ t = 1 + η u θ ( s t , a t ) and the weights π ( a ′ ∣ s t ) \pi(a' \mid s_t) π ( a ′ ∣ s t ) of the mean ∑ a ′ π ( a ′ ∣ s t ) f θ ( s t , a ′ ) \sum_{a'}\pi(a' \mid s_t)f_\theta(s_t, a') ∑ a ′ π ( a ′ ∣ s t ) f θ ( s t , a ′ ) do not depend on θ \theta θ , so:
∇ θ ρ t = η ∇ θ u θ ( s t , a t ) = η 2 [ ∇ θ f θ ( s t , a t ) − ∑ a ′ π ( a ′ ∣ s t ) ∇ θ f θ ( s t , a ′ ) ] . (38) \nabla_\theta\rho_t = \eta\nabla_\theta u_\theta(s_t, a_t) = \frac{\eta}{2}\left[\nabla_\theta f_\theta(s_t, a_t) - \sum_{a'}\pi(a' \mid s_t)\nabla_\theta f_\theta(s_t, a')\right]. \tag*{(38)} ∇ θ ρ t = η ∇ θ u θ ( s t , a t ) = 2 η [ ∇ θ f θ ( s t , a t ) − a ′ ∑ π ( a ′ ∣ s t ) ∇ θ f θ ( s t , a ′ ) ] . (38)
(c) Line 1 of Algorithm 2 computes G t G_t G t backward from G H + 1 = 0 G_{H+1} = 0 G H + 1 = 0 . At the start of iteration t t t of the loop:
U = U t , ∇ U = ∇ θ U t , w = ∏ t ′ < t ρ t ′ , Z 2 = ∑ t ′ < t ( ρ t ′ − 1 ) ( 1 + U t ′ ) ( G t ′ − b ) , Z I S = ∑ t ′ < t ( ∏ t ′ ′ ≤ t ′ ρ t ′ ′ − 1 ) r t ′ ( s t ′ , a t ′ ) . U = U_t,\qquad\nabla U = \nabla_\theta U_t,\qquad w = \prod_{t'<t}\rho_{t'},\qquad Z_2 = \sum_{t'<t}(\rho_{t'} - 1)(1 + U_{t'})(G_{t'} - b),\qquad Z_{\mathrm{IS}} = \sum_{t'<t}\left(\prod_{t''\leq t'}\rho_{t''} - 1\right)r_{t'}(s_{t'}, a_{t'}). U = U t , ∇ U = ∇ θ U t , w = t ′ < t ∏ ρ t ′ , Z 2 = t ′ < t ∑ ( ρ t ′ − 1 ) ( 1 + U t ′ ) ( G t ′ − b ) , Z IS = t ′ < t ∑ ( t ′′ ≤ t ′ ∏ ρ t ′′ − 1 ) r t ′ ( s t ′ , a t ′ ) .
This holds at t = 1 t = 1 t = 1 by the initialization. If it holds at t t t , iteration t t t sets ( ρ , ∇ ρ ) = ( ρ t , ∇ θ ρ t ) (\rho,\nabla\rho) = (\rho_t,\nabla_\theta\rho_t) ( ρ , ∇ ρ ) = ( ρ t , ∇ θ ρ t ) by (38) , adds the t t t -th terms of (9) and (37) with ( U , ∇ U ) = ( U t , ∇ θ U t ) (U,\nabla U) = (U_t,\nabla_\theta U_t) ( U , ∇ U ) = ( U t , ∇ θ U t ) , sets ( U , ∇ U ) = ( U t + 1 , ∇ θ U t + 1 ) (U,\nabla U) = (U_{t+1},\nabla_\theta U_{t+1}) ( U , ∇ U ) = ( U t + 1 , ∇ θ U t + 1 ) , and sets w = ∏ t ′ ≤ t ρ t ′ w = \prod_{t'\le t}\rho_{t'} w = ∏ t ′ ≤ t ρ t ′ before adding the t t t -th reward term of (11) ; so it holds at t + 1 t+1 t + 1 . At t = H + 1 t = H+1 t = H + 1 , w = ∏ t ′ ≤ H ρ t ′ w = \prod_{t'\le H}\rho_{t'} w = ∏ t ′ ≤ H ρ t ′ , and the last line subtracts the baseline term of (11) . Each iteration costs O ( 1 ) O(1) O ( 1 ) , so the time is O ( H ) O(H) O ( H ) .
Improving Steps Exist
Theorem 3.6 (Existence of improving steps). Fix θ \theta θ and write μ 1 = μ 1 ( θ ) \mu_1 = \mu_1(\theta) μ 1 = μ 1 ( θ ) and μ 2 = μ 2 ( θ ) \mu_2 = \mu_2(\theta) μ 2 = μ 2 ( θ ) . (i) If μ 1 > 0 \mu_1 > 0 μ 1 > 0 , then any η ∈ ( 0 , 1 ] \eta\in(0,1] η ∈ ( 0 , 1 ] with η ( ∣ μ 2 ∣ + ϵ ( 1 ) ) ≤ μ 1 / 2 \eta(|\mu_2|+\epsilon(1)) \le\mu_1/2 η ( ∣ μ 2 ∣ + ϵ ( 1 )) ≤ μ 1 /2 guarantees J ( π θ , η ) − J ( π ) ≥ η μ 1 / 2 J(\pi_{\theta,\eta})-J(\pi) \ge\eta\mu_1/2 J ( π θ , η ) − J ( π ) ≥ η μ 1 /2 . (ii) If μ 1 = 0 < μ 2 \mu_1 = 0 < \mu_2 μ 1 = 0 < μ 2 , then any η ∈ ( 0 , 1 ] \eta\in(0,1] η ∈ ( 0 , 1 ] with η ϵ ( 1 ) ≤ μ 2 / 2 \eta\epsilon(1) \le\mu_2/2 η ϵ ( 1 ) ≤ μ 2 /2 guarantees J ( π θ , η ) − J ( π ) ≥ η 2 μ 2 / 2 J(\pi_{\theta,\eta})-J(\pi) \ge\eta^2\mu_2/2 J ( π θ , η ) − J ( π ) ≥ η 2 μ 2 /2 .
Furthermore, if the score class can represent f θ ( s t , a ) = Q t π ( s t , a ) / G ˉ f_\theta(s_t,a) = Q_t^\pi(s_t,a)/\bar{G} f θ ( s t , a ) = Q t π ( s t , a ) / G ˉ , then:
μ 1 = 1 2 G ˉ E π [ ∑ t : i t = 0 A t π ( s t , a t ) 2 ] . \mu_1 = \frac{1}{2\bar{G}}\mathbb{E}_\pi\left[\sum_{t:i_t=0} A_t^\pi(s_t,a_t)^2\right]. μ 1 = 2 G ˉ 1 E π [ t : i t = 0 ∑ A t π ( s t , a t ) 2 ] . which is positive whenever the advantage is nonzero with positive probability at an orchestrator decision.
Proof. By Lemma E.1 :
J ( π θ , η ) − J ( π ) ≥ η μ 1 + η 2 μ 2 − ϵ ( η ) . J(\pi_{\theta,\eta})-J(\pi) \ge\eta\mu_1+\eta^2\mu_2-\epsilon(\eta). J ( π θ , η ) − J ( π ) ≥ η μ 1 + η 2 μ 2 − ϵ ( η ) .
For η ∈ [ 0 , 1 ] \eta\in[0,1] η ∈ [ 0 , 1 ] and m ≥ 3 m \ge3 m ≥ 3 , ( η / 2 ) m = η m 2 − m ≤ η 3 2 − m (\eta/2)^m=\eta^m2^{-m}\le\eta^32^{-m} ( η /2 ) m = η m 2 − m ≤ η 3 2 − m , so by (10) :
ϵ ( η ) ≤ η 3 G ˉ ∑ m = 3 K + 1 ( K + 1 m ) 2 − m = η 3 ϵ ( 1 ) . (39) \epsilon(\eta)\le\eta^3\bar{G}\sum_{m=3}^{K+1}\binom{K+1}{m}2^{-m}=\eta^3\epsilon(1). \tag*{(39)} ϵ ( η ) ≤ η 3 G ˉ m = 3 ∑ K + 1 ( m K + 1 ) 2 − m = η 3 ϵ ( 1 ) . (39)
If μ 1 > 0 \mu_1>0 μ 1 > 0 and η ( ∣ μ 2 ∣ + ϵ ( 1 ) ) ≤ μ 1 / 2 \eta(|\mu_2|+\epsilon(1))\le\mu_1/2 η ( ∣ μ 2 ∣ + ϵ ( 1 )) ≤ μ 1 /2 :
η μ 1 + η 2 μ 2 − ϵ ( η ) ≥ ( a ) η μ 1 − η 2 ∣ μ 2 ∣ − η 2 ϵ ( 1 ) = η [ μ 1 − η ( ∣ μ 2 ∣ + ϵ ( 1 ) ) ] ≥ ( b ) η μ 1 2 , \eta\mu_1+\eta^2\mu_2-\epsilon(\eta)\mathrel{\overset{(a)}{\ge}}\eta\mu_1-\eta^2|\mu_2|-\eta^2\epsilon(1)=\eta\left[\mu_1-\eta\left(|\mu_2|+\epsilon(1)\right)\right]\mathrel{\overset{(b)}{\ge}}\frac{\eta\mu_1}{2}, η μ 1 + η 2 μ 2 − ϵ ( η ) ≥ ( a ) η μ 1 − η 2 ∣ μ 2 ∣ − η 2 ϵ ( 1 ) = η [ μ 1 − η ( ∣ μ 2 ∣ + ϵ ( 1 ) ) ] ≥ ( b ) 2 η μ 1 ,
where (a) uses η 2 μ 2 ≥ − η 2 ∣ μ 2 ∣ \eta^2\mu_2\ge-\eta^2|\mu_2| η 2 μ 2 ≥ − η 2 ∣ μ 2 ∣ , (39) , and η 3 ≤ η 2 \eta^3\le\eta^2 η 3 ≤ η 2 , and (b) uses η ( ∣ μ 2 ∣ + ϵ ( 1 ) ) ≤ μ 1 / 2 \eta(|\mu_2|+\epsilon(1))\le\mu_1/2 η ( ∣ μ 2 ∣ + ϵ ( 1 )) ≤ μ 1 /2 .
If μ 1 = 0 < μ 2 \mu_1=0<\mu_2 μ 1 = 0 < μ 2 and η ϵ ( 1 ) ≤ μ 2 / 2 \eta\epsilon(1)\le\mu_2/2 η ϵ ( 1 ) ≤ μ 2 /2 , then (39) gives:
η 2 μ 2 − ϵ ( η ) ≥ η 2 μ 2 − η 3 ϵ ( 1 ) = η 2 ( μ 2 − η ϵ ( 1 ) ) ≥ η 2 μ 2 2 . \eta^2\mu_2-\epsilon(\eta)\ge\eta^2\mu_2-\eta^3\epsilon(1)=\eta^2\left(\mu_2-\eta\epsilon(1)\right)\ge\frac{\eta^2\mu_2}{2}. η 2 μ 2 − ϵ ( η ) ≥ η 2 μ 2 − η 3 ϵ ( 1 ) = η 2 ( μ 2 − η ϵ ( 1 ) ) ≥ 2 η 2 μ 2 .
By Lemma B.2 (c), ∣ Q t π ∣ ≤ G ˉ |Q_t^\pi|\le\bar{G} ∣ Q t π ∣ ≤ G ˉ , so f θ ( s t , a ) = Q t π ( s t , a ) / G ˉ ∈ [ − 1 , 1 ] f_\theta(s_t,a)=Q_t^\pi(s_t,a)/\bar{G}\in[-1,1] f θ ( s t , a ) = Q t π ( s t , a ) / G ˉ ∈ [ − 1 , 1 ] , a function of ( s , a ) (s,a) ( s , a ) as s s s determines t t t . By Lemma B.2 (b), ∑ a ′ π ( a ′ ∣ s t ) f θ ( s t , a ′ ) = V t π ( s t ) / G ˉ \sum_{a'}\pi(a'|s_t)f_\theta(s_t,a')=V_t^\pi(s_t)/\bar{G} ∑ a ′ π ( a ′ ∣ s t ) f θ ( s t , a ′ ) = V t π ( s t ) / G ˉ , so if i t = 0 i_t=0 i t = 0 :
u t = 1 2 ( Q t π ( s t , a t ) G ˉ − V t π ( s t ) G ˉ ) = A t π ( s t , a t ) 2 G ˉ . u_t=\frac{1}{2}\left(\frac{Q_t^\pi(s_t,a_t)}{\bar{G}}-\frac{V_t^\pi(s_t)}{\bar{G}}\right)=\frac{A_t^\pi(s_t,a_t)}{2\bar{G}}. u t = 2 1 ( G ˉ Q t π ( s t , a t ) − G ˉ V t π ( s t ) ) = 2 G ˉ A t π ( s t , a t ) .
Since u t u_t u t is a function of τ t \tau_t τ t :
E π [ u t G t ] = ( a ) E π [ u t Q t π ( s t , a t ) ] = ( b ) E π [ u t A t π ( s t , a t ) ] , (40) \mathbb{E}_\pi[u_tG_t]\mathrel{\overset{(a)}{=}}\mathbb{E}_\pi[u_tQ_t^\pi(s_t,a_t)]\mathrel{\overset{(b)}{=}}\mathbb{E}_\pi[u_tA_t^\pi(s_t,a_t)], \tag*{(40)} E π [ u t G t ] = ( a ) E π [ u t Q t π ( s t , a t )] = ( b ) E π [ u t A t π ( s t , a t )] , (40)
where (a) uses the tower rule and Lemma B.2 (a), and (b) uses Q t π = A t π + V t π Q_t^\pi=A_t^\pi+V_t^\pi Q t π = A t π + V t π and, by Lemma B.2 (b):
E π [ A t π ( s t , a t ) V t π ( s t ) ∣ s t ] = V t π ( s t ) ∑ a π t ( a ∣ s t ) A t π ( s t , a ) = 0. \mathbb{E}_\pi\left[A_t^\pi(s_t,a_t)V_t^\pi(s_t)\mid s_t\right]=V_t^\pi(s_t)\sum_a\pi_t(a\mid s_t)A_t^\pi(s_t,a)=0. E π [ A t π ( s t , a t ) V t π ( s t ) ∣ s t ] = V t π ( s t ) a ∑ π t ( a ∣ s t ) A t π ( s t , a ) = 0.
Summing (40) over t t t , with u t = A t π ( s t , a t ) / ( 2 G ˉ ) u_t=A_t^\pi(s_t,a_t)/(2\bar{G}) u t = A t π ( s t , a t ) / ( 2 G ˉ ) if i t = 0 i_t=0 i t = 0 and u t = 0 u_t=0 u t = 0 otherwise:
μ 1 = ∑ t E π [ u t A t π ( s t , a t ) ] = 1 2 G ˉ E π [ ∑ t : i t = 0 A t π ( s t , a t ) 2 ] . \mu_1=\sum_t\mathbb{E}_\pi[u_tA_t^\pi(s_t,a_t)]=\frac{1}{2\bar{G}}\mathbb{E}_\pi\left[\sum_{t:i_t=0}A_t^\pi(s_t,a_t)^2\right]. μ 1 = t ∑ E π [ u t A t π ( s t , a t )] = 2 G ˉ 1 E π [ t : i t = 0 ∑ A t π ( s t , a t ) 2 ] .
which is positive unless A t π ( s t , a t ) = 0 A_t^\pi(s_t,a_t)=0 A t π ( s t , a t ) = 0 almost surely at every orchestrator step.
Means and Ranges of the Statistics
Lemma E.3 (Means and ranges). For the candidate π η ′ \pi'_{\eta} π η ′ , let Δ η : = J ( π η ′ ) − J ( π ) \Delta_{\eta} := J(\pi'_{\eta}) - J(\pi) Δ η := J ( π η ′ ) − J ( π ) , and let b b b be fixed. Then:
(a) E π [ Z I S ] = Δ η \mathbb{E}_{\pi}[Z_{\mathrm{IS}}] = \Delta_{\eta} E π [ Z IS ] = Δ η and ∣ E π [ Z 2 ] − Δ η ∣ ≤ ϵ ( η ) |\mathbb{E}_{\pi}[Z_2] - \Delta_{\eta}| \le\epsilon(\eta) ∣ E π [ Z 2 ] − Δ η ∣ ≤ ϵ ( η ) ;
(b) ∣ Z 2 ∣ ≤ B 2 , η |Z_2| \le B_{2,\eta} ∣ Z 2 ∣ ≤ B 2 , η and ∣ Z I S ∣ ≤ B I S , η |Z_{\mathrm{IS}}| \le B_{\mathrm{IS},\eta} ∣ Z IS ∣ ≤ B IS , η , where:
B 2 , η : = ( G ˉ + ∣ b ∣ ) ( ( K + 1 ) η + ( K + 1 2 ) η 2 ) , B I S , η : = ( G ˉ + ∣ b ∣ ) ( ( 1 + η ) K + 1 − 1 ) . B_{2,\eta} := (\bar{G} + |b|)\left((K+1)\eta+ \binom{K+1}{2}\eta^2\right), \qquad B_{\mathrm{IS},\eta} := (\bar{G} + |b|)\left((1+\eta)^{K+1}-1\right). B 2 , η := ( G ˉ + ∣ b ∣ ) ( ( K + 1 ) η + ( 2 K + 1 ) η 2 ) , B IS , η := ( G ˉ + ∣ b ∣ ) ( ( 1 + η ) K + 1 − 1 ) .
Proof. For (a), Lemma B.3 with f = r t ( s t , a t ) f = r_t(s_t, a_t) f = r t ( s t , a t ) and with f = 1 f = 1 f = 1 gives:
E π [ ( ∏ t ′ ≤ t ρ t ′ − 1 ) r t ( s t , a t ) ] = E π η ′ [ r t ( s t , a t ) ] − E π [ r t ( s t , a t ) ] , E π [ ∏ t ≤ H ρ t − 1 ] = 1 − 1 = 0. \mathbb{E}_{\pi}\left[\left(\prod_{t'\le t}\rho_{t'}-1\right)r_t(s_t,a_t)\right] = \mathbb{E}_{\pi'_{\eta}}[r_t(s_t,a_t)]-\mathbb{E}_{\pi}[r_t(s_t,a_t)], \qquad\mathbb{E}_{\pi}\left[\prod_{t\le H}\rho_t-1\right]=1-1=0. E π [ ( t ′ ≤ t ∏ ρ t ′ − 1 ) r t ( s t , a t ) ] = E π η ′ [ r t ( s t , a t )] − E π [ r t ( s t , a t )] , E π [ t ≤ H ∏ ρ t − 1 ] = 1 − 1 = 0.
Summing over t t t , Lemma B.1 gives E π [ Z I S ] = J ( π η ′ ) − J ( π ) = Δ η \mathbb{E}_{\pi}[Z_{\mathrm{IS}}] = J(\pi'_{\eta}) - J(\pi) = \Delta_{\eta} E π [ Z IS ] = J ( π η ′ ) − J ( π ) = Δ η . For Z 2 Z_2 Z 2 :
∣ E π [ Z 2 ] − Δ η ∣ = ( a ) ∣ η μ 1 + η 2 μ 2 − Δ η ∣ ≤ ( b ) ϵ ( η ) , \left|\mathbb{E}_{\pi}[Z_2]-\Delta_{\eta}\right| \overset{(a)}{=} |\eta\mu_1+\eta^2\mu_2-\Delta_{\eta}| \overset{(b)}{\le} \epsilon(\eta), ∣ E π [ Z 2 ] − Δ η ∣ = ( a ) ∣ η μ 1 + η 2 μ 2 − Δ η ∣ ≤ ( b ) ϵ ( η ) ,
where (a) uses Lemma E.2(a) and (b) uses Lemma E.1.
For (b), on every trajectory ∣ G t − b ∣ ≤ G ˉ + ∣ b ∣ |G_t-b|\le\bar{G}+|b| ∣ G t − b ∣ ≤ G ˉ + ∣ b ∣ , ∣ { t : ρ t ≠ 1 } ∣ ≤ K + 1 |\{t:\rho_t\ne1\}|\le K+1 ∣ { t : ρ t = 1 } ∣ ≤ K + 1 (one per orchestrator step), and ∣ ρ t − 1 ∣ ≤ η |\rho_t-1|\le\eta ∣ ρ t − 1∣ ≤ η by Proposition 3.5. By Lemma E.2(a), Z 2 Z_2 Z 2 has at most K + 1 K+1 K + 1 nonzero single terms and ( K + 1 2 ) \binom{K+1}{2} ( 2 K + 1 ) nonzero pair terms, so:
∣ Z 2 ∣ ≤ ( G ˉ + ∣ b ∣ ) ( ( K + 1 ) η + ( K + 1 2 ) η 2 ) = B 2 , η . |Z_2|\le(\bar{G}+|b|)\left((K+1)\eta+\binom{K+1}{2}\eta^2\right)=B_{2,\eta}. ∣ Z 2 ∣ ≤ ( G ˉ + ∣ b ∣ ) ( ( K + 1 ) η + ( 2 K + 1 ) η 2 ) = B 2 , η .
Since at most K + 1 K+1 K + 1 ratios differ from 1 and ρ t ′ ∈ [ 1 − η , 1 + η ] \rho_{t'}\in[1-\eta,1+\eta] ρ t ′ ∈ [ 1 − η , 1 + η ] , ( 1 − η ) K + 1 ≤ ∏ t ′ ≤ t ρ t ′ ≤ ( 1 + η ) K + 1 (1-\eta)^{K+1}\le\prod_{t'\le t}\rho_{t'}\le(1+\eta)^{K+1} ( 1 − η ) K + 1 ≤ ∏ t ′ ≤ t ρ t ′ ≤ ( 1 + η ) K + 1 . By convexity of x ↦ x K + 1 x\mapsto x^{K+1} x ↦ x K + 1 , ( 1 + η ) K + 1 + ( 1 − η ) K + 1 ≥ 2 (1+\eta)^{K+1}+(1-\eta)^{K+1}\ge2 ( 1 + η ) K + 1 + ( 1 − η ) K + 1 ≥ 2 , so ∣ ∏ t ′ ≤ t ρ t ′ − 1 ∣ ≤ ( 1 + η ) K + 1 − 1 \left|\prod_{t'\le t}\rho_{t'}-1\right|\le(1+\eta)^{K+1}-1 ∏ t ′ ≤ t ρ t ′ − 1 ≤ ( 1 + η ) K + 1 − 1 and:
∣ Z I S ∣ ≤ ( ∑ t ∣ r t ∣ + ∣ b ∣ ) ( ( 1 + η ) K + 1 − 1 ) ≤ ( G ˉ + ∣ b ∣ ) ( ( 1 + η ) K + 1 − 1 ) = B I S , η . |Z_{\mathrm{IS}}|\le\left(\sum_t|r_t|+|b|\right)\left((1+\eta)^{K+1}-1\right)\le(\bar{G}+|b|)\left((1+\eta)^{K+1}-1\right)=B_{\mathrm{IS},\eta}. ∣ Z IS ∣ ≤ ( t ∑ ∣ r t ∣ + ∣ b ∣ ) ( ( 1 + η ) K + 1 − 1 ) ≤ ( G ˉ + ∣ b ∣ ) ( ( 1 + η ) K + 1 − 1 ) = B IS , η .
Certified Selection
Theorem 3.7 (Certified selection). With probability at least 1 − δ 1-\delta 1 − δ , the following statements hold simultaneously. (i) For every η ∈ Λ \eta\in\Lambda η ∈ Λ , L η ≤ J ( π η ′ ) − J ( π ) L_{\eta}\le J(\pi'_{\eta})-J(\pi) L η ≤ J ( π η ′ ) − J ( π ) ; consequently, J ( π ^ ) ≥ J ( π ) J(\hat{\pi})\ge J(\pi) J ( π ^ ) ≥ J ( π ) . (ii) The selected policy satisfies:
J ( π ^ ) − J ( π ) ≥ max { 0 , max η ∈ Λ [ J ( π η ′ ) − J ( π ) − 2 c η ] } . J(\hat{\pi})-J(\pi)\ge\max\left\{0,\max_{\eta\in\Lambda}\left[J(\pi'_{\eta})-J(\pi)-2c_{\eta}\right]\right\}. J ( π ^ ) − J ( π ) ≥ max { 0 , η ∈ Λ max [ J ( π η ′ ) − J ( π ) − 2 c η ] } . (iii) If some η ∈ Λ \eta\in\Lambda η ∈ Λ satisfies J ( π η ′ ) > J ( π ) J(\pi'_{\eta})>J(\pi) J ( π η ′ ) > J ( π ) and n − 1 > 256 B I S , η 2 log ( 8 ∣ Λ ∣ / δ ) / [ J ( π η ′ ) − J ( π ) ] 2 n-1>256B_{\mathrm{IS},\eta}^{2}\log(8|\Lambda|/\delta)/[J(\pi'_{\eta})-J(\pi)]^2 n − 1 > 256 B IS , η 2 log ( 8∣Λ∣/ δ ) / [ J ( π η ′ ) − J ( π ) ] 2 , then π ^ ≠ π \hat{\pi}\ne\pi π ^ = π .
Lemma E.4 (Empirical Bernstein bound (Maurer and Pontil [24 ] ). Let Y 1 , … , Y n Y_1,\ldots,Y_n Y 1 , … , Y n , n ≥ 2 n\ge2 n ≥ 2 , be i.i.d. copies of Y ∈ [ − B , B ] Y\in[-B,B] Y ∈ [ − B , B ] with sample mean Y ˉ \bar{Y} Y ˉ and sample variance σ ^ 2 = 1 n − 1 ∑ ℓ = 1 n ( Y ℓ − Y ˉ ) 2 \hat{\sigma}^2=\frac{1}{n-1}\sum_{\ell=1}^{n}(Y_{\ell}-\bar{Y})^2 σ ^ 2 = n − 1 1 ∑ ℓ = 1 n ( Y ℓ − Y ˉ ) 2 . For every δ ′ ∈ ( 0 , 1 ) \delta'\in(0,1) δ ′ ∈ ( 0 , 1 ) , with probability ≥ 1 − δ ′ \ge1-\delta' ≥ 1 − δ ′ :
E [ Y ] − Y ˉ ≤ σ ^ 2 log ( 2 / δ ′ ) n + 14 B log ( 2 / δ ′ ) 3 ( n − 1 ) . \mathbb{E}[Y]-\bar{Y}\le\hat{\sigma}\sqrt{\frac{2\log(2/\delta')}{n}}+\frac{14B\log(2/\delta')}{3(n-1)}. E [ Y ] − Y ˉ ≤ σ ^ n 2 log ( 2/ δ ′ ) + 3 ( n − 1 ) 14 B log ( 2/ δ ′ ) .
This is Theorem 4 of Maurer and Pontil [24 ] applied to ( Y ℓ + B ) / ( 2 B ) ∈ [ 0 , 1 ] (Y_{\ell}+B)/(2B)\in[0,1] ( Y ℓ + B ) / ( 2 B ) ∈ [ 0 , 1 ] and rescaled by 2 B 2B 2 B .
Proof of Theorem 3.7. Let ι : = log ( 8 ∣ Λ ∣ / δ ) \iota:=\log(8|\Lambda|/\delta) ι := log ( 8∣Λ∣/ δ ) , so (12) reads ξ η ( Z ) = σ ^ Z 2 ι / n + 14 B ι / ( 3 ( n − 1 ) ) \xi_{\eta}(Z)=\hat{\sigma}_{Z}\sqrt{2\iota/n}+14B\iota/(3(n-1)) ξ η ( Z ) = σ ^ Z 2 ι / n + 14 B ι / ( 3 ( n − 1 )) with B = B Z , η B=B_{Z,\eta} B = B Z , η . Set π 0 ′ : = π \pi'_0:=\pi π 0 ′ := π and Δ 0 = L 0 : = 0 \Delta_0=L_0:=0 Δ 0 = L 0 := 0 , so π ^ = π η ^ ′ \hat{\pi}=\pi'_{\hat{\eta}} π ^ = π η ^ ′ with η ^ = arg max η ∈ Λ ∪ { 0 } L η \hat{\eta}=\arg\max_{\eta\in\Lambda\cup\{0\}}L_{\eta} η ^ = arg max η ∈ Λ ∪ { 0 } L η .
The candidates are fixed before the n n n validation trajectories are drawn i.i.d. from π \pi π , so each statistic is i.i.d. with bound B B B from Lemma E.3(b). Lemma E.4, applied to Z Z Z and − Z -Z − Z with δ ′ = 2 e − ι \delta'=2e^{-\iota} δ ′ = 2 e − ι (so log ( 2 / δ ′ ) = ι \log(2/\delta')=\iota log ( 2/ δ ′ ) = ι ), gives:
Pr ( ∣ Z ˉ − E [ Z ] ∣ > ξ η ( Z ) ) ≤ 2 δ ′ = 4 e − ι . \Pr\left(|\bar{Z}-\mathbb{E}[Z]|>\xi_{\eta}(Z)\right)\le2\delta'=4e^{-\iota}. Pr ( ∣ Z ˉ − E [ Z ] ∣ > ξ η ( Z ) ) ≤ 2 δ ′ = 4 e − ι .
By the union bound over the 2 ∣ Λ ∣ 2|\Lambda| 2∣Λ∣ statistics, all intervals hold except with probability:
2 ∣ Λ ∣ ⋅ 4 e − ι = 8 ∣ Λ ∣ ⋅ δ 8 ∣ Λ ∣ = δ . 2|\Lambda|\cdot4e^{-\iota}=8|\Lambda|\cdot\frac{\delta}{8|\Lambda|}=\delta. 2∣Λ∣ ⋅ 4 e − ι = 8∣Λ∣ ⋅ 8∣Λ∣ δ = δ .
On this event:
Z ˉ 2 − ξ η ( Z 2 ) − ϵ ( η ) ≤ ( a ) E [ Z 2 ] − ϵ ( η ) ≤ ( b ) Δ η ≤ ( b ) E [ Z 2 ] + ϵ ( η ) ≤ ( a ) Z ˉ 2 + ξ η ( Z 2 ) + ϵ ( η ) , \bar{Z}_2-\xi_\eta(Z_2)-\epsilon(\eta)\mathrel{\overset{(a)}{\le}}\mathbb{E}[Z_2]-\epsilon(\eta)\mathrel{\overset{(b)}{\le}}\Delta_\eta\mathrel{\overset{(b)}{\le}}\mathbb{E}[Z_2]+\epsilon(\eta)\mathrel{\overset{(a)}{\le}}\bar{Z}_2+\xi_\eta(Z_2)+\epsilon(\eta), Z ˉ 2 − ξ η ( Z 2 ) − ϵ ( η ) ≤ ( a ) E [ Z 2 ] − ϵ ( η ) ≤ ( b ) Δ η ≤ ( b ) E [ Z 2 ] + ϵ ( η ) ≤ ( a ) Z ˉ 2 + ξ η ( Z 2 ) + ϵ ( η ) ,
where (a) uses the event and (b) uses Lemma E.3(a). Hence:
0 ≤ Δ η − ( Z ˉ 2 − ξ η ( Z 2 ) − ϵ ( η ) ) ≤ 2 ξ η ( Z 2 ) + 2 ϵ ( η ) . (41) 0\leq\Delta_\eta-\left(\bar{Z}_2-\xi_\eta(Z_2)-\epsilon(\eta)\right)\leq2\xi_\eta(Z_2)+2\epsilon(\eta). \tag*{(41)} 0 ≤ Δ η − ( Z ˉ 2 − ξ η ( Z 2 ) − ϵ ( η ) ) ≤ 2 ξ η ( Z 2 ) + 2 ϵ ( η ) . (41)
Likewise, by E [ Z I S ] = Δ η \mathbb{E}[Z_{\mathrm{IS}}]=\Delta_\eta E [ Z IS ] = Δ η (Lemma E.3(a)):
0 ≤ Δ η − ( Z ˉ I S − ξ η ( Z I S ) ) ≤ 2 ξ η ( Z I S ) . (42) 0\leq\Delta_\eta-\left(\bar{Z}_{\mathrm{IS}}-\xi_\eta(Z_{\mathrm{IS}})\right)\leq2\xi_\eta(Z_{\mathrm{IS}}). \tag*{(42)} 0 ≤ Δ η − ( Z ˉ IS − ξ η ( Z IS ) ) ≤ 2 ξ η ( Z IS ) . (42)
For (i), L η L_\eta L η is the larger of the two lower bounds in (41) and (42) , so L η ≤ Δ η L_\eta\leq\Delta_\eta L η ≤ Δ η and:
Δ η − L η ≤ 2 min { ξ η ( Z 2 ) + ϵ ( η ) , ξ η ( Z I S ) } . (43) \Delta_\eta-L_\eta\leq2\min\{\xi_\eta(Z_2)+\epsilon(\eta),\xi_\eta(Z_{\mathrm{IS}})\}. \tag*{(43)} Δ η − L η ≤ 2 min { ξ η ( Z 2 ) + ϵ ( η ) , ξ η ( Z IS )} . (43)
Since L η ≥ L 0 = 0 L_\eta\geq L_0=0 L η ≥ L 0 = 0 :
J ( π ^ ) − J ( π ) = Δ η ^ ≥ L η ^ ≥ 0. (44) J(\hat{\pi})-J(\pi)=\Delta_{\hat{\eta}}\geq L_{\hat{\eta}}\geq0. \tag*{(44)} J ( π ^ ) − J ( π ) = Δ η ^ ≥ L η ^ ≥ 0. (44)
For (ii), J ( π ^ ) − J ( π ) ≥ 0 J(\hat{\pi})-J(\pi)\geq0 J ( π ^ ) − J ( π ) ≥ 0 by (i), and for every η ∈ Λ \eta\in\Lambda η ∈ Λ :
J ( π ^ ) − J ( π ) ≥ ( a ) L η ^ ≥ ( b ) L η ≥ ( c ) Δ η − 2 min { ξ η ( Z 2 ) + ϵ ( η ) , ξ η ( Z I S ) } , J(\hat{\pi})-J(\pi)\mathrel{\overset{(a)}{\geq}}L_{\hat{\eta}}\mathrel{\overset{(b)}{\geq}}L_\eta\mathrel{\overset{(c)}{\geq}}\Delta_\eta-2\min\{\xi_\eta(Z_2)+\epsilon(\eta),\xi_\eta(Z_{\mathrm{IS}})\}, J ( π ^ ) − J ( π ) ≥ ( a ) L η ^ ≥ ( b ) L η ≥ ( c ) Δ η − 2 min { ξ η ( Z 2 ) + ϵ ( η ) , ξ η ( Z IS )} ,
where (a) uses (44) , (b) the definition of η ^ \hat{\eta} η ^ , and (c) (43) . This is Δ η − 2 c η \Delta_\eta-2c_\eta Δ η − 2 c η ; maximizing over η \eta η and with 0 gives (ii).
For (iii), let Y 1 , … , Y n ∈ [ − B , B ] Y_1,\ldots,Y_n\in[-B,B] Y 1 , … , Y n ∈ [ − B , B ] be the validation values of Z Z Z , with mean Z ˉ \bar{Z} Z ˉ , so Y ℓ 2 ≤ B 2 Y_\ell^2\leq B^2 Y ℓ 2 ≤ B 2 . Averaging over ℓ \ell ℓ :
1 n ∑ ℓ = 1 n ( Y ℓ − Z ˉ ) 2 = 1 n ∑ ℓ = 1 n Y ℓ 2 − Z ˉ 2 ≤ B 2 , so σ ^ 2 ≤ n n − 1 B 2 . (45) \frac{1}{n}\sum_{\ell=1}^{n}(Y_\ell-\bar{Z})^2=\frac{1}{n}\sum_{\ell=1}^{n}Y_\ell^2-\bar{Z}^2\leq B^2,\qquad\text{so}\qquad\hat{\sigma}^2\leq\frac{n}{n-1}B^2. \tag*{(45)} n 1 ℓ = 1 ∑ n ( Y ℓ − Z ˉ ) 2 = n 1 ℓ = 1 ∑ n Y ℓ 2 − Z ˉ 2 ≤ B 2 , so σ ^ 2 ≤ n − 1 n B 2 . (45)
Put x : = ι / ( n − 1 ) x:=\sqrt{\iota/(n-1)} x := ι / ( n − 1 ) and assume n − 1 > ι n-1>\iota n − 1 > ι , so x < 1 x<1 x < 1 and:
ξ η ( Z ) = σ ^ 2 ι n + 14 B ι 3 ( n − 1 ) ≤ ( a ) B 2 ι n − 1 + 14 3 B x 2 = 2 B x + 14 3 B x 2 ≤ ( b ) ( 2 + 14 3 ) B x ≤ 8 B x , (46) \begin{aligned}
\xi_\eta(Z)=\hat{\sigma}\sqrt{\frac{2\iota}{n}}+\frac{14B\iota}{3(n-1)}
\mathrel{\overset{(a)}{\leq}}B\sqrt{\frac{2\iota}{n-1}}+\frac{14}{3}Bx^2 \\
&=\sqrt{2}Bx+\frac{14}{3}Bx^2\mathrel{\overset{(b)}{\leq}}\left(\sqrt{2}+\frac{14}{3}\right)Bx\leq8Bx, \tag*{(46)}
\end{aligned} ξ η ( Z ) = σ ^ n 2 ι + 3 ( n − 1 ) 14 B ι ≤ ( a ) B n − 1 2 ι + 3 14 B x 2 = 2 B x + 3 14 B x 2 ≤ ( b ) ( 2 + 3 14 ) B x ≤ 8 B x , (46)
where (a) uses (45) and (b) uses x < 1 x<1 x < 1 . Since ∣ Δ η ∣ = ∣ E [ Z I S ] ∣ ≤ B |\Delta_\eta|=|\mathbb{E}[Z_{\mathrm{IS}}]|\leq B ∣ Δ η ∣ = ∣ E [ Z IS ] ∣ ≤ B , n − 1 > 256 B 2 ι / Δ η 2 ⇒ n − 1 > 256 ι > ι n-1>256B^2\iota/\Delta_\eta^2\Rightarrow n-1>256\iota>\iota n − 1 > 256 B 2 ι / Δ η 2 ⇒ n − 1 > 256 ι > ι , so (46) applies with B = B I S , η B=B_{\mathrm{IS},\eta} B = B IS , η and:
Δ η − L η ≤ ( a ) 2 ξ η ( Z I S ) ≤ ( b ) 16 B ι n − 1 < ( c ) Δ η , \Delta_\eta-L_\eta\mathrel{\overset{(a)}{\leq}}2\xi_\eta(Z_{\mathrm{IS}})\mathrel{\overset{(b)}{\leq}}16B\sqrt{\frac{\iota}{n-1}}\mathrel{\overset{(c)}{<}}\Delta_\eta, Δ η − L η ≤ ( a ) 2 ξ η ( Z IS ) ≤ ( b ) 16 B n − 1 ι < ( c ) Δ η ,
where (a) uses (43) , (b) (46) , and (c) n − 1 > 256 B 2 ι / Δ η 2 n-1>256B^2\iota/\Delta_\eta^2 n − 1 > 256 B 2 ι / Δ η 2 . Hence L η ^ ≥ L η > 0 = L 0 L_{\hat{\eta}}\geq L_\eta>0=L_0 L η ^ ≥ L η > 0 = L 0 , so η ^ ≠ 0 \hat{\eta}\neq0 η ^ = 0 and π ^ ≠ π \hat{\pi}\neq\pi π ^ = π .
Corollary E.5 (Every update improves). If round κ \kappa κ of Algorithm 1 uses fresh trajectories and level δ / 2 κ \delta/2^\kappa δ / 2 κ , then with probability at least 1 − δ 1-\delta 1 − δ , every update made by Algorithm 1 strictly improves J J J .
Proof. In round κ \kappa κ , Theorem 3.7 at level δ / 2 κ \delta/2^\kappa δ / 2 κ gives, with probability at least 1 − δ / 2 κ 1-\delta/2^\kappa 1 − δ / 2 κ , L η > 0 ⇒ Δ η ≥ L η > 0 L_\eta>0\Rightarrow\Delta_\eta\geq L_\eta>0 L η > 0 ⇒ Δ η ≥ L η > 0 for every candidate, and updates require L η ^ > 0 L_{\hat{\eta}}>0 L η ^ > 0 . A union bound over rounds bounds the failure probability by ∑ κ ≥ 1 δ / 2 κ = δ \sum_{\kappa\geq1}\delta/2^\kappa=\delta ∑ κ ≥ 1 δ / 2 κ = δ .
Theorem 3.7 needs only candidates fixed before the validation data (the split T , V \mathcal{T},\mathcal{V} T , V in Algorithm 1) and known bounds (Lemma E.3(b)), not a good optimizer of Z 2 Z_2 Z 2 .
A Betting Confidence Bound
Item (i) of Theorem 3.7 also holds with Z ˉ − ξ η ( Z ) \bar{Z}-\xi_{\eta}(Z) Z ˉ − ξ η ( Z ) replaced by a betting lower bound [40 ] , often tighter for skewed statistics such as Z I S Z_{\mathrm{IS}} Z IS . For a statistic ∣ Z ∣ ≤ B |Z| \le B ∣ Z ∣ ≤ B with validation values Y 1 , … , Y n Y_1,\ldots,Y_n Y 1 , … , Y n , a finite set F ⊂ ( 0 , 1 ] F \subset(0,1] F ⊂ ( 0 , 1 ] of bet sizes, and z ∈ ( − B , B ] z \in(-B,B] z ∈ ( − B , B ] , define:
W ( z ) : = 1 ∣ F ∣ ∑ ν ∈ F ∏ ℓ = 1 n ( 1 + ν ( Y ℓ + B z + B − 1 ) ) , \mathcal{W}(z) := \frac{1}{|F|}\sum_{\nu\in F}\prod_{\ell=1}^{n}\left(1+\nu\left(\frac{Y_{\ell}+B}{z+B}-1\right)\right), W ( z ) := ∣ F ∣ 1 ν ∈ F ∑ ℓ = 1 ∏ n ( 1 + ν ( z + B Y ℓ + B − 1 ) ) ,
and:
Z ˉ b e t : = sup { z ∈ ( − B , B ] : W ( z ) ≥ 2 ∣ Λ ∣ δ } , (47) \bar{Z}^{\mathrm{bet}} := \sup\left\{z\in(-B,B] : \mathcal{W}(z)\ge\frac{2|\Lambda|}{\delta}\right\}, \tag*{(47)} Z ˉ bet := sup { z ∈ ( − B , B ] : W ( z ) ≥ δ 2∣Λ∣ } , (47)
with Z ˉ b e t : = − B \bar{Z}^{\mathrm{bet}} := -B Z ˉ bet := − B if the set is empty.
Proposition E.6 (Validity of the betting bound). For each statistic , Pr ( Z ˉ b e t > E [ Z ] ) ≤ δ / ( 2 ∣ Λ ∣ ) \Pr(\bar{Z}^{\mathrm{bet}}>\mathbb{E}[Z])\le\delta/(2|\Lambda|) Pr ( Z ˉ bet > E [ Z ]) ≤ δ / ( 2∣Λ∣ ) . Hence item (i) of Theorem 3.7 holds with Z ˉ b e t \bar{Z}^{\mathrm{bet}} Z ˉ bet in place of Z ˉ − ξ η ( Z ) \bar{Z}-\xi_{\eta}(Z) Z ˉ − ξ η ( Z ) .
Proof. Let μ Z : = E [ Z ] \mu_Z := \mathbb{E}[Z] μ Z := E [ Z ] , fix z ≥ μ Z z\ge\mu_Z z ≥ μ Z with z > − B z>-B z > − B , and let X ℓ : = ( Y ℓ + B ) / ( z + B ) ≥ 0 X_{\ell}:=(Y_{\ell}+B)/(z+B)\ge0 X ℓ := ( Y ℓ + B ) / ( z + B ) ≥ 0 . For ν ∈ ( 0 , 1 ] \nu\in(0,1] ν ∈ ( 0 , 1 ] :
E [ X ℓ ] = μ Z + B z + B ≤ 1 , 1 + ν ( X ℓ − 1 ) ≥ 1 − ν ≥ 0. \mathbb{E}[X_{\ell}]=\frac{\mu_Z+B}{z+B}\le1,\qquad1+\nu(X_{\ell}-1)\ge1-\nu\ge0. E [ X ℓ ] = z + B μ Z + B ≤ 1 , 1 + ν ( X ℓ − 1 ) ≥ 1 − ν ≥ 0.
Hence, for each ν ∈ F \nu\in F ν ∈ F , ( ∏ ℓ ≤ ℓ ′ ( 1 + ν ( X ℓ − 1 ) ) ) ℓ ′ = 0 n \left(\prod_{\ell\leq\ell'}(1+\nu(X_{\ell}-1))\right)_{\ell'=0}^{n} ( ∏ ℓ ≤ ℓ ′ ( 1 + ν ( X ℓ − 1 )) ) ℓ ′ = 0 n is a nonnegative supermartingale starting at 1 1 1 , and so is its average over F F F . By Ville’s inequality [37 ] with threshold 2 ∣ Λ ∣ / δ 2|\Lambda|/\delta 2∣Λ∣/ δ :
Pr ( W ( z ) ≥ 2 ∣ Λ ∣ δ ) ≤ δ 2 ∣ Λ ∣ . \Pr\left(\mathcal{W}(z)\ge\frac{2|\Lambda|}{\delta}\right)\le\frac{\delta}{2|\Lambda|}. Pr ( W ( z ) ≥ δ 2∣Λ∣ ) ≤ 2∣Λ∣ δ .
Each factor is nonincreasing in z z z , so W \mathcal{W} W is nonincreasing and Z ˉ b e t > μ Z ⇒ ∃ z > μ Z : W ( z ) ≥ 2 ∣ Λ ∣ / δ ⇒ W ( μ Z ) ≥ 2 ∣ Λ ∣ / δ \bar{Z}^{\mathrm{bet}}>\mu_Z\Rightarrow\exists z>\mu_Z:\mathcal{W}(z)\ge2|\Lambda|/\delta\Rightarrow\mathcal{W}(\mu_Z)\ge2|\Lambda|/\delta Z ˉ bet > μ Z ⇒ ∃ z > μ Z : W ( z ) ≥ 2∣Λ∣/ δ ⇒ W ( μ Z ) ≥ 2∣Λ∣/ δ . If μ Z = − B \mu_Z=-B μ Z = − B , then Y ℓ = − B Y_{\ell}=-B Y ℓ = − B almost surely, every factor equals 1 − ν 1-\nu 1 − ν , and W ≤ 1 < 2 ∣ Λ ∣ / δ \mathcal{W}\le1<2|\Lambda|/\delta W ≤ 1 < 2∣Λ∣/ δ . In both cases, Pr ( Z ˉ b e t > μ Z ) ≤ Pr ( W ( μ Z ) ≥ 2 ∣ Λ ∣ / δ ) ≤ δ / ( 2 ∣ Λ ∣ ) \Pr(\bar{Z}^{\mathrm{bet}}>\mu_Z)\le\Pr(\mathcal{W}(\mu_Z)\ge2|\Lambda|/\delta)\le\delta/(2|\Lambda|) Pr ( Z ˉ bet > μ Z ) ≤ Pr ( W ( μ Z ) ≥ 2∣Λ∣/ δ ) ≤ δ / ( 2∣Λ∣ ) .
By the union bound over the 2 ∣ Λ ∣ 2|\Lambda| 2∣Λ∣ statistics, with probability at least 1 − δ 1-\delta 1 − δ the lower sides of (41) and (42) hold with Z ˉ b e t \bar{Z}^{\mathrm{bet}} Z ˉ bet in place of Z ˉ − ξ η ( Z ) \bar{Z}-\xi_{\eta}(Z) Z ˉ − ξ η ( Z ) , so L η ≤ Δ η L_{\eta}\le\Delta_{\eta} L η ≤ Δ η for every η \eta η . Items (ii) and (iii) use the two-sided radius, so they are stated for the empirical Bernstein bound only, and the betting bound is not claimed to satisfy them.
A Stationary Point of the First-Order Objective
Policies at which J ( 1 ) J^{(1)} J ( 1 ) has no improving direction (the second case of Theorem 3.6) arise from complementary decisions.
Proposition E.7 (Complementarity). Two agents independently choose 1 with probabilities φ 1 , φ 2 \varphi_1,\varphi_2 φ 1 , φ 2 , and J = 3 φ 1 φ 2 − φ 1 − φ 2 J=3\varphi_1\varphi_2-\varphi_1-\varphi_2 J = 3 φ 1 φ 2 − φ 1 − φ 2 . Start from ( p , p ) (p,p) ( p , p ) with p ∈ ( 0 , 1 3 ) p\in(0,\frac{1}{3}) p ∈ ( 0 , 3 1 ) , and let both agents move to p + α p+\alpha p + α with α ∈ ( 0 , 1 − p ) \alpha\in(0,1-p) α ∈ ( 0 , 1 − p ) . Then:
(a) each single move lowers J J J , and projected gradient flow on [ 0 , 1 ] 2 [0,1]^2 [ 0 , 1 ] 2 from ( p , p ) (p,p) ( p , p ) reaches the strict local maximum ( 0 , 0 ) (0,0) ( 0 , 0 ) ;
(b) J ( 1 ) = 2 α ( 3 p − 1 ) < 0 J^{(1)}=2\alpha(3p-1)<0 J ( 1 ) = 2 α ( 3 p − 1 ) < 0 and J ( 2 ) = 3 α 2 J^{(2)}=3\alpha^2 J ( 2 ) = 3 α 2 , and the joint move gains if and only if 3 ( 2 p + α ) > 2 3(2p+\alpha)>2 3 ( 2 p + α ) > 2 ;
(c) with probability at least 1 − δ 1-\delta 1 − δ , the sign of J ( 2 ) J^{(2)} J ( 2 ) is identified from n ≥ 1 / [ 9 δ p 2 ( 1 − p ) 2 ] n\ge1/[9\delta p^2(1-p)^2] n ≥ 1/ [ 9 δ p 2 ( 1 − p ) 2 ] trajectories.
Proof. With r ( 1 , 1 ) = 1 r(1,1)=1 r ( 1 , 1 ) = 1 , r ( 0 , 0 ) = 0 r(0,0)=0 r ( 0 , 0 ) = 0 , r ( 1 , 0 ) = r ( 0 , 1 ) = − 1 r(1,0)=r(0,1)=-1 r ( 1 , 0 ) = r ( 0 , 1 ) = − 1 and independence:
J = φ 1 φ 2 ⋅ 1 + φ 1 ( 1 − φ 2 ) ⋅ ( − 1 ) + ( 1 − φ 1 ) φ 2 ⋅ ( − 1 ) = 3 φ 1 φ 2 − φ 1 − φ 2 . J=\varphi_1\varphi_2\cdot1+\varphi_1(1-\varphi_2)\cdot(-1)+(1-\varphi_1)\varphi_2\cdot(-1)=3\varphi_1\varphi_2-\varphi_1-\varphi_2. J = φ 1 φ 2 ⋅ 1 + φ 1 ( 1 − φ 2 ) ⋅ ( − 1 ) + ( 1 − φ 1 ) φ 2 ⋅ ( − 1 ) = 3 φ 1 φ 2 − φ 1 − φ 2 .
For (a), since p < 1 3 p<\frac{1}{3} p < 3 1 , a single move of agent 1 gives:
J ( p + α , p ) − J ( p , p ) = ( 3 ( p + α ) p − ( p + α ) − p ) − ( 3 p 2 − 2 p ) = α ( 3 p − 1 ) < 0 , J(p+\alpha,p)-J(p,p)=\left(3(p+\alpha)p-(p+\alpha)-p\right)-\left(3p^2-2p\right)=\alpha(3p-1)<0, J ( p + α , p ) − J ( p , p ) = ( 3 ( p + α ) p − ( p + α ) − p ) − ( 3 p 2 − 2 p ) = α ( 3 p − 1 ) < 0 ,
and agent 2 is symmetric. Since ∂ φ 1 J = 3 φ 2 − 1 \partial_{\varphi_1}J=3\varphi_2-1 ∂ φ 1 J = 3 φ 2 − 1 and ∂ φ 2 J = 3 φ 1 − 1 \partial_{\varphi_2}J=3\varphi_1-1 ∂ φ 2 J = 3 φ 1 − 1 , the flow from ( p , p ) (p,p) ( p , p ) stays on φ 1 = φ 2 = φ \varphi_1=\varphi_2=\varphi φ 1 = φ 2 = φ with φ ˙ = 3 φ − 1 < 0 \dot{\varphi}=3\varphi-1<0 φ ˙ = 3 φ − 1 < 0 for φ < 1 3 \varphi<\frac{1}{3} φ < 3 1 , so φ \varphi φ reaches 0 0 0 in finite time, and the projection keeps it there since ∂ φ i J = − 1 < 0 \partial_{\varphi_i}J=-1<0 ∂ φ i J = − 1 < 0 at ( 0 , 0 ) (0,0) ( 0 , 0 ) . Near ( 0 , 0 ) (0,0) ( 0 , 0 ) in [ 0 , 1 ] 2 [0,1]^2 [ 0 , 1 ] 2 , J = 3 φ 1 φ 2 − φ 1 − φ 2 < 0 = J ( 0 , 0 ) J=3\varphi_1\varphi_2-\varphi_1-\varphi_2<0=J(0,0) J = 3 φ 1 φ 2 − φ 1 − φ 2 < 0 = J ( 0 , 0 ) unless φ 1 = φ 2 = 0 \varphi_1=\varphi_2=0 φ 1 = φ 2 = 0 , so ( 0 , 0 ) (0,0) ( 0 , 0 ) is a strict local maximum of J J J on [ 0 , 1 ] 2 [0,1]^2 [ 0 , 1 ] 2 .
For (b), a single move changes one decision per trajectory, so by Corollary D.4 its gain is L i = α ( 3 p − 1 ) L_i=\alpha(3p-1) L i = α ( 3 p − 1 ) and J ( 1 ) = L 1 + L 2 = 2 α ( 3 p − 1 ) J^{(1)}=L_1+L_2=2\alpha(3p-1) J ( 1 ) = L 1 + L 2 = 2 α ( 3 p − 1 ) . The joint move changes two decisions per trajectory, so J ( 1 ) + J ( 2 ) J^{(1)}+J^{(2)} J ( 1 ) + J ( 2 ) is exact by Corollary D.4 :
J ( 2 ) = J ( p + α , p + α ) − J ( p , p ) − 2 α ( 3 p − 1 ) = ( 3 ( p + α ) 2 − 2 ( p + α ) ) − ( 3 p 2 − 2 p ) − 6 p α + 2 α = 3 α 2 , J^{(2)}=J(p+\alpha,p+\alpha)-J(p,p)-2\alpha(3p-1)=\left(3(p+\alpha)^2-2(p+\alpha)\right)-\left(3p^2-2p\right)-6p\alpha+2\alpha=3\alpha^2, J ( 2 ) = J ( p + α , p + α ) − J ( p , p ) − 2 α ( 3 p − 1 ) = ( 3 ( p + α ) 2 − 2 ( p + α ) ) − ( 3 p 2 − 2 p ) − 6 p α + 2 α = 3 α 2 ,
J ( 1 ) + J ( 2 ) = 2 α ( 3 p − 1 ) + 3 α 2 = α ( 6 p − 2 + 3 α ) = α ( 3 ( 2 p + α ) − 2 ) , J^{(1)}+J^{(2)}=2\alpha(3p-1)+3\alpha^2=\alpha(6p-2+3\alpha)=\alpha\left(3(2p+\alpha)-2\right), J ( 1 ) + J ( 2 ) = 2 α ( 3 p − 1 ) + 3 α 2 = α ( 6 p − 2 + 3 α ) = α ( 3 ( 2 p + α ) − 2 ) ,
so J ( 1 ) + J ( 2 ) > 0 ⟺ 3 ( 2 p + α ) > 2 J^{(1)}+J^{(2)}>0\Longleftrightarrow3(2p+\alpha)>2 J ( 1 ) + J ( 2 ) > 0 ⟺ 3 ( 2 p + α ) > 2 .
For (c), under π \pi π , ρ − 1 = p + α p − 1 = α p \rho-1=\frac{p+\alpha}{p}-1=\frac{\alpha}{p} ρ − 1 = p p + α − 1 = p α with probability p p p and ρ − 1 = 1 − p − α 1 − p − 1 = − α 1 − p \rho-1=\frac{1-p-\alpha}{1-p}-1=-\frac{\alpha}{1-p} ρ − 1 = 1 − p 1 − p − α − 1 = − 1 − p α with probability 1 − p 1-p 1 − p , so:
E [ ( ρ − 1 ) 2 ] = p ⋅ α 2 p 2 + ( 1 − p ) ⋅ α 2 ( 1 − p ) 2 = α 2 p ( 1 − p ) . \mathbb{E}\left[(\rho-1)^2\right]=p\cdot\frac{\alpha^2}{p^2}+(1-p)\cdot\frac{\alpha^2}{(1-p)^2}=\frac{\alpha^2}{p(1-p)}. E [ ( ρ − 1 ) 2 ] = p ⋅ p 2 α 2 + ( 1 − p ) ⋅ ( 1 − p ) 2 α 2 = p ( 1 − p ) α 2 .
For Z : = ( ρ 1 − 1 ) ( ρ 2 − 1 ) G 2 Z:=(\rho_1-1)(\rho_2-1)G_2 Z := ( ρ 1 − 1 ) ( ρ 2 − 1 ) G 2 with ∣ G 2 ∣ ≤ 1 \lvert G_2\rvert\le1 ∣ G 2 ∣ ≤ 1 , independence gives E [ Z ] = J ( 2 ) = 3 α 2 \mathbb{E}[Z]=J^{(2)}=3\alpha^2 E [ Z ] = J ( 2 ) = 3 α 2 and:
E [ Z 2 ] ≤ E [ ( ρ 1 − 1 ) 2 ] E [ ( ρ 2 − 1 ) 2 ] = α 4 p 2 ( 1 − p ) 2 . \mathbb{E}[Z^2]\le\mathbb{E}\left[(\rho_1-1)^2\right]\mathbb{E}\left[(\rho_2-1)^2\right]=\frac{\alpha^4}{p^2(1-p)^2}. E [ Z 2 ] ≤ E [ ( ρ 1 − 1 ) 2 ] E [ ( ρ 2 − 1 ) 2 ] = p 2 ( 1 − p ) 2 α 4 .
The sample mean Z ˉ \bar{Z} Z ˉ of n n n trajectories satisfies:
Pr ( Z ˉ ≤ 0 ) ≤ Pr ( ∣ Z ˉ − 3 α 2 ∣ ≥ 3 α 2 ) ≤ ( a ) E [ Z 2 ] 9 n α 4 ≤ 1 9 n p 2 ( 1 − p ) 2 , \Pr(\bar{Z}\le0)\le\Pr\left(\lvert\bar{Z}-3\alpha^2\rvert\ge3\alpha^2\right)\underset{(a)}{\le}\frac{\mathbb{E}[Z^2]}{9n\alpha^4}\le\frac{1}{9np^2(1-p)^2}, Pr ( Z ˉ ≤ 0 ) ≤ Pr ( ∣ Z ˉ − 3 α 2 ∣ ≥ 3 α 2 ) ( a ) ≤ 9 n α 4 E [ Z 2 ] ≤ 9 n p 2 ( 1 − p ) 2 1 ,
where (a) is Chebyshev’s inequality, and the right side is at most δ \delta δ for n ≥ 1 / [ 9 δ p 2 ( 1 − p ) 2 ] n\ge1/[9\delta p^2(1-p)^2] n ≥ 1/ [ 9 δ p 2 ( 1 − p ) 2 ] .
At p = 1 3 p=\frac{1}{3} p = 3 1 , J ( 1 ) = 0 J^{(1)}=0 J ( 1 ) = 0 for every α \alpha α while J ( 2 ) = 3 α 2 > 0 J^{(2)}=3\alpha^2>0 J ( 2 ) = 3 α 2 > 0 , the case μ 1 = 0 < μ 2 \mu_1=0<\mu_2 μ 1 = 0 < μ 2 of Theorem 3.6; for an orchestrator with frozen workers, the same holds for two complementary decisions, with I 0 → 0 I_{0\to0} I 0 → 0 in place of I 1 → 2 I_{1\to2} I 1 → 2 .
Certified Leaps
Throughout, for p , q ∈ ( 0 , 1 ) p,q\in(0,1) p , q ∈ ( 0 , 1 ) , kl ( p ∥ q ) \operatorname{kl}(p\|q) kl ( p ∥ q ) is the Kullback–Leibler divergence between Bern ( p ) \operatorname{Bern}(p) Bern ( p ) and Bern ( q ) \operatorname{Bern}(q) Bern ( q ) . We use two standard facts [36 ] :
chi-square bound: kl ( p ∥ q ) ≤ χ 2 ( p ∥ q ) = ( p − q ) 2 / ( q ( 1 − q ) ) \operatorname{kl}(p\|q)\le\chi^2(p\|q)=(p-q)^2/(q(1-q)) kl ( p ∥ q ) ≤ χ 2 ( p ∥ q ) = ( p − q ) 2 / ( q ( 1 − q )) ;
data processing: for laws P , Q \mathbb{P},\mathbb{Q} P , Q of the data and any event A A A , kl ( P ( A ) ∥ Q ( A ) ) ≤ KL ( P ∥ Q ) \operatorname{kl}(\mathbb{P}(A)\|\mathbb{Q}(A))\le\operatorname{KL}(\mathbb{P}\|\mathbb{Q}) kl ( P ( A ) ∥ Q ( A )) ≤ KL ( P ∥ Q ) , and KL ( P ⊗ n ∥ Q ⊗ n ) = n KL ( P ∥ Q ) \operatorname{KL}(\mathbb{P}^{\otimes n}\|\mathbb{Q}^{\otimes n})=n\operatorname{KL}(\mathbb{P}\|\mathbb{Q}) KL ( P ⊗ n ∥ Q ⊗ n ) = n KL ( P ∥ Q ) ;
For p ≥ q p\ge q p ≥ q , kl ( p ∥ q ) \operatorname{kl}(p\|q) kl ( p ∥ q ) is nondecreasing in p p p and nonincreasing in q q q . Hence, if P ( A ) ≥ 1 2 \mathbb{P}(A)\ge\frac{1}{2} P ( A ) ≥ 2 1 and Q ( A ) ≤ δ ≤ 1 2 \mathbb{Q}(A)\le\delta\le\frac{1}{2} Q ( A ) ≤ δ ≤ 2 1 :
KL ( P ∥ Q ) ≥ kl ( 1 2 ∥ δ ) = 1 2 log 1 4 δ ( 1 − δ ) ≥ 1 2 log 1 4 δ . (48) \operatorname{KL}(\mathbb{P}\|\mathbb{Q})\ge\operatorname{kl}\left(\frac{1}{2}\|\delta\right)=\frac{1}{2}\log\frac{1}{4\delta(1-\delta)}\ge\frac{1}{2}\log\frac{1}{4\delta}. \tag*{(48)} KL ( P ∥ Q ) ≥ kl ( 2 1 ∥ δ ) = 2 1 log 4 δ ( 1 − δ ) 1 ≥ 2 1 log 4 δ 1 . (48)
Certified Leaps
Fix a plan , a sequence of orchestrator actions, and let:
ν = Pr π ( τ follows the plan ) , R ( τ ) ∼ Bern ( ν + g 1 { τ follows the plan } ) , 0 < v < v + g ≤ 1 , \nu=\Pr_{\pi}(\tau\text{ follows the plan}),\qquad R(\tau)\sim\operatorname{Bern}\left(\nu+g\,1\{\tau\text{ follows the plan}\}\right),\qquad0<v<v+g\le1, ν = π Pr ( τ follows the plan ) , R ( τ ) ∼ Bern ( ν + g 1 { τ follows the plan } ) , 0 < v < v + g ≤ 1 ,
with return G 1 = R G_1=R G 1 = R . Write F ( τ ) = 1 { τ follows the plan } F(\tau)=1\{\tau\text{ follows the plan}\} F ( τ ) = 1 { τ follows the plan } , so that J ( π ) = v + g ν J(\pi)=v+g\nu J ( π ) = v + g ν .
Proposition F.1 (What a trust region cannot certify). In this setting:
(i) a rule that certifies a gain with probability at most δ ≤ 1 2 \delta\le\frac{1}{2} δ ≤ 2 1 when g = 0 g=0 g = 0 , and with probability at least 1 2 \frac{1}{2} 2 1 here, needs at least v ( 1 − v ) log 1 4 δ / ( 2 ν g 2 ) v(1-v)\log\frac{1}{4\delta}/(2\nu g^2) v ( 1 − v ) log 4 δ 1 / ( 2 ν g 2 ) trajectories of π \pi π ;
(ii) a step of (7) gains at most g ν ( ( 1 + η ) K + 1 − 1 ) g\nu((1+\eta)^{K+1}-1) g ν (( 1 + η ) K + 1 − 1 ) ;
Controlled Experiments
(iii) if the actions of the plan are always legal, the policy that follows it gains g ( 1 − ν ) g(1-\nu) g ( 1 − ν ) , and for every ν < 1 \nu< 1 ν < 1 it is certified from n ′ n' n ′ of its own trajectories, paired with trajectories of π \pi π on the same initial states, once n ′ − 1 > 256 log ( 4 / δ ) / ( g ( 1 − ν ) ) 2 n' - 1 > 256\log(4/\delta)/(g(1-\nu))^2 n ′ − 1 > 256 log ( 4/ δ ) / ( g ( 1 − ν ) ) 2 .
Proof. For (i), let P g \mathbb{P}_g P g be the law of one trajectory of π \pi π in this setting and P 0 \mathbb{P}_0 P 0 the law with g = 0 g=0 g = 0 . The two laws share the law of the actions and states, and differ only in the law of R R R when F = 1 F=1 F = 1 :
KL ( P g ∥ P 0 ) = ν kl ( v + g ∥ v ) ≤ ( a ) ν g 2 v ( 1 − v ) . \operatorname{KL}(\mathbb{P}_g\Vert\mathbb{P}_0)=\nu\operatorname{kl}(v+g\Vert v)\mathrel{\overset{(a)}{\leq}}\frac{\nu g^2}{v(1-v)}. KL ( P g ∥ P 0 ) = ν kl ( v + g ∥ v ) ≤ ( a ) v ( 1 − v ) ν g 2 .
where (a) is the chi-square bound. Let A A A be the event that the rule certifies a gain from n n n trajectories. Then P g n ( A ) ≥ 1 2 \mathbb{P}_g^n(A)\geq\frac{1}{2} P g n ( A ) ≥ 2 1 and P 0 n ( A ) ≤ δ \mathbb{P}_0^n(A)\leq\delta P 0 n ( A ) ≤ δ , so by (48) and data processing:
1 2 log 1 4 δ ≤ n KL ( P g ∥ P 0 ) ≤ n ν g 2 v ( 1 − v ) ⟹ n ≥ v ( 1 − v ) 2 ν g 2 log 1 4 δ . \frac{1}{2}\log\frac{1}{4\delta}\leq n\operatorname{KL}(\mathbb{P}_g\Vert\mathbb{P}_0)\leq\frac{n\nu g^2}{v(1-v)}
\quad\Longrightarrow\quad
n\geq\frac{v(1-v)}{2\nu g^2}\log\frac{1}{4\delta}. 2 1 log 4 δ 1 ≤ n KL ( P g ∥ P 0 ) ≤ v ( 1 − v ) n ν g 2 ⟹ n ≥ 2 ν g 2 v ( 1 − v ) log 4 δ 1 .
For (ii), let π ′ = π θ , η ′ \pi'=\pi'_{\theta,\eta} π ′ = π θ , η ′ and ν ′ = Pr π ′ ( F = 1 ) \nu'=\Pr_{\pi'}(F=1) ν ′ = Pr π ′ ( F = 1 ) . A trajectory has at most K + 1 K+1 K + 1 orchestrator steps, and ρ t ≤ 1 + η \rho_t\leq1+\eta ρ t ≤ 1 + η at each of them by Proposition 3.5. Hence, by Lemma B.3:
J ( π ′ ) − J ( π ) = g ( ν ′ − ν ) = g E π [ F ( ∏ t ≤ H ρ t − 1 ) ] ≤ g E π [ F ( ( 1 + η ) K + 1 − 1 ) ] = g ν ( ( 1 + η ) K + 1 − 1 ) . J(\pi')-J(\pi)=g(\nu'-\nu)=g\mathbb{E}_{\pi}\left[F\left(\prod_{t\leq H}\rho_t-1\right)\right]\leq g\mathbb{E}_{\pi}\left[F\left((1+\eta)^{K+1}-1\right)\right]=g\nu\left((1+\eta)^{K+1}-1\right). J ( π ′ ) − J ( π ) = g ( ν ′ − ν ) = g E π [ F ( t ≤ H ∏ ρ t − 1 ) ] ≤ g E π [ F ( ( 1 + η ) K + 1 − 1 ) ] = g ν ( ( 1 + η ) K + 1 − 1 ) .
For (iii), the policy π ~ \tilde{\pi} π ~ that follows the plan has F = 1 F=1 F = 1 almost surely, so J ( π ~ ) − J ( π ) = ( v + g ) − ( v + g ν ) = g ( 1 − ν ) J(\tilde{\pi})-J(\pi)=(v+g)-(v+g\nu)=g(1-\nu) J ( π ~ ) − J ( π ) = ( v + g ) − ( v + g ν ) = g ( 1 − ν ) . For i ≤ n ′ i\leq n' i ≤ n ′ , τ i \tau_i τ i and τ ~ i \tilde{\tau}_i τ ~ i share s 1 ∼ d 0 s_1\sim d_0 s 1 ∼ d 0 , and the pairs are i.i.d. Hence d i = G 1 ( τ ~ i ) − G 1 ( τ i ) ∈ [ − 1 , 1 ] d_i=G_1(\tilde{\tau}_i)-G_1(\tau_i)\in[-1,1] d i = G 1 ( τ ~ i ) − G 1 ( τ i ) ∈ [ − 1 , 1 ] are i.i.d. with:
E [ d i ] = E s 1 ∼ d 0 [ E π ~ [ G 1 ∣ s 1 ] − E π [ G 1 ∣ s 1 ] ] = J ( π ~ ) − J ( π ) = g ( 1 − ν ) . \mathbb{E}[d_i]=\mathbb{E}_{s_1\sim d_0}\left[\mathbb{E}_{\tilde{\pi}}[G_1\mid s_1]-\mathbb{E}_{\pi}[G_1\mid s_1]\right]=J(\tilde{\pi})-J(\pi)=g(1-\nu). E [ d i ] = E s 1 ∼ d 0 [ E π ~ [ G 1 ∣ s 1 ] − E π [ G 1 ∣ s 1 ] ] = J ( π ~ ) − J ( π ) = g ( 1 − ν ) .
Let ι = log ( 4 / δ ) \iota=\log(4/\delta) ι = log ( 4/ δ ) and let ξ ( d ) \xi(d) ξ ( d ) be (12) with B = 1 B=1 B = 1 , n ′ n' n ′ samples, and ι \iota ι in place of log ( 8 ∣ Λ ∣ / δ ) \log(8|\Lambda|/\delta) log ( 8∣Λ∣/ δ ) . Lemma E.4 applied to d d d and − d -d − d with δ ′ = 2 e − ι \delta'=2e^{-\iota} δ ′ = 2 e − ι gives ∣ d ˉ − E [ d ] ∣ ≤ ξ ( d ) |\bar d-\mathbb{E}[d]|\leq\xi(d) ∣ d ˉ − E [ d ] ∣ ≤ ξ ( d ) except with probability 4 e − ι = δ 4e^{-\iota}=\delta 4 e − ι = δ . On this event, L π ~ = d ˉ − ξ ( d ) ≤ E [ d ] L_{\tilde{\pi}}=\bar d-\xi(d)\leq\mathbb{E}[d] L π ~ = d ˉ − ξ ( d ) ≤ E [ d ] , and, since n ′ − 1 > 256 ι / ( g ( 1 − ν ) ) 2 ≥ 256 ι > ι n'-1>256\iota/(g(1-\nu))^2\geq256\iota>\iota n ′ − 1 > 256 ι / ( g ( 1 − ν ) ) 2 ≥ 256 ι > ι , (44) with B = 1 B=1 B = 1 gives:
L π ~ ≥ E [ d ] − 2 ξ ( d ) ≥ g ( 1 − ν ) − 16 ι n ′ − 1 > 0. L_{\tilde{\pi}}\geq\mathbb{E}[d]-2\xi(d)\geq g(1-\nu)-16\sqrt{\frac{\iota}{n'-1}}>0. L π ~ ≥ E [ d ] − 2 ξ ( d ) ≥ g ( 1 − ν ) − 16 n ′ − 1 ι > 0.
As ν → 0 \nu\to0 ν → 0 , this threshold stays bounded, while the bound in (i) grows as 1 / ν 1/\nu 1/ ν .
A leap is a policy π ~ \tilde{\pi} π ~ that need not be close to π \pi π . The set Π ~ \widetilde{\Pi} Π of leaps is fixed before τ 1 , … , τ n ∼ π \tau_1,\ldots,\tau_n\sim\pi τ 1 , … , τ n ∼ π are drawn. For a leap π ~ \tilde{\pi} π ~ and i ≤ n ′ ≤ n i\leq n'\leq n i ≤ n ′ ≤ n , τ ~ i ∼ π ~ \tilde{\tau}_i\sim\tilde{\pi} τ ~ i ∼ π ~ starts at the initial state of τ i \tau_i τ i , and the certificate of π ~ \tilde{\pi} π ~ applies (12) to the paired returns:
L π ~ = d ˉ − ξ ( d ) , d i = G 1 ( τ ~ i ) − G 1 ( τ i ) , (49) L_{\tilde{\pi}}=\bar d-\xi(d),\qquad d_i=G_1(\tilde{\tau}_i)-G_1(\tau_i), \tag*{(49)} L π ~ = d ˉ − ξ ( d ) , d i = G 1 ( τ ~ i ) − G 1 ( τ i ) , (49)
with d ˉ \bar d d ˉ and ξ ( d ) \xi(d) ξ ( d ) over i ≤ n ′ i\leq n' i ≤ n ′ and B = 2 G ˉ ≥ ∣ d i ∣ B=2\bar G\geq|d_i| B = 2 G ˉ ≥ ∣ d i ∣ . Plans are chosen by successive halving (Karnin et al., [16 ] ) on separate trajectories. In practice, a leap plays π \pi π with small probability, so that bounded steps from it can still reach every action of π \pi π .
Corollary F.2 (Certified selection with leaps). Theorem 3.7 holds when the leaps are added to the candidates with the certificates (47), 8 ∣ Λ ∣ 8|\Lambda| 8∣Λ∣ is replaced by 4 ( 2 ∣ Λ ∣ + ∣ Π ~ ∣ ) 4(2|\Lambda|+|\widetilde{\Pi}|) 4 ( 2∣Λ∣ + ∣ Π ∣ ) in (12) and in item (iii), and c π ~ = ξ ( d ) c_{\tilde{\pi}}=\xi(d) c π ~ = ξ ( d ) in item (ii).
Proof. Let ι = log ( 4 ( 2 ∣ Λ ∣ + ∣ Π ~ ∣ ) / δ ) \iota=\log(4(2|\Lambda|+|\widetilde{\Pi}|)/\delta) ι = log ( 4 ( 2∣Λ∣ + ∣ Π ∣ ) / δ ) . The leaps are fixed before the validation data, and each d d d is i.i.d. over i ≤ n ′ i\leq n' i ≤ n ′ with E [ d ] = J ( π ~ ) − J ( π ) \mathbb{E}[d]=J(\tilde{\pi})-J(\pi) E [ d ] = J ( π ~ ) − J ( π ) , as in the proof of Proposition F.1 (iii). As in the proof of Theorem 3.7, each of the 2 ∣ Λ ∣ + ∣ Π ~ ∣ 2|\Lambda|+|\widetilde{\Pi}| 2∣Λ∣ + ∣ Π ∣ statistics violates its two-sided interval with probability at most 4 e − ι 4e^{-\iota} 4 e − ι , so all intervals hold except with probability:
( 2 ∣ Λ ∣ + ∣ Π ~ ∣ ) ⋅ 4 e − ι = δ . (2|\Lambda|+|\widetilde{\Pi}|)\cdot4e^{-\iota}=\delta. ( 2∣Λ∣ + ∣ Π ∣ ) ⋅ 4 e − ι = δ .
On this event, 0 ≤ ( J ( π ~ ) − J ( π ) ) − L π ~ ≤ 2 ξ ( d ) 0\leq(J(\tilde{\pi})-J(\pi))-L_{\tilde{\pi}}\leq2\xi(d) 0 ≤ ( J ( π ~ ) − J ( π )) − L π ~ ≤ 2 ξ ( d ) for every leap, which is (40) with d d d in place of Z Z Z . The proof of Theorem 3.7 then applies with the leaps among the candidates.
Controlled Experiments
Each controlled task shares one probability q q q over N N N decisions with frozen binary workers, and the reward arrives at the end: Complementarity (N = 2 N=2 N = 2 , p = 1 / 3 p=1/3 p = 1/3 , reward 3 x 0 x 1 − x 0 − x 1 3x_0x_1-x_0-x_1 3 x 0 x 1 − x 0 − x 1 ), Redundancy (N = 2 N=2 N = 2 , p = 0.2 p=0.2 p = 0.2 , reward − x 0 x 1 + 0.3 ( x 0 + x 1 ) -x_0x_1+0.3(x_0+x_1) − x 0 x 1 + 0.3 ( x 0 + x 1 ) ) and Evidence threshold (N = 6 N=6 N = 6 , p = 0.3 p=0.3 p = 0.3 , reward 1 { ∑ t x t ≥ 3 } − 0.05 ∑ t x t 1\{\sum_t x_t \ge3\}-0.05\sum_t x_t 1 { ∑ t x t ≥ 3 } − 0.05 ∑ t x t ). A candidate with step s ∈ [ − 1 , 1 ] s \in[-1,1] s ∈ [ − 1 , 1 ] plays q = p + s p ( 1 − p ) q=p+s p(1-p) q = p + s p ( 1 − p ) , which is (7) with η = ∣ s ∣ \eta=|s| η = ∣ s ∣ , and all 2 N 2^N 2 N trajectories are enumerated, with 500 repetitions at δ = 0.05 \delta=0.05 δ = 0.05 . On Complementarity the linear term vanishes at p = 1 3 p=\frac{1}{3} p = 3 1 and a step gains 4 s 2 / 27 4s^2/27 4 s 2 /27 , so the first-order score keeps the reference while the second-order score reaches 0.148. On Redundancy a step gains 0.032 s − 0.0256 s 2 0.032s-0.0256s^2 0.032 s − 0.0256 s 2 : the first-order score takes s = 1 s=1 s = 1 and gains 0.0064, whereas the second-order score takes s = 1 2 s=\frac{1}{2} s = 2 1 on the grid and gains 0.0096.
On Evidence threshold the truncation bound ϵ = 1.728 \epsilon=1.728 ϵ = 1.728 exceeds the second-order gain 0.423 at s = 1 s=1 s = 1 , so the order-two certificate rejects that step at every sample size and accepts only a smaller one (gain 0.087); the importance-sampling certificate accepts the oracle step from n = 4096 n=4096 n = 4096 (gain 0.356). Across the 6,000 validation runs, no candidate with negative gain was accepted.
Environments, Roles and Protocol
Benchmarks. Each problem is stored as a target unit and a question in AFlow’s format [49 ] . Its long input shuffles the target unit among D D D units of other problems of the same split, using a generator seeded by a hash of the problem text, so that the document is a fixed function of the problem. Units stay whole and numbered, which is what the evidence finder retrieves. The setting follows GSM-IC [32 ] , carried to document length. The baselines read the whole document, and the team reaches it only through the evidence finder or the generalist; both receive the same question text, including the opening words of the target unit.
Table 2. The five environments. Prompt tokens are the mean over test of a single call that reads the whole input.
Training problems come from the official training splits and test problems from AFlow’s test splits. GSM8K uses the first 2,000 official training problems in SHA-256 order and 800 of AFlow’s 1,055 test problems (790 with a statement before the question). MATH uses 1,131 official level 4 to 5 problems of MaAS’s four subjects [48 ] and 336 test problems (220 with at least two sentences). Because AFlow’s DROP training split has only 200 questions, IAPO trains on 1,000 questions of the official split whose passages are disjoint from AFlow’s splits. MBPP keeps the 333 test problems whose first assertion names the function. HumanEval has no training split, so IAPO trains on 577 problems, AFlow’s 33 validation problems and 544 MBPP problems rewritten as HumanEval prompts [2 ] ; none of the test problems is in this training pool.
MATH answers are scored by a matcher that takes the last balanced \boxed{...} from prediction and reference and applies MaAS’s MATH equality (identical strings, numbers within 10 − 3 10^{-3} 1 0 − 3 , or a SymPy difference that simplifies to zero). It replaced a matcher that misread some answers, and every MATH arm was rerun with it.
Roles. Question answering and mathematics use eight roles and code adds Test (Table 3 ). Teams run in a pipeline regime with typed interfaces: a worker’s product reaches only its designated consumer at the next decision, and candidates are combined only when the selector is chosen. The evidence finder ranks the numbered units with BM25 [27 ] , and Test is MaAS’s public-test operator as released, with up to three repairs. Which roles are legal depends on the decision index and a few flags, and at the last decision only stopping is legal, which leaves 2,016 open-loop plans at five decisions and 432 at four. The orchestrator observes a MiniLM embedding of the task, the role signals, the previous action, the position and the share of the token budget spent, but never a score.
Table 3. The roles. “Right after X” means at the decision immediately following role X.
Protocol. Every call is a chat completion of Qwen2.5-7B-Instruct in bfloat16 on vLLM, one engine per H200, with a 24,576-token context window, at most 4,096 completion tokens and temperature 1, and both arms run through one metered environment that follows MaAS’s protocol. Tokens are the prompt and completion tokens of every request, retries included. IAPO’s trajectories are capped at 32,768 tokens and the baselines’ at 131,072 unless Appendix H says otherwise. A trajectory that exhausts its budget or fails with known usage scores 0 at its paid cost, as in MaAS’s evaluator; this happened only on MBPP among IAPO’s test runs (at most 3 of 1,332 episodes per run). The reward is s − λ N / 32,768 s - \lambda N/32{,}768 s − λ N /32 , 768 for score s s s and N N N tokens, with λ = 0.076 \lambda= 0.076 λ = 0.076 (config A, IAPO), a value fixed in an early pilot so that one extra revision step of about 1,500 tokens breaks even, or λ = 1 \lambda= 1 λ = 1 (CW1, IAPO-Lite).
Baselines. Table 4 lists how each system was run.
Table 4. How each baseline was run. Budget is the trajectory cap in tokens; “Rep.” is attempts per test problem.
Each system runs as released, adapted only where the shared protocol or the 7B executor forces it, and is then frozen. Systems that search or train read only the benchmark’s training split (MBPP’s for HumanEval), and their search cost is not counted in tokens per problem. On DROP the baselines’ training file, AFlow’s validation split, shares 120 of its 161 passages with the test split. For LLM-Blender both the fused output and the top-ranked candidate are kept and the higher is reported. MaAS’s solving prompt for MATH reaches the model with \boxed as a backspace and “oxed”, so prompting baselines on MATH run with the backslash restored. The strongest baseline of each environment received a second seed where one was run (Debate on GSM8K and DROP, DyLAN on MATH and MBPP, LLM-Blender on DROP and HumanEval, the benchmarks where it was the strongest).
IAPO Settings, Certificates and Statistics
All 42 IAPO runs (21 pairs of IAPO and its first-order variant) share the settings of Table 5 within each environment family, apart from the draw salt and the interaction switch.
Table 5. IAPO settings of every reported run.
Certificates
With three steps and three leaps there are M = 9 M = 9 M = 9 statistics per run, each at level δ 1 / 9 ≈ 2.8 × 10 − 3 \delta_1/9 \approx2.8 \times10^{-3} δ 1 /9 ≈ 2.8 × 1 0 − 3 for δ 1 = 0.025 \delta_1 = 0.025 δ 1 = 0.025 . A step is certified by L n = max { L ( Z 2 ) − ϵ ( η ) , L ( Z 1 ) } L_n = \max\{L(Z_2) - \epsilon(\eta), L(Z_1)\} L n = max { L ( Z 2 ) − ϵ ( η ) , L ( Z 1 )} with the betting bound of Appendix E.7, found by 48 bisection steps and rounded down; the interval widths and ϵ ( η ) \epsilon(\eta) ϵ ( η ) are in Table 6 . A leap is certified by the betting bound on the paired differences d i = G 1 ( τ ~ i ) − G 1 ( τ i ) d_i = G_1(\tilde{\tau}_i) - G_1(\tau_i) d i = G 1 ( τ ~ i ) − G 1 ( τ i ) for i ≤ 4 , 000 i \le4,000 i ≤ 4 , 000 , where τ ~ i \tilde{\tau}_i τ ~ i starts from the problem of validation draw τ i \tau_i τ i with its own seeds. The deployed policy is the reference, step or leap with the largest certificate, and a candidate with a positive certificate was deployed in all 42 runs. One MBPP run spends about 2.9 × 10 7 2.9 \times10^{7} 2.9 × 1 0 7 tokens on proposal trajectories, 3.6 × 10 7 3.6 \times10^{7} 3.6 × 1 0 7 on validation and 1.07 × 10 8 1.07 \times10^{8} 1.07 × 1 0 8 on the three leaps, about 1.7 × 10 8 1.7 \times10^{8} 1.7 × 1 0 8 in total, which Table 1 does not include.
Table 6. Known interval lengths b b b of Z 2 Z_2 Z 2 and Z I S Z_{\mathrm{IS}} Z IS (with the per-problem baseline) and the remainder ϵ ( η ) \epsilon(\eta) ϵ ( η ) , per family and per cost weight of the objective.
Per-problem baseline. A constant baseline removes the mean return but not the spread between problems, so in validation trajectory i i i we use b i ∈ [ g − , g + ] b_i \in[g_- , g_+] b i ∈ [ g − , g + ] , the mean return of the earlier validation trajectories on the same problem, or b b b if there is none. Because b i b_i b i depends only on trajectories 1 , … , i − 1 1,\ldots,i-1 1 , … , i − 1 and the initial state of trajectory i i i , and E π [ ∏ j ≤ t ρ j − 1 ∣ s 1 ] = E π [ ρ t − 1 − 1 ∣ τ t − 1 , s t ] = 0 \mathbb{E}_{\pi}[\prod_{j\leq t}\rho_j-1 \mid s_1] = \mathbb{E}_{\pi}[\rho_{t-1}-1 \mid\tau_{t-1},s_t] = 0 E π [ ∏ j ≤ t ρ j − 1 ∣ s 1 ] = E π [ ρ t − 1 − 1 ∣ τ t − 1 , s t ] = 0 , the b i b_i b i terms have conditional mean zero and each statistic keeps its mean given the earlier trajectories. The payoff then lies in [ g − − g + , g + − g − ] [g_- - g_+, g_+ - g_-] [ g − − g + , g + − g − ] , which widens the intervals of Lemma E.3. The values are martingale differences rather than i.i.d.; the proof of Proposition E.6 needs only that the conditional mean is at most z z z , so it holds with [ z − , z + ] [z_- , z_+] [ z − , z + ] in place of [ − B , B ] [-B,B] [ − B , B ] , whereas the empirical Bernstein bound would need independence. On competition mathematics with a seven-operator workflow this baseline lowers the standard deviation of Z I S Z_{\mathrm{IS}} Z IS by about a quarter at every step of the grid.
Leaps. Every declared plan is scored by the proposal statistic at its leap policy, the plan mixed with ϵ = 0.001 \epsilon= 0.001 ϵ = 0.001 of the reference, as the mean over the 6,400 proposal trajectories minus two standard errors. On question answering and mathematics the 50 top-ranked plans each run on 1,000 fresh validation draws and the three with the largest mean net return become the leaps; on code the three top-ranked plans become the leaps directly. The certificate never reads the screening draws. The expansion serves here as a proposal heuristic for distant plans, so the remainder bound does not apply to the leap ratios, while their certificates stay valid because they use paired draws of their own.
Matched budget. The frozen deployed policy is rerun on the test split with a fresh salt and 10 to 200 repeats per problem. On answer tasks a vote takes the most common non-empty answer key over the first K K K episodes, with ties to the earliest; on code the episodes split into groups of K K K , each reduced to one program by the MaAS’s HumanEval-template selector with its tokens added. K K K is the largest count whose tokens per vote fit the strongest baseline’s tokens per episode, for every listed seed. The budgets are 28.6k (GSM8K), 53.8k (MATH), 57.1k (DROP) and 26.5k tokens (HumanEval); on MBPP it is 35.4k, below DyLAN’s 121.9k.
Statistics. Comparisons are paired by problem on the full test split: per-problem differences of arm means give d ˉ \bar d d ˉ , S E = s d / n \mathrm{SE}=s_d/\sqrt{n} SE = s d / n and z = d ˉ / S E z=\bar d/\mathrm{SE} z = d ˉ / SE , called significant when z > 2 z>2 z > 2 . The ablation is scored on net return s − λ s-\lambda s − λ tokens/32,768 with each configuration’s own λ \lambda λ . An arm’s score is averaged over its runs per problem before the differences are taken, and no run of a reported configuration is left out.
Full Results
Table 7 gives one episode of each deployed policy and Table 8 the paired comparisons. Against every baseline, IAPO’s single-episode accuracy on MBPP is higher by at least 8.8 points (best baseline DyLAN, 72.3%), and on HumanEval, apart from MaAS (83.8%), by at least 8.1 points (next best Debate, 75.6%).
Table 7: Single episode of the deployed policy, mean over runs: accuracy (%) and tokens per problem (thousands) of IAPO and of its first-order variant (FO).
Table 7.
Table 8 : Paired comparisons, with z z z in parentheses and bold for z > 2 z > 2 z > 2 . (a) Matched budget against the strongest baseline (mean over runs): accuracy (%) and tokens per problem, and the difference in points. ‡Against DyLAN, at the smaller budget of Appendix I. (b) IAPO minus its first-order variant on net return and accuracy.
(a) Matched budget
(b) Interaction ablation
Table 8.
Figure 4. Every system on each benchmark. Top: accuracy, with the strongest baseline’s accuracy dashed; bottom: tokens per episode (log scale). IAPO and IAPO-Lite are shaded; as in Figure 2 , their accuracy is that of the voted plan within the strongest baseline’s budget.
Figure 5. Left: gain over the reference of the generalist alone (the choice of the first-order score), the analyst alone, and the evidence finder followed by the analyst. Right: runs that deploy the pair, with the interaction term and with first order only, for each benchmark.
Figure 6. Plan deployed by each IAPO run, with the interaction term (left) and with first order only (right): roles in call order, with accuracy and tokens of a single episode. Each row is a separately trained run. evid.: evidence finder; general.: second answerer reading the whole document; calc.: calculator; select: selector.