Introduction
Policy optimization algorithms are the reinforcement learning (RL) methods of choice in practice, with successes spanning dexterous robotic manipulation and control ([47 , 84 , 66 , 87 ] ), superhuman performance in strategic games ([97 , 108 , 80 , 82 , 4 ] ), and, most notably, frontier reasoning in language models (LMs) ([81 , 79 , 33 , 50 ] ). Across these applications, post-training has emerged as a successful paradigm in which policy optimization uses reward feedback to finetune an already capable base model , developed through large-scale supervised pretraining. The base model provides a foundation for learning complex tasks, while reward feedback offers a way to improve beyond the limits of available demonstrations.
Despite these successes, principled guidance for policy optimization methodology remains limited [44 , 2 ] , while the range of interventions continues to grow [96 , 116 , 64 , 120 , 75 , 125 ] . Tuning any given method requires extensive trial and error [42 , 23 , 65 ] , making it difficult to evaluate the effects of specific interventions [81 , 48 , 99 ] . Even training recipes for leading open models show little consensus on core algorithmic choices, such as which objective to optimize [33 , 104 , 125 , 49 ] . Principled insights are therefore increasingly important for focusing limited resources on promising interventions and hyperparameter regimes.
Theory has, in the past, provided key design principles for policy optimization methods. TRPO [92 ] , followed by PPO [93 ] , directly drew from the proximal updates and policy improvement guarantees established for early theoretical algorithms [45 , 46 ] . In post-training, however, the central goal is to learn a performant policy while making the most of the base model, a goal that existing theory has left largely unaddressed. Most guarantees either require a uniform initial policy, sidestepping the role of the base model altogether [70 , 72 , 55 ] , or cannot preclude poor performance even with a high-quality base model [1 , 72 , 73 , 109 ] . They therefore offer little guidance for understanding and designing policy optimization methods in post-training.
Motivated by this gap, we revisit the theory of policy optimization in post-training, and ask:
Which policy optimization algorithms effectively leverage the base model for learning?
To capture this goal, we will seek algorithms whose performance is governed by the quality of the base model. This is especially important given the substantial resources invested in developing pretrained base models with strong post-training performance [32 , 20 ] . We formalize our question and summarize our contributions below.
Contributions
We ask which policy optimization algorithms learn near-optimal policies using total samples and gradient steps governed by the base model’s coverage over optimal responses, a guarantee that we call coverage-preserving (Section 2.2). This captures the basic post-training goal of leveraging the base model for learning and is achieved by methods such as Best-of-N sampling (Section 1.3), but has not been established in policy optimization. By designing algorithms that achieve this guarantee, we recover key interventions in modern post-training and help explain their practical efficacy.
Coverage collapse with a fixed reference policy or no regularization (Section 2.1). When initialized from a base model with good (constant) coverage, REINFORCE may require exponentially many gradient steps to converge, even with KL regularization to the base model. The cause is coverage collapse , in which coverage over optimal responses deteriorates even as reward improves, causing the gradient signal to vanish (Figure 2). This failure is intrinsic to the RL objective—it occurs with exact gradients and without parameter sharing—motivating interventions that preserve coverage for efficient learning.
Approximate Policy Mirror Descent (APMD) preserves coverage (Section 3.1). We show that algorithms built around a single intervention—resetting the reference policy—are coverage-preserving. We formalize this in APMD, a meta-algorithm that approximates mirror descent by iteratively setting the reference to the most recent policy, then running an inner optimization loop for a regularized RL subproblem. APMD algorithms can heavily regularize the learner to the reference, making each subproblem easy to solve. While the learner improves only locally, over successive resets it reaches global optimality (Figure 1).
Applying APMD: Coverage-preserving algorithms. APMD is a template for algorithm design and analysis, and can be combined with existing methods or directly applied to analyze them. We give end-to-end convergence guarantees for softmax linear policies, a model of learning with frozen pretrained features.
(a) PPO, GRPO, and variants (Sections 3.2 and 3.3). We analyze variants of PPO and GRPO with reference resets, as in ProRL [62 ] and DeepSeek [96 , 33 ] . In PPO-NoClip (Algorithm 1), which does not clip the importance weights, regularization alone enables efficient learning by controlling the policy ratios (Corollary 3.1). We then add clipping to obtain PPO-Lite (Algorithm 2). Our analysis shows that clipping complements regularization by limiting the statistical cost of large importance weights, rather than directly improving optimization (Corollary 3.2 ).
(b) Squared-loss algorithms (§ 4 ). These algorithms fit log policy ratios to rewards using squared-loss objectives. We first show that Kimi-style algorithms (Kimi Team et al. [50 ] ), which shift the rewards by an estimated baseline function, are coverage-preserving for two common baseline choices, the mean reward in KimiPO (Corollary 4.1) and the normalization function in KimiPO-Norm (Corollary K.1). Lastly, we show that Rebel-OnPolicy, an on-policy variant of Rebel that eliminates the baseline using paired responses, is also coverage-preserving (Corollary 4.2 ).
Implications of our theory (§ 5 ). Our results show that optimization in a local neighborhood of the reference policy makes improvement tractable, whether regularization is explicitly used or implicit. This role of the reference in anchoring optimization is distinct from its use in importance weighting or batch reuse. Compared to fixed-reference (or unregularized) policy optimization, algorithms that regularize the learner to a periodically updated reference converge using substantially fewer gradient steps and even fewer rounds of batch generation.
Background: Policy Optimization in Post-Training
We study policy optimization in a contextual bandit setting motivated by RL post-training, where reward feedback is used to finetune the weights of a pretrained base model. A language model (LM) is a policy π θ : X → Δ ( Y ) \pi_\theta: \mathcal{X} \to\Delta(\mathcal{Y}) π θ : X → Δ ( Y ) that maps a prompt x ∈ X x \in\mathcal{X} x ∈ X to a distribution over complete responses, denoted y y y . Given a prompt distribution μ ∈ Δ ( X ) \mu\in\Delta(\mathcal{X}) μ ∈ Δ ( X ) and bounded reward function r : X × Y → [ 0 , R max ] r : \mathcal{X} \times\mathcal{Y} \to[0,R_{\max}] r : X × Y → [ 0 , R m a x ] , the goal of policy optimization is to find
arg max π θ ∈ Π Θ J ( π θ ) , where J ( π ) : = E x ∼ μ , y ∼ π ( ⋅ ∣ x ) [ r ( x , y ) ] . \underset{\pi_\theta\in\Pi_\Theta}{\operatorname{arg\,max}}\ J(\pi_\theta), \qquad\text{where}\quad J(\pi) := \mathbb{E}_{x\sim\mu,y\sim\pi(\cdot|x)}[r(x,y)]. π θ ∈ Π Θ arg max J ( π θ ) , where J ( π ) := E x ∼ μ , y ∼ π ( ⋅ ∣ x ) [ r ( x , y )] .
The objective J ( π ) J(\pi) J ( π ) is the expected reward, and Π Θ = { π θ : θ ∈ Θ } \Pi_\Theta= \{\pi_\theta: \theta\in\Theta\} Π Θ = { π θ : θ ∈ Θ } denotes the policy class induced by a fixed architecture with weights in Θ ⊆ R d \Theta\subseteq\mathbb{R}^d Θ ⊆ R d . Reward feedback is assigned to a complete response y y y , the sequence of tokens generated for prompt x x x . This abstraction captures RL for one-shot reasoning.
In post-training, policy optimization starts from a pretrained base model π 0 = π b a s e = π θ b a s e ∈ Π Θ \pi_0 = \pi_{\mathrm{base}} = \pi_{\theta_{\mathrm{base}}} \in\Pi_{\Theta} π 0 = π base = π θ base ∈ Π Θ and produces the policy iterates { π t = π θ t } t = 0 T \{\pi_t = \pi_{\theta_t}\}_{t=0}^{T} { π t = π θ t } t = 0 T over T T T iterations. Objectives and update rules vary across algorithms, but we focus on first-order methods that update π t \pi_t π t to π t + 1 \pi_{t+1} π t + 1 using gradients of reward-based objectives. Regardless of the algorithm, we evaluate the learned policy by the suboptimality of its expected reward. For any ε ∈ ( 0 , R max ] \varepsilon\in(0, R_{\max}] ε ∈ ( 0 , R m a x ] , we say that a policy π \pi π is ε \varepsilon ε -optimal if J ⋆ − J ( π ) ≤ ε J^\star- J(\pi) \le\varepsilon J ⋆ − J ( π ) ≤ ε , where J ⋆ : = max π : X → Δ ( Y ) J ( π ) J^\star:= \max_{\pi:\mathcal{X}\to\Delta(\mathcal{Y})} J(\pi) J ⋆ := max π : X → Δ ( Y ) J ( π ) is the maximum reward achievable by any conditional distribution.
Example: REINFORCE. REINFORCE [111 , 102 ] is a foundational policy optimization algorithm that performs stochastic gradient ascent on the expected reward J ( π θ ) J(\pi_\theta) J ( π θ ) . For a fixed learning rate α > 0 \alpha> 0 α > 0 , the parameter update with exact gradients is θ t + 1 = θ t + α ∇ J ( π t ) \theta_{t+1} = \theta_t + \alpha\nabla J(\pi_t) θ t + 1 = θ t + α ∇ J ( π t ) , where ∇ \nabla ∇ denotes differentiation with respect to the policy parameters and ∇ J ( π t ) \nabla J(\pi_t) ∇ J ( π t ) is evaluated at θ = θ t \theta= \theta_t θ = θ t ,
∇ J ( π t ) = E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ∇ log π t ( y ∣ x ) r ( x , y ) ] . \nabla J(\pi_t) = \mathbb{E}_{x\sim\mu,y\sim\pi_t(\cdot\mid x)}[\nabla\log\pi_t(y\mid x)r(x,y)]. ∇ J ( π t ) = E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ∇ log π t ( y ∣ x ) r ( x , y )] .
In practice, REINFORCE estimates ∇ J ( π t ) \nabla J(\pi_t) ∇ J ( π t ) using responses sampled from π t \pi_t π t . Modern policy gradient methods build on this update with additions such as KL regularization to the base model [81 , 96 ] , estimated advantage functions, and clipped importance weights for batch reuse [93 , 96 ] . In § 2.1 , we study REINFORCE to identify optimization challenges specific to RL. These challenges motivate from first principles the design of modern post-training methods such as PPO and GRPO (§ 3 ).
Background: Coverage and the Role of the Base Model
Policy optimization in post-training begins from a base model, and we measure the quality of the base model by its response coverage , which quantifies how often it generates optimal responses.
Definition 1.1 (Response coverage). For any prompt x ∈ X x \in\mathcal{X} x ∈ X , let Y x ⋆ : = { y ∈ Y : r ( x , y ) = max y ′ r ( x , y ′ ) } \mathcal{Y}^{\star}_x := \{y \in\mathcal{Y}: r(x,y) = \max_{y'} r(x,y')\} Y x ⋆ := { y ∈ Y : r ( x , y ) = max y ′ r ( x , y ′ )} denote the set of optimal responses. Then the response coverage of a policy π \pi π is measured by
C π : = E x ∼ μ [ 1 π ( Y x ⋆ ∣ x ) ] . C_\pi:= \mathbb{E}_{x\sim\mu}\left[\frac{1}{\pi(\mathcal{Y}^{\star}_x\mid x)}\right]. C π := E x ∼ μ [ π ( Y x ⋆ ∣ x ) 1 ] .
For the base model, we use C b a s e ≡ C π b a s e C_{\mathrm{base}} \equiv C_{\pi_{\mathrm{base}}} C base ≡ C π base .
For a policy π \pi π and prompt x x x , the expected number of generations needed to observe an optimal response is π ( Y x ⋆ ∣ x ) − 1 \pi(\mathcal{Y}^{\star}_x\mid x)^{-1} π ( Y x ⋆ ∣ x ) − 1 , and C π C_\pi C π averages this cost over prompts. Smaller values of C π C_\pi C π indicate better response coverage, meaning that fewer generations are needed to find an optimal response. A central premise of post-training is that pretraining imbues the base model with favorable coverage (a moderate C b a s e C_{\mathrm{base}} C base ). This has helped explain the success of inference-time sampling [9 , 40 ] and supervised finetuning on filtered generations [98 , 56 , 39 ] .
Coverage and Best-of-N. The base model’s response coverage governs the performance of Best-of-N sampling, a widely used inference-time method for improving response quality without training. Given a prompt x x x , Best-of-N draws N N N independent responses from π b a s e ( ⋅ ∣ x ) \pi_{\mathrm{base}}(\cdot\mid x) π base ( ⋅ ∣ x ) and returns the one with the highest reward r ( x , y ) r(x,y) r ( x , y ) . It fails to return an optimal response only when none of the N N N candidates is optimal, an event with probability E x [ ( 1 − π b a s e ( Y x ⋆ ∣ x ) ) N ] ≤ C b a s e / N \mathbb{E}_x[(1-\pi_{\mathrm{base}}(\mathcal{Y}^{\star}_x\mid x))^N] \le C_{\mathrm{base}}/N E x [( 1 − π base ( Y x ⋆ ∣ x ) ) N ] ≤ C base / N . Thus, by using N = Θ ~ ( C b a s e / ε ) N = \widetilde{\Theta}(C_{\mathrm{base}}/\varepsilon) N = Θ ( C base / ε ) responses per prompt, Best-of-N returns an optimal response with probability at least 1 − ε 1-\varepsilon 1 − ε [40 , 16 ] .
Coverage and policy optimization. The Best-of-N guarantee above suggests that policy optimization algorithms should learn near-optimal policies using total samples controlled by C b a s e C_{\mathrm{base}} C base . However, unlike Best-of-N, which always samples from π b a s e \pi_{\mathrm{base}} π base , in policy optimization responses are sampled from the learner, whose coverage evolves during training. For example, with binary rewards r ( x , y ) ∈ { 0 , 1 } r(x,y) \in\{0,1\} r ( x , y ) ∈ { 0 , 1 } , the estimated REINFORCE gradient ∇ J ( π t ) = E π t [ ∇ log π t ( y ∣ x ) r ( x , y ) ] \nabla J(\pi_t) = \mathbb{E}_{\pi_t}[\nabla\log\pi_t(y\mid x)r(x,y)] ∇ J ( π t ) = E π t [ ∇ log π t ( y ∣ x ) r ( x , y )] is zero unless the batch contains a correct response. It is therefore the coverage of π t \pi_t π t that governs convergence in policy optimization. While the initial policy inherits the base model’s coverage, it is unclear whether this coverage, which supports the learning signal, persists during training.
Motivation: The Coverage Collapse Phenomenon
We therefore ask whether policy optimization algorithms can learn efficiently when initialized from a high-quality base model, the basic goal of post-training. We first show that REINFORCE can suffer coverage collapse , causing exponentially slow convergence despite favorable initial coverage (§ 2.1 ). This failure motivates our central question in Section 2.2, where we ask which policy optimization methods learn near-optimal policies with total samples and gradient steps controlled by the base model’s coverage, a guarantee we call coverage-preserving.
Lower Bound: Coverage Collapse in Naïve Policy Optimization Methods
We now investigate whether REINFORCE converges efficiently when initialized from a base model with good coverage. Throughout this section, we use exact rather than estimated gradients to isolate the effect of coverage on the dynamics of policy optimization. We also allow KL regularization to the base model, with regularization parameter η ∈ ( 0 , ∞ ] \eta\in(0,\infty] η ∈ ( 0 , ∞ ] , giving the objective
J η ( π θ ) : = J ( π θ ) − 1 η D K L ( π θ ∥ π b a s e ) , where D K L ( π ∥ π b a s e ) = E x ∼ μ , y ∼ π ( ⋅ ∣ x ) [ log π ( y ∣ x ) π b a s e ( y ∣ x ) ] . (1) J_\eta(\pi_\theta) := J(\pi_\theta) - \frac{1}{\eta}D_{\mathrm{KL}}(\pi_\theta\mathbin\Vert\pi_{\mathrm{base}}), \qquad\text{where}\quad D_{\mathrm{KL}}(\pi\mathbin\Vert\pi_{\mathrm{base}}) = \mathbb{E}_{x\sim\mu,\,y\sim\pi(\cdot\mid x)}\left[\log\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right]. \tag*{(1)} J η ( π θ ) := J ( π θ ) − η 1 D KL ( π θ ∥ π base ) , where D KL ( π ∥ π base ) = E x ∼ μ , y ∼ π ( ⋅ ∣ x ) [ log π base ( y ∣ x ) π ( y ∣ x ) ] . (1)
where η = ∞ \eta= \infty η = ∞ recovers the unregularized objective J J J . Starting from π 0 = π b a s e \pi_0 = \pi_{\mathrm{base}} π 0 = π base , REINFORCE performs gradient ascent with a fixed learning rate α > 0 \alpha> 0 α > 0 ,
θ t + 1 = θ t + α ∇ J η ( π t ) , where ∇ J η ( π t ) = E π t [ ∇ log π t ( y ∣ x ) ( r ( x , y ) − 1 η log π t ( y ∣ x ) π b a s e ( y ∣ x ) ) ] . (2) \theta_{t+1} = \theta_t + \alpha\nabla J_\eta(\pi_t), \qquad\text{where}\quad\nabla J_\eta(\pi_t) = \mathbb{E}_{\pi_t}\left[\nabla\log\pi_t(y\mid x)\left(r(x,y)-\frac{1}{\eta}\log\frac{\pi_t(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right)\right]. \tag*{(2)} θ t + 1 = θ t + α ∇ J η ( π t ) , where ∇ J η ( π t ) = E π t [ ∇ log π t ( y ∣ x ) ( r ( x , y ) − η 1 log π base ( y ∣ x ) π t ( y ∣ x ) ) ] . (2)
KL regularization is widely used in post-training to stabilize optimization and help preserve pre-existing behavior ([96 , 52 , 62 ] ). Because it discourages deviations from the base model, one might expect π t \pi_t π t to stay close to π b a s e \pi_{\mathrm{base}} π base in distribution and retain the base model’s coverage, especially since improving performance involves assigning greater probability to higher-reward responses.
However, Theorem 2.1 shows that the policy’s coverage can deteriorate until it is exponentially worse than the base model’s, causing convergence to be exponentially slow. Favorable coverage in the base model therefore does not guarantee efficient convergence of REINFORCE, with or without KL regularization.
Theorem 2.1 (Coverage collapse in REINFORCE). Fix any ε ∈ ( 0 , 1 64 ] \varepsilon\in(0,\frac{1}{64}] ε ∈ ( 0 , 64 1 ] . There exists a bandit instance with prompts X = { x } \mathcal{X} = \{x\} X = { x } , responses Y \mathcal{Y} Y , a reward function r r r with R max ≤ 1 R_{\max} \le1 R m a x ≤ 1 , and a base model π b a s e \pi_{\mathrm{base}} π base with C b a s e ≤ 6 C_{\mathrm{base}} \le6 C base ≤ 6 for which the following holds. Consider the tabular softmax policy class Π Θ = { π θ : π θ ( y ∣ x ) ∝ exp ( θ y ) , θ ∈ R ∣ Y ∣ } \Pi_\Theta= \{\pi_\theta: \pi_\theta(y\mid x) \propto\exp(\theta_y),\ \theta\in\mathbb{R}^{|\mathcal{Y}|}\} Π Θ = { π θ : π θ ( y ∣ x ) ∝ exp ( θ y ) , θ ∈ R ∣ Y ∣ } . Initialize π 0 = π b a s e \pi_0 = \pi_{\mathrm{base}} π 0 = π base and run the REINFORCE update in Eq. (2) with any regularization parameter η ∈ ( 0 , ∞ ] \eta\in(0,\infty] η ∈ ( 0 , ∞ ] and any learning rate 0 < α ≤ 1 / 5 0 < \alpha\le1/5 0 < α ≤ 1/5 . If the policy is ε \varepsilon ε -optimal from iteration T T T onward, then there exists 0 < t < T 0 < t < T 0 < t < T such that
T = Ω ( C π t α ε ) and C π t = exp ( Θ ( 1 ε ) ) . T = \Omega\left(\frac{C_{\pi_t}}{\alpha\sqrt{\varepsilon}}\right) \qquad\text{and}\qquad C_{\pi_t} = \exp\left(\Theta\left(\frac{1}{\sqrt{\varepsilon}}\right)\right). T = Ω ( α ε C π t ) and C π t = exp ( Θ ( ε 1 ) ) .
Theorem 2.1 shows that the convergence of REINFORCE depends on the policy’s coverage along the optimization path, measured by C π t C_{\pi_t} C π t , rather than just the base model’s. The problem is that, in the first phase of training, C π t C_{\pi_t} C π t grows from C b a s e ≤ 6 C_{\mathrm{base}} \le6 C base ≤ 6 to an exponentially large value—meaning that the optimal response has exponentially small probability—even though expected reward increases monotonically. This coverage collapse makes the gradient signal vanish, and REINFORCE requires exponentially many steps to regain coverage and converge. This can be seen in the prolonged reward plateau of Figure 2 a (yellow). Given the simplicity of our construction, we expect coverage collapse to persist under the complex optimization landscapes and stochastic noise encountered in practice.
Figure 2. Coverage collapse stalls optimization with a fixed reference policy. We compare REINFORCE (Eq. (2) ) and APMD using exact gradients on a modified instance from Theorem 2.1. APMD runs PPO-NoClip (Algorithm 1) with a reference reset every K = 100 K = 100 K = 100 inner steps. (a) In REINFORCE (black), coverage collapse (red) causes a prolonged reward plateau (yellow). The shaded regions are schematic. APMD (blue) converges faster as coverage keeps improving (smaller C π t \mathcal{C}_{\pi_t} C π t ). (b,c) The REINFORCE trajectory plotted on the optimization landscape, with surface height J η J_{\eta} J η and color indicating reward (b) or response coverage (c), over the 2D parameter plane defined in Appendix § 9 .§ 1 .
Coverage collapse: Nonconcavity in RL. The dependence of convergence on coverage along the optimization path results from the nonconcavity particular to RL (Figure 2 b). The objective J ( π θ ) J(\pi_\theta) J ( π θ ) is nonconcave for softmax policies, even without parameter sharing (as in Theorem 2.1). Correspondingly, convergence analyses for REINFORCE use Łojasiewicz-type inequalities of the form ([70 , 72 , 1 ] ),
J ⋆ − J ( π t ) ≤ c ⋅ ∥ π ⋆ π t ∥ ∞ ⋅ ∥ ∇ J ( π t ) ∥ . (3) J^\star- J(\pi_t) \le c \cdot\left\lVert\frac{\pi^\star}{\pi_t}\right\rVert_\infty\cdot\lVert\nabla J(\pi_t)\rVert. \tag*{(3)} J ⋆ − J ( π t ) ≤ c ⋅ π t π ⋆ ∞ ⋅ ∥ ∇ J ( π t )∥ . (3)
This bound shows how near-stationarity can still imply near-optimality, depending on a coefficient ∥ π ⋆ / π t ∥ ∞ \left\lVert\pi^\star/\pi_t\right\rVert_\infty ∥ π ⋆ / π t ∥ ∞ that measures π t \pi_t π t ’s coverage of an optimal policy π ⋆ \pi^\star π ⋆ . During coverage collapse this coefficient becomes exponentially large (Figure 2 c). Eq. (3) then permits π t \pi_t π t to be nearly stationary yet highly suboptimal—precisely the behavior exhibited by Theorem 2.1 and Figure 2 a (yellow).
Why regularization to π b a s e \pi_{\mathrm{base}} π base fails. Because KL regularization anchors the policy to the base model, it must be weak enough (1 η = O ( ε ) \frac{1}{\eta} = O(\varepsilon) η 1 = O ( ε ) ) for the policy to improve substantially and achieve J ⋆ − J ( π T ) ≤ ε J^{\star} - J(\pi_T) \le\varepsilon J ⋆ − J ( π T ) ≤ ε . This freedom also permits large deviations that reduce the probability of optimal responses, which is the mechanism behind the lower bound in Theorem 2.1. There, KL regularization exerts a significant restoring force only after coverage has degraded exponentially (see Figure 1 a).
Figure 1. Overview of APMD: Continued progress through reference resets. Shaded regions depict KL neighborhoods around the chosen reference policy. Each arrow marks the learner’s progress after a fixed number of gradient steps. (a) With weak regularization to the base model, coverage can collapse and cause optimization to stall. (b) Strong regularization controls coverage and keeps the learner close to the reference policy, but far from optimality (marked ☆). (c) APMD resets the reference to the most recent policy, reaching optimality through successive local improvements.
Remark 2.1 (Comparison to prior work). To our knowledge, Theorem 2.1 is the first result isolating the failure of REINFORCE from coverage collapse, given that we use tabular parameters and exact gradients. Related results either (i) show asymptotic failure without characterizing coverage ([59 ] ); (ii) have additional sources of hardness, such as stochastic noise ([17 , 71 , 5 , 54 ] ), parameter interference ([91 ] ), or multi-step dynamics ([57 ] ); or (iii) scale with the base model’s coverage, remaining mild when this coverage is constant ([1 , 86 , 69 ] ). In contrast, our lower bound is exponential in 1 / ε 1/\sqrt{\varepsilon} 1/ ε even when C b a s e \mathcal{C}_{\mathrm{base}} C base is constant.
Question: Which Policy Optimization Algorithms Are Coverage-Preserving?
In light of Theorem 2.1, we ask which policy optimization methods learn efficiently when initialized from a base model and avoid coverage collapse. Concretely, this means that they output near-optimal policies using N s a m p l e N_{\mathrm{sample}} N sample samples (prompt-response pairs) and N s t e p N_{\mathrm{step}} N step gradient steps, with both quantities controlled by the response coverage C b a s e \mathcal{C}_{\mathrm{base}} C base . We call algorithms with this guarantee coverage-preserving , as formalized below.
Which policy optimization algorithms are coverage-preserving?
For a given policy class Π Θ \Pi_{\Theta} Π Θ with complexity comp ( Π Θ ) \operatorname{comp}(\Pi_{\Theta}) comp ( Π Θ ) , an algorithm is coverage-preserving if, for every ε ∈ ( 0 , R max ] \varepsilon\in(0, R_{\max}] ε ∈ ( 0 , R m a x ] and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , it returns a policy π p o s t \pi_{\mathrm{post}} π post satisfying J ⋆ − J ( π p o s t ) ≤ ε J^{\star} - J(\pi_{\mathrm{post}}) \le\varepsilon J ⋆ − J ( π post ) ≤ ε with probability at least 1 − δ 1-\delta 1 − δ , using total gradient steps N s t e p ( ε , δ ) N_{\mathrm{step}}(\varepsilon,\delta) N step ( ε , δ ) and total samples N s a m p l e ( ε , δ ) N_{\mathrm{sample}}(\varepsilon,\delta) N sample ( ε , δ ) bounded as follows:
N s t e p ( ε , δ ) ≤ poly ( C b a s e , comp ( Π Θ ) , ε − 1 , log ( 1 / δ ) ) , N_{\mathrm{step}}(\varepsilon,\delta) \le\operatorname{poly}\left(\mathcal{C}_{\mathrm{base}},\operatorname{comp}(\Pi_{\Theta}),\varepsilon^{-1},\log(1/\delta)\right), N step ( ε , δ ) ≤ poly ( C base , comp ( Π Θ ) , ε − 1 , log ( 1/ δ ) ) ,
N s a m p l e ( ε , δ ) ≤ poly ( C b a s e , comp ( Π Θ ) , ε − 1 , log ( 1 / δ ) ) . (4) N_{\mathrm{sample}}(\varepsilon,\delta) \le\operatorname{poly}\left(\mathcal{C}_{\mathrm{base}},\operatorname{comp}(\Pi_{\Theta}),\varepsilon^{-1},\log(1/\delta)\right). \tag*{(4)} N sample ( ε , δ ) ≤ poly ( C base , comp ( Π Θ ) , ε − 1 , log ( 1/ δ ) ) . (4)
For coverage-preserving algorithms, the cost of learning is controlled by the base model’s coverage. The guarantee therefore formalizes the basic post-training goal of amplifying high-reward responses from the base model, sometimes called “sharpening”1 [39 , 124 , 119 ] . Analogous sample complexity guarantees already hold for Best-of-N N N sampling (§ 1.3 ) and SFT on reward-filtered samples from π b a s e \pi_{\mathrm{base}} π base [39 , 61 ] . Indeed, in the construction of Theorem 2.1, where C b a s e = O ( 1 ) C_{\mathrm{base}} = O(1) C base = O ( 1 ) , Best-of-N N N produces an optimal response using only O ( 1 ) O(1) O ( 1 ) generations, whereas REINFORCE requires exponentially many gradient steps to output a near-optimal policy due to coverage collapse. This failure motivates interventions that preserve the base model’s coverage throughout training (hence the name coverage-preserving).
Focus: Online policy optimization. We focus on policy optimization algorithms that train on samples generated by the learner, as in PPO, GRPO, and other widely used post-training methods. We seek algorithms that are coverage-preserving without additional samples from the base model or other fixed distributions with favorable coverage, as standard procedures do not use them.
Theoretical testbed: Softmax linear policies. While many of our results apply to general policy parameterizations, our end-to-end complexity analyses work with softmax linear policy classes , defined below. We explicitly invoke Assumption 2.1 in results that require it.
Assumption 2.1 (Softmax linear policies). For a fixed feature map ϕ : X × Y → R d \phi: \mathcal{X} \times\mathcal{Y} \to\mathbb{R}^{d} ϕ : X × Y → R d , the policy class is
Π Θ : = { π θ : θ ∈ R d } , π θ ( y ∣ x ) = exp ( ϕ ( x , y ) ⊤ θ ) ∑ y ′ ∈ Y exp ( ϕ ( x , y ′ ) ⊤ θ ) . \Pi_{\Theta} := \left\{\pi_{\theta} : \theta\in\mathbb{R}^{d}\right\}, \qquad\pi_{\theta}(y \mid x) = \frac{\exp\left(\phi(x,y)^{\top}\theta\right)}{\sum_{y' \in\mathcal{Y}} \exp\left(\phi(x,y')^{\top}\theta\right)}. Π Θ := { π θ : θ ∈ R d } , π θ ( y ∣ x ) = ∑ y ′ ∈ Y exp ( ϕ ( x , y ′ ) ⊤ θ ) exp ( ϕ ( x , y ) ⊤ θ ) . Here, y ∈ Y y \in\mathcal{Y} y ∈ Y denotes a complete response consisting of a sequence of tokens. The reward function is linear in the same features, meaning there exists some θ ⋆ ∈ R d \theta_{\star} \in\mathbb{R}^{d} θ ⋆ ∈ R d such that
r ( x , y ) = ϕ ( x , y ) ⊤ θ ⋆ , ∀ ( x , y ) ∈ X × Y . r(x,y) = \phi(x,y)^{\top}\theta_{\star}, \qquad\forall(x,y) \in\mathcal{X} \times\mathcal{Y}. r ( x , y ) = ϕ ( x , y ) ⊤ θ ⋆ , ∀ ( x , y ) ∈ X × Y . We write B ϕ : = max x ∈ X , y ∈ Y ∥ ϕ ( x , y ) ∥ 2 B_{\phi} := \max_{x \in\mathcal{X},\, y \in\mathcal{Y}} \lVert\phi(x,y)\rVert_{2} B ϕ := max x ∈ X , y ∈ Y ∥ ϕ ( x , y ) ∥ 2 and B ⋆ : = ∥ θ ⋆ ∥ 2 B_{\star} := \lVert\theta_{\star}\rVert_{2} B ⋆ := ∥ θ ⋆ ∥ 2 .
Assumption 2.1 is the simplest function approximation setting for policies, where shared parameters allow learning from observed samples to transfer to unseen prompts. It models post-training as finetuning weights over the base model’s frozen features ϕ \phi ϕ . For softmax linear policies, the cross-entropy loss used in pretraining is convex [16 ] , whereas the expected reward J ( π θ ) J(\pi_{\theta}) J ( π θ ) is nonconcave in θ \theta θ [1 , 70 ] . The nonconcavity is therefore particular to the RL objective rather than the parameterization itself, and is reflected in the coverage collapse exhibited by Theorem 2.1, where the policy class is a special case of Assumption 2.1 with one-hot features ϕ \phi ϕ .
Reference Resets Preserve Coverage: APMD Framework
We now show that algorithms built around a single intervention—resetting the reference policy used for regularization—are coverage-preserving in the sense of Eq. (4) . Recall from § 2.1 that KL regularization to the base model does not prevent coverage collapse, as regularization must be weak enough for the learner to improve substantially (Figure 1 a). Stronger regularization would mitigate loss of coverage but bias the learner toward the base model, limiting potential improvement (Figure 1 b). When the reference is periodically reset to the most recent policy, however, the learner can keep improving and reach optimality (Figure 1 c).
We formalize this intervention as a meta-algorithm, Approximate Policy Mirror Descent (APMD), which solves a sequence of RL subproblems regularized to the most recent policy. As long as each RL subproblem is solved accurately enough, the policies preserve the base model’s coverage throughout training and converge to optimality (§ 3.1 ). We then instantiate APMD with stylized versions of PPO and GRPO, without (§ 3.2 ) and with (§ 3.3 ) clipping. In § 4 , we study a second family of APMD algorithms based on squared-loss objectives, including those used to train the Kimi models [49 b] .
Framework: Approximate Policy Mirror Descent
Approximate Policy Mirror Descent (APMD), defined below, starts from the base model, π 0 = π b a s e \pi_0 = \pi_{\mathrm{base}} π 0 = π base . At each outer iteration t t t , it first resets the reference —the policy to which regularization is anchored—to the most recent policy π t \pi_t π t . This sets the optimization target to π η , t ⋆ ∝ π t e η r \pi_{\eta,t}^{\star} \propto\pi_t e^{\eta r} π η , t ⋆ ∝ π t e η r , the exact mirror descent update to π t \pi_t π t under KL regularization.2 APMD then trains the next policy π t + 1 \pi_{t+1} π t + 1 to approximate π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ using an algorithm-specific inner loop.
Table 2.
All APMD algorithms perform the reference reset in Step 1 and share the optimization target π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ . They differ in the RL subproblem and inner optimization procedure used for Step 2. For example, the algorithm in § 3.2 uses gradient ascent on the KL-regularized expected reward in Eq. (5). The algorithms in § 4 instead minimize squared-loss objectives to approximate the same target.
If every RL subproblem were solved exactly, so that π t + 1 = π η , t ⋆ \pi_{t+1} = \pi_{\eta,t}^{\star} π t + 1 = π η , t ⋆ for all t t t , the policy iterates would follow the mirror descent path, satisfying (Appendix § 8 .§ 3 )
π t ( y ∣ x ) ∝ π b a s e ( y ∣ x ) e t η r ( x , y ) , and J ⋆ − J ( π t ) ≤ log C b a s e η t , ∀ 0 < t ≤ T . (5) \pi_t(y\mid x) \propto\pi_{\mathrm{base}}(y\mid x)e^{t\eta r(x,y)}, \qquad\text{and} \qquad J^{\star}-J(\pi_t)\le\frac{\log\mathcal{C}_{\mathrm{base}}}{\eta t}, \qquad\forall0<t\le T. \tag*{(5)} π t ( y ∣ x ) ∝ π base ( y ∣ x ) e t η r ( x , y ) , and J ⋆ − J ( π t ) ≤ η t log C base , ∀0 < t ≤ T . (5)
The policies converge at the sublinear mirror descent rate above, which depends on the base model only through log C b a s e \log\mathcal{C}_{\mathrm{base}} log C base . Moreover, both expected reward and coverage improve monotonically in t t t ; each iterate π t \pi_t π t draws optimal responses at least as often as the base model, preserving useful learning signals and avoiding the coverage collapse under fixed-reference policy optimization in Theorem 2.1. The challenge is maintaining these properties when π t + 1 \pi_{t+1} π t + 1 only approximates π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ , given that sampling and optimization errors compound across iterations.
The following theorem, a central result of the paper, shows that the final policy π T \pi_T π T retains the mirror descent rate in Eq. (5) , provided each π t + 1 \pi_{t+1} π t + 1 is sufficiently close to π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ in KL divergence. The learned policies also preserve coverage , meaning that, for most prompts, every iterate generates an optimal response at least half as often as the base model.3
Theorem 3.1 (APMD preserves coverage). Given any η > 0 \eta> 0 η > 0 and any sequence of policies { π t } t = 0 T \{\pi_t\}_{t=0}^{T} { π t } t = 0 T with π 0 = π b a s e \pi_0=\pi_{\mathrm{base}} π 0 = π base and π t > 0 \pi_t>0 π t > 0 , suppose that
D K L ( π t + 1 ∥ π η , t ⋆ ) ≤ ε i n : = 1 64 C b a s e T 2 log ( e T ) , ∀ 0 ≤ t < T . (6) D_{\mathrm{KL}}(\pi_{t+1}\Vert\pi_{\eta,t}^{\star}) \le\varepsilon_{\mathrm{in}} := \frac{1}{64\mathcal{C}_{\mathrm{base}}T^2\log(eT)}, \qquad\forall0\le t<T. \tag*{(6)} D KL ( π t + 1 ∥ π η , t ⋆ ) ≤ ε in := 64 C base T 2 log ( e T ) 1 , ∀0 ≤ t < T . (6) Then the final policy converges at the mirror descent rate,
J ⋆ − J ( π T ) ≤ log ( 2 C b a s e ) η T , (7) J^{\star}-J(\pi_T)\le\frac{\log(2\mathcal{C}_{\mathrm{base}})}{\eta T}, \tag*{(7)} J ⋆ − J ( π T ) ≤ η T log ( 2 C base ) , (7) and all policies preserve coverage: with probability at least 1 − 1 2 log ( e T ) 1-\frac{1}{2\sqrt{\log(eT)}} 1 − 2 l o g ( e T ) 1 over x ∼ μ x\sim\mu x ∼ μ ,
π t ( Y x ⋆ ∣ x ) ≥ 1 2 π b a s e ( Y x ⋆ ∣ x ) , ∀ 0 ≤ t ≤ T , \pi_t(\mathcal{Y}_x^{\star}\mid x)\ge\frac{1}{2}\pi_{\mathrm{base}}(\mathcal{Y}_x^{\star}\mid x), \qquad\forall0\le t\le T, π t ( Y x ⋆ ∣ x ) ≥ 2 1 π base ( Y x ⋆ ∣ x ) , ∀0 ≤ t ≤ T , where Y x ⋆ \mathcal{Y}_x^{\star} Y x ⋆ is the set of optimal responses for the prompt x x x (Definition 1.1).
Theorem 3.1 provides a criterion for identifying coverage-preserving APMD algorithms. It applies to any policy sequence { π t } t = 0 T \{\pi_t\}_{t=0}^{T} { π t } t = 0 T satisfying the inner K L KL K L tolerance ϵ i n \epsilon_{\mathrm{in}} ϵ in in Eq. (6) . For any η > 0 \eta> 0 η > 0 , the final policy is ϵ \epsilon ϵ -optimal when
T = ⌈ log ( 2 C b a s e ) η ϵ ⌉ , D K L ( π t + 1 ∥ π η , t ⋆ ) ≤ ϵ i n = O ~ ( η 2 ϵ 2 C b a s e ) , ∀ 0 ≤ t < T . (8) T = \left\lceil\frac{\log(2C_{\mathrm{base}})}{\eta\epsilon} \right\rceil,\qquad D_{\mathrm{KL}}\left(\pi_{t+1}\,\middle\|\,\pi_{\eta,t}^{\star}\right) \leq\epsilon_{\mathrm{in}} = \widetilde{O}\left(\frac{\eta^{2}\epsilon^{2}}{C_{\mathrm{base}}}\right),\qquad\forall0 \leq t < T. \tag*{(8)} T = ⌈ η ϵ log ( 2 C base ) ⌉ , D KL ( π t + 1 π η , t ⋆ ) ≤ ϵ in = O ( C base η 2 ϵ 2 ) , ∀0 ≤ t < T . (8)
Thus, for an APMD algorithm to be coverage-preserving, the inner optimization loop should meet the KL tolerance above using a polynomial number of samples and gradient steps, for a suitable choice of η \eta η . Following this approach, we will show that stylized variants of PPO and GRPO are coverage-preserving in § 3.2 and § 3.3 .
Intuition: APMD algorithms are coverage-preserving. Across our results, strong regularization to the reference—typically with η = Θ ( 1 ) \eta= \Theta(1) η = Θ ( 1 ) , compared to η = Ω ( 1 / ϵ ) \eta= \Omega(1/\epsilon) η = Ω ( 1/ ϵ ) when the reference is fixed (§ 2.1 )—is key to obtaining coverage-preserving guarantees for APMD algorithms. This sets the target π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ close to the reference π t \pi_t π t and stabilizes the inner loop, making each RL subproblem relatively easy to solve (Figure 1 b). Correspondingly, as we will see in § 3.2 , APMD algorithms converge using exponentially fewer gradient steps than REINFORCE in the construction of Theorem 2.1, illustrated in Figure 2 a.
Reference resets in APMD make global optimality possible under strong regularization. Each π t + 1 ≈ π η , t ⋆ \pi_{t+1} \approx\pi_{\eta,t}^{\star} π t + 1 ≈ π η , t ⋆ is only a local improvement to π t \pi_t π t , but these improvements accumulate over successive resets (Figure 1 c). Eq. (8) makes this precise, showing that the same performance can be achieved under stronger regularization by proportionally increasing the number of outer iterations T T T . This role of reference resets echoes empirical observations from Liu et al. [62 ] , who report that the KL penalty to a fixed reference can dominate the loss and slow policy updates, and periodically reset the reference to counteract this effect.
Instantiating APMD: Importance-Weighted Policy Optimization (PPO-NoClip)
PPO and GRPO are the most recognizable policy optimization algorithms in post-training, yet the individual roles of key design elements—importance-weighted gradients, clipping, and KL regularization—are not well understood. To disentangle their effects, we analyze stylized variants using APMD. We begin with PPO-NoClip, which uses KL regularization with reference resets but omits clipping. We then add clipping in § 3.3 to obtain PPO-Lite. This approach reveals how KL regularization controls the optimization landscape, while clipping controls the statistical cost of importance weighting.
In each outer iteration t t t , PPO-NoClip aims to maximize the KL-regularized objective with reference π t \pi_t π t ,
J η , t ( π ) : = E π t [ r ( x , y ) ] − 1 η D K L ( π ∥ π t ) . (9) J_{\eta,t}(\pi) := \mathbb{E}_{\pi_t}[r(x,y)] - \frac{1}{\eta}D_{\mathrm{KL}}(\pi\,\|\, \pi_t). \tag*{(9)} J η , t ( π ) := E π t [ r ( x , y )] − η 1 D KL ( π ∥ π t ) . (9)
This objective is uniquely maximized by π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ (Eq. (5)), so PPO-NoClip implements APMD. Note that we use observed rewards rather than advantage estimates, as our focus is on the effects of regularization and clipping.
To optimize this objective, the inner loop of PPO-NoClip performs K K K steps of full-batch gradient ascent, starting from π t 0 = π t \pi_t^0 = \pi_t π t 0 = π t . It reuses a pre-collected batch of N N N prompt-response pairs from π t \pi_t π t to estimate these gradients, following standard practice in PPO and GRPO to reduce generation costs (Schulman et al., [93 ] ; Shao et al., [96 ] ). At inner step k k k , the estimated gradient uses importance weights π t k ( y ∣ x ) / π t ( y ∣ x ) \pi_t^k(y \mid x)/\pi_t(y \mid x) π t k ( y ∣ x ) / π t ( y ∣ x ) to correct for distribution shift between the learner π t k \pi_t^k π t k and the sampling policy π t \pi_t π t (Eq. (11) ). This procedure is equivalent to gradient ascent on the importance-weighted formulation of Eq. (9) below, which resembles the PPO objective without clipping and with regularization to π t \pi_t π t .
J η , t ( π θ ) = E π t [ π θ ( y ∣ x ) π t ( y ∣ x ) ( r ( x , y ) − 1 η log π θ ( y ∣ x ) π t ( y ∣ x ) ) ] . (10) J_{\eta,t}(\pi_\theta) = \mathbb{E}_{\pi_t}\left[\frac{\pi_\theta(y \mid x)}{\pi_t(y \mid x)}\left(r(x,y) - \frac{1}{\eta}\log\frac{\pi_\theta(y \mid x)}{\pi_t(y \mid x)}\right)\right]. \tag*{(10)} J η , t ( π θ ) = E π t [ π t ( y ∣ x ) π θ ( y ∣ x ) ( r ( x , y ) − η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) ) ] . (10)
Finally, PPO-NoClip sets π t + 1 = π t K \pi_{t+1} = \pi_t^K π t + 1 = π t K , which becomes the reference policy, sampling policy, and starting point for the next outer iteration.
The following result shows that PPO-NoClip is coverage-preserving under Assumption 2.1.
Algorithm 1 PPO with Reference Resets, without Clipping (PPO-NoClip)
Require: Base model π b a s e \pi_{\mathrm{base}} π base , outer iterations T T T , regularization parameter η > 0 \eta> 0 η > 0 , prompt batch size N N N , inner steps K K K , and learning rate α \alpha α .
Initialize π 0 = π b a s e = π θ 0 \pi_0 = \pi_{\mathrm{base}} = \pi_{\theta_0} π 0 = π base = π θ 0 .
for t = 0 , 1 , … , T − 1 t = 0, 1, \ldots, T - 1 t = 0 , 1 , … , T − 1 do
Draw a batch D t = { ( x i , y i , r i ) } i = 1 N D_t = \{(x_i, y_i, r_i)\}_{i=1}^{N} D t = {( x i , y i , r i ) } i = 1 N , where x i ∼ i i d μ x_i \mathrel{\overset{\mathrm{iid}}{\sim}} \mu x i ∼ iid μ , y i ∼ π t ( ⋅ ∣ x i ) y_i \sim\pi_t(\cdot\mid x_i) y i ∼ π t ( ⋅ ∣ x i ) , and r i = r ( x i , y i ) r_i = r(x_i, y_i) r i = r ( x i , y i ) .
Initialize θ t 0 = θ t \theta_t^0 = \theta_t θ t 0 = θ t and π t 0 = π t \pi_t^0 = \pi_t π t 0 = π t .
for k = 0 , 1 , … , K − 1 k = 0, 1, \ldots, K - 1 k = 0 , 1 , … , K − 1 do
Compute the gradient estimate
∇ J η , t ( π t k ) = 1 N ∑ i = 1 N π t k ( y i ∣ x i ) π t ( y i ∣ x i ) ∇ log π t k ( y i ∣ x i ) ( r i − 1 η log π t k ( y i ∣ x i ) π t ( y i ∣ x i ) ) . (11) \nabla J_{\eta,t}(\pi_t^k) = \frac{1}{N} \sum_{i=1}^{N} \frac{\pi_t^k(y_i \mid x_i)}{\pi_t(y_i \mid x_i)} \nabla\log\pi_t^k(y_i \mid x_i) \left(r_i - \frac{1}{\eta} \log\frac{\pi_t^k(y_i \mid x_i)}{\pi_t(y_i \mid x_i)}\right). \tag*{(11)} ∇ J η , t ( π t k ) = N 1 i = 1 ∑ N π t ( y i ∣ x i ) π t k ( y i ∣ x i ) ∇ log π t k ( y i ∣ x i ) ( r i − η 1 log π t ( y i ∣ x i ) π t k ( y i ∣ x i ) ) . (11)
Update θ t k + 1 = θ t k + α ∇ J η , t ( π t k ) \theta_t^{k+1} = \theta_t^k + \alpha\nabla J_{\eta,t}(\pi_t^k) θ t k + 1 = θ t k + α ∇ J η , t ( π t k ) and π t k + 1 = π θ t k + 1 \pi_t^{k+1} = \pi_{\theta_t^{k+1}} π t k + 1 = π θ t k + 1 .
Set θ t + 1 = θ t K \theta_{t+1} = \theta_t^K θ t + 1 = θ t K and π t + 1 = π t K \pi_{t+1} = \pi_t^K π t + 1 = π t K .
return π p o s t = π T \pi_{\mathrm{post}} = \pi_T π post = π T .
Corollary 3.1 (PPO-NoClip is coverage-preserving). Suppose that Assumption 2.1 holds, and let B : = B ϕ B ⋆ B := B_\phi B_\star B := B ϕ B ⋆ . In Algorithm 1, set η = 1 / B \eta= 1/B η = 1/ B . Then, for any ϵ ∈ ( 0 , R max ] \epsilon\in(0, R_{\max}] ϵ ∈ ( 0 , R m a x ] and δ ∈ ( 0 , 1 ) \delta\in(0, 1) δ ∈ ( 0 , 1 ) , there exist choices of α \alpha α , N N N , and K K K (see Corollary G.1), with T T T specified by Eq. (8) , such that PPO-NoClip returns a policy π p o s t \pi_{\mathrm{post}} π post satisfying J ⋆ − J ( π p o s t ) ≤ ϵ J^\star- J(\pi_{\mathrm{post}}) \le\epsilon J ⋆ − J ( π post ) ≤ ϵ with probability at least 1 − δ 1 - \delta 1 − δ , using
N s t e p ( ϵ , δ ) = O ~ ( C b a s e B 3 ϵ 3 ) and N s a m p l e ( ϵ , δ ) = O ~ ( C b a s e B 3 ( d + log ( 1 / δ ) ) ϵ 3 ) . N_{\mathrm{step}}(\epsilon, \delta) = \widetilde{O}\left(\frac{C_{\mathrm{base}}B^3}{\epsilon^3}\right)
\qquad\text{and}\qquad
N_{\mathrm{sample}}(\epsilon, \delta) = \widetilde{O}\left(\frac{C_{\mathrm{base}}B^3(d + \log(1/\delta))}{\epsilon^3}\right). N step ( ϵ , δ ) = O ( ϵ 3 C base B 3 ) and N sample ( ϵ , δ ) = O ( ϵ 3 C base B 3 ( d + log ( 1/ δ )) ) .
Here, N s t e p = K T N_{\mathrm{step}} = KT N step = K T and N s a m p l e = N T N_{\mathrm{sample}} = NT N sample = N T .
In the lower bound construction (Theorem 2.1), where C b a s e B 3 = O ( 1 ) C_{\mathrm{base}}B^3 = O(1) C base B 3 = O ( 1 ) , Corollary 3.1 gives N s t e p = O ~ ( ϵ − 3 ) N_{\mathrm{step}} = \widetilde{O}(\epsilon^{-3}) N step = O ( ϵ − 3 ) , while REINFORCE on the fixed-reference objective requires exp ( Ω ( 1 / ϵ ) ) \exp(\Omega(1/\sqrt{\epsilon})) exp ( Ω ( 1/ ϵ )) steps to converge even with exact gradients. Figure 2 a illustrates this performance gap with exact gradients for both methods. The difference is that PPO-NoClip uses strong regularization, η = 1 / B \eta= 1/B η = 1/ B , thereby avoiding coverage collapse. The proof sketch below makes clear how optimization benefits from this regularization.
Overview of analysis for PPO-NoClip (Corollary 3.1) By Theorem 3.1 , it suffices to show that each inner loop returns a policy with D K L ( π t + 1 ∥ π η , t ⋆ ) ≤ ϵ i n D_{\mathrm{KL}}(\pi_{t+1} \mathbin\Vert\pi_{\eta,t}^\star) \le\epsilon_{\mathrm{in}} D KL ( π t + 1 ∥ π η , t ⋆ ) ≤ ϵ in . Maximizing J η , t J_{\eta,t} J η , t (Eq. (9) ) directly controls this KL divergence, which is proportional to the suboptimality of π t + 1 \pi_{t+1} π t + 1 on the RL subproblem (Lemma G.2 ),
1 η D K L ( π ∥ π η , t ⋆ ) = J η , t ( π η , t ⋆ ) − J η , t ( π ) . \frac{1}{\eta}D_{\mathrm{KL}}(\pi\mathbin\Vert\pi_{\eta,t}^\star) = J_{\eta,t}(\pi_{\eta,t}^\star) - J_{\eta,t}(\pi). η 1 D KL ( π ∥ π η , t ⋆ ) = J η , t ( π η , t ⋆ ) − J η , t ( π ) .
We therefore bound the batch size N N N and gradient steps K K K needed for π t + 1 \pi_{t+1} π t + 1 to be ϵ i n / η \epsilon_{\mathrm{in}}/\eta ϵ in / η -suboptimal.
(Optimization benefits of regularization) Although J η , t J_{\eta,t} J η , t is nonconcave in θ \theta θ , a local form of concavity holds at each inner iterate π t k \pi_t^k π t k (Lemma G.6 ),
J η , t ( π η , t ⋆ ) − J η , t ( π t k ) ≤ 1 2 ∥ π η , t ⋆ π t k ∥ ∞ ⟨ ∇ J η , t ( π t k ) , θ t − θ t k + η θ ⋆ ⟩ . (12) J_{\eta,t}(\pi_{\eta,t}^\star) - J_{\eta,t}(\pi_t^k) \le\frac{1}{2}\left\|\frac{\pi_{\eta,t}^\star}{\pi_t^k}\right\|_\infty\left\langle\nabla J_{\eta,t}(\pi_t^k), \theta_t - \theta_t^k + \eta\theta_\star\right\rangle. \tag*{(12)} J η , t ( π η , t ⋆ ) − J η , t ( π t k ) ≤ 2 1 π t k π η , t ⋆ ∞ ⟨ ∇ J η , t ( π t k ) , θ t − θ t k + η θ ⋆ ⟩ . (12)
Like the Łojasiewicz inequality in Eq. (3) , this bound weakens when π t k \pi_t^k π t k covers the target poorly and the ratio π η , t ⋆ / π t k \pi_{\eta,t}^\star/\pi_t^k π η , t ⋆ / π t k is large. This allows for suboptimal policies that are also near-stationary, consistent with the training plateaus caused by coverage collapse in § 2.1 . Convergence also depends on a second policy ratio, the importance weight π t k / π t \pi_t^k/\pi_t π t k / π t , which, when large, worsens smoothness and amplifies gradient noise.
KL regularization controls both ratios under Assumption 2.1, and we show that employing strong regularization keeps all inner iterates close both to the reference and sampling policy π t \pi_t π t and to the target π η , t ⋆ \pi_{\eta,t}^\star π η , t ⋆ , in the sense that
π t k ( y ∣ x ) π t ( y ∣ x ) ≤ e 8 η B and π t , η ∗ ( y ∣ x ) π t k ( y ∣ x ) ≤ e 4 η B , ∀ k ∈ [ K ] . (13) \frac{\pi_t^k(y \mid x)}{\pi_t(y \mid x)} \le e^{8\eta B}
\quad\text{and}\quad
\frac{\pi_{t,\eta}^*(y \mid x)}{\pi_t^k(y \mid x)} \le e^{4\eta B}, \quad\forall k \in[K]. \tag*{(13)} π t ( y ∣ x ) π t k ( y ∣ x ) ≤ e 8 η B and π t k ( y ∣ x ) π t , η ∗ ( y ∣ x ) ≤ e 4 η B , ∀ k ∈ [ K ] . (13)
With η = 1 / B \eta= 1/B η = 1/ B , both ratios are bounded by absolute constants throughout the inner optimization loop, giving uniform control over gradient smoothness and the local concavity in Eq. (12) , as illustrated by the optimization landscape of APMD in Figure 3 c.
Figure 3. APMD continues improving by resetting the reference policy. We plot the optimization landscapes for the APMD run from Figure 2 in (c). Each surface shows the objective J η , t J_{\eta,t} J η , t for a selected reference policy π t \pi_t π t , with color indicating reward. APMD runs PPO-NoClip with a reference reset every K = 100 K=100 K = 100 inner steps. For visualization purposes we add height offsets to make the trajectory continuous over resets, which does not change curvature or gradients. (a,b) The corresponding fixed-reference landscapes with weak and strong KL regularization. Panel (a) matches Figure 2 b, while (b) shows the initial objective in (c).
(Inner loop convergence) Adapting standard gradient ascent analysis shows it suffices to take
K = O ~ ( 1 ε i n ) , N = O ~ ( d + log ( T / δ ) ε i n ) . K = \widetilde{O}\left(\frac{1}{\varepsilon_{\mathrm{in}}}\right), \qquad
N = \widetilde{O}\left(\frac{d+\log(T/\delta)}{\varepsilon_{\mathrm{in}}}\right). K = O ( ε in 1 ) , N = O ( ε in d + log ( T / δ ) ) .
With T T T and ε i n \varepsilon_{\mathrm{in}} ε in as in Eq. (8) for η = 1 / B \eta= 1/B η = 1/ B , this gives the total gradient steps N s t e p ( ε , δ ) = K T N_{\mathrm{step}}(\varepsilon,\delta)=KT N step ( ε , δ ) = K T and samples N s a m p l e ( ε , δ ) = N T N_{\mathrm{sample}}(\varepsilon,\delta)=NT N sample ( ε , δ ) = N T stated in Corollary 3.1. The full proof appears in Appendix G.§ 3 ; our analyses of other algorithms will follow the template sketched here.
On-policy PPO-NoClip. KL regularization to the reference improves convergence by controlling the distribution shift (Eq. (13) ) that governs the optimization landscape (Eq. (12) ). This role of the reference policy—anchoring optimization to a local region where it is well-behaved—is distinct from its use in the importance weight. To see this distinction, consider the on-policy variant of PPO-NoClip, which is also of independent interest given recent empirical work on on-policy training (Xiang et al., [112 ] ; Marek and Ryabinin, [67 ] ). This variant also regularizes to π t \pi_t π t , but draws a fresh batch from π t k \pi_t^k π t k at each inner step to estimate the on-policy gradient,
∇ J η , t ( π t k ) = E π t k [ ∇ log π t k ( y ∣ x ) ( r ( x , y ) − 1 η log π t k ( y ∣ x ) π t ( y ∣ x ) ) ] . \nabla J_{\eta,t}(\pi_t^k)
=
\mathbb{E}_{\pi_t^k}\left[
\nabla\log\pi_t^k(y \mid x)
\left(
r(x,y)-\frac{1}{\eta}\log\frac{\pi_t^k(y \mid x)}{\pi_t(y \mid x)}
\right)
\right]. ∇ J η , t ( π t k ) = E π t k [ ∇ log π t k ( y ∣ x ) ( r ( x , y ) − η 1 log π t ( y ∣ x ) π t k ( y ∣ x ) ) ] .
This removes importance weights, but the objective J η , t J_{\eta,t} J η , t remains the same and the local concavity bound in Eq. (12) still applies. Strong regularization to π t \pi_t π t therefore remains key for convergence of the inner loop. Our analysis readily adapts to establish that on-policy PPO-NoClip is coverage-preserving under Assumption 2.1.
Instantiating APMD: A Stylized Variant of PPO with Clipping (PPO-Lite)
For softmax linear policies, we showed that KL regularization keeps policy ratios controlled throughout training. This may not hold for more complex nonlinear policies, and to limit their effects we now add clipping to the importance weights. Clipping is a core component of PPO and GRPO widely used in practice, but its benefits are not well understood theoretically. We establish a concrete statistical benefit by obtaining sample complexity guarantees for general policy parameterizations without requiring bounded policy ratios throughout training.
PPO-Lite clips the importance weights in the PPO-NoClip objective (Eq. (10) ), using separate lower and upper thresholds, 1 − τ − 1-\tau_- 1 − τ − and 1 + τ + 1+\tau_+ 1 + τ + . We denote the clipping operator by [ z ] 1 − τ − 1 + τ + : = min { 1 + τ + , max { 1 − τ − , z } } [z]_{1-\tau_-}^{1+\tau_+}:=\min\{1+\tau_+,\max\{1-\tau_-,z\}\} [ z ] 1 − τ − 1 + τ + := min { 1 + τ + , max { 1 − τ − , z }} . Writing w t π ( x , y ) : = π ( y ∣ x ) / π t ( y ∣ x ) w_t^\pi(x,y):=\pi(y \mid x)/\pi_t(y \mid x) w t π ( x , y ) := π ( y ∣ x ) / π t ( y ∣ x ) for the importance weight, we define the clipped objective as
J t η , c l i p ( π ) : = E π t [ min { w t π ( x , y ) r ( x , y ) , [ w t π ( x , y ) ] 1 − τ − 1 + τ + r ( x , y ) } ] − 1 η D K L ( π ∥ π t ) . (14) J_t^{\eta,\mathrm{clip}}(\pi)
:=
\mathbb{E}_{\pi_t}\left[
\min\left\{
w_t^\pi(x,y)r(x,y),
[w_t^\pi(x,y)]_{1-\tau_-}^{1+\tau_+}r(x,y)
\right\}
\right]
-\frac{1}{\eta}D_{\mathrm{KL}}(\pi\,\|\, \pi_t). \tag*{(14)} J t η , clip ( π ) := E π t [ min { w t π ( x , y ) r ( x , y ) , [ w t π ( x , y ) ] 1 − τ − 1 + τ + r ( x , y ) } ] − η 1 D KL ( π ∥ π t ) . (14)
Eq. (14) uses the same clipping scheme as PPO (Schulman et al., [93 ] ), taking the minimum of the clipped and unclipped importance-weighted rewards. It also allows for the asymmetric clipping widely adopted in RL post-training (Yu et al., [116 ] ; Zheng et al., [125 ] ; Yue et al., [120 ] ; Liu et al., [62 ] ). Like PPO-NoClip, PPO-Lite (Algorithm 2) optimizes J t η , c l i p J_t^{\eta,\mathrm{clip}} J t η , clip using a pre-collected batch of N N N prompt-response pairs from π t \pi_t π t at each outer iteration t t t . We leave the inner optimization loop abstract (Line 5); it returns a policy π t + 1 \pi_{t+1} π t + 1 that is approximately optimal for J ^ t η , c l i p \widehat{J}_t^{\eta,\mathrm{clip}} J t η , clip , the empirical version of J t η , c l i p J_t^{\eta,\mathrm{clip}} J t η , clip , up to optimization error ε o p t \varepsilon_{\mathrm{opt}} ε opt .
The following result bounds the samples needed for PPO-Lite to learn an ε \varepsilon ε -optimal policy, provided each inner loop meets the prescribed optimization tolerance. In this sense, the sample complexity of PPO-Lite is coverage-preserving.
Algorithm 2 PPO with Reference Resets and Clipping (PPO-Lite)
Require: Base model π b a s e \pi_{\mathrm{base}} π base , outer iterations T T T , regularization parameter η > 0 \eta> 0 η > 0 , prompt batch size N N N , clipping thresholds τ − \tau_{-} τ − and τ + \tau_{+} τ + , and inner optimization tolerance ϵ o p t \epsilon_{\mathrm{opt}} ϵ opt .
Initialize π 0 = π b a s e \pi_{0} = \pi_{\mathrm{base}} π 0 = π base .
for t = 0 , 1 , … , T − 1 t = 0, 1, \ldots, T - 1 t = 0 , 1 , … , T − 1 do
Draw a batch D t = { ( x i , y i , r i ) } i = 1 N D_{t} = \{(x_{i}, y_{i}, r_{i})\}_{i=1}^{N} D t = {( x i , y i , r i ) } i = 1 N where x i ∼ i i d μ x_{i} \overset{\mathrm{iid}}{\sim} \mu x i ∼ iid μ , y i ∼ π t ( ⋅ ∣ x i ) y_{i} \sim\pi_{t}(\cdot\mid x_{i}) y i ∼ π t ( ⋅ ∣ x i ) , and r i = r ( x i , y i ) r_{i} = r(x_{i}, y_{i}) r i = r ( x i , y i ) .
Define the empirical clipped objective
J ^ t η , c l i p ( π ) : = 1 N ∑ i = 1 N min { π ( y i ∣ x i ) π t ( y i ∣ x i ) r i , [ π ( y i ∣ x i ) π t ( y i ∣ x i ) ] 1 − τ − 1 + τ + r i } − 1 η D K L ( π ∥ π t ) . \widehat{J}_{t}^{\eta,\mathrm{clip}}(\pi) := \frac{1}{N}\sum_{i=1}^{N}\min\left\{\frac{\pi(y_{i}\mid x_{i})}{\pi_{t}(y_{i}\mid x_{i})}r_{i},\left[\frac{\pi(y_{i}\mid x_{i})}{\pi_{t}(y_{i}\mid x_{i})}\right]_{1-\tau_{-}}^{1+\tau_{+}}r_{i}\right\}-\frac{1}{\eta}D_{\mathrm{KL}}(\pi\parallel\pi_{t}). J t η , clip ( π ) := N 1 i = 1 ∑ N min { π t ( y i ∣ x i ) π ( y i ∣ x i ) r i , [ π t ( y i ∣ x i ) π ( y i ∣ x i ) ] 1 − τ − 1 + τ + r i } − η 1 D KL ( π ∥ π t ) .
Find π t + 1 = π θ ∈ Π Θ \pi_{t+1} = \pi_{\theta} \in\Pi_{\Theta} π t + 1 = π θ ∈ Π Θ satisfying
J ^ t η , c l i p ( π t + 1 ) ≥ sup π ∈ Π Θ J ^ t η , c l i p ( π ) − ϵ o p t . (15) \widehat{J}_{t}^{\eta,\mathrm{clip}}(\pi_{t+1}) \ge\sup_{\pi\in\Pi_{\Theta}}\widehat{J}_{t}^{\eta,\mathrm{clip}}(\pi)-\epsilon_{\mathrm{opt}}. \tag*{(15)} J t η , clip ( π t + 1 ) ≥ π ∈ Π Θ sup J t η , clip ( π ) − ϵ opt . (15)
return π p o s t = π T \pi_{\mathrm{post}} = \pi_{T} π post = π T .
Corollary 3.2 (PPO-Lite is coverage-preserving (Corollary H.1, informal)). Suppose that Π Θ \Pi_{\Theta} Π Θ is finite,4 r ( x , y ) ∈ [ − R max , R max ] r(x,y)\in[-R_{\max},R_{\max}] r ( x , y ) ∈ [ − R m a x , R m a x ] , and π η , t ⋆ ∈ Π Θ \pi_{\eta,t}^{\star}\in\Pi_{\Theta} π η , t ⋆ ∈ Π Θ for every t t t . In Algorithm 2, set η = 1 / R max \eta=1/R_{\max} η = 1/ R m a x and the clipping thresholds to τ − = 1 − e − 2 η R max \tau_{-}=1-e^{-2\eta R_{\max}} τ − = 1 − e − 2 η R m a x and τ + = e 2 η R max − 1 \tau_{+}=e^{2\eta R_{\max}}-1 τ + = e 2 η R m a x − 1 . Then, for any ϵ ∈ ( 0 , 2 R max ] \epsilon\in(0,2R_{\max}] ϵ ∈ ( 0 , 2 R m a x ] and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , set T T T and ϵ i n \epsilon_{\mathrm{in}} ϵ in as specified by Eq. (8) . If each inner loop satisfies Eq. (15) with optimization tolerance ϵ o p t ≤ ϵ i n / ( 8 η ) \epsilon_{\mathrm{opt}}\le\epsilon_{\mathrm{in}}/(8\eta) ϵ opt ≤ ϵ in / ( 8 η ) , then PPO-Lite returns a policy π p o s t \pi_{\mathrm{post}} π post satisfying J ⋆ − J ( π p o s t ) ≤ ϵ J^{\star}-J(\pi_{\mathrm{post}})\le\epsilon J ⋆ − J ( π post ) ≤ ϵ with probability at least 1 − δ 1-\delta 1 − δ , using
N s a m p l e ( ϵ , δ ) = N T = O ~ ( C b a s e R max 3 ( log ∣ Π Θ ∣ + log ( 1 / δ ) ) ϵ 3 ) . N_{\mathrm{sample}}(\epsilon,\delta)=NT=\widetilde{O}\left(\frac{C_{\mathrm{base}}R_{\max}^{3}\left(\log|\Pi_{\Theta}|+\log(1/\delta)\right)}{\epsilon^{3}}\right). N sample ( ϵ , δ ) = N T = O ( ϵ 3 C base R m a x 3 ( log ∣ Π Θ ∣ + log ( 1/ δ ) ) ) .
Clipping thresholds & KL regularization. Our guarantee relies on both regularization and clipping. The strength of regularization defines how close the target π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ is to the reference π t \pi_{t} π t , with π η , t ⋆ / π t ∈ [ e − 2 η R max , e 2 η R max ] \pi_{\eta,t}^{\star}/\pi_{t}\in[e^{-2\eta R_{\max}},e^{2\eta R_{\max}}] π η , t ⋆ / π t ∈ [ e − 2 η R m a x , e 2 η R m a x ] . Corollary 3.2 chooses the narrowest clipping interval guaranteed to leave π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ unclipped, which therefore remains a maximizer of the clipped objective. Although clipping was introduced in PPO as an alternative to KL regularization [93 ] , here it complements rather than replaces regularization. Regularization defines the optimization target itself and sets the difficulty of the problem, while clipping on its own does not directly provide the same control.
Benefits and costs of clipping. To understand the benefits of clipping, let us compare Corollary 3.2 with the guarantee for PPO-NoClip in Corollary 3.1, which holds for softmax linear policies. There, sufficiently strong KL regularization keeps policy ratios bounded by a constant throughout training. If clipping were added using similar thresholds to Corollary 3.2 it would bring no theoretical benefits, as the policy ratios would remain unclipped. Indeed, Corollary H.2, our end-to-end analysis for PPO-Lite under Assumption 2.1, recovers Corollary 3.1.
In contrast, Corollary H.1 applies to any final policy obtained by approximately maximizing the empirical objective J ^ t η , c l i p \widehat{J}_{t}^{\eta,\mathrm{clip}} J t η , clip at each outer iteration, regardless of the policy parameterization, the inner optimization procedure, or how large the importance weights become. Clipping controls the contribution of importance-weighted rewards, so the required batch size depends on the preset clipping thresholds rather than the magnitude of the policy ratios during training. To our knowledge, this benefit of clipping has not been isolated in previous theory. We note that these statistical benefits come with potentially higher optimization costs as the objective is non-smooth at clipping thresholds. We therefore do not analyze specific optimization procedures for the inner loop when clipping may be active, leaving it to future work.
Remark 3.1 (Connection to practice). PPO-Lite resembles practical methods that periodically reset the reference policy and add KL regularization to a clipped importance-weighted objective, including the DeepSeek training procedures [96 , 33 ] and ProRL and its variants [62 , 35 ] 2026). Our analysis offers theoretical insight into this practice and suggests tuning the clipping thresholds together with the regularization strength, where stronger regularization may improve the optimization landscape. In particular, Corollary H.1 sets the clipping interval to [ e − 2 η R max , e 2 η R max ] [e^{-2\eta R_{\max}}, e^{2\eta R_{\max}}] [ e − 2 η R m a x , e 2 η R m a x ] , so stronger regularization calls for tighter clipping.
Additional results for § 3 : General performance target. Theorem F.1 in Appendix § 11.3 .§ 1 extends Theorem 3.1 to provide performance guarantees against any comparator policy π \pi π . In particular, it bounds J ( π ) − J ( π T ) J(\pi)-J(\pi_T) J ( π ) − J ( π T ) under an inner KL tolerance ϵ i n \epsilon_{\mathrm{in}} ϵ in that depends on the base model’s coverage of π \pi π . Combined with the analyses of PPO-NoClip or PPO-Lite, this gives guarantees on samples and/or gradient steps (see Eq. (101) ) for attaining reward at least J ( π ) − ϵ J(\pi)-\epsilon J ( π ) − ϵ .
Squared-Loss Algorithms Preserve Coverage: Kimi and Rebel
In this section, we study a second family of algorithms that instead minimize squared-loss objectives to approximate the APMD target π η , t ∗ \pi^*_{\eta,t} π η , t ∗ . These objectives are derived from the identity below, obtained by rearranging the target’s closed form π η , t ∗ ( y ∣ x ) ∝ π t ( y ∣ x ) e η r ( x , y ) \pi^*_{\eta,t}(y \mid x) \propto\pi_t(y \mid x)e^{\eta r(x,y)} π η , t ∗ ( y ∣ x ) ∝ π t ( y ∣ x ) e η r ( x , y ) .
1 η log π η , t ∗ ( y ∣ x ) π t ( y ∣ x ) = r ( x , y ) − b t ∗ ( x ) ∀ ( x , y ) . (16) \frac{1}{\eta}\log\frac{\pi^*_{\eta,t}(y \mid x)}{\pi_t(y \mid x)}=r(x,y)-b^*_t(x)\qquad\forall(x,y). \tag*{(16)} η 1 log π t ( y ∣ x ) π η , t ∗ ( y ∣ x ) = r ( x , y ) − b t ∗ ( x ) ∀ ( x , y ) . (16)
The reference π t \pi_t π t now appears in the denominator of the log policy ratio, and b t ∗ ( x ) = 1 η log E y ′ ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ′ ) ] b^*_t(x)=\frac{1}{\eta}\log\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[e^{\eta r(x,y')}] b t ∗ ( x ) = η 1 log E y ′ ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ′ ) ] is the normalization function that ensures π η , t ∗ ( ⋅ ∣ x ) \pi^*_{\eta,t}(\cdot\mid x) π η , t ∗ ( ⋅ ∣ x ) is a valid distribution. We can therefore approximate π η , t ∗ \pi^*_{\eta,t} π η , t ∗ by fitting the log-ratios of a policy to the shifted rewards r ( x , y ) − b t ∗ ( x ) r(x,y)-b^*_t(x) r ( x , y ) − b t ∗ ( x ) . Take, for example, the squared-loss objective
L t ( π θ ) : = E π t [ ( 1 η log π θ ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + b t ∗ ( x ) ) 2 ] . (17) L_t(\pi_\theta):=\mathbb{E}_{\pi_t}\left[\left(\frac{1}{\eta}\log\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+b^*_t(x)\right)^2\right]. \tag*{(17)} L t ( π θ ) := E π t [ ( η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) − r ( x , y ) + b t ∗ ( x ) ) 2 ] . (17)
This objective averages the squared residuals of Eq. (16) over π t \pi_t π t , and can therefore be optimized without importance weights. It is uniquely minimized by π η , t ∗ \pi^*_{\eta,t} π η , t ∗ and therefore implements APMD. Although the objective has no explicit KL penalty, it implicitly regularizes to π t \pi_t π t in the log policy ratio.
Overview. We analyze squared-loss algorithms with objectives based on Eq. (17) , which differ in how they handle the unknown normalization function b t ∗ ( x ) b^*_t(x) b t ∗ ( x ) . Kimi-style objectives estimate a proxy baseline, while pairwise objectives eliminate b t ∗ b^*_t b t ∗ using paired responses. We first establish conditions on the squared-loss error that ensure convergence under APMD (§ 4.1 ), then use this to show that three practical algorithms are coverage-preserving.
Kimi-style objectives: An estimated baseline (§ 4.2 ). Kimi-style objectives replace b t ∗ ( x ) b^*_t(x) b t ∗ ( x ) in Eq. (17) with a baseline b t ( x ) b_t(x) b t ( x ) , typically estimated using samples from π t \pi_t π t . This gives the objective
L t K i m i ( π θ ) : = E π t [ ( 1 η log π θ ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + b t ( x ) ) 2 ] . (18) L_t^{\mathrm{Kimi}}(\pi_\theta):=\mathbb{E}_{\pi_t}\left[\left(\frac{1}{\eta}\log\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+b_t(x)\right)^2\right]. \tag*{(18)} L t Kimi ( π θ ) := E π t [ ( η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) − r ( x , y ) + b t ( x ) ) 2 ] . (18)
The earliest uses of this formulation were, to our knowledge, in Kimi K1.5 ([50 ] ) and Kimi K2 ([49 ] ). It has since been adopted in algorithms such as OAPL ([89 ] ) and KARL ([15 ] ). These methods differ in their choice of baseline, which generally biases the learned policies away from π η , t ∗ \pi^*_{\eta,t} π η , t ∗ . We use APMD to analyze the two most common choices, characterizing the additional samples and gradient steps needed to control this bias.
KimiPO (§ 4.2 ) uses a mean baseline b t ( x ) = r ( x , π t ) b_t(x)=r(x,\pi_t) b t ( x ) = r ( x , π t ) , the expected reward under π t \pi_t π t ([50 a] ). The bias arises from how much r ( x , π t ) r(x,\pi_t) r ( x , π t ) underestimates b t ∗ b^*_t b t ∗ , and we show that stronger regularization controls this error.
KimiPO-Norm (Appendix § 17.1 ) uses the normalizer baseline , which estimates b t ∗ b^*_t b t ∗ using responses from π t \pi_t π t ([50 , 18 , 89 , 15 ] ). This estimator is biased but consistent, and we instead use additional responses to control its error.
Pairwise objectives: Removing the baseline with paired responses (§ 4.3 ). Pairwise objectives eliminate b t ∗ ( x ) b_t^*(x) b t ∗ ( x ) by comparing two responses to the same prompt, an approach popularized by DPO (Rafailov et al., [85 ] ). We analyze Rebel-OnPolicy, a variant of Rebel (Gao et al., [27 ] ) tailored to online RL with observed rewards. It minimizes the following objective with both responses drawn from π t \pi_t π t ,
L t R e b e l - O n ( π θ ) : = E x ∼ μ , y , y ′ ∼ π t ( ⋅ ∣ x ) [ ( 1 η log π θ ( y ∣ x ) π t ( y ∣ x ) − 1 η log π θ ( y ′ ∣ x ) π t ( y ′ ∣ x ) − ( r ( x , y ) − r ( x , y ′ ) ) ) 2 ] . (19) L_t^{\mathrm{Rebel\text{-}On}}(\pi_\theta) := \mathbb{E}_{x\sim\mu,\,y,y'\sim\pi_t(\cdot\mid x)}\left[\left(\frac{1}{\eta}\log\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}-\frac{1}{\eta}\log\frac{\pi_\theta(y'\mid x)}{\pi_t(y'\mid x)}-(r(x,y)-r(x,y'))\right)^2\right]. \tag*{(19)} L t Rebel - On ( π θ ) := E x ∼ μ , y , y ′ ∼ π t ( ⋅ ∣ x ) [ ( η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) − η 1 log π t ( y ′ ∣ x ) π θ ( y ′ ∣ x ) − ( r ( x , y ) − r ( x , y ′ )) ) 2 ] . (19)
L t R e b e l - O n L_t^{\mathrm{Rebel\text{-}On}} L t Rebel - On is unbiased with respect to π η , t ∗ \pi_{\eta,t}^* π η , t ∗ , making Rebel-OnPolicy a theoretically clean instance of APMD.
Natural policy gradient. NPG is a foundational policy optimization method with a squared-loss formulation closely related to the objectives studied here. We discuss these connections in Appendix § 17.6 .
Specializing APMD: A Mean-Squared Error Criterion
The algorithms in § 3 approximate the APMD target π η , t ∗ \pi_{\eta,t}^* π η , t ∗ in KL divergence. For squared-loss algorithms, which fit log policy ratios rather than distributions, we instead use the mean-squared error (MSE), defined below in terms of log-ratios centered under π t \pi_t π t . Centering removes response-independent normalization terms that are handled differently across squared-loss algorithms, yielding a common criterion for analysis. For a fixed iteration t t t and reference policy π t \pi_t π t , let
l t , η π ( x , y ) : = 1 η log π ( y ∣ x ) π t ( y ∣ x ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ 1 η log π ( y ′ ∣ x ) π t ( y ′ ∣ x ) ] l_{t,\eta}^{\pi}(x,y):=\frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[\frac{1}{\eta}\log\frac{\pi(y'\mid x)}{\pi_t(y'\mid x)}\right] l t , η π ( x , y ) := η 1 log π t ( y ∣ x ) π ( y ∣ x ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ η 1 log π t ( y ′ ∣ x ) π ( y ′ ∣ x ) ]
denote the centered log-ratios of a candidate policy π \pi π . The mean-squared error is then
MSE t ( π ) : = E π t [ ( l t , η π ( x , y ) − ( r ( x , y ) − r ( x , π t ) ) ) 2 ] . \operatorname{MSE}_t(\pi):=\mathbb{E}_{\pi_t}\left[\left(l_{t,\eta}^{\pi}(x,y)-(r(x,y)-r(x,\pi_t))\right)^2\right]. MSE t ( π ) := E π t [ ( l t , η π ( x , y ) − ( r ( x , y ) − r ( x , π t )) ) 2 ] .
The MSE compares the centered log-ratios of π \pi π with those of the target π η , t ∗ \pi_{\eta,t}^* π η , t ∗ . Indeed, centering Eq. (16) under π t \pi_t π t gives l t , η π η , t ∗ ( x , y ) = r ( x , y ) − r ( x , π t ) l_{t,\eta}^{\pi_{\eta,t}^*}(x,y)=r(x,y)-r(x,\pi_t) l t , η π η , t ∗ ( x , y ) = r ( x , y ) − r ( x , π t ) , so MSE t ( π ) = E π t [ ( l t , η π ( x , y ) − l t , η π η , t ∗ ( x , y ) ) 2 ] \operatorname{MSE}_t(\pi)=\mathbb{E}_{\pi_t}\left[(l_{t,\eta}^{\pi}(x,y)-l_{t,\eta}^{\pi_{\eta,t}^*}(x,y))^2\right] MSE t ( π ) = E π t [ ( l t , η π ( x , y ) − l t , η π η , t ∗ ( x , y ) ) 2 ] . Thus, π η , t ∗ \pi_{\eta,t}^* π η , t ∗ uniquely minimizes MSE t \operatorname{MSE}_t MSE t with zero error, and MSE t ( π ) \operatorname{MSE}_t(\pi) MSE t ( π ) measures how well π \pi π approximates π η , t ∗ \pi_{\eta,t}^* π η , t ∗ .
Later, we will show how the Kimi-style (Eq. (18) ) and pairwise objectives (Eq. (19) ) control MSE t ( π ) \operatorname{MSE}_t(\pi) MSE t ( π ) and yield coverage-preserving algorithms. These results rely on the following theorem, a counterpart to Theorem 3.1 , that ensures convergence and coverage preservation when MSE t ( π t + 1 ) \operatorname{MSE}_t(\pi_{t+1}) MSE t ( π t + 1 ) is sufficiently small at every iteration.
Theorem 4.1 (APMD: Mean-squared error criterion). Given any η > 0 \eta>0 η > 0 and any sequence of policies { π t } t = 0 T \{\pi_t\}_{t=0}^{T} { π t } t = 0 T with π 0 = π b a s e \pi_0=\pi_{\mathrm{base}} π 0 = π base and π t > 0 \pi_t>0 π t > 0 , suppose that
MSE t ( π t + 1 ) ≤ 1 4 η 2 T 2 ( C b a s e log ( e T ) + e η S ) , ∀ 0 ≤ t < T , (20) \operatorname{MSE}_t(\pi_{t+1})\leq\frac{1}{4\eta^2T^2(C_{\mathrm{base}}\log(eT)+e^{\eta S})},\qquad\forall\,0\leq t<T, \tag*{(20)} MSE t ( π t + 1 ) ≤ 4 η 2 T 2 ( C base log ( e T ) + e η S ) 1 , ∀ 0 ≤ t < T , (20) where S : = max 0 ≤ t < T max x , y ∣ l t , η π t + 1 ( x , y ) − ( r ( x , y ) − r ( x , π t ) ) ∣ S:=\max_{0\leq t<T}\max_{x,y}\left|l_{t,\eta}^{\pi_{t+1}}(x,y)-(r(x,y)-r(x,\pi_t))\right| S := max 0 ≤ t < T max x , y l t , η π t + 1 ( x , y ) − ( r ( x , y ) − r ( x , π t )) . Then J ∗ − J ( π T ) ≤ log ( 2 C b a s e ) η T J^*-J(\pi_T)\leq\frac{\log(2C_{\mathrm{base}})}{\eta T} J ∗ − J ( π T ) ≤ η T l o g ( 2 C base ) , and with probability at least 1 − 7 8 log ( e T ) 1-\frac{7}{8\sqrt{\log(eT)}} 1 − 8 l o g ( e T ) 7 over x ∼ μ x\sim\mu x ∼ μ , π t + 1 ( Y x ∗ ∣ x ) ≥ 1 2 π b a s e ( Y x ∗ ∣ x ) \pi_{t+1}(\mathcal{Y}_x^*\mid x)\geq\frac{1}{2}\pi_{\mathrm{base}}(\mathcal{Y}_x^*\mid x) π t + 1 ( Y x ∗ ∣ x ) ≥ 2 1 π base ( Y x ∗ ∣ x ) for all 0 ≤ t < T 0\leq t<T 0 ≤ t < T .
Because MSE t \operatorname{MSE}_t MSE t is measured under the sampling policy π t \pi_t π t , squared-loss algorithms can drive it down using a pre-collected batch without importance weights. However, the inner MSE tolerance shrinks with the term e η S e^{\eta S} e η S in its denominator (Eq. (20) ), where S S S bounds the magnitude of the residual error. This cost arises from distribution shift when translating small MSE t ( π t + 1 ) \operatorname{MSE}_t(\pi_{t+1}) MSE t ( π t + 1 ) into guarantees for the learned policy π t + 1 \pi_{t+1} π t + 1 . In our results, sufficiently strong regularization ensures e η S = O ( C b a s e ) e^{\eta S}=O(C_{\mathrm{base}}) e η S = O ( C base ) , making this term relatively harmless.
We now apply this criterion to squared-loss algorithms, starting with the Kimi-style objectives.
Instantiating APMD: Kimi-Style Policy Optimization Algorithms
We first study the Kimi-style objective (Eq. (18) ) with the mean baseline b t ( x ) = r ( x , π t ) b_t(x)=r(x,\pi_t) b t ( x ) = r ( x , π t ) used in Kimi K2 (Kimi Team et al., [50 ] ). Here r ( x , π t ) = E y ′ ∼ π t ( ⋅ ∣ x ) [ r ( x , y ′ ) ] r(x,\pi_t)=\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[r(x,y')] r ( x , π t ) = E y ′ ∼ π t ( ⋅ ∣ x ) [ r ( x , y ′ )] is the expected reward under π t \pi_t π t for prompt x x x .
Algorithm 3 Kimi Policy Optimization (KimiPO)
Require: Base model $\pi_{\mathrm{base}}$, outer iterations $T$, regularization parameter $\eta > 0$, prompt batch size $N$, response group size $G$, inner steps $K$, and learning rate $\alpha$.
1: Initialize $\pi_0 = \pi_{\mathrm{base}} = \pi_{\theta_0}$.
2: for $t = 0, 1, \ldots, T - 1$ do
3: Draw a batch $\mathcal{D}_t = \{(x_i, y_i, r_i, \bar{r}_i)\}_{i=1}^{N}$, where $x_i \overset{\mathrm{iid}}{\sim} \mu$ and $y_i, y'_{i,1}, \ldots, y'_{i,G} \sim \pi_t(\cdot \mid x_i)$ independently.
Set $r_i = r(x_i, y_i)$ and estimate the mean baseline by $\bar{r}_i = \frac{1}{G}\sum_{j=1}^{G} r(x_i, y'_{i,j})$.
4: Define the empirical objective
$$
\widehat{L}^{\mathrm{Kimi}}_t(\pi_\theta) := \frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{\eta}\log\frac{\pi_\theta(y_i \mid x_i)}{\pi_t(y_i \mid x_i)} - (r_i-\bar{r}_i)\right)^2.
$$
5: Initialize $\theta_t^0 = \theta_t$ and $\pi_t^0 = \pi_t$.
6: for $k = 0, 1, \ldots, K - 1$ do
7: Compute the empirical gradient $\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)$ (e.g., via backpropagation).
8: Update $\theta_t^{k+1} = \theta_t^k - \alpha\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)$ and $\pi_t^{k+1} = \pi_{\theta_t^{k+1}}$.
9: Set $\theta_{t+1} = \theta_t^K$ and $\pi_{t+1} = \pi_t^K$.
10: return $\pi_{\mathrm{post}} = \pi_T$.
The resulting algorithm, KimiPO (Algorithm 3), uses the following objective in each outer iteration t t t ,
L t K i m i ( π θ ) = E π t [ ( 1 η log π θ ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + r ( x , π t ) ) 2 ] . L_t^{\mathrm{Kimi}}(\pi_\theta) = \mathbb{E}_{\pi_t}\left[\left(\frac{1}{\eta}\log\frac{\pi_\theta(y \mid x)}{\pi_t(y \mid x)} - r(x,y) + r(x,\pi_t)\right)^2\right]. L t Kimi ( π θ ) = E π t [ ( η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) − r ( x , y ) + r ( x , π t ) ) 2 ] .
To minimize this objective, KimiPO uses a pre-collected batch of N N N prompts and G + 1 G+1 G + 1 responses per prompt from π t \pi_t π t . For each prompt, it estimates the mean baseline by averaging the rewards of G G G responses (Line 3), then combines this estimate with the remaining response to form the empirical objective L ^ t K i m i \widehat{L}_t^{\mathrm{Kimi}} L t Kimi (Line 4). Starting from π t \pi_t π t , the inner loop takes K K K full-batch gradient descent steps on L ^ t K i m i \widehat{L}_t^{\mathrm{Kimi}} L t Kimi and sets π t + 1 = π t K \pi_{t+1}=\pi_t^K π t + 1 = π t K .5
The following result shows that KimiPO is coverage-preserving under Assumption 2.1.
Corollary 4.1 (KimiPO is coverage-preserving). Suppose that Assumption 2.1 holds, and let B : = B ϕ B ⋆ B := B_\phi B_\star B := B ϕ B ⋆ . For any ε ∈ ( 0 , R max ] \varepsilon\in(0,R_{\max}] ε ∈ ( 0 , R m a x ] and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , in Algorithm 3, set G = 1 G=1 G = 1 and η = O ~ ( ε / ( C b a s e 1 / 4 B 3 / 2 ) ) \eta=\widetilde{O}\left(\sqrt{\varepsilon}/(C_{\mathrm{base}}^{1/4}B^{3/2})\right) η = O ( ε / ( C base 1/4 B 3/2 ) ) . Then there exist choices of α \alpha α , N N N , and K K K (see Corollary J.1), with T T T specified by Eq. (8) , such that KimiPO returns a policy π p o s t \pi_{\mathrm{post}} π post satisfying J ⋆ − J ( π p o s t ) ≤ ε J^\star-J(\pi_{\mathrm{post}})\leq\varepsilon J ⋆ − J ( π post ) ≤ ε with probability at least 1 − δ 1-\delta 1 − δ , using
N s t e p ( ε , δ ) = O ~ ( C b a s e 5 / 4 B 7 / 2 ε 7 / 2 ) and N s a m p l e ( ε , δ ) = O ~ ( C b a s e 5 / 4 B 7 / 2 ( d + log ( 1 / δ ) ) ε 7 / 2 ) . N_{\mathrm{step}}(\varepsilon,\delta)=\widetilde{O}\left(\frac{C_{\mathrm{base}}^{5/4}B^{7/2}}{\varepsilon^{7/2}}\right)
\qquad\text{and}\qquad
N_{\mathrm{sample}}(\varepsilon,\delta)=\widetilde{O}\left(\frac{C_{\mathrm{base}}^{5/4}B^{7/2}(d+\log(1/\delta))}{\varepsilon^{7/2}}\right). N step ( ε , δ ) = O ( ε 7/2 C base 5/4 B 7/2 ) and N sample ( ε , δ ) = O ( ε 7/2 C base 5/4 B 7/2 ( d + log ( 1/ δ )) ) . Here, N s t e p = K T N_{\mathrm{step}}=KT N step = K T and N s a m p l e = N ( G + 1 ) T N_{\mathrm{sample}}=N(G+1)T N sample = N ( G + 1 ) T .
To our knowledge, Corollary 4.1 gives the first end-to-end global convergence guarantee for a Kimi-style algorithm. As in § 3 , this guarantee relies on reference resets, which KimiPO implements by updating π t \pi_t π t in the log policy ratio. This design choice is employed by both Kimi K1.5 and Kimi K2 (Kimi Team et al. [50 ] ), as well as specific instantiations of OAPL (Ritter et al. [89 ] ) and KARL (Chang et al. [15 ] ), all of which can be viewed as APMD algorithms. By contrast, A⋆ ^\star ⋆ -PO (Brantley et al. [8 ] ) and SPO (Cohen et al. [18 ] ) retain a fixed reference policy in the denominator of the log-ratio and do not implement APMD.
We highlight specific features of this result and our analysis below.
Bias from the baseline function. Our bounds on samples and gradient steps scale as O ~ ( ε − 7 / 2 ) \widetilde{O}(\varepsilon^{-7/2}) O ( ε − 7/2 ) for KimiPO, compared with O ~ ( ε − 3 ) \widetilde{O}(\varepsilon^{-3}) O ( ε − 3 ) for PPO-NoClip (Corollary 3.1). This extra dependence on ε \varepsilon ε arises because Corollary 4.1 uses stronger regularization (η = Θ ~ ( ε ) \eta= \widetilde{\Theta}(\sqrt{\varepsilon}) η = Θ ( ε ) rather than η = 1 / B \eta= 1/B η = 1/ B ) and thereby requires more outer iterations. KimiPO needs stronger regularization to control the bias introduced by the mean baseline, as the following decomposition shows (Lemma J.1),
L t K i m i ( π ) = MSE t ( π ) + 1 η 2 E x ∼ μ [ D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) 2 ] . L_t^{\mathrm{Kimi}}(\pi) = \operatorname{MSE}_t(\pi) + \frac{1}{\eta^2} \mathbb{E}_{x \sim\mu}\left[D_{\mathrm{KL}}\left(\pi_t(\cdot\mid x) \mathbin\Vert\pi(\cdot\mid x)\right)^2\right]. L t Kimi ( π ) = MSE t ( π ) + η 2 1 E x ∼ μ [ D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) 2 ] .
The first term is the MSE from Theorem 4.1 that KimiPO must control. The second favors policies close to π t \pi_t π t , consistent with the implicit regularization identified by Xu et al. [115 ] , so even exact minimization of L t K i m i L_t^{\mathrm{Kimi}} L t Kimi may not recover the target π η , t ∗ \pi_{\eta,t}^* π η , t ∗ . We show that this bias increases MSE t \operatorname{MSE}_t MSE t by O ( η 4 B 6 ) O(\eta^4 B^6) O ( η 4 B 6 ) (Lemma J.5 )—an error that is controlled by stronger regularization. Indeed, Corollary 4.1 holds for any G ≥ 1 G \ge1 G ≥ 1 , and choosing η = Θ ~ ( ε ) \eta= \widetilde{\Theta}(\sqrt{\varepsilon}) η = Θ ( ε ) keeps the bias small enough to meet the MSE criterion in Theorem 4.1.
Optimization landscape of Kimi-style objectives. Although the mean baseline introduces bias, it makes the KimiPO objective globally convex in θ \theta θ under Assumption 2.1 (Lemma J.3 ). By contrast, the PPO-NoClip objective is unbiased but satisfies only a local form of concavity around π t \pi_t π t (Eq. (12) ). The curvature of Kimi-style objectives depends on the baseline choice (Lemma I.4); the KimiPO-Norm objective studied below is likewise only locally convex.
Additional result: KimiPO-Norm estimates b t ∗ b_t^* b t ∗ . In Appendix § 17.1 , we analyze KimiPO-Norm (Algorithm 4), which optimizes the Kimi-style objective with baseline b t = b t ∗ b_t = b_t^* b t = b t ∗ . As in prior work (Kimi Team et al. [50 ] ; Cohen et al. [18 ] ; Ritter et al. [89 ] ; Chang et al. [15 ] ), the empirical objective uses a plug-in estimate of b t ∗ ( x ) b_t^*(x) b t ∗ ( x ) computed over G G G responses sampled from π t \pi_t π t ,
b ^ t ( x ) : = 1 η log ( 1 G ∑ j = 1 G e r ( x , y j ′ ) ) . \widehat{b}_t(x) := \frac{1}{\eta} \log\left(\frac{1}{G}\sum_{j=1}^{G} e^{r(x,y_j')}\right). b t ( x ) := η 1 log ( G 1 j = 1 ∑ G e r ( x , y j ′ ) ) .
This estimate is consistent but biased, and increasing G G G is what controls the resulting error (Lemma K.2 ). Corollary K.1 shows that KimiPO-Norm is coverage-preserving under Assumption 2.1 with G = Θ ~ ( 1 / ε ) G = \widetilde{\Theta}(1/\varepsilon) G = Θ ( 1/ ε ) and η = Θ ( 1 / B ) \eta= \Theta(1/B) η = Θ ( 1/ B ) . Compared with KimiPO, which takes η = Θ ~ ( ε ) \eta= \widetilde{\Theta}(\sqrt{\varepsilon}) η = Θ ( ε ) (Corollary 4.1), KimiPO-Norm uses significantly weaker regularization but more responses for accurate baseline estimation. This saves a factor of 1 / ε 1/\sqrt{\varepsilon} 1/ ε in N s t e p N_{\mathrm{step}} N step but increases N s a m p l e N_{\mathrm{sample}} N sample by the same factor (full comparison in Table 1 ).
Table 1. Parameter and complexity bounds as ε → 0 \varepsilon\to0 ε → 0 with problem parameters fixed, from Corollaries J.1 and K.1. Total responses are 2 N T 2NT 2 N T for KimiPO and N ( G + 1 ) T N(G+1)T N ( G + 1 ) T for KimiPO-Norm.
Instantiating APMD: Pairwise Squared-Loss Algorithms (Rebel-OnPolicy)
Finally, we study pairwise squared-loss algorithms , which do not require baseline functions. Our analysis focuses on Rebel-OnPolicy, a variant of Rebel (Gao et al. [27 ] ) that draws two independent responses from π t \pi_t π t for each prompt. At outer iteration t t t , the objective is
L t R e b e l − O n ( π θ ) : = E x ∼ μ , y , y ′ ∼ π t ( ⋅ ∣ x ) [ ( 1 η log π θ ( y ∣ x ) π t ( y ∣ x ) − 1 η log π θ ( y ′ ∣ x ) π t ( y ′ ∣ x ) − ( r ( x , y ) − r ( x , y ′ ) ) ) 2 ] . (21) L_t^{\mathrm{Rebel-On}}(\pi_\theta) := \mathbb{E}_{x \sim\mu,y,y' \sim\pi_t(\cdot\mid x)}\left[\left(\frac{1}{\eta}\log\frac{\pi_\theta(y \mid x)}{\pi_t(y \mid x)}-\frac{1}{\eta}\log\frac{\pi_\theta(y' \mid x)}{\pi_t(y' \mid x)}-(r(x,y)-r(x,y'))\right)^2\right]. \tag*{(21)} L t Rebel − On ( π θ ) := E x ∼ μ , y , y ′ ∼ π t ( ⋅ ∣ x ) [ ( η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) − η 1 log π t ( y ′ ∣ x ) π θ ( y ′ ∣ x ) − ( r ( x , y ) − r ( x , y ′ )) ) 2 ] . (21)
To optimize this objective, Rebel-OnPolicy (Algorithm 5) draws a batch of N N N prompts and two independent responses per prompt from π t \pi_t π t . It then takes K K K full-batch gradient descent steps on the empirical objective starting from π t \pi_t π t , and sets π t + 1 = π t K \pi_{t+1} = \pi_t^K π t + 1 = π t K .
The following result shows that Rebel-OnPolicy is coverage-preserving under Assumption 2.1.
Corollary 4.2 (Rebel-OnPolicy is coverage-preserving). Suppose that Assumption 2.1 holds, and let B : = B Φ B ⋆ B := B_{\Phi}B_{\star} B := B Φ B ⋆ . In Algorithm 5, set η = log ( 2 c b a s e ) / ( 2 B ) \eta= \log(2c_{\mathrm{base}})/(2B) η = log ( 2 c base ) / ( 2 B ) . Then, for any ε ∈ ( 0 , R max ) \varepsilon\in(0,R_{\max}) ε ∈ ( 0 , R m a x ) and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , there exist choices of α \alpha α , N N N , and K K K (Corollary D.1, with T T T specified by Eq. (8) ), such that Rebel-OnPolicy returns a policy π p o s t \pi_{\mathrm{post}} π post satisfying J ∗ − J ( π p o s t ) ≤ ε J^* - J(\pi_{\mathrm{post}}) \le\varepsilon J ∗ − J ( π post ) ≤ ε with probability at least 1 − δ 1-\delta 1 − δ , using
N s t e p ( ε , δ ) = O ~ ( c b a s e B 3 ε 3 ) and N s a m p l e ( ε , δ ) = O ~ ( c b a s e B 3 ε 3 ) . N_{\mathrm{step}}(\varepsilon,\delta) = \widetilde{O}\left(\frac{c_{\mathrm{base}}B^3}{\varepsilon^3}\right)
\qquad\text{and}\qquad
N_{\mathrm{sample}}(\varepsilon,\delta) = \widetilde{O}\left(\frac{c_{\mathrm{base}}B^3}{\varepsilon^3}\right). N step ( ε , δ ) = O ( ε 3 c base B 3 ) and N sample ( ε , δ ) = O ( ε 3 c base B 3 ) . Here, N s t e p = K T N_{\mathrm{step}} = KT N step = K T and N s a m p l e = 2 N T N_{\mathrm{sample}} = 2NT N sample = 2 N T . N s a m p l e N_{\mathrm{sample}} N sample depends only logarithmically on d d d and 1 / δ 1/\delta 1/ δ .
Rebel-OnPolicy is unbiased. The Rebel-OnPolicy objective directly controls the mean-squared error,
L t R e b e l − O n ( π ) = 2 MSE t ( π ) L_t^{\mathrm{Rebel-On}}(\pi) = 2\operatorname{MSE}_t(\pi) L t Rebel − On ( π ) = 2 MSE t ( π )
(Lemma L.1). It is therefore uniquely minimized by the target π η , t ∗ \pi_{\eta,t}^{*} π η , t ∗ and unbiased. Rebel-OnPolicy circumvents the baseline estimation in Kimi-style algorithms by taking the difference of residuals for paired responses, thereby canceling the shared normalization function b t ∗ ( x ) b_t^{*}(x) b t ∗ ( x ) . Rebel-OnPolicy is also a particularly clean instance of APMD under Assumption 2.1, because every empirical objective is a globally convex quadratic in θ \theta θ , on which π η , t ∗ \pi_{\eta,t}^{*} π η , t ∗ has zero loss (Lemma L.2).
Comparison: Pairwise algorithms with a fixed reference policy. Reference resets distinguish Rebel-OnPolicy from pairwise methods that use a fixed reference, such as DPO (Rafailov et al., [85 ] ), IPO (Gheshlaghi Azar et al., [31 ] ), and their online variants (Calandriello et al., [12 ] ). For example, online DPO draws response pairs from π t \pi_t π t but retains π b a s e \pi_{\mathrm{base}} π base in the denominator of the log-ratio. Consider its squared-loss formulation for observed rewards,
L t O n l i n e − D P O ( π θ ) : = E x ∼ μ , y , y ′ ∼ π t ( ⋅ ∣ x ) [ ( 1 η log π θ ( y ∣ x ) π b a s e ( y ∣ x ) − 1 η log π θ ( y ′ ∣ x ) π b a s e ( y ′ ∣ x ) − ( r ( x , y ) − r ( x , y ′ ) ) ) 2 ] . L_t^{\mathrm{Online-DPO}}(\pi_\theta) := \mathbb{E}_{x\sim\mu,\,y,y'\sim\pi_t(\cdot\mid x)}\left[\left(\frac{1}{\eta}\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}-\frac{1}{\eta}\log\frac{\pi_\theta(y'\mid x)}{\pi_{\mathrm{base}}(y'\mid x)}-(r(x,y)-r(x,y'))\right)^2\right]. L t Online − DPO ( π θ ) := E x ∼ μ , y , y ′ ∼ π t ( ⋅ ∣ x ) [ ( η 1 log π base ( y ∣ x ) π θ ( y ∣ x ) − η 1 log π base ( y ′ ∣ x ) π θ ( y ′ ∣ x ) − ( r ( x , y ) − r ( x , y ′ )) ) 2 ] .
This objective differs from L t R e b e l − O n L_t^{\mathrm{Rebel-On}} L t Rebel − On only in using π b a s e \pi_{\mathrm{base}} π base instead of π t \pi_t π t as the reference policy. Since it has the same solution as the KL-regularized objective with fixed reference π b a s e \pi_{\mathrm{base}} π base in Eq. (1) , it is prone to the same coverage collapse highlighted by our lower bound (Theorem 2.1).
Conclusion
We develop a theory of policy optimization in post-training, where the goal is to learn efficiently from a capable base model. We formalize this goal by asking which algorithms are coverage-preserving, with learning costs governed by the base model’s response coverage, a criterion that proves discerning for algorithm design. Basic methods such as REINFORCE can suffer coverage collapse, converging exponentially slowly despite a base model with good coverage. APMD, a meta-algorithm built around reference resets, addresses this failure and yields coverage-preserving algorithms. We apply it to establish global convergence of stylized PPO and GRPO variants and squared-loss methods, including Kimi-style algorithms.
Our theory provides insights that apply to policy optimization more generally.
Optimization benefits of regularization to the reference. Optimization in RL is challenging because the gradient signal depends on the learner’s distribution, whose coverage can deteriorate during training and thereby slow convergence to a given target. If we instead aim to optimize the policy in the proximity of a reference policy, coverage is controlled and local improvement becomes relatively inexpensive to realize. Updating the reference is what translates these local improvements into global optimality over successive resets. The reference policy therefore serves to anchor optimization, a role distinct from its use in importance weighting or batch reuse. In APMD, KL divergence defines the proximal region around the reference, but other interventions or divergences may play a similar role.
These optimization benefits also translate to improved sample efficiency, even if the batch is not reused in the inner loop. With reuse, algorithms adopting the APMD structure can improve more with a given batch in the inner loop, and may therefore require significantly fewer generation rounds. This is of particular importance in RL, where sampling can be the computational bottleneck (Qin et al., [83 ] ; He et al., [34 ] ; Hu et al., [36 ] ).
Implications for practical methodology. These insights also provide principles underpinning practical methodology. They suggest that the inner loop should be viewed as a local improvement step on the reference. This step is more stable the more strongly it is regularized to the reference and should be solved to a sufficient level to mitigate error compounding. In contrast, it is common to use 1–4 gradient steps for the inner loop (Shao et al., [96 ] ; Hu et al., [38 ] ; Liu et al., [62 ] ; Lambert et al., [52 ] ) and to either omit
regularization ([116 , 38 , 104 ] ) or treat it similarly to fixed-reference regularization ([81 , 52 ] ). Our theory also suggests that hyperparameters such as the clipping thresholds, number of inner steps, and batch size depend on the regularization parameter and should be tuned together.
Design choices within APMD. Finally, while we unify different post-training algorithms under APMD, we have also highlighted differences in the behavior of their objectives and in their sampling and optimization costs, which may be consequential in practice and merit empirical study. More broadly, APMD offers a starting point for designing and analyzing algorithms as new practical considerations arise.
Limitations and future work. Our end-to-end guarantees are limited to softmax linear policies, which we view as a useful but simplified model of learning with pretrained features. While we expect the core insights to be relevant more broadly, richer policy parameterizations may introduce challenges beyond the scope of our framework.
In addition, our theory focuses on the one-shot reasoning setting ([52 , 33 ] ) and treats each response as a single unit. In reality, responses are token sequences that vary in length and structure, and policy ratios over full responses can be large. These practical considerations can significantly affect training dynamics ([125 , 103 , 94 , 116 , 64 ] ), and how they interact with our theory is an interesting open question. Beyond the one-shot setting, our results serve as a basis for understanding policy optimization in multi-step reasoning, such as agentic tasks that involve many rounds of interaction with an environment ([49 , 15 ] ), and extending them is a natural next step.
Lastly, response coverage is only one basic notion of base model quality, and our central question addresses what we view as the first-order consideration. Ideally, post-training would do more than amplify existing behaviors ([124 , 119 ] ). Understanding when this is possible ([62 , 117 ] ) and which properties of the base model matter ([16 , 25 ] ) is an important direction for future work.
Statement on AI usage. The core results of this paper, including coverage collapse (Theorem 2.1) and coverage preservation under APMD (Theorem 3.1 ), were developed by the authors. AI models (GPT-5.5 and GPT-5.6) were used to greatly speed up the exploration of how to instantiate these results for specific algorithms. AI tools (Claude and GPT-6) were also used for writing assistance, including drafting, editing, and generating figures. The authors take full responsibility for all content in the final version of the paper, including its results, proofs, and writing.
Acknowledgments
We thank Alex Damian and Wen Sun for illuminating discussions that helped shape this paper. This work was enabled in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. Part of this research was performed while AH was visiting the Institute for Mathematical and Statistical Innovation (IMSI), which is supported by the National Science Foundation (Grant No. DMS-2425650), and while visiting the Simons Institute for the Theory of Computing. AH gratefully acknowledges their support.
Related Work
Convergence theory for policy gradient methods. A large body of work establishes global convergence of policy gradient methods, despite the nonconcavity of the expected reward. This includes results for tabular and softmax linear policies with exact gradients ([1 , 70 , 7 ] ), natural policy gradient (NPG) ([45 , 1 , 14 ] ), and policy mirror descent ([95 , 53 , 113 ] ), with extensions to log-linear and neural policies and to stochastic gradients ([118 , 13 , 109 , 60 , 51 , 73 , 74 , 3 , 55 ] ).
These guarantees differ in how they account for initialization. Some specialize to a uniform initial policy ([1 , 70 , 53 ] ), while others depend on properties of the policy iterates, such as the smallest probability assigned to an optimal action during training ([70 , 109 , 51 , 3 , 13 , 58 ] ). Such quantities need not remain favorable even when the base model has good coverage (Theorem 2.1). Exact mirror descent can instead depend on the initial policy through its KL divergence to an optimal policy ([113 ] ); related bounds hold for log-linear NPG with compatible function approximation ([118 ] ).
Inexact mirror-descent analyses also give sample-complexity guarantees. Xiao [113 ] uses a generative model to control value-estimation error, while Shani et al. [95 ] use uniform action sampling and a restart distribution that covers the states visited by an optimal policy. Lan [53 ] analyzes both generative sampling and single-trajectory sampling under exploration conditions, as well as approximate subproblem solutions. Yuan et al. [118 ] give complexity bounds for solving the compatible regression problems under concentrability assumptions and lower bounds on feature covariance or Fisher matrices. Our focus is the dependence on the base model’s response coverage. In the contextual-bandit setting, our KL and MSE criteria ensure that approximate updates preserve this coverage (Theorems 3.1 and 4.1), and our algorithm analyses bound the total samples and gradient steps needed to implement them (Corollaries 3.1, 4.1 and 4.2 ).
Analogy to first-order convex optimization. The coverage-preserving criterion is motivated by the dependence on initialization in classical convex optimization guarantees. For an L L L -smooth convex function f f f , Euclidean gradient descent with learning rate 1 / L 1/L 1/ L satisfies f ( x T ) − f ( x ∗ ) ≤ L ∥ x 0 − x ∗ ∥ 2 2 / ( 2 T ) f(x_T)-f(x_\ast) \le L\lVert x_0-x_\ast\rVert_2^2/(2T) f ( x T ) − f ( x ∗ ) ≤ L ∥ x 0 − x ∗ ∥ 2 2 / ( 2 T ) , where x ∗ x_\ast x ∗ is a minimizer ([11 ] ). This bound depends on the initial distance to an optimum rather than on uncontrolled properties of later iterates. The exact APMD guarantee in Eq. (5) has a similar form, with log C b a s e \log C_{\mathrm{base}} log C base measuring the quality of the initialization. Our question is whether implementable policy optimization methods retain a favorable dependence on the base model after accounting for both sampling and numerical optimization. Theorem 2.1 shows why this does not follow from good initialization alone.
Lower bounds for policy gradient methods. Mei et al. [69 ] show that softmax policy gradient with exact gradients can remain near a suboptimal deterministic policy for a number of gradient steps inversely proportional to the initial probability of the optimal action and the reward gap. This bound grows as the initial coverage worsens. In contrast, Theorem 2.1 starts with constant coverage (C b a s e ≤ 6 C_{\mathrm{base}} \le6 C base ≤ 6 ), but coverage deteriorates while reward increases. Attaining and maintaining ε \varepsilon ε -optimality then requires a number of gradient steps exponential in 1 / ε 1/\sqrt{\varepsilon} 1/ ε , with or without KL regularization to the base model. § 2.1 compares this result to other policy-gradient lower bounds.
Other results identify limitations of exploration from a weak base model. Foster et al. [25 ] prove coverage-dependent lower bounds on sampling-oracle complexity for KL-regularized alignment, distinguishing the number of sampling-oracle calls from the number of reward queries. Mousavi-Hosseini and Erdogdu [76 ] establish reward-query lower bounds for post-training linear autoregressive models, with the difficulty governed by the base model’s likelihood of correct responses. In reasoning-tree models, Liu et al. [63 ] prove an exponential query lower bound in the tree depth without hints and a polynomial guarantee with hints. Bu et al. [10 ] establish an exponential-to-polynomial sample-complexity separation between a direct gradient update and staged curriculum updates. Huang et al. [43 ] study a different barrier in compositional reasoning, showing that long horizons can produce exponentially small policy-gradient signals and that training on a mixture of difficulties can enable progress from easier to harder tasks. Our construction instead isolates a failure of exact-gradient optimization in a tabular bandit despite good initial coverage.
Coverage in post-training. The coverage of the base model governs the success of several post-training and inference-time methods. For Best-of-N sampling, the chance of finding a high-reward response depends on the probability that the base model assigns to such responses [9 , 6 , 40 ] . Huang et al. [39 ] formalize self-improvement as sharpening the model toward high-scoring responses and give coverage-dependent guarantees for supervised finetuning on selected generations, as used in rejection-sampling methods [121 , 98 , 22 , 114 ] . Chen et al. [16 ] explain why pretraining can provide this favorable initialization, showing that next-token prediction implicitly promotes coverage.
Reference-policy quality also enters statistical analyses of RL post-training. Zhao et al. [123 ] give coverage-dependent sample bounds for KL-regularized contextual bandits and RLHF, while the bounds for Q ♯ Q^{\sharp} Q ♯ depend on the variance of returns under the reference policy [126 ] . The analyses of Rebel and A⋆ ^{\star} ⋆ -PO likewise relate policy performance to regression accuracy or online-learning regret, with coverage conditions on the sampling or reference policy [27 , 8 ] . These guarantees use optimization or online-learning oracles, or assume a bound on regression error. Our algorithm analyses additionally bound the number of gradient steps needed to solve the subproblems and show how response coverage is preserved throughout training.
Trust regions and mirror descent. Limiting how far each update moves the policy is a classical principle in policy optimization. Conservative policy iteration [46 ] and TRPO [92 ] relate policy improvement to a surrogate objective and the distance between consecutive policies. PPO [93 ] replaces an explicit trust-region constraint with a clipped surrogate or an adaptive KL penalty. Neu et al. [77 ] interpret entropy-regularized methods, including an exact version of TRPO, through mirror descent and dual averaging, and Geist et al. [30 ] develop a theory of regularized MDPs that includes these connections.
The KL specialization of the tabular update analyzed by Shani et al. [95 ] maximizes expected advantage minus a KL penalty toward the current policy; subsequent work sharpens policy mirror descent rates [53 , 113 ] . Closest to our algorithms, on-policy mirror descent policy optimization (MDPO; Tomar et al. [105 ] ) approximately solves each KL-regularized subproblem using several gradient steps on an importance-weighted objective computed from current-policy samples. In contextual bandits, this resembles PPO-NoClip (Section 3.2). For PPO-NoClip, our analysis bounds the number of samples and gradient steps under softmax linear parameterization and proves coverage preservation. Lan [53 ] uses the name approximate policy mirror descent for a variant with a diminishing initial-policy KL perturbation of the reward and update. In our usage, “approximate” refers to inexact solutions of the KL-regularized subproblems; Lan also separately studies inexact subproblem solutions.
Reference resets in practice. KL regularization toward a fixed reference, typically the initial model, is commonly used to limit reward-model overoptimization [100 , 81 , 26 ] . Several training recipes instead update the reference during training. DeepSeekMath resets it when retraining the reward model [96 ] , while ProRL and DeepSeek-R1 use periodic resets [62 , 33 ] . EMA Policy Gradient uses an exponential moving average of the policy parameters as its reference and studies both its stability and empirical benefits [122 ] . Our analysis studies resetting the reference to the current policy at each outer iteration. It shows how this choice allows strong regularization to preserve coverage without stalling progress (Theorem 3.1).
Training without KL regularization. Successful reasoning pipelines such as DAPO and Open-Reasoner-Zero omit the KL penalty and use clipped policy-gradient objectives [116 , 38 ] . Other interventions promote exploration through forward-KL regularization [78 , 21 ] , entropy-aware token updates [19 , 110 ] , or protection of low-probability tokens [41 ] . Our clipping analysis identifies a statistical benefit of clipping. In PPO-Lite (Section 3.3), clipping controls the contribution of large importance weights to the empirical objective, allowing sample-efficient learning without a uniform bound on policy ratios. This need not keep consecutive policies close. Empirical studies find heavy-tailed gradients in PPO [29 ] and violations of the nominal ratio limits [24 ] ; alternatives include early stopping [101 ] and sequence-level clipping [125 ] . Our guarantee for PPO-Lite still uses KL regularization toward the current policy. Whether clipping alone can ensure coverage preservation remains open.
Squared-loss objectives. Kimi K1.5 and K2 regress the log-ratio between the new and current policies onto rewards after subtracting a baseline [50 a] . K1.5 considers both a sampled log-partition estimate and the sample mean reward, while K2 uses the sample mean. Sampled log-partition estimates also appear in A⋆ ^\star ⋆ -PO, OAPL, KARL, and SPO [8 , 89 , 15 , 18 ] . Xu et al. [115 ] characterize the implicit regularization induced by the mean baseline and report greater training stability than with a sampled log-partition baseline in their math-reasoning experiments. Our analysis in Appendices I to K quantifies how baseline choice affects objective bias, curvature, and the sample and gradient step complexities under softmax linear parameterization.
These methods also differ in the reference policy used in the log-ratio. Kimi K1.5 and K2 use the current policy, while OAPL and KARL periodically replace the reference with the learned policy. The original formulations of A⋆ ^\star ⋆ -PO and SPO instead retain a fixed reference. Related single-response regression methods DRO [88 ] and QRPO [68 ] also retain a fixed reference; DRO learns a value baseline, whereas QRPO uses quantile rewards to obtain an analytic partition function. Thus the use of online samples alone does not determine whether an algorithm implements reference resets.
Pairwise regression eliminates the baseline by fitting differences of log policy ratios to differences in rewards. Rebel [27 ] and its multi-turn extension REFUEL [28 ] take these ratios with respect to the current policy. Related preference losses in DPO and IPO [85 , 31 ] use a fixed reference. The online IPO formulation of Calandriello et al. [12 ] retains this reference in the loss while sampling fresh responses from the current policy. By contrast, DNO [90 ] uses the current policy as the reference for each batch of updates. This distinction separates fitting a target defined by a fixed reference from the successive policy-improvement targets analyzed in § 4.3 .
Technical Tools: Exact Mirror Descent and Gradient Methods
This section collects standard optimization results that our analysis builds on.
Appendix § 8 .§ 1 includes standard gradient descent convergence guarantees for convex and smooth objectives.
Appendix § 8 .§ 2 proves convergence of exact policy gradient ascent for the KL-regularized objective under Assumption 2.1. Although the expected reward J ( π θ ) J(\pi_\theta) J ( π θ ) is nonconcave in θ \theta θ , forms of local concavity (Lemma B.3) and local smoothness (Lemma B.4 ) hold. With these properties, the rest of the proof closely follows the gradient descent analysis.
Appendix § 8 .§ 3 establishes convergence of exact policy mirror descent in RL, which provides the basis for the Approximate Policy Mirror Descent framework results in § 3.1 .
Appendix § 8 .§ 4 collects basic properties of softmax linear policies.
Notation. Throughout this section, we use α > 0 \alpha> 0 α > 0 to denote the learning rate, and we always use exact gradients.
Exact Gradient Descent
This section develops standard gradient descent analysis for smooth and convex functions. In this subsection, X \mathcal{X} X denotes the optimization domain. We consider the optimization problem of finding
x ⋆ ∈ arg min x ∈ X f ( x ) , x_\star\in\arg\min_{x \in\mathcal{X}} f(x), x ⋆ ∈ arg x ∈ X min f ( x ) ,
and provide convergence analysis for population gradient descent with a fixed learning rate α > 0 \alpha> 0 α > 0 , when the function f : X → R f : \mathcal{X} \to\mathbb{R} f : X → R is convex and smooth. For the remainder of this subsection, we assume that X ⊆ R d \mathcal{X} \subseteq\mathbb{R}^d X ⊆ R d is a convex set with x ⋆ ∈ X x_\star\in\mathcal{X} x ⋆ ∈ X , and f f f is differentiable, convex, and smooth.
Definition B.1 (Convex function). A function f : X → R f : \mathcal{X} \to\mathbb{R} f : X → R is convex if, for every x , y ∈ X x,y \in\mathcal{X} x , y ∈ X and λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , it holds that f ( λ x + ( 1 − λ ) y ) ≤ λ f ( x ) + ( 1 − λ ) f ( y ) f(\lambda x + (1-\lambda)y) \leq\lambda f(x) + (1-\lambda)f(y) f ( λ x + ( 1 − λ ) y ) ≤ λ f ( x ) + ( 1 − λ ) f ( y ) . When f f f is differentiable, this is equivalent to
f ( y ) ≥ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ , x , y ∈ X . f(y) \geq f(x) + \langle\nabla f(x), y - x \rangle, \qquad x,y \in\mathcal{X}. f ( y ) ≥ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ , x , y ∈ X .
Definition B.2 (Smooth function). A differentiable function f : X → R f : \mathcal{X} \to\mathbb{R} f : X → R is L L L -smooth with respect to the Euclidean norm if L > 0 L > 0 L > 0 and
∥ ∇ f ( x ) − ∇ f ( y ) ∥ 2 ≤ L ∥ x − y ∥ 2 , x , y ∈ X . \lVert\nabla f(x) - \nabla f(y) \rVert_2 \le L\lVert x-y\rVert_2, \qquad x,y \in\mathcal{X}. ∥ ∇ f ( x ) − ∇ f ( y ) ∥ 2 ≤ L ∥ x − y ∥ 2 , x , y ∈ X . Equivalently, L L L -smoothness gives the quadratic upper bound
f ( y ) ≤ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ + L 2 ∥ y − x ∥ 2 2 , x , y ∈ X . f(y) \le f(x) + \langle\nabla f(x), y-x \rangle+ \frac{L}{2}\lVert y-x\rVert_2^2, \qquad x,y \in\mathcal{X}. f ( y ) ≤ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ + 2 L ∥ y − x ∥ 2 2 , x , y ∈ X .
Population gradient descent. Given an initial point x 0 ∈ X x_0 \in\mathcal{X} x 0 ∈ X and a learning rate α > 0 \alpha> 0 α > 0 , run population gradient descent for T ≥ 1 T \ge1 T ≥ 1 gradient steps using the update
x t + 1 = x t − α ∇ f ( x t ) , t = 0 , … , T − 1. (22) x_{t+1} = x_t - \alpha\nabla f(x_t), \qquad t = 0,\ldots,T-1. \tag*{(22)} x t + 1 = x t − α ∇ f ( x t ) , t = 0 , … , T − 1. (22)
Assume that the iterates remain in X \mathcal{X} X .
Theorem B.1 (Gradient descent convergence). Suppose that f f f is convex and L L L -smooth. Fix T ≥ 1 T \ge1 T ≥ 1 and a learning rate 0 < α ≤ 1 / L 0 < \alpha\le1/L 0 < α ≤ 1/ L . Then the iterates in Eq. (22) satisfy
f ( x T ) − f ( x ⋆ ) ≤ ∥ x 0 − x ⋆ ∥ 2 2 2 α T . f(x_T) - f(x_\star) \le\frac{\lVert x_0-x_\star\rVert_2^2}{2\alpha T}. f ( x T ) − f ( x ⋆ ) ≤ 2 α T ∥ x 0 − x ⋆ ∥ 2 2 . In particular, choosing α = 1 / L \alpha= 1/L α = 1/ L gives f ( x T ) − f ( x ⋆ ) ≤ L ∥ x 0 − x ⋆ ∥ 2 2 2 T f(x_T) - f(x_\star) \le\frac{L\lVert x_0-x_\star\rVert_2^2}{2T} f ( x T ) − f ( x ⋆ ) ≤ 2 T L ∥ x 0 − x ⋆ ∥ 2 2 .
The central inequality to the proof is the following one-step progress bound,
f ( x t ) − f ( x ⋆ ) ≤ ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 2 α + f ( x t ) − f ( x t + 1 ) , f(x_t) - f(x_\star) \le\frac{\lVert x_t-x_\star\rVert_2^2-\lVert x_{t+1}-x_\star\rVert_2^2}{2\alpha} + f(x_t) - f(x_{t+1}), f ( x t ) − f ( x ⋆ ) ≤ 2 α ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 + f ( x t ) − f ( x t + 1 ) ,
equivalent to Eq. (24) when rearranged. In words, an update to x t x_t x t must either cause the objective to decrease by at least half the current suboptimality, or the squared Euclidean distance to decrease by at least α \alpha α times the current suboptimality. The inequality combines convexity, which states that parameter distance decreases after taking an α \alpha α -step in the − ∇ f ( x t ) -\nabla f(x_t) − ∇ f ( x t ) direction (Eq. (23) ), subject to a curvature correction that grows with ∥ ∇ f ( x t ) ∥ 2 2 \lVert\nabla f(x_t)\rVert_2^2 ∥ ∇ f ( x t ) ∥ 2 2 , and smoothness (Eq. (25) ), which guarantees the existence of a learning rate α \alpha α that decreases the objective value proportionally to ∥ ∇ f ( x t ) ∥ 2 2 \lVert\nabla f(x_t)\rVert_2^2 ∥ ∇ f ( x t ) ∥ 2 2 .
Proof of Theorem B.1. Write g t : = ∇ f ( x t ) g_t := \nabla f(x_t) g t := ∇ f ( x t ) . Summing the second inequality in Lemma B.1 for all 0 ≤ t < T 0 \le t < T 0 ≤ t < T gives
∑ t = 0 T − 1 ( f ( x t + 1 ) − f ( x ⋆ ) ) ≤ ∥ x 0 − x ⋆ ∥ 2 2 − ∥ x T − x ⋆ ∥ 2 2 2 α ≤ ∥ x 0 − x ⋆ ∥ 2 2 2 α . \sum_{t=0}^{T-1} \left(f(x_{t+1}) - f(x_\star)\right) \le\frac{\lVert x_0-x_\star\rVert_2^2-\lVert x_T-x_\star\rVert_2^2}{2\alpha}
\le\frac{\lVert x_0-x_\star\rVert_2^2}{2\alpha}. t = 0 ∑ T − 1 ( f ( x t + 1 ) − f ( x ⋆ ) ) ≤ 2 α ∥ x 0 − x ⋆ ∥ 2 2 − ∥ x T − x ⋆ ∥ 2 2 ≤ 2 α ∥ x 0 − x ⋆ ∥ 2 2 .
Moreover, Lemma B.2 and α ≤ 1 / L \alpha\le1/L α ≤ 1/ L imply f ( x t + 1 ) ≤ f ( x t ) f(x_{t+1}) \le f(x_t) f ( x t + 1 ) ≤ f ( x t ) for every 0 ≤ t < T 0 \le t < T 0 ≤ t < T . Therefore,
T ( f ( x T ) − f ( x ⋆ ) ) ≤ ∑ t = 0 T − 1 ( f ( x t + 1 ) − f ( x ⋆ ) ) ≤ ∥ x 0 − x ⋆ ∥ 2 2 2 α . T\left(f(x_T)-f(x_\star)\right) \le\sum_{t=0}^{T-1}\left(f(x_{t+1})-f(x_\star)\right) \le\frac{\lVert x_0-x_\star\rVert_2^2}{2\alpha}. T ( f ( x T ) − f ( x ⋆ ) ) ≤ t = 0 ∑ T − 1 ( f ( x t + 1 ) − f ( x ⋆ ) ) ≤ 2 α ∥ x 0 − x ⋆ ∥ 2 2 .
Dividing by T T T proves the first claim. The second follows by setting α = 1 / L \alpha= 1/L α = 1/ L . □ \square □
Lemma B.1 (One-step progress). Suppose that f : X → R f:\mathcal{X}\to\mathbb{R} f : X → R is convex. Then the iterates of Eq. (22) for all 0 ≤ t < T 0\le t<T 0 ≤ t < T satisfy,
f ( x t ) − f ( x ⋆ ) ≤ ⟨ g t , x t − x ⋆ ⟩ = ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 2 α + α 2 ∥ ∇ f ( x t ) ∥ 2 2 . (23) f(x_t)-f(x_\star)\le\langle g_t,x_t-x_\star\rangle=\frac{\|x_t-x_\star\|_2^2-\|x_{t+1}-x_\star\|_2^2}{2\alpha}+\frac{\alpha}{2}\|\nabla f(x_t)\|_2^2. \tag*{(23)} f ( x t ) − f ( x ⋆ ) ≤ ⟨ g t , x t − x ⋆ ⟩ = 2 α ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 + 2 α ∥∇ f ( x t ) ∥ 2 2 . (23) If, additionally, f f f is L L L -smooth and α ≤ 1 L \alpha\le\frac{1}{L} α ≤ L 1 ,
f ( x t + 1 ) − f ( x ⋆ ) ≤ ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 2 α . (24) f(x_{t+1})-f(x_\star)\le\frac{\|x_t-x_\star\|_2^2-\|x_{t+1}-x_\star\|_2^2}{2\alpha}. \tag*{(24)} f ( x t + 1 ) − f ( x ⋆ ) ≤ 2 α ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 . (24)
Proof of Lemma B.1. Write g t : = ∇ f ( x t ) g_t:=\nabla f(x_t) g t := ∇ f ( x t ) . Expanding the update in Eq. (22) gives
∥ x t + 1 − x ⋆ ∥ 2 2 = ∥ x t − x ⋆ − α g t ∥ 2 2 = ∥ x t − x ⋆ ∥ 2 2 − 2 α ⟨ g t , x t − x ⋆ ⟩ + α 2 ∥ g t ∥ 2 2 . \begin{aligned}
\|x_{t+1}-x_\star\|_2^2&=\|x_t-x_\star-\alpha g_t\|_2^2\\
&=\|x_t-x_\star\|_2^2-2\alpha\langle g_t,x_t-x_\star\rangle+\alpha^2\|g_t\|_2^2.
\end{aligned} ∥ x t + 1 − x ⋆ ∥ 2 2 = ∥ x t − x ⋆ − α g t ∥ 2 2 = ∥ x t − x ⋆ ∥ 2 2 − 2 α ⟨ g t , x t − x ⋆ ⟩ + α 2 ∥ g t ∥ 2 2 .
Rearranging this identity and using convexity yields
f ( x t ) − f ( x ⋆ ) ≤ ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 2 α + α 2 ∥ g t ∥ 2 2 , f(x_t)-f(x_\star)\le\frac{\|x_t-x_\star\|_2^2-\|x_{t+1}-x_\star\|_2^2}{2\alpha}+\frac{\alpha}{2}\|g_t\|_2^2, f ( x t ) − f ( x ⋆ ) ≤ 2 α ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 + 2 α ∥ g t ∥ 2 2 ,
which proves the first claim.
Now suppose additionally that f f f is L L L -smooth and α ≤ 1 / L \alpha\le1/L α ≤ 1/ L . By Lemma B.2 and the preceding bound,
f ( x t + 1 ) − f ( x ⋆ ) = f ( x t ) − f ( x ⋆ ) + f ( x t + 1 ) − f ( x t ) ≤ ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 2 α + α 2 ∥ g t ∥ 2 2 − α ( 1 − L α 2 ) ∥ g t ∥ 2 2 = ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 2 α − α 2 ( 1 − L α ) ∥ g t ∥ 2 2 ≤ ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 2 α , \begin{aligned}
f(x_{t+1})-f(x_\star)&=f(x_t)-f(x_\star)+f(x_{t+1})-f(x_t)\\
&\le\frac{\|x_t-x_\star\|_2^2-\|x_{t+1}-x_\star\|_2^2}{2\alpha}+\frac{\alpha}{2}\|g_t\|_2^2-\alpha\left(1-\frac{L\alpha}{2}\right)\|g_t\|_2^2\\
&=\frac{\|x_t-x_\star\|_2^2-\|x_{t+1}-x_\star\|_2^2}{2\alpha}-\frac{\alpha}{2}(1-L\alpha)\|g_t\|_2^2\\
&\le\frac{\|x_t-x_\star\|_2^2-\|x_{t+1}-x_\star\|_2^2}{2\alpha},
\end{aligned} f ( x t + 1 ) − f ( x ⋆ ) = f ( x t ) − f ( x ⋆ ) + f ( x t + 1 ) − f ( x t ) ≤ 2 α ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 + 2 α ∥ g t ∥ 2 2 − α ( 1 − 2 Lα ) ∥ g t ∥ 2 2 = 2 α ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 − 2 α ( 1 − Lα ) ∥ g t ∥ 2 2 ≤ 2 α ∥ x t − x ⋆ ∥ 2 2 − ∥ x t + 1 − x ⋆ ∥ 2 2 ,
where the last inequality uses α ≤ 1 / L \alpha\le1/L α ≤ 1/ L .
Lemma B.2 (Descent lemma). Suppose that f : X → R f:\mathcal{X}\to\mathbb{R} f : X → R is L L L -smooth. Then for all 0 ≤ t < T 0\le t<T 0 ≤ t < T , the iterates of Eq. (22) satisfy
f ( x t + 1 ) − f ( x t ) ≤ − α ( 1 − L α 2 ) ∥ ∇ f ( x t ) ∥ 2 2 . (25) f(x_{t+1})-f(x_t)\le-\alpha\left(1-\frac{L\alpha}{2}\right)\|\nabla f(x_t)\|_2^2. \tag*{(25)} f ( x t + 1 ) − f ( x t ) ≤ − α ( 1 − 2 Lα ) ∥∇ f ( x t ) ∥ 2 2 . (25)
Proof of Lemma B.2. Applying L L L -smoothness with x = x t x=x_t x = x t and y = x t + 1 y=x_{t+1} y = x t + 1 gives
f ( x t + 1 ) − f ( x t ) ≤ ⟨ ∇ f ( x t ) , x t + 1 − x t ⟩ + L 2 ∥ x t + 1 − x t ∥ 2 2 . f(x_{t+1})-f(x_t)\le\langle\nabla f(x_t),x_{t+1}-x_t\rangle+\frac{L}{2}\|x_{t+1}-x_t\|_2^2. f ( x t + 1 ) − f ( x t ) ≤ ⟨ ∇ f ( x t ) , x t + 1 − x t ⟩ + 2 L ∥ x t + 1 − x t ∥ 2 2 .
Substituting x t + 1 − x t = − α ∇ f ( x t ) x_{t+1}-x_t=-\alpha\nabla f(x_t) x t + 1 − x t = − α ∇ f ( x t ) from Eq. (22) yields
f ( x t + 1 ) − f ( x t ) ≤ − α ∥ ∇ f ( x t ) ∥ 2 2 + L α 2 2 ∥ ∇ f ( x t ) ∥ 2 2 = − α ( 1 − L α 2 ) ∥ ∇ f ( x t ) ∥ 2 2 . \begin{aligned}
f(x_{t+1})-f(x_t)&\le-\alpha\|\nabla f(x_t)\|_2^2+\frac{L\alpha^2}{2}\|\nabla f(x_t)\|_2^2\\
&=-\alpha\left(1-\frac{L\alpha}{2}\right)\|\nabla f(x_t)\|_2^2.
\end{aligned} f ( x t + 1 ) − f ( x t ) ≤ − α ∥∇ f ( x t ) ∥ 2 2 + 2 L α 2 ∥∇ f ( x t ) ∥ 2 2 = − α ( 1 − 2 Lα ) ∥∇ f ( x t ) ∥ 2 2 .
Exact Policy Gradient Ascent
We analyze the convergence of gradient ascent on the regularized policy gradient objective over softmax linear policies. Assume we are in the (non-contextual) bandit setting and Assumption 2.1 holds, and define the feature diameter
D ϕ : = max y , y ′ ∈ Y ∥ ϕ ( y ) − ϕ ( y ′ ) ∥ 2 . D_{\phi} := \max_{y,y'\in\mathcal{Y}} \lVert\phi(y)-\phi(y')\rVert_{2}. D ϕ := y , y ′ ∈ Y max ∥ ϕ ( y ) − ϕ ( y ′ ) ∥ 2 .
Fix a regularization parameter η > 0 \eta> 0 η > 0 and a reference policy π r e f = π θ r e f ∈ Π Θ \pi_{\mathrm{ref}}=\pi_{\theta_{\mathrm{ref}}}\in\Pi_{\Theta} π ref = π θ ref ∈ Π Θ , and define
J η ( π ) : = E y ∼ π [ r ( y ) ] − 1 η D K L ( π ∥ π r e f ) . J_{\eta}(\pi) := \mathbb{E}_{y\sim\pi}[r(y)]-\frac{1}{\eta}D_{\mathrm{KL}}(\pi\mathbin\Vert\pi_{\mathrm{ref}}). J η ( π ) := E y ∼ π [ r ( y )] − η 1 D KL ( π ∥ π ref ) .
Here π r e f \pi_{\mathrm{ref}} π ref is a general reference policy, so J η J_{\eta} J η specializes to the fixed-reference objective in Eq. (1) when π r e f = π b a s e \pi_{\mathrm{ref}}=\pi_{\mathrm{base}} π ref = π base and to the objective J η , t J_{\eta,t} J η , t in Eq. (9) when π r e f \pi_{\mathrm{ref}} π ref is the most recent policy. We consider the optimization problem of finding
π η ⋆ : = arg max π θ ∈ Π Θ J η ( π θ ) . \pi_{\eta}^{\star}:=\mathop{\arg\max}_{\pi_{\theta}\in\Pi_{\Theta}}J_{\eta}(\pi_{\theta}). π η ⋆ := arg max π θ ∈ Π Θ J η ( π θ ) .
Under Assumption 2.1, π η ⋆ = π θ η ⋆ \pi_{\eta}^{\star}=\pi_{\theta_{\eta}^{\star}} π η ⋆ = π θ η ⋆ with θ η ⋆ : = θ r e f + η θ ⋆ \theta_{\eta}^{\star}:=\theta_{\mathrm{ref}}+\eta\theta^{\star} θ η ⋆ := θ ref + η θ ⋆ (Lemma B.11).
Policy gradient. Given a learning rate α > 0 \alpha>0 α > 0 and an integer T ≥ 1 T\geq1 T ≥ 1 , initialize θ 0 = θ r e f \theta_{0}=\theta_{\mathrm{ref}} θ 0 = θ ref and run
θ t + 1 = θ t + α ∇ J η ( π θ t ) , t = 0 , … , T − 1. (26) \theta_{t+1}=\theta_{t}+\alpha\nabla J_{\eta}(\pi_{\theta_{t}}),\qquad t=0,\ldots,T-1. \tag*{(26)} θ t + 1 = θ t + α ∇ J η ( π θ t ) , t = 0 , … , T − 1. (26)
The algorithm returns π θ T \pi_{\theta_{T}} π θ T . Defining Σ θ : = E π θ [ ϕ ~ θ ( y ) ϕ ~ θ ( y ) T ] \Sigma_{\theta}:=\mathbb{E}_{\pi_{\theta}}\left[\widetilde{\phi}_{\theta}(y)\widetilde{\phi}_{\theta}(y)^{\mathsf{T}}\right] Σ θ := E π θ [ ϕ θ ( y ) ϕ θ ( y ) T ] , where ϕ ~ θ ( y ) : = ϕ ( y ) − E y ′ ∼ π θ [ ϕ ( y ′ ) ] \widetilde{\phi}_{\theta}(y):=\phi(y)-\mathbb{E}_{y'\sim\pi_{\theta}}[\phi(y')] ϕ θ ( y ) := ϕ ( y ) − E y ′ ∼ π θ [ ϕ ( y ′ )] , the gradient is given by
∇ J η ( π θ ) = 1 η Σ θ ( θ η ⋆ − θ ) . \nabla J_{\eta}(\pi_{\theta})=\frac{1}{\eta}\Sigma_{\theta}(\theta_{\eta}^{\star}-\theta). ∇ J η ( π θ ) = η 1 Σ θ ( θ η ⋆ − θ ) .
The objective J η J_{\eta} J η is not globally concave, and its global smoothness constant can grow as the minimum response probability under the reference policy decreases. However, if the parameters remain in the ball
B R : = { θ ∈ R d : ∥ θ − θ ⋆ ∥ 2 ≤ R } , \mathcal{B}_{R}:=\left\{\theta\in\mathbb{R}^{d}:\lVert\theta-\theta^{\star}\rVert_{2}\leq R\right\}, B R := { θ ∈ R d : ∥ θ − θ ⋆ ∥ 2 ≤ R } ,
then J η J_{\eta} J η satisfies local notions of concavity (Lemma B.3) and smoothness (Lemma B.4 ). The guarantees in both these inequalities become looser as R R R grows.
Lemma B.3 (Single-policy concavity). For any θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d ,
J η ( π η ⋆ ) − J η ( π θ ) ≤ 1 2 ∥ π η ⋆ π θ ∥ ∞ ⟨ ∇ J η ( π θ ) , θ η ⋆ − θ ⟩ ⏟ = 1 η ∥ θ η ⋆ − θ ∥ Σ θ 2 . J_{\eta}(\pi_{\eta}^{\star})-J_{\eta}(\pi_{\theta})\leq\frac{1}{2}\left\lVert\frac{\pi_{\eta}^{\star}}{\pi_{\theta}}\right\rVert_{\infty}\underbrace{\left\langle\nabla J_{\eta}(\pi_{\theta}),\theta_{\eta}^{\star}-\theta\right\rangle}_{=\frac{1}{\eta}\lVert\theta_{\eta}^{\star}-\theta\rVert_{\Sigma_{\theta}}^{2}}. J η ( π η ⋆ ) − J η ( π θ ) ≤ 2 1 π θ π η ⋆ ∞ = η 1 ∥ θ η ⋆ − θ ∥ Σ θ 2 ⟨ ∇ J η ( π θ ) , θ η ⋆ − θ ⟩ . Moreover, for R > 0 R>0 R > 0 define
W R : = e D ϕ R . W_{R}:=e^{D_{\phi}R}. W R := e D ϕ R . If θ ∈ B R \theta\in\mathcal{B}_{R} θ ∈ B R , then
J η ( π η ⋆ ) − J η ( π θ ) ≤ W R 2 ⟨ ∇ J η ( π θ ) , θ η ⋆ − θ ⟩ . J_{\eta}(\pi_{\eta}^{\star})-J_{\eta}(\pi_{\theta})\leq\frac{W_{R}}{2}\left\langle\nabla J_{\eta}(\pi_{\theta}),\theta_{\eta}^{\star}-\theta\right\rangle. J η ( π η ⋆ ) − J η ( π θ ) ≤ 2 W R ⟨ ∇ J η ( π θ ) , θ η ⋆ − θ ⟩ .
Lemma B.4 (Local smoothness). Fix R > 0 R>0 R > 0 and define
L R : = D ϕ 2 η ( 1 + D ϕ R ) . L_{R}:=\frac{D_{\phi}^{2}}{\eta}(1+D_{\phi}R). L R := η D ϕ 2 ( 1 + D ϕ R ) . Then θ ↦ J η ( π θ ) \theta\mapsto J_{\eta}(\pi_{\theta}) θ ↦ J η ( π θ ) is L R L_{R} L R -smooth on B R \mathcal{B}_{R} B R . In particular, for any θ , θ ′ ∈ B R \theta,\theta'\in\mathcal{B}_{R} θ , θ ′ ∈ B R ,
J η ( π θ ′ ) ≥ J η ( π θ ) + ⟨ ∇ J η ( π θ ) , θ ′ − θ ⟩ − L R 2 ∥ θ ′ − θ ∥ 2 2 . J_{\eta}(\pi_{\theta'})\geq J_{\eta}(\pi_{\theta})+\langle\nabla J_{\eta}(\pi_{\theta}),\theta'-\theta\rangle-\frac{L_{R}}{2}\lVert\theta'-\theta\rVert_{2}^{2}. J η ( π θ ′ ) ≥ J η ( π θ ) + ⟨ ∇ J η ( π θ ) , θ ′ − θ ⟩ − 2 L R ∥ θ ′ − θ ∥ 2 2 .
With these properties, the proof follows using essentially the same steps as in the convex analysis of Theorem B.1 (with concavity in place of convexity, since we maximize J η J_\eta J η ). It is crucial that the iterates remain inside the ball, and there this follows purely from the population policy gradient update for softmax linear policies, not the local concavity. (Linearity here gives us extra structure; for general functions concavity is needed for this to hold.) The factors L R L_R L R and W R W_R W R carry into the final guarantee, and both grow with R R R . Because R R R is an analytical construct we can set R = η ∥ θ ⋆ ∥ 2 R = \eta\|\theta_\star\|_2 R = η ∥ θ ⋆ ∥ 2 .
Theorem B.2 (Policy gradient convergence). Fix R ≥ η ∥ θ ⋆ ∥ 2 R \ge\eta\|\theta_\star\|_2 R ≥ η ∥ θ ⋆ ∥ 2 and T ≥ 1 T \ge1 T ≥ 1 , and let L R L_R L R and W R W_R W R be defined as in Lemmas B.3 and B.4 , respectively. Let { θ t } t = 0 T \{\theta_t\}_{t=0}^T { θ t } t = 0 T follow Eq. (26) with learning rate 0 < α ≤ 1 L R 0 < \alpha\le\frac{1}{L_R} 0 < α ≤ L R 1 . Then the iterates remain localized and the objective is nondecreasing, that is, for every 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T ,
∥ θ t − θ η ⋆ ∥ 2 ≤ η ∥ θ ⋆ ∥ 2 ≤ R , and for every 0 ≤ t < T , J η ( π θ t + 1 ) ≥ J η ( π θ t ) . \|\theta_t-\theta_\eta^\star\|_2 \le\eta\|\theta_\star\|_2 \le R,\qquad\text{and for every }0 \le t<T,\qquad J_\eta(\pi_{\theta_{t+1}}) \ge J_\eta(\pi_{\theta_t}). ∥ θ t − θ η ⋆ ∥ 2 ≤ η ∥ θ ⋆ ∥ 2 ≤ R , and for every 0 ≤ t < T , J η ( π θ t + 1 ) ≥ J η ( π θ t ) . Moreover, the final iterate satisfies
J η ( π η ⋆ ) − J η ( π θ T ) ≤ e D ϕ R η 2 ∥ θ ⋆ ∥ 2 2 2 α ( 2 − α D ϕ 2 / η ) T . J_\eta(\pi_{\eta}^\star)-J_\eta(\pi_{\theta_T}) \le\frac{e^{D_\phi R}\eta^2\|\theta_\star\|_2^2}{2\alpha\left(2-\alpha D_\phi^2/\eta\right)T}. J η ( π η ⋆ ) − J η ( π θ T ) ≤ 2 α ( 2 − α D ϕ 2 / η ) T e D ϕ R η 2 ∥ θ ⋆ ∥ 2 2 . In particular, choosing α = 1 L R \alpha=\frac{1}{L_R} α = L R 1 gives
J η ( π η ⋆ ) − J η ( π θ T ) ≤ W R L R η 2 ∥ θ ⋆ ∥ 2 2 2 T . J_\eta(\pi_{\eta}^\star)-J_\eta(\pi_{\theta_T}) \le\frac{W_RL_R\eta^2\|\theta_\star\|_2^2}{2T}. J η ( π η ⋆ ) − J η ( π θ T ) ≤ 2 T W R L R η 2 ∥ θ ⋆ ∥ 2 2 . Setting R = η B ⋆ R=\eta B_\star R = η B ⋆ , where B ⋆ = ∥ θ ⋆ ∥ 2 B_\star=\|\theta_\star\|_2 B ⋆ = ∥ θ ⋆ ∥ 2 , and using D ⋆ : = D ϕ B ⋆ D_\star:=D_\phi B_\star D ⋆ := D ϕ B ⋆ ,
J η ( π η ⋆ ) − J η ( π θ T ) ≤ η e η D ⋆ D ⋆ 2 ( 1 + η D ⋆ ) 2 T . J_\eta(\pi_{\eta}^\star)-J_\eta(\pi_{\theta_T}) \le\frac{\eta e^{\eta D_\star}D_\star^2(1+\eta D_\star)}{2T}. J η ( π η ⋆ ) − J η ( π θ T ) ≤ 2 T η e η D ⋆ D ⋆ 2 ( 1 + η D ⋆ ) .
Proof of Theorem B.2. Since L R = D ϕ 2 η ( 1 + D ϕ R ) L_R=\frac{D_\phi^2}{\eta}(1+D_\phi R) L R = η D ϕ 2 ( 1 + D ϕ R ) , the condition on α \alpha α implies that 0 < α D ϕ 2 / η < 2 0<\alpha D_\phi^2/\eta<2 0 < α D ϕ 2 / η < 2 , so the RHS of the claim is well-behaved.
We first prove localization by induction. The base case follows from θ 0 = θ r e f \theta_0=\theta_{\mathrm{ref}} θ 0 = θ ref , since θ η ⋆ − θ r e f = η θ ⋆ \theta_\eta^\star-\theta_{\mathrm{ref}}=\eta\theta_\star θ η ⋆ − θ ref = η θ ⋆ . Suppose that ∥ θ t − θ η ⋆ ∥ 2 ≤ η ∥ θ ⋆ ∥ 2 \|\theta_t-\theta_\eta^\star\|_2\le\eta\|\theta_\star\|_2 ∥ θ t − θ η ⋆ ∥ 2 ≤ η ∥ θ ⋆ ∥ 2 . By the gradient identity and Σ θ t ⪰ 0 \Sigma_{\theta_t}\succeq0 Σ θ t ⪰ 0 ,
⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ = 1 η ∥ θ η ⋆ − θ t ∥ Σ θ t 2 ≥ 0. \left\langle\nabla J_\eta(\pi_{\theta_t}),\theta_\eta^\star-\theta_t\right\rangle=\frac{1}{\eta}\|\theta_\eta^\star-\theta_t\|_{\Sigma_{\theta_t}}^2\ge0. ⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ = η 1 ∥ θ η ⋆ − θ t ∥ Σ θ t 2 ≥ 0.
Applying Lemma B.5 and using the positivity of α ( 2 − α D ϕ 2 / η ) \alpha\left(2-\alpha D_\phi^2/\eta\right) α ( 2 − α D ϕ 2 / η ) therefore gives
∥ θ t + 1 − θ η ⋆ ∥ 2 2 ≤ ∥ θ t − θ η ⋆ ∥ 2 2 ≤ η 2 ∥ θ ⋆ ∥ 2 2 . \|\theta_{t+1}-\theta_\eta^\star\|_2^2\le\|\theta_t-\theta_\eta^\star\|_2^2\le\eta^2\|\theta_\star\|_2^2. ∥ θ t + 1 − θ η ⋆ ∥ 2 2 ≤ ∥ θ t − θ η ⋆ ∥ 2 2 ≤ η 2 ∥ θ ⋆ ∥ 2 2 .
We can now apply Lemma B.6 at every iteration. Since α L R ≤ 1 \alpha L_R\le1 α L R ≤ 1 , it yields
J η ( π θ t + 1 ) − J η ( π θ t ) ≥ α ( 1 − α L R 2 ) ∥ ∇ J η ( π θ t ) ∥ 2 2 ≥ α 2 ∥ ∇ J η ( π θ t ) ∥ 2 2 ≥ 0. \begin{aligned}
J_\eta(\pi_{\theta_{t+1}})-J_\eta(\pi_{\theta_t})\ge\alpha\left(1-\frac{\alpha L_R}{2}\right)\|\nabla J_\eta(\pi_{\theta_t})\|_2^2 \\
&\ge\frac{\alpha}{2}\|\nabla J_\eta(\pi_{\theta_t})\|_2^2\ge0.
\end{aligned} J η ( π θ t + 1 ) − J η ( π θ t ) ≥ α ( 1 − 2 α L R ) ∥∇ J η ( π θ t ) ∥ 2 2 ≥ 2 α ∥∇ J η ( π θ t ) ∥ 2 2 ≥ 0.
This proves monotonicity. Finally, summing Lemma B.5 for every 0 ≤ t < T 0\le t<T 0 ≤ t < T and telescoping gives
∑ t = 0 T − 1 ( J η ( π η ⋆ ) − J η ( π θ t ) ) ≤ W R 2 α ( 2 − α D ϕ 2 / η ) ∑ t = 0 T − 1 ( ∥ θ t − θ η ⋆ ∥ 2 2 − ∥ θ t + 1 − θ η ⋆ ∥ 2 2 ) ≤ W R η 2 ∥ θ ⋆ ∥ 2 2 2 α ( 2 − α D ϕ 2 / η ) . \begin{aligned}
\sum_{t=0}^{T-1}\left(J_\eta(\pi_{\eta}^\star)-J_\eta(\pi_{\theta_t})\right)\le\frac{W_R}{2\alpha\left(2-\alpha D_\phi^2/\eta\right)}\sum_{t=0}^{T-1}\left(\|\theta_t-\theta_\eta^\star\|_2^2-\|\theta_{t+1}-\theta_\eta^\star\|_2^2\right) \\
&\le\frac{W_R\eta^2\|\theta_\star\|_2^2}{2\alpha\left(2-\alpha D_\phi^2/\eta\right)}.
\end{aligned} t = 0 ∑ T − 1 ( J η ( π η ⋆ ) − J η ( π θ t ) ) ≤ 2 α ( 2 − α D ϕ 2 / η ) W R t = 0 ∑ T − 1 ( ∥ θ t − θ η ⋆ ∥ 2 2 − ∥ θ t + 1 − θ η ⋆ ∥ 2 2 ) ≤ 2 α ( 2 − α D ϕ 2 / η ) W R η 2 ∥ θ ⋆ ∥ 2 2 .
Combining the last two displays and dividing by T T T proves the final guarantee.
Lemma B.5 (One-step population PG progress). Let { θ t } t = 0 T \{\theta_t\}_{t=0}^{T} { θ t } t = 0 T follow Eq. (26) with 0 < α < 2 η / D ϕ 2 0 < \alpha< 2\eta/D_{\phi}^{2} 0 < α < 2 η / D ϕ 2 . Then, for every 0 ≤ t < T 0 \le t < T 0 ≤ t < T ,
⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ ≤ ∥ θ t − θ η ⋆ ∥ 2 2 − ∥ θ t + 1 − θ η ⋆ ∥ 2 2 α ( 2 − α D ϕ 2 / η ) . \left\langle\nabla J_{\eta}(\pi_{\theta_t}),\theta_{\eta}^{\star}-\theta_t\right\rangle\le\frac{\left\|\theta_t-\theta_{\eta}^{\star}\right\|_2^2-\left\|\theta_{t+1}-\theta_{\eta}^{\star}\right\|_2^2}{\alpha\left(2-\alpha D_{\phi}^{2}/\eta\right)}. ⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ ≤ α ( 2 − α D ϕ 2 / η ) θ t − θ η ⋆ 2 2 − θ t + 1 − θ η ⋆ 2 2 . If, additionally, θ t ∈ B R \theta_t \in B_R θ t ∈ B R for some R > 0 R > 0 R > 0 , then applying Lemma B.3 gives
J η ( π η ⋆ ) − J η ( π θ t ) ≤ W R 2 ∥ θ t − θ η ⋆ ∥ 2 2 − ∥ θ t + 1 − θ η ⋆ ∥ 2 2 α ( 2 − α D ϕ 2 / η ) . J_{\eta}(\pi_{\eta}^{\star})-J_{\eta}(\pi_{\theta_t}) \le\frac{W_R}{2}\frac{\left\|\theta_t-\theta_{\eta}^{\star}\right\|_2^2-\left\|\theta_{t+1}-\theta_{\eta}^{\star}\right\|_2^2}{\alpha\left(2-\alpha D_{\phi}^{2}/\eta\right)}. J η ( π η ⋆ ) − J η ( π θ t ) ≤ 2 W R α ( 2 − α D ϕ 2 / η ) θ t − θ η ⋆ 2 2 − θ t + 1 − θ η ⋆ 2 2 .
Proof of Lemma B.5. Fix 0 ≤ t < T 0 \le t < T 0 ≤ t < T . Expanding the update gives
∥ θ t + 1 − θ η ⋆ ∥ 2 2 = ∥ θ t − θ η ⋆ ∥ 2 2 − 2 α ⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ + α 2 ∥ ∇ J η ( π θ t ) ∥ 2 2 . \left\|\theta_{t+1}-\theta_{\eta}^{\star}\right\|_2^2=\left\|\theta_t-\theta_{\eta}^{\star}\right\|_2^2-2\alpha\left\langle\nabla J_{\eta}(\pi_{\theta_t}),\theta_{\eta}^{\star}-\theta_t\right\rangle+\alpha^2\left\|\nabla J_{\eta}(\pi_{\theta_t})\right\|_2^2. θ t + 1 − θ η ⋆ 2 2 = θ t − θ η ⋆ 2 2 − 2 α ⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ + α 2 ∥ ∇ J η ( π θ t ) ∥ 2 2 .
Since ∇ J η ( π θ t ) = 1 η Σ θ t ( θ η ⋆ − θ t ) \nabla J_{\eta}(\pi_{\theta_t})=\frac{1}{\eta}\Sigma_{\theta_t}(\theta_{\eta}^{\star}-\theta_t) ∇ J η ( π θ t ) = η 1 Σ θ t ( θ η ⋆ − θ t ) and 0 ⪯ Σ θ t ⪯ D ϕ 2 I 0\preceq\Sigma_{\theta_t}\preceq D_{\phi}^{2}I 0 ⪯ Σ θ t ⪯ D ϕ 2 I , we have
∥ ∇ J η ( π θ t ) ∥ 2 2 = 1 η 2 ( θ η ⋆ − θ t ) T Σ θ t 2 ( θ η ⋆ − θ t ) ≤ D ϕ 2 η 2 ( θ η ⋆ − θ t ) T Σ θ t ( θ η ⋆ − θ t ) = D ϕ 2 η ⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ . \begin{aligned}
\left\|\nabla J_{\eta}(\pi_{\theta_t})\right\|_2^2=\frac{1}{\eta^2}(\theta_{\eta}^{\star}-\theta_t)^{\mathsf{T}}\Sigma_{\theta_t}^{2}(\theta_{\eta}^{\star}-\theta_t) \\
&\le\frac{D_{\phi}^{2}}{\eta^2}(\theta_{\eta}^{\star}-\theta_t)^{\mathsf{T}}\Sigma_{\theta_t}(\theta_{\eta}^{\star}-\theta_t) \\
&= \frac{D_{\phi}^{2}}{\eta}\left\langle\nabla J_{\eta}(\pi_{\theta_t}),\theta_{\eta}^{\star}-\theta_t\right\rangle.
\end{aligned} ∥ ∇ J η ( π θ t ) ∥ 2 2 = η 2 1 ( θ η ⋆ − θ t ) T Σ θ t 2 ( θ η ⋆ − θ t ) ≤ η 2 D ϕ 2 ( θ η ⋆ − θ t ) T Σ θ t ( θ η ⋆ − θ t ) = η D ϕ 2 ⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ .
Therefore,
∥ θ t − θ η ⋆ ∥ 2 2 − ∥ θ t + 1 − θ η ⋆ ∥ 2 2 ≥ α ( 2 − α D ϕ 2 / η ) ⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ , \left\|\theta_t-\theta_{\eta}^{\star}\right\|_2^2-\left\|\theta_{t+1}-\theta_{\eta}^{\star}\right\|_2^2\ge\alpha\left(2-\alpha D_{\phi}^{2}/\eta\right)\left\langle\nabla J_{\eta}(\pi_{\theta_t}),\theta_{\eta}^{\star}-\theta_t\right\rangle, θ t − θ η ⋆ 2 2 − θ t + 1 − θ η ⋆ 2 2 ≥ α ( 2 − α D ϕ 2 / η ) ⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ ,
and rearranging proves the first claim.
Now suppose θ t ∈ B R \theta_t \in B_R θ t ∈ B R . Using the local concavity in Lemma B.3,
J η ( π η ⋆ ) − J η ( π θ t ) ≤ W R 2 ⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ . J_{\eta}(\pi_{\eta}^{\star})-J_{\eta}(\pi_{\theta_t})\le\frac{W_R}{2}\left\langle\nabla J_{\eta}(\pi_{\theta_t}),\theta_{\eta}^{\star}-\theta_t\right\rangle. J η ( π η ⋆ ) − J η ( π θ t ) ≤ 2 W R ⟨ ∇ J η ( π θ t ) , θ η ⋆ − θ t ⟩ .
:::
Lemma B.6 (Population PG ascent). Fix R > 0 R > 0 R > 0 and let L R L_R L R be defined as in Lemma B.4. Let { θ t } t = 0 T \{\theta_t\}_{t=0}^{T} { θ t } t = 0 T follow Eq. (26) , and suppose that θ t ∈ B R \theta_t \in B_R θ t ∈ B R for all 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T . Then, for every 0 ≤ t < T 0 \le t < T 0 ≤ t < T ,
J η ( π θ t + 1 ) − J η ( π θ t ) ≥ α ( 1 − α L R 2 ) ∥ ∇ J η ( π θ t ) ∥ 2 2 . J_{\eta}(\pi_{\theta_{t+1}})-J_{\eta}(\pi_{\theta_t})\ge\alpha\left(1-\frac{\alpha L_R}{2}\right)\left\|\nabla J_{\eta}(\pi_{\theta_t})\right\|_2^2. J η ( π θ t + 1 ) − J η ( π θ t ) ≥ α ( 1 − 2 α L R ) ∥ ∇ J η ( π θ t ) ∥ 2 2 .
Proof of Lemma B.6.
Fix 0 ≤ t < T 0 \le t < T 0 ≤ t < T . By assumption, both θ t \theta_t θ t and θ t + 1 \theta_{t+1} θ t + 1 belong to the ball B R B_R B R . Since this ball is convex, the line segment joining the two iterates also lies in the ball, so the local smoothness inequality in Lemma B.4 applies and gives
J η ( π θ t + 1 ) − J η ( π θ t ) ≥ ⟨ ∇ J η ( π θ t ) , θ t + 1 − θ t ⟩ − L R 2 ∥ θ t + 1 − θ t ∥ 2 2 . J_{\eta}(\pi_{\theta_{t+1}})-J_{\eta}(\pi_{\theta_t})\ge\left\langle\nabla J_{\eta}(\pi_{\theta_t}),\theta_{t+1}-\theta_t\right\rangle-\frac{L_R}{2}\left\|\theta_{t+1}-\theta_t\right\|_2^2. J η ( π θ t + 1 ) − J η ( π θ t ) ≥ ⟨ ∇ J η ( π θ t ) , θ t + 1 − θ t ⟩ − 2 L R ∥ θ t + 1 − θ t ∥ 2 2 .
::: Substituting θ t + 1 − θ t = α ∇ J η ( π θ t ) \theta_{t+1}-\theta_t=\alpha\nabla J_{\eta}(\pi_{\theta_t}) θ t + 1 − θ t = α ∇ J η ( π θ t ) from Eq. (26) yields
J η ( π θ t + 1 ) − J η ( π θ t ) ≥ α ∥ ∇ J η ( π θ t ) ∥ 2 2 − α 2 L R 2 ∥ ∇ J η ( π θ t ) ∥ 2 2 = α ( 1 − α L R 2 ) ∥ ∇ J η ( π θ t ) ∥ 2 2 . \begin{aligned}
J_{\eta}(\pi_{\theta_{t+1}})-J_{\eta}(\pi_{\theta_t}) \ge\alpha\|\nabla J_{\eta}(\pi_{\theta_t})\|_2^2-\frac{\alpha^2L_R}{2}\|\nabla J_{\eta}(\pi_{\theta_t})\|_2^2 \\
&= \alpha\left(1-\frac{\alpha L_R}{2}\right)\|\nabla J_{\eta}(\pi_{\theta_t})\|_2^2.
\end{aligned} J η ( π θ t + 1 ) − J η ( π θ t ) ≥ α ∥∇ J η ( π θ t ) ∥ 2 2 − 2 α 2 L R ∥∇ J η ( π θ t ) ∥ 2 2 = α ( 1 − 2 α L R ) ∥∇ J η ( π θ t ) ∥ 2 2 .
as claimed.
Proof of Lemma B.3. Fix θ \theta θ and write
δ θ ( y ) : = η r ( y ) − ϕ ( y ) T ( θ − θ r e f ) = ϕ ( y ) T ( θ η ⋆ − θ ) . \delta_{\theta}(y):=\eta r(y)-\phi(y)^{\mathsf{T}}(\theta-\theta_{\mathrm{ref}})=\phi(y)^{\mathsf{T}}(\theta_{\eta}^{\star}-\theta). δ θ ( y ) := η r ( y ) − ϕ ( y ) T ( θ − θ ref ) = ϕ ( y ) T ( θ η ⋆ − θ ) .
For s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] , let π s ( y ) ∝ π θ ( y ) e s δ θ ( y ) \pi_s(y)\propto\pi_{\theta}(y)e^{s\delta_{\theta}(y)} π s ( y ) ∝ π θ ( y ) e s δ θ ( y ) be the exponential tilt of π θ \pi_{\theta} π θ by s δ θ s\delta_{\theta} s δ θ , so that π 1 = π η ⋆ \pi_1=\pi_{\eta}^{\star} π 1 = π η ⋆ . We can write
J η ( π η ⋆ ) − J η ( π θ ) = 1 η D K L ( π θ ∥ π η ⋆ ) = 1 η ∫ 0 1 ( 1 − s ) V y ∼ π s ( δ θ ( y ) ) d s . J_{\eta}(\pi_{\eta}^{\star})-J_{\eta}(\pi_{\theta})=\frac{1}{\eta}D_{\mathrm{KL}}(\pi_{\theta}\Vert\pi_{\eta}^{\star})=\frac{1}{\eta}\int_0^1(1-s)\,\mathbb{V}_{y\sim\pi_s}(\delta_{\theta}(y))\,\mathrm{d}s. J η ( π η ⋆ ) − J η ( π θ ) = η 1 D KL ( π θ ∥ π η ⋆ ) = η 1 ∫ 0 1 ( 1 − s ) V y ∼ π s ( δ θ ( y )) d s .
By a change of measure and since ∥ π s / π θ ∥ ∞ \|\pi_s/\pi_{\theta}\|_{\infty} ∥ π s / π θ ∥ ∞ is nondecreasing in s s s ,
V y ∼ π s ( δ θ ( y ) ) ≤ ∥ π s π θ ∥ ∞ V y ∼ π θ ( δ θ ( y ) ) ≤ ∥ π η ⋆ π θ ∥ ∞ V y ∼ π θ ( δ θ ( y ) ) . \mathbb{V}_{y\sim\pi_s}(\delta_{\theta}(y))\le\left\|\frac{\pi_s}{\pi_{\theta}}\right\|_{\infty}\mathbb{V}_{y\sim\pi_{\theta}}(\delta_{\theta}(y))\le\left\|\frac{\pi_{\eta}^{\star}}{\pi_{\theta}}\right\|_{\infty}\mathbb{V}_{y\sim\pi_{\theta}}(\delta_{\theta}(y)). V y ∼ π s ( δ θ ( y )) ≤ π θ π s ∞ V y ∼ π θ ( δ θ ( y )) ≤ π θ π η ⋆ ∞ V y ∼ π θ ( δ θ ( y )) .
Combining the above two displays and integrating,
J η ( π η ⋆ ) − J η ( π θ ) ≤ 1 2 η ∥ π η ⋆ π θ ∥ ∞ V y ∼ π θ ( δ θ ( y ) ) . J_{\eta}(\pi_{\eta}^{\star})-J_{\eta}(\pi_{\theta})\le\frac{1}{2\eta}\left\|\frac{\pi_{\eta}^{\star}}{\pi_{\theta}}\right\|_{\infty}\mathbb{V}_{y\sim\pi_{\theta}}(\delta_{\theta}(y)). J η ( π η ⋆ ) − J η ( π θ ) ≤ 2 η 1 π θ π η ⋆ ∞ V y ∼ π θ ( δ θ ( y )) .
Since
V y ∼ π θ ( δ θ ( y ) ) = ∥ θ η ⋆ − θ ∥ Σ θ 2 = η ⟨ ∇ J η ( π θ ) , θ η ⋆ − θ ⟩ , \mathbb{V}_{y\sim\pi_{\theta}}(\delta_{\theta}(y))=\|\theta_{\eta}^{\star}-\theta\|_{\Sigma_{\theta}}^2=\eta\langle\nabla J_{\eta}(\pi_{\theta}),\theta_{\eta}^{\star}-\theta\rangle, V y ∼ π θ ( δ θ ( y )) = ∥ θ η ⋆ − θ ∥ Σ θ 2 = η ⟨ ∇ J η ( π θ ) , θ η ⋆ − θ ⟩ ,
this proves the first inequality. Finally, range ( δ θ ) ≤ D ϕ R \operatorname{range}(\delta_{\theta})\le D_{\phi}R range ( δ θ ) ≤ D ϕ R implies
∥ π η ⋆ π θ ∥ ∞ ≤ e D ϕ R , \left\|\frac{\pi_{\eta}^{\star}}{\pi_{\theta}}\right\|_{\infty}\le e^{D_{\phi}R}, π θ π η ⋆ ∞ ≤ e D ϕ R ,
proving the second inequality.
Proof of Lemma B.4 . For every h ∈ R d h\in\mathbb{R}^d h ∈ R d ,
h T ∇ 2 J η ( π θ ) h = − 1 η E y ∼ π θ [ ( h T ϕ ~ θ ( y ) ) 2 ] + 1 η E y ∼ π θ [ ( h T ϕ ~ θ ( y ) ) 2 ( θ η ⋆ − θ ) T ϕ ~ θ ( y ) ] . h^{\mathsf{T}}\nabla^2J_{\eta}(\pi_{\theta})h=-\frac{1}{\eta}\mathbb{E}_{y\sim\pi_{\theta}}\left[\left(h^{\mathsf{T}}\widetilde{\phi}_{\theta}(y)\right)^2\right]
+\frac{1}{\eta}\mathbb{E}_{y\sim\pi_{\theta}}\left[\left(h^{\mathsf{T}}\widetilde{\phi}_{\theta}(y)\right)^2(\theta_{\eta}^{\star}-\theta)^{\mathsf{T}}\widetilde{\phi}_{\theta}(y)\right]. h T ∇ 2 J η ( π θ ) h = − η 1 E y ∼ π θ [ ( h T ϕ θ ( y ) ) 2 ] + η 1 E y ∼ π θ [ ( h T ϕ θ ( y ) ) 2 ( θ η ⋆ − θ ) T ϕ θ ( y ) ] .
The definitions of the ball and D ϕ D_{\phi} D ϕ imply
∣ ( θ η ⋆ − θ ) T ϕ ~ θ ( y ) ∣ ≤ D ϕ R , ∣ h T ϕ ~ θ ( y ) ∣ ≤ D ϕ ∥ h ∥ 2 . \left|(\theta_{\eta}^{\star}-\theta)^{\mathsf{T}}\widetilde{\phi}_{\theta}(y)\right|\le D_{\phi}R,\qquad\left|h^{\mathsf{T}}\widetilde{\phi}_{\theta}(y)\right|\le D_{\phi}\|h\|_2. ( θ η ⋆ − θ ) T ϕ θ ( y ) ≤ D ϕ R , h T ϕ θ ( y ) ≤ D ϕ ∥ h ∥ 2 .
Therefore,
∣ h T ∇ 2 J η ( π θ ) h ∣ ≤ 1 η ( 1 + D ϕ R ) E y ∼ π θ [ ( h T ϕ ~ θ ( y ) ) 2 ] ≤ L R ∥ h ∥ 2 2 , \begin{aligned}
\left|h^{\mathsf{T}}\nabla^2J_{\eta}(\pi_{\theta})h\right|\le\frac{1}{\eta}(1+D_{\phi}R)\mathbb{E}_{y\sim\pi_{\theta}}\left[\left(h^{\mathsf{T}}\widetilde{\phi}_{\theta}(y)\right)^2\right] \\
&\le L_R\|h\|_2^2,
\end{aligned} h T ∇ 2 J η ( π θ ) h ≤ η 1 ( 1 + D ϕ R ) E y ∼ π θ [ ( h T ϕ θ ( y ) ) 2 ] ≤ L R ∥ h ∥ 2 2 ,
and ∥ ∇ 2 J η ( π θ ) ∥ o p ≤ L R \lVert\nabla^{2}J_{\eta}(\pi_{\theta})\rVert_{\mathrm{op}} \le L_{R} ∥ ∇ 2 J η ( π θ ) ∥ op ≤ L R throughout the ball. To conclude, fix θ , θ ′ ∈ B R \theta,\theta' \in\mathcal{B}_{R} θ , θ ′ ∈ B R . Since the ball is convex, the Hessian bound holds at θ + s ( θ ′ − θ ) \theta+s(\theta'-\theta) θ + s ( θ ′ − θ ) for every s ∈ [ 0 , 1 ] s \in[0,1] s ∈ [ 0 , 1 ] . Therefore,
∥ ∇ J η ( π θ ′ ) − ∇ J η ( π θ ) ∥ 2 = ∥ ∫ 0 1 ∇ 2 J η ( π θ + s ( θ ′ − θ ) ) ( θ ′ − θ ) d s ∥ 2 ≤ L R ∥ θ ′ − θ ∥ 2 . \lVert\nabla J_{\eta}(\pi_{\theta'})-\nabla J_{\eta}(\pi_{\theta})\rVert_{2}
=
\left\lVert\int_{0}^{1}\nabla^{2}J_{\eta}(\pi_{\theta+s(\theta'-\theta)})(\theta'-\theta)\,\mathrm{d}s \right\rVert_{2}
\le L_{R}\lVert\theta'-\theta\rVert_{2}. ∥ ∇ J η ( π θ ′ ) − ∇ J η ( π θ ) ∥ 2 = ∫ 0 1 ∇ 2 J η ( π θ + s ( θ ′ − θ ) ) ( θ ′ − θ ) d s 2 ≤ L R ∥ θ ′ − θ ∥ 2 .
Similarly, Taylor’s theorem with integral remainder gives
J η ( π θ ′ ) − J η ( π θ ) − ⟨ ∇ J η ( π θ ) , θ ′ − θ ⟩ = ∫ 0 1 ( 1 − s ) ( θ ′ − θ ) T ∇ 2 J η ( π θ + s ( θ ′ − θ ) ) ( θ ′ − θ ) d s ≥ − L R ∥ θ ′ − θ ∥ 2 2 ∫ 0 1 ( 1 − s ) d s = − L R 2 ∥ θ ′ − θ ∥ 2 2 . J_{\eta}(\pi_{\theta'})-J_{\eta}(\pi_{\theta})-\langle\nabla J_{\eta}(\pi_{\theta}),\theta'-\theta\rangle
=
\int_{0}^{1}(1-s)(\theta'-\theta)^{\mathsf{T}}\nabla^{2}J_{\eta}(\pi_{\theta+s(\theta'-\theta)})(\theta'-\theta)\,\mathrm{d}s
\ge-L_{R}\lVert\theta'-\theta\rVert_{2}^{2}\int_{0}^{1}(1-s)\,\mathrm{d}s
=
-\frac{L_{R}}{2}\lVert\theta'-\theta\rVert_{2}^{2}. J η ( π θ ′ ) − J η ( π θ ) − ⟨ ∇ J η ( π θ ) , θ ′ − θ ⟩ = ∫ 0 1 ( 1 − s ) ( θ ′ − θ ) T ∇ 2 J η ( π θ + s ( θ ′ − θ ) ) ( θ ′ − θ ) d s ≥ − L R ∥ θ ′ − θ ∥ 2 2 ∫ 0 1 ( 1 − s ) d s = − 2 L R ∥ θ ′ − θ ∥ 2 2 .
This proves both claims.
Exact Mirror Descent
We analyze exact KL-regularized mirror descent for contextual bandits. These are the updates obtained when every RL subproblem of APMD is solved exactly.
Exact mirror descent. Fix η > 0 \eta>0 η > 0 and initialize π 0 = π b a s e > 0 \pi_{0}=\pi_{\mathrm{base}}>0 π 0 = π base > 0 . For every integer t ≥ 0 t\ge0 t ≥ 0 , update
π t + 1 ( ⋅ ∣ x ) = π η , t ( ⋅ ∣ x ) = arg max π ( ⋅ ∣ x ) ∈ Δ ( Y ) { J x ( π ) − 1 η D K L ( π ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x ) ) } , x ∈ X . (27) \pi_{t+1}(\cdot\mid x)=\pi_{\eta,t}(\cdot\mid x)=\underset{\pi(\cdot\mid x)\in\Delta(\mathcal{Y})}{\arg\max}\left\{J_{x}(\pi)-\frac{1}{\eta}D_{\mathrm{KL}}\bigl(\pi(\cdot\mid x)\mathbin{\|}\pi_{t}(\cdot\mid x)\bigr)\right\},\qquad x\in\mathcal{X}. \tag*{(27)} π t + 1 ( ⋅ ∣ x ) = π η , t ( ⋅ ∣ x ) = π ( ⋅ ∣ x ) ∈ Δ ( Y ) arg max { J x ( π ) − η 1 D KL ( π ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x ) ) } , x ∈ X . (27)
Lemma B.7 gives the closed form of this update and of the resulting policy sequence. We write J x ( π ) : = E y ∼ π ( ⋅ ∣ x ) [ r ( x , y ) ] J_{x}(\pi):=\mathbb{E}_{y\sim\pi(\cdot\mid x)}[r(x,y)] J x ( π ) := E y ∼ π ( ⋅ ∣ x ) [ r ( x , y )] and r x ⋆ : = max y ∈ Y r ( x , y ) r_{x}^{\star}:=\max_{y\in\mathcal{Y}}r(x,y) r x ⋆ := max y ∈ Y r ( x , y ) , and define the partition function for the rewards centered by the mean baseline r ( x , π t ) = J x ( π t ) r(x,\pi_{t})=J_{x}(\pi_{t}) r ( x , π t ) = J x ( π t ) by
Z t ( x ) : = E y ∼ π t ( ⋅ ∣ x ) [ e η ( r ( x , y ) − r ( x , π t ) ) ] . Z_{t}(x):=\mathbb{E}_{y\sim\pi_{t}(\cdot\mid x)}\left[\mathrm{e}^{\eta(r(x,y)-r(x,\pi_{t}))}\right]. Z t ( x ) := E y ∼ π t ( ⋅ ∣ x ) [ e η ( r ( x , y ) − r ( x , π t )) ] .
Subtracting r ( x , π t ) r(x,\pi_{t}) r ( x , π t ) from every reward on prompt x x x leaves the exact update unchanged.
Theorem B.3 (Convergence of exact mirror descent). Let p 0 ( x ) : = π b a s e ( Y x ⋆ ∣ x ) p_{0}(x):=\pi_{\mathrm{base}}(Y_{x}^{\star}\mid x) p 0 ( x ) := π base ( Y x ⋆ ∣ x ) denote the probability of an optimal response under the base policy. For the exact mirror descent iterates in Eq. (27) , for every t > 0 t>0 t > 0 , it holds that
J ⋆ − J ( π t ) ≤ E x ∼ μ [ log ( 1 / p 0 ( x ) ) ] η t = D K L ( π ⋆ ∥ π b a s e ) η t ≤ log C b a s e η t . J^{\star}-J(\pi_{t})\le\frac{\mathbb{E}_{x\sim\mu}[\log(1/p_{0}(x))]}{\eta t}=\frac{D_{\mathrm{KL}}(\pi^{\star}\mathbin{\|}\pi_{\mathrm{base}})}{\eta t}\le\frac{\log C_{\mathrm{base}}}{\eta t}. J ⋆ − J ( π t ) ≤ η t E x ∼ μ [ log ( 1/ p 0 ( x ))] = η t D KL ( π ⋆ ∥ π base ) ≤ η t log C base . where π ⋆ ( y ∣ x ) : = π b a s e ( y ∣ x ) I [ y ∈ Y x ⋆ ] π b a s e ( Y x ⋆ ∣ x ) \pi^{\star}(y\mid x):=\frac{\pi_{\mathrm{base}}(y\mid x)\mathbb{I}[y\in Y_{x}^{\star}]}{\pi_{\mathrm{base}}(Y_{x}^{\star}\mid x)} π ⋆ ( y ∣ x ) := π base ( Y x ⋆ ∣ x ) π base ( y ∣ x ) I [ y ∈ Y x ⋆ ] is the optimal policy induced by conditioning π b a s e \pi_{\mathrm{base}} π base on optimal responses.
Proof of Theorem B.3. For this proof and the supporting lemmas below, write p t ( x ) : = π t ( Y x ⋆ ∣ x ) p_{t}(x):=\pi_{t}(Y_{x}^{\star}\mid x) p t ( x ) := π t ( Y x ⋆ ∣ x ) for t ≥ 0 t\ge0 t ≥ 0 . Fix a prompt x x x . By Lemma B.9 , the probability p t ( x ) p_{t}(x) p t ( x ) of an optimal response is nondecreasing in t t t , so p t ( x ) ≥ p 0 ( x ) > 0 p_{t}(x)\ge p_{0}(x)>0 p t ( x ) ≥ p 0 ( x ) > 0 for every t ≥ 0 t\ge0 t ≥ 0 .
Fix an iteration t ≥ 1 t\ge1 t ≥ 1 . Applying Lemma B.10 at s = 0 , … , t − 1 s=0,\ldots,t-1 s = 0 , … , t − 1 and summing gives
η ∑ s = 1 t ( r x ⋆ − J x ( π s ) ) ≤ ∑ s = 0 t − 1 ( log p s + 1 ( x ) − log p s ( x ) ) = log p t ( x ) − log p 0 ( x ) ≤ log 1 p 0 ( x ) . (28) \eta\sum_{s=1}^{t}(r_{x}^{\star}-J_{x}(\pi_{s}))\le\sum_{s=0}^{t-1}(\log p_{s+1}(x)-\log p_{s}(x))
=\log p_{t}(x)-\log p_{0}(x)\le\log\frac{1}{p_{0}(x)}. \tag*{(28)} η s = 1 ∑ t ( r x ⋆ − J x ( π s )) ≤ s = 0 ∑ t − 1 ( log p s + 1 ( x ) − log p s ( x )) = log p t ( x ) − log p 0 ( x ) ≤ log p 0 ( x ) 1 . (28)
Since J x ( π s ) J_x(\pi_s) J x ( π s ) is nondecreasing in s s s by Lemma B.8 ,
t ( r x ⋆ − J x ( π t ) ) ≤ ∑ s = 1 t ( r x ⋆ − J x ( π s ) ) . t(r_x^\star-J_x(\pi_t))\le\sum_{s=1}^{t}(r_x^\star-J_x(\pi_s)). t ( r x ⋆ − J x ( π t )) ≤ s = 1 ∑ t ( r x ⋆ − J x ( π s )) .
Combining these inequalities gives
r x ⋆ − J x ( π t ) ≤ log ( 1 / p 0 ( x ) ) η t . r_x^\star-J_x(\pi_t)\le\frac{\log(1/p_0(x))}{\eta t}. r x ⋆ − J x ( π t ) ≤ η t log ( 1/ p 0 ( x )) .
Taking expectations over prompts gives
J ⋆ − J ( π t ) ≤ E x ∼ μ [ log ( 1 / p 0 ( x ) ) ] η t . J^\star-J(\pi_t)\le\frac{\mathbb{E}_{x\sim\mu}[\log(1/p_0(x))]}{\eta t}. J ⋆ − J ( π t ) ≤ η t E x ∼ μ [ log ( 1/ p 0 ( x ))] .
The equality in the theorem statement follows from the definition of π ⋆ \pi^\star π ⋆ .
D K L ( π ⋆ ∥ π b a s e ) = E x ∼ μ [ E y ∼ π ⋆ ( ⋅ ∣ x ) [ log 1 p 0 ( x ) ] ] = E x ∼ μ [ log 1 p 0 ( x ) ] . D_{\mathrm{KL}}(\pi^\star\Vert\pi_{\mathrm{base}})=\mathbb{E}_{x\sim\mu}\left[\mathbb{E}_{y\sim\pi^\star(\cdot\mid x)}\left[\log\frac{1}{p_0(x)}\right]\right]=\mathbb{E}_{x\sim\mu}\left[\log\frac{1}{p_0(x)}\right]. D KL ( π ⋆ ∥ π base ) = E x ∼ μ [ E y ∼ π ⋆ ( ⋅ ∣ x ) [ log p 0 ( x ) 1 ] ] = E x ∼ μ [ log p 0 ( x ) 1 ] .
Finally, Jensen’s inequality gives
E x ∼ μ [ log 1 p 0 ( x ) ] ≤ log E x ∼ μ [ 1 p 0 ( x ) ] = log C b a s e . \mathbb{E}_{x\sim\mu}\left[\log\frac{1}{p_0(x)}\right]\le\log\mathbb{E}_{x\sim\mu}\left[\frac{1}{p_0(x)}\right]=\log C_{\mathrm{base}}. E x ∼ μ [ log p 0 ( x ) 1 ] ≤ log E x ∼ μ [ p 0 ( x ) 1 ] = log C base .
Lemma B.7 (Closed form of exact mirror descent). For any current policy π t \pi_t π t and η > 0 \eta>0 η > 0 , the exact update in Eq. (27) satisfies, for every prompt x x x and response y y y ,
π η , t ⋆ ( y ∣ x ) = π t ( y ∣ x ) e η ( r ( x , y ) − r ( x , π t ) ) Z t ( x ) . (29) \pi_{\eta,t}^\star(y\mid x)=\frac{\pi_t(y\mid x)e^{\eta(r(x,y)-r(x,\pi_t))}}{Z_t(x)}. \tag*{(29)} π η , t ⋆ ( y ∣ x ) = Z t ( x ) π t ( y ∣ x ) e η ( r ( x , y ) − r ( x , π t )) . (29) If π 0 = π b a s e \pi_0=\pi_{\mathrm{base}} π 0 = π base and π t + 1 = π η , t ⋆ \pi_{t+1}=\pi_{\eta,t}^\star π t + 1 = π η , t ⋆ at every iteration, then
π t ( y ∣ x ) = π b a s e ( y ∣ x ) e t η r ( x , y ) E y ′ ∼ π b a s e ( ⋅ ∣ x ) [ e t η r ( x , y ′ ) ] , t ≥ 0. (30) \pi_t(y\mid x)=\frac{\pi_{\mathrm{base}}(y\mid x)e^{t\eta r(x,y)}}{\mathbb{E}_{y'\sim\pi_{\mathrm{base}}(\cdot\mid x)}[e^{t\eta r(x,y')}]},\qquad t\ge0. \tag*{(30)} π t ( y ∣ x ) = E y ′ ∼ π base ( ⋅ ∣ x ) [ e t η r ( x , y ′ ) ] π base ( y ∣ x ) e t η r ( x , y ) , t ≥ 0. (30) In particular, π η , t ⋆ ( ⋅ ∣ x ) \pi_{\eta,t}^\star(\cdot\mid x) π η , t ⋆ ( ⋅ ∣ x ) and π t ( ⋅ ∣ x ) \pi_t(\cdot\mid x) π t ( ⋅ ∣ x ) have the same support, which remains equal to that of π b a s e ( ⋅ ∣ x ) \pi_{\mathrm{base}}(\cdot\mid x) π base ( ⋅ ∣ x ) along the sequence.
Proof of Lemma B.7 . Fix a prompt x x x . A policy assigning positive probability outside the support of π t ( ⋅ ∣ x ) \pi_t(\cdot\mid x) π t ( ⋅ ∣ x ) has infinite KL divergence in Eq. (27) . For any policy π \pi π with finite KL divergence, we have
J x ( π ) − 1 η D K L ( π ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x ) ) = r ( x , π t ) + log Z t ( x ) η − 1 η ∑ y : π t ( y ∣ x ) > 0 π ( y ∣ x ) log π ( y ∣ x ) Z t ( x ) π t ( y ∣ x ) e η ( r ( x , y ) − r ( x , π t ) ) ≤ r ( x , π t ) + log Z t ( x ) η . \begin{aligned}
J_x(\pi)-\frac{1}{\eta}D_{\mathrm{KL}}(\pi(\cdot\mid x)\Vert\pi_t(\cdot\mid x)) \\
&=r(x,\pi_t)+\frac{\log Z_t(x)}{\eta}-\frac{1}{\eta}\sum_{y:\pi_t(y\mid x)>0}\pi(y\mid x)\log\frac{\pi(y\mid x)Z_t(x)}{\pi_t(y\mid x)e^{\eta(r(x,y)-r(x,\pi_t))}} \\
&\le r(x,\pi_t)+\frac{\log Z_t(x)}{\eta}.
\end{aligned} J x ( π ) − η 1 D KL ( π ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x )) = r ( x , π t ) + η log Z t ( x ) − η 1 y : π t ( y ∣ x ) > 0 ∑ π ( y ∣ x ) log π t ( y ∣ x ) e η ( r ( x , y ) − r ( x , π t )) π ( y ∣ x ) Z t ( x ) ≤ r ( x , π t ) + η log Z t ( x ) .
The sum is the KL divergence from π ( ⋅ ∣ x ) \pi(\cdot\mid x) π ( ⋅ ∣ x ) to the distribution in Eq. (29), whose probabilities sum to one by the definition of Z t ( x ) Z_t(x) Z t ( x ) . It is nonnegative, with equality precisely at that distribution. This proves the closed form of π η , t ⋆ \pi_{\eta,t}^\star π η , t ⋆ .
For the policy sequence, Eq. (30) holds at t = 0 t = 0 t = 0 . If it holds at iteration t t t , substituting into Eq. (29) and canceling the common normalization factor gives
π t + 1 ( y ∣ x ) = π b a s e ( y ∣ x ) e t η r ( x , y ) e η r ( x , y ) ∑ y ′ ∈ Y π b a s e ( y ′ ∣ x ) e t η r ( x , y ′ ) e η r ( x , y ′ ) = π b a s e ( y ∣ x ) e ( t + 1 ) η r ( x , y ) E y ′ ∼ π b a s e ( ⋅ ∣ x ) [ e ( t + 1 ) η r ( x , y ′ ) ] . \pi_{t+1}(y \mid x) = \frac{\pi_{\mathrm{base}}(y \mid x)e^{t\eta r(x,y)}e^{\eta r(x,y)}}{\sum_{y'\in\mathcal{Y}}\pi_{\mathrm{base}}(y' \mid x)e^{t\eta r(x,y')}e^{\eta r(x,y')}}
= \frac{\pi_{\mathrm{base}}(y \mid x)e^{(t+1)\eta r(x,y)}}{\mathbb{E}_{y'\sim\pi_{\mathrm{base}}(\cdot\mid x)}\left[e^{(t+1)\eta r(x,y')}\right]}. π t + 1 ( y ∣ x ) = ∑ y ′ ∈ Y π base ( y ′ ∣ x ) e t η r ( x , y ′ ) e η r ( x , y ′ ) π base ( y ∣ x ) e t η r ( x , y ) e η r ( x , y ) = E y ′ ∼ π base ( ⋅ ∣ x ) [ e ( t + 1 ) η r ( x , y ′ ) ] π base ( y ∣ x ) e ( t + 1 ) η r ( x , y ) .
Induction proves the second claim. The support claims follow because the exponential factors are strictly positive.
The following improvement bound is the contextual bandit analogue of Agarwal et al. [1 ] , Lemma 17.
Lemma B.8 (Improvement lower bound). For the exact mirror descent iterates with η > 0 \eta> 0 η > 0 , at every iteration t ≥ 0 t \ge0 t ≥ 0 ,
J ( π t + 1 ) − J ( π t ) ≥ 1 η E x ∼ μ [ log Z t ( x ) ] ≥ 0. J(\pi_{t+1}) - J(\pi_t) \ge\frac{1}{\eta}\mathbb{E}_{x\sim\mu}[\log Z_t(x)] \ge0. J ( π t + 1 ) − J ( π t ) ≥ η 1 E x ∼ μ [ log Z t ( x )] ≥ 0. For each prompt x ∈ X x \in\mathcal{X} x ∈ X ,
J x ( π t + 1 ) − J x ( π t ) ≥ log Z t ( x ) η ≥ 0. J_x(\pi_{t+1}) - J_x(\pi_t) \ge\frac{\log Z_t(x)}{\eta} \ge0. J x ( π t + 1 ) − J x ( π t ) ≥ η log Z t ( x ) ≥ 0.
Proof of Lemma B.8 . Fix a prompt x x x . Jensen’s inequality gives
log Z t ( x ) = log E y ∼ π t ( ⋅ ∣ x ) [ e η ( r ( x , y ) − r ( x , π t ) ) ] ≥ η E y ∼ π t ( ⋅ ∣ x ) [ r ( x , y ) − r ( x , π t ) ] = 0. \log Z_t(x) = \log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta(r(x,y)-r(x,\pi_t))}\right] \ge\eta\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[r(x,y)-r(x,\pi_t)] = 0. log Z t ( x ) = log E y ∼ π t ( ⋅ ∣ x ) [ e η ( r ( x , y ) − r ( x , π t )) ] ≥ η E y ∼ π t ( ⋅ ∣ x ) [ r ( x , y ) − r ( x , π t )] = 0.
By Lemma B.7 , for every response in the support of π t ( ⋅ ∣ x ) \pi_t(\cdot\mid x) π t ( ⋅ ∣ x ) ,
log π t + 1 ( y ∣ x ) π t ( y ∣ x ) = η ( r ( x , y ) − r ( x , π t ) ) − log Z t ( x ) . \log\frac{\pi_{t+1}(y\mid x)}{\pi_t(y\mid x)} = \eta(r(x,y)-r(x,\pi_t))-\log Z_t(x). log π t ( y ∣ x ) π t + 1 ( y ∣ x ) = η ( r ( x , y ) − r ( x , π t )) − log Z t ( x ) .
Taking the expectation under y ∼ π t + 1 ( ⋅ ∣ x ) y\sim\pi_{t+1}(\cdot\mid x) y ∼ π t + 1 ( ⋅ ∣ x ) gives
D K L ( π t + 1 ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x ) ) = η ( J x ( π t + 1 ) − J x ( π t ) ) − log Z t ( x ) . (31) D_{\mathrm{KL}}(\pi_{t+1}(\cdot\mid x)\mathbin\Vert\pi_t(\cdot\mid x)) = \eta(J_x(\pi_{t+1})-J_x(\pi_t))-\log Z_t(x). \tag*{(31)} D KL ( π t + 1 ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x )) = η ( J x ( π t + 1 ) − J x ( π t )) − log Z t ( x ) . (31)
Therefore,
J x ( π t + 1 ) − J x ( π t ) = 1 η D K L ( π t + 1 ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x ) ) + log Z t ( x ) η ≥ log Z t ( x ) η ≥ 0. J_x(\pi_{t+1})-J_x(\pi_t) = \frac{1}{\eta}D_{\mathrm{KL}}(\pi_{t+1}(\cdot\mid x)\mathbin\Vert\pi_t(\cdot\mid x))+\frac{\log Z_t(x)}{\eta}
\ge\frac{\log Z_t(x)}{\eta}\ge0. J x ( π t + 1 ) − J x ( π t ) = η 1 D KL ( π t + 1 ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x )) + η log Z t ( x ) ≥ η log Z t ( x ) ≥ 0.
Averaging over prompts proves the bound on J ( π t + 1 ) − J ( π t ) J(\pi_{t+1})-J(\pi_t) J ( π t + 1 ) − J ( π t ) .
Lemma B.9 (Response coverage improvement). For the exact mirror descent iterates with η > 0 \eta> 0 η > 0 , at every iteration t ≥ 0 t \ge0 t ≥ 0 ,
p t + 1 ( x ) ≥ p t ( x ) , x ∈ X . p_{t+1}(x)\ge p_t(x), \qquad x\in\mathcal{X}. p t + 1 ( x ) ≥ p t ( x ) , x ∈ X . Consequently,
C π t + 1 ≤ C π t . C_{\pi_{t+1}}\le C_{\pi_t}. C π t + 1 ≤ C π t .
Proof of Lemma B.9. Fix a prompt x x x . Since r ( x , y ) ≤ r x ⋆ r(x,y) \le r_x^\star r ( x , y ) ≤ r x ⋆ for every response y y y ,
Z t ( x ) = E y ∼ π t ( ⋅ ∣ x ) [ e η ( r ( x , y ) − r ( x , π t ) ) ] ≤ e η ( r x ⋆ − r ( x , π t ) ) . Z_t(x) = \mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta(r(x,y)-r(x,\pi_t))}\right] \le e^{\eta(r_x^\star-r(x,\pi_t))}. Z t ( x ) = E y ∼ π t ( ⋅ ∣ x ) [ e η ( r ( x , y ) − r ( x , π t )) ] ≤ e η ( r x ⋆ − r ( x , π t )) .
Summing Eq. (29) over the optimal responses gives
p t + 1 ( x ) = ∑ y ∈ Y ⋆ π t ( y ∣ x ) e η ( r ( x , y ) − r ( x , π t ) ) Z t ( x ) = p t ( x ) e η ( r x ⋆ − r ( x , π t ) ) Z t ( x ) ≥ p t ( x ) . (32) \begin{aligned}
p_{t+1}(x) = \sum_{y\in\mathcal{Y}^\star}\frac{\pi_t(y\mid x)e^{\eta(r(x,y)-r(x,\pi_t))}}{Z_t(x)} \\
&= \frac{p_t(x)e^{\eta(r_x^\star-r(x,\pi_t))}}{Z_t(x)} \ge p_t(x). \tag*{(32)}
\end{aligned} p t + 1 ( x ) = y ∈ Y ⋆ ∑ Z t ( x ) π t ( y ∣ x ) e η ( r ( x , y ) − r ( x , π t )) = Z t ( x ) p t ( x ) e η ( r x ⋆ − r ( x , π t )) ≥ p t ( x ) . (32)
Taking reciprocals and averaging over prompts gives the response coverage bound.
Lemma B.10 (One-step suboptimality). For the exact mirror descent iterates with η > 0 \eta> 0 η > 0 , fix an iteration t ≥ 0 t \ge0 t ≥ 0 and a prompt x x x with p t ( x ) > 0 p_t(x) > 0 p t ( x ) > 0 . Then
η ( r x ⋆ − J x ( π t + 1 ) ) ≤ log p t + 1 ( x ) p t ( x ) . \eta(r_x^\star-J_x(\pi_{t+1})) \le\log\frac{p_{t+1}(x)}{p_t(x)}. η ( r x ⋆ − J x ( π t + 1 )) ≤ log p t ( x ) p t + 1 ( x ) .
Proof of Lemma B.10. Taking logarithms in Eq. (32) gives
log p t + 1 ( x ) p t ( x ) = η ( r x ⋆ − r ( x , π t ) ) − log Z t ( x ) . \log\frac{p_{t+1}(x)}{p_t(x)} = \eta(r_x^\star-r(x,\pi_t))-\log Z_t(x). log p t ( x ) p t + 1 ( x ) = η ( r x ⋆ − r ( x , π t )) − log Z t ( x ) .
Subtracting Eq. (31) cancels the log partition function, yielding
η ( r x ⋆ − J x ( π t + 1 ) ) = log p t + 1 ( x ) p t ( x ) − D K L ( π t + 1 ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x ) ) ≤ log p t + 1 ( x ) p t ( x ) . \begin{aligned}
\eta(r_x^\star-J_x(\pi_{t+1})) = \log\frac{p_{t+1}(x)}{p_t(x)}-D_{\mathrm{KL}}(\pi_{t+1}(\cdot\mid x)\mathbin{\|}\pi_t(\cdot\mid x)) \\
&\le\log\frac{p_{t+1}(x)}{p_t(x)}.
\end{aligned} η ( r x ⋆ − J x ( π t + 1 )) = log p t ( x ) p t + 1 ( x ) − D KL ( π t + 1 ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x )) ≤ log p t ( x ) p t + 1 ( x ) .
Softmax Linear Policies
Linear Reward and Mirror Descent Realizability
Lemma B.11 (Target realizability). Consider the softmax linear class Π Θ \Pi_\Theta Π Θ with Θ = R d \Theta= \mathbb{R}^d Θ = R d . Fix π θ ∈ Π Θ \pi_\theta\in\Pi_\Theta π θ ∈ Π Θ and η > 0 \eta> 0 η > 0 , and define
π η ⋆ ( y ∣ x ) = π θ ( y ∣ x ) e η r ( x , y ) ∑ y ′ π θ ( y ′ ∣ x ) e η r ( x , y ′ ) . \pi_\eta^\star(y\mid x)=\frac{\pi_\theta(y\mid x)e^{\eta r(x,y)}}{\sum_{y'}\pi_\theta(y'\mid x)e^{\eta r(x,y')}}. π η ⋆ ( y ∣ x ) = ∑ y ′ π θ ( y ′ ∣ x ) e η r ( x , y ′ ) π θ ( y ∣ x ) e η r ( x , y ) . Then π η ⋆ ∈ Π Θ \pi_\eta^\star\in\Pi_\Theta π η ⋆ ∈ Π Θ if and only if there exist θ ⋆ ∈ R d \theta_\star\in\mathbb{R}^d θ ⋆ ∈ R d and c : X → R c:\mathcal{X}\to\mathbb{R} c : X → R such that
r ( x , y ) = ϕ ( x , y ) ⊤ θ ⋆ + c ( x ) for all ( x , y ) ∈ X × Y . (33) r(x,y)=\phi(x,y)^\top\theta_\star+c(x)\quad\text{for all }(x,y)\in\mathcal{X}\times\mathcal{Y}. \tag*{(33)} r ( x , y ) = ϕ ( x , y ) ⊤ θ ⋆ + c ( x ) for all ( x , y ) ∈ X × Y . (33) In this case,
π η ⋆ = π θ + η θ ⋆ . (34) \pi_\eta^\star=\pi_{\theta+\eta\theta_\star}. \tag*{(34)} π η ⋆ = π θ + η θ ⋆ . (34)
The inner loop of APMD approximates π η , t ⋆ \pi_{\eta,t}^\star π η , t ⋆ within the policy class Π Θ \Pi_\Theta Π Θ . The linear reward assumption in Assumption 2.1 ensures that this target is realizable at every iteration. Lemma B.11 shows that target realizability is equivalent to reward linearity up to a prompt-dependent additive constant, which cancels when the target is normalized.
Proof of Lemma B.11. Suppose π η ⋆ = π θ ′ \pi_{\eta}^{\star}=\pi_{\theta'} π η ⋆ = π θ ′ for some θ ′ ∈ R d \theta' \in\mathbb{R}^{d} θ ′ ∈ R d . For every prompt x x x and responses y , y ′ y,y' y , y ′ , the definition of π η ⋆ \pi_{\eta}^{\star} π η ⋆ and the softmax formula give
η ( r ( x , y ) − r ( x , y ′ ) ) = log π θ ′ ( y ∣ x ) π θ ′ ( y ′ ∣ x ) − log π θ ( y ∣ x ) π θ ( y ′ ∣ x ) = ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) T ( θ ′ − θ ) . \begin{aligned}
\eta(r(x,y)-r(x,y'))=\log\frac{\pi_{\theta'}(y\mid x)}{\pi_{\theta'}(y'\mid x)}-\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\theta}(y'\mid x)} \\
&=(\phi(x,y)-\phi(x,y'))^{\mathsf{T}}(\theta'-\theta).
\end{aligned} η ( r ( x , y ) − r ( x , y ′ )) = log π θ ′ ( y ′ ∣ x ) π θ ′ ( y ∣ x ) − log π θ ( y ′ ∣ x ) π θ ( y ∣ x ) = ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) T ( θ ′ − θ ) .
Set θ ⋆ = ( θ ′ − θ ) / η \theta_{\star}=(\theta'-\theta)/\eta θ ⋆ = ( θ ′ − θ ) / η . The displayed identity shows that r ( x , y ) − ϕ ( x , y ) T θ ⋆ r(x,y)-\phi(x,y)^{\mathsf{T}}\theta_{\star} r ( x , y ) − ϕ ( x , y ) T θ ⋆ is independent of y y y for each prompt x x x . Denoting this value by c ( x ) c(x) c ( x ) gives Eq. (33) .
Conversely, suppose Eq. (33) holds. The factor e η c ( x ) e^{\eta c(x)} e η c ( x ) is common to all responses to x x x and cancels in the normalization. Thus
π η ⋆ ( y ∣ x ) = π θ ( y ∣ x ) e η r ( x , y ) ∑ y ′ π θ ( y ′ ∣ x ) e η r ( x , y ′ ) = exp ( ϕ ( x , y ) T ( θ + η θ ⋆ ) ) ∑ y ′ exp ( ϕ ( x , y ′ ) T ( θ + η θ ⋆ ) ) = π θ + η θ ⋆ ( y ∣ x ) , \begin{aligned}
\pi_{\eta}^{\star}(y\mid x)=\frac{\pi_{\theta}(y\mid x)e^{\eta r(x,y)}}{\sum_{y'}\pi_{\theta}(y'\mid x)e^{\eta r(x,y')}} \\
&=\frac{\exp(\phi(x,y)^{\mathsf{T}}(\theta+\eta\theta_{\star}))}{\sum_{y'}\exp(\phi(x,y')^{\mathsf{T}}(\theta+\eta\theta_{\star}))}=\pi_{\theta+\eta\theta_{\star}}(y\mid x),
\end{aligned} π η ⋆ ( y ∣ x ) = ∑ y ′ π θ ( y ′ ∣ x ) e η r ( x , y ′ ) π θ ( y ∣ x ) e η r ( x , y ) = ∑ y ′ exp ( ϕ ( x , y ′ ) T ( θ + η θ ⋆ )) exp ( ϕ ( x , y ) T ( θ + η θ ⋆ )) = π θ + η θ ⋆ ( y ∣ x ) ,
proving Eq. (34) and membership in Π Θ \Pi_{\Theta} Π Θ .
Properties of Softmax Linear Policies
We now prove various properties of the softmax linear parameterization in Assumption 2.1. Define the feature diameter
D ϕ : = max x ∈ X , y , y ′ ∈ Y ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 . D_{\phi}:=\max_{x\in\mathcal{X},\,y,y'\in\mathcal{Y}}\|\phi(x,y)-\phi(x,y')\|_{2}. D ϕ := x ∈ X , y , y ′ ∈ Y max ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 .
The following properties hold for each prompt x ∈ X x\in\mathcal{X} x ∈ X .
Lemma B.12 (Softmax policy ratio bound). For every θ , θ ′ ∈ R d \theta,\theta'\in\mathbb{R}^{d} θ , θ ′ ∈ R d and ( x , y ) ∈ X × Y (x,y)\in\mathcal{X}\times\mathcal{Y} ( x , y ) ∈ X × Y ,
π θ ′ ( y ∣ x ) π θ ( y ∣ x ) ≤ exp ( D ϕ ∥ θ ′ − θ ∥ 2 ) . \frac{\pi_{\theta'}(y\mid x)}{\pi_{\theta}(y\mid x)}\leq\exp(D_{\phi}\|\theta'-\theta\|_{2}). π θ ( y ∣ x ) π θ ′ ( y ∣ x ) ≤ exp ( D ϕ ∥ θ ′ − θ ∥ 2 ) .
Proof of Lemma B.12 . The softmax formula gives
π θ ′ ( y ∣ x ) π θ ( y ∣ x ) = exp ( ϕ ( x , y ) T ( θ ′ − θ ) ) E y ′ ∼ π θ ( ⋅ ∣ x ) [ exp ( ϕ ( x , y ′ ) T ( θ ′ − θ ) ) ] = 1 E y ′ ∼ π θ ( ⋅ ∣ x ) [ exp ( ( ϕ ( x , y ′ ) − ϕ ( x , y ) ) T ( θ ′ − θ ) ) ] ≤ exp ( D ϕ ∥ θ ′ − θ ∥ 2 ) , \begin{aligned}
\frac{\pi_{\theta'}(y\mid x)}{\pi_{\theta}(y\mid x)}
=\frac{\exp(\phi(x,y)^{\mathsf{T}}(\theta'-\theta))}{\mathbb{E}_{y'\sim\pi_{\theta}(\cdot\mid x)}[\exp(\phi(x,y')^{\mathsf{T}}(\theta'-\theta))]} \\
&=\frac{1}{\mathbb{E}_{y'\sim\pi_{\theta}(\cdot\mid x)}[\exp((\phi(x,y')-\phi(x,y))^{\mathsf{T}}(\theta'-\theta))]} \\
&\leq\exp(D_{\phi}\|\theta'-\theta\|_{2}),
\end{aligned} π θ ( y ∣ x ) π θ ′ ( y ∣ x ) = E y ′ ∼ π θ ( ⋅ ∣ x ) [ exp ( ϕ ( x , y ′ ) T ( θ ′ − θ ))] exp ( ϕ ( x , y ) T ( θ ′ − θ )) = E y ′ ∼ π θ ( ⋅ ∣ x ) [ exp (( ϕ ( x , y ′ ) − ϕ ( x , y ) ) T ( θ ′ − θ ))] 1 ≤ exp ( D ϕ ∥ θ ′ − θ ∥ 2 ) ,
where the last inequality follows from ( ϕ ( x , y ′ ) − ϕ ( x , y ) ) T ( θ ′ − θ ) ≥ − D ϕ ∥ θ ′ − θ ∥ 2 (\phi(x,y')-\phi(x,y))^{\mathsf{T}}(\theta'-\theta)\geq-D_{\phi}\|\theta'-\theta\|_{2} ( ϕ ( x , y ′ ) − ϕ ( x , y ) ) T ( θ ′ − θ ) ≥ − D ϕ ∥ θ ′ − θ ∥ 2 for every y ′ ∈ Y y'\in\mathcal{Y} y ′ ∈ Y by the Cauchy–Schwarz inequality.
Lemma B.13 (Softmax KL bound). For every θ , θ ′ ∈ R d \theta,\theta'\in\mathbb{R}^{d} θ , θ ′ ∈ R d and x ∈ X x\in\mathcal{X} x ∈ X ,
D K L ( π θ ( ⋅ ∣ x ) ∥ π θ ′ ( ⋅ ∣ x ) ) ≤ D ϕ 2 8 ∥ θ ′ − θ ∥ 2 2 . D_{\mathrm{KL}}(\pi_{\theta}(\cdot\mid x)\mathbin\Vert\pi_{\theta'}(\cdot\mid x))\leq\frac{D_{\phi}^{2}}{8}\|\theta'-\theta\|_{2}^{2}. D KL ( π θ ( ⋅ ∣ x ) ∥ π θ ′ ( ⋅ ∣ x )) ≤ 8 D ϕ 2 ∥ θ ′ − θ ∥ 2 2 .
Proof of Lemma B.13 . Fix x ∈ X x\in\mathcal{X} x ∈ X . Under y ∼ π θ ( ⋅ ∣ x ) y\sim\pi_{\theta}(\cdot\mid x) y ∼ π θ ( ⋅ ∣ x ) , the random variable ϕ ( x , y ) T ( θ ′ − θ ) \phi(x,y)^{\mathsf{T}}(\theta'-\theta) ϕ ( x , y ) T ( θ ′ − θ ) has range of length at most D ϕ ∥ θ ′ − θ ∥ 2 D_{\phi}\|\theta'-\theta\|_{2} D ϕ ∥ θ ′ − θ ∥ 2 by the definition of D ϕ D_{\phi} D ϕ . The softmax formula and Hoeffding’s lemma therefore give
D K L ( π θ ( ⋅ ∣ x ) ∥ π θ ′ ( ⋅ ∣ x ) ) = log E y ∼ π θ ( ⋅ ∣ x ) [ e ϕ ( x , y ) T ( θ ′ − θ ) ] − E y ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ) T ( θ ′ − θ ) ] ≤ D ϕ 2 8 ∥ θ ′ − θ ∥ 2 2 . \begin{aligned}
D_{\mathrm{KL}}(\pi_{\theta}(\cdot\mid x)\mathbin\Vert\pi_{\theta'}(\cdot\mid x))
=\log\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[e^{\phi(x,y)^{\mathsf{T}}(\theta'-\theta)}\right]-\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}[\phi(x,y)^{\mathsf{T}}(\theta'-\theta)] \\
&\leq\frac{D_{\phi}^{2}}{8}\|\theta'-\theta\|_{2}^{2}.
\end{aligned} D KL ( π θ ( ⋅ ∣ x ) ∥ π θ ′ ( ⋅ ∣ x )) = log E y ∼ π θ ( ⋅ ∣ x ) [ e ϕ ( x , y ) T ( θ ′ − θ ) ] − E y ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ) T ( θ ′ − θ )] ≤ 8 D ϕ 2 ∥ θ ′ − θ ∥ 2 2 .
Lemma B.14 (Softmax score identity). For every θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d and ( x , y ) ∈ X × Y (x,y) \in\mathcal{X} \times\mathcal{Y} ( x , y ) ∈ X × Y ,
∇ log π θ ( y ∣ x ) = ϕ ( x , y ) − E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] . \nabla\log\pi_{\theta}(y \mid x) = \phi(x,y) - \mathbb{E}_{y'\sim\pi_{\theta}(\cdot\mid x)}[\phi(x,y')]. ∇ log π θ ( y ∣ x ) = ϕ ( x , y ) − E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] .
Proof of Lemma B.14 . Differentiating the log policy gives
∇ log π θ ( y ∣ x ) = ∇ ( ϕ ( x , y ) ⊤ θ − log ∑ y ′ ∈ Y exp ( ϕ ( x , y ′ ) ⊤ θ ) ) = ϕ ( x , y ) − E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] . \begin{aligned}
\nabla\log\pi_{\theta}(y \mid x)
&= \nabla\left(\phi(x,y)^{\top}\theta- \log\sum_{y'\in\mathcal{Y}}\exp(\phi(x,y')^{\top}\theta)\right) \\
&= \phi(x,y) - \mathbb{E}_{y'\sim\pi_{\theta}(\cdot\mid x)}[\phi(x,y')].
\end{aligned} ∇ log π θ ( y ∣ x ) = ∇ ϕ ( x , y ) ⊤ θ − log y ′ ∈ Y ∑ exp ( ϕ ( x , y ′ ) ⊤ θ ) = ϕ ( x , y ) − E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] .
Lemma B.15 (Softmax score bound). For every θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d and ( x , y ) ∈ X × Y (x,y) \in\mathcal{X} \times\mathcal{Y} ( x , y ) ∈ X × Y ,
∥ ∇ log π θ ( y ∣ x ) ∥ 2 ≤ D ϕ . \lVert\nabla\log\pi_{\theta}(y \mid x)\rVert_{2} \le D_{\phi}. ∥ ∇ log π θ ( y ∣ x ) ∥ 2 ≤ D ϕ .
Proof of Lemma B.15 . The score identity in Lemma B.14 gives
∥ ∇ log π θ ( y ∣ x ) ∥ 2 = ∥ E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ) − ϕ ( x , y ′ ) ] ∥ 2 ≤ E y ′ ∼ π θ ( ⋅ ∣ x ) [ ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 ] ≤ D ϕ . \begin{aligned}
\lVert\nabla\log\pi_{\theta}(y \mid x)\rVert_{2}
&= \left\lVert\mathbb{E}_{y'\sim\pi_{\theta}(\cdot\mid x)}[\phi(x,y)-\phi(x,y')]\right\rVert_{2} \\
&\le\mathbb{E}_{y'\sim\pi_{\theta}(\cdot\mid x)}[\lVert\phi(x,y)-\phi(x,y')\rVert_{2}] \le D_{\phi}.
\end{aligned} ∥ ∇ log π θ ( y ∣ x ) ∥ 2 = E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ) − ϕ ( x , y ′ )] 2 ≤ E y ′ ∼ π θ ( ⋅ ∣ x ) [∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 ] ≤ D ϕ .
Lemma B.16 (Softmax log-policy Hessian). For every θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d and ( x , y ) ∈ X × Y (x,y) \in\mathcal{X} \times\mathcal{Y} ( x , y ) ∈ X × Y ,
− ∇ 2 log π θ ( y ∣ x ) = Cov y ′ ∼ π θ ( ⋅ ∣ x ) ( ϕ ( x , y ′ ) ) . -\nabla^{2}\log\pi_{\theta}(y \mid x) = \operatorname{Cov}_{y'\sim\pi_{\theta}(\cdot\mid x)}(\phi(x,y')). − ∇ 2 log π θ ( y ∣ x ) = Cov y ′ ∼ π θ ( ⋅ ∣ x ) ( ϕ ( x , y ′ )) .
Proof of Lemma B.16 . Differentiating the score identity in Lemma B.14 yields
− ∇ 2 log π θ ( y ∣ x ) = ∑ y ′ ∈ Y π θ ( y ′ ∣ x ) ϕ ( x , y ′ ) ( ∇ log π θ ( y ′ ∣ x ) ) ⊤ = E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ϕ ( x , y ′ ) ⊤ ] − E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] ⊤ = Cov y ′ ∼ π θ ( ⋅ ∣ x ) ( ϕ ( x , y ′ ) ) . \begin{aligned}
-\nabla^{2}\log\pi_{\theta}(y \mid x)
&= \sum_{y'\in\mathcal{Y}}\pi_{\theta}(y'\mid x)\phi(x,y')(\nabla\log\pi_{\theta}(y'\mid x))^{\top} \\
&= \mathbb{E}_{y'\sim\pi_{\theta}(\cdot\mid x)}[\phi(x,y')\phi(x,y')^{\top}] \\
&\quad- \mathbb{E}_{y'\sim\pi_{\theta}(\cdot\mid x)}[\phi(x,y')]\mathbb{E}_{y'\sim\pi_{\theta}(\cdot\mid x)}[\phi(x,y')]^{\top} \\
&= \operatorname{Cov}_{y'\sim\pi_{\theta}(\cdot\mid x)}(\phi(x,y')).
\end{aligned} − ∇ 2 log π θ ( y ∣ x ) = y ′ ∈ Y ∑ π θ ( y ′ ∣ x ) ϕ ( x , y ′ ) ( ∇ log π θ ( y ′ ∣ x ) ) ⊤ = E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ϕ ( x , y ′ ) ⊤ ] − E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] ⊤ = Cov y ′ ∼ π θ ( ⋅ ∣ x ) ( ϕ ( x , y ′ )) .
Coverage Collapse under RL with a Fixed Reference (Proof of Theorem 2.1)
This section provides the supplementary material for § 2.1 and Figure 2 .
First, in Appendix § 9 .§ 1 we provide the experimental details behind Figure 2 and Figure 3 as well as the details for the visualization. Then, in Appendix § 9 .§ 2 , we provide the full statement of Theorem 2.1 and the proof, including the construction and explicit constants for the result.
Details for Figs. 2 and 3
Instance. Figure 2 uses a modification of the single-prompt bandit constructed for Theorem 2.1 in Appendix § 9 .§ 2 . There are m + 2 m+2 m + 2 responses, with m = 100 m=100 m = 100 zero-reward responses. The rewards and base model are
r ( 1 ) = 1 , r ( 2 ) = 0.8 , r ( i ) = 0 ( i ≥ 3 ) , r(1)=1,\qquad r(2)=0.8,\qquad r(i)=0\quad(i\ge3), r ( 1 ) = 1 , r ( 2 ) = 0.8 , r ( i ) = 0 ( i ≥ 3 ) ,
π b a s e ( 1 ) = 1 16 , π b a s e ( 2 ) = 1 3 , π b a s e ( i ) = 1 − 1 / 16 − 1 / 3 m ( i ≥ 3 ) . \pi_{\mathrm{base}}(1)=\frac{1}{16},\qquad\pi_{\mathrm{base}}(2)=\frac{1}{3},\qquad\pi_{\mathrm{base}}(i)=\frac{1-1/16-1/3}{m}\quad(i\ge3). π base ( 1 ) = 16 1 , π base ( 2 ) = 3 1 , π base ( i ) = m 1 − 1/16 − 1/3 ( i ≥ 3 ) .
We use tabular softmax policies π θ ( i ) = exp ( θ i ) / ∑ j = 1 m + 2 exp ( θ j ) \pi_\theta(i)=\exp(\theta_i)/\sum_{j=1}^{m+2}\exp(\theta_j) π θ ( i ) = exp ( θ i ) / ∑ j = 1 m + 2 exp ( θ j ) , where θ i \theta_i θ i is the unnormalized score, or logit, for response i i i . Let θ b a s e \theta_{\mathrm{base}} θ base denote the initial logit vector representing π b a s e \pi_{\mathrm{base}} π base . We fix the target suboptimality at ε = 0.01 \varepsilon=0.01 ε = 0.01 . Relative to the lower-bound construction, we increase the reward gap between the optimal response and the closest alternative from 3 ε 3\varepsilon 3 ε to 0.2 0.2 0.2 , and reduce the base model’s probability of the optimal response from 1 / 6 1/6 1/6 to 1 / 16 1/16 1/16 . These changes make coverage collapse and the reward plateau more visually apparent than in the construction used for the worst-case analysis.
Reading Figure 2 . The horizontal axes in panels (b,c) are coordinates ( u , v ) (u,v) ( u , v ) in a two-dimensional parameter plane. In the displayed view, u u u increases from left to right, while v v v increases from the lower foreground to the upper background. Increasing u u u raises the optimal response’s logit relative to that of response 2. Increasing v v v raises both rewarding responses’ logits relative to the zero-reward logits, without changing the relative probability of responses 1 and 2.
Surface height gives the fixed-reference objective J η J_\eta J η (Eq. (1) ), while color shows expected reward in (b) and coverage C π θ C_{\pi_\theta} C π θ in (c).
Parameter coordinates. These coordinates represent the training trajectories exactly. Both methods use exact policy gradients, as specified below. The zero-reward logits start equal and receive equal updates by symmetry, so they remain equal throughout training. The gradient coordinates in Eq. (43) sum to zero, so ∑ i = 1 m + 2 θ i \sum_{i=1}^{m+2}\theta_i ∑ i = 1 m + 2 θ i remains fixed at its initial value. The same identity holds with the current reference policy in place of π b a s e \pi_{\mathrm{base}} π base for APMD, so both trajectories lie in the plane through θ b a s e \theta_{\mathrm{base}} θ base with orthonormal basis
b 1 = ( 1 , − 1 , 0 , … , 0 ) 2 , b 2 = ( 1 , 1 , − 2 / m , … , − 2 / m ) 2 + 4 / m . b_1=\frac{(1,-1,0,\ldots,0)}{\sqrt{2}},\qquad b_2=\frac{(1,1,-2/m,\ldots,-2/m)}{\sqrt{2+4/m}}. b 1 = 2 ( 1 , − 1 , 0 , … , 0 ) , b 2 = 2 + 4/ m ( 1 , 1 , − 2/ m , … , − 2/ m ) .
For a policy π θ \pi_\theta π θ with logits θ \theta θ in this plane, its coordinates and reconstruction are
u = b 1 T ( θ − θ b a s e ) , v = b 2 T ( θ − θ b a s e ) , u=b_1^{\mathsf T}(\theta-\theta_{\mathrm{base}}),\qquad v=b_2^{\mathsf T}(\theta-\theta_{\mathrm{base}}), u = b 1 T ( θ − θ base ) , v = b 2 T ( θ − θ base ) ,
θ ( u , v ) = θ b a s e + u b 1 + v b 2 . \theta(u,v)=\theta_{\mathrm{base}}+u b_1+v b_2. θ ( u , v ) = θ base + u b 1 + v b 2 .
Thus, the initial base model has ( u , v ) = ( 0 , 0 ) (u,v)=(0,0) ( u , v ) = ( 0 , 0 ) . Because the basis is orthonormal, the coordinates preserve Euclidean distances and gradient steps within the plane. Coverage is measured by C π θ ( u , v ) = 1 / π θ ( u , v ) ( 1 ) C_{\pi_{\theta(u,v)}}=1/\pi_{\theta(u,v)}(1) C π θ ( u , v ) = 1/ π θ ( u , v ) ( 1 ) , where smaller values indicate better coverage.
Methods and tuning. Both methods start from π b a s e \pi_{\mathrm{base}} π base and perform 10,000 gradient steps with a constant learning rate α = 1 \alpha=1 α = 1 . REINFORCE performs exact gradient ascent on J η ( π θ ) = J ( π θ ) − η − 1 D K L ( π θ ∥ π b a s e ) J_\eta(\pi_\theta)=J(\pi_\theta)-\eta^{-1}D_{\mathrm{KL}}(\pi_\theta\mathbin\Vert\pi_{\mathrm{base}}) J η ( π θ ) = J ( π θ ) − η − 1 D KL ( π θ ∥ π base ) . APMD instantiates Algorithm 1 with exact population gradients in place of the sampled gradient estimates. At each outer iteration, it takes K = 100 K=100 K = 100 gradient ascent steps on J η , t ( π θ ) = J ( π θ ) − η − 1 D K L ( π θ ∥ π t ) J_{\eta,t}(\pi_\theta)=J(\pi_\theta)-\eta^{-1}D_{\mathrm{KL}}(\pi_\theta\mathbin\Vert\pi_t) J η , t ( π θ ) = J ( π θ ) − η − 1 D KL ( π θ ∥ π t ) , then uses π t + 1 = π t K \pi_{t+1}=\pi_t^K π t + 1 = π t K as the next reference policy. It therefore performs T = 100 T=100 T = 100 outer iterations. The horizontal axis in Figure 2 a counts individual gradient steps n = t K + k n=tK+k n = t K + k , not reference resets.
For REINFORCE, we choose η ≈ 23.64513 \eta\approx23.64513 η ≈ 23.64513 by solving 1 − J ( π η ⋆ ) = 0.9 ϵ 1-J(\pi_\eta^\star)=0.9\epsilon 1 − J ( π η ⋆ ) = 0.9 ϵ , where π η ⋆ ( i ) ∝ π b a s e ( i ) e η r ( i ) \pi_\eta^\star(i)\propto\pi_{\mathrm{base}}(i)e^{\eta r(i)} π η ⋆ ( i ) ∝ π base ( i ) e η r ( i ) . This is the strongest regularization for which the optimal regularized policy π η ⋆ \pi_\eta^\star π η ⋆ has suboptimality at least 10% below the target ϵ \epsilon ϵ . For APMD, we fix η = 1 / ( B ϕ B ⋆ ) = 1 / 1 + 0.8 2 ≈ 0.78087 \eta=1/(B_\phi B_\star)=1/\sqrt{1+0.8^2}\approx0.78087 η = 1/ ( B ϕ B ⋆ ) = 1/ 1 + 0. 8 2 ≈ 0.78087 , using standard basis vectors as features and the reward parameter θ ⋆ = r \theta_\star=r θ ⋆ = r .
To choose the learning rate, we compare α ∈ { 0.1 , 0.3 , 1 } \alpha\in\{0.1,0.3,1\} α ∈ { 0.1 , 0.3 , 1 } in runs with a fixed reference at both values of η \eta η . We use 100,000 steps for η ≈ 23.64513 \eta\approx23.64513 η ≈ 23.64513 and 1,000 steps for η ≈ 0.78087 \eta\approx0.78087 η ≈ 0.78087 . The selected learning rate α = 1 \alpha=1 α = 1 gives the fastest reward improvement in both sweeps.
With α = 1 \alpha=1 α = 1 , we test K ∈ { 1 , 5 , 10 , 50 , 100 } K\in\{1,5,10,50,100\} K ∈ { 1 , 5 , 10 , 50 , 100 } for APMD over 10,000 steps. The choice K = 10 K=10 K = 10 reaches reward 0.99 fastest, in 520 steps, compared with 2,911 for K = 100 K=100 K = 100 . We nevertheless use K = 100 K=100 K = 100 to show more optimization within each subproblem. The common budget of 10,000 steps extends beyond REINFORCE’s first attainment of reward 0.99, at step 7,364.
Optimization landscapes. Figure 3 c displays the optimization landscape of APMD in the same run as Figure 2 . Figure 3 a matches Figure 2 b, while Figure 3 b is the strongly regularized version that matches the lowest manifold in Figure 3 c.
Proof of 2.1
Instance and notation. Fix ε ∈ ( 0 , 1 64 ] \varepsilon\in(0,\frac{1}{64}] ε ∈ ( 0 , 64 1 ] and consider a single-prompt bandit with response set Y = [ m + 2 ] \mathcal{Y} = [m+2] Y = [ m + 2 ] , where m = ⌈ 1 / ε ⌉ m = \lceil1/\varepsilon\rceil m = ⌈ 1/ ε ⌉ is the number of zero-reward responses. The rewards are
r ( 1 ) = 1 , r ( 2 ) = 1 − 3 ε , r ( i ) = 0 for every i ≥ 3. (35) r(1)=1,\qquad r(2)=1-3\varepsilon,\qquad r(i)=0\quad\text{for every }i\ge3. \tag*{(35)} r ( 1 ) = 1 , r ( 2 ) = 1 − 3 ε , r ( i ) = 0 for every i ≥ 3. (35)
The base model assigns twice as much probability to response 2 as to the optimal response 1, and divides the remaining probability equally among the zero-reward responses,
π b a s e ( 1 ) = 1 6 , π b a s e ( 2 ) = 1 3 , π b a s e ( i ) = 1 2 m for every i ≥ 3. (36) \pi_{\mathrm{base}}(1)=\frac{1}{6},\qquad\pi_{\mathrm{base}}(2)=\frac{1}{3},\qquad\pi_{\mathrm{base}}(i)=\frac{1}{2m}\quad\text{for every }i\ge3. \tag*{(36)} π base ( 1 ) = 6 1 , π base ( 2 ) = 3 1 , π base ( i ) = 2 m 1 for every i ≥ 3. (36)
Thus J ⋆ = 1 J^\star=1 J ⋆ = 1 and C π b a s e = 1 / π b a s e ( 1 ) = 6 C_{\pi_{\mathrm{base}}}=1/\pi_{\mathrm{base}}(1)=6 C π base = 1/ π base ( 1 ) = 6 . We use the tabular softmax policy class
π θ ( i ) = exp ( θ i ) ∑ j = 1 m + 2 exp ( θ j ) . \pi_\theta(i)=\frac{\exp(\theta_i)}{\sum_{j=1}^{m+2}\exp(\theta_j)}. π θ ( i ) = ∑ j = 1 m + 2 exp ( θ j ) exp ( θ i ) .
Together with a one-hot feature vector for each response, this instance satisfies Assumption 2.1 with R max = 1 R_{\max}=1 R m a x = 1 .
For η ∈ ( 0 , ∞ ] \eta\in(0,\infty] η ∈ ( 0 , ∞ ] , write β : = 1 / η \beta:=1/\eta β := 1/ η , with 1 / ∞ = 0 1/\infty=0 1/∞ = 0 . The exact policy-gradient update in Eq. (2) optimizes the fixed-reference objective
J η ( π ) : = J ( π ) − β D K L ( π ∥ π b a s e ) , θ t + 1 = θ t + α ∇ J η ( π t ) . J_\eta(\pi):=J(\pi)-\beta D_{\mathrm{KL}}(\pi\mathbin\Vert\pi_{\mathrm{base}}),\qquad\theta_{t+1}=\theta_t+\alpha\nabla J_\eta(\pi_t). J η ( π ) := J ( π ) − β D KL ( π ∥ π base ) , θ t + 1 = θ t + α ∇ J η ( π t ) .
We denote its maximizer over the policy simplex by
π η ⋆ : = arg max π J η ( π ) . \pi_\eta^\star:=\underset{\pi}{\arg\max}\ J_\eta(\pi). π η ⋆ := π arg max J η ( π ) .
The following theorem is the full version of Theorem 2.1. It shows that the instance above has constant initial coverage, but becoming and remaining ε \varepsilon ε -optimal requires passing through a policy with exponentially worse coverage.
Theorem C.1 (Coverage collapse in REINFORCE, full version of Theorem 2.1). Fix ε ∈ ( 0 , 1 64 ] \varepsilon\in(0,\frac{1}{64}] ε ∈ ( 0 , 64 1 ] , and consider the single-prompt bandit in Eqs. (35) and (36) with the tabular softmax policy class. Fix any η > 0 \eta>0 η > 0 , initialize π 0 = π b a s e \pi_0=\pi_{\mathrm{base}} π 0 = π base , and run Eq. (2) with any fixed learning rate α ∈ ( 0 , 1 5 ] \alpha\in(0,\frac{1}{5}] α ∈ ( 0 , 5 1 ] . For every T ≥ 1 T\ge1 T ≥ 1 , if J ⋆ − J ( π t ) ≤ ε J^\star-J(\pi_t)\le\varepsilon J ⋆ − J ( π t ) ≤ ε for every t ≥ T t\ge T t ≥ T , then there exists an iteration ρ \rho ρ with 0 < ρ < T 0<\rho<T 0 < ρ < T such that
1 7 e 1 / ( 4 ε ) < C π ρ ≤ 3 e 1 / ( 4 ε ) and T ≥ e 1 / ( 4 ε ) 630 α ε ≥ C π ρ 1890 α ε . (37) \frac{1}{7}e^{1/(4\sqrt{\varepsilon})}<C_{\pi_\rho}\le3e^{1/(4\sqrt{\varepsilon})}
\qquad\text{and}\qquad
T\ge\frac{e^{1/(4\sqrt{\varepsilon})}}{630\alpha\sqrt{\varepsilon}}\ge\frac{C_{\pi_\rho}}{1890\alpha\sqrt{\varepsilon}}. \tag*{(37)} 7 1 e 1/ ( 4 ε ) < C π ρ ≤ 3 e 1/ ( 4 ε ) and T ≥ 630 α ε e 1/ ( 4 ε ) ≥ 1890 α ε C π ρ . (37) For this same ρ \rho ρ , the expected reward increases strictly throughout the coverage-collapse phase,
J ( π t + 1 ) > J ( π t ) for every 0 ≤ t < ρ . (38) J(\pi_{t+1})>J(\pi_t)\qquad\text{for every }0\le t<\rho. \tag*{(38)} J ( π t + 1 ) > J ( π t ) for every 0 ≤ t < ρ . (38) These conclusions also hold for η = ∞ \eta=\infty η = ∞ . In this case, the lower bound on the number of gradient steps strengthens to
T ≥ 2 315 α exp ( 1 16 ε ) . (39) T\ge\frac{2}{315\alpha}\exp\left(\frac{1}{16\varepsilon}\right). \tag*{(39)} T ≥ 315 α 2 exp ( 16 ε 1 ) . (39)
Theorem 2.1 follows by suppressing the numerical constants.
We first show that eventual ε \varepsilon ε -optimality requires $\beta\le 3\varepsilon/\log 4 ({xref}Lemma C.1 and {xref}Eq. (40)`). By symmetry, the zero-reward responses have equal probabilities throughout training, so the dynamics reduce to the probabilities of responses 1, 2, and the total probability of the zero-reward responses. We then analyze two phases.
During coverage collapse, response 2 gains probability relative to response 1 as probability moves away from the zero-reward responses. We show that the probability of response 1 falls to order e − 1 / ( 4 ε ) e^{-1/(4\sqrt{\varepsilon})} e − 1/ ( 4 ε ) , giving the coverage bounds in Eqs. (58) and (59) .
To reach an ϵ \epsilon ϵ -optimal policy, response 1 must become at least as likely as response 2 (Eq. (48) ). We bound how quickly π t ( 2 ) / π t ( 1 ) \pi_t(2)/\pi_t(1) π t ( 2 ) / π t ( 1 ) can decrease from its exponentially large value after coverage collapse. Summing this bound gives the lower bound on the number of gradient steps in Eq. (64) .
Lemma C.1 (Sustained accuracy requires weak regularization). Fix ϵ ∈ ( 0 , 1 / 64 ] \epsilon\in(0,1/64] ϵ ∈ ( 0 , 1/64 ] and m = ⌈ 1 / ϵ ⌉ m=\lceil1/\epsilon\rceil m = ⌈ 1/ ϵ ⌉ . On Y = [ m + 2 ] \mathcal{Y}=[m+2] Y = [ m + 2 ] , let
r ( 1 ) = 1 , r ( 2 ) = 1 − 3 ϵ , r ( i ) = 0 ( i ≥ 3 ) , r(1)=1,\qquad r(2)=1-3\epsilon,\qquad r(i)=0\quad(i\ge3), r ( 1 ) = 1 , r ( 2 ) = 1 − 3 ϵ , r ( i ) = 0 ( i ≥ 3 ) , π b a s e ( 1 ) = 1 6 , π b a s e ( 2 ) = 1 3 , π b a s e ( i ) = 1 2 m ( i ≥ 3 ) . \pi_{\mathrm{base}}(1)=\frac{1}{6},\qquad\pi_{\mathrm{base}}(2)=\frac{1}{3},\qquad\pi_{\mathrm{base}}(i)=\frac{1}{2m}\quad(i\ge3). π base ( 1 ) = 6 1 , π base ( 2 ) = 3 1 , π base ( i ) = 2 m 1 ( i ≥ 3 ) . For η > 0 \eta>0 η > 0 , define J ( π ) = ∑ i π ( i ) r ( i ) J(\pi)=\sum_i\pi(i)r(i) J ( π ) = ∑ i π ( i ) r ( i ) and J η ( π ) = J ( π ) − η − 1 D K L ( π ∥ π b a s e ) J_\eta(\pi)=J(\pi)-\eta^{-1}D_{\mathrm{KL}}(\pi\mathbin\Vert\pi_{\mathrm{base}}) J η ( π ) = J ( π ) − η − 1 D KL ( π ∥ π base ) . Starting from finite logits θ 0 \theta_0 θ 0 with π θ 0 = π b a s e \pi_{\theta_0}=\pi_{\mathrm{base}} π θ 0 = π base , run exact gradient ascent with a fixed 0 < α ≤ 1 / 5 0<\alpha\le1/5 0 < α ≤ 1/5 :
π t ( i ) = e θ t , i ∑ j e θ t , j , θ t + 1 = θ t + α ∇ J η ( π θ ) ∣ θ = θ t . \pi_t(i)=\frac{e^{\theta_{t,i}}}{\sum_j e^{\theta_{t,j}}},\qquad\theta_{t+1}=\theta_t+\alpha\nabla J_\eta(\pi_\theta)\big|_{\theta=\theta_t}. π t ( i ) = ∑ j e θ t , j e θ t , i , θ t + 1 = θ t + α ∇ J η ( π θ ) θ = θ t . If 1 − J ( π t ) ≤ ϵ 1-J(\pi_t)\le\epsilon 1 − J ( π t ) ≤ ϵ for every t ≥ T t\ge T t ≥ T for some finite T T T , then
1 − J ( π η ⋆ ) ≤ ϵ , η ≥ log 4 3 ϵ , where π η ⋆ ( i ) = π b a s e ( i ) e η r ( i ) ∑ j π b a s e ( j ) e η r ( j ) . 1-J(\pi_\eta^\star)\le\epsilon,\qquad\eta\ge\frac{\log4}{3\epsilon},\qquad\text{where}\qquad\pi_\eta^\star(i)=\frac{\pi_{\mathrm{base}}(i)e^{\eta r(i)}}{\sum_j\pi_{\mathrm{base}}(j)e^{\eta r(j)}}. 1 − J ( π η ⋆ ) ≤ ϵ , η ≥ 3 ϵ log 4 , where π η ⋆ ( i ) = ∑ j π base ( j ) e η r ( j ) π base ( i ) e η r ( i ) .
Proof of Lemma C.1. Define
Q t ( i ) = r ( i ) − η − 1 log π t ( i ) π b a s e ( i ) , Q_t(i)=r(i)-\eta^{-1}\log\frac{\pi_t(i)}{\pi_{\mathrm{base}}(i)}, Q t ( i ) = r ( i ) − η − 1 log π base ( i ) π t ( i ) ,
g t , i = π t ( i ) ( Q t ( i ) − ∑ j π t ( j ) Q t ( j ) ) . g_{t,i}=\pi_t(i)\left(Q_t(i)-\sum_j\pi_t(j)Q_t(j)\right). g t , i = π t ( i ) ( Q t ( i ) − j ∑ π t ( j ) Q t ( j ) ) .
Differentiating the objective gives θ t + 1 , i = θ t , i + α g t , i \theta_{t+1,i}=\theta_{t,i}+\alpha g_{t,i} θ t + 1 , i = θ t , i + α g t , i . In particular, ∑ i g t , i = 0 \sum_i g_{t,i}=0 ∑ i g t , i = 0 , so ∑ i θ t , i \sum_i\theta_{t,i} ∑ i θ t , i is constant. The zero-reward logits remain equal by symmetry of the initialization and update.
We first rule out η < 1 / 5 \eta<1/5 η < 1/5 under the stated accuracy condition. For every t ≥ T t\ge T t ≥ T ,
ϵ ≥ 1 − J ( π t ) = 3 ϵ π t ( 2 ) + ∑ i ≥ 3 π t ( i ) ≥ 3 ϵ ( 1 − π t ( 1 ) ) . \epsilon\ge1-J(\pi_t)=3\epsilon\pi_t(2)+\sum_{i\ge3}\pi_t(i)\ge3\epsilon(1-\pi_t(1)). ϵ ≥ 1 − J ( π t ) = 3 ϵ π t ( 2 ) + i ≥ 3 ∑ π t ( i ) ≥ 3 ϵ ( 1 − π t ( 1 )) .
Hence π t ( 1 ) ≥ 2 / 3 \pi_t(1)\ge2/3 π t ( 1 ) ≥ 2/3 , π t ( 2 ) ≤ 1 / 3 \pi_t(2)\le1/3 π t ( 2 ) ≤ 1/3 , and ∑ i ≥ 3 π t ( i ) ≤ ϵ \sum_{i\ge3}\pi_t(i)\le\epsilon ∑ i ≥ 3 π t ( i ) ≤ ϵ . Using symmetry for the zero-reward responses gives
π t ( 1 ) π b a s e ( 1 ) ≥ 4 , π t ( 2 ) π b a s e ( 2 ) ≤ 1 , \frac{\pi_t(1)}{\pi_{\mathrm{base}}(1)}\ge4,\qquad\frac{\pi_t(2)}{\pi_{\mathrm{base}}(2)}\le1, π base ( 1 ) π t ( 1 ) ≥ 4 , π base ( 2 ) π t ( 2 ) ≤ 1 ,
π t ( i ) π b a s e ( i ) = 2 ∑ j ≥ 3 π t ( j ) ≤ 2 ϵ ≤ 1 ( i ≥ 3 ) . \frac{\pi_t(i)}{\pi_{\mathrm{base}}(i)}=2\sum_{j\ge3}\pi_t(j)\le2\epsilon\le1\qquad(i\ge3). π base ( i ) π t ( i ) = 2 j ≥ 3 ∑ π t ( j ) ≤ 2 ϵ ≤ 1 ( i ≥ 3 ) .
Therefore, for every i ≠ 1 i\ne1 i = 1 ,
Q t ( 1 ) − Q t ( i ) ≤ 1 − log 4 η = − c , c : = log 4 η − 1 > 0 , Q_t(1)-Q_t(i)\le1-\frac{\log4}{\eta}=-c,\qquad c:=\frac{\log4}{\eta}-1>0, Q t ( 1 ) − Q t ( i ) ≤ 1 − η log 4 = − c , c := η log 4 − 1 > 0 ,
g t , 1 ≤ − c π t ( 1 ) ( 1 − π t ( 1 ) ) < 0. g_{t,1}\le-c\pi_t(1)(1-\pi_t(1))<0. g t , 1 ≤ − c π t ( 1 ) ( 1 − π t ( 1 )) < 0.
Thus θ t , 1 \theta_{t,1} θ t , 1 decreases for t ≥ T t\ge T t ≥ T . Since π t ( 1 ) ≥ 2 / 3 \pi_t(1)\ge2/3 π t ( 1 ) ≥ 2/3 , it remains the largest logit. Writing S = ∑ i θ 0 , i S=\sum_i\theta_{0,i} S = ∑ i θ 0 , i , conservation of the sum implies
S − ( m + 1 ) θ T , 1 ≤ θ t , i ≤ θ T , 1 ( t ≥ T , i ∈ [ m + 2 ] ) . S-(m+1)\theta_{T,1}\le\theta_{t,i}\le\theta_{T,1}\qquad(t\ge T,\ i\in[m+2]). S − ( m + 1 ) θ T , 1 ≤ θ t , i ≤ θ T , 1 ( t ≥ T , i ∈ [ m + 2 ]) .
All probabilities are consequently bounded away from zero uniformly in t ≥ T t\ge T t ≥ T . The bound on g t , 1 g_{t,1} g t , 1 then makes θ t + 1 , 1 − θ t , 1 \theta_{t+1,1}-\theta_{t,1} θ t + 1 , 1 − θ t , 1 bounded above by a strictly negative constant, contradicting the displayed lower bound.
It remains to consider η ≥ 1 / 5 \eta\ge1/5 η ≥ 1/5 , for which α ≤ η \alpha\le\eta α ≤ η . We show directly that π t → π η ⋆ \pi_t \to\pi_\eta^\star π t → π η ⋆ . Set
θ η , i ⋆ = log π b a s e ( i ) + η r ( i ) , s t , i = θ t , i − θ η , i ⋆ \theta_{\eta,i}^\star= \log\pi_{\mathrm{base}}(i) + \eta r(i), \qquad s_{t,i} = \theta_{t,i} - \theta_{\eta,i}^\star θ η , i ⋆ = log π base ( i ) + η r ( i ) , s t , i = θ t , i − θ η , i ⋆
s ˉ t = ∑ i π t ( i ) s t , i , h = α η ∈ ( 0 , 1 ] . \bar{s}_t = \sum_i \pi_t(i)s_{t,i}, \qquad h = \frac{\alpha}{\eta} \in(0,1]. s ˉ t = i ∑ π t ( i ) s t , i , h = η α ∈ ( 0 , 1 ] .
Substitution into the gradient formula yields
s t + 1 , i = ( 1 − h π t ( i ) ) s t , i + h π t ( i ) s ˉ t . s_{t+1,i} = (1-h\pi_t(i))s_{t,i} + h\pi_t(i)\bar{s}_t. s t + 1 , i = ( 1 − h π t ( i )) s t , i + h π t ( i ) s ˉ t .
Each new error is a convex combination of the old errors. If m t = min i s t , i m_t=\min_i s_{t,i} m t = min i s t , i and M t = max i s t , i M_t=\max_i s_{t,i} M t = max i s t , i , then m 0 ≤ m t ≤ M t ≤ M 0 m_0 \le m_t \le M_t \le M_0 m 0 ≤ m t ≤ M t ≤ M 0 and
π t ( i ) = π η ⋆ ( i ) e s t , i ∑ j π η ⋆ ( j ) e s t , j ≥ e m 0 − M 0 π η ⋆ ( i ) ≥ p , p : = e m 0 − M 0 min i π η ⋆ ( i ) > 0. \pi_t(i) = \frac{\pi_\eta^\star(i)e^{s_{t,i}}}{\sum_j \pi_\eta^\star(j)e^{s_{t,j}}} \ge e^{m_0-M_0}\pi_\eta^\star(i) \ge p,\qquad p := e^{m_0-M_0}\min_i\pi_\eta^\star(i)>0. π t ( i ) = ∑ j π η ⋆ ( j ) e s t , j π η ⋆ ( i ) e s t , i ≥ e m 0 − M 0 π η ⋆ ( i ) ≥ p , p := e m 0 − M 0 i min π η ⋆ ( i ) > 0.
The common averaging term therefore gives
m t + 1 ≥ ( 1 − h p ) m t + h p s ˉ t , M t + 1 ≤ ( 1 − h p ) M t + h p s ˉ t , M t + 1 − m t + 1 ≤ ( 1 − h p ) ( M t − m t ) . \begin{aligned}
m_{t+1} &\ge(1-hp)m_t + hp\bar{s}_t,\\
M_{t+1} &\le(1-hp)M_t + hp\bar{s}_t,\\
M_{t+1}-m_{t+1} &\le(1-hp)(M_t-m_t).
\end{aligned} m t + 1 M t + 1 M t + 1 − m t + 1 ≥ ( 1 − h p ) m t + h p s ˉ t , ≤ ( 1 − h p ) M t + h p s ˉ t , ≤ ( 1 − h p ) ( M t − m t ) .
It follows that M t − m t → 0 M_t-m_t\to0 M t − m t → 0 , and the displayed expression for π t ( i ) \pi_t(i) π t ( i ) gives π t → π η ⋆ \pi_t\to\pi_\eta^\star π t → π η ⋆ . By continuity of J J J , the accuracy condition implies 1 − J ( π η ⋆ ) ≤ ε 1-J(\pi_\eta^\star)\le\varepsilon 1 − J ( π η ⋆ ) ≤ ε . Finally, the explicit Gibbs weights give
1 − J ( π η ⋆ ) = 6 ε e − 3 ε η + 3 e − η 1 + 2 e − 3 ε η + 3 e − η . 1-J(\pi_\eta^\star)=\frac{6\varepsilon e^{-3\varepsilon\eta}+3e^{-\eta}}{1+2e^{-3\varepsilon\eta}+3e^{-\eta}}. 1 − J ( π η ⋆ ) = 1 + 2 e − 3 ε η + 3 e − η 6 ε e − 3 ε η + 3 e − η .
Consequently,
4 ε e − 3 ε η + 3 ( 1 − ε ) e − η ≤ ε , e − 3 ε η ≤ 1 4 , 4\varepsilon e^{-3\varepsilon\eta}+3(1-\varepsilon)e^{-\eta}\le\varepsilon,\qquad e^{-3\varepsilon\eta}\le\frac{1}{4}, 4 ε e − 3 ε η + 3 ( 1 − ε ) e − η ≤ ε , e − 3 ε η ≤ 4 1 ,
which implies η ≥ log 4 / ( 3 ε ) \eta\ge\log4/(3\varepsilon) η ≥ log 4/ ( 3 ε ) .
Proof of Theorem C.1.
Constraint on regularization. Fix any T ≥ 1 T\ge1 T ≥ 1 such that J ⋆ − J ( π t ) ≤ ε J^\star-J(\pi_t)\le\varepsilon J ⋆ − J ( π t ) ≤ ε for every t ≥ T t\ge T t ≥ T . If η < ∞ \eta<\infty η < ∞ , the instance and update satisfy all the hypotheses of Lemma C.1. Hence
0 < β = 1 η ≤ 3 ε log 4 . (40) 0<\beta=\frac{1}{\eta}\le\frac{3\varepsilon}{\log4}. \tag*{(40)} 0 < β = η 1 ≤ log 4 3 ε . (40)
For η = ∞ \eta=\infty η = ∞ , we have β = 0 \beta=0 β = 0 , so the same upper bound holds automatically. In either case, π T \pi_T π T is ε \varepsilon ε -optimal. The remaining argument uses only this fact and 0 ≤ β ≤ 3 ε / log 4 0\le\beta\le3\varepsilon/\log4 0 ≤ β ≤ 3 ε / log 4 .
Exact dynamics of the probability ratios. By symmetry, all arms in { 3 , … , m + 2 } \{3,\ldots,m+2\} { 3 , … , m + 2 } have the same logit and probability at every iteration. It therefore suffices to track
p t : = π t ( 1 ) , q t : = π t ( 2 ) , b t : = ∑ i = 3 m + 2 π t ( i ) , p_t:=\pi_t(1),\qquad q_t:=\pi_t(2),\qquad b_t:=\sum_{i=3}^{m+2}\pi_t(i), p t := π t ( 1 ) , q t := π t ( 2 ) , b t := i = 3 ∑ m + 2 π t ( i ) ,
together with the ratios
u t : = p t b t , x t : = b t q t , y t : = u t x t = p t q t , D t : = 1 + x t + y t . (41) u_t:=\frac{p_t}{b_t},\qquad x_t:=\frac{b_t}{q_t},\qquad y_t:=u_tx_t=\frac{p_t}{q_t},\qquad D_t:=1+x_t+y_t. \tag*{(41)} u t := b t p t , x t := q t b t , y t := u t x t = q t p t , D t := 1 + x t + y t . (41)
In particular,
p t = y t D t , q t = 1 D t , b t = x t D t . p_t = \frac{y_t}{D_t}, \qquad q_t = \frac{1}{D_t}, \qquad b_t = \frac{x_t}{D_t}. p t = D t y t , q t = D t 1 , b t = D t x t .
and the initialization in Eq. (36) gives
u 0 = 1 3 , x 0 = 3 2 , y 0 = 1 2 . (42) u_0 = \frac{1}{3}, \qquad x_0 = \frac{3}{2}, \qquad y_0 = \frac{1}{2}. \tag*{(42)} u 0 = 3 1 , x 0 = 2 3 , y 0 = 2 1 . (42)
For an arm i i i , define
Q t ( i ) : = r ( i ) − β log π t ( i ) π b a s e ( i ) . Q_t(i) := r(i) - \beta\log\frac{\pi_t(i)}{\pi_{\mathrm{base}}(i)}. Q t ( i ) := r ( i ) − β log π base ( i ) π t ( i ) .
The softmax gradient has coordinate
[ ∇ J η ( π θ ) ] i ∣ θ = θ t = π t ( i ) ( Q t ( i ) − E j ∼ π t [ Q t ( j ) ] ) . (43) \left[\nabla J_\eta(\pi_\theta)\right]_i\bigg|_{\theta=\theta_t}
= \pi_t(i)\left(Q_t(i) - \mathbb{E}_{j\sim\pi_t}[Q_t(j)]\right). \tag*{(43)} [ ∇ J η ( π θ ) ] i θ = θ t = π t ( i ) ( Q t ( i ) − E j ∼ π t [ Q t ( j )] ) . (43)
Writing Q t ( 0 ) Q_t(0) Q t ( 0 ) for the common value of Q t ( i ) Q_t(i) Q t ( i ) on the arms i ≥ 3 i \ge3 i ≥ 3 , the three relevant gaps are
Q t ( 1 ) − Q t ( 2 ) = 3 ε − β log ( 2 y t ) , Q_t(1) - Q_t(2) = 3\varepsilon- \beta\log(2y_t), Q t ( 1 ) − Q t ( 2 ) = 3 ε − β log ( 2 y t ) ,
Q t ( 1 ) − Q t ( 0 ) = 1 − β log ( 3 u t ) , Q_t(1) - Q_t(0) = 1 - \beta\log(3u_t), Q t ( 1 ) − Q t ( 0 ) = 1 − β log ( 3 u t ) ,
Q t ( 2 ) − Q t ( 0 ) = 1 − 3 ε + β log 2 x t 3 . Q_t(2) - Q_t(0) = 1 - 3\varepsilon+ \beta\log\frac{2x_t}{3}. Q t ( 2 ) − Q t ( 0 ) = 1 − 3 ε + β log 3 2 x t .
Substituting these gaps into Eq. (43) yields
u t + 1 = u t exp ( α A t ) , x t + 1 = x t exp ( − α B t ) , y t + 1 = y t exp ( α C t ) , (44) u_{t+1} = u_t \exp(\alpha A_t), \qquad x_{t+1} = x_t \exp(-\alpha B_t), \qquad y_{t+1} = y_t \exp(\alpha C_t), \tag*{(44)} u t + 1 = u t exp ( α A t ) , x t + 1 = x t exp ( − α B t ) , y t + 1 = y t exp ( α C t ) , (44)
where C t = A t − B t C_t = A_t - B_t C t = A t − B t and
A t u t = x t D t 2 [ x t + 3 ε − β log ( 2 y t ) − β x t log ( 3 u t ) ] + x t m D t 2 [ x t + 1 − 3 ε + β log ( 2 x t / 3 ) u t − β x t log ( 3 u t ) ] . (45) \frac{A_t}{u_t}
= \frac{x_t}{D_t^2}\left[x_t + 3\varepsilon- \beta\log(2y_t) - \beta x_t \log(3u_t)\right]
+ \frac{x_t}{mD_t^2}\left[x_t + \frac{1 - 3\varepsilon+ \beta\log(2x_t/3)}{u_t} - \beta x_t \log(3u_t)\right]. \tag*{(45)} u t A t = D t 2 x t [ x t + 3 ε − β log ( 2 y t ) − β x t log ( 3 u t ) ] + m D t 2 x t [ x t + u t 1 − 3 ε + β log ( 2 x t /3 ) − β x t log ( 3 u t ) ] . (45)
B t = x t D t 2 [ 1 − 3 ε ( 1 + u t ) + β u t log ( 2 y t ) + β log 2 x t 3 + x t m ( y t + 1 − 3 ε − β y t log ( 3 u t ) + β log 2 x t 3 ) ] . (46) B_t
= \frac{x_t}{D_t^2}\left[1 - 3\varepsilon(1+u_t) + \beta u_t \log(2y_t) + \beta\log\frac{2x_t}{3}
+ \frac{x_t}{m}\left(y_t + 1 - 3\varepsilon- \beta y_t \log(3u_t) + \beta\log\frac{2x_t}{3}\right)\right]. \tag*{(46)} B t = D t 2 x t [ 1 − 3 ε ( 1 + u t ) + β u t log ( 2 y t ) + β log 3 2 x t + m x t ( y t + 1 − 3 ε − β y t log ( 3 u t ) + β log 3 2 x t ) ] . (46)
C t = 1 D t 2 [ x t ( y t − 1 ) + 3 ε ( x t + 2 y t ) − β ( ( x t + 2 ) y t log ( 2 y t ) − ( y t − 1 ) x t log 2 x t 3 ) ] . (47) C_t
= \frac{1}{D_t^2}\left[x_t(y_t-1) + 3\varepsilon(x_t+2y_t) - \beta\left((x_t+2)y_t\log(2y_t) - (y_t-1)x_t\log\frac{2x_t}{3}\right)\right]. \tag*{(47)} C t = D t 2 1 [ x t ( y t − 1 ) + 3 ε ( x t + 2 y t ) − β ( ( x t + 2 ) y t log ( 2 y t ) − ( y t − 1 ) x t log 3 2 x t ) ] . (47)
The unregularized reward gap can also be written in terms of these ratios,
J ⋆ − J ( π t ) = b t + 3 ε q t = x t + 3 ε 1 + x t + y t . J^\star- J(\pi_t) = b_t + 3\varepsilon q_t = \frac{x_t + 3\varepsilon}{1+x_t+y_t}. J ⋆ − J ( π t ) = b t + 3 ε q t = 1 + x t + y t x t + 3 ε .
Consequently, whenever y t < 1 y_t < 1 y t < 1 ,
J ⋆ − J ( π t ) > x t + 3 ε 2 + x t > ε . (48) J^\star- J(\pi_t) > \frac{x_t + 3\varepsilon}{2+x_t} > \varepsilon. \tag*{(48)} J ⋆ − J ( π t ) > 2 + x t x t + 3 ε > ε . (48)
Coverage collapse
Define the log-ratio change and the threshold for the collapse phase by
ℓ t : = log x 0 x t , L ε : = 1 4 ε . \ell_t := \log\frac{x_0}{x_t}, \qquad L_\varepsilon:= \frac{1}{4\sqrt{\varepsilon}}. ℓ t := log x t x 0 , L ε := 4 ε 1 .
We first record the bounds that drive the initial phase. As long as
1 3 ≤ u t < 4 , y t ≤ 1 2 , 0 ≤ ℓ t ≤ L ε , (49) \frac{1}{3} \le u_t < 4,\qquad y_t \le\frac{1}{2},\qquad0 \le\ell_t \le L_\varepsilon, \tag*{(49)} 3 1 ≤ u t < 4 , y t ≤ 2 1 , 0 ≤ ℓ t ≤ L ε , (49)
the quantities in Eqs. (45) to (47) satisfy
0 ≤ A t ≤ 1 8 , 3 4 x t D t 2 ≤ B t ≤ 1 5 , 0 ≤ A t u t ≤ x t D t 2 [ x t + 15 2 ε ( 1 + ℓ t ) ] , C t < 0. (50) 0 \le A_t \le\frac{1}{8},
\qquad
\frac{3}{4}\frac{x_t}{D_t^2} \le B_t \le\frac{1}{5},
\qquad
0 \le\frac{A_t}{u_t} \le\frac{x_t}{D_t^2}\left[x_t+\frac{15}{2}\varepsilon(1+\ell_t)\right],
\qquad
C_t<0. \tag*{(50)} 0 ≤ A t ≤ 8 1 , 4 3 D t 2 x t ≤ B t ≤ 5 1 , 0 ≤ u t A t ≤ D t 2 x t [ x t + 2 15 ε ( 1 + ℓ t ) ] , C t < 0. (50)
We verify these bounds directly. The identities x t = x 0 e − ℓ t x_t=x_0e^{-\ell_t} x t = x 0 e − ℓ t and 2 y t = 3 u t e − ℓ t 2y_t=3u_te^{-\ell_t} 2 y t = 3 u t e − ℓ t imply
x t ≤ 3 y t ≤ 3 2 , 0 ≤ log ( 3 u t ) ≤ ℓ t , − log ( 2 y t ) = ℓ t − log ( 3 u t ) , (51) x_t \le3y_t \le\frac{3}{2},
\qquad
0 \le\log(3u_t) \le\ell_t,
\qquad
-\log(2y_t)=\ell_t-\log(3u_t), \tag*{(51)} x t ≤ 3 y t ≤ 2 3 , 0 ≤ log ( 3 u t ) ≤ ℓ t , − log ( 2 y t ) = ℓ t − log ( 3 u t ) , (51)
while m − 1 ≤ ε m^{-1}\le\varepsilon m − 1 ≤ ε and, by Eq. (40) ,
β ℓ t ≤ 3 ε L ε log 4 ≤ 3 32 log 4 < 1 14 . \beta\ell_t\le\frac{3\varepsilon L_\varepsilon}{\log4}\le\frac{3}{32\log4}<\frac{1}{14}. β ℓ t ≤ log 4 3 ε L ε ≤ 32 log 4 3 < 14 1 .
Using these relations in Eq. (45) and dropping its negative terms gives
A t u t ≤ x t D t 2 [ x t + 3 ε + β ℓ t + ε x t + 3 ε ] ≤ x t D t 2 [ x t + 15 2 ε ( 1 + ℓ t ) ] . (52) \frac{A_t}{u_t}\le\frac{x_t}{D_t^2}\left[x_t+3\varepsilon+\beta\ell_t+\varepsilon x_t+3\varepsilon\right]
\le\frac{x_t}{D_t^2}\left[x_t+\frac{15}{2}\varepsilon(1+\ell_t)\right]. \tag*{(52)} u t A t ≤ D t 2 x t [ x t + 3 ε + β ℓ t + ε x t + 3 ε ] ≤ D t 2 x t [ x t + 2 15 ε ( 1 + ℓ t ) ] . (52)
Both brackets in Eq. (45) are nonnegative. Indeed, x t ≤ 1 / ( 2 u t ) x_t\le1/(2u_t) x t ≤ 1/ ( 2 u t ) and sup u > 0 log ( 3 u ) / u = 3 / e \sup_{u>0}\log(3u)/u=3/e sup u > 0 log ( 3 u ) / u = 3/ e give
β x t log ( 3 u t ) ≤ β 2 log ( 3 u t ) u t ≤ 3 β 2 e < 3 ε . \beta x_t\log(3u_t)\le\frac{\beta}{2}\frac{\log(3u_t)}{u_t}\le\frac{3\beta}{2e}<3\varepsilon. β x t log ( 3 u t ) ≤ 2 β u t log ( 3 u t ) ≤ 2 e 3 β < 3 ε .
Hence the first bracket is at least x t + 3 ε − 3 β / ( 2 e ) > 0 x_t+3\varepsilon-3\beta/(2e)>0 x t + 3 ε − 3 β / ( 2 e ) > 0 . The second bracket is at least
x t + 1 − 3 ε − 3 2 β ℓ t u t > 0 , x_t+\frac{1-3\varepsilon-\frac{3}{2}\beta\ell_t}{u_t}>0, x t + u t 1 − 3 ε − 2 3 β ℓ t > 0 ,
where we again used x t ≤ 1 / ( 2 u t ) x_t\le1/(2u_t) x t ≤ 1/ ( 2 u t ) and log ( 3 u t ) ≤ ℓ t \log(3u_t)\le\ell_t log ( 3 u t ) ≤ ℓ t . Thus A t ≥ 0 A_t\ge0 A t ≥ 0 .
For B t B_t B t , the m − 1 m^{-1} m − 1 -bracket in Eq. (46) obeys
y t + 1 − 3 ε − β y t log ( 3 u t ) + β log 2 x t 3 ≥ 1 − 3 ε − 3 2 β ℓ t > 4 5 . y_t+1-3\varepsilon-\beta y_t\log(3u_t)+\beta\log\frac{2x_t}{3}
\ge1-3\varepsilon-\frac{3}{2}\beta\ell_t>\frac{4}{5}. y t + 1 − 3 ε − β y t log ( 3 u t ) + β log 3 2 x t ≥ 1 − 3 ε − 2 3 β ℓ t > 5 4 .
Substituting Eq. (51) into the other bracket in Eq. (46) gives
D t 2 x t B t ≥ 1 − 3 ε ( 1 + u t ) + β u t log ( 3 u t ) − β ( u t + 1 ) L ε . (53) \frac{D_t^2}{x_t}B_t\ge1-3\varepsilon(1+u_t)+\beta u_t\log(3u_t)-\beta(u_t+1)L_\varepsilon. \tag*{(53)} x t D t 2 B t ≥ 1 − 3 ε ( 1 + u t ) + β u t log ( 3 u t ) − β ( u t + 1 ) L ε . (53)
To lower-bound this expression uniformly, let z = ε ≤ 1 / 8 z=\sqrt{\varepsilon}\le1/8 z = ε ≤ 1/8 , a = log ( 3 u t ) a=\log(3u_t) a = log ( 3 u t ) , and κ = 3 / log 4 \kappa=3/\log4 κ = 3/ log 4 . Since a ≤ L ε a\le L_\varepsilon a ≤ L ε , the coefficient of β \beta β is negative, so the right-hand side is minimized at β = κ ε \beta=\kappa\varepsilon β = κ ε . The resulting loss from 1 1 1 is
3 z 2 ( 1 + u t ) − κ z 2 u t a + κ z 4 ( 1 + u t ) . 3z^2(1+u_t)-\kappa z^2u_ta+\frac{\kappa z}{4}(1+u_t). 3 z 2 ( 1 + u t ) − κ z 2 u t a + 4 κ z ( 1 + u t ) .
This loss is increasing in a a a because a ≤ L ε a \le L_{\varepsilon} a ≤ L ε and its derivative has the sign of
κ 4 + z [ 3 − κ ( a + 1 ) ] ≥ κ 4 + z [ 3 − κ ( L ε + 1 ) ] = z κ ( log 4 − 1 ) > 0. \frac{\kappa}{4}+z[3-\kappa(a+1)] \ge\frac{\kappa}{4}+z[3-\kappa(L_{\varepsilon}+1)]=z\kappa(\log4-1)>0. 4 κ + z [ 3 − κ ( a + 1 )] ≥ 4 κ + z [ 3 − κ ( L ε + 1 )] = z κ ( log 4 − 1 ) > 0.
If L ε ≤ log 12 L_{\varepsilon}\le\log12 L ε ≤ log 12 , its maximum is therefore attained at a = L ε a=L_{\varepsilon} a = L ε , where it is
3 + e L ε 16 L ε 2 + κ 16 L ε ≤ 3 + e 2 64 + κ 32 < 1 4 . \frac{3+e^{L_{\varepsilon}}}{16L_{\varepsilon}^{2}}+\frac{\kappa}{16L_{\varepsilon}}\le\frac{3+e^{2}}{64}+\frac{\kappa}{32}<\frac{1}{4}. 16 L ε 2 3 + e L ε + 16 L ε κ ≤ 64 3 + e 2 + 32 κ < 4 1 .
Here both terms are decreasing on L ε ∈ [ 2 , log 12 ] L_{\varepsilon}\in[2,\log12] L ε ∈ [ 2 , log 12 ] . If L ε > log 12 L_{\varepsilon}>\log12 L ε > log 12 , the maximum is attained at a = log 12 a=\log12 a = log 12 and is increasing in z ≤ 1 / ( 4 log 12 ) z\le1/(4\log12) z ≤ 1/ ( 4 log 12 ) , so the same bound follows by evaluating at L ε = log 12 L_{\varepsilon}=\log12 L ε = log 12 . Thus the right-hand side of Eq. (53) is at least 3 / 4 3/4 3/4 .
Conversely, the nonpositive logarithmic terms in Eq. (46) and the relation x t ≤ 3 y t x_t\le3y_t x t ≤ 3 y t imply
B t ≤ x t D t 2 ( 1 + 1 + y t m ) ≤ 3 16 ( 1 + 3 ε 2 ) < 1 5 . B_t\le\frac{x_t}{D_t^2}\left(1+\frac{1+y_t}{m}\right)\le\frac{3}{16}\left(1+\frac{3\varepsilon}{2}\right)<\frac{1}{5}. B t ≤ D t 2 x t ( 1 + m 1 + y t ) ≤ 16 3 ( 1 + 2 3 ε ) < 5 1 .
Here we used x t / D t 2 ≤ x t / ( 1 + 4 x t / 3 ) 2 ≤ 3 / 16 x_t/D_t^2\le x_t/(1+4x_t/3)^2\le3/16 x t / D t 2 ≤ x t / ( 1 + 4 x t /3 ) 2 ≤ 3/16 .
Multiplying Eq. (45) by u t u_t u t and again dropping negative terms gives the envelope
A t ≤ x t y t + 3 ε y t − β y t log ( 2 y t ) + ε x t ( 1 + y t ) D t 2 . A_t\le\frac{x_ty_t+3\varepsilon y_t-\beta y_t\log(2y_t)+\varepsilon x_t(1+y_t)}{D_t^2}. A t ≤ D t 2 x t y t + 3 ε y t − β y t log ( 2 y t ) + ε x t ( 1 + y t ) .
On the region in Eq. (49) , these terms are at most 1 / 12 1/12 1/12 , 3 ε / 2 3\varepsilon/2 3 ε /2 , 3 ε / ( 2 e log 4 ) 3\varepsilon/(2e\log4) 3 ε / ( 2 e log 4 ) , and ε / 4 \varepsilon/4 ε /4 , respectively. Thus A t < 1 / 8 A_t<1/8 A t < 1/8 . It remains to verify the sign of C t C_t C t . Let
a t : = log ( 3 u t ) , δ t : = ℓ t − a t = − log ( 2 y t ) . a_t:=\log(3u_t),\qquad\delta_t:=\ell_t-a_t=-\log(2y_t). a t := log ( 3 u t ) , δ t := ℓ t − a t = − log ( 2 y t ) .
Dividing Eq. (47) by x t / D t 2 x_t/D_t^2 x t / D t 2 and using Eq. (51) gives
D t 2 x t C t = y t − 1 + 3 ε ( 1 + 2 u t ) + β [ ( 1 + 2 u t ) ℓ t − ( y t + 2 u t ) a t ] . \frac{D_t^2}{x_t}C_t=y_t-1+3\varepsilon(1+2u_t)+\beta[(1+2u_t)\ell_t-(y_t+2u_t)a_t]. x t D t 2 C t = y t − 1 + 3 ε ( 1 + 2 u t ) + β [( 1 + 2 u t ) ℓ t − ( y t + 2 u t ) a t ] .
Dropping the term − β y t a t -\beta y_ta_t − β y t a t bounds the right-hand side by
h ( ℓ t , δ t ) : = 1 2 exp ( − δ t ) − 1 + 3 ε [ 1 + 2 3 exp ( ℓ t − δ t ) ] + β [ ℓ t + 2 3 exp ( ℓ t − δ t ) δ t ] . h(\ell_t,\delta_t):=\frac{1}{2}\exp(-\delta_t)-1+3\varepsilon\left[1+\frac{2}{3}\exp(\ell_t-\delta_t)\right]+\beta\left[\ell_t+\frac{2}{3}\exp(\ell_t-\delta_t)\delta_t\right]. h ( ℓ t , δ t ) := 2 1 exp ( − δ t ) − 1 + 3 ε [ 1 + 3 2 exp ( ℓ t − δ t ) ] + β [ ℓ t + 3 2 exp ( ℓ t − δ t ) δ t ] .
For fixed ℓ t \ell_t ℓ t , this function is strictly decreasing in δ t \delta_t δ t : when δ t ≤ 1 \delta_t\le1 δ t ≤ 1 , this follows from β < 3 ε \beta<3\varepsilon β < 3 ε , and when δ t ≥ 1 \delta_t\ge1 δ t ≥ 1 , every term in its derivative is nonpositive. The constraints u t < 4 u_t<4 u t < 4 and y t ≤ 1 / 2 y_t\le1/2 y t ≤ 1/2 give δ t ≥ max { 0 , ℓ t − log 12 } \delta_t\ge\max\{0,\ell_t-\log12\} δ t ≥ max { 0 , ℓ t − log 12 } , so it remains to check the two boundary cases. If ℓ t ≤ log 12 \ell_t\le\log12 ℓ t ≤ log 12 , then
h ( ℓ t , 0 ) = − 1 2 + 3 ε [ 1 + 2 3 exp ( ℓ t ) ] + β ℓ t < − 1 8 . h(\ell_t,0)=-\frac{1}{2}+3\varepsilon\left[1+\frac{2}{3}\exp(\ell_t)\right]+\beta\ell_t<-\frac{1}{8}. h ( ℓ t , 0 ) = − 2 1 + 3 ε [ 1 + 3 2 exp ( ℓ t ) ] + β ℓ t < − 8 1 .
Indeed, if L ε ≤ log 12 L_{\varepsilon}\le\log12 L ε ≤ log 12 , the positive part is at most
3 + 2 e L ε 16 L ε 2 + κ 16 L ε ≤ 3 + 2 e 2 64 + κ 32 < 3 8 . \frac{3+2e^{L_{\varepsilon}}}{16L_{\varepsilon}^{2}}+\frac{\kappa}{16L_{\varepsilon}}\le\frac{3+2e^{2}}{64}+\frac{\kappa}{32}<\frac{3}{8}. 16 L ε 2 3 + 2 e L ε + 16 L ε κ ≤ 64 3 + 2 e 2 + 32 κ < 8 3 .
The derivative of the displayed upper bound changes sign at most once, from negative to positive, on L ε ∈ [ 2 , log 12 ] L_{\varepsilon}\in[2,\log12] L ε ∈ [ 2 , log 12 ] ; its maximum is therefore attained at an endpoint, and the value at L ε = 2 L_{\varepsilon}=2 L ε = 2 is larger. If L ε > log 12 L_{\varepsilon}>\log12 L ε > log 12 , the same bound follows by evaluating at L ε = log 12 L_{\varepsilon}=\log12 L ε = log 12 .
If ℓ t ≥ log 12 \ell_t\ge\log12 ℓ t ≥ log 12 , then
h ( ℓ t , ℓ t − log 12 ) = 6 exp ( − ℓ t ) − 1 + 27 ε + β [ 9 ℓ t − 8 log 12 ] . h(\ell_t,\ell_t-\log12)=6\exp(-\ell_t)-1+27\varepsilon+\beta[9\ell_t-8\log12]. h ( ℓ t , ℓ t − log 12 ) = 6 exp ( − ℓ t ) − 1 + 27 ε + β [ 9 ℓ t − 8 log 12 ] .
This is convex in ℓ t \ell_t ℓ t , so its maximum on [ log 12 , L ε ] [\log12, L_\varepsilon] [ log 12 , L ε ] occurs at an endpoint. At ℓ t = log 12 \ell_t = \log12 ℓ t = log 12 , it is smaller than − 1 / 8 -1/8 − 1/8 by the preceding bound. At ℓ t = L ε \ell_t = L_\varepsilon ℓ t = L ε , substituting ε = 1 / ( 16 L ε 2 ) \varepsilon= 1/(16L_\varepsilon^2) ε = 1/ ( 16 L ε 2 ) and β ≤ κ / ( 16 L ε 2 ) \beta\le\kappa/(16L_\varepsilon^2) β ≤ κ / ( 16 L ε 2 ) shows that the expression is decreasing for L ε ≥ log 12 L_\varepsilon\ge\log12 L ε ≥ log 12 ; hence it is again at most its value at L ε = log 12 L_\varepsilon= \log12 L ε = log 12 .
Thus C t < 0 C_t < 0 C t < 0 , completing the proof of Eq. (50) .
Let
ρ : = min { t ≥ 0 : u t ≥ 4 or ℓ t > L ε } . \rho:= \min\{t \ge0 : u_t \ge4 \text{ or } \ell_t > L_\varepsilon\}. ρ := min { t ≥ 0 : u t ≥ 4 or ℓ t > L ε } .
The initialization in Eq. (42) and Eq. (50) imply inductively that, for every t < ρ t < \rho t < ρ , u t u_t u t is nondecreasing, x t x_t x t is nonincreasing, and y t ≤ 1 / 2 y_t \le1/2 y t ≤ 1/2 . If this hitting time is infinite, then these properties hold for all t t t . By Eq. (48) , no iterate is ε \varepsilon ε -optimal. We may therefore suppose that ρ < ∞ \rho< \infty ρ < ∞ .
We next show that the exit occurs through ℓ ρ > L ε \ell_\rho> L_\varepsilon ℓ ρ > L ε . Suppose instead that u ρ ≥ 4 u_\rho\ge4 u ρ ≥ 4 and ℓ ρ ≤ L ε \ell_\rho\le L_\varepsilon ℓ ρ ≤ L ε . Define
s t : = 1 u t , Δ t : = ℓ t + 1 − ℓ t = α B t . s_t := \frac{1}{u_t}, \qquad\Delta_t := \ell_{t+1} - \ell_t = \alpha B_t. s t := u t 1 , Δ t := ℓ t + 1 − ℓ t = α B t .
For every t < ρ t < \rho t < ρ , Eq. (50) and α ≤ 1 / 5 \alpha\le1/5 α ≤ 1/5 give 0 ≤ Δ t ≤ 1 / 25 0 \le\Delta_t \le1/25 0 ≤ Δ t ≤ 1/25 . Hence
s t + 1 − s t = 1 u t ( e − α A t − 1 ) ≥ − α A t u t ≥ − 4 3 Δ t [ x t + 15 2 ε ( 1 + ℓ t ) ] . (54) \begin{aligned}
s_{t+1} - s_t &= \frac{1}{u_t}\left(e^{-\alpha A_t}-1\right) \\
&\ge-\alpha\frac{A_t}{u_t} \\
&\ge-\frac{4}{3}\Delta_t\left[x_t + \frac{15}{2}\varepsilon(1+\ell_t)\right]. \tag*{(54)}
\end{aligned} s t + 1 − s t = u t 1 ( e − α A t − 1 ) ≥ − α u t A t ≥ − 3 4 Δ t [ x t + 2 15 ε ( 1 + ℓ t ) ] . (54)
Also, 1 − e − z ≥ ( 24 / 25 ) z 1-e^{-z} \ge(24/25)z 1 − e − z ≥ ( 24/25 ) z for z ∈ [ 0 , 1 / 25 ] z \in[0,1/25] z ∈ [ 0 , 1/25 ] implies
∑ t = 0 ρ − 1 x t Δ t ≤ 25 24 ∑ t = 0 ρ − 1 ( x t − x t + 1 ) ≤ 25 16 . (55) \sum_{t=0}^{\rho-1} x_t\Delta_t \le\frac{25}{24}\sum_{t=0}^{\rho-1}(x_t-x_{t+1}) \le\frac{25}{16}. \tag*{(55)} t = 0 ∑ ρ − 1 x t Δ t ≤ 24 25 t = 0 ∑ ρ − 1 ( x t − x t + 1 ) ≤ 16 25 . (55)
Since ℓ t \ell_t ℓ t is nondecreasing, a left Riemann sum gives
∑ t = 0 ρ − 1 ( 1 + ℓ t ) Δ t ≤ ℓ ρ + 1 2 ℓ ρ 2 ≤ L ε + 1 2 L ε 2 . (56) \sum_{t=0}^{\rho-1}(1+\ell_t)\Delta_t \le\ell_\rho+ \frac{1}{2}\ell_\rho^2 \le L_\varepsilon+ \frac{1}{2}L_\varepsilon^2. \tag*{(56)} t = 0 ∑ ρ − 1 ( 1 + ℓ t ) Δ t ≤ ℓ ρ + 2 1 ℓ ρ 2 ≤ L ε + 2 1 L ε 2 . (56)
Summing Eq. (54) and using Eqs. (55) and (56) yields
s ρ ≥ 3 − 25 12 − 10 ε ( L ε + 1 2 L ε 2 ) s_\rho\ge3-\frac{25}{12}-10\varepsilon\left(L_\varepsilon+\frac{1}{2}L_\varepsilon^2\right) s ρ ≥ 3 − 12 25 − 10 ε ( L ε + 2 1 L ε 2 )
≥ 3 − 25 12 − 5 8 = 7 24 . \ge3-\frac{25}{12}-\frac{5}{8}=\frac{7}{24}. ≥ 3 − 12 25 − 8 5 = 24 7 .
This contradicts u ρ ≥ 4 u_\rho\ge4 u ρ ≥ 4 , which would imply s ρ ≤ 1 / 4 s_\rho\le1/4 s ρ ≤ 1/4 . Therefore ℓ ρ > L ε \ell_\rho> L_\varepsilon ℓ ρ > L ε .
Since u ρ − 1 < 4 u_{\rho-1}<4 u ρ − 1 < 4 , A ρ − 1 ≤ 1 / 8 A_{\rho-1}\le1/8 A ρ − 1 ≤ 1/8 , and α ≤ 1 / 5 \alpha\le1/5 α ≤ 1/5 ,
u ρ < 4 e 1 / 40 . u_\rho< 4e^{1/40}. u ρ < 4 e 1/40 .
Together with x ρ = x 0 e − ℓ ρ x_\rho=x_0e^{-\ell_\rho} x ρ = x 0 e − ℓ ρ , this gives
y ρ = u ρ x ρ < 6 exp ( 1 / 40 − L ε ) < 7 exp ( − L ε ) . (57) y_\rho=u_\rho x_\rho< 6\exp(1/40-L_\varepsilon)<7\exp(-L_\varepsilon). \tag*{(57)} y ρ = u ρ x ρ < 6 exp ( 1/40 − L ε ) < 7 exp ( − L ε ) . (57)
In particular, p ρ = y ρ / D ρ < y ρ p_\rho=y_\rho/D_\rho<y_\rho p ρ = y ρ / D ρ < y ρ , and hence
C π ρ = 1 p ρ > 1 7 exp ( 1 4 ε ) . (58) C_{\pi_\rho}=\frac{1}{p_\rho}>\frac{1}{7}\exp\left(\frac{1}{4\sqrt{\varepsilon}}\right). \tag*{(58)} C π ρ = p ρ 1 > 7 1 exp ( 4 ε 1 ) . (58)
Conversely, since ℓ ρ − 1 ≤ L ε \ell_{\rho-1} \le L_{\varepsilon} ℓ ρ − 1 ≤ L ε and ℓ ρ − ℓ ρ − 1 = α B ρ − 1 ≤ 1 / 25 \ell_{\rho} - \ell_{\rho-1} = \alpha B_{\rho-1} \le1/25 ℓ ρ − ℓ ρ − 1 = α B ρ − 1 ≤ 1/25 , while u ρ ≥ u 0 = 1 / 3 u_{\rho} \ge u_0 = 1/3 u ρ ≥ u 0 = 1/3 , we have
C π ρ = 1 + 1 u ρ + 1 y ρ = 1 + 1 u ρ + 2 exp ( ℓ ρ ) 3 u ρ ≤ 4 + 2 exp ( L ε + 1 25 ) ≤ 3 exp ( L ε ) . (59) C_{\pi_{\rho}} = 1 + \frac{1}{u_{\rho}} + \frac{1}{y_{\rho}} = 1 + \frac{1}{u_{\rho}} + \frac{2\exp(\ell_{\rho})}{3u_{\rho}}
\le4 + 2\exp\left(L_{\varepsilon} + \frac{1}{25}\right) \le3\exp(L_{\varepsilon}). \tag*{(59)} C π ρ = 1 + u ρ 1 + y ρ 1 = 1 + u ρ 1 + 3 u ρ 2 exp ( ℓ ρ ) ≤ 4 + 2 exp ( L ε + 25 1 ) ≤ 3 exp ( L ε ) . (59)
The last inequality uses L ε ≥ 2 L_{\varepsilon} \ge2 L ε ≥ 2 .
We also verify Eq. (38) . For every t < ρ t < \rho t < ρ , Eqs. (44) and (50) give u t + 1 ≥ u t u_{t+1} \ge u_t u t + 1 ≥ u t and x t + 1 < x t x_{t+1} < x_t x t + 1 < x t . In the coordinates of Eq. (41) , the expected reward is
J ( π t ) = u t x t + 1 − 3 ε 1 + ( 1 + u t ) x t . J(\pi_t) = \frac{u_t x_t + 1 - 3\varepsilon}{1 + (1 + u_t)x_t}. J ( π t ) = 1 + ( 1 + u t ) x t u t x t + 1 − 3 ε .
This expression is increasing in u t u_t u t . Hence
J ( π t + 1 ) − J ( π t ) ≥ u t x t + 1 + 1 − 3 ε 1 + ( 1 + u t ) x t + 1 − u t x t + 1 − 3 ε 1 + ( 1 + u t ) x t = ( 1 − 3 ε − 3 ε u t ) ( x t − x t + 1 ) ( 1 + ( 1 + u t ) x t + 1 ) ( 1 + ( 1 + u t ) x t ) > 0. J(\pi_{t+1}) - J(\pi_t) \ge\frac{u_t x_{t+1} + 1 - 3\varepsilon}{1 + (1 + u_t)x_{t+1}} - \frac{u_t x_t + 1 - 3\varepsilon}{1 + (1 + u_t)x_t}
= \frac{(1 - 3\varepsilon- 3\varepsilon u_t)(x_t - x_{t+1})}{(1 + (1 + u_t)x_{t+1})(1 + (1 + u_t)x_t)} > 0. J ( π t + 1 ) − J ( π t ) ≥ 1 + ( 1 + u t ) x t + 1 u t x t + 1 + 1 − 3 ε − 1 + ( 1 + u t ) x t u t x t + 1 − 3 ε = ( 1 + ( 1 + u t ) x t + 1 ) ( 1 + ( 1 + u t ) x t ) ( 1 − 3 ε − 3 ε u t ) ( x t − x t + 1 ) > 0.
The last inequality follows from x t + 1 < x t x_{t+1} < x_t x t + 1 < x t and 1 − 3 ε − 3 ε u t > 1 − 15 ε ≥ 49 / 64 1 - 3\varepsilon- 3\varepsilon u_t > 1 - 15\varepsilon\ge49/64 1 − 3 ε − 3 ε u t > 1 − 15 ε ≥ 49/64 , since u t < 4 u_t < 4 u t < 4 and ε ≤ 1 / 64 \varepsilon\le1/64 ε ≤ 1/64 .
Gradient steps required to reach an ε \varepsilon ε -optimal policy. We first establish two bounds that hold whenever u t ≥ 1 / 3 u_t \ge1/3 u t ≥ 1/3 and y t < 1 y_t < 1 y t < 1 . Rewriting Eq. (45) using x t = y t / u t x_t = y_t/u_t x t = y t / u t gives
A t = x t D t 2 R ( u t , y t ) , A_t = \frac{x_t}{D_t^2}R(u_t,y_t), A t = D t 2 x t R ( u t , y t ) ,
where
R ( u , y ) : = 3 ε u − β u log ( 2 y ) + ( 1 + 1 m ) y ( 1 − β log ( 3 u ) ) + 1 − 3 ε + β log ( 2 y / ( 3 u ) ) m . R(u,y) := 3\varepsilon u - \beta u\log(2y) + \left(1 + \frac{1}{m}\right)y(1 - \beta\log(3u)) + \frac{1 - 3\varepsilon+ \beta\log(2y/(3u))}{m}. R ( u , y ) := 3 ε u − β u log ( 2 y ) + ( 1 + m 1 ) y ( 1 − β log ( 3 u )) + m 1 − 3 ε + β log ( 2 y / ( 3 u )) .
Write a = log ( 3 u ) a = \log(3u) a = log ( 3 u ) and s = log ( 1 / y ) s = \log(1/y) s = log ( 1/ y ) . Grouping the terms proportional to s s s rewrites this expression as
R ( u , y ) = u ( 3 ε − β log 2 ) + β ( u − 1 m ) s + ( 1 + 1 m ) y ( 1 − β a ) + 1 − 3 ε + β log 2 − β a m . R(u,y) = u(3\varepsilon- \beta\log2) + \beta\left(u - \frac{1}{m}\right)s
+ \left(1 + \frac{1}{m}\right)y(1 - \beta a) + \frac{1 - 3\varepsilon+ \beta\log2 - \beta a}{m}. R ( u , y ) = u ( 3 ε − β log 2 ) + β ( u − m 1 ) s + ( 1 + m 1 ) y ( 1 − β a ) + m 1 − 3 ε + β log 2 − β a .
By Eq. (40) ,
3 ε − β log 2 ≥ 3 2 ε . 3\varepsilon- \beta\log2 \ge\frac{3}{2}\varepsilon. 3 ε − β log 2 ≥ 2 3 ε .
Every term in the expression for R ( u , y ) R(u,y) R ( u , y ) is therefore nonnegative for u ∈ [ 1 / 3 , 4 ] u \in[1/3,4] u ∈ [ 1/3 , 4 ] and y < 1 y < 1 y < 1 , and the first term is positive. Here we also used β log 12 < 1 \beta\log12 < 1 β log 12 < 1 and m − 1 ≤ ε m^{-1} \le\varepsilon m − 1 ≤ ε . For u ≥ 4 u \ge4 u ≥ 4 , direct differentiation gives
∂ u R ( u , y ) ≥ 3 2 ε − β 4 ( 1 + 2 m ) > 0. \partial_u R(u,y) \ge\frac{3}{2}\varepsilon- \frac{\beta}{4}\left(1 + \frac{2}{m}\right) > 0. ∂ u R ( u , y ) ≥ 2 3 ε − 4 β ( 1 + m 2 ) > 0.
Since R ( 4 , y ) > 0 R(4,y) > 0 R ( 4 , y ) > 0 by the preceding case, this proves
A t > 0 whenever u t ≥ 1 3 and y t < 1. (60) A_t > 0 \qquad\text{whenever} \qquad u_t \ge\frac{1}{3} \text{ and } y_t < 1. \tag*{(60)} A t > 0 whenever u t ≥ 3 1 and y t < 1. (60)
Under the same conditions, x t = y t / u t ≤ 3 y t x_t = y_t/u_t \le3y_t x t = y t / u t ≤ 3 y t . Applying this relation to Eq. (47) yields
C t ≤ 18 ε y t ( 1 + log 1 y t ) . (61) C_t \le18\varepsilon y_t\left(1 + \log\frac{1}{y_t}\right). \tag*{(61)} C t ≤ 18 ε y t ( 1 + log y t 1 ) . (61)
To see this, the reward term contributes at most 15 ε y t 15\varepsilon y_t 15 ε y t . The term containing − log ( 2 y t ) -\log(2y_t) − log ( 2 y t ) is nonpositive when y t ≥ 1 / 2 y_t \ge1/2 y t ≥ 1/2 ; when y t < 1 / 2 y_t < 1/2 y t < 1/2 , it contributes at most 8 ε y t log ( 1 / y t ) 8\varepsilon y_t\log(1/y_t) 8 ε y t log ( 1/ y t ) . Finally, the remaining logarithmic term is positive only when x t < 3 / 2 x_t < 3/2 x t < 3/2 . Writing x t = c t y t x_t=c_ty_t x t = c t y t with c t ≤ 3 c_t \le3 c t ≤ 3 gives
x t log 3 2 x t = c t y t log 1 y t + y t c t log 3 2 c t x_t\log\frac{3}{2x_t}=c_ty_t\log\frac{1}{y_t}+y_tc_t\log\frac{3}{2c_t} x t log 2 x t 3 = c t y t log y t 1 + y t c t log 2 c t 3
≤ 3 y t log 1 y t + y t . \le3y_t\log\frac{1}{y_t}+y_t. ≤ 3 y t log y t 1 + y t .
By Eq. (40) , its contribution is at most 3 ε y t + 7 ε y t log ( 1 / y t ) 3\varepsilon y_t+7\varepsilon y_t\log(1/y_t) 3 ε y t + 7 ε y t log ( 1/ y t ) . Combining the three bounds proves Eq. (61) .
Let
τ ∗ : = min { t ≥ ρ : y t ≥ 1 } . \tau_*:=\min\{t\ge\rho:y_t\ge1\}. τ ∗ := min { t ≥ ρ : y t ≥ 1 } .
If τ ∗ = ∞ \tau_*=\infty τ ∗ = ∞ , no iterate after ρ \rho ρ is ε \varepsilon ε -optimal, so the theorem follows. Otherwise, Eq. (60) and the fact that u ρ ≥ u 0 = 1 / 3 u_\rho\ge u_0=1/3 u ρ ≥ u 0 = 1/3 show that u t ≥ 1 / 3 u_t\ge1/3 u t ≥ 1/3 for every t ∈ [ ρ , τ ∗ ] t\in[\rho,\tau_*] t ∈ [ ρ , τ ∗ ] . For t < τ ∗ t<\tau_* t < τ ∗ , the inequality e − z ≥ 1 − z e^{-z}\ge1-z e − z ≥ 1 − z and Eq. (61) therefore give the reciprocal recursion
1 y t + 1 = 1 y t e − α C t ≥ 1 y t ( 1 − α C t ) ≥ 1 y t − 18 α ε ( 1 + log 1 y t ) . (62) \begin{aligned}
\frac{1}{y_{t+1}}&=\frac{1}{y_t}e^{-\alpha C_t}\\
&\ge\frac{1}{y_t}(1-\alpha C_t)\\
&\ge\frac{1}{y_t}-18\alpha\varepsilon\left(1+\log\frac{1}{y_t}\right). \tag*{(62)}
\end{aligned} y t + 1 1 = y t 1 e − α C t ≥ y t 1 ( 1 − α C t ) ≥ y t 1 − 18 α ε ( 1 + log y t 1 ) . (62)
Choose t max ∈ [ ρ , τ ∗ ) t_{\max}\in[\rho,\tau_*) t m a x ∈ [ ρ , τ ∗ ) maximizing 1 / y t 1/y_t 1/ y t . Summing Eq. (62) from t max t_{\max} t m a x to τ ∗ − 1 \tau_*-1 τ ∗ − 1 , and using y τ ∗ − 1 ≤ 1 y_{\tau_*}^{-1}\le1 y τ ∗ − 1 ≤ 1 and y t − 1 ≤ y t max − 1 y_t^{-1}\le y_{t_{\max}}^{-1} y t − 1 ≤ y t m a x − 1 for t < τ ∗ t<\tau_* t < τ ∗ , gives
τ ∗ − ρ ≥ y t max − 1 − 1 18 α ε ( 1 + log ( y t max − 1 ) ) . (63) \tau_*-\rho\ge\frac{y_{t_{\max}}^{-1}-1}{18\alpha\varepsilon(1+\log(y_{t_{\max}}^{-1}))}. \tag*{(63)} τ ∗ − ρ ≥ 18 α ε ( 1 + log ( y t m a x − 1 )) y t m a x − 1 − 1 . (63)
The function f ( z ) = ( z − 1 ) / ( 1 + log z ) f(z)=(z-1)/(1+\log z) f ( z ) = ( z − 1 ) / ( 1 + log z ) is increasing on z > 1 z>1 z > 1 because
f ′ ( z ) = log z + 1 / z ( 1 + log z ) 2 > 0. f'(z)=\frac{\log z+1/z}{(1+\log z)^2}>0. f ′ ( z ) = ( 1 + log z ) 2 log z + 1/ z > 0.
By Eq. (57) , y t max − 1 ≥ y ρ − 1 > e L ε / 7 y_{t_{\max}}^{-1}\ge y_\rho^{-1}>e^{L_\varepsilon}/7 y t m a x − 1 ≥ y ρ − 1 > e L ε /7 . Since L ε ≥ 2 L_\varepsilon\ge2 L ε ≥ 2 ,
f ( e L ε 7 ) = e L ε 1 / 7 − e − L ε 1 + L ε − log 7 ≥ e L ε 140 L ε . f\left(\frac{e^{L_\varepsilon}}{7}\right)=e^{L_\varepsilon}\frac{1/7-e^{-L_\varepsilon}}{1+L_\varepsilon-\log7}\ge\frac{e^{L_\varepsilon}}{140L_\varepsilon}. f ( 7 e L ε ) = e L ε 1 + L ε − log 7 1/7 − e − L ε ≥ 140 L ε e L ε .
Here we used 1 / 7 − e − L ε ≥ 1 / 7 − e − 2 > 1 / 140 1/7-e^{-L_\varepsilon}\ge1/7-e^{-2}>1/140 1/7 − e − L ε ≥ 1/7 − e − 2 > 1/140 and 1 + L ε − log 7 ≤ L ε 1+L_\varepsilon-\log7\le L_\varepsilon 1 + L ε − log 7 ≤ L ε . Substituting this into Eq. (63) and using ε L ε = ε / 4 \varepsilon L_\varepsilon=\sqrt{\varepsilon}/4 ε L ε = ε /4 yields
τ ∗ − ρ ≥ 1 630 α ε exp ( 1 4 ε ) . (64) \tau_*-\rho\ge\frac{1}{630\alpha\sqrt{\varepsilon}}\exp\left(\frac{1}{4\sqrt{\varepsilon}}\right). \tag*{(64)} τ ∗ − ρ ≥ 630 α ε 1 exp ( 4 ε 1 ) . (64)
By Eq. (48) , every ε \varepsilon ε -optimal iterate must occur at or after τ ∗ \tau_* τ ∗ . Hence any T T T satisfying J ⋆ − J ( π T ) ≤ ε J^\star-J(\pi_T)\le\varepsilon J ⋆ − J ( π T ) ≤ ε obeys
T ≥ τ ∗ ≥ τ ∗ − ρ ≥ 1 630 α ε exp ( 1 4 ε ) ≥ C π ρ 1890 α ε . T\ge\tau_*\ge\tau_*-\rho\ge\frac{1}{630\alpha\sqrt{\varepsilon}}\exp\left(\frac{1}{4\sqrt{\varepsilon}}\right)\ge\frac{C_{\pi_\rho}}{1890\alpha\sqrt{\varepsilon}}. T ≥ τ ∗ ≥ τ ∗ − ρ ≥ 630 α ε 1 exp ( 4 ε 1 ) ≥ 1890 α ε C π ρ .
Moreover, Eqs. (58) and (59) hold at the same index ρ < τ ∗ ≤ T \rho<\tau_*\le T ρ < τ ∗ ≤ T , proving that
C π ρ = exp ( Θ ( 1 ε ) ) . C_{\pi_\rho}=\exp\left(\Theta\left(\frac{1}{\sqrt{\varepsilon}}\right)\right). C π ρ = exp ( Θ ( ε 1 ) ) .
Since u 0 = 1 / 3 < 4 u_0 = 1/3 < 4 u 0 = 1/3 < 4 and ℓ 0 = 0 \ell_0 = 0 ℓ 0 = 0 , the hitting time satisfies ρ ≥ 1 \rho\ge1 ρ ≥ 1 , so 0 < ρ < T 0 < \rho< T 0 < ρ < T . Together with the preceding display, this establishes Eq. (37) , and Theorem 2.1 follows by suppressing the constants.
Stronger bound without regularization. When β = 0 \beta= 0 β = 0 , the term growing with ℓ t \ell_t ℓ t in Eq. (52) disappears, allowing a longer collapse phase. For this final argument, set L ε = 1 / ( 16 ε ) L_\varepsilon= 1/(16\varepsilon) L ε = 1/ ( 16 ε ) and redefine ρ \rho ρ using this threshold. Let τ ∗ = min { t ≥ ρ : y t ≥ 1 } \tau_* = \min\{t \ge\rho: y_t \ge1\} τ ∗ = min { t ≥ ρ : y t ≥ 1 } . On the region in Eq. (49) , Eq. (45) gives
0 ≤ A t u t ≤ x t D t 2 [ x t + 3 ε + ε x t + 3 ε ] ≤ x t D t 2 [ x t + 15 2 ε ] 0 \le\frac{A_t}{u_t} \le\frac{x_t}{D_t^2}\left[x_t + 3\varepsilon+ \varepsilon x_t + 3\varepsilon\right] \le\frac{x_t}{D_t^2}\left[x_t + \frac{15}{2}\varepsilon\right] 0 ≤ u t A t ≤ D t 2 x t [ x t + 3 ε + ε x t + 3 ε ] ≤ D t 2 x t [ x t + 2 15 ε ]
The other bounds in Eq. (50) remain valid; Eq. (46) gives B t ≥ ( 1 − 15 ε ) x t / D t 2 ≥ ( 49 / 64 ) x t / D t 2 B_t \ge(1 - 15\varepsilon)x_t/D_t^2 \ge(49/64)x_t/D_t^2 B t ≥ ( 1 − 15 ε ) x t / D t 2 ≥ ( 49/64 ) x t / D t 2 , and Eq. (47) gives C t ≤ ( − 1 / 2 + 27 ε ) x t / D t 2 < 0 C_t \le(-1/2 + 27\varepsilon)x_t/D_t^2 < 0 C t ≤ ( − 1/2 + 27 ε ) x t / D t 2 < 0 . The upper bounds on A t A_t A t and B t B_t B t above do not depend on the threshold when β = 0 \beta= 0 β = 0 . The same induction therefore keeps the iterates in the collapse region for t < ρ t < \rho t < ρ . Moreover, ρ < ∞ \rho< \infty ρ < ∞ , since otherwise Eq. (48) would rule out the ε \varepsilon ε -optimality of π T \pi_T π T . Thus, if u ρ ≥ 4 u_\rho\ge4 u ρ ≥ 4 and ℓ ρ ≤ L ε \ell_\rho\le L_\varepsilon ℓ ρ ≤ L ε , repeating the arguments for Eqs. (54) and (55) with the sharper bound on A t / u t A_t/u_t A t / u t and using ∑ t < ρ Δ t = ℓ ρ \sum_{t<\rho}\Delta_t = \ell_\rho ∑ t < ρ Δ t = ℓ ρ yields
s ρ ≥ 3 − 4 3 ∑ t < ρ x t Δ t − 10 ε ∑ t < ρ Δ t s_\rho\ge3 - \frac{4}{3}\sum_{t<\rho}x_t\Delta_t - 10\varepsilon\sum_{t<\rho}\Delta_t s ρ ≥ 3 − 3 4 t < ρ ∑ x t Δ t − 10 ε t < ρ ∑ Δ t
≥ 3 − 25 12 − 10 ε L ε = 7 24 > 1 4 . \ge3 - \frac{25}{12} - 10\varepsilon L_\varepsilon= \frac{7}{24} > \frac{1}{4}. ≥ 3 − 12 25 − 10 ε L ε = 24 7 > 4 1 .
Hence the exit again occurs through ℓ ρ > L ε \ell_\rho> L_\varepsilon ℓ ρ > L ε , and the argument for Eq. (57) gives y ρ < 7 e − L ε y_\rho< 7e^{-L_\varepsilon} y ρ < 7 e − L ε . Since L ε ≥ 4 L_\varepsilon\ge4 L ε ≥ 4 , repeating the argument for Eq. (63) and using the monotonicity of f f f and f ( e L ε / 7 ) ≥ e L ε / ( 140 L ε ) f(e^{L_\varepsilon}/7) \ge e^{L_\varepsilon}/(140L_\varepsilon) f ( e L ε /7 ) ≥ e L ε / ( 140 L ε ) gives
T ≥ τ ∗ − ρ ≥ e L ε 2520 α ε L ε = 2 315 α exp ( 1 16 ε ) , T \ge\tau_* - \rho\ge\frac{e^{L_\varepsilon}}{2520\alpha\varepsilon L_\varepsilon} = \frac{2}{315\alpha}\exp\left(\frac{1}{16\varepsilon}\right), T ≥ τ ∗ − ρ ≥ 2520 α ε L ε e L ε = 315 α 2 exp ( 16 ε 1 ) ,
proving Eq. (39) .
I Approximate Policy Mirror Descent
The KL Divergence Criterion (Proof of Theorem 3.1 )
This section proves Theorem 3.1 , which establishes the KL divergence criterion sufficient for convergence and coverage preservation in APMD. In Appendix D.§ 1 we first develop supporting lemmas, then prove Theorem 3.1 in Appendix D.§ 2 .
Notation. Throughout this section, fix η > 0 \eta> 0 η > 0 , integer T ≥ 1 T \ge1 T ≥ 1 , and a given sequence of policies { π t } t = 0 T \{\pi_t\}_{t=0}^{T} { π t } t = 0 T with π 0 = π b a s e \pi_0 = \pi_{\mathrm{base}} π 0 = π base . Recall that for all 0 ≤ t < T 0 \le t < T 0 ≤ t < T , the exact APMD target is given by
π η , t ⋆ ( y ∣ x ) : = π t ( y ∣ x ) e η ( r ( x , y ) − b t ⋆ ( x ) ) , \pi_{\eta,t}^{\star}(y \mid x) := \pi_t(y \mid x)e^{\eta(r(x,y)-b_t^{\star}(x))}, π η , t ⋆ ( y ∣ x ) := π t ( y ∣ x ) e η ( r ( x , y ) − b t ⋆ ( x )) ,
where
b t ⋆ ( x ) : = 1 η log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] . b_t^{\star}(x) := \frac{1}{\eta}\log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta r(x,y)}\right]. b t ⋆ ( x ) := η 1 log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] .
We also introduce notation for a fixed prompt x x x . Denote the expected reward under π \pi π given x x x to be J x ( π ) : = E y ∼ π ( ⋅ ∣ x ) [ r ( x , y ) ] J_x(\pi) := \mathbb{E}_{y\sim\pi(\cdot\mid x)}[r(x,y)] J x ( π ) := E y ∼ π ( ⋅ ∣ x ) [ r ( x , y )] , and the maximum reward at a prompt x x x to be r x ⋆ : = max y ∈ Y r ( x , y ) r_x^{\star} := \max_{y\in\mathcal{Y}}r(x,y) r x ⋆ := max y ∈ Y r ( x , y ) . In addition, we denote p t ( x ) p_t(x) p t ( x ) to be probability mass of π t \pi_t π t over optimal responses,
p t ( x ) : = π t ( Y x ⋆ ∣ x ) , p_t(x) := \pi_t(\mathcal{Y}_x^{\star}\mid x), p t ( x ) := π t ( Y x ⋆ ∣ x ) ,
where
Y x ⋆ : = { y ∈ Y : r ( x , y ) = r x ⋆ } , \mathcal{Y}_x^{\star} := \{y\in\mathcal{Y}:r(x,y)=r_x^{\star}\}, Y x ⋆ := { y ∈ Y : r ( x , y ) = r x ⋆ } ,
where Y x ⋆ \mathcal{Y}_x^{\star} Y x ⋆ is the set of optimal responses at x x x . Lastly, for shorthand we use the following to denote the KL divergence error of π t + 1 \pi_{t+1} π t + 1 ,
K L t ( x ) : = D K L ( π t + 1 ( ⋅ ∣ x ) ∥ π η , t ⋆ ( ⋅ ∣ x ) ) , \mathrm{KL}_t(x) := D_{\mathrm{KL}}\left(\pi_{t+1}(\cdot\mid x)\,\|\,\pi_{\eta,t}^{\star}(\cdot\mid x)\right), KL t ( x ) := D KL ( π t + 1 ( ⋅ ∣ x ) ∥ π η , t ⋆ ( ⋅ ∣ x ) ) ,
K L t : = E x ∼ μ [ K L t ( x ) ] , \mathrm{KL}_t := \mathbb{E}_{x\sim\mu}[\mathrm{KL}_t(x)], KL t := E x ∼ μ [ KL t ( x )] ,
and the average error over T T T iterations to be,
K L ‾ ( x ) : = 1 T ∑ t = 0 T − 1 K L t ( x ) , \overline{\mathrm{KL}}(x) := \frac{1}{T}\sum_{t=0}^{T-1}\mathrm{KL}_t(x), KL ( x ) := T 1 t = 0 ∑ T − 1 KL t ( x ) ,
K L ‾ : = 1 T ∑ t = 0 T − 1 K L t . \overline{\mathrm{KL}} := \frac{1}{T}\sum_{t=0}^{T-1}\mathrm{KL}_t. KL := T 1 t = 0 ∑ T − 1 KL t .
Supporting Lemmas
We first develop supporting estimates for approximate updates.
Later, Appendix § 11.3 reuses these lemmas to extend Theorem 3.1 to general target rewards and comparator policies, while Appendix § 11.2 adapts their arguments to mean-squared approximation error. They are organized as follows.
(Approximate improvement) Lemma D.1 shows that the KL divergence D K L ( π t + 1 ∥ π η , t ⋆ ) D_{\mathrm{KL}}(\pi_{t+1}\|\pi_{\eta,t}^{\star}) D KL ( π t + 1 ∥ π η , t ⋆ ) controls how much π t + 1 \pi_{t+1} π t + 1 can decrease in reward relative to π t \pi_t π t . It is the approximate version of Lemma B.8 .
(Coverage approximately preserved) Lemma D.2 bounds how much the probabilities of upper reward quantiles degrade from π 0 \pi_0 π 0 to π t \pi_t π t , in terms of the accumulated KL divergence error.
(Error accumulation) Lemma D.3 quantifies how approximation errors perturb sequential exponential reweighting under the exact APMD updates. The perturbation is measured through a given function of interest f f f and bounded in terms of the cumulative KL error.
(Technical lemmas) Lemma D.4 and Lemma D.5 .
Lemma D.1 (Approximate improvement). Fix x ∈ X x \in\mathcal{X} x ∈ X . For every 0 ≤ t < T 0 \le t < T 0 ≤ t < T ,
J x ( π t ) ≤ b t ⋆ ( x ) ≤ J x ( π t + 1 ) + K L t ( x ) η . J_x(\pi_t) \le b_t^{\star}(x) \le J_x(\pi_{t+1}) + \frac{\mathrm{KL}_t(x)}{\eta}. J x ( π t ) ≤ b t ⋆ ( x ) ≤ J x ( π t + 1 ) + η KL t ( x ) .
Proof of Lemma D.1. We obtain the first inequality using Jensen’s inequality,
η J x ( π t ) ≤ log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] = η b t ⋆ ( x ) . \eta J_x(\pi_t) \le\log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta r(x,y)}\right] = \eta b_t^\star(x). η J x ( π t ) ≤ log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] = η b t ⋆ ( x ) .
For the second inequality, expanding the KL divergence gives
K L t ( x ) = E y ∼ π t + 1 ( ⋅ ∣ x ) [ log π t + 1 ( y ∣ x ) π t ( y ∣ x ) − η r ( x , y ) + η b t ⋆ ( x ) ] = D K L ( π t + 1 ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x ) ) − η J x ( π t + 1 ) + η b t ⋆ ( x ) ≥ − η J x ( π t + 1 ) + η b t ⋆ ( x ) , \begin{aligned}
\mathrm{KL}_t(x) &= \mathbb{E}_{y\sim\pi_{t+1}(\cdot\mid x)}\left[\log\frac{\pi_{t+1}(y\mid x)}{\pi_t(y\mid x)}-\eta r(x,y)+\eta b_t^\star(x)\right]\\
&=D_{\mathrm{KL}}(\pi_{t+1}(\cdot\mid x)\mathbin\Vert\pi_t(\cdot\mid x))-\eta J_x(\pi_{t+1})+\eta b_t^\star(x)\\
&\ge-\eta J_x(\pi_{t+1})+\eta b_t^\star(x),
\end{aligned} KL t ( x ) = E y ∼ π t + 1 ( ⋅ ∣ x ) [ log π t ( y ∣ x ) π t + 1 ( y ∣ x ) − η r ( x , y ) + η b t ⋆ ( x ) ] = D KL ( π t + 1 ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x )) − η J x ( π t + 1 ) + η b t ⋆ ( x ) ≥ − η J x ( π t + 1 ) + η b t ⋆ ( x ) ,
by nonnegativity of KL divergence. Rearranging gives the second inequality.
Lemma D.2 (Coverage from accumulated KL error). Fix x ∈ X x\in\mathcal{X} x ∈ X and a ∈ [ 0 , R max ] a\in[0,R_{\max}] a ∈ [ 0 , R m a x ] , and let Y a ( x ) = { y ∈ Y : r ( x , y ) ≥ a } \mathcal{Y}_a(x)=\{y\in\mathcal{Y}:r(x,y)\ge a\} Y a ( x ) = { y ∈ Y : r ( x , y ) ≥ a } . Then, for every 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T ,
π t ( Y a ( x ) ∣ x ) ≥ π b a s e ( Y a ( x ) ∣ x ) − ∑ s = 0 t − 1 K L s ( x ) . \sqrt{\pi_t(\mathcal{Y}_a(x)\mid x)}\ge\sqrt{\pi_{\mathrm{base}}(\mathcal{Y}_a(x)\mid x)}-\sum_{s=0}^{t-1}\sqrt{\mathrm{KL}_s(x)}. π t ( Y a ( x ) ∣ x ) ≥ π base ( Y a ( x ) ∣ x ) − s = 0 ∑ t − 1 KL s ( x ) .
Proof of Lemma D.2.
For every 0 ≤ t < T 0\le t<T 0 ≤ t < T , Lemma D.5 gives
∑ y ( π t ⋆ ( y ∣ x ) − π t + 1 ( y ∣ x ) ) 2 ≤ K L t ( x ) . \sum_y\left(\sqrt{\pi_{t}^{\star}(y\mid x)}-\sqrt{\pi_{t+1}(y\mid x)}\right)^2\le\mathrm{KL}_t(x). y ∑ ( π t ⋆ ( y ∣ x ) − π t + 1 ( y ∣ x ) ) 2 ≤ KL t ( x ) .
Applying the reverse triangle inequality to the vectors ( π t + 1 ( y ∣ x ) ) y ∈ Y a ( x ) \left(\sqrt{\pi_{t+1}(y\mid x)}\right)_{y\in\mathcal{Y}_a(x)} ( π t + 1 ( y ∣ x ) ) y ∈ Y a ( x ) and ( π t ⋆ ( y ∣ x ) ) y ∈ Y a ( x ) \left(\sqrt{\pi_{t}^{\star}(y\mid x)}\right)_{y\in\mathcal{Y}_a(x)} ( π t ⋆ ( y ∣ x ) ) y ∈ Y a ( x ) then gives
π t + 1 ( Y a ( x ) ∣ x ) ≥ π t , t ⋆ ( Y a ( x ) ∣ x ) − ( ∑ y ∈ Y a ( x ) ( π t ⋆ ( y ∣ x ) − π t + 1 ( y ∣ x ) ) 2 ) 1 / 2 ≥ π t , t ⋆ ( Y a ( x ) ∣ x ) − K L t ( x ) . (65) \begin{aligned}
\sqrt{\pi_{t+1}(\mathcal{Y}_a(x)\mid x)}\ge\sqrt{\pi_{t,t}^{\star}(\mathcal{Y}_a(x)\mid x)}-\left(\sum_{y\in\mathcal{Y}_a(x)}\left(\sqrt{\pi_{t}^{\star}(y\mid x)}-\sqrt{\pi_{t+1}(y\mid x)}\right)^2\right)^{1/2} \\
&\ge\sqrt{\pi_{t,t}^{\star}(\mathcal{Y}_a(x)\mid x)}-\sqrt{\mathrm{KL}_t(x)}. \tag*{(65)}
\end{aligned} π t + 1 ( Y a ( x ) ∣ x ) ≥ π t , t ⋆ ( Y a ( x ) ∣ x ) − y ∈ Y a ( x ) ∑ ( π t ⋆ ( y ∣ x ) − π t + 1 ( y ∣ x ) ) 2 1/2 ≥ π t , t ⋆ ( Y a ( x ) ∣ x ) − KL t ( x ) . (65)
The exact update also satisfies
e η b t ⋆ ( x ) ( π t , t ⋆ ( Y a ( x ) ∣ x ) − π t ( Y a ( x ) ∣ x ) ) = ∑ y ∈ Y a ( x ) ∑ y ′ ∉ Y a ( x ) π t ( y ∣ x ) π t ( y ′ ∣ x ) ( e η r ( x , y ) − e η r ( x , y ′ ) ) ≥ 0. (66) \begin{aligned}
e^{\eta b_t^\star(x)}\left(\pi_{t,t}^{\star}(\mathcal{Y}_a(x)\mid x)-\pi_t(\mathcal{Y}_a(x)\mid x)\right)
&=\sum_{y\in\mathcal{Y}_a(x)}\sum_{y'\notin\mathcal{Y}_a(x)}\pi_t(y\mid x)\pi_t(y'\mid x)\left(e^{\eta r(x,y)}-e^{\eta r(x,y')}\right)\ge0. \tag*{(66)}
\end{aligned} e η b t ⋆ ( x ) ( π t , t ⋆ ( Y a ( x ) ∣ x ) − π t ( Y a ( x ) ∣ x ) ) = y ∈ Y a ( x ) ∑ y ′ ∈ / Y a ( x ) ∑ π t ( y ∣ x ) π t ( y ′ ∣ x ) ( e η r ( x , y ) − e η r ( x , y ′ ) ) ≥ 0. (66)
Combining with Eq. (65) gives
π t + 1 ( Y a ( x ) ∣ x ) ≥ π t ( Y a ( x ) ∣ x ) − K L t ( x ) . \sqrt{\pi_{t+1}(\mathcal{Y}_a(x)\mid x)}\ge\sqrt{\pi_t(\mathcal{Y}_a(x)\mid x)}-\sqrt{\mathrm{KL}_t(x)}. π t + 1 ( Y a ( x ) ∣ x ) ≥ π t ( Y a ( x ) ∣ x ) − KL t ( x ) .
For any 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T , summing over 0 ≤ s < t 0\le s<t 0 ≤ s < t gives
π t ( Y a ( x ) ∣ x ) ≥ π b a s e ( Y a ( x ) ∣ x ) − ∑ s = 0 t − 1 K L s ( x ) . \sqrt{\pi_t(\mathcal{Y}_a(x)\mid x)}\ge\sqrt{\pi_{\mathrm{base}}(\mathcal{Y}_a(x)\mid x)}-\sum_{s=0}^{t-1}\sqrt{\mathrm{KL}_s(x)}. π t ( Y a ( x ) ∣ x ) ≥ π base ( Y a ( x ) ∣ x ) − s = 0 ∑ t − 1 KL s ( x ) .
The intuition for the following lemma is that under exact updates, π t ( y ∣ x ) π b a s e ( y ∣ x ) = e η ∑ s = 0 t − 1 ( r ( x , y ) − b s ∗ ( x ) ) \frac{\pi_t(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}=e^{\eta\sum_{s=0}^{t-1}(r(x,y)-b_s^*(x))} π base ( y ∣ x ) π t ( y ∣ x ) = e η ∑ s = 0 t − 1 ( r ( x , y ) − b s ∗ ( x )) . This result characterizes how this relationship is perturbed by cumulative approximation errors, as measured in expectation over π b a s e \pi_{\mathrm{base}} π base for a given function of interest f ( x , y ) f(x,y) f ( x , y ) . The square-root form is better suited to accounting for approximation errors measured in KL divergence, which accumulate additively when updates are unrolled.
Lemma D.3 (Accumulation of KL error). Let f : X × Y → [ 0 , ∞ ) f:\mathcal{X}\times\mathcal{Y}\to[0,\infty) f : X × Y → [ 0 , ∞ ) be bounded. For every 1 ≤ t ≤ T 1\le t\le T 1 ≤ t ≤ T ,
E π b a s e [ f ( x , y ) π t ( y ∣ x ) π b a s e ( y ∣ x ) ] ≥ E π b a s e [ f ( x , y ) e η 2 ∑ s = 0 t − 1 ( r ( x , y ) − b s ∗ ( x ) ) ] − ∑ s = 1 t K L s − 1 ( x ) ( E π b a s e [ f ( x , y ) 2 e η ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x ) ) ] ) 1 / 2 . (67) \mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)\sqrt{\frac{\pi_t(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}}\right]\geq\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)e^{\frac{\eta}{2}\sum_{s=0}^{t-1}(r(x,y)-b_s^*(x))}\right]-\sum_{s=1}^{t}\sqrt{\mathrm{KL}_{s-1}(x)}\left(\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)^2e^{\eta\sum_{u=s}^{t-1}(r(x,y)-b_u^*(x))}\right]\right)^{1/2}. \tag*{(67)} E π base [ f ( x , y ) π base ( y ∣ x ) π t ( y ∣ x ) ] ≥ E π base [ f ( x , y ) e 2 η ∑ s = 0 t − 1 ( r ( x , y ) − b s ∗ ( x )) ] − s = 1 ∑ t KL s − 1 ( x ) ( E π base [ f ( x , y ) 2 e η ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x )) ] ) 1/2 . (67)
Proof of Lemma D.3. By Lemma D.5, for every x x x and 0 ≤ t < T 0\leq t<T 0 ≤ t < T ,
∑ y ( π η , t ∗ ( y ∣ x ) − π t + 1 ( y ∣ x ) ) 2 ≤ K L t ( x ) . (68) \sum_y\left(\sqrt{\pi_{\eta,t}^*(y\mid x)}-\sqrt{\pi_{t+1}(y\mid x)}\right)^2\leq\mathrm{KL}_t(x). \tag*{(68)} y ∑ ( π η , t ∗ ( y ∣ x ) − π t + 1 ( y ∣ x ) ) 2 ≤ KL t ( x ) . (68)
At each ( x , y ) (x,y) ( x , y ) , the definition of the exact update π η , t ∗ \pi_{\eta,t}^* π η , t ∗ gives
π t + 1 ( y ∣ x ) = π t ( y ∣ x ) e η 2 ( r ( x , y ) − b t ∗ ( x ) ) − ( π η , t ∗ ( y ∣ x ) − π t + 1 ( y ∣ x ) ) . \sqrt{\pi_{t+1}(y\mid x)}=\sqrt{\pi_t(y\mid x)}e^{\frac{\eta}{2}(r(x,y)-b_t^*(x))}-\left(\sqrt{\pi_{\eta,t}^*(y\mid x)}-\sqrt{\pi_{t+1}(y\mid x)}\right). π t + 1 ( y ∣ x ) = π t ( y ∣ x ) e 2 η ( r ( x , y ) − b t ∗ ( x )) − ( π η , t ∗ ( y ∣ x ) − π t + 1 ( y ∣ x ) ) .
For any 1 ≤ t ≤ T 1\leq t\leq T 1 ≤ t ≤ T , unrolling over 0 ≤ s < t 0\leq s<t 0 ≤ s < t gives
π t ( y ∣ x ) = π b a s e ( y ∣ x ) e η 2 ∑ s = 0 t − 1 ( r ( x , y ) − b s ∗ ( x ) ) − ∑ s = 1 t ( π η , s − 1 ∗ ( y ∣ x ) − π s ( y ∣ x ) ) e η 2 ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x ) ) . (69) \sqrt{\pi_t(y\mid x)}=\sqrt{\pi_{\mathrm{base}}(y\mid x)}e^{\frac{\eta}{2}\sum_{s=0}^{t-1}(r(x,y)-b_s^*(x))}
-\sum_{s=1}^{t}\left(\sqrt{\pi_{\eta,s-1}^*(y\mid x)}-\sqrt{\pi_s(y\mid x)}\right)e^{\frac{\eta}{2}\sum_{u=s}^{t-1}(r(x,y)-b_u^*(x))}. \tag*{(69)} π t ( y ∣ x ) = π base ( y ∣ x ) e 2 η ∑ s = 0 t − 1 ( r ( x , y ) − b s ∗ ( x )) − s = 1 ∑ t ( π η , s − 1 ∗ ( y ∣ x ) − π s ( y ∣ x ) ) e 2 η ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x )) . (69)
Fix x x x , multiply by π b a s e ( y ∣ x ) f ( x , y ) \sqrt{\pi_{\mathrm{base}}(y\mid x)}f(x,y) π base ( y ∣ x ) f ( x , y ) , and sum over responses. For each term in the sum on the RHS, the Cauchy–Schwarz inequality and Eq. (68) give
∑ y π b a s e ( y ∣ x ) f ( x , y ) ( π η , s − 1 ∗ ( y ∣ x ) − π s ( y ∣ x ) ) e η 2 ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x ) ) ≤ ( ∑ y ( π η , s − 1 ∗ ( y ∣ x ) − π s ( y ∣ x ) ) 2 ) 1 / 2 ( E y ∼ π b a s e ( ⋅ ∣ x ) [ f ( x , y ) 2 e η ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x ) ) ] ) 1 / 2 ≤ K L s − 1 ( x ) ( E y ∼ π b a s e ( ⋅ ∣ x ) [ f ( x , y ) 2 e η ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x ) ) ] ) 1 / 2 . \begin{aligned}
&\sum_y\sqrt{\pi_{\mathrm{base}}(y\mid x)}f(x,y)\left(\sqrt{\pi_{\eta,s-1}^*(y\mid x)}-\sqrt{\pi_s(y\mid x)}\right)e^{\frac{\eta}{2}\sum_{u=s}^{t-1}(r(x,y)-b_u^*(x))}\\
&\leq\left(\sum_y\left(\sqrt{\pi_{\eta,s-1}^*(y\mid x)}-\sqrt{\pi_s(y\mid x)}\right)^2\right)^{1/2}\left(\mathbb{E}_{y\sim\pi_{\mathrm{base}}(\cdot\mid x)}\left[f(x,y)^2e^{\eta\sum_{u=s}^{t-1}(r(x,y)-b_u^*(x))}\right]\right)^{1/2}\\
&\leq\sqrt{\mathrm{KL}_{s-1}(x)}\left(\mathbb{E}_{y\sim\pi_{\mathrm{base}}(\cdot\mid x)}\left[f(x,y)^2e^{\eta\sum_{u=s}^{t-1}(r(x,y)-b_u^*(x))}\right]\right)^{1/2}.
\end{aligned} y ∑ π base ( y ∣ x ) f ( x , y ) ( π η , s − 1 ∗ ( y ∣ x ) − π s ( y ∣ x ) ) e 2 η ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x )) ≤ ( y ∑ ( π η , s − 1 ∗ ( y ∣ x ) − π s ( y ∣ x ) ) 2 ) 1/2 ( E y ∼ π base ( ⋅ ∣ x ) [ f ( x , y ) 2 e η ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x )) ] ) 1/2 ≤ KL s − 1 ( x ) ( E y ∼ π base ( ⋅ ∣ x ) [ f ( x , y ) 2 e η ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x )) ] ) 1/2 .
Substituting into Eq. (69) yields
E y ∼ π b a s e ( ⋅ ∣ x ) [ f ( x , y ) π t ( y ∣ x ) π b a s e ( y ∣ x ) ] ≥ E y ∼ π b a s e ( ⋅ ∣ x ) [ f ( x , y ) e η 2 ∑ s = 0 t − 1 ( r ( x , y ) − b s ∗ ( x ) ) ] − ∑ s = 1 t K L s − 1 ( x ) ( E y ∼ π b a s e ( ⋅ ∣ x ) [ f ( x , y ) 2 e η ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x ) ) ] ) 1 / 2 . \begin{aligned}
\mathbb{E}_{y\sim\pi_{\mathrm{base}}(\cdot\mid x)}\left[f(x,y)\sqrt{\frac{\pi_t(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}}\right]
&\geq\mathbb{E}_{y\sim\pi_{\mathrm{base}}(\cdot\mid x)}\left[f(x,y)e^{\frac{\eta}{2}\sum_{s=0}^{t-1}(r(x,y)-b_s^*(x))}\right]\\
&\quad-\sum_{s=1}^{t}\sqrt{\mathrm{KL}_{s-1}(x)}\left(\mathbb{E}_{y\sim\pi_{\mathrm{base}}(\cdot\mid x)}\left[f(x,y)^2e^{\eta\sum_{u=s}^{t-1}(r(x,y)-b_u^*(x))}\right]\right)^{1/2}.
\end{aligned} E y ∼ π base ( ⋅ ∣ x ) [ f ( x , y ) π base ( y ∣ x ) π t ( y ∣ x ) ] ≥ E y ∼ π base ( ⋅ ∣ x ) [ f ( x , y ) e 2 η ∑ s = 0 t − 1 ( r ( x , y ) − b s ∗ ( x )) ] − s = 1 ∑ t KL s − 1 ( x ) ( E y ∼ π base ( ⋅ ∣ x ) [ f ( x , y ) 2 e η ∑ u = s t − 1 ( r ( x , y ) − b u ∗ ( x )) ] ) 1/2 .
Taking the expectation over x ∼ μ x\sim\mu x ∼ μ and applying the Cauchy–Schwarz inequality to each error term proves Eq. (67) .
Lemma D.4. Fix x ∈ X x \in\mathcal{X} x ∈ X . For every 0 ≤ t < T 0 \le t < T 0 ≤ t < T ,
b t ⋆ ( x ) ≤ J x ( π T ) + T K L ‾ ( x ) η . b_t^\star(x) \le J_x(\pi_T) + \frac{T\overline{\mathrm{KL}}(x)}{\eta}. b t ⋆ ( x ) ≤ J x ( π T ) + η T KL ( x ) .
Proof of Lemma D.4 . Fix 0 ≤ t < T 0 \le t < T 0 ≤ t < T . For every t < s < T t < s < T t < s < T , Lemma D.1 gives
J x ( π s ) − J x ( π s + 1 ) ≤ K L s ( x ) η . J_x(\pi_s) - J_x(\pi_{s+1}) \le\frac{\mathrm{KL}_s(x)}{\eta}. J x ( π s ) − J x ( π s + 1 ) ≤ η KL s ( x ) .
Summing these inequalities over t < s < T t < s < T t < s < T , we obtain
J x ( π t + 1 ) − J x ( π T ) = ∑ s = t + 1 T − 1 ( J x ( π s ) − J x ( π s + 1 ) ) ≤ 1 η ∑ s = t + 1 T − 1 K L s ( x ) . J_x(\pi_{t+1}) - J_x(\pi_T) = \sum_{s=t+1}^{T-1} \left(J_x(\pi_s) - J_x(\pi_{s+1})\right) \le\frac{1}{\eta}\sum_{s=t+1}^{T-1}\mathrm{KL}_s(x). J x ( π t + 1 ) − J x ( π T ) = s = t + 1 ∑ T − 1 ( J x ( π s ) − J x ( π s + 1 ) ) ≤ η 1 s = t + 1 ∑ T − 1 KL s ( x ) .
Then, applying the second inequality in Lemma D.1 at iteration t t t and combining with the preceding inequality, we have
b t ⋆ ( x ) ≤ J x ( π t + 1 ) + K L t ( x ) η ≤ J x ( π T ) + 1 η ∑ s = t T − 1 K L s ( x ) . b_t^\star(x) \le J_x(\pi_{t+1}) + \frac{\mathrm{KL}_t(x)}{\eta} \le J_x(\pi_T) + \frac{1}{\eta}\sum_{s=t}^{T-1}\mathrm{KL}_s(x). b t ⋆ ( x ) ≤ J x ( π t + 1 ) + η KL t ( x ) ≤ J x ( π T ) + η 1 s = t ∑ T − 1 KL s ( x ) .
The result follows from ∑ s = t T − 1 K L s ( x ) ≤ ∑ s = 0 T − 1 K L s ( x ) = T K L ‾ ( x ) \sum_{s=t}^{T-1}\mathrm{KL}_s(x) \le\sum_{s=0}^{T-1}\mathrm{KL}_s(x) = T\overline{\mathrm{KL}}(x) ∑ s = t T − 1 KL s ( x ) ≤ ∑ s = 0 T − 1 KL s ( x ) = T KL ( x ) .
Lemma D.5 (KL–Hellinger lower bound). For any probability distributions p , q ∈ Δ ( Y ) p,q \in\Delta(\mathcal{Y}) p , q ∈ Δ ( Y ) ,
D K L ( p ∥ q ) ≥ ∑ y ∈ Y ( p ( y ) − q ( y ) ) 2 . D_{\mathrm{KL}}(p \mathbin\Vert q) \ge\sum_{y \in\mathcal{Y}}\left(\sqrt{p(y)}-\sqrt{q(y)}\right)^2. D KL ( p ∥ q ) ≥ y ∈ Y ∑ ( p ( y ) − q ( y ) ) 2 .
Proof of Lemma D.5 . The inequality − log u ≥ 2 ( 1 − u ) -\log u \ge2(1-\sqrt{u}) − log u ≥ 2 ( 1 − u ) gives
D K L ( p ∥ q ) ≥ 2 ∑ y : p ( y ) > 0 p ( y ) ( 1 − q ( y ) p ( y ) ) = 2 − 2 ∑ y p ( y ) q ( y ) = ∑ y ( p ( y ) − q ( y ) ) 2 . \begin{aligned}
D_{\mathrm{KL}}(p \mathbin\Vert q) \ge2\sum_{y:p(y)>0}p(y)\left(1-\sqrt{\frac{q(y)}{p(y)}}\right) \\
&= 2-2\sum_y\sqrt{p(y)q(y)}=\sum_y\left(\sqrt{p(y)}-\sqrt{q(y)}\right)^2.
\end{aligned} D KL ( p ∥ q ) ≥ 2 y : p ( y ) > 0 ∑ p ( y ) ( 1 − p ( y ) q ( y ) ) = 2 − 2 y ∑ p ( y ) q ( y ) = y ∑ ( p ( y ) − q ( y ) ) 2 .
If q ( y ) = 0 < p ( y ) q(y)=0<p(y) q ( y ) = 0 < p ( y ) for some y y y , the KL divergence is infinite so the claim in the lemma is immediate.
Proof of 3.1
We first state Theorem D.1, which extends the exact policy mirror descent analysis in Theorem B.3 to approximate updates, where approximation error is quantified by the KL divergence D K L ( π t + 1 ∥ π t ⋆ ) D_{\mathrm{KL}}(\pi_{t+1}\mathbin\Vert\pi^\star_{t}) D KL ( π t + 1 ∥ π t ⋆ ) . The proof of Theorem 3.1 directly follows below; substituting the prescribed inner KL tolerance into Theorem D.1 yields the main theorem. Finally, we prove Theorem D.1.
Theorem D.1 (Global convergence under APMD). The final policy π T \pi_T π T satisfies
J ⋆ − J ( π T ) ≤ E x ∼ μ [ log ( 1 / p 0 ( x ) ) ] η T + 1 η ∑ t = 0 T − 1 K L t + 2 e η C b a s e ∑ t = 0 T − 1 K L t t + 1 . (70) J^\star-J(\pi_T) \le\frac{\mathbb{E}_{x\sim\mu}[\log(1/p_0(x))]}{\eta T}+\frac{1}{\eta}\sum_{t=0}^{T-1}\mathrm{KL}_t+\frac{2\sqrt{e}}{\eta}\sqrt{C_{\mathrm{base}}\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t}{t+1}}. \tag*{(70)} J ⋆ − J ( π T ) ≤ η T E x ∼ μ [ log ( 1/ p 0 ( x ))] + η 1 t = 0 ∑ T − 1 KL t + η 2 e C base t = 0 ∑ T − 1 t + 1 KL t . (70)
Moreover, for any c ∈ ( 0 , 1 ) c \in(0,1) c ∈ ( 0 , 1 ) , with probability at least 1 − C b a s e 1 − c ∑ t = 0 T − 1 K L t 1-\frac{\sqrt{C_{\mathrm{base}}}}{1-\sqrt{c}}\sum_{t=0}^{T-1}\sqrt{\mathrm{KL}_t} 1 − 1 − c C base ∑ t = 0 T − 1 KL t over x ∼ μ x \sim\mu x ∼ μ ,
π t ( Y x ⋆ ∣ x ) ≥ c π b a s e ( Y x ⋆ ∣ x ) , ∀ 0 ≤ t ≤ T . (71) \pi_t(\mathcal{Y}^{\star}_x\mid x) \ge c\pi_{\mathrm{base}}(\mathcal{Y}^{\star}_x\mid x), \qquad\forall0 \le t \le T. \tag*{(71)} π t ( Y x ⋆ ∣ x ) ≥ c π base ( Y x ⋆ ∣ x ) , ∀0 ≤ t ≤ T . (71)
Proof of Theorem 3.1. We first consider the coverage claim in Theorem 3.1. We apply Theorem D.1 with c = 1 / 2 c=1/2 c = 1/2 and the prescribed inner KL tolerance,
K L t ≤ ε i n = 1 64 C b a s e T 2 log ( e T ) , 0 ≤ t < T . (72) \mathrm{KL}_t \le\varepsilon_{\mathrm{in}}=\frac{1}{64C_{\mathrm{base}}T^2\log(eT)}, \qquad0 \le t<T. \tag*{(72)} KL t ≤ ε in = 64 C base T 2 log ( e T ) 1 , 0 ≤ t < T . (72)
Since ∑ t = 0 T − 1 K L t ≤ T ε i n \sum_{t=0}^{T-1}\sqrt{\mathrm{KL}_t}\le T\sqrt{\varepsilon_{\mathrm{in}}} ∑ t = 0 T − 1 KL t ≤ T ε in , the theorem gives
P x ∼ μ ( π t ( Y x ⋆ ∣ x ) ≥ 1 2 π b a s e ( Y x ⋆ ∣ x ) , ∀ 0 ≤ t ≤ T ) ≥ 1 − ( 2 + 2 ) T C b a s e ε i n = 1 − 2 + 2 8 log ( e T ) ≥ 1 − 1 2 log ( e T ) , \begin{aligned}
\mathbb{P}_{x\sim\mu}\left(\pi_t(\mathcal{Y}^{\star}_x\mid x)\ge\frac{1}{2}\pi_{\mathrm{base}}(\mathcal{Y}^{\star}_x\mid x),\ \forall0\le t\le T\right)\ge1-(2+\sqrt{2})T\sqrt{C_{\mathrm{base}}\varepsilon_{\mathrm{in}}} \\
&=1-\frac{2+\sqrt{2}}{8\sqrt{\log(eT)}}\ge1-\frac{1}{2\sqrt{\log(eT)}},
\end{aligned} P x ∼ μ ( π t ( Y x ⋆ ∣ x ) ≥ 2 1 π base ( Y x ⋆ ∣ x ) , ∀0 ≤ t ≤ T ) ≥ 1 − ( 2 + 2 ) T C base ε in = 1 − 8 log ( e T ) 2 + 2 ≥ 1 − 2 log ( e T ) 1 ,
which is the claimed coverage guarantee. To prove the convergence guarantee in Theorem 3.1, we again apply Theorem D.1 with Eq. (72) . Since E x ∼ μ [ log ( 1 / p 0 ( x ) ) ] ≤ log C b a s e \mathbb{E}_{x\sim\mu}[\log(1/p_0(x))]\le\log C_{\mathrm{base}} E x ∼ μ [ log ( 1/ p 0 ( x ))] ≤ log C base by Jensen’s inequality and ∑ t = 1 T 1 / t ≤ log ( e T ) \sum_{t=1}^{T}1/t\le\log(eT) ∑ t = 1 T 1/ t ≤ log ( e T ) , we obtain
J ⋆ − J ( π T ) ≤ log C b a s e η T + T ε i n η + 2 e η C b a s e ε i n log ( e T ) ≤ log C b a s e + 1 / 64 + e / 4 η T ≤ log ( 2 C b a s e ) η T . \begin{aligned}
J^{\star}-J(\pi_T)\le\frac{\log C_{\mathrm{base}}}{\eta T}+\frac{T\varepsilon_{\mathrm{in}}}{\eta}+\frac{2\sqrt{e}}{\eta}\sqrt{C_{\mathrm{base}}\varepsilon_{\mathrm{in}}\log(eT)} \\
&\le\frac{\log C_{\mathrm{base}}+1/64+\sqrt{e}/4}{\eta T}\le\frac{\log(2C_{\mathrm{base}})}{\eta T}.
\end{aligned} J ⋆ − J ( π T ) ≤ η T log C base + η T ε in + η 2 e C base ε in log ( e T ) ≤ η T log C base + 1/64 + e /4 ≤ η T log ( 2 C base ) .
This proves the convergence claim in Eq. (7) .
Next, we prove Theorem D.1, the approximate analogue of the mirror descent convergence guarantee in Theorem B.3. We use Lemma D.6, proved below, for the suboptimality bound and Lemma D.2 for the coverage guarantee.
Proof of Theorem D.1. We first prove the convergence bound by applying Lemma D.6 with t = T t=T t = T and averaging over prompts. Since J ⋆ − J ( π T ) = E x ∼ μ [ r x ⋆ − J x ( π T ) ] J^{\star}-J(\pi_T)=\mathbb{E}_{x\sim\mu}[r_x^{\star}-J_x(\pi_T)] J ⋆ − J ( π T ) = E x ∼ μ [ r x ⋆ − J x ( π T )] , this gives
J ⋆ − J ( π T ) ≤ E x ∼ μ [ log ( 1 / p 0 ( x ) ) ] η T + 1 η ∑ t = 0 T − 1 K L t + 2 e η E x ∼ μ [ 1 p 0 ( x ) ∑ t = 0 T − 1 K L t ( x ) t + 1 ] . J^{\star}-J(\pi_T)\le\frac{\mathbb{E}_{x\sim\mu}[\log(1/p_0(x))]}{\eta T}+\frac{1}{\eta}\sum_{t=0}^{T-1}\mathrm{KL}_t+\frac{2\sqrt{e}}{\eta}\mathbb{E}_{x\sim\mu}\left[\sqrt{\frac{1}{p_0(x)}\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t(x)}{t+1}}\right]. J ⋆ − J ( π T ) ≤ η T E x ∼ μ [ log ( 1/ p 0 ( x ))] + η 1 t = 0 ∑ T − 1 KL t + η 2 e E x ∼ μ p 0 ( x ) 1 t = 0 ∑ T − 1 t + 1 KL t ( x ) .
To further bound the last term, the Cauchy–Schwarz inequality gives
E x ∼ μ [ 1 p 0 ( x ) ∑ t = 0 T − 1 K L t ( x ) t + 1 ] ≤ E x ∼ μ [ 1 p 0 ( x ) ] E x ∼ μ [ ∑ t = 0 T − 1 K L t ( x ) t + 1 ] = C b a s e ∑ t = 0 T − 1 K L t t + 1 . \begin{aligned}
\mathbb{E}_{x\sim\mu}\left[\sqrt{\frac{1}{p_0(x)}\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t(x)}{t+1}}\right]\le\sqrt{\mathbb{E}_{x\sim\mu}\left[\frac{1}{p_0(x)}\right]\mathbb{E}_{x\sim\mu}\left[\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t(x)}{t+1}\right]} \\
&=\sqrt{C_{\mathrm{base}}\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t}{t+1}}.
\end{aligned} E x ∼ μ p 0 ( x ) 1 t = 0 ∑ T − 1 t + 1 KL t ( x ) ≤ E x ∼ μ [ p 0 ( x ) 1 ] E x ∼ μ [ t = 0 ∑ T − 1 t + 1 KL t ( x ) ] = C base t = 0 ∑ T − 1 t + 1 KL t .
Combining the above two panels gives Eq. (70) .
Next, to prove the coverage guarantee, fix c ∈ ( 0 , 1 ) c\in(0,1) c ∈ ( 0 , 1 ) and apply Lemma D.2 with a = r x ⋆ a=r_x^{\star} a = r x ⋆ . For every prompt x x x satisfying ∑ s = 0 T − 1 K L s ( x ) / p 0 ( x ) ≤ 1 − c \sum_{s=0}^{T-1}\sqrt{\mathrm{KL}_s(x)/p_0(x)}\le1-\sqrt{c} ∑ s = 0 T − 1 KL s ( x ) / p 0 ( x ) ≤ 1 − c , the lemma gives
p t ( x ) ≥ p 0 ( x ) − ∑ s = 0 t − 1 K L s ( x ) ≥ c p 0 ( x ) , ∀ 0 ≤ t ≤ T . \sqrt{p_t(x)}\ge\sqrt{p_0(x)}-\sum_{s=0}^{t-1}\sqrt{\mathrm{KL}_s(x)}\ge\sqrt{cp_0(x)},\qquad\forall0\le t\le T. p t ( x ) ≥ p 0 ( x ) − s = 0 ∑ t − 1 KL s ( x ) ≥ c p 0 ( x ) , ∀0 ≤ t ≤ T .
Thus, π t ( Y x ⋆ ∣ x ) ≥ c π b a s e ( Y x ⋆ ∣ x ) \pi_t(\mathcal{Y}^{\star}_x \mid x) \ge c\pi_{\mathrm{base}}(\mathcal{Y}^{\star}_x \mid x) π t ( Y x ⋆ ∣ x ) ≥ c π base ( Y x ⋆ ∣ x ) for all 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T on every prompt satisfying the above condition. It remains to bound the probability that this condition does not hold, or ∑ s = 0 T − 1 K L s ( x ) / p 0 ( x ) > 1 − c \sum_{s=0}^{T-1}\sqrt{\mathrm{KL}_s(x)/p_0(x)} > 1-\sqrt{c} ∑ s = 0 T − 1 KL s ( x ) / p 0 ( x ) > 1 − c . Markov’s inequality and the Cauchy–Schwarz inequality give
P x ∼ μ ( ∑ s = 0 T − 1 K L s ( x ) p 0 ( x ) > 1 − c ) ≤ 1 1 − c ∑ s = 0 T − 1 E x ∼ μ [ K L s ( x ) p 0 ( x ) ] ≤ C b a s e 1 − c ∑ s = 0 T − 1 K L s . \begin{aligned}
\mathbb{P}_{x\sim\mu}\left(\sum_{s=0}^{T-1}\sqrt{\frac{\mathrm{KL}_s(x)}{p_0(x)}} > 1-\sqrt{c}\right) \le\frac{1}{1-\sqrt{c}}\sum_{s=0}^{T-1}\mathbb{E}_{x\sim\mu}\left[\sqrt{\frac{\mathrm{KL}_s(x)}{p_0(x)}}\right] \\
&\le\frac{\sqrt{\mathcal{C}_{\mathrm{base}}}}{1-\sqrt{c}}\sum_{s=0}^{T-1}\sqrt{\mathrm{KL}_s}.
\end{aligned} P x ∼ μ ( s = 0 ∑ T − 1 p 0 ( x ) KL s ( x ) > 1 − c ) ≤ 1 − c 1 s = 0 ∑ T − 1 E x ∼ μ [ p 0 ( x ) KL s ( x ) ] ≤ 1 − c C base s = 0 ∑ T − 1 KL s .
This proves the stated coverage guarantee.
We now prove the key lemma used in the proof above, an approximate version of the per-prompt suboptimality bound in Theorem B.3. The intuition is as follows. Under exact updates, the inequality p t ( x ) ≥ p 0 ( x ) e η t ( r x ⋆ − J x ( π t ) ) p_t(x) \ge p_0(x)e^{\eta t(r_x^{\star}-J_x(\pi_t))} p t ( x ) ≥ p 0 ( x ) e η t ( r x ⋆ − J x ( π t )) yields the mirror descent rate by rearranging. For approximate APMD updates, we use Lemma D.5 to bound the change in the probability of Y x ⋆ \mathcal{Y}^{\star}_x Y x ⋆ at each iteration.
Lemma D.6. For every prompt x x x and every 1 ≤ t ≤ T 1 \le t \le T 1 ≤ t ≤ T ,
r x ⋆ − J x ( π t ) ≤ log ( 1 / p 0 ( x ) ) η t + 1 η ∑ s = 0 t − 1 K L s ( x ) + 2 e η 1 p 0 ( x ) ∑ s = 0 t − 1 K L s ( x ) s + 1 . (73) r_x^{\star}-J_x(\pi_t) \le\frac{\log(1/p_0(x))}{\eta t}+\frac{1}{\eta}\sum_{s=0}^{t-1}\mathrm{KL}_s(x)+\frac{2\sqrt{e}}{\eta}\sqrt{\frac{1}{p_0(x)}\sum_{s=0}^{t-1}\frac{\mathrm{KL}_s(x)}{s+1}}. \tag*{(73)} r x ⋆ − J x ( π t ) ≤ η t log ( 1/ p 0 ( x )) + η 1 s = 0 ∑ t − 1 KL s ( x ) + η 2 e p 0 ( x ) 1 s = 0 ∑ t − 1 s + 1 KL s ( x ) . (73)
Proof of Lemma D.6 . Fix x ∈ X x \in\mathcal{X} x ∈ X and 1 ≤ t ≤ T 1 \le t \le T 1 ≤ t ≤ T . We first bound p s + 1 ( x ) p_{s+1}(x) p s + 1 ( x ) in terms of p s ( x ) p_s(x) p s ( x ) and K L s ( x ) \mathrm{KL}_s(x) KL s ( x ) . Since r ( x , y ) = r x ⋆ r(x,y)=r_x^{\star} r ( x , y ) = r x ⋆ on Y x ⋆ \mathcal{Y}^{\star}_x Y x ⋆ , the exact update satisfies
π η , s ⋆ ( Y x ⋆ ∣ x ) = p s ( x ) e η ( r x ⋆ − b x ⋆ ( x ) ) . \pi^{\star}_{\eta,s}(\mathcal{Y}^{\star}_x\mid x)=p_s(x)e^{\eta(r_x^{\star}-b_x^{\star}(x))}. π η , s ⋆ ( Y x ⋆ ∣ x ) = p s ( x ) e η ( r x ⋆ − b x ⋆ ( x )) .
For 0 ≤ s < t 0 \le s < t 0 ≤ s < t , apply the reverse triangle inequality to the vectors ( π s + 1 ( y ∣ x ) ) y ∈ Y x ⋆ (\sqrt{\pi_{s+1}(y\mid x)})_{y\in\mathcal{Y}^{\star}_x} ( π s + 1 ( y ∣ x ) ) y ∈ Y x ⋆ and ( π η , s ⋆ ( y ∣ x ) ) y ∈ Y x ⋆ (\sqrt{\pi^{\star}_{\eta,s}(y\mid x)})_{y\in\mathcal{Y}^{\star}_x} ( π η , s ⋆ ( y ∣ x ) ) y ∈ Y x ⋆ . Then using Lemma D.5 , we obtain
p s + 1 ( x ) ≥ π η , s ⋆ ( Y x ⋆ ∣ x ) − ( ∑ y ∈ Y x ⋆ ( π s + 1 ( y ∣ x ) − π η , s ⋆ ( y ∣ x ) ) 2 ) 1 / 2 ≥ p s ( x ) e η 2 ( r x ⋆ − b x ⋆ ( x ) ) − K L s ( x ) . \begin{aligned}
\sqrt{p_{s+1}(x)} \ge\sqrt{\pi^{\star}_{\eta,s}(\mathcal{Y}^{\star}_x\mid x)}-\left(\sum_{y\in\mathcal{Y}^{\star}_x}\left(\sqrt{\pi_{s+1}(y\mid x)}-\sqrt{\pi^{\star}_{\eta,s}(y\mid x)}\right)^2\right)^{1/2} \\
&\ge\sqrt{p_s(x)e^{\frac{\eta}{2}(r_x^{\star}-b_x^{\star}(x))}}-\sqrt{\mathrm{KL}_s(x)}.
\end{aligned} p s + 1 ( x ) ≥ π η , s ⋆ ( Y x ⋆ ∣ x ) − y ∈ Y x ⋆ ∑ ( π s + 1 ( y ∣ x ) − π η , s ⋆ ( y ∣ x ) ) 2 1/2 ≥ p s ( x ) e 2 η ( r x ⋆ − b x ⋆ ( x )) − KL s ( x ) .
Rearranging this inequality and unrolling from iterations t t t to 0 0 0 gives
p 0 ( x ) ≤ p t ( x ) e − η 2 ∑ u = 0 t − 1 ( r x ⋆ − b u ⋆ ( x ) ) + ∑ s = 1 t K L s − 1 ( x ) e − η 2 ∑ u = 0 s − 1 ( r x ⋆ − b u ⋆ ( x ) ) . \sqrt{p_0(x)}\le\sqrt{p_t(x)e^{-\frac{\eta}{2}\sum_{u=0}^{t-1}(r_x^{\star}-b_u^{\star}(x))}}+\sum_{s=1}^{t}\sqrt{\mathrm{KL}_{s-1}(x)e^{-\frac{\eta}{2}\sum_{u=0}^{s-1}(r_x^{\star}-b_u^{\star}(x))}}. p 0 ( x ) ≤ p t ( x ) e − 2 η ∑ u = 0 t − 1 ( r x ⋆ − b u ⋆ ( x )) + s = 1 ∑ t KL s − 1 ( x ) e − 2 η ∑ u = 0 s − 1 ( r x ⋆ − b u ⋆ ( x )) .
Next, we translate the exponents in the above bound to r x ⋆ − J x ( π t ) r_x^{\star}-J_x(\pi_t) r x ⋆ − J x ( π t ) . Applying Lemma D.4 gives, for every 0 ≤ s < t 0 \le s < t 0 ≤ s < t , b s ⋆ ( x ) ≤ J x ( π t ) + η − 1 ∑ u = 0 t − 1 K L u ( x ) b_s^{\star}(x)\le J_x(\pi_t)+\eta^{-1}\sum_{u=0}^{t-1}\mathrm{KL}_u(x) b s ⋆ ( x ) ≤ J x ( π t ) + η − 1 ∑ u = 0 t − 1 KL u ( x ) , so
p 0 ( x ) ≤ p t ( x ) e − η t 2 ( r x ⋆ − J x ( π t ) ) + t 2 ∑ u = 0 t − 1 K L u ( x ) + ∑ s = 1 t K L s − 1 ( x ) e − η s 2 ( r x ⋆ − J x ( π t ) ) + s 2 ∑ u = 0 t − 1 K L u ( x ) . (74) \begin{aligned}
\sqrt{p_0(x)}\le\sqrt{p_t(x)e^{-\frac{\eta t}{2}(r_x^{\star}-J_x(\pi_t))+\frac{t}{2}\sum_{u=0}^{t-1}\mathrm{KL}_u(x)}} \\
&+\sum_{s=1}^{t}\sqrt{\mathrm{KL}_{s-1}(x)e^{-\frac{\eta s}{2}(r_x^{\star}-J_x(\pi_t))+\frac{s}{2}\sum_{u=0}^{t-1}\mathrm{KL}_u(x)}}. \tag*{(74)}
\end{aligned} p 0 ( x ) ≤ p t ( x ) e − 2 η t ( r x ⋆ − J x ( π t )) + 2 t ∑ u = 0 t − 1 KL u ( x ) + s = 1 ∑ t KL s − 1 ( x ) e − 2 η s ( r x ⋆ − J x ( π t )) + 2 s ∑ u = 0 t − 1 KL u ( x ) . (74)
Finally, to prove the lemma claim from Eq. (74) , define
z t ( x ) : = η ( r x ⋆ − J x ( π t ) ) − ∑ s = 0 t − 1 K L s ( x ) − log ( 1 / p 0 ( x ) ) t , z_t(x):=\eta(r_x^{\star}-J_x(\pi_t))-\sum_{s=0}^{t-1}\mathrm{KL}_s(x)-\frac{\log(1/p_0(x))}{t}, z t ( x ) := η ( r x ⋆ − J x ( π t )) − s = 0 ∑ t − 1 KL s ( x ) − t log ( 1/ p 0 ( x )) ,
A t ( x ) : = 1 p 0 ( x ) ∑ s = 1 t K L s − 1 ( x ) s . A_t(x):=\sqrt{\frac{1}{p_0(x)}\sum_{s=1}^{t}\frac{\mathrm{KL}_{s-1}(x)}{s}}. A t ( x ) := p 0 ( x ) 1 s = 1 ∑ t s KL s − 1 ( x ) .
By the definition of z t ( x ) z_t(x) z t ( x ) , it suffices to prove
z t ( x ) ≤ 2 A t ( x ) , (75) z_t(x) \le2A_t(x), \tag*{(75)} z t ( x ) ≤ 2 A t ( x ) , (75)
which we return to in Eq. (77) . We will assume going forwards that z t ( x ) > 0 z_t(x)>0 z t ( x ) > 0 , otherwise Eq. (75) is immediate.
Dividing the bound in Eq. (74) by p 0 ( x ) \sqrt{p_0(x)} p 0 ( x ) and substituting the definition of z t ( x ) z_t(x) z t ( x ) gives
1 ≤ p t ( x ) e − t z t ( x ) / 2 + 1 p 0 ( x ) ∑ s = 1 t K L s − 1 ( x ) p 0 ( x ) s / ( 2 t ) e − s z t ( x ) / 2 . 1 \le\sqrt{p_t(x)}e^{-tz_t(x)/2}+\frac{1}{\sqrt{p_0(x)}}\sum_{s=1}^{t}\sqrt{\mathrm{KL}_{s-1}(x)}p_0(x)^{s/(2t)}e^{-sz_t(x)/2}. 1 ≤ p t ( x ) e − t z t ( x ) /2 + p 0 ( x ) 1 s = 1 ∑ t KL s − 1 ( x ) p 0 ( x ) s / ( 2 t ) e − s z t ( x ) /2 .
Using p 0 ( x ) , p t ( x ) ≤ 1 p_0(x),p_t(x)\le1 p 0 ( x ) , p t ( x ) ≤ 1 and rearranging, we obtain
1 − e − t z t ( x ) / 2 ≤ 1 p 0 ( x ) ∑ s = 1 t K L s − 1 ( x ) e − s z t ( x ) / 2 ≤ 1 p 0 ( x ) ( ∑ s = 1 t K L s − 1 ( x ) s ) 1 / 2 ( ∑ s = 1 t s e − s z t ( x ) ) 1 / 2 = A t ( x ) ( ∑ s = 1 t s e − s z t ( x ) ) 1 / 2 , (76) \begin{aligned}
1-e^{-tz_t(x)/2}
&\le\frac{1}{\sqrt{p_0(x)}}\sum_{s=1}^{t}\sqrt{\mathrm{KL}_{s-1}(x)}e^{-sz_t(x)/2}\\
&\le\frac{1}{\sqrt{p_0(x)}}\left(\sum_{s=1}^{t}\frac{\mathrm{KL}_{s-1}(x)}{s}\right)^{1/2}\left(\sum_{s=1}^{t}se^{-sz_t(x)}\right)^{1/2}\\
&= A_t(x)\left(\sum_{s=1}^{t}se^{-sz_t(x)}\right)^{1/2}, \tag*{(76)}
\end{aligned} 1 − e − t z t ( x ) /2 ≤ p 0 ( x ) 1 s = 1 ∑ t KL s − 1 ( x ) e − s z t ( x ) /2 ≤ p 0 ( x ) 1 ( s = 1 ∑ t s KL s − 1 ( x ) ) 1/2 ( s = 1 ∑ t s e − s z t ( x ) ) 1/2 = A t ( x ) ( s = 1 ∑ t s e − s z t ( x ) ) 1/2 , (76)
where we have used the Cauchy-Schwarz inequality in the second line, followed by the definition of A t ( x ) A_t(x) A t ( x ) . Next, to bound the sum on the RHS,
∑ s = 1 t s e − s z t ( x ) = ∑ s = 1 t ( ∑ j = 1 s e − s z t ( x ) / 2 ) e − s z t ( x ) / 2 ≤ ∑ s = 1 t ( ∑ j = 1 s e − j z t ( x ) / 2 ) e − s z t ( x ) / 2 ≤ ∑ s = 1 t ( ∑ j = 1 t e − j z t ( x ) / 2 ) e − s z t ( x ) / 2 = ( ∑ s = 1 t e − s z t ( x ) / 2 ) 2 . \begin{aligned}
\sum_{s=1}^{t}se^{-sz_t(x)}
&=\sum_{s=1}^{t}\left(\sum_{j=1}^{s}e^{-sz_t(x)/2}\right)e^{-sz_t(x)/2}\\
&\le\sum_{s=1}^{t}\left(\sum_{j=1}^{s}e^{-jz_t(x)/2}\right)e^{-sz_t(x)/2}\\
&\le\sum_{s=1}^{t}\left(\sum_{j=1}^{t}e^{-jz_t(x)/2}\right)e^{-sz_t(x)/2}
=\left(\sum_{s=1}^{t}e^{-sz_t(x)/2}\right)^2.
\end{aligned} s = 1 ∑ t s e − s z t ( x ) = s = 1 ∑ t ( j = 1 ∑ s e − s z t ( x ) /2 ) e − s z t ( x ) /2 ≤ s = 1 ∑ t ( j = 1 ∑ s e − j z t ( x ) /2 ) e − s z t ( x ) /2 ≤ s = 1 ∑ t ( j = 1 ∑ t e − j z t ( x ) /2 ) e − s z t ( x ) /2 = ( s = 1 ∑ t e − s z t ( x ) /2 ) 2 .
Substituting the above panel back into Eq. (76) , we obtain
1 − e − t z t ( x ) / 2 ≤ A t ( x ) ( ∑ s = 1 t e − s z t ( x ) / 2 ) = A t ( x ) 1 − e − t z t ( x ) / 2 e z t ( x ) / 2 − 1 . 1-e^{-tz_t(x)/2}\le A_t(x)\left(\sum_{s=1}^{t}e^{-sz_t(x)/2}\right)=A_t(x)\frac{1-e^{-tz_t(x)/2}}{e^{z_t(x)/2}-1}. 1 − e − t z t ( x ) /2 ≤ A t ( x ) ( s = 1 ∑ t e − s z t ( x ) /2 ) = A t ( x ) e z t ( x ) /2 − 1 1 − e − t z t ( x ) /2 .
Finally, rearranging the above gives Eq. (75) ,
z t ( x ) ≤ 2 log ( 1 + A t ( x ) ) ≤ 2 A t ( x ) . z_t(x)\le2\log(1+A_t(x))\le2A_t(x). z t ( x ) ≤ 2 log ( 1 + A t ( x )) ≤ 2 A t ( x ) .
This proves the lemma claim since, using the definition of z t ( x ) z_t(x) z t ( x ) ,
η ( r x ⋆ − J x ( π t ) ) = z t ( x ) + ∑ s = 0 t − 1 K L s ( x ) + log ( 1 / p 0 ( x ) ) t ≤ 2 A t ( x ) + ∑ s = 0 t − 1 K L s ( x ) + log ( 1 / p 0 ( x ) ) t = log ( 1 / p 0 ( x ) ) t + ∑ s = 0 t − 1 K L s ( x ) + 2 1 p 0 ( x ) ∑ s = 0 t − 1 K L s ( x ) s + 1 , (77) \begin{aligned}
\eta(r_x^\star-J_x(\pi_t))
&=z_t(x)+\sum_{s=0}^{t-1}\mathrm{KL}_s(x)+\frac{\log(1/p_0(x))}{t}\\
&\le2A_t(x)+\sum_{s=0}^{t-1}\mathrm{KL}_s(x)+\frac{\log(1/p_0(x))}{t}\\
&=\frac{\log(1/p_0(x))}{t}+\sum_{s=0}^{t-1}\mathrm{KL}_s(x)+2\sqrt{\frac{1}{p_0(x)}\sum_{s=0}^{t-1}\frac{\mathrm{KL}_s(x)}{s+1}}, \tag*{(77)}
\end{aligned} η ( r x ⋆ − J x ( π t )) = z t ( x ) + s = 0 ∑ t − 1 KL s ( x ) + t log ( 1/ p 0 ( x )) ≤ 2 A t ( x ) + s = 0 ∑ t − 1 KL s ( x ) + t log ( 1/ p 0 ( x )) = t log ( 1/ p 0 ( x )) + s = 0 ∑ t − 1 KL s ( x ) + 2 p 0 ( x ) 1 s = 0 ∑ t − 1 s + 1 KL s ( x ) , (77)
where we have used z t ( x ) ≤ 2 A t ( x ) z_t(x)\le2A_t(x) z t ( x ) ≤ 2 A t ( x ) in the second line and the definition of A t ( x ) A_t(x) A t ( x ) in the last equality. Dividing both sides by η \eta η yields Eq. (73) .
The Mean-Squared Error Criterion (Proof of Theorem 4.1)
This section proves Theorem 4.1, which establishes the mean-squared error criterion sufficient for convergence and coverage preservation in APMD. We first develop supporting lemmas in Appendix § 11.2 .§ 1 , then prove Theorem 4.1 in Appendix § 11.2 .§ 2 .
Notation. Throughout this section, fix η > 0 \eta> 0 η > 0 and T ≥ 1 T \ge1 T ≥ 1 , and a given sequence of policies { π t } t = 0 T \{\pi_t\}_{t=0}^{T} { π t } t = 0 T with π 0 = π b a s e \pi_0 = \pi_{\mathrm{base}} π 0 = π base and π t > 0 \pi_t > 0 π t > 0 . We retain the notation of Appendix D, including for a fixed prompt x x x the notation J x ( π ) J_x(\pi) J x ( π ) , r x ∗ r_x^* r x ∗ , γ x ∗ \gamma_x^* γ x ∗ , p t ( x ) p_t(x) p t ( x ) , and Y a ( x ) Y_a(x) Y a ( x ) . For 0 ≤ t < T 0 \le t < T 0 ≤ t < T , define the centered residual of π t + 1 \pi_{t+1} π t + 1 to be
s t ( x , y ) : = 1 η log π t + 1 ( y ∣ x ) π t ( y ∣ x ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ 1 η log π t + 1 ( y ′ ∣ x ) π t ( y ′ ∣ x ) ] − ( r ( x , y ) − J x ( π t ) ) , (78) s_t(x,y) := \frac{1}{\eta}\log\frac{\pi_{t+1}(y \mid x)}{\pi_t(y \mid x)}-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[\frac{1}{\eta}\log\frac{\pi_{t+1}(y' \mid x)}{\pi_t(y' \mid x)}\right]-(r(x,y)-J_x(\pi_t)), \tag*{(78)} s t ( x , y ) := η 1 log π t ( y ∣ x ) π t + 1 ( y ∣ x ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ η 1 log π t ( y ′ ∣ x ) π t + 1 ( y ′ ∣ x ) ] − ( r ( x , y ) − J x ( π t )) , (78)
and denote S : = max 0 ≤ t < T ∥ s t ∥ ∞ S := \max_{0\le t<T}\lVert s_t\rVert_\infty S := max 0 ≤ t < T ∥ s t ∥ ∞ as in Theorem 4.1.
Under Assumption 2.1, the definition of s t s_t s t gives
s t ( x , y ) = ( ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] ) ⊤ ( θ t + 1 − θ t η − θ ⋆ ) . (79) s_t(x,y)=\left(\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\phi(x,y')]\right)^\top\left(\frac{\theta_{t+1}-\theta_t}{\eta}-\theta_\star\right). \tag*{(79)} s t ( x , y ) = ( ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] ) ⊤ ( η θ t + 1 − θ t − θ ⋆ ) . (79)
We denote the mean-squared error conditional on x x x by MSE t ( x ) \operatorname{MSE}_t(x) MSE t ( x ) ,
MSE t ( x ) : = E y ∼ π t ( ⋅ ∣ x ) [ s t ( x , y ) 2 ] , \operatorname{MSE}_t(x):=\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[s_t(x,y)^2], MSE t ( x ) := E y ∼ π t ( ⋅ ∣ x ) [ s t ( x , y ) 2 ] ,
where MSE t : = E x ∼ μ [ MSE t ( x ) ] \operatorname{MSE}_t:=\mathbb{E}_{x\sim\mu}[\operatorname{MSE}_t(x)] MSE t := E x ∼ μ [ MSE t ( x )] .
Throughout this section, we will call upon the following rearrangement of π t + 1 \pi_{t+1} π t + 1 for convenience, and use the shifted normalization constant b ~ t \tilde b_t b ~ t . Define this to be
b ~ t ( x ) : = 1 η log E y ∼ π t ( ⋅ ∣ x ) [ e η ( r ( x , y ) + s t ( x , y ) ) ] , \tilde b_t(x):=\frac{1}{\eta}\log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta(r(x,y)+s_t(x,y))}\right], b ~ t ( x ) := η 1 log E y ∼ π t ( ⋅ ∣ x ) [ e η ( r ( x , y ) + s t ( x , y )) ] ,
then rearranging the definition of s t s_t s t gives
π t + 1 ( y ∣ x ) = π t ( y ∣ x ) e η ( r ( x , y ) + s t ( x , y ) − b ~ t ( x ) ) . (80) \pi_{t+1}(y\mid x)=\pi_t(y\mid x)e^{\eta(r(x,y)+s_t(x,y)-\tilde b_t(x))}. \tag*{(80)} π t + 1 ( y ∣ x ) = π t ( y ∣ x ) e η ( r ( x , y ) + s t ( x , y ) − b ~ t ( x )) . (80)
Supporting Lemmas
We first state and prove the supporting lemmas used in Appendix § 11.2 .§ 2 , adapting the arguments of Appendix D.§ 1 to the mean-squared error. They are organized as follows.
(Approximate improvement) Lemma E.2 bounds how much performance can drop from π t \pi_t π t to any π s \pi_s π s by the cumulative mean-squared errors.
(Coverage approximately preserved) Lemma E.3 bounds how much the probabilities of upper reward quantiles degrade from π 0 \pi_0 π 0 to π t \pi_t π t , in terms of the accumulated mean-squared error.
Lemma E.1 (Exponential moment of the centered residual). Fix x ∈ X x \in\mathcal{X} x ∈ X . For every 0 ≤ t < T 0 \le t < T 0 ≤ t < T ,
log E y ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ) ] ≤ η 2 e η S 2 MSE t ( x ) . \log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y)}\right]\le\frac{\eta^2 e^{\eta S}}{2}\operatorname{MSE}_t(x). log E y ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ) ] ≤ 2 η 2 e η S MSE t ( x ) .
Proof of Lemma E.1. Fix 0 ≤ t < T 0 \le t < T 0 ≤ t < T . Since ∣ s t ( x , y ) ∣ ≤ S \lvert s_t(x,y)\rvert\le S ∣ s t ( x , y )∣ ≤ S , Taylor’s theorem gives
e η s t ( x , y ) ≤ 1 + η s t ( x , y ) + η 2 e η S 2 s t ( x , y ) 2 . e^{\eta s_t(x,y)}\le1+\eta s_t(x,y)+\frac{\eta^2e^{\eta S}}{2}s_t(x,y)^2. e η s t ( x , y ) ≤ 1 + η s t ( x , y ) + 2 η 2 e η S s t ( x , y ) 2 .
We therefore obtain
log E y ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ) ] ≤ log ( 1 + η 2 e η S 2 E y ∼ π t ( ⋅ ∣ x ) [ s t ( x , y ) 2 ] ) ≤ η 2 e η S 2 MSE t ( x ) , \begin{aligned}
\log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y)}\right]
&\le\log\left(1+\frac{\eta^2e^{\eta S}}{2}\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[s_t(x,y)^2]\right)\\
&\le\frac{\eta^2e^{\eta S}}{2}\operatorname{MSE}_t(x),
\end{aligned} log E y ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ) ] ≤ log ( 1 + 2 η 2 e η S E y ∼ π t ( ⋅ ∣ x ) [ s t ( x , y ) 2 ] ) ≤ 2 η 2 e η S MSE t ( x ) ,
where the first inequality uses the fact that E y ′ ∼ π t ( ⋅ ∣ x ) [ s t ( x , y ′ ) ] = 0 \mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[s_t(x,y')]=0 E y ′ ∼ π t ( ⋅ ∣ x ) [ s t ( x , y ′ )] = 0 , and the second uses log ( 1 + u ) ≤ u \log(1+u)\le u log ( 1 + u ) ≤ u .
Lemma E.2 (Approximate improvement). Fix x ∈ X x \in\mathcal{X} x ∈ X . For every 0 ≤ t < T 0 \le t < T 0 ≤ t < T ,
J x ( π t ) ≤ b ~ t ( x ) ≤ J x ( π t + 1 ) + η e η S 2 MSE t ( x ) . (81) J_x(\pi_t) \le\widetilde{b}_t(x) \le J_x(\pi_{t+1}) + \frac{\eta e^{\eta S}}{2}\operatorname{MSE}_t(x). \tag*{(81)} J x ( π t ) ≤ b t ( x ) ≤ J x ( π t + 1 ) + 2 η e η S MSE t ( x ) . (81) More generally, for every $0 \le s < t \le T$,
J x ( π s ) ≤ b ~ s ( x ) ≤ J x ( π t ) + η e η S 2 ∑ u = s t − 1 MSE u ( x ) . (82) J_x(\pi_s) \le\widetilde{b}_s(x) \le J_x(\pi_t) + \frac{\eta e^{\eta S}}{2}\sum_{u=s}^{t-1}\operatorname{MSE}_u(x). \tag*{(82)} J x ( π s ) ≤ b s ( x ) ≤ J x ( π t ) + 2 η e η S u = s ∑ t − 1 MSE u ( x ) . (82)
Proof of Lemma E.2. Fix any 0 ≤ t < T 0 \le t < T 0 ≤ t < T . We first prove the lower bounds in both Eq. (81) and Eq. (82) . Jensen’s inequality and the definition of s t ( x , y ) s_t(x,y) s t ( x , y ) give
η b ~ t ( x ) ≥ η E y ∼ π t ( ⋅ ∣ x ) [ r ( x , y ) + s t ( x , y ) ] = η J x ( π t ) . (83) \eta\widetilde{b}_t(x) \ge\eta\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[r(x,y)+s_t(x,y)] = \eta J_x(\pi_t). \tag*{(83)} η b t ( x ) ≥ η E y ∼ π t ( ⋅ ∣ x ) [ r ( x , y ) + s t ( x , y )] = η J x ( π t ) . (83)
We next prove the upper bound in Eq. (81) . Recall that
π t + 1 ( y ∣ x ) = π t ( y ∣ x ) e η ( r ( x , y ) + s t ( x , y ) − b ~ t ( x ) ) . \pi_{t+1}(y\mid x)=\pi_t(y\mid x)e^{\eta(r(x,y)+s_t(x,y)-\widetilde{b}_t(x))}. π t + 1 ( y ∣ x ) = π t ( y ∣ x ) e η ( r ( x , y ) + s t ( x , y ) − b t ( x )) .
Taking logarithms and rearranging, we obtain
η b ~ t ( x ) = η r ( x , y ) + η s t ( x , y ) − log π t + 1 ( y ∣ x ) π t ( y ∣ x ) . \eta\widetilde{b}_t(x)=\eta r(x,y)+\eta s_t(x,y)-\log\frac{\pi_{t+1}(y\mid x)}{\pi_t(y\mid x)}. η b t ( x ) = η r ( x , y ) + η s t ( x , y ) − log π t ( y ∣ x ) π t + 1 ( y ∣ x ) .
Taking expectation under π t + 1 ( ⋅ ∣ x ) \pi_{t+1}(\cdot\mid x) π t + 1 ( ⋅ ∣ x ) gives
η b ~ t ( x ) = η J x ( π t + 1 ) + η E y ∼ π t + 1 ( ⋅ ∣ x ) [ s t ( x , y ) ] − D K L ( π t + 1 ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x ) ) = η J x ( π t + 1 ) + E y ∼ π t + 1 ( ⋅ ∣ x ) [ log ( e η s t ( x , y ) π t ( y ∣ x ) π t + 1 ( y ∣ x ) ) ] ≤ η J x ( π t + 1 ) + log E y ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ) ] ≤ η J x ( π t + 1 ) + η 2 e η S 2 MSE t ( x ) . (84) \begin{aligned}
\eta\widetilde{b}_t(x)
&=\eta J_x(\pi_{t+1})+\eta\mathbb{E}_{y\sim\pi_{t+1}(\cdot\mid x)}[s_t(x,y)]-D_{\mathrm{KL}}(\pi_{t+1}(\cdot\mid x)\mathbin{\|}\pi_t(\cdot\mid x))\\
&=\eta J_x(\pi_{t+1})+\mathbb{E}_{y\sim\pi_{t+1}(\cdot\mid x)}\left[\log\left(e^{\eta s_t(x,y)}\frac{\pi_t(y\mid x)}{\pi_{t+1}(y\mid x)}\right)\right]\\
&\le\eta J_x(\pi_{t+1})+\log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y)}\right]\\
&\le\eta J_x(\pi_{t+1})+\frac{\eta^2e^{\eta S}}{2}\operatorname{MSE}_t(x). \tag*{(84)}
\end{aligned} η b t ( x ) = η J x ( π t + 1 ) + η E y ∼ π t + 1 ( ⋅ ∣ x ) [ s t ( x , y )] − D KL ( π t + 1 ( ⋅ ∣ x ) ∥ π t ( ⋅ ∣ x )) = η J x ( π t + 1 ) + E y ∼ π t + 1 ( ⋅ ∣ x ) [ log ( e η s t ( x , y ) π t + 1 ( y ∣ x ) π t ( y ∣ x ) ) ] ≤ η J x ( π t + 1 ) + log E y ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ) ] ≤ η J x ( π t + 1 ) + 2 η 2 e η S MSE t ( x ) . (84)
The first inequality uses Jensen’s inequality, and the second uses Lemma E.1. Thus, combining Eq. (83) and Eq. (84) , we have shown that
J x ( π t ) ≤ b ~ t ( x ) ≤ J x ( π t + 1 ) + η e η S 2 MSE t ( x ) , (85) J_x(\pi_t)\le\widetilde{b}_t(x)\le J_x(\pi_{t+1})+\frac{\eta e^{\eta S}}{2}\operatorname{MSE}_t(x), \tag*{(85)} J x ( π t ) ≤ b t ( x ) ≤ J x ( π t + 1 ) + 2 η e η S MSE t ( x ) , (85)
which proves the claim in Eq. (81) . Finally, we prove the upper bound for general 0 ≤ s < t ≤ T 0\le s<t\le T 0 ≤ s < t ≤ T in Eq. (82) . We can write
b ~ s ( x ) − J x ( π t ) = b ~ s ( x ) − J x ( π s + 1 ) + ∑ u = s + 1 t − 1 ( J x ( π u ) − J x ( π u + 1 ) ) ≤ η e η S 2 MSE s ( x ) + η e η S 2 ∑ u = s + 1 t − 1 MSE u ( x ) = η e η S 2 ∑ u = s t − 1 MSE u ( x ) , \begin{aligned}
\widetilde{b}_s(x)-J_x(\pi_t)
&=\widetilde{b}_s(x)-J_x(\pi_{s+1})+\sum_{u=s+1}^{t-1}\left(J_x(\pi_u)-J_x(\pi_{u+1})\right)\\
&\le\frac{\eta e^{\eta S}}{2}\operatorname{MSE}_s(x)+\frac{\eta e^{\eta S}}{2}\sum_{u=s+1}^{t-1}\operatorname{MSE}_u(x)\\
&=\frac{\eta e^{\eta S}}{2}\sum_{u=s}^{t-1}\operatorname{MSE}_u(x),
\end{aligned} b s ( x ) − J x ( π t ) = b s ( x ) − J x ( π s + 1 ) + u = s + 1 ∑ t − 1 ( J x ( π u ) − J x ( π u + 1 ) ) ≤ 2 η e η S MSE s ( x ) + 2 η e η S u = s + 1 ∑ t − 1 MSE u ( x ) = 2 η e η S u = s ∑ t − 1 MSE u ( x ) ,
where we have used Eq. (85) for each s ≤ u ≤ t − 1 s\le u\le t-1 s ≤ u ≤ t − 1 in the second line. Rearranging proves the upper bound in Eq. (82) .
Lemma E.3 (Coverage preserved approximately). Fix x ∈ X x \in\mathcal{X} x ∈ X and let Y a ( x ) = { y ∈ Y : r ( x , y ) ≥ a } \mathcal{Y}_a(x) = \{y \in\mathcal{Y}: r(x,y) \ge a\} Y a ( x ) = { y ∈ Y : r ( x , y ) ≥ a } for a ∈ [ 0 , R max ] a \in[0, R_{\max}] a ∈ [ 0 , R m a x ] . Then, for every 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T ,
π t ( Y a ( x ) ∣ x ) ≥ e − η 2 e η S 4 ∑ s = 0 t − 1 MSE s ( x ) π b a s e ( Y a ( x ) ∣ x ) − η 2 ∑ s = 0 t − 1 MSE s ( x ) . \sqrt{\pi_t(\mathcal{Y}_a(x)\mid x)} \ge e^{-\frac{\eta^2 e^{\eta S}}{4}\sum_{s=0}^{t-1}\operatorname{MSE}_s(x)}\sqrt{\pi_{\mathrm{base}}(\mathcal{Y}_a(x)\mid x)}-\frac{\eta}{2}\sum_{s=0}^{t-1}\sqrt{\operatorname{MSE}_s(x)}. π t ( Y a ( x ) ∣ x ) ≥ e − 4 η 2 e η S ∑ s = 0 t − 1 MSE s ( x ) π base ( Y a ( x ) ∣ x ) − 2 η s = 0 ∑ t − 1 MSE s ( x ) .
Proof of Lemma E.3 . For 0 ≤ t < T 0 \le t < T 0 ≤ t < T , define the conditional distribution
q t ( y ∣ x ) : = π t ( y ∣ x ) e η s t ( x , y ) E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] q_t(y\mid x) := \frac{\pi_t(y\mid x)e^{\eta s_t(x,y)}}{\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[e^{\eta s_t(x,y')}]} q t ( y ∣ x ) := E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] π t ( y ∣ x ) e η s t ( x , y )
Note that π t + 1 ( y ∣ x ) = q t ( y ∣ x ) e η r ( x , y ) E y ′ ∼ q t ( ⋅ ∣ x ) [ e η r ( x , y ′ ) ] \pi_{t+1}(y\mid x) = \frac{q_t(y\mid x)e^{\eta r(x,y)}}{\mathbb{E}_{y'\sim q_t(\cdot\mid x)}[e^{\eta r(x,y')}] } π t + 1 ( y ∣ x ) = E y ′ ∼ q t ( ⋅ ∣ x ) [ e η r ( x , y ′ ) ] q t ( y ∣ x ) e η r ( x , y ) from Eq. (80) . Therefore, the same calculation as in Eq. (66) gives
π t + 1 ( Y a ( x ) ∣ x ) ≥ q t ( Y a ( x ) ∣ x ) . (86) \pi_{t+1}(\mathcal{Y}_a(x)\mid x) \ge q_t(\mathcal{Y}_a(x)\mid x). \tag*{(86)} π t + 1 ( Y a ( x ) ∣ x ) ≥ q t ( Y a ( x ) ∣ x ) . (86)
Next, since 1 ≤ e u − u ≤ e u + ∣ u ∣ 1 \le e^u-u \le e^u+|u| 1 ≤ e u − u ≤ e u + ∣ u ∣ ,
π t ( y ∣ x ) ≤ π t ( y ∣ x ) e η s t ( x , y ) / 2 + η 2 π t ( y ∣ x ) ∣ s t ( x , y ) ∣ = q t ( y ∣ x ) ( E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] ) 1 / 2 + η 2 π t ( y ∣ x ) ∣ s t ( x , y ) ∣ , \begin{aligned}
\sqrt{\pi_t(y\mid x)} \le\sqrt{\pi_t(y\mid x)e^{\eta s_t(x,y)/2}}+\frac{\eta}{2}\sqrt{\pi_t(y\mid x)}|s_t(x,y)| \\
&= \sqrt{q_t(y\mid x)}\left(\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y')}\right]\right)^{1/2}+\frac{\eta}{2}\sqrt{\pi_t(y\mid x)}|s_t(x,y)|,
\end{aligned} π t ( y ∣ x ) ≤ π t ( y ∣ x ) e η s t ( x , y ) /2 + 2 η π t ( y ∣ x ) ∣ s t ( x , y ) ∣ = q t ( y ∣ x ) ( E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] ) 1/2 + 2 η π t ( y ∣ x ) ∣ s t ( x , y ) ∣ ,
where the second line uses the definition of q t q_t q t .
Then applying the above inequality to each y ∈ Y a ( x ) y \in\mathcal{Y}_a(x) y ∈ Y a ( x ) ,
π t ( Y a ( x ) ∣ x ) = ∑ y ∈ Y a ( x ) π t ( y ∣ x ) ≤ q t ( Y a ( x ) ∣ x ) E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] + η 2 4 ∑ y ∈ Y a ( x ) π t ( y ∣ x ) s t ( x , y ) 2 + η ( E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] ) 1 / 2 ∑ y ∈ Y a ( x ) q t ( y ∣ x ) π t ( y ∣ x ) ∣ s t ( x , y ) ∣ ≤ q t ( Y a ( x ) ∣ x ) E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] + η 2 4 ∑ y ∈ Y a ( x ) π t ( y ∣ x ) s t ( x , y ) 2 + η q t ( Y a ( x ) ∣ x ) ( E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] ) 1 / 2 ( ∑ y ∈ Y a ( x ) π t ( y ∣ x ) s t ( x , y ) 2 ) 1 / 2 ≤ q t ( Y a ( x ) ∣ x ) E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] + η 2 4 MSE t ( x ) + η q t ( Y a ( x ) ∣ x ) ( E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] ) 1 / 2 MSE t ( x ) = ( q t ( Y a ( x ) ∣ x ) ( E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] ) 1 / 2 + η 2 MSE t ( x ) ) 2 . \begin{aligned}
\pi_t(\mathcal{Y}_a(x)\mid x)
&=\sum_{y\in\mathcal{Y}_a(x)}\pi_t(y\mid x)\\
&\le q_t(\mathcal{Y}_a(x)\mid x)\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y')}\right]
+\frac{\eta^2}{4}\sum_{y\in\mathcal{Y}_a(x)}\pi_t(y\mid x)s_t(x,y)^2\\
&\quad+\eta\left(\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y')}\right]\right)^{1/2}
\sum_{y\in\mathcal{Y}_a(x)}\sqrt{q_t(y\mid x)\pi_t(y\mid x)}|s_t(x,y)|\\
&\le q_t(\mathcal{Y}_a(x)\mid x)\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y')}\right]
+\frac{\eta^2}{4}\sum_{y\in\mathcal{Y}_a(x)}\pi_t(y\mid x)s_t(x,y)^2\\
&\quad+\eta\sqrt{q_t(\mathcal{Y}_a(x)\mid x)}
\left(\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y')}\right]\right)^{1/2}
\left(\sum_{y\in\mathcal{Y}_a(x)}\pi_t(y\mid x)s_t(x,y)^2\right)^{1/2}\\
&\le q_t(\mathcal{Y}_a(x)\mid x)\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y')}\right]
+\frac{\eta^2}{4}\operatorname{MSE}_t(x)\\
&\quad+\eta\sqrt{q_t(\mathcal{Y}_a(x)\mid x)}
\left(\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y')}\right]\right)^{1/2}\sqrt{\operatorname{MSE}_t(x)}\\
&=\left(\sqrt{q_t(\mathcal{Y}_a(x)\mid x)}
\left(\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y')}\right]\right)^{1/2}
+\frac{\eta}{2}\sqrt{\operatorname{MSE}_t(x)}\right)^2.
\end{aligned} π t ( Y a ( x ) ∣ x ) = y ∈ Y a ( x ) ∑ π t ( y ∣ x ) ≤ q t ( Y a ( x ) ∣ x ) E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] + 4 η 2 y ∈ Y a ( x ) ∑ π t ( y ∣ x ) s t ( x , y ) 2 + η ( E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] ) 1/2 y ∈ Y a ( x ) ∑ q t ( y ∣ x ) π t ( y ∣ x ) ∣ s t ( x , y ) ∣ ≤ q t ( Y a ( x ) ∣ x ) E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] + 4 η 2 y ∈ Y a ( x ) ∑ π t ( y ∣ x ) s t ( x , y ) 2 + η q t ( Y a ( x ) ∣ x ) ( E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] ) 1/2 y ∈ Y a ( x ) ∑ π t ( y ∣ x ) s t ( x , y ) 2 1/2 ≤ q t ( Y a ( x ) ∣ x ) E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] + 4 η 2 MSE t ( x ) + η q t ( Y a ( x ) ∣ x ) ( E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] ) 1/2 MSE t ( x ) = ( q t ( Y a ( x ) ∣ x ) ( E y ′ ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ′ ) ] ) 1/2 + 2 η MSE t ( x ) ) 2 .
Therefore,
π t ( Y a ( x ) ∣ x ) ≤ q t ( Y a ( x ) ∣ x ) ( E y ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ) ] ) 1 / 2 + η 2 MSE t ( x ) . \sqrt{\pi_t(\mathcal{Y}_a(x)\mid x)}
\le\sqrt{q_t(\mathcal{Y}_a(x)\mid x)}
\left(\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y)}\right]\right)^{1/2}
+\frac{\eta}{2}\sqrt{\operatorname{MSE}_t(x)}. π t ( Y a ( x ) ∣ x ) ≤ q t ( Y a ( x ) ∣ x ) ( E y ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ) ] ) 1/2 + 2 η MSE t ( x ) .
For the RHS, Eq. (86) gives q t ( Y a ( x ) ∣ x ) ≤ π t + 1 ( Y a ( x ) ∣ x ) q_t(\mathcal{Y}_a(x)\mid x) \le\pi_{t+1}(\mathcal{Y}_a(x)\mid x) q t ( Y a ( x ) ∣ x ) ≤ π t + 1 ( Y a ( x ) ∣ x ) , and Lemma E.1 gives
log E y ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ) ] ≤ η 2 e η S 2 MSE t ( x ) . \log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta s_t(x,y)}\right]\le\frac{\eta^2e^{\eta S}}{2}\operatorname{MSE}_t(x). log E y ∼ π t ( ⋅ ∣ x ) [ e η s t ( x , y ) ] ≤ 2 η 2 e η S MSE t ( x ) .
Therefore,
π t ( Y a ( x ) ∣ x ) ≤ e η 2 e η S 4 MSE t ( x ) π t + 1 ( Y a ( x ) ∣ x ) + η 2 MSE t ( x ) . (87) \sqrt{\pi_t(\mathcal{Y}_a(x)\mid x)} \le e^{\frac{\eta^2 e^{\eta S}}{4}\operatorname{MSE}_t(x)}\sqrt{\pi_{t+1}(\mathcal{Y}_a(x)\mid x)}+\frac{\eta}{2}\sqrt{\operatorname{MSE}_t(x)}. \tag*{(87)} π t ( Y a ( x ) ∣ x ) ≤ e 4 η 2 e η S MSE t ( x ) π t + 1 ( Y a ( x ) ∣ x ) + 2 η MSE t ( x ) . (87)
For 1 ≤ t ≤ T 1\le t\le T 1 ≤ t ≤ T , applying Eq. (87) at iteration t − 1 t-1 t − 1 and unrolling for 0 ≤ s < t 0\le s<t 0 ≤ s < t gives
π t ( Y a ( x ) ∣ x ) ≥ e − η 2 e η S 4 MSE t − 1 ( x ) π t − 1 ( Y a ( x ) ∣ x ) − η 2 e − η 2 e η S 4 MSE t − 1 ( x ) MSE t − 1 ( x ) ≥ e − η 2 e η S 4 ∑ s = 0 t − 1 MSE s ( x ) π b a s e ( Y a ( x ) ∣ x ) − η 2 ∑ s = 0 t − 1 e − η 2 e η S 4 ∑ u = s t − 1 MSE u ( x ) MSE s ( x ) ≥ e − η 2 e η S 4 ∑ s = 0 t − 1 MSE s ( x ) π b a s e ( Y a ( x ) ∣ x ) − η 2 ∑ s = 0 t − 1 MSE s ( x ) . \begin{aligned}
\sqrt{\pi_t(\mathcal{Y}_a(x)\mid x)}
&\ge e^{-\frac{\eta^2 e^{\eta S}}{4}\operatorname{MSE}_{t-1}(x)}\sqrt{\pi_{t-1}(\mathcal{Y}_a(x)\mid x)}
-\frac{\eta}{2}e^{-\frac{\eta^2 e^{\eta S}}{4}\operatorname{MSE}_{t-1}(x)}\sqrt{\operatorname{MSE}_{t-1}(x)}\\
&\ge e^{-\frac{\eta^2 e^{\eta S}}{4}\sum_{s=0}^{t-1}\operatorname{MSE}_s(x)}\sqrt{\pi_{\mathrm{base}}(\mathcal{Y}_a(x)\mid x)}
-\frac{\eta}{2}\sum_{s=0}^{t-1}e^{-\frac{\eta^2 e^{\eta S}}{4}\sum_{u=s}^{t-1}\operatorname{MSE}_u(x)}\sqrt{\operatorname{MSE}_s(x)}\\
&\ge e^{-\frac{\eta^2 e^{\eta S}}{4}\sum_{s=0}^{t-1}\operatorname{MSE}_s(x)}\sqrt{\pi_{\mathrm{base}}(\mathcal{Y}_a(x)\mid x)}
-\frac{\eta}{2}\sum_{s=0}^{t-1}\sqrt{\operatorname{MSE}_s(x)}.
\end{aligned} π t ( Y a ( x ) ∣ x ) ≥ e − 4 η 2 e η S MSE t − 1 ( x ) π t − 1 ( Y a ( x ) ∣ x ) − 2 η e − 4 η 2 e η S MSE t − 1 ( x ) MSE t − 1 ( x ) ≥ e − 4 η 2 e η S ∑ s = 0 t − 1 MSE s ( x ) π base ( Y a ( x ) ∣ x ) − 2 η s = 0 ∑ t − 1 e − 4 η 2 e η S ∑ u = s t − 1 MSE u ( x ) MSE s ( x ) ≥ e − 4 η 2 e η S ∑ s = 0 t − 1 MSE s ( x ) π base ( Y a ( x ) ∣ x ) − 2 η s = 0 ∑ t − 1 MSE s ( x ) .
The case t = 0 t=0 t = 0 holds with equality.
Proof of 4.1
We first state Theorem E.1, which extends the exact policy mirror descent analysis in Theorem B.3 to approximate updates using mean-squared error. The proof of Theorem 4.1 directly follows; following that, we prove Theorem E.1.
Theorem E.1 (Global convergence under APMD). The final policy π T \pi_T π T satisfies
J ⋆ − J ( π T ) ≤ E x ∼ μ [ log ( 1 / p 0 ( x ) ) ] η T + η e η S 2 ∑ t = 0 T − 1 MSE t + C b a s e ∑ t = 0 T − 1 MSE t t + 1 . (88) J^\star-J(\pi_T)\le\frac{\mathbb{E}_{x\sim\mu}[\log(1/p_0(x))]}{\eta T}+\frac{\eta e^{\eta S}}{2}\sum_{t=0}^{T-1}\operatorname{MSE}_t+\sqrt{C_{\mathrm{base}}\sum_{t=0}^{T-1}\frac{\operatorname{MSE}_t}{t+1}}. \tag*{(88)} J ⋆ − J ( π T ) ≤ η T E x ∼ μ [ log ( 1/ p 0 ( x ))] + 2 η e η S t = 0 ∑ T − 1 MSE t + C base t = 0 ∑ T − 1 t + 1 MSE t . (88) Moreover, for any c ∈ ( 0 , 1 ) c\in(0,1) c ∈ ( 0 , 1 ) , with probability at least
1 − η 2 e η S 4 ( 1 − c ) ∑ t = 0 T − 1 MSE t − η C b a s e 2 ( 1 − c ) ∑ t = 0 T − 1 MSE t 1-\frac{\eta^2e^{\eta S}}{4(1-\sqrt{c})}\sum_{t=0}^{T-1}\operatorname{MSE}_t-\frac{\eta\sqrt{C_{\mathrm{base}}}}{2(1-\sqrt{c})}\sum_{t=0}^{T-1}\sqrt{\operatorname{MSE}_t} 1 − 4 ( 1 − c ) η 2 e η S t = 0 ∑ T − 1 MSE t − 2 ( 1 − c ) η C base t = 0 ∑ T − 1 MSE t over x ∼ μ x\sim\mu x ∼ μ ,
π t ( Y x ⋆ ∣ x ) ≥ c π b a s e ( Y x ⋆ ∣ x ) , ∀ 0 ≤ t ≤ T . (89) \pi_t(\mathcal{Y}_x^\star\mid x)\ge c\pi_{\mathrm{base}}(\mathcal{Y}_x^\star\mid x),\qquad\forall0\le t\le T. \tag*{(89)} π t ( Y x ⋆ ∣ x ) ≥ c π base ( Y x ⋆ ∣ x ) , ∀0 ≤ t ≤ T . (89)
Proof of Theorem 4.1. We first prove the coverage claim by applying Theorem E.1 with c = 1 / 2 c=1/2 c = 1/2 and the prescribed MSE tolerance in Eq. (20) ,
MSE t ≤ 1 4 η 2 T 2 ( C b a s e log ( e T ) + e η S ) , 0 ≤ t < T . (90) \operatorname{MSE}_t\le\frac{1}{4\eta^2T^2(C_{\mathrm{base}}\log(eT)+e^{\eta S})},\qquad0\le t<T. \tag*{(90)} MSE t ≤ 4 η 2 T 2 ( C base log ( e T ) + e η S ) 1 , 0 ≤ t < T . (90)
Substituting into the theorem gives
P x ∼ μ ( ∃ 0 ≤ t ≤ T : π t ( Y x ⋆ ∣ x ) < 1 2 π b a s e ( Y x ⋆ ∣ x ) ) ≤ ( 2 + 2 ) ( e η S 16 T ( C b a s e log ( e T ) + e η S ) + 1 4 C b a s e C b a s e log ( e T ) + e η S ) ≤ 2 + 2 8 log ( e T ) ( e η S C b a s e log ( e T ) + e η S + 1 + C b a s e log ( e T ) C b a s e log ( e T ) + e η S ) = 2 + 2 4 log ( e T ) ≤ 7 8 log ( e T ) . \begin{aligned}
\mathbb{P}_{x\sim\mu}\left(\exists\,0\le t\le T:\pi_t(\mathcal{Y}_x^\star\mid x)<\frac{1}{2}\pi_{\mathrm{base}}(\mathcal{Y}_x^\star\mid x)\right)
&\le(2+\sqrt{2})\left(\frac{e^{\eta S}}{16T(C_{\mathrm{base}}\log(eT)+e^{\eta S})}+\frac{1}{4}\sqrt{\frac{C_{\mathrm{base}}}{C_{\mathrm{base}}\log(eT)+e^{\eta S}}}\right)\\
&\le\frac{2+\sqrt{2}}{8\sqrt{\log(eT)}}\left(\frac{e^{\eta S}}{C_{\mathrm{base}}\log(eT)+e^{\eta S}}+1+\frac{C_{\mathrm{base}}\log(eT)}{C_{\mathrm{base}}\log(eT)+e^{\eta S}}\right)\\
&=\frac{2+\sqrt{2}}{4\sqrt{\log(eT)}}\le\frac{7}{8\sqrt{\log(eT)}}.
\end{aligned} P x ∼ μ ( ∃ 0 ≤ t ≤ T : π t ( Y x ⋆ ∣ x ) < 2 1 π base ( Y x ⋆ ∣ x ) ) ≤ ( 2 + 2 ) ( 16 T ( C base log ( e T ) + e η S ) e η S + 4 1 C base log ( e T ) + e η S C base ) ≤ 8 log ( e T ) 2 + 2 ( C base log ( e T ) + e η S e η S + 1 + C base log ( e T ) + e η S C base log ( e T ) ) = 4 log ( e T ) 2 + 2 ≤ 8 log ( e T ) 7 .
The second inequality uses u ≤ ( 1 + u ) / 2 \sqrt{u} \le(1+u)/2 u ≤ ( 1 + u ) /2 for u ≥ 0 u \ge0 u ≥ 0 . This proves the coverage claim.
To prove convergence, we again substitute Eq. (90) into Theorem E.1. Since E x ∼ μ [ log ( 1 / p 0 ( x ) ) ] ≤ log C b a s e \mathbb{E}_{x\sim\mu}[\log(1/p_0(x))] \le\log\mathcal{C}_{\mathrm{base}} E x ∼ μ [ log ( 1/ p 0 ( x ))] ≤ log C base by Jensen’s inequality, we obtain
J ⋆ − J ( π T ) ≤ log C b a s e η T + e η S 8 η T ( C b a s e log ( e T ) + e η S ) + 1 2 η T C b a s e ∑ t = 1 T 1 / t C b a s e log ( e T ) + e η S ≤ log C b a s e η T + e η S 8 η T ( C b a s e log ( e T ) + e η S ) + 1 2 η T C b a s e log ( e T ) C b a s e log ( e T ) + e η S ≤ log C b a s e + 1 / 8 + 1 / 2 η T ≤ log ( 2 C b a s e ) η T . \begin{aligned}
J^\star-J(\pi_T) \le\frac{\log\mathcal{C}_{\mathrm{base}}}{\eta T}+\frac{e^{\eta S}}{8\eta T(\mathcal{C}_{\mathrm{base}}\log(eT)+e^{\eta S})}+\frac{1}{2\eta T}\sqrt{\frac{\mathcal{C}_{\mathrm{base}}\sum_{t=1}^{T}1/t}{\mathcal{C}_{\mathrm{base}}\log(eT)+e^{\eta S}}} \\
&\le\frac{\log\mathcal{C}_{\mathrm{base}}}{\eta T}+\frac{e^{\eta S}}{8\eta T(\mathcal{C}_{\mathrm{base}}\log(eT)+e^{\eta S})}+\frac{1}{2\eta T}\sqrt{\frac{\mathcal{C}_{\mathrm{base}}\log(eT)}{\mathcal{C}_{\mathrm{base}}\log(eT)+e^{\eta S}}} \\
&\le\frac{\log\mathcal{C}_{\mathrm{base}}+1/8+1/2}{\eta T}\le\frac{\log(2\mathcal{C}_{\mathrm{base}})}{\eta T}.
\end{aligned} J ⋆ − J ( π T ) ≤ η T log C base + 8 η T ( C base log ( e T ) + e η S ) e η S + 2 η T 1 C base log ( e T ) + e η S C base ∑ t = 1 T 1/ t ≤ η T log C base + 8 η T ( C base log ( e T ) + e η S ) e η S + 2 η T 1 C base log ( e T ) + e η S C base log ( e T ) ≤ η T log C base + 1/8 + 1/2 ≤ η T log ( 2 C base ) .
The second inequality uses ∑ t = 1 T 1 / t ≤ log ( e T ) \sum_{t=1}^{T}1/t\le\log(eT) ∑ t = 1 T 1/ t ≤ log ( e T ) . This proves the convergence claim.
Next, we prove Theorem E.1, which extends the exact mirror descent convergence guarantee in Theorem B.3 to handle approximation error measured in M S E t \mathrm{MSE}_t MSE t . We use Lemma E.4 , proved below, for the suboptimality bound and Lemma E.3 for the coverage guarantee.
Proof of Theorem E.1. We first prove the convergence bound by applying Lemma E.4 with t = T t=T t = T and averaging over prompts. Since J ⋆ − J ( π T ) = E x ∼ μ [ r x ⋆ − J x ( π T ) ] J^\star-J(\pi_T)=\mathbb{E}_{x\sim\mu}[r_x^\star-J_x(\pi_T)] J ⋆ − J ( π T ) = E x ∼ μ [ r x ⋆ − J x ( π T )] , this gives
J ⋆ − J ( π T ) ≤ E x ∼ μ [ log ( 1 / p 0 ( x ) ) ] η T + η e η S 2 ∑ t = 0 T − 1 M S E t + E x ∼ μ [ 1 p 0 ( x ) ∑ t = 0 T − 1 M S E t ( x ) t + 1 ] ≤ E x ∼ μ [ log ( 1 / p 0 ( x ) ) ] η T + η e η S 2 ∑ t = 0 T − 1 M S E t + C b a s e ∑ t = 0 T − 1 M S E t t + 1 , \begin{aligned}
J^\star-J(\pi_T)\le\frac{\mathbb{E}_{x\sim\mu}[\log(1/p_0(x))]}{\eta T}+\frac{\eta e^{\eta S}}{2}\sum_{t=0}^{T-1}\mathrm{MSE}_t+\mathbb{E}_{x\sim\mu}\left[\sqrt{\frac{1}{p_0(x)}\sum_{t=0}^{T-1}\frac{\mathrm{MSE}_t(x)}{t+1}}\right] \\
&\le\frac{\mathbb{E}_{x\sim\mu}[\log(1/p_0(x))]}{\eta T}+\frac{\eta e^{\eta S}}{2}\sum_{t=0}^{T-1}\mathrm{MSE}_t+\sqrt{\mathcal{C}_{\mathrm{base}}\sum_{t=0}^{T-1}\frac{\mathrm{MSE}_t}{t+1}},
\end{aligned} J ⋆ − J ( π T ) ≤ η T E x ∼ μ [ log ( 1/ p 0 ( x ))] + 2 η e η S t = 0 ∑ T − 1 MSE t + E x ∼ μ p 0 ( x ) 1 t = 0 ∑ T − 1 t + 1 MSE t ( x ) ≤ η T E x ∼ μ [ log ( 1/ p 0 ( x ))] + 2 η e η S t = 0 ∑ T − 1 MSE t + C base t = 0 ∑ T − 1 t + 1 MSE t ,
using Cauchy–Schwarz in the second line, which gives Eq. (88) .
Next, to prove the coverage guarantee, fix c ∈ ( 0 , 1 ) c\in(0,1) c ∈ ( 0 , 1 ) and apply Lemma E.3 with a = r x ⋆ a=r_x^\star a = r x ⋆ . For every 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T , we obtain
p t ( x ) p 0 ( x ) ≥ e − η 2 e η S 4 ∑ s = 0 t − 1 M S E s ( x ) − η 2 ∑ s = 0 t − 1 M S E s ( x ) p 0 ( x ) \sqrt{\frac{p_t(x)}{p_0(x)}}\ge e^{-\frac{\eta^2e^{\eta S}}{4}\sum_{s=0}^{t-1}\mathrm{MSE}_s(x)-\frac{\eta}{2}\sum_{s=0}^{t-1}\sqrt{\frac{\mathrm{MSE}_s(x)}{p_0(x)}}} p 0 ( x ) p t ( x ) ≥ e − 4 η 2 e η S ∑ s = 0 t − 1 MSE s ( x ) − 2 η ∑ s = 0 t − 1 p 0 ( x ) MSE s ( x )
≥ 1 − η 2 e η S 4 ∑ s = 0 T − 1 M S E s ( x ) − η 2 ∑ s = 0 T − 1 M S E s ( x ) p 0 ( x ) . \ge1-\frac{\eta^2e^{\eta S}}{4}\sum_{s=0}^{T-1}\mathrm{MSE}_s(x)-\frac{\eta}{2}\sum_{s=0}^{T-1}\sqrt{\frac{\mathrm{MSE}_s(x)}{p_0(x)}}. ≥ 1 − 4 η 2 e η S s = 0 ∑ T − 1 MSE s ( x ) − 2 η s = 0 ∑ T − 1 p 0 ( x ) MSE s ( x ) .
The second inequality uses e − u ≥ 1 − u e^{-u}\ge1-u e − u ≥ 1 − u and t ≤ T t\le T t ≤ T . Thus, every prompt satisfying
η 2 e η S 4 ∑ s = 0 T − 1 M S E s ( x ) + η 2 ∑ s = 0 T − 1 M S E s ( x ) p 0 ( x ) ≤ 1 − c (91) \frac{\eta^2e^{\eta S}}{4}\sum_{s=0}^{T-1}\mathrm{MSE}_s(x)+\frac{\eta}{2}\sum_{s=0}^{T-1}\sqrt{\frac{\mathrm{MSE}_s(x)}{p_0(x)}}\le1-\sqrt{c} \tag*{(91)} 4 η 2 e η S s = 0 ∑ T − 1 MSE s ( x ) + 2 η s = 0 ∑ T − 1 p 0 ( x ) MSE s ( x ) ≤ 1 − c (91)
has
p t ( x ) p 0 ( x ) ≥ c , \sqrt{\frac{p_t(x)}{p_0(x)}}\ge\sqrt{c}, p 0 ( x ) p t ( x ) ≥ c ,
and hence π t ( y x ⋆ ∣ x ) ≥ c π b a s e ( y x ⋆ ∣ x ) \pi_t(y_x^\star\mid x)\ge c\pi_{\mathrm{base}}(y_x^\star\mid x) π t ( y x ⋆ ∣ x ) ≥ c π base ( y x ⋆ ∣ x ) , for all 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T . It remains to bound the probability that Eq. (91) does not hold. We have
P x ∼ μ ( η 2 e η S 4 ∑ s = 0 T − 1 MSE s ( x ) + η 2 ∑ s = 0 T − 1 MSE s ( x ) p 0 ( x ) > 1 − c ) ≤ 1 1 − c ( η 2 e η S 4 ∑ s = 0 T − 1 MSE s + η 2 ∑ s = 0 T − 1 E x ∼ μ [ MSE s ( x ) p 0 ( x ) ] ) ≤ 1 1 − c ( η 2 e η S 4 ∑ s = 0 T − 1 MSE s + η C b a s e 2 ∑ s = 0 T − 1 MSE s ) . \begin{aligned}
\mathbb{P}_{x\sim\mu}\left(\frac{\eta^2 e^{\eta S}}{4}\sum_{s=0}^{T-1}\operatorname{MSE}_s(x)+\frac{\eta}{2}\sum_{s=0}^{T-1}\sqrt{\frac{\operatorname{MSE}_s(x)}{p_0(x)}}>1-\sqrt{c}\right) \\
&\le\frac{1}{1-\sqrt{c}}\left(\frac{\eta^2 e^{\eta S}}{4}\sum_{s=0}^{T-1}\operatorname{MSE}_s+\frac{\eta}{2}\sum_{s=0}^{T-1}\mathbb{E}_{x\sim\mu}\left[\sqrt{\frac{\operatorname{MSE}_s(x)}{p_0(x)}}\right]\right) \\
&\le\frac{1}{1-\sqrt{c}}\left(\frac{\eta^2 e^{\eta S}}{4}\sum_{s=0}^{T-1}\operatorname{MSE}_s+\frac{\eta\sqrt{C_{\mathrm{base}}}}{2}\sum_{s=0}^{T-1}\sqrt{\operatorname{MSE}_s}\right).
\end{aligned} P x ∼ μ ( 4 η 2 e η S s = 0 ∑ T − 1 MSE s ( x ) + 2 η s = 0 ∑ T − 1 p 0 ( x ) MSE s ( x ) > 1 − c ) ≤ 1 − c 1 ( 4 η 2 e η S s = 0 ∑ T − 1 MSE s + 2 η s = 0 ∑ T − 1 E x ∼ μ [ p 0 ( x ) MSE s ( x ) ] ) ≤ 1 − c 1 ( 4 η 2 e η S s = 0 ∑ T − 1 MSE s + 2 η C base s = 0 ∑ T − 1 MSE s ) .
The first inequality uses Markov’s inequality, and the second uses Cauchy–Schwarz and the definition of C b a s e C_{\mathrm{base}} C base . This proves the coverage guarantee in Eq. (89) .
The remaining per-prompt bound uses the residual form of the update on Y x ∗ \mathcal{Y}_x^* Y x ∗ .
Lemma E.4. For every prompt x x x and every 1 ≤ t ≤ T 1\le t\le T 1 ≤ t ≤ T ,
r x ∗ − J x ( π t ) ≤ log ( 1 / p 0 ( x ) ) η t + η e η S 2 ∑ s = 0 t − 1 MSE s ( x ) + 1 p 0 ( x ) ∑ s = 0 t − 1 MSE s ( x ) s + 1 . r_x^*-J_x(\pi_t)\le\frac{\log(1/p_0(x))}{\eta t}+\frac{\eta e^{\eta S}}{2}\sum_{s=0}^{t-1}\operatorname{MSE}_s(x)+\sqrt{\frac{1}{p_0(x)}\sum_{s=0}^{t-1}\frac{\operatorname{MSE}_s(x)}{s+1}}. r x ∗ − J x ( π t ) ≤ η t log ( 1/ p 0 ( x )) + 2 η e η S s = 0 ∑ t − 1 MSE s ( x ) + p 0 ( x ) 1 s = 0 ∑ t − 1 s + 1 MSE s ( x ) .
Proof of Lemma E.4. Fix x ∈ X x\in\mathcal{X} x ∈ X and 1 ≤ t ≤ T 1\le t\le T 1 ≤ t ≤ T . We bound the excess over the first two terms in the claimed inequality. Define
z t ( x ) : = η ( r x ∗ − J x ( π t ) ) − η 2 e η S 2 ∑ s = 0 t − 1 MSE s ( x ) − log ( 1 / p 0 ( x ) ) t . z_t(x):=\eta(r_x^*-J_x(\pi_t))-\frac{\eta^2e^{\eta S}}{2}\sum_{s=0}^{t-1}\operatorname{MSE}_s(x)-\frac{\log(1/p_0(x))}{t}. z t ( x ) := η ( r x ∗ − J x ( π t )) − 2 η 2 e η S s = 0 ∑ t − 1 MSE s ( x ) − t log ( 1/ p 0 ( x )) .
The desired bound is equivalent to
z t ( x ) ≤ η 1 p 0 ( x ) ∑ s = 0 t − 1 MSE s ( x ) s + 1 , (92) z_t(x)\le\eta\sqrt{\frac{1}{p_0(x)}\sum_{s=0}^{t-1}\frac{\operatorname{MSE}_s(x)}{s+1}}, \tag*{(92)} z t ( x ) ≤ η p 0 ( x ) 1 s = 0 ∑ t − 1 s + 1 MSE s ( x ) , (92)
and it suffices to prove that Eq. (92) holds. We may assume z t ( x ) > 0 z_t(x)>0 z t ( x ) > 0 , since otherwise it holds immediately.
First, observe that by Lemma E.2, for every 0 ≤ s < t 0\le s<t 0 ≤ s < t ,
η ( r x ∗ − b ~ s ( x ) ) ≥ η ( r x ∗ − J x ( π t ) ) − η 2 e η S 2 ∑ u = s t − 1 MSE u ( x ) ≥ z t ( x ) + log ( 1 / p 0 ( x ) ) t . (93) \eta(r_x^*-\widetilde b_s(x))\ge\eta(r_x^*-J_x(\pi_t))-\frac{\eta^2e^{\eta S}}{2}\sum_{u=s}^{t-1}\operatorname{MSE}_u(x)\ge z_t(x)+\frac{\log(1/p_0(x))}{t}. \tag*{(93)} η ( r x ∗ − b s ( x )) ≥ η ( r x ∗ − J x ( π t )) − 2 η 2 e η S u = s ∑ t − 1 MSE u ( x ) ≥ z t ( x ) + t log ( 1/ p 0 ( x )) . (93)
Using Eq. (93) , we lower bound p s + 1 ( x ) \sqrt{p_{s+1}(x)} p s + 1 ( x ) in terms of z t ( x ) z_t(x) z t ( x ) . Since r ( x , y ) = r x ∗ r(x,y)=r_x^* r ( x , y ) = r x ∗ on Y x ∗ \mathcal{Y}_x^* Y x ∗ , the update for π s + 1 \pi_{s+1} π s + 1 in Eq. (80) gives
p s + 1 ( x ) = p s ( x ) e η 2 ( r x ∗ − b ~ s ( x ) ) ( ∑ y ∈ Y x ∗ π s ( y ∣ x ) p s ( x ) e η s s ( x , y ) ) 1 / 2 ≥ p 0 ( x ) − 1 / ( 2 t ) p s ( x ) e z t ( x ) / 2 ( ∑ y ∈ Y x ∗ π s ( y ∣ x ) p s ( x ) e η s s ( x , y ) ) 1 / 2 , \begin{aligned}
\sqrt{p_{s+1}(x)}=\sqrt{p_s(x)}e^{\frac{\eta}{2}(r_x^*-\widetilde b_s(x))}\left(\sum_{y\in\mathcal{Y}_x^*}\frac{\pi_s(y\mid x)}{p_s(x)}e^{\eta s_s(x,y)}\right)^{1/2} \\
&\ge p_0(x)^{-1/(2t)}\sqrt{p_s(x)}e^{z_t(x)/2}\left(\sum_{y\in\mathcal{Y}_x^*}\frac{\pi_s(y\mid x)}{p_s(x)}e^{\eta s_s(x,y)}\right)^{1/2},
\end{aligned} p s + 1 ( x ) = p s ( x ) e 2 η ( r x ∗ − b s ( x )) y ∈ Y x ∗ ∑ p s ( x ) π s ( y ∣ x ) e η s s ( x , y ) 1/2 ≥ p 0 ( x ) − 1/ ( 2 t ) p s ( x ) e z t ( x ) /2 y ∈ Y x ∗ ∑ p s ( x ) π s ( y ∣ x ) e η s s ( x , y ) 1/2 ,
where in the inequality we use Eq. (93) . We next bound the contribution of s s ( x , y ) s_s(x,y) s s ( x , y ) by MSE s ( x ) \operatorname{MSE}_s(x) MSE s ( x ) . Using Jensen’s inequality on the sum in the parentheses,
p s + 1 ( x ) ≥ p 0 ( x ) − 1 / ( 2 t ) p s ( x ) exp ( z t ( x ) 2 + η 2 p s ( x ) ∑ y ∈ Y ⋆ π s ( y ∣ x ) s s ( x , y ) ) ≥ p 0 ( x ) − 1 / ( 2 t ) ( ( 1 + z t ( x ) 2 ) p s ( x ) + η 2 p s ( x ) ∑ y ∈ Y ⋆ π s ( y ∣ x ) s s ( x , y ) ) ≥ p 0 ( x ) − 1 / ( 2 t ) ( ( 1 + z t ( x ) 2 ) p s ( x ) − η 2 MSE s ( x ) ) , (94) \begin{aligned}
\sqrt{p_{s+1}(x)} &\ge p_0(x)^{-1/(2t)}\sqrt{p_s(x)}\exp\left(\frac{z_t(x)}{2}+\frac{\eta}{2p_s(x)}\sum_{y\in\mathcal{Y}_\star}\pi_s(y\mid x)s_s(x,y)\right)\\
&\ge p_0(x)^{-1/(2t)}\left(\left(1+\frac{z_t(x)}{2}\right)\sqrt{p_s(x)}+\frac{\eta}{2\sqrt{p_s(x)}}\sum_{y\in\mathcal{Y}_\star}\pi_s(y\mid x)s_s(x,y)\right)\\
&\ge p_0(x)^{-1/(2t)}\left(\left(1+\frac{z_t(x)}{2}\right)\sqrt{p_s(x)}-\frac{\eta}{2}\sqrt{\operatorname{MSE}_s(x)}\right), \tag*{(94)}
\end{aligned} p s + 1 ( x ) ≥ p 0 ( x ) − 1/ ( 2 t ) p s ( x ) exp 2 z t ( x ) + 2 p s ( x ) η y ∈ Y ⋆ ∑ π s ( y ∣ x ) s s ( x , y ) ≥ p 0 ( x ) − 1/ ( 2 t ) ( 1 + 2 z t ( x ) ) p s ( x ) + 2 p s ( x ) η y ∈ Y ⋆ ∑ π s ( y ∣ x ) s s ( x , y ) ≥ p 0 ( x ) − 1/ ( 2 t ) ( ( 1 + 2 z t ( x ) ) p s ( x ) − 2 η MSE s ( x ) ) , (94)
where the second inequality above uses e u ≥ 1 + u e^u\ge1+u e u ≥ 1 + u , and the third uses Cauchy–Schwarz,
∑ y ∈ Y ⋆ π s ( y ∣ x ) s s ( x , y ) ≥ − ∑ y ∈ Y ⋆ π s ( y ∣ x ) ∑ y ∈ Y ⋆ π s ( y ∣ x ) s s ( x , y ) 2 ≥ − p s ( x ) MSE s ( x ) . \begin{aligned}
\sum_{y\in\mathcal{Y}_\star}\pi_s(y\mid x)s_s(x,y)
&\ge-\sqrt{\sum_{y\in\mathcal{Y}_\star}\pi_s(y\mid x)}\sqrt{\sum_{y\in\mathcal{Y}_\star}\pi_s(y\mid x)s_s(x,y)^2}\\
&\ge-\sqrt{p_s(x)\operatorname{MSE}_s(x)}.
\end{aligned} y ∈ Y ⋆ ∑ π s ( y ∣ x ) s s ( x , y ) ≥ − y ∈ Y ⋆ ∑ π s ( y ∣ x ) y ∈ Y ⋆ ∑ π s ( y ∣ x ) s s ( x , y ) 2 ≥ − p s ( x ) MSE s ( x ) .
We now use Eq. (94) to prove Eq. (92) . Rearranging the bound on p s + 1 ( x ) \sqrt{p_{s+1}(x)} p s + 1 ( x ) gives
p s ( x ) ≤ p 0 ( x ) 1 / ( 2 t ) 1 + z t ( x ) / 2 p s + 1 ( x ) + η MSE s ( x ) 2 ( 1 + z t ( x ) / 2 ) . \sqrt{p_s(x)}\le\frac{p_0(x)^{1/(2t)}}{1+z_t(x)/2}\sqrt{p_{s+1}(x)}+\frac{\eta\sqrt{\operatorname{MSE}_s(x)}}{2(1+z_t(x)/2)}. p s ( x ) ≤ 1 + z t ( x ) /2 p 0 ( x ) 1/ ( 2 t ) p s + 1 ( x ) + 2 ( 1 + z t ( x ) /2 ) η MSE s ( x ) .
Unrolling this inequality through iterations t − 1 , … , 0 t-1,\ldots,0 t − 1 , … , 0 and dividing by p 0 ( x ) \sqrt{p_0(x)} p 0 ( x ) , we obtain
1 ≤ p t ( x ) ( 1 + z t ( x ) / 2 ) t + η 2 p 0 ( x ) ∑ s = 1 t p 0 ( x ) ( s − 1 ) / ( 2 t ) MSE s − 1 ( x ) ( 1 + z t ( x ) / 2 ) s . 1\le\frac{\sqrt{p_t(x)}}{(1+z_t(x)/2)^t}+\frac{\eta}{2\sqrt{p_0(x)}}\sum_{s=1}^{t}\frac{p_0(x)^{(s-1)/(2t)}\sqrt{\operatorname{MSE}_{s-1}(x)}}{(1+z_t(x)/2)^s}. 1 ≤ ( 1 + z t ( x ) /2 ) t p t ( x ) + 2 p 0 ( x ) η s = 1 ∑ t ( 1 + z t ( x ) /2 ) s p 0 ( x ) ( s − 1 ) / ( 2 t ) MSE s − 1 ( x ) .
Using p 0 ( x ) , p t ( x ) ≤ 1 p_0(x),p_t(x)\le1 p 0 ( x ) , p t ( x ) ≤ 1 and rearranging,
1 − ( 1 + z t ( x ) / 2 ) − t ≤ η 2 p 0 ( x ) ∑ s = 1 t MSE s − 1 ( x ) ( 1 + z t ( x ) / 2 ) s ≤ η 2 1 p 0 ( x ) ∑ s = 1 t MSE s − 1 ( x ) s ( ∑ s = 1 t s ( 1 + z t ( x ) / 2 ) 2 s ) , (95) \begin{aligned}
1-(1+z_t(x)/2)^{-t}\le\frac{\eta}{2\sqrt{p_0(x)}}\sum_{s=1}^{t}\frac{\sqrt{\operatorname{MSE}_{s-1}(x)}}{(1+z_t(x)/2)^s} \\
&\le\frac{\eta}{2}\sqrt{\frac{1}{p_0(x)}\sum_{s=1}^{t}\frac{\operatorname{MSE}_{s-1}(x)}{s}\left(\sum_{s=1}^{t}\frac{s}{(1+z_t(x)/2)^{2s}}\right)}, \tag*{(95)}
\end{aligned} 1 − ( 1 + z t ( x ) /2 ) − t ≤ 2 p 0 ( x ) η s = 1 ∑ t ( 1 + z t ( x ) /2 ) s MSE s − 1 ( x ) ≤ 2 η p 0 ( x ) 1 s = 1 ∑ t s MSE s − 1 ( x ) ( s = 1 ∑ t ( 1 + z t ( x ) /2 ) 2 s s ) , (95)
where the second inequality uses Cauchy–Schwarz. To bound the sum in the rightmost term of Eq. (95) ,
∑ s = 1 t s ( 1 + z t ( x ) / 2 ) 2 s ≤ ∑ s = 1 t ∑ j = 1 s 1 ( 1 + z t ( x ) / 2 ) s + j ≤ ( ∑ s = 1 t 1 ( 1 + z t ( x ) / 2 ) s ) 2 = 4 z t ( x ) 2 ( 1 − ( 1 + z t ( x ) / 2 ) − t ) 2 . \begin{aligned}
\sum_{s=1}^{t}\frac{s}{(1+z_t(x)/2)^{2s}}
\le\sum_{s=1}^{t}\sum_{j=1}^{s}\frac{1}{(1+z_t(x)/2)^{s+j}} \\
&\le\left(\sum_{s=1}^{t}\frac{1}{(1+z_t(x)/2)^s}\right)^2
=\frac{4}{z_t(x)^2}\left(1-(1+z_t(x)/2)^{-t}\right)^2.
\end{aligned} s = 1 ∑ t ( 1 + z t ( x ) /2 ) 2 s s ≤ s = 1 ∑ t j = 1 ∑ s ( 1 + z t ( x ) /2 ) s + j 1 ≤ ( s = 1 ∑ t ( 1 + z t ( x ) /2 ) s 1 ) 2 = z t ( x ) 2 4 ( 1 − ( 1 + z t ( x ) /2 ) − t ) 2 .
Substituting this bound into Eq. (95) gives
1 − ( 1 + z t ( x ) / 2 ) − t ≤ η z t ( x ) 1 p 0 ( x ) ∑ s = 1 t MSE s − 1 ( x ) s ( 1 − ( 1 + z t ( x ) / 2 ) − t ) . 1-(1+z_t(x)/2)^{-t}\le\frac{\eta}{z_t(x)}\sqrt{\frac{1}{p_0(x)}\sum_{s=1}^{t}\frac{\operatorname{MSE}_{s-1}(x)}{s}}\left(1-(1+z_t(x)/2)^{-t}\right). 1 − ( 1 + z t ( x ) /2 ) − t ≤ z t ( x ) η p 0 ( x ) 1 s = 1 ∑ t s MSE s − 1 ( x ) ( 1 − ( 1 + z t ( x ) /2 ) − t ) .
Since z t ( x ) > 0 z_t(x)>0 z t ( x ) > 0 , 1 − ( 1 + z t ( x ) / 2 ) − t > 0 1-(1+z_t(x)/2)^{-t}>0 1 − ( 1 + z t ( x ) /2 ) − t > 0 . Multiplying both sides by z t ( x ) / ( 1 − ( 1 + z t ( x ) / 2 ) − t ) z_t(x)/(1-(1+z_t(x)/2)^{-t}) z t ( x ) / ( 1 − ( 1 + z t ( x ) /2 ) − t ) gives
z t ( x ) ≤ η 1 p 0 ( x ) ∑ s = 1 t MSE s − 1 ( x ) s = η 1 p 0 ( x ) ∑ s = 0 t − 1 MSE s ( x ) s + 1 . z_t(x)\le\eta\sqrt{\frac{1}{p_0(x)}\sum_{s=1}^{t}\frac{\operatorname{MSE}_{s-1}(x)}{s}}
=\eta\sqrt{\frac{1}{p_0(x)}\sum_{s=0}^{t-1}\frac{\operatorname{MSE}_s(x)}{s+1}}. z t ( x ) ≤ η p 0 ( x ) 1 s = 1 ∑ t s MSE s − 1 ( x ) = η p 0 ( x ) 1 s = 0 ∑ t − 1 s + 1 MSE s ( x ) .
This proves Eq. (92) and the claimed lemma statement.
Additional Result: KL Divergence Criterion for General Comparators
In this section, we extend Theorem 3.1 to general comparator policies and performance targets.
(Appendix § 11.3 .§ 1 ) In Theorem F.1, we establish sufficient conditions in terms of T T T and the inner KL tolerance D K L ( π t + 1 ∥ π t ⋆ ) ≤ ε i n D_{\mathrm{KL}}(\pi_{t+1}\mathbin\Vert\pi^\star_{t}) \leq\varepsilon_{\mathrm{in}} D KL ( π t + 1 ∥ π t ⋆ ) ≤ ε in for J ( π ) − J ( π T ) ≤ ε J(\pi)-J(\pi_T)\leq\varepsilon J ( π ) − J ( π T ) ≤ ε . For a given π \pi π the tolerance ε i n \varepsilon_{\mathrm{in}} ε in scales with the coverage of the base model over π \pi π , and the guarantee holds for every comparator π \pi π . The conditions in Eq. (100) also guarantee J − J ( π T ) ≤ ε J-J(\pi_T)\leq\varepsilon J − J ( π T ) ≤ ε , where J ∈ [ 0 , J ⋆ ] J\in[0,J^\star] J ∈ [ 0 , J ⋆ ] is a target performance.
We omit analogous results for the mean-squared error criterion (Theorem 4.1), as well as end-to-end analyses for algorithms such as PPO-NoClip, which follow straightforwardly from the results here.
(Appendix § 11.3 .§ 2 ) Here we prove the results in Appendix § 11.3 .§ 1 . The proofs in this section build on the supporting lemmas in Appendix D.§ 1 .
Notation. Fix an η > 0 \eta>0 η > 0 and T ≥ 1 T\geq1 T ≥ 1 , as well as a sequence of policies { π t } t = 0 T \{\pi_t\}_{t=0}^{T} { π t } t = 0 T with π 0 = π b a s e \pi_0=\pi_{\mathrm{base}} π 0 = π base with π t > 0 \pi_t>0 π t > 0 for all t t t . We retain the setup and notation of Appendix D, including the per-prompt rewards J x ( π ) J_x(\pi) J x ( π ) and approximation errors K L t ( x ) \mathrm{KL}_t(x) KL t ( x ) and K L ( x ) \mathrm{KL}(x) KL ( x ) . The inner KL tolerance ε i n ≥ 0 \varepsilon_{\mathrm{in}}\geq0 ε in ≥ 0 is a free parameter in this section.
Extension of 3.1 to General Comparator Policies
In this section, we write π p o s t = π T \pi_{\mathrm{post}}=\pi_T π post = π T and consider performance targets of the form J ( π ) − J ( π T ) J(\pi)-J(\pi_T) J ( π ) − J ( π T ) , where π \pi π is a given comparator policy. Our convergence guarantees will scale with C b a s e π C_{\mathrm{base}}^\pi C base π , the response coverage of π b a s e \pi_{\mathrm{base}} π base over π \pi π , defined below.
Definition F.1 (Coverage coefficient). Fix a comparator policy π \pi π . The coverage of π \pi π by another policy π ′ \pi' π ′ is measured by
C π π ′ : = E x ∼ μ [ ∑ y ∈ Y π ( y ∣ x ) 2 π ′ ( y ∣ x ) ] = E π ′ [ π ( y ∣ x ) π ′ ( y ∣ x ) ] . (96) C_{\pi}^{\pi'} := \mathbb{E}_{x\sim\mu}\left[\sum_{y\in\mathcal{Y}}\frac{\pi(y\mid x)^2}{\pi'(y\mid x)}\right]=\mathbb{E}_{\pi'}\left[\frac{\pi(y\mid x)}{\pi'(y\mid x)}\right]. \tag*{(96)} C π π ′ := E x ∼ μ y ∈ Y ∑ π ′ ( y ∣ x ) π ( y ∣ x ) 2 = E π ′ [ π ′ ( y ∣ x ) π ( y ∣ x ) ] . (96) For the base model, we use C b a s e π ≡ C π π b a s e C_{\mathrm{base}}^\pi\equiv C_{\pi}^{\pi_{\mathrm{base}}} C base π ≡ C π π base . For the policy iterate π t \pi_t π t we use C t π ≡ C π π t C_t^\pi\equiv C_{\pi}^{\pi_t} C t π ≡ C π π t .
The comparator policy π \pi π need not be optimal or belong to the learner’s policy class. An example is the distribution induced by Best-of-N N N on the base model with N N N responses per prompt, denoted π N B o N \pi_N^{\mathrm{BoN}} π N BoN , for which C b a s e π N B o N ≤ N C_{\mathrm{base}}^{\pi_N^{\mathrm{BoN}}}\leq N C base π N BoN ≤ N (Huang et al., [40 ] ). Another example is the optimal policy induced by conditioning the base model on optimal responses, given by π b a s e ⋆ : = π b a s e ( y ∣ x ) I [ y ∈ Y π b a s e ⋆ ] π b a s e ( Y π b a s e ⋆ ∣ x ) \pi_{\mathrm{base}}^{\star}:=\frac{\pi_{\mathrm{base}}(y\mid x)\mathbb{I}[y\in\mathcal{Y}_{\pi_{\mathrm{base}}}^{\star}]}{\pi_{\mathrm{base}}(\mathcal{Y}_{\pi_{\mathrm{base}}}^{\star}\mid x)} π base ⋆ := π base ( Y π base ⋆ ∣ x ) π base ( y ∣ x ) I [ y ∈ Y π base ⋆ ] . This comparator policy has J ( π b a s e ⋆ ) = J ⋆ J(\pi_{\mathrm{base}}^{\star})=J^\star J ( π base ⋆ ) = J ⋆ ; moreover,
C b a s e π b a s e ⋆ = C b a s e ⏟ Definition 1.1 . (97) C_{\mathrm{base}}^{\pi_{\mathrm{base}}^\star}=\underbrace{C_{\mathrm{base}}}_{\text{Definition 1.1}}. \tag*{(97)} C base π base ⋆ = Definition 1.1 C base . (97) Therefore, the performance target of the main body (Eq. (4) ) is a special case of the formulation here with comparator policy π = π b a s e ⋆ \pi=\pi_{\mathrm{base}}^\star π = π base ⋆ .
We now extend the convergence guarantee and KL criterion of Theorem 3.1 to hold for any comparator policy π \pi π , where the inner tolerance ε i n \varepsilon_{\mathrm{in}} ε in now depends on C b a s e π C_{\mathrm{base}}^\pi C base π , the coverage of the base model over the comparator.
Theorem F.1 (Convergence relative to a comparator). Fix η > 0 \eta>0 η > 0 and a sequence of policies { π t } t = 0 T \{\pi_t\}_{t=0}^{T} { π t } t = 0 T with π 0 = π b a s e \pi_0=\pi_{\mathrm{base}} π 0 = π base and π t > 0 \pi_t>0 π t > 0 for all t t t . Recall that K L t = D K L ( π t + 1 ∥ π t ⋆ ) \mathrm{KL}_t=D_{\mathrm{KL}}(\pi_{t+1}\Vert\pi_t^\star) KL t = D KL ( π t + 1 ∥ π t ⋆ ) . Then it holds for every comparator policy π \pi π that
J ( π ) − J ( π T ) ≤ D K L ( π ∥ π b a s e ) η T + 1 η ∑ t = 0 T − 1 K L t + 4 η C b a s e π ∑ t = 0 T − 1 K L t t + 1 . (98) J(\pi)-J(\pi_T)\leq\frac{D_{\mathrm{KL}}(\pi\Vert\pi_{\mathrm{base}})}{\eta T}+\frac{1}{\eta}\sum_{t=0}^{T-1}\mathrm{KL}_t+\frac{4}{\eta}\sqrt{C_{\mathrm{base}}^\pi\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t}{t+1}}. \tag*{(98)} J ( π ) − J ( π T ) ≤ η T D KL ( π ∥ π base ) + η 1 t = 0 ∑ T − 1 KL t + η 4 C base π t = 0 ∑ T − 1 t + 1 KL t . (98)
where C b a s e π C_{\mathrm{base}}^{\pi} C base π is given in Definition F.1. In particular, for any target suboptimality ε ∈ ( 0 , R max ] \varepsilon\in(0, R_{\max}] ε ∈ ( 0 , R m a x ] , we have J ( π ) − J ( π T ) ≤ ε J(\pi)-J(\pi_T)\leq\varepsilon J ( π ) − J ( π T ) ≤ ε whenever
T ≥ ⌈ log ( 2 C b a s e π ) η ε ⌉ , D K L ( π t + 1 ∥ π η , t ⋆ ) ≤ ε i n : = 1 64 C b a s e π T 2 log ( e T ) , ∀ 0 ≤ t < T . (99) T \geq\left\lceil\frac{\log(2C_{\mathrm{base}}^{\pi})}{\eta\varepsilon} \right\rceil,\qquad D_{\mathrm{KL}}(\pi_{t+1}\Vert\pi_{\eta,t}^{\star})\leq\varepsilon_{\mathrm{in}}:=\frac{1}{64C_{\mathrm{base}}^{\pi}T^2\log(eT)},\qquad\forall0\leq t<T. \tag*{(99)} T ≥ ⌈ η ε log ( 2 C base π ) ⌉ , D KL ( π t + 1 ∥ π η , t ⋆ ) ≤ ε in := 64 C base π T 2 log ( e T ) 1 , ∀0 ≤ t < T . (99)
Theorem F.1 holds simultaneously for all comparator policies π \pi π . Taking π = π b a s e ⋆ \pi=\pi_{\mathrm{base}}^{\star} π = π base ⋆ (Eq. (97) ) recovers the convergence guarantee of Theorem 3.1. More generally, smaller KL errors D K L ( π t + 1 ∥ π η , t ⋆ ) D_{\mathrm{KL}}(\pi_{t+1}\Vert\pi_{\eta,t}^{\star}) D KL ( π t + 1 ∥ π η , t ⋆ ) allow the final policy to compete with comparators having larger coverage coefficients.
In particular, to achieve J − J ( π T ) ≤ ε J-J(\pi_T)\leq\varepsilon J − J ( π T ) ≤ ε for a target performance J ∈ [ 0 , J ⋆ ] J\in[0,J^{\star}] J ∈ [ 0 , J ⋆ ] , it suffices to have an inner KL tolerance that scales inversely with the best response coverage (lowest C b a s e π C_{\mathrm{base}}^{\pi} C base π ) of any policy with J ( π ) ≥ J J(\pi)\geq J J ( π ) ≥ J ,
T = ⌈ inf π : J ( π ) ≥ J log ( 2 C b a s e π ) η ε ⌉ , D K L ( π t + 1 ∥ π η , t ⋆ ) ≤ 1 64 ( inf π : J ( π ) ≥ J C b a s e π ) T 2 log ( e T ) , ∀ 0 ≤ t < T . (100) T=\left\lceil\frac{\inf_{\pi:J(\pi)\geq J}\log(2C_{\mathrm{base}}^{\pi})}{\eta\varepsilon}\right\rceil,\qquad D_{\mathrm{KL}}(\pi_{t+1}\Vert\pi_{\eta,t}^{\star})\leq\frac{1}{64\left(\inf_{\pi:J(\pi)\geq J}C_{\mathrm{base}}^{\pi}\right)T^2\log(eT)},\qquad\forall0\leq t<T. \tag*{(100)} T = ⌈ η ε inf π : J ( π ) ≥ J log ( 2 C base π ) ⌉ , D KL ( π t + 1 ∥ π η , t ⋆ ) ≤ 64 ( inf π : J ( π ) ≥ J C base π ) T 2 log ( e T ) 1 , ∀0 ≤ t < T . (100)
Lastly, under Assumption 2.1, combining Theorem F.1 with the inner loop convergence result for PPO-NoClip (Theorem G.1) yields end-to-end sample complexity and gradient step complexity guarantees. Choosing η = 1 / ( B ϕ B ⋆ ) \eta=1/(B_{\phi}B_{\star}) η = 1/ ( B ϕ B ⋆ ) , for any comparator policy π \pi π , PPO-NoClip with other parameters set according to C b a s e π C_{\mathrm{base}}^{\pi} C base π returns a final iterate satisfying J ( π ) − J ( π p o s t ) ≤ ε J(\pi)-J(\pi_{\mathrm{post}})\leq\varepsilon J ( π ) − J ( π post ) ≤ ε with probability at least 1 − δ 1-\delta 1 − δ , using gradient steps and sampled responses, respectively,
N s t e p ( ε , δ ; π ) = O ~ ( C b a s e π B ϕ 3 B ⋆ 3 ε 3 ) , N s a m p l e ( ε , δ ; π ) = O ~ ( C b a s e π B ϕ 3 B ⋆ 3 ε 3 [ d + log 1 δ ] ) . (101) N_{\mathrm{step}}(\varepsilon,\delta;\pi)=\widetilde{O}\left(\frac{C_{\mathrm{base}}^{\pi}B_{\phi}^{3}B_{\star}^{3}}{\varepsilon^{3}}\right),\qquad N_{\mathrm{sample}}(\varepsilon,\delta;\pi)=\widetilde{O}\left(\frac{C_{\mathrm{base}}^{\pi}B_{\phi}^{3}B_{\star}^{3}}{\varepsilon^{3}}\left[d+\log\frac{1}{\delta}\right]\right). \tag*{(101)} N step ( ε , δ ; π ) = O ( ε 3 C base π B ϕ 3 B ⋆ 3 ) , N sample ( ε , δ ; π ) = O ( ε 3 C base π B ϕ 3 B ⋆ 3 [ d + log δ 1 ] ) . (101)
Proofs for F.1
We first prove Theorem F.1, then the reward-target guarantee under Eq. (100) .
Proof of Theorem F.1. We will use the following reward threshold from Lemma F.1. Define
τ π ( x , y ) = J x ( π T ) + 1 η ∑ t = 0 T − 1 K L t ( x ) + log ( π ( y ∣ x ) / π b a s e ( y ∣ x ) ) − 1 η T , \tau_{\pi}(x,y)=J_x(\pi_T)+\frac{1}{\eta}\sum_{t=0}^{T-1}\mathrm{KL}_t(x)+\frac{\log(\pi(y\mid x)/\pi_{\mathrm{base}}(y\mid x))-1}{\eta T}, τ π ( x , y ) = J x ( π T ) + η 1 t = 0 ∑ T − 1 KL t ( x ) + η T log ( π ( y ∣ x ) / π base ( y ∣ x )) − 1 ,
where π ( y ∣ x ) > 0 \pi(y\mid x)>0 π ( y ∣ x ) > 0 , and τ π ( x , y ) = 0 \tau_{\pi}(x,y)=0 τ π ( x , y ) = 0 otherwise. Using r ( x , y ) ≤ τ π ( x , y ) + [ r ( x , y ) − τ π ( x , y ) ] + r(x,y)\leq\tau_{\pi}(x,y)+[r(x,y)-\tau_{\pi}(x,y)]_{+} r ( x , y ) ≤ τ π ( x , y ) + [ r ( x , y ) − τ π ( x , y ) ] + , we obtain
J ( π ) − J ( π T ) ≤ E π [ τ π ( x , y ) − J x ( π T ) ] + E π [ [ r ( x , y ) − τ π ( x , y ) ] + ] = D K L ( π ∥ π b a s e ) − 1 η T + 1 η ∑ t = 0 T − 1 K L t + E π [ [ r ( x , y ) − τ π ( x , y ) ] + ] . \begin{aligned}
J(\pi)-J(\pi_T)
&\leq\mathbb{E}_{\pi}[\tau_{\pi}(x,y)-J_x(\pi_T)]+\mathbb{E}_{\pi}[[r(x,y)-\tau_{\pi}(x,y)]_{+}]\\
&=\frac{D_{\mathrm{KL}}(\pi\Vert\pi_{\mathrm{base}})-1}{\eta T}+\frac{1}{\eta}\sum_{t=0}^{T-1}\mathrm{KL}_t+\mathbb{E}_{\pi}[[r(x,y)-\tau_{\pi}(x,y)]_{+}].
\end{aligned} J ( π ) − J ( π T ) ≤ E π [ τ π ( x , y ) − J x ( π T )] + E π [[ r ( x , y ) − τ π ( x , y ) ] + ] = η T D KL ( π ∥ π base ) − 1 + η 1 t = 0 ∑ T − 1 KL t + E π [[ r ( x , y ) − τ π ( x , y ) ] + ] .
Lemma F.1 bounds the last term as
E π [ [ r ( x , y ) − τ π ( x , y ) ] + ] ≤ 1 η T + 2 η e C b a s e π ∑ t = 0 T − 1 K L t t + 1 , \mathbb{E}_{\pi}[[r(x,y)-\tau_{\pi}(x,y)]_{+}]\leq\frac{1}{\eta T}+\frac{2}{\eta}\sqrt{eC_{\mathrm{base}}^{\pi}\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t}{t+1}}, E π [[ r ( x , y ) − τ π ( x , y ) ] + ] ≤ η T 1 + η 2 e C base π t = 0 ∑ T − 1 t + 1 KL t ,
and substituting gives
J ( π ) − J ( π T ) ≤ D K L ( π ∥ π b a s e ) η T + 1 η ∑ t = 0 T − 1 K L t + 2 η e C b a s e π ∑ t = 0 T − 1 K L t t + 1 . J(\pi)-J(\pi_T)\leq\frac{D_{\mathrm{KL}}(\pi\Vert\pi_{\mathrm{base}})}{\eta T}+\frac{1}{\eta}\sum_{t=0}^{T-1}\mathrm{KL}_t+\frac{2}{\eta}\sqrt{eC_{\mathrm{base}}^{\pi}\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t}{t+1}}. J ( π ) − J ( π T ) ≤ η T D KL ( π ∥ π base ) + η 1 t = 0 ∑ T − 1 KL t + η 2 e C base π t = 0 ∑ T − 1 t + 1 KL t .
This proves the first theorem claim in Eq. (98) , which uses 2 e ≤ 4 2\sqrt{e}\leq4 2 e ≤ 4 .
Next, for the second claim in the theorem, Jensen’s inequality gives
D K L ( π ∥ π b a s e ) = E π [ log π ( y ∣ x ) π b a s e ( y ∣ x ) ] ≤ log E π [ π ( y ∣ x ) π b a s e ( y ∣ x ) ] = log C b a s e π . (102) D_{\mathrm{KL}}(\pi\Vert\pi_{\mathrm{base}})=\mathbb{E}_{\pi}\left[\log\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right]\leq\log\mathbb{E}_{\pi}\left[\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right]=\log C_{\mathrm{base}}^{\pi}. \tag*{(102)} D KL ( π ∥ π base ) = E π [ log π base ( y ∣ x ) π ( y ∣ x ) ] ≤ log E π [ π base ( y ∣ x ) π ( y ∣ x ) ] = log C base π . (102)
Under the KL tolerance D K L ( π t + 1 ∥ π η ⋆ ) ≤ ε i n D_{\mathrm{KL}}(\pi_{t+1}\Vert\pi^\star_{\eta})\leq\varepsilon_{\mathrm{in}} D KL ( π t + 1 ∥ π η ⋆ ) ≤ ε in from Eq. (99) ,
∑ t = 0 T − 1 K L t ≤ T ε i n , ∑ t = 0 T − 1 K L t t + 1 ≤ ε i n ∑ t = 1 T 1 t ≤ ε i n log ( e T ) . \sum_{t=0}^{T-1}\mathrm{KL}_t\leq T\varepsilon_{\mathrm{in}},\qquad\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t}{t+1}\leq\varepsilon_{\mathrm{in}}\sum_{t=1}^{T}\frac{1}{t}\leq\varepsilon_{\mathrm{in}}\log(eT). t = 0 ∑ T − 1 KL t ≤ T ε in , t = 0 ∑ T − 1 t + 1 KL t ≤ ε in t = 1 ∑ T t 1 ≤ ε in log ( e T ) .
Substituting these bounds into Eq. (98) and the bounds on T T T and ε i n \varepsilon_{\mathrm{in}} ε in from Eq. (99) gives
J ( π ) − J ( π T ) ≤ log C b a s e π η T + T ε i n η + 4 η C b a s e π ε i n log ( e T ) = 1 η T ( log C b a s e π + 1 64 C b a s e π log ( e T ) + 1 2 ) ≤ log ( 2 C b a s e π ) η T ≤ ε . \begin{aligned}
J(\pi)-J(\pi_T)\leq\frac{\log C_{\mathrm{base}}^\pi}{\eta T}+\frac{T\varepsilon_{\mathrm{in}}}{\eta}+\frac{4}{\eta}\sqrt{C_{\mathrm{base}}^\pi\varepsilon_{\mathrm{in}}\log(eT)} \\
&=\frac{1}{\eta T}\left(\log C_{\mathrm{base}}^\pi+\frac{1}{64C_{\mathrm{base}}^\pi\log(eT)}+\frac{1}{2}\right) \\
&\leq\frac{\log(2C_{\mathrm{base}}^\pi)}{\eta T}\leq\varepsilon.
\end{aligned} J ( π ) − J ( π T ) ≤ η T log C base π + η T ε in + η 4 C base π ε in log ( e T ) = η T 1 ( log C base π + 64 C base π log ( e T ) 1 + 2 1 ) ≤ η T log ( 2 C base π ) ≤ ε .
This proves the second theorem claim.
Proof of Eq. (100) . Fix a target performance J ∈ [ 0 , J ⋆ ] J\in[0,J^\star] J ∈ [ 0 , J ⋆ ] and write C : = inf π : J ( π ) ≥ J C b a s e π C:=\inf_{\pi:J(\pi)\geq J}C_{\mathrm{base}}^\pi C := inf π : J ( π ) ≥ J C base π . The choices in Eq. (100) give T = ⌈ log ( 2 C ) / ( η ε ) ⌉ T=\lceil\log(2C)/(\eta\varepsilon)\rceil T = ⌈ log ( 2 C ) / ( η ε )⌉ and K L t ≤ ε i n : = 1 / ( 64 C T 2 log ( e T ) ) \mathrm{KL}_t\leq\varepsilon_{\mathrm{in}}:=1/(64CT^2\log(eT)) KL t ≤ ε in := 1/ ( 64 C T 2 log ( e T )) for every 0 ≤ t < T 0\leq t<T 0 ≤ t < T . For any comparator policy π \pi π with J ( π ) ≥ J J(\pi)\geq J J ( π ) ≥ J , applying Eq. (98) with the KL bound Eq. (102) gives
J − J ( π T ) ≤ J ( π ) − J ( π T ) ≤ log C b a s e π η T + T ε i n η + 4 η C b a s e π ε i n log ( e T ) , \begin{aligned}
J-J(\pi_T)\leq J(\pi)-J(\pi_T) \\
&\leq\frac{\log C_{\mathrm{base}}^\pi}{\eta T}+\frac{T\varepsilon_{\mathrm{in}}}{\eta}+\frac{4}{\eta}\sqrt{C_{\mathrm{base}}^\pi\varepsilon_{\mathrm{in}}\log(eT)},
\end{aligned} J − J ( π T ) ≤ J ( π ) − J ( π T ) ≤ η T log C base π + η T ε in + η 4 C base π ε in log ( e T ) ,
where we used K L t ≤ ε i n \mathrm{KL}_t\leq\varepsilon_{\mathrm{in}} KL t ≤ ε in and ∑ t = 1 T 1 / t ≤ log ( e T ) \sum_{t=1}^{T}1/t\leq\log(eT) ∑ t = 1 T 1/ t ≤ log ( e T ) . The right-hand side is continuous and increasing in C b a s e π C_{\mathrm{base}}^\pi C base π , so taking the infimum over these comparators and substituting ε i n \varepsilon_{\mathrm{in}} ε in yields
J − J ( π T ) ≤ log C η T + T ε i n η + 4 η C ε i n log ( e T ) = 1 η T ( log C + 1 64 C log ( e T ) + 1 2 ) ≤ log ( 2 C ) η T ≤ ε . \begin{aligned}
J-J(\pi_T)\leq\frac{\log C}{\eta T}+\frac{T\varepsilon_{\mathrm{in}}}{\eta}+\frac{4}{\eta}\sqrt{C\varepsilon_{\mathrm{in}}\log(eT)} \\
&=\frac{1}{\eta T}\left(\log C+\frac{1}{64C\log(eT)}+\frac{1}{2}\right)\leq\frac{\log(2C)}{\eta T}\leq\varepsilon.
\end{aligned} J − J ( π T ) ≤ η T log C + η T ε in + η 4 C ε in log ( e T ) = η T 1 ( log C + 64 C log ( e T ) 1 + 2 1 ) ≤ η T log ( 2 C ) ≤ ε .
The last inequality uses the choice of T T T in Eq. (100) .
We now state and prove the supporting lemmas used in the proof of Theorem F.1.
Lemma F.1. For any comparator policy π \pi π , define the reward threshold τ π : X × Y → ( − ∞ , ∞ ) \tau_\pi:\mathcal{X}\times\mathcal{Y}\to(-\infty,\infty) τ π : X × Y → ( − ∞ , ∞ ) to be
τ π ( x , y ) : = J x ( π T ) + 1 η ∑ t = 0 T − 1 K L t ( x ) + log ( π ( y ∣ x ) / π b a s e ( y ∣ x ) ) − 1 η T , \tau_\pi(x,y):=J_x(\pi_T)+\frac{1}{\eta}\sum_{t=0}^{T-1}\mathrm{KL}_t(x)+\frac{\log(\pi(y\mid x)/\pi_{\mathrm{base}}(y\mid x))-1}{\eta T}, τ π ( x , y ) := J x ( π T ) + η 1 t = 0 ∑ T − 1 KL t ( x ) + η T log ( π ( y ∣ x ) / π base ( y ∣ x )) − 1 , where π ( y ∣ x ) > 0 \pi(y\mid x)>0 π ( y ∣ x ) > 0 , and τ π ( x , y ) = 0 \tau_\pi(x,y)=0 τ π ( x , y ) = 0 otherwise. Then
E π [ [ r ( x , y ) − τ π ( x , y ) ] + ] ≤ 1 η T + 2 η e C b a s e π ∑ t = 0 T − 1 K L t t + 1 . (103) \mathbb{E}_{\pi}[[r(x,y)-\tau_\pi(x,y)]_+]\leq\frac{1}{\eta T}+\frac{2}{\eta}\sqrt{e^{C_{\mathrm{base}}^\pi}\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t}{t+1}}. \tag*{(103)} E π [[ r ( x , y ) − τ π ( x , y ) ] + ] ≤ η T 1 + η 2 e C base π t = 0 ∑ T − 1 t + 1 KL t . (103)
Proof of Lemma F.1. To bound Eq. (103) , define
f ( x , y ) : = π ( y ∣ x ) π b a s e ( y ∣ x ) [ r ( x , y ) − τ π ( x , y ) ] + . f(x,y):=\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}[r(x,y)-\tau_\pi(x,y)]_+. f ( x , y ) := π base ( y ∣ x ) π ( y ∣ x ) [ r ( x , y ) − τ π ( x , y ) ] + .
Where π ( y ∣ x ) > 0 \pi(y\mid x)>0 π ( y ∣ x ) > 0 , the definition of τ π \tau_\pi τ π gives
0 ≤ f ( x , y ) ≤ π ( y ∣ x ) η T π b a s e ( y ∣ x ) [ 1 + η T R max − log π ( y ∣ x ) π b a s e ( y ∣ x ) ] + ≤ e η T R max η T , 0\leq f(x,y)\leq\frac{\pi(y\mid x)}{\eta T\pi_{\mathrm{base}}(y\mid x)}
\left[1+\eta T R_{\max}-\log\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right]_+
\leq\frac{e^{\eta T R_{\max}}}{\eta T}, 0 ≤ f ( x , y ) ≤ η T π base ( y ∣ x ) π ( y ∣ x ) [ 1 + η T R m a x − log π base ( y ∣ x ) π ( y ∣ x ) ] + ≤ η T e η T R m a x ,
where the last inequality uses sup u > 0 { u [ 1 + η T R max − log u ] + } = e η T R max \sup_{u>0}\{u[1+\eta T R_{\max}-\log u]_{+}\}=e^{\eta T R_{\max}} sup u > 0 { u [ 1 + η T R m a x − log u ] + } = e η T R m a x . Since f ( x , y ) = 0 f(x,y)=0 f ( x , y ) = 0 otherwise, f f f is nonnegative and bounded, with
E π b a s e [ f ( x , y ) ] = E π [ [ r ( x , y ) − τ π ( x , y ) ] + ] < ∞ . \mathbb{E}_{\pi_{\mathrm{base}}}[f(x,y)]=\mathbb{E}_{\pi}[[r(x,y)-\tau_{\pi}(x,y)]_{+}]<\infty. E π base [ f ( x , y )] = E π [[ r ( x , y ) − τ π ( x , y ) ] + ] < ∞.
Then, applying Lemma D.3 at t = T t=T t = T to the bounded nonnegative function f ( x , y ) e η 2 ∑ s = 0 T − 1 ( b s ∗ ( x ) − r ( x , y ) ) f(x,y)e^{\frac{\eta}{2}\sum_{s=0}^{T-1}(b_s^*(x)-r(x,y))} f ( x , y ) e 2 η ∑ s = 0 T − 1 ( b s ∗ ( x ) − r ( x , y )) , we obtain
E π b a s e [ f ( x , y ) ] ≤ E π b a s e [ f ( x , y ) e η 2 ∑ s = 0 T − 1 ( b s ∗ ( x ) − r ( x , y ) ) π T ( y ∣ x ) π b a s e ( y ∣ x ) ] + ∑ t = 1 T K L t − 1 ( E π b a s e [ f ( x , y ) 2 e η ∑ s = 0 t − 1 ( b s ∗ ( x ) − r ( x , y ) ) ] ) ≤ ( E π b a s e [ f ( x , y ) 2 e − η T ( r ( x , y ) − J x ( π T ) ) + T ∑ s = 0 T − 1 K L s ( x ) ] ) 1 / 2 + ∑ t = 1 T K L t − 1 ( E π b a s e [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T ) ) + t ∑ s = 0 T − 1 K L s ( x ) ] ) . (104) \begin{aligned}
\mathbb{E}_{\pi_{\mathrm{base}}}[f(x,y)]
&\leq\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)e^{\frac{\eta}{2}\sum_{s=0}^{T-1}(b_s^*(x)-r(x,y))}\sqrt{\frac{\pi_T(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}}\right]\\
&\quad+\sum_{t=1}^{T}\sqrt{\mathrm{KL}_{t-1}\left(\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)^2e^{\eta\sum_{s=0}^{t-1}(b_s^*(x)-r(x,y))}\right]\right)}\\
&\leq\left(\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)^2e^{-\eta T(r(x,y)-J_x(\pi_T))+T\sum_{s=0}^{T-1}\mathrm{KL}_s(x)}\right]\right)^{1/2}\\
&\quad+\sum_{t=1}^{T}\sqrt{\mathrm{KL}_{t-1}\left(\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)^2e^{-\eta t(r(x,y)-J_x(\pi_T))+t\sum_{s=0}^{T-1}\mathrm{KL}_s(x)}\right]\right)}. \tag*{(104)}
\end{aligned} E π base [ f ( x , y )] ≤ E π base [ f ( x , y ) e 2 η ∑ s = 0 T − 1 ( b s ∗ ( x ) − r ( x , y )) π base ( y ∣ x ) π T ( y ∣ x ) ] + t = 1 ∑ T KL t − 1 ( E π base [ f ( x , y ) 2 e η ∑ s = 0 t − 1 ( b s ∗ ( x ) − r ( x , y )) ] ) ≤ ( E π base [ f ( x , y ) 2 e − η T ( r ( x , y ) − J x ( π T )) + T ∑ s = 0 T − 1 KL s ( x ) ] ) 1/2 + t = 1 ∑ T KL t − 1 ( E π base [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T )) + t ∑ s = 0 T − 1 KL s ( x ) ] ) . (104)
The second inequality uses Cauchy–Schwarz, and the following inequality from Lemma D.4 to bound the exponents, which states for every 1 ≤ t ≤ T 1\leq t\leq T 1 ≤ t ≤ T that,
η ∑ s = 0 t − 1 ( b s ∗ ( x ) − r ( x , y ) ) ≤ − η t ( r ( x , y ) − J x ( π T ) ) + t ∑ s = 0 T − 1 K L s ( x ) . \eta\sum_{s=0}^{t-1}(b_s^*(x)-r(x,y))\leq-\eta t(r(x,y)-J_x(\pi_T))+t\sum_{s=0}^{T-1}\mathrm{KL}_s(x). η s = 0 ∑ t − 1 ( b s ∗ ( x ) − r ( x , y )) ≤ − η t ( r ( x , y ) − J x ( π T )) + t s = 0 ∑ T − 1 KL s ( x ) .
We now bound the expectations on the RHS of Eq. (104) . Where π ( y ∣ x ) > 0 \pi(y\mid x)>0 π ( y ∣ x ) > 0 , the definition of τ π \tau_{\pi} τ π gives
− η t ( r ( x , y ) − J x ( π T ) ) + t ∑ s = 0 T − 1 K L s ( x ) = − η t ( r ( x , y ) − τ π ( x , y ) ) + t T ( 1 − log π ( y ∣ x ) π b a s e ( y ∣ x ) ) . -\eta t(r(x,y)-J_x(\pi_T))+t\sum_{s=0}^{T-1}\mathrm{KL}_s(x)
=-\eta t(r(x,y)-\tau_{\pi}(x,y))+\frac{t}{T}\left(1-\log\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right). − η t ( r ( x , y ) − J x ( π T )) + t s = 0 ∑ T − 1 KL s ( x ) = − η t ( r ( x , y ) − τ π ( x , y )) + T t ( 1 − log π base ( y ∣ x ) π ( y ∣ x ) ) .
Since f ( x , y ) = 0 f(x,y)=0 f ( x , y ) = 0 otherwise, substituting this identity and the definition of f f f gives, for every 1 ≤ t ≤ T 1\leq t\leq T 1 ≤ t ≤ T ,
E π b a s e [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T ) ) + t ∑ s = 0 T − 1 K L s ( x ) ] = e t / T E π [ ( π ( y ∣ x ) π b a s e ( y ∣ x ) ) 1 − t / T [ r ( x , y ) − τ π ( x , y ) ] + 2 e − η t ( r ( x , y ) − τ π ( x , y ) ) ] . (105) \begin{aligned}
\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)^2e^{-\eta t(r(x,y)-J_x(\pi_T))+t\sum_{s=0}^{T-1}\mathrm{KL}_s(x)}\right]
=e^{t/T}\mathbb{E}_{\pi}\left[\left(\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right)^{1-t/T}[r(x,y)-\tau_{\pi}(x,y)]_{+}^{2}e^{-\eta t(r(x,y)-\tau_{\pi}(x,y))}\right]. \tag*{(105)}
\end{aligned} E π base [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T )) + t ∑ s = 0 T − 1 KL s ( x ) ] = e t / T E π [ ( π base ( y ∣ x ) π ( y ∣ x ) ) 1 − t / T [ r ( x , y ) − τ π ( x , y ) ] + 2 e − η t ( r ( x , y ) − τ π ( x , y )) ] . (105)
For the first term on the RHS of Eq. (104) , taking t = T t=T t = T in Eq. (105) gives
E π b a s e [ f ( x , y ) 2 e − η T ( r ( x , y ) − J x ( π T ) ) + T ∑ s = 0 T − 1 K L s ( x ) ] = e E π [ [ r ( x , y ) − τ π ( x , y ) ] + 2 e − η T ( r ( x , y ) − τ π ( x , y ) ) ] ≤ E π [ [ r ( x , y ) − τ π ( x , y ) ] + ] η T , (106) \mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)^2e^{-\eta T(r(x,y)-J_x(\pi_T))+T\sum_{s=0}^{T-1}\mathrm{KL}_s(x)}\right]
=e\mathbb{E}_{\pi}\left[[r(x,y)-\tau_{\pi}(x,y)]_{+}^{2}e^{-\eta T(r(x,y)-\tau_{\pi}(x,y))}\right]
\leq\frac{\mathbb{E}_{\pi}[[r(x,y)-\tau_{\pi}(x,y)]_{+}]}{\eta T}, \tag*{(106)} E π base [ f ( x , y ) 2 e − η T ( r ( x , y ) − J x ( π T )) + T ∑ s = 0 T − 1 KL s ( x ) ] = e E π [ [ r ( x , y ) − τ π ( x , y ) ] + 2 e − η T ( r ( x , y ) − τ π ( x , y )) ] ≤ η T E π [[ r ( x , y ) − τ π ( x , y ) ] + ] , (106)
where the last inequality uses sup z ≥ 0 z e − η T z = 1 / ( e η T ) \sup_{z\geq0}ze^{-\eta Tz}=1/(e\eta T) sup z ≥ 0 z e − η T z = 1/ ( eη T ) .
Next, for the second term of Eq. (104) , the Cauchy–Schwarz inequality gives
( ∑ t = 1 T K L t − 1 ( E π b a s e [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T ) ) + t ∑ s = 0 T − 1 K L s ( x ) ] ) ) 2 ≤ ( ∑ t = 1 T K L t − 1 t ) ∑ t = 1 T t E π b a s e [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T ) ) + t ∑ s = 0 T − 1 K L s ( x ) ] . (107) \left(\sum_{t=1}^{T}\sqrt{\mathrm{KL}_{t-1}\left(\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)^2e^{-\eta t(r(x,y)-J_x(\pi_T))+t\sum_{s=0}^{T-1}\mathrm{KL}_s(x)}\right]\right)}\right)^2
\leq\left(\sum_{t=1}^{T}\frac{\mathrm{KL}_{t-1}}{t}\right)\sum_{t=1}^{T}t\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)^2e^{-\eta t(r(x,y)-J_x(\pi_T))+t\sum_{s=0}^{T-1}\mathrm{KL}_s(x)}\right]. \tag*{(107)} ( t = 1 ∑ T KL t − 1 ( E π base [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T )) + t ∑ s = 0 T − 1 KL s ( x ) ] ) ) 2 ≤ ( t = 1 ∑ T t KL t − 1 ) t = 1 ∑ T t E π base [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T )) + t ∑ s = 0 T − 1 KL s ( x ) ] . (107)
To bound the second sum on the RHS of Eq. (107) , we use the following two inequalities. For u > 0 u>0 u > 0 and s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] ,
e s u 2 − s = ( u 2 ) 1 − s ( e u ) s ≤ max { u 2 , e u } ≤ e 2 ( 1 + u 2 ) , (108) e^{s}u^{2-s}=(u^2)^{1-s}(eu)^s\le\max\{u^2,eu\}\le\frac{e}{2}(1+u^2), \tag*{(108)} e s u 2 − s = ( u 2 ) 1 − s ( e u ) s ≤ max { u 2 , e u } ≤ 2 e ( 1 + u 2 ) , (108)
which also holds for u = 0 u=0 u = 0 . Further, for z > 0 z>0 z > 0 ,
z 2 ∑ t = 1 ∞ t e − t z = z 2 e − z ( 1 − e − z ) 2 = ( z e z / 2 − e − z / 2 ) 2 ≤ 1 , (109) z^2\sum_{t=1}^{\infty}te^{-tz}=\frac{z^2e^{-z}}{(1-e^{-z})^2}=\left(\frac{z}{e^{z/2}-e^{-z/2}}\right)^2\le1, \tag*{(109)} z 2 t = 1 ∑ ∞ t e − t z = ( 1 − e − z ) 2 z 2 e − z = ( e z /2 − e − z /2 z ) 2 ≤ 1 , (109)
since e z / 2 − e − z / 2 ≥ z e^{z/2}-e^{-z/2}\ge z e z /2 − e − z /2 ≥ z . Rewriting Eq. (105) as an expectation under π b a s e \pi_{\mathrm{base}} π base and applying these inequalities gives
∑ t = 1 T t E π b a s e [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T ) ) + t ∑ s = 0 T − 1 K L s ( x ) ] = E π b a s e [ ( r ( x , y ) − τ π ( x , y ) ) 2 ∑ t = 1 T t e t / T ( π ( y ∣ x ) π b a s e ( y ∣ x ) ) 2 − t / T e − η t ( r ( x , y ) − τ π ( x , y ) ) ] ≤ e 2 E π b a s e [ ( 1 + ( π ( y ∣ x ) π b a s e ( y ∣ x ) ) 2 ) [ r ( x , y ) − τ π ( x , y ) ] 2 ∑ t = 1 T e − η t ( r ( x , y ) − τ π ( x , y ) ) ] ≤ e 2 η 2 E π b a s e [ 1 + ( π ( y ∣ x ) π b a s e ( y ∣ x ) ) 2 ] = e ( 1 + C b a s e π ) 2 η 2 ≤ e C b a s e π η 2 . (110) \begin{aligned}
\sum_{t=1}^{T}t\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)^2e^{-\eta t(r(x,y)-J_x(\pi_T))+t\sum_{s=0}^{T-1}\mathrm{KL}_s(x)}\right] \\
&=\mathbb{E}_{\pi_{\mathrm{base}}}\left[(r(x,y)-\tau_\pi(x,y))^2\sum_{t=1}^{T}te^{t/T}\left(\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right)^{2-t/T}e^{-\eta t(r(x,y)-\tau_\pi(x,y))}\right] \\
&\le\frac{e}{2}\mathbb{E}_{\pi_{\mathrm{base}}}\left[\left(1+\left(\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right)^2\right)[r(x,y)-\tau_\pi(x,y)]^2\sum_{t=1}^{T}e^{-\eta t(r(x,y)-\tau_\pi(x,y))}\right] \\
&\le\frac{e}{2\eta^2}\mathbb{E}_{\pi_{\mathrm{base}}}\left[1+\left(\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right)^2\right]=\frac{e(1+C_{\mathrm{base}}^\pi)}{2\eta^2}\le\frac{eC_{\mathrm{base}}^\pi}{\eta^2}. \tag*{(110)}
\end{aligned} t = 1 ∑ T t E π base [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T )) + t ∑ s = 0 T − 1 KL s ( x ) ] = E π base [ ( r ( x , y ) − τ π ( x , y ) ) 2 t = 1 ∑ T t e t / T ( π base ( y ∣ x ) π ( y ∣ x ) ) 2 − t / T e − η t ( r ( x , y ) − τ π ( x , y )) ] ≤ 2 e E π base [ ( 1 + ( π base ( y ∣ x ) π ( y ∣ x ) ) 2 ) [ r ( x , y ) − τ π ( x , y ) ] 2 t = 1 ∑ T e − η t ( r ( x , y ) − τ π ( x , y )) ] ≤ 2 η 2 e E π base [ 1 + ( π base ( y ∣ x ) π ( y ∣ x ) ) 2 ] = 2 η 2 e ( 1 + C base π ) ≤ η 2 e C base π . (110)
The first inequality above uses u = π ( y ∣ x ) / π b a s e ( y ∣ x ) u=\pi(y\mid x)/\pi_{\mathrm{base}}(y\mid x) u = π ( y ∣ x ) / π base ( y ∣ x ) and s = t / T s=t/T s = t / T in Eq. (108) . The second uses z = η ( r ( x , y ) − τ π ( x , y ) ) z=\eta(r(x,y)-\tau_\pi(x,y)) z = η ( r ( x , y ) − τ π ( x , y )) where r ( x , y ) > τ π ( x , y ) r(x,y)>\tau_\pi(x,y) r ( x , y ) > τ π ( x , y ) in Eq. (109) .
Substituting Eq. (110) into Eq. (107) , we obtain
( ∑ t = 1 T K L t − 1 ( E π b a s e [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T ) ) + t ∑ s = 0 T − 1 K L s ( x ) ] ) ) 2 ≤ e C b a s e π η 2 ∑ t = 1 T K L t − 1 t . (111) \left(\sum_{t=1}^{T}\sqrt{\mathrm{KL}_{t-1}\left(\mathbb{E}_{\pi_{\mathrm{base}}}\left[f(x,y)^2e^{-\eta t(r(x,y)-J_x(\pi_T))+t\sum_{s=0}^{T-1}\mathrm{KL}_s(x)}\right]\right)}\right)^2
\le\frac{eC_{\mathrm{base}}^\pi}{\eta^2}\sum_{t=1}^{T}\frac{\mathrm{KL}_{t-1}}{t}. \tag*{(111)} ( t = 1 ∑ T KL t − 1 ( E π base [ f ( x , y ) 2 e − η t ( r ( x , y ) − J x ( π T )) + t ∑ s = 0 T − 1 KL s ( x ) ] ) ) 2 ≤ η 2 e C base π t = 1 ∑ T t KL t − 1 . (111)
Substituting Eqs. (106) and (111) into Eq. (104) gives
E π [ [ r ( x , y ) − τ π ( x , y ) ] + ] ≤ E π [ [ r ( x , y ) − τ π ( x , y ) ] + ] η T + 1 η e C b a s e π ∑ t = 0 T − 1 K L t t + 1 ≤ 1 2 E π [ [ r ( x , y ) − τ π ( x , y ) ] + ] + 1 2 η T + 1 η e C b a s e π ∑ t = 0 T − 1 K L t t + 1 . \begin{aligned}
\mathbb{E}_{\pi}[[r(x,y)-\tau_\pi(x,y)]_+]\le\sqrt{\frac{\mathbb{E}_{\pi}[[r(x,y)-\tau_\pi(x,y)]_+]}{\eta T}}+\frac{1}{\eta}\sqrt{eC_{\mathrm{base}}^\pi\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t}{t+1}} \\
&\le\frac{1}{2}\mathbb{E}_{\pi}[[r(x,y)-\tau_\pi(x,y)]_+]+\frac{1}{2\eta T}+\frac{1}{\eta}\sqrt{eC_{\mathrm{base}}^\pi\sum_{t=0}^{T-1}\frac{\mathrm{KL}_t}{t+1}}.
\end{aligned} E π [[ r ( x , y ) − τ π ( x , y ) ] + ] ≤ η T E π [[ r ( x , y ) − τ π ( x , y ) ] + ] + η 1 e C base π t = 0 ∑ T − 1 t + 1 KL t ≤ 2 1 E π [[ r ( x , y ) − τ π ( x , y ) ] + ] + 2 η T 1 + η 1 e C base π t = 0 ∑ T − 1 t + 1 KL t .
Rearranging proves Eq. (103) .
Lemma F.2 (Coverage of the optimal reward). The policy π b a s e ∗ \pi_{\mathrm{base}}^* π base ∗ defined in Appendix § 11.3 .§ 1 satisfies J ( π b a s e ∗ ) = J ∗ J(\pi_{\mathrm{base}}^*)=J^* J ( π base ∗ ) = J ∗ and
C b a s e π b a s e ∗ = inf π : J ( π ) ≥ J b a s e ∗ C b a s e π = E x ∼ μ [ 1 π b a s e ( Y x ∗ ∣ x ) ] = C b a s e . C_{\mathrm{base}}^{\pi_{\mathrm{base}}^*}=\inf_{\pi:J(\pi)\ge J_{\mathrm{base}}^*}C_{\mathrm{base}}^\pi=\mathbb{E}_{x\sim\mu}\left[\frac{1}{\pi_{\mathrm{base}}(Y_x^*\mid x)}\right]=C_{\mathrm{base}}. C base π base ∗ = π : J ( π ) ≥ J base ∗ inf C base π = E x ∼ μ [ π base ( Y x ∗ ∣ x ) 1 ] = C base .
Proof of Lemma F.2 . By definition, π b a s e ∗ \pi_{\mathrm{base}}^* π base ∗ is supported on Y x ∗ Y_x^* Y x ∗ , so J ( π b a s e ∗ ) = J ∗ J(\pi_{\mathrm{base}}^*)=J^* J ( π base ∗ ) = J ∗ . Substituting its probabilities into Eq. (96) gives
C b a s e π b a s e ∗ = E x ∼ μ [ ∑ y ∈ Y x ∗ π b a s e ( y ∣ x ) π b a s e ( Y x ∗ ∣ x ) 2 ] = E x ∼ μ [ 1 π b a s e ( Y x ∗ ∣ x ) ] = C b a s e . C_{\mathrm{base}}^{\pi_{\mathrm{base}}^*}=\mathbb{E}_{x\sim\mu}\left[\sum_{y\in Y_x^*}\frac{\pi_{\mathrm{base}}(y\mid x)}{\pi_{\mathrm{base}}(Y_x^*\mid x)^2}\right]=\mathbb{E}_{x\sim\mu}\left[\frac{1}{\pi_{\mathrm{base}}(Y_x^*\mid x)}\right]=C_{\mathrm{base}}. C base π base ∗ = E x ∼ μ y ∈ Y x ∗ ∑ π base ( Y x ∗ ∣ x ) 2 π base ( y ∣ x ) = E x ∼ μ [ π base ( Y x ∗ ∣ x ) 1 ] = C base .
Thus the infimum over policies with reward at least J ⋆ J^\star J ⋆ is at most C b a s e C_{\mathrm{base}} C base . Conversely, any such policy π \pi π has J ( π ) = J ⋆ J(\pi) = J^\star J ( π ) = J ⋆ and π ( Y x ⋆ ∣ x ) = 1 \pi(\mathcal{Y}_x^\star\mid x) = 1 π ( Y x ⋆ ∣ x ) = 1 for μ \mu μ -almost every prompt. Cauchy–Schwarz therefore gives
∑ y ∈ Y x ⋆ π ( y ∣ x ) 2 π b a s e ( y ∣ x ) ≥ ( ∑ y ∈ Y x ⋆ π ( y ∣ x ) ) 2 π b a s e ( Y x ⋆ ∣ x ) = 1 π b a s e ( Y x ⋆ ∣ x ) . \sum_{y \in\mathcal{Y}_x^\star} \frac{\pi(y \mid x)^2}{\pi_{\mathrm{base}}(y \mid x)} \ge\frac{\left(\sum_{y \in\mathcal{Y}_x^\star} \pi(y \mid x)\right)^2}{\pi_{\mathrm{base}}(\mathcal{Y}_x^\star\mid x)} = \frac{1}{\pi_{\mathrm{base}}(\mathcal{Y}_x^\star\mid x)}. y ∈ Y x ⋆ ∑ π base ( y ∣ x ) π ( y ∣ x ) 2 ≥ π base ( Y x ⋆ ∣ x ) ( ∑ y ∈ Y x ⋆ π ( y ∣ x ) ) 2 = π base ( Y x ⋆ ∣ x ) 1 .
Averaging over x ∼ μ x \sim\mu x ∼ μ yields C ˉ b a s e ≥ C b a s e \bar{C}_{\mathrm{base}} \ge C_{\mathrm{base}} C ˉ base ≥ C base . Taking the infimum over these policies proves the reverse inequality.
Part II
PPO-Style Algorithms
PPO-NoClip: Global Convergence (Proof of Corollary 3.1)
This section proves Corollary 3.1, which establishes that PPO-NoClip is coverage-preserving under Assumption 2.1. We first show that the inner loop of Algorithm 1, which reuses a fixed batch, approximately maximizes the regularized population objective. The proof adapts the population policy gradient argument in Appendix § 8 .§ 2 . We control the distance to a parameter realizing the exact mirror descent update, then use this control to bound the objective gap of the final inner iterate. The KL divergence criterion in Theorem D.1 converts this inner guarantee into global convergence and coverage preservation. The section is organized as follows.
Appendix G.§ 1 develops the supporting lemmas for the population and empirical objectives.
Appendix G.§ 2 proves convergence of the inner optimization loop of Algorithm 1 on a fixed batch.
Appendix G.§ 3 states the full version of Corollary 3.1 and proves global convergence using the KL divergence criterion in Theorem D.1.
Notation. Throughout this section, suppose that Assumption 2.1 holds, and define the feature diameter
D ϕ : = max x , y , y ′ ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 ≤ 2 B ϕ . D_{\phi} := \max_{x,y,y'} \lVert\phi(x,y)-\phi(x,y') \rVert_{2} \le2B_{\phi}. D ϕ := x , y , y ′ max ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 ≤ 2 B ϕ .
For an outer iteration t t t and regularization parameter η > 0 \eta> 0 η > 0 , the exact mirror descent update is
π η , t ⋆ ( y ∣ x ) = π t ( y ∣ x ) e η ( r ( x , y ) − b t ⋆ ( x ) ) , b t ⋆ ( x ) : = 1 η log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] . (112) \pi_{\eta,t}^{\star}(y \mid x) = \pi_{t}(y \mid x)e^{\eta(r(x,y)-b_{t}^{\star}(x))}, \qquad b_{t}^{\star}(x) := \frac{1}{\eta}\log\mathbb{E}_{y\sim\pi_{t}(\cdot\mid x)}\left[e^{\eta r(x,y)}\right]. \tag*{(112)} π η , t ⋆ ( y ∣ x ) = π t ( y ∣ x ) e η ( r ( x , y ) − b t ⋆ ( x )) , b t ⋆ ( x ) := η 1 log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] . (112)
For a candidate parameter θ \theta θ , define the importance weight and regularized reward,
w t π θ ( x , y ) : = π θ ( y ∣ x ) π t ( y ∣ x ) , ℓ θ ( x , y ) : = r ( x , y ) − 1 η log w t π θ ( x , y ) . w_{t}^{\pi_{\theta}}(x,y) := \frac{\pi_{\theta}(y\mid x)}{\pi_{t}(y\mid x)}, \qquad\ell_{\theta}(x,y) := r(x,y)-\frac{1}{\eta}\log w_{t}^{\pi_{\theta}}(x,y). w t π θ ( x , y ) := π t ( y ∣ x ) π θ ( y ∣ x ) , ℓ θ ( x , y ) := r ( x , y ) − η 1 log w t π θ ( x , y ) .
We also write
ϕ ~ θ ( x , y ) : = ϕ ( x , y ) − E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] , Σ θ : = E x ∼ μ , y ∼ π θ ( ⋅ ∣ x ) [ ϕ ~ θ ( x , y ) ϕ ~ θ ( x , y ) T ] . \widetilde{\phi}_{\theta}(x,y) := \phi(x,y)-\mathbb{E}_{y'\sim\pi_{\theta}(\cdot\mid x)}[\phi(x,y')], \qquad\Sigma_{\theta} := \mathbb{E}_{x\sim\mu,\,y\sim\pi_{\theta}(\cdot\mid x)}\left[\widetilde{\phi}_{\theta}(x,y)\widetilde{\phi}_{\theta}(x,y)^{\mathsf{T}}\right]. ϕ θ ( x , y ) := ϕ ( x , y ) − E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] , Σ θ := E x ∼ μ , y ∼ π θ ( ⋅ ∣ x ) [ ϕ θ ( x , y ) ϕ θ ( x , y ) T ] .
By Lemma B.14 , ϕ ~ θ = ∇ log π θ \widetilde{\phi}_{\theta} = \nabla\log\pi_{\theta} ϕ θ = ∇ log π θ .
We define the empirical version of the objective in Eq. (9) by
J ^ η , t ( π θ ) : = 1 N ∑ i = 1 N [ w t π θ ( x i , y i ) r i − 1 η ( w t π θ ( x i , y i ) log w t π θ ( x i , y i ) − w t π θ ( x i , y i ) + 1 ) ] . (113) \widehat{J}_{\eta,t}(\pi_{\theta}) := \frac{1}{N}\sum_{i=1}^{N}\left[w_{t}^{\pi_{\theta}}(x_i,y_i)r_i-\frac{1}{\eta}\left(w_{t}^{\pi_{\theta}}(x_i,y_i)\log w_{t}^{\pi_{\theta}}(x_i,y_i)-w_{t}^{\pi_{\theta}}(x_i,y_i)+1\right)\right]. \tag*{(113)} J η , t ( π θ ) := N 1 i = 1 ∑ N [ w t π θ ( x i , y i ) r i − η 1 ( w t π θ ( x i , y i ) log w t π θ ( x i , y i ) − w t π θ ( x i , y i ) + 1 ) ] . (113)
where D t = { ( x i , y i , r i ) } i = 1 N D_t=\{(x_i,y_i,r_i)\}_{i=1}^{N} D t = {( x i , y i , r i ) } i = 1 N is the batch drawn in Algorithm 1. Although Algorithm 1 does not explicitly use this objective, the gradient of Eq. (113) is exactly the gradient estimate used in Algorithm 1. We use J ^ η , t \widehat{J}_{\eta,t} J η , t in the proof to prove that the inner optimization loop converges relative to π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ on the empirical objective, which we then transfer to performance on the population objective J η , t J_{\eta,t} J η , t .
Supporting Lemmas
The supporting lemmas are organized as follows.
Lemma G.2 identifies the population objective gap with the KL divergence to π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ .
Lemma G.3 establishes realizability of π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ and defines its target parameter θ η , t ⋆ \theta_{\eta,t}^{\star} θ η , t ⋆ .
Lemma G.1, Lemma G.4 and Lemma G.5 establish properties of the gradient and gradient norm.
Lemma G.6 establishes the local concavity of the population RL objective.
Lemma G.7 bounds the smoothness of the empirical objective on a parameter ball.
Lemma G.8 and Lemma G.9 are concentration results for the empirical objective and functions of the empirical gradient.
Throughout this subsection, we fix an outer iteration t t t and condition on the history before its batch is drawn, so π t \pi_t π t is fixed. All probabilistic statements are over the batch D t \mathcal{D}_t D t drawn in Algorithm 1.
Lemma G.1. Let J ^ η , t \widehat{J}_{\eta,t} J η , t be the empirical objective in Eq. (113) . For every θ ∈ R d \theta\in\mathbb{R}^d θ ∈ R d ,
∇ J ^ η , t ( π θ ) = 1 N ∑ i = 1 N w t π θ ( x i , y i ) ℓ θ ( x i , y i ) ∇ log π θ ( y i ∣ x i ) . (114) \nabla\widehat{J}_{\eta,t}(\pi_\theta) = \frac{1}{N}\sum_{i=1}^{N} w_t^{\pi_\theta}(x_i,y_i)\ell_\theta(x_i,y_i)\nabla\log\pi_\theta(y_i \mid x_i). \tag*{(114)} ∇ J η , t ( π θ ) = N 1 i = 1 ∑ N w t π θ ( x i , y i ) ℓ θ ( x i , y i ) ∇ log π θ ( y i ∣ x i ) . (114) In particular, the estimate in Eq. (11) equals ∇ J ^ η , t ( π t k ) \nabla\widehat{J}_{\eta,t}(\pi_t^k) ∇ J η , t ( π t k ) at each inner iterate of Algorithm 1.
Proof of Lemma G.1. For a sample ( x , y ) (x,y) ( x , y ) , let F θ ( x , y ) F_\theta(x,y) F θ ( x , y ) denote the expression inside the brackets in Eq. (113) . Since π t \pi_t π t is fixed, ∇ w t π θ = w t π θ ∇ log π θ \nabla w_t^{\pi_\theta}=w_t^{\pi_\theta}\nabla\log\pi_\theta ∇ w t π θ = w t π θ ∇ log π θ . Differentiating gives
∇ F θ ( x , y ) = [ r ( x , y ) − 1 η ( log w t π θ ( x , y ) + 1 − 1 ) ] ∇ w t π θ ( x , y ) = w t π θ ( x , y ) ℓ θ ( x , y ) ϕ ~ θ ( x , y ) , (115) \begin{aligned}
\nabla F_\theta(x,y)=\left[r(x,y)-\frac{1}{\eta}\left(\log w_t^{\pi_\theta}(x,y)+1-1\right)\right]\nabla w_t^{\pi_\theta}(x,y) \\
&= w_t^{\pi_\theta}(x,y)\ell_\theta(x,y)\widetilde{\phi}_\theta(x,y), \tag*{(115)}
\end{aligned} ∇ F θ ( x , y ) = [ r ( x , y ) − η 1 ( log w t π θ ( x , y ) + 1 − 1 ) ] ∇ w t π θ ( x , y ) = w t π θ ( x , y ) ℓ θ ( x , y ) ϕ θ ( x , y ) , (115)
where ϕ ~ θ = ∇ log π θ \widetilde{\phi}_\theta=\nabla\log\pi_\theta ϕ θ = ∇ log π θ by Lemma B.14 . Averaging over D t \mathcal{D}_t D t proves Eq. (114) ; substituting θ = θ t k \theta=\theta_t^k θ = θ t k gives the estimate in Eq. (11) . □
Lemma G.2. For any η > 0 \eta>0 η > 0 , π t > 0 \pi_t>0 π t > 0 , and policy π \pi π ,
J η , t ( π η , t ⋆ ) − J η , t ( π ) = 1 η D K L ( π ∥ π η , t ⋆ ) . (116) J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi)=\frac{1}{\eta}D_{\mathrm{KL}}(\pi\mathbin\Vert\pi_{\eta,t}^{\star}). \tag*{(116)} J η , t ( π η , t ⋆ ) − J η , t ( π ) = η 1 D KL ( π ∥ π η , t ⋆ ) . (116)
Proof of Lemma G.2 . The closed form of π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ in Lemma B.7 gives log ( π η , t ⋆ ( y ∣ x ) / π t ( y ∣ x ) ) = η ( r ( x , y ) − b t ⋆ ( x ) ) \log(\pi_{\eta,t}^{\star}(y\mid x)/\pi_t(y\mid x))=\eta(r(x,y)-b_t^{\star}(x)) log ( π η , t ⋆ ( y ∣ x ) / π t ( y ∣ x )) = η ( r ( x , y ) − b t ⋆ ( x )) . Hence
D K L ( π ∥ π η , t ⋆ ) = E π [ log π ( y ∣ x ) π t ( y ∣ x ) − η r ( x , y ) + η b t ⋆ ( x ) ] = − η J η , t ( π ) + η E x ∼ μ [ b t ⋆ ( x ) ] . \begin{aligned}
D_{\mathrm{KL}}(\pi\mathbin\Vert\pi_{\eta,t}^{\star})=\mathbb{E}_{\pi}\left[\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-\eta r(x,y)+\eta b_t^{\star}(x)\right] \\
&=-\eta J_{\eta,t}(\pi)+\eta\mathbb{E}_{x\sim\mu}[b_t^{\star}(x)].
\end{aligned} D KL ( π ∥ π η , t ⋆ ) = E π [ log π t ( y ∣ x ) π ( y ∣ x ) − η r ( x , y ) + η b t ⋆ ( x ) ] = − η J η , t ( π ) + η E x ∼ μ [ b t ⋆ ( x )] .
Substituting π = π η , t ⋆ \pi=\pi_{\eta,t}^{\star} π = π η , t ⋆ gives J η , t ( π η , t ⋆ ) = E x ∼ μ [ b t ⋆ ( x ) ] J_{\eta,t}(\pi_{\eta,t}^{\star})=\mathbb{E}_{x\sim\mu}[b_t^{\star}(x)] J η , t ( π η , t ⋆ ) = E x ∼ μ [ b t ⋆ ( x )] , proving the identity. □
Lemma G.3. Suppose that Assumption 2.1 holds, and let Π Θ \Pi_\Theta Π Θ be the softmax linear policy class. For every outer iteration t t t and η > 0 \eta>0 η > 0 , define the target parameter
θ η , t ⋆ : = θ t + η θ ⋆ . \theta_{\eta,t}^{\star}:=\theta_t+\eta\theta_{\star}. θ η , t ⋆ := θ t + η θ ⋆ . Then π η , t ⋆ = π θ η , t ⋆ ∈ Π Θ \pi_{\eta,t}^{\star}=\pi_{\theta_{\eta,t}^{\star}}\in\Pi_\Theta π η , t ⋆ = π θ η , t ⋆ ∈ Π Θ is realizable.
Proof of Lemma G.3 . Linear realizability and the softmax formula give
π θ η , t ⋆ ( y ∣ x ) = exp ( ϕ ( x , y ) T θ t + η r ( x , y ) ) ∑ y ′ exp ( ϕ ( x , y ′ ) T θ t + η r ( x , y ′ ) ) = π η , t ⋆ ( y ∣ x ) . \pi_{\theta_{\eta,t}^{\star}}(y\mid x)=\frac{\exp(\phi(x,y)^{\mathsf{T}}\theta_t+\eta r(x,y))}{\sum_{y'}\exp(\phi(x,y')^{\mathsf{T}}\theta_t+\eta r(x,y'))}=\pi_{\eta,t}^{\star}(y\mid x). π θ η , t ⋆ ( y ∣ x ) = ∑ y ′ exp ( ϕ ( x , y ′ ) T θ t + η r ( x , y ′ )) exp ( ϕ ( x , y ) T θ t + η r ( x , y )) = π η , t ⋆ ( y ∣ x ) .
□
Lemma G.4. For every η > 0 \eta> 0 η > 0 and θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d , with θ η , t ⋆ \theta^{\star}_{\eta,t} θ η , t ⋆ defined in Lemma G.3 ,
∇ J η , t ( π θ ) = 1 η Σ θ ( θ η , t ⋆ − θ ) . \nabla J_{\eta,t}(\pi_{\theta}) = \frac{1}{\eta}\Sigma_{\theta}(\theta^{\star}_{\eta,t}-\theta). ∇ J η , t ( π θ ) = η 1 Σ θ ( θ η , t ⋆ − θ ) . Therefore,
⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ = 1 η ∥ θ η , t ⋆ − θ ∥ Σ θ 2 . (117) \left\langle\nabla J_{\eta,t}(\pi_{\theta}),\theta^{\star}_{\eta,t}-\theta\right\rangle= \frac{1}{\eta}\left\|\theta^{\star}_{\eta,t}-\theta\right\|_{\Sigma_{\theta}}^{2}. \tag*{(117)} ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ = η 1 θ η , t ⋆ − θ Σ θ 2 . (117)
Proof of Lemma G.4 . We separate the part of the regularized reward that varies with the response from the part that depends only on the prompt.
ℓ θ ( x , y ) = 1 η ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) + 1 η log E y ′ ∼ π t ( ⋅ ∣ x ) [ e ϕ ( x , y ′ ) ⊤ ( θ − θ t ) ] . \ell_{\theta}(x,y) = \frac{1}{\eta}\phi(x,y)^{\top}(\theta^{\star}_{\eta,t}-\theta) + \frac{1}{\eta}\log\mathbb{E}_{y'\sim\pi_{t}(\cdot\mid x)}\left[e^{\phi(x,y')^{\top}(\theta-\theta_{t})}\right]. ℓ θ ( x , y ) = η 1 ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) + η 1 log E y ′ ∼ π t ( ⋅ ∣ x ) [ e ϕ ( x , y ′ ) ⊤ ( θ − θ t ) ] .
Since the score has conditional mean zero under π θ \pi_{\theta} π θ , the second term makes no contribution to the population gradient. Thus
∇ J η , t ( π θ ) = E π θ [ ϕ ~ θ ( x , y ) ℓ θ ( x , y ) ] = 1 η E π θ [ ϕ ~ θ ( x , y ) ϕ ~ θ ( x , y ) ⊤ ] ( θ η , t ⋆ − θ ) = 1 η Σ θ ( θ η , t ⋆ − θ ) . \begin{aligned}
\nabla J_{\eta,t}(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}}\left[\widetilde{\phi}_{\theta}(x,y)\ell_{\theta}(x,y)\right] \\
&= \frac{1}{\eta}\mathbb{E}_{\pi_{\theta}}\left[\widetilde{\phi}_{\theta}(x,y)\widetilde{\phi}_{\theta}(x,y)^{\top}\right](\theta^{\star}_{\eta,t}-\theta) = \frac{1}{\eta}\Sigma_{\theta}(\theta^{\star}_{\eta,t}-\theta).
\end{aligned} ∇ J η , t ( π θ ) = E π θ [ ϕ θ ( x , y ) ℓ θ ( x , y ) ] = η 1 E π θ [ ϕ θ ( x , y ) ϕ θ ( x , y ) ⊤ ] ( θ η , t ⋆ − θ ) = η 1 Σ θ ( θ η , t ⋆ − θ ) .
Taking the inner product with θ η , t ⋆ − θ \theta^{\star}_{\eta,t}-\theta θ η , t ⋆ − θ proves the alignment identity.
Lemma G.5. For every η > 0 \eta> 0 η > 0 and θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d ,
∥ ∇ J η , t ( π θ ) ∥ 2 2 ≤ D ϕ 2 η ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ . (118) \left\|\nabla J_{\eta,t}(\pi_{\theta})\right\|_{2}^{2} \leq\frac{D_{\phi}^{2}}{\eta}\left\langle\nabla J_{\eta,t}(\pi_{\theta}),\theta^{\star}_{\eta,t}-\theta\right\rangle. \tag*{(118)} ∥ ∇ J η , t ( π θ ) ∥ 2 2 ≤ η D ϕ 2 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ . (118)
Proof of Lemma G.5 . By Lemma B.15 , 0 ⪯ Σ θ ⪯ D ϕ 2 I 0 \preceq\Sigma_{\theta} \preceq D_{\phi}^{2}I 0 ⪯ Σ θ ⪯ D ϕ 2 I . Combining this with Lemma G.4 gives
∥ ∇ J η , t ( π θ ) ∥ 2 2 = 1 η 2 ( θ η , t ⋆ − θ ) ⊤ Σ θ 2 ( θ η , t ⋆ − θ ) ≤ D ϕ 2 η 2 ∥ θ η , t ⋆ − θ ∥ Σ θ 2 = D ϕ 2 η ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ . \begin{aligned}
\left\|\nabla J_{\eta,t}(\pi_{\theta})\right\|_{2}^{2} = \frac{1}{\eta^{2}}(\theta^{\star}_{\eta,t}-\theta)^{\top}\Sigma_{\theta}^{2}(\theta^{\star}_{\eta,t}-\theta) \\
&\leq\frac{D_{\phi}^{2}}{\eta^{2}}\left\|\theta^{\star}_{\eta,t}-\theta\right\|_{\Sigma_{\theta}}^{2} = \frac{D_{\phi}^{2}}{\eta}\left\langle\nabla J_{\eta,t}(\pi_{\theta}),\theta^{\star}_{\eta,t}-\theta\right\rangle.
\end{aligned} ∥ ∇ J η , t ( π θ ) ∥ 2 2 = η 2 1 ( θ η , t ⋆ − θ ) ⊤ Σ θ 2 ( θ η , t ⋆ − θ ) ≤ η 2 D ϕ 2 θ η , t ⋆ − θ Σ θ 2 = η D ϕ 2 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ .
This bound controls the squared gradient term in the recursion for the squared distance to θ η , t ⋆ \theta^{\star}_{\eta,t} θ η , t ⋆ in Lemma G.10.
Lemma G.6 (Adapted from Lemma B.3). For every η > 0 \eta> 0 η > 0 and θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d ,
J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ≤ 1 2 ∥ π η , t ⋆ π θ ∥ ∞ ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ . J_{\eta,t}(\pi^{\star}_{\eta,t}) - J_{\eta,t}(\pi_{\theta}) \leq\frac{1}{2}\left\|\frac{\pi^{\star}_{\eta,t}}{\pi_{\theta}}\right\|_{\infty}\left\langle\nabla J_{\eta,t}(\pi_{\theta}),\theta^{\star}_{\eta,t}-\theta\right\rangle. J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ≤ 2 1 π θ π η , t ⋆ ∞ ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ . For a parameter radius R > 0 R > 0 R > 0 , define W R : = e D ϕ R W_{R} := e^{D_{\phi}R} W R := e D ϕ R . In particular, if ∥ θ − θ η , t ⋆ ∥ 2 ≤ R \left\|\theta-\theta^{\star}_{\eta,t}\right\|_{2} \leq R θ − θ η , t ⋆ 2 ≤ R , then
J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ≤ W R 2 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ . (119) J_{\eta,t}(\pi^{\star}_{\eta,t}) - J_{\eta,t}(\pi_{\theta}) \leq\frac{W_{R}}{2}\left\langle\nabla J_{\eta,t}(\pi_{\theta}),\theta^{\star}_{\eta,t}-\theta\right\rangle. \tag*{(119)} J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ≤ 2 W R ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ . (119)
Proof of Lemma G.6 .
By Eq. (116) , it suffices to bound D K L ( π θ ∥ π η , t ⋆ ) D_{\mathrm{KL}}(\pi_{\theta}\|\pi^{\star}_{\eta,t}) D KL ( π θ ∥ π η , t ⋆ ) . We express this divergence as an integral of feature variances along the segment from θ \theta θ to θ η , t ⋆ \theta^{\star}_{\eta,t} θ η , t ⋆ , then compare them with the variance under π θ \pi_{\theta} π θ . For a fixed prompt x x x , let F x ( s ) : = log E y ∼ π θ ( ⋅ ∣ x ) [ e s ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) ] F_{x}(s) := \log\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[e^{s\phi(x,y)^{\top}(\theta^{\star}_{\eta,t}-\theta)}\right] F x ( s ) := log E y ∼ π θ ( ⋅ ∣ x ) [ e s ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) ] . Differentiating twice and applying Taylor’s theorem gives
D K L ( π θ ( ⋅ ∣ x ) ∥ π η , t ⋆ ( ⋅ ∣ x ) ) = F x ( 1 ) − F x ( 0 ) − F x ′ ( 0 ) = ∫ 0 1 ( 1 − s ) V y ∼ π θ + s ( θ η , t ⋆ − θ ) ( ⋅ ∣ x ) ( ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) ) d s . \begin{aligned}
D_{\mathrm{KL}}\left(\pi_{\theta}(\cdot\mid x)\,\middle\|\,\pi^{\star}_{\eta,t}(\cdot\mid x)\right) = F_{x}(1)-F_{x}(0)-F'_{x}(0) \\
&= \int_{0}^{1}(1-s)\operatorname{V}_{y\sim\pi_{\theta+s(\theta^{\star}_{\eta,t}-\theta)}(\cdot\mid x)}\left(\phi(x,y)^{\top}(\theta^{\star}_{\eta,t}-\theta)\right)\,ds.
\end{aligned} D KL ( π θ ( ⋅ ∣ x ) π η , t ⋆ ( ⋅ ∣ x ) ) = F x ( 1 ) − F x ( 0 ) − F x ′ ( 0 ) = ∫ 0 1 ( 1 − s ) V y ∼ π θ + s ( θ η , t ⋆ − θ ) ( ⋅ ∣ x ) ( ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) ) d s .
To compare these variances with the variance under π θ \pi_\theta π θ , write m x : = sup y ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) m_x := \sup_y \phi(x,y)^\top(\theta^\star_{\eta,t}-\theta) m x := sup y ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) . The softmax formula gives, for 0 ≤ s ≤ 1 0 \le s \le1 0 ≤ s ≤ 1 ,
∥ π θ + s ( θ η , t ⋆ − θ ) ( ⋅ ∣ x ) π θ ( ⋅ ∣ x ) ∥ ∞ = 1 E y ∼ π θ ( ⋅ ∣ x ) [ e s ( ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) − m x ) ] ≤ 1 E y ∼ π θ ( ⋅ ∣ x ) [ e ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) − m x ] = ∥ π η , t ⋆ ( ⋅ ∣ x ) π θ ( ⋅ ∣ x ) ∥ ∞ . \left\|\frac{\pi_{\theta+s(\theta^\star_{\eta,t}-\theta)}(\cdot\mid x)}{\pi_\theta(\cdot\mid x)}\right\|_\infty
= \frac{1}{\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\left[e^{s(\phi(x,y)^\top(\theta^\star_{\eta,t}-\theta)-m_x)}\right]}
\le\frac{1}{\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\left[e^{\phi(x,y)^\top(\theta^\star_{\eta,t}-\theta)-m_x}\right]}
= \left\|\frac{\pi^\star_{\eta,t}(\cdot\mid x)}{\pi_\theta(\cdot\mid x)}\right\|_\infty. π θ ( ⋅ ∣ x ) π θ + s ( θ η , t ⋆ − θ ) ( ⋅ ∣ x ) ∞ = E y ∼ π θ ( ⋅ ∣ x ) [ e s ( ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) − m x ) ] 1 ≤ E y ∼ π θ ( ⋅ ∣ x ) [ e ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) − m x ] 1 = π θ ( ⋅ ∣ x ) π η , t ⋆ ( ⋅ ∣ x ) ∞ .
where the inequality uses ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) − m x ≤ 0 \phi(x,y)^\top(\theta^\star_{\eta,t}-\theta)-m_x\le0 ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) − m x ≤ 0 . A variance is no larger than the second moment about any fixed center. Centering at the mean under π θ \pi_\theta π θ and changing measure therefore gives
V y ∼ π θ + s ( θ η , t ⋆ − θ ) ( ⋅ ∣ x ) ( ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) ) ≤ ∥ π η , t ⋆ ( ⋅ ∣ x ) π θ ( ⋅ ∣ x ) ∥ ∞ E y ∼ π θ ( ⋅ ∣ x ) [ ( ϕ ~ θ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) ) 2 ] . \mathbb{V}_{y\sim\pi_{\theta+s(\theta^\star_{\eta,t}-\theta)}(\cdot\mid x)}\left(\phi(x,y)^\top(\theta^\star_{\eta,t}-\theta)\right)
\le\left\|\frac{\pi^\star_{\eta,t}(\cdot\mid x)}{\pi_\theta(\cdot\mid x)}\right\|_\infty
\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\left[\left(\widetilde{\phi}_\theta(x,y)^\top(\theta^\star_{\eta,t}-\theta)\right)^2\right]. V y ∼ π θ + s ( θ η , t ⋆ − θ ) ( ⋅ ∣ x ) ( ϕ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) ) ≤ π θ ( ⋅ ∣ x ) π η , t ⋆ ( ⋅ ∣ x ) ∞ E y ∼ π θ ( ⋅ ∣ x ) [ ( ϕ θ ( x , y ) ⊤ ( θ η , t ⋆ − θ ) ) 2 ] .
Integrating over s s s and averaging over x x x , we conclude that
J η , t ( π η , t ⋆ ) − J η , t ( π θ ) = 1 η D K L ( π θ ∥ π η , t ⋆ ) ≤ 1 2 η ∥ π η , t ⋆ π θ ∥ ∞ ∥ θ η , t ⋆ − θ ∥ Σ θ 2 = 1 2 ∥ π η , t ⋆ π θ ∥ ∞ ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ , \begin{aligned}
J_{\eta,t}(\pi^\star_{\eta,t})-J_{\eta,t}(\pi_\theta)
= \frac{1}{\eta}D_{\mathrm{KL}}(\pi_\theta\Vert\pi^\star_{\eta,t}) \\
&\le\frac{1}{2\eta}\left\|\frac{\pi^\star_{\eta,t}}{\pi_\theta}\right\|_\infty\|\theta^\star_{\eta,t}-\theta\|_{\Sigma_\theta}^2 \\
&= \frac{1}{2}\left\|\frac{\pi^\star_{\eta,t}}{\pi_\theta}\right\|_\infty
\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle,
\end{aligned} J η , t ( π η , t ⋆ ) − J η , t ( π θ ) = η 1 D KL ( π θ ∥ π η , t ⋆ ) ≤ 2 η 1 π θ π η , t ⋆ ∞ ∥ θ η , t ⋆ − θ ∥ Σ θ 2 = 2 1 π θ π η , t ⋆ ∞ ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ,
using Lemma G.2 and Eq. (117) in the first and last steps. If ∥ θ − θ η , t ⋆ ∥ 2 ≤ R \|\theta-\theta^\star_{\eta,t}\|_2\le R ∥ θ − θ η , t ⋆ ∥ 2 ≤ R , then ∥ π η , t ⋆ / π θ ∥ ∞ ≤ W R \|\pi^\star_{\eta,t}/\pi_\theta\|_\infty\le W_R ∥ π η , t ⋆ / π θ ∥ ∞ ≤ W R by Lemma B.12 , proving Eq. (119) .
Lemma G.7. Let J ^ η , t \widehat{J}_{\eta,t} J η , t be defined in Eq. (113) . Fix η > 0 \eta>0 η > 0 and a parameter radius R > 0 R>0 R > 0 , and define the parameter ball
B R : = { θ ∈ R d : ∥ θ − θ η , t ⋆ ∥ 2 ≤ R } . \mathcal{B}_R:=\left\{\theta\in\mathbb{R}^d:\|\theta-\theta^\star_{\eta,t}\|_2\le R\right\}. B R := { θ ∈ R d : ∥ θ − θ η , t ⋆ ∥ 2 ≤ R } . For every θ ∈ B R \theta\in\mathcal{B}_R θ ∈ B R ,
w t π θ ( x , y ) : = π θ ( y ∣ x ) π t ( y ∣ x ) ≤ W η , R : = W R e η B ϕ B ⋆ , ∀ ( x , y ) , (120) w_t^{\pi_\theta}(x,y):=\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}\le W_{\eta,R}:=W_R e^{\eta B_\phi B_\star},\qquad\forall(x,y), \tag*{(120)} w t π θ ( x , y ) := π t ( y ∣ x ) π θ ( y ∣ x ) ≤ W η , R := W R e η B ϕ B ⋆ , ∀ ( x , y ) , (120) where W R = e D ϕ R W_R=e^{D_\phi R} W R = e D ϕ R is defined in Lemma G.6 .
For every batch, J ^ η , t \widehat{J}_{\eta,t} J η , t is L ^ η , R \widehat{L}_{\eta,R} L η , R -smooth on B R \mathcal{B}_R B R . In particular, for any θ , θ ′ ∈ B R \theta,\theta'\in\mathcal{B}_R θ , θ ′ ∈ B R ,
J ^ η , t ( π θ ′ ) ≥ J ^ η , t ( π θ ) + ⟨ ∇ J ^ η , t ( π θ ) , θ ′ − θ ⟩ − L ^ η , R 2 ∥ θ ′ − θ ∥ 2 , (121) \widehat{J}_{\eta,t}(\pi_{\theta'})\ge\widehat{J}_{\eta,t}(\pi_\theta)+\left\langle\nabla\widehat{J}_{\eta,t}(\pi_\theta),\theta'-\theta\right\rangle-\frac{\widehat{L}_{\eta,R}}{2}\|\theta'-\theta\|^2, \tag*{(121)} J η , t ( π θ ′ ) ≥ J η , t ( π θ ) + ⟨ ∇ J η , t ( π θ ) , θ ′ − θ ⟩ − 2 L η , R ∥ θ ′ − θ ∥ 2 , (121) where
L ^ η , R : = D ϕ 2 η W η , R ( 1 + 5 4 log W η , R ) . (122) \widehat{L}_{\eta,R}:=\frac{D_\phi^2}{\eta}W_{\eta,R}\left(1+\frac{5}{4}\log W_{\eta,R}\right). \tag*{(122)} L η , R := η D ϕ 2 W η , R ( 1 + 4 5 log W η , R ) . (122)
Proof of Lemma G.7 .
Fix θ ∈ B R \theta\in\mathcal{B}_R θ ∈ B R . Since 0 ≤ r ( x , y ) ≤ B ϕ B ⋆ 0\le r(x,y)\le B_\phi B_\star 0 ≤ r ( x , y ) ≤ B ϕ B ⋆ , we have 0 ≤ b t ⋆ ( x ) ≤ B ϕ B ⋆ 0\le b_t^\star(x)\le B_\phi B_\star 0 ≤ b t ⋆ ( x ) ≤ B ϕ B ⋆ . The identity π η , t ⋆ / π t = e η ( r − b t ⋆ ) \pi^\star_{\eta,t}/\pi_t=e^{\eta(r-b_t^\star)} π η , t ⋆ / π t = e η ( r − b t ⋆ ) and Lemma B.12 then give
w t π θ ( x , y ) = π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) π θ ( y ∣ x ) π η , t ⋆ ( y ∣ x ) = π θ ( y ∣ x ) π η , t ⋆ ( y ∣ x ) e η ( r ( x , y ) − b t ⋆ ( x ) ) ≤ e D ϕ R + η B ϕ B ⋆ = W η , R , w_t^{\pi_\theta}(x,y)=\frac{\pi^\star_{\eta,t}(y\mid x)}{\pi_t(y\mid x)}\frac{\pi_\theta(y\mid x)}{\pi^\star_{\eta,t}(y\mid x)}
=\frac{\pi_\theta(y\mid x)}{\pi^\star_{\eta,t}(y\mid x)}e^{\eta(r(x,y)-b_t^\star(x))}
\le e^{D_\phi R+\eta B_\phi B_\star}=W_{\eta,R}, w t π θ ( x , y ) = π t ( y ∣ x ) π η , t ⋆ ( y ∣ x ) π η , t ⋆ ( y ∣ x ) π θ ( y ∣ x ) = π η , t ⋆ ( y ∣ x ) π θ ( y ∣ x ) e η ( r ( x , y ) − b t ⋆ ( x )) ≤ e D ϕ R + η B ϕ B ⋆ = W η , R ,
∣ ℓ θ ( x , y ) ∣ = ∣ b t ⋆ ( x ) + 1 η log π η , t ⋆ ( y ∣ x ) π θ ( y ∣ x ) ∣ ≤ B ϕ B ⋆ + D ϕ R η = log W η , R η . |\ell_\theta(x,y)|=\left|b_t^\star(x)+\frac{1}{\eta}\log\frac{\pi^\star_{\eta,t}(y\mid x)}{\pi_\theta(y\mid x)}\right|
\le B_\phi B_\star+\frac{D_\phi R}{\eta}
=\frac{\log W_{\eta,R}}{\eta}. ∣ ℓ θ ( x , y ) ∣ = b t ⋆ ( x ) + η 1 log π θ ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ≤ B ϕ B ⋆ + η D ϕ R = η log W η , R .
Let F θ ( x , y ) F_\theta(x,y) F θ ( x , y ) denote the expression inside the brackets in Eq. (113) . Its gradient is given by Eq. (115) . Since ∇ w t π θ = w t π θ ϕ ~ θ \nabla w_t^{\pi_\theta}=w_t^{\pi_\theta}\widetilde{\phi}_\theta ∇ w t π θ = w t π θ ϕ θ and ∇ ℓ θ = − ϕ ~ θ / η \nabla\ell_\theta=-\widetilde{\phi}_\theta/\eta ∇ ℓ θ = − ϕ θ / η , differentiating once more gives
∇ 2 F θ ( x , y ) = w t π θ ( x , y ) [ ℓ θ ( x , y ) ∇ 2 log π θ ( y ∣ x ) + ( ℓ θ ( x , y ) − 1 η ) ϕ ~ θ ( x , y ) ϕ ~ θ ( x , y ) ⊤ ] . \nabla^2 F_\theta(x,y)=w_t^{\pi_\theta}(x,y)\left[\ell_\theta(x,y)\nabla^2\log\pi_\theta(y\mid x)+\left(\ell_\theta(x,y)-\frac{1}{\eta}\right)\widetilde{\phi}_\theta(x,y)\widetilde{\phi}_\theta(x,y)^\top\right]. ∇ 2 F θ ( x , y ) = w t π θ ( x , y ) [ ℓ θ ( x , y ) ∇ 2 log π θ ( y ∣ x ) + ( ℓ θ ( x , y ) − η 1 ) ϕ θ ( x , y ) ϕ θ ( x , y ) ⊤ ] .
By Lemma B.16, for every unit vector v v v ,
− v ⊤ ∇ 2 log π θ ( y ∣ x ) v = V y ′ ∼ π θ ( ⋅ ∣ x ) ( v ⊤ ϕ ( x , y ′ ) ) ≤ D ϕ 2 4 . -v^\top\nabla^2\log\pi_\theta(y\mid x)v=\mathbb{V}_{y'\sim\pi_\theta(\cdot\mid x)}\left(v^\top\phi(x,y')\right)\leq\frac{D_\phi^2}{4}. − v ⊤ ∇ 2 log π θ ( y ∣ x ) v = V y ′ ∼ π θ ( ⋅ ∣ x ) ( v ⊤ ϕ ( x , y ′ ) ) ≤ 4 D ϕ 2 .
The last inequality follows by bounding variance by the squared deviation from the midpoint of an interval of width at most D ϕ D_\phi D ϕ . Together with ∥ ϕ ~ θ ( x , y ) ∥ 2 ≤ D ϕ \left\|\widetilde{\phi}_\theta(x,y)\right\|_2\leq D_\phi ϕ θ ( x , y ) 2 ≤ D ϕ , this yields
∥ ∇ 2 F θ ( x , y ) ∥ o p ≤ w t π θ ( x , y ) [ D ϕ 2 4 ∣ ℓ θ ( x , y ) ∣ + D ϕ 2 ( ∣ ℓ θ ( x , y ) ∣ + 1 η ) ] ≤ D ϕ 2 η W η , R ( 1 + 5 4 log W η , R ) = L ^ η , R . \begin{aligned}
\left\|\nabla^2F_\theta(x,y)\right\|_{\mathrm{op}}\leq w_t^{\pi_\theta}(x,y)\left[\frac{D_\phi^2}{4}\lvert\ell_\theta(x,y)\rvert+D_\phi^2\left(\lvert\ell_\theta(x,y)\rvert+\frac{1}{\eta}\right)\right] \\
&\leq\frac{D_\phi^2}{\eta}W_{\eta,R}\left(1+\frac{5}{4}\log W_{\eta,R}\right)=\widehat{L}_{\eta,R}.
\end{aligned} ∇ 2 F θ ( x , y ) op ≤ w t π θ ( x , y ) [ 4 D ϕ 2 ∣ ℓ θ ( x , y )∣ + D ϕ 2 ( ∣ ℓ θ ( x , y )∣ + η 1 ) ] ≤ η D ϕ 2 W η , R ( 1 + 4 5 log W η , R ) = L η , R .
Averaging over the batch gives the same bound on the Hessian of J ^ η , t \widehat{J}_{\eta,t} J η , t throughout B R \mathcal{B}_R B R . For θ , θ ′ ∈ B R \theta,\theta'\in\mathcal{B}_R θ , θ ′ ∈ B R , the segment joining them lies in B R \mathcal{B}_R B R by convexity, so Taylor’s theorem proves Eq. (121) .
Lemma G.8. Let J ^ η , t \widehat{J}_{\eta,t} J η , t be defined in Eq. (113) . Fix η , R > 0 \eta,R>0 η , R > 0 , N ≥ 1 N\geq1 N ≥ 1 , and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and let W η , R W_{\eta,R} W η , R and L ^ η , R \widehat{L}_{\eta,R} L η , R be as in Eqs. (120) and (122) . Define
ε d i r : = W η , R log W η , R η N ( 2 log W η , R + 2 3 D ϕ R ) ( d log ( 1 + 2 N ) + log 4 δ ) + 9 R 4 N ( R L ^ η , R + D ϕ W η , R log W η , R η ) , \varepsilon_{\mathrm{dir}}:=\frac{W_{\eta,R}\log W_{\eta,R}}{\eta N}\left(2\log W_{\eta,R}+\frac{2}{3}D_\phi R\right)\left(d\log(1+2N)+\log\frac{4}{\delta}\right)
+\frac{9R}{4N}\left(R\widehat{L}_{\eta,R}+\frac{D_\phi W_{\eta,R}\log W_{\eta,R}}{\eta}\right), ε dir := η N W η , R log W η , R ( 2 log W η , R + 3 2 D ϕ R ) ( d log ( 1 + 2 N ) + log δ 4 ) + 4 N 9 R ( R L η , R + η D ϕ W η , R log W η , R ) , ε g r a d : = 2 D ϕ W η , R log W η , R η 2 N ( d log ( 5 ( 1 + 2 N ) ) + log 4 δ ) + 2 R L ^ η , R N . (123) \varepsilon_{\mathrm{grad}}:=\frac{2D_\phi W_{\eta,R}\log W_{\eta,R}}{\eta}\sqrt{\frac{2}{N}\left(d\log(5(1+2N))+\log\frac{4}{\delta}\right)}+\frac{2R\widehat{L}_{\eta,R}}{N}. \tag*{(123)} ε grad := η 2 D ϕ W η , R log W η , R N 2 ( d log ( 5 ( 1 + 2 N )) + log δ 4 ) + N 2 R L η , R . (123) With probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every θ \theta θ with ∥ θ − θ η , t ⋆ ∥ 2 ≤ R \left\|\theta-\theta_{\eta,t}^\star\right\|_2\leq R θ − θ η , t ⋆ 2 ≤ R ,
∣ ⟨ ∇ J ^ η , t ( π θ ) − ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ∣ ≤ 1 4 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + ε d i r , (124) \left|\left\langle\nabla\widehat{J}_{\eta,t}(\pi_\theta)-\nabla J_{\eta,t}(\pi_\theta),\theta_{\eta,t}^\star-\theta\right\rangle\right|\leq\frac{1}{4}\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta_{\eta,t}^\star-\theta\right\rangle+\varepsilon_{\mathrm{dir}}, \tag*{(124)} ⟨ ∇ J η , t ( π θ ) − ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ≤ 4 1 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + ε dir , (124) ∥ ∇ J ^ η , t ( π θ ) − ∇ J η , t ( π θ ) ∥ 2 ≤ ε g r a d . (125) \left\|\nabla\widehat{J}_{\eta,t}(\pi_\theta)-\nabla J_{\eta,t}(\pi_\theta)\right\|_2\leq\varepsilon_{\mathrm{grad}}. \tag*{(125)} ∇ J η , t ( π θ ) − ∇ J η , t ( π θ ) 2 ≤ ε grad . (125)
Proof of Lemma G.8. We first control the gradient error in the direction θ η , t ⋆ − θ \theta_{\eta,t}^\star-\theta θ η , t ⋆ − θ , then bound its Euclidean norm on the same parameter ball. By Eq. (115) , the contribution of one sample to the empirical gradient is
Z θ ( x , y ) : = w t π θ ( x , y ) ℓ θ ( x , y ) ϕ ~ θ ( x , y ) . Z_\theta(x,y):=w_t^{\pi_\theta}(x,y)\ell_\theta(x,y)\widetilde{\phi}_\theta(x,y). Z θ ( x , y ) := w t π θ ( x , y ) ℓ θ ( x , y ) ϕ θ ( x , y ) .
Its expectation under π t \pi_t π t is ∇ J η , t ( π θ ) \nabla J_{\eta,t}(\pi_\theta) ∇ J η , t ( π θ ) , while its average over D t \mathcal{D}_t D t is ∇ J ^ η , t ( π θ ) \nabla\widehat{J}_{\eta,t}(\pi_\theta) ∇ J η , t ( π θ ) . The bounds in Lemma G.7 and its proof give, throughout the ball,
∥ Z θ ( x , y ) ∥ 2 ≤ D ϕ W η , R log W η , R η , (126) \left\|Z_\theta(x,y)\right\|_2\leq\frac{D_\phi W_{\eta,R}\log W_{\eta,R}}{\eta}, \tag*{(126)} ∥ Z θ ( x , y ) ∥ 2 ≤ η D ϕ W η , R log W η , R , (126)
∥ Z θ ( x , y ) − Z θ ′ ( x , y ) ∥ 2 ≤ L ^ η , R ∥ θ − θ ′ ∥ 2 . (127) \left\|Z_\theta(x,y)-Z_{\theta'}(x,y)\right\|_2\leq\widehat{L}_{\eta,R}\left\|\theta-\theta'\right\|_2. \tag*{(127)} ∥ Z θ ( x , y ) − Z θ ′ ( x , y ) ∥ 2 ≤ L η , R ∥ θ − θ ′ ∥ 2 . (127)
Changing measure to π θ \pi_\theta π θ bounds the second moment of this directional contribution in terms of the population gradient’s alignment with θ η , t ⋆ − θ \theta^\star_{\eta,t}-\theta θ η , t ⋆ − θ .
E π θ [ ⟨ Z θ ( x , y ) , θ η , t ⋆ − θ ⟩ 2 ] = E π θ [ ( w η , R π θ ( x , y ) ℓ θ ( x , y ) ) 2 ⟨ ϕ ~ θ ( x , y ) , θ η , t ⋆ − θ ⟩ 2 ] ≤ W η , R ( log W η , R ) 2 η 2 ∥ θ η , t ⋆ − θ ∥ Σ θ 2 = W η , R ( log W η , R ) 2 η ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ . \begin{aligned}
\mathbb{E}_{\pi_\theta}\left[\left\langle Z_\theta(x,y),\theta^\star_{\eta,t}-\theta\right\rangle^2\right]
=
\mathbb{E}_{\pi_\theta}\left[\left(w_{\eta,R}^{\pi_\theta}(x,y)\ell_\theta(x,y)\right)^2\left\langle\widetilde{\phi}_\theta(x,y),\theta^\star_{\eta,t}-\theta\right\rangle^2\right] \\
&\leq\frac{W_{\eta,R}(\log W_{\eta,R})^2}{\eta^2}\left\|\theta^\star_{\eta,t}-\theta\right\|_{\Sigma_\theta}^2 \\
&= \frac{W_{\eta,R}(\log W_{\eta,R})^2}{\eta}\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle.
\end{aligned} E π θ [ ⟨ Z θ ( x , y ) , θ η , t ⋆ − θ ⟩ 2 ] = E π θ [ ( w η , R π θ ( x , y ) ℓ θ ( x , y ) ) 2 ⟨ ϕ θ ( x , y ) , θ η , t ⋆ − θ ⟩ 2 ] ≤ η 2 W η , R ( log W η , R ) 2 θ η , t ⋆ − θ Σ θ 2 = η W η , R ( log W η , R ) 2 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ .
where the last equality is Eq. (117) . Also, ∣ ⟨ Z θ ( x , y ) , θ η , t ⋆ − θ ⟩ ∣ ≤ R D ϕ W η , R log W η , R / η \left|\left\langle Z_\theta(x,y),\theta^\star_{\eta,t}-\theta\right\rangle\right|\leq RD_\phi W_{\eta,R}\log W_{\eta,R}/\eta ⟨ Z θ ( x , y ) , θ η , t ⋆ − θ ⟩ ≤ R D ϕ W η , R log W η , R / η . Thus, for a fixed θ \theta θ and u > 0 u>0 u > 0 , Bernstein’s inequality (Vershynin, [107 ] , Section 2.8) and a b ≤ a / 4 + b \sqrt{ab}\leq a/4+b ab ≤ a /4 + b give, with probability at least 1 − 2 e − u 1-2e^{-u} 1 − 2 e − u ,
∣ ⟨ ∇ J ^ η , t ( π θ ) − ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ∣ ≤ 2 W η , R ( log W η , R ) 2 u η N ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + 2 R D ϕ W η , R log W η , R 3 η N u ≤ 1 4 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + W η , R log W η , R η N ( 2 log W η , R + 2 3 R D ϕ ) u . (128) \begin{aligned}
\left|\left\langle\widehat{\nabla J}_{\eta,t}(\pi_\theta)-\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle\right|
\leq
\sqrt{\frac{2W_{\eta,R}(\log W_{\eta,R})^2u}{\eta N}\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle+\frac{2RD_\phi W_{\eta,R}\log W_{\eta,R}}{3\eta N}u} \\
&\leq\frac{1}{4}\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle+\frac{W_{\eta,R}\log W_{\eta,R}}{\eta N}\left(2\log W_{\eta,R}+\frac{2}{3}RD_\phi\right)u. \tag*{(128)}
\end{aligned} ⟨ ∇ J η , t ( π θ ) − ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ≤ η N 2 W η , R ( log W η , R ) 2 u ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + 3 η N 2 R D ϕ W η , R log W η , R u ≤ 4 1 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + η N W η , R log W η , R ( 2 log W η , R + 3 2 R D ϕ ) u . (128)
Take a Euclidean R / N R/N R / N -net N R \mathcal{N}_R N R of the ball, with ∣ N R ∣ ≤ ( 1 + 2 N ) d |\mathcal{N}_R|\leq(1+2N)^d ∣ N R ∣ ≤ ( 1 + 2 N ) d (Vershynin, [107 ] , Section 4.2). Applying Eq. (128) on this net with u = d log ( 1 + 2 N ) + log ( 4 / δ ) u=d\log(1+2N)+\log(4/\delta) u = d log ( 1 + 2 N ) + log ( 4/ δ ) gives an event of probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 . To extend the bound to the whole ball, Eqs. (126) and (127) imply
∣ ⟨ Z θ ( x , y ) , θ η , t ⋆ − θ ⟩ − ⟨ Z θ ′ ( x , y ) , θ η , t ⋆ − θ ′ ⟩ ∣ ≤ ∥ Z θ ( x , y ) − Z θ ′ ( x , y ) ∥ 2 ∥ θ η , t ⋆ − θ ∥ 2 + ∥ Z θ ′ ( x , y ) ∥ 2 ∥ θ − θ ′ ∥ 2 ≤ ( R L ^ η , R + D ϕ W η , R log W η , R η ) ∥ θ − θ ′ ∥ 2 . \begin{aligned}
\left|\left\langle Z_\theta(x,y),\theta^\star_{\eta,t}-\theta\right\rangle-\left\langle Z_{\theta'}(x,y),\theta^\star_{\eta,t}-\theta'\right\rangle\right|
\leq\|Z_\theta(x,y)-Z_{\theta'}(x,y)\|_2\|\theta^\star_{\eta,t}-\theta\|_2+\|Z_{\theta'}(x,y)\|_2\|\theta-\theta'\|_2 \\
&\leq\left(R\widehat{L}_{\eta,R}+\frac{D_\phi W_{\eta,R}\log W_{\eta,R}}{\eta}\right)\|\theta-\theta'\|_2.
\end{aligned} ⟨ Z θ ( x , y ) , θ η , t ⋆ − θ ⟩ − ⟨ Z θ ′ ( x , y ) , θ η , t ⋆ − θ ′ ⟩ ≤ ∥ Z θ ( x , y ) − Z θ ′ ( x , y ) ∥ 2 ∥ θ η , t ⋆ − θ ∥ 2 + ∥ Z θ ′ ( x , y ) ∥ 2 ∥ θ − θ ′ ∥ 2 ≤ ( R L η , R + η D ϕ W η , R log W η , R ) ∥ θ − θ ′ ∥ 2 .
For any θ \theta θ in the ball, choose θ ˉ ∈ N R \bar{\theta}\in\mathcal{N}_R θ ˉ ∈ N R within distance R / N R/N R / N . Replacing θ ˉ \bar{\theta} θ ˉ by θ \theta θ in the empirical and population directional terms on the left of Eq. (128) , and in the population alignment on the right, adds at most
( 2 + 1 4 ) ( R L ^ η , R + D ϕ W η , R log W η , R η ) R N . \left(2+\frac{1}{4}\right)\left(R\widehat{L}_{\eta,R}+\frac{D_\phi W_{\eta,R}\log W_{\eta,R}}{\eta}\right)\frac{R}{N}. ( 2 + 4 1 ) ( R L η , R + η D ϕ W η , R log W η , R ) N R .
This is the second term in ϵ d i r \epsilon_{\mathrm{dir}} ϵ dir , proving Eq. (124) .
For the Euclidean norm, let V \mathcal{V} V be a 1 / 2 1/2 1/2 -net of the unit sphere with ∣ V ∣ ≤ 5 d |\mathcal{V}|\leq5^d ∣ V ∣ ≤ 5 d . By Eq. (126) and Hoeffding’s inequality (Vershynin, [107 ] , Section 2.2), for fixed ( θ ˉ , v ) ∈ N R × V (\bar{\theta},v)\in\mathcal{N}_R\times\mathcal{V} ( θ ˉ , v ) ∈ N R × V ,
∣ ⟨ ∇ J ^ η , t ( π θ ˉ ) − ∇ J η , t ( π θ ˉ ) , v ⟩ ∣ ≤ D ϕ W η , R log W η , R η 2 u N \left|\left\langle\widehat{\nabla J}_{\eta,t}(\pi_{\bar{\theta}})-\nabla J_{\eta,t}(\pi_{\bar{\theta}}),v\right\rangle\right|
\leq\frac{D_\phi W_{\eta,R}\log W_{\eta,R}}{\eta}\sqrt{\frac{2u}{N}} ⟨ ∇ J η , t ( π θ ˉ ) − ∇ J η , t ( π θ ˉ ) , v ⟩ ≤ η D ϕ W η , R log W η , R N 2 u
with probability at least 1 − 2 e − u 1-2e^{-u} 1 − 2 e − u . Setting u = d log ( 5 ( 1 + 2 N ) ) + log ( 4 / δ ) u=d\log(5(1+2N))+\log(4/\delta) u = d log ( 5 ( 1 + 2 N )) + log ( 4/ δ ) and taking a union bound gives an event of probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 on which this holds for every ( θ ˉ , v ) ∈ N R × V (\bar{\theta},v)\in\mathcal{N}_R\times\mathcal{V} ( θ ˉ , v ) ∈ N R × V . Approximating an arbitrary unit vector by v ∈ V v\in\mathcal{V} v ∈ V loses at most one half of the gradient-error norm, so
∥ ∇ J ^ η , t ( π θ ˉ ) − ∇ J η , t ( π θ ˉ ) ∥ 2 ≤ 2 D ϕ W η , R log W η , R η 2 u N , θ ˉ ∈ N R . \left\|\widehat{\nabla J}_{\eta,t}(\pi_{\bar{\theta}})-\nabla J_{\eta,t}(\pi_{\bar{\theta}})\right\|_2
\leq\frac{2D_\phi W_{\eta,R}\log W_{\eta,R}}{\eta}\sqrt{\frac{2u}{N}},
\qquad\bar{\theta}\in\mathcal{N}_R. ∇ J η , t ( π θ ˉ ) − ∇ J η , t ( π θ ˉ ) 2 ≤ η 2 D ϕ W η , R log W η , R N 2 u , θ ˉ ∈ N R .
Finally, Eq. (127) bounds the change in each of the empirical and population gradients by R L ^ η , R / N R\widehat{L}_{\eta,R}/N R L η , R / N when passing from θ ˉ \bar{\theta} θ ˉ to θ \theta θ . This proves Eq. (125) ; intersecting the two events establishes the simultaneous guarantee.
Lemma G.9. Let J ^ η , t \widehat{J}_{\eta,t} J η , t be defined in Eq. (113) . Fix η , R > 0 \eta, R > 0 η , R > 0 , N ≥ 1 N \ge1 N ≥ 1 , and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and define
ε g a p : = W η , R η N [ ( 8 η 2 ( B ϕ B ⋆ ) 2 + 4 D ϕ R + 2 3 log W η , R ) ( d log ( 1 + 2 N ) + log 2 δ ) + 9 4 D ϕ R log W η , R ] . (129) \varepsilon_{\mathrm{gap}} := \frac{W_{\eta,R}}{\eta N}\left[\left(8\eta^{2}(B_{\phi}B_{\star})^{2}+4D_{\phi}R+\frac{2}{3}\log W_{\eta,R}\right)\left(d\log(1+2N)+\log\frac{2}{\delta}\right)+\frac{9}{4}D_{\phi}R\log W_{\eta,R}\right]. \tag*{(129)} ε gap := η N W η , R [ ( 8 η 2 ( B ϕ B ⋆ ) 2 + 4 D ϕ R + 3 2 log W η , R ) ( d log ( 1 + 2 N ) + log δ 2 ) + 4 9 D ϕ R log W η , R ] . (129) where W η , R W_{\eta,R} W η , R is defined in Eq. (120) . With probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every θ \theta θ with ∥ θ − θ η , t ⋆ ∥ 2 ≤ R \lVert\theta-\theta_{\eta,t}^{\star}\rVert_{2}\le R ∥ θ − θ η , t ⋆ ∥ 2 ≤ R ,
∣ ( J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π θ ) ) − ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) ∣ ≤ 1 4 ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) + ε g a p . (130) \left|\left(\widehat{J}_{\eta,t}(\pi_{\eta,t}^{\star})-\widehat{J}_{\eta,t}(\pi_{\theta})\right)-\left(J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_{\theta})\right)\right|\le\frac{1}{4}\left(J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_{\theta})\right)+\varepsilon_{\mathrm{gap}}. \tag*{(130)} ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) − ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) ≤ 4 1 ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) + ε gap . (130)
Proof of Lemma G.9. We bound the second moment of each sample’s contribution to the empirical objective gap by a multiple of the population objective gap. Bernstein’s inequality and a parameter net then give a uniform concentration bound, as in Lemma G.8. Let Y θ ( x , y ) Y_{\theta}(x,y) Y θ ( x , y ) be the difference between the contributions to Eq. (113) at θ η , t ⋆ \theta_{\eta,t}^{\star} θ η , t ⋆ and θ \theta θ , and write v θ ( x , y ) : = π θ ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) = w t π θ ( x , y ) / w t π η , t ⋆ ( x , y ) v_{\theta}(x,y):=\pi_{\theta}(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x)=w_{t}^{\pi_{\theta}}(x,y)/w_{t}^{\pi_{\eta,t}^{\star}}(x,y) v θ ( x , y ) := π θ ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) = w t π θ ( x , y ) / w t π η , t ⋆ ( x , y ) . Substituting log w t π η , t ⋆ = η ( r − b t ⋆ ) \log w_{t}^{\pi_{\eta,t}^{\star}}=\eta(r-b_{t}^{\star}) log w t π η , t ⋆ = η ( r − b t ⋆ ) gives
Y θ ( x , y ) = w t π η , t ⋆ ( x , y ) [ b t ⋆ ( x ) ( 1 − v θ ( x , y ) ) + 1 η ( v θ ( x , y ) log v θ ( x , y ) − v θ ( x , y ) + 1 ) ] . (131) Y_{\theta}(x,y)=w_{t}^{\pi_{\eta,t}^{\star}}(x,y)\left[b_{t}^{\star}(x)(1-v_{\theta}(x,y))+\frac{1}{\eta}\left(v_{\theta}(x,y)\log v_{\theta}(x,y)-v_{\theta}(x,y)+1\right)\right]. \tag*{(131)} Y θ ( x , y ) = w t π η , t ⋆ ( x , y ) [ b t ⋆ ( x ) ( 1 − v θ ( x , y )) + η 1 ( v θ ( x , y ) log v θ ( x , y ) − v θ ( x , y ) + 1 ) ] . (131)
Its expectation under π t \pi_{t} π t is J η , t ( π η , t ⋆ ) − J η , t ( π θ ) J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_{\theta}) J η , t ( π η , t ⋆ ) − J η , t ( π θ ) , while its average over D t \mathcal{D}_{t} D t is J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π θ ) \widehat{J}_{\eta,t}(\pi_{\eta,t}^{\star})-\widehat{J}_{\eta,t}(\pi_{\theta}) J η , t ( π η , t ⋆ ) − J η , t ( π θ ) . The term containing b t ⋆ ( x ) b_{t}^{\star}(x) b t ⋆ ( x ) has conditional mean zero under π t \pi_{t} π t , since E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ v θ ( x , y ) ] = 1 \mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[v_{\theta}(x,y)]=1 E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ v θ ( x , y )] = 1 , but it must still be included in the second moment. On the parameter ball,
w t π η , t ⋆ ( x , y ) ≤ e η B ϕ B ⋆ , W R − 1 ≤ v θ ( x , y ) ≤ W R , 0 ≤ b t ⋆ ( x ) ≤ B ϕ B ⋆ . w_{t}^{\pi_{\eta,t}^{\star}}(x,y)\le e^{\eta B_{\phi}B_{\star}},\qquad W_{R}^{-1}\le v_{\theta}(x,y)\le W_{R},\qquad0\le b_{t}^{\star}(x)\le B_{\phi}B_{\star}. w t π η , t ⋆ ( x , y ) ≤ e η B ϕ B ⋆ , W R − 1 ≤ v θ ( x , y ) ≤ W R , 0 ≤ b t ⋆ ( x ) ≤ B ϕ B ⋆ .
For any v ∈ [ W R − 1 , W R ] v\in[W_{R}^{-1},W_{R}] v ∈ [ W R − 1 , W R ] ,
∣ 1 − v ∣ ≤ W R , 0 ≤ v log v − v + 1 ≤ W R log W R , ( v − 1 ) 2 ≤ 2 W R ( v log v − v + 1 ) . (132) |1-v|\le W_{R},\qquad0\le v\log v-v+1\le W_{R}\log W_{R},\qquad(v-1)^{2}\le2W_{R}(v\log v-v+1). \tag*{(132)} ∣1 − v ∣ ≤ W R , 0 ≤ v log v − v + 1 ≤ W R log W R , ( v − 1 ) 2 ≤ 2 W R ( v log v − v + 1 ) . (132)
The last inequality follows because v log v − v + 1 v\log v-v+1 v log v − v + 1 and its first derivative vanish at v = 1 v=1 v = 1 , while its second derivative is 1 / v ≥ 1 / W R 1/v\ge1/W_{R} 1/ v ≥ 1/ W R on this interval. These bounds and Eq. (131) imply
∣ Y θ ( x , y ) ∣ ≤ e η B ϕ B ⋆ W R ( B ϕ B ⋆ + log W R η ) = W η , R log W η , R η . |Y_{\theta}(x,y)|\le e^{\eta B_{\phi}B_{\star}}W_{R}\left(B_{\phi}B_{\star}+\frac{\log W_{R}}{\eta}\right)=\frac{W_{\eta,R}\log W_{\eta,R}}{\eta}. ∣ Y θ ( x , y ) ∣ ≤ e η B ϕ B ⋆ W R ( B ϕ B ⋆ + η log W R ) = η W η , R log W η , R .
Moreover, using ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^{2}\le2a^{2}+2b^{2} ( a + b ) 2 ≤ 2 a 2 + 2 b 2 and changing measure to π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ gives
E π t [ Y θ 2 ] ≤ 2 e η B ϕ B ⋆ ( B ϕ B ⋆ ) 2 E π η , t ⋆ [ ( 1 − v θ ) 2 ] + 2 e η B ϕ B ⋆ η 2 E π η , t ⋆ [ ( v θ log v θ − v θ + 1 ) 2 ] ≤ W η , R ( 4 ( B ϕ B ⋆ ) 2 + 2 log W R η 2 ) E π η , t ⋆ [ v θ log v θ − v θ + 1 ] = W η , R ( 4 η ( B ϕ B ⋆ ) 2 + 2 D ϕ R η ) ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) . (133) \begin{aligned}
\mathbb{E}_{\pi_{t}}[Y_{\theta}^{2}]
&\le2e^{\eta B_{\phi}B_{\star}}(B_{\phi}B_{\star})^{2}\mathbb{E}_{\pi_{\eta,t}^{\star}}[(1-v_{\theta})^{2}]\\
&\quad+\frac{2e^{\eta B_{\phi}B_{\star}}}{\eta^{2}}\mathbb{E}_{\pi_{\eta,t}^{\star}}[(v_{\theta}\log v_{\theta}-v_{\theta}+1)^{2}]\\
&\le W_{\eta,R}\left(4(B_{\phi}B_{\star})^{2}+\frac{2\log W_{R}}{\eta^{2}}\right)\mathbb{E}_{\pi_{\eta,t}^{\star}}[v_{\theta}\log v_{\theta}-v_{\theta}+1]\\
&=W_{\eta,R}\left(4\eta(B_{\phi}B_{\star})^{2}+\frac{2D_{\phi}R}{\eta}\right)\left(J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_{\theta})\right). \tag*{(133)}
\end{aligned} E π t [ Y θ 2 ] ≤ 2 e η B ϕ B ⋆ ( B ϕ B ⋆ ) 2 E π η , t ⋆ [( 1 − v θ ) 2 ] + η 2 2 e η B ϕ B ⋆ E π η , t ⋆ [( v θ log v θ − v θ + 1 ) 2 ] ≤ W η , R ( 4 ( B ϕ B ⋆ ) 2 + η 2 2 log W R ) E π η , t ⋆ [ v θ log v θ − v θ + 1 ] = W η , R ( 4 η ( B ϕ B ⋆ ) 2 + η 2 D ϕ R ) ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) . (133)
The second step uses Eq. (132) ; the last step uses
E π η , t ⋆ [ v θ log v θ − v θ + 1 ] = E π θ [ log π θ π η , t ⋆ ] = D K L ( π θ ∥ π η , t ⋆ ) = η ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) . \mathbb{E}_{\pi_{\eta,t}^{\star}}[v_{\theta}\log v_{\theta}-v_{\theta}+1]=\mathbb{E}_{\pi_{\theta}}\left[\log\frac{\pi_{\theta}}{\pi_{\eta,t}^{\star}}\right]=D_{\mathrm{KL}}(\pi_{\theta}\Vert\pi_{\eta,t}^{\star})=\eta\left(J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_{\theta})\right). E π η , t ⋆ [ v θ log v θ − v θ + 1 ] = E π θ [ log π η , t ⋆ π θ ] = D KL ( π θ ∥ π η , t ⋆ ) = η ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) .
where the first equality again uses that both policies have total mass one, and the last equality is Lemma G.2 .
Take a Euclidean R / N R/N R / N -net N R \mathcal{N}_{R} N R with ∣ N R ∣ ≤ ( 1 + 2 N ) d \lvert\mathcal{N}_{R}\rvert\leq(1+2N)^{d} ∣ N R ∣ ≤ ( 1 + 2 N ) d , and set u = d log ( 1 + 2 N ) + log ( 2 / δ ) u=d\log(1+2N)+\log(2/\delta) u = d log ( 1 + 2 N ) + log ( 2/ δ ) . Bernstein’s inequality [107 Section 2.8] , Eq. (133) , and a union bound imply that, with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every θ ˉ ∈ N R \bar{\theta}\in\mathcal{N}_{R} θ ˉ ∈ N R ,
∣ ( J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π θ ˉ ) ) − ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ˉ ) ) ∣ ≤ 2 W η , R u N ( 4 η ( B ϕ B ⋆ ) 2 + 2 D ϕ R η ) ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ˉ ) ) + 2 W η , R log W η , R 3 η N u ≤ 1 4 ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ˉ ) ) + W η , R η N ( 8 η 2 ( B ϕ B ⋆ ) 2 + 4 D ϕ R + 2 3 log W η , R ) u , \begin{aligned}
\left\lvert\left(\widehat{J}_{\eta,t}(\pi^{\star}_{\eta,t})-\widehat{J}_{\eta,t}(\pi_{\bar{\theta}})\right)-\left(J_{\eta,t}(\pi^{\star}_{\eta,t})-J_{\eta,t}(\pi_{\bar{\theta}})\right)\right\rvert
\leq\sqrt{\frac{2W_{\eta,R}u}{N}\left(4\eta(B_{\phi}B_{\star})^{2}+\frac{2D_{\phi}R}{\eta}\right)\left(J_{\eta,t}(\pi^{\star}_{\eta,t})-J_{\eta,t}(\pi_{\bar{\theta}})\right)}+\frac{2W_{\eta,R}\log W_{\eta,R}}{3\eta N}u \\
&\leq\frac{1}{4}\left(J_{\eta,t}(\pi^{\star}_{\eta,t})-J_{\eta,t}(\pi_{\bar{\theta}})\right)+\frac{W_{\eta,R}}{\eta N}\left(8\eta^{2}(B_{\phi}B_{\star})^{2}+4D_{\phi}R+\frac{2}{3}\log W_{\eta,R}\right)u,
\end{aligned} ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ˉ ) ) − ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ˉ ) ) ≤ N 2 W η , R u ( 4 η ( B ϕ B ⋆ ) 2 + η 2 D ϕ R ) ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ˉ ) ) + 3 η N 2 W η , R log W η , R u ≤ 4 1 ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ˉ ) ) + η N W η , R ( 8 η 2 ( B ϕ B ⋆ ) 2 + 4 D ϕ R + 3 2 log W η , R ) u ,
where the last step is a b ≤ a / 4 + b \sqrt{ab}\leq a/4+b ab ≤ a /4 + b . To extend to the whole ball, Eq. (115) gives
∇ Y θ ( x , y ) = − w t π θ ( x , y ) ℓ θ ( x , y ) ϕ ~ θ ( x , y ) , ∥ ∇ Y θ ( x , y ) ∥ 2 ≤ D ϕ W η , R log W η , R η . \nabla Y_{\theta}(x,y)=-w_{t}^{\pi_{\theta}}(x,y)\ell_{\theta}(x,y)\widetilde{\phi}_{\theta}(x,y),\qquad\lVert\nabla Y_{\theta}(x,y)\rVert_{2}\leq\frac{D_{\phi}W_{\eta,R}\log W_{\eta,R}}{\eta}. ∇ Y θ ( x , y ) = − w t π θ ( x , y ) ℓ θ ( x , y ) ϕ θ ( x , y ) , ∥ ∇ Y θ ( x , y ) ∥ 2 ≤ η D ϕ W η , R log W η , R .
For each θ \theta θ in the ball, choose θ ˉ ∈ N R \bar{\theta}\in\mathcal{N}_{R} θ ˉ ∈ N R within distance R / N R/N R / N . Replacing θ ˉ \bar{\theta} θ ˉ by θ \theta θ in the empirical and population gaps on the left and the population gap on the right adds at most
( 2 + 1 4 ) D ϕ W η , R log W η , R η R N = 9 D ϕ R W η , R log W η , R 4 η N . \left(2+\frac{1}{4}\right)\frac{D_{\phi}W_{\eta,R}\log W_{\eta,R}}{\eta}\frac{R}{N}=\frac{9D_{\phi}RW_{\eta,R}\log W_{\eta,R}}{4\eta N}. ( 2 + 4 1 ) η D ϕ W η , R log W η , R N R = 4 η N 9 D ϕ R W η , R log W η , R .
This is the remaining term in ε g a p \varepsilon_{\mathrm{gap}} ε gap , proving the claim.
Inner Loop Convergence
We now prove convergence of the inner loop of Algorithm 1, which reuses one batch for all K K K gradient steps. The proof extends the population policy gradient argument in Appendix § 8 .§ 2 by controlling the parameter drift caused by sampling error. A sufficiently large batch keeps the iterates near the target parameter, where the empirical objective is smooth and its gradient and objective gap concentrate uniformly. Throughout the subsection, we condition on the history before the batch D t D_t D t is drawn.
Theorem G.1 (Inner loop convergence of PPO-NoClip). Fix an outer iteration t t t , integers K , N ≥ 1 K,N\geq1 K , N ≥ 1 , and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Run the inner loop of Algorithm 1, initialized at θ t 0 = θ t \theta_{t}^{0}=\theta_{t} θ t 0 = θ t , with
η = 1 B ϕ B ⋆ , α = 1 60 e 5 B ϕ 3 B ⋆ . \eta=\frac{1}{B_{\phi}B_{\star}},\qquad\alpha=\frac{1}{60e^{5}B_{\phi}^{3}B_{\star}}. η = B ϕ B ⋆ 1 , α = 60 e 5 B ϕ 3 B ⋆ 1 . If
N ≥ 2 K log ( 8 [ 5 ( 1 + 2 N ) ] d δ ) (134) N\geq2K\log\left(\frac{8[5(1+2N)]^{d}}{\delta}\right) \tag*{(134)} N ≥ 2 K log ( δ 8 [ 5 ( 1 + 2 N ) ] d ) (134) then, with probability at least 1 − δ 1-\delta 1 − δ , for every 0 ≤ k ≤ K 0\leq k\leq K 0 ≤ k ≤ K and every ( x , y ) (x,y) ( x , y ) ,
∥ θ t k − θ η , t ⋆ ∥ 2 ≤ R = 2 η B ⋆ , \lVert\theta_{t}^{k}-\theta_{\eta,t}^{\star}\rVert_{2}\leq R=2\eta B_{\star}, ∥ θ t k − θ η , t ⋆ ∥ 2 ≤ R = 2 η B ⋆ , π t k ( y ∣ x ) π t ( y ∣ x ) ≤ e D ϕ R + η B ϕ B ⋆ , π η , t ⋆ ( y ∣ x ) π t k ( y ∣ x ) ≤ e D ϕ R , (135) \frac{\pi_{t}^{k}(y\mid x)}{\pi_{t}(y\mid x)}\leq e^{D_{\phi}R+\eta B_{\phi}B_{\star}},\qquad\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_{t}^{k}(y\mid x)}\leq e^{D_{\phi}R}, \tag*{(135)} π t ( y ∣ x ) π t k ( y ∣ x ) ≤ e D ϕ R + η B ϕ B ⋆ , π t k ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ≤ e D ϕ R , (135) and the final policy satisfies
J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ≤ 150 e 9 B ϕ B ⋆ [ 1 K + 1 N log ( 8 [ 5 ( 1 + 2 N ) ] d δ ) ] . (136) J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_{t}^{K})\leq150e^{9}B_{\phi}B_{\star}\left[\frac{1}{K}+\frac{1}{N}\log\left(\frac{8[5(1+2N)]^{d}}{\delta}\right)\right]. \tag*{(136)} J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ≤ 150 e 9 B ϕ B ⋆ [ K 1 + N 1 log ( δ 8 [ 5 ( 1 + 2 N ) ] d ) ] . (136)
Proof of Theorem G.1.
Set R = 2 η B ⋆ = 2 / B ϕ R = 2\eta B_\star= 2/B_\phi R = 2 η B ⋆ = 2/ B ϕ . Recall the policy ratio bounds
W R = e D ϕ R , W η , R = e D ϕ R + η B ϕ B ⋆ . W_R = e^{D_\phi R}, \qquad W_{\eta,R} = e^{D_\phi R+\eta B_\phi B_\star}. W R = e D ϕ R , W η , R = e D ϕ R + η B ϕ B ⋆ .
from Lemma G.6 and Eq. (120) . On the ball of radius R R R around θ η , t ⋆ \theta_{\eta,t}^\star θ η , t ⋆ , these bound π η , t ⋆ / π θ \pi_{\eta,t}^\star/\pi_\theta π η , t ⋆ / π θ and π θ / π t \pi_\theta/\pi_t π θ / π t , respectively. The empirical objective is smooth on this ball with constant L ^ η , R \widehat{L}_{\eta,R} L η , R defined in Eq. (122) . Since D ϕ R ≤ 4 D_\phi R \le4 D ϕ R ≤ 4 and η B ϕ B ⋆ = 1 \eta B_\phi B_\star= 1 η B ϕ B ⋆ = 1 ,
W R ≤ e 4 , W η , R ≤ e 5 , L ^ η , R ≤ 29 e 5 B ϕ 3 B ⋆ < 1 α . (137) W_R \le e^4, \qquad W_{\eta,R} \le e^5, \qquad\widehat{L}_{\eta,R} \le29e^5B_\phi^3B_\star< \frac{1}{\alpha}. \tag*{(137)} W R ≤ e 4 , W η , R ≤ e 5 , L η , R ≤ 29 e 5 B ϕ 3 B ⋆ < α 1 . (137)
Also, α D ϕ 2 / η ≤ 1 / ( 15 e 5 ) < 3 / 4 \alpha D_\phi^2/\eta\le1/(15e^5) < 3/4 α D ϕ 2 / η ≤ 1/ ( 15 e 5 ) < 3/4 , which verifies the learning rate condition in Lemma G.10.
Apply Lemmas G.8 and G.9 with failure probability δ / 2 \delta/2 δ /2 each. By a union bound, with probability at least 1 − δ 1-\delta 1 − δ , the directional and Euclidean norm bounds on the gradient error in Eqs. (124) and (125) and the objective-gap bound in Eq. (130) hold simultaneously for every θ \theta θ in the ball. Fix a batch for which these inequalities hold, and let ε d i r \varepsilon_{\rm dir} ε dir , ε g r a d \varepsilon_{\rm grad} ε grad , and ε g a p \varepsilon_{\rm gap} ε gap denote the errors in Eqs. (123) and (129) evaluated with failure probability δ / 2 \delta/2 δ /2 . Abbreviate the common logarithmic factor by
u : = log ( 8 [ 5 ( 1 + 2 N ) ] d δ ) . u := \log\left(\frac{8[5(1+2N)]^d}{\delta}\right). u := log ( δ 8 [ 5 ( 1 + 2 N ) ] d ) .
Then the sample condition is N ≥ 2 K u N \ge2Ku N ≥ 2 K u , and u ≥ log 120 > 9 / 2 u \ge\log120 > 9/2 u ≥ log 120 > 9/2 . Substituting Eq. (137) into the error bounds gives
ε d i r ≤ e 5 B ϕ B ⋆ N ( 190 3 u + 306 ) ≤ 394 e 5 B ϕ B ⋆ 3 N u , \varepsilon_{\rm dir} \le\frac{e^5B_\phi B_\star}{N}\left(\frac{190}{3}u+306\right) \le\frac{394e^5B_\phi B_\star}{3N}u, ε dir ≤ N e 5 B ϕ B ⋆ ( 3 190 u + 306 ) ≤ 3 N 394 e 5 B ϕ B ⋆ u ,
ε g r a d ≤ e 5 B ϕ 2 B ⋆ ( 20 2 u N + 116 N ) ≤ 33 e 5 B ϕ 2 B ⋆ 2 u N , \varepsilon_{\rm grad} \le e^5B_\phi^2B_\star\left(20\sqrt{\frac{2u}{N}}+\frac{116}{N}\right) \le33e^5B_\phi^2B_\star\sqrt{\frac{2u}{N}}, ε grad ≤ e 5 B ϕ 2 B ⋆ ( 20 N 2 u + N 116 ) ≤ 33 e 5 B ϕ 2 B ⋆ N 2 u ,
ε g a p ≤ e 5 B ϕ B ⋆ N ( 82 3 u + 45 ) ≤ 112 e 5 B ϕ B ⋆ 3 N u . (138) \varepsilon_{\rm gap} \le\frac{e^5B_\phi B_\star}{N}\left(\frac{82}{3}u+45\right) \le\frac{112e^5B_\phi B_\star}{3N}u. \tag*{(138)} ε gap ≤ N e 5 B ϕ B ⋆ ( 3 82 u + 45 ) ≤ 3 N 112 e 5 B ϕ B ⋆ u . (138)
The last bounds for ε d i r \varepsilon_{\rm dir} ε dir and ε g a p \varepsilon_{\rm gap} ε gap use u > 9 / 2 u>9/2 u > 9/2 . For ε g r a d \varepsilon_{\rm grad} ε grad , the condition N ≥ 2 K u ≥ 2 u N \ge2Ku \ge2u N ≥ 2 K u ≥ 2 u gives 116 / 2 N u ≤ 58 / u < 13 116/\sqrt{2Nu} \le58/u < 13 116/ 2 N u ≤ 58/ u < 13 , which bounds its N − 1 N^{-1} N − 1 term by a multiple of 2 u / N \sqrt{2u/N} 2 u / N .
We first show that all iterates remain in the ball of radius R R R around θ η , t ⋆ \theta_{\eta,t}^\star θ η , t ⋆ . The accumulated error in Eq. (141) satisfies
K ( 2 α ε d i r + 2 α 2 ε g r a d 2 ) ≤ 5029 900 K u B ϕ 2 N ≤ 5029 1800 B ϕ 2 < 3 B ϕ 2 . (139) K(2\alpha\varepsilon_{\rm dir}+2\alpha^2\varepsilon_{\rm grad}^2) \le\frac{5029}{900}\frac{Ku}{B_\phi^2N} \le\frac{5029}{1800B_\phi^2} < \frac{3}{B_\phi^2}. \tag*{(139)} K ( 2 α ε dir + 2 α 2 ε grad 2 ) ≤ 900 5029 B ϕ 2 N K u ≤ 1800 B ϕ 2 5029 < B ϕ 2 3 . (139)
The initialization satisfies ∥ θ t 0 − θ η , t ⋆ ∥ 2 2 = η 2 B 2 = 1 / B ϕ 2 \lVert\theta_t^0-\theta_{\eta,t}^\star\rVert_2^2=\eta^2B^2=1/B_\phi^2 ∥ θ t 0 − θ η , t ⋆ ∥ 2 2 = η 2 B 2 = 1/ B ϕ 2 . Whenever the current iterate lies in the ball, Lemma G.10 bounds the increase in squared distance by 2 α ε d i r + 2 α 2 ε g r a d 2 2\alpha\varepsilon_{\rm dir}+2\alpha^2\varepsilon_{\rm grad}^2 2 α ε dir + 2 α 2 ε grad 2 . Induction therefore gives
∥ θ t k − θ η , t ⋆ ∥ 2 2 ≤ 1 B ϕ 2 + k ( 2 α ε d i r + 2 α 2 ε g r a d 2 ) < 4 B ϕ 2 = R 2 , 0 ≤ k ≤ K . \lVert\theta_t^k-\theta_{\eta,t}^\star\rVert_2^2 \le\frac{1}{B_\phi^2}+k(2\alpha\varepsilon_{\rm dir}+2\alpha^2\varepsilon_{\rm grad}^2) < \frac{4}{B_\phi^2}=R^2,\qquad0\le k\le K. ∥ θ t k − θ η , t ⋆ ∥ 2 2 ≤ B ϕ 2 1 + k ( 2 α ε dir + 2 α 2 ε grad 2 ) < B ϕ 2 4 = R 2 , 0 ≤ k ≤ K .
Indeed, Eq. (139) ensures that the next iterate stays strictly inside the ball whenever the recursion is applied. Applying Lemmas B.12 and G.7 on this ball yields
π t k ( y ∣ x ) π t ( y ∣ x ) ≤ W η , R , π η , t ⋆ ( y ∣ x ) π t k ( y ∣ x ) ≤ W R . \frac{\pi_t^k(y\mid x)}{\pi_t(y\mid x)}\le W_{\eta,R},\qquad
\frac{\pi_{\eta,t}^\star(y\mid x)}{\pi_t^k(y\mid x)}\le W_R. π t ( y ∣ x ) π t k ( y ∣ x ) ≤ W η , R , π t k ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ≤ W R .
Substituting the definitions of W R W_R W R and W η , R W_{\eta,R} W η , R proves Eq. (135) . Both endpoints of every inner update now lie in the ball, and α L ^ η , R < 1 \alpha\widehat{L}_{\eta,R}<1 α L η , R < 1 by Eq. (137) . Together with the uniform concentration bounds fixed above, these conditions verify the hypotheses of Lemma G.11 for every inner update. Its proof also gives the empirical ascent bound Eq. (145) , so the empirical objective is nondecreasing.
We next bound the final objective gap. Sum Eq. (142) over 0 ≤ k < K 0 \le k < K 0 ≤ k < K and telescope the squared distances and empirical objective increments. Using W R ≤ e 4 W_R \le e^4 W R ≤ e 4 and the initial distance gives
∑ k = 0 K − 1 ( J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π t k ) ) ≤ 5 e 4 η 2 B ⋆ 2 12 α + 5 e 4 6 ( J ^ η , t ( π t K ) − J ^ η , t ( π t 0 ) ) + K ( 5 e 4 6 ε d i r + ε g a p ) . \sum_{k=0}^{K-1}\left(\widehat{J}_{\eta,t}(\pi_{\eta,t}^{\star})-\widehat{J}_{\eta,t}(\pi_t^k)\right)\le\frac{5e^4\eta^2B_{\star}^2}{12\alpha}+\frac{5e^4}{6}\left(\widehat{J}_{\eta,t}(\pi_t^K)-\widehat{J}_{\eta,t}(\pi_t^0)\right)+K\left(\frac{5e^4}{6}\varepsilon_{\mathrm{dir}}+\varepsilon_{\mathrm{gap}}\right). k = 0 ∑ K − 1 ( J η , t ( π η , t ⋆ ) − J η , t ( π t k ) ) ≤ 12 α 5 e 4 η 2 B ⋆ 2 + 6 5 e 4 ( J η , t ( π t K ) − J η , t ( π t 0 ) ) + K ( 6 5 e 4 ε dir + ε gap ) .
Each empirical gap in the sum is at least the final empirical gap, because the empirical objective is nondecreasing. Moving that gap to the left gives
( K + 5 e 4 6 ) ( J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π t K ) ) ≤ 5 e 4 η 2 B ⋆ 2 12 α + 5 e 4 6 ( J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π t 0 ) ) + K ( 5 e 4 6 ε d i r + ε g a p ) . (140) \left(K+\frac{5e^4}{6}\right)\left(\widehat{J}_{\eta,t}(\pi_{\eta,t}^{\star})-\widehat{J}_{\eta,t}(\pi_t^K)\right)
\le\frac{5e^4\eta^2B_{\star}^2}{12\alpha}+\frac{5e^4}{6}\left(\widehat{J}_{\eta,t}(\pi_{\eta,t}^{\star})-\widehat{J}_{\eta,t}(\pi_t^0)\right)+K\left(\frac{5e^4}{6}\varepsilon_{\mathrm{dir}}+\varepsilon_{\mathrm{gap}}\right). \tag*{(140)} ( K + 6 5 e 4 ) ( J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ) ≤ 12 α 5 e 4 η 2 B ⋆ 2 + 6 5 e 4 ( J η , t ( π η , t ⋆ ) − J η , t ( π t 0 ) ) + K ( 6 5 e 4 ε dir + ε gap ) . (140)
The initial population gap is at most B ϕ B ⋆ B_{\phi}B_{\star} B ϕ B ⋆ , because J η , t ( π η , t ⋆ ) ≤ J ( π η , t ⋆ ) ≤ B ϕ B ⋆ J_{\eta,t}(\pi_{\eta,t}^{\star}) \le J(\pi_{\eta,t}^{\star}) \le B_{\phi}B_{\star} J η , t ( π η , t ⋆ ) ≤ J ( π η , t ⋆ ) ≤ B ϕ B ⋆ and J η , t ( π t ) = J ( π t ) ≥ 0 J_{\eta,t}(\pi_t)=J(\pi_t)\ge0 J η , t ( π t ) = J ( π t ) ≥ 0 . By Lemma G.9 ,
J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π t 0 ) ≤ 5 4 B ϕ B ⋆ + ε g a p , \widehat{J}_{\eta,t}(\pi_{\eta,t}^{\star})-\widehat{J}_{\eta,t}(\pi_t^0)\le\frac{5}{4}B_{\phi}B_{\star}+\varepsilon_{\mathrm{gap}}, J η , t ( π η , t ⋆ ) − J η , t ( π t 0 ) ≤ 4 5 B ϕ B ⋆ + ε gap ,
J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ≤ 4 3 ( J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π t K ) ) + 4 3 ε g a p . J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_t^K)\le\frac{4}{3}\left(\widehat{J}_{\eta,t}(\pi_{\eta,t}^{\star})-\widehat{J}_{\eta,t}(\pi_t^K)\right)+\frac{4}{3}\varepsilon_{\mathrm{gap}}. J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ≤ 3 4 ( J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ) + 3 4 ε gap .
Substitute the initial empirical gap bound into Eq. (140) and divide by K + 5 e 4 / 6 K+5e^4/6 K + 5 e 4 /6 . The two coefficients of ε g a p \varepsilon_{\mathrm{gap}} ε gap sum to one, giving
J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π t K ) ≤ 5 e 4 η 2 B ⋆ 2 / ( 12 α ) + 25 e 4 B ϕ B ⋆ / 24 K + 5 e 4 / 6 + 5 e 4 K 6 K + 5 e 4 ε d i r + ε g a p . \widehat{J}_{\eta,t}(\pi_{\eta,t}^{\star})-\widehat{J}_{\eta,t}(\pi_t^K)
\le\frac{5e^4\eta^2B_{\star}^2/(12\alpha)+25e^4B_{\phi}B_{\star}/24}{K+5e^4/6}+\frac{5e^4K}{6K+5e^4}\varepsilon_{\mathrm{dir}}+\varepsilon_{\mathrm{gap}}. J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ≤ K + 5 e 4 /6 5 e 4 η 2 B ⋆ 2 / ( 12 α ) + 25 e 4 B ϕ B ⋆ /24 + 6 K + 5 e 4 5 e 4 K ε dir + ε gap .
Converting to the population gap and using K + 5 e 4 / 6 ≥ K K+5e^4/6\ge K K + 5 e 4 /6 ≥ K yields
J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ≤ ( 100 e 9 3 + 25 e 4 18 ) B ϕ B ⋆ K + 10 e 4 9 ε d i r + 8 3 ε g a p . J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_t^K)\le\left(\frac{100e^9}{3}+\frac{25e^4}{18}\right)\frac{B_{\phi}B_{\star}}{K}+\frac{10e^4}{9}\varepsilon_{\mathrm{dir}}+\frac{8}{3}\varepsilon_{\mathrm{gap}}. J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ≤ ( 3 100 e 9 + 18 25 e 4 ) K B ϕ B ⋆ + 9 10 e 4 ε dir + 3 8 ε gap .
Substituting Eq. (138) and the definition of u u u proves Eq. (136) , since
100 e 9 3 + 25 e 4 18 < 150 e 9 , 3940 e 9 27 + 896 e 5 9 < 150 e 9 . \frac{100e^9}{3}+\frac{25e^4}{18}<150e^9,\qquad\frac{3940e^9}{27}+\frac{896e^5}{9}<150e^9. 3 100 e 9 + 18 25 e 4 < 150 e 9 , 27 3940 e 9 + 9 896 e 5 < 150 e 9 .
Lemma G.10 (Control of the parameter distance, adapted from Lemma B.5). Let J ^ η , t \widehat{J}_{\eta,t} J η , t be defined in Eq. (113) . Fix η \eta η , R > 0 R>0 R > 0 , N ≥ 1 N\ge1 N ≥ 1 , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and α > 0 \alpha>0 α > 0 with 4 α D ϕ 2 ≤ 3 η 4\alpha D_{\phi}^2\le3\eta 4 α D ϕ 2 ≤ 3 η . Let ε d i r \varepsilon_{\mathrm{dir}} ε dir and ε g r a d \varepsilon_{\mathrm{grad}} ε grad be defined in Eq. (123) with failure probability δ / 2 \delta/2 δ /2 . With probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 , the directional and norm bounds on the gradient error in Lemma G.8 hold throughout the ball of radius R R R around θ η , t ⋆ \theta_{\eta,t}^{\star} θ η , t ⋆ . On this event, for every θ \theta θ in the ball, the empirical gradient update θ + = θ + α ∇ J ^ η , t ( π θ ) \theta^+=\theta+\alpha\nabla\widehat{J}_{\eta,t}(\pi_\theta) θ + = θ + α ∇ J η , t ( π θ ) satisfies
∥ θ + − θ η , t ⋆ ∥ 2 2 ≤ ∥ θ − θ η , t ⋆ ∥ 2 2 + 2 α ε d i r + 2 α 2 ε g r a d 2 . (141) \left\|\theta^+-\theta_{\eta,t}^{\star}\right\|_2^2\le\left\|\theta-\theta_{\eta,t}^{\star}\right\|_2^2+2\alpha\varepsilon_{\mathrm{dir}}+2\alpha^2\varepsilon_{\mathrm{grad}}^2. \tag*{(141)} θ + − θ η , t ⋆ 2 2 ≤ θ − θ η , t ⋆ 2 2 + 2 α ε dir + 2 α 2 ε grad 2 . (141)
Proof of Lemma G.10. We bound the increase in squared distance caused by replacing the population gradient with its empirical estimate. Expanding the update gives
∥ θ + − θ η , t ⋆ ∥ 2 2 = ∥ θ − θ η , t ⋆ ∥ 2 2 − 2 α ⟨ ∇ J ^ η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + α 2 ∥ ∇ J ^ η , t ( π θ ) ∥ 2 2 . \left\|\theta^+-\theta_{\eta,t}^{\star}\right\|_2^2=\left\|\theta-\theta_{\eta,t}^{\star}\right\|_2^2-2\alpha\left\langle\nabla\widehat{J}_{\eta,t}(\pi_\theta),\theta_{\eta,t}^{\star}-\theta\right\rangle+\alpha^2\left\|\nabla\widehat{J}_{\eta,t}(\pi_\theta)\right\|_2^2. θ + − θ η , t ⋆ 2 2 = θ − θ η , t ⋆ 2 2 − 2 α ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + α 2 ∇ J η , t ( π θ ) 2 2 .
On the concentration event, the directional and norm bounds give
− 2 α ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ≤ − 3 α 2 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + 2 α ε d i r , -2\alpha\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle\le-\frac{3\alpha}{2}\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle+2\alpha\varepsilon_{\mathrm{dir}}, − 2 α ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ≤ − 2 3 α ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + 2 α ε dir ,
∥ ∇ J ^ η , t ( π θ ) ∥ 2 2 ≤ 2 ∥ ∇ J η , t ( π θ ) ∥ 2 2 + 2 ε g r a d 2 ≤ 2 D ϕ 2 η ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + 2 ε g r a d 2 , \begin{aligned}
\left\|\nabla\widehat{J}_{\eta,t}(\pi_\theta)\right\|_2^2 \le2\left\|\nabla J_{\eta,t}(\pi_\theta)\right\|_2^2+2\varepsilon_{\mathrm{grad}}^2 \\
&\le\frac{2D_\phi^2}{\eta}\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle+2\varepsilon_{\mathrm{grad}}^2,
\end{aligned} ∇ J η , t ( π θ ) 2 2 ≤ 2 ∥ ∇ J η , t ( π θ ) ∥ 2 2 + 2 ε grad 2 ≤ η 2 D ϕ 2 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + 2 ε grad 2 ,
where the last inequality uses Eq. (118) . Substituting into the distance expansion yields
∥ θ + − θ η , t ⋆ ∥ 2 2 ≤ ∥ θ − θ η , t ⋆ ∥ 2 2 − α ( 3 2 − 2 α D ϕ 2 η ) ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + 2 α ε d i r + 2 α 2 ε g r a d 2 . \begin{aligned}
\left\|\theta^+-\theta^\star_{\eta,t}\right\|_2^2\le\left\|\theta-\theta^\star_{\eta,t}\right\|_2^2-\alpha\left(\frac{3}{2}-\frac{2\alpha D_\phi^2}{\eta}\right)\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle\\
&+2\alpha\varepsilon_{\mathrm{dir}}+2\alpha^2\varepsilon_{\mathrm{grad}}^2.
\end{aligned} θ + − θ η , t ⋆ 2 2 ≤ θ − θ η , t ⋆ 2 2 − α ( 2 3 − η 2 α D ϕ 2 ) ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + 2 α ε dir + 2 α 2 ε grad 2 .
The alignment is nonnegative by Eq. (117) , and 4 α D ϕ 2 ≤ 3 η 4\alpha D_\phi^2\le3\eta 4 α D ϕ 2 ≤ 3 η makes its coefficient nonpositive. Dropping this term proves the claim.
Lemma G.11 (One-step objective progress). Let J ^ η , t \widehat{J}_{\eta,t} J η , t be defined in Eq. (113) . Fix η \eta η , R > 0 R>0 R > 0 , N ≥ 1 N\ge1 N ≥ 1 , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and α > 0 \alpha>0 α > 0 with α L J ^ η , R ≤ 1 \alpha L_{\widehat{J}_{\eta,R}}\le1 α L J η , R ≤ 1 . Let ε d i r \varepsilon_{\mathrm{dir}} ε dir and ε g a p \varepsilon_{\mathrm{gap}} ε gap be defined in Eqs. (123) and (129) with failure probability δ / 2 \delta/2 δ /2 . With probability at least 1 − δ 1-\delta 1 − δ , the directional gradient error and objective-gap error satisfy Eqs. (124) and (130) throughout the ball of radius R R R around θ η , t ⋆ \theta^\star_{\eta,t} θ η , t ⋆ . On this event, every empirical gradient update θ + = θ + α ∇ J ^ η , t ( π θ ) \theta^+=\theta+\alpha\nabla\widehat{J}_{\eta,t}(\pi_\theta) θ + = θ + α ∇ J η , t ( π θ ) with ∥ θ − θ η , t ⋆ ∥ 2 ≤ R \left\|\theta-\theta^\star_{\eta,t}\right\|_2\le R θ − θ η , t ⋆ 2 ≤ R and ∥ θ + − θ η , t ⋆ ∥ 2 ≤ R \left\|\theta^+-\theta^\star_{\eta,t}\right\|_2\le R θ + − θ η , t ⋆ 2 ≤ R satisfies
J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π θ ) ≤ 5 W R 12 α ( ∥ θ − θ η , t ⋆ ∥ 2 2 − ∥ θ + − θ η , t ⋆ ∥ 2 2 ) \widehat{J}_{\eta,t}(\pi_{\eta,t}^\star)-\widehat{J}_{\eta,t}(\pi_\theta)\le\frac{5W_R}{12\alpha}\left(\left\|\theta-\theta^\star_{\eta,t}\right\|_2^2-\left\|\theta^+-\theta^\star_{\eta,t}\right\|_2^2\right) J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ≤ 12 α 5 W R ( θ − θ η , t ⋆ 2 2 − θ + − θ η , t ⋆ 2 2 ) + 5 W R 6 ( J ^ η , t ( π θ + ) − J ^ η , t ( π θ ) ) + 5 W R 6 ε d i r + ε g a p . (142) +\frac{5W_R}{6}\left(\widehat{J}_{\eta,t}(\pi_{\theta^+})-\widehat{J}_{\eta,t}(\pi_\theta)\right)+\frac{5W_R}{6}\varepsilon_{\mathrm{dir}}+\varepsilon_{\mathrm{gap}}. \tag*{(142)} + 6 5 W R ( J η , t ( π θ + ) − J η , t ( π θ ) ) + 6 5 W R ε dir + ε gap . (142)
Proof of Lemma G.11. Apply Lemmas G.8 and G.9 with failure probability δ / 2 \delta/2 δ /2 each. By a union bound, with probability at least 1 − δ 1-\delta 1 − δ , the directional gradient error bound Eq. (124) and the objective gap error bound Eq. (130) hold simultaneously for every θ \theta θ in the ball of radius R R R around θ η , t ⋆ \theta^\star_{\eta,t} θ η , t ⋆ . Fix a batch for which these bounds hold and a parameter θ \theta θ in this ball.
We first bound the empirical objective gap by the population objective gap. Taking the upper bound on the difference between these gaps in Eq. (130) gives
J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π θ ) ≤ ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) + 1 4 ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) + ε g a p = 5 4 ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) + ε g a p . \begin{aligned}
\widehat{J}_{\eta,t}(\pi_{\eta,t}^\star)-\widehat{J}_{\eta,t}(\pi_\theta)\le\left(J_{\eta,t}(\pi_{\eta,t}^\star)-J_{\eta,t}(\pi_\theta)\right)+\frac{1}{4}\left(J_{\eta,t}(\pi_{\eta,t}^\star)-J_{\eta,t}(\pi_\theta)\right)+\varepsilon_{\mathrm{gap}} \\
&=\frac{5}{4}\left(J_{\eta,t}(\pi_{\eta,t}^\star)-J_{\eta,t}(\pi_\theta)\right)+\varepsilon_{\mathrm{gap}}.
\end{aligned} J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ≤ ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) + 4 1 ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) + ε gap = 4 5 ( J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ) + ε gap .
Since ∥ θ − θ η , t ⋆ ∥ 2 ≤ R \left\|\theta-\theta^\star_{\eta,t}\right\|_2\le R θ − θ η , t ⋆ 2 ≤ R , the local concavity bound Eq. (119) gives
J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ≤ W R 2 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ . J_{\eta,t}(\pi_{\eta,t}^\star)-J_{\eta,t}(\pi_\theta)\le\frac{W_R}{2}\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle. J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ≤ 2 W R ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ .
Substituting this bound for the population objective gap yields
J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π θ ) ≤ 5 W R 8 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + ε g a p . (143) \widehat{J}_{\eta,t}(\pi_{\eta,t}^\star)-\widehat{J}_{\eta,t}(\pi_\theta)\le\frac{5W_R}{8}\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle+\varepsilon_{\mathrm{gap}}. \tag*{(143)} J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ≤ 8 5 W R ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + ε gap . (143)
We next replace the population gradient in this bound by the empirical gradient. The lower bound on the directional gradient error in Eq. (124) implies
⟨ ∇ J ^ η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ≥ ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ − 1 4 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ − ε d i r = 3 4 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ − ε d i r . \begin{aligned}
\left\langle\nabla\widehat{J}_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle\ge\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle-\frac{1}{4}\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle-\varepsilon_{\mathrm{dir}} \\
&=\frac{3}{4}\left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle-\varepsilon_{\mathrm{dir}}.
\end{aligned} ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ≥ ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ − 4 1 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ − ε dir = 4 3 ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ − ε dir .
Moving the error term to the left and dividing by 3 / 4 3/4 3/4 gives
⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ≤ 4 3 ( ⟨ ∇ J ^ η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + ϵ d i r ) . \left\langle\nabla J_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle\le\frac{4}{3}\left(\left\langle\nabla\widehat{J}_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle+\epsilon_{\mathrm{dir}}\right). ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ ≤ 3 4 ( ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + ϵ dir ) .
Substituting into Eq. (143) and distributing the factor 5 W R 8 ⋅ 4 3 = 5 W R 6 \frac{5W_R}{8}\cdot\frac{4}{3}=\frac{5W_R}{6} 8 5 W R ⋅ 3 4 = 6 5 W R , we obtain
J ^ η , t ( π η , t ⋆ ) − J ^ η , t ( π θ ) ≤ 5 W R 8 ⋅ 4 3 ( ⟨ ∇ J ^ η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + ϵ d i r ) + ϵ g a p = 5 W R 6 ⟨ ∇ J ^ η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + 5 W R 6 ϵ d i r + ϵ g a p . (144) \begin{aligned}
\widehat{J}_{\eta,t}(\pi^\star_{\eta,t})-\widehat{J}_{\eta,t}(\pi_\theta)
&\le\frac{5W_R}{8}\cdot\frac{4}{3}\left(\left\langle\nabla\widehat{J}_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle+\epsilon_{\mathrm{dir}}\right)+\epsilon_{\mathrm{gap}}\\
&= \frac{5W_R}{6}\left\langle\nabla\widehat{J}_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle+\frac{5W_R}{6}\epsilon_{\mathrm{dir}}+\epsilon_{\mathrm{gap}}. \tag*{(144)}
\end{aligned} J η , t ( π η , t ⋆ ) − J η , t ( π θ ) ≤ 8 5 W R ⋅ 3 4 ( ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + ϵ dir ) + ϵ gap = 6 5 W R ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ + 6 5 W R ϵ dir + ϵ gap . (144)
Both parameters lie in the ball of radius R R R around θ η , t ⋆ \theta^\star_{\eta,t} θ η , t ⋆ . Smoothness of the empirical objective (Lemma G.7 ) and α ^ L J ^ η , R ≤ 1 \widehat{\alpha}L_{\widehat{J}_{\eta,R}}\le1 α L J η , R ≤ 1 therefore imply
J ^ η , t ( π θ + ) − J ^ η , t ( π θ ) ≥ α ( 1 − α ^ L J ^ η , R 2 ) ∥ ∇ J ^ η , t ( π θ ) ∥ 2 2 ≥ α 2 ∥ ∇ J ^ η , t ( π θ ) ∥ 2 2 . (145) \widehat{J}_{\eta,t}(\pi_{\theta^+})-\widehat{J}_{\eta,t}(\pi_\theta)\ge\alpha\left(1-\frac{\widehat{\alpha}L_{\widehat{J}_{\eta,R}}}{2}\right)\left\|\nabla\widehat{J}_{\eta,t}(\pi_\theta)\right\|_2^2\ge\frac{\alpha}{2}\left\|\nabla\widehat{J}_{\eta,t}(\pi_\theta)\right\|_2^2. \tag*{(145)} J η , t ( π θ + ) − J η , t ( π θ ) ≥ α ( 1 − 2 α L J η , R ) ∇ J η , t ( π θ ) 2 2 ≥ 2 α ∇ J η , t ( π θ ) 2 2 . (145)
Using this bound for the squared gradient term in the update identity gives
⟨ ∇ J ^ η , t ( π θ ) , θ η , t ⋆ − θ ⟩ = ∥ θ − θ η , t ⋆ ∥ 2 2 − ∥ θ + − θ η , t ⋆ ∥ 2 2 2 α + α 2 ∥ ∇ J ^ η , t ( π θ ) ∥ 2 2 ≤ ∥ θ − θ η , t ⋆ ∥ 2 2 − ∥ θ + − θ η , t ⋆ ∥ 2 2 2 α + J ^ η , t ( π θ + ) − J ^ η , t ( π θ ) . \begin{aligned}
\left\langle\nabla\widehat{J}_{\eta,t}(\pi_\theta),\theta^\star_{\eta,t}-\theta\right\rangle
= \frac{\|\theta-\theta^\star_{\eta,t}\|_2^2-\|\theta^+-\theta^\star_{\eta,t}\|_2^2}{2\alpha}+\frac{\alpha}{2}\left\|\nabla\widehat{J}_{\eta,t}(\pi_\theta)\right\|_2^2 \\
&\le\frac{\|\theta-\theta^\star_{\eta,t}\|_2^2-\|\theta^+-\theta^\star_{\eta,t}\|_2^2}{2\alpha}+\widehat{J}_{\eta,t}(\pi_{\theta^+})-\widehat{J}_{\eta,t}(\pi_\theta).
\end{aligned} ⟨ ∇ J η , t ( π θ ) , θ η , t ⋆ − θ ⟩ = 2 α ∥ θ − θ η , t ⋆ ∥ 2 2 − ∥ θ + − θ η , t ⋆ ∥ 2 2 + 2 α ∇ J η , t ( π θ ) 2 2 ≤ 2 α ∥ θ − θ η , t ⋆ ∥ 2 2 − ∥ θ + − θ η , t ⋆ ∥ 2 2 + J η , t ( π θ + ) − J η , t ( π θ ) .
Substituting into Eq. (144) proves the claim.
Proof of 3.1
We now state the full version of Corollary 3.1. Its proof combines the inner loop guarantee with the KL divergence criterion in Theorem D.1, which bounds suboptimality and coverage loss in terms of the error in each approximate mirror descent update. By Lemma G.2 , an inner objective gap of at most ϵ i n / η \epsilon_{\mathrm{in}}/\eta ϵ in / η gives the required KL tolerance ϵ i n \epsilon_{\mathrm{in}} ϵ in . Under Assumption 2.1, 0 ≤ r ( x , y ) = ϕ ( x , y ) ⊤ θ ⋆ ≤ B ϕ B ⋆ 0\le r(x,y)=\phi(x,y)^\top\theta^\star\le B_\phi B_\star 0 ≤ r ( x , y ) = ϕ ( x , y ) ⊤ θ ⋆ ≤ B ϕ B ⋆ , so we take the reward bound to satisfy R max ≤ B ϕ B ⋆ R_{\max}\le B_\phi B_\star R m a x ≤ B ϕ B ⋆ .
Corollary G.1 (PPO-NoClip is coverage-preserving). Suppose that Assumption 2.1 holds and C b a s e < ∞ C_{\mathrm{base}}<\infty C base < ∞ . Fix ϵ ∈ ( 0 , R max ] \epsilon\in(0,R_{\max}] ϵ ∈ ( 0 , R m a x ] and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Run Algorithm 1 from π 0 = π b a s e = π θ 0 \pi_0=\pi_{\mathrm{base}}=\pi_{\theta_0} π 0 = π base = π θ 0 with
η = 1 B ϕ B ⋆ , T = ⌈ B ϕ B ⋆ log ( 2 C b a s e ) ϵ ⌉ , α = 1 60 e 5 B ϕ 3 B ⋆ , \eta=\frac{1}{B_\phi B_\star},\qquad T=\left\lceil\frac{B_\phi B_\star\log(2C_{\mathrm{base}})}{\epsilon}\right\rceil,\qquad\alpha=\frac{1}{60e^5B_\phi^3B_\star}, η = B ϕ B ⋆ 1 , T = ⌈ ϵ B ϕ B ⋆ log ( 2 C base ) ⌉ , α = 60 e 5 B ϕ 3 B ⋆ 1 ,
ϵ i n = 1 64 C b a s e T 2 log ( e T ) , (146) \epsilon_{\mathrm{in}}=\frac{1}{64C_{\mathrm{base}}T^2\log(eT)}, \tag*{(146)} ϵ in = 64 C base T 2 log ( e T ) 1 , (146)
K = ⌈ 225 e 9 ϵ i n ⌉ , N = ⌈ 4 K ( d log ( 80 K d ) + log 8 T δ ) ⌉ . (147) K=\left\lceil\frac{225e^9}{\epsilon_{\mathrm{in}}}\right\rceil,\qquad N=\left\lceil4K\left(d\log(80Kd)+\log\frac{8T}{\delta}\right)\right\rceil. \tag*{(147)} K = ⌈ ϵ in 225 e 9 ⌉ , N = ⌈ 4 K ( d log ( 80 K d ) + log δ 8 T ) ⌉ . (147)
Here ϵ i n \epsilon_{\mathrm{in}} ϵ in is the tolerance on D K L ( π t + 1 ∥ π η , t ⋆ ) D_{\mathrm{KL}}(\pi_{t+1}\mathbin\Vert\pi^\star_{\eta,t}) D KL ( π t + 1 ∥ π η , t ⋆ ) in Eq. (72) . With probability at least 1 − δ 1-\delta 1 − δ , the returned policy satisfies
J ⋆ − J ( π p o s t ) ≤ ϵ . J^\star-J(\pi_{\mathrm{post}})\le\epsilon. J ⋆ − J ( π post ) ≤ ϵ .
On the same event, the policy sequence satisfies
P x ∼ μ ( π t ( Y x ⋆ ∣ x ) ≥ 1 2 π b a s e ( Y x ⋆ ∣ x ) for all 0 ≤ t ≤ T ) ≥ 1 − 2 + 2 8 log ( e T ) . (148) \mathbb{P}_{x\sim\mu}\left(\pi_t(Y^\star_x\mid x)\ge\frac{1}{2}\pi_{\mathrm{base}}(Y^\star_x\mid x)\text{ for all }0\le t\le T\right)\ge1-\frac{2+\sqrt{2}}{8\sqrt{\log(eT)}}. \tag*{(148)} P x ∼ μ ( π t ( Y x ⋆ ∣ x ) ≥ 2 1 π base ( Y x ⋆ ∣ x ) for all 0 ≤ t ≤ T ) ≥ 1 − 8 log ( e T ) 2 + 2 . (148)
The total numbers of gradient steps and sampled responses satisfy
N s t e p ( ϵ , δ ) = O ~ ( C b a s e ( B ϕ B ⋆ ) 3 ϵ 3 ) , N s a m p l e ( ϵ , δ ) = O ~ ( C b a s e ( B ϕ B ⋆ ) 3 ( d + log ( 1 / δ ) ) ϵ 3 ) . (149) N_{\mathrm{step}}(\epsilon,\delta)=\widetilde{O}\left(\frac{C_{\mathrm{base}}(B_\phi B_\star)^3}{\epsilon^3}\right),\qquad N_{\mathrm{sample}}(\epsilon,\delta)=\widetilde{O}\left(\frac{C_{\mathrm{base}}(B_\phi B_\star)^3(d+\log(1/\delta))}{\epsilon^3}\right). \tag*{(149)} N step ( ϵ , δ ) = O ( ϵ 3 C base ( B ϕ B ⋆ ) 3 ) , N sample ( ϵ , δ ) = O ( ϵ 3 C base ( B ϕ B ⋆ ) 3 ( d + log ( 1/ δ )) ) . (149)
Proof of Corollary G.1. We first verify that every inner loop reaches the KL tolerance required by Theorem D.1. Condition on the history before outer iteration t t t and apply Theorem G.1 to its fresh batch with failure probability δ / T \delta/T δ / T . To verify the sample condition, the choice of N N N in Eq. (147) gives
4 K ( d log ( 80 K d ) + log 8 T δ ) ≤ N ≤ 5 K ( d log ( 80 K d ) + log 8 T δ ) . 4K\left(d\log(80Kd)+\log\frac{8T}{\delta}\right)\le N\le5K\left(d\log(80Kd)+\log\frac{8T}{\delta}\right). 4 K ( d log ( 80 K d ) + log δ 8 T ) ≤ N ≤ 5 K ( d log ( 80 K d ) + log δ 8 T ) .
Consequently,
log ( 8 T [ 5 ( 1 + 2 N ) ] d δ ) ≤ log 8 T δ + d log ( 75 K ( d log ( 80 K d ) + log 8 T δ ) ) = log 8 T δ + d log ( 80 K d ) + d log ( 15 16 d ( d log ( 80 K d ) + log 8 T δ ) ) \begin{aligned}
\log\left(\frac{8T[5(1+2N)]^{d}}{\delta}\right)\le\log\frac{8T}{\delta}+d\log\left(75K\left(d\log(80Kd)+\log\frac{8T}{\delta}\right)\right) \\
&=\log\frac{8T}{\delta}+d\log(80Kd)+d\log\left(\frac{15}{16d}\left(d\log(80Kd)+\log\frac{8T}{\delta}\right)\right)
\end{aligned} log ( δ 8 T [ 5 ( 1 + 2 N ) ] d ) ≤ log δ 8 T + d log ( 75 K ( d log ( 80 K d ) + log δ 8 T ) ) = log δ 8 T + d log ( 80 K d ) + d log ( 16 d 15 ( d log ( 80 K d ) + log δ 8 T ) )
≤ 2 ( d log ( 80 K d ) + log 8 T δ ) , \le2\left(d\log(80Kd)+\log\frac{8T}{\delta}\right), ≤ 2 ( d log ( 80 K d ) + log δ 8 T ) ,
where the last inequality uses log z ≤ z \log z\le z log z ≤ z for z > 0 z>0 z > 0 . This verifies Eq. (134) and bounds the logarithmic term in Eq. (136) by N / ( 2 K ) N/(2K) N / ( 2 K ) . Therefore, with conditional probability at least 1 − δ / T 1-\delta/T 1 − δ / T ,
J η , t ( π η , t ⋆ ) − J η , t ( π t + 1 ) ≤ 150 e 9 B ϕ B ⋆ ( 1 K + 1 2 K ) = 225 e 9 B ϕ B ⋆ K ≤ ε i n η . J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_{t+1})\le150e^{9}B_{\phi}B_{\star}\left(\frac{1}{K}+\frac{1}{2K}\right)=\frac{225e^{9}B_{\phi}B_{\star}}{K}\le\frac{\varepsilon_{\mathrm{in}}}{\eta}. J η , t ( π η , t ⋆ ) − J η , t ( π t + 1 ) ≤ 150 e 9 B ϕ B ⋆ ( K 1 + 2 K 1 ) = K 225 e 9 B ϕ B ⋆ ≤ η ε in .
Taking expectations over the history and a union bound over outer iterations shows that this holds simultaneously for all 0 ≤ t < T 0\le t<T 0 ≤ t < T with probability at least 1 − δ 1-\delta 1 − δ .
We next use the KL error bound to prove reward convergence and coverage preservation. On this event, Lemma G.2 gives
D K L ( π t + 1 ∥ π η , t ⋆ ) = η ( J η , t ( π η , t ⋆ ) − J η , t ( π t + 1 ) ) ≤ ε i n = 1 64 C b a s e T 2 log ( e T ) , 0 ≤ t < T . D_{\mathrm{KL}}(\pi_{t+1}\mathbin\Vert\pi_{\eta,t}^{\star})=\eta\left(J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_{t+1})\right)\le\varepsilon_{\mathrm{in}}=\frac{1}{64C_{\mathrm{base}}T^{2}\log(eT)},\qquad0\le t<T. D KL ( π t + 1 ∥ π η , t ⋆ ) = η ( J η , t ( π η , t ⋆ ) − J η , t ( π t + 1 ) ) ≤ ε in = 64 C base T 2 log ( e T ) 1 , 0 ≤ t < T .
The policies are nonzero everywhere under the softmax parameterization and π 0 = π b a s e \pi_{0}=\pi_{\mathrm{base}} π 0 = π base , so Theorem D.1 applies. Using Jensen’s inequality and ∑ t = 0 T − 1 ( t + 1 ) − 1 ≤ log ( e T ) \sum_{t=0}^{T-1}(t+1)^{-1}\le\log(eT) ∑ t = 0 T − 1 ( t + 1 ) − 1 ≤ log ( e T ) , we obtain
J ⋆ − J ( π p o s t ) ≤ log C b a s e η T + T ε i n η + 2 e η C b a s e ε i n log ( e T ) ≤ log C b a s e + 1 / 64 + e / 4 η T ≤ log ( 2 C b a s e ) η T ≤ ε . \begin{aligned}
J^{\star}-J(\pi_{\mathrm{post}})\le\frac{\log C_{\mathrm{base}}}{\eta T}+\frac{T\varepsilon_{\mathrm{in}}}{\eta}+\frac{2\sqrt{e}}{\eta}\sqrt{C_{\mathrm{base}}\varepsilon_{\mathrm{in}}\log(eT)} \\
&\le\frac{\log C_{\mathrm{base}}+1/64+\sqrt{e}/4}{\eta T}\le\frac{\log(2C_{\mathrm{base}})}{\eta T}\le\varepsilon.
\end{aligned} J ⋆ − J ( π post ) ≤ η T log C base + η T ε in + η 2 e C base ε in log ( e T ) ≤ η T log C base + 1/64 + e /4 ≤ η T log ( 2 C base ) ≤ ε .
The second line substitutes ε i n \varepsilon_{\mathrm{in}} ε in and the choice of T T T .
For coverage, apply Eq. (71) with c = 1 / 2 c=1/2 c = 1/2 on the same event. Since each KL error is at most ε i n \varepsilon_{\mathrm{in}} ε in ,
P x ∼ μ ( π t ( y x ⋆ ∣ x ) ≥ 1 2 π b a s e ( y x ⋆ ∣ x ) for all 0 ≤ t ≤ T ) ≥ 1 − ( 2 + 2 ) T C b a s e ε i n = 1 − 2 + 2 8 log ( e T ) . \mathbb{P}_{x\sim\mu}\left(\pi_{t}(y_{x}^{\star}\mid x)\ge\frac{1}{2}\pi_{\mathrm{base}}(y_{x}^{\star}\mid x)\text{ for all }0\le t\le T\right)\ge1-(2+\sqrt{2})T\sqrt{C_{\mathrm{base}}\varepsilon_{\mathrm{in}}}=1-\frac{2+\sqrt{2}}{8\sqrt{\log(eT)}}. P x ∼ μ ( π t ( y x ⋆ ∣ x ) ≥ 2 1 π base ( y x ⋆ ∣ x ) for all 0 ≤ t ≤ T ) ≥ 1 − ( 2 + 2 ) T C base ε in = 1 − 8 log ( e T ) 2 + 2 .
This proves Eq. (148) .
Finally, each outer iteration uses K K K gradient steps and N N N sampled responses, so the total counts are T K TK T K and T N TN T N , respectively. Since ε ≤ R max ≤ B ϕ B ⋆ \varepsilon\le R_{\max}\le B_{\phi}B_{\star} ε ≤ R m a x ≤ B ϕ B ⋆ and C b a s e ≥ 1 C_{\mathrm{base}}\ge1 C base ≥ 1 , the choices of T T T and ε i n \varepsilon_{\mathrm{in}} ε in in Eq. (146) give
B ϕ B ⋆ log ( 2 C b a s e ) ε ≤ T ≤ 3 B ϕ B ⋆ log ( 2 C b a s e ) ε , 1 ε i n = 64 C b a s e T 2 log ( e T ) = O ~ ( C b a s e ( B ϕ B ⋆ ) 2 ε 2 ) . \frac{B_{\phi}B_{\star}\log(2C_{\mathrm{base}})}{\varepsilon}\le T\le\frac{3B_{\phi}B_{\star}\log(2C_{\mathrm{base}})}{\varepsilon},\qquad\frac{1}{\varepsilon_{\mathrm{in}}}=64C_{\mathrm{base}}T^{2}\log(eT)=\widetilde{O}\left(\frac{C_{\mathrm{base}}(B_{\phi}B_{\star})^{2}}{\varepsilon^{2}}\right). ε B ϕ B ⋆ log ( 2 C base ) ≤ T ≤ ε 3 B ϕ B ⋆ log ( 2 C base ) , ε in 1 = 64 C base T 2 log ( e T ) = O ( ε 2 C base ( B ϕ B ⋆ ) 2 ) .
Substituting these bounds into the definitions of K K K and N N N in Eq. (147) gives
K = O ~ ( C b a s e ( B ϕ B ⋆ ) 2 ε 2 ) , N = O ~ ( K ( d + log ( T / δ ) ) ) . K=\widetilde{O}\left(\frac{C_{\mathrm{base}}(B_{\phi}B_{\star})^{2}}{\varepsilon^{2}}\right),\qquad N=\widetilde{O}\left(K\left(d+\log(T/\delta)\right)\right). K = O ( ε 2 C base ( B ϕ B ⋆ ) 2 ) , N = O ( K ( d + log ( T / δ ) ) ) .
Multiplying by T T T proves Eq. (149) .
Corollary 3.1 follows from Corollary G.1 upon writing B = B ϕ B ⋆ B=B_{\phi}B_{\star} B = B ϕ B ⋆ and choosing K , N K,N K , N as in Eq. (147) . The explicit learning rate is α = 1 / ( 60 e 5 B ϕ 2 B ) = Θ ( 1 / ( B ϕ 2 B ) ) \alpha=1/(60e^{5}B_{\phi}^{2}B)=\Theta(1/(B_{\phi}^{2}B)) α = 1/ ( 60 e 5 B ϕ 2 B ) = Θ ( 1/ ( B ϕ 2 B )) , and Eq. (149) gives the stated gradient step and sample bounds.
PPO-Lite: Global Convergence (Proof of Corollary 3.2)
This section proves Corollary 3.2 for PPO-Lite (Algorithm 2). The section is organized as follows.
Appendix § 14.2 .§ 1 develops the supporting lemmas for the clipped objective.
Appendix § 14.2 .§ 2 states the full corollary and proves global convergence and coverage.
Appendix § 14.2 .§ 3.1 gives the gradient of the clipped objective.
Appendix § 14.2 .§ 3.2 proves inner and global convergence for softmax linear policies with an estimated KL penalty and sufficiently wide clipping thresholds.
Appendix § 14.2 .§ 4 compares the objectives of PPO-NoClip and PPO-Lite with those of PPO and GRPO.
We assume throughout this section that the policy class is finite with cardinality ∣ Π Θ ∣ |\Pi_{\Theta}| ∣ Π Θ ∣ . Extensions to continuous policy classes require uniform concentration bounds for the chosen class.
Notation and objectives. Throughout this section, we use signed rewards,
− R max ≤ r ( x , y ) ≤ R max , ∀ ( x , y ) ∈ X × Y . -R_{\max}\le r(x,y)\le R_{\max},\qquad\forall(x,y)\in\mathcal{X}\times\mathcal{Y}. − R m a x ≤ r ( x , y ) ≤ R m a x , ∀ ( x , y ) ∈ X × Y .
For an outer iteration t t t , a regularization parameter η > 0 \eta>0 η > 0 , and π t > 0 \pi_t>0 π t > 0 , write
w t π ( x , y ) : = π ( y ∣ x ) π t ( y ∣ x ) , J η , t ( π ) : = J ( π ) − 1 η D K L ( π ∥ π t ) . w_t^\pi(x,y):=\frac{\pi(y\mid x)}{\pi_t(y\mid x)},\qquad J_{\eta,t}(\pi):=J(\pi)-\frac{1}{\eta}D_{\mathrm{KL}}(\pi\parallel\pi_t). w t π ( x , y ) := π t ( y ∣ x ) π ( y ∣ x ) , J η , t ( π ) := J ( π ) − η 1 D KL ( π ∥ π t ) .
For 0 ≤ τ − < 1 0\le\tau_-<1 0 ≤ τ − < 1 and τ + ≥ 0 \tau_+\ge0 τ + ≥ 0 , use the clipping thresholds from Algorithm 2,
[ z ] 1 − τ − 1 + τ + : = min { 1 + τ + , max { 1 − τ − , z } } . [z]_{1-\tau_-}^{1+\tau_+}:=\min\{1+\tau_+,\max\{1-\tau_-,z\}\}. [ z ] 1 − τ − 1 + τ + := min { 1 + τ + , max { 1 − τ − , z }} .
Recall the population objective in Eq. (14) ,
J t η , c l i p ( π ) : = E π t [ min { w t π ( x , y ) r ( x , y ) , [ w t π ( x , y ) ] 1 − τ − 1 + τ + r ( x , y ) } ] − 1 η D K L ( π ∥ π t ) . (150) J_t^{\eta,\mathrm{clip}}(\pi):=\mathbb{E}_{\pi_t}\left[\min\left\{w_t^\pi(x,y)r(x,y),[w_t^\pi(x,y)]_{1-\tau_-}^{1+\tau_+}r(x,y)\right\}\right]-\frac{1}{\eta}D_{\mathrm{KL}}(\pi\parallel\pi_t). \tag*{(150)} J t η , clip ( π ) := E π t [ min { w t π ( x , y ) r ( x , y ) , [ w t π ( x , y ) ] 1 − τ − 1 + τ + r ( x , y ) } ] − η 1 D KL ( π ∥ π t ) . (150)
Supporting Lemmas
The supporting lemmas are as follows.
Lemma H.1 derives clipping thresholds that guarantee π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ maximizes J t η , c l i p J_t^{\eta,\mathrm{clip}} J t η , clip , which then implies that maximizing J t η , c l i p J_t^{\eta,\mathrm{clip}} J t η , clip controls D K L ( π ∥ π η , t ⋆ ) D_{\mathrm{KL}}(\pi\parallel\pi_{\eta,t}^{\star}) D KL ( π ∥ π η , t ⋆ ) .
Lemma H.2 proves the concentration inequalities for J ^ t η , c l i p \widehat{J}_t^{\eta,\mathrm{clip}} J t η , clip .
Lemma H.3 then bounds suboptimality on J t η , c l i p J_t^{\eta,\mathrm{clip}} J t η , clip by the statistical error and the empirical ε o p t \varepsilon_{\mathrm{opt}} ε opt (Line 5).
Throughout the rest of this subsection, fix η \eta η , t t t , and a given π t \pi_t π t .
Lemma H.1. If
1 − τ − ≤ e − 2 η R max , 1 + τ + ≥ e 2 η R max , 1-\tau_-\le e^{-2\eta R_{\max}},\qquad1+\tau_+\ge e^{2\eta R_{\max}}, 1 − τ − ≤ e − 2 η R m a x , 1 + τ + ≥ e 2 η R m a x ,
then it holds that 1 − τ − ≤ π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) ≤ 1 + τ + 1-\tau_-\le\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_t(y\mid x)}\le1+\tau_+ 1 − τ − ≤ π t ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ≤ 1 + τ + for all ( x , y ) (x,y) ( x , y ) , that is, π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ is not clipped, and for every π ∈ Π Θ \pi\in\Pi_{\Theta} π ∈ Π Θ
J t η , c l i p ( π η , t ⋆ ) − J t η , c l i p ( π ) ≥ J η , t ( π η , t ⋆ ) − J η , t ( π ) = 1 η D K L ( π ∥ π η , t ⋆ ) . (151) J_t^{\eta,\mathrm{clip}}(\pi_{\eta,t}^{\star})-J_t^{\eta,\mathrm{clip}}(\pi)\ge J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi)=\frac{1}{\eta}D_{\mathrm{KL}}(\pi\parallel\pi_{\eta,t}^{\star}). \tag*{(151)} J t η , clip ( π η , t ⋆ ) − J t η , clip ( π ) ≥ J η , t ( π η , t ⋆ ) − J η , t ( π ) = η 1 D KL ( π ∥ π η , t ⋆ ) . (151)
Proof of Lemma H.1. Recall that π η , t ⋆ ( y ∣ x ) = π t ( y ∣ x ) e η r ( x , y ) E y ′ ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ′ ) ] \pi_{\eta,t}^{\star}(y \mid x) = \frac{\pi_t(y \mid x)e^{\eta r(x,y)}}{\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[e^{\eta r(x,y')]} } π η , t ⋆ ( y ∣ x ) = E y ′ ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ′ )] π t ( y ∣ x ) e η r ( x , y ) . Since − R max ≤ r ( x , y ) ≤ R max -R_{\max} \le r(x,y) \le R_{\max} − R m a x ≤ r ( x , y ) ≤ R m a x , we have
e − η R max ≤ E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] ≤ e η R max , e^{-\eta R_{\max}} \le\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta r(x,y)}\right] \le e^{\eta R_{\max}}, e − η R m a x ≤ E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] ≤ e η R m a x ,
e − 2 η R max ≤ w t π η , t ⋆ ( x , y ) = e η r ( x , y ) E y ′ ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ′ ) ] ≤ e 2 η R max . e^{-2\eta R_{\max}} \le w_t^{\pi^\star_{\eta,t}}(x,y) = \frac{e^{\eta r(x,y)}}{\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[e^{\eta r(x,y')]}} \le e^{2\eta R_{\max}}. e − 2 η R m a x ≤ w t π η , t ⋆ ( x , y ) = E y ′ ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ′ )] e η r ( x , y ) ≤ e 2 η R m a x .
Thus clipping leaves the objective value for π η , t ⋆ \pi^\star_{\eta,t} π η , t ⋆ unchanged. For any other policy, taking the minimum in Eq. (150) can only decrease its objective value, so
J t η , c l i p ( π η , t ⋆ ) − J η , t ( π η , t ⋆ ) = 0 , J t η , c l i p ( π ) − J η , t ( π ) ≤ 0. J_t^{\eta,\mathrm{clip}}(\pi^\star_{\eta,t})-J_{\eta,t}(\pi^\star_{\eta,t})=0,\qquad J_t^{\eta,\mathrm{clip}}(\pi)-J_{\eta,t}(\pi)\le0. J t η , clip ( π η , t ⋆ ) − J η , t ( π η , t ⋆ ) = 0 , J t η , clip ( π ) − J η , t ( π ) ≤ 0.
Combining these inequalities proves the first part of Eq. (151) , and the second follows from the identity established for the unclipped objective in Lemma G.2 .
Lemma H.2. Suppose that 1 − τ − ≤ e − 2 η R max 1-\tau_- \le e^{-2\eta R_{\max}} 1 − τ − ≤ e − 2 η R m a x and 1 + τ + ≥ e 2 η R max 1+\tau_+ \ge e^{2\eta R_{\max}} 1 + τ + ≥ e 2 η R m a x . For any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every π ∈ Π Θ \pi\in\Pi_\Theta π ∈ Π Θ ,
J t η , c l i p ( π η , t ⋆ ) − J t η , c l i p ( π ) ≤ 4 ( J ^ t η , c l i p ( π η , t ⋆ ) − J ^ t η , c l i p ( π ) ) + ε s t a t ( δ ) , (152) J_t^{\eta,\mathrm{clip}}(\pi^\star_{\eta,t})-J_t^{\eta,\mathrm{clip}}(\pi) \le4\left(\widehat{J}_t^{\eta,\mathrm{clip}}(\pi^\star_{\eta,t})-\widehat{J}_t^{\eta,\mathrm{clip}}(\pi)\right)+\varepsilon_{\mathrm{stat}}(\delta), \tag*{(152)} J t η , clip ( π η , t ⋆ ) − J t η , clip ( π ) ≤ 4 ( J t η , clip ( π η , t ⋆ ) − J t η , clip ( π ) ) + ε stat ( δ ) , (152) where
ε s t a t ( δ ) : = ( 16 η R max max { 1 + τ + , e 1 + 3 η R max } + 8 3 ( 1 + τ + + e 1 + 3 η R max ) ) × R max log ( ∣ Π Θ ∣ / δ ) N . (153) \varepsilon_{\mathrm{stat}}(\delta):=\left(16\eta R_{\max}\max\{1+\tau_+,e^{1+3\eta R_{\max}}\}+\frac{8}{3}(1+\tau_++e^{1+3\eta R_{\max}})\right)\times\frac{R_{\max}\log(|\Pi_\Theta|/\delta)}{N}. \tag*{(153)} ε stat ( δ ) := ( 16 η R m a x max { 1 + τ + , e 1 + 3 η R m a x } + 3 8 ( 1 + τ + + e 1 + 3 η R m a x ) ) × N R m a x log ( ∣ Π Θ ∣/ δ ) . (153)
Proof of Lemma H.2. We will bound the difference between the population and empirical objective gaps uniformly over Π Θ \Pi_\Theta Π Θ . Fix a policy π ∈ Π Θ \pi\in\Pi_\Theta π ∈ Π Θ . Write the population and empirical objective gaps as
Δ π : = J t η , c l i p ( π η , t ⋆ ) − J t η , c l i p ( π ) , Δ ^ π : = J ^ t η , c l i p ( π η , t ⋆ ) − J ^ t η , c l i p ( π ) . \Delta_\pi:=J_t^{\eta,\mathrm{clip}}(\pi^\star_{\eta,t})-J_t^{\eta,\mathrm{clip}}(\pi),\qquad\widehat{\Delta}_\pi:=\widehat{J}_t^{\eta,\mathrm{clip}}(\pi^\star_{\eta,t})-\widehat{J}_t^{\eta,\mathrm{clip}}(\pi). Δ π := J t η , clip ( π η , t ⋆ ) − J t η , clip ( π ) , Δ π := J t η , clip ( π η , t ⋆ ) − J t η , clip ( π ) .
We write E t = E π t \mathbb{E}_t=\mathbb{E}_{\pi_t} E t = E π t , and let E ^ t \widehat{\mathbb{E}}_t E t denote the empirical average over D t \mathcal{D}_t D t . Since both J t η , c l i p ( π ) J_t^{\eta,\mathrm{clip}}(\pi) J t η , clip ( π ) and J ^ t η , c l i p ( π ) \widehat{J}_t^{\eta,\mathrm{clip}}(\pi) J t η , clip ( π ) use the population KL divergence − η − 1 D K L ( π ∥ π t ) -\eta^{-1}D_{\mathrm{KL}}(\pi\parallel\pi_t) − η − 1 D KL ( π ∥ π t ) , this term cancels when we subtract the two gaps above. Moreover, the clipping thresholds ensure that π η , t ⋆ \pi^\star_{\eta,t} π η , t ⋆ is not clipped (Lemma H.1). We therefore have
Δ π − Δ ^ π = ( E t − E ^ t ) [ Z π ( x , y ) ] , (154) \Delta_\pi-\widehat{\Delta}_\pi=(\mathbb{E}_t-\widehat{\mathbb{E}}_t)[Z_\pi(x,y)], \tag*{(154)} Δ π − Δ π = ( E t − E t ) [ Z π ( x , y )] , (154)
where
Z π ( x , y ) : = w t π η , t ⋆ ( x , y ) r ( x , y ) − min { w t π ( x , y ) r ( x , y ) , [ w t π ( x , y ) ] 1 + τ + 1 − τ − r ( x , y ) } . Z_\pi(x,y):=w_t^{\pi^\star_{\eta,t}}(x,y)r(x,y)-\min\left\{w_t^\pi(x,y)r(x,y),[w_t^\pi(x,y)]_{\frac{1+\tau_+}{1-\tau_-}}r(x,y)\right\}. Z π ( x , y ) := w t π η , t ⋆ ( x , y ) r ( x , y ) − min { w t π ( x , y ) r ( x , y ) , [ w t π ( x , y ) ] 1 − τ − 1 + τ + r ( x , y ) } .
For this proof, define
r + ( x , y ) : = max { r ( x , y ) , 0 } , r − ( x , y ) : = max { − r ( x , y ) , 0 } . r_+(x,y):=\max\{r(x,y),0\},\qquad r_-(x,y):=\max\{-r(x,y),0\}. r + ( x , y ) := max { r ( x , y ) , 0 } , r − ( x , y ) := max { − r ( x , y ) , 0 } .
We expand the clipped reward according to the sign of the reward, rewriting it in terms of r + ( x , y ) r_+(x,y) r + ( x , y ) and r − ( x , y ) r_-(x,y) r − ( x , y ) ,
min { w t π ( x , y ) r ( x , y ) , [ w t π ( x , y ) ] 1 + τ + 1 − τ − r ( x , y ) } = r + ( x , y ) min { w t π ( x , y ) , 1 + τ + } − r − ( x , y ) max { w t π ( x , y ) , 1 − τ − } . \min\left\{w_t^\pi(x,y)r(x,y),[w_t^\pi(x,y)]_{\frac{1+\tau_+}{1-\tau_-}}r(x,y)\right\}=r_+(x,y)\min\{w_t^\pi(x,y),1+\tau_+\}-r_-(x,y)\max\{w_t^\pi(x,y),1-\tau_-\}. min { w t π ( x , y ) r ( x , y ) , [ w t π ( x , y ) ] 1 − τ − 1 + τ + r ( x , y ) } = r + ( x , y ) min { w t π ( x , y ) , 1 + τ + } − r − ( x , y ) max { w t π ( x , y ) , 1 − τ − } .
Substituting into the definition of Z π ( x , y ) Z_\pi(x,y) Z π ( x , y ) gives
Z π ( x , y ) = r + ( x , y ) w t π η , t ⋆ ( x , y ) ( 1 − min { π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) , 1 + τ + w t π η , t ⋆ ( x , y ) } ) + r − ( x , y ) w t π η , t ⋆ ( x , y ) ( max { π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) , 1 − τ − w t π η , t ⋆ ( x , y ) } − 1 ) . (155) Z_\pi(x,y)=r_+(x,y)w_t^{\pi^\star_{\eta,t}}(x,y)\left(1-\min\left\{\frac{\pi(y\mid x)}{\pi^\star_{\eta,t}(y\mid x)},\frac{1+\tau_+}{w_t^{\pi^\star_{\eta,t}}(x,y)}\right\}\right)+r_-(x,y)w_t^{\pi^\star_{\eta,t}}(x,y)\left(\max\left\{\frac{\pi(y\mid x)}{\pi^\star_{\eta,t}(y\mid x)},\frac{1-\tau_-}{w_t^{\pi^\star_{\eta,t}}(x,y)}\right\}-1\right). \tag*{(155)} Z π ( x , y ) = r + ( x , y ) w t π η , t ⋆ ( x , y ) ( 1 − min { π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) , w t π η , t ⋆ ( x , y ) 1 + τ + } ) + r − ( x , y ) w t π η , t ⋆ ( x , y ) ( max { π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) , w t π η , t ⋆ ( x , y ) 1 − τ − } − 1 ) . (155)
In Eq. (155) , the second term containing r + ( x , y ) r_{+}(x,y) r + ( x , y ) is bounded, whereas the first term containing r − ( x , y ) r_{-}(x,y) r − ( x , y ) can grow with π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) \frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)} π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) . To apply Bernstein’s inequality, we first replace Z π ( x , y ) Z_{\pi}(x,y) Z π ( x , y ) by a bounded random variable. Letting M η : = e 1 + 2 η R max M_{\eta}:=e^{1+2\eta R_{\max}} M η := e 1 + 2 η R m a x , we define this to be
Z ~ π ( x , y ) : = Z π ( x , y ) I [ r ( x , y ) ≥ 0 or π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ≤ M η ] . (156) \widetilde{Z}_{\pi}(x,y):=Z_{\pi}(x,y)\mathbb{I}\left[r(x,y)\geq0\ \text{or}\ \frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\leq M_{\eta}\right]. \tag*{(156)} Z π ( x , y ) := Z π ( x , y ) I [ r ( x , y ) ≥ 0 or π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) ≤ M η ] . (156)
We next bound the second moment of Z ~ π ( x , y ) \widetilde{Z}_{\pi}(x,y) Z π ( x , y ) in terms of Δ π \Delta_{\pi} Δ π , which will control the variance in Bernstein’s inequality, and the following steps culminate in Eq. (163) . Squaring the two terms with disjoint supports in Eq. (155) and applying the truncation gives
Z ~ π ( x , y ) 2 = r + ( x , y ) 2 ( w η , t ⋆ ( x , y ) − min { w t π ( x , y ) , 1 + τ + } ) 2 + r − ( x , y ) 2 ( max { w t π ( x , y ) , 1 − τ − } − w t π η , t ⋆ ( x , y ) ) 2 I [ π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ≤ M η ] . \begin{aligned}
\widetilde{Z}_{\pi}(x,y)^2={}&r_{+}(x,y)^2\left(w_{\eta,t}^{\star}(x,y)-\min\left\{w_{t}^{\pi}(x,y),1+\tau_{+}\right\}\right)^2\\
&+r_{-}(x,y)^2\left(\max\left\{w_{t}^{\pi}(x,y),1-\tau_{-}\right\}-w_{t}^{\pi_{\eta,t}^{\star}}(x,y)\right)^2\mathbb{I}\left[\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\leq M_{\eta}\right].
\end{aligned} Z π ( x , y ) 2 = r + ( x , y ) 2 ( w η , t ⋆ ( x , y ) − min { w t π ( x , y ) , 1 + τ + } ) 2 + r − ( x , y ) 2 ( max { w t π ( x , y ) , 1 − τ − } − w t π η , t ⋆ ( x , y ) ) 2 I [ π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) ≤ M η ] .
We will bound the above in terms of the KL divergence, for which we first establish useful bounds for Z ~ π \widetilde{Z}_{\pi} Z π , which will use the following inequality. Define h ( u ) : = u log u − u + 1 h(u):=u\log u-u+1 h ( u ) := u log u − u + 1 , with h ( 0 ) = 1 h(0)=1 h ( 0 ) = 1 by continuity. Integrating h ′ ′ ( u ) = 1 / u h^{\prime\prime}(u)=1/u h ′′ ( u ) = 1/ u twice from 1 1 1 , where h ( 1 ) = h ′ ( 1 ) = 0 h(1)=h^{\prime}(1)=0 h ( 1 ) = h ′ ( 1 ) = 0 , gives
h ( u ) : = u log u − u + 1 ≥ ( u − 1 ) 2 2 max { u , 1 } , u ≥ 0. (157) h(u):=u\log u-u+1\geq\frac{(u-1)^2}{2\max\{u,1\}},\qquad u\geq0. \tag*{(157)} h ( u ) := u log u − u + 1 ≥ 2 max { u , 1 } ( u − 1 ) 2 , u ≥ 0. (157)
Next, we handle two cases separately based on the sign of the rewards. Suppose r ( x , y ) ≥ 0 r(x,y)\geq0 r ( x , y ) ≥ 0 . Then Eq. (156) gives Z ~ π ( x , y ) = Z π ( x , y ) \widetilde{Z}_{\pi}(x,y)=Z_{\pi}(x,y) Z π ( x , y ) = Z π ( x , y ) . To apply Eq. (157) , take u = min { w t π ( x , y ) , 1 + τ + } / w η , t ⋆ ( x , y ) u=\min\{w_{t}^{\pi}(x,y),1+\tau_{+}\}/w_{\eta,t}^{\star}(x,y) u = min { w t π ( x , y ) , 1 + τ + } / w η , t ⋆ ( x , y ) . By Lemma H.1, w η , t ⋆ ( x , y ) ≤ 1 + τ + w_{\eta,t}^{\star}(x,y)\leq1+\tau_{+} w η , t ⋆ ( x , y ) ≤ 1 + τ + , so max { 1 , u } ≤ ( 1 + τ + ) / w η , t ⋆ ( x , y ) \max\{1,u\}\leq(1+\tau_{+})/w_{\eta,t}^{\star}(x,y) max { 1 , u } ≤ ( 1 + τ + ) / w η , t ⋆ ( x , y ) . If w t π ( x , y ) ≤ 1 + τ + w_{t}^{\pi}(x,y)\leq1+\tau_{+} w t π ( x , y ) ≤ 1 + τ + , then u = π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) u=\pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x) u = π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ; otherwise, 1 ≤ u ≤ π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) 1\leq u\leq\pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x) 1 ≤ u ≤ π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) . Since h h h is increasing on [ 1 , ∞ ) [1,\infty) [ 1 , ∞ ) , in either case h ( u ) ≤ h ( π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ) h(u)\leq h\left(\pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x)\right) h ( u ) ≤ h ( π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ) . Substituting this choice of u u u into Eq. (157) therefore gives
( w η , t ⋆ ( x , y ) − min { w t π ( x , y ) , 1 + τ + } ) 2 = ( w η , t ⋆ ( x , y ) ) 2 ( 1 − u ) 2 ≤ 2 ( w η , t ⋆ ( x , y ) ) 2 max { 1 , u } h ( u ) ≤ 2 ( 1 + τ + ) w η , t ⋆ ( x , y ) h ( π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ) . \begin{aligned}
\left(w_{\eta,t}^{\star}(x,y)-\min\left\{w_{t}^{\pi}(x,y),1+\tau_{+}\right\}\right)^2
&=\left(w_{\eta,t}^{\star}(x,y)\right)^2(1-u)^2\\
&\leq2\left(w_{\eta,t}^{\star}(x,y)\right)^2\max\{1,u\}h(u)\\
&\leq2(1+\tau_{+})w_{\eta,t}^{\star}(x,y)h\left(\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\right).
\end{aligned} ( w η , t ⋆ ( x , y ) − min { w t π ( x , y ) , 1 + τ + } ) 2 = ( w η , t ⋆ ( x , y ) ) 2 ( 1 − u ) 2 ≤ 2 ( w η , t ⋆ ( x , y ) ) 2 max { 1 , u } h ( u ) ≤ 2 ( 1 + τ + ) w η , t ⋆ ( x , y ) h ( π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) ) .
Multiplying both sides by r ( x , y ) 2 = r + ( x , y ) 2 r(x,y)^2=r_{+}(x,y)^2 r ( x , y ) 2 = r + ( x , y ) 2 and using ∣ r ( x , y ) ∣ ≤ R max \lvert r(x,y)\rvert\leq R_{\max} ∣ r ( x , y )∣ ≤ R m a x on the RHS gives
Z ~ π ( x , y ) 2 ≤ 2 R max 2 ( 1 + τ + ) w η , t ⋆ ( x , y ) h ( π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ) . (158) \widetilde{Z}_{\pi}(x,y)^2\leq2R_{\max}^2(1+\tau_{+})w_{\eta,t}^{\star}(x,y)h\left(\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\right). \tag*{(158)} Z π ( x , y ) 2 ≤ 2 R m a x 2 ( 1 + τ + ) w η , t ⋆ ( x , y ) h ( π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) ) . (158)
Now suppose r ( x , y ) < 0 r(x,y)<0 r ( x , y ) < 0 . If π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) > M η \pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x)>M_{\eta} π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) > M η then Z ~ π ( x , y ) = 0 \widetilde{Z}_{\pi}(x,y)=0 Z π ( x , y ) = 0 by Eq. (156) . Therefore, we assume that both r ( x , y ) < 0 r(x,y)<0 r ( x , y ) < 0 and π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ≤ M η \pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x)\leq M_{\eta} π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ≤ M η for the next set of derivations.
Apply Eq. (157) with u = max { w t π ( x , y ) , 1 − τ − } / w η , t ⋆ ( x , y ) u=\max\{w_{t}^{\pi}(x,y),1-\tau_{-}\}/w_{\eta,t}^{\star}(x,y) u = max { w t π ( x , y ) , 1 − τ − } / w η , t ⋆ ( x , y ) . Since 1 − τ − ≤ w η , t ⋆ ( x , y ) 1-\tau_{-}\leq w_{\eta,t}^{\star}(x,y) 1 − τ − ≤ w η , t ⋆ ( x , y ) by Lemma H.1, we have max { 1 , u } ≤ M η \max\{1,u\}\leq M_{\eta} max { 1 , u } ≤ M η . If w t π ( x , y ) ≥ 1 − τ − w_{t}^{\pi}(x,y)\geq1-\tau_{-} w t π ( x , y ) ≥ 1 − τ − , then u = π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) u=\pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x) u = π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ; otherwise, π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ≤ u ≤ 1 \pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x)\leq u\leq1 π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ≤ u ≤ 1 . Since h h h is decreasing on [ 0 , 1 ] [0,1] [ 0 , 1 ] , in either case h ( u ) ≤ h ( π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ) h(u)\leq h\left(\pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x)\right) h ( u ) ≤ h ( π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ) . Thus,
( max { w t π ( x , y ) , 1 − τ − } − w η , t ⋆ ( x , y ) ) 2 = ( w η , t ⋆ ( x , y ) ) 2 ( u − 1 ) 2 ≤ 2 ( w η , t ⋆ ( x , y ) ) 2 max { 1 , u } h ( u ) ≤ 2 M η ( w η , t ⋆ ( x , y ) ) 2 h ( π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ) . (159) \begin{aligned}
\left(\max\left\{w_{t}^{\pi}(x,y),1-\tau_{-}\right\}-w_{\eta,t}^{\star}(x,y)\right)^2
&=\left(w_{\eta,t}^{\star}(x,y)\right)^2(u-1)^2\\
&\leq2\left(w_{\eta,t}^{\star}(x,y)\right)^2\max\{1,u\}h(u)\\
&\leq2M_{\eta}\left(w_{\eta,t}^{\star}(x,y)\right)^2h\left(\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\right). \tag*{(159)}
\end{aligned} ( max { w t π ( x , y ) , 1 − τ − } − w η , t ⋆ ( x , y ) ) 2 = ( w η , t ⋆ ( x , y ) ) 2 ( u − 1 ) 2 ≤ 2 ( w η , t ⋆ ( x , y ) ) 2 max { 1 , u } h ( u ) ≤ 2 M η ( w η , t ⋆ ( x , y ) ) 2 h ( π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) ) . (159)
The last line above uses π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ≤ M η \pi(y \mid x)/\pi_{\eta,t}^{\star}(y \mid x) \le M_{\eta} π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ≤ M η , as, in the second line, the lower clipping threshold alone does not bound u u u from above. For negative rewards, e η r ( x , y ) ≤ 1 e^{\eta r(x,y)} \le1 e η r ( x , y ) ≤ 1 and r ( x , y ′ ) ≥ − R max r(x,y') \ge-R_{\max} r ( x , y ′ ) ≥ − R m a x for every y ′ y' y ′ , so
w ~ t π η , t ⋆ ( x , y ) = e η r ( x , y ) E y ′ ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ′ ) ] ≤ e η R max . (160) \widetilde{w}_{t}^{\pi_{\eta,t}^{\star}}(x,y) = \frac{e^{\eta r(x,y)}}{\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[e^{\eta r(x,y')}\right]} \le e^{\eta R_{\max}}. \tag*{(160)} w t π η , t ⋆ ( x , y ) = E y ′ ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ′ ) ] e η r ( x , y ) ≤ e η R m a x . (160)
Multiplying Eq. (159) by r ( x , y ) 2 r(x,y)^2 r ( x , y ) 2 and using Eq. (160) gives the following bound when π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ≤ M η \pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x) \le M_{\eta} π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ≤ M η .
Z ~ π ( x , y ) 2 ≤ 2 R max 2 M η e η R max w ~ t π η , t ⋆ ( x , y ) h ( π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ) . (161) \widetilde{Z}_{\pi}(x,y)^2 \le2R_{\max}^2M_{\eta}e^{\eta R_{\max}}\widetilde{w}_{t}^{\pi_{\eta,t}^{\star}}(x,y)h\left(\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\right). \tag*{(161)} Z π ( x , y ) 2 ≤ 2 R m a x 2 M η e η R m a x w t π η , t ⋆ ( x , y ) h ( π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) ) . (161)
Since Z ~ π ( x , y ) = 0 \widetilde{Z}_{\pi}(x,y)=0 Z π ( x , y ) = 0 when π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) > M η \pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x)>M_{\eta} π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) > M η , it therefore holds for all x , y x,y x , y such that r ( x , y ) < 0 r(x,y)<0 r ( x , y ) < 0 .
We now bound the second moment of Z ~ π \widetilde{Z}_{\pi} Z π in terms of the KL divergence. Combining Eq. (158) and Eq. (161) and averaging under π t \pi_t π t gives
E t [ Z ~ π ( x , y ) 2 ] ≤ 2 R max 2 max { 1 + τ + , M η e η R max } E π η , t ⋆ [ h ( π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ) ] . \mathbb{E}_t\left[\widetilde{Z}_{\pi}(x,y)^2\right]\le2R_{\max}^2\max\left\{1+\tau_+,M_{\eta}e^{\eta R_{\max}}\right\}\mathbb{E}_{\pi_{\eta,t}^{\star}}\left[h\left(\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\right)\right]. E t [ Z π ( x , y ) 2 ] ≤ 2 R m a x 2 max { 1 + τ + , M η e η R m a x } E π η , t ⋆ [ h ( π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) ) ] .
For the expectation on the right, since π t ( y ∣ x ) w ~ t π η , t ⋆ ( x , y ) = π η , t ⋆ ( y ∣ x ) \pi_t(y\mid x)\widetilde{w}_{t}^{\pi_{\eta,t}^{\star}}(x,y)=\pi_{\eta,t}^{\star}(y\mid x) π t ( y ∣ x ) w t π η , t ⋆ ( x , y ) = π η , t ⋆ ( y ∣ x ) , first we have the identity that
E π η , t ⋆ [ h ( π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ) ] = E π η , t ⋆ [ π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) log π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) − π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) + 1 ] = D K L ( π ∥ π η , t ⋆ ) . (162) \mathbb{E}_{\pi_{\eta,t}^{\star}}\left[h\left(\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\right)\right]=\mathbb{E}_{\pi_{\eta,t}^{\star}}\left[\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\log\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}-\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}+1\right]
=\mathit{D}_{\mathrm{KL}}(\pi\parallel\pi_{\eta,t}^{\star}). \tag*{(162)} E π η , t ⋆ [ h ( π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) ) ] = E π η , t ⋆ [ π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) log π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) − π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) + 1 ] = D KL ( π ∥ π η , t ⋆ ) . (162)
Combining the above two panels we obtain
E t [ Z ~ π ( x , y ) 2 ] ≤ 2 R max 2 max { 1 + τ + , e 1 + 3 η R max } D K L ( π ∥ π η , t ⋆ ) ≤ 2 η R max 2 max { 1 + τ + , e 1 + 3 η R max } Δ π . (163) \mathbb{E}_t\left[\widetilde{Z}_{\pi}(x,y)^2\right]\le2R_{\max}^2\max\left\{1+\tau_+,e^{1+3\eta R_{\max}}\right\}\mathit{D}_{\mathrm{KL}}(\pi\parallel\pi_{\eta,t}^{\star})
\le2\eta R_{\max}^2\max\left\{1+\tau_+,e^{1+3\eta R_{\max}}\right\}\Delta_{\pi}. \tag*{(163)} E t [ Z π ( x , y ) 2 ] ≤ 2 R m a x 2 max { 1 + τ + , e 1 + 3 η R m a x } D KL ( π ∥ π η , t ⋆ ) ≤ 2 η R m a x 2 max { 1 + τ + , e 1 + 3 η R m a x } Δ π . (163)
The last inequality uses Lemma H.1 and gives our bound on the second moment.
In addition to this, for Bernstein’s inequality we need a bound on the range of the random variables. For r ( x , y ) ≥ 0 r(x,y)\ge0 r ( x , y ) ≥ 0 , use 0 ≤ min { w ~ t π ⋆ ( x , y ) , 1 + τ + } ≤ 1 + τ + 0\le\min\{\widetilde{w}_{t}^{\pi^{\star}}(x,y),1+\tau_+\}\le1+\tau_+ 0 ≤ min { w t π ⋆ ( x , y ) , 1 + τ + } ≤ 1 + τ + and w ~ t π η , t ⋆ ( x , y ) ≤ e 2 η R max \widetilde{w}_{t}^{\pi_{\eta,t}^{\star}}(x,y)\le e^{2\eta R_{\max}} w t π η , t ⋆ ( x , y ) ≤ e 2 η R m a x from Lemma H.1. For r ( x , y ) < 0 r(x,y)<0 r ( x , y ) < 0 with π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ≤ M η \pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x)\le M_{\eta} π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) ≤ M η , use 0 ≤ max { w ~ t π ⋆ ( x , y ) , 1 − τ − } ≤ M η w ~ t π η , t ⋆ ( x , y ) 0\le\max\{\widetilde{w}_{t}^{\pi^{\star}}(x,y),1-\tau_-\}\le M_{\eta}\widetilde{w}_{t}^{\pi_{\eta,t}^{\star}}(x,y) 0 ≤ max { w t π ⋆ ( x , y ) , 1 − τ − } ≤ M η w t π η , t ⋆ ( x , y ) and Eq. (160) . When r ( x , y ) < 0 r(x,y)<0 r ( x , y ) < 0 and π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) > M η \pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x)>M_{\eta} π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) > M η , we have Z ~ π ( x , y ) = 0 \widetilde{Z}_{\pi}(x,y)=0 Z π ( x , y ) = 0 . Together, these bounds give
− ( 1 + τ + ) R max ≤ Z ~ π ( x , y ) ≤ e 2 η R max R max , r ( x , y ) ≥ 0 , -(1+\tau_+)R_{\max}\le\widetilde{Z}_{\pi}(x,y)\le e^{2\eta R_{\max}}R_{\max},\qquad r(x,y)\ge0, − ( 1 + τ + ) R m a x ≤ Z π ( x , y ) ≤ e 2 η R m a x R m a x , r ( x , y ) ≥ 0 ,
− e η R max R max ≤ Z ~ π ( x , y ) ≤ M η e η R max R max , r ( x , y ) < 0. -e^{\eta R_{\max}}R_{\max}\le\widetilde{Z}_{\pi}(x,y)\le M_{\eta}e^{\eta R_{\max}}R_{\max},\qquad r(x,y)<0. − e η R m a x R m a x ≤ Z π ( x , y ) ≤ M η e η R m a x R m a x , r ( x , y ) < 0.
Using 1 + τ + ≥ e 2 η R max 1+\tau_+\ge e^{2\eta R_{\max}} 1 + τ + ≥ e 2 η R m a x and M η = e 1 + 2 η R max M_{\eta}=e^{1+2\eta R_{\max}} M η = e 1 + 2 η R m a x , we then have
− ( 1 + τ + ) R max ≤ Z ~ π ( x , y ) ≤ e 1 + 3 η R max R max , -(1+\tau_+)R_{\max}\le\widetilde{Z}_{\pi}(x,y)\le e^{1+3\eta R_{\max}}R_{\max}, − ( 1 + τ + ) R m a x ≤ Z π ( x , y ) ≤ e 1 + 3 η R m a x R m a x ,
and
∣ Z ~ π ( x , y ) − E t [ Z ~ π ( x , y ) ] ∣ ≤ R max ( 1 + τ + + e 1 + 3 η R max ) . \left|\widetilde{Z}_{\pi}(x,y)-\mathbb{E}_t[\widetilde{Z}_{\pi}(x,y)]\right|\le R_{\max}\left(1+\tau_++e^{1+3\eta R_{\max}}\right). Z π ( x , y ) − E t [ Z π ( x , y )] ≤ R m a x ( 1 + τ + + e 1 + 3 η R m a x ) .
For a fixed π \pi π , Bernstein’s inequality (Vershynin, [107 ] , Section 2.8) therefore gives, with probability at least 1 − δ / ∣ Π Θ ∣ 1-\delta/|\Pi_{\Theta}| 1 − δ /∣ Π Θ ∣ ,
( E t − E ^ t ) [ Z ~ π ( x , y ) ] ≤ 2 V π t [ Z ~ π ( x , y ) ] log ( ∣ Π Θ ∣ / δ ) N + 2 R max ( 1 + τ + + e 1 + 3 η R max ) log ( ∣ Π Θ ∣ / δ ) 3 N . (164) \left(\mathbb{E}_t-\widehat{\mathbb{E}}_t\right)\left[\widetilde{Z}_{\pi}(x,y)\right]\le\sqrt{\frac{2\mathbb{V}_{\pi_t}[\widetilde{Z}_{\pi}(x,y)]\log(|\Pi_{\Theta}|/\delta)}{N}}+\frac{2R_{\max}\left(1+\tau_++e^{1+3\eta R_{\max}}\right)\log(|\Pi_{\Theta}|/\delta)}{3N}. \tag*{(164)} ( E t − E t ) [ Z π ( x , y ) ] ≤ N 2 V π t [ Z π ( x , y )] log ( ∣ Π Θ ∣/ δ ) + 3 N 2 R m a x ( 1 + τ + + e 1 + 3 η R m a x ) log ( ∣ Π Θ ∣/ δ ) . (164)
The variance is bounded by the second moment, and using Eq. (163) gives
V π t [ Z ~ π ( x , y ) ] ≤ E t [ Z ~ π ( x , y ) 2 ] ≤ 2 η R max 2 max { 1 + τ + , e 1 + 3 η R max } Δ π . \mathbb{V}_{\pi_t}[\widetilde{Z}_{\pi}(x,y)] \le\mathbb{E}_t[\widetilde{Z}_{\pi}(x,y)^2]
\le2\eta R_{\max}^2 \max\{1+\tau_+,e^{1+3\eta R_{\max}}\}\Delta_{\pi}. V π t [ Z π ( x , y )] ≤ E t [ Z π ( x , y ) 2 ] ≤ 2 η R m a x 2 max { 1 + τ + , e 1 + 3 η R m a x } Δ π .
Substituting this bound into Eq. (164) and taking a union bound over Π Θ \Pi_{\Theta} Π Θ gives, with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every π ∈ Π Θ \pi\in\Pi_{\Theta} π ∈ Π Θ ,
( E t − E ^ t ) [ Z ~ π ( x , y ) ] ≤ 2 η R max 2 max { 1 + τ + , e 1 + 3 η R max } log ( ∣ Π Θ ∣ / δ ) N Δ π + 2 R max ( 1 + τ + + e 1 + 3 η R max ) log ( ∣ Π Θ ∣ / δ ) 3 N . (165) (\mathbb{E}_t-\widehat{\mathbb{E}}_t)[\widetilde{Z}_{\pi}(x,y)]
\le2\sqrt{\frac{\eta R_{\max}^2\max\{1+\tau_+,e^{1+3\eta R_{\max}}\}\log(|\Pi_{\Theta}|/\delta)}{N}\Delta_{\pi}}
+\frac{2R_{\max}(1+\tau_++e^{1+3\eta R_{\max}})\log(|\Pi_{\Theta}|/\delta)}{3N}. \tag*{(165)} ( E t − E t ) [ Z π ( x , y )] ≤ 2 N η R m a x 2 max { 1 + τ + , e 1 + 3 η R m a x } log ( ∣ Π Θ ∣/ δ ) Δ π + 3 N 2 R m a x ( 1 + τ + + e 1 + 3 η R m a x ) log ( ∣ Π Θ ∣/ δ ) . (165)
Finally, we translate this bound into one on the objective gap using the relation in Eq. (154) . To do so, we will show that Z π ( x , y ) − Z ~ π ( x , y ) Z_{\pi}(x,y)-\widetilde{Z}_{\pi}(x,y) Z π ( x , y ) − Z π ( x , y ) is nonnegative and has expectation at most Δ π / 2 \Delta_{\pi}/2 Δ π /2 under π t \pi_t π t . Whenever π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) > M η > 1 \pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x)>M_{\eta}>1 π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) > M η > 1 , Lemma H.1 gives w t π ( x , y ) > w t π η , t ⋆ ( x , y ) ≥ 1 − τ − w^{\pi}_{t}(x,y)>w^{\pi_{\eta,t}^{\star}}_{t}(x,y)\ge1-\tau_- w t π ( x , y ) > w t π η , t ⋆ ( x , y ) ≥ 1 − τ − . Using Eqs. (155) and (156) , we therefore have
Z π ( x , y ) − Z ~ π ( x , y ) = r − ( x , y ) ( w t π ( x , y ) − w t π η , t ⋆ ( x , y ) ) I [ π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) > M η ] ≥ 0. (166) Z_{\pi}(x,y)-\widetilde{Z}_{\pi}(x,y)
=r_-(x,y)\left(w^{\pi}_{t}(x,y)-w^{\pi_{\eta,t}^{\star}}_{t}(x,y)\right)\mathbb{I}\left[\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}>M_{\eta}\right]\ge0. \tag*{(166)} Z π ( x , y ) − Z π ( x , y ) = r − ( x , y ) ( w t π ( x , y ) − w t π η , t ⋆ ( x , y ) ) I [ π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) > M η ] ≥ 0. (166)
To bound the expectation of this difference, recall that h ( u ) ≥ 0 h(u)\ge0 h ( u ) ≥ 0 and h ( u ) / u = log u − 1 + 1 / u h(u)/u=\log u-1+1/u h ( u ) / u = log u − 1 + 1/ u is increasing for u > 1 u>1 u > 1 . It follows that
u I [ u > M η ] ≤ h ( u ) log M η − 1 + 1 / M η , u ≥ 0. u\mathbb{I}[u>M_{\eta}]\le\frac{h(u)}{\log M_{\eta}-1+1/M_{\eta}},\qquad u\ge0. u I [ u > M η ] ≤ log M η − 1 + 1/ M η h ( u ) , u ≥ 0.
Since π t ( y ∣ x ) w t π η , t ⋆ ( x , y ) = π η , t ⋆ ( y ∣ x ) \pi_t(y\mid x)w_t^{\pi_{\eta,t}^{\star}}(x,y)=\pi_{\eta,t}^{\star}(y\mid x) π t ( y ∣ x ) w t π η , t ⋆ ( x , y ) = π η , t ⋆ ( y ∣ x ) , taking expectations in Eq. (166) gives
E t [ Z π ( x , y ) − Z ~ π ( x , y ) ] = E π η , t ⋆ [ r − ( x , y ) ( π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) − 1 ) I [ π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) > M η ] ] ≤ R max E π η , t ⋆ [ π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) I [ π ( y ∣ x ) π η , t ⋆ ( y ∣ x ) > M η ] ] ≤ R max D K L ( π ∥ π η , t ⋆ ) log M η − 1 + 1 / M η ≤ η R max 2 η R max + e − 1 − 2 η R max Δ π ≤ 1 2 Δ π . (167) \begin{aligned}
\mathbb{E}_t\left[Z_{\pi}(x,y)-\widetilde{Z}_{\pi}(x,y)\right]
&=\mathbb{E}_{\pi_{\eta,t}^{\star}}\left[r_-(x,y)\left(\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}-1\right)\mathbb{I}\left[\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}>M_{\eta}\right]\right]\\
&\le R_{\max}\mathbb{E}_{\pi_{\eta,t}^{\star}}\left[\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\mathbb{I}\left[\frac{\pi(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}>M_{\eta}\right]\right]\\
&\le\frac{R_{\max}D_{\mathrm{KL}}(\pi\Vert\pi_{\eta,t}^{\star})}{\log M_{\eta}-1+1/M_{\eta}}\\
&\le\frac{\eta R_{\max}}{2\eta R_{\max}+e^{-1-2\eta R_{\max}}}\Delta_{\pi}\le\frac{1}{2}\Delta_{\pi}. \tag*{(167)}
\end{aligned} E t [ Z π ( x , y ) − Z π ( x , y ) ] = E π η , t ⋆ [ r − ( x , y ) ( π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) − 1 ) I [ π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) > M η ] ] ≤ R m a x E π η , t ⋆ [ π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) I [ π η , t ⋆ ( y ∣ x ) π ( y ∣ x ) > M η ] ] ≤ log M η − 1 + 1/ M η R m a x D KL ( π ∥ π η , t ⋆ ) ≤ 2 η R m a x + e − 1 − 2 η R m a x η R m a x Δ π ≤ 2 1 Δ π . (167)
The first inequality uses 0 ≤ r − ( x , y ) ≤ R max 0\le r_-(x,y)\le R_{\max} 0 ≤ r − ( x , y ) ≤ R m a x . The second applies the scalar inequality above with u = π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) u=\pi(y\mid x)/\pi_{\eta,t}^{\star}(y\mid x) u = π ( y ∣ x ) / π η , t ⋆ ( y ∣ x ) and then uses Eq. (162) . The last line uses Lemma H.1 and log M η − 1 + 1 / M η = 2 η R max + e − 1 − 2 η R max \log M_{\eta}-1+1/M_{\eta}=2\eta R_{\max}+e^{-1-2\eta R_{\max}} log M η − 1 + 1/ M η = 2 η R m a x + e − 1 − 2 η R m a x .
Under the event that Eq. (165) holds for every π ∈ Π Θ \pi\in\Pi_{\Theta} π ∈ Π Θ , by Eq. (166) , E ^ t [ Z π ( x , y ) − Z ~ π ( x , y ) ] ≥ 0 \widehat{\mathbb{E}}_t[Z_{\pi}(x,y)-\widetilde{Z}_{\pi}(x,y)]\ge0 E t [ Z π ( x , y ) − Z π ( x , y )] ≥ 0 . Combining this with Eqs. (165) and (167) in Eq. (154) gives
Δ π − Δ ^ π = ( E t − E ^ t ) [ Z ~ π ( x , y ) ] + ( E t − E ^ t ) [ Z π ( x , y ) − Z ~ π ( x , y ) ] ≤ ( E t − E ^ t ) [ Z ~ π ( x , y ) ] + E t [ Z π ( x , y ) − Z ~ π ( x , y ) ] ≤ 1 2 Δ π + 2 η R max 2 max { 1 + τ + , e 1 + 3 η R max } log ( ∣ Π Θ ∣ / δ ) N Δ π + 2 R max ( 1 + τ + + e 1 + 3 η R max ) log ( ∣ Π Θ ∣ / δ ) 3 N ≤ 3 4 Δ π + 4 η R max 2 max { 1 + τ + , e 1 + 3 η R max } log ( ∣ Π Θ ∣ / δ ) N + 2 R max ( 1 + τ + + e 1 + 3 η R max ) log ( ∣ Π Θ ∣ / δ ) 3 N = 3 4 Δ π + 1 4 ϵ s t a t ( δ ) . \begin{aligned}
\Delta_{\pi}-\widehat{\Delta}_{\pi}
&=(\mathbb{E}_t-\widehat{\mathbb{E}}_t)[\widetilde{Z}_{\pi}(x,y)]+(\mathbb{E}_t-\widehat{\mathbb{E}}_t)[Z_{\pi}(x,y)-\widetilde{Z}_{\pi}(x,y)]\\
&\le(\mathbb{E}_t-\widehat{\mathbb{E}}_t)[\widetilde{Z}_{\pi}(x,y)]+\mathbb{E}_t[Z_{\pi}(x,y)-\widetilde{Z}_{\pi}(x,y)]\\
&\le\frac{1}{2}\Delta_{\pi}+2\sqrt{\frac{\eta R_{\max}^2\max\{1+\tau_+,e^{1+3\eta R_{\max}}\}\log(|\Pi_{\Theta}|/\delta)}{N}\Delta_{\pi}}\\
&\qquad+\frac{2R_{\max}(1+\tau_++e^{1+3\eta R_{\max}})\log(|\Pi_{\Theta}|/\delta)}{3N}\\
&\le\frac{3}{4}\Delta_{\pi}+\frac{4\eta R_{\max}^2\max\{1+\tau_+,e^{1+3\eta R_{\max}}\}\log(|\Pi_{\Theta}|/\delta)}{N}\\
&\qquad+\frac{2R_{\max}(1+\tau_++e^{1+3\eta R_{\max}})\log(|\Pi_{\Theta}|/\delta)}{3N}\\
&=\frac{3}{4}\Delta_{\pi}+\frac{1}{4}\epsilon_{\mathrm{stat}}(\delta).
\end{aligned} Δ π − Δ π = ( E t − E t ) [ Z π ( x , y )] + ( E t − E t ) [ Z π ( x , y ) − Z π ( x , y )] ≤ ( E t − E t ) [ Z π ( x , y )] + E t [ Z π ( x , y ) − Z π ( x , y )] ≤ 2 1 Δ π + 2 N η R m a x 2 max { 1 + τ + , e 1 + 3 η R m a x } log ( ∣ Π Θ ∣/ δ ) Δ π + 3 N 2 R m a x ( 1 + τ + + e 1 + 3 η R m a x ) log ( ∣ Π Θ ∣/ δ ) ≤ 4 3 Δ π + N 4 η R m a x 2 max { 1 + τ + , e 1 + 3 η R m a x } log ( ∣ Π Θ ∣/ δ ) + 3 N 2 R m a x ( 1 + τ + + e 1 + 3 η R m a x ) log ( ∣ Π Θ ∣/ δ ) = 4 3 Δ π + 4 1 ϵ stat ( δ ) .
The last inequality uses 2 a b ≤ a / 4 + 4 b 2\sqrt{ab} \le a/4 + 4b 2 ab ≤ a /4 + 4 b with a = Δ π a = \Delta_{\pi} a = Δ π and b = η R max 2 max { 1 + τ + , e 1 + 3 η R max } log ( ∣ Π Θ ∣ / δ ) / N b = \eta R_{\max}^{2}\max\{1+\tau_{+},e^{1+3\eta R_{\max}}\}\log(|\Pi_{\Theta}|/\delta)/N b = η R m a x 2 max { 1 + τ + , e 1 + 3 η R m a x } log ( ∣ Π Θ ∣/ δ ) / N . The final equality uses the definition of ϵ s t a t ( δ ) \epsilon_{\mathrm{stat}}(\delta) ϵ stat ( δ ) in Eq. (153) . Rearranging gives Δ π ≤ 4 Δ ^ π + ϵ s t a t ( δ ) \Delta_{\pi} \le4\widehat{\Delta}_{\pi}+\epsilon_{\mathrm{stat}}(\delta) Δ π ≤ 4 Δ π + ϵ stat ( δ ) , proving Eq. (152) .
Lemma H.3. Suppose that 1 − τ − ≤ e − 2 η R max 1-\tau_{-}\le e^{-2\eta R_{\max}} 1 − τ − ≤ e − 2 η R m a x and 1 + τ + ≥ e 2 η R max 1+\tau_{+}\ge e^{2\eta R_{\max}} 1 + τ + ≥ e 2 η R m a x . If π t + 1 ∈ Π Θ \pi_{t+1}\in\Pi_{\Theta} π t + 1 ∈ Π Θ satisfies
J ^ t η , c l i p ( π t + 1 ) ≥ sup π ∈ Π Θ J ^ t η , c l i p ( π ) − ϵ o p t , \widehat{J}_{t}^{\eta,\mathrm{clip}}(\pi_{t+1})\ge\sup_{\pi\in\Pi_{\Theta}}\widehat{J}_{t}^{\eta,\mathrm{clip}}(\pi)-\epsilon_{\mathrm{opt}}, J t η , clip ( π t + 1 ) ≥ π ∈ Π Θ sup J t η , clip ( π ) − ϵ opt ,
then, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ ,
J η , t ( π η , t ⋆ ) − J η , t ( π t + 1 ) = 1 η D K L ( π t + 1 ∥ π η , t ⋆ ) ≤ 4 ϵ o p t + ϵ s t a t ( δ ) , J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_{t+1})=\frac{1}{\eta}D_{\mathrm{KL}}(\pi_{t+1}\Vert\pi_{\eta,t}^{\star})\le4\epsilon_{\mathrm{opt}}+\epsilon_{\mathrm{stat}}(\delta), J η , t ( π η , t ⋆ ) − J η , t ( π t + 1 ) = η 1 D KL ( π t + 1 ∥ π η , t ⋆ ) ≤ 4 ϵ opt + ϵ stat ( δ ) ,
where ϵ s t a t ( δ ) \epsilon_{\mathrm{stat}}(\delta) ϵ stat ( δ ) is defined in Eq. (153) .
Proof of Lemma H.3. Since π η , t ⋆ ∈ Π Θ \pi_{\eta,t}^{\star}\in\Pi_{\Theta} π η , t ⋆ ∈ Π Θ , Line 5 gives
J ^ t η , c l i p ( π η , t ⋆ ) − J ^ t η , c l i p ( π t + 1 ) ≤ ϵ o p t . \widehat{J}_{t}^{\eta,\mathrm{clip}}(\pi_{\eta,t}^{\star})-\widehat{J}_{t}^{\eta,\mathrm{clip}}(\pi_{t+1})\le\epsilon_{\mathrm{opt}}. J t η , clip ( π η , t ⋆ ) − J t η , clip ( π t + 1 ) ≤ ϵ opt .
Since the concentration inequality in Lemma H.2 applies to every policy in Π Θ \Pi_{\Theta} Π Θ , including π t + 1 \pi_{t+1} π t + 1 , combining it with Lemma H.1 and the above panel give
1 η D K L ( π t + 1 ∥ π η , t ⋆ ) ≤ J t η , c l i p ( π η , t ⋆ ) − J t η , c l i p ( π t + 1 ) \frac{1}{\eta}D_{\mathrm{KL}}(\pi_{t+1}\Vert\pi_{\eta,t}^{\star})\le J_{t}^{\eta,\mathrm{clip}}(\pi_{\eta,t}^{\star})-J_{t}^{\eta,\mathrm{clip}}(\pi_{t+1}) η 1 D KL ( π t + 1 ∥ π η , t ⋆ ) ≤ J t η , clip ( π η , t ⋆ ) − J t η , clip ( π t + 1 )
≤ 4 ( J ^ t η , c l i p ( π η , t ⋆ ) − J ^ t η , c l i p ( π t + 1 ) ) + ϵ s t a t ( δ ) ≤ 4 ϵ o p t + ϵ s t a t ( δ ) . \le4\left(\widehat{J}_{t}^{\eta,\mathrm{clip}}(\pi_{\eta,t}^{\star})-\widehat{J}_{t}^{\eta,\mathrm{clip}}(\pi_{t+1})\right)+\epsilon_{\mathrm{stat}}(\delta)\le4\epsilon_{\mathrm{opt}}+\epsilon_{\mathrm{stat}}(\delta). ≤ 4 ( J t η , clip ( π η , t ⋆ ) − J t η , clip ( π t + 1 ) ) + ϵ stat ( δ ) ≤ 4 ϵ opt + ϵ stat ( δ ) .
Proof of 3.2
We now give the full version of Corollary 3.2 . The result depends on the complexity of the finite policy classes considered in this section, given by log ∣ Π Θ ∣ \log|\Pi_{\Theta}| log ∣ Π Θ ∣ . We first use this result to prove Corollary 3.2 then prove Corollary H.1.
Corollary H.1 (PPO-Lite is coverage-preserving). Suppose that r ( x , y ) ∈ [ − R max , R max ] r(x,y)\in[-R_{\max},R_{\max}] r ( x , y ) ∈ [ − R m a x , R m a x ] and Π Θ \Pi_{\Theta} Π Θ is finite. Assume that every policy in Π Θ \Pi_{\Theta} Π Θ has full support, D K L ( π ∥ π ′ ) < ∞ D_{\mathrm{KL}}(\pi\Vert\pi')<\infty D KL ( π ∥ π ′ ) < ∞ for all π , π ′ ∈ Π Θ \pi,\pi'\in\Pi_{\Theta} π , π ′ ∈ Π Θ , and π η , t ⋆ ∈ Π Θ \pi_{\eta,t}^{\star}\in\Pi_{\Theta} π η , t ⋆ ∈ Π Θ at every outer iteration. Fix η > 0 \eta>0 η > 0 , ϵ ∈ ( 0 , 2 R max ] \epsilon\in(0,2R_{\max}] ϵ ∈ ( 0 , 2 R m a x ] , and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Run Algorithm 2 with π 0 = π b a s e ∈ Π Θ \pi_{0}=\pi_{\mathrm{base}}\in\Pi_{\Theta} π 0 = π base ∈ Π Θ and
τ − = 1 − e − 2 η R max , τ + = e 2 η R max − 1 , T = ⌈ log ( 2 C b a s e ) η ϵ ⌉ . (168) \tau_{-}=1-e^{-2\eta R_{\max}},\qquad\tau_{+}=e^{2\eta R_{\max}}-1,\qquad T=\left\lceil\frac{\log(2C_{\mathrm{base}})}{\eta\epsilon}\right\rceil. \tag*{(168)} τ − = 1 − e − 2 η R m a x , τ + = e 2 η R m a x − 1 , T = ⌈ η ϵ log ( 2 C base ) ⌉ . (168)
Set
ϵ i n = 1 64 C b a s e T 2 log ( e T ) , ϵ o p t ≤ ϵ i n 8 η , (169) \epsilon_{\mathrm{in}}=\frac{1}{64C_{\mathrm{base}}T^{2}\log(eT)},\qquad\epsilon_{\mathrm{opt}}\le\frac{\epsilon_{\mathrm{in}}}{8\eta}, \tag*{(169)} ϵ in = 64 C base T 2 log ( e T ) 1 , ϵ opt ≤ 8 η ϵ in , (169)
N = ⌈ 16 η R max 3 ϵ i n e 2 η R max [ 1 + ( 1 + 6 η R max ) e 1 + η R max ] log ∣ Π Θ ∣ T δ ⌉ . (170) N=\left\lceil\frac{16\eta R_{\max}}{3\epsilon_{\mathrm{in}}}e^{2\eta R_{\max}}\left[1+(1+6\eta R_{\max})e^{1+\eta R_{\max}}\right]\log\frac{|\Pi_{\Theta}|T}{\delta}\right\rceil. \tag*{(170)} N = ⌈ 3 ϵ in 16 η R m a x e 2 η R m a x [ 1 + ( 1 + 6 η R m a x ) e 1 + η R m a x ] log δ ∣ Π Θ ∣ T ⌉ . (170)
Here ϵ i n \epsilon_{\mathrm{in}} ϵ in is the right-hand side of Eq. (72) . With probability at least 1 − δ 1-\delta 1 − δ over the batches, the returned policy satisfies
J ⋆ − J ( π p o s t ) ≤ ϵ , (171) J^{\star}-J(\pi_{\mathrm{post}})\le\epsilon, \tag*{(171)} J ⋆ − J ( π post ) ≤ ϵ , (171)
and the policy sequence satisfies
P x ∼ μ ( π t ( y x ⋆ ∣ x ) ≥ 1 2 π b a s e ( y x ⋆ ∣ x ) for all 0 ≤ t ≤ T ) ≥ 1 − 2 + 2 8 log ( e T ) . (172) \mathbb{P}_{x\sim\mu}\left(\pi_t(y_x^\star\mid x) \ge\frac{1}{2}\pi_{\mathrm{base}}(y_x^\star\mid x)\ \text{for all }0 \le t \le T\right) \ge1-\frac{2+\sqrt{2}}{8\sqrt{\log(eT)}}. \tag*{(172)} P x ∼ μ ( π t ( y x ⋆ ∣ x ) ≥ 2 1 π base ( y x ⋆ ∣ x ) for all 0 ≤ t ≤ T ) ≥ 1 − 8 log ( e T ) 2 + 2 . (172)
In particular, with η = 1 / R max \eta= 1/R_{\max} η = 1/ R m a x , the number of sampled responses is
N s a m p l e ( ε , δ ) = T N = O ~ ( C b a s e R max 3 ε 3 log 2 ∣ Π Θ ∣ δ ) . (173) N_{\mathrm{sample}}(\varepsilon,\delta)=TN=\widetilde{O}\left(\frac{C_{\mathrm{base}}R_{\max}^{3}}{\varepsilon^{3}}\log\frac{2|\Pi_\Theta|}{\delta}\right). \tag*{(173)} N sample ( ε , δ ) = T N = O ( ε 3 C base R m a x 3 log δ 2∣ Π Θ ∣ ) . (173)
Proof of Corollary 3.2 . Corollary 3.2 follows from Corollary H.1 by setting η = 1 / R max \eta= 1/R_{\max} η = 1/ R m a x and choosing N N N as in Eq. (170) . The KL tolerance in Eq. (169) is sufficient for Eq. (6) (Theorem 3.1 ), and the same inner optimization requirement ε o p t ≤ ε i n / ( 8 η ) \varepsilon_{\mathrm{opt}} \le\varepsilon_{\mathrm{in}}/(8\eta) ε opt ≤ ε in / ( 8 η ) applies. The sample complexity bound is then given by Eq. (173) .
Proof of Corollary H.1. We first show that the KL errors are at most ε i n \varepsilon_{\mathrm{in}} ε in simultaneously over all outer iterations with probability at least 1 − δ 1-\delta 1 − δ . We will then apply Theorem D.1 to obtain convergence and coverage.
The thresholds in Eq. (168) satisfy the conditions of Lemma H.3. Substituting 1 + τ + = e 2 η R max 1+\tau_+=e^{2\eta R_{\max}} 1 + τ + = e 2 η R m a x into Eq. (153) gives
ε s t a t ( δ T ) = 8 R max 3 N e 2 η R max [ 1 + ( 1 + 6 η R max ) e 1 + η R max ] log ∣ Π Θ ∣ T δ ≤ ε i n 2 η , \varepsilon_{\mathrm{stat}}\left(\frac{\delta}{T}\right)=\frac{8R_{\max}}{3N}e^{2\eta R_{\max}}\left[1+(1+6\eta R_{\max})e^{1+\eta R_{\max}}\right]\log\frac{|\Pi_\Theta|T}{\delta}
\le\frac{\varepsilon_{\mathrm{in}}}{2\eta}, ε stat ( T δ ) = 3 N 8 R m a x e 2 η R m a x [ 1 + ( 1 + 6 η R m a x ) e 1 + η R m a x ] log δ ∣ Π Θ ∣ T ≤ 2 η ε in ,
where the inequality uses the choice of N N N in Eq. (170) . Fix an outer iteration t t t and condition on the preceding batches D 0 , … , D t − 1 \mathcal{D}_0,\ldots,\mathcal{D}_{t-1} D 0 , … , D t − 1 , which fixes π t \pi_t π t before D t \mathcal{D}_t D t is drawn. Applying Lemma H.3 with failure probability δ / T \delta/T δ / T gives
P ( D K L ( π t + 1 ∥ π η , t ⋆ ) ≤ η ( 4 ε o p t + ε s t a t ( δ / T ) ) ∣ D 0 , … , D t − 1 ) ≥ 1 − δ T . \mathbb{P}\left(D_{\mathrm{KL}}(\pi_{t+1}\Vert\pi_{\eta,t}^{\star})\le\eta(4\varepsilon_{\mathrm{opt}}+\varepsilon_{\mathrm{stat}}(\delta/T))\mid\mathcal{D}_0,\ldots,\mathcal{D}_{t-1}\right)\ge1-\frac{\delta}{T}. P ( D KL ( π t + 1 ∥ π η , t ⋆ ) ≤ η ( 4 ε opt + ε stat ( δ / T )) ∣ D 0 , … , D t − 1 ) ≥ 1 − T δ .
Since 4 ε o p t ≤ ε i n / ( 2 η ) 4\varepsilon_{\mathrm{opt}}\le\varepsilon_{\mathrm{in}}/(2\eta) 4 ε opt ≤ ε in / ( 2 η ) by Eq. (169) , the statistical and optimization errors together give a KL error of at most ε i n \varepsilon_{\mathrm{in}} ε in . Averaging over the preceding batches and taking a union bound over 0 ≤ t < T 0\le t<T 0 ≤ t < T , we obtain, with probability at least 1 − δ 1-\delta 1 − δ ,
D K L ( π t + 1 ∥ π η , t ⋆ ) ≤ η ( 4 ε o p t + ε s t a t ( δ / T ) ) ≤ ε i n = 1 64 C b a s e T 2 log ( e T ) , 0 ≤ t < T . (174) D_{\mathrm{KL}}(\pi_{t+1}\Vert\pi_{\eta,t}^{\star})\le\eta(4\varepsilon_{\mathrm{opt}}+\varepsilon_{\mathrm{stat}}(\delta/T))\le\varepsilon_{\mathrm{in}}=\frac{1}{64C_{\mathrm{base}}T^2\log(eT)},\qquad0\le t<T. \tag*{(174)} D KL ( π t + 1 ∥ π η , t ⋆ ) ≤ η ( 4 ε opt + ε stat ( δ / T )) ≤ ε in = 64 C base T 2 log ( e T ) 1 , 0 ≤ t < T . (174)
Fix any policy sequence satisfying Eq. (174) for the remainder of the proof.
To obtain the reward guarantee, apply Theorem D.1 to r ( x , y ) + R max ∈ [ 0 , 2 R max ] r(x,y)+R_{\max}\in[0,2R_{\max}] r ( x , y ) + R m a x ∈ [ 0 , 2 R m a x ] . Adding this constant preserves the exact targets, KL errors, reward gaps, and sets of optimal responses, so the theorem applies to the same policy sequence. Jensen’s inequality gives E x ∼ μ [ log ( 1 / π b a s e ( y x ⋆ ∣ x ) ) ] ≤ log C b a s e \mathbb{E}_{x\sim\mu}[\log(1/\pi_{\mathrm{base}}(y_x^\star\mid x))]\le\log C_{\mathrm{base}} E x ∼ μ [ log ( 1/ π base ( y x ⋆ ∣ x ))] ≤ log C base . Combining this with Eq. (174) and ∑ t = 0 T − 1 ( t + 1 ) − 1 ≤ log ( e T ) \sum_{t=0}^{T-1}(t+1)^{-1}\le\log(eT) ∑ t = 0 T − 1 ( t + 1 ) − 1 ≤ log ( e T ) in Theorem D.1 yields
J ⋆ − J ( π p o s t ) ≤ log C b a s e η T + T ε i n η + 2 e η C b a s e ε i n log ( e T ) ≤ log C b a s e + 1 / 64 + e / 4 η T ≤ log ( 2 C b a s e ) η T ≤ ε . \begin{aligned}
J^\star-J(\pi_{\mathrm{post}})\le\frac{\log C_{\mathrm{base}}}{\eta T}+\frac{T\varepsilon_{\mathrm{in}}}{\eta}+\frac{2\sqrt{e}}{\eta}\sqrt{C_{\mathrm{base}}\varepsilon_{\mathrm{in}}\log(eT)} \\
&\le\frac{\log C_{\mathrm{base}}+1/64+\sqrt{e}/4}{\eta T}\le\frac{\log(2C_{\mathrm{base}})}{\eta T}\le\varepsilon.
\end{aligned} J ⋆ − J ( π post ) ≤ η T log C base + η T ε in + η 2 e C base ε in log ( e T ) ≤ η T log C base + 1/64 + e /4 ≤ η T log ( 2 C base ) ≤ ε .
The second line substitutes ε i n \varepsilon_{\mathrm{in}} ε in and uses C b a s e ≥ 1 C_{\mathrm{base}}\ge1 C base ≥ 1 and log ( e T ) ≥ 1 \log(eT)\ge1 log ( e T ) ≥ 1 . The remaining inequalities use 1 / 64 + e / 4 ≤ log 2 1/64+\sqrt{e}/4\le\log2 1/64 + e /4 ≤ log 2 and the choice of T T T , proving Eq. (171) .
The coverage guarantee in Theorem D.1 also applies to this policy sequence. Taking c = 1 / 2 c = 1/2 c = 1/2 and using the bound ε i n \varepsilon_{\mathrm{in}} ε in on each KL error gives
P x ∼ μ ( π t ( Y x ∗ ∣ x ) ≥ 1 2 π b a s e ( Y x ∗ ∣ x ) for all 0 ≤ t ≤ T ) ≥ 1 − ( 2 + 2 ) T C b a s e ε i n = 1 − 2 + 2 8 log ( e T ) . \mathbb{P}_{x\sim\mu}\left(\pi_t(Y_x^* \mid x) \ge\frac{1}{2}\pi_{\mathrm{base}}(Y_x^* \mid x)\text{ for all }0 \le t \le T\right)
\ge1-(2+\sqrt{2})T\sqrt{C_{\mathrm{base}}\varepsilon_{\mathrm{in}}}
=1-\frac{2+\sqrt{2}}{8\sqrt{\log(eT)}}. P x ∼ μ ( π t ( Y x ∗ ∣ x ) ≥ 2 1 π base ( Y x ∗ ∣ x ) for all 0 ≤ t ≤ T ) ≥ 1 − ( 2 + 2 ) T C base ε in = 1 − 8 log ( e T ) 2 + 2 .
This proves Eq. (172) .
Finally, each of the T T T outer iterations draws N N N responses. Substituting ε i n \varepsilon_{\mathrm{in}} ε in from Eq. (169) into the choice of N N N in Eq. (170) , and accounting for the ceiling, gives
T N ≤ T + 1024 3 η R max C b a s e e 2 η R max [ 1 + ( 1 + 6 η R max ) e 1 + η R max ] T 3 log ( e T ) log ∣ Π Θ ∣ T δ . TN \le T+\frac{1024}{3}\eta R_{\max}C_{\mathrm{base}}e^{2\eta R_{\max}}\left[1+(1+6\eta R_{\max})e^{1+\eta R_{\max}}\right]T^3\log(eT)\log\frac{|\Pi_\Theta|T}{\delta}. T N ≤ T + 3 1024 η R m a x C base e 2 η R m a x [ 1 + ( 1 + 6 η R m a x ) e 1 + η R m a x ] T 3 log ( e T ) log δ ∣ Π Θ ∣ T .
In particular, taking η = 1 / R max \eta=1/R_{\max} η = 1/ R m a x and using ε ≤ 2 R max \varepsilon\le2R_{\max} ε ≤ 2 R m a x yields
T = O ( R max log ( 2 C b a s e ) ε ) , T=O\left(\frac{R_{\max}\log(2C_{\mathrm{base}})}{\varepsilon}\right), T = O ( ε R m a x log ( 2 C base ) ) ,
T N = O ( C b a s e R max 3 log 3 ( 2 C b a s e ) ε 3 log ( e T ) log 2 ∣ Π Θ ∣ T δ ) . TN=O\left(\frac{C_{\mathrm{base}}R_{\max}^3\log^3(2C_{\mathrm{base}})}{\varepsilon^3}\log(eT)\log\frac{2|\Pi_\Theta|T}{\delta}\right). T N = O ( ε 3 C base R m a x 3 log 3 ( 2 C base ) log ( e T ) log δ 2∣ Π Θ ∣ T ) .
Suppressing logarithmic factors gives Eq. (173) .
Additional Result: Gradient Ascent for Softmax Linear Policies
Corollary H.1 assumes that each inner loop meets the optimization tolerance in Eq. (15) , but leaves the optimization procedure abstract. We now study gradient ascent as a concrete way to implement the inner loop. Appendix § 14.2 .§ 3.1 defines the inner loop for differentiable policies, using the same batch to estimate the reward and KL gradients. Appendix § 14.2 .§ 3.2 then gives an end-to-end guarantee for these updates under softmax linear policies and sufficiently wide clipping thresholds.
Gradient Ascent on the Clipped Objective
Fix an outer iteration t t t and a differentiable policy class Π Θ = { π θ } \Pi_\Theta=\{\pi_\theta\} Π Θ = { π θ } , with π t = π θ t \pi_t=\pi_{\theta_t} π t = π θ t . For the discussion below, assume that the clipped reward term is differentiable for almost every ( x , y ) (x,y) ( x , y ) under π t \pi_t π t and that we can differentiate under the expectation.
Gradient computation. The population objective in Eq. (150) has gradient
∇ J t η , c l i p ( π θ ) = E π t [ w t π θ ( x , y ) r ( x , y ) ∇ log π θ ( y ∣ x ) I [ r ( x , y ) ≥ 0 , w t π θ ( x , y ) < 1 + τ + ] ] \nabla J_t^{\eta,\mathrm{clip}}(\pi_\theta)
= \mathbb{E}_{\pi_t}\left[w_t^{\pi_\theta}(x,y)r(x,y)\nabla\log\pi_\theta(y\mid x)\mathbb{I}\left[r(x,y)\ge0,\ w_t^{\pi_\theta}(x,y)<1+\tau_+\right]\right] ∇ J t η , clip ( π θ ) = E π t [ w t π θ ( x , y ) r ( x , y ) ∇ log π θ ( y ∣ x ) I [ r ( x , y ) ≥ 0 , w t π θ ( x , y ) < 1 + τ + ] ]
+ E π t [ w t π θ ( x , y ) r ( x , y ) ∇ log π θ ( y ∣ x ) I [ r ( x , y ) < 0 , w t π θ ( x , y ) > 1 − τ − ] ] − 1 η ∇ D K L ( π θ ∥ π t ) . \quad+\mathbb{E}_{\pi_t}\left[w_t^{\pi_\theta}(x,y)r(x,y)\nabla\log\pi_\theta(y\mid x)\mathbb{I}\left[r(x,y)<0,\ w_t^{\pi_\theta}(x,y)>1-\tau_-\right]\right]
-\frac{1}{\eta}\nabla D_{\mathrm{KL}}(\pi_\theta\parallel\pi_t). + E π t [ w t π θ ( x , y ) r ( x , y ) ∇ log π θ ( y ∣ x ) I [ r ( x , y ) < 0 , w t π θ ( x , y ) > 1 − τ − ] ] − η 1 ∇ D KL ( π θ ∥ π t ) .
Clipping sets a response’s contribution to the reward gradient to zero when its policy ratio exceeds 1 + τ + 1+\tau_+ 1 + τ + for a positive reward or falls below 1 − τ − 1-\tau_- 1 − τ − for a negative reward. It leaves the KL gradient unchanged.
For the inner loop, we estimate both the gradient using the batch D t \mathcal{D}_t D t drawn as in Algorithm 2, where the gradient estimate is
∇ J ^ t , e s t η , c l i p ( π θ ) = 1 N ∑ i = 1 N w t π θ ( x i , y i ) r i ∇ log π θ ( y i ∣ x i ) I [ r i ≥ 0 , w t π θ ( x i , y i ) < 1 + τ + ] + 1 N ∑ i = 1 N w t π θ ( x i , y i ) r i ∇ log π θ ( y i ∣ x i ) I [ r i < 0 , w t π θ ( x i , y i ) > 1 − τ − ] − 1 η N ∑ i = 1 N w t π θ ( x i , y i ) log w t π θ ( x i , y i ) ∇ log π θ ( y i ∣ x i ) . (175) \nabla\widehat{J}^{\eta,\mathrm{clip}}_{t,\mathrm{est}}(\pi_{\theta}) = \frac{1}{N}\sum_{i=1}^{N} w_{t}^{\pi_{\theta}}(x_i,y_i)r_i\nabla\log\pi_{\theta}(y_i\mid x_i)\mathbb{I}[r_i\ge0,\ w_{t}^{\pi_{\theta}}(x_i,y_i)<1+\tau_{+}]
+\frac{1}{N}\sum_{i=1}^{N}w_{t}^{\pi_{\theta}}(x_i,y_i)r_i\nabla\log\pi_{\theta}(y_i\mid x_i)\mathbb{I}[r_i<0,\ w_{t}^{\pi_{\theta}}(x_i,y_i)>1-\tau_{-}]
-\frac{1}{\eta N}\sum_{i=1}^{N}w_{t}^{\pi_{\theta}}(x_i,y_i)\log w_{t}^{\pi_{\theta}}(x_i,y_i)\nabla\log\pi_{\theta}(y_i\mid x_i). \tag*{(175)} ∇ J t , est η , clip ( π θ ) = N 1 i = 1 ∑ N w t π θ ( x i , y i ) r i ∇ log π θ ( y i ∣ x i ) I [ r i ≥ 0 , w t π θ ( x i , y i ) < 1 + τ + ] + N 1 i = 1 ∑ N w t π θ ( x i , y i ) r i ∇ log π θ ( y i ∣ x i ) I [ r i < 0 , w t π θ ( x i , y i ) > 1 − τ − ] − η N 1 i = 1 ∑ N w t π θ ( x i , y i ) log w t π θ ( x i , y i ) ∇ log π θ ( y i ∣ x i ) . (175)
Similar to PPO-NoClip, this is the gradient of the empirical version of the objective given by
J ^ t , e s t η , c l i p ( π ) : = 1 N ∑ i = 1 N [ min { w t π ( x i , y i ) r i , [ w t π ( x i , y i ) ] 1 − τ − 1 + τ + r i } − 1 η [ w t π ( x i , y i ) log w t π ( x i , y i ) − w t π ( x i , y i ) + 1 ] ] . \widehat{J}^{\eta,\mathrm{clip}}_{t,\mathrm{est}}(\pi):=\frac{1}{N}\sum_{i=1}^{N}\left[\min\left\{w_{t}^{\pi}(x_i,y_i)r_i,\left[w_{t}^{\pi}(x_i,y_i)\right]_{1-\tau_{-}}^{1+\tau_{+}}r_i\right\}-\frac{1}{\eta}\left[w_{t}^{\pi}(x_i,y_i)\log w_{t}^{\pi}(x_i,y_i)-w_{t}^{\pi}(x_i,y_i)+1\right]\right]. J t , est η , clip ( π ) := N 1 i = 1 ∑ N [ min { w t π ( x i , y i ) r i , [ w t π ( x i , y i ) ] 1 − τ − 1 + τ + r i } − η 1 [ w t π ( x i , y i ) log w t π ( x i , y i ) − w t π ( x i , y i ) + 1 ] ] .
Inner optimization loop. The inner loop initializes at θ t \theta_t θ t and reuses D t \mathcal{D}_t D t for K K K steps with the gradient in Eq. (175) ,
θ t 0 = θ t , θ t k + 1 = θ t k + α ∇ J ^ t , e s t η , c l i p ( π t k ) , π t k = π θ t k , 0 ≤ k < K , (176) \theta_t^0=\theta_t,\qquad\theta_t^{k+1}=\theta_t^k+\alpha\nabla\widehat{J}^{\eta,\mathrm{clip}}_{t,\mathrm{est}}(\pi_t^k),\qquad\pi_t^k=\pi_{\theta_t^k},\qquad0\le k<K, \tag*{(176)} θ t 0 = θ t , θ t k + 1 = θ t k + α ∇ J t , est η , clip ( π t k ) , π t k = π θ t k , 0 ≤ k < K , (176)
and returns θ t + 1 = θ t K \theta_{t+1}=\theta_t^K θ t + 1 = θ t K and π t + 1 = π t K \pi_{t+1}=\pi_t^K π t + 1 = π t K .
Convergence for Softmax Linear Policies
To bound the sample and gradient step complexity for the full algorithm, we adopt the setting of Corollary 3.1 with softmax linear policies (Assumption 2.1) and nonnegative rewards 0 ≤ r ( x , y ) ≤ R max 0\le r(x,y)\le R_{\max} 0 ≤ r ( x , y ) ≤ R m a x . Note that here the policy class is not finite.
The analysis of PPO-NoClip already shows that, with high probability, KL regularization keeps its inner iterates close to the target π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ , and bounds the policy ratios as absolute constants throughout training. Thus, with sufficiently side clipping thresholds of constant magnitude the ratios will remain unclipped, and we can largely reuse the analysis of PPO-NoClip in Appendix G, as the clipping in PPO-Lite leaves every gradient update unchanged. The following corollary therefore recovers the complexity guarantees of PPO-NoClip.
Corollary H.2. Suppose that Assumption 2.1 holds and 0 ≤ r ( x , y ) ≤ R max 0\le r(x,y)\le R_{\max} 0 ≤ r ( x , y ) ≤ R m a x . Fix an outer iteration t t t and run Eq. (176) with η , α , K , N , δ \eta,\alpha,K,N,\delta η , α , K , N , δ satisfying the conditions of Theorem G.1. If
1 − τ − ≤ e − 5 , 1 + τ + ≥ e 5 , (177) 1-\tau_{-}\le e^{-5},\qquad1+\tau_{+}\ge e^{5}, \tag*{(177)} 1 − τ − ≤ e − 5 , 1 + τ + ≥ e 5 , (177) then, with probability at least 1 − δ 1-\delta 1 − δ , clipping is inactive at every inner iterate and
J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ≤ 190 e 9 B ϕ B ⋆ [ 1 K + 1 N log ( 8 [ 5 ( 1 + 2 N ) ] d δ ) ] . J_{\eta,t}(\pi_{\eta,t}^{\star})-J_{\eta,t}(\pi_t^K)\le190e^9B_{\phi}B_{\star}\left[\frac{1}{K}+\frac{1}{N}\log\left(\frac{8[5(1+2N)]^d}{\delta}\right)\right]. J η , t ( π η , t ⋆ ) − J η , t ( π t K ) ≤ 190 e 9 B ϕ B ⋆ [ K 1 + N 1 log ( δ 8 [ 5 ( 1 + 2 N ) ] d ) ] . Consequently, under the assumptions and parameter choices of Corollary G.1, replacing its inner updates by Eq. (176) with Eq. (177) gives, with probability at least 1 − δ 1-\delta 1 − δ ,
J ⋆ − J ( π p o s t ) ≤ ε . J^{\star}-J(\pi_{\mathrm{post}})\le\varepsilon. J ⋆ − J ( π post ) ≤ ε . On the same event, the policy sequence satisfies
P x ∼ μ ( π t ( Y x ⋆ ∣ x ) ≥ 1 2 π b a s e ( Y x ⋆ ∣ x ) for all 0 ≤ t ≤ T ) ≥ 1 − 2 + 2 8 log ( e T ) . \mathbb{P}_{x\sim\mu}\left(\pi_t(Y_x^{\star}\mid x)\ge\frac{1}{2}\pi_{\mathrm{base}}(Y_x^{\star}\mid x)\text{ for all }0\le t\le T\right)\ge1-\frac{2+\sqrt{2}}{8\sqrt{\log(eT)}}. P x ∼ μ ( π t ( Y x ⋆ ∣ x ) ≥ 2 1 π base ( Y x ⋆ ∣ x ) for all 0 ≤ t ≤ T ) ≥ 1 − 8 log ( e T ) 2 + 2 . The numbers of gradient steps and sampled responses satisfy the bounds in Eq. (149) .
Discussion: Comparison to Corollary 3.2 . Corollary H.2 bounds both the samples and gradient steps needed for convergence, without assuming that the inner loop satisfies Eq. (15) . Its guarantee relies on clipping remaining inactive, so it does not establish an additional benefit from clipping in this regime. Corollary H.1 allows clipping to be active and gives a guarantee for any inner procedure that meets the empirical optimization tolerance, but does not bound the number of gradient steps needed to do so.
Proof of Corollary H.2. We show that the clipped updates coincide with those of PPO-NoClip, so its convergence guarantees apply. Fix an outer iteration t t t and condition on the history before drawing D t D_t D t .
Let { θ ~ t k } k = 0 K \{\widetilde{\theta}_t^k\}_{k=0}^{K} { θ t k } k = 0 K denote the iterates of Algorithm 1 using the same batch and initialization at θ t \theta_t θ t , with all other parameters the same. By Lemma G.3 , the parameter θ η , t ⋆ = θ t + η θ ⋆ \theta_{\eta,t}^{\star}=\theta_t+\eta\theta_{\star} θ η , t ⋆ = θ t + η θ ⋆ realizes π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ . Theorem G.1 gives, with probability at least 1 − δ 1-\delta 1 − δ ,
∥ θ ~ t k − θ η , t ⋆ ∥ 2 ≤ 2 η B ⋆ , 0 ≤ k ≤ K . (178) \left\|\widetilde{\theta}_t^k-\theta_{\eta,t}^{\star}\right\|_2 \le2\eta B_{\star}, \qquad0\le k\le K. \tag*{(178)} θ t k − θ η , t ⋆ 2 ≤ 2 η B ⋆ , 0 ≤ k ≤ K . (178)
Going forward we condition on the event that Eq. (178) holds. Under this event, we next show that clipping remains inactive for all policy iterates by comparing each iterate with π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ , which obeys the following policy ratio bound. Since 0 ≤ r ( x , y ) ≤ B ϕ B ⋆ 0\le r(x,y)\le B_{\phi}B_{\star} 0 ≤ r ( x , y ) ≤ B ϕ B ⋆ , Eq. (112) gives, for every ( x , y ) (x,y) ( x , y ) ,
e − η B ϕ B ⋆ ≤ π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) ≤ e η B ϕ B ⋆ . e^{-\eta B_{\phi}B_{\star}} \le\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_t(y\mid x)} \le e^{\eta B_{\phi}B_{\star}}. e − η B ϕ B ⋆ ≤ π t ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ≤ e η B ϕ B ⋆ .
Applying Lemma B.12 in both directions, with feature diameter at most 2 B ϕ 2B_{\phi} 2 B ϕ , now gives for θ ~ t k \widetilde{\theta}_t^k θ t k ,
∣ log π ~ t k ( y ∣ x ) π t ( y ∣ x ) ∣ ≤ ∣ log π ~ t k ( y ∣ x ) π η , t ⋆ ( y ∣ x ) ∣ + ∣ log π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) ∣ \left|\log\frac{\widetilde{\pi}_t^k(y\mid x)}{\pi_t(y\mid x)}\right|
\le\left|\log\frac{\widetilde{\pi}_t^k(y\mid x)}{\pi_{\eta,t}^{\star}(y\mid x)}\right|+\left|\log\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_t(y\mid x)}\right| log π t ( y ∣ x ) π t k ( y ∣ x ) ≤ log π η , t ⋆ ( y ∣ x ) π t k ( y ∣ x ) + log π t ( y ∣ x ) π η , t ⋆ ( y ∣ x )
≤ 2 B ϕ ∥ θ ~ t k − θ η , t ⋆ ∥ 2 + η B ϕ B ⋆ < 5 η B ϕ B ⋆ = 5. (179) \le2B_{\phi}\left\|\widetilde{\theta}_t^k-\theta_{\eta,t}^{\star}\right\|_2+\eta B_{\phi}B_{\star}<5\eta B_{\phi}B_{\star}=5. \tag*{(179)} ≤ 2 B ϕ θ t k − θ η , t ⋆ 2 + η B ϕ B ⋆ < 5 η B ϕ B ⋆ = 5. (179)
The clipped updates therefore produce the same inner iterates as PPO-NoClip. By Eqs. (177) and (179) , clipping leaves the gradient unchanged at every θ ~ t k \widetilde{\theta}_t^k θ t k . Thus, starting from the common initialization θ t \theta_t θ t , induction gives θ t k = θ ~ t k \theta_t^k=\widetilde{\theta}_t^k θ t k = θ t k for all 0 ≤ k ≤ K 0\le k\le K 0 ≤ k ≤ K for the inner optimization loop. Thus, PPO-Lite also inherits the global optimality guarantee of PPO-NoClip.
Comparison with PPO and GRPO
Like PPO and GRPO, both PPO-NoClip and PPO-Lite reuse a batch sampled from π t \pi_t π t throughout the inner policy updates. The objectives below use our contextual bandit notation and probabilities of complete responses. They omit token averaging and normalization by the group standard deviation.
PPO. For a fixed value estimate V ^ t ( x ) \widehat{V}_t(x) V t ( x ) , the contextual bandit advantage estimate is A ^ t ( x , y ) : = r ( x , y ) − V ^ t ( x ) \widehat{A}_t(x,y):=r(x,y)-\widehat{V}_t(x) A t ( x , y ) := r ( x , y ) − V t ( x ) . The clipped surrogate of Schulman et al. [93 ] then takes the form
J η , t P P O ( π ) = E π t [ min { w t π ( x , y ) A ^ t ( x , y ) , [ w t π ( x , y ) ] 1 − τ − 1 + τ + A ^ t ( x , y ) } ] . \mathcal{J}_{\eta,t}^{\mathrm{PPO}}(\pi)=\mathbb{E}_{\pi_t}\left[\min\left\{w_t^\pi(x,y)\widehat{A}_t(x,y),\left[w_t^\pi(x,y)\right]_{1-\tau_-}^{1+\tau_+}\widehat{A}_t(x,y)\right\}\right]. J η , t PPO ( π ) = E π t [ min { w t π ( x , y ) A t ( x , y ) , [ w t π ( x , y ) ] 1 − τ − 1 + τ + A t ( x , y ) } ] .
The usual symmetric clipping interval corresponds to τ − = τ + \tau_-=\tau_+ τ − = τ + . This surrogate clips the importance weights applied to the advantages; an explicit KL penalty is a separate design choice.
GRPO. In place of a value estimate, GRPO uses the mean reward of a group of responses as its baseline ([96 ] ). For y 1 , … , y G y^1,\ldots,y^G y 1 , … , y G drawn independently from π t ( ⋅ ∣ x ) \pi_t(\cdot\mid x) π t ( ⋅ ∣ x ) , define
r ˉ t ( x ) : = 1 G ∑ j = 1 G r ( x , y j ) , A ^ t ( x , y j ) : = r ( x , y j ) − r ˉ t ( x ) . \bar{r}_t(x):=\frac{1}{G}\sum_{j=1}^{G}r(x,y^j), \qquad\widehat{A}_t(x,y^j):=r(x,y^j)-\bar{r}_t(x). r ˉ t ( x ) := G 1 j = 1 ∑ G r ( x , y j ) , A t ( x , y j ) := r ( x , y j ) − r ˉ t ( x ) .
To display the choice of reference explicitly, we write a surrogate for complete responses with a population KL penalty to π b a s e \pi_{\mathrm{base}} π base ,
J η , t G R P O ( π ) = E x ∼ μ , y 1 , … , y G ∼ i i d π t ( ⋅ ∣ x ) [ 1 G ∑ j = 1 G min { w t π ( x , y j ) A ^ t ( x , y j ) , [ w t π ( x , y j ) ] 1 − τ − 1 + τ + A ^ t ( x , y j ) } ] − 1 η D K L ( π ∥ π b a s e ) . (180) J_{\eta,t}^{\mathrm{GRPO}}(\pi)=\mathbb{E}_{\substack{x\sim\mu,\\ y^1,\ldots,y^G\mathrel{\overset{\mathrm{iid}}{\sim}}\pi_t(\cdot\mid x)}}\left[\frac{1}{G}\sum_{j=1}^{G}\min\left\{w_t^\pi(x,y^j)\widehat{A}_t(x,y^j),\left[w_t^\pi(x,y^j)\right]_{1-\tau_-}^{1+\tau_+}\widehat{A}_t(x,y^j)\right\}\right]-\frac{1}{\eta}D_{\mathrm{KL}}(\pi\mathbin\Vert\pi_{\mathrm{base}}). \tag*{(180)} J η , t GRPO ( π ) = E x ∼ μ , y 1 , … , y G ∼ iid π t ( ⋅ ∣ x ) [ G 1 j = 1 ∑ G min { w t π ( x , y j ) A t ( x , y j ) , [ w t π ( x , y j ) ] 1 − τ − 1 + τ + A t ( x , y j ) } ] − η 1 D KL ( π ∥ π base ) . (180)
The original GRPO algorithm normalizes the centered rewards by their group standard deviation and uses token-level ratios and a sampled KL penalty. Our simplified objective in Eq. (180) makes the roles of the baseline and regularization reference explicit for comparison with PPO-Lite.
Regularization, clipping, and baselines. APMD resets the regularization reference to π t \pi_t π t at each outer iteration, and the inner optimization tolerance ensures that π t + 1 \pi_{t+1} π t + 1 approximates the resulting target π η , t ∗ \pi_{\eta,t}^* π η , t ∗ . For PPO-Lite, clipping and the population KL penalty together control the statistical error without requiring bounded candidate policy ratios (Lemma H.2). Clipping modifies the objective; it does not constrain those ratios to lie in the clipping interval.
For a fixed prompt-dependent baseline V ^ t ( x ) \widehat{V}_t(x) V t ( x ) , the unclipped objective satisfies
E π t [ w t π ( x , y ) ( r ( x , y ) − V ^ t ( x ) ) ] = J ( π ) − E x ∼ μ [ V ^ t ( x ) ] . \mathbb{E}_{\pi_t}\left[w_t^\pi(x,y)\left(r(x,y)-\widehat{V}_t(x)\right)\right]=J(\pi)-\mathbb{E}_{x\sim\mu}\left[\widehat{V}_t(x)\right]. E π t [ w t π ( x , y ) ( r ( x , y ) − V t ( x ) ) ] = J ( π ) − E x ∼ μ [ V t ( x ) ] .
The baseline term is independent of π \pi π , so subtracting it leaves the population gradient unchanged. With clipping, subtracting a baseline can change the sign of an advantage and thus which clipping threshold applies, altering the objective and potentially its maximizer. The result proved here therefore concerns the objective using observed rewards in Eq. (150) ; extending it to estimated advantages or advantages normalized within each group requires accounting for those changes.
Part III
Squared-Loss Algorithms
Overview of Kimi-Style Algorithms
This section characterizes Kimi-style objectives with general baselines and establishes supporting results for their analysis.
Appendix § 16.1 .§ 1 describes the Kimi-style objective for general baseline functions, which is biased relative to the APMD target π η , t ⋆ \pi^{\star}_{\eta,t} π η , t ⋆ . We then bound the error from this bias and also characterize the objective curvature for any baseline.
Appendix § 16.1 .§ 2 gives an overview of the mean and normalizer baselines, and characterizes their individual objective biases and curvature. We then summarize our theoretical results in Table 1 and compare the sample and gradient step complexities of these two baseline choices.
In Appendix § 16.1 .§ 3 we state and prove supporting lemmas for general baselines, used for the above characterizations and in our baseline-specific convergence proofs. These results can be used to extend our results to other baseline choices.
General Kimi-Style Objectives
Fix an iteration t t t , a current policy π t > 0 \pi_t > 0 π t > 0 , and η > 0 \eta> 0 η > 0 . Given a baseline function b t : X → R b_t:\mathcal{X} \to\mathbb{R} b t : X → R that is independent of the candidate policy π \pi π , recall that the Kimi objective (Eq. (18) ) is
L t K i m i , b ( π ) = E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( 1 η log π ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + b t ( x ) ) 2 ] . L_t^{\mathrm{Kimi},b}(\pi) = \mathbb{E}_{x\sim\mu,\,y\sim\pi_t(\cdot|x)}\left[\left(\frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+b_t(x)\right)^2\right]. L t Kimi , b ( π ) = E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( η 1 log π t ( y ∣ x ) π ( y ∣ x ) − r ( x , y ) + b t ( x ) ) 2 ] .
The normalization function and the exact APMD update are
b t ∗ ( x ) : = 1 η log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] , (181) b_t^*(x) := \frac{1}{\eta}\log\mathbb{E}_{y\sim\pi_t(\cdot|x)}\left[e^{\eta r(x,y)}\right], \tag*{(181)} b t ∗ ( x ) := η 1 log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] , (181)
π η , t ∗ ( y ∣ x ) = π t ( y ∣ x ) e η ( r ( x , y ) − b t ∗ ( x ) ) . (182) \pi_{\eta,t}^*(y\mid x)=\pi_t(y\mid x)e^{\eta(r(x,y)-b_t^*(x))}. \tag*{(182)} π η , t ∗ ( y ∣ x ) = π t ( y ∣ x ) e η ( r ( x , y ) − b t ∗ ( x )) . (182)
Objective bias. Kimi-style objectives are generally biased. In particular, by Lemma I.1, the loss at the exact APMD update π η , t ∗ \pi_{\eta,t}^* π η , t ∗ is
L t K i m i , b ( π η , t ∗ ) = E x ∼ μ [ ( b t ∗ ( x ) − b t ( x ) ) 2 ] . L_t^{\mathrm{Kimi},b}(\pi_{\eta,t}^*)=\mathbb{E}_{x\sim\mu}\left[(b_t^*(x)-b_t(x))^2\right]. L t Kimi , b ( π η , t ∗ ) = E x ∼ μ [ ( b t ∗ ( x ) − b t ( x ) ) 2 ] .
Thus π η , t ∗ \pi_{\eta,t}^* π η , t ∗ minimizes the objective with zero loss when b t = b t ∗ b_t=b_t^* b t = b t ∗ , while a different baseline need not preserve this minimizer. We show (Lemma I.1) that every policy π > 0 \pi>0 π > 0 satisfies
MSE t ( π ) ≤ L t K i m i , b ( π ) − L t K i m i , b ( π η , t ∗ ) + E x ∼ μ [ ( b t ∗ ( x ) − b t ( x ) ) 2 − ( r ( x , π t ) − b t ( x ) ) 2 ] . \operatorname{MSE}_t(\pi)\le L_t^{\mathrm{Kimi},b}(\pi)-L_t^{\mathrm{Kimi},b}(\pi_{\eta,t}^*)+\mathbb{E}_{x\sim\mu}\left[(b_t^*(x)-b_t(x))^2-(r(x,\pi_t)-b_t(x))^2\right]. MSE t ( π ) ≤ L t Kimi , b ( π ) − L t Kimi , b ( π η , t ∗ ) + E x ∼ μ [ ( b t ∗ ( x ) − b t ( x ) ) 2 − ( r ( x , π t ) − b t ( x ) ) 2 ] .
The correction in this upper bound is independent of the candidate policy. Consequently, small excess loss guarantees small MSE up to an error controlled by the baseline. This bound need not be tight for a particular baseline; Appendix § 16.1 .§ 2 gives the comparisons used for the mean and normalizer baselines.
Curvature. Under the softmax linear parameterization in Assumption 2.1, Lemma I.4 shows that L t K i m i , b ( π θ ) L_t^{\mathrm{Kimi},b}(\pi_\theta) L t Kimi , b ( π θ ) is globally convex whenever b t ( x ) ≤ r ( x , π t ) b_t(x)\le r(x,\pi_t) b t ( x ) ≤ r ( x , π t ) for every x x x . More generally, fix a radius R > 0 R>0 R > 0 and suppose that b t ( x ) − r ( x , π t ) ≤ a b_t(x)-r(x,\pi_t)\le a b t ( x ) − r ( x , π t ) ≤ a for every x x x and some a ≥ 0 a\ge0 a ≥ 0 . The same lemma gives convexity on the ball ∥ θ − θ t ∥ 2 ≤ R \lVert\theta-\theta_t\rVert_2\le R ∥ θ − θ t ∥ 2 ≤ R whenever η a e D ϕ R ≤ 1 \eta a e^{D_\phi R}\le1 η a e D ϕ R ≤ 1 , where D ϕ D_\phi D ϕ is the feature diameter in Eq. (187) . Thus the comparison with b t ∗ b_t^* b t ∗ controls the objective bias, while the comparison with r ( x , π t ) r(x,\pi_t) r ( x , π t ) controls these curvature guarantees.
Mean and Normalizer Baselines: Bias, Curvature, and Complexity
We consider two choices of baseline. The comparisons below use Assumption 2.1 and the notation of Appendix § 16.1 .§ 3 .
Mean baseline (Appendix J). The method KimiPO (Algorithm 3) uses the mean baseline b t ( x ) = r ( x , π t ) b_t(x)=r(x,\pi_t) b t ( x ) = r ( x , π t ) . For this choice, we write L t K i m i = L t K i m i , b L_t^{\mathrm{Kimi}}=L_t^{\mathrm{Kimi},b} L t Kimi = L t Kimi , b , so the population objective is
L t K i m i ( π ) = E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( 1 η log π ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + r ( x , π t ) ) 2 ] . L_t^{\mathrm{Kimi}}(\pi)=\mathbb{E}_{x\sim\mu,\,y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+r(x,\pi_t)\right)^2\right]. L t Kimi ( π ) = E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( η 1 log π t ( y ∣ x ) π ( y ∣ x ) − r ( x , y ) + r ( x , π t ) ) 2 ] .
The algorithm estimates r ( x , π t ) r(x,\pi_t) r ( x , π t ) by averaging the rewards of G G G additional responses sampled independently of the response used to evaluate the loss.
Bias. Increasing G G G does not change the expected baseline r ( x , π t ) r(x,\pi_t) r ( x , π t ) or its difference from b t ⋆ ( x ) b_t^\star(x) b t ⋆ ( x ) . The sampled mean is unbiased, and its variance cancels from the expected excess loss by Lemma I.2. The remaining baseline error satisfies 0 ≤ b t ⋆ ( x ) − r ( x , π t ) ≤ η D ⋆ 2 / 8 0\leq b_t^\star(x)-r(x,\pi_t)\leq\eta D_\star^2/8 0 ≤ b t ⋆ ( x ) − r ( x , π t ) ≤ η D ⋆ 2 /8 by Eq. (215) . The excess-loss comparison in Lemma J.5 gives
MSE t ( π ) ≤ 4 3 ( L t K i m i ( π ) − L t K i m i ( π η , t ⋆ ) ) + η 4 D ⋆ 6 e η D ⋆ 48 . \operatorname{MSE}_t(\pi)\leq\frac{4}{3}\left(L_t^{\mathrm{Kimi}}(\pi)-L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^\star)\right)+\frac{\eta^4D_\star^6e^{\eta D_\star}}{48}. MSE t ( π ) ≤ 3 4 ( L t Kimi ( π ) − L t Kimi ( π η , t ⋆ ) ) + 48 η 4 D ⋆ 6 e η D ⋆ .
The inner convergence theorem requires η 4 D ⋆ 6 ≤ ε i n s q ≤ D ⋆ 2 \eta^4D_\star^6\leq\varepsilon_{\mathrm{in}}^{\mathrm{sq}}\leq D_\star^2 η 4 D ⋆ 6 ≤ ε in sq ≤ D ⋆ 2 (Eq. (225) ), which implies η D ⋆ ≤ 1 \eta D_\star\leq1 η D ⋆ ≤ 1 and controls this bias by a constant multiple of ε i n s q \varepsilon_{\mathrm{in}}^{\mathrm{sq}} ε in sq .
Curvature. The population objective L t K i m i ( π θ ) L_t^{\mathrm{Kimi}}(\pi_\theta) L t Kimi ( π θ ) is globally convex by Lemma J.3 . By Lemma I.5 with A = D ⋆ A=D_\star A = D ⋆ , it is also 2 D ϕ 2 / η 2 2D_\phi^2/\eta^2 2 D ϕ 2 / η 2 -smooth on the ball ∥ θ − θ t ∥ 2 ≤ 4 η B ⋆ \lVert\theta-\theta_t\rVert_2\leq4\eta B_\star ∥ θ − θ t ∥ 2 ≤ 4 η B ⋆ when η D ⋆ ≤ 1 / 3 \eta D_\star\leq1/3 η D ⋆ ≤ 1/3 .
Normalizer baseline (Appendix § 17.1 ). The method KimiPO-Norm (Algorithm 4) approximates the exact baseline b t ⋆ b_t^\star b t ⋆ in Eq. (181) . We write L t n o r m = L t K i m i , b L_t^{\mathrm{norm}}=L_t^{\mathrm{Kimi},b} L t norm = L t Kimi , b for this exact baseline. The algorithm uses G G G independent responses y 1 , … , y G ∼ π t ( ⋅ ∣ x ) y_1,\ldots,y_G\sim\pi_t(\cdot\mid x) y 1 , … , y G ∼ π t ( ⋅ ∣ x ) and sets
b ^ t G ( x ) : = 1 η log ( 1 G ∑ j = 1 G e η r ( x , y j ) ) . \widehat b_t^G(x):=\frac{1}{\eta}\log\left(\frac{1}{G}\sum_{j=1}^{G}e^{\eta r(x,y_j)}\right). b t G ( x ) := η 1 log ( G 1 j = 1 ∑ G e η r ( x , y j ) ) .
These responses are sampled independently of the response used to evaluate the loss. The auxiliary population objective L ~ t n o r m \widetilde L_t^{\mathrm{norm}} L t norm uses b ^ t G ( x ) = E [ b ^ t G ( x ) ∣ x ] \widehat b_t^G(x)=\mathbb{E}[\widehat b_t^G(x)\mid x] b t G ( x ) = E [ b t G ( x ) ∣ x ] as its baseline (Eq. (186) and Appendix § 17.1 .§ 2 ). By Lemma I.2, it differs from E [ L ^ t n o r m ] \mathbb{E}[\widehat L_t^{\mathrm{norm}}] E [ L t norm ] only by the baseline variance and hence has the same gradients and Hessians. These generally differ from those of L t n o r m L_t^{\mathrm{norm}} L t norm , whose baseline is the exact b t ⋆ b_t^\star b t ⋆ .
0 ≤ b t ⋆ ( x ) − E [ b ^ t G ( x ) ∣ x ] ≤ ( e η D ⋆ − 1 ) 2 8 η G . 0\leq b_t^\star(x)-\mathbb{E}\left[\widehat b_t^G(x)\mid x\right]\leq\frac{\left(e^{\eta D_\star}-1\right)^2}{8\eta G}. 0 ≤ b t ⋆ ( x ) − E [ b t G ( x ) ∣ x ] ≤ 8 η G ( e η D ⋆ − 1 ) 2 .
The decomposition in Eq. (191) gives MSE t ( π ) ≤ L ~ t n o r m ( π ) \operatorname{MSE}_t(\pi)\leq\widetilde L_t^{\mathrm{norm}}(\pi) MSE t ( π ) ≤ L t norm ( π ) , and the bias bound above yields
L ~ t n o r m ( π η , t ⋆ ) ≤ ( e η D ⋆ − 1 ) 4 64 η 2 G 2 \widetilde L_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\leq\frac{\left(e^{\eta D_\star}-1\right)^4}{64\eta^2G^2} L t norm ( π η , t ⋆ ) ≤ 64 η 2 G 2 ( e η D ⋆ − 1 ) 4
as in Lemma K.3 .
Curvature. The expected baseline satisfies b ^ t G ( x ) − r ( x , π t ) ≤ η D ⋆ 2 / 8 \widehat b_t^G(x)-r(x,\pi_t)\leq\eta D_\star^2/8 b t G ( x ) − r ( x , π t ) ≤ η D ⋆ 2 /8 by Lemmas J.2 and K.2 . Applying Lemma I.4 with a = η D ⋆ 2 / 8 a=\eta D_\star^2/8 a = η D ⋆ 2 /8 gives convexity of L ~ t n o r m \widetilde L_t^{\mathrm{norm}} L t norm on the ball ∥ θ − θ t ∥ 2 ≤ 4 η B ⋆ \lVert\theta-\theta_t\rVert_2\leq4\eta B_\star ∥ θ − θ t ∥ 2 ≤ 4 η B ⋆ when η D ⋆ ≤ 1 / 3 \eta D_\star\leq1/3 η D ⋆ ≤ 1/3 . On this ball, Lemma I.5 with A = D ⋆ A=D_\star A = D ⋆ also gives 2 D ϕ 2 / η 2 2D_\phi^2/\eta^2 2 D ϕ 2 / η 2 -smoothness.
Comparison of sample and gradient step complexities (Table 1 ). The two baselines lead to different ways of controlling approximation error. The mean baseline reduces bias by decreasing η \eta η , while the normalizer baseline does so by increasing the number of additional responses G G G . In particular, Corollary K.1 keeps η = 1 / ( 3 R max + 4 D ⋆ ) \eta= 1/(3R_{\max} + 4D_{\star}) η = 1/ ( 3 R m a x + 4 D ⋆ ) fixed and chooses G G G according to Eq. (295) to meet the regression accuracy ε i n S q \varepsilon_{\mathrm{in}}^{\mathrm{Sq}} ε in Sq in Eq. (294) . This gives T = Θ ~ ( ( R max + D ⋆ ) / ε ) T = \widetilde{\Theta}((R_{\max} + D_{\star})/\varepsilon) T = Θ (( R m a x + D ⋆ ) / ε ) outer iterations.
For fixed problem parameters as ε → 0 \varepsilon\to0 ε → 0 , the bounds for the normalizer baseline save a factor of 1 / ε 1/\sqrt{\varepsilon} 1/ ε in outer iterations and gradient steps, while the bound on sampled responses is larger by a factor of 1 / ε 1/\sqrt{\varepsilon} 1/ ε , up to logarithmic factors. Table 1 gives the parameter choices and complexities from Corollaries J.1 and K.1. Since the additional responses are used only to form the baseline, they can be collected in parallel and are not needed for gradient computation.
Supporting Lemmas for a General Baseline
Notation. Fix an outer iteration t t t , a current policy π t > 0 \pi_t > 0 π t > 0 , and η > 0 \eta> 0 η > 0 . Let r : X × Y → [ 0 , R max ] r : \mathcal{X} \times\mathcal{Y} \to[0,R_{\max}] r : X × Y → [ 0 , R m a x ] and let b t : X → R b_t : \mathcal{X} \to\mathbb{R} b t : X → R be the baseline being estimated. For G ≥ 1 G \ge1 G ≥ 1 , form b ^ t G ( x ) \widehat{b}_t^G(x) b t G ( x ) from independent responses y 1 ′ , … , y G ′ ∼ π t ( ⋅ ∣ x ) y'_1,\ldots,y'_G \sim\pi_t(\cdot\mid x) y 1 ′ , … , y G ′ ∼ π t ( ⋅ ∣ x ) . Both b t b_t b t and the law of b ^ t G ( x ) \widehat{b}_t^G(x) b t G ( x ) are independent of the candidate policy. The conditional mean and variance of the estimate are
b ~ t G ( x ) : = E y 1 ′ , … , y G ′ ∼ π t ( ⋅ ∣ x ) [ b ^ t G ( x ) ] , \widetilde{b}_t^G(x) := \mathbb{E}_{y'_1,\ldots,y'_G\sim\pi_t(\cdot\mid x)}\left[\widehat{b}_t^G(x)\right], b t G ( x ) := E y 1 ′ , … , y G ′ ∼ π t ( ⋅ ∣ x ) [ b t G ( x ) ] ,
( σ t G ( x ) ) 2 : = V y 1 ′ , … , y G ′ ∼ π t ( ⋅ ∣ x ) [ b ^ t G ( x ) ] . (183) \left(\sigma_t^G(x)\right)^2 := \mathbb{V}_{y'_1,\ldots,y'_G\sim\pi_t(\cdot\mid x)}\left[\widehat{b}_t^G(x)\right]. \tag*{(183)} ( σ t G ( x ) ) 2 := V y 1 ′ , … , y G ′ ∼ π t ( ⋅ ∣ x ) [ b t G ( x ) ] . (183)
We leave the additional response samples implicit in b ^ t G \widehat{b}_t^G b t G . The population objective with baseline b t b_t b t is
L t K i m i , b ( π ) : = E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( 1 η log π ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + b t ( x ) ) 2 ] . (184) L_t^{\mathrm{Kimi},b}(\pi) := \mathbb{E}_{x\sim\mu,\ y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+b_t(x)\right)^2\right]. \tag*{(184)} L t Kimi , b ( π ) := E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( η 1 log π t ( y ∣ x ) π ( y ∣ x ) − r ( x , y ) + b t ( x ) ) 2 ] . (184)
We abbreviate this joint expectation by E π t \mathbb{E}_{\pi_t} E π t . Draw N N N independent triples ( x i , y i , b ^ t G ( x i ) ) (x_i,y_i,\widehat{b}_t^G(x_i)) ( x i , y i , b t G ( x i )) , where x i ∼ μ x_i\sim\mu x i ∼ μ , y i ∼ π t ( ⋅ ∣ x i ) y_i\sim\pi_t(\cdot\mid x_i) y i ∼ π t ( ⋅ ∣ x i ) , and each b ^ t G ( x i ) \widehat{b}_t^G(x_i) b t G ( x i ) is formed from G G G additional responses independently of y i y_i y i given x i x_i x i . Define the empirical and auxiliary population objectives by
L ^ t K i m i , b ( π ) : = 1 N ∑ i = 1 N ( 1 η log π ( y i ∣ x i ) π t ( y i ∣ x i ) − r ( x i , y i ) + b ^ t G ( x i ) ) 2 , (185) \widehat{L}_t^{\mathrm{Kimi},b}(\pi) := \frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{\eta}\log\frac{\pi(y_i\mid x_i)}{\pi_t(y_i\mid x_i)}-r(x_i,y_i)+\widehat{b}_t^G(x_i)\right)^2, \tag*{(185)} L t Kimi , b ( π ) := N 1 i = 1 ∑ N ( η 1 log π t ( y i ∣ x i ) π ( y i ∣ x i ) − r ( x i , y i ) + b t G ( x i ) ) 2 , (185)
L ~ t K i m i , b ( π ) : = E π t [ ( 1 η log π ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + b ~ t G ( x ) ) 2 ] . (186) \widetilde{L}_t^{\mathrm{Kimi},b}(\pi) := \mathbb{E}_{\pi_t}\left[\left(\frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+\widetilde{b}_t^G(x)\right)^2\right]. \tag*{(186)} L t Kimi , b ( π ) := E π t [ ( η 1 log π t ( y ∣ x ) π ( y ∣ x ) − r ( x , y ) + b t G ( x ) ) 2 ] . (186)
The superscript b b b identifies the baseline being estimated in all three objectives, while the auxiliary objective uses b ~ t G \widetilde{b}_t^G b t G .
Recall from Assumption 2.1 that B ϕ : = max x , y ∥ ϕ ( x , y ) ∥ 2 B_{\phi} := \max_{x,y}\|\phi(x,y)\|_{2} B ϕ := max x , y ∥ ϕ ( x , y ) ∥ 2 and B ⋆ : = ∥ θ ⋆ ∥ 2 B_{\star} := \|\theta_{\star}\|_{2} B ⋆ := ∥ θ ⋆ ∥ 2 . We also use the feature diameter and the resulting bound on the reward range,
D ϕ : = max x , y , y ′ ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 , D ⋆ : = D ϕ B ⋆ . (187) D_{\phi} := \max_{x,y,y'}\|\phi(x,y)-\phi(x,y')\|_{2}, \qquad D_{\star} := D_{\phi}B_{\star}. \tag*{(187)} D ϕ := x , y , y ′ max ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 , D ⋆ := D ϕ B ⋆ . (187)
Thus D ϕ ≤ 2 B ϕ D_{\phi} \le2B_{\phi} D ϕ ≤ 2 B ϕ and D ⋆ ≤ 2 B ϕ B ⋆ D_{\star} \le2B_{\phi}B_{\star} D ⋆ ≤ 2 B ϕ B ⋆ . Under the linear reward assumption in Assumption 2.1,
∣ r ( x , y ) − r ( x , y ′ ) ∣ ≤ D ⋆ , ∀ x ∈ X , y , y ′ ∈ Y . (188) |r(x,y)-r(x,y')| \le D_{\star}, \qquad\forall x \in\mathcal{X},\ y,y' \in\mathcal{Y}. \tag*{(188)} ∣ r ( x , y ) − r ( x , y ′ ) ∣ ≤ D ⋆ , ∀ x ∈ X , y , y ′ ∈ Y . (188)
We retain R max R_{\max} R m a x and D ⋆ D_{\star} D ⋆ separately in the appendix bounds. The main-body bounds use B : = B ϕ B ⋆ B := B_{\phi}B_{\star} B := B ϕ B ⋆ . Under linear rewards, we can take R max ≤ B R_{\max} \le B R m a x ≤ B , and D ⋆ ≤ 2 B D_{\star} \le2B D ⋆ ≤ 2 B . Throughout the Kimi analyses, ε i n S q \varepsilon_{\mathrm{in}}^{\mathrm{Sq}} ε in Sq denotes a local regression accuracy, while ε i n \varepsilon_{\mathrm{in}} ε in retains its meaning as the KL tolerance in Eq. (6) .
For a positive policy π \pi π write its centered logits and residual as
l t , η π ( x , y ) : = 1 η log π ( y ∣ x ) π t ( y ∣ x ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ 1 η log π ( y ′ ∣ x ) π t ( y ′ ∣ x ) ] , l_{t,\eta}^{\pi}(x,y) := \frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_{t}(y\mid x)}-\mathbb{E}_{y'\sim\pi_{t}(\cdot\mid x)}\left[\frac{1}{\eta}\log\frac{\pi(y'\mid x)}{\pi_{t}(y'\mid x)}\right], l t , η π ( x , y ) := η 1 log π t ( y ∣ x ) π ( y ∣ x ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ η 1 log π t ( y ′ ∣ x ) π ( y ′ ∣ x ) ] ,
s t π ( x , y ) : = l t , η π ( x , y ) − ( r ( x , y ) − r ( x , π t ) ) . (189) s_{t}^{\pi}(x,y) := l_{t,\eta}^{\pi}(x,y)-\left(r(x,y)-r(x,\pi_{t})\right). \tag*{(189)} s t π ( x , y ) := l t , η π ( x , y ) − ( r ( x , y ) − r ( x , π t ) ) . (189)
The conditional and average mean-squared errors are
MSE t ( π ; x ) : = E y ∼ π t ( ⋅ ∣ x ) [ ( s t π ( x , y ) ) 2 ] , MSE t ( π ) : = E x ∼ μ [ MSE t ( π ; x ) ] . \operatorname{MSE}_{t}(\pi;x) := \mathbb{E}_{y\sim\pi_{t}(\cdot\mid x)}\left[(s_{t}^{\pi}(x,y))^{2}\right], \qquad\operatorname{MSE}_{t}(\pi) := \mathbb{E}_{x\sim\mu}[\operatorname{MSE}_{t}(\pi;x)]. MSE t ( π ; x ) := E y ∼ π t ( ⋅ ∣ x ) [ ( s t π ( x , y ) ) 2 ] , MSE t ( π ) := E x ∼ μ [ MSE t ( π ; x )] .
For the feature map in Assumption 2.1, write
Σ t : = E x ∼ μ [ Cov y ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ) ] ] . (190) \Sigma_{t} := \mathbb{E}_{x\sim\mu}\left[\operatorname{Cov}_{y\sim\pi_{t}(\cdot\mid x)}[\phi(x,y)]\right]. \tag*{(190)} Σ t := E x ∼ μ [ Cov y ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y )] ] . (190)
Under Assumption 2.1, the centered residual s t π s_{t}^{\pi} s t π has the representation in Eq. (79) with θ t + 1 \theta_{t+1} θ t + 1 replaced by θ \theta θ .
Lemma I.1. For any baseline b t b_t b t , every policy π > 0 \pi> 0 π > 0 satisfies
MSE t ( π ) ≤ L t K i m i , b ( π ) = MSE t ( π ) + E x ∼ μ [ ( r ( x , π t ) − b t ( x ) + 1 η D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) ) 2 ] . (191) \operatorname{MSE}_{t}(\pi) \le L_{t}^{\mathrm{Kimi},b}(\pi) = \operatorname{MSE}_{t}(\pi) + \mathbb{E}_{x\sim\mu}\left[\left(r(x,\pi_t)-b_t(x)+\frac{1}{\eta}D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\parallel\pi(\cdot\mid x))\right)^2\right]. \tag*{(191)} MSE t ( π ) ≤ L t Kimi , b ( π ) = MSE t ( π ) + E x ∼ μ [ ( r ( x , π t ) − b t ( x ) + η 1 D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) ) 2 ] . (191) L t K i m i , b ( π t , η ⋆ ) = E x ∼ μ [ ( b t ⋆ ( x ) − b t ( x ) ) 2 ] . (192) L_{t}^{\mathrm{Kimi},b}(\pi_{t,\eta}^{\star}) = \mathbb{E}_{x\sim\mu}\left[(b_t^{\star}(x)-b_t(x))^2\right]. \tag*{(192)} L t Kimi , b ( π t , η ⋆ ) = E x ∼ μ [ ( b t ⋆ ( x ) − b t ( x ) ) 2 ] . (192)
The same identities hold for L ~ t K i m i , b \widetilde{L}_{t}^{\mathrm{Kimi},b} L t Kimi , b with b t b_t b t replaced by b ~ t G \widetilde{b}_{t}^{G} b t G .
Proof of Lemma I.1. The definition of s t π s_{t}^{\pi} s t π in Eq. (189) gives
1 η log π ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + b t ( x ) = s t π ( x , y ) − ( r ( x , π t ) − b t ( x ) + 1 η D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) ) . \frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+b_t(x)=s_t^\pi(x,y)-\left(r(x,\pi_t)-b_t(x)+\frac{1}{\eta}D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\parallel\pi(\cdot\mid x))\right). η 1 log π t ( y ∣ x ) π ( y ∣ x ) − r ( x , y ) + b t ( x ) = s t π ( x , y ) − ( r ( x , π t ) − b t ( x ) + η 1 D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) ) .
Substituting into the definition of L t K i m i , b L_t^{\mathrm{Kimi},b} L t Kimi , b (Eq. (184) ) gives
L t K i m i , b ( π ) = E π t [ s t π ( x , y ) 2 ] − 2 E x ∼ μ [ ( r ( x , π t ) − b t ( x ) + 1 η D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) ) E y ∼ π t ( ⋅ ∣ x ) [ s t π ( x , y ) ] ] + E x ∼ μ [ ( r ( x , π t ) − b t ( x ) + 1 η D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) ) 2 ] = MSE t ( π ) + E x ∼ μ [ ( r ( x , π t ) − b t ( x ) + 1 η D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) ) 2 ] ≥ MSE t ( π ) . \begin{aligned}
L_t^{\mathrm{Kimi},b}(\pi)
&=\mathbb{E}_{\pi_t}\left[s_t^\pi(x,y)^2\right]
-2\mathbb{E}_{x\sim\mu}\left[\left(r(x,\pi_t)-b_t(x)+\frac{1}{\eta}D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\parallel\pi(\cdot\mid x))\right)\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[s_t^\pi(x,y)]\right]\\
&\quad+\mathbb{E}_{x\sim\mu}\left[\left(r(x,\pi_t)-b_t(x)+\frac{1}{\eta}D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\parallel\pi(\cdot\mid x))\right)^2\right]\\
&=\operatorname{MSE}_t(\pi)+\mathbb{E}_{x\sim\mu}\left[\left(r(x,\pi_t)-b_t(x)+\frac{1}{\eta}D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\parallel\pi(\cdot\mid x))\right)^2\right]\ge\operatorname{MSE}_t(\pi).
\end{aligned} L t Kimi , b ( π ) = E π t [ s t π ( x , y ) 2 ] − 2 E x ∼ μ [ ( r ( x , π t ) − b t ( x ) + η 1 D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) ) E y ∼ π t ( ⋅ ∣ x ) [ s t π ( x , y )] ] + E x ∼ μ [ ( r ( x , π t ) − b t ( x ) + η 1 D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) ) 2 ] = MSE t ( π ) + E x ∼ μ [ ( r ( x , π t ) − b t ( x ) + η 1 D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) ) 2 ] ≥ MSE t ( π ) .
The second equality uses E y ∼ π t ( ⋅ ∣ x ) [ s t π ( x , y ) ] = 0 \mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[s_t^\pi(x,y)]=0 E y ∼ π t ( ⋅ ∣ x ) [ s t π ( x , y )] = 0 . This proves Eq. (191) . For π t , η ⋆ \pi_{t,\eta}^{\star} π t , η ⋆ , the exact update in Eq. (182) gives
1 η log π t , η ⋆ ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + b t ( x ) = b t ( x ) − b t ⋆ ( x ) . \frac{1}{\eta}\log\frac{\pi_{t,\eta}^{\star}(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+b_t(x)=b_t(x)-b_t^{\star}(x). η 1 log π t ( y ∣ x ) π t , η ⋆ ( y ∣ x ) − r ( x , y ) + b t ( x ) = b t ( x ) − b t ⋆ ( x ) .
Substituting this into L t K i m i , b L_t^{\mathrm{Kimi},b} L t Kimi , b proves Eq. (192) . Finally, the result for L ~ t K i m i , b \widetilde{L}_t^{\mathrm{Kimi},b} L t Kimi , b follows by substituting b t = b ~ t G b_t=\widetilde{b}_t^G b t = b t G . ∎
Lemma I.2. For any G ≥ 1 G \ge1 G ≥ 1 , let L ^ t K i m i , b \widehat{L}_{t}^{\mathrm{Kimi},b} L t Kimi , b (Eq. (185) ) be computed from N N N independent triples ( x i , y i , b ^ t G ( x i ) ) (x_i,y_i,\widehat{b}_{t}^{G}(x_i)) ( x i , y i , b t G ( x i )) , with b ^ t G ( x i ) \widehat{b}_{t}^{G}(x_i) b t G ( x i ) formed from responses y i , 1 ′ , … , y i , G ′ y'_{i,1},\ldots,y'_{i,G} y i , 1 ′ , … , y i , G ′ . For every policy π > 0 \pi> 0 π > 0 fixed independently of this batch,
E x i ∼ μ , 1 ≤ i ≤ N y i , y i , 1 ′ , … , y i , G ′ ∼ π t ( ⋅ ∣ x i ) [ L ^ t K i m i , b ( π ) ] = L ~ t K i m i , b ( π ) + E x ∼ μ [ ( σ t G ( x ) ) 2 ] . \mathbb{E}_{\substack{x_i\sim\mu,\ 1\le i\le N\\ y_i,y'_{i,1},\ldots,y'_{i,G}\sim\pi_t(\cdot\mid x_i)}}\left[\widehat{L}_{t}^{\mathrm{Kimi},b}(\pi)\right]=\widetilde{L}_{t}^{\mathrm{Kimi},b}(\pi)+\mathbb{E}_{x\sim\mu}\left[\left(\sigma_{t}^{G}(x)\right)^2\right]. E x i ∼ μ , 1 ≤ i ≤ N y i , y i , 1 ′ , … , y i , G ′ ∼ π t ( ⋅ ∣ x i ) [ L t Kimi , b ( π ) ] = L t Kimi , b ( π ) + E x ∼ μ [ ( σ t G ( x ) ) 2 ] .
Proof of Lemma I.2. Since b ^ t G ( x i ) \widehat{b}_{t}^{G}(x_i) b t G ( x i ) is independent of y i y_i y i conditional on x i x_i x i , its conditional mean and variance given ( x i , y i ) (x_i,y_i) ( x i , y i ) are b ~ t G ( x i ) \widetilde{b}_{t}^{G}(x_i) b t G ( x i ) and ( σ t G ( x i ) ) 2 (\sigma_{t}^{G}(x_i))^2 ( σ t G ( x i ) ) 2 (Eq. (183) ). Thus
E x i ∼ μ , 1 ≤ i ≤ N y i , y i , 1 ′ , … , y i , G ′ ∼ π t ( ⋅ ∣ x i ) [ ( 1 η log π ( y i ∣ x i ) π t ( y i ∣ x i ) − r ( x i , y i ) + b ^ t G ( x i ) ) 2 ] = E x i ∼ μ , y i ∼ π t ( ⋅ ∣ x i ) [ ( 1 η log π ( y i ∣ x i ) π t ( y i ∣ x i ) − r ( x i , y i ) + b ~ t G ( x i ) ) 2 ] + E x i ∼ μ [ ( σ t G ( x i ) ) 2 ] . \begin{aligned}
\mathbb{E}_{\substack{x_i\sim\mu,\ 1\le i\le N\\ y_i,y'_{i,1},\ldots,y'_{i,G}\sim\pi_t(\cdot\mid x_i)}}\left[\left(\frac{1}{\eta}\log\frac{\pi(y_i\mid x_i)}{\pi_t(y_i\mid x_i)}-r(x_i,y_i)+\widehat{b}_{t}^{G}(x_i)\right)^2\right] \\
&=\mathbb{E}_{x_i\sim\mu,\ y_i\sim\pi_t(\cdot\mid x_i)}\left[\left(\frac{1}{\eta}\log\frac{\pi(y_i\mid x_i)}{\pi_t(y_i\mid x_i)}-r(x_i,y_i)+\widetilde{b}_{t}^{G}(x_i)\right)^2\right]+\mathbb{E}_{x_i\sim\mu}\left[\left(\sigma_{t}^{G}(x_i)\right)^2\right].
\end{aligned} E x i ∼ μ , 1 ≤ i ≤ N y i , y i , 1 ′ , … , y i , G ′ ∼ π t ( ⋅ ∣ x i ) [ ( η 1 log π t ( y i ∣ x i ) π ( y i ∣ x i ) − r ( x i , y i ) + b t G ( x i ) ) 2 ] = E x i ∼ μ , y i ∼ π t ( ⋅ ∣ x i ) [ ( η 1 log π t ( y i ∣ x i ) π ( y i ∣ x i ) − r ( x i , y i ) + b t G ( x i ) ) 2 ] + E x i ∼ μ [ ( σ t G ( x i ) ) 2 ] .
Averaging over i i i proves the result by the definitions of L ^ t K i m i , b \widehat{L}_{t}^{\mathrm{Kimi},b} L t Kimi , b (Eq. (185) ) and L ~ t K i m i , b \widetilde{L}_{t}^{\mathrm{Kimi},b} L t Kimi , b (Eq. (186) ).
:::
Lemma I.3. For the softmax linear policy class in Assumption 2.1, fix a parameter radius R > 0 R>0 R > 0 . Every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \|\theta-\theta_t\|_2\le R ∥ θ − θ t ∥ 2 ≤ R satisfies
E y ∼ π t ( ⋅ ∣ x ) [ ( π θ ( y ∣ x ) π t ( y ∣ x ) − 1 ) 2 ] ≤ D ϕ 2 R 2 e D ϕ R 4 , ∀ x ∈ X . \mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}-1\right)^2\right]\le\frac{D_\phi^2R^2e^{D_\phi R}}{4},\qquad\forall x\in\mathcal{X}. E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π θ ( y ∣ x ) − 1 ) 2 ] ≤ 4 D ϕ 2 R 2 e D ϕ R , ∀ x ∈ X .
Proof of Lemma I.3. Fix x ∈ X x\in\mathcal{X} x ∈ X and θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \|\theta-\theta_t\|_2\le R ∥ θ − θ t ∥ 2 ≤ R . Under the softmax linear parameterization,
π θ ( y ∣ x ) / π t ( y ∣ x ) π θ ( y ′ ∣ x ) / π t ( y ′ ∣ x ) = e ⟨ ϕ ( x , y ) − ϕ ( x , y ′ ) , θ − θ t ⟩ ≤ e D ϕ R , y , y ′ ∈ Y . \frac{\pi_\theta(y\mid x)/\pi_t(y\mid x)}{\pi_\theta(y'\mid x)/\pi_t(y'\mid x)}=e^{\langle\phi(x,y)-\phi(x,y'),\theta-\theta_t\rangle}\le e^{D_\phi R},\qquad y,y'\in\mathcal{Y}. π θ ( y ′ ∣ x ) / π t ( y ′ ∣ x ) π θ ( y ∣ x ) / π t ( y ∣ x ) = e ⟨ ϕ ( x , y ) − ϕ ( x , y ′ ) , θ − θ t ⟩ ≤ e D ϕ R , y , y ′ ∈ Y .
Let m = inf y π θ ( y ∣ x ) / π t ( y ∣ x ) > 0 m=\inf_y\pi_\theta(y\mid x)/\pi_t(y\mid x)>0 m = inf y π θ ( y ∣ x ) / π t ( y ∣ x ) > 0 . The preceding inequality gives m ≤ π θ ( y ∣ x ) / π t ( y ∣ x ) ≤ e D ϕ R m m\le\pi_\theta(y\mid x)/\pi_t(y\mid x)\le e^{D_\phi R}m m ≤ π θ ( y ∣ x ) / π t ( y ∣ x ) ≤ e D ϕ R m , hence
( π θ ( y ∣ x ) π t ( y ∣ x ) ) 2 ≤ ( 1 + e D ϕ R ) m π θ ( y ∣ x ) π t ( y ∣ x ) − e D ϕ R m 2 . \left(\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}\right)^2\le(1+e^{D_\phi R})m\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}-e^{D_\phi R}m^2. ( π t ( y ∣ x ) π θ ( y ∣ x ) ) 2 ≤ ( 1 + e D ϕ R ) m π t ( y ∣ x ) π θ ( y ∣ x ) − e D ϕ R m 2 .
Since E y ∼ π t ( ⋅ ∣ x ) [ π θ ( y ∣ x ) / π t ( y ∣ x ) ] = 1 \mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[\pi_\theta(y\mid x)/\pi_t(y\mid x)]=1 E y ∼ π t ( ⋅ ∣ x ) [ π θ ( y ∣ x ) / π t ( y ∣ x )] = 1 ,
E y ∼ π t ( ⋅ ∣ x ) [ ( π θ ( y ∣ x ) π t ( y ∣ x ) − 1 ) 2 ] = E y ∼ π t ( ⋅ ∣ x ) [ ( π θ ( y ∣ x ) π t ( y ∣ x ) ) 2 ] − 1 ≤ ( 1 + e D ϕ R ) m − e D ϕ R m 2 − 1 ≤ ( e D ϕ R − 1 ) 2 4 e D ϕ R ≤ D ϕ 2 R 2 e D ϕ R 4 . \begin{aligned}
\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}-1\right)^2\right]=\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}\right)^2\right]-1 \\
&\le(1+e^{D_\phi R})m-e^{D_\phi R}m^2-1\le\frac{(e^{D_\phi R}-1)^2}{4e^{D_\phi R}}\le\frac{D_\phi^2R^2e^{D_\phi R}}{4}.
\end{aligned} E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π θ ( y ∣ x ) − 1 ) 2 ] = E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π θ ( y ∣ x ) ) 2 ] − 1 ≤ ( 1 + e D ϕ R ) m − e D ϕ R m 2 − 1 ≤ 4 e D ϕ R ( e D ϕ R − 1 ) 2 ≤ 4 D ϕ 2 R 2 e D ϕ R .
The last inequality uses e a − 1 ≤ a e a e^a-1\le ae^a e a − 1 ≤ a e a for a ≥ 0 a\ge0 a ≥ 0 .
:::
Lemma I.4. Suppose that π θ \pi_\theta π θ has the softmax linear parameterization in Assumption 2.1. Fix η , R > 0 \eta,R>0 η , R > 0 and a ≥ 0 a\ge0 a ≥ 0 such that b t ( x ) − r ( x , π t ) ≤ a b_t(x)-r(x,\pi_t)\le a b t ( x ) − r ( x , π t ) ≤ a for every x x x . Every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \|\theta-\theta_t\|_2\le R ∥ θ − θ t ∥ 2 ≤ R satisfies
∇ 2 L t K i m i , b ( π θ ) ⪰ 2 η 2 ( 1 − η a e D ϕ R ) Σ t . \nabla^2 L_{t}^{\mathrm{Kimi},b}(\pi_\theta)\succeq\frac{2}{\eta^2}(1-\eta ae^{D_\phi R})\Sigma_t. ∇ 2 L t Kimi , b ( π θ ) ⪰ η 2 2 ( 1 − η a e D ϕ R ) Σ t . In particular, if b t ( x ) ≤ r ( x , π t ) b_t(x)\le r(x,\pi_t) b t ( x ) ≤ r ( x , π t ) for every x x x , then L t K i m i , b ( π θ ) L_{t}^{\mathrm{Kimi},b}(\pi_\theta) L t Kimi , b ( π θ ) is globally convex and
∇ 2 L t K i m i , b ( π θ ) ⪰ 2 η 2 Σ t , θ ∈ R d . (193) \nabla^2L_{t}^{\mathrm{Kimi},b}(\pi_\theta)\succeq\frac{2}{\eta^2}\Sigma_t,\qquad\theta\in\mathbb{R}^d. \tag*{(193)} ∇ 2 L t Kimi , b ( π θ ) ⪰ η 2 2 Σ t , θ ∈ R d . (193) The same statements hold for L ~ t K i m i , b \widetilde{L}_{t}^{\mathrm{Kimi},b} L t Kimi , b with b t b_t b t replaced by b ~ t G \widetilde{b}_{t}^{G} b t G .
Proof of Lemma I.4. Fix θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \lVert\theta-\theta_t\rVert_2 \le R ∥ θ − θ t ∥ 2 ≤ R . We differentiate the decomposition of L t K i m i , b L_t^{\mathrm{Kimi},b} L t Kimi , b in Eq. (191) of Lemma I.1. Write
q θ ( x ) : = r ( x , π t ) − b t ( x ) + 1 η D K L ( π t ( ⋅ ∣ x ) ∥ π θ ( ⋅ ∣ x ) ) . q_\theta(x) := r(x,\pi_t)-b_t(x)+\frac{1}{\eta}D_{\mathrm{KL}}\left(\pi_t(\cdot\mid x)\mathbin{\|}\pi_\theta(\cdot\mid x)\right). q θ ( x ) := r ( x , π t ) − b t ( x ) + η 1 D KL ( π t ( ⋅ ∣ x ) ∥ π θ ( ⋅ ∣ x ) ) .
Then L t K i m i , b ( π θ ) = MSE t ( π θ ) + E x ∼ μ [ q θ ( x ) 2 ] L_t^{\mathrm{Kimi},b}(\pi_\theta)=\operatorname{MSE}_t(\pi_\theta)+\mathbb{E}_{x\sim\mu}[q_\theta(x)^2] L t Kimi , b ( π θ ) = MSE t ( π θ ) + E x ∼ μ [ q θ ( x ) 2 ] .
Using ∇ log π θ ( y ∣ x ) = ϕ ( x , y ) − E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] \nabla\log\pi_\theta(y\mid x)=\phi(x,y)-\mathbb{E}_{y'\sim\pi_\theta(\cdot\mid x)}[\phi(x,y')] ∇ log π θ ( y ∣ x ) = ϕ ( x , y ) − E y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] (Lemma B.14) in the definition of s t π θ s_t^{\pi_\theta} s t π θ (Eq. (189) ) gives
∇ s t π θ ( x , y ) = 1 η ( ∇ log π θ ( y ∣ x ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ∇ log π θ ( y ′ ∣ x ) ] ) = 1 η ( ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] ) . \begin{aligned}
\nabla s_t^{\pi_\theta}(x,y)=\frac{1}{\eta}\left(\nabla\log\pi_\theta(y\mid x)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\nabla\log\pi_\theta(y'\mid x)]\right) \\
&=\frac{1}{\eta}\left(\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\phi(x,y')]\right).
\end{aligned} ∇ s t π θ ( x , y ) = η 1 ( ∇ log π θ ( y ∣ x ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ∇ log π θ ( y ′ ∣ x )] ) = η 1 ( ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] ) .
This gradient is independent of θ \theta θ , so
∇ 2 MSE t ( π θ ) = 2 E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ∇ s t π θ ( x , y ) ∇ s t π θ ( x , y ) ⊤ ] = 2 η 2 Σ t . \nabla^2\operatorname{MSE}_t(\pi_\theta)=2\mathbb{E}_{x\sim\mu,\,y\sim\pi_t(\cdot\mid x)}\left[\nabla s_t^{\pi_\theta}(x,y)\nabla s_t^{\pi_\theta}(x,y)^\top\right]=\frac{2}{\eta^2}\Sigma_t. ∇ 2 MSE t ( π θ ) = 2 E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ∇ s t π θ ( x , y ) ∇ s t π θ ( x , y ) ⊤ ] = η 2 2 Σ t .
Also, − ∇ 2 log π θ ( y ∣ x ) = Cov y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] -\nabla^2\log\pi_\theta(y\mid x)=\operatorname{Cov}_{y'\sim\pi_\theta(\cdot\mid x)}[\phi(x,y')] − ∇ 2 log π θ ( y ∣ x ) = Cov y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] (Lemma B.16), so ∇ 2 q θ ( x ) = η − 1 Cov y ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ) ] \nabla^2q_\theta(x)=\eta^{-1}\operatorname{Cov}_{y\sim\pi_\theta(\cdot\mid x)}[\phi(x,y)] ∇ 2 q θ ( x ) = η − 1 Cov y ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y )] . Hence
∇ 2 L t K i m i , b ( π θ ) = 2 η 2 Σ t + 2 E x ∼ μ [ ∇ q θ ( x ) ∇ q θ ( x ) ⊤ + q θ ( x ) η Cov y ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ) ] ] \nabla^2 L_t^{\mathrm{Kimi},b}(\pi_\theta)=\frac{2}{\eta^2}\Sigma_t+2\mathbb{E}_{x\sim\mu}\left[\nabla q_\theta(x)\nabla q_\theta(x)^\top+\frac{q_\theta(x)}{\eta}\operatorname{Cov}_{y\sim\pi_\theta(\cdot\mid x)}[\phi(x,y)]\right] ∇ 2 L t Kimi , b ( π θ ) = η 2 2 Σ t + 2 E x ∼ μ [ ∇ q θ ( x ) ∇ q θ ( x ) ⊤ + η q θ ( x ) Cov y ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y )] ]
⪰ 2 η 2 Σ t − 2 a η E x ∼ μ [ Cov y ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ) ] ] . (194) \succeq\frac{2}{\eta^2}\Sigma_t-\frac{2a}{\eta}\mathbb{E}_{x\sim\mu}\left[\operatorname{Cov}_{y\sim\pi_\theta(\cdot\mid x)}[\phi(x,y)]\right]. \tag*{(194)} ⪰ η 2 2 Σ t − η 2 a E x ∼ μ [ Cov y ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y )] ] . (194)
The inequality uses q θ ( x ) ≥ r ( x , π t ) − b t ( x ) ≥ − a q_\theta(x)\ge r(x,\pi_t)-b_t(x)\ge-a q θ ( x ) ≥ r ( x , π t ) − b t ( x ) ≥ − a and ∇ q θ ( x ) ∇ q θ ( x ) ⊤ ⪰ 0 \nabla q_\theta(x)\nabla q_\theta(x)^\top\succeq0 ∇ q θ ( x ) ∇ q θ ( x ) ⊤ ⪰ 0 .
We now bound the second term in Eq. (194) using π θ ( y ∣ x ) ≤ e D ϕ R π t ( y ∣ x ) \pi_\theta(y\mid x)\le e^{D_\phi R}\pi_t(y\mid x) π θ ( y ∣ x ) ≤ e D ϕ R π t ( y ∣ x ) (Lemma B.12). For every h ∈ R d h\in\mathbb{R}^d h ∈ R d ,
V y ∼ π θ ( ⋅ ∣ x ) [ h ⊤ ϕ ( x , y ) ] ≤ E y ∼ π θ ( ⋅ ∣ x ) [ ( h ⊤ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ h ⊤ ϕ ( x , y ′ ) ] ) 2 ] ≤ e D ϕ R V y ∼ π t ( ⋅ ∣ x ) [ h ⊤ ϕ ( x , y ) ] . \begin{aligned}
\operatorname{V}_{y\sim\pi_\theta(\cdot\mid x)}[h^\top\phi(x,y)]\le\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\left[\left(h^\top\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[h^\top\phi(x,y')]\right)^2\right] \\
&\le e^{D_\phi R}\operatorname{V}_{y\sim\pi_t(\cdot\mid x)}[h^\top\phi(x,y)].
\end{aligned} V y ∼ π θ ( ⋅ ∣ x ) [ h ⊤ ϕ ( x , y )] ≤ E y ∼ π θ ( ⋅ ∣ x ) [ ( h ⊤ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ h ⊤ ϕ ( x , y ′ )] ) 2 ] ≤ e D ϕ R V y ∼ π t ( ⋅ ∣ x ) [ h ⊤ ϕ ( x , y )] .
The first inequality uses V ( Z ) ≤ E [ ( Z − c ) 2 ] \operatorname{V}(Z)\le\mathbb{E}[(Z-c)^2] V ( Z ) ≤ E [( Z − c ) 2 ] for any constant c c c . Thus E x ∼ μ [ Cov y ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ) ] ] ⪯ e D ϕ R Σ t \mathbb{E}_{x\sim\mu}[\operatorname{Cov}_{y\sim\pi_\theta(\cdot\mid x)}[\phi(x,y)]]\preceq e^{D_\phi R}\Sigma_t E x ∼ μ [ Cov y ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y )]] ⪯ e D ϕ R Σ t . Substituting into Eq. (194) gives
∇ 2 L t K i m i , b ( π θ ) ⪰ 2 η 2 Σ t − 2 a e D ϕ R η Σ t = 2 η 2 ( 1 − η a e D ϕ R ) Σ t . \nabla^2 L_t^{\mathrm{Kimi},b}(\pi_\theta)\succeq\frac{2}{\eta^2}\Sigma_t-\frac{2ae^{D_\phi R}}{\eta}\Sigma_t=\frac{2}{\eta^2}\left(1-\eta ae^{D_\phi R}\right)\Sigma_t. ∇ 2 L t Kimi , b ( π θ ) ⪰ η 2 2 Σ t − η 2 a e D ϕ R Σ t = η 2 2 ( 1 − η a e D ϕ R ) Σ t .
If b t ( x ) ≤ r ( x , π t ) b_t(x)\le r(x,\pi_t) b t ( x ) ≤ r ( x , π t ) for every x x x , take a = 0 a=0 a = 0 . The bound is then independent of R R R , so it holds for every θ \theta θ , and Σ t ⪰ 0 \Sigma_t\succeq0 Σ t ⪰ 0 gives global convexity. Substituting b ~ t G \widetilde b_t^G b t G for b t b_t b t proves the statements for L ~ t K i m i , b \widetilde L_t^{\mathrm{Kimi},b} L t Kimi , b .
Lemma I.5. Suppose that π θ \pi_\theta π θ has the softmax linear parameterization in Assumption 2.1. Fix η , R > 0 \eta,R>0 η , R > 0 and A ≥ 0 A\ge0 A ≥ 0 such that ∣ b t ( x ) − r ( x , y ) ∣ ≤ A \lvert b_t(x)-r(x,y)\rvert\le A ∣ b t ( x ) − r ( x , y )∣ ≤ A for all x , y x,y x , y . Every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \lVert\theta-\theta_t\rVert_2\le R ∥ θ − θ t ∥ 2 ≤ R satisfies
∥ ∇ L t K i m i , b ( π θ ) ∥ 2 ≤ 2 D ϕ η 2 ( D ϕ R + η A ) , (195) \left\lVert\nabla L_t^{\mathrm{Kimi},b}(\pi_\theta)\right\rVert_2\le\frac{2D_\phi}{\eta^2}(D_\phi R+\eta A), \tag*{(195)} ∇ L t Kimi , b ( π θ ) 2 ≤ η 2 2 D ϕ ( D ϕ R + η A ) , (195) ∥ ∇ 2 L t K i m i , b ( π θ ) ∥ o p ≤ D ϕ 2 η 2 max { 2 , 1 + D ϕ R + η A 2 } . (196) \left\lVert\nabla^2L_t^{\mathrm{Kimi},b}(\pi_\theta)\right\rVert_{\mathrm{op}}\le\frac{D_\phi^2}{\eta^2}\max\left\{2,1+\frac{D_\phi R+\eta A}{2}\right\}. \tag*{(196)} ∇ 2 L t Kimi , b ( π θ ) op ≤ η 2 D ϕ 2 max { 2 , 1 + 2 D ϕ R + η A } . (196) The same statement holds for L ^ t K i m i , b \widehat L_t^{\mathrm{Kimi},b} L t Kimi , b with b t b_t b t replaced by b ^ t G \widehat b_t^G b t G , and for L ~ t K i m i , b \widetilde L_t^{\mathrm{Kimi},b} L t Kimi , b with b t b_t b t replaced by b ~ t G \widetilde b_t^G b t G .
Proof of Lemma I.5. Fix θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \lVert\theta-\theta_t\rVert_2\le R ∥ θ − θ t ∥ 2 ≤ R and x , y x,y x , y , and write
q θ ( x , y ) : = 1 η log π θ ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + b t ( x ) . q_\theta(x,y):=\frac{1}{\eta}\log\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+b_t(x). q θ ( x , y ) := η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) − r ( x , y ) + b t ( x ) .
We use ∣ log ( π θ ( y ∣ x ) / π t ( y ∣ x ) ) ∣ ≤ D ϕ R |\log(\pi_\theta(y\mid x)/\pi_t(y\mid x))|\le D_\phi R ∣ log ( π θ ( y ∣ x ) / π t ( y ∣ x )) ∣ ≤ D ϕ R (Lemma B.12 ) and ∥ ∇ log π θ ( y ∣ x ) ∥ 2 ≤ D ϕ \|\nabla\log\pi_\theta(y\mid x)\|_2\le D_\phi ∥∇ log π θ ( y ∣ x ) ∥ 2 ≤ D ϕ (Lemma B.15 ) to obtain
∣ q θ ( x , y ) ∣ ≤ 1 η ∣ log π θ ( y ∣ x ) π t ( y ∣ x ) ∣ + ∣ b t ( x ) − r ( x , y ) ∣ ≤ A + D ϕ R η , (197) |q_\theta(x,y)|\le\frac{1}{\eta}\left|\log\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}\right|+|b_t(x)-r(x,y)|\le A+\frac{D_\phi R}{\eta}, \tag*{(197)} ∣ q θ ( x , y ) ∣ ≤ η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) + ∣ b t ( x ) − r ( x , y ) ∣ ≤ A + η D ϕ R , (197)
∥ ∇ q θ ( x , y ) ∥ 2 = 1 η ∥ ∇ log π θ ( y ∣ x ) ∥ 2 ≤ D ϕ η . (198) \|\nabla q_\theta(x,y)\|_2=\frac{1}{\eta}\|\nabla\log\pi_\theta(y\mid x)\|_2\le\frac{D_\phi}{\eta}. \tag*{(198)} ∥∇ q θ ( x , y ) ∥ 2 = η 1 ∥∇ log π θ ( y ∣ x ) ∥ 2 ≤ η D ϕ . (198)
Consequently,
∥ ∇ ( q θ ( x , y ) 2 ) ∥ 2 = 2 ∣ q θ ( x , y ) ∣ ∥ ∇ q θ ( x , y ) ∥ 2 ≤ 2 D ϕ η 2 ( D ϕ R + η A ) . \|\nabla(q_\theta(x,y)^2)\|_2=2|q_\theta(x,y)|\|\nabla q_\theta(x,y)\|_2\le\frac{2D_\phi}{\eta^2}(D_\phi R+\eta A). ∥∇ ( q θ ( x , y ) 2 ) ∥ 2 = 2∣ q θ ( x , y ) ∣∥∇ q θ ( x , y ) ∥ 2 ≤ η 2 2 D ϕ ( D ϕ R + η A ) .
To bound the Hessian, fix ∥ h ∥ 2 = 1 \|h\|_2=1 ∥ h ∥ 2 = 1 . Using − ∇ 2 log π θ ( y ∣ x ) = Cov y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] -\nabla^2\log\pi_\theta(y\mid x)=\operatorname{Cov}_{y'\sim\pi_\theta(\cdot\mid x)}[\phi(x,y')] − ∇ 2 log π θ ( y ∣ x ) = Cov y ′ ∼ π θ ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] (Lemma B.16 ), differentiation gives
h T ∇ 2 ( q θ ( x , y ) 2 ) h = 2 η 2 [ ( ∇ log π θ ( y ∣ x ) , h ) 2 − η q θ ( x , y ) V y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] ] . (199) h^\mathsf{T}\nabla^2(q_\theta(x,y)^2)h=\frac{2}{\eta^2}\left[(\nabla\log\pi_\theta(y\mid x),h)^2-\eta q_\theta(x,y)\mathbb{V}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')]\right]. \tag*{(199)} h T ∇ 2 ( q θ ( x , y ) 2 ) h = η 2 2 [ ( ∇ log π θ ( y ∣ x ) , h ) 2 − η q θ ( x , y ) V y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] ] . (199)
To control these two terms together, let c c c be the midpoint of the range of h T ϕ ( x , ⋅ ) h^\mathsf{T}\phi(x,\cdot) h T ϕ ( x , ⋅ ) , so
c ≔ 1 2 ( inf y ′ h T ϕ ( x , y ′ ) + sup y ′ h T ϕ ( x , y ′ ) ) , ∣ h T ϕ ( x , y ′ ) − c ∣ ≤ D ϕ 2 , y ′ ∈ Y . c\coloneqq\frac{1}{2}\left(\inf_{y'}h^\mathsf{T}\phi(x,y')+\sup_{y'}h^\mathsf{T}\phi(x,y')\right),\qquad|h^\mathsf{T}\phi(x,y')-c|\le\frac{D_\phi}{2},\quad y'\in\mathcal{Y}. c : = 2 1 ( y ′ inf h T ϕ ( x , y ′ ) + y ′ sup h T ϕ ( x , y ′ ) ) , ∣ h T ϕ ( x , y ′ ) − c ∣ ≤ 2 D ϕ , y ′ ∈ Y .
It follows that
V y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] = E y ′ ∼ π θ ( ⋅ ∣ x ) [ ( h T ϕ ( x , y ′ ) − c ) 2 ] − ( E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] − c ) 2 ≤ D ϕ 2 4 − ( E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] − c ) 2 ≤ D ϕ 2 4 . (200) \begin{aligned}
\mathbb{V}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')]=\mathbb{E}_{y'\sim\pi_\theta(\cdot\mid x)}[(h^\mathsf{T}\phi(x,y')-c)^2]-(\mathbb{E}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')]-c)^2 \\
&\le\frac{D_\phi^2}{4}-(\mathbb{E}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')]-c)^2\le\frac{D_\phi^2}{4}. \tag*{(200)}
\end{aligned} V y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] = E y ′ ∼ π θ ( ⋅ ∣ x ) [( h T ϕ ( x , y ′ ) − c ) 2 ] − ( E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] − c ) 2 ≤ 4 D ϕ 2 − ( E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] − c ) 2 ≤ 4 D ϕ 2 . (200)
Since ⟨ ∇ log π θ ( y ∣ x ) , h ⟩ = h T ϕ ( x , y ) − E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] \langle\nabla\log\pi_\theta(y\mid x),h\rangle=h^\mathsf{T}\phi(x,y)-\mathbb{E}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')] ⟨ ∇ log π θ ( y ∣ x ) , h ⟩ = h T ϕ ( x , y ) − E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] (Lemma B.14 ), we also obtain
⟨ ∇ log π θ ( y ∣ x ) , h ⟩ 2 + 2 V y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] ≤ ( D ϕ 2 + ∣ E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] − c ∣ ) 2 + D ϕ 2 2 − 2 ( E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] − c ) 2 = D ϕ 2 − ( D ϕ 2 − ∣ E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] − c ∣ ) 2 ≤ D ϕ 2 . (201) \begin{aligned}
\langle\nabla\log\pi_\theta(y\mid x),h\rangle^2+2\mathbb{V}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')] \\
&\le\left(\frac{D_\phi}{2}+|\mathbb{E}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')]-c|\right)^2+\frac{D_\phi^2}{2}-2(\mathbb{E}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')]-c)^2 \\
&=D_\phi^2-\left(\frac{D_\phi}{2}-|\mathbb{E}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')]-c|\right)^2\le D_\phi^2. \tag*{(201)}
\end{aligned} ⟨ ∇ log π θ ( y ∣ x ) , h ⟩ 2 + 2 V y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] ≤ ( 2 D ϕ + ∣ E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] − c ∣ ) 2 + 2 D ϕ 2 − 2 ( E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] − c ) 2 = D ϕ 2 − ( 2 D ϕ − ∣ E y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] − c ∣ ) 2 ≤ D ϕ 2 . (201)
Returning to Eq. (199) , we obtain
∣ h T ∇ 2 ( q θ ( x , y ) 2 ) h ∣ ≤ 2 η 2 ( ⟨ ∇ log π θ ( y ∣ x ) , h ⟩ 2 + η ∣ q θ ( x , y ) ∣ V y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] ) ≤ 2 η 2 ( D ϕ 2 + ( η ∣ q θ ( x , y ) ∣ − 2 ) V y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ ) ] ) ≤ D ϕ 2 η 2 max { 2 , 1 + D ϕ R + η A 2 } . \begin{aligned}
|h^\mathsf{T}\nabla^2(q_\theta(x,y)^2)h|\le\frac{2}{\eta^2}\left(\langle\nabla\log\pi_\theta(y\mid x),h\rangle^2+\eta|q_\theta(x,y)|\mathbb{V}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')]\right) \\
&\le\frac{2}{\eta^2}\left(D_\phi^2+(\eta|q_\theta(x,y)|-2)\mathbb{V}_{y'\sim\pi_\theta(\cdot\mid x)}[h^\mathsf{T}\phi(x,y')]\right) \\
&\le\frac{D_\phi^2}{\eta^2}\max\left\{2,1+\frac{D_\phi R+\eta A}{2}\right\}.
\end{aligned} ∣ h T ∇ 2 ( q θ ( x , y ) 2 ) h ∣ ≤ η 2 2 ( ⟨ ∇ log π θ ( y ∣ x ) , h ⟩ 2 + η ∣ q θ ( x , y ) ∣ V y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] ) ≤ η 2 2 ( D ϕ 2 + ( η ∣ q θ ( x , y ) ∣ − 2 ) V y ′ ∼ π θ ( ⋅ ∣ x ) [ h T ϕ ( x , y ′ )] ) ≤ η 2 D ϕ 2 max { 2 , 1 + 2 D ϕ R + η A } .
The second inequality uses Eq. (201) . The last uses the variance bound Eq. (200) and ∣ q θ ( x , y ) ∣ ≤ A + D ϕ R / η |q_\theta(x,y)|\le A+D_\phi R/\eta ∣ q θ ( x , y ) ∣ ≤ A + D ϕ R / η from Eq. (197) . Taking the supremum over ∥ h ∥ 2 = 1 \|h\|_2=1 ∥ h ∥ 2 = 1 gives
∥ ∇ 2 ( q θ ( x , y ) 2 ) ∥ o p = sup ∥ h ∥ 2 = 1 ∣ h T ∇ 2 ( q θ ( x , y ) 2 ) h ∣ ≤ D ϕ 2 η 2 max { 2 , 1 + D ϕ R + η A 2 } . \|\nabla^2(q_\theta(x,y)^2)\|_{\mathrm{op}}=\sup_{\|h\|_2=1}|h^\mathsf{T}\nabla^2(q_\theta(x,y)^2)h|\le\frac{D_\phi^2}{\eta^2}\max\left\{2,1+\frac{D_\phi R+\eta A}{2}\right\}. ∥ ∇ 2 ( q θ ( x , y ) 2 ) ∥ op = ∥ h ∥ 2 = 1 sup ∣ h T ∇ 2 ( q θ ( x , y ) 2 ) h ∣ ≤ η 2 D ϕ 2 max { 2 , 1 + 2 D ϕ R + η A } .
The definition of L t K i m i , b L_t^{\mathrm{Kimi},b} L t Kimi , b (Eq. (184) ) now gives
∥ ∇ L t K i m i , b ( π θ ) ∥ 2 ≤ E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ∥ ∇ ( q θ ( x , y ) 2 ) ∥ 2 ] ≤ 2 D ϕ η 2 ( D ϕ R + η A ) , \|\nabla L_t^{\mathrm{Kimi},b}(\pi_\theta)\|_2\le\mathbb{E}_{x\sim\mu,y\sim\pi_t(\cdot\mid x)}[\|\nabla(q_\theta(x,y)^2)\|_2]\le\frac{2D_\phi}{\eta^2}(D_\phi R+\eta A), ∥∇ L t Kimi , b ( π θ ) ∥ 2 ≤ E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ∥∇ ( q θ ( x , y ) 2 ) ∥ 2 ] ≤ η 2 2 D ϕ ( D ϕ R + η A ) ,
∥ ∇ 2 L t K i m i , b ( π θ ) ∥ o p ≤ E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ∥ ∇ 2 ( q θ ( x , y ) 2 ) ∥ o p ] ≤ D ϕ 2 η 2 max { 2 , 1 + D ϕ R + η A 2 } . \|\nabla^2L_t^{\mathrm{Kimi},b}(\pi_\theta)\|_{\mathrm{op}}\le\mathbb{E}_{x\sim\mu,y\sim\pi_t(\cdot\mid x)}[\|\nabla^2(q_\theta(x,y)^2)\|_{\mathrm{op}}]\le\frac{D_\phi^2}{\eta^2}\max\left\{2,1+\frac{D_\phi R+\eta A}{2}\right\}. ∥ ∇ 2 L t Kimi , b ( π θ ) ∥ op ≤ E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ∥ ∇ 2 ( q θ ( x , y ) 2 ) ∥ op ] ≤ η 2 D ϕ 2 max { 2 , 1 + 2 D ϕ R + η A } .
This proves Eqs. (195) and (196) . For L ^ t K i m i , b \widehat{L}_t^{\mathrm{Kimi},b} L t Kimi , b (Eq. (185) ), replace b t b_t b t by b ^ t G \widehat{b}_t^G b t G in q θ q_\theta q θ . The same bounds hold at each sampled ( x i , y i ) (x_i,y_i) ( x i , y i ) , so
∥ ∇ L ^ t K i m i , b ( π θ ) ∥ 2 ≤ 1 N ∑ i = 1 N ∥ ∇ ( q θ ( x i , y i ) 2 ) ∥ 2 ≤ 2 D ϕ η 2 ( D ϕ R + η A ) . \left\|\nabla\widehat{L}_t^{\mathrm{Kimi},b}(\pi_\theta)\right\|_2 \le\frac{1}{N}\sum_{i=1}^{N}\left\|\nabla(q_\theta(x_i,y_i)^2)\right\|_2 \le\frac{2D_\phi}{\eta^2}\left(D_\phi R+\eta A\right). ∇ L t Kimi , b ( π θ ) 2 ≤ N 1 i = 1 ∑ N ∇ ( q θ ( x i , y i ) 2 ) 2 ≤ η 2 2 D ϕ ( D ϕ R + η A ) .
∥ ∇ 2 L ^ t K i m i , b ( π θ ) ∥ o p ≤ 1 N ∑ i = 1 N ∥ ∇ 2 ( q θ ( x i , y i ) 2 ) ∥ o p ≤ D ϕ 2 η 2 max { 2 , 1 + D ϕ R + η A 2 } . \left\|\nabla^2\widehat{L}_t^{\mathrm{Kimi},b}(\pi_\theta)\right\|_{\mathrm{op}} \le\frac{1}{N}\sum_{i=1}^{N}\left\|\nabla^2(q_\theta(x_i,y_i)^2)\right\|_{\mathrm{op}} \le\frac{D_\phi^2}{\eta^2}\max\left\{2,1+\frac{D_\phi R+\eta A}{2}\right\}. ∇ 2 L t Kimi , b ( π θ ) op ≤ N 1 i = 1 ∑ N ∇ 2 ( q θ ( x i , y i ) 2 ) op ≤ η 2 D ϕ 2 max { 2 , 1 + 2 D ϕ R + η A } .
The bounds for L ~ t K i m i , b \widetilde{L}_t^{\mathrm{Kimi},b} L t Kimi , b (Eq. (186) ) follow by substituting b t = b ~ t G b_t=\widetilde{b}_t^G b t = b t G in the population calculation. If ∣ b ^ t G ( x ) − r ( x , y ) ∣ ≤ A \left|\widehat{b}_t^G(x)-r(x,y)\right|\le A b t G ( x ) − r ( x , y ) ≤ A , then
∣ b ~ t G ( x ) − r ( x , y ) ∣ ≤ E y 1 ′ , … , y G ′ ∼ π t ( ⋅ ∣ x ) [ ∣ b ^ t G ( x ) − r ( x , y ) ∣ ] ≤ A . \left|\widetilde{b}_t^G(x)-r(x,y)\right|\le\mathbb{E}_{y'_1,\ldots,y'_G\sim\pi_t(\cdot|x)}\left[\left|\widehat{b}_t^G(x)-r(x,y)\right|\right]\le A. b t G ( x ) − r ( x , y ) ≤ E y 1 ′ , … , y G ′ ∼ π t ( ⋅ ∣ x ) [ b t G ( x ) − r ( x , y ) ] ≤ A .
□
Lemma I.6. Suppose that Assumption 2.1 holds. Let A ≥ 0 A\ge0 A ≥ 0 satisfy ∣ b ^ t G ( x ) − r ( x , y ) ∣ ≤ A \left|\widehat{b}_t^G(x)-r(x,y)\right|\le A b t G ( x ) − r ( x , y ) ≤ A for all x , y x,y x , y . Fix η , R > 0 \eta,R>0 η , R > 0 , an integer N ≥ 1 N\ge1 N ≥ 1 , and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with R ≥ η B ⋆ R\ge\eta B_\star R ≥ η B ⋆ . Define
ζ N ( δ ; R , A ) : = 160 e D ϕ R N ( A + D ϕ R η ) 2 ( d log ( 1 + 2 N ) + log 4 δ ) . (202) \zeta_N(\delta;R,A):=\frac{160e^{D_\phi R}}{N}\left(A+\frac{D_\phi R}{\eta}\right)^2\left(d\log(1+2N)+\log\frac{4}{\delta}\right). \tag*{(202)} ζ N ( δ ; R , A ) := N 160 e D ϕ R ( A + η D ϕ R ) 2 ( d log ( 1 + 2 N ) + log δ 4 ) . (202) With probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \|\theta-\theta_t\|_2\le R ∥ θ − θ t ∥ 2 ≤ R ,
∣ ⟨ ∇ L ^ t K i m i , b ( π θ ) − ∇ L ~ t K i m i , b ( π θ ) , θ − θ t − η θ ⋆ ⟩ ∣ ≤ 1 64 MSE t ( π θ ) + ζ N ( δ ; R , A ) , (203) \left|\left\langle\nabla\widehat{L}_t^{\mathrm{Kimi},b}(\pi_\theta)-\nabla\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_\theta),\theta-\theta_t-\eta\theta_\star\right\rangle\right|\le\frac{1}{64}\operatorname{MSE}_t(\pi_\theta)+\zeta_N(\delta;R,A), \tag*{(203)} ⟨ ∇ L t Kimi , b ( π θ ) − ∇ L t Kimi , b ( π θ ) , θ − θ t − η θ ⋆ ⟩ ≤ 64 1 MSE t ( π θ ) + ζ N ( δ ; R , A ) , (203) ∣ L ^ t K i m i , b ( π θ ) − L ^ t K i m i , b ( π η , t ⋆ ) − L ~ t K i m i , b ( π θ ) + L ~ t K i m i , b ( π η , t ⋆ ) ∣ ≤ 1 64 MSE t ( π θ ) + ζ N ( δ ; R , A ) . (204) \left|\widehat{L}_t^{\mathrm{Kimi},b}(\pi_\theta)-\widehat{L}_t^{\mathrm{Kimi},b}(\pi_{\eta,t}^\star)-\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_\theta)+\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_{\eta,t}^\star)\right|\le\frac{1}{64}\operatorname{MSE}_t(\pi_\theta)+\zeta_N(\delta;R,A). \tag*{(204)} L t Kimi , b ( π θ ) − L t Kimi , b ( π η , t ⋆ ) − L t Kimi , b ( π θ ) + L t Kimi , b ( π η , t ⋆ ) ≤ 64 1 MSE t ( π θ ) + ζ N ( δ ; R , A ) . (204)
Proof of Lemma I.6. We first apply Bernstein’s inequality at a fixed parameter, then extend the bounds uniformly over ∥ θ − θ t ∥ 2 ≤ R \|\theta-\theta_t\|_2\le R ∥ θ − θ t ∥ 2 ≤ R . For each sample, write
q θ , i : = 1 η log π θ ( y i ∣ x i ) π t ( y i ∣ x i ) − r ( x i , y i ) + b ^ t G ( x i ) . q_{\theta,i}:=\frac{1}{\eta}\log\frac{\pi_\theta(y_i\mid x_i)}{\pi_t(y_i\mid x_i)}-r(x_i,y_i)+\widehat{b}_t^G(x_i). q θ , i := η 1 log π t ( y i ∣ x i ) π θ ( y i ∣ x i ) − r ( x i , y i ) + b t G ( x i ) .
The bound ∣ q θ , i ∣ ≤ A + D ϕ R / η |q_{\theta,i}|\le A+D_\phi R/\eta ∣ q θ , i ∣ ≤ A + D ϕ R / η in Eq. (197) applies with b t = b ^ t G b_t=\widehat{b}_t^G b t = b t G . Write S = A + D ϕ R / η S=A+D_\phi R/\eta S = A + D ϕ R / η , so ∣ q θ , i ∣ ≤ S |q_{\theta,i}|\le S ∣ q θ , i ∣ ≤ S whenever ∥ θ − θ t ∥ 2 ≤ R \|\theta-\theta_t\|_2\le R ∥ θ − θ t ∥ 2 ≤ R . This also holds at θ t + η θ ⋆ \theta_t+\eta\theta_\star θ t + η θ ⋆ , since R ≥ η B ⋆ R\ge\eta B_\star R ≥ η B ⋆ , and π θ t + η θ ⋆ = π η , t ⋆ \pi_{\theta_t+\eta\theta_\star}=\pi_{\eta,t}^\star π θ t + η θ ⋆ = π η , t ⋆ by Lemma B.11.
Fix θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \|\theta-\theta_t\|_2\le R ∥ θ − θ t ∥ 2 ≤ R and write h = θ − θ t − η θ ⋆ h=\theta-\theta_t-\eta\theta_\star h = θ − θ t − η θ ⋆ . We will bound the second moments of the sample derivatives and loss differences in terms of MSE t ( π θ ) \operatorname{MSE}_t(\pi_\theta) MSE t ( π θ ) . By reward linearity and the definitions of s t π θ s_t^{\pi_\theta} s t π θ (Eq. (189) ) and Σ t \Sigma_t Σ t (Eq. (190) ),
s t π θ ( x , y ) = 1 η ⟨ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] , h ⟩ , MSE t ( π θ ) = 1 η 2 h ⊤ Σ t h . (205) s_t^{\pi_\theta}(x,y)=\frac{1}{\eta}\left\langle\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot|x)}[\phi(x,y')],h\right\rangle,\qquad\operatorname{MSE}_t(\pi_\theta)=\frac{1}{\eta^2}h^\top\Sigma_t h. \tag*{(205)} s t π θ ( x , y ) = η 1 ⟨ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] , h ⟩ , MSE t ( π θ ) = η 2 1 h ⊤ Σ t h . (205)
For any v v v with ∥ v − θ t ∥ 2 ≤ R \|v-\theta_t\|_2\le R ∥ v − θ t ∥ 2 ≤ R , the identity ∇ q v , i = η − 1 ( ϕ ( x i , y i ) − E y ∼ π v ( ⋅ ∣ x i ) [ ϕ ( x i , y ) ] ) \nabla q_{v,i}=\eta^{-1}(\phi(x_i,y_i)-\mathbb{E}_{y\sim\pi_v(\cdot|x_i)}[\phi(x_i,y)]) ∇ q v , i = η − 1 ( ϕ ( x i , y i ) − E y ∼ π v ( ⋅ ∣ x i ) [ ϕ ( x i , y )]) (Lemma B.14 ) gives
⟨ ∇ q v , i , h ⟩ = s t π θ ( x i , y i ) − E y ∼ π v ( ⋅ ∣ x i ) [ s t π θ ( x i , y ) ] . \langle\nabla q_{v,i},h\rangle=s_t^{\pi_\theta}(x_i,y_i)-\mathbb{E}_{y\sim\pi_v(\cdot|x_i)}[s_t^{\pi_\theta}(x_i,y)]. ⟨ ∇ q v , i , h ⟩ = s t π θ ( x i , y i ) − E y ∼ π v ( ⋅ ∣ x i ) [ s t π θ ( x i , y )] .
Since E y ∼ π t ( ⋅ ∣ x ) [ s t π θ ( x , y ) ] = 0 \mathbb{E}_{y\sim\pi_t(\cdot|x)}[s_t^{\pi_\theta}(x,y)]=0 E y ∼ π t ( ⋅ ∣ x ) [ s t π θ ( x , y )] = 0 , conditioning on x i = x x_i=x x i = x gives
E y i ∼ π t ( ⋅ ∣ x ) [ ⟨ ∇ q v , i , h ⟩ 2 ] = MSE t ( π θ ; x ) + ( E y ∼ π v ( ⋅ ∣ x ) [ s t π θ ( x , y ) ] ) 2 = MSE t ( π θ ; x ) + ( E y ∼ π t ( ⋅ ∣ x ) [ ( π v ( y ∣ x ) π t ( y ∣ x ) − 1 ) s t π θ ( x , y ) ] ) 2 ≤ [ 1 + E y ∼ π t ( ⋅ ∣ x ) [ ( π v ( y ∣ x ) π t ( y ∣ x ) − 1 ) 2 ] ] MSE t ( π θ ; x ) = E y ∼ π t ( ⋅ ∣ x ) [ ( π v ( y ∣ x ) π t ( y ∣ x ) ) 2 ] MSE t ( π θ ; x ) ≤ e D ϕ R MSE t ( π θ ; x ) . (206) \begin{aligned}
\mathbb{E}_{y_i\sim\pi_t(\cdot|x)}\left[\langle\nabla q_{v,i},h\rangle^2\right]
&=\operatorname{MSE}_t(\pi_\theta;x)+\left(\mathbb{E}_{y\sim\pi_v(\cdot|x)}[s_t^{\pi_\theta}(x,y)]\right)^2\\
&=\operatorname{MSE}_t(\pi_\theta;x)+\left(\mathbb{E}_{y\sim\pi_t(\cdot|x)}\left[\left(\frac{\pi_v(y\mid x)}{\pi_t(y\mid x)}-1\right)s_t^{\pi_\theta}(x,y)\right]\right)^2\\
&\le\left[1+\mathbb{E}_{y\sim\pi_t(\cdot|x)}\left[\left(\frac{\pi_v(y\mid x)}{\pi_t(y\mid x)}-1\right)^2\right]\right]\operatorname{MSE}_t(\pi_\theta;x)\\
&=\mathbb{E}_{y\sim\pi_t(\cdot|x)}\left[\left(\frac{\pi_v(y\mid x)}{\pi_t(y\mid x)}\right)^2\right]\operatorname{MSE}_t(\pi_\theta;x)\le e^{D_\phi R}\operatorname{MSE}_t(\pi_\theta;x). \tag*{(206)}
\end{aligned} E y i ∼ π t ( ⋅ ∣ x ) [ ⟨ ∇ q v , i , h ⟩ 2 ] = MSE t ( π θ ; x ) + ( E y ∼ π v ( ⋅ ∣ x ) [ s t π θ ( x , y )] ) 2 = MSE t ( π θ ; x ) + ( E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π v ( y ∣ x ) − 1 ) s t π θ ( x , y ) ] ) 2 ≤ [ 1 + E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π v ( y ∣ x ) − 1 ) 2 ] ] MSE t ( π θ ; x ) = E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π v ( y ∣ x ) ) 2 ] MSE t ( π θ ; x ) ≤ e D ϕ R MSE t ( π θ ; x ) . (206)
The first inequality is Cauchy–Schwarz. The last line uses π v ( y ∣ x ) / π t ( y ∣ x ) ≤ e D ϕ R \pi_v(y \mid x)/\pi_t(y \mid x) \le e^{D_\phi R} π v ( y ∣ x ) / π t ( y ∣ x ) ≤ e D ϕ R from Lemma B.12 .
We next bound the second moment of q θ , i − q θ t + η θ ⋆ , i q_{\theta,i} - q_{\theta_t+\eta\theta_\star,i} q θ , i − q θ t + η θ ⋆ , i . Let θ u = ( 1 − u ) ( θ t + η θ ⋆ ) + u θ \theta_u = (1-u)(\theta_t+\eta\theta_\star)+u\theta θ u = ( 1 − u ) ( θ t + η θ ⋆ ) + u θ for 0 ≤ u ≤ 1 0 \le u \le1 0 ≤ u ≤ 1 . Then ∥ θ u − θ t ∥ 2 ≤ R \lVert\theta_u-\theta_t\rVert_2 \le R ∥ θ u − θ t ∥ 2 ≤ R , and integrating then applying Jensen’s inequality, we have
E x i ∼ μ , y i ∼ π t ( ⋅ ∣ x i ) [ ( q θ , i − q θ t + η θ ⋆ , i ) 2 ] = E x i ∼ μ , y i ∼ π t ( ⋅ ∣ x i ) [ ( ∫ 0 1 ⟨ ∇ q θ u , i , h ⟩ d u ) 2 ] ≤ ∫ 0 1 E x i ∼ μ , y i ∼ π t ( ⋅ ∣ x i ) [ ⟨ ∇ q θ u , i , h ⟩ 2 ] d u ≤ e D ϕ R MSE t ( π θ ) . (207) \begin{aligned}
\mathbb{E}_{x_i\sim\mu,\,y_i\sim\pi_t(\cdot\mid x_i)}\left[(q_{\theta,i}-q_{\theta_t+\eta\theta_\star,i})^2\right] = \mathbb{E}_{x_i\sim\mu,\,y_i\sim\pi_t(\cdot\mid x_i)}\left[\left(\int_0^1 \langle\nabla q_{\theta_u,i},h\rangle\,du\right)^2\right] \\
&\le\int_0^1 \mathbb{E}_{x_i\sim\mu,\,y_i\sim\pi_t(\cdot\mid x_i)}\left[\langle\nabla q_{\theta_u,i},h\rangle^2\right]\,du \le e^{D_\phi R}\operatorname{MSE}_t(\pi_\theta). \tag*{(207)}
\end{aligned} E x i ∼ μ , y i ∼ π t ( ⋅ ∣ x i ) [ ( q θ , i − q θ t + η θ ⋆ , i ) 2 ] = E x i ∼ μ , y i ∼ π t ( ⋅ ∣ x i ) [ ( ∫ 0 1 ⟨ ∇ q θ u , i , h ⟩ d u ) 2 ] ≤ ∫ 0 1 E x i ∼ μ , y i ∼ π t ( ⋅ ∣ x i ) [ ⟨ ∇ q θ u , i , h ⟩ 2 ] d u ≤ e D ϕ R MSE t ( π θ ) . (207)
where we use Eq. (206) in the last line.
For the gradient bound Eq. (203) and the excess loss bound Eq. (204) , respectively, define
Z i g r a d ( θ ) = 2 q θ , i ⟨ ∇ q θ , i , h ⟩ , Z i l o s s ( θ ) = ( q θ , i + q θ t + η θ ⋆ , i ) ( q θ , i − q θ t + η θ ⋆ , i ) . Z_i^{\mathrm{grad}}(\theta)=2q_{\theta,i}\langle\nabla q_{\theta,i},h\rangle,\qquad Z_i^{\mathrm{loss}}(\theta)=(q_{\theta,i}+q_{\theta_t+\eta\theta_\star,i})(q_{\theta,i}-q_{\theta_t+\eta\theta_\star,i}). Z i grad ( θ ) = 2 q θ , i ⟨ ∇ q θ , i , h ⟩ , Z i loss ( θ ) = ( q θ , i + q θ t + η θ ⋆ , i ) ( q θ , i − q θ t + η θ ⋆ , i ) .
For j ∈ { g r a d , l o s s } j\in\{\mathrm{grad},\mathrm{loss}\} j ∈ { grad , loss } , define
F j ( θ ) : = 1 N ∑ i = 1 N ( Z i j ( θ ) − E x i ∼ μ y i , y i ′ , … , y i ′ ∼ π t ( ⋅ ∣ x i ) [ Z i j ( θ ) ] ) . F^j(\theta):=\frac{1}{N}\sum_{i=1}^{N}\left(Z_i^j(\theta)-\mathbb{E}_{\substack{x_i\sim\mu\\y_i,y_i',\ldots,y_i'\sim\pi_t(\cdot\mid x_i)}}[Z_i^j(\theta)]\right). F j ( θ ) := N 1 i = 1 ∑ N ( Z i j ( θ ) − E x i ∼ μ y i , y i ′ , … , y i ′ ∼ π t ( ⋅ ∣ x i ) [ Z i j ( θ )] ) .
By Lemma I.2,
F g r a d ( θ ) = ⟨ ∇ L ^ t K i m i , b ( π θ ) − ∇ L ~ t K i m i , b ( π θ ) , h ⟩ , F^{\mathrm{grad}}(\theta)=\left\langle\nabla\widehat{L}_t^{\mathrm{Kimi,b}}(\pi_\theta)-\nabla\widetilde{L}_t^{\mathrm{Kimi,b}}(\pi_\theta),h\right\rangle, F grad ( θ ) = ⟨ ∇ L t Kimi , b ( π θ ) − ∇ L t Kimi , b ( π θ ) , h ⟩ ,
F l o s s ( θ ) = L ^ t K i m i , b ( π θ ) − L ^ t K i m i , b ( π η , t ) − L ~ t K i m i , b ( π θ ) + L ~ t K i m i , b ( π η , t ) . (208) F^{\mathrm{loss}}(\theta)=\widehat{L}_t^{\mathrm{Kimi,b}}(\pi_\theta)-\widehat{L}_t^{\mathrm{Kimi,b}}(\pi_{\eta,t})-\widetilde{L}_t^{\mathrm{Kimi,b}}(\pi_\theta)+\widetilde{L}_t^{\mathrm{Kimi,b}}(\pi_{\eta,t}). \tag*{(208)} F loss ( θ ) = L t Kimi , b ( π θ ) − L t Kimi , b ( π η , t ) − L t Kimi , b ( π θ ) + L t Kimi , b ( π η , t ) . (208)
Here E x ∼ μ [ ( σ t G ( x ) ) 2 ] \mathbb{E}_{x\sim\mu}[(\sigma^G_t(x))^2] E x ∼ μ [( σ t G ( x ) ) 2 ] is independent of θ \theta θ , so it vanishes upon differentiation or subtraction. The gradient bound in Lemma I.5 justifies interchanging differentiation and expectation.
Since ∣ q θ , i ∣ ≤ S |q_{\theta,i}|\le S ∣ q θ , i ∣ ≤ S and ∣ q θ , i + q θ t + η θ ⋆ , i ∣ ≤ 2 S |q_{\theta,i}+q_{\theta_t+\eta\theta_\star,i}|\le2S ∣ q θ , i + q θ t + η θ ⋆ , i ∣ ≤ 2 S , we obtain, for each j j j ,
E [ Z i j ( θ ) 2 ] ≤ 4 e D ϕ R S 2 MSE t ( π θ ) , ∣ Z i j ( θ ) ∣ ≤ 4 S 2 . \mathbb{E}[Z_i^j(\theta)^2]\le4e^{D_\phi R}S^2\operatorname{MSE}_t(\pi_\theta),\qquad|Z_i^j(\theta)|\le4S^2. E [ Z i j ( θ ) 2 ] ≤ 4 e D ϕ R S 2 MSE t ( π θ ) , ∣ Z i j ( θ ) ∣ ≤ 4 S 2 .
The second moment bound uses Eq. (206) for Z i g r a d Z_i^{\mathrm{grad}} Z i grad and Eq. (207) for Z i l o s s Z_i^{\mathrm{loss}} Z i loss . For the range bound on Z i g r a d Z_i^{\mathrm{grad}} Z i grad , we also use ∥ h ∥ 2 ≤ 2 R \lVert h\rVert_2\le2R ∥ h ∥ 2 ≤ 2 R and ∥ ∇ q θ , i ∥ 2 ≤ D ϕ / η \lVert\nabla q_{\theta,i}\rVert_2\le D_\phi/\eta ∥ ∇ q θ , i ∥ 2 ≤ D ϕ / η from Eq. (198) . Thus V ( Z i j ( θ ) ) ≤ 4 e D ϕ R S 2 MSE t ( π θ ) \mathbb{V}(Z_i^j(\theta))\le4e^{D_\phi R}S^2\operatorname{MSE}_t(\pi_\theta) V ( Z i j ( θ )) ≤ 4 e D ϕ R S 2 MSE t ( π θ ) and ∣ Z i j ( θ ) − E [ Z i j ( θ ) ] ∣ ≤ 8 S 2 |Z_i^j(\theta)-\mathbb{E}[Z_i^j(\theta)]|\le8S^2 ∣ Z i j ( θ ) − E [ Z i j ( θ )] ∣ ≤ 8 S 2 . Bernstein’s inequality (Vershynin, [107 ] , Section 2.8) gives, for each j j j and u > 0 u>0 u > 0 , with failure probability at most 2 e − u 2e^{-u} 2 e − u ,
∣ F j ( θ ) ∣ ≤ 2 V ( Z i j ( θ ) ) u N + 16 S 2 u 3 N ≤ 8 e D ϕ R S 2 MSE t ( π θ ) u N + 16 S 2 u 3 N ≤ 1 64 MSE t ( π θ ) + 144 e D ϕ R S 2 u N . (209) \begin{aligned}
|F^j(\theta)|\le\sqrt{\frac{2\mathbb{V}(Z_i^j(\theta))u}{N}}+\frac{16S^2u}{3N} \\
&\le\sqrt{\frac{8e^{D_\phi R}S^2\operatorname{MSE}_t(\pi_\theta)u}{N}+\frac{16S^2u}{3N}}\le\frac{1}{64}\operatorname{MSE}_t(\pi_\theta)+\frac{144e^{D_\phi R}S^2u}{N}. \tag*{(209)}
\end{aligned} ∣ F j ( θ ) ∣ ≤ N 2 V ( Z i j ( θ )) u + 3 N 16 S 2 u ≤ N 8 e D ϕ R S 2 MSE t ( π θ ) u + 3 N 16 S 2 u ≤ 64 1 MSE t ( π θ ) + N 144 e D ϕ R S 2 u . (209)
The last inequality is Young’s inequality. Choose a finite set Θ 0 ⊆ { θ : ∥ θ − θ t ∥ 2 ≤ R } \Theta_0\subseteq\{\theta:\lVert\theta-\theta_t\rVert_2\le R\} Θ 0 ⊆ { θ : ∥ θ − θ t ∥ 2 ≤ R } such that every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \lVert\theta-\theta_t\rVert_2\le R ∥ θ − θ t ∥ 2 ≤ R has some θ ~ ∈ Θ 0 \widetilde{\theta}\in\Theta_0 θ ∈ Θ 0 satisfying ∥ θ − θ ~ ∥ 2 ≤ R / N \lVert\theta-\widetilde{\theta}\rVert_2\le R/N ∥ θ − θ ∥ 2 ≤ R / N . Such a set exists with ∣ Θ 0 ∣ ≤ ( 1 + 2 N ) d |\Theta_0|\le(1+2N)^d ∣ Θ 0 ∣ ≤ ( 1 + 2 N ) d (Vershynin, [107 ] , Section 4.2). Set u = d log ( 1 + 2 N ) + log ( 4 / δ ) u=d\log(1+2N)+\log(4/\delta) u = d log ( 1 + 2 N ) + log ( 4/ δ ) . A union bound makes Eq. (209) hold for both F g r a d F^{\mathrm{grad}} F grad and F l o s s F^{\mathrm{loss}} F loss at every θ ~ ∈ Θ 0 \widetilde{\theta}\in\Theta_0 θ ∈ Θ 0 with probability at least 1 − δ 1-\delta 1 − δ . We condition on this event.
We now extend Eq. (209) to every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \lVert\theta-\theta_t\rVert_2\le R ∥ θ − θ t ∥ 2 ≤ R . Differentiating F g r a d F^{\mathrm{grad}} F grad and F l o s s F^{\mathrm{loss}} F loss in Eq. (208) gives
∇ F g r a d ( θ ) = ( ∇ 2 L ^ t K i m i , b ( π θ ) − ∇ 2 L ~ t K i m i , b ( π θ ) ) h + ∇ L ^ t K i m i , b ( π θ ) − ∇ L ~ t K i m i , b ( π θ ) , \nabla F^{\mathrm{grad}}(\theta)=\left(\nabla^2\widehat{L}_t^{\mathrm{Kimi,b}}(\pi_\theta)-\nabla^2\widetilde{L}_t^{\mathrm{Kimi,b}}(\pi_\theta)\right)h+\nabla\widehat{L}_t^{\mathrm{Kimi,b}}(\pi_\theta)-\nabla\widetilde{L}_t^{\mathrm{Kimi,b}}(\pi_\theta), ∇ F grad ( θ ) = ( ∇ 2 L t Kimi , b ( π θ ) − ∇ 2 L t Kimi , b ( π θ ) ) h + ∇ L t Kimi , b ( π θ ) − ∇ L t Kimi , b ( π θ ) ,
∇ F l o s s ( θ ) = ∇ L ^ t K i m i , b ( π θ ) − ∇ L ~ t K i m i , b ( π θ ) . \nabla F^{\mathrm{loss}}(\theta)=\nabla\widehat{L}_t^{\mathrm{Kimi,b}}(\pi_\theta)-\nabla\widetilde{L}_t^{\mathrm{Kimi,b}}(\pi_\theta). ∇ F loss ( θ ) = ∇ L t Kimi , b ( π θ ) − ∇ L t Kimi , b ( π θ ) .
Thus ∥ h ∥ 2 ≤ 2 R \|h\|_2 \le2R ∥ h ∥ 2 ≤ 2 R and the gradient norm bounds in Lemma I.5 give, for each j j j ,
∥ ∇ F j ( θ ) ∥ 2 ≤ 2 R ( ∥ ∇ 2 L ~ t K i m i , b ( π θ ) ∥ o p + ∥ ∇ 2 L ^ t K i m i , b ( π θ ) ∥ o p ) + ∥ ∇ L ^ t K i m i , b ( π θ ) ∥ 2 + ∥ ∇ L ~ t K i m i , b ( π θ ) ∥ 2 ≤ 4 R D ϕ 2 η 2 ( 2 + η S 2 ) + 4 D ϕ S η . (210) \begin{aligned}
\left\|\nabla F^j(\theta)\right\|_2 \le2R\left(\left\|\nabla^2\widetilde{L}^{\mathrm{Kimi,b}}_t(\pi_\theta)\right\|_{\mathrm{op}}+\left\|\nabla^2\widehat{L}^{\mathrm{Kimi,b}}_t(\pi_\theta)\right\|_{\mathrm{op}}\right)+\left\|\nabla\widehat{L}^{\mathrm{Kimi,b}}_t(\pi_\theta)\right\|_2+\left\|\nabla\widetilde{L}^{\mathrm{Kimi,b}}_t(\pi_\theta)\right\|_2 \\
&\le\frac{4RD_\phi^2}{\eta^2}\left(2+\frac{\eta S}{2}\right)+\frac{4D_\phi S}{\eta}. \tag*{(210)}
\end{aligned} ∇ F j ( θ ) 2 ≤ 2 R ( ∇ 2 L t Kimi , b ( π θ ) op + ∇ 2 L t Kimi , b ( π θ ) op ) + ∇ L t Kimi , b ( π θ ) 2 + ∇ L t Kimi , b ( π θ ) 2 ≤ η 2 4 R D ϕ 2 ( 2 + 2 η S ) + η 4 D ϕ S . (210)
Since V y ∼ π t ( ⋅ ∣ x ) [ v ⊤ ϕ ( x , y ) ] ≤ D ϕ 2 / 4 \mathop{\mathrm{V}}_{y\sim\pi_t(\cdot\mid x)}[v^\top\phi(x,y)]\le D_\phi^2/4 V y ∼ π t ( ⋅ ∣ x ) [ v ⊤ ϕ ( x , y )] ≤ D ϕ 2 /4 for every ∥ v ∥ 2 = 1 \|v\|_2=1 ∥ v ∥ 2 = 1 by Eq. (200) , the definition of Σ t \Sigma_t Σ t in Eq. (190) gives Σ t ⪯ D ϕ 2 I / 4 \Sigma_t\preceq D_\phi^2I/4 Σ t ⪯ D ϕ 2 I /4 . Differentiating MSE t ( π θ ) = h ⊤ Σ t h / η 2 \operatorname{MSE}_t(\pi_\theta)=h^\top\Sigma_th/\eta^2 MSE t ( π θ ) = h ⊤ Σ t h / η 2 from Eq. (205) therefore yields
∥ ∇ MSE t ( π θ ) ∥ 2 = 2 η 2 ∥ Σ t ( θ − θ t − η θ ⋆ ) ∥ 2 ≤ D ϕ 2 R η 2 . (211) \|\nabla\operatorname{MSE}_t(\pi_\theta)\|_2=\frac{2}{\eta^2}\|\Sigma_t(\theta-\theta_t-\eta\theta_\star)\|_2\le\frac{D_\phi^2R}{\eta^2}. \tag*{(211)} ∥∇ MSE t ( π θ ) ∥ 2 = η 2 2 ∥ Σ t ( θ − θ t − η θ ⋆ ) ∥ 2 ≤ η 2 D ϕ 2 R . (211)
For any θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \|\theta-\theta_t\|_2\le R ∥ θ − θ t ∥ 2 ≤ R , choose θ ~ ∈ Θ 0 \widetilde{\theta}\in\Theta_0 θ ∈ Θ 0 with ∥ θ − θ ~ ∥ 2 ≤ R / N \|\theta-\widetilde{\theta}\|_2\le R/N ∥ θ − θ ∥ 2 ≤ R / N . The bounds on ∥ ∇ F j ( θ ) ∥ 2 \|\nabla F^j(\theta)\|_2 ∥∇ F j ( θ ) ∥ 2 (Eq. (210) ) and ∥ ∇ MSE t ( π θ ) ∥ 2 \|\nabla\operatorname{MSE}_t(\pi_\theta)\|_2 ∥∇ MSE t ( π θ ) ∥ 2 (Eq. (211) ), together with D ϕ R / η ≤ S D_\phi R/\eta\le S D ϕ R / η ≤ S , give
∣ F j ( θ ) − F j ( θ ~ ) ∣ ≤ R N [ 4 R D ϕ 2 η 2 ( 2 + η S 2 ) + 4 D ϕ S η ] ≤ ( 12 + 2 D ϕ R ) S 2 N , |F^j(\theta)-F^j(\widetilde{\theta})|\le\frac{R}{N}\left[\frac{4RD_\phi^2}{\eta^2}\left(2+\frac{\eta S}{2}\right)+\frac{4D_\phi S}{\eta}\right]\le\frac{(12+2D_\phi R)S^2}{N}, ∣ F j ( θ ) − F j ( θ ) ∣ ≤ N R [ η 2 4 R D ϕ 2 ( 2 + 2 η S ) + η 4 D ϕ S ] ≤ N ( 12 + 2 D ϕ R ) S 2 ,
∣ MSE t ( π θ ) − MSE t ( π θ ~ ) ∣ ≤ D ϕ 2 R 2 η 2 N ≤ S 2 N . |\operatorname{MSE}_t(\pi_\theta)-\operatorname{MSE}_t(\pi_{\widetilde{\theta}})|\le\frac{D_\phi^2R^2}{\eta^2N}\le\frac{S^2}{N}. ∣ MSE t ( π θ ) − MSE t ( π θ ) ∣ ≤ η 2 N D ϕ 2 R 2 ≤ N S 2 .
Combining these bounds with Eq. (209) at θ ~ \widetilde{\theta} θ yields
∣ F j ( θ ) ∣ ≤ ∣ F j ( θ ~ ) ∣ + ( 12 + 2 D ϕ R ) S 2 N ≤ 1 64 MSE t ( π θ ~ ) + 144 e D ϕ R S 2 u N + ( 12 + 2 D ϕ R ) S 2 N ≤ 1 64 MSE t ( π θ ) + 144 e D ϕ R S 2 u N + ( 12 + 2 D ϕ R + 1 / 64 ) S 2 N ≤ 1 64 MSE t ( π θ ) + 160 e D ϕ R S 2 u N . \begin{aligned}
|F^j(\theta)|\le|F^j(\widetilde{\theta})|+\frac{(12+2D_\phi R)S^2}{N} \\
&\le\frac{1}{64}\operatorname{MSE}_t(\pi_{\widetilde{\theta}})+\frac{144e^{D_\phi R}S^2u}{N}+\frac{(12+2D_\phi R)S^2}{N} \\
&\le\frac{1}{64}\operatorname{MSE}_t(\pi_\theta)+\frac{144e^{D_\phi R}S^2u}{N}+\frac{(12+2D_\phi R+1/64)S^2}{N} \\
&\le\frac{1}{64}\operatorname{MSE}_t(\pi_\theta)+\frac{160e^{D_\phi R}S^2u}{N}.
\end{aligned} ∣ F j ( θ ) ∣ ≤ ∣ F j ( θ ) ∣ + N ( 12 + 2 D ϕ R ) S 2 ≤ 64 1 MSE t ( π θ ) + N 144 e D ϕ R S 2 u + N ( 12 + 2 D ϕ R ) S 2 ≤ 64 1 MSE t ( π θ ) + N 144 e D ϕ R S 2 u + N ( 12 + 2 D ϕ R + 1/64 ) S 2 ≤ 64 1 MSE t ( π θ ) + N 160 e D ϕ R S 2 u .
The last inequality uses u ≥ 1 u\ge1 u ≥ 1 and 12 + 2 D ϕ R + 1 / 64 ≤ 13 e D ϕ R 12+2D_\phi R+1/64\le13e^{D_\phi R} 12 + 2 D ϕ R + 1/64 ≤ 13 e D ϕ R . Substituting S = A + D ϕ R / η S=A+D_\phi R/\eta S = A + D ϕ R / η and u = d log ( 1 + 2 N ) + log ( 4 / δ ) u=d\log(1+2N)+\log(4/\delta) u = d log ( 1 + 2 N ) + log ( 4/ δ ) gives Eqs. (203) and (204) simultaneously for every θ \theta θ in the stated range. □
Lemma I.7. Under the assumptions of Lemma I.6, let a ≥ 0 a\ge0 a ≥ 0 satisfy b ~ t G ( x ) − r ( x , π t ) ≤ a \widetilde{b}^{\,G}_t(x)-r(x,\pi_t)\le a b t G ( x ) − r ( x , π t ) ≤ a for every x x x . On the event in that lemma, every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \|\theta-\theta_t\|_2\le R ∥ θ − θ t ∥ 2 ≤ R satisfies
⟨ ∇ L ^ t K i m i , b ( π θ ) , θ − θ t − η θ ⋆ ⟩ ≥ L ^ t K i m i , b ( π θ ) − L ^ t K i m i , b ( π η , t ) + ( 31 32 − η a e D ϕ R ) MSE t ( π θ ) − 2 ζ N ( δ ; R , A ) , (212) \left\langle\nabla\widehat{L}^{\mathrm{Kimi,b}}_t(\pi_\theta),\theta-\theta_t-\eta\theta_\star\right\rangle\ge\widehat{L}^{\mathrm{Kimi,b}}_t(\pi_\theta)-\widehat{L}^{\mathrm{Kimi,b}}_t(\pi_{\eta,t})+\left(\frac{31}{32}-\eta ae^{D_\phi R}\right)\operatorname{MSE}_t(\pi_\theta)-2\zeta_N(\delta;R,A), \tag*{(212)} ⟨ ∇ L t Kimi , b ( π θ ) , θ − θ t − η θ ⋆ ⟩ ≥ L t Kimi , b ( π θ ) − L t Kimi , b ( π η , t ) + ( 32 31 − η a e D ϕ R ) MSE t ( π θ ) − 2 ζ N ( δ ; R , A ) , (212) where ζ N ( δ ; R , A ) \zeta_N(\delta;R,A) ζ N ( δ ; R , A ) is defined in Eq. (202) .
Proof of Lemma I.7 . Condition on the event in Lemma I.6. Fix θ \theta θ with ∥ θ − θ t ∥ 2 ≤ R \|\theta-\theta_t\|_2\le R ∥ θ − θ t ∥ 2 ≤ R and write h = θ − θ t − η θ ⋆ h=\theta-\theta_t-\eta\theta_\star h = θ − θ t − η θ ⋆ . To bound ⟨ ∇ L ~ t K i m i , b ( π θ ) , h ⟩ \langle\nabla\widetilde{L}^{\mathrm{Kimi,b}}_t(\pi_\theta),h\rangle ⟨ ∇ L t Kimi , b ( π θ ) , h ⟩ , we use Lemma I.4, which gives
∇ v 2 L ~ t K i m i , b ( π v ) ⪰ 2 η 2 ( 1 − η a e D ϕ R ) Σ t , ∥ v − θ t ∥ 2 ≤ R . (213) \nabla_v^2\widetilde{L}^{\mathrm{Kimi,b}}_t(\pi_v)\succeq\frac{2}{\eta^2}(1-\eta ae^{D_\phi R})\Sigma_t,\qquad\|v-\theta_t\|_2\le R. \tag*{(213)} ∇ v 2 L t Kimi , b ( π v ) ⪰ η 2 2 ( 1 − η a e D ϕ R ) Σ t , ∥ v − θ t ∥ 2 ≤ R . (213)
We have π η , t ⋆ = π θ t + η θ ⋆ \pi^\star_{\eta,t}=\pi_{\theta_t+\eta\theta_\star} π η , t ⋆ = π θ t + η θ ⋆ (Lemma B.11). For v = ( 1 − u ) θ + u ( θ t + η θ ⋆ ) v=(1-u)\theta+u(\theta_t+\eta\theta_\star) v = ( 1 − u ) θ + u ( θ t + η θ ⋆ ) with 0 ≤ u ≤ 1 0\le u\le1 0 ≤ u ≤ 1 ,
∥ v − θ t ∥ 2 ≤ ( 1 − u ) ∥ θ − θ t ∥ 2 + u η B ⋆ ≤ R . \|v-\theta_t\|_2\le(1-u)\|\theta-\theta_t\|_2+u\eta B_\star\le R. ∥ v − θ t ∥ 2 ≤ ( 1 − u ) ∥ θ − θ t ∥ 2 + u η B ⋆ ≤ R .
Thus Taylor’s theorem at θ \theta θ gives, for some such v v v ,
L ~ t K i m i , b ( π η , t ∗ ) = L ~ t K i m i , b ( π θ ) − ⟨ ∇ L ~ t K i m i , b ( π θ ) , h ⟩ + 1 2 h ⊤ ∇ v 2 L ~ t K i m i , b ( π v ) h ≥ L ~ t K i m i , b ( π θ ) − ⟨ ∇ L ~ t K i m i , b ( π θ ) , h ⟩ + ( 1 − η a e D ϕ R ) MSE t ( π θ ) . (214) \begin{aligned}
\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_{\eta,t}^{*}) = \widetilde{L}_t^{\mathrm{Kimi},b}(\pi_\theta) - \left\langle\nabla\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_\theta), h \right\rangle+ \frac{1}{2}h^\top\nabla_v^2\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_v)h \\
&\ge\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_\theta) - \left\langle\nabla\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_\theta), h \right\rangle+ \left(1-\eta a e^{D_\phi R}\right)\operatorname{MSE}_t(\pi_\theta). \tag*{(214)}
\end{aligned} L t Kimi , b ( π η , t ∗ ) = L t Kimi , b ( π θ ) − ⟨ ∇ L t Kimi , b ( π θ ) , h ⟩ + 2 1 h ⊤ ∇ v 2 L t Kimi , b ( π v ) h ≥ L t Kimi , b ( π θ ) − ⟨ ∇ L t Kimi , b ( π θ ) , h ⟩ + ( 1 − η a e D ϕ R ) MSE t ( π θ ) . (214)
The inequality uses Eq. (213) and h ⊤ Σ t h / η 2 = MSE t ( π θ ) h^\top\Sigma_t h/\eta^2 = \operatorname{MSE}_t(\pi_\theta) h ⊤ Σ t h / η 2 = MSE t ( π θ ) from Eq. (205) . Applying the two concentration bounds in Lemma I.6 and the preceding inequality yields
⟨ ∇ L ~ t K i m i , b ( π θ ) , h ⟩ ≥ ⟨ ∇ L ~ t K i m i , b ( π θ ) , h ⟩ − 1 64 MSE t ( π θ ) − ζ N ( δ ; R , A ) ≥ L ~ t K i m i , b ( π θ ) − L ~ t K i m i , b ( π η , t ∗ ) + ( 63 64 − η a e D ϕ R ) MSE t ( π θ ) − ζ N ( δ ; R , A ) ≥ L ^ t K i m i , b ( π θ ) − L ^ t K i m i , b ( π η , t ∗ ) + ( 31 32 − η a e D ϕ R ) MSE t ( π θ ) − 2 ζ N ( δ ; R , A ) . \begin{aligned}
\left\langle\nabla\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_\theta),h \right\rangle\ge\left\langle\nabla\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_\theta),h \right\rangle- \frac{1}{64}\operatorname{MSE}_t(\pi_\theta)-\zeta_N(\delta;R,A) \\
&\ge\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_\theta)-\widetilde{L}_t^{\mathrm{Kimi},b}(\pi_{\eta,t}^{*})+\left(\frac{63}{64}-\eta a e^{D_\phi R}\right)\operatorname{MSE}_t(\pi_\theta)-\zeta_N(\delta;R,A) \\
&\ge\widehat{L}_t^{\mathrm{Kimi},b}(\pi_\theta)-\widehat{L}_t^{\mathrm{Kimi},b}(\pi_{\eta,t}^{*})+\left(\frac{31}{32}-\eta a e^{D_\phi R}\right)\operatorname{MSE}_t(\pi_\theta)-2\zeta_N(\delta;R,A).
\end{aligned} ⟨ ∇ L t Kimi , b ( π θ ) , h ⟩ ≥ ⟨ ∇ L t Kimi , b ( π θ ) , h ⟩ − 64 1 MSE t ( π θ ) − ζ N ( δ ; R , A ) ≥ L t Kimi , b ( π θ ) − L t Kimi , b ( π η , t ∗ ) + ( 64 63 − η a e D ϕ R ) MSE t ( π θ ) − ζ N ( δ ; R , A ) ≥ L t Kimi , b ( π θ ) − L t Kimi , b ( π η , t ∗ ) + ( 32 31 − η a e D ϕ R ) MSE t ( π θ ) − 2 ζ N ( δ ; R , A ) .
The first inequality uses the concentration result in Eq. (203) of Lemma I.6, the second uses Eq. (214) above, and the third uses the concentration result in Eq. (204) of Lemma I.6.
KimiPO: Global Convergence (Proof of Corollary 4.1)
This section proves the convergence guarantees for KimiPO (Algorithm 3) with the mean baseline, and is organized as follows.
Appendix J.§ 1 states and proves supporting lemmas.
Appendix J.§ 2 establishes convergence of the inner optimization loop for KimiPO (Algorithm 3) in Theorem J.1.
Appendix J.§ 3 combines the inner loop guarantee with the mean-squared error criterion in Theorem 4.1 to prove Corollary J.1.
Notation. Throughout this section, we use the same notation as § 4.2 . Recall that KimiPO uses the baseline function b t ( x ) = r ( x , π t ) b_t(x)=r(x,\pi_t) b t ( x ) = r ( x , π t ) , and at an outer iteration t t t with π t \pi_t π t , the objective is
L t K i m i ( π ) = E π t [ ( 1 η log π ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + r ( x , π t ) ) 2 ] . L_t^{\mathrm{Kimi}}(\pi)=\mathbb{E}_{\pi_t}\left[\left(\frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+r(x,\pi_t)\right)^2\right]. L t Kimi ( π ) = E π t [ ( η 1 log π t ( y ∣ x ) π ( y ∣ x ) − r ( x , y ) + r ( x , π t ) ) 2 ] .
We also follow the notation of KimiPO in Algorithm 3. To compute the empirical objective, for each sampled prompt x i ∼ μ x_i\sim\mu x i ∼ μ recall that KimiPO independently draws one response y i y_i y i to estimate the objective and a group of G G G additional responses y i , 1 ′ , … , y i , G ′ y'_{i,1},\ldots,y'_{i,G} y i , 1 ′ , … , y i , G ′ to estimate the baseline from π t ( ⋅ ∣ x i ) \pi_t(\cdot\mid x_i) π t ( ⋅ ∣ x i ) . The baseline estimate is
r ˉ i = 1 G ∑ j = 1 G r ( x i , y i , j ′ ) , \bar{r}_i=\frac{1}{G}\sum_{j=1}^{G}r(x_i,y'_{i,j}), r ˉ i = G 1 j = 1 ∑ G r ( x i , y i , j ′ ) ,
which is a consistent and unbiased estimator of the baseline function, E [ r ˉ i ∣ x i ] = r ( x i , π t ) \mathbb{E}[\bar{r}_i\mid x_i]=r(x_i,\pi_t) E [ r ˉ i ∣ x i ] = r ( x i , π t ) . The empirical objective is then computed as
L ^ t K i m i ( π ) = 1 N ∑ i = 1 N ( 1 η log π ( y i ∣ x i ) π t ( y i ∣ x i ) − r i + r ˉ i ) 2 . \widehat{L}_t^{\mathrm{Kimi}}(\pi)=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{\eta}\log\frac{\pi(y_i\mid x_i)}{\pi_t(y_i\mid x_i)}-r_i+\bar{r}_i\right)^2. L t Kimi ( π ) = N 1 i = 1 ∑ N ( η 1 log π t ( y i ∣ x i ) π ( y i ∣ x i ) − r i + r ˉ i ) 2 .
Supporting Lemmas
We now state and prove the supporting results for the inner convergence guarantee in Appendix J.§ 2 .
Lemma J.1 decomposes the population objective into mean-squared error and a squared KL divergence term.
Lemma J.2 bounds the difference between the mean baseline r ( x , π t ) r(x,\pi_t) r ( x , π t ) and the normalizer baseline b t ⋆ ( x ) b_t^\star(x) b t ⋆ ( x ) .
Lemma J.3 establishes global convexity by specializing Lemma I.4.
Lemma J.4 bounds the gradient at the exact update, which controls the bias when converting excess loss to mean-squared error.
Lemma J.5 lower-bounds excess loss in terms of mean-squared error.
Notation. Fix an outer iteration t t t and adopt the notation of Appendix § 16.1 .§ 3 . For the mean baseline and every integer G ≥ 1 G \ge1 G ≥ 1 , we have
b ^ t G ( x ) = 1 G ∑ j = 1 G r ( x , y j ′ ) , b ~ t G ( x ) = b t ( x ) = r ( x , π t ) . \widehat{b}_t^G(x) = \frac{1}{G}\sum_{j=1}^{G} r(x,y'_j), \qquad\widetilde{b}_t^G(x) = b_t(x) = r(x,\pi_t). b t G ( x ) = G 1 j = 1 ∑ G r ( x , y j ′ ) , b t G ( x ) = b t ( x ) = r ( x , π t ) .
The sample mean satisfies min y r ( x , y ) ≤ b ^ t G ( x ) ≤ max y r ( x , y ) \min_y r(x,y) \le\widehat{b}_t^G(x) \le\max_y r(x,y) min y r ( x , y ) ≤ b t G ( x ) ≤ max y r ( x , y ) . Recall the empirical objective is L ^ t K i m i \widehat{L}_t^{\mathrm{Kimi}} L t Kimi , and the auxiliary objective defined in Eq. (186) of Appendix § 16.1 .§ 3 is the same as the population objective, L ~ t K i m i , b = L t K i m i \widetilde{L}_t^{\mathrm{Kimi},b} = L_t^{\mathrm{Kimi}} L t Kimi , b = L t Kimi . We use the feature covariance Σ t \Sigma_t Σ t defined in Eq. (190) .
Lemma J.1. Every policy π > 0 \pi> 0 π > 0 satisfies
L t K i m i ( π ) = MSE t ( π ) + E x ∼ μ [ ( 1 η D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) ) 2 ] . L_t^{\mathrm{Kimi}}(\pi) = \operatorname{MSE}_t(\pi) + \mathbb{E}_{x\sim\mu}\left[\left(\frac{1}{\eta}D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\mathbin\Vert\pi(\cdot\mid x))\right)^2\right]. L t Kimi ( π ) = MSE t ( π ) + E x ∼ μ [ ( η 1 D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) ) 2 ] .
Proof of Lemma J.1. Apply Eq. (191) in Lemma I.1 with b ~ t G ( x ) = r ( x , π t ) \widetilde{b}_t^G(x) = r(x,\pi_t) b t G ( x ) = r ( x , π t ) and L ~ t K i m i , b = L t K i m i \widetilde{L}_t^{\mathrm{Kimi},b} = L_t^{\mathrm{Kimi}} L t Kimi , b = L t Kimi . □ \square □
Lemma J.2. Let η > 0 \eta> 0 η > 0 and π t > 0 \pi_t > 0 π t > 0 , and suppose that r ( x , y ) ∈ [ 0 , R max ] r(x,y) \in[0,R_{\max}] r ( x , y ) ∈ [ 0 , R m a x ] . For every prompt x x x , the mean baseline satisfies
0 ≤ b t ⋆ ( x ) − r ( x , π t ) ≤ η R max 2 8 . (215) 0 \le b_t^\star(x) - r(x,\pi_t) \le\frac{\eta R_{\max}^2}{8}. \tag*{(215)} 0 ≤ b t ⋆ ( x ) − r ( x , π t ) ≤ 8 η R m a x 2 . (215) Under Assumption 2.1, the bound also holds with D ⋆ D_\star D ⋆ in place of R max R_{\max} R m a x .
Proof of Lemma J.2 . For every prompt x x x , Jensen’s inequality gives
b t ⋆ ( x ) = 1 η log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] ≥ E y ∼ π t ( ⋅ ∣ x ) [ r ( x , y ) ] = r ( x , π t ) . b_t^\star(x) = \frac{1}{\eta}\log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta r(x,y)}\right] \ge\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[r(x,y)] = r(x,\pi_t). b t ⋆ ( x ) = η 1 log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] ≥ E y ∼ π t ( ⋅ ∣ x ) [ r ( x , y )] = r ( x , π t ) .
Applying Hoeffding’s lemma to the centered reward r ( x , y ) − r ( x , π t ) r(x,y)-r(x,\pi_t) r ( x , y ) − r ( x , π t ) , whose range has length at most R max R_{\max} R m a x , then yields
0 ≤ b t ⋆ ( x ) − r ( x , π t ) = 1 η log E y ∼ π t ( ⋅ ∣ x ) [ e η ( r ( x , y ) − r ( x , π t ) ) ] ≤ η R max 2 8 . 0 \le b_t^\star(x) - r(x,\pi_t) = \frac{1}{\eta}\log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[e^{\eta(r(x,y)-r(x,\pi_t))}\right] \le\frac{\eta R_{\max}^2}{8}. 0 ≤ b t ⋆ ( x ) − r ( x , π t ) = η 1 log E y ∼ π t ( ⋅ ∣ x ) [ e η ( r ( x , y ) − r ( x , π t )) ] ≤ 8 η R m a x 2 .
Under Assumption 2.1, Eq. (188) bounds the reward range by D ⋆ D_\star D ⋆ , so the same argument gives the bound with D ⋆ D_\star D ⋆ in place of R max R_{\max} R m a x . □ \square □
Lemma J.3. Suppose that Assumption 2.1 holds. For any η > 0 \eta> 0 η > 0 , L t K i m i ( π θ ) L_t^{\mathrm{Kimi}}(\pi_\theta) L t Kimi ( π θ ) is globally convex in θ \theta θ , with
∇ 2 L t K i m i ( π θ ) ⪰ 2 η 2 Σ t , ∀ θ ∈ R d . (216) \nabla^2 L_t^{\mathrm{Kimi}}(\pi_\theta) \succeq\frac{2}{\eta^2}\Sigma_t,\qquad\forall\theta\in\mathbb{R}^d. \tag*{(216)} ∇ 2 L t Kimi ( π θ ) ⪰ η 2 2 Σ t , ∀ θ ∈ R d . (216)
Proof of Lemma J.3 . For the mean baseline, b t ( x ) = r ( x , π t ) b_t(x)=r(x,\pi_t) b t ( x ) = r ( x , π t ) and L t K i m i , b = L t K i m i L_t^{\mathrm{Kimi},b}=L_t^{\mathrm{Kimi}} L t Kimi , b = L t Kimi . The global curvature bound in Eq. (193) of Lemma I.4 gives the result.
Lemma J.4. Suppose that Assumption 2.1 holds. For every η > 0 \eta>0 η > 0 and θ ∈ R d \theta\in\mathbb{R}^d θ ∈ R d ,
∣ ⟨ ∇ L t K i m i ( π η , t ⋆ ) , θ − θ t − η θ ⋆ ⟩ ∣ ≤ η 2 D ⋆ 2 e η D ⋆ / 2 8 MSE t ( π θ ) . \left|\left\langle\nabla L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^{\star}),\theta-\theta_t-\eta\theta_{\star}\right\rangle\right|\leq\frac{\eta^2D_{\star}^2e^{\eta D_{\star}/2}}{8}\sqrt{\operatorname{MSE}_t(\pi_{\theta})}. ⟨ ∇ L t Kimi ( π η , t ⋆ ) , θ − θ t − η θ ⋆ ⟩ ≤ 8 η 2 D ⋆ 2 e η D ⋆ /2 MSE t ( π θ ) .
Proof of Lemma J.4 . By Lemma B.11, π η , t ⋆ = π θ t + η θ ⋆ \pi_{\eta,t}^{\star}=\pi_{\theta_t+\eta\theta_{\star}} π η , t ⋆ = π θ t + η θ ⋆ . The exact update in Eq. (182) gives
1 η log π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + r ( x , π t ) = r ( x , π t ) − b t ⋆ ( x ) . \frac{1}{\eta}\log\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_t(y\mid x)}-r(x,y)+r(x,\pi_t)=r(x,\pi_t)-b_t^{\star}(x). η 1 log π t ( y ∣ x ) π η , t ⋆ ( y ∣ x ) − r ( x , y ) + r ( x , π t ) = r ( x , π t ) − b t ⋆ ( x ) .
Differentiating L t K i m i L_t^{\mathrm{Kimi}} L t Kimi at θ = θ t + η θ ⋆ \theta=\theta_t+\eta\theta_{\star} θ = θ t + η θ ⋆ and using the softmax linear policy gradient in Lemma B.14 therefore gives
⟨ ∇ L t K i m i ( π η , t ⋆ ) , θ − θ t − η θ ⋆ ⟩ = 2 η E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) ⟨ E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ ϕ ( x , y ) ] − E y ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ) ] , θ − θ t − η θ ⋆ ⟩ ] = 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π θ ( x , y ) ] ] . (217) \begin{aligned}
\left\langle\nabla L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^{\star}),\theta-\theta_t-\eta\theta_{\star}\right\rangle
&=\frac{2}{\eta}\mathbb{E}_{x\sim\mu}\left[\left(b_t^{\star}(x)-r(x,\pi_t)\right)\left\langle\mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[\phi(x,y)]-\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[\phi(x,y)],\theta-\theta_t-\eta\theta_{\star}\right\rangle\right]\\
&=2\mathbb{E}_{x\sim\mu}\left[\left(b_t^{\star}(x)-r(x,\pi_t)\right)\mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[s_t^{\pi_{\theta}}(x,y)]\right]. \tag*{(217)}
\end{aligned} ⟨ ∇ L t Kimi ( π η , t ⋆ ) , θ − θ t − η θ ⋆ ⟩ = η 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) ⟨ E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ ϕ ( x , y )] − E y ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y )] , θ − θ t − η θ ⋆ ⟩ ] = 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π θ ( x , y )] ] . (217)
For the last equality, the definition of s t π θ s_t^{\pi_{\theta}} s t π θ in Eq. (189) and reward linearity give
s t π θ ( x , y ) = 1 η ⟨ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] , θ − θ t − η θ ⋆ ⟩ . s_t^{\pi_{\theta}}(x,y)=\frac{1}{\eta}\left\langle\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\phi(x,y')],\theta-\theta_t-\eta\theta_{\star}\right\rangle. s t π θ ( x , y ) = η 1 ⟨ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] , θ − θ t − η θ ⋆ ⟩ .
We bound E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π θ ( x , y ) ] \mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[s_t^{\pi_{\theta}}(x,y)] E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π θ ( x , y )] in Eq. (217) by first converting the expectation to π t \pi_t π t . For π η , t ⋆ = π θ t + η θ ⋆ \pi_{\eta,t}^{\star}=\pi_{\theta_t+\eta\theta_{\star}} π η , t ⋆ = π θ t + η θ ⋆ , Lemma I.3 with R = η B ⋆ R=\eta B_{\star} R = η B ⋆ gives, since D ϕ B ⋆ = D ⋆ D_{\phi}B_{\star}=D_{\star} D ϕ B ⋆ = D ⋆ ,
E y ∼ π t ( ⋅ ∣ x ) [ ( π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) − 1 ) 2 ] ≤ ( e η D ⋆ − 1 ) 2 4 e η D ⋆ ≤ η 2 D ⋆ 2 e η D ⋆ 4 . (218) \mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_t(y\mid x)}-1\right)^2\right]\leq\frac{(e^{\eta D_{\star}}-1)^2}{4e^{\eta D_{\star}}}\leq\frac{\eta^2D_{\star}^2e^{\eta D_{\star}}}{4}. \tag*{(218)} E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π η , t ⋆ ( y ∣ x ) − 1 ) 2 ] ≤ 4 e η D ⋆ ( e η D ⋆ − 1 ) 2 ≤ 4 η 2 D ⋆ 2 e η D ⋆ . (218)
Since E y ∼ π t ( ⋅ ∣ x ) [ s t π θ ( x , y ) ] = 0 \mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[s_t^{\pi_{\theta}}(x,y)]=0 E y ∼ π t ( ⋅ ∣ x ) [ s t π θ ( x , y )] = 0 , Cauchy–Schwarz and Eq. (218) give
∣ E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π θ ( x , y ) ] ∣ = ∣ E y ∼ π t ( ⋅ ∣ x ) [ ( π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) − 1 ) s t π θ ( x , y ) ] ∣ ≤ E y ∼ π t ( ⋅ ∣ x ) [ ( π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) − 1 ) 2 ] MSE t ( π θ ; x ) ≤ η D ⋆ e η D ⋆ / 2 2 MSE t ( π θ ; x ) . (219) \begin{aligned}
\left|\mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[s_t^{\pi_{\theta}}(x,y)]\right|
&=\left|\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_t(y\mid x)}-1\right)s_t^{\pi_{\theta}}(x,y)\right]\right|\\
&\leq\sqrt{\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_t(y\mid x)}-1\right)^2\right]}\sqrt{\operatorname{MSE}_t(\pi_{\theta};x)}\\
&\leq\frac{\eta D_{\star}e^{\eta D_{\star}/2}}{2}\sqrt{\operatorname{MSE}_t(\pi_{\theta};x)}. \tag*{(219)}
\end{aligned} E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π θ ( x , y )] = E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π η , t ⋆ ( y ∣ x ) − 1 ) s t π θ ( x , y ) ] ≤ E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π η , t ⋆ ( y ∣ x ) − 1 ) 2 ] MSE t ( π θ ; x ) ≤ 2 η D ⋆ e η D ⋆ /2 MSE t ( π θ ; x ) . (219)
Returning to Eq. (217) , we obtain
∣ ⟨ ∇ L t K i m i ( π η , t ⋆ ) , θ − θ t − η θ ⋆ ⟩ ∣ ≤ 2 E x ∼ μ [ ∣ ( b t ⋆ ( x ) − r ( x , π t ) ) E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π θ ( x , y ) ] ∣ ] ≤ 2 E x ∼ μ [ η D ⋆ 2 8 ⋅ η D ⋆ e η D ⋆ / 2 2 MSE t ( π θ ; x ) ] = η 2 D ⋆ 3 e η D ⋆ / 2 8 E x ∼ μ [ MSE t ( π θ ; x ) ] ≤ η 2 D ⋆ 3 e η D ⋆ / 2 8 MSE t ( π θ ) . \begin{aligned}
\left|\left\langle\nabla L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^{\star}),\theta-\theta_t-\eta\theta_{\star}\right\rangle\right|
&\leq2\mathbb{E}_{x\sim\mu}\left[\left|\left(b_t^{\star}(x)-r(x,\pi_t)\right)\mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[s_t^{\pi_{\theta}}(x,y)]\right|\right]\\
&\leq2\mathbb{E}_{x\sim\mu}\left[\frac{\eta D_{\star}^2}{8}\cdot\frac{\eta D_{\star}e^{\eta D_{\star}/2}}{2}\sqrt{\operatorname{MSE}_t(\pi_{\theta};x)}\right]\\
&=\frac{\eta^2D_{\star}^3e^{\eta D_{\star}/2}}{8}\mathbb{E}_{x\sim\mu}\left[\sqrt{\operatorname{MSE}_t(\pi_{\theta};x)}\right]\\
&\leq\frac{\eta^2D_{\star}^3e^{\eta D_{\star}/2}}{8}\sqrt{\operatorname{MSE}_t(\pi_{\theta})}.
\end{aligned} ⟨ ∇ L t Kimi ( π η , t ⋆ ) , θ − θ t − η θ ⋆ ⟩ ≤ 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π θ ( x , y )] ] ≤ 2 E x ∼ μ [ 8 η D ⋆ 2 ⋅ 2 η D ⋆ e η D ⋆ /2 MSE t ( π θ ; x ) ] = 8 η 2 D ⋆ 3 e η D ⋆ /2 E x ∼ μ [ MSE t ( π θ ; x ) ] ≤ 8 η 2 D ⋆ 3 e η D ⋆ /2 MSE t ( π θ ) .
Here 0 ≤ b t ⋆ ( x ) − r ( x , π t ) ≤ η D ⋆ 2 / 8 0\leq b_t^{\star}(x)-r(x,\pi_t)\leq\eta D_{\star}^2/8 0 ≤ b t ⋆ ( x ) − r ( x , π t ) ≤ η D ⋆ 2 /8 by Lemma J.2 . The second inequality also uses Eq. (219) , and the last is Cauchy–Schwarz over x ∼ μ x\sim\mu x ∼ μ .
Lemma J.5. Let η > 0 \eta> 0 η > 0 and π t > 0 \pi_t > 0 π t > 0 . If r ( x , y ) ∈ [ 0 , R max ] r(x,y) \in[0,R_{\max}] r ( x , y ) ∈ [ 0 , R m a x ] , every policy π > 0 \pi> 0 π > 0 satisfies
L t K i m i ( π ) − L t K i m i ( π η , t ⋆ ) ≥ 3 4 MSE t ( π ) − η 4 R max 2 e η R max 64 . L_t^{\mathrm{Kimi}}(\pi)-L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^{\star}) \ge\frac{3}{4}\operatorname{MSE}_t(\pi)-\frac{\eta^4 R_{\max}^2 e^{\eta R_{\max}}}{64}. L t Kimi ( π ) − L t Kimi ( π η , t ⋆ ) ≥ 4 3 MSE t ( π ) − 64 η 4 R m a x 2 e η R m a x . Under Assumption 2.1, this bound also holds with D ⋆ D_{\star} D ⋆ in place of R max R_{\max} R m a x .
Proof of Lemma J.5 . By Lemma I.1 with b ~ t G ( x ) = r ( x , π t ) \tilde{b}_t^G(x)=r(x,\pi_t) b ~ t G ( x ) = r ( x , π t ) ,
L t K i m i ( π ) − L t K i m i ( π η , t ⋆ ) = MSE t ( π ) + E x ∼ μ [ ( 1 η D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) ) 2 − ( b t ⋆ ( x ) − r ( x , π t ) ) 2 ] ≥ MSE t ( π ) + 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) ( 1 η D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) − b t ⋆ ( x ) + r ( x , π t ) ) ] . (220) \begin{aligned}
L_t^{\mathrm{Kimi}}(\pi)-L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^{\star})
&=\operatorname{MSE}_t(\pi)+\mathbb{E}_{x\sim\mu}\left[\left(\frac{1}{\eta}D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\mathbin\Vert\pi(\cdot\mid x))\right)^2-\left(b_t^{\star}(x)-r(x,\pi_t)\right)^2\right]\\
&\ge\operatorname{MSE}_t(\pi)+2\mathbb{E}_{x\sim\mu}\left[\left(b_t^{\star}(x)-r(x,\pi_t)\right)\left(\frac{1}{\eta}D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\mathbin\Vert\pi(\cdot\mid x))-b_t^{\star}(x)+r(x,\pi_t)\right)\right]. \tag*{(220)}
\end{aligned} L t Kimi ( π ) − L t Kimi ( π η , t ⋆ ) = MSE t ( π ) + E x ∼ μ [ ( η 1 D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) ) 2 − ( b t ⋆ ( x ) − r ( x , π t ) ) 2 ] ≥ MSE t ( π ) + 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) ( η 1 D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) − b t ⋆ ( x ) + r ( x , π t ) ) ] . (220)
To lower bound the second term in Eq. (220) , the definition of s t π s_t^{\pi} s t π in Eq. (189) gives, for each prompt x x x ,
E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ e η s t π ( x , y ) ] = E y ∼ π t ( ⋅ ∣ x ) [ π ( y ∣ x ) π t ( y ∣ x ) e D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) − η b t ⋆ ( x ) + η r ( x , π t ) ] = e D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) − η b t ⋆ ( x ) + η r ( x , π t ) . \mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}\left[e^{\eta s_t^{\pi}(x,y)}\right]
=\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[\frac{\pi(y\mid x)}{\pi_t(y\mid x)}e^{D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\mathbin\Vert\pi(\cdot\mid x))-\eta b_t^{\star}(x)+\eta r(x,\pi_t)}\right]
=e^{D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\mathbin\Vert\pi(\cdot\mid x))-\eta b_t^{\star}(x)+\eta r(x,\pi_t)}. E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ e η s t π ( x , y ) ] = E y ∼ π t ( ⋅ ∣ x ) [ π t ( y ∣ x ) π ( y ∣ x ) e D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) − η b t ⋆ ( x ) + η r ( x , π t ) ] = e D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) − η b t ⋆ ( x ) + η r ( x , π t ) .
Taking logarithms and applying Jensen’s inequality yields
1 η D K L ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x ) ) − b t ⋆ ( x ) + r ( x , π t ) = 1 η log E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ e η s t π ( x , y ) ] ≥ E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π ( x , y ) ] . (221) \frac{1}{\eta}D_{\mathrm{KL}}(\pi_t(\cdot\mid x)\mathbin\Vert\pi(\cdot\mid x))-b_t^{\star}(x)+r(x,\pi_t)
=\frac{1}{\eta}\log\mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}\left[e^{\eta s_t^{\pi}(x,y)}\right]
\ge\mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[s_t^{\pi}(x,y)]. \tag*{(221)} η 1 D KL ( π t ( ⋅ ∣ x ) ∥ π ( ⋅ ∣ x )) − b t ⋆ ( x ) + r ( x , π t ) = η 1 log E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ e η s t π ( x , y ) ] ≥ E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π ( x , y )] . (221)
Since b t ⋆ ( x ) − r ( x , π t ) ≥ 0 b_t^{\star}(x)-r(x,\pi_t)\ge0 b t ⋆ ( x ) − r ( x , π t ) ≥ 0 by Lemma J.2 , substituting Eq. (221) into Eq. (220) yields
L t K i m i ( π ) − L t K i m i ( π η , t ⋆ ) ≥ MSE t ( π ) + 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π ( x , y ) ] ] ≥ MSE t ( π ) − 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) ∣ E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π ( x , y ) ] ∣ ] . (222) \begin{aligned}
L_t^{\mathrm{Kimi}}(\pi)-L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^{\star})
&\ge\operatorname{MSE}_t(\pi)+2\mathbb{E}_{x\sim\mu}\left[(b_t^{\star}(x)-r(x,\pi_t))\mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[s_t^{\pi}(x,y)]\right]\\
&\ge\operatorname{MSE}_t(\pi)-2\mathbb{E}_{x\sim\mu}\left[(b_t^{\star}(x)-r(x,\pi_t))\left|\mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[s_t^{\pi}(x,y)]\right|\right]. \tag*{(222)}
\end{aligned} L t Kimi ( π ) − L t Kimi ( π η , t ⋆ ) ≥ MSE t ( π ) + 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t )) E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π ( x , y )] ] ≥ MSE t ( π ) − 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t )) E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π ( x , y )] ] . (222)
Next, we bound ∣ E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π ( x , y ) ] ∣ \left|\mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[s_t^{\pi}(x,y)]\right| E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π ( x , y )] in the RHS of Eq. (222) in terms of MSE t \operatorname{MSE}_t MSE t . To transfer the expectation from π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ to π t \pi_t π t , we use E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] = e η b t ⋆ ( x ) \mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[e^{\eta r(x,y)}]=e^{\eta b_t^{\star}(x)} E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] = e η b t ⋆ ( x ) to obtain
E y ∼ π t ( ⋅ ∣ x ) [ ( π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) − 1 ) 2 ] = E y ∼ π t ( ⋅ ∣ x ) [ ( e η r ( x , y ) − e η b t ⋆ ( x ) ) 2 ] e 2 η b t ⋆ ( x ) ≤ E y ∼ π t ( ⋅ ∣ x ) [ ( 1 + e η R max ) e η r ( x , y ) − e η R max − e 2 η b t ⋆ ( x ) ] e 2 η b t ⋆ ( x ) = ( e η R max − e η b t ⋆ ( x ) ) ( e η b t ⋆ ( x ) − 1 ) e 2 η b t ⋆ ( x ) ≤ ( e η R max − 1 ) 2 4 e η R max ≤ η 2 R max 2 e η R max 4 . (223) \begin{aligned}
\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_t(y\mid x)}-1\right)^2\right]
&=\frac{\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[(e^{\eta r(x,y)}-e^{\eta b_t^{\star}(x)})^2\right]}{e^{2\eta b_t^{\star}(x)}}\\
&\le\frac{\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[(1+e^{\eta R_{\max}})e^{\eta r(x,y)}-e^{\eta R_{\max}}-e^{2\eta b_t^{\star}(x)}\right]}{e^{2\eta b_t^{\star}(x)}}\\
&=\frac{(e^{\eta R_{\max}}-e^{\eta b_t^{\star}(x)})(e^{\eta b_t^{\star}(x)}-1)}{e^{2\eta b_t^{\star}(x)}}\\
&\le\frac{(e^{\eta R_{\max}}-1)^2}{4e^{\eta R_{\max}}}\le\frac{\eta^2R_{\max}^2e^{\eta R_{\max}}}{4}. \tag*{(223)}
\end{aligned} E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π η , t ⋆ ( y ∣ x ) − 1 ) 2 ] = e 2 η b t ⋆ ( x ) E y ∼ π t ( ⋅ ∣ x ) [ ( e η r ( x , y ) − e η b t ⋆ ( x ) ) 2 ] ≤ e 2 η b t ⋆ ( x ) E y ∼ π t ( ⋅ ∣ x ) [ ( 1 + e η R m a x ) e η r ( x , y ) − e η R m a x − e 2 η b t ⋆ ( x ) ] = e 2 η b t ⋆ ( x ) ( e η R m a x − e η b t ⋆ ( x ) ) ( e η b t ⋆ ( x ) − 1 ) ≤ 4 e η R m a x ( e η R m a x − 1 ) 2 ≤ 4 η 2 R m a x 2 e η R m a x . (223)
Then Cauchy–Schwarz yields
∣ E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π ( x , y ) ] ∣ = ∣ E y ∼ π t ( ⋅ ∣ x ) [ ( π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) − 1 ) s t π ( x , y ) ] ∣ ≤ E y ∼ π t ( ⋅ ∣ x ) [ ( π η , t ⋆ ( y ∣ x ) π t ( y ∣ x ) − 1 ) 2 ] MSE t ( π ; x ) ≤ η R max e η R max / 2 2 MSE t ( π ; x ) , (224) \begin{aligned}
\left|\mathbb{E}_{y\sim\pi_{\eta,t}^{\star}(\cdot\mid x)}[s_t^{\pi}(x,y)]\right|
&=\left|\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_t(y\mid x)}-1\right)s_t^{\pi}(x,y)\right]\right|\\
&\le\sqrt{\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{\pi_{\eta,t}^{\star}(y\mid x)}{\pi_t(y\mid x)}-1\right)^2\right]}\sqrt{\operatorname{MSE}_t(\pi;x)}\\
&\le\frac{\eta R_{\max}e^{\eta R_{\max}/2}}{2}\sqrt{\operatorname{MSE}_t(\pi;x)}, \tag*{(224)}
\end{aligned} E y ∼ π η , t ⋆ ( ⋅ ∣ x ) [ s t π ( x , y )] = E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π η , t ⋆ ( y ∣ x ) − 1 ) s t π ( x , y ) ] ≤ E y ∼ π t ( ⋅ ∣ x ) [ ( π t ( y ∣ x ) π η , t ⋆ ( y ∣ x ) − 1 ) 2 ] MSE t ( π ; x ) ≤ 2 η R m a x e η R m a x /2 MSE t ( π ; x ) , (224)
where in the last inequality we have used Eq. (223) . Substituting Eq. (224) into Eq. (222) and again applying Cauchy–Schwarz, we obtain
L t K i m i ( π ) − L t K i m i ( π η , t ⋆ ) ≥ MSE t ( π ) − η R max e η R max / 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) MSE t ( π ; x ) ] ≥ MSE t ( π ) − η R max e η R max / 2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) 2 ] MSE t ( π ) ≥ MSE t ( π ) − η 2 R max 3 e η R max / 2 8 MSE t ( π ) ≥ 3 4 MSE t ( π ) − η 4 R max 6 e η R max 64 . \begin{aligned}
L_t^{\mathrm{Kimi}}(\pi)-L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^\star)
&\ge\operatorname{MSE}_t(\pi)-\eta R_{\max}e^{\eta R_{\max}/2}\mathbb{E}_{x\sim\mu}\left[\left(b_t^\star(x)-r(x,\pi_t)\right)\sqrt{\operatorname{MSE}_t(\pi;x)}\right]\\
&\ge\operatorname{MSE}_t(\pi)-\eta R_{\max}e^{\eta R_{\max}/2}\sqrt{\mathbb{E}_{x\sim\mu}\left[\left(b_t^\star(x)-r(x,\pi_t)\right)^2\right]}\sqrt{\operatorname{MSE}_t(\pi)}\\
&\ge\operatorname{MSE}_t(\pi)-\frac{\eta^2R_{\max}^3e^{\eta R_{\max}/2}}{8}\sqrt{\operatorname{MSE}_t(\pi)}\\
&\ge\frac{3}{4}\operatorname{MSE}_t(\pi)-\frac{\eta^4R_{\max}^6e^{\eta R_{\max}}}{64}.
\end{aligned} L t Kimi ( π ) − L t Kimi ( π η , t ⋆ ) ≥ MSE t ( π ) − η R m a x e η R m a x /2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) MSE t ( π ; x ) ] ≥ MSE t ( π ) − η R m a x e η R m a x /2 E x ∼ μ [ ( b t ⋆ ( x ) − r ( x , π t ) ) 2 ] MSE t ( π ) ≥ MSE t ( π ) − 8 η 2 R m a x 3 e η R m a x /2 MSE t ( π ) ≥ 4 3 MSE t ( π ) − 64 η 4 R m a x 6 e η R m a x .
The third line uses 0 ≤ b t ⋆ ( x ) − r ( x , π t ) ≤ η R max 2 / 8 0\le b_t^\star(x)-r(x,\pi_t)\le\eta R_{\max}^2/8 0 ≤ b t ⋆ ( x ) − r ( x , π t ) ≤ η R m a x 2 /8 from Lemma J.2, and the last uses Young’s inequality. Under Assumption 2.1, the same calculation applies with D ⋆ D_\star D ⋆ in place of R max R_{\max} R m a x .
Inner Loop Convergence
We now prove inner loop convergence for the mean baseline in Algorithm 3. The proof uses empirical descent to bound the excess loss of the last iterate, then converts this bound to one on its MSE.
Theorem J.1 (Inner convergence for the mean baseline). There is a universal constant C > 0 C>0 C > 0 such that the following holds. Fix an outer iteration t t t and suppose that Assumption 2.1 holds. Let δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , let G ≥ 1 G\ge1 G ≥ 1 be an integer, and let η , ε i n S q > 0 \eta,\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}>0 η , ε in Sq > 0 satisfy
η 4 D ⋆ 6 ≤ ε i n S q ≤ D ⋆ 2 . (225) \eta^4D_\star^6\le\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\le D_\star^2. \tag*{(225)} η 4 D ⋆ 6 ≤ ε in Sq ≤ D ⋆ 2 . (225) Run the inner loop of Algorithm 3 with
0 < α ≤ η 2 D ϕ 2 ( 2 + η D ⋆ ) , 0<\alpha\le\frac{\eta^2}{D_\phi^2(2+\eta D_\star)}, 0 < α ≤ D ϕ 2 ( 2 + η D ⋆ ) η 2 , K : = max { 1 , ⌈ 2 η 2 3 α ε i n S q ( B ⋆ 2 − 4 ε i n S q D ϕ 2 ) ⌉ } . (226) K:=\max\left\{1,\left\lceil\frac{2\eta^2}{3\alpha\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\left(B_\star^2-\frac{4\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{D_\phi^2}\right)\right\rceil\right\}. \tag*{(226)} K := max { 1 , ⌈ 3 α ε in Sq 2 η 2 ( B ⋆ 2 − D ϕ 2 4 ε in Sq ) ⌉ } . (226) Choose an integer N ≥ 1 N\ge1 N ≥ 1 such that
N ≥ C D ⋆ 2 ε i n S q [ d log ( 1 + 2 N ) + log 4 δ ] . (227) N\ge C\frac{D_\star^2}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\left[d\log(1+2N)+\log\frac{4}{\delta}\right]. \tag*{(227)} N ≥ C ε in Sq D ⋆ 2 [ d log ( 1 + 2 N ) + log δ 4 ] . (227) Conditional on the history preceding iteration t t t , with probability at least 1 − δ 1-\delta 1 − δ over the batch and its additional baseline responses,
MSE t ( π t K ) ≤ ε i n S q , max 0 ≤ k ≤ K ∥ s t π t k ∥ ∞ ≤ 101 96 D ⋆ . \operatorname{MSE}_t(\pi_t^K)\le\varepsilon_{\mathrm{in}}^{\mathrm{Sq}},\qquad
\max_{0\le k\le K}\left\|s_t^{\pi_t^k}\right\|_\infty\le\sqrt{\frac{101}{96}}D_\star. MSE t ( π t K ) ≤ ε in Sq , 0 ≤ k ≤ K max s t π t k ∞ ≤ 96 101 D ⋆ .
With the largest allowed learning rate in Eq. (226) , the theorem requires O ( D ⋆ 2 / ε i n S q ) O(D_\star^2/\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}) O ( D ⋆ 2 / ε in Sq ) inner steps and a batch of size O ~ ( D ⋆ 2 ε i n S q ( d + log ( 1 / δ ) ) ) \widetilde{O}\left(\frac{D_\star^2}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}(d+\log(1/\delta))\right) O ( ε in Sq D ⋆ 2 ( d + log ( 1/ δ )) ) . These guarantees hold for every G ≥ 1 G\ge1 G ≥ 1 .
The proof below proceeds as follows. We first use global convexity of L t K i m i L_t^{\mathrm{Kimi}} L t Kimi and concentration to show that θ t k \theta_t^k θ t k stays within a bounded ball throughout the inner loop (Lemma J.6), and moreover the empirical loss L ^ t K i m i ( π t k ) \widehat{L}_t^{\mathrm{Kimi}}(\pi_t^k) L t Kimi ( π t k ) decreases monotonically (Lemma J.7). We use these guarantees to show that π t K \pi_t^K π t K converges on the empirical loss (Eq. (235) ), which we then transfer to the population excess loss L t K i m i ( π t K ) − L t K i m i ( π η , t ⋆ ) L_t^{\mathrm{Kimi}}(\pi_t^K)-L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^\star) L t Kimi ( π t K ) − L t Kimi ( π η , t ⋆ ) and finally to MSE t ( π t K ) \operatorname{MSE}_t(\pi_t^K) MSE t ( π t K ) . The last step uses a loss–MSE comparison showing that, although π η , t ⋆ \pi_{\eta,t}^\star π η , t ⋆ need not minimize L t K i m i L_t^{\mathrm{Kimi}} L t Kimi , small population excess loss guarantees small MSE up to an error that shrinks with η \eta η (Eq. (236) ).
Proof of Theorem J.1
Proof of Theorem J.1. Condition on the history preceding iteration t t t . We first establish useful convexity and concentration inequalities for our analysis.
We apply the concentration results in Lemma I.6 for general Kimi-style objectives to the mean baseline where b ^ t G ( x ) = r ( x , π t ) \hat{b}^{G}_{t}(x)=r(x,\pi_t) b ^ t G ( x ) = r ( x , π t ) and min y r ( x , y ) ≤ b ^ t G ( x ) ≤ max y r ( x , y ) \min_y r(x,y)\leq\hat{b}^{G}_{t}(x)\leq\max_y r(x,y) min y r ( x , y ) ≤ b ^ t G ( x ) ≤ max y r ( x , y ) . We will invoke it using the theorem parameter conditions, A = D ⋆ A=D_{\star} A = D ⋆ , and R = 13 η B ⋆ / 4 R=13\eta B_{\star}/4 R = 13 η B ⋆ /4 , which specifies the parameter radius over which Lemma I.6 holds. With N N N as in Eq. (227) and η D ⋆ ≤ 1 \eta D_{\star}\leq1 η D ⋆ ≤ 1 from Eq. (225) , the statistical error in Eq. (202) satisfies, for sufficiently large C C C ,
ζ N ( δ ; R , D ⋆ ) ≤ ε i n S q 128 ≔ ζ . (228) \zeta_N(\delta;R,D_{\star})\leq\frac{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{128}\coloneqq\zeta. \tag*{(228)} ζ N ( δ ; R , D ⋆ ) ≤ 128 ε in Sq : = ζ . (228)
Then Lemma I.6 states that the following inequality holds with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every θ \theta θ such that ∥ θ − θ t ∥ 2 ≤ 13 η B ⋆ / 4 \lVert\theta-\theta_t\rVert_2\leq13\eta B_{\star}/4 ∥ θ − θ t ∥ 2 ≤ 13 η B ⋆ /4 ,
∣ L ^ t K i m i ( π θ ) − L ^ t K i m i ( π η , t ⋆ ) − L t K i m i ( π θ ) + L t K i m i ( π η , t ⋆ ) ∣ ≤ 1 64 MSE t ( π θ ) + ζ . (229) \left|\widehat{L}^{\mathrm{Kimi}}_t(\pi_\theta)-\widehat{L}^{\mathrm{Kimi}}_t(\pi_{\eta,t}^{\star})-L^{\mathrm{Kimi}}_t(\pi_\theta)+L^{\mathrm{Kimi}}_t(\pi_{\eta,t}^{\star})\right|\leq\frac{1}{64}\operatorname{MSE}_t(\pi_\theta)+\zeta. \tag*{(229)} L t Kimi ( π θ ) − L t Kimi ( π η , t ⋆ ) − L t Kimi ( π θ ) + L t Kimi ( π η , t ⋆ ) ≤ 64 1 MSE t ( π θ ) + ζ . (229)
Since b ~ t G ( x ) = r ( x , π t ) \widetilde{b}^{G}_{t}(x)=r(x,\pi_t) b t G ( x ) = r ( x , π t ) , we apply the empirical first-order convexity bound in Eq. (212) of Lemma I.7 with a = 0 a=0 a = 0 , A = D ⋆ A=D_{\star} A = D ⋆ , and R = 13 η B ⋆ / 4 R=13\eta B_{\star}/4 R = 13 η B ⋆ /4 . Together with Eq. (228) , this gives on the same event, for every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ 13 η B ⋆ / 4 \lVert\theta-\theta_t\rVert_2\leq13\eta B_{\star}/4 ∥ θ − θ t ∥ 2 ≤ 13 η B ⋆ /4 ,
⟨ ∇ L ^ t K i m i ( π θ ) , θ − θ t − η θ ⋆ ⟩ ≥ L ^ t K i m i ( π θ ) − L ^ t K i m i ( π η , t ⋆ ) + 31 32 MSE t ( π θ ) − 2 ζ . (230) \left\langle\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_\theta),\theta-\theta_t-\eta\theta_{\star}\right\rangle\geq\widehat{L}^{\mathrm{Kimi}}_t(\pi_\theta)-\widehat{L}^{\mathrm{Kimi}}_t(\pi_{\eta,t}^{\star})+\frac{31}{32}\operatorname{MSE}_t(\pi_\theta)-2\zeta. \tag*{(230)} ⟨ ∇ L t Kimi ( π θ ) , θ − θ t − η θ ⋆ ⟩ ≥ L t Kimi ( π θ ) − L t Kimi ( π η , t ⋆ ) + 32 31 MSE t ( π θ ) − 2 ζ . (230)
We condition on Eqs. (229) and (230) holding for the remainder of the proof.
Eq. (230) holds whenever ∥ θ − θ t ∥ 2 ≤ 13 η B ⋆ / 4 \lVert\theta-\theta_t\rVert_2\leq13\eta B_{\star}/4 ∥ θ − θ t ∥ 2 ≤ 13 η B ⋆ /4 . The following lemma, proved at the end of this section, shows that all inner iterates satisfy this condition.
Lemma J.6 (Localization for the mean baseline). Under the assumptions and parameter choices of Theorem J.1, suppose that Eqs. (229) and (230) hold. Then
max 0 ≤ k ≤ K ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 ≤ 101 96 η 2 B ⋆ 2 , max 0 ≤ k ≤ K ∥ s t π k ∥ ∞ ≤ 101 96 D ⋆ . (231) \max_{0\leq k\leq K}\left\lVert\theta_t^k-\theta_t-\eta\theta_{\star}\right\rVert_2^2\leq\frac{101}{96}\eta^2B_{\star}^2,\qquad\max_{0\leq k\leq K}\left\lVert s_t^{\pi^k}\right\rVert_{\infty}\leq\sqrt{\frac{101}{96}D_{\star}}. \tag*{(231)} 0 ≤ k ≤ K max θ t k − θ t − η θ ⋆ 2 2 ≤ 96 101 η 2 B ⋆ 2 , 0 ≤ k ≤ K max s t π k ∞ ≤ 96 101 D ⋆ . (231)
The bound on ∥ s t π K ∥ ∞ \lVert s_t^{\pi^K}\rVert_{\infty} ∥ s t π K ∥ ∞ in the theorem statement follows directly from Eq. (231) . In addition, by the triangle inequality, Lemma J.6 gives
∥ θ t k − θ t ∥ 2 ≤ 9 4 η B ⋆ , 0 ≤ k ≤ K . (232) \left\lVert\theta_t^k-\theta_t\right\rVert_2\leq\frac{9}{4}\eta B_{\star},\qquad0\leq k\leq K. \tag*{(232)} θ t k − θ t 2 ≤ 4 9 η B ⋆ , 0 ≤ k ≤ K . (232)
Convergence of the empirical loss. We next bound L ^ t K i m i ( π t K ) − L ^ t K i m i ( π η , t ⋆ ) \widehat{L}^{\mathrm{Kimi}}_t(\pi_t^K)-\widehat{L}^{\mathrm{Kimi}}_t(\pi_{\eta,t}^{\star}) L t Kimi ( π t K ) − L t Kimi ( π η , t ⋆ ) . We apply Lemma J.7 , stated and proved later in this section, with the learning rate in the theorem preconditions (Eq. (226) ). For every realization of the batch and its additional baseline responses, Lemma J.7 gives, for every 0 ≤ k < K 0\leq k<K 0 ≤ k < K ,
L ^ t K i m i ( π t k ) − L ^ t K i m i ( π t k + 1 ) ≥ α 2 ∥ ∇ L ^ t K i m i ( π t k ) ∥ 2 2 . (233) \widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)-\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^{k+1})\geq\frac{\alpha}{2}\left\lVert\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)\right\rVert_2^2. \tag*{(233)} L t Kimi ( π t k ) − L t Kimi ( π t k + 1 ) ≥ 2 α ∇ L t Kimi ( π t k ) 2 2 . (233)
Expanding the update in Algorithm 3 gives, for 0 ≤ k < K 0\leq k<K 0 ≤ k < K ,
∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 = − 2 α ⟨ ∇ L ^ t K i m i ( π t k ) , θ t k − θ t − η θ ⋆ ⟩ + α 2 ∥ ∇ L ^ t K i m i ( π t k ) ∥ 2 2 ≤ − 2 α [ L ^ t K i m i ( π t k ) − L ^ t K i m i ( π η , t ⋆ ) + 31 32 MSE t ( π t k ) − 2 ζ ] + α 2 ∥ ∇ L ^ t K i m i ( π t k ) ∥ 2 2 ≤ − 2 α [ L ^ t K i m i ( π t k + 1 ) − L ^ t K i m i ( π η , t ⋆ ) + 31 32 MSE t ( π t k ) − 2 ζ ] . (234) \begin{aligned}
\left\lVert\theta_t^{k+1}-\theta_t-\eta\theta_{\star}\right\rVert_2^2-\left\lVert\theta_t^k-\theta_t-\eta\theta_{\star}\right\rVert_2^2
&=-2\alpha\left\langle\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k),\theta_t^k-\theta_t-\eta\theta_{\star}\right\rangle+\alpha^2\left\lVert\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)\right\rVert_2^2\\
&\leq-2\alpha\left[\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)-\widehat{L}^{\mathrm{Kimi}}_t(\pi_{\eta,t}^{\star})+\frac{31}{32}\operatorname{MSE}_t(\pi_t^k)-2\zeta\right]+\alpha^2\left\lVert\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)\right\rVert_2^2\\
&\leq-2\alpha\left[\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^{k+1})-\widehat{L}^{\mathrm{Kimi}}_t(\pi_{\eta,t}^{\star})+\frac{31}{32}\operatorname{MSE}_t(\pi_t^k)-2\zeta\right]. \tag*{(234)}
\end{aligned} θ t k + 1 − θ t − η θ ⋆ 2 2 − θ t k − θ t − η θ ⋆ 2 2 = − 2 α ⟨ ∇ L t Kimi ( π t k ) , θ t k − θ t − η θ ⋆ ⟩ + α 2 ∇ L t Kimi ( π t k ) 2 2 ≤ − 2 α [ L t Kimi ( π t k ) − L t Kimi ( π η , t ⋆ ) + 32 31 MSE t ( π t k ) − 2 ζ ] + α 2 ∇ L t Kimi ( π t k ) 2 2 ≤ − 2 α [ L t Kimi ( π t k + 1 ) − L t Kimi ( π η , t ⋆ ) + 32 31 MSE t ( π t k ) − 2 ζ ] . (234)
The first inequality applies the empirical first-order convexity bound we proved in Eq. (230) , which holds for π t k \pi_t^k π t k due to the parameter norm bound in Eq. (232) . The second uses the empirical descent bound in Eq. (233) above.
The empirical descent bound in Eq. (233) also gives L ^ t K i m i ( π t K ) ≤ L ^ t K i m i ( π t k + 1 ) \widehat{L}_t^{\mathrm{Kimi}}(\pi_t^K) \le\widehat{L}_t^{\mathrm{Kimi}}(\pi_t^{k+1}) L t Kimi ( π t K ) ≤ L t Kimi ( π t k + 1 ) for every 0 ≤ k < K 0 \le k < K 0 ≤ k < K . Hence
L ^ t K i m i ( π t K ) − L ^ t K i m i ( π η , t ∗ ) ≤ 1 K ∑ k = 0 K − 1 ( L ^ t K i m i ( π t k + 1 ) − L ^ t K i m i ( π η , t ∗ ) ) ≤ 1 K ∑ k = 0 K − 1 [ ∥ θ t k − θ t − η θ ∗ ∥ 2 2 − ∥ θ t k + 1 − θ t − η θ ∗ ∥ 2 2 2 α − 31 32 MSE t ( π t k ) + 2 ζ ] = η 2 B ∗ 2 − ∥ θ t K − θ t − η θ ∗ ∥ 2 2 2 α K − 31 32 K ∑ k = 0 K − 1 MSE t ( π t k ) + 2 ζ . (235) \begin{aligned}
\widehat{L}_t^{\mathrm{Kimi}}(\pi_t^K) - \widehat{L}_t^{\mathrm{Kimi}}(\pi_{\eta,t}^*) \le\frac{1}{K}\sum_{k=0}^{K-1}\left(\widehat{L}_t^{\mathrm{Kimi}}(\pi_t^{k+1})-\widehat{L}_t^{\mathrm{Kimi}}(\pi_{\eta,t}^*)\right) \\
&\le\frac{1}{K}\sum_{k=0}^{K-1}\left[\frac{\left\|\theta_t^k-\theta_t-\eta\theta_*\right\|_2^2-\left\|\theta_t^{k+1}-\theta_t-\eta\theta_*\right\|_2^2}{2\alpha}-\frac{31}{32}\operatorname{MSE}_t(\pi_t^k)+2\zeta\right] \\
&= \frac{\eta^2 B_*^2-\left\|\theta_t^K-\theta_t-\eta\theta_*\right\|_2^2}{2\alpha K}-\frac{31}{32K}\sum_{k=0}^{K-1}\operatorname{MSE}_t(\pi_t^k)+2\zeta. \tag*{(235)}
\end{aligned} L t Kimi ( π t K ) − L t Kimi ( π η , t ∗ ) ≤ K 1 k = 0 ∑ K − 1 ( L t Kimi ( π t k + 1 ) − L t Kimi ( π η , t ∗ ) ) ≤ K 1 k = 0 ∑ K − 1 [ 2 α θ t k − θ t − η θ ∗ 2 2 − θ t k + 1 − θ t − η θ ∗ 2 2 − 32 31 MSE t ( π t k ) + 2 ζ ] = 2 α K η 2 B ∗ 2 − θ t K − θ t − η θ ∗ 2 2 − 32 K 31 k = 0 ∑ K − 1 MSE t ( π t k ) + 2 ζ . (235)
The second inequality substitutes Eq. (234) at each k k k . The last line follows by telescoping and using θ t 0 = θ t \theta_t^0=\theta_t θ t 0 = θ t and ∥ θ ∗ ∥ 2 = B ∗ \left\|\theta_*\right\|_2=B_* ∥ θ ∗ ∥ 2 = B ∗ .
Excess empirical loss to MSE. Finally, we translate the empirical excess loss of π t K \pi_t^K π t K to MSE t \operatorname{MSE}_t MSE t , first going through the population excess loss. Recall that π η , t ∗ = π θ t + η θ ∗ \pi_{\eta,t}^*=\pi_{\theta_t+\eta\theta_*} π η , t ∗ = π θ t + η θ ∗ (Lemma B.11). For every θ ∈ R d \theta\in\mathbb{R}^d θ ∈ R d , Taylor’s theorem at θ t + η θ ∗ \theta_t+\eta\theta_* θ t + η θ ∗ gives, for some θ ′ ∈ R d \theta'\in\mathbb{R}^d θ ′ ∈ R d ,
L t K i m i ( π θ ) − L t K i m i ( π η , t ∗ ) = ⟨ ∇ L t K i m i ( π η , t ∗ ) , θ − θ t − η θ ∗ ⟩ + 1 2 ( θ − θ t − η θ ∗ ) ⊤ ∇ 2 L t K i m i ( π θ ′ ) ( θ − θ t − η θ ∗ ) ≥ MSE t ( π θ ) + ⟨ ∇ L t K i m i ( π η , t ∗ ) , θ − θ t − η θ ∗ ⟩ ≥ MSE t ( π θ ) − η 2 D ∗ 3 e η D ∗ / 2 8 MSE t ( π θ ) . (236) \begin{aligned}
L_t^{\mathrm{Kimi}}(\pi_\theta)-L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^*)=\left\langle\nabla L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^*),\theta-\theta_t-\eta\theta_*\right\rangle+\frac{1}{2}(\theta-\theta_t-\eta\theta_*)^\top\nabla^2L_t^{\mathrm{Kimi}}(\pi_{\theta'})(\theta-\theta_t-\eta\theta_*) \\
&\ge\operatorname{MSE}_t(\pi_\theta)+\left\langle\nabla L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^*),\theta-\theta_t-\eta\theta_*\right\rangle\\
&\ge\operatorname{MSE}_t(\pi_\theta)-\frac{\eta^2D_*^3e^{\eta D_*/2}}{8}\sqrt{\operatorname{MSE}_t(\pi_\theta)}. \tag*{(236)}
\end{aligned} L t Kimi ( π θ ) − L t Kimi ( π η , t ∗ ) = ⟨ ∇ L t Kimi ( π η , t ∗ ) , θ − θ t − η θ ∗ ⟩ + 2 1 ( θ − θ t − η θ ∗ ) ⊤ ∇ 2 L t Kimi ( π θ ′ ) ( θ − θ t − η θ ∗ ) ≥ MSE t ( π θ ) + ⟨ ∇ L t Kimi ( π η , t ∗ ) , θ − θ t − η θ ∗ ⟩ ≥ MSE t ( π θ ) − 8 η 2 D ∗ 3 e η D ∗ /2 MSE t ( π θ ) . (236)
The first inequality uses ∇ 2 L t K i m i ( π θ ′ ) ⪰ 2 Σ t / η 2 \nabla^2L_t^{\mathrm{Kimi}}(\pi_{\theta'})\succeq2\Sigma_t/\eta^2 ∇ 2 L t Kimi ( π θ ′ ) ⪰ 2 Σ t / η 2 from Lemma J.3 (Eq. (216) ) and the quadratic form of the MSE (Eq. (205) ); the second uses Lemma J.4.
Next, we convert the above to the empirical loss. Since θ t K \theta_t^K θ t K satisfies the parameter norm bound in Eq. (232) , we apply the concentration inequality in Eq. (229) above. Combined with Eq. (236) this gives
L ^ t K i m i ( π t K ) − L ^ t K i m i ( π η , t ∗ ) ≥ L t K i m i ( π t K ) − L t K i m i ( π η , t ∗ ) − 1 64 MSE t ( π t K ) − ζ ≥ 63 64 MSE t ( π t K ) − η 2 D ∗ 3 e η D ∗ / 2 8 MSE t ( π t K ) − ζ . \begin{aligned}
\widehat{L}_t^{\mathrm{Kimi}}(\pi_t^K)-\widehat{L}_t^{\mathrm{Kimi}}(\pi_{\eta,t}^*)\ge L_t^{\mathrm{Kimi}}(\pi_t^K)-L_t^{\mathrm{Kimi}}(\pi_{\eta,t}^*)-\frac{1}{64}\operatorname{MSE}_t(\pi_t^K)-\zeta\\
&\ge\frac{63}{64}\operatorname{MSE}_t(\pi_t^K)-\frac{\eta^2D_*^3e^{\eta D_*/2}}{8}\sqrt{\operatorname{MSE}_t(\pi_t^K)}-\zeta.
\end{aligned} L t Kimi ( π t K ) − L t Kimi ( π η , t ∗ ) ≥ L t Kimi ( π t K ) − L t Kimi ( π η , t ∗ ) − 64 1 MSE t ( π t K ) − ζ ≥ 64 63 MSE t ( π t K ) − 8 η 2 D ∗ 3 e η D ∗ /2 MSE t ( π t K ) − ζ .
Rearranging this lower bound and then substituting the empirical upper bound Eq. (235) gives
63 64 MSE t ( π t K ) ≤ L ^ t K i m i ( π t K ) − L ^ t K i m i ( π η , t ∗ ) + η 2 D ∗ 3 e η D ∗ / 2 8 MSE t ( π t K ) + ζ ≤ η 2 B ∗ 2 − ∥ θ t K − θ t − η θ ∗ ∥ 2 2 2 α K − 31 32 K ∑ k = 0 K − 1 MSE t ( π t k ) + η 2 D ∗ 3 e η D ∗ / 2 8 MSE t ( π t K ) + 3 ζ ≤ η 2 B ∗ 2 − ∥ θ t K − θ t − η θ ∗ ∥ 2 2 2 α K + η 2 D ∗ 3 e η D ∗ / 2 8 MSE t ( π t K ) + 3 ζ . (237) \begin{aligned}
\frac{63}{64}\operatorname{MSE}_t(\pi_t^K)\le\widehat{L}_t^{\mathrm{Kimi}}(\pi_t^K)-\widehat{L}_t^{\mathrm{Kimi}}(\pi_{\eta,t}^*)+\frac{\eta^2D_*^3e^{\eta D_*/2}}{8}\sqrt{\operatorname{MSE}_t(\pi_t^K)}+\zeta\\
&\le\frac{\eta^2B_*^2-\left\|\theta_t^K-\theta_t-\eta\theta_*\right\|_2^2}{2\alpha K}-\frac{31}{32K}\sum_{k=0}^{K-1}\operatorname{MSE}_t(\pi_t^k)
+\frac{\eta^2D_*^3e^{\eta D_*/2}}{8}\sqrt{\operatorname{MSE}_t(\pi_t^K)}+3\zeta\\
&\le\frac{\eta^2B_*^2-\left\|\theta_t^K-\theta_t-\eta\theta_*\right\|_2^2}{2\alpha K}+\frac{\eta^2D_*^3e^{\eta D_*/2}}{8}\sqrt{\operatorname{MSE}_t(\pi_t^K)}+3\zeta. \tag*{(237)}
\end{aligned} 64 63 MSE t ( π t K ) ≤ L t Kimi ( π t K ) − L t Kimi ( π η , t ∗ ) + 8 η 2 D ∗ 3 e η D ∗ /2 MSE t ( π t K ) + ζ ≤ 2 α K η 2 B ∗ 2 − θ t K − θ t − η θ ∗ 2 2 − 32 K 31 k = 0 ∑ K − 1 MSE t ( π t k ) + 8 η 2 D ∗ 3 e η D ∗ /2 MSE t ( π t K ) + 3 ζ ≤ 2 α K η 2 B ∗ 2 − θ t K − θ t − η θ ∗ 2 2 + 8 η 2 D ∗ 3 e η D ∗ /2 MSE t ( π t K ) + 3 ζ . (237)
To lower bound ∥ θ t K − θ t − η θ ∗ ∥ 2 2 \left\|\theta_t^K-\theta_t-\eta\theta_*\right\|_2^2 θ t K − θ t − η θ ∗ 2 2 in Eq. (237) , we use the quadratic form of the MSE in Eq. (205) at θ t K \theta_t^K θ t K to obtain
MSE t ( π t K ) = 1 η 2 ( θ t K − θ t − η θ ∗ ) ⊤ Σ t ( θ t K − θ t − η θ ∗ ) ≤ D ϕ 2 4 η 2 ∥ θ t K − θ t − η θ ∗ ∥ 2 2 . (238) \operatorname{MSE}_t(\pi_t^K)=\frac{1}{\eta^2}(\theta_t^K-\theta_t-\eta\theta_*)^\top\Sigma_t(\theta_t^K-\theta_t-\eta\theta_*)\le\frac{D_\phi^2}{4\eta^2}\left\|\theta_t^K-\theta_t-\eta\theta_*\right\|_2^2. \tag*{(238)} MSE t ( π t K ) = η 2 1 ( θ t K − θ t − η θ ∗ ) ⊤ Σ t ( θ t K − θ t − η θ ∗ ) ≤ 4 η 2 D ϕ 2 θ t K − θ t − η θ ∗ 2 2 . (238)
Here Σ t ⪯ D ϕ 2 I / 4 \Sigma_t \preceq D_\phi^2 I/4 Σ t ⪯ D ϕ 2 I /4 , since V y ∼ π t ( ⋅ ∣ x ) [ h ⊤ ϕ ( x , y ) ] ≤ D ϕ 2 ∥ h ∥ 2 2 / 4 \mathbb{V}_{y\sim\pi_t(\cdot\mid x)}[h^\top\phi(x,y)] \le D_\phi^2\lVert h\rVert_2^2/4 V y ∼ π t ( ⋅ ∣ x ) [ h ⊤ ϕ ( x , y )] ≤ D ϕ 2 ∥ h ∥ 2 2 /4 for every x x x and h ∈ R d h\in\mathbb{R}^d h ∈ R d . Substituting Eq. (238) into Eq. (237) and rearranging gives
( 63 64 + 2 η 2 α K D ϕ 2 ) MSE t ( π t K ) ≤ η 2 B ⋆ 2 2 α K + η 2 D ⋆ 3 e η D ⋆ / 2 8 MSE t ( π t K ) + 3 ζ . (239) \left(\frac{63}{64}+\frac{2\eta^2}{\alpha K D_\phi^2}\right)\operatorname{MSE}_t(\pi_t^K) \le\frac{\eta^2 B_\star^2}{2\alpha K}+\frac{\eta^2D_\star^3e^{\eta D_\star/2}}{8}\sqrt{\operatorname{MSE}_t(\pi_t^K)}+3\zeta. \tag*{(239)} ( 64 63 + α K D ϕ 2 2 η 2 ) MSE t ( π t K ) ≤ 2 α K η 2 B ⋆ 2 + 8 η 2 D ⋆ 3 e η D ⋆ /2 MSE t ( π t K ) + 3 ζ . (239)
By Young’s inequality,
η 2 D ⋆ 3 e η D ⋆ / 2 8 MSE t ( π t K ) + 3 ζ ≤ 1 8 MSE t ( π t K ) + η 4 D ⋆ 6 e η D ⋆ 32 + 3 ζ ≤ 1 8 MSE t ( π t K ) + 7 64 ε i n S q . \begin{aligned}
\frac{\eta^2D_\star^3e^{\eta D_\star/2}}{8}\sqrt{\operatorname{MSE}_t(\pi_t^K)}+3\zeta\le\frac{1}{8}\operatorname{MSE}_t(\pi_t^K)+\frac{\eta^4D_\star^6e^{\eta D_\star}}{32}+3\zeta\\
&\le\frac{1}{8}\operatorname{MSE}_t(\pi_t^K)+\frac{7}{64}\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}.
\end{aligned} 8 η 2 D ⋆ 3 e η D ⋆ /2 MSE t ( π t K ) + 3 ζ ≤ 8 1 MSE t ( π t K ) + 32 η 4 D ⋆ 6 e η D ⋆ + 3 ζ ≤ 8 1 MSE t ( π t K ) + 64 7 ε in Sq .
The second inequality uses the range of ε i n S q \varepsilon_{\mathrm{in}}^{\mathrm{Sq}} ε in Sq in Eq. (225) from the theorem preconditions, e η D ⋆ ≤ e < 11 / 4 e^{\eta D_\star}\le e<11/4 e η D ⋆ ≤ e < 11/4 , and ζ = ε i n S q / 128 \zeta=\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}/128 ζ = ε in Sq /128 from Eq. (228) . Substituting the above panel into Eq. (239) and rearranging gives
( 55 64 + 2 η 2 α K D ϕ 2 ) MSE t ( π t K ) ≤ η 2 B ⋆ 2 2 α K + 7 64 ε i n S q ≤ ( 55 64 + 2 η 2 α K D ϕ 2 ) ε i n S q . \left(\frac{55}{64}+\frac{2\eta^2}{\alpha K D_\phi^2}\right)\operatorname{MSE}_t(\pi_t^K) \le\frac{\eta^2B_\star^2}{2\alpha K}+\frac{7}{64}\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\le\left(\frac{55}{64}+\frac{2\eta^2}{\alpha K D_\phi^2}\right)\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}. ( 64 55 + α K D ϕ 2 2 η 2 ) MSE t ( π t K ) ≤ 2 α K η 2 B ⋆ 2 + 64 7 ε in Sq ≤ ( 64 55 + α K D ϕ 2 2 η 2 ) ε in Sq .
The last inequality uses the choice of K K K in Eq. (226) . Thus MSE t ( π t K ) ≤ ε i n S q \operatorname{MSE}_t(\pi_t^K)\le\varepsilon_{\mathrm{in}}^{\mathrm{Sq}} MSE t ( π t K ) ≤ ε in Sq .
Empirical Descent
Lemma J.7 (Empirical descent for the mean baseline). Fix an outer iteration t t t and suppose that Assumption 2.1 holds. Let η > 0 \eta>0 η > 0 and let N , G , K ≥ 1 N,G,K\ge1 N , G , K ≥ 1 be integers. Run the inner loop of Algorithm 3 with learning rate
0 < α ≤ η 2 D ϕ 2 ( 2 + η D ⋆ ) . (240) 0<\alpha\le\frac{\eta^2}{D_\phi^2(2+\eta D_\star)}. \tag*{(240)} 0 < α ≤ D ϕ 2 ( 2 + η D ⋆ ) η 2 . (240) For every realization of the batch and its additional baseline responses, L ^ t K i m i ( π t k ) ≤ D ⋆ 2 \widehat{L}_t^{\mathrm{Kimi}}(\pi_t^k)\le D_\star^2 L t Kimi ( π t k ) ≤ D ⋆ 2 for 0 ≤ k ≤ K 0\le k\le K 0 ≤ k ≤ K , and
L ^ t K i m i ( π t k ) − L ^ t K i m i ( π t k + 1 ) ≥ α 2 ∥ ∇ L ^ t K i m i ( π t k ) ∥ 2 2 , \widehat{L}_t^{\mathrm{Kimi}}(\pi_t^k)-\widehat{L}_t^{\mathrm{Kimi}}(\pi_t^{k+1})\ge\frac{\alpha}{2}\left\lVert\nabla\widehat{L}_t^{\mathrm{Kimi}}(\pi_t^k)\right\rVert_2^2, L t Kimi ( π t k ) − L t Kimi ( π t k + 1 ) ≥ 2 α ∇ L t Kimi ( π t k ) 2 2 , ∥ θ t k + 1 − θ t k ∥ 2 ≤ 2 α D ϕ D ⋆ η ≤ η B ⋆ , 0 ≤ k < K . (241) \left\lVert\theta_t^{k+1}-\theta_t^k\right\rVert_2\le\frac{2\alpha D_\phi D_\star}{\eta}\le\eta B_\star,\qquad0\le k<K. \tag*{(241)} θ t k + 1 − θ t k 2 ≤ η 2 α D ϕ D ⋆ ≤ η B ⋆ , 0 ≤ k < K . (241)
Proof of Lemma J.7 . We first bound the gradient and Hessian in terms of the empirical loss. For each sample, write
q θ , i : = 1 η log π θ ( y i ∣ x i ) π t ( y i ∣ x i ) − r i + r ˉ i , L ^ t K i m i ( π θ ) = 1 N ∑ i = 1 N q θ , i 2 . q_{\theta,i}:=\frac{1}{\eta}\log\frac{\pi_\theta(y_i\mid x_i)}{\pi_t(y_i\mid x_i)}-r_i+\bar r_i,\qquad\widehat{L}_t^{\mathrm{Kimi}}(\pi_\theta)=\frac{1}{N}\sum_{i=1}^{N}q_{\theta,i}^2. q θ , i := η 1 log π t ( y i ∣ x i ) π θ ( y i ∣ x i ) − r i + r ˉ i , L t Kimi ( π θ ) = N 1 i = 1 ∑ N q θ , i 2 .
By Lemmas B.15 and B.16,
∥ ∇ q θ , i ∥ 2 ≤ D ϕ η , ∥ ∇ 2 q θ , i ∥ o p ≤ D ϕ 2 4 η , θ ∈ R d . \lVert\nabla q_{\theta,i}\rVert_2\le\frac{D_\phi}{\eta},\qquad\lVert\nabla^2q_{\theta,i}\rVert_{\mathrm{op}}\le\frac{D_\phi^2}{4\eta},\qquad\theta\in\mathbb{R}^d. ∥ ∇ q θ , i ∥ 2 ≤ η D ϕ , ∥ ∇ 2 q θ , i ∥ op ≤ 4 η D ϕ 2 , θ ∈ R d .
The second bound uses V y ∼ π θ ( ⋅ ∣ x ) [ h ⊤ ϕ ( x , y ) ] ≤ D ϕ 2 / 4 \mathbb{V}_{y\sim\pi_\theta(\cdot\mid x)}[h^\top\phi(x,y)]\le D_\phi^2/4 V y ∼ π θ ( ⋅ ∣ x ) [ h ⊤ ϕ ( x , y )] ≤ D ϕ 2 /4 for ∥ h ∥ 2 = 1 \lVert h\rVert_2=1 ∥ h ∥ 2 = 1 , which follows from the feature diameter. Differentiating the squared loss therefore gives
∥ ∇ L ^ t K i m i ( π θ ) ∥ 2 = ∥ 2 N ∑ i = 1 N q θ , i ∇ q θ , i ∥ 2 ≤ 2 D ϕ η L ^ t K i m i ( π θ ) , (242) \left\lVert\nabla\widehat{L}_t^{\mathrm{Kimi}}(\pi_\theta)\right\rVert_2=\left\lVert\frac{2}{N}\sum_{i=1}^{N}q_{\theta,i}\nabla q_{\theta,i}\right\rVert_2\le\frac{2D_\phi}{\eta}\sqrt{\widehat{L}_t^{\mathrm{Kimi}}(\pi_\theta)}, \tag*{(242)} ∇ L t Kimi ( π θ ) 2 = N 2 i = 1 ∑ N q θ , i ∇ q θ , i 2 ≤ η 2 D ϕ L t Kimi ( π θ ) , (242)
∥ ∇ 2 L ^ t K i m i ( π θ ) ∥ o p = ∥ 2 N ∑ i = 1 N [ ∇ q θ , i ∇ q θ , i ⊤ + q θ , i ∇ 2 q θ , i ] ∥ o p ≤ 2 D ϕ 2 η 2 + D ϕ 2 2 η L ^ t K i m i ( π θ ) . (243) \begin{aligned}
\left\lVert\nabla^2\widehat{L}_t^{\mathrm{Kimi}}(\pi_\theta)\right\rVert_{\mathrm{op}}=\left\lVert\frac{2}{N}\sum_{i=1}^{N}\left[\nabla q_{\theta,i}\nabla q_{\theta,i}^{\top}+q_{\theta,i}\nabla^2q_{\theta,i}\right]\right\rVert_{\mathrm{op}} \\
&\le\frac{2D_\phi^2}{\eta^2}+\frac{D_\phi^2}{2\eta}\sqrt{\widehat{L}_t^{\mathrm{Kimi}}(\pi_\theta)}. \tag*{(243)}
\end{aligned} ∇ 2 L t Kimi ( π θ ) op = N 2 i = 1 ∑ N [ ∇ q θ , i ∇ q θ , i ⊤ + q θ , i ∇ 2 q θ , i ] op ≤ η 2 2 D ϕ 2 + 2 η D ϕ 2 L t Kimi ( π θ ) . (243)
Both inequalities use N − 1 ∑ i ∣ q θ , i ∣ ≤ L ^ t K i m i ( π θ ) N^{-1}\sum_i |q_{\theta,i}| \le\sqrt{\widehat{L}^{\mathrm{Kimi}}_t(\pi_\theta)} N − 1 ∑ i ∣ q θ , i ∣ ≤ L t Kimi ( π θ ) .
The gradient bound on q θ , i q_{\theta,i} q θ , i gives ∣ q θ ′ , i − q θ , i ∣ ≤ ( D ϕ / η ) ∥ θ ′ − θ ∥ 2 |q_{\theta',i}-q_{\theta,i}| \le(D_\phi/\eta)\|\theta'-\theta\|_2 ∣ q θ ′ , i − q θ , i ∣ ≤ ( D ϕ / η ) ∥ θ ′ − θ ∥ 2 . Thus the triangle inequality yields
L ^ t K i m i ( π θ ′ ) ≤ L ^ t K i m i ( π θ ) + 1 N ∑ i = 1 N ( q θ ′ , i − q θ , i ) 2 ≤ L ^ t K i m i ( π θ ) + D ϕ η ∥ θ ′ − θ ∥ 2 . (244) \sqrt{\widehat{L}^{\mathrm{Kimi}}_t(\pi_{\theta'})} \le\sqrt{\widehat{L}^{\mathrm{Kimi}}_t(\pi_\theta)+\sqrt{\frac{1}{N}\sum_{i=1}^{N}(q_{\theta',i}-q_{\theta,i})^2}} \le\sqrt{\widehat{L}^{\mathrm{Kimi}}_t(\pi_\theta)}+\frac{D_\phi}{\eta}\|\theta'-\theta\|_2. \tag*{(244)} L t Kimi ( π θ ′ ) ≤ L t Kimi ( π θ ) + N 1 i = 1 ∑ N ( q θ ′ , i − q θ , i ) 2 ≤ L t Kimi ( π θ ) + η D ϕ ∥ θ ′ − θ ∥ 2 . (244)
We now prove descent and L ^ t K i m i ( π t k ) ≤ D ⋆ 2 \widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k) \le D_\star^2 L t Kimi ( π t k ) ≤ D ⋆ 2 by induction. Initially, L ^ t K i m i ( π t ) ≤ D ⋆ 2 \widehat{L}^{\mathrm{Kimi}}_t(\pi_t) \le D_\star^2 L t Kimi ( π t ) ≤ D ⋆ 2 because ∣ r ˉ i − r i ∣ ≤ D ⋆ |\bar r_i-r_i| \le D_\star ∣ r ˉ i − r i ∣ ≤ D ⋆ by Eq. (188) . Suppose L ^ t K i m i ( π t k ) ≤ D ⋆ 2 \widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k) \le D_\star^2 L t Kimi ( π t k ) ≤ D ⋆ 2 for some k < K k<K k < K . The gradient bound Eq. (242) and the learning rate in Eq. (240) give
∥ θ t k + 1 − θ t k ∥ 2 = α ∥ ∇ L ^ t K i m i ( π t k ) ∥ 2 ≤ 2 α D ϕ D ⋆ η ≤ η B ⋆ . \|\theta_t^{k+1}-\theta_t^k\|_2=\alpha\|\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)\|_2\le\frac{2\alpha D_\phi D_\star}{\eta}\le\eta B_\star. ∥ θ t k + 1 − θ t k ∥ 2 = α ∥∇ L t Kimi ( π t k ) ∥ 2 ≤ η 2 α D ϕ D ⋆ ≤ η B ⋆ .
To apply Taylor’s theorem to this update, we use Eqs. (243) and (244) to obtain, for every θ ′ \theta' θ ′ with ∥ θ ′ − θ t k ∥ 2 ≤ η B ⋆ \|\theta'-\theta_t^k\|_2\le\eta B_\star ∥ θ ′ − θ t k ∥ 2 ≤ η B ⋆ ,
L ^ t K i m i ( π θ ′ ) ≤ D ⋆ + D ϕ η ⋅ η B ⋆ = 2 D ⋆ , ∥ ∇ 2 L ^ t K i m i ( π θ ′ ) ∥ o p ≤ D ϕ 2 η 2 ( 2 + η D ⋆ ) . \sqrt{\widehat{L}^{\mathrm{Kimi}}_t(\pi_{\theta'})}\le D_\star+\frac{D_\phi}{\eta}\cdot\eta B_\star=2D_\star,\qquad
\|\nabla^2\widehat{L}^{\mathrm{Kimi}}_t(\pi_{\theta'})\|_{\mathrm{op}}\le\frac{D_\phi^2}{\eta^2}(2+\eta D_\star). L t Kimi ( π θ ′ ) ≤ D ⋆ + η D ϕ ⋅ η B ⋆ = 2 D ⋆ , ∥ ∇ 2 L t Kimi ( π θ ′ ) ∥ op ≤ η 2 D ϕ 2 ( 2 + η D ⋆ ) .
This Hessian bound holds at every convex combination of θ t k \theta_t^k θ t k and θ t k + 1 \theta_t^{k+1} θ t k + 1 . Taylor’s theorem therefore gives
L ^ t K i m i ( π t k ) − L ^ t K i m i ( π t k + 1 ) ≥ − ⟨ ∇ L ^ t K i m i ( π t k ) , θ t k + 1 − θ t k ⟩ − D ϕ 2 ( 2 + η D ⋆ ) 2 η 2 ∥ θ t k + 1 − θ t k ∥ 2 2 = α [ 1 − α D ϕ 2 ( 2 + η D ⋆ ) 2 η 2 ] ∥ ∇ L ^ t K i m i ( π t k ) ∥ 2 2 ≥ α 2 ∥ ∇ L ^ t K i m i ( π t k ) ∥ 2 2 . \begin{aligned}
\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)-\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^{k+1})\ge-\left\langle\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k),\theta_t^{k+1}-\theta_t^k\right\rangle-\frac{D_\phi^2(2+\eta D_\star)}{2\eta^2}\|\theta_t^{k+1}-\theta_t^k\|_2^2 \\
&=\alpha\left[1-\frac{\alpha D_\phi^2(2+\eta D_\star)}{2\eta^2}\right]\|\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)\|_2^2\ge\frac{\alpha}{2}\|\nabla\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)\|_2^2.
\end{aligned} L t Kimi ( π t k ) − L t Kimi ( π t k + 1 ) ≥ − ⟨ ∇ L t Kimi ( π t k ) , θ t k + 1 − θ t k ⟩ − 2 η 2 D ϕ 2 ( 2 + η D ⋆ ) ∥ θ t k + 1 − θ t k ∥ 2 2 = α [ 1 − 2 η 2 α D ϕ 2 ( 2 + η D ⋆ ) ] ∥∇ L t Kimi ( π t k ) ∥ 2 2 ≥ 2 α ∥∇ L t Kimi ( π t k ) ∥ 2 2 .
The learning rate bound in Eq. (240) gives the last inequality. Since L ^ t K i m i ( π t k + 1 ) ≤ L ^ t K i m i ( π t k ) ≤ D ⋆ 2 \widehat{L}^{\mathrm{Kimi}}_t(\pi_t^{k+1})\le\widehat{L}^{\mathrm{Kimi}}_t(\pi_t^k)\le D_\star^2 L t Kimi ( π t k + 1 ) ≤ L t Kimi ( π t k ) ≤ D ⋆ 2 , the induction is complete. □
Proof of Lemma J.6
Proof of Lemma J.6. We condition on the inequalities in Eqs. (229) and (230) from the theorem proof holding, and use ζ = ε i n S / 128 \zeta=\varepsilon_{\mathrm{in}}^{\mathrm{S}}/128 ζ = ε in S /128 from Eq. (228) . We will prove by induction that
∥ θ t k − θ t − η θ ⋆ ∥ 2 2 ≤ η 2 B ⋆ 2 + 10 k α ζ , 0 ≤ k ≤ K . (245) \|\theta_t^k-\theta_t-\eta\theta_\star\|_2^2\le\eta^2B_\star^2+10k\alpha\zeta,\qquad0\le k\le K. \tag*{(245)} ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 ≤ η 2 B ⋆ 2 + 10 k α ζ , 0 ≤ k ≤ K . (245)
The choice of K K K in Eq. (226) gives
α K ζ ≤ max { α ζ , η 2 B ⋆ 2 192 − ( 8 η 2 3 D ϕ 2 − α ) ζ } ≤ η 2 B ⋆ 2 192 . (246) \alpha K\zeta\le\max\left\{\alpha\zeta,\frac{\eta^2B_\star^2}{192}-\left(\frac{8\eta^2}{3D_\phi^2}-\alpha\right)\zeta\right\}\le\frac{\eta^2B_\star^2}{192}. \tag*{(246)} α K ζ ≤ max { α ζ , 192 η 2 B ⋆ 2 − ( 3 D ϕ 2 8 η 2 − α ) ζ } ≤ 192 η 2 B ⋆ 2 . (246)
Here α ≤ η 2 / ( 2 D ϕ 2 ) \alpha\le\eta^2/(2D_\phi^2) α ≤ η 2 / ( 2 D ϕ 2 ) and ε i n S ≤ D ⋆ 2 \varepsilon_{\mathrm{in}}^{\mathrm{S}}\le D_\star^2 ε in S ≤ D ⋆ 2 give α ζ ≤ η 2 B ⋆ 2 / 256 \alpha\zeta\le\eta^2B_\star^2/256 α ζ ≤ η 2 B ⋆ 2 /256 . Thus Eq. (245) will imply the parameter bound in Eq. (231) .
The initial bound holds because θ t 0 = θ t \theta_t^0=\theta_t θ t 0 = θ t . Suppose Eq. (245) holds at k < K k<K k < K . By Eq. (246) and the update bound Eq. (241) from Lemma J.7 ,
∥ θ t k − θ t ∥ 2 ≤ η B ⋆ + η 2 B ⋆ 2 + 10 k α ζ < 9 4 η B ⋆ , \|\theta_t^k-\theta_t\|_2\le\eta B_\star+\sqrt{\eta^2B_\star^2+10k\alpha\zeta}<\frac{9}{4}\eta B_\star, ∥ θ t k − θ t ∥ 2 ≤ η B ⋆ + η 2 B ⋆ 2 + 10 k α ζ < 4 9 η B ⋆ ,
∥ θ t k + 1 − θ t ∥ 2 ≤ ∥ θ t k − θ t ∥ 2 + ∥ θ t k + 1 − θ t k ∥ 2 < 13 4 η B ⋆ . \|\theta_t^{k+1}-\theta_t\|_2\le\|\theta_t^k-\theta_t\|_2+\|\theta_t^{k+1}-\theta_t^k\|_2<\frac{13}{4}\eta B_\star. ∥ θ t k + 1 − θ t ∥ 2 ≤ ∥ θ t k − θ t ∥ 2 + ∥ θ t k + 1 − θ t k ∥ 2 < 4 13 η B ⋆ .
Hence the empirical first-order bound Eq. (230) applies at θ t k \theta_t^k θ t k , and the excess loss concentration bound Eq. (229) applies at θ t k + 1 \theta_t^{k+1} θ t k + 1 .
Proof of 4.1
We now bound the increase in the squared distance in Eq. (245) . Expanding the gradient update and using Eq. (230) and empirical descent from Lemma J.7 gives
∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 = − 2 α ⟨ ∇ L ^ t K i m i ( π t k ) , θ t k − θ t − η θ ⋆ ⟩ + α 2 ∥ ∇ L ^ t K i m i ( π t k ) ∥ 2 2 ≤ − 2 α [ L ^ t K i m i ( π t k ) − L ^ t K i m i ( π η , t ⋆ ) + 31 32 MSE t ( π t k ) − 2 ζ ] + 2 α ( L ^ t K i m i ( π t k ) − L ^ t K i m i ( π t k + 1 ) ) = − 2 α [ L ^ t K i m i ( π t k + 1 ) − L ^ t K i m i ( π η , t ⋆ ) + 31 32 MSE t ( π t k ) − 2 ζ ] . (247) \begin{aligned}
\left\|\theta_t^{k+1}-\theta_t-\eta\theta_\star\right\|_2^2-\left\|\theta_t^k-\theta_t-\eta\theta_\star\right\|_2^2
= -2\alpha\left\langle\nabla\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_t^k\right),\theta_t^k-\theta_t-\eta\theta_\star\right\rangle+\alpha^2\left\|\nabla\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_t^k\right)\right\|_2^2 \\
&\le-2\alpha\left[\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_t^k\right)-\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_{\eta,t}^\star\right)+\frac{31}{32}\operatorname{MSE}_t\left(\pi_t^k\right)-2\zeta\right]+2\alpha\left(\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_t^k\right)-\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_t^{k+1}\right)\right) \\
&= -2\alpha\left[\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_t^{k+1}\right)-\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_{\eta,t}^\star\right)+\frac{31}{32}\operatorname{MSE}_t\left(\pi_t^k\right)-2\zeta\right]. \tag*{(247)}
\end{aligned} θ t k + 1 − θ t − η θ ⋆ 2 2 − θ t k − θ t − η θ ⋆ 2 2 = − 2 α ⟨ ∇ L t Kimi ( π t k ) , θ t k − θ t − η θ ⋆ ⟩ + α 2 ∇ L t Kimi ( π t k ) 2 2 ≤ − 2 α [ L t Kimi ( π t k ) − L t Kimi ( π η , t ⋆ ) + 32 31 MSE t ( π t k ) − 2 ζ ] + 2 α ( L t Kimi ( π t k ) − L t Kimi ( π t k + 1 ) ) = − 2 α [ L t Kimi ( π t k + 1 ) − L t Kimi ( π η , t ⋆ ) + 32 31 MSE t ( π t k ) − 2 ζ ] . (247)
We now lower bound L ^ t K i m i ( π t k + 1 ) − L ^ t K i m i ( π η , t ⋆ ) \widehat{L}_t^{\mathrm{Kimi}}\left(\pi_t^{k+1}\right)-\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_{\eta,t}^\star\right) L t Kimi ( π t k + 1 ) − L t Kimi ( π η , t ⋆ ) in Eq. (247) . Applying Eq. (229) at θ t k + 1 \theta_t^{k+1} θ t k + 1 and then the population lower bound Eq. (236) from the theorem proof gives
L ^ t K i m i ( π t k + 1 ) − L ^ t K i m i ( π η , t ⋆ ) ≥ L t K i m i ( π t k + 1 ) − L t K i m i ( π η , t ⋆ ) − 1 64 MSE t ( π t k + 1 ) − ζ ≥ 63 64 MSE t ( π t k + 1 ) − η 2 D ⋆ 3 e η D ⋆ / 2 8 MSE t ( π t k + 1 ) − ζ ≥ 15 64 MSE t ( π t k + 1 ) − η 4 D ⋆ 6 e η D ⋆ 192 − ζ ≥ − 3 ζ . (248) \begin{aligned}
\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_t^{k+1}\right)-\widehat{L}_t^{\mathrm{Kimi}}\left(\pi_{\eta,t}^\star\right)\ge L_t^{\mathrm{Kimi}}\left(\pi_t^{k+1}\right)-L_t^{\mathrm{Kimi}}\left(\pi_{\eta,t}^\star\right)-\frac{1}{64}\operatorname{MSE}_t\left(\pi_t^{k+1}\right)-\zeta\\
&\ge\frac{63}{64}\operatorname{MSE}_t\left(\pi_t^{k+1}\right)-\frac{\eta^2D_\star^3e^{\eta D_\star/2}}{8}\sqrt{\operatorname{MSE}_t\left(\pi_t^{k+1}\right)}-\zeta\\
&\ge\frac{15}{64}\operatorname{MSE}_t\left(\pi_t^{k+1}\right)-\frac{\eta^4D_\star^6e^{\eta D_\star}}{192}-\zeta\ge-3\zeta. \tag*{(248)}
\end{aligned} L t Kimi ( π t k + 1 ) − L t Kimi ( π η , t ⋆ ) ≥ L t Kimi ( π t k + 1 ) − L t Kimi ( π η , t ⋆ ) − 64 1 MSE t ( π t k + 1 ) − ζ ≥ 64 63 MSE t ( π t k + 1 ) − 8 η 2 D ⋆ 3 e η D ⋆ /2 MSE t ( π t k + 1 ) − ζ ≥ 64 15 MSE t ( π t k + 1 ) − 192 η 4 D ⋆ 6 e η D ⋆ − ζ ≥ − 3 ζ . (248)
The last line uses Young’s inequality, η 4 D ⋆ 6 ≤ ε i n 4 \eta^4D_\star^6\le\varepsilon_{\mathrm{in}}^4 η 4 D ⋆ 6 ≤ ε in 4 , e η D ⋆ < 3 e^{\eta D_\star}<3 e η D ⋆ < 3 , and ζ = ε i n 4 / 128 \zeta=\varepsilon_{\mathrm{in}}^4/128 ζ = ε in 4 /128 . Substituting Eq. (248) into Eq. (247) yields
∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 ≤ ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 + 10 α ζ ≤ η 2 B ⋆ 2 + 10 ( k + 1 ) α ζ . \left\|\theta_t^{k+1}-\theta_t-\eta\theta_\star\right\|_2^2\le\left\|\theta_t^k-\theta_t-\eta\theta_\star\right\|_2^2+10\alpha\zeta\le\eta^2B_\star^2+10(k+1)\alpha\zeta. θ t k + 1 − θ t − η θ ⋆ 2 2 ≤ θ t k − θ t − η θ ⋆ 2 2 + 10 α ζ ≤ η 2 B ⋆ 2 + 10 ( k + 1 ) α ζ .
This completes the induction. Together with Eq. (246) , it gives
max 0 ≤ k ≤ K ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 ≤ η 2 B ⋆ 2 + 10 K α ζ ≤ 101 96 η 2 B ⋆ 2 . \max_{0\le k\le K}\left\|\theta_t^k-\theta_t-\eta\theta_\star\right\|_2^2\le\eta^2B_\star^2+10K\alpha\zeta\le\frac{101}{96}\eta^2B_\star^2. 0 ≤ k ≤ K max θ t k − θ t − η θ ⋆ 2 2 ≤ η 2 B ⋆ 2 + 10 K α ζ ≤ 96 101 η 2 B ⋆ 2 .
For the residual bound, the definition of s t π t k s_t^{\pi_t^k} s t π t k in Eq. (189) gives, under reward linearity,
∣ s t π t k ( x , y ) ∣ = 1 η ∣ ⟨ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] , θ t k − θ t − η θ ⋆ ⟩ ∣ ≤ D ϕ η ∥ θ t k − θ t − η θ ⋆ ∥ 2 ≤ 101 96 D ⋆ . \begin{aligned}
\left|s_t^{\pi_t^k}(x,y)\right|=\frac{1}{\eta}\left|\left\langle\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\phi(x,y')],\theta_t^k-\theta_t-\eta\theta_\star\right\rangle\right| \\
&\le\frac{D_\phi}{\eta}\left\|\theta_t^k-\theta_t-\eta\theta_\star\right\|_2\le\sqrt{\frac{101}{96}}D_\star.
\end{aligned} s t π t k ( x , y ) = η 1 ⟨ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] , θ t k − θ t − η θ ⋆ ⟩ ≤ η D ϕ θ t k − θ t − η θ ⋆ 2 ≤ 96 101 D ⋆ .
The first inequality uses the feature diameter. Taking the supremum over x , y x,y x , y proves the residual bound for every 0 ≤ k ≤ K 0\le k\le K 0 ≤ k ≤ K .
Proof of Corollary 4.1
We combine the inner convergence guarantee in Theorem J.1 with the MSE criterion in Theorem 4.1. The corollary uses G = 1 G=1 G = 1 , so each sampled prompt requires one additional response to estimate the mean baseline.
Corollary J.1 (KimiPO is coverage-preserving). Suppose that Assumption 2.1 holds. Fix ε ∈ ( 0 , R max ] \varepsilon\in(0,R_{\max}] ε ∈ ( 0 , R m a x ] and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . If ε ≥ D ⋆ \varepsilon\ge D_\star ε ≥ D ⋆ , take T = 0 T=0 T = 0 and return π p o s t = π b a s e \pi_{\mathrm{post}}=\pi_{\mathrm{base}} π post = π base without any updates or samples. Otherwise, run Algorithm 3 with π 0 = π b a s e \pi_0=\pi_{\mathrm{base}} π 0 = π base , G = 1 G=1 G = 1 , and
T = ⌈ 2 C b a s e 1 / 4 ( D ⋆ ε ) 3 / 2 log 3 / 2 ( 2 C b a s e ) log 1 / 4 ( e C b a s e D ⋆ / ε ) ⌉ , η = log ( 2 C b a s e ) ε T . (249) T=\left\lceil2C_{\mathrm{base}}^{1/4}\left(\frac{D_\star}{\varepsilon}\right)^{3/2}\log^{3/2}(2C_{\mathrm{base}})\log^{1/4}\left(eC_{\mathrm{base}}D_\star/\varepsilon\right)\right\rceil,\qquad\eta=\frac{\log(2C_{\mathrm{base}})}{\varepsilon T}. \tag*{(249)} T = ⌈ 2 C base 1/4 ( ε D ⋆ ) 3/2 log 3/2 ( 2 C base ) log 1/4 ( e C base D ⋆ / ε ) ⌉ , η = εT log ( 2 C base ) . (249)
Set
ε i n S q = 1 4 η 2 T 2 ( C b a s e log ( e T ) + e η D ⋆ 101 / 96 ) (250) \varepsilon_{\mathrm{in}}^{\mathrm{Sq}}=\frac{1}{4\eta^{2}T^{2}\left(C_{\mathrm{base}}\log(eT)+e^{\eta D_{\star}\sqrt{101/96}}\right)} \tag*{(250)} ε in Sq = 4 η 2 T 2 ( C base log ( e T ) + e η D ⋆ 101/96 ) 1 (250)
α = η 2 D ϕ 2 ( 2 + η D ⋆ ) , K = max { 1 , ⌈ 2 ( 2 + η D ⋆ ) 3 ( D ϕ 2 ε i n S q − 4 ) ⌉ } (251) \alpha=\frac{\eta^{2}}{D_{\phi}^{2}(2+\eta D_{\star})},\qquad K=\max\left\{1,\left\lceil\frac{2(2+\eta D_{\star})}{3}\left(\frac{D_{\phi}^{2}}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}-4\right)\right\rceil\right\} \tag*{(251)} α = D ϕ 2 ( 2 + η D ⋆ ) η 2 , K = max { 1 , ⌈ 3 2 ( 2 + η D ⋆ ) ( ε in Sq D ϕ 2 − 4 ) ⌉ } (251)
and let N N N be the smallest positive integer satisfying
N ≥ C D ⋆ 2 ε i n S q [ d log ( 1 + 2 N ) + log 4 T δ ] , (252) N\geq\frac{CD_{\star}^{2}}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\left[d\log(1+2N)+\log\frac{4T}{\delta}\right], \tag*{(252)} N ≥ ε in Sq C D ⋆ 2 [ d log ( 1 + 2 N ) + log δ 4 T ] , (252)
where C C C is the universal constant in Theorem J.1. With probability at least 1 − δ 1-\delta 1 − δ over the sampled training data, the returned policy π p o s t = π T \pi_{\mathrm{post}}=\pi_T π post = π T satisfies
J ⋆ − J ( π p o s t ) ≤ ε . J^{\star}-J(\pi_{\mathrm{post}})\leq\varepsilon. J ⋆ − J ( π post ) ≤ ε .
On the same event, when ε < D ⋆ \varepsilon<D_{\star} ε < D ⋆ , with probability at least 1 − ( 2 + 2 ) / ( 4 log ( e T ) ) 1-(2+\sqrt{2})/(4\sqrt{\log(eT)}) 1 − ( 2 + 2 ) / ( 4 log ( e T ) ) over x ∼ μ x\sim\mu x ∼ μ ,
π t ( Y x ⋆ ∣ x ) ≥ 1 2 π b a s e ( Y x ⋆ ∣ x ) , 0 ≤ t ≤ T . \pi_t(Y_x^{\star}\mid x)\geq\frac{1}{2}\pi_{\mathrm{base}}(Y_x^{\star}\mid x),\qquad0\leq t\leq T. π t ( Y x ⋆ ∣ x ) ≥ 2 1 π base ( Y x ⋆ ∣ x ) , 0 ≤ t ≤ T .
In this case, the numbers of gradient steps and sampled responses satisfy
N s t e p ( ε , δ ) = K T = O ~ ( C b a s e 5 / 4 D ⋆ 7 / 2 ε 7 / 2 ) , N_{\mathrm{step}}(\varepsilon,\delta)=KT=\widetilde{O}\left(\frac{C_{\mathrm{base}}^{5/4}D_{\star}^{7/2}}{\varepsilon^{7/2}}\right), N step ( ε , δ ) = K T = O ( ε 7/2 C base 5/4 D ⋆ 7/2 ) ,
N s a m p l e ( ε , δ ) = 2 N T = O ~ ( C b a s e 5 / 4 D ⋆ 7 / 2 ε 7 / 2 ( d + log ( 1 / δ ) ) ) . (253) N_{\mathrm{sample}}(\varepsilon,\delta)=2NT=\widetilde{O}\left(\frac{C_{\mathrm{base}}^{5/4}D_{\star}^{7/2}}{\varepsilon^{7/2}}\left(d+\log(1/\delta)\right)\right). \tag*{(253)} N sample ( ε , δ ) = 2 N T = O ( ε 7/2 C base 5/4 D ⋆ 7/2 ( d + log ( 1/ δ ) ) ) . (253)
The outer parameters in Eq. (249) satisfy η = Θ ~ ( ε / ( C b a s e D ⋆ 3 / 2 ) ) \eta=\widetilde{\Theta}\left(\sqrt{\varepsilon}/(C_{\mathrm{base}}D_{\star}^{3/2})\right) η = Θ ( ε / ( C base D ⋆ 3/2 ) ) . Since D ⋆ ≤ 2 B ϕ B ⋆ D_{\star}\leq2B_{\phi}B_{\star} D ⋆ ≤ 2 B ϕ B ⋆ , Corollary J.1 also gives the complexity bounds in Corollary 4.1 with B = B ϕ B ⋆ B=B_{\phi}B_{\star} B = B ϕ B ⋆ .
Proof of Corollary J.1. If ε ≥ D ⋆ \varepsilon\geq D_{\star} ε ≥ D ⋆ , then J ⋆ − J ( π b a s e ) ≤ D ⋆ ≤ ε J^{\star}-J(\pi_{\mathrm{base}})\leq D_{\star}\leq\varepsilon J ⋆ − J ( π base ) ≤ D ⋆ ≤ ε by Eq. (188) . It remains to consider ε < D ⋆ \varepsilon<D_{\star} ε < D ⋆ . We first verify that Theorem J.1 applies at each outer iteration with failure probability δ / T \delta/T δ / T . The choices of α \alpha α , K K K , and N N N in Eqs. (251) and (252) satisfy Eqs. (226) and (227) , so it remains to check η 4 D ⋆ 6 ≤ ε i n S q ≤ D ⋆ 2 \eta^{4}D_{\star}^{6}\leq\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\leq D_{\star}^{2} η 4 D ⋆ 6 ≤ ε in Sq ≤ D ⋆ 2 . The outer parameters in Eq. (249) give
T ≤ 4 C b a s e 1 / 4 ( D ⋆ ε ) 3 / 2 log 3 / 2 ( 2 C b a s e ) log 1 / 4 ( e C b a s e D ⋆ / ε ) , (254) T\leq4C_{\mathrm{base}}^{1/4}\left(\frac{D_{\star}}{\varepsilon}\right)^{3/2}\log^{3/2}(2C_{\mathrm{base}})\log^{1/4}(eC_{\mathrm{base}}D_{\star}/\varepsilon), \tag*{(254)} T ≤ 4 C base 1/4 ( ε D ⋆ ) 3/2 log 3/2 ( 2 C base ) log 1/4 ( e C base D ⋆ / ε ) , (254)
log ( e T ) ≤ 2 log ( e C b a s e D ⋆ / ε ) , η D ⋆ ≤ 1 2 log 2 , e η D ⋆ 101 / 96 < 2. (255) \log(eT)\leq2\log(eC_{\mathrm{base}}D_{\star}/\varepsilon),\qquad\eta D_{\star}\leq\frac{1}{2\sqrt{\log2}},\qquad e^{\eta D_{\star}\sqrt{101/96}}<2. \tag*{(255)} log ( e T ) ≤ 2 log ( e C base D ⋆ / ε ) , η D ⋆ ≤ 2 log 2 1 , e η D ⋆ 101/96 < 2. (255)
Here the bound on T T T uses ⌈ u ⌉ ≤ 2 u \lceil u\rceil\leq2u ⌈ u ⌉ ≤ 2 u for u ≥ 1 u\geq1 u ≥ 1 , and the bound on log ( e T ) \log(eT) log ( e T ) follows from log u ≤ u − 1 \log u\leq u-1 log u ≤ u − 1 . Substituting Eqs. (249) and (255) into the tolerance in Eq. (250) yields
η 4 D ⋆ 6 = log 4 ( 2 C b a s e ) D ⋆ 6 ε 4 T 4 ≤ ε 2 16 C b a s e log 2 ( 2 C b a s e ) log ( e C b a s e D ⋆ / ε ) ≤ ε 2 4 log 2 ( 2 C b a s e ) ( C b a s e log ( e T ) + e η D ⋆ 101 / 96 ) = ε i n S q ≤ ε 2 < D ⋆ 2 . (256) \begin{aligned}
\eta^{4}D_{\star}^{6}=\frac{\log^{4}(2C_{\mathrm{base}})D_{\star}^{6}}{\varepsilon^{4}T^{4}}\leq\frac{\varepsilon^{2}}{16C_{\mathrm{base}}\log^{2}(2C_{\mathrm{base}})\log(eC_{\mathrm{base}}D_{\star}/\varepsilon)} \\
&\leq\frac{\varepsilon^{2}}{4\log^{2}(2C_{\mathrm{base}})\left(C_{\mathrm{base}}\log(eT)+e^{\eta D_{\star}\sqrt{101/96}}\right)}=\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\leq\varepsilon^{2}<D_{\star}^{2}. \tag*{(256)}
\end{aligned} η 4 D ⋆ 6 = ε 4 T 4 log 4 ( 2 C base ) D ⋆ 6 ≤ 16 C base log 2 ( 2 C base ) log ( e C base D ⋆ / ε ) ε 2 ≤ 4 log 2 ( 2 C base ) ( C base log ( e T ) + e η D ⋆ 101/96 ) ε 2 = ε in Sq ≤ ε 2 < D ⋆ 2 . (256)
We may therefore apply Theorem J.1 conditional on the history preceding each outer iteration. A union bound gives, with probability at least 1 − δ 1-\delta 1 − δ over the sampled training data,
MSE t ( π t + 1 ) ≤ ε i n S q , ∥ s π t + 1 ∥ ∞ ≤ 101 96 D ⋆ , 0 ≤ t < T . (257) \operatorname{MSE}_{t}(\pi_{t+1})\leq\varepsilon_{\mathrm{in}}^{\mathrm{Sq}},\qquad\left\|s^{\pi_{t+1}}\right\|_{\infty}\leq\sqrt{\frac{101}{96}}D_{\star},\qquad0\leq t<T. \tag*{(257)} MSE t ( π t + 1 ) ≤ ε in Sq , ∥ s π t + 1 ∥ ∞ ≤ 96 101 D ⋆ , 0 ≤ t < T . (257)
We condition on this event for the remainder of the proof.
We now use Theorem 4.1 to bound J ⋆ − J ( π p o s t ) J^\star-J(\pi_{\mathrm{post}}) J ⋆ − J ( π post ) . By Eqs. (189) and (257) , S = max 0 ≤ t < T ∥ s t + 1 π ∥ ∞ ≤ 101 / 96 D ⋆ S=\max_{0\le t<T}\lVert s_{t+1}^{\pi}\rVert_\infty\le\sqrt{101/96}D_\star S = max 0 ≤ t < T ∥ s t + 1 π ∥ ∞ ≤ 101/96 D ⋆ . The choice of ε i n S q \varepsilon_{\mathrm{in}}^{\mathrm{Sq}} ε in Sq in Eq. (250) therefore gives the required MSE bound,
MSE t ( π t + 1 ) ≤ ε i n S q ≤ 1 4 η T 2 ( C b a s e log ( e T ) + e η S ) , 0 ≤ t < T . (258) \operatorname{MSE}_t(\pi_{t+1})\le\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\le\frac{1}{4\eta T^2(C_{\mathrm{base}}\log(eT)+e^{\eta S})},\qquad0\le t<T. \tag*{(258)} MSE t ( π t + 1 ) ≤ ε in Sq ≤ 4 η T 2 ( C base log ( e T ) + e η S ) 1 , 0 ≤ t < T . (258)
Since π 0 = π b a s e \pi_0=\pi_{\mathrm{base}} π 0 = π base and softmax policies are positive, applying Theorem 4.1 and substituting Eq. (249) gives
J ⋆ − J ( π p o s t ) ≤ log ( 2 C b a s e ) η T = ε . J^\star-J(\pi_{\mathrm{post}})\le\frac{\log(2C_{\mathrm{base}})}{\eta T}=\varepsilon. J ⋆ − J ( π post ) ≤ η T log ( 2 C base ) = ε .
For coverage, we apply Theorem E.1 with c = 1 / 2 c=1/2 c = 1/2 and s t = s t π t + 1 s_t=s_t^{\pi_{t+1}} s t = s t π t + 1 , as defined in Eqs. (78) and (189) . Substituting Eq. (258) into its coverage bound gives
P x ∼ μ ( ∃ 0 ≤ t ≤ T : π t ( Y ⋆ ∣ x ) < 1 2 π b a s e ( Y ⋆ ∣ x ) ) ≤ ( 2 + 2 ) ( e η S 16 T ( C b a s e log ( e T ) + e η S ) + 1 4 C b a s e C b a s e log ( e T ) + e η S ) ≤ 2 + 2 8 log ( e T ) ( e η S C b a s e log ( e T ) + e η S + 1 + C b a s e log ( e T ) C b a s e log ( e T ) + e η S ) = 2 + 2 4 log ( e T ) . \begin{aligned}
\mathbb{P}_{x\sim\mu}\left(\exists\,0\le t\le T:\ \pi_t(Y^\star\mid x)<\frac{1}{2}\pi_{\mathrm{base}}(Y^\star\mid x)\right) \\
&\le(2+\sqrt{2})\left(\frac{e^{\eta S}}{16T(C_{\mathrm{base}}\log(eT)+e^{\eta S})}+\frac{1}{4}\sqrt{\frac{C_{\mathrm{base}}}{C_{\mathrm{base}}\log(eT)+e^{\eta S}}}\right) \\
&\le\frac{2+\sqrt{2}}{8\sqrt{\log(eT)}}\left(\frac{e^{\eta S}}{C_{\mathrm{base}}\log(eT)+e^{\eta S}}+1+\frac{C_{\mathrm{base}}\log(eT)}{C_{\mathrm{base}}\log(eT)+e^{\eta S}}\right)=\frac{2+\sqrt{2}}{4\sqrt{\log(eT)}}.
\end{aligned} P x ∼ μ ( ∃ 0 ≤ t ≤ T : π t ( Y ⋆ ∣ x ) < 2 1 π base ( Y ⋆ ∣ x ) ) ≤ ( 2 + 2 ) ( 16 T ( C base log ( e T ) + e η S ) e η S + 4 1 C base log ( e T ) + e η S C base ) ≤ 8 log ( e T ) 2 + 2 ( C base log ( e T ) + e η S e η S + 1 + C base log ( e T ) + e η S C base log ( e T ) ) = 4 log ( e T ) 2 + 2 .
The second inequality uses u ≤ ( 1 + u ) / 2 \sqrt{u}\le(1+u)/2 u ≤ ( 1 + u ) /2 and log ( e T ) ≤ 2 T \sqrt{\log(eT)}\le2T log ( e T ) ≤ 2 T .
Finally, we bound the numbers of gradient steps and sampled responses. By Eqs. (254) and (256) ,
T = O ~ ( C b a s e 1 / 4 D ⋆ 3 / 2 ε 3 / 2 ) , D ⋆ 2 ε i n S q ≤ 16 C b a s e D ⋆ 2 log 2 ( 2 C b a s e ) log ( e C b a s e D ⋆ / ε ) ε 2 . T=\widetilde{O}\left(\frac{C_{\mathrm{base}}^{1/4}D_\star^{3/2}}{\varepsilon^{3/2}}\right),\qquad
\frac{D_\star^2}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\le\frac{16C_{\mathrm{base}}D_\star^2\log^2(2C_{\mathrm{base}})\log(eC_{\mathrm{base}}D_\star/\varepsilon)}{\varepsilon^2}. T = O ( ε 3/2 C base 1/4 D ⋆ 3/2 ) , ε in Sq D ⋆ 2 ≤ ε 2 16 C base D ⋆ 2 log 2 ( 2 C base ) log ( e C base D ⋆ / ε ) .
The choices of K K K and N N N in Eqs. (251) and (252) then give
K = O ( D ⋆ 2 ε i n S q ) , N = O ~ ( D ⋆ 2 ε i n S q ( d + log ( T / δ ) ) ) . K=O\left(\frac{D_\star^2}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\right),\qquad
N=\widetilde{O}\left(\frac{D_\star^2}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}(d+\log(T/\delta))\right). K = O ( ε in Sq D ⋆ 2 ) , N = O ( ε in Sq D ⋆ 2 ( d + log ( T / δ )) ) .
Each outer iteration uses K K K gradient steps and, since G = 1 G=1 G = 1 , 2 N 2N 2 N sampled responses. Multiplying these bounds by T T T proves Eq. (253) .
Additional Result: KimiPO-Norm with a Normalizer Baseline
This section introduces KimiPO-Norm, which estimates the normalizer baseline b t ⋆ b_t^\star b t ⋆ instead of the mean reward used by KimiPO in § 4.2 and Appendix J. We use the superscript norm to distinguish its objectives from those of KimiPO.
Appendix § 17.1 .§ 1 defines the objectives and algorithm, and states the global convergence guarantee in Corollary K.1.
Appendix § 17.1 .§ 2 bounds the baseline estimation bias and develops the supporting lemmas for the inner optimization loop.
Appendix § 17.1 .§ 3 proves the inner loop convergence guarantee.
Appendix § 17.1 .§ 4 combines this guarantee with Theorem 4.1 to prove Corollary K.1.
We use the notation and mean-squared error from Appendix § 16.1 , and assume the softmax linear parameterization and linear rewards in Assumption 2.1, with r ( x , y ) ∈ [ 0 , R max ] r(x,y)\in[0,R_{\max}] r ( x , y ) ∈ [ 0 , R m a x ] .
Algorithm 4 KimiPO-Norm
Require: Base model π b a s e \pi_{\mathrm{base}} π base , outer iterations T T T , regularization parameter η > 0 \eta> 0 η > 0 , prompt batch size N N N , response group size G G G , inner steps K K K , and learning rate α \alpha α .
Initialize π 0 = π b a s e = π θ 0 \pi_0 = \pi_{\mathrm{base}} = \pi_{\theta_0} π 0 = π base = π θ 0 .
for t = 0 , 1 , … , T − 1 t = 0, 1, \ldots, T - 1 t = 0 , 1 , … , T − 1 do
Draw a batch of N N N prompts x i ∼ i i d μ x_i \mathrel{\overset{\mathrm{iid}}{\sim}} \mu x i ∼ iid μ .
For each prompt x i x_i x i , draw y i , y i , 1 ′ , … , y i , G ′ ∼ π t ( ⋅ ∣ x i ) y_i, y'_{i,1}, \ldots, y'_{i,G} \sim\pi_t(\cdot\mid x_i) y i , y i , 1 ′ , … , y i , G ′ ∼ π t ( ⋅ ∣ x i ) independently.
Set r i = r ( x i , y i ) r_i = r(x_i,y_i) r i = r ( x i , y i ) and estimate the normalizer baseline by
b ^ t , i = 1 η log ( 1 G ∑ j = 1 G e η r ( x i , y i , j ′ ) ) . \widehat{b}_{t,i} = \frac{1}{\eta}\log\left(\frac{1}{G}\sum_{j=1}^{G} e^{\eta r(x_i,y'_{i,j})}\right). b t , i = η 1 log ( G 1 j = 1 ∑ G e η r ( x i , y i , j ′ ) ) .
Initialize θ t 0 = θ t \theta_t^0 = \theta_t θ t 0 = θ t and π t 0 = π t \pi_t^0 = \pi_t π t 0 = π t .
for k = 0 , 1 , … , K − 1 k = 0, 1, \ldots, K - 1 k = 0 , 1 , … , K − 1 do
Compute the empirical gradient (e.g., via backpropagation)
∇ L ^ t n o r m ( π t k ) = 2 η N ∑ i = 1 N ( 1 η log π t k ( y i ∣ x i ) π t ( y i ∣ x i ) − r i + b ^ t , i ) ∇ log π t k ( y i ∣ x i ) . \nabla\widehat{L}^{\mathrm{norm}}_t(\pi_t^k) = \frac{2}{\eta N}\sum_{i=1}^{N}\left(\frac{1}{\eta}\log\frac{\pi_t^k(y_i \mid x_i)}{\pi_t(y_i \mid x_i)} - r_i + \widehat{b}_{t,i}\right)\nabla\log\pi_t^k(y_i \mid x_i). ∇ L t norm ( π t k ) = η N 2 i = 1 ∑ N ( η 1 log π t ( y i ∣ x i ) π t k ( y i ∣ x i ) − r i + b t , i ) ∇ log π t k ( y i ∣ x i ) .
Update θ t k + 1 = θ t k − α ∇ L ^ t n o r m ( π t k ) \theta_t^{k+1} = \theta_t^k - \alpha\nabla\widehat{L}^{\mathrm{norm}}_t(\pi_t^k) θ t k + 1 = θ t k − α ∇ L t norm ( π t k ) and π t k + 1 = π θ t k + 1 \pi_t^{k+1} = \pi_{\theta_t^{k+1}} π t k + 1 = π θ t k + 1 .
Set θ t + 1 = θ t K \theta_{t+1} = \theta_t^K θ t + 1 = θ t K and π t + 1 = π t K \pi_{t+1} = \pi_t^K π t + 1 = π t K .
return π p o s t = π T \pi_{\mathrm{post}} = \pi_T π post = π T .
KimiPO-Norm Algorithm and the Normalizer Baseline
KimiPO-Norm uses the Kimi-style objective with the normalizer baseline b t ∗ ( x ) = η − 1 log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] b_t^*(x) = \eta^{-1}\log\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[e^{\eta r(x,y)}] b t ∗ ( x ) = η − 1 log E y ∼ π t ( ⋅ ∣ x ) [ e η r ( x , y ) ] ,
L t n o r m ( π θ ) : = E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( 1 η log π θ ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + b t ∗ ( x ) ) 2 ] . (259) L_t^{\mathrm{norm}}(\pi_\theta) := \mathbb{E}_{x\sim\mu,\,y\sim\pi_t(\cdot\mid x)}\left[\left(\frac{1}{\eta}\log\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)} - r(x,y) + b_t^*(x)\right)^2\right]. \tag*{(259)} L t norm ( π θ ) := E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) − r ( x , y ) + b t ∗ ( x ) ) 2 ] . (259)
By Eq. (16) , the APMD target π η , t ∗ \pi_{\eta,t}^* π η , t ∗ has zero loss and minimizes Eq. (259) .
As in prior Kimi-style algorithms ([50 , 18 , 89 , 15 ] ), KimiPO-Norm replaces the unknown b t ∗ ( x ) b_t^*(x) b t ∗ ( x ) with a plug-in estimate from G G G additional responses,
b ^ t ( x ) : = 1 η log ( 1 G ∑ j = 1 G e η r ( x , y j ′ ) ) , y 1 ′ , … , y G ′ ∼ i i d π t ( ⋅ ∣ x ) . (260) \widehat{b}_t(x) := \frac{1}{\eta}\log\left(\frac{1}{G}\sum_{j=1}^{G}e^{\eta r(x,y'_j)}\right), \qquad y'_1,\ldots,y'_G \mathrel{\overset{\mathrm{iid}}{\sim}} \pi_t(\cdot\mid x). \tag*{(260)} b t ( x ) := η 1 log ( G 1 j = 1 ∑ G e η r ( x , y j ′ ) ) , y 1 ′ , … , y G ′ ∼ iid π t ( ⋅ ∣ x ) . (260)
The logarithm makes b ^ t ( x ) \widehat{b}_t(x) b t ( x ) biased relative to b t ∗ b_t^* b t ∗ ; in particular, b ^ t ( x ) \widehat{b}_t(x) b t ( x ) underestimates b t ∗ b_t^* b t ∗ in expectation. Lemma K.2 shows that this bias is O ( 1 / G ) O(1/G) O ( 1/ G ) ,
0 ≤ b t ∗ ( x ) − E [ b ^ t ( x ) ∣ x ] ≤ ( e η D ⋆ − 1 ) 2 8 η G , ∀ x ∈ X , (261) 0 \le b_t^*(x) - \mathbb{E}\left[\widehat{b}_t(x)\mid x\right] \le\frac{(e^{\eta D_\star}-1)^2}{8\eta G}, \qquad\forall x\in\mathcal{X}, \tag*{(261)} 0 ≤ b t ∗ ( x ) − E [ b t ( x ) ∣ x ] ≤ 8 η G ( e η D ⋆ − 1 ) 2 , ∀ x ∈ X , (261)
where D ⋆ D_\star D ⋆ bounds the reward difference between responses to the same prompt (Eq. (188) ). The analysis must therefore account for how much the gap between b ^ t \widehat{b}_t b t and b t ∗ b_t^* b t ∗ biases the objective solution away from π η , t ∗ \pi_{\eta,t}^* π η , t ∗ , and controlling this error will require a larger response group size G G G .
Algorithm. At each outer iteration t t t , KimiPO-Norm (Algorithm 4) draws N N N prompts x i ∼ μ x_i \sim\mu x i ∼ μ and G + 1 G+1 G + 1 independent responses from π t \pi_t π t for each prompt (Line 4). It uses G G G responses to form b ^ t , i \widehat{b}_{t,i} b t , i according to Eq. (260) , and the remaining response y i y_i y i , with reward r i = r ( x i , y i ) r_i = r(x_i,y_i) r i = r ( x i , y i ) , to form the empirical objective
L ^ t n o r m ( π θ ) : = 1 N ∑ i = 1 N ( 1 η log π θ ( y i ∣ x i ) π t ( y i ∣ x i ) − r i + b ^ t , i ) 2 . (262) \widehat{L}_t^{\mathrm{norm}}(\pi_\theta) := \frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{\eta}\log\frac{\pi_\theta(y_i\mid x_i)}{\pi_t(y_i\mid x_i)} - r_i + \widehat{b}_{t,i}\right)^2. \tag*{(262)} L t norm ( π θ ) := N 1 i = 1 ∑ N ( η 1 log π t ( y i ∣ x i ) π θ ( y i ∣ x i ) − r i + b t , i ) 2 . (262)
Starting from θ t 0 = θ t \theta_t^0 = \theta_t θ t 0 = θ t , the inner loop takes K K K full-batch gradient descent steps on Eq. (262) with learning rate α \alpha α (Line 9). The batch, baseline estimates, and reference policy π t \pi_t π t remain fixed during these steps. The last inner iterate becomes π t + 1 = π t K \pi_{t+1} = \pi_t^K π t + 1 = π t K , and the procedure repeats for T T T outer iterations.
The following result shows that KimiPO-Norm is coverage-preserving under Assumption 2.1. The proof and exact parameter choices are given in Appendix § 17.1 .§ 4 .
Corollary K.1 (KimiPO-Norm is coverage-preserving). Suppose that Assumption 2.1 holds. For any ε ∈ ( 0 , R max ] \varepsilon\in(0, R_{\max}] ε ∈ ( 0 , R m a x ] and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , run Algorithm 4 with π 0 = π b a s e \pi_0 = \pi_{\mathrm{base}} π 0 = π base and η = 1 / ( 3 R max + 4 D ⋆ ) \eta= 1/(3R_{\max} + 4D_\star) η = 1/ ( 3 R m a x + 4 D ⋆ ) . There exist choices of α \alpha α , G G G , N N N , K K K , and T T T such that the returned policy π p o s t \pi_{\mathrm{post}} π post satisfies
J ⋆ − J ( π p o s t ) ≤ ε J^\star- J(\pi_{\mathrm{post}}) \le\varepsilon J ⋆ − J ( π post ) ≤ ε with probability at least 1 − δ 1-\delta 1 − δ , using
N s t e p ( ε , δ ) = K T = O ~ ( C b a s e D ⋆ 2 η ε 3 ) , N_{\mathrm{step}}(\varepsilon,\delta) = KT = \widetilde{O}\left(\frac{C_{\mathrm{base}}D_\star^2}{\eta\varepsilon^3}\right), N step ( ε , δ ) = K T = O ( η ε 3 C base D ⋆ 2 ) , N s a m p l e ( ε , δ ) = N ( G + 1 ) T = O ~ ( C b a s e D ⋆ 2 ( d + log ( 1 / δ ) ) η ε 3 ( 1 + η C b a s e D ⋆ 2 ε ) ) . N_{\mathrm{sample}}(\varepsilon,\delta) = N(G+1)T = \widetilde{O}\left(\frac{C_{\mathrm{base}}D_\star^2(d+\log(1/\delta))}{\eta\varepsilon^3}\left(1+\frac{\eta\sqrt{C_{\mathrm{base}}D_\star^2}}{\varepsilon}\right)\right). N sample ( ε , δ ) = N ( G + 1 ) T = O ( η ε 3 C base D ⋆ 2 ( d + log ( 1/ δ )) ( 1 + ε η C base D ⋆ 2 ) ) .
Parameter choices. The regularization parameter η \eta η in Corollary K.1 is independent of the target accuracy ε \varepsilon ε . With problem parameters fixed, the choices in the proof satisfy T = Θ ~ ( 1 / ε ) T = \widetilde{\Theta}(1/\varepsilon) T = Θ ( 1/ ε ) and N , K = Θ ~ ( 1 / ε 2 ) N,K = \widetilde{\Theta}(1/\varepsilon^2) N , K = Θ ( 1/ ε 2 ) as ε → 0 \varepsilon\to0 ε → 0 . Thus KimiPO-Norm uses K T = O ~ ( 1 / ε 3 ) KT = \widetilde{O}(1/\varepsilon^3) K T = O ( 1/ ε 3 ) gradient steps, compared with O ~ ( 1 / ε 7 / 2 ) \widetilde{O}(1/\varepsilon^{7/2}) O ( 1/ ε 7/2 ) for KimiPO (Corollary 4.1). The improvement comes from keeping η \eta η fixed, whereas KimiPO decreases η \eta η to control the bias of the mean baseline and consequently needs more outer iterations.
Responses for baseline estimation. To obtain the guarantee in Corollary K.1, we choose
G = Θ ~ ( 1 + η C b a s e D ⋆ 2 ε ) . G = \widetilde{\Theta}\left(1+\frac{\eta\sqrt{C_{\mathrm{base}}D_\star^2}}{\varepsilon}\right). G = Θ ( 1 + ε η C base D ⋆ 2 ) .
This choice makes the squared bias in Eq. (261) small enough to meet the mean-squared error tolerance required by Theorem 4.1. For fixed problem parameters, G = Θ ~ ( 1 / ε ) G = \widetilde{\Theta}(1/\varepsilon) G = Θ ( 1/ ε ) , so the total number of sampled responses is N ( G + 1 ) T = O ~ ( 1 / ε 4 ) N(G+1)T = \widetilde{O}(1/\varepsilon^4) N ( G + 1 ) T = O ( 1/ ε 4 ) . Thus KimiPO-Norm uses fewer gradient steps but more responses than KimiPO, which uses G = 1 G=1 G = 1 and O ~ ( 1 / ε 7 / 2 ) \widetilde{O}(1/\varepsilon^{7/2}) O ( 1/ ε 7/2 ) responses. Table 1 gives the full dependence on the problem parameters.
Supporting Lemmas
We instantiate the supporting lemmas from the general baseline results in Appendix § 16.1 .§ 3 to the normalizer baseline. As an overview,
Lemma K.1 and Lemma K.2 bound the range and bias of the estimated baseline, the latter as a function of G G G .
Lemma K.3 bounds loss of π η , t ⋆ \pi_{\eta,t}^\star π η , t ⋆ on the auxiliary loss L ~ t n o r m \widetilde{L}_t^{\mathrm{norm}} L t norm used in our proofs, defined below.
Notation. We adopt the notation of Appendix § 16.1 .§ 3 . Given G ≥ 1 G \ge1 G ≥ 1 , for the baseline estimate in Eq. (260) and its expectation, respectively, we write
b ^ t G ( x ) = 1 η log ( 1 G ∑ j = 1 G e η r ( x , y j ′ ) ) , b ~ t G ( x ) = E y 1 ′ , … , y G ′ ∼ π t ( ⋅ ∣ x ) [ b ^ t G ( x ) ] . \widehat{b}_t^G(x) = \frac{1}{\eta}\log\left(\frac{1}{G}\sum_{j=1}^{G}e^{\eta r(x,y'_j)}\right), \qquad\widetilde{b}_t^G(x) = \mathbb{E}_{y'_1,\ldots,y'_G\sim\pi_t(\cdot\mid x)}\left[\widehat{b}_t^G(x)\right]. b t G ( x ) = η 1 log ( G 1 j = 1 ∑ G e η r ( x , y j ′ ) ) , b t G ( x ) = E y 1 ′ , … , y G ′ ∼ π t ( ⋅ ∣ x ) [ b t G ( x ) ] .
Recall the empirical KimiPO-Norm objective L ^ t n o r m \widehat{L}_t^{\mathrm{norm}} L t norm from Eq. (262) . Following Eq. (186) , in our proofs we use the auxiliary population objective defined using b ~ t G \widetilde{b}_t^G b t G rather than the true estimation target b t ⋆ b_t^\star b t ⋆ ,
L ~ t n o r m ( π ) : = E π t [ ( 1 η log π ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) + b ~ t G ( x ) ) 2 ] . \widetilde{L}_t^{\mathrm{norm}}(\pi) := \mathbb{E}_{\pi_t}\left[\left(\frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)} - r(x,y) + \widetilde{b}_t^G(x)\right)^2\right]. L t norm ( π ) := E π t [ ( η 1 log π t ( y ∣ x ) π ( y ∣ x ) − r ( x , y ) + b t G ( x ) ) 2 ] .
Below, we state and prove the supporting lemmas for the inner loop convergence result and proof in Appendix § 17.1 .§ 3 .
Lemma K.1. For η > 0 \eta> 0 η > 0 and every integer G ≥ 1 G \ge1 G ≥ 1 , the estimated normalizer baseline satisfies
min y r ( x , y ) ≤ b ^ t G ( x ) ≤ max y r ( x , y ) , x ∈ X . \min_y r(x,y) \le\hat{b}_t^G(x) \le\max_y r(x,y), \qquad x \in\mathcal{X}. y min r ( x , y ) ≤ b ^ t G ( x ) ≤ y max r ( x , y ) , x ∈ X .
Proof of Lemma K.1. For every prompt x x x , each sampled reward lies between min y r ( x , y ) \min_y r(x,y) min y r ( x , y ) and max y r ( x , y ) \max_y r(x,y) max y r ( x , y ) . Hence
e η min y r ( x , y ) ≤ 1 G ∑ j = 1 G e η r ( x , y j ′ ) ≤ e η max y r ( x , y ) . e^{\eta\min_y r(x,y)} \le\frac{1}{G}\sum_{j=1}^{G}e^{\eta r(x,y'_j)} \le e^{\eta\max_y r(x,y)}. e η m i n y r ( x , y ) ≤ G 1 j = 1 ∑ G e η r ( x , y j ′ ) ≤ e η m a x y r ( x , y ) .
The claim follows by taking the logarithm of the above and dividing by η \eta η .
Lemma K.2. Suppose that Assumption 2.1 holds. For η > 0 \eta> 0 η > 0 and every integer G ≥ 1 G \ge1 G ≥ 1 ,
0 ≤ b t ⋆ ( x ) − b ~ t G ( x ) ≤ ( e η D ⋆ − 1 ) 2 8 η G , x ∈ X . 0 \le b_t^\star(x) - \widetilde{b}_t^G(x) \le\frac{(e^{\eta D_\star}-1)^2}{8\eta G}, \qquad x \in\mathcal{X}. 0 ≤ b t ⋆ ( x ) − b t G ( x ) ≤ 8 η G ( e η D ⋆ − 1 ) 2 , x ∈ X .
Proof of Lemma K.2 . Fix x x x , with expectations over the group of G G G additional responses sampled for x x x . Subtracting min y r ( x , y ) \min_y r(x,y) min y r ( x , y ) from the rewards shifts b t ⋆ ( x ) b_t^\star(x) b t ⋆ ( x ) , b ^ t G ( x ) \hat{b}_t^G(x) b ^ t G ( x ) , and b ~ t G ( x ) \widetilde{b}_t^G(x) b t G ( x ) by the same amount, so the bias is unchanged. By Assumption 2.1, the shifted rewards lie in [ 0 , D ⋆ ] [0,D_\star] [ 0 , D ⋆ ] . We use the same notation for these shifted quantities.
From the definition of b ^ t G ( x ) \hat{b}_t^G(x) b ^ t G ( x ) (Eq. (260) ), we have E π t [ e η b ^ t G ( x ) ] = e η b t ⋆ ( x ) \mathbb{E}_{\pi_t}\left[e^{\eta\hat{b}_t^G(x)}\right] = e^{\eta b_t^\star(x)} E π t [ e η b ^ t G ( x ) ] = e η b t ⋆ ( x ) . Jensen’s inequality therefore yields
b ~ t G ( x ) = E π t [ b ^ t G ( x ) ] ≤ 1 η log E π t [ e η b ^ t G ( x ) ] = b t ⋆ ( x ) , \widetilde{b}_t^G(x) = \mathbb{E}_{\pi_t}\left[\hat{b}_t^G(x)\right] \le\frac{1}{\eta}\log\mathbb{E}_{\pi_t}\left[e^{\eta\hat{b}_t^G(x)}\right] = b_t^\star(x), b t G ( x ) = E π t [ b ^ t G ( x ) ] ≤ η 1 log E π t [ e η b ^ t G ( x ) ] = b t ⋆ ( x ) ,
proving the lower bound in the lemma statement.
Next, both e η b ^ t G ( x ) e^{\eta\hat{b}_t^G(x)} e η b ^ t G ( x ) and e η b t ⋆ ( x ) e^{\eta b_t^\star(x)} e η b t ⋆ ( x ) lie in [ 1 , e η D ⋆ ] [1,e^{\eta D_\star}] [ 1 , e η D ⋆ ] by Lemma K.1 and the identity above. Since the second derivative of − log -\log − log is at most 1 on this interval, Taylor’s theorem around e η b t ⋆ ( x ) e^{\eta b_t^\star(x)} e η b t ⋆ ( x ) gives
η ( b t ⋆ ( x ) − b ^ t G ( x ) ) ≤ 1 − e η ( b ^ t G ( x ) − b t ⋆ ( x ) ) + 1 2 ( e η b ^ t G ( x ) − e η b t ⋆ ( x ) ) 2 . \eta\left(b_t^\star(x)-\hat{b}_t^G(x)\right) \le1-e^{\eta(\hat{b}_t^G(x)-b_t^\star(x))}+\frac{1}{2}\left(e^{\eta\hat{b}_t^G(x)}-e^{\eta b_t^\star(x)}\right)^2. η ( b t ⋆ ( x ) − b ^ t G ( x ) ) ≤ 1 − e η ( b ^ t G ( x ) − b t ⋆ ( x )) + 2 1 ( e η b ^ t G ( x ) − e η b t ⋆ ( x ) ) 2 .
Taking expectations and again using the fact that E π t [ e η b t ⋆ ( x ) ] = e η b t ⋆ ( x ) \mathbb{E}_{\pi_t}[e^{\eta b_t^\star(x)}] = e^{\eta b_t^\star(x)} E π t [ e η b t ⋆ ( x ) ] = e η b t ⋆ ( x ) removes the linear term. Since b ~ t G ( x ) = E π t [ b ^ t G ( x ) ] \widetilde{b}_t^G(x) = \mathbb{E}_{\pi_t}[\hat{b}_t^G(x)] b t G ( x ) = E π t [ b ^ t G ( x )] , we obtain
η ( b t ⋆ ( x ) − b ~ t G ( x ) ) ≤ 1 2 E π t [ ( e η b ^ t G ( x ) − e η b t ⋆ ( x ) ) 2 ] = 1 2 V a r π t [ e η b ^ t G ( x ) ] . (263) \eta\left(b_t^\star(x)-\widetilde{b}_t^G(x)\right) \le\frac{1}{2}\mathbb{E}_{\pi_t}\left[\left(e^{\eta\hat{b}_t^G(x)}-e^{\eta b_t^\star(x)}\right)^2\right] = \frac{1}{2}\mathop{\mathrm{Var}}_{\pi_t}\left[e^{\eta\hat{b}_t^G(x)}\right]. \tag*{(263)} η ( b t ⋆ ( x ) − b t G ( x ) ) ≤ 2 1 E π t [ ( e η b ^ t G ( x ) − e η b t ⋆ ( x ) ) 2 ] = 2 1 Var π t [ e η b ^ t G ( x ) ] . (263)
Finally, e η b ^ t G ( x ) e^{\eta\hat{b}_t^G(x)} e η b ^ t G ( x ) is the average of G G G independent copies of e η r ( x , y ) e^{\eta r(x,y)} e η r ( x , y ) sampled from π t ( ⋅ ∣ x ) \pi_t(\cdot\mid x) π t ( ⋅ ∣ x ) . By independence,
V a r π t [ e η b ^ t G ( x ) ] = 1 G V a r π t [ e η r ( x , y ) ] . \mathop{\mathrm{Var}}_{\pi_t}\left[e^{\eta\hat{b}_t^G(x)}\right] = \frac{1}{G}\mathop{\mathrm{Var}}_{\pi_t}\left[e^{\eta r(x,y)}\right]. Var π t [ e η b ^ t G ( x ) ] = G 1 Var π t [ e η r ( x , y ) ] .
Since the mean minimizes expected squared distance, using the midpoint of [ 1 , e η D ⋆ ] [1,e^{\eta D_\star}] [ 1 , e η D ⋆ ] we obtain
V a r π t [ e η r ( x , y ) ] ≤ E π t [ ( e η r ( x , y ) − 1 + e η D ⋆ 2 ) 2 ] ≤ ( e η D ⋆ − 1 ) 2 4 . \mathop{\mathrm{Var}}_{\pi_t}\left[e^{\eta r(x,y)}\right] \le\mathbb{E}_{\pi_t}\left[\left(e^{\eta r(x,y)}-\frac{1+e^{\eta D_\star}}{2}\right)^2\right] \le\frac{(e^{\eta D_\star}-1)^2}{4}. Var π t [ e η r ( x , y ) ] ≤ E π t [ ( e η r ( x , y ) − 2 1 + e η D ⋆ ) 2 ] ≤ 4 ( e η D ⋆ − 1 ) 2 .
The last inequality holds because every value of e η r ( x , y ) e^{\eta r(x,y)} e η r ( x , y ) lies within ( e η D ⋆ − 1 ) / 2 (e^{\eta D_\star}-1)/2 ( e η D ⋆ − 1 ) /2 of this midpoint. Hence V a r π t [ e η b ^ t G ( x ) ] ≤ ( e η D ⋆ − 1 ) 2 / ( 4 G ) \mathop{\mathrm{Var}}_{\pi_t}\left[e^{\eta\hat{b}_t^G(x)}\right] \le(e^{\eta D_\star}-1)^2/(4G) Var π t [ e η b ^ t G ( x ) ] ≤ ( e η D ⋆ − 1 ) 2 / ( 4 G ) . Substituting this variance bound into Eq. (263) and dividing by η \eta η proves the upper bound.
Lemma K.3. Suppose that Assumption 2.1 holds. For η > 0 \eta> 0 η > 0 and every integer G ≥ 1 G \ge1 G ≥ 1 ,
L ~ t n o r m ( π η , t ⋆ ) ≤ ( e η D ⋆ − 1 ) 4 64 η 2 G 2 . \widetilde{L}^{\mathrm{norm}}_t(\pi^\star_{\eta,t}) \le\frac{(e^{\eta D_\star}-1)^4}{64\eta^2G^2}. L t norm ( π η , t ⋆ ) ≤ 64 η 2 G 2 ( e η D ⋆ − 1 ) 4 .
Proof of Lemma K.3. Applying Eq. (192) in Lemma I.1 to the auxiliary objective L ~ t n o r m \widetilde{L}^{\mathrm{norm}}_t L t norm gives
L ~ t n o r m ( π η , t ⋆ ) = E x ∼ μ [ ( b t ⋆ ( x ) − b ~ t G ( x ) ) 2 ] . (264) \widetilde{L}^{\mathrm{norm}}_t(\pi^\star_{\eta,t}) = \mathbb{E}_{x\sim\mu}\left[\left(b^\star_t(x)-\widetilde{b}^{G}_t(x)\right)^2\right]. \tag*{(264)} L t norm ( π η , t ⋆ ) = E x ∼ μ [ ( b t ⋆ ( x ) − b t G ( x ) ) 2 ] . (264)
By Lemma K.2, for every x ∈ X x \in\mathcal{X} x ∈ X ,
0 ≤ b t ⋆ ( x ) − b ~ t G ( x ) ≤ ( e η D ⋆ − 1 ) 2 8 η G . 0 \le b^\star_t(x)-\widetilde{b}^{G}_t(x) \le\frac{(e^{\eta D_\star}-1)^2}{8\eta G}. 0 ≤ b t ⋆ ( x ) − b t G ( x ) ≤ 8 η G ( e η D ⋆ − 1 ) 2 .
Squaring the above inequality and taking expectation over x ∼ μ x \sim\mu x ∼ μ yields
E x ∼ μ [ ( b t ⋆ ( x ) − b ~ t G ( x ) ) 2 ] ≤ ( ( e η D ⋆ − 1 ) 2 8 η G ) 2 = ( e η D ⋆ − 1 ) 4 64 η 2 G 2 . \mathbb{E}_{x\sim\mu}\left[\left(b^\star_t(x)-\widetilde{b}^{G}_t(x)\right)^2\right] \le\left(\frac{(e^{\eta D_\star}-1)^2}{8\eta G}\right)^2 = \frac{(e^{\eta D_\star}-1)^4}{64\eta^2G^2}. E x ∼ μ [ ( b t ⋆ ( x ) − b t G ( x ) ) 2 ] ≤ ( 8 η G ( e η D ⋆ − 1 ) 2 ) 2 = 64 η 2 G 2 ( e η D ⋆ − 1 ) 4 .
Substituting the above into Eq. (264) proves the claim. □
Inner Loop Convergence
Using the supporting lemmas in Appendix § 17.1 .§ 2 with the general supporting lemmas in Appendix § 16.1 .§ 3 , instantiated for the normalizer baseline, we now prove convergence of the inner optimization loop of KimiPO-Norm (Algorithm 4).
Theorem K.1 (Inner loop convergence of KimiPO-Norm). Suppose that Assumption 2.1 holds. Fix an outer iteration 0 ≤ t < T 0 \le t < T 0 ≤ t < T , η ∈ ( 0 , 1 / ( 3 D ⋆ ) ] \eta\in(0,1/(3D_\star)] η ∈ ( 0 , 1/ ( 3 D ⋆ )] , and ε i n S q ∈ ( 0 , D ⋆ 2 ] \varepsilon^{\mathrm{Sq}}_{\mathrm{in}} \in(0,D_\star^2] ε in Sq ∈ ( 0 , D ⋆ 2 ] . Let the response group size satisfy
G ≥ ( e η D ⋆ − 1 ) 2 4 η ε i n S q . (265) G \ge\frac{(e^{\eta D_\star}-1)^2}{4\eta\sqrt{\varepsilon^{\mathrm{Sq}}_{\mathrm{in}}}}. \tag*{(265)} G ≥ 4 η ε in Sq ( e η D ⋆ − 1 ) 2 . (265) There is a universal constant C > 0 C > 0 C > 0 such that, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , if the inner loop hyperparameters satisfy
N ≥ C D ⋆ 2 ε i n S q ( d log ( 1 + 8 N ) + log 4 ( d + 1 ) δ ) , K = ⌈ η 2 B ⋆ 2 α ε i n S q ⌉ , 0 < α ≤ η 2 8 D ϕ 2 , (266) N \ge\frac{CD_\star^2}{\varepsilon^{\mathrm{Sq}}_{\mathrm{in}}}\left(d\log(1+8N)+\log\frac{4(d+1)}{\delta}\right), \qquad K = \left\lceil\frac{\eta^2B_\star^2}{\alpha\varepsilon^{\mathrm{Sq}}_{\mathrm{in}}}\right\rceil, \qquad0 < \alpha\le\frac{\eta^2}{8D_\phi^2}, \tag*{(266)} N ≥ ε in Sq C D ⋆ 2 ( d log ( 1 + 8 N ) + log δ 4 ( d + 1 ) ) , K = ⌈ α ε in Sq η 2 B ⋆ 2 ⌉ , 0 < α ≤ 8 D ϕ 2 η 2 , (266) then the resulting policy π t + 1 = π t K \pi_{t+1}=\pi_t^K π t + 1 = π t K satisfies, with probability at least 1 − δ 1-\delta 1 − δ ,
MSE t ( π t + 1 ) ≤ ε i n S q , ∥ s t π t + 1 ∥ ∞ ≤ 2 D ⋆ . (267) \operatorname{MSE}_t(\pi_{t+1}) \le\varepsilon^{\mathrm{Sq}}_{\mathrm{in}}, \qquad\lVert s_t^{\pi_{t+1}}\rVert_\infty\le2D_\star. \tag*{(267)} MSE t ( π t + 1 ) ≤ ε in Sq , ∥ s t π t + 1 ∥ ∞ ≤ 2 D ⋆ . (267)
The inner convergence complexity and the results in Eq. (267) can then be combined with Theorem 4.1 to establish global convergence, which we proceed to do in Appendix § 17.1 .§ 4 .
The proof below proceeds as follows. First, we show jointly that throughout the inner loop, the parameters θ t k \theta_t^k θ t k stay within a ball of constant radius; moreover, the empirical objective L ^ t n o r m ( π t k ) \widehat{L}^{\mathrm{norm}}_t(\pi_t^k) L t norm ( π t k ) decreases (Lemma K.4). Using this, we prove that π t K \pi_t^K π t K converges on L ^ t n o r m \widehat{L}^{\mathrm{norm}}_t L t norm (Eq. (275) ), which we then transfer to the auxiliary population objective L ~ t n o r m \widetilde{L}^{\mathrm{norm}}_t L t norm (Eq. (278) ) and finally to MSE t \operatorname{MSE}_t MSE t (Eq. (281) ).
Proof of Theorem K.1
Proof of Theorem K.1. By Lemma K.1 and the reward bound in Eq. (188) ,
∣ b ^ t G ( x ) − r ( x , y ) ∣ ≤ max y ′ r ( x , y ′ ) − min y ′ r ( x , y ′ ) ≤ D ⋆ , x ∈ X , y ∈ Y . (268) \left|\widehat{b}^{G}_{t}(x)-r(x,y)\right| \le\max_{y'} r(x,y')-\min_{y'} r(x,y') \le D_{\star}, \qquad x\in\mathcal{X},\ y\in\mathcal{Y}. \tag*{(268)} b t G ( x ) − r ( x , y ) ≤ y ′ max r ( x , y ′ ) − y ′ min r ( x , y ′ ) ≤ D ⋆ , x ∈ X , y ∈ Y . (268)
Applying Lemma I.6 with A = D ⋆ A=D_{\star} A = D ⋆ and R = 4 η B ⋆ R=4\eta B_{\star} R = 4 η B ⋆ , with probability at least 1 − δ 1-\delta 1 − δ , the following inequalities hold simultaneously for every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ 4 η B ⋆ \lVert\theta-\theta_t\rVert_2\le4\eta B_{\star} ∥ θ − θ t ∥ 2 ≤ 4 η B ⋆ ,
∣ ⟨ ∇ L ^ t n o r m ( π θ ) − ∇ ~ L t n o r m ( π θ ) , θ − θ t − η θ ⋆ ⟩ ∣ ≤ 1 64 MSE t ( π θ ) + ζ N ( δ ; 4 η B ⋆ , D ⋆ ) , \left|\left\langle\nabla\widehat{L}^{\mathrm{norm}}_t(\pi_\theta)-\widetilde{\nabla}L^{\mathrm{norm}}_t(\pi_\theta),\theta-\theta_t-\eta\theta^{\star}\right\rangle\right|\le\frac{1}{64}\operatorname{MSE}_t(\pi_\theta)+\zeta_N(\delta;4\eta B_{\star},D_{\star}), ⟨ ∇ L t norm ( π θ ) − ∇ L t norm ( π θ ) , θ − θ t − η θ ⋆ ⟩ ≤ 64 1 MSE t ( π θ ) + ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ,
∣ L ^ t n o r m ( π θ ) − L ^ t n o r m ( π η , t ⋆ ) − L ~ t n o r m ( π θ ) + L ~ t n o r m ( π η , t ⋆ ) ∣ ≤ 1 64 MSE t ( π θ ) + ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (269) \left|\widehat{L}^{\mathrm{norm}}_t(\pi_\theta)-\widehat{L}^{\mathrm{norm}}_t(\pi^{\star}_{\eta,t})-\widetilde{L}^{\mathrm{norm}}_t(\pi_\theta)+\widetilde{L}^{\mathrm{norm}}_t(\pi^{\star}_{\eta,t})\right|\le\frac{1}{64}\operatorname{MSE}_t(\pi_\theta)+\zeta_N(\delta;4\eta B_{\star},D_{\star}). \tag*{(269)} L t norm ( π θ ) − L t norm ( π η , t ⋆ ) − L t norm ( π θ ) + L t norm ( π η , t ⋆ ) ≤ 64 1 MSE t ( π θ ) + ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (269)
Here ζ N \zeta_N ζ N is the statistical error defined in Eq. (202) .
For the remainder of this proof we condition on Eq. (269) holding. By Lemmas J.2 and K.2,
b ~ t G ( x ) − r ( x , π t ) ≤ b t ⋆ ( x ) − r ( x , π t ) ≤ η D ⋆ 2 8 , x ∈ X . \widetilde{b}^{G}_{t}(x)-r(x,\pi_t)\le b^{\star}_{t}(x)-r(x,\pi_t)\le\frac{\eta D_{\star}^{2}}{8}, \qquad x\in\mathcal{X}. b t G ( x ) − r ( x , π t ) ≤ b t ⋆ ( x ) − r ( x , π t ) ≤ 8 η D ⋆ 2 , x ∈ X .
We apply Eq. (212) in Lemma I.7 with A = D ⋆ A=D_{\star} A = D ⋆ , a = η D ⋆ 2 / 8 a=\eta D_{\star}^{2}/8 a = η D ⋆ 2 /8 , and R = 4 η B ⋆ R=4\eta B_{\star} R = 4 η B ⋆ . Then simultaneously for every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ 4 η B ⋆ \lVert\theta-\theta_t\rVert_2\le4\eta B_{\star} ∥ θ − θ t ∥ 2 ≤ 4 η B ⋆ ,
⟨ ∇ L ^ t n o r m ( π θ ) , θ − θ t − η θ ⋆ ⟩ ≥ L ^ t n o r m ( π θ ) − L ^ t n o r m ( π η , t ⋆ ) + ( 31 32 − e 4 / 3 72 ) MSE t ( π θ ) − 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≥ L ^ t n o r m ( π θ ) − L ^ t n o r m ( π η , t ⋆ ) − 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (270) \begin{aligned}
\left\langle\nabla\widehat{L}^{\mathrm{norm}}_t(\pi_\theta),\theta-\theta_t-\eta\theta^{\star}\right\rangle\ge\widehat{L}^{\mathrm{norm}}_t(\pi_\theta)-\widehat{L}^{\mathrm{norm}}_t(\pi^{\star}_{\eta,t})+\left(\frac{31}{32}-\frac{e^{4/3}}{72}\right)\operatorname{MSE}_t(\pi_\theta)-2\zeta_N(\delta;4\eta B_{\star},D_{\star}) \\
&\ge\widehat{L}^{\mathrm{norm}}_t(\pi_\theta)-\widehat{L}^{\mathrm{norm}}_t(\pi^{\star}_{\eta,t})-2\zeta_N(\delta;4\eta B_{\star},D_{\star}). \tag*{(270)}
\end{aligned} ⟨ ∇ L t norm ( π θ ) , θ − θ t − η θ ⋆ ⟩ ≥ L t norm ( π θ ) − L t norm ( π η , t ⋆ ) + ( 32 31 − 72 e 4/3 ) MSE t ( π θ ) − 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≥ L t norm ( π θ ) − L t norm ( π η , t ⋆ ) − 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (270)
Under the above inequalities, we have the following lemma, which is proved at the end of this section. It says that, throughout the inner loop, the inner policy iterates stay within a parameter ball of constant radius, and have monotonically decreasing empirical loss L ^ t n o r m \widehat{L}^{\mathrm{norm}}_t L t norm .
Lemma K.4 (Localization and empirical descent for the normalizer baseline). Under the conditions on η \eta η , G G G , N N N , α \alpha α , and K K K in Theorem K.1, the concentration event Eq. (269) implies
max 0 ≤ k ≤ K ∥ θ t k − θ t η − θ ⋆ ∥ 2 ≤ 2 B ⋆ , max 0 ≤ k ≤ K ∥ π t k ∥ S t ≤ 2 D ⋆ . (271) \max_{0\le k\le K}\left\lVert\frac{\theta_t^k-\theta_t}{\eta}-\theta^{\star}\right\rVert_2\le2B_{\star}, \qquad\max_{0\le k\le K}\left\lVert\pi_t^k\right\rVert_{\mathcal{S}_t}\le2D_{\star}. \tag*{(271)} 0 ≤ k ≤ K max η θ t k − θ t − θ ⋆ 2 ≤ 2 B ⋆ , 0 ≤ k ≤ K max π t k S t ≤ 2 D ⋆ . (271) Moreover, for every 0 ≤ k < K 0\le k<K 0 ≤ k < K ,
L ^ t n o r m ( π t k ) − L ^ t n o r m ( π t k + 1 ) ≥ α 2 ∥ ∇ L ^ t n o r m ( π t k ) ∥ 2 2 . (272) \widehat{L}^{\mathrm{norm}}_t(\pi_t^k)-\widehat{L}^{\mathrm{norm}}_t(\pi_t^{k+1})\ge\frac{\alpha}{2}\left\lVert\nabla\widehat{L}^{\mathrm{norm}}_t(\pi_t^k)\right\rVert_2^2. \tag*{(272)} L t norm ( π t k ) − L t norm ( π t k + 1 ) ≥ 2 α ∇ L t norm ( π t k ) 2 2 . (272)
The parameter bound in Eq. (271) of Lemma K.4 therefore gives
∥ θ t k − θ t ∥ 2 ≤ 3 η B ⋆ , 0 ≤ k ≤ K . \left\lVert\theta_t^k-\theta_t\right\rVert_2\le3\eta B_{\star}, \qquad0\le k\le K. θ t k − θ t 2 ≤ 3 η B ⋆ , 0 ≤ k ≤ K .
Convergence on empirical loss. Using this and Lemma K.4, we next prove that π t k \pi_t^k π t k converges on the empirical loss L ^ t n o r m \widehat{L}^{\mathrm{norm}}_t L t norm . Applying the empirical first-order convexity bound in Eq. (270) at θ t k \theta_t^k θ t k , then expanding the squared distance after the gradient update, gives, for every 0 ≤ k < K 0\le k<K 0 ≤ k < K ,
L ^ t n o r m ( π t k ) − L ^ t n o r m ( π η , t ⋆ ) ≤ ⟨ ∇ L ^ t n o r m ( π t k ) , θ t k − θ t − η θ ⋆ ⟩ + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) = ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 2 α + α 2 ∥ ∇ L ~ t n o r m ( π t k ) ∥ 2 2 + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (273) \begin{aligned}
\widehat{L}^{\mathrm{norm}}_t(\pi_t^k)-\widehat{L}^{\mathrm{norm}}_t(\pi^{\star}_{\eta,t})\le\left\langle\nabla\widehat{L}^{\mathrm{norm}}_t(\pi_t^k),\theta_t^k-\theta_t-\eta\theta^{\star}\right\rangle+2\zeta_N(\delta;4\eta B_{\star},D_{\star}) \\
&=\frac{\left\lVert\theta_t^k-\theta_t-\eta\theta^{\star}\right\rVert_2^2-\left\lVert\theta_t^{k+1}-\theta_t-\eta\theta^{\star}\right\rVert_2^2}{2\alpha}+\frac{\alpha}{2}\left\lVert\nabla\widetilde{L}^{\mathrm{norm}}_t(\pi_t^k)\right\rVert_2^2+2\zeta_N(\delta;4\eta B_{\star},D_{\star}). \tag*{(273)}
\end{aligned} L t norm ( π t k ) − L t norm ( π η , t ⋆ ) ≤ ⟨ ∇ L t norm ( π t k ) , θ t k − θ t − η θ ⋆ ⟩ + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) = 2 α θ t k − θ t − η θ ⋆ 2 2 − θ t k + 1 − θ t − η θ ⋆ 2 2 + 2 α ∇ L t norm ( π t k ) 2 2 + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (273)
Using this result, below we bound the excess empirical loss at π t k + 1 \pi_t^{k+1} π t k + 1 in terms of the difference in parameter norm between π t k \pi_t^k π t k and π t k + 1 \pi_t^{k+1} π t k + 1 .
L ^ t n o r m ( π t k + 1 ) − L ^ t n o r m ( π η , t ⋆ ) = L ^ t n o r m ( π t k ) − L ^ t n o r m ( π η , t ⋆ ) − ( L ^ t n o r m ( π t k ) − L ^ t n o r m ( π t k + 1 ) ) ≤ L ^ t n o r m ( π t k ) − L ^ t n o r m ( π η , t ⋆ ) − α 2 ∥ ∇ L ^ t n o r m ( π t k ) ∥ 2 2 ≤ ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 2 α + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (274) \begin{aligned}
\widehat{L}_t^{\mathrm{norm}}(\pi_t^{k+1})-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)=\widehat{L}_t^{\mathrm{norm}}(\pi_t^k)-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)-\left(\widehat{L}_t^{\mathrm{norm}}(\pi_t^k)-\widehat{L}_t^{\mathrm{norm}}(\pi_t^{k+1})\right) \\
&\leq\widehat{L}_t^{\mathrm{norm}}(\pi_t^k)-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)-\frac{\alpha}{2}\left\|\nabla\widehat{L}_t^{\mathrm{norm}}(\pi_t^k)\right\|_2^2 \\
&\leq\frac{\left\|\theta_t^k-\theta_t-\eta\theta_\star\right\|_2^2-\left\|\theta_t^{k+1}-\theta_t-\eta\theta_\star\right\|_2^2}{2\alpha}+2\zeta_N(\delta;4\eta B_\star,D_\star). \tag*{(274)}
\end{aligned} L t norm ( π t k + 1 ) − L t norm ( π η , t ⋆ ) = L t norm ( π t k ) − L t norm ( π η , t ⋆ ) − ( L t norm ( π t k ) − L t norm ( π t k + 1 ) ) ≤ L t norm ( π t k ) − L t norm ( π η , t ⋆ ) − 2 α ∇ L t norm ( π t k ) 2 2 ≤ 2 α θ t k − θ t − η θ ⋆ 2 2 − θ t k + 1 − θ t − η θ ⋆ 2 2 + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (274)
The second line uses Eq. (272) from Lemma K.4 , and we substitute in Eq. (273) to obtain the last line.
We now invoke Eq. (274) to prove that π t K \pi_t^K π t K has small excess empirical loss,
L ^ t n o r m ( π t K ) − L ^ t n o r m ( π η , t ⋆ ) ≤ 1 K ∑ k = 0 K − 1 ( L ^ t n o r m ( π t k + 1 ) − L ^ t n o r m ( π η , t ⋆ ) ) ≤ 1 2 α K ∑ k = 0 K − 1 [ ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 ] + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) = ∥ θ t 0 − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t K − θ t − η θ ⋆ ∥ 2 2 2 α K + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . \begin{aligned}
\widehat{L}_t^{\mathrm{norm}}(\pi_t^K)-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\leq\frac{1}{K}\sum_{k=0}^{K-1}\left(\widehat{L}_t^{\mathrm{norm}}(\pi_t^{k+1})-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\right) \\
&\leq\frac{1}{2\alpha K}\sum_{k=0}^{K-1}\left[\left\|\theta_t^k-\theta_t-\eta\theta_\star\right\|_2^2-\left\|\theta_t^{k+1}-\theta_t-\eta\theta_\star\right\|_2^2\right]+2\zeta_N(\delta;4\eta B_\star,D_\star) \\
&=\frac{\left\|\theta_t^0-\theta_t-\eta\theta_\star\right\|_2^2-\left\|\theta_t^K-\theta_t-\eta\theta_\star\right\|_2^2}{2\alpha K}+2\zeta_N(\delta;4\eta B_\star,D_\star).
\end{aligned} L t norm ( π t K ) − L t norm ( π η , t ⋆ ) ≤ K 1 k = 0 ∑ K − 1 ( L t norm ( π t k + 1 ) − L t norm ( π η , t ⋆ ) ) ≤ 2 α K 1 k = 0 ∑ K − 1 [ θ t k − θ t − η θ ⋆ 2 2 − θ t k + 1 − θ t − η θ ⋆ 2 2 ] + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) = 2 α K θ t 0 − θ t − η θ ⋆ 2 2 − θ t K − θ t − η θ ⋆ 2 2 + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) .
The first line uses L ^ t n o r m ( π t k ) − L ^ t n o r m ( π t k + 1 ) ≥ 0 \widehat{L}_t^{\mathrm{norm}}(\pi_t^k)-\widehat{L}_t^{\mathrm{norm}}(\pi_t^{k+1})\geq0 L t norm ( π t k ) − L t norm ( π t k + 1 ) ≥ 0 from Eq. (272) in Lemma K.4 , and the second line uses Eq. (274) . Finally, using θ t 0 = θ t \theta_t^0=\theta_t θ t 0 = θ t and ∥ θ ⋆ ∥ 2 = B ⋆ \|\theta_\star\|_2=B_\star ∥ θ ⋆ ∥ 2 = B ⋆ , we obtain
L ^ t n o r m ( π t K ) − L ^ t n o r m ( π η , t ⋆ ) ≤ ∥ θ t 0 − θ t − η θ ⋆ ∥ 2 2 2 α K + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ η 2 B ⋆ 2 2 α K + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (275) \begin{aligned}
\widehat{L}_t^{\mathrm{norm}}(\pi_t^K)-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\leq\frac{\left\|\theta_t^0-\theta_t-\eta\theta_\star\right\|_2^2}{2\alpha K}+2\zeta_N(\delta;4\eta B_\star,D_\star) \\
&\leq\frac{\eta^2B_\star^2}{2\alpha K}+2\zeta_N(\delta;4\eta B_\star,D_\star). \tag*{(275)}
\end{aligned} L t norm ( π t K ) − L t norm ( π η , t ⋆ ) ≤ 2 α K θ t 0 − θ t − η θ ⋆ 2 2 + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ 2 α K η 2 B ⋆ 2 + 2 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (275)
Small mean-squared error from empirical loss. We now use Eq. (275) to bound MSE t ( π t K ) \operatorname{MSE}_t(\pi_t^K) MSE t ( π t K ) . By Eq. (191) in Lemma I.1,
MSE t ( π t K ) ≤ L ~ t n o r m ( π t K ) . (276) \operatorname{MSE}_t(\pi_t^K)\leq\widetilde{L}_t^{\mathrm{norm}}(\pi_t^K). \tag*{(276)} MSE t ( π t K ) ≤ L t norm ( π t K ) . (276)
It therefore suffices to bound L ~ t n o r m ( π t K ) \widetilde{L}_t^{\mathrm{norm}}(\pi_t^K) L t norm ( π t K ) through L ^ t n o r m ( π t K ) \widehat{L}_t^{\mathrm{norm}}(\pi_t^K) L t norm ( π t K ) . Applying the second inequality in Eq. (269) , which relates L ~ t n o r m \widetilde{L}_t^{\mathrm{norm}} L t norm to L ^ t n o r m \widehat{L}_t^{\mathrm{norm}} L t norm , at θ t K \theta_t^K θ t K and then substituting the empirical convergence result in Eq. (275) gives
L ~ t n o r m ( π t K ) − L ~ t n o r m ( π η , t ⋆ ) ≤ L ^ t n o r m ( π t K ) − L ^ t n o r m ( π η , t ⋆ ) + 1 64 MSE t ( π t K ) + ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ η 2 B ⋆ 2 2 α K + 1 64 MSE t ( π t K ) + 3 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ η 2 B ⋆ 2 2 α K + 1 64 L ~ t n o r m ( π t K ) + 3 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (277) \begin{aligned}
\widetilde{L}_t^{\mathrm{norm}}(\pi_t^K)-\widetilde{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\leq\widehat{L}_t^{\mathrm{norm}}(\pi_t^K)-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)+\frac{1}{64}\operatorname{MSE}_t(\pi_t^K)+\zeta_N(\delta;4\eta B_\star,D_\star) \\
&\leq\frac{\eta^2B_\star^2}{2\alpha K}+\frac{1}{64}\operatorname{MSE}_t(\pi_t^K)+3\zeta_N(\delta;4\eta B_\star,D_\star) \\
&\leq\frac{\eta^2B_\star^2}{2\alpha K}+\frac{1}{64}\widetilde{L}_t^{\mathrm{norm}}(\pi_t^K)+3\zeta_N(\delta;4\eta B_\star,D_\star). \tag*{(277)}
\end{aligned} L t norm ( π t K ) − L t norm ( π η , t ⋆ ) ≤ L t norm ( π t K ) − L t norm ( π η , t ⋆ ) + 64 1 MSE t ( π t K ) + ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ 2 α K η 2 B ⋆ 2 + 64 1 MSE t ( π t K ) + 3 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ 2 α K η 2 B ⋆ 2 + 64 1 L t norm ( π t K ) + 3 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (277)
where we use Eq. (276) in the last line above. Rearranging Eq. (277) , we obtain
63 64 L ~ t n o r m ( π t K ) ≤ η 2 B ⋆ 2 2 α K + L ~ t n o r m ( π η , t ⋆ ) + 3 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ ε i n s q 2 + L ~ t n o r m ( π η , t ⋆ ) + 3 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (278) \begin{aligned}
\frac{63}{64}\widetilde{L}_t^{\mathrm{norm}}(\pi_t^K)\leq\frac{\eta^2B_\star^2}{2\alpha K}+\widetilde{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)+3\zeta_N(\delta;4\eta B_\star,D_\star) \\
&\leq\frac{\varepsilon_{\mathrm{in}}^{\mathrm{sq}}}{2}+\widetilde{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)+3\zeta_N(\delta;4\eta B_\star,D_\star). \tag*{(278)}
\end{aligned} 64 63 L t norm ( π t K ) ≤ 2 α K η 2 B ⋆ 2 + L t norm ( π η , t ⋆ ) + 3 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ 2 ε in sq + L t norm ( π η , t ⋆ ) + 3 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (278)
The last line above uses the choice of K K K in Eq. (266) , which gives η 2 B ⋆ 2 / ( 2 α K ) ≤ ε i n s q / 2 \eta^2B_\star^2/(2\alpha K)\leq\varepsilon_{\mathrm{in}}^{\mathrm{sq}}/2 η 2 B ⋆ 2 / ( 2 α K ) ≤ ε in sq /2 . We bound the remaining terms on the RHS of Eq. (278) . Invoking Lemma K.3 , which upper bounds L ~ t n o r m ( π η , t ⋆ ) \widetilde{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star) L t norm ( π η , t ⋆ ) in terms of G G G , and combining with the theorem condition on G G G (Eq. (265) ), we obtain
L ~ t n o r m ( π η , t ⋆ ) ≤ ( e η D ⋆ − 1 ) 4 64 η 2 G 2 ≤ ε i n s q 4 . (279) \widetilde{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\leq\frac{(e^{\eta D_\star}-1)^4}{64\eta^2G^2}\leq\frac{\varepsilon_{\mathrm{in}}^{\mathrm{sq}}}{4}. \tag*{(279)} L t norm ( π η , t ⋆ ) ≤ 64 η 2 G 2 ( e η D ⋆ − 1 ) 4 ≤ 4 ε in sq . (279)
Finally, since η D ⋆ ≤ 1 / 3 \eta D_{\star} \le1/3 η D ⋆ ≤ 1/3 , the definition of ζ N \zeta_N ζ N in Eq. (202) at R = 4 η B ⋆ R = 4\eta B_{\star} R = 4 η B ⋆ and the bound on N N N in Eq. (266) give
ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ ε i n S q 128 (280) \zeta_N(\delta;4\eta B_{\star},D_{\star}) \le\frac{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{128} \tag*{(280)} ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ 128 ε in Sq (280)
for sufficiently large C C C . Substituting the previous two panels into Eq. (278) yields
MSE t ( π t K ) ≤ L ~ t n o r m ( π t K ) ≤ 64 63 ( 1 2 + 1 4 + 3 128 ) ε i n S q ≤ ε i n S q . (281) \operatorname{MSE}_t(\pi_t^K) \le\widetilde{L}_t^{\mathrm{norm}}(\pi_t^K) \le\frac{64}{63}\left(\frac{1}{2}+\frac{1}{4}+\frac{3}{128}\right)\varepsilon_{\mathrm{in}}^{\mathrm{Sq}} \le\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}. \tag*{(281)} MSE t ( π t K ) ≤ L t norm ( π t K ) ≤ 63 64 ( 2 1 + 4 1 + 128 3 ) ε in Sq ≤ ε in Sq . (281)
This proves the first result in Eq. (267) of Theorem K.1.
Finally, Eq. (271) from Lemma K.4 directly gives ∥ s t π t + 1 ∥ ∞ = ∥ s t π t K ∥ ∞ < 2 D ⋆ \lVert s_t^{\pi_{t+1}}\rVert_{\infty}=\lVert s_t^{\pi_t^K}\rVert_{\infty}<2D_{\star} ∥ s t π t + 1 ∥ ∞ = ∥ s t π t K ∥ ∞ < 2 D ⋆ , the other result in Theorem K.1. This completes the proof.
Proof of Lemma K.4
Proof of Lemma K.4. Throughout this proof, we condition on the event that the concentration inequalities in Eq. (269) hold.
Under this event, we will prove by induction that all policy iterates π t k \pi_t^k π t k satisfy Eq. (272) and stay within the parameter ball given by Eq. (271) . Initially, θ t 0 = θ t \theta_t^0=\theta_t θ t 0 = θ t and ∥ θ t 0 − θ t − η θ ⋆ ⋆ ∥ 2 2 = η 2 B ⋆ 2 \lVert\theta_t^0-\theta_t-\eta\theta_{\star}^{\star}\rVert_2^2=\eta^2B_{\star}^2 ∥ θ t 0 − θ t − η θ ⋆ ⋆ ∥ 2 2 = η 2 B ⋆ 2 . Suppose ∥ θ t j − θ t − η θ ⋆ ⋆ ∥ 2 ≤ 2 η B ⋆ \lVert\theta_t^j-\theta_t-\eta\theta_{\star}^{\star}\rVert_2\le2\eta B_{\star} ∥ θ t j − θ t − η θ ⋆ ⋆ ∥ 2 ≤ 2 η B ⋆ for every 0 ≤ j ≤ k 0\le j\le k 0 ≤ j ≤ k , where k < K k<K k < K , which implies that ∥ θ t k − θ t ∥ 2 ≤ 3 η B ⋆ \lVert\theta_t^k-\theta_t\rVert_2\le3\eta B_{\star} ∥ θ t k − θ t ∥ 2 ≤ 3 η B ⋆ . We will then show for the k + 1 k+1 k + 1 th inner step that ∥ θ t k + 1 − θ t − η θ ⋆ ⋆ ∥ 2 ≤ 2 η B ⋆ \lVert\theta_t^{k+1}-\theta_t-\eta\theta_{\star}^{\star}\rVert_2\le2\eta B_{\star} ∥ θ t k + 1 − θ t − η θ ⋆ ⋆ ∥ 2 ≤ 2 η B ⋆ and that L ~ t n o r m ( π t k ) − L ~ t n o r m ( π t k + 1 ) ≥ α 2 ∥ ∇ L ~ t n o r m ( π t k ) ∥ 2 2 \widetilde{L}_t^{\mathrm{norm}}(\pi_t^k)-\widetilde{L}_t^{\mathrm{norm}}(\pi_t^{k+1})\ge\frac{\alpha}{2}\lVert\nabla\widetilde{L}_t^{\mathrm{norm}}(\pi_t^k)\rVert_2^2 L t norm ( π t k ) − L t norm ( π t k + 1 ) ≥ 2 α ∥ ∇ L t norm ( π t k ) ∥ 2 2 .
Proving empirical descent (Eq. (272) ). We first prove the latter statement. The estimator bound Eq. (268) allows us to apply Eq. (195) in Lemma I.5 with A = D ⋆ A=D_{\star} A = D ⋆ and R = 3 η B ⋆ R=3\eta B_{\star} R = 3 η B ⋆ , giving
∥ ∇ L ~ t n o r m ( π t k ) ∥ 2 ≤ 2 D ϕ η 2 ( 3 η D ⋆ + η D ⋆ ) = 8 D ϕ D ⋆ η . \left\lVert\nabla\widetilde{L}_t^{\mathrm{norm}}(\pi_t^k)\right\rVert_2\le\frac{2D_{\phi}}{\eta^2}(3\eta D_{\star}+\eta D_{\star})=\frac{8D_{\phi}D_{\star}}{\eta}. ∇ L t norm ( π t k ) 2 ≤ η 2 2 D ϕ ( 3 η D ⋆ + η D ⋆ ) = η 8 D ϕ D ⋆ .
Then, for the update in Line 9 of Algorithm 4, using the above inequalities we obtain
∥ θ t k + 1 − θ t ∥ 2 = ∥ θ t k − α ∇ L ~ t n o r m ( π t k ) − θ t ∥ 2 ≤ ∥ θ t k − θ t ∥ 2 + α ∥ ∇ L ~ t n o r m ( π t k ) ∥ 2 ≤ 3 η B ⋆ + 8 α D ϕ D ⋆ η ≤ 4 η B ⋆ , (282) \begin{aligned}
\lVert\theta_t^{k+1}-\theta_t\rVert_2
&=\left\lVert\theta_t^k-\alpha\nabla\widetilde{L}_t^{\mathrm{norm}}(\pi_t^k)-\theta_t\right\rVert_2\\
&\le\lVert\theta_t^k-\theta_t\rVert_2+\alpha\left\lVert\nabla\widetilde{L}_t^{\mathrm{norm}}(\pi_t^k)\right\rVert_2\\
&\le3\eta B_{\star}+\frac{8\alpha D_{\phi}D_{\star}}{\eta}\\
&\le4\eta B_{\star}, \tag*{(282)}
\end{aligned} ∥ θ t k + 1 − θ t ∥ 2 = θ t k − α ∇ L t norm ( π t k ) − θ t 2 ≤ ∥ θ t k − θ t ∥ 2 + α ∇ L t norm ( π t k ) 2 ≤ 3 η B ⋆ + η 8 α D ϕ D ⋆ ≤ 4 η B ⋆ , (282)
where in the last inequality we use the theorem condition on α \alpha α (Eq. (266) ).
We now use Taylor’s theorem to bound L ~ t n o r m ( π t k ) − L ~ t n o r m ( π t k + 1 ) \widetilde{L}_t^{\mathrm{norm}}(\pi_t^k)-\widetilde{L}_t^{\mathrm{norm}}(\pi_t^{k+1}) L t norm ( π t k ) − L t norm ( π t k + 1 ) . The inductive hypothesis and Eq. (282) give
∥ θ t k − θ t ∥ 2 ≤ 4 η B ⋆ , ∥ θ t k + 1 − θ t ∥ 2 ≤ 4 η B ⋆ . \lVert\theta_t^k-\theta_t\rVert_2\le4\eta B_{\star},\qquad\lVert\theta_t^{k+1}-\theta_t\rVert_2\le4\eta B_{\star}. ∥ θ t k − θ t ∥ 2 ≤ 4 η B ⋆ , ∥ θ t k + 1 − θ t ∥ 2 ≤ 4 η B ⋆ .
We again invoke Lemma I.5, but with A = D ⋆ A=D_{\star} A = D ⋆ and R = 4 η B ⋆ R=4\eta B_{\star} R = 4 η B ⋆ , and apply the upper bound on the Hessian in Eq. (196) . Since η D ⋆ ≤ 1 / 3 \eta D_{\star}\le1/3 η D ⋆ ≤ 1/3 , for every θ \theta θ with ∥ θ − θ t ∥ 2 ≤ 4 η B ⋆ \lVert\theta-\theta_t\rVert_2\le4\eta B_{\star} ∥ θ − θ t ∥ 2 ≤ 4 η B ⋆ , this gives,
∥ ∇ 2 L ~ t n o r m ( π θ ) ∥ o p ≤ D ϕ 2 η 2 max { 2 , 1 + 5 η D ⋆ 2 } = 2 D ϕ 2 η 2 . (283) \left\lVert\nabla^2\widetilde{L}_t^{\mathrm{norm}}(\pi_\theta)\right\rVert_{\mathrm{op}}\le\frac{D_{\phi}^2}{\eta^2}\max\left\{2,1+\frac{5\eta D_{\star}}{2}\right\}=\frac{2D_{\phi}^2}{\eta^2}. \tag*{(283)} ∇ 2 L t norm ( π θ ) op ≤ η 2 D ϕ 2 max { 2 , 1 + 2 5 η D ⋆ } = η 2 2 D ϕ 2 . (283)
By convexity of ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 , the Hessian bound in Eq. (283) holds for convex combinations of θ t k \theta_t^k θ t k and θ t k + 1 \theta_t^{k+1} θ t k + 1 . Applying Taylor’s theorem at θ t k \theta_t^k θ t k , we obtain
L ~ t n o r m ( π t k ) − L ~ t n o r m ( π t k + 1 ) ≥ − ⟨ ∇ L ~ t n o r m ( π t k ) , θ t k + 1 − θ t k ⟩ − D ϕ 2 η 2 ∥ θ t k + 1 − θ t k ∥ 2 2 = α ( 1 − α D ϕ 2 η 2 ) ∥ ∇ L ~ t n o r m ( π t k ) ∥ 2 2 ≥ α 2 ∥ ∇ L ~ t n o r m ( π t k ) ∥ 2 2 . (284) \begin{aligned}
\widetilde{L}_t^{\mathrm{norm}}(\pi_t^k)-\widetilde{L}_t^{\mathrm{norm}}(\pi_t^{k+1})
&\ge-\left\langle\nabla\widetilde{L}_t^{\mathrm{norm}}(\pi_t^k),\theta_t^{k+1}-\theta_t^k\right\rangle-\frac{D_{\phi}^2}{\eta^2}\lVert\theta_t^{k+1}-\theta_t^k\rVert_2^2\\
&=\alpha\left(1-\frac{\alpha D_{\phi}^2}{\eta^2}\right)\left\lVert\nabla\widetilde{L}_t^{\mathrm{norm}}(\pi_t^k)\right\rVert_2^2\\
&\ge\frac{\alpha}{2}\left\lVert\nabla\widetilde{L}_t^{\mathrm{norm}}(\pi_t^k)\right\rVert_2^2. \tag*{(284)}
\end{aligned} L t norm ( π t k ) − L t norm ( π t k + 1 ) ≥ − ⟨ ∇ L t norm ( π t k ) , θ t k + 1 − θ t k ⟩ − η 2 D ϕ 2 ∥ θ t k + 1 − θ t k ∥ 2 2 = α ( 1 − η 2 α D ϕ 2 ) ∇ L t norm ( π t k ) 2 2 ≥ 2 α ∇ L t norm ( π t k ) 2 2 . (284)
The second line uses θ t k + 1 − θ t k = − α ∇ L ^ t n o r m ( π t k ) \theta_t^{k+1}-\theta_t^k=-\alpha\nabla\widehat{L}_t^{\mathrm{norm}}(\pi_t^k) θ t k + 1 − θ t k = − α ∇ L t norm ( π t k ) from Line 9 of Algorithm 4. The last inequality uses α ≤ η 2 / ( 8 D ϕ 2 ) \alpha\le\eta^2/(8D_\phi^2) α ≤ η 2 / ( 8 D ϕ 2 ) from the theorem conditions in Eq. (266) . This completes the proof of Eq. (272) at step k k k .
Proving the parameter bound (Eq. (271) ). It remains to prove ∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 ≤ 2 η B ⋆ \lVert\theta_t^{k+1}-\theta_t-\eta\theta_\star\rVert_2\le2\eta B_\star ∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 ≤ 2 η B ⋆ . The inductive hypothesis gives ∥ θ t k − θ t ∥ 2 ≤ 3 η B ⋆ \lVert\theta_t^k-\theta_t\rVert_2\le3\eta B_\star ∥ θ t k − θ t ∥ 2 ≤ 3 η B ⋆ . Expanding the squared distance after the gradient update and applying the empirical first-order convexity bound in Eq. (270) at θ t k \theta_t^k θ t k , we obtain
∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 ≤ − 2 α ( L ^ t n o r m ( π t k ) − L ^ t n o r m ( π η , t ⋆ ) ) + α 2 ∥ ∇ L ^ t n o r m ( π t k ) ∥ 2 2 + 4 α ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (285) \lVert\theta_t^{k+1}-\theta_t-\eta\theta_\star\rVert_2^2-\lVert\theta_t^k-\theta_t-\eta\theta_\star\rVert_2^2\le-2\alpha\left(\widehat{L}_t^{\mathrm{norm}}(\pi_t^k)-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\right)+\alpha^2\left\lVert\nabla\widehat{L}_t^{\mathrm{norm}}(\pi_t^k)\right\rVert_2^2+4\alpha\zeta_N(\delta;4\eta B_\star,D_\star). \tag*{(285)} ∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 ≤ − 2 α ( L t norm ( π t k ) − L t norm ( π η , t ⋆ ) ) + α 2 ∇ L t norm ( π t k ) 2 2 + 4 α ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (285)
We now use Eq. (284) to bound α 2 ∥ ∇ L ^ t n o r m ( π t k ) ∥ 2 2 \alpha^2\left\lVert\nabla\widehat{L}_t^{\mathrm{norm}}(\pi_t^k)\right\rVert_2^2 α 2 ∇ L t norm ( π t k ) 2 2 in Eq. (285) , yielding
∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 ≤ − 2 α ( L ^ t n o r m ( π t k ) − L ^ t n o r m ( π η , t ⋆ ) ) + 2 α ( L ^ t n o r m ( π t k ) − L ^ t n o r m ( π t k + 1 ) ) + 4 α ζ N ( δ ; 4 η B ⋆ , D ⋆ ) = − 2 α ( L ^ t n o r m ( π t k + 1 ) − L ^ t n o r m ( π η , t ⋆ ) ) + 4 α ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (286) \begin{aligned}
\lVert\theta_t^{k+1}-\theta_t-\eta\theta_\star\rVert_2^2-\lVert\theta_t^k-\theta_t-\eta\theta_\star\rVert_2^2
&\le-2\alpha\left(\widehat{L}_t^{\mathrm{norm}}(\pi_t^k)-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\right) \\
&\quad+2\alpha\left(\widehat{L}_t^{\mathrm{norm}}(\pi_t^k)-\widehat{L}_t^{\mathrm{norm}}(\pi_t^{k+1})\right)+4\alpha\zeta_N(\delta;4\eta B_\star,D_\star) \\
&= -2\alpha\left(\widehat{L}_t^{\mathrm{norm}}(\pi_t^{k+1})-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\right)+4\alpha\zeta_N(\delta;4\eta B_\star,D_\star). \tag*{(286)}
\end{aligned} ∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 ≤ − 2 α ( L t norm ( π t k ) − L t norm ( π η , t ⋆ ) ) + 2 α ( L t norm ( π t k ) − L t norm ( π t k + 1 ) ) + 4 α ζ N ( δ ; 4 η B ⋆ , D ⋆ ) = − 2 α ( L t norm ( π t k + 1 ) − L t norm ( π η , t ⋆ ) ) + 4 α ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (286)
Rearranging Eq. (286) gives
∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 ≤ ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 − 2 α ( L ^ t n o r m ( π t k + 1 ) − L ^ t n o r m ( π η , t ⋆ ) ) + 4 α ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (287) \lVert\theta_t^{k+1}-\theta_t-\eta\theta_\star\rVert_2^2\le\lVert\theta_t^k-\theta_t-\eta\theta_\star\rVert_2^2-2\alpha\left(\widehat{L}_t^{\mathrm{norm}}(\pi_t^{k+1})-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\right)+4\alpha\zeta_N(\delta;4\eta B_\star,D_\star). \tag*{(287)} ∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 ≤ ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 − 2 α ( L t norm ( π t k + 1 ) − L t norm ( π η , t ⋆ ) ) + 4 α ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (287)
We now lower bound L ^ t n o r m ( π t k + 1 ) − L ^ t n o r m ( π η , t ⋆ ) \widehat{L}_t^{\mathrm{norm}}(\pi_t^{k+1})-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star) L t norm ( π t k + 1 ) − L t norm ( π η , t ⋆ ) in Eq. (287) . Since ∥ θ t k + 1 − θ t ∥ 2 ≤ 4 η B ⋆ \lVert\theta_t^{k+1}-\theta_t\rVert_2\le4\eta B_\star ∥ θ t k + 1 − θ t ∥ 2 ≤ 4 η B ⋆ by Eq. (282) , the second concentration inequality in Eq. (269) applies at θ t k + 1 \theta_t^{k+1} θ t k + 1 , giving
L ^ t n o r m ( π t k + 1 ) − L ^ t n o r m ( π η , t ⋆ ) ≥ L ~ t n o r m ( π t k + 1 ) − L ~ t n o r m ( π η , t ⋆ ) − 1 64 MSE t ( π t k + 1 ) − ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≥ 63 64 L ~ t n o r m ( π t k + 1 ) − L ~ t n o r m ( π η , t ⋆ ) − ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≥ − L ~ t n o r m ( π η , t ⋆ ) − ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (288) \begin{aligned}
\widehat{L}_t^{\mathrm{norm}}(\pi_t^{k+1})-\widehat{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)
&\ge\widetilde{L}_t^{\mathrm{norm}}(\pi_t^{k+1})-\widetilde{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)-\frac{1}{64}\operatorname{MSE}_t(\pi_t^{k+1})-\zeta_N(\delta;4\eta B_\star,D_\star) \\
&\ge\frac{63}{64}\widetilde{L}_t^{\mathrm{norm}}(\pi_t^{k+1})-\widetilde{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)-\zeta_N(\delta;4\eta B_\star,D_\star) \\
&\ge-\widetilde{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)-\zeta_N(\delta;4\eta B_\star,D_\star). \tag*{(288)}
\end{aligned} L t norm ( π t k + 1 ) − L t norm ( π η , t ⋆ ) ≥ L t norm ( π t k + 1 ) − L t norm ( π η , t ⋆ ) − 64 1 MSE t ( π t k + 1 ) − ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≥ 64 63 L t norm ( π t k + 1 ) − L t norm ( π η , t ⋆ ) − ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≥ − L t norm ( π η , t ⋆ ) − ζ N ( δ ; 4 η B ⋆ , D ⋆ ) . (288)
The second inequality uses MSE t ( π t k + 1 ) ≤ L ~ t n o r m ( π t k + 1 ) \operatorname{MSE}_t(\pi_t^{k+1})\le\widetilde{L}_t^{\mathrm{norm}}(\pi_t^{k+1}) MSE t ( π t k + 1 ) ≤ L t norm ( π t k + 1 ) from Eq. (276) . Substituting Eq. (288) into Eq. (287) now gives
∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 ≤ ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 + 2 α ( L ~ t n o r m ( π η , t ⋆ ) + 3 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ) . (289) \lVert\theta_t^{k+1}-\theta_t-\eta\theta_\star\rVert_2^2\le\lVert\theta_t^k-\theta_t-\eta\theta_\star\rVert_2^2+2\alpha\left(\widetilde{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)+3\zeta_N(\delta;4\eta B_\star,D_\star)\right). \tag*{(289)} ∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 ≤ ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 + 2 α ( L t norm ( π η , t ⋆ ) + 3 ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ) . (289)
By Eqs. (279) and (280) in the proof of Theorem K.1,
L ~ t n o r m ( π η , t ⋆ ) ≤ ε i n S q 4 , ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ ε i n S q 128 . \widetilde{L}_t^{\mathrm{norm}}(\pi_{\eta,t}^\star)\le\frac{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{4},\qquad\zeta_N(\delta;4\eta B_\star,D_\star)\le\frac{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{128}. L t norm ( π η , t ⋆ ) ≤ 4 ε in Sq , ζ N ( δ ; 4 η B ⋆ , D ⋆ ) ≤ 128 ε in Sq .
Substituting these bounds into Eq. (289) yields
∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 ≤ ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 + 2 α ( ε i n S q 4 + 3 ε i n S q 128 ) = ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 + 35 α ε i n S q 64 . (290) \begin{aligned}
\lVert\theta_t^{k+1}-\theta_t-\eta\theta_\star\rVert_2^2
&\le\lVert\theta_t^k-\theta_t-\eta\theta_\star\rVert_2^2+2\alpha\left(\frac{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{4}+\frac{3\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{128}\right) \\
&=\lVert\theta_t^k-\theta_t-\eta\theta_\star\rVert_2^2+\frac{35\alpha\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{64}. \tag*{(290)}
\end{aligned} ∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 ≤ ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 + 2 α ( 4 ε in Sq + 128 3 ε in Sq ) = ∥ θ t k − θ t − η θ ⋆ ∥ 2 2 + 64 35 α ε in Sq . (290)
The choices of K K K and α \alpha α in Eq. (266) and ε i n S q ≤ D ⋆ 2 \varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\le D_\star^2 ε in Sq ≤ D ⋆ 2 give
α K ε i n S q ≤ η 2 B ⋆ 2 + ε i n S q ≤ 9 8 η 2 B ⋆ 2 . (291) \alpha K\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\le\eta^2B_\star^2+\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\le\frac{9}{8}\eta^2B_\star^2. \tag*{(291)} α K ε in Sq ≤ η 2 B ⋆ 2 + ε in Sq ≤ 8 9 η 2 B ⋆ 2 . (291)
Proof of K.1
Because the inductive hypothesis assumes ∥ θ t j − θ t − η θ ⋆ ∥ 2 ≤ 2 η B ⋆ \left\|\theta_t^j-\theta_t-\eta\theta_\star\right\|_2\leq2\eta B_\star θ t j − θ t − η θ ⋆ 2 ≤ 2 η B ⋆ for every 0 ≤ j ≤ k 0\leq j\leq k 0 ≤ j ≤ k , Eq. (290) also holds for every 0 ≤ j ≤ k 0\leq j\leq k 0 ≤ j ≤ k . Then summing Eq. (290) over these steps gives
∥ θ t k + 1 − θ t − η θ ⋆ ∥ 2 2 = ∥ θ t 0 − θ t − η θ ⋆ ∥ 2 2 + ∑ j = 0 k [ ∥ θ t j + 1 − θ t − η θ ⋆ ∥ 2 2 − ∥ θ t j − θ t − η θ ⋆ ∥ 2 2 ] ≤ η 2 B ⋆ 2 + 35 α ( k + 1 ) ε i n S q 64 ≤ η 2 B ⋆ 2 + 35 α K ε i n S q 64 < 4 η 2 B ⋆ 2 . \begin{aligned}
\left\|\theta_t^{k+1}-\theta_t-\eta\theta_\star\right\|_2^2
&=\left\|\theta_t^0-\theta_t-\eta\theta_\star\right\|_2^2+\sum_{j=0}^{k}\left[\left\|\theta_t^{j+1}-\theta_t-\eta\theta_\star\right\|_2^2-\left\|\theta_t^j-\theta_t-\eta\theta_\star\right\|_2^2\right]\\
&\leq\eta^2B_\star^2+\frac{35\alpha(k+1)\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{64}\\
&\leq\eta^2B_\star^2+\frac{35\alpha K\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{64}<4\eta^2B_\star^2.
\end{aligned} θ t k + 1 − θ t − η θ ⋆ 2 2 = θ t 0 − θ t − η θ ⋆ 2 2 + j = 0 ∑ k [ θ t j + 1 − θ t − η θ ⋆ 2 2 − θ t j − θ t − η θ ⋆ 2 2 ] ≤ η 2 B ⋆ 2 + 64 35 α ( k + 1 ) ε in Sq ≤ η 2 B ⋆ 2 + 64 35 α K ε in Sq < 4 η 2 B ⋆ 2 .
The first inequality uses Eq. (290) and θ t 0 = θ t \theta_t^0=\theta_t θ t 0 = θ t , and the last line uses Eq. (291) . This completes the induction and proves the parameter bound in Eq. (271) ,
max 0 ≤ k ≤ K ∥ θ t k − θ t η − θ ⋆ ∥ 2 ≤ 2 B ⋆ . (292) \max_{0\leq k\leq K}\left\|\frac{\theta_t^k-\theta_t}{\eta}-\theta_\star\right\|_2\leq2B_\star. \tag*{(292)} 0 ≤ k ≤ K max η θ t k − θ t − θ ⋆ 2 ≤ 2 B ⋆ . (292)
Finally, we use Eq. (292) to prove ∥ s t π t k ∥ ∞ ≤ 2 D ⋆ \left\|s_t^{\pi_t^k}\right\|_\infty\leq2D_\star s t π t k ∞ ≤ 2 D ⋆ . The definition of s t π t k s_t^{\pi_t^k} s t π t k in Eq. (189) and Assumption 2.1 give, for every 0 ≤ k ≤ K 0\leq k\leq K 0 ≤ k ≤ K ,
s t π t k ( x , y ) = 1 η log π t k ( y ∣ x ) π t ( y ∣ x ) − r ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ 1 η log π t k ( y ′ ∣ x ) π t ( y ′ ∣ x ) − r ( x , y ′ ) ] = ⟨ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] , θ t k − θ t η − θ ⋆ ⟩ . \begin{aligned}
s_t^{\pi_t^k}(x,y)
&=\frac{1}{\eta}\log\frac{\pi_t^k(y\mid x)}{\pi_t(y\mid x)}-r(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[\frac{1}{\eta}\log\frac{\pi_t^k(y'\mid x)}{\pi_t(y'\mid x)}-r(x,y')\right]\\
&=\left\langle\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\phi(x,y')],\frac{\theta_t^k-\theta_t}{\eta}-\theta_\star\right\rangle.
\end{aligned} s t π t k ( x , y ) = η 1 log π t ( y ∣ x ) π t k ( y ∣ x ) − r ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ η 1 log π t ( y ′ ∣ x ) π t k ( y ′ ∣ x ) − r ( x , y ′ ) ] = ⟨ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] , η θ t k − θ t − θ ⋆ ⟩ .
By the definition of D ϕ D_\phi D ϕ , ∥ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] ∥ 2 ≤ D ϕ \left\|\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\phi(x,y')]\right\|_2\leq D_\phi ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] 2 ≤ D ϕ . Cauchy–Schwarz and Eq. (292) therefore give
∥ s t π t k ∥ ∞ ≤ D ϕ ∥ θ t k − θ t η − θ ⋆ ∥ 2 ≤ 2 D ⋆ , 0 ≤ k ≤ K . \left\|s_t^{\pi_t^k}\right\|_\infty\leq D_\phi\left\|\frac{\theta_t^k-\theta_t}{\eta}-\theta_\star\right\|_2\leq2D_\star,\qquad0\leq k\leq K. s t π t k ∞ ≤ D ϕ η θ t k − θ t − θ ⋆ 2 ≤ 2 D ⋆ , 0 ≤ k ≤ K .
This proves Eq. (271) .
Proof of Corollary K.1
Corollary K.2 (KimiPO-Norm is coverage-preserving (full version of Corollary K.1).) Suppose that Assumption 2.1 holds. Fix ε ∈ ( 0 , R max ] \varepsilon\in(0,R_{\max}] ε ∈ ( 0 , R m a x ] and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Run Algorithm 4 with π 0 = π b a s e \pi_0=\pi_{\mathrm{base}} π 0 = π base and η = 1 / ( 3 R max + 4 D ⋆ ) \eta=1/(3R_{\max}+4D_\star) η = 1/ ( 3 R m a x + 4 D ⋆ ) . If ε ≥ D ⋆ \varepsilon\geq D_\star ε ≥ D ⋆ , take T = 0 T=0 T = 0 , with arbitrary α > 0 \alpha>0 α > 0 and integers G , N , K ≥ 1 G,N,K\geq1 G , N , K ≥ 1 . Otherwise, set
T = ⌈ log ( 2 C b a s e ) η ε ⌉ , (293) T=\left\lceil\frac{\log(2C_{\mathrm{base}})}{\eta\varepsilon}\right\rceil, \tag*{(293)} T = ⌈ η ε log ( 2 C base ) ⌉ , (293)
and choose the inner loop parameters as
ε i n S q = 1 12 C b a s e η 2 T 2 log ( e T ) , α = η 2 8 D ϕ 2 , K = ⌈ 8 D ⋆ 2 ε i n S q ⌉ , (294) \varepsilon_{\mathrm{in}}^{\mathrm{Sq}}=\frac{1}{12C_{\mathrm{base}}\eta^2T^2\log(eT)},\qquad
\alpha=\frac{\eta^2}{8D_\phi^2},\qquad
K=\left\lceil\frac{8D_\star^2}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\right\rceil, \tag*{(294)} ε in Sq = 12 C base η 2 T 2 log ( e T ) 1 , α = 8 D ϕ 2 η 2 , K = ⌈ ε in Sq 8 D ⋆ 2 ⌉ , (294)
and let G , N G,N G , N be the smallest positive integers satisfying
G ≥ ( e η D ⋆ − 1 ) 2 4 η ε i n S q , N ≥ C D ⋆ 2 ε i n S q ( d log ( 1 + 8 N ) + log 4 ( d + 1 ) T δ ) , (295) G\geq\frac{(e^{\eta D_\star}-1)^2}{4\eta\sqrt{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}},\qquad
N\geq\frac{CD_\star^2}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\left(d\log(1+8N)+\log\frac{4(d+1)T}{\delta}\right), \tag*{(295)} G ≥ 4 η ε in Sq ( e η D ⋆ − 1 ) 2 , N ≥ ε in Sq C D ⋆ 2 ( d log ( 1 + 8 N ) + log δ 4 ( d + 1 ) T ) , (295)
where C C C is the universal constant in Theorem K.1. Then the returned policy π p o s t = π T \pi_{\mathrm{post}}=\pi_T π post = π T satisfies
J ⋆ − J ( π p o s t ) ≤ ε (296) J^\star-J(\pi_{\mathrm{post}})\leq\varepsilon \tag*{(296)} J ⋆ − J ( π post ) ≤ ε (296)
with probability at least 1 − δ 1-\delta 1 − δ over the sampled training data. The numbers of gradient steps and sampled responses are
N s t e p ( ε , δ ) = K T = O ~ ( C b a s e D ⋆ 2 η ε 3 ) , N_{\mathrm{step}}(\varepsilon,\delta)=KT=\widetilde{O}\left(\frac{C_{\mathrm{base}}D_{\star}^{2}}{\eta\varepsilon^{3}}\right), N step ( ε , δ ) = K T = O ( η ε 3 C base D ⋆ 2 ) ,
N s a m p l e ( ε , δ ) = N ( G + 1 ) T = O ~ ( C b a s e D ⋆ 2 ( d + log ( 1 / δ ) ) η ε 3 ( 1 + η C b a s e D ⋆ 2 ε ) ) . N_{\mathrm{sample}}(\varepsilon,\delta)=N(G+1)T=\widetilde{O}\left(\frac{C_{\mathrm{base}}D_{\star}^{2}(d+\log(1/\delta))}{\eta\varepsilon^{3}}\left(1+\frac{\eta\sqrt{C_{\mathrm{base}}D_{\star}^{2}}}{\varepsilon}\right)\right). N sample ( ε , δ ) = N ( G + 1 ) T = O ( η ε 3 C base D ⋆ 2 ( d + log ( 1/ δ )) ( 1 + ε η C base D ⋆ 2 ) ) .
Proof of Corollary K.2. We prove the result supposing ε < D ⋆ \varepsilon<D_{\star} ε < D ⋆ as otherwise we immediately have J ⋆ − J ( π b a s e ) ≤ D ⋆ ≤ ε J^{\star}-J(\pi_{\mathrm{base}})\le D_{\star}\le\varepsilon J ⋆ − J ( π base ) ≤ D ⋆ ≤ ε from Eq. (188) .
We show that the hyperparameter choices in Corollary K.2 satisfy the conditions for Theorem 4.1. For each outer iteration t t t , we invoke Theorem K.1, the inner convergence guarantee, with failure probability δ / T \delta/T δ / T . We check that the corollary parameters match the theorem requirements. First, the bounds on G G G and N N N in Eq. (295) in Corollary K.2 match those in Theorem K.1 (Eqs. (265) and (266) ) with δ / T \delta/T δ / T . The inner learning rate and number of steps in Eq. (294) ,
α = η 2 8 D ϕ 2 , K = ⌈ 8 D ⋆ 2 ε i n S q ⌉ = ⌈ η 2 D ⋆ 2 α ε i n S q ⌉ , \alpha=\frac{\eta^{2}}{8D_{\phi}^{2}},\qquad K=\left\lceil\frac{8D_{\star}^{2}}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\right\rceil=\left\lceil\frac{\eta^{2}D_{\star}^{2}}{\alpha\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\right\rceil, α = 8 D ϕ 2 η 2 , K = ⌈ ε in Sq 8 D ⋆ 2 ⌉ = ⌈ α ε in Sq η 2 D ⋆ 2 ⌉ ,
also satisfy Eq. (266) . Finally, η D ⋆ ≤ 1 / 4 \eta D_{\star}\le1/4 η D ⋆ ≤ 1/4 , and Eqs. (293) and (294) give 0 < ε i n S q ≤ ε 2 < D ⋆ 2 0<\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\le\varepsilon^{2}<D_{\star}^{2} 0 < ε in Sq ≤ ε 2 < D ⋆ 2 . Applying Theorem K.1 conditional on the preceding history at each iteration and taking a union bound over the T T T iterations gives, with probability at least 1 − δ 1-\delta 1 − δ over the sampled training data,
MSE t ( π t + 1 ) ≤ ε i n S q , ∥ s t π t + 1 ∥ ∞ ≤ 2 D ⋆ , 0 ≤ t < T . (297) \operatorname{MSE}_{t}(\pi_{t+1})\le\varepsilon_{\mathrm{in}}^{\mathrm{Sq}},\qquad\lVert s_{t}^{\pi_{t+1}}\rVert_{\infty}\le2D_{\star},\qquad0\le t<T. \tag*{(297)} MSE t ( π t + 1 ) ≤ ε in Sq , ∥ s t π t + 1 ∥ ∞ ≤ 2 D ⋆ , 0 ≤ t < T . (297)
We condition on this event for the remainder of the proof.
We next verify Eq. (20) in Theorem 4.1. By Eqs. (189) and (297) , S = max 0 ≤ t < T ∥ s t π t + 1 ∥ ∞ ≤ 2 D ⋆ S=\max_{0\le t<T}\lVert s_{t}^{\pi_{t+1}}\rVert_{\infty}\le2D_{\star} S = max 0 ≤ t < T ∥ s t π t + 1 ∥ ∞ ≤ 2 D ⋆ , so e η S ≤ e 1 / 2 < 2 e^{\eta S}\le e^{1/2}<2 e η S ≤ e 1/2 < 2 . Using Eqs. (294) and (297) ,
MSE t ( π t + 1 ) ≤ ε i n S q = 1 12 C b a s e η 2 T 2 log ( e T ) ≤ 1 4 η 2 T 2 ( C b a s e log ( e T ) + e η S ) , 0 ≤ t < T . \operatorname{MSE}_{t}(\pi_{t+1})\le\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}=\frac{1}{12C_{\mathrm{base}}\eta^{2}T^{2}\log(eT)}\le\frac{1}{4\eta^{2}T^{2}(C_{\mathrm{base}}\log(eT)+e^{\eta S})},\qquad0\le t<T. MSE t ( π t + 1 ) ≤ ε in Sq = 12 C base η 2 T 2 log ( e T ) 1 ≤ 4 η 2 T 2 ( C base log ( e T ) + e η S ) 1 , 0 ≤ t < T .
The last inequality uses C b a s e log ( e T ) + e η S ≤ 3 C b a s e log ( e T ) C_{\mathrm{base}}\log(eT)+e^{\eta S}\le3C_{\mathrm{base}}\log(eT) C base log ( e T ) + e η S ≤ 3 C base log ( e T ) , since C b a s e log ( e T ) ≥ 1 C_{\mathrm{base}}\log(eT)\ge1 C base log ( e T ) ≥ 1 .
Next, under Assumption 2.1 we have π t > 0 \pi_{t}>0 π t > 0 for all 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T , and π 0 = π b a s e \pi_{0}=\pi_{\mathrm{base}} π 0 = π base . Applying Theorem 4.1 and substituting Eq. (293) therefore gives
J ⋆ − J ( π p o s t ) = J ⋆ − J ( π T ) ≤ log ( 2 C b a s e ) η T ≤ ε . J^{\star}-J(\pi_{\mathrm{post}})=J^{\star}-J(\pi_{T})\le\frac{\log(2C_{\mathrm{base}})}{\eta T}\le\varepsilon. J ⋆ − J ( π post ) = J ⋆ − J ( π T ) ≤ η T log ( 2 C base ) ≤ ε .
This proves Eq. (296) with probability at least 1 − δ 1-\delta 1 − δ .
Finally, we bound the numbers of gradient steps and sampled responses. Since η ε ≤ η R max ≤ 1 / 3 \eta\varepsilon\le\eta R_{\max}\le1/3 η ε ≤ η R m a x ≤ 1/3 and C b a s e ≥ 1 C_{\mathrm{base}}\ge1 C base ≥ 1 , the choice of T T T gives
log ( 2 C b a s e ) η ε ≤ T ≤ 2 log ( 2 C b a s e ) η ε . (298) \frac{\log(2C_{\mathrm{base}})}{\eta\varepsilon}\le T\le\frac{2\log(2C_{\mathrm{base}})}{\eta\varepsilon}. \tag*{(298)} η ε log ( 2 C base ) ≤ T ≤ η ε 2 log ( 2 C base ) . (298)
Substituting Eq. (298) into the definition of ε i n S q \varepsilon_{\mathrm{in}}^{\mathrm{Sq}} ε in Sq in Eq. (294) gives
1 ε i n S q = 12 C b a s e η 2 T 2 log ( e T ) = O ~ ( C b a s e ε 2 ) . (299) \frac{1}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}=12C_{\mathrm{base}}\eta^{2}T^{2}\log(eT)=\widetilde{O}\left(\frac{C_{\mathrm{base}}}{\varepsilon^{2}}\right). \tag*{(299)} ε in Sq 1 = 12 C base η 2 T 2 log ( e T ) = O ( ε 2 C base ) . (299)
By Eq. (299) , the choice of K K K in Eq. (294) and the smallest admissible N N N in Eq. (295) satisfy
K = O ~ ( C b a s e D ⋆ 2 ε 2 ) , N = O ~ ( C b a s e D ⋆ 2 ε 2 ( d + log ( T / δ ) ) ) . (300) K=\widetilde{O}\left(\frac{C_{\mathrm{base}}D_{\star}^{2}}{\varepsilon^{2}}\right),\qquad N=\widetilde{O}\left(\frac{C_{\mathrm{base}}D_{\star}^{2}}{\varepsilon^{2}}\left(d+\log(T/\delta)\right)\right). \tag*{(300)} K = O ( ε 2 C base D ⋆ 2 ) , N = O ( ε 2 C base D ⋆ 2 ( d + log ( T / δ ) ) ) . (300)
The log ( 1 + 8 N ) \log(1 + 8N) log ( 1 + 8 N ) term in Eq. (295) contributes a logarithmic factor to this bound on N N N . Since e η D ⋆ − 1 ≤ 2 η D ⋆ e^{\eta D_\star} - 1 \le2\eta D_\star e η D ⋆ − 1 ≤ 2 η D ⋆ for η D ⋆ ≤ 1 / 4 \eta D_\star\le1/4 η D ⋆ ≤ 1/4 , the choice of G G G in Eq. (295) gives
G ≤ 1 + η D ⋆ 2 ε i n S q = O ~ ( 1 + η C b a s e D ⋆ 2 ε ) , (301) G \le1 + \frac{\eta D_\star^2}{\sqrt{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}} = \widetilde{O}\left(1 + \frac{\eta\sqrt{C_{\mathrm{base}}D_\star^2}}{\varepsilon}\right), \tag*{(301)} G ≤ 1 + ε in Sq η D ⋆ 2 = O ( 1 + ε η C base D ⋆ 2 ) , (301)
where the last bound uses Eq. (299) . Each outer iteration uses K K K gradient steps and samples N ( G + 1 ) N(G+1) N ( G + 1 ) responses. Substituting Eqs. (298) , (300) and (301) into K T KT K T and N ( G + 1 ) T N(G+1)T N ( G + 1 ) T gives
N s t e p ( ε , δ ) = K T = O ~ ( C b a s e D ⋆ 2 η ε 3 ) , N_{\mathrm{step}}(\varepsilon,\delta) = KT = \widetilde{O}\left(\frac{C_{\mathrm{base}}D_\star^2}{\eta\varepsilon^3}\right), N step ( ε , δ ) = K T = O ( η ε 3 C base D ⋆ 2 ) ,
N s a m p l e ( ε , δ ) = N ( G + 1 ) T = O ~ ( C b a s e D ⋆ 2 ( d + log ( 1 / δ ) ) η ε 3 ( 1 + η C b a s e D ⋆ 2 ε ) ) . N_{\mathrm{sample}}(\varepsilon,\delta) = N(G+1)T = \widetilde{O}\left(\frac{C_{\mathrm{base}}D_\star^2(d+\log(1/\delta))}{\eta\varepsilon^3}\left(1+\frac{\eta\sqrt{C_{\mathrm{base}}D_\star^2}}{\varepsilon}\right)\right). N sample ( ε , δ ) = N ( G + 1 ) T = O ( η ε 3 C base D ⋆ 2 ( d + log ( 1/ δ )) ( 1 + ε η C base D ⋆ 2 ) ) .
Rebel-OnPolicy: Global Convergence (Proof of Corollary 4.2)
This section gives the algorithm and convergence analysis for Rebel-OnPolicy, and is organized as follows.
Appendix § 17.5 .§ 1 describes the full Rebel-OnPolicy algorithm (Algorithm 5), and Corollary L.1 is the formal global convergence result summarized in Corollary 4.2 .
Appendix § 17.5 .§ 2 contains the supporting lemmas for the proof of Corollary L.1.
Appendix § 17.5 .§ 3 establishes convergence of the inner optimization loop in Algorithm 5.
Appendix § 17.5 .§ 4 proves Corollary L.1 by combining Theorem E.1 with the inner loop convergence in Theorem L.1.
Notation. We assume the softmax linear policies and linear rewards in Assumption 2.1, with r ( x , y ) ∈ [ 0 , R max ] r(x,y) \in[0,R_{\max}] r ( x , y ) ∈ [ 0 , R m a x ] , and write
B ⋆ : = ∥ θ ⋆ ∥ 2 , D ϕ : = max x , y , y ′ ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 , D ⋆ : = D ϕ B ⋆ . B_\star:= \lVert\theta_\star\rVert_2,\qquad D_\phi:= \max_{x,y,y'}\lVert\phi(x,y)-\phi(x,y')\rVert_2,\qquad D_\star:= D_\phi B_\star. B ⋆ := ∥ θ ⋆ ∥ 2 , D ϕ := x , y , y ′ max ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 , D ⋆ := D ϕ B ⋆ .
Thus ∣ r ( x , y ) − r ( x , y ′ ) ∣ ≤ D ⋆ \lvert r(x,y)-r(x,y')\rvert\le D_\star ∣ r ( x , y ) − r ( x , y ′ )∣ ≤ D ⋆ for responses to the same prompt. In this section we use D ⋆ D_\star D ⋆ as the more natural bound on reward differences, while the bounds in the main body use B : = B ϕ B ⋆ B := B_\phi B_\star B := B ϕ B ⋆ to be consistent with § 3 . Note that D ⋆ ≤ 2 B D_\star\le2B D ⋆ ≤ 2 B .
We will also denote an error tolerance on M S E t \mathrm{MSE}_t MSE t as ε i n S q \varepsilon_{\mathrm{in}}^{\mathrm{Sq}} ε in Sq , reserving ε i n \varepsilon_{\mathrm{in}} ε in for the KL tolerance in Eq. (6) .
Rebel-OnPolicy Algorithm
Recall the Rebel-OnPolicy objective in Eq. (21) ,
L t R e b e l − O n ( π ) : = E x ∼ μ , y , y ′ ∼ i i d π t ( ⋅ ∣ x ) [ ( 1 η log π ( y ∣ x ) π t ( y ∣ x ) − 1 η log π ( y ′ ∣ x ) π t ( y ′ ∣ x ) − ( r ( x , y ) − r ( x , y ′ ) ) ) 2 ] . (302) L_t^{\mathrm{Rebel-On}}(\pi) := \mathbb{E}_{x\sim\mu,\;y,y'\mathrel{\overset{\mathrm{iid}}{\sim}}\pi_t(\cdot\mid x)}\left[\left(\frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-\frac{1}{\eta}\log\frac{\pi(y'\mid x)}{\pi_t(y'\mid x)}-(r(x,y)-r(x,y'))\right)^2\right]. \tag*{(302)} L t Rebel − On ( π ) := E x ∼ μ , y , y ′ ∼ iid π t ( ⋅ ∣ x ) [ ( η 1 log π t ( y ∣ x ) π ( y ∣ x ) − η 1 log π t ( y ′ ∣ x ) π ( y ′ ∣ x ) − ( r ( x , y ) − r ( x , y ′ )) ) 2 ] . (302)
The full Rebel-OnPolicy algorithm is displayed in Algorithm 5. At outer iteration t t t , draw N N N independent prompts x i ∼ μ x_i\sim\mu x i ∼ μ and, conditionally on each x i x_i x i , two independent responses y i , y i ′ ∼ π t ( ⋅ ∣ x i ) y_i,y_i'\sim\pi_t(\cdot\mid x_i) y i , y i ′ ∼ π t ( ⋅ ∣ x i ) . Write r i = r ( x i , y i ) r_i=r(x_i,y_i) r i = r ( x i , y i ) and r i ′ = r ( x i , y i ′ ) r_i'=r(x_i,y_i') r i ′ = r ( x i , y i ′ ) . The empirical objective is
L ^ t R e b e l − O n ( π ) : = 1 N ∑ i = 1 N ( 1 η log π ( y i ∣ x i ) π t ( y i ∣ x i ) − 1 η log π ( y i ′ ∣ x i ) π t ( y i ′ ∣ x i ) − ( r i − r i ′ ) ) 2 . \widehat{L}_t^{\mathrm{Rebel-On}}(\pi) := \frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{\eta}\log\frac{\pi(y_i\mid x_i)}{\pi_t(y_i\mid x_i)}-\frac{1}{\eta}\log\frac{\pi(y_i'\mid x_i)}{\pi_t(y_i'\mid x_i)}-(r_i-r_i')\right)^2. L t Rebel − On ( π ) := N 1 i = 1 ∑ N ( η 1 log π t ( y i ∣ x i ) π ( y i ∣ x i ) − η 1 log π t ( y i ′ ∣ x i ) π ( y i ′ ∣ x i ) − ( r i − r i ′ ) ) 2 .
Starting from θ 0 t = θ t \theta_0^t=\theta_t θ 0 t = θ t , Algorithm 5 takes K K K gradient descent steps on this objective and returns the last iterate as π t + 1 \pi_{t+1} π t + 1 . The batch and the policy π t \pi_t π t in the denominator remain fixed during these steps.
Algorithm 5 Rebel-OnPolicy
Require: Base model π b a s e \pi_{\mathrm{base}} π base , outer iterations T T T , regularization parameter η > 0 \eta> 0 η > 0 , prompt batch size N N N , inner steps K K K , and learning rate α \alpha α .
Initialize π 0 = π b a s e = π θ 0 \pi_0 = \pi_{\mathrm{base}} = \pi_{\theta_0} π 0 = π base = π θ 0 .
for t = 0 , 1 , … , T − 1 t = 0, 1, \ldots, T - 1 t = 0 , 1 , … , T − 1 do
Draw a batch of N N N prompts x i ∼ i i d μ x_i \overset{\mathrm{iid}}{\sim} \mu x i ∼ iid μ .
For each prompt x i x_i x i , draw y i , y i ′ ∼ π t ( ⋅ ∣ x i ) y_i, y_i' \sim\pi_t(\cdot\mid x_i) y i , y i ′ ∼ π t ( ⋅ ∣ x i ) independently, and set r i = r ( x i , y i ) r_i = r(x_i,y_i) r i = r ( x i , y i ) and r i ′ = r ( x i , y i ′ ) r_i' = r(x_i,y_i') r i ′ = r ( x i , y i ′ ) .
Initialize θ t 0 = θ t \theta_t^0 = \theta_t θ t 0 = θ t and π t 0 = π t \pi_t^0 = \pi_t π t 0 = π t .
for k = 0 , 1 , … , K − 1 k = 0, 1, \ldots, K - 1 k = 0 , 1 , … , K − 1 do
Compute the empirical gradient (e.g., via backpropagation)
∇ L ^ t R e b e l − O n ( π t k ) = 2 η N ∑ i = 1 N ( ∇ log π t k ( y i ∣ x i ) − ∇ log π t k ( y i ′ ∣ x i ) ) ( 1 η log π t k ( y i ∣ x i ) π t ( y i ∣ x i ) − 1 η log π t k ( y i ′ ∣ x i ) π t ( y i ′ ∣ x i ) − ( r i − r i ′ ) ) . \widehat{\nabla L}^{\mathrm{Rebel-On}}_t(\pi_t^k) = \frac{2}{\eta N} \sum_{i=1}^{N} \left(\nabla\log\pi_t^k(y_i \mid x_i) - \nabla\log\pi_t^k(y_i' \mid x_i)\right)
\left(\frac{1}{\eta}\log\frac{\pi_t^k(y_i \mid x_i)}{\pi_t(y_i \mid x_i)} - \frac{1}{\eta}\log\frac{\pi_t^k(y_i' \mid x_i)}{\pi_t(y_i' \mid x_i)} - (r_i-r_i')\right). ∇ L t Rebel − On ( π t k ) = η N 2 i = 1 ∑ N ( ∇ log π t k ( y i ∣ x i ) − ∇ log π t k ( y i ′ ∣ x i ) ) ( η 1 log π t ( y i ∣ x i ) π t k ( y i ∣ x i ) − η 1 log π t ( y i ′ ∣ x i ) π t k ( y i ′ ∣ x i ) − ( r i − r i ′ ) ) .
Update θ t k + 1 = θ t k − α ∇ L ^ t R e b e l − O n ( π t k ) \theta_t^{k+1} = \theta_t^k - \alpha\widehat{\nabla L}^{\mathrm{Rebel-On}}_t(\pi_t^k) θ t k + 1 = θ t k − α ∇ L t Rebel − On ( π t k ) and π t k + 1 = π θ t k + 1 \pi_t^{k+1} = \pi_{\theta_t^{k+1}} π t k + 1 = π θ t k + 1 .
Set θ t + 1 = θ t K \theta_{t+1} = \theta_t^K θ t + 1 = θ t K and π t + 1 = π t K \pi_{t+1} = \pi_t^K π t + 1 = π t K .
return π p o s t = π T \pi_{\mathrm{post}} = \pi_T π post = π T .
The following corollary shows that Rebel-OnPolicy is coverage-preserving, and is the full version of the result presented in Corollary 4.2 .
Corollary L.1 (Rebel-OnPolicy is coverage-preserving). Suppose that Assumption 2.1 holds. Fix ϵ ∈ ( 0 , R max ] \epsilon\in(0,R_{\max}] ϵ ∈ ( 0 , R m a x ] with ϵ < D ⋆ \epsilon< D_\star ϵ < D ⋆ and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Run Algorithm 5 with π 0 = π b a s e \pi_0 = \pi_{\mathrm{base}} π 0 = π base and
η = log ( 2 C b a s e ) D ⋆ , T = ⌈ log ( 2 C b a s e ) η ϵ ⌉ . (303) \eta= \frac{\log(2C_{\mathrm{base}})}{D_\star}, \qquad T = \left\lceil\frac{\log(2C_{\mathrm{base}})}{\eta\epsilon} \right\rceil. \tag*{(303)} η = D ⋆ log ( 2 C base ) , T = ⌈ η ϵ log ( 2 C base ) ⌉ . (303) Set
ε i n S q : = 1 12 C b a s e η 2 T 2 log ( e T ) , α = η 2 2 D ϕ 2 , (304) \varepsilon_{\mathrm{in}}^{\mathrm{Sq}} := \frac{1}{12C_{\mathrm{base}}\eta^2T^2\log(eT)}, \qquad\alpha= \frac{\eta^2}{2D_\phi^2}, \tag*{(304)} ε in Sq := 12 C base η 2 T 2 log ( e T ) 1 , α = 2 D ϕ 2 η 2 , (304) K = ⌈ 2 D ⋆ 2 ε i n S q ⌉ , N = ⌈ 28 D ⋆ 2 3 ε i n S q log 2 d T δ ⌉ . (305) K = \left\lceil\frac{2D_\star^2}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}} \right\rceil, \qquad N = \left\lceil\frac{28D_\star^2}{3\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\log\frac{2dT}{\delta} \right\rceil. \tag*{(305)} K = ⌈ ε in Sq 2 D ⋆ 2 ⌉ , N = ⌈ 3 ε in Sq 28 D ⋆ 2 log δ 2 d T ⌉ . (305) Then the returned policy satisfies
J ⋆ − J ( π p o s t ) ≤ ϵ (306) J^\star- J(\pi_{\mathrm{post}}) \le\epsilon \tag*{(306)} J ⋆ − J ( π post ) ≤ ϵ (306) with probability at least 1 − δ 1-\delta 1 − δ over the sampled training data. On the same event, with probability at least 1 − ( 2 + 2 ) / ( 4 log ( e T ) ) 1-(2+\sqrt{2})/(4\sqrt{\log(eT)}) 1 − ( 2 + 2 ) / ( 4 log ( e T ) ) over x ∼ μ x \sim\mu x ∼ μ ,
π t ( Y x ⋆ ∣ x ) ≥ 1 2 π b a s e ( Y x ⋆ ∣ x ) , ∀ 0 ≤ t ≤ T . (307) \pi_t(Y_x^\star\mid x) \ge\frac{1}{2}\pi_{\mathrm{base}}(Y_x^\star\mid x), \qquad\forall0 \le t \le T. \tag*{(307)} π t ( Y x ⋆ ∣ x ) ≥ 2 1 π base ( Y x ⋆ ∣ x ) , ∀0 ≤ t ≤ T . (307) The numbers of gradient steps and sampled responses are
N s t e p ( ϵ , δ ) = K T = O ~ ( C b a s e D 3 ϵ 3 ) , N_{\mathrm{step}}(\epsilon,\delta) = KT = \widetilde{O}\left(\frac{C_{\mathrm{base}}D^3}{\epsilon^3}\right), N step ( ϵ , δ ) = K T = O ( ϵ 3 C base D 3 ) , N s a m p l e ( ϵ , δ ) = 2 N T = O ~ ( C b a s e D 3 ϵ 3 log 2 d δ ) . (308) N_{\mathrm{sample}}(\epsilon,\delta) = 2NT = \widetilde{O}\left(\frac{C_{\mathrm{base}}D^3}{\epsilon^3}\log\frac{2d}{\delta}\right). \tag*{(308)} N sample ( ϵ , δ ) = 2 N T = O ( ϵ 3 C base D 3 log δ 2 d ) . (308)
Parameter choices. The inner loop convergence analysis permits any η > 0 \eta> 0 η > 0 . The choice in Eq. (303) balances the number of outer iterations against the exponential factor in Theorem E.1, giving e η D ⋆ = 2 C b a s e e^{\eta D_\star} = 2C_{\mathrm{base}} e η D ⋆ = 2 C base . For fixed problem parameters, both the number of gradient steps and the number of sampled responses are O ~ ( 1 / ϵ 3 ) \widetilde{O}(1/\epsilon^3) O ( 1/ ϵ 3 ) , and the response count depends only logarithmically on d d d and 1 / δ 1/\delta 1/ δ .
Supporting Lemmas
The following lemmas connect optimization of the empirical objective to the mean-squared error needed for the global convergence guarantee.
Lemma L.1 shows that L t R e b e l − O n ( π ) = 2 MSE t ( π ) L_t^{\mathrm{Rebel-On}}(\pi)=2\operatorname{MSE}_t(\pi) L t Rebel − On ( π ) = 2 MSE t ( π ) , so minimizing the population loss directly minimizes the mean-squared error.
Lemma L.2 shows that the population and empirical losses are convex quadratics and identifies the exact update π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ as a common minimizer with zero loss.
Lemma L.3 bounds the smoothness of both losses.
Lemma L.4 bounds the population loss by the empirical loss and an error proportional to the squared parameter distance from the exact update, uniformly over θ \theta θ .
Fix an outer iteration t t t and η > 0 \eta>0 η > 0 throughout this subsection. Probabilities and expectations over the batch are conditional on π t \pi_t π t . Recall that for any π > 0 \pi>0 π > 0 , the residual and mean-squared error are defined as
s t π ( x , y ) : = 1 η log π ( y ∣ x ) π t ( y ∣ x ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ 1 η log π ( y ′ ∣ x ) π t ( y ′ ∣ x ) ] − ( r ( x , y ) − r ( x , π t ) ) , (309) s_t^\pi(x,y):=\frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[\frac{1}{\eta}\log\frac{\pi(y'\mid x)}{\pi_t(y'\mid x)}\right]-(r(x,y)-r(x,\pi_t)), \tag*{(309)} s t π ( x , y ) := η 1 log π t ( y ∣ x ) π ( y ∣ x ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ η 1 log π t ( y ′ ∣ x ) π ( y ′ ∣ x ) ] − ( r ( x , y ) − r ( x , π t )) , (309)
MSE t ( π ) : = E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( s t π ( x , y ) ) 2 ] . \operatorname{MSE}_t(\pi):=\mathbb{E}_{x\sim\mu,\,y\sim\pi_t(\cdot\mid x)}\left[(s_t^\pi(x,y))^2\right]. MSE t ( π ) := E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( s t π ( x , y ) ) 2 ] .
Lemma L.1. For every policy π > 0 \pi>0 π > 0 , the Rebel–OnPolicy loss Eq. (302) satisfies
MSE t ( π ) = 1 2 L t R e b e l − O n ( π ) . \operatorname{MSE}_t(\pi)=\frac{1}{2}L_t^{\mathrm{Rebel-On}}(\pi). MSE t ( π ) = 2 1 L t Rebel − On ( π ) .
Proof of Lemma L.1. For a fixed prompt x x x , the definition of s t π s_t^\pi s t π in Eq. (309) gives for a pair of responses ( y , y ′ ) (y,y') ( y , y ′ ) ,
s t π ( x , y ) − s t π ( x , y ′ ) = 1 η log π ( y ∣ x ) π t ( y ∣ x ) − 1 η log π ( y ′ ∣ x ) π t ( y ′ ∣ x ) − r ( x , y ) + r ( x , y ′ ) . s_t^\pi(x,y)-s_t^\pi(x,y')=\frac{1}{\eta}\log\frac{\pi(y\mid x)}{\pi_t(y\mid x)}-\frac{1}{\eta}\log\frac{\pi(y'\mid x)}{\pi_t(y'\mid x)}-r(x,y)+r(x,y'). s t π ( x , y ) − s t π ( x , y ′ ) = η 1 log π t ( y ∣ x ) π ( y ∣ x ) − η 1 log π t ( y ′ ∣ x ) π ( y ′ ∣ x ) − r ( x , y ) + r ( x , y ′ ) .
Thus, since E y ∼ π t ( ⋅ ∣ x ) [ s t π ( x , y ) ] = 0 \mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[s_t^\pi(x,y)]=0 E y ∼ π t ( ⋅ ∣ x ) [ s t π ( x , y )] = 0 and the two responses are conditionally independent given x x x ,
L t R e b e l − O n ( π ) = E x ∼ μ , y , y ′ ∼ i i d π t ( ⋅ ∣ x ) [ ( s t π ( x , y ) − s t π ( x , y ′ ) ) 2 ] = 2 E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( s t π ( x , y ) ) 2 ] = 2 MSE t ( π ) . \begin{aligned}
L_t^{\mathrm{Rebel-On}}(\pi)=\mathbb{E}_{x\sim\mu,\,y,y'\stackrel{\mathrm{iid}}{\sim}\pi_t(\cdot\mid x)}\left[(s_t^\pi(x,y)-s_t^\pi(x,y'))^2\right] \\
&=2\mathbb{E}_{x\sim\mu,\,y\sim\pi_t(\cdot\mid x)}\left[(s_t^\pi(x,y))^2\right]=2\operatorname{MSE}_t(\pi).
\end{aligned} L t Rebel − On ( π ) = E x ∼ μ , y , y ′ ∼ iid π t ( ⋅ ∣ x ) [ ( s t π ( x , y ) − s t π ( x , y ′ ) ) 2 ] = 2 E x ∼ μ , y ∼ π t ( ⋅ ∣ x ) [ ( s t π ( x , y ) ) 2 ] = 2 MSE t ( π ) .
For the remaining lemmas, define the population and empirical covariance matrices of the feature differences by
Σ t O n : = E x ∼ μ , y , y ′ ∼ i i d π t ( ⋅ ∣ x ) [ ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ⊤ ] , (310) \Sigma_t^{\mathrm{On}}:=\mathbb{E}_{x\sim\mu,\,y,y'\stackrel{\mathrm{iid}}{\sim}\pi_t(\cdot\mid x)}\left[(\phi(x,y)-\phi(x,y'))(\phi(x,y)-\phi(x,y'))^\top\right], \tag*{(310)} Σ t On := E x ∼ μ , y , y ′ ∼ iid π t ( ⋅ ∣ x ) [ ( ϕ ( x , y ) − ϕ ( x , y ′ )) ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ⊤ ] , (310)
Σ ^ t O n : = 1 N ∑ i = 1 N ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ⊤ , (311) \widehat{\Sigma}_t^{\mathrm{On}}:=\frac{1}{N}\sum_{i=1}^{N}(\phi(x_i,y_i)-\phi(x_i,y_i'))(\phi(x_i,y_i)-\phi(x_i,y_i'))^\top, \tag*{(311)} Σ t On := N 1 i = 1 ∑ N ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ )) ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ⊤ , (311)
where the batch is sampled as in Algorithm 5.
Lemma L.2. Suppose that Assumption 2.1 holds. For every θ ∈ R d \theta\in\mathbb{R}^d θ ∈ R d , the population and empirical losses satisfy
L t R e b e l − O n ( π θ ) = ( θ − θ t η − θ ⋆ ) ⊤ Σ t O n ( θ − θ t η − θ ⋆ ) , (312) L_t^{\mathrm{Rebel-On}}(\pi_\theta)=\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right)^\top\Sigma_t^{\mathrm{On}}\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right), \tag*{(312)} L t Rebel − On ( π θ ) = ( η θ − θ t − θ ⋆ ) ⊤ Σ t On ( η θ − θ t − θ ⋆ ) , (312)
L ^ t R e b e l − O n ( π θ ) = ( θ − θ t η − θ ⋆ ) ⊤ Σ ^ t O n ( θ − θ t η − θ ⋆ ) . (313) \widehat{L}_t^{\mathrm{Rebel-On}}(\pi_\theta)=\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right)^\top\widehat{\Sigma}_t^{\mathrm{On}}\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right). \tag*{(313)} L t Rebel − On ( π θ ) = ( η θ − θ t − θ ⋆ ) ⊤ Σ t On ( η θ − θ t − θ ⋆ ) . (313)
Both losses are convex as functions of θ \theta θ , and π η , t ⋆ = π θ t + η θ ⋆ \pi_{\eta,t}^{\star}=\pi_{\theta_t+\eta\theta_\star} π η , t ⋆ = π θ t + η θ ⋆ minimizes both with zero loss. The empirical claims hold for every realization of the batch.
Proof of Lemma L.2. For every prompt x x x and pair of responses ( y , y ′ ) (y,y') ( y , y ′ ) , Assumption 2.1 gives
1 η log π θ ( y ∣ x ) π t ( y ∣ x ) − 1 η log π θ ( y ′ ∣ x ) π t ( y ′ ∣ x ) − ( r ( x , y ) − r ( x , y ′ ) ) = ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ⊤ ( θ − θ t η − θ ⋆ ) . (314) \frac{1}{\eta}\log\frac{\pi_\theta(y\mid x)}{\pi_t(y\mid x)}-\frac{1}{\eta}\log\frac{\pi_\theta(y'\mid x)}{\pi_t(y'\mid x)}-\left(r(x,y)-r(x,y')\right)=\left(\phi(x,y)-\phi(x,y')\right)^\top\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right). \tag*{(314)} η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) − η 1 log π t ( y ′ ∣ x ) π θ ( y ′ ∣ x ) − ( r ( x , y ) − r ( x , y ′ ) ) = ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ⊤ ( η θ − θ t − θ ⋆ ) . (314)
Squaring Eq. (314) and taking the population expectation gives
L t R e b e l − O n ( π θ ) = E x ∼ μ , y , y ′ ∼ i i d π t ( ⋅ ∣ x ) [ ( ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ⊤ ( θ − θ t η − θ ⋆ ) ) 2 ] = ( θ − θ t η − θ ⋆ ) ⊤ E x ∼ μ , y , y ′ ∼ i i d π t ( ⋅ ∣ x ) [ ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ⊤ ] ( θ − θ t η − θ ⋆ ) = ( θ − θ t η − θ ⋆ ) ⊤ Σ ^ t O n ( θ − θ t η − θ ⋆ ) , \begin{aligned}
L_t^{\mathrm{Rebel-On}}(\pi_\theta)=\mathbb{E}_{\substack{x\sim\mu,\\y,y'\mathrel{\overset{\mathrm{iid}}{\sim}}\pi_t(\cdot\mid x)}}\left[\left(\left(\phi(x,y)-\phi(x,y')\right)^\top\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right)\right)^2\right] \\
&=\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right)^\top\mathbb{E}_{\substack{x\sim\mu,\\y,y'\mathrel{\overset{\mathrm{iid}}{\sim}}\pi_t(\cdot\mid x)}}\left[\left(\phi(x,y)-\phi(x,y')\right)\left(\phi(x,y)-\phi(x,y')\right)^\top\right]\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right) \\
&=\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right)^\top\widehat{\Sigma}_t^{\mathrm{On}}\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right),
\end{aligned} L t Rebel − On ( π θ ) = E x ∼ μ , y , y ′ ∼ iid π t ( ⋅ ∣ x ) [ ( ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ⊤ ( η θ − θ t − θ ⋆ ) ) 2 ] = ( η θ − θ t − θ ⋆ ) ⊤ E x ∼ μ , y , y ′ ∼ iid π t ( ⋅ ∣ x ) [ ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ⊤ ] ( η θ − θ t − θ ⋆ ) = ( η θ − θ t − θ ⋆ ) ⊤ Σ t On ( η θ − θ t − θ ⋆ ) ,
where the last equality uses the definition of Σ ^ t O n \widehat{\Sigma}_t^{\mathrm{On}} Σ t On in Eq. (310) . Averaging over the batch instead gives
L ^ t R e b e l − O n ( π θ ) = 1 N ∑ i = 1 N ( ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ⊤ ( θ − θ t η − θ ⋆ ) ) 2 = ( θ − θ t η − θ ⋆ ) ⊤ Σ ^ ^ t O n ( θ − θ t η − θ ⋆ ) , \begin{aligned}
\widehat{L}_t^{\mathrm{Rebel-On}}(\pi_\theta)=\frac{1}{N}\sum_{i=1}^{N}\left(\left(\phi(x_i,y_i)-\phi(x_i,y_i')\right)^\top\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right)\right)^2 \\
&=\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right)^\top\widehat{\widehat{\Sigma}}_t^{\mathrm{On}}\left(\frac{\theta-\theta_t}{\eta}-\theta_\star\right),
\end{aligned} L t Rebel − On ( π θ ) = N 1 i = 1 ∑ N ( ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ⊤ ( η θ − θ t − θ ⋆ ) ) 2 = ( η θ − θ t − θ ⋆ ) ⊤ Σ t On ( η θ − θ t − θ ⋆ ) ,
by the definition of Σ ^ ^ t O n \widehat{\widehat{\Sigma}}_t^{\mathrm{On}} Σ t On in Eq. (311) . Since Σ ^ t O n ⪰ 0 \widehat{\Sigma}_t^{\mathrm{On}}\succeq0 Σ t On ⪰ 0 and Σ ^ ^ t O n ⪰ 0 \widehat{\widehat{\Sigma}}_t^{\mathrm{On}}\succeq0 Σ t On ⪰ 0 , both losses are convex in θ \theta θ . The exact update π η , t ⋆ = π θ t + η θ ⋆ \pi_{\eta,t}^\star=\pi_{\theta_t+\eta\theta_\star} π η , t ⋆ = π θ t + η θ ⋆ from Lemma B.7 has zero loss under both objectives. □
Lemma L.3. Suppose that Assumption 2.1 holds. The population loss L t R e b e l − O n ( π θ ) L_t^{\mathrm{Rebel-On}}(\pi_\theta) L t Rebel − On ( π θ ) and empirical loss L ^ t R e b e l − O n ( π θ ) \widehat{L}_t^{\mathrm{Rebel-On}}(\pi_\theta) L t Rebel − On ( π θ ) are 2 D ϕ 2 / η 2 2D_\phi^2/\eta^2 2 D ϕ 2 / η 2 -smooth as functions of θ ∈ R d \theta\in\mathbb{R}^d θ ∈ R d . This holds for every L ^ t R e b e l − O n \widehat{L}_t^{\mathrm{Rebel-On}} L t Rebel − On , regardless of the batch size or samples.
Proof of Lemma L.3. Differentiating the quadratic formulas in Lemma L.2 twice gives
∇ 2 L t R e b e l − O n ( π θ ) = 2 η 2 Σ ^ t O n , \nabla^2L_t^{\mathrm{Rebel-On}}(\pi_\theta)=\frac{2}{\eta^2}\widehat{\Sigma}_t^{\mathrm{On}}, ∇ 2 L t Rebel − On ( π θ ) = η 2 2 Σ t On ,
∇ 2 L ^ t R e b e l − O n ( π θ ) = 2 η 2 Σ ^ ^ t O n . (315) \nabla^2\widehat{L}_t^{\mathrm{Rebel-On}}(\pi_\theta)=\frac{2}{\eta^2}\widehat{\widehat{\Sigma}}_t^{\mathrm{On}}. \tag*{(315)} ∇ 2 L t Rebel − On ( π θ ) = η 2 2 Σ t On . (315)
To bound their operator norms, for every v ∈ R d v\in\mathbb{R}^d v ∈ R d , Cauchy–Schwarz and the definition of D ϕ D_\phi D ϕ give
0 ≤ v ⊤ Σ ^ t O n v = E x ∼ μ , y , y ′ ∼ i i d π t ( ⋅ ∣ x ) [ ( v ⊤ ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ) 2 ] ≤ D ϕ 2 ∥ v ∥ 2 2 , 0\leq v^\top\widehat{\Sigma}_t^{\mathrm{On}}v=\mathbb{E}_{\substack{x\sim\mu,\\y,y'\mathrel{\overset{\mathrm{iid}}{\sim}}\pi_t(\cdot\mid x)}}\left[\left(v^\top\left(\phi(x,y)-\phi(x,y')\right)\right)^2\right]\leq D_\phi^2\lVert v\rVert_2^2, 0 ≤ v ⊤ Σ t On v = E x ∼ μ , y , y ′ ∼ iid π t ( ⋅ ∣ x ) [ ( v ⊤ ( ϕ ( x , y ) − ϕ ( x , y ′ ) ) ) 2 ] ≤ D ϕ 2 ∥ v ∥ 2 2 ,
0 ≤ v ⊤ Σ ^ ^ t O n v = 1 N ∑ i = 1 N ( v ⊤ ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ) 2 ≤ D ϕ 2 ∥ v ∥ 2 2 . (316) 0\leq v^\top\widehat{\widehat{\Sigma}}_t^{\mathrm{On}}v=\frac{1}{N}\sum_{i=1}^{N}\left(v^\top\left(\phi(x_i,y_i)-\phi(x_i,y_i')\right)\right)^2\leq D_\phi^2\lVert v\rVert_2^2. \tag*{(316)} 0 ≤ v ⊤ Σ t On v = N 1 i = 1 ∑ N ( v ⊤ ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ) 2 ≤ D ϕ 2 ∥ v ∥ 2 2 . (316)
Hence both Hessians in Eq. (315) have operator norm at most 2 D ϕ 2 / η 2 2D_\phi^2/\eta^2 2 D ϕ 2 / η 2 , proving the stated smoothness bound. □
Lemma L.4. Suppose that Assumption 2.1 holds. Fix δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and draw a batch as in Algorithm 5, with N ≥ 28 δ log 2 d δ N\geq\frac{28}{\delta}\log\frac{2d}{\delta} N ≥ δ 28 log δ 2 d prompts and two responses per prompt. With probability at least 1 − δ 1-\delta 1 − δ over the batch, simultaneously for every θ ∈ R d \theta\in\mathbb{R}^d θ ∈ R d ,
L t R e b e l − O n ( π θ ) ≤ 2 L ^ t R e b e l − O n ( π θ ) + 28 D ϕ 2 3 η 2 N log 2 d δ ∥ θ − θ t − η θ ⋆ ∥ 2 2 . L_t^{\mathrm{Rebel-On}}(\pi_\theta)\leq2\widehat{L}_t^{\mathrm{Rebel-On}}(\pi_\theta)+\frac{28D_\phi^2}{3\eta^2N}\log\frac{2d}{\delta}\left\lVert\theta-\theta_t-\eta\theta_\star\right\rVert_2^2. L t Rebel − On ( π θ ) ≤ 2 L t Rebel − On ( π θ ) + 3 η 2 N 28 D ϕ 2 log δ 2 d ∥ θ − θ t − η θ ⋆ ∥ 2 2 .
Proof of Lemma L.4. By Lemma L.2, both losses are quadratic forms in ( θ − θ t ) / η − θ ⋆ (\theta-\theta_t)/\eta-\theta_\star ( θ − θ t ) / η − θ ⋆ . We first prove that, with probability at least 1 − δ 1-\delta 1 − δ ,
Σ ^ t O n ⪰ 2 Σ ^ t O n + λ I , λ : = 28 D ϕ 2 3 N log 2 d δ . (317) \widehat{\Sigma}^{\mathrm{On}}_t \succeq2\widehat{\Sigma}^{\mathrm{On}}_t+\lambda I,\qquad\lambda:= \frac{28D_\phi^2}{3N}\log\frac{2d}{\delta}. \tag*{(317)} Σ t On ⪰ 2 Σ t On + λ I , λ := 3 N 28 D ϕ 2 log δ 2 d . (317)
We will obtain Eq. (317) by bounding the operator norm of
( Σ ^ t O n + λ I ) − 1 / 2 ( Σ ^ t O n − Σ t O n ) ( Σ ^ t O n + λ I ) − 1 / 2 = 1 N ∑ i = 1 N ( Y i − E [ Y i ] ) , (318) \left(\widehat{\Sigma}^{\mathrm{On}}_t+\lambda I\right)^{-1/2}\left(\widehat{\Sigma}^{\mathrm{On}}_t-\Sigma^{\mathrm{On}}_t\right)\left(\widehat{\Sigma}^{\mathrm{On}}_t+\lambda I\right)^{-1/2}=\frac{1}{N}\sum_{i=1}^{N}(Y_i-\mathbb{E}[Y_i]), \tag*{(318)} ( Σ t On + λ I ) − 1/2 ( Σ t On − Σ t On ) ( Σ t On + λ I ) − 1/2 = N 1 i = 1 ∑ N ( Y i − E [ Y i ]) , (318)
where
Y i : = ( Σ t O n + λ I ) − 1 / 2 ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ⊤ ( Σ t O n + λ I ) − 1 / 2 . Y_i:=\left(\Sigma^{\mathrm{On}}_t+\lambda I\right)^{-1/2}\left(\phi(x_i,y_i)-\phi(x_i,y_i')\right)\left(\phi(x_i,y_i)-\phi(x_i,y_i')\right)^\top\left(\Sigma^{\mathrm{On}}_t+\lambda I\right)^{-1/2}. Y i := ( Σ t On + λ I ) − 1/2 ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ( ϕ ( x i , y i ) − ϕ ( x i , y i ′ ) ) ⊤ ( Σ t On + λ I ) − 1/2 .
To apply matrix Bernstein, we bound the norm and variance of Y i − E [ Y i ] Y_i-\mathbb{E}[Y_i] Y i − E [ Y i ] . The definition of D ϕ D_\phi D ϕ and ( Σ t O n + λ I ) − 1 ⪯ λ − 1 I (\Sigma^{\mathrm{On}}_t+\lambda I)^{-1}\preceq\lambda^{-1}I ( Σ t On + λ I ) − 1 ⪯ λ − 1 I give
0 ⪯ Y i ⪯ D ϕ 2 λ I , 0\preceq Y_i\preceq\frac{D_\phi^2}{\lambda}I, 0 ⪯ Y i ⪯ λ D ϕ 2 I ,
0 ⪯ E [ Y i ] = ( Σ t O n + λ I ) − 1 / 2 Σ t O n ( Σ t O n + λ I ) − 1 / 2 ⪯ I . 0\preceq\mathbb{E}[Y_i]=(\Sigma^{\mathrm{On}}_t+\lambda I)^{-1/2}\Sigma^{\mathrm{On}}_t(\Sigma^{\mathrm{On}}_t+\lambda I)^{-1/2}\preceq I. 0 ⪯ E [ Y i ] = ( Σ t On + λ I ) − 1/2 Σ t On ( Σ t On + λ I ) − 1/2 ⪯ I .
Since the sample size condition ensures 0 < λ ≤ D ϕ 2 0<\lambda\le D_\phi^2 0 < λ ≤ D ϕ 2 , it follows that
∥ Y i − E [ Y i ] ∥ o p ≤ D ϕ 2 λ , \left\|Y_i-\mathbb{E}[Y_i]\right\|_{\mathrm{op}}\le\frac{D_\phi^2}{\lambda}, ∥ Y i − E [ Y i ] ∥ op ≤ λ D ϕ 2 ,
E [ ( Y i − E [ Y i ] ) 2 ] = E [ Y i 2 ] − ( E [ Y i ] ) 2 ⪯ E [ Y i 2 ] ⪯ D ϕ 2 λ E [ Y i ] ⪯ D ϕ 2 λ I . \mathbb{E}\left[(Y_i-\mathbb{E}[Y_i])^2\right]=\mathbb{E}[Y_i^2]-(\mathbb{E}[Y_i])^2\preceq\mathbb{E}[Y_i^2]\preceq\frac{D_\phi^2}{\lambda}\mathbb{E}[Y_i]\preceq\frac{D_\phi^2}{\lambda}I. E [ ( Y i − E [ Y i ] ) 2 ] = E [ Y i 2 ] − ( E [ Y i ] ) 2 ⪯ E [ Y i 2 ] ⪯ λ D ϕ 2 E [ Y i ] ⪯ λ D ϕ 2 I .
The matrices Y i − E [ Y i ] Y_i-\mathbb{E}[Y_i] Y i − E [ Y i ] are independent and have mean zero. Matrix Bernstein ([106 Theorem 6.1.1] ) therefore gives
P ( ∥ 1 N ∑ i = 1 N ( Y i − E [ Y i ] ) ∥ o p ≥ 1 2 ) ≤ 2 d exp ( − N / 4 2 D ϕ 2 / λ + D ϕ 2 / ( 3 λ ) ) = 2 d exp ( − 3 N λ 28 D ϕ 2 ) = δ . \begin{aligned}
\mathbb{P}\left(\left\|\frac{1}{N}\sum_{i=1}^{N}(Y_i-\mathbb{E}[Y_i])\right\|_{\mathrm{op}}\ge\frac{1}{2}\right)\le2d\exp\left(-\frac{N/4}{2D_\phi^2/\lambda+D_\phi^2/(3\lambda)}\right) \\
&=2d\exp\left(-\frac{3N\lambda}{28D_\phi^2}\right)=\delta.
\end{aligned} P N 1 i = 1 ∑ N ( Y i − E [ Y i ]) op ≥ 2 1 ≤ 2 d exp ( − 2 D ϕ 2 / λ + D ϕ 2 / ( 3 λ ) N /4 ) = 2 d exp ( − 28 D ϕ 2 3 N λ ) = δ .
Thus, with probability at least 1 − δ 1-\delta 1 − δ , ∥ 1 N ∑ i = 1 N ( Y i − E [ Y i ] ) ∥ o p < 1 2 \left\|\frac{1}{N}\sum_{i=1}^{N}(Y_i-\mathbb{E}[Y_i])\right\|_{\mathrm{op}}<\frac{1}{2} N 1 ∑ i = 1 N ( Y i − E [ Y i ]) op < 2 1 . On this event, Eq. (318) gives
( Σ t O n + λ I ) − 1 / 2 ( Σ ^ t O n − Σ t O n ) ( Σ t O n + λ I ) − 1 / 2 ⪰ − 1 2 I . (319) \left(\Sigma^{\mathrm{On}}_t+\lambda I\right)^{-1/2}\left(\widehat{\Sigma}^{\mathrm{On}}_t-\Sigma^{\mathrm{On}}_t\right)\left(\Sigma^{\mathrm{On}}_t+\lambda I\right)^{-1/2}\succeq-\frac{1}{2}I. \tag*{(319)} ( Σ t On + λ I ) − 1/2 ( Σ t On − Σ t On ) ( Σ t On + λ I ) − 1/2 ⪰ − 2 1 I . (319)
Multiplying both sides of Eq. (319) on the left and right by ( Σ t O n + λ I ) 1 / 2 (\Sigma^{\mathrm{On}}_t+\lambda I)^{1/2} ( Σ t On + λ I ) 1/2 gives
Σ ^ t O n − Σ t O n ⪰ − 1 2 ( Σ t O n + λ I ) , \widehat{\Sigma}^{\mathrm{On}}_t-\Sigma^{\mathrm{On}}_t\succeq-\frac{1}{2}(\Sigma^{\mathrm{On}}_t+\lambda I), Σ t On − Σ t On ⪰ − 2 1 ( Σ t On + λ I ) ,
which rearranges to Eq. (317) . Using Eq. (317) in Eqs. (312) and (313) gives
L t R e b e l − O n ( π θ ) ≤ 2 L ^ t R e b e l − O n ( π θ ) + λ η 2 ∥ θ − θ t − η θ ⋆ ∥ 2 2 = 2 L ^ t R e b e l − O n ( π θ ) + 28 D ϕ 2 3 η 2 N log 2 d δ ∥ θ − θ t − η θ ⋆ ∥ 2 2 . \begin{aligned}
L_t^{\mathrm{Rebel-On}}(\pi_\theta)\le2\widehat{L}_t^{\mathrm{Rebel-On}}(\pi_\theta)+\frac{\lambda}{\eta^2}\left\|\theta-\theta_t-\eta\theta_\star\right\|_2^2 \\
&=2\widehat{L}_t^{\mathrm{Rebel-On}}(\pi_\theta)+\frac{28D_\phi^2}{3\eta^2N}\log\frac{2d}{\delta}\left\|\theta-\theta_t-\eta\theta_\star\right\|_2^2.
\end{aligned} L t Rebel − On ( π θ ) ≤ 2 L t Rebel − On ( π θ ) + η 2 λ ∥ θ − θ t − η θ ⋆ ∥ 2 2 = 2 L t Rebel − On ( π θ ) + 3 η 2 N 28 D ϕ 2 log δ 2 d ∥ θ − θ t − η θ ⋆ ∥ 2 2 .
The event depends only on the batch, so this bound holds simultaneously for every θ ∈ R d \theta\in\mathbb{R}^d θ ∈ R d .
Inner Loop Convergence
We next prove the convergence of the inner optimization loop in Algorithm 5.
Theorem L.1 (Inner loop convergence of Rebel-OnPolicy). Fix an outer iteration t t t , any η > 0 \eta> 0 η > 0 , ε i n S q ∈ ( 0 , D ⋆ 2 ] \varepsilon_{\mathrm{in}}^{\mathrm{Sq}} \in(0, D_{\star}^{2}] ε in Sq ∈ ( 0 , D ⋆ 2 ] , and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Run the inner loop of Algorithm 5 with
N ≥ 28 D ⋆ 2 3 ε i n S q log 2 d δ , K ≥ 2 D ⋆ 2 ε i n S q , α = η 2 2 D ϕ 2 . (320) N \geq\frac{28D_{\star}^{2}}{3\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\log\frac{2d}{\delta}, \qquad K \geq\frac{2D_{\star}^{2}}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}, \qquad\alpha= \frac{\eta^{2}}{2D_{\phi}^{2}}. \tag*{(320)} N ≥ 3 ε in Sq 28 D ⋆ 2 log δ 2 d , K ≥ ε in Sq 2 D ⋆ 2 , α = 2 D ϕ 2 η 2 . (320) With probability at least 1 − δ 1-\delta 1 − δ , the returned policy satisfies
MSE t ( π t K ) ≤ ε i n S q , ∥ s t π t K ∥ ∞ ≤ D ⋆ . \operatorname{MSE}_{t}(\pi_{t}^{K}) \leq\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}, \qquad\left\lVert s_{t}^{\pi_{t}^{K}}\right\rVert_{\infty} \leq D_{\star}. MSE t ( π t K ) ≤ ε in Sq , s t π t K ∞ ≤ D ⋆ .
Because every empirical realization of L ^ t R e b e l − O n \widehat{L}_{t}^{\mathrm{Rebel-On}} L t Rebel − On is convex and is minimized by π η , t ⋆ \pi_{\eta,t}^{\star} π η , t ⋆ , Lemma L.2, the proof below first proves convergence of gradient descent on the empirical objective. It then transfers this to the population objective by Lemma L.4 . Because the empirical covariance Σ ^ t O n \widehat{\Sigma}_{t}^{\mathrm{On}} Σ t On is the only estimated term in L ^ t R e b e l − O n \widehat{L}_{t}^{\mathrm{Rebel-On}} L t Rebel − On , regardless of the parameter θ \theta θ , the concentration inequality in Lemma L.4 scales with log d \log d log d and not d d d .
Proof of Theorem L.1. For every realization of the batch, Lemmas L.2 and L.3 show that L ^ t R e b e l − O n ( π θ ) \widehat{L}_{t}^{\mathrm{Rebel-On}}(\pi_{\theta}) L t Rebel − On ( π θ ) is convex and 2 D ϕ 2 / η 2 2D_{\phi}^{2}/\eta^{2} 2 D ϕ 2 / η 2 -smooth, with minimum value zero at θ = θ t + η θ ⋆ \theta=\theta_{t}+\eta\theta_{\star} θ = θ t + η θ ⋆ . Applying Theorem B.1 from θ t 0 = θ t \theta_{t}^{0}=\theta_{t} θ t 0 = θ t with learning rate α = η 2 / ( 2 D ϕ 2 ) \alpha=\eta^{2}/(2D_{\phi}^{2}) α = η 2 / ( 2 D ϕ 2 ) gives
L ^ t R e b e l − O n ( π t K ) ≤ ∥ θ t − ( θ t + η θ ⋆ ) ∥ 2 2 2 α K = D ⋆ 2 K . (321) \widehat{L}_{t}^{\mathrm{Rebel-On}}(\pi_{t}^{K}) \leq\frac{\left\lVert\theta_{t}-(\theta_{t}+\eta\theta_{\star})\right\rVert_{2}^{2}}{2\alpha K}=\frac{D_{\star}^{2}}{K}. \tag*{(321)} L t Rebel − On ( π t K ) ≤ 2 α K ∥ θ t − ( θ t + η θ ⋆ ) ∥ 2 2 = K D ⋆ 2 . (321)
We next bound MSE t ( π t K ) \operatorname{MSE}_{t}(\pi_{t}^{K}) MSE t ( π t K ) in terms of L ^ t R e b e l − O n \widehat{L}_{t}^{\mathrm{Rebel-On}} L t Rebel − On using Lemma L.4 , which holds uniformly over θ \theta θ and therefore applies to θ t K \theta_{t}^{K} θ t K . The lemma’s requirement on N N N follows from ε i n S q ≤ D ⋆ 2 \varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\leq D_{\star}^{2} ε in Sq ≤ D ⋆ 2 and theorem conditions in Eq. (320) . Together with Lemma L.1, this gives, with probability at least 1 − δ 1-\delta 1 − δ ,
MSE t ( π t K ) = 1 2 L ^ t R e b e l − O n ( π t K ) ≤ L ^ t R e b e l − O n ( π t K ) + 14 D ϕ 2 3 η 2 N log 2 d δ ∥ θ t K − θ t − η θ ⋆ ∥ 2 2 . (322) \operatorname{MSE}_{t}(\pi_{t}^{K})=\frac{1}{2}\widehat{L}_{t}^{\mathrm{Rebel-On}}(\pi_{t}^{K})
\leq\widehat{L}_{t}^{\mathrm{Rebel-On}}(\pi_{t}^{K})+\frac{14D_{\phi}^{2}}{3\eta^{2}N}\log\frac{2d}{\delta}\left\lVert\theta_{t}^{K}-\theta_{t}-\eta\theta_{\star}\right\rVert_{2}^{2}. \tag*{(322)} MSE t ( π t K ) = 2 1 L t Rebel − On ( π t K ) ≤ L t Rebel − On ( π t K ) + 3 η 2 N 14 D ϕ 2 log δ 2 d θ t K − θ t − η θ ⋆ 2 2 . (322)
We condition on this event for the remainder of the proof.
We now bound ∥ θ t K − θ t − η θ ⋆ ∥ 2 \left\lVert\theta_{t}^{K}-\theta_{t}-\eta\theta_{\star}\right\rVert_{2} θ t K − θ t − η θ ⋆ 2 in Eq. (322) . Differentiating Eq. (313) gives
∇ L ^ t R e b e l − O n ( π θ ) = 2 η 2 Σ ^ t O n ( θ − θ t − η θ ⋆ ) . \nabla\widehat{L}_{t}^{\mathrm{Rebel-On}}(\pi_{\theta})=\frac{2}{\eta^{2}}\widehat{\Sigma}_{t}^{\mathrm{On}}(\theta-\theta_{t}-\eta\theta_{\star}). ∇ L t Rebel − On ( π θ ) = η 2 2 Σ t On ( θ − θ t − η θ ⋆ ) .
Substituting into the gradient descent update in Line 8 of Algorithm 5 with α = η 2 / ( 2 D ϕ 2 ) \alpha=\eta^{2}/(2D_{\phi}^{2}) α = η 2 / ( 2 D ϕ 2 ) gives
θ t k + 1 − θ t − η θ ⋆ = ( I − Σ ^ t O n D ϕ 2 ) ( θ t k − θ t − η θ ⋆ ) . (323) \theta_{t}^{k+1}-\theta_{t}-\eta\theta_{\star}=\left(I-\frac{\widehat{\Sigma}_{t}^{\mathrm{On}}}{D_{\phi}^{2}}\right)(\theta_{t}^{k}-\theta_{t}-\eta\theta_{\star}). \tag*{(323)} θ t k + 1 − θ t − η θ ⋆ = ( I − D ϕ 2 Σ t On ) ( θ t k − θ t − η θ ⋆ ) . (323)
By Eq. (316) in the proof of Lemma L.3 , 0 ⪯ Σ ^ t O n / D ϕ 2 ⪯ I 0\preceq\widehat{\Sigma}_{t}^{\mathrm{On}}/D_{\phi}^{2}\preceq I 0 ⪯ Σ t On / D ϕ 2 ⪯ I , so ∥ I − Σ ^ t O n / D ϕ 2 ∥ o p ≤ 1 \left\lVert I-\widehat{\Sigma}_{t}^{\mathrm{On}}/D_{\phi}^{2}\right\rVert_{\mathrm{op}}\leq1 I − Σ t On / D ϕ 2 op ≤ 1 . Iterating Eq. (323) from θ t 0 = θ t \theta_{t}^{0}=\theta_{t} θ t 0 = θ t therefore gives
∥ θ t k − θ t − η θ ⋆ ∥ 2 ≤ η ∥ θ ⋆ ∥ 2 = η B ⋆ , 0 ≤ k ≤ K . (324) \left\lVert\theta_{t}^{k}-\theta_{t}-\eta\theta_{\star}\right\rVert_{2}\leq\eta\left\lVert\theta_{\star}\right\rVert_{2}=\eta B_{\star}, \qquad0\leq k\leq K. \tag*{(324)} θ t k − θ t − η θ ⋆ 2 ≤ η ∥ θ ⋆ ∥ 2 = η B ⋆ , 0 ≤ k ≤ K . (324)
Substituting Eqs. (321) and (324) into Eq. (322) gives
MSE t ( π t K ) ≤ D ⋆ 2 K + 14 D ⋆ 2 3 N log 2 d δ ≤ ε i n S q , \operatorname{MSE}_{t}(\pi_{t}^{K})\leq\frac{D_{\star}^{2}}{K}+\frac{14D_{\star}^{2}}{3N}\log\frac{2d}{\delta}\leq\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}, MSE t ( π t K ) ≤ K D ⋆ 2 + 3 N 14 D ⋆ 2 log δ 2 d ≤ ε in Sq ,
where the last inequality uses the choices of K K K and N N N in Eq. (320) .
Finally, we bound the residual s t π t k s_t^{\pi_t^k} s t π t k and rewrite it under Assumption 2.1 as
s t π t k ( x , y ) = ( ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] ) ⊤ ( θ t k − θ t η − θ ⋆ ) . (325) s_t^{\pi_t^k}(x,y)=\left(\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\phi(x,y')]\right)^\top\left(\frac{\theta_t^k-\theta_t}{\eta}-\theta_\star\right). \tag*{(325)} s t π t k ( x , y ) = ( ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] ) ⊤ ( η θ t k − θ t − θ ⋆ ) . (325)
For every prompt x x x and response y y y , the definition of D ϕ D_\phi D ϕ gives
∥ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] ∥ 2 ≤ E y ′ ∼ π t ( ⋅ ∣ x ) [ ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 ] ≤ D ϕ . \left\|\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\phi(x,y')]\right\|_2\leq\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\|\phi(x,y)-\phi(x,y')\|_2]\leq D_\phi. ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] 2 ≤ E y ′ ∼ π t ( ⋅ ∣ x ) [ ∥ ϕ ( x , y ) − ϕ ( x , y ′ ) ∥ 2 ] ≤ D ϕ .
Applying Cauchy–Schwarz to Eq. (325) and using the preceding bound together with Eq. (324) gives
∥ s t π t k ∥ ∞ ≤ sup x , y ∥ ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ ) ] ∥ 2 ∥ θ t k − θ t η − θ ⋆ ∥ 2 ≤ D ϕ B ⋆ = D ⋆ , 0 ≤ k ≤ K . \begin{aligned}
\left\|s_t^{\pi_t^k}\right\|_\infty\leq\sup_{x,y}\left\|\phi(x,y)-\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}[\phi(x,y')]\right\|_2\left\|\frac{\theta_t^k-\theta_t}{\eta}-\theta_\star\right\|_2 \\
&\leq D_\phi B_\star=D_\star,\qquad0\leq k\leq K.
\end{aligned} s t π t k ∞ ≤ x , y sup ϕ ( x , y ) − E y ′ ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ′ )] 2 η θ t k − θ t − θ ⋆ 2 ≤ D ϕ B ⋆ = D ⋆ , 0 ≤ k ≤ K .
Proof of 4.2
Proof of Corollary L.1. The parameter choices in Eqs. (303) and (304) give 0 < ε i n S q ≤ ε 2 < D ⋆ 2 0<\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}\leq\varepsilon^2<D_\star^2 0 < ε in Sq ≤ ε 2 < D ⋆ 2 . At each outer iteration t t t , condition on the batches sampled in iterations 0 , … , t − 1 0,\ldots,t-1 0 , … , t − 1 . The parameter choices in Eqs. (304) and (305) satisfy the conditions of Theorem L.1 with failure probability δ / T \delta/T δ / T . A union bound therefore gives, with probability at least 1 − δ 1-\delta 1 − δ ,
MSE t ( π t + 1 ) ≤ ε i n S q , ∥ π t + 1 π t k ∥ ∞ ≤ D ⋆ , ∀ 0 ≤ t < T . (326) \operatorname{MSE}_t(\pi_{t+1})\leq\varepsilon_{\mathrm{in}}^{\mathrm{Sq}},\qquad\left\|\pi_{t+1}^{\pi_t^k}\right\|_\infty\leq D_\star,\qquad\forall\,0\leq t<T. \tag*{(326)} MSE t ( π t + 1 ) ≤ ε in Sq , π t + 1 π t k ∞ ≤ D ⋆ , ∀ 0 ≤ t < T . (326)
We condition on the event in Eq. (326) for the remainder of the proof.
Using Eq. (326) in Theorem E.1 with S = D ⋆ S=D_\star S = D ⋆ , together with E x ∼ μ [ log ( 1 / π b a s e ( Y ⋆ ∣ x ) ) ] ≤ log C b a s e \mathbb{E}_{x\sim\mu}[\log(1/\pi_{\mathrm{base}}(Y^\star\mid x))]\leq\log C_{\mathrm{base}} E x ∼ μ [ log ( 1/ π base ( Y ⋆ ∣ x ))] ≤ log C base by Jensen’s inequality and ∑ t = 1 T 1 / t ≤ log ( e T ) \sum_{t=1}^{T}1/t\leq\log(eT) ∑ t = 1 T 1/ t ≤ log ( e T ) , gives
J ⋆ − J ( π p o s t ) ≤ log C b a s e η T + e η D ⋆ T 2 ε i n S q + C b a s e log ( e T ) ε i n S q . (327) J^\star-J(\pi_{\mathrm{post}})\leq\frac{\log C_{\mathrm{base}}}{\eta T}+\frac{e^{\eta D_\star}T}{2}\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}+\sqrt{C_{\mathrm{base}}\log(eT)}\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}. \tag*{(327)} J ⋆ − J ( π post ) ≤ η T log C base + 2 e η D ⋆ T ε in Sq + C base log ( e T ) ε in Sq . (327)
Since e η D ⋆ = 2 C b a s e e^{\eta D_\star}=2C_{\mathrm{base}} e η D ⋆ = 2 C base , substituting ε i n S q \varepsilon_{\mathrm{in}}^{\mathrm{Sq}} ε in Sq from Eq. (304) into Eq. (327) yields
J ⋆ − J ( π p o s t ) ≤ log C b a s e + 1 / 12 + 1 / 12 η T ≤ log ( 2 C b a s e ) η T ≤ ε , J^\star-J(\pi_{\mathrm{post}})\leq\frac{\log C_{\mathrm{base}}+1/12+1/\sqrt{12}}{\eta T}\leq\frac{\log(2C_{\mathrm{base}})}{\eta T}\leq\varepsilon, J ⋆ − J ( π post ) ≤ η T log C base + 1/12 + 1/ 12 ≤ η T log ( 2 C base ) ≤ ε ,
where the last inequality uses the choice of T T T in Eq. (303) . On the same event, the coverage bound in Theorem E.1 with c = 1 / 2 c=1/2 c = 1/2 and the same parameter choices gives
P x ∼ μ ( ∃ 0 ≤ t ≤ T : π t ( Y ⋆ ∣ x ) < 1 2 π b a s e ( Y ⋆ ∣ x ) ) ≤ ( 2 + 2 ) ( η 2 e η D ⋆ T ε i n S q 4 + η T C b a s e ε i n S q 2 ) ≤ ( 2 + 2 ) ( 1 24 T log ( e T ) + 1 4 3 log ( e T ) ) ≤ 2 + 2 4 log ( e T ) . (328) \begin{aligned}
\mathbb{P}_{x\sim\mu}\left(\exists\,0\leq t\leq T:\ \pi_t(Y^\star\mid x)<\frac{1}{2}\pi_{\mathrm{base}}(Y^\star\mid x)\right) \\
&\leq(2+\sqrt{2})\left(\frac{\eta^2e^{\eta D_\star}T\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}{4}+\frac{\eta T\sqrt{C_{\mathrm{base}}\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}}{2}\right) \\
&\leq(2+\sqrt{2})\left(\frac{1}{24T\log(eT)}+\frac{1}{4\sqrt{3\log(eT)}}\right) \\
&\leq\frac{2+\sqrt{2}}{4\sqrt{\log(eT)}}. \tag*{(328)}
\end{aligned} P x ∼ μ ( ∃ 0 ≤ t ≤ T : π t ( Y ⋆ ∣ x ) < 2 1 π base ( Y ⋆ ∣ x ) ) ≤ ( 2 + 2 ) 4 η 2 e η D ⋆ T ε in Sq + 2 η T C base ε in Sq ≤ ( 2 + 2 ) ( 24 T log ( e T ) 1 + 4 3 log ( e T ) 1 ) ≤ 4 log ( e T ) 2 + 2 . (328)
This proves Eqs. (306) and (307) .
Finally, Algorithm 5 uses one batch of 2 N 2N 2 N responses for K K K gradient steps at each outer iteration. The choices of K K K and N N N in Eq. (305) therefore give
N s t e p ( ε , δ ) = K T = O ( T D ⋆ 2 ε i n S q ) , N_{\mathrm{step}}(\varepsilon,\delta)=KT=O\left(\frac{TD_{\star}^{2}}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\right), N step ( ε , δ ) = K T = O ( ε in Sq T D ⋆ 2 ) ,
N s a m p l e ( ε , δ ) = 2 N T = O ( T D ⋆ 2 ε i n S q log 2 d T δ ) . (329) N_{\mathrm{sample}}(\varepsilon,\delta)=2NT=O\left(\frac{TD_{\star}^{2}}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}\log\frac{2dT}{\delta}\right). \tag*{(329)} N sample ( ε , δ ) = 2 N T = O ( ε in Sq T D ⋆ 2 log δ 2 d T ) . (329)
Since ε < D ⋆ \varepsilon<D_{\star} ε < D ⋆ , the choices in Eqs. (303) and (304) give
D ⋆ ε ≤ T ≤ 2 D ⋆ ε , \frac{D_{\star}}{\varepsilon}\leq T\leq\frac{2D_{\star}}{\varepsilon}, ε D ⋆ ≤ T ≤ ε 2 D ⋆ ,
D ⋆ 2 ε i n S q = 12 C b a s e D ⋆ 2 η 2 T 2 log ( e T ) ≤ 48 C b a s e D ⋆ 2 log 2 ( 2 C b a s e ) log ( e T ) ε 2 . (330) \frac{D_{\star}^{2}}{\varepsilon_{\mathrm{in}}^{\mathrm{Sq}}}=12C_{\mathrm{base}}D_{\star}^{2}\eta^{2}T^{2}\log(eT)\leq\frac{48C_{\mathrm{base}}D_{\star}^{2}\log^{2}(2C_{\mathrm{base}})\log(eT)}{\varepsilon^{2}}. \tag*{(330)} ε in Sq D ⋆ 2 = 12 C base D ⋆ 2 η 2 T 2 log ( e T ) ≤ ε 2 48 C base D ⋆ 2 log 2 ( 2 C base ) log ( e T ) . (330)
Substituting Eq. (330) into Eq. (329) yields
K T = O ( C b a s e D ⋆ 3 log 2 ( 2 C b a s e ) log ( e T ) ε 3 ) , KT=O\left(\frac{C_{\mathrm{base}}D_{\star}^{3}\log^{2}(2C_{\mathrm{base}})\log(eT)}{\varepsilon^{3}}\right), K T = O ( ε 3 C base D ⋆ 3 log 2 ( 2 C base ) log ( e T ) ) ,
2 N T = O ( C b a s e D ⋆ 3 log 2 ( 2 C b a s e ) log ( e T ) ε 3 log 4 d D ⋆ ε δ ) . 2NT=O\left(\frac{C_{\mathrm{base}}D_{\star}^{3}\log^{2}(2C_{\mathrm{base}})\log(eT)}{\varepsilon^{3}}\log\frac{4dD_{\star}}{\varepsilon\delta}\right). 2 N T = O ( ε 3 C base D ⋆ 3 log 2 ( 2 C base ) log ( e T ) log ε δ 4 d D ⋆ ) .
Writing these bounds in O ~ \widetilde{O} O notation gives Eq. (308) .
To obtain Corollary 4.2 , let B = B ϕ B ⋆ B=B_{\phi}B_{\star} B = B ϕ B ⋆ and choose
η = log ( 2 C b a s e ) 2 B , \eta=\frac{\log(2C_{\mathrm{base}})}{2B}, η = 2 B log ( 2 C base ) ,
with the remaining parameters defined as in Corollary L.1 using this value of η \eta η . Since D ⋆ ≤ 2 B D_{\star}\leq2B D ⋆ ≤ 2 B , we have e η D ⋆ ≤ 2 C b a s e e^{\eta D_{\star}}\leq2C_{\mathrm{base}} e η D ⋆ ≤ 2 C base , while the definition of T T T still gives η T ≥ log ( 2 C b a s e ) / ε \eta T\geq\log(2C_{\mathrm{base}})/\varepsilon η T ≥ log ( 2 C base ) / ε . The same substitutions in Eqs. (327) and (328) therefore give the reward and coverage guarantees in Eqs. (306) and (307) .
Moreover, ε ≤ R max ≤ B \varepsilon\leq R_{\max}\leq B ε ≤ R m a x ≤ B implies T = ⌈ 2 B / ε ⌉ = O ( B / ε ) T=\lceil2B/\varepsilon\rceil=O(B/\varepsilon) T = ⌈ 2 B / ε ⌉ = O ( B / ε ) and η T = O ( log ( 2 C b a s e ) / ε ) \eta T=O(\log(2C_{\mathrm{base}})/\varepsilon) η T = O ( log ( 2 C base ) / ε ) . Using these bounds, D ⋆ ≤ 2 B D_{\star}\leq2B D ⋆ ≤ 2 B , and the definition of ε i n S q \varepsilon_{\mathrm{in}}^{\mathrm{Sq}} ε in Sq in Eq. (304) , the complexities in Eq. (329) give
N s t e p ( ε , δ ) = O ~ ( C b a s e B 3 ε 3 ) , N s a m p l e ( ε , δ ) = O ~ ( C b a s e B 3 ε 3 log 2 d δ ) . N_{\mathrm{step}}(\varepsilon,\delta)=\widetilde{O}\left(\frac{C_{\mathrm{base}}B^{3}}{\varepsilon^{3}}\right),\qquad
N_{\mathrm{sample}}(\varepsilon,\delta)=\widetilde{O}\left(\frac{C_{\mathrm{base}}B^{3}}{\varepsilon^{3}}\log\frac{2d}{\delta}\right). N step ( ε , δ ) = O ( ε 3 C base B 3 ) , N sample ( ε , δ ) = O ( ε 3 C base B 3 log δ 2 d ) .
Discussion: Connection to Natural Policy Gradient
Under Assumption 2.1, the objectives of § 4 are closely related to natural policy gradient (NPG) (Kakade, [45 ] ). Write a candidate policy as π θ \pi_{\theta} π θ with θ = θ t + η g \theta=\theta_t+\eta g θ = θ t + η g , and let ϕ ˉ t ( x ) : = E y ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y ) ] \bar{\phi}_t(x):=\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}[\phi(x,y)] ϕ ˉ t ( x ) := E y ∼ π t ( ⋅ ∣ x ) [ ϕ ( x , y )] . Then
1 η log π θ ( y ∣ x ) π t ( y ∣ x ) = ϕ ( x , y ) ⊤ g − 1 η log E y ′ ∼ π t ( ⋅ ∣ x ) [ e η ϕ ( x , y ′ ) ⊤ g ] . \frac{1}{\eta}\log\frac{\pi_{\theta}(y\mid x)}{\pi_t(y\mid x)}
=\phi(x,y)^{\top}g-\frac{1}{\eta}\log\mathbb{E}_{y'\sim\pi_t(\cdot\mid x)}\left[e^{\eta\phi(x,y')^{\top}g}\right]. η 1 log π t ( y ∣ x ) π θ ( y ∣ x ) = ϕ ( x , y ) ⊤ g − η 1 log E y ′ ∼ π t ( ⋅ ∣ x ) [ e η ϕ ( x , y ′ ) ⊤ g ] .
so the idealized objective Eq. (17) is a linear regression of r − b t ⋆ r-b_t^{\star} r − b t ⋆ onto the features, plus a nonlinear term from the policy’s normalizer. Both this term and b t ⋆ ( x ) b_t^{\star}(x) b t ⋆ ( x ) are log-mean-exponentials, and replacing each by its first-order approximation in η \eta η (ϕ ˉ t ( x ) ⊤ g \bar{\phi}_t(x)^{\top}g ϕ ˉ t ( x ) ⊤ g and r ( x , π t ) r(x,\pi_t) r ( x , π t ) , respectively) turns the objective into centered linear regression,
min g E π t [ ( ( ϕ ( x , y ) − ϕ ˉ t ( x ) ) ⊤ g − ( r ( x , y ) − r ( x , π t ) ) ) 2 ] . (331) \min_g\ \mathbb{E}_{\pi_t}\left[\left((\phi(x,y)-\bar{\phi}_t(x))^{\top}g-(r(x,y)-r(x,\pi_t))\right)^2\right]. \tag*{(331)} g min E π t [ ( ( ϕ ( x , y ) − ϕ ˉ t ( x ) ) ⊤ g − ( r ( x , y ) − r ( x , π t )) ) 2 ] . (331)
Since centering removes the normalizer, the objective in Eq. (331) is exactly MSE t ( π θ ) \operatorname{MSE}_t(\pi_\theta) MSE t ( π θ ) , and its minimizers satisfy F t g = ∇ J ( π t ) F_tg = \nabla J(\pi_t) F t g = ∇ J ( π t ) , where F t : = E π t [ ( ϕ − ϕ ˉ t ) ( ϕ − ϕ ˉ t ) T ] F_t := \mathbb{E}_{\pi_t}\left[(\phi-\bar{\phi}_t)(\phi-\bar{\phi}_t)^{\mathsf{T}}\right] F t := E π t [ ( ϕ − ϕ ˉ t ) ( ϕ − ϕ ˉ t ) T ] is the Fisher information matrix. Reward realizability gives ∇ J ( π t ) = F t θ ⋆ \nabla J(\pi_t) = F_t\theta_\star ∇ J ( π t ) = F t θ ⋆ , so g = θ ⋆ g = \theta_\star g = θ ⋆ is a minimizer and yields the NPG update [45 , 1 ] ,
θ t + 1 = θ t + η θ ⋆ . \theta_{t+1} = \theta_t + \eta\theta_\star. θ t + 1 = θ t + η θ ⋆ .
Exact minimization of the population Rebel-OnPolicy objective therefore implements an NPG step. The mean-baseline objective of KimiPO adds a nonlinear correction; we emphasize however that this connection is only precise under Assumption 2.1 and the softmax linear parameterization.