Introduction
Learning a transferable representation of expert behavior makes it possible to predict how the expert would act in previously unseen circumstances. Inverse Reinforcement Learning (IRL) [1 ] formalizes this by inferring a reward function from expert demonstrations. Unlike imitation learning methods such as Behavior Cloning (BC) [2 ] , which directly replicate observed behavior, IRL aims to recover the underlying objective the expert is optimizing. In contrast to a cloned policy tied to its training dynamics, a reward can, in principle, transfer to unseen dynamics.
Despite its appeal, IRL is fundamentally ill-posed [1 ] . Under entropy regularization [3 ] , the standard setting in identifiability works [4 , 5 ] , demonstrations from a fixed Markov decision process (MDP) identify the reward only up to subspace specific to that MDP. Consequently, such rewards transfer only to dynamics whose identifiable subspace sufficiently overlaps with that of the original MDP [6 ] . Recent works show that observing expert demonstrations in as few as two sufficiently different dynamics resolves this ambiguity [4 , 5 , 6 ] . However, these works leave open how to actively select such dynamics to reduce the ambiguity with as few expert demonstrations as possible.
In practice, expert demonstrations are costly, and only a fixed set of dynamics is available. We must therefore choose carefully which dynamics to query the expert in. This raises the following central question:
How should we select dynamics to learn a transferable reward from a minimal number of expert demonstrations?
To address this question, we propose an active IRL setting, in which the learner chooses the dynamics under which the expert demonstrates the task. This setting is natural whenever reconfiguring the environment is cheap compared to collecting expert demonstrations. For example, a robotics lab can cheaply vary the initial placement, mass, or friction of manipulated objects [7 ] , or a driving simulator can vary road conditions [8 ] ; collecting demonstrations, by contrast, requires a skilled human operator for each one [9 ] .
Closest to our work, Buening et al. [10 ] adaptively design dynamics to identify the expert’s reward. They empirically show that, compared to randomly chosen environments, this yields more transferable rewards from fewer demonstrations, but provide no theoretical guarantees. To the best of our knowledge, our work provides the first active IRL algorithms with sample complexity guarantees for learning transferable rewards through dynamics selection.
Contributions. Our main contributions are:
We formalize active IRL as a sequential problem: over n n n rounds, the learner chooses dynamics from a finite set on which to request an expert demonstration, aiming to recover a reward that transfers as much as possible across the set (Equation (1) ).
We derive novel confidence sets for the expert reward (Lemma 3.1) and show that transfer to each dynamics is controlled by how informative they are relative to the information already accumulated (Lemma 3.2).
This yields a two-phase algorithm: cycle through the set of dynamics until the reward estimate is reliable (Lemma 3.3), then query the expert where information gain is greatest. After n n n rounds of dynamics selection and expert demonstrations, this results in a transfer error of O ~ ( 1 / n ) \widetilde{O}(1/n) O ( 1/ n ) across the set of available dynamics, independent of its cardinality (Theorem 3.6). The same rate extends to any dynamics, up to an irreducible bias (Theorem 3.7).
We validate our approach empirically against imitation learning and IRL baselines, as well as dynamics design for IRL [10 ] . Our active approach learns an equally transferable reward using up to five times fewer demonstrations than the baselines, and its learned reward generalizes best to unseen dynamics (§ 4 ).
Problem formulation
Notations. Let ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 denote the Euclidean norm. For a symmetric positive semidefinite matrix X ∈ R d × d X \in\mathbb{R}^{d \times d} X ∈ R d × d , ∥ X ∥ 2 \lVert X\rVert_2 ∥ X ∥ 2 denotes its spectral norm, X † X^\dagger X † its Moore–Penrose inverse, and ∥ v ∥ X : = ⟨ v , X v ⟩ \lVert v\rVert_X := \sqrt{\langle v, Xv\rangle} ∥ v ∥ X := ⟨ v , X v ⟩ the associated Mahalanobis norm of a vector v ∈ R d v \in\mathbb{R}^d v ∈ R d . For a finite set Z \mathcal{Z} Z , Δ Z \Delta^{\mathcal{Z}} Δ Z denotes the probability simplex over Z \mathcal{Z} Z .
Regularized MDPs. A regularized MDP is a tuple M = ( S , A , T , ν 1 , r , H , η ) \mathcal{M} = (\mathcal{S}, \mathcal{A}, T, \nu_1, r, H, \eta) M = ( S , A , T , ν 1 , r , H , η ) . Here, S \mathcal{S} S and A \mathcal{A} A are finite state and action spaces, T : S × A → Δ S T : \mathcal{S} \times\mathcal{A} \to\Delta^{\mathcal{S}} T : S × A → Δ S is the transition function, ν 1 ∈ Δ S \nu_1 \in\Delta^{\mathcal{S}} ν 1 ∈ Δ S denotes the initial state distribution, r : S × A → R r : \mathcal{S} \times\mathcal{A} \to\mathbb{R} r : S × A → R denotes the reward, and H ∈ N + H \in\mathbb{N}^{+} H ∈ N + is the horizon. We consider entropy-regularized MDPs, with η > 0 \eta> 0 η > 0 denoting the inverse temperature that controls the strength of the regularization. Starting from state s 1 ∼ ν 1 s_1 \sim\nu_1 s 1 ∼ ν 1 , an agent interacting with the MDP takes at each timestep h h h an action a h ∈ A a_h \in\mathcal{A} a h ∈ A , receives the reward r ( s h , a h ) r(s_h, a_h) r ( s h , a h ) and the environment transitions to a new state s h + 1 ∼ T ( ⋅ ∣ s h , a h ) s_{h+1} \sim T(\cdot\mid s_h, a_h) s h + 1 ∼ T ( ⋅ ∣ s h , a h ) .
Denoting the dynamics by P = ( ν 1 , T ) P = (\nu_1, T) P = ( ν 1 , T ) , the optimal solution of M \mathcal{M} M is a policy π = ( π h ) h = 1 H \pi= (\pi_h)_{h=1}^{H} π = ( π h ) h = 1 H , with π h : S → Δ A \pi_h : \mathcal{S} \to\Delta^{\mathcal{A}} π h : S → Δ A , maximizing the regularized value V P π ( r ) : = E P T [ ∑ h = 1 H r ( s h , a h ) − η − 1 log ( π h ( a h ∣ s h ) ) ] V^\pi_P(r) := \mathbb{E}^T_P\left[\sum_{h=1}^{H} r(s_h,a_h) - \eta^{-1}\log\bigl(\pi_h(a_h\mid s_h)\bigr)\right] V P π ( r ) := E P T [ ∑ h = 1 H r ( s h , a h ) − η − 1 log ( π h ( a h ∣ s h ) ) ] . A state s s s is unreachable at timestep h h h if no policy visits it with positive probability at h h h . As a convention, all policies follow an arbitrarily fixed action distribution (e.g., uniform) on unreachable states, which leaves trajectory laws and values unchanged. The MDP M \mathcal{M} M then admits a unique optimal policy [3 ] . For a reward r r r and dynamics P P P , we write V P ∗ ( r ) V^*_P(r) V P ∗ ( r ) and π P ( r ) \pi_P(r) π P ( r ) for the corresponding optimal value and policy.
Setting. Given an unknown reward r E r^{\mathrm{E}} r E , an expert follows the optimal policy π P E : = π P ( r E ) \pi_P^{\mathrm{E}} := \pi_P(r^{\mathrm{E}}) π P E := π P ( r E ) under any chosen dynamics P P P . We consider a fixed, finite family of MDPs { M i } i \{\mathcal{M}_i\}_i { M i } i that share r E r^{\mathrm{E}} r E but differ in their dynamics P : = { P i } i \mathcal{P} := \{P_i\}_i P := { P i } i . A demonstration in M i \mathcal{M}_i M i is a trajectory τ = ( ( s h , a h ) ) h = 1 H \tau= ((s_h,a_h))_{h=1}^{H} τ = (( s h , a h ) ) h = 1 H generated by π P i E \pi_{P_i}^{\mathrm{E}} π P i E .
We consider an active setting in which a learner selects dynamics sequentially, and an expert is queried after each selection. Over a budget of n ∈ N + n \in\mathbb{N}^{+} n ∈ N + rounds, the learner selects at each round k k k dynamics P k ∈ P P_k \in\mathcal{P} P k ∈ P , and observes a batch of m m m expert trajectories generated under P k P_k P k , for a total of n m nm nm trajectories, gathering them in a dataset D E : = ⋃ k = 1 n { τ k , j } j = 1 m \mathcal{D}^{\mathrm{E}} := \bigcup_{k=1}^{n}\{\tau_{k,j}\}_{j=1}^{m} D E := ⋃ k = 1 n { τ k , j } j = 1 m .
Objective. Given this budget, our goal is to select dynamics ( P k ) k = 1 n (P_k)_{k=1}^{n} ( P k ) k = 1 n from P \mathcal{P} P and estimate a reward r r r from D E \mathcal{D}^{\mathrm{E}} D E that is transferable across P \mathcal{P} P . We quantify transferability by the worst-case suboptimality,
max P ∈ P SubOpt ( r , P ) = max P ∈ P V P ∗ ( r E ) − V P π P ( r ) ( r E ) , (1) \max_{P\in\mathcal{P}} \operatorname{SubOpt}(r,P) = \max_{P\in\mathcal{P}} V^*_P(r^{\mathrm{E}}) - V^{\pi_P(r)}_P(r^{\mathrm{E}}), \tag*{(1)} P ∈ P max SubOpt ( r , P ) = P ∈ P max V P ∗ ( r E ) − V P π P ( r ) ( r E ) , (1)
and we seek a reward r r r minimizing (1). This objective captures the ground-truth performance of the recovered optimal policy π P ( r ) \pi_P(r) π P ( r ) across P \mathcal{P} P and coincides with the notion of transferability studied by Schlaginhaufen and Kamgarpour [6 ] for fixed dynamics. Furthermore, it can be viewed as a regularized analogue of the imitation gap studied in imitation learning [11 ] .
Structural assumption. Before presenting the algorithm for selecting the dynamics P k P_k P k and estimating the reward, we introduce our main structural assumption, which enables efficient estimation of the expert reward r E r^{\mathrm{E}} r E from expert demonstrations D E \mathcal{D}^{\mathrm{E}} D E .
Assumption 2.1 (Expert bounded linear rewards). Let ϕ : S × A → R d \phi: \mathcal{S} \times\mathcal{A} \to\mathbb{R}^{d} ϕ : S × A → R d be a known feature map with ∥ ϕ ( s , a ) ∥ 2 ≤ B ϕ \lVert\phi(s,a)\rVert_2 \le B_\phi ∥ ϕ ( s , a ) ∥ 2 ≤ B ϕ for all ( s , a ) ∈ S × A (s,a) \in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A , and let Θ : = { θ ∈ R d : ∥ θ ∥ 2 ≤ B θ } \Theta:= \{\theta\in\mathbb{R}^{d} : \lVert\theta\rVert_2 \le B_\theta\} Θ := { θ ∈ R d : ∥ θ ∥ 2 ≤ B θ } , where B ϕ , B θ > 0 B_\phi,B_\theta> 0 B ϕ , B θ > 0 are known constants. We assume the expert reward is linear in unknown parameters θ E ∈ Θ \theta^{\mathrm{E}} \in\Theta θ E ∈ Θ , i.e. r E ( s , a ) = ⟨ θ E , ϕ ( s , a ) ⟩ r^{\mathrm{E}}(s,a) = \langle\theta^{\mathrm{E}},\phi(s,a)\rangle r E ( s , a ) = ⟨ θ E , ϕ ( s , a )⟩ .
Linear rewards are standard in RL [12 ] and inverse RL [1 ] . We index rewards r θ = ⟨ θ , ϕ ⟩ r_\theta= \langle\theta,\phi\rangle r θ = ⟨ θ , ϕ ⟩ by their parame-
Algorithm 1 Active IRL
1: input: set of dynamics $\mathcal{P}$, warm-up length $n_0$
2: // Warm-up phase
3: for round $k = 1,\ldots,n_0$ do
4: Select $P_k$ round-robin over $\mathcal{P}$ (6)
5: Observe $m$ trajectories in $P_k$
6: Estimate $\theta_{n_0}$ by minimizing (2) (Section 3.1)
7: // Active phase
8: for round $k = n_0+1,\ldots,n$ do
9: Actively select $P_k$ via (7) (Section 3.2)
10: Observe $m$ trajectories in $P_k$
11: Estimate $\theta_k$ by minimizing (2) (Section 3.1)
ters θ \theta θ , and write ϕ ( τ ) = ∑ ( s , a ) ∈ τ ϕ ( s , a ) \phi(\tau)=\sum_{(s,a)\in\tau}\phi(s,a) ϕ ( τ ) = ∑ ( s , a ) ∈ τ ϕ ( s , a ) for trajectory features.
Finally, we assume that the expert is optimal for θ E \theta^{\mathrm{E}} θ E , with a known temperature. Appendix G extends our results to misspecified experts, e.g., when the assumed temperature is incorrect, in which case all our transferability results hold with an additional error proportional to the expert’s suboptimality.
Actively learning transferable rewards
We now present our active method for learning transferable rewards minimizing (1), summarized in Algorithm 1. Its key idea is to query the expert under the most informative dynamics, i.e., those revealing parts of the reward that matter for transfer across P \mathcal{P} P but remain poorly determined by previous demonstrations.
The method proceeds in two phases. In a warm-up phase, the learner has too little data to tell which dynamics are informative, so it queries the expert under each available dynamics in turn. It then switches to an active phase, in which it uses its reward estimate to select the most informative dynamics.
To make this strategy precise, we first quantify the accuracy of reward estimates from adaptively collected demonstrations through ellipsoidal confidence bounds (§ 3.1 ). These bounds determine when the estimate is reliable enough to guide dynamics selection. In § 3.2 , we use them to construct the two-phase selection algorithm, before deriving transfer guarantees for the learned reward in § 3.3 .
Reward estimation
To select informative dynamics, we need to quantify how much information each dynamics provides about θ E \theta^{\mathrm{E}} θ E , and how much has already been accumulated. We first introduce the loss and use its curvature to characterize the information provided under each dynamics.
We then show how the accumulated information yields an ellipsoidal confidence bound for θ E \theta^{\mathrm{E}} θ E (Lemma 3.1). Proofs are deferred to Appendix B.
IRL loss. After k k k rounds, we estimate θ E \theta^{\mathrm{E}} θ E by minimizing the empirical loss over Θ \Theta Θ ,
L k ( θ ) : = ∑ i = 1 k l i ( θ ) , l i ( θ ) : = V P i ∗ ( Π P i θ ) − ⟨ θ , Π P i ϕ ^ i ⟩ , (2) L_k(\theta) := \sum_{i=1}^{k} l_i(\theta), \qquad l_i(\theta) := V_{P_i}^{*}(\Pi_{P_i}\theta)-\langle\theta,\Pi_{P_i}\hat{\phi}_i\rangle, \tag*{(2)} L k ( θ ) := i = 1 ∑ k l i ( θ ) , l i ( θ ) := V P i ∗ ( Π P i θ ) − ⟨ θ , Π P i ϕ ^ i ⟩ , (2)
and write θ k ∈ arg min θ ∈ Θ L k ( θ ) \theta_k\in\arg\min_{\theta\in\Theta}L_k(\theta) θ k ∈ arg min θ ∈ Θ L k ( θ ) . Here P i P_i P i are the dynamics selected at round i i i , ϕ ^ i = 1 m ∑ j = 1 m ϕ ( τ i , j ) \hat{\phi}_i=\frac{1}{m}\sum_{j=1}^{m}\phi(\tau_{i,j}) ϕ ^ i = m 1 ∑ j = 1 m ϕ ( τ i , j ) is the empirical mean of the expert’s trajectory features, and Π P i \Pi_{P_i} Π P i is the orthogonal projector onto the identifiable subspace U P i U_{P_i} U P i defined below. Without the projections, each l i l_i l i is an entropy-regularized version of the min-max IRL loss [13 ] under P i P_i P i .
Curvature and identifiability. The optimal policy induced by a reward depends on the underlying dynamics. Hence, by selecting the dynamics under which the expert demonstrates, the learner chooses which parts of the reward can be estimated, and how efficiently. We characterize this through the Hessian of the optimal value function,
H P ( θ ) : = ∇ θ 2 V P ∗ ( θ ) , \mathcal{H}_{P}(\theta):=\nabla_{\theta}^{2}V_{P}^{*}(\theta), H P ( θ ) := ∇ θ 2 V P ∗ ( θ ) ,
which is the information provided by P P P . It is also the curvature of the per-round loss l i l_i l i when P = P i P=P_i P = P i , since it differs from V P i ∗ V_{P_i}^{*} V P i ∗ only by a linear term. For a direction v ∈ R d v\in\mathbb{R}^{d} v ∈ R d , the curvature ∥ v ∥ H P ( θ ) 2 \|v\|_{\mathcal{H}_{P}(\theta)}^{2} ∥ v ∥ H P ( θ ) 2 quantifies the sensitivity of the optimal policy π P ( θ ) \pi_P(\theta) π P ( θ ) to reward perturbations along v v v . The larger the curvature, the fewer demonstrations under P P P are needed to estimate the reward along v v v . With zero curvature, the reward along v v v cannot be estimated under P P P , regardless of how many trajectories are collected. Such directions arise from potential shaping [14 ] and from reward changes at unreachable states. We therefore define the identifiable subspace U P : = im H P ( θ ) U_P:=\operatorname{im}\mathcal{H}_P(\theta) U P := im H P ( θ ) , the orthogonal complement of the zero-curvature directions.3 Demonstrations under P P P only inform the components of θ E \theta^{\mathrm{E}} θ E in U P U_P U P . The projections in (2) thus prevent sampling noise in ϕ ^ i \hat{\phi}_i ϕ ^ i from drifting the estimate along unidentifiable directions (Appendix B). Across all available dynamics, the identifiable components of θ E \theta^{\mathrm{E}} θ E lie in the joint identifiable subspace U : = ∑ P ∈ P U P U:=\sum_{P\in\mathcal{P}}U_P U := ∑ P ∈ P U P , of dimension d U ≤ d d_U\le d d U ≤ d .
Confidence sequence. After k k k rounds, the information accumulated about θ E \theta^{\mathrm{E}} θ E is given by the curvature of L k L_k L k (2) at θ E \theta^{\mathrm{E}} θ E and its smallest eigenvalue on U U U ,
H k : = ∑ i = 1 k H P i ( θ E ) , λ k : = min v ∈ U ∥ v ∥ 2 = 1 ∥ v ∥ H k 2 , \mathcal{H}_k:=\sum_{i=1}^{k}\mathcal{H}_{P_i}(\theta^{\mathrm{E}}), \qquad\lambda_k:=\min_{\substack{v\in U\\ \|v\|_2=1}}\|v\|_{\mathcal{H}_k}^{2}, H k := i = 1 ∑ k H P i ( θ E ) , λ k := v ∈ U ∥ v ∥ 2 = 1 min ∥ v ∥ H k 2 ,
where λ k \lambda_k λ k measures the information in the least-informed identifiable direction. Our main estimation result is a confidence sequence for θ E \theta^{\mathrm{E}} θ E , whose radius depends on the effective dimension
d ⋆ : = d U max P ∈ P ∥ H P ( θ E ) † / 2 Σ P H P ( θ E ) † / 2 ∥ 2 , (3) d_\star:= d_U \max_{P \in\mathcal{P}} \left\| \mathcal{H}_P(\theta^{\mathrm{E}})^{\dagger/2} \Sigma_P \mathcal{H}_P(\theta^{\mathrm{E}})^{\dagger/2} \right\|_2, \tag*{(3)} d ⋆ := d U P ∈ P max H P ( θ E ) † /2 Σ P H P ( θ E ) † /2 2 , (3)
where Σ P : = Cov π P E ( Π P ϕ ( τ ) ) \Sigma_P := \operatorname{Cov}_{\pi_P}^{\mathrm{E}}(\Pi_P \phi(\tau)) Σ P := Cov π P E ( Π P ϕ ( τ )) is the covariance of the projected expert trajectory features.
Lemma 3.1. Fix n ∈ N + n \in\mathbb{N}^{+} n ∈ N + , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and R : = η B ϕ H ( H + 1 ) R := \eta B_\phi H(H+1) R := η B ϕ H ( H + 1 ) . Let n δ n_\delta n δ be the first round k ≤ n k \le n k ≤ n such that
λ k ≥ 32 R 2 d ⋆ ( 1 − e − 1 ) 2 m log 3 η d ⋆ n m δ , (4) \lambda_k \ge\frac{32R^2d_\star}{(1-e^{-1})^2m}\log\frac{3\eta d_\star nm}{\delta}, \tag*{(4)} λ k ≥ ( 1 − e − 1 ) 2 m 32 R 2 d ⋆ log δ 3 η d ⋆ nm , (4) with n δ = ∞ n_\delta= \infty n δ = ∞ if no such round exists. Then, with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for all n δ ≤ k ≤ n n_\delta\le k \le n n δ ≤ k ≤ n ,
∥ θ k − θ E ∥ H k 2 ≤ 16 d ⋆ ( 1 − e − 1 ) 2 m log 3 η d ⋆ k m δ = : β k 2 ( δ ) . \left\|\theta_k-\theta^{\mathrm{E}}\right\|_{\mathcal{H}_k}^2 \le\frac{16d_\star}{(1-e^{-1})^2m}\log\frac{3\eta d_\star km}{\delta} =: \beta_k^2(\delta). θ k − θ E H k 2 ≤ ( 1 − e − 1 ) 2 m 16 d ⋆ log δ 3 η d ⋆ k m =: β k 2 ( δ ) .
Proof sketch. We combine concentration of the projected empirical features under adaptive selection with the pseudo self-concordance of L k L_k L k [15 ] . After the warm-up, the localization approach of Schlaginhaufen and Kamgarpour [16 ] converts sampling variation into an ellipsoidal parameter-error bound.
Lemma 3.1 quantifies the uncertainty in our reward estimate through a confidence ellipsoid shaped by the accumulated information H k \mathcal{H}_k H k . The ellipsoid is wide along directions where little information has been collected, and narrow where much has. Querying the expert under P P P adds H P ( θ E ) \mathcal{H}_P(\theta^{\mathrm{E}}) H P ( θ E ) to the accumulated information, and is therefore informative when P P P provides information along directions where the ellipsoid is wide.
The ellipsoid, however, is only guaranteed to contain θ E \theta^{\mathrm{E}} θ E once enough information has been accumulated in every identifiable direction, as prescribed by the warm-up condition (4). Since L k L_k L k is nonquadratic, its curvature changes with the reward parameters at a rate controlled by R R R [15 ] . The warm-up ensures that θ k \theta_k θ k lies in a region where the curvature remains comparable to that at θ E \theta^{\mathrm{E}} θ E , allowing the approximately local quadratic analysis that yields the confidence bound [17 , 18 ] .
The effective dimension d ⋆ d_\star d ⋆ compares the variability of the expert’s features to the information they carry, and plays the role of the parameter dimension in the confidence radius. It satisfies d ⋆ ≥ d U / η d_\star\ge d_U/\eta d ⋆ ≥ d U / η , with equality under deterministic dynamics, where the projections also have no effect on the loss (2). Together with the curvature variation controlled by R R R , it determines the required information level (4). Through R 2 R^2 R 2 , the threshold has an explicit quartic dependence on the horizon H H H and quadratic dependence on the feature scale B ϕ B_\phi B ϕ . Up to logarithmic factors, both the threshold and the squared confidence radius β k 2 ( δ ) \beta_k^2(\delta) β k 2 ( δ ) scale as 1 / m 1/m 1/ m . Larger batches therefore relax the warm-up requirement and shrink the confidence ellipsoid by reducing the sampling noise in the empirical features.
Lemma 3.1 thus splits dynamics selection into two goals: first accumulating enough information in every direction of U U U to complete the warm-up, then adding information where it matters for transfer across P \mathcal{P} P , which we address next.
Dynamics selection
We now use the confidence sequence to select dynamics so that the learned reward transfers across P \mathcal{P} P (1) . We bound the suboptimality under each P ∈ P P \in\mathcal{P} P ∈ P , then derive a two-phase algorithm querying the dynamics where this bound is worst. Proofs are in Appendix C.
From information to suboptimality. Lemma 3.1 bounds the error θ k − θ E \theta_k-\theta^{\mathrm{E}} θ k − θ E in the geometry of the accumulated information H k \mathcal{H}_k H k . Transfer to P P P , however, is governed by the geometry of H P ( θ E ) \mathcal{H}_P(\theta^{\mathrm{E}}) H P ( θ E ) , which measures how sensitive the optimal policy under P P P is to reward errors (Section 3.1). We compare the two through the information gain of P P P ,
g k ( P ) : = ∥ H k † / 2 H P ( θ E ) H k † / 2 ∥ 2 , (5) g_k(P) := \left\| \mathcal{H}_k^{\dagger/2}\mathcal{H}_P(\theta^{\mathrm{E}})\mathcal{H}_k^{\dagger/2} \right\|_2, \tag*{(5)} g k ( P ) := H k † /2 H P ( θ E ) H k † /2 2 , (5)
which is large when P P P provides much information along directions where little has been accumulated. Errors along such directions remain large, and since they are important for transfer under P P P , they are amplified in the suboptimality under P P P .
Lemma 3.2. Fix n ∈ N + n \in\mathbb{N}^{+} n ∈ N + and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . With probability at least 1 − δ 1-\delta 1 − δ , it holds for all P ∈ P P \in\mathcal{P} P ∈ P and n δ ≤ k ≤ n n_\delta\le k \le n n δ ≤ k ≤ n that
SubOpt ( θ k , P ) ≤ g k ( P ) β k 2 ( δ ) . \operatorname{SubOpt}(\theta_k,P) \le g_k(P)\beta_k^2(\delta). SubOpt ( θ k , P ) ≤ g k ( P ) β k 2 ( δ ) .
Proof sketch. Let u k : = θ k − θ E u_k := \theta_k-\theta^{\mathrm{E}} u k := θ k − θ E . The suboptimality is the Bregman divergence of V P ⋆ V_P^\star V P ⋆ between θ E \theta^{\mathrm{E}} θ E and θ k \theta_k θ k , i.e., SubOpt ( θ k , P ) = ∫ 0 1 ( 1 − t ) ∥ u k ∥ H P ( θ k − t u k ) 2 d t \operatorname{SubOpt}(\theta_k,P) = \int_0^1 (1-t)\left\|u_k\right\|_{\mathcal{H}_P(\theta_k-tu_k)}^2\,\mathrm{d}t SubOpt ( θ k , P ) = ∫ 0 1 ( 1 − t ) ∥ u k ∥ H P ( θ k − t u k ) 2 d t . On the event of Lemma 3.1, for every P ∈ P P \in\mathcal{P} P ∈ P and n δ ≤ k ≤ n n_\delta\le k \le n n δ ≤ k ≤ n ,
SubOpt ( θ k , P ) ≲ (i) ∥ u k ∥ H P ( θ E ) 2 ≤ (ii) g k ( P ) ∥ u k ∥ H k 2 . \operatorname{SubOpt}(\theta_k,P) \underset{\text{(i)}}{\lesssim} \left\|u_k\right\|_{\mathcal{H}_P(\theta^{\mathrm{E}})}^2 \underset{\text{(ii)}}{\le} g_k(P)\left\|u_k\right\|_{\mathcal{H}_k}^2. SubOpt ( θ k , P ) (i) ≲ ∥ u k ∥ H P ( θ E ) 2 (ii) ≤ g k ( P ) ∥ u k ∥ H k 2 .
(i) holds by pseudo self-concordance of V P ⋆ V_P^\star V P ⋆ , since θ k \theta_k θ k is close to θ E \theta^{\mathrm{E}} θ E after the warm-up. (ii) changes the geometry from H P ( θ E ) \mathcal{H}_P(\theta^{\mathrm{E}}) H P ( θ E ) to H k \mathcal{H}_k H k at a cost g k ( P ) g_k(P) g k ( P ) . Lemma 3.1 bounds the last term by β k 2 ( δ ) \beta_k^2(\delta) β k 2 ( δ ) .
Lemma 3.2 suggests that transfer is hardest to the dynamics with the most information left to gain, which are thus the most valuable to query. Since β k 2 ( δ ) \beta_k^2(\delta) β k 2 ( δ ) does not depend on dynamics, our goal is to make g k ( P ) g_k(P) g k ( P ) small across P \mathcal{P} P . However, g k ( P ) g_k(P) g k ( P ) depends on the unknown θ E \theta^{\mathrm{E}} θ E , and plugging in θ k \theta_k θ k is only reliable after the warm-up, so we first collect data passively.
Warm-up. During the warm-up, we cycle through the available dynamics using a round-robin rule that does not rely on reward estimates. Writing k ( P ) k(P) k ( P ) for the number of times P P P was selected in the first k k k rounds, we choose the next dynamics according to
P k + 1 ∈ arg min P ∈ P k ( P ) . (6) P_{k+1} \in\arg\min_{P\in\mathcal{P}} k(P). \tag*{(6)} P k + 1 ∈ arg P ∈ P min k ( P ) . (6)
This simple rule guarantees that the warm-up condition is met within a bounded number of rounds.
Lemma 3.3. Fix n ∈ N + n \in\mathbb{N}^{+} n ∈ N + and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Under round-robin selection (6), the warm-up length satisfies
n δ ≤ ∣ P ∣ ⌈ 2 R 2 β n 2 ( δ ) λ ∣ P ∣ ⌉ , n_{\delta} \leq|\mathcal{P}|\left\lceil\frac{2R^{2}\beta_{n}^{2}(\delta)}{\lambda_{|\mathcal{P}|}}\right\rceil, n δ ≤ ∣ P ∣ ⌈ λ ∣ P ∣ 2 R 2 β n 2 ( δ ) ⌉ , whenever the right-hand side is at most n n n .
Proof sketch. After k k k rounds, every P ∈ P P \in\mathcal{P} P ∈ P was picked at least ⌊ k / ∣ P ∣ ⌋ \lfloor k/|\mathcal{P}|\rfloor ⌊ k /∣ P ∣ ⌋ times, so λ k ≥ ⌊ k / ∣ P ∣ ⌋ λ ∣ P ∣ \lambda_k \geq\lfloor k/|\mathcal{P}|\rfloor\lambda_{|\mathcal{P}|} λ k ≥ ⌊ k /∣ P ∣ ⌋ λ ∣ P ∣ .
The threshold (4) equals 2 R 2 β n 2 ( δ ) 2R^{2}\beta_n^{2}(\delta) 2 R 2 β n 2 ( δ ) , and λ ∣ P ∣ \lambda_{|\mathcal{P}|} λ ∣ P ∣ is the information in the least-informed direction after one sweep over P \mathcal{P} P . The bound thus counts the sweeps needed before Lemma 3.1 holds. It depends on the unknown θ E \theta^{\mathrm{E}} θ E , although a loose θ E \theta^{\mathrm{E}} θ E -independent version can be computed (Appendix F). A similar dependence appears in active learning for maximum-likelihood estimation [17 ] , where a passive phase provides an initial estimate of the Fisher information used to select informative labels. Here, the warm-up makes the estimated Hessians reliable enough to guide dynamics selection. Being worst-case, the bound is conservative, and Section 4 shows that the active phase performs well after a much shorter warm-up.
Active selection. After the warm-up, θ k \theta_k θ k is localized around θ E \theta^{\mathrm{E}} θ E (Lemma 3.1). This is not yet enough to transfer well to dynamics that rely on weakly informed directions, but it is enough to estimate information gains, since the Hessians at θ k \theta_k θ k then match those at θ E \theta^{\mathrm{E}} θ E up to a constant factor. We thus estimate g k ( P ) g_k(P) g k ( P ) by g ^ k ( P ) \hat{g}_k(P) g ^ k ( P ) , defined as in (5) with Hessians at θ k \theta_k θ k instead of θ E \theta^{\mathrm{E}} θ E , and query the dynamics with the largest estimated information gain,
P k + 1 ∈ arg max P ∈ P g ^ k ( P ) . (7) P_{k+1} \in\arg\max_{P\in\mathcal{P}} \hat{g}_k(P). \tag*{(7)} P k + 1 ∈ arg P ∈ P max g ^ k ( P ) . (7)
Querying the selected dynamics adds its information to H k H_k H k , lowering its information gain as well as that of every dynamics sharing its identifiable directions. Section 3.3 shows that this rule drives the worst-case suboptimality down across P \mathcal{P} P .
Remark 3.4. Minimizing max P ∈ P g k ( P ) \max_{P\in\mathcal{P}} g_k(P) max P ∈ P g k ( P ) relates to G-optimal designs [19 , 20 ] . If each H P ( θ ) H_P(\theta) H P ( θ ) were rank-one and parameter-independent, our rule (7) would reduce to greedy D-optimal design. More generally, Theorem 3.6 still holds if the spectral norm in the selection rule (7) is replaced by the trace or by log det ( I + ⋅ ) \log\det(I + \cdot) log det ( I + ⋅ ) . Our experiments use the latter.
Remark 3.5 (Offline active learning). When sequential interaction is impractical, one can select all dynamics in advance by using θ 0 = 0 \theta_0 = 0 θ 0 = 0 in place of θ k \theta_k θ k (Appendix E).
Computational complexity. The information H P ( θ ) = η E [ A P θ ( τ ) A P θ ( τ ) T ] H_P(\theta) = \eta\,\mathbb{E}\left[A_P^\theta(\tau)A_P^\theta(\tau)^{\mathsf{T}}\right] H P ( θ ) = η E [ A P θ ( τ ) A P θ ( τ ) T ] is the second moment of the feature advantages A P θ ( τ ) A_P^\theta(\tau) A P θ ( τ ) along trajectories of π P ( θ ) \pi_P(\theta) π P ( θ ) (Appendix B). It is estimated by sampling such trajectories and computing advantages through one policy evaluation. Since U P U_P U P does not depend on θ \theta θ , each projector Π P = H P ( 0 ) H P ( 0 ) † \Pi_P = H_P(0)H_P(0)^\dagger Π P = H P ( 0 ) H P ( 0 ) † is computable once at the start, using the uniform policy π P ( 0 ) \pi_P(0) π P ( 0 ) (Appendix H). As in any IRL method, the cost of Algorithm 1 is dominated by the RL solves in lines 6 and 11. Active selection only adds line 9, which reuses the policies computed by IRL to estimate the information, whereas Buening et al. [10 ] require O ( k ∣ P ∣ ) O(k|\mathcal{P}|) O ( k ∣ P ∣ ) RL solves at round k k k to choose dynamics.
Transfer guarantees
We now show that our two-phase approach yields a reward that transfers across P \mathcal{P} P at a rate independent of ∣ P ∣ |\mathcal{P}| ∣ P ∣ , and extend this guarantee to dynamics outside P \mathcal{P} P . Proofs of this section are deferred to Appendix D.
Transfer across P \mathcal{P} P . The following theorem shows that, as active rounds accumulate, less and less information remains to be gained under any P ∈ P P \in\mathcal{P} P ∈ P , and the worst-case suboptimality decreases accordingly.
Theorem 3.6. Fix n ∈ N + n \in\mathbb{N}^{+} n ∈ N + , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and follow the active rule (7) after round n δ n_{\delta} n δ . With probability at least 1 − δ 1-\delta 1 − δ , it holds for all n δ < k ≤ n n_{\delta} < k \leq n n δ < k ≤ n and P ∈ P P \in\mathcal{P} P ∈ P that
g k ( P ) ≲ ( 1 + m η d ⋆ H 2 ) d U log ( k m ) k − n δ . g_k(P) \lesssim\left(1+\frac{m}{\eta d_{\star}H^2}\right)\frac{d_U\log(km)}{k-n_{\delta}}. g k ( P ) ≲ ( 1 + η d ⋆ H 2 m ) k − n δ d U log ( k m ) . If moreover m ≤ η d ⋆ H 2 m \leq\eta d_{\star}H^2 m ≤ η d ⋆ H 2 and n ≥ 2 n δ n \geq2n_{\delta} n ≥ 2 n δ ,
SubOpt ( θ n , P ) ≲ d ⋆ d U log ( n m ) log ( η d ⋆ n m / δ ) n m . (8) \operatorname{SubOpt}(\theta_n,P) \lesssim\frac{d_{\star}d_U\log(nm)\log(\eta d_{\star}nm/\delta)}{nm}. \tag*{(8)} SubOpt ( θ n , P ) ≲ nm d ⋆ d U log ( nm ) log ( η d ⋆ nm / δ ) . (8) Proof sketch. For all P ∈ P P \in\mathcal{P} P ∈ P , H P ( θ k ) H_P(\theta_k) H P ( θ k ) and H P ( θ E ) H_P(\theta^{\mathrm{E}}) H P ( θ E ) are within an ϵ \epsilon ϵ factor of each other. An elliptical potential argument relating the spectral norm to the log-determinant bounds g k ( P ) g_k(P) g k ( P ) for all P ∈ P P \in\mathcal{P} P ∈ P . For m ≤ η d ⋆ H 2 m \leq\eta d_{\star}H^2 m ≤ η d ⋆ H 2 , the front factor is at most 2, and n ≥ 2 n δ n \geq2n_{\delta} n ≥ 2 n δ gives n − n δ ≥ n / 2 n-n_{\delta} \geq n/2 n − n δ ≥ n /2 . Lemma 3.2 then yields (8).
Theorem 3.6 shows that, after the warm-up, the information gain of every P ∈ P P \in\mathcal{P} P ∈ P decays inversely with
the number of active rounds k − n δ k - n_{\delta} k − n δ , so that progressively no dynamics in P \mathcal{P} P is left with much to reveal. With at most η d ⋆ H 2 \eta d_{\star} H^{2} η d ⋆ H 2 trajectories per round, and as long as the warm-up uses at most half the budget, the returned reward thus achieves a worst-case suboptimality of O ~ ( d ⋆ d U / ( n m ) ) \widetilde{O}(d_{\star}d_{U}/(nm)) O ( d ⋆ d U / ( nm )) over P \mathcal{P} P , i.e., inversely proportional to the total number of trajectories.
Importantly, ∣ P ∣ |\mathcal{P}| ∣ P ∣ only affects the warm-up length (Lemma 3.3), not the suboptimality rate. Round-robin instead pays for ∣ P ∣ |\mathcal{P}| ∣ P ∣ at every round. If only one P ∈ P P \in\mathcal{P} P ∈ P is informative and the others are redundant, our analysis only guarantees a suboptimality of O ~ ( d ⋆ ∣ P ∣ / ( n m ) ) \widetilde{O}(d_{\star}|\mathcal{P}|/(nm)) O ( d ⋆ ∣ P ∣/ ( nm )) for round-robin, as it keeps querying dynamics with little information to offer. Active selection thus replaces the number of dynamics ∣ P ∣ |\mathcal{P}| ∣ P ∣ by the dimension d U d_{U} d U , an improvement whenever ∣ P ∣ > d U |\mathcal{P}| > d_{U} ∣ P ∣ > d U .
Intuitively, the condition m ≤ η d ⋆ H 2 m \le\eta d_{\star} H^{2} m ≤ η d ⋆ H 2 states that when reconfiguring the dynamics is costly, the expert can demonstrate up to η d ⋆ H 2 \eta d_{\star} H^{2} η d ⋆ H 2 trajectories per round without loss. Beyond this threshold, the noise in the empirical features ϕ ^ i \hat{\phi}_{i} ϕ ^ i in (2) still shrinks, but the information each round contributes is set by its dynamics and does not grow with m m m , so our bound no longer improves.
Transfer beyond P \mathcal{P} P . We now extend our guarantee to target dynamics P P P outside P \mathcal{P} P . Such dynamics may require information along directions that no dynamics in P \mathcal{P} P identifies, which we measure by g ⊥ ( P ) : = ∥ Π U ⊥ H P ( θ E ) Π U ⊥ ∥ 2 g_{\perp}(P) := \lVert\Pi_{U^{\perp}} \mathcal{H}_{P}(\theta^{E}) \Pi_{U^{\perp}} \rVert_{2} g ⊥ ( P ) := ∥ Π U ⊥ H P ( θ E ) Π U ⊥ ∥ 2 , where U ⊥ U^{\perp} U ⊥ is the orthogonal complement of the identifiable subspace U U U .
Theorem 3.7. Fix δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and follow the active rule (7) after round n δ n_{\delta} n δ . If m ≤ η d ⋆ H 2 m \le\eta d_{\star} H^{2} m ≤ η d ⋆ H 2 and n ≥ 2 n δ n \ge2n_{\delta} n ≥ 2 n δ , then with probability at least 1 − δ 1-\delta 1 − δ , for any P P P ,
SubOpt ( θ n , P ) = O ~ ( d ⋆ d U n m ) + O ( g ⊥ ( P ) B θ 2 ) . \operatorname{SubOpt}(\theta_{n},P) = \widetilde{O}\left(\frac{d_{\star}d_{U}}{nm}\right) + O\left(g_{\perp}(P)B_{\theta}^{2}\right). SubOpt ( θ n , P ) = O ( nm d ⋆ d U ) + O ( g ⊥ ( P ) B θ 2 ) .
Proof sketch. Let θ n − θ E = u + v \theta_{n}-\theta^{E}=u+v θ n − θ E = u + v with u ∈ U u \in U u ∈ U and v ∈ U ⊥ v \in U^{\perp} v ∈ U ⊥ . Theorem 3.6 bounds the error along u u u , up to how much information P P P provides along the directions that matter under P P P . Since ∥ v ∥ 2 ≤ 2 B θ \lVert v\rVert_{2} \le2B_{\theta} ∥ v ∥ 2 ≤ 2 B θ , we have ∥ v ∥ H P ( θ E ) 2 ≤ 4 g ⊥ ( P ) B θ 2 \lVert v\rVert_{\mathcal{H}_{P}(\theta^{E})}^{2} \le4g_{\perp}(P)B_{\theta}^{2} ∥ v ∥ H P ( θ E ) 2 ≤ 4 g ⊥ ( P ) B θ 2 . Pseudo self-concordance then yields the result, stated in full in Appendix § 11 . □
Theorem 3.7 splits suboptimality under any P P P into two terms. The first shrinks with the number of demonstrations, and corresponds to directions shared by U P U_{P} U P and U U U . The second is a transfer bias, and corresponds to directions that no dynamics in P \mathcal{P} P cover. It therefore cannot be reduced by observing the expert in P \mathcal{P} P . Interestingly, our active rule helps reduce the first term, but only the choice of P P P can mitigate the second. When U P ⊆ U U_{P} \subseteq U U P ⊆ U , the bias term vanishes, and in particular for P ∈ P P \in\mathcal{P} P ∈ P , we exactly recover (8).
Experiments
Our experiments serve two purposes: (i) to evaluate when active learning improves sample efficiency in terms of expert demonstrations, and (ii) to identify which dynamics are most useful for efficiently learning transferable rewards.
We instantiate our two-phase strategy (§ 3.2 ) with the log-determinant criterion (Remark 3.4), which performed best among valid criteria we compare in Appendix § 16 . Since n δ n_{\delta} n δ cannot be computed, we treat the warm-up length as a hyperparameter, set to one third of the budget, and study other choices in Appendix § 16 . We compare against baselines that isolate the benefits of (a) the warm-up phase, (b) leveraging multiple dynamics, and (c) learning transferable rewards rather than directly imitating behavior.
To evaluate (a), we run an Active only ablation that follows the active rule (7) from the start.4 To evaluate (b), we compare against the environment design (ED) approach of Buening et al. [10 ] , a round-robin baseline following (6), and methods that learn from one fixed dynamics in P \mathcal{P} P : Fixed dynamics IRL and GAIL [21 ] . All IRL methods except GAIL optimize the min-max IRL objective (2). Finally, to evaluate (c), we compare against imitation methods: Behavioral Cloning (BC), trained on a single dynamics like the methods above, and robust imitation via RIME [22 ] .
All methods follow the same evaluation protocol. At each round k k k , the learner selects dynamics P k ∈ P P_{k} \in\mathcal{P} P k ∈ P and observes m = 1 m = 1 m = 1 expert trajectory, generated by rolling out, under P k P_{k} P k , the entropy-regularized optimal policy for the true reward parameters θ E \theta^{E} θ E . For fixed-dynamics methods, the environment is uniformly sampled once from P \mathcal{P} P and then fixed. Evaluation proceeds by learning a reward from collected data and deploying a policy optimized for each tested dynamics; imitation methods (BC, RIME) instead deploy a single policy across all dynamics. In all experiments, state-action features are one-hot encodings over states and actions.
We measure performance by the worst-case suboptimality (1) and by the reward error up to constants, min c ∈ R ∥ θ E − θ n + c 1 ∥ 2 \min_{c \in\mathbb{R}}\lVert\theta^{E}-\theta_{n}+c\mathbf{1}\rVert_{2} min c ∈ R ∥ θ E − θ n + c 1 ∥ 2 . We report means and standard errors over 20 randomly seeded runs. These experiments focus on tractable environments that require no approximation, leaving scalability to future work.
Controlled sets of dynamics
We construct training sets P \mathcal{P} P in which we control how well rewards learned in one dynamics transfer to the others. This is quantified by principal angles between unidentifiable subspaces [23 , 6 ] (see also Appendix § 16 .§ 2 ).
We consider four types of sets: (1) Equivalent dynamics, where rewards transfer perfectly across P \mathcal{P} P ; (2) Similar dynamics, where rewards transfer consistently well; (3) Uniformly diverse dynamics, where transfer is uniformly weak across pairs; (4) Arbitrarily diverse dynamics, where transferability is heterogeneous across the set. For each random seed, we generate a set of ∣ P ∣ = 10 |\mathcal{P}| = 10 ∣ P ∣ = 10 dynamics. After n = 100 n = 100 n = 100 demonstrations, methods are also evaluated on unseen dynamics. The MDPs all have ∣ S ∣ = 20 |\mathcal{S}| = 20 ∣ S ∣ = 20 states and ∣ A ∣ = 5 |\mathcal{A}| = 5 ∣ A ∣ = 5 actions.
Results. With equivalent dynamics (Figure 1 (a)), all methods perform similarly. Since every dynamics reveals the same information about the reward, dynamics selection offers no advantage, and imitation methods remain competitive with reward learning. As dynamics become more diverse (Figure 1 (b)–(Figure 1 d)), fixed-dynamics methods (IRL, GAIL, and BC) increasingly stall. For reward learning, this is the bias of Theorem 3.7 with a single training dynamics, which grows with the diversity of P \mathcal{P} P . Active selection is most beneficial in arbitrarily diverse sets (Figure 1 (d)). There, the Active only variant performs best, both on P \mathcal{P} P and on unseen dynamics, where it substantially outperforms ED (Table 1). This suggests that our warm-up length is conservative, a gap we discuss in § 6 .
Figure 1. Worst-case suboptimality over the training set P \mathcal{P} P as a function of the number of observed expert trajectories on randomly generated MDPs. Each plot corresponds to a different type of training set.
Figure 2 shows which dynamics are selected. In uniformly diverse sets, selection closely matches round-robin. In arbitrarily diverse sets, it concentrates on dynamics that either transfer well on average across P \mathcal{P} P (P 1 , … , P 5 P_1,\ldots,P_5 P 1 , … , P 5 ) or are highly isolated from the rest (P 10 P_{10} P 10 ).
Figure 2. Number of times each dynamics in the training set P \mathcal{P} P is selected after 100 rounds, for two types of training set. Dynamics are ordered by increasing average dissimilarity to the rest of P \mathcal{P} P , so that P 10 P_{10} P 10 corresponds to the most isolated dynamics.
Random mazes
We next consider a maze navigation task [10 ] , where the agent is rewarded for reaching and waiting in one of three goal states. This task makes the selected dynamics easy to visualize (Figure 3 ). We generate sets of ∣ P ∣ = 100 |\mathcal{P}| = 100 ∣ P ∣ = 100 mazes by randomly placing walls, ranging from nearly empty layouts to highly constrained or even unsolvable ones (e.g., blocked goals). All mazes in a set share the initial state and the reward, with goal rewards sampled uniformly in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Here, ∣ S ∣ = 49 |\mathcal{S}| = 49 ∣ S ∣ = 49 and ∣ A ∣ = 5 |\mathcal{A}| = 5 ∣ A ∣ = 5 (four moves and a waiting action).
Figure 3. Mazes most frequently selected by Active only after 100 rounds, on a representative seed. The blue triangle marks the initial state and stars mark goal states, labeled with their expert rewards. The heatmap shows the state occupancy of policies optimized for the inferred reward.
Results. Active selection is clearly more sample-efficient (Figure 4 ). Without warm-up, it reaches with 20 trajectories the suboptimality that round-robin attains with 100. The two-phase strategy catches up after about 60 trajectories, again suggesting that the warm-up is conservative.
Figure 4. Worst-case suboptimality over the training set P \mathcal{P} P of 100 random mazes, and reward error, as a function of the number of observed expert trajectories.
Figure 3 shows that the active rule often selects unsolvable mazes. When goals are all unreachable but the layout remains open, the entropy-regularized expert acts uniformly, revealing that the reward is constant across reachable non-goal states. A maze can thus be informative without being solvable.
Related Work
Active IRL aims to minimize expert interactions, the main practical bottleneck, by querying the expert selectively. Prior work has mostly focused on fixed dynamics. Lopes et al. [24 ] and Cui and Niekum [25 ] select states or trajectories that maximize information about the reward, Bajgar et al. [26 ] provide PAC guarantees for such Bayesian state queries, and Lindner et al. [27 ] use confidence sets over feasible rewards to guide exploration. Our setting instead queries the expert under chosen dynamics.
Table 1. Worst-case suboptimality (× 100 \times100 × 100 ) over 1000 unseen dynamics after learning from n = 100 n = 100 n = 100 expert trajectories, collected on either a uniformly or an arbitrarily diverse training set of dynamics.
Learning from given dynamics. In the entropy-regularized setting, sufficiently diverse dynamics allow recovering rewards up to constants when the expert policy is known [4 , 5 ] . For transfer from demonstrations, Schlaginhaufen and Kamgarpour [6 ] give sufficient conditions, and Hao et al. [28 ] derive finite-sample guarantees from one source to a given target environment. These results require the dynamics to be well covered, through ergodicity [4 , 5 , 6 ] or through source data covering the target’s occupancy [28 ] , whereas none of our results require either assumption. Our confidence sets extend the fast rates of Schlaginhaufen and Kamgarpour [16 ] from fixed dynamics to adaptively selected ones. Without regularization, rewards are only partially identifiable, and recent work selects rewards that remain robust to this ambiguity [29 , 30 ] . Beyond reward learning, Chae et al. [22 ] learn a single policy robust to dynamics variations by imitating experts on randomly sampled dynamics. None of these works lets the learner choose the dynamics in which to collect demonstrations.
Choosing dynamics
Amin et al. [31 ] show that an agent choosing the tasks can identify the expert’s reward when the expert policy is revealed in each, and bound the number of mistakes when tasks are chosen adversarially. Nitschke et al. [32 ] maximize behavior entropy, Bajgar et al. [33 ] maximize expected information gain, and Buening et al. [10 ] target where the current reward estimate is expected to perform worst. These methods focus on state-only rewards or deterministic experts, rely on expensive selection rules, or lack sample complexity guarantees for dynamics selection. In contrast, our work provides a practical and provably efficient framework for actively choosing dynamics to learn transferable rewards.
Conclusion
We introduced an active learning method for inferring transferable rewards, which adaptively selects the dynamics under which the expert demonstrates. After a round-robin warm-up, it queries the expert under the dynamics with the largest information gain. The resulting reward transfers across all available dynamics at a rate governed by the dimension of the identifiable reward subspace rather than their number, and to unseen dynamics up to a bias from the directions left uncovered. Empirically, it requires substantially fewer demonstrations than passive selection and outperforms the most closely related active approach [10 ] .
Limitations and future work. First, our guarantees rely on a conservative warm-up, while active selection performs well without it in practice, possibly because it naturally first selects the dynamics informing directions not yet covered. Additional assumptions on the information gathered, ensuring that our local curvature argument applies early, or data-driven switching criteria could close this gap. Second, IRL across multiple dynamics requires one optimal policy per dynamics, which is costly for large ∣ P ∣ |\mathcal{P}| ∣ P ∣ , and dynamics-conditioned policies could amortize this cost. Finally, our study is restricted to finite sets of tabular environments and linear rewards. A promising direction is to model rewards with neural networks whose last layer is linear, using the last hidden layer as learned low-dimensional features ϕ \phi ϕ and computing Hessians with respect to the last layer only, as in Ash et al. [34 ] .
Checklist
For all models and algorithms presented, check if you include:
(a) A clear description of the mathematical setting, assumptions, algorithm, and/or model. Yes. § 2 for the setting, assumptions and models. § 3.2 for algorithms.
(b) An analysis of the properties and complexity (time, space, sample size) of any algorithm. Yes. Sample complexity is analyzed in Lemma 3.3 and Theorem 3.6, and computational complexity in the “Computational complexity” paragraph of § 3.2 and Appendix § 15 .
(c) (Optional) Anonymized source code, with specification of all dependencies, including external libraries. Yes. Link to the anonymized source code provided in Appendix § 16 .
For any theoretical claim, check if you include:
(a) Statements of the full set of assumptions of all theoretical results. Yes. See § 2 .
(b) Complete proofs of all theoretical results. Yes, all proofs are provided in Appendices § 9 and § 10 .
(c) Clear explanations of any assumptions. Yes.
For all figures and tables that present empirical results, check if you include:
(a) The code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL). Yes. We provide a link to the code used in experiments in Appendix I.
(b) All the training details (e.g., data splits, hyperparameters, how they were chosen). Yes, hyperparameters are specified both in Appendix I, and through the opensourced code used for our experiments.
(c) A clear definition of the specific measure or statistics and error bars (e.g., with respect to the random seed after running experiments multiple times). Yes. We consistently provide the number of seeds used, and what error bars represent in both the main text (Section 4) and appendix.
(d) A description of the computing infrastructure used. (e.g., type of GPUs, internal cluster, or cloud provider). Yes. Information is included in Appendix I.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets, check if you include:
(a) Citations of the creator If your work uses existing assets. Not Applicable.
(b) The license information of the assets, if applicable. Not Applicable.
(c) New assets either in the supplemental material or as a URL, if applicable. Not applicable.
(d) Information about consent from data providers/curators. Not Applicable.
(e) Discussion of sensible content if applicable, e.g., personally identifiable information or offensive content. Not Applicable.
If you used crowdsourcing or conducted research with human subjects, check if you include:
(a) The full text of instructions given to participants and screenshots. Not Applicable.
(b) Descriptions of potential participant risks, with links to Institutional Review Board (IRB) approvals if applicable. Not Applicable.
(c) The estimated hourly wage paid to participants and the total amount spent on participant compensation. Not Applicable.
Notations and Preliminaries
Notations
We enumerate below the most important notations used in our appendix.
Table 1 (PDF p. 12).
Preliminaries
MDPs. Throughout this appendix, we work under the same regularized MDP setup introduced in Section 2. For a policy π = ( π h ) h = 1 H \pi=(\pi_h)_{h=1}^{H} π = ( π h ) h = 1 H , we define the state and state-action occupancies at each timestep h ∈ { 1 , … , H } h\in\{1,\ldots,H\} h ∈ { 1 , … , H } as
ν h , P π ( s ) : = P P π [ s h = s ] , μ h , P π ( s , a ) : = ν h , P π ( s ) π h ( a ∣ s ) . \nu^{\pi}_{h,P}(s):=\mathbb{P}^{\pi}_{P}[s_h=s],\qquad\mu^{\pi}_{h,P}(s,a):=\nu^{\pi}_{h,P}(s)\pi_h(a\mid s). ν h , P π ( s ) := P P π [ s h = s ] , μ h , P π ( s , a ) := ν h , P π ( s ) π h ( a ∣ s ) .
The regularized value then reads
V P π ( r ) = ∑ h = 1 H ( ⟨ r , μ h , P π ⟩ + 1 η Ent h ( π ) ) , Ent h ( π ) : = − ∑ s , a μ h , P π ( s , a ) log π h ( a ∣ s ) , V^{\pi}_{P}(r)=\sum_{h=1}^{H}\left(\langle r,\mu^{\pi}_{h,P}\rangle+\frac{1}{\eta}\operatorname{Ent}_{h}(\pi)\right),\qquad\operatorname{Ent}_{h}(\pi):=-\sum_{s,a}\mu^{\pi}_{h,P}(s,a)\log\pi_h(a\mid s), V P π ( r ) = h = 1 ∑ H ( ⟨ r , μ h , P π ⟩ + η 1 Ent h ( π ) ) , Ent h ( π ) := − s , a ∑ μ h , P π ( s , a ) log π h ( a ∣ s ) ,
where Ent h ( π ) \operatorname{Ent}_{h}(\pi) Ent h ( π ) is the expected Shannon entropy of π \pi π at timestep h h h under P P P . The optimal policy in regularized MDPs has a softmax characterization [3 ] ,
π h , P θ ( a ∣ s ) = exp ( η Q h , P θ ( s , a ) ) ∑ a ′ ∈ A exp ( η Q h , P θ ( s , a ′ ) ) (9) \pi^{\theta}_{h,P}(a\mid s)=\frac{\exp\left(\eta Q^{\theta}_{h,P}(s,a)\right)}{\sum_{a'\in\mathcal{A}}\exp\left(\eta Q^{\theta}_{h,P}(s,a')\right)} \tag*{(9)} π h , P θ ( a ∣ s ) = ∑ a ′ ∈ A exp ( η Q h , P θ ( s , a ′ ) ) exp ( η Q h , P θ ( s , a ) ) (9)
where Q h , P θ Q^{\theta}_{h,P} Q h , P θ and V h , P θ V^{\theta}_{h,P} V h , P θ are the optimal regularized Q Q Q -function and value, defined by backward recursion from V H + 1 , P θ : = 0 V^{\theta}_{H+1,P}:=0 V H + 1 , P θ := 0 ,
Q h , P θ ( s , a ) : = ⟨ θ , ϕ ( s , a ) ⟩ + ∑ s ′ ∈ S T ( s ′ ∣ s , a ) V h + 1 , P θ ( s ′ ) , V h , P θ ( s ) : = 1 η log ∑ a ∈ A exp ( η Q h , P θ ( s , a ) ) . Q^{\theta}_{h,P}(s,a):=\langle\theta,\phi(s,a)\rangle+\sum_{s'\in\mathcal{S}}T(s'\mid s,a)V^{\theta}_{h+1,P}(s'),\qquad V^{\theta}_{h,P}(s):=\frac{1}{\eta}\log\sum_{a\in\mathcal{A}}\exp\left(\eta Q^{\theta}_{h,P}(s,a)\right). Q h , P θ ( s , a ) := ⟨ θ , ϕ ( s , a )⟩ + s ′ ∈ S ∑ T ( s ′ ∣ s , a ) V h + 1 , P θ ( s ′ ) , V h , P θ ( s ) := η 1 log a ∈ A ∑ exp ( η Q h , P θ ( s , a ) ) .
Eigenvalues. The smallest eigenvalue of X X X is denoted as λ min ( X ) : = min ∥ v ∥ 2 = 1 ∥ v ∥ X 2 \lambda_{\min}(X):=\min_{\|v\|_2=1}\|v\|_{X}^{2} λ m i n ( X ) := min ∥ v ∥ 2 = 1 ∥ v ∥ X 2 and the smallest eigenvalue restricted to a subspace X ⊂ R d \mathcal{X}\subset\mathbb{R}^{d} X ⊂ R d as λ min X ( X ) : = min ∥ v ∥ 2 = 1 , v ∈ X ∥ v ∥ X 2 \lambda^{\mathcal{X}}_{\min}(X):=\min_{\|v\|_2=1,v\in\mathcal{X}}\|v\|_{X}^{2} λ m i n X ( X ) := min ∥ v ∥ 2 = 1 , v ∈ X ∥ v ∥ X 2 .
Features. For convenience, we use the ϕ ( τ ) = ∑ h = 1 H ϕ ( s h , a h ) \phi(\tau)=\sum_{h=1}^{H}\phi(s_h,a_h) ϕ ( τ ) = ∑ h = 1 H ϕ ( s h , a h ) to denote trajectory features, and ϕ P ( θ ) = E P π P ( θ ) [ ϕ ( τ ) ] \phi_{P}(\theta)=\mathbb{E}_{P}^{\pi_{P}(\theta)}[\phi(\tau)] ϕ P ( θ ) = E P π P ( θ ) [ ϕ ( τ )] to denote feature occupancies, with ϕ P E = ϕ P ( θ E ) \phi_{P}^{E}=\phi_{P}(\theta^{E}) ϕ P E = ϕ P ( θ E ) .
Fréchet derivatives. For a function f : R d → R f:\mathbb{R}^{d}\to\mathbb{R} f : R d → R that is three times differentiable at x ∈ R d x\in\mathbb{R}^{d} x ∈ R d , we denote the first three directional derivatives along u , v , w ∈ R d u,v,w\in\mathbb{R}^{d} u , v , w ∈ R d by
D f ( x ) [ u ] : = ⟨ ∇ f ( x ) , u ⟩ , D 2 f ( x ) [ u , v ] : = ⟨ u , ∇ 2 f ( x ) v ⟩ , Df(x)[u]:=\langle\nabla f(x),u\rangle,\qquad D^{2}f(x)[u,v]:=\langle u,\nabla^{2}f(x)v\rangle, D f ( x ) [ u ] := ⟨ ∇ f ( x ) , u ⟩ , D 2 f ( x ) [ u , v ] := ⟨ u , ∇ 2 f ( x ) v ⟩ ,
D 3 f ( x ) [ u , v , w ] : = D ( D 2 f ( x ) [ u , v ] ) [ w ] . D^{3}f(x)[u,v,w]:=D\left(D^{2}f(x)[u,v]\right)[w]. D 3 f ( x ) [ u , v , w ] := D ( D 2 f ( x ) [ u , v ] ) [ w ] .
where D n f ( x ) D^{n}f(x) D n f ( x ) denotes the n n n -th Fréchet derivative of f f f .
Proofs of section 3.1
Reward estimation preliminaries
In this paper, we consider minimizing the following aggregated empirical IRL loss to estimate rewards:
L k ( θ ) : = ∑ i = 1 k l i ( θ ) , l i ( θ ) = V Π P i ⋆ ( θ ) − ⟨ Π P i θ , ϕ ^ i ⟩ , L_k(\theta):=\sum_{i=1}^{k}l_i(\theta),\qquad l_i(\theta)=V^{\star}_{\Pi_{P_i}}(\theta)-\langle\Pi_{P_i}\theta,\hat{\phi}_i\rangle, L k ( θ ) := i = 1 ∑ k l i ( θ ) , l i ( θ ) = V Π P i ⋆ ( θ ) − ⟨ Π P i θ , ϕ ^ i ⟩ ,
where ϕ ^ i = 1 m ∑ j = 1 m ϕ ( τ i , j ) \hat{\phi}_i=\frac{1}{m}\sum_{j=1}^{m}\phi(\tau_{i,j}) ϕ ^ i = m 1 ∑ j = 1 m ϕ ( τ i , j ) , and τ i , j \tau_{i,j} τ i , j is the j j j -th trajectory generated by the expert at round i i i . We denote a minimizer over Θ \Theta Θ by θ k ∈ arg min θ ∈ Θ L k ( θ ) \theta_k\in\arg\min_{\theta\in\Theta}L_k(\theta) θ k ∈ arg min θ ∈ Θ L k ( θ ) . We furthermore define the population loss
L k p o p ( θ ) : = ∑ i = 1 k ( V Π P i ⋆ ( θ ) − ⟨ Π P i θ , ϕ P i E ⟩ ) . (10) L_k^{\mathrm{pop}}(\theta):=\sum_{i=1}^{k}\left(V^{\star}_{\Pi_{P_i}}(\theta)-\langle\Pi_{P_i}\theta,\phi^{E}_{P_i}\rangle\right). \tag*{(10)} L k pop ( θ ) := i = 1 ∑ k ( V Π P i ⋆ ( θ ) − ⟨ Π P i θ , ϕ P i E ⟩ ) . (10)
Here ϕ P E : = E P E [ ϕ ( τ ) ] \phi^{\mathrm{E}}_{P} := \mathbb{E}^{\mathrm{E}}_{P}[\phi(\tau)] ϕ P E := E P E [ ϕ ( τ )] . Under adaptive dynamics selection, the population loss L k p o p L^{\mathrm{pop}}_{k} L k pop is a sum of per-round conditional population losses. It is minimized at the expert parameters θ E \theta^{\mathrm{E}} θ E .
Defining S k : = ∑ i = 1 k Π P i ( ϕ ^ i − ϕ P i E ) S_k := \sum_{i=1}^{k}\Pi_{P_i}(\hat{\phi}_i-\phi^{\mathrm{E}}_{P_i}) S k := ∑ i = 1 k Π P i ( ϕ ^ i − ϕ P i E ) , the aggregated empirical loss decomposes as
L k ( θ ) = L k p o p ( θ ) − ⟨ θ , S k ⟩ . L_k(\theta) = L^{\mathrm{pop}}_{k}(\theta) - \langle\theta, S_k\rangle. L k ( θ ) = L k pop ( θ ) − ⟨ θ , S k ⟩ .
Additionally, we introduce the following key quantities of our analysis:
H P ( θ ) : = ∇ θ 2 V P ⋆ ( θ ) , H k : = ∇ 2 L k ( θ E ) = ∇ 2 L k p o p ( θ E ) = ∑ i = 1 k H P i ( θ E ) , \mathcal{H}_{P}(\theta) := \nabla_{\theta}^{2}V^{\star}_{P}(\theta), \qquad\mathcal{H}_{k} := \nabla^{2}L_k(\theta^{\mathrm{E}}) = \nabla^{2}L^{\mathrm{pop}}_{k}(\theta^{\mathrm{E}}) = \sum_{i=1}^{k}\mathcal{H}_{P_i}(\theta^{\mathrm{E}}), H P ( θ ) := ∇ θ 2 V P ⋆ ( θ ) , H k := ∇ 2 L k ( θ E ) = ∇ 2 L k pop ( θ E ) = i = 1 ∑ k H P i ( θ E ) ,
λ k : = λ min U ( H k ) , R : = η H ( H + 1 ) B ϕ , R k : = R λ k , \lambda_k := \lambda_{\min}^{\mathrm{U}}(\mathcal{H}_k), \qquad R := \eta H(H+1)B_{\phi}, \qquad R_k := \frac{R}{\sqrt{\lambda_k}}, λ k := λ m i n U ( H k ) , R := η H ( H + 1 ) B ϕ , R k := λ k R ,
ρ k : = ∥ θ k − θ E ∥ H k , ω k : = ∥ S k ∥ H k † . \rho_k := \lVert\theta_k-\theta^{\mathrm{E}}\rVert_{\mathcal{H}_k}, \qquad\omega_k := \lVert S_k\rVert_{\mathcal{H}_k^{\dagger}}. ρ k := ∥ θ k − θ E ∥ H k , ω k := ∥ S k ∥ H k † .
Here, H k † \mathcal{H}_k^{\dagger} H k † denotes the Moore-Penrose inverse of H k \mathcal{H}_k H k . The quantity R k R_k R k is used only when λ k > 0 \lambda_k > 0 λ k > 0 . The stochastic self-concordance constant R R R is justified in Lemma B.6.
Recall that Π P \Pi_P Π P is the orthogonal projector onto im ( H P ( 0 ) ) \operatorname{im}(\mathcal{H}_P(0)) im ( H P ( 0 )) , which equals im ( H P ( θ ) ) \operatorname{im}(\mathcal{H}_P(\theta)) im ( H P ( θ )) for every θ \theta θ by Proposition B.5. The parameter-independent kernel implies
Π P H P ( Π P θ ) Π P = H P ( θ ) . \Pi_P\mathcal{H}_P(\Pi_P\theta)\Pi_P = \mathcal{H}_P(\theta). Π P H P ( Π P θ ) Π P = H P ( θ ) .
Thus the projected loss has the same Hessian as the original loss, justifying the expression for H k \mathcal{H}_k H k above.
Moreover, projection ensures
S k ∈ ∑ i = 1 k im ( H P i ( θ E ) ) = im ( H k ) . S_k \in\sum_{i=1}^{k}\operatorname{im}(\mathcal{H}_{P_i}(\theta^{\mathrm{E}})) = \operatorname{im}(\mathcal{H}_k). S k ∈ i = 1 ∑ k im ( H P i ( θ E )) = im ( H k ) .
This support property allows the noise to be controlled in the inverse Hessian norm without requiring full identifiability.
Supporting lemmas
We present several technical lemmas laying out various properties of the optimal regularized value function and population loss 10. To make the dependence of the optimal value function on reward parameters explicit, we sometimes use the notation V θ , P ⋆ = V P ⋆ ( θ ) V^{\star}_{\theta,P}=V^{\star}_{P}(\theta) V θ , P ⋆ = V P ⋆ ( θ ) .
Lemma B.1 (Value derivatives [16 ] ). Let Assumption 2.1 hold. Then, for any dynamics P P P and any θ , u , v ∈ R d \theta,u,v \in\mathbb{R}^{d} θ , u , v ∈ R d ,
D V P ⋆ ( θ ) [ v ] = ⟨ v , ϕ P ( θ ) ⟩ , DV^{\star}_{P}(\theta)[v] = \langle v,\phi_P(\theta)\rangle, D V P ⋆ ( θ ) [ v ] = ⟨ v , ϕ P ( θ )⟩ , D 2 V P ⋆ ( θ ) [ v , v ] = ∥ v ∥ H P ( θ ) 2 = η E P π P ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ] , D^{2}V^{\star}_{P}(\theta)[v,v] = \lVert v\rVert^{2}_{\mathcal{H}_{P}(\theta)} = \eta\,\mathbb{E}^{\pi_P(\theta)}_{P}\left[\langle v,A^{\theta}_{P}(\tau)\rangle^{2}\right], D 2 V P ⋆ ( θ ) [ v , v ] = ∥ v ∥ H P ( θ ) 2 = η E P π P ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ] , D 3 V P ⋆ ( θ ) [ v , v , u ] = η 2 E P π P ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ⟨ u , A P θ ( τ ) ⟩ ] , D^{3}V^{\star}_{P}(\theta)[v,v,u] = \eta^{2}\,\mathbb{E}^{\pi_P(\theta)}_{P}\left[\langle v,A^{\theta}_{P}(\tau)\rangle^{2}\langle u,A^{\theta}_{P}(\tau)\rangle\right], D 3 V P ⋆ ( θ ) [ v , v , u ] = η 2 E P π P ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ⟨ u , A P θ ( τ )⟩ ] , where ϕ P ( θ ) : = E P π P ( θ ) [ ϕ ( τ ) ] \phi_P(\theta) := \mathbb{E}^{\pi_P(\theta)}_{P}[\phi(\tau)] ϕ P ( θ ) := E P π P ( θ ) [ ϕ ( τ )] is the expected feature vector of the optimal policy, and A P θ ( τ ) : = ∑ h = 1 H A h , P θ ( s h , a h ) A^{\theta}_{P}(\tau) := \sum_{h=1}^{H}A^{\theta}_{h,P}(s_h,a_h) A P θ ( τ ) := ∑ h = 1 H A h , P θ ( s h , a h ) is the unregularized feature advantage, with
A h , P θ ( s , a ) : = E P π P ( θ ) [ ∑ t = h H ϕ ( s t , a t ) ∣ s h = s , a h = a ] − E P π P ( θ ) [ ∑ t = h H ϕ ( s t , a t ) ∣ s h = s ] . A^{\theta}_{h,P}(s,a) := \mathbb{E}^{\pi_P(\theta)}_{P}\left[\left.\sum_{t=h}^{H}\phi(s_t,a_t)\right|s_h=s,a_h=a\right] - \mathbb{E}^{\pi_P(\theta)}_{P}\left[\left.\sum_{t=h}^{H}\phi(s_t,a_t)\right|s_h=s\right]. A h , P θ ( s , a ) := E P π P ( θ ) [ t = h ∑ H ϕ ( s t , a t ) s h = s , a h = a ] − E P π P ( θ ) [ t = h ∑ H ϕ ( s t , a t ) s h = s ] . In particular, E P π P ( θ ) [ A P θ ( τ ) ] = 0 \mathbb{E}^{\pi_P(\theta)}_{P}[A^{\theta}_{P}(\tau)] = 0 E P π P ( θ ) [ A P θ ( τ )] = 0 , since E a ∼ π h , P ( θ ) ( ⋅ ∣ s ) [ A h , P θ ( s , a ) ] = 0 \mathbb{E}_{a\sim\pi_{h,P}(\theta)(\cdot\mid s)}[A^{\theta}_{h,P}(s,a)] = 0 E a ∼ π h , P ( θ ) ( ⋅ ∣ s ) [ A h , P θ ( s , a )] = 0 for every h h h and s s s .
Proof. The three identities follow from Schlaginhaufen and Kamgarpour [16 Lemma A.13] under the linear reward structure of Assumption 2.1 .
Corollary B.2 (Deterministic transitions). Let Assumption 2.1 hold, and let P P P have deterministic transitions and a deterministic initial state. Then, for any θ , u , v ∈ R d \theta, u, v \in\mathbb{R}^{d} θ , u , v ∈ R d ,
D V P ⋆ ( θ ) [ v ] = ⟨ v , ϕ P ( θ ) ⟩ , D V^\star_{P}(\theta)[v] = \langle v, \phi_{P}(\theta)\rangle, D V P ⋆ ( θ ) [ v ] = ⟨ v , ϕ P ( θ )⟩ , D 2 V P ⋆ ( θ ) [ v , v ] = ∥ v ∥ H P ( θ ) 2 = η E P π P ( θ ) [ ⟨ v , ϕ ( τ ) − ϕ P ( θ ) ⟩ 2 ] , D^{2}V^\star_{P}(\theta)[v,v] = \|v\|^{2}_{\mathcal{H}_{P}(\theta)} = \eta\mathbb{E}^{\pi_{P}(\theta)}_{P}\left[\langle v,\phi(\tau)-\phi_{P}(\theta)\rangle^{2}\right], D 2 V P ⋆ ( θ ) [ v , v ] = ∥ v ∥ H P ( θ ) 2 = η E P π P ( θ ) [ ⟨ v , ϕ ( τ ) − ϕ P ( θ ) ⟩ 2 ] , D 3 V P ⋆ ( θ ) [ v , v , u ] = η 2 E P π P ( θ ) [ ⟨ v , ϕ ( τ ) − ϕ P ( θ ) ⟩ 2 ⟨ u , ϕ ( τ ) − ϕ P ( θ ) ⟩ ] . D^{3}V^\star_{P}(\theta)[v,v,u] = \eta^{2}\mathbb{E}^{\pi_{P}(\theta)}_{P}\left[\langle v,\phi(\tau)-\phi_{P}(\theta)\rangle^{2}\langle u,\phi(\tau)-\phi_{P}(\theta)\rangle\right]. D 3 V P ⋆ ( θ ) [ v , v , u ] = η 2 E P π P ( θ ) [ ⟨ v , ϕ ( τ ) − ϕ P ( θ ) ⟩ 2 ⟨ u , ϕ ( τ ) − ϕ P ( θ )⟩ ] . Proof. By Lemma B.1, it suffices to show that A P θ ( τ ) = ϕ ( τ ) − ϕ P ( θ ) A^{\theta}_{P}(\tau)=\phi(\tau)-\phi_{P}(\theta) A P θ ( τ ) = ϕ ( τ ) − ϕ P ( θ ) . For h ∈ { 1 , … , H } h\in\{1,\ldots,H\} h ∈ { 1 , … , H } , let
Φ h ( s ) : = E P π P ( θ ) [ ∑ t = h H ϕ ( s t , a t ) ∣ s h = s ] , Φ H + 1 : = 0. \Phi_{h}(s):=\mathbb{E}^{\pi_{P}(\theta)}_{P}\left[\left.\sum_{t=h}^{H}\phi(s_{t},a_{t})\right|s_{h}=s\right],\qquad\Phi_{H+1}:=0. Φ h ( s ) := E P π P ( θ ) [ t = h ∑ H ϕ ( s t , a t ) s h = s ] , Φ H + 1 := 0. Since transitions are deterministic, ( s h , a h ) (s_{h},a_{h}) ( s h , a h ) determines s h + 1 s_{h+1} s h + 1 , so by the Markov property
E P π P ( θ ) [ ∑ t = h H ϕ ( s t , a t ) ∣ s h , a h ] = ϕ ( s h , a h ) + Φ h + 1 ( s h + 1 ) , \mathbb{E}^{\pi_{P}(\theta)}_{P}\left[\left.\sum_{t=h}^{H}\phi(s_{t},a_{t})\right|s_{h},a_{h}\right]=\phi(s_{h},a_{h})+\Phi_{h+1}(s_{h+1}), E P π P ( θ ) [ t = h ∑ H ϕ ( s t , a t ) s h , a h ] = ϕ ( s h , a h ) + Φ h + 1 ( s h + 1 ) , and hence A h , P θ ( s h , a h ) = ϕ ( s h , a h ) + Φ h + 1 ( s h + 1 ) − Φ h ( s h ) A^{\theta}_{h,P}(s_{h},a_{h})=\phi(s_{h},a_{h})+\Phi_{h+1}(s_{h+1})-\Phi_{h}(s_{h}) A h , P θ ( s h , a h ) = ϕ ( s h , a h ) + Φ h + 1 ( s h + 1 ) − Φ h ( s h ) . Summing over h h h , the terms in Φ \Phi Φ telescope:
A P θ ( τ ) = ∑ h = 1 H ϕ ( s h , a h ) − Φ 1 ( s 1 ) = ϕ ( τ ) − Φ 1 ( s 1 ) . A^{\theta}_{P}(\tau)=\sum_{h=1}^{H}\phi(s_{h},a_{h})-\Phi_{1}(s_{1})=\phi(\tau)-\Phi_{1}(s_{1}). A P θ ( τ ) = h = 1 ∑ H ϕ ( s h , a h ) − Φ 1 ( s 1 ) = ϕ ( τ ) − Φ 1 ( s 1 ) . Since the initial state is deterministic, Φ 1 ( s 1 ) = E P π P ( θ ) [ ϕ ( τ ) ] = ϕ P ( θ ) \Phi_{1}(s_{1})=\mathbb{E}^{\pi_{P}(\theta)}_{P}[\phi(\tau)]=\phi_{P}(\theta) Φ 1 ( s 1 ) = E P π P ( θ ) [ ϕ ( τ )] = ϕ P ( θ ) , which concludes the proof. □
Lemma B.3 (Covariance decomposition). Let Assumption 2.1 hold. For any dynamics P P P and θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d ,
Cov P π P ( θ ) ( ϕ ( τ ) ) = η − 1 H P ( θ ) + Σ P d y n ( θ ) , (11) \operatorname{Cov}^{\pi_{P}(\theta)}_{P}(\phi(\tau))=\eta^{-1}\mathcal{H}_{P}(\theta)+\Sigma^{\mathrm{dyn}}_{P}(\theta), \tag*{(11)} Cov P π P ( θ ) ( ϕ ( τ )) = η − 1 H P ( θ ) + Σ P dyn ( θ ) , (11) where,
Σ P d y n ( θ ) : = Cov P π P ( θ ) ( ϕ ( τ ) − A P θ ( τ ) ) ⪰ 0. \Sigma^{\mathrm{dyn}}_{P}(\theta):=\operatorname{Cov}^{\pi_{P}(\theta)}_{P}\left(\phi(\tau)-A^{\theta}_{P}(\tau)\right)\succeq0. Σ P dyn ( θ ) := Cov P π P ( θ ) ( ϕ ( τ ) − A P θ ( τ ) ) ⪰ 0. The matrix Σ P d y n ( θ ) \Sigma^{\mathrm{dyn}}_{P}(\theta) Σ P dyn ( θ ) captures the randomness of the transitions and of the initial state.
Proof. Since E P π P ( θ ) [ A P θ ( τ ) ] = 0 \mathbb{E}^{\pi_{P}(\theta)}_{P}[A^{\theta}_{P}(\tau)]=0 E P π P ( θ ) [ A P θ ( τ )] = 0 , Lemma B.1 gives Cov P π P ( θ ) ( A P θ ( τ ) ) = E P π P ( θ ) [ A P θ ( τ ) A P θ ( τ ) T ] = η − 1 H P ( θ ) \operatorname{Cov}^{\pi_{P}(\theta)}_{P}(A^{\theta}_{P}(\tau))=\mathbb{E}^{\pi_{P}(\theta)}_{P}[A^{\theta}_{P}(\tau)A^{\theta}_{P}(\tau)^{\mathsf{T}}]=\eta^{-1}\mathcal{H}_{P}(\theta) Cov P π P ( θ ) ( A P θ ( τ )) = E P π P ( θ ) [ A P θ ( τ ) A P θ ( τ ) T ] = η − 1 H P ( θ ) . Moreover, in the decomposition
ϕ ( τ ) − ϕ P ( θ ) = A P θ ( τ ) + ( ϕ ( τ ) − A P θ ( τ ) − ϕ P ( θ ) ) , \phi(\tau)-\phi_{P}(\theta)=A^{\theta}_{P}(\tau)+\left(\phi(\tau)-A^{\theta}_{P}(\tau)-\phi_{P}(\theta)\right), ϕ ( τ ) − ϕ P ( θ ) = A P θ ( τ ) + ( ϕ ( τ ) − A P θ ( τ ) − ϕ P ( θ ) ) , the two terms are uncorrelated by Schlaginhaufen and Kamgarpour [16 Lemma A.7] . Taking covariances on both sides proves (11). □
Proposition B.4 (Hessian upper bound). Let Assumption 2.1 hold. For any dynamics P P P and θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d ,
∥ H P ( θ ) ∥ 2 ≤ η H 2 B ϕ 2 . \|\mathcal{H}_{P}(\theta)\|_{2}\leq\eta H^{2}B_{\phi}^{2}. ∥ H P ( θ ) ∥ 2 ≤ η H 2 B ϕ 2 . Proof. Let v ∈ R d v\in\mathbb{R}^{d} v ∈ R d with ∥ v ∥ 2 = 1 \|v\|_{2}=1 ∥ v ∥ 2 = 1 . By Lemma B.1 and the covariance decomposition of Lemma B.3,
∥ v ∥ H P ( θ ) 2 = η E P π P ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ] ≤ η Var P π P ( θ ) ( ⟨ v , ϕ ( τ ) ⟩ ) ≤ η E P π P ( θ ) [ ⟨ v , ϕ ( τ ) ⟩ 2 ] ≤ η H 2 B ϕ 2 , \|v\|^{2}_{\mathcal{H}_{P}(\theta)}=\eta\mathbb{E}^{\pi_{P}(\theta)}_{P}\left[\langle v,A^{\theta}_{P}(\tau)\rangle^{2}\right]\leq\eta\operatorname{Var}^{\pi_{P}(\theta)}_{P}\left(\langle v,\phi(\tau)\rangle\right)\leq\eta\mathbb{E}^{\pi_{P}(\theta)}_{P}\left[\langle v,\phi(\tau)\rangle^{2}\right]\leq\eta H^{2}B_{\phi}^{2}, ∥ v ∥ H P ( θ ) 2 = η E P π P ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ] ≤ η Var P π P ( θ ) ( ⟨ v , ϕ ( τ )⟩ ) ≤ η E P π P ( θ ) [ ⟨ v , ϕ ( τ ) ⟩ 2 ] ≤ η H 2 B ϕ 2 , where the last step uses ∥ ϕ ( τ ) ∥ 2 ≤ ∑ h = 1 H ∥ ϕ ( s h , a h ) ∥ 2 ≤ H B ϕ \|\phi(\tau)\|_{2}\leq\sum_{h=1}^{H}\|\phi(s_{h},a_{h})\|_{2}\leq HB_{\phi} ∥ ϕ ( τ ) ∥ 2 ≤ ∑ h = 1 H ∥ ϕ ( s h , a h ) ∥ 2 ≤ H B ϕ . Since H P ( θ ) \mathcal{H}_{P}(\theta) H P ( θ ) is positive semidefinite, ∥ H P ( θ ) ∥ 2 = max ∥ v ∥ 2 = 1 ∥ v ∥ H P ( θ ) 2 \|\mathcal{H}_{P}(\theta)\|_{2}=\max_{\|v\|_{2}=1}\|v\|^{2}_{\mathcal{H}_{P}(\theta)} ∥ H P ( θ ) ∥ 2 = max ∥ v ∥ 2 = 1 ∥ v ∥ H P ( θ ) 2 , which proves the claim. □
Proposition B.5 (Hessian image [16 ] ). *Let Assumption 2.1 hold and let P P P be any dynamics. Then ker ( H P ( θ ) ) \operatorname{ker}(\mathcal{H}_{P}(\theta)) ker ( H P ( θ )) consists of the directions v ∈ R d v\in\mathbb{R}^{d} v ∈ R d whose reward perturbation r v ( s , a ) : = ⟨ v , ϕ ( s , a ) ⟩ r_{v}(s,a):=\langle v,\phi(s,a)\rangle r v ( s , a ) := ⟨ v , ϕ ( s , a )⟩ preserves the optimal trajectory law under P P P , and is independent of θ \theta θ . In particular, U P = im ( H P ( θ ) ) = im ( H P ( 0 ) ) U_{P}=\operatorname{im}(\mathcal{H}_{P}(\theta))=\operatorname{im}(\mathcal{H}_{P}(0)) U P = im ( H P ( θ )) = im ( H P ( 0 )) for all θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d .
Proof. By Lemma B.1 , for any v ∈ R d v \in\mathbb{R}^{d} v ∈ R d ,
v ∈ ker ( H P ( θ ) ) ⟺ E P π P π ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ] = 0 v \in\ker(\mathcal{H}_{P}(\theta)) \Longleftrightarrow\mathbb{E}_{P}^{\pi_{P}^{\pi(\theta)}}\left[\langle v,A_{P}^{\theta}(\tau)\rangle^{2}\right]=0 v ∈ ker ( H P ( θ )) ⟺ E P π P π ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ] = 0
⟺ ⟨ v , A P θ ( τ ) ⟩ = 0 P P π P π ( θ ) -almost surely . \Longleftrightarrow\langle v,A_{P}^{\theta}(\tau)\rangle=0\quad\mathbb{P}_{P}^{\pi_{P}^{\pi(\theta)}}\text{-almost surely}. ⟺ ⟨ v , A P θ ( τ )⟩ = 0 P P π P π ( θ ) -almost surely .
By linearity, ⟨ v , A P θ ( τ ) ⟩ \langle v,A_{P}^{\theta}(\tau)\rangle ⟨ v , A P θ ( τ )⟩ is the cumulative unregularized advantage of r v r_v r v under π P ( θ ) \pi_{P}(\theta) π P ( θ ) . By Schlaginhaufen and Kamgarpour [16 Corollary A.16] , it vanishes almost surely if and only if r v r_v r v preserves the optimal trajectory law under P P P , a condition that does not depend on θ \theta θ since soft-optimal policies have full support. Hence ker ( H P ( θ ) ) = ker ( H P ( 0 ) ) \ker(\mathcal{H}_{P}(\theta))=\ker(\mathcal{H}_{P}(0)) ker ( H P ( θ )) = ker ( H P ( 0 )) for all θ \theta θ . Since H P ( θ ) \mathcal{H}_{P}(\theta) H P ( θ ) is symmetric, im ( H P ( θ ) ) = ker ( H P ( θ ) ) ⊥ \operatorname{im}(\mathcal{H}_{P}(\theta))=\ker(\mathcal{H}_{P}(\theta))^{\perp} im ( H P ( θ )) = ker ( H P ( θ ) ) ⊥ , which concludes the proof. □ \square □
Lemma B.6 (Pseudo self-concordance). Let Assumption 2.1 hold, and define R : = η H ( H + 1 ) B ϕ R:=\eta H(H+1)B_{\phi} R := η H ( H + 1 ) B ϕ . For any dynamics P P P and θ , u , v ∈ R d \theta,u,v\in\mathbb{R}^{d} θ , u , v ∈ R d , the optimal regularized value function satisfies
∣ D 3 V P ⋆ ( θ ) [ v , v , u ] ∣ ≤ R ∥ u ∥ Π P D 2 V P ⋆ ( θ ) [ v , v ] . \left|D^{3}V_{P}^{\star}(\theta)[v,v,u]\right|\leq R\|u\|_{\Pi_{P}}D^{2}V_{P}^{\star}(\theta)[v,v]. D 3 V P ⋆ ( θ ) [ v , v , u ] ≤ R ∥ u ∥ Π P D 2 V P ⋆ ( θ ) [ v , v ] .
Proof. By Lemma B.1 and Proposition B.5 , ⟨ w , A P θ ( τ ) ⟩ = 0 \langle w,A_{P}^{\theta}(\tau)\rangle=0 ⟨ w , A P θ ( τ )⟩ = 0 almost surely for each w w w in a basis of ker ( H P ( θ ) ) = U P ⊥ \ker(\mathcal{H}_{P}(\theta))=U_{P}^{\perp} ker ( H P ( θ )) = U P ⊥ . Hence A P θ ( τ ) ∈ U P A_{P}^{\theta}(\tau)\in U_{P} A P θ ( τ ) ∈ U P almost surely, and
⟨ u , A P θ ( τ ) ⟩ = ⟨ Π P u , A P θ ( τ ) ⟩ almost surely . \langle u,A_{P}^{\theta}(\tau)\rangle=\langle\Pi_{P}u,A_{P}^{\theta}(\tau)\rangle\quad\text{almost surely}. ⟨ u , A P θ ( τ )⟩ = ⟨ Π P u , A P θ ( τ )⟩ almost surely .
At each step h h h , the feature advantage A h , P θ A_{h,P}^{\theta} A h , P θ is the difference of two conditional expectations of the remaining feature sum, each of norm at most ( H − h + 1 ) B ϕ (H-h+1)B_{\phi} ( H − h + 1 ) B ϕ . Therefore,
∥ A P θ ( τ ) ∥ 2 ≤ 2 B ϕ ∑ h = 1 H ( H − h + 1 ) = H ( H + 1 ) B ϕ almost surely . \left\|A_{P}^{\theta}(\tau)\right\|_{2}\leq2B_{\phi}\sum_{h=1}^{H}(H-h+1)=H(H+1)B_{\phi}\quad\text{almost surely}. A P θ ( τ ) 2 ≤ 2 B ϕ h = 1 ∑ H ( H − h + 1 ) = H ( H + 1 ) B ϕ almost surely .
Using Lemma B.1 and the Cauchy–Schwarz inequality,
∣ D 3 V P ⋆ ( θ ) [ v , v , u ] ∣ = η 2 ∣ E P π P ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ⟨ Π P u , A P θ ( τ ) ⟩ ] ∣ ≤ η 2 H ( H + 1 ) B ϕ ∥ Π P u ∥ 2 E P π P ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ] = R ∥ u ∥ Π P D 2 V P ⋆ ( θ ) [ v , v ] . \begin{aligned}
\left|D^{3}V_{P}^{\star}(\theta)[v,v,u]\right|=\eta^{2}\left|\mathbb{E}_{P}^{\pi_{P}(\theta)}\left[\langle v,A_{P}^{\theta}(\tau)\rangle^{2}\langle\Pi_{P}u,A_{P}^{\theta}(\tau)\rangle\right]\right| \\
&\leq\eta^{2}H(H+1)B_{\phi}\|\Pi_{P}u\|_{2}\mathbb{E}_{P}^{\pi_{P}(\theta)}\left[\langle v,A_{P}^{\theta}(\tau)\rangle^{2}\right] \\
&=R\|u\|_{\Pi_{P}}D^{2}V_{P}^{\star}(\theta)[v,v].
\end{aligned} D 3 V P ⋆ ( θ ) [ v , v , u ] = η 2 E P π P ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ⟨ Π P u , A P θ ( τ )⟩ ] ≤ η 2 H ( H + 1 ) B ϕ ∥ Π P u ∥ 2 E P π P ( θ ) [ ⟨ v , A P θ ( τ ) ⟩ 2 ] = R ∥ u ∥ Π P D 2 V P ⋆ ( θ ) [ v , v ] .
□ \square □
Lemma B.7 (Curvature stability). Let Assumption 2.1 hold and let P P P be any dynamics. Fix θ 1 , θ 2 ∈ R d \theta_{1},\theta_{2}\in\mathbb{R}^{d} θ 1 , θ 2 ∈ R d and set a : = R ∥ θ 2 − θ 1 ∥ Π P a:=R\|\theta_{2}-\theta_{1}\|_{\Pi_{P}} a := R ∥ θ 2 − θ 1 ∥ Π P . Define
ψ ( x ) : = ∫ 0 1 t e x t d t = { x e x − e x + 1 x 2 , x ≠ 0 , 1 / 2 , x = 0 , G P ( θ 1 , θ 2 ) : = ∫ 0 1 ( 1 − t ) H P ( t θ 2 + ( 1 − t ) θ 1 ) d t . \psi(x):=\int_{0}^{1}te^{xt}\,\mathrm{d}t=
\begin{cases}
\dfrac{xe^{x}-e^{x}+1}{x^{2}}, & x\ne0,\\
1/2, & x=0,
\end{cases}
\qquad
G_{P}(\theta_{1},\theta_{2}):=\int_{0}^{1}(1-t)\mathcal{H}_{P}(t\theta_{2}+(1-t)\theta_{1})\,\mathrm{d}t. ψ ( x ) := ∫ 0 1 t e x t d t = ⎩ ⎨ ⎧ x 2 x e x − e x + 1 , 1/2 , x = 0 , x = 0 , G P ( θ 1 , θ 2 ) := ∫ 0 1 ( 1 − t ) H P ( t θ 2 + ( 1 − t ) θ 1 ) d t . Then:
e − a H P ( θ 2 ) ⪯ H P ( θ 1 ) ⪯ e a H P ( θ 2 ) e^{-a}\mathcal{H}_{P}(\theta_{2})\preceq\mathcal{H}_{P}(\theta_{1})\preceq e^{a}\mathcal{H}_{P}(\theta_{2}) e − a H P ( θ 2 ) ⪯ H P ( θ 1 ) ⪯ e a H P ( θ 2 ) ;
ψ ( − a ) H P ( θ 2 ) ⪯ G P ( θ 1 , θ 2 ) ⪯ ψ ( a ) H P ( θ 2 ) \psi(-a)\mathcal{H}_{P}(\theta_{2})\preceq G_{P}(\theta_{1},\theta_{2})\preceq\psi(a)\mathcal{H}_{P}(\theta_{2}) ψ ( − a ) H P ( θ 2 ) ⪯ G P ( θ 1 , θ 2 ) ⪯ ψ ( a ) H P ( θ 2 ) ;
e − a ψ ( a ) H P ( θ 1 ) ⪯ G P ( θ 1 , θ 2 ) ⪯ e a ψ ( − a ) H P ( θ 1 ) e^{-a}\psi(a)\mathcal{H}_{P}(\theta_{1})\preceq G_{P}(\theta_{1},\theta_{2})\preceq e^{a}\psi(-a)\mathcal{H}_{P}(\theta_{1}) e − a ψ ( a ) H P ( θ 1 ) ⪯ G P ( θ 1 , θ 2 ) ⪯ e a ψ ( − a ) H P ( θ 1 ) .
Proof. Let θ ˉ ( t ) : = t θ 2 + ( 1 − t ) θ 1 \bar{\theta}(t):=t\theta_{2}+(1-t)\theta_{1} θ ˉ ( t ) := t θ 2 + ( 1 − t ) θ 1 for t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] , and recall that D 2 V P ⋆ ( θ ) [ v , v ] = v T H P ( θ ) v D^{2}V_{P}^{\star}(\theta)[v,v]=v^{\mathsf{T}}\mathcal{H}_{P}(\theta)v D 2 V P ⋆ ( θ ) [ v , v ] = v T H P ( θ ) v .
Reduction to v ∈ U P ∖ { 0 } v\in U_{P}\setminus\{0\} v ∈ U P ∖ { 0 } . Let v ∈ R d v\in\mathbb{R}^{d} v ∈ R d . By Proposition B.5 , ( I − Π P ) v ∈ ker ( H P ( θ ) ) (I-\Pi_{P})v\in\ker(\mathcal{H}_{P}(\theta)) ( I − Π P ) v ∈ ker ( H P ( θ )) for every θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d , so by symmetry of H P ( θ ) \mathcal{H}_{P}(\theta) H P ( θ ) ,
v T H P ( θ ) v = ( Π P v ) T H P ( θ ) ( Π P v ) for all θ ∈ R d , v^{\mathsf{T}}\mathcal{H}_{P}(\theta)v=(\Pi_{P}v)^{\mathsf{T}}\mathcal{H}_{P}(\theta)(\Pi_{P}v)\quad\text{for all }\theta\in\mathbb{R}^{d}, v T H P ( θ ) v = ( Π P v ) T H P ( θ ) ( Π P v ) for all θ ∈ R d ,
and integrating this identity gives the same for G P ( θ 1 , θ 2 ) G_{P}(\theta_{1},\theta_{2}) G P ( θ 1 , θ 2 ) . If Π P v = 0 \Pi_{P}v=0 Π P v = 0 , every quadratic form in the claim vanishes. It therefore suffices to prove the scalar inequalities for v ∈ U P ∖ { 0 } v\in U_{P}\setminus\{0\} v ∈ U P ∖ { 0 } , which we fix from now on. Pointwise bounds. By Proposition B.5, q ( t ) : = D 2 V P ∗ ( θ ˉ ( t ) ) [ v , v ] > 0 q(t) := D^2V_P^*(\bar{\theta}(t))[v,v] > 0 q ( t ) := D 2 V P ∗ ( θ ˉ ( t )) [ v , v ] > 0 for all t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] . Since q ′ ( t ) = D 3 V P ∗ ( θ ˉ ( t ) ) [ v , v , θ 2 − θ 1 ] q'(t) = D^3V_P^*(\bar{\theta}(t))[v,v,\theta_2-\theta_1] q ′ ( t ) = D 3 V P ∗ ( θ ˉ ( t )) [ v , v , θ 2 − θ 1 ] , Lemma B.6 gives
∣ d d t log q ( t ) ∣ ≤ a for all t ∈ [ 0 , 1 ] . \left|\frac{\mathrm{d}}{\mathrm{d}t}\log q(t)\right| \le a \qquad\text{for all } t \in[0,1]. d t d log q ( t ) ≤ a for all t ∈ [ 0 , 1 ] .
Hence ∣ log q ( t ) − log q ( s ) ∣ ≤ ∣ t − s ∣ a |\log q(t)-\log q(s)| \le|t-s|\,a ∣ log q ( t ) − log q ( s ) ∣ ≤ ∣ t − s ∣ a for all s , t ∈ [ 0 , 1 ] s,t \in[0,1] s , t ∈ [ 0 , 1 ] . Taking s = 1 s=1 s = 1 and s = 0 s=0 s = 0 and exponentiating, we get, for all t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] ,
e − ( 1 − t ) a q ( 1 ) ≤ q ( t ) ≤ e ( 1 − t ) a q ( 1 ) , (12) e^{-(1-t)a}q(1) \le q(t) \le e^{(1-t)a}q(1), \tag*{(12)} e − ( 1 − t ) a q ( 1 ) ≤ q ( t ) ≤ e ( 1 − t ) a q ( 1 ) , (12)
e − t a q ( 0 ) ≤ q ( t ) ≤ e t a q ( 0 ) . (13) e^{-ta}q(0) \le q(t) \le e^{ta}q(0). \tag*{(13)} e − t a q ( 0 ) ≤ q ( t ) ≤ e t a q ( 0 ) . (13)
Item 1. This is (12) at t = 0 t=0 t = 0 .
Items 2 and 3. Note that v ⊤ G P ( θ 1 , θ 2 ) v = ∫ 0 1 ( 1 − t ) q ( t ) d t v^\top G_P(\theta_1,\theta_2)v = \int_0^1(1-t)q(t)\,\mathrm{d}t v ⊤ G P ( θ 1 , θ 2 ) v = ∫ 0 1 ( 1 − t ) q ( t ) d t . Multiplying (12) and (13) by ( 1 − t ) ≥ 0 (1-t) \ge0 ( 1 − t ) ≥ 0 and integrating over [ 0 , 1 ] [0,1] [ 0 , 1 ] preserves the inequalities. With the substitution s = 1 − t s=1-t s = 1 − t and the definition of ψ \psi ψ , the resulting weights are
∫ 0 1 ( 1 − t ) e ± ( 1 − t ) a d t = ∫ 0 1 s e ± a s d s = ψ ( ± a ) , \int_0^1(1-t)e^{\pm(1-t)a}\,\mathrm{d}t = \int_0^1s e^{\pm as}\,\mathrm{d}s = \psi(\pm a), ∫ 0 1 ( 1 − t ) e ± ( 1 − t ) a d t = ∫ 0 1 s e ± a s d s = ψ ( ± a ) ,
∫ 0 1 ( 1 − t ) e ± t a d t = e ± a ∫ 0 1 s e ∓ a s d s = e ± a ψ ( ∓ a ) . \int_0^1(1-t)e^{\pm ta}\,\mathrm{d}t = e^{\pm a}\int_0^1s e^{\mp as}\,\mathrm{d}s = e^{\pm a}\psi(\mp a). ∫ 0 1 ( 1 − t ) e ± t a d t = e ± a ∫ 0 1 s e ∓ a s d s = e ± a ψ ( ∓ a ) .
This yields items 2 and 3, respectively.
Lemma B.8. Let Assumption 2.1 hold, and define χ ( x ) : = e x − 1 x \chi(x) := \frac{e^x-1}{x} χ ( x ) := x e x − 1 for x ≠ 0 x \ne0 x = 0 , with χ ( 0 ) : = 1 \chi(0) := 1 χ ( 0 ) := 1 . Then, for any k ≥ 1 k \ge1 k ≥ 1 with λ k > 0 \lambda_k > 0 λ k > 0 ,
χ ( − R k ρ k ) ρ k 2 ≤ ⟨ θ k − θ E , ∇ L k p o p ( θ k ) − ∇ L k p o p ( θ E ) ⟩ ≤ χ ( R k ρ k ) ρ k 2 . \chi(-R_k\rho_k)\rho_k^2 \le\langle\theta_k-\theta^{\mathrm{E}},\nabla L_k^{\mathrm{pop}}(\theta_k)-\nabla L_k^{\mathrm{pop}}(\theta^{\mathrm{E}})\rangle\le\chi(R_k\rho_k)\rho_k^2. χ ( − R k ρ k ) ρ k 2 ≤ ⟨ θ k − θ E , ∇ L k pop ( θ k ) − ∇ L k pop ( θ E )⟩ ≤ χ ( R k ρ k ) ρ k 2 .
Proof. Let u : = θ k − θ E u := \theta_k-\theta^{\mathrm{E}} u := θ k − θ E and θ ˉ ( t ) : = θ E + t u \bar{\theta}(t) := \theta^{\mathrm{E}}+tu θ ˉ ( t ) := θ E + t u for t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] . Since H k \mathcal{H}_k H k vanishes on U ⊥ U^\perp U ⊥ and ∥ w ∥ H k 2 ≥ λ k ∥ w ∥ 2 2 \|w\|_{\mathcal{H}_k}^2 \ge\lambda_k\|w\|_2^2 ∥ w ∥ H k 2 ≥ λ k ∥ w ∥ 2 2 for w ∈ U w \in U w ∈ U , we have H k ⪰ λ k Π U \mathcal{H}_k \succeq\lambda_k\Pi_U H k ⪰ λ k Π U . As U P i ⊆ U U_{P_i} \subseteq U U P i ⊆ U , this gives for every i ≤ k i \le k i ≤ k
a i : = R ∥ u ∥ Π P i 2 ≤ R ∥ Π U u ∥ 2 2 ≤ R ∥ u ∥ H k 2 λ k = R k ρ k . a_i := R\|u\|_{\Pi_{P_i}}^2 \le R\|\Pi_Uu\|_2^2 \le R\frac{\|u\|_{\mathcal{H}_k}^2}{\sqrt{\lambda_k}} = R_k\rho_k. a i := R ∥ u ∥ Π P i 2 ≤ R ∥ Π U u ∥ 2 2 ≤ R λ k ∥ u ∥ H k 2 = R k ρ k .
Applying (13) from the proof of Lemma B.7 to each P i P_i P i with θ 1 = θ E \theta_1=\theta^{\mathrm{E}} θ 1 = θ E and θ 2 = θ k \theta_2=\theta_k θ 2 = θ k , and summing over i i i , we get for any v ∈ R d v \in\mathbb{R}^d v ∈ R d and t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ]
e − t R k ρ k ∥ v ∥ H k 2 ≤ D 2 L k p o p ( θ ˉ ( t ) ) [ v , v ] ≤ e t R k ρ k ∥ v ∥ H k 2 , (14) e^{-tR_k\rho_k}\|v\|_{\mathcal{H}_k}^2 \le D^2L_k^{\mathrm{pop}}(\bar{\theta}(t))[v,v] \le e^{tR_k\rho_k}\|v\|_{\mathcal{H}_k}^2, \tag*{(14)} e − t R k ρ k ∥ v ∥ H k 2 ≤ D 2 L k pop ( θ ˉ ( t )) [ v , v ] ≤ e t R k ρ k ∥ v ∥ H k 2 , (14)
where we used a i ≤ R k ρ k a_i \le R_k\rho_k a i ≤ R k ρ k and D 2 L k p o p ( θ ) = ∑ i = 1 k H P i ( θ ) D^2L_k^{\mathrm{pop}}(\theta)=\sum_{i=1}^k\mathcal{H}_{P_i}(\theta) D 2 L k pop ( θ ) = ∑ i = 1 k H P i ( θ ) .
By the fundamental theorem of calculus,
⟨ u , ∇ L k p o p ( θ k ) − ∇ L k p o p ( θ E ) ⟩ = ∫ 0 1 D 2 L k p o p ( θ ˉ ( t ) ) [ u , u ] d t . \langle u,\nabla L_k^{\mathrm{pop}}(\theta_k)-\nabla L_k^{\mathrm{pop}}(\theta^{\mathrm{E}})\rangle= \int_0^1D^2L_k^{\mathrm{pop}}(\bar{\theta}(t))[u,u]\,\mathrm{d}t. ⟨ u , ∇ L k pop ( θ k ) − ∇ L k pop ( θ E )⟩ = ∫ 0 1 D 2 L k pop ( θ ˉ ( t )) [ u , u ] d t .
Sandwiching the integrand using (14) with v = u v=u v = u , and using ∫ 0 1 e ± x t d t = χ ( ± x ) \int_0^1e^{\pm xt}\,\mathrm{d}t=\chi(\pm x) ∫ 0 1 e ± x t d t = χ ( ± x ) and ∥ u ∥ H k = ρ k \|u\|_{\mathcal{H}_k}=\rho_k ∥ u ∥ H k = ρ k , gives the result.
Ellipsoidal confidence sets
Remark B.9. For any u ∈ R d u \in\mathbb{R}^d u ∈ R d ,
⟨ S k , u ⟩ = ⟨ ( H k † ) 1 / 2 S k , H k 1 / 2 u ⟩ . (15) \langle S_k,u\rangle= \langle(\mathcal{H}_k^{\dagger})^{1/2}S_k,\mathcal{H}_k^{1/2}u\rangle. \tag*{(15)} ⟨ S k , u ⟩ = ⟨( H k † ) 1/2 S k , H k 1/2 u ⟩ . (15)
Indeed, let v ∈ ker ( H k ) v \in\ker(\mathcal{H}_k) v ∈ ker ( H k ) . Since H k = ∑ i = 1 k H P i ( θ E ) \mathcal{H}_k=\sum_{i=1}^k\mathcal{H}_{P_i}(\theta^{\mathrm{E}}) H k = ∑ i = 1 k H P i ( θ E ) is a sum of positive semidefinite matrices, v ⊤ H P i ( θ E ) v = 0 v^\top\mathcal{H}_{P_i}(\theta^{\mathrm{E}})v=0 v ⊤ H P i ( θ E ) v = 0 and hence v ∈ ker ( H P i ( θ E ) ) v \in\ker(\mathcal{H}_{P_i}(\theta^{\mathrm{E}})) v ∈ ker ( H P i ( θ E )) for every i i i . By Proposition B.5,
ker ( H P i ( θ E ) ) = U P i ⊥ = ker ( Π P i ) , \ker\left(\mathcal{H}_{P_i}(\theta^{\mathrm{E}})\right)=U_{P_i}^{\perp}=\ker(\Pi_{P_i}), ker ( H P i ( θ E ) ) = U P i ⊥ = ker ( Π P i ) ,
so Π P i v = 0 \Pi_{P_i}v=0 Π P i v = 0 for every i i i . By definition of S k S_k S k and symmetry of the orthogonal projectors,
⟨ v , S k ⟩ = ∑ i = 1 k ⟨ v , Π P i ( ϕ ^ i − ϕ P i E ) ⟩ = ∑ i = 1 k ⟨ Π P i v , ϕ ^ i − ϕ P i E ⟩ = 0. \langle v,S_k\rangle=\sum_{i=1}^{k}\langle v,\Pi_{P_i}(\hat{\phi}_i-\phi_{P_i}^{\mathrm{E}})\rangle=\sum_{i=1}^{k}\langle\Pi_{P_i}v,\hat{\phi}_i-\phi_{P_i}^{\mathrm{E}}\rangle=0. ⟨ v , S k ⟩ = i = 1 ∑ k ⟨ v , Π P i ( ϕ ^ i − ϕ P i E )⟩ = i = 1 ∑ k ⟨ Π P i v , ϕ ^ i − ϕ P i E ⟩ = 0.
Therefore S k ∈ ker ( H k ) ⊥ = im ( H k ) S_k\in\operatorname{ker}(\mathcal{H}_k)^\perp=\operatorname{im}(\mathcal{H}_k) S k ∈ ker ( H k ) ⊥ = im ( H k ) . Since H k † 1 / 2 H k 1 / 2 \mathcal{H}_k^{\dagger1/2}\mathcal{H}_k^{1/2} H k † 1/2 H k 1/2 is the orthogonal projector onto im ( H k ) \operatorname{im}(\mathcal{H}_k) im ( H k ) , the right-hand side of (15) equals ⟨ S k , u ⟩ \langle S_k,u\rangle ⟨ S k , u ⟩ .
To proceed, we use the first-order optimality of θ k \theta_k θ k for L k L_k L k over Θ \Theta Θ and of θ E \theta^{\mathrm{E}} θ E for L k p o p L_k^{\mathrm{pop}} L k pop to derive a deterministic localization bound.
Lemma B.10 (Deterministic localization bound). Let Assumption 2.1 hold and θ E ∈ Θ \theta^{\mathrm{E}}\in\Theta θ E ∈ Θ . For all k ≥ 1 k\ge1 k ≥ 1 with λ k > 0 \lambda_k>0 λ k > 0 ,
χ ( − R k ρ k ) ρ k ≤ ω k . (16) \chi(-R_k\rho_k)\rho_k\le\omega_k. \tag*{(16)} χ ( − R k ρ k ) ρ k ≤ ω k . (16) Additionally, on the localization event
E k : = { ω k ≤ 1 − e − 1 R k } , (17) \mathcal{E}_k:=\left\{\omega_k\le\frac{1-e^{-1}}{R_k}\right\}, \tag*{(17)} E k := { ω k ≤ R k 1 − e − 1 } , (17) we have R k ρ k ≤ 1 R_k\rho_k\le1 R k ρ k ≤ 1 and
ρ k ≤ 1 1 − e − 1 ω k . (18) \rho_k\le\frac{1}{1-e^{-1}}\omega_k. \tag*{(18)} ρ k ≤ 1 − e − 1 1 ω k . (18)
Proof. Since θ k \theta_k θ k minimizes L k L_k L k over Θ \Theta Θ , the first-order optimality condition gives
⟨ ∇ L k ( θ k ) , θ − θ k ⟩ ≥ 0 ∀ θ ∈ Θ . \langle\nabla L_k(\theta_k),\theta-\theta_k\rangle\ge0\quad\forall\ \theta\in\Theta. ⟨ ∇ L k ( θ k ) , θ − θ k ⟩ ≥ 0 ∀ θ ∈ Θ.
By well-specification and feature matching for the projected population loss, ∇ L k p o p ( θ E ) = 0 \nabla L_k^{\mathrm{pop}}(\theta^{\mathrm{E}})=0 ∇ L k pop ( θ E ) = 0 . Using the decomposition ∇ L k ( θ k ) = ∇ L k p o p ( θ k ) − S k \nabla L_k(\theta_k)=\nabla L_k^{\mathrm{pop}}(\theta_k)-S_k ∇ L k ( θ k ) = ∇ L k pop ( θ k ) − S k and taking θ = θ E \theta=\theta^{\mathrm{E}} θ = θ E , we have
⟨ ∇ L k p o p ( θ k ) − S k , θ E − θ k ⟩ ≥ 0. \langle\nabla L_k^{\mathrm{pop}}(\theta_k)-S_k,\theta^{\mathrm{E}}-\theta_k\rangle\ge0. ⟨ ∇ L k pop ( θ k ) − S k , θ E − θ k ⟩ ≥ 0.
Rearranging and subtracting ⟨ ∇ L k p o p ( θ E ) , θ k − θ E ⟩ = 0 \langle\nabla L_k^{\mathrm{pop}}(\theta^{\mathrm{E}}),\theta_k-\theta^{\mathrm{E}}\rangle=0 ⟨ ∇ L k pop ( θ E ) , θ k − θ E ⟩ = 0 ,
⟨ ∇ L k p o p ( θ k ) − ∇ L k p o p ( θ E ) , θ k − θ E ⟩ ≤ ⟨ S k , θ k − θ E ⟩ . (19) \langle\nabla L_k^{\mathrm{pop}}(\theta_k)-\nabla L_k^{\mathrm{pop}}(\theta^{\mathrm{E}}),\theta_k-\theta^{\mathrm{E}}\rangle\le\langle S_k,\theta_k-\theta^{\mathrm{E}}\rangle. \tag*{(19)} ⟨ ∇ L k pop ( θ k ) − ∇ L k pop ( θ E ) , θ k − θ E ⟩ ≤ ⟨ S k , θ k − θ E ⟩ . (19)
By Remark B.9 and the Cauchy–Schwarz inequality, the right-hand side of (19) satisfies
⟨ S k , θ k − θ E ⟩ = ⟨ ( H k † ) 1 / 2 S k , H k 1 / 2 ( θ k − θ E ) ⟩ ≤ ω k ρ k . (20) \langle S_k,\theta_k-\theta^{\mathrm{E}}\rangle=\langle(\mathcal{H}_k^{\dagger})^{1/2}S_k,\mathcal{H}_k^{1/2}(\theta_k-\theta^{\mathrm{E}})\rangle\le\omega_k\rho_k. \tag*{(20)} ⟨ S k , θ k − θ E ⟩ = ⟨( H k † ) 1/2 S k , H k 1/2 ( θ k − θ E )⟩ ≤ ω k ρ k . (20)
Combining the lower bound of Lemma B.8 with (19) and (20),
χ ( − R k ρ k ) ρ k 2 ≤ ω k ρ k . \chi(-R_k\rho_k)\rho_k^2\le\omega_k\rho_k. χ ( − R k ρ k ) ρ k 2 ≤ ω k ρ k .
Dividing by ρ k \rho_k ρ k (if ρ k > 0 \rho_k>0 ρ k > 0 ; the bound is trivial otherwise) yields (16).
Finally, on the event E k \mathcal{E}_k E k , inequality (16) gives
1 − e − R k ρ k R k = χ ( − R k ρ k ) ρ k ≤ ω k ≤ 1 − e − 1 R k , \frac{1-e^{-R_k\rho_k}}{R_k}=\chi(-R_k\rho_k)\rho_k\le\omega_k\le\frac{1-e^{-1}}{R_k}, R k 1 − e − R k ρ k = χ ( − R k ρ k ) ρ k ≤ ω k ≤ R k 1 − e − 1 ,
hence 1 − e − R k ρ k ≤ 1 − e − 1 1-e^{-R_k\rho_k}\le1-e^{-1} 1 − e − R k ρ k ≤ 1 − e − 1 , i.e. R k ρ k ≤ 1 R_k\rho_k\le1 R k ρ k ≤ 1 . Since χ \chi χ is increasing, χ ( − R k ρ k ) ≥ χ ( − 1 ) = 1 − e − 1 \chi(-R_k\rho_k)\ge\chi(-1)=1-e^{-1} χ ( − R k ρ k ) ≥ χ ( − 1 ) = 1 − e − 1 , and therefore
( 1 − e − 1 ) ρ k ≤ χ ( − R k ρ k ) ρ k ≤ ω k , (1-e^{-1})\rho_k\le\chi(-R_k\rho_k)\rho_k\le\omega_k, ( 1 − e − 1 ) ρ k ≤ χ ( − R k ρ k ) ρ k ≤ ω k ,
which gives (18). □
In the next step, we are bounding the excess risk on the event E \mathcal{E} E .
Lemma B.11 (Excess risk bound on E k \mathcal{E}_k E k ). Let Assumption 2.1 hold and θ E ∈ Θ \theta^{\mathrm{E}}\in\Theta θ E ∈ Θ . Fix k ≥ 1 k\ge1 k ≥ 1 with λ k > 0 \lambda_k>0 λ k > 0 . On the event E k \mathcal{E}_k E k defined in (17),
L k p o p ( θ k ) − L k p o p ( θ E ) ≤ 1 1 − e − 1 ω k 2 . (21) L_k^{\mathrm{pop}}(\theta_k)-L_k^{\mathrm{pop}}(\theta^{\mathrm{E}})\le\frac{1}{1-e^{-1}}\omega_k^2. \tag*{(21)} L k pop ( θ k ) − L k pop ( θ E ) ≤ 1 − e − 1 1 ω k 2 . (21)
Proof. Since θ k \theta_k θ k minimizes L k L_k L k over Θ \Theta Θ and θ E ∈ Θ \theta^{\mathrm{E}} \in\Theta θ E ∈ Θ , we have L k ( θ k ) ≤ L k ( θ E ) L_k(\theta_k) \le L_k(\theta^{\mathrm{E}}) L k ( θ k ) ≤ L k ( θ E ) . Using L k ( θ ) = L k p o p ( θ ) − ⟨ θ , S k ⟩ L_k(\theta) = L_k^{\mathrm{pop}}(\theta) - \langle\theta, S_k\rangle L k ( θ ) = L k pop ( θ ) − ⟨ θ , S k ⟩ ,
L k p o p ( θ k ) − L k p o p ( θ E ) = ⟨ θ k − θ E , S k ⟩ + ( L k ( θ k ) − L k ( θ E ) ) ≤ ⟨ θ k − θ E , S k ⟩ . \begin{aligned}
L_k^{\mathrm{pop}}(\theta_k) - L_k^{\mathrm{pop}}(\theta^{\mathrm{E}}) = \langle\theta_k - \theta^{\mathrm{E}}, S_k\rangle+ \left(L_k(\theta_k) - L_k(\theta^{\mathrm{E}})\right) \\
&\le\langle\theta_k - \theta^{\mathrm{E}}, S_k\rangle.
\end{aligned} L k pop ( θ k ) − L k pop ( θ E ) = ⟨ θ k − θ E , S k ⟩ + ( L k ( θ k ) − L k ( θ E ) ) ≤ ⟨ θ k − θ E , S k ⟩ .
By (20) and, on E k \mathcal{E}_k E k , Lemma B.10 ,
⟨ θ k − θ E , S k ⟩ ≤ ω k ρ k ≤ 1 1 − e − 1 ω k 2 . \langle\theta_k - \theta^{\mathrm{E}}, S_k\rangle\le\omega_k \rho_k \le\frac{1}{1-e^{-1}}\omega_k^2. ⟨ θ k − θ E , S k ⟩ ≤ ω k ρ k ≤ 1 − e − 1 1 ω k 2 .
Fix the number of trajectories per round m ∈ N + m \in\mathbb{N}^{+} m ∈ N + and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Recall that U = ∑ P ∈ P U P U = \sum_{P \in\mathcal{P}} U_P U = ∑ P ∈ P U P has dimension d U ≤ d d_U \le d d U ≤ d . We assume d U ≥ 1 d_U \ge1 d U ≥ 1 : if U = { 0 } U = \{0\} U = { 0 } , all projected scores and Hessians vanish and there is nothing to estimate. At each round k ≥ 1 k \ge1 k ≥ 1 , P k P_k P k is selected based on the history, and m m m fresh, conditionally independent expert trajectories are drawn under P k P_k P k . Recall
H k : = ∑ i = 1 k H P i ( θ E ) , λ k : = min v ∈ U , ∥ v ∥ 2 = 1 ∥ v ∥ H k 2 , \mathcal{H}_k := \sum_{i=1}^{k} \mathcal{H}_{P_i}(\theta^{\mathrm{E}}), \qquad\lambda_k := \min_{\substack{v \in U,\\ \lVert v\rVert_2=1}} \lVert v\rVert_{\mathcal{H}_k}^{2}, H k := i = 1 ∑ k H P i ( θ E ) , λ k := v ∈ U , ∥ v ∥ 2 = 1 min ∥ v ∥ H k 2 ,
Σ P : = Cov π P τ ∈ P ( Π P ϕ ( τ ) ) , d ⋆ : = d U max P ∈ P ∥ H P ( θ E ) † / 2 Σ P H P ( θ E ) † / 2 ∥ 2 . \Sigma_P := \operatorname{Cov}_{\pi_P}^{\tau\in P}\left(\Pi_P \phi(\tau)\right), \qquad d_{\star} := d_U \max_{P \in\mathcal{P}} \left\lVert\mathcal{H}_P(\theta^{\mathrm{E}})^{\dagger/2}\Sigma_P\mathcal{H}_P(\theta^{\mathrm{E}})^{\dagger/2}\right\rVert_2. Σ P := Cov π P τ ∈ P ( Π P ϕ ( τ ) ) , d ⋆ := d U P ∈ P max H P ( θ E ) † /2 Σ P H P ( θ E ) † /2 2 .
Note that λ k = 0 \lambda_k = 0 λ k = 0 until the selected dynamics cover all of U U U .
Since Σ P \Sigma_P Σ P is supported on U P = im H P ( θ E ) U_P = \operatorname{im}\mathcal{H}_P(\theta^{\mathrm{E}}) U P = im H P ( θ E ) , the definition of d ⋆ d_{\star} d ⋆ gives
Σ P ⪯ d ⋆ d U H P ( θ E ) for every P ∈ P . (22) \Sigma_P \preceq\frac{d_{\star}}{d_U}\mathcal{H}_P(\theta^{\mathrm{E}}) \qquad\text{for every } P \in\mathcal{P}. \tag*{(22)} Σ P ⪯ d U d ⋆ H P ( θ E ) for every P ∈ P . (22)
Projecting the covariance decomposition (11) onto U P U_P U P gives
Σ P = η − 1 H P ( θ E ) + Π P Σ P d y n ( θ E ) Π P ⪰ η − 1 H P ( θ E ) . \Sigma_P = \eta^{-1}\mathcal{H}_P(\theta^{\mathrm{E}}) + \Pi_P\Sigma_P^{\mathrm{dyn}}(\theta^{\mathrm{E}})\Pi_P \succeq\eta^{-1}\mathcal{H}_P(\theta^{\mathrm{E}}). Σ P = η − 1 H P ( θ E ) + Π P Σ P dyn ( θ E ) Π P ⪰ η − 1 H P ( θ E ) .
Thus d ⋆ ≥ d U / η d_{\star} \ge d_U/\eta d ⋆ ≥ d U / η , with equality for deterministic transitions and initial state (Corollary B.2 ). The ratio d ⋆ / d U d_{\star}/d_U d ⋆ / d U measures the largest sampling variance relative to curvature along an identifiable direction, and stochastic transitions can only increase it.
We define the analysis regularization and logarithmic factor
γ : = 4 d U H 2 B ϕ 2 d ⋆ m , ℓ k , δ : = log 1 δ + d U log 2 + 1 2 log det ( I + H k γ ) + 1 4 . (23) \gamma:= \frac{4d_UH^2B_{\phi}^2}{d_{\star}m}, \qquad\ell_{k,\delta} := \log\frac{1}{\delta} + d_U\log2 + \frac{1}{2}\log\det\left(I + \frac{\mathcal{H}_k}{\gamma}\right) + \frac{1}{4}. \tag*{(23)} γ := d ⋆ m 4 d U H 2 B ϕ 2 , ℓ k , δ := log δ 1 + d U log 2 + 2 1 log det ( I + γ H k ) + 4 1 . (23)
The matrix inside the determinant is the identity on U ⊥ U^\perp U ⊥ , so only directions in U U U contribute. For the subsequent selection guarantees, we fix the budget n n n .
Lemma B.12 (Anytime self-normalized concentration). Let Assumption 2.1 hold, and define
X i , j : = 1 m Π P i ( ϕ ( τ i , j ) − ϕ P i E ) , S k : = ∑ i = 1 k ∑ j = 1 m X i , j , ω k : = ∥ S k ∥ H k † . X_{i,j} := \frac{1}{m}\Pi_{P_i}\left(\phi(\tau_{i,j})-\phi_{P_i}^{\mathrm{E}}\right), \qquad S_k := \sum_{i=1}^{k}\sum_{j=1}^{m}X_{i,j}, \qquad\omega_k := \lVert S_k\rVert_{\mathcal{H}_k^\dagger}. X i , j := m 1 Π P i ( ϕ ( τ i , j ) − ϕ P i E ) , S k := i = 1 ∑ k j = 1 ∑ m X i , j , ω k := ∥ S k ∥ H k † . With probability at least 1 − δ 1-\delta 1 − δ , simultaneously for all k ≥ 1 k \ge1 k ≥ 1 ,
∥ S k ∥ ( H k + γ I ) − 1 ≤ 2 d ⋆ ℓ k , δ d U m + 8 H B ϕ ℓ k , δ m λ k + γ . (24) \lVert S_k\rVert_{(\mathcal{H}_k+\gamma I)^{-1}} \le2\sqrt{\frac{d_{\star}\ell_{k,\delta}}{d_Um}}+\frac{8HB_{\phi}\ell_{k,\delta}}{m\sqrt{\lambda_k+\gamma}}. \tag*{(24)} ∥ S k ∥ ( H k + γ I ) − 1 ≤ 2 d U m d ⋆ ℓ k , δ + m λ k + γ 8 H B ϕ ℓ k , δ . (24) On the same event, every round k k k with λ k ≥ γ \lambda_k \ge\gamma λ k ≥ γ satisfies
ω k ≤ 2 2 d ⋆ ℓ k , δ d U m + 8 H B ϕ ℓ k , δ m λ k . (25) \omega_k \le2\sqrt{\frac{2d_{\star}\ell_{k,\delta}}{d_Um}}+\frac{8HB_{\phi}\ell_{k,\delta}}{m\sqrt{\lambda_k}}. \tag*{(25)} ω k ≤ 2 d U m 2 d ⋆ ℓ k , δ + m λ k 8 H B ϕ ℓ k , δ . (25)
Proof. Since the Hessian images do not depend on the parameters, U = ∑ P ∈ P im H P ( 0 ) U = \sum_{P \in\mathcal{P}} \operatorname{im} H_P(0) U = ∑ P ∈ P im H P ( 0 ) is fixed before sampling. In this proof, we work in a fixed orthonormal basis of U U U , with all vectors and matrices restricted to U U U (and I I I the identity on U U U ) unless otherwise indicated. Each increment X i , j X_{i,j} X i , j lies in U U U , as it is projected onto U P i ⊆ U U_{P_i} \subseteq U U P i ⊆ U . We use the shorthands
c : = d ⋆ d U m , r : = m 2 H B ϕ , c := \frac{d_\star}{d_U m}, \qquad r := \frac{m}{2 H B_{\phi}}, c := d U m d ⋆ , r := 2 H B ϕ m ,
so that γ c = r − 2 \gamma c = r^{-2} γ c = r − 2 by (23) .
Step 1: supermartingale. Let G i , j − 1 \mathcal{G}_{i,j-1} G i , j − 1 contain all information available before trajectory j j j of round i i i , including P i P_i P i and any learner randomization used to select it. Fresh sampling gives
E [ X i , j ∣ G i , j − 1 ] = 0 , ∥ X i , j ∥ 2 ≤ 2 H B ϕ m = 1 r , \mathbb{E}[X_{i,j} \mid\mathcal{G}_{i,j-1}] = 0, \qquad\lVert X_{i,j} \rVert_2 \leq\frac{2 H B_{\phi}}{m} = \frac{1}{r}, E [ X i , j ∣ G i , j − 1 ] = 0 , ∥ X i , j ∥ 2 ≤ m 2 H B ϕ = r 1 ,
and, by (22) ,
E [ X i , j X i , j T ∣ G i , j − 1 ] = Σ P i m 2 ⪯ c m H P i ( θ E ) . \mathbb{E}[X_{i,j}X_{i,j}^{\mathsf{T}} \mid\mathcal{G}_{i,j-1}] = \frac{\Sigma_{P_i}}{m^2} \preceq\frac{c}{m} H_{P_i}(\theta^E). E [ X i , j X i , j T ∣ G i , j − 1 ] = m 2 Σ P i ⪯ m c H P i ( θ E ) .
For every fixed z ∈ U z \in U z ∈ U with ∥ z ∥ 2 ≤ r \lVert z \rVert_2 \leq r ∥ z ∥ 2 ≤ r , we have ∣ z T X i , j ∣ ≤ 1 \lvert z^{\mathsf{T}} X_{i,j} \rvert\leq1 ∣ z T X i , j ∣ ≤ 1 . Using e x ≤ 1 + x + x 2 e^x \leq1 + x + x^2 e x ≤ 1 + x + x 2 for ∣ x ∣ ≤ 1 \lvert x \rvert\leq1 ∣ x ∣ ≤ 1 , we obtain
E [ e z T X i , j ∣ G i , j − 1 ] ≤ 1 + E [ ( z T X i , j ) 2 ∣ G i , j − 1 ] ≤ exp ( c m z T H P i ( θ E ) z ) . \mathbb{E}[e^{z^{\mathsf{T}}X_{i,j}} \mid\mathcal{G}_{i,j-1}] \leq1 + \mathbb{E}[(z^{\mathsf{T}}X_{i,j})^2 \mid\mathcal{G}_{i,j-1}] \leq\exp\left(\frac{c}{m} z^{\mathsf{T}} H_{P_i}(\theta^E) z\right). E [ e z T X i , j ∣ G i , j − 1 ] ≤ 1 + E [( z T X i , j ) 2 ∣ G i , j − 1 ] ≤ exp ( m c z T H P i ( θ E ) z ) .
Iterating this bound over the m m m trajectories of each round shows that
M k ( z ) : = exp ( z T S k − c z T H k z ) M_k(z) := \exp(z^{\mathsf{T}}S_k - c z^{\mathsf{T}}H_k z) M k ( z ) := exp ( z T S k − c z T H k z )
is a nonnegative supermartingale with M 0 ( z ) = 1 M_0(z) = 1 M 0 ( z ) = 1 .
Step 2: mixture and Ville’s inequality. Let ν 0 \nu_0 ν 0 be the uniform distribution on { z ∈ U : ∥ z ∥ 2 ≤ r } \{z \in U : \lVert z \rVert_2 \leq r\} { z ∈ U : ∥ z ∥ 2 ≤ r } . The mixture ∫ M k ( z ) ν 0 ( d z ) \int M_k(z)\,\nu_0(dz) ∫ M k ( z ) ν 0 ( d z ) is a nonnegative supermartingale starting at one. By Ville’s inequality, with probability at least 1 − δ 1 - \delta 1 − δ ,
∫ M k ( z ) ν 0 ( d z ) ≤ 1 / δ for all k ≥ 0. \int M_k(z)\,\nu_0(dz) \leq1/\delta\quad\text{for all } k \geq0. ∫ M k ( z ) ν 0 ( d z ) ≤ 1/ δ for all k ≥ 0.
On this event, the Donsker–Varadhan variational inequality gives, for every round k k k and every probability distribution ν ≪ ν 0 \nu\ll\nu_0 ν ≪ ν 0 ,
E z ∼ ν [ z T S k − c z T H k z ] ≤ KL ( ν ∥ ν 0 ) + log 1 δ . (26) \mathbb{E}_{z \sim\nu}[z^{\mathsf{T}}S_k - c z^{\mathsf{T}}H_k z] \leq\operatorname{KL}(\nu\Vert\nu_0) + \log\frac{1}{\delta}. \tag*{(26)} E z ∼ ν [ z T S k − c z T H k z ] ≤ KL ( ν ∥ ν 0 ) + log δ 1 . (26)
Since this holds for all such ν \nu ν simultaneously, ν \nu ν may depend on the data.
Step 3: choice of ν \nu ν . Fix k k k and define the positive definite matrix on U U U
W k : = c H k + r − 2 I = c ( H k + γ I ) . W_k := c H_k + r^{-2} I = c(H_k + \gamma I). W k := c H k + r − 2 I = c ( H k + γ I ) .
In particular, λ min ( W k ) = c ( λ k + γ ) ≥ r − 2 \lambda_{\min}(W_k) = c(\lambda_k + \gamma) \geq r^{-2} λ m i n ( W k ) = c ( λ k + γ ) ≥ r − 2 . If S k = 0 S_k = 0 S k = 0 , the claimed bounds at this round hold trivially. Otherwise, set
α k : = min { 1 2 , r λ min ( W k ) 2 ∥ S k ∥ W k − 1 } , \alpha_k := \min\left\{\frac{1}{2}, \frac{r\sqrt{\lambda_{\min}(W_k)}}{2\lVert S_k\rVert_{W_k^{-1}}}\right\}, α k := min { 2 1 , 2 ∥ S k ∥ W k − 1 r λ m i n ( W k ) } ,
and let ν k \nu_k ν k be uniform on the ellipsoid
{ α k W k − 1 S k + 1 2 W k − 1 / 2 u : u ∈ U , ∥ u ∥ 2 ≤ 1 } . \left\{\alpha_k W_k^{-1} S_k + \frac{1}{2}W_k^{-1/2}u : u \in U,\ \lVert u\rVert_2 \leq1\right\}. { α k W k − 1 S k + 2 1 W k − 1/2 u : u ∈ U , ∥ u ∥ 2 ≤ 1 } .
Its center and radius satisfy
∥ α k W k − 1 S k ∥ 2 ≤ α k ∥ S k ∥ W k − 1 λ min ( W k ) ≤ r 2 , 1 2 ∥ W k − 1 / 2 ∥ 2 ≤ r 2 , \lVert\alpha_k W_k^{-1} S_k\rVert_2 \leq\frac{\alpha_k \lVert S_k\rVert_{W_k^{-1}}}{\sqrt{\lambda_{\min}(W_k)}} \leq\frac{r}{2}, \qquad\frac{1}{2}\lVert W_k^{-1/2}\rVert_2 \leq\frac{r}{2}, ∥ α k W k − 1 S k ∥ 2 ≤ λ m i n ( W k ) α k ∥ S k ∥ W k − 1 ≤ 2 r , 2 1 ∥ W k − 1/2 ∥ 2 ≤ 2 r ,
where the second bound uses λ min ( W k ) ≥ r − 2 \lambda_{\min}(W_k) \geq r^{-2} λ m i n ( W k ) ≥ r − 2 . Hence ν k ≪ ν 0 \nu_k \ll\nu_0 ν k ≪ ν 0 . Its mean and covariance are
E ν k [ z ] = α k W k − 1 S k , Cov ν k ( z ) = W k − 1 4 ( d U + 2 ) , \mathbb{E}_{\nu_k}[z] = \alpha_k W_k^{-1}S_k, \qquad\operatorname{Cov}_{\nu_k}(z) = \frac{W_k^{-1}}{4(d_U + 2)}, E ν k [ z ] = α k W k − 1 S k , Cov ν k ( z ) = 4 ( d U + 2 ) W k − 1 ,
and, since r 2 W k = I + H k / γ r^{2}W_k=I+\mathcal{H}_k/\gamma r 2 W k = I + H k / γ , the ratio of support volumes gives
KL ( ν k ∥ ν 0 ) = d U log 2 + 1 2 log det U ( I + H k γ ) . \operatorname{KL}(\nu_k\Vert\nu_0)=d_U\log2+\frac{1}{2}\log\det_U\left(I+\frac{\mathcal{H}_k}{\gamma}\right). KL ( ν k ∥ ν 0 ) = d U log 2 + 2 1 log U det ( I + γ H k ) .
This restricted determinant equals the full-space determinant in (23), since H k \mathcal{H}_k H k vanishes on U ⊥ U^\perp U ⊥ .
Step 4: solving for S k S_k S k . Since 0 ⪯ c H k ⪯ W k 0\preceq c\mathcal{H}_k\preceq W_k 0 ⪯ c H k ⪯ W k ,
E ν k [ z ⊤ S k − c z ⊤ H k z ] = α k ∥ S k ∥ W k − 1 2 − c α k 2 S k ⊤ W k − 1 H k W k − 1 S k − c tr ( H k W k − 1 ) 4 ( d U + 2 ) ≥ α k ( 1 − α k ) ∥ S k ∥ W k − 1 2 − 1 4 . \begin{aligned}
\mathbb{E}_{\nu_k}\left[z^\top S_k-cz^\top\mathcal{H}_kz\right]
=\alpha_k\lVert S_k\rVert_{W_k^{-1}}^2-c\alpha_k^2S_k^\top W_k^{-1}\mathcal{H}_kW_k^{-1}S_k-\frac{c\operatorname{tr}(\mathcal{H}_kW_k^{-1})}{4(d_U+2)} \\
&\geq\alpha_k(1-\alpha_k)\lVert S_k\rVert_{W_k^{-1}}^2-\frac{1}{4}.
\end{aligned} E ν k [ z ⊤ S k − c z ⊤ H k z ] = α k ∥ S k ∥ W k − 1 2 − c α k 2 S k ⊤ W k − 1 H k W k − 1 S k − 4 ( d U + 2 ) c tr ( H k W k − 1 ) ≥ α k ( 1 − α k ) ∥ S k ∥ W k − 1 2 − 4 1 .
Substituting into (26) yields α k ( 1 − α k ) ∥ S k ∥ W k − 1 2 ≤ ℓ k , δ \alpha_k(1-\alpha_k)\lVert S_k\rVert_{W_k^{-1}}^2\leq\ell_{k,\delta} α k ( 1 − α k ) ∥ S k ∥ W k − 1 2 ≤ ℓ k , δ . If α k = 1 / 2 \alpha_k=1/2 α k = 1/2 , then ∥ S k ∥ W k − 1 ≤ 2 ℓ k , δ \lVert S_k\rVert_{W_k^{-1}}\leq2\sqrt{\ell_{k,\delta}} ∥ S k ∥ W k − 1 ≤ 2 ℓ k , δ . Otherwise, 1 − α k ≥ 1 / 2 1-\alpha_k\geq1/2 1 − α k ≥ 1/2 and the definition of α k \alpha_k α k gives ∥ S k ∥ W k − 1 ≤ 4 ℓ k , δ / ( r λ min ( W k ) ) \lVert S_k\rVert_{W_k^{-1}}\leq4\ell_{k,\delta}/(r\sqrt{\lambda_{\min}(W_k)}) ∥ S k ∥ W k − 1 ≤ 4 ℓ k , δ / ( r λ m i n ( W k ) ) . Combining the two cases,
∥ S k ∥ W k − 1 ≤ 2 ℓ k , δ + 4 ℓ k , δ r λ min ( W k ) . \lVert S_k\rVert_{W_k^{-1}}\leq2\sqrt{\ell_{k,\delta}}+\frac{4\ell_{k,\delta}}{r\sqrt{\lambda_{\min}(W_k)}}. ∥ S k ∥ W k − 1 ≤ 2 ℓ k , δ + r λ m i n ( W k ) 4 ℓ k , δ .
Since ∥ S k ∥ W k − 1 = c − 1 / 2 ∥ S k ∥ ( H k + γ I ) − 1 \lVert S_k\rVert_{W_k^{-1}}=c^{-1/2}\lVert S_k\rVert_{(\mathcal{H}_k+\gamma I)^{-1}} ∥ S k ∥ W k − 1 = c − 1/2 ∥ S k ∥ ( H k + γ I ) − 1 and λ min ( W k ) = c ( λ k + γ ) \lambda_{\min}(W_k)=c(\lambda_k+\gamma) λ m i n ( W k ) = c ( λ k + γ ) , rescaling proves (24). The norm agrees with its full-space version because S k ∈ U S_k\in U S k ∈ U .
Step 5: removing the regularization. If λ k ≥ γ \lambda_k\geq\gamma λ k ≥ γ , then H k ⪰ λ k I \mathcal{H}_k\succeq\lambda_kI H k ⪰ λ k I on U U U , so H k − 1 ⪯ ( 1 + γ / λ k ) ( H k + γ I ) − 1 \mathcal{H}_k^{-1}\preceq(1+\gamma/\lambda_k)(\mathcal{H}_k+\gamma I)^{-1} H k − 1 ⪯ ( 1 + γ / λ k ) ( H k + γ I ) − 1 there, and
ω k ≤ 1 + γ λ k ∥ S k ∥ ( H k + γ I ) − 1 . \omega_k\leq\sqrt{1+\frac{\gamma}{\lambda_k}}\lVert S_k\rVert_{(\mathcal{H}_k+\gamma I)^{-1}}. ω k ≤ 1 + λ k γ ∥ S k ∥ ( H k + γ I ) − 1 .
Applying (24) and 1 + γ / λ k ≤ 2 1+\gamma/\lambda_k\leq2 1 + γ / λ k ≤ 2 gives (25). The linear term simplifies exactly, because 1 + γ / λ k / λ k + γ = 1 / λ k \sqrt{1+\gamma/\lambda_k}/\sqrt{\lambda_k+\gamma}=1/\sqrt{\lambda_k} 1 + γ / λ k / λ k + γ = 1/ λ k . All bounds hold on the same event for every round k k k .
Let Θ \Theta Θ be convex with θ E ∈ Θ \theta^{\mathrm{E}}\in\Theta θ E ∈ Θ , and let θ k \theta_k θ k minimize the projected empirical loss over Θ \Theta Θ . Let R ≥ 2 η H B ϕ R\geq2\eta HB_\phi R ≥ 2 η H B ϕ be a valid pseudo-self-concordance constant from Lemma B.6. For the stochastic setting, the established choice is R = η H ( H + 1 ) B ϕ R=\eta H(H+1)B_\phi R = η H ( H + 1 ) B ϕ . The deterministic localization lemma remains a pathwise result and requires no additional concentration argument.
Lemma 3.1. Fix n ∈ N + n\in\mathbb{N}^{+} n ∈ N + , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and R : = η B ϕ H ( H + 1 ) R:=\eta B_\phi H(H+1) R := η B ϕ H ( H + 1 ) . Let n δ n_\delta n δ be the first round k ≤ n k\leq n k ≤ n such that
λ k ≥ 32 R 2 d ⋆ ( 1 − e − 1 ) 2 m log 3 η d ⋆ n m δ (27) \lambda_k\geq\frac{32R^2d_\star}{(1-e^{-1})^2m}\log\frac{3\eta d_\star nm}{\delta} \tag*{(27)} λ k ≥ ( 1 − e − 1 ) 2 m 32 R 2 d ⋆ log δ 3 η d ⋆ nm (27) with n δ = ∞ n_\delta=\infty n δ = ∞ if no such round exists. Then, with probability at least $1-\delta, s i m u l t a n e o u s l y f o r a l l , simultaneously for all , s im u l t an eo u s l y f or a l l n_\delta\leq k\leq n,
∥ θ k − θ E ∥ H k 2 ≤ 16 d ⋆ ( 1 − e − 1 ) 2 m log 3 η d ⋆ k m δ = : β k 2 ( δ ) . \lVert\theta_k-\theta^{\mathrm{E}}\rVert_{\mathcal{H}_k}^2\leq\frac{16d_\star}{(1-e^{-1})^2m}\log\frac{3\eta d_\star km}{\delta}=:\beta_k^2(\delta). ∥ θ k − θ E ∥ H k 2 ≤ ( 1 − e − 1 ) 2 m 16 d ⋆ log δ 3 η d ⋆ k m =: β k 2 ( δ ) .
Proof. Work on the event from Lemma B.12. Throughout, we use that d ⋆ ≥ d U / η d_\star\geq d_U/\eta d ⋆ ≥ d U / η and R = η H ( H + 1 ) B ϕ ≥ 2 η H B ϕ R=\eta H(H+1)B_\phi\geq2\eta HB_\phi R = η H ( H + 1 ) B ϕ ≥ 2 η H B ϕ , so that
R d ⋆ d U ≥ R η ≥ 2 H B ϕ . (28) R\frac{d_\star}{d_U}\geq\frac{R}{\eta}\geq2HB_\phi. \tag*{(28)} R d U d ⋆ ≥ η R ≥ 2 H B ϕ . (28)
The argument relies on the condition
λ k ≥ 32 R 2 d ⋆ ℓ k , δ ( 1 − e − 1 ) 2 d U m . (29) \lambda_k\geq\frac{32R^2d_\star\ell_{k,\delta}}{(1-e^{-1})^2d_Um}. \tag*{(29)} λ k ≥ ( 1 − e − 1 ) 2 d U m 32 R 2 d ⋆ ℓ k , δ . (29)
Step 1: the warm-up condition implies (28). By Proposition B.4, every eigenvalue of H k \mathcal{H}_k H k is at most k η H 2 B ϕ 2 k\eta H^2B_\phi^2 k η H 2 B ϕ 2 . With γ = 4 d U H 2 B ϕ 2 / ( d ⋆ m ) \gamma=4d_UH^2B_\phi^2/(d_\star m) γ = 4 d U H 2 B ϕ 2 / ( d ⋆ m ) , every eigenvalue of H k / γ \mathcal{H}_k/\gamma H k / γ is thus at most x / 4 x/4 x /4 , where x : = k m η d ⋆ / d U ≥ 1 x:=km\eta d_\star/d_U\geq1 x := k m η d ⋆ / d U ≥ 1 . Since the determinant in ℓ k , δ \ell_{k,\delta} ℓ k , δ only involves the d U d_U d U dimensions of U U U ,
ℓ k , δ ≤ log 1 δ + d U log 2 + d U 2 log ( 1 + x 4 ) + 1 4 = log 1 δ + d U 2 log ( 4 + x ) + 1 4 . \ell_{k,\delta}\leq\log\frac{1}{\delta}+d_U\log2+\frac{d_U}{2}\log\left(1+\frac{x}{4}\right)+\frac{1}{4}
=\log\frac{1}{\delta}+\frac{d_U}{2}\log(4+x)+\frac{1}{4}. ℓ k , δ ≤ log δ 1 + d U log 2 + 2 d U log ( 1 + 4 x ) + 4 1 = log δ 1 + 2 d U log ( 4 + x ) + 4 1 .
Since x ≥ 1 x \ge1 x ≥ 1 , we have 9 x 2 / ( 4 + x ) ≥ 9 / 5 9x^2/(4+x) \ge9/5 9 x 2 / ( 4 + x ) ≥ 9/5 and 1 2 log 9 5 ≥ 1 4 \frac{1}{2}\log\frac{9}{5} \ge\frac{1}{4} 2 1 log 5 9 ≥ 4 1 , hence d U 2 log ( 4 + x ) + 1 4 ≤ d U log ( 3 x ) \frac{d_U}{2}\log(4+x)+\frac{1}{4} \le d_U\log(3x) 2 d U log ( 4 + x ) + 4 1 ≤ d U log ( 3 x ) . With log 1 δ ≤ d U log 1 δ \log\frac{1}{\delta} \le d_U\log\frac{1}{\delta} log δ 1 ≤ d U log δ 1 and x ≤ k m η d ⋆ x \le km\eta d_{\star} x ≤ k m η d ⋆ , this gives
ℓ k , δ ≤ d U log ( 3 x / δ ) ≤ d U log ( 3 η d ⋆ k m / δ ) . (30) \ell_{k,\delta} \le d_U\log(3x/\delta) \le d_U\log(3\eta d_{\star}km/\delta). \tag*{(30)} ℓ k , δ ≤ d U log ( 3 x / δ ) ≤ d U log ( 3 η d ⋆ k m / δ ) . (30)
In particular, ℓ k , δ ≤ d U log ( 3 η d ⋆ n m / δ ) \ell_{k,\delta} \le d_U\log(3\eta d_{\star}nm/\delta) ℓ k , δ ≤ d U log ( 3 η d ⋆ nm / δ ) for every 1 ≤ k ≤ n 1 \le k \le n 1 ≤ k ≤ n . Since H k ⪰ H n δ \mathcal{H}_k \succeq\mathcal{H}_{n_\delta} H k ⪰ H n δ for k ≥ n δ k \ge n_\delta k ≥ n δ , the warm-up condition (4) implies
λ k ≥ λ n δ ≥ 32 R 2 d ⋆ ( 1 − e − 1 ) 2 m log 3 η d ⋆ n m δ ≥ 32 R 2 d ⋆ ℓ k , δ ( 1 − e − 1 ) 2 d U m \lambda_k \ge\lambda_{n_\delta} \ge\frac{32R^2d_{\star}}{(1-e^{-1})^2m}\log\frac{3\eta d_{\star}nm}{\delta} \ge\frac{32R^2d_{\star}\ell_{k,\delta}}{(1-e^{-1})^2d_Um} λ k ≥ λ n δ ≥ ( 1 − e − 1 ) 2 m 32 R 2 d ⋆ log δ 3 η d ⋆ nm ≥ ( 1 − e − 1 ) 2 d U m 32 R 2 d ⋆ ℓ k , δ
for every n δ ≤ k ≤ n n_\delta\le k \le n n δ ≤ k ≤ n .
Thus (29) holds throughout the remaining budget, independently of the dynamics selected after the warm-up.
Step 2: localization. Fix a round n δ ≤ k ≤ n n_\delta\le k \le n n δ ≤ k ≤ n . Using ℓ k , δ ≥ 1 / 4 \ell_{k,\delta} \ge1/4 ℓ k , δ ≥ 1/4 , the right-hand side of (29) is at least 8 R 2 d ⋆ / ( d U m ) 8R^2d_{\star}/(d_Um) 8 R 2 d ⋆ / ( d U m ) , which is at least γ \gamma γ since ( R d ⋆ / d U ) 2 ≥ 4 H 2 B ϕ 2 (Rd_{\star}/d_U)^2 \ge4H^2B_\phi^2 ( R d ⋆ / d U ) 2 ≥ 4 H 2 B ϕ 2 by (28) . Hence λ k ≥ γ \lambda_k \ge\gamma λ k ≥ γ and (25) applies. We bound both of its terms by ( 1 − e − 1 ) λ k / ( 2 R ) (1-e^{-1})\sqrt{\lambda_k}/(2R) ( 1 − e − 1 ) λ k / ( 2 R ) . For the first, this is exactly (29) :
2 2 d ⋆ ℓ k , δ d U m ≤ ( 1 − e − 1 ) λ k 2 R . 2\sqrt{\frac{2d_{\star}\ell_{k,\delta}}{d_Um}} \le\frac{(1-e^{-1})\sqrt{\lambda_k}}{2R}. 2 d U m 2 d ⋆ ℓ k , δ ≤ 2 R ( 1 − e − 1 ) λ k .
For the second, it holds whenever λ k ≥ 16 R H B ϕ ℓ k , δ / ( ( 1 − e − 1 ) m ) \lambda_k \ge16RHB_\phi\ell_{k,\delta}/((1-e^{-1})m) λ k ≥ 16 R H B ϕ ℓ k , δ / (( 1 − e − 1 ) m ) , which (29) implies by (28) . Consequently, ω k ≤ ( 1 − e − 1 ) λ k / R = ( 1 − e − 1 ) / R k \omega_k \le(1-e^{-1})\sqrt{\lambda_k}/R = (1-e^{-1})/R_k ω k ≤ ( 1 − e − 1 ) λ k / R = ( 1 − e − 1 ) / R k , so the event E k \mathcal{E}_k E k holds, and Lemma B.10 yields
R λ k ∥ θ k − θ E ∥ H k ≤ 1 , ∥ θ k − θ E ∥ H k ≤ ω k 1 − e − 1 . \frac{R}{\sqrt{\lambda_k}}\|\theta_k-\theta^E\|_{\mathcal{H}_k} \le1,\qquad\|\theta_k-\theta^E\|_{\mathcal{H}_k} \le\frac{\omega_k}{1-e^{-1}}. λ k R ∥ θ k − θ E ∥ H k ≤ 1 , ∥ θ k − θ E ∥ H k ≤ 1 − e − 1 ω k .
Step 3: confidence bound. Using (29) and (28) again,
8 H B ϕ ℓ k , δ m λ k ≤ 1 − e − 1 2 d ⋆ ℓ k , δ d U m , hence ω k ≤ ( 2 2 + 1 − e − 1 2 ) d ⋆ ℓ k , δ d U m ≤ 4 d ⋆ ℓ k , δ d U m . \frac{8HB_\phi\ell_{k,\delta}}{m\sqrt{\lambda_k}} \le\frac{1-e^{-1}}{\sqrt{2}}\sqrt{\frac{d_{\star}\ell_{k,\delta}}{d_Um}},
\qquad\text{hence}\quad\omega_k \le\left(2\sqrt{2}+\frac{1-e^{-1}}{\sqrt{2}}\right)\sqrt{\frac{d_{\star}\ell_{k,\delta}}{d_Um}} \le4\sqrt{\frac{d_{\star}\ell_{k,\delta}}{d_Um}}. m λ k 8 H B ϕ ℓ k , δ ≤ 2 1 − e − 1 d U m d ⋆ ℓ k , δ , hence ω k ≤ ( 2 2 + 2 1 − e − 1 ) d U m d ⋆ ℓ k , δ ≤ 4 d U m d ⋆ ℓ k , δ .
Combining the last two bounds with (30) gives
∥ θ k − θ E ∥ H k 2 ≤ 16 d ⋆ ℓ k , δ ( 1 − e − 1 ) 2 d U m ≤ 16 d ⋆ log ( 3 η d ⋆ k m / δ ) ( 1 − e − 1 ) 2 m . \|\theta_k-\theta^E\|_{\mathcal{H}_k}^2 \le\frac{16d_{\star}\ell_{k,\delta}}{(1-e^{-1})^2d_Um} \le\frac{16d_{\star}\log(3\eta d_{\star}km/\delta)}{(1-e^{-1})^2m}. ∥ θ k − θ E ∥ H k 2 ≤ ( 1 − e − 1 ) 2 d U m 16 d ⋆ ℓ k , δ ≤ ( 1 − e − 1 ) 2 m 16 d ⋆ log ( 3 η d ⋆ k m / δ ) .
Every step holds simultaneously for all n δ ≤ k ≤ n n_\delta\le k \le n n δ ≤ k ≤ n on the event from Lemma B.12, which has probability at least 1 − δ 1-\delta 1 − δ .
Proofs of Section 3.2
We now lay out the results employed to prove Lemma 3.2, Lemma 3.3, Theorem 3.6 and Theorem 3.7.
Supporting lemmas
We first present useful identities of the suboptimality (1) .
Proposition C.1 (Remainder form). Let Assumption 2.1 hold and let P P P be any dynamics. For any θ 1 , θ 2 ∈ R d \theta_1,\theta_2 \in\mathbb{R}^d θ 1 , θ 2 ∈ R d ,
V P ⋆ ( θ 2 ) − V P π P ( θ 1 ) ( θ 2 ) = ∥ θ 2 − θ 1 ∥ G P ( θ 1 , θ 2 ) 2 , V_P^\star(\theta_2)-V_P^{\pi_P(\theta_1)}(\theta_2)=\|\theta_2-\theta_1\|_{G_P(\theta_1,\theta_2)}^2, V P ⋆ ( θ 2 ) − V P π P ( θ 1 ) ( θ 2 ) = ∥ θ 2 − θ 1 ∥ G P ( θ 1 , θ 2 ) 2 , where G P G_P G P is defined in Lemma B.7.
Proof. Let u : = θ 2 − θ 1 u:=\theta_2-\theta_1 u := θ 2 − θ 1 . A Taylor expansion of V P ⋆ V_P^\star V P ⋆ around θ 1 \theta_1 θ 1 with integral remainder, together with Lemma B.1, gives
V P ⋆ ( θ 2 ) = V P ⋆ ( θ 1 ) + D V P ⋆ ( θ 1 ) [ u ] + ∫ 0 1 ( 1 − t ) D 2 V P ⋆ ( θ 1 + t u ) [ u , u ] d t = V P ⋆ ( θ 1 ) + ⟨ u , ϕ P ( θ 1 ) ⟩ + ∥ u ∥ G P ( θ 1 , θ 2 ) 2 . \begin{aligned}
V_P^\star(\theta_2)&=V_P^\star(\theta_1)+DV_P^\star(\theta_1)[u]+\int_0^1(1-t)D^2V_P^\star(\theta_1+tu)[u,u]\,dt\\
&=V_P^\star(\theta_1)+\langle u,\phi_P(\theta_1)\rangle+\|u\|_{G_P(\theta_1,\theta_2)}^2.
\end{aligned} V P ⋆ ( θ 2 ) = V P ⋆ ( θ 1 ) + D V P ⋆ ( θ 1 ) [ u ] + ∫ 0 1 ( 1 − t ) D 2 V P ⋆ ( θ 1 + t u ) [ u , u ] d t = V P ⋆ ( θ 1 ) + ⟨ u , ϕ P ( θ 1 )⟩ + ∥ u ∥ G P ( θ 1 , θ 2 ) 2 .
On the other hand, the regularized value of the fixed policy π P ( θ 1 ) \pi_P(\theta_1) π P ( θ 1 ) is affine in the reward parameters, with slope ϕ P ( θ 1 ) \phi_P(\theta_1) ϕ P ( θ 1 ) , and equals V P ∗ ( θ 1 ) V_P^*(\theta_1) V P ∗ ( θ 1 ) at θ 1 \theta_1 θ 1 . Hence
V P π P ( θ 1 ) ( θ 2 ) = V P ∗ ( θ 1 ) + ⟨ u , ϕ P ( θ 1 ) ⟩ . V_P^{\pi_P(\theta_1)}(\theta_2)=V_P^*(\theta_1)+\langle u,\phi_P(\theta_1)\rangle. V P π P ( θ 1 ) ( θ 2 ) = V P ∗ ( θ 1 ) + ⟨ u , ϕ P ( θ 1 )⟩ .
Subtracting the two expressions gives the result.
Proposition C.2 (Pseudo self-concordant suboptimality bound). Let Assumption 2.1 hold, let P P P be any dynamics, and let θ 1 , θ 2 ∈ R d \theta_1,\theta_2\in\mathbb{R}^d θ 1 , θ 2 ∈ R d . Then, for any symmetric positive semidefinite matrix X ∈ R d × d X\in\mathbb{R}^{d\times d} X ∈ R d × d with im H P ( θ 2 ) ⊆ im X \operatorname{im}\mathcal{H}_P(\theta_2)\subseteq\operatorname{im}X im H P ( θ 2 ) ⊆ im X ,
V P ∗ ( θ 2 ) − V P π P ( θ 1 ) ( θ 2 ) ≤ ψ ( R ∥ θ 2 − θ 1 ∥ Π P ) ∥ X † / 2 H P ( θ 2 ) X † / 2 ∥ 2 ∥ θ 2 − θ 1 ∥ X 2 . V_P^*(\theta_2)-V_P^{\pi_P(\theta_1)}(\theta_2)\leq\psi\left(R\lVert\theta_2-\theta_1\rVert_{\Pi_P}\right)\left\lVert X^{\dagger/2}\mathcal{H}_P(\theta_2)X^{\dagger/2}\right\rVert_2\lVert\theta_2-\theta_1\rVert_X^2. V P ∗ ( θ 2 ) − V P π P ( θ 1 ) ( θ 2 ) ≤ ψ ( R ∥ θ 2 − θ 1 ∥ Π P ) X † /2 H P ( θ 2 ) X † /2 2 ∥ θ 2 − θ 1 ∥ X 2 . where ψ \psi ψ is defined in Lemma B.7 .
Proof. Set u : = θ 2 − θ 1 u:=\theta_2-\theta_1 u := θ 2 − θ 1 and ψ u : = ψ ( R ∥ u ∥ Π P ) \psi_u:=\psi\left(R\lVert u\rVert_{\Pi_P}\right) ψ u := ψ ( R ∥ u ∥ Π P ) . By Proposition C.1 and item 2 of Lemma B.7 ,
V P ∗ ( θ 2 ) − V P π P ( θ 1 ) ( θ 2 ) = ∥ u ∥ G P ( θ 1 , θ 2 ) 2 ≤ ψ u ∥ u ∥ H P ( θ 2 ) 2 . V_P^*(\theta_2)-V_P^{\pi_P(\theta_1)}(\theta_2)=\lVert u\rVert^2_{G_P(\theta_1,\theta_2)}\leq\psi_u\lVert u\rVert^2_{\mathcal{H}_P(\theta_2)}. V P ∗ ( θ 2 ) − V P π P ( θ 1 ) ( θ 2 ) = ∥ u ∥ G P ( θ 1 , θ 2 ) 2 ≤ ψ u ∥ u ∥ H P ( θ 2 ) 2 .
Since X 1 / 2 X † / 2 = X † / 2 X 1 / 2 X^{1/2}X^{\dagger/2}=X^{\dagger/2}X^{1/2} X 1/2 X † /2 = X † /2 X 1/2 is the orthogonal projector onto im X \operatorname{im}X im X , and im H P ( θ 2 ) ⊆ im X \operatorname{im}\mathcal{H}_P(\theta_2)\subseteq\operatorname{im}X im H P ( θ 2 ) ⊆ im X , we have H P ( θ 2 ) = X 1 / 2 ( X † / 2 H P ( θ 2 ) X † / 2 ) X 1 / 2 \mathcal{H}_P(\theta_2)=X^{1/2}\left(X^{\dagger/2}\mathcal{H}_P(\theta_2)X^{\dagger/2}\right)X^{1/2} H P ( θ 2 ) = X 1/2 ( X † /2 H P ( θ 2 ) X † /2 ) X 1/2 . Hence
∥ u ∥ H P ( θ 2 ) 2 = ( X 1 / 2 u ) T ( X † / 2 H P ( θ 2 ) X † / 2 ) ( X 1 / 2 u ) ≤ ∥ X † / 2 H P ( θ 2 ) X † / 2 ∥ 2 ∥ u ∥ X 2 , \lVert u\rVert^2_{\mathcal{H}_P(\theta_2)}=\left(X^{1/2}u\right)^{\mathsf{T}}\left(X^{\dagger/2}\mathcal{H}_P(\theta_2)X^{\dagger/2}\right)\left(X^{1/2}u\right)\leq\left\lVert X^{\dagger/2}\mathcal{H}_P(\theta_2)X^{\dagger/2}\right\rVert_2\lVert u\rVert_X^2, ∥ u ∥ H P ( θ 2 ) 2 = ( X 1/2 u ) T ( X † /2 H P ( θ 2 ) X † /2 ) ( X 1/2 u ) ≤ X † /2 H P ( θ 2 ) X † /2 2 ∥ u ∥ X 2 ,
which concludes the proof.
We now show that once the warm-up phase has terminated, i.e, condition (4), pseudo self-concordance factors (Lemma B.6 ) are upper bounded by their localized constants.
Lemma C.3 (Localized self-concordance constants). Let Assumption 2.1 hold and θ E ∈ Θ \theta^E\in\Theta θ E ∈ Θ . On the event of Lemma B.12 , for every round n δ ≤ k ≤ n n_\delta\leq k\leq n n δ ≤ k ≤ n , where n δ n_\delta n δ is defined by the warm-up condition (4), and every P ∈ P P\in\mathcal{P} P ∈ P ,
ψ ( R ∥ θ E − θ k ∥ Π P ) ≤ 1 , exp ( R ∥ θ E − θ k ∥ Π P ) ≤ e . \psi\left(R\lVert\theta^E-\theta_k\rVert_{\Pi_P}\right)\leq1,\qquad\exp\left(R\lVert\theta^E-\theta_k\rVert_{\Pi_P}\right)\leq e. ψ ( R ∥ θ E − θ k ∥ Π P ) ≤ 1 , exp ( R ∥ θ E − θ k ∥ Π P ) ≤ e .
Proof. Fix P ∈ P P\in\mathcal{P} P ∈ P and n δ ≤ k ≤ n n_\delta\leq k\leq n n δ ≤ k ≤ n . By Step 1 of the proof of Lemma 3.1 , λ k ≥ λ n δ > 0 \lambda_k\geq\lambda_{n_\delta}>0 λ k ≥ λ n δ > 0 . Since U P ⊆ U U_P\subseteq U U P ⊆ U and H k ⪰ λ k Π U \mathcal{H}_k\succeq\lambda_k\Pi_U H k ⪰ λ k Π U (see the proof of Lemma B.8 ),
R ∥ θ E − θ k ∥ Π P ≤ R ∥ θ E − θ k ∥ Π U ≤ R ∥ θ E − θ k ∥ H k λ k = R k ρ k . R\lVert\theta^E-\theta_k\rVert_{\Pi_P}\leq R\lVert\theta^E-\theta_k\rVert_{\Pi_U}\leq R\frac{\lVert\theta^E-\theta_k\rVert_{\mathcal{H}_k}}{\sqrt{\lambda_k}}=R_k\rho_k. R ∥ θ E − θ k ∥ Π P ≤ R ∥ θ E − θ k ∥ Π U ≤ R λ k ∥ θ E − θ k ∥ H k = R k ρ k .
By Step 2 of the proof of Lemma 3.1 , the localization event E k \mathcal{E}_k E k (17) holds, so Lemma B.10 gives R k ρ k ≤ 1 R_k\rho_k\leq1 R k ρ k ≤ 1 . Since ψ \psi ψ and exp \exp exp are increasing,
ψ ( R ∥ θ E − θ k ∥ Π P ) ≤ ψ ( 1 ) = 1 and exp ( R ∥ θ E − θ k ∥ Π P ) ≤ e . \psi\left(R\lVert\theta^E-\theta_k\rVert_{\Pi_P}\right)\leq\psi(1)=1\qquad\text{and}\qquad\exp\left(R\lVert\theta^E-\theta_k\rVert_{\Pi_P}\right)\leq e. ψ ( R ∥ θ E − θ k ∥ Π P ) ≤ ψ ( 1 ) = 1 and exp ( R ∥ θ E − θ k ∥ Π P ) ≤ e .
Additionally, we provide a characterization of the lower bound of optimal value Hessians evaluated at the expert parameter θ E \theta^E θ E , as well as the resulting once-cycle coverage λ ∣ P ∣ \lambda_{|\mathcal{P}|} λ ∣ P ∣ obtained after following round-robin for ∣ P ∣ |\mathcal{P}| ∣ P ∣ rounds.
Lemma C.4 (Hessian lower bound). Assume finite S S S and A A A , and U ≠ { 0 } U\neq\{0\} U = { 0 } . Let ν h , P E ( s ) \nu^E_{h,P}(s) ν h , P E ( s ) denote the probability that the expert visits s s s at timestep h h h under P P P , and call ( h , s ) (h,s) ( h , s ) reachable under P P P when this probability is positive. For v ∈ R d v\in\mathbb{R}^d v ∈ R d , let α h , P v ( s , a ) : = ⟨ v , A h , P θ E ( s , a ) ⟩ \alpha^v_{h,P}(s,a):=\langle v,A^{\theta^E}_{h,P}(s,a)\rangle α h , P v ( s , a ) := ⟨ v , A h , P θ E ( s , a )⟩ , and define the minimal expert occupancy and the minimal identifiable advantage gap
μ min : = min P ∈ P , a ∈ A ( h , s ) r e a c h a b l e ν h , P E ( s ) π h , P E ( a ∣ s ) , Δ : = min P ∈ P : U P ≠ { 0 } v ∈ U P , ∥ v ∥ 2 = 1 min ( h , s ) r e a c h a b l e a ∈ A ∣ α h , P v ( s , a ) ∣ . \mu_{\min}:=\min_{\substack{P\in\mathcal{P},\ a\in A\\(h,s)\ \mathrm{reachable}}\!}\nu^E_{h,P}(s)\pi^E_{h,P}(a\mid s),\qquad
\Delta:=\min_{\substack{P\in\mathcal{P}: \ U_P\neq\{0\}\\v\in U_P,\ \lVert v\rVert_2=1}}\ \min_{\substack{(h,s)\ \mathrm{reachable}\\a\in A}}\left|\alpha^v_{h,P}(s,a)\right|. μ m i n := P ∈ P , a ∈ A ( h , s ) reachable min ν h , P E ( s ) π h , P E ( a ∣ s ) , Δ := P ∈ P : U P = { 0 } v ∈ U P , ∥ v ∥ 2 = 1 min ( h , s ) reachable a ∈ A min α h , P v ( s , a ) . Then μ min > 0 \mu_{\min}>0 μ m i n > 0 , Δ > 0 \Delta>0 Δ > 0 , and for every P ∈ P P\in\mathcal{P} P ∈ P ,
H P ( θ E ) ⪰ κ Π P , κ : = η μ min Δ 2 . \mathcal{H}_P(\theta^E)\succeq\kappa\Pi_P,\qquad\kappa:=\eta\mu_{\min}\Delta^2. H P ( θ E ) ⪰ κ Π P , κ := η μ m i n Δ 2 .
Proof. Fix P ∈ P P \in\mathcal{P} P ∈ P and v ∈ R d v \in\mathbb{R}^d v ∈ R d .
Step 1 (per-step decomposition). By definition of the feature advantage, E a ∼ π h , P E ( ⋅ ∣ s ) [ α h , P v ( s , a ) ] = 0 \mathbb{E}_{a \sim\pi^E_{h,P}(\cdot\mid s)}\left[\alpha^v_{h,P}(s,a)\right] = 0 E a ∼ π h , P E ( ⋅ ∣ s ) [ α h , P v ( s , a ) ] = 0 for every ( h , s ) (h,s) ( h , s ) . Hence ( α h , P v ( s h , a h ) ) h = 1 H \left(\alpha^v_{h,P}(s_h,a_h)\right)_{h=1}^{H} ( α h , P v ( s h , a h ) ) h = 1 H is a martingale difference sequence along the expert trajectory, and its terms are uncorrelated. Since π P E = π P ( θ E ) \pi^E_P = \pi_P(\theta^E) π P E = π P ( θ E ) , Lemma B.1 gives
∥ v ∥ H P ( θ E ) 2 = η E [ ( ∑ h = 1 H α h , P v ( s h , a h ) ) 2 ] = η ∑ h = 1 H ∑ s , a ν h , P E ( s ) π h , P E ( a ∣ s ) ( α h , P v ( s , a ) ) 2 . \|v\|^2_{\mathcal{H}_P(\theta^E)} = \eta\mathbb{E}\left[\left(\sum_{h=1}^{H}\alpha^v_{h,P}(s_h,a_h)\right)^2\right] = \eta\sum_{h=1}^{H}\sum_{s,a}\nu^E_{h,P}(s)\pi^E_{h,P}(a \mid s)(\alpha^v_{h,P}(s,a))^2. ∥ v ∥ H P ( θ E ) 2 = η E ( h = 1 ∑ H α h , P v ( s h , a h ) ) 2 = η h = 1 ∑ H s , a ∑ ν h , P E ( s ) π h , P E ( a ∣ s ) ( α h , P v ( s , a ) ) 2 .
Step 2 (positivity of μ min \mu_{\min} μ m i n and Δ \Delta Δ ). Since π h , P E \pi^E_{h,P} π h , P E has full support (9) and S \mathcal{S} S , A \mathcal{A} A and P \mathcal{P} P are finite, μ min > 0 \mu_{\min} > 0 μ m i n > 0 .
By Step 1, ∥ v ∥ H P ( θ E ) 2 \|v\|^2_{\mathcal{H}_P(\theta^E)} ∥ v ∥ H P ( θ E ) 2 is a sum of non-negative terms with positive weights on reachable pairs. Hence v ∈ ker ( H P ( θ E ) ) v \in\ker(\mathcal{H}_P(\theta^E)) v ∈ ker ( H P ( θ E )) if and only if α h , P v ( s , a ) = 0 \alpha^v_{h,P}(s,a) = 0 α h , P v ( s , a ) = 0 for every h h h , every s s s reachable at h h h , and every a a a . Since U P = ker ( H P ( θ E ) ) ⊥ U_P = \ker(\mathcal{H}_P(\theta^E))^\perp U P = ker ( H P ( θ E ) ) ⊥ (Proposition B.5), every unit v ∈ U P v \in U_P v ∈ U P lies outside the kernel, so the inner maximum defining Δ \Delta Δ is strictly positive. This maximum is a maximum of finitely many continuous functions of v v v , hence continuous, and the unit sphere of U P U_P U P is compact. The minimum over v v v is therefore attained and strictly positive, and finiteness of P \mathcal{P} P gives Δ > 0 \Delta> 0 Δ > 0 .
Step 3 (conclusion). If U P = { 0 } U_P = \{0\} U P = { 0 } , then H P ( θ E ) = 0 = Π P \mathcal{H}_P(\theta^E) = 0 = \Pi_P H P ( θ E ) = 0 = Π P and the claim holds trivially. Otherwise, let w ∈ U P w \in U_P w ∈ U P be a unit vector, and let ( h ⋆ , s ⋆ , a ⋆ ) (h^\star,s^\star,a^\star) ( h ⋆ , s ⋆ , a ⋆ ) attain the inner maximum in the definition of Δ \Delta Δ for w w w . Keeping only that term in Step 1,
∥ w ∥ H P ( θ E ) 2 ≥ η ν h ⋆ , P E ( s ⋆ ) π h ⋆ , P E ( a ⋆ ∣ s ⋆ ) ( α h ⋆ , P w ( s ⋆ , a ⋆ ) ) 2 ≥ η μ min Δ 2 = κ . \|w\|^2_{\mathcal{H}_P(\theta^E)} \ge\eta\nu^E_{h^\star,P}(s^\star)\pi^E_{h^\star,P}(a^\star\mid s^\star)(\alpha^w_{h^\star,P}(s^\star,a^\star))^2 \ge\eta\mu_{\min}\Delta^2 = \kappa. ∥ w ∥ H P ( θ E ) 2 ≥ η ν h ⋆ , P E ( s ⋆ ) π h ⋆ , P E ( a ⋆ ∣ s ⋆ ) ( α h ⋆ , P w ( s ⋆ , a ⋆ ) ) 2 ≥ η μ m i n Δ 2 = κ .
For general v ∈ R d v \in\mathbb{R}^d v ∈ R d , since H P ( θ E ) \mathcal{H}_P(\theta^E) H P ( θ E ) vanishes on U P ⊥ U_P^\perp U P ⊥ , ∥ v ∥ H P ( θ E ) 2 = ∥ Π P v ∥ H P ( θ E ) 2 ≥ κ ∥ Π P v ∥ 2 2 = κ v ⊤ Π P v \|v\|^2_{\mathcal{H}_P(\theta^E)} = \|\Pi_Pv\|^2_{\mathcal{H}_P(\theta^E)} \ge\kappa\|\Pi_Pv\|_2^2 = \kappa v^\top\Pi_Pv ∥ v ∥ H P ( θ E ) 2 = ∥ Π P v ∥ H P ( θ E ) 2 ≥ κ ∥ Π P v ∥ 2 2 = κ v ⊤ Π P v , which is H P ( θ E ) ⪰ κ Π P \mathcal{H}_P(\theta^E) \succeq\kappa\Pi_P H P ( θ E ) ⪰ κ Π P .
Corollary C.5. (Information coverage after one cycle). Assume U ≠ { 0 } U \ne\{0\} U = { 0 } . After selecting each dynamics P ∈ P P \in\mathcal{P} P ∈ P once,
λ ∣ P ∣ U = λ min U ( ∑ P ∈ P H P ( θ E ) ) ≥ η μ min Δ 2 min v ∈ U ∥ v ∥ 2 = 1 ∑ P ∈ P ∥ Π P v ∥ 2 2 > 0 , \lambda^{U}_{|\mathcal{P}|} = \lambda_{\min}^{U}\left(\sum_{P \in\mathcal{P}}\mathcal{H}_P(\theta^E)\right) \ge\eta\mu_{\min}\Delta^2 \min_{\substack{v \in U \\ \|v\|_2=1}}\sum_{P \in\mathcal{P}}\|\Pi_Pv\|_2^2 > 0, λ ∣ P ∣ U = λ m i n U ( P ∈ P ∑ H P ( θ E ) ) ≥ η μ m i n Δ 2 v ∈ U ∥ v ∥ 2 = 1 min P ∈ P ∑ ∥ Π P v ∥ 2 2 > 0 , where μ min \mu_{\min} μ m i n and Δ \Delta Δ are defined in Lemma C.4 .
Proof. After one complete cycle, H ∣ P ∣ = ∑ P ∈ P H P ( θ E ) \mathcal{H}_{|\mathcal{P}|} = \sum_{P \in\mathcal{P}}\mathcal{H}_P(\theta^E) H ∣ P ∣ = ∑ P ∈ P H P ( θ E ) . By Lemma C.4 ,
H ∣ P ∣ ⪰ η μ min Δ 2 ∑ P ∈ P Π P , \mathcal{H}_{|\mathcal{P}|} \succeq\eta\mu_{\min}\Delta^2\sum_{P \in\mathcal{P}}\Pi_P, H ∣ P ∣ ⪰ η μ m i n Δ 2 P ∈ P ∑ Π P ,
and since v ⊤ Π P v = ∥ Π P v ∥ 2 2 v^\top\Pi_Pv = \|\Pi_Pv\|_2^2 v ⊤ Π P v = ∥ Π P v ∥ 2 2 , taking the minimum of the quadratic form over unit vectors in U U U gives the lower bound. Moreover, since each Π P \Pi_P Π P is an orthogonal projector and U = ∑ P ∈ P U P U = \sum_{P \in\mathcal{P}}U_P U = ∑ P ∈ P U P ,
ker ( ∑ P ∈ P Π P ) = ⋂ P ∈ P U P ⊥ = U ⊥ , \ker\left(\sum_{P \in\mathcal{P}}\Pi_P\right) = \bigcap_{P \in\mathcal{P}}U_P^\perp= U^\perp, ker ( P ∈ P ∑ Π P ) = P ∈ P ⋂ U P ⊥ = U ⊥ ,
so ∑ P ∈ P Π P \sum_{P \in\mathcal{P}}\Pi_P ∑ P ∈ P Π P is positive definite on U U U , and its minimum quadratic form over the unit sphere of U U U is strictly positive.
Proposition C.6. Let a , b > 0 a,b > 0 a , b > 0 with a b ≥ 1 ab \ge1 ab ≥ 1 . Then x ≥ a log ( b x ) x \ge a\log(bx) x ≥ a log ( b x ) for every x ≥ 2 a log ( 2 a b ) x \ge2a\log(2ab) x ≥ 2 a log ( 2 ab ) .
Proof. The map f ( x ) : = x − a log ( b x ) f(x) := x - a\log(bx) f ( x ) := x − a log ( b x ) is increasing for x ≥ a x \ge a x ≥ a , and x 0 : = 2 a log ( 2 a b ) ≥ a x_0 := 2a\log(2ab) \ge a x 0 := 2 a log ( 2 ab ) ≥ a since a b ≥ 1 ab \ge1 ab ≥ 1 . Using log y ≤ y \log y \le y log y ≤ y , we get f ( x 0 ) = a ( log ( 2 a b ) − log log ( 2 a b ) ) ≥ 0 f(x_0) = a\left(\log(2ab) - \log\log(2ab)\right) \ge0 f ( x 0 ) = a ( log ( 2 ab ) − log log ( 2 ab ) ) ≥ 0 .
Lemma 3.2. Fix n ∈ N + n \in\mathbb{N}^+ n ∈ N + and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . With probability at least 1 − δ 1-\delta 1 − δ , it holds for all P ∈ P P \in\mathcal{P} P ∈ P and n δ ≤ k ≤ n n_\delta\le k \le n n δ ≤ k ≤ n that
SubOpt ( θ k , P ) ≤ g k ( P ) β k 2 ( δ ) . \operatorname{SubOpt}(\theta_k,P) \le g_k(P)\beta_k^2(\delta). SubOpt ( θ k , P ) ≤ g k ( P ) β k 2 ( δ ) .
Proof. We work on the event of Lemma B.12, which has probability at least 1 − δ 1-\delta 1 − δ . Fix P ∈ P P \in\mathcal{P} P ∈ P and a round n δ ≤ k ≤ n n_{\delta} \le k \le n n δ ≤ k ≤ n , and let u k : = θ k − θ E u_k := \theta_k-\theta^{\mathrm{E}} u k := θ k − θ E . Recall that
SubOpt ( θ k , P ) = V P ⋆ ( θ E ) − V P π P ( θ k ) ( θ E ) , g k ( P ) = ∥ H k † / 2 H P ( θ E ) H k † / 2 ∥ 2 . \operatorname{SubOpt}(\theta_k,P)=V_P^\star(\theta^{\mathrm{E}})-V_P^{\pi_P(\theta_k)}(\theta^{\mathrm{E}}), \qquad g_k(P)=\left\|\mathcal{H}_k^{\dagger/2}\mathcal{H}_P(\theta^{\mathrm{E}})\mathcal{H}_k^{\dagger/2}\right\|_2. SubOpt ( θ k , P ) = V P ⋆ ( θ E ) − V P π P ( θ k ) ( θ E ) , g k ( P ) = H k † /2 H P ( θ E ) H k † /2 2 .
Step 1 (image condition). By Step 1 of the proof of Lemma 3.1, λ k ≥ λ n δ > 0 \lambda_k \ge\lambda_{n_{\delta}}>0 λ k ≥ λ n δ > 0 . Since H k \mathcal{H}_k H k vanishes on U ⊥ U^\perp U ⊥ and H k ⪰ λ k Π U \mathcal{H}_k \succeq\lambda_k\Pi_U H k ⪰ λ k Π U , we have im H k = U \operatorname{im}\mathcal{H}_k=U im H k = U . As im H P ( θ E ) = U P ⊆ U \operatorname{im}\mathcal{H}_P(\theta^{\mathrm{E}})=U_P\subseteq U im H P ( θ E ) = U P ⊆ U by Proposition B.5 , it follows that im H P ( θ E ) ⊆ im H k \operatorname{im}\mathcal{H}_P(\theta^{\mathrm{E}})\subseteq\operatorname{im}\mathcal{H}_k im H P ( θ E ) ⊆ im H k .
Step 2 (change of geometry). By Step 1, Proposition C.2 applies with θ 1 = θ k \theta_1=\theta_k θ 1 = θ k , θ 2 = θ E \theta_2=\theta^{\mathrm{E}} θ 2 = θ E , and X = H k X=\mathcal{H}_k X = H k :
SubOpt ( θ k , P ) ≤ ψ ( R ∥ θ E − θ k ∥ Π P ) ∥ H k † / 2 H P ( θ E ) H k † / 2 ∥ 2 ∥ u k ∥ H k 2 = ψ ( R ∥ θ E − θ k ∥ Π P ) g k ( P ) ∥ u k ∥ H k 2 . \operatorname{SubOpt}(\theta_k,P)\leq\psi\left(R\left\|\theta^{\mathrm{E}}-\theta_k\right\|_{\Pi_P}\right)\left\|\mathcal{H}_k^{\dagger/2}\mathcal{H}_P(\theta^{\mathrm{E}})\mathcal{H}_k^{\dagger/2}\right\|_2\left\|u_k\right\|_{\mathcal{H}_k}^2=\psi\left(R\left\|\theta^{\mathrm{E}}-\theta_k\right\|_{\Pi_P}\right)g_k(P)\left\|u_k\right\|_{\mathcal{H}_k}^2. SubOpt ( θ k , P ) ≤ ψ ( R θ E − θ k Π P ) H k † /2 H P ( θ E ) H k † /2 2 ∥ u k ∥ H k 2 = ψ ( R θ E − θ k Π P ) g k ( P ) ∥ u k ∥ H k 2 .
Step 3 (localization). By Lemma C.3 , ψ ( R ∥ θ E − θ k ∥ Π P ) ≤ 1 \psi\left(R\left\|\theta^{\mathrm{E}}-\theta_k\right\|_{\Pi_P}\right)\leq1 ψ ( R θ E − θ k Π P ) ≤ 1 , hence
SubOpt ( θ k , P ) ≤ g k ( P ) ∥ u k ∥ H k 2 . \operatorname{SubOpt}(\theta_k,P)\leq g_k(P)\left\|u_k\right\|_{\mathcal{H}_k}^2. SubOpt ( θ k , P ) ≤ g k ( P ) ∥ u k ∥ H k 2 .
Step 4 (confidence bound). By Lemma 3.1, ∥ u k ∥ H k 2 ≤ β k 2 ( δ ) \left\|u_k\right\|_{\mathcal{H}_k}^2\leq\beta_k^2(\delta) ∥ u k ∥ H k 2 ≤ β k 2 ( δ ) , which gives
SubOpt ( θ k , P ) ≤ g k ( P ) β k 2 ( δ ) . \operatorname{SubOpt}(\theta_k,P)\leq g_k(P)\beta_k^2(\delta). SubOpt ( θ k , P ) ≤ g k ( P ) β k 2 ( δ ) .
All steps hold on the same event, simultaneously for every P ∈ P P\in\mathcal{P} P ∈ P and n δ ≤ k ≤ n n_{\delta}\leq k\leq n n δ ≤ k ≤ n , which concludes the proof. □
Lemma 3.3. Fix n ∈ N + n\in\mathbb{N}^{+} n ∈ N + and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Under round-robin selection (6), the warm-up length satisfies
n δ ≤ ∣ P ∣ ⌈ 2 R 2 β n 2 ( δ ) λ ∣ P ∣ ⌉ , n_{\delta}\leq|\mathcal{P}|\left\lceil\frac{2R^2\beta_n^2(\delta)}{\lambda_{|\mathcal{P}|}}\right\rceil, n δ ≤ ∣ P ∣ ⌈ λ ∣ P ∣ 2 R 2 β n 2 ( δ ) ⌉ , whenever the right-hand side is at most n n n .
Proof. By definition of β n ( δ ) \beta_n(\delta) β n ( δ ) in Lemma 3.1, the warm-up condition (4) reads
λ k ≥ 2 R 2 β n 2 ( δ ) . \lambda_k\geq2R^2\beta_n^2(\delta). λ k ≥ 2 R 2 β n 2 ( δ ) .
Fix a round k ≤ n k\leq n k ≤ n and let j : = ⌊ k / ∣ P ∣ ⌋ j:=\lfloor k/|\mathcal{P}|\rfloor j := ⌊ k /∣ P ∣ ⌋ be the number of completed round-robin cycles. Each P ∈ P P\in\mathcal{P} P ∈ P has been selected at least j j j times, so H k ⪰ j H ∣ P ∣ \mathcal{H}_k\succeq j\mathcal{H}_{|\mathcal{P}|} H k ⪰ j H ∣ P ∣ and hence λ k ≥ j λ ∣ P ∣ \lambda_k\geq j\lambda_{|\mathcal{P}|} λ k ≥ j λ ∣ P ∣ , where λ ∣ P ∣ > 0 \lambda_{|\mathcal{P}|}>0 λ ∣ P ∣ > 0 by Corollary C.5. The warm-up condition therefore holds at round k k k whenever
j ≥ 2 R 2 β n 2 ( δ ) λ ∣ P ∣ , j\geq\frac{2R^2\beta_n^2(\delta)}{\lambda_{|\mathcal{P}|}}, j ≥ λ ∣ P ∣ 2 R 2 β n 2 ( δ ) ,
which is the case for k = ∣ P ∣ ⌈ 2 R 2 β n 2 ( δ ) / λ ∣ P ∣ ⌉ k=|\mathcal{P}|\left\lceil2R^2\beta_n^2(\delta)/\lambda_{|\mathcal{P}|}\right\rceil k = ∣ P ∣ ⌈ 2 R 2 β n 2 ( δ ) / λ ∣ P ∣ ⌉ . If this round is at most n n n , the definition of n δ n_{\delta} n δ as the first round satisfying (4) gives the claim. □
Proofs of section 3.3
Having stated the required lemmas, we subsequently prove our main transfer results. We first present the general transfer guarantee.
Theorem 3.6. Fix n ∈ N + n\in\mathbb{N}^{+} n ∈ N + , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and follow the active rule (7) after round n δ n_{\delta} n δ . With probability at least 1 − δ 1-\delta 1 − δ , it holds for all n δ < k ≤ n n_{\delta}<k\leq n n δ < k ≤ n and P ∈ P P\in\mathcal{P} P ∈ P that
g k ( P ) ≲ ( 1 + m η d ⋆ H 2 ) d U log ( k m ) k − n δ . g_k(P)\lesssim\left(1+\frac{m}{\eta d_{\star}H^2}\right)\frac{d_U\log(km)}{k-n_{\delta}}. g k ( P ) ≲ ( 1 + η d ⋆ H 2 m ) k − n δ d U log ( k m ) . If moreover m ≤ η d ⋆ H 2 m\leq\eta d_{\star}H^2 m ≤ η d ⋆ H 2 and n ≥ 2 n δ n\geq2n_{\delta} n ≥ 2 n δ ,
SubOpt ( θ n , P ) ≲ d ⋆ d U log ( n m ) log ( η d ⋆ n m / δ ) n m . (31) \operatorname{SubOpt}(\theta_n,P)\lesssim\frac{d_{\star}d_U\log(nm)\log(\eta d_{\star}nm/\delta)}{nm}. \tag*{(31)} SubOpt ( θ n , P ) ≲ nm d ⋆ d U log ( nm ) log ( η d ⋆ nm / δ ) . (31)
Proof. Setup. We work on the event of Lemma 3.1, which holds with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for all rounds. Let Λ : = 32 R 2 d ⋆ / ( ( 1 − e − 1 ) 2 m ) \Lambda:= 32R^2d_{\star}/((1-e^{-1})^2m) Λ := 32 R 2 d ⋆ / (( 1 − e − 1 ) 2 m ) , which lower-bounds the right-hand side of (4) since 3 η d ⋆ n m / δ ≥ e 3\eta d_{\star}nm/\delta\ge e 3 η d ⋆ nm / δ ≥ e . By definition of n δ n_{\delta} n δ , every round k ≥ n δ k \ge n_{\delta} k ≥ n δ satisfies λ k ≥ Λ > 0 \lambda_k \ge\Lambda> 0 λ k ≥ Λ > 0 , so H k \mathcal{H}_k H k is invertible on U U U , and the confidence and localization bounds apply. The same holds for the plug-in matrix H ^ k : = ∑ i = 1 k H P i ( θ k ) \widehat{\mathcal{H}}_k := \sum_{i=1}^k \mathcal{H}_{P_i}(\theta_k) H k := ∑ i = 1 k H P i ( θ k ) , which has the same image U U U since Hessian images do not depend on the parameters (Proposition B.5 ). For k ≥ n δ k \ge n_{\delta} k ≥ n δ , we write
T k ( P ) : = H k † / 2 H P ( θ E ) H k † / 2 , T ^ k ( P ) : = H ^ k † / 2 H P ( θ k ) H ^ k † / 2 , \mathcal{T}_k(P) := \mathcal{H}_k^{\dagger/2}\mathcal{H}_P(\theta^{\mathrm{E}})\mathcal{H}_k^{\dagger/2}, \qquad\widehat{\mathcal{T}}_k(P) := \widehat{\mathcal{H}}_k^{\dagger/2}\mathcal{H}_P(\theta_k)\widehat{\mathcal{H}}_k^{\dagger/2}, T k ( P ) := H k † /2 H P ( θ E ) H k † /2 , T k ( P ) := H k † /2 H P ( θ k ) H k † /2 ,
so that g k ( P ) = ∥ T k ( P ) ∥ 2 g_k(P)=\lVert\mathcal{T}_k(P)\rVert_2 g k ( P ) = ∥ T k ( P ) ∥ 2 and g ^ k ( P ) = ∥ T ^ k ( P ) ∥ 2 \widehat{g}_k(P)=\lVert\widehat{\mathcal{T}}_k(P)\rVert_2 g k ( P ) = ∥ T k ( P ) ∥ 2 . All matrices below are restricted to U U U . Since ∥ H P ( θ E ) ∥ 2 ≤ η H 2 B ϕ 2 \lVert\mathcal{H}_P(\theta^{\mathrm{E}})\rVert_2 \le\eta H^2B_{\phi}^2 ∥ H P ( θ E ) ∥ 2 ≤ η H 2 B ϕ 2 (Proposition B.4 ) and H k ⪰ Λ I \mathcal{H}_k \succeq\Lambda I H k ⪰ Λ I on U U U ,
T k ( P ) ⪯ c m I , c m : = η H 2 B ϕ 2 Λ = ( 1 − e − 1 ) 2 m 32 ( H + 1 ) 2 η d ⋆ , (32) \mathcal{T}_k(P) \preceq c_m I, \qquad c_m := \frac{\eta H^2B_{\phi}^2}{\Lambda} = \frac{(1-e^{-1})^2m}{32(H+1)^2\eta d_{\star}}, \tag*{(32)} T k ( P ) ⪯ c m I , c m := Λ η H 2 B ϕ 2 = 32 ( H + 1 ) 2 η d ⋆ ( 1 − e − 1 ) 2 m , (32)
where we used R 2 = η 2 H 2 ( H + 1 ) 2 B ϕ 2 R^2=\eta^2H^2(H+1)^2B_{\phi}^2 R 2 = η 2 H 2 ( H + 1 ) 2 B ϕ 2 .
Step 1 (localization of g k g_k g k ). Fix k ≥ n δ k \ge n_{\delta} k ≥ n δ . By the proof of Lemma C.3 , R ∥ θ k − θ E ∥ P ≤ 1 R\lVert\theta_k-\theta^{\mathrm{E}}\rVert_{\mathcal{P}}\le1 R ∥ θ k − θ E ∥ P ≤ 1 for every P ∈ P P\in\mathcal{P} P ∈ P , so item 1 of Lemma B.7 gives e − 1 H P ( θ E ) ⪯ H P ( θ k ) ⪯ e H P ( θ E ) e^{-1}\mathcal{H}_P(\theta^{\mathrm{E}})\preceq\mathcal{H}_P(\theta_k)\preceq e\mathcal{H}_P(\theta^{\mathrm{E}}) e − 1 H P ( θ E ) ⪯ H P ( θ k ) ⪯ e H P ( θ E ) . Summing over the selected dynamics, e − 1 H k ⪯ H ^ k ⪯ e H k e^{-1}\mathcal{H}_k\preceq\widehat{\mathcal{H}}_k\preceq e\mathcal{H}_k e − 1 H k ⪯ H k ⪯ e H k .
For positive definite B B B and positive semidefinite A A A on U U U , the Courant–Fischer theorem gives that the i i i -th largest eigenvalue of B − 1 / 2 A B − 1 / 2 B^{-1/2}AB^{-1/2} B − 1/2 A B − 1/2 equals max dim V = i min v ∈ V ∖ { 0 } ∥ v ∥ A 2 / ∥ v ∥ B 2 \max_{\dim V=i}\min_{v\in V\setminus\{0\}}\lVert v\rVert_A^2/\lVert v\rVert_B^2 max d i m V = i min v ∈ V ∖ { 0 } ∥ v ∥ A 2 / ∥ v ∥ B 2 , and each such ratio increases with A A A and decreases with B B B in the Loewner order. Hence every eigenvalue of T ^ k ( P ) \widehat{\mathcal{T}}_k(P) T k ( P ) lies within a factor e 2 e^2 e 2 of the corresponding eigenvalue of T k ( P ) \mathcal{T}_k(P) T k ( P ) , and in particular
e − 2 g k ( P ) ≤ g ^ k ( P ) ≤ e 2 g k ( P ) for every P ∈ P . (33) e^{-2}g_k(P)\le\widehat{g}_k(P)\le e^2g_k(P)\qquad\text{for every }P\in\mathcal{P}. \tag*{(33)} e − 2 g k ( P ) ≤ g k ( P ) ≤ e 2 g k ( P ) for every P ∈ P . (33)
Step 2 (greedy rule) For k ≥ n δ k\ge n_{\delta} k ≥ n δ , the rule selects P k + 1 ∈ arg max P ∈ P g ^ k ( P ) P_{k+1}\in\arg\max_{P\in\mathcal{P}}\widehat{g}_k(P) P k + 1 ∈ arg max P ∈ P g k ( P ) . By (33) , for every P ∈ P P\in\mathcal{P} P ∈ P ,
g k ( P ) ≤ e 2 g ^ k ( P ) ≤ e 2 g ^ k ( P k + 1 ) ≤ e 4 g k ( P k + 1 ) . g_k(P)\le e^2\widehat{g}_k(P)\le e^2\widehat{g}_k(P_{k+1})\le e^4g_k(P_{k+1}). g k ( P ) ≤ e 2 g k ( P ) ≤ e 2 g k ( P k + 1 ) ≤ e 4 g k ( P k + 1 ) .
Since log ( 1 + x ) ≥ x / ( 1 + x ) \log(1+x)\ge x/(1+x) log ( 1 + x ) ≥ x / ( 1 + x ) , every x ∈ [ 0 , c m ] x\in[0,c_m] x ∈ [ 0 , c m ] satisfies x ≤ c ˉ m log ( 1 + x ) x\le\bar{c}_m\log(1+x) x ≤ c ˉ m log ( 1 + x ) with c ˉ m : = 1 + c m \bar{c}_m:=1+c_m c ˉ m := 1 + c m . Applying this to the largest eigenvalue of T k ( P k + 1 ) \mathcal{T}_k(P_{k+1}) T k ( P k + 1 ) , which lies in [ 0 , c m ] [0,c_m] [ 0 , c m ] by (32) , and using that the other eigenvalues contribute non-negatively, gives
g k ( P ) ≤ e 4 c ˉ m log det ( I + T k ( P k + 1 ) ) for every P ∈ P . (34) g_k(P)\le e^4\bar{c}_m\log\det\left(I+\mathcal{T}_k(P_{k+1})\right)\qquad\text{for every }P\in\mathcal{P}. \tag*{(34)} g k ( P ) ≤ e 4 c ˉ m log det ( I + T k ( P k + 1 ) ) for every P ∈ P . (34)
Step 3 (elliptical potential lemma). Fix P ∈ P P\in\mathcal{P} P ∈ P and n δ < k ≤ n n_{\delta}<k\le n n δ < k ≤ n . Since H k ⪰ H i \mathcal{H}_k\succeq\mathcal{H}_i H k ⪰ H i on U U U for i ≤ k i\le k i ≤ k , we have g k ( P ) ≤ g i ( P ) g_k(P)\le g_i(P) g k ( P ) ≤ g i ( P ) . Averaging over i ∈ { n δ , … , k − 1 } i\in\{n_{\delta},\ldots,k-1\} i ∈ { n δ , … , k − 1 } and applying (34) ,
g k ( P ) ≤ e 4 c ˉ m k − n δ ∑ i = n δ k − 1 log det ( I + T i ( P i + 1 ) ) . g_k(P)\le\frac{e^4\bar{c}_m}{k-n_{\delta}}\sum_{i=n_{\delta}}^{k-1}\log\det\left(I+\mathcal{T}_i(P_{i+1})\right). g k ( P ) ≤ k − n δ e 4 c ˉ m i = n δ ∑ k − 1 log det ( I + T i ( P i + 1 ) ) .
Since H i + 1 = H i + H P i + 1 ( θ E ) \mathcal{H}_{i+1}=\mathcal{H}_i+\mathcal{H}_{P_{i+1}}(\theta^{\mathrm{E}}) H i + 1 = H i + H P i + 1 ( θ E ) , we have det H i + 1 = det H i det ( I + T i ( P i + 1 ) ) \det\mathcal{H}_{i+1}=\det\mathcal{H}_i\det\left(I+\mathcal{T}_i(P_{i+1})\right) det H i + 1 = det H i det ( I + T i ( P i + 1 ) ) on U U U , so the sum telescopes to log ( det H k / det H n δ ) \log(\det\mathcal{H}_k/\det\mathcal{H}_{n_{\delta}}) log ( det H k / det H n δ ) . On U U U , every eigenvalue of H k \mathcal{H}_k H k is at most k η H 2 B ϕ 2 k\eta H^2B_{\phi}^2 k η H 2 B ϕ 2 (Proposition B.4 ) and every eigenvalue of H n δ \mathcal{H}_{n_{\delta}} H n δ is at least Λ \Lambda Λ , so every eigenvalue of H n δ − 1 / 2 H k H n δ − 1 / 2 \mathcal{H}_{n_{\delta}}^{-1/2}\mathcal{H}_k\mathcal{H}_{n_{\delta}}^{-1/2} H n δ − 1/2 H k H n δ − 1/2 is at most k c m kc_m k c m . Since U U U has dimension d U d_U d U , log ( det H k / det H n δ ) ≤ d U log ( 1 + k c m ) \log(\det\mathcal{H}_k/\det\mathcal{H}_{n_{\delta}})\le d_U\log(1+kc_m) log ( det H k / det H n δ ) ≤ d U log ( 1 + k c m ) . Taking the maximum over P ∈ P P\in\mathcal{P} P ∈ P ,
max P ∈ P g k ( P ) ≤ e 4 c ˉ m d U log ( 1 + k c m ) k − n δ . (35) \max_{P\in\mathcal{P}}g_k(P)\le e^4\bar{c}_m\frac{d_U\log(1+kc_m)}{k-n_{\delta}}. \tag*{(35)} P ∈ P max g k ( P ) ≤ e 4 c ˉ m k − n δ d U log ( 1 + k c m ) . (35)
Since η d ⋆ ≥ 1 \eta d_{\star}\ge1 η d ⋆ ≥ 1 , we have c m ≤ m c_m\le m c m ≤ m and c m ≲ m / ( η d ⋆ H 2 ) c_m\lesssim m/(\eta d_{\star}H^2) c m ≲ m / ( η d ⋆ H 2 ) , and log ( 1 + k c m ) ≤ log ( 1 + k m ) ≤ 2 log ( k m ) \log(1+kc_m)\le\log(1+km)\le2\log(km) log ( 1 + k c m ) ≤ log ( 1 + k m ) ≤ 2 log ( k m ) as k m ≥ 2 km\ge2 k m ≥ 2 . This proves the first claim.
Step 4 (suboptimality bound). By Lemma 3.2 and (35) at k = n k=n k = n , and since β n 2 ( δ ) ≲ d ⋆ log ( η d ⋆ n m / δ ) / m \beta_n^2(\delta)\lesssim d_{\star}\log(\eta d_{\star}nm/\delta)/m β n 2 ( δ ) ≲ d ⋆ log ( η d ⋆ nm / δ ) / m and c m d ⋆ / m ≲ 1 / ( η ( H + 1 ) 2 ) c_m d_{\star}/m\lesssim1/(\eta(H+1)^2) c m d ⋆ / m ≲ 1/ ( η ( H + 1 ) 2 ) , it holds for every P ∈ P P\in\mathcal{P} P ∈ P that
SubOpt ( θ n , P ) ≤ g n ( P ) β n 2 ( δ ) ≲ ( d ⋆ m + 1 η ( H + 1 ) 2 ) d U log ( n m ) log ( η d ⋆ n m / δ ) n − n δ . \operatorname{SubOpt}(\theta_n,P)\le g_n(P)\beta_n^2(\delta)\lesssim\left(\frac{d_{\star}}{m}+\frac{1}{\eta(H+1)^2}\right)\frac{d_U\log(nm)\log(\eta d_{\star}nm/\delta)}{n-n_{\delta}}. SubOpt ( θ n , P ) ≤ g n ( P ) β n 2 ( δ ) ≲ ( m d ⋆ + η ( H + 1 ) 2 1 ) n − n δ d U log ( nm ) log ( η d ⋆ nm / δ ) .
If m ≤ η d ⋆ H 2 m\le\eta d_{\star}H^2 m ≤ η d ⋆ H 2 , then 1 η ( H + 1 ) 2 ≤ 1 η H 2 ≤ d ⋆ m \frac{1}{\eta(H+1)^2}\le\frac{1}{\eta H^2}\le\frac{d_{\star}}{m} η ( H + 1 ) 2 1 ≤ η H 2 1 ≤ m d ⋆ , and if n ≥ 2 n δ n\ge2n_{\delta} n ≥ 2 n δ , then n − n δ ≥ n / 2 n-n_{\delta}\ge n/2 n − n δ ≥ n /2 . Together, these give (8), on the same event.
We now provide the proof of Theorem 3.7 which allows us to transfer the recovered reward to arbitrary unseen dynamics. We introduce a couple of quantities required for our analysis. We measure how P P P relates to the training set P \mathcal{P} P through the following quantities:
g n ( P ) : = max P ′ ∈ P g n ( P ′ ) , s P : = ∥ Π P Π U ⊥ ∥ 2 , γ ( P ) : = max w ∈ U ∖ { 0 } ∥ w ∥ H P ( θ E ) 2 max Q ∈ P ∥ w ∥ H Q ( θ E ) 2 . g_n(P):=\max_{P'\in\mathcal{P}}g_n(P'),\qquad s_P:=\left\|\Pi_P\Pi_{U^\perp}\right\|_2,\qquad\gamma(P):=\max_{w\in U\setminus\{0\}}\frac{\|w\|_{\mathcal{H}_P(\theta^{\mathrm{E}})}^2}{\max_{Q\in\mathcal{P}}\|w\|_{\mathcal{H}_Q(\theta^{\mathrm{E}})}^2}. g n ( P ) := P ′ ∈ P max g n ( P ′ ) , s P := ∥ Π P Π U ⊥ ∥ 2 , γ ( P ) := w ∈ U ∖ { 0 } max max Q ∈ P ∥ w ∥ H Q ( θ E ) 2 ∥ w ∥ H P ( θ E ) 2 .
Here, s P ∈ [ 0 , 1 ] s_P\in[0,1] s P ∈ [ 0 , 1 ] measures how far the identifiable subspace of P P P leaves U U U , and γ ( P ) \gamma(P) γ ( P ) compares the curvature of P P P along U U U with that of the training dynamics. The ratio defining γ ( P ) \gamma(P) γ ( P ) is well defined. If w ∈ U w\in U w ∈ U satisfies ∥ w ∥ H Q ( θ E ) = 0 \|w\|_{\mathcal{H}_Q(\theta^{\mathrm{E}})}=0 ∥ w ∥ H Q ( θ E ) = 0 for every Q ∈ P Q\in\mathcal{P} Q ∈ P , then w ∈ ⋂ Q ∈ P U Q = U ⊥ w\in\bigcap_{Q\in\mathcal{P}}U_Q=U^\perp w ∈ ⋂ Q ∈ P U Q = U ⊥ by Proposition B.5, so w = 0 w=0 w = 0 . By homogeneity, the maximum may be taken over the unit sphere of U U U , hence it is attained and finite. Moreover, γ ( P ) ≤ 1 \gamma(P)\leq1 γ ( P ) ≤ 1 for every P ∈ P P\in\mathcal{P} P ∈ P , since P P P itself appears in the inner maximum.
Note that s P = 0 s_P=0 s P = 0 and g ⊥ ( P ) = 0 g_\perp(P)=0 g ⊥ ( P ) = 0 are both equivalent to U P ⊆ U U_P\subseteq U U P ⊆ U .
Theorem 3.7. Fix δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and follow the active rule (7) after round n δ n_\delta n δ . If m ≤ n δ H 2 m\leq n_\delta H^2 m ≤ n δ H 2 and n ≥ 2 n δ n\geq2n_\delta n ≥ 2 n δ , then with probability at least 1 − δ 1-\delta 1 − δ , for any P P P ,
SubOpt ( θ n , P ) = O ~ ( d ⋆ d U n m ) + O ( g ⊥ ( P ) B θ 2 ) . \operatorname{SubOpt}(\theta_n,P)=\widetilde{O}\left(\frac{d_\star d_U}{nm}\right)+O\left(g_\perp(P)B_\theta^2\right). SubOpt ( θ n , P ) = O ( nm d ⋆ d U ) + O ( g ⊥ ( P ) B θ 2 ) .
Proof. We work on the confidence event of Lemma 3.1. Since n ≥ 2 n δ ≥ n δ n\geq2n_\delta\geq n_\delta n ≥ 2 n δ ≥ n δ , its bounds and those of Lemma C.3 hold at round n n n . Let
Δ : = θ n − θ E = u + v , u : = Π U Δ , v : = Π U ⊥ Δ . \Delta:=\theta_n-\theta^{\mathrm{E}}=u+v,\qquad u:=\Pi_U\Delta,\qquad v:=\Pi_{U^\perp}\Delta. Δ := θ n − θ E = u + v , u := Π U Δ , v := Π U ⊥ Δ.
Since H n \mathcal{H}_n H n vanishes on U ⊥ U^\perp U ⊥ and H n ⪰ λ n Π U \mathcal{H}_n\succeq\lambda_n\Pi_U H n ⪰ λ n Π U , the confidence bound, the localization of Lemma C.3, and boundedness of Θ \Theta Θ give
∥ u ∥ H n ≤ β n ( δ ) , R ∥ u ∥ 2 ≤ 1 , ∥ v ∥ 2 ≤ 2 B θ . \|u\|_{\mathcal{H}_n}\leq\beta_n(\delta),\qquad R\|u\|_2\leq1,\qquad\|v\|_2\leq2B_\theta. ∥ u ∥ H n ≤ β n ( δ ) , R ∥ u ∥ 2 ≤ 1 , ∥ v ∥ 2 ≤ 2 B θ .
Since im H Q ( θ E ) ⊆ U = im H n \operatorname{im}\mathcal{H}_Q(\theta^{\mathrm{E}}) \subseteq U = \operatorname{im}\mathcal{H}_n im H Q ( θ E ) ⊆ U = im H n for every Q ∈ P Q \in\mathcal{P} Q ∈ P , the proof of Lemma 3.2 gives ∥ u ∥ H Q ( θ E ) ≤ g n ( Q ) ∥ u ∥ H n \lVert u\rVert_{\mathcal{H}_Q(\theta^{\mathrm{E}})} \le g_n(Q)\lVert u\rVert_{\mathcal{H}_n} ∥ u ∥ H Q ( θ E ) ≤ g n ( Q ) ∥ u ∥ H n . Thus, by definition of γ ( P ) \gamma(\mathcal{P}) γ ( P ) and since u ∈ U u \in U u ∈ U , it follows that
∥ u ∥ H P ( θ E ) 2 ≤ γ ( P ) max Q ∈ P ∥ u ∥ H Q ( θ E ) 2 ≤ γ ( P ) g n ( P ) ∥ u ∥ H n 2 ≤ γ ( P ) g n ( P ) β n 2 ( δ ) . \lVert u\rVert_{\mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})}^2 \le\gamma(\mathcal{P}) \max_{Q \in\mathcal{P}} \lVert u\rVert_{\mathcal{H}_Q(\theta^{\mathrm{E}})}^2
\le\gamma(\mathcal{P})g_n(\mathcal{P})\lVert u\rVert_{\mathcal{H}_n}^2
\le\gamma(\mathcal{P})g_n(\mathcal{P})\beta_n^2(\delta). ∥ u ∥ H P ( θ E ) 2 ≤ γ ( P ) Q ∈ P max ∥ u ∥ H Q ( θ E ) 2 ≤ γ ( P ) g n ( P ) ∥ u ∥ H n 2 ≤ γ ( P ) g n ( P ) β n 2 ( δ ) .
Moreover, v = Π U ⊥ v v = \Pi_{U^\perp}v v = Π U ⊥ v implies
∥ v ∥ H P ( θ E ) 2 = v ⊤ Π U ⊥ H P ( θ E ) Π U ⊥ v ≤ g ⊥ ( P ) ∥ v ∥ 2 2 ≤ 4 g ⊥ ( P ) B θ 2 . \lVert v\rVert_{\mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})}^2 = v^\top\Pi_{U^\perp}\mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})\Pi_{U^\perp}v
\le g_\perp(\mathcal{P})\lVert v\rVert_2^2
\le4g_\perp(\mathcal{P})B_\theta^2. ∥ v ∥ H P ( θ E ) 2 = v ⊤ Π U ⊥ H P ( θ E ) Π U ⊥ v ≤ g ⊥ ( P ) ∥ v ∥ 2 2 ≤ 4 g ⊥ ( P ) B θ 2 .
Consequently,
∥ Δ ∥ H P ( θ E ) 2 ≤ 2 ∥ u ∥ H P ( θ E ) 2 + 2 ∥ v ∥ H P ( θ E ) 2 ≤ 2 γ ( P ) g n ( P ) β n 2 ( δ ) + 8 g ⊥ ( P ) B θ 2 . \lVert\Delta\rVert_{\mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})}^2 \le2\lVert u\rVert_{\mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})}^2 + 2\lVert v\rVert_{\mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})}^2
\le2\gamma(\mathcal{P})g_n(\mathcal{P})\beta_n^2(\delta) + 8g_\perp(\mathcal{P})B_\theta^2. ∥ Δ ∥ H P ( θ E ) 2 ≤ 2 ∥ u ∥ H P ( θ E ) 2 + 2 ∥ v ∥ H P ( θ E ) 2 ≤ 2 γ ( P ) g n ( P ) β n 2 ( δ ) + 8 g ⊥ ( P ) B θ 2 .
Furthermore, v = Π U ⊥ v v = \Pi_{U^\perp}v v = Π U ⊥ v and the definition of s P s_{\mathcal{P}} s P yield
R ∥ Π P Δ ∥ 2 ≤ R ∥ u ∥ 2 + R ∥ Π P Π U ⊥ v ∥ 2 ≤ R ∥ u ∥ 2 + R s P ∥ v ∥ 2 ≤ 1 + 2 R B θ s P . R\lVert\Pi_{\mathcal{P}}\Delta\rVert_2 \le R\lVert u\rVert_2 + R\lVert\Pi_{\mathcal{P}}\Pi_{U^\perp}v\rVert_2
\le R\lVert u\rVert_2 + Rs_{\mathcal{P}}\lVert v\rVert_2
\le1 + 2RB_\theta s_{\mathcal{P}}. R ∥ Π P Δ ∥ 2 ≤ R ∥ u ∥ 2 + R ∥ Π P Π U ⊥ v ∥ 2 ≤ R ∥ u ∥ 2 + R s P ∥ v ∥ 2 ≤ 1 + 2 R B θ s P .
Applying Proposition C.1 and item 2 of Lemma B.7 with θ 1 = θ n \theta_1 = \theta_n θ 1 = θ n and θ 2 = θ E \theta_2 = \theta^{\mathrm{E}} θ 2 = θ E , the monotonicity of ψ \psi ψ , and ψ ( 1 + a ) ≤ e ψ ( a ) \psi(1+a) \le e\psi(a) ψ ( 1 + a ) ≤ e ψ ( a ) for a ≥ 0 a \ge0 a ≥ 0 , which follows from ψ ( x ) = ∫ 0 1 t x d t \psi(x) = \int_0^1 t^x \, \mathrm{d}t ψ ( x ) = ∫ 0 1 t x d t , yields
SubOpt ( θ n , P ) ≤ ψ ( R ∥ Π P Δ ∥ 2 ) ∥ Δ ∥ H P ( θ E ) 2 ≤ ψ ( 1 + 2 R B θ s P ) ∥ Δ ∥ H P ( θ E ) 2 ≤ 2 e ψ ( 2 R B θ s P ) [ γ ( P ) g n ( P ) β n 2 ( δ ) + 4 g ⊥ ( P ) B θ 2 ] . (36) \operatorname{SubOpt}(\theta_n,\mathcal{P}) \le\psi(R\lVert\Pi_{\mathcal{P}}\Delta\rVert_2)\lVert\Delta\rVert_{\mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})}^2
\le\psi(1+2RB_\theta s_{\mathcal{P}})\lVert\Delta\rVert_{\mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})}^2
\le2e\psi(2RB_\theta s_{\mathcal{P}})\left[\gamma(\mathcal{P})g_n(\mathcal{P})\beta_n^2(\delta) + 4g_\perp(\mathcal{P})B_\theta^2\right]. \tag*{(36)} SubOpt ( θ n , P ) ≤ ψ ( R ∥ Π P Δ ∥ 2 ) ∥ Δ ∥ H P ( θ E ) 2 ≤ ψ ( 1 + 2 R B θ s P ) ∥ Δ ∥ H P ( θ E ) 2 ≤ 2 e ψ ( 2 R B θ s P ) [ γ ( P ) g n ( P ) β n 2 ( δ ) + 4 g ⊥ ( P ) B θ 2 ] . (36)
Since m ≤ η d ⋆ H 2 m \le\eta d_\star H^2 m ≤ η d ⋆ H 2 and n ≥ 2 n δ n \ge2n_\delta n ≥ 2 n δ , Theorem 3.6 and the definition of β n ( δ ) \beta_n(\delta) β n ( δ ) give
g n ( P ) β n 2 ( δ ) ≲ d ⋆ d U log ( n m ) log ( η d ⋆ n m / δ ) n m , g_n(\mathcal{P})\beta_n^2(\delta) \lesssim\frac{d_\star d_U\log(nm)\log(\eta d_\star nm/\delta)}{nm}, g n ( P ) β n 2 ( δ ) ≲ nm d ⋆ d U log ( nm ) log ( η d ⋆ nm / δ ) ,
which, plugged into (34), yields the claim. The confidence event does not depend on P \mathcal{P} P , so the bound holds simultaneously for all P \mathcal{P} P .
Finally, U P ⊆ U U_{\mathcal{P}} \subseteq U U P ⊆ U implies s P = g ⊥ ( P ) = 0 s_{\mathcal{P}} = g_\perp(\mathcal{P}) = 0 s P = g ⊥ ( P ) = 0 , so the transfer bias vanishes, and ψ ( 0 ) = 1 / 2 \psi(0) = 1/2 ψ ( 0 ) = 1/2 . For P ∈ P \mathcal{P} \in\mathcal{P} P ∈ P , the definition of γ ( P ) \gamma(\mathcal{P}) γ ( P ) directly gives γ ( P ) ≤ 1 \gamma(\mathcal{P}) \le1 γ ( P ) ≤ 1 , and we recover Theorem 3.6.
Remark D.2 (Connection to principal angles). Suppose dim U P ≤ dim U \dim U_{\mathcal{P}} \le\dim U dim U P ≤ dim U and denote α P : = α max ( U P , U ) \alpha_{\mathcal{P}} := \alpha_{\max}(U_{\mathcal{P}}, U) α P := α m a x ( U P , U ) for the maximal principal angle between U P U_{\mathcal{P}} U P and U U U . Then
s P = ∥ Π P Π U ⊥ ∥ 2 = sin α P , s_{\mathcal{P}} = \lVert\Pi_{\mathcal{P}}\Pi_{U^\perp}\rVert_2 = \sin\alpha_{\mathcal{P}}, s P = ∥ Π P Π U ⊥ ∥ 2 = sin α P ,
so the constant in (34) reads ψ ( 2 R B θ sin α P ) \psi(2RB_\theta\sin\alpha_{\mathcal{P}}) ψ ( 2 R B θ sin α P ) . In particular, since ψ ( 1 ) = 1 \psi(1) = 1 ψ ( 1 ) = 1 , it is at most one whenever 2 R B θ sin α P ≤ 1 2RB_\theta\sin\alpha_{\mathcal{P}} \le1 2 R B θ sin α P ≤ 1 . If instead dim U P > dim U \dim U_{\mathcal{P}} > \dim U dim U P > dim U , then U P ∩ U ⊥ ≠ { 0 } U_{\mathcal{P}} \cap U^\perp\ne\{0\} U P ∩ U ⊥ = { 0 } and s P = 1 s_{\mathcal{P}} = 1 s P = 1 .
Moreover, since H P ( θ E ) 1 / 2 = H P ( θ E ) 1 / 2 Π P \mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})^{1/2} = \mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})^{1/2}\Pi_{\mathcal{P}} H P ( θ E ) 1/2 = H P ( θ E ) 1/2 Π P ,
g ⊥ ( P ) = ∥ H P ( θ E ) 1 / 2 Π P Π U ⊥ ∥ 2 2 ≤ ∥ H P ( θ E ) ∥ 2 sin 2 α P . g_\perp(\mathcal{P}) = \left\lVert\mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})^{1/2}\Pi_{\mathcal{P}}\Pi_{U^\perp}\right\rVert_2^2
\le\left\lVert\mathcal{H}_{\mathcal{P}}(\theta^{\mathrm{E}})\right\rVert_2 \sin^2\alpha_{\mathcal{P}}. g ⊥ ( P ) = H P ( θ E ) 1/2 Π P Π U ⊥ 2 2 ≤ H P ( θ E ) 2 sin 2 α P .
Both the constant and the transfer bias are thus controlled by α P \alpha_P α P . For a single training dynamics P 0 P^0 P 0 with dim U P 0 = dim U P \dim U_{P^0}=\dim U_P dim U P 0 = dim U P , we have
α P = α max ( U P 0 ⊥ , U P ⊥ ) . \alpha_P=\alpha_{\max}\left(U_{P^0}^{\perp},U_P^{\perp}\right). α P = α m a x ( U P 0 ⊥ , U P ⊥ ) .
This is the angle between unidentifiable subspaces appearing in the transferability guarantee of Schlaginhaufen and Kamgarpour [6 Theorem 3.11] .
Offline Active Learning
In some cases, sequential interaction with the expert may be impractical, for instance due to limited expert availability or high environment setup costs. We therefore also consider an offline variant, where the entire sequence of dynamics is selected in advance. Setting θ 0 : = 0 \theta_0:=0 θ 0 := 0 and ε : = η H 2 B ϕ 2 \varepsilon:=\eta H^2B_\phi^2 ε := η H 2 B ϕ 2 , we construct the offline information proxy and offline transfer operator
H k 0 : = ε Π U + ∑ i = 1 k H P i ( θ 0 ) , T k 0 ( P ) : = ( H k 0 ) † / 2 H P ( θ 0 ) ( H k 0 ) † / 2 , \mathcal{H}_k^0:=\varepsilon\Pi_U+\sum_{i=1}^{k}\mathcal{H}_{P_i}(\theta_0),\qquad\mathcal{T}_k^0(P):=(\mathcal{H}_k^0)^{\dagger/2}\mathcal{H}_P(\theta_0)(\mathcal{H}_k^0)^{\dagger/2}, H k 0 := ε Π U + i = 1 ∑ k H P i ( θ 0 ) , T k 0 ( P ) := ( H k 0 ) † /2 H P ( θ 0 ) ( H k 0 ) † /2 ,
and select dynamics with
P k + 1 ∈ arg max P ∈ P J ( T k 0 ( P ) ) , with J ∈ { ∥ ⋅ ∥ 2 , tr , log det ( I + ⋅ ) } . (37) P_{k+1}\in\mathop{\arg\max}_{P\in\mathcal{P}}J\left(\mathcal{T}_k^0(P)\right),\qquad\text{with }J\in\{\|\cdot\|_2,\operatorname{tr},\log\det(I+\cdot)\}. \tag*{(37)} P k + 1 ∈ arg max P ∈ P J ( T k 0 ( P ) ) , with J ∈ { ∥ ⋅ ∥ 2 , tr , log det ( I + ⋅ )} . (37)
Since U U U and H P ( θ 0 ) \mathcal{H}_P(\theta_0) H P ( θ 0 ) do not depend on expert demonstrations, this rule constructs the full sequence ( P k ) k = 1 n (P_k)_{k=1}^{n} ( P k ) k = 1 n without any interaction with the expert.
Theorem E.1. Let Assumption 2.1 hold, assume finite S \mathcal{S} S and A \mathcal{A} A and U ≠ { 0 } U\ne\{0\} U = { 0 } , and follow (35) at every round. Let κ \kappa κ be defined in Lemma C.4, and ξ : = min v ∈ U , ∥ v ∥ 2 = 1 max P ∈ P ∥ v ∥ P 2 > 0 \xi:=\min_{v\in U,\|v\|_2=1}\max_{P\in\mathcal{P}}\|v\|_P^2>0 ξ := min v ∈ U , ∥ v ∥ 2 = 1 max P ∈ P ∥ v ∥ P 2 > 0 . Then, for every n ≥ n δ n\ge n_\delta n ≥ n δ and P ∈ P P\in\mathcal{P} P ∈ P ,
g n ( P ) ≤ 2 e 2 R B θ ( 1 + m η d ⋆ H 2 ) d U log ( 1 + n ) n , g_n(P)\le2e^{2RB_\theta}\left(1+\frac{m}{\eta d_\star H^2}\right)\frac{d_U\log(1+n)}{n}, g n ( P ) ≤ 2 e 2 R B θ ( 1 + η d ⋆ H 2 m ) n d U log ( 1 + n ) , and n δ ≤ ⌈ 2 K log ( 4 K ) ⌉ n_\delta\le\lceil2K\log(4K)\rceil n δ ≤ ⌈ 2 K log ( 4 K )⌉ whenever the right-hand side is at most n n n , where
K : = 2 d U e 2 R B θ ( 2 R 2 β n 2 ( δ ) + η H 2 B ϕ 2 ) κ ξ . K:=\frac{2d_Ue^{2RB_\theta}\left(2R^2\beta_n^2(\delta)+\eta H^2B_\phi^2\right)}{\kappa\xi}. K := κ ξ 2 d U e 2 R B θ ( 2 R 2 β n 2 ( δ ) + η H 2 B ϕ 2 ) .
Proof. All matrices below are positive semidefinite and vanish on U ⊥ U^\perp U ⊥ , so we compare them in the Loewner order on U U U , where H k 0 ⪰ ε Π U \mathcal{H}_k^0\succeq\varepsilon\Pi_U H k 0 ⪰ ε Π U is invertible and ( H k 0 ) † (\mathcal{H}_k^0)^\dagger ( H k 0 ) † is its inverse. Determinants det U \det_U det U are taken on U U U , of dimension d U d_U d U . Let t k : = max P ∈ P ∥ T k 0 ( P ) ∥ 2 t_k:=\max_{P\in\mathcal{P}}\|\mathcal{T}_k^0(P)\|_2 t k := max P ∈ P ∥ T k 0 ( P ) ∥ 2 and a : = R B θ a:=RB_\theta a := R B θ .
Step 1 (elliptical potential lemma). Since H k 0 ⪰ ε Π U \mathcal{H}_k^0\succeq\varepsilon\Pi_U H k 0 ⪰ ε Π U and H P ( θ 0 ) ⪯ η H 2 B ϕ 2 Π U \mathcal{H}_P(\theta_0)\preceq\eta H^2B_\phi^2\Pi_U H P ( θ 0 ) ⪯ η H 2 B ϕ 2 Π U (Proposition B.4), every eigenvalue of T k 0 ( P ) \mathcal{T}_k^0(P) T k 0 ( P ) lies in [ 0 , 1 ] [0,1] [ 0 , 1 ] . As x ≤ 2 log ( 1 + x ) x\le2\log(1+x) x ≤ 2 log ( 1 + x ) on [ 0 , 1 ] [0,1] [ 0 , 1 ] , for every P ∈ P P\in\mathcal{P} P ∈ P and each criterion J J J ,
∥ T k 0 ( P ) ∥ 2 ≤ 2 log det ( I + T k 0 ( P k + 1 ) ) . \|\mathcal{T}_k^0(P)\|_2\le2\log\det\left(I+\mathcal{T}_k^0(P_{k+1})\right). ∥ T k 0 ( P ) ∥ 2 ≤ 2 log det ( I + T k 0 ( P k + 1 ) ) .
For J = log det ( I + ⋅ ) J=\log\det(I+\cdot) J = log det ( I + ⋅ ) , this follows from ∥ T k 0 ( P ) ∥ 2 ≤ 2 log det ( I + T k 0 ( P ) ) \|\mathcal{T}_k^0(P)\|_2\le2\log\det(I+\mathcal{T}_k^0(P)) ∥ T k 0 ( P ) ∥ 2 ≤ 2 log det ( I + T k 0 ( P )) and the choice of P k + 1 P_{k+1} P k + 1 . For J = ∥ ⋅ ∥ 2 J=\|\cdot\|_2 J = ∥ ⋅ ∥ 2 and J = tr J=\operatorname{tr} J = tr , it follows from ∥ T k 0 ( P ) ∥ 2 ≤ J ( T k 0 ( P ) ) ≤ J ( T k 0 ( P k + 1 ) ) \|\mathcal{T}_k^0(P)\|_2\le J(\mathcal{T}_k^0(P))\le J(\mathcal{T}_k^0(P_{k+1})) ∥ T k 0 ( P ) ∥ 2 ≤ J ( T k 0 ( P )) ≤ J ( T k 0 ( P k + 1 )) and the scalar bound applied to each eigenvalue of T k 0 ( P k + 1 ) \mathcal{T}_k^0(P_{k+1}) T k 0 ( P k + 1 ) . Since H k 0 \mathcal{H}_k^0 H k 0 is nondecreasing in k k k , ∥ T n 0 ( P ) ∥ 2 ≤ ∥ T k 0 ( P ) ∥ 2 \|\mathcal{T}_n^0(P)\|_2\le\|\mathcal{T}_k^0(P)\|_2 ∥ T n 0 ( P ) ∥ 2 ≤ ∥ T k 0 ( P ) ∥ 2 for k ≤ n k\le n k ≤ n . Averaging over k ∈ { 0 , … , n − 1 } k\in\{0,\ldots,n-1\} k ∈ { 0 , … , n − 1 } and using det U H k + 1 0 = det U H k 0 det ( I + T k 0 ( P k + 1 ) ) \det_U\mathcal{H}_{k+1}^0=\det_U\mathcal{H}_k^0\det(I+\mathcal{T}_k^0(P_{k+1})) det U H k + 1 0 = det U H k 0 det ( I + T k 0 ( P k + 1 )) ,
t n ≤ 2 n ∑ k = 0 n − 1 log det ( I + T k 0 ( P k + 1 ) ) = 2 n log det U H n 0 ε d U ≤ 2 d U log ( 1 + n ) n , t_n\le\frac{2}{n}\sum_{k=0}^{n-1}\log\det\left(I+\mathcal{T}_k^0(P_{k+1})\right)=\frac{2}{n}\log\frac{\det_U\mathcal{H}_n^0}{\varepsilon^{d_U}}\le\frac{2d_U\log(1+n)}{n}, t n ≤ n 2 k = 0 ∑ n − 1 log det ( I + T k 0 ( P k + 1 ) ) = n 2 log ε d U det U H n 0 ≤ n 2 d U log ( 1 + n ) ,
where the last step uses that every eigenvalue of H n 0 \mathcal{H}_n^0 H n 0 on U U U is at most ε + n η H 2 B ϕ 2 = ( 1 + n ) ε \varepsilon+n\eta H^2B_\phi^2=(1+n)\varepsilon ε + n η H 2 B ϕ 2 = ( 1 + n ) ε .
Step 2 (from θ 0 \theta_0 θ 0 to θ E \theta^E θ E ). Since ∥ θ E ∥ P ≤ ∥ θ E ∥ 2 ≤ B θ \|\theta^E\|_{\mathcal{P}}\le\|\theta^E\|_2\le B_\theta ∥ θ E ∥ P ≤ ∥ θ E ∥ 2 ≤ B θ for every P ∈ P P\in\mathcal{P} P ∈ P , item 1 of Lemma B.7 with θ 1 = θ E \theta_1=\theta^E θ 1 = θ E and θ 2 = θ 0 \theta_2=\theta_0 θ 2 = θ 0 gives e − a H P ( θ 0 ) ⪯ H P ( θ E ) ⪯ e a H P ( θ 0 ) e^{-a}\mathcal{H}_P(\theta_0)\preceq\mathcal{H}_P(\theta^E)\preceq e^a\mathcal{H}_P(\theta_0) e − a H P ( θ 0 ) ⪯ H P ( θ E ) ⪯ e a H P ( θ 0 ) . Summing over the selected dynamics,
H k 0 ⪯ ε Π U + e a H k . (38) \mathcal{H}_k^0\preceq\varepsilon\Pi_U+e^a\mathcal{H}_k. \tag*{(38)} H k 0 ⪯ ε Π U + e a H k . (38)
Step 3 (transfer bound). Let n ≥ n δ n \ge n_{\delta} n ≥ n δ , so that λ n ≥ Λ \lambda_n \ge\Lambda λ n ≥ Λ , with Λ \Lambda Λ as in the proof of Theorem 3.6, and ε Π U ⪯ ( ε / Λ ) H n = c m H n \varepsilon\Pi_U \preceq(\varepsilon/\Lambda)\mathcal{H}_n = c_m\mathcal{H}_n ε Π U ⪯ ( ε /Λ ) H n = c m H n . By definition of t n t_n t n , Step 2 and (36),
H P ( θ E ) ⪯ e a H P ( θ 0 ) ⪯ e a t n H n 0 ⪯ e a t n ( c m + e a ) H n . \mathcal{H}_P(\theta^{\mathrm{E}}) \preceq e^a\mathcal{H}_P(\theta_0) \preceq e^a t_n\mathcal{H}_n^0 \preceq e^a t_n(c_m+e^a)\mathcal{H}_n. H P ( θ E ) ⪯ e a H P ( θ 0 ) ⪯ e a t n H n 0 ⪯ e a t n ( c m + e a ) H n .
hence g n ( P ) = ∥ H n † / 2 H P ( θ E ) H n † / 2 ∥ 2 ≤ e a ( e a + c m ) t n ≤ e 2 a ( 1 + c m ) t n g_n(P)=\left\|\mathcal{H}_n^{\dagger/2}\mathcal{H}_P(\theta^{\mathrm{E}})\mathcal{H}_n^{\dagger/2}\right\|_2 \le e^a(e^a+c_m)t_n \le e^{2a}(1+c_m)t_n g n ( P ) = H n † /2 H P ( θ E ) H n † /2 2 ≤ e a ( e a + c m ) t n ≤ e 2 a ( 1 + c m ) t n . By (30), c m ≤ m / ( η d ⋆ H 2 ) c_m \le m/(\eta d_{\star}H^2) c m ≤ m / ( η d ⋆ H 2 ) , and Step 1 gives the first claim.
Step 4 (offline warm-up). Let v ∈ U v \in U v ∈ U be a unit vector and P ⋆ ∈ arg max P ∈ P ∥ Π P v ∥ 2 2 P^{\star} \in\operatorname*{arg\,max}_{P\in\mathcal{P}}\|\Pi_Pv\|_2^2 P ⋆ ∈ arg max P ∈ P ∥ Π P v ∥ 2 2 , so that ∥ Π P ⋆ v ∥ 2 2 ≥ ξ \|\Pi_{P^{\star}}v\|_2^2 \ge\xi ∥ Π P ⋆ v ∥ 2 2 ≥ ξ . By Lemma C.4 and Step 2, H P ⋆ ( θ 0 ) ⪰ e − a H P ⋆ ( θ E ) ⪰ e − a κ Π P ⋆ \mathcal{H}_{P^{\star}}(\theta_0) \succeq e^{-a}\mathcal{H}_{P^{\star}}(\theta^{\mathrm{E}}) \succeq e^{-a}\kappa\Pi_{P^{\star}} H P ⋆ ( θ 0 ) ⪰ e − a H P ⋆ ( θ E ) ⪰ e − a κ Π P ⋆ , and by definition of t k t_k t k , H P ⋆ ( θ 0 ) ⪯ t k H k 0 \mathcal{H}_{P^{\star}}(\theta_0) \preceq t_k\mathcal{H}_k^0 H P ⋆ ( θ 0 ) ⪯ t k H k 0 . Hence v T H k 0 v ≥ e − a κ ξ / t k v^{\mathsf{T}}\mathcal{H}_k^0v \ge e^{-a}\kappa\xi/t_k v T H k 0 v ≥ e − a κ ξ / t k , and by (36),
λ k ≥ e − a ( e − a κ ξ t k − ε ) ≥ e − 2 a κ ξ t k − ε . \lambda_k \ge e^{-a}\left(\frac{e^{-a}\kappa\xi}{t_k}-\varepsilon\right) \ge\frac{e^{-2a}\kappa\xi}{t_k}-\varepsilon. λ k ≥ e − a ( t k e − a κ ξ − ε ) ≥ t k e − 2 a κ ξ − ε .
By Step 1, the warm-up condition (4), which reads λ k ≥ 2 R 2 β n 2 ( δ ) \lambda_k \ge2R^2\beta_n^2(\delta) λ k ≥ 2 R 2 β n 2 ( δ ) , therefore holds as soon as
k log ( 1 + k ) ≥ 2 d U e 2 a ( 2 R 2 β n 2 ( δ ) + ε ) κ ξ = K . \frac{k}{\log(1+k)} \ge\frac{2d_Ue^{2a}\left(2R^2\beta_n^2(\delta)+\varepsilon\right)}{\kappa\xi}=K. log ( 1 + k ) k ≥ κ ξ 2 d U e 2 a ( 2 R 2 β n 2 ( δ ) + ε ) = K .
Since log ( 1 + k ) ≤ log ( 2 k ) \log(1+k)\le\log(2k) log ( 1 + k ) ≤ log ( 2 k ) , it suffices that k ≥ K log ( 2 k ) k\ge K\log(2k) k ≥ K log ( 2 k ) . As κ Π P ⪯ H P ( θ E ) ⪯ ε Π P \kappa\Pi_P\preceq\mathcal{H}_P(\theta^{\mathrm{E}})\preceq\varepsilon\Pi_P κ Π P ⪯ H P ( θ E ) ⪯ ε Π P gives κ ≤ ε \kappa\le\varepsilon κ ≤ ε , and ξ ≤ 1 \xi\le1 ξ ≤ 1 , we have K ≥ 2 K\ge2 K ≥ 2 , so Lemma C.6 shows that this holds for every k ≥ 2 K log ( 4 K ) k\ge2K\log(4K) k ≥ 2 K log ( 4 K ) . Since n δ n_{\delta} n δ is the first round satisfying the warm-up condition, n δ ≤ ⌈ 2 K log ( 4 K ) ⌉ n_{\delta}\le\lceil2K\log(4K)\rceil n δ ≤ ⌈ 2 K log ( 4 K )⌉ whenever the right-hand side is at most n n n .
While looser due to the less precise offline information H 0 k \mathcal{H}_0^k H 0 k , this strategy retains a transfer rate that does not depend on ∣ P ∣ |\mathcal{P}| ∣ P ∣ , and provides a practical alternative when interaction is limited. The resulting offline algorithm is given in Appendix § 15 .
Tractable warm-up length bound
The bound of Lemma 3.3 depends on θ E \theta^{\mathrm{E}} θ E through λ ∣ P ∣ \lambda_{|\mathcal{P}|} λ ∣ P ∣ and d ⋆ d_{\star} d ⋆ . Both can be bounded using only the Hessians at θ = 0 \theta=0 θ = 0 , which only involve the uniform policy and are computed with the projectors (Appendix § 15 ).
Corollary F.1. Let λ ˉ : = e − R B θ λ min U ( ∑ P ∈ P H P ( 0 ) ) \bar{\lambda}:=e^{-RB_{\theta}}\lambda_{\min}^{U}\left(\sum_{P\in\mathcal{P}}\mathcal{H}_P(0)\right) λ ˉ := e − R B θ λ m i n U ( ∑ P ∈ P H P ( 0 ) ) and d ˉ ⋆ : = d U e R B θ H 2 B ϕ 2 max P ∈ P λ min U ( H P ( 0 ) ) − 1 \bar{d}_{\star}:=d_Ue^{RB_{\theta}H^2B_{\phi}^2}\max_{P\in\mathcal{P}}\lambda_{\min}^{U}\left(\mathcal{H}_P(0)\right)^{-1} d ˉ ⋆ := d U e R B θ H 2 B ϕ 2 max P ∈ P λ m i n U ( H P ( 0 ) ) − 1 , and let β ˉ n 2 ( δ ) \bar{\beta}_n^2(\delta) β ˉ n 2 ( δ ) be defined as β n 2 ( δ ) \beta_n^2(\delta) β n 2 ( δ ) with d ˉ ⋆ \bar{d}_{\star} d ˉ ⋆ in place of d ⋆ d_{\star} d ⋆ . Under round-robin selection (6),
n δ ≤ ∣ P ∣ ⌈ 2 R 2 β ˉ n 2 ( δ ) λ ˉ ⌉ . n_{\delta}\le|\mathcal{P}|\left\lceil\frac{2R^2\bar{\beta}_n^2(\delta)}{\bar{\lambda}}\right\rceil. n δ ≤ ∣ P ∣ ⌈ λ ˉ 2 R 2 β ˉ n 2 ( δ ) ⌉ .
Proof. Step 1 (information after one sweep). Since ∥ θ E ∥ Π P ≤ ∥ θ E ∥ 2 ≤ B θ \|\theta^{\mathrm{E}}\|_{\Pi_P}\le\|\theta^{\mathrm{E}}\|_2\le B_{\theta} ∥ θ E ∥ Π P ≤ ∥ θ E ∥ 2 ≤ B θ for every P ∈ P P\in\mathcal{P} P ∈ P , item 1 of Lemma B.7 with θ 1 = θ E \theta_1=\theta^{\mathrm{E}} θ 1 = θ E and θ 2 = 0 \theta_2=0 θ 2 = 0 gives
H P ( θ E ) ⪰ e − R B θ H P ( 0 ) . \mathcal{H}_P(\theta^{\mathrm{E}})\succeq e^{-RB_{\theta}}\mathcal{H}_P(0). H P ( θ E ) ⪰ e − R B θ H P ( 0 ) .
Summing over P P P yields λ ∣ P ∣ ≥ λ ˉ \lambda_{|\mathcal{P}|}\ge\bar{\lambda} λ ∣ P ∣ ≥ λ ˉ .
Step 2 (effective dimension). Since ∥ ϕ ( τ ) ∥ 2 ≤ H B ϕ \|\phi(\tau)\|_2\le HB_{\phi} ∥ ϕ ( τ ) ∥ 2 ≤ H B ϕ , we have ∥ Σ P ∥ 2 ≤ E ∥ Π P ϕ ( τ ) ∥ 2 2 ≤ H 2 B ϕ 2 \|\Sigma_P\|_2\le\mathbb{E}\|\Pi_P\phi(\tau)\|_2^2\le H^2B_{\phi}^2 ∥ Σ P ∥ 2 ≤ E ∥ Π P ϕ ( τ ) ∥ 2 2 ≤ H 2 B ϕ 2 . Moreover, Step 1 gives
H P ( θ E ) † ⪯ e R B θ λ min U ( H P ( 0 ) ) − 1 Π P , \mathcal{H}_P(\theta^{\mathrm{E}})^{\dagger}\preceq e^{RB_{\theta}}\lambda_{\min}^{U}\left(\mathcal{H}_P(0)\right)^{-1}\Pi_P, H P ( θ E ) † ⪯ e R B θ λ m i n U ( H P ( 0 ) ) − 1 Π P ,
so that d ⋆ ≤ d ˉ ⋆ d_{\star}\le\bar{d}_{\star} d ⋆ ≤ d ˉ ⋆ . Since the threshold (4) is increasing in d ⋆ d_{\star} d ⋆ , it is at most 2 R 2 β ˉ n 2 ( δ ) 2R^2\bar{\beta}_n^2(\delta) 2 R 2 β ˉ n 2 ( δ ) .
Step 3 (warm-up length). Under round-robin selection, after k k k rounds every P ∈ P P\in\mathcal{P} P ∈ P was selected at least ⌊ k / ∣ P ∣ ⌋ \lfloor k/|\mathcal{P}|\rfloor ⌊ k /∣ P ∣ ⌋ times, so that
λ k ≥ ⌊ k ∣ P ∣ ⌋ λ ∣ P ∣ ≥ ⌊ k ∣ P ∣ ⌋ λ ˉ . \lambda_k\ge\left\lfloor\frac{k}{|\mathcal{P}|}\right\rfloor\lambda_{|\mathcal{P}|}\ge\left\lfloor\frac{k}{|\mathcal{P}|}\right\rfloor\bar{\lambda}. λ k ≥ ⌊ ∣ P ∣ k ⌋ λ ∣ P ∣ ≥ ⌊ ∣ P ∣ k ⌋ λ ˉ .
By Step 2, the warm-up condition thus holds at round k = ∣ P ∣ ⌈ 2 R 2 β ˉ n 2 ( δ ) / λ ˉ ⌉ k=|\mathcal{P}|\left\lceil2R^2\bar{\beta}_n^2(\delta)/\bar{\lambda}\right\rceil k = ∣ P ∣ ⌈ 2 R 2 β ˉ n 2 ( δ ) / λ ˉ ⌉ , which proves the claim.
The factors e R B θ e^{RB_{\theta}} e R B θ make this bound loose, which is why we treat the warm-up length as a hyperparameter in practice (§ 4 ).
Misspecified experts
So far, we assumed that the expert is soft-optimal for θ E \theta^{\mathrm E} θ E under the learner’s temperature η \eta η . We now only require its behavior to be explained by a nearby reward.
Assumption G.1 (Misspecified expert). There exists θ ~ E ∈ Θ \widetilde{\theta}^{\mathrm E} \in\Theta θ E ∈ Θ with R ∥ θ ~ E − θ E ∥ Π U ≤ 1 R\left\|\widetilde{\theta}^{\mathrm E}-\theta^{\mathrm E}\right\|_{\Pi_U} \le1 R θ E − θ E Π U ≤ 1 such that, for every P ∈ P P \in\mathcal{P} P ∈ P , the expert follows π ~ P = π P ( θ ~ E ) \widetilde{\pi}_P=\pi_P(\widetilde{\theta}^{\mathrm E}) π P = π P ( θ E ) . We denote its suboptimality by
ε m i s : = max P ∈ P ( V P ∗ ( θ E ) − V ~ P π ~ P ( θ E ) ) . \varepsilon_{\mathrm{mis}}:=\max_{P\in\mathcal{P}}\left(V_P^*(\theta^{\mathrm E})-\widetilde{V}_P^{\widetilde{\pi}_P}(\theta^{\mathrm E})\right). ε mis := P ∈ P max ( V P ∗ ( θ E ) − V P π P ( θ E ) ) .
This covers a human demonstrator who is slightly mistaken about the task’s objective, or whose level of randomness differs from the one the learner assumes. The latter simply amounts to a rescaled reward: an expert acting with an inverse temperature η E ≠ η \eta^{\mathrm E}\ne\eta η E = η follows π P ( θ ~ E ) \pi_P(\widetilde{\theta}^{\mathrm E}) π P ( θ E ) with θ ~ E = ( η E / η ) θ E \widetilde{\theta}^{\mathrm E}=(\eta^{\mathrm E}/\eta)\theta^{\mathrm E} θ E = ( η E / η ) θ E , since soft-optimal policies depend on the reward only through η θ \eta\theta η θ , provided ( η E / η ) ∥ θ E ∥ Θ ≤ B θ (\eta^{\mathrm E}/\eta)\left\|\theta^{\mathrm E}\right\|_{\Theta}\le B_\theta ( η E / η ) θ E Θ ≤ B θ . Its suboptimality then grows quadratically with the mismatch,
ε m i s ≤ ( η E / η − 1 ) 2 max P ∈ P ∥ θ E ∥ H P ( θ E ) 2 . \varepsilon_{\mathrm{mis}}\le(\eta^{\mathrm E}/\eta-1)^2\max_{P\in\mathcal{P}}\left\|\theta^{\mathrm E}\right\|_{\mathcal{H}_P(\theta^{\mathrm E})}^2. ε mis ≤ ( η E / η − 1 ) 2 P ∈ P max θ E H P ( θ E ) 2 .
Since the demonstrations are well-specified for θ ~ E \widetilde{\theta}^{\mathrm E} θ E , all our results apply to θ ~ E \widetilde{\theta}^{\mathrm E} θ E . Transferring to θ E \theta^{\mathrm E} θ E then only adds the expert’s own suboptimality.
Lemma G.2. Let Assumption G.1 hold, and let d ⋆ d_\star d ⋆ and n δ n_\delta n δ be defined with respect to the actual expert, i.e., at θ E \theta^{\mathrm E} θ E . With probability at least 1 − δ 1-\delta 1 − δ , it holds for all P ∈ P P\in\mathcal{P} P ∈ P and n δ ≤ k ≤ n n_\delta\le k\le n n δ ≤ k ≤ n that
SubOpt ( θ k , P ) ≲ g k ( P ) β k 2 ( δ ) + ε m i s . \operatorname{SubOpt}(\theta_k,P)\lesssim g_k(P)\beta_k^2(\delta)+\varepsilon_{\mathrm{mis}}. SubOpt ( θ k , P ) ≲ g k ( P ) β k 2 ( δ ) + ε mis . Proof. Let w : = θ ~ E − θ E w:=\widetilde{\theta}^{\mathrm E}-\theta^{\mathrm E} w := θ E − θ E denote the misspecification and u k : = θ k − θ ~ E u_k:=\theta_k-\widetilde{\theta}^{\mathrm E} u k := θ k − θ E the estimation error, so that θ k − θ E = u k + w \theta_k-\theta^{\mathrm E}=u_k+w θ k − θ E = u k + w , and write H ~ P : = H P ( θ ~ E ) \widetilde{\mathcal{H}}_P:=\mathcal{H}_P(\widetilde{\theta}^{\mathrm E}) H P := H P ( θ E ) .
Step 1 (comparison of curvatures). Since U P ⊂ U U_P\subset U U P ⊂ U , we have R ∥ w ∥ Π P ≤ R ∥ w ∥ Π U ≤ 1 R\|w\|_{\Pi_P}\le R\|w\|_{\Pi_U}\le1 R ∥ w ∥ Π P ≤ R ∥ w ∥ Π U ≤ 1 for every P ∈ P P\in\mathcal{P} P ∈ P , and item 1 of Lemma B.7 gives
e − 1 H P ( θ E ) ⪯ H ~ P ⪯ e H P ( θ E ) . e^{-1}\mathcal{H}_P(\theta^{\mathrm E})\preceq\widetilde{\mathcal{H}}_P\preceq e\mathcal{H}_P(\theta^{\mathrm E}). e − 1 H P ( θ E ) ⪯ H P ⪯ e H P ( θ E ) . Hence, the information gain evaluated at θ ~ E \widetilde{\theta}^{\mathrm E} θ E is within a factor e 2 e^2 e 2 of g k ( P ) g_k(P) g k ( P ) , by the argument of Step 1 in the proof of Theorem 3.6.
Step 2 (estimation error). Since the demonstrations are generated by π P ( θ E ) \pi_P(\theta^{\mathrm E}) π P ( θ E ) with θ ~ E ∈ Θ \widetilde{\theta}^{\mathrm E}\in\Theta θ E ∈ Θ , Lemma 3.1 applies with θ ~ E \widetilde{\theta}^{\mathrm E} θ E in place of θ E \theta^{\mathrm E} θ E for all n δ ≤ k ≤ n n_\delta\le k\le n n δ ≤ k ≤ n . On its event, the proof of Lemma 3.2 and Step 1 yield
∥ u k ∥ H ~ P 2 ≤ e 2 g k ( P ) β k 2 ( δ ) and R ∥ u k ∥ Π P ≤ 1 , \|u_k\|_{\widetilde{\mathcal{H}}_P}^2\le e^2g_k(P)\beta_k^2(\delta)
\qquad\text{and}\qquad
R\|u_k\|_{\Pi_P}\le1, ∥ u k ∥ H P 2 ≤ e 2 g k ( P ) β k 2 ( δ ) and R ∥ u k ∥ Π P ≤ 1 , hence R ∥ u k + w ∥ Π P ≤ 2 R\|u_k+w\|_{\Pi_P}\le2 R ∥ u k + w ∥ Π P ≤ 2 for all $n_\delta\le k\le n.
Step 3 (decomposition). By Proposition C.1 and item 2 of Lemma B.7,
SubOpt ( θ k , P ) ≤ ψ ( 2 ) ∥ u k + w ∥ H P ( θ E ) 2 ≤ 2 ψ ( 2 ) ( ∥ u k ∥ H P ( θ E ) 2 + ∥ w ∥ H P ( θ E ) 2 ) . \operatorname{SubOpt}(\theta_k,P)\le\psi(2)\|u_k+w\|_{\mathcal{H}_P(\theta^{\mathrm E})}^2\le2\psi(2)\left(\|u_k\|_{\mathcal{H}_P(\theta^{\mathrm E})}^2+\|w\|_{\mathcal{H}_P(\theta^{\mathrm E})}^2\right). SubOpt ( θ k , P ) ≤ ψ ( 2 ) ∥ u k + w ∥ H P ( θ E ) 2 ≤ 2 ψ ( 2 ) ( ∥ u k ∥ H P ( θ E ) 2 + ∥ w ∥ H P ( θ E ) 2 ) . By Steps 1 and 2, the first term satisfies ∥ u k ∥ H P ( θ E ) 2 ≤ e ∥ u k ∥ H ~ P 2 ≲ g k ( P ) β k 2 ( δ ) \|u_k\|_{\mathcal{H}_P(\theta^{\mathrm E})}^2\le e\|u_k\|_{\widetilde{\mathcal{H}}_P}^2\lesssim g_k(P)\beta_k^2(\delta) ∥ u k ∥ H P ( θ E ) 2 ≤ e ∥ u k ∥ H P 2 ≲ g k ( P ) β k 2 ( δ ) . For the second, Proposition C.1 and item 2 of Lemma B.7 with θ 1 = θ ~ E \theta_1=\widetilde{\theta}^{\mathrm E} θ 1 = θ E and θ 2 = θ E \theta_2=\theta^{\mathrm E} θ 2 = θ E give, since R ∥ w ∥ Π P ≤ 1 R\|w\|_{\Pi_P}\le1 R ∥ w ∥ Π P ≤ 1 ,
ε m i s ≥ V P ∗ ( θ E ) − V ~ P π ~ P ( θ E ) = ∥ w ∥ G P ( θ ~ E , θ E ) 2 ≥ ψ ( − 1 ) ∥ w ∥ H P ( θ E ) 2 , \varepsilon_{\mathrm{mis}}\ge V_P^*(\theta^{\mathrm E})-\widetilde{V}_P^{\widetilde{\pi}_P}(\theta^{\mathrm E})=\|w\|_{\mathcal{G}_P(\widetilde{\theta}^{\mathrm E},\theta^{\mathrm E})}^2\ge\psi(-1)\|w\|_{\mathcal{H}_P(\theta^{\mathrm E})}^2, ε mis ≥ V P ∗ ( θ E ) − V P π P ( θ E ) = ∥ w ∥ G P ( θ E , θ E ) 2 ≥ ψ ( − 1 ) ∥ w ∥ H P ( θ E ) 2 , which concludes the proof. \hfill □ \square □
Lemma G.2 is Lemma 3.2 plus a bias. Since θ k \theta_k θ k remains within distance 2 / R 2/R 2/ R of θ E \theta^{\mathrm E} θ E along every U P U_P U P after the warm-up (Step 2), Theorem 3.6 carries over with larger constants and the same bias. In short, the learned reward transfers as well as the expert itself, up to the usual rate. No amount of demonstrations removes this bias.
The same holds for Theorem 3.7, for any target dynamics P P P ,
SubOpt ( θ n , P ) = O ~ ( d ⋆ d U n m ) + O ( ε m i s + g ⊥ ( P ) B θ 2 ) \operatorname{SubOpt}(\theta_n,P)=\widetilde{O}\left(\frac{d_\star d_U}{nm}\right)+O\left(\varepsilon_{\mathrm{mis}}+g_\perp(P)B_\theta^2\right) SubOpt ( θ n , P ) = O ( nm d ⋆ d U ) + O ( ε mis + g ⊥ ( P ) B θ 2 )
since the misspecification only enters the error along U U U , which the proof of Theorem 3.7 bounds through the dynamics in P P P , while the error along U ⊥ U^\perp U ⊥ is bounded as before. Remark G.3 (Unregularized experts). Deterministic experts (η → ∞ \eta\to\infty η → ∞ ) lie outside our framework, as entropy regularization is what makes the reward identifiable along each U P U_P U P [4 , 6 ] . They can be approximated with a large η \eta η , at the cost of constants growing with η \eta η , such as R R R .
Algorithms
Hessian estimation. Algorithm 2 estimates the information H P ( θ ) \mathcal{H}_P(\theta) H P ( θ ) of a dynamics P P P from trajectories of its optimal policy π P ( θ ) \pi_P(\theta) π P ( θ ) , using the feature advantages of Lemma B.1 , which require one policy evaluation of the features. Under deterministic transitions and initial state, the advantages reduce to centered trajectory features (Corollary B.2 ), so no policy evaluation is needed. Under stochastic dynamics, the advantages cannot be replaced by trajectory features, whose covariance also contains the dynamics noise Σ P d y n \Sigma_P^{\mathrm{dyn}} Σ P dyn (Lemma B.3 ), which may lie outside U P U_P U P .
Projectors. Since U P U_P U P does not depend on θ \theta θ (Proposition B.5 ), each projector is computed once, before any demonstration, from the information at θ = 0 \theta=0 θ = 0 , whose optimal policy π P ( 0 ) \pi_P(0) π P ( 0 ) is uniform. We run Algorithm 2 with π P ( 0 ) \pi_P(0) π P ( 0 ) to obtain H ^ P ( 0 ) \widehat{\mathcal{H}}_P(0) H P ( 0 ) , compute its eigendecomposition H ^ P ( 0 ) = ∑ i = 1 d λ ^ i u ^ i u ^ i ⊤ \widehat{\mathcal{H}}_P(0)=\sum_{i=1}^{d}\widehat{\lambda}_i\widehat{u}_i\widehat{u}_i^\top H P ( 0 ) = ∑ i = 1 d λ i u i u i ⊤ , and set
Π ^ P : = ∑ i : λ ^ i > ε λ u ^ i u ^ i ⊤ \widehat{\Pi}_P:=\sum_{i:\widehat{\lambda}_i>\varepsilon_\lambda}\widehat{u}_i\widehat{u}_i^\top Π P := i : λ i > ε λ ∑ u i u i ⊤
With exact Hessians, this recovers Π P = H P ( 0 ) H P ( 0 ) † \Pi_P=\mathcal{H}_P(0)\mathcal{H}_P(0)^\dagger Π P = H P ( 0 ) H P ( 0 ) † for any ε \varepsilon ε below the smallest positive eigenvalue of H P ( 0 ) \mathcal{H}_P(0) H P ( 0 ) . The threshold ε > 0 \varepsilon>0 ε > 0 discards the small eigenvalues that sampling noise adds along unidentifiable directions. For our experiments, we fixed ε λ = 5 ⋅ 10 − 4 \varepsilon_\lambda=5\cdot10^{-4} ε λ = 5 ⋅ 1 0 − 4 .
Two-phase active IRL. Algorithm 4 details our two-phase approach. After computing the projectors, it selects dynamics round-robin (6) during the first n 0 n_0 n 0 rounds, observing m m m expert trajectories in each, and then computes θ n 0 \theta_{n_0} θ n 0 with Algorithm 3. At every subsequent round k k k , it
estimates H P ( θ k − 1 ) \mathcal{H}_P(\theta_{k-1}) H P ( θ k − 1 ) for each P ∈ P P\in\mathcal{P} P ∈ P with Algorithm 2, using the policies π P ( θ k − 1 ) \pi_P(\theta_{k-1}) π P ( θ k − 1 ) returned by Algorithm 3;
selects P k P_k P k by (7) and observes m m m expert trajectories in P k P_k P k ;
updates θ k \theta_k θ k with Algorithm 3 on all data collected so far.
Offline active IRL. Algorithm 5 selects the whole sequence ( P k ) k = 1 n (P_k)_{k=1}^{n} ( P k ) k = 1 n before observing the expert, using θ 0 = 0 \theta_0=0 θ 0 = 0 in place of θ k − 1 \theta_{k-1} θ k − 1 and no warm-up. The required Hessians H P ( 0 ) \mathcal{H}_P(0) H P ( 0 ) are those already computed for the projectors. The expert is then observed on the selected dynamics, and the reward is estimated once with Algorithm 3.
Computational complexity. The main cost in our setting is the number of expert demonstrations, which our analysis minimizes. All other computations only require access to the dynamics in P \mathcal{P} P , and we do not optimize them further. For completeness:
Projectors (once): ∣ P ∣ |\mathcal{P}| ∣ P ∣ runs of Algorithm 2, each with one policy evaluation of d d d -dimensional features and one rollout batch, followed by an eigendecomposition of a d × d d\times d d × d matrix.
Information estimation (per active round): ∣ P ∣ |\mathcal{P}| ∣ P ∣ runs of Algorithm 2 with the current policies, which Algorithm 3 already returns, so no additional RL solve is needed.
Dynamics selection (per active round): one inverse square root of a d × d d\times d d × d matrix and ∣ P ∣ |\mathcal{P}| ∣ P ∣ evaluations of J J J , i.e., O ( ∣ P ∣ d 3 ) O(|\mathcal{P}|d^3) O ( ∣ P ∣ d 3 ) operations.
Algorithm 2 Hessian estimation
Input: dynamics P P P , policy π = π P ( θ ) \pi= \pi_P(\theta) π = π P ( θ ) , rollouts N N N
Set Φ ˉ H + 1 : = 0 \bar{\Phi}_{H+1} := 0 Φ ˉ H + 1 := 0
for h = H , … , 1 h = H, \ldots, 1 h = H , … , 1 do
for a ∈ A , s ∈ S a \in\mathcal{A}, s \in\mathcal{S} a ∈ A , s ∈ S do
Φ h ( s , a ) ← ϕ ( s , a ) + ∑ s ′ T ( s ′ ∣ s , a ) Φ ˉ h + 1 ( s ′ ) \Phi_h(s,a) \leftarrow\phi(s,a) + \sum_{s'} T(s' \mid s,a)\bar{\Phi}_{h+1}(s') Φ h ( s , a ) ← ϕ ( s , a ) + ∑ s ′ T ( s ′ ∣ s , a ) Φ ˉ h + 1 ( s ′ )
Φ ˉ h ( s ) ← ∑ a π h ( a ∣ s ) Φ h ( s , a ) \bar{\Phi}_h(s) \leftarrow\sum_a \pi_h(a \mid s)\Phi_h(s,a) Φ ˉ h ( s ) ← ∑ a π h ( a ∣ s ) Φ h ( s , a )
Sample τ 1 , … , τ N \tau_1,\ldots,\tau_N τ 1 , … , τ N with π \pi π in P P P
A ( τ i ) ← ∑ h = 1 H ( Φ h ( s h , a h ) − Φ ˉ h ( s h ) ) A(\tau_i) \leftarrow\sum_{h=1}^{H}\left(\Phi_h(s_h,a_h)-\bar{\Phi}_h(s_h)\right) A ( τ i ) ← ∑ h = 1 H ( Φ h ( s h , a h ) − Φ ˉ h ( s h ) ) (Feature advantage)
return: H ^ P ( θ ) ← η N ∑ i = 1 N A ( τ i ) A ( τ i ) ⊤ \widehat{H}_P(\theta) \leftarrow\frac{\eta}{N}\sum_{i=1}^{N} A(\tau_i)A(\tau_i)^{\top} H P ( θ ) ← N η ∑ i = 1 N A ( τ i ) A ( τ i ) ⊤ (Lemma B.1)
Algorithm 3 Projected min-max IRL with multiple dynamics
Input: expert data { ( P i , ϕ ^ i ) } i = 1 k \left\{(P_i,\hat{\phi}_i)\right\}_{i=1}^{k} { ( P i , ϕ ^ i ) } i = 1 k , projectors { Π ^ P } P ∈ P \left\{\hat{\Pi}_P\right\}_{P\in\mathcal{P}} { Π ^ P } P ∈ P , step size α \alpha α , iterations K K K , rollouts N N N
Initialize: θ ( 0 ) ← 0 \theta^{(0)} \leftarrow0 θ ( 0 ) ← 0
for t = 0 , … , K − 1 t = 0,\ldots,K-1 t = 0 , … , K − 1 do
for each distinct P ∈ { P 1 , … , P k } P \in\{P_1,\ldots,P_k\} P ∈ { P 1 , … , P k } do
π ← π P ( θ ( t ) ) \pi\leftarrow\pi_P(\theta^{(t)}) π ← π P ( θ ( t ) ) (RL)
ϕ ˉ P t ← 1 N ∑ j = 1 N ϕ ( τ j ) \bar{\phi}^{t}_P \leftarrow\frac{1}{N}\sum_{j=1}^{N}\phi(\tau_j) ϕ ˉ P t ← N 1 ∑ j = 1 N ϕ ( τ j ) , with τ j ∼ π \tau_j \sim\pi τ j ∼ π in P P P (Rollouts)
g t ← ∑ i = 1 k Π ^ P i ( ϕ ˉ P i t − ϕ ^ i ) g_t \leftarrow\sum_{i=1}^{k}\hat{\Pi}_{P_i}\left(\bar{\phi}^{t}_{P_i}-\hat{\phi}_i\right) g t ← ∑ i = 1 k Π ^ P i ( ϕ ˉ P i t − ϕ ^ i ) (Gradient of (2))
θ ( t + 1 ) ← Π Θ ( θ ( t ) − α g t ) \theta^{(t+1)} \leftarrow\Pi_{\Theta}\left(\theta^{(t)}-\alpha g_t\right) θ ( t + 1 ) ← Π Θ ( θ ( t ) − α g t )
θ k ← 1 K ∑ t = 0 K − 1 θ ( t ) \theta_k \leftarrow\frac{1}{K}\sum_{t=0}^{K-1}\theta^{(t)} θ k ← K 1 ∑ t = 0 K − 1 θ ( t )
return: θ k , { π P ( θ k ) } P ∈ P \theta_k,\{\pi_P(\theta_k)\}_{P\in\mathcal{P}} θ k , { π P ( θ k ) } P ∈ P
Algorithm 4 Active IRL
Input: P \mathcal{P} P , budget n n n , warm-up length n 0 ≥ 0 n_0 \ge0 n 0 ≥ 0 , trajectories per round m m m , regularization ε > 0 \varepsilon> 0 ε > 0 , criterion J J J
Initialize: D 0 E ← ∅ D^E_0 \leftarrow\emptyset D 0 E ← ∅ , k ( P ) ← 0 k(P) \leftarrow0 k ( P ) ← 0 for all P ∈ P P \in\mathcal{P} P ∈ P , θ 0 ← 0 \theta_0 \leftarrow0 θ 0 ← 0
Compute H ^ P ( 0 ) \widehat{H}_P(0) H P ( 0 ) and Π ^ P \hat{\Pi}_P Π ^ P for all P ∈ P P \in\mathcal{P} P ∈ P from π P ( 0 ) \pi_P(0) π P ( 0 ) (Algorithm 2)
for k = 1 , … , n k = 1,\ldots,n k = 1 , … , n do
if k ≤ n 0 k \le n_0 k ≤ n 0 then (Warm-up)
P k ∈ arg min P ∈ P k ( P ) P_k \in\arg\min_{P\in\mathcal{P}} k(P) P k ∈ arg min P ∈ P k ( P )
else (Active selection)
Estimate H ^ P ( θ k − 1 ) \widehat{H}_P(\theta_{k-1}) H P ( θ k − 1 ) for all P ∈ P P \in\mathcal{P} P ∈ P from π P ( θ k − 1 ) \pi_P(\theta_{k-1}) π P ( θ k − 1 ) (Algorithm 2)
H ^ k − 1 ← ε Π ^ U + ∑ P ∈ P k ( P ) H ^ P ( θ k − 1 ) \widehat{H}_{k-1} \leftarrow\varepsilon\hat{\Pi}_{\mathcal{U}}+\sum_{P\in\mathcal{P}}k(P)\widehat{H}_P(\theta_{k-1}) H k − 1 ← ε Π ^ U + ∑ P ∈ P k ( P ) H P ( θ k − 1 )
P k ∈ arg max P ∈ P J ( H ^ k − 1 † / 2 H ^ P ( θ k − 1 ) H ^ k − 1 † / 2 ) P_k \in\arg\max_{P\in\mathcal{P}} J\left(\widehat{H}_{k-1}^{\dagger/2}\widehat{H}_P(\theta_{k-1})\widehat{H}_{k-1}^{\dagger/2}\right) P k ∈ arg max P ∈ P J ( H k − 1 † /2 H P ( θ k − 1 ) H k − 1 † /2 )
k ( P k ) ← k ( P k ) + 1 k(P_k) \leftarrow k(P_k)+1 k ( P k ) ← k ( P k ) + 1
D k E ← D k − 1 E ∪ { τ k , j } j = 1 m D^E_k \leftarrow D^E_{k-1}\cup\{\tau_{k,j}\}_{j=1}^{m} D k E ← D k − 1 E ∪ { τ k , j } j = 1 m , with τ k , j \tau_{k,j} τ k , j generated by π P k E \pi^{E}_{P_k} π P k E under P k P_k P k (Observe expert)
if k ≥ n 0 k \ge n_0 k ≥ n 0 then
θ k , { π P ( θ k ) } P ∈ P ← IRL ( D k E , { Π ^ P } P ∈ P ) \theta_k,\{\pi_P(\theta_k)\}_{P\in\mathcal{P}} \leftarrow\operatorname{IRL}\left(D^E_k,\{\hat{\Pi}_P\}_{P\in\mathcal{P}}\right) θ k , { π P ( θ k ) } P ∈ P ← IRL ( D k E , { Π ^ P } P ∈ P ) (Algorithm 3)
return: θ n , { π P ( θ n ) } P ∈ P \theta_n,\{\pi_P(\theta_n)\}_{P\in\mathcal{P}} θ n , { π P ( θ n ) } P ∈ P
Reward estimation (per round): K K K gradient steps of Algorithm 3, each requiring one RL solve and one rollout batch per distinct dynamics selected so far, i.e., at most min ( k , ∣ P ∣ ) \min(k,|\mathcal{P}|) min ( k , ∣ P ∣ ) , plus ∣ P ∣ |\mathcal{P}| ∣ P ∣ final solves for the returned policies. In our finite-horizon tabular setting, each RL solve is a single backward pass of soft value iteration [3 ] , i.e., O ( H ∣ S ∣ 2 ∣ A ∣ ) \mathcal{O}(H|\mathcal{S}|^2|\mathcal{A}|) O ( H ∣ S ∣ 2 ∣ A ∣ ) operations. Alternatively, each solve can be replaced by a single soft policy iteration step per reward update, which still converges [35 ] and avoids nested RL loops in larger problems.
Memory is dominated by the ∣ P ∣ |\mathcal{P}| ∣ P ∣ Hessians and projectors, i.e., O ( ∣ P ∣ d 2 ) \mathcal{O}(|\mathcal{P}|d^2) O ( ∣ P ∣ d 2 ) . In comparison, the environment design approach of Buening et al. [10 ] requires O ( k ∣ P ∣ ) \mathcal{O}(k|\mathcal{P}|) O ( k ∣ P ∣ ) RL solves at round k k k for selection alone.
Algorithm 5 Offline Active IRL
Input: P \mathcal{P} P , budget n n n , trajectories per round m m m , regularization ε > 0 \varepsilon> 0 ε > 0 , criterion J J J
Compute H ^ P ( 0 ) \widehat{\mathcal{H}}_{\mathcal{P}}(0) H P ( 0 ) and Π ^ P \widehat{\Pi}_{\mathcal{P}} Π P for all P ∈ P \mathcal{P} \in\mathcal{P} P ∈ P from π P ( 0 ) \pi_{\mathcal{P}}(0) π P ( 0 ) (Algorithm 2)
H 0 0 ← ε Π U \mathcal{H}^{0}_{0} \gets\varepsilon\Pi_{U} H 0 0 ← ε Π U , with ε : = η H 2 B ϕ 2 \varepsilon:= \eta H^{2}B_{\phi}^{2} ε := η H 2 B ϕ 2
for k = 1 , … , n k = 1,\ldots,n k = 1 , … , n do
P k ∈ arg max P ∈ P J ( ( H k − 1 0 ) † / 2 H ^ P ( 0 ) ( H k − 1 0 ) † / 2 ) \mathcal{P}_{k} \in\arg\max_{\mathcal{P}\in\mathcal{P}} J\left((\mathcal{H}^{0}_{k-1})^{\dagger/2}\widehat{\mathcal{H}}_{\mathcal{P}}(0)(\mathcal{H}^{0}_{k-1})^{\dagger/2}\right) P k ∈ arg max P ∈ P J ( ( H k − 1 0 ) † /2 H P ( 0 ) ( H k − 1 0 ) † /2 ) (Offline active selection (35))
H k 0 ← H k − 1 0 + H ^ P k ( 0 ) \mathcal{H}^{0}_{k} \gets\mathcal{H}^{0}_{k-1} + \widehat{\mathcal{H}}_{\mathcal{P}_{k}}(0) H k 0 ← H k − 1 0 + H P k ( 0 )
D E ← ⋃ k = 1 n { τ k , j } j = 1 m \mathcal{D}^{\mathrm{E}} \gets\bigcup_{k=1}^{n}\{\tau_{k,j}\}_{j=1}^{m} D E ← ⋃ k = 1 n { τ k , j } j = 1 m , with τ k , j \tau_{k,j} τ k , j generated by π P k E \pi^{\mathrm{E}}_{\mathcal{P}_{k}} π P k E under P k \mathcal{P}_{k} P k (Observe expert)
θ n , { π P ( θ n ) } P ∈ P ← IRL ( D E , { Π ^ P } P ∈ P ) \theta_{n},\{\pi_{\mathcal{P}}(\theta_{n})\}_{\mathcal{P}\in\mathcal{P}} \gets\operatorname{IRL}\left(\mathcal{D}^{\mathrm{E}},\{\widehat{\Pi}_{\mathcal{P}}\}_{\mathcal{P}\in\mathcal{P}}\right) θ n , { π P ( θ n ) } P ∈ P ← IRL ( D E , { Π P } P ∈ P ) (Algorithm 3)
return: θ n , { π P ( θ n ) } P ∈ P \theta_{n},\{\pi_{\mathcal{P}}(\theta_{n})\}_{\mathcal{P}\in\mathcal{P}} θ n , { π P ( θ n ) } P ∈ P
Additional experimental details and results
Common Experimental Details
All expert trajectories have length H = 100 H = 100 H = 100 . We set η = 20 \eta= 20 η = 20 and use B ϕ = B θ = 1 B_{\phi} = B_{\theta} = 1 B ϕ = B θ = 1 throughout all experiments. Policies corresponding to every combination of dynamics and reward functions are computed using soft policy iteration.
In all experiments, the feature map ϕ \phi ϕ is chosen as the indicator function over state–action pairs. Consequently, the reward function takes the form
r θ ( s , a ) = ∑ s ′ , a ′ 1 { s ′ = s , a ′ = a } θ s ′ , a ′ . r_{\theta}(s,a) = \sum_{s',a'} \mathbb{1}\{s' = s, a' = a\}\theta_{s',a'}. r θ ( s , a ) = s ′ , a ′ ∑ 1 { s ′ = s , a ′ = a } θ s ′ , a ′ .
Reported suboptimality is evaluated as the worst-case value over the considered set of dynamics P \mathcal{P} P , max P ∈ P SubOpt ( θ n , P ) \max_{\mathcal{P}\in\mathcal{P}}\operatorname{SubOpt}(\theta_n,\mathcal{P}) max P ∈ P SubOpt ( θ n , P ) .
Implementation details. The implementation is publicly available at https://anonymous.4open.science/r/transferable-active-irl.
Compute resources. All experiments were conducted on a single AMD EPYC 7302P machine.
Controlled sets of dynamics
Principal angles. Here we provide the details on principal angles, used for the first experimental setup. Intuitively, principal angles [23 ] can be seen as a natural generalization of angles between two lines or planes to higher dimensional subspaces. Formally we define them as follows.
Definition I.1 (Principal angles [23 , 6 ] ). Let V , W ⊆ R d \mathcal{V},\mathcal{W} \subseteq\mathbb{R}^{d} V , W ⊆ R d be two subspaces. The principal angles 0 ≤ α 1 ( V , W ) ≤ ⋯ ≤ α m ( V , W ) : = α max ( V , W ) ≤ π 2 0 \leq\alpha_{1}(\mathcal{V},\mathcal{W}) \leq\cdots\leq\alpha_{m}(\mathcal{V},\mathcal{W}) := \alpha_{\max}(\mathcal{V},\mathcal{W}) \leq\frac{\pi}{2} 0 ≤ α 1 ( V , W ) ≤ ⋯ ≤ α m ( V , W ) := α m a x ( V , W ) ≤ 2 π between V \mathcal{V} V and W \mathcal{W} W are defined recursively by
cos ( α i ( V , W ) ) = max v ∈ V , w ∈ W ⟨ v , w ⟩ \cos\left(\alpha_{i}(\mathcal{V},\mathcal{W})\right) = \max_{\substack{v\in\mathcal{V},\,w\in\mathcal{W}}} \langle v,w\rangle cos ( α i ( V , W ) ) = v ∈ V , w ∈ W max ⟨ v , w ⟩ s.t. ∥ v ∥ 2 = ∥ w ∥ 2 = 1 , \text{s.t.}\quad\lVert v\rVert_{2} = \lVert w\rVert_{2} = 1, s.t. ∥ v ∥ 2 = ∥ w ∥ 2 = 1 , ⟨ v , v j ⟩ = 0 , ⟨ w , w j ⟩ = 0 , j = 1 , … , i − 1 , \langle v,v_{j}\rangle= 0,\quad\langle w,w_{j}\rangle= 0,\quad j=1,\ldots,i-1, ⟨ v , v j ⟩ = 0 , ⟨ w , w j ⟩ = 0 , j = 1 , … , i − 1 , where v j ∈ V v_{j}\in\mathcal{V} v j ∈ V and w j ∈ W w_{j}\in\mathcal{W} w j ∈ W are maximizers corresponding to α j ( V , W ) \alpha_{j}(\mathcal{V},\mathcal{W}) α j ( V , W ) .
In the context of IRL, our work builds on the geometric perspective introduced by Schlaginhaufen and Kamgarpour [6 ] . In particular, their Theorems 3.10 and 3.11 characterize reward transferability through the relative geometry of the unidentifiable subspaces associated with different transition dynamics, quantified via principal angles. More precisely, Theorem 3.10 shows that when the second principal angle between two unidentifiable subspaces U P ⊥ U_{\mathcal{P}}^{\perp} U P ⊥ and U P ′ ⊥ U_{\mathcal{P}'}^{\perp} U P ′ ⊥ is sufficiently large, observing the expert on both dynamics enables the recovery of rewards that transfer to arbitrary dynamics. Conversely, Theorem 3.11 establishes a complementary regime: when the largest principal angle between U P U_P U P and U P ′ U_{P'} U P ′ is sufficiently small, a reward learned from demonstrations in one dynamics transfers reliably to the other. Taken together, these results reveal a geometric trade-off. Large diversity between environments, captured by a large second principal angle, improves global identifiability and enables broad transfer. In contrast, strong similarity between dynamics, captured by a small maximal principal angle, guarantees local transferability of learned rewards across environments.
Let α i ( P , P ′ ) \alpha_i(P,P') α i ( P , P ′ ) denote the i i i -th principal angle between the subspaces U P ⊥ U_P^\perp U P ⊥ and U P ′ ⊥ U_{P'}^\perp U P ′ ⊥ , and let α ˉ i ( P ) = 1 ∣ P ∣ ∑ P ′ ∈ P α i ( P , P ′ ) \bar{\alpha}_i(P)=\frac{1}{|\mathcal{P}|}\sum_{P'\in\mathcal{P}}\alpha_i(P,P') α ˉ i ( P ) = ∣ P ∣ 1 ∑ P ′ ∈ P α i ( P , P ′ ) be the average i i i -th principal angle of P P P over the set P \mathcal{P} P .
Details. In our experiments, we constructed four classes of dynamics sets, each characterized by the geometry of their unidentifiable subspaces through principal-angle statistics. For a set of dynamics P = { P 1 , … , P ∣ P ∣ } \mathcal{P}=\{P_1,\ldots,P_{|\mathcal{P}|}\} P = { P 1 , … , P ∣ P ∣ } , we define
μ i ( P ) : = 1 ∣ P ∣ ∑ P ∈ P α ˉ i ( P ) , σ i 2 ( P ) : = 1 ∣ P ∣ ∑ P ∈ P ( α ˉ i ( P ) − μ i ( P ) ) 2 . \mu_i(\mathcal{P}) := \frac{1}{|\mathcal{P}|}\sum_{P\in\mathcal{P}}\bar{\alpha}_i(P), \qquad\sigma_i^2(\mathcal{P}) := \frac{1}{|\mathcal{P}|}\sum_{P\in\mathcal{P}}\left(\bar{\alpha}_i(P)-\mu_i(\mathcal{P})\right)^2. μ i ( P ) := ∣ P ∣ 1 P ∈ P ∑ α ˉ i ( P ) , σ i 2 ( P ) := ∣ P ∣ 1 P ∈ P ∑ ( α ˉ i ( P ) − μ i ( P ) ) 2 .
All training dynamics are deterministic: every state-action pair ( s , a ) (s,a) ( s , a ) maps to a single next state. For each seed, a set P \mathcal{P} P of ∣ P ∣ = 10 |\mathcal{P}|=10 ∣ P ∣ = 10 such dynamics is obtained by an evolutionary search over candidate sets, driven by a fitness defined on principal-angle statistics. A class is specified by a target angle rank i ∈ { 2 , max } i\in\{2,\max\} i ∈ { 2 , max } , a target angle α ⋆ \alpha^\star α ⋆ , and a heterogeneity weight λ ≥ 0 \lambda\ge0 λ ≥ 0 . For a candidate set P \mathcal{P} P , let
m min ( P ) : = 1 ∣ P ∣ ∑ P ∈ P min P ′ ≠ P α i ( P , P ′ ) , m max ( P ) : = 1 ∣ P ∣ ∑ P ∈ P max P ′ ≠ P α i ( P , P ′ ) , m_{\min}(\mathcal{P}) := \frac{1}{|\mathcal{P}|}\sum_{P\in\mathcal{P}}\min_{P'\ne P}\alpha_i(P,P'), \qquad m_{\max}(\mathcal{P}) := \frac{1}{|\mathcal{P}|}\sum_{P\in\mathcal{P}}\max_{P'\ne P}\alpha_i(P,P'), m m i n ( P ) := ∣ P ∣ 1 P ∈ P ∑ P ′ = P min α i ( P , P ′ ) , m m a x ( P ) := ∣ P ∣ 1 P ∈ P ∑ P ′ = P max α i ( P , P ′ ) ,
and let h ( P ) : = π / 2 − 1 ∣ P ∣ − 1 ∑ j = 1 ∣ P ∣ − 1 α ˉ i , ( j + 1 ) − α ˉ i , ( j ) h(\mathcal{P}):=\sqrt{\pi/2}-\frac{1}{|\mathcal{P}|-1}\sum_{j=1}^{|\mathcal{P}|-1}\sqrt{\bar{\alpha}_{i,(j+1)}-\bar{\alpha}_{i,(j)}} h ( P ) := π /2 − ∣ P ∣ − 1 1 ∑ j = 1 ∣ P ∣ − 1 α ˉ i , ( j + 1 ) − α ˉ i , ( j ) , where α ˉ i , ( 1 ) ≤ ⋯ ≤ α ˉ i , ( ∣ P ∣ ) \bar{\alpha}_{i,(1)}\le\cdots\le\bar{\alpha}_{i,(|\mathcal{P}|)} α ˉ i , ( 1 ) ≤ ⋯ ≤ α ˉ i , ( ∣ P ∣ ) are the sorted average angles α ˉ i ( P ) \bar{\alpha}_i(P) α ˉ i ( P ) , P ∈ P P\in\mathcal{P} P ∈ P . The fitness to minimize is
f ( P ) : = max { ( m min ( P ) − α ⋆ ) 2 , ( m max ( P ) − α ⋆ ) 2 } + λ h ( P ) . f(\mathcal{P}) := \max\left\{(m_{\min}(\mathcal{P})-\alpha^\star)^2,\ (m_{\max}(\mathcal{P})-\alpha^\star)^2\right\}+\lambda h(\mathcal{P}). f ( P ) := max { ( m m i n ( P ) − α ⋆ ) 2 , ( m m a x ( P ) − α ⋆ ) 2 } + λh ( P ) .
The first term drives both the closest and the farthest pair of every dynamics toward the target angle, so that transferability is uniform across pairs; the second term, when active, rewards large gaps between the sorted average angles, so that transferability is heterogeneous across the set. The four classes correspond to
Equivalent: i = max i=\max i = max , α ⋆ = 0 \alpha^\star=0 α ⋆ = 0 , λ = 0 \lambda=0 λ = 0 ;
Similar: i = 2 i=2 i = 2 , α ⋆ = 0 \alpha^\star=0 α ⋆ = 0 , λ = 0 \lambda=0 λ = 0 ;
Uniformly diverse: i = 2 i=2 i = 2 , α ⋆ = π / 4 \alpha^\star=\pi/4 α ⋆ = π /4 , λ = 0 \lambda=0 λ = 0 ;
Arbitrarily diverse: i = 2 i=2 i = 2 , α ⋆ = π / 3 \alpha^\star=\pi/3 α ⋆ = π /3 , λ = 10 4 \lambda=10^4 λ = 1 0 4 .
The search maintains a population of 300 candidate sets, initialized as copies of a set whose ∣ P ∣ |\mathcal{P}| ∣ P ∣ dynamics are all equal to one random deterministic dynamics (next states drawn uniformly at random for every ( s , a ) (s,a) ( s , a ) ). At every generation, the 30 fittest candidates survive and 270 offspring are generated, each a copy of a uniformly drawn survivor in which every transition is independently redirected to the initial state with probability 1 / ( ∣ S ∣ ∣ A ∣ ) 1/(|\mathcal{S}||\mathcal{A}|) 1/ ( ∣ S ∣∣ A ∣ ) ; when i = 2 i=2 i = 2 , each dynamics of the offspring is additionally replaced by a fresh random deterministic dynamics with probability 0.1 / ∣ P ∣ 0.1/|\mathcal{P}| 0.1/∣ P ∣ . The search runs for at most 500 generations, stops early once f ( P ) < 10 − 5 f(\mathcal{P})<10^{-5} f ( P ) < 1 0 − 5 , and returns the fittest candidate, with dynamics indexed by increasing α ˉ i ( P ) \bar{\alpha}_i(P) α ˉ i ( P ) . Table 2 summarizes the statistics of the resulting dynamics sets, averaged over 20 runs.
Table 2. Principal-angle statistics (in radian) of the controlled sets of dynamics, averaged over 20 runs.
For each seed, we evaluate the recovered reward on a set of 1000 held-out stochastic dynamics { P ~ 1 , … , P ~ 1000 } \{\tilde{P}_1,\ldots,\tilde{P}_{1000}\} { P ~ 1 , … , P ~ 1000 } , drawn independently of the training set P \mathcal{P} P . For each P ~ k \tilde{P}_k P ~ k , a concentration vector β ∈ R ∣ S ∣ \beta\in\mathbb{R}^{|\mathcal{S}|} β ∈ R ∣ S ∣ is drawn with log β s ′ ∼ Unif [ − 2 , 0.1 ] \log\beta_{s'}\sim\operatorname{Unif}[-2,0.1] log β s ′ ∼ Unif [ − 2 , 0.1 ] independently for every next state s ′ s' s ′ , and every row P ~ k ( ⋅ ∣ s , a ) \tilde{P}_k(\cdot\mid s,a) P ~ k ( ⋅ ∣ s , a ) , ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A , is then sampled independently from Dirichlet(β \beta β ). Since β s ′ < 0.1 \beta_{s'}<0.1 β s ′ < 0.1 , most rows concentrate near the same next states, so the held-out dynamics are genuinely stochastic yet far from uniform, and none of them coincides with a deterministic training dynamics.
Regarding rewards, for each different seed, θ E \theta^E θ E is uniformly sampled in [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d , and then rescaled such that ∥ r E ∥ 2 ≤ 1 \lVert r^E\rVert_2\le1 ∥ r E ∥ 2 ≤ 1 . To estimate rewards, we run Algorithm 3 for 100 iterations, with α = 0.02 \alpha=0.02 α = 0.02 .
Results. Figure 5 reports the suboptimality attained by our methods when using either the log det ( I + ⋅ ) \log\det(I + \cdot) log det ( I + ⋅ ) , tr \operatorname{tr} tr , or ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 criterion for the active selection objective J J J defined in Equation (7) . In these experiments, all methods operate without a warm-up phase, meaning that dynamics are selected actively from the very first round. We additionally compare against the offline variant introduced in Section E.
Figure 5. Worst-case suboptimality over the training set P \mathcal{P} P as a function of the number of observed expert trajectories on randomly generated MDPs, and of the criterion J J J used to pick dynamics online (7) or offline (37) . Each plot corresponds to a different structure of P \mathcal{P} P .
We first observe that all methods achieve comparable performance when the dynamics set is uniformly distributed with respect to principal angles. Differences only emerge in the arbitrarily distributed setting, where the spectral norm criterion and the offline trace-based variant perform slightly worse, reaching approximately twice the suboptimality of the best-performing methods. Interestingly, the offline variant remains competitive in this setting.
Moreover, Table 3 shows that the log-determinant and trace criteria provide the strongest transfer performance to previously unseen dynamics. In contrast, the offline variants generally exhibit weaker transferability overall, although the trace-based offline criterion performs best among the three offline alternatives.
Table 3. Worst-case suboptimality on sets of held-out stochastic dynamics after learning from n = 100 n = 100 n = 100 expert trajectories collected on either uniformly or arbitrarily diverse sets of dynamics. Worst-case suboptimality (× 100 \times100 × 100 ) on held-out dynamics.
Figure 6 reports the worst-case suboptimality after n = 100 n = 100 n = 100 rounds as a function of the warm-up share n 0 / n n_0/n n 0 / n , on the arbitrarily diverse dynamics sets, using the log det ( I + ⋅ ) \log\det(I + \cdot) log det ( I + ⋅ ) criterion for the active phase. The two endpoints recover the Active only (n 0 = 0 n_0 = 0 n 0 = 0 ) and round-robin (n 0 = n n_0 = n n 0 = n ) selection rules. Suboptimality increases monotonically with the warm-up share. Actively selecting dynamics from the very first round is the best choice, and round-robin selection ends about three times worse. The degradation is however mild for short warm-ups: the share of 1 / 3 1/3 1/3 used in the main paper costs roughly 15% relative to no warm-up, and shares up to 1 / 2 1/2 1/2 remain within 30%. Most of the gap opens once the warm-up exceeds half of the budget, where the active phase no longer has enough rounds to rebalance the counts toward the most informative dynamics. The warm-up therefore acts as a cheap safeguard against poor early Hessian estimates rather than as a source of performance.
Figure 6. Worst-case suboptimality over the training set P \mathcal{P} P after n = 100 n = 100 n = 100 rounds, as the warmup share n 0 / n n_0/n n 0 / n increases. The vertical gray dashed line corresponds to the share of 1 / 3 1/3 1/3 used in the main paper.
Random mazes
Details. Mazes are l × l l \times l l × l grids with l = 7 l = 7 l = 7 , i.e., ∣ S ∣ = 49 |\mathcal{S}| = 49 ∣ S ∣ = 49 cells, with ∣ A ∣ = 5 |\mathcal{A}| = 5 ∣ A ∣ = 5 actions corresponding to four moves and a waiting action. Transitions are deterministic, a move into a wall leaves the agent in place, and the initial state is the top-left cell. For each seed, we draw three goal cells s 1 , s 2 , s 3 s_1, s_2, s_3 s 1 , s 2 , s 3 uniformly among the non-initial cells, set r E ( s j , a w a i t ) ∼ U [ 0 , 1 ] r^{\mathrm{E}}(s_j, a_{\mathrm{wait}}) \sim U[0, 1] r E ( s j , a wait ) ∼ U [ 0 , 1 ] , and r E ( s , a ) = 0 r^{\mathrm{E}}(s, a) = 0 r E ( s , a ) = 0 elsewhere. The reward vector is then rescaled to a uniformly random norm in [ 0 , 1 ] [0, 1] [ 0 , 1 ] , so that ∥ r E ∥ 2 ≤ 1 \lVert r^{\mathrm{E}}\rVert_2 \le1 ∥ r E ∥ 2 ≤ 1 .
A set P \mathcal{P} P of ∣ P ∣ |\mathcal{P}| ∣ P ∣ mazes is generated incrementally. The i i i -th maze (i = 0 , … , ∣ P ∣ − 1 i = 0, \ldots, |\mathcal{P}| - 1 i = 0 , … , ∣ P ∣ − 1 ) starts from the walls of the ( i − 1 ) (i - 1) ( i − 1 ) -th and is modified by a randomized wall-adding procedure that targets a shortest path from the initial state to the nearest goal of length ℓ i : = ( i ∣ P ∣ − 1 ) ℓ 2 \ell_i := \left(\frac{i}{|\mathcal{P}|-1}\right)\ell^2 ℓ i := ( ∣ P ∣ − 1 i ) ℓ 2 , accepted within ± 5 % \pm5\% ± 5% . Walls are proposed uniformly at random and kept only if they do not shorten this path, so the path length is non decreasing in i i i . The procedure stops after 15,000 proposals and returns the maze reached, so that ℓ i \ell_i ℓ i acts as a target rather than a constraint. In practice, for ∣ P ∣ = 100 |\mathcal{P}| = 100 ∣ P ∣ = 100 , shortest paths range from 4 to about 44 steps, well below a Hamiltonian path. For odd i i i , the first wall that disconnects every goal from the initial state is kept, yielding an unsolvable maze. Roughly a third of each set is unsolvable, so that rewards must be inferred partly from dynamics in which the goals are unreachable.
For reward estimation, we run Algorithm 3 with 100 inner iterations per round using SGD with step size 0.02.
Results. Beyond the results presented in the main paper, we additionally evaluate settings with dynamics sets of size ∣ P ∣ = 5 |\mathcal{P}| = 5 ∣ P ∣ = 5 and ∣ P ∣ = 20 |\mathcal{P}| = 20 ∣ P ∣ = 20 in order to study the impact of the number of candidate dynamics on the effectiveness of our active learning strategy.
Figure 7 reports both the suboptimality and reward estimation error achieved by our methods (using the log-determinant criterion), together with passive learning and ED [10 ] , for ∣ P ∣ ∈ { 5 , 20 , 100 } |\mathcal{P}| \in\{5, 20, 100\} ∣ P ∣ ∈ { 5 , 20 , 100 } . Across all values of ∣ P ∣ |\mathcal{P}| ∣ P ∣ , our active selection rule (with and without warm-up) consistently outperforms the competing approaches. This suggests that actively selecting dynamics improves sample efficiency even when the set of candidate dynamics is relatively small. However, the offline variant (see Section E) performs worse than passively selecting dynamics, hinting that in general, it is hard to entirely choose dynamics offline for expert demonstrations to learn the expert reward.
We also observe that the passive warm-up phase used in our two-phase method is not beneficial. In all tested settings, removing this warm-up phase leads to improved sample efficiency, with the gains becoming even more pronounced as the number of dynamics increases.
Figure 7. Worst-case suboptimality over the training set P \mathcal{P} P of randomly generated mazes, and the corresponding reward error, as a function of the number of observed expert trajectories, and of the size ∣ P ∣ |\mathcal{P}| ∣ P ∣ of the dynamics set.