Introduction
Latent Action Models (LAMs) have emerged as a promising paradigm for learning control representations from action-free video by inferring latent actions that explain observed state transitions [39 , 8 , 7 ] . Because latent-action learning captures transition and outcome structure rather than task optimality, it can naturally exploit large collections of diverse and suboptimal video, including failed trajectories that would be poor demonstrations but remain informative for learning how the environment changes. Yet existing LAM pipelines typically train the resulting latent policies with behavioral cloning (BC) [36 , 39 , 7 , 4 ] , without explicitly optimizing latent policies for improved task return. This creates a gap between the broad data that LAMs can learn from and the policies that current pipelines can extract from it.
Offline reinforcement learning (RL) offers a natural way to improve beyond the observed behavior [23 , 22 , 13 , 21 , 26 ] , but applying standard offline RL directly in a learned latent action space is not straightforward. We identify three properties that distinguish learned latent actions from physical actions. The first two concern realizability —whether the transition outcome represented by a latent action can be physically produced from the current state: (i) state-conditioned realizability , whereby a latent action realizable at one state can be unrealizable at another; and (ii) non-convex realizable set , whereby interpolating between realizable latent actions can yield unrealizable ones. The third, (iii) outcome-based abstraction , arises because distinct physical controls that reach the same outcome can map to the same latent action.
These properties expose two distinct failure modes of naive offline RL in latent action spaces. The first is an optimization–execution mismatch. Because the realizable latent set is state-dependent and potentially non-convex, standard offline RL can move outside this set by directly maximizing latent-action value or through unimodal policy fitting, producing latent actions that cannot be faithfully realized by physical execution. This raises the question: when does latent-space policy improvement transfer to decoded physical execution? The second is an objective mismatch. Because outcome-based abstraction can merge physical controls with different rewards, the optimal latent policy need not correspond to the optimal physical policy. This raises a separate question: when do the latent and physical policy optima align?
Figure 1. Overview of LAPIS. Left: a Latent Action Model learns latent actions from action-free trajectories and a decoder from limited action labels. Middle: naive offline RL can produce unrealizable latent actions through value maximization or unimodal policy fitting, while outcome-based abstraction can merge reward-distinct controls. Right: LAPIS performs support-aware policy improvement with critic-guided filtering and a multimodal flow policy, instantiated with an outcome-compatible goal-conditioned objective.
To answer these questions, we analyze paired physical- and latent-action batch MDPs constructed from the same observed transitions. For the first question, we show that batch-constrained latent policy improvement transfers to physical execution under deterministic dynamics and supported outcome identifiability. For the second, we show that latent and physical optima align under an outcome-determined objective within the decoder-covered, batch-constrained physical policy class. These results characterize when latent-space policy improvement remains valid and provide the principles behind our method.
Building on these insights, we introduce LAPIS (Latent Action Policy Improvement on Support), a latent policy improvement framework motivated by these failure modes. Rather than directly optimizing a latent actor toward arbitrary high-value actions, LAPIS grounds value-based policy improvement in observed state–latent transitions and uses a conditional flow policy to capture their potentially multimodal structure. LAPIS is instantiated with a goal-conditioned objective based on future states, which is naturally compatible with the outcome semantics captured by latent actions. At inference, classifier-free guidance provides a simple way to adjust the emphasis between behavior and high-value policy branches. With the same LAM and 5% action labels, LAPIS achieves the highest success among latent-space methods on all five OGBench visual manipulation tasks and outperforms prior semi-supervised offline RL baselines on nearly all tasks with only 5% or 1% action labels. Further experiments examine the contributions of these design choices.
Related Work
Latent action models from action-free video. A growing body of work learns controllable representations from action-free video by inferring latent actions that explain observed transitions (Edwards et al., 2019; Rybkin et al., [35 ] ; Baker et al., [3 ] ; Bruce et al., [4 ] ; Google DeepMind, [17 ] ). Recent approaches learn such representations using discrete or continuous latent codes and extend latent action modeling to richer video and vision-language-action settings (Schmidt & Jiang, [36 ] ; Ye et al., [39 ] ; Chen et al., [7 ] ; Gao et al., [14 ] ; Zhang et al., [40 ] ; Chen et al., [8 ] ). Their control pipelines predominantly train latent policies by behavioral cloning and use a small labeled set to decode latent actions into physical controls. LAPO (Schmidt & Jiang, [36 ] ) additionally incorporates reinforcement learning by applying online PPO to fine-tune the physical action policy after latent-action pretraining. In contrast, we study policy improvement directly in the learned latent action space using offline data, with scarce action labels used for physical action decoding.
Action-free and semi-supervised reinforcement learning. A related line of work studies learning control from trajectories with missing or limited action annotations, using inferred actions, latent representations, or other forms of auxiliary supervision to exploit action-free experience (Torabi et al., [37 ] ; Baker et al., [3 ] ; Ghosh et al., [16 ] ; Park et al., [29 ] ; Zheng et al., [42 ] ). Most closely related is LAQ (Chang et al., [6 ] ), which performs Q-learning over latent actions from state-only data but relies on a hand-specified controller for execution. In contrast, we learn a latent action space from action-free data together with a physical-action decoder from scarce action labels, and study offline policy improvement directly in this learned latent action space. This setting makes the relationship between latent-space optimization and decoded physical execution central, motivating our analysis of when latent policy improvement transfers to physical control. Further related work is discussed in Appendix § 9 .
Preliminaries
Goal-conditioned reinforcement learning We model the environment as an MDP M = ( S , A , p , r , γ ) \mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, \gamma) M = ( S , A , p , r , γ ) , with dynamics p ( s ′ ∣ s , a ) p(s' \mid s, a) p ( s ′ ∣ s , a ) , bounded reward r r r , and initial-state distribution μ 0 \mu_0 μ 0 . We write ( s , a , s ′ ) = ( s t , a t , s t + 1 ) (s, a, s') = (s_t, a_t, s_{t+1}) ( s , a , s ′ ) = ( s t , a t , s t + 1 ) for a transition. A policy π ( a ∣ s ) \pi(a \mid s) π ( a ∣ s ) induces the usual state-action and state value functions Q M π Q^\pi_{\mathcal{M}} Q M π and V M π V^\pi_{\mathcal{M}} V M π , with expected return J M ( π ) = E s 0 ∼ μ 0 [ V M π ( s 0 ) ] J_{\mathcal{M}}(\pi) = \mathbb{E}_{s_0 \sim\mu_0}[V^\pi_{\mathcal{M}}(s_0)] J M ( π ) = E s 0 ∼ μ 0 [ V M π ( s 0 )] . We refer to M \mathcal{M} M as the ground MDP to distinguish it from the batch MDPs introduced in § 5 . In the goal-conditioned setting, the policy additionally conditions on a goal g g g , π ( a ∣ s , g ) \pi(a \mid s, g) π ( a ∣ s , g ) . Contrastive RL (Eysenbach et al., [10 ] ) casts goal-conditioned control as density-ratio estimation, bypassing explicit modeling of transition dynamics or reward functions. The goal-reaching reward is ( 1 − γ ) p ( s ′ = g ∣ s , a ) (1-\gamma)p(s'=g \mid s,a) ( 1 − γ ) p ( s ′ = g ∣ s , a ) , and the corresponding goal-conditioned value equals the discounted future-state occupancy:
Q g π ( s , a ) = p π ( s t + k = g ∣ s , a ) = ( 1 − γ ) ∑ k = 1 ∞ γ k − 1 p π ( s t + k = g ∣ s t = s , a t = a ) , (1) Q^\pi_g(s,a) = p^\pi(s_{t+k}=g \mid s,a) = (1-\gamma)\sum_{k=1}^{\infty}\gamma^{k-1}p^\pi(s_{t+k}=g \mid s_t=s,a_t=a), \tag*{(1)} Q g π ( s , a ) = p π ( s t + k = g ∣ s , a ) = ( 1 − γ ) k = 1 ∑ ∞ γ k − 1 p π ( s t + k = g ∣ s t = s , a t = a ) , (1)
where the right-hand side denotes the discounted occupancy of future states visited from ( s , a ) (s,a) ( s , a ) under π \pi π . To illustrate the contrastive principle, consider a critic parameterized as a bilinear score f ϕ ( s , a , g ) = ϕ ( s , a ) T ψ ( g ) / d f_\phi(s,a,g) = \phi(s,a)^\mathsf{T}\psi(g)/\sqrt{d} f ϕ ( s , a , g ) = ϕ ( s , a ) T ψ ( g ) / d and trained by noise-contrastive binary classification, with a positive goal g + g^+ g + sampled from the future-state occupancy and a negative goal g − ∼ p B g^- \sim p_\mathcal{B} g − ∼ p B sampled from the dataset state distribution,
max ϕ , ψ E ( s , a ) , g + , g − [ log σ ( f ϕ ( s , a , g + ) ) + log ( 1 − σ ( f ϕ ( s , a , g − ) ) ) ] . (2) \max_{\phi,\psi}\mathbb{E}_{(s,a),g^+,g^-}\left[\log\sigma(f_\phi(s,a,g^+))+\log(1-\sigma(f_\phi(s,a,g^-)))\right]. \tag*{(2)} ϕ , ψ max E ( s , a ) , g + , g − [ log σ ( f ϕ ( s , a , g + )) + log ( 1 − σ ( f ϕ ( s , a , g − ))) ] . (2)
For the ideal binary classification problem, the optimal score is the log density ratio, f ∗ = log Q g π − log p B ( g ) f^* = \log Q^\pi_g - \log p_\mathcal{B}(g) f ∗ = log Q g π − log p B ( g ) wherever the ratio is defined. LAPIS uses a TD-InfoNCE estimator, detailed in Appendix § 13 .§ 3 . In the offline setting, directly maximizing this critic can drive the actor outside the support of the fixed behavior data, so offline CRL regularizes policy improvement with behavioral cloning,
L C R L ( π ) = E ( s , a , g ) ∼ D [ − f ϕ ( s , π ( s , g ) , g ) + α ∥ π ( s , g ) − a ∥ 2 2 ] , \mathcal{L}_{\mathrm{CRL}}(\pi) = \mathbb{E}_{(s,a,g)\sim\mathcal{D}}\left[-f_\phi(s,\pi(s,g),g)+\alpha\left\lVert\pi(s,g)-a\right\rVert_2^2\right], L CRL ( π ) = E ( s , a , g ) ∼ D [ − f ϕ ( s , π ( s , g ) , g ) + α ∥ π ( s , g ) − a ∥ 2 2 ] ,
improving the actor toward high-value actions while keeping it close to the observed behavior.
Latent action models A Latent Action Model (LAM) learns a latent action space Z Z Z from an action-free dataset B = { ( s i , s i ′ ) } i = 1 N \mathcal{B}=\{(s_i,s'_i)\}_{i=1}^{N} B = {( s i , s i ′ ) } i = 1 N by training an inverse dynamics model I I I and a forward dynamics model F F F , with z = I ( s , s ′ ) z=I(s,s') z = I ( s , s ′ ) and s ′ = F ( s , z ) s'=F(s,z) s ′ = F ( s , z ) with a bottlenecked latent representation,
min I , F E ( s , s ′ ) ∼ B [ ∥ s ′ − F ( s , I ( s , s ′ ) ) ∥ 2 2 ] . (3) \min_{I,F}\mathbb{E}_{(s,s')\sim\mathcal{B}}\left[\left\lVert s'-F(s,I(s,s'))\right\rVert_2^2\right]. \tag*{(3)} I , F min E ( s , s ′ ) ∼ B [ ∥ s ′ − F ( s , I ( s , s ′ )) ∥ 2 2 ] . (3)
Relabeling each transition by its latent action z i = I ( s i , s i ′ ) z_i = I(s_i, s'_i) z i = I ( s i , s i ′ ) yields the latent dataset B z = { ( s i , z i , s i ′ ) } i = 1 N \mathcal{B}^z = \{(s_i, z_i, s'_i)\}_{i=1}^{N} B z = {( s i , z i , s i ′ ) } i = 1 N . A separate small labeled dataset B d e c = { ( s i , a i , s i ′ ) } i = 1 N d e c \mathcal{B}^{\mathrm{dec}} = \{(s_i, a_i, s'_i)\}_{i=1}^{N_{\mathrm{dec}}} B dec = {( s i , a i , s i ′ ) } i = 1 N dec with N d e c ≪ N N_{\mathrm{dec}} \ll N N dec ≪ N supplies the action supervision for the decoder.
The deployed system composes a latent policy with a decoder . The latent policy π z ( z ∣ s ) \pi_z(z \mid s) π z ( z ∣ s ) selects a latent transition at the current state, and the state-marginal decoder Ψ ( a ∣ z ) \Psi(a \mid z) Ψ ( a ∣ z ) maps the chosen latent to a physical action, trained on B d e c \mathcal{B}^{\mathrm{dec}} B dec by max Ψ E B d e c [ log Ψ ( a ∣ z ) ] \max_{\Psi} \mathbb{E}_{\mathcal{B}^{\mathrm{dec}}}[\log\Psi(a \mid z)] max Ψ E B dec [ log Ψ ( a ∣ z )] and frozen thereafter. Deployment samples z ∼ π z ( ⋅ ∣ s ) z \sim\pi_z(\cdot\mid s) z ∼ π z ( ⋅ ∣ s ) then a ∼ Ψ ( ⋅ ∣ z ) a \sim\Psi(\cdot\mid z) a ∼ Ψ ( ⋅ ∣ z ) , giving the deployed policy π ˉ = Ψ ∘ π z \bar{\pi} = \Psi\circ\pi_z π ˉ = Ψ ∘ π z . Section 5 introduces an ideal state-conditioned batch decoder Ψ ∗ ( a ∣ s , z ) \Psi^*(a \mid s, z) Ψ ∗ ( a ∣ s , z ) only for the transfer analysis.
Latent Action Structure and Failures
A latent policy selects a latent action representing a desired transition outcome, which must ultimately be realized through physical control at deployment. Section 4.1 characterizes the realizability and abstraction structure of learned latent actions. Section 4.2 then shows how realizability can break the connection between latent-space optimization and physical execution, while outcome abstraction can misalign latent and physical objectives.
Structural properties of learned latent action spaces
To quantify whether a target latent can be physically realized from the current state, we define its realization error as the infimum of its distance to the latents of reachable outcomes. We use it as a diagnostic (Figure 2 , Appendix B) and to bound the transfer gap outside the data (Appendix D.9).
Figure 2. Structural properties of learned latent action spaces. Top row: concept schematics for (a) state-conditioned realizability (the same latent can be realizable at s 1 s_1 s 1 but not s 2 s_2 s 2 ), (b) the non-convex realizable set (interpolating two realizable latents can pass through unrealizable ones), and (c) outcome-based abstraction (distinct physical controls reaching the same successor share one latent). Bottom row: empirical diagnostics for the experimental LAM on visual-cube-single-play-v0; the diagnostic protocol is in Appendix § 10 .
Definition 1 (Latent realizability). Let R ( s ) : = ⋃ a ∈ A supp p ( ⋅ ∣ s , a ) R(s) := \bigcup_{a \in\mathcal{A}} \operatorname{supp} p(\cdot\mid s, a) R ( s ) := ⋃ a ∈ A supp p ( ⋅ ∣ s , a ) be the set of successor states reachable from s s s . The latent realization error of a target latent z z z at state s s s is
δ ( s , z ) = inf s ′ ∈ R ( s ) ∥ I ( s , s ′ ) − z ∥ 2 , \delta(s, z) = \inf_{s' \in R(s)} \lVert I(s, s') - z \rVert_2, δ ( s , z ) = s ′ ∈ R ( s ) inf ∥ I ( s , s ′ ) − z ∥ 2 , and z z z is ϵ \epsilon ϵ -realizable at s s s if δ ( s , z ) ≤ ϵ \delta(s, z) \le\epsilon δ ( s , z ) ≤ ϵ , defining the realizable set Z r e a l ϵ ( s ) = { z : δ ( s , z ) ≤ ϵ } \mathcal{Z}_{\mathrm{real}}^{\epsilon}(s) = \{z : \delta(s, z) \le\epsilon\} Z real ϵ ( s ) = { z : δ ( s , z ) ≤ ϵ } .
The observed set B z \mathcal{B}^z B z contains only a subset of realizable state–latent pairs; absence from B z \mathcal{B}^z B z does not imply unrealizability. Figure 2 illustrates the properties discussed below.
State-conditioned realizability Whether a latent is realizable depends on the current state: the same z z z can be realizable from s 1 s_1 s 1 but not s 2 s_2 s 2 , so realizability is a property of the pair ( s , z ) (s, z) ( s , z ) , not of z z z alone (Z r e a l ϵ ( s 1 ) ≠ Z r e a l ϵ ( s 2 ) \mathcal{Z}_{\mathrm{real}}^{\epsilon}(s_1) \ne\mathcal{Z}_{\mathrm{real}}^{\epsilon}(s_2) Z real ϵ ( s 1 ) = Z real ϵ ( s 2 ) ). In Figure 2(a), we take latents with known realizations at their source states and search for physical realizations of the same latents from other states. Latents have near-zero realization error at their source states, while 73% of transfers to other states exceed the diagnostic threshold under finite search. LAPO ([36 Sec. 4.1] ) notes that the action behind a latent can depend on the state; this is harmless when imitating the latents observed at s s s , but offline RL moves away from them and can select latents that are realizable only from other states.
Non-convex realizable set Even at a fixed state, the space between two realizable latents need not be realizable. For z 1 , z 2 ∈ Z r e a l ϵ ( s ) z_1, z_2 \in\mathcal{Z}_{\mathrm{real}}^{\epsilon}(s) z 1 , z 2 ∈ Z real ϵ ( s ) , an interpolation z α = ( 1 − α ) z 1 + α z 2 z_{\alpha} = (1-\alpha)z_1 + \alpha z_2 z α = ( 1 − α ) z 1 + α z 2 need not be realizable, so z α ∉ Z r e a l ϵ ( s ) z_{\alpha} \notin\mathcal{Z}_{\mathrm{real}}^{\epsilon}(s) z α ∈ / Z real ϵ ( s ) for α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) . In Figure 2 (b), we select pairs of realizable endpoint latents at the same state and search for physical realizations along the interpolation between them. The endpoints admit known physical realizations, while interior errors exceed the diagnostic threshold.
Outcome-based abstraction The encoder z = I ( s , s ′ ) z = I(s, s') z = I ( s , s ′ ) labels the transition outcome rather than the physical command, so distinct actions reaching the same successor from s s s share a latent. In Figure 2 (c), we search for substantially different action chunks that reach each target outcome within a fixed tolerance; 95% of sampled targets admit at least two such realizations.
Why the properties break naive offline RL
These properties become consequential when offline RL is applied in the learned latent space, giving rise to two distinct failure modes: an optimization–execution mismatch and an objective mismatch. Optimization–execution mismatch State-conditioned realizability and a non-convex realizable set imply that physically realizable latent actions depend on the current state, while interpolations between realizable latents can be unrealizable. As a result, directly optimizing the actor to maximize learned value can move it toward unrealizable latents despite behavior regularization, while a unimodal actor can place probability between separated realizable modes and generate unrealizable latents. Thus, even when the critic favors a high-value latent action z z z , if z z z is unrealizable, decoding and physical execution can reach a different successor than the outcome represented by z z z . Consequently, latent-space policy improvement need not translate to improvement after decoding,
J l a t e n t ( π z ′ ) ≥ J l a t e n t ( π z ) ⇏ J M ( Ψ ∘ π z ′ ) ≥ J M ( Ψ ∘ π z ) . \mathcal{J}_{\mathrm{latent}}(\pi'_z) \ge\mathcal{J}_{\mathrm{latent}}(\pi_z) \;\not\Rightarrow\; \mathcal{J}_{M}(\Psi\circ\pi'_z) \ge\mathcal{J}_{M}(\Psi\circ\pi_z). J latent ( π z ′ ) ≥ J latent ( π z ) ⇒ J M ( Ψ ∘ π z ′ ) ≥ J M ( Ψ ∘ π z ) .
Here J l a t e n t \mathcal{J}_{\mathrm{latent}} J latent denotes the learned latent-space optimization objective.
Objective mismatch Outcome-based abstraction can erase reward-relevant distinctions between physical controls. When distinct controls a 1 ≠ a 2 a_1 \ne a_2 a 1 = a 2 reach the same successor s ′ s' s ′ , they share the latent action z = I ( s , s ′ ) z = I(s,s') z = I ( s , s ′ ) , even if an action-dependent reward assigns r ( s , a 1 , s ′ ) ≠ r ( s , a 2 , s ′ ) r(s,a_1,s') \ne r(s,a_2,s') r ( s , a 1 , s ′ ) = r ( s , a 2 , s ′ ) . Because the latent policy can select only z z z , it cannot distinguish between these reward-distinct physical controls. Consequently, a policy that is optimal in the latent-action MDP need not decode to an optimal physical-action policy,
π z ∗ ∈ arg max π z ∈ Π B z J M B z ( π z ) ⇏ ( Ψ ∗ ∘ π z ∗ ) ∈ arg max π ∈ Π B a J M ( π ) . \pi_z^* \in\underset{\pi_z \in\Pi_{B_z}}{\arg\max}\; \mathcal{J}_{M_{B_z}}(\pi_z) \;\not\Rightarrow\; (\Psi^* \circ\pi_z^*) \in\underset{\pi\in\Pi_{B_a}}{\arg\max}\; \mathcal{J}_{M}(\pi). π z ∗ ∈ π z ∈ Π B z arg max J M B z ( π z ) ⇒ ( Ψ ∗ ∘ π z ∗ ) ∈ π ∈ Π B a arg max J M ( π ) .
Here, Π B a \Pi_{B_a} Π B a and Π B z \Pi_{B_z} Π B z denote the physical and latent policy classes restricted to observed state–action and state–latent pairs, respectively, and Ψ ∗ \Psi^* Ψ ∗ is the batch-exact decoder defined in § 5 .
These failure modes raise two questions: when does latent-space policy improvement transfer to physical execution, and when do latent and physical policy optima align? § 5 establishes conditions for both, which motivate the design of LAPIS in § 6 .
When Does Latent Policy Improvement Work?
To answer these questions, we construct paired batch MDPs from the same observed transitions under physical and latent action representations.
M B a = ( S ∪ { s a b s } , A , p B a , r B a , γ ) , M B z = ( S ∪ { s a b s } , Z , p B z , r B z , γ ) . \mathcal{M}_{B_a} = (S \cup\{s_{\mathrm{abs}}\}, A, p_{B_a}, r_{B_a}, \gamma), \qquad\mathcal{M}_{B_z} = (S \cup\{s_{\mathrm{abs}}\}, Z, p_{B_z}, r_{B_z}, \gamma). M B a = ( S ∪ { s abs } , A , p B a , r B a , γ ) , M B z = ( S ∪ { s abs } , Z , p B z , r B z , γ ) .
The two MDPs are constructed from B a = { ( s i , a i , s i ′ ) } i = 1 N \mathcal{B}^{a} = \{(s_i, a_i, s'_i)\}_{i=1}^{N} B a = {( s i , a i , s i ′ ) } i = 1 N and its latent relabeling B z = { ( s i , z i , s i ′ ) } i = 1 N \mathcal{B}^{z} = \{(s_i, z_i, s'_i)\}_{i=1}^{N} B z = {( s i , z i , s i ′ ) } i = 1 N , with z i = I ( s i , s i ′ ) z_i = I(s_i, s'_i) z i = I ( s i , s i ′ ) , while unobserved pairs transition to the zero-reward absorbing state s a b s s_{\mathrm{abs}} s abs . Because realizability is state-conditioned, support in M B z \mathcal{M}_{\mathcal{B}^{z}} M B z is over observed pairs ( s , z ) (s, z) ( s , z ) , not z z z globally. We further assume supported outcome identifiability : each supported ( s , z ) (s, z) ( s , z ) corresponds to a unique successor F B ( s , z ) F_{\mathcal{B}}(s, z) F B ( s , z ) , so that a supported latent action has a well-defined transition outcome at each state. This supported successor map is distinct from the learned forward predictor F F F . For the analysis, we deploy a latent policy through the batch-exact state-conditioned decoder Ψ ∗ ( a ∣ s , z ) \Psi^{*}(a \mid s, z) Ψ ∗ ( a ∣ s , z ) , which selects physical actions observed at s s s to produce the successor represented by z z z in the batch. The support domain, initial-state and terminal conventions, batch rewards, and decoder construction are specified in Appendix § 11 .§ 1 . All exact results below use this common construction. Appendix § 9 .§ 4 discusses how our analysis extends classical batch-constrained RL and MDP abstraction to learned latent actions.
When does latent policy improvement transfer?
We first ask when an improvement measured in the latent batch MDP is preserved by decoded physical execution. To avoid selecting unrealizable latent actions, we restrict the latent policy to state–latent transitions observed in the batch.
Definition 2 (Batch-constrained policy). A policy is batch-constrained if its trajectory uses only observed state–action pairs in B a \mathcal{B}^{a} B a . A latent batch-constrained policy is defined analogously over B z \mathcal{B}^{z} B z .
Every observed state–latent pair ( s , z ) ∈ B z (s, z) \in\mathcal{B}^{z} ( s , z ) ∈ B z has a physical realization by construction (δ ( s , z ) = 0 \delta(s, z) = 0 δ ( s , z ) = 0 in Definition 1), and the batch-exact decoder Ψ ∗ \Psi^{*} Ψ ∗ selects physical actions observed at s s s that realize this outcome. A batch-constrained latent policy selects only latent actions that are physically realizable at the current state.
Support alone, however, does not ensure exact transfer. Under stochastic dynamics, the same decoded action can reach a different successor from the one used to evaluate z z z in the latent batch MDP, breaking the correspondence between latent evaluation and physical execution. We therefore assume deterministic ground dynamics, which leads to Theorem 1.
Assumption 1 (Deterministic ground dynamics). The successor is a deterministic function of the current state and physical action, s ′ = T ( s , a ) s' = T(s, a) s ′ = T ( s , a ) .
Theorem 1 (On-support policy improvement transfer). Under Assumption 1, let π z \pi_z π z be a batch-constrained latent policy. Then the latent return equals the decoded physical return, J M B z ( π z ) = J M ( Ψ ∗ ∘ π z ) J_{\mathcal{M}_{\mathcal{B}^{z}}}(\pi_z) = J_{\mathcal{M}}(\Psi^{*} \circ\pi_z) J M B z ( π z ) = J M ( Ψ ∗ ∘ π z ) , and consequently, for any two such latent policies π z \pi_z π z and π z ′ \pi'_z π z ′ ,
J M B z ( π z ′ ) ≥ J M B z ( π z ) ⟹ J M ( Ψ ∗ ∘ π z ′ ) ≥ J M ( Ψ ∗ ∘ π z ) . J_{\mathcal{M}_{\mathcal{B}^{z}}}(\pi'_z) \ge J_{\mathcal{M}_{\mathcal{B}^{z}}}(\pi_z) \quad\Longrightarrow\quad J_{\mathcal{M}}(\Psi^{*} \circ\pi'_z) \ge J_{\mathcal{M}}(\Psi^{*} \circ\pi_z). J M B z ( π z ′ ) ≥ J M B z ( π z ) ⟹ J M ( Ψ ∗ ∘ π z ′ ) ≥ J M ( Ψ ∗ ∘ π z ) .
Proof. See Appendix § 11 .§ 2 .
Thus, return improvement between batch-constrained latent policies transfers to their batch-exact decoded policies. Appendix § 12 .§ 3 extends this exact result to stochastic dynamics, bounding the transfer gap by the mismatch between the latent batch transition and the transition induced by decoded execution. Appendix § 12 .9 bounds the gap outside the data via the realization error (Definition 1).
When do latent and physical optima align?
Even when latent policy improvement transfers to decoded execution as in Theorem 1, this alone does not guarantee alignment between the optimal latent and physical-action policies. Under a generic action-dependent reward, outcome-based abstraction can collapse reward-distinct physical controls into the same latent action, removing potentially necessary distinctions for physical optimality.
Assumption 2 (Outcome-determined reward). The reward depends on a transition only through the current and next states, r ( s , a , s ′ ) = r ( s , s ′ ) r(s, a, s') = r(s, s') r ( s , a , s ′ ) = r ( s , s ′ ) .
Under Assumption 2, physical actions reaching the same successor from the same state receive the same immediate reward, removing the reward ambiguity introduced by outcome-based abstraction. We therefore obtain the following alignment result.
Algorithm 1 LAPIS training and deployment
Require: transition batch B \mathcal{B} B , labeled set B d e c \mathcal{B}_{\mathrm{dec}} B dec , guidance scale w w w
Train IDM I ( s , s ′ ) I(s,s') I ( s , s ′ ) , FDM F ( s , z ) F(s,z) F ( s , z ) , and decoder Ψ ( a ∣ z ) \Psi(a \mid z) Ψ ( a ∣ z ) ; form B z \mathcal{B}^{z} B z
repeat
Score with f ϕ ( s , z , g ) f_{\phi}(s,z,g) f ϕ ( s , z , g ) and gate with G ( s , z , g ) G(s,z,g) G ( s , z , g ) in Eq. (4)
Fit v θ ( x t , t ; s , g , o ) v_{\theta}(x_t,t;s,g,o) v θ ( x t , t ; s , g , o ) by minimizing L a c t o r \mathcal{L}_{\mathrm{actor}} L actor in Eq. (5)
until convergence
Deploy: generate z z z with v w C F G v_{w}^{\mathrm{CFG}} v w CFG and execute a ∼ Ψ ( ⋅ ∣ z ) a \sim\Psi(\cdot\mid z) a ∼ Ψ ( ⋅ ∣ z )
Theorem 2 (Optimal-policy alignment). Under Assumptions 1–2, for the same initial distribution μ 0 \mu_0 μ 0 ,
π z ∗ ∈ arg max π z ∈ Π B z J M B z ( π z ) ⟹ ( Ψ ∗ ∘ π z ∗ ) ∈ arg max π ∈ Π B a J M ( π ) . \pi_z^* \in\operatorname*{arg\,max}_{\pi_z \in\Pi_{\mathcal{B}^z}} J_{\mathcal{M}_{\mathcal{B}^z}}(\pi_z) \quad\Longrightarrow\quad(\Psi^* \circ\pi_z^*) \in\operatorname*{arg\,max}_{\pi\in\Pi_{\mathcal{B}^a}} J_{\mathcal{M}}(\pi). π z ∗ ∈ π z ∈ Π B z arg max J M B z ( π z ) ⟹ ( Ψ ∗ ∘ π z ∗ ) ∈ π ∈ Π B a arg max J M ( π ) . Proof. See Appendix § 11 .§ 3 .
LAPIS: Policy Improvement on Support
Section 5 motivates batch-constrained latent policy improvement over supported state–latent transitions. LAPIS follows this principle by using critic-guided filtering to select high-value transitions from the observed state–latent pairs and a conditional flow policy to capture their potentially multimodal distribution. We instantiate LAPIS in goal-conditioned RL, using future states as goals to obtain a reward-label-free objective compatible with outcome-based latent actions. Appendix § 12 relates the practical algorithm to the supported-improvement analysis and explains how the learned components instantiate its principles.
Critic-guided transition filtering LAPIS fits a goal-conditioned contrastive critic f ϕ ( s , z , g ) f_{\phi}(s,z,g) f ϕ ( s , z , g ) for the behavior latent policy π B z \pi_{\mathcal{B}}^{z} π B z and uses it to score only the state–latent transitions observed in B z \mathcal{B}^{z} B z . We retain high-value transitions using a learned state–goal threshold,
G ( s , z , g ) : = 1 [ f ϕ ( s , z , g ) > V ^ ( s , g ) ] , (4) G(s,z,g) := \mathbf{1}\left[f_{\phi}(s,z,g) > \widehat{V}(s,g)\right], \tag*{(4)} G ( s , z , g ) := 1 [ f ϕ ( s , z , g ) > V ( s , g ) ] , (4)
where V ^ ( s , g ) \widehat{V}(s,g) V ( s , g ) is learned by expectile regression in the critic’s score space (Appendix § 13 .§ 3 ). We show in Appendix § 11 .§ 4 that, for the optimal density-ratio score f ∗ f^* f ∗ , critic-guided filtering corresponds to a supported reweighting that preserves support and improves return. For the TD-InfoNCE critic used in practice, this guarantee does not apply directly: its score is determined only up to an offset c ( s , z ) c(s,z) c ( s , z ) , which can change the ordering of latents (Appendix § 12 .§ 7 ). We therefore compare the raw score with an offset-free score in § 7.6 .
Multimodal policy extraction To capture the potentially multimodal distribution of gated high-value transitions, LAPIS uses conditional flow matching (Lipman et al. [25 ] ). For x 0 ∼ N ( 0 , I ) x_0 \sim\mathcal{N}(0,I) x 0 ∼ N ( 0 , I ) and target latent z z z , we use the linear path x t = ( 1 − t ) x 0 + t z x_t=(1-t)x_0+tz x t = ( 1 − t ) x 0 + t z with target velocity z − x 0 z-x_0 z − x 0 . The velocity model v θ ( x t , t ; s , g , o ) v_{\theta}(x_t,t;s,g,o) v θ ( x t , t ; s , g , o ) is trained with two branches: o = 0 o=0 o = 0 models all observed transitions, while o = 1 o=1 o = 1 models only the high-value transitions selected by G ( s , z , g ) G(s,z,g) G ( s , z , g ) ,
L a c t o r = E [ G ( s , z , g ) ∥ v θ ( x t , t ; s , g , 1 ) − ( z − x 0 ) ∥ 2 2 + λ ∥ v θ ( x t , t ; s , g , 0 ) − ( z − x 0 ) ∥ 2 2 ] . (5) \mathcal{L}_{\mathrm{actor}} = \mathbb{E}\left[G(s,z,g)\left\lVert v_{\theta}(x_t,t;s,g,1)-(z-x_0)\right\rVert_2^2 + \lambda\left\lVert v_{\theta}(x_t,t;s,g,0)-(z-x_0)\right\rVert_2^2\right]. \tag*{(5)} L actor = E [ G ( s , z , g ) ∥ v θ ( x t , t ; s , g , 1 ) − ( z − x 0 ) ∥ 2 2 + λ ∥ v θ ( x t , t ; s , g , 0 ) − ( z − x 0 ) ∥ 2 2 ] . (5)
At inference, classifier-free guidance (Ho & Salimans [20 ] ) combines the behavior and high-value branches,
v w C F G = ( 1 − w ) v θ ( ⋅ ; 0 ) + w v θ ( ⋅ ; 1 ) , v_w^{\mathrm{CFG}}=(1-w)v_{\theta}(\cdot;0)+wv_{\theta}(\cdot;1), v w CFG = ( 1 − w ) v θ ( ⋅ ; 0 ) + w v θ ( ⋅ ; 1 ) ,
where w = 0 w=0 w = 0 recovers the behavior branch, w = 1 w=1 w = 1 the high-value branch, and w > 1 w>1 w > 1 further emphasizes the high-value branch.
Experiments
Our experiments address six questions: (Q1) How does LAPIS perform in learned latent action spaces, and does standard goal-conditioned offline RL remain effective? (Q2) How does LAPIS compare with goal-conditioned baselines adapted to limited action supervision? (Q3) How important is it to keep latent policy improvement on supported state–latent transitions? (Q4) Which components of LAPIS drive its gains? (Q5) How does decoder conditioning affect performance across action-label budgets? (Q6) Does the offset in the TD-InfoNCE score change the gate’s decisions?
Benchmarks We evaluate on the OGBench visual suite ([30 ] ), covering cube and scene manipulation as well as antmaze navigation, and additionally on visual D4RL Franka Kitchen ([12 ] ; [18 ] ) for multi-subtask manipulation. All tasks are evaluated in a goal-conditioned offline RL setting with 64 × 64 64 \times64 64 × 64 RGB observations; example observations from OGBench and Kitchen are shown in Figure 5 and Figure 6 , respectively. We use x % x\% x % labels to denote a setting where x % x\% x % of the trajectories have action annotations while the remaining 100 − x % 100-x\% 100 − x % are available as action-free data, and evaluate at 100%, 5%, and 1% labels.
Figure 5. Example observations, rendered from our own environment instances, spanning the cube and scene manipulation families and the antmaze navigation family.
Figure 6. Example observations from visual Franka Kitchen. Rows show partial (top) and mixed (bottom); columns show steps t = 0 t = 0 t = 0 , 225, 450 of illustrative LAPIS rollouts with guidance w = 3 w = 3 w = 3 and reset seed 0. The displayed simulator states are rendered at 512 × 512 using the evaluation camera, while policy observations remain 64 × 64. These illustrative 450-step rollouts are separate from the 280-step D4RL evaluations reported in Table 2 .
Baselines and evaluation In Table 1 –Table 3 , magenta method names indicate the use of a LAM. We compare LAPIS against GCBC, flow-based GCBC (F-GCBC), GCIVL ([30 ] ), goal-conditioned IQL (GCIQL; [22 ] ), ORL ([38 ] ), HIQL ([29 ] ), and contrastive RL (CRL; [10 ] ) in Table 1 . Table 3 extends the comparison to goal-conditioned baselines adapted to limited action supervision, adding SS-ORL ([42 ] ) and VPT-RL ([3 ] ). For OGBench, we report success rates as mean ± std over four seeds, evaluating 20 episodes on each of five tasks per environment. All latent action models use a temporal gap of H = 3 H=3 H = 3 , with z t = I ( s t , s t + H ) z_t=I(s_t,s_{t+H}) z t = I ( s t , s t + H ) . The controller executes only the first decoded action before replanning, and Theorem 1′ in Appendix § 11 .§ 5 extends Theorem 1 to this setting. Detailed implementation settings for each algorithm are provided in Appendix § 13 .
Table 1. Success rate (%) on the OGBench visual suite. Gray columns use real actions with 100% labels; the others use the LAM with 5% labels. LAPIS{sup}† uses the same critic with IDQL-style selection ({xref}Appendix E.8`).
Table 3. Success rate (%) on the OGBench visual suite with 5% and 1% action labels. We compare LAPIS with goal-conditioned methods adapted to learning under limited physical-action supervision. With 1% labels, the LAM, decoder, and policy of LAPIS are all trained with 1% labels, unlike the decoder study in Table 5 .
Goal-conditioned offline RL in learned latent action spaces (Q1)
Table 1 compares goal-conditioned offline RL methods in the learned latent action space with 5% labels, with a 100%-label oracle as a reference; results without LAM pretraining are in Appendix § 13 .10. LAPIS is best on all five manipulation tasks and, compared with LAPIS trained on real actions with 100% labels, loses little performance on every task except scene. CRL maximizes the critic with a unimodal BC-regularized actor, so it can leave the observed latent modes for unrealizable high-value latents or average separated modes; LAPIS instead filters observed state–latent transitions and fits their multimodal distribution with a flow policy. On antmaze, LAM+HIQL exceeds LAPIS, as long-horizon navigation favors less conservative extraction than the LAPIS gate (Appendix § 14 .§ 1 ). With the same critic, IDQL-style selection (LAPIS{sup}†`) closes this gap on all four antmaze tasks.
Beyond OGBench: visual Franka Kitchen Table 2 : Normalized score (%) on visual D4RL Franka Kitchen with 100% and 5% action labels. We further evaluate on visual D4RL Franka Kitchen, a multi-subtask manipulation benchmark. Table 2 reports the percentage of required subtasks completed on the partial and mixed target-task sets; details are provided in Appendix § 13 .§ 1 . LAPIS also consistently outperforms GCIQL and CRL on both target-task sets with 100% and 5% action labels.
Table 2. Normalized score (%) on visual D4RL Franka Kitchen with 100% and 5% action labels.
Comparison under limited action supervision (Q2)
We next compare LAPIS with goal-conditioned baselines adapted to learning under limited physical-action supervision. These adaptations use hindsight goal relabeling, with implementation details provided in Appendix § 13 .§ 4 . Apart from antmaze-giant, where every method stays near zero at both label budgets, LAPIS is the strongest method with 5% labels, and with 1% labels it still succeeds while the competing methods largely collapse (Table 3 ). These results show that LAPIS can effectively exploit action-free trajectories even when physical action supervision is extremely limited.
Effect of supported policy improvement (Q3)
We examine whether successful latent policy improvement remains close to the behavior-supported latent region. Starting from behavior-like policies, we strengthen critic-driven improvement by decreasing CRL’s BC coefficient α \alpha α and increasing LAPIS guidance w w w . Figure 3 measures d B C d_{\mathrm{BC}} d BC , the distance from a policy latent to the nearest latent sampled from the BC policy at the same state–goal pair, and d r o u n d d_{\mathrm{round}} d round , the round-trip latent error obtained by decoding a selected latent into physical control, executing it in the environment, and measuring the distance between the original latent and the latent re-encoded from the resulting transition; lower values indicate more faithful physical realization. As CRL becomes more aggressive, both distances increase and task success collapses, whereas LAPIS remains closer to the behavior-supported region.
Figure 3. Support–success diagnostic (antmaze). We sweep CRL’s BC coefficient α \alpha α and LAPIS guidance w w w , measuring support distance d B C d_{\mathrm{BC}} d BC , realization round-trip error d r o u n d d_{\mathrm{round}} d round , and task success.
Which components drive LAPIS? (Q4)
LAPIS combines critic-guided filtering with a multimodal flow actor. To isolate their contributions, we perform a 2 × 2 2 \times2 2 × 2 ablation crossing the policy-improvement mechanism { max - Q , g a t e } \{\max\text{-}Q, \mathrm{gate}\} { max - Q , gate } and actor parameterization { d e t e r m i n i s t i c , f l o w } \{\mathrm{deterministic}, \mathrm{flow}\} { deterministic , flow } , while keeping the LAM, decoder, critic, and training budget fixed (Table 4 (a)). Critic-guided filtering consistently improves success with either actor parameterization, while the flow actor provides a further gain when combined with the gate.
Table 4: Success rate (%) with 5% action labels (mean ± \pm ± std over four seeds); bold marks the best value in each column of each panel. (a) Improvement × \times × actor. (b) Critic × \times × extraction rule with the same LAMs and decoders.
Table 4.
Extraction rule versus critic. Table 4 (b) crosses two critics (TD-InfoNCE, IQL) with four extraction rules (gate, AWR [32 ] , IDQL-style selection [19 ] , DDPG+BC [31 ] ); Appendix § 13 .§ 8 at fixed LAM, decoder, and labels. The critic matters most: under the same rule, switching to the IQL critic lowers success in 10 of the 12 comparable cells and leaves the other two similar. With the TD-InfoNCE critic, the gate is best on manipulation, whereas IDQL-style selection is best on antmaze-medium (LAPIS† in Table 1 ). DDPG+BC, which maximizes the critic directly, is among the lowest on manipulation, consistent with the max-Q Q Q rows of Table 4 (a).
Effect of decoder conditioning (Q5)
We compare state-marginal Ψ ( a ∣ z ) \Psi(a \mid z) Ψ ( a ∣ z ) and state-conditioned Ψ ( a ∣ s , z ) \Psi(a \mid s,z) Ψ ( a ∣ s , z ) decoders across label budgets, keeping the 5%-label LAM fixed and varying only decoder supervision (Table 5 ). With 5% and 1% action labels, the marginal decoder achieves higher success rates across all three domains. With full action supervision, the state-conditioned decoder performs better in every domain. Conditioning on the state closes the transfer gap in theory (Theorem 1), but in practice it pays off only when labels are plentiful. LAPO (Schmidt & Jiang [36 ] , Sec. 6.2) attributes the plateau of latent-only decoders to state dependence; with scarce labels, however, a state-conditioned decoder does not help, so LAPIS uses Ψ ( a ∣ z ) \Psi(a \mid z) Ψ ( a ∣ z ) .
Table 5: Task success (%) across decoder types and action-label budgets.
Table 5.
Does the score offset change the gate? (Q6)
The LAPIS gate (Equation (4) ) uses the raw TD-InfoNCE score f f f , which is determined only up to an offset c ( s , z ) c(s,z) c ( s , z ) . The offset-free score f ˉ \bar{f} f ˉ removes this offset but requires a log-partition estimate (Appendix § 12 .§ 7 ). To see whether the offset matters in practice, we compare the two scores on candidate latents in trained LAPIS critics (Table 6 ). They rank the candidates similarly, and gates on the two scores make the same keep/drop decision for about 90% of transitions. The raw score is thus a practical surrogate for f ˉ \bar{f} f ˉ .
Table 6: f f f versus f ˉ \bar{f} f ˉ in trained critics.
Table 6.
Conclusion
We study offline policy improvement in latent action spaces learned from action-free data. We show that state-conditioned realizability, non-convex realizable sets, and outcome-based abstraction can make standard offline RL unreliable, and characterize when latent improvement transfers to physical execution. In goal-conditioned RL, LAPIS performs support-aware policy improvement through critic-guided filtering over observed state–latent transitions and a multimodal flow policy. With only 5% action labels, LAPIS recovers much of the performance of its fully supervised counterpart and outperforms baselines on most tasks, especially in manipulation.
Extended Related Work
Offline RL in Learned Latent or Skill Spaces
Prior work has used learned latent or skill representations to constrain policy optimization to behavior-like controls. PLAS [43 ] learns a continuous latent action space with a conditional VAE and performs offline policy optimization through a behavior-trained decoder. Skill-based methods such as OPAL [1 ] , SPiRL [33 ] , and latent planning approaches [27 ] instead learn temporally extended action representations and optimize high-level policies over them. These approaches typically rely on action-labeled experience to learn the latent control representation or its low-level execution model. In contrast, we study offline policy improvement in a latent action space learned primarily from action-free trajectories, using scarce action labels only to ground latent decisions in physical control.
Action-Free and Semi-Supervised Reinforcement Learning
Learning control from trajectories with missing actions has been studied through imitation from observation [37 ] , action inference from video [3 ] , latent-intention learning [16 ] , hierarchical goal-conditioned learning [29 ] , and semi-supervised offline RL [42 ] . These approaches exploit action-free experience through inferred actions, state or goal abstractions, or value learning without complete action supervision. Our setting differs in that policy improvement is performed directly in a learned latent action space and physical actions are needed only for decoding. This makes the relationship between latent-space optimization and physical execution a central part of the problem.
Offline Goal-Conditioned and Contrastive RL
Offline goal-conditioned RL learns multi-goal policies from fixed datasets without additional environment interaction. Contrastive RL [10 ] formulates goal-conditioned control through density-ratio estimation of future-state occupancy, while subsequent work develops related contrastive objectives for offline control [41 ] . LAPIS adopts this goal-conditioned formulation in the learned latent-action space, using future states as goals and a contrastive critic to estimate which observed state–latent transitions are useful for reaching them.
Recent work has also explored expressive generative policies and guidance-based policy extraction [25 , 20 , 11 ] . Most closely related, CFGRL [11 ] derives classifier-free guidance as a controllable policy-improvement operator; in its value-free goal-conditioned formulation, guidance combines unconditional and goal-conditioned generative policy branches without an explicit value function. LAPIS shares guided flow-based extraction, but differs in where policy improvement is introduced. LAPIS first estimates goal-conditioned future-state occupancy with a TD-InfoNCE critic and filters only observed state–latent transitions to construct a high-value distribution whose support remains within the observed conditional latent support. The flow policy then models this support-aware target, and classifier-free guidance is applied only at inference.
This distinction is particularly important in learned latent-action spaces. Unlike physical actions, a latent action is a learned transition label whose usefulness depends on state-conditioned realizability and on whether decoding faithfully produces the represented outcome. Thus, improving a generative policy over latents does not by itself imply improved physical execution. Our transfer analysis characterizes when latent-space improvement survives decoding, while the direct comparison in Appendix § 13 .§ 5 shows that increasing goal guidance can improve physical-action CFGRL but does not provide consistent improvement when the same construction is applied directly in the learned latent space.
Relation to Batch-Constrained RL and MDP Abstraction
Our theoretical analysis is related to classical results in batch-constrained reinforcement learning and MDP abstraction, but addresses a different cross-representation question. Batch-constrained RL studies policy evaluation and improvement when the policy is restricted to actions represented in a fixed offline dataset, thereby avoiding extrapolation to unsupported physical actions [13 ] . In our setting, however, the policy acts in a learned action space: observed physical transitions are relabeled by an inverse model as latent actions, and each latent decision must subsequently be decoded back to physical control. Consequently, restricting the policy to observed latent actions is not sufficient by itself to establish correspondence with physical execution. The latent support is state-dependent, a supported pair ( s , z ) (s, z) ( s , z ) must identify the transition outcome represented by z z z , and the decoder must realize that outcome through physical actions.
Theorem 1 characterizes this additional structure. We construct physical- and latent-action batch MDPs from the same observed transitions and introduce supported outcome identifiability together with a batch-exact decoder that links a supported latent decision to its physical realizations. Under deterministic dynamics, this construction yields exact equality between the return of a batch-constrained latent policy and that of its decoded physical policy. Thus, the result extends the batch-constrained perspective from policy optimization within a fixed action space to policy improvement through a learned action representation whose semantics and physical execution must themselves be related.
Our optimality analysis is also connected to MDP homomorphism and abstraction theory, which studies when an abstract decision process preserves the value structure of the underlying MDP [34 , 24 ] . The latent actions considered here induce an outcome-based action abstraction learned from observed transitions: distinct physical controls can map to the same latent action whenever they produce the same represented outcome. This abstraction can preserve execution while discarding distinctions that matter for reward. Theorem 2 identifies the corresponding condition for optimality alignment: when reward is determined by the transition outcome, an optimal latent policy decodes to an optimal policy within the decoder-covered, batch-constrained physical policy class. The counterexample in Appendix § 11 .§ 3 shows that this condition is substantive: latent-to-physical return transfer can hold exactly even when latent and physical optima do not align.
Taken together, these results connect batch-constrained and abstraction-based reasoning to learned latent actions, where the representation is inferred from action-free transitions rather than given a priori. The additional questions are therefore not only whether a policy remains within dataset support or whether an abstraction preserves an MDP, but whether a learned latent label identifies a physically executable outcome and whether the distinctions removed by that representation are compatible with the downstream objective.
Empirical realizability diagnostics
The bottom-row diagnostics in Figure 2 evaluate the three structural properties discussed in § 4.1 on visual-cube-single-play-v0. All latent-space distances are computed in the standardized, quantized latent coordinates used throughout the experiments.
State-conditioned realizability. For Figure 2 (a), we select latents with known physical realizations at their source states and evaluate whether the same latents can be realized from other query states. For each state–latent pair, realization error is approximated by finite search over physical action chunks. The candidate set combines known realizing chunks, dataset chunks retrieved from nearby states and re-executed at the query state, and multi-start cross-entropy-method (CEM) refinements within the environment’s action bounds. Rows correspond to query states and columns to source latents.
Non-convex realizable set. For Figure 2 (b), we select pairs of separated latents that are both realizable from the same state and linearly interpolate between them. Realization error is evaluated on a uniform interpolation grid. The action chunks realizing both endpoints are included in the candidate set at every interpolation point, together with the same dataset-based and CEM search used above. Confidence bands are computed across latent pairs.
Diagnostic threshold. Panels (a–b) use a common threshold calibrated from two finite-search error samples. Let D s e l f D_{\mathrm{self}} D self contain errors for known-realizable source latents searched at their source states without supplying their original realizing chunks, and let D c r o s s D_{\mathrm{cross}} D cross contain errors for those latents searched from different states. Using 16 anchor states and CEM population 256 with 5 iterations, we set
ε = Q 0.9 ( D s e l f ) + Q 0.1 ( D c r o s s ) 2 ≈ 3.14 , \varepsilon= \frac{Q_{0.9}(D_{\mathrm{self}}) + Q_{0.1}(D_{\mathrm{cross}})}{2} \approx3.14, ε = 2 Q 0.9 ( D self ) + Q 0.1 ( D cross ) ≈ 3.14 ,
where Q p Q_p Q p denotes the empirical p p p -quantile. We use the rounded value 3.14 in both panels.
Outcome-based abstraction. For Figure 2 (c), we test whether substantially different physical action chunks can realize the same target outcome. For each target, targeted CEM search minimizes outcome distance, with restarts repelled from previously discovered chunks to encourage distinct realizations. Outcome distance is Euclidean in simulator generalized-position coordinates standardized using a candidate rollout pool at each anchor state, while action distance is Euclidean over flattened chunks. A candidate is retained when its outcome distance is at most τ o u t \tau_{\mathrm{out}} τ out and its distance from every previously retained chunk is at least τ a \tau_a τ a . We use 12 anchor states with 8 target outcomes each and 20 CEM restarts per target, all run without early stopping. At each anchor, τ o u t \tau_{\mathrm{out}} τ out is set to the 1st percentile of pairwise outcome distances and τ a \tau_a τ a to the 40th percentile of pairwise action-chunk distances. Each target’s observed source chunk is included as the first realization.
Search-budget robustness. Because realization error is estimated through finite action-space search, we additionally repeat the search with a substantially larger CEM budget. The default search uses population 256 and 5 iterations, while the enlarged search uses population 1024 and 12 iterations. We also evaluate a known-realizable floor by searching for source latents without providing their original realizing action chunks. This comparison tests whether the conclusions of panels (a–b) persist under substantially stronger search, without interpreting failure to find a realization as proof of global unrealizability. Results are summarized in Table 7 .
Table 7: Realization error under different search budgets (lower is better; ϵ = 3.14 \epsilon= 3.14 ϵ = 3.14 ). Interpolation denotes interior points in Figure 2 (b), and state swap denotes off-diagonal pairs in Figure 2 (a).
Table 7. Realization error under different search budgets (lower is better; ϵ = 3.14 \epsilon= 3.14 ϵ = 3.14 ). Interpolation denotes interior points in Figure 2 (b), and state swap denotes off-diagonal pairs in Figure 2 (a).
Proofs and theoretical derivations
We first formalize the paired physical and latent batch MDPs, then establish policy improvement transfer, optimal-policy alignment, and the supported reweighting guarantee underlying LAPIS. Appendix C.1 fixes notation and gives the exact construction of the paired physical and latent batch MDPs together with the batch-exact decoder Ψ ∗ \Psi^* Ψ ∗ . Appendix C.2 proves the on-support transfer result (Theorem 1). Appendix C.3 defines the decoder-covered policy class and proves optimal-policy alignment (Theorem 2). Appendix C.4 establishes the supported reweighting guarantee underlying LAPIS’s policy improvement principle. The correspondence between this analysis and practical LAPIS, together with the associated diagnostics, is discussed in Appendix D. We discuss the relationship of these results to classical batch-constrained RL and MDP abstraction theory in Appendix A.4.
Batch-MDP definitions and decoder
Throughout the analysis, we assume bounded rewards, R max : = sup s , a , s ′ ∣ r ( s , a , s ′ ) ∣ < ∞ R_{\max} := \sup_{s,a,s'} |r(s, a, s')| < \infty R m a x := sup s , a , s ′ ∣ r ( s , a , s ′ ) ∣ < ∞ , and a discount factor 0 ≤ γ < 1 0 \le\gamma< 1 0 ≤ γ < 1 . For a policy π \pi π in an MDP N \mathcal{N} N , we write J N ( π ) = E s 0 ∼ μ 0 [ V N ( s 0 ) ] J_{\mathcal{N}}(\pi) = \mathbb{E}_{s_0 \sim\mu_0}[V^{\mathcal{N}}(s_0)] J N ( π ) = E s 0 ∼ μ 0 [ V N ( s 0 )] , using the same initial-state distribution μ 0 \mu_0 μ 0 for all comparisons. For the analysis, B a \mathcal{B}^a B a pairs each observed transition with the physical action that generated it, whether or not that action label is available to the learning algorithm. The batch-MDP rewards are defined for the analysis; the learning algorithm does not require reward labels.
Support domain and termination. Let S B S_{\mathcal{B}} S B be the set of states and successors observed in the batch, and let N ( s , a , s ′ ) N(s, a, s') N ( s , a , s ′ ) denote the number of occurrences of ( s , a , s ′ ) (s, a, s') ( s , a , s ′ ) in B a \mathcal{B}^a B a . The observed physical and latent actions at state s s s are
A B ( s ) : = { a : ∑ s ′ N ( s , a , s ′ ) > 0 } , Z B ( s ) : = { I ( s , s ′ ) : ∑ a N ( s , a , s ′ ) > 0 } . A_{\mathcal{B}}(s) := \left\{a : \sum_{s'} N(s, a, s') > 0\right\}, \qquad Z_{\mathcal{B}}(s) := \left\{I(s, s') : \sum_a N(s, a, s') > 0\right\}. A B ( s ) := { a : s ′ ∑ N ( s , a , s ′ ) > 0 } , Z B ( s ) := { I ( s , s ′ ) : a ∑ N ( s , a , s ′ ) > 0 } .
We assume supp μ 0 ⊆ S B \operatorname{supp}\mu_0 \subseteq S_{\mathcal{B}} supp μ 0 ⊆ S B and consider a nonempty class of policies that remain on observed pairs until true termination. True terminal states have zero-reward absorbing self-loops in both the physical and latent processes. A trajectory cutoff is not treated as termination: a policy that reaches a nonterminal state with no observed continuation is excluded from the comparison class. All exact comparisons concern states reachable from μ 0 \mu_0 μ 0 under the policies being compared.
Both batch MDPs are defined on S ∪ { s a b s } S \cup\{s_{\mathrm{abs}}\} S ∪ { s abs } , with unobserved pairs routed to the zero-reward absorbing state s a b s s_{\mathrm{abs}} s abs . Starting from S B S_{\mathcal{B}} S B , their transitions remain within the finite set S B ∪ { s a b s } S_{\mathcal{B}} \cup\{s_{\mathrm{abs}}\} S B ∪ { s abs } , even when the ground state space is continuous. For a fixed policy, the Bellman operator is a γ \gamma γ -contraction on bounded functions and has a unique fixed point. Sums in the exact analysis are over finite empirical support; in the continuous-kernel extension of Appendix D.3, they are interpreted as integrals.
Definition 3 (Real action batch MDP). The physical-action batch MDP M B a = ( S ∪ { s a b s } , A , p B a , r B a , γ ) \mathcal{M}_{\mathcal{B}^a} = (S \cup\{s_{\mathrm{abs}}\}, A, p_{\mathcal{B}^a}, r_{\mathcal{B}^a}, \gamma) M B a = ( S ∪ { s abs } , A , p B a , r B a , γ ) uses the empirical transition probabilities
p B a ( s ′ ∣ s , a ) = N ( s , a , s ′ ) ∑ s ~ N ( s , a , s ~ ) p_{\mathcal{B}^a}(s' \mid s,a) = \frac{N(s,a,s')}{\sum_{\widetilde{s}}N(s,a,\widetilde{s})} p B a ( s ′ ∣ s , a ) = ∑ s N ( s , a , s ) N ( s , a , s ′ ) for a ∈ A B ( s ) a \in A_{\mathcal{B}}(s) a ∈ A B ( s ) .
On observed transitions , r B a ( s , a , s ′ ) = r ( s , a , s ′ ) r_{\mathcal{B}^a}(s,a,s') = r(s,a,s') r B a ( s , a , s ′ ) = r ( s , a , s ′ ) . For unobserved pairs, p B a ( s a b s ∣ s , a ) = 1 p_{\mathcal{B}^a}(s_{\mathrm{abs}} \mid s,a) = 1 p B a ( s abs ∣ s , a ) = 1 . The sink is absorbing, and all transitions into or within it receive zero reward. Consequently, ∥ V M B a π ∥ ∞ ≤ R max / ( 1 − γ ) \lVert V^{\pi}_{\mathcal{M}_{\mathcal{B}^a}}\rVert_{\infty} \le R_{\max}/(1-\gamma) ∥ V M B a π ∥ ∞ ≤ R m a x / ( 1 − γ ) for every policy π \pi π .
Definition 4 (Supported outcome identifiability). At a fixed state s s s , any two observed successors s 1 ′ , s 2 ′ s'_1,s'_2 s 1 ′ , s 2 ′ satisfy
I ( s , s 1 ′ ) = I ( s , s 2 ′ ) ⟹ s 1 ′ = s 2 ′ . I(s,s'_1) = I(s,s'_2) \quad\Longrightarrow\quad s'_1 = s'_2. I ( s , s 1 ′ ) = I ( s , s 2 ′ ) ⟹ s 1 ′ = s 2 ′ . Thus, each z ∈ Z B ( s ) z \in Z_{\mathcal{B}}(s) z ∈ Z B ( s ) identifies a unique observed successor, denoted F B ( s , z ) F_{\mathcal{B}}(s,z) F B ( s , z ) .
This condition applies only to observed successors at the same state. It allows distinct physical actions to share a latent when they reach the same successor, but excludes different observed successors from sharing a latent. The map F B F_{\mathcal{B}} F B is an exact lookup on the supported transition graph, not the learned forward predictor F F F in Equation (3) . Its definition does not require exact reconstruction by F F F , and the analysis evaluates F B F_{\mathcal{B}} F B only at supported pairs.
Definition 5 (Latent action batch MDP). Relabel each observed transition with z i = I ( s i , s i ′ ) z_i = I(s_i,s'_i) z i = I ( s i , s i ′ ) to obtain B z = { ( s i , z i , s i ′ ) } i = 1 n \mathcal{B}^z = \{(s_i,z_i,s'_i)\}_{i=1}^n B z = {( s i , z i , s i ′ ) } i = 1 n , and let N z ( s , z , s ′ ) N^z(s,z,s') N z ( s , z , s ′ ) denote the relabeled transition counts. Under Definition 4, the latent-action batch MDP M B z = ( S ∪ { s a b s } , Z , p B z , r B z , γ ) \mathcal{M}_{\mathcal{B}^z} = (S \cup\{s_{\mathrm{abs}}\}, Z, p_{\mathcal{B}^z}, r_{\mathcal{B}^z}, \gamma) M B z = ( S ∪ { s abs } , Z , p B z , r B z , γ ) has, for z ∈ Z B ( s ) z \in Z_{\mathcal{B}}(s) z ∈ Z B ( s ) ,
p B z ( ⋅ ∣ s , z ) = δ F B ( s , z ) , p_{\mathcal{B}^z}(\cdot\mid s,z) = \delta_{F_{\mathcal{B}}(s,z)}, p B z ( ⋅ ∣ s , z ) = δ F B ( s , z ) , r B z ( s , z ) = ∑ a N ( s , a , F B ( s , z ) ) r ( s , a , F B ( s , z ) ) ∑ a N ( s , a , F B ( s , z ) ) . r_{\mathcal{B}^z}(s,z) = \frac{\sum_a N(s,a,F_{\mathcal{B}}(s,z))r(s,a,F_{\mathcal{B}}(s,z))}{\sum_a N(s,a,F_{\mathcal{B}}(s,z))}. r B z ( s , z ) = ∑ a N ( s , a , F B ( s , z )) ∑ a N ( s , a , F B ( s , z )) r ( s , a , F B ( s , z )) . The transition kernel is equivalently obtained by normalizing the relabeled counts. The reward averages the physical rewards associated with the represented successor, using their empirical frequencies. For unsupported pairs, p B z ( s a b s ∣ s , z ) = 1 p_{\mathcal{B}^z}(s_{\mathrm{abs}} \mid s,z) = 1 p B z ( s abs ∣ s , z ) = 1 and r B z ( s , z ) = 0 r_{\mathcal{B}^z}(s,z) = 0 r B z ( s , z ) = 0 , with the same absorbing-sink convention as in M B a \mathcal{M}_{\mathcal{B}^a} M B a . For every latent policy π z \pi_z π z , ∥ V M B z π z ∥ ∞ ≤ R max / ( 1 − γ ) \lVert V^{\pi_z}_{\mathcal{M}_{\mathcal{B}^z}}\rVert_{\infty} \le R_{\max}/(1-\gamma) ∥ V M B z π z ∥ ∞ ≤ R m a x / ( 1 − γ ) .
Batch-exact decoder. For a supported pair ( s , z ) (s,z) ( s , z ) , write s z ′ = F B ( s , z ) s'_z = F_{\mathcal{B}}(s,z) s z ′ = F B ( s , z ) . The batch-exact decoder selects physical actions according to their observed frequencies among transitions from s s s to s z ′ s'_z s z ′ :
Ψ ∗ ( a ∣ s , z ) = N ( s , a , s z ′ ) ∑ b N ( s , b , s z ′ ) = π B , B ( a ∣ s ) p B a ( s z ′ ∣ s , a ) ∑ b π B , B ( b ∣ s ) p B a ( s z ′ ∣ s , b ) (6) \Psi^*(a \mid s,z) = \frac{N(s,a,s'_z)}{\sum_b N(s,b,s'_z)} = \frac{\pi_{\mathcal{B},\mathcal{B}}(a \mid s)p_{\mathcal{B}^a}(s'_z \mid s,a)}{\sum_b \pi_{\mathcal{B},\mathcal{B}}(b \mid s)p_{\mathcal{B}^a}(s'_z \mid s,b)} \tag*{(6)} Ψ ∗ ( a ∣ s , z ) = ∑ b N ( s , b , s z ′ ) N ( s , a , s z ′ ) = ∑ b π B , B ( b ∣ s ) p B a ( s z ′ ∣ s , b ) π B , B ( a ∣ s ) p B a ( s z ′ ∣ s , a ) (6)
Here, π B , B ( a ∣ s ) = ∑ s ′ N ( s , a , s ′ ) / ∑ b , s ′ N ( s , b , s ′ ) \pi_{\mathcal{B},\mathcal{B}}(a \mid s) = \sum_{s'}N(s,a,s')/\sum_{b,s'}N(s,b,s') π B , B ( a ∣ s ) = ∑ s ′ N ( s , a , s ′ ) / ∑ b , s ′ N ( s , b , s ′ ) is the empirical action distribution at s s s . The denominators in Equation (6) are positive on the supported domain.
(6) is an empirical Bayes decoder: it conditions the batch action distribution on the represented successor using p B a p_{\mathcal{B}^a} p B a , rather than the ground transition kernel p p p . Under deterministic ground dynamics, these kernels agree on observed state–action pairs, so the decoded actions reproduce the represented successor. Under stochastic dynamics, an action observed to reach s z ′ s'_z s z ′ may reach a different successor when executed again. This analytical decoder Ψ ∗ ( a ∣ s , z ) \Psi^*(a \mid s,z) Ψ ∗ ( a ∣ s , z ) is distinct from the learned state-marginal decoder Ψ ( a ∣ z ) \Psi(a \mid z) Ψ ( a ∣ z ) used by LAPIS; their relationship is discussed in Appendix § 12 .§ 4 .
Illustrative example. Figure 4 illustrates how the same observed transitions define the physical- and latent-action batch MDPs. The physical-action batch MDP retains the empirical successor distribution of each observed action and routes the unobserved action a 3 a_3 a 3 to the absorbing sink, even though it is valid in the ground MDP. Relabeling by z = I ( s , s ′ ) z = I(s,s') z = I ( s , s ′ ) separates the two observed successors of a 1 a_1 a 1 into distinct latents z 1 z_1 z 1 and z 2 z_2 z 2 , while assigning the transitions of both a 1 a_1 a 1 and a 2 a_2 a 2 to s 2 s_2 s 2 the same latent z 2 z_2 z 2 . The batch-exact decoder selects physical actions according to their frequencies for that successor: the three a 1 a_1 a 1 -transitions and five a 2 a_2 a 2 -transitions to s 2 s_2 s 2 give decoding probabilities 3 / 8 3/8 3/8 and 5 / 8 5/8 5/8 , respectively. Because a 1 a_1 a 1 is stochastic, decoding z 2 z_2 z 2 need not reproduce s 2 s_2 s 2 , despite its unique successor in the latent batch MDP. This distinction motivates the deterministic-dynamics assumption used for exact transfer in the following subsection.
Figure 4. Paired real-action and latent-action batch MDPs. (a) A real action in the ground MDP can induce multiple outcomes. (b,c) The finite dataset defines an empirical real-action batch MDP, with unseen actions routed to an absorbing sink. (d) Relabeling transitions by z = I ( s , s ′ ) z = I(s, s') z = I ( s , s ′ ) separates distinct outcomes while merging different real actions that reach the same successor.
Proof of Theorem 1
We first establish that batch constraints are preserved by decoding: selecting only observed latent actions at a state also restricts physical execution to actions observed at that state.
Under deterministic dynamics, these decoded actions also reproduce the successor represented by the selected latent. Together with the batch reward construction, this correspondence yields equality between latent and decoded physical returns.
Theorem 1 (On-support policy improvement transfer). Under Assumption 1, every batch-constrained latent policy π z \pi_z π z satisfies J M B z ( π z ) = J M ( Ψ ∗ ∘ π z ) J_{M_B^z}(\pi_z)=J_M(\Psi^*\circ\pi_z) J M B z ( π z ) = J M ( Ψ ∗ ∘ π z ) . Consequently, return improvement between two such latent policies transfers to their batch-exact decoded physical policies.
Proof. Fix a supported pair ( s , z ) (s,z) ( s , z ) and write s z ′ = F B ( s , z ) s'_z=F_B(s,z) s z ′ = F B ( s , z ) . Every action selected by Ψ ∗ ( ⋅ ∣ s , z ) \Psi^*(\cdot\mid s,z) Ψ ∗ ( ⋅ ∣ s , z ) was observed to produce s z ′ s'_z s z ′ from s s s . Under Assumption 1, each such action satisfies T ( s , a ) = s z ′ T(s,a)=s'_z T ( s , a ) = s z ′ , giving
∑ a Ψ ∗ ( a ∣ s , z ) p ( ⋅ ∣ s , a ) = δ s z ′ = p B z ( ⋅ ∣ s , z ) . \sum_a \Psi^*(a \mid s,z)p(\cdot\mid s,a)=\delta_{s'_z}=p_{B^z}(\cdot\mid s,z). a ∑ Ψ ∗ ( a ∣ s , z ) p ( ⋅ ∣ s , a ) = δ s z ′ = p B z ( ⋅ ∣ s , z ) . The decoder weights also match those used to define the latent reward:
∑ a Ψ ∗ ( a ∣ s , z ) r ( s , a , s z ′ ) = ∑ a N ( s , a , s z ′ ) r ( s , a , s z ′ ) ∑ a N ( s , a , s z ′ ) = r B z ( s , z ) . \sum_a \Psi^*(a \mid s,z)r(s,a,s'_z)=\frac{\sum_a N(s,a,s'_z)r(s,a,s'_z)}{\sum_a N(s,a,s'_z)}=r_{B^z}(s,z). a ∑ Ψ ∗ ( a ∣ s , z ) r ( s , a , s z ′ ) = ∑ a N ( s , a , s z ′ ) ∑ a N ( s , a , s z ′ ) r ( s , a , s z ′ ) = r B z ( s , z ) . Thus, the one-step transition and expected reward agree for every supported pair. Individual decoded actions may receive different rewards, but their expectation matches the latent reward; Assumption 2 is not required.
Starting from the common initial distribution μ 0 \mu_0 μ 0 , the latent policy selects only supported latents, and Remark 1 ensures that decoding selects observed physical actions. The transition identity above then implies, by induction, that the two processes have the same state distribution at every step. The continuation and terminal conventions in Appendix § 11 .§ 1 ensure that both processes remain well-defined on this common reachable support.
Averaging over π z \pi_z π z , the two processes therefore share the state-level transition kernel and expected reward
P π z ( ⋅ ∣ s ) = ∑ z π z ( z ∣ s ) δ F B ( s , z ) , r ˉ π z ( s ) = ∑ z π z ( z ∣ s ) r B z ( s , z ) . P_{\pi_z}(\cdot\mid s)=\sum_z \pi_z(z \mid s)\delta_{F_B(s,z)},\qquad\bar r_{\pi_z}(s)=\sum_z \pi_z(z \mid s)r_{B^z}(s,z). P π z ( ⋅ ∣ s ) = z ∑ π z ( z ∣ s ) δ F B ( s , z ) , r ˉ π z ( s ) = z ∑ π z ( z ∣ s ) r B z ( s , z ) . Both V M B z π z V_{M_B^z}^{\pi_z} V M B z π z and V M Ψ ∗ ∘ π z V_M^{\Psi^*\circ\pi_z} V M Ψ ∗ ∘ π z satisfy
V = r ˉ π z + γ P π z V V=\bar r_{\pi_z}+\gamma P_{\pi_z}V V = r ˉ π z + γ P π z V on the common reachable support. Since γ < 1 \gamma<1 γ < 1 , this Bellman equation has a unique bounded solution, so the two value functions agree there. Averaging over μ 0 \mu_0 μ 0 establishes return equality. Applying this equality separately to π z \pi_z π z and π z ′ \pi'_z π z ′ proves the improvement implication.
Proof of Theorem 2 and the decoder-covered policy class
Theorem 1 establishes return equality between a latent policy and its own decoded physical policy. This does not yet imply that an optimal latent policy attains the best return available to batch-constrained physical policies.
To see why, consider a deterministic one-step problem with two equally frequent observed actions satisfying T ( s , a 1 ) = T ( s , a 2 ) = s t e r m T(s,a_1)=T(s,a_2)=s_{\mathrm{term}} T ( s , a 1 ) = T ( s , a 2 ) = s term , but receiving rewards 1 and 0, respectively. Both actions share one latent, and the batch-exact decoder selects each with probability 1 / 2 1/2 1/2 . The latent and decoded returns are therefore both 1 / 2 1/2 1/2 , consistent with Theorem 1. A physical policy can instead select a 1 a_1 a 1 and obtain return 1. Thus, outcome-based abstraction can create an optimality gap even without off-support actions or decoder error.
To establish alignment, we construct a value-equivalent latent counterpart for each physical policy in the comparison class. We first define the required coverage in terms of represented successors.
Definition 6 (Decoder-covered policy). Under deterministic dynamics, a physical policy π \pi π is decoder-covered from μ 0 \mu_0 μ 0 if, at every nonterminal state reachable under π \pi π from μ 0 \mu_0 μ 0 , each action in its support satisfies
T ( s , a ) = F B ( s , z ) for some z ∈ Z B ( s ) . T(s,a) = F_B(s,z) \quad\text{for some } z \in Z_B(s). T ( s , a ) = F B ( s , z ) for some z ∈ Z B ( s ) .
This condition requires the successor itself to be represented in the batch, rather than merely requiring its code I ( s , T ( s , a ) ) I(s,T(s,a)) I ( s , T ( s , a )) to have appeared. Under Assumption 1, every batch-constrained physical policy is decoder-covered because each observed action reaches its recorded successor. Coverage can also include unobserved actions that reach the same successors as observed actions, although Theorem 2 compares only the batch-constrained subclass.
Under Assumption 2, actions reaching the same successor from a given state also receive the same reward. We can therefore group a decoder-covered policy’s action probabilities by successor without changing its return.
Lemma 1 (Decoder-covered lift). Under Assumptions 1–2, a decoder-covered physical policy π \pi π has the latent lift
π z ( z ∣ s ) = ∑ a π ( a ∣ s ) 1 [ T ( s , a ) = F B ( s , z ) ] , z ∈ Z B ( s ) , \pi_z(z \mid s) = \sum_a \pi(a \mid s)1[T(s,a) = F_B(s,z)], \qquad z \in Z_B(s), π z ( z ∣ s ) = a ∑ π ( a ∣ s ) 1 [ T ( s , a ) = F B ( s , z )] , z ∈ Z B ( s ) , on its reachable support. This lift is batch-constrained and satisfies J M ( π ) = J M ( Ψ ∗ ∘ π z ) = J M B z ( π z ) J_M(\pi) = J_M(\Psi^* \circ\pi_z) = J_{M_{B_z}}(\pi_z) J M ( π ) = J M ( Ψ ∗ ∘ π z ) = J M B z ( π z ) . The lift may be extended arbitrarily outside the reachable domain. For a continuous physical-action policy, the sum over a a a is interpreted as an integral; the supported latent set remains finite.
Proof. Decoder coverage ensures that every selected action reaches a represented successor. Since the encoder is deterministic and supported outcome identifiability holds, each action is assigned to exactly one supported latent. The displayed probabilities therefore sum to one and define a policy supported on Z B ( s ) Z_B(s) Z B ( s ) .
By construction, at every reachable nonterminal state,
∑ z π z ( z ∣ s ) δ F B ( s , z ) = ∑ a π ( a ∣ s ) δ T ( s , a ) . \sum_z \pi_z(z \mid s)\delta_{F_B(s,z)} = \sum_a \pi(a \mid s)\delta_{T(s,a)}. z ∑ π z ( z ∣ s ) δ F B ( s , z ) = a ∑ π ( a ∣ s ) δ T ( s , a ) .
The left-hand side is also the successor distribution induced by Ψ ∗ ∘ π z \Psi^* \circ\pi_z Ψ ∗ ∘ π z , as established in the proof of Theorem 1. Thus, π \pi π and its lifted-and-decoded policy have the same successor distribution. Under Assumption 2, they also have the same expected reward.
Starting from μ 0 \mu_0 μ 0 , these identities imply that the two policies induce the same state distributions at every step. Their value functions satisfy the same Bellman equation on the common reachable support and hence agree there, giving
J M ( π ) = J M ( Ψ ∗ ∘ π z ) . J_M(\pi) = J_M(\Psi^* \circ\pi_z). J M ( π ) = J M ( Ψ ∗ ∘ π z ) .
The lift selects only observed latents along this reachable support and satisfies the continuation convention of Appendix § 11 .§ 1 , so it is batch-constrained. Theorem 1 then gives the remaining equality with J M B z ( π z ) J_{M_{B_z}}(\pi_z) J M B z ( π z ) . □ \square □
Every batch-constrained physical policy therefore has a latent counterpart with the same return. Conversely, decoding a batch-constrained latent policy preserves its return and produces a batch-constrained physical policy. These two correspondences establish optimal-policy alignment.
Theorem 2 (Optimal-policy alignment). Under Assumptions 1–2, for the same initial distribution μ 0 \mu_0 μ 0 ,
π z ∗ ∈ arg max π z ∈ Π B z J M B z ( π z ) ⟹ ( Ψ ∗ ∘ π z ∗ ) ∈ arg max π ∈ Π B a J M ( π ) . \pi_z^* \in\underset{\pi_z \in\Pi_{B_z}}{\arg\max}\; J_{M_{B_z}}(\pi_z)
\quad\Longrightarrow\quad
(\Psi^* \circ\pi_z^*) \in\underset{\pi\in\Pi_{B_a}}{\arg\max}\; J_M(\pi). π z ∗ ∈ π z ∈ Π B z arg max J M B z ( π z ) ⟹ ( Ψ ∗ ∘ π z ∗ ) ∈ π ∈ Π B a arg max J M ( π ) .
Proof. Let π z ∗ \pi_z^* π z ∗ maximize J M B z J_{M_{\mathcal{B}_z}} J M B z over Π B z \Pi_{\mathcal{B}_z} Π B z . For any π ∈ Π B a \pi\in\Pi_{\mathcal{B}_a} π ∈ Π B a , decoder coverage follows from Assumption 1, and Lemma 1 provides a lift π ~ z ∈ Π B z \tilde{\pi}_z \in\Pi_{\mathcal{B}_z} π ~ z ∈ Π B z with the same return. Consequently,
J M ( π ) = J M B z ( π ~ z ) ≤ J M B z ( π z ∗ ) = J M ( Ψ ∗ ∘ π z ∗ ) . J_M(\pi) = J_{M_{\mathcal{B}_z}}(\tilde{\pi}_z) \leq J_{M_{\mathcal{B}_z}}(\pi_z^*) = J_M(\Psi^* \circ\pi_z^*). J M ( π ) = J M B z ( π ~ z ) ≤ J M B z ( π z ∗ ) = J M ( Ψ ∗ ∘ π z ∗ ) .
where the inequality follows from latent optimality and the last equality follows from Theorem 1.
Moreover, ( Ψ ∗ ∘ π z ∗ ) ∈ Π B a (\Psi^* \circ\pi_z^*) \in\Pi_{\mathcal{B}_a} ( Ψ ∗ ∘ π z ∗ ) ∈ Π B a by Remark 1 and the reachable-support argument in Theorem 1. The decoded policy therefore belongs to the physical comparison class and attains at least as much return as every policy in that class, proving the claimed optimality. In particular,
J M ( Ψ ∗ ∘ π z ∗ ) = max π ∈ Π B a J M ( π ) = J M B z ( π z ∗ ) . J_M(\Psi^* \circ\pi_z^*) = \max_{\pi\in\Pi_{\mathcal{B}_a}} J_M(\pi) = J_{M_{\mathcal{B}_z}}(\pi_z^*). J M ( Ψ ∗ ∘ π z ∗ ) = π ∈ Π B a max J M ( π ) = J M B z ( π z ∗ ) .
The alignment concerns the stated batch-constrained policy classes, not the unrestricted physical optimum.
Supported reweighting guarantee
We now formalize the policy-improvement principle underlying critic-guided filtering. The update reweights a fixed reference policy toward higher-value latent actions without introducing actions outside its support. Throughout this subsection, we fix a goal g g g and use the common closed supported comparison domain of Appendix § 11 .§ 1 .
We first show that a reweighting that favors higher-value actions cannot decrease the expected action value under the reference policy.
Lemma 2 (Co-monotone reweighting). Fix ( s , g ) (s, g) ( s , g ) and a reference latent policy μ ( ⋅ ∣ s , g ) \mu(\cdot\mid s, g) μ ( ⋅ ∣ s , g ) on the finite supported latent set. Let ω ( z ) \omega(z) ω ( z ) be bounded, nonnegative, and nondecreasing in Q μ ( s , z , g ) Q^\mu(s, z, g) Q μ ( s , z , g ) , with C ω : = E μ [ ω ] > 0 C_\omega:= \mathbb{E}_\mu[\omega] > 0 C ω := E μ [ ω ] > 0 . Then the reweighted policy
μ ω ( z ∣ s , g ) = μ ( z ∣ s , g ) ω ( z ) C ω \mu^\omega(z \mid s, g) = \frac{\mu(z \mid s, g)\omega(z)}{C_\omega} μ ω ( z ∣ s , g ) = C ω μ ( z ∣ s , g ) ω ( z ) satisfies
E μ ω [ Q μ ( s , z , g ) ] ≥ E μ [ Q μ ( s , z , g ) ] = V μ ( s , g ) . \mathbb{E}_{\mu^\omega}[Q^\mu(s, z, g)] \geq\mathbb{E}_\mu[Q^\mu(s, z, g)] = V^\mu(s, g). E μ ω [ Q μ ( s , z , g )] ≥ E μ [ Q μ ( s , z , g )] = V μ ( s , g ) . Proof. Let z , z ~ z, \tilde{z} z , z ~ be independent samples from μ ( ⋅ ∣ s , g ) \mu(\cdot\mid s, g) μ ( ⋅ ∣ s , g ) , and abbreviate Q z = Q μ ( s , z , g ) Q_z = Q^\mu(s, z, g) Q z = Q μ ( s , z , g ) . Since the weights are nondecreasing in value,
0 ≤ 1 2 E [ ( Q z − Q z ~ ) ( ω ( z ) − ω ( z ~ ) ) ] = E μ [ Q ω ] − E μ [ Q ] E μ [ ω ] . 0 \leq\frac{1}{2}\mathbb{E}[(Q_z - Q_{\tilde{z}})(\omega(z) - \omega(\tilde{z}))] = \mathbb{E}_\mu[Q\omega] - \mathbb{E}_\mu[Q]\mathbb{E}_\mu[\omega]. 0 ≤ 2 1 E [( Q z − Q z ~ ) ( ω ( z ) − ω ( z ~ ))] = E μ [ Q ω ] − E μ [ Q ] E μ [ ω ] . Dividing by C ω > 0 C_\omega> 0 C ω > 0 gives
E μ ω [ Q ] = E μ [ Q ω ] C ω ≥ E μ [ Q ] , \mathbb{E}_{\mu^\omega}[Q] = \frac{\mathbb{E}_\mu[Q\omega]}{C_\omega} \geq\mathbb{E}_\mu[Q], E μ ω [ Q ] = C ω E μ [ Q ω ] ≥ E μ [ Q ] , which proves the claim.
The weights need not depend directly on the numerical value of Q μ Q^\mu Q μ . Advantage-based weights and thresholded scores also satisfy the required ordering.
Corollary 1. A bounded, nonnegative, nondecreasing function h h h of the advantage
A μ ( s , z , g ) = Q μ ( s , z , g ) − V μ ( s , g ) A^\mu(s, z, g) = Q^\mu(s, z, g) - V^\mu(s, g) A μ ( s , z , g ) = Q μ ( s , z , g ) − V μ ( s , g ) defines weights ω ( z ) = h ( A μ ( s , z , g ) ) \omega(z) = h(A^\mu(s, z, g)) ω ( z ) = h ( A μ ( s , z , g )) satisfying Lemma 2 whenever the normalizing mass is positive. More generally, the thresholded weights
ω ( z ) = 1 [ f ( s , z , g ) > b ( s , g ) ] \omega(z) = \mathbf{1}[f(s, z, g) > b(s, g)] ω ( z ) = 1 [ f ( s , z , g ) > b ( s , g )] satisfy Lemma 2 whenever f ( s , ⋅ , g ) f(s, \cdot, g) f ( s , ⋅ , g ) is nondecreasing in Q μ ( s , ⋅ , g ) Q^\mu(s, \cdot, g) Q μ ( s , ⋅ , g ) , b b b is independent of z z z , and the normalizing mass is positive.
Proof. Subtracting a baseline independent of z z z preserves the value ordering. Applying a nondecreasing function or thresholding a nondecreasing score at a z z z -independent cutoff likewise preserves this ordering. The resulting weights therefore satisfy Lemma 2 under the stated positivity condition.
This observation connects value-based reweighting to the contrastive score used to motivate LAPIS. Remark 2 (Density-ratio scores and value ordering). For the binary density-ratio objective introduced in § 3 , the optimal score is
f ⋆ ( s , z , g ) = log Q μ ( s , z , g ) − log p B ( g ) f^{\star}(s,z,g)=\log Q^{\mu}(s,z,g)-\log p_{\mathrm{B}}(g) f ⋆ ( s , z , g ) = log Q μ ( s , z , g ) − log p B ( g )
wherever the ratio is defined. At fixed ( s , g ) (s,g) ( s , g ) , the term − log p B ( g ) -\log p_{\mathrm{B}}(g) − log p B ( g ) is independent of z z z , so f ⋆ f^{\star} f ⋆ preserves the ordering of Q μ Q^{\mu} Q μ . A z z z -independent score threshold therefore produces nonnegative weights that are nondecreasing in value; the threshold need not equal V μ ( s , g ) V^{\mu}(s,g) V μ ( s , g ) . Appendix § 12 .§ 7 relates this score-space filtering principle to the learned critic.
Applying the statewise inequality at every state yields a return guarantee for the reweighted policy. When all weights vanish, retaining the reference policy makes the update well-defined.
Proposition 1 (Goal-conditioned supported improvement). Let π β z \pi_{\beta}^{z} π β z be a batch-constrained reference policy, with advantage A β z A_{\beta}^{z} A β z in M B z \mathcal{M}_{\mathrm{B}z} M B z . For η > 0 \eta>0 η > 0 , define
π η z ( z ∣ s , g ) ∝ π β z ( z ∣ s , g ) h ( A β z ( s , z , g ) ) η , \pi_{\eta}^{z}(z\mid s,g)\propto\pi_{\beta}^{z}(z\mid s,g)h(A_{\beta}^{z}(s,z,g))^{\eta}, π η z ( z ∣ s , g ) ∝ π β z ( z ∣ s , g ) h ( A β z ( s , z , g ) ) η , where h h h is bounded, nonnegative, and nondecreasing. At states with zero normalizing mass, set π η z = π β z \pi_{\eta}^{z}=\pi_{\beta}^{z} π η z = π β z , and define π 0 z = π β z \pi_{0}^{z}=\pi_{\beta}^{z} π 0 z = π β z . Then
supp π η z ( ⋅ ∣ s , g ) ⊆ supp π β z ( ⋅ ∣ s , g ) , J M B z ( π η z ; g ) ≥ J M B z ( π β z ; g ) . \operatorname{supp}\pi_{\eta}^{z}(\cdot\mid s,g)\subseteq\operatorname{supp}\pi_{\beta}^{z}(\cdot\mid s,g),\qquad J_{\mathcal{M}_{\mathrm{B}z}}(\pi_{\eta}^{z};g)\geq J_{\mathcal{M}_{\mathrm{B}z}}(\pi_{\beta}^{z};g). supp π η z ( ⋅ ∣ s , g ) ⊆ supp π β z ( ⋅ ∣ s , g ) , J M B z ( π η z ; g ) ≥ J M B z ( π β z ; g ) . Under the conditions of Theorem 1, the same return ordering holds for their batch-exact decoded physical policies.
Proof. Reweighting cannot assign positive probability to a latent action outside the reference policy’s support, and the fallback leaves the reference policy unchanged. The updated policy therefore remains batch-constrained on the common supported domain.
For η > 0 \eta>0 η > 0 , the weights h ( A β z ) η h(A_{\beta}^{z})^{\eta} h ( A β z ) η remain nonnegative and nondecreasing in the reference action value. At each state with positive normalizing mass, Lemma 2 gives
E z ∼ π η z ( ⋅ ∣ s , g ) [ Q M B z π β z ( s , z , g ) ] ≥ V M B z π β z ( s , g ) . \mathbb{E}_{z\sim\pi_{\eta}^{z}(\cdot\mid s,g)}\left[Q_{\mathcal{M}_{\mathrm{B}z}}^{\pi_{\beta}^{z}}(s,z,g)\right]\geq V_{\mathcal{M}_{\mathrm{B}z}}^{\pi_{\beta}^{z}}(s,g). E z ∼ π η z ( ⋅ ∣ s , g ) [ Q M B z π β z ( s , z , g ) ] ≥ V M B z π β z ( s , g ) .
At fallback states, or when η = 0 \eta=0 η = 0 , equality holds. Writing V β = V M B z π β z ( ⋅ , g ) V_{\beta}=V_{\mathcal{M}_{\mathrm{B}z}}^{\pi_{\beta}^{z}}(\cdot,g) V β = V M B z π β z ( ⋅ , g ) and T π η z T^{\pi_{\eta}^{z}} T π η z for the updated policy’s Bellman operator, this inequality is
T π η z V β ≥ V β . T^{\pi_{\eta}^{z}}V_{\beta}\geq V_{\beta}. T π η z V β ≥ V β .
By monotonicity, repeated application of T π η z T^{\pi_{\eta}^{z}} T π η z produces a nondecreasing sequence starting from V β V_{\beta} V β . Since the operator is a γ \gamma γ -contraction, this sequence converges to V M B z π η z ( ⋅ , g ) V_{\mathcal{M}_{\mathrm{B}z}}^{\pi_{\eta}^{z}}(\cdot,g) V M B z π η z ( ⋅ , g ) , establishing pointwise value improvement. Averaging over μ 0 \mu_{0} μ 0 proves the return inequality.
Both policies are batch-constrained, so Theorem 1 transfers this ordering to their decoded physical returns under its assumptions. The argument holds for each fixed goal and therefore also after averaging over a common goal distribution.
The exponent η \eta η controls distributional reweighting and is distinct from the flow-guidance scale w w w in § 6 . In particular, for the hard gate h = 1 [ ⋅ > 0 ] h=\mathbf{1}[\cdot>0] h = 1 [ ⋅ > 0 ] , we have h η = h h^{\eta}=h h η = h for every η > 0 \eta>0 η > 0 , so the reweighted policy is independent of η \eta η over this range. The proposition concerns this exact distributional update; guided flow samples need not follow the same distribution. Their relationship is discussed in Appendix § 12 .§ 8 .
H-step endpoint latents: proof of Theorem 1'
Theorem 1′ (On-support transfer for H H H -step latents). Let M B H ~ z \mathcal{M}_{\mathrm{B}_{\tilde{H}}}^{z} M B H ~ z be the latent batch MDP over observed endpoint latents with the empirical one-step transition p B , H ( 1 ) p_{\mathrm{B},H}^{(1)} p B , H ( 1 ) and reward r B , H ( 1 ) r_{\mathrm{B},H}^{(1)} r B , H ( 1 ) given ( s t , z t ) (s_t,z_t) ( s t , z t ) , and Ψ H ⋆ ( a ∣ s , z ) \Psi_{H}^{\star}(a\mid s,z) Ψ H ⋆ ( a ∣ s , z ) the empirical first-action decoder (Definition 7 ). Under Assumption 1, every latent policy π z \pi_z π z batch-constrained in M B H ~ z \mathcal{M}_{\mathrm{B}_{\tilde{H}}}^{z} M B H ~ z satisfies J M B H ~ z ( π z ) = J M ( Ψ H ⋆ ∘ π z ) J_{\mathcal{M}_{\mathrm{B}_{\tilde{H}}}^{z}}(\pi_z)=J_{\mathcal{M}}(\Psi_{H}^{\star}\circ\pi_z) J M B H ~ z ( π z ) = J M ( Ψ H ⋆ ∘ π z ) , so return improvement transfers to the decoded policies. Supported outcome identifiability is not required, and p B , H ( 1 ) ( ⋅ ∣ s , z ) p_{\mathrm{B},H}^{(1)}(\cdot\mid s,z) p B , H ( 1 ) ( ⋅ ∣ s , z ) may be stochastic. For H = 1 H=1 H = 1 under supported outcome identifiability, M B 1 z = M B z \mathcal{M}_{\mathrm{B}_{1}^{z}}=\mathcal{M}_{\mathrm{B}z} M B 1 z = M B z and Ψ 1 ⋆ = Ψ ⋆ \Psi_{1}^{\star}=\Psi^{\star} Ψ 1 ⋆ = Ψ ⋆ , recovering Theorem 1.
The deployed controller labels each recorded step with an H H H -step endpoint latent and executes only the first decoded action before replanning (Appendix § 12 .§ 6 ). This subsection defines the corresponding latent batch MDP and proves Theorem 1′. We keep the conventions of Appendix § 11 .§ 1 for rewards, the absorbing sink, initial states, and termination.
Setting. For every recorded step ( s t , a t , s t + 1 ) (s_t,a_t,s_{t+1}) ( s t , a t , s t + 1 ) whose endpoint s t + H s_{t+H} s t + H is also recorded, let z t = I ( s t , s t + H ) z_t = I(s_t,s_{t+H}) z t = I ( s t , s t + H ) , and let N ( s , a , z , s ′ ) N(s,a,z,s') N ( s , a , z , s ′ ) count the occurrences of ( s t , a t , z t , s t + 1 ) = ( s , a , z , s ′ ) (s_t,a_t,z_t,s_{t+1})=(s,a,z,s') ( s t , a t , z t , s t + 1 ) = ( s , a , z , s ′ ) . Marginal counts omit arguments, for example N ( s , z ) = ∑ a , s ′ N ( s , a , z , s ′ ) N(s,z)=\sum_{a,s'}N(s,a,z,s') N ( s , z ) = ∑ a , s ′ N ( s , a , z , s ′ ) and N ( s , a , z ) = ∑ s ′ N ( s , a , z , s ′ ) N(s,a,z)=\sum_{s'}N(s,a,z,s') N ( s , a , z ) = ∑ s ′ N ( s , a , z , s ′ ) .
Definition 7 (H H H -step endpoint latent batch MDP). The latent batch MDP M B H z = ( S ∪ { s a b s } , Z , p B , H ( 1 ) , r B , H ( 1 ) , γ ) \mathcal{M}_{B_H^z}=(\mathcal{S}\cup\{s_{\mathrm{abs}}\},Z,p_{B,H}^{(1)},r_{B,H}^{(1)},\gamma) M B H z = ( S ∪ { s abs } , Z , p B , H ( 1 ) , r B , H ( 1 ) , γ ) has observed latents Z B , H ( s ) = { z : N ( s , z ) > 0 } Z_{B,H}(s)=\{z:N(s,z)>0\} Z B , H ( s ) = { z : N ( s , z ) > 0 } and, for z ∈ Z B , H ( s ) z\in Z_{B,H}(s) z ∈ Z B , H ( s ) ,
p B , H ( 1 ) ( s ′ ∣ s , z ) = N ( s , z , s ′ ) N ( s , z ) , r B , H ( 1 ) ( s , z ) = ∑ a , s ′ N ( s , a , z , s ′ ) r ( s , a , s ′ ) N ( s , z ) . p_{B,H}^{(1)}(s'\mid s,z)=\frac{N(s,z,s')}{N(s,z)},\qquad r_{B,H}^{(1)}(s,z)=\frac{\sum_{a,s'}N(s,a,z,s')r(s,a,s')}{N(s,z)}. p B , H ( 1 ) ( s ′ ∣ s , z ) = N ( s , z ) N ( s , z , s ′ ) , r B , H ( 1 ) ( s , z ) = N ( s , z ) ∑ a , s ′ N ( s , a , z , s ′ ) r ( s , a , s ′ ) . Unsupported pairs transition to s a b s s_{\mathrm{abs}} s abs with zero reward, as in Definition 5 . The first-action batch decoder is
Ψ H ∗ ( a ∣ s , z ) = N ( s , a , z ) N ( s , z ) . \Psi_H^*(a\mid s,z)=\frac{N(s,a,z)}{N(s,z)}. Ψ H ∗ ( a ∣ s , z ) = N ( s , z ) N ( s , a , z ) .
The kernel p B , H ( 1 ) p_{B,H}^{(1)} p B , H ( 1 ) is the one-step successor distribution conditioned on the endpoint latent introduced in Appendix § 12 .§ 6 . No identifiability condition is imposed: the same z z z at s s s may be followed by different first actions and different successors, so p B , H ( 1 ) ( ⋅ ∣ s , z ) p_{B,H}^{(1)}(\cdot\mid s,z) p B , H ( 1 ) ( ⋅ ∣ s , z ) can be stochastic even under deterministic ground dynamics. Because r B , H ( 1 ) r_{B,H}^{(1)} r B , H ( 1 ) averages bounded rewards, ∥ V M B H z π z ∥ ∞ ≤ R max / ( 1 − γ ) \lVert V_{\mathcal{M}_{B_H^z}}^{\pi_z}\rVert_\infty\le R_{\max}/(1-\gamma) ∥ V M B H z π z ∥ ∞ ≤ R m a x / ( 1 − γ ) for every latent policy. A latent policy is batch-constrained in M B H z \mathcal{M}_{B_H^z} M B H z if it selects only z ∈ Z B , H ( s ) z\in Z_{B,H}(s) z ∈ Z B , H ( s ) along its reachable states, with the continuation convention of Appendix § 11 .§ 1 . Decoded execution samples z ∼ π z ( ⋅ ∣ s ) z\sim\pi_z(\cdot\mid s) z ∼ π z ( ⋅ ∣ s ) and a ∼ Ψ H ∗ ( ⋅ ∣ s , z ) a\sim\Psi_H^*(\cdot\mid s,z) a ∼ Ψ H ∗ ( ⋅ ∣ s , z ) , executes a a a , and samples again at the next state.
The case H = 1 H=1 H = 1 . Then z t = I ( s t , s t + 1 ) z_t=I(s_t,s_{t+1}) z t = I ( s t , s t + 1 ) and N ( s , a , z , s ′ ) = N ( s , a , s ′ ) 1 [ z = I ( s , s ′ ) ] N(s,a,z,s')=N(s,a,s')\mathbf{1}[z=I(s,s')] N ( s , a , z , s ′ ) = N ( s , a , s ′ ) 1 [ z = I ( s , s ′ )] . Under supported outcome identifiability (Definition 4 ), N ( s , z , s ′ ) > 0 N(s,z,s')>0 N ( s , z , s ′ ) > 0 for z ∈ Z B ( s ) z\in Z_B(s) z ∈ Z B ( s ) only if s ′ = F B ( s , z ) s'=F_B(s,z) s ′ = F B ( s , z ) . Hence M B 1 z = M B z \mathcal{M}_{B_1^z}=\mathcal{M}_{B^z} M B 1 z = M B z and
Ψ 1 ∗ ( a ∣ s , z ) = N ( s , a , F B ( s , z ) ) ∑ b N ( s , b , F B ( s , z ) ) = Ψ ∗ ( a ∣ s , z ) , \Psi_1^*(a\mid s,z)=\frac{N(s,a,F_B(s,z))}{\sum_bN(s,b,F_B(s,z))}=\Psi^*(a\mid s,z), Ψ 1 ∗ ( a ∣ s , z ) = ∑ b N ( s , b , F B ( s , z )) N ( s , a , F B ( s , z )) = Ψ ∗ ( a ∣ s , z ) ,
and Theorem 1 is the case H = 1 H=1 H = 1 of Theorem 1′.
Proof of Theorem 1′. Batch constraints under decoding. Ψ H ∗ ( a ∣ s , z ) > 0 \Psi_H^*(a\mid s,z)>0 Ψ H ∗ ( a ∣ s , z ) > 0 implies N ( s , a , z ) > 0 N(s,a,z)>0 N ( s , a , z ) > 0 , hence a ∈ A B ( s ) a\in A_B(s) a ∈ A B ( s ) , as in Remark 1.
Transition and reward identities. Under Assumption 1, every recorded step from ( s , a ) (s,a) ( s , a ) reaches T ( s , a ) T(s,a) T ( s , a ) , so N ( s , a , z , s ′ ) = N ( s , a , z ) 1 [ s ′ = T ( s , a ) ] N(s,a,z,s')=N(s,a,z)\mathbf{1}[s'=T(s,a)] N ( s , a , z , s ′ ) = N ( s , a , z ) 1 [ s ′ = T ( s , a )] . For a supported pair ( s , z ) (s,z) ( s , z ) ,
∑ a Ψ H ∗ ( a ∣ s , z ) p ( s ′ ∣ s , a ) = ∑ a N ( s , a , z ) 1 [ T ( s , a ) = s ′ ] N ( s , z ) = ∑ a N ( s , a , z , s ′ ) N ( s , z ) = p B , H ( 1 ) ( s ′ ∣ s , z ) , \begin{aligned}
\sum_a\Psi_H^*(a\mid s,z)p(s'\mid s,a)=\sum_a\frac{N(s,a,z)\mathbf{1}[T(s,a)=s']}{N(s,z)} \\
&=\sum_a\frac{N(s,a,z,s')}{N(s,z)}=p_{B,H}^{(1)}(s'\mid s,z),
\end{aligned} a ∑ Ψ H ∗ ( a ∣ s , z ) p ( s ′ ∣ s , a ) = a ∑ N ( s , z ) N ( s , a , z ) 1 [ T ( s , a ) = s ′ ] = a ∑ N ( s , z ) N ( s , a , z , s ′ ) = p B , H ( 1 ) ( s ′ ∣ s , z ) ,
∑ a Ψ H ∗ ( a ∣ s , z ) r ( s , a , T ( s , a ) ) = ∑ a , s ′ N ( s , a , z , s ′ ) r ( s , a , s ′ ) N ( s , z ) = r B , H ( 1 ) ( s , z ) . \sum_a\Psi_H^*(a\mid s,z)r(s,a,T(s,a))=\frac{\sum_{a,s'}N(s,a,z,s')r(s,a,s')}{N(s,z)}=r_{B,H}^{(1)}(s,z). a ∑ Ψ H ∗ ( a ∣ s , z ) r ( s , a , T ( s , a )) = N ( s , z ) ∑ a , s ′ N ( s , a , z , s ′ ) r ( s , a , s ′ ) = r B , H ( 1 ) ( s , z ) .
Thus the decoded one-step transition and expected reward agree with those of M B H z \mathcal{M}_{B_H^z} M B H z at every supported pair.
Return equality. The remainder is the proof of Theorem 1 with δ F B ( s , z ) \delta_{F_B(s,z)} δ F B ( s , z ) and r B z r_{B^z} r B z replaced by p B , H ( 1 ) ( ⋅ ∣ s , z ) p_{B,H}^{(1)}(\cdot\mid s,z) p B , H ( 1 ) ( ⋅ ∣ s , z ) and r B , H ( 1 ) r_{B,H}^{(1)} r B , H ( 1 ) . Averaging over π z \pi_z π z , both processes have the state-level kernel P π z ( ⋅ ∣ s ) = ∑ z π z ( z ∣ s ) p B , H ( 1 ) ( ⋅ ∣ s , z ) P_{\pi_z}(\cdot\mid s)=\sum_z\pi_z(z\mid s)p_{B,H}^{(1)}(\cdot\mid s,z) P π z ( ⋅ ∣ s ) = ∑ z π z ( z ∣ s ) p B , H ( 1 ) ( ⋅ ∣ s , z ) and expected reward r ˉ π z ( s ) = ∑ z π z ( z ∣ s ) r B , H ( 1 ) ( s , z ) \bar r_{\pi_z}(s)=\sum_z\pi_z(z\mid s)r_{B,H}^{(1)}(s,z) r ˉ π z ( s ) = ∑ z π z ( z ∣ s ) r B , H ( 1 ) ( s , z ) on the common reachable support, so V M B H z π z V_{\mathcal{M}_{B_H^z}}^{\pi_z} V M B H z π z and V M Ψ H ∗ ∘ π z V_{\mathcal{M}}^{\Psi_H^*\circ\pi_z} V M Ψ H ∗ ∘ π z solve the same Bellman equation V = r ˉ π z + γ P π z V V=\bar r_{\pi_z}+\gamma P_{\pi_z}V V = r ˉ π z + γ P π z V , whose bounded solution is unique for γ < 1 \gamma<1 γ < 1 . Averaging over μ 0 \mu_0 μ 0 gives return equality, and applying it to two policies gives the improvement implication. □
Correspondence with practical LAPIS. LAPIS selects z t z_t z t at s t s_t s t , executes only the first decoded action (E = 1 E = 1 E = 1 ), and selects again at s t + 1 s_{t+1} s t + 1 , which is decoded execution in M B H z \mathcal{M}_{B^z_H} M B H z . Its critic propagates value from ( s t , z t ) (s_t,z_t) ( s t , z t ) to ( s t + 1 , z t + 1 ) (s_{t+1},z_{t+1}) ( s t + 1 , z t + 1 ) with the recorded continuation z t + 1 = I ( s t + 1 , s t + 1 + H ) z_{t+1}=I(s_{t+1},s_{t+1+H}) z t + 1 = I ( s t + 1 , s t + 1 + H ) (Appendix § 12 .§ 6 ), so it evaluates the behavior latent policy in the same MDP. Theorem 1 covers the batch-exact first-action decoder; Proposition 2 with reference kernel p B , H ( 1 ) p^{(1)}_{B,H} p B , H ( 1 ) covers learned decoders (Appendix § 12 .§ 3 ).
Theory–Practice Correspondence
The analysis in § 5 isolates the conditions under which latent policy improvement remains valid after physical execution. Practical LAPIS realizes these conditions through learned latent representations, decoding, value estimation, and policy extraction. This appendix makes each correspondence explicit and extends the exact analysis to the learned components used in practice. Exact transfer arises as the zero-mismatch case of the transition and reward correspondence below; supported reweighting defines the target distribution approximated by the learned policy.
Learned representations and supported outcome identifiability
Supported outcome identifiability (Definition 4 ) requires each supported state–latent pair ( s , z ) (s,z) ( s , z ) to identify a unique observed successor F B ( s , z ) F_B(s,z) F B ( s , z ) . In practice, this structure is learned through the inverse model I I I and forward model F F F . Reconstruction connects the learned representation to outcome identification. If two observed successors are assigned the same latent, I ( s , s 1 ′ ) = I ( s , s 2 ′ ) = z I(s,s'_1)=I(s,s'_2)=z I ( s , s 1 ′ ) = I ( s , s 2 ′ ) = z , then
∥ s 1 ′ − s 2 ′ ∥ 2 ≤ ∥ s 1 ′ − F ( s , z ) ∥ 2 + ∥ s 2 ′ − F ( s , z ) ∥ 2 . \lVert s'_1-s'_2\rVert_2 \le\lVert s'_1-F(s,z)\rVert_2+\lVert s'_2-F(s,z)\rVert_2. ∥ s 1 ′ − s 2 ′ ∥ 2 ≤ ∥ s 1 ′ − F ( s , z ) ∥ 2 + ∥ s 2 ′ − F ( s , z ) ∥ 2 .
Thus, when both transitions are reconstructed accurately, successors sharing a latent are close in the reconstruction metric. The theorem states supported outcome identifiability as an exact structural condition, while the reconstruction objective encourages this structure on observed transitions.
The experimental policy operates on quantized latent codes. Identifiability concerns whether the code supplied to the policy and decoder distinguishes the observed outcomes relevant for control. The learned predictor F F F and the exact supported lookup F B F_B F B play complementary roles: F F F is trained to reconstruct outcomes, whereas F B F_B F B states the exact correspondence on the observed transition graph. Accurate reconstruction encourages outcome consistency; exact identifiability additionally requires that the supported code distinguish these outcomes.
State representations and Markov control
The analysis in § 5 uses Markov states to isolate the relationship between latent decisions and physical execution from perceptual representation learning. The experiments use single-frame RGB observations processed by learned visual encoders. Practical LAPIS applies the latent policy improvement principle to this learned observation representation.
Different underlying states can produce indistinguishable observations, so the observation-level process need not be exactly Markov. This concerns the state representation, whereas latent-action realizability concerns whether a selected latent transition can be physically executed from the current state. The exact results characterize the control-level correspondence under Markov state information. Temporal or recurrent representations provide possible ways to retain additional state information; the analysis applies when the resulting representation satisfies the stated Markov and transition conditions.
Approximate transfer under execution mismatch
Theorem 1 gives exact return equality when supported latent transitions are executed through the batch-exact decoder under deterministic dynamics. More generally, the same correspondence admits a quantitative extension whenever decoded execution induces transitions and rewards close to those used for latent evaluation. We state the bound for a decoder Ψ ^ ( a ∣ s , z ) \widehat{\Psi}(a \mid s,z) Ψ ( a ∣ s , z ) , which includes both Ψ ∗ \Psi^{*} Ψ ∗ and the state-marginal choice Ψ ~ ( a ∣ s , z ) = Ψ ( a ∣ z ) \widetilde{\Psi}(a \mid s,z)=\Psi(a \mid z) Ψ ( a ∣ s , z ) = Ψ ( a ∣ z ) .
Reference latent process. The bound compares decoded execution with a reference latent batch MDP. For one-step latents this is M B z \mathcal{M}_{B_z} M B z with ( Z B , p B z , r B z ) (Z_B,p_{B_z},r_{B_z}) ( Z B , p B z , r B z ) . For the H H H -step endpoint latents of the deployed controller it is M B H \mathcal{M}_{B_H} M B H with ( Z B , H , p B , H ( 1 ) , r B , H ( 1 ) ) (Z_{B,H},p^{(1)}_{B,H},r^{(1)}_{B,H}) ( Z B , H , p B , H ( 1 ) , r B , H ( 1 ) ) (Definition 7 ). We write the statements below for M B z \mathcal{M}_{B_z} M B z . They hold verbatim for M B H \mathcal{M}_{B_H} M B H after replacing p B z p_{B_z} p B z and r B z r_{B_z} r B z by p B , H ( 1 ) p^{(1)}_{B,H} p B , H ( 1 ) and r B , H ( 1 ) r^{(1)}_{B,H} r B , H ( 1 ) , since the argument uses only the two Bellman equations and the reward bound, not the form of the reference kernel. Because the controller executes only the first decoded action before replanning (Appendix § 12 .§ 6 ), Ψ ^ ( a ∣ s , z ) \widehat{\Psi}(a \mid s,z) Ψ ( a ∣ s , z ) denotes the distribution of this first action; for a chunk decoder, it is the first-action marginal of the decoded chunk.
Fix a latent policy π z \pi_z π z that is batch-constrained in M B z \mathcal{M}_{B^z} M B z . We compare the two processes on a common Markov state space and decision interval, with the same γ \gamma γ , reward bound R max R_{\max} R m a x , and initial distribution μ 0 \mu_0 μ 0 . Use the ambient-space batch kernels of Appendix § 11 .§ 1 , extend the ground process with the same absorbing sink, and define π z \pi_z π z and Ψ ^ \widehat{\Psi} Ψ as probability kernels throughout this space. Where an empirical policy or decoder is undefined, fix an arbitrary probability-kernel extension. The occupancy below includes any off-support visits under this specified execution. Sums over continuous variables denote integrals.
The decoder induces
p ~ Ψ ^ ( s ′ ∣ s , z ) : = ∑ a Ψ ^ ( a ∣ s , z ) p ( s ′ ∣ s , a ) , \widetilde{p}_{\widehat{\Psi}}(s' \mid s,z) := \sum_a \widehat{\Psi}(a \mid s,z)p(s' \mid s,a), p Ψ ( s ′ ∣ s , z ) := a ∑ Ψ ( a ∣ s , z ) p ( s ′ ∣ s , a ) ,
r ~ Ψ ^ ( s , z ) : = ∑ a Ψ ^ ( a ∣ s , z ) ∑ s ′ p ( s ′ ∣ s , a ) r ( s , a , s ′ ) . \widetilde{r}_{\widehat{\Psi}}(s,z) := \sum_a \widehat{\Psi}(a \mid s,z)\sum_{s'}p(s' \mid s,a)r(s,a,s'). r Ψ ( s , z ) := a ∑ Ψ ( a ∣ s , z ) s ′ ∑ p ( s ′ ∣ s , a ) r ( s , a , s ′ ) .
For this fixed decoder, define the mismatches
ϵ p ( s , z ) : = D T V ( p B z ( ⋅ ∣ s , z ) , p ~ Ψ ^ ( ⋅ ∣ s , z ) ) , \epsilon_p(s,z) := D_{\mathrm{TV}}\left(p_{B^z}(\cdot\mid s,z),\widetilde{p}_{\widehat{\Psi}}(\cdot\mid s,z)\right), ϵ p ( s , z ) := D TV ( p B z ( ⋅ ∣ s , z ) , p Ψ ( ⋅ ∣ s , z ) ) ,
ϵ r ( s , z ) : = ∣ r B z ( s , z ) − r ~ Ψ ^ ( s , z ) ∣ , \epsilon_r(s,z) := \left|r_{B^z}(s,z)-\widetilde{r}_{\widehat{\Psi}}(s,z)\right|, ϵ r ( s , z ) := ∣ r B z ( s , z ) − r Ψ ( s , z ) ∣ ,
where D T V ( P , Q ) = sup A ∣ P ( A ) − Q ( A ) ∣ D_{\mathrm{TV}}(P,Q)=\sup_A|P(A)-Q(A)| D TV ( P , Q ) = sup A ∣ P ( A ) − Q ( A ) ∣ . Let d Ψ ^ ∘ π z d^{\widehat{\Psi}\circ\pi_z} d Ψ ∘ π z be the normalized γ \gamma γ -discounted occupancy of ( s , z ) (s,z) ( s , z ) under decoded execution in M \mathcal{M} M from μ 0 \mu_0 μ 0 , with z ∼ π z ( ⋅ ∣ s ) z\sim\pi_z(\cdot\mid s) z ∼ π z ( ⋅ ∣ s ) . Write ϵ ‾ p = E d Ψ ^ ∘ π z [ ϵ p ] \overline{\epsilon}_p=\mathbb{E}_{d^{\widehat{\Psi}\circ\pi_z}}[\epsilon_p] ϵ p = E d Ψ ∘ π z [ ϵ p ] and ϵ ‾ r = E d Ψ ^ ∘ π z [ ϵ r ] \overline{\epsilon}_r=\mathbb{E}_{d^{\widehat{\Psi}\circ\pi_z}}[\epsilon_r] ϵ r = E d Ψ ∘ π z [ ϵ r ] .
Proposition 2 (Approximate transfer under execution mismatch). For a batch-constrained latent policy π z \pi_z π z and the fixed, globally defined decoder Ψ ^ \widehat{\Psi} Ψ above.
∣ J M ( Ψ ^ ∘ π z ) − J M B z ( π z ) ∣ ≤ ϵ ‾ r 1 − γ + 2 γ R max ( 1 − γ ) 2 ϵ ‾ p . (7) \left|J_{\mathcal{M}}(\widehat{\Psi}\circ\pi_z)-J_{\mathcal{M}_{B^z}}(\pi_z)\right|
\leq\frac{\overline{\epsilon}_r}{1-\gamma}
+\frac{2\gamma R_{\max}}{(1-\gamma)^2}\overline{\epsilon}_p. \tag*{(7)} J M ( Ψ ∘ π z ) − J M B z ( π z ) ≤ 1 − γ ϵ r + ( 1 − γ ) 2 2 γ R m a x ϵ p . (7) Zero transition and reward mismatch yields exact return equality. In particular, Ψ ^ = Ψ ∗ \widehat{\Psi}=\Psi^* Ψ = Ψ ∗ under Assumption 1 recovers Theorem 1. With reference M B H z \mathcal{M}_{B_H^z} M B H z , Ψ ^ = Ψ H ∗ \widehat{\Psi}=\Psi_H^* Ψ = Ψ H ∗ under Assumption 1 makes both mismatches zero by the identities in the proof of Theorem 1', which recovers Theorem 1'.
Proof. Step 1: Bellman residual. Write V 1 = V M Ψ ^ ∘ π z V_1=V_{\mathcal{M}}^{\widehat{\Psi}\circ\pi_z} V 1 = V M Ψ ∘ π z and V 2 = V M B z π z V_2=V_{\mathcal{M}_{B^z}}^{\pi_z} V 2 = V M B z π z . Their Bellman equations are
V 1 ( s ) = ∑ z π z ( z ∣ s ) [ r ~ Ψ ^ ( s , z ) + γ ∑ s ′ p ~ Ψ ^ ( s ′ ∣ s , z ) V 1 ( s ′ ) ] , V_1(s)=\sum_z\pi_z(z\mid s)\left[\widetilde{r}_{\widehat{\Psi}}(s,z)+\gamma\sum_{s'}\widetilde{p}_{\widehat{\Psi}}(s'\mid s,z)V_1(s')\right], V 1 ( s ) = z ∑ π z ( z ∣ s ) [ r Ψ ( s , z ) + γ s ′ ∑ p Ψ ( s ′ ∣ s , z ) V 1 ( s ′ ) ] ,
V 2 ( s ) = ∑ z π z ( z ∣ s ) [ r B z ( s , z ) + γ ∑ s ′ p B z ( s ′ ∣ s , z ) V 2 ( s ′ ) ] . V_2(s)=\sum_z\pi_z(z\mid s)\left[r_{B^z}(s,z)+\gamma\sum_{s'}p_{B^z}(s'\mid s,z)V_2(s')\right]. V 2 ( s ) = z ∑ π z ( z ∣ s ) [ r B z ( s , z ) + γ s ′ ∑ p B z ( s ′ ∣ s , z ) V 2 ( s ′ ) ] .
Let ( P ‾ Ψ ^ h ) ( s ) = ∑ z , s ′ π z ( z ∣ s ) p ~ Ψ ^ ( s ′ ∣ s , z ) h ( s ′ ) (\overline{P}_{\widehat{\Psi}}h)(s)=\sum_{z,s'}\pi_z(z\mid s)\widetilde{p}_{\widehat{\Psi}}(s'\mid s,z)h(s') ( P Ψ h ) ( s ) = ∑ z , s ′ π z ( z ∣ s ) p Ψ ( s ′ ∣ s , z ) h ( s ′ ) . Subtracting the equations gives Δ = δ + γ P ‾ Ψ ^ Δ \Delta=\delta+\gamma\overline{P}_{\widehat{\Psi}}\Delta Δ = δ + γ P Ψ Δ , where Δ = V 1 − V 2 \Delta=V_1-V_2 Δ = V 1 − V 2 and
δ ( s ) = ∑ z π z ( z ∣ s ) [ r ~ Ψ ^ ( s , z ) − r B z ( s , z ) + γ ∑ s ′ ( p ~ Ψ ^ − p B z ) ( s ′ ∣ s , z ) V 2 ( s ′ ) ] . \delta(s)=\sum_z\pi_z(z\mid s)\left[\widetilde{r}_{\widehat{\Psi}}(s,z)-r_{B^z}(s,z)+\gamma\sum_{s'}\left(\widetilde{p}_{\widehat{\Psi}}-p_{B^z}\right)(s'\mid s,z)V_2(s')\right]. δ ( s ) = z ∑ π z ( z ∣ s ) [ r Ψ ( s , z ) − r B z ( s , z ) + γ s ′ ∑ ( p Ψ − p B z ) ( s ′ ∣ s , z ) V 2 ( s ′ ) ] .
Step 2: bounding the residual. Definition 5 gives ∥ V 2 ∥ ∞ ≤ R max / ( 1 − γ ) \lVert V_2\rVert_\infty\leq R_{\max}/(1-\gamma) ∥ V 2 ∥ ∞ ≤ R m a x / ( 1 − γ ) . The total-variation inequality ∣ E P h − E Q h ∣ ≤ 2 ∥ h ∥ ∞ D T V ( P , Q ) \lvert\mathbb{E}_P h-\mathbb{E}_Q h\rvert\leq2\lVert h\rVert_\infty D_{\mathrm{TV}}(P,Q) ∣ E P h − E Q h ∣ ≤ 2 ∥ h ∥ ∞ D TV ( P , Q ) therefore yields
∣ δ ( s ) ∣ ≤ ∑ z π z ( z ∣ s ) [ ϵ r ( s , z ) + 2 γ R max 1 − γ ϵ p ( s , z ) ] . |\delta(s)|\leq\sum_z\pi_z(z\mid s)\left[\epsilon_r(s,z)+\frac{2\gamma R_{\max}}{1-\gamma}\epsilon_p(s,z)\right]. ∣ δ ( s ) ∣ ≤ z ∑ π z ( z ∣ s ) [ ϵ r ( s , z ) + 1 − γ 2 γ R m a x ϵ p ( s , z ) ] .
Step 3: occupancy-weighted bound. Since P ‾ Ψ ^ \overline{P}_{\widehat{\Psi}} P Ψ is a stochastic operator and γ < 1 \gamma<1 γ < 1 , unrolling the residual along decoded execution gives
∣ J M ( Ψ ^ ∘ π z ) − J M B z ( π z ) ∣ ≤ ∑ t ≥ 0 γ t E [ ∣ δ ( s t ) ∣ ] ≤ 1 1 − γ E d Ψ ^ ∘ π z [ ϵ r + 2 γ R max 1 − γ ϵ p ] , (8) \begin{aligned}
\left|J_{\mathcal{M}}(\widehat{\Psi}\circ\pi_z)-J_{\mathcal{M}_{B^z}}(\pi_z)\right|
\leq\sum_{t\geq0}\gamma^t\mathbb{E}[|\delta(s_t)|] \\
&\leq\frac{1}{1-\gamma}\mathbb{E}_{d^{\widehat{\Psi}\circ\pi_z}}\left[\epsilon_r+\frac{2\gamma R_{\max}}{1-\gamma}\epsilon_p\right], \tag*{(8)}
\end{aligned} J M ( Ψ ∘ π z ) − J M B z ( π z ) ≤ t ≥ 0 ∑ γ t E [ ∣ δ ( s t ) ∣ ] ≤ 1 − γ 1 E d Ψ ∘ π z [ ϵ r + 1 − γ 2 γ R m a x ϵ p ] , (8)
which is the stated bound. If Ψ ^ = Ψ ∗ \widehat{\Psi}=\Psi^{*} Ψ = Ψ ∗ and Assumption 1 holds, the supported transition and expected-reward identities in the proof of Theorem 1 make both mismatches zero along the rollout. This recovers exact transfer without requiring Assumption 2.
For the fixed decoder, let ϵ Ψ ^ ( π z ) \epsilon_{\widehat{\Psi}}(\pi_z) ϵ Ψ ( π z ) denote the right-hand side of Equation (7) , using the occupancy of Ψ ^ ∘ π z \widehat{\Psi}\circ\pi_z Ψ ∘ π z .
Corollary 2 (Approximate improvement transfer). For batch-constrained latent policies π z , π z ′ \pi_z,\pi'_z π z , π z ′ and the same decoder Ψ ^ \widehat{\Psi} Ψ , define Δ z = J M B z ( π z ′ ) − J M B z ( π z ) \Delta_z=J_{\mathcal{M}_{B_z}}(\pi'_z)-J_{\mathcal{M}_{B_z}}(\pi_z) Δ z = J M B z ( π z ′ ) − J M B z ( π z ) . Then
J M ( Ψ ^ ∘ π z ′ ) − J M ( Ψ ^ ∘ π z ) ≥ Δ z − ϵ Ψ ^ ( π z ′ ) − ϵ Ψ ^ ( π z ) . J_{\mathcal{M}}(\widehat{\Psi}\circ\pi'_z)-J_{\mathcal{M}}(\widehat{\Psi}\circ\pi_z)\geq\Delta_z-\epsilon_{\widehat{\Psi}}(\pi'_z)-\epsilon_{\widehat{\Psi}}(\pi_z). J M ( Ψ ∘ π z ′ ) − J M ( Ψ ∘ π z ) ≥ Δ z − ϵ Ψ ( π z ′ ) − ϵ Ψ ( π z ) . Thus a latent improvement exceeding the two execution-gap bounds transfers to physical improvement.
Proposition 2 and Corollary 2 hold unchanged with M B z H \mathcal{M}_{B_z}^{\mathsf{H}} M B z H as the reference.
Proof. Add and subtract the two latent returns and apply Proposition 2 separately to π z ′ \pi'_z π z ′ and $\pi_z.
Interpretation. The bound accounts jointly for stochastic dynamics and decoder approximation through their induced transition and reward mismatch. Its quantitative usefulness depends on this mismatch being small in the stated metric: a point mass and an atomless transition law have total-variation distance one even if they are geometrically close. Appendix § 12 .§ 6 describes the experimental controller.
Learned decoding under scarce action supervision
The batch-exact decoder establishes an exact correspondence, while successful execution fundamentally requires realizing the represented successor. Recovering a particular recorded action distribution is unnecessary when alternative controls induce the same transition and reward. Under Assumptions 1–2, a decoder satisfying
Ψ ^ ( { a : T ( s , a ) = F B ( s , z ) } ∣ s , z ) = 1 \widehat{\Psi}\left(\{a:T(s,a)=F_B(s,z)\}\mid s,z\right)=1 Ψ ( { a : T ( s , a ) = F B ( s , z )} ∣ s , z ) = 1
on visited supported pairs induces the same successor and reward as the latent transition. Hence ϵ p = ϵ r = 0 \epsilon_p=\epsilon_r=0 ϵ p = ϵ r = 0 in Proposition 2, even if its action distribution differs from the empirical weights of Ψ ∗ \Psi^* Ψ ∗ . In this outcome-compatible setting, consistent execution of the latent outcome, rather than recovery of a particular recorded action, is sufficient for transfer.
Explicit state conditioning is therefore not a requirement of the transfer argument when the decoder realizes the represented outcome on every visited supported pair. LAPIS learns a state-marginal decoder Ψ ( a ∣ z ) \Psi(a\mid z) Ψ ( a ∣ z ) from scarce action labels. Table 5 compares it with a state-conditioned decoder using the same frozen 5%-label LAM across decoder-label budgets; within each budget, the remaining policy and training setup is shared between decoder variants. Marginal decoding achieves higher success across all three domains at 1% and 5% labels, while state conditioning performs better with full supervision. These results support marginal decoding for the scarce-label setting studied here.
Goal-conditioned objectives and outcome alignment
Theorem 2 requires physical controls that produce the same transition outcome to receive the same reward. The goal-conditioned objective used by LAPIS satisfies this condition under the deterministic dynamics of Assumption 1. Contrastive RL uses the goal-reaching reward
r g ( s , a ) = ( 1 − γ ) p ( s ′ = g ∣ s , a ) . r_g(s,a)=(1-\gamma)p(s'=g\mid s,a). r g ( s , a ) = ( 1 − γ ) p ( s ′ = g ∣ s , a ) .
Under Assumption 1, s ′ = T ( s , a ) s'=T(s,a) s ′ = T ( s , a ) , so
r g ( s , a ) = ( 1 − γ ) 1 [ T ( s , a ) = g ] . r_g(s,a)=(1-\gamma)\mathbf{1}[T(s,a)=g]. r g ( s , a ) = ( 1 − γ ) 1 [ T ( s , a ) = g ] .
Consequently, if two physical actions produce the same successor,
T ( s , a 1 ) = T ( s , a 2 ) = s ′ , T(s,a_1)=T(s,a_2)=s', T ( s , a 1 ) = T ( s , a 2 ) = s ′ ,
then
r g ( s , a 1 ) = r g ( s , a 2 ) = ( 1 − γ ) 1 [ s ′ = g ] . r_g(s,a_1)=r_g(s,a_2)=(1-\gamma)\mathbf{1}[s'=g]. r g ( s , a 1 ) = r g ( s , a 2 ) = ( 1 − γ ) 1 [ s ′ = g ] .
Thus, in the deterministic setting analyzed in § 5 , the goal-conditioned reward depends on a physical control only through its transition outcome. Outcome-based abstraction therefore preserves the reward distinctions relevant to the goal-conditioned objective, providing the objective-alignment condition used in Theorem 2.
Representation horizon and receding-horizon control
The analysis in § 5 is stated at the policy decision interval, where a latent action represents the successor of a single transition. In visual latent-action learning, however, the temporal gap used to construct the latent representation also controls how much transition information is visible to the encoder. We distinguish three quantities: the LAM temporal gap H H H in z t = I ( s t , s t + H ) z_t=I(s_t,s_{t+H}) z t = I ( s t , s t + H ) ; the physical-action chunk size C C C , the number of actions predicted by a real-action policy or a latent-action decoder; and the execution horizon E E E , the number of actions executed before replanning. The symbol H H H refers only to the LAM temporal gap and does not apply to real-action policies without a LAM.
Our default LAM uses a representation horizon of H = 3 H=3 H = 3 ,
z t = I ( s t , s t + 3 ) , z_t=I(s_t,s_{t+3}), z t = I ( s t , s t + 3 ) ,
but LAPIS remains a one-step receding-horizon controller: the decoder predicts a chunk of C = 3 C=3 C = 3 actions, only the first action is executed (E = 1 E=1 E = 1 ), and the policy replans from s t + 1 s_{t+1} s t + 1 . The critic likewise propagates value at this one-step interval using s t + 1 s_{t+1} s t + 1 and the recorded continuation latent z t + 1 = I ( s t + 1 , s t + 4 ) z_{t+1}=I(s_{t+1},s_{t+4}) z t + 1 = I ( s t + 1 , s t + 4 ) . Thus, the longer temporal gap changes the representation used for control, rather than requiring longer open-loop execution.
Representation and execution horizons. We empirically separate these two roles by varying the representation horizon and the number of decoded actions executed before replanning. Both H = 1 H=1 H = 1 and our default H = 3 H=3 H = 3 controller execute a single action before replanning. With this execution scheme, reducing the representation horizon from H = 3 H=3 H = 3 to H = 1 H=1 H = 1 sharply reduces task success across all three representative domains. Conversely, keeping the H = 3 H=3 H = 3 representation but executing the full three-action chunk does not account for the benefit of the longer horizon: one-step replanning achieves higher mean performance in all three domains, with a particularly large difference on AntMaze.
Table 8 : LAM temporal gap H H H and execution horizon E E E . Success rate (%) on representative tasks. The decoder chunk size is C = H C=H C = H in these ablations; E E E is the number of decoded actions executed before replanning.
Table 8. LAM temporal gap H H H and execution horizon E E E . Success rate (%) on representative tasks.
The control results are consistent with the representation diagnostics. With matched frozen LAMs, increasing the temporal gap from H = 1 H=1 H = 1 to H = 3 H=3 H = 3 raises held-out action-decodability R 2 R^2 R 2 from 0.501 to 0.597 on AntMaze, 0.586 to 0.778 on Cube, and 0.628 to 0.811 on Scene (Table 18 ). Adjacent visual observations can contain only small action-dependent changes, whereas a longer gap exposes more visible transition structure to the latent bottleneck. Together, the representation probes and control results indicate that H = 3 H=3 H = 3 is useful primarily as a representation horizon , while one-step replanning remains effective for closed-loop execution.
Table 18. Temporal gap and action decodability. Variance-weighted test R 2 R^2 R 2 for an MLP predicting a t a_t a t from z e z_e z e , using matched samples and frozen 500k LAMs.
Connection to the one-step analysis. When H = 1 H=1 H = 1 , the encoded endpoint coincides with the successor at the control interval, giving the direct latent–successor correspondence analyzed in § 5 . For H > 1 H>1 H > 1 , an endpoint latent can in principle summarize trajectories with different intermediate successors even though execution and value propagation remain one-step. The appropriate practical object is therefore the one-step successor distribution conditioned on the endpoint latent,
p B , H ( 1 ) ( s + ∣ s , z ) : = Pr B ( s t + 1 = s + ∣ s t = s , I ( s t , s t + H ) = z ) . p_{\mathcal{B},H}^{(1)}(s^+\mid s,z):=\Pr_{\mathcal{B}}(s_{t+1}=s^+\mid s_t=s,I(s_t,s_{t+H})=z). p B , H ( 1 ) ( s + ∣ s , z ) := B Pr ( s t + 1 = s + ∣ s t = s , I ( s t , s t + H ) = z ) .
The corresponding physical process is the one-step transition induced by the first decoded action. Theorem 1′ gives exact transfer for this object: in the H H H -step endpoint latent batch MDP M B H z \mathcal{M}_{\mathcal{B}_H^z} M B H z , whose transition kernel is p B , H ( 1 ) p_{\mathcal{B},H}^{(1)} p B , H ( 1 ) , decoding with the first-action batch decoder Ψ H ∗ \Psi_H^* Ψ H ∗ preserves return under deterministic dynamics (Appendix § 11 .§ 5 ). For learned decoders, Proposition 2 with reference kernel p B , H ( 1 ) p_{\mathcal{B},H}^{(1)} p B , H ( 1 ) bounds the gap (Appendix § 12 .§ 3 ). The critic bootstraps from ( s t , z t ) (s_t,z_t) ( s t , z t ) to ( s t + 1 , z t + 1 ) (s_{t+1},z_{t+1}) ( s t + 1 , z t + 1 ) and therefore evaluates latent policies in this same MDP.
This distinction allows the analytical and practical roles of the temporal horizon to be separated cleanly: the batch-MDP analysis (Theorems 1 and 1′) isolates the conditions required for exact latent-to-physical correspondence, while the longer observation gap provides a substantially more informative representation for the visual domains studied here.
First-step execution fidelity. Although the H = 3 H=3 H = 3 latent represents an endpoint three steps ahead, the deployed controller executes only the first decoded action before replanning. We therefore directly evaluate whether this longer-horizon representation preserves accurate one-step execution. On 100 held-out transitions per environment, we encode the corresponding latent, decode the action chunk, execute only its first action, and measure the resulting next-state cube-position error. The deployed H = 3 H=3 H = 3 configuration achieves lower error than the H = 1 H=1 H = 1 configuration on both Cube (1.51 vs. 3.26 mm) and Scene (0.80 vs. 1.22 mm). The same trend holds on transitions where the cube actually moves, with 3.59 vs. 7.69 mm on Cube and 4.20 vs. 6.42 mm on Scene (Table 9 ). These results provide direct evidence that the practical H = 3 H=3 H = 3 representation can preserve accurate first-step execution of the measured object transition despite encoding a longer-horizon endpoint, supporting its use with one-step receding-horizon control. This diagnostic compares existing configurations using held-out dataset latents and a geometric next-state metric; the configurations differ in temporal gap, decoder chunking, and training augmentation.
Table 9. First-step cube-position error (mm, ↓ \downarrow ↓ ) on matched held-out transitions using seed-0 models (C = H C=H C = H , E = 1 E=1 E = 1 ). Moving subsets have recorded cube displacement > 1 >1 > 1 mm.
Learned value ordering and critic-guided reweighting
Proposition 1 formalizes policy improvement as reweighting a supported reference distribution toward higher-value latent actions. LAPIS implements this principle through the learned critic and the gate in Equation (4) . For a fixed critic and threshold snapshot, let q B ( z ∣ s , g ) q_B(z\mid s,g) q B ( z ∣ s , g ) denote the conditional distribution of observed latent transitions presented to the actor, including its goal-relabeling procedure. Define the selected mass
Z G ( s , g ) = E z ∼ q B ( ⋅ ∣ s , g ) [ G ( s , z , g ) ] . Z_G(s,g)=\mathbb{E}_{z\sim q_B(\cdot\mid s,g)}[G(s,z,g)]. Z G ( s , g ) = E z ∼ q B ( ⋅ ∣ s , g ) [ G ( s , z , g )] .
and, whenever Z G ( s , g ) > 0 Z_G(s,g)>0 Z G ( s , g ) > 0 , the gated distribution
q G ( z ∣ s , g ) = q B ( z ∣ s , g ) G ( s , z , g ) Z G ( s , g ) . (9) q_G(z\mid s,g)=\frac{q_B(z\mid s,g)G(s,z,g)}{Z_G(s,g)}. \tag*{(9)} q G ( z ∣ s , g ) = Z G ( s , g ) q B ( z ∣ s , g ) G ( s , z , g ) . (9)
Because the gate only removes examples,
supp q G ( ⋅ ∣ s , g ) ⊆ supp q B ( ⋅ ∣ s , g ) ⊆ Z B ( s ) \operatorname{supp}q_G(\cdot\mid s,g)\subseteq\operatorname{supp}q_B(\cdot\mid s,g)\subseteq\mathcal{Z}_B(s) supp q G ( ⋅ ∣ s , g ) ⊆ supp q B ( ⋅ ∣ s , g ) ⊆ Z B ( s )
at sampled state–goal pairs. Thus, at the target-distribution level, critic-guided filtering is exactly a supported reweighting of observed state–latent transitions.
The improvement result does not require the critic score to be a calibrated numerical estimate of Q Q Q ; it requires the score to preserve the ordering of the reference action values over z z z at fixed ( s , g ) (s,g) ( s , g ) . For the ideal binary density-ratio objective in Remark 2,
f ⋆ ( s , z , g ) = log Q μ ( s , z , g ) − log p B ( g ) , f^\star(s,z,g)=\log Q^\mu(s,z,g)-\log p_B(g), f ⋆ ( s , z , g ) = log Q μ ( s , z , g ) − log p B ( g ) ,
and the second term is independent of z z z . Hence f ⋆ f^\star f ⋆ preserves the ordering of Q μ Q^\mu Q μ , and thresholding it at any z z z -independent state–goal threshold produces weights that are nondecreasing in value. In the supported comparison of Appendix § 11 .§ 4 , take the reference policy to be
μ ( ⋅ ∣ s , g ) = q B ( ⋅ ∣ s , g ) , \mu(\cdot\mid s,g)=q_B(\cdot\mid s,g), μ ( ⋅ ∣ s , g ) = q B ( ⋅ ∣ s , g ) ,
with q G = μ q_G=\mu q G = μ when Z G ( s , g ) = 0 Z_G(s,g)=0 Z G ( s , g ) = 0 . Whenever the score preserves the ordering of
Q M B z μ ( s , ⋅ , g ) , Q^\mu_{M_{Bz}}(s,\cdot,g), Q M B z μ ( s , ⋅ , g ) ,
Corollary 1 and Proposition 1 give
J M B z ( q G ; g ) ≥ J M B z ( q B ; g ) . J_{M_{Bz}}(q_G;g)\ge J_{M_{Bz}}(q_B;g). J M B z ( q G ; g ) ≥ J M B z ( q B ; g ) .
Under the conditions of Theorem 1, this ordering transfers to batch-exact decoded execution, while Corollary 2 provides the corresponding approximate-transfer statement for learned decoding.
Practical LAPIS uses the TD-InfoNCE critic and the learned score-space threshold described in Appendix § 13 .§ 3 . The gate is applied only to observed state–latent transitions, so for every fixed critic and threshold snapshot it defines an exact supported reweighting q G q_G q G of the actor-training distribution q B q_B q B ; this support property holds for any score. During joint training, the critic and hence the reweighted target evolve, while the actor is trained toward the target defined by the current gate. Whether this reweighting also improves the reference depends on whether the learned score preserves the reference-value ordering over z z z , which the TD-InfoNCE objective does not determine, as follows.
Score offset in TD-InfoNCE. Both cross-entropy terms in Equation (12) are softmaxes over candidate goals within the same row ( s i , z i ) (s_i,z_i) ( s i , z i ) , and the bootstrap weights W i j W_{ij} W ij are softmaxes over goals as well. Adding any function c ( s , z ) c(s,z) c ( s , z ) to the score therefore leaves the loss unchanged, so the optimum is determined only up to such an offset, as noted for TD-InfoNCE by Zheng et al. [41 ] . An offset that depends on z z z can change the ordering of latents at fixed ( s , g ) (s,g) ( s , g ) and hence the raw-score gate in Equation (4) . The L2-normalized bilinear parameterization of Appendix § 13 .§ 3 bounds the score, ∣ f ∣ ≤ 1 / τ r e p \lvert f\rvert\le1/\tau_{\mathrm{rep}} ∣ f ∣ ≤ 1/ τ rep , but does not remove the offset: a component of the normalized goal features shared across goals contributes a term that depends only on ( s , z ) (s,z) ( s , z ) . Neither the ensemble minimum nor the threshold V ^ ( s , g ) \widehat V(s,g) V ( s , g ) , which does not depend on z z z , removes it. Preservation of value ordering by the learned score is therefore an empirical property of the trained critic, not a consequence of the objective.
Offset-free score. Let p c p_c p c denote the critic’s candidate-goal distribution (Appendix § 13 .§ 3 ) and define
f ˉ ( s , z , g ) = f ( s , z , g ) − log E g ′ ∼ p c [ exp f ( s , z , g ′ ) ] . \bar f(s,z,g)=f(s,z,g)-\log\mathbb{E}_{g'\sim p_c}\left[\exp f(s,z,g')\right]. f ˉ ( s , z , g ) = f ( s , z , g ) − log E g ′ ∼ p c [ exp f ( s , z , g ′ ) ] .
The log-partition in f ˉ \bar f f ˉ can be estimated by a log-sum-exp over the candidate goals in a minibatch, the same normalization TD-InfoNCE uses for its importance weights, but this finite-sample estimate is biased and noisy. We therefore measure how closely the raw score f f f , which the gate uses, follows f ˉ \bar f f ˉ in trained LAPIS critics (Table 6 in § 7.6 ). The protocol for that table is as follows. We use antmaze-medium, cube-single, and scene with four seeds each, average the 300k, 400k, and 500k checkpoints within each seed, and report the mean over seeds. For each training state, we score 16 candidate latents, the dataset latent and 15 samples from the behavior branch, and estimate the log-partition of f ˉ \bar f f ˉ with 16,384 goals from p c p_c p c , averaged over four independent draws. threshold rule, a stand-in for V ^ \hat{V} V ^ , keeps the dataset latent if its score exceeds the 0.9-expectile of the scores of the 15 sampled latents. Standard deviations across seeds are below 0.01 for the correlation and below 1% for the agreement. Table 4 (b) compares the TD-InfoNCE critic with an IQL critic, whose regression targets make values directly comparable across z z z , under fixed extraction rules.
Consistent with this supported-update view, Table 4 (a) shows that replacing direct max-Q Q Q actor optimization with critic-guided filtering improves performance for both deterministic and flow actors.
Flow-based extraction and guided inference
The remaining correspondence is between the supported target distribution defined by critic-guided filtering and the learned flow policy. For a fixed gate snapshot, define
ℓ θ ( o ) ( s , z , g ) = E x 0 , t [ ∥ v θ ( x t , t ; s , g , o ) − ( z − x 0 ) ∥ 2 2 ] , x t = ( 1 − t ) x 0 + t z , \ell_{\theta}^{(o)}(s,z,g)=\mathbb{E}_{x_0,t}\left[\left\|v_{\theta}(x_t,t;s,g,o)-(z-x_0)\right\|_2^2\right], \qquad x_t=(1-t)x_0+tz, ℓ θ ( o ) ( s , z , g ) = E x 0 , t [ ∥ v θ ( x t , t ; s , g , o ) − ( z − x 0 ) ∥ 2 2 ] , x t = ( 1 − t ) x 0 + t z ,
where noise and time are sampled as in Equation (5) . Conditioning the actor objective on ( s , g ) (s,g) ( s , g ) gives
L a c t o r ( s , g ) = E q B [ G ℓ θ ( 1 ) + λ ℓ θ ( 0 ) ] = Z G ( s , g ) E q G [ ℓ θ ( 1 ) ] + λ E q B [ ℓ θ ( 0 ) ] . (10) \begin{aligned}
\mathcal{L}_{\mathrm{actor}}(s,g) &= \mathbb{E}_{q_B}\left[G\ell_{\theta}^{(1)}+\lambda\ell_{\theta}^{(0)}\right] \\
&= Z_G(s,g)\mathbb{E}_{q_G}\left[\ell_{\theta}^{(1)}\right]+\lambda\mathbb{E}_{q_B}\left[\ell_{\theta}^{(0)}\right]. \tag*{(10)}
\end{aligned} L actor ( s , g ) = E q B [ G ℓ θ ( 1 ) + λ ℓ θ ( 0 ) ] = Z G ( s , g ) E q G [ ℓ θ ( 1 ) ] + λ E q B [ ℓ θ ( 0 ) ] . (10)
Thus, the high-value branch is trained toward q G q_G q G , while the behavior branch is trained toward q B q_B q B . The full loss averages over sampled ( s , g ) (s,g) ( s , g ) , with Z G ( s , g ) Z_G(s,g) Z G ( s , g ) retaining the original weighting across state–goal contexts. When Z G ( s , g ) = 0 Z_G(s,g)=0 Z G ( s , g ) = 0 , the gated loss term vanishes; the reference-policy fallback in Appendix § 12 .§ 7 is used only to define the ideal policy comparison. Equation (10) therefore gives the direct correspondence between the practical flow-matching objective and the supported reweighting analyzed in Appendix § 11 .§ 4 .
The support inclusion established in Appendix § 12 .§ 7 is exact for the target distributions q B q_B q B and q G q_G q G . A finite-capacity neural flow approximates these conditional distributions while sharing statistical strength across visual states. At inference, generated continuous latents are projected to the learned product codebook before decoding. This projection restricts samples to the learned code space, but does not by itself guarantee state-conditioned support; that depends on how faithfully the learned policy represents its conditional target.
Classifier-free guidance combines the two learned velocity fields,
v w C F G = ( 1 − w ) v θ ( ⋅ ; 0 ) + w v θ ( ⋅ ; 1 ) . v_w^{\mathrm{CFG}}=(1-w)v_{\theta}(\cdot;0)+wv_{\theta}(\cdot;1). v w CFG = ( 1 − w ) v θ ( ⋅ ; 0 ) + w v θ ( ⋅ ; 1 ) .
At w = 0 w=0 w = 0 , inference uses the branch trained toward q B q_B q B , while w = 1 w=1 w = 1 uses the branch trained toward q G q_G q G . Values w > 1 w>1 w > 1 extrapolate the velocity field toward the high-value branch and therefore need not preserve the exact support guarantee of q G q_G q G . This extrapolation differs from direct max-Q Q Q actor optimization, however. Direct max-Q Q Q updates the selected latent by following critic values in the continuous latent space, including regions where the critic is not grounded by observed transitions. In LAPIS, the critic is used first to select high-value examples only among observed state–latent transitions, producing q G ⊆ q B q_G\subseteq q_B q G ⊆ q B ; guidance then extrapolates between policy fields trained on q B q_B q B and this supported high-value reweighting. The guidance direction is therefore anchored to distributions constructed entirely from observed state–latent transitions, although sufficiently strong extrapolation can still move generated samples away from their exact support.
The supported-reweighting result thus identifies q G q_G q G at w = 1 w=1 w = 1 as the theoretical anchor, while w > 1 w>1 w > 1 provides an inference-time extension that trades stricter adherence to this target for stronger emphasis on the high-value branch. The guidance scale w w w is distinct from the distributional exponent η \eta η in Proposition 1, and the supported-improvement guarantee does not by itself extend to arbitrary w > 1 w>1 w > 1 .
The experiments provide complementary evidence for these stages. Table 4 (a) shows that the flow actor improves policy extraction when paired with critic-guided filtering. Table 14 further separates the supported high-value branch from guidance extrapolation. Across the visual manipulation tasks, w = 1 w=1 w = 1 , which uses the branch trained toward q G q_G q G , already captures most of the improvement from the behavior branch w = 0 w=0 w = 0 to the default w = 3 w=3 w = 3 policy. Stronger guidance therefore provides an additional empirical refinement rather than being solely responsible for the gains of LAPIS. § 7.3 further compares the effect of stronger direct max-Q Q Q optimization and stronger guidance through behavior-support distance d B C d_{\mathrm{BC}} d BC , realization round-trip error d r o u n d d_{\mathrm{round}} d round , and task success. As CRL becomes more aggressive, its support distance and realization error increase while success collapses, whereas LAPIS remains closer to the behavior-supported region over the guidance sweep. These results show that guided inference stays closer to the behavior-supported region than unconstrained critic maximization in this comparison.
Table 14. Success rate (%) of latent-action LAPIS with 5% action labels at different guidance scales.
The direct CFGRL comparison in Appendix § 13 .§ 5 further illustrates why this distinction matters in a learned latent action space. Goal-conditioned CFGRL benefits from increasing guidance when applied to physical actions, most clearly on AntMaze, but provides no consistent improvement when the same construction is applied directly to learned latent actions. In contrast, LAPIS obtains large gains already at w = 1 w = 1 w = 1 , before extrapolative guidance, with its high-value branch trained toward the critic-filtered supported target q G q_G q G rather than the unfiltered goal-conditioned behavior distribution. Together with the component ablation in Table 4 (a), this provides empirical evidence that support-aware target construction, rather than guidance alone, is consequential for latent policy improvement.
Realization error and return
Proposition 2 measures execution mismatch in total variation, which equals one between two distinct point masses even when the two successors are geometrically close. The analysis of § 5 restricts policies to observed state–latent pairs, where δ ( s , z ) = 0 \delta(s,z) = 0 δ ( s , z ) = 0 (Definition 1). Here we bound how far latent and decoded returns can separate when the executed latents are not realized exactly, under Lipschitz conditions that are standard in the analysis of latent and model-based RL ([2 , 15 ] ).
Let M F \mathcal{M}^{F} M F denote the latent outcome process that, from state s s s and latent z z z , moves to the predicted successor F ( s , z ) F(s,z) F ( s , z ) with reward r ( s , F ( s , z ) ) r(s,F(s,z)) r ( s , F ( s , z )) . For a decoder Ψ ^ ( a ∣ s , z ) \hat{\Psi}(a \mid s,z) Ψ ^ ( a ∣ s , z ) , state-conditioned or not, define its expected one-step round-trip error and its excess over the realization error,
η Ψ ^ ( s , z ) = E a ∼ Ψ ^ ( ⋅ ∣ s , z ) ∥ I ( s , T ( s , a ) ) − z ∥ 2 , ϵ Ψ ^ ( s , z ) = η Ψ ^ ( s , z ) − δ ( s , z ) ≥ 0 , \eta_{\hat{\Psi}}(s,z) = \mathbb{E}_{a \sim\hat{\Psi}(\cdot\mid s,z)} \left\| I(s,T(s,a)) - z \right\|_2,\qquad\epsilon_{\hat{\Psi}}(s,z) = \eta_{\hat{\Psi}}(s,z) - \delta(s,z) \ge0, η Ψ ^ ( s , z ) = E a ∼ Ψ ^ ( ⋅ ∣ s , z ) ∥ I ( s , T ( s , a )) − z ∥ 2 , ϵ Ψ ^ ( s , z ) = η Ψ ^ ( s , z ) − δ ( s , z ) ≥ 0 ,
where the inequality holds because T ( s , a ) ∈ R ( s ) T(s,a) \in\mathcal{R}(s) T ( s , a ) ∈ R ( s ) .
Proposition 3 (Return gap under realization error). Suppose Assumptions 1 and 2 hold, and (i) the forward model is accurate on reachable outcomes up to ϵ F ≥ 0 \epsilon_F \ge0 ϵ F ≥ 0 , ∥ F ( s , I ( s , s ′ ) ) − s ′ ∥ 2 ≤ ϵ F \|F(s,I(s,s')) - s'\|_2 \le\epsilon_F ∥ F ( s , I ( s , s ′ )) − s ′ ∥ 2 ≤ ϵ F for every s ′ ∈ R ( s ) s' \in\mathcal{R}(s) s ′ ∈ R ( s ) ; (ii) F ( s , ⋅ ) F(s,\cdot) F ( s , ⋅ ) is L F L_F L F -Lipschitz, ∥ F ( s , z ) − F ( s , z ′ ) ∥ 2 ≤ L F ∥ z − z ′ ∥ 2 \|F(s,z) - F(s,z')\|_2 \le L_F\|z-z'\|_2 ∥ F ( s , z ) − F ( s , z ′ ) ∥ 2 ≤ L F ∥ z − z ′ ∥ 2 ; (iii) r ( s , ⋅ ) r(s,\cdot) r ( s , ⋅ ) , extended to all states with ∣ r ∣ ≤ R max |r| \le R_{\max} ∣ r ∣ ≤ R m a x , is L r L_r L r -Lipschitz, and the latent value V M F π z V_{\mathcal{M}^{F}}^{\pi_z} V M F π z is L V L_V L V -Lipschitz in the state. Then, for any latent policy π z \pi_z π z and decoder Ψ ^ \hat{\Psi} Ψ ^ ,
∣ J M F ( π z ) − J M ( Ψ ^ ∘ π z ) ∣ ≤ ( L r + γ L V ) [ L F ∑ t ≥ 0 γ t E [ δ ( s t , z t ) + ϵ Ψ ^ ( s t , z t ) ] + ϵ F 1 − γ ] . \left|J_{\mathcal{M}^{F}}(\pi_z) - J_{\mathcal{M}}(\hat{\Psi} \circ\pi_z)\right| \le(L_r + \gamma L_V) \left[ L_F \sum_{t \ge0} \gamma^t \mathbb{E}\left[\delta(s_t,z_t) + \epsilon_{\hat{\Psi}}(s_t,z_t)\right] + \frac{\epsilon_F}{1-\gamma} \right]. J M F ( π z ) − J M ( Ψ ^ ∘ π z ) ≤ ( L r + γ L V ) [ L F t ≥ 0 ∑ γ t E [ δ ( s t , z t ) + ϵ Ψ ^ ( s t , z t ) ] + 1 − γ ϵ F ] . where the expectation is over decoded execution of Ψ ^ ∘ π z \hat{\Psi} \circ\pi_z Ψ ^ ∘ π z in M \mathcal{M} M from μ 0 \mu_0 μ 0 .
Proof. Write V F = V M F π z V_F = V_{\mathcal{M}^{F}}^{\pi_z} V F = V M F π z and V E = V M Ψ ^ ∘ π z V_E = V_{\mathcal{M}}^{\hat{\Psi}\circ\pi_z} V E = V M Ψ ^ ∘ π z . For a state s s s , with z ∼ π z ( ⋅ ∣ s ) z \sim\pi_z(\cdot\mid s) z ∼ π z ( ⋅ ∣ s ) , a ∼ Ψ ^ ( ⋅ ∣ s , z ) a \sim\hat{\Psi}(\cdot\mid s,z) a ∼ Ψ ^ ( ⋅ ∣ s , z ) and s ′ = T ( s , a ) s' = T(s,a) s ′ = T ( s , a ) , the Bellman equations give
V F ( s ) − V E ( s ) = E [ r ( s , F ( s , z ) ) − r ( s , s ′ ) + γ ( V F ( F ( s , z ) ) − V F ( s ′ ) ) + γ ( V F ( s ′ ) − V E ( s ′ ) ) ] . V_F(s) - V_E(s) = \mathbb{E}\left[r(s,F(s,z)) - r(s,s') + \gamma(V_F(F(s,z)) - V_F(s')) + \gamma(V_F(s') - V_E(s'))\right]. V F ( s ) − V E ( s ) = E [ r ( s , F ( s , z )) − r ( s , s ′ ) + γ ( V F ( F ( s , z )) − V F ( s ′ )) + γ ( V F ( s ′ ) − V E ( s ′ )) ] .
Since s ′ ∈ R ( s ) s' \in\mathcal{R}(s) s ′ ∈ R ( s ) , (i) and (ii) give
∥ F ( s , z ) − s ′ ∥ 2 ≤ ∥ F ( s , z ) − F ( s , I ( s , s ′ ) ) ∥ 2 + ∥ F ( s , I ( s , s ′ ) ) − s ′ ∥ 2 ≤ L F ∥ z − I ( s , s ′ ) ∥ 2 + ϵ F . \|F(s,z) - s'\|_2 \le\|F(s,z) - F(s,I(s,s'))\|_2 + \|F(s,I(s,s')) - s'\|_2 \le L_F\|z-I(s,s')\|_2 + \epsilon_F. ∥ F ( s , z ) − s ′ ∥ 2 ≤ ∥ F ( s , z ) − F ( s , I ( s , s ′ )) ∥ 2 + ∥ F ( s , I ( s , s ′ )) − s ′ ∥ 2 ≤ L F ∥ z − I ( s , s ′ ) ∥ 2 + ϵ F .
By (iii), the first two terms are bounded in absolute value by ( L r + γ L V ) ( L F ∥ z − I ( s , s ′ ) ∥ 2 + ϵ F ) (L_r + \gamma L_V)(L_F\|z-I(s,s')\|_2 + \epsilon_F) ( L r + γ L V ) ( L F ∥ z − I ( s , s ′ ) ∥ 2 + ϵ F ) , and the conditional expectation of ∥ z − I ( s , s ′ ) ∥ 2 \|z-I(s,s')\|_2 ∥ z − I ( s , s ′ ) ∥ 2 given ( s , z ) (s,z) ( s , z ) is η Ψ ^ ( s , z ) = δ ( s , z ) + ϵ Ψ ^ ( s , z ) \eta_{\hat{\Psi}}(s,z) = \delta(s,z) + \epsilon_{\hat{\Psi}}(s,z) η Ψ ^ ( s , z ) = δ ( s , z ) + ϵ Ψ ^ ( s , z ) . Hence
∣ V F ( s ) − V E ( s ) ∣ ≤ ( L r + γ L V ) ( L F E [ δ ( s , z ) + ϵ Ψ ^ ( s , z ) ] + ϵ F ) + γ E ∣ V F ( s ′ ) − V E ( s ′ ) ∣ . |V_F(s) - V_E(s)| \le(L_r + \gamma L_V)\left(L_F\mathbb{E}[\delta(s,z) + \epsilon_{\hat{\Psi}}(s,z)] + \epsilon_F\right) + \gamma\mathbb{E}|V_F(s') - V_E(s')|. ∣ V F ( s ) − V E ( s ) ∣ ≤ ( L r + γ L V ) ( L F E [ δ ( s , z ) + ϵ Ψ ^ ( s , z )] + ϵ F ) + γ E ∣ V F ( s ′ ) − V E ( s ′ ) ∣.
Unrolling this inequality along decoded execution, using that both values are bounded by R max / ( 1 − γ ) R_{\max}/(1-\gamma) R m a x / ( 1 − γ ) so the remainder vanishes, and averaging over s 0 ∼ μ 0 s_0 \sim\mu_0 s 0 ∼ μ 0 gives the claim.
Interpretation. The bound has three terms. The realization error δ \delta δ depends only on which latents the policy selects; it is zero for latents of transitions observed at the current state. The excess ϵ Ψ b \epsilon_{\Psi_b} ϵ Ψ b depends only on the decoder; with the batch-exact decoder Ψ ∗ \Psi^* Ψ ∗ , execution of a batch-constrained policy stays on observed transitions, and both terms vanish. The last term is the forward model’s own error and does not depend on the policy. With a finite codebook, distinct reachable outcomes can share a latent, so ϵ F > 0 \epsilon_F > 0 ϵ F > 0 in general; ϵ F = 0 \epsilon_F = 0 ϵ F = 0 would require a global form of supported outcome identifiability (Definition 4 ). Theorem 1 states the batch-constrained case exactly for M B z \mathcal{M}_{\mathcal{B}_z} M B z , without a forward model. For a fixed forward model, decoder and Lipschitz constants, the bound therefore grows linearly with the realization error of the latents the policy selects. The H H H -step analogue of η Ψ b \eta_{\Psi_b} η Ψ b is the round-trip error d r o u n d d_{\mathrm{round}} d round of Figure 3 . As for Lipschitz bounds in model-based RL, condition (iii) is an idealization; in particular, it excludes 0/1 success rewards, so for our tasks the statement is qualitative. Proposition 3 is stated for one-step latents, and the same argument extends it to H H H -step latents, with the H H H -step transition and the discounted H H H -step reward in place of T T T and r r r , and γ H \gamma^H γ H in place of γ \gamma γ .
Experimental details and additional analyses
This appendix provides the experimental setup and supplementary analyses. We describe the benchmarks and evaluation protocols (Appendix § 13 .§ 1 ), latent-action representation learning and decoding (E.2), LAPIS critic and policy learning (E.3), and baseline implementations (E.4). We then report additional sampling, control, and guidance analyses (E.5), analyze action decodability and temporal-gap effects in the learned latent representation (E.6), and present qualitative rollouts (E.7).
Datasets and evaluation
Benchmarks and label regimes. We use the OGBench ([30 ] ) visual cube, scene, and antmaze-navigate domains with 64 × 64 64 \times64 64 × 64 RGB observations. Manipulation actions have five dimensions (end-effector position change, yaw change, and gripper control), while antmaze actions are eight-dimensional joint torques. For the ablations in Table 4 –Table 6 , we use one representative task from each family: visual-cube-single-play, visual-scene-play, and visual-antmaze-medium-navigate; Table 4 (b) also includes visual-cube-double-play. The diagnostics in Figure 3 focus on visual-antmaze-medium-navigate. Latent-action methods use all observed transitions for LAM and latent-policy learning, while the data used by the action-scarce baselines are specified in Appendix § 13 .§ 4 .
OGBench evaluation. Each of four seeds is evaluated on 20 episodes per task at the 300k, 400k, and 500k checkpoints. We average these checkpoint scores within each seed, then report the mean and standard deviation across seeds. All entries of Table 1 follow this protocol.
Visual Franka Kitchen. Kitchen uses 64 × 64 64 \times64 64 × 64 rendered RGB observations, nine-dimensional bounded actions, and the original D4RL simulator with a 280-step episode limit. The partial variant targets microwave, kettle, light switch, and slide cabinet; mixed targets microwave, kettle, bottom burner, and light switch. Goal images are rendered with the evaluation camera after setting the required objects to their target configurations and the robot to its initial posture. Completed subtasks are removed from the remaining-task set, and an episode terminates when all required subtasks are completed or the time limit is reached.
Table 2 reports the percentage of required subtasks completed, averaged over evaluation episodes:
C ˉ = 1 N e v a l ∑ j = 1 N e v a l 100 n c , j N j , \bar{C} = \frac{1}{N_{\mathrm{eval}}} \sum_{j=1}^{N_{\mathrm{eval}}} 100 \frac{n_{c,j}}{N_j}, C ˉ = N eval 1 j = 1 ∑ N eval 100 N j n c , j ,
where n c , j n_{c,j} n c , j of the N j = 4 N_j = 4 N j = 4 required subtasks are completed in episode j j j . All latent policies use the canonical partial training split and are evaluated against both target-task sets; mixed does not denote a separately trained latent policy. Each seed uses its own frozen LAM and decoder trained for 500k updates, with the H = 3 H = 3 H = 3 , C = 3 C = 3 C = 3 , E = 1 E = 1 E = 1 receding-horizon setup described in Appendix § 12 .§ 6 . For seeds 0–3, we average the 300k, 400k, and 500k policy-checkpoint scores (20 episodes per target-task set per checkpoint), then report the mean and population standard deviation across the four seed averages. Table 2 reports LAPIS at guidance w = 3 w = 3 w = 3 ; the latent LAPIS run uses a TD-InfoNCE critic and a log-sum-exp value baseline. The real-action oracle results are reported separately from the latent policies.
Latent action model and decoder
Representation learning. The IDM/FDM pair is a product-quantized transition autoencoder trained by pixel reconstruction with independent cross-crop inputs. It uses single-frame observations, temporal gap H = 3 H = 3 H = 3 , and IDM encoder scale 4. The codebook configuration is M : H z : D : K = 2 : 4 : 8 : 128 M:H_z:D:K = 2:4:8:128 M : H z : D : K = 2 : 4 : 8 : 128 : two codebooks, four slots per codebook, eight embedding coordinates, and 128 entries per codebook, giving 64 flattened latent coordinates. We denote the continuous encoder output by z e z_e z e and its quantization by z q z_q z q .
Latent coordinates and codebook projection. The policy, critic, and decoder use quantized latents standardized per codebook and embedding dimension:
[ N ( z q ) ] m h d = ( z q ) m h d − μ m d σ m d , (11) [\mathcal{N}(z_q)]_{mhd} = \frac{(z_q)_{mhd} - \mu_{md}}{\sigma_{md}}, \tag*{(11)} [ N ( z q ) ] mh d = σ m d ( z q ) mh d − μ m d , (11)
where the frozen means μ m d \mu_{md} μ m d and stabilized standard deviations σ m d \sigma_{md} σ m d are shared across slots h h h . The same transform is applied to the codebook entries. Generated continuous latents are projected to the nearest product-codebook entries in these coordinates before decoding.
Action decoding. The experimental state-marginal flow decoder Ψ ( a ∣ z ) \Psi(a \mid z) Ψ ( a ∣ z ) is trained on labeled examples to predict three-action chunks from standardized quantized latents and is frozen for deployment. Predicted actions are clipped to the environment bounds and executed with the receding-horizon procedure of Appendix § 12 .§ 6 . The state-conditioned variant in Table 5 additionally takes s s s as input. For Table 5 , we keep the same frozen 5%-label LAM across all decoder-label budgets and vary only decoder supervision.
Critic and policy learning
Shared policy-learning settings. Table 10 consolidates the optimization, architecture, critic, flow-policy, and evaluation hyperparameters used by LAPIS on OGBench. In Table 1 , the physical-action chunk sizes C C C for F-GCBC, GCIQL, CRL, and LAPIS are ( 3 , 1 , 1 , 3 ) (3, 1, 1, 3) ( 3 , 1 , 1 , 3 ) , respectively, while their latent-action counterparts use C = 3 C = 3 C = 3 . All latent-action methods use the H = 3 H = 3 H = 3 LAM of Appendix § 13 .§ 2 , and all settings execute only the first physical action before replanning (E = 1 E = 1 E = 1 ; Appendix § 12 .§ 6 ).
Table 10. LAPIS hyperparameters for OGBench policy learning.
Categorical TD objective. LAPIS uses categorical TD-InfoNCE on both visual manipulation and antmaze. For a minibatch of n n n recorded transitions, let s i + = s t i + 1 s_i^{+} = s_{t_i+1} s i + = s t i + 1 and let z i + z_i^{+} z i + be the LAM label inferred from its recorded continuation. Critic candidate goals are sampled from a mixture of current states, future states in recorded trajectories, and random dataset states. With ensemble index e ∈ { 1 , 2 } e \in\{1, 2\} e ∈ { 1 , 2 } , define
L i j e n e x t = f ϕ , e ( s i , z i , s j + ) , L i j e g o a l = f ϕ , e ( s i , z i , g j ) , L_{ije}^{\mathrm{next}} = f_{\phi,e}(s_i, z_i, s_j^{+}), \qquad L_{ije}^{\mathrm{goal}} = f_{\phi,e}(s_i, z_i, g_j), L ij e next = f ϕ , e ( s i , z i , s j + ) , L ij e goal = f ϕ , e ( s i , z i , g j ) ,
W i j = sg [ softmax j ( min e f ϕ ˉ , e ( s i + , z i + , g j ) ) ] . W_{ij} = \operatorname{sg}\left[\operatorname{softmax}_j\left(\min_e f_{\bar{\phi},e}(s_i^{+}, z_i^{+}, g_j)\right)\right]. W ij = sg [ softmax j ( e min f ϕ ˉ , e ( s i + , z i + , g j ) ) ] .
where sg \operatorname{sg} sg denotes stop-gradient and ϕ ˉ \bar{\phi} ϕ ˉ is the target critic. For CE ( ℓ , y ) = − ∑ j y j log softmax j ( ℓ ) \operatorname{CE}(\ell, y) = -\sum_j y_j \log\operatorname{softmax}_j(\ell) CE ( ℓ , y ) = − ∑ j y j log softmax j ( ℓ ) and I i j = 1 [ i = j ] I_{ij} = 1[i = j] I ij = 1 [ i = j ] , the loss is
L T D = 1 2 n ∑ i , e [ ( 1 − γ ) CE ( L i , : , e n e x t , I i , : ) + γ CE ( L i , : , e g o a l , W i , : ) ] . (12) \mathcal{L}_{\mathrm{TD}} = \frac{1}{2n}\sum_{i,e}\left[(1-\gamma)\operatorname{CE}(L_{i,:,e}^{\mathrm{next}}, I_{i,:}) + \gamma\operatorname{CE}(L_{i,:,e}^{\mathrm{goal}}, W_{i,:})\right]. \tag*{(12)} L TD = 2 n 1 i , e ∑ [ ( 1 − γ ) CE ( L i , : , e next , I i , : ) + γ CE ( L i , : , e goal , W i , : ) ] . (12)
The first term predicts immediate successors; the second propagates future-state occupancy through behavior continuations using a detached categorical bootstrap [41 ] . Appendix § 13 .§ 5 reports the goal-candidate sampling ablation.
Score-space gate. The score f ϕ f_\phi f ϕ in Equation (4) is the ensemble minimum f min ( s , z , g ) = min e f ϕ , e ( s , z , g ) f_{\min}(s,z,g)=\min_e f_{\phi,e}(s,z,g) f m i n ( s , z , g ) = min e f ϕ , e ( s , z , g ) . For actor-training goals g g g , we fit the state–goal threshold by expectile regression,
L V ^ = E [ ρ τ ( sg [ f min ( s , z , g ) − V ^ ( s , g ) ] ) ] , ρ τ ( u ) = ∣ τ − 1 [ u < 0 ] ∣ u 2 , (13) \mathcal{L}_{\hat{V}}=\mathbb{E}\left[\rho_\tau\left(\operatorname{sg}[f_{\min}(s,z,g)-\hat{V}(s,g)]\right)\right],\qquad\rho_\tau(u)=\left|\tau-\mathbf{1}[u<0]\right|u^2, \tag*{(13)} L V ^ = E [ ρ τ ( sg [ f m i n ( s , z , g ) − V ^ ( s , g )] ) ] , ρ τ ( u ) = ∣ τ − 1 [ u < 0 ] ∣ u 2 , (13)
with the expectile τ \tau τ specified in Table 10 . The gate is sg [ 1 [ f min > V ^ ] ] \operatorname{sg}[\mathbf{1}[f_{\min}>\hat{V}]] sg [ 1 [ f m i n > V ^ ]] . The critic, score threshold, and actor are updated jointly as in Algorithm 1.
Critic features and flow policy. The bilinear critic uses L2-normalized features with temperature τ r e p \tau_{\mathrm{rep}} τ rep :
f ϕ , e ( s , z , g ) = ⟨ ϕ e ( s , z ) / ∥ ϕ e ( s , z ) ∥ 2 , ψ e ( g ) / ∥ ψ e ( g ) ∥ 2 ⟩ τ r e p . f_{\phi,e}(s,z,g)=\frac{\left\langle\phi_e(s,z)/\lVert\phi_e(s,z)\rVert_2,\psi_e(g)/\lVert\psi_e(g)\rVert_2\right\rangle}{\tau_{\mathrm{rep}}}. f ϕ , e ( s , z , g ) = τ rep ⟨ ϕ e ( s , z ) / ∥ ϕ e ( s , z ) ∥ 2 , ψ e ( g ) / ∥ ψ e ( g ) ∥ 2 ⟩ .
The critic dimensions and all flow-training and guided-inference hyperparameters are summarized in Table 10 .
Baseline implementations
Table 1 baselines. All entries of Table 1 are from our runs. The fully supervised GCBC, GCIVL, GCIQL, QRL, and HIQL use the OGBench implementations and pixel-based hyperparameters [30 ] . With 5% labels, GCIVL, QRL, and HIQL use the same implementations and hyperparameters in the latent action space, with the same LAM and decoder as LAPIS. LAPIS† is described in Appendix § 13 .§ 8 . F-GCBC replaces the Gaussian GCBC head with our flow-matching actor and is not an OGBench baseline. GCIQL in Table 1 follows standard goal-conditioned IQL, learning an action-conditioned critic Q ( s , a , g ) Q(s,a,g) Q ( s , a , g ) and value function V ( s , g ) V(s,g) V ( s , g ) and extracting the actor from their advantage. CRL follows the upstream OGBench implementation with a DDPG+BC actor and InfoMax/InfoNCE critic, with hyperparameters tuned per domain. The domain-specific BC coefficient is reported in Table 11 . F-GCBC, CRL, and LAPIS use the shared visual encoder, batch size, and augmentation settings within a domain. The method-specific action-chunk sizes used in Table 1 are summarized in Appendix § 13 .§ 3 , and their sensitivity is examined in Table 13 .
Table 11. CRL-specific OGBench hyperparameter.
Table 13. Success rate (%) of real-action policies with 100% action labels (mean ± standard deviation over four seeds). Results for C = 1 C = 1 C = 1 and LAPIS are reproduced from Table 1. New C = 3 C = 3 C = 3 results average the 300k, 400k, and 500k checkpoints within each seed.
Action-scarce baselines. Table 3 uses goal-conditioned physical-action adaptations designed for limited action supervision. In particular, its GCIVL entry, a value-only variant that the original submission labeled GCIQL, is distinct from the standard action-conditioned GCIQL used in Table 1 ; Appendix § 14 .§ 2 discusses this protocol difference. Let D L \mathcal{D}_L D L contain genuinely labeled transitions and B \mathcal{B} B the action-free transitions, with goal-reaching reward r ( s , g ) r(s,g) r ( s , g ) . We use the following value-learning, labeled-action, and inverse-prediction objectives:
L V = E s , s ′ , g [ L 2 τ ( r ( s , g ) + γ V ˉ ( s ′ , g ) − V ( s , g ) ) ] , \mathcal{L}_V=\mathbb{E}_{s,s',g}\left[\mathcal{L}_2^\tau(r(s,g)+\gamma\bar{V}(s',g)-V(s,g))\right], L V = E s , s ′ , g [ L 2 τ ( r ( s , g ) + γ V ˉ ( s ′ , g ) − V ( s , g )) ] ,
L 2 τ ( u ) = ∣ τ − 1 [ u < 0 ] ∣ u 2 , \mathcal{L}_2^\tau(u)=\left|\tau-\mathbf{1}[u<0]\right|u^2, L 2 τ ( u ) = ∣ τ − 1 [ u < 0 ] ∣ u 2 ,
L π A W R = E ( s , a , g ) ∼ D L [ − exp ( α A ) log π ( a ∣ s , g ) ] , \mathcal{L}_{\pi}^{\mathrm{AWR}}=\mathbb{E}_{(s,a,g)\sim\mathcal{D}_L}\left[-\exp(\alpha A)\log\pi(a\mid s,g)\right], L π AWR = E ( s , a , g ) ∼ D L [ − exp ( α A ) log π ( a ∣ s , g ) ] ,
L I D M = E ( s , a , s ′ ) ∼ D L [ ∥ a ( s , s ′ ) − a ^ ∥ 2 2 ] , a ^ = I a ( s , s ′ ) on B , \mathcal{L}_{\mathrm{IDM}}=\mathbb{E}_{(s,a,s')\sim\mathcal{D}_L}\left[\lVert a(s,s')-\hat{a}\rVert_2^2\right],\qquad\hat{a}=I_a(s,s')\text{ on }\mathcal{B}, L IDM = E ( s , a , s ′ ) ∼ D L [ ∥ a ( s , s ′ ) − a ^ ∥ 2 2 ] , a ^ = I a ( s , s ′ ) on B ,
L B C = E [ − log π ( a ∣ s , g ) ] , \mathcal{L}_{\mathrm{BC}}=\mathbb{E}[-\log\pi(a\mid s,g)], L BC = E [ − log π ( a ∣ s , g )] ,
where V ˉ \bar{V} V ˉ is the target value and τ \tau τ is the upper expectile. The methods combine these objectives as follows:
GCIVL [30 , 22 ] : L V \mathcal{L}_V L V plus a flat AWR actor with the action-free advantage A = V ( s ′ , g ) − V ( s , g ) A=V(s',g)-V(s,g) A = V ( s ′ , g ) − V ( s , g ) . It learns no action-conditioned Q Q Q and corresponds to GCIVL in OGBench.
HIQL [29 ] : L V \mathcal{L}_V L V plus two AWR actors—a high-level actor over subgoals (A h = V ( s t + k , g ) − V ( s , g ) A^h = V(s_{t+k}, g) - V(s, g) A h = V ( s t + k , g ) − V ( s , g ) ) and a low-level actor over actions toward the subgoal (A ℓ = V ( s ′ , s t + k ) − V ( s , s t + k ) A^\ell= V(s', s_{t+k}) - V(s, s_{t+k}) A ℓ = V ( s ′ , s t + k ) − V ( s , s t + k ) , on D L \mathcal{D}_L D L ). We use the OGBench HIQL implementation with its learned subgoal representation, which the high-level actor predicts and the low-level actor conditions on. All HIQL runs in Table 3 use this representation.
VPT-RL (VPT-style action inference) [3 ] : train I a I_a I a via L I D M \mathcal{L}_{\mathrm{IDM}} L IDM , pseudo-label B \mathcal{B} B , then extract a value-guided (classifier-free-guided flow) RL policy from the (pseudo-)labeled real actions—the LAPIS policy objective of Equation (5) applied to actions instead of latents.
SS-ORL [42 ] : offline RL (L V \mathcal{L}_V L V + AWR actor) on the union D L ∪ { ( s , a ^ , s ′ ) : ( s , s ′ ) ∈ B } \mathcal{D}_L \cup\{(s, \hat{a}, s') : (s,s') \in\mathcal{B}\} D L ∪ {( s , a ^ , s ′ ) : ( s , s ′ ) ∈ B } of true and IDM-pseudo-labeled actions.
Action-free value objectives use all transitions. GCIVL’s physical-action actor and HIQL’s low-level actor use genuine labels; HIQL’s subgoal learning also uses action-free trajectories. The inverse-action predictor is fitted on genuine labels, and pseudo-label-based policy objectives use the union of genuine and inferred actions. Architecture and optimization settings follow the shared recipe where applicable.
Additional control and sampling ablations
Goal-candidate sampling. Because the LAM represents H = 3 H = 3 H = 3 endpoint transitions, we test whether future-state candidates for critic training should also be restricted to the same temporal grid. We compare the standard sampling scheme, which allows arbitrary future offsets, with a grid-restricted variant that rounds each trajectory-future offset k k k to H max ( 1 , round ( k / H ) ) H\max(1,\operatorname{round}(k/H)) H max ( 1 , round ( k / H )) , with H = 3 H = 3 H = 3 . Current-state and random-state candidates, actor-goal sampling, the next-state positive, and one-step bootstrapping are unchanged. The ablation therefore changes only the temporal sampling of future candidates in Equation (12) . Table 12 uses the standard LAPIS reference from Table 1 . Standard sampling achieves higher success than grid-restricted sampling on cube (62 vs. 53), scene (29 vs. 23), and antmaze (49 vs. 34), supporting future-goal candidates without a temporal-grid restriction.
Table 12. Goal-candidate sampling ablation for LAPIS (success rate %). Only the trajectory-future candidate component is grid-restricted; the standard reference is from Table 1.
Action chunk size in real-action policies. Table 13 compares action chunk sizes C = 1 C = 1 C = 1 and C = 3 C = 3 C = 3 on three representative tasks, with one action executed before replanning in both settings. This ablation helps separate the policy-improvement mechanism from the method-specific action chunking used in the fully supervised block of Table 1 .
Guidance scale. Table 14 evaluates LAPIS at w ∈ { 0 , 1 , 3 } w \in\{0, 1, 3\} w ∈ { 0 , 1 , 3 } . Here w = 0 w = 0 w = 0 uses the behavior branch, w = 1 w = 1 w = 1 uses the high-value branch trained toward the supported target q G q_G q G , and w = 3 w = 3 w = 3 further extrapolates in the high-value direction. On all visual manipulation tasks, most of the improvement from w = 0 w = 0 w = 0 to the default w = 3 w = 3 w = 3 is already obtained at w = 1 w = 1 w = 1 . For example, success increases from 13 to 53 to 62 on cube-single, from 13 to 29 to 31 on cube-double, and from 18 to 25 to 29 on scene. Thus, stronger guidance provides additional gains in some tasks, but the supported high-value branch itself accounts for the dominant improvement in these manipulation domains.
CFGRL in physical and learned latent action spaces. Because LAPIS adopts classifier-free guided policy extraction, we additionally compare with the value-free goal-conditioned formulation of CFGRL [11 ] . In this setting, standard flow-based GCBC corresponds to w = 1 w = 1 w = 1 , while w > 1 w > 1 w > 1 strengthens goal guidance at inference without learning an explicit value function. We apply the same CFGRL construction both to physical actions and directly to learned latent actions, and sweep w ∈ { 0 , 1 , 3 } w \in\{0, 1, 3\} w ∈ { 0 , 1 , 3 } . In the latent-action setting, CFGRL uses the same latent action model and decoder as LAPIS. Table 15 reports checkpoint-averaged success rates.
Table 15. Goal-conditioned CFGRL in physical and learned latent action spaces. Success rate (%). The comparison focuses on the effect of increasing guidance within each action representation.
Increasing guidance improves physical-action CFGRL most clearly on AntMaze, where success rises from 7.2 at w = 0 w = 0 w = 0 to 24.1 at w = 3 w = 3 w = 3 . In contrast, directly applying the same goal-conditioned guidance mechanism in the learned latent space produces no consistent improvement across the three domains: performance remains nearly unchanged, or decreases, as w w w increases. Thus, the gains associated with classifier-free guidance in an executable physical action space do not automatically transfer when the guided variable is a learned latent action.
This difference is consistent with the cross-representation issue studied in § 5 . CFGRL’s goal-conditioned construction treats the generated variable as the action whose future-goal occupancy defines the improvement signal. For a physical action, this action is executed directly. A learned latent action, however, is an outcome label inferred from an observed transition and must subsequently be decoded into physical control. Improving a distribution over latent actions therefore does not by itself guarantee improvement after decoding: the selected latent must be appropriate for the current state and its represented transition must be faithfully realized by the decoder. These are precisely the additional correspondences captured by supported outcome identifiability and latent-to-physical execution in Theorem 1.
LAPIS addresses this issue before guided inference. Rather than using guidance itself to construct an improved latent policy, LAPIS first scores only observed state–latent transitions and forms the critic-filtered distribution q G ( z ∣ s , g ) q_G(z \mid s,g) q G ( z ∣ s , g ) , whose support is contained in the observed conditional distribution q B ( z ∣ s , g ) q_B(z \mid s,g) q B ( z ∣ s , g ) . The high-value flow branch is then trained toward q G q_G q G , and classifier-free guidance is applied only after this support-aware target has been constructed. This distinction is already visible without extrapolative guidance: at w = 1 w=1 w = 1 , latent CFGRL remains weak on AntMaze-medium, Cube-single, and Scene, whereas the LAPIS high-value branch reaches 51%, 53%, and 25%, respectively (Table 14 ). Thus, the main difference is not the use of classifier-free guidance itself, but the construction of the latent policy-improvement target on observed state–latent transitions.
Action decodability of the latent representation
Probe protocol. We fit a two-hidden-layer MLP (256 units per layer, ReLU activations) from the frozen, unquantized latent z e z_e z e to the current action a t a_t a t . For each OGBench task, we use independently trained H = 3 H=3 H = 3 LAMs at 500k training steps. The MLP is trained on the labeled 5% of training trajectories. For OGBench, probe validation and test trajectories are disjoint subsets of the official validation dataset. For D4RL Kitchen, we use four independently trained partial-data LAMs (seeds 0–3) at 500k steps with H = 3 H=3 H = 3 , and split held-out rendered episodes into disjoint probe validation and test sets. Latent standardization uses probe training data only; we select the MLP by validation MSE over at most 200 epochs and report test R 2 R^2 R 2 in Table 16 and Table 17 .
Table 16.
Table 17.
For action coordinate d d d , let S S E d = ∑ i ( a i d − a ^ i d ) 2 \mathrm{SSE}_d=\sum_i(a_{id}-\hat{a}_{id})^2 SSE d = ∑ i ( a i d − a ^ i d ) 2 and S S T d = ∑ i ( a i d − a ˉ d ) 2 \mathrm{SST}_d=\sum_i(a_{id}-\bar{a}_d)^2 SST d = ∑ i ( a i d − a ˉ d ) 2 on the test set. We report R d 2 = 1 − S S E d / S S T d R_d^2=1-\mathrm{SSE}_d/\mathrm{SST}_d R d 2 = 1 − SSE d / SST d and the variance-weighted aggregate
R O v e r a l l 2 = 1 − ∑ d S S E d ∑ d S S T d . R_{\mathrm{Overall}}^2=1-\frac{\sum_d \mathrm{SSE}_d}{\sum_d \mathrm{SST}_d}. R Overall 2 = 1 − ∑ d SST d ∑ d SSE d .
For Table 16 –Table 18 , we first average the three probe runs within each LAM seed. Per-coordinate entries are then averaged across LAM seeds, while Overall reports the mean ± \pm ± sample standard deviation across LAM seeds.
Table 16 : MLP action-decodability R 2 R^2 R 2 on held-out OGBench trajectories (z e → a t z_e \to a_t z e → a t , H = 3 H=3 H = 3 ). Overall is the variance-weighted R 2 R^2 R 2 . Cube/scene actions have five coordinates; antmaze actions have eight.
Table 17 : MLP action-decodability R 2 R^2 R 2 on held-out D4RL Kitchen trajectories (z e → a t z_e \to a_t z e → a t , H = 3 H=3 H = 3 ). Overall is the variance-weighted R 2 R^2 R 2 .
Temporal gap and action decodability. We compare H = 1 H=1 H = 1 and H = 3 H=3 H = 3 representations using matched samples and frozen 500k LAMs. Both probes predict the same current action a t a_t a t from the unquantized latent z e z_e z e using the MLP protocol above. Table 18 reports variance-weighted test R 2 R^2 R 2 . H = 3 H = 3 H = 3 representation gives higher action decodability in all three domains. A plausible explanation is that a longer observation gap exposes more visible action-dependent changes, making the transition signal easier to retain under visual augmentation and quantization. These probes quantify the action information recoverable from the learned representations.
Qualitative rollouts
Figure 7 compares rollouts from a shared start state with the goal of closing the window, opening the drawer, and keeping the cube unchanged. Each panel overlays the gripper trajectory and two interaction frames on the final state, with per-subtask outcomes below. Recorded simulator positions confirm that LAM+BC opens then re-closes the drawer and leaves the window unchanged; LAM+CRL neither opens the drawer nor closes the window and moves the cube off-target; LAPIS completes all three subtasks.
Figure 7. Qualitative rollouts on scene (5% labels), shared start and goal.
Extraction rule and critic
Table 4 (b) separates the contribution of the extraction rule from that of the critic in the same latent space. All entries use the same frozen LAMs and decoders, 5% action labels, flow-actor architecture, training budget, and evaluation protocol as LAPIS (four seeds; the 300k, 400k, and 500k checkpoints are averaged within each seed). The TD-InfoNCE critic is the LAPIS critic. The IQL critic is goal-conditioned IQL (Kostrikov et al., [22 ] ) over latent actions with expectile 0.9. The extraction rules are as follows.
Gate + flow: the LAPIS actor of Equation (5) , evaluated at w = 3 w = 3 w = 3 . With the TD-InfoNCE critic this is LAPIS (Table 1 ).
AWR-weighted flow: the high-value branch is trained with unnormalized weights exp ( β A ) \exp(\beta A) exp ( β A ) , clipped at 100, instead of the gate (Peng et al., [32 ] ), with a per-environment temperature β \beta β ; the behavior branch is unchanged. Evaluated at w = 3 w = 3 w = 3 .
IDQL-style selection: 32 latents are sampled from the behavior branch (w = 0 w = 0 w = 0 ) of the trained flow actor, and the one with the highest critic value is executed (Hansen-Estruch et al., [19 ] ). With the TD-InfoNCE critic this is LAPIS† (Table 1 ).
DDPG+BC flow: a one-step policy is trained to maximize the critic while being distilled from a flow behavior policy, as in FQL (Park et al., [31 ] ).
Results are in Table 4 (b) (§ 7.4 ); this grid was Table 18 in the previous revision.
Execution failure of generated latents
Protocol. We measure whether latents generated by a policy are executed faithfully, on cube-single and scene with four seeds, at every decision step of the evaluation episodes (five tasks and 20 episodes per task). At each decision step we save the full simulator state, decode the selected latent into its three-action chunk, execute the chunk open loop, re-encode the start observation and the observation three steps later, and record the round-trip error ρ = ∥ I ( o t , o ~ t + 3 ) − z ∥ 2 \rho= \lVert I(o_t, \tilde{o}_{t+3}) - z \rVert_2 ρ = ∥ I ( o t , o ~ t + 3 ) − z ∥ 2 in the standardized quantized latent coordinates used for d r o u n d d_{\mathrm{round}} d round in Figure 3 . We then restore the saved state and continue the episode; for every run we checked that restoring reproduces the simulator state exactly and that probing does not change the episode.
Reference and failure rate. For each task and LAM, we set the simulator to 2,000 dataset states and execute the decoded chunk of the recorded dataset latent z = I ( s t , s t + 3 ) z = I(s_t, s_{t+3}) z = I ( s t , s t + 3 ) , which gives a reference distribution of ρ \rho ρ . A decision step fails when ρ \rho ρ exceeds the 95th percentile of this reference distribution, so dataset latents fail 5% of the time by construction. The threshold is computed for the LAM and decoder that each policy uses.
Settings. LAPIS at w = 3 w = 3 w = 3 (the reported setting); LAPIS at w = 3 w = 3 w = 3 without codebook projection, where the generated continuous latent is decoded directly; AWR-weighted flow (Appendix § 13 .§ 8 ) at w = 3 w = 3 w = 3 ; and max-Q Q Q , CRL-style critic maximization with a flow actor ([10 ] ), as in Table 4 (a).
Table 19. Execution failure rate (%, ↓) of generated latents: the fraction of decision steps whose round-trip error exceeds the 95th percentile of dataset latents. Mean ± std over four seeds. Bold marks the lowest rate among policies in each row.
Results. LAPIS latents fail at nearly the dataset rate on cube-single (5.8% vs. 5%) and at the lowest rate among the compared methods on scene (12.6%), where max-Q Q Q fails about twice as often (24.0%). Removing codebook projection leaves the failure rate unchanged, and the success difference lies within seed variation (Table 20 ). Executability and performance therefore come from the latents generated by the policy itself, not from the projection. For guidance strength, the d r o u n d d_{\mathrm{round}} d round panel of Figure 3 reports the mean round-trip error of LAPIS over w w w on antmaze-medium.
Table 20. Effect of codebook projection on LAPIS at w = 3 w = 3 w = 3 , evaluated on the same checkpoints with and without projection. The success difference is paired by seed (mean ± std over four seeds).
5% labels without LAM
Table 21 reports the 5%-label setting without LAM pretraining, where policies are trained on the labeled trajectories only.
Table 21. Success rate (%) on the OGBench visual suite with 5% action labels and without LAM pretraining.
Additional Discussion of Experimental Settings
This appendix provides additional context for several experimental comparisons whose interpretation involves differences in policy-improvement mechanisms, supervision protocols, decoder training, or environment dynamics. These details are complementary to the implementation specifications in Appendix § 13 and the theory–practice correspondence in Appendix § 12 .
Table 1 shows a substantial gap between fully supervised real-action CRL and real-action LAPIS on AntMaze, particularly on the larger mazes. These results highlight a tradeoff of the support-aware update used by LAPIS under full physical-action supervision.
LAPIS is motivated by a failure specific to learned latent-action spaces: direct value maximization can select latent actions whose represented transitions are not physically realizable after decoding. The latent realizability failure underlying this mismatch is absent when the policy directly outputs physical actions, although ordinary offline-RL distribution shift remains. In this setting, restricting policy improvement toward behavior-supported transitions can be unnecessarily conservative, while more aggressive critic-driven optimization may be beneficial. Conversely, in the learned latent-action setting, direct maximization can move the policy toward unrealizable latents, motivating the supported update used by LAPIS.
The larger gap on more difficult AntMaze tasks is therefore consistent with a cost of conservative policy extraction in long-horizon navigation. Accordingly, the fully supervised results provide physical-action references for the limited-supervision comparisons. Our main question is whether effective policy improvement can be retained after replacing physical actions with learned latent actions under limited action supervision.
Protocol differences across baseline tables
The GCIQL entries in Table 1 and the GCIVL entries in Table 3 (labeled GCIQL in the original submission) differ in their value-learning and policy-extraction objectives. Table 1 uses an action-conditioned critic and a state–goal value function, with policy extraction based on the Q − V Q - V Q − V advantage. This formulation is applied to physical actions at the stated label budgets and to learned latent actions in the LAM setting.
Table 3 uses an action-scarce adaptation that learns a state–goal value function from all observed transitions, without requiring action labels for value learning. Its physical-action policy is trained only on labeled transitions, weighted by the value difference between successive states. The two tables therefore use different methods rather than repeated evaluations of the same procedure. Because the Table 3 variant learns no action-conditioned Q Q Q , it corresponds to GCIVL in OGBench (Park et al., [30 ] ), and we name it accordingly. GCIVL in Table 1 and Table 2 is the standard OGBench GCIVL, trained only on transitions with (physical or latent) actions; the Table 3 variant also learns its value from action-free transitions.
Interpreting decoder label-budget trends
Table 5 isolates decoder supervision from latent representation learning: all label budgets use the same frozen LAM trained with 5% action labels, while only the amount of action supervision available to the decoder is varied. Thus, the non-monotonic performance of the state-marginal decoder across 1%, 5%, and 100% decoder labels is not caused by changes in the learned latent representation.
The state-marginal decoder Ψ ( a ∣ z ) \Psi(a \mid z) Ψ ( a ∣ z ) aggregates physical realizations across states without observing the current state. If the same latent code admits different appropriate controls at different states, broader coverage can make the empirical conditional p ( a ∣ z ) p(a \mid z) p ( a ∣ z ) more heterogeneous, partially offsetting the benefit of additional labeled examples. By contrast, the state-conditioned decoder Ψ ( a ∣ s , z ) \Psi(a \mid s, z) Ψ ( a ∣ s , z ) can use the current state to resolve such variation when sufficient supervision is available.
This provides a plausible interpretation of the pattern in Table 5 : the marginal decoder performs better under 1% and 5% supervision, where sharing action information across states can be beneficial, while the state-conditioned decoder performs better with full supervision. The fact that the marginal cube score at 5% exceeds its 100%-label score should therefore not be interpreted as evidence that fewer labels are intrinsically preferable; rather, performance reflects the interaction between decoder conditioning, labeled-data coverage, and finite-sample optimization. LAPO [36 Sec. 6.2] reports the related observation that a decoder with access only to the latent action plateaus as labeled data grows, which it attributes to latents that are not state-invariant; it does not compare against a state-conditioned decoder. Table 5 thus measures the gap between the analyzed decoder Ψ ∗ \Psi^{*} Ψ ∗ and the implemented Ψ ( a ∣ z ) \Psi(a \mid z) Ψ ( a ∣ z ) , and shows that it depends on the label budget.
Stochastic teleport dynamics
The visual-antmaze-teleport benchmark provides a concrete case in which the deterministic-dynamics assumption used for the exact transfer result does not hold globally. In the OGBench teleport environment, entering an inbound teleport region sends the agent to a randomly selected outbound portal. Thus, the same physical state and control can induce different successor states at a teleport transition. This violates Assumption 1 and means that the exact equality in Theorem 1 is not expected to apply at such transitions. The stochastic teleport implementation can be verified directly in OGBench.
This case is instead described by the approximate-transfer analysis in Appendix § 12 .§ 3 . For a decoded latent policy, Proposition 2 bounds the return discrepancy by
∣ J M ( Ψ ^ ∘ π z ) − J M B z ( π z ) ∣ ≤ ϵ ˉ r 1 − γ + 2 γ R max ( 1 − γ ) 2 ϵ ˉ p \left|J_{\mathcal{M}}(\widehat{\Psi} \circ\pi_z) - J_{\mathcal{M}_{B_z}}(\pi_z)\right| \leq\frac{\bar{\epsilon}_r}{1-\gamma} + \frac{2\gamma R_{\max}}{(1-\gamma)^2}\bar{\epsilon}_p J M ( Ψ ∘ π z ) − J M B z ( π z ) ≤ 1 − γ ϵ ˉ r + ( 1 − γ ) 2 2 γ R m a x ϵ ˉ p
where ϵ ˉ p \bar{\epsilon}_p ϵ ˉ p measures the transition mismatch between latent evaluation and decoded physical execution and ϵ ˉ r \bar{\epsilon}_r ϵ ˉ r measures the corresponding reward mismatch. Random teleportation can increase ϵ p \epsilon_p ϵ p because a latent transition inferred from one observed teleport outcome need not reproduce that same outcome when the decoded action is executed again. The exact deterministic result is recovered when these mismatches vanish.
The teleport task evaluates performance under stochastic transitions, for which the transition- and reward-mismatch framework in Appendix § 12 .§ 3 provides the relevant theoretical perspective.
Standalone F-GCBC and the LAPIS behavior branch
Table 1 reports F-GCBC as a standalone flow-based goal-conditioned behavioral-cloning baseline, whereas w = 0 w = 0 w = 0 in Table 14 evaluates the behavior-conditioned branch of a jointly trained LAPIS actor. Although both are trained from observed latent transitions using flow matching, they are not the same trained policy. LAPIS uses a single conditional velocity model v θ ( ⋅ ; o ) v_{\theta}(\cdot;o) v θ ( ⋅ ; o ) whose o = 0 o = 0 o = 0 branch is trained toward the observed distribution q B q_{\mathcal{B}} q B and whose o = 1 o = 1 o = 1 branch is simultaneously trained toward the critic-filtered distribution q G q_{\mathcal{G}} q G through Equation (5) . The two losses therefore update shared model parameters.
Consequently, setting w = 0 w = 0 w = 0 at inference selects the behavior-conditioned velocity field of the jointly trained two-branch model. Table 14 is an inference-time guidance ablation within a fixed LAPIS actor, whereas Table 1 evaluates a separately trained F-GCBC policy. Their different training objectives and parameter sharing explain the difference between their scores.
Future Work and Limitations
Future Work
A promising direction is to extend LAPIS from goal-conditioned to language-conditioned control. Prior work has shown that language instructions can be grounded into visual goal representations and executed through goal-conditioned policies [28 , 5 ] . This suggests that LAPIS could retain its latent-space policy-improvement framework while replacing explicit visual goals with language-derived goal representations, enabling policy improvement for instruction-following and vision-language-action settings. Extending the framework and its transfer analysis to partially observed settings using temporal or recurrent state representations is another important direction.
Limitations
Our experiments are limited to simulated visual manipulation and navigation benchmarks, and we focus on goal-conditioned control with limited physical-action supervision. Consequently, we do not empirically study objectives with action-dependent rewards, where outcome-based abstraction can create a mismatch between latent and physical policy objectives. Real-robot deployment and language-conditioned control also remain outside the scope of the current evaluation.
Our exact transfer and alignment results rely on deterministic dynamics, supported outcome identifiability, batch-constrained execution through the batch-exact decoder, and Markov state information. Practical LAPIS uses learned visual state representations, a learned state-marginal decoder, an H = 3 H = 3 H = 3 latent representation with one-step replanning, and classifier-free guidance, so these conditions are not instantiated exactly. We explicitly study these departures in Appendix § 12 and through targeted empirical analyses. The approximate-transfer analysis accounts for transition and decoder mismatch, while decoder-conditioning, temporal-horizon, and first-step execution analyses examine learned decoding and the H > 1 H > 1 H > 1 receding-horizon controller. For guided inference, the w = 1 w = 1 w = 1 high-value branch provides the supported target underlying our analysis, and the guidance sweep shows that it already captures most of the improvement on the manipulation tasks; w > 1 w > 1 w > 1 provides an additional inference-time extrapolation beyond this exact support guarantee. The direct CFGRL comparison further shows that guidance alone does not provide consistent improvement when applied directly in the learned latent space, reinforcing the role of support-aware target construction in practical LAPIS.