betaFeedback

WorksComputer Science

Conditional Optimal Transport and Triangular Maps for Imitation Learning

Abstract

We introduce a unified formulation of imitation learning from demonstrations as a conditional sampling problem solved by triangular transport: a policy is the second block of a triangular map that transports a reference distribution to the conditional distribution of the action chunk given the observation history. Unlike in Diffusion Policy, where it is a fixed Gaussian, the reference becomes a design variable; we take it to be the empirical distribution of the demonstrated action chunks and introduce three instances of the formulation: an optimal transport policy (OTP), a flow matching policy (FMP), and a stochastic interpolant policy (SIP). On the Push-T and robomimic Square tasks, the three policies are competitive in task performance with Diffusion Policy and Consistency Policy while using considerably smaller networks and shorter training, and the informative reference accounts for a substantial part of their performance when few network evaluations are used. The three instances offer different trade-offs rather than a single best method: OTP generates an action chunk with a single evaluation of the smallest network (0.23 million parameters on Push-T) and is suitable for settings in which inference time and model size matter, whereas SIP attains the highest scores among the proposed policies at a higher cost.

Introduction

Imitation learning aims to compute a control policy that reproduces the behavior of a demonstrator. Its most widely used form, behavior cloning, casts the problem as supervised learning, with recorded observations as inputs and demonstrated actions as targets ([38]). The conditional action distribution, however, can be multimodal, with several distinct actions being valid under the same observation ([19, 51, 42, 50]). This diversity motivates policies that represent the entire conditional distribution of the action given the observation, so that the problem is one of conditional sampling: given only joint samples of observations and actions, generate actions distributed according to their conditional distribution given a new observation, fast enough to be executed online.

Gaussian-reference policies. Diffusion Policy (DP) [13] represents the conditional distribution of the action by a diffusion model [23] conditioned on the recent observations, and it has become a strong baseline for manipulation tasks. Its sampler, however, generates every chunk of future actions by iteratively denoising a Gaussian sample, at the cost of one evaluation of a large network per integration step: 100 steps in its simulation benchmarks, reduced to 10 by deterministic sampler [54] in its real-world experiments. This cost has motivated samplers with fewer steps [60], distilled policies such as Consistency Policy (CP) [46], and flow-matching policies (see Appendix § 6 for details). All of these start from a Gaussian distribution, which carries no information about the task.

Informative references. A growing body of work replaces the Gaussian reference with a more informative distribution, which we simply refer to as the reference. The adaptation of the reference distribution has already been considered in BRIDGeR [11], which interpolates from a learned or hand-crafted reference policy and reports gains at a small number of integration steps, as well as other methods that construct the reference from previously executed actions [28, 27] or learn it as a function of the observation [15, 35]. All of these approaches, however, require a separately trained model, domain knowledge, or the pairing of every training target with its own historical data (see Appendix § 6 for details). This raises the question of whether a reference can be informative and yet independent of both the observation and the previously executed actions, which would lead to more flexible and efficient algorithms.

In this paper, we build upon the generative filters of Al-Jarrah et al. [4], in which the analysis step of nonlinear filtering is realized by a generative model that transports the forecast distribution of the state to its posterior. The problem solved by a policy, namely sampling from the conditional distribution of the action given the observation using only joint samples of the two, coincides with this conditioning problem. Following a triangular transport formulation [6] [57, 2], we represent a policy as the second block of a triangular map, and we take the reference to be the empirical distribution of the demonstrated action chunks, so that every action chunk is obtained by transporting one that the demonstrator executed. To our knowledge, no existing policy takes as reference the empirical distribution of the demonstrated action chunks itself (see Appendix § 6 for details). Three policies are derived in this manner, each from one of the filters of Al-Jarrah et al. [4]: the optimal transport filter (OTF) (Al-Jarrah et al., [2]) yields the optimal transport policy (OTP), which requires a single network evaluation per action chunk; the flow matching filter (FMF) yields the flow matching policy (FMP), realized by an ordinary differential equation (ODE) (Lipman et al., [34]); and the stochastic interpolant filter (SIF) yields the stochastic interpolant policy (SIP), realized by a stochastic differential equation (SDE) (Albergo et al., [5]). In addition, the stochastic map filter of Spantini et al. [55], which is based on the Knothe–Rosenblatt rearrangement and is considered in Al-Jarrah et al. [4], yields the Knothe–Rosenblatt policy (KRP), which we include as a Gaussian-reference counterpart of the proposed policies. We refer to these methods as policies rather than filters, since each conditions the action chunk on the observation history at a single re-planning instant, without the recursive prediction and update in filters.

The contributions of this paper are as follows.

  1. Formulation. We formulate imitation learning with action chunks as a conditional sampling problem and represent a policy as the second block of a triangular transport map, so that the reference distribution becomes a design variable and DP and CP are recovered as the instances with a Gaussian reference (§ 3.1 and Appendix § 9).

  1. Policies, reference, and deployment. Within this formulation, we derive three policies, OTP, FMP, and SIP, which learn the map by conditional optimal transport (OT), flow matching, and stochastic interpolants, respectively, with the reference PAP_A, which requires no auxiliary model and no pairing of the training data, together with a deployment protocol under which the reference has the same distribution during training and deployment (§ 3); OTP is learned in a single training stage and evaluates, on Push-T, a network of 0.23 million parameters once per action chunk.

  1. Evaluation. We compare the proposed policies with KRP, DP, and CP under a common protocol, on the Push-T benchmark and on the Square task of robomimic, in terms of task performance and of training and online cost (§ 4), and we show that the reference PAP_A is responsible for a substantial part of their performance, in particular with few network evaluations, the regime of interest for online deployment.

The remainder of this paper is organized as follows. § 2 formulates imitation learning with action chunks as a conditional sampling problem. § 3 presents the triangular formulation, the proposed policies, and the deployment protocol. § 4 compares the proposed policies with the Gaussian-reference policies on Push-T and Square, and § 5 discusses the results, the limitations, and future work. The appendices provide supplementary material.

Problem Formulation

Consider a system operated by a demonstrator at the discrete time instants k=0,1,2,…k = 0, 1, 2, \ldots, where ok∈Rdoo_k \in\mathbb{R}^{d_o} denotes the observation and ak∈Rdaa_k \in\mathbb{R}^{d_a} the action applied at time kk. A demonstration data set D={(okj,akj):k=0,…,Kj−1, j=1,…,J}\mathcal{D} = \{(o_k^j, a_k^j) : k = 0, \ldots, K_j - 1,\ j = 1, \ldots, J\} consists of JJ episodes, the jj-th of which has length KjK_j. Following Zhao et al. [61]; ); Chi et al. [13], the policy returns a chunk of future actions conditioned on a short history of observations; to this end, we define the observation history and the action chunk

Sk:=(ok−Ho+1,…,ok)∈Rm,Ak:=(ak,…,ak+Hp−1)∈Rn,(1)S_k := (o_{k-H_o+1},\ldots,o_k) \in\mathbb{R}^{m}, \qquad A_k := (a_k,\ldots,a_{k+H_p-1}) \in\mathbb{R}^{n}, \tag*{(1)}

where HoH_o is the observation horizon, HpH_p is the prediction horizon, m=Hodom = H_od_o, and n=Hpdan = H_pd_a. We take SkS_k to consist of observations only, since a policy conditioned on its own past actions tends to copy them rather than respond to the observations [59, 16]. Collecting the pairs (1) from the demonstration data set D\mathcal{D} yields the training data {(Si,Ai)}i=1M\{(S^i,A^i)\}_{i=1}^{M}, where MM denotes the number of available training data pairs. We regard the training data as samples of the joint distribution PS,AP_{S,A} of the observation history and the action chunk under the demonstrator, with marginals PSP_S and PAP_A; every expectation with respect to these distributions is approximated by an average over the training data, and samples of PAP_A are drawn uniformly from the MM demonstrated action chunks. At deployment, the policy is executed in a receding-horizon fashion: at a re-planning instant kk, an action chunk A^k=(a^k,…,a^k+Hp−1)\hat{A}_k = (\hat{a}_k,\ldots,\hat{a}_{k+H_p-1}) is sampled given SkS_k, its first Ha≤HpH_a \le H_p actions are applied, and the next chunk is sampled at time k+Hak + H_a. At the first instants of an episode, k<Ho−1k < H_o - 1, the missing observations of SkS_k are replaced by the first observation o0o_0, as in the training data. The horizons HoH_o, HpH_p, and HaH_a are selected separately for each policy (Section 4.2).

Problem statement. Let {(Si,Ai)}i=1M\{(S^i,A^i)\}_{i=1}^{M} be the training data, and let PZP_Z be a reference distribution on Rn\mathbb{R}^{n} from which samples can be drawn. The goal is to construct a map TT that reproduces the conditional distribution of the action chunk given the observation history:

Given:(Si,Ai)i=1M,(Si,Ai)∼PS,A,Find:a map T:Rm×Rn→Rn such that T(s,⋅)#PZ=PA∣S=s for PS-a.e. s∈Rm,\begin{aligned} \text{Given:}\quad&(S^i,A^i)_{i=1}^{M}, \quad(S^i,A^i) \sim P_{S,A},\\ \text{Find:}\quad&\text{a map }T:\mathbb{R}^{m}\times\mathbb{R}^{n}\to\mathbb{R}^{n}\text{ such that }T(s,\cdot)_{\#}P_Z=P_{A\mid S=s}\text{ for }P_S\text{-a.e. }s\in\mathbb{R}^{m}, \end{aligned}

where #\# denotes the push-forward operator. A solution defines a stochastic policy, in which the action chunk A^k=T(Sk,Zk)\hat{A}_k = T(S_k,Z_k) is computed from a reference sample Zk∼PZZ_k \sim P_Z drawn afresh, independently of SkS_k, at every re-planning instant.

This problem coincides with the conditioning problem of the analysis step of nonlinear filtering [4], with the observation history in place of the observation and the action chunk in place of the state, except that the map is learned once, offline, from recorded demonstrations, and that, at deployment, the observation history is generated in closed loop and may leave the support of PSP_S, a phenomenon known as covariate shift [49]. Consequently, a policy is assessed by the closed-loop performance on the task (Section 4) rather than by a distance between conditional distributions.

Generative Transport Policies

Conditioning with triangular maps

The policies studied in this paper share one element: each is realized by a triangular map, learned from the training data by a generative modeling framework. We say that a map G:Rm×Rn→Rm×RnG:\mathbb{R}^{m}\times\mathbb{R}^{n}\to\mathbb{R}^{m}\times\mathbb{R}^{n} is triangular if there exists a map T:Rm×Rn→RnT:\mathbb{R}^{m}\times\mathbb{R}^{n}\to\mathbb{R}^{n} such that

G(s,z)=(s,T(s,z)),∀ (s,z)∈Rm×Rn,(2)G(s,z) = (s,T(s,z)), \qquad\forall\,(s,z)\in\mathbb{R}^{m}\times\mathbb{R}^{n}, \tag*{(2)}

that is, GG holds the observation history fixed. For some of the policies, such as the stochastic samplers of Appendices § 8.§ 3 and § 9.§ 2, the map TT depends on auxiliary randomness that is independent of (s,z)(s,z). In such cases, without loss of generality, the reference is extended by the auxiliary variable. Triangular maps are useful for conditioning because of the following result ([6, 2]), whose short proof is given in Appendix § 7.

Proposition 1 Let PS,AP_{S,A} be a probability distribution on Rm×Rn\mathbb{R}^{m} \times\mathbb{R}^{n} with marginal PSP_S on Rm\mathbb{R}^{m}, let PZP_Z be a probability distribution on Rn\mathbb{R}^{n}, and let GG be a measurable triangular map of the form (2). If G#(PS⊗PZ)=PS,AG_{\#}(P_S \otimes P_Z) = P_{S,A}, then T(s,⋅)#PZ=PA∣S=sT(s,\cdot)_{\#}P_Z = P_{A\mid S=s} for PSP_S-almost every ss, where PA∣S=sP_{A\mid S=s} denotes the conditional distribution obtained by disintegrating PS,AP_{S,A} with respect to PSP_S.

This result suggests a recipe for converting a generative model into a policy (Al-Jarrah et al. [4]): (i) let the target be the joint distribution PS,AP_{S,A} of the training data; (ii) let the source be the product distribution PS⊗PZP_S \otimes P_Z, whose samples are obtained by pairing every observation history with a reference sample drawn independently of it; and (iii) train a generative model to learn a (deterministic or stochastic) map of the triangular form (2) that transports the source to the target. Any such map satisfies the hypothesis of Proposition 1, and its second block consequently solves the problem of § 2. This condition does not specify TT uniquely, and the policies presented below differ precisely in how a particular solution is selected and learned.

The reference PZP_Z is a design choice that Proposition 1 leaves entirely free: DP, CP, and the flow-matching and distilled policies take PZ=N(0,In)P_Z = \mathcal{N}(0,I_n), whereas the informative references of § 1 depend on the observation or on the previously executed actions, so that the source is no longer of the form PS⊗PZP_S \otimes P_Z. We instead take PZ=PAP_Z = P_A, so that the policy reads

A^k=T(Sk,A‾k),A‾k∼PA,(3)\hat{A}_k = T(S_k,\overline{A}_k), \qquad\overline{A}_k \sim P_A, \tag*{(3)}

where A‾k\overline{A}_k is drawn independently of SkS_k at every re-planning instant. This reference is supported on the actions applied by the demonstrator, which are confined to the reachable workspace and vary smoothly in time, properties that a Gaussian does not possess (Figure 1). Moreover, since PAP_A is the mixture of the conditional distributions PA∣S=sP_{A\mid S=s} over s∼PSs \sim P_S, the transport to PA∣S=sP_{A\mid S=s} is expected to require a smaller displacement from PAP_A than from a Gaussian, which would simplify the learning of the map and reduce the number of integration steps, consistently with the observations of Chen et al. [11]; Kang et al. [28] for informative references. For OTP and FMP, whose maps are deterministic, the randomness of the policy is thus inherited entirely from the demonstrations, and, for SIP, it is supplemented by the noise of the sampler.

Figure 1. One Push-T episode under OTP (left) and CP 1-step (right) from the same observation ss, up to step 61, at which OTP completes the task. Every action chunk is the image of a fresh sample of the reference, PAP_A for OTP and N(0,In)\mathcal{N}(0,I_n) for CP (gray; about half of the points of the latter lie outside the frame), and dotted lines connect the first action of each sample to that of its image. The executed actions are colored by the score after their execution, and CP completes this episode at step 265 (see Appendix § 12 for details).

The proposed policies

Following Al-Jarrah et al. [4], we obtain three policies with the reference PAP_A, each of which realizes the map TT of (3) by the generative modeling framework of one of the filters of that work (see Appendix § 8 for details). As summarized in Table 1, they differ in the optimization problem by which TT is learned, in the form of the transport at deployment, and in the randomness of the policy at a fixed reference sample: OTP, derived from the optimal transport filter (Al-Jarrah et al. [2]), learns the conditional OT map from a max–min problem and transports the reference by a one-shot map; FMP, derived from the flow matching filter, learns a vector field by least-squares regression (Lipman et al. [34]) and transports the reference along a deterministic flow, an ODE; and SIP, derived from the stochastic interpolant filter, learns a vector field and a score by two least-squares regressions [5] and transports the reference along a stochastic flow, an SDE. Consequently, OTP trades the stability of a regression objective for an online cost of a single network evaluation, whereas FMP and SIP are trained by regression at the price of 200 network evaluations; we note that SIP is formally the instance of BRIDGeR [11] whose source policy is the action marginal.

PolicyReferenceTransport / couplingOptimizationTransport at deployment (NFE)
OTPPAP_AOT map, independent couplingmax–min (4)one-shot map (1)
FMPPAP_Adeterministic interpolant (7)least squares (6)deterministic flow, ODE (5), Heun (200)
SIPPAP_Astochastic interpolant (8)least squares (6), (9)stochastic flow, SDE (10)†, stochastic Heun (200)
KRPN(0,In)\mathcal{N}(0, I_n)Knothe–Rosenblatt map, inverse direction (11)maximum likelihood (12)inverse map, coordinate-wise Newton (≤192\le192)‡
DPN(0,In)\mathcal{N}(0, I_n)noising processdenoising (13)stochastic chain, ancestral sampler† (100)
DPRN(0,In)\mathcal{N}(0, I_n)noising process, continuous timedenoising (13)deterministic flow, probability-flow ODE, Heun (200)
CPdistilled from a diffusion teacherconsistency distillationone-shot map (1) or three-step map† (3)

Table 1. Summary of the policies evaluated in this paper; related policies are classified in Table 4.

Deployment protocol

The deployment of the proposed policies is summarized by Algorithm 1 in Appendix E. The reference sample Aˉk\bar{A}_k is drawn afresh from the demonstrations at every re-planning instant, independently of the observation history, so that the reference has the same distribution during training and deployment. Reusing instead the action chunk produced at the previous re-planning instant violates this independence, and it was markedly inferior in preliminary experiments.

Numerical Results

In this section, we list the policies that are compared, describe the benchmarks and the evaluation protocol, and present the results, with an emphasis on the trade-off between task performance and computational cost. The code that reproduces all the experiments is publicly available at https://github.com/Mohd9485/transport-policies.

Policies compared

Table 1 lists the policies, which fall into two groups. The first comprises the proposed policies OTP, FMP, and SIP of § 3.2, which transport the reference PAP_A. The second comprises four policies that transport the Gaussian reference N(0,In)\mathcal{N}(0,I_n), each of which is an instance of the formulation of § 3.1: KRP learns the triangular map in the inverse direction, from the data to the reference, by maximum likelihood, and inverts it coordinate by coordinate at deployment; DP [13] generates an action chunk by 100 steps of an ancestral sampler; DPR (Diffusion Policy with a residual network) trains the objective of DP on the residual network and the integrator of FMP and SIP, so that its comparison with SIP separates the effect of the reference from that of the network; and CP [46] distills a diffusion teacher into a student that is evaluated once (CP 1-step) or three times (CP 3-step; see Appendix § 9 for details). Two variations are, in addition, examined. On Push-T, OTP, FMP, and SIP are trained with the reference N(0,In)\mathcal{N}(0,I_n) in place of PAP_A, and FMP, SIP, and DPR are deployed with fewer integration steps. On Square, every policy is evaluated without and with two modifications, the coordinate-wise mean of 8 sampled chunks and Gaussian noise added to the state of the robot during training (§ 4.3).

Benchmarks and evaluation protocol

We consider the state-based Push-T task [19, 13], in which a circular end effector moving in a plane must push a T-shaped block from a randomized initial pose into a fixed target pose, with ak∈R2a_k \in\mathbb{R}^2 and ok∈R20o_k \in\mathbb{R}^{20}. The data set consists of J=206J = 206 human demonstrations with M=25,650M = 25{,}650 time instants in total. As in Chi et al. [13], the reward is rk=min⁡{ck/0.95,1}r_k = \min\{c_k/0.95, 1\}, where ck∈[0,1]c_k \in[0,1] is the fraction of the target region covered by the block at time kk, an episode terminates once ck>0.95c_k > 0.95 or after 300 steps, its score is max⁡krk\max_k r_k, and it

NFE: network evaluations per action chunk. †Stochastic inference: repeated evaluation at a fixed reference sample yields different action chunks. ‡Evaluations of a single component of Φθ\Phi_\theta, each at 16 quadrature nodes: at most 12 per coordinate of the action chunk (2 to bracket the root and at most 10 Newton steps), for n=16n = 16, after one evaluation of the whole map that yields the reference sample.

is called a success if its score equals one; the score of a policy is its mean episode score. Every policy is trained with three training seeds on all 206 demonstrations and evaluated, with the final weights of every training run and without any selection on the evaluation episodes, on the same 500 held-out initial conditions, so that each policy is assessed on the same 1,500 episodes. The hyperparameters of the proposed policies are listed in Table 5, the baselines are used with their published settings, and the horizons (Ho,Hp,HaH_o, H_p, H_a) are (1,4,4)(1, 4, 4) for OTP and FMP, (1,8,8)(1, 8, 8) for SIP and KRP, and (2,16,8)(2, 16, 8) for DP and CP (see Appendices E and § 12 for details). Figure 4 in Appendix § 12 shows the final configurations of nine held-out episodes for every policy.

OTPFMPSIP
Horizons (Ho,Hp,HaH_o, H_p, H_a)(1,4,4)(1, 4, 4)(1,4,4)(1, 4, 4)(1,8,8)(1, 8, 8)
Networksff: width 576, 3 residual blocks; TT: width 192, 3 residual blockswidth 896, 5 residual blocks, layer normalizationwidth 512, 5 residual blocks, layer normalization
Activationmishmishrelu
Time embedding (dimension)–2010
OptimizerAdamW, weight decay 0AdamW, weight decay 0.2AdamW, weight decay 0.01
Learning rate5.97×10−55.97 \times10^{-5} (ff), 7.23×10−47.23 \times10^{-4} (TT)9.33×10−49.33 \times10^{-4}1.92×10−31.92 \times10^{-3}
Schedulecosine, 5% warm-upcosine, 2% warm-upcosine, 2% warm-up
Gradient clipping0.251.00.25
Batch size576256320
Iterations16,384, with Kin=25K_{\mathrm{in}} = 2516,38465,536
Noise level ε\varepsilon–00.115
Samplerone evaluationHeun, Nτ=100N_\tau= 100stochastic Heun, Nτ=100N_\tau= 100, λ=1\lambda= 1

Table 5. Hyperparameters of the proposed policies on Push-T.

Figure 4. Final configurations of nine held-out Push-T episodes, drawn at random and identical for every policy, under the first training seed of each policy: target pose (green), T block at the start (gray), end effector at the start (circle), and T block at the end of the episode (dashed); the number in each scene is the score of the episode. Above each grid: mean score ± one standard error over the 1,500 episodes of Table 2, and the time per episode.

We consider, in addition, the Square task of robomimic ([38]), simulated in robosuite ([62]), in which a robot arm must grasp a square nut and place it on a peg; the data set consists of the 200 proficient-human demonstrations (30,154 time instants), with ok∈R23o_k \in\mathbb{R}^{23} and ak∈R10a_k \in\mathbb{R}^{10}, and an episode is a success if the nut is placed within 400 steps. The evaluation protocol is the same, and the two modifications of § 4.1 were selected on a disjoint block of initial conditions. Figure 5 shows the final configurations of six held-out episodes for every policy, without and with the two modifications (see Appendix § 13 for details).

Figure 5. Final configurations of six held-out Square episodes, seen from above, drawn at random and identical for every policy and configuration, under the first training seed of each policy; for each policy, the left 3×23 \times2 block is the policy without modification and the right block the policy with both modifications (state noise and mean of 8 sampled chunks). Each scene shows the square peg (green), the nut at the start (gray), the end effector at the start (circle), and the nut at the end of the episode (dashed); a check mark denotes a success and a cross a failure. Above each block: success rate ±\pm one standard error over the 1,500 episodes of Table 3; above each policy: its time per episode.

Results

Results on Push-T. Table 2 reports the task performance and the computational cost on Push-T, and Figure 2 relates the two on both tasks. On Push-T, the three proposed policies attain scores at least as high as those of the Gaussian-reference baselines DP, KRP, and CP, with SIP the highest overall, while using networks an order of magnitude smaller than the U-Nets of DP and CP and training that requires minutes rather than hours; the differences are assessed by a bootstrap over the initial conditions and the training seeds (see Appendix § 12 for details). The comparison of DPR with SIP indicates that, with 100 or more network evaluations, the gap between DP and the proposed policies is due mainly to the network and its training rather than to the reference.

PolicyScore↑\uparrowSeed spread↓\downarrowSuccess [%]↑\uparrowFailure [%]↓\downarrowNFE↓\downarrowParam. [M]↓\downarrowTraining [min]↓\downarrowTime [s]↓\downarrow
OTP0.943 ±\pm 0.0050.01651.24.710.23a^{a}10.60.12
FMP0.947 ±\pm 0.0040.00852.14.42008.100.63.74
SIP0.965 ±\pm 0.0040.00566.73.12002.671.92.10
KRP0.914 ±\pm 0.0060.01855.68.2≤192b\le192^{b}6.887.31.25
DP0.915 ±\pm 0.0070.00466.78.610065.8268.35.40
DPR0.963 ±\pm 0.0040.00956.63.12008.472.42.84
CP 1-step0.938 ±\pm 0.0060.01163.05.9169.5c^{c}309.6c^{c}0.17
CP 3-step0.940 ±\pm 0.0060.01162.75.73as CP 1-stepas CP 1-step0.29

Table 2. Task performance and computational cost on Push-T, over three training seeds and 500 held-out initial conditions (1,500 episodes per policy).

Figure 2. Score (Push-T) and success rate (Square) against the time per episode, mean ±\pm one standard error over 1,500 episodes; on Square, every policy is trained with state noise and executes the mean of 8 sampled chunks (§ 4.3), and the open circle marks OTP without these modifications.

Score: mean episode score ±\pm one standard error over the 1,500 episodes. Seed spread: population standard deviation of the three per-seed mean scores. Success: score equal to one. Failure: score below 0.5. NFE: network evaluations per action chunk. Param.: parameters of the network evaluated at inference. Training: wall-clock time of one training run, mean over the three training seeds, each run alone on one GPU. Time: wall-clock time of the evaluation divided by its 1,500 episodes, including the simulation of the environment, a measure of throughput rather than of latency. The open-loop replay of the 206 demonstrations scores 0.947.

Effect of the reference. The reference becomes decisive when the number of network evaluations is small. The two policies that produce an action chunk with a single network evaluation, OTP and CP 1-step, attain comparable scores at comparable online times, but they obtain the single evaluation differently: CP distills a diffusion teacher into a one-step student in two training stages, whereas OTP learns the conditional OT map from PAP_A directly, in a single stage and with a network approximately 300 times smaller. Figure 3 isolates the effect of the reference: with 100 integration steps, PAP_A and N(0,In)\mathcal{N}(0,I_n) attain comparable scores, whereas, as the number of steps is reduced, PAP_A retains the score markedly better, and, for OTP, replacing PAP_A by N(0,In)\mathcal{N}(0, I_n) reduces the score from 0.943 to 0.380. The reference PAP_A therefore accounts for a substantial part of the performance of the proposed policies in the regime of few network evaluations, which is the regime of interest for online deployment (see Appendix § 12 for details).

Figure 3. Effect of the reference on Push-T: score against the number of integration steps NτN_{\tau} with PAP_A (filled markers, solid lines) and N(0,In)\mathcal{N}(0, I_n) (open markers, dashed lines); mean ±\pm one standard error (see Appendix § 12 for details). Arrows: the gain of OTP from N(0,In)\mathcal{N}(0, I_n) to PAP_A, and the gap between DPR and SIP with N(0,In)\mathcal{N}(0, I_n) at Nτ=3N_{\tau}=3.

Results on Square. Square requires the nut to be placed with a precision of a few millimeters, and Table 3 shows that, without modification, the single-sample OTP falls behind SIP and DP. Its deficit was traced to two causes: the variation of a sampled chunk with the reference sample, which the mean of 8 chunks reduces, and the extrapolation of the map away from the demonstrated states, which the state noise reduces (see Appendix § 13 for details). The state noise improves every policy, whereas the averaging is beneficial only when the variation with the reference sample dominates the error, as for OTP. With both modifications, OTP matches the other one-step policy, CP 1-step, in success rate and online time, and it trades a deficit of 0.03 to 0.05 in success rate, relative to SIP and DP, for an order of magnitude in online time.

Table 3: Success rate and computational cost on Square, over three training seeds and 500 held-out initial conditions (1,500 episodes per entry), without and with the two modifications of § 4.1.

PolicyPlain↑\uparrowMean of 8↑\uparrowState noise↑\uparrowBoth↑\uparrowParam. [M]↓\downarrowTraining [min]↓\downarrowTime [s]↓\downarrow
OTP0.768 ±\pm 0.0110.871 ±\pm 0.0090.841 ±\pm 0.0090.895 ±\pm 0.00811.0a113.70.12
FMP0.856 ±\pm 0.0090.865 ±\pm 0.0090.907 ±\pm 0.0080.901 ±\pm 0.00812.818.60.98
SIP0.913 ±\pm 0.0070.897 ±\pm 0.0080.936 ±\pm 0.0060.921 ±\pm 0.00724.134.91.24
KRP0.753 ±\pm 0.0110.721 ±\pm 0.0120.821 ±\pm 0.0100.761 ±\pm 0.01118.261.10.84
DP0.938 ±\pm 0.0060.939 ±\pm 0.0060.953 ±\pm 0.0050.947 ±\pm 0.00665.9322.31.24
DPR0.896 ±\pm 0.0080.891 ±\pm 0.0080.942 ±\pm 0.0060.935 ±\pm 0.00618.246.01.05
CP 1-step0.879 ±\pm 0.0080.880 ±\pm 0.0080.889 ±\pm 0.0080.900 ±\pm 0.00869.6c317.0c0.12
CP 3-step0.887 ±\pm 0.0080.881 ±\pm 0.0080.897 ±\pm 0.0080.901 ±\pm 0.008as CP 1-step0.13

Table 3. Success rate ±\pm one standard error over the 1,500 episodes. Mean of 8: the executed chunk is the coordinate-wise mean of 8 sampled chunks. State noise: Gaussian noise of standard deviation 0.02, in normalized units, added to the state of the robot during training. Both: state noise and mean of 8. The largest entry of each column is in bold. Training: with state noise. Time: one sample per chunk, with state noise. Param., Training, Time, aa, and cc: as in Table 2.

Discussion

The results of § 4 indicate that transporting demonstrated action chunks yields policies that are competitive in task performance with DP and CP on both tasks, while using considerably smaller networks and shorter training.

As for the corresponding filters in Al-Jarrah et al. [4], OTP is the least expensive and SIP the most accurate of the proposed policies. In particular, OTP produces an action chunk with a single evaluation of a network of 0.23 million parameters on Push-T, learned in a single training stage, without the teacher and the distillation that CP requires, which makes it suited to online settings in which an action must be produced without delay. We note that these results were obtained with deliberately small networks, of at most 8.1 million parameters on Push-T and 24.1 million on Square, against 65.8 and 65.9 million for DP, respectively, drawn from a common family of fully connected networks, so that the reported scores do not represent the best attainable by the proposed policies (see Appendix § 14 for details).

Moreover, the results on Square indicate that the rule by which the executed action is selected from the distribution represented by the policy deserves attention in its own right: the mean of sampled chunks benefits OTP at an unchanged time per episode, but not SIP or DP, and it is not expected to be adequate for multimodal conditional distributions. Several limitations of the present study are noted: the comparison is restricted to two state-based tasks, of which Push-T has a conditional action distribution close to unimodal (Mazza et al. [40]), so that the representation of multimodal conditional distributions, which motivates the formulation, is not tested directly; the baselines are used with their published settings, whereas the settings of the proposed policies were chosen for the present study; the policies are exposed to covariate shift; Proposition 1 provides no guarantee for approximately trained maps or for the closed-loop system; and OTP produces, at a fixed observation history, a distribution supported on at most MM demonstrated chunks (see Appendix § 14 for the corresponding remedies). Future work will focus on these extensions, on fast and more sophisticated sampling rules that exploit the distribution of action chunks represented by the policy, on observations from images, and on the evaluation of the policies on physical systems.

Generative AI Use

Generative AI tools were used for coding and experimental implementation, literature search and synthesis, and writing and editing. The authors reviewed and verified all AI-assisted content and take responsibility for the final content of the paper.

Acknowledgments

This work was supported by the National Science Foundation (NSF) awards EPCN-2347358 and CAREER-2440861 and the Air Force Office of Scientific Research (AFOSR) award FA9550261B219.

Extended Related Work

In this appendix, we complete the comparison of § 1 with the generative policies whose references or couplings are closest to those of the proposed policies; Table 4 classifies them by the elements of the triangular formulation of § 3.1.

PolicyReference PZP_ZCoupling during trainingLearningInferenceNFEAuxiliary model
IMLE Policy ([47])Gaussian latentnearest generated sampleimplicit maximum likelihoodone-shot map1none
COT Policy ((Sochopoulos et al., 2025))Gaussianconditional minibatch OTflow matchingODE2OT solver in training
DA-SIP ([14])Gaussianindependentinterpolant regressionODE or SDE†1–100difficulty classifier
BRIDGeR ([11])source policy π0(⋅∣s)\pi_0(\cdot\mid s): Gaussian, CVAE, or heuristicconditionally independent given ssinterpolant regressionSDE†, Euler–Maruyama5–160‡source policy
WarmPrior ([28])Gaussian centered at the previous chunkpaired in timeflow matchingODE1–9none
Temporal Policy ([41])state historypaired in timeinterpolant regressionODE or SDE†10none
LeaP ([15])learned Gaussian, conditioned on proprioceptionconditionally independent given ssflow matching, likelihood and contrastive alignmentODE6‡prior network
BridgePolicy ([35])embedding of the observationpaireddiffusion bridgebridge sampler†10fusion module, semantic aligner

Table 4. Generative policies related to the proposed policies, classified by the elements of the triangular formulation of § 3.1; the entries are taken from the cited works, and these policies are not evaluated in this paper.

Gaussian-reference policies. The cost of the sampler of DP (§ 1) has motivated deterministic samplers with fewer steps [54], distillation into one or a few steps [46, 58], flow-matching policies [34] whose number of integration steps is reduced by adaptive, consistency, mean-velocity, or minibatch optimal transport techniques [24, 14, 52] (Zhang et al. 2025; Sochopoulos et al. 2025), and one-step generators trained by implicit maximum likelihood estimation [47]. In all of these approaches, the generation starts from a Gaussian distribution, which carries no information about the task, so that the structure of the action distribution must be produced entirely by the learned map. In the formulation of § 3.1, DP and the flow-matching, consistency, and distilled policies of [24, 9, 46, 58] take PZ=N(0,In)P_Z = \mathcal{N}(0, I_n).

Informative references. BRIDGeR [11] connects a source policy π0(⋅∣s)\pi_0(\cdot\mid s) to the demonstrated actions through a stochastic interpolant, and reports that an informative source improves the performance at a small number of integration steps; similar observations are made for diffusion bridges in visual navigation [48]. On its Sphere Push-T task, the Riemannian flow-matching policy of Ding et al. [17] uses a uniform prior on the sphere, which, like PAP_A, is independent of both the observation and the history and outperforms a wrapped Gaussian prior at a single integration step, but which is fixed by the geometry of the action space rather than drawn from the demonstrations. A second group of methods constructs the reference from the recent behavior of the agent: WarmPrior [28] centers a Gaussian at the previously executed action chunk, Streaming Flow Policy [27] starts every flow at the last action, A2A [26] encodes the executed actions into a latent source, and Temporal Policy [41] constructs an interpolant from the state history of the robot, so that in each case the reference is paired with the target through time and its distribution at deployment is determined by the closed-loop behavior of the policy. A third group learns a reference that depends on the observation, namely a parametric Gaussian [15], a library of Gaussians selected by a latent action [36], or an embedding of the observation itself [35, 20]. Finally, the reverse process of a pretrained DP has been shortened by starting it from an informative chunk, namely the demonstrated action whose recorded observation is nearest to the current observation [44] or a chunk predicted by an auxiliary network from the current observation and the previous chunk STEP [32], and Streaming DP [25], RNR-DP [12], and real-time chunking [10] reuse previously generated actions across re-planning instants; these methods retain the learned transport of a diffusion or flow policy trained with the Gaussian reference.

Optimal transport in generative policies. The max–min formulation (4) follows the approach of Makkuva et al. [37] to the computation of OT maps, extended to conditional, block-triangular maps in [1, 2]. OT has previously been incorporated into generative policies through minibatch couplings between a Gaussian reference and the demonstrated actions (Sochopoulos et al., 2025), where it was shown that a coupling that ignores the observation fails in conditional tasks, and, in reinforcement learning, through the interpretation of a diffusion policy as a transport map whose cost is given by the critic [56]; in (4), by contrast, the conditional OT problem is posed between PAP_A and PA∣S=sP_{A\mid S=s} for every ss simultaneously, so that the conditioning is respected by construction and no assignment problem is solved during training. We also remark that the adversarial character of (4) is distinct from that of generative adversarial imitation learning [22], which matches the occupancy measures of states and actions through interaction with the environment, whereas (4) is solved offline, from the demonstrations alone.

Stochastic interpolant policies. Stochastic interpolants have previously been used for policies with a Gaussian reference [14], with the state history of the robot as reference [41], and with an arbitrary source policy π0(⋅∣s)\pi_0(\cdot\mid s) [11]. SIP is the instance of the last of these constructions whose source policy is π0(⋅∣s)≡PA\pi_0(\cdot\mid s) \equiv P_A for every ss, up to the noise schedule, the discretization, and the network; it differs from the instances evaluated in [11], whose sources are a Gaussian, a conditional variational autoencoder trained on the demonstrations, or a hand-crafted policy, in that its reference is independent of the observation and requires no model other than the data. In particular, the SDE of Chen et al. [11] is integrated by the Euler–Maruyama scheme with 5 to 160 steps, whereas (10) is integrated by the stochastic Heun scheme with Nτ=100N_\tau= 100 steps, a choice that § 4.3 revisits with NτN_\tau between 3 and 100.

Position of the proposed policies. Since every policy whose reference is drawn independently of the observation, DP included, realizes a triangular map (Appendix § 9.§ 2), the proposed policies are distinguished not by the triangular structure but by the reference PAP_A, which requires no auxiliary model, no domain knowledge, and no pairing of the training data, and whose distribution at deployment coincides with its distribution during training; moreover, OTP combines this reference with a single network evaluation, learned in a single training stage as a conditional OT map, whereas existing one-step policies start from a Gaussian. We note that SIP is formally the instance of BRIDGeR [11] whose source policy is the action marginal, up to the noise schedule, the discretization, and the network, so that its contribution lies in this choice of reference rather than in the interpolant itself.

Proof of Proposition 1

Proof. Step 1: Let (S,Z)∼PS⊗PZ(S,Z) \sim P_S \otimes P_Z, and let φ:Rm→R\varphi: \mathbb{R}^{m} \to\mathbb{R} and ψ:Rn→R\psi: \mathbb{R}^{n} \to\mathbb{R} be bounded and measurable. The hypothesis G#(PS⊗PZ)=PS,AG_{\#}(P_S \otimes P_Z) = P_{S,A} gives E[φ(S)ψ(T(S,Z))]=EPS,A[φ(S)ψ(A)]\mathbb{E}[\varphi(S)\psi(T(S,Z))] = \mathbb{E}_{P_{S,A}}[\varphi(S)\psi(A)].

Step 2: Since SS and ZZ are independent, the left-hand side equals the first of the integrals below, while the disintegration of PS,AP_{S,A} with respect to PSP_S shows that the right-hand side equals the second:

∫φ(s)(∫ψ[T(s,⋅)#PZ] )PS(ds)and∫φ(s)(∫ψ dPA∣S=s)PS(ds).\int\varphi(s)\left(\int\psi\mathopen{}\left[\mathopen{}T(s,\cdot)_{\#}P_Z\right]\mathclose{}\,\right)P_S(\mathrm{d}s) \quad\text{and}\quad \int\varphi(s)\left(\int\psi\,\mathrm{d}P_{A\mid S=s}\right)P_S(\mathrm{d}s).

Step 3: Since φ\varphi is arbitrary, the two inner integrals coincide for PSP_S-almost every ss. Applying this conclusion to each element ψ\psi of a countable convergence-determining class of bounded continuous functions, and using the fact that a countable union of PSP_S-null sets is PSP_S-null, one verifies that T(s,⋅)#PZT(s,\cdot)_{\#}P_Z and PA∣S=sP_{A\mid S=s} coincide for PSP_S-almost every ss, which concludes the proof. ▪

NFE: network evaluations per action chunk, as stated by each work; DA-SIP and BRIDGeR report the number of integration steps instead. †Stochastic inference: repeated evaluation at a fixed reference sample yields different action chunks. ‡In addition to one evaluation of the source policy. CVAE: conditional variational autoencoder.

Derivation of the Proposed Policies

In this appendix, we derive the three proposed policies summarized in § 3.2.

Optimal transport policy (OTP)

OTP, derived from the optimal transport filter of Al-Jarrah et al. [2], selects T(s,⋅)T(s,\cdot) to be the OT map from PAP_A to PA∣S=sP_{A\mid S=s} for every observation history ss. If the OT cost is quadratic, the OT map is obtained as the solution of the max–min problem Al-Jarrah et al. [1, 2]

max⁡fmin⁡TJ(f,T):=E[f(S,A)]+E[12∥T(S,Aˉ)−Aˉ∥2−f(S,T(S,Aˉ))],(4)\max_f \min_T \mathcal{J}(f,T) := \mathbb{E}[f(S,A)] + \mathbb{E}\left[\frac{1}{2}\lVert T(S,\bar{A})-\bar{A}\rVert^2 - f(S,T(S,\bar{A}))\right], \tag*{(4)}

where the first expectation is taken with respect to PS,AP_{S,A}, the second with respect to the independent coupling (S,Aˉ)∼PS⊗PA(S,\bar{A})\sim P_S\otimes P_A, and f:Rm×Rn→Rf:\mathbb{R}^m\times\mathbb{R}^n\to\mathbb{R} is a potential function. In a numerical implementation, ff and TT are parameterized by residual networks and trained adversarially (Appendix E); at deployment, the policy (3) requires a single evaluation of TT per action chunk, and ff is discarded.

Flow matching policy (FMP)

In FMP, derived from the flow matching filter of Al-Jarrah et al. [4], the map TT is realized as the time-one flow of an ODE (Lipman et al., [34]): for each (s,aˉ)∈Rm×Rn(s,\bar{a}) \in\mathbb{R}^{m} \times\mathbb{R}^{n}, we set T(s,aˉ)=ξ1T(s,\bar{a}) = \xi_{1}, where

dξτdτ=bθ(s,ξτ,τ),ξ0=aˉ,(5)\frac{\mathrm{d}\xi_{\tau}}{\mathrm{d}\tau} = b_{\theta}(s,\xi_{\tau},\tau), \qquad\xi_{0} = \bar{a}, \tag*{(5)}

and bθ:Rm×Rn×[0,1]→Rnb_{\theta} : \mathbb{R}^{m} \times\mathbb{R}^{n} \times[0,1] \to\mathbb{R}^{n} is a learned vector field. The observation history ss enters (5) as a parameter and is held fixed along the flow, so that the associated map is triangular in the sense of (2) by construction. The vector field is obtained as the solution to the regression problem

min⁡θE[∥bθ(S,Aτ,τ)−∂τAτ∥2],(6)\min_{\theta} \mathbb{E}\left[\left\|b_{\theta}(S,A_{\tau},\tau)-\partial_{\tau}A_{\tau}\right\|^{2}\right], \tag*{(6)}

where the expectation is taken over τ∼Unif⁡[0,1]\tau\sim\operatorname{Unif}[0,1], (S,A)∼PS,A(S,A) \sim P_{S,A}, and Aˉ∼PA\bar{A} \sim P_{A} drawn independently of (S,A)(S,A), and where

Aτ=(1−τ)Aˉ+τA,τ∈[0,1],(7)A_{\tau} = (1-\tau)\bar{A}+\tau A, \qquad\tau\in[0,1], \tag*{(7)}

is the linear interpolant between the reference and the target, whose derivative ∂τAτ=A−Aˉ\partial_{\tau}A_{\tau}=A-\bar{A} is available in closed form. The policy (3) is evaluated by integrating (5) from ξ0=Aˉk\xi_{0}=\bar{A}_{k} with s=Sks=S_{k} fixed, by the Heun scheme with Nτ=100N_{\tau}=100 uniform steps, that is, with 200 network evaluations.

Stochastic interpolant policy (SIP)

SIP, derived from the stochastic interpolant filter of Al-Jarrah et al. [4], generalizes FMP along two independent axes (Albergo et al., [5]): the interpolant connecting the reference to the target may carry noise, and the generation may be carried out by an SDE rather than an ODE. The linear interpolant (7) is generalized to

Aτ=(1−τ)Aˉ+τA+γ(τ)ζ,γ(τ)=ετ(1−τ),(8)A_{\tau}=(1-\tau)\bar{A}+\tau A+\gamma(\tau)\zeta, \qquad\gamma(\tau)=\varepsilon\sqrt{\tau(1-\tau)}, \tag*{(8)}

where ζ∼N(0,In)\zeta\sim\mathcal{N}(0,I_{n}) is independent of (S,A,Aˉ)(S,A,\bar{A}) and ε≥0\varepsilon\ge0 sets the noise level; the case ε=0\varepsilon=0 recovers (7). The velocity field is learned exactly as in (6), with the regression target now ∂τAτ=A−Aˉ+γ˙(τ)ζ\partial_{\tau}A_{\tau}=A-\bar{A}+\dot{\gamma}(\tau)\zeta. A second function ηθ\eta_{\theta}, realized as a second output head of the same network, is trained to recover the latent noise,

min⁡θE[∥ηθ(S,Aτ,τ)−ζ∥2],(9)\min_{\theta} \mathbb{E}\left[\left\|\eta_{\theta}(S,A_{\tau},\tau)-\zeta\right\|^{2}\right], \tag*{(9)}

whose minimizer η\eta determines the conditional score of the interpolant (Albergo et al., [5]), namely ∇alog⁡ρτ(a∣s)=−η(s,a,τ)/γ(τ)\nabla_{a}\log\rho_{\tau}(a\mid s)=-\eta(s,a,\tau)/\gamma(\tau) whenever γ(τ)>0\gamma(\tau)>0, where ρτ(⋅∣s)\rho_{\tau}(\cdot\mid s) denotes the density of AτA_{\tau} given S=sS=s. Given the score, the deterministic flow (5) may be replaced by the SDE

dξτ=bˉθ(s,ξτ,τ) dτ+λγ(τ) dWτ,ξ0=aˉ,(10)\mathrm{d}\xi_{\tau}=\bar{b}_{\theta}(s,\xi_{\tau},\tau)\,\mathrm{d}\tau+\lambda\gamma(\tau)\,\mathrm{d}W_{\tau}, \qquad\xi_{0}=\bar{a}, \tag*{(10)}

with drift bˉθ(s,a,τ)=bθ(s,a,τ)−12λ2γ(τ)ηθ(s,a,τ)\bar{b}_{\theta}(s,a,\tau)=b_{\theta}(s,a,\tau)-\frac{1}{2}\lambda^{2}\gamma(\tau)\eta_{\theta}(s,a,\tau), where WτW_{\tau} denotes a standard Wiener process on Rn\mathbb{R}^{n} and λ≥0\lambda\ge0 is a parameter that controls the diffusion coefficient. The score correction in (10) compensates exactly for the injected noise, so that, at the minimizers of (6) and (9), (10) and (5) share the time-τ\tau marginals of the interpolant (8) conditioned on S=sS=s for every choice of λ\lambda; in particular, ξ1∼PA∣S=s\xi_1 \sim P_{A\mid S=s}, since the interpolant starts from PAP_A and ends at PA∣S=sP_{A\mid S=s} by the independence of Aˉ\bar{A} and SS.

In the experiments, SIP is realized by the SDE (10) with ϵ>0\epsilon>0 and λ=1\lambda=1, integrated by a stochastic Heun scheme with Nτ=100N_\tau=100 uniform steps, so that SIP, like FMP, requires 200 network evaluations per action chunk (Appendix E). We note that SIP is formally the instance of BRIDGeR (Chen et al. [11]) whose source policy is the action marginal (Appendix § 6).

Gaussian-Reference Policies

In this appendix, we describe the three policies with which the proposed policies are compared in § 4, namely KRP and the two baselines DP and CP, together with DPR, a variant of DP realized with the network and the sampler of FMP and SIP, all of which transport the Gaussian reference PZ=N(0,In)P_Z=\mathcal{N}(0,I_n), and identify them as instances of the triangular formulation of § 3.1. Table 1 summarizes the seven policies evaluated in this paper.

Knothe–Rosenblatt policy (KRP)

As an instance with the Gaussian reference PZ=N(0,In)P_Z=\mathcal{N}(0,I_n), which is also the reference of DP and CP (Appendices § 9.§ 2 and § 9.§ 4), we consider KRP, derived from the stochastic map filter of Spantini et al. [55], which is based on the Knothe–Rosenblatt rearrangement and is considered in Al-Jarrah et al. [4], in which the triangular map is learned in the inverse direction, from the data to the reference, by maximum likelihood. More precisely, KRP learns a map Φθ:Rm×Rn→Rn\Phi_\theta:\mathbb{R}^m \times\mathbb{R}^n \to\mathbb{R}^n that is lower triangular and monotone in the action chunk, with components

Φθ,j(s,a)=cj(s,a<j)+∫0aj(softplus⁡(gj(s,a<j,t))+δ) dt,j=1,…,n,(11)\Phi_{\theta,j}(s,a)=c_j(s,a_{<j})+\int_0^{a_j}\left(\operatorname{softplus}(g_j(s,a_{<j},t))+\delta\right)\,\mathrm{d}t,\qquad j=1,\ldots,n, \tag*{(11)}

where a<j=(a1,…,aj−1)a_{<j}=(a_1,\ldots,a_{j-1}), cjc_j and gjg_j are neural networks, and δ>0\delta>0, so that Φθ,j\Phi_{\theta,j} is strictly increasing in aja_j by construction (Spantini et al. [55]; ); Baptista et al. [7]). The parameters are obtained by maximizing the likelihood of the training data under the push-forward of the reference, that is, by solving

min⁡θE[12∥Φθ(S,A)∥2−∑j=1nlog⁡∂ajΦθ,j(S,A)],(12)\min_\theta\mathbb{E}\left[\frac{1}{2}\|\Phi_\theta(S,A)\|^2-\sum_{j=1}^{n}\log\partial_{a_j}\Phi_{\theta,j}(S,A)\right], \tag*{(12)}

where (S,A)∼PS,A(S,A)\sim P_{S,A} and the partial derivatives are the integrands of (11), available in closed form. At the minimizer, the map (s,a)↦(s,Φθ(s,a))(s,a)\mapsto(s,\Phi_\theta(s,a)) transports PS,AP_{S,A} to PS⊗N(0,In)P_S\otimes\mathcal{N}(0,I_n), so that its inverse is triangular in the sense of (2), with T(s,⋅)=Φθ(s,⋅)−1T(s,\cdot)=\Phi_\theta(s,\cdot)^{-1}, and Proposition 1 applies; since Φθ(s,⋅)\Phi_\theta(s,\cdot) is moreover lower triangular and monotone, T(s,⋅)T(s,\cdot) is the Knothe–Rosenblatt rearrangement of N(0,In)\mathcal{N}(0,I_n) to PA∣S=sP_{A\mid S=s} (Marzouk et al. [39]). The policy is therefore A^k=Φθ(Sk,⋅)−1(Zk)\hat{A}_k=\Phi_\theta(S_k,\cdot)^{-1}(Z_k), where, as in the analysis step of Spantini et al. [55], the reference sample is Zk=Φθ(Si,Ai)Z_k=\Phi_\theta(S^i,A^i) for a demonstrated pair (Si,Ai)(S^i,A^i) drawn uniformly from the training data, which is distributed as N(0,In)\mathcal{N}(0,I_n) at the minimizer of (12), and the inverse is computed coordinate by coordinate, each coordinate being the root of a scalar monotone equation, by a safeguarded Newton iteration (Appendix E). KRP differs from the analysis step of Spantini et al. [55] in that the functions cjc_j and gjg_j are neural networks rather than linear combinations of basis functions, so that (12) is not convex. On Push-T, drawing ZkZ_k from N(0,In)\mathcal{N}(0,I_n) instead changes the score of KRP from 0.914 to 0.918, a difference of 0.004 ±\pm 0.005. In contrast with DP, KRP is a normalizing flow [39] rather than a denoising model, which provides a second Gaussian-reference comparison. The Knothe–Rosenblatt rearrangement filter is compared in [4] as an established alternative to the filters from which the proposed policies are derived.

Diffusion Policy (DP) as a Gaussian-reference instance

DP [13] is the instance of the formulation of § 3.1 with PZ=N(0,In)P_Z = \mathcal{N}(0,I_n). Given a noise schedule 0<β1<⋯<βL<10 < \beta_1 < \cdots< \beta_L < 1 and αˉℓ:=∏j≤ℓ(1−βj)\bar{\alpha}_{\ell} := \prod_{j \leq\ell}(1-\beta_j), a network eθe_\theta is trained to recover the noise from the noised action chunk,

min⁡θ E[∥eθ(αˉℓA+1−αˉℓZ,ℓ,S)−Z∥2],(13)\min_{\theta}\ \mathbb{E}\left[\left\|e_\theta\left(\sqrt{\bar{\alpha}_{\ell}}A+\sqrt{1-\bar{\alpha}_{\ell}}Z,\ell,S\right)-Z\right\|^2\right], \tag*{(13)}

where ℓ∼Unif⁡{1,…,L}\ell\sim\operatorname{Unif}\{1,\ldots,L\}, (S,A)∼PS,A(S,A) \sim P_{S,A}, and Z∼N(0,In)Z \sim\mathcal{N}(0,I_n) are mutually independent, and an action chunk is generated from A(L)∼N(0,In)A^{(L)} \sim\mathcal{N}(0,I_n) by L=100L=100 steps of the ancestral sampler (Appendix E), a stochastic map that leaves ss unchanged and is therefore triangular. The difference from the proposed policies lies not in the conditioning structure but in the reference, in a path prescribed by a noising process that is independent of the data, rather than by an OT map or by an interpolant between demonstrated actions, and in the LL evaluations of eθe_\theta, a one-dimensional convolutional U-Net; we remark that (8) with a Gaussian sample in place of Aˉ\bar{A} defines the Gaussian-reference counterpart of SIP [5], which isolates the effect of the reference (§ 4.3).

Diffusion Policy with a Residual Network (DPR)

Since DP differs from the proposed policies not only in its reference but also in its network, its sampler, and its settings, we consider, in addition, a variant of DP, which we refer to as DPR, that retains the defining elements of DP, namely the Gaussian reference N(0,In)\mathcal{N}(0,I_n) drawn independently of the observation history, the cosine noise schedule [43], and the noise-prediction objective (13), and takes every other element from FMP and SIP, namely the residual network, the training procedure, and the integration in a continuous artificial time. More precisely, with τ∈[0,1]\tau\in[0,1] running from the reference (τ=0\tau=0) to the data (τ=1\tau=1), the noised action chunk is Aτ=α(τ)A+σ(τ)ZA_\tau=\alpha(\tau)A+\sigma(\tau)Z, where Z∼N(0,In)Z\sim\mathcal{N}(0,I_n) and α(τ)2+σ(τ)2=1\alpha(\tau)^2+\sigma(\tau)^2=1 are given by the cosine schedule, and a network eθ(S,Aτ,τ)e_\theta(S,A_\tau,\tau) is trained to recover ZZ, as in (13) with the discrete index ℓ\ell replaced by the continuous time τ∼Unif⁡[0,1]\tau\sim\operatorname{Unif}[0,1]. An action chunk is generated from Zk∼N(0,In)Z_k\sim\mathcal{N}(0,I_n) by integrating the corresponding probability-flow ODE, in the coordinates of the deterministic sampler of Song et al. [54], by the Heun scheme of Karras et al. [29] with NT=100N_T=100 uniform steps, that is, with 200 network evaluations per action chunk; as in DP, the predicted chunk is clipped to the normalized range of the demonstrated actions at every evaluation. The resulting map leaves ss unchanged and is therefore triangular in the sense of (2). On Push-T, DPR evaluates a network of 8.5 million parameters (width 1024, four residual blocks) with the horizons (Ho,Hp,Ha)=(1,8,8)(H_o,H_p,H_a)=(1,8,8), and, on Square, a network of 18.2 million parameters (width 1728, three residual blocks) with the horizons (2,4,4)(2,4,4). The network uses layer normalization, the mish activation, and a time embedding of dimension 10, and it is trained with AdamW and a cosine schedule, without an exponential moving average of the weights; on Push-T, at the learning rate 3.68×10−43.68\times10^{-4} with 2% warm-up, a weight decay of 0.1, gradient clipping at 0.1, a batch size of 256, and 65,536 iterations, and, on Square, at the learning rate 4.24×10−44.24 \times10^{-4} with 5% warm-up, a weight decay of 0.01, gradient clipping at 0.5, a batch size of 1088, and 262,144 iterations.

On Push-T, under the protocol of § 4.2, DPR attains a score of 0.963±0.0040.963 \pm0.004, with a seed spread of 0.009, 56.6% successes, and 3.1% failures; its paired difference from SIP is −0.002±0.005-0.002 \pm0.005. It is trained in 2.4 minutes per seed and requires 2.84 seconds per episode. On Square, without modification, it succeeds in 89.6%±0.8%89.6\% \pm0.8\% of the episodes, and in 89.1%±0.8%89.1\% \pm0.8\% with the mean of 8 chunks. With state noise, it succeeds in 94.2%±0.6%94.2\% \pm0.6\% of the episodes, and in 93.5%±0.6%93.5\% \pm0.6\% with the mean of 8 chunks, which is comparable with SIP and DP under the same training; it is trained in 44 minutes per seed without state noise and in 46 minutes with it, and requires 1.05 seconds per episode with state noise, against 1.24 for SIP and DP (Table 3). On Push-T, its score is retained down to 10 integration steps (0.932) and drops to 0.226 with 5 steps (Figure 3).

Consistency Policy (CP)

CP ([46]) is likewise an instance of the formulation of § 3.1 with PZ=N(0,In)P_Z = \mathcal{N}(0, I_n), obtained in two training stages: a diffusion model in the parameterization of Karras et al. [29] is first trained as a teacher, and a student network, initialized from the teacher, is then distilled by the consistency trajectory objective of [46], so that it maps a noised action chunk at a given noise level directly to a chunk at a lower noise level along the trajectories of the probability-flow ODE of the teacher. At deployment, CP 1-step maps a Gaussian sample at the largest noise level to an action chunk in a single evaluation of the student, and CP 3-step re-noises this output twice, to intermediate noise levels, and maps it back each time, which yields a stochastic sampler with three evaluations. Both samplers leave ss unchanged and are therefore triangular in the sense of (2), the reference of CP 3-step being extended by the re-noising variables. In contrast with OTP, the other one-step policy of this paper, CP starts from a Gaussian, requires a teacher and a second training stage, and evaluates a U-Net of 69.5 million parameters (Table 2); the details of the implementation are given in Appendix E.

Implementation Details

In this appendix, we describe the implementation of the six policies and of the deployment protocol; the hyperparameters of the proposed policies are listed in Table 5 of Appendix § 11.

Training data. The training data of § 2 are obtained by collecting the pairs (1) over every time instant of every episode, with the first observation and the last action of an episode repeated where the windows extend beyond its boundaries; at deployment, the observation history is likewise completed by repeating the first observation of the episode. Since the observation histories of the data set are, in general, distinct, the empirical conditional distribution at each of them is a Dirac measure, so that the stochasticity of a learned policy relies on the generalization of the map across neighboring observation histories.

Proposed policies. In OTP, the potential ff and the map TT are residual networks whose input is the concatenation of their two arguments, and the map is parameterized as the displacement T(s,aˉ)=aˉ+Δθ(s,aˉ)T(s,\bar{a}) = \bar{a}+\Delta_{\theta}(s,\bar{a}), with the output layer of Δθ\Delta_{\theta} initialized at zero; at every iteration, a minibatch of training pairs is drawn with replacement, the observation histories are permuted within the minibatch to realize the independent coupling, and Kin=25K_{\mathrm{in}} = 25 gradient steps on TT precede one gradient step on ff. In FMP and SIP, the networks are residual multilayer perceptrons with layer normalization, whose input is the concatenation of aa, a sinusoidal embedding of τ\tau, and ss; the network of SIP has the two output heads bθb_\theta and ηθ\eta_\theta, and that of FMP the first only. Every step of the Heun scheme of FMP and of the stochastic Heun scheme of SIP evaluates the network twice, and, at every training iteration, the index of the target pair and the index of the reference chunk are drawn independently, with replacement, from the training data; the two terms (6) and (9) of SIP are weighted equally. In (5), the artificial time τ∈[0,1]\tau\in[0,1] is distinct from the time kk of the system. In (10), setting λ=0\lambda= 0 recovers the ODE. The equivalence of (10) and (5) holds for the exact fields; with the learned fields bθb_\theta and ηθ\eta_\theta and a numerical integrator, the two samplers agree only up to the approximation and discretization errors.

Training. The hyperparameters of the proposed policies are listed in Table 5 (Appendix § 11). The baselines are used with their published settings: DP with the configuration published in Chi et al. [13], trained for 505,000 gradient steps (5,000 epochs at a batch size of 256) and deployed with the exponential moving average of its weights, and CP (Prasad et al., [46]) with the settings shared by its released robomimic configurations and the U-Net, the horizons, and the batch size of DP, its teacher (Karras et al., [29]) and its student each being trained for 150,000 gradient steps (see the paragraph on CP below); the student is deployed with one step (CP 1-step) or, re-noising its output twice, with three steps (CP 3-step). All six policies are trained on all 206 demonstrations (Appendix § 12), and the horizons (Ho,Hp,Ha)(H_o, H_p, H_a) are (1,4,4)(1, 4, 4) for OTP and FMP, (1,8,8)(1, 8, 8) for SIP and KRP, and (2,16,8)(2, 16, 8) for DP and CP.

DP. The ancestral sampler of Appendix § 9.§ 2 applies, for ℓ=L,…,1\ell= L,\ldots,1, the step

A(ℓ−1)=11−βℓ(A(ℓ)−βℓ1−αˉℓϵθ(A(ℓ),ℓ,s))+σℓζℓ,(14)A^{(\ell-1)} = \frac{1}{\sqrt{1-\beta_\ell}}\left(A^{(\ell)}-\frac{\beta_\ell}{\sqrt{1-\bar{\alpha}_\ell}}\epsilon_\theta(A^{(\ell)},\ell,s)\right)+\sigma_\ell\zeta_\ell, \tag*{(14)}

with ζℓ∼N(0,In)\zeta_\ell\sim\mathcal{N}(0,I_n) and σℓ2=βℓ(1−αˉℓ−1)/(1−αˉℓ)\sigma_\ell^2 = \beta_\ell(1-\bar{\alpha}_{\ell-1})/(1-\bar{\alpha}_\ell), the output being A(0)A^{(0)}. In the implementation of Chi et al. [13], used here, every step first forms the estimate of the clean chunk implied by ϵθ\epsilon_\theta and clips it to the normalized range [−1,1]n[-1,1]^n, so that the step coincides with (14) whenever the clipping is inactive. The noise schedule is the squared-cosine schedule of Nichol and Dhariwal [43], with βℓ\beta_\ell capped at 0.999; the U-Net, conditioned on ss and ℓ\ell by feature-wise linear modulation (Perez et al., [45]), has the channel widths 256, 512, and 1024, a kernel size of 5, and diffusion-step embeddings of dimension 256, and it is trained with AdamW at the learning rate 10−410^{-4}, with 500 warm-up steps and a weight decay of 10−610^{-6}, and an exponential moving average of the weights is maintained for deployment, all as in the published configuration.

CP. The implementation is a self-contained module extracted from the code of Prasad et al. [46], and it was verified against that code, to numerical precision, for the network, the noise scalings, the losses, the initialization of the student from the teacher, and the sampler. The teacher is a diffusion model in the parameterization of Karras et al. [29], with noise levels in [0.02,80][0.02,80] and a pseudo-Huber loss; the student, initialized from the teacher, is distilled by the consistency trajectory objective of Prasad et al. [46], combined in equal proportions with a denoising score-matching term. CP 1-step maps the noise level 80 to the level 0.02 in one evaluation, and CP 3-step re-noises the output to the levels 27 and 54 and maps it back each time. Each stage is trained for 150,000 gradient steps at a batch size of 256, as for DP, so that the two stages together require approximately the training time of DP.

KRP. The nn components of (11) are evaluated in parallel: the networks cjc_j and gjg_j of all components are stacked into batched weight tensors, and the triangular structure is imposed by a fixed binary mask on the input (s,a)(s,a), so that component jj receives ss and a<ja_{<j} only. Every cjc_j and gjg_j is a multilayer perceptron with three hidden layers of width 320 and the tanh activation, δ=10−3\delta= 10^{-3}, and the integral of (11) is evaluated by the Gauss–Legendre rule with 16 nodes. The map is trained on (12) with AdamW at the learning rate 2.88×10−32.88 \times10^{-3} and a weight decay of 10−310^{-3}, a cosine schedule with 5% warm-up, gradient clipping at 5.0, a batch size of 320, and 32,768 iterations, with the horizons (1,8,8)(1, 8, 8), so that n=16n = 16. At deployment, coordinate jj of Φθ(Sk,⋅)−1(Zk)\Phi_\theta(S_k,\cdot)^{-1}(Z_k) is the root of the scalar equation Φθ,j(Sk,a^<j,aj)=Zk,j\Phi_{\theta,j}(S_k,\hat{a}_{<j},a_j) = Z_{k,j}, which is bracketed in [−3,3][-3,3], in normalized coordinates, with the bracket doubled until it contains the root, and solved by at most 10 safeguarded Newton steps, in which a step that leaves the bracket is replaced by a bisection step; the derivative required by Newton’s method is the integrand of (11). The reference sample is Φθ(Si,Ai)\Phi_\theta(S^i,A^i) for a demonstrated pair drawn uniformly. On Square, with n=40n = 40, every cjc_j and gjg_j has three hidden layers of width 320 with residual connections and the mish activation, and the map is trained with AdamW at the learning rate 3×10−33 \times10^{-3} and a weight decay of 10−210^{-2}, a cosine schedule without warm-up, gradient clipping at 1.0, a batch size of 576, and 49,152 iterations, with an exponential moving average of the weights of decay 0.999 and the horizons (2,4,4)(2,4,4).

Algorithm 1: Deployment of a generative transport policy

Input: map TT, learned on normalized coordinates; demonstrated action chunks {Ai}i=1M\{A^i\}_{i=1}^{M}; horizons HoH_o, HpH_p, and HaH_a; episode length Kmax⁡K_{\max}

Observe o0o_0, set o−Ho+1=⋯=o−1:=o0o_{-H_o+1} = \cdots= o_{-1} := o_0, and set k←0k \leftarrow0

while k<Kmax⁡k < K_{\max} and the task is not completed do

Sk←(ok−Ho+1,…,ok)S_k \leftarrow(o_{k-H_o+1},\ldots,o_k); draw i∼Unif⁡{1,…,M}i \sim\operatorname{Unif}\{1,\ldots,M\} and set Aˉk←Ai\bar{A}_k \leftarrow A^i

(a^k,…,a^k+Hp−1)←T(Sk,Aˉk)(\hat{a}_k,\ldots,\hat{a}_{k+H_p-1}) \leftarrow T(S_k,\bar{A}_k) \hfill // normalized inputs, denormalized output

for j=k,…,k+Ha−1j = k,\ldots,k+H_a-1 do

Apply clip⁡(a^j)\operatorname{clip}(\hat{a}_j), clipped per coordinate to the demonstrated range, and observe oj+1o_{j+1}

end

k←k+Hak \leftarrow k + H_a

end

Deployment. Algorithm 1 summarizes the deployment protocol of § 3.3. The observation histories and the action chunks are normalized affinely to [−1,1][-1,1] per coordinate, using the ranges of the demonstrations, so that the quadratic cost of (4) and the isotropic noise of (8) are defined in these coordinates, and the executed actions are clipped, per coordinate, to the range of the demonstrated actions. Since the observation history is generated in closed loop, the protocol of § 3.3 reproduces the independence of the reference from the observation history and the distribution PAP_A of the reference, but the coupling PS⊗PAP_S \otimes P_A itself only insofar as SkS_k remains distributed as in the demonstrations. The alternative in which the reference is the action chunk produced at the previous re-planning instant violates this independence, since that chunk is correlated with the current observation history, and it was markedly inferior in preliminary experiments. The history-based policies of Kang et al. [28]; Miller and Jagersand [41]; Jiang et al. [27] avoid the mismatch of the previous-chunk reference by training on references paired with their targets through time, although the distribution of the reference at deployment is then determined by the closed-loop behavior of the policy. The clipping box of Algorithm 1, namely the per-coordinate range of the 25,650 demonstrated actions, is [12,511]×[25,511][12, 511] \times[25, 511] in pixels, which covers almost the whole workspace [0,512]2[0, 512]^2; the clipping therefore confines each coordinate of the action to the demonstrated range, rather than the action to a demonstrated region.

Conditioning on the observations only. We remark that the conditioning variable SkS_k may, in principle, also contain the most recently executed actions ak−Ho,…,ak−1a_{k-H_o}, \ldots, a_{k-1}; this is, however, not recommended, since a policy conditioned on its own past actions tends to copy them rather than respond to the observations, a failure mode known as the copycat problem [59] and, more generally, as causal confusion [16]. Accordingly, we take SkS_k to consist of observations only. Finally, the policies are conditioned on the observations only (§ 2), since conditioning on the most recently executed actions degraded the performance in preliminary experiments, in agreement with the causal confusion of de Haan et al. [16] and the copycat problem of Wen et al. [59].

Hyperparameters

Table 5 lists the hyperparameters of the proposed policies on Push-T. These values were selected with the Bayesian optimization package SMAC3 [33], which searches over the network and training hyperparameters for the configuration that maximizes the evaluation score; the same procedure yields the values on Square in Table 7.

OTPFMPSIP
Horizons (Ho,Hp,HaH_o, H_p, H_a)(2, 4, 4)(2, 4, 4)(2, 4, 4)
Networksff: width 896, 3 residual blocks; TT: width 1344, 3 residual blockswidth 1024, 6 residual blocks, layer normalizationwidth 1408, 6 residual blocks, layer normalization
Activationmishmishmish
Time embedding (dimension)–3040
OptimizerAdamW, weight decay 0AdamW, weight decay 0.001AdamW, weight decay 0.01
Learning rate6.81×10−56.81 \times10^{-5} (ff), 2.48×10−42.48 \times10^{-4} (TT)3.62×10−43.62 \times10^{-4}5.44×10−45.44 \times10^{-4}
Schedulecosine, 5% warm-upcosine, 5% warm-upcosine, 5% warm-up
Gradient clipping1.00.51.0
Batch size115212801216
Iterations32,768, with Kin=25K_{\mathrm{in}} = 25131,072131,072
Moving average of the weights (decay)0.9990.99990.9999
Noise level ε\varepsilon–00.087
Samplerone evaluationHeun, Nτ=100N_{\tau} = 100stochastic Heun, Nτ=100N_{\tau} = 100, λ=1\lambda= 1

Table 7. Hyperparameters of the proposed policies on Square.

Additional Evaluation Details

Environment and computation. On Push-T, the action ak∈R2a_k \in\mathbb{R}^2 is the commanded position of the end effector, applied at 10 Hz, and the observation ok∈R20o_k \in\mathbb{R}^{20} comprises the planar positions of nine keypoints of the block and of the end effector; the environment is the reference implementation of Chi et al. [13]. Two modifications of the code of Chi et al. [13] were required for compatibility with current software versions, neither of which affects the simulation: the gym import is replaced by gymnasium, and a deprecated matplotlib colormap accessor is replaced by its current equivalent. The experiments were performed on a workstation with three NVIDIA RTX 5000 Ada Generation GPUs and an AMD Ryzen Threadripper PRO 7965WX CPU with 24 cores, with PyTorch 2.11, Gymnasium 1.3.0, and pymunk 6.11.1. The training times of Table 2 and Table 3 were measured with one training run per GPU and vary by less than 8% across training seeds; the split of the training time of CP between the teacher and the student is estimated from the times at which the checkpoints of the two stages were written. In the evaluation, the 500 episodes of each training seed are distributed over the three GPUs, so that the time per episode of Table 2 and Table 3 is a measure of throughput, which also depends on the number of steps per episode, smaller for a policy that completes the task earlier.

Figure 1. Figure 1 illustrates the policy (3) for OTP on Push-T: the reference chunks, spread over the workspace, are transported to chunks near the end effector, which depend on the observation history. The reference samples of CP are drawn over the first 8 actions of each chunk. In the normalized coordinates of CP, in which every coordinate of the demonstrated actions ranges over [−1,1][-1,1], each coordinate of a sample of N(0,In)\mathcal{N}(0,I_n) has a standard deviation equal to half the range of the demonstrated actions, approximately 250 pixels, and consecutive actions of a sample are independent; consequently, about half of the sampled points lie outside the frame, and the reference of CP carries no information on the location of the demonstrated actions, in contrast with PAP_A. Only the executed actions are shown, namely 4 per chunk for OTP and the first 8 of 16 for CP, colored by the score after their execution. The block at the start (gray) and after step 61 (dashed) and the target pose (green) are shaded; CP completes this episode at step 265.

Figure 3. In Figure 3, OTP requires one network evaluation. Each point is the mean over 1,500 episodes, namely three training seeds and 500 held-out initial conditions, and error bars denote one standard error. The curves show SIP, FMP, and DPR as functions of NτN_{\tau}, and OTP is the single point at Nτ=1N_{\tau}=1; the vertical axis is logarithmic in 1−score1-\mathrm{score}.

Example episodes. Figure 4 shows the final configuration of nine held-out episodes, drawn at random from the initial conditions 10000–10499 and identical for every policy, under the first training seed of each policy.

Statistical analysis. The intervals of Table 6 are percentile intervals of 10,000 bootstrap replicates of the difference of the mean scores, in which the 500 initial conditions are resampled jointly for both policies and the three training seeds are resampled independently for each policy. The seed spread of Table 2 is the population standard deviation of the three per-seed mean scores; the corresponding sample standard deviations are 0.006 for SIP, 0.020 for OTP, 0.009 for FMP, 0.004 for DP, 0.010 for DPR, 0.013 for both variants of CP, and 0.022 for KRP. Since the two variants of CP share the trained students, the comparison CP 3-step – CP 1-step in Table 6 resamples the training seeds jointly for both variants.

ComparisonDifferenceIntervalComparisonDifferenceInterval
SIP−DP\mathrm{SIP} - \mathrm{DP}0.050[0.030, 0.072]FMP−OTP\mathrm{FMP} - \mathrm{OTP}0.004[−0.021, 0.030]
SIP−OTP\mathrm{SIP} - \mathrm{OTP}0.022[−0.001, 0.046]OTP−CP\mathrm{OTP} - \mathrm{CP} 1-step0.005[−0.023, 0.034]
SIP−FMP\mathrm{SIP} - \mathrm{FMP}0.018[0.001, 0.036]OTP−CP\mathrm{OTP} - \mathrm{CP} 3-step0.003[−0.026, 0.031]
SIP−CP\mathrm{SIP} - \mathrm{CP} 1-step0.027[0.007, 0.050]FMP−CP\mathrm{FMP} - \mathrm{CP} 1-step0.009[−0.014, 0.034]
SIP−CP\mathrm{SIP} - \mathrm{CP} 3-step0.025[0.005, 0.047]FMP−CP\mathrm{FMP} - \mathrm{CP} 3-step0.007[−0.016, 0.031]
OTP−DP\mathrm{OTP} - \mathrm{DP}0.028[−0.001, 0.057]CP\mathrm{CP} 1-step −DP- \mathrm{DP}0.023[−0.001, 0.044]
FMP−DP\mathrm{FMP} - \mathrm{DP}0.032[0.009, 0.056]CP\mathrm{CP} 3-step −DP- \mathrm{DP}0.025[0.003, 0.047]
SIP−KRP\mathrm{SIP} - \mathrm{KRP}0.051[0.024, 0.081]KRP−DP\mathrm{KRP} - \mathrm{DP}−0.001[−0.033, 0.029]
OTP−KRP\mathrm{OTP} - \mathrm{KRP}0.029[−0.004, 0.064]CP\mathrm{CP} 1-step −KRP- \mathrm{KRP}0.024[−0.007, 0.057]
FMP−KRP\mathrm{FMP} - \mathrm{KRP}0.033[0.004, 0.065]CP\mathrm{CP} 3-step −KRP- \mathrm{KRP}0.027[−0.004, 0.060]
CP\mathrm{CP} 3-step −CP- \mathrm{CP} 1-step0.003[−0.006, 0.011]

Table 6. Differences of the mean scores of Table 2, with 95% bootstrap intervals.

Task performance on Push-T. Table 2 reports the performance of the seven policies. It is observed that SIP attains the highest score, 0.965, followed by FMP (0.947), OTP (0.943), the two variants of CP (0.938 and 0.940), and DP (0.915). Since episodes that share a training seed or an initial condition are correlated, the differences are assessed by a bootstrap over the initial conditions

and the training seeds (Appendix § 12, Table 6), whose 95% intervals indicate that SIP exceeds DP, FMP, and both variants of CP, and that FMP and CP 3-step exceed DP, whereas the advantages of SIP over OTP and of OTP and CP 1-step over DP are marginal, and OTP, FMP, and CP 1-step are mutually indistinguishable. KRP, the second policy with a Gaussian reference, attains 0.914, which is indistinguishable from the score of DP. DPR (Diffusion Policy with a residual network), which trains the objective of DP on the residual network and the integrator of FMP and SIP (Appendix § 9.§ 3), attains 0.963, and its paired difference from SIP, −0.002±0.005-0.002 \pm0.005, is not significant; this suggests that, with 100 or more network evaluations, the gap between DP and the proposed policies is largely due to the network and its training rather than to the reference, whose effect is examined in § 4.3. The reproduced score of DP agrees with the average over its last ten checkpoints reported in Chi et al. [13], 0.91 (Appendix § 12). The policies differ mainly in their failures and in the completion of the task (Appendix § 12): DP completes the task as often as SIP, in 66.7% of the episodes, but fails in 8.6% of them, against 3.1% for SIP, whereas the deficit of OTP and FMP lies in near-completions.

Cost on Push-T. Table 2 reports the offline and online cost of the policies, and Figure 2 (Appendix § 12) relates the online cost to the score. On Push-T, the proposed policies are trained in minutes, against 4.5 hours for DP and 5.2 hours for CP, which reflects both the number of gradient steps and the size of the networks, namely U-Nets of 65.8 and 69.5 million parameters against at most 8.1 million. DPR, whose network is of the same type as that of FMP and SIP (8.5 million parameters), is likewise trained in 2.4 minutes, which indicates that the training cost of DP is due to its network and its number of gradient steps rather than to its objective. Online, the time per episode follows the number of network evaluations and the size of the network: the one-step policies OTP and CP 1-step require 0.12 and 0.17 seconds per episode, against 2.1 to 5.4 seconds for the policies with 100 to 200 evaluations per action chunk, and they attain comparable scores with networks whose sizes differ by a factor of approximately 300. The two policies differ, however, in the manner in which the single evaluation is obtained: CP learns its one-step map from a Gaussian reference by distillation, which requires a diffusion teacher of 65.8 million parameters and a student of 69.5 million parameters, trained in two stages for 310 minutes, whereas OTP learns its one-step map directly, as the conditional OT map from PAP_A, in a single training stage of 10.6 minutes and with a network of 0.23 million parameters (Appendix § 12).

Effect of the reference. The reference PAP_A is expected to simplify the learning of the map and to reduce the number of integration steps (§ 3.1). To examine this effect, OTP, FMP, and SIP are trained with the reference PAP_A replaced by N(0,In)\mathcal{N}(0,I_n), every other setting being held fixed, and FMP and SIP are deployed with Nτ∈{100,50,20,10,5,3}N_\tau\in\{100,50,20,10,5,3\} integration steps; the settings were not adapted to the Gaussian reference, and the evaluation protocol is that of § 4.2. Figure 3 reports the results, together with those of DPR (Appendix § 9.§ 3), a diffusion policy with the network of FMP and SIP and the reference N(0,In)\mathcal{N}(0,I_n). With 100 steps, the two references attain comparable scores, 0.965 and 0.960 for SIP, a paired difference of 0.005 ±\pm 0.005, and 0.947 and 0.955 for FMP. As the number of steps is reduced, however, the reference PAP_A retains the score more effectively: with 3 steps, SIP attains 0.798 with PAP_A and 0.752 with N(0,In)\mathcal{N}(0,I_n), and FMP attains 0.871 and 0.785, paired differences of 0.047 ±\pm 0.008 and 0.086 ±\pm 0.007, whereas the score of DPR drops to 0.226 with 5 steps and to 0.176 with 3 steps. For OTP, which transports the reference with a single evaluation of the map, the effect is decisive: the score drops from 0.943 with PAP_A to 0.380 with N(0,In)\mathcal{N}(0,I_n), a paired difference of 0.563 ±\pm 0.008, and the fraction of failed episodes increases from 4.7% to 71.3%; with one evaluation, OTP exceeds FMP with 3 steps by 0.072 ±\pm 0.006. These results indicate that the reference PAP_A accounts for a substantial part of the performance of the proposed policies, and that its benefit is largest when the transport is computed with few network evaluations, which is the regime of interest for online deployment.

Comparison of the scores. In addition to the comparisons of § 4.3, the intervals of Table 6 indicate the following. KRP, the second policy with a Gaussian reference, attains 0.914, which is indistinguishable from the score of DP (difference −0.001-0.001, interval [−0.033-0.033, 0.029]); it is exceeded by SIP ([0.0240.024, 0.081]) and by FMP ([0.0040.004, 0.065]), whereas its deficits relative to OTP and to both variants of CP are not significant.

Breakdown of the scores. The breakdown of the scores indicates that the policies differ mainly in their failures and in the completion of the task. DP completes the task as often as SIP, in 66.7% of the episodes, but fails, with a score below 0.5, in 8.6% of them, against 3.1% for SIP. OTP and FMP complete the task in only 51.2% and 52.1% of the episodes but fail in only 4.7% and 4.4% of them; their deficit is concentrated in near-completions, since 35.5% and 34.0% of their episodes attain a score of at least 0.95 but below one, against 26.5% for SIP and 22.9% for DP. KRP completes the task about as often as OTP and FMP, in 55.6% of the episodes, with 31.3% near-completions, but fails in 8.2% of them, a rate close to that of DP, so that its lower score is accounted for by its failures rather than by its completions.

Computational cost. In addition to the comparison of § 4.3, KRP is likewise trained in 7.3 minutes; however, its sampler inverts Φθ(s,⋅)\Phi_\theta(s,\cdot) one coordinate at a time, with up to 192 sequential evaluations of a component network per action chunk, so that its time per episode, 1.25 seconds, lies between those of the one-step policies and those of the policies with 100 to 200 evaluations. We remark that OTP and CP 1-step are the only policies of Table 2 that produce an action chunk with a single network evaluation, and, accordingly, their online times are comparable (0.12 and 0.17 seconds per episode), as are their scores (0.943 and 0.938, a difference of 0.005 with 95% interval [−0.023-0.023, 0.034]). The same comparison holds on Square (§ 4.3), where the two policies attain comparable success rates at the same online time, whereas the training of CP requires 5.3 hours per seed, against 1.9 hours for OTP. Figure 2 relates the online cost to the score on both tasks. The cost entries of Table 2 and Table 3 are obtained as follows. Gradient steps: 16,384 iterations for OTP, each with 25 updates of the map TT and one of the potential ff; 16,384 for FMP; 65,536 for SIP; 32,768 for KRP; 505,000 for DP; 150,000 + 150,000 for CP (teacher, then student). The parameter count of OTP is that of the map TT; the potential ff, used in training only, has 2.01 million parameters. For KRP, Param. counts the weights that the triangular structure leaves active. For CP, training: teacher, then student (80 + 230 minutes on Push-T). With the mean of 8 chunks, the time per episode on Square changes by at most 0.11 seconds.

Comparability of published results. Published results on Push-T differ in the metric, in the observation modality, and in the selection of checkpoints: the metric is the maximum coverage in Chi et al. [13]; Prasad et al. [46], the final intersection over union in Lee et al. [31], and a success rate in Rana et al. [47]; Gao et al. [20]. In particular, Chi et al. [13] report, for state-based Push-T, 0.95 as the maximum over checkpoints evaluated on the test initial conditions themselves and 0.91 as the average over the last ten checkpoints, of which only the latter, which, like the present protocol, involves no selection on the evaluation episodes, is comparable with the scores of Table 2, and the state-based results reported for CP in Prasad et al. [46], 0.82 with one step and 0.84 with three steps, are averages over 200 episodes of the best checkpoint. Every comparison in this paper is therefore made under the protocol of § 4.2, and a published value is quoted only together with its protocol. We note that the released configuration of Chi et al. [13] for state-based Push-T restricts training to a random subset of 90 demonstrations, whereas Table III of Chi et al. [13] lists 200 demonstrations for this task; all policies of Table 2, DP included, are trained on the 206 demonstrations of the data set.

Replayed demonstrations. The replayed demonstrations of Table 2 score 0.947, below SIP; since the replay applies the recorded actions in open loop from the recorded initial states, whereas the policies act in closed loop and are scored by the best coverage attained, this value is a property of the data rather than an upper bound on the score of a policy.

Clipping of the executed actions. Since the executed actions of every policy are clipped (§ 3.3), we verified that the clipping is inactive for DP, whose sampler already confines its output to the demonstrated range: none of the 2,504 actions executed over ten held-out episodes was modified.

Horizons. The horizons of the policies differ both in the length of the observation history and in the length of the chunk (§ 4.2).

The Square Task: Settings, Averaging, and State Noise

In this appendix, we describe the evaluation on Square (§ 4.3), the diagnosis of the single-sample OTP on this task, and the two modifications of Table 3.

Environment and protocol. The data set is the low-dimensional proficient-human data set of Square released by Chi et al. [13], with absolute actions, and the environment is robosuite 1.2.0 (Zhu et al., [62]) through robomimic 0.2.0 (Mandlekar et al., [38]), the versions with which the data were collected, with the controller set to absolute poses as in the evaluation code of Chi et al. [13]. The policies receive the 23-dimensional observation, namely the 14-dimensional state of the nut and the position (3), orientation quaternion (4), and gripper joint positions (2) of the end effector, and output 10-dimensional actions, namely the position (3), the rotation in the six-dimensional representation (6), and the gripper command (1); the rotation is converted back to an axis-angle command before it is applied. An episode succeeds if the nut is placed on the peg within 400 steps, and it stops at the first success. Every entry of Table 3 uses the final weights of three training runs (seeds 0, 1, and 2), each evaluated on the episode seeds 10000–10499. The two modifications below were selected on the disjoint episode seeds 20000–20499; DP uses its published configuration for Square, trained for 590,000 gradient steps, with its own normalization. CP uses the settings of Appendix § 9, with the horizons (2, 16, 8), 150,000 gradient steps for each of its teacher and student, and the normalization of DP. The hyperparameters of the proposed policies on Square, with the state noise of § 4.3, are listed in Table 7.

Summary of the results. Table 3 reports the success rates. Without modification, OTP succeeds in 76.8% of the episodes, against 91.3% for SIP and 93.8% for DP. Unlike Push-T, Square requires the nut to be placed with a precision of a few millimeters, and two deficits of the single-sample OTP were identified (Appendix § 13): the error of a sampled chunk, about three times that of SIP, is dominated by its variation with the reference sample, and most failures begin when a small error moves the state away from the demonstrations, where the map extrapolates poorly. Accordingly, we consider two modifications, each directed at one deficit: the coordinate-wise mean of 8 chunks sampled at the same state, which reduces the variation due to the reference sample, and Gaussian noise added to the state of the robot during training, which exposes the map to states near, rather than on, the demonstrations. It is observed that each modification improves OTP, to 0.871 and 0.841, respectively, and that together they raise its success rate to 0.895. Since neither modification is specific to OTP, both are applied to every policy of Table 3: the state noise raises the success rate of every policy, whereas the averaging is beneficial only when the variation of the output with the reference sample dominates its error, as is the case for OTP but not for SIP. DPR follows the same pattern: the state noise raises its success rate from 0.896 to 0.942, which is comparable with SIP (0.936, a paired difference of 0.006±0.0080.006 \pm0.008) and DP (0.953, −0.011±0.008-0.011 \pm0.008), whereas the averaging does not improve it. With both modifications, OTP is comparable to the other one-step policy, CP 1-step, in success rate (a difference of 0.005±0.0130.005 \pm0.013) and in online time, and it trades a deficit of 0.03 to 0.05 in success rate, relative to SIP and DP, for an order of magnitude in online time (Table 3), which the mean of 8 chunks leaves unchanged, since the 8 reference samples are mapped in one batched evaluation.

Results. The observation of § 4.3 comprises the pose of the nut and the position, orientation, and gripper state of the end effector, and the action is the absolute pose command of the end effector, with the rotation in the continuous six-dimensional representation, and the gripper command, as in Chi et al. [13]. The evaluation protocol is that of § 4.2, namely three training seeds and the same 500 held-out initial conditions for every policy. Table 3 reports the success rates. Without modification, OTP succeeds in 76.8% of the episodes, against 91.3% for SIP and 93.8% for DP. Unlike Push-T, Square requires the nut to be placed with a precision of a few millimeters, and two deficits of the single-sample OTP were identified (see the diagnosis below): the error of a sampled chunk, about three times that of SIP, is dominated by its variation with the reference sample, and most failures begin when a small error moves the state away from the demonstrations, where the map extrapolates poorly. Accordingly, we consider two modifications, each directed at one deficit: the coordinate-wise mean of 8 chunks sampled at the same state, which reduces the variation due to the reference sample, and Gaussian noise added to the state of the robot during training, which exposes the map to states near, rather than on, the demonstrations. It is observed that each modification improves OTP, to 0.871 and 0.841, respectively, and that together they raise its success rate to 0.895, which reduces its deficit relative to SIP without modification from 0.145 to 0.019±0.0120.019 \pm0.012. Since neither modification is specific to OTP, both are applied to every policy of Table 3. For SIP, the state noise raises the success rate to 0.936 (+0.023±0.010+0.023 \pm0.010), whereas the mean of 8 chunks lowers it to 0.897 (−0.017±0.010-0.017 \pm0.010), and, for DP, the state noise raises it to 0.953 (+0.015±0.007+0.015 \pm0.007), whereas the mean of 8 leaves it unchanged (0.939); for KRP, the state noise raises it from 0.753 to 0.821 (+0.068±0.014+0.068 \pm0.014), whereas the mean of 8 lowers it, to 0.721 without noise and 0.761 with the state noise. This suggests that the state noise is beneficial beyond OTP, whereas the averaging is beneficial only when the variation of the output with the reference sample dominates its error, as is the case for OTP but not for SIP, whose samples vary about three times less. Table 3 reports the corresponding costs: OTP requires 0.12 seconds per episode, about ten times less than SIP and DP (1.24 seconds), and the mean of 8 chunks leaves this time unchanged, since the 8 reference samples are mapped in a single batched evaluation of the network. For CP, the state noise raises the success rate to 0.889 and 0.897 for one and three steps, respectively, and the mean of 8 chunks is ineffective without it (0.880 and 0.881), whereas, combined with it, it yields 0.900 and 0.901. The other one-step policy, CP 1-step, is therefore comparable to OTP both in time (0.12 seconds) and, with both modifications, in success rate (0.900, against 0.895 for OTP, a difference of 0.005±0.0130.005 \pm0.013); however, its two-stage distillation of a U-Net requires 5.3 hours of training per seed. With both modifications applied to every policy, OTP trades a deficit of 0.03 to 0.05 in success rate, relative to SIP and DP, for an order of magnitude in online time. We remark, moreover, that the mean of sampled chunks is a simple remedy: since the proposed policies represent a distribution of action chunks rather than a single chunk, a more sophisticated sampling rule for the online setting, which exploits this distribution at a cost comparable to that of one network evaluation, is a direction worth investigating, and it is the subject of future work.

Example episodes. Figure 5 shows, from above, the final configuration of six held-out episodes, drawn at random from the initial conditions 10000–10499 and identical for every policy and for both configurations, under the first training seed of each policy, without modification (left) and with both modifications (right).

Diagnosis of the single-sample OTP. On 256 demonstrated states, with 64 reference samples each, the median error of the first commanded position is 15.6 mm for OTP against 4.9 mm for SIP; it consists of the variation of the output across the reference samples at a fixed state, 10.0 mm against 3.3 mm, and of a bias of the mean output, 12.0 mm against 3.9 mm, which is largely shared across the training seeds. Across twelve OTP checkpoints, the single-sample error correlates with the success rate with a coefficient of −0.94-0.94. In the closed loop, 118 of the 121 failed episodes of one training seed contain a jump of the commanded position of more than 10 cm between consecutive chunks, against 12 of its 379 successful episodes; such a jump typically follows a small error, for instance a grasp that misses the nut, after which the state leaves the region covered by the demonstrations and the map, evaluated outside its training data, produces outputs far outside the demonstrated range. These two observations motivate the two modifications below, which are directed at the variation of the output and at its sensitivity to states near the demonstrations, respectively; neither removes the shared bias.

Mean of sampled chunks. At every query, N=8N = 8 reference chunks Aˉk(1),…,Aˉk(N)\bar{A}_k^{(1)}, \ldots, \bar{A}_k^{(N)} are drawn independently, and the executed chunk is

A^k=1N∑i=1Nclip⁡[−1,1](T(Sk,Aˉk(i))),(15)\hat{A}_k = \frac{1}{N}\sum_{i=1}^{N}\operatorname{clip}_{[-1,1]}\left(T(S_k,\bar{A}_k^{(i)})\right), \tag*{(15)}

computed in normalized coordinates, where each sample is first clipped to the normalized range of the demonstrated actions, so that one extrapolated sample cannot dominate the mean; the gripper command is averaged with the other coordinates, so that its sign is decided by a majority of the samples. For the stochastic policies, the NN samples are drawn at the same state with independent noise. The mean (15) is a Monte Carlo estimate of the mean of the conditional distribution represented by the policy, in the same way as the ensemble mean in the ensemble Kalman filter (Evensen, [18]); we are not aware of a policy-learning method that uses it in this form, and it differs from the temporal ensembling of Zhao et al. [61], which averages the chunks predicted at successive time steps for the same future step. Its cost is one batched evaluation of the network with NN inputs, which left the time per episode of OTP unchanged. On the selection block, with three training seeds, the success rate of OTP was 0.763 for N=1N = 1, 0.823 for N=2N = 2, 0.842 for N=4N = 4, 0.875 for N=8N = 8, 0.866 for N=16N = 16, 0.859 for N=32N = 32, and 0.849 for N=64N = 64, and the coordinate-wise median and trimmed means of 16 samples did not exceed the mean of 8; temporal ensembling attained 0.846 alone and did not improve the mean of 8. On the evaluation block, the mean of 8 lowered the success rate of SIP from 0.913 to 0.897 (−0.017±0.010-0.017 \pm0.010) and left that of DP unchanged (0.938 against 0.939); since the samples of these policies vary little at a fixed state (3.3 mm for SIP), the mean removes little error, and it may, moreover, merge distinct chunks at the states at which the demonstrated actions are not unimodal. Since a mean of samples lies between the modes of a multimodal conditional distribution, this modification is suited to the near-unimodal distribution of the demonstrated actions at a given state of Square, and it is not proposed as a general replacement for sampling; a fast sampling rule that exploits the represented distribution more effectively than its

mean, for instance by selecting among the samples or by accounting for the previously executed chunk, is left for future work.

State noise. During training, the state of every minibatch, on which the policy is conditioned, is replaced by S+σDξS + \sigma D \xi, with ξ∼N(0,I)\xi\sim\mathcal{N}(0, I) drawn afresh at every iteration and σ=0.02\sigma= 0.02 in the normalized coordinates of OTP, FMP, and SIP, in which each coordinate of the state spans [−1,1][-1, 1]; the diagonal matrix DD restricts the noise to the state of the robot, namely the position, quaternion, and gripper joints of the end effector, at both steps of the observation history, and the target actions are left unchanged. In OTP, the noisy states are also the ones permuted within the minibatch to realize the independent coupling. DP normalizes the state differently, by a single scale factor, and its standard deviations were therefore set per coordinate so that the noise is the same in physical units. Training with noisy inputs is a known regularizer, equivalent for small noise to a penalty on the derivatives of the model with respect to its input ([8]); here, it is applied to the conditioning variable only, so that the map is fitted on states near, rather than on, the demonstrations, which are the states reached after a small error in the closed loop. It is related to the noise injection of Laskey et al. [30], which perturbs the actions of the demonstrator during data collection so that the demonstrations cover such states, whereas the present noise requires no additional demonstrations and leaves the data set unchanged. On the selection block, the success rate of OTP with the mean of 8 was 0.892 for σ=0.005\sigma= 0.005 (one training seed), 0.891 for σ=0.01\sigma= 0.01, 0.903 for σ=0.02\sigma= 0.02, and 0.914 for σ=0.03\sigma= 0.03 (one training seed), against 0.875 without noise, and, with a single sample, 0.803 for σ=0.01\sigma= 0.01 and 0.833 for σ=0.02\sigma= 0.02, against 0.763; the value σ=0.02\sigma= 0.02 was retained on the basis of three training seeds. Applied to SIP, the same noise raised its success rate on the evaluation block from 0.913 to 0.936 (+0.023 ±\pm 0.010, paired over the 1,500 episodes) with a single sample, and to 0.921 with the mean of 8.

Extended Discussion

In this appendix, we give the complete discussion of the results, of which § 5 retains the conclusions.

Results. The results of § 4 indicate that transporting demonstrated action chunks yields policies that are competitive with DP and CP on both tasks considered, at a fraction of their computational cost. On Push-T, the three proposed policies attain scores between 0.943 and 0.965, against 0.915 for DP and 0.938 to 0.940 for CP; on Square, with the state noise and the mean of 8 sampled chunks applied to every policy, they attain success rates between 0.895 and 0.921, against 0.947 for DP and 0.900 to 0.901 for CP. As for the corresponding filters in Al-Jarrah et al. [4], OTP is the least expensive and SIP the most accurate of the proposed policies. In particular, on Push-T, OTP produces an action chunk with a single evaluation of a network of 0.23 million parameters, approximately 300 times fewer than the one-step CP; it is learned in a single training stage of 11 minutes, without the teacher and the distillation that CP requires; and its time per episode on Square is approximately one tenth of those of SIP and DP. These properties make OTP suited to online settings in which the environment requires an action at every re-planning instant and a delay in its computation cannot be tolerated. We note that these results were obtained with deliberately small networks, of at most 8.1 million parameters on Push-T and 24.1 million on Square, against 65.8 and 65.9 million for DP, respectively, drawn from a common family of fully connected networks without architectures specific to each policy, since the purpose of the present study is a unified application of the transport methods of Al-Jarrah et al. [4] to imitation learning rather than the attainment of the highest score; the reported scores therefore do not represent the best attainable by the proposed policies, and larger networks and architectures adapted to each policy may improve them further, possibly beyond those of current policies. The advantage of SIP over FMP, whose 95% interval [0.001, 0.036] excludes zero only marginally, suggests that the latent noise and the stochastic sampler improve the score, although the two policies use separate settings and differ also in their horizons and networks; similarly, the spread of OTP across training seeds, three times that of SIP, suggests a sensitivity to the adversarial training of (4), which an exponential moving average of the weights or a gradient penalty on ff may reduce, an attribution that remains a hypothesis.

Selection of the executed action. Moreover, the results on Square indicate that the rule by which the executed action is selected deserves attention in its own right. The proposed policies represent a conditional distribution of action chunks rather than a single chunk; if the map is exact, executing one sample reproduces the behavior of the demonstrator, whereas, if the map is approximate, other rules may be preferable, so that the selection of the executed action from the distribution represented by the policy remains, in general, an open question. It is observed that the coordinate-wise mean of 8 sampled chunks raises the success rate of OTP from 0.768 to 0.871, and to 0.895 together with the state noise, at an unchanged time per episode, since the 8 reference samples are mapped in a single batched evaluation of the network (§ 4.3). The same rule, however, lowers the success rate of SIP and leaves that of DP unchanged, and, since the mean of chunks drawn from distinct modes need not be an admissible action, it is not expected to be adequate for multimodal conditional distributions. This suggests that the benefit of a selection rule depends on the policy and, in particular, that, for OTP, whose samples are obtained at the cost of a single batched network evaluation, more sophisticated rules may be applied without increasing the online computational time; the design of such rules is the subject of future work.

Limitations and future work. Several limitations of the present study are noted, each with the corresponding remedy. The comparison is restricted to two state-based tasks, of which Push-T has a conditional action distribution close to unimodal ([40]), a restriction that tasks with verified conditional multimodality would address; we note that the common alternatives, Block Pushing ([19, 51]) and Franka Kitchen ([21]), are also estimated to be close to unimodal by Mazza et al. [40]; the baselines are used with their published settings, whereas the settings of the proposed policies were chosen for the present study within the restricted network sizes noted above, an asymmetry that a search over the hyperparameters of the baselines would remove; and the policies are exposed to covariate shift, against which remedies that act on the data set, such as data aggregation when the demonstrator can be queried ([49]) or noise injection during the demonstrations ([30]), leave the formulation of § 3.1 unchanged. Moreover, Proposition 1 provides no guarantee for approximately trained maps or for the closed-loop system, for which the error analysis of triangular transport maps for filtering ([3]) indicates a direction, and OTP produces, at a fixed observation history, a distribution supported on at most MM demonstrated chunks, so that the dependence of the performance on the size of the data set should be quantified. Future work will focus on these extensions, on fast and more sophisticated sampling rules that exploit, in the closed loop, the distribution of action chunks represented by the policy, in place of the mean of sampled chunks used on Square (§ 4.3), on observations from images, and on the evaluation of the policies on physical systems.

References

  1. [1]Mohammad Al-Jarrah, Bamdad Hosseini, and Amirhossein Taghvaei. Optimal transport particle filters. In 2023 62nd IEEE Conference on Decision and Control (CDC), pages 6798–6805. IEEE, 2023.arxiv.org/abs/2304.00392
  2. [2]Mohammad Al-Jarrah, Niyizhen Jin, Bamdad Hosseini, and Amirhossein Taghvaei. Nonlinear filtering with Brenier optimal transport maps. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2310.13886
  3. [3]Mohammad Al-Jarrah, Bamdad Hosseini, Niyizhen Jin, Michele Martino, and Amirhossein Taghvaei. Error analysis of triangular optimal transport maps for filtering. arXiv preprint arXiv:2510.19283, 2025.arxiv.org/abs/2510.19283
  4. [4]Mohammad Al-Jarrah, Wei Deng, Bamdad Hosseini, and Amirhossein Taghvaei. Generative models for simulation based filtering: Formulations and empirical comparisons. arXiv preprint arXiv:2609.16317, 2026.arxiv.org/abs/2609.16317
  5. [5]Michael Albergo, Nicholas M. Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. Journal of Machine Learning Research, 26(209):1–80, 2025.
  6. [6]Ricardo Baptista, Bamdad Hosseini, Nikola Kovachki, and Youssef Marzouk. Conditional sampling with monotone gans: From generative models to likelihood-free inference. SIAM/ASA Journal on Uncertainty Quantification, 12(3), 08 2024a. ISSN ISSN 2166-2525. doi: 10.1137/23m1581546.DOI
  7. [7]Ricardo Baptista, Youssef Marzouk, and Olivier Zahm. On the representation and learning of monotone triangular transport maps. Foundations of Computational Mathematics, 24(6):2063–2108, 2024b.
  8. [8]Christopher M. Bishop. Training with noise is equivalent to Tikhonov regularization. Neural Computation, 7(1):108–116, 1995. doi: 10.1162/neco.1995.7.1.108.DOI
  9. [9]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xieoyang Shi, Laura Smith, James Tanner, Quan Vuong, Anna Walling, Haoquan Wang, and Ury Zhilinsky. π₀: A vision-language-action flow model for general robot control. In Proceedings of Robotics: Science and Systems, 2025a. doi: 10.15607/RSS.2025.XXI.010.DOI
  10. [10]Kevin Black, Manuel Y. Galliker, and Sergey Levine. Real-time execution of action chunking flow policies. In Advances in Neural Information Processing Systems, 2025b.arxiv.org/abs/2506.07339
  11. [11]Kaiqi Chen, Eugene Lim, Kelvin Lin, Yiyang Chen, and Harold Soh. Don’t start from scratch: Behavioral refinement via interpolant-based policy diffusion. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024. doi: 10.15607/RSS.2024.XX.122.DOI
  12. [12]Zhuoqun Chen, Xiu Yuan, Tongzhou Mu, and Hao Su. Responsive noise-relaying diffusion policy: Responsive and efficient visuomotor control. Transactions on Machine Learning Research, 2025.DOI
  13. [13]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, 2023. doi: 10.15607/RSS.2023. XIX.026.
  14. [14]Inkook Chun, Seungjae Lee, Michael S. Albergo, Saining Xie, and Eric Vanden-Eijnden. Dynamic test-time compute scaling in control policy: Difficulty-aware stochastic interpolant policy. In Advances in Neural Information Processing Systems, volume 38, 2025.DOI
  15. [15]Meipo Dai, Qiyuan Zhuang, He-Yang Xu, Ying-Jie Shuai, Yijun Wang, Qi Dou, and Xiu-Shen Wei. Where should action generation begin? A learnable source prior for generative robot policies. arXiv preprint arXiv:2606.17408, 2026.
  16. [16]Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1905.11979
  17. [17]Haoran Ding, Noémie Jaquier, Jan Peters, and Leonel Rozo. Fast and robust visuomotor Riemannian flow matching policy. IEEE Transactions on Robotics, 41:5327–5343, 2025. doi: 10.1109/TRO.2025.3601293.
  18. [18]Geir Evensen. Data Assimilation: The Ensemble Kalman Filter, volume 2. Springer, 2009.
  19. [19]Pete Florence, Corey Lynch, Andy Zeng, Oscar A. Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 158–168, 2022.arxiv.org/abs/2109.00137
  20. [20]Dechen Gao, Boqi Zhao, Andrew Lee, Ian Chuang, Hanchu Zhou, Hang Wang, Zhe Zhao, Jun-shan Zhang, and Iman Soltani. VITA: Vision-to-action flow matching policy. In International Conference on Learning Representations, volume 2026, pages 100903–100933, 2026.arxiv.org/abs/2507.13231
  21. [21]Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 1025–1037, 2020.arxiv.org/abs/1910.11956
  22. [22]Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, volume 29, 2016.arxiv.org/abs/1606.03476
  23. [23]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851, 2020.arxiv.org/abs/2006.11239
  24. [24]Xixi Hu, Bo Liu, Xingchao Liu, and Qiang Liu. AdaFlow: Imitation learning with variance-adaptive flow-based policies. In Advances in Neural Information Processing Systems, volume 37, pages 138836–138858, 2024.arxiv.org/abs/2402.04292
  25. [25]Sigmund H. Høeg, Yilun Du, and Olav Egeland. Fast policy synthesis with variable noise diffusion models. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 4821–4828, 2025. doi: 10.1109/ICRA55743.2025.11127858.DOI
  26. [26]Jindou Jia, Gen Li, Xiangyu Chen, Tuo An, Yuxuan Hu, Jingliang Li, Xinying Guo, and Jianfei Yang. A2A: Action-to-Action Flow Matching Policy. In Proceedings of Robotics: Science and Systems, Sydney, Australia, July 2026.
  27. [27]Sunshine Jiang, Xiaolin Fang, Nicholas Roy, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Siddharth Ancha. Streaming flow policy: Simplifying diffusion/flow-matching policies by treating action trajectories as flow trajectories. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 238–257, 2025.arxiv.org/abs/2505.21851
  28. [28]Sinjae Kang, Chanyoung Kim, Kaixin Wang, Li Zhao, and Kimin Lee. WarmPrior: Straightening flow-matching policies with temporal priors. arXiv preprint arXiv:2605.13959, 2026.arxiv.org/abs/2605.13959
  29. [29]Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, volume 35, pages 26565–26577, 2022.arxiv.org/abs/2206.00364
  30. [30]Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. DART: Noise injection for robust imitation learning. In Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 143–156. PMLR, 2017.arxiv.org/abs/1703.09327
  31. [31]Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 26991–27008, 2024.arxiv.org/abs/2403.03181
  32. [32]Jinhao Li, Yuxuan Cong, Yingqiao Wang, Hao Xia, Shan Huang, Yijia Zhang, Ningyi Xu, and Guohao Dai. STEP: Warm-started visuomotor policies with spatiotemporal consistency prediction. In International Conference on Machine Learning (ICML), 2026.arxiv.org/abs/2602.08245
  33. [33]Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. Smac3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23 (54):1–9, 2022.arxiv.org/abs/2109.09831
  34. [34]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
  35. [35]Zhaoyang Liu, Mokai Pan, Zhongyi Wang, Kaizhen Zhu, Haotao Lu, Haipeng Zhang, Jingya Wang, and Ye Shi. Sample from what you see: Visuomotor policy learning via diffusion bridge with observation-embedded stochastic differential equation. In International Conference on Machine Learning (ICML), 2026.arxiv.org/abs/2512.07212
  36. [36]Bruno Machado, Alexandre Chapin, Emmanuel Dellandrea, and Liming Chen. Flowing with purpose: Latent action guided flow matching policies for robotic manipulation. arXiv preprint arXiv:2606.23420, 2026.arxiv.org/abs/2606.23420
  37. [37]Ashok Makkuva, Amirhossein Taghvaei, Sewoong Oh, and Jason Lee. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning, pages 6672–6681. PMLR, 2020.arxiv.org/abs/1908.10962
  38. [38]Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1678–1690, 2022.arxiv.org/abs/2108.03298
  39. [39]Youssef Marzouk, Tarek Moselhy, Matthew Parno, and Alessio Spantini. Sampling via measure transport: An introduction. Springer Books, pages 785–825, 2017.
  40. [40]Lorenzo Mazza, Massimiliano Datres, Ariel Rodriguez, Sebastian Bodenstedt, Gitta Kutyniok, and Stefanie Speidel. Understanding multimodality in generative behavioral cloning. In Advances in Neural Information Processing Systems, 2026.arxiv.org/abs/2605.22493
  41. [41]Dylan Miller and Martin Jagersand. Temporal policy: History-initialized action generation for robotic learning from demonstration. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026.arxiv.org/abs/2607.29482
  42. [42]Kazuki Mizuta and Karen Leung. Unified Generation-Refinement Planning: Bridging Guided Flow Matching and Sampling-Based MPC for Social Navigation. In Proceedings IEEE Conference on Robotics and Automation, 2026.arxiv.org/abs/2508.01192
  43. [43]Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162–8171. PMLR, 2021.arxiv.org/abs/2102.09672
  44. [44]Sodtavilan Odonchimed, Tatsuya Matsushima, Simon Holk, Yusuke Iwasawa, and Yutaka Matsuo. Retrieve-augmented generation for speeding up diffusion policy without additional training. arXiv preprint arXiv:2507.21452, 2025.arxiv.org/abs/2507.21452
  45. [45]Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. doi: 10.1609/aaai.v32i1.11671.DOI
  46. [46]Aaditya Prasad, Kevin Lin, Jimmy Wu, Linqi Zhou, and Jeannette Bohg. Consistency policy: Accelerated visuomotor policies via consistency distillation. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024. doi: 10.15607/RSS.2024.XX.071.DOI
  47. [47]Krishan Rana, Robert Lee, David Pershouse, and Niko Suenderhauf. IMLE policy: Fast and sample efficient visuomotor policy learning via implicit maximum likelihood estimation. In Proceedings of Robotics: Science and Systems, 2025. doi: 10.15607/RSS.2025.XXI.158.DOI
  48. [48]Hao Ren, Yiming Zeng, Zetong Bi, Zhaoliang Wan, Junlong Huang, and Hui Cheng. Prior does matter: Visual navigation via denoising diffusion bridge models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12100–12110, 2025.arxiv.org/abs/2504.10041
  49. [49]Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 627–635, 2011.arxiv.org/abs/1011.0686
  50. [50]Edward Schmerling, Karen Leung, Wolf Vollprecht, and Marco Pavone. Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction. In Proceedings IEEE Conference on Robotics and Automation, 2018.arxiv.org/abs/1710.09483
  51. [51]Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. In Advances in Neural Information Processing Systems, volume 35, pages 22955–22968, 2022.arxiv.org/abs/2206.11251
  52. [52]Juyi Sheng, Ziyi Wang, Peiming Li, and Mengyuan Liu. MP1: MeanFlow tames policy learning in 1-step for robotic manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 18532–18539, 2026. doi: 10.1609/aaai.v40i22.38919.DOI
  53. [54]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021.arxiv.org/abs/2010.02502
  54. [55]Alessio Spantini, Ricardo Baptista, and Youssef Marzouk. Coupling techniques for nonlinear ensemble filtering. SIAM Review, 64(4):921–953, 2022.arxiv.org/abs/1907.00389
  55. [56]Mingyang Sun, Pengxiang Ding, Weinan Zhang, and Donglin Wang. Score-based diffusion policy compatible with reinforcement learning via optimal transport. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 57374–57390. PMLR, 13–19 Jul 2025.arxiv.org/abs/2502.12631
  56. [57]Amirhossein Taghvaei and Bamdad Hosseini. An optimal transport formulation of Bayes’ law for nonlinear filtering algorithms. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 6608–6613. IEEE, 2022.arxiv.org/abs/2203.11869
  57. [58]Zhendong Wang, Max Li, Ajay Mandlekar, Zhenjia Xu, Jiaojiao Fan, Yashraj Narang, Linxi Fan, Yuke Zhu, Yogesh Balaji, Mingyuan Zhou, Ming-Yu Liu, and Yu Zeng. One-step diffusion policy: Fast visuomotor policies via diffusion distillation. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 63399–63416, 2025.arxiv.org/abs/2410.21257
  58. [59]Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting copycat agents in behavioral cloning from observation histories. In Advances in Neural Information Processing Systems, volume 33, pages 2564–2575, 2020.arxiv.org/abs/2010.14876
  59. [60]Qingsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2204.13902
  60. [61]Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems, 2023. doi: 10.15607/RSS.2023.XIX.016.DOI
  61. [62]Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Kevin Lin, Abhiram Maddukuri, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020.arxiv.org/abs/2009.12293

Paper details

vlnc:W_scw92zed
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents