Introduction
Generating text and molecular graphs requires learning dependencies between discrete coordinates. Continuous (jump) diffusion models approach this task by corrupting token embeddings and learning to predict their endpoints [23 , 10 , 16 , 20 ] . Yet a coordinate’s own noisy embedding can reveal its target without requiring the model to learn the surrounding dependencies. A masked position in masked diffusion or flow matching provides no such local token evidence [34 , 13 ] . For continuous models, increasing noise to remove this shortcut can also destroy useful context [31 ] . We ask whether excluding a coordinate from its own prediction, so called leave-one-out (LOO) prediction, encourages useful contextual learning, and how to do so while preserving the intended probability path.
We study LOO prediction in embedding-space jump diffusions, where continuous motion is combined with jumps to token embeddings. Let Y = ( Y 1 , … , Y L ) Y = (Y_1,\ldots,Y_L) Y = ( Y 1 , … , Y L ) be a token sequence, X t = ( X 1 , t , … , X L , t ) X_t = (X_{1,t},\ldots,X_{L,t}) X t = ( X 1 , t , … , X L , t ) its corrupted embeddings in continuous space at time t t t , and denote by X − i , t X_{-i,t} X − i , t all coordinates except i i i . For a token a a a , the full and LOO posteriors are
p i , t ( a ∣ x ) = P ( Y i = a ∣ X t = x ) , p_{i,t}(a \mid x) = \mathbb{P}(Y_i = a \mid X_t = x), p i , t ( a ∣ x ) = P ( Y i = a ∣ X t = x ) ,
q i , t ( a ∣ x − i ) = P ( Y i = a ∣ X − i , t = x − i ) . q_{i,t}(a \mid x_{-i}) = \mathbb{P}(Y_i = a \mid X_{-i,t} = x_{-i}). q i , t ( a ∣ x − i ) = P ( Y i = a ∣ X − i , t = x − i ) .
Predicting q i , t q_{i,t} q i , t requires estimating Y i Y_i Y i without access to X i , t X_{i,t} X i , t . Our four-token parity experiment illustrates why this can matter: low full-state prediction loss can precede learning the dependence needed for joint generation (Table 1).
Excluding local inputs is established in blind-spot denoising [1 , 22 ] and discrete diffusion [40 , 2 ] (Gouvritch et al. 2026). For a broader overview of related work, see Appendix § 7 . The gap we address is how to use these q i , t ( a ∣ x − i ) q_{i,t}(a \mid x_{-i}) q i , t ( a ∣ x − i ) predictions in processes with both a continuous drift and coordinate resets, i.e. jumps to the token embeddings. In this case the input of the network provides noisy contextual information before discrete decisions. Continuous motion resumes after each reset, and later resets can replace an earlier, possibly wrong, hypothesis. LOO makes the last point particularly meaningful: the learned reset distribution q θ , i , t ( a ∣ x − i ) q_{\theta,i,t}(a \mid x_{-i}) q θ , i , t ( a ∣ x − i ) depends only on the surrounding state x − i x_{-i} x − i , so an incorrect local state x i x_i x i at time t t t cannot directly influence this reset distribution.
The distinction between p i , t p_{i,t} p i , t and q i , t q_{i,t} q i , t is essential. Posterior averaging of conditional generators uses the full posterior p i , t ( a ∣ x ) p_{i,t}(a \mid x) p i , t ( a ∣ x ) [17 ] , whereas LOO training learns q i , t ( a ∣ x − i ) q_{i,t}(a \mid x_{-i}) q i , t ( a ∣ x − i ) without using x i x_i x i . We show that substituting the latter into the drift can change the prescribed marginal path (Example C.2). For jump diffusion, however, exact LOO jump probabilities q i , t ( a ∣ x − i ) q_{i,t}(a \mid x_{-i}) q i , t ( a ∣ x − i ) preserve the correct probability curve when the drift retains the full posterior, under the assumptions of Proposition 3.2. Analytic likelihood completion combines LOO predictions with the local corruption likelihood to recover the full posterior. Thus one network can supply both prediction roles without additional network evaluations.
We compare unrestricted, raw LOO, and completed predictors on Text8, LM1B, and ZINC250k. LOO improves key generation metrics over matched unrestricted controls, while completion’s effect depends on the path and dataset. Local-state and subject–verb interventions test contextual information use, see § 4 .
Our contributions are:
A marginal-preservation result for complete coordinate resets using LOO destinations and full-posterior drift, with a counterexample for substituting LOO probabilities into the drift.
A repeated-reset Brownian bridge whose marginals converge to the data law, using one LOO network and analytic likelihood completion for the two prediction roles.
Matched-budget experiments on Text8, LM1B, and ZINC250k showing when LOO improves over unrestricted prediction and when raw LOO drift outperforms completion despite lacking a marginal-preservation guarantee.
We demonstrate empirically that LOO drift without analytic completion can help recover from wrong predictions at earlier time steps, offering a possible explanation for its better performance in some experiments despite lacking a marginal-preservation guarantee. Controlled LM1B interventions show that a token’s own noisy input can dominate unmasked predictions and final token choices, even when it conflicts with context. On LM1B, LOO improves subject tracking and the usefulness of older word order in generated text.
Conditional paths and leave-one-out prediction
Let V \mathcal{V} V be a finite vocabulary and let p d a t a ∈ P ( V L ) p_{\mathrm{data}} \in\mathcal{P}(\mathcal{V}^{L}) p data ∈ P ( V L ) be a data distribution on sequences of length L L L . Choose a random variable Y = ( Y 1 , … , Y L ) ∼ p d a t a Y = (Y_1,\ldots,Y_L) \sim p_{\mathrm{data}} Y = ( Y 1 , … , Y L ) ∼ p data , and consider an embedding e : V → R d e:\mathcal{V} \to\mathbb{R}^{d} e : V → R d . For recovery of token sequences, we assume that e e e is injective. Write E ( Y ) = ( e ( Y 1 ) , … , e ( Y L ) ) E(Y) = (e(Y_1),\ldots,e(Y_L)) E ( Y ) = ( e ( Y 1 ) , … , e ( Y L )) and for x = ( x 1 , … , x L ) ∈ R d L x = (x_1,\ldots,x_L) \in\mathbb{R}^{dL} x = ( x 1 , … , x L ) ∈ R d L let x i ← z x^{i \gets z} x i ← z denote x x x with position i i i replaced by z z z , i.e. x i ← z = ( x 1 , … , x i − 1 , z , … , x L ) x^{i \gets z} = (x_1,\ldots,x_{i-1},z,\ldots,x_L) x i ← z = ( x 1 , … , x i − 1 , z , … , x L ) . We write x − i = ( x 1 , … , x i − 1 , x i + 1 , … , x L ) x_{-i} = (x_1,\ldots,x_{i-1},x_{i+1},\ldots,x_L) x − i = ( x 1 , … , x i − 1 , x i + 1 , … , x L ) for the sequence with position i i i omitted, and use X − i , t X_{-i,t} X − i , t analogously. Time runs from the source at t = 0 t=0 t = 0 toward data at t = 1 t=1 t = 1 . For each process below, we write p t = L ( X t ) p_t = \mathcal{L}(X_t) p t = L ( X t ) for its state law and p i , t ( a ∣ x ) = P ( Y i = a ∣ X t = x ) p_{i,t}(a \mid x) = \mathbb{P}(Y_i = a \mid X_t = x) p i , t ( a ∣ x ) = P ( Y i = a ∣ X t = x ) for the endpoint posterior at position i i i .
Discrete flow matching
Discrete flow matching (DFM) uses a probability path on token sequences and learns a jump rate kernel that generates its marginals [13 ] . Appendix B.1 summarizes the terminology. For the masked path, let V \mathcal{V} V be a mask and we consider the following discrete process X t y ∈ ( V ∪ { m } ) L X_t^y \in(\mathcal{V} \cup\{m\})^L X t y ∈ ( V ∪ { m } ) L starting from X 0 y = ( m , … , m ) X_0^y = (m,\ldots,m) X 0 y = ( m , … , m ) . Let κ : [ 0 , 1 ] → [ 0 , 1 ] \kappa:[0,1] \to[0,1] κ : [ 0 , 1 ] → [ 0 , 1 ] be continuous and nondecreasing, continuously differentiable on [ 0 , 1 ) [0,1) [ 0 , 1 ) , with κ ( 0 ) = 0 \kappa(0)=0 κ ( 0 ) = 0 , κ ( 1 ) = 1 \kappa(1)=1 κ ( 1 ) = 1 , and κ ( t ) < 1 \kappa(t)<1 κ ( t ) < 1 for t < 1 t<1 t < 1 . For a fixed endpoint y y y , take independent reveal times T 1 , … , T L T_1,\ldots,T_L T 1 , … , T L with P ( T i ≤ t ) = κ ( t ) \mathbb{P}(T_i \le t)=\kappa(t) P ( T i ≤ t ) = κ ( t ) and define the conditional process X t y = ( X 1 , t y , … , X L , t y ) X_t^y=(X_{1,t}^y,\ldots,X_{L,t}^y) X t y = ( X 1 , t y , … , X L , t y ) by setting X i , t y = m X_{i,t}^y=m X i , t y = m for t < T i t<T_i t < T i and X i , t y = y i X_{i,t}^y=y_i X i , t y = y i for t ≥ T i t\ge T_i t ≥ T i . Each position is therefore revealed independently with probability κ ( t ) \kappa(t) κ ( t ) at time t t t , so
P ( X t y = x ) = ∏ i = 1 L [ ( 1 − κ ( t ) ) 1 { x i = m } + κ ( t ) 1 { x i = y i } ] . (1) \mathbb{P}(X_t^y=x)=\prod_{i=1}^{L}\left[(1-\kappa(t))\mathbf{1}\{x_i=m\}+\kappa(t)\mathbf{1}\{x_i=y_i\}\right]. \tag*{(1)} P ( X t y = x ) = i = 1 ∏ L [ ( 1 − κ ( t )) 1 { x i = m } + κ ( t ) 1 { x i = y i } ] . (1)
Since κ ( 1 ) = 1 \kappa(1)=1 κ ( 1 ) = 1 , X 1 y = y X^{y}_{1}=y X 1 y = y almost surely. Hence the mixture p t = L ( X t Y ) p_t=\mathcal{L}(X^{Y}_{t}) p t = L ( X t Y ) satisfies p 1 = p d a t a p_1=p_{\mathrm{data}} p 1 = p data . This conditional process reveals y i y_i y i at rate κ ′ ( t ) / ( 1 − κ ( t ) ) \kappa'(t)/(1-\kappa(t)) κ ′ ( t ) / ( 1 − κ ( t )) while position i i i is masked and then leaves it fixed. Gat et al. [13 ] obtain the jump rate kernel for the marginal process X t X_t X t with law p t p_t p t through the per-position rates
r i , t ( x i → a ∣ x ) = 1 { x i = m } κ ′ ( t ) 1 − κ ( t ) p i , t ( a ∣ x ) , a ∈ V . (2) r_{i,t}(x_i\to a\mid x)=\mathbf{1}\{x_i=\mathrm{m}\}\frac{\kappa'(t)}{1-\kappa(t)}p_{i,t}(a\mid x),\qquad a\in\mathcal{V}. \tag*{(2)} r i , t ( x i → a ∣ x ) = 1 { x i = m } 1 − κ ( t ) κ ′ ( t ) p i , t ( a ∣ x ) , a ∈ V . (2)
Thus generation requires p i , t ( a ∣ x ) p_{i,t}(a\mid x) p i , t ( a ∣ x ) , a categorical endpoint posterior at each position, conditioned on the entire current sequence. These posteriors can be learned by cross-entropy on samples of (1) (Gat et al., [13 ] ; Sahoo et al., [34 ] ). At a masked position, the local mask provides no evidence about its target token.
Continuous embeddings and endpoint posteriors
We briefly review continuous diffusion for categorical data using endpoint-posterior prediction (Dieleman et al., [10 ] ) and posterior averaging of conditional generators (Holderrieth et al., [17 ] ). We illustrate these established ideas with Brownian bridges (Liu et al., [24 ] ), which also provide the continuous component of our later construction. For each endpoint y ∈ V L y\in\mathcal{V}^{L} y ∈ V L , consider Brownian bridges X t y = ( X 1 , t y , … , X L , t y ) X^{y}_{t}=(X^{y}_{1,t},\ldots,X^{y}_{L,t}) X t y = ( X 1 , t y , … , X L , t y ) satisfying, for 0 ≤ t < 1 0\leq t<1 0 ≤ t < 1 ,
d X i , t y = e ( y i ) − X i , t y 1 − t d t + σ d d W i , t , X i , 0 y ∼ N ( 0 , σ s 2 I d ) , σ d > 0 , σ s ≥ 0 dX^{y}_{i,t}=\frac{e(y_i)-X^{y}_{i,t}}{1-t}\,dt+\sigma_d\,dW_{i,t},\qquad X^{y}_{i,0}\sim\mathcal{N}(0,\sigma_s^2I_d),\qquad\sigma_d>0,\ \sigma_s\geq0 d X i , t y = 1 − t e ( y i ) − X i , t y d t + σ d d W i , t , X i , 0 y ∼ N ( 0 , σ s 2 I d ) , σ d > 0 , σ s ≥ 0
These paths extend continuously to t = 1 t=1 t = 1 with X 1 y = E ( y ) X^{y}_{1}=E(y) X 1 y = E ( y ) almost surely. Let p t = L ( X t Y ) p_t=\mathcal{L}(X^{Y}_{t}) p t = L ( X t Y ) for Y ∼ p d a t a Y\sim p_{\mathrm{data}} Y ∼ p data . Posterior averaging gives the marginal SDE d X i , t = b i , t ( X t ) d t + σ d d W i , t dX_{i,t}=b_{i,t}(X_t)\,dt+\sigma_d\,dW_{i,t} d X i , t = b i , t ( X t ) d t + σ d d W i , t , with X 0 ∼ N ( 0 , σ s 2 I d L ) X_0\sim\mathcal{N}(0,\sigma_s^2I_{dL}) X 0 ∼ N ( 0 , σ s 2 I d L ) , X 1 ∼ L ( E ( Y ) ) X_1\sim\mathcal{L}(E(Y)) X 1 ∼ L ( E ( Y )) and
b i , t ( x ) = E [ e ( Y i ) ∣ X t Y = x ] − x i 1 − t = ∑ a ∈ V p i , t ( a ∣ x ) e ( a ) − x i 1 − t . b_{i,t}(x)=\frac{\mathbb{E}[e(Y_i)\mid X^{Y}_{t}=x]-x_i}{1-t}=\frac{\sum_{a\in\mathcal{V}}p_{i,t}(a\mid x)e(a)-x_i}{1-t}. b i , t ( x ) = 1 − t E [ e ( Y i ) ∣ X t Y = x ] − x i = 1 − t ∑ a ∈ V p i , t ( a ∣ x ) e ( a ) − x i .
Under the usual existence and uniqueness assumptions for the forward equation, the resulting process has law p t p_t p t at each time t t t . We can learn p i , t ( a ∣ x ) p_{i,t}(a\mid x) p i , t ( a ∣ x ) with the loss
L C E ( θ ) = E t , Y , X t Y [ − ∑ i log p θ , i , t ( Y i ∣ X t Y ) ] . \mathcal{L}_{\mathrm{CE}}(\theta)=\mathbb{E}_{t,Y,X^{Y}_{t}}\left[-\sum_i\log p_{\theta,i,t}(Y_i\mid X^{Y}_{t})\right]. L CE ( θ ) = E t , Y , X t Y [ − i ∑ log p θ , i , t ( Y i ∣ X t Y ) ] .
An endpoint-directed bridge with repeated resets
For a fixed endpoint y y y , consider the conditional jump diffusion X t y = ( X 1 , t y , … , X L , t y ) X^{y}_{t}=(X^{y}_{1,t},\ldots,X^{y}_{L,t}) X t y = ( X 1 , t y , … , X L , t y ) given by
d X i , t y = e ( y i ) − X i , t − y 1 − t d t + σ d d W i , t + ( e ( y i ) − X i , t − y ) d N i , t , X i , 0 y ∼ N ( 0 , σ s 2 I d ) . (3) dX^{y}_{i,t}=\frac{e(y_i)-X^{y}_{i,t-}}{1-t}\,dt+\sigma_d\,dW_{i,t}+\left(e(y_i)-X^{y}_{i,t-}\right)dN_{i,t},\qquad X^{y}_{i,0}\sim\mathcal{N}(0,\sigma_s^2I_d). \tag*{(3)} d X i , t y = 1 − t e ( y i ) − X i , t − y d t + σ d d W i , t + ( e ( y i ) − X i , t − y ) d N i , t , X i , 0 y ∼ N ( 0 , σ s 2 I d ) . (3)
Each coordinate follows Brownian bridge dynamics between jumps, resets exactly to e ( y i ) e(y_i) e ( y i ) at every jump, and resumes diffusing immediately afterwards. Here N i N_i N i are independent Poisson counting processes with a nonnegative deterministic rate λ ( t ) \lambda(t) λ ( t ) satisfying ∫ 0 t λ ( s ) d s < ∞ \int_0^t\lambda(s)\,ds<\infty ∫ 0 t λ ( s ) d s < ∞ for every t < 1 t<1 t < 1 . The notation X i , t − y X^{y}_{i,t-} X i , t − y denotes the state just before time t t t . For example, λ γ ( t ) = γ t γ − 1 / ( 1 − t γ ) \lambda_\gamma(t)=\gamma t^{\gamma-1}/(1-t^\gamma) λ γ ( t ) = γ t γ − 1 / ( 1 − t γ ) , γ ∈ { 1 , 2 , 4 } \gamma\in\{1,2,4\} γ ∈ { 1 , 2 , 4 } , gives first-reset probability P ( N i , t ≥ 1 ) = t γ \mathbb{P}(N_{i,t}\geq1)=t^\gamma P ( N i , t ≥ 1 ) = t γ . Figure 1 illustrates one choice; Appendix § 8 .§ 2 gives the conditional transition laws.
Figure 1. Four paths, shown against time. States are projected onto e ( b ) e(b) e ( b ) ; colors distinguish the antipodal unit-circle endpoints e ( b ) e(b) e ( b ) and e ( e ) e(e) e ( e ) . Dashed segments mark jumps. Masked diffusion reveals once; the schematic sticky jump diffusion [20 ] stays fixed after commitment. The illustrated repeated-reset trajectories use independent clocks with λ ( t ) = ( 1 − t ) − 1 \lambda(t)=(1-t)^{-1} λ ( t ) = ( 1 − t ) − 1 and continue diffusing after each reset.
Remark 2.1. (Bridge clock). We write (3) and the derivations below for κ ( t ) = t \kappa(t)=t κ ( t ) = t . With a continuous bridge clock κ = κ ( t ) \kappa=\kappa(t) κ = κ ( t ) , the drift factor becomes κ ′ ( t ) / ( 1 − κ ( t ) ) \kappa'(t)/(1-\kappa(t)) κ ′ ( t ) / ( 1 − κ ( t )) and the diffusion coefficient becomes σ d κ ′ ( t ) \sigma_d\sqrt{\kappa'(t)} σ d κ ′ ( t ) . The reset rate λ ( t ) \lambda(t) λ ( t ) is specified independently in physical time. The Text8 and ZINC250k conditional-path comparisons use κ ( t ) = t 2 \kappa(t)=t^2 κ ( t ) = t 2 , giving drift factor 2 t / ( 1 − t 2 ) 2t/(1-t^2) 2 t / ( 1 − t 2 ) and diffusion coefficient σ d 2 t \sigma_d\sqrt{2t} σ d 2 t ; the jump-schedule comparisons vary only λ γ ( t ) \lambda_\gamma(t) λ γ ( t ) . Training states and analytic likelihoods use the conditional laws for the same clock.
Sampling Y ∼ p d a t a Y\sim p_{\mathrm{data}} Y ∼ p data defines the mixture X t Y X^{Y}_{t} X t Y and its marginal curve p t = L ( X t Y ) p_t=\mathcal{L}(X^{Y}_{t}) p t = L ( X t Y ) , with p 0 = N ( 0 , σ s 2 I d L ) p_0=\mathcal{N}(0,\sigma_s^2I_{dL}) p 0 = N ( 0 , σ s 2 I d L ) . Posterior averaging (Holderrieth et al., [17 ] , Proposition 1) gives the following operator, whose weak forward equation is satisfied by p t p_t p t (proved directly in Appendix § 9 .§ 2 ):
A t f ( x ) = ∑ i [ ∑ a ∈ V P ( Y i = a ∣ X t Y = x ) e ( a ) − x i 1 − t ⋅ ∇ i f ( x ) + σ 2 2 Δ i f ( x ) ] + λ ( t ) ∑ i , a P ( Y i = a ∣ X t Y = x ) [ f ( x i ← e ( a ) ) − f ( x ) ] . (4) \mathcal{A}_t f(x)=\sum_i\left[\frac{\sum_{a\in V}\mathbb{P}(Y_i=a\mid X_t^Y=x)e(a)-x_i}{1-t}\cdot\nabla_i f(x)+\frac{\sigma^2}{2}\Delta_i f(x)\right]+\lambda(t)\sum_{i,a}\mathbb{P}(Y_i=a\mid X_t^Y=x)\left[f\left(x^{i\leftarrow e(a)}\right)-f(x)\right]. \tag*{(4)} A t f ( x ) = i ∑ [ 1 − t ∑ a ∈ V P ( Y i = a ∣ X t Y = x ) e ( a ) − x i ⋅ ∇ i f ( x ) + 2 σ 2 Δ i f ( x ) ] + λ ( t ) i , a ∑ P ( Y i = a ∣ X t Y = x ) [ f ( x i ← e ( a ) ) − f ( x ) ] . (4)
Here ∇ i \nabla_i ∇ i and Δ i \Delta_i Δ i are the gradient and Laplacian with respect to x i x_i x i . The conditional paths concentrate on their endpoints as t ↑ 1 t \uparrow1 t ↑ 1 , so the marginal curve extends to p 1 = L ( E ( Y ) ) p_1=\mathcal{L}(E(Y)) p 1 = L ( E ( Y )) , or p 1 = p d a t a p_1=p_{\mathrm{data}} p 1 = p data when identifying tokens with their embeddings.
The only unknown quantities in (4) are p i , t ( a ∣ x ) p_{i,t}(a\mid x) p i , t ( a ∣ x ) , which determine both the drift and the reset probabilities. We therefore learn p θ , i , t p_{\theta,i,t} p θ , i , t from conditional-path samples using
L f u l l ( θ ) = E t , Y , X t Y [ − ∑ i log p θ , i , t ( Y i ∣ X t Y ) ] . (5) \mathcal{L}_{\mathrm{full}}(\theta)=\mathbb{E}_{t,Y,X_t^Y}\left[-\sum_i\log p_{\theta,i,t}(Y_i\mid X_t^Y)\right]. \tag*{(5)} L full ( θ ) = E t , Y , X t Y [ − i ∑ log p θ , i , t ( Y i ∣ X t Y ) ] . (5)
Its unrestricted minimizer is p i , t p_{i,t} p i , t . Appendices C.1 and D give the posterior-learning argument and sampling details, respectively.
Remark 2.2 (Sampling training states). For y i = a y_i=a y i = a , write p i , t ( ⋅ ∣ a ) = L ( X i , t Y ) p_{i,t}(\cdot\mid a)=\mathcal{L}(X_{i,t}^Y) p i , t ( ⋅ ∣ a ) = L ( X i , t Y ) for the conditional state law; the argument order distinguishes it from the endpoint posterior p i , t ( a ∣ x ) p_{i,t}(a\mid x) p i , t ( a ∣ x ) . To estimate (5), draw t t t and Y ∼ p d a t a Y\sim p_{\mathrm{data}} Y ∼ p data , then sample the coordinates independently from p i , t ( ⋅ ∣ Y i ) p_{i,t}(\cdot\mid Y_i) p i , t ( ⋅ ∣ Y i ) . Each coordinate requires only the last reset time (or no reset) and a Gaussian draw. Appendix § 8 .§ 2 gives the mixture law, and Proposition B.2 gives its closed form when λ ( t ) = κ ′ ( t ) / ( 1 − κ ( t ) ) \lambda(t)=\kappa'(t)/(1-\kappa(t)) λ ( t ) = κ ′ ( t ) / ( 1 − κ ( t )) .
Remark 2.3 (Local endpoint information). For uniform four-token even-parity data, Figure 5 in Appendix § 12 shows that the current local state already reveals much of its endpoint: nearest-embedding decoding from X i , t X_{i,t} X i , t alone is 93.9% accurate at t = 0.2 t=0.2 t = 0.2 and 99.9% at t = 0.5 t=0.5 t = 0.5 . Thus, endpoint prediction can be easy without learning the dependence between positions required for joint generation.
Figure 5. Local information in conditional paths. Repeated-reset bridges with e ( a ) = − 1 e(a)=-1 e ( a ) = − 1 , e ( b ) = 1 e(b)=1 e ( b ) = 1 , X 0 = 0 X_0=0 X 0 = 0 , σ d = 0.35 \sigma_d=0.35 σ d = 0.35 , and λ ( t ) = ( 1 − t ) − 1 \lambda(t)=(1-t)^{-1} λ ( t ) = ( 1 − t ) − 1 . Left: 48 coordinate paths, colored by endpoint; dashed segments indicate jumps, after which diffusion continues. Right: endpoint recovery using only the current local state.
Learning contextual resets
Figure 5 in Appendix § 12 shows that the current local state can already reveal most of its endpoint token, making endpoint prediction easy without learning dependencies between positions. To remove this shortcut from the jump predictor, we exclude X i , t X_{i,t} X i , t when predicting the reset destination at position i i i . This restriction changes the posterior learned by cross-entropy, so using it in the dynamics requires care. We show that it can be used while still preserving p t p_t p t , provided the continuous drift retains p i , t ( ⋅ ∣ X t ) p_{i,t}(\cdot\mid X_t) p i , t ( ⋅ ∣ X t ) .
Endpoint prediction without local information
Following contextual token prediction ([38 ] ) and leave-one-out denoising in discrete diffusion ([15 ] ), we consider a leave-one-out (LOO) predictor at position i i i : a categorical kernel q θ , i , t ( a ∣ x − i ) q_{\theta,i,t}(a \mid x_{-i}) q θ , i , t ( a ∣ x − i ) invariant to every change of x i x_i x i . Given samples of the conditional path, we minimize
L L O O ( θ ) = E t , Y , X t [ − ∑ i = 1 L log q θ , i , t ( Y i ∣ X − i , t ) ] (6) \mathcal{L}_{\mathrm{LOO}}(\theta)=\mathbb{E}_{t,Y,X_t}\left[-\sum_{i=1}^{L}\log q_{\theta,i,t}(Y_i\mid X_{-i,t})\right] \tag*{(6)} L LOO ( θ ) = E t , Y , X t [ − i = 1 ∑ L log q θ , i , t ( Y i ∣ X − i , t ) ] (6)
Equation (12) gives their law for the chosen reset schedule.
Over all measurable categorical kernels, the minimizers of (6) are versions of
q i , t ( a ∣ x − i ) = P ( Y i = a ∣ X − i , t = x − i ) (7) q_{i,t}(a \mid x_{-i})=\mathbb{P}(Y_i=a\mid X_{-i,t}=x_{-i}) \tag*{(7)} q i , t ( a ∣ x − i ) = P ( Y i = a ∣ X − i , t = x − i ) (7)
In general, q i , t ( ⋅ ∣ x − i ) ≠ p i , t ( ⋅ ∣ x ) q_{i,t}(\cdot\mid x_{-i})\ne p_{i,t}(\cdot\mid x) q i , t ( ⋅ ∣ x − i ) = p i , t ( ⋅ ∣ x ) .
Preserving the marginal path
Remark 3.1. Replacing p i , t ( ⋅ ∣ x ) p_{i,t}(\cdot\mid x) p i , t ( ⋅ ∣ x ) by q i , t ( ⋅ ∣ x − i ) q_{i,t}(\cdot\mid x_{-i}) q i , t ( ⋅ ∣ x − i ) changes the bridge drift by
b i , t L O O ( x ) − b i , t ( x ) = E [ e ( Y i ) ∣ X − i , t = x − i ] − E [ e ( Y i ) ∣ X t = x ] 1 − t . b^{\mathrm{LOO}}_{i,t}(x)-b_{i,t}(x)=\frac{\mathbb{E}[e(Y_i)\mid X_{-i,t}=x_{-i}]-\mathbb{E}[e(Y_i)\mid X_t=x]}{1-t}. b i , t LOO ( x ) − b i , t ( x ) = 1 − t E [ e ( Y i ) ∣ X − i , t = x − i ] − E [ e ( Y i ) ∣ X t = x ] .
This substitution does not in general preserve the marginal path, even if the jump probabilities are left unchanged. In Example C.2 in Appendix § 9 .§ 3 , the jump rate kernel is unchanged, but the second-moment equation for X 1 , t X_{1,t} X 1 , t fails. Thus the discrepancy is caused by the drift alone.
Keeping the drift b i , t b_{i,t} b i , t while using q i , t q_{i,t} q i , t for the jump part preserves p t p_t p t , although the resulting jump rate kernel differs from that obtained by generator matching. The trajectory laws can differ while the one-time laws remain p t p_t p t .
Proposition 3.2 (Drift b i , t b_{i,t} b i , t and LOO jumps). For the conditional path (3) with any rate λ ( t ) \lambda(t) λ ( t ) satisfying the conditions above, consider the generator
A t c t x f ( x ) = ∑ i [ ∑ a p i , t ( a ∣ x ) e ( a ) − x i 1 − t ⋅ ∇ i f ( x ) + σ d 2 2 Δ i f ( x ) ] + λ ( t ) ∑ i , a q i , t ( a ∣ x − i ) [ f ( x i ← e ( a ) ) − f ( x ) ] (8) \mathcal{A}^{\mathrm{ctx}}_t f(x)=\sum_i\left[\frac{\sum_a p_{i,t}(a\mid x)e(a)-x_i}{1-t}\cdot\nabla_i f(x)+\frac{\sigma_d^2}{2}\Delta_i f(x)\right]+\lambda(t)\sum_{i,a}q_{i,t}(a\mid x_{-i})\left[f(x^{i\leftarrow e(a)})-f(x)\right] \tag*{(8)} A t ctx f ( x ) = i ∑ [ 1 − t ∑ a p i , t ( a ∣ x ) e ( a ) − x i ⋅ ∇ i f ( x ) + 2 σ d 2 Δ i f ( x ) ] + λ ( t ) i , a ∑ q i , t ( a ∣ x − i ) [ f ( x i ← e ( a ) ) − f ( x ) ] (8) Under the assumptions stated in Appendix § 9 .§ 2 , the resulting process started from p 0 p_0 p 0 has law p t p_t p t for every t < 1 t<1 t < 1 . Moreover,
p t ⇒ L ( E ( Y ) ) as t ↑ 1. p_t\Rightarrow\mathcal{L}(\mathbb{E}(Y))\qquad\text{as }t\uparrow1. p t ⇒ L ( E ( Y )) as t ↑ 1. The precise statement and proof are given in Appendix § 9 .§ 2 .
Input-isolated contextual prediction
We adapt the attention construction of T-T A ([38 ] ) to noisy embeddings and path time. Each position has a source feature s j = ϕ ( x j , j , t ) s_j=\phi(x_j,j,t) s j = ϕ ( x j , j , t ) depending only on its own input, position, and time. The sources are fixed across layers. A query h i ( 0 ) = ψ ( i , t ) h_i^{(0)}=\psi(i,t) h i ( 0 ) = ψ ( i , t ) has no state input and is updated by
h i ( ℓ + 1 ) = F ℓ ( h i ( ℓ ) , Attn ℓ ( h i ( ℓ ) , { s j : j ≠ i } , t ) , t ) , q θ , i , t = softmax ( W q h i ( D ) + b q ) , (9) h_i^{(\ell+1)}=F_\ell\left(h_i^{(\ell)},\operatorname{Attn}_\ell(h_i^{(\ell)},\{s_j:j\ne i\},t),t\right),\qquad q_{\theta,i,t}=\operatorname{softmax}(W_qh_i^{(D)}+b_q), \tag*{(9)} h i ( ℓ + 1 ) = F ℓ ( h i ( ℓ ) , Attn ℓ ( h i ( ℓ ) , { s j : j = i } , t ) , t ) , q θ , i , t = softmax ( W q h i ( D ) + b q ) , (9)
Here F ℓ F_\ell F ℓ acts separately at each position; its residual and feed-forward structure is specified in Appendix § 10 .§ 1 . Evolving query features never supply source keys or values.
For fixed parameters, q θ , i , t q_{\theta,i,t} q θ , i , t in (9) is invariant by construction to changes of x i x_i x i , so ∂ q θ , i , t / ∂ x i = 0 \partial q_{\theta,i,t}/\partial x_i=0 ∂ q θ , i , t / ∂ x i = 0 wherever differentiable.
Diagonal masking alone in ordinary self-attention does not ensure this property: another position can receive x i x_i x i and return it in a later layer. Fixed token-local sources prevent that route; SwinIA uses the same principle for blind-spot image denoising [27 ] . All L L L predictions are computed in one forward pass, with O ( L 2 ) O(L^2) O ( L 2 ) attention cost per layer at fixed width. Our unmasked control uses the same fixed-source architecture and allows the additional source s i s_i s i . Another possibility is to target invariance to x i x_i x i through the loss: for uniform discrete diffusion, Gourevitch et al. [15 ] give objectives whose expected loss is minimized over unrestricted predictors by the LOO posterior, even when the predictor receives X t X_t X t . This invariance need not hold after finite training [15 Appendix H] .
Recovering the posterior for the drift
Proposition 3.2 justifies LOO reset probabilities. To obtain the posterior π i , t ( a ∣ x ) \pi_{i,t}(a \mid x) π i , t ( a ∣ x ) required by the continuous drift, we apply analytic likelihood completion to the LOO predictions. For token-factorizing discrete corruption, the analogous cavity-to-denoiser conversion is given by Casado Noguerales et al. [2 ] .
The conditional paths (3) split into products i.e. p t ( d x ∣ y ) = ∏ i p i , t ( d x i ∣ y i ) p_t(dx \mid y) = \prod_i p_{i,t}(dx_i \mid y_i) p t ( d x ∣ y ) = ∏ i p i , t ( d x i ∣ y i ) . Let μ i , t \mu_{i,t} μ i , t be such that p i , t ( ⋅ ∣ a ) ≪ μ i , t p_{i,t}(\cdot\mid a) \ll\mu_{i,t} p i , t ( ⋅ ∣ a ) ≪ μ i , t is absolutely continuous for every a a a and set ℓ i , t ( x i ∣ a ) = d p i , t ( ⋅ ∣ a ) d μ i , t ( x i ) \ell_{i,t}(x_i \mid a) = \frac{dp_{i,t}(\cdot\mid a)}{d\mu_{i,t}(x_i)} ℓ i , t ( x i ∣ a ) = d μ i , t ( x i ) d p i , t ( ⋅ ∣ a ) . For example, μ i , t = ∑ a p i , t ( ⋅ ∣ a ) \mu_{i,t} = \sum_a p_{i,t}(\cdot\mid a) μ i , t = ∑ a p i , t ( ⋅ ∣ a ) is a valid choice.
Proposition 3.3 (Likelihood completion). For conditional paths as in (3) , p i , t p_{i,t} p i , t satisfies
p i , t ( a ∣ x ) = q i , t ( a ∣ x − i ) ℓ i , t ( x i ∣ a ) ∑ b q i , t ( b ∣ x − i ) ℓ i , t ( x i ∣ b ) (10) p_{i,t}(a \mid x) = \frac{q_{i,t}(a \mid x_{-i})\ell_{i,t}(x_i \mid a)}{\sum_b q_{i,t}(b \mid x_{-i})\ell_{i,t}(x_i \mid b)} \tag*{(10)} p i , t ( a ∣ x ) = ∑ b q i , t ( b ∣ x − i ) ℓ i , t ( x i ∣ b ) q i , t ( a ∣ x − i ) ℓ i , t ( x i ∣ a ) (10) for p t p_t p t -almost every x x x where the denominator is positive.
The proof is given in Appendix § 9 .§ 4 . Changing λ \lambda λ changes the training law and its local likelihood; the learned posteriors and likelihood completion must use that same law.
We insert the completed probabilities only into the drift of (8) . The reset term continues to use q θ , i , t q_{\theta,i,t} q θ , i , t , so a single network supplies both prediction roles. Algorithms 1 and 2 in Appendix § 10 summarize simulation-free training and numerical generation. An alternative with a learned drift predictor is described in Appendix § 10 .§ 2 .
Table 1. Cross-entropy loss (CE) and percentage of invalid generated strings at step 250 on the four-token even-parity task. Results average three seeds, with 8,192 generated strings per seed.
Numerical experiments
We compare unrestricted, raw LOO, and Bayesian-completed predictions on Text8 [26 ] , LM1B [4 ] , and ZINC250k [39 , 19 ] . Local-state replacements and subject–verb interventions test reliance on local evidence, contextual prediction, and recovery. Metrics and protocols are in Appendices § 13 .§ 2 , § 13 , § 15 , and § 17 .
Prediction variants. Repeated-reset jump diffusion (RJD) follows (3) ; pure diffusion (PD) uses its continuous bridge without jumps, while absorbing jump diffusion (AJD) stops each coordinate at its first reset (Appendix § 11 ).
Table 36.
BC denotes Bayesian completion via (10) and reintroduces x i x_i x i .
Text8 experiments
Repeated resets. RJD-LOO improves all four metrics over unrestricted prediction (Table 2 ). Compared with completion, raw LOO drift gives higher validity and lower J S D 4 \mathrm{JSD}_4 JSD 4 , while completion gives higher D 5 D_5 D 5 and E 5 E_5 E 5 . Each ordering holds in all three seeds.
Table 2. Text8 at 128 NFE after 60,000 training updates. 4,096 samples per seed, mean ± sample standard deviation over three training seeds. Bold marks the best mean. The protocol is in Appendix § 13 ; metrics are in Appendix § 13 .§ 2 .
Pure diffusion. For pure diffusion, LOO improves all four metrics over both unrestricted prediction and completion. PD-LOO has the highest validity; RJD-LOO has higher D 5 D_5 D 5 and E 5 E_5 E 5 and lower J S D 4 \mathrm{JSD}_4 JSD 4 than PD-LOO.
Comparison methods. Our comparison methods are masked discrete flow matching (DFM) ([13 ] ) and embedded pure jumps (EPJ), with the same backbone after the input encoder (Appendix § 11 ).
DFM and EPJ give similar results; DFM has the lowest mean J S D 4 \mathrm{JSD}_4 JSD 4 despite much lower validity. Thus four-gram agreement alone does not capture lexical quality. Additional AJD results are in Appendix § 13 .§ 5 .
Sampling and budget ablations are in Appendices § 13 .7 and § 13 .8; Appendix § 14 .§ 3 reports a CADD-derived comparison.
LM1B experiments
We use length-128 LM1B windows with the BERT uncased tokenizer. Figure 2 plots GPT-2-large generative perplexity against within-window token entropy ([13 , 32 ] ) across a top-p p p sweep ([18 ] ), which retains the smallest set of highest-probability tokens with cumulative probability at least p p p . Top-p p p applies to jump destinations for RJD/EPJ and drift probabilities for PD. Appendix § 15 .§ 3 describes the sampling protocol and the approximation used for Bayesian completion.
Figure 2. LM1B at 750k updates and 128 NFE on a uniform time grid. Curves connect all p ∈ { 0.7 , 0.8 , 0.9 , 1 } p \in\{0.7, 0.8, 0.9, 1\} p ∈ { 0.7 , 0.8 , 0.9 , 1 } in p p p order; filled markers denote p = 1 p = 1 p = 1 . Points show means over three generation seeds (5,120 windows each); error bars indicate ± 1 \pm1 ± 1 sample standard deviation and are mostly smaller than the markers. All models use one training seed. RJD-SA uses standard self-attention.
Within the entropy range of natural text, curves closer to the lower right indicate a better perplexity–diversity trade-off: lower PPL without sacrificing within-window token diversity. Low PPL alone can reflect limited diversity. For reference, Chemseddine et al. [5 ] report natural LM1B entropy of H ≈ 4.34 H \approx4.34 H ≈ 4.34 nats.
MAUVE ([29 ] ) compares generated text with test LM1B text, with higher scores indicating closer agreement. Table 3 reports the best mean score over the same top-p p p sweep;
Table 3. Best observed MAUVE over p ∈ { 0.7 , 0.8 , 0.9 , 1 } p \in\{0.7, 0.8, 0.9, 1\} p ∈ { 0.7 , 0.8 , 0.9 , 1 } for each method in Figure 2 . Entries are mean ± \pm ± sample SD over three generation seeds.
Appendix § 15 .§ 2 defines the metrics. RJD-LOO and EPJ attain similar best MAUVE (0.9039 and 0.8965). LOO achieves higher best MAUVE than unrestricted prediction, both with and without Bayesian completion. Bayesian completion gives lower best MAUVE than raw LOO in both RJD and PD. Appendix § 15 .§ 3 discusses the full top-p p p sweep and NFE comparisons are in Appendix § 15 .§ 4 .
Self-attention control. RJD-SA replaces fixed-source attention with standard self-attention, computing keys and values from evolving hidden states at each layer. It uses unrestricted predictions with the same RJD path, depth, width, and training budget (Appendix § 15 .§ 1 ). Its best MAUVE is 0.4704, compared with 0.2019 for RJD (Table 3 ). This suggests that fixed-source attention limits performance. RJD-LOO nevertheless reaches 0.9039, motivating more expressive architectures that preserve LOO invariance.
ZINC250k experiments
On ZINC250k (Sterling & Irwin [39 ] ), we compare unrestricted EPJ with unrestricted and LOO predictors for PD and AJD. All five models receive 40,000 updates for each of two training seeds; Table 4 evaluates them at 128 NFE. Appendix § 17 gives the representation, architecture, and evaluation protocol. The two-seed results show that the effect of raw LOO drift depends on the conditional path. For AJD, raw LOO drift increases the fraction of valid and connected outputs over unrestricted prediction and achieves the lowest mean raw and corrected FCD. Completing its drift reduces connected validity but improves NSPDK agreement. For PD, completion improves raw validity, connected validity, raw FCD, and NSPDK over raw LOO drift; corrected FCD improves only slightly on average. PD-BC has the highest mean raw validity, whereas AJD-LOO has the highest mean fraction of valid and connected molecules.
Table 4: ZINC250k test results for 128 NFE: mean ± \pm ± sample standard deviation over training seeds 0 and 1, with 10,000 generated graphs per seed. Validity percentages use all generated graphs before correction. Raw distributional metrics include valid disconnected outputs; corrected FCD uses bond repair and largest-component selection. NSPDK is squared MMD (Appendix § 17 .§ 3 ).
Table 4.
Testing Contextual Learning
Local shortcuts. We first investigate whether the model learns a shortcut: predicting the token at position i i i from its own noisy input X i , t X_{i,t} X i , t without relying on relations between positions. The intervention tests whether conflicting local evidence at X i , t X_{i,t} X i , t overrides context and persists during continued generation. For an LM1B sequence y = ( y 1 , … , y L ) y = (y_1, \ldots, y_L) y = ( y 1 , … , y L ) , draw x x x at time t t t from the conditional process (3). Replace only x i x_i x i by the state x i d x_i^d x i d for a donor d ≠ y i d \ne y_i d = y i , using the same Gaussian residual and reset times, then continue all positions freely. At t = 0.50 t = 0.50 t = 0.50 , unrestricted RJD and RJD-SA almost always retain d d d , whereas LOO more often recovers y i y_i y i but also changes correct inputs (Table 5 ). These interventions show that unrestricted predictions can remain dominated by local evidence despite conflicting context. The start-time sweep is in Appendix § 15 .§ 5 .
Table 5: LM1B local-state intervention at t = 0.50 t = 0.50 t = 0.50 and 750k updates (one training seed/model), with 128 further calls on a uniform time grid. Reset top-p p p is 0.8 for RJD-LOO and 0.7 for RJD and RJD-SA. Percentages use 1,024 text pairs with two draws each. Protocol and start-time sweep: Appendix § 15 .§ 5 .
Table 5.
Bayesian completion and recovery. To test whether completion can hinder recovery from misleading local evidence, we replace a Text8 character state by an independently corrupted state for a donor d ≠ y i d \ne y_i d = y i and compare raw and completed drift using the same LOO checkpoint. Equation (10) explains the immediate effect: replacing only the local state leaves the LOO prediction unchanged, while completion multiplies its odds of donor versus original by the local likelihood ratio. Results are in Table 6 : PD-BC retains the donor almost always, whereas RJD-BC-D can replace it through resets; completion lowers the final word-check pass rate in both families. Earlier-time results and controls are in Appendix § 13 .§ 4 .
Table 6: Text8 donor replacement at t = 0.6973 t = 0.6973 t = 0.6973 . Percentages are mean ± \pm ± sample SD over three training seeds (957 character replacements per seed). The local word check accepts corpus-attested alternatives and retains unresolved cases in the denominator. Immediate argmax refers to the probabilities used in the drift, before continued generation.
Table 6.
LM1B subject–verb agreement. To test whether contextual predictions capture grammatical dependencies, we change the subject’s number while holding the local verb state and all other noisy inputs fixed. For example, changing “the key near the cabinets” to “the keys near the cabinets” should switch the preferred verb from “is” to “are”. We measure whether the preferred form is correct in both versions, using the two number variants rather than the full-vocabulary argmax. Figure 3 shows that LOO tracks the subject more often under conflicting local evidence, but performance deteriorates on longer and harder templates. Distance also varies with wording and intervening nouns; Appendix § 15 .6 separates these configurations and reports all four tested times.
Figure 3. Paired subject tracking with LM1B models at 750k updates. Success requires the correct verb-form preference before and after changing only the subject; the local verb state is initially correct and held fixed. Error bars are 95% lexical-cluster bootstrap intervals. Each distance uses 128 templates with two noise draws, except distance one (32 templates). Distances count WordPieces; wording and intervening-noun configurations also vary.
Context in generated text. We test whether older word order helps predict later tokens. For illustration, GPT-2-large scores “oven” in “The chef mixed dough and put it in the oven” before and after shuffling “chef mixed dough and”, keeping “put it in the” fixed. A drop after shuffling indicates useful older word order. On generated texts from the LM1B experiments, comparisons favor RJD-LOO over RJD, RJD-SA, and EPJ with 4 or 8 recent GPT-2 tokens fixed, see Appendix § 15 .7.
Conclusion
Across Text8, LM1B, and ZINC250k, LOO-based variants improve key generation metrics over matched unrestricted controls, with the choice between raw and Bayesian-completed predictions depending on the task and sampler. In the LM1B interventions, unrestricted RJD relies heavily on its own-position input at the tested intermediate and late times, retaining incorrect local tokens despite conflicting context.
Limitations and future work. Our fixed-source architecture is restrictive: ordinary self-attention improves the paired, single-seed EPJ control (Table 17 ). LOO’s existing gains over matched unrestricted controls make more expressive architectures preserving LOO invariance particularly promising.
Table 17. Pure-jump predictors with matched layers, 60,000-update EMA checkpoints, training and sampling seed 0, and 1,024 length-128 sequences per setting. LOO is the fixed-source EPJ predictor; SA is ordinary self-attention. All six sampling settings are shown. Counts D 5 D_5 D 5 are exact and E 5 E_5 E 5 is rounded to one decimal.
Our repeated-reset sampler uses state-independent rates and full coordinate resets. Rates and jump kernels adapted to disagreement between x i , t x_{i,t} x i , t and contextual predictions could target likely errors more selectively, but would require new marginal-preservation conditions.
Recovery is tested from prescribed corrupted states, and LM1B uses short sequences and one training seed. More seeds, longer contexts, and interventions along generated trajectories would assess robustness and how local errors affect ordinary generation. Comparisons with exact posteriors on tractable joint distributions would help separate changes to the probability path from learning and numerical errors.
Acknowledgments
GENCI provided computing and storage resources on Jean Zay at IDRIS (grant 2026-AD011017753) and on Adastra at CINES (grant 2026-AD011018512).
Extended related work
Discrete and continuous representations. Masked diffusion models such as MDLM and MD4 [34 , 37 ] , and discrete flow matching [13 ] , evolve discrete states. A masked position supplies no target identity, so prediction uses other positions. Continuous models diffuse embeddings, as in Diffusion-LM [23 ] , continuous diffusion for categorical data [10 ] , and Plaid [16 ] . LangFlow and RePlaid report competitive language-modeling performance [6 , 42 ] . Our motivation concerns training information: an identifiable noisy embedding can permit accurate local prediction before the model learns dependencies needed for joint generation.
Hybrid processes and repeated resets. CADD, CANDI, and CCDD combine discrete states with continuous representations [44 , 31 , 45 ] . CANDI highlights a relevant tension: preserving continuous context can leave denoising too easy, while stronger corruption can remove that context [31 ] . Sticky Jump Diffusions combine continuous evolution with absorbing jumps to token anchors and permit cross-position blending in the corruption process [20 ] . Our endpoint-directed Brownian bridge resets coordinates completely to their endpoint embeddings, resumes continuous motion, and permits further resets. Its drift and reset destinations both require predictions, whose information requirements need not coincide. The CADD-derived experiments in Appendix § 14 .§ 3 examine input exclusion within a hybrid training recipe.
Architectures that exclude local information. Blind-spot image denoising excludes a pixel’s noisy observation from its own prediction to enable self-supervised restoration [1 , 22 ] . We adapt T-TA’s input-isolated attention [38 ] to noisy embeddings and path time. Related constructions appear in Hollow Transformers for uniform discrete corruption [40 ] and informed correctors for masked diffusion [43 ] . Input exclusion must hold across the entire network: diagonal attention masking alone permits information to return through another position in a later layer. Our fixed token-local sources prevent this route; SwinIA uses the same principle for blind-spot image denoising [27 ] .
LOO posteriors in discrete diffusion. Input exclusion changes the population prediction target. Gourveitch et al. (2026) identify the LOO posterior as the target of the plug-in ELBO for uniform diffusion and derive conversions between this posterior, the denoiser, and the score. Their objectives can have a LOO population minimizer without architectural input exclusion, although finite training need not yield exact invariance. Casado Noguerales et al. [2 ] give corresponding conversions for general token-factorizing discrete corruption using the cavity posterior, which excludes the local corrupted token. Dmitriev et al. [11 ] analyze LOO samplers for uniform and remasking diffusion, with guarantees depending on the target’s dependence structure. We study LOO prediction when continuous evolution occurs between coordinate resets.
Likelihood completion and the generator. Combining contextual predictions with local likelihoods has precedents in blind-spot denoising (Laine et al., [22 ] ), Plaid’s output parameterization (Gulrajani & Hashimoto, [16 ] , Section 3.3), and the cavity-to-denoiser conversion for discrete corruption (Casado Noguerales et al., [2 ] , Eq. (42)). For our token-factorizing conditional paths, Bayes’ rule likewise recovers p i , t p_{i,t} p i , t from q i , t q_{i,t} q i , t and the local likelihood (Proposition 3.3 ).
Conditional law and analytic likelihood
Jump rate kernels and Markov generators
Jump rate kernels and destination probabilities. Let S S S denote the state space. A jump rate kernel K t ( x , d z ) K_t(x,dz) K t ( x , d z ) is a nonnegative measure in the destination d z dz d z and measurable in x x x . It specifies the instantaneous rates of jump events into subsets of S S S , rather than probabilities over a fixed time interval (Conforti & Léonard, [7 ] ). For the processes considered here, its total mass λ t ( x ) = K t ( x , S ) \lambda_t(x)=K_t(x,S) λ t ( x ) = K t ( x , S ) is finite at each t < 1 t<1 t < 1 and gives the total jump intensity. When λ t ( x ) > 0 \lambda_t(x)>0 λ t ( x ) > 0 , J t ( x , d z ) = K t ( x , d z ) / λ t ( x ) J_t(x,dz)=K_t(x,dz)/\lambda_t(x) J t ( x , d z ) = K t ( x , d z ) / λ t ( x ) is the destination probability kernel conditional on an event. When λ t ( x ) = 0 \lambda_t(x)=0 λ t ( x ) = 0 , the instantaneous event rate is zero and this destination law may be chosen arbitrarily.
For a finite-state pure-jump process with rates continuous in time, let N t N_t N t count all jump events, including self-resets. As h ↓ 0 h\downarrow0 h ↓ 0 ,
P ( N t + h − N t = 0 ∣ X t = x ) = 1 − λ t ( x ) h + o ( h ) , \mathbb{P}(N_{t+h}-N_t=0\mid X_t=x)=1-\lambda_t(x)h+o(h), P ( N t + h − N t = 0 ∣ X t = x ) = 1 − λ t ( x ) h + o ( h ) ,
P ( N t + h − N t ≥ 1 ∣ X t = x ) = λ t ( x ) h + o ( h ) . \mathbb{P}(N_{t+h}-N_t\ge1\mid X_t=x)=\lambda_t(x)h+o(h). P ( N t + h − N t ≥ 1 ∣ X t = x ) = λ t ( x ) h + o ( h ) .
Thus λ t ( x ) \lambda_t(x) λ t ( x ) is a rate, and λ t ( x ) h \lambda_t(x)h λ t ( x ) h approximates the probability of an event over a short interval of length h h h . Self-resets may be included in K t K_t K t ; the rate of actual state changes is then K t ( x , S ∖ { x } ) = λ t ( x ) − K t ( x , { x } ) K_t(x,S\setminus\{x\})=\lambda_t(x)-K_t(x,\{x\}) K t ( x , S ∖ { x }) = λ t ( x ) − K t ( x , { x }) . Accordingly,
P ( X t + h = x ∣ X t = x ) = 1 − h [ λ t ( x ) − K t ( x , { x } ) ] + o ( h ) , \mathbb{P}(X_{t+h}=x\mid X_t=x)=1-h\left[\lambda_t(x)-K_t(x,\{x\})\right]+o(h), P ( X t + h = x ∣ X t = x ) = 1 − h [ λ t ( x ) − K t ( x , { x }) ] + o ( h ) ,
P ( X t + h = z ∣ X t = x ) = h K t ( x , { z } ) + o ( h ) , z ≠ x . \mathbb{P}(X_{t+h}=z\mid X_t=x)=hK_t(x,\{z\})+o(h),\qquad z\ne x. P ( X t + h = z ∣ X t = x ) = h K t ( x , { z }) + o ( h ) , z = x .
Without self-resets, the first probability is 1 − λ t ( x ) h + o ( h ) 1-\lambda_t(x)h+o(h) 1 − λ t ( x ) h + o ( h ) ; the second is h λ t ( x ) J t ( x , { z } ) + o ( h ) h\lambda_t(x)J_t(x,\{z\})+o(h) h λ t ( x ) J t ( x , { z }) + o ( h ) when λ t ( x ) > 0 \lambda_t(x)>0 λ t ( x ) > 0 . For a jump diffusion, no reset does not imply an unchanged state, since the drift and Brownian motion continue between resets. We use jump rate kernel for K t K_t K t and destination probabilities for normalized laws such as J t J_t J t .
Discrete transitions and embedding resets. On a discrete state space, the masses K t ( x , { z } ) K_t(x,\{z\}) K t ( x , { z }) , z ≠ x z\ne x z = x , are also called transition rates or transition intensities. In the masked model, the per-position rates in (2) specify
K t ( x , d z ) = ∑ i = 1 L ∑ a ∈ V r i , t ( x i → a ∣ x ) δ x i ← a ( d z ) . K_t(x,dz)=\sum_{i=1}^{L}\sum_{a\in\mathcal{V}}r_{i,t}(x_i\to a\mid x)\delta_{x_{i\leftarrow a}}(dz). K t ( x , d z ) = i = 1 ∑ L a ∈ V ∑ r i , t ( x i → a ∣ x ) δ x i ← a ( d z ) .
where δ z \delta_z δ z is the point mass at z z z , and coordinate replacement is understood here for token sequences. For the embedding-space resets, the kernels in (4) and (8) are, respectively,
K t ( x , d z ) = λ ( t ) ∑ i , a p i , t ( a ∣ x ) δ x i ← e ( a ) ( d z ) , K_t(x,dz)=\lambda(t)\sum_{i,a}p_{i,t}(a\mid x)\delta_{x_{i\leftarrow e(a)}}(dz), K t ( x , d z ) = λ ( t ) i , a ∑ p i , t ( a ∣ x ) δ x i ← e ( a ) ( d z ) ,
K t c t x ( x , d z ) = λ ( t ) ∑ i , a q i , t ( a ∣ x − i ) δ x i ← e ( a ) ( d z ) . K_t^{\mathrm{ctx}}(x,dz)=\lambda(t)\sum_{i,a}q_{i,t}(a\mid x_{-i})\delta_{x_{i\leftarrow e(a)}}(dz). K t ctx ( x , d z ) = λ ( t ) i , a ∑ q i , t ( a ∣ x − i ) δ x i ← e ( a ) ( d z ) .
Both have total reset intensity λ t ( x ) = L λ ( t ) \lambda_t(x)=L\lambda(t) λ t ( x ) = L λ ( t ) ; λ ( t ) \lambda(t) λ ( t ) is the per-position clock intensity. The categorical laws p i , t p_{i,t} p i , t and q i , t q_{i,t} q i , t specify destination tokens conditional on resetting position i i i ; multiplying by the clock intensity gives the corresponding contribution to the jump rate kernel.
Markov generators. The infinitesimal generator , or Markov generator , describes the instantaneous change in expected test functions [17 Section 4] . For our jump diffusions,
A t f ( x ) = ∑ i [ b i , t ( x ) ⋅ ∇ i f ( x ) + σ d 2 2 Δ i f ( x ) ] + ∫ S [ f ( z ) − f ( x ) ] K t ( x , d z ) . \mathcal{A}_t f(x) = \sum_i \left[b_{i,t}(x) \cdot\nabla_i f(x) + \frac{\sigma_d^2}{2} \Delta_i f(x)\right] + \int_S [f(z) - f(x)]K_t(x,dz). A t f ( x ) = i ∑ [ b i , t ( x ) ⋅ ∇ i f ( x ) + 2 σ d 2 Δ i f ( x ) ] + ∫ S [ f ( z ) − f ( x )] K t ( x , d z ) .
For a pure-jump process, the continuous terms are absent. On a finite state space, the generator matrix has off-diagonal entries K t ( x , { z } ) K_t(x,\{z\}) K t ( x , { z }) and diagonal entry − K t ( x , S ∖ { x } ) -K_t(x,S \setminus\{x\}) − K t ( x , S ∖ { x }) ; its rows sum to zero. Self-resets contribute zero to the generator.
Conditional jump-diffusion SDE
For background on stochastic differential equations with jumps and their generators, see [21 Chapters 3–4] . We derive the conditional state law used in Remark 2.2 by conditioning on the last reset time, as in last-renewal representations of stochastic resetting [12 ] .
As in the main construction, let λ ( t ) ≥ 0 \lambda(t) \ge0 λ ( t ) ≥ 0 satisfy ∫ 0 t λ ( s ) d s < ∞ \int_0^t \lambda(s)\,ds < \infty ∫ 0 t λ ( s ) d s < ∞ for every t < 1 t < 1 t < 1 . Fix y ∈ V L y \in\mathcal{V}^L y ∈ V L . Let W i W_i W i be independent d d d -dimensional Brownian motions and N i N_i N i independent inhomogeneous Poisson counting processes with intensity measure λ ( t ) d t \lambda(t)\,dt λ ( t ) d t . Recall the Jump SDE from (3)
d X i , t y = e ( y i ) − X i , t − y 1 − t d t + σ d d W i , t + ( e ( y i ) − X i , t − y ) d N i , t , X i , 0 y ∼ N ( 0 , σ s 2 I d ) . (11) dX^y_{i,t} = \frac{e(y_i) - X^y_{i,t-}}{1-t}\,dt + \sigma_d\,dW_{i,t} + (e(y_i) - X^y_{i,t-})\,dN_{i,t}, \qquad X^y_{i,0} \sim\mathcal{N}(0,\sigma_s^2 I_d). \tag*{(11)} d X i , t y = 1 − t e ( y i ) − X i , t − y d t + σ d d W i , t + ( e ( y i ) − X i , t − y ) d N i , t , X i , 0 y ∼ N ( 0 , σ s 2 I d ) . (11)
We take σ d > 0 \sigma_d > 0 σ d > 0 and σ s ≥ 0 \sigma_s \ge0 σ s ≥ 0 , with a base law N ( 0 , σ s 2 I d ) \mathcal{N}(0,\sigma_s^2 I_d) N ( 0 , σ s 2 I d ) independent of y y y . Here X i , t − y X^y_{i,t-} X i , t − y is the left limit. At a jump of N i N_i N i , the increment is e ( y i ) − X i , t − y e(y_i)-X^y_{i,t-} e ( y i ) − X i , t − y , so X i , t y = e ( y i ) X^y_{i,t}=e(y_i) X i , t y = e ( y i ) exactly. The Brownian motion and counting process continue after this event. On each [ 0 , T ] [0,T] [ 0 , T ] , T < 1 T < 1 T < 1 , there are almost surely finitely many resets, and the between-reset linear SDE has a unique solution. Write p t ( ⋅ ∣ y ) = L ( X t y ) p_t(\cdot\mid y)=\mathcal{L}(X^y_t) p t ( ⋅ ∣ y ) = L ( X t y ) .
Lemma B.1 (Conditional state law). Consider (11) with deterministic λ \lambda λ and mutually independent initial states, Brownian motions, and Poisson processes. For every 0 < t < 1 0 < t < 1 0 < t < 1 and y i = a y_i=a y i = a , the law p i , t ( ⋅ ∣ a ) = L ( X i , t y ) p_{i,t}(\cdot\mid a)=\mathcal{L}(X^y_{i,t}) p i , t ( ⋅ ∣ a ) = L ( X i , t y ) is the Gaussian mixture
p i , t ( d x ∣ a ) = S ( 0 , t ) N ( d x ; t e ( a ) , v 0 , t I d ) + ∫ 0 t λ ( r ) S ( r , t ) N ( d x ; e ( a ) , v r , t I d ) d r , (12) p_{i,t}(dx \mid a) = S(0,t)\mathcal{N}(dx;te(a),v_{0,t}I_d) + \int_0^t \lambda(r)S(r,t)\mathcal{N}(dx;e(a),v_{r,t}I_d)\,dr, \tag*{(12)} p i , t ( d x ∣ a ) = S ( 0 , t ) N ( d x ; t e ( a ) , v 0 , t I d ) + ∫ 0 t λ ( r ) S ( r , t ) N ( d x ; e ( a ) , v r , t I d ) d r , (12) where S ( r , t ) = exp ( − ∫ r t λ ( u ) d u ) S(r,t)=\exp\left(-\int_r^t\lambda(u)\,du\right) S ( r , t ) = exp ( − ∫ r t λ ( u ) d u ) , v 0 , t = ( 1 − t ) 2 σ s 2 + σ d 2 t ( 1 − t ) v_{0,t}=(1-t)^2\sigma_s^2+\sigma_d^2t(1-t) v 0 , t = ( 1 − t ) 2 σ s 2 + σ d 2 t ( 1 − t ) , and v r , t = σ d 2 ( t − r ) ( 1 − t ) / ( 1 − r ) v_{r,t}=\sigma_d^2(t-r)(1-t)/(1-r) v r , t = σ d 2 ( t − r ) ( 1 − t ) / ( 1 − r ) for 0 < r < t 0<r<t 0 < r < t . The coordinates are conditionally independent, so p t ( d x ∣ y ) = ∏ i p i , t ( d x i ∣ y i ) p_t(dx\mid y)=\prod_i p_{i,t}(dx_i\mid y_i) p t ( d x ∣ y ) = ∏ i p i , t ( d x i ∣ y i ) .
Proof. For 0 ≤ s < t < 1 0 \le s < t < 1 0 ≤ s < t < 1 , if no reset occurs during ( s , t ] (s,t] ( s , t ] , solving the linear SDE gives
X i , t y = e ( y i ) + 1 − t 1 − s ( X i , s y − e ( y i ) ) + σ d ( 1 − t ) ∫ s t d W i , u 1 − u . X^y_{i,t}=e(y_i)+\frac{1-t}{1-s}\left(X^y_{i,s}-e(y_i)\right)+\sigma_d(1-t)\int_s^t\frac{dW_{i,u}}{1-u}. X i , t y = e ( y i ) + 1 − s 1 − t ( X i , s y − e ( y i ) ) + σ d ( 1 − t ) ∫ s t 1 − u d W i , u . The last term is centered Gaussian with covariance σ d 2 ( t − s ) ( 1 − t ) / ( 1 − s ) I d \sigma_d^2(t-s)(1-t)/(1-s)I_d σ d 2 ( t − s ) ( 1 − t ) / ( 1 − s ) I d . If there is no reset before t t t , set s = 0 s=0 s = 0 and use the independent Gaussian initial state. This gives mean t e ( a ) te(a) t e ( a ) and covariance v 0 , t I d v_{0,t}I_d v 0 , t I d , where v 0 , t = ( 1 − t ) 2 σ s 2 + σ d 2 t ( 1 − t ) v_{0,t}=(1-t)^2\sigma_s^2+\sigma_d^2t(1-t) v 0 , t = ( 1 − t ) 2 σ s 2 + σ d 2 t ( 1 − t ) . If the last reset is at r ∈ ( 0 , t ) r\in(0,t) r ∈ ( 0 , t ) , then X i , r y = e ( a ) X^y_{i,r}=e(a) X i , r y = e ( a ) . Setting s = r s=r s = r therefore gives mean e ( a ) e(a) e ( a ) and covariance v r , t I d v_{r,t}I_d v r , t I d , where v r , t = σ d 2 ( t − r ) ( 1 − t ) / ( 1 − r ) v_{r,t}=\sigma_d^2(t-r)(1-t)/(1-r) v r , t = σ d 2 ( t − r ) ( 1 − t ) / ( 1 − r ) .
It remains to weight these Gaussian laws by the reset probabilities. Let τ i , t \tau_{i,t} τ i , t be the last reset time of coordinate i i i up to time t t t , with τ i , t = 0 \tau_{i,t}=0 τ i , t = 0 if no reset has occurred. For 0 ≤ r < t 0\le r<t 0 ≤ r < t ,
P ( τ i , t ≤ r ) = P ( N i , t − N i , r = 0 ) = exp ( − ∫ r t λ ( u ) d u ) = S ( r , t ) . \mathbb{P}(\tau_{i,t}\le r)=\mathbb{P}(N_{i,t}-N_{i,r}=0)=\exp\left(-\int_r^t\lambda(u)\,du\right)=S(r,t). P ( τ i , t ≤ r ) = P ( N i , t − N i , r = 0 ) = exp ( − ∫ r t λ ( u ) d u ) = S ( r , t ) . Thus τ i , t \tau_{i,t} τ i , t has an atom of mass S ( 0 , t ) S(0,t) S ( 0 , t ) at zero and density ∂ r S ( r , t ) = λ ( r ) S ( r , t ) \partial_rS(r,t)=\lambda(r)S(r,t) ∂ r S ( r , t ) = λ ( r ) S ( r , t ) for almost every r ∈ ( 0 , t ) r\in(0,t) r ∈ ( 0 , t ) . Independence of the Poisson process from the Brownian motion and initial state ensures that conditioning on these reset times leaves the Gaussian laws above unchanged. Weighting the no-reset law by S ( 0 , t ) S(0,t) S ( 0 , t ) and integrating the last-reset law against λ ( r ) S ( r , t ) d r \lambda(r)S(r,t)\,dr λ ( r ) S ( r , t ) d r proves (12). The weights sum to one because S ( 0 , t ) + ∫ 0 t λ ( r ) S ( r , t ) d r = 1 S(0,t)+\int_0^t\lambda(r)S(r,t)\,dr=1 S ( 0 , t ) + ∫ 0 t λ ( r ) S ( r , t ) d r = 1 . Finally, each coordinate depends only on its own initial state and drivers, which proves conditional coordinate independence. □ \square □
Since ( t − r ) / ( 1 − r ) ≤ t (t-r)/(1-r) \le t ( t − r ) / ( 1 − r ) ≤ t for 0 ≤ r < t < 1 0 \le r < t < 1 0 ≤ r < t < 1 , the Gaussian mixture above gives
E [ ∥ X i , t y i − e ( y i ) ∥ 2 ] ≤ ( 1 − t ) 2 ( ∥ e ( y i ) ∥ 2 + d σ s 2 ) + d σ d 2 t ( 1 − t ) ⟶ 0. \mathbb{E}[\lVert X_{i,t}^{y_i}-e(y_i)\rVert^2] \le(1-t)^2(\lVert e(y_i)\rVert^2+d\sigma_s^2)+d\sigma_d^2t(1-t)\longrightarrow0. E [∥ X i , t y i − e ( y i ) ∥ 2 ] ≤ ( 1 − t ) 2 (∥ e ( y i ) ∥ 2 + d σ s 2 ) + d σ d 2 t ( 1 − t ) ⟶ 0.
Summing over positions and averaging over Y Y Y yields p t ⇒ L ( E ( Y ) ) p_t \Rightarrow\mathcal{L}(E(Y)) p t ⇒ L ( E ( Y )) as t ↑ 1 t \uparrow1 t ↑ 1 . For a bridge clock κ \kappa κ , replace t t t by κ ( t ) \kappa(t) κ ( t ) in this bound.
The closed form extends beyond the linear clock: it applies whenever the bridge clock and the first-reset CDF are both κ ( t ) \kappa(t) κ ( t ) . For the baseline reset schedule, we use it for the repeated-reset training law and for likelihood completion in RJD-BC-D, with κ ( t ) = t 2 \kappa(t)=t^2 κ ( t ) = t 2 .
Proposition B.2 (Conditional law and likelihood). Let κ \kappa κ satisfy the clock conditions in § 2.1 . Consider the repeated-reset bridge under the independence and noise assumptions of Lemma B.1, with drift factor κ ′ ( t ) / ( 1 − κ ( t ) ) \kappa'(t)/(1-\kappa(t)) κ ′ ( t ) / ( 1 − κ ( t )) , diffusion coefficient σ d κ ′ ( t ) \sigma_d\sqrt{\kappa'(t)} σ d κ ′ ( t ) , and reset rate λ ( t ) = κ ′ ( t ) / ( 1 − κ ( t ) ) \lambda(t)=\kappa'(t)/(1-\kappa(t)) λ ( t ) = κ ′ ( t ) / ( 1 − κ ( t )) . For y i = a y_i=a y i = a and u = κ ( t ) ∈ ( 0 , 1 ) u=\kappa(t)\in(0,1) u = κ ( t ) ∈ ( 0 , 1 ) , put v u = σ d 2 u ( 1 − u ) v_u=\sigma_d^2u(1-u) v u = σ d 2 u ( 1 − u ) and v 0 , u = ( 1 − u ) 2 σ s 2 + v u v_{0,u}=(1-u)^2\sigma_s^2+v_u v 0 , u = ( 1 − u ) 2 σ s 2 + v u . Then
p i , t ( d x ∣ a ) = ( 1 − u ) N ( d x ; u e ( a ) , v 0 , u I d ) + u ∫ 0 1 N ( d x ; e ( a ) , v u z I d ) d z . (13) p_{i,t}(dx\mid a)=(1-u)\mathcal{N}(dx;ue(a),v_{0,u}I_d)+u\int_0^1\mathcal{N}(dx;e(a),v_uzI_d)\,dz. \tag*{(13)} p i , t ( d x ∣ a ) = ( 1 − u ) N ( d x ; u e ( a ) , v 0 , u I d ) + u ∫ 0 1 N ( d x ; e ( a ) , v u z I d ) d z . (13)
Its moments satisfy
E [ X i , t ∣ Y i = a ] = u ( 2 − u ) e ( a ) , \mathbb{E}[X_{i,t}\mid Y_i=a]=u(2-u)e(a), E [ X i , t ∣ Y i = a ] = u ( 2 − u ) e ( a ) ,
E [ ∥ X i , t − e ( a ) ∥ 2 ∣ Y i = a ] = ( 1 − u ) 3 ( ∥ e ( a ) ∥ 2 + d σ s 2 ) + d σ d 2 u ( 1 − u ) ( 1 − u / 2 ) . (14) \mathbb{E}[\lVert X_{i,t}-e(a)\rVert^2\mid Y_i=a]=(1-u)^3(\lVert e(a)\rVert^2+d\sigma_s^2)+d\sigma_d^2u(1-u)(1-u/2). \tag*{(14)} E [∥ X i , t − e ( a ) ∥ 2 ∣ Y i = a ] = ( 1 − u ) 3 (∥ e ( a ) ∥ 2 + d σ s 2 ) + d σ d 2 u ( 1 − u ) ( 1 − u /2 ) . (14)
For even d ≥ 4 d\ge4 d ≥ 4 and x ≠ e ( a ) x\ne e(a) x = e ( a ) , let n = d / 2 − 1 n=d/2-1 n = d /2 − 1 and β = ∥ x − e ( a ) ∥ 2 / ( 2 v u ) > 0 \beta=\lVert x-e(a)\rVert^2/(2v_u)>0 β = ∥ x − e ( a ) ∥ 2 / ( 2 v u ) > 0 . Conditional on at least one reset, the density is
∫ 0 1 N ( x ; e ( a ) , v u z I d ) d z = ( 2 π v u ) − d / 2 β − n Γ ( n , β ) , \int_0^1\mathcal{N}(x;e(a),v_uzI_d)\,dz=(2\pi v_u)^{-d/2}\beta^{-n}\Gamma(n,\beta), ∫ 0 1 N ( x ; e ( a ) , v u z I d ) d z = ( 2 π v u ) − d /2 β − n Γ ( n , β ) ,
Γ ( n , β ) = ( n − 1 ) ! e − β ∑ k = 0 n − 1 β k k ! . (15) \Gamma(n,\beta)=(n-1)!e^{-\beta}\sum_{k=0}^{n-1}\frac{\beta^k}{k!}. \tag*{(15)} Γ ( n , β ) = ( n − 1 )! e − β k = 0 ∑ n − 1 k ! β k . (15)
Here Γ ( n , β ) = ∫ β ∞ w n − 1 e − w d w \Gamma(n,\beta)=\int_\beta^\infty w^{n-1}e^{-w}\,dw Γ ( n , β ) = ∫ β ∞ w n − 1 e − w d w is the upper incomplete gamma function.
Proof. The process is the deterministic time change u = κ ( t ) u=\kappa(t) u = κ ( t ) of the linear-clock bridge with reset rate ( 1 − u ) − 1 (1-u)^{-1} ( 1 − u ) − 1 . Lemma B.1 therefore gives no-reset weight 1 − u 1-u 1 − u and last-reset density ( 1 − u ) / ( 1 − τ ) 2 (1-u)/(1-\tau)^2 ( 1 − u ) / ( 1 − τ ) 2 , for 0 < τ < u 0<\tau<u 0 < τ < u . Following a last reset at τ \tau τ , the mean is e ( a ) e(a) e ( a ) and the variance is σ d 2 ( u − τ ) ( 1 − u ) / ( 1 − τ ) \sigma_d^2(u-\tau)(1-u)/(1-\tau) σ d 2 ( u − τ ) ( 1 − u ) / ( 1 − τ ) . The substitution z = ( u − τ ) / ( u ( 1 − τ ) ) z=(u-\tau)/(u(1-\tau)) z = ( u − τ ) / ( u ( 1 − τ )) transforms this variance into v u z v_uz v u z and the last-reset measure into u d z u\,dz u d z on [ 0 , 1 ] [0,1] [ 0 , 1 ] , with reversed limits. Together with the no-reset Gaussian, this proves (13) .
The component means give ( 1 − u ) u e ( a ) + u e ( a ) = u ( 2 − u ) e ( a ) (1-u)ue(a)+ue(a)=u(2-u)e(a) ( 1 − u ) u e ( a ) + u e ( a ) = u ( 2 − u ) e ( a ) . The expected squared distance from e ( a ) e(a) e ( a ) is ( 1 − u ) [ ( 1 − u ) 2 ∥ e ( a ) ∥ 2 + d v 0 , u ] + u d v u / 2 (1-u)[(1-u)^2\lVert e(a)\rVert^2+dv_{0,u}]+udv_u/2 ( 1 − u ) [( 1 − u ) 2 ∥ e ( a ) ∥ 2 + d v 0 , u ] + u d v u /2 , since ∫ 0 1 z d z = 1 / 2 \int_0^1z\,dz=1/2 ∫ 0 1 z d z = 1/2 . Expanding gives (14) . For the density, substituting w = β / z w=\beta/z w = β / z yields
∫ 0 1 z − d / 2 e − β / z d z = β − n ∫ β ∞ w n − 1 e − w d w . \int_0^1z^{-d/2}e^{-\beta/z}\,dz=\beta^{-n}\int_\beta^\infty w^{n-1}e^{-w}\,dw. ∫ 0 1 z − d /2 e − β / z d z = β − n ∫ β ∞ w n − 1 e − w d w .
Repeated integration by parts for the positive integer n n n gives the finite sum in (15) .
To sample (13) , use the first Gaussian with probability 1 − u 1-u 1 − u ; otherwise draw z ∼ Unif ( 0 , 1 ) z\sim\operatorname{Unif}(0,1) z ∼ Unif ( 0 , 1 ) and ξ ∼ N ( 0 , I d ) \xi\sim\mathcal{N}(0,I_d) ξ ∼ N ( 0 , I d ) independently and set x = e ( a ) + v u z ξ x=e(a)+\sqrt{v_uz}\,\xi x = e ( a ) + v u z ξ . In dimension 64, the density formula is evaluated in log space without reset-time quadrature. For even d ≥ 4 d\ge4 d ≥ 4 and 0 < u < 1 0<u<1 0 < u < 1 , the density is singular at an anchor, but the law has no atom there; a posterior version may be chosen on these zero-measure points. At u = 0 u=0 u = 0 , including σ s = 0 \sigma_s=0 σ s = 0 , the common base law is handled directly. The mean squared distance in (14) tends to zero at the data endpoint.
Other jump schedules. The proposition covers λ γ ( t ) = γ t γ − 1 / ( 1 − t γ ) \lambda_\gamma(t)=\gamma t^{\gamma-1}/(1-t^\gamma) λ γ ( t ) = γ t γ − 1 / ( 1 − t γ ) when the bridge also uses κ ( t ) = t γ \kappa(t)=t^\gamma κ ( t ) = t γ . If the bridge remains at κ ( t ) = t 2 \kappa(t)=t^2 κ ( t ) = t 2 while γ \gamma γ changes, the repeated-reset process has no-reset weight 1 − t γ 1-t^\gamma 1 − t γ and last-reset density γ r γ − 1 ( 1 − t γ ) / ( 1 − r γ ) 2 \gamma r^{\gamma-1}(1-t^\gamma)/(1-r^\gamma)^2 γ r γ − 1 ( 1 − t γ ) / ( 1 − r γ ) 2 on ( 0 , t ) (0,t) ( 0 , t ) . For repeated resets, these weights must be combined with the Gaussian laws in bridge time: the no-reset mean and variance are κ ( t ) e ( a ) \kappa(t)e(a) κ ( t ) e ( a ) and v 0 , κ ( t ) v_{0,\kappa(t)} v 0 , κ ( t ) , and a last reset at physical time r r r gives mean e ( a ) e(a) e ( a ) and variance σ d 2 ( κ ( t ) − κ ( r ) ) ( 1 − κ ( t ) ) / ( 1 − κ ( r ) ) \sigma_d^2(\kappa(t)-\kappa(r))(1-\kappa(t))/(1-\kappa(r)) σ d 2 ( κ ( t ) − κ ( r )) ( 1 − κ ( t )) / ( 1 − κ ( r )) . This is the last-reset construction of Lemma B.1; the uniform-mixture formula above does not generally apply. Our non-absorbing schedule experiments also use γ = 1 , 4 \gamma=1,4 γ = 1 , 4 with the bridge fixed at κ ( t ) = t 2 \kappa(t)=t^2 κ ( t ) = t 2 . Training draws from this schedule-specific last-reset mixture. For completed drift, the local likelihood is evaluated by numerical quadrature; γ = 2 \gamma=2 γ = 2 retains the closed form above. The quadrature uses 256 generalized Gauss–Laguerre nodes and checks the completed posterior against 128 nodes at every sampled position, aborting if the maximum ℓ 1 \ell_1 ℓ 1 difference exceeds 0.002. This is a numerical convergence check, not an exact likelihood evaluation. The raw LOO and unmasked sampling arms do not evaluate the local likelihood.
For analytic completion with a learned predictor, we add log ℓ i , t ( x i ∣ a ) \log\ell_{i,t}(x_i\mid a) log ℓ i , t ( x i ∣ a ) to contextual logits and normalize over a a a . The local likelihood is computed from the conditional training path using the learned token embeddings, which also define reset destinations and nearest-embedding decoding.
Proofs and a counterexample
Minimizers of the training losses
All expectations below use the training pair ( Y , X t Y ) (Y,X_t^Y) ( Y , X t Y ) . Logarithms are natural.
Full-posterior prediction. Recall the loss (5),
L f u l l ( θ ) = E t , Y , X t Y [ − ∑ i = 1 L log p θ , i , t ( Y i ∣ X t Y ) ] . (16) \mathcal{L}_{\mathrm{full}}(\theta)=\mathbb{E}_{t,Y,X_t^Y}\left[-\sum_{i=1}^{L}\log p_{\theta,i,t}(Y_i\mid X_t^Y)\right]. \tag*{(16)} L full ( θ ) = E t , Y , X t Y [ − i = 1 ∑ L log p θ , i , t ( Y i ∣ X t Y ) ] . (16)
Condition on ( t , X t Y ) = ( t , x ) (t,X_t^Y)=(t,x) ( t , X t Y ) = ( t , x ) . At position i i i , the conditional loss from predicting p θ , i , t ( ⋅ ∣ x ) p_{\theta,i,t}(\cdot\mid x) p θ , i , t ( ⋅ ∣ x ) instead of p i , t ( ⋅ ∣ x ) p_{i,t}(\cdot\mid x) p i , t ( ⋅ ∣ x ) is
KL ( p i , t ( ⋅ ∣ x ) ∥ p θ , i , t ( ⋅ ∣ x ) ) ≥ 0. \operatorname{KL}\left(p_{i,t}(\cdot\mid x)\mathbin{\|}p_{\theta,i,t}(\cdot\mid x)\right)\ge0. KL ( p i , t ( ⋅ ∣ x ) ∥ p θ , i , t ( ⋅ ∣ x ) ) ≥ 0.
Averaging over training times and states and summing over positions therefore shows that the unrestricted minimizers equal p i , t p_{i,t} p i , t almost everywhere. The learned drift is
b θ , i , t ( x ) = ∑ a ∈ V p θ , i , t ( a ∣ x ) e ( a ) − x i 1 − t . b_{\theta,i,t}(x)=\frac{\sum_{a\in V}p_{\theta,i,t}(a\mid x)e(a)-x_i}{1-t}. b θ , i , t ( x ) = 1 − t ∑ a ∈ V p θ , i , t ( a ∣ x ) e ( a ) − x i .
and the reset probabilities are p θ , i , t p_{\theta,i,t} p θ , i , t . When p θ , i , t = p i , t p_{\theta,i,t}=p_{i,t} p θ , i , t = p i , t , these coefficients and the corresponding reset clock recover the posterior-averaged generator for the chosen path.
Leave-one-out prediction. Write X − i , t Y X_{-i,t}^Y X − i , t Y for X t Y X_t^Y X t Y with position i i i omitted. The loss (6) is
L L O O ( θ ) = E t , Y , X t Y [ − ∑ i = 1 L log q θ , i , t ( Y i ∣ X − i , t Y ) ] . (17) \mathcal{L}_{\mathrm{LOO}}(\theta)=\mathbb{E}_{t,Y,X_t^Y}\left[-\sum_{i=1}^{L}\log q_{\theta,i,t}(Y_i\mid X_{-i,t}^Y)\right]. \tag*{(17)} L LOO ( θ ) = E t , Y , X t Y [ − i = 1 ∑ L log q θ , i , t ( Y i ∣ X − i , t Y ) ] . (17)
Conditioning on ( t , X − i , t Y ) (t,X_{-i,t}^Y) ( t , X − i , t Y ) gives the conditional loss
KL ( q i , t ( ⋅ ∣ X − i , t Y ) ∥ q θ , i , t ( ⋅ ∣ X − i , t Y ) ) . \operatorname{KL}\left(q_{i,t}(\cdot\mid X_{-i,t}^Y)\mathbin{\|}q_{\theta,i,t}(\cdot\mid X_{-i,t}^Y)\right). KL ( q i , t ( ⋅ ∣ X − i , t Y ) ∥ q θ , i , t ( ⋅ ∣ X − i , t Y ) ) .
The same argument shows that the unrestricted minimizers equal q i , t q_{i,t} q i , t almost everywhere under the training law, establishing (7).
Assumptions and proof of Proposition 3.2
Proposition C.1. Consider the conditional path (3) under the assumptions of Appendix § 8 .§ 2 , with any nonnegative deterministic reset rate λ ( t ) \lambda(t) λ ( t ) satisfying ∫ 0 t λ ( s ) d s → ∞ \int_0^t\lambda(s)\,\mathrm{d}s\to\infty ∫ 0 t λ ( s ) d s → ∞ for every t < 1 t<1 t < 1 . Let p t ( d x ) = ∑ y p d a t a ( y ) p t ( d x ∣ y ) p_t(\mathrm{d}x)=\sum_y p_{\mathrm{data}}(y)p_t(\mathrm{d}x\mid y) p t ( d x ) = ∑ y p data ( y ) p t ( d x ∣ y ) , and let p i , t p_{i,t} p i , t and q i , t q_{i,t} q i , t be versions of P ( Y i ∈ ⋅ ∣ X t Y ) \mathbb{P}(Y_i\in\cdot\mid X_t^Y) P ( Y i ∈ ⋅ ∣ X t Y ) and P ( Y i ∈ ⋅ ∣ X − i , t Y ) \mathbb{P}(Y_i\in\cdot\mid X_{-i,t}^Y) P ( Y i ∈ ⋅ ∣ X − i , t Y )
under this mixture. On every [ 0 , T ] [0,T] [ 0 , T ] , T < 1 T<1 T < 1 , assume that (8) admits a Markov process started from p 0 p_0 p 0 whose laws satisfy its integrated weak forward equation against smooth, compactly supported test functions, and that this equation has a unique probability-law solution from p 0 p_0 p 0 . Then this Markov process has law p t p_t p t for every t < 1 t<1 t < 1 , and p t ⇒ L ( E ( Y ) ) p_t \Rightarrow\mathcal{L}(E(Y)) p t ⇒ L ( E ( Y )) as t ↑ 1 t \uparrow1 t ↑ 1 .
Proof. Fix T < 1 T<1 T < 1 . We establish two claims for every smooth, compactly supported test function f : R d L → R f:\mathbb{R}^{dL}\to\mathbb{R} f : R d L → R .
Claim 1. The mixture p t p_t p t satisfies the integrated weak forward equation for the averaged operator A t \mathcal{A}_t A t :
∫ f d p t − ∫ f d p 0 = ∫ 0 t ∫ A s f ( x ) p s ( d x ) d s , 0 ≤ t ≤ T . \int f\,dp_t-\int f\,dp_0=\int_0^t\int\mathcal{A}_s f(x)\,p_s(dx)\,ds,\qquad0\leq t\leq T. ∫ f d p t − ∫ f d p 0 = ∫ 0 t ∫ A s f ( x ) p s ( d x ) d s , 0 ≤ t ≤ T .
Proof of Claim 1. For a fixed endpoint y ∈ V L y\in\mathcal{V}^L y ∈ V L , the conditional generator of (3) is
A s y f ( x ) = ∑ i = 1 L [ e ( y i ) − x i 1 − s ⋅ ∇ i f ( x ) + σ d 2 2 Δ i f ( x ) ] \mathcal{A}_s^y f(x)=\sum_{i=1}^{L}\left[\frac{e(y_i)-x_i}{1-s}\cdot\nabla_i f(x)+\frac{\sigma_d^2}{2}\Delta_i f(x)\right] A s y f ( x ) = i = 1 ∑ L [ 1 − s e ( y i ) − x i ⋅ ∇ i f ( x ) + 2 σ d 2 Δ i f ( x ) ]
+ λ ( s ) ∑ i = 1 L [ f ( x i ← e ( y i ) ) − f ( x ) ] . \qquad+\lambda(s)\sum_{i=1}^{L}\left[f\left(x^{i\leftarrow e(y_i)}\right)-f(x)\right]. + λ ( s ) i = 1 ∑ L [ f ( x i ← e ( y i ) ) − f ( x ) ] .
Here ∇ i \nabla_i ∇ i and Δ i \Delta_i Δ i are the gradient and Laplacian with respect to the block x i ∈ R d x_i\in\mathbb{R}^d x i ∈ R d , and x i ← e ( y i ) x^{i\leftarrow e(y_i)} x i ← e ( y i ) replaces that block by e ( y i ) e(y_i) e ( y i ) . The continuous terms are uniformly bounded over s ≤ T s\leq T s ≤ T , x x x , and y y y : the derivatives of f f f have compact support, 1 − s ≥ 1 − T 1-s\geq1-T 1 − s ≥ 1 − T , and the finite vocabulary makes the embeddings bounded. The jump term is bounded in absolute value by 2 L λ ( s ) ∥ f ∥ ∞ 2L\lambda(s)\|f\|_\infty 2 L λ ( s ) ∥ f ∥ ∞ , where ∥ f ∥ ∞ = sup x ∣ f ( x ) ∣ \|f\|_\infty=\sup_x|f(x)| ∥ f ∥ ∞ = sup x ∣ f ( x ) ∣ . These bounds and ∫ 0 T λ ( s ) d s < ∞ \int_0^T\lambda(s)\,ds<\infty ∫ 0 T λ ( s ) d s < ∞ justify taking expectations in the jump Itô formula and interchanging the time integral with expectation; the Brownian and compensated jump integrals have zero expectation. There are no jumps at any fixed deterministic time almost surely, since the Poisson intensity measures are λ ( s ) d s \lambda(s)\,ds λ ( s ) d s , so the left limits in that formula can be replaced by X s Y X_s^Y X s Y under expectation.
Now draw Y ∼ p d a t a Y\sim p_{\mathrm{data}} Y ∼ p data and write A s Y \mathcal{A}^Y_s A s Y for the conditional generator with endpoint Y Y Y . Since P ( Y i = a ∣ X s Y = x ) = p i , s ( a ∣ x ) \mathbb{P}(Y_i=a\mid X_s^Y=x)=p_{i,s}(a\mid x) P ( Y i = a ∣ X s Y = x ) = p i , s ( a ∣ x ) , conditioning A s Y f ( X s Y ) \mathcal{A}^Y_s f(X_s^Y) A s Y f ( X s Y ) on X s Y = x X_s^Y=x X s Y = x gives exactly A s f ( x ) \mathcal{A}_s f(x) A s f ( x ) in (4). Consequently, for 0 ≤ t ≤ T 0\leq t\leq T 0 ≤ t ≤ T , taking expectation over Y Y Y and its conditional process gives
∫ f d p t − ∫ f d p 0 = ∫ 0 t E [ A s Y f ( X s Y ) ] d s = ∫ 0 t ∫ A s f ( x ) p s ( d x ) d s . \begin{aligned}
\int f\,dp_t-\int f\,dp_0=\int_0^t\mathbb{E}\left[\mathcal{A}_s^Yf(X_s^Y)\right]\,ds \\
&=\int_0^t\int\mathcal{A}_s f(x)\,p_s(dx)\,ds.
\end{aligned} ∫ f d p t − ∫ f d p 0 = ∫ 0 t E [ A s Y f ( X s Y ) ] d s = ∫ 0 t ∫ A s f ( x ) p s ( d x ) d s .
This proves Claim 1 and establishes posterior averaging directly, following the conditional-expectation argument of [17 Proposition 1] .
Claim 2. Replacing the full-posterior reset probabilities by the LOO probabilities leaves the generator integral under p t p_t p t unchanged:
∫ A t f d p t = ∫ A t c t x f d p t , 0 ≤ t ≤ T . \int\mathcal{A}_t f\,dp_t=\int\mathcal{A}^{\mathrm{ctx}}_t f\,dp_t,\qquad0\leq t\leq T. ∫ A t f d p t = ∫ A t ctx f d p t , 0 ≤ t ≤ T .
Proof of Claim 2. From (4) and (8), using ∑ a p i , t ( a ∣ x ) = ∑ a q i , t ( a ∣ x − i ) = 1 \sum_a p_{i,t}(a\mid x)=\sum_a q_{i,t}(a\mid x_{-i})=1 ∑ a p i , t ( a ∣ x ) = ∑ a q i , t ( a ∣ x − i ) = 1 ,
( A t − A t c t x ) f ( x ) = λ ( t ) ∑ i , a [ p i , t ( a ∣ x ) − q i , t ( a ∣ x − i ) ] f ( x i ← e ( a ) ) . (\mathcal{A}_t-\mathcal{A}^{\mathrm{ctx}}_t)f(x)=\lambda(t)\sum_{i,a}\left[p_{i,t}(a\mid x)-q_{i,t}(a\mid x_{-i})\right]f\left(x^{i\leftarrow e(a)}\right). ( A t − A t ctx ) f ( x ) = λ ( t ) i , a ∑ [ p i , t ( a ∣ x ) − q i , t ( a ∣ x − i ) ] f ( x i ← e ( a ) ) .
The tower property gives
∫ p i , t ( a ∣ x ) p t ( d x i ∣ x − i ) = q i , t ( a ∣ x − i ) . \int p_{i,t}(a\mid x)\,p_t(dx_i\mid x_{-i})=q_{i,t}(a\mid x_{-i}). ∫ p i , t ( a ∣ x ) p t ( d x i ∣ x − i ) = q i , t ( a ∣ x − i ) .
Since f ( x i ← e ( a ) ) f(x^{i\leftarrow e(a)}) f ( x i ← e ( a ) ) does not depend on x i x_i x i ,
∫ ( A t − A t c t x ) f d p t = λ ( t ) ∑ i , a ∫ f ( x i ← e ( a ) ) [ ∫ p i , t ( a ∣ x ) p t ( d x i ∣ x − i ) − q i , t ( a ∣ x − i ) ] p t ( d x − i ) = 0 , \int(\mathcal{A}_t-\mathcal{A}^{\mathrm{ctx}}_t)f\,\mathrm{d}p_t
=\lambda(t)\sum_{i,a}\int f(x^{i\leftarrow e(a)})\left[\int p_{i,t}(a\mid x)\,p_t(\mathrm{d}x_i\mid x_{-i})-q_{i,t}(a\mid x_{-i})\right]p_t(\mathrm{d}x_{-i})=0, ∫ ( A t − A t ctx ) f d p t = λ ( t ) i , a ∑ ∫ f ( x i ← e ( a ) ) [ ∫ p i , t ( a ∣ x ) p t ( d x i ∣ x − i ) − q i , t ( a ∣ x − i ) ] p t ( d x − i ) = 0 ,
which proves Claim 2.
Combining Claims 1 and 2, p t p_t p t solves the integrated weak forward equation for A t c t x \mathcal{A}^{\mathrm{ctx}}_t A t ctx . By the assumed uniqueness, the Markov process started from p 0 p_0 p 0 therefore has law p t p_t p t on [ 0 , T ] [0,T] [ 0 , T ] . Since T < 1 T<1 T < 1 was arbitrary, this holds for every t < 1 t<1 t < 1 . Endpoint convergence follows from the bound in Appendix B.2.
A counterexample for LOO drift
Example C.2 (LOO drift with informative context). Take L = 2 L=2 L = 2 , d = 1 d=1 d = 1 , and Y = ( B , B ) Y=(B,B) Y = ( B , B ) , where B B B is uniform on { − 1 , 1 } \{-1,1\} { − 1 , 1 } ; let e ( a ) = a e(a)=a e ( a ) = a . Conditional on B B B , let the coordinates follow independent copies of the repeated-reset bridge (3), with X 1 , 0 = X 2 , 0 = 0 X_{1,0}=X_{2,0}=0 X 1 , 0 = X 2 , 0 = 0 , σ d = 1 \sigma_d=1 σ d = 1 , and λ ( t ) = ( 1 − t ) − 1 \lambda(t)=(1-t)^{-1} λ ( t ) = ( 1 − t ) − 1 . Both positions therefore carry noisy information about the same endpoint. Define, under this prescribed conditional-path mixture,
m t ( x ) = E [ B ∣ X 2 , t = x ] , M t ( x 1 , x 2 ) = E [ B ∣ X 1 , t = x 1 , X 2 , t = x 2 ] . m_t(x)=\mathbb{E}[B\mid X_{2,t}=x],\qquad M_t(x_1,x_2)=\mathbb{E}[B\mid X_{1,t}=x_1,X_{2,t}=x_2]. m t ( x ) = E [ B ∣ X 2 , t = x ] , M t ( x 1 , x 2 ) = E [ B ∣ X 1 , t = x 1 , X 2 , t = x 2 ] .
By symmetry, m t m_t m t is also the posterior mean given the first coordinate alone. The drift b i , t b_{i,t} b i , t at position i i i is ( M t ( x 1 , x 2 ) − x i ) / ( 1 − t ) (M_t(x_1,x_2)-x_i)/(1-t) ( M t ( x 1 , x 2 ) − x i ) / ( 1 − t ) ; its LOO replacement is ( m t ( x 3 − i ) − x i ) / ( 1 − t ) (m_t(x_{3-i})-x_i)/(1-t) ( m t ( x 3 − i ) − x i ) / ( 1 − t ) . Let A ~ t \widetilde{\mathcal{A}}_t A t change only these drifts in (4), retaining the exact jump probabilities p i , t p_{i,t} p i , t , reset clocks, and diffusion coefficients. The original process with generator A t \mathcal{A}_t A t has marginal law p t p_t p t . Write p ~ t \widetilde{p}_t p t for the marginal law of the process with generator A ~ t \widetilde{\mathcal{A}}_t A t , started from the same initial law p ~ 0 = p 0 = δ ( 0 , 0 ) \widetilde{p}_0=p_0=\delta_{(0,0)} p 0 = p 0 = δ ( 0 , 0 ) .
Claim. Replacing the drift in this way does not preserve the prescribed marginal path ( p t ) 0 ≤ t < 1 (p_t)_{0\le t<1} ( p t ) 0 ≤ t < 1 , even though the jump rate kernel remains exact.
Proof. Take f ( x ) = x 1 2 f(x)=x_1^2 f ( x ) = x 1 2 . All second moments are finite on compact time intervals. Each generator determines the second-moment derivative under its own marginal law:
d d t E p t [ X 1 2 ] = ∫ A t f d p t , \frac{\mathrm{d}}{\mathrm{d}t}\mathbb{E}_{p_t}[X_1^2]=\int\mathcal{A}_t f\,\mathrm{d}p_t, d t d E p t [ X 1 2 ] = ∫ A t f d p t ,
d d t E p ~ t [ X 1 2 ] = ∫ A ~ t f d p ~ t . \frac{\mathrm{d}}{\mathrm{d}t}\mathbb{E}_{\widetilde{p}_t}[X_1^2]=\int\widetilde{\mathcal{A}}_t f\,\mathrm{d}\widetilde{p}_t. d t d E p t [ X 1 2 ] = ∫ A t f d p t .
Suppose, for contradiction, that p ~ t = p t \widetilde{p}_t=p_t p t = p t for every t < 1 t<1 t < 1 . Then these second moments, and hence their derivatives, must agree.
To compare these derivatives, we first compute the generator difference. The continuous diffusion and jump terms of the two generators coincide. For f ( x ) = x 1 2 f(x)=x_1^2 f ( x ) = x 1 2 , we have ∂ 1 f = 2 x 1 \partial_1f=2x_1 ∂ 1 f = 2 x 1 and ∂ 2 f = 0 \partial_2f=0 ∂ 2 f = 0 , so only the difference between the first-coordinate drifts contributes:
( A ~ t − A t ) f ( x ) = 2 x 1 1 − t ( m t ( x 2 ) − M t ( x 1 , x 2 ) ) . (\widetilde{\mathcal{A}}_t-\mathcal{A}_t)f(x)=\frac{2x_1}{1-t}\bigl(m_t(x_2)-M_t(x_1,x_2)\bigr). ( A t − A t ) f ( x ) = 1 − t 2 x 1 ( m t ( x 2 ) − M t ( x 1 , x 2 ) ) .
Integrating against p t p_t p t gives
∫ ( A ~ t − A t ) f d p t = 2 1 − t ( E [ X 1 , t m t ( X 2 , t ) ] − E [ X 1 , t M t ( X 1 , t , X 2 , t ) ] ) . \int(\widetilde{\mathcal{A}}_t-\mathcal{A}_t)f\,\mathrm{d}p_t
=\frac{2}{1-t}\left(\mathbb{E}[X_{1,t}m_t(X_{2,t})]-\mathbb{E}[X_{1,t}M_t(X_{1,t},X_{2,t})]\right). ∫ ( A t − A t ) f d p t = 1 − t 2 ( E [ X 1 , t m t ( X 2 , t )] − E [ X 1 , t M t ( X 1 , t , X 2 , t )] ) .
It remains to evaluate the two expectations on the right. Both are under the prescribed joint law of ( B , X t ) (B,X_t) ( B , X t ) , whose state marginal is p t p_t p t . With κ ( t ) = t \kappa(t)=t κ ( t ) = t , (14) gives E [ X 1 , t ∣ B ] = t ( 2 − t ) B \mathbb{E}[X_{1,t}\mid B]=t(2-t)B E [ X 1 , t ∣ B ] = t ( 2 − t ) B .
For the first expectation, involving the LOO term, condition on both B B B and X 2 , t X_{2,t} X 2 , t . The coordinates are independent conditional on B B B , so observing X 2 , t X_{2,t} X 2 , t provides no further information about X 1 , t X_{1,t} X 1 , t once B B B is known. Thus
E [ X 1 , t ∣ B , X 2 , t ] = E [ X 1 , t ∣ B ] = t ( 2 − t ) B . \mathbb{E}[X_{1,t}\mid B,X_{2,t}]=\mathbb{E}[X_{1,t}\mid B]=t(2-t)B. E [ X 1 , t ∣ B , X 2 , t ] = E [ X 1 , t ∣ B ] = t ( 2 − t ) B .
Since m t ( X 2 , t ) m_t(X_{2,t}) m t ( X 2 , t ) is determined by X 2 , t X_{2,t} X 2 , t , it can be taken outside this conditional expectation. The tower property therefore gives
E [ X 1 , t m t ( X 2 , t ) ] = E [ E [ X 1 , t m t ( X 2 , t ) ∣ B , X 2 , t ] ] = E [ m t ( X 2 , t ) E [ X 1 , t ∣ B , X 2 , t ] ] = t ( 2 − t ) E [ B m t ( X 2 , t ) ] . \mathbb{E}[X_{1,t}m_t(X_{2,t})] = \mathbb{E}[\mathbb{E}[X_{1,t}m_t(X_{2,t}) \mid B, X_{2,t}]]
= \mathbb{E}[m_t(X_{2,t})\mathbb{E}[X_{1,t} \mid B, X_{2,t}]]
= t(2-t)\mathbb{E}[Bm_t(X_{2,t})]. E [ X 1 , t m t ( X 2 , t )] = E [ E [ X 1 , t m t ( X 2 , t ) ∣ B , X 2 , t ]] = E [ m t ( X 2 , t ) E [ X 1 , t ∣ B , X 2 , t ]] = t ( 2 − t ) E [ B m t ( X 2 , t )] .
Next condition on X 2 , t X_{2,t} X 2 , t alone and use the definition m t ( X 2 , t ) = E [ B ∣ X 2 , t ] m_t(X_{2,t}) = \mathbb{E}[B \mid X_{2,t}] m t ( X 2 , t ) = E [ B ∣ X 2 , t ] :
E [ B m t ( X 2 , t ) ] = E [ E [ B m t ( X 2 , t ) ∣ X 2 , t ] ] = E [ m t ( X 2 , t ) E [ B ∣ X 2 , t ] ] = E [ m t ( X 2 , t ) 2 ] . \mathbb{E}[Bm_t(X_{2,t})] = \mathbb{E}[\mathbb{E}[Bm_t(X_{2,t}) \mid X_{2,t}]]
= \mathbb{E}[m_t(X_{2,t})\mathbb{E}[B \mid X_{2,t}]]
= \mathbb{E}[m_t(X_{2,t})^2]. E [ B m t ( X 2 , t )] = E [ E [ B m t ( X 2 , t ) ∣ X 2 , t ]] = E [ m t ( X 2 , t ) E [ B ∣ X 2 , t ]] = E [ m t ( X 2 , t ) 2 ] .
Combining these two calculations yields
E [ X 1 , t m t ( X 2 , t ) ] = t ( 2 − t ) E [ m t ( X 2 , t ) 2 ] . \mathbb{E}[X_{1,t}m_t(X_{2,t})] = t(2-t)\mathbb{E}[m_t(X_{2,t})^2]. E [ X 1 , t m t ( X 2 , t )] = t ( 2 − t ) E [ m t ( X 2 , t ) 2 ] .
For the second expectation, involving the full-posterior term, recall that M t ( X 1 , t , X 2 , t ) = E [ B ∣ X 1 , t , X 2 , t ] M_t(X_{1,t}, X_{2,t}) = \mathbb{E}[B \mid X_{1,t}, X_{2,t}] M t ( X 1 , t , X 2 , t ) = E [ B ∣ X 1 , t , X 2 , t ] . Here X 1 , t X_{1,t} X 1 , t is known when conditioning on both coordinates, so
E [ X 1 , t M t ( X 1 , t , X 2 , t ) ] = E [ X 1 , t E [ B ∣ X 1 , t , X 2 , t ] ] = E [ E [ X 1 , t B ∣ X 1 , t , X 2 , t ] ] = E [ X 1 , t B ] . \mathbb{E}[X_{1,t}M_t(X_{1,t}, X_{2,t})] = \mathbb{E}[X_{1,t}\mathbb{E}[B \mid X_{1,t}, X_{2,t}]]
= \mathbb{E}[\mathbb{E}[X_{1,t}B \mid X_{1,t}, X_{2,t}]]
= \mathbb{E}[X_{1,t}B]. E [ X 1 , t M t ( X 1 , t , X 2 , t )] = E [ X 1 , t E [ B ∣ X 1 , t , X 2 , t ]] = E [ E [ X 1 , t B ∣ X 1 , t , X 2 , t ]] = E [ X 1 , t B ] .
Finally, conditioning on B B B and using B 2 = 1 B^2 = 1 B 2 = 1 gives
E [ X 1 , t B ] = E [ B E [ X 1 , t ∣ B ] ] = t ( 2 − t ) E [ B 2 ] = t ( 2 − t ) . \mathbb{E}[X_{1,t}B] = \mathbb{E}[B\mathbb{E}[X_{1,t} \mid B]]
= t(2-t)\mathbb{E}[B^2] = t(2-t). E [ X 1 , t B ] = E [ B E [ X 1 , t ∣ B ]] = t ( 2 − t ) E [ B 2 ] = t ( 2 − t ) .
Substituting these two expectations into the generator difference gives
∫ ( A ~ t − A t ) f d p t = 2 t ( 2 − t ) 1 − t ( E [ m t ( X 2 , t ) 2 ] − 1 ) = − 2 t ( 2 − t ) 1 − t E [ 1 − m t ( X 2 , t ) 2 ] < 0 , 0 < t < 1. (18) \int(\widetilde{A}_t-A_t)f\,\mathrm{d}p_t
= \frac{2t(2-t)}{1-t}\left(\mathbb{E}[m_t(X_{2,t})^2]-1\right)
= -\frac{2t(2-t)}{1-t}\mathbb{E}\left[1-m_t(X_{2,t})^2\right] < 0,\qquad0<t<1. \tag*{(18)} ∫ ( A t − A t ) f d p t = 1 − t 2 t ( 2 − t ) ( E [ m t ( X 2 , t ) 2 ] − 1 ) = − 1 − t 2 t ( 2 − t ) E [ 1 − m t ( X 2 , t ) 2 ] < 0 , 0 < t < 1. (18)
To justify the strict inequality in (18) , we must show that E [ 1 − m t ( X 2 , t ) 2 ] > 0 \mathbb{E}[1-m_t(X_{2,t})^2]>0 E [ 1 − m t ( X 2 , t ) 2 ] > 0 . For either endpoint B = ± 1 B=\pm1 B = ± 1 , the local law of X 2 , t X_{2,t} X 2 , t contains the no-reset component N ( ± t , t ( 1 − t ) ) \mathcal{N}(\pm t,t(1-t)) N ( ± t , t ( 1 − t )) with weight 1 − t > 0 1-t>0 1 − t > 0 . For 0 < t < 1 0<t<1 0 < t < 1 , both Gaussian densities are positive everywhere. Hence both values of B B B retain positive conditional probability after observing X 2 , t X_{2,t} X 2 , t , so ∣ m t ( X 2 , t ) ∣ < 1 |m_t(X_{2,t})|<1 ∣ m t ( X 2 , t ) ∣ < 1 almost surely. Since B 2 = 1 B^2=1 B 2 = 1 , this gives
1 − m t ( X 2 , t ) 2 = Var ( B ∣ X 2 , t ) > 0 almost surely . 1-m_t(X_{2,t})^2=\operatorname{Var}(B\mid X_{2,t})>0\quad\text{almost surely}. 1 − m t ( X 2 , t ) 2 = Var ( B ∣ X 2 , t ) > 0 almost surely .
Its expectation is therefore strictly positive, while the prefactor − 2 t ( 2 − t ) / ( 1 − t ) -2t(2-t)/(1-t) − 2 t ( 2 − t ) / ( 1 − t ) is strictly negative. This proves the strict negativity of the residual. Figure 4 plots this residual as a function of time.
Figure 4. LOO drift violates the moment equation with informative context. The residual in (18) for Example C.2, with Y = ( B , B ) Y = (B, B) Y = ( B , B ) and σ d = 1 \sigma_d = 1 σ d = 1 . Preserving the prescribed marginal path requires a zero residual; replacing only the drift posterior by LOO gives a negative residual. The expectation is evaluated by deterministic quadrature of the exact local conditional law, not by trajectory simulation.
Under the assumed equality p ~ t = p t \widetilde{p}_t=p_t p t = p t , this yields
d d t ( E p ~ t [ X 1 2 ] − E p t [ X 1 2 ] ) = ∫ ( A ~ t − A t ) f d p t < 0 , 0 < t < 1 , \frac{d}{dt}\left(\mathbb{E}_{\widetilde{p}_t}[X_1^2]-\mathbb{E}_{p_t}[X_1^2]\right)
=\int(\widetilde{A}_t-A_t)f\,\mathrm{d}p_t<0,\qquad0<t<1, d t d ( E p t [ X 1 2 ] − E p t [ X 1 2 ] ) = ∫ ( A t − A t ) f d p t < 0 , 0 < t < 1 ,
contradicting equality of the derivatives. Thus the two marginal paths cannot coincide.
Proof of Proposition 3.3
Proof. Fix i i i and t t t . Given Y Y Y , the coordinates are independent, and the law of X i , t X_{i,t} X i , t depends on Y Y Y only through Y i Y_i Y i . Averaging over the other endpoint tokens therefore gives X i , t ⊥ X − i , t ∣ Y i X_{i,t}\perp X_{-i,t}\mid Y_i X i , t ⊥ X − i , t ∣ Y i . Consequently, for almost every context x − i x_{-i} x − i ,
P ( Y i = a , X i , t ∈ d x i ∣ X − i , t = x − i ) = q i , t ( a ∣ x − i ) p i , t ( d x i ∣ a ) = q i , t ( a ∣ x − i ) ℓ i , t ( x i ∣ a ) μ i , t ( d x i ) . \mathbb{P}(Y_i=a, X_{i,t}\in\mathrm{d}x_i\mid X_{-i,t}=x_{-i})=q_{i,t}(a\mid x_{-i})p_{i,t}(\mathrm{d}x_i\mid a)
=q_{i,t}(a\mid x_{-i})\ell_{i,t}(x_i\mid a)\mu_{i,t}(\mathrm{d}x_i). P ( Y i = a , X i , t ∈ d x i ∣ X − i , t = x − i ) = q i , t ( a ∣ x − i ) p i , t ( d x i ∣ a ) = q i , t ( a ∣ x − i ) ℓ i , t ( x i ∣ a ) μ i , t ( d x i ) .
Summing over a a a shows that the conditional density of X i , t X_{i,t} X i , t given X − i , t = x − i X_{-i,t} = x_{-i} X − i , t = x − i , with respect to μ i , t \mu_{i,t} μ i , t , is
∑ b q i , t ( b ∣ x − i ) ℓ i , t ( x i ∣ b ) . \sum_b q_{i,t}(b \mid x_{-i})\ell_{i,t}(x_i \mid b). b ∑ q i , t ( b ∣ x − i ) ℓ i , t ( x i ∣ b ) .
Bayes’ rule divides the joint density by this sum, yielding (10) for p t p_t p t -almost every x x x where the denominator is positive.
Architecture, training, and sampling
Algorithms 1 and 2 summarize training and generation for the repeated-reset model with bridge clock κ \kappa κ and deterministic physical-time reset rate λ \lambda λ . The numerical sampler freezes predictions on each grid interval and uses the no-reset probability and last-reset distribution described below. Sampling stops before t = 1 t = 1 t = 1 , where the chosen reset schedules have infinite cumulative intensity.
The positionwise update in (9) is
F ℓ ( h , a , t ) = h + a + F F ℓ ( L N ℓ ( h + a ) , t ) . F_{\ell}(h,a,t)=h+a+\mathrm{FF}_{\ell}(\mathrm{LN}_{\ell}(h+a),t). F ℓ ( h , a , t ) = h + a + FF ℓ ( LN ℓ ( h + a ) , t ) .
Here L N ℓ \mathrm{LN}_{\ell} LN ℓ is positionwise layer normalization and F F ℓ \mathrm{FF}_{\ell} FF ℓ is a positionwise feed-forward network. Time conditioning and residual gates are included in A t t n ℓ \mathrm{Attn}_{\ell} Attn ℓ and F F ℓ \mathrm{FF}_{\ell} FF ℓ ; attention also includes positionwise normalization of its queries and sources.
A separate head for predicting p i , t p_{i,t} p i , t
An alternative to analytic likelihood completion is to learn the drift posterior. Let h i ( D ) ( x − i , t ) h_i^{(D)}(x_{-i},t) h i ( D ) ( x − i , t ) denote the contextual feature defined by the input-isolated architecture in (9) . A positionwise head receives this feature together with the local observation,
r θ , i ( ⋅ ∣ x , t ) = softmax ( g θ ( h i ( D ) ( x − i , t ) , x i , t ) ) . r_{\theta,i}(\cdot\mid x,t)=\operatorname{softmax}(g_{\theta}(h_i^{(D)}(x_{-i},t),x_i,t)). r θ , i ( ⋅ ∣ x , t ) = softmax ( g θ ( h i ( D ) ( x − i , t ) , x i , t )) .
L d u a l = L L O O + E [ − ∑ i log r θ , i ( Y i ∣ X t , t ) ] . \mathcal{L}_{\mathrm{dual}}=\mathcal{L}_{\mathrm{LOO}}+\mathbb{E}\left[-\sum_i\log r_{\theta,i}(Y_i\mid X_t,t)\right]. L dual = L LOO + E [ − i ∑ log r θ , i ( Y i ∣ X t , t ) ] .
For a fixed conditional path, including fixed embeddings and corruption schedules, assume that the model class jointly represents both posteriors. Every global minimizer of the expected loss then satisfies q θ , i , t = q i , t q_{\theta,i,t}=q_{i,t} q θ , i , t = q i , t and r θ , i = p i , t r_{\theta,i}=p_{i,t} r θ , i = p i , t almost everywhere under the training law of ( t , X t Y ) (t,X_t^Y) ( t , X t Y ) , recovering the coefficients of (8) under that law. This statement does not constrain predictions outside the support of the training law, in particular beyond the training-time cutoff. Marginal preservation over a given time interval still requires exact posterior coefficients on that interval and the existence and uniqueness assumptions of Proposition C.1. The local input and the drift-head activations never enter the jump branch. Both losses may update the shared contextual parameters.
Simulation-free training
Algorithm 1 samples training states directly from the conditional law in Lemma B.1 , using bridge time κ ( t ) \kappa(t) κ ( t ) . Conditional on Y Y Y and t t t , reset times and Gaussian draws are independent across positions; examples are sampled independently. The atom τ i = 0 \tau_i=0 τ i = 0 represents no reset; the predictor observes only x x x and t t t , without reset times or indicators. Gradients pass through the sampled states to the embedding parameters.
The Text8 experiments use t max = 0.999 t_{\max}=0.999 t m a x = 0.999 . For λ ( t ) = κ ′ ( t ) / ( 1 − κ ( t ) ) \lambda(t)=\kappa'(t)/(1-\kappa(t)) λ ( t ) = κ ′ ( t ) / ( 1 − κ ( t )) , the equivalent mixture (13) can also be sampled directly. The unrestricted control uses the same state sampler and replaces q θ , i , t q_{\theta,i,t} q θ , i , t by p θ , i , t p_{\theta,i,t} p θ , i , t in the loss. Pure diffusion uses λ = 0 \lambda=0 λ = 0 in both training and sampling.
Frozen-coefficient interval update
For the linear bridge clock κ ( t ) = t \kappa(t)=t κ ( t ) = t , consider an interval [ a , b ] ⊂ [ 0 , 1 ) [a,b]\subset[0,1) [ a , b ] ⊂ [ 0 , 1 ) and freeze the contextual reset distribution q i q_i q i and drift endpoint mean m i m_i m i at its left endpoint. With S ( a , b ) = exp ( − ∫ a b λ ( r ) d r ) S(a,b)=\exp\left(-\int_a^b\lambda(r)\,\mathrm{d}r\right) S ( a , b ) = exp ( − ∫ a b λ ( r ) d r ) , the probability of no reset is S ( a , b ) S(a,b) S ( a , b ) , and the unconditional density of the last reset time τ ∈ ( a , b ) \tau\in(a,b) τ ∈ ( a , b ) is λ ( τ ) S ( τ , b ) \lambda(\tau)S(\tau,b) λ ( τ ) S ( τ , b ) . Its destination is sampled from q i q_i q i , independently of the clock. Earlier resets can be integrated out because the last reset replaces the state completely.
Set τ = a \tau=a τ = a and z i = X i , a z_i=X_{i,a} z i = X i , a if no reset occurs; otherwise let τ \tau τ be the last reset time and z i z_i z i its destination embedding. The exact update for this frozen-coefficient process is
X i , b = m i + α ( z i − m i ) + σ d ( b − τ ) α ξ i , α = 1 − b 1 − τ , ξ i ∼ N ( 0 , I d ) . (19) X_{i,b}=m_i+\alpha(z_i-m_i)+\sigma_d\sqrt{(b-\tau)\alpha}\,\xi_i,\qquad\alpha=\frac{1-b}{1-\tau},\qquad\xi_i\sim\mathcal{N}(0,I_d). \tag*{(19)} X i , b = m i + α ( z i − m i ) + σ d ( b − τ ) α ξ i , α = 1 − τ 1 − b , ξ i ∼ N ( 0 , I d ) . (19)
Algorithm 1 Simulation-free training of the RJD LOO predictor
Require: Data law p d a t a p_{\mathrm{data}} p data ; trainable predictor q θ , i , t q_{\theta,i,t} q θ , i , t and embedding e e e ; bridge clock κ \kappa κ , reset rate λ \lambda λ , noise scales σ s , σ d \sigma_s,\sigma_d σ s , σ d ; batch size B B B and training cutoff 0 < t max < 1 0 < t_{\max} < 1 0 < t m a x < 1 .
while training budget remains do
for b = 1 , … , B b = 1, \ldots, B b = 1 , … , B do
Draw Y ∼ p d a t a Y \sim p_{\mathrm{data}} Y ∼ p data , t ∼ Unif ( 0 , t max ) t \sim\operatorname{Unif}(0,t_{\max}) t ∼ Unif ( 0 , t m a x ) ; set u ← κ ( t ) u \leftarrow\kappa(t) u ← κ ( t ) .
for i = 1 , … , L i = 1, \ldots, L i = 1 , … , L do
Draw τ i ∈ [ 0 , t ] \tau_i \in[0,t] τ i ∈ [ 0 , t ] with P ( τ i ≤ h ) = S ( h , t ) \mathbb{P}(\tau_i \le h) = S(h,t) P ( τ i ≤ h ) = S ( h , t ) for 0 ≤ h ≤ t 0 \le h \le t 0 ≤ h ≤ t .
Draw ξ i ∼ N ( 0 , I d ) \xi_i \sim\mathcal{N}(0,I_d) ξ i ∼ N ( 0 , I d ) independently.
if τ i = 0 \tau_i = 0 τ i = 0 then
x i ← u e ( Y i ) + ( 1 − u ) 2 σ s 2 + σ d 2 u ( 1 − u ) ξ i x_i \leftarrow u e(Y_i) + \sqrt{(1-u)^2 \sigma_s^2 + \sigma_d^2 u(1-u)}\,\xi_i x i ← u e ( Y i ) + ( 1 − u ) 2 σ s 2 + σ d 2 u ( 1 − u ) ξ i .
else
x i ← e ( Y i ) + σ d ( u − κ ( τ i ) ) ( 1 − u ) 1 − κ ( τ i ) ξ i x_i \leftarrow e(Y_i) + \sigma_d \sqrt{\frac{(u-\kappa(\tau_i))(1-u)}{1-\kappa(\tau_i)}}\,\xi_i x i ← e ( Y i ) + σ d 1 − κ ( τ i ) ( u − κ ( τ i )) ( 1 − u ) ξ i .
end if
end for
One forward pass gives q θ , i , t ( ⋅ ∣ x − i ) q_{\theta,i,t}(\cdot\mid x_{-i}) q θ , i , t ( ⋅ ∣ x − i ) for all i i i .
L b ← − 1 L ∑ i = 1 L log q θ , i , t ( Y i ∣ x − i ) \mathcal{L}_b \leftarrow-\frac{1}{L}\sum_{i=1}^{L}\log q_{\theta,i,t}(Y_i \mid x_{-i}) L b ← − L 1 ∑ i = 1 L log q θ , i , t ( Y i ∣ x − i ) .
end for
Update θ \theta θ and the embedding parameters using B − 1 ∑ b = 1 B L b B^{-1}\sum_{b=1}^{B}\mathcal{L}_b B − 1 ∑ b = 1 B L b .
end while
For a general bridge clock κ \kappa κ , keep a a a , b b b , τ \tau τ in physical time when computing S S S and sampling resets, and apply (19) with b , τ b,\tau b , τ replaced by κ ( b ) , κ ( τ ) \kappa(b),\kappa(\tau) κ ( b ) , κ ( τ ) in α \alpha α and the Gaussian variance. This is the construction used in Algorithm 2 and does not require κ \kappa κ to be invertible. Alternatively, to sample resets directly in bridge time u = κ ( t ) u=\kappa(t) u = κ ( t ) , suppose κ \kappa κ is strictly increasing with κ ′ ( t ) > 0 \kappa'(t)>0 κ ′ ( t ) > 0 for 0 < t < 1 0<t<1 0 < t < 1 . The reset rate per unit bridge time is then
λ b r i d g e ( u ) = λ ( κ − 1 ( u ) ) κ ′ ( κ − 1 ( u ) ) , 0 < u < 1 , \lambda_{\mathrm{bridge}}(u)=\frac{\lambda\left(\kappa^{-1}(u)\right)}{\kappa'\left(\kappa^{-1}(u)\right)}, \qquad0<u<1, λ bridge ( u ) = κ ′ ( κ − 1 ( u ) ) λ ( κ − 1 ( u ) ) , 0 < u < 1 ,
where λ \lambda λ remains the physical-time rate. In particular, ∫ κ ( a ) κ ( b ) λ b r i d g e ( u ) d u = ∫ a b λ ( t ) d t \int_{\kappa(a)}^{\kappa(b)}\lambda_{\mathrm{bridge}}(u)\,\mathrm{d}u=\int_a^b\lambda(t)\,\mathrm{d}t ∫ κ ( a ) κ ( b ) λ bridge ( u ) d u = ∫ a b λ ( t ) d t , so both clocks give the same no-reset probability. For κ ( t ) = t 2 \kappa(t)=t^2 κ ( t ) = t 2 , the baseline physical rate 2 t / ( 1 − t 2 ) 2t/(1-t^2) 2 t / ( 1 − t 2 ) becomes λ b r i d g e ( u ) = 1 / ( 1 − u ) \lambda_{\mathrm{bridge}}(u)=1/(1-u) λ bridge ( u ) = 1/ ( 1 − u ) , whereas the physical rate 1 / ( 1 − t ) 1/(1-t) 1/ ( 1 − t ) becomes 1 / [ 2 u ( 1 − u ) ] 1/[2\sqrt{u}(1-\sqrt{u})] 1/ [ 2 u ( 1 − u )] .
The post-reset drift uses m i m_i m i , not the last sampled embedding as an additional hidden endpoint. Predictions are recomputed at the next grid point. This uses one contextual backbone call per interval and no terminal prediction call. Freezing state-dependent predictions, truncating before the endpoint, and decoding at the cutoff are numerical approximations.
Repeated-reset generation
Algorithm 2 uses physical reset times and applies (19) in bridge time. The local likelihood ℓ i , t \ell_{i,t} ℓ i , t is defined in Proposition 3.3 and must match the training path and embedding. Appendices § 8 .§ 2 and § 8 .§ 3 give its Gaussian-mixture form and evaluation. All predictions are computed from the state at the interval’s left endpoint and held fixed throughout that interval. Reset times, destination draws, and Gaussian noises are independent across positions conditional on those predictions.
Algorithm 2 RJD generation with LOO resets and completed or raw LOO drift
Require: Trained q θ , i , t q_{\theta,i,t} q θ , i , t and e e e ; κ \kappa κ , λ \lambda λ , σ s \sigma_s σ s , σ d \sigma_d σ d ; grid 0 = t 0 < ⋯ < t M < 1 0=t_0<\cdots<t_M<1 0 = t 0 < ⋯ < t M < 1 ; drift choice; local likelihood ℓ i , t \ell_{i,t} ℓ i , t if completed.
Draw x i ∼ N ( 0 , σ s 2 I d ) x_i\sim\mathcal{N}(0,\sigma_s^2I_d) x i ∼ N ( 0 , σ s 2 I d ) independently for i = 1 , … , L i=1,\ldots,L i = 1 , … , L .
for j = 0 , … , M − 1 j=0,\ldots,M-1 j = 0 , … , M − 1 do
Set s ← t j s\leftarrow t_j s ← t j , r ← t j + 1 r\leftarrow t_{j+1} r ← t j + 1 .
One forward pass gives q i ( a ) ← q θ , i , s ( a ∣ x − i ) q_i(a)\leftarrow q_{\theta,i,s}(a\mid x_{-i}) q i ( a ) ← q θ , i , s ( a ∣ x − i ) for all i , a i,a i , a .
if using completed drift (RJD-BC-D) then
m i ← ∑ a ∈ V q i ( a ) ℓ i , s ( x i ∣ a ) e ( a ) ∑ a ∈ V q i ( a ) ℓ i , s ( x i ∣ a ) m_i\leftarrow\frac{\sum_{a\in\mathcal{V}}q_i(a)\ell_{i,s}(x_i\mid a)e(a)}{\sum_{a\in\mathcal{V}}q_i(a)\ell_{i,s}(x_i\mid a)} m i ← ∑ a ∈ V q i ( a ) ℓ i , s ( x i ∣ a ) ∑ a ∈ V q i ( a ) ℓ i , s ( x i ∣ a ) e ( a ) for all i i i .
else ▹ \triangleright ▹ raw LOO drift (RJD-LOO)
m i ← ∑ a ∈ V q i ( a ) e ( a ) m_i\leftarrow\sum_{a\in\mathcal{V}}q_i(a)e(a) m i ← ∑ a ∈ V q i ( a ) e ( a ) for all i i i .
end if
for i = 1 , … , L i=1,\ldots,L i = 1 , … , L do
Draw τ i ∈ [ s , r ] \tau_i\in[s,r] τ i ∈ [ s , r ] with P ( τ i ≤ h ) = S ( h , r ) \mathbb{P}(\tau_i\le h)=S(h,r) P ( τ i ≤ h ) = S ( h , r ) for s ≤ h ≤ r s\le h\le r s ≤ h ≤ r .
if τ i = s \tau_i=s τ i = s then ▹ \triangleright ▹ no reset
z i ← x i z_i\leftarrow x_i z i ← x i .
else
Draw A i ∼ q i A_i\sim q_i A i ∼ q i ; set z i ← e ( A i ) z_i\leftarrow e(A_i) z i ← e ( A i ) .
end if
Set α i ← 1 − κ ( r ) 1 − κ ( τ i ) \alpha_i\leftarrow\frac{1-\kappa(r)}{1-\kappa(\tau_i)} α i ← 1 − κ ( τ i ) 1 − κ ( r ) ; draw ξ i ∼ N ( 0 , I d ) \xi_i\sim\mathcal{N}(0,I_d) ξ i ∼ N ( 0 , I d ) independently.
x i ′ ← m i + α i ( z i − m i ) + σ d ( κ ( r ) − κ ( τ i ) ) α i ξ i x_i'\leftarrow m_i+\alpha_i(z_i-m_i)+\sigma_d\sqrt{(\kappa(r)-\kappa(\tau_i))\alpha_i}\,\xi_i x i ′ ← m i + α i ( z i − m i ) + σ d ( κ ( r ) − κ ( τ i )) α i ξ i .
end for
x ← x ′ x\leftarrow x' x ← x ′ .
end for
Decode Y ^ i ∈ arg min a ∈ V ∥ x i − e ( a ) ∥ 2 \widehat{Y}_i\in\arg\min_{a\in\mathcal{V}}\lVert x_i-e(a)\rVert^2 Y i ∈ arg min a ∈ V ∥ x i − e ( a ) ∥ 2 for each i i i ; return Y ^ \widehat{Y} Y .
The law of τ i \tau_i τ i has an atom of mass S ( s , r ) S(s,r) S ( s , r ) at s s s and unconditional density λ ( h ) S ( h , r ) \lambda(h)S(h,r) λ ( h ) S ( h , r ) on ( s , r ) (s,r) ( s , r ) . Thus each interval includes any number of resets through its last-reset law, and every coordinate remains eligible for later resets. The algorithm uses M M M network evaluations and no terminal prediction call. Raw LOO drift is the empirical variant and is not covered by Proposition 3.2 .
Baseline paths and information access
For Text8, masked discrete flow matching (DFM) [13 ] uses m → A i m\to A_i m → A i and embedded pure jumps (EPJ) use 0 → e ( A i ) 0\to e(A_i) 0 → e ( A i ) . Both use q θ , i , t q_{\theta,i,t} q θ , i , t and the same backbone after the input encoder; each position jumps once and remains fixed, with reveal probability κ ( t ) = t 2 \kappa(t)=t^2 κ ( t ) = t 2 .
The pure-jump reference starts with every coordinate unresolved, commits to e ( Y i ) e(Y_i) e ( Y i ) at rate κ ′ ( t ) / ( 1 − κ ( t ) ) \kappa'(t)/(1-\kappa(t)) κ ′ ( t ) / ( 1 − κ ( t )) , and remains fixed after commitment. The baseline experiments use κ ( t ) = t 2 \kappa(t)=t^2 κ ( t ) = t 2 . Its augmented state records commitment. At an unresolved position the local state carries no token information, so p i , t p_{i,t} p i , t and q i , t q_{i,t} q i , t agree there in the idealized model. Their learned architectures and losses can nevertheless behave differently.
Pure diffusion follows (3) with λ = 0 \lambda=0 λ = 0 . Its conditional law is N ( t e ( Y i ) , [ ( 1 − t ) 2 σ s 2 + σ d 2 t ( 1 − t ) ] I d ) \mathcal{N}\left(te(Y_i),[(1-t)^2\sigma_s^2+\sigma_d^2t(1-t)]I_d\right) N ( t e ( Y i ) , [( 1 − t ) 2 σ s 2 + σ d 2 t ( 1 − t )] I d ) . A predictor of p i , t p_{i,t} p i , t or exact likelihood-completed LOO supplies its drift mean; raw LOO is an empirical ablation without that justification. Absorbing jump diffusion follows this bridge until its first reset, then holds both the anchor and an observed commitment indicator fixed. Completion for that control includes the observed status. Proposition 3.2 concerns repeated resets without such an indicator and is not its proof.
The numerical experiments use the time change κ ( t ) = t 2 \kappa(t)=t^2 κ ( t ) = t 2 for the continuous bridge.
Four-token parity diagnostic
For Table 1 , we use V = { 0 , 1 } \mathcal{V} = \{0,1\} V = { 0 , 1 } , L = 4 L = 4 L = 4 , scalar embeddings e ( a ) = 2 a − 1 e(a) = 2a - 1 e ( a ) = 2 a − 1 , and p d a t a p_{\mathrm{data}} p data uniform on { y ∈ V 4 : ∑ i y i ≡ 0 ( m o d 2 ) } \{y \in\mathcal{V}^{4} : \sum_i y_i \equiv0 \pmod{2}\} { y ∈ V 4 : ∑ i y i ≡ 0 ( mod 2 )} . Each token is marginally uniform, but the other three determine it exactly. A generated string is invalid if it has odd parity; independently sampling the four marginal distributions gives 50% invalid strings. We use λ ( t ) = ( 1 − t ) − 1 \lambda(t) = (1-t)^{-1} λ ( t ) = ( 1 − t ) − 1 and train equally sized MLPs for 250 updates. Table 1 reports cross-entropy and the percentage of invalid strings, averaged over three seeds with 8,192 generated strings per seed.
Text8 experimental details and additional results
Training and sampling protocol
We use 90 million training characters and a nominal five-million-character validation split of Text8 [26 ] , divided into length-128 sequences with incomplete chunks discarded. The alphabet consists of the 26 lower-case letters and space. Each model is trained for 60,000 updates with batch size 512 and uniform training times on [ 0 , .999 ] [0,.999] [ 0 , .999 ] . The endpoint cross-entropy includes all positions. AdamW uses learning rate 10 − 3 10^{-3} 1 0 − 3 , no warmup or weight decay, coefficients ( 0.9 , 0.999 ) (0.9, 0.999) ( 0.9 , 0.999 ) , numerical constant 10 − 8 10^{-8} 1 0 − 8 , and gradient-norm clipping at one. Evaluation uses the network and embedding EMA with decay 0.9999. Training and evaluation use FP32 and the math attention backend.
The shared backbone has eight blocks of width 256, eight attention heads, 32 time features, AdaLN-Zero conditioning [28 ] , SwiGLU width 704 [36 ] , and dropout 0.1. The LOO and unrestricted variants have the same 6,878,880 trainable parameters, including the normalized 27 × 64 27 \times64 27 × 64 embedding matrix. The LOO construction excludes the local input through the fixed-source keys and values in (9) ; it is not obtained by masking only the diagonal of ordinary multilayer self-attention. DFM replaces the geometric input encoder with a token-plus-mask lookup and has 6,867,616 parameters, approximately 0.164% fewer. Its backbone after the input encoder and its token-level conditional path match EPJ.
The continuous bridge uses the fixed schedule κ ( t ) = t 2 \kappa(t)=t^2 κ ( t ) = t 2 . During continuous motion, the conditional drift in physical time is 2 t ( e ( y i ) − x i ) / ( 1 − t 2 ) 2t(e(y_i)-x_i)/(1-t^2) 2 t ( e ( y i ) − x i ) / ( 1 − t 2 ) and the diffusion coefficient is σ d 2 t \sigma_d\sqrt{2t} σ d 2 t . The jump-schedule comparison keeps each bridge fixed and vary only the physical-time reset rate λ γ ( t ) = γ t γ − 1 / ( 1 − t γ ) \lambda_\gamma(t)=\gamma t^{\gamma-1}/(1-t^\gamma) λ γ ( t ) = γ t γ − 1 / ( 1 − t γ ) , γ ∈ { 1 , 2 , 4 } \gamma\in\{1,2,4\} γ ∈ { 1 , 2 , 4 } .
Unless specified otherwise, we use the cosine grid
t j = .999 [ 1 − cos ( π j 2 M ) ] , j = 0 , … , M , t_j = .999\left[1-\cos\left(\frac{\pi j}{2M}\right)\right], \qquad j=0,\ldots,M, t j = .999 [ 1 − cos ( 2 M π j ) ] , j = 0 , … , M ,
where M M M is the number of predictor intervals. NFE counts actual backbone evaluations, including terminal predictions for unabsorbed coordinates. Pure diffusion and RJD are decoded to the nearest token embedding at the cutoff; absorbing jump models use the stated token rule for remaining unabsorbed positions. The conditional-path comparisons use temperature one and CAT for jumps; the separately labelled TOPP variant changes only reset destinations. AJD-BC-D and AJD-BC reuse the AJD-LOO checkpoints; they change posterior routing without additional training or backbone evaluations. Their local likelihood is Gaussian at an unabsorbed coordinate and includes the token atom and observed status after absorption. PD-BC similarly reuses PD-LOO.
Repeated resets. RJD uses (3) with ( σ s , σ d ) = ( 0 , 1 ) (\sigma_s,\sigma_d)=(0,1) ( σ s , σ d ) = ( 0 , 1 ) and independent reset clocks of rate 1 / ( 1 − u ) 1/(1-u) 1/ ( 1 − u ) in path time u = κ ( t ) = t 2 u=\kappa(t)=t^2 u = κ ( t ) = t 2 , or physical-time intensity κ ′ ( t ) / ( 1 − κ ( t ) ) = 2 t / ( 1 − t 2 ) \kappa'(t)/(1-\kappa(t))=2t/(1-t^2) κ ′ ( t ) / ( 1 − κ ( t )) = 2 t / ( 1 − t 2 ) . Training samples follow (13) with u = κ ( t ) u=\kappa(t) u = κ ( t ) ; no absorption indicator is observed. RJD-LOO and RJD-BC-D reuse the same repeated-reset LOO checkpoints. For RJD-BC-D, the local likelihood is evaluated using (15), without reset-time quadrature. Sampling freezes the predictions on each grid interval and uses the last-reset update (17) between κ ( t j ) \kappa(t_j) κ ( t j ) and κ ( t j + 1 ) \kappa(t_{j+1}) κ ( t j + 1 ) . The last-reset law integrates out any earlier resets within that interval; diffusion continues afterwards. Each interval costs one network evaluation, and nearest-embedding decoding costs none. RJD-BC-D has the prescribed marginals with exact posteriors and dynamics; learned prediction and finite-step sampling introduce approximations. RJD-LOO additionally replaces the drift posterior by q θ , i , t q_{\theta,i,t} q θ , i , t and is an empirical ablation. RJD and AJD use path-specific training, different observed states, and different terminal rules, so their comparison does not isolate the effect of absorption alone.
Main evaluation. The eight settings in Table 2 and RJD-LOO TOPP in Table 7 use 4,096 sequences per training seed at 128 NFE, generated in batches of 64 on A100 GPUs with TF32 disabled. Both tables round D 5 D_5 D 5 and E 5 E_5 E 5 to integers. Batch b ∈ { 0 , … , 63 } b\in\{0,\ldots,63\} b ∈ { 0 , … , 63 } for training seed s s s uses generation seed 625000 + 1000s s s + b b b . RJD and PD use 128 predictor intervals; DFM and EPJ use 127 intervals and one terminal prediction. Actual backbone calls are checked in every batch. The checkpoints are unchanged across sampling variants; fresh generation adds no training replicates.
Table 7. Calibration on 4,096 length-128 sequences per replicate. Model rows use three training seeds at 128 NFE; real text uses three validation blocks. All rows share the dictionary and validation reference. Counts are rounded to integers. Model rows add V 6 V_{6} V 6 to the main comparison and include RJD-LOO with TOPP.
The main comparison uses validation characters [3,000,000 : 4,000,000] as the 4-gram reference. The separate 1,024-sample ablations use the first million validation characters unless stated otherwise. The fixed word dictionary is derived from the test split and has been reused during development; this is not an untouched test evaluation. Dictionary membership is a lexical metric, not a measure of sentence grammaticality. We compute each metric separately for each seed, then report its arithmetic mean and sample standard deviation across seeds 0, 1, and 2. Repeated evaluations of one checkpoint are not additional training seeds. The path, decoding, and clock comparisons have separate evaluation records, which accounts for small differences between their CAT results.
Evaluation metrics
Word validity and diversity. Words are sequences of lowercase letters a–z, separated by spaces or other non-letter characters. Word fragments at sample boundaries are retained. For each seed, let W ℓ \mathcal{W}_{\ell} W ℓ be the multiset of generated words of length at least ℓ ∈ { 5 , 6 } \ell\in\{5,6\} ℓ ∈ { 5 , 6 } , and D \mathcal{D} D the fixed dictionary from the Text8 test split. Define
V ℓ = ∑ w ∈ W ℓ 1 { w ∈ D } ∣ W ℓ ∣ , V_{\ell}=\frac{\sum_{w\in\mathcal{W}_{\ell}}\mathbf{1}\{w\in\mathcal{D}\}}{|\mathcal{W}_{\ell}|}, V ℓ = ∣ W ℓ ∣ ∑ w ∈ W ℓ 1 { w ∈ D } ,
D ℓ = ∣ { w : w ∈ W ℓ , w ∈ D } ∣ . D_{\ell}=|\{w:w\in\mathcal{W}_{\ell},\,w\in\mathcal{D}\}|. D ℓ = ∣ { w : w ∈ W ℓ , w ∈ D } ∣.
Compare D ℓ D_{\ell} D ℓ only at matched sample counts and sequence lengths.
Effective valid-word count. If f ℓ ( w ) f_{\ell}(w) f ℓ ( w ) is the relative frequency of w w w among valid words in W ℓ \mathcal{W}_{\ell} W ℓ , define
E ℓ = exp ( − ∑ w ∈ D : f ℓ ( w ) > 0 f ℓ ( w ) log f ℓ ( w ) ) , E_{\ell}=\exp\left(-\sum_{\substack{w\in\mathcal{D}:\\ f_{\ell}(w)>0}}f_{\ell}(w)\log f_{\ell}(w)\right), E ℓ = exp − w ∈ D : f ℓ ( w ) > 0 ∑ f ℓ ( w ) log f ℓ ( w ) ,
setting E ℓ = 0 E_{\ell}=0 E ℓ = 0 when no valid words occur.
Exact duplicates and training overlap. The duplicate fraction is one minus the number of distinct complete sequences divided by the sample count. The exact training-match fraction is the fraction of generated sequences occurring as contiguous length-128 segments anywhere in the full training text, including across stored row boundaries. It does not test partial or approximate copying.
Character 4-gram divergence. Let C g e n ( g ) C_{\mathrm{gen}}(g) C gen ( g ) and C r e f ( g ) C_{\mathrm{ref}}(g) C ref ( g ) count overlapping character 4-grams g g g (including spaces) in generated samples and the validation reference. With
S = { g : C g e n ( g ) + C r e f ( g ) > 0 } , ε = 10 − 8 , S = \{g : C_{\mathrm{gen}}(g) + C_{\mathrm{ref}}(g) > 0\}, \qquad\varepsilon= 10^{-8}, S = { g : C gen ( g ) + C ref ( g ) > 0 } , ε = 1 0 − 8 ,
use the smoothed distributions
P ( g ) = C g e n ( g ) + ε ∑ h ∈ S C g e n ( h ) + ε ∣ S ∣ , Q ( g ) = C r e f ( g ) + ε ∑ h ∈ S C r e f ( h ) + ε ∣ S ∣ P(g) = \frac{C_{\mathrm{gen}}(g) + \varepsilon}{\sum_{h \in S} C_{\mathrm{gen}}(h) + \varepsilon|S|}, \qquad Q(g) = \frac{C_{\mathrm{ref}}(g) + \varepsilon}{\sum_{h \in S} C_{\mathrm{ref}}(h) + \varepsilon|S|} P ( g ) = ∑ h ∈ S C gen ( h ) + ε ∣ S ∣ C gen ( g ) + ε , Q ( g ) = ∑ h ∈ S C ref ( h ) + ε ∣ S ∣ C ref ( g ) + ε
and report
JSD 4 = 1 2 ∑ g ∈ S P ( g ) log P ( g ) M ( g ) + 1 2 ∑ g ∈ S Q ( g ) log Q ( g ) M ( g ) , M ( g ) = P ( g ) + Q ( g ) 2 . \operatorname{JSD}_{4} = \frac{1}{2}\sum_{g \in S} P(g) \log\frac{P(g)}{M(g)} + \frac{1}{2}\sum_{g \in S} Q(g) \log\frac{Q(g)}{M(g)}, \qquad M(g) = \frac{P(g) + Q(g)}{2}. JSD 4 = 2 1 g ∈ S ∑ P ( g ) log M ( g ) P ( g ) + 2 1 g ∈ S ∑ Q ( g ) log M ( g ) Q ( g ) , M ( g ) = 2 P ( g ) + Q ( g ) .
All logarithms are natural, so 0 ≤ JSD 4 ≤ log 2 0 \leq\operatorname{JSD}_{4} \leq\log2 0 ≤ JSD 4 ≤ log 2 . Counts never cross generated-sample boundaries; the reference is contiguous across stored chunks. The main 4,096-sample comparison and Appendix § 13 .9 use validation characters [3,000,000 : 4,000,000]. Unless stated otherwise, the separate 1,024-sample ablations use the first 10 6 10^{6} 1 0 6 validation characters. Compare scores only at matched reference slices and sample counts.
Aggregation across seeds. Compute each metric on each seed’s complete sample set. For values r s r_s r s from training seeds s = 0 , 1 , 2 s = 0, 1, 2 s = 0 , 1 , 2 , report r ˉ ± SD ( r ) \bar{r} \pm\operatorname{SD}(r) r ˉ ± SD ( r ) , where
r ˉ = 1 3 ∑ s = 0 2 r s , SD ( r ) = 1 2 ∑ s = 0 2 ( r s − r ˉ ) 2 . \bar{r} = \frac{1}{3}\sum_{s=0}^{2} r_s, \qquad\operatorname{SD}(r) = \sqrt{\frac{1}{2}\sum_{s=0}^{2}(r_s - \bar{r})^2}. r ˉ = 3 1 s = 0 ∑ 2 r s , SD ( r ) = 2 1 s = 0 ∑ 2 ( r s − r ˉ ) 2 .
All transformations, including the exponential defining E ℓ E_{\ell} E ℓ , precede averaging. Report V ℓ V_{\ell} V ℓ as percentages and round count statistics only for display.
Real-text calibration and supplementary diagnostics
Table 7 adds V 6 V_{6} V 6 and real-text controls to the main 4,096-sample comparison. The eight main rows reuse samples from Table 2 ; the additional RJD-LOO row applies TOPP (p = 0.9 p = 0.9 p = 0.9 ) to reset destinations, retaining the cosine grid and temperature-one raw LOO drift. For real text, we take three disjoint blocks of 128 × 4096 128 \times4096 128 × 4096 validation characters, starting at offsets 1,100,000, 1,700,000, and 2,300,000, and divide each into length-128 passages. These blocks are disjoint from the common 4-gram reference [3,000,000 : 4,000,000]. We report mean and sample standard deviation across training seeds for models and across text blocks for the real controls.
RJD-BC-D has higher D 5 D_{5} D 5 and E 5 E_{5} E 5 than RJD-LOO, while RJD-LOO has higher validity and lower four-gram divergence. TOPP improves RJD-LOO’s validity and divergence with similar D 5 D_{5} D 5 but slightly lower E 5 E_{5} E 5 . Real-text scores vary across blocks, and their four-gram divergence can exceed that of generated sets; they do not define a universal lower bound.
Within each of the 27 generated sets of 4,096 samples, all complete sequences are distinct, and none exactly matches a contiguous 128-character segment of the training text. This does not exclude partial or approximate copying. Real-text controls have an exact training-match fraction of 0.285%± \pm ± 0.322% across blocks. These diagnostics concern lexical statistics, short-range character agreement, and exact repetition, rather than grammaticality or coherence.
Text8 interventions with full-vocabulary completion
Using 60k-update PD-LOO and RJD-LOO predictors, we compare raw drift with Bayesian completion. We replace a selected noisy state with an independently corrupted state for a different donor character d d d , initially preserving its surroundings, then continue all positions to t = 0.999 t = 0.999 t = 0.999 with 128 network calls. Raw and completed variants share a LOO checkpoint. We use three training seeds and 957 replacements per seed, each using a donor character different from the original.
The immediate argmax uses the drift probabilities just after replacement: q θ , i , t q_{\theta,i,t} q θ , i , t for PD-LOO and RJD-LOO, or its likelihood completion via (10) for PD-BC and RJD-BC-D. Both RJD variants use raw LOO jump probabilities.
The local word check uses the Text8 test lexicon of all word lengths. A letter’s complete containing word, or both nonempty complete words beside a space, must belong to it. Clipped or missing required words remain in the denominator as unresolved cases.
Replacing the local state leaves the immediate LOO prediction unchanged, while completion can favor the donor through its new local likelihood (Table 8 ). PD-BC tends to retain this conflicting character, whereas RJD-BC-D can replace it through further resets. Completion lowers final word-check pass rates in both families. Nearly perfect recovery in unchanged and same-character controls implicates conflicting character evidence rather than replacement noise alone. These interventions suggest that Bayesian completion can hinder recovery by reinforcing misleading local evidence.
Table 8. Text8 local-state replacement: percentages, mean ± \pm ± sample SD over three training seeds. Bold marks the higher final word-check pass rate within each path family.
Absorbing jump diffusion
Absorbing jump diffusion (AJD) follows the time-changed Brownian bridge until its first jump, then remains at the sampled embedding. Its rate is 2 t / ( 1 − t 2 ) 2t/(1-t^2) 2 t / ( 1 − t 2 ) while unabsorbed. We compare unrestricted prediction in both drift and jump probabilities (AJD), LOO prediction in both (AJD-LOO), completion only in the drift (AJD-BC-D), and completion in both terms (AJD-BC). The full predictor observes the absorption indicators; q θ , i , t q_{\theta,i,t} q θ , i , t excludes both X i , t X_{i,t} X i , t and its indicator. Completion uses the local likelihood of this absorbing path.
Table 9 compares the four AJD variants at 128 NFE, using 1,024 samples per seed and the first million validation characters as reference. LOO improves all four reported metrics over unrestricted AJD and both completion variants. This is a separate prediction ablation; its D 5 D_5 D 5 and JSD 4 \operatorname{JSD}_4 JSD 4 values should not be compared directly with the 4,096-sample main table.
Table 9.
Table 9: AJD prediction ablation at 128 NFE and 1,024 samples per seed. We report mean ± \pm ± sample standard deviation over three training seeds; D 5 D_5 D 5 counts are rounded to the nearest integer. All paths use κ ( t ) = t 2 \kappa(t) = t^2 κ ( t ) = t 2 , jumps use CAT, and predictions have temperature one.
Jump-time schedules for absorbing jump diffusion
Keeping the continuous schedule κ ( t ) = t 2 \kappa(t) = t^2 κ ( t ) = t 2 fixed, we compare independent first-jump times T i T_i T i with
P ( T i ≤ t ) = t γ , λ γ ( t ) = γ t γ − 1 1 − t γ , γ ∈ { 1 , 2 , 4 } . \mathbb{P}(T_i \le t) = t^\gamma,\qquad\lambda_\gamma(t) = \frac{\gamma t^{\gamma-1}}{1-t^\gamma},\qquad\gamma\in\{1,2,4\}. P ( T i ≤ t ) = t γ , λ γ ( t ) = 1 − t γ γ t γ − 1 , γ ∈ { 1 , 2 , 4 } .
The baseline is γ = 2 \gamma= 2 γ = 2 ; larger γ \gamma γ delays jumps. For absorbing jump diffusion, each model is trained and sampled with the same γ \gamma γ . An unabsorbed coordinate jumps over [ s , t ] [s,t] [ s , t ] with probability ( t γ − s γ ) / ( 1 − s γ ) (t^\gamma-s^\gamma)/(1-s^\gamma) ( t γ − s γ ) / ( 1 − s γ ) . We keep the cosine time grid and network budget fixed, stop at t = .999 t = .999 t = .999 , and apply the chosen token rule to any remaining unabsorbed positions, counting terminal network evaluations.
At 128 NFE, delaying absorption increases validity and diversity: V 5 V_5 V 5 rises from 58.07% to 77.10% between γ = 1 \gamma= 1 γ = 1 and 4. The lowest mean JSD 4 \operatorname{JSD}_4 JSD 4 occurs at γ = 2 \gamma= 2 γ = 2 , close to γ = 4 \gamma= 4 γ = 4 (Table 10 ).
Table 10.
Table 10: AJD-LOO with CAT, 128 NFE, and 1,024 samples per seed (mean ± \pm ± sample standard deviation over three seeds). D 5 D_5 D 5 counts are rounded to the nearest integer. The continuous schedule is fixed at κ ( t ) = t 2 \kappa(t) = t^2 κ ( t ) = t 2 ; the γ = 1 , 4 \gamma= 1, 4 γ = 1 , 4 models are trained with their respective jump clocks.
Table 11 supplements the 128-NFE clock comparison in Table 10 . Increasing γ \gamma γ improves V 5 V_5 V 5 at both additional budgets. At 32 NFE, γ = 4 \gamma= 4 γ = 4 gives a slightly lower mean JSD 4 \operatorname{JSD}_4 JSD 4 than γ = 2 \gamma= 2 γ = 2 ; at 512 NFE, γ = 2 \gamma= 2 γ = 2 gives the lowest mean. The continuous schedule and numerical grid remain fixed. At the common cutoff, the theoretical unabsorbed fractions are 1 − .999 γ 1 - .999^\gamma 1 − .99 9 γ , approximately 0.001, 0.002, and 0.004 for γ = 1 , 2 , 4 \gamma= 1, 2, 4 γ = 1 , 2 , 4 ; terminal decoding therefore affects different fractions.
Table 11: Additional budgets for AJD-LOO with CAT and trained jump CDF t γ t^\gamma t γ . Mean ± \pm ± sample standard deviation over three seeds, with 1,024 samples per seed and fixed continuous schedule κ ( t ) = t 2 \kappa(t) = t^2 κ ( t ) = t 2 .
Table 11.
Jump sampling rules and complete comparisons
Five jump rules reuse each path’s 60,000-update EMA checkpoints. Destination rules are categorical sampling A i ∼ q θ , i , t A_i \sim q_{\theta,i,t} A i ∼ q θ , i , t (CAT), A i = arg max a q θ , i , t ( a ) A_i = \operatorname*{arg\,max}_a q_{\theta,i,t}(a) A i = arg max a q θ , i , t ( a ) (MAX), probabilities proportional to q θ , i , t ( a ) 1 / 0.7 q_{\theta,i,t}(a)^{1/0.7} q θ , i , t ( a ) 1/0.7 (TEMP), and categorical sampling from the smallest set of highest-probability tokens with total mass at least 0.9 (TOPP, or nucleus sampling) (Holtzman et al. [18 ] , Sections 3.1 and 3.3). Confidence selection (CONF) reassigns the positions selected by independent clocks to those with largest max a q θ , i , t ( a ) \max_a q_{\theta,i,t}(a) max a q θ , i , t ( a ) , then samples destinations categorically, adapting Mask-Predict Ghazvininejad et al. [14 ] and MaskGIT Chang et al. [3 ] . AJD-LOO and RJD-LOO retain the temperature-one raw LOO drift for every rule. The modified rules change the generator.
Comparisons use 1,024 length-128 sequences per seed, the same dictionary, the first million validation characters as reference, and the cosine grid. NFE includes EPJ and AJD-LOO terminal prediction calls for unabsorbed positions; RJD-LOO decodes to nearest embeddings at t = .999 t = .999 t = .999 without an extra call. TOPP improves validity and four-gram agreement across the tested paths; the effects of TEMP and CONF depend on the path and budget. Appendices § 13 .8 and § 13 .9 give additional NFE budgets and other samplers with 4,096 samples per seed.
Confidence selection. For EPJ and AJD-LOO, CONF draws the number of jumps from the original independent clocks and assigns them to the most confident unabsorbed positions; committed positions remain fixed. For RJD-LOO, all positions remain eligible on every interval. On the path-time interval [ u , v ] [u, v] [ u , v ] , we first sample the independent reset indicators and last-reset times from Appendix § 10 . Their count satisfies
K ∼ Binomial ( L , v − u 1 − u ) . K \sim\operatorname{Binomial}\left(L, \frac{v-u}{1-u}\right). K ∼ Binomial ( L , 1 − u v − u ) .
Here K K K counts positions with at least one reset, not the total number of Poisson events. CONF chooses the K K K positions with largest max a q θ , i , t ( a ) \max_a q_{\theta,i,t}(a) max a q θ , i , t ( a ) , breaking ties with fresh independent uniforms, and reassigns the sampled last-reset times to them. The multiset of times is preserved, and their assignment does not depend on the time values. The interval update (19) then includes diffusion after each reset. The drift uses the unchanged LOO mean, rather than the last sampled token as a hidden endpoint. This finite-grid rule preserves the sampled count but changes the position-specific reset probabilities; it is not a marginal-preserving modification.
At 128 NFE, CAT and CONF both produce about 4.19 reset-containing intervals per position. The fraction of positions reset in multiple intervals is 96.5% for CAT and 48.9% for CONF; CONF’s selected positions have mean predicted confidence 0.994. Thus CONF concentrates resets on fewer positions. This is consistent with, but does not establish the cause of, the lower word diversity. These diagnostics count intervals with resets, not individual Poisson events within an interval.
Complete sampling results. Tables 12–14 report all five rules, including the effective word count E 5 E_5 E 5 . At 128 NFE, AJD-LOO with TOPP has lower JSD 4 \operatorname{JSD}_4 JSD 4 than RJD-LOO with TOPP, but lower validity and diversity; neither path improves every metric. EPJ and AJD-LOO have all three training seeds at every budget. For the RJD-LOO sampling study, 32 and 128 NFE have seeds 0, 1, and 2; the 512-NFE results use only seeds 0 and 1. The separate CAT path comparison in Table 15 has all three seeds.
Table 15: Additional budgets for the conditional-path comparison. Mean ± \pm ± sample standard deviation over three seeds, with 1,024 samples per seed, temperature one, CAT, and κ ( t ) = t 2 \kappa(t) = t^2 κ ( t ) = t 2 . All checkpoints are evaluated after 60,000 training updates.
Table 15.
At 128 NFE, MAX raises RJD-LOO validity over CAT but reduces E 5 E_5 E 5 by about 25%; CONF reduces it by about 19%. None of the RJD-LOO evaluations contains duplicate full sequences; the reduction concerns word diversity.
For budget comparisons involving 512 NFE, we match seeds 0 and 1 at every budget. RJD-LOO with TOPP has V 5 = 70.27 % , 79.96 % , 82.31 % V_5 = 70.27\%, 79.96\%, 82.31\% V 5 = 70.27% , 79.96% , 82.31% and JSD 4 = 0.0874 , 0.0771 , 0.0750 \operatorname{JSD}_4 = 0.0874, 0.0771, 0.0750 JSD 4 = 0.0874 , 0.0771 , 0.0750 at 32, 128, and 512 NFE. At 512 NFE, PD-LOO on these same two seeds has V 5 = 85.37 % V_5 = 85.37\% V 5 = 85.37% , D 5 = 2904 D_5 = 2904 D 5 = 2904 , E 5 = 1432 E_5 = 1432 E 5 = 1432 , and JSD 4 = 0.0811 \operatorname{JSD}_4 = 0.0811 JSD 4 = 0.0811 , compared with D 5 = 3210 D_5 = 3210 D 5 = 3210 and E 5 = 1654 E_5 = 1654 E 5 = 1654 for RJD-LOO with TOPP. The tradeoff remains; these two-seed estimates should not be compared directly with the three-seed means as if the seed sets were identical.
Table 12. EPJ sampling at 32, 128, and 512 NFE with 1,024 sequences per seed. Mean ± sample standard deviation; count metrics are rounded to integers. Best means within each budget are bold. All budgets use three training seeds.
Table 13. AJD-LOO sampling at 32, 128, and 512 NFE with 1,024 sequences per seed. Mean ± sample standard deviation; count metrics are rounded to integers. Best means within each budget are bold. All budgets use three training seeds.
Additional network budgets
Table 15 gives the complete comparison at 512 NFE and a partial comparison at 2,048 NFE. DFM, AJD-BC-D, and RJD-LOO were not evaluated at 2,048 NFE. There is no complete comparison at 32 NFE: PD, PD-BC, AJD, AJD-BC, AJD-BC-D, RJD, and RJD-BC-D have no such evaluation in this set; the separate RJD-LOO sampling results are in Table 14 . All values below use 1,024 samples per seed. At 512 NFE, RJD-LOO has D 5 = 3193 ± 17 D_5 = 3193 \pm17 D 5 = 3193 ± 17 , compared with 2939 ± 23 for AJD-LOO and 3209 ± 23 for RJD-BC-D. Thus the validity advantage of RJD-LOO over RJD-BC-D does not imply higher word diversity: RJD-BC-D has slightly higher D 5 D_5 D 5 and higher E 5 E_5 E 5 at this budget (1911 ± 22 versus 1686 ± 6).
Table 14. RJD-LOO sampling at 32, 128, and 512 NFE with 1,024 sequences per seed. Mean ± sample standard deviation; count metrics are rounded to integers. Best means within each budget are bold. The 32- and 128-NFE results use three seeds; 512 NFE uses only seeds 0 and 1. The nonzero CAT standard deviation for JSD4 _4 4 at 512 NFE is shown in scientific notation.
At 128 NFE, EPJ with MAX gives 96.87% word validity but only about 11 effective valid-word types, compared with about 1,098 for CAT. Here the effective count is exp ( H ) \exp(H) exp ( H ) , where H H H is the empirical entropy of valid words of length at least five; it is calculated per seed before averaging. This concentration explains why a high V 5 V_5 V 5 need not imply a close match to the reference text distribution.
Additional samplers with 4,096 samples per seed
This separate evaluation covers five sampler settings, the same three training seeds, and budgets of 128 and 512 NFE. All checkpoints are the 60,000-update EMA checkpoints. We generate 4,096 sequences per seed with generation seeds 51000 + s 51000 + s 51000 + s , where s ∈ { 0 , 1 , 2 } s \in\{0, 1, 2\} s ∈ { 0 , 1 , 2 } is the training seed. The dictionary and validation reference [3,000,000 : 4,000,000] match the main comparison, but the grids, jump rules, and correction steps below differ. These are complete-sampler comparisons using existing checkpoints, not additional training seeds.
PD-LOO uses a grid uniform in κ ( t ) = t 2 \kappa(t) = t^2 κ ( t ) = t 2 ; the other settings use the cosine grid. All AJD variants retain raw LOO drift. The final setting adds 16 LOO updates at times in [ 2 / 3 , 1 ] [2/3, 1] [ 2/3 , 1 ] : each update chooses one position per sequence, draws its token from q θ , i , t q_{\theta,i,t} q θ , i , t , and redraws its complete conditional state, including its absorption indicator. These are 16 single-position updates, not 16 sweeps over the sequence. At total budget B B B , this setting uses B − 17 B - 17 B − 17 predictor intervals, 16 correction calls, and one terminal call. The selected settings are fixed before this confirmation; they compare complete samplers rather than isolating one change to the conditional path.
Table 16: Additional samplers with 4,096 generated sequences per seed. Mean ± \pm ± sample standard deviation over three training seeds; the dictionary and 4-gram reference match the main comparison. All correction and terminal calls count toward NFE.
Table 16.
Generated Text8 samples
The panels below show the first four generated sequences from training seed 0 for each model in Table 7 , using the 60,000-update EMA checkpoints and 128 NFE. They come from the first batch, with generation seed 625000, of the main 4,096-sample evaluations. The real-text panel shows the first four passages from the first calibration block. Every sequence has 128 characters; spelling and spacing are retained, with line wrapping only for display. No sample is selected by quality. Text8 chunks may begin or end inside a word and need not contain complete sentences.
RJD
e dumb and viner moders in londom ishue eight seven true tradicated in rather from the condect one nine isles that citigating an
sex mperet copics miles the hzrio seven with e glor out and afforms what pertihally obstylor dissogovernation has game millions
ore s for scends are and agoin profound whone in the work irs and it christ of alk ind with the pus assi son the moduls awice sho
s soon an ive a rind th for diale orseural our eight freed basc time samier unite forcrection was indual numes two zerological
RJD-LOO
lm an increased at never until one nine zero zero s missed into the from the germans seven the size with their but conditions in
ne nine from the similar mey lije finding in treaty are federatorial from master others leading system and and so two spirit in
c wicked as they three poland beyond lanes to with city award and stand its black ards s hulm premiere of kelrey could arrive wo
lt at the bbc s case of diamond one four three singer had a links situationes the bhm trishment both saturated in twins in the t
RJD-BC-D
dm and the six nine microbably the much five zero s protestants the from the one one depart on mars with their dint eduction sym
ne five root the similar taxation finding of traits are need italian crime mo orname ballistine system appsara a yff cld nations
nglabour is that finally have a sack antident with no times terrain him one which own a coast julians cpu of continues refer com
t wae the end a core or ted airlinear used to exclaim nsiek manuse igration scales greenish will tried segwod war complete the r
DFM
the geoglopes have cpublic grand children cummon from any alternet tee also goes a mitchem abablical hugaleth three of covement
lly even products while is olderchears and dandry minstur five last invaserabolations und novel tamp artial and theory prejertiv
ked place the icon unsoruence a like sotrague are excell museumus very proper i five zero fivr kx three eighteen agree mcass as
gly class for no fish race humans batant tu merolsec french chances of inderstand s s movonecy but estment of the acid and as e
EPJ
ch proootypes whom the diccs and children cummon lock and alternet the also goes a discreas biblical questleth three of constant
lly john products would be kide cietre and landon the fur five last invoker word it sound constant pourtheI in its of rkjers a
ked electrolysis of unbrousics a like sourage earloes allo usedmus very proper s live articuiar exported either aka men carious
g include for no fish race humans balant toe e pollic speech chances of inderstand s complecacy bettes term of the acid and acle
PD
ellow willing printers used an a host qz o rge work wood the suveriations in use at mouke the devells web or the taffairable pun
ines technical uperators as a justicially electrbst delphan s also litylcha tustman pathinal his most zero six somes posite diag
n to athmic as long a can carediamed in waftgaivil that all ardent of whichment of completitiea buduals targe base it beespresen
rm nately of brone for a i to st author of bell or or terrable or a masteri s anotherain of two time construttle golgia of aomon
PD-LOO
ther way if and is required and his three in the much lead the unable two zero with fresco known explants botany where bible by
in combinatings and known incomes with more volume turkiest south the faced back and from a pictures functive in a zero six foll
family concerned with led the handpoint and of break train later and german is in nine door is violence west for three the part
selling roman holding deputies the threats the root of british unificialized in the official companion islanding man and four f
PD-BC
ry arvasion god be nasourew as whichborre with robbonder external planing were heingening of sean explated but onothers libdiness
ided mmigh mary and brown in one white clocter over rarring years howevered contrast s am ibrans intemulceptic to where firstwir
eft day converses logist diated a insult and required for foltwerna decamillies on the amate native abcoaster togethrew the land
mportational civil its families one three e well of hopes uzes it inv iding to in the old that cormits is standard but ael founli
RJD-LOO, TOPP
cd and spreaded an generatical one nine zero zero somerged into the from the germans seven the life with their all conditions in
ne nine from the similar may life finding in treaty are federatorial from master others leading system and and so two shend cond
d passed as they three poland beyond lined to with city award and stand its black ards a hulm premiere on the us could argued wo
temit the abdul work os studies one four three loved from a livks in party by these largestment both saturated in twins in the t
Real Text8
n nine seven five ad also malay sailors knew of mauritius as early as the one zero th century portuguese sailors one five zero s
even one five one three mauritius was discovered and visited by the portuguese between one five zero seven and one five one thre
e an official world map by diego ribeiro described from west to east the first island mascarenhas the second santa apolonia and
the third domingo friz one the three islands r union mauritius and Rodrigues were discovered some years earlier by chance during
Additional Text8 controls
Pure-jump control with ordinary self-attention
We additionally evaluate a pure-jump predictor with ordinary self-attention, using the matched-layer DDiT control. Queries, keys, and values are recomputed from the evolving hidden states at every layer, and attention includes the diagonal. This differs from both the LOO and unmasked fixed-source predictors in the main comparison. The remaining layers match our backbone: eight blocks of width 256, eight heads of dimension 32, SwiGLU width 704, time conditioning, rotary positional embeddings, normalization, dropout, and learned 64-dimensional token embeddings. The control has 6,878,624 trainable parameters, versus 6,878,880 for structural LOO; the difference is the latter’s learned 256-dimensional initial query.
The run uses the same embedded pure-jump path as EPJ: coordinates start at zero, jump with CDF t 2 t^2 t 2 , and remain at their sampled token embeddings. It is trained from scratch on an A100 for 60,000 updates with seed 0, length-128 sequences, batch size 512, uniform training times on [ 0 , .999 ] [0, .999] [ 0 , .999 ] , and ordinary cross-entropy over all positions. AdamW, learning rate 10 − 3 10^{-3} 1 0 − 3 , FP32/math attention, and EMA decay 0.9999 match the main protocol. No likelihood completion or LOO loss correction is used. The mean training loss over the final 1,000 updates is 1.5089; final EMA validation CE is 1.4885 over all positions, 1.8254 for the per-sequence average over unabsorbed positions, and 5.20 × 10 − 5 5.20 \times10^{-5} 5.20 × 1 0 − 5 over absorbed positions. These are denoising losses, not language-model likelihoods or generation-quality scores.
Table 17 compares this control with the EPJ structural-LOO checkpoint for the same training seed. Each evaluation uses the 60,000-update EMA, sampling seed 0, 1,024 sequences, the cosine grid, and categorical destinations at temperature one or 0.7. NFE includes terminal classification of any remaining unabsorbed positions. The validation-reference, lexicon, and character-map hashes agree across all twelve evaluations. These evaluations share two trained checkpoints; there is no training-seed standard deviation.
Ordinary self-attention gives higher V 5 V_5 V 5 and V 6 V_6 V 6 and lower J S D 4 \mathrm{JSD}_4 JSD 4 in all six matched sampling settings, while the word-diversity comparisons are mixed. At 128 NFE and temperature one, V 5 V_5 V 5 increases from 44.35% to 54.08% and J S D 4 \mathrm{JSD}_4 JSD 4 decreases from 0.08215 to 0.07284; both D 5 D_5 D 5 and E 5 E_5 E 5 also increase. Lowering the self-attention predictor’s sampling temperature to 0.7 raises V 5 V_5 V 5 to 73.41%, but reduces diversity and increases divergence. At temperature one, increasing the budget from 512 to 2,048 NFE gives little additional validity and reduces D 5 D_5 D 5 and E 5 E_5 E 5 . No evaluation contains duplicate full sequences.
This provides a stronger pure-jump reference than the structural-LOO EPJ predictor in this seed. At an unabsorbed pure-jump position, the local zero state carries no endpoint information, so the local denoising shortcut that motivates LOO for continuous paths is absent. Structural LOO therefore need not improve this baseline. The result is specific to this single-seed comparison and does not establish an ordering across seeds or continuous paths.
CADD-derived corruption
We adapt the masked categorical supervision and Gaussian interpolation of CADD [44 Section 4.3 and Appendix B] , with detached hints and 10% whole-hint dropout from its released training guide. Our architecture and normalized embeddings are adaptations; the control is not a reproduction of the original CADD backbone. Using clean-progress time t t t (the paper’s noise time is 1 − t 1-t 1 − t ), draw t ∼ Unif [ 0 , .999 ) t \sim\operatorname{Unif}[0, .999) t ∼ Unif [ 0 , .999 ) per sequence and independent R i , t ∼ Bernoulli ( t ) R_{i,t} \sim\operatorname{Bernoulli}(t) R i , t ∼ Bernoulli ( t ) , where R i , t = 1 R_{i,t}=1 R i , t = 1 means resolved. For independent ϵ i ∼ N ( 0 , I 64 ) \epsilon_i \sim\mathcal{N}(0, I_{64}) ϵ i ∼ N ( 0 , I 64 ) , set
X i , t = R i , t e ( Y i ) + ( 1 − R i , t ) sg [ H ( t e ( Y i ) + ( 1 − t ) ϵ i ) ] , H ∼ Bernoulli ( 0.9 ) . X_{i,t}=R_{i,t}e(Y_i)+(1-R_{i,t})\operatorname{sg}\left[H\left(te(Y_i)+(1-t)\epsilon_i\right)\right], \qquad H\sim\operatorname{Bernoulli}(0.9). X i , t = R i , t e ( Y i ) + ( 1 − R i , t ) sg [ H ( t e ( Y i ) + ( 1 − t ) ϵ i ) ] , H ∼ Bernoulli ( 0.9 ) .
Here sg \operatorname{sg} sg stops gradients and one H H H is shared by the entire minibatch. Resolved inputs still supply embedding gradients. Before hint dropout, the unresolved conditional covariance is ( 1 − t ) 2 I 64 (1-t)^2I_{64} ( 1 − t ) 2 I 64 , with a Gaussian initial state.
CADD-derived training with one shared predictor
With CADD-derived training [44 ] , one predictor supplies continuous and discrete updates. The paired seed-0 comparison changes only the attention mask. With posterior-mean (soft) endpoints, LOO improves validity, distinct valid-word counts, and four-gram agreement; with argmax (hard) endpoints, it improves validity but worsens divergence (Table 18 , 128 NFE with CAT; other budgets below).
Table 18: CADD-derived shared-predictor decoding at 128 NFE: 60,000-update EMA, training and sampling seed 0, and 1,024 sequences per cell. Hard and soft specify the continuous endpoint; the discrete rule is varied independently. Bold marks the better value in each matched control/LOO pair. No training-seed standard deviation is available.
Table 18.
This comparison uses a matched pair trained for 60,000 updates with seed 0. One transformer stream and one categorical head supply the predictions for both continuous and discrete updates. Control and LOO share initialization, data order, parameter count, corruption, optimizer, and training budget; only the attention diagonal differs. Both use fixed position-local sources for keys and values at every layer and queries initialized independently of the inputs. LOO excludes the position’s own input and status from both predictions, including indirect routes within one network call. Resolved positions receive no loss and remain fixed during sampling; their embeddings remain visible to other positions.
We use the 90/5/5-million-character Text8 split, ∣ V ∣ = 27 |\mathcal{V}|=27 ∣ V ∣ = 27 , L = 128 L=128 L = 128 , and normalized embeddings in R 64 \mathbb{R}^{64} R 64 with norm 8. The network has eight blocks of width 256, eight attention heads, SwiGLU width 704, rotary positions, AdaLN-Zero conditioning, and feature/attention dropout 0.1. There are 6,878,880 trainable parameters including the embeddings. Training uses batch size 512, AdamW with constant learning rate 10 − 3 10^{-3} 1 0 − 3 , coefficients ( 0.9 , 0.999 ) (0.9, 0.999) ( 0.9 , 0.999 ) , numerical constant 10 − 8 10^{-8} 1 0 − 8 , no warmup or weight decay, gradient clipping at one, and EMA decay 0.9999 for network and embeddings. The corruption is the CADD-derived Gaussian interpolation, linear masking, detached hints, and 10% minibatch hint dropout described in Appendix H.2. The loss is a single categorical cross-entropy term at unresolved positions, divided by B L BL B L for a batch of B B B sequences, using the one shared prediction. Our fixed-source architecture and normalized embeddings are adaptations of the CADD recipe [44 ] ; this control does not reproduce the published model.
Continuous estimates and discrete rules. Sampling starts with independent Gaussian hints and no resolved positions. On a uniform grid from zero to .999, one call at time t t t gives logits z i , t z_{i,t} z i , t for both updates. An unresolved position commits over [ t , r ] [t,r] [ t , r ] with probability ( r − t ) / ( 1 − t ) (r-t)/(1-t) ( r − t ) / ( 1 − t ) , except that the final interval commits all survivors. Uncommitted positions receive a fresh hint with independent ϵ i ′ ∼ N ( 0 , I 64 ) \epsilon'_i \sim\mathcal{N}(0,I_{64}) ϵ i ′ ∼ N ( 0 , I 64 ) ,
X i , r = r m i , t + ( 1 − r ) ϵ i ′ , m i , t = { e ( arg max a z i , t ( a ) ) , hard , ∑ a softmax ( z i , t ) a e ( a ) , soft . X_{i,r}=r m_{i,t}+(1-r)\epsilon'_i,\qquad
m_{i,t}=
\begin{cases}
e(\operatorname*{arg\,max}_{a} z_{i,t}(a)), & \text{hard},\\
\sum_a \operatorname{softmax}(z_{i,t})_a e(a), & \text{soft}.
\end{cases} X i , r = r m i , t + ( 1 − r ) ϵ i ′ , m i , t = { e ( arg max a z i , t ( a )) , ∑ a softmax ( z i , t ) a e ( a ) , hard , soft .
These are fresh-noise updates rather than Brownian-bridge increments. The endpoint choice is independent of the discrete rule: CAT uses raw probabilities, TEMP uses temperature 0.7, TOPP uses mass 0.9, and MAX uses argmax only for the committed token. In particular, TEMP and TOPP never alter the soft continuous mean. CONF preserves each sequence’s independently drawn commitment count and selects that many unresolved positions with largest raw predicted confidence, with exchangeable random tie-breaking; destinations remain categorical. Positions already resolved stay at their committed embeddings. These sampling changes, and the LOO continuous prediction without Bayesian completion, carry no guarantee of preserving the prescribed marginals.
One NFE counts one transformer call, including the last commitment call; there is no additional terminal call. All 60 combinations of two predictors, two continuous estimates, five discrete rules, and 32/128/512 NFE are complete for training seed 0. Each generates 1,024 length-128 sequences with sampling seed 0, batch size 64, EMA network and embeddings, FP32 and math attention on a V100. The dictionary, validation reference, and metric implementation match Appendix § 13 .§ 2 . Different decoding cells reuse checkpoints and do not supply independent training seeds. CAT at 128 NFE is the primary comparison; the sweep is reported in full below, rather than selecting the best observed setting.
Results. LOO improves V 5 V_5 V 5 in all 30 matched comparisons, V 6 V_6 V 6 in 29, D 5 D_5 D 5 in 21, E 5 E_5 E 5 in eight, and J S D 4 \mathrm{JSD}_4 JSD 4 in nine. At 128 NFE with CAT, soft LOO improves all five metrics over its control. Hard LOO increases V 5 V_5 V 5 and D 5 D_5 D 5 but reduces E 5 E_5 E 5 from 417.2 to 379.5 and worsens J S D 4 \mathrm{JSD}_4 JSD 4 from 0.10837 to 0.12347. For LOO, soft endpoints have larger E 5 E_5 E 5 and lower divergence than hard endpoints for every tested rule and budget. TEMP and TOPP increase soft LOO validity over CAT at 128 NFE, but reduce its diversity and worsen its divergence. Increasing NFE from 128 to 512 raises V 5 V_5 V 5 , reduces E 5 E_5 E 5 , and worsens divergence for all ten LOO settings. For soft LOO with CAT, V 5 V_5 V 5 increases from 67.55% to 71.27%, while E 5 E_5 E 5 falls from 761.8 to 542.3 and divergence rises from 0.08610 to 0.09606. These are paired observations for one training seed.
Table 19: CADD-derived shared-predictor decoding at 32 NFE: 60,000-update EMA, training and sampling seed 0, and 1,024 sequences per cell. Hard and soft specify the continuous endpoint; the discrete rule is varied independently. Bold marks the better value in each matched control/LOO pair. No training-seed standard deviation is available.
Table 19.
Table 20: CADD-derived shared-predictor decoding at 512 NFE: 60,000-update EMA, training and sampling seed 0, and 1,024 sequences per cell. Hard and soft specify the continuous endpoint; the discrete rule is varied independently. Bold marks the better value in each matched control/LOO pair. No training-seed standard deviation is available.
Table 20.
Word concentration. MAX and CONF concentrate the word distribution. In every matched setting, CONF has lower E 5 E_5 E 5 and higher divergence than CAT. Table 21 reports the share of valid word occurrences accounted for by the three and ten most frequent words. The three most frequent are three, seven, and eight, in varying order. With hard LOO and MAX, they account for 86.92% of valid word occurrences of length at least five, despite V 5 = 97.40 % V_5 = 97.40\% V 5 = 97.40% . These are lexical concentration measurements, not fractions of identical generated sequences.
Table 21: Word concentration for shared-predictor LOO at 128 NFE. Percentages use dictionary-valid occurrences of words of at least five letters as the denominator. The checkpoints and samples are those in Table 18 .
Table 21.
LM1B experimental details and additional results
Data, architecture, and training
Our LM1B experiments [4 ] use the BERT uncased tokenizer [9 ] , a 30,522-token WordPiece vocabulary [41 ] , and length-128 windows, matching the tokenizer and window size of Chemseddine et al. [5 ] . We compare RJD-LOO, unrestricted RJD, embedded pure jumps (EPJ), and pure-diffusion and completion variants. RJD and RJD-LOO use fixed-source attention and differ in access to the current position’s noisy input. Raw RJD-LOO predictions drive both drift and resets; EPJ uses unrestricted prediction and absorbing jumps without continuous motion. RJD, RJD-LOO, and EPJ all use κ ( t ) = t 2 \kappa(t) = t^{2} κ ( t ) = t 2 and start from zero; the two RJD variants use diffusion scale σ d = 1 \sigma_{d} = 1 σ d = 1 .
We use a DDiT-style backbone [25 ] , adapted to continuous token embeddings and fixed-source attention. RJD-LOO, RJD, and EPJ have 12 layers, width and embedding dimension 768, 12 attention heads, and learned embeddings of norm 14.18. All LM1B predictors have approximately 116M trainable parameters, including the learned token embeddings. As in Chemseddine et al. [5 ] , the output head shares the learned token embeddings with the conditional path. Given the network’s output vector f i f_i f i , token probabilities are obtained by applying softmax over logits e ( a ) ⊤ f i + b a e(a)^{\top}f_i+b_a e ( a ) ⊤ f i + b a , where b a b_a b a is a learned token bias. All LM1B models use batch size 512 and AdamW with peak learning rate 3 × 10 − 4 3 \times10^{-4} 3 × 1 0 − 4 , 2,500 linear warmup updates, and cosine decay to 3 × 10 − 5 3 \times10^{-5} 3 × 1 0 − 5 at 1M updates. EMA decay is 0.9999. We use one training seed per model (seed 0) and evaluate 750,000-update EMA weights.
Self-attention control (RJD-SA). RJD-SA computes queries, keys, and values from evolving hidden states, initialized from the corrupted input, with all positions visible. It matches the fixed-source models’ depth, width, heads, time conditioning, and tied embedding/readout. It uses the same RJD path, all-position cross-entropy loss, and raw unrestricted predictions for drift and resets. Training starts from random initialization and follows the shared protocol above. Training uses NVIDIA A100 and H100 GPUs and AMD MI300A accelerators; evaluation uses NVIDIA A100 GPUs with the common protocol. The single training run and mixed training hardware prevent isolating architecture from training variability.
Evaluation metrics
Generative perplexity (PPL) exponentiates the mean negative log-probability assigned to GPT-2-tokenized samples by external GPT-2-large [33 ] . Lower means more predictable to this scorer. Reported PPL is the arithmetic mean of per-generation-seed perplexities. Token entropy is H = − ∑ a f ( a ) log f ( a ) H = -\sum_{a} f(a) \log f(a) H = − ∑ a f ( a ) log f ( a ) [35 ] , where f ( a ) f(a) f ( a ) is the relative frequency of BERT token a a a within a 128-token window, including special tokens. We average over windows in nats. Higher values indicate less concentrated token frequencies. MAUVE [29 ] compares generated and reference distributions in a quantized embedding space, capturing both generated text unlike the reference and missing reference content. Scores range from zero to one; higher is closer to the reference. We use GPT-2-large features with a maximum input length of 256 tokens, 512 clusters, and 5,120 fixed held-out windows (reference seed 20261029). Each corpus score averages results from clustering seeds 25, 26, and 27. Reported SEDs describe variation across generation seeds, not training runs. Low PPL can accompany repetition [18 ] .
Figure 2 uses 750,000-update EMA weights, one training seed (0), and 128 predictor calls on a uniform physical-time grid. RJD and PD end at t = 0.999 t = 0.999 t = 0.999 and use nearest-embedding readout; EPJ ends at t = 1 t = 1 t = 1 with categorical output. We sweep p ∈ { 1 , 0.9 , 0.8 , 0.7 } p \in\{1, 0.9, 0.8, 0.7\} p ∈ { 1 , 0.9 , 0.8 , 0.7 } . Completion changes the drift. Top-p p p filters jump destinations for RJD/EPJ, leaving RJD drift probabilities unfiltered; for PD it filters drift probabilities after any completion.
All eight methods cover every p p p , with generation seeds 42, 43, and 44 and 5,120 length-128 windows per seed, giving 32 settings and 96 corpora.
Candidate-restricted Bayesian completion. RJD-BC-D and PD-BC approximate (10) to reduce the cost of likelihood evaluation over the 30,522-token vocabulary. Raw and completed variants share the same trained checkpoint. Let Top K ( f ) \operatorname{Top}_{K}(f) Top K ( f ) denote the tokens with the K K K largest values of f f f . For 0 < t < 1 0 < t < 1 0 < t < 1 , define
C i , t K ( x ) = Top K ( q θ , i , t ( ⋅ ∣ x − i ) ) ∪ Top K ( ℓ i , t ( x i ∣ ⋅ ) ) . C_{i,t}^{K}(x) = \operatorname{Top}_{K}\left(q_{\theta,i,t}(\cdot\mid x_{-i})\right) \cup\operatorname{Top}_{K}\left(\ell_{i,t}(x_i \mid\cdot)\right). C i , t K ( x ) = Top K ( q θ , i , t ( ⋅ ∣ x − i ) ) ∪ Top K ( ℓ i , t ( x i ∣ ⋅ ) ) .
We use K = 64 K = 64 K = 64 for each ranking and count duplicate tokens once, so ∣ C i , t K ( x ) ∣ ≤ 2 K = 128 \lvert C_{i,t}^{K}(x)\rvert\leq2K = 128 ∣ C i , t K ( x )∣ ≤ 2 K = 128 . The completed prediction before nucleus filtering is
p ^ θ , i , t K ( a ∣ x ) = 1 { a ∈ C i , t K ( x ) } q θ , i , t ( a ∣ x − i ) ℓ i , t ( x i ∣ a ) ∑ b ∈ C i , t K ( x ) q θ , i , t ( b ∣ x − i ) ℓ i , t ( x i ∣ b ) . (20) \widehat{p}_{\theta,i,t}^{K}(a \mid x) = \frac{\mathbb{1}_{\{a\in C_{i,t}^{K}(x)\}}q_{\theta,i,t}(a \mid x_{-i})\ell_{i,t}(x_i \mid a)}{\sum_{b\in C_{i,t}^{K}(x)}q_{\theta,i,t}(b \mid x_{-i})\ell_{i,t}(x_i \mid b)}. \tag*{(20)} p θ , i , t K ( a ∣ x ) = ∑ b ∈ C i , t K ( x ) q θ , i , t ( b ∣ x − i ) ℓ i , t ( x i ∣ b ) 1 { a ∈ C i , t K ( x )} q θ , i , t ( a ∣ x − i ) ℓ i , t ( x i ∣ a ) . (20)
Here ℓ i , t \ell_{i,t} ℓ i , t is the likelihood of the corresponding training path: RJD uses (13) , and PD uses the Gaussian bridge likelihood. With equal-norm embeddings, the local ranking can be computed from x i T e ( a ) x_i^{\mathsf{T}}e(a) x i T e ( a ) . The drift uses the embedding mean of (20) , with subsequent top-p p p filtering for PD. At t = 0 t = 0 t = 0 , the local likelihood is token-independent and we use the full-vocabulary LOO distribution before any nucleus filtering.
Results across the sweep. Raw and completed LOO improve PPL and MAUVE over unrestricted fixed-source prediction throughout the sweep, while completion lowers MAUVE relative to raw LOO (Table 22). Top-p p p filtering improves RJD-LOO; stronger filtering can further lower PD-LOO’s PPL while reducing its MAUVE. RJD-SA improves on RJD, but its MAUVE changes across the sweep are small relative to seed variation. The settings in Table 3 are selected from this sweep.
Table 22. MAUVE across the uniform-time top-p p p sweep at 750k updates and 128 NFE. Entries are mean ± \pm ± sample SD across three generation seeds, with 5,120 windows per seed. Higher is better and bold marks the highest mean in each column.
We compare 32, 64, and 128 predictor calls using the same 750k EMA checkpoints and uniform-time sampling and scoring protocol of Appendix § 15 .§ 3 . For each recipe, top-p p p is selected by the highest observed mean MAUVE at 128 NFE and held fixed across budgets: p = 0.8 p = 0.8 p = 0.8 for RJD-LOO, RJD-BC-D, and PD-LOO, and p = 0.9 p = 0.9 p = 0.9 for EPJ. RJD-BC-D uses the candidate-restricted completion defined in Appendix § 15 .§ 3 . Table 23 reports the resulting budget comparison.
Table 23. LM1B generation on uniform physical-time grids at 750k updates. Entries are mean ± \pm ± sample SD over three generation seeds (42, 43, 44), with 5,120 length-128 windows per seed and one training seed per model. Bold marks the lowest mean PPL and highest mean MAUVE at each NFE.
More calls improve PPL and MAUVE for every recipe. RJD-LOO and EPJ have similar MAUVE at lower budgets, without a clear ranking across generation seeds. Completion lowers both PPL and MAUVE relative to raw RJD-LOO; PD-LOO has the lowest PPL but substantially lower MAUVE.
Recovery after inserting a wrong local state
To probe a potential shortcut through a token’s own noisy input, we test whether models can recover from misleading local evidence using the surrounding context.
RJD-LOO, RJD, and RJD-SA use 750k EMA checkpoints, one training seed each, and the same 1,024 held-out text pairs, focal sites, donor tokens, and two noise draws per pair. Let y i y_i y i be the original token at position i i i and d ≠ y i d \ne y_i d = y i a donor token. After drawing a noisy text at time t t t , replace its local state by
x i ′ = x i + c ( e ( d ) − e ( y i ) ) , c = { t 2 , before any reset , 1 , after a reset . x_i' = x_i + c(e(d) - e(y_i)), \qquad c =
\begin{cases}
t^2, & \text{before any reset},\\
1, & \text{after a reset}.
\end{cases} x i ′ = x i + c ( e ( d ) − e ( y i )) , c = { t 2 , 1 , before any reset , after a reset .
This preserves the Gaussian residual and reset times while substituting the donor token. All 128 positions then evolve freely to 0.999 with 128 further calls on a uniform grid over the remaining interval. Top-p p p filters reset destinations with p = 0.8 p = 0.8 p = 0.8 for RJD-LOO and p = 0.7 p = 0.7 p = 0.7 for RJD and RJD-SA; drift probabilities remain raw. Table 24 records the final focal token.
Table 24. Recovery after donor replacement at 750k updates, using 128 further calls and the model-specific top-p p p settings stated above. Percentages use all 2,048 continuations per row.
LOO more often recovers the original token after replacement; unrestricted models retain the donor at later times.
LM1B subject–verb interventions
To test whether the models capture longer-range grammatical dependencies, we examine whether verb predictions track the subject across intervening words and distracting nouns.
We change only the subject’s number and measure immediate verb-form preferences of the 750k EMA RJD-LOO and RJD checkpoints.
Paired subject intervention. In “The key near the cabinets is still here”, changing only “key” to “keys” should change the preferred verb to “are”, despite the intervening noun. We use 800 templates embedded in length-128 windows with held-out background text: 32 controls without an intervening noun and 768 templates with one or two intervening nouns. Each template has two noise draws at t ∈ { 0.25 , 0.40 , 0.55 , 0.70 } t \in\{0.25, 0.40, 0.55, 0.70\} t ∈ { 0.25 , 0.40 , 0.55 , 0.70 } . The cohort uses eight subject-noun pairs in 16 subject–noun lexical blocks and eight control blocks. Blocks requiring multi-WordPiece nouns are excluded beforehand. Templates and corruption draws are shared across models and subject variants.
The local verb state follows the conditional corruption law for the form agreeing with the original subject. Changing only the subject’s number preserves its Gaussian residual and reset times and fixes every other state, so local verb evidence conflicts with the new subject. We compare “is” with “are”, or “was” with “were”. Success requires strictly preferring the agreeing form in both variants; ties, unchanged preferences, and changes in the wrong direction fail.
Controls. Controls use an initially wrong local verb with the same noise and reset history, or an artificial candidate-independent zero-mean Gaussian state. Each stays fixed across the subject change; “correct” and “wrong” refer to the original subject and reverse after it changes. Table 25 distinguishes original-subject accuracy from paired success. LOO’s invariance to local-state controls is architectural while following the subject is learned.
Table 25.
Table 25 : Immediate agreement preferences at 750k updates, as percentages of 1,600 cases per time. The first three rows concern the original subject and give ties half credit. The last requires correct preferences in both subject variants with the initially correct local verb held fixed; ties fail. LOO denotes RJD-LOO. Bold marks the higher accuracy within each condition and time.
Distance and intervening nouns. Figure 3 stratifies paired success by WordPiece position difference at early and later times, with initially correct local verbs. Each distance has 128 templates and two draws, except the 32 distance-one controls. Intervals use 1,000 bootstrap resamples of lexical clusters, keeping noise draws and subject variants together: 16 clusters per nontrivial distance and eight for controls. They condition on one checkpoint per model and lack multiplicity correction.
LOO’s success generally decreases with distance, while unrestricted RJD rarely follows both subject variants. Table 26 separates intervening-noun count and number; matching versus opposite number has no uniform ordering. Wording and noun configurations also vary across distances.
Table 26.
Table 26 : RJD-LOO paired subject-tracking success (%) by intervening-noun configuration at all four times. M/O means the intervening noun(s) match or oppose the original subject’s number; that relation reverses when the subject changes. Each M/O cell uses 64 templates with two draws. For unrestricted RJD, all corresponding rates are zero except 0.78% for one opposite-number noun at distance six and t = 0.25 t = 0.25 t = 0.25 .
Worked example for Table 26 . In “The key near the cabinets is still here”, “cabinets” is the single intervening noun. The sequence “key / near / the / cabinets / is” spans four position steps, so this the first row. Singular “key” and plural “cabinets” place it in O. Replacing “cabinets” with “cabinet” gives an M example.
At t = 0.55 t = 0.55 t = 0.55 , the first noisy input contains local evidence for “is”. The second changes only “key” to “keys”, fixing all other states. Success requires preferring “is” first and “are” second. Each cell aggregates 64 templates with two noise draws. M/O refers to the original subject and reverses after its number changes.
Blinded LLM assessment and context diagnostic
We assess whether generated LM1B sentences are understandable and internally consistent, and whether earlier context helps predict later tokens.
Samples and blinding. We sampled 300 128-token texts per method (RJD-LOO, RJD, RJD-SA, and EPJ): 100 uniformly without replacement from each generation seed 42, 43, and 44. We also include 100 held-out real texts with the same decoding. Generated samples use the 750k, 128-NFE, p = 1 p = 1 p = 1 setting of Appendix I.3, with one training seed per model. LLM judges A (gpt-6.1-sol) and B (gpt-6.1-terra) used high reasoning effort in separate fresh Codex sessions. They received anonymous texts in different orders with the same rubric and returned scores, reasons, and verbatim evidence. Ratings were sealed before joining source labels.
Rubric and primary outcome. Judges scored clarity, internal consistency, grammar, and disruptive repetition from zero to four; higher is better except for repetition. Illustratively, “The chef bake the bread” conveys a clear meaning despite a grammatical error; “The box is empty and contains three apples” is grammatical but internally inconsistent. Acceptance requires clarity and consistency of at least three: mostly understandable content with mostly consistent relationships. Texts with no interpretable proposition are excluded from consistency means but remain in the acceptance denominator. Cosmetic formatting and clipped outer clauses are ignored; texts are rated as written.
We report each judge separately. For each of 2,000 bootstrap replicates, we resample texts with replacement within generation seeds and recompute the mean score or acceptance rate. The 2.5th and 97.5th percentiles give the 95% confidence interval. Intervals condition on the fixed judges and one training run per model; there is no human validation.
Generation-quality results. Both judges favor RJD-LOO on acceptance, clarity, and grammar, while all generators remain below real text (Table 27 ). RJD’s high consistency under judge B concerns its assessable texts, despite poor clarity overall.
Table 27. Blinded LLM assessment of 128-token LM1B texts at 750k updates and p = 1 p = 1 p = 1 : means and 95% bootstrap intervals. Consistency uses assessable texts; U U U is the unassessable percentage. Accept. is the percentage passing the joint clarity/consistency criterion, using all 300 texts per method or 100 real texts. Bold marks the highest clarity, grammar, and acceptance means among generators within each judge. Zero-pass bootstrap bounds collapse to zero without establishing a zero population rate.
Controls and reliability. Real-text controls shuffle words (98), splice equal-length word blocks from other real texts (96), or repeat blocks (98), preserving sentence markers and outer fragments. Eight ineffective controls were omitted. Each judge also rated 159 hidden repeats, excluded from method counts; control intervals resample parent–control pairs. Both judges detected perturbations and largely agreed on acceptance, but differed in individual scores (Table 28 ). Judge B often repeated entire responses to matching items, so within-session recognition or reuse remains possible.
Table 28. Acceptance contrast, control effects, and judge agreement. Intervals are 95% bootstrap intervals; control changes are relative to paired real text. Repeats use 159 pairs per judge; between-judge agreement uses 1,592 base items, or 1,293 jointly assessable items for consistency. All-score agreement includes assessability; full-response matches also require identical evidence and explanation.
Table 28: Acceptance contrast, control effects, and judge agreement. Intervals are 95% bootstrap intervals; control changes are relative to paired real text. Repeats use 159 pairs per judge; between-judge agreement uses 1,592 base items, or 1,293 jointly assessable items for consistency. All-score agreement includes assessability; full-response matches also require identical evidence and explanation.
External-context diagnostic. We used GPT-2-large as an external reader to ask: Does the earlier part of a generated sentence help predict a later token, beyond what the most recent tokens already tell us? We scored each selected token under three versions of its preceding context.
Consider the illustrative sentence “The chef mixed the dough and put it in the oven.” Treating each word as one token for this example, we score “oven” with k = 4 k = 4 k = 4 recent tokens, “put it in the”:
Table 37.
Shuffling preserves the initial word, recent tokens, and older word multiset. Hypothetical target probabilities of 20%, 5%, and 10% under these contexts give gains of log ( 0.20 / 0.05 ) ≈ 1.39 \log(0.20/0.05) \approx1.39 log ( 0.20/0.05 ) ≈ 1.39 nats for retaining older context and log ( 0.20 / 0.10 ) ≈ 0.69 \log(0.20/0.10) \approx0.69 log ( 0.20/0.10 ) ≈ 0.69 nats for retaining its order. Positive gains favor the original context; negative gains favor the altered context.
We apply this test to all 1,300 original texts, including real text, using GPT-2-large in FP32 and its subword tokens, with k ∈ { 4 , 8 , 16 } k \in\{4, 8, 16\} k ∈ { 4 , 8 , 16 } . We use spans between internal sentence markers and, with a fixed seed, uniformly sample at most 16 targets per text and k k k that admit a nonidentity shuffle. Shuffling keeps word groups intact and fixes the initial group, recent k k k tokens, and any group crossing the old/recent boundary; truncation retains original position indices. Target sets can differ across k k k .
G k s h o r t G_k^{\mathrm{short}} G k short and G k s h u f f l e G_k^{\mathrm{shuffle}} G k shuffle average the full-minus-truncated and full-minus-shuffled log-probabilities over targets. For 95% percentile intervals, we resample texts within generation seeds, retaining all distances, conditions, and texts with no eligible targets: 1,000 replicates for gains and 10,000 for post-hoc pairwise differences, sampling methods independently. Intervals condition on one training run per model and the fixed scorer; pairwise intervals are pointwise and unadjusted for nine comparisons.
Context results and interpretation. RJD-LOO has the largest shuffle gain among generators, with a clearly positive absolute gain only for k = 4 k = 4 k = 4 . Real text benefits at all three values of k k k (Table 29 ). Pairwise intervals favor RJD-LOO over all competitors for k = 4 , 8 k = 4, 8 k = 4 , 8 , and over RJD alone for k = 16 k = 16 k = 16 (Table 30 ). Truncation gains can reflect boundary and position-calibration effects; length-preserving shuffling controls these changes, while repetition and topic can still affect predictability.
Table 29. External-context diagnostic: gains in nats per selected target with 95% bootstrap intervals. Coverage is the percentage of original texts with eligible targets; Targets counts selected tokens. Bold marks the largest mean shuffle gain among generators at each k k k . Truncation can change scorer calibration.
Table 30. Difference in context-shuffling gain: RJD-LOO minus the specified competitor, in nats per selected target. Positive values favor RJD-LOO. Pointwise 95% bootstrap intervals appear underneath; these post-hoc intervals are not adjusted for multiple comparisons.
ZINC250k experimental details
Data, representation, and predictor
We use the GDSS benchmark CSV and test indices at commit 24cc490e0c5b [19 ] . From its 224,568 training examples, we reserve 5,000 for development using split seed 314159 and grouping identical non-isomeric canonical SMILES, leaving 219,568 fitting examples. Together with the 24,887 official test examples, these comprise 249,455 molecules. The input molecules are connected. Fitting and development identities do not overlap. The official test split is unchanged; 295 test entries share fitting identities after non-isomeric canonicalization. This inherited overlap is retained and recorded. Test examples supply neither training batches nor the atom-count distribution. Holding out development examples reduces the fitting set relative to methods using the entire published training pool.
Graphs have at most 38 heavy atoms and are represented by L = 38 + ( 38 2 ) = 741 L = 38 + \binom{38}{2} = 741 L = 38 + ( 2 38 ) = 741 coordinates. Atom counts are sampled from the fitting-set distribution. The nine atom types are {C, N, O, F, P, S, Cl, Br, I}; bond labels are no bond, single, double, and triple. Kekulization encodes aromatic bonds through single and double bonds. Hydrogens are implicit; the model predicts neither stereochemistry nor three-dimensional coordinates nor separate formal-charge labels. Decoding follows the GDSS positive-charge convention where applicable. Given the sampled atom count n n n , the first n n n atom slots and all ( n 2 ) \binom{n}{2} ( 2 n ) corresponding pair slots are active, including pairs labeled “no bond.” Padding is an external mask, not an atom category; inactive slots are excluded from the loss and attention sources.
The atom and bond embeddings are learned vectors in R 64 \mathbb{R}^{64} R 64 , each normalized to norm 8.
The predictor has eight blocks, width 256, eight attention heads, 32 time features, 32 context channels, dropout 0.1, AdaLN conditioning, and SwiGLU layers. The atom and bond classifiers share this backbone; they are not separate drift and jump heads. Its 6,923,552 parameters and the ( 9 + 4 ) × 64 = 832 (9 + 4) \times64 = 832 ( 9 + 4 ) × 64 = 832 embedding parameters total 6,924,384 for each of the five trained models. Parameters have matched shapes and initializations within each seed.
An atom query aggregates features of other atoms and their incident bonds; a bond query uses symmetric endpoint features and pairs of bonds through a third atom. Each such context excludes the target coordinate’s state and absorption indicator. All features use noisy observations, without access to the clean adjacency. Attention keys and values are fixed functions of individual input coordinates, as in (9) ; updated queries never serve as sources. LOO excludes the target source in every layer. Allowing this source is the only architectural change for predictors of p i , t ( ⋅ ∣ X t ) p_{i,t}(\cdot\mid X_t) p i , t ( ⋅ ∣ X t ) . Shared features and the absence of positional embeddings give equivariance under joint permutations of atom slots and their induced pair slots.
Training and sampling
With u = t 2 u = t^2 u = t 2 , a continuous coordinate before absorption has conditional law N ( u e ( y i ) , u ( 1 − u ) I 64 ) \mathcal{N}(u e(y_i), u(1-u)I_{64}) N ( u e ( y i ) , u ( 1 − u ) I 64 ) and starts at zero. Jump coordinates have first-jump CDF t 2 t^2 t 2 and hazard 2 t / ( 1 − t 2 ) 2t/(1-t^2) 2 t / ( 1 − t 2 ) ; after jumping they remain at the selected anchor. An observed indicator records absorption. In EPJ, the coordinate remains zero before its jump. For unabsorbed continuous coordinates, Bayesian completion uses the Gaussian likelihood above. PD-BC completes the LOO prediction for the drift, and AJD-BC-D does so only for the drift, retaining LOO jump probabilities. EPJ uses ρ θ , i , t \rho_{\theta,i,t} ρ θ , i , t . Table 31 maps the seven sampling variants to the five trained models. PD-LOO and PD-BC share one checkpoint, as do AJD-LOO and AJD-BC-D; completion is a sampling-time operation. The absorbing process differs from RJD: continuous motion stops after a jump, and Proposition 3.2 is not a marginal-preservation result for this augmented absorbing process.
Table 31. ZINC250k methods. All predictors have the same architecture and parameter count. Unrestricted and LOO prediction differ only in allowing or excluding the target input as an attention source. BC denotes analytic likelihood completion; all jump destinations use CAT in the test comparison.
Training samples t t t uniformly from [ 0 , 0.999 ) [0, 0.999) [ 0 , 0.999 ) and minimizes 1 2 L a t o m + 1 2 L b o n d \frac{1}{2}\mathcal{L}_{\mathrm{atom}} + \frac{1}{2}\mathcal{L}_{\mathrm{bond}} 2 1 L atom + 2 1 L bond , where each cross-entropy is normalized over its eligible coordinates. These are all active coordinates for PD and only unabsorbed active coordinates for jump paths. No additional bond-class weights are used. AdamW uses batch size 512, accumulated in microbatches of 128, learning rate 10 − 3 10^{-3} 1 0 − 3 after 1,000 warmup updates, cosine decay to 10 − 4 10^{-4} 1 0 − 4 at 40,000 updates, zero weight decay, and gradient-norm clipping at one. EMA includes predictor and anchors, with decay min ( 0.999 , ( 1 + s ) / ( 10 + s ) ) \min(0.999, (1+s)/(10+s)) min ( 0.999 , ( 1 + s ) / ( 10 + s )) at update s s s . The test comparison uses the EMA weights at exactly 40,000 updates, rather than selecting checkpoints by validation loss. Computation uses FP32 with TF32 disabled.
For M M M NFE, the sampling grid is t j = 1 − cos ( π j / ( 2 M ) ) t_j = 1 - \cos(\pi j/(2M)) t j = 1 − cos ( π j / ( 2 M )) , j = 0 , … , M j = 0, \ldots, M j = 0 , … , M , with u j = t j 2 u_j = t_j^2 u j = t j 2 . There is one predictor call at each left endpoint. Given its predicted anchor mean m m m , the continuous update is
x ′ = m + α ( x − m ) + ( u j + 1 − u j ) α ε , α = 1 − u j + 1 1 − u j , ε ∼ N ( 0 , I 64 ) . x' = m + \alpha(x - m) + \sqrt{(u_{j+1} - u_j)\alpha}\,\varepsilon,\qquad\alpha= \frac{1-u_{j+1}}{1-u_j},\qquad\varepsilon\sim\mathcal{N}(0,I_{64}). x ′ = m + α ( x − m ) + ( u j + 1 − u j ) α ε , α = 1 − u j 1 − u j + 1 , ε ∼ N ( 0 , I 64 ) .
An eligible coordinate jumps with probability ( u j + 1 − u j ) / ( 1 − u j ) (u_{j+1} - u_j)/(1-u_j) ( u j + 1 − u j ) / ( 1 − u j ) . Thus the final interval reaches u = 1 u=1 u = 1 without a network call at t = 1 t=1 t = 1 , and M M M NFE counts exactly M M M backbone calls. Pure diffusion is decoded by nearest anchor, equivalently maximum anchor inner product. Test jump destinations use CAT at temperature one, without correction steps. Except for the evaluation specified below, atom counts and generation seeds are paired across methods, although different samplers consume random numbers differently. Evaluation uses batches of 32 graphs and sampling seed base 1729. For AJD-LOO at 512 NFE with training seed 1, we use an independent H100 evaluation with batch size 256 and sampling seed base 271828, keeping the same 40,000-update EMA checkpoint, strict LOO predictor, and CAT sampler without BC. This evaluation has the same sample count and test reference but does not share the paired atom-count and sampling-seed realization. EPJ, PD, PD-BC, AJD, and AJD-BC-D use H100 for both seeds. PD-LOO and AJD-LOO evaluations use V100 for seed 0 and H100 for seed 1 at 64 and 128 NFE; 512-NFE evaluations also use A100 GPUs. Table 33 reports generation time and hardware separately for each seed, excluding scoring and queue waits. These times are not a comparison of sampler speed on a common GPU and batch size.
Table 33. Secondary ZINC250k metrics and generation time for the same two-seed test samples as Table 32. GDSS valence is checked before correction. Uniqueness uses valid outputs as denominator; pruning uses all attempted graphs. All four metric columns are percentages, with the mean above ± sample standard deviation. Each time column shows generation minutes for 10,000 graphs above the GPU used, separately for each seed; scoring and queue waits are excluded. A slash indicates generation used both listed GPU types. Batch size is 32 except for seed-1 AJD-LOO at 512 NFE (256).
Molecular evaluation metrics
Validity (no correction) is the number of outputs passing full RDKit sanitization before repair divided by the number generated. Valid & connected additionally requires exactly one connected component, with the same denominator; it is not connectivity conditional on validity. For comparison with GDSS ([19 ] ), we separately report the fraction passing its valence check before bond correction, labeled GDSS valence (no correction). This uses the GDSS check_valency routine rather than full sanitization; the two checks need not agree, and neither requires a connected graph. Corrected outputs use GDSS bond downgrading followed by largest-component selection; we compute their metrics separately from raw outputs.
FCD compares the empirical means and covariances of pretrained ChemNet features ([30 ] ), using fcd-torch 1.0.7. For NSPDK, we use normalized GDSS/EDedN graph features at complexity four ([8 ] ; [19 ] ). If ϕ ( G ) \phi(G) ϕ ( G ) denotes these features, the reported statistic is
M M D ^ 2 = ∥ 1 N ∑ r = 1 N ϕ ( G r ) − 1 N r e f ∑ r = 1 N r e f ϕ ( G r r e f ) ∥ 2 , \widehat{\mathrm{MMD}}^2 = \left\lVert\frac{1}{N}\sum_{r=1}^{N}\phi(G_r) - \frac{1}{N_{\mathrm{ref}}}\sum_{r=1}^{N_{\mathrm{ref}}}\phi(G_r^{\mathrm{ref}}) \right\rVert^2, MMD 2 = N 1 r = 1 ∑ N ϕ ( G r ) − N ref 1 r = 1 ∑ N ref ϕ ( G r ref ) 2 ,
the biased empirical squared MMD for the corresponding linear kernel. Here N N N counts valid raw outputs or valid corrected outputs, respectively; invalid outputs do not enter either distributional metric. Their frequency is therefore reported separately by validity. The FCD reference retains the benchmark SMILES, including stereochemical information where present.
Uniqueness is the number of distinct valid non-isomeric canonical SMILES divided by the number of valid outputs. Novelty is the fraction of distinct valid identities absent from the training reference, reported against both the fitting set and the full published training pool. The fitting-copy fraction instead counts repetitions and divides the number of valid samples matching fitting identities by the valid sample count. Raw identities retain all components; corrected identities retain only the selected component. Component pruning is the fraction of generated graphs for which decoding discards components after bond repair.
The test evaluation uses 10,000 attempted graphs per method, NFE budget, and training seed, with all 24,887 test molecules as reference. Invalid outputs are not replaced to reach a target number of valid molecules. The tables report training seeds 0 and 1. Each metric is computed separately per seed and summarized by its mean and sample standard deviation; generated collections are not pooled for FCD. The development sweep instead uses seed 0, 1,000 graphs, and all 5,000 development references. Absolute FCD values across these protocols are not a matched measure of training progress. Evaluation uses RDKit 2025.03.6, NumPy 2.2.6, NetworkX 3.4.2, and a fixed Python hash seed of zero for the NSPDK feature map.
Test results at 40,000 updates
Table 32 extends the main comparison to 64 and 512 NFE and includes corrected NSPDK. At 128 NFE, raw LOO drift gives AJD the lowest mean FCD, while completion gives it the lowest mean NSPDK; these metrics therefore rank the drift choices differently. Increasing NFE improves EPJ and AJD-BC-D raw validity and connected validity, whereas PD and PD-BC change less. At 512 NFE, AJD-LOO has the highest mean connected validity and lowest mean raw FCD, while AJD-BC-D has the lowest mean corrected FCD and NSPDK. These rankings describe means over two seeds.
Table 32. ZINC250k test results: 40,000-update EMA weights, training seeds 0 and 1, 10,000 attempted graphs per seed, and all 24,887 test references. Each cell shows the mean above ± sample standard deviation. All jumps use CAT and κ ( t ) = t 2 \kappa(t)=t^2 κ ( t ) = t 2 . The validity percentages use all generated graphs as denominator. NSPDK is squared MMD. Corrected metrics use bond repair and largest-component selection. The seed-1 AJD-LOO evaluation at 512 NFE uses the independent sampling realization described above.
Secondary metrics and generation time. All 42 test evaluations have 100% corrected validity and no decoding errors. Novelty against both training references is 100%, with one exception: PD-LOO at 128 NFE for seed 1 contains one fitting-set copy, giving raw and corrected novelty of 99.9825% and 99.9867%, respectively. Its raw and corrected fitting-copy fractions are 1/5921 and 1/10000; all other evaluations have zero observed fitting copies. Raw uniqueness is 100% except for PD-LOO; corrected uniqueness is also 100% except for PD-LOO and AJD at 128 NFE for seed 0 and 512 NFE for seed 1 (99.99% for each of these two AJD evaluations). Table 33 gives the nonconstant secondary metrics. The low uniqueness of PD-LOO shows an additional limitation of its raw LOO drift. High novelty or corrected validity alone does not establish good raw validity, connectivity, or distributional agreement.
Development sampling ablations at 20,000 updates
We separately compare the five sampling rules from Appendix § 13 .7 on fixed 20,000-update EMA checkpoints: CAT, MAX, TEMP with temperature 0.7, TOPP with mass 0.9, and CONF. For CONF, event counts are preserved separately for atom and bond coordinates, then assigned to the most confident eligible coordinates within each group. The drift remains at temperature one and retains the prediction choice in Table 31 ; only jump sampling is varied. These rules change the sampling dynamics and are not covered by the repeated-reset marginal-preservation guarantee.
Table 34 and Table 35 report the complete seed-0 sweep with 1,000 generated graphs per setting and 5,000 development references. MAX markedly increases EPJ validity but worsens FCD and NSPDK. For AJD-LOO at 128 NFE, TEMP improves validity, connected validity, and FCD over CAT, while worsening NSPDK; TOPP improves all six reported metrics over CAT. Thus a sampling rule that improves validity need not improve distributional agreement, and these development results do not substitute for the 40,000-update test comparison.
Table 34. ZINC250k development sampling sweep at 64 NFE: 20,000-update EMA weights, seed 0, 1,000 attempted graphs, and 5,000 development references. Validity percentages use all generated graphs. NSPDK is squared MMD. These scores use a different checkpoint, sample count, and reference from the test comparison.
Table 35. ZINC250k development sampling sweep at 128 NFE, with the same 20,000-update checkpoints, seed 0, 1,000 attempted graphs, and 5,000 development references as Table 34. Metric definitions and sampling rules are unchanged.