betaFeedback

WorksComputer Science

BCT: Bayes-Consistent Tempering for Flow Language Models

Abstract

Temperature scaling is typically applied as a post-hoc transformation of model probabilities. In posterior-induced flow language models, this creates two mismatches. First, posterior tempering implicitly changes both the endpoint distribution and the distribution of intermediate noisy states, while the denoiser remains trained on the original states. Second, the tempered posterior weights are combined with transport dynamics derived for the untempered flow. These mismatches are reflected empirically: lowering temperature improves GSM8K performance much less reliably in flow LMs than in discrete diffusion LMs. We introduce Bayes-Consistent Tempering (BCT), which makes temperature explicit in both the distributions and dynamics of the generative flow. Its learned realization trains the denoiser on tempered intermediate states and tempers the endpoint distribution using the corresponding tempered dynamics. We also derive training-free variants for conventionally trained checkpoints. Across hyperspherical flow LMs with two denoiser architectures and a von Mises–Fisher flow LM, BCT improves GSM8K generation across broad temperature ranges and achieves the strongest tempering performance. The gains also extend to Sudoku, demonstrating benefits on tightly constrained generation. At very low temperatures, the learned realization is also markedly more robust than the training-free variants, supporting temperature as a property of the generative flow across training and inference rather than a post-hoc probability adjustment. Code and model weights are available at https://daioba.github.io/bct .

Figure 1. Conceptual overview of Bayes-Consistent Tempering (BCT). Posterior tempering implicitly changes both the endpoint distribution and the conditional bridge, while standard sampling retains the original conditional velocities and denoiser training states. BCT restores consistency between the bridge and sampling dynamics, and its learned realization additionally aligns training with the tempered bridge.

Introduction

Temperature scaling is one of the simplest controls for language generation [1, 17]. In autoregressive models, dividing next-token logits by T<1T < 1 directly sharpens the distribution from which the next token is sampled. In many flow language models for discrete data, however, generation is driven by a denoising posterior: a model predicts the clean endpoint token from an intermediate noisy state, and this posterior induces the velocity that transports the state toward token embeddings [35, 21, 12]. Applying temperature only to this posterior creates two mismatches. First, by Bayes’ rule, posterior tempering implicitly changes not only the endpoint distribution but also the conditional bridge—the time-indexed distribution of intermediate noisy states given each endpoint. The denoiser, however, is trained on states from the untempered bridge, so tempering changes the states encountered at inference without a corresponding change in training. Second, tempering changes the posterior weights over endpoints but leaves their conditional velocities unchanged, even though those velocities are designed for the original, untempered bridges. Thus, temperature changes the implied generative distribution without consistently changing how the model is trained or sampled.

This issue is also visible empirically. On GSM8K [10], prior work reports that flow language models gain little from lowering the temperature, in contrast to discrete diffusion models such as MDLM [31] and Duo [32], whose accuracy improves substantially under low-temperature decoding [12]. This discrepancy motivates us to ask whether temperature is being incorporated correctly into posterior-induced flows.

We introduce Bayes-Consistent Tempering (BCT), which treats temperature as part of the generative flow rather than as a post-hoc transformation of the denoising posterior (§ 3). Its learned realization trains the denoiser on states from a tempered conditional bridge and, at inference, applies the Bayes correction needed to temper the endpoint distribution while preserving that tempered bridge, together with the dynamics induced by the same bridge. We call the training component Tempered-Flow Training (TFT) and the corresponding sampler BCT-L.

We further investigate how much inference-time consistency can be recovered without retraining through two variants for existing checkpoints: BCT-P tempers both the endpoint distribution and bridge with matched dynamics, whereas BCT-E tempers only the endpoint distribution, retaining the original bridge and dynamics (§ 3.2).

For posterior-induced flow language models, BCT is not merely a conceptual framework: applying it requires deriving the tempered bridge and its corresponding sampling dynamics, and these quantities are tractable in concrete models. For S-FLM [12], for example, the hyperspherical geometry lets us derive both explicitly. The same principle applies to SFCD [7], which uses a von Mises–Fisher bridge with its own analytic tempering rule and corresponding dynamics. Thus, BCT is not specific to a particular denoiser architecture; its construction is determined by the bridge that defines the flow.

Across hyperspherical flow LMs with two denoiser architectures (S-arch/DiT) [12] and a von Mises–Fisher flow LM [7], BCT improves GSM8K over broad temperature ranges (§ 4). BCT-L is particularly robust at very low temperatures, outperforming the training-free variants and highlighting the benefit of temperature-aware training. Gains also extend to Sudoku [5, 20], where standard tempering does not benefit from lower temperatures, suggesting applicability to constrained generation.

Preliminaries

Posterior-induced flows

We consider a single token position with finite vocabulary V\mathcal{V} and endpoint distribution π(x)\pi(x). Let zz denote the continuous latent state and t∈[0,1]t \in[0,1] run from noise to data. For each endpoint token xx, a conditional bridge qt(z∣x)q_t(z \mid x) describes the distribution of intermediate states at time tt, starting from a token-independent source p0p_0 and converging toward its embedding exe_x. All bridge densities are defined with respect to a fixed reference measure μ\mu on the latent space. A corresponding **conditional velocity ut(z∣x)u_t(z \mid x) transports states so that their distribution follows qt(⋅∣x)q_t(\cdot\mid x) over time; formally, qtq_t and utu_t satisfy the continuity equation.

Bayes’ rule gives the denoising posterior

rt(x∣z)=π(x)qt(z∣x)∑y∈Vπ(y)qt(z∣y)(1)r_t(x \mid z) = \frac{\pi(x)q_t(z \mid x)}{\sum_{y\in\mathcal{V}}\pi(y)q_t(z \mid y)} \tag*{(1)}

and the generative velocity is its posterior-weighted average of the conditional velocities,

ut(z)=∑x∈Vrt(x∣z)ut(z∣x).(2)u_t(z) = \sum_{x\in\mathcal{V}} r_t(x \mid z)u_t(z \mid x). \tag*{(2)}

A denoiser rtη(x∣z)r_t^\eta(x \mid z) is trained to approximate rtr_t and replaces it in Eq. (2) at generation time. We call flows constructed in this way posterior-induced flows.

Standard tempering

Writing β=1/T\beta= 1/T, standard temperature scaling replaces the denoising posterior by

rtT(x∣z)∝rt(x∣z)β,utT(z)=∑xrtT(x∣z)ut(z∣x),(3)r_t^T(x \mid z) \propto r_t(x \mid z)^\beta,\qquad u_t^T(z) = \sum_x r_t^T(x \mid z)u_t(z \mid x), \tag*{(3)}

thus changing the posterior weights while leaving the conditional velocities unchanged. In practice, this is implemented by dividing the denoiser logits by TT.

Using Eq. (1),

rt(x∣z)β∝π(x)βqt(z∣x)β,(4)r_t(x \mid z)^\beta\propto\pi(x)^\beta q_t(z \mid x)^\beta, \tag*{(4)}

so posterior tempering acts algebraically on both the endpoint factor and the bridge likelihood. Define

πT(x)∝π(x)β,qt,T(z∣x)=qt(z∣x)βZt,T(x),Zt,T(x)=∫qt(z∣x)β dμ(z).(5)\pi_T(x) \propto\pi(x)^\beta,\qquad q_{t,T}(z \mid x) = \frac{q_t(z \mid x)^\beta}{Z_{t,T}(x)},\qquad Z_{t,T}(x) = \int q_t(z \mid x)^\beta\,\mathrm{d}\mu(z). \tag*{(5)}

The posterior induced by (πT,qt,T)(\pi_T,q_{t,T}) differs from standard posterior tempering only by the token-dependent normalizer Zt,T(x)Z_{t,T}(x). Hence the two coincide whenever Zt,T(x)Z_{t,T}(x) is independent of xx, as for the symmetric bridge families studied in this work.

For clarity, we present all derivations for a single token position. For sequences, the same decomposition applies position-wise, with π(x)\pi(x) replaced by the context-conditioned endpoint distribution induced by the remaining positions; see Appendix § 9.

Bayes-Consistent Tempering

Once an endpoint distribution and a conditional bridge are specified, the posterior weights and sampling dynamics should correspond to that same generative model. In particular, for the tempered pair (πT,qt,T)(\pi_T,q_{t,T}), the sampler should use its denoising posterior, conditional velocities ut,T(z∣x)u_{t,T}(z \mid x) that transport qt,Tq_{t,T}, and the source q0,Tq_{0,T} of that bridge. However, standard tempering does not generally satisfy this alignment: while its posterior corresponds to the tempered pair under the condition described in § 2, it continues to use the conditional velocities ut(z∣x)u_t(z \mid x) of the original bridge.

In contrast, our proposal, Bayes-Consistent Tempering (BCT), treats the endpoint distribution, conditional bridge, and sampling dynamics as components of the same generative flow. We first develop its learned realization, which also aligns the distribution of states seen during training (§ 3.1), and then derive post-hoc variants for conventionally trained models (§ 3.2).

Learned BCT

The learned realization of BCT, BCT-L, aligns training with the tempered bridge used at inference. We first train the denoiser on intermediate states drawn from qt,Tq_{t,T}, rather than from the original bridge qtq_t. Given a training distribution p(T)p(T) over temperatures, Tempered-Flow Training (TFT) minimizes the usual denoising loss,

L(η)=ET∼p(T), t, x∼π, z∼qt,T(⋅∣x)[−log⁡rt,Tη(x∣z)],(6)\mathcal{L}(\eta)=\mathbb{E}_{T\sim p(T),\,t,\,x\sim\pi,\,z\sim q_{t,T}(\cdot\mid x)}\left[-\log r_{t,T}^{\eta}(x\mid z)\right], \tag*{(6)}

where the denoiser is conditioned on TT. Since the endpoint token is still drawn from π\pi while its intermediate state is drawn from tempered bridge qt,Tq_{t,T}, the Bayes-optimal denoiser is

rt,T(x∣z)=π(x) qt,T(z∣x)∑yπ(y) qt,T(z∣y).(7)r_{t,T}(x\mid z)=\frac{\pi(x)\,q_{t,T}(z\mid x)}{\sum_y\pi(y)\,q_{t,T}(z\mid y)}. \tag*{(7)}

Thus, TFT incorporates the tempered bridge into the learned posterior, but does not itself temper the endpoint distribution.

At inference, BCT targets the pair (πT,qt,T)(\pi_T,q_{t,T}). Its tempered posterior can be written directly in terms of the TFT posterior as

rt,TBCT−L(x∣z)∝rt,T(x∣z)βqt,T(z∣x)1−β.(8)r_{t,T}^{\mathrm{BCT-L}}(x\mid z)\propto r_{t,T}(x\mid z)^\beta q_{t,T}(z\mid x)^{1-\beta}. \tag*{(8)}

The second factor is the Bayes correction that tempers the endpoint while preserving the tempered bridge already represented by the denoiser. In practice, learned posteriors are approximate, so we allow the strength of this correction to be selected on held-out data:

ℓxBCT−L=βℓxη+λ(1−β)log⁡qt,T(z∣x),0≤λ≤1,(9)\ell_x^{\mathrm{BCT-L}}=\beta\ell_x^\eta+\lambda(1-\beta)\log q_{t,T}(z\mid x),\qquad0\le\lambda\le1, \tag*{(9)}

where one λ\lambda is selected per trained model and shared across temperatures.

Finally, BCT-L uses the tempered conditional velocity ut,T(z∣x)u_{t,T}(z\mid x) that transports the corresponding tempered bridge qt,Tq_{t,T}:

ut,TBCT−L(z)=∑xrt,TBCT−L(x∣z)ut,T(z∣x).(10)u_{t,T}^{\mathrm{BCT-L}}(z)=\sum_x r_{t,T}^{\mathrm{BCT-L}}(x\mid z)u_{t,T}(z\mid x). \tag*{(10)}

with sampling initialized from the tempered source q0,Tq_{0,T}. The form of ut,Tu_{t,T} is model-dependent; we derive it for the flow constructions studied here in § 3.3. Thus, TFT and BCT-L align the intermediate states seen during training, the endpoint distribution, and the sampling dynamics within the same generative flow.

Post-hoc BCT Variants for Conventionally Trained Models

We next ask how much of BCT can be recovered after a model has already been trained on the original bridge qtq_t. We consider two complementary post-hoc variants—BCT-P and BCT-E.

Path tempering (BCT-P). BCT-P retains posterior tempering, which implicitly tempers both the endpoint distribution and the bridge, but replaces the original conditional dynamics with those of the tempered bridge. Its posterior is

rt,TBCT−P(x∣z)∝rt(x∣z)βZt,T(x),(11)r_{t,T}^{\mathrm{BCT-P}}(x\mid z)\propto\frac{r_t(x\mid z)^\beta}{Z_{t,T}(x)}, \tag*{(11)}

and the corresponding velocity is

ut,TBCT−P(z)=∑xrt,TBCT−P(x∣z)ut,T(z∣x).(12)u_{t,T}^{\mathrm{BCT-P}}(z)=\sum_x r_{t,T}^{\mathrm{BCT-P}}(x\mid z)u_{t,T}(z\mid x). \tag*{(12)}

When Zt,T(x)Z_{t,T}(x) is independent of xx, BCT-P uses exactly the same posterior weights as standard tempering; the difference is that its conditional velocities are matched to the tempered bridge. Unlike BCT-L, however, its denoiser was trained only on states from the original bridge qtq_t, leaving a train–inference mismatch in the intermediate states.

Endpoint tempering (BCT-E). BCT-E avoids tempering the bridge altogether. It keeps untempered bridge and conditional dynamics, and tempers only the endpoint distribution π\pi. By Bayes’ rule, its posterior can be written using the original denoiser as

rt,TBCT−E(x∣z)∝rt(x∣z)βqt(z∣x)1−β,(13)r_{t,T}^{\mathrm{BCT-E}}(x\mid z)\propto r_t(x\mid z)^\beta q_t(z\mid x)^{1-\beta}, \tag*{(13)}

or, in logits,

ℓxE=βℓx+(1−β)log⁡qt(z∣x).(14)\ell_x^{\mathrm{E}} = \beta\ell_x + (1-\beta)\log q_t(z \mid x). \tag*{(14)}

These weights are combined with the original conditional velocities ut(z∣x)u_t(z \mid x).

Both variants require no retraining. BCT-P restores consistency by matching the sampling dynamics to the tempered bridge, whereas BCT-E restores consistency by keeping the bridge and sampling dynamics untempered.

Instantiations

We instantiate BCT on two posterior-induced flow constructions with different conditional bridges.

Hyperspherical flow. S-FLM ([12]) is based on hyperspherical flow, which places token embeddings and latents on the unit sphere and transports a uniform source toward each endpoint along geodesics. Let θx(z)=arccos⁡⟨z,ex⟩\theta_x(z) = \arccos\langle z, e_x \rangle and at=1−αta_t = 1-\alpha_t denote the remaining fraction of the geodesic at time tt. Its conditional bridge has density

qt(z∣x)∝sin⁡d−2(θx/at)sin⁡d−2θx1{θx<atπ},(15)q_t(z \mid x) \propto\frac{\sin^{d-2}(\theta_x/a_t)}{\sin^{d-2}\theta_x}\mathbb{1}\{\theta_x < a_t\pi\}, \tag*{(15)}

with respect to the spherical surface measure. This density gives the bridge likelihood needed by BCT-E and TFT. Rotational symmetry also makes Zt,T(x)Z_{t,T}(x) independent of xx, so BCT-P uses the same posterior weights as standard tempering.

The tempered bridge qt,Tq_{t,T} remains rotationally symmetric around exe_x. Its corresponding conditional velocity therefore follows the same geodesic direction as the original S-FLM velocity, with a temperature-dependent change in speed,

ut,T(z∣x)=wt,T(θx)ut(z∣x),(16)u_{t,T}(z \mid x) = w_{t,T}(\theta_x)u_t(z \mid x), \tag*{(16)}

where wt,Tw_{t,T} is obtained from a one-dimensional continuity equation; we give its derivation in Appendix § 10.§ 3. The same rotational symmetry reduces exact sampling from qt,Tq_{t,T} to a one-dimensional transformation of the polar angle, allowing TFT to draw training states directly from the tempered bridge (Appendix § 10.§ 4).

von Mises–Fisher flow. Chemseddine et al. [7] use a von Mises–Fisher conditional bridge,

qt(z∣x)∝exp⁡(κt⟨ex,z⟩).(17)q_t(z \mid x) \propto\exp(\kappa_t \langle e_x,z\rangle). \tag*{(17)}

Tempering stays within the same family:

qt,T(z∣x)∝exp⁡(κtT⟨ex,z⟩),(18)q_{t,T}(z \mid x) \propto\exp\left(\frac{\kappa_t}{T}\langle e_x,z\rangle\right), \tag*{(18)}

so BCT simply replaces κt\kappa_t by κt/T\kappa_t/T and uses the corresponding conditional dynamics (Appendix § 15). TFT likewise draws training states directly from the vMF distribution with concentration κt/T\kappa_t/T. The normalizer is independent of xx, and BCT-P therefore again shares its posterior weights with standard tempering.

For von Mises–Fisher flow, the tempered bridge evidence gt,T(x,z)=(κt/T)⟨ex,z⟩g_{t,T}(x,z) = (\kappa_t/T)\langle e_x,z\rangle is available analytically. For TFT, we therefore include this term explicitly in the denoiser logits, ℓx=aη,x+gt,T(x,z)\ell_x = a_{\eta,x} + g_{t,T}(x,z), and learn only the remaining component aη,xa_{\eta,x}. This is a parameterization of the same TFT posterior and BCT-L sampler, not a separate training method.

Experiments

Experimental Setup

Main setting. Our main experiments evaluate BCT on GSM8K ([10]) with S-FLM ([12]). Following its setup, we adopt the TinyGSM protocol ([25]): for each of the 1319 test problems, the model generates a Python program, and the problem counts as solved if executing the program returns the reference answer. We use its released TinyGSM checkpoints for two denoiser backbones: S-arch, a Transformer whose intermediate activations lie on the hypersphere, and the standard Diffusion Transformer (DiT; [29]).

We refer to either checkpoint as S-FLM and specify the backbone where relevant. The two backbones share the bridge, the posterior-induced velocity, and all samplers, so comparing them varies only the denoiser architecture.

Additional settings. Two additional experiments test whether our conclusions generalize to other settings (§ 4.4 and § 4.5). First, we change the task while keeping the hyperspherical flow, using Sudoku-Hard [20, 5], which requires tightly constrained generation. Since no Sudoku checkpoint is released, we train a DiT-based S-FLM with its released training recipe (Appendix § 14). Second, we change the flow structure while keeping the task: we apply BCT to SFCD [7], which uses a von Mises–Fisher bridge instead of the geodesic bridge of S-FLM, and evaluate its released TinyGSM checkpoint on the same GSM8K problems (Appendix § 15). Unless noted otherwise in their sections, both follow the protocol below.

Samplers. All samplers use the exact posterior-induced velocity with 1024 function evaluations (NFE) and otherwise follow the released sampler; they differ only in the posterior weights and conditional velocities. We compare standard tempering Eq. (3), BCT-E Eq. (14), BCT-P Eq. (12), and BCT-L, which is described below. As a reference, we report the Top-1 velocity [12], which moves only toward the most likely token; it is invariant to temperature and is the T→0T \to0 limit of standard tempering. We evaluate T∈{1,0.8,0.6,0.5,0.4,0.3,0.2,0.1}T \in\{1, 0.8, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1\} in all experiments.

TFT and BCT-L. For each backbone, we continue training the released checkpoint with TFT Eq. (6) for 2,000 updates, 0.8% of its 250k pretraining updates, and refer to the resulting model as TFT. The temperature distribution p(T)p(T) draws T=1T = 1 with probability 0.2, T=0.1T = 0.1 with probability 0.1, and otherwise β∼U(1.25,5)\beta\sim U(1.25, 5), so it covers every evaluated temperature. As a matched training control, Cont. continues training for the same 2,000 updates with the original recipe. All models are evaluated with EMA weights. λ\lambda in Eq. (9) is updated once per model on TinyGSM validation problems and shared across temperatures (Appendix § 11).

Main Results

Figure 2 and Table 1 show the results for the hyperspherical flow, whose bridge moves latents along geodesics on the sphere (§ 3.3), with both denoiser backbones. Since TFT continues a pretrained model rather than training from scratch, the fair baseline for BCT-L is Continual Training (Cont.), which receives the same additional update steps with the original recipe; the two differ only in whether temperature enters training. The continuation itself changes little: standard tempering on Cont. stays within one point of the released checkpoint at every temperature, and the training-free variants behave alike on the two checkpoints.

GSM8K accuracy against the sampling temperature on the two backbones

Figure 2. GSM8K accuracy against the sampling temperature on the two backbones (NFE = 1024). Lines are the mean over three generation seeds and bands ± one sample standard deviation. Curves without a prefix use the released checkpoint; Cont. (dashed) is the matched continuation. At T=1T = 1 every BCT correction is the identity. Values in Table 1.

Table 1: GSM8K accuracy (%), mean ±\pm sample standard deviation over three generation seeds (1319 problems, NFE = 1024).

TTTFT BCT-LCont. StandardCont. BCT-ECont. BCT-PS-FLM StandardS-FLM BCT-ES-FLM BCT-P
112.6 ±\pm 1.112.0 ±\pm 0.512.0 ±\pm 0.512.0 ±\pm 0.512.5 ±\pm 0.512.5 ±\pm 0.512.5 ±\pm 0.5
0.816.3 ±\pm 0.214.1 ±\pm 0.515.0 ±\pm 0.515.9 ±\pm 0.214.7 ±\pm 0.715.0 ±\pm 1.116.7 ±\pm 0.2
0.620.4 ±\pm 0.415.3 ±\pm 0.616.5 ±\pm 0.420.2 ±\pm 0.315.6 ±\pm 0.216.9 ±\pm 0.319.7 ±\pm 0.5
0.521.5 ±\pm 0.316.8 ±\pm 0.217.4 ±\pm 0.421.7 ±\pm 0.616.3 ±\pm 0.517.7 ±\pm 0.321.2 ±\pm 0.6
0.423.0 ±\pm 0.917.3 ±\pm 0.718.3 ±\pm 0.523.2 ±\pm 0.517.1 ±\pm 0.518.0 ±\pm 1.122.9 ±\pm 0.6
0.323.9 ±\pm 1.017.3 ±\pm 0.618.4 ±\pm 0.222.3 ±\pm 0.617.4 ±\pm 0.618.5 ±\pm 0.623.0 ±\pm 0.5
0.225.1 ±\pm 0.917.5 ±\pm 0.318.7 ±\pm 0.116.6 ±\pm 1.117.6 ±\pm 0.318.3 ±\pm 0.118.3 ±\pm 0.9
0.127.0 ±\pm 1.017.5 ±\pm 0.518.8 ±\pm 0.13.9 ±\pm 0.517.6 ±\pm 0.518.3 ±\pm 0.15.0 ±\pm 0.4
111.2 ±\pm 1.311.7 ±\pm 1.211.7 ±\pm 1.211.7 ±\pm 1.212.0 ±\pm 1.012.0 ±\pm 1.012.0 ±\pm 1.0
0.816.2 ±\pm 0.814.3 ±\pm 0.415.2 ±\pm 1.017.0 ±\pm 0.414.0 ±\pm 0.614.8 ±\pm 0.917.3 ±\pm 0.5
0.620.2 ±\pm 0.516.5 ±\pm 0.117.9 ±\pm 0.420.8 ±\pm 0.616.4 ±\pm 0.118.1 ±\pm 0.620.8 ±\pm 0.9
0.522.0 ±\pm 0.216.6 ±\pm 1.118.9 ±\pm 1.023.9 ±\pm 0.916.9 ±\pm 0.718.9 ±\pm 1.623.8 ±\pm 0.9
0.424.1 ±\pm 0.317.0 ±\pm 0.619.3 ±\pm 0.326.2 ±\pm 0.617.0 ±\pm 0.919.6 ±\pm 1.125.7 ±\pm 0.7
0.325.2 ±\pm 1.117.8 ±\pm 0.919.7 ±\pm 0.626.7 ±\pm 0.217.2 ±\pm 0.919.4 ±\pm 0.526.7 ±\pm 0.7
0.225.4 ±\pm 0.518.0 ±\pm 0.719.5 ±\pm 0.727.8 ±\pm 0.617.2 ±\pm 0.819.5 ±\pm 0.827.3 ±\pm 0.6
0.126.0 ±\pm 0.617.3 ±\pm 0.219.4 ±\pm 0.617.0 ±\pm 0.617.2 ±\pm 0.719.2 ±\pm 0.616.7 ±\pm 0.4

Table 1. GSM8K accuracy (%), mean ±\pm sample standard deviation over three generation seeds (1319 problems, NFE = 1024).

BCT-L makes low-temperature decoding effective. With conventional training, lowering the temperature helps little: standard tempering stays at 17–18% for every T≤0.4T \le0.4 on both backbones, close to the Top-1 reference, its T→0T \to0 limit. BCT-L instead improves steadily as the temperature decreases, reaching 27.0 ±\pm 1.0% on S-arch and 26.0 ±\pm 0.6% on DiT at T=0.1T = 0.1, and it is more accurate than standard tempering at every T<1T < 1.

Matching the dynamics at inference gives large gains at moderate temperatures. Without training, BCT-P already improves substantially on standard tempering, raising its best accuracy by about 6 points on S-arch and 10 points on DiT, and at moderate temperatures it is comparable to BCT-L, slightly ahead on DiT.

Strong tempering requires training on the tempered bridge. Under strong tempering, however, BCT-P collapses and falls below standard tempering at T=0.1T = 0.1 on both backbones. This is consistent with its train–inference mismatch (§ 3.2): the latents follow the tempered bridge, on whose states the denoiser was not trained, and the mismatch grows as TT decreases. BCT-L, whose denoiser is trained on the tempered bridge, has no such mismatch and is the most accurate sampler at T=0.1T = 0.1. The results at T=0.1T = 0.1 thus particularly highlight the importance of removing the mismatch between the intermediate states seen in training and those visited during sampling.

Keeping the bridge is stable but gains less. BCT-E keeps the bridge on which the denoiser was trained and therefore does not degrade at low temperature, but it improves only modestly on standard tempering and stays below BCT-L at every T<1T < 1, increasingly so as the temperature decreases.

Gains Beyond Reduced Diversity

Lowering the temperature concentrates the sampler, so the low-temperature gain of BCT-L could merely trade sample diversity for single-sample accuracy ([8]), which a less concentrated sampler would recover with more samples. We test this on S-arch by comparing BCT-L with standard tempering on Cont. in Pass@K for K∈{1,2,4,8}K \in\{1, 2, 4, 8\}, estimated from eight generations per problem with the unbiased estimator of Chen et al. [8] (Figure 3; details in Appendix § 12).

Pass@1, Pass@2, Pass@4, and Pass@8 on GSM8K

Figure 3. Pass@KK on GSM8K (S-arch, NFE = 1024), estimated from eight generations per problem. Bands are 95% bootstrap intervals over problems; the panels share the vertical axis.

For every T<1T < 1 and K≤8K \le8, BCT-L has the higher Pass@K, and the 95% paired bootstrap interval of the difference lies above zero. More samples raise Pass@K for both samplers, but not enough for standard tempering to catch up. The low-temperature gain of BCT-L is therefore not explained by reduced diversity alone.

Beyond GSM8K: Sudoku-Hard

Sudoku-Hard tests whether our conclusions carry over to tightly constrained generation, in which each puzzle has a single valid completion. Because no Sudoku checkpoint is released, we train the base model from scratch and obtain TFT by continuing it; as in § 4.2, the fair baseline for BCT-L is therefore Cont. (Figure 4). Following the released configuration of S-FLM [12], we sample with 180 NFE.

Sudoku-Hard exact solve rate

Figure 4. Sudoku-Hard exact solve rate (NFE = 180): mean and ± one sample standard deviation over three generation seeds; base-checkpoint results are in Table 5.

TTBase checkpointCont.TFT
StandardBCT-EBCT-PStandardBCT-EBCT-PBCT-L
142.9 ± 1.542.9 ± 1.642.9 ± 1.541.7 ± 1.241.7 ± 1.241.7 ± 1.241.7 ± 1.1
0.842.6 ± 1.643.7 ± 1.443.6 ± 1.041.6 ± 1.242.6 ± 1.242.4 ± 1.042.7 ± 0.7
0.642.3 ± 1.344.1 ± 1.244.2 ± 1.141.4 ± 1.143.1 ± 1.042.7 ± 1.244.3 ± 1.2
0.542.2 ± 1.344.4 ± 1.344.4 ± 1.241.3 ± 1.043.3 ± 1.243.0 ± 0.845.1 ± 1.3
0.442.1 ± 1.244.8 ± 1.444.8 ± 1.141.2 ± 1.043.5 ± 1.243.2 ± 0.545.6 ± 1.3
0.342.0 ± 1.245.0 ± 1.444.7 ± 0.941.2 ± 1.043.7 ± 1.243.3 ± 0.545.9 ± 1.1
0.241.9 ± 1.345.2 ± 1.444.8 ± 0.441.2 ± 1.044.0 ± 1.243.6 ± 0.146.4 ± 1.1
0.141.7 ± 1.345.5 ± 1.444.9 ± 0.441.0 ± 1.044.2 ± 1.443.7 ± 0.546.8 ± 1.2

Table 5. Sudoku-Hard exact solve rate (%), mean ± standard deviation over generation seeds 1, 11 and 21 (2,000 test puzzles, NFE = 180). BCT-L uses the held-out λ=1\lambda= 1. The seed standard deviation is dominated by a shift shared by all samplers.

Lowering the temperature slightly hurts standard tempering, whereas BCT-L improves monotonically to 46.8%. The training-free variants also benefit from lower temperatures, and on this task BCT-P does not collapse; still, BCT-L is the most accurate at every T≤0.6T \le0.6, with a margin that widens as TT decreases, and the base checkpoint shows the same pattern (Appendix § 14).

Beyond the Geodesic Bridge: von Mises–Fisher Flows

We finally apply BCT to SFCD [7], whose von Mises–Fisher bridge replaces the geodesic bridge of S-FLM (§ 3.3). We use its released TinyGSM checkpoint and sampler with NFE = 1024, and updates for TFT and Cont., and average over five generation seeds (Appendix § 15). Unlike S-FLM, this flow already benefits from standard tempering: lowering the temperature raises its accuracy by 9 points between T=1T = 1 and T=0.3T = 0.3 (Figure 5), suggesting that inconsistent tempering harms the vMF bridge less. We therefore ask whether consistent tempering still improves a model whose standard tempering is already effective.

GSM8K accuracy of BCT on the vMF flow LM

Figure 5. GSM8K accuracy of BCT on the vMF flow LM (NFE = 1024): mean and ± one sample standard deviation over five generation seeds. BCT-E falls below the plotted range for T≤0.3T \le0.3; more detailed results are in Table 7.

It does, mainly at moderate temperatures. BCT-L improves on standard tempering at every T∈[0.2,0.6]T \in[0.2, 0.6] and attains the highest accuracy among the evaluated samplers, 34.5±0.5%34.5 \pm0.5\% at T=0.3T = 0.3, against 32.6±0.6%32.6 \pm0.6\% for standard tempering, a margin well beyond the seed standard deviations.

Under strong tempering, the learned realization remains stable while the training-free variants fail. BCT-L declines at the lowest temperature, a trend absent on S-FLM, but only to the level of standard tempering (31.7% vs. 31.3% at T=0.1T = 0.1). Both training-free variants instead fall below it: BCT-P as on S-FLM (28.5%), and BCT-E, which is stable on S-FLM, collapses to essentially zero accuracy for T≤0.2T \le0.2, likely because on this bridge its correction −(β−1)κt⟨ex,z⟩-(\beta- 1)\kappa_t\langle e_x,z\rangle amplifies any mismatch between the learned logits and the analytic bridge term by β\beta and κt\kappa_t. BCT-L is thus the only variant that never falls below standard tempering for T<1T < 1: consistent tempering improves even a flow that already responds well to temperature, and training on the tempered bridge keeps it reliable under strong tempering.

Related Work

Posterior-induced flow language models. Non-autoregressive language models include discrete diffusion [3, 26, 31, 33, 32, 2, 27] and continuous diffusion and flow models, which map tokens to continuous representations such as learned embeddings or one-hot vectors [24, 13, 15, 21, 18]. In many continuous models, a denoiser trained with cross-entropy predicts the clean token, and its posterior weights conditional velocities, drifts, or scores. Among spherical constructions [11, 19], S-FLM moves latents along geodesics of the hypersphere [12, 9], whereas SFCD uses von Mises–Fisher bridges [7]; we instantiate BCT for both.

Temperature and consistency. In autoregressive models and discrete diffusion, temperature rescales the categorical distribution from which tokens are sampled [1, 17], and lowering it substantially improves discrete diffusion language models on GSM8K [12]; we reproduce this in Appendix § 13. Continuous diffusion and flow models instead temper the score or the token posterior [13, 15, 12, 4], and for continuous data tempering is known not to commute with noising: the scaled score is not the score of the noised tempered distribution [14]. Work on this inconsistency targets other settings: Feynman–Kac correctors reweight and resample populations of particles along annealed marginals [34, 16], Li et al. [23] correct score scaling under Gaussian noising for flattening (T>1T > 1), and temporal score rescaling controls a local temperature that leaves the mixture weights unchanged [36]. None of them addresses flows induced by a token posterior over a finite vocabulary, where tempering the posterior implicitly changes the conditional bridge while leaving the conditional velocities untempered. BCT corrects this through the bridge itself, along a single sampling trajectory without particles, and extends the correction to training.

Train–inference mismatch. Models evaluated on states that differ from those seen in training suffer from exposure bias, both in autoregressive models [30, 6] and in diffusion models, where it is reduced by perturbing training inputs or shifting sampling time steps [28, 22]. There, the mismatch arises from the model’s own prediction errors. Under tempering, it is instead introduced by a known change of the bridge, so TFT can draw its training states exactly from the tempered bridge that the sampler follows.

Conclusion

We introduced Bayes-Consistent Tempering (BCT), which treats temperature as part of the generative flow rather than as a post-hoc transformation of the denoising posterior. BCT aligns the tempered endpoint distribution, intermediate-state distribution, training distribution, and sampling dynamics; its training-free variants recover parts of this consistency for conventionally trained checkpoints. Across hyperspherical flow LMs with two denoiser architectures and the von Mises–Fisher flow LM, BCT improves GSM8K generation over broad temperature ranges, while its learned realization remains robust under aggressive tempering where the post-hoc variants deteriorate. The gains also extend to Sudoku, suggesting that BCT’s benefits extend to tightly constrained generation. Overall, our results indicate that temperature in posterior-induced flow language models is better treated as part of the flow itself, across both training and inference.

Limitations and Discussion

BCT applies to posterior-induced flows for which the tempered bridge and its corresponding sampling dynamics can be derived or approximated; the concrete derivations in this work cover the hyperspherical bridge of S-FLM and the von Mises–Fisher bridge of SFCD. Extending BCT to other bridge families is an important direction for future work.

Acknowledgments

This research was conducted by AIST and Institute of Science Tokyo under FRONTia, a Japanese national program led by the Ministry of Economy, Trade and Industry of Japan (METI) and the New Energy and Industrial Technology Development Organization (NEDO), aimed at developing a domestic multimodal foundation model for AI robots and physical AI. This work was also supported in part by JSPS KAKENHI Grant Number 25H01137 and JST K Program Japan Grant Number JPMJKP24C3.

Sequences

For a sequence x=(x1,…,xL)x=(x^1,\ldots,x^L) with joint endpoint distribution π(x)\pi(x) and a bridge that factorizes over positions, the posterior at position ll marginalizes the other positions:

rt(xl∣z)=qt(zl∣xl)π~t(xl;z−l)∑yqt(zl∣y)π~t(y;z−l),π~t(xl;z−l)=∑x−lπ(x)∏j≠lqt(zj∣xj).(19)r_t(x^l\mid z)=\frac{q_t(z^l\mid x^l)\tilde{\pi}_t(x^l;z^{-l})}{\sum_y q_t(z^l\mid y)\tilde{\pi}_t(y;z^{-l})},\qquad\tilde{\pi}_t(x^l;z^{-l})=\sum_{x^{-l}}\pi(x)\prod_{j\ne l}q_t(z^j\mid x^j). \tag*{(19)}

This is Eq. (1) at position ll with π\pi replaced by the context-conditioned endpoint π~t(⋅;z−l)\tilde{\pi}_t(\cdot;z^{-l}), which does not depend on zlz^l.

Every construction of § 3 therefore applies position-wise and tempers π~t\tilde{\pi}_t; under TFT, π~t\tilde{\pi}_t is formed with the tempered bridges qt,Tq_{t,T} at the other positions. For any fixed endpoint sequence (x)(x), the product of the position-wise conditional bridges is transported by the concatenation of their corresponding conditional velocities, so the joint conditional continuity equation holds by the product rule. None of them tempers the joint endpoint: that would require the context factor ∑x−lπ(x)β∏j≠lqt′(zj∣xj)\sum_{x^{-l}}\pi(x)^\beta\prod_{j\ne l}q'_t(z^j\mid x^j), with qt′q'_t the bridge of the target pair, which in general is not proportional to the power of π~\tilde{\pi} that every variant uses. Per-token temperature in autoregressive models has the same property.

The hyperspherical bridge

Throughout, θ=θx(z)\theta=\theta_x(z) is the polar angle about exe_x, ν=d−2\nu=d-2, σ\sigma is the surface measure on Sd−1\mathbb{S}^{d-1}, and a=at∈(0,1]a=a_t\in(0,1], which equals 1 at the source and decreases in tt; the bridge depends on tt only through aa, and the results below hold for every a∈(0,1]a\in(0,1].

Bridge density

S-FLM draws z0z_0 uniformly on Sd−1\mathbb{S}^{d-1} and sets zt=SLERP⁡(z0,ex;αt)z_t=\operatorname{SLERP}(z_0,e_x;\alpha_t). About exe_x, write z=cos⁡θ ex+sin⁡θ ξz=\cos\theta\,e_x+\sin\theta\,\xi with ξ\xi in the unit sphere Sd−2\mathbb{S}^{d-2} of the orthogonal complement of exe_x. The surface measure factorizes as

dσ(z)=sin⁡νθ dθ dσd−2(ξ).(20)d\sigma(z)=\sin^\nu\theta\,d\theta\,d\sigma_{d-2}(\xi). \tag*{(20)}

Let ω=θx(z0)\omega=\theta_x(z_0). Then SLERP⁡(z0,ex;α)=[sin⁡(aω)z0+sin⁡((1−a)ω)ex]/sin⁡ω\operatorname{SLERP}(z_0,e_x;\alpha)=[\sin(a\omega)z_0+\sin((1-a)\omega)e_x]/\sin\omega, and substituting z0=cos⁡ω ex+sin⁡ω ξz_0=\cos\omega\,e_x+\sin\omega\,\xi gives zt=cos⁡(aω)ex+sin⁡(aω)ξz_t=\cos(a\omega)e_x+\sin(a\omega)\xi: SLERP keeps ξ\xi and contracts the polar angle by the factor aa. For uniform z0z_0, (ω,ξ)(\omega,\xi) has density sin⁡νω/∣Sd−1∣\sin^\nu\omega/\lvert\mathbb{S}^{d-1}\rvert with respect to dω dσd−2(ξ)d\omega\,d\sigma_{d-2}(\xi). The map ω↦aω\omega\mapsto a\omega has Jacobian aa, and dividing by the factor sin⁡νθ\sin^\nu\theta of Eq. (20) gives

qt(z∣x)=sin⁡ν(θ/a)a∣Sd−1∣sin⁡νθ1{θ<aπ},(21)q_t(z\mid x)=\frac{\sin^\nu(\theta/a)}{a\lvert\mathbb{S}^{d-1}\rvert\sin^\nu\theta}\mathbf{1}\{\theta<a\pi\}, \tag*{(21)}

which is Eq. (15).

Tempered bridge

By Eq. (20), under qt,T(⋅∣x)∝qt(⋅∣x)βq_{t,T}(\cdot\mid x)\propto q_t(\cdot\mid x)^\beta the direction ξ\xi is uniform on Sd−2\mathbb{S}^{d-2} and independent of the polar angle, whose marginal is

ma,T(θ)=ga,T(θ)Ca,T,ga,T(θ)=sin⁡νβ(θa)sin⁡ν(1−β)θ,θ∈(0,aπ),(22)m_{a,T}(\theta)=\frac{g_{a,T}(\theta)}{C_{a,T}},\qquad g_{a,T}(\theta)=\sin^{\nu\beta}\left(\frac{\theta}{a}\right)\sin^{\nu(1-\beta)}\theta,\qquad\theta\in(0,a\pi), \tag*{(22)}

with Ca,T=∫0aπga,TC_{a,T}=\int_0^{a\pi}g_{a,T}. Rotations preserve σ\sigma and map the bridges of different tokens onto one another, so Zt,TZ_{t,T} depends only on (a,T,d)(a,T,d). Since ga,Tg_{a,T} behaves like θνβ\theta^{\nu\beta} as θ→0\theta\to0 and like (aπ−θ)νβ(a\pi-\theta)^{\nu\beta} as θ→aπ\theta\to a\pi (a<1a<1), Zt,T<∞Z_{t,T}<\infty and qt,Tq_{t,T} is a valid density for every T>0T>0. Its support does not depend on TT. At a=1a=1, g1,T=sin⁡νθg_{1,T}=\sin^\nu\theta for every TT, so the tempered source q0,Tq_{0,T} is the uniform source of S-FLM.

Tempered conditional velocity

We write the polar dynamics as dθ/da=va,T(θ)\mathrm{d}\theta/\mathrm{d}a = v_{a,T}(\theta).

Symmetry reduction. qt,T(⋅∣x)q_{t,T}(\cdot\mid x) is invariant under the orthogonal transformations fixing exe_x, and we take its conditional velocity to be equivariant, u(Rz)=Ru(z)u(Rz) = Ru(z) for all such RR. The transformations fixing both exe_x and zz therefore fix u(z)u(z), and for d≥3d \ge3 the only tangent vectors at zz they fix are multiples of eθe_\theta, the unit tangent pointing away from exe_x along the geodesic. Equivariance under transformations that move ξ\xi makes the magnitude a function of θ\theta alone, so u=θ˙eθu = \dot{\theta}e_\theta.

One-dimensional continuity equation. For a density ρ\rho with respect to σ\sigma, ∇⋅(ρveθ)=sin⁡−νθ ∂θ(sin⁡νθ ρv)\nabla\cdot(\rho v e_\theta) = \sin^{-\nu}\theta\,\partial_\theta(\sin^\nu\theta\,\rho v). Multiplying ∂aρ+∇⋅(ρveθ)=0\partial_a\rho+ \nabla\cdot(\rho v e_\theta) = 0 by ∣Sd−2∣sin⁡νθ|\mathbb{S}^{d-2}|\sin^\nu\theta, which turns ρ\rho into the polar marginal ma,Tm_{a,T}, gives

∂ama,T(θ)+∂θ(ma,T(θ)va,T(θ))=0.(23)\partial_a m_{a,T}(\theta) + \partial_\theta\left(m_{a,T}(\theta)v_{a,T}(\theta)\right) = 0. \tag*{(23)}

With no flux through the pole (mv→0mv \to0 as θ→0\theta\to0), integrating over (0,θ)(0,\theta) gives the unique solution va,T=−∂aMa,T/ma,Tv_{a,T} = -\partial_a M_{a,T}/m_{a,T}, where Ma,T(θ)=∫0θma,TM_{a,T}(\theta) = \int_0^\theta m_{a,T}.

Closed form. From Eq. (22), ∂aga,T(θ)=−(νβ/a2)θcot⁡(θ/a) ga,T(θ)\partial_a g_{a,T}(\theta) = -(\nu\beta/a^2)\theta\cot(\theta/a)\,g_{a,T}(\theta). Since ga,T(aπ)=0g_{a,T}(a\pi) = 0, differentiating Ca,TC_{a,T} produces no boundary term, and ∂aCa,T=−(νβ/a2)H‾a,TCa,T\partial_a C_{a,T} = -(\nu\beta/a^2)\overline{H}_{a,T}C_{a,T} with H‾a,T=Eθ∼ma,T[θcot⁡(θ/a)]\overline{H}_{a,T} = \mathbb{E}_{\theta\sim m_{a,T}}[\theta\cot(\theta/a)]. Hence

va,T(θ)=νβa2∫0θ[ϑcot⁡(ϑ/a)−H‾a,T] ga,T(ϑ) dϑga,T(θ).(24)v_{a,T}(\theta) = \frac{\nu\beta}{a^2}\frac{\int_0^\theta[\vartheta\cot(\vartheta/a)-\overline{H}_{a,T}]\,g_{a,T}(\vartheta)\,\mathrm{d}\vartheta}{g_{a,T}(\theta)}. \tag*{(24)}

Velocity field. As tt increases, da/dt=−α˙t\mathrm{d}a/\mathrm{d}t = -\dot{\alpha}_t, so z˙=−α˙tva,T(θ)eθ\dot{z} = -\dot{\alpha}_t v_{a,T}(\theta)e_\theta. Since Log⁡z(ex)=−θeθ\operatorname{Log}_z(e_x) = -\theta e_\theta, this equals

ut,T(z∣x)=wt,T(θ)ut(z∣x),wt,T(θ)=atvat,T(θ)θ,ut(z∣x)=α˙tatLog⁡z(ex),u_{t,T}(z \mid x) = w_{t,T}(\theta)u_t(z \mid x), \qquad w_{t,T}(\theta) = \frac{a_t v_{a_t,T}(\theta)}{\theta}, \qquad u_t(z \mid x) = \frac{\dot{\alpha}_t}{a_t}\operatorname{Log}_z(e_x),

which is Eq. (16). At T=1T = 1, integration by parts with s=θ/as = \theta/a gives ∫0πscos⁡ssin⁡ν−1s ds=−1ν∫0πsin⁡νs ds\int_0^\pi s\cos s\sin^{\nu-1}s\,\mathrm{d}s = -\frac{1}{\nu}\int_0^\pi\sin^\nu s\,\mathrm{d}s, hence H‾a,1=−a/ν\overline{H}_{a,1} = -a/\nu, and ddθ[θaga,1(θ)]=νa2[θcot⁡(θ/a)−H‾a,1]ga,1(θ)\frac{\mathrm{d}}{\mathrm{d}\theta}\left[\frac{\theta}{a}g_{a,1}(\theta)\right] = \frac{\nu}{a^2}[\theta\cot(\theta/a)-\overline{H}_{a,1}]g_{a,1}(\theta). The integral in Eq. (24) thus equals aθθga,1(θ)\frac{a}{\theta}\theta g_{a,1}(\theta), so va,1=θ/av_{a,1} = \theta/a and wt,1≡1w_{t,1} \equiv1: utu_t is the S-FLM conditional velocity, and the tempered construction reduces to it at T=1T = 1.

Exact sampling of the tempered bridge

Use the normalized angles ψ=ω/π\psi= \omega/\pi and ϕ=θ/(aπ)\phi= \theta/(a\pi) in (0,1)(0,1). Let FsrcF_{\mathrm{src}} be the CDF of the density ∝sin⁡ν(πψ)\propto\sin^\nu(\pi\psi), and Fa,TF_{a,T} the CDF of the density ∝sin⁡νβ(πϕ)sin⁡ν(1−β)(aπϕ)\propto\sin^{\nu\beta}(\pi\phi)\sin^{\nu(1-\beta)}(a\pi\phi), which is Eq. (22) in the variable ϕ\phi. Define

Qa,T(ω)=aπFa,T−1(Fsrc(ω/π)).(25)Q_{a,T}(\omega) = a\pi F_{a,T}^{-1}\left(F_{\mathrm{src}}(\omega/\pi)\right). \tag*{(25)}

Under both the source and the tempered bridge, ξ\xi is uniform on Sd−2\mathbb{S}^{d-2} and independent of the polar angle. Hence z=cos⁡θ ex+sin⁡θ ξz = \cos\theta\,e_x + \sin\theta\,\xi with θ=Qa,T(ω)\theta= Q_{a,T}(\omega) is an exact sample of qt,T(⋅∣x)q_{t,T}(\cdot\mid x). At β=1\beta= 1, Fa,1=FsrcF_{a,1} = F_{\mathrm{src}} and θ=aω\theta= a\omega, which reproduces SLERP for the same z0z_0. Equivalently, z=SLERP⁡(z0,ex;1−θ/ω)z = \operatorname{SLERP}(z_0,e_x;1-\theta/\omega), a SLERP with a position-specific parameter. For TFT we build the target CDF for each (at,T)(a_t,T) pair in the batch on a grid refined geometrically toward both ends of [0,atπ][0,a_t\pi] and invert it at the drawn angles, with no interpolation table over (a,T)(a,T).

Held-out selection of λ\lambda

Held-out set. The TinyGSM data used by S-FLM are split into a training part, which the TFT continuation reads, and a validation part of 118,216 problems, which it does not. From a random subset of 768 validation problems, we remove 11 prompts that also occur verbatim in the training part and 30 problems without a reliably recoverable reference answer (seven fall in both groups), leaving 634 problems, none of which matches a GSM8K test problem. The reference answer of each problem is the value returned by its TinyGSM reference program, and generated programs are graded as on GSM8K.

Protocol. For each backbone, the sweep uses the TFT checkpoint, generation seed 1 and the GSM8K sampling protocol. Each λ∈{0,0.25,0.5,0.75,1}\lambda\in\{0, 0.25, 0.5, 0.75, 1\} is scored by its held-out accuracy averaged over T∈{0.4,0.2,0.1}T \in\{0.4, 0.2, 0.1\}; the maximizer, with ties broken toward the larger λ\lambda, is used at every temperature. This selects λ=0.5\lambda= 0.5 for S-arch and λ=0.25\lambda= 0.25 for DiT (Figure 6, Table 2).

Held-out TinyGSM accuracy as a function of correction strength for S-arch and DiT

Figure 6. Held-out selection of λ\lambda for BCT-L (634 held-out TinyGSM problems, TFT, seed 1). Thin lines: accuracy at each selection temperature; thick line: their mean, the selection score. The dotted line marks the selected λ\lambda. λ=0\lambda= 0 applies no Bayes correction and λ=1\lambda= 1 the full correction. Values in Table 2.

λ\lambda\multicolumn{4}{c}{S-arch}\multicolumn{4}{c}{DiT}DiT (500 upd.)
T=0.4T=0.40.20.20.10.1ScoreT=0.4T=0.40.20.20.10.1ScoreScore
027928528244.4827127828743.9542.27
0.2527929229045.2727829531646.7448.74
0.528330430346.7928129929546.0048.32
0.7528430529546.4828229525743.8545.79
128630028946.0027927918138.8543.22

Table 2. Held-out λ\lambda sweep: correct answers out of 634 at each selection temperature and the selection score (mean accuracy over the three temperatures, %); bold: selected value. The last column is a DiT TFT checkpoint after 500 updates, not used for selection.

Pass@K on S-arch

This appendix gives the setup and values behind § 4.3.

Setup. We compare Cont. with standard tempering and TFT with BCT-L (λ=0.5\lambda= 0.5) on S-arch, using the checkpoints, sampler, prompts and grader of Table 1 at the same eight temperatures. The sampler draws one generation per problem, so the eight generations per problem come from eight runs with generation seeds 1+10j1 + 10j, j=0,…,7j = 0, \ldots, 7; the first three are the seeds of Table 1 and reproduce its generations exactly. For a problem with cc correct generations out of n=8n = 8, we use the unbiased estimator 1−(n−ck)(nk)1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} ([8]), averaged over the 1319 problems. Intervals are 95% percentile bootstrap intervals over problems (10,000 resamples), paired for differences.

TTCont. standard tempering K=1K = 1248TFT BCT-L K=1K = 1248
TTK=1K = 1248K=1K = 1248
112.319.628.337.512.720.229.037.7
0.814.422.532.142.016.525.134.744.5
0.615.824.334.144.520.229.840.351.3
0.516.925.836.046.821.831.341.251.1
0.417.426.336.747.523.332.942.852.5
0.317.626.736.947.324.433.743.252.5
0.217.726.636.647.225.334.543.451.9
0.117.826.837.047.527.135.743.951.1

Table 3. Pass@K (%) of Figure 3: unbiased estimator over n=8n = 8 generations per problem (generation seeds 1, 11, …\ldots, 71), S-arch, 1319 GSM8K problems. Pass@1 is the mean single-generation accuracy over the eight draws and agrees with the three-seed mean of Table 1 to within 0.6 points.

Differences. For every T<1T < 1 and K≤8K \le8, the paired interval of the difference (BCT-L minus standard tempering) lies above zero. At T=0.1T = 0.1 the difference is 9.2 [8.1, 10.3] points at K=1K = 1 and 3.6 [1.4, 5.8] at K=8K = 8. The number of problems solved by at least one of the eight generations is 693 and 674 for BCT-L at T=0.3T = 0.3 and 0.1, against 624 and 627 for standard tempering.

Standard temperature scaling on discrete diffusion baselines

We sweep the sampling temperature of the MDLM [31] and Duo [32] TinyGSM checkpoints released by Deschenaux & Gulcehre [12], without any training or tuning. Both use the ancestral samplers of that release: at every step the denoiser logits are divided by TT before the softmax, and the tempered clean-token distribution enters the unchanged reverse transition (absorbing state for MDLM, uniform state for Duo). At T=1T = 1 this reproduces the released sampler exactly (1319 of 1319 identical programs for both models, checked with 32 steps). We use 1024 steps, generation seed 1 and the GSM8K evaluation of § 4.1; the S-FLM values are the standard-tempering results at seed 1.

At T=0.1T = 0.1, MDLM and Duo solve 32.7% and 36.2%, in line with the 33% and 36% reported by Deschenaux & Gulcehre [12]; at T=1T = 1 they solve 15.0% and 18.6%, against the reported 18.0% and 17.2%.

Sudoku-Hard

This appendix gives the setup and values behind § 4.4.

Task. We use the Sudoku-Hard task and puzzle generator of S-FLM ([12]) unchanged. Every puzzle has 30 clues and a unique solution (checked with a solution counter), and the 48,000 training and 2,000 test puzzles share no puzzle or solution. A sequence is the puzzle followed by its solution, each written as 81 digits with row separators (a vocabulary of 12 tokens). The puzzle is a fixed prefix during sampling, and a test puzzle counts as solved only if the generated solution equals the reference.

Model and training. We train with the released Sudoku recipe of S-FLM unchanged: a DiT denoiser with 8 blocks and hidden size 512, the geodesic bridge with its truncated adaptive noise schedule, 20,000 updates with a batch of 256 puzzles on one GPU, AdamW with a constant learning rate of 3×10−43 \times10^{-4} after warmup, EMA decay 0.9999, and training seed 1. Our code reproduces the released code bit for bit on this recipe (weights, EMA weights and optimizer state after 1,200 updates). We call the result the base checkpoint. Cont. and TFT resume its complete training state for 2,000 updates; TFT uses the TFT setting of § 4.1 with TT-conditioning. Each checkpoint is a single training run.

Evaluation. All samplers use the released Sudoku sampler (180 NFE, exact velocity, argmax decoding, EMA weights, batches of 64 puzzles on one GPU) with generation seeds 1, 11 and 21; only the temperature and the BCT rule change between runs, and for a given seed all samplers start from the same noise. At T=1T = 1 the training-free samplers coincide up to floating-point differences, which change the number of solved puzzles by at most 2 per seed.

Selection of λ\lambda. We select λ\lambda with the protocol of Appendix § 11 on 2,000 new puzzles from the same generator with a different seed, disjoint from the training and test puzzles. Held-out accuracy increases with λ\lambda at every selection temperature (Table 6), so λ=1\lambda= 1.

λ\lambdaT=0.4T = 0.4T=0.2T = 0.2T=0.1T = 0.1Mean
041.840.840.140.9
0.2542.843.543.343.2
0.544.044.544.944.5
0.7544.945.446.945.7
145.646.347.546.5

Table 6. Held-out selection of λ\lambda for BCT-L on Sudoku-Hard: exact solve rate (%) of TFT on 2,000 held-out puzzles (generation seed 1, NFE = 180). Mean: the selection score. Bold: selected. No test puzzle was used.

BCT for vMF flows

This appendix gives the instantiation and setup of § 4.5. SFCD [7] keeps learned token embeddings on the unit sphere but uses the von Mises–Fisher bridge qt(z∣x)=vMF⁡(ex,κt)q_t(z \mid x) = \operatorname{vMF}(e_x, \kappa_t) with increasing concentration and κ0=0\kappa_0 = 0. Its sampler integrates the posterior-induced velocity Eq. (2) on a time warp chosen by the release. We use the released TinyGSM checkpoint (250k updates) with its architecture, warp and sampler unchanged, and change only the velocity.

Tempering the vMF bridge. The tempered bridge is vMF⁡(ex,κt/T)\operatorname{vMF}(e_x, \kappa_t/T) Eq. (18), whose normalizer does not depend on xx, so BCT-P keeps the posterior weights of standard tempering. Because the vMF bridge is indexed by its concentration, the released conditional velocity has the form ut(z∣x)=κ˙tVκt(z∣x)u_t(z \mid x) = \dot{\kappa}_t V_{\kappa_t}(z \mid x), where Vκ(⋅∣x)V_\kappa(\cdot\mid x) transports vMF⁡(ex,κ)\operatorname{vMF}(e_x, \kappa) as κ\kappa increases. The tempered bridge is the same family at concentration κt/T\kappa_t/T, so its conditional velocity is

ut,T(z∣x)=κ˙tTVκt/T(z∣x)(26)u_{t,T}(z \mid x) = \frac{\dot{\kappa}_t}{T} V_{\kappa_t/T}(z \mid x) \tag*{(26)}

the released velocity evaluated at concentration κt/T\kappa_t/T, with the concentration advancing 1/T1/T times as fast. BCT-P and BCT-L integrate this velocity on the warp grid of the release. BCT-E keeps the original bridge and dynamics and, since log⁡qt(z∣x)=κt⟨ex,z⟩\log q_t(z \mid x) = \kappa_t\langle e_x, z\rangle up to a constant in xx, adds (1−β)κt⟨ex,z⟩(1-\beta)\kappa_t\langle e_x,z\rangle to the tempered logits. At T=1T = 1 every variant reduces to the released sampler.

TFT and BCT-L. TFT uses the parameterization of § 3.3: the network outputs a remainder aθa_\theta, the posterior is softmax(aθ+g)(a_\theta+ g) with the analytic term gx=(κt/T)⟨ex,z⟩g_x = (\kappa_t/T)\langle e_x,z\rangle, and training uses the cross-entropy of Eq. (6). The temperature enters the network through κt/(Tκmax⁡)\kappa_t/(T\kappa_{\max}). BCT-L Eq. (9) then takes the form ℓx=βaθ,x+[β+λ(1−β)]gx\ell_x = \beta a_{\theta,x} + [\beta+ \lambda(1-\beta)]g_x.

Continued training. Both continuations resume the released checkpoint for 20,000 updates: Cont. with the original recipe (with the warp frozen, as in TFT) and TFT with the temperature distribution of § 4.1 and latents drawn from qt,Tq_{t,T}. Both are evaluated with an EMA accumulated over the continuation (decay 0.9999); the released checkpoint uses its released EMA weights. Each continuation is a single training run.

Evaluation. We use the released SFCD evaluation of the 1319 GSM8K test problems (warp-aware ODE sampler, NFE = 1024, batches of 32, and its own answer checker) with generation seeds 42–46; every cell of Table 7 is complete over the five seeds. The separate T=1T = 1 runs of BCT-E on the released checkpoint reproduce standard tempering exactly.

TTReleasedCont.TFT
StandardBCT-EBCT-PStandardBCT-EBCT-PBCT-L
123.55 ± 0.7923.55 ± 0.7923.55 ± 0.7923.59 ± 0.7123.59 ± 0.7123.59 ± 0.7121.55 ± 0.86
0.827.79 ± 0.8329.19 ± 1.4329.39 ± 0.4827.99 ± 0.2729.80 ± 0.7529.28 ± 0.7428.19 ± 0.97
0.630.34 ± 0.6331.71 ± 0.6431.63 ± 0.7730.61 ± 0.7731.54 ± 0.7832.43 ± 0.4532.05 ± 0.47
0.530.80 ± 0.8031.40 ± 0.6232.36 ± 1.0331.80 ± 0.7731.21 ± 0.4733.28 ± 0.3933.54 ± 0.63
0.431.19 ± 0.9928.26 ± 0.5233.39 ± 0.9032.57 ± 0.8326.37 ± 0.3233.93 ± 0.5834.33 ± 0.97
0.331.75 ± 0.9115.30 ± 0.6733.63 ± 0.6932.62 ± 0.612.93 ± 0.2533.54 ± 0.7534.47 ± 0.55
0.231.71 ± 0.400.08 ± 0.1132.05 ± 0.6732.16 ± 0.400.02 ± 0.0332.24 ± 0.2833.24 ± 1.12
0.130.99 ± 0.550.00 ± 0.0026.63 ± 0.8531.25 ± 0.420.00 ± 0.0028.49 ± 0.5431.68 ± 0.53

Table 7. BCT on SFCD: GSM8K accuracy (%), mean ± sample standard deviation over generation seeds 42–46 (1319 problems, NFE = 1024). Released: the SFCD TinyGSM checkpoint with its released EMA weights. Cont. and TFT: 20,000 updates from it, evaluated with an adaptation-local EMA. BCT-L uses the held-out λ=1\lambda= 1. At T=1T = 1 every correction is the identity, so the three columns of a model coincide.

Selection of λ\lambda. We select λ\lambda with the protocol of Appendix § 11 on 1024 TinyGSM validation problems that the evaluation did not read and that share no problem with the GSM8K test set. The score increases with λ\lambda (51.92, 53.19, 53.87, 54.07 and 54.39% for λ=0,0.25,0.5,0.75,1\lambda= 0, 0.25, 0.5, 0.75, 1), so λ=1\lambda= 1.

Standard temperature scaling on the TinyGSM baselines

Figure 7. Standard temperature scaling on the TinyGSM baselines released with S-FLM (NFE = 1024, seed 1). Gray: S-FLM (§ 4.2). Values in Table 4.

TTMDLMDuoS-FLM (S-arch)S-FLM (DiT)
115.0 (198)18.6 (245)13.0 (172)11.7 (154)
0.822.5 (297)23.0 (304)15.0 (198)13.9 (183)
0.628.5 (376)28.3 (373)15.8 (208)16.3 (215)
0.529.3 (386)29.5 (389)16.5 (218)17.3 (228)
0.430.2 (398)32.1 (423)17.5 (231)17.0 (224)
0.332.6 (430)34.0 (448)18.0 (238)16.9 (223)
0.233.7 (445)35.3 (465)18.0 (237)17.1 (226)
0.132.7 (431)36.2 (477)17.3 (228)16.9 (223)

Table 4. Accuracy (%) and correct answers out of 1319 for Figure 7. S-FLM uses the exact velocity.

References

  1. [1]David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for Boltz-mann machines. Cognitive Science, 9(1):147–169, 1985.
  2. [2]Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub-ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore-gressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025.
  3. [3]Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Struc-tured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2107.03006
  4. [4]Iskander Azangulov and Leo Zhang. Sampling from flow language models via marginal-conditioned bridges. arXiv preprint arXiv:2605.13681, 2026.arxiv.org/abs/2605.13681
  5. [5]Heli Ben-Hamu, Itai Gat, Daniel Severo, Niklas Nolte, and Brian Karrer. Accelerated sampling from masked diffusion models via entropy bounded unmasking, 2025. URL https://arxiv.org/abs/2505.24857.
  6. [6]Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Sys-tems, 2015.arxiv.org/abs/1506.03099
  7. [7]Jannis Chemseddine, Gregor Kornhardt, and Gabriele Steidl. Spherical flows for sampling categor-ical data. arXiv preprint arXiv:2605.05629, 2026.arxiv.org/abs/2605.05629
  8. [8]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  9. [9]Ricky T. Q. Chen and Yaron Lipman. Flow matching on general geometries. In The Twelfth Inter-national Conference on Learning Representations, 2024.arxiv.org/abs/2302.03660
  10. [10]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  11. [11]Oscar Davis, Samuel Kessler, Mircea Petrache, İsmail İlkan Ceylan, Michael Bronstein, and Avishek Joey Bose. Fisher flow matching for generative modeling over discrete data. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2405.14664
  12. [12]Justin Deschenaux and Caglar Gulcehre. Language modeling with hyperspherical flows. arXiv preprint arXiv:2605.11125, 2026.arxiv.org/abs/2605.11125
  13. [13]Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089, 2022.arxiv.org/abs/2211.15089
  14. [14]Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and MCMC. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 8489–8510, 2023.arxiv.org/abs/2302.11552
  15. [15]Ishaan Gulrajani and Tatsunori B. Hashimoto. Likelihood-based diffusion language models. In Advances in Neural Information Processing Systems, volume 36, 2023.arxiv.org/abs/2305.18619
  16. [16]Mohsin Hasan, Viktor Ohanesian, Artem Gazizov, Yoshua Bengio, Alán Aspuru-Guzik, Roberto Bondesan, Marta Skreta, and Kirill Neklyudov. Discrete Feynman-Kac correctors. arXiv preprint arXiv:2601.10403, 2026.arxiv.org/abs/2601.10403
  17. [17]Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
  18. [18]Keya Hu, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li, Yoon Kim, Jacob Andreas, and Kaiming He. Embedded language flows. arXiv preprint arXiv:2605.10938, 2026.arxiv.org/abs/2605.10938
  19. [19]Jaehyeong Jo and Sung Ju Hwang. Continuous diffusion model for language modeling. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2502.11564
  20. [20]Jaeyeon Kim, Seunggeun Kim, Taekyun Lee, David Z. Pan, Hyeji Kim, Sham Kakade, and Sitan Chen. Fine-tuning masked diffusion for provable self-correction. In International Conference on Machine Learning, 2026.arxiv.org/abs/2510.01384
  21. [21]Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah, Jerry Huang, Aditi Raghunathan, Seung-hoon Hong, Nicholas M Boffi, and Jinwoo Kim. Flow map language models: One-step language modeling via continuous denoising. arXiv preprint arXiv:2602.16813, 2026.arxiv.org/abs/2602.16813
  22. [22]Mingxiao Li, Tingyu Qu, Ruicong Yao, Wei Sun, and Marie-Francine Moens. Alleviating exposure bias in diffusion models through sampling with shifted time steps. In International Conference on Learning Representations, 2024.arxiv.org/abs/2305.15583
  23. [23]Peizhuo Li, Emre Aksan, Alexandru-Eugen Ichim, Thabo Beeler, and Olga Sorkine-Hornung. Diversify diffusion with temperature sampling and variance-corrective time shifting. arXiv preprint arXiv:2607.10853, 2026.arxiv.org/abs/2607.10853
  24. [24]Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-LM improves controllable text generation. Advances in neural information processing systems, 35: 4328–4343, 2022.arxiv.org/abs/2205.14217
  25. [25]Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. TinyGSM: achieving >80% on GSM8K with small language models. arXiv preprint arXiv:2312.09241, 2023.arxiv.org/abs/2312.09241
  26. [26]Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, 2024.arxiv.org/abs/2310.16834
  27. [27]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025.arxiv.org/abs/2502.09992
  28. [28]Mang Ning, Enver Sangineto, Angelo Porrello, Simone Caldera, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. In International Conference on Machine Learning, 2023.arxiv.org/abs/2301.11706
  29. [29]William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. URL https://arxiv.org/abs/2212.09748.
  30. [30]Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In International Conference on Learning Representations, 2016.
  31. [31]Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136–130184, 2024.arxiv.org/abs/2406.07524
  32. [32]Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T Chiu, and Volodymyr Kuleshov. The diffusion duality. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=9P9Y8FOSok.arxiv.org/abs/2506.10892
  33. [33]Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37:103131–103167, 2024.arxiv.org/abs/2406.04329
  34. [34]Marta Skreta, Tara Akhound-Sadegh, Viktor Ohanesian, Roberto Bondesan, Alán Aspuru-Guzik, Arnaud Doucet, Rob Brekelmans, Alexander Tong, and Kirill Neklyudov. Feynman-Kac correctors in diffusion: Annealing, guidance, and product of experts. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 55906–55949, 2025.arxiv.org/abs/2503.02819
  35. [35]Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. In Proceedings of the 41st International Conference on Machine Learning, pp. 46495–46513, 2024.arxiv.org/abs/2402.05841
  36. [36]Yanbo Xu, Yu Wu, Sungjae Park, Zhizhuo Zhou, and Shubham Tulsiani. Temporal score rescaling for temperature sampling in diffusion and flow models. In International Conference on Machine Learning, 2026.arxiv.org/abs/2510.01184

Paper details

vlnc:W_5fnn7h2h
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents