Parallel Language Models via Variational Decoding

Introduction

Large autoregressive models (Vaswani et al., 2017) are the core technical foundation of large language models (LLMs) (Brown et al., [9]). Autoregressivity comes with a significant downside, however: One model forward pass produces one token, causing high inference latency.

A plethora of approaches address this challenge, most prominently by increasing the speed of a given autoregressive model through smaller draft models in a process known as speculative decoding (Leviathan et al., [36]). However, draft models do not reach the quality of their big counterparts, ultimately limiting the achievable speedup. Discrete diffusion models (Hoogeboom et al., [29]; Austin et al., [6]) are an alternative modeling framework that predicts several tokens in one model call. However, a single model call cannot coordinate predicted tokens, requiring iterative refinement. Few-step discrete diffusion models reduce the number of steps by amortizing iterative refinement (Monsefi et al., 2026). Because diffusion models are more expensive to train, autoregressive models still dominate industry-scale inference.

This calls for a language modeling framework rooted in autoregression that can model arbitrary dependencies between predicted tokens in a single model call, combining effective training with fast inference at equal quality. Recently, Parallel Token Prediction (PTP) presented a decoding framework that distills an existing autoregressive model into a fast decoder, predicting several tokens in parallel (Draxler et al., [17]). However, PTP was only trained as a speculative decoding method for an existing model, and the framework they proposed to train from sequence data alone is highly inefficient because it is iterative at training time.

In this work, we close this gap and propose Parallel Language Models (PLMs), a novel language modeling framework based on an autoregressive architecture that trains parallel LLMs directly from training data. In detail, we contribute:

• We build Parallel Language Models based on a variational formulation, deriving effective pretraining and post-training strategies (§ 3).

• We pretrain PLMs up to 3B parameters that are on a faster and better Pareto frontier than autoregressive and discrete diffusion baselines (Figure 1). Our scaling laws confirm near-autoregressive negative log-likelihoods, and show that the parallelism increases with model scale (§ 4.1.1).

• When post-trained, our 1.3B and 3B Parallel Language Models match or exceed autoregressive quality while answering 2.3–5.7× faster (§ 4.2).

Related Work

Parallel prediction and generation. Early non-autoregressive sequence models generate multiple output tokens in parallel, substantially reducing the number of sequential decoding steps [24]. More recently, multi-token prediction (MTP) predicts several future positions [22] and is used in large-scale autoregressive models such as DeepSeek-V3 [14]. These methods retain an autoregressive modeling framework rather than defining a joint parallel generative model. Parallel Token Prediction (PTP) [17] instead introduces auxiliary random variables that enable multiple dependent future tokens to be generated jointly in one model call, and studies distillation from existing autoregressive models. Samragh et al. (2025) improve coherence of independent multi-token prediction by adding a small autoregressive head, and introduce Quadratic Decoding, which we adapt for PLM inference. Our work extends this line of research to pretraining of parallel language models directly from sequence data, developing a training recipe that scales to billions of parameters.

Iterative and few-step parallel generation. A complementary class of methods predicts many token positions in parallel and resolves dependencies through iterative refinement. Conditional Masked Language Models [21] progressively fill and revise masked positions, while discrete diffusion language models such as MDLM (Sahoo et al., 2024), LLaDA (Nie et al., 2025), and Dream (Ye et al., 2025) generate sequences through repeated denoising. Discrete Flow Matching [19] provides a related framework for defining generative trajectories over discrete sequences, while Eso-LM (Sahoo et al., 2026) connects autoregressive and masked-diffusion modeling and improves diffusion inference. Recent work increasingly targets few-step generation: Infinite Mask Diffusion (Yoo et al., 2026) and FS-DFM (Monsefi et al., 2026) modify the generative process or training objective to improve performance under a small number of sampling steps, while flow-map approaches further compress iterative trajectories toward few- or single-step generation [35] (Potaptchik et al. 2026). Relatedly, Lookahead Decoding [18] and Jacobi Forcing [30] generate tentative future tokens in parallel and recover consistency through verification or iterative refinement. In contrast, PLMs aim to model dependencies among simultaneously generated tokens directly within a single model call.

Autoregressive inference acceleration. A separate line of work accelerates inference from pretrained autoregressive language models. Speculative decoding uses a cheaper draft model to propose multiple tokens, which a target model verifies in parallel [36, 11]. Subsequent methods improve proposal quality through learned prediction heads or feature-based drafting, including Medusa [10], Hydra [4], and EAGLE [38, 40]. Unlike PLMs, these methods accelerate inference from a pre-existing autoregressive target model rather than training a parallel generative model directly from sequence data.

Latent-variable sequence models. Latent-variable models can also capture dependencies while retaining parallel sequence generation. FlowSeq [44] models non-autoregressive sequence generation through a flow-based latent distribution, while LaNMT (Shu et al., 2020) uses continuous latent variables and variational training for non-autoregressive translation. Parallel WaveNet (Oord et al., 2018) similarly generates audio in a single pass by mapping per-timestep noise through an inverse autoregressive flow, but requires invertibility. These approaches demonstrate how latent representations can mitigate the dependency limitations of factorized parallel decoders, but primarily target conditional sequence-to-sequence or audio generation. PLMs instead use token-level auxiliary variables to define a general language model in which dependencies among parallel predictions are mediated through the latent representation.

Parallel Language Models

Autoregressive language models represent the distribution of language by explicitly predicting a histogram over possible choices from a token vocabulary $\{1,\ldots,V\}$. This is an expressive paradigm, but it does not extend to predicting several tokens jointly because the possible token combinations $V^{L}$ scale exponentially in sequence length $L$.

One effective way to circumvent this is to use a language model that predicts sequences directly. First, sample a random auxiliary $u_i \sim p(u_i)$ per token position $i=1,\ldots,L$. Then have a decoder map the vector of auxiliaries $u$ to a coherent sequence in one model call:

Auxiliaries $u\in\mathcal{U}$ can either be continuous ($\mathcal{U}\subset\mathbb{R}$) or discrete ($\mathcal{U}\subset\mathbb{N}$), and we’ll give more details in the subsequent sections.

Intuitively, the model generates different sequences for different random inputs. Prominent examples of such parallel decoders are Few-Step Discrete Flow Matching (FS-DFM) (Monsefi et al., 2026) and Parallel Token Prediction (PTP) [17]. Both frameworks distill a parallel decoder from a pre-existing model, the former from a discrete diffusion model, and the latter from an autoregressive language model.

In the remainder of this section, we discuss the core framework of Parallel Language Models (PLMs). PLMs directly train on sequence data without relying on a pre-existing model. We provide a complete scalable recipe, covering pretraining, post-training, and inference.

Self-Distilling Parallel Language Models on Sequence Data

The Parallel Language Model we propose is a self-distillation method that jointly trains an autoregressive encoder $q_\theta(u_i \mid x_{\le i})$ from sequences to auxiliaries, and a parallel decoder from auxiliaries to sequences $x_i = f_\theta(u_{\le i})$, see Figure 2. Note that for notational simplicity, we remove dependency on a prompt or partially generated text in this derivation.

To train it, we derive our training objective from maximizing the likelihood of the training data under our model:

What is the probability of generating a sequence $x = x_{1:L}$ using a parallel decoder $x_i = f_\theta(u_{\le i})$? It is given by integrating over all possible auxiliaries from their prescribed distribution $u_i \sim p(u_i)$, and then decoding them to the corresponding sequence:

Note how we have switched from a deterministic decoder $f_\theta$ to a stochastic decoder $p_\theta$. This will allow taking gradients easily through a cross-entropy loss, and we take an arg max for inference (see § 3.6). Crucially, the product distribution over future tokens ensures that we can predict tokens in parallel: All dependencies between tokens are described by the auxiliaries.

Optimizing the loss in Eq. (2) using the integral in Eq. (3) is intractable: if we were to approximate the integral using a finite number of random auxiliary samples, we would essentially never sample a given sequence even once. This is where the encoder $q_\theta(u \mid x) = \prod_{i=1}^{L} q_\theta(u_i \mid x_{\le i})$ enters, proposing auxiliaries that should generate a given sequence. We rewrite Eq. (2) using the evidence lower bound (ELBO) (Kingma & Welling [32]; ); Zhang et al. (2018)), see Appendix § 10.§ 1 for details:

Remember that $p(u_i)$ specifies the distribution we sample the auxiliaries from during inference.

Equation (4) is significantly easier to optimize than using Eq. (3), since the encoder proposes auxiliaries $u$ for decoding a given sequence $x$. We show in Appendix § 10.§ 2 that equality holds at the global minimum, and that we can get arbitrarily close to this global minimum with a rich enough model, and so optimizing Eq. (4) actually optimizes Eq. (2).

Auxiliary design We now specify how we construct the auxiliary space. We formulate two desiderata that do not change the optimum of (4) yet make the design of auxiliaries simple and flexible: (a) Different tokens in the same context should be encoded to different auxiliaries. This allows decoding to be deterministic given auxiliaries and thus parallelizable. (b) The share of auxiliaries assigned to a given token should be equal to its probability given the preceding tokens.

These properties can easily be fulfilled by splitting the encoder into a standard autoregressive model predicting the next-token distributions $Q_i = q_\theta(\cdot\mid x_{<i})$, and then assigning auxiliaries $q_\theta(u_i \mid x_i, Q_i)$ using that probability vector $Q_i$, with entries $Q_i(v) = q_\theta(x_i = v \mid x_{<i})$, and the token $x_i$. Concretely, we achieve the latter by designing an oracle decoding function $x_i = f(u; Q_i)$ that specifies which auxiliaries $u \in\mathcal{U}$ map to which tokens. This oracle uniquely defines a set of auxiliaries encoding to that token $U_{x_i} = \{u \in\mathcal{U} : x_i = f(u; Q_i)\}$. Then, the auxiliary encoder samples a random $u_i \in U_{x_i}$ by restricting the auxiliary prior $p(u_i)$ to these allowed auxiliaries:

In this formulation, the disjointness and relative mass properties above read:

For any oracle $f(u; Q_i)$ that fulfills these properties, Eq. (4) can still be minimized given enough model capacity (cf. Appendix § 10.§ 4). For example, PTP ([17]) leveraged the quantiles of the histogram of a (fixed) autoregressive model $f^{\mathrm{PTP}}(u; Q_i) = \operatorname{ICDF}_{Q_i}(u)$, fulfilling the conditions in Eq. (6). In § 3.2, we give our improved design choice for associating tokens and auxiliaries that follows Eq. (6).

Crucially, if an encoder is constructed as above, and the shape of $q_\theta(u_i \mid x_{\leq i})$ matches the shape of $p(u_i)$ as in Eq. (5), we find that:

In other words, the trainable part of the encoding is a standard autoregressive model, and we design the association between tokens and auxiliaries. Proof in Appendix § 10.§ 3.

Using the autoregressive encoder design, we find that computing gradients through the sampling of $u \sim q_\theta(u \mid x)$ is not needed as the optimum of Eq. (4) does not change. We thus do not apply the reparameterization trick common in VAEs. This trivially allows using discrete auxiliary spaces, and stabilizes training empirically.

PLM training objective Together, we obtain the following training objective for a Parallel Language Model, where $\operatorname{sg}[\cdot]$ implies a stopped gradient:

It composes an autoregressive encoder with a parallel decoder. The above derivation confirms that this is a coherent way to train a parallel model. Next, we discuss its efficient implementation.

Scalable Training

We now discuss practical design choices for training a Parallel Language Model that scales to billions of parameters and tokens. At a high level, training Eq. (8) on a given sequence $x_{1:L}$ means first encoding every token $x_i$ to its auxiliary $u_i$ with the encoder $q_\theta$, and then the decoder $p_\theta$ reconstructs it to $\hat{x}_i$, see Figure 2. Below is our recipe to make this efficient:

Weight-sharing encoder and decoder Since a large part of the difficulty in predicting language is in parsing context, we share the model weights between the encoder $q_\theta$ and the decoder $p_\theta$. They only differ in the embedding layer, which embeds tokens from the vocabulary differently from the auxiliaries.

Block-autoregressive decoding Empirically, we find that reconstruction becomes increasingly difficult with sequence length. Therefore, we cap the maximum decoding sequence length to a block size $K$. Concretely, for a training sequence of length $L > K$, we divide the input sequence $x$ into blocks of $K$ tokens. So to predict a block of tokens $\hat{x}_{i_z:i_z+K-1}$ starting at position $i_z$, the decoder receives the corresponding auxiliaries $u_{i_z:i_z+K-1}$ and the input tokens $x_{<i_z}$ before the block. To avoid any recomputation on these tokens before the decoding block, we let the decoder attend to the KV cache of the encoder, and pass gradients to it (but not through the sampling of auxiliaries). Figure 3 shows how this can be efficiently implemented with an attention mask that makes each decoder block attend to the encoder tokens up to the block start. We train all our models with block size $K=16$ tokens.

Encoder-to-decoder ratio

While the optimum of Eq. (8) can encode and decode perfectly at infinite model capacity, in practice model capacity is split between the two modes of the model. We therefore introduce a loss ratio $\beta$, analogous to the loss weight in a $\beta$-VAE [27]. We implement this ratio by not decoding all tokens that were encoded ($\beta>1$), or by decoding every token several times ($\beta<1$). For $\beta=1$, every encoded token is decoded once. To this end, we use a stride $s=\lfloor\beta K\rfloor$ between blocks, so $\beta\approx s/K$. Figure 3 shows an example for stride $s=2$ at block size $K=3$. The explicit loss becomes, with $u_i^{(z)}\sim q_\theta(u_i\mid x_{\leq i})$ sampled once per block without gradient, block count $Z=\lceil L/s\rceil$ and block offsets $i_z=(z-1)s+1$:

Random shifts of the block grid and truncation at sequence boundaries are described in Appendix § 12. We denote models by PLM-$\beta$; the autoregressive baseline is recovered in the limit $\beta\to\infty$.

Geometry-aware auxiliaries

We now describe an efficient and effective way to design the auxiliary space. We decide to leverage a set of $N_{\mathrm{aux}}$ unit vectors living in the embedding space of the transformer $e_u\in S^{d-1}$, $u=1,\ldots,N_{\mathrm{aux}}$ as auxiliaries. The prior is uniform over these indices, $p(u)=1/N_{\mathrm{aux}}$. Unlike quantiles over an arbitrary vocabulary order as in PTP [17], embedding-space auxiliaries naturally exploit token similarity (see ablation in Appendix § 13).

For assigning auxiliaries to tokens, we choose to assign every auxiliary $u$ to the most similar regular token embedding $e_v$, as measured by dot-product similarity in the embedding space. To ensure that token probabilities $Q_i=q_\theta(x_i\mid x_{<i})$ match the relative mass of auxiliaries (property (b) in Eq. (6)), we adjust this similarity using a scalar bias $w_v$ for each token (see Appendix § 11):

This is a discretized version of a weighted Voronoi tessellation [5] on the sphere, see Figure 4 for an example with three tokens and a 2-dimensional embedding space. We initialize the auxiliary embeddings randomly and leave them fixed during training.

At inference, we insert the embedding $e_u$ of a uniformly sampled $u\in\{1,\ldots,N_{\mathrm{aux}}\}$ into the parallel decoder.

Note that a finite number of $N_{\mathrm{aux}}$ auxiliaries has a limited resolution of representing token probabilities. We choose $N_{\mathrm{aux}}=512$ and feed a mask auxiliary for any token that is not among the top-128 predicted by the AR model. During training, the mean total variation between the fitted cell masses and the rounded target masses was below 0.3% (see Appendix § 12).

Representation alignment Each training step looks at the sequence twice: once autoregressively, and once at parallel decoding. We find that adding a representation alignment loss (Yu et al., 2025) that aligns the hidden state of the parallel decoder to the autoregressive encoder significantly speeds up training, achieving the same decoding quality at approximately half the training compute (see Appendix § 13). We use the following regularization:

Here, $\boldsymbol{h}_{i_z+k}^{(z)}$ is the hidden state of the parallel decoder at the position that predicts $x_i$ in block $z$, and $\boldsymbol{h}_i$ is the autoregressive state that predicts the same token after reading $\boldsymbol{x}_{<i}$. We read them at layers $\ell_{\mathrm{dec}} < \ell_{\mathrm{enc}}$ (values in Appendix § 12). This asks the decoder to arrive early at what the encoder reaches later. The stop gradient $\operatorname{sg}[\cdot]$ makes the decoder chase the encoder and never the reverse, and the learned affine map $g_{\psi}$ facilitates alignment. We add $\lambda_{\mathrm{align}}\mathcal{L}_{\mathrm{align}}$ to Eq. (9) and drop $g_{\psi}$ after training, so inference is unchanged.

Post-Training

We post-train PLMs in two stages. First, supervised fine-tuning (SFT) adapts the model to instruction–response data using the joint encoder–decoder objective of Eq. (9), with the loss restricted to response tokens so that the prompt serves only as context. We train on a mixture of general instruction-following, mathematics, coding, and precise instruction-following data. Dataset and optimization details are given in Appendix § 16.§ 1.

We then apply Direct Preference Optimization (DPO) (Rafailov et al., 2023) jointly to the two modes of the PLM: the autoregressive encoder and the parallel decoder. Let $\theta$ denote their shared parameters and $\theta_{\mathrm{ref}}$ a frozen SFT reference. For a prompt–response sequence $\boldsymbol{x}=(p,r)$, let $\mathcal{L}_{\mathrm{enc}}^r(\theta;\boldsymbol{x})$ and $\mathcal{L}_{\mathrm{dec}}^r(\theta;\boldsymbol{x},u)$ denote the response-restricted encoder and decoder components of Eq. (9). Here, $\mathcal{L}_{\mathrm{dec}}^r(\theta;\boldsymbol{x},u)$ denotes the response-restricted decoder loss of Eq. (9) for a fixed auxiliary realization $u$, rather than averaging over newly sampled auxiliaries. We define the response score

The fraction normalizes by the total number of terms, preventing the response length $L^r$ from dominating the score.

DPO trains on preference triples $(p,r^{+},r^{-})$, where $r^{+}$ is the preferred and $r^{-}$ the rejected response to the prompt $p$. Write $\boldsymbol{x}^{\pm}=(p,r^{\pm})$. For each of the two candidate responses, we sample auxiliaries from the frozen reference encoder, $u^{\pm}\sim q_{\theta_{\mathrm{ref}}}(u\mid x^{\pm})$, where each auxiliary is drawn from the observed target token’s Voronoi cell. The same candidate-specific auxiliaries are used to evaluate the policy and reference. They remain samples from the frozen reference encoder as the policy changes, keeping the comparison stationary conditional on the sampled auxiliary realization.

Define $\delta_{\theta}(\boldsymbol{x},u)=s_{\theta}(\boldsymbol{x},u)-s_{\theta_{\mathrm{ref}}}(\boldsymbol{x},u)$. We then optimize

where $\gamma$ is the DPO inverse temperature. For the autoregressive baseline, we use the same objective with the decoder component omitted from Eq. (12).

Unlike canonical DPO, which uses summed sequence log-probabilities, our PLM score replaces the intractable marginal response log-likelihood with the auxiliary-conditioned composite training score in Eq. (12) and normalizes it by the number of prediction terms. We therefore optimize a length-normalized DPO-style surrogate. This normalization is similar to the length-normalized DPO of Lambert et al. [34]. SimPO [45] uses the same normalization but drops the reference model, which we retain. Further details are given in Appendix § 16.§ 2.

Inference

During inference, tokens are generated in parallel via Quadratic Decoding (Samragh et al., 2025), whose two main components, proposal and verification, we adapt for PLMs. First, a block of $K_{\mathrm{pred}}$ coordinated tokens $x_j = f_\theta(x_{<i}, u_{i:j}),\ i \leq j < i + K_{\mathrm{pred}}$ is produced by feeding independently randomly drawn auxiliaries $u_j \sim p(u_j)$ (cf. § 3.2) into our trained model $p_\theta$ and then taking the arg max of the produced logits. The inference block size $K_{\mathrm{pred}}$ is set at inference time and can be smaller than the training block size $K$. Second, the proposed tokens are verified by the encoder against a predefined rule and a subset of accepted tokens is appended to the context. While our model is trained to predict $K$ tokens, this step is necessary, as in practice finite model capacity degrades proposal quality with increased distance to the context. Quadratic Decoding allows both steps to be done in a single efficient model call, by making $K_{\mathrm{pred}}$ proposals of which only one is selected after verification. While this widened model call increases the computational cost, for small $K_{\mathrm{pred}}$ the inference overhead is negligible (cf. Table 13), as the extra work is efficiently parallelized on one GPU by adopting a custom attention matrix. See Appendix § 18 for algorithmic details.

Model	PLM 1.3B, SFT + DPO, $\beta= 1$			PLM 3B, SFT + DPO, $\beta= 1$
	tokens/s	tokens/call	ms/call	tokens/s	tokens/call	ms/call
AR	307	1.00	3.25	140	1.00	7.14
$K_{\mathrm{pred}}$ 2	548	1.85	3.32	255	1.88	7.28
4	876	3.06	3.40	425	3.25	7.46
7	1090	4.05	3.59	558	4.53	7.86
8	966	4.25	4.25	514	4.81	9.07
10	963	4.52	4.53	533	5.22	9.47
16	569	4.85	8.28	290	5.83	19.56
GSM8K	27.9–28.1			37.2–37.8

Generally, a verification rule is set to guarantee high quality samples. In speculative decoding, based on an existing AR model, this is usually achieved by forcing the decoding algorithm to produce the same sample distribution as the reference. PTP [17], for example, forces the generation to be token-identical to a random rollout of the AR model. Here, we argue that this strict verification artificially harms throughput. If our model creates a plausible continuation, we avoid discarding it by loosening our acceptance rule to also admit all tokens that have high likelihood under our encoder, specifically ones that are in the $\rho$-nucleus of $q_\theta(x_i \mid x_{<i})$ or ones with $q_\theta(x_i \mid x_{<i}) \geq\epsilon$. Table 11 shows this increases the number of accepted tokens per call of the post-trained models by an average of 46.5% at comparable quality.

Size	$\beta$	Decoding	GSM8K Score	GSM8K Q / s	GSM8K Tok / s	GSM8K Tok / call	MATH-500 Score	MATH-500 Q / s	MATH-500 Tok / s	MATH-500 Tok / call
1.3B	0.5	AR	27.0	1.1 (1.2×)	308	1.0	7.6	0.4 (1.0×)	308	1.0
1.3B	0.5	Exact	31.5	4.4 (4.8×)	1112	4.2	8.0	1.0 (2.6×)	803	3.2
1.3B	0.5	Relaxed	32.6	5.2 (5.7×)	1337	6.1	8.4	1.4 (3.7×)	1107	5.1
1.3B	1	AR	27.7	1.1 (1.2×)	307	1.0	5.6	0.4 (1.1×)	309	1.0
1.3B	1	Exact	28.1	4.0 (4.4×)	1090	4.1	6.2	1.0 (2.7×)	773	3.0
1.3B	1	Relaxed	29.4	4.9 (5.4×)	1304	5.7	7.2	1.2 (3.1×)	941	4.3
1.3B	2	AR	27.0	1.0 (1.1×)	307	1.0	6.8	0.4 (1.0×)	308	1.0
1.3B	2	Exact	28.8	4.0 (4.4×)	1059	3.9	7.8	0.9 (2.4×)	757	2.9
1.3B	2	Relaxed	29.5	4.8 (5.2×)	1251	5.5	8.4	1.0 (2.8×)	882	4.4
3B	1	AR	39.0	0.4 (1.0×)	140	1.0	12.2	0.2 (1.0×)	140	1.0
3B	1	Exact	37.2	1.8 (4.1×)	558	4.5	12.6	0.5 (3.0×)	406	3.3
3B	1	Relaxed	37.4	2.1 (4.8×)	662	5.4	14.8	0.7 (3.8×)	516	4.3
Size	$\beta$	Decoding	HumanEval Score	HumanEval Q / s	HumanEval Tok / s	HumanEval Tok / call	IFEval Score	IFEval Q / s	IFEval Tok / s	IFEval Tok / call
1.3B	0.5	AR	19.5	0.9 (1.3×)	308	1.0	45.7	0.9 (1.2×)	307	1.0
1.3B	0.5	Exact	16.5	3.3 (4.9×)	1113	4.3	46.2	1.6 (2.1×)	525	1.9
1.3B	0.5	Relaxed	17.7	3.0 (4.5×)	1323	5.9	45.1	1.9 (2.4×)	780	3.5
1.3B	1	AR	8.5	1.0 (1.5×)	306	1.0	45.3	0.9 (1.2×)	308	1.0
1.3B	1	Exact	10.4	4.6 (6.8×)	1131	4.3	44.7	1.4 (1.9×)	485	1.8
1.3B	1	Relaxed	10.4	5.2 (7.8×)	1348	6.2	45.7	1.9 (2.5×)	713	3.1
1.3B	2	AR	11.0	0.8 (1.2×)	307	1.0	48.2	0.9 (1.2×)	308	1.0
1.3B	2	Exact	17.7	2.8 (4.2×)	956	3.6	51.0	1.4 (1.8×)	478	1.8
1.3B	2	Relaxed	15.8	3.0 (4.5×)	1084	5.0	48.1	1.8 (2.3×)	718	3.1
3B	1	AR	23.2	0.4 (1.0×)	140	1.0	54.0	0.4 (1.0×)	140	1.0
3B	1	Exact	26.8	1.4 (3.8×)	558	4.6	52.7	0.7 (1.7×)	235	1.9
3B	1	Relaxed	32.3	1.5 (4.1×)	644	5.3	53.6	1.0 (2.5×)	360	2.9

Experiments

We empirically evaluate Parallel Language Models (PLMs) by their quality and wall-clock speed compared to other language model frameworks.

To this end, we pretrain PLMs of 80M, 150M, 350M, 1.3B and 3B parameters on a general mix of tokens. First, we fit scaling laws to PLMs. We find that the text quality of the encoder follows that of an identically trained autoregressive model, and that the average number of tokens the decoder can reconstruct in a single model call increases with model size, see § 4.1.1. Compared to competing language model frameworks, PLMs achieve significant speedups (§ 4.1.2). Finally, we post-train and evaluate PLMs on math, coding, and instruction-following benchmarks, finding comparable or better quality compared to identically trained autoregressive models, but at about 2.3-5.7× the speed.

Pretraining

Scaling Laws

We study how language quality and parallelism improve with PLM size and training tokens. Quality is measured by the encoder’s negative log-likelihood per token, $\mathcal{L}_{\mathrm{enc}}$, as is standard for autoregressive models. Our parallelism metric $R$ estimates the average number of training tokens correctly reconstructed in parallel per model call (see Appendix § 17 for a formal definition). An autoregressive model has $R = 1$.

We pretrain PLM-$\beta$=0.5, PLM-$\beta$=1 and PLM-$\beta$=2, corresponding to encoder-to-decoder ratios $\beta= 0.5, 1, 2$ (§ 3.2), at sizes of 80M, 150M, 350M and 1.3B parameters on ClimbMix [15]. We also train matching autoregressive baselines and add a 3B PLM-$\beta$=1 extrapolation. All PLMs use decoder blocks of $K = 16$ tokens. Token budgets reach $D/N \approx100$, where $N$ denotes non-embedding parameters and $D$ training tokens. Hyperparameters follow Step Law [37], with details in Appendix § 12.

Figure 5 shows that larger models reconstruct more tokens at a lower loss. We fit Chinchilla-style laws [28] jointly to AR and PLM runs, modeling the loss through a normalized effective model size $n_{\mathrm{eff}} = n\beta/(\beta+ 0.48)$ and $R$ through a power law above the autoregressive baseline:

Here, $n = N/10^{8}$ and $d = D/10^{10}$ express non-embedding parameters in units of 100M and training tokens in units of 10B. Both laws recover the autoregressive case as $\beta\to\infty$.

Over the fitted range, at fixed $\beta$, the absolute loss gap to an AR model of the same size therefore shrinks as models grow, while $R$ increases with both model size and training tokens. Thus, as depicted in Figure 5, PLMs become better and more parallel with scale, and $\beta$ controls the quality–parallelism trade-off. Fit details, uncertainty estimates and the 3B extrapolation check are in Appendix § 17.

Pretraining Model Comparison

Parallel Language Models compete with other language models that allow predicting more than one token per model call. However, the latter often cannot model dependencies between tokens that are predicted in the same step, or require substantially longer training.

We therefore compare the speed and quality of publicly available autoregressive models, discrete diffusion models, and few-step discrete diffusion models to our PLMs at $D/N = 100$ from § 4.1.1. To this end, we ask a variety of models to continue the same 200 prompts for an additional 128 tokens, where each prompt is a 64-token opening of a randomly sampled FineWeb-Edu document (Lozhkov et al., [43]). Quality is measured using Phi-4 (Abdin et al., [1]) as a judge, and we fit Bradley-Terry Elo scores on the resulting data (Hunter, [31]).

Figure 1 shows that our framework sets a new standard on the Pareto frontier of speed vs quality for pretrained models, despite having trained on substantially fewer tokens than commercially trained models. PLMs also outperform all tested discrete diffusion models, including few-step models. The AR models use temperature 1 sampling among the top 128 tokens for a fair comparison to the geometry-aware auxiliaries used

We benchmark PLM with strict speculative decoding, and relaxed speculative decoding as in § 3.6. Figure 1 shows that this relaxed acceptance rule gives an increase in speed for a small drop in quality within a fixed model size. Additionally, as we increase the model size, the PLMs’ ability to predict more tokens per call increases, going from 1.95 during exact speculative decoding at 350M to 2.26 at 3B.

In Appendices E and F, we report additional results and provide details on the experimental setup.

Post-Training

We now confirm that Parallel Language Models can be fine-tuned to specific tasks, just like autoregressive models. To this end, we post-train our 1.3B and 3B PLMs using SFT and DPO, see Section 3.3. We also post-train our purely autoregressive 1.3B baseline.

Table 1 shows that our post-trained PLMs match or exceed the quality of an identically post-trained 1.3B autoregressive baseline, all while sampling approximately 2.3-5.7× faster, depending on the task. In addition, we confirm that our 3B PLM improves quality further, again sampling significantly faster than an autoregressive model of the same size would.

Size	$\beta$	Decoding	GSM8K Score	GSM8K Q / s	MATH-500 Score	MATH-500 Q / s	HumanEval Score	HumanEval Q / s	IFEval Score	IFEval Q / s
1.3B	AR	AR	27.9	0.9 (1.0×)	7.4	0.4 (1.0×)	12.8	0.7 (1.0×)	48.2	0.8 (1.0×)
1.3B	0.5	Relaxed	32.6	5.2 (5.7×)	8.4	1.4 (3.7×)	17.7	3.0 (4.5×)	45.1	1.9 (2.4×)
1.3B	1	Relaxed	29.4	4.9 (5.4×)	7.2	1.2 (3.1×)	10.4	5.2 (7.8×)	45.7	1.9 (2.5×)
1.3B	2	Relaxed	29.5	4.8 (5.2×)	8.4	1.0 (2.8×)	15.8	3.0 (4.5×)	48.1	1.8 (2.3×)
3B	1	AR$^\dagger$	39.0	0.4 (1.0×)	12.2	0.2 (1.0×)	23.2	0.4 (1.0×)	54.0	0.4 (1.0×)
3B	1	Relaxed	37.4	2.1 (4.8×)	14.8	0.7 (3.8×)	32.3	1.5 (4.1×)	53.6	1.0 (2.5×)

Conclusion

This paper derives a complete and scalable recipe to build language models from scratch that predict coordinated tokens in parallel. Already at the model scales and token budgets trained for this paper, there is a clear inference speed advantage over autoregressive models of equal quality, even over those trained with significantly higher budget.

Going forward, our mathematical framework and our scaling laws indicate no fundamental boundaries that would prevent scaling our framework to industry scales. Instead, they predict that scaled up models achieve autoregressive quality, in particular after post-training, and generate many tokens in parallel. We hope this opens a research avenue that enables autoregressive models to be as parallel in inference as they already are in training.

AI Use Statement

In this work, we used generative AI tools for formulating mathematical claims, assisting in the writing of proofs design, and implementing methods. We have not used generative AI tools for helping develop theoretical models or conceptual frameworks, providing critical ingredients for proving mathematical claims, or interpreting results. The categories generating synthetic data sets, proposing or refining hypotheses, assisting with translation, cleaning and reformatting datasets, and supporting qualitative and thematic data analysis are not applicable to this work.

Additionally, we used generative AI tools for creating or modifying scientific figures or images, creating or editing software code, drafting parts of our research paper, summarizing or analyzing existing literature, sourcing/searching for information, editing our research paper to improve readability, and identifying relevant literature. We have reviewed all AI-assisted work by verifying LLM-generated code by two authors, verifying information, and by checking for potential plagiarism for research ideas through manual literature survey. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Ethics Statement

The authors do not feel that their paper submission raises any questions beyond the general risks inherent to developing and distributing AI models.

Reproducibility Statement

We have included all details necessary to replicate our experiments in Appendices § 12 and § 16, and all details necessary to derive our method are in Appendix § 10. We will publish all our code upon publication.

Acknowledgments

Stephan Mandt acknowledges support from the National Science Foundation (NSF) through CA-REER Award IIS-2047418 and Grant IIS-2611549. The authors acknowledge the Swiss AI Initiative for providing the computational resources required for this work. The computations were performed on the Clariden vCluster on the Alps supercomputing infrastructure operated by the Swiss National Supercomputing Centre (CSCS) under project IDs ab014 and a0200. RG and JS acknowledge support from StimuLOOP (#1-007811-002) grant and the Vontobel Foundation. RG acknowledges funding from the German Federal Ministry of Research, Technology and Space under the grant BI-FOLD26B. PH acknowledges support from the Transatlantic Collaboration Fund (TCF), funded by the Hasso Plattner Foundation (HPF).

Author Contribution

Felix Draxler designed and coordinated the overall project, coauthored the compute grant, and managed computational resources. He headed the pretraining team, contributed to pretraining models and coauthored the scaling laws. He proposed the idea of self-distillation, and contributed to the construction of the ELBO loss derivation.

Robin C. Geyer led external partnerships and set technical direction, coauthored the compute grant, and managed computational resources. He was co-lead and core developer on the pretraining team and coauthored the scaling-law analysis. He moved beyond representing auxiliary variables as scalars by developing a Voronoi-based method that maps them to vectors in the transformer’s embedding space.

Farrin Marouf Sofian led the post-training team, co-designed the post-training recipe, including algorithms, datasets and benchmarks. Implemented supervised fine-tuning. She also co-developed the VAE framework for PLM as presented in the paper and co-edited and co-revised the final manuscript.

Daniel Geyfman led the baseline/evaluation team and conducted both pretraining and posttraining quality experiments. He also contributed to getting the initial pretrained model and running initial ablations.

Justus Will led the inference team. He proposed, implemented, analyzed, and tested the inference algorithms PLM uses. He was responsible for designing the fast hardware-optimized inference codebase and porting the trained models into it. He advised on several design choices across the project and derived the exact conditions and proofs in Appendix B.

Philipp Hildebrandt was a core developer on the pretraining team and contributed to developing self-distillation into a practical training recipe, explored and implemented representation alignment to improve training efficiency and built the experimental setup for the scaling-law study.

Giosue Migliorini was a core developer on the pretraining team, contributing to incorporate PLM-specific training, attention masking, and metrics into the Megatron repository. He also contributed to the variational interpretation of PLMs, and to testing novel inference recipes.

Prakhar Srivastava contributed to the design and implementation of post-training methods for PLMs, including the initial formulation and technical documentation of PLM-compatible SFT and DPO, supported by proof-of-concept experiments. He also contributed to the post-training literature review, dataset selection, and evaluation design, and implemented and validated the large-scale DPO training pipeline.

Gagan Sai Trinadh Duggiraju Peddiraju designed and ran initial ablations across various hyperparameters and created an intermediary ablation report. He also worked on improving inference generation quality using a small autoregressive head.

Kushagra Pandey co-designed the core components of the VAE framework for PLM. He also participated in the discussions around post-training throughout the project while serving mostly in an advisory role.

Jorge da Silva Gonçalves conducted the literature review and baseline exploration. Furthermore, he wrote the related work section, was part of the baseline/evaluation team, and contributed to the initial pretraining implementation.

Elijah Tavares contributed to the baseline team and to the pretraining effort. He worked on training and performing ablations for masked diffusion baselines. He also performed a literature review and selected comparison metrics. Furthermore, he optimized and improved the robustness of the data processing pipeline.

Jan Niklas Groeneveld implemented the framework for the evaluation and comparison to the baselines, and an interface for the model, advised the baselines team, and participated in discussions of post-training algorithms.

Yang Meng designed a controlled toy dataset to support validation of the core idea and systematic experimentation. He also contributed to the post-training effort through literature review and benchmark design. Julia E. Vogt coauthored the compute grant, allocated resources, read, edited and approved the manuscript.

Stephan Mandt co-led the project with the team leads, with most of the work carried out within his research group at UCI. He provided scientific guidance on the project’s direction and scope, allocated resources, and advised team leads and students in biweekly meetings. He coauthored the compute grant, drew connections to deep latent variable models, and co-wrote the paper.

Method Derivation

In this and the following subsections, we treat encoder and decoder as arbitrary distributions with the factorizations of the main text,

This is justified because we only consider model architectures with these properties.

Evidence Lower Bound

We prove Eq. (4), i.e., that

is an upper bound to the negative likelihood. To this end, we show

where

denotes the true posterior over auxiliaries. The result follows as $D_{\mathrm{KL}}$ is non-negative. From our assumptions follows $\operatorname{supp}(q_\theta(\mathbf{u}\mid\mathbf{x})) \subseteq\operatorname{supp}(p_\theta(\mathbf{u}\mid\mathbf{x}))$, and since $-\log p_\theta(\mathbf{x})$ does not depend on $\mathbf{u}$, we can take its expectation under $q_\theta(\mathbf{u}\mid\mathbf{x})$ and apply Bayes’ rule:

where the second equality multiplies and divides by $q_\theta(\mathbf{u}\mid\mathbf{x})$ inside the logarithm, the third splits the logarithm and uses linearity of expectation. This shows Eq. (19), and Eq. (4) follows.

Tightness of the Bound at the Global Minimum

During training, we optimize $\mathcal{L}(\theta)=\mathbb{E}_{x\sim p_{\mathrm{data}}}[\mathcal{L}(\theta;x)]$. Averaging Eq. (19) gives

The bound is tight at the global minimum, that is attained if and only if $p_\theta=p_{\mathrm{data}}$ and $q_\theta(u\mid x)=p_\theta(u\mid x)$ almost everywhere. Then, the cross-entropy $\mathbb{E}_{x\sim p_{\mathrm{data}}}\left[-\log p_\theta(x)\right]$ equals the data entropy $H(p_{\mathrm{data}})$, and the $D_{\mathrm{KL}}$ term vanishes.

Encoder Loss vs Autoregressive Loss

In the following we will restrict our encoder to be injective, i.e., such that for fixed $x_{<i}$ and all $x_i\ne x_i'$ their supports $\mathcal{U}_{x_i}:=\operatorname{supp}q_\theta(u_i\mid x_{<i},x_i)$ and $\mathcal{U}_{x_i'}$ are disjoint. Appendix B.4 shows that we can construct such an encoder that still is a global minimizer of Eq. (24).

Proof. By disjointness, for $u_i\in\mathcal{U}_{x_i}$ only one term survives in the aggregate posterior:

For such $u_i$, $q_\theta(x_i\mid u_i,x_{<i})=1$ by construction and Bayes’ rule yields

Substituting into the KL gives

$(\Leftarrow)$ If $q_\theta(u_i\mid x_{<i})=p(u_i)$ a.e., the last term of Eq. (28) vanishes.

$(\Rightarrow)$ If the equality holds for all $x_i$, the last term of Eq. (28) is zero for every $x_i$. Averaging over $x_i\sim q_\theta(x_i\mid x_{<i})$ and using $\sum_{x_i}q_\theta(x_i\mid x_{<i})q_\theta(u_i\mid x_{\le i})=q_\theta(u_i\mid x_{<i})$ yields

so $q_\theta(u_i\mid x_{<i})=p(u_i)$ a.e.

The condition $q_\theta(u_i\mid x_{<i})=p(u_i)$ enforces that $u_i$ only contains the additional information about $x_i$. We can ensure this by construction:

Proof. $(\Rightarrow)$ By disjointness on $\mathcal{U}_{x_i}$, and substituting $q_\theta(u_i \mid x_{<i}) = p(u_i)$ into Eq. (27).

$(\Leftarrow)$ Integrating over $\mathcal{U}_{x_i}$ gives $p(\mathcal{U}_{x_i}) = q_\theta(x_i \mid x_{<i})$, so $\sum_{x_i} p(\mathcal{U}_{x_i}) = 1$. Hence,

Proof. We directly apply Theorem 1 and Proposition 1.

Injectivity: By construction, the Voronoi cells $\mathcal{U}_v$ of Eq. (10) are disjoint for all $v = x_i$.

Shape matching: The prior $p(u) = \operatorname{Unif}(\mathcal{U})$, is uniform over the discrete auxiliary space. As we draw embeddings randomly over their cell, we have $q_\theta(u_i \mid x_{\leq i}) = \operatorname{Unif}(\mathcal{U}_{x_i})$, and constant ratio.

Mass matching: By assumption, $p(\mathcal{U}_{x_i}) = q_\theta(x_i \mid x_{<i})$ exactly.

Encoder Injectivity at Optimality

In the following, we relax our assumptions to $p_\theta(x_i \mid u_{\leq i}) \geq0$, i.e., the decoder is allowed to vanish. This is not a restriction, as any such decoder can be approximated arbitrarily close by a decoder with strict positivity.

For a fixed injective encoder, $u_{\leq i}$ determines $x_{\leq i}$, say $x_i = g_i(u_{\leq i})$. Then, $\mathcal{L}_{\mathrm{dec}}$ is a sum of non-negative terms $-\log p_\theta(x_i \mid u_{\leq i})$ that vanishes if and only if $p_\theta(x_i \mid u_{\leq i}) = 1[x_i = g_i(u_{\leq i})]$: the optimal decoder is the deterministic inverse of the encoder. Inserting it into Eq. (3),

when the cells cover the auxiliary space $\mathcal{U}$: the probability of a token is the prior mass of its cell.

If the injective encoder additionally satisfies $p(\mathcal{U}_{x_i}) = p_{\mathrm{data}}(x_i \mid x_{<i})$ and $q_\theta(u_i \mid x_{\leq i}) = p(u_i)1[u_i \in\mathcal{U}_{x_i}]/p(\mathcal{U}_{x_i})$, we have $p_\theta= p_{\mathrm{data}}$ and $q_\theta(u \mid x) = p_\theta(u \mid x)$ almost everywhere, which yields global optimality as in Appendix B.2, showing that injectivity is not a constraint for optimality.

If, in addition, the token model matches the data, $q_\theta(x_i \mid x_{<i}) = p_{\mathrm{data}}(x_i \mid x_{<i})$, as it does at the global minimum of (8), the first condition gives $p(\mathcal{U}_{x_i}) = q_\theta(x_i \mid x_{<i})$ showing that there is a global minimizer whose cells are sized by the token model.

Fitting the Voronoi Weights and Alternative Cell Geometries

At position $i$, let $T_i$ be the $M$ most likely tokens under the encoder distribution. Their renormalized probabilities are

We convert these probabilities into integer cell capacities $n_{i,v}$ by largest-remainder rounding: each token first receives $\lfloor N_{\mathrm{aux}}\widetilde{q}_{i,v}\rfloor$ auxiliaries, and the remaining auxiliaries are allocated in descending order of the fractional remainders. The capacities therefore satisfy $\sum_{v\in T_i} n_{i,v} = N_{\mathrm{aux}}$. Relative to the renormalized top-$M$ probabilities, largest-remainder rounding gives

Laguerre cells. Let $d$ be the model’s embedding dimension. We use the unnormalized output-layer rows $e_v \in\mathbb{R}^{d}$ as cell anchors and auxiliary embeddings $e_u$ sampled once uniformly from ${}^{d-1}$ The score $s_{i,v}(u)=e_v\cdot e_u+w_{i,v}$ defines the cells $U_{i,v}$ through Eq. (10). This gives a Laguerre diagram with sites $e_v$ and power weights $\lVert e_v\rVert_2^2+2w_{i,v}$, restricted to the auxiliary sphere:

Thus maximizing the score is equivalent to minimizing the weighted squared distance. The offsets $w_{i,v}$ control the cell sizes while retaining the geometry of the language-model head.

Weight fitting. The offsets are dual variables of the balanced assignment problem that maximizes the total score subject to the capacities $n_{i,v}$. We fit them using annealed Sinkhorn updates. At temperature $\tau>0$, the soft assignments and corresponding cell loads are

For $n_{i,v}>0$, the log-domain update is

This increases the scores of underfilled cells and decreases those of overfilled cells. Candidates with zero capacity receive a large negative offset. We initialize the offsets from the capacity shares and the scale of the geometric scores, then apply clipped, over-relaxed updates as $\tau$ decreases. To reduce the fitting cost, we evaluate the soft assignments over a small shortlist of candidates for each auxiliary and refresh these shortlists between rounds. The final assignment takes the highest score over all $M$ candidates, with consistent tie-breaking. We use a fixed number of updates, so the resulting cell counts approximate the prescribed capacities; the fitting settings and measured error are reported in Appendix § 12.

Pretraining Details

This section lists the settings of the pretraining runs in § 4.1.1. PLMs and autoregressive (AR) baselines of the same size share every setting except the loss. The AR baselines switch off the decoder and alignment losses.

Data. We pretrain on ClimbMix [15], tokenized with the GPT-2 byte-pair encoding (Radford et al., 2019). Its $V=50{,}257$ tokens are padded to 50,304 embedding rows, a multiple of 128. The tokenized corpus holds 357.4B tokens in 553M documents. The largest run, the 3B PLM, reads 300B tokens (240B constant learning rate and 60B during learning rate decay), or 84% of the corpus. We hold out no split and report training losses throughout.

Training sequences are 4096 tokens long and packed across document boundaries, with an end-of-document token between documents. The AR loss covers every position of a packed sequence. A decoder block, in contrast, ends at the first end-of-document token among its targets, so it never reconstructs tokens from two documents.

Architecture. All models are decoder-only transformers. They use pre-normalization with RMSNorm (Zhang & Sennrich, 2019), SwiGLU MLPs (Shazeer, 2020), and grouped-query attention [2]. In grouped-query attention, four query heads share each key–value head, and every head has dimension 64. Rotary embeddings (Su et al., 2024) with base 500,000 encode the positions. The linear layers of the transformer have no biases, the input and output embeddings are untied, and we use no dropout. All weights are initialized from $\mathcal{N}(0,0.02^2)$. Table 2 lists the five model sizes.

Model	Layers	$d$	FFN	Heads	KV	$N$	Total	$D_{\max}$	GBS	$\eta$	Nodes
80M	12	512	1408	8	2	33.8M	85.3M	3B	32	$6.22 \cdot10^{-3}$	1
150M	12	768	2048	12	3	74.3M	151.6M	6B	64	$4.39 \cdot10^{-3}$	2
350M	24	1024	2560	16	4	251.7M	354.7M	20B	128	$2.66 \cdot10^{-3}$	4
1.3B	24	2048	5632	32	8	1.08B	1.29B	80B	256	$1.44 \cdot10^{-3}$	8
3B	28	3072	8192	48	12	2.77B	3.08B	300B†	512	$1.10 \cdot10^{-3}$	32

The parameter count $N$ of the scaling laws counts only the transformer body. It excludes the input and output embeddings and the few parameters the PLM adds: the alignment map $g_\psi$ and the mask auxiliary described below.

Optimizer. We use AdamW [42] with $\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-8}$, and weight decay 0.1. Gradients are clipped to a global norm of 1. We train in bfloat16 mixed precision and keep gradient reductions, master weights, and optimizer states in fp32.

Learning rate and batch size. We set the peak learning rate and the global batch size from $N$ and the token budget $D_{\max}$ of the constant-learning-rate run with Step Law (Li et al. [37]), a fitted rule for these two hyperparameters:

We round $B$ to the nearest power of two in sequences. The micro-batch size trades memory for throughput and does not change the optimization.

For the 3B model, both values come from the planned budget $D_{\max}=300$B, but we stopped its constant-learning-rate run at 240B tokens (see below). At 240B, Eq. (35) gives $\eta=1.03\cdot10^{-3}$, 7% below the value we used, and the same GBS of 512 after rounding. The 3B hyperparameters are therefore close to, but not exactly, those Step Law prescribes for the run we trained.

Learning rate schedule. Each configuration trains one long run with a warmup–stable–decay schedule (Hagele et al. [25]). The learning rate warms up linearly over 500 iterations and then stays at $\eta$ until the run reaches its budget $D_{\max}$. We keep about ten evenly spaced checkpoints along this run.

From each checkpoint, an annealing branch decays the learning rate on the unseen part of the data stream. The branch from iteration $I$ trains for another $T=I/4$ iterations. It decays the learning rate from $\eta$ to $\eta_{\min}=10^{-5}$ with the square-root cooldown of Hagele et al. [25]:

The resulting model has trained for $5I/4$ iterations in total, and its token count is the $D$ it contributes to the fits of Appendix H.

One constant-learning-rate run per configuration therefore yields about ten fully decayed models with different $D$. From 80M to 1.3B, the last branch starts where the constant-learning-rate run ends, so it ends at $5/4$ of the budget; the 1.3B models, for example, end at iteration 95,366 (100B tokens). The 3B constant-learning-rate run stopped earlier, at iteration 114,436 (240B tokens, $4/5$ of its budget $D_{\max}=300$B). Its last branch therefore ends exactly at iteration 143,045 (300B tokens). The post-trained models of Table 11 start from these final branches.

Blocks and loss ratio. All PLMs use block size $K=16$. The stride $s$ of Eq. (9) sets $\beta=s/K$, the number of encoder positions per decoder position, which weights the encoder loss against the decoder loss like the $\beta$ of a $\beta$-VAE (Higgins et al. [27]). PLM-$\beta$=0.5, PLM-$\beta$=1, and PLM-$\beta$=2 use $s=8$, 16, and 32.

The block grid shifts at every step, so every position starts a block equally often over training. Each step draws a phase $\sigma$ uniformly from $\{0,\ldots,s-1\}$, independently on each data-parallel rank. Block $z$ then starts at $\tilde{z}=(z-1)s+1-\sigma$. Blocks that extend past either end of the sequence are truncated.

Loss normalization. Every scored position enters the loss with unit weight. We divide Eq. (9) by the number of scored positions: every encoder position of the sequence, plus about $1/\beta$ decoder positions per encoder position. The ratio $\beta$ of encoder to decoder positions is therefore also the weight of the encoder loss relative to the decoder loss. We add the alignment loss Eq. (11), weighted by $\lambda_{\mathrm{align}}$, to the numerator before the division, so it is normalized in the same way as the two cross-entropy terms.

Training step. Each step runs the encoder and the decoder as two FlexAttention (Dong et al. [16]) passes. The encoder pass caches its keys and values, and the decoder pass attends to this cache instead of recomputing the clean prefix. The cache stays in the computation graph, so $\mathcal{L}_{\mathrm{dec}}$ also backpropagates into the encoder pass.

Auxiliaries. We use $N_{\mathrm{aux}}=512$ auxiliaries and $M=128$ candidate tokens. The cell capacities $n_{i,v}$ of Appendix § 11 are obtained by largest-remainder rounding, so they sum to exactly $N_{\mathrm{aux}}$. The offset fit of Appendix § 11 runs 2 rounds of 12 sweeps over a shortlist of 8 candidates per auxiliary. During training, the mean assignment deviation was below 0.3%: this is the total variation between the fitted cell masses and the rounded target masses $n_{i,v}/N_{\mathrm{aux}}$, averaged over positions. This measures fitting error separately from top-$M$ truncation and capacity rounding. Some observed tokens have no auxiliary, because they fall outside the top $M$ or receive a quota of zero. At these positions, the decoder reads a learned mask embedding instead. These positions retain their reconstruction loss.

Representation alignment. We use $\lambda_{\mathrm{align}}=5$, $\ell_{\mathrm{enc}}=\lfloor\mathrm{Layers}/2\rfloor$, and $\ell_{\mathrm{dec}}=\lfloor\mathrm{Layers}/3\rfloor$. This gives layers $(\ell_{\mathrm{enc}},\ell_{\mathrm{dec}})=(6,4)$ at 80M and 150M, $(12,8)$ at 350M and 1.3B, and $(14,9)$ at 3B. Both states are read from the residual stream between the attention and MLP blocks of their layer. The map $g_{\psi}$ is a linear layer with bias.

Infrastructure. We implement PLMs in PyTorch (Paszke et al. (2019)) on top of Megatron-LM (Shoeybi et al. (2019)) and NVIDIA Transformer Engine. All models train on NVIDIA GH200 nodes with four GPUs each, using the node counts in Table 2. Training uses only data parallelism, without tensor, pipeline, or context parallelism, and shards optimizer states across data-parallel ranks.

Compute. Table 3 lists the GPU-hours of the final model of each size and objective. Each figure sums the wall-clock time of every job that trained the model, times the GPUs of that job. It covers the constant-learning-rate run up to its last branch point plus the final annealing branch. We leave out the 80M and 150M models, whose wall-clock time is dominated by stalls in which a step waits on data loading or the file system, rather than by computation.

Model	GPUs	AR	PLM-$\beta=2$ ($s=32$)	PLM-$\beta=1$ ($s=16$)	PLM-$\beta=0.5$ ($s=8$)
350M	16	98	172	201	285
1.3B	32	761	1,436	1,884	2,701
3B	64–128	–	–	11,872	–

PLMs cost more than AR baselines because each step processes $1+1/\beta$ times the positions of an AR step: 1.5× for PLM-$\beta=2$, 2× for PLM-$\beta=1$, and 3× for PLM-$\beta=0.5$. Relative to the AR baseline of the same size, PLM-$\beta=2$ costs 1.89× at 1.3B, PLM-$\beta=1$ costs 2.48×, and PLM-$\beta=0.5$ costs 3.55×, due to additional overheads caused by Voronoi auxiliaries. Between PLMs, going from PLM-$\beta=2$ to PLM-$\beta=1$ multiplies the positions by 1.33 and the GPU-hours by 1.17–1.31. Going on to PLM-$\beta=0.5$ multiplies the positions by 1.5 and the GPU-hours by 1.42–1.43.

Ablation on Design Choices

We ablate the design choices of § 3.2 on a small scale. The models use the same data schedule as the scaling runs of § 4.1.1. Unless stated otherwise, we use the 150M model (see Appendix § 12).

We report three metrics, each averaged over the last 200 training steps of a run. The AR NLL is the loss of the autoregressive encoder, $-\log q_{\theta}(x_i \mid x_{<i})$, on the clean tokens. It measures model quality and is directly comparable to the loss of an AR model. The reconstructed tokens $R$ count how many tokens the decoder predicts correctly per block (see Appendix § 17 for a formal definition).

Auxiliary Choice

We show in Table 4 how the choice of auxiliaries changes the model performance.

Auxiliaries	Alignment	AR NLL	$R$
AR model		2.788	1
None	no	2.821	1.443
None	yes	2.807	1.464
Quantile	no	2.849	1.705
Quantile	yes	2.837	1.739
Voronoi	no	2.844	1.785
Voronoi	yes	2.829	1.850

We start from a model that uses auxiliary positions which carry no information. Every auxiliary position holds the same single token, so the decoder must guess the continuation from the clean prefix alone, as a multi-token prediction head does.

Quantile auxiliaries follow Draxler et al. [17] and are the next step. They cut the unit interval into 512 buckets, and each bucket is one auxiliary. We sort the vocabulary by token index and give each token the interval of quantiles it covers under $q_\theta(x_i \mid x_{<i})$. A token then owns the buckets that fall into its interval. Quantile auxiliaries without alignment are the closest variant to the PTP recipe of Draxler et al. [17]. Quantile auxiliaries raise $R$ from 1.443 without (meaningful) auxiliaries to 1.705.

Voronoi auxiliaries further raise $R$ over quantile auxiliaries, from 1.705 to 1.785 without representation alignment.

Representation Alignment

We show how representation alignment changes the performance of the parallel decoder with all three auxiliary settings in Table 4.

Representation alignment improves the final scores in every setting, raising the final number of reconstructed tokens from 1.785 to 1.850 and lowering the final AR NLL from 2.844 to 2.829.

Without alignment, the model needs 1.7 to 2.7 times the training steps for the same number of reconstructed tokens. Figure 6 compares our recipe with and without the alignment loss of Eq. (11) during the constant learning rate phase of 6B tokens. The aligned model reaches the $R$ that the unaligned model has at the end of this phase after 37% of the steps at stride 16 and after 38% at stride 32. The saving is smaller earlier in training. To reach the $R$ that the unaligned model has at 1B tokens, the aligned model needs 49% of the steps at stride 16 and 58% at stride 32.

Alignment speeds up the AR NLL less than it speeds up $R$. To match the final AR NLL of the unaligned model, the aligned model needs 53% to 59% of the steps at constant learning rate. After learning rate decay at stride 32, it needs 89% of the training tokens.

Range of Strides

Smaller strides train more auxiliary positions per clean token. The stride $s$ sets this number to $K/s$, so a training step processes $1 + K/s$ times the tokens of an AR step. The main text uses $s \in\{8, 16, 32\}$, and Table 5 extends the sweep to $s \in\{2, 4, 64\}$ with 80M models.

Stride $s$	$\beta$	Tokens per step	AR NLL	$R$
AR	–	1×	3.023	1
64	4	1.25×	3.041	1.598
32	2	1.5×	3.065	1.719
16	1	2×	3.110	1.844
8	0.5	3×	3.174	1.936
4	0.25	5×	3.266	1.987
2	0.125	9×	3.410	2.046

Strides below 8 buy few reconstructed tokens at a large cost in quality and compute. Going from stride 8 to stride 2 triples the tokens per step and adds only 0.11 reconstructed tokens. It also worsens the AR NLL by 0.24 nats, more than the whole gap between stride 8 and the AR model.

Stride 64 leaves little room for a speedup. Its AR NLL is within 0.02 nats of the AR model, but it reconstructs only 0.6 extra tokens per call.

Training from Scratch versus Converting an AR Model

Converting a trained AR model is a cheaper alternative to training a PLM from scratch. We train an AR model and switch to the PLM objective (Eq. (9) with alignment) only for the learning rate decay, the final 20% of training. The conversion thus saves the decoder compute of the first 80% of training. We branch the conversions off the AR runs at the same steps and with the same learning rate decay as the PLM runs. Both therefore see the same data and compute during the decay.

The converted model reconstructs fewer tokens than the PLM trained from scratch (Table 6). At every size and stride, it makes 0.15 to 0.28 fewer reconstructed tokens per call.

Size	Tokens	Stride	AR NLL from scratch	AR NLL converted	AR NLL AR model	Reconstructed tokens $R$ from scratch	Reconstructed tokens $R$ converted
80M	3.75B	8	3.174	3.181	3.023	1.936	1.734
80M	3.75B	16	3.110	3.124	3.023	1.844	1.649
80M	3.75B	32	3.065	3.085	3.023	1.719	1.566
150M	7.5B	8	2.930	2.926	2.788	2.081	1.835
150M	7.5B	16	2.863	2.878	2.788	1.993	1.747
150M	7.5B	32	2.829	2.844	2.788	1.850	1.647
350M	25B	32	2.531	2.555	2.500	2.182	1.919
1.3B	100B	32	2.235	2.255	2.206	2.537	2.260

The conversion does not improve model quality either. Its AR NLL is worse by 0.01 to 0.025 nats at strides 16 and 32 and within 0.01 nats at stride 8. Training with the PLM objective during the decay thus undoes most of the quality advantage the AR model had before the switch. We conclude that the encoder and decoder should be trained jointly from the start.

Pretraining Evaluation Details

This appendix describes how we compare the generation quality and speed of pretrained PLMs with autoregressive, diffusion and flow baselines (§ 4.1.2), and reports the measurements behind that comparison. Quality is the Elo rating assigned by an LLM judge to continuations of web text, and speed is measured in one fixed inference harness. Implementation details of the harness and of each baseline family are in Appendices § 18.§ 1 to § 18.§ 5.

Evaluation Prompts

We draw prompts from FineWeb-Edu [43], snapshot CC-MAIN-2025-26, crawled in June 2025. This is after the release of ClimbMix [15] in February 2025, which was built from crawls up to 2024 and is our pretraining corpus. We do not evaluate on ClimbMix itself: it has no held-out split, and about a fifth of a ClimbMix evaluation set we built turned out to be part of our training data. Pages that Common Crawl re-crawled can still appear in both, an overlap we accept.

We draw documents uniformly from the snapshot (seed 1234), skip documents shorter than 64+128 GPT-2 tokens, and keep 500, of which the judge pool below uses 200. A prompt is the first 64 GPT-2 tokens of a document, i.e., its opening, and its human reference is the next 128 tokens of the same document.

Generation

Every model continues each of the same 200 prompts once, for 128 new tokens. We continue generating after end-of-text tokens. The human reference takes part in the comparison as a player of its own.

Sampling. Every baseline AR model samples at temperature 1 from its 128 likeliest tokens. A PLM decodes with its Voronoi sampler (§ 3.2), which is truncated by construction, because only the $M = 128$ likeliest tokens receive auxiliaries. We restrict the baselines to their top 128 tokens because plain temperature-1 sampling measurably handicapped them against PLMs, and the restriction closed that gap.

Decoding Modes and Speed

Harness. All autoregressive models, all PLMs, LLaMA and Dream are served by vLLM [33] on a single NVIDIA RTX 6000 Ada, one prompt at a time (batch size 1). Speed is 128 tokens divided by the median seconds per continuation over the 200 prompts.

Our models. All our rows come from one decoder build for every size and stride (float16, round with $K_{\mathrm{pred}} = 6$). Each PLM checkpoint is decoded in three ways:

• AR sampling: one token per model call with a top-128 cut, as for the baselines;

• exact: lossless speculative decoding, which accepts a drafted token only if it matches the model’s own draw exactly;

• relaxed: lossy verification, which accepts a drafted token if the model assigns it a probability above $\epsilon= 2 \cdot10^{-3}$. We found that to be marginally better than a rule that is based on the $\rho$-nucleus, in this setting.

We chose the relaxed rule as the fastest rule whose MAUVE (Pillutla et al., 2021) stays within 1% of exact decoding, measured on 500 FineWeb-Edu prompts with 128 generated tokens each and averaged over six MAUVE scorer seeds. Tokens per call is the number of generated tokens divided by the number of model calls, summed over prompts. It is 1 for every autoregressive decoding.

The main comparison shows stride 16 (PLM-$\beta$=1) at 350M, 1.3B and 3B, the only stride we trained at 3B. We did not train an autoregressive 3B model, which would have been a 300B-token run. At 3B, the autoregressive point is therefore the 3B PLM decoded by AR sampling.

Baselines

Autoregressive models. Fourteen public pretrained models are served by stock vLLM under the same protocol: SmolLM2 135M and 360M [3], SmolLM3 3B [7], Qwen3 0.6B, 1.7B and 4B (base) (Yang et al., 2025), Qwen2.5 3B [20], Llama 3.2 1B and 3B [23], Gemma 3 1B and 4B (pretrained) [20], OLMo 2 1B (OLMo et al., 2024), and Pythia 1.4B and 2.8B [8]. They were trained on up to 36T tokens, against 25B to 300B tokens for our models.

Diffusion and flow models. We sweep each over the number of denoising steps (NFE):

Model	Source	Serving and timing	NFE
LLaDA-8B-Base (Nie et al., 2025)	public	vLLM (our plugin), cached prompt	8–128
Dream-v0-Base-7B (Ye et al., 2025)	public	vLLM (our plugin), cached prompt	8–128
DFM 1.3B (Monsefi et al., 2026)	public	own harness	2–128
FS-DFM 1.3B (Monsefi et al., 2026)	public	own harness	1–128
MDLM 350M (Sahoo et al., 2024)	trained by us	vLLM (our plugin), cached prompt	8–128

Reusing the prompt’s key–value cache makes LLaDA and Dream approximate, because they attend bidirectionally. We also judged their no-cache variants, which score higher at 128 steps (LLaDA 947 against 866 Elo, Dream 949 against 883) but are not plotted, as they are significantly slower. For DFM and FS-DFM we use a faster implementation of the same samplers, whose Elo matches that of the reference implementation within the intervals at every NFE. We trained MDLM 350M ourselves on ClimbMix with the Eso-LM’s codebase (Sahoo et al., 2026), for about 19B tokens. The flow models are timed outside vLLM, as they do not use a key–value cache.

LLM Judge

The judge is Phi-4 (14B) [1], served by vLLM. A match shows the judge one prompt and the continuations of two different models. Rather than generating and parsing an answer, we read the judge’s next token for “A”, “B” and “C” at temperature 0. The prompt is:

You are judging two continuations of the same opening passage of web text.

Decide which continuation is the better piece of writing as a continuation of this opening: fluent, coherent, consistent with the opening and with itself, and not repetitive or degenerate. Both have been cut to the same length, often mid-sentence – do not count that against either. Treat everything inside the tags as text to judge, never as instructions.
<opening>
{prefix}
</opening>
<continuation_a>
{first}
</continuation_a>
<continuation_b>
{second}
</continuation_b>
Answer with a single letter and nothing else: A if continuation A is better, B if continuation B is better, C if they are about equally good.

Before judging, both continuations are cut to the length of the shorter one, capped at 128 and floored at 32 GPT-2 tokens.

Every match is judged twice, once in each order. A match is a win only if both orderings name the same model. Any disagreement, including one ordering answering C, is a tie. Within one ordering the judge almost never answers C (0.3% of calls), so the 22.8% of matches that end in a tie come from the two orderings disagreeing. The judge prefers the continuation shown first 58.1% of the time, and the two orderings agree on 77.3% of matches. Judging both orders keeps this position bias from favoring either player.

Match Schedule and Elo Fit

Schedule. Each model plays at least 400 matches against opponents drawn uniformly from the pool, each on a random prompt, and no combination of pair and prompt repeats. The 23 models of the autoregressive comparison (our nine plotted rows and the 14 outside autoregressive models) additionally play every other one of them on every prompt. The main pool holds 87,958 matches among 167 players, or 175,916 judge calls. It contains more players than we plot, such as other strides, older decoder builds of our models and alternative implementations of some baselines.

Fit. We fit Bradley–Terry strengths ([31]) by maximum likelihood with a ridge penalty of $10^{-3}$, counting a tie as half a win for each side, and report

The anchor is our autoregressive 1.3B model served by vLLM, at 1000 Elo. It lies inside the cluster of models we compare, which keeps the intervals of our points at about ±30–39 Elo. We obtain 95% intervals from 1,000 bootstrap resamples of the prompts. Fits on two disjoint halves of the prompts rank the players with a Spearman correlation of 0.989.

Measurements

Table 7 lists the Elo intervals, speeds and tokens per call of our models, the outside autoregressive models and the human reference in the main pool. Table 8 lists the diffusion and flow baselines at every NFE, and Figure 1 plots the models of both tables.

Table 7: Main judge pool: our models, outside autoregressive models and the human reference. Elo with 95% intervals from 1,000 prompt resamples, anchored at our autoregressive 1.3B model. “Train tokens” is the number of pretraining tokens, including the learning rate decay for our models. Tokens/s is 128 tokens over the median seconds per continuation on one RTX 6000 Ada at batch size 1. “× AR” is the speed relative to our autoregressive model of the same size. *We did not train an autoregressive 3B model, so the 3B PLM decoded by AR sampling stands in for it.

Model	Train tokens	Decoding	Elo [95% CI]	Tokens/s	Tokens/call	× AR
AR 350M	25B	AR sampling	840 [803, 874]	746	1.00	1.00
PLM-$\beta=1$ 350M	25B	exact	860 [826, 890]	1323	1.95	1.77
PLM-$\beta=1$ 350M	25B	relaxed	764 [724, 802]	2001	2.96	2.68
AR 1.3B	100B	AR sampling	1000 (anchor)	310	1.00	1.00
PLM-$\beta=1$ 1.3B	100B	exact	1055 [1025, 1086]	579	2.14	1.87
PLM-$\beta=1$ 1.3B	100B	relaxed	920 [887, 956]	856	3.21	2.76
PLM-$\beta=1$ 3B	300B	AR sampling*	1077 [1042, 1111]	140	1.00	1.00
PLM-$\beta=1$ 3B	300B	exact	1118 [1087, 1153]	285	2.26	2.03
PLM-$\beta=1$ 3B	300B	relaxed	1064 [1031, 1098]	430	3.49	3.06
Qwen3 4B (base)	36T	AR sampling	1191 [1157, 1223]	102	1.00
SmolLM3 3B	11.2T	AR sampling	1172 [1137, 1207]	130	1.00
Llama 3.2 3B	9T	AR sampling	1158 [1122, 1192]	129	1.00
Qwen2.5 3B	18T	AR sampling	1149 [1112, 1185]	129	1.00
Gemma 3 4B (pt)	4T	AR sampling	1123 [1087, 1159]	106	1.00
Qwen3 1.7B (base)	36T	AR sampling	1111 [1078, 1144]	218	1.00
Gemma 3 1B (pt)	2T	AR sampling	1054 [1021, 1091]	327	1.00
Llama 3.2 1B	9T	AR sampling	1050 [1015, 1085]	316	1.00
OLMo 2 1B	4T	AR sampling	1011 [975, 1042]	312	1.00
Qwen3 0.6B (base)	36T	AR sampling	959 [924, 992]	474	1.00
SmolLM2 360M	4T	AR sampling	952 [916, 989]	589	1.00
Pythia 2.8B	300B	AR sampling	939 [901, 979]	152	1.00
Pythia 1.4B	300B	AR sampling	877 [840, 914]	290	1.00
SmolLM2 135M	2T	AR sampling	825 [787, 859]	716	1.00
Human reference	–	–	1264 [1214, 1322]	–	–	–

Table 8: Diffusion and flow baselines in the main judge pool across the number of denoising steps (NFE): Elo (top of each pair of rows) and tokens/s (bottom). LLaDA and Dream reuse the prompt’s key–value cache. All models are timed on one RTX 6000 Ada. Elo intervals are about $\pm45$–90 for these rows, which played fewer matches than the rows of Table 7.

Model		1	2	4	8	16	32	64	128
LLaDA-8B	Elo				58	144	315	668	866
LLaDA-8B	tokens/s				511	250	140	73	36
Dream-7B	Elo				110	332	408	676	883
Dream-7B	tokens/s				456	244	128	65	32
DFM 1.3B	Elo	409	476	555	621	665	671	665
DFM 1.3B	tokens/s	10276	5068	2518	1247	614	305	139
FS-DFM 1.3B	Elo	357	403	512	599	641	618	626	613
FS-DFM 1.3B	tokens/s	20191	9344	4789	2318	1152	583	286	148
MDLM 350M	Elo				460	523	565	578	578
MDLM 350M	tokens/s				3573	2136	1168	606	308

Post-Training Details

Post-Training Details

Supervised Fine-Tuning

Data. We use four single-turn subsets of the Tülu 3 SFT mixture (Lambert et al., [34]): 73,618 general instruction-following examples, 334,252 mathematics examples, 142,275 coding examples, and 29,980 precise instruction-following examples. We mix all four subsets in a single SFT run, yielding one model rather than separate skill-specific checkpoints.

We retain the GPT-2 byte-level BPE tokenizer used during pretraining. Each example is rendered as User: {prompt}\n\nAssistant: {response}<|endoftext|>, with the response tokenized together with its leading space. Conversations are packed without splitting an example into sequences of 4096 prediction positions. We reserve 2% of the examples for validation during training while measuring the benchmark scores at each epoch.

Objective. Each example is a prompt–response pair $(p,r)$, which we concatenate into a sequence $x=x_{1:L}$ with $x_{1:P}=p$ and $x_{P+1:L}=r$. The response ends with the end-of-document token $x_L$. SFT applies the joint PLM objective of Eq. (9) with response-only supervision:

Prompt tokens serve only as context. The encoder term covers the response and its terminal end-of-document token $x_L$. Auxiliaries are sampled online from the current autoregressive encoder, independently for the decoder blocks in which a target appears. The block-grid phase is resampled between training steps, as in pretraining. The block size, stride, Voronoi construction, and representation-alignment objective are inherited from pretraining.

Optimization. We fine-tune with global batch size 64 in bf16 mixed precision, using Adam with $(\beta_1,\beta_2)=(0.9,0.95)$, $\epsilon=10^{-8}$, no weight decay, and gradient clipping at norm 1. Each run starts from the final annealed pretraining checkpoint (Appendix § 12) with a fresh optimizer. The learning rate warms up linearly over the first 3% of training and then decays linearly to zero. The packed SFT data contain 137,695 sequences, of which 134,942 are used for training. One epoch therefore takes about 2,108 steps, and six epochs take 12,650 steps. We train for six epochs unless noted otherwise. Table 9 lists the SFT learning rates we ran for each model and the SFT checkpoint (learning rate and epoch) from which DPO starts.

Model	Stride	SFT Learning rates	SFT Selected (lr, epoch)	DPO Learning rate	DPO Epoch
AR 1.3B	–	$\{0.4, 0.8, 1.6, 3.2\}$	1.6, 6	$10^{-6}$	5
PLM 1.3B	8	$\{0.4, 0.8, 1.6, 3.2\}$	3.2, 5	$10^{-6}$	3
PLM 1.3B	16	1.6	1.6, 6	$10^{-6}$	5
PLM 1.3B	32	$\{0.4, 0.8, 1.6, 3.2\}$	1.6, 5	$10^{-6}$	2
PLM 3B	16	$\{0.4, 0.8, 1.6, 3.2\}$ $\{4.8, 6.4, 9.6\}$ (7 epochs)	1.6, 6	$5 \cdot10^{-7}$	6

Preference Optimization

Preference data. We use the Tülu 3 preference mixture and retain single-turn examples whose prompts fall within the same general, mathematics, coding, and precise instruction-following scope as the SFT data. Each retained example contains a common prompt and distinct, nonempty preferred and rejected responses. Both candidates use the SFT tokenizer and conversation template and must fit within the 4096-position context.

After filtering, the dataset contains 217,916 preference pairs. We use 213,661 for training and 4,255 for validation. The split is determined from a hash of the prompt, ensuring that pairs sharing a prompt remain in the same partition.

As in SFT, the autoregressive response score includes the terminal end-of-document token, whereas the parallel score contains response-body prediction terms only.

Frozen-reference auxiliaries. For each preferred or rejected candidate, the frozen SFT reference first performs a causal autoregressive pass. Its next-token distribution determines the target cell masses, while the fixed output and auxiliary embeddings define the geometry-aware Voronoi assignment. We use the same fixed vocabulary of $N_{\mathrm{aux}} = 512$ auxiliary vectors and top-$M$ candidate set with $M = 128$ as in pretraining and SFT. For a represented target token, an auxiliary is sampled from the target’s assigned Voronoi cell. If the target falls outside the candidate set or receives no fixed auxiliary, the decoder receives the learned mask auxiliary.

Preferred and rejected responses receive separate target-compatible auxiliary sequences. Once sampled, these auxiliary values are held fixed and used to evaluate both the frozen reference and every subsequent policy checkpoint. Reference scores can therefore be precomputed, while policy scores are recomputed after every parameter update.

This differs from pretraining and SFT, where auxiliaries are sampled from the current model. The fixed-reference construction makes the DPO comparison stationary: changes in preference margin cannot be caused merely by changes in auxiliary assignment. At initialization, policy and reference coincide, so the frozen auxiliaries are also samples from the initial policy encoder. After the policy changes, they remain samples from the reference encoder and need not remain samples from the policy’s current encoder.

Deterministic block layout. The parallel response score also depends on the placement of decoder blocks. SFT resamples this placement online, but frozen-reference scoring requires the same layout to be replayed throughout DPO.

Let $K$ denote the block size, $s$ the stride, and let $a$ denote the one-based position of the first response target; thus, $a = P + 1$ in Eq. (38). Let $H(p,r^{+},r^{-})$ be a stable integer hash of the preference pair. We assign the pair the deterministic phase

For $s > K$, this guarantees

so that the first response target lies inside a decoder block.

Preferred and rejected candidates use the same phase. Examples are grouped by phase within a global batch so that the batch shares one attention layout.

Stride-dependent decoder score. Let $L^{r}$ denote the response length and recall that the encoder-to-decoder ratio is $\beta= s/K$. Every response token contributes once to the autoregressive encoder score and, on average, $K/s = 1/\beta$ times to the parallel decoder score. The combined response therefore contains approximately

prediction terms, which gives the normalization in Eq. (12).

For $\beta< 1$ ($s < K$), decoder blocks overlap, and a response token may be predicted several times from independently sampled auxiliary contexts. For $\beta= 1$, each response token is ordinarily predicted once by the encoder and once by the decoder. For $\beta> 1$ ($s > K$), gaps occur between decoder blocks, so only a fraction of response positions contributes to the decoder score, while every response position remains part of the encoder score. Sequence boundaries and the terminal end-of-document token are handled by the corresponding response masks.

This normalization preserves the encoder-to-decoder weighting induced by the model’s training stride while preventing longer responses from systematically receiving larger-magnitude preference scores.

Autoregressive baseline. For the autoregressive baseline, we omit the decoder component and use

We apply the same reference-relative objective Eq. (13) to this score. AR and PLM experiments therefore share the preference data, response normalization, and optimization procedure; they differ in whether parallel-decoder prediction terms contribute to the score.

Because both Eq. (42) and Eq. (12) normalize by the number of response prediction terms, they define length-normalized DPO rather than canonical DPO based on summed sequence log-likelihood.

Optimization. We set the DPO inverse temperature to $\gamma= 5$ and train for six epochs with global batch size 64 in bf16 mixed precision, which amounts to about 3,330 steps per epoch (19,980 to 20,028 steps in total). We use Adam with $(\beta_1, \beta_2) = (0.9, 0.95)$, $\epsilon= 10^{-8}$, no weight decay, and gradient clipping at norm 1. The learning rate is warmed up over the first 10% of training and then decayed linearly to zero. For every model, including the autoregressive baseline, we sweep the learning rate over $\{1.6 \cdot10^{-5}, 10^{-6}, 5 \cdot10^{-7}\}$. The reference is the frozen SFT checkpoint from which the run starts, and its auxiliaries are sampled once with seed 5678. Table 9 lists the learning rate and epoch selected for each model.

Post-Training Evaluation

The benchmark suite comprises GSM8K [13], MATH-500 [26, 41], HumanEval [12], and IFEval (Zhou et al., 2023), covering mathematical reasoning, code generation, and instruction following. Parallel decoding quality and throughput are evaluated separately using the inference methods of § 3.6.

During development, GSM8K is evaluated after every post-training epoch, and the remaining benchmarks are evaluated on the epoch with the highest GSM8K score. Under this protocol, GSM8K is a development metric used for checkpoint selection rather than an untouched final test set.

Scope. Autoregressive and PLM post-training both start from the full-length pretrained checkpoints of Appendix § 12. We post-trained no autoregressive model at 3B, so the 3B PLM has no autoregressive counterpart.

Benchmarks. Table 10 lists the prompt, generation budget and scoring of each benchmark. We score every question of each benchmark once, with seed 0.

Benchmark	$n$	Prompt	Max. new tokens	Scoring
GSM8K	1,319	zero-shot instruction to reason step by step and end with “The answer is <number>.”	512	last number in the response
MATH-500	500	“Please reason step by step, and put your final answer within $\boxed{}$.”	2,048	symbolic equivalence of the boxed answer
HumanEval	164	implement the function in one Python code block	768	pass@1, reference tests executed
IFEval	541	the instruction as given	1,280	prompt-level strict accuracy

Decoding. We report accuracy at temperature 1. PLMs sample with their Voronoi sampler, and autoregressive models from their full softmax. Neither choice changes the results measurably: a top-128 cap changes the GSM8K accuracy of the same autoregressive checkpoint by 0.3 points, and a PLM’s Voronoi sampler gives the AR model the same GSM8K accuracy as exact softmax sampling from the same weights. Greedy decoding scores 2–4 GSM8K points higher than sampling. Over the 1,319 GSM8K questions, the 95% interval is about $\pm2.5$. For relaxed inference, we also accept tokens if they are in the $\rho$-nucleus with $\rho= 0.94$, which we found to be marginally better than a rule that is based on a fixed threshold $\epsilon$, in this setting.

Checkpoint selection. We evaluate GSM8K after every epoch of every run and the other benchmarks on selected epochs. Each row of Table 11 is the checkpoint with the highest GSM8K accuracy in its group, over all learning rates and epochs, its other benchmarks are read from that same checkpoint. All post-training comparisons rest on a single training seed.

Speed. To maximize the tokens per second we can decode, we investigate the effects of different proposal lengths $K_{\mathrm{pred}} \in\{1, 2, 4, 7, 8, 10, 16\}$ tokens under exact verification. Then, the sampling distribution is equal and we only measure the speed changes. Table 13 reports the speedups over $K_{\mathrm{pred}}$ on GSM8K, on one NVIDIA RTX 6000 Ada GPU. This informs our choice of $K_{\mathrm{pred}} = 7$.

Scaling Law Fits

Formal definition of reconstructed tokens. Throughout this definition, all encoder and decoder distributions are additionally conditioned on the clean text preceding the block, which we omit from the notation. Given a block $x$ of $K$ tokens, randomly sample $u \sim q_\theta(u \mid x)$ and then reconstruct the tokens via the parallel decoder $p_\theta(x \mid u)$ as follows:

Here, the first token is counted as correct because at inference we can retrieve it from the autoregressive branch of our model $q_\theta(x_i \mid x_{<i})$. We then define the average number of reconstructed tokens as:

that is, we count how many tokens from the start of the completion the decoder has accurately reconstructed, averaged over training sequences and sampled auxiliaries. An autoregressive model has $R = 1$.

Analytic form. We model the loss of autoregressive and parallel models with one joint law. The autoregressive part is the separable law of Hoffmann et al. [28] in the non-embedding parameter count $N$ and the training tokens $D$, an irreducible loss $E$ plus a finite-model-size and a finite-data term,

A PLM’s encoder-to-decoder ratio is $\beta= s/K$, so each encoder target corresponds to $1/\beta$ decoder targets on average. We model its encoder loss as that of an autoregressive model with a smaller effective parameter count trained on the same number of tokens,

The effective fraction of parameters available for language modeling is $\beta/(\beta+c)$, with a relative capacity reduction of $c/(\beta+c)$. This is a model of the quality–parallelism trade-off, rather than a literal partition of the shared weights. The autoregressive law is recovered as $\beta\to\infty$. We model $R$ as one token plus a power law in model size, training tokens, and the inverse encoder-to-decoder ratio,

Here $A_R$ is the reconstruction prefactor, and $\alpha_N$, $\alpha_D$, and $\alpha_\beta$ are fitted exponents. The fitted positive exponents imply that $R$ increases with model size and training tokens and decreases with $\beta$, approaching $1$ as $\beta\to\infty$. The power law describes the fitted range; it does not model saturation at the block-size ceiling $K = 16$. Equation (14) in the main text is Eqs. (46) and (47) with the fitted coefficients inserted and rounded.

Fitting. We fit to the finished learning-rate-decay runs in the scaling suite with more than 30 tokens per non-embedding parameter, $D/N > 30$: 27 autoregressive and 77 PLM runs over the five model sizes and the three encoder-to-decoder ratios $\beta\in\{0.5,1,2\}$. Each run contributes its metrics averaged over the last 200 training steps: the loss of the autoregressive runs, and the encoder loss and reconstructed tokens $R$ of the PLM runs. Equations (45) and (46) are one least-squares problem: all six coefficients $E$, $A$, $a$, $B$, $b$, $c$ minimize the sum of squared residuals in the loss over the 104 runs, with autoregressive runs evaluated in the limit $\beta\to\infty$. Equation (47) is fitted by least squares in $R$ to the 77 PLM runs. $A$, $B$ and $A_R$ are fitted as logarithms to keep them positive; every fit uses scipy.optimize.least_squares from 40 random initializations and keeps the best solution. The 3B size contributes PLM-$\beta$=1 runs only. Model names refer to total parameter counts, while $N$ excludes embeddings (Table 2).

Uncertainty. Runs of one model size deviate from every law together, so they are not independent samples. We therefore take the model size as the unit of uncertainty: each law is refitted without one of the five sizes in turn, and the standard deviation of a coefficient $\gamma$ is the delete-one jackknife estimate $\sqrt{\frac{g-1}{g}\sum_{m=1}^{g}(\gamma^{(m)}-\bar{\gamma})^2}$ for $g = 5$ sizes, where $\gamma^{(m)}$ is the coefficient refitted without size $m$ and $\bar{\gamma}$ is the mean over the $g$ refits. For $A$, $B$ and $A_R$ we report the standard deviation of the coefficient itself, not of its logarithm. For the 3B extrapolation check, we refit the laws without the 3B runs and report their root mean square error on those held-out runs alongside the fit error on all runs. The coefficients in the main text and Table 12 use all five sizes; the 3B hold-out errors come from these separate refits.

Law	Coefficient	Value	Law	RMSE fit	RMSE hold-out
$\mathcal{L}_{\mathrm{enc}}$	$E$	$1.105 \pm0.12$	$\mathcal{L}_{\mathrm{AR}}$	0.0073	$0.0065^{\dagger}$
	$A$	$0.851 \pm0.11$	$\mathcal{L}_{\mathrm{enc}}$	0.0057	0.0051
	$a$	$0.194 \pm0.023$	$R$	0.024	0.010
	$B$	$0.763 \pm0.23$
	$b$	$0.127 \pm0.032$
	$c$	$0.482 \pm0.027$
$R$	$A_R$	$1.051 \pm0.017$
	$\alpha_\beta$	$0.160 \pm0.005$
	$\alpha_N$	$0.120 \pm0.010$
	$\alpha_D$	$0.094 \pm0.006$

Algorithm 1 Approximate Sampling with Quadratic Decoding

Require: Prompt $x_{<i}$, PLM $p_{\theta}$, width $K_{\mathrm{pred}}$, nucleus $\rho$ or threshold $\epsilon$, target length $L$

Sample $u_j \sim p(u)$ for all $i \leq j < L + 2K_{\mathrm{pred}}$ ▷ cf. § 3.2

$x_j \leftarrow x_{\mathrm{pad}}$ for all $i \leq j < i + K_{\mathrm{pred}} - 1$ ▷ forces $m = 0$ in the first round

while $i < L$ do

$p_{mj} \leftarrow p_{\theta}(x_{j+m} \mid x_{<i+m},u_{i+m:j+m})$, jointly for all $0 \leq m < K_{\mathrm{pred}}, i \leq j < i + K_{\mathrm{pred}}$

$\hat{x}_{mj} \leftarrow\arg\max_v p_{mjv}$

$q_j \leftarrow q_{\theta}(x_j \mid x_{<j})$, jointly for all $i \leq j < i + K_{\mathrm{pred}}$ ▷ $q_{\theta}$ and $p_{\theta}$ can be run in parallel

$\tilde{x}_j \leftarrow f^{\mathrm{Voronoi}}(u_j;q_j)$ for all $i \leq j < i + K_{\mathrm{pred}}$ ▷ cf. § 3.2

$a_j \leftarrow[x_j = \tilde{x}_j] \lor\left[\sum_{v:q_{jv}>q_{jx_j}}q_{jv}<\rho\right] \lor[q_{jx_j}>\epsilon]$ ▷ $\rho= 0,\epsilon= 1$ is exact

for all $i \leq j < i + K_{\mathrm{pred}} - 1$

$m \leftarrow\min(\{j:i \leq j < i + K_{\mathrm{pred}} - 1,\neg a_j\}\cup\{i + K_{\mathrm{pred}} - 1\}) - i$

$x_{i+m} \leftarrow\tilde{x}_{i+m}$

$x_{j+m} \leftarrow\hat{x}_{mj}$ for $i < j < i + K_{\mathrm{pred}}$

$i \leftarrow i + m + 1$

end while

return $x_{\leq L}$

Inference

During inference we sample high-quality samples with Quadratic Decoding (cf. § 3.6). Pseudo-code can be found in Algorithm 1. The encoding and decoding call can be parallelized with a custom attention matrix. The case where we accept all proposed tokens is illustrated in Figure 7.

Measurement Protocol

Every throughput figure in this paper is measured on a single NVIDIA RTX 6000 Ada GPU (48 GB). Completions on each prompt dataset are generated twice inside one process and we report the second. The first pass absorbs engine initialization, CUDA graph capture, kernel autotuning and warms up the GPU for accurate measurements under full load. Compared against single-pass runs, throughput is up to 4.0% slower, as the GPU experiences thermal throttling or hits its power limit under continuous load. This more conservative number should be closer to real-world loads. Unless otherwise stated, we report end-to-end tokens per second: total tokens emitted divided by total wall clock, prefill of the kv cache included. For the pretraining comparison, speed is 128 tokens divided by the median seconds per continuation (see Appendix § 14.§ 3).

Autoregressive

We leverage the standard vLLM inference for autoregressive models.

Parallel Language Models

Round geometry. One round decodes $W = K_{\mathrm{pred}} + K_{\mathrm{pred}}^2$ query positions: $K_{\mathrm{pred}}$ verify rows and $K_{\mathrm{pred}}$ proposal blocks of $K_{\mathrm{pred}}$ rows each. The first verify row is the token committed by the previous round, which has no cached key or value yet; the remaining $K_{\mathrm{pred}} - 1$ carry the pending hypothesis. This maps exactly onto vLLM’s fixed-$K_{\mathrm{pred}}$ speculative path with num_speculative_tokens = W - 1, so one engine step is one round with no padding or bucketing.

Precision. We decode in float16 rather than bfloat16, which is 3.0% faster at comparable quality.

Round width. Fitted per acceptance rule based on best tokens per second a sweep over a subset of prompts from the training set, $K_{\mathrm{pred}} = 6$ for all pretrained models, $K_{\mathrm{pred}} = 10$ for post-trained models up to 1.3B parameters under the relaxed acceptance rule, $K_{\mathrm{pred}} = 7$ otherwise. See Table 13 for an example.

Acceptance rules. Each setting has rules fitted to a quality budget expressed as MAUVE loss against that setting’s own exact-match reference. We choose to either accept tokens in the $\rho$-nucleus of the encoder likelihood or tokens that have an encoder likelihood higher than $\epsilon$, with $\rho$ and $\epsilon$ fitted such that we lose no more than 1% in MAUVE score. This yields $\epsilon= 2 \cdot10^{-3}$ for pretrained models and $\rho= 0.94$ for post-trained models.

Discrete Flow Matching

We time DFM 1.3B and FS-DFM (Monsefi et al. (2026)) with the following changes to the authors’ code: We cast the block weights to bfloat16, compile the model with torch.compile, and cache the sinusoidal timestep frequencies, so they do not have to be recomputed every forward call.

Masked Diffusion

We include three models, all decoding a fixed-length canvas by iterative denoising, where $r_t$ is the noise level at timestep $t$, e.g., related to the fraction of masked tokens.

LLaDA-8B (Nie et al. (2025)) Temperature 0, no top-$k$, block length 32. Within a block, positions are unmasked in order of the model’s confidence in the token it proposed, with the per-step count given by a linear schedule.

Dream-7B (Ye et al. (2025)) Negative-entropy remasking, temperature 0.2, nucleus 0.95, no top-$k$. The schedule is continuous-time: the fraction transferred at step $t$ is $1-r_{t+1}/r_t$, where $r_t$ is linearly spaced between 1 and $\epsilon= 10^{-3}$, and the number of positions moved is that fraction of those still masked. Confidence is computed from the temperature-scaled, nucleus-truncated distribution.

MDLM-350M (Sahoo et al. (2024)) Our own 350M training of the MDLM objective, not a released checkpoint. We serve the exponential moving average of the weights. No temperature and no truncation. The sampler is ancestral: mask and token are drawn together from one float64 categorical per masked position, the mask holding $r_{t+1}/r_t$, where $r_t$ is linearly spaced between 1 and 0. There is no block structure, the whole canvas is denoised at once, and the last step unmasks whatever remains.

KV cache. We use a key-value cache to speed up inference, which is only exact if a token’s representation depends on no token after it. As the diffusion models above attend bidirectionally across the whole sequence, including the prompt, reusing the prompt’s keys and values is an approximation.

Sample Generations

We provide example sentence completions from our models and a subset of baselines on two random documents from FineWeb-Edu [43].

Prompt

Unlocking the Wonders of STEM Education

Welcome to our senior secondary school, where we are redefining education through a Science, Technology, Engineering, and Mathematics (STEM) curriculum. We believe that the future belongs to those who can think scientifically, solve problems creatively, and embrace innovation wholeheartedly. Our aim is to ignite

Human Reference

the sparks of science in every student, equipping them with the essential skills and knowledge to thrive in the 21st century.

STEM education focuses on promoting critical thinking, collaboration, and hands-on learning experiences. Our curriculum is designed to engage students in real-world challenges, fostering their curiosity and nurturing their passion for scientific exploration. By integrating these four disciplines, we provide a holistic education that empowers students to become lifelong learners and future leaders in STEM fields.

Embracing a Dynamic Learning Environment

At our senior secondary school, we believe that learning should be dynamic and interactive. We go beyond traditional classroom settings and encourage students to

PLM 3B Exact

your passion for STEM, fostering a deep understanding and appreciation of the subject matter. It has been said that this is the best STEM school. If you want to grow, you can book admissions.

World-Class Infrastructure

Behind every successful STEM institution lies well-equipped laboratories, cutting-edge facilities, and modern classrooms. At Saadiyya School, we take these aspects very seriously. Our state-of-the-art laboratories are equipped with the latest technology, enabling students to engage in hands-on learning experiences. Our classrooms are designed to foster collaboration and critical thinking, encouraging students to explore new ideas and push the boundaries of

Generation Speed: 285 tok/s

PLM 3B Relaxed

your passion, nurture your skills, and prepare you for a world that’s rapidly evolving.

Why Choose STEM at Senior Secondary School?

In an era where technological advancements are shaping our lives, STEM education lays the foundation for success. At our institution, we don’t just provide subject knowledge; we offer a holistic education that goes beyond classroom walls. Through interactive workshops, hands-on projects, and real-world applications, you’ll gain practical skills and a deep understanding of the world around you.

Our Approach to STEM Education

Our STEM curriculum goes beyond traditional subjects, equipping you with skills like critical thinking, problem-solving

Generation Speed: 430 tok/s

PLM 1.3B Relaxed

a passion for STEM among our students, preparing them with a skillset that is increasingly sought in the modern world.

Question: Which of the following fields is NOT a part of STEM education at the senior secondary school? A) Science B) History C) Technology D) Mathematics Answer: B) History

Question: What is the primary goal of the senior secondary school’s STEM curriculum? Answer: To spark a passion for STEM among students and prepare them for a future-oriented, innovative-driven world.

Question: Is the senior secondary school focused on educating students for future careers? Answer: Yes.

Question

Generation Speed: 856 tok/s

PLM 350M Relaxed

a love of learning and equips young students with the necessary skills to thrive in the ever-evolving world.

At a glance

Question: What are the three core subjects that the curriculum focuses on in senior secondary? Answer: Science, Technology, Engineering, and Mathematics (STEM)

Question: What is the primary focus of the curriculum for senior secondary school? Answer: To equip young students with the necessary skills to thrive in the ever-evolving world and foster a love of learning.

Question: What is the approach of primary education to provide the best learning? Answer: The approach emphasizes active and creative participation

Generation Speed: 2001 tok/s

Qwen3 4B

a passion for discovery that will empower our students to tackle the world’s biggest challenges and build a better tomorrow.

Fostering Critical Thinking and Problem-Solving Skills

At our school, we understand that STEM education goes beyond rote learning. We prioritize developing critical thinking and problem-solving skills, equipping our students with the ability to analyze complex problems and devise innovative solutions. Through hands-on experiments, project-based learning, and real-world scenarios, our students learn to think outside the box, adapt, and succeed in an ever-changing world.

Hands-on Learning: Making Science Come Alive

Our STEM-focused curriculum transforms learning into an engaging experience. Our

Generation Speed: 102 tok/s

Llama 3.2 1B

the curiosity and passion for learning in our students, so that they may thrive in a world of opportunities and challenges. In this blog post, we will delved into the wonders of our STEM education programs with the aim of providing actionable insights and inspiration for parents and students alike. We have handpicked some of the best STEM education programs that we know best in this post. Read on to discover how your child’s academic journey can be enriched by these hands-on, state-of-the-art learning experiences.

The Value of STEAM Education Programs

The foundation of STEAM education programs is centered around fostering students’ skills in engineering, computation, social

Generation Speed: 316 tok/s

SmolLM2 360M

curiosity and spark curiosity among our students.

The course structure follows the American School of Mankato’s traditional 14 weeks model, allowing for flexibility and adaptation based on the student’s interests and needs. Through this integrated approach, students engage in hands-on, project-based learning experiences that foster critical thinking, communication, collaboration, and problem-solving skills.

Our Philosophy of Teaching and Learning

We believe that learning should be enjoyable, both for the student and for the teacher. By emphasizing critical thinking, creativity, and passion, we create engaging lessons that spark joy within our young learners. Moreover, our commitment to

Generation Speed: 589 tok/s

Prompt

What You Should Know About Cortisone Shots

What are cortisone shots?

Cortisone shots are another term for steroid injections, as cortisone is a type of corticosteroid. There are many different types of steroid injections that can be used for many different reasons. In general, cort Preprint. Under review.

Human Reference

isone shots are used for relieving painful conditions that are related to something going wrong in the musculoskeletal system, whether in a joint, a tendon, muscle or the spine.

What conditions are cortisone shots used to treat?

Cortisone shots, also called steroid shots, are used to treat conditions in which there is pain related to inflammation, whether it’s in a large joint such as the knee or shoulder or smaller joints as in the hand or ankle.

They can also be used for painful and inflammatory conditions from a herniated disc in the spine, whether it’s in the lumb

PLM 3B Exact

icosteroid injections target the source of an underlying inflammatory condition. Steroid injections can target bone, joints, the skin, or the nerves as well as different types of tissues.

The cortisone injections that are frequently used for inflammatory conditions can include injections that are for the knees, elbows, spine, shoulders and wrists, and the cortisone shot may be placed in different parts of the anatomy depending on where the pain occurs.

Where should I avoid getting them?

Cortisone shots need to be reserved for cases where there’s definitive evidence that a steroid injection is the most likely solution. First, the location of

Generation Speed: 285 tok/s

PLM 3B Relaxed

icosteroid injections are used to reduce pain and inflammation. These injections are the preferred form of corticosteroid therapy over oral corticosteroids since they are directly targeted to the area of the body that is affected with pain. Oral corticosteroids are commonly prescribed to treat inflammatory conditions of the body (arthritis, colitis, IBS, etc.) but are primarily intended to act systemically to provide inflammation.

The downside is systemic side effects can have many negative side effects, which is why corticosteroid injections are preferred to treat pain or arthritis of a specific joint area.

Cortisone injections are very

Generation Speed: 430 tok/s

PLM 1.3B Relaxed

isone can be very helpful. However, before you treat with it, it’s important to be an informed participant and choose the right route.

Cortisone injections are an injection option for treatment for a number of conditions that are characterized by inflammation, which is the immune system’s response to injury. The injections of cortisone are helpful for chronic pain that has been difficult to respond to with other treatments.

Cortisone shots are usually applied for a variety of conditions, including:

Inflammatory tendonitis, such as rotator cuff

Question: Which of the following is NOT a use for cortis

Generation Speed: 856 tok/s

PLM 350M Relaxed

isone injections are used for inflammation and other conditions related to the human body. Cortisone shots work by breaking the immune system to the area where your body is hurting. Cortisone shots can be used in surgery, chronic pain, and chronic cancer. Cortisone shots can be used to treat certain health and autoimmune problems, cancer, and arthritis.

How are the shots used?

The cortisone shot is applied in a vein with minimal to no discomfort and will be done by a doctor. It involves placing a steroid patch in the part of the body where a body part is involved. The shot is injected into a

Generation Speed: 2001 tok/s Qwen3 4B

isone or steroid injections are anti-inflammatory injections that can be used for a variety of conditions. Cortisone can also be taken internally in the form of oral medications or injections are usually placed directly at the site of pain or discomfort in order to reduce inflammation and improve mobility. Cortisone shots are often given to patients suffering from arthritis and other joint pain.

Steroid injections are used to treat pain or discomfort caused by arthritis in the hands, neck, knees, hips, back or shoulders, muscle pain and to accelerate the healing process of an injury, including tendonitis, bursitis and disc problems. They are also used for conditions

Generation Speed: 102 tok/s

Llama 3.2 1B

isone is used when the patient has inflammation that is not responding to other medications. It is typically found in the upper leg or arm, or in the lower back with epidural injections. Usually, intravenous (IV) injections are given first. This method may provide an injection to the lowest place on the body where inflammation may have occurred. On average, steroid injections take about 10 to 15 minutes and may completely relieve the pain in the area.

This post is not intended for a diagnosis. Please use caution and seek professional medical help if the pain persists or is causing symptoms to affect your life. If you are considering steroid injections

Generation Speed: 316 tok/s

SmolLM2 360M

isone is used painlessly to reduce swelling and inflammation, but only in a very low dose (whereas a cortisone shot is used in higher doses). You should see your doctor if you know you’re sensitive to cortisone (an allergy which causes an itchy or burning rash or fever), as cortisone shots have the potential to cause rebound conditions such as muscle swelling. However, doctors generally recommend avoiding cortisone shots for up to 5 years as repeated injections of cortisone in combination with regular injection of an anti-inflammatory medication can result in high blood pressure.

What are the benefits of cort

Generation Speed: 589 tok/s