betaFeedback

WorksComputer Science

On numerical issues of divergences between nearly equal distributions

Abstract

Barron et al. (2026) recently identified numerical issues in the softmax and soft cross-entropy loss. We show that similar issues are present in standard divergence operations between categorical distributions. This includes the forward and reverse Kullback-Leibler (KL) divergence and generalized Jensen-Shannon divergence (JSD), as well as Monte Carlo KL estimates and entropy. When the distributions are nearly equal, rounding and cancellation issues affect the forward and gradient computations. These issues can affect settings using divergences in optimization objectives, such as knowledge distillation, reinforcement learning, and proximal methods. We propose numerically stable reformulations and provide example implementations for each setting, validated by property-based tests.

Introduction

Standard implementations of the softmax and soft cross-entropy loss subtract the maximum logit to avoid overflow, but their gradients can still suffer from rounding and cancellation. Barron et al. [3] analyze and fix two such issues. First, softmax collapse, previously identified by Prieto et al. [36], occurs for confident hard-target predictions. The target probability rounds to one, making its cross-entropy gradient zero even though the other categories retain positive probabilities and nonzero gradients. Retaining the small probability tail avoids this failure. Second, softmax cancellation affects soft cross-entropy when the prediction and target distributions are nearly equal. The gradient depends on small differences between their probabilities, which separate normalization and rounding can erase. Computed target probabilities need not sum exactly to one, so differentiation can even produce a nonzero gradient for identical logits. Reconstructing probability differences from logit differences, together with a zero-sum correction, addresses this second failure.

These results motivate further studying numerical accuracy in divergences. The forward KL inherits the soft cross-entropy gradient issue because their gradients coincide when the reference distribution is fixed. Divergences themselves also vanish as the distributions approach equality, so their forward computations can suffer cancellation as well. Applications potentially affected by these errors include knowledge distillation, reinforcement learning, and proximal optimization, where these quantities serve as fitting terms and regularizers. § 2 expands on these.

Contributions of this paper. For nearly equal distributions, we identify five numerical issues in standard implementations of the forward and gradient computations of 1) the forward and reverse KL divergences, including Monte Carlo KL estimators and entropy penalties (§ 4), and 2) the generalized Jensen–Shannon divergence (§ 5). To address these issues, we cover eight reformulations and provide executable JAX examples validated by property-based tests. Table 1 summarizes the settings.

SettingForward issueGradient issue
Forward KL (§ 4.1)KL-sum cancellationSoftmax subtraction
Reverse KL (§ 4.2)KL-sum cancellationLog-softmax subtraction
Entropy via reverse KL to uniform (§ 4.2.1)No issue presentLog-probability centering
Monte Carlo K1: log-ratio (§ 4.3.3)Log-softmax subtractionNo issue present
Monte Carlo K2: squared log-ratio (§ 4.3.4)Log-softmax subtractionInherits forward issue
Monte Carlo K3: control variate (§ 4.3.5)Remainder cancellationRounded exponential
Generalized JSD (§ 5)KL-sum cancellationMixture log-ratio rounding

Table 1. Numerical issues for nearly equal (categorical) distributions.

Does it matter in practice? The examples isolate the arithmetic rather than measure large-scale training outcomes. § 3 further discusses their practical significance.

Notation and setup. Because this is an ML paper, we consider a model fθf_\theta with parameters θ\theta that maps an input xx to logits z=fθ(x)∈RKz = f_\theta(x) \in\mathbb{R}^{K} over KK categories (e.g., vocabulary entries), although our main analysis here only considers the logits regardless of how they were obtained. The logits define a categorical distribution q=softmax⁡(z)q = \operatorname{softmax}(z), and p=softmax⁡(zref)p = \operatorname{softmax}(z_{\mathrm{ref}}) is a fixed reference distribution with logits zref∈RKz_{\mathrm{ref}} \in\mathbb{R}^{K}, potentially also generated by another model. We analyze the forward computation of each divergence and its gradient with respect to θ\theta. Gradients with respect to θ\theta follow by the chain rule. Section A gives further details on the numerical conventions, experimental protocol, and helper implementations.

Applications of optimizing divergences

Divergences appear in many optimization objectives, both as fitting terms and as regularizers. The consequences of numerical error depend on the weight of the divergence term and how its gradient propagates through the model and optimizer.

Knowledge distillation and on-policy distillation. Knowledge distillation trains a student to match a teacher’s predictions ([7, 18, 21]), with on-policy variants using student-generated contexts ([15] (Agarwal et al. 2024)). A student with parameters θ\theta predicts qθ(⋅∣x)q_\theta(\cdot\mid x) at context xx. The context distribution μ\mu comes from data or student rollouts and is held fixed during each update. The objective compares the student’s predictions with a fixed teacher p(⋅∣x)p(\cdot\mid x):

min⁡θ Ex∼μ[D(p(⋅∣x),qθ(⋅∣x))].(1)\min_{\theta}\ \mathbb{E}_{x\sim\mu}\left[D\left(p(\cdot\mid x), q_\theta(\cdot\mid x)\right)\right]. \tag*{(1)}

where DD is a divergence commonly chosen as the forward KL, reverse KL, or generalized JSD, as in Agarwal et al. (2024, Section 4.1). Here, the numerical issues discussed in this paper can arise near optimality as the student’s predictions approach the teacher’s.

Reinforcement learning with reverse-KL or entropy regularization. A policy with parameters θ\theta generates outputs y∼πθy \sim\pi_\theta. With penalty weight λ>0\lambda> 0, the objective balances reward r(y)r(y) against a reverse-KL penalty that encourages the policy to remain close to a fixed reference policy πref\pi_{\mathrm{ref}}:

max⁡θ Ey∼πθ[r(y)]−λKL⁡(πθ∥πref).(2)\max_{\theta}\ \mathbb{E}_{y\sim\pi_\theta}[r(y)]-\lambda\operatorname{KL}(\pi_\theta\mathbin\Vert\pi_{\mathrm{ref}}). \tag*{(2)}

Reverse-KL penalties regularize control and policy optimization ([46, 14, 34]). The numerical issues discussed in this paper can arise when the policy is close to its reference. Entropy bonuses encourage exploration ([49, 31]). For a categorical policy, maximizing entropy is equivalent to penalizing the reverse KL to the uniform distribution. Similarly, entropy gradients can become inaccurate near a uniform policy. These errors may have limited practical impact when the task favors nonuniform behavior or behavior far from the reference policy. In these cases, the gradient may be dominated by that of the expected reward.

Proximal optimization with reverse KL. Proximal methods can minimize a base objective FF with a reverse-KL penalty that discourages large changes from the current distribution qθkq_{\theta_k}, which remains fixed during each update (Teboulle [44]):

θk+1∈arg⁡min⁡θ{F(θ)+ηk−1KL⁡(qθ∥qθk)},ηk>0.(3)\theta_{k+1}\in\arg\min_{\theta}\left\{F(\theta)+\eta_k^{-1}\operatorname{KL}(q_\theta\mathbin\Vert q_{\theta_k})\right\},\qquad\eta_k>0. \tag*{(3)}

Reverse-KL regularization of successive iterates also motivates natural-gradient methods ([2, 30]). Related policy-search methods constrain the KL between successive policies ([35, 40]). The numerical issues discussed in this paper can arise when successive distributions are nearly equal, although the gradient may be dominated by that of FF.

Discussion on the scope and practical significance

Before developing the reformulations, we discuss where they apply and how to interpret their numerical accuracy in optimization.

⚠ When to use the reformulations. Some reformulations also require nearly equal distributions and may be numerically unstable outside this regime. Full production implementations are outside the scope of this paper, but we indicate where guards and fallback evaluations are needed. For example, the soft cross-entropy implementation of Barron et al. [3] uses the correction only for log-ratios with magnitude below one and switches to direct probability subtraction otherwise. § 4.1.1 illustrates this limitation for forward KL: although the reformulation is algebraically valid, its intermediate computations can overflow away from equality. The same limitation applies to other reformulations that share this log-ratio reconstruction.

Numerical issues are exacerbated at lower precision. Our examples use FP32. Barron et al. [3] compare BF16, FP32, and FP64 in soft-target distillation and observe larger residual gradient errors and parameter drift in BF16. Lower precision can make rounding dominate at larger separations between distributions, when the divergence and its gradient are larger. Other lower-precision formats, such as FP8 and FP4, could make these issues even more prevalent when used to compute divergences and their gradients. At higher precision, large relative errors may still have very small absolute magnitude.

Adam’s stabilizing constant. Barron et al. [3] study how the stabilizing constant ϵ\epsilon in Adam (Kingma & Ba, [23]) limits the amplification of small gradients. Our examples examine numerical accuracy without measuring its effects on training. In their soft-target distillation experiment, Adam amplifies rounding residuals and moves a student away from an initially identical teacher. The divergences considered here raise a similar concern: their true gradients vanish at equality, but numerical errors can produce nonzero updates. The practical effect depends on the parameter-gradient statistics used by Adam, after backpropagation through the model and combination with other objective terms.

Remaining rounding errors. Accuracy also depends on normalization, reductions, and input storage. A reformulation cannot recover a displacement already rounded away in its inputs. Compensated summation can retain rounding residuals in the sampled log-ratio (reformulation 5), improving accuracy when its two differences nearly cancel (Ogita et al., [33]). This costs additional arithmetic and cannot correct errors already present in the supplied log-normalizers.

Fixing Kullback–Leibler divergences

From the setup, p=softmax⁡(zref)p = \operatorname{softmax}(z_{\mathrm{ref}}) is the fixed categorical reference and q=softmax⁡(z)q = \operatorname{softmax}(z) is the distribution being optimized. The forward and reverse KL are defined as:

Definition 1 (Forward KL).

KL⁡(p∥q)=Ea∼p[log⁡pa−log⁡qa]=∑ipi(log⁡pi−log⁡qi).(4)\operatorname{KL}(p \parallel q) = \mathbb{E}_{a \sim p}[\log p_a - \log q_a] = \sum_i p_i(\log p_i - \log q_i). \tag*{(4)}

Definition 2 (Reverse KL).

KL⁡(q∥p)=Ea∼q[log⁡qa−log⁡pa]=∑iqi(log⁡qi−log⁡pi).(5)\operatorname{KL}(q \parallel p) = \mathbb{E}_{a \sim q}[\log q_a - \log p_a] = \sum_i q_i(\log q_i - \log p_i). \tag*{(5)}

The only difference is the order of the arguments. The names forward and reverse refer to that order when optimizing qq with pp fixed. Both are nonnegative and vanish at q=pq = p. Optimization can favor different solutions when the model family is not expressive enough, e.g., the forward KL is typically mode-covering and the reverse KL is often mode-seeking.

Basic derivatives. Holding pp fixed, differentiating with respect to zz gives a probability difference for the forward KL and a centered log-ratio weighted by qq for the reverse KL:

Forward KL gradient.

∇zKL⁡(p∥q)=q−p.(6)\nabla_z \operatorname{KL}(p \parallel q) = q - p. \tag*{(6)}

Reverse KL gradient.

∇zKL⁡(q∥p)=q(s−qTs),where s=log⁡q−log⁡p.(7)\nabla_z \operatorname{KL}(q \parallel p) = q(s - q^\mathsf{T}s), \qquad\text{where } s = \log q - \log p. \tag*{(7)}

This section. The two KL directions require different numerical evaluations. We begin with forward KL and its connection to soft cross-entropy (§ 4.1), then turn to reverse KL (§ 4.2). Monte Carlo estimates replace the KL sums with samples, with expected gradients that depend on the estimator and sampling direction (§ 4.3). Entropy regularization is another instance of the reverse KL, with a uniform reference (§ 4.2.1).

Fixing the forward KL

The forward KL connects directly to the soft cross-entropy analysis of Barron et al. [3]. With pp fixed, the (soft) cross-entropy CE⁡(p,q)=−∑ipilog⁡qi\operatorname{CE}(p,q) = -\sum_i p_i \log q_i equals KL⁡(p∥q)+H(p)\operatorname{KL}(p \parallel q) + H(p), where H(p)=−∑ipilog⁡piH(p) = -\sum_i p_i \log p_i is the reference entropy. Both require a shared gradient q−pq - p (see equation (6)). The direct computation first subtracts log-probabilities, then sums their weighted differences. Errors in either step can overwhelm the small divergence. We examine these two issues before turning to the shared gradient.

Numerical issue 1 (Log-softmax subtraction, e.g., as in equations (4) and (5)). Evaluating the log-softmax separately for the logits zz and zrefz_{\mathrm{ref}} can round distinct log-probabilities to the same value. Subtracting the rounded log-softmax outputs can then erase the small entries of the log-ratio s=log⁡q−log⁡ps = \log q - \log p. Subtracting the maximum logit stabilizes each normalization but does not prevent this rounding error.

Barron et al. [3] compute ss from centered logit differences to avoid this subtraction. For δ=z−zref\delta= z-z_{\mathrm{ref}} and d=δ−∑ipiδid = \delta- \sum_i p_i\delta_i, their expression is

s=d−log⁡1p(∑ipiexpm1⁡(di)).(8)s = d - \log1p\left(\sum_i p_i\operatorname{expm1}(d_i)\right). \tag*{(8)}

This identity follows from q/p=eδ/∑ipieδiq/p = e^\delta/\sum_i p_i e^{\delta_i}. Subtracting a common scalar from δ\delta leaves this ratio unchanged. Choosing ∑ipiδi\sum_i p_i\delta_i gives q/p=ed/∑ipiediq/p = e^d/\sum_i p_i e^{d_i}, using the shift invariance of softmax (Blanchard et al., [5]). The expm1 and log1p routines retain the small increments in this identity without first rounding ede^d or ∑ipiedi\sum_i p_i e^{d_i} near one.

Even with an accurate ss, the signed sum KL⁡(p∥q)=−∑ipisi\operatorname{KL}(p\mathbin\Vert q) = -\sum_i p_i s_i can still suffer cancellation.

Numerical issue 2 (KL-sum cancellation, e.g., as in equation (4)). Near equality, each summand satisfies

−pisi=pi(log⁡pi−log⁡qi)≈(pi−qi)+(qi−pi)22pi.(9)-p_i s_i = p_i(\log p_i-\log q_i) \approx(p_i-q_i) + \frac{(q_i-p_i)^2}{2p_i}. \tag*{(9)}

The terms −pisi-p_i s_i have mixed signs and can be much larger than their sum: the pi−qip_i-q_i contributions cancel because ∑ipi=∑iqi=1\sum_i p_i=\sum_i q_i=1, leaving the smaller squared terms. Small errors in the log-ratios, the products, or the summation need not cancel and can overwhelm the remaining divergence.

The forward computation therefore needs both an accurate log-ratio (issue 1) and a sum that avoids cancellation between categories (issue 2). Adding qi−piq_i-p_i to each summand removes the cancelling contributions without changing the total, since ∑i(qi−pi)=0\sum_i(q_i-p_i)=0. Each corrected summand is pi(esi−1−si)=piE2(si)p_i(e^{s_i}-1-s_i)=p_iE_2(s_i), where E2(x)=ex−1−xE_2(x)=e^x-1-x.

Numerical reformulation 1 (Fixing issues 1 and 2 for the forward KL computation in equation (4)). Use ss from equation (8) to avoid subtracting rounded log-probabilities. Replace the signed KL sum by the nonnegative-remainder identity:

KL⁡(p∥q)=∑ipiE2(si).(10)\operatorname{KL}(p\mathbin\Vert q)=\sum_i p_iE_2(s_i). \tag*{(10)}

Evaluate E2E_2 near zero using its series expansion to avoid cancellation.

The same correction is used as a control variate in Monte Carlo KL estimation ([39]). Since ex≥1+xe^x\ge1+x, every remainder E2(si)E_2(s_i) is nonnegative, avoiding cancellation between categories. Computing E2(x)E_2(x) as expm1⁡(x)−x\operatorname{expm1}(x)-x still subtracts nearly equal quantities. Near zero, evaluate it directly using a Taylor expansion, as in the expm1x routine of Mächler [28]:

E2(x)=x2(12+x6+x224+⋯ ),E2′(x)=expm1⁡(x).(11)E_2(x)=x^2\left(\frac{1}{2}+\frac{x}{6}+\frac{x^2}{24}+\cdots\right), \qquad E_2'(x)=\operatorname{expm1}(x). \tag*{(11)}

Section A gives the e2 implementation and its connection to established remainder evaluations.

Gradient computation. A similar rounding issue affects the gradient in equation (6), which requires the probability difference q−pq-p. Separate softmax evaluations can round nearby probabilities to the same value.

Numerical issue 3 (Softmax subtraction, e.g., as in equation (6)). The difference q−p=softmax⁡(z)−softmax⁡(zref)q-p=\operatorname{softmax}(z)-\operatorname{softmax}(z_{\mathrm{ref}}) subtracts softmax outputs, whereas issue 1 subtracts log-softmax outputs. Either subtraction can erase a small difference when the outputs are rounded separately, even if each output is individually accurate. Differentiating the direct KL sum through log-softmax produces the probability difference q−pq-p. Normalization roundoff can also leave a nonzero residual at identical logits.

The retained log-ratio recovers the small probability difference that softmax subtraction can erase (issue 3). The final correction uses the gradient’s zero-sum identity to reduce accumulated roundoff.

Numerical reformulation 2 (Fixing issue 3 for the forward KL gradient in equation (6)). The soft-target correction of Barron et al. [3] evaluates q−pq-p from the reconstructed ss in equation (8) and corrects its numerical sum residual:

u=pexpm1⁡(s),∇zKL⁡(p∥q)=u−q(∑iui).(12)u=p\operatorname{expm1}(s), \qquad\nabla_z\operatorname{KL}(p\mathbin\Vert q)=u-q\left(\sum_i u_i\right). \tag*{(12)}

In exact arithmetic, u=q−pu=q-p and ∑iui=0\sum_i u_i=0. The final term reduces the rounding residual.

Listing 1 combines the remainder with this explicit gradient. Figure 1 tests both as the logits approach equality. The helper categorical implements equation (8) and returns pp, qq, log⁡p\log p, log⁡q\log q, ss (section A). These examples focus on numerical accuracy. An optimized implementation can avoid computing quantities that the loss does not need. Ordinary autodiff of the complete remainder-based graph is also accurate in the tested sweep. Retaining the ordinary cross-entropy sum instead needs the backward correction despite an accurate forward cross-entropy.

Loss error and gradient error for forward KL under perturbation

Figure 1. Rounding and cancellation distort forward KL and its gradient. This example uses fixed reference logits zref=(−1,0,1)z_{\mathrm{ref}} = (-1, 0, 1) and varying logits z=(−1,h,1)z = (-1, h, 1). The reformulation retains the divergence and gradient as qq approaches pp.

Pitfalls of the reformulations

The full-distribution KL and JSD reformulations share the log-ratio reconstruction in equation (8). Its exponentials can overflow away from equality, and its logit subtraction can erase small differences even near equality. We illustrate both limitations with forward KL. The closeness guard addresses the first but does not resolve the second.

⚠ Overflow away from equality. The log-ratio identity in equation (8) holds for finite logits, but its exponential evaluation requires a guard restricting its use to nearly equal distributions. Otherwise, intermediate quantities can overflow even when the desired result is finite. With zref=(−h,0)z_{\mathrm{ref}} = (-h, 0) and z=(h,0)z = (h, 0), the true log-ratio is s=(h,−h)s = (h, -h). At h=50h = 50, the reconstruction forms an exponential near e100e^{100} and overflows in FP32, although the log-ratio and KL are finite (Figure 2). A small KL does not bound every probability ratio. The implementations use assert_close (section A) to restrict this evaluation to nearly equal distributions. Full-range use requires a fallback, such as the hybrid evaluation of Barron et al. [3].

Plot of KL divergence versus logit separation, comparing Basic, Fixed, and Reference

Figure 2. ⚠ Using the fixed forward-KL formula can cause overflow when the distributions are not nearly equal. This example uses reference logits zref=(−h,0)z_{\mathrm{ref}} = (-h, 0) and varying logits z=(h,0)z = (h, 0). As hh increases, the reformulation overflows while the basic log-probability sum remains finite.

⚠ Rounding can remain near equality. Take zref=(0,h)z_{\mathrm{ref}} = (0, h) and z=(1,1)z = (1, 1) with h=2−25h = 2^{-25}. The inputs retain hh exactly, but FP32 subtraction gives δ=z−zref=(1,1)\delta= z - z_{\mathrm{ref}} = (1, 1). Centering this constant vector gives zero, so the fixed forward KL and its gradient both vanish even though p≠qp \ne q (Figure 3). The reverse-KL and JSD evaluations share this failure because they use the same logit differences. It occurs before the stable elementary functions are applied and passes the assert_close check. The reconstruction avoids subtracting separately rounded log-probabilities, but its own logit subtraction can still erase the signal. Max and mean centering, anchoring to one category, and aligning the logit offsets caused regressions or left rounding failures unresolved in our tests. Finding a simple FP32 reformulation without these trade-offs remains an open problem.

Plots of loss error and gradient norm versus perturbation

Figure 3. ⚠ Rounding can remain in the fixed forward KL. With zref=(0,h)z_{\mathrm{ref}} = (0, h) and z=(1,1)z = (1, 1), FP32 subtraction rounds both logit differences to one for sufficiently small hh. The fixed divergence and gradient then vanish although p≠qp \ne q.

Fixing the reverse KL

The reverse KL averages under qq, which depends on the logits zz. Its forward computation inherits log-softmax subtraction (issue 1) and KL-sum cancellation (issue 2). With pp and qq exchanged in equation (9), the signed differences qi−piq_i - p_i again sum to zero. The remainder identity in equation (10) applies with −s-s in place of ss.

Forward KL example code

Figure 1 (PDF p. 5). Forward KL example code

Basic
1 def forward_kl_basic(z, z_ref):
2     p, _, lp, lq, _ = categorical(z, z_ref)
3     return jnp.sum(p * (lp - lq))

Fixed
1 @custom_gradient
2 def forward_kl_fixed(z, z_ref):
3     assert_close(z, z_ref)
4     p, q, _, _, s = categorical(z, z_ref)
5     u = p * jnp.expm1(s)
6     value = jnp.sum(p * e2(s))
7     gradient = u - q * jnp.sum(u)
8     return value, gradient

Numerical reformulation 3 (Fixing issues 1 and 2 for the reverse KL computation in equation (5)). Adding qi(e−si−1)=pi−qiq_i(e^{-s_i}-1)=p_i-q_i to each summand qisiq_i s_i leaves the total unchanged because ∑i(pi−qi)=0\sum_i(p_i-q_i)=0. Using ss from equation (8), this replaces the signed sum in equation (5) with nonnegative remainders:

KL⁡(q∥p)=∑iqiE2(−si).(13)\operatorname{KL}(q \mathbin{\|} p)=\sum_i q_i E_2(-s_i). \tag*{(13)}

Evaluate E2E_2 with equation (11) near zero.

Gradient computation. The reverse-KL gradient in equation (7) centers the log-ratio ss, whose small entries can be erased by log-softmax subtraction (issue 1). The log-normalizer offset is common to every category and cancels under centering. Removing it before evaluation avoids rounding away the logit differences that remain. Differentiating the exponential-family KL identity using the log-partition derivatives of Wainwright & Jordan [47] gives the centered gradient in terms of the logit differences δ=z−zref\delta= z-z_{\mathrm{ref}}.

Numerical reformulation 4 (Fixing issue 1 for the reverse KL gradient in equation (7)). Center the logit differences instead of the separately computed log-ratio:

u=q(δ−∑iqiδi),∇zKL⁡(q∥p)=u−q(∑iui).(14)u=q\left(\delta-\sum_i q_i\delta_i\right), \qquad\nabla_z \operatorname{KL}(q \mathbin{\|} p)=u-q\left(\sum_i u_i\right). \tag*{(14)}

In the implementation, first subtract δ1\delta_1 from every entry of δ\delta, as in listing 2. This removes a common logit offset before centering. The final term applies the zero-sum correction from equation (12).

The following proposition justifies replacing ss by δ\delta in equation (14).

Proposition 1 (Centering log-ratios and logit differences). For finite logits z,zref∈RKz, z_{\mathrm{ref}} \in\mathbb{R}^{K}, let q=softmax⁡(z)q = \operatorname{softmax}(z) and p=softmax⁡(zref)p = \operatorname{softmax}(z_{\mathrm{ref}}). Define the log-ratio s=log⁡q−log⁡ps = \log q-\log p and the logit differences δ=z−zref\delta= z-z_{\mathrm{ref}}. Then

s−∑iqisi=δ−∑iqiδi.(15)s-\sum_i q_i s_i=\delta-\sum_i q_i\delta_i. \tag*{(15)}

Listing 2: Reverse KL example code

Basic
1 def reverse_kl_basic(z, z_ref):
2     _, q, lp, lq, _ = categorical(z, z_ref)
3     return jnp.sum(q * (lq - lp))

Fixed
1 @custom_gradient
2 def reverse_kl_fixed(z, z_ref):
3     assert_close(z, z_ref)
4     _, q, _, _, s = categorical(z, z_ref)
5     delta = z - z_ref
6     delta = delta - delta[0]
7     value = jnp.sum(q * e2(-s))
8     centered = delta - jnp.sum(q * delta)
9     gradient = q * centered
10    gradient = gradient - q * jnp.sum(gradient)
11    return value, gradient

Proof. Expanding si=log⁡qi−log⁡pis_i = \log q_i - \log p_i using the two softmax distributions gives

si=δi+log⁡∑jezref,j−log⁡∑jezjs_i = \delta_i + \log\sum_j e^{z_{\mathrm{ref},j}} - \log\sum_j e^{z_j}

The log-normalizer difference is the same for every category. Since ∑jqj=1\sum_j q_j = 1, subtracting the qq-weighted mean removes this common offset:

si−∑jqjsj=δi−∑jqjδjs_i - \sum_j q_j s_j = \delta_i - \sum_j q_j \delta_j

Subtracting a common scalar from every logit difference also leaves the centered gradient unchanged in exact arithmetic. Listing 2 uses δ1\delta_1 for convenience, avoiding an additional maximum reduction. Subtracting max⁡iδi\max_i \delta_i is also valid. Either choice bounds the shifted magnitudes by the span of δ\delta, and this centering step does not require nonpositive entries.

In exact arithmetic, the centered vector uu satisfies ∑iui=0\sum_i u_i = 0, so the final correction in equation (14) leaves the gradient unchanged. The reverse-KL gradient is also zero whenever q=pq = p, including when the logits differ by a common offset. In floating point, the choice of offset and the accuracy of normalization and centering still affect the result. Computed probabilities need not sum to exactly one, so a common shift can leave a residual proportional to the normalization error. The final correction reduces this residual with an extra sum.

Listing 2 reconstructs ss for the forward remainder and applies the two centering steps for the gradient. Figure 4 compares these evaluations with the direct KL sum. Ordinary autodiff of the complete remainder-based forward graph also passes the tests. The explicit rule gives a shorter derivative and supports other formulas.

Two plots comparing relative error, loss error, and reverse-KL gradient norm for Basic, Fixed, and Reference implementations

Figure 4. Centering logit differences preserves the reverse-KL gradient. This example uses fixed reference logits zref=(−1,0,1)z_{\mathrm{ref}} = (-1, 0, 1) and varying logits z=(−1,h,1)z = (-1, h, 1). The reformulation retains the divergence and gradient as qq approaches pp.

Fixing the entropy gradient

Entropy regularization is a special case of reverse KL. For KK categories and a uniform reference ui=1/Ku_i = 1/K, entropy satisfies the identity [10]:

H(q)=log⁡K−KL(q∥u)=−∑iqilog⁡qi.(16)H(q) = \log K - \mathrm{KL}(q \mathbin{\|} u) = -\sum_i q_i \log q_i . \tag*{(16)}

Differentiating this identity gives ∇zH(q)=−∇zKL(q∥u)\nabla_z H(q) = -\nabla_z \mathrm{KL}(q \mathbin{\|} u), so reformulation 4 applies with a uniform reference and a sign change. Near uniformity, the entropy stays near log⁡K\log K while its gradient vanishes. Centering separately computed log-probabilities can round this small gradient to zero, just as for reverse KL.

# Basic
1 def categorical_entropy_basic(z):
2     log_q = jax.nn.log_softmax(z)
3     q = jnp.exp(log_q)
4     return -jnp.sum(q * log_q)

# Fixed
1 @custom_gradient
2 def categorical_entropy_fixed(z):
3     log_q = jax.nn.log_softmax(z)
4     q = jnp.exp(log_q)
5     value = -jnp.sum(q * log_q)
6     a = z - z[0]
7     centered = a - jnp.sum(q * a)
8     gradient = -q * centered
9     gradient = gradient - q * jnp.sum(gradient)
10     return value, gradient
Entropy gradient accuracy plot

Figure 5. An accurate entropy value can accompany an inaccurate gradient. This example uses two-category logits z=(0,h)z=(0,h), so qq approaches the uniform distribution as hh approaches zero. Reversing the fixed reverse-KL gradient retains the small derivative that ordinary autodiff rounds to zero.

For z=(0,10−8)z=(0,10^{-8}), directly evaluating and differentiating the entropy sum in FP32 returns an accurate entropy near log⁡2\log2 but rounds its nonzero gradient to zero. Listing 3 specializes the reverse-KL gradient rule, including its final zero-sum correction (equation (14)), while keeping the usual entropy sum for the forward computation. Figure 5 illustrates the resulting gradient accuracy near uniformity. The entropy deficit log⁡K−H(q)\log K-H(q), if needed, should be evaluated through the reverse-KL remainder in equation (13).

A practical note. This issue may be less consequential in practice when entropy is used as a regularizer. The numerical failure studied here occurs near the uniform distribution, whereas the main task may favor nonuniform predictions. Accurately retaining this small entropy gradient may therefore matter less than retaining a divergence gradient near a teacher or reference distribution.

Fixing Monte Carlo KL estimators

Summing over every category can be expensive when the distribution is large. Monte Carlo estimation instead evaluates log-probabilities at sampled categories, a longstanding approach to KL estimation [8, 16]. At a fixed context, write sa=log⁡qa−log⁡pas_a=\log q_a-\log p_a for the log-ratio at a sampled category aa. Three commonly used estimators, as notated by Schulman [39], are:

Definition 3 (Monte Carlo KL estimators).

K1: log-ratioK2: squared log-ratioK3: control variate
k1(sa)=sak_1(s_a)=s_ak2(sa)=sa2/2k_2(s_a)=s_a^2/2k3(sa)=sa+e−sa−1k_3(s_a)=s_a+e^{-s_a}-1

For a minibatch of NN independent categories an∼qa_n\sim q, the Monte Carlo estimate to the reverse KL is

1N∑n=1Nk(san),(17)\frac{1}{N}\sum_{n=1}^{N}k(s_{a_n}), \tag*{(17)}

where kk is any of the three estimators. The Monte Carlo estimate to the forward KL instead uses samples an∼pa_n\sim p and evaluates k(−san)k(-s_{a_n}). When this average is used as an optimization objective, autodiff differentiates it while treating the sampled categories as fixed. The choice of estimator and sampling direction determines whether the value and gradient estimates are unbiased, as discussed in § 4.3.1.

⚠ K1 should not be directly differentiated as a reverse-KL penalty: its expected gradient is zero.

Forward and reverse Monte Carlo KL estimators

Expected values. The Monte Carlo estimates to the reverse and forward KL have expected values Ea∼q[k(sa)]\mathbb{E}_{a\sim q}[k(s_a)] and Ea∼p[k(−sa)]\mathbb{E}_{a\sim p}[k(-s_a)], respectively. K1 and K3 are unbiased in both directions, whereas K2 matches the corresponding KL only to second order near equality ([39]). In on-policy distillation, sampling from qq is called student-sampled estimation because qq is the student distribution ([25]).

Expected gradients. Table 2 compares the expected gradients in exact arithmetic. The reverse KL depends on zz through both the log-ratio and the sampling distribution qq. Treating the sampled categories as fixed omits the score-function gradient associated with changes in the sampling probabilities. Thus, differentiating an unbiased forward estimate may not give an unbiased gradient estimate. With forward-KL sampling, pp is fixed, so there is no sampling-distribution derivative to omit. The reverse-KL sampling identities are established by Tang & Munos [43]. For forward-KL sampling, differentiating 12∑apasa2\frac{1}{2}\sum_a p_a s_a^2 gives the K2 entry, which generally differs from either KL gradient. K2 therefore minimizes squared log-probability errors under the reference distribution, with a gradient that matches the forward-KL gradient to first order near equality. K1 and K3 give unbiased forward-KL gradients.

EstimatorReverse KL (Ea∼q[∇zkj(sa)]\mathbb{E}_{a\sim q}[\nabla_z k_j(s_a)])Forward KL (Ea∼p[∇zkj(−sa)]\mathbb{E}_{a\sim p}[\nabla_z k_j(-s_a)])
k1k_100∇zKL(p∣q)\nabla_z \mathrm{KL}(p\mathbin{|}q)
k2k_2∇zKL(q∣p)\nabla_z \mathrm{KL}(q\mathbin{|}p)ps−q∑apasap s-q\sum_a p_a s_a
k3k_3∇zKL(p∣q)\nabla_z \mathrm{KL}(p\mathbin{|}q)∇zKL(p∣q)\nabla_z \mathrm{KL}(p\mathbin{|}q)

Table 2. Expected gradients of Monte Carlo KL estimators.

Applications in reinforcement learning for language models. In reinforcement learning for language models, large vocabularies motivate using these estimators. For example, Kimi k1.5 uses K2 in a surrogate update for a reverse-KL-regularized objective ([22 Section 2.3.2, Eqs. (2)–(3]), while DeepSeekMath uses K3 for reverse-KL regularization in GRPO ([42 Eq. (4]). StableReinforce uses forward KL from the sampler to the learner, with K1 by default or K3 ([45 Sampler–learner KL]).

Shared log-ratio inputs

Both sampling directions use the same scalar log-ratio and remainder operations. We evaluate the numerical issues in both directions and compare the results with each estimator’s expected value and gradient to separate numerical error from estimator bias. The listings show Monte Carlo estimates to the reverse KL. Forward-KL implementations follow by negating the log-ratio before evaluating and differentiating the estimate. K2 uses the same code in either direction.

The log-ratio uses the student logit zaz_a and reference logit zref,az_{\mathrm{ref},a} for the same sampled category aa, together with their log-normalizers log⁡Z=log⁡∑iezi\log Z=\log\sum_i e^{z_i} and log⁡Zref=log⁡∑iezref,i\log Z_{\mathrm{ref}}=\log\sum_i e^{z_{\mathrm{ref},i}} over all KK categories. Both use a numerically stable log-sum-exp evaluation that shifts by the maximum logit to avoid overflow ([5]). These four scalars determine the sampled log-ratio:

sa=(za−zref,a)−(log⁡Z−log⁡Zref).(18)s_a=(z_a-z_{\mathrm{ref},a})-(\log Z-\log Z_{\mathrm{ref}}). \tag*{(18)}

Each model computes its own log-normalizer, so evaluating the penalty requires only the sampled logit and log-normalizer from each model. This also reduces communication when the models run separately, such as in distributed setups. Each supplies two scalars per sampled category rather than its full KK-dimensional logit vector. Both student inputs remain differentiable. Throughout, the input derivatives of an estimator are its partial derivatives with respect to these two student inputs, zaz_a and log⁡Z\log Z, holding the sampled category and the reference inputs fixed.

All three estimators inherit the log-softmax subtraction issue ((issue 1)): separately computing za−log⁡Zz_a-\log Z and zref,a−log⁡Zrefz_{\mathrm{ref},a}-\log Z_{\mathrm{ref}} can round the two log-probabilities to the same value before subtraction. Regrouping the four scalar inputs avoids forming these intermediate log-probabilities.

Numerical reformulation 5 (Fixing issue 1 for the sampled log-ratio in equation (18)). Evaluate equation (18) by first computing za−zref,az_a-z_{\mathrm{ref},a} and log⁡Z−log⁡Zref\log Z-\log Z_{\mathrm{ref}}, then subtracting these differences without forming the log-probabilities. Like the full-distribution reconstruction in equation (8), this forms logit differences before log-probabilities. It retains small logit changes that separate log-probability evaluations can round away.

Listing 4: Sampled log-ratio example code

Basic
1 def sampled_log_ratio_basic(
2     z_a, log_Z, z_ref_a, log_Z_ref
3 ):
4     log_q_a = z_a - log_Z
5     log_p_a = z_ref_a - log_Z_ref
6     return log_q_a - log_p_a
Fixed
1 def sampled_log_ratio_fixed(
2     z_a, log_Z, z_ref_a, log_Z_ref
3 ):
4     z_diff = z_a - z_ref_a
5     log_Z_diff = log_Z - log_Z_ref
6     return z_diff - log_Z_diff

Stable log-sum-exp still rounds the supplied log-normalizers, and the two differences can introduce further rounding. The regrouping therefore does not guarantee small relative error for every nonzero log-ratio.

For za=hz_a = h, zref,a=0z_{\mathrm{ref},a} = 0, and log⁡Z=log⁡Zref=1\log Z = \log Z_{\mathrm{ref}} = 1, the log-ratio is sa=hs_a = h. At h=10−8h = 10^{-8}, the two log-probabilities round to the same value, so direct subtraction returns zero, whereas listing 4 retains hh.

Numerical evaluation. The three estimators use the same example, with zref=(−1,0,1)z_{\mathrm{ref}} = (-1, 0, 1) and z=(−1,h,1)z = (-1, h, 1). All four inputs and both log-normalizer computations use FP32. For reverse-KL sampling, enumerating every category eliminates Monte Carlo sampling error from the expected estimate and the expected gradient:

Ea∼q[k(sa)]=∑aqak(sa),Ea∼q[∇zk(sa)]=∑aqa∇zk(sa).(19)\mathbb{E}_{a\sim q}[k(s_a)] = \sum_a q_a k(s_a), \qquad\mathbb{E}_{a\sim q}[\nabla_z k(s_a)] = \sum_a q_a \nabla_z k(s_a). \tag*{(19)}

For forward-KL sampling, the weights are pap_a and the estimates are k(−sa)k(-s_a). The value panels compare arithmetic at the supplied FP32 normalizers. The gradient panels compare against the exact distributions defined by the stored logits, including errors from the normalizers and their derivatives.

Fixing the sampled log-ratio estimator (K1)

K1 returns the sampled log-ratio directly, inheriting the rounding issue in issue 1. The Monte Carlo estimate to the forward KL negates this log-ratio. The input derivatives are constant in both directions, so the regrouping introduces no change in the gradient computation. Under sampling from qq, K1’s expected logit gradient remains zero. Under sampling from pp, it is the forward-KL gradient in exact arithmetic (Table 2). FP32 backpropagation through the student normalizer can still make this small expected gradient inaccurate. The log-ratio regrouping does not correct that separate error.

Figure 6 shows the reduced log-ratio error, but regrouping cannot recover differences already rounded away in the supplied normalizers. For sufficiently small hh in the figure’s example, both normalizers round to the same value, leaving a log-ratio of hh only for the perturbed category. In both directions, the expected estimate then scales linearly with hh in magnitude, whereas the true KL is quadratic. The figure’s reference holds the supplied normalizers fixed, isolating error in the subsequent log-ratio calculation.

Reverse and forward sampling value error plots

Figure 6. Regrouping reduces K1’s log-ratio error in both directions. Left: reverse-KL sampling. Right: forward-KL sampling. Exact averages for zref=(−1,0,1)z_{\mathrm{ref}} = (-1, 0, 1) and z=(−1,h,1)z = (-1, h, 1). Value errors isolate arithmetic at the supplied FP32 normalizers.

Fixing the squared log-ratio estimator (K2)

K2 in definition 3 squares the shared log-ratio from equation (18). Rounding in this input propagates to both the penalty and its two input derivatives.

Rounding the log-ratio to zero in the scalar example of § 4.3.2 also erases K2 and both input derivatives. Listing 5 preserves them by regrouping the log-ratio before squaring it. The square is unchanged by reversing the log-ratio, but the sampling weights differ. Given an accurate log-ratio, neither the square nor its derivative introduces further cancellation, although sufficiently small outputs can underflow. Figure 7 compares both sampling directions with their respective expected gradients in Table 2. For sampling from qq, if the score gradients ∇zlog⁡qa\nabla_{z}\log q_a are evaluated exactly, a log-normalizer error shared by every category cancels from this expected gradient because Ea∼q[∇zlog⁡qa]=0\mathbb{E}_{a\sim q}[\nabla_{z}\log q_a]=0. It still changes the expected penalty.

K2 error plots for reverse- and forward-KL sampling

Figure 7. K2 inherits log-ratio error in both sampling directions. First row: reverse-KL sampling. Second row: forward-KL sampling. Exact averages for zref=(−1,0,1)z_{\mathrm{ref}}=(-1,0,1) and z=(−1,h,1)z=(-1,h,1). Value errors isolate arithmetic at the supplied FP32 normalizers. Gradient references are the expected derivatives of each estimator under the exact distributions.

Squared log-ratio (K2) example code

Figure 5 (PDF p. 11). Squared log-ratio (K2) example code

Fixing the control-variate estimator (K3)

The K3 estimator in definition 3 also appears as an estimator of the extended KL divergence in Nielsen & Hadjeres [32], with pp and qq interchanged. The extended KL includes the terms pi−qip_i-q_i, which sum to zero for normalized distributions. Schulman [39] explains the same correction as a zero-mean control variate added to K1.

For reverse-KL sampling, a basic evaluation using the log-ratio from equation (18) and its input derivatives are

k3(sa)=expm1⁡(−sa)+sa,∂k3∂za=1−e−sa,∂k3∂log⁡Z=e−sa−1.(20)k_3(s_a)=\operatorname{expm1}(-s_a)+s_a,\qquad\frac{\partial k_3}{\partial z_a}=1-e^{-s_a},\qquad\frac{\partial k_3}{\partial\log Z}=e^{-s_a}-1. \tag*{(20)}

Although expm1 retains the small exponential increment, adding sas_a still subtracts nearly equal quantities: expm1⁡(−sa)=−sa+sa2/2+O(sa3)\operatorname{expm1}(-s_a)=-s_a+s_a^2/2+O(s_a^3), so the linear terms cancel, leaving a much smaller quadratic remainder. For forward-KL sampling, the same cancellation occurs in k3(−sa)=expm1⁡(sa)−sak_3(-s_a)=\operatorname{expm1}(s_a)-s_a. Its input derivatives are esa−1e^{s_a}-1 and 1−esa1-e^{s_a}.

Gradient computation. The input derivatives in equation (20) also subtract nearly equal quantities when e−sae^{-s_a} is close to one. This is the cancellation in ex−1e^x-1 analyzed by Beebe [4].

Numerical issue 4 (Remainder cancellation and rounded exponential, e.g., as in equation (20)). Rounding in expm1⁡(−sa)\operatorname{expm1}(-s_a) can overwhelm the quadratic remainder after adding sas_a. In the derivatives, e−sae^{-s_a} can round to one, making both derivatives zero. Both failures can occur even with an accurate sas_a.

To address remainder cancellation and rounding of the exponential in the derivative (issue 4), evaluate the remainder using equation (11) and the input derivatives using expm1.

Listing 6: Control-variate estimator (K3) example code

Basic

1 def reverse_k3_basic(
2 z_a, log_Z, z_ref_a, log_Z_ref
3 ):
4     s = sampled_log_ratio_basic(
5         z_a, log_Z, z_ref_a, log_Z_ref)
6     return jnp.expm1(-s) + s

Fixed

1 @custom_gradient(argnums=(0, 1))
2 def reverse_k3_fixed(
3 z_a, log_Z, z_ref_a, log_Z_ref
4 ):
5     s = sampled_log_ratio_fixed(
6         z_a, log_Z, z_ref_a, log_Z_ref)
7     assert_small_log_ratio(s)
8     g = -jnp.expm1(-s)
9     return e2(-s), (g, -g)

Numerical reformulation 6 (Fixing issue 4 for the K3 computation and gradient in equation (20)). Evaluate the remainder with the Taylor strategy of Mächler (2026, pp. 36–37) and its derivative with expm1 Beebe [4]:

k3(sa)=E2(−sa),∂k3∂za=−expm1⁡(−sa),∂k3∂log⁡Z=expm1⁡(−sa).(21)\begin{aligned} k_3(s_a) &= E_2(-s_a), \\ \frac{\partial k_3}{\partial z_a} &= -\operatorname{expm1}(-s_a), \\ \frac{\partial k_3}{\partial\log Z} &= \operatorname{expm1}(-s_a). \tag*{(21)} \end{aligned}

For forward-KL sampling, evaluate k3(−sa)=E2(sa)k_3(-s_a) = E_2(s_a) with input derivatives expm1⁡(sa)\operatorname{expm1}(s_a) and −expm1⁡(sa)-\operatorname{expm1}(s_a).

Listing 6 implements the basic and fixed evaluations in equations (20) and (21). Both take the sampled logits and log-normalizers, and the fixed version checks ∣sa∣≤1/2|s_a| \le1/2.

To isolate K3’s additional cancellation, take za=−hz_a = -h, zref,a=−2hz_{\mathrm{ref},a} = -2h, and log⁡Z=log⁡Zref=0\log Z = \log Z_{\mathrm{ref}} = 0. The direct log-ratio sa=hs_a = h is accurate, but at h=10−8h = 10^{-8} the direct K3 value and both input derivatives are zero. The reformulation preserves the small nonzero penalty and both input derivatives. Ordinary autodiff of the polynomial remainder also preserves these derivatives.

Expected K3 gradients. Figure 8 compares both sampling directions. Both expected gradients have the forward-KL reference in Table 2, although their expected values estimate different KL directions. The fixed update stays close to the reference as the distributions approach equality, including where the direct update rounds to zero.

Four plots comparing basic, fixed, and reference K3 value errors and expected gradients

Figure 8. K3 needs a stable remainder and derivative in both directions. First row: reverse-KL sampling. Second row: forward-KL sampling. Exact averages for zref=(−1,0,1)z_{\mathrm{ref}} = (-1, 0, 1) and z=(−1,h,1)z = (-1, h, 1). Value errors isolate arithmetic at the supplied FP32 normalizers. Gradient references are the expected derivatives of each estimator under the exact distributions.

Fixing the generalized Jensen–Shannon divergence

The generalized JSD compares pp and qq with a shared mixture (Lin, [24], Section IV), and is used for distillation by Agarwal et al. (2024, Section 4.1). Its forward computation inherits the cancellation within each KL term, while its gradient depends on a small mixture log-ratio. The remainder and centering identities from the KL sections address these two issues.

Definition 4 (Generalized JSD). For a mixture weight β∈(0,1)\beta\in(0,1), the generalized JSD is defined by

Lβ=βKL⁡(p∥m)+(1−β)KL⁡(q∥m),m=βp+(1−β)q.(22)L_\beta= \beta\operatorname{KL}(p \mathbin{\|} m) + (1-\beta)\operatorname{KL}(q \mathbin{\|} m), \qquad m = \beta p + (1-\beta)q. \tag*{(22)}

Dividing by β(1−β)\beta(1-\beta) recovers the forward KL as β→0\beta\to0 and the reverse KL as β→1\beta\to1, so the generalized JSD interpolates between the two KL directions (Huszár, [20], Section 6, Eqs. (13)–(16)).

Both KL terms in generalized JSD are susceptible to KL-sum cancellation (issue 2). Raj & Wiggins [37] address this issue using nonnegative series. Here, we apply the KL remainder identity to each term, which requires accurate log-ratios between the distributions and their mixture.

Following Barron et al. [3], we first reconstruct s=log⁡q−log⁡ps = \log q - \log p from logit differences (equation (8)) to avoid log-softmax subtraction (issue 1). The remaining quantity is the mixture log-ratio h=log⁡q−log⁡mh = \log q - \log m. Unlike reverse KL, this log-ratio is a nonlinear function of ss: m/q=1+β(e−s−1)m/q = 1 + \beta(e^{-s} - 1). Centering after this transformation cannot generally remove the common log-normalizer offset in ss, so replacing ss by δ\delta changes the gradient. Reconstructing ss accurately is necessary, but the conversion to hh can introduce further rounding.

Numerical issue 5 (Mixture log-ratio rounding, e.g., as in equation (22)). Even with an accurate ss, forming m/q=1+β(e−s−1)m/q = 1 + \beta(e^{-s} - 1) can round the ratio to one, making h=−log⁡(m/q)h = -\log(m/q) zero. Evaluating log⁡q−log⁡m\log q - \log m instead can subtract separately rounded log-probabilities, as in issue 1. Computing log⁡m\log m with log-sum-exp does not prevent that subtraction error.

Use expm1 to retain e−s−1e^{-s} - 1 and log1p to take the logarithm without first rounding the sum with one:

h=−log⁡1p(βexpm1⁡(−s)).(23)h = -\log1p\left(\beta\operatorname{expm1}(-s)\right). \tag*{(23)}

The two KL terms use log⁡(m/p)=s−h\log(m/p) = s - h and log⁡(m/q)=−h\log(m/q) = -h. With these log-ratios retained, cancellation within each signed KL sum (issue 2) remains. The remainder identity avoids this cancellation.

Numerical reformulation 7 (Fixing issues 1, 2 and 5 for the JSD computation in equation (22)). Using the reconstructed ss and hh from equations (8) and (23), apply the KL remainder identity in equation (10):

Lβ=β∑ipiE2(si−hi)+(1−β)∑iqiE2(−hi).(24)L_\beta= \beta\sum_i p_i E_2(s_i-h_i) + (1-\beta)\sum_i q_i E_2(-h_i). \tag*{(24)}

Evaluate E2E_2 using equation (11).

Gradient computation. The same mixture log-ratio determines the gradient. Differentiating both KL terms in equation (22), including the dependence of mm on qq, gives ∇qLβ=(1−β)h\nabla_q L_\beta= (1-\beta)h. The softmax Jacobian then centers hh under qq, as it centers ss for reverse KL:

∇zLβ=(1−β)q(h−∑iqihi),h=log⁡q−log⁡m.(25)\nabla_z L_\beta= (1-\beta)q\left(h-\sum_i q_i h_i\right), \qquad h = \log q-\log m. \tag*{(25)}

Evaluating hh through equation (23) avoids the log-probability subtraction and ratio rounding in issue 5, retaining the small differences that centering needs.

Numerical reformulation 8 (Fixing issue 5 for the JSD gradient in equation (25)). Evaluate hh with the log1p–expm1 expression in equation (23), then use it in equation (25). The gradient formula is unchanged. The correction preserves the small mixture log-ratio before centering it.

Listing 7 combines the remainder evaluation with the centered gradient rule. Figure 9 compares it with the direct KL sums using a log-sum-exp mixture.

Loss error and gradient norm comparison for Basic, Fixed, and Reference methods

Figure 9. Rounding mixture log-ratios can make JSD gradients inaccurate. This example uses mixture weight β=1/2\beta= 1/2, fixed reference logits zref=(−1,0,1)z_{\mathrm{ref}} = (-1, 0, 1) and varying logits z=(−1,h,1)z = (-1, h, 1).

Related work

Numerical stability of divergences and probability computations. The reformulations use classical techniques for avoiding cancellation (Higham, [17], Sections 1.7 and 1.18), together with stable evaluations of elementary functions (Brent & Zimmermann, [6], Chapter 4). Computational divided differencing similarly Listing 7: Generalized JSD example code

Basic
1 def jsd_basic(z, z_ref, beta):
2     p, q, lp, lq, _ = categorical(z, z_ref)
3     lm = jnp.logaddexp(
4         jnp.log(beta) + lp,
5         jnp.log1p(-beta) + lq,
   )
7     terms = beta * p * (lp - lm)
8     terms += (1 - beta) * q * (lq - lm)
9     return jnp.sum(terms)

Fixed
1 @custom_gradient
2 def jsd_fixed(z, z_ref, beta):
3     assert_close(z, z_ref)
4     p, q, _, _, s = categorical(z, z_ref)
5     hq = -jnp.log1p(beta * jnp.expm1(-s))
6     terms = (
7         beta * p * e2(s - hq)
8         + (1 - beta) * q * e2(-hq)
   )
10    value = jnp.sum(terms)
11    centered = hq - jnp.sum(q * hq)
12    gradient = (1 - beta) * q * centered
13    return value, gradient

transforms function differences to avoid subtracting nearby evaluations [38]. Blanchard et al. [5] analyze rounding errors in softmax and log-sum-exp, which supply the normalization primitives for the categorical objectives here. Accurate normalization alone does not ensure an accurate difference between two nearby probabilities. Barron et al. [3] address this distinction for soft cross-entropy gradients. The reverse-KL and generalized JSD evaluations in § 4.2 and § 5 build on their use of logit differences. Maaz [26] used property-based tests to find NaN gradients in the softmax code in the first version of Barron et al. [3]. Version 3 corrects this bug. We use similar tests to validate our reformulations.

Forward divergence computations also have established numerical treatments. Raj & Wiggins [37] derive a series with nonnegative grouped terms for small weighted JSD, a direct precedent for the forward analysis in § 5. Mächler [27] discusses stable evaluation of xlog⁡(x/M)+M−xx \log(x/M) + M - x, the scalar extended-KL term. The note also analyzes the log1pmx implementation of Welinder [48] for stably evaluating log⁡(1+x)−x\log(1 + x) - x. Using log1p⁡(x)\operatorname{log1p}(x) alone still leaves cancellation when subtracting xx, just as expm1⁡(x)−x\operatorname{expm1}(x) - x can lose the quadratic remainder in our KL, JSD, and K3 formulas. The expm1x routine of Mächler [28] stably evaluates this same exponential remainder using a Taylor expansion near zero. Our reformulations use this established strategy to evaluate divergences.

Numerical accuracy of derivatives. Chen et al. [9] study how BF16 rounding in attention backward can break the zero-sum constraint on softmax score gradients and propose a correction to the resulting query and key gradients.

Monte Carlo gradient estimation is a separate topic, with different bias and variance tradeoffs even in exact arithmetic. Tang & Munos [43] and Shah et al. [41] analyze the expected updates produced by Monte Carlo estimates to the reverse KL with samples from the optimized policy. As discussed in § 4.3, estimator bias can persist in exact arithmetic, whereas the numerical issues concern evaluating the chosen estimator. The reformulations preserve its expected update without correcting bias or removing sampling variability.

Conclusion and further extensions

Nearly equal distributions expose numerical errors in both divergences and their derivatives. This paper develops reformulations for categorical KL, generalized JSD, Monte Carlo KL estimates, and entropy, together with examples that illustrate their benefits and remaining limitations.

These settings are only part of the broader problem. Related numerical questions arise for continuous distributions and broader divergence families. Exponential-family KL identities [47] connect these questions to derivatives of log-partition functions. General ff-divergences and Bregman divergences introduce other differences of nearby function values or gradients. Entropy-regularized optimal transport [11], its debiased Sinkhorn divergences [13], and differentiable dynamic time warping [12] provide further settings in which small discrepancies and their derivatives must be retained. Extending the comparisons to these settings would also need to account for numerical integration, matrix computations, and solver accuracy alongside scalar cancellation.

Acknowledgments

I would like to thank Lucas Maystre, Oliver Liu, Risto Vuorio, and Sam Buchanan for insightful comments and discussions. Language models helped with writing, code, and numerical checks.

Numerical setup and helper implementations

Numerically stable primitives. We assume access to the standard routines expm1⁡(x)\operatorname{expm1}(x) and log1p⁡(x)\operatorname{log1p}(x), which evaluate ex−1e^x-1 and log⁡(1+x)\log(1+x) in a numerically stable way. Brent & Zimmermann [6] describe how series and range reduction preserve these increments without first rounding exe^x or 1+x1+x near one.

Experiments and code. The paper provides small numerical examples to illustrate the issues and fixes, comparing basic and fixed FP32 evaluations on a CPU. Reference values and gradients use mpmath with 80 decimal digits. Following Maaz [26], we use Hypothesis (MacIver et al. [29]) for property-based testing of all FP32 reformulations against these references. The categorical checks cover equality, category permutations, representable common logit shifts, and forward and gradient accuracy for K∈{2,3,4,8,16}K \in\{2, 3, 4, 8, 16\}.

Custom gradients. The custom_gradient decorator wraps a function returning (value, gradient), with the derivative taken with respect to its first input by default. The decorated function returns only the value and uses the supplied derivative for backpropagation. For K3 (listing 6), argnums=(0, 1) selects zaz_a and log⁡Z\log Z, with derivatives returned in that order as (g,−g)(g, -g). The remaining inputs are held fixed.

Log-ratio reconstruction. The helper categorical reconstructs the log-ratio in equation (8) from logit differences and returns pp, qq, log⁡p\log p, log⁡q\log q, ss.

def categorical(z, z_ref):
    lp = jax.nn.log_softmax(z_ref)
    lq = jax.nn.log_softmax(z)
    p, q = jnp.exp(lp), jnp.exp(lq)
    delta = z - z_ref
    delta = delta - delta[0]
    d = delta - jnp.sum(p * delta)
    s = d - jnp.log1p(jnp.sum(p * jnp.expm1(d)))
    return p, q, lp, lq, s

Exponential remainder. The helper e2 evaluates E2(x)=ex−1−xE_2(x) = e^x - 1 - x using the series in equation (11) for ∣x∣<1/2|x| < 1/2 and expm1⁡(x)−x\operatorname{expm1}(x) - x otherwise. Its degree-12 polynomial retains the quadratic leading term without subtracting first-order quantities. Mächler [28] (expm1x, pp. 36–37) uses the same Taylor strategy. Exponential remainders also appear in exponential integrators (Hochbruck & Ostermann, [19], Eqs. (2.10)–(2.11)). Ordinary autodiff of the polynomial retains the small derivative in the plotted examples. An explicit backward rule can instead use E2′(x)=expm1⁡(x)E_2'(x) = \operatorname{expm1}(x), as in listing 6.

def e2(x, degree=12, cutoff=0.5):
    poly = jnp.full_like(x, 1 / math.factorial(degree))
    for k in range(degree - 1, 1, -1):
        coeff = 1 / math.factorial(k)
        poly = coeff + x * poly
    return jnp.where(jnp.abs(x) < cutoff, x * x * poly, jnp.expm1(x) - x)

Closeness guard. The full-distribution KL and JSD implementations call assert_close before reconstructing the log-ratio. It checks that the difference between the largest and smallest entries of δ=z−zref\delta= z - z_{\mathrm{ref}} is finite and at most limit, which defaults to 1/21/2. In exact arithmetic, this bounds every ∣si∣|s_i| by the same limit: log⁡(∑ipieδi)\log\left(\sum_i p_i e^{\delta_i}\right) lies between the smallest and largest δi\delta_i. The condition is invariant to common logit shifts in exact arithmetic. The implemented check uses the rounded logit differences and controls the exponential range without guaranteeing numerical accuracy. The listing uses an ordinary assertion. The compiled JAX examples check the same condition with checkify.

In the rounding example of Figure 3, the exact logit differences have a span of h=2−25h = 2^{-25} and the rounded differences (1,1)(1, 1) have a span of zero, so both pass the guard. Both FP32 softmax outputs also equal (1/2,1/2)(1/2, 1/2), so checking the probabilities would not detect this failure.

def assert_close(z, z_ref, limit=0.5):
    delta = z - z_ref
    span = jnp.max(delta) - jnp.min(delta)
    assert jnp.isfinite(span) & (span <= limit)

References

  1. [2]Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251–276, 1998.
  2. [3]Jonathan T. Barron, Dan Gnanapragasam, and Michael Tschannen. A fix for softmax and cross-entropy. arXiv preprint arXiv:2610.31642, 2026. Version 3, October 29, 2026. See also the accompanying implementation.arxiv.org/abs/2610.31642
  3. [4]Nelson H. F. Beebe. Computation of expm1(x) = exp(x) − 1. Technical report, University of Utah, 2002.
  4. [5]Pierre Blanchard, Desmond J. Higham, and Nicholas J. Higham. Accurately computing the log-sum-exp and softmax functions. IMA Journal of Numerical Analysis, 41(4):2311–2330, 2021.DOI
  5. [6]Richard P. Brent and Paul Zimmermann. Modern Computer Arithmetic, volume 18 of Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2010.DOI
  6. [7]Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535–541, 2006.DOI
  7. [8]Bradley P. Carlin and Nicholas G. Polson. Monte Carlo Bayesian methods for discrete regression models and categorical time series. In Bayesian Statistics 4, pp. 577–586. Oxford University Press, 1992.DOI
  8. [9]Junlin Chen, Daize Dong, Huanwei Di, Haolong Jia, Jiawei Wu, and others. Broken symmetry in BF16 attention: Why Flash Attention gradients blow up late in training. arXiv preprint arXiv:2609.34272, 2026. Version 1.arxiv.org/abs/2609.34272
  9. [10]Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley & Sons, second edition, 2006.
  10. [11]Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, volume 26, 2013.
  11. [12]Marco Cuturi and Mathieu Blondel. Soft-DTW: A differentiable loss function for time-series. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 894–903, 2017.arxiv.org/abs/1703.01541
  12. [13]Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and others. Interpolating between optimal transport and MMD using Sinkhorn divergences. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 2681–2690, 2019.arxiv.org/abs/1810.08278
  13. [14]Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2160–2169, 2019.arxiv.org/abs/1901.11275
  14. [15]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In International Conference on Learning Representations, 2024.DOI
  15. [16]John R. Hershey and Pedar A. Olsen. Approximating the Kullback Leibler divergence between Gaussian mixture models. In IEEE International Conference on Acoustics, Speech and Signal Processing, volume 4, pp. IV–317–IV–320, 2007.DOI
  16. [17]Nicholas J. Higham. Accuracy and Stability of Numerical Algorithms. Society for Industrial and Applied Mathematics, second edition, 2002.DOI
  17. [18]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
  18. [19]Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica, 19:209–286, 2010.DOI
  19. [20]Ferenc Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? arXiv preprint arXiv:1511.05101, 2015.arxiv.org/abs/1511.05101
  20. [21]Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327. Association for Computational Linguistics, 2016.arxiv.org/abs/1606.07947
  21. [22]Kimi Team. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12599, 2025.arxiv.org/abs/2501.12599
  22. [23]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.arxiv.org/abs/1412.6980
  23. [24]Jianhua Lin. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1): 145–151, 1991.DOI
  24. [25]Kevin Lu. On-policy distillation. Thinking Machines Lab, 2025. October 27, 2025.
  25. [26]Muhammad Maaz. Property-based testing for robust softmax and cross-entropy. arXiv preprint arXiv:2610.36427, 2026.arxiv.org/abs/2610.36427
  26. [27]Martin Mächler. log1pmx(), bd0(), stirrer(): Computing Poisson, binomial, gamma probabilities in R. Technical report, ETH Zurich, 2021. Updated September 2026. Documents Morten Welinder’s 2005 logarithmic-remainder algorithm.
  27. [28]Martin Mächler. DPQ: Density, Probability, Quantile Computations, 2026. R package version 0.6-2. expm1x documentation, pp. 36–37.
  28. [29]David R. MacIver, Zac Hatfield-Dodds, and Many Other Contributors. Hypothesis: A new approach to property-based testing. Journal of Open Source Software, 4(43):1891, 2019.
  29. [30]James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 (146):1–76, 2020.
  30. [31]Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, and others. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1928–1937, 2016.arxiv.org/abs/1602.01783
  31. [32]Frank Nielsen and Gaëtan Hadjeres. Monte Carlo information geometry: The dually flat case. arXiv preprint arXiv:1803.07225, 2018.DOI
  32. [33]Takeshi Ogita, Siegfried M. Rump, and Shin’ichi Oishi. Accurate sum and dot product. SIAM Journal on Scientific Computing, 26(6):1955–1988, 2005.DOI
  33. [34]Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, and others. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.arxiv.org/abs/2203.02155
  34. [35]Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 1607–1612, 2010.DOI
  35. [36]Lucas Prieto, Melih Barsbey, Pedro A. M. Mediano, and Tolga Birdal. Grokking at the edge of numerical stability. In International Conference on Learning Representations, 2025.arxiv.org/abs/2501.04697
  36. [37]Anil Raj and Chris H. Wiggins. A non-negative expansion for small Jensen–Shannon divergences. arXiv preprint arXiv:0810.5117, 2008.arxiv.org/abs/0810.5117
  37. [38]Thomas W. Reps and Louis B. Rall. Computational divided differencing and divided-difference arithmetics. Higher-Order and Symbolic Computation, 16(1–2):93–149, 2003.
  38. [39]John Schulman. Approximating KL divergence. Online note, 2020.
  39. [40]John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. In International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889–1897, 2015.arxiv.org/abs/1502.05477
  40. [41]Vedant Shah, Johan Obando-Ceron, Vineet Jain, Brian R. Bartoldson, Bhavya Kailkhura, and others. A comedy of estimators: On KL regularization in RL training of LLMs. arXiv preprint arXiv:2512.21852, 2025. Version 4, revised August 25, 2026.arxiv.org/abs/2512.21852
  41. [42]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, and others. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  42. [43]Yunhao Tang and Rémi Munos. On a few pitfalls in KL divergence gradient estimation for RL. arXiv preprint arXiv:2506.09477, 2025.arxiv.org/abs/2506.09477
  43. [44]Marc Teboulle. Entropic proximal mappings with applications to nonlinear programming. Mathematics of Operations Research, 17(3):670–690, 1992.DOI
  44. [45]Thinking Machines Lab. Tinker documentation, 2026. Accessed November 6, 2026.
  45. [46]Emanuel Todorov. Linearly-solvable Markov decision problems. In Advances in Neural Information Processing Systems, volume 19, 2006.DOI
  46. [47]Martin J. Wainwright and Michael I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1–2):1–305, 2008.
  47. [48]Morten Welinder. pgamma discontinuity (PR#7307). R-devel mailing list, January 11, 2005. Includes the log1pmx implementation.
  48. [49]Ronald J. Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268, 1991.DOI

Paper details

vlnc:W_jepe7774
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents