CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Introduction

Knowledge distillation (KD) transfers knowledge from a teacher to a smaller student, typically by matching their predictive distributions [1]. In language models, standard token-level KD assumes a shared vocabulary and aligned prediction positions. When teacher and student use different tokenizers, the same text is segmented differently which requires token level alignment, and their distributions assign probability mass to different token outcomes. Direct distribution matching therefore requires both aligning prediction contexts and establishing corresponding outcomes across vocabularies. These requirements complicate token-level KD and restrict the choice of teachers.

We tackle the issue of logits distillation between models of different families and different tokenizers. Existing approaches differ in how they handle vocabulary and segmentation mismatches. GOLD [2] directly matches teacher and student probabilities for tokens with identical strings in both vocabularies. For tokens without such a match, it sorts each model’s remaining probabilities and compares them by rank, ignoring token identity. X-Token identifies two limitations of this partition: loss of task-critical token identity and suppressive gradients on unmatched tokens [3]. Its P-KL and H-KL modes use projection and relaxed hybrid matching, respectively. However, selecting between them relies on a manual coverage audit for each tokenizer pair. Approximate Likelihood Matching (ALM) provides a uniform formulation across tokenizer pairs by matching the probability assigned to each observed aligned chunk and aggregating all remaining probability mass into a single residual event [4]. While broadly applicable, this aggregation discards fine-grained dark knowledge about how the teacher distributes probability among alternative continuations.

We introduce Common-Support Distillation (CSD), which makes teacher and student predictions comparable by organizing them over a shared support of text-equivalent continuation events. We first align token chunks that decode to the same observed text span. For one-to-one chunks, the support consists of common-token pairs from the two vocabularies. For one-to-many or many-to-one chunks, each event pairs a single token from one model with a token sequence from the other. For example, given the observed alignment ($120 \leftrightarrow[1] [2] [0]$), fixing the prefix [1] [2] and varying the final token yields $120 \leftrightarrow[1] [2] [0], \ldots, 129 \leftrightarrow[1] [2] [9]$. We fix the within-chunk prefix to [1] [2] because the forward pass on the observed sequence provides the conditional distributions along this prefix. A sequence event is assigned the product of its constituent conditional token probabilities. For accepted many-to-many chunks, the observed alignment forms the sole explicit event. Finally, a residual event captures probability mass outside the selected support, yielding normalized distributions that CSD matches with a single objective across all chunk types and tokenizer pairs.

Our contributions are summarized as follows, and the implementation of CSD is publicly available¹.

• A uniform cross-tokenizer objective. CSD matches teacher and student distributions over a shared support of paired token sequences plus a residual event. The same distribution-matching objective applies across chunk shapes and tokenizer pairs without pair-specific loss selection.

• Consistent gains across model families. Across four settings Llama-3.2-1B and Gemma3-1B-PT students and Qwen3-4B and Phi-4-mini teachers, CSD achieves the highest average accuracy, outperforming X-Token by 0.66–5.38 points and ALM by 1.33–3.30 points, while also surpassing continued pretraining and same-tokenizer KD.

• Efficient top-$k$ distillation. In the Qwen–Llama setting, using only the top 16 or 128 teacher tokens reduces the five-task average by just 1.74% or 0.74% relative to full logits, substantially lowering training-step time with minimal accuracy loss.

• Gains at larger scales and training stages. For the Llama-3.1-8B base student, CSD outperforms continued pretraining by 2.17–4.43 points and ALM by 2.61–4.23 points across two teachers. For Llama-3.1-8B-Instruct, it improves over continued training and ALM by 3.90 and 1.60 points.

Related Work

Existing cross-tokenizer distillation methods differ in how they make teacher and student predictions comparable. We group these approaches into representation-based methods and logits-based methods.

Representation-based methods. These methods access hidden states to align different output spaces. Dual-Space Knowledge Distillation (DSKD) learns projections between teacher and student representations and uses cross-model attention to align sequences before comparing predictions [5]. Byte-Level Distillation (BLD) attaches a byte-level prediction head to student hidden states and trains it using teacher probabilities mapped to byte-level outcomes [6].

Logits-based methods. These methods derive supervision from existing output logits, using tokenizer information for alignment. Several approaches transfer probabilities through token correspondences: MinED uses edit-distance alignment [7]; Contextual Dynamic Mapping (CDM) refines sequence and vocabulary mappings using contextual predictions [8]; and SeDi combines graph-based token-span alignment, vocabulary re-encoding, and entropy alignment [9]. Other approaches change the quantities being compared. Universal Logit Distillation (ULD) matches sorted probability vectors, discarding token identity [10], and MultiLevelOT extends this optimal-transport formulation to both the token and sequence levels [11]. GOLD utilizes a hybrid loss formulation where KL is used for common tokens in both vocabularies and ULD loss for tokens without such a match [2]. X-Token addresses limitations of GOLD through two loss modes: projection (P-KL) and relaxed hybrid (H-KL) matching. However, it requires a manual coverage audit to decide the loss mode for each tokenizer pair [3]. Approximate Likelihood Matching (ALM) instead uses a common chunk-level formulation across tokenizer pairs, matching distributions over the observed aligned chunk and its complement [4]. However, it does not individually match the probabilities of alternative tokens.

Cross-tokenizer on-policy distillation. A recent line of work extends these ideas to on-policy distillation (OPD), where the teacher scores trajectories generated by the student. Beyond GOLD, DPCA aligns student and teacher sequences into minimal synchronized chunks for on-policy training [12]. SimCT compares the two models on short continuations that both tokenizers can express, supervising only the positions and candidates where such continuations exist [13]. Byte-Prefix Marginalization (BPM) re-expresses the teacher’s next-token distribution over the full student vocabulary by routing each teacher token’s probability to the longest student token whose bytes prefix it, with unmatched mass assigned to a residual category [14].

CSD shares with ALM and BPM the use of a residual event to keep the matched distributions normalized. It differs in how the explicit support is formed. ALM matches only the observed chunk, and BPM maps teacher mass into the student vocabulary through byte-prefix routing. CSD instead constructs a shared support on both sides: common-token pairs with identical decoded text for one-to-one chunks, and chain-rule events over alternative final tokens for one-to-many and many-to-one chunks. This symmetric construction requires only text equivalence and makes no assumption about how either tokenizer segments a candidate continuation. Although we study CSD in the off-policy setting, its per-chunk construction depends only on the observed text and both models’ predictions, so it applies unchanged to on-policy trajectories.

Common-Support Distillation

CSD matches teacher and student predictions of corresponding textual outcomes, conditioned on the same preceding text. It comprises three stages. First, sequence alignment identifies paired token ranges covering the same substring using character offsets and decoded-text checks. Second, common-support construction maps each chunk’s predictions onto shared events defined by common-token pairs or token sequences representing the same text (Figure 1). Event probabilities are computed directly from token predictions or through the autoregressive chain rule. A residual event retains all remaining mass, ensuring normalization. Finally, distribution matching applies a uniform objective across chunk shapes and teacher–student tokenizer pairs. Algorithm 1 in Appendix C summarizes the computation of the CSD training objective for one batch.

Sequence alignment

Token-level predictions. Let a text sequence $x$ be tokenized by the student and teacher as $\mathbf{s} = (s_1, \ldots, s_{n_S})$, $\mathbf{t} = (t_1, \ldots, t_{n_T})$, with vocabularies $\mathcal{V}_S$ and $\mathcal{V}_T$, respectively. Their next-token distributions are denoted by $p_S(\cdot\mid s_{<i})$ over $\mathcal{V}_S$ and $p_T(\cdot\mid t_{<j})$ over $\mathcal{V}_T$. In this case, even when $i = j$, the histories may represent different text. Moreover, their distributions assign probabilities to different vocabulary tokens. Thus, direct token-level matching is not suitable here.

Aligned chunks. We first define $(i,j)$ as a common boundary if the histories $\mathbf{s}_{<i}$ and $\mathbf{t}_{<j}$ cover the same prefix of the original text. Then, given two consecutive common boundaries $(i,j)$ and $(i + M, j + N)$, with $M,N \ge1$, we can define an aligned chunk $c = (\mathbf{s}^{c}, \mathbf{t}^{c})$ covering the intervening substring, where $\mathbf{s}^{c} = (s_i, \ldots, s_{i+M-1}) = (s^{c}_1, \ldots, s^{c}_M)$, $\mathbf{t}^{c} = (t_j, \ldots, t_{j+N-1}) = (t^{c}_1, \ldots, t^{c}_N)$. The shape $M:N$ records the student and teacher token counts for the aligned chunk. We term aligned chunks with $M = N = 1$ as one-to-one and chunks with $M > 1$ or $N > 1$ as non-one-to-one. In cross-tokenizer distillation, the aim is to match the probabilities of corresponding student and teacher realizations within each aligned chunk.

Finding aligned chunks with character offsets. For student tokens $s_r$ and teacher tokens $t_r$, their tokenizers provide character spans $[a^S_r, b^S_r)$ and $[a^T_r, b^T_r)$ in the original text. Thus, we use a two-pointer scan that starts from tokens with the same start offset and extends whichever range ends earlier until the end offsets first coincide. That is, for an aligned chunk $c=(s^{c},t^{c})$, the two token ranges share the same character span $[a_c,b_c)$, with $a_c:=a_i^S=a_j^T$, $b_c:=b_{i+M-1}^S=b_{j+N-1}^T$, and $a_c<b_c$. The retained set $\mathcal{C}$ excludes unmatched ranges and candidates that fail the decoded-chunk checks, which allow Unicode normalization and specified whitespace equivalences. We provide the full procedure for offset-based alignment in Appendix § 7.

Common-support construction

Alignment identifies the observed token ranges and their histories. We next define the events whose probabilities are matched within each aligned chunk. This event set forms the common support $\mathcal{E}_c$, constructed as shown in Figure 1. For one-to-one chunks, each explicit event pairs student and teacher realizations of the same decoded text. Each retained token-pair set contains no repeated student or teacher token IDs. For non-one-to-one chunks, we build virtual events whose probabilities can be computed for both models. These events, together with a residual event, define the common support for the distributions $P_S^c$ and $P_T^c$ compared in distillation.

One-to-one chunks

Common-token pairs. For one-to-one chunks (Figure 1(a)), the common support can be built by checking the token pairs in the student vocabulary $\mathcal{V}_S$ and teacher vocabulary $\mathcal{V}_T$ that decode to the same text fragment. For $u\in\mathcal{V}_S$ and $v\in\mathcal{V}_T$, we define the common-token pairs as: $\mathcal{M}=\{(u_k,v_k)\}_{k=1}^{K}\subseteq\mathcal{V}_S\times\mathcal{V}_T$, where $K$ denotes the total number of retained token pairs. This relation is shared across one-to-one chunks for a tokenizer pair and includes alternatives beyond the observed tokens.

Normalized event distributions. Given the token pairs, each event $e_k^c$ pairs student token $u_k$ with teacher token $v_k$, while $e_{\mathrm{res}}^c$ collects unpaired tokens. The common support is $\mathcal{E}_c=\{e_1^c,\ldots,e_K^c,e_{\mathrm{res}}^c\}$. Let $h_S^c=s_{<i}$ and $h_T^c=t_{<j}$ denote the student and teacher token histories preceding chunk $c$. These histories represent the same preceding text. In this event order, their distributions can be represented as $P_S^c=\left(p_S(u_1\mid h_S^c),\ldots,p_S(u_K\mid h_S^c),1-\sum_{k=1}^{K}p_S(u_k\mid h_S^c)\right)$, $P_T^c=\left(p_T(v_1\mid h_T^c),\ldots,p_T(v_K\mid h_T^c),1-\sum_{k=1}^{K}p_T(v_k\mid h_T^c)\right)$. The residual incorporates unmatched tokens through their aggregate probability mass, avoiding potentially inaccurate token-level correspondences as in GOLD [2]. X-Token [3] identifies suppression in GOLD’s original common-token loss: a common-token-only loss computed from full-vocabulary softmax probabilities pushes unmatched student logits downward. For that loss, including the residual makes the gradient depend on the teacher–student residual-mass mismatch, allowing either upward or downward pressure. Thus, even a task-critical token is not always penalized for lacking a common-token counterpart. ALM uses the observed chunk as its sole explicit event [4], whereas CSD expands this to include alternative common tokens (Appendix § 9).

Non-one-to-one chunks

Virtual common support. When segmentation differs, corresponding text requires multiple tokens on at least one side. We therefore construct a virtual common support from paired token sequences. For a chunk of shape $M:N$ with $(M,N)\ne(1,1)$, we fix the observed prefixes $s_{<M}^c$ and $t_{<N}^c$ and allow only the final tokens to vary. The prefixes contain $M-1$ and $N-1$ tokens, respectively, and are empty on a single-token side $(M=1\text{ or }N=1)$.

Let $\operatorname{Dec}_S$ and $\operatorname{Dec}_T$ denote the decoding functions and $\Vert$ token-sequence concatenation. A final-token pair $(u,v)\in\mathcal{V}_S\times\mathcal{V}_T$ defines a paired completion if $\operatorname{Dec}_S(s_{<M}^c\Vert u)=\operatorname{Dec}_T(t_{<N}^c\Vert v)$. The paired sequences must decode to the same text, which need not equal the observed chunk text; their token counts remain $M$ and $N$. Let the retained valid final-token pairs form the candidate set $\mathcal{A}_c=\{(u_k^c,v_k^c)\}_{k=1}^{K_c}$, with no repeated student or teacher token IDs. Each shared event $e_k^c$ pairs $s_{<M}^c\Vert u_k^c$ and $t_{<N}^c\Vert v_k^c$.

We define shared events this way, with fixed prefixes of $M-1$ and $N-1$ tokens, so that their probabilities can be computed from the observed-prefix predictions using the chain rule:

An empty product equals one. Each $q_S^c(k)$ and $q_T^c(k)$ is the probability of a specific token-sequence realization, computed by the chain rule without length normalization.

For one-to-many or many-to-one chunks ($M = 1$ or $N = 1$, Figure 1(b)), one side represents each candidate text with a single token. We can therefore precompute bidirectional mappings between student and teacher tokens, enabling $O(1)$ lookup per mapping entry when retrieving valid final-token pairs. In contrast, for many-to-many chunks ($M, N > 1$, Figure 1(c)), valid pairs depend on the observed prefixes on both sides, making exhaustive precomputation impractical. Constructing a larger support therefore requires additional chunk-specific matching. To avoid this cost, we retain only the observed final-token pair $(s_M^c, t_N^c)$. This is a computational choice rather than a requirement of the objective. We evaluate richer many-to-many support in § 4.3.

Normalized event distributions. Each pair defines an event $e_k^c$ with probabilities $q_S^c(k)$ and $q_T^c(k)$. In the order $(e_1^c, \ldots, e_{K_c}^c, e_{\mathrm{res}}^c)$, $P_m^c = \left(q_m^c(1), \ldots, q_m^c(K_c), 1 - \sum_{k=1}^{K_c} q_m^c(k)\right)$, $m \in\{S,T\}$. Within each model, candidates share a fixed prefix and have distinct final tokens. They therefore define mutually exclusive token-sequence events whose total probability is at most one, ensuring a nonnegative residual. For accepted many-to-many chunks, we have $K_c = 1$, resulting in a binary distribution similar to ALM [4]. For one-to-one, one-to-many, and many-to-one with $K_c > 1$, the expanded support provides finer-grained supervision by preserving the relative probabilities of multiple alternative events. Appendix § 6.§ 4 reports the empirical support-size distributions.

Top-$k$ support selection

CSD is compatible with top-$k$ selection on either or both models, or directly over paired events. We can treat top-$k$ selection as restricting the candidate vocabulary used for common-support construction at each prediction position. For student position $i$ and teacher position $j$, define $\mathcal{V}_{S,k} = \operatorname{TopK}_{u \in\mathcal{V}_S} p_S(u \mid s_{<i})$, $\mathcal{V}_{T,k} = \operatorname{TopK}_{v \in\mathcal{V}_T} p_T(v \mid t_{<j})$, where TopK returns the set of $k$ highest-probability tokens.

The reduced vocabularies restrict which token pairs enter the common support. For one-to-one chunks, only common-token pairs whose tokens remain in the candidate vocabularies contribute. For non-one-to-one chunks, selection restricts the final-token pairs at student position $i + M - 1$ and teacher position $j + N - 1$. The observed within-chunk prefixes remain fixed, and their conditional probabilities must also be available for chain-rule scoring. Thus, top-$k$ selection reduces the explicit common support.

Retained probabilities come from the original full-vocabulary distributions without renormalization. Excluded mass is absorbed by the existing residual event on each side, so both event distributions remain normalized and the matching objective retains the same. Our experiments evaluate teacher-side top-$k$ in § 4.4.

Distribution matching

Both single-token and multi-token constructions yield normalized event distributions $P_T^c$ and $P_S^c$, allowing us to use one matching rule. Although support sizes may differ across chunks, the teacher and student distributions refer to the same events within each chunk. A retained chunk is accepted for KD if its support contains at least one explicit event ($K_c \ge1$). Chunks that do not meet this criterion contribute no KD term.

Per-chunk loss. For an accepted chunk $c$, we use $\ell(c) = \mathcal{D}(P_T^c, P_S^c)$, where $\mathcal{D}$ can be any suitable divergence, including forward KL (FKL), reverse KL (RKL), or Jensen–Shannon divergence (JSD). The loss compares the probabilities of all explicit events together with the residual mass, retaining aggregate supervision for outcomes outside the explicit support. The same selected $\mathcal{D}$ applies across chunk shapes and tokenizer pairs, without pair-specific loss selection.

Batch loss. Let $\mathcal{C}_{1:1}$ and $\mathcal{C}_{\ne}$ be the accepted one-to-one and non-one-to-one chunks in a batch. Each accepted chunk contributes one loss term. For a nonempty accepted set, we aggregate these terms as $\mathcal{L}_{\mathrm{KD}}^{\beta,\mathcal{D}} = \left(\sum_{c \in\mathcal{C}_{1:1}} \ell(c) + \beta\sum_{c \in\mathcal{C}_{\ne}} \ell(c)\right)/(|\mathcal{C}_{1:1}| + |\mathcal{C}_{\ne}|)$. The configurable weight $\beta\ge0$ controls the relative contribution of non-one-to-one chunks. It weights their losses without changing the underlying support. The denominator counts accepted chunks independently of $\beta$. If none is accepted, we set $\mathcal{L}_{\mathrm{KD}}^{\beta,\mathcal{D}} = 0$.

Dynamic joint training. Following X-Token [3], we combine the student’s next-token cross-entropy $\mathcal{L}_{\mathrm{CE}}$ with dynamically rescaled KD. Cross-entropy supervises the observed student tokens, while KD transfers the teacher’s probabilities over the shared events. For $\mathcal{L}_{\mathrm{KD}}^{\beta,\mathcal{D}} > 0$, $\mathcal{L}_{\mathrm{train}} = \operatorname{sg}\left(\mathcal{L}_{\mathrm{CE}}/\mathcal{L}_{\mathrm{KD}}^{\beta,\mathcal{D}}\right)\mathcal{L}_{\mathrm{KD}}^{\beta,\mathcal{D}} + \mathcal{L}_{\mathrm{CE}}$, where

Main Results at 1B Scale

sg stops gradients through the entire ratio. The scaling factor is computed from the current batch losses and treated as a constant during backpropagation. This balances scalar loss values, not necessarily gradient magnitudes. Student gradients flow through all retained chain factors and explicit and residual probabilities; the teacher remains fixed. If KD is zero, training uses $\mathcal{L}_{\mathrm{CE}}$ alone.

Experiments

Experimental Setup

Models and data. Our main experiments use Llama-3.2-1B [15] and Gemma3-1B-PT [16] as students and four teachers: Llama-3.2-3B and Gemma3-4B-PT are tokenizer-matched teachers for their respective students, while Qwen3-4B-Base [17] and Phi-4-mini [18] are cross-tokenizer teachers. To evaluate larger students at different training stages, we also use Llama-3.1-8B: we distill the base checkpoint from Qwen3-14B-Base and Phi-4-14B, and the instruction-tuned checkpoint from Qwen3-14B. We use the NVIDIA CLIMB corpus [19] for all base-student experiments and Nemotron-Cascade-SFT [20] for the instruction-tuned experiment.

Training. The 1B experiments run for 30,000 steps with an effective batch size of 768 and a context length of 4096. We use AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, gradient clipping at 1.0, an initial student learning rate of $5 \times10^{-5}$, and a weight decay of 0.001. The 8B scaling experiments use the same settings and run for 2,000 steps. The default CSD configuration uses $\mathcal{D} = \mathrm{JSD}$ and $\beta= 5$.

Evaluation. Our primary evaluation for base models comprises MMLU [21], GSM8K [22], MATH [23], WinoGrande [24], and MBPP [25]. We use 3-shot prompting, greedy decoding for generative tasks, and the same lm-evaluation-harness configuration for every method. For the instruction-tuned student, we evaluate 0-shot GSM8K, MATH, IFEval [26], MBPP, and MBPP+ [27], following task-specific instruction-tuned protocols held fixed across methods.

Baselines. We compare against the unmodified base model, continued pretraining (CPT) without a teacher, and same-tokenizer KD. Cross-tokenizer baselines include ALM [4], ULD [10], GOLD [2], and X-Token [3]. For X-Token, P-KL and H-KL denote its projection and hybrid configurations, respectively. We select P-KL or H-KL for each teacher–student pair following the coverage analysis in the original work. The selected configuration is indicated in each table. For ALM, we use chunk temperature $\tau= 10$ and binary cross-entropy (BCE), without its hidden-state distillation component. Appendix § 9 compares the differences between CSD and baselines. In our experiments, all cross-tokenizer methods use the same offset-based alignment, and GOLD, X-Token, and CSD use JSD in the main comparisons. For each student setting, all continued-training runs use the same data and update budget. Standalone teacher scores are reported in Table 11 for reference and are excluded from comparisons among student models.

Table 11 | Standalone teacher performance under the same 3-shot evaluation protocol used for the 1B students. Avg.-5 is the unweighted mean over the five displayed tasks.

Teacher	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5
Llama-3.2-3B	55.97	70.01	24.94	8.82	38.20	39.59
Gemma3-4B-PT	59.34	73.16	35.78	15.26	45.80	45.87
Qwen3-4B-Base	72.43	72.30	84.61	27.76	67.60	64.94
Phi-4-mini	68.32	74.90	82.71	19.30	56.00	60.25

Main Results at 1B Scale

Llama-3.2-1B student. Table 1 reports the results for Llama-3.2-1B. GOLD can suffer from an uncommon-token failure: task-critical tokens, including numeral tokens important for mathematical reasoning, may fall outside its matched set and receive harmful supervision, as discussed by X-Token [3]. X-Token handles the two teacher–student pairs through different loss modes: P-KL with the Qwen teacher and H-KL with the Phi teacher, both outperforming their corresponding GOLD baselines. In contrast, CSD uses the same event-matching objective for both pairs and further improves over X-Token by 1.38 and 0.66 average points, reaching the highest five-task averages of 33.84 and 35.84, respectively. Across the two teachers, CSD also improves over ULD by 3.38–4.66 points, over ALM by 1.92–3.30 points, and over GOLD by 0.84–2.80 points. It further exceeds CPT by 3.05–5.05 points and same-tokenizer KD from Llama-3.2-3B by 2.85–4.85 points. These results show that CSD benefits from both cross-tokenizer teachers, outperforming CPT and same-tokenizer KD without pair-specific loss selection.

Setting	Method	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5
No distillation	Llama-3.2-1B (base)	32.05	61.48	5.69	5.48	26.40	26.22
	Continued pretraining	42.14	62.27	14.71	7.82	27.00	30.79
Same tokenizer	Llama-3.2-3B	43.94	62.75	12.28	7.76	28.20	30.99
Cross-tokenizer Qwen3-4B	ULD	39.19	61.96	12.74	6.40	25.60	29.18
	ALM	42.81	62.98	16.07	8.74	29.00	31.92
	GOLD	45.71	<u>62.43</u>	11.68	5.56	29.80	31.04
	X-Token (P-KL)†	<u>45.63</u>	62.19	15.69	7.78	<u>31.00</u>	<u>32.46</u>
	CSD (ours)	45.22	61.80	21.30	<u>8.26</u>	32.60	33.84
Cross-tokenizer Phi-4-mini	ULD	41.64	63.38	19.94	6.54	<u>30.80</u>	32.46
	ALM	44.41	62.35	18.88	8.68	28.40	32.54
	GOLD	46.42	62.98	26.46	<u>9.34</u>	29.80	35.00
	X-Token (H-KL)†	46.65	<u>63.22</u>	26.08	9.36	30.60	<u>35.18</u>
	CSD (ours)	<u>46.47</u>	62.98	26.91	9.22	33.60	35.84

Gemma3-1B-PT student. Gemma3-1B-PT provides a challenging transfer setting with its SentencePiece-based tokenizer. As shown in Table 2, GOLD yields zero MBPP accuracy with both teachers and weak mathematical reasoning results, consistent with the uncommon-token failure discussed above (see Appendix § 6.§ 2). As a result, X-Token uses P-KL for both teacher–student pairs. This configuration is effective with Qwen, reaching a five-task average of 27.04, but remains below CPT with Phi: its average is 23.60, with GSM8K and MATH scores of only 1.74 and 3.40.

Setting	Method	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5
No distillation	Gemma3-1B-PT (base)	26.44	61.72	1.59	3.58	8.20	20.31
	Continued pretraining	36.00	58.96	10.77	5.10	15.00	25.17
Same tokenizer	Gemma3-4B-PT	38.51	61.25	9.25	5.30	18.40	26.54
Cross-tokenizer Qwen3-4B	ULD	25.99	<u>60.85</u>	4.47	2.96	13.20	21.49
	ALM	37.25	<u>60.46</u>	11.30	5.08	<u>17.80</u>	26.38
	GOLD	41.23	60.69	4.32	4.62	0.00	22.17
	X-Token (P-KL)†	39.40	60.93	<u>12.59</u>	5.10	17.20	27.04
	CSD (ours)	<u>39.56</u>	60.93	13.04	5.40	19.60	27.71
Cross-tokenizer Phi-4-mini	ULD	31.63	61.07	7.88	4.08	14.00	23.73
	ALM	38.12	61.33	<u>12.36</u>	<u>5.02</u>	<u>19.00</u>	<u>27.17</u>
	GOLD	<u>41.21</u>	60.93	1.36	3.92	0.00	21.48
	X-Token (P-KL)†	39.72	62.35	1.74	3.40	10.80	23.60
	CSD (ours)	41.28	<u>61.64</u>	15.62	5.76	20.60	28.98

In contrast, CSD uses the same common-support objective for both teachers and achieves the highest five-task averages of 27.71 with Qwen and 28.98 with Phi, exceeding X-Token by 0.67 and 5.38 points, respectively. Across the two teachers, CSD also improves over ULD by 5.26–6.22 points and over ALM by 1.33–1.81 points, while outperforming GOLD in both settings. It further exceeds CPT by 2.54–3.81 points and same-tokenizer KD from Gemma3-4B-PT by 1.17–2.44 points. Across all four 1B teacher–student settings, CSD uses the same objective and achieves the highest Avg.-5 among the evaluated methods.

Support Selection for Non-One-to-One Chunks

Table 3 compares three support policies for non-one-to-one chunks: realized-only support for all such chunks, the proposed shape-dependent support, and full support for all such chunks. Realized-only support retains the observed sequence pair, whereas full support includes all retained admissible final-token pairs, with the observed prefixes fixed. The proposed policy uses full support for one-to-many and many-to-one chunks and realized-only support for many-to-many chunks. All variants include a residual event. For each teacher, the one-to-one construction, divergence, $\beta$, student, and training settings remain unchanged across variants.

Support policy	$1:N$	$M:1$	$M:N$ ($M,N>1$)	Qwen3-4B Avg.-5	Phi-4-mini Avg.-5	Avg. time/step (s) Qwen / Phi
Realized-only for all	Realized-only	Realized-only		33.58	35.22	7.45 / 6.99
CSD (proposed)	Full	Realized-only		33.84	35.84	7.45 / 7.06
Full support for all	Full	Full		33.69	35.55	21.31 / 27.64

CSD achieves the highest Avg.-5 with both teachers, increasing the score from 33.58 to 33.84 with Qwen and Table 3 | Support-selection ablation for non-one-to-one chunks with the Llama-3.2-1B student. Realized-only retains the observed sequence pair; full retains all admissible final-token pairs with fixed observed prefixes. All variants include a residual event. One-to-one support is unchanged across variants. Avg.-5 is the unweighted average of MMLU, WinoGrande, GSM8K, MATH, and MBPP. Bold marks the highest mean for each teacher.

from 35.22 to 35.84 with Phi compared with realized-only support, at nearly unchanged step time. These gains suggest that matching alternative continuations in one-to-many and many-to-one chunks provides useful additional supervision. Extending full support to many-to-many chunks yields lower Avg.-5 scores while requiring 2.86× and 3.92× as much time per step compared with CSD. Thus, expanding the support does not necessarily improve accuracy in the evaluated settings. These results support using full support for one-to-many and many-to-one chunks while retaining only the observed pair for many-to-many chunks, balancing distillation quality and computational cost.

Top-$k$ Robustness and Training Efficiency

Figure 2 compares Avg.-5 and average training-step time for Qwen3-4B→Llama-3.2-1B. We evaluate teacher top-$k$ selection with $k \in\{16, 128, 8192\}$ alongside full-logit CSD and baselines.

With $k = 128$, CSD achieves an Avg.-5 of 33.59, only 0.25 points below the full-logit score of 33.84, while reducing step time from 7.45 to 3.66 seconds, a 2.04× speedup. Further reducing $k$ to 16 gives 33.45 at 3.60 seconds per step. With $k = 8192$, Avg.-5 reaches 33.66, only 0.18 points below full logits, at 4.03 seconds per step. The $k = 16$ and $k = 128$ configurations achieve higher Avg.-5 with lower step time than ULD, GOLD, and X-Token. ALM remains slightly faster, but CSD with $k = 16$ improves Avg.-5 by 1.33 points while requiring only 0.10 additional seconds per step. These results support a favorable accuracy–efficiency trade-off under teacher top-$k$ truncation in this setting.

Scaling to Larger and Instruction-Tuned Students

To evaluate whether the gains extend beyond 1B students, we study both Llama-3.1-8B base and instruction-tuned checkpoints. Given the higher training cost, all 8B runs use a budget of 2,000 steps.

Base student. We distill the Llama-3.1-8B base student from Qwen3-14B-Base and Phi-4-14B. Table 4 shows that CSD improves over both CPT and ALM on all four tasks with either teacher. With Qwen, CSD reaches an Avg.-4 of 56.72, exceeding CPT and ALM by 2.17 and 2.61 points, respectively. With Phi, it reaches 58.98, with corresponding gains of 4.43 and 4.23 points. The largest gains over ALM occur on MATH: 6.66 points with Qwen and 7.42 points with Phi. These gains extend to the larger base student within the evaluated training budget.

Setting	Method	MMLU	GSM8K	MATH	MBPP	Avg.-4
No distillation	Llama-3.1-8B (base)	64.98	54.13	21.16	48.60	47.22
	Continued pretraining	66.19	68.39	32.42	51.20	54.55
Teachers (reference)	Phi-4-14B	80.37	91.96	50.98	72.20	73.88
	Qwen3-14B-Base	80.71	86.73	54.84	74.20	74.12
Cross-tokenizer	Qwen3-14B-Base, ALM	65.91	69.45	29.08	52.00	54.11
	Qwen3-14B-Base, CSD	66.35	71.19	35.74	53.60	56.72
	Phi-4-14B, ALM	66.12	72.48	29.58	50.80	54.75
	Phi-4-14B, CSD	67.74	75.59	37.00	55.60	58.98

Instruction-tuned student. We further evaluate CSD with Llama-3.1-8B-Instruct as the student and Qwen3-14B as the teacher. Table 5 shows that CSD achieves the highest student Avg.-6 of 67.79, outperforming the initial checkpoint, continued training, and ALM by 2.00, 3.90, and 1.60 points, respectively. It exceeds ALM on four of six metrics, including gains of 6.65 and 5.39 points on IFEval-p and IFEval-i. Together, the base and instruction-tuned results show that CSD improves average performance over teacher-free continued training and ALM for 8B students, extending the gains observed at 1B scale to larger students at different training stages.

Setting	Method	GSM8K	MATH	IFEval-p	IFEval-i	MBPP	MBPP+	Avg.-6
No distillation	Llama-3.1-8B-Inst	75.36	46.18	74.49	82.01	64.02	52.65	65.79
	Continued training	72.48	40.72	71.72	78.30	64.20	55.90	63.89
Teacher (reference)	Qwen3-14B	78.62	78.76	84.47	89.57	83.79	72.34	81.26
Cross-tokenizer	Qwen3-14B, ALM	79.76	46.72	70.06	78.06	65.66	56.89	66.19
	Qwen3-14B, CSD	76.50	47.76	76.71	83.45	65.91	56.39	67.79

Conclusion

We introduced CSD, a cross-tokenizer KD method that matches normalized teacher and student distributions over a shared support of common-token and multi-token events, with a residual event capturing the remaining probability mass. Its unified objective applies across chunk shapes and tokenizer pairs without pair-specific loss selection, matching teacher probabilities over alternative events where available.

CSD achieves the highest average score among the evaluated baselines across the four 1B teacher–student settings and outperforms continued training and ALM for both base and instruction-tuned 8B students. For the instruction-tuned student, it also improves all six reported metrics over the initial checkpoint, including instruction-following accuracy. The ablations support using full support for one-to-many and many-to-one chunks and realized-only support for many-to-many chunks, which achieves the highest mean accuracy without the additional cost of expanding many-to-many support. In the Qwen–Llama setting, teacher top-$k$ truncation substantially reduces training-step time with only small changes in accuracy.

Our evaluation focuses primarily on English tasks, and the residual event provides aggregate supervision without distinguishing the teacher’s preferences among outcomes outside the explicit support. Future work will explore more efficient construction of richer common support, particularly for many-to-many chunks, task-specific choices of $\beta$, and generalization across languages and tokenizer families.

Tokenizer and Alignment Statistics

We characterize vocabulary overlap, task-relevant mapping gaps, and aligned-chunk statistics across the four teacher–student pairs with 1B students. Pair labels use teacher→student order, whereas chunk shapes use student:teacher token counts.

Tokenizer Overlap

Table 6 summarizes the common-token map for each pair. The common-pair count, $\lvert\mathcal{M}\rvert$, is the number of retained one-to-one token correspondences used to construct explicit events for one-to-one chunks. Because each token appears in at most one pair, this count also gives the number of covered tokens on each side. Student and teacher coverage divide this count by their respective text-distillation vocabulary sizes.

Teacher $\to$ student	Student vocab. size	Teacher vocab. size	Common pairs	Student covered	Teacher covered
Qwen3-4B $\to$ Llama-3.2-1B	128,256	151,669	108,581	84.66%	71.59%
Phi-4-mini $\to$ Llama-3.2-1B	128,256	200,029	102,149	79.64%	51.07%
Qwen3-4B $\to$ Gemma3-1B-PT	262,144	151,669	94,932	36.21%	62.59%
Phi-4-mini $\to$ Gemma3-1B-PT	262,144	200,029	130,704	49.86%	65.34%

Table 6 | Vocabulary overlap for the evaluated teacher–student pairs. Common pairs are retained one-to-one token correspondences. Student and teacher coverage divide the common-pair count by the respective text-distillation vocabulary size.

Common pairs cover 79.64%–84.66% of the Llama student vocabulary, compared with 36.21%–49.86% for Gemma. These statistics measure vocabulary-entry coverage, not probability mass or aligned-chunk frequency.

Token	Occurrences	Examples
Qwen3-4B $\to$ Llama-3.2-1B
10	1,228	359
20	1,222	334
12	909	256
Phi-4-mini $\to$ Llama-3.2-1B
/week	40	14
/hour	28	17
Subtract	14	13
Qwen3-4B $\to$ Gemma3-1B-PT
/day	55	21
/h	29	18
%\n	24	18
Phi-4-mini $\to$ Gemma3-1B-PT
10	1,228	359
20	1,222	334
12	909	256

Token	Occurrences	Examples
Qwen3-4B $\to$ Llama-3.2-1B
10	35	18
100	7	6
26	5	3
Phi-4-mini $\to$ Llama-3.2-1B
␣heapq	21	9
rlen	11	1
_odd	10	6
Qwen3-4B $\to$ Gemma3-1B-PT
␣\n	908	216
):\n	431	314
\t	314	71
Phi-4-mini $\to$ Gemma3-1B-PT
␣\n	908	216
):\n	433	314
\t	314	71

Unpaired tokens contribute through the residual event in the one-to-one construction, while virtual support for non-one-to-one chunks is constructed separately. Despite differences in vocabulary overlap, CSD uses the same distribution-matching objective for all four pairs.

Task-Relevant Tokens Outside the Common Map

Tables 7 and 8 list the three most frequent entries in each pair’s exported audit example list. Occurrences counts token appearances, while Examples counts distinct examples containing the token. GSM8K references include reasoning and final answers; MBPP references are programs with line endings normalized to LF. Spaces, tabs, and newlines are shown as ␣, \t, and \n, respectively.

GSM8K. The audited examples include numeral tokens for Qwen–Llama and Phi–Gemma, and unit or rate fragments for the other pairs. Losing token-specific supervision for numerals can weaken supervision of the quantities used in intermediate calculations and final answers. The numeral mapping gaps therefore suggest a plausible explanation for GOLD’s low GSM8K and MATH scores in the Qwen–Llama and Phi–Gemma settings (Table 1 and Table 2).

MBPP. For pairs with Gemma students, the audited unmatched tokens include frequent whitespace and code-syntax fragments, whereas the Llama examples include numerals and identifier fragments. The whitespace and syntax fragments encode indentation, line breaks, and block boundaries needed for valid Python programs. Inadequate supervision of these elements can make otherwise correct solutions syntactically invalid. These frequent mapping gaps therefore suggest a plausible explanation for GOLD’s zero MBPP accuracy with both Gemma teacher–student pairs (Table 2): losing token-specific supervision for code structure can undermine programming performance.

Chunk-Shape Distribution

Table 9 reports recorded chunk-shape frequencies. Chunk shapes use student:teacher token counts. One-to-many chunks have shape $1:N$ with $N > 1$, many-to-one chunks have shape $M:1$ with $M > 1$, and many-to-many chunks have shape $M:N$ with $M,N > 1$. Many-to-many chunks need not contain equal numbers of tokens on the two sides.

One-to-one chunks dominate for every pair. Non-one-to-one chunks are nevertheless more frequent with Gemma (9.45–10.25%) than with Llama (1.83–2.95%), indicating greater segmentation differences with the evaluated teachers. One-to-many and many-to-one chunks account for most non-one-to-one chunks; many-to-many chunks constitute at most 1.58% of all recorded chunks.

Teacher → student	1:1	1:N	M:1	M:N	All non-1:1
Qwen3-4B → Llama-3.2-1B	98.17	1.61	0.16	0.06	1.83
Phi-4-mini → Llama-3.2-1B	97.05	0.62	1.70	0.63	2.95
Qwen3-4B → Gemma3-1B-PT	90.55	3.00	4.87	1.58	9.45
Phi-4-mini → Gemma3-1B-PT	89.75	2.49	6.32	1.44	10.25

Common-Support Size Distribution

We measure explicit common-support sizes for one-to-many and many-to-one chunks using the same 500 text samples across all four teacher–student pairs, a context length of 4096, and no top-$k$ truncation. For each recorded chunk, $K$ counts deduplicated explicit events with the observed prefix and multi-token length fixed. The residual event is excluded, so the corresponding event distribution has $K + 1$ entries.

Figure 3 shows empirical cumulative distributions, normalized separately for each pair and chunk shape. Each recorded chunk occurrence has equal weight. The curves include all recorded chunks, including those with no explicit event ($K = 0$), which do not contribute to KD. Chunks with $K = 1$ are accepted and form a binary distribution consisting of one explicit event and the residual.

One-to-many supports are typically small, with median sizes ranging from 3 to 10 across the four pairs. Many-to-one supports vary more substantially: the median sizes are 8 and 6 for the Llama student with Qwen and Phi teachers, respectively, but increase to 1,740 and 175 for the corresponding Gemma pairs. The largest recorded supports also occur in the Gemma many-to-one setting, reaching 5,294 events with Qwen and 4,362 with Phi. Thus, although the same common-support construction applies across pairs, the number of explicit events depends strongly on both the tokenizer pair and the direction of the segmentation mismatch.

Offset-Based Sequence Alignment

Inputs and outputs

Our alignment procedure uses the character spans returned by the student and teacher tokenizers for the same text $x$. Following § 3.1, these spans are $[a_r^S,b_r^S)$ for student tokens and $[a_r^T,b_r^T)$ for teacher tokens. Offsets refer to character positions in the original text, rather than byte positions or lengths of decoded prefixes. We retain the original token indices to associate each chunk with its model predictions.

The procedure returns a set of valid text chunks $\mathcal{C}$. Each chunk records its starting indices $(i,j)$, the indices immediately after it $(i+M,j+N)$, and its shared source interval $[a_c,b_c)$. These records identify the token ranges, their shape $M:N$, and their preceding histories $h_S^{}=\mathbf{s}_{<i}$ and $h_T^{}=\mathbf{t}_{<j}$. Unmatched ranges and candidates that fail verification are marked invalid.

We scan both token sequences in order, maintaining one pointer per sequence:

1. Locate a shared start. A candidate begins when the current student and teacher tokens have the same start offset. Denoting their indices by $i$ and $j$, we set $a_c=a_i^S=a_j^T$. Ranges without corresponding source coverage are marked as orphans, and the scan continues to the next matchable ranges.

2. Initialize the ranges. Include one token from each side, setting $M=N=1$. The current end offsets are $b_{i+M-1}^S$ and $b_{j+N-1}^T$.

3. Extend the earlier-ending range. If $b_{i+M-1}^S<b_{j+N-1}^T$, include the next student token and increment $M$. Otherwise, if $b_{j+N-1}^T<b_{i+M-1}^S$, include the next teacher token and increment $N$. Compare the end offsets after each extension; the side being extended may change as the scan proceeds.

4. Close the candidate. Stop at the first coincident end offset for which both ranges cover the same source interval, and set $b_c=b_{i+M-1}^S=b_{j+N-1}^T$. The index pairs $(i,j)$ and $(i+M,j+N)$ delimit the candidate. If matching coverage cannot be established, mark the unmatched range invalid.

5. Verify the decoded text. Apply the checks below to the candidate token sequences $\mathbf{s}^c$ and $\mathbf{t}^c$. Add the chunk to $\mathcal{C}$ only if verification succeeds.

6. Continue the scan. Resume after the processed ranges. Pointers never move backward, and processed tokens are not reconsidered. Any trailing range without a counterpart is marked as an orphan.

Every candidate passed to verification satisfies

The intervening token sequences are $\mathbf{s}^c$ and $\mathbf{t}^c$, as defined in § 3.1. Their lengths determine whether the chunk is one-to-one, one-to-many, many-to-one, or many-to-many; no shape is prescribed in advance. Extending to a shared end offset accommodates different segmentations without requiring correspondence between internal token positions.

Invalid ranges are excluded from aligned KD, but their tokens remain in the original sequences and in subsequent conditioning histories.

Decoded-text verification

Matching source coverage identifies a candidate but does not automatically validate it. We verify each complete candidate range rather than repeatedly decoding the preceding histories. Each range is decoded with its own tokenizer and normalized using Unicode NFC:

The comparison uses decoded text, not vocabulary token spellings. Decoding the complete range also avoids requiring individual tokens to represent complete text fragments.

A candidate passes verification if any of the following conditions holds:

Algorithm 1 Common-Support Distillation: one training batch

Require: Text batch $\mathcal{X}$; student $S$ and frozen teacher $T$; maps $\mathcal{M},\mathcal{B}$ Require: Divergence $\mathcal{D}$; non-one-to-one weight $\beta\ge0$ Ensure: Joint training loss $\mathcal{L}_{\mathrm{train}}$

1. Tokenize $\mathcal{X}$ with both tokenizers, retaining character offsets

2. Compute full-vocabulary probabilities $p_{S,r}$ and $p_{T,r}$ using the observed token histories

3. $\mathcal{L}_{\mathrm{CE}} \leftarrow$ student next-token cross-entropy on the batch

4. $L_{\Sigma} \leftarrow0$; $n_{\mathrm{acc}} \leftarrow0$

5. for each text $x \in\mathcal{X}$ with token sequences $s,t$ do

6. $\mathcal{C}(x) \leftarrow\operatorname{AlignOffsets}(s,t,\text{offsets})$

7. for each chunk $c=(i,j,M,N) \in\mathcal{C}(x)$ do

8. $s^c \leftarrow s_{i:i+M-1}$; $t^c \leftarrow t_{j:j+N-1}$

9. if $M=N=1$ then ◁ One-to-one chunks

10. $\mathcal{A}_c \leftarrow\mathcal{M}$

11. else if $\min(M,N)=1$ then ◁ One-to-many or many-to-one chunks

12. $\mathcal{A}_c \leftarrow\operatorname{PairedCompletions}(\mathcal{B},s^c_{<M},t^c_{<N},M,N)$

13. else ◁ Many-to-many chunks

14. $\mathcal{A}_c \leftarrow\{(s^c_M,t^c_N)\}$ if $\operatorname{Dec}_S(s^c)=\operatorname{Dec}_T(t^c)$; otherwise $\mathcal{A}_c \leftarrow\varnothing$

15. end if

16. if $\mathcal{A}_c=\varnothing$ then ◁ Skip chunks with no explicit event

17. continue

18. end if

19. Enumerate $\mathcal{A}_c=\{(u_a,v_a)\}_{a=1}^{K_c}$

20. if $M=N=1$ then ◁ Direct next-token probabilities

21. $q_S^{(a)} \leftarrow p_{S,i}(u_a)$; $q_T^{(a)} \leftarrow p_{T,j}(v_a)$, for $a=1,\ldots,K_c$

22. else ◁ Chain-rule probabilities

23. $\alpha_S \leftarrow\prod_{r=1}^{M-1}p_{S,i+r-1}(s^c_r)$; $\alpha_T \leftarrow\prod_{r=1}^{N-1}p_{T,j+r-1}(t^c_r)$

24. $q_S^{(a)} \leftarrow\alpha_S p_{S,i+M-1}(u_a)$; $q_T^{(a)} \leftarrow\alpha_T p_{T,j+N-1}(v_a)$, for $a=1,\ldots,K_c$

25. end if

26. ◁ Collect all remaining mass in a residual event $P_m^c \leftarrow(q_m^c(1),\ldots,q_m^c(K_c),1-\sum_{a=1}^{K_c}q_m^c(a))$, for $m \in\{S,T\}$

27. $w_c \leftarrow1$ if $M=N=1$; otherwise $w_c \leftarrow\beta$

28. $L_{\Sigma} \leftarrow L_{\Sigma}+w_c\mathcal{D}(P_T^c,P_S^c)$; $n_{\mathrm{acc}} \leftarrow n_{\mathrm{acc}}+1$

29. end for

30. end for ◁ Average over accepted chunks, with an unweighted count

31. $\mathcal{L}_{\mathrm{KD}} \leftarrow L_{\Sigma}/\max(1,n_{\mathrm{acc}})$

32. if $\mathcal{L}_{\mathrm{KD}}>0$ then ◁ Dynamically balance KD and cross-entropy

33. $\mathcal{L}_{\mathrm{train}} \leftarrow\operatorname{sg}(\mathcal{L}_{\mathrm{CE}}/\mathcal{L}_{\mathrm{KD}})\mathcal{L}_{\mathrm{KD}}+\mathcal{L}_{\mathrm{CE}}$

34. else

35. $\mathcal{L}_{\mathrm{train}} \leftarrow\mathcal{L}_{\mathrm{CE}}$

36. end if

37. return $\mathcal{L}_{\mathrm{train}}$

• Exact match: $d_S^c=d_T^c$.

• Whitespace-stripped match: the whitespace-stripped forms of $d_S^c$ and $d_T^c$ are identical.

• Pure-whitespace match: both decoded strings consist entirely of whitespace.

These checks accommodate tokenizer-specific whitespace and decoding artifacts. Candidates that fail all three checks are marked invalid.

A valid text chunk therefore requires both matching source coverage and an accepted decoded-text comparison. NFC normalization and whitespace equivalence are used only for verification: they do not modify the original offsets, token ranges, or histories used to compute model probabilities.

Special tokens and the KD interface

BOS, EOS, and padding tokens are matched separately by semantic role, rather than by character spans. EOS and padding may be treated as equivalent when permitted by the tokenizer-role configuration; their token IDs need not coincide across models.

Alignment supplies the valid observed token ranges, their histories, and their chunk shapes. Common-support construction then determines the explicit events and their probabilities, as described in § 3.2. The whitespace allowances used during alignment do not replace the text-matching conditions used to construct that support. A valid aligned chunk with no retained explicit event contributes no KD term, as defined in

Notes. A residual explicitly collects mass outside individually matched events; its absence does not imply unnormalized distributions. X-Token selects P-KL or H-KL through a tokenizer-pair coverage audit. Our evaluated ULD baseline additionally uses the same chunk-merging procedure as GOLD before applying the ULD loss.

Section 3.3.

CSD Training Algorithm

Algorithm 1 summarizes the computation of the CSD training objective for one batch under the default support policy, without teacher top-$k$ truncation. Teacher top-$k$ selection is described separately in § 3.2.3.

Alignment. AlignOffsets follows Appendix § 7: it identifies shared character boundaries and retains chunks that pass the decoded-text checks.

Precomputed support index. We construct vocabulary-conversion tables in both directions. For each ordinary vocabulary token, we first seek a canonicalized token-surface match; otherwise, we decode the token and retokenize the result with the other tokenizer without adding special tokens. Special tokens are matched by semantic role, including EOS. Each direction stores one deterministic conversion per successfully mapped source token.

For a stored sequence $(u_1,\ldots,u_n)$, the index key consists of its length $n$ and token-ID prefix $(u_1,\ldots,u_{n-1})$. The associated entry pairs the source token with $u_n$. For one-to-many chunks, PairedCompletions queries the student-to-teacher index using the observed teacher prefix and token count. For many-to-one chunks, it queries the reverse index using the observed student prefix and token count. Within each bucket, we process source tokens in increasing ID order and retain pairs without repeated token IDs on either side. The indices are built once per process and cached.

Here, full support refers to all retained candidates in the corresponding index bucket, rather than an enumeration of every possible tokenization. Given a constructed prefix key, bucket retrieval takes expected $O(1)$ time. Constructing the key depends on prefix length, while gathering and scoring $K$ candidates requires $O(K)$ work in addition to prefix scoring.

Probability computation and optimization. Empty prefix products equal one, so the same probability computation also applies to one-to-one chunks. The residual contains all mass outside the retained support. The denominator counts accepted chunks without weighting by $\beta$. During backpropagation, sg stops gradients through the entire scaling ratio, while student gradients flow through all retained chain factors and residual probabilities; the teacher remains fixed.

Comparison with Logits-Based Distillation Methods

Table 10 compares CSD with the evaluated logits-based baselines, distinguishing the distribution-matching rule from the construction of supervision for non-one-to-one chunks. The table follows published formulations; experimental adaptations are described separately in § 4.1.

Method	Matching rule	Handling non-one-to-one chunks	Residual event	Pair-specific loss choice
ULD [10]	$L_1$ between sorted, zero-padded probability vectors	No chunk construction; index-wise comparison up to the shorter sequence	No	No
GOLD [2]	KL on exact token-string matches + sorted $L_1$ on unmatched entries	Scales the first-position probability vector by multiplying subsequent realized token probabilities	No	No
X-Token [3]	P-KL: vocabulary projection + KL; H-KL: KL on relaxed token matching + sorted $L_1$ on unmatched entries	Scales the first-position probability vector by multiplying subsequent realized token probabilities	No	Yes
ALM [4]	Binary divergence on the observed event and a residual	Chain-rule observed-only probability	Yes	No
CSD (ours)	Divergence between distributions over explicit paired events and a residual	Chain-rule common-support probabilities	Yes	No

Like ALM, CSD includes a residual event without pair-specific loss-mode selection. The difference lies in the explicit support: ALM matches the observed aligned chunk and its complement, whereas CSD can match multiple alternative events for one-to-one, one-to-many, and many-to-one chunks. Under its default support policy, CSD retains binary support for many-to-many chunks, consisting of the observed sequence pair and the residual.

More Evaluations

Table 11 reports standalone scores for all teachers used in the 1B experiments under the same 3-shot evaluation protocol. Teacher scores are reported separately and are not included in the rankings of student methods.

Both cross-tokenizer teachers achieve higher Avg.-5 than the same-tokenizer teachers, providing an opportunity to transfer knowledge from stronger models outside the student’s family. Qwen has the highest average (64.94) and leads on four tasks, while Phi (60.25) performs best on WinoGrande. This ranking does not carry over directly to distilled students: with CSD, Phi produces higher student averages than Qwen for both Llama and Gemma (Table 1 and Table 2). Standalone teacher accuracy therefore does not by itself determine transfer effectiveness.

Divergence and Chunk Weighting

Table 12 examines the divergence and non-one-to-one chunk weight $\beta$ with the Qwen3-4B teacher and Llama-3.2-1B student. At $\beta= 5$, JSD achieves an Avg.-5 of 33.84, compared with 32.65 for FKL and 33.77 for RKL. JSD has the highest reported average among the three divergences, with performance close to RKL. The common-support construction accommodates all three choices.

Table 12 | Divergence and chunk-weight ablations with Qwen3-4B and Llama-3.2-1B. Divergence comparisons fix $\beta= 5$; weight comparisons use JSD. FKL and RKL denote forward and reverse KL. All other settings are fixed. $^† marks the shared default, reported once. Avg.-5 averages the five displayed tasks. Bold marks the highest reported average.

Teacher	Divergence	$\beta$	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5
Qwen3-4B	JSD	$5$$† | 45.22 | 61.80 | 21.30 | 8.26 | 32.60 | 33.84
Qwen3-4B	FKL	5	44.55	62.59	19.71	8.60	27.80	32.65
Qwen3-4B	RKL	5	45.48	62.43	23.05	7.30	30.60	33.77
Qwen3-4B	JSD	1	45.24	62.35	19.71	7.12	30.00	32.88
Qwen3-4B	JSD	20	45.03	62.51	23.20	8.82	30.00	33.91

With JSD, increasing $\beta$ from 1 to 5 raises Avg.-5 from 32.88 to 33.84. Increasing it further to 20 yields 33.91, a further small increase. These results support upweighting non-one-to-one chunks in this setting, but do not establish a universally optimal weight. We retain $\beta= 5$ as the default and leave it configurable.

Detailed Support-Selection Ablation

Detailed Support-Selection Ablation

Table 13 provides per-task results for the support-selection ablation in § 4.3. We compare three policies for non-one-to-one chunks. Realized-only support retains only the observed sequence pair for every such chunk. The proposed policy uses full support for one-to-many and many-to-one chunks, and realized-only support for many-to-many chunks. Full support for all uses all retained admissible final-token pairs, with the observed prefixes fixed, for every non-one-to-one chunk. The one-to-one construction is unchanged, and all policies include a residual event.

Support policy	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5	Time (s)
Qwen3-4B-Base → Llama-3.2-1B
Realized-only for all	45.16	61.80	21.61	7.94	31.40	33.58	7.45
Proposed	45.22	61.80	21.30	8.26	32.60	33.84	7.45
Full support for all	45.02	61.48	22.59	8.38	31.00	33.69	21.31
Phi-4-mini → Llama-3.2-1B
Realized-only for all	46.55	63.54	24.94	9.08	32.00	35.22	6.99
Proposed	46.47	62.98	26.91	9.22	33.60	35.84	7.06
Full support for all	46.09	64.17	25.40	9.08	33.00	35.55	27.64

The proposed policy achieves the highest Avg.-5 and MBPP score with both teachers. Compared with realized-only support, it raises Avg.-5 from 33.58 to 33.84 with Qwen and from 35.22 to 35.84 with Phi, at nearly unchanged step time. Using full support for many-to-many chunks produces mixed task-level changes. With Qwen, it increases GSM8K and MATH scores relative to the proposed policy, but lowers MBPP from 32.60 to 31.00, resulting in a lower Avg.-5.

Full support for all also requires 2.86$\times$ and 3.92$\times$ as much time per step as the proposed policy with Qwen and Phi, respectively. These results favor using full support for one-to-many and many-to-one chunks while retaining realized-only support for many-to-many chunks in the evaluated settings.

Detailed Teacher Top-$k$ Results

Table 14 provides per-task scores and training-step times for the configurations in § 4.4. Using Qwen3-4B and Llama-3.2-1B, we compare teacher-side truncation at $k = 16$, 128, and 8192 with full logits. All other settings remain fixed.

Teacher support	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5	Time (s)
$k = 16$	44.91	62.75	19.71	8.30	30.60	33.25	3.60
$k = 128$	45.16	62.60	21.15	8.24	30.80	33.59	3.66
$k = 8192$	44.77	62.83	21.61	8.48	30.60	33.66	4.03
Full logits	45.22	61.80	21.30	8.26	32.60	33.84	7.45

All three top-$k$ settings remain within 0.59 average points of full-logit performance while reducing training-step time. At $k = 128$, Avg.-5 is 33.59, only 0.25 points below full logits, and step time falls from 7.45 to 3.66 seconds, a 2.04$\times$ speedup. Increasing $k$ to 8192 raises Avg.-5 to 33.66 and step time to 4.03 seconds. Individual task scores are not monotonic in $k$, but aggregate accuracy remains close to the full-logit result. These observations support teacher top-$k$ truncation as an effective accuracy–efficiency trade-off in this setting.

Timing protocol. The production timing benchmarks use 256 NVIDIA H100 GPUs across 32 nodes, with a per-GPU microbatch size of 1, gradient accumulation of 3, a global batch size of 768, and a sequence length of 4096. Teacher predictions are computed online. Measurements include teacher and student forward passes, loss and support computation, backward propagation, distributed communication, optimizer updates, and data loading and transfer. They exclude offline data preparation and tokenizer-table construction, process and model startup, checkpoint writing, and additional measurement synchronization.