betaFeedback

WorksComputer Science

Label Smoothing in Disguise? Gated Knowledge Distillation for Low-Resource Bangla Generation

Abstract

Gated knowledge distillation (KD) weights teacher signals per token or example, aiming to trust an imperfect teacher only where it seems reliable. We test two gates: EWAD (entropy-weighted agreement-aware distillation), which gates tokens by teacher confidence, and CHAD (counterfactual harm-aware distillation), which gates examples by gradient alignment with validation data. Across six Bangla-centric tasks, three student models (60M to about 2B parameters), and two teacher families, no gate significantly improves plain KD in 45 comparisons. KD also fails to beat cross-entropy (CE) fine-tuning on gold outputs in 40 single-seed comparisons or three-seed runs on four tasks; CE leads on every summarisation metric. The main benefit of KD is regularisation: over ten fixed epochs, CE students lose 0.030 to 0.044 token-F1 from their early-stopped checkpoint, versus 0.005 to 0.025 for distilled students. Teacher-free label smoothing closes most of this gap. In one comparison, a teacher 0.29 token-F1 worse yields students at most 0.009 worse. Thus, gating may change regularisation strength but has little teacher knowledge to select. Findings are limited to Bangla-centric tasks and small students. We will release code, data splits, predictions, and scores.

Introduction

Knowledge distillation (KD) trains a small student model to match the output distribution of a larger teacher (Hinton et al., [15]). Kim and Rush [20] extended it to sequence generation, and it is now a standard way to compress language models ([18] (Sanh et al. 2020; Wang et al. 2020)). The premise is that a distilled student outperforms the same student fine-tuned with cross-entropy (CE) on the gold outputs alone.

For low-resource languages, this premise is harder to rely on (Joshi et al., [19]). Teachers are trained on little data and make more mistakes, and the capacity gap between teacher and student is often large. One response is to gate the KD signal, trusting the teacher more where it seems reliable ([23] (Wen et al. 2023)). For Bangla summarisation, Sumit et al. (2026) proposed two such gates. EWAD (entropy-weighted agreement-aware distillation)1 lowers the weight of the KD loss on tokens where the teacher is not confident or where two teachers disagree. CHAD (counterfactual harm-aware distillation) lowers the weight of training examples whose KD gradient points away from the gradient of the validation loss, which was the case for roughly half of the BanSum training data. Both gates were reported to improve summarisation, with the main results coming from single runs on one task with one student.

We study these two gates for three reasons. They are, to our knowledge, the only gated KD methods designed for and evaluated on low-resource generation. They cover the two main kinds of gate: per token, on teacher confidence or agreement, and per example, on an estimate of training influence. And their positive results make them a fair test of whether gating helps. We do not test other ways of selecting the teacher signal, such as choosing among divergences (Wen et al., 2023) or switching between teacher and student (Koo et al., [23]).

We evaluate both gates on six Bangla-centric tasks with three students and two teacher families, and we measure seed and implementation variation explicitly. The reported gains do not carry over to this setting. In 45 comparisons, no gate significantly beats plain KD (§ 5), and the differences are the size of seed noise (§ 6). Distillation of any kind also fails to beat CE fine-tuning of the same student (§ 7). We argue that KD acts here mainly as a regulariser. It slows overfitting when the student is trained for many epochs, and label smoothing without a teacher reproduces most of that effect (§ 8). The quality of the teacher barely reaches the student (§ 9). A gate applied to such a signal can change the amount of regularisation, but it has little teacher knowledge to select. Summarisation, the task the gates were designed for, shows the same pattern (§ 10), and § 11 discusses how our results relate to the original ones. All tasks are Bangla-centric, the students are small, and the teachers are fine-tuned on the same data as the students. Our claims are restricted to this setting.

Related Work

Distillation and gating. Beyond output-level KD [15, 20], students can be distilled from intermediate layers (Jiao et al., 2020; Sanh et al., 2020), from relations between examples [28], from several teachers (You et al., 2017; Fukuda et al., 2017) or through a teacher assistant [26], including for summarisation (Shleifer and Rush, 2020). These methods weight every example and token equally. Selective methods choose among divergences (Wen et al., 2023), switch between teacher and student guidance by token-level disagreement [23], or gate by teacher confidence and agreement (EWAD) or by gradient alignment with validation data (CHAD; Sumit et al., 2026). CHAD’s score is an estimate of training-data influence [22, 31]. All of these methods assume that the KD signal carries example- or token-specific knowledge worth selecting. This is the assumption we test.

KD for low-resource languages. Distillation is widely used to build smaller models for low-resource languages, for example in machine translation together with transfer learning [7], in multilingual sentence encoders and bitext mining [35, 14], and in compact language-specific models for cross-lingual transfer [2]. Our question is narrower: whether, for a pretrained student fine-tuned on a few thousand examples, the teacher signal adds anything beyond the gold labels, and whether gating it helps.

KD as regularisation. Several studies argue that much of KD’s benefit is regularisation. Label smoothing and KD have related effects on representations [27], and confidence penalties act as output regularisers [29]. A poorly trained teacher, or even a hand-designed smooth distribution, gives most of KD’s gain (Yuan et al., 2020). Self-distillation behaves like instance-specific label smoothing (Zhang and Sabuncu, 2020; Tang et al., 2020), and students often do not match their teachers closely even when they generalise well (Stanton et al., 2021). These studies are mostly on image classification. We test the question for low-resource generation, with a weak-teacher comparison and a direct label-smoothing control.

Variance. Reported NLP gains often fall within the variation caused by seeds, data order or implementation choices [34, 9, 10, 5]. We measure these sources for our own pipeline and compare every effect against them.

Methods Under Test

Notation. Let xx be an input with gold output y=(y1,…,yL)y = (y_1,\ldots,y_L). At position tt the student θ\theta predicts pθS(⋅∣y<t,x)p_\theta^S(\cdot\mid y_{<t},x) and the teacher pT(⋅∣y<t,x)p^T(\cdot\mid y_{<t},x). The CE and KD losses at tt are

ℓtCE=−log⁡pθS(yt∣y<t,x),(1)\ell_t^{\mathrm{CE}} = -\log p_\theta^S(y_t \mid y_{<t},x), \tag*{(1)}
ℓtKD=τ2KL⁡(pτT∥pθ,τS),(2)\ell_t^{\mathrm{KD}} = \tau^2 \operatorname{KL}(p_\tau^T \parallel p_{\theta,\tau}^S), \tag*{(2)}

with temperature τ\tau. The BanglaT5 student uses its teacher’s full distribution. The two large language model (LLM) students (§ 4.2) read the teacher’s top-20 log-probabilities from a cache; the KL is computed exactly over those tokens, and the remaining mass is treated as one tail bucket. All methods share the student, data, optimiser, schedule and decoding. They differ only in how the KD loss is weighted (Table 1).

methodweight on the KD lossused in
CEnoneall students
plain KDconstant λ\lambdaall students
EWAD-confper token, teacher confidence5 tasks
EWAD-agreeper token, teacher agreementsumm.
+ CPDPadds a second-teacher termall students
CHADper example, gradient alignment5 tasks
entropy, semantic gateper example, heuristicBanglaT5
agreement-harm gateper token, agreementsumm.

Table 1. Methods compared. EWAD, CPDP (capacity-proportional divergence preservation) and CHAD follow Sumit et al. (2026). The entropy and semantic gates are cheaper variants of CHAD’s gate, and the agreement-harm gate is a forward-only proxy of CHAD used for summarisation. Details and diagrams are in Appendix § 14.

CE and plain KD. CE trains on LCE=∑tℓtCE\mathcal{L}_{\mathrm{CE}} = \sum_t \ell_t^{\mathrm{CE}} and never sees a teacher. Plain KD adds the KD loss with weight λ=0.5\lambda= 0.5 at τ=2\tau= 2, LKD=∑t(ℓtCE+λℓtKD)\mathcal{L}_{\mathrm{KD}} = \sum_t(\ell_t^{\mathrm{CE}} + \lambda\ell_t^{\mathrm{KD}}). Because of the factor τ2\tau^2, the KD term carries twice the weight of the CE term.

EWAD-conf. This gate trusts the teacher only where it is confident. With the teacher’s top-1 probability pmax⁡,tTp_{\max,t}^T,

gt=σ(k(pmax⁡,tT−δ)),αt=gt(1−η),(3)g_t = \sigma\left(k\left(p_{\max,t}^T-\delta\right)\right), \qquad\alpha_t = g_t(1-\eta), \tag*{(3)}
LEWAD=∑t((1−αt)ℓtCE+αtℓtKD).(4)\mathcal{L}_{\mathrm{EWAD}} = \sum_t\left((1-\alpha_t)\ell_t^{\mathrm{CE}}+\alpha_t\ell_t^{\mathrm{KD}}\right). \tag*{(4)}
methodweight on the KD lossused in
CEnoneall students
plain KDconstant λ\lambdaall students
EWAD-confper token, teacher confidence5 tasks
EWAD-agreeper token, teacher agreementsumm.
+ CPDPadds a second-teacher termall students
CHADper example, gradient alignment5 tasks
entropy, semantic gateper example, heuristicBanglaT5
agreement-harm gateper token, agreementsumm.

where k=10k = 10 sets the sharpness of the gate and δ\delta is the confidence at which the gate equals 0.5. The floor η=0.3\eta= 0.3 keeps a CE weight of at least 0.3 at every token, and the KL uses τ=1\tau= 1. Unlike plain KD, EWAD mixes the two losses instead of adding them. For BanglaT5, δ=0.5\delta= 0.5. For the LLM students, whose top-20 distributions are renormalised, δ\delta is the teacher’s median top-1 probability on the training set.

CPDP. CPDP adds a second teacher T2T_2. It asks the student to keep its distances to the two teachers consistent with the distance between the teachers. With mean-pooled hidden states projected to a shared space and cosine distance dd, the added loss is β(dST1−dST2−dT1T2)2\beta(d_{ST_1} - d_{ST_2} - d_{T_1T_2})^2 with β=0.05\beta= 0.05. For BanglaT5, the states are encoder outputs and T2T_2 is an mT5 model (Xue et al. (2021)). For the LLM students, they are prompt states and T2T_2 is the smaller teacher of the same family.

CHAD. CHAD gives more KD weight to training examples whose KD signal helps on validation data. Each example is scored by

s(x)=1∣C∣∑c∈Ccos⁡(∇Lval(θc),∇ℓKD(x;θc))(5)s(x) = \frac{1}{\lvert C\rvert}\sum_{c\in C}\cos\left(\nabla\mathcal{L}_{\mathrm{val}}(\theta_c), \nabla\ell^{\mathrm{KD}}(x;\theta_c)\right) \tag*{(5)}

where ℓKD(x)=∑tℓtKD\ell^{\mathrm{KD}}(x) = \sum_t \ell_t^{\mathrm{KD}}, gradients are with respect to the student’s parameters, and CC is a set of student checkpoints, as in TracIn (Pruthi et al. [31]). Scores are mapped to weights h(x)∈[0,1]h(x) \in[0,1] by their rank and renormalised within each batch BB, so the gate redistributes the KD weight between examples without reducing it:

LCHAD=LCE+λ∑x∈Bh(x)ℓKD(x)∑x∈Bh(x)(6)\mathcal{L}_{\mathrm{CHAD}} = \mathcal{L}_{\mathrm{CE}} + \lambda\frac{\sum_{x\in B}h(x)\ell^{\mathrm{KD}}(x)}{\sum_{x\in B}h(x)} \tag*{(6)}

For the LLM students, only the LoRA parameters receive gradients, so s(x)s(x) is computed exactly for every example. For BanglaT5, it is computed on a probe of 1,500 examples, and a gradient-boosting regressor on 13 cheap features extends it to the rest. Two cheaper variants, the entropy gate and the semantic gate, use only teacher-confidence or teacher–gold-overlap features. The scores are weakly determined. The regressor’s held-out R2R^2 is at most 0.19, and for the LLM students the weights from two seeds correlate at only 0.50 to 0.81 (Appendix § 20).

Summarisation variants. The summarisation pipeline uses two teachers in every KD term and their top-50 log-probabilities. EWAD-agree gates each token on how much the two teachers agree, not on confidence. EWAD-agree+CPDP-κl adds a CPDP term on output divergences. The agreement-harm gate weights the KD term by teacher agreement times the teachers’ probability of the gold token, a forward-only stand-in for CHAD. As a control, we also run EWAD-agree with its gate fixed at its average value (0.666). Formulas are in Appendix § 14.

Experimental Setup

Tasks and data

We use six Bangla-centric tasks (statistics in Appendix § 15). Extractive question answering (qa) uses 5,000 question–answer pairs over 316 passages from the two releases of UDDIPOK (Aurpa et al. [3]), split by passage. Bangla-to-English translation (mt) uses 5,000 sentence pairs sampled from the Bangla–English part of Samanantar (Ramesh et al. [33]). Paraphrase generation (pp) uses 5,000 pairs from Bangla Paraphrase (Akil et al. [1]), without identical pairs. Conversational reply (cc) uses all 574 unique question–answer pairs of a public Bangla chat dataset (Spedrox Sac (2024)). Sentence-type classification (bt) uses the 3,775 unique sentences of BTSD (Das et al. [8]), with the label generated as one word (majority class 41%). News summarisation (summ) uses 5,000 BanSum articles of at most 1,000 tokens (Hasan et al. [13]). For the first five tasks, we remove duplicates after text normalisation and make our own 70/10/20 split with seed 42; summ is split 80/10/10. With 115 test examples, cc cannot detect effects much below 0.025 token-F1, so we report it but do not rely on it.

Models

Students. All students start from pretrained weights, and a separate student is trained for each task and method. We use BanglaT5-small (60M parameters, full fine-tuning; Bhattacharjee et al., [4]), gemma-4-E2B (Gemma Team, Google DeepMind, [12]) and Qwen3-1.7B (Qwen Team, [32]). We call the last two the LLM students. They are trained with low-rank adapters (LoRA; Hu et al., [17]) at rank 16 on the five tasks and with 4-bit quantised LoRA (QLoRA) at rank 32 on summ.

Teachers. BanglaT5 is distilled from BanglaT5-base (247M), fully fine-tuned on each task. Each LLM student has a teacher of its own family: gemma-4-31B-it for gemma and Qwen3-32B for Qwen. These teachers are fine-tuned with 4-bit LoRA on the same data, and their outputs are cached once with teacher forcing. The smaller gemma-4-12B and Qwen3-14B serve only as the second CPDP teacher on the five tasks, not as an ensemble. In summ, both gemma teachers enter every KD term. Every teacher had to pass a validation check before distillation (Appendix § 24). The teacher beats its CE-trained student in 13 of the 15 student–task pairs, by up to 0.16 token-F1. The exceptions are gemma on cc and bt, where the teacher is 0.009 and 0.016 behind (Table 6). A weak teacher therefore cannot explain the results in 13 of the 15 pairs.

taskBanglaT5 TBanglaT5 CEBanglaT5 KDgemma Tgemma CEgemma KDQwen TQwen CEQwen KD
qa0.9700.9530.9490.9810.9530.9530.9340.9180.902
mt0.4970.3940.3540.5520.5160.5140.5350.4370.438
pp0.5430.4720.4680.5390.5140.5200.5000.4430.447
cc0.6360.4790.4710.708*0.7170.7280.7230.6720.684
bt0.9550.9500.9510.946*0.9620.9560.9630.9470.959

Table 6. Test token-F1 (accuracy for bt) of the teacher (T), the CE-trained student (CE) and the plain-KD student (KD). *Teacher below the CE student. Single seed; the gemma mt CE value is seed 42 of the seeded study.

Experiments

We use method for a training objective (CE, plain KD or a gate) and comparison for the difference between two methods on the same student and task.

Grid. Each student is distilled on the five tasks with plain KD and three gates. The LLM students use EWAD-conf, EWAD-conf+CPDP and CHAD; BanglaT5 uses CHAD and the two heuristic gates. This gives 45 single-seed comparisons of a gated with a plain-KD student. EWAD variants were also trained for BanglaT5, in a separate implementation that has no plain-KD student (§ 6, Appendix § 17).

CE baselines. Every LLM student also has a CE student on every task, trained with the grid recipe and seed 42.

Seeded study. The gemma student is trained with seeds 42, 7 and 13 for exactly ten epochs, with early stopping disabled and an adapter saved after every epoch. From each run we evaluate two checkpoints: the one that early stopping with patience 3 would have kept, found by replaying the rule on the run’s own validation log, and the final one. Both therefore come from the same run. On mt we train CE, plain KD, EWAD-conf, EWAD-conf+CPDP-d, CHAD, and CE with label smoothing (LS) at ε∈{0.05,0.1,0.2}\varepsilon\in\{0.05, 0.1, 0.2\}. On pp, bt and cc we train CE, plain KD, EWAD-conf and CHAD. qa, where all students are near the ceiling, was not seeded.

Summarisation. The gemma student is trained for one epoch in each of the six summarisation configurations with three seeds. The gold target tokens are identical across all 18 runs.

Evaluation

One script scores every prediction file. On the five tasks, the primary metric is token-F1 after Bangla-aware normalisation, and we also report exact match. On summ, the primary metric is ROUGE-L (Lin, [25]) on whitespace tokens. We also compute sacreBLEU (Post, [30]), BERTScore (Zhang et al., 2020) with xlmr-roberta-base, and the cosine similarity of BGE-M3 embeddings (Chen et al., [6]), which we call semsim. Single-seed comparisons use a paired bootstrap (Koehn, [21]) and a permutation test, with 10,000 resamples each, and McNemar’s test on exact match. Multi-seed comparisons use Welch’s tt-test on seed means. We apply Holm’s correction (Holm, [16]) within each family of comparisons. We test equivalence with two one-sided tests (TOST; Lakens, [24]) at ±\pm0.032 token-F1, the largest difference we measured between two correct CE implementations (§ 6), and at a stricter ±\pm0.020.

Gated KD Does Not Beat Plain KD

Of the 45 comparisons of a gated with a plain-KD student, one is significant after correction. It is a loss: CHAD on pp with the gemma student, −0.013-0.013 token-F1 (pHolm=0.024p_{\mathrm{Holm}} = 0.024), still inside the equivalence margin. No comparison shows a significant gain. Of 45, 42 are equivalent to plain KD within ±\pm0.032 and 39 within ±\pm0.020 (Appendix § 16). The largest positive effect is CHAD on qa with the Qwen student (+0.019+0.019, permutation p=0.003p = 0.003). It does not survive correction (pHolm=0.093p_{\mathrm{Holm}} = 0.093), and § 7 shows that it reflects a weakness of plain KD. The picture does not depend on the metric. On BERTScore, one comparison is significant and none is a gain. On semsim, four are significant, two gains (+0.008, +0.007) and two losses, and all 45 are equivalent on both metrics.

Seed and Implementation Variation

Seed variation. In the seeded study, the pooled within-method standard deviation of token-F1 at the early-stopped checkpoint is 0.0063 on mt, 0.0030 on pp, 0.0068 on bt and 0.0088 on cc (Table 3). Against the mt value, 34 of the 45 grid effects (76%) lie within one standard deviation and 38 (84%) within two (Figure 1). For pure noise of that spread, the expected shares are 68% and 95%. For the gemma student, on which the spread was measured, the counts are 11 and 14 of 15. Three seeds estimate the spread only roughly, so these counts are descriptive. Still, the grid effects are of the size that reseeding alone produces.

Figure 1. The 45 gated-minus-plain-KD token-F1 effects with 95% bootstrap intervals, against one and two seed standard deviations measured on mt (σ=0.0063\sigma= 0.0063, shaded bands).

taskmethodF1 at early stopΔ\Delta vs CEpHolmp_{\mathrm{Holm}}F1 at epoch 10degradation
mt (n=1000n=1000, σ=0.0063\sigma=0.0063)CE0.5216 ± 0.0048––0.4774−0.0442 ± 0.0082
plain KD0.5114 ± 0.0009−0.01020.4300.5013−0.0101 ± 0.0035
EWAD-conf0.5213 ± 0.0109−0.00031.0000.4996−0.0217 ± 0.0089
EWAD-conf+CPDP-d0.5220 ± 0.0067+0.00041.0000.4969−0.0251 ± 0.0062
CHAD0.5135 ± 0.0038−0.00810.5050.5016−0.0120 ± 0.0049
pp (n=1000n=1000, σ=0.0030\sigma=0.0030)CE0.5147 ± 0.0018––0.4843−0.0304 ± 0.0013
plain KD0.5101 ± 0.0033−0.00450.3870.5055−0.0047 ± 0.0021
EWAD-conf0.5162 ± 0.0042+0.00150.6120.5085−0.0077 ± 0.0056
CHAD0.5118 ± 0.0020−0.00290.3870.5032−0.0086 ± 0.0053
bt (n=755n=755, σ=0.0068\sigma=0.0068)CE0.9558 ± 0.0043––0.9558+0.0000 ± 0.0066
plain KD0.9479 ± 0.0073−0.00790.3910.9563+0.0084 ± 0.0061
EWAD-conf0.9501 ± 0.0099−0.00570.4320.9554+0.0053 ± 0.0093
CHAD0.9497 ± 0.0035−0.00620.3800.9572+0.0075 ± 0.0028
cc (n=115n=115, σ=0.0088\sigma=0.0088)CE0.7177 ± 0.0126––0.7184+0.0007 ± 0.0114
plain KD0.7115 ± 0.0106−0.00621.0000.7223+0.0108 ± 0.0092
EWAD-conf0.7241 ± 0.0054+0.00641.0000.7253+0.0012 ± 0.0106
CHAD0.7194 ± 0.0029+0.00171.0000.7158−0.0037 ± 0.0018

Table 3. Seeded study (gemma student, seeds 42/7/13), test token-F1 as mean ± standard deviation over seeds. “Early stop” is the checkpoint that patience-3 early stopping keeps, “epoch 10” the final checkpoint of the same run, and degradation their difference. Δ\Delta vs CE and pHolmp_{\mathrm{Holm}} are for the early-stopped checkpoint (Welch’s test on seed means). σ\sigma is the pooled within-method standard deviation. Per-seed values are in Appendix § 18.

Implementation variation. The BanglaT5 student has two CE baselines, trained with the recipes of the two codebases in which the gates were originally built (learning rate 5×10−55 \times10^{-5}, batch 16, up to 15 epochs; and 3×10−43 \times10^{-4}, batch 32, up to 6 epochs). On identical data and scoring, they differ significantly on two of five tasks, in opposite directions. On mt the difference is +0.032 (pHolm<0.001p_{\mathrm{Holm}} < 0.001); one recipe produces 47 empty outputs in 1,000, and without them the gap is +0.020. On qa it is −0.013 (pHolm=0.008p_{\mathrm{Holm}} = 0.008). We scored the ten students trained in the EWAD codebase against each CE baseline in turn. For eight of the ten, switching the baseline changes the sign of the effect, moves it across the ±0.032 margin or changes its significance (Appendix § 17). The choice between two reasonable baselines is thus enough to change most of these conclusions, so we report both wherever two

exist.

Software environment. Re-running the mt seed-42 students on the software stack of the seeded study moves token-F1 by 0.001 to 0.004 (0.014 for CHAD, whose scores are recomputed). The same CE recipe on two different GPUs gives 0.514 and 0.513 on pp and 0.962 and 0.959 on bt.

Distillation Does Not Beat Fine-Tuning

The grid compares gated with plain KD. A more basic question is whether distillation helps at all. Table 2 compares every distilled LLM student with a CE student trained with the same recipe. None of the 40 differences is significant after correction, and all lie within ±0.020 token-F1. Before correction, the confidence interval excludes zero in ten cases, seven of which favour CE. In the seeded study, no distilled method differs significantly from CE at the early-stopped checkpoint on any of mt, pp, bt or cc (smallest pHolm=0.38p_{\mathrm{Holm}} = 0.38; Table 3).

studenttaskCE F1plain KDEWAD-conf+CPDPCHAD
gemmaqa0.9532+0.0002+0.0046+0.0002+0.0031
gemmamt0.5161−0.0023−0.0054+0.0019−0.0124†
gemmapp0.5144+0.0057+0.0009+0.0010−0.0075†
gemmacc0.7168+0.0116+0.0164+0.0132+0.0091
gemmabt0.9616−0.0053−0.0146†−0.0159†−0.0026
Qwenqa0.9181−0.0162†−0.0197†−0.0187†+0.0030
Qwenmt0.4370+0.0012+0.0032+0.0020−0.0010
Qwenpp0.4431+0.0041+0.0017+0.0030+0.0018
Qwencc0.6718+0.0127−0.0096−0.0013−0.0074
Qwenbt0.9470+0.0119†+0.0079+0.0132†+0.0132†

Table 2. Distilled minus CE student, token-F1, single seed (42). None of the 40 differences is significant after Holm’s correction within each student. †Bootstrap 95% interval excludes zero before correction.

These baselines also explain the largest gated effect in § 5. For Qwen on qa, CE scores 0.918, plain KD 0.902 and CHAD 0.921. CHAD’s advantage of 0.019 over plain KD is almost exactly what plain KD lost relative to CE (0.016), and CHAD does not differ from CE (+0.003, pHolm=1.0p_{\mathrm{Holm}} = 1.0). The gate recovers a loss caused by distillation; it does not improve on CE. Plain KD is likewise below CE in four of the five BanglaT5 student–task pairs (Table 6).

Distillation Acts as a Regulariser

Although KD does not improve on CE at the early-stopped checkpoint, it does change how training proceeds. To measure this, the seeded study compares the early-stopped checkpoint of each run with its checkpoint after a fixed ten epochs (Table 3, Figure 2).

Figure 2. Change in test token-F1 from the early-stopped checkpoint to epoch ten, per seed (points) and on average (bars), on mt and pp. CE degrades most; the distilled methods and teacher-free label smoothing (LS, mt only) degrade much less.

Under a fixed budget, CE overfits most. Training past the early-stopping point hurts every method on mt, but it hurts CE most. CE loses 0.044 ± 0.008 token-F1 by epoch ten, while the distilled methods lose between 0.010 (plain KD) and 0.025 (EWAD-conf+CPDP-d). At epoch ten, every distilled method is therefore 0.020 to 0.024 ahead of CE (pHolmp_{\mathrm{Holm}} = 0.019). pp shows the same pattern with less noise. CE loses 0.030 ± 0.001 and the distilled methods 0.005 to 0.009, which leaves them 0.019 to 0.024 ahead at epoch ten (pHolmp_{\mathrm{Holm}} = 0.004). The direct test of whether CE degrades more than plain KD is significant on pp (pHolmp_{\mathrm{Holm}} < 0.001) but not on mt after correction (pHolmp_{\mathrm{Holm}} = 0.067). The gated methods do not degrade differently from plain KD (Appendix § 19). CE’s validation score peaks in the first epoch in every seed on both tasks, whereas on pp plain KD and EWAD-conf keep improving until the fifth to tenth epoch. On cc and bt, no method degrades: with 402 training examples (cc) or a three-way label (bt), ten epochs is not a long budget.

Label smoothing reproduces most of the effect. If the teacher’s distribution acts mainly as a regulariser, smoothing the targets without a teacher should give similar protection. On mt, we trained CE with label smoothing ([27]) at ε∈{0.05,0.1,0.2}\varepsilon\in\{0.05, 0.1, 0.2\} under the same budget. These students degrade by 0.018, 0.017 and 0.009, against 0.044 for CE and 0.010 for plain

KD. Label smoothing therefore removes a share f=(∣dCE∣−∣dLS∣)/(∣dCE∣−∣dKD∣)f = (|d_{\mathrm{CE}}| - |d_{\mathrm{LS}}|)/(|d_{\mathrm{CE}}| - |d_{\mathrm{KD}}|) of 0.77, 0.80 and 1.03 of the CE-to-KD gap, computed on seed means (0.40 to 1.58 per seed; Appendix § 19). At ε=0.2\varepsilon= 0.2, the degradation is indistinguishable from that of plain KD (pHolmp_{\mathrm{Holm}} = 1.0). At the early-stopped checkpoint, label smoothing does not help either: it is 0.006 to 0.011 below CE, which is not significant. A single smoothing constant, with no teacher, thus provides most or all of the protection that distillation provides here. This also addresses the objection that parity with CE is an artefact of early stopping. With longer training, KD pulls ahead only because CE degrades, and every method is then worse than at its own early-stopped checkpoint.

Teacher Quality Barely Reaches the Student

If KD mainly regularises, a much worse teacher should barely change the student. We test this in one setting, BanglaT5 on qa with a single seed. The pipeline was run twice, once with an under-trained BanglaT5-base teacher and once with a converged one. The under-trained teacher comes from an earlier run whose teacher training stopped too soon (ten epochs at learning rate 5×10−55 \times10^{-5}, still improving). The converged teacher was trained at learning rate 10−410^{-4} for up to 40 epochs. They reach 0.645 and 0.960 test exact match. Data, split, student recipe and CE student are identical, and CHAD’s scores are recomputed for each teacher. Because the CE student uses no teacher, this gives a difference-in-differences comparison. The weaker teacher is 0.291 token-F1 worse [−0.320, −0.262], but its students are only 0.002 (entropy gate) to 0.009 (semantic gate) worse, an attenuation of about 31 to 1. Only the semantic-gate difference survives correction on token-F1 (pHolm=0.034p_{\mathrm{Holm}} = 0.034), and none does on BERTScore or semsim (Appendix § 21). This is the pattern the regularisation account predicts. Because it rests on one task, one student and one seed, we treat it as supporting evidence, not as a test on its own.

Summarisation

The gates were designed for summarisation, where outputs are long and teacher distributions are less peaked, so this is where they should help most.

No configuration beats CE. CE has the highest ROUGE-L (0.279 ±\pm 0.002), BERTScore and semsim (Table 4). Every distilled configuration is below CE on ROUGE-L. The difference is significant after correction for plain KD (−0.019), the agreement-harm gate (−0.011) and the constant gate (−0.033), and not significant for EWAD-agree (−0.006) or EWAD-agree+CPDP-kl (−0.017). EWAD-agree does beat plain KD (+0.013, pHolm=0.027p_{\mathrm{Holm}} = 0.027). Measured against plain KD, the gate therefore shows a gain; measured against CE, it does not.

methodROUGE-LΔ\Delta vs CEpHolmp_{\mathrm{Holm}}BERTScoresemsimwords
CE0.2787 ±\pm 0.0023––0.89330.848769.2
EWAD-agree0.2731 ±\pm 0.0020−0.00560.0950.89220.848475.8
agreement-harm gate0.2679 ±\pm 0.0031−0.01080.0400.89190.844266.4
EWAD-agree+CPDP-kl0.2621 ±\pm 0.0062−0.01660.0950.89290.843260.2
plain KD0.2602 ±\pm 0.0030−0.01850.0080.89200.840656.4
EWAD-agree, constant gate0.2458 ±\pm 0.0033−0.03290.0020.88700.835061.3

Table 4. Summarisation (gemma student, seeds 42/7/13), ROUGE-L as mean ±\pm sd; BERTScore and semsim are seed means, and “words” the mean output length (references: 73.1). Δ\Delta and pHolmp_{\mathrm{Holm}} are against CE (Welch’s test on seed means, Holm over eight contrasts). All contrasts on all metrics are in Appendix § 22.

The learned gate beats a constant gate. Fixing EWAD-agree’s gate at its average value keeps the average KD weight but removes the variation between tokens. The learned gate beats this constant by 0.027 ROUGE-L (pHolm=0.005p_{\mathrm{Holm}} = 0.005). At that average, both put about eight times more weight on the KD term than on CE, compared with two times for plain KD. Spread uniformly, this much KD gives the worst configuration. The learned gate concentrates it on tokens where the teachers agree and returns the rest of the weight to CE. This brings the student back towards CE, but not past it.

Length and metrics. CE’s summaries (69.2 words) are close to the references in length (73.1); EWAD-agree writes 75.8 words and plain KD 56.4. Truncating every output to 47 words shrinks all ROUGE-L gaps without changing any sign (EWAD-agree minus CE moves from −0.006 to −0.004). ROUGE-L, BERTScore and semsim all rank CE first and the constant gate last (Spearman ρ=1.0\rho= 1.0 between ROUGE-L and semsim, 0.71 between BERTScore and either). On the five tasks, all three metrics agree on the sign of 10 of the 12 seeded CE-versus-KD differences, and the other two are smaller than the seed standard deviation (Appendix § 23).

Relation to the Original Results

Sumit et al. (2026) report that CHAD improves ROUGE-L by 0.018 and EWAD+CPDP by 0.022 over a CE student of the same architecture on BanSum. Our study does not repeat that setting. They distil BanglaT5-small (60M) from BanglaT5 (247M) on about 141,000 BanSum articles, roughly 28 times our summarisation training set. Our summarisation student is a 2B model, distilled from 31B and 12B teachers through their top 50 tokens, and our BanglaT5 students see at most 5,000 examples of other tasks. Our results therefore do not show that their gains are wrong in their setting. They show that the gains do not carry over to ours, and they identify what a test in their setting would need.

Three of our controls are relevant. Seeds: gains of 0.018 and 0.022 are at least three times the summarisation seed standard deviation we measure (0.002 to 0.006), so seed noise alone is an unlikely explanation. However, their main configurations were run once, and their multi-seed check covered a 20,000-example subset, not those configurations. Baseline: two correct CE recipes for our BanglaT5 student differ by up to 0.032 token-F1 and change the conclusion for eight of ten students (§ 6). A gain over a single CE run therefore depends on that run’s recipe. Constant gate: in our summarisation experiments, the learned gate’s advantage comes from where it places the KD weight, and it recovers towards CE without passing it (§ 10). A gain over CE needs this control before it can be attributed to the gate. Checkpoint selection is comparable in the two studies (300 validation articles there, 150 or 200 here) and does not explain the difference.

Our configuration closest to theirs is BanglaT5 with CPDP. It gives our largest single-seed gain: +0.048 token-F1 on mt over the CE student of its own codebase (pHolm<0.001p_{\mathrm{Holm}} < 0.001). That CE student is the one with 47 empty outputs. Against the other CE student, the gain is +0.016 [+0.007, +0.026], inside the stricter equivalence margin. Gating may matter more with far more data, a smaller capacity gap or full teacher distributions. A multi-seed test against both baselines and a constant gate in that setting is the natural next step.

Discussion and Conclusion

Taken together, the results are consistent with one explanation. In this setting, the quality of the teacher barely reaches the student (§ 9, one task and one seed). What distillation provides is protection against overfitting, and this protection does not need a teacher: one label-smoothing constant reproduces most of it (§ 8). A gate applied to such a signal can change how much regularisation is applied, but it has little teacher knowledge to select. This explanation predicts parity with plain KD (§ 5) at the size of seed noise (§ 6), no advantage over CE at the early-stopped checkpoint (§ 7), and a learned gate whose measurable benefit is to repair damage that distillation itself causes (§ 10).

For low-resource generation at this scale, CE fine-tuning with early stopping is a strong baseline that no form of distillation beat in our experiments. A claim that a gate improves distillation should be supported by four controls: a comparison with a CE student, not only with plain KD; a constant gate of the same average weight; at least three seeds; and, where two reasonable baseline implementations exist, both of them. Each of these controls changed a conclusion in this study. We do not claim that distillation is useless in general. For Bangla-centric tasks, small students and teachers fine-tuned on the same data, its effect is mainly a regularisation that is available more cheaply.

Limitations

Seeds. The 45-comparison grid and the CE baselines of the LLM students use a single seed. The seeded study uses three seeds for the gemma student on four of the five tasks (not qa), and the summarisation study uses three seeds. Three seeds estimate the seed variation only roughly, and Holm’s correction is strict at this sample size; we therefore report uncorrected and corrected pp-values and give every per-seed value in Appendix § 18. The cc test set (n=115n = 115) is too small to detect effects below about 0.025 token-F1.

Scope of the regularisation evidence. The fixed-budget effect appears on mt and pp; on cc and bt no method overfits within ten epochs, so those tasks neither support nor contradict it. The label-smoothing control was run on mt only.

Weak-teacher comparison. The teacher-quality comparison (§ 9) comes from one task, one student and one seed, and the weak teacher was under-trained rather than deliberately corrupted. Because CHAD’s scores and gates were recomputed for each teacher, the gated students differ in more than the teacher.

Evaluation. All results use automatic metrics; we did not run a human or LLM-judge evaluation. A human study limited to summarisation could not support claims across the six tasks, and on every task the lexical, contextual and embedding-based metrics we report agree on the direction of the effects wherever there is an effect larger than the seed variation (§ 10, Appendix § 23). Whether human readers would prefer CE or distilled summaries remains open.

Generalisation. All tasks are Bangla-centric, students have 60M to about 2B parameters, and teachers are fine-tuned on the same data as the students. The LLM students see only the top 20 (five tasks) or top 50 (summarisation) teacher tokens. We do not reproduce the original BanSum setting of Sumit et al. (2026), which uses far more data and a smaller capacity gap (§ 11). Settings with much larger capacity gaps, out-of-domain teachers, much more training data or full teacher distributions may behave differently; the weak-teacher and label-smoothing designs used here can be reused to test them. We test two gate families and do not claim that other ways of selecting the teacher signal would fail.

Ethics Statement

All datasets are publicly available and were used in accordance with their licences; we collected no new data and recruited no human participants. Bangla news and conversation data may contain social biases that generated text can reproduce, and deployed systems should include human review. Experiments ran on single consumer GPUs. Code, data splits, per-example predictions and scores, and pinned environment specifications will be released publicly; the data splits keep the licences of the original datasets. AI assistants were used for coding and writing support.

Method Details

Figure 3 and Figure 4 show EWAD-conf, CPDP and CHAD as used on the five tasks.

EWAD-conf and CPDP diagrams. EWAD-conf: teacher top-1 $p^{T}_{\max,t}$ leads to $g_t=\sigma(k(p^{T}_{\max,t}-\delta))$; $(1-\alpha_t)\ell^{\mathrm{CE}}_t+\alpha_t\ell^{\mathrm{KD}}_t$ leads to $\alpha_t=g_t(1-\eta)\le 1-\eta$. CPDP: $z_S,z_{T_1},z_{T_2}$ (mean-pooled) leads to project to 256d, normalise; $\beta(d_{ST_1}-d_{ST_2}-d_{T_1T_2})^2$ and $d_{AB}=1-z_A^{T}z_B$.

Figure 3. EWAD-conf (top) and CPDP (bottom). The gate turns the teacher’s confidence at a token into the weight αt\alpha_t that the KD loss receives there, and the floor η\eta reserves weight for the gold label. CPDP adds a second teacher through distances between projected hidden states, which stays defined when T2T_2 has a different vocabulary. The two terms are summed into one objective.

Figure 4. CHAD. Each training example is scored by how well its KD gradient aligns with the validation gradient, averaged over student checkpoints, and the scores become per-example weights by their rank. The weights are renormalised within the batch, so CHAD redistributes the KD term between examples rather than reducing it. For the BanglaT5 student, the scores are computed for a probe of 1,500 examples and a regressor generalises them to the rest.

CPDP variants. For the BanglaT5 student (CPDP-geo), the hidden states are encoder outputs, mean-pooled with the attention mask, each passed through its own linear projection to 256 dimensions (orthogonally initialised and trained with the student) and L2L_2-normalised; T2T_2 is an mT5 model fine-tuned on the same task, whose vocabulary differs from the student’s. For the LLM students (CPDP-d), the same loss is applied to mean-pooled hidden states of the prompt, taken from the cached teacher passes, with T2T_2 the smaller teacher of the same family. These teachers share the student’s vocabulary, so a KL to T2T_2 would also be possible; we keep the representation-distance form so that the method matches its original design.

CHAD for the BanglaT5 student. The probe contains 1,500 training examples, the validation gradient is taken over 300 validation examples, and CC is the pretrained student and up to three checkpoints of the CE-trained student. A gradient-boosting regressor predicts s(x)s(x) from 13 features (teacher entropy, top-1 probability and top-2 margin; student CE loss and entropy; teacher–student KL; teacher–gold ROUGE-L; and length, compression and novelty features) and scores every training example. The entropy gate uses only teacher entropy, top-1 probability, top-2 margin and source and target length; the semantic gate uses only teacher–gold ROUGE-L and the two lengths. For the LLM students, CC is the plain-KD student’s adapters after its first three epochs, and the scores are recomputed for every seed.

Summarisation variants. Plain KD uses the fixed mixture 0.7 KL1+0.3 KL20.7\,\mathrm{KL}_1 + 0.3\,\mathrm{KL}_2 of the two teachers. EWAD-agree measures teacher agreement as At=∑vmin⁡(pvT1,pvT2)A_t = \sum_v \min(p_v^{T_1}, p_v^{T_2}), a bounded proxy for one minus the Jensen–Shannon divergence, gates with gt=σ(k(At−0.5))g_t = \sigma(k(A_t - 0.5)), and uses the loss gtτ2(w1KL1+w2KL2)+(1−gt)ℓtCEg_t \tau^2(w_1\mathrm{KL}_1 + w_2\mathrm{KL}_2) + (1 - g_t)\ell_t^{\mathrm{CE}}, where the weights wiw_i are a softmax over the teachers’ normalised confidence. EWAD-agree+CPDP-kl adds μ((KL1−KL2)/H(pt)−KL(pT1∥pT2))2\mu\left((\mathrm{KL}_1 - \mathrm{KL}_2)/H(p^t) - \mathrm{KL}(p^{T_1}\mathbin{\|}p^{T_2})\right)^2 with μ=0.05\mu= 0.05. The agreement-harm gate uses ℓtCE+htτ2(w1KL1+w2KL2)\ell_t^{\mathrm{CE}} + h_t\tau^2(w_1\mathrm{KL}_1 + w_2\mathrm{KL}_2) with ht=η+(1−η)Atmth_t = \eta+ (1-\eta)A_t m_t, where mtm_t is the teachers’ average probability of the gold token and η=0.3\eta= 0.3. The constant-gate control sets gt=0.666g_t = 0.666, the gate’s average over the training targets. With τ=2\tau= 2, plain KD weights KD to CE at 2 : 1 and the constant gate at about 8 : 1.

Data and Teachers

Table Table 5 gives the size of every split and Table 6 the quality of every teacher relative to the students it distils into.

tasktrainvaltestref. wordstype
qa36064889061.4extractive answer
mt3500500100011.5Bn→En translation
pp3500500100011.8paraphrase
cc4025711510.1conversational reply
bt26423787551.03-way classification
summ400050050073.1news summarisation

Table 5. Task statistics. “ref. words” is the mean length of the test references in whitespace-separated words.

All Gated-versus-Plain-KD Comparisons

Table Table 7 summarises the comparisons of § 5, and Table 8 lists each one with its paired-bootstrap 95% interval, permutation pp-value, Holm-corrected pp-value (within each group) and TOST equivalence at ±0.032\pm0.032.

studenttaskgateΔ\DeltaF195% CIppermp_{\mathrm{perm}}pHolmp_{\mathrm{Holm}}equiv.
BanglaT5qaentropy gate−0.0005[−0.0050, +0.0040]0.8411.000✓
BanglaT5qasemantic gate+0.0053[+0.0009, +0.0106]0.0370.552✓
BanglaT5qaCHAD+0.0031[−0.0001, +0.0072]0.1041.000✓
BanglaT5mtentropy gate−0.0028[−0.0103, +0.0046]0.4701.000✓
BanglaT5mtsemantic gate−0.0017[−0.0118, +0.0088]0.7421.000✓
BanglaT5mtCHAD−0.0018[−0.0087, +0.0050]0.6051.000✓
BanglaT5ppentropy gate+0.0012[−0.0025, +0.0049]0.5421.000✓
BanglaT5ppsemantic gate−0.0029[−0.0072, +0.0012]0.1761.000✓
BanglaT5ppCHAD−0.0016[−0.0046, +0.0017]0.3311.000✓
BanglaT5cc (n=115n=115)entropy gate−0.0045[−0.0229, +0.0130]0.6241.000✓
BanglaT5cc (n=115n=115)semantic gate−0.0158[−0.0332, +0.0015]0.0781.000✓
BanglaT5cc (n=115n=115)CHAD−0.0166[−0.0378, +0.0043]0.1301.000
BanglaT5btentropy gate+0.0000[−0.0053, +0.0053]1.0001.000✓
BanglaT5btsemantic gate−0.0040[−0.0106, +0.0026]0.4561.000✓
BanglaT5btCHAD−0.0066[−0.0132, −0.0013]0.0660.931✓
gemmaqaEWAD-conf+0.0045[−0.0044, +0.0135]0.3411.000✓
gemmaqa+CPDP−0.0000[−0.0084, +0.0085]1.0001.000✓
gemmaqaCHAD+0.0029[−0.0048, +0.0109]0.4791.000✓
gemmamtEWAD-conf−0.0031[−0.0127, +0.0066]0.5161.000✓
gemmamt+CPDP+0.0041[−0.0054, +0.0138]0.3991.000✓
gemmamtCHAD−0.0102[−0.0196, −0.0007]0.0340.887✓
gemmappEWAD-conf−0.0048[−0.0126, +0.0029]0.2291.000✓
gemmapp+CPDP−0.0047[−0.0119, +0.0025]0.2021.000✓
gemmappCHAD−0.0132[−0.0205, −0.0058]< .0010.024✓
gemmacc (n=115n=115)EWAD-conf+0.0049[−0.0081, +0.0181]0.4801.000✓
gemmacc (n=115n=115)+CPDP+0.0017[−0.0130, +0.0164]0.8311.000✓
gemmacc (n=115n=115)CHAD−0.0025[−0.0195, +0.0146]0.7921.000✓
gemmabtEWAD-conf−0.0093[−0.0185, −0.0013]0.0651.000✓
gemmabt+CPDP−0.0106[−0.0185, −0.0040]0.0080.216✓
gemmabtCHAD+0.0026[−0.0040, +0.0093]0.6851.000✓
QwenqaEWAD-conf−0.0035[−0.0171, +0.0097]0.6161.000✓
Qwenqa+CPDP−0.0025[−0.0155, +0.0099]0.6991.000✓
QwenqaCHAD+0.0192[+0.0069, +0.0320]0.0030.093✓
QwenmtEWAD-conf+0.0020[−0.0074, +0.0112]0.6671.000✓
Qwenmt+CPDP+0.0008[−0.0085, +0.0099]0.8711.000✓
QwenmtCHAD−0.0022[−0.0113, +0.0068]0.6331.000✓
QwenppEWAD-conf−0.0025[−0.0107, +0.0058]0.5601.000✓
Qwenpp+CPDP−0.0011[−0.0093, +0.0067]0.7781.000✓
QwenppCHAD−0.0023[−0.0090, +0.0046]0.4981.000✓
Qwencc (n=115n=115)EWAD-conf−0.0223[−0.0446, +0.0002]0.0521.000
Qwencc (n=115n=115)+CPDP−0.0139[−0.0313, +0.0035]0.1161.000✓
Qwencc (n=115n=115)CHAD−0.0201[−0.0389, −0.0019]0.0320.867✓
QwenbtEWAD-conf−0.0040[−0.0132, +0.0040]0.5451.000✓
Qwenbt+CPDP+0.0013[−0.0093, +0.0119]1.0001.000✓
QwenbtCHAD+0.0013[−0.0079, +0.0106]1.0001.000✓

Table 8. All 45 gated-minus-plain-KD comparisons, token-F1, single seed. “equiv.” marks equivalence at ±0.032 by TOST.

EWAD for the BanglaT5 Student

EWAD-conf and EWAD-conf+CPDP-geo were trained for the BanglaT5 student in a separate implementation (learning rate 3×10−43 \times10^{-4}, effective batch 32, up to 6 epochs, early stopping on validation ROUGE-L) without a plain-KD student, so they are not part of the grid. Table 9 compares each with the CE student of the same implementation and with the CE student of the other implementation (§ 6).

taskmodelΔ\Delta vs own CE95% CIpHolmp_{\mathrm{Holm}}Δ\Delta vs other CE95% CI
qaEWAD-conf−0.0058[−0.0133, +0.0013]0.625+0.0068[+0.0005, +0.0138]
qa+CPDP-geo−0.0038[−0.0119, +0.0041]1.000+0.0088[+0.0010, +0.0172]
mtEWAD-conf−0.0127[−0.0250, −0.0004]0.248−0.0447[−0.0562, −0.0333]
mt+CPDP-geo+0.0484[+0.0366, +0.0606]< .001+0.0164[+0.0069, +0.0261]
ppEWAD-conf+0.0079[+0.0022, +0.0138]0.060+0.0139[+0.0080, +0.0198]
pp+CPDP-geo−0.0082[−0.0135, −0.0029]0.018+0.0022[−0.0079, +0.0035]
ccEWAD-conf+0.0219[−0.0062, +0.0501]0.625−0.0024[−0.0305, +0.0264]
cc+CPDP-geo+0.0396[+0.0151, +0.0645]0.017+0.0152[−0.0131, +0.0435]
btEWAD-conf−0.0013[−0.0093, +0.0066]1.000+0.0026[−0.0053, +0.0106]
bt+CPDP-geo+0.0013[−0.0053, +0.0079]1.000+0.0053[−0.0013, +0.0132]

Table 9. EWAD variants for the BanglaT5 student, single seed, token-F1. pHolmp_{\mathrm{Holm}} is for the comparison with the CE student of the same implementation (Holm over the ten comparisons).

Per-Seed Values

Tables 10 and 11 give the value of every individual run in the seeded study and the summarisation study.

Fixed-Budget Contrasts

Tables 12 and 13 give the contrasts behind § 8. Figure 5 shows the validation token-F1 of every mt method over the ten epochs, averaged over seeds.

Figure 5. Validation token-F1 on mt over ten epochs, mean of three seeds.

CHAD Score Quality

Table 14 gives the diagnostics summarised in § 3: how well CHAD’s score can be predicted for the BanglaT5 student, and how stable the exact scores are across seeds for the gemma student. A gate

qamtppccbt
helpful share.93.66.21.73.71
held-out R2R^2−.12.19.03−.24.00
held-out AUC.47.65.64.59.63
gemma: rank corr. across seeds–0.52–0.700.74–0.810.50–0.650.69–0.78

Table 14. CHAD score quality. Upper rows: the BanglaT5 student, where a regressor predicts the score; “helpful share” is the fraction of probe examples with s(x)>0s(x) > 0, and R2R^2 and AUC are on held-out probe examples. Last row: gemma student, where scores are exact; range of the Spearman correlation of the CHAD weights between pairs of seeds.

studentnnsig.gainsequiv. ±.032equiv. ±.020
LLM students (gemma, Qwen)30102826
BanglaT5 student15001413
total45104239

caption: Gated versus plain KD on token-F1, single seed. “sig.” counts comparisons significant after Holm’s correction within each group, “gains” those that are significant and favour the gate, and “equiv.” those shown equivalent by TOST at each margin.

whose held-out R2R^2 is at most 0.19, and whose weights correlate at 0.50 to 0.81 between seeds, is close to a noisy reweighting of plain KD. The share of examples with s(x)>0s(x) > 0 ranges from 21% (pp) to 93% (qa), against about half on BanSum in Sumit et al. (2026).

Weak versus Converged Teacher

Table Table 15 gives the full comparison behind § 9: for the teacher and for each student, the difference between the run with the under-trained teacher and the run with the converged one, on the same test set and with the CE student shared between the two conditions. The teacher moves by −0.29 token-F1 and no student by more than 0.009; only the semantic gate’s drop survives Holm’s correction on token-F1, and the BERTScore column shows the same ordering at a tenth of the scale.

modelΔF195% CIpHolmp_{\mathrm{Holm}}ΔBERTScore
teacher−0.2909[−0.3202, −0.2624]–−0.0379
CE (no teacher)+0.0000[+0.0000, +0.0000]–+0.0000
plain KD−0.0054[−0.0134, +0.0022]0.353−0.0007
entropy gate−0.0024[−0.0099, +0.0048]0.533−0.0001
semantic gate−0.0094[−0.0167, −0.0027]0.034−0.0014
CHAD−0.0064[−0.0136, +0.0002]0.220−0.0010

Table 15. Under-trained minus converged teacher on qa (BanglaT5 student, single seed), for the teacher and each student. pHolmp_{\mathrm{Holm}} is over the four distilled students.

Summarisation Contrasts

Table Table 16 lists all eight contrasts of § 10 on each of the three metrics, rather than only the comparisons with CE reported in the main text. The three metrics agree on the two effects that matter for the argument: the learned gate beats the constant gate of the same average weight, and no configuration beats CE. They disagree on magnitude only, with BERTScore compressing every difference into a range of 0.006.

metriccontrastΔ\Delta95% CIpHolmp_{\mathrm{Holm}}
rougeLEWAD-agree, constant gate − CE−0.0329[−0.0397, −0.0261]0.002
rougeLplain KD − CE−0.0185[−0.0248, −0.0123]0.008
rougeLEWAD-agree+CPDP-kl − CE−0.0166[−0.0302, −0.0031]0.095
rougeLEWAD-agree − EWAD-agree, constant gate+0.0273[+0.0206, +0.0341]0.005
rougeLagreement-harm gate − CE−0.0108[−0.0172, −0.0044]0.040
rougeLEWAD-agree − plain KD+0.0129[+0.0068, +0.0191]0.027
rougeLagreement-harm gate − plain KD+0.0077[+0.0008, +0.0146]0.095
rougeLEWAD-agree − CE−0.0056[−0.0106, −0.0006]0.095
bertscoreEWAD-agree, constant gate − CE−0.0063[−0.0108, −0.0018]0.178
bertscoreplain KD − CE−0.0013[−0.0023, −0.0003]0.175
bertscoreEWAD-agree+CPDP-kl − CE−0.0004[−0.0015, +0.0006]0.948
bertscoreEWAD-agree − EWAD-agree, constant gate+0.0052[+0.0004, +0.0100]0.242
bertscoreagreement-harm gate − CE−0.0015[−0.0029, −0.0000]0.242
bertscoreEWAD-agree − plain KD+0.0002[−0.0005, +0.0010]0.948
bertscoreagreement-harm gate − plain KD−0.0001[−0.0016, +0.0013]0.948
bertscoreEWAD-agree − CE−0.0011[−0.0021, −0.0001]0.242
semsimEWAD-agree, constant gate − CE−0.0137[−0.0170, −0.0104]0.008
semsimplain KD − CE−0.0082[−0.0129, −0.0035]0.056
semsimEWAD-agree+CPDP-kl − CE−0.0055[−0.0124, +0.0014]0.250
semsimEWAD-agree − EWAD-agree, constant gate+0.0134[+0.0087, +0.0180]0.028
semsimagreement-harm gate − CE−0.0046[−0.0087, −0.0005]0.148
semsimEWAD-agree − plain KD+0.0078[+0.0028, +0.0129]0.061
semsimagreement-harm gate − plain KD+0.0036[−0.0012, +0.0085]0.250
semsimEWAD-agree − CE−0.0003[−0.0048, +0.0041]0.837

Table 16. Summarisation contrasts on all three metrics (Welch’s test on seed means, Holm over the eight contrasts of each metric).

Agreement Between Metrics

Because this study reports no human or LLM-judge evaluation, Table 17 checks whether the automatic metrics could be hiding an effect from each other. For every distilled method of the seeded study it gives the difference from CE on token-F1, BERTScore and semantic similarity. In 10 of the 12 comparisons all three metrics agree on the sign; in the two exceptions the token-F1 difference is smaller than that task’s seed standard deviation, so the disagreement is within noise rather than between measures.

taskmethodΔ\DeltaF1Δ\DeltaBERTScoreΔ\Deltasemsimsame sign∣ΔF1∣<σ|\Delta\mathrm{F1}| < \sigma
btplain KD−0.0079−0.0008−0.0037yesno
btEWAD-conf−0.0057−0.0008−0.0025yesyes
btCHAD−0.0062−0.0007−0.0026yesyes
ccplain KD−0.0062−0.0010−0.0012yesyes
ccEWAD-conf+0.0064+0.0007+0.0030yesyes
ccCHAD+0.0017+0.0010+0.0001yesyes
mtplain KD−0.0102−0.0012−0.0013yesno
mtEWAD-conf−0.0003−0.0001+0.0007noyes
mtCHAD−0.0081−0.0011−0.0011yesno
ppplain KD−0.0045−0.0008−0.0034yesno
ppEWAD-conf+0.0015−0.0002−0.0007noyes
ppCHAD−0.0029−0.0007−0.0028yesyes

Table 17. Seeded study: distilled method minus CE at the early-stopped checkpoint, on three metrics (seed means). The last column marks whether the token-F1 difference is below the task’s seed standard deviation.

Hyperparameters and Environment

Teachers. The LLM teachers are fine-tuned with 4-bit LoRA at learning rate 10−410^{-4} and must reach validation token-F1 of at least 0.40 before their outputs are cached. The BanglaT5 teacher is fully fine-tuned at learning rate 10−410^{-4} for up to 40 epochs and must reach, on validation, exact match ≥0.80\ge0.80 (qa) or token-F1 ≥0.25\ge0.25 (mt, pp), ≥0.45\ge0.45 (bt, above the 0.41 majority class) or ≥0.15\ge0.15 (cc).

CHAD. BanglaT5 student: probe of 1,500 training and 300 validation examples, gradient-boosting gate with 300 trees of depth 4 and learning rate 0.05, trained on 1,200 probe examples. LLM students: exact scores for every training example over the LoRA parameters.

Summarisation. gemma-4-E2B student with 4-bit QLoRA (rank 32, α=64\alpha= 64), batch 2×82 \times8, learning rate 2×10−42 \times10^{-4} with cosine schedule, one epoch, λ=0.5\lambda= 0.5, τ=2\tau= 2; source up to 1,024 and target up to 256 tokens; teachers gemma-4-31B-it and gemma-4-12B through cached top-50 teacher-forced log-probabilities; greedy decoding with up to 200 new tokens.

Environment. torch 2.10.0 with CUDA 12.8, transformers 5.14.1, bitsandbytes 0.50.0, bert_score 0.3.12, sacreBLEU 2.6.0. The mt seeded runs, the CE baselines and summarisation ran on an RTX 5090 (32 GB); the pp, bt and cc seeded runs on an RTX 5070 Ti (16 GB) with gradient checkpointing, which changes memory use but not the computed gradients. Model revisions are pinned on every load.

Reproducing the results. The released material contains the data splits, every prediction file, the per-example scores, the run configurations and the scripts that produce each table and figure from them: analysis/score_all.py (scoring), analyze_grid.py (§ 5, § 6, § 9), analyze_A_mechanism.py (§ 8), analyze_B_ce.py (§ 7), analyze_C_summ.py (§ 10), metric_agreement.py (Appendix § 23), and paper/make_tables.py, which writes every table in this paper. acceptance_check.py verifies that a new environment reproduces the archived numbers.

taskmethodF1 at early stop s42F1 at early stop s7F1 at early stop s13F1 at epoch 10 s42F1 at epoch 10 s7F1 at epoch 10 s13ES epochs
mtCE0.51610.52360.52510.48080.47220.47910,0,0
mtplain KD0.51240.51100.51060.49870.50120.50391,0,0
mtEWAD-conf0.50880.52720.52800.49670.50360.49840,0,0
mtEWAD-conf+CPDP-d0.51450.52740.52410.49620.50070.49380,0,0
mtCHAD0.51720.50970.51370.50040.50270.50161,3,1
mtCE + LS 0.050.51030.52820.50770.49960.49340.49930,0,1
mtCE + LS 0.100.50520.52220.51370.49050.50140.49900,0,0
mtCE + LS 0.200.50460.51070.51730.50340.50460.49780,0,0
ppCE0.51340.51390.51670.48170.48340.48770,0,0
ppplain KD0.51320.50660.51060.50620.50330.50700,4,4
ppEWAD-conf0.51840.51130.51870.50680.51010.50871,0,0
ppCHAD0.50970.51360.51200.50690.50040.50240,2,5
btCE0.95890.95760.95100.95230.95760.95761,4,3
btplain KD0.94300.94440.95630.95500.95630.95761,0,2
btEWAD-conf0.96030.94040.94970.95500.95230.95891,2,3
btCHAD0.94570.95100.95230.95630.95630.95894,1,6
ccCE0.73030.71790.70500.72000.71780.71746,5,5
ccplain KD0.71730.71790.69930.72260.72350.72075,2,0
ccEWAD-conf0.72160.73030.72030.72950.71930.72705,2,1
ccCHAD0.71680.71890.72250.71300.71710.71725,3,4

Table 10. Seeded study, test token-F1 of every run at the early-stopped and final checkpoints, and the epoch (0-based) that early stopping selected in each seed.

methodROUGE-LBERTScore
s42s7s13s42s7s13
CE0.27770.28140.27710.89310.89390.8930
EWAD-agree0.27510.27310.27110.89200.89250.8923
agreement-harm gate0.26810.27090.26470.89210.89240.8911
EWAD-agree+CPDP-kl0.26340.26760.25530.89310.89320.8924
plain KD0.25950.25760.26350.89190.89170.8924
EWAD-agree, constant gate0.24230.24880.24630.88480.88790.8885

Table 11. Summarisation, every run.

taskmethodfinal Δ vs CEpHolmp_{\mathrm{Holm}}degr. Δ vs plain KDpHolmp_{\mathrm{Holm}}
mtplain KD+0.02390.019––
mtEWAD-conf+0.02220.019−0.01160.560
mtEWAD-conf+CPDP-d+0.01950.019−0.01500.192
mtCHAD+0.02420.019−0.00191.000
mtCE––−0.03420.067
ppplain KD+0.02120.004––
ppEWAD-conf+0.02430.004−0.00300.659
ppCHAD+0.01900.004−0.00390.659
ppCE––−0.0257< .001

Table 12. Fixed ten-epoch budget. Left: epoch-ten token-F1 minus CE. Right: degradation minus that of plain KD (negative means the method degrades more). Welch’s test on seed means, Holm within each task.

methoddegradationff per seedff on meansearly-stop Δ vs CEpp
CE−0.0442 ± 0.0082––––
CE + LS 0.05−0.0180 ± 0.01461.14, 0.40, 0.960.77−0.00620.449
CE + LS 0.10−0.0168 ± 0.00350.95, 0.74, 0.800.80−0.00790.252
CE + LS 0.20−0.0089 ± 0.00951.58, 1.09, 0.681.03−0.01070.084
plain KD−0.0101 ± 0.0035––––

Table 13. Label smoothing on mt (three seeds). Degradation is epoch-ten minus early-stopped token-F1 (mean ± sd). ff is the share of the CE-to-plain-KD degradation gap removed. The last two columns compare the early-stopped checkpoint with CE (Welch’s test, uncorrected).

settingLLM studentsBanglaT5 student
adaptationLoRA rr=16, bf16full fine-tuning
optimiserAdamWAdamW
learning rate2×10−42 \times10^{-4}5×10−55 \times10^{-5}
effective batch4×4=164 \times4 = 168×2=168 \times2 = 16
epochs / patience10 / 415 / 6 evaluations
KD λ\lambda / τ\tau0.5 / 2.00.5 / 2.0
EWAD κ\kappa / δ\delta / η\eta10 / median / 0.310 / 0.5 / 0.3
CPDP weight0.050.05
teacher distributioncached top-20full vocabulary
decodinggreedybeam 4

Table 18. Core hyperparameters on the five tasks. qa uses batch 2×82 \times8 for the LLM students. Source/target token budgets: qa 512/32, mt, pp and cc 128/128, bt 128/16.

References

  1. [1]Ajwad Akil, Najrin Sultana, Abhik Bhattacharjee, and Rifat Shahriyar. 2022. BanglaParaphrase: A high-quality Bangla paraphrase dataset. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers).
  2. [2]Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić. 2023. Distilling efficient language-specific models for cross-lingual transfer. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8147–8165, Toronto, Canada. Association for Computational Linguistics.arxiv.org/abs/2306.01709
  3. [3]Tanjim Taharat Aurpa, Md Shoaib Ahmed, Richita Khandakar Rifat, Md Musfique Anwar, and A B M Shawkat Ali. 2023. UDDIPOK: A reading comprehension based question answering dataset in Bangla language. Data in Brief, 47:108933.DOI
  4. [4]Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, and Rifat Shahriyar. 2023. BanglaNLG and BanglaT5: Benchmarks and resources for evaluating low-resource natural language generation in Bangla. In Findings of the Association for Computational Linguistics: EACL 2023, pages 726–735, Dubrovnik, Croatia. Association for Computational Linguistics.
  5. [5]Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vincent. 2021. Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems, volume 3.
  6. [6]Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE M3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.arxiv.org/abs/2402.03216
  7. [7]Raj Dabre and Atsushi Fujita. 2020. Combining sequence distillation and transfer learning for efficient low-resource neural machine translation models. In Proceedings of the Fifth Conference on Machine Translation, pages 492–502, Online. Association for Computational Linguistics.DOI
  8. [8]Rajesh Kumar Das, Mirajul Islam, and Sharun Akter Khushbu. 2023. BTSD: A curated transformation of sentence dataset for text classification in Bangla language. Data in Brief, 50:109445.
  9. [9]Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2185–2194.arxiv.org/abs/1909.03004
  10. [10]Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannah Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. Preprint, arXiv:2002.06305.arxiv.org/abs/2002.06305
  11. [12]Gemma Team, Google DeepMind. 2026. Gemma 4 technical report. Preprint, arXiv:2607.02770.
  12. [13]Mahmudul Hasan, Abdullah Ibne Hanif Arean, and Md Mosaddek Khan. 2024. BanSum: A dataset for Bangla abstractive article summarization with multiple sentences. Dataset.
  13. [14]Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence representations for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2101–2112, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  14. [15]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. Preprint, arXiv:1503.02531.arxiv.org/abs/1503.02531
  15. [16]Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70.DOI
  16. [17]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
  17. [18]Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174, Online. Association for Computational Linguistics.arxiv.org/abs/1909.10351
  18. [19]Pratik Joshi, Sebastian Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.arxiv.org/abs/2004.09095
  19. [20]Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. Association for Computational Linguistics.arxiv.org/abs/1606.07947
  20. [21]Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395.
  21. [22]Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885–1894. PMLR.arxiv.org/abs/1703.04730
  22. [23]Jahyun Koo, Yerin Hwang, Yongil Kim, Taegwan Kang, Hyunkyung Bae, and Kyomin Jung. 2025. SWITCH: Studying with teacher for knowledge distillation of large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3733–3746, Albuquerque, New Mexico. Association for Computational Linguistics.
  23. [24]Daniël Lakens. 2017. Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4):355–362.
  24. [25]Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  25. [26]Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2019. Improved knowledge distillation via teacher assistant. Preprint, arXiv:1902.03393.arxiv.org/abs/1902.03393
  26. [27]Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. 2019. When does label smoothing help? In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.arxiv.org/abs/1906.02629
  27. [28]Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. 2019. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).arxiv.org/abs/1904.05068
  28. [29]Gabriel Pereyra, George Tucker, Ian Choro​​wski, Łukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. Preprint, arXiv:1701.06548.arxiv.org/abs/1701.06548
  29. [30]Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191.arxiv.org/abs/1804.08771
  30. [31]Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, volume 33, pages 19920–19930. Curran Associates, Inc.arxiv.org/abs/2002.08484
  31. [32]Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  32. [33]Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Beehmaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. 2022. Samanantar: The largest publicly available parallel corpora collection for 11 Indic languages. Transactions of the Association for Computational Linguistics, 10:145–162.
  33. [34]Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338–348.arxiv.org/abs/1707.09861
  34. [35]Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics.arxiv.org/abs/2004.09813

Paper details

vlnc:W_j8b9ae7d
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents