WorksComputer Science
Watermark Basins: Black-Box Detection of Distilled LLMs
Abstract
Proprietary language models and their reasoning traces are vulnerable to API-based distillation, where an adversary trains a student model on the teacher model's outputs. To obtain evidence of distillation, model providers can watermark the teacher's outputs, aiming to leave a radioactive signal in the student's outputs. Existing black-box detectors search for such radioactivity using token-level statistics of the suspect model's outputs. However, token-level signals are not always detectable, for instance when the distillation traces are sampled at a low temperature. In this work, we discover the watermark basin, a persistent signal that enables more reliable watermark detection: during fine-tuning, watermarked teacher traces embed a distinct, key-specific geometric signal in parameter space that lies orthogonal to the ordinary task-specific fine-tuning subspace. Building on this, we introduce Watermark Basin Alignment (WM-Basin), a black-box watermark detector that queries the suspect model through its API, without access to its weights or logits. WM-Basin maps generated text to gradient representations using a fixed clean probe model, projects out the ordinary fine-tuning task directions, and tests whether the residual gradient aligns with the watermark basin induced by the watermark key. We evaluate WM-Basin extensively across diverse teacher models and student architectures (from 135M to 8.2B parameters) distilled on mathematical reasoning. At a strict 0.1% false-positive rate, WM-Basin detects 79% of distilled models at n-gram=1 and 45% at n-gram=3, substantially improving over the state-of-the-art token-level baseline (29% and 1.3%, respectively). Because the watermark signal resides in a persistent basin, it survives post-distillation modifications including quantization, pruning, and reinforcement-learning post-training.
Introduction
Proprietary language models are increasingly targeted by API-based distillation: an adversary queries a closed teacher model and fine-tunes a competing student model on its outputs. This attack extends black-box model extraction, first demonstrated against ML-as-a-service systems Tramèr et al. [51], and has become a major threat to reasoning LLMs [6, 18, 43]. Providers have responded by withholding reasoning traces, but concealment does not prevent distillation. Hidden traces can be extracted at scale from encrypted response blocks Panfilov et al. [36], while answers and short summaries can be expanded into synthetic traces that recover much of the benefit of training on the originals Zhang et al. [58]. Other defenses make extraction harder rather than detectable [6, 18, 48, 31]. We review related work in full in § 4.
In this work, we focus on output watermarking, which is encouraged by emerging regulatory requirements European Commission [14]. Anthropic, for example, embeds watermarks in texts generated by Claude models released beginning in August 2026 Anthropic [6]. Such watermarks can survive distillation: Sander et al. [46] show that students trained on watermarked teacher outputs retain a “radioactive” signal. Their black-box detector, to our knowledge the only existing one for this setting and our baseline throughout, tests suspect outputs for an excess of watermark-favored tokens, but only where watermark-conditioned contexts recur. We find that this dependence is fragile: detection degrades sharply with larger context windows and loses most of its power under low-temperature teacher sampling. We therefore seek a watermark signal that does not require exact token-level recurrence.
Our main finding is the watermark basin: supervised fine-tuning on clean (watermark-free) teacher traces moves a student along a broad task direction, whereas watermarked traces add a faint, key-specific watermark direction Figure 1, leading the student into a key-specific region of parameter space. Post-training updates cannot easily push the student out of this basin. Building on this geometry, we introduce Watermark Basin Alignment (WM-Basin), a black-box detector that represents the suspect model’s outputs and the watermarked teacher’s outputs as gradients of one fixed, clean probe model. After removing the task subspace estimated from clean models, the distilled student should align with the watermark key more closely than with decoy keys that were never used to watermark the teacher. WM-Basin turns this relative rank into a calibrated -value using a non-parametric ranking test Dwass [12]: a small value is evidence that the suspect was distilled from the claimed teacher and key on the audited domain Figure 2.

Figure 1. The watermark basin. Training on watermarked traces moves the student into a persistent, key-specific watermark basin, whereas clean traces from the same teacher move it only along the task direction. Watermark Basin Alignment (WM-Basin) recovers this geometry from black-box outputs alone to detect distillation.

Figure 2. WM-Basin: black-box distillation detection via watermark basin alignment. The three panels are the three stages of § 2.4. (i) The verifier collects suspect outputs on test prompts and, on separate build prompts, teacher outputs under the claimed watermark key and decoy keys. Backpropagating each collection through one fixed probe model gives the suspect gradient and one reference gradient per key. (ii) Raw gradients share large common task components that inflate the cosine similarities, so every key appears to overlap with the suspect. Projecting out the shared task directions leaves the residuals and , among which only the claimed key stays close to the suspect; is the angle between them. (iii) Each candidate key receives the alignment score , and the claimed key’s rank among the decoys gives the -value . The claimed key ranking at the top is evidence of distillation; a decoy key at the top is not.
We evaluate WM-Basin along many axes: the student architecture, the teacher model, the watermarking scheme, the sampling temperature and watermarked fraction of the distillation traces, the fine-tuning recipe, and the task domain. Among 1,200 student models distilled from DeepSeek-R1-Distill-Qwen-14B [21] on OpenR1-Math reasoning traces [24], at a 0.1% false-positive rate, WM-Basin detects 79% of distilled models at -gram=1 and 45% at -gram=3, against 29% and 1.3% for the state-of-the-art black-box detector [46]. Quantization, pruning, and reinforcement-learning post-training all leave detection intact, as none of them moves the student out of the watermark basin. Like any black-box method, WM-Basin reads only the suspect model’s text, so detection depends on that text carrying the teacher’s token distribution. Continued training on clean data can reshape that distribution and therefore degrade black-box detection, although the basin itself survives in the weights and remains easy to identify under white-box access.
Our contributions are as follows:
Discovery of watermark basins. We find that watermarked distillation places every distilled student in a persistent basin, spanned by the ordinary task direction and a faint, key-specific watermark direction. Leaving this basin without knowledge of the key is difficult: post-training operations (quantization, pruning, supervised fine-tuning, RL) displace weights primarily along the task axis and leave the key-specific component intact, which we exploit for detection.
A parameter-space black-box detector (WM-Basin). WM-Basin looks for distillation evidence in parameter space. It maps text outputs to gradients of a fixed clean probe model, removes the shared task subspace, and scores the residual alignment against decoy keys. The watermark becomes far more detectable than in token space, with exact finite-sample false-positive control and no access to the suspect’s parameters.
Robust detection at scale. Across 24 student architectures (135M to 8.2B parameters), 4 teachers, and 4 watermarking schemes, WM-Basin achieves 79% TPR at 0.1% FPR versus 29% for the state-of-the-art token-level detector, and remains effective under quantization, pruning, and RL post-training.
Watermark-Basin Detection
Threat Model and Statistical Setup
Distillation setting and adversary model. A proprietary teacher model serves completions watermarked under a secret key , drawn from the watermark scheme’s key-generation procedure. Following deployed systems [1, 47, 6] (Datathatri et al. 2024), we consider distortion-free schemes: the teacher’s marginal next-token distribution is preserved, but for each prompt the output is coupled to the pseudorandomness induced by . The adversary adaptively chooses prompts , receives watermarked traces , builds , and trains a student by supervised fine-tuning on . Without knowledge of , watermarked completions are indistinguishable from clean text, so the default adversary is unaware of the watermark and uses the traces as-is. After fine-tuning, the adversary may further post-train on clean data, e.g., via continued SFT or GRPO [21].
Verifier regime. An audit is opened by the verifier, either the model owner or an authorized third party, and begins by naming one capability the owner sells and the suspect appears to compete on. We call it the audited domain (e.g., math) and let be its induced prompt distribution; the verifier chooses both and draws every prompt it uses from . It holds two disjoint pools: a build pool used to construct the detector, and a test pool used to read the suspect. Neither overlaps the adversary’s prompts, which the verifier never sees, and keeping build and test prompts apart avoids score inflation from prompt reuse. The verifier’s access to is black-box: it observes only texts sampled on the test prompts. The access is also unsupervised in the sense of Sander et al. [46]: the verifier does not hold the traces was trained on, only fresh teacher traces on the build prompts (unwatermarked, watermarked under , and watermarked under independent decoy keys ), together with a pool of public clean checkpoints. It knows the system prompts used to query and the teacher, because it chooses them, and the number of test prompts sent to the suspect is the query budget.
Statistical objective. Let be any scalar alignment score computed from the suspect’s completions on test prompts and from teacher traces on build prompts under key ; § 2.4 instantiates it. Its absolute magnitude is not comparable across suspects, prompt pools, or probe models, so an audit reads only where the claimed key ranks among decoy keys. Our null hypothesis is that the suspect’s outputs on the audited domain contain no information about the production key . If the texts do depend on , the dependence can only arise through the distillation path . Rejecting therefore establishes watermark radioactivity: the suspect was trained on the teacher’s keyed text, which separates it from any model not distilled under . A failure to reject is scoped to domain ; we measure detection power beyond the distillation domain in § 3.2.
Watermark Basins in Parameter Space
We first use a white-box setting to investigate what distinguishes a student trained on watermarked traces from one trained on clean traces, and thus what makes a model radioactive. Figure 1 presented the watermark basin; here we test that geometry directly with access to the student weights. We fix the true key and fine-tune the same base model either on teacher traces watermarked with , giving , or on unwatermarked traces, giving a clean control . The teacher, prompts, initialization, and training recipe are otherwise identical, so both students learn the same task while only one sees the watermark. Let be the total update from watermarked training and the clean task update. We isolate the watermark-specific part of the update by removing from its component along :
The subtracted term is the part of the watermarked update that the clean student also makes, i.e., learning the task. What remains, , is the part that only the watermark can explain. We normalize and to obtain the watermark axis and the task axis .
To map the basin, we start at and perturb the weights along each axis, , over a grid of step sizes . At each point, we apply the open-weight detector of Sander et al. [46] to the perturbed student and record , where a smaller -value is stronger evidence of distillation. Figure 3 shows that the surface is strongly anisotropic: a small step along the watermark axis makes the watermark signal collapse, whereas much larger displacements along the task axis leave it detectable. The basin is broad along but has a sharp boundary along .
Figure 3. The anisotropic geometry of the watermark basin, measured as the white-box detection significance () around a radioactive student (red star; in the text) fine-tuned on watermarked teacher traces. Left to right: 3D landscape, 2D contour map, and 1D slices through . The signal falls to the detector’s floor within a few steps along the watermark direction (red curve), while remaining detectable over a much broader range along the task direction (green curve).
This asymmetry gives the two axes different roles. The width along suggests why ordinary post-training can alter task behavior without erasing the basin, a hypothesis we test under compression and continued training in § 3.3. The narrowness along makes the basin key-specific, which motivates comparing against decoy keys rather than thresholding a generic watermark score. This analysis relies on access to the student’s parameters, which a black-box verifier does not have. Next, we show how to detect the same key-specific signal using only text sampled from the suspect.
WM-Basin: Black-Box Estimation via Probe Gradients
WM-Basin: Black-Box Estimation via Probe Gradients
A black-box verifier cannot inspect the suspect’s parameters and observes only its generated text. WM-Basin maps suspect, keyed-teacher, and clean-model outputs into a shared gradient representation, removes the variation common to clean models, and ranks the claimed key by the remaining alignment. Figure 2 gives the overview and Alg. 1 the full procedure.
Stage (i): from text to probe gradients. We use a small clean model as the probe, warm its LoRA parameters on held-out unwatermarked teacher traces, and then freeze them. A prompt–completion pair is represented by the gradient , the parameter nudge that would make the probe more likely to produce given . We average these gradients over the suspect’s completions on the test prompts, and over the teacher’s completions under each key on the build prompts:
If the suspect inherited key , the nudges induced by its outputs should share the key-specific component of the nudges induced by teacher outputs under ; this alignment is the signal we measure. The gradients give a common coordinate system for comparing first-order fine-tuning tendencies; they do not reconstruct the suspect’s hidden weight update (Xia et al., [54]). A single fixed random projection maps every gradient to dimensions while approximately preserving inner products (Johnson et al., [25]). This stage needs only keyed teacher traces, not the watermark scheme’s own detector.
Stage (ii): removing the shared task subspace. Raw probe gradients share large components caused by the task, the response format, and generic language modeling. These common components inflate every cosine similarity and hide the small differences that distinguish one key from another. We therefore adapt the All-but-the-Top post-processing used to dehub learned representations (Mu et al., [33]). Clean checkpoints generate responses on held-out build prompts, and PCA on their probe gradients gives the top directions of clean-model variation, collected in the orthonormal basis . We project every gradient off this basis and score each key by the cosine of the residuals:
The projection keeps only what clean models do not have in common, so the cosine asks whether the suspect’s residual points the same way as key ’s residual. We fit on clean models alone, never on suspects, and use (App. § 7.§ 5).
Stage (iii): ranking against decoy keys. Because has no absolute scale, the verifier evaluates it at the claimed key and at independent decoy keys , and reads the claimed key’s rank:
In words, is the fraction of the candidate keys, the claimed key included, that score at least as high as the claimed key. Under , the claimed key and the decoys are independent outputs of the same key generator and pass through the same probe, random projection, basis , and prompts. Their scores are therefore exchangeable, which gives exact finite-sample validity marginally over the draw of (Dwass, [12]; Pesarin and Salmaso, [39]) (App. § 6). Under and absent ties, is uniform on the grid ; a very small lets the verifier conclude distillation from with high confidence. The smallest attainable value is , so certifying or requires or 999 decoys.
Build once, audit many suspects. Alg. 1 separates what a verifier computes once from what it pays per suspect. The build-once block never touches the suspect: it reads only the candidate keys, the build pool , and the clean checkpoints, and produces the warmed probe, the reference bank , and the task basis (stages (i) and (ii)). The detect block is the only part paid per suspect: it samples the suspect on , maps that text through the same probe and projection, and ranks the claimed key (stage (iii)). The claimed key enters exactly where each decoy does, as an index that selects reference traces, which is what makes the rank test exact. No step invokes the watermark scheme’s own test, so
Algorithm 1 WM-Basin: black-box detection of watermark distillation via basin alignment.
Require: Query access to the suspect model $M_{\mathrm{sus}}$ (black-box API) and the watermarked teacher; claimed key $k^\star$ and $M$ decoy keys $k_1,\ldots,k_M \leftarrow \operatorname{KeyGen}$; disjoint build and test prompt pools $\mathcal{X}_{\mathrm{build}},\mathcal{X}_{\mathrm{test}}$ from the audited domain $\mathcal{D}$; a pool of clean off-the-shelf models; significance level $\alpha$.
Ensure: $p$-value $p_{\mathrm{rank}}$; report the suspect as distilled into the basin of $k^\star$ when $p_{\mathrm{rank}}\leq\alpha$.
Build once (offline; independent of the suspect)
1: $\theta_{\mathrm{probe}}\leftarrow$ clean small model warmed by a short LoRA pass on unwatermarked teacher traces from $\mathcal{X}_{\mathrm{build}}$
2: for each key $k\in\{k^\star,k_1,\ldots,k_M\}$ do
3: Sample teacher responses under key $k$ on $\mathcal{X}_{\mathrm{build}}$
4: $\Delta_k\leftarrow$ average probe gradient of these responses ▷ Eq. {xref}`Eq. (2)`
5: end for
6: $U\leftarrow$ top-$q$ principal directions of the clean pool’s probe gradients on $\mathcal{X}_{\mathrm{build}}$
7: for each key $k\in\{k^\star,k_1,\ldots,k_M\}$ do
8: $\widetilde{\Delta}_k\leftarrow(I-UU^\top)\Delta_k$ ▷ Residual reference gradient
9: end for
Detect (online; once per suspect)
10: Sample answers from $M_{\mathrm{sus}}$ on $\mathcal{X}_{\mathrm{test}}$ ▷ Unused by every step above
11: $\Delta_{\mathrm{sus}}\leftarrow$ average probe gradient of the suspect’s answers ▷ Eq. {xref}`Eq. (2)`
12: $\widetilde{\Delta}_{\mathrm{sus}}\leftarrow(I-UU^\top)\Delta_{\mathrm{sus}}$ ▷ Residual suspect gradient
13: for each key $k\in\{k^\star,k_1,\ldots,k_M\}$ do
14: $S(k)\leftarrow\cos\left(\widetilde{\Delta}_{\mathrm{sus}},\widetilde{\Delta}_k\right)$ ▷ Eq. {xref}`Eq. (3)`
15: end for
16: $p_{\mathrm{rank}}\leftarrow\frac{1+\sum_{j=1}^{M}\mathbb{I}(S(k_j)\geq S(k^\star))}{M+1}$ ▷ Eq. {xref}`Eq. (4)`
17: return $p_{\mathrm{rank}}$; report “distilled into the basin of $k^\star$” if $p_{\mathrm{rank}}\leq\alpha$
the same algorithm applies unchanged to any keyed watermark. The computational cost is dominated by the one-time generation of teacher traces under keys (App. § 7.§ 4).
Experiments and Results
Most main-body tables report all three -gram windows of the watermark () at three operating points. Unless stated otherwise, the text quotes -gram=3, the hardest window for both detectors.
Experimental Setup
Distillation. We evaluate distillation detection across 24 suspect model architectures, 4 teacher models, and 4 watermarking schemes. In the default setting, the teacher is DeepSeek-R1-Distill-Qwen14B (Guo et al. [21]) watermarked with TextSeal (Sander et al. [47]), a dual-key scheme that samples each token under one of two secret keys, mixed at a fixed rate . Following standard reasoning-distillation protocols, an adversary fine-tunes a Qwen2.5-3B base model (Yang et al. [56]) for math on 1,900 quality-filtered OpenR1-Math-220k traces (Hugging Face [24]) from that teacher. All traces are generated under one fixed math-teacher system prompt, on which the suspect model is also conditioned during fine-tuning. We evaluate capability on held-out GSM8K (Cobbe et al. [9]) and Math-500 (Hendrycks et al. [22]); App. § 7.§ 2 gives the full distillation setup.
Sampling suspect model outputs. The verifier chooses one domain to audit and draws every prompt it uses for detection from that domain; it never sees the adversary’s prompts or traces (§ 2.1). By default the audited domain is OpenR1-Math-220k (Hugging Face [24]), held out from the adversary’s distillation prompts, and every audit stage uses the same system prompt as the distillation traces, which we call a matched-prompt audit; § 3.2 varies both the domain and the system prompt. We read each suspect on 1,900 prompts from the test pool , disjoint from the distillation prompts , and score the first 512 tokens of every response, around 1M tokens per suspect.
Detector. WM-Basin turns text into a gradient with a probe model, by default a clean Qwen2.5-3B base warmed on around 450 clean teacher traces from the audited domain (App. § 8.6 studies both the choice of probe and the warm-up). On 512 build-pool prompts we query the teacher under the claimed watermark key and 999 decoys, then map those responses and the suspect generations through the probe, giving a 1,000-key reference gradient bank and the suspect gradient scored against it. Before scoring, we project both away from a PCA task subspace fit on 43 off-the-shelf checkpoints, which leaves the key-specific alignment that the rank test reads (§ 2.4). All three stages draw separately from the build pool , disjoint from . Ranking the claimed key gives -values down to 0.001; unless stated otherwise we report the true positive rate (TPR) at , i.e., at a 1% false positive rate (FPR). App. § 7.6 validates the false positive rate empirically.
Baseline. We compare with the black-box sampled-token detector of Sander et al. [46], to our knowledge the only existing detector for this setting. It reads the same suspect text as WM-Basin, so every comparison is paired. We run it under its own protocol with both filters that give it power: a position is scored only when its context window occurs in a held-out corpus of 5,000 teacher traces regenerated under the key being scored, and each distinct (context, token) pair is scored once. Its -value is computed analytically from the score’s null distribution rather than by ranking decoys, and we threshold both detectors at the same -value. App. § 7.§ 4 gives the test used for each watermark scheme and the remaining settings.
Detection Performance
Effect of the student architecture. Does detection depend on which student architecture the adversary selects? We vary the student across 24 architectures spanning 6 families (135M to 8.2B parameters) and distill 50 models per architecture, for 1,200 distilled models per -gram window, holding the teacher, distillation recipe, probe model, and reference gradient bank fixed. Pooled over all architectures (the Overall row of Table 1), WM-Basin outperforms the baseline at every window and operating point, and at the 5% and 1% thresholds its margin widens with the window. At -gram=3 and , the baseline recovers almost nothing (1.3%) while WM-Basin detects 45% of distilled models. Detectability is governed by task acquisition rather than architecture: WM-Basin mostly misses students that failed to learn the task, so evading detection means giving up the capability that motivated distillation.
| Student structure | -gram=1 Acc. | -gram=1 WM-Basin TPR@FPR 5% | 1% | .1% | -gram=1 Black-box TPR@FPR 5% | 1% | .1% | -gram=2 Acc. | -gram=2 WM-Basin TPR@FPR 5% | 1% | .1% | -gram=2 Black-box TPR@FPR 5% | 1% | .1% | -gram=3 Acc. | -gram=3 WM-Basin TPR@FPR 5% | 1% | .1% | -gram=3 Black-box TPR@FPR 5% | 1% | .1% |
| Qwen3-4B-Instruct | 88.7 | 100 | 100 | 100 | 63 | 26 | 9 | 89.5 | 100 | 100 | 100 | 38 | 18 | 8 | 91.6 | 100 | 100 | 88 | 8 | 2 | 0 |
| Qwen3-4B-Instruct-2507 | 91.4 | 100 | 100 | 100 | 56 | 33 | 14 | 91.6 | 100 | 100 | 100 | 36 | 22 | 8 | 90.8 | 98 | 96 | 90 | 14 | 4 | 0 |
| Qwen3-8B-Instruct | 89.6 | 100 | 100 | 100 | 49 | 19 | 2 | 90.8 | 100 | 100 | 100 | 20 | 12 | 0 | 89.9 | 100 | 100 | 92 | 6 | 0 | 0 |
| Qwen2.5-7B-Instruct | 87.8 | 100 | 100 | 100 | 70 | 47 | 12 | 87.9 | 100 | 100 | 98 | 42 | 24 | 8 | 88.8 | 100 | 92 | 80 | 18 | 2 | 0 |
| Qwen2.5-Math-7B-Instruct | 83.1 | 100 | 100 | 100 | 42 | 23 | 2 | 85.6 | 100 | 100 | 96 | 34 | 10 | 2 | 88.0 | 100 | 100 | 76 | 12 | 4 | 0 |
| DeepSeek-R1-Distill-Qwen-7B | 86.7 | 100 | 100 | 100 | 44 | 7 | 0 | 87.4 | 100 | 100 | 98 | 24 | 4 | 2 | 87.0 | 100 | 96 | 68 | 0 | 0 | 0 |
| Qwen3-8B-Base | 88.6 | 95 | 91 | 74 | 51 | 28 | 5 | 89.1 | 80 | 70 | 46 | 40 | 20 | 8 | 87.0 | 62 | 48 | 24 | 10 | 6 | 0 |
| Llama-3.1-8B-Instruct | 84.1 | 100 | 100 | 100 | 91 | 77 | 42 | 84.5 | 100 | 100 | 88 | 80 | 52 | 18 | 84.4 | 98 | 80 | 66 | 24 | 4 | 4 |
| Qwen3-1.7B-Instruct | 77.6 | 100 | 100 | 100 | 42 | 19 | 5 | 80.1 | 100 | 100 | 96 | 28 | 16 | 2 | 81.7 | 98 | 88 | 70 | 4 | 0 | 0 |
| Qwen2.5-3B-Instruct | 77.7 | 100 | 100 | 100 | 72 | 56 | 19 | 77.1 | 100 | 98 | 94 | 50 | 26 | 8 | 79.7 | 100 | 90 | 58 | 18 | 6 | 0 |
| DeepSeek-R1-Distill-Qwen-1.5B | 75.7 | 100 | 100 | 100 | 33 | 2 | 2 | 77.3 | 100 | 100 | 94 | 12 | 2 | 0 | 74.8 | 100 | 94 | 76 | 4 | 0 | 0 |
| Llama-Guard-3-8B | 61.7 | 100 | 100 | 100 | 100 | 84 | 51 | 62.6 | 100 | 96 | 76 | 82 | 64 | 22 | 63.0 | 92 | 70 | 52 | 24 | 6 | 2 |
| Qwen3-0.6B-Instruct | 61.3 | 100 | 100 | 100 | 72 | 42 | 16 | 62.3 | 100 | 98 | 88 | 30 | 16 | 2 | 59.6 | 96 | 74 | 50 | 10 | 0 | 0 |
| Qwen2.5-1.5B-Base | 55.9 | 100 | 100 | 100 | 74 | 35 | 7 | 52.2 | 98 | 90 | 66 | 38 | 20 | 6 | 58.5 | 82 | 62 | 38 | 12 | 6 | 0 |
| Qwen2.5-1.5B-Instruct | 58.2 | 100 | 100 | 98 | 74 | 44 | 16 | 58.5 | 92 | 80 | 54 | 42 | 10 | 0 | 54.6 | 68 | 52 | 34 | 8 | 2 | 0 |
| Llama-3.2-1B-Instruct | 36.0 | 100 | 100 | 98 | 70 | 42 | 16 | 35.8 | 98 | 92 | 72 | 52 | 30 | 16 | 38.7 | 96 | 76 | 44 | 14 | 6 | 0 |
| Qwen2.5-0.5B-Base | 31.9 | 100 | 100 | 91 | 88 | 65 | 44 | 30.4 | 92 | 78 | 56 | 66 | 44 | 12 | 36.8 | 88 | 62 | 20 | 16 | 6 | 0 |
| Qwen2.5-0.5B-Instruct | 30.7 | 100 | 100 | 95 | 79 | 54 | 19 | 31.6 | 94 | 84 | 58 | 54 | 28 | 4 | 30.4 | 92 | 64 | 26 | 18 | 4 | 2 |
| SmolLM2-1.7B-Base | 18.5 | 95 | 86 | 56 | 26 | 5 | 2 | 18.0 | 82 | 58 | 34 | 6 | 0 | 0 | 18.0 | 64 | 34 | 12 | 4 | 0 | 0 |
| Gemma-2-2B-Base | 10.8 | 47 | 19 | 7 | 100 | 98 | 88 | 10.4 | 12 | 6 | 2 | 86 | 72 | 44 | 5.9 | 14 | 4 | 0 | 30 | 14 | 0 |
| Llama-Guard-3-1B | 3.8 | 100 | 84 | 56 | 86 | 54 | 21 | 4.4 | 88 | 64 | 38 | 46 | 24 | 10 | 5.8 | 80 | 42 | 16 | 16 | 6 | 2 |
| Gemma-3-1B-Base | 1.8 | 56 | 26 | 2 | 84 | 44 | 14 | 1.7 | 38 | 14 | 4 | 46 | 20 | 0 | 1.8 | 32 | 14 | 4 | 16 | 4 | 0 |
| SmolLM2-135M-Instruct | 1.5 | 56 | 28 | 7 | 7 | 5 | 0 | 1.6 | 50 | 18 | 10 | 6 | 4 | 0 | 1.8 | 34 | 16 | 2 | 4 | 0 | 0 |
| SmolLM2-135M-Base | 1.1 | 56 | 26 | 2 | 12 | 5 | 0 | 1.4 | 48 | 18 | 6 | 6 | 2 | 0 | 1.3 | 32 | 12 | 4 | 0 | 0 | 0 |
| Overall | 92 | 86 | 79 | 71 | 52 | 29 | 86 | 78 | 66 | 52 | 29 | 12 | 80 | 65 | 45 | 20 | 7 | 1 |
Table 1. Table 1 Distillation detection across 24 student architectures. A single TextSeal-watermarked teacher (DeepSeek-R1-Distill-Qwen-14B) is distilled into 24 students. Each cell is the percentage of up to 50 distilled models per architecture flagged at a -value of 5%, 1%, or 0.1%; Overall pools all models in that column (1,200 for each), and Acc. is post-distillation GSM8K accuracy.
Effect of the distillation temperature. What happens when the adversary samples its distillation traces at a temperature the verifier does not know? We generate the teacher’s traces at , 0.7, and 0.5 and distill 100 students from each setting. A lower concentrates the teacher’s output distribution on its most likely tokens. For WM-Basin, the reference gradient bank remains fixed at , creating a mismatch at the two lower temperatures. Table 2 shows that lowering the distillation temperature costs the black-box baseline most of its power at every window, whereas WM-Basin at still flags every suspect at -gram=1 and about half of them at -gram=3, even at . Temperature thus governs what the tokens reveal, not what the gradient reveals: lowering it is a cheap evasion against the baseline but not against WM-Basin.
| Temp. | \multicolumn{6}{c}{n-gram=1} | \multicolumn{6}{c}{n-gram=2} | \multicolumn{6}{c}{n-gram=3} | |||||||||||||||
| WM-Basin 5% | WM-Basin 1% | WM-Basin .1% | Black-box 5% | Black-box 1% | Black-box .1% | WM-Basin 5% | WM-Basin 1% | WM-Basin .1% | Black-box 5% | Black-box 1% | Black-box .1% | WM-Basin 5% | WM-Basin 1% | WM-Basin .1% | Black-box 5% | Black-box 1% | Black-box .1% | |
| 1.0 | 100 | 100 | 100 | 72 | 43 | 26 | 100 | 100 | 98 | 50 | 25 | 9 | 100 | 99 | 88 | 23 | 9 | 1 |
| 0.7 | 100 | 100 | 100 | 37 | 16 | 2 | 100 | 97 | 88 | 25 | 12 | 0 | 99 | 93 | 62 | 12 | 2 | 0 |
| 0.5 | 100 | 100 | 99 | 24 | 7 | 2 | 94 | 78 | 40 | 8 | 2 | 1 | 71 | 51 | 22 | 4 | 2 | 0 |
Table 2. Distillation detection across distillation temperatures. Qwen2.5-3B is distilled from DeepSeek-R1-Distill-Qwen-14B on traces sampled at each temperature, while the detector’s reference bank stays fixed at . Each cell is the percentage of up to 100 distilled models (one per watermark key) flagged at a -value threshold of 5%, 1%, or 0.1%, i.e., the TPR at that false-positive rate.
Effect of the teacher model. Are some teachers harder to protect than others? We distill the same Qwen2.5-3B student from each of four teachers spanning three families and 8B to 120B parameters: DeepSeek-R1-Distill-Qwen-14B, gpt-oss-120B ([3]), QwQ-32B ((Team, 2025)), and Llama-3.1-8B ((Grattafiori et al., 2024)), rebuilding the reference bank for each. Table 3 shows that, at , WM-Basin detects 96 to 100% of distilled models for every teacher. The baseline loses most of its power on two teachers and approaches WM-Basin on the other two. Its power is thus inherited from the teacher and varies sharply with it, whereas WM-Basin is robust across teachers; App. § 8.§ 2 relates this to how often each student repeats its contexts.
| Teacher | Acc. | WM-Basin 5% | WM-Basin 1% | WM-Basin .1% | Black-box 5% | Black-box 1% | Black-box .1% |
| DeepSeek-R1-Distill-Qwen-14B | 68.9 | 100 | 99 | 88 | 23 | 9 | 1 |
| gpt-oss-120B | 77.2 | 96 | 76 | 30 | 18 | 10 | 2 |
| QwQ-32B | 80.8 | 99 | 97 | 91 | 88 | 74 | 42 |
| Llama-3.1-8B | 74.9 | 100 | 100 | 100 | 93 | 87 | 64 |
Table 3. Distillation detection across teacher models. The same Qwen2.5-3B student is distilled at from each of four teachers spanning three families and 8B to 120B parameters, and the detector’s reference bank is rebuilt for each teacher. Cells are read as in Table 2, at -gram=3, over up to 100 distilled models per teacher; Acc. is mean student GSM8K accuracy. The thresholds are nominal -values on the DeepSeek row, as elsewhere, and empirical false-positive rates on the three swapped-teacher rows. Teacher properties are in Table 12.
Effect of the watermark scheme. Is detection tied to the watermark the teacher happens to deploy? We replace TextSeal in the teacher’s decoder with GumbelMax (Aaronson, [1]), KGW () (Kirchenbauer et al., [26]), and SynthID (Dathathri et al., [10]), rebuilding the reference bank for each and distilling 50 students per scheme and window. Tab. 4 shows that all three transfer: WM-Basin flags every student at and -gram=1 under each scheme, and it outperforms the baseline in every cell, by a margin that widens with the window at . WM-Basin needs only traces the teacher generated under a key, not the watermark’s own detection test. The baseline, in contrast, scores the suspect’s tokens with that test, which must be derived scheme by scheme (App. B.4). To audit a new keyed watermark, the verifier therefore only generates teacher traces under its keys, rebuilds the reference bank, and runs the same detector.
Effect of the audited domain and system prompt. How close must the audited domain be to the adversary’s distillation data? We hold the 100 suspects fixed and change the domain the verifier audits, from the math domain they were distilled on (OpenR1-Math, MATH, GSM8K) to two domains outside it (code and chat). For each domain we rebuild the probe warm-up, the task subspace, and both detectors from that domain’s own prompts, none of them the teacher’s training system prompt; App. C.4 lists the exact prompts. Tab. 5 shows that WM-Basin’s detection power follows where the suspects were distilled: it ranks every math domain above both non-math ones in every cell, whereas the baseline ranks code above GSM8K at -gram=1 and 2. A hit therefore tells the verifier on which domain the suspect’s text carries that watermark, not only that it carries one; a low score elsewhere means nothing was found there, not that the model is clean.
Effect of the watermarked fraction and the training recipe. What in the adversary’s data and training recipe lets the watermark reach the student? At -gram=3 we sweep one knob at a time around the default recipe: the watermarked fraction of a fixed 1,900-trace training set from 1 to 100% (the remainder filled with clean traces from the same teacher), the distillation LoRA rank from 16 to 1,024, and 1, 2, or 3 training epochs. Figure 4 shows that detection power depends mostly on the watermarked fraction and the LoRA rank, and much less on how long the student trains. (1) WM-Basin performs at chance when at most 10% of the traces are watermarked and exceeds 90% TPR once 75% are. (2) Its TPR climbs from 16% at to 100% at . (3) A single epoch already yields 78% TPR. The black-box baseline stays at or below 2% at every fraction and reaches its highest value in the figure at the largest rank; WM-Basin outperforms it in every cell of the rank and epoch sweeps.
Figure 4. What governs detection: each knob swept on its own axis (-gram=3). Each panel varies one knob, with its value on the -axis and the TPR at a fixed 1% FPR on the -axis, for WM-Basin and the black-box baseline; the grey dashed line marks chance (TPR=FPR=1%). Each configuration trains 50 students, each with a different watermark key. Left: watermarked fraction of the training traces. Middle: LoRA rank . Right: training epochs. The all-clean control (no watermarked traces) is not plotted; none of its 50 students is flagged at any of the three thresholds.
Post-processing and Basin Dynamics
In § 2.2, we showed that a watermarked student sits in a narrow basin: a small step along the watermark direction (Eq. (1)) leaves it, while a much larger step along the task direction does not. We now examine in which direction standard post-processing moves the student, and whether it removes the watermark.
Effect of model compression. Can an adversary quantize or prune the watermark out of a deployed student? We apply int8, int4, or nf4 quantization, or 10 to 30% unstructured pruning, to the same 100 suspects. Table 6 shows that only int8 and 10% pruning leave a usable model, and both cost WM-Basin at most three points at any window and operating point. The other operations weaken detection but also leave the student unusable. The weight-space decomposition explains why (Table 16): every operation displaces the student at least three orders of magnitude further orthogonally to the watermark direction than along it. Compression therefore does not move the student out of the watermark basin, and it remains detectable.
| Compression | Acc. | WM-Basin TPR@FPR 5% | WM-Basin TPR@FPR 1% | WM-Basin TPR@FPR .1% | Black-box TPR@FPR 5% | Black-box TPR@FPR 1% | Black-box TPR@FPR .1% | Acc. | WM-Basin TPR@FPR 5% | WM-Basin TPR@FPR 1% | WM-Basin TPR@FPR .1% | Black-box TPR@FPR 5% | Black-box TPR@FPR 1% | Black-box TPR@FPR .1% | Acc. | WM-Basin TPR@FPR 5% | WM-Basin TPR@FPR 1% | WM-Basin TPR@FPR .1% | Black-box TPR@FPR 5% | Black-box TPR@FPR 1% | Black-box TPR@FPR .1% |
| \multicolumn{6}{c | }{-gram=1} | \multicolumn{6}{c | }{-gram=2} | \multicolumn{6}{c}{-gram=3} | |||||||||||||||||
| none | 70.7 | 100 | 100 | 100 | 81 | 50 | 17 | 70.7 | 100 | 100 | 99 | 44 | 19 | 5 | 70.7 | 100 | 95 | 80 | 8 | 4 | 2 |
| int8 | 72.2 | 100 | 100 | 100 | 75 | 50 | 17 | 71.0 | 100 | 100 | 99 | 50 | 27 | 6 | 70.0 | 100 | 99 | 85 | 24 | 7 | 1 |
| prune10 | 77.1 | 100 | 100 | 100 | 76 | 49 | 17 | 76.2 | 100 | 100 | 98 | 52 | 27 | 9 | 77.3 | 99 | 94 | 77 | 21 | 8 | 1 |
| prune20 | 59.5 | 100 | 100 | 100 | 70 | 44 | 20 | 61.2 | 100 | 96 | 82 | 44 | 24 | 10 | 57.6 | 91 | 82 | 49 | 20 | 4 | 0 |
| int4 | 50.4 | 79 | 59 | 31 | 50 | 25 | 11 | 50.8 | 43 | 25 | 12 | 38 | 16 | 2 | 45.4 | 46 | 25 | 10 | 7 | 1 | 0 |
| nf4 | 55.8 | 100 | 100 | 99 | 70 | 46 | 23 | 42.9 | 100 | 95 | 76 | 48 | 26 | 4 | 41.2 | 95 | 79 | 40 | 8 | 3 | 0 |
| prune30 | 37.8 | 93 | 71 | 36 | 52 | 30 | 8 | 39.6 | 75 | 51 | 12 | 32 | 13 | 1 | 28.9 | 49 | 26 | 7 | 14 | 1 | 0 |
Table 6. Table 6 Distillation detection under compression. Each row applies one operation to the same 100 distilled suspects and regenerates their text, holding the probe and the reference bank fixed. Cells are read as in Table 2; Acc. is mean GSM8K accuracy per window. Operations below the mid-rule leave an unusable model, with accuracy under 0.9× the uncompressed 70.7. The black-box cells of the uncompressed row predate the token matching used for the six operations, so compare the black-box columns only across operations. The weight-space displacement of every operation is in Table 16 and the setup in App. § 9.§ 1.
| Cond. | Orthogonal | Usable? | Acc. | Along | Acc. | Along | Acc. | Along |
| none | 0 | ✓ | 70.7 | 0 | 70.7 | 0 | 70.7 | 0 |
| int8 | 14.09 | ✓ | 72.2 | −0.0001 | 71.0 | −0.0002 | 70.0 | −0.0001 |
| int4 | 152.07 | ✗ | 50.4 | −0.0113 | 50.8 | −0.0112 | 45.4 | −0.0117 |
| nf4 | 125.91 | ✗ | 55.8 | −0.0291 | 42.9 | −0.0281 | 41.2 | −0.0266 |
| prune10 | 27.23 | ✓ | 77.1 | −0.0063 | 76.2 | −0.0061 | 77.3 | −0.0058 |
| prune20 | 77.74 | ✗ | 59.5 | −0.0390 | 61.2 | −0.0368 | 57.6 | −0.0348 |
| prune30 | 144.85 | ✗ | 37.8 | −0.1022 | 39.6 | −0.0945 | 28.9 | −0.0887 |
Table 16. Table 16 Compression moves almost entirely orthogonally to the watermark direction. Across 100 students per window, Acc. is mean GSM8K (%) and Usable? denotes Acc.≥0.9×70.7=63.6. We decompose into its signed projection along the watermark key direction (Eq. (1)) and the orthogonal remainder. The remainder is a property of the compressed weights alone and is identical at the three windows; the usability verdict is read off each window’s own Acc. and comes out the same in every row. Both are therefore printed once. The along component is at least three orders of magnitude smaller and increases in magnitude within the ordered pruning sweep. -gram=1, -gram=2, -gram=3.
Effect of continued SFT and RL. Does continued training on clean data move the distilled model out of its watermark basin? We continue training the suspects via SFT on cross-domain data and via GRPO (Shao et al., [49]) on verifiable tasks (details in App. § 9.§ 2), holding the probe and the reference bank fixed. Table 7 shows that continued RL leaves the student detectable while continued SFT weakens the signal: WM-Basin detects every distilled model in every GRPO arm at and every window, whereas both SFT arms substantially reduce its power. This ordering matches how far each update moves the model along the watermark direction: SFT on general reasoning traces moves it furthest, SFT on multilingual math less, and GRPO barely at all. The open-weight detector, teacher-forced on the distillation traces, nevertheless still detects the SFT-trained students. Continued SFT thus weakens the signal observable in the student’s outputs without moving the model out of the watermark basin.
| Cond. | Acc. | Along | WM-Basin | Black-box | Open-weight | ||||||
| TPR@FPR 5% | 1% | .1% | TPR@FPR 5% | 1% | .1% | TPR@FPR 5% | 1% | .1% | |||
| n-gram=1 | |||||||||||
| SFT (reason) | +4.8 | −0.252 | 3 | 1 | 0 | 39 | 15 | 3 | 92 | 82 | 52 |
| SFT (multi.) | +7.0 | −0.013 | 33 | 16 | 3 | 63 | 34 | 13 | 99 | 97 | 89 |
| GRPO (GSM8K) | +2.4 | 0.000 | 100 | 100 | 100 | 58 | 42 | 25 | 100 | 100 | 92 |
| GRPO (K&K) | +0.5 | 0.000 | 100 | 100 | 100 | 57 | 14 | 0 | 100 | 100 | 86 |
| GRPO (IFEval) | −1.4 | 0.000 | 100 | 100 | 100 | 71 | 57 | 14 | 100 | 100 | 86 |
| n-gram=2 | |||||||||||
| SFT (reason) | +1.0 | −0.182 | 4 | 2 | 0 | 22 | 9 | 1 | 99 | 97 | 76 |
| SFT (multi.) | +4.6 | −0.009 | 27 | 13 | 3 | 36 | 18 | 5 | 100 | 100 | 99 |
| GRPO (GSM8K) | +2.2 | 0.000 | 100 | 100 | 87 | 40 | 40 | 7 | 100 | 100 | 100 |
| GRPO (K&K) | +0.9 | 0.000 | 100 | 100 | 100 | 40 | 30 | 20 | 100 | 100 | 100 |
| GRPO (IFEval) | +1.0 | 0.000 | 100 | 100 | 100 | 30 | 30 | 10 | 100 | 100 | 100 |
| n-gram=3 | |||||||||||
| SFT (reason) | +2.4 | −0.146 | 6 | 1 | 0 | 7 | 3 | 1 | 98 | 95 | 73 |
| SFT (multi.) | +6.0 | −0.007 | 21 | 4 | 0 | 15 | 5 | 0 | 100 | 100 | 97 |
| GRPO (GSM8K) | +0.5 | 0.000 | 100 | 93 | 53 | 0 | 0 | 0 | 100 | 100 | 100 |
| GRPO (K&K) | −0.2 | 0.000 | 100 | 100 | 50 | 0 | 0 | 0 | 100 | 100 | 100 |
| GRPO (IFEval) | −0.3 | 0.000 | 100 | 100 | 50 | 10 | 0 | 0 | 100 | 100 | 100 |
Table 7. Table 7 Distillation detection under continued training. Each row continues training the same distilled suspects on clean data, holding the probe, the reference bank, and the generation settings fixed. Cells are read as in Table 2, one block per -gram window. Acc. is the paired per-key change in GSM8K accuracy, and Along is the signed projection of the weight change on the watermark direction (Eq. (1)). Open-weight ([46]) is teacher-forced on the adversary’s own distillation traces, an idealized read that our black-box verifier cannot run (App. § 9.§ 2). Cohort sizes and the full weight-space decomposition of every row are in Table 17.
Related Work
Model extraction and unauthorized distillation. Frontier providers, threat-intelligence groups, policy analysts, and security researchers now document distillation-oriented misuse in the wild: adversaries harvest reasoning traces at scale from deployed APIs to train competing models (Anthropic, [6]; Google Threat Intelligence Group, [18]; Panfilov et al., [36]). The underlying mechanism is knowledge distillation, which transfers capability from a teacher to a student model (Hinton et al., [23]). When
exposed through an API, it becomes model extraction: black-box query access alone, without the target’s parameters or training data, suffices to replicate the teacher’s functionality ([51]). The threat is sharpest for reasoning models, whose step-by-step traces provide dense supervision. Capability also transfers through synthetic instructions, model-generated reasoning, rubric or reward feedback, and agent trajectories ([53, 21, 20, 57]). We study supervised fine-tuning on collected reasoning traces, motivated by evidence that small, high-quality reasoning sets sharply improve students ([34]).
Defenses against extraction. Existing defenses restrict what the API returns, monitor extraction-like querying ([6, 18]), or make outputs less useful for training via antidistillation sampling or trace rewriting ([48, 31]). Such methods trade away utility for benign users or assume that the attacker stays observable during collection. That assumption is fragile in practice: adversaries spread collection across many accounts and third-party API resellers, so no single query stream looks like extraction ([6]). Detection is complementary: it acts after the fact and leaves the deployed model untouched. A parallel black-box line, dataset inference ([32]), resolves ownership by testing whether a suspect shares private knowledge of the victim’s data, without inserting a watermark; it observes that inserted watermarks are off-distribution and therefore not always preserved under stealing. Classical model watermarks instead hide a signal directly in the weights or via backdoors ([52, 2]), read from the parameters or by querying a trigger set; both assume the owner watermarked the suspect model under test, whereas our suspect is a model the owner never trained.
Watermarking and radioactivity. Text watermarking embeds a secret statistical signal into generated text ([26, 28, 47]). The idea of tracing data through a trained model originates in computer vision: Sablayrolles et al. [44] show that imperceptible “radioactive” watermarks in a dataset leave a statistical residue in models trained on it, detectable with a calibrated -value even when only 1% of the data is marked. Our rank test follows the same framework. Sander et al. [46] carry radioactivity over to language models, showing that watermarked distillation leaves a key-specific residue in the student, and others study how well such watermarks survive distillation ([47, 59, 35]). The black-box verifier of Sander et al. [46], which counts watermark-favored tokens in a suspect’s sampled text, defines the setting we work in and is, to the best of our knowledge, the only existing method for the task; it is our baseline throughout. Its limits were not previously characterized, and establishing them is part of our contribution. The statistic accumulates evidence only where exact watermarked contexts recur, which we show is fragile to wider windows, lower decoding temperatures, and larger text budgets (§ 3.2, App. § 8.§ 5), the same failure mode that paraphrase attacks exploit against surface-level watermark detection ([45]). WM-Basin keeps the access level and the generations, and replaces the token statistic with a weight-space one.
Weight-space geometry and basins. WM-Basin relies on the fact that students distilled under one key settle into a shared, key-specific region of weight space. This connects to the geometry of neural loss landscapes: independently trained optima are joined by simple, near-constant-loss curves ([17]), and once permutation symmetries are accounted for, the landscape resembles a single wide basin, so separately trained models can be aligned by permutation and merged in weight space ([4]). Large language models exhibit the same basin-like structure ([7, 30, 38]). WM-Basin treats the watermark key as carving out one such basin and asks whether a suspect has fallen into it.
Gradient attribution and representation post-processing. To read this geometric signal from text alone, WM-Basin backpropagates a suspect’s completions through a fixed probe model. This draws on first-order data attribution: influence functions trace a prediction back to the training points responsible for it using only gradient access ([27]), an idea scaled by gradient tracing ([41]) and random-projected attribution ([37]), and adapted to instruction tuning with low-rank projected gradients ([54]). Intuitively, a training example’s gradient captures how it would nudge the model, so aligned gradients indicate similar examples. Raw gradient cosines, however, are dominated by a few shared directions (generic language and formatting): a handful of top components inflate the similarity between otherwise unrelated items ([13, 16, 42, 11]). We remove these with an “All-but-the-Top” projection ([33]), stripping the few dominant yet uninformative directions so that the faint key-specific signal stands out from the generic task geometry.
Conclusion and Future Work
In this work, we introduce the watermark basin: distilling on watermarked traces encodes the teacher’s secret key as a faint direction in parameter space, displacing the student’s weights along that key direction and into a region that later training does not easily escape. Building on this geometry, we design WM-Basin, which reformulates black-box detection of unauthorized distillation from a fragile, surface-level token count into a geometric alignment test in parameter space. Across a wide range of evaluations, spanning six student families, four teachers, and four watermarking schemes, WM-Basin outperforms the existing baseline by a large margin, and it survives quantization, pruning, and RL post-training.
Scope and outlook. A watermark can be present in a distilled model but unreadable in its text: continued SFT on clean data suppresses the text-level signal while an open-weight probe still recovers the residual, and a supplementary study (App. § 9.§ 3) finds the same for rewriting the teacher’s traces before distillation. WM-Basin is also an in-domain test: a hit tells the verifier on which domain the suspect’s text carries the watermark, not only that it carries one, and a low score elsewhere means nothing was found there, not that the model is clean (§ 3.2). Both are limits on what a black-box verifier can observe, not on what distillation transfers. Narrowing the gap between the black-box read and the open-weight probe is thus a promising direction for future work.
Ethics Statement
This work studies whether a model provider can establish, from API access alone, that a suspect model was distilled from its watermarked teacher. It involves no human subjects and no personal or user data: the distillation corpus (OpenR1-Math-220k), the utility benchmarks (GSM8K and Math-500), and every teacher, student, probe, and clean-pool checkpoint are publicly released artefacts used under their own licenses, and every experiment runs on models we host ourselves. Because the method produces evidence of provenance, we report how it behaves on models that are not guilty rather than a bare score. We also measure what weakens the audit, including rewriting the teacher’s traces before distillation, continued fine-tuning, and probing away from the distillation domain. These are variants of attacks already described in the text-watermarking literature, and we report them because a verifier that does not know the limits of its own test is the more likely harm.
Reproducibility Statement
The configuration behind every number in the paper is recorded in App. § 7. Table 8 gives the default models, watermark and sampling settings, distillation data and LoRA hyperparameters, probe warm-up, and detector and baseline settings, down to the key ranges and seeds. Table 9 indexes every study, the factor it varies, the cohort and sample size, and where its result is reported, so a single arm is reproduced by reading the defaults plus the one row that changes them. The detector is specified procedurally in Alg. 1, and App. § 7.§ 2 documents trace generation and the quality filter that takes 5,000 sampled traces to the 1,900 used for training.
| Component | Setting |
| Models | |
| Teacher (default) | DeepSeek-R1-Distill-Qwen-14B |
| Teacher (swaps) | gpt-oss-120B, QwQ-32B, Llama-3.1-8B |
| Student (default) | Qwen2.5-3B |
| Student (cross-arch.) | 24 structures, 135M to 8.2B (Llama(-Guard), Qwen2.5/3, QwQ, DeepSeek-Distill, Gemma, SmolLM2) |
| Probe model | Qwen2.5-3B (base, warmed) |
| Clean PCA pool | 43 off-the-shelf checkpoints (15 families, 135M to 120B; 129 cells total; Table 10) |
| Watermark & teacher sampling | |
| Scheme | TextSeal, |
| Sampling | vLLM; , top-; tokens |
| Distillation temperatures | (reference fixed at ) |
| Distillation (student) | |
| Dataset | OpenR1-Math-220k |
| System prompt | one fixed math-teacher prompt for every trace; by default every verifier stage (warm-up, bank, clean pool, audit) renders this same prompt, so the default audit is prompt-matched. Only the domain sweep replaces it (App. § 8.§ 4) |
| Traces | 5,000 sampled → quality-filtered → 1,900 used for training |
| Trace filter | closing </think>, \boxed{}, no 3× repetition, math_verify match |
| Fine-tuning | LoRA , lr 2e−5, 3 epochs, effective batch 16, max sequence length 8192 |
| LoRA targets | |
| Eval benchmarks | GSM8K, Math-500 |
| Probe warm-up | |
| Data | 450 clean teacher traces, on build-pool prompts from the domain being audited (App. § 8.§ 4) |
| Adapter | LoRA , 4 epochs, lr 2e−4, batch 1 |
| Detector (WM-Basin) | |
| Reference bank | 100 deployed keys, one student distilled per key; each ranked against a 1,000-key bank ( decoys) |
| Per-key gradient | average of 512 traces (512 tokens) on build-pool prompts |
| Projection | fixed random projection to 8,192-dim (TRAK) |
| Task subspace | top- PCA (all-but-the-top), fit on clean pool |
| -value floor | 0.001 |
| Operating point | threshold (TPR@1% FPR) |
| Baseline (Sander et al., [46]) | |
| Black-box -gram | corpus from keyed teacher traces on 5,000 held-out prompts |
| Windows | -gram |
| Deduplication | each distinct (context, token) pair scored once across traces |
| Component | Setting |
| Test | baseline’s own analytic -value: two-key score fused at the generation , variance-corrected Gamma tail (App. § 7.§ 4); plain, not entropy-weighted |
| Suspect text (shared by every detector) | |
| Budget | 1,900 held-out prompts median 1,875 scored responses, 10M sampled tokens of which 1M are scored (App. § 7.§ 3) |
| Prompts | test pool , disjoint from and (App. § 7.§ 3) |
| System prompt | the distillation prompt above, i.e. matched; replaced only in App. § 8.§ 4 |
| Sampling | , top-, new tokens, repetition penalty 1.0, seed 42 |
Table 8. Default experimental configuration. Models, watermark, distillation, probe, and detector settings used throughout § 3 unless stated otherwise; see App. § 7.§ 2 for full details. The TextSeal parameter is held fixed in every experiment.
| Experiment | Varied factor and levels | Model cohort / sample size | Result |
| Student architecture | 24 structures, 135M to 8.2B parameters | 50 students per architecture (1,200 total) | Table 1 |
| Distillation temperature | Teacher sampling ; reference bank fixed at | 100 students per temperature | Table 2 |
| Teacher model | 4 teachers spanning 3 families, 8B to 120B; bank rebuilt per teacher; the three swapped teachers distil for 8 epochs, not 3 | 100 students per teacher | Table 3, Table 12 |
| Watermark scheme | 4 watermark schemes at -gram | 50 students per scheme and window | Table 4, Table 13 |
| Audited domain | 5 domains; every detector stage and the system prompt rebuilt per domain | Fixed pool of 100 students | Table 5 |
| Model compression | int8, int4 and nf4 quantization; 10 to 30% pruning | 100 students per operation | Table 6, Table 16; Figure 10 |
| Continued SFT and RL | Post-distillation training: SFT domain, or GRPO task | 100 students (SFT); 10 to 20 (GRPO) | Table 7, Table 17, Table 18 |
| Probe model | 10 probe checkpoints; reference bank rebuilt per probe | Fixed pool of 100 students | Table 14 (PDF p. 28) |
| Probe warm-up | Presence, corpus domain, LoRA rank, learning rate, epochs; one knob at a time, -gram=3 only | Fixed pool of 100 scored keys, reused across all 15 arms | Table 15 |
| Trace paraphrasing | Same-family Qwen paraphraser at 3 strengths | 20 students per strength | Table 19 |
| Watermarked data fraction | Proportion of watermarked traces, at a fixed 1,900-trace training budget | 50 students per fraction | Figure 4 |
| Student training recipe | LoRA rank , epochs | 50 students per configuration | Figure 4 |
| Distillation utility | Clean vs. watermarked teacher traces | 100 watermarked students per -gram window, plus a clean control | Figure 5 |
| Operating-point ROC | Detection threshold, at -gram | 100 watermarked students against 6,500 null cases | Figure 8 |
| Suspect-text budget | Cumulative prefixes of 5 to 1,878 generations (0.03 to 10.3M tokens supplied) | Fixed pool of 100 students | Figure 9 |
| Per-key false-positive rate | Which bank key is read as the deployed one, at and -gram | All 1,000 bank keys, each across 129 held-out clean model-temperature cells | Table 11 |
| Watermark basin geometry | Weight-space offset along and | One student distilled under a single key | Figure 3 |
| Task-subspace removal | Removed components ; seven settings, from no PCA through mean only to | 1,200 distilled students at all -grams, against 129 clean model-temperature cells | Figure 6 |
| Rank -value calibration | Significance threshold , swept under the null | 100 deployed keys across 65 held-out clean model-temperature cells | Figure 7 |
Table 9. Per-experiment settings. What each study varies, the cohort used, and where the result is reported; varied factors are read against the defaults of Table 8.
| Task | -gram=1 Pre | -gram=1 Post | -gram=1 | -gram=2 Pre | -gram=2 Post | -gram=2 | -gram=3 Pre | -gram=3 Post | -gram=3 |
| K&K | 10.0 | 14.4 | +4.4 | 9.0 | 13.4 | +4.4 | 10.1 | 15.2 | +5.1 |
| IFEval | 18.4 | 21.3 | +2.9 | 18.3 | 21.6 | +3.3 | 18.2 | 22.1 | +3.9 |
Table 18. Cross-domain GRPO improves the task it optimises. Pre, post and are task accuracy in percent on greedy held-out evaluation (400 K&K puzzles, 1,000 IFEval prompts). Each task and window pools 10 keys, except IFEval at -gram=3 (11). Counted over the three windows together, 10 keys improved on K&K and 10 on IFEval. The same students’ weight-space displacement and their off-target GSM8K and Math-500 changes are in Table 17.
The Key-Randomization Rank Test
Token-level detectors read significance from a parametric null for their score: a Gaussian or binomial tail for a green-list count [26, 46], or a Gamma tail for a Gumbel score [15], which is the form our black-box baseline uses (App. § 7.§ 4). WM-Basin assumes no such model for and reads significance from the rank of the claimed key among decoy keys drawn from the same generator [12, 40].
Let be the null hypothesis of § 2.1: the suspect’s outputs on the audited domain carry no information about the claimed key , as holds when the suspect was never trained within that key’s basin. To test it, we fix the suspect model, the prompts, and the detection pipeline, then draw independent decoy keys . We evaluate at and at each decoy, where a higher score means stronger basin alignment, and form the empirical -value of Eq. (4), which measures how uniquely the suspect aligns with relative to random basins. The leading counts among the candidates, which prevents and keeps Type-I error control valid under finite Monte Carlo sampling [40].
The -value is valid because nothing in the pipeline can favor the claimed key under . The claimed key and the decoys are drawn the same way, and the scoring rule uses a key only as an index that selects reference traces; the probe, the random projection, the build prompts, and the fitted are identical for all candidates. The claimed key is therefore equally likely to occupy any of the ranks, assuming no ties. Thresholding its rank -value then controls the FPR in finite samples: the FPR is at most for any threshold, and equals when lies on the attainable rank grid. The in Eq. (4) counts ties against rejection, so ties cost power rather than validity. None of this needs a parametric model of ([39]): Eq. (4) places no restriction on the score, so the guarantee survives arbitrary non-linear transformations of the suspect model. App. § 7.6 measures the realized rate against the nominal one for every deployed key.
The smallest attainable -value is , so the decoy count sets the finest significance an audit can certify: reporting at requires decoys. Decoy scoring is offline and reuses the build-once artefacts, so improving the resolution can be cheap in practice.
Experimental Setup
This section presents the experiment setup in § 3.
Configuration Summary
Table 8 gives in full the default models and numeric settings that § 3.1 states in summary, and Table 9 indexes the studies built on them, what each varies and where it is reported.
Distillation and Student Training
The student follows the fine-tuning recipe of Table 8. A teacher trace is retained only if it contains a closing </think> tag and a answer, has no substring repeated times under the sliding 100-character repetition check, and its extracted answer matches the gold answer under math_verify.
The probe is warmed up on clean teacher traces with LoRA.
Distillation performance. Watermarking the teacher does not cost the student the utility distillation buys. Figure 5 compares clean and TextSeal distillation under this recipe on GSM8K (Cobbe et al., [9]) and Math-500 (Hendrycks et al., [22]): distillation lifts both benchmarks well above the un-distilled base, and the watermarked students match the clean-distilled control at trace-filter pass rates within 8 points of it.
Figure 5. Watermarking preserves the utility gained through distillation. DeepSeek-R1-Distill-Qwen-14B (Guo et al., [21]) is distilled into Qwen2.5-3B (Yang et al., [56]) using clean traces or TextSeal traces at three -gram windows. The left panel reports the fraction of 5,000 traces passing the quality filter; the other panels report GSM8K and Math-500 accuracy. Watermarked bars average 100 keyed students with standard-deviation error bars, while No WM is one student trained on clean traces; dashed lines mark the un-distilled base.
Prompt Pools and Suspect Sampling
Section 2.1 splits the audited domain into disjoint training, build and test pools. In the following, we explain how we split the prompts in our evaluation.
The training pool is the adversary’s: OpenR1-Math-220k prompts sampled under the fixed math-teacher system prompt, yielding 5,000 traces, of which 40 to 48% pass the filter of App. § 7.§ 2 (Figure 5) and 1,900 become the distillation set.
The build pool is the verifier’s offline side and supplies the three artefacts WM-Basin is made of: the clean teacher traces that warm the probe, the 512 keyed traces averaged into each of the 1,000 reference directions, and the free generations of the 43 clean checkpoints that fit the task subspace. The baseline’s -gram corpus is a fourth offline artefact, drawn by construction from held-out teacher traces the suspect is never read on.
The test pool is 1,900 held-out prompts and is the only pool a verdict is ever read on. No prompt the adversary trained on is used to build either detector, and no prompt used to build either detector is used to score a suspect.
Detector and Baseline Implementation
This section gives the implementation behind the detector and baseline of § 3.1.
Reference key gradient bank and projection. For each candidate key, we generate teacher traces on the build pool (App. § 7.§ 3), discard responses below the minimum token length in Table 8, and average the surviving reference gradients into . A single fixed random projection, shared by all keys and applied unchanged to the suspect gradient so that every cosine rank test compares is taken in one coordinate space, approximately preserves their inner products (Johnson et al., [25]; Xia et al., [54]). We compute this with TRAK (Park et al., [37]). The deployed key and i.i.d. decoys form the bank used by the rank test in Eq. (4), and its size sets the reported -value floor.
Compute cost. Nearly all of the cost is one-time and suspect-independent. Warming the probe is one short LoRA pass (App. § 8.6). Building the bank dominates: each projected direction averages 512 reference traces at about 27 minutes of single-GPU time, so around 450 GPU-hours for the full 1,000-key bank, which occupies 31.2 MiB once projected. Both artefacts are reused by every subsequent audit and are rebuilt only when what they describe changes, namely the teacher or the probe for the bank, and the probe or the prompt distribution for the task subspace (App. § 7.§ 5). Auditing one suspect then costs sampling its text, the ≈1M sampled tokens of App. § 7.§ 3, a query cost identical for every detector, plus one backward pass and projection per completion; at the throughput above, the median 1,875-completion budget is roughly 1.6 GPU-hours on one device.
Sampled-token baselines. The baselines of Sander et al. [46] score a single token per position. In the open-weight setting, teacher-forced reading evaluates the student’s top-1 prediction on teacher-generated test text. Their black-box variant scores tokens sampled by the student. We score a token only if its preceding -gram appears in held-out teacher text generated with the key being tested. We exclude tokens whose preceding -gram also appears in the prompt. Following Sander et al. [46], both variants score distinct (context, token) pairs across traces. The open-weight variant additionally requires distinct contexts within each trace.
Seeing only text, the verifier re-tokenizes each suspect response with the teacher’s tokenizer and scores every position of that stream: the context -gram and the scored token both come from the suspect’s own tokens, with no teacher forcing and no student logits. TextSeal’s secret is a key pair , and the generator draws its Gumbel noise from with probability and from otherwise, so no token reveals which key drove it. We score both and weight them by that generation prior,
at the deployed . Here is the keyed pseudorandom value of the scored token: a watermarked teacher prefers tokens with large , so a radioactive student produces positions where is larger than chance. Under each and the two keys are independent, so but . Matching over the retained positions to a Gamma of mean and variance (shape , scale ) gives
with the upper regularized incomplete gamma function. The -value is the probability that a clean model accumulates a total score of at least over positions, and is the same evidence expressed in standard deviations above the null mean.
The three other watermark schemes in App. § 8.§ 3 are read on the same principle: GumbelMax is the single-key case, where leaves , whereas KGW and SynthID score each position with a keyed-vocabulary indicator and a tournament -value rather than a Gumbel variate.
Removing Shared Task Directions
Section 2.3 projects the shared task subspace out of every gradient before scoring; this section gives how that subspace is fit and how many components we remove.
The idea of removing the shared task direction is aligned with removing the shared common vector for the word embeddings studied in Mu et al. [33]. Following their postprocessing, we fit PCA to gradients from the clean checkpoint pool and keep its top principal directions as the shared task basis , then project and every onto the orthogonal complement of before computing the cosine similarity score in stage (iii). The number of removed directions, , is the one free parameter of this procedure.
What the clean pool is. Table 10 lists all models used for removing the task direction: 43 checkpoints are off-the-shelf, never-distilled causal LMs spanning 15 pretraining families and 135M to 120B parameters. None has been fine-tuned on the audited domain and none has ever been exposed to a watermark key. Every checkpoint contributes one cell per decoding temperature (0.7, 1.0, 1.3), giving the 129 cells of Table 8. A cell free-generates on the build pool , and its gradients are taken at the same probe model , so a clean cell and a suspect differ only in the text they produced. For fixed-threshold TPR evaluation, clean models are not evaluation models, so the PCA fit uses all 129 cells.
| Family | Checkpoints | Ckpts. |
| Qwen | Qwen2.5-0.5B, -0.5B-Instruct, -1.5B, -1.5B-Instruct, -3B, -3B-Instruct, -7B-Instruct, -Math-7B; Qwen3-0.6B, -1.7B, -4B, -4B-Instruct, -8B, -8B-Base, -32B; QwQ-32B; Qwen1.5-MoE-A2.7B | 17 |
| Llama | Llama-2-7B, Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, Llama-Guard-3-1B, Llama-Guard-3-8B | 5 |
| DeepSeek | DeepSeek-R1-Distill-Qwen-1.5B, -7B, -14B{sup}†`; DeepSeek-MoE-16B-Base | 4 |
| SmolLM2 | SmolLM2-135M, -135M-Instruct, -1.7B | 3 |
| Gemma | Gemma-2-2B, Gemma-3-1B-PT | 2 |
| Phi | Phi-3.5-mini-Instruct, Phi-4 | 2 |
| OLMo | OLMo-2-1B, OLMoE-1B-7B | 2 |
| Mistral | Mistral-7B-Instruct | 1 |
| Yi | Yi-1.5-9B-Chat | 1 |
| Falcon | Falcon3-7B-Instruct | 1 |
| Granite | Granite-3.1-8B-Instruct | 1 |
| StarCoder | StarCoder2-7B | 1 |
| Nemotron | Nemotron-Mini-4B | 1 |
| Moonlight | Moonlight-16B-A3B | 1 |
| gpt-oss | gpt-oss-120B | 1 |
| 15 families | 43 |
Table 10. The clean PCA pool. The 43 off-the-shelf checkpoints whose gradients fit the shared task subspace of App. § 7.§ 5, grouped by pretraining lineage. Each model generates on 1,000 build-pool prompts at each of three decoding temperatures (0.7, 1.0, 1.3).
The ROC curves and the per-key calibration curves use the balanced subset, i.e., 43 checkpoints at three temperatures split into disjoint fit and evaluation halves.
How many components to remove. We choose on clean models alone, and we read the false-positive rate of each single deployed key. Figure 6 sweeps from no PCA to =100 and reads three keys of the 1,000-key bank against TPR: the typical key, the key at the 95th percentile of the per-key rate in terms of FPR, and the worst key which has the highest FPR. Without the projection those three disagree completely, the typical key firing on no clean cell at all while the 95th-percentile key fires on 38 to 44% of them. Removing =100 directions closes that spread onto the nominal line and lifts TPR at =5% from 70.2 to 91.7 at -gram=1. A small does neither, because the leading directions carry watermark signal as well as shared task structure: the projection pays for itself only from 10, after which power climbs monotonically. We deploy =100, and App. § 7.6 reports the per-key calibration that remains at that setting.

Figure 6. Choosing : detection power against three order statistics of the key bank. The projection sweeps from no PCA through mean only (subtract the pool mean, remove no subspace) to =100 removed components; the path visits the seven settings in that order and marker shade runs light to dark with . Columns are the -gram window. Rows are which key of the 1,000-key bank the false-positive rate is read off: the typical key, the 95th-percentile tail key and the worst key. : TPR at =5% over the 1,200 distilled students. : the percentage of the 129 clean model-temperature cells on which that key fires at the same threshold.
Calibration of the Rank -Value
By the exchangeability argument of Appendix § 6, thresholding at should falsely accuse an fraction of models that were not distilled from the protected teacher, which is what § 3.1 relies on when it reads operating points off the nominal -value. We test it on 100 deployed keys and 65 clean model-temperature cells held out from the 64 used to fit the task subspace (Appendix § 7.§ 5), ranking each key among the full 1,000-key bank. Figure 7 plots the empirical rate against for each deployed key, for WM-Basin (Figure 7a) and for the black-box baseline (Figure 7b).
Figure 7. Nominal -value thresholds match empirical false-positive rates. Each faint curve is one of the 100 deployed keys, reporting its empirical FPR over the 65 held-out clean model-temperature cells at -gram=3 as the threshold varies; the solid curve is the worst key and the dashed line is the calibrated target FPR=.
The per-key false-positive distribution. Table 11 reports the empirical false-positive rate at =5% over the 1,000 keys of the bank, each read across the 129 clean model-temperature cells, after the task-subspace projection and under the same 4-fold cross-fitting by base model as Appendix § 7.§ 5. The typical key is calibrated and the tail is not, which is the distinction the mean over the bank hides: the median key sits at 4.3% against a nominal 5% in all three windows and the 90th percentile at 9.9%, while the worst key reaches 23.4 to 26.2%. That tail is bounded by the size of the pool: the floor column is the level a perfectly exchangeable bank of this size reaches on its own (Appendix § 7.§ 5), and against it the worst observed key is high by the worst / floor factor, where the unprojected scores are high by 6.
| -gram | median | p90 | p95 | worst | floor | worst / floor |
| 1 | 4.3 | 9.9 | 12.1 | 23.4 | 13.5 | 1.74 |
| 2 | 4.3 | 9.9 | 12.8 | 25.5 | 13.5 | 1.89 |
| 3 | 4.3 | 9.9 | 12.1 | 26.2 | 13.5 | 1.95 |
Table 11.
Table 11 Per-key false-positive rate at , after removing the task subspace. Each of the 1,000 bank keys is read over the 129 clean model-temperature cells, with 4-fold projectors cross-fitted by base model so no cell is scored by a subspace fit on its own model. All values are percentages except the last column. Floor is the 95th percentile of the maximum under exact uniformity at over 129 cells: the level a perfectly exchangeable bank of this size would still reach, and the reference the worst key should be judged against.
Additional Detection Results
This section follows the sweep order of § 3.2, adding the operating points and factors the main tables leave out, together with two studies the main text states only in passing: the suspect-text budget behind the fragility claim of § 1, and the choice of probe model, together with the warm-up it is given, named in the detector setup of § 3.1. The configuration of App. B is the reference point throughout, and each row of Table 9 names what it varies.
Operating Points
Figure 8 sweeps the detection threshold continuously over the same distillation arm that Table 2 reads at three fixed thresholds, on an empirical false-positive axis rather than that table’s nominal -values. Both detectors are scored on one shared set of students and innocent cells here, all 100 keys rather than the per-window cohorts the tables report, so the comparison to make is between the two curves rather than between a curve and a cell of Table 2: WM-Basin sits at or above the black-box baseline throughout the plotted range at every window, and strictly above it everywhere below the false-positive rate at which both saturate. The baseline’s curve is the read that preceded the token matching both tables now use; that matching lifts eight of the nine black-box cells of Table 2 at 5% FPR, by up to 18 points, so the separation plotted here is wider than the matched one.

Figure 8. Detection performance of the two closed-model detectors across all operating points. ROC (log , linear ), one panel per -gram window, matched on the 100 distillation keys 1001 to 1100. TPR is over the 100 watermarked students (one per key, scored with its own key); FPR is over the same 100 keys × the 65 held-out innocent model-temperature cells (the evaluation half of the 129-cell clean pool: 43 off-the-shelf base models × 3 sampling temperatures; 6,500 null cases). Sweeping the threshold traces each curve; WM-Basin (ours) is drawn in vermilion and Black-box (baseline) in blue. The grey dashed line marks chance (TPR=FPR).
Cross-Teacher Detection
§ 3.2 reports cross-teacher detection in Table 3, whose three thresholds are empirical false-positive rates. The sampled-token baseline can only accumulate evidence at positions whose context window recurs, and student diversity orders it: gpt-oss students repeat only 65.3% of their 2-word contexts and the baseline reaches 18% there, against 92.1% and 93% on the most repetitive Llama students. Neither tokenizer sharing, trace length nor utility separates the teachers the same way (Table 12), and gpt-oss gives the second-largest GSM8K gain alongside the weakest sampled-token result. WM-Basin aggregates the watermark’s gradient direction over all positions instead, and stays at 96 to 100% in the 5% column on every teacher.
| gpt-oss-120B | QwQ-32B | Llama-3.1-8B | |
| Teacher | |||
| Family / tokenizer | MoE / 200k native | Qwen2.5 / DS tok | Llama / native |
| Median trace chars | 10,552 | 23,095 | 2,969 |
| Student utility | |||
| GSM8K (%) | 77.2 ± 1.7 | 80.8 ± 2.7 | 74.9 ± 1.2 |
| Δ GSM8K | +28.7 | +32.3 | +26.4 |
| Math-500 (%) | 47.9 ± 1.5 | 52.1 ± 1.5 | 36.5 ± 1.5 |
| Δ Math-500 | +14.1 | +18.3 | +2.7 |
| Detection, TPR at the 5% threshold | |||
| WM-Basin | 96 | 99 | 100 |
| Black-box | 18 | 88 | 93 |
Table 12. Teacher properties, student utility and detection, for the three swapped-in teachers. is the DeepSeek-R1-Distill tokenizer. Accuracy is mean ± standard deviation over 100 keyed Qwen2.5-3B students, and Δ is against the un-distilled base (GSM8K 48.5%, Math-500 33.8%). The two detection rows are the FPR=5% column of Table 3.
Watermark Scheme
Table 4 replaces the teacher’s TextSeal decoder with three other watermarking schemes. TextSeal, the dual-key scheme used everywhere else in the paper ([47]), draws each token’s Gumbel noise from one of two keys with probability ; GumbelMax ([1]) is the single-key Gumbel scheme; and SynthID ([10]) resolves each position through a tournament of depth 10 between keyed -values. KGW ([26]) instead adds a fixed logit bias to a keyed half of the vocabulary () at every position, the one scheme here that modifies the teacher’s logits rather than the sampling rule. Each is generated at -gram=1, 2 and 3 and distilled into 50 students, with all detectors reading the same 1M-token budget, the reference bank rebuilt per scheme and window, and everything else at the defaults of Table 8.
| Watermark | \multicolumn{6}{c}{n-gram=1} | \multicolumn{6}{c}{n-gram=2} | \multicolumn{6}{c}{n-gram=3} | |||||||||||||||
| WM-Basin 5% | WM-Basin 1% | WM-Basin .1% | Black-box 5% | Black-box 1% | Black-box .1% | WM-Basin 5% | WM-Basin 1% | WM-Basin .1% | Black-box 5% | Black-box 1% | Black-box .1% | WM-Basin 5% | WM-Basin 1% | WM-Basin .1% | Black-box 5% | Black-box 1% | Black-box .1% | |
| GumbelMax | 100 | 100 | 100 | 72 | 44 | 28 | 100 | 98 | 98 | 52 | 32 | 6 | 100 | 100 | 100 | 22 | 10 | 0 |
| KGW | 100 | 100 | 100 | 58 | 26 | 8 | 94 | 86 | 56 | 40 | 20 | 4 | 90 | 68 | 34 | 16 | 2 | 2 |
| SynthID | 100 | 100 | 100 | 64 | 38 | 8 | 100 | 100 | 98 | 44 | 16 | 0 | 100 | 96 | 78 | 24 | 8 | 4 |
Table 4. Distillation detection across watermarking schemes. The teacher’s default TextSeal decoder is replaced by each of three other schemes, with the detector’s reference bank rebuilt to match. Cells are read as in Table 2, over 50 distilled models per scheme and window.
Does WM-Basin’s margin hold when the baseline is read in its strongest form? The black-box statistic can weight each scored token by the teacher’s entropy at that position, concentrating the test on the high-entropy positions where the watermark is able to bias the sample at all, as in [47]; Table 13 gives the baseline that weighted read as a second row per scheme, beside the plain statistic reported in Table 4. The weighting is worth 8 to 30 points to the baseline at across all twelve scheme and window cells, yet WM-Basin still leads at every level in every cell where that level is defined. Its margin in the main body is therefore not an artefact of reading the baseline in its plain form.
| Watermark | -gram=1 WM-Basin TPR@FPR 5% 1% .1% | -gram=1 Black-box TPR@FPR 5% 1% .1% | -gram=2 WM-Basin TPR@FPR 5% 1% .1% | -gram=2 Black-box TPR@FPR 5% 1% .1% | -gram=3 WM-Basin TPR@FPR 5% 1% .1% | -gram=3 Black-box TPR@FPR 5% 1% .1% |
| TextSeal | 100 100 — | 76 44 28 | 100 100 98 | 52 30 6 | 100 98 84 | 22 8 2 |
| TextSeal with entropy | 96 88 60 | 74 54 40 | 46 24 2 | |||
| GumbelMax | 100 100 100 | 72 44 28 | 100 98 98 | 52 32 6 | 100 100 100 | 22 10 0 |
| GumbelMax with entropy | 96 88 62 | 80 54 28 | 46 18 6 | |||
| KGW | 100 100 100 | 58 26 8 | 94 86 56 | 40 20 4 | 90 68 34 | 16 2 2 |
| KGW with entropy | 66 46 28 | 54 32 8 | 38 10 0 | |||
| SynthID | 100 100 100 | 64 38 8 | 100 100 98 | 44 16 0 | 100 96 78 | 24 8 4 |
| SynthID with entropy | 94 68 38 | 66 38 12 | 42 22 6 |
Table 13. Distillation detection across watermarking schemes, with the baseline’s entropy-weighted statistic. The full form of Table 4: TextSeal, the teacher’s default scheme, joins the three swapped-in ones, and each watermark occupies two rows, the black-box baseline [46] reading the suspect text with its plain statistic on the first and with its entropy-weighted statistic on the second. The weighting belongs to that statistic, so one WM-Basin run serves both rows and its cells span the pair. Cells are read as in Table 4, and the dash marks a level the 991-key TextSeal bank at -gram=1 cannot resolve.
Audited Domain and System Prompt
The audited-domain sweep of § 3.2 reads Table 5; this section gives the per-domain rebuild and the exact prompts behind it. Every row only uses the prompt from the audit domain: the probe warm-up corpus, the PCA task subspace removal, the decoy bank, and the black-box baseline’s per-key corpus filter are all built from that row’s own questions and system prompt, never the teacher’s math-teacher prompt, so neither detector inherits an advantage from the distillation domain.
| Audited domain | -gram=1 | -gram=2 | -gram=3 | |||||||||||||||
| WM-Basin | Black-box | WM-Basin | Black-box | WM-Basin | Black-box | |||||||||||||
| TPR@FPR | TPR@FPR | TPR@FPR | TPR@FPR | TPR@FPR | TPR@FPR | |||||||||||||
| 5% | 1% | .1% | 5% | 1% | .1% | 5% | 1% | .1% | 5% | 1% | .1% | 5% | 1% | .1% | 5% | 1% | .1% | |
| OpenR1-Math | 100 | 100 | 96 | 76 | 49 | 19 | 98 | 92 | 69 | 51 | 24 | 9 | 95 | 83 | 45 | 26 | 12 | 2 |
| MATH | 91 | 74 | 56 | 74 | 49 | 20 | 71 | 48 | 15 | 60 | 28 | 7 | 65 | 38 | 9 | 24 | 7 | 2 |
| GSM8K | 29 | 14 | 3 | 35 | 13 | 2 | 18 | 7 | 1 | 16 | 9 | 1 | 22 | 10 | 1 | 7 | 3 | 0 |
| Code | 12 | 5 | 0 | 50 | 19 | 4 | 5 | 1 | 0 | 31 | 11 | 2 | 8 | 4 | 0 | 7 | 2 | 0 |
| Chat | 11 | 1 | 0 | 22 | 4 | 1 | 2 | 0 | 0 | 15 | 4 | 0 | 4 | 1 | 0 | 13 | 3 | 1 |
Table 5. Distillation detection across the audited domain. The same 100 suspects, all distilled on OpenR1-Math (Hugging Face, [24]), are audited on three math datasets from the distillation domain (top block) and two domains outside it (bottom block), with both detectors built entirely from the audited domain’s own prompts and system prompt. Cells are read as in Tab. 1. Detailed setup in App. C.4.
Each suspect is ranked against its own key plus the 999 decoys of that row’s bank, and 100 students are scored in every cell. All five rows read that row’s own held-out test pool, the three math-dataset rows under the shared math_neutral system prompt and the other two under their own, so wording is constant inside the top block and the dataset alone separates those rows. On OpenR1-Math, the distilled dataset itself with held-out questions read under a prompt no suspect trained under, WM-Basin reaches 95%/83%/45% when context window is 1/2/3, so a new system prompt does not by itself cost the verifier its read.
| Training domain (math): what the students actually learned on | |
| You are a math teacher. You will be given a math problem and you will solve it step by step. You will output your final solution like . Be sure to include relevant units within the brackets and fully evaluate arithmetic expressions. | |
| math_neutral (OOD, closest to training) | |
| You are a helpful assistant. Solve the following problem step by step, then give the final answer. | |
| code (OOD) | |
| You are an expert programmer. Read the problem and write a complete, correct Python solution. Think step by step, then provide the final code. | |
| chat (OOD, farthest from training) | |
| You are a helpful, honest, and courteous assistant. Follow the user’s instruction and give a clear, complete response. |
Table 20.
Suspect-Text Budget
How much suspect text does a verdict need, and does reading more always help? § 3.1 fixes the default budget in the in-domain tables are read at. We read the same fixed pool of 100 students, at cumulative prefixes of their generations, from 5 to 1,878 generations (0.03 to 10.3M tokens supplied), holding the probe, bank, task subspace and threshold fixed, and score both detectors on every prefix at 1% FPR (Figure 9). WM-Basin accumulates evidence monotonically at every window, so more text never costs the verifier anything and no stopping rule is needed: it reaches 100% TPR by 0.6M tokens at -gram 1 and 2, and at -gram=3 it is within 8 points of its ceiling by that budget and spends the twenty times more text that follows gaining the rest. The baseline is not monotone in the budget at all: it peaks within the first ≈0.5M tokens and decays afterwards, because its scored-token denominator keeps growing while deduplication caps the signal it can collect.
Figure 9. Impact of suspect text budget on detection performance. We plot True Positive Rate at 1% FPR (TPR@1%; share of distilled models detected at rank-≤0.01, one model per key) against the amount of suspect text evaluated (log scale, million tokens). Both WM-Basin (ours) and Black-box ([46]) are evaluated on identical generations at the same 1% FPR operating point.
Probe Model and Warm-Up
The probe model turns text into a gradient signal. In this section, we study the impact of the probe model on the performance of WM-Basin.
Choice of probe model. Does the verifier need a probe model from the suspect’s own family? Table 14 (PDF p. 28) varies only the probe that § 3.1 holds at a clean Qwen2.5-3B base, across ten checkpoints from 0.5B to 8B spanning four Qwen2.5 scales or post-training states, three Llama-3 checkpoints, and two families whose tokenizers sit farthest from the student’s 152k vocabulary. Every probe receives the same warm-up, and because each defines a different gradient coordinate space its reference bank is rebuilt with the default recipe; the suspect cohort, clean PCA pool and three -gram windows stay fixed. Family costs the detector nothing: every probe recovers all suspects at -gram=1, and at -gram=3 and 0.1% FPR the three Llama-3 probes land inside or just under the band the Qwen probes span, despite sharing no tokenizer with the student. The loss instead tracks how far the probe’s vocabulary sits from the student’s, the smallest vocabulary losing most while still staying an order of magnitude above the baseline in the same cell, and falling further behind the Qwen probes as the threshold tightens at -gram=2 and 3. The verifier can therefore audit with whatever checkpoint it holds, preferring a nearby vocabulary.
| Probe model | -gram=1 | -gram=2 | -gram=3 | ||||||
| 5% | 1% | .1% | 5% | 1% | .1% | 5% | 1% | .1% | |
| Black-box baseline (no ) | 72 | 34 | 11 | 44 | 28 | 9 | 26 | 13 | 3 |
| Qwen2.5-3B-Base (default) | 100 | 100 | 100 | 100 | 100 | 98 | 100 | 99 | 89 |
| Qwen2.5-1.5B-Base | 100 | 100 | 100 | 100 | 100 | 99 | 97 | 90 | 72 |
| Qwen2.5-0.5B-Base | 100 | 100 | 100 | 100 | 99 | 97 | 99 | 96 | 81 |
| Qwen2.5-3B-Instruct | 100 | 100 | 100 | 100 | 100 | 98 | 100 | 99 | 84 |
| Qwen2.5-7B-Instruct | 100 | 100 | 100 | 100 | 100 | 96 | 96 | 87 | 69 |
| Llama-3.1-8B-Instruct (128k vocab) | 100 | 100 | 100 | 100 | 100 | 92 | 100 | 92 | 67 |
| Llama-Guard-3-8B (128k vocab) | 100 | 100 | 100 | 100 | 100 | 94 | 97 | 91 | 69 |
| Llama-3.2-1B-Instruct (128k vocab) | 100 | 100 | 100 | 100 | 100 | 93 | 99 | 88 | 72 |
| Gemma-2-2B-Base (256k vocab) | 100 | 100 | 100 | 100 | 99 | 91 | 91 | 74 | 57 |
| SmolLM2-1.7B-Base (49k vocab) | 100 | 100 | 100 | 100 | 97 | 71 | 90 | 68 | 34 |
Table 14 (PDF p. 28). Effect of the probe model on detection. The same 100 distilled suspects are held fixed while the verifier changes the probe model and rebuilds the reference bank. Probes vary in scale, post-training, family, and tokenizer; the Black-box baseline [46] uses no probe and is token-matched, reading exactly the same suspect text as WM-Basin. Each cell reports True Positive Rate (TPR, %) at FPR (5%, 1%, 0.1%), over the suspects whose generations both detectors can score, 87/95/90 of the 100 keys at -gram 1/2/3. At -gram=3 and 0.1% FPR the five Qwen2.5 probes span 69 to 89 and the three Llama-3 probes 67 to 72; vocabulary size is given beside each cross-family row.
Impact of warm-up. The probe warm-up of § 3.1 is one short LoRA pass on clean teacher traces, run before any key gradient is taken; Table 15 sweeps it one knob at a time from the default, in a single scoring pass per arm. We report -gram=3 because it is the only window that separates the arms: at -gram=1 every probe in Table 14 (PDF p. 28) already reads 100 at all three levels.
| Probe warm-up | WM-Basin | Probe warm-up | WM-Basin | ||||
| 5% | 1% | .1% | 5% | 1% | .1% | ||
| (a) Warm-up presence | (c) Adapter rank | ||||||
| Trained warm-up (default) | 100 | 99 | 89 | Rank 32 | 100 | 100 | 89 |
| Untrained random adapter | 82 | 59 | 33 | Rank 128 (default) | 100 | 99 | 89 |
| No adapter (bare base model) | 30 | 21 | 6 | Rank 512 | 100 | 100 | 86 |
| (b) Warm-up corpus | (d) Learning rate | ||||||
| OpenR1-Math (default) | 100 | 99 | 89 | 5e−5 | 100 | 98 | 90 |
| MATH | 100 | 97 | 76 | 2e−4 (default) | 100 | 99 | 89 |
| GSM8K | 99 | 96 | 68 | 5e−4 | 100 | 100 | 93 |
| MMLU | 99 | 97 | 69 | 1e−3 (diverged) | 7 | 2 | 0 |
| Chat | 99 | 94 | 78 | (e) Warm-up epochs | |||
| Code | 88 | 73 | 37 | 1 epoch | 100 | 99 | 92 |
| 4 epochs (default) | 100 | 99 | 89 | ||||
| 8 epochs | 100 | 97 | 81 |
Table 15. The probe warm-up is necessary, but almost none of its details are. All rows are WM-Basin at -gram=3 on the same 90 watermark keys, the same suspects, and the same suspect-text budget; only the probe warm-up changes, one knob at a time from the default. Cells are TPR (%) at 5%, 1%, and 0.1% FPR, scored on the task-neutral residual gradients of § 2.4. Defaults in Table 8.
Removing the warm-up costs most of the detector’s power at 0.1% FPR: an untrained, randomly initialised adapter keeps about a third of the default’s TPR there, and the bare base model almost none. Gradients are taken with respect to the adapter parameters (§ 2.4), so a random adapter still defines a usable coordinate space and retains real signal at 5% FPR, whereas the bare model has no adapter to differentiate against. The warm-up therefore sharpens a signal the probe already carries rather than creating one.
Warming on prompts from different domains and datasets, i.e., chat, MMLU, GSM8K or MATH instead of the audited OpenR1-Math distribution costs 11 to 21 points at 0.1% FPR and at most one key at 5%. Code is the single exception (37%), and it is the warm-up corpus farthest from the reasoning traces the reference bank is built from. This is a separate question from where the verifier draws its detection prompts, which do have to come from the audited domain or a similar one (§ 3.2).
Adapter rank and learning rate each move TPR at 0.1% FPR by at most 4 points, and epoch count by 11; none of the three moves the 5% column at all. Only divergence is fatal: at learning rate 1e−3 the warm-up run diverges and detection collapses at every level, a training failure the verifier can see without running the detector at all. The cheapest arms we ran are among the best at every level, so the setting to deploy is whatever converges: rank 32 matches the default at 0.1% FPR and beats it at 1%, and one epoch beats it at 0.1%. We did not run the two together, so rank 32 at one epoch is an inference from two single-knob arms rather than a measured configuration.
Additional Robustness Results
This section gives the protocols and weight-space analyses behind § 3.3, followed by a robustness analysis against a paraphrasing attack that rewrites the teacher’s traces before distillation. Each modifies student or the traces it was distilled on and re-reads it with the same probe and reference bank, at the defaults of App. § 7 unless this section or a caption records otherwise.
Compression
This section gives the protocol behind Table 6 and the weight-space displacement of every operation. We apply per-output-channel symmetric int8, uniform group-64 int4, nf4, or 10 to 30% unstructured magnitude pruning to 100 keyed students at each -gram window, regenerating detector inputs after each operation. Quantized weights are dequantized to fp16 for evaluation, and pruning excludes lm_head and embed_tokens. A model counts as usable when its mean GSM8K accuracy is at least 0.9× the distilled baseline, which int8 and prune10 clear and int4, nf4, prune20 and prune30 do not. The none row of Table 6 is the uncompressed reference rather than a compression condition, so its black-box cells were never re-read under the token matching the six operation rows use; that row therefore supports no black-box comparison, either with its own WM-Basin cells or down the column.
Compression costs the watermark only where it costs the model. Int8 and prune10 retain both utility and strong WM-Basin detection, and the larger detection losses occur only beyond the usability gate, with nf4 and prune20 the two operations where the gradient signal outlasts utility (Figure 10 at -gram=3; Table 6 and Table 16 at all three windows). The weight-space displacement is ordered the same way: the along component of every compression operation stays below 0.11 against orthogonal movement of 14 to 152, and grows with the pruning fraction. Therefore, compression does not move the model outside the watermark basin, and detection remains effective.
Figure 10. Robustness to post-processing. TextSeal, -gram=3, 100 keyed students under int8, int4 and nf4 quantization and 10/20/30% pruning; base is uncompressed. (a) TPR at 1% FPR for WM-Basin and Black-box; (b) gsm8k accuracy, dotted line is the 0.9× usable gate; (c) weight-space move along the watermark direction vs orthogonal residual. Details in Table 16.
Continued Fine-Tuning and RL
This section gives the training recipes behind Table 7 and the weight-space displacement of every arm. We continue training the keyed Qwen2.5-3B students of each -gram window on clean data. We then regenerate suspect text and re-measure detection, utility and geometry. Continued SFT uses one of two clean corpora. The reason corpus is 1,900 traces, 950 code and 950 science, from open-r1/Mixture-of-Thoughts ([24]). The multilingual corpus is 1,206 traces on math questions in eight languages ([8]). Both pass the think-closure and repetition filters of App. § 7.§ 2.
SFT keeps the distillation adapter shape. It runs 2 epochs at global batch 16 and maximum sequence length 8192, on a cosine learning rate of with 0.05 warmup. Continued RL uses GRPO ([49]). The in-domain arm optimises gsm8k with a verifiable correctness reward. The two cross-domain arms use learning rate on rule-verified K&K logic puzzles ([55]) or on RLVR-IFEval instruction following ([29]). Both cross-domain rewards are all-or-nothing: K&K requires every inhabitant to be labeled correctly in the final boxed answer, and IFEval requires the requested constraint to hold. We measure task accuracy greedily on 400 held-out puzzles. and 1,000 held-out prompts. Detector inputs keep the standard , top- sampling.
For every condition we decompose against , the key-specific watermark direction of Eq. (1). The Open-weight column (Sander et al., [46]) is teacher-forced on the adversary’s own distillation traces. That is the ideal case, and not a read our verifier can run. It shows that those traces stay memorised after continued training, not that the watermark survives in the weights.
Which of these updates removes the watermark? One of the two SFT arms does, and no RL arm does (Table 7). The reason arm puts WM-Basin at its wrong-key floor at all three windows. The multilingual arm only degrades it, and part of the cohort is still recovered at 5% FPR. WM-Basin stays flat against each GRPO cohort’s pre-attack detection at 5% FPR, at every window. The Open-weight column stays at 86 or above in 41 of its 45 cells, and its four exceptions are all in the reason arm. The same detector scored on held-out prompts falls by 41 to 87 points. Every update is nearly orthogonal to , SFT and GRPO alike. Its projection on the watermark direction is at most 5.3% of its length under SFT and 0.0% under every GRPO arm (Table 17). The watermark component of the weights is left in place, and under GRPO this is the text read. Continued SFT changes what the student writes instead. WM-Basin and the black-box baseline score that text alone, so both fall on the reason arm at 5% FPR.
GRPO preserves utility with far smaller weight movement than SFT. Every row is paired against its own cohort’s pre-attack students, whose accuracy is printed as base: the cohorts differ in size () and do not share a baseline, so is the mean per-key change and not the difference of the two pooled columns beside it. GSM8K and Math-500 are percentages and is in points. Move (total) is , Along is its signed projection on the key direction , and Along/total is the absolute percentage share. GRPO projections are under 0.0005 in magnitude and print as 0.000. The SFT rows share the full 100-key pool as their base; the GRPO rows are smaller cohorts carrying their own, which run from 0.6 points above it to 2.8 below, which is why Table 7 reports accuracy as a paired change rather than an absolute.
| Cond. | -gram | GSM8K base | GSM8K post | GSM8K | Math-500 base | Math-500 post | Math-500 | Move total | Move along | along/total | |
| reason | 1 | 100 | 67.2 | 72.0 | +4.77 | 41.9 | 40.8 | −1.11 | 4.80 | −0.252 | 5.3% |
| reason | 2 | 100 | 69.0 | 70.0 | +0.96 | 43.0 | 39.4 | −3.60 | 4.70 | −0.182 | 3.9% |
| reason | 3 | 100 | 69.0 | 71.3 | +2.37 | 43.2 | 40.6 | −2.62 | 4.66 | −0.146 | 3.1% |
| multilingual | 1 | 100 | 67.2 | 74.2 | +6.98 | 41.9 | 46.9 | +4.99 | 2.72 | −0.013 | 0.5% |
| multilingual | 2 | 100 | 69.0 | 73.6 | +4.59 | 43.0 | 46.7 | +3.68 | 2.71 | −0.009 | 0.3% |
| multilingual | 3 | 100 | 69.0 | 75.0 | +6.03 | 43.2 | 47.5 | +4.29 | 2.70 | −0.007 | 0.3% |
| rl (gsm8k, GRPO) | 1 | 20 | 67.8 | 70.1 | +2.35 | 41.3 | 42.2 | +0.90 | 0.051 | 0.000 | 0.0% |
| rl (gsm8k, GRPO) | 2 | 20 | 69.3 | 71.5 | +2.17 | 42.5 | 43.2 | +0.74 | 0.049 | 0.000 | 0.0% |
| rl (gsm8k, GRPO) | 3 | 20 | 68.0 | 68.5 | +0.52 | 43.4 | 44.0 | +0.67 | 0.050 | 0.000 | 0.0% |
| rl (K&K, GRPO) | 1 | 10 | 66.4 | 66.9 | +0.48 | 40.9 | 42.4 | +1.44 | 0.228 | 0.000 | 0.0% |
| rl (K&K, GRPO) | 2 | 10 | 67.8 | 68.7 | +0.93 | 42.9 | 43.4 | +0.54 | 0.226 | 0.000 | 0.0% |
| rl (K&K, GRPO) | 3 | 10 | 66.2 | 66.0 | −0.20 | 43.7 | 44.8 | +1.04 | 0.241 | 0.000 | 0.0% |
| rl (IFEval, GRPO) | 1 | 10 | 66.4 | 65.0 | −1.42 | 40.9 | 40.3 | −0.58 | 0.242 | 0.000 | 0.0% |
| rl (IFEval, GRPO) | 2 | 10 | 67.8 | 68.7 | +0.96 | 42.9 | 42.1 | −0.76 | 0.251 | 0.000 | 0.0% |
| rl (IFEval, GRPO) | 3 | 10 | 66.3 | 66.1 | −0.26 | 43.3 | 41.6 | −1.71 | 0.246 | 0.000 | 0.0% |
Table 17.
Paraphrasing
We paraphrase only the reasoning body between <think> and </think>, preserving the tags and the final answer, then reapply the standard quality filter before distillation. A same-family (Qwen) paraphraser rewrites the traces under mild, moderate and aggressive instructions at , and respectively, with top- and at most 8,192 generated tokens. Each setting trains 20 keyed students. Rewriting costs the adversary no capability: mean GSM8K accuracy exceeds 77% at every strength and window against 68 to 69% for the unmodified control, and every strength stays above its matched clean student in the paired per-key comparison.
Each -gram window is a block of Table 19. Mild rewriting leaves the watermark readable from text: WM-Basin holds 100, 100 and 90% TPR at 5% FPR at -gram=1, 2 and 3, against 47, 20 and 10% for the black-box baseline. Moderate and aggressive rewriting bring both text detectors down together, to at most 33% TPR at 5% FPR for WM-Basin and 15% for the baseline. What is left then depends on which traces the open-weight read [46] is teacher-forced on. On the original teacher traces, the set a verifier actually holds, it falls with the text detectors (53, 55 and 55% at 5% FPR under moderate rewriting). On the rewritten traces the student trained on, it recovers the moderate arm almost in full (87, 100 and 100% at 5% FPR, and 95% at 1% FPR at the two longer windows). The aggressive arm is gone under either read, at best 33 to 55% at 5% FPR. Moderate rewriting therefore leaves the watermark in the weights and only hides it from the traces the verifier can obtain.
| Strength | WM-Basin TPR@FPR 5% | WM-Basin TPR@FPR 1% | WM-Basin TPR@FPR .1% | Black-box TPR@FPR 5% | Black-box TPR@FPR 1% | Black-box TPR@FPR .1% | Open-weight orig. 5% | Open-weight orig. 1% | Open-weight orig. .1% | Open-weight ideal 5% | Open-weight ideal 1% | Open-weight ideal .1% |
| -gram=1 (15 of 20 keys; strictest column is 0.101% FPR) | ||||||||||||
| none | 100 | 100 | 100 | 60 | 20 | 0 | 100 | 93 | 93 | 100 | 93 | 93 |
| mild | 100 | 100 | 100 | 47 | 13 | 0 | 93 | 87 | 60 | 100 | 100 | 100 |
| moderate | 33 | 0 | 0 | 7 | 0 | 0 | 53 | 13 | 0 | 87 | 73 | 20 |
| aggressive | 13 | 0 | 0 | 7 | 0 | 0 | 7 | 7 | 0 | 33 | 13 | 7 |
| -gram=2 | ||||||||||||
| none | 100 | 100 | 100 | 60 | 25 | 5 | 100 | 100 | 100 | 100 | 100 | 100 |
| mild | 100 | 85 | 65 | 20 | 10 | 5 | 100 | 100 | 95 | 100 | 100 | 100 |
| moderate | 15 | 15 | 5 | 15 | 5 | 5 | 55 | 20 | 0 | 100 | 95 | 85 |
| aggressive | 15 | 10 | 5 | 5 | 5 | 0 | 5 | 5 | 0 | 55 | 40 | 10 |
| -gram=3 | ||||||||||||
| none | 100 | 100 | 85 | 35 | 10 | 0 | 100 | 100 | 100 | 100 | 100 | 100 |
| mild | 90 | 70 | 25 | 10 | 5 | 0 | 100 | 100 | 95 | 100 | 100 | 100 |
| moderate | 5 | 0 | 0 | 5 | 0 | 0 | 55 | 25 | 15 | 100 | 95 | 80 |
| aggressive | 0 | 0 | 0 | 0 | 0 | 0 | 10 | 0 | 0 | 55 | 25 | 10 |
Table 19. Effect of paraphrasing teacher traces before distillation, at all three -gram windows. Every watermarked trace is rewritten at one of three strengths with a same-family (Qwen) paraphraser; none is the unmodified control, repeated in each block because each window is a separate training set. WM-Basin (ours) and Black-box (Sander et al., [46]) score text, while Open-weight (Sander et al., [46]) teacher-forces the original teacher traces (orig.) or the rewritten traces the student actually trained on (ideal), the latter an upper bound that assumes the verifier already holds the adversary’s rewritten training set. Each cell reports TPR (%) at FPR (5%, 1%, 0.1%) over 20 students, except at -gram=1, which is scored over the 15 of 20 keys whose reference direction at that window is clean and whose strictest column is 0.101% FPR, the finest rate that bank resolves.
References
- [1]Scott Aaronson. My AI safety lecture for UT effective altruism, 2022. Blog post. Accessed: 2026-03-17.
- [2]Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In 27th USENIX security symposium (USENIX Security 18), pages 1615–1631, 2018.arxiv.org/abs/1802.04633
- [3]Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025.arxiv.org/abs/2508.10925
- [4]Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836, 2022.arxiv.org/abs/2209.04836
- [6]Anthropic. How Claude marks AI-generated content, 2026b. Accessed: 2026-08-13.
- [7]Huanran Chen, Zeming Wei, Yao Huang, Yichi Zhang, Yinpeng Dong, and Jun Zhu. Unveiling the basin-like loss landscape in large language models. In International Conference on Learning Representations, pages 148112–148146, 2026.arxiv.org/abs/2505.17646
- [8]Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7001–7016, 2024.
- [9]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
- [10]Sumanth Dhathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, et al. Scalable watermarking for identifying large language model outputs. Nature, 634(8035):818–823, 2024.
- [11]Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. Improving zero-shot learning by mitigating the hubness problem. arXiv preprint arXiv:1412.6568, 2014.DOI
- [12]Meyer Dwass. Modified randomization tests for nonparametric hypotheses. The Annals of Mathematical Statistics, pages 181–187, 1957.DOI
- [13]Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 55–65, 2019.arxiv.org/abs/1909.00512
- [14]European Commission. Code of Practice on Transparency of AI-Generated Content, 2026.
- [15]Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three bricks to consolidate watermarks for large language models. In 2023 IEEE international workshop on information forensics and security (WIFS), pages 1–6. IEEE, 2023.arxiv.org/abs/2308.00113
- [16]Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009, 2019.arxiv.org/abs/1907.12009
- [17]Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1802.10026
- [18]Google Threat Intelligence Group. GTIG AI threat tracker: Distillation, experimentation, and (continued) integration of AI for adversarial use, 2026. Accessed: 2026-02-26.
- [20]Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746, 2025.arxiv.org/abs/2507.17746
- [21]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.DOI
- [22]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
- [23]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
- [24]Hugging Face. Open r1: A fully open reproduction of deepseek-r1, 2025.
- [25]William B Johnson, Joram Lindenstrauss, et al. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189-206):1, 1984.
- [26]John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In International conference on machine learning, pages 17061–17084. PMLR, 2023.arxiv.org/abs/2301.10226
- [27]Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. Pmlr, 2017.arxiv.org/abs/1703.04730
- [28]Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free watermarks for language models. arXiv preprint arXiv:2307.15593, 2023.arxiv.org/abs/2307.15593
- [29]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024.arxiv.org/abs/2411.15124
- [30]Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1712.09913
- [31]Xinhang Ma, William Yeoh, Ning Zhang, and Yevgeniy Vorobeychik. Protecting language models against unauthorized distillation through trace rewriting. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11307–11324, 2026.
- [32]Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. arXiv preprint arXiv:2104.10706, 2021.arxiv.org/abs/2104.10706
- [33]Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. arXiv preprint arXiv:1702.01417, 2017.arxiv.org/abs/1702.01417
- [34]Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannah Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20286–20332, 2025.arxiv.org/abs/2501.19393
- [35]Leyi Pan, Aiwei Liu, Shiyu Huang, Yijian Lu, Xuming Hu, Lijie Wen, Irwin King, and Philip S Yu. Can llm watermarks robustly prevent unauthorized knowledge distillation? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13228–13251, 2025.arxiv.org/abs/2502.11598
- [36]Alexander Panfilov, David Schmotz, Ilia Shumailov, Luca Beurer-Kellner, Joachim Schaeffer, Ameya Prabhu, Jonas Geiping, and Maksym Andriushchenko. Stealing reasoning traces from proprietary llm apis. arXiv preprint arXiv:2608.09867, 2026.arxiv.org/abs/2608.09867
- [37]Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186, 2023.arxiv.org/abs/2303.14186
- [39]Fortunato Pesarin and Luigi Salmaso. Permutation tests for complex data: theory, applications and software. John Wiley & Sons, 2010.
- [40]Belinda Phipson and Gordon K Smyth. Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn. arXiv preprint arXiv:1603.05766, 2016.DOI
- [41]Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in neural information processing systems, 33:19920–19930, 2020.arxiv.org/abs/2002.08484
- [42]Milos Radovanovic, Alexandros Nanopoulos, and Mirjana Ivanovic. Hubs in space: Popular nearest neighbors in high-dimensional data. Journal of machine learning research, 11(sept):2487–2531, 2010.
- [43]Daniel Remler and Ben Hayum. Adversarial distillation. Center for a New American Security (CNAS) Report, 2026. Accessed: 2026-08-01.
- [44]Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Radioactive data: tracing through training. In International Conference on Machine Learning, pages 8326–8335. PMLR, 2020.
- [45]Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023.arxiv.org/abs/2303.11156
- [46]Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze, and Teddy Furon. Watermarking makes language models radioactive. Advances in Neural Information Processing Systems, 37:21079–21113, 2024.arxiv.org/abs/2402.14904
- [47]Tom Sander, Hongyan Chang, Tomáš Souček, Tuan Tran, Valeriu Lacatusu, Sylvestre-Alvise Rebuffi, Alexandre Mourachko, Surya Parimi, Christophe Ropers, Rashel Moritz, et al. Textseal: A localized llm watermark for provenance & distillation protection. arXiv preprint arXiv:2605.12456, 2026.arxiv.org/abs/2605.12456
- [48]Yash Savani, Asher Trockman, Zhili Feng, Yixuan Xu, Avi Schwarzschild, Alexander Robey, Marc Finzi, and Zico Kolter. Antidistillation sampling. Advances in Neural Information Processing Systems, 38:117800–117827, 2026.arxiv.org/abs/2504.13146
- [49]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [51]Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction {APIs}. In 25th USENIX security symposium (USENIX Security 16), pages 601–618, 2016.arxiv.org/abs/1609.02943
- [52]Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin’ichi Satoh. Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM on international conference on multimedia retrieval, pages 269–277, 2017.DOI
- [53]Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 13484–13508, 2023.arxiv.org/abs/2212.10560
- [54]Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024.arxiv.org/abs/2402.04333
- [55]Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 2742–2785, 2025.
- [56]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuanchang Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.arxiv.org/abs/2412.15115
- [57]Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3053–3077, 2024.arxiv.org/abs/2310.12823
- [58]Tingwei Zhang, John X Morris, and Vitaly Shmatikov. How to steal reasoning without reasoning traces. arXiv preprint arXiv:2603.07267, 2026.arxiv.org/abs/2603.07267
- [59]Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Distillation-resistant watermarking for model protection in nlp. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5044–5055, 2022.arxiv.org/abs/2210.03312





