WorksComputer Science
The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device Use
Abstract
Open large language models (LLMs) and speech models ship vocabularies of up to 262k tokens, and every user downloads the tables those tokens index (the input embedding, an untied output head and, in recent on-device models, per-layer embeddings), whatever languages they need. We call the bytes in rows a deployment never uses the vocabulary tax and measure it, without downloading weights, for 1,244 generative and speech-recognition models on the Hugging Face Hub. For a Thai and English deployment it is a median 4.4% of model bytes below 0.5B parameters and 2.1% above 8B, and 1.93 GB in Gemma 4 E2B: it falls hardest on the small models that edge devices run. Keeping only the rows of the deployment's writing systems is exact. Keeping the tokens seen in the user's own text removes much more, a median 29.0% below 0.5B, but the usual way of doing it breaks BPE tokenizers: it drops the intermediate tokens that merges build on, so new text is split differently 1.9 times as often as the standard check reports. We repair this with merge closure, at a cost of 5.9% more rows, and add a trim certificate: a Good-Turing estimate, from the trimming text alone, of how much new text the trimmed tokenizer will split differently (correlation 0.996 over 300 settings in 22 languages) and of how much more text a target needs. Without test data our method never exceeds a 2% target, while prior keep rules checked on test data exceed it in 9-13 of 44 settings. On six models of 270M to 3B parameters in 11 languages, trimmed models keep their predictions: where the certificate passes, a median 89.7% of rows is removed and bits per character rise by at most 2.0%. The census, the trimming and the certificate are released as an open Python package, and the per-model tax for 41 languages as an open dataset.
Playground: https://joessattes.github.io/ai-model-explorer/
Code: https://github.com/JoesSattes/ai-model-explorer
Data: huggingface.co/datasets/JoesSattes/ai-vocabulary-tax
Python: pip install ai-model-explorer AI Model Explorer
Introduction
A team that deploys an open model to a phone, a call-center server or a kiosk usually serves a handful of languages. The model it downloads was trained for many more. Its vocabulary covers dozens of writing systems, and every table indexed by the vocabulary has one row per piece: the input embedding, the output head when it is not tied to the embedding, and, in recent on-device models such as Gemma 3n and Gemma 4, per-layer embedding tables that hold nearly half of the model’s parameters. A Thai and English deployment of Gemma 4 E2B stores 1.93 GB of rows that no Thai or English text can ever select. We call these bytes the vocabulary tax.
The tax is easy to state and has not been measured. Work on tokenizer fairness measures how many tokens a sentence costs in each language [1, 16, 19], a cost in compute and context length. Work on vocabulary trimming removes unused rows from particular models and shows that accuracy on the kept languages is preserved [4, 10, 22] [25]. Neither says how large the memory cost is across the models people download, who pays it, or how a user can tell, before deploying, whether a trimmed model will receive their text as the original did.
We answer these questions by building on our census of 12,974 Hugging Face repositories, which reads each model’s anatomy from its weight headers alone [21]. In doing so we find that the standard corpus-based way of trimming is unsound for the tokenizers most open models now use. Trimming by corpus keeps the vocabulary pieces that appear as tokens in a sample of target text. In byte-pair encoding (BPE) [20], a piece is produced only by merging two smaller pieces, and many of these smaller pieces never appear as tokens because they are always merged further. A corpus rule removes them, and every piece built on them becomes unreachable: its row is kept but can never be selected, and the words it covered fall apart into smaller pieces than the trimming intended. The damage is invisible to the usual check, which counts the tokens of held-out text whose rows were removed.
The contributions are as follows.
A weight-free census of the vocabulary tax for 1,244 generative and speech-recognition models (§ 5.1). The tax is regressive: the median falls from 4.4% of model bytes below 0.5B parameters to 2.1% above 8B (). Across tokenizer families it trades against the compute tax of long token sequences.
An exact keep rule by writing system and the conditions that make it exact, including Latin pieces, without which 3.5–6.6% of Thai tokens are lost (§ 5.2).
Merge closure (§ 5.3). Across 22 languages and four BPE families, corpus trimming leaves a median of 828 kept rows (5.0%) unreachable, and the real share of new text that is split differently is a median 1.9 times the nominal share, in 95 of 101 measurable settings more than 1.2 times; sequences grow by up to 62%. Adding the merge parents of every kept piece costs a median 5.9% more rows and brings the real rate to 0.94 of the nominal one. Against four keep rules of prior work, prior rules re-split 1.30–1.32 times as much new text at the same compression, and ours removes the most rows at a 1% target (§ 5.4).
A trim certificate (§ 5.7): a Good–Turing estimate of the real re-split rate of new text and an extrapolation of how much text would bring it under a target, both from the trimming text alone, validated against rebuilt tokenizers; its error is 4.5–6.6%, against 23–27% for hold-out and cross-validation estimates.
A 22-language study (§ 5.5): every language can remove about the same share of bytes, but the text needed to do so safely differs by two orders of magnitude, ordered by how much of the language’s vocabulary it shares with English. The certificate also holds per language in mixed-language deployments (median ratio 1.03; § 5.6), and we quantify how it degrades under domain shift and how to guard against it.
A quality evaluation of trimmed models (§ 5.8): six edge-sized models, 11 languages, bits per character, greedy continuations and Belebele. The script rule leaves the models’ predictions unchanged up to renormalization; under the corpus rule, the loss in bits per character is about 2.5 times the re-split rate, so the certificate, computed before any model is run, also predicts the quality cost.
We also report how many edge-sized models fall under common memory budgets once trimmed (§ 5.9). The measurement, the merge-closed trimming and the certificate are part of the open Python package ai-model-explorer, and the corpus rule with its certificate also runs in the browser on the user’s own text, without uploading it. The tax of 2,061 Hub models for 41 languages, the tokenizer-family results, the certificate and merge-closure measurements and the quality evaluation are released as a dataset (https://huggingface.co/datasets/JoeSSattes/ai-vocabulary-tax).
Related work
Vocabulary trimming. Ushio et al. [25] remove the vocabulary entries of a multilingual model that do not appear in a target-language corpus and show that accuracy on the target language is retained without retraining; their models use Unigram vocabularies, which have no merges. Bogoychev et al. [4] compare a script-based and a corpus-based selection of sub-vocabularies at inference time, the two rules we study, and measure memory and speed gains for several models. SqueezeBits [22] report latency gains of 15–21% for Gemma 3 1B. Gurgurov et al. [10] keep the most frequent pieces of a target-language corpus as part of compressing multilingual encoders, COMPACT [14] removes rare tokens together with the merge rules that produce them, alongside channel pruning, VocabTailor [?] selects an output vocabulary per task and input, and Ben Shoham [3] trim the vocabulary of a speculative-decoding draft model. These works establish, on the models they chose, that trimming preserves accuracy when the kept set covers the target text. None examines whether a frequency-selected set of BPE pieces can still be produced by the rebuilt tokenizer, and none tells a user, without held-out data, whether their text is enough. Under-trained tokens are related: Land and Bartolo [15] find that intermediate BPE pieces are often under-trained because they rarely occur as tokens, the same property that makes a corpus rule remove them.
Tokenization cost across languages. Petrov et al. [16] and Ahia et al. [1] show that the same content costs up to an order of magnitude more tokens in some languages than in English, and Rust et al. [19] relate the quality of a multilingual tokenizer for a language to downstream performance. This is the compute side of the cost of a multilingual vocabulary. We measure the memory side and show that the two trade against each other across tokenizer families (Figure 2).
Figure 2. The memory tax (rows a Thai and English deployment never uses) against the compute tax (Thai tokens per English token for the same sentence) for 19 tokenizer families. Dot area: models in P1 using the family; grey: none.
Estimating unseen mass. The Good–Turing estimator [7, 9] estimates the probability of events not seen in a sample from the number of events seen once. Heaps’ law [11] describes how the number of distinct types grows with sample size. We apply both to a question they have not been used for: how much of a deployment’s future text will fall outside a vocabulary chosen from a sample of that text.
Reading models without weights. The tensor manifest of a safetensors file [12] records the name, shape and storage type of every tensor. Our previous work recovers a model’s anatomy from the manifest alone and reports a census of 12,974 Hub repositories [21]; we take the token tables and model sizes of this paper from that census.
Problem and method
The vocabulary tax
A model stores token tables , each with rows (the vocabulary size) and bytes. They are found in the manifest as the tensors whose leading dimension equals and that are used as an input embedding, an untied output head, or a per-layer embedding [21]. A deployment for a set of languages needs a set of rows. The vocabulary tax of for is
where is the stored size of the model and is the number of pieces in the tokenizer file. Rows past the end of the tokenizer are kept: they may be padding or special tokens that the file does not list. The tax depends on the keep rule that defines .
Two keep rules
Script rule. Each piece is decoded to text (undoing byte-level encoding and the SentencePiece word marker). A piece is kept if every character in it has a Unicode script [24] in the scripts of , or in the Common or Inherited scripts, or is a format character (category Cf), and pieces that are special tokens, byte fallbacks, digits or punctuation are always kept. Latin is added to every by default because names, numbers, code and loanwords appear in Latin letters in the text of every language. Under this rule, any text written in the scripts of is tokenized into kept pieces only, so it is tokenized exactly as before, provided the tokenizer is rebuilt consistently (below). Each of the inclusions repairs a failure we observed: the Japanese length mark choon (U+30FC) and the Arabic tatweel (U+0640) belong to the Common script, combining marks to Inherited, and zero-width spaces to Cf.
Corpus rule. Given lines of target text , keep a piece if the script rule keeps it and, in addition, it is a special token, it is a single character, or it occurs at least times in the tokenization of (we use unless stated). The single-character pieces guarantee that any word can still be spelled, so no text fails to encode. Pieces seen fewer than times are removed, and any later word that needed them is split into smaller kept pieces, which changes the model’s input. We call the share of tokens in new text that fall in removed rows the re-split rate; it is zero for the script rule on text in ’s scripts.
Rebuilding the tokenizer and merge closure
Trimming rows is only safe if the rebuilt tokenizer maps text to the intended pieces. For BPE tokenizers [17, 20] we remove the vocabulary entries of removed pieces and every merge whose inputs or output were removed; for SentencePiece models [13] we remove the pieces from the model protocol buffer. Kept pieces are re-indexed in their original order, and a list of kept ids maps new ids to original rows, so the token tables are sliced with one indexing operation.
A BPE piece is created by one merge of two pieces, its merge parents, and the tokenizer can produce only if it can produce and . A keep set is merge-closed if it contains the parents of each of its pieces. The script rule is closed by construction for whole characters, since the parents of a piece are substrings of it and have the same scripts; the corpus rule is not, because it keeps pieces by their frequency as tokens, and the parents of a frequent piece are often never tokens themselves. In a set that is not closed, a kept piece whose parent was removed is an orphan: its row is never selected, and every word that used it is split into smaller pieces. We therefore close every keep set: starting from , we add the parents of each kept piece, recursively. The parents are read from the merge list of tokenizer.json and GGUF files, and for tiktoken and Tekken vocabularies, which store ranks instead of merges, from the last merge that BPE performs when it encodes the piece by itself.
The trim certificate
Under the corpus rule, the user needs to know the real re-split rate of text they have not yet seen: the share of the tokens the original tokenizer would produce that the trimmed tokenizer does not reproduce. Holding out part of measures it, but costs text and, as we show, overestimates the rate of the final keep set, which is built from all of . Instead we estimate it from itself. Let be the number of tokens in the tokenization of , and the number of distinct candidate pieces (kept by the script rule, not special and longer than one character) that occur exactly times. The corpus rule removes the candidates seen fewer than times. By the Good–Turing estimate [9], the probability that a new token is a candidate seen times in is , Algorithm 1 Merge-closed corpus trimming with a certificate
Require: tokenizer with pieces , languages , text ( lines), minimum count , target
▷ specials, bytes, Common, Inherited, Cf always kept
piece counts of ;
repeat add the merge parents of every to until no change ▷ merge closure
over candidates of ▷ Eq. (2)
the same on the first lines;
if ▷ Eq. (3)
rebuild on (drop merges touching removed pieces); kept ids in original order
verify: test sentences of tokenize identically under the script rule
return , kept ids, , ,

Figure 1. Pipeline. The tax is computed from the manifest and the tokenizer without weights. The corpus rule refines the script rule with the user’s text, and the certificate estimates, from that text alone, how much new text the trimmed tokenizer will split differently.
so the re-split rate of new text from the same source is estimated by
To say how much text would reach a target rate , we compute on the first half and on all of ( lines), fit , which is the form Heaps’ law [11] gives for a Zipfian sample, and report
The certificate reports , whether it is below (default 1%), and when it is not. It is computed in the time the tokenization of takes. Algorithm 1 summarizes the corpus rule with merge closure and the certificate.
Experimental setup
Models. We start from the 12,974 repositories of the Hub census of Singkul [21], whose token tables and stored sizes come from manifests. We assign each repository’s tokenizer to one of 24 reference tokenizer families by model type and vocabulary size, using the vocabulary files distributed with llama.cpp [8], the Whisper assets [18], and the o200k_base and cl100k_base encodings used by GPT-OSS and OLMo. This maps 2,171 repositories. Of these, 1,244 are generative or speech-recognition models (pipeline tags text generation, image-text-to-text, any-to-any and automatic speech recognition); we call them population P1. P1 is recent: 55% of its models were created in 2025 or 2026, including Gemma 4, Qwen3, Ministral 3, GPT-OSS and OLMo 3. The newest families whose tokenizers are distributed only through the Hub (Qwen3.5 and later, Nemotron-H, GLM-5, DeepSeek-V3.2 and V4) are not yet mapped; they account for most of the downloads of models created in 2026, and the package computes their tax directly from their own tokenizer files. Population P2 is the 909 models of P1 whose tokenizer belongs to one of the four BPE families we rebuild exactly from their merge lists (Gemma, Qwen, Llama 3 and GPT-2), 73% of P1. Edge-sized models are those with fewer than 4B parameters.
Text. For the script rule we use the 21 languages of the Parallel Universal Dependencies treebanks [5], 1,000 parallel sentences each. For the corpus rule we take, for each of 22 languages, the largest Universal Dependencies treebank with sentence text (CC BY-SA or similar licenses), shuffle it, keep the first 1,000 sentences as held-out text and use the rest as trimming text; for Thai we add the Wisesight social-media corpus [23]. We use up to 15,000 trimming lines per language. Held-out text comes from the same source as the trimming text, which is the situation a user who trims for their own domain is in.
Baselines. We compare against the keep rules of prior work, each run on the same text and rebuilt the same way: the script rule of Bogoychev et al. [4] and SqueezeBits [22]; rank top-K, which keeps the first pieces of the vocabulary ( = 10k–80k); frequency top-N, which keeps the most frequent pieces of the target text as in Bogoychev et al. [4], Gurgurov et al. [10] and the vocabulary part of COMPACT [14] ( = 1k–16k); and seen in corpus, the rule of Ushio et al. [25], with minimum counts = 1–10. All rules keep special, byte and single-character pieces. Our method is the corpus rule with merge closure, with chosen by the certificate. For the certificate we compare against estimating the re-split rate on a 20% hold-out of the trimming text and by 5-fold cross-validation, and for the extrapolation against fixed decay exponents.
Model quality. We evaluate six edge-sized models with different tokenizers: Gemma 3 270M and 1B, Qwen3 0.6B and 1.7B, Llama 3.2 1B and SmolLM3 3B, the last two sharing one tokenizer. For each of 11 languages (Thai, Chinese, Japanese, Korean, Arabic, Hindi, Russian, German, French, Spanish, English) we compare the original model with the script rule and with the merge-closed corpus rule on 4,000 and 15,000 trimming lines (all available lines where a treebank has fewer: 3,997 for Chinese, 7,100 for Japanese, 6,526 for Arabic, 14,525 for English; Chinese has no 15,000-line condition). A trimmed model is evaluated exactly without editing weights: the trimmed tokenizer’s ids are mapped to the original rows and every softmax runs over the kept rows only, which is the same function as a model whose embedding, head and per-layer tables are sliced. We measure bits per character on 300 held-out lines, which compares models with different tokenizations; the share of half-lines whose 24-token greedy continuation matches the original model’s exactly; and zero-shot Belebele accuracy [2] on 200 questions, choosing the answer with the lowest negative log-likelihood per character.
Tokenizers and the real re-split rate. We rebuild the reference tokenizers with the Hugging Face tokenizers and sentencepiece libraries and check them against llama.cpp. For every trimmed keep set we rebuild the trimmed tokenizer and tokenize the held-out text with both; the real re-split rate is the share of original tokens whose piece and character span do not appear in the trimmed tokenization, and the nominal rate is the share of original tokens whose row was removed, the check used in prior work. Statistics are medians with 95% bootstrap confidence intervals [6] where intervals are shown.
Results
The tax is regressive and falls on small models
Table 1 lists the 24 tokenizer families. Between 0.5% (GPT-2) and 56% (Baichuan) of their rows are unused by Thai and English text. Families with large multilingual vocabularies carry the largest tax; families with small, English-centered vocabularies carry almost none, but split Thai into many more tokens. Figure 2 plots the two costs against each other. GPT-2, Falcon and GPT-NeoX need 4.6–8.3 times as many tokens for Thai as for English and waste almost no memory; Gemma and XLM-R need 1.1–1.5 times as many tokens and leave 35–52% of their rows unused, and the newest large vocabulary in the census, GPT-OSS’s o200k, sits between them (1.8 times, 27%). No family in the census avoids both: a Thai user pays a compute tax or a memory tax, and the tokenizer decides which.
| Family | Vocabulary | Rows unused (%) | Token ratio TH/EN | Median tax (%) | |
| Baichuan | 64,000 | 55.8 | 3.86 | 0 | – |
| Aquila | 100,008 | 53.2 | 4.66 | 0 | – |
| XLM-R | 250,048 | 51.9 | 1.06 | 0 | – |
| Gemma 3/3n/4 | 262,144 | 35.0 | 1.53 | 170 | 5.3 |
| Qwen 2–3.5 | 151,936 | 28.8 | 2.36 | 496 | 4.1 |
| Command R | 256,000 | 28.1 | 4.31 | 1 | 1.7 |
| Mistral Tekken | 131,072 | 27.2 | 2.21 | 26 | 2.9 |
| GPT-OSS (o200k) | 199,998 | 27.1 | 1.84 | 17 | 4.5 |
| DeepSeek LLM | 102,400 | 24.8 | 7.51 | 8 | 0.6 |
| DeepSeek Coder | 32,256 | 20.5 | 7.97 | 0 | – |
| Llama 3 | 128,256 | 15.7 | 2.04 | 205 | 2.1 |
| Whisper | 51,866 | 14.1 | 3.49 | 42 | 1.2 |
| Llama 2 | 32,000 | 13.2 | 4.01 | 169 | 0.5 |
| Phi-3 | 32,064 | 13.2 | 4.01 | 12 | 0.7 |
| Mistral v3 | 32,768 | 12.3 | 3.87 | 6 | 1.0 |
| StarCoder | 49,152 | 7.3 | 3.37 | 5 | 0.4 |
| BERT | 30,522 | 4.2 | 0.17 | 0 | – |
| GPT-NeoX | 50,432 | 2.4 | 4.60 | 24 | 0.6 |
| Falcon | 65,024 | 2.4 | 7.54 | 7 | 0.2 |
| OLMo (cl100k) | 100,256 | 2.2 | 4.03 | 18 | 0.2 |
| GPT-2 | 50,257 | 0.5 | 8.29 | 38 | 0.1 |
Table 1. Table 1: Reference tokenizer families, sorted by the share of vocabulary rows unused by Thai and English text under the script rule. Token ratio: Thai over English tokens for the same PUD sentence. : generative and speech models of P1 using the family. Tax: their median share of model bytes for a Thai and English deployment.
Across P1, the median tax for a Thai and English deployment is 4.4% (95% CI [2.3, 7.9]) of model bytes for models below 0.5B parameters (), 4.3% for 0.5–2B, 2.8% for 2–8B and 2.1% above 8B (). The 90th percentile below 0.5B is 22.1%. Weighted by downloads, edge-sized models carry a tax of 4.8%. Across all of P1, which was downloaded 318 million times in the month before the census, a median 2.8% of the downloaded bytes over the 40 target languages (1.5% for Chinese) are rows a single-language deployment never uses under the script rule, and 8.2% in P2 under the Thai corpus rule; Qwen3 0.6B alone was downloaded 23 million times, each copy carrying 0.18 GB of rows a Thai and English deployment never uses (0.54 GB under the corpus rule). The decline is significant (Spearman between parameters and tax over all 1,244 models, ; one-sided Mann–Whitney test of models under 0.5B against models over 8B, ). The token tables themselves hold a median 13.1% of model bytes in P1 and 41.2% below 0.5B. The tax is regressive for a structural reason: the token tables grow with the vocabulary and the width, while the transformer layers grow with the width squared times the depth, so the tables are a larger share of a small model. Per-layer embeddings make this extreme. In Gemma 4 E2B the token tables hold 2.75B of 5.12B parameters, 4.70 GB of them in per-layer tables, and 1.93 GB of its 10.25 GB (18.8%) are rows a Thai and English user never uses (Table 13); the 41 models of P1 with per-layer embeddings have a median tax of 15.3%.
| Model | Size (GB) | Script TH+EN (%) | Script TH+EN+ZH (%) | Corpus TH (%) |
| Gemma 3 270M | 0.54 | 21.9 (0.12) | 17.3 | 55.4 (0.30) |
| Qwen3 0.6B | 1.50 | 11.9 (0.18) | 5.0 | 35.8 (0.54) |
| Gemma 3 1B | 2.00 | 10.6 (0.21) | 8.4 | 26.7 (0.53) |
| Qwen3 1.7B | 4.06 | 8.8 (0.36) | 3.7 | 26.5 (1.08) |
| Whisper large-v3-turbo | 1.62 | 1.2 (0.02) | 0.9 | – |
| Gemma 4 E2B | 10.25 | 18.8 (1.93) | 14.9 | 47.5 (4.87) |
| Ministral 3 3B | 4.67 | 4.7 (0.22) | 4.2 | – |
| Whisper medium | 3.06 | 1.0 (0.03) | 0.8 | – |
Table 13. Reference models. Tax as a share of stored model bytes (GB removed in brackets) for Thai and English, for Thai, English and Chinese, and under the merge-closed Thai corpus rule (15,000 lines; families outside P2: –).
The tax barely depends on which language is kept. Repeating the census for 40 languages (Appendix Table 14), the median below 0.5B is 4.4–4.5% for 35 of them; it is lower only for languages whose script occupies a large part of common vocabularies (Korean 4.0%, Chinese 3.6%, Japanese 3.3%, Russian and Ukrainian 2.4%). For Lao and Punjabi the script rule keeps no more rows than for English alone, and for Amharic, Burmese and Khmer hardly more: the reference vocabularies contain hardly any pieces in these scripts, so their speakers pay the compute tax in full instead.
The script rule is exact
Over 189 language–tokenizer pairs (21 PUD languages, 9 families), at most 0.008% of held-out tokens fall in rows the script rule removes, and the misses are rare symbols outside any declared script (Appendix Table 15). Table 2 reports the full round trip for five families: we rebuild each tokenizer, trim it to Thai and English, and tokenize every PUD sentence and 4,000 Thai social-media posts with both. Thai and the 14 Latin-script languages, English among them, tokenize identically in 100% of sentences, every language round-trips to the same text, and the removed scripts still encode, at 1.0–7.2 times as many tokens. The rebuilt untrimmed tokenizers agree with llama.cpp on all sentences for Qwen, GPT-2 and Gemma and on 97.9% for Llama 3, where they differ in pre-tokenization of a few character classes; the identity test compares the trimmed and untrimmed tokenizers built the same way.
| Family | Kept / vocabulary | Rebuild | Same TH | Same Latin | Round trip | Other scripts |
| Qwen 2–3.5 | 108,232 / 151,936 | 100.0 | 100 | 100 | 100 | 1.30–4.97 |
| Llama 3 | 108,101 / 128,256 | 97.9 | 100 | 100 | 100 | 2.04–5.43 |
| GPT-2 | 50,021 / 50,257 | 100.0 | 100 | 100 | 100 | 1.00–1.63 |
| Gemma 3/3n/4 | 170,274 / 262,144 | 100.0 | 100 | 100 | 100 | 3.62–7.23 |
| Mistral v3 | 28,726 / 32,768 | – | 100 | 100 | 100 | 1.91–4.65 |
Table 2. Tokenizer surgery to Thai and English; all columns but the last in %. Rebuild: share of sentences on which the rebuilt untrimmed tokenizer matches llama.cpp. Same: share of sentences tokenized identically by the trimmed and untrimmed tokenizer (Thai: PUD and 4,000 social-media posts; Latin: 14 Latin-script PUD languages). Round trip: share of all sentences decoded back to the input. Other scripts: token count after trimming over before, for Arabic, Hindi, Japanese, Korean, Russian and Chinese. Mistral v3 is a SentencePiece model, trimmed in its protocol buffer and compared with the sentencepiece library only.
Keeping Latin matters. Trimming to the Thai script alone removes 91–95% of the rows instead of 12–35% (0.5% for GPT-2), but 3.5–6.6% of the tokens in Thai text are then lost (1.8% for GPT-2), because Thai text contains Latin names, numbers written with Latin pieces, URLs and English words. With Latin kept, no Thai token is lost.
Frequency-based trimming breaks BPE tokenizers
Figure 3 and Table 3 compare the usual corpus rule, which keeps the pieces seen as tokens, with its merge-closed version, measuring both on rebuilt tokenizers. Without closure, between 13 and 6,198 kept rows per setting are orphans (median 828, 5.0% of the kept rows): they cost memory and can never be selected. The real re-split rate exceeds the nominal rate in almost every setting, by a median factor of 1.9 and by more than 1.2 in 95 of the 101 settings with a measurable rate. The effect is largest for Latin-script text, where long words are built by chains of merges: for German on Qwen the nominal rate is 2.4% and the real rate 7.1%; for English web text the real rate is 23–27% and held-out sequences grow by 49–62%, against a nominal rate of about 9%. With closure the real rate is a median 0.94 of the nominal rate (below 1 because the added parents also catch some words that the removed pieces would have covered), and sequences grow by a median 0.9% instead of 2.5%. Closure costs a median 5.9% more kept rows (up to 36% for English), a small price next to the 80–90% of rows the corpus rule removes. The script rule needs almost no closure: on Qwen with Thai and English it adds 92 parents to 108,232 kept pieces: single Cyrillic, Arabic, Hangul and CJK characters that are parents of byte-fragment pieces, which the rule always keeps.
Figure 3. Corpus trimming with and without merge closure, BPE families, 4,000 trimming lines. (a) Real re-split rate of 1,000 held-out lines after rebuilding the tokenizer, against the nominal rate (tokens whose row was removed). Keeping the pieces seen as tokens (gold) gives a real rate a median 1.9 times the nominal rate; adding merge parents (blue) brings it to or below the nominal rate. (b) Extra tokens per held-out text, without and with closure.
| Family | Orphans | Closure cost (%) | Real / nominal usual | Real / nominal closed | Extra tokens (%) usual | Extra tokens (%) closed |
| Gemma 3/3n/4 | 1,146 | 8.7 | 2.05 | 0.95 | 6.31 / 62 | 1.80 / 12 |
| Qwen 2–3.5 | 989 | 4.7 | 2.10 | 0.95 | 2.86 / 52 | 0.72 / 9 |
| Llama 3 | 638 | 11.6 | 1.81 | 0.91 | 2.07 / 49 | 0.88 / 10 |
| GPT-2 | 567 | 8.1 | 1.67 | 0.94 | 0.76 / 51 | 0.36 / 9 |
Table 3. Merge closure, medians over 22 languages per family (4,000 trimming lines, 2,000 for Chinese and Urdu). Orphans: kept rows the rebuilt tokenizer can no longer produce without closure. Closure cost: extra kept rows. Real / nominal: real re-split rate over nominal rate, without and with closure (settings with a nominal rate above 0.2%). Extra tokens: median / maximum growth of held-out sequences (%).
Merge-closed trimming beats prior keep rules
Figure 4 and Table 4 compare our rule with the keep rules of prior work. At the same number of rows removed, the seen-in-corpus rule re-splits a median 1.30 times as much new text as ours and frequency top- 1.32 times as much, worse in 91% and 90% of the comparisons; rank top-, which ignores the target language, re-splits 8.4 times as much. Choosing each rule’s best parameter with knowledge of the real rate, ours removes 52.6% of the rows on average at a real rate of at most 1%, against 46.6% for seen-in-corpus, 46.5% for frequency top-, 22.4% for rank top- and 17.3% for the script rule alone.
Figure 4. Rows removed against the real re-split rate of new text for each keep rule, swept over its parameter (4,000 trimming lines, rebuilt tokenizers). Lower and to the right is better. Merge closure moves both corpus-based rules down and to the left of their usual versions; rank top- keeps the wrong pieces altogether.
Table 4. Keep rules compared on 44 settings (4 families 11 languages: Thai, Chinese, Japanese, Korean, Arabic, Hindi, Russian, German, French, Spanish, English; 4,000 trimming lines). Matched: real re-split rate of the rule at the rows our rule removes, interpolated along its curve, as a multiple of ours (median). Orphans: unreachable kept rows (median). Oracle: mean rows removed (%) at a real re-split rate , choosing each rule’s best parameter with knowledge of the real rate, script rule as fallback. Deployed: each rule chooses its parameter as a user would, prior rules by the nominal rate on 1,000 held-out lines (test data), ours by the certificate on the trimming text alone (no test data); mean rows removed (%) and number of settings whose real re-split rate exceeds the target.
The deployed comparison is the one a user faces. A user of a prior rule checks the nominal rate on held-out text and keeps the most aggressive setting that passes; because the nominal rate understates the real one, this setting overshoots the target in 13 of 44 settings for seen-in-corpus and 9 for frequency top- at a 2% target, although these users had test data. Our user, without test data, chooses by the certificate and overshoots in 0 settings while removing about as many rows (68.2% against 68.1–68.5%). At a 1% target the certificate overshoots in 3 settings, as many as frequency top- (3) and half as many as seen-in-corpus (6); our overshoots reach at most 1.18%, those of the prior rules up to 1.58% and 2.53%. Every prior rule also keeps unreachable rows, the script rule included (a median of 87 byte-fragment pieces whose parents it removes); ours keeps none.
Every language can shed the same share, at very different cost
With 15,000 lines of Thai text, the merge-closed corpus rule removes 79%–89% of the rows of the four families at a real re-split rate of 0.10%–0.67% (Table 5); each doubling of the text lowers the rate by about a third. In P2 the median tax rises from 7.9% under the script rule to 29.0% below 0.5B parameters, and from 2.1% to 9.1% above 8B (Figure 5). For Gemma 4 E2B the corpus rule removes 4.87 GB of 10.25 GB (47.5%), and for Qwen 3 0.6B 35.8% (Table 13).
| Family | Script | 1,000 | 2,000 | 4,000 | 15,000 |
| Gemma 3/3n/4 | 35.0 | 92.1 / 3.33 | 91.4 / 2.35 | 90.7 / 1.59 | 88.5 / 0.67 |
| Qwen 2–3.5 | 28.8 | 90.9 / 2.00 | 90.0 / 1.46 | 89.3 / 0.92 | 86.5 / 0.40 |
| Llama 3 | 15.7 | 89.5 / 2.28 | 88.3 / 1.65 | 87.3 / 1.05 | 83.5 / 0.44 |
| GPT-2 | 0.5 | 90.5 / 0.58 | 88.4 / 0.41 | 86.3 / 0.28 | 79.1 / 0.10 |
Table 5. Merge-closed corpus rule for Thai: share of vocabulary rows removed / real re-split rate of 1,000 held-out lines (%), by number of trimming lines. Script: rows removed by the script rule alone.
Figure 5. The tax is regressive. Median share of model bytes removable for a Thai and English deployment by model size in P2, under the script rule and the merge-closed corpus rule with 15,000 Thai lines. Bars: 95% bootstrap confidence intervals.
Figure 6 repeats this for 22 languages. The share of bytes removable below 0.5B is 27.8%–31.1% in every language: each language uses a small fraction of a large vocabulary. The cost of removing it is not equal. With 4,000 lines the median real re-split rate is at most 0.01% for Hebrew and Hindi, 0.04% for Arabic, 0.39% for Russian and 0.99% for Thai, but 2.26% for German, 3.04% for French and 7.69% for English web text. The order follows how much of a language’s vocabulary lies in pieces shared with English. Pieces in a script used by few languages are few and frequent, so a few thousand lines see nearly all of those a language needs. Latin pieces are numerous and spread over every Latin-script language, so a sample of German sees a smaller share of the German pieces it will need; English, whose pieces are the bulk of most vocabularies, is the extreme. Appendix Table 16 gives every language and family, with and without closure.
Figure 6. 22 languages, P2 (909 models), merge-closed corpus rule with 4,000 trimming lines (Urdu and Chinese: 2,000). Left: median share of bytes removable for models under 0.5B. Right: real re-split rate of new text, median and range of the four families, log scale. Languages are sorted by the median.
Mixed-language deployments
Most deployments serve more than one language. Table 6 repeats the analysis for five mixtures, each with English. Under the script rule the tax shrinks when the mixture includes a script that occupies much of the vocabulary: Thai with Chinese leaves a median 3.3% below 0.5B and 1.52 GB in Gemma 4 E2B, against 4.4% and 1.93 GB for Thai alone. The merge-closed corpus rule still removes 80–90% of the rows with 2,000 lines per language. The pooled re-split rate hides large differences between the languages of a mixture (French reaches 8.1% next to Arabic’s 0.5% on Gemma), so the certificate should be read per language. We compute it per language by counting, in each language’s text, the singletons that no other language of the mixture uses; it predicts the per-language real rate with a median ratio of 1.03 (interquartile range 0.96–1.10, 40 language–family pairs), and the pooled estimate predicts the pooled rate with a median ratio of 1.02. Without merge closure the pooled real rate is 1.65 times higher.
| Languages | Tax <0.5B | Gemma 4 E2B | Removed | Real (pooled) | Worst language | Cert./real |
| th, zh, en | 3.3 | 1.52 | 85.7 | 1.93 | 7.08 | 0.98 |
| ar, fr, en | 4.4 | 1.80 | 83.2 | 1.86 | 8.08 | 1.04 |
| ja, ko, zh, en | 2.7 | 1.37 | 85.3 | 1.37 | 6.66 | 0.96 |
| es, pt, fr, en | 4.5 | 1.97 | 79.9 | 1.99 | 6.26 | 1.09 |
| ru, uk, en | 2.4 | 1.69 | 89.7 | 0.47 | 2.91 | 1.11 |
Table 6. Deployments serving several languages (ISO codes; English always included). Script rule: median tax (%) of models under 0.5B in P1, and GB removable from Gemma 4 E2B. Corpus rule: merge-closed, 2,000 trimming lines per language, medians over the four families of rows removed (%) and pooled real re-split rate (%), the worst real rate of any single language and family (%), and the median ratio of the per-language certificate to the per-language real rate.
The certificate predicts the real re-split rate without test data
We evaluate the certificate on 22 languages, the four families and 1k, 2k, 4k and, where the treebank allows, 15k trimming lines: 300 settings, each against the real re-split rate of the merge-closed trimmed tokenizer on the language’s 1,000 held-out lines. The estimate of Eq. (2) agrees with the real rate with correlation 0.996 on a log scale and a median ratio of 1.01 (interquartile range 0.94–1.06), and it falls on the right side of a 1% threshold in 297 of 300 settings (Figure 7a). Holding out 20% of the trimming text, the alternative that needs no theory, has a median relative error of 22.6%, 24.4% and 25.9% at 1k, 2k and 4k lines, against 6.6%, 5.9% and 4.5% for the estimate. Five-fold cross-validation, which uses all of the text for estimation, does no better (median error 25–27%; Table 7): every fold builds its keep set from 80% of the text, while the deployed set uses all of it, so both resampling estimates are biased upward. The Good–Turing estimate has no such bias because it describes the keep set actually deployed.
| Lines | Hold-out | 5-fold CV | Ours | |
| 1,000 | 22.6 | 24.9 | 6.6 | 76 |
| 2,000 | 24.4 | 25.5 | 5.9 | 72 |
| 4,000 | 25.9 | 26.7 | 4.5 | 62 |
| 15,000 | – | 25.8 | 6.5 | 32 |
Table 7. Median relative error (%) of re-split estimates against the real rate, by trimming lines. : settings with a real rate above 0.2%.
Figure 7. Trim certificate, against the real re-split rate of merge-closed trimming. (a) Rate estimated from the trimming text (Eq. (2), blue) or measured on a 20% hold-out of it (gold), against the real rate on 1,000 new lines; 22 languages, 4 families, 1k, 2k and 4k lines. (b) Rate at 15,000 lines predicted from 2,000 and 4,000 lines (Eq. (3)) against the real rate. Dashed: perfect agreement.
To test the extrapolation of Eq. (3), we predict the real rate at 15,000 lines from the estimates at 2,000 and 4,000 lines for the 11 languages whose treebanks are large enough (35 language–family pairs with a rate above 0.1%). The median ratio of predicted to real rate is 1.03 (interquartile range 0.94–1.12; 1.06 for the 30 pairs with a rate of at least 0.3%). A fixed exponent does markedly worse ( error 0.32 for against 0.10 fitted; Table 8, so the decay must be fitted per language and tokenizer. Below 0.3% the prediction can be low by up to a factor of four (Figure 7b), because the estimate becomes noisy when few singletons remain; at a 1% target this region is already certified. Table 9 turns the extrapolation into a guide: the median number of lines needed for a 1% rate is about 4,000 or fewer for the languages written in Arabic, Hebrew, Devanagari, Korean, Cyrillic or Thai script, 5,000 for Japanese, 7,000–20,000 for the Latin-script languages, and about 190,000 for English web text.
| Predictor | Ratio | err. |
| No change (rate at 4k lines) | 2.75 | 1.01 |
| Fixed exponent | 1.42 | 0.39 |
| Fixed exponent | 0.73 | 0.32 |
| Fitted (Eq. (3)) | 1.03 | 0.10 |
Table 8. Predicting the real rate at 15k lines from 2k and 4k lines (35 pairs): median ratio to the real rate and median error.
| Language | Measured at 4k (%) | Lines needed (median) | Lines needed (worst) |
| Arabic | 0.04 | ≤4k | ≤4k |
| Persian | 0.14 | ≤4k | ≤4k |
| Hebrew | 0.00 | ≤4k | ≤4k |
| Hindi | 0.00 | ≤4k | 8k |
| Korean | 0.41 | ≤4k | ≤4k |
| Russian | 0.39 | ≤4k | 6k |
| Ukrainian | 0.40 | ≤4k | 4k |
| Thai | 0.99 | 4k | 7k |
| Japanese | 0.79 | 5k | 6k |
| Spanish | 1.56 | 7k | 16k |
| Polish | 1.54 | 7k | 13k |
| Czech | 1.76 | 9k | 18k |
| Italian | 1.95 | 10k | 20k |
| Portuguese | 2.21 | 12k | 27k |
| German | 2.26 | 14k | 25k |
| Turkish | 2.42 | 15k | 25k |
| Indonesian | 2.74 | 18k | 55k |
| Dutch | 2.78 | 19k | 35k |
| French | 3.04 | 20k | 43k |
| English (web) | 7.69 | 191k | 275k |
Table 9. Lines of in-domain text needed for a real re-split rate of at most 1% after merge-closed trimming, extrapolated from 2,000 and 4,000 lines (Eq. (3)); median and worst of the four families. Measured: median real rate at 4,000 lines (%).
Shifting domain. The certificate describes new text from the source of the trimming text. To measure what happens when the deployment text differs, we trim on each language’s main treebank (for Thai, on social-media posts only) and test on the PUD sentences of the same language, translated news and Wikipedia text (Table 10). Across the 16 languages the real rate on the shifted domain is a median 1.7 times the in-domain rate and up to 3.4 times (Korean); it is roughly unchanged for French, Indonesian and Japanese. Adding 200 lines of the new domain to the trimming text lowers it by only 6%. Thai moves the other way, from 1.47% to 0.12%, because informal posts mix in English, slang and spelling variants that make formal news easy to cover. The practical remedy is cheap because of merge closure: with a closed keep set, the nominal rate measured on just 200 lines of deployment text is a median 0.86 of the real rate on 1,000 lines, so a user can check a new domain directly; without closure the same check would understate the damage about twofold (§ 5.3).
| Language | Certificate | Real, in-domain | Real, PUD | PUD / in-domain | Nominal 200 / real |
| Arabic | 0.06 | 0.04 | 0.15 | – | 1.22 |
| Czech | 1.87 | 1.76 | 3.14 | 1.75 | 0.77 |
| English | 8.95 | 7.69 | 13.65 | 1.78 | 0.82 |
| French | 3.10 | 3.04 | 2.96 | 0.98 | 0.96 |
| German | 2.35 | 2.26 | 4.74 | 2.10 | 0.78 |
| Hindi | 0.00 | 0.00 | 0.10 | 2.13 | 0.85 |
| Indonesian | 2.52 | 2.74 | 2.77 | 1.00 | 0.84 |
| Italian | 2.08 | 1.95 | 3.69 | 1.89 | 0.89 |
| Japanese | 0.88 | 0.79 | 0.88 | 1.12 | 1.05 |
| Korean | 0.24 | 0.41 | 1.26 | 2.79 | 1.26 |
| Polish | 1.49 | 1.54 | 3.69 | 2.34 | 0.82 |
| Portuguese | 2.33 | 2.21 | 3.51 | 1.59 | 0.95 |
| Russian | 0.40 | 0.39 | 0.86 | 2.10 | 1.12 |
| Spanish | 1.61 | 1.56 | 3.09 | 1.99 | 0.83 |
| Thai | 1.01 | 1.47 | 0.12 | 0.08 | 0.50 |
| Turkish | 2.54 | 2.42 | 4.27 | 1.75 | 0.72 |
Table 10. Domain shift, 4,000 trimming lines from the main treebank (Thai: social-media posts), medians over four families. Certificate and real re-split rate on 1,000 in-domain lines and on the 1,000 PUD sentences (%), their ratio, and the nominal rate on 200 PUD lines over the real rate on all 1,000. Chinese is omitted because its PUD is in Traditional characters.
What did not help. A natural way to need less text is to keep, in addition to the pieces seen, the first pieces of the vocabulary, since BPE merge order follows pretraining frequency. With this lowers the nominal re-split rate of Qwen on 1,000 French lines from 8.6% to 5.3% but keeps 7.6 percentage points more rows, a worse trade than adding text (Appendix § 11). We keep the plain corpus rule and let the certificate say how much text is needed.
Trimmed models keep their predictions
Table 11 and Figure 8 show what trimming does to the models themselves.
Figure 8. Corpus rule on six models and 11 languages (122 settings). (a) The certificate, computed from the trimming text, against the re-split rate measured on the evaluation lines (correlation 0.994). Models that share a tokenizer overlap. (b) Change in bits per character against the measured re-split rate; dashed: least-squares line without SmolLM3. SmolLM3 points below zero at low re-split are the renormalization effect discussed in the text.
| Model | Rule | n | Removed (%) | Re-split (%) | ΔBPC (%) | Range (%) | Greedy (%) | ΔBelebele (pp) |
| Gemma 3 270M | Script | 11 | 34.1 | 0.00 | −0.01 | −0.1, +0.4 | 100 | 0.0 |
| Corpus 4k | 11 | 88.5 | 1.58 | +2.93 | −0.2, +23.3 | 63 | +0.2 | |
| Corpus 15k | 10 | 87.9 | 0.80 | +1.13 | −0.3, +8.1 | 80 | +0.2 | |
| Qwen3 0.6B | Script | 11 | 28.9 | 0.00 | −0.20 | −0.4, +0.1 | 100 | 0.0 |
| Corpus 4k | 10 | 89.1 | 0.95 | +0.54 | −1.9, +14.0 | 76 | +0.5 | |
| Corpus 15k | 7 | 86.7 | 0.42 | −1.04 | −1.8, +0.4 | 85 | +0.3 | |
| Gemma 3 1B | Script | 11 | 34.1 | 0.00 | −0.07 | −0.6, +0.6 | 99 | 0.0 |
| Corpus 4k | 11 | 87.6 | 1.74 | +2.96 | −0.2, +21.3 | 53 | −0.5 | |
| Corpus 15k | 10 | 87.6 | 0.81 | +1.15 | −0.6, +9.2 | 75 | −0.2 | |
| Llama 3.2 1B | Script | 11 | 15.6 | 0.00 | −0.03 | −0.3, +0.3 | 100 | 0.0 |
| Corpus 4k | 11 | 88.0 | 0.60 | +0.69 | −2.2, +16.7 | 72 | −1.4 | |
| Corpus 15k | 10 | 85.4 | 0.35 | −0.49 | −2.2, +4.5 | 85 | −0.8 | |
| Qwen3 1.7B | Script | 11 | 28.9 | 0.00 | −0.39 | −0.7, 0.0 | 99 | 0.0 |
| Corpus 4k | 11 | 88.8 | 0.98 | +2.27 | −0.9, +16.3 | 72 | −0.2 | |
| Corpus 15k | 10 | 87.5 | 0.47 | −0.13 | −0.9, +4.5 | 86 | +0.3 | |
| SmolLM3 3B | Script | 11 | 15.6 | 0.00 | −0.02 | −0.1, +0.3 | 100 | 0.0 |
| Corpus 4k | 11 | 88.0 | 0.60 | −8.75 | −13.8, +14.2 | 72 | −2.0 | |
| Corpus 15k | 10 | 85.4 | 0.35 | −8.37 | −13.8, +3.8 | 83 | −0.9 |
Table 11. Quality of trimmed models, over the 11 languages of each condition (n). Rows removed and measured re-split rate on the evaluation lines: medians. Change in bits per character (BPC): median and range. Greedy: share of greedy continuations identical to the original model’s, mean. Belebele: mean change in accuracy, percentage points (original accuracy 0.20–0.39, chance 0.25).
The script rule. Across the 66 model and language pairs, bits per character change by between −0.71% and +0.55%, at least 93% of greedy continuations are identical to the original model’s (median 100%), and Belebele answers are identical in 61 cases and differ by one question in the rest. When the tokenization is unchanged, the trimmed model is the original model renormalized over the kept rows, so every token’s probability can only rise and bits per character can only fall. The few small increases are in Korean and French, whose held-out text contains characters outside the kept scripts, such as Han characters in Korean, which the trimmed tokenizer spells in bytes (0.3% more tokens).
The corpus rule and the certificate. The certificate predicts the re-split rate on the evaluation lines as it did on the tokenizer experiments (correlation 0.994, measured rate a median 0.88 times the prediction), and the change in bits per character follows the re-split rate almost linearly. For the five models other than SmolLM3, the least-squares line is (correlation 0.975, n=101): each percent of re-split tokens costs about 2.5% in bits per character, and renormalization gives back about 1%. The certificate therefore predicts the quality cost before any model is run. In the 68 settings that it certifies (predicted re-split under 1%), the corpus rule removes a median 89.7% of rows, bits per character change by a median −0.65% and at most +1.99%, and a median 91% of greedy continuations are identical. In the 54 settings where it asks for more text, the median change is +3.41%, and the worst, English with 4,000 lines, reaches +23.3% at a re-split rate of 8.8%. Greedy agreement is the strictest of the three measures, since one different token among 24 counts as a mismatch; it falls with the re-split rate in the same way.
Renormalisation is a property of the model. SmolLM3 uses the Llama 3 tokenizer, so its keep sets and re-split rates are identical to those of Llama 3.2 1B. Yet where the re-split rate is under 0.2%, its bits per character change by −8.0% to −13.8%, against falls of at most 2.3% for Llama 3.2 1B. With the tokenization unchanged, the change equals the log of the probability the original model gives to kept pieces, so SmolLM3 places far more probability on pieces that never occur in the target language. This is not a better model: its Belebele accuracy does not rise. A drop in bits per character after trimming should therefore be read together with the re-split rate, and comparisons of trimmed models are clearest when made against the same model’s script rule, which isolates the renormalization.
Belebele. These models are close to chance on Belebele (0.20–0.39 against 0.25), and with 200 questions two binomial standard errors are about 0.06. Only 3 of 188 changes exceed that, all drops for Llama 3.2 1B under the corpus rule in Chinese and Japanese (largest 0.075). Belebele passages come from Wikipedia through FLORES, not from the treebanks the keep sets were built on, and we did not measure the re-split rate on them; the drop is consistent with the domain effect of Table 10, which the 200-line check on deployment text is meant to catch. At this model scale Belebele detects only large changes; bits per character and greedy agreement are the more sensitive measures.
Trimming moves edge models under memory budgets
Table 12 counts edge-sized models of P2 whose weights exceed a budget and fit after trimming. At stored precision and a 1 GB budget, 3 of 182 over-budget models fit after the script rule and 25 after the corpus rule; at INT4 and 1 GB, 10 and 21 of 54. Qwen3 0.6B (1.50 GB) fits in 1 GB only after the corpus rule, and so does Qwen3 1.7B at INT4. Table 13 shows reference models. Adding Chinese to Thai and English reduces the tax of Qwen3 0.6B from 11.9% to 5.0%, because Chinese pieces are a large share of its vocabulary, but reduces that of Gemma 3 270M only from 21.9% to 17.3%.
| Budget | Over budget | Fit after script rule | Fit after corpus rule |
| Stored precision, 1 GB | 182 | 3 | 25 |
| Stored precision, 2 GB | 131 | 5 | 16 |
| Stored precision, 4 GB | 64 | 9 | 19 |
| INT8, 1 GB | 124 | 3 | 6 |
| INT8, 2 GB | 54 | 10 | 21 |
| INT4, 1 GB | 54 | 10 | 21 |
Table 12. Edge-sized models in P2 () that exceed a weight budget, and how many fit after trimming to Thai and English. Weight bytes at the stored precision, or recomputed at INT8 or INT4. Corpus rule: merge-closed, 15,000 Thai lines.
Discussion and limitations
Model quality. § 5.8 measures quality without retraining, on models up to 3B parameters and on text from the same source as the trimming text. Larger models, generation quality judged by people, and fine-tuning after trimming are not covered, and Belebele has little power at this scale. The relation between re-split rate and bits per character was close to linear for five of six models, but its intercept depends on how much probability a model places outside the kept pieces, as SmolLM3 shows. The certificate estimates how much of the input changes at all, and merge closure matters for the same reason: without it, two to three times more of the input changes than a user who checks the nominal rate would believe.
Bytes on disk and bytes in memory. The tax is measured in stored bytes. Runtimes that memory-map weights page in only the embedding rows they touch, so for them the input embedding costs disk and download size more than memory, while the output head is read in full at every decoding step and costs memory bandwidth and compute. The deployment counts of Table 12 are therefore for weight files; the reduction in resident memory depends on the runtime and on whether the head is tied.
Scope. Tokenizer families are assigned by model type and vocabulary size, which can misassign a model that ships a custom tokenizer with a standard size; the package reads each model’s own tokenizer when it is used directly. The certificate is for in-domain text; on a shifted domain the real rate was a median 1.7 times higher in our test, and a 200-line nominal check on deployment text is the recommended safeguard. The newest Hub-only tokenizer families (Qwen3.5 and later, Nemotron-H, GLM-5, DeepSeek-V3.2 and V4) are not in the census populations. Treebanks for Tamil, Bengali, Marathi and Telugu are too small for the corpus experiments. The corpus-rule experiments cover the four BPE families we rebuild from merge lists; the package also closes tiktoken, Tekken and SentencePiece-BPE vocabularies, taking the parents of a SentencePiece-BPE piece from the highest-score merge order (on Thai text no reachable kept piece of the Mistral v3 or Llama 2 vocabulary was left unreachable after closure), and SentencePiece Unigram needs no closure. These extensions are not part of the measured experiments. Encoder models and the vision and audio towers of multimodal models are outside the populations studied here.
Conclusion
Multilingual vocabularies have a memory cost that single-language deployments pay in full. Measured from manifests and tokenizer files, it is a few percent of a large model and over a quarter of a small one under the corpus rule, it is about the same for every language, and it trades against the extra tokens that small vocabularies cost non-English languages. Removing it by writing system is exact when the keep rule includes the shared character classes and Latin. Removing it with the user’s text goes further, but only if the keep set is closed under BPE merges; the usual frequency rule silently strands rows and changes two to three times more of the input than its check reports. With closure, the amount of text a deployment needs is predictable from that text itself: a Good–Turing estimate and a fitted decay tell the user, before deployment, how much of their future input the trimmed tokenizer will split differently and how much more text would fix it, and on the models we tested the loss in bits per character follows that rate. The census, the merge-closed trimming and the certificate are available in ai-model-explorer, and the tables behind this paper in the ai-vocabulary-tax dataset.
Script rule for 40 languages
| Languages | <0.5B | 0.5–2B | 2–8B | >8B | Edge, weighted | Gemma 4 E2B (GB) |
| Dutch, English, French, German, Indonesian, Italian, Lao, Malay, Polish, Portuguese, Punjabi, Spanish, Swahili, Tagalog, Turkish, Vietnamese | 4.5 | 4.6 | 2.9 | 2.2 | 5.1 | 1.97 |
| Bengali | 4.5 | 4.6 | 2.9 | 2.2 | 5.1 | 1.77 |
| Telugu | 4.5 | 4.6 | 2.9 | 2.2 | 5.1 | 1.94 |
| Gujarati, Kannada | 4.5 | 4.6 | 2.9 | 2.2 | 5.1 | 1.95 |
| Amharic, Burmese, Khmer | 4.5 | 4.6 | 2.9 | 2.2 | 5.1 | 1.96 |
| Thai | 4.4 | 4.3 | 2.8 | 2.1 | 4.8 | 1.93 |
| Arabic, Persian, Urdu | 4.4 | 4.2 | 2.7 | 1.8 | 4.7 | 1.80 |
| Hindi, Marathi | 4.4 | 4.6 | 2.9 | 2.1 | 5.1 | 1.69 |
| Tamil | 4.4 | 4.6 | 2.9 | 2.2 | 5.1 | 1.89 |
| Malayalam | 4.4 | 4.6 | 2.9 | 2.2 | 5.1 | 1.93 |
| Hebrew | 4.4 | 4.3 | 2.8 | 2.2 | 4.8 | 1.96 |
| Greek | 4.4 | 4.6 | 2.9 | 2.1 | 5.0 | 1.94 |
| Georgian | 4.4 | 4.6 | 2.9 | 2.2 | 5.1 | 1.96 |
| Armenian | 4.4 | 4.6 | 2.9 | 2.2 | 5.1 | 1.97 |
| Korean | 4.0 | 4.2 | 2.7 | 2.0 | 4.7 | 1.88 |
| Chinese | 3.6 | 2.5 | 1.4 | 1.5 | 2.6 | 1.57 |
| Japanese | 3.3 | 2.2 | 1.3 | 1.3 | 2.3 | 1.47 |
| Russian, Ukrainian | 2.4 | 4.2 | 2.7 | 1.5 | 4.5 | 1.69 |
Table 14. Median vocabulary tax (% of model bytes) under the script rule (target language plus Latin) in P1, by model size; download-weighted tax of edge-sized models; tax of Gemma 4 E2B in GB. Languages with identical rows are grouped.
Script rule validation
| Family | ar | cs | de | en | es | fi | fr | gl | hi | id | is | it | ja | ko | pl | pt | ru | sv | th | tr | zh |
| Gemma 3/3n/4 | 33 | 36 | 36 | 36 | 36 | 36 | 36 | 36 | 31 | 36 | 36 | 36 | 27 | 34 | 36 | 36 | 31 | 36 | 35 | 36 | 29 |
| Qwen 2–3.5 | 28 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 12 | 28 | 30 | 30 | 28 | 30 | 29 | 30 | 14 |
| Llama 3 | 14 | 17 | 17 | 17 | 17 | 17 | 17 | 17 | 16 | 17 | 17 | 17 | 13 | 15 | 17 | 17 | 12 | 17 | 16 | 17 | 13 |
| Llama 2 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 11 | 13 | 13 | 13 | 4 | 13 | 13 | 13 | 11 |
| Mistral Tekken | 20 | 28 | 28 | 28 | 28 | 28 | 28 | 28 | 26 | 28 | 28 | 28 | 24 | 24 | 28 | 28 | 22 | 28 | 27 | 28 | 25 |
| Whisper | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 11 | 11 | 14 | 14 | 9 | 14 | 14 | 14 | 12 |
| GPT-2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| XLM-R | 48 | 54 | 54 | 54 | 54 | 54 | 54 | 54 | 51 | 54 | 54 | 54 | 45 | 51 | 54 | 54 | 41 | 54 | 52 | 54 | 47 |
| Mistral v3 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 8 | 11 | 12 | 12 | 7 | 12 | 12 | 12 | 8 |
Table 15. Vocabulary rows removed (%) when keeping each PUD language plus Latin, for nine families. The share of held-out tokens in removed rows is at most 0.008% in every cell.
Corpus rule for 22 languages
| Language | Gemma | Qwen | Llama 3 | GPT-2 |
| Arabic | 92/0.14/0.23 | 91/0.05/1.09 | 90/0.03/0.05 | 94/0.00/0.00 |
| Chinese (2k) | 88/3.25/3.69 | 86/7.24/10.19 | 89/1.01/1.43 | 92/0.25/0.29 |
| Czech | 90/2.72/4.76 | 88/1.38/2.25 | 84/2.14/3.96 | 83/1.06/1.61 |
| Dutch | 89/4.10/7.77 | 88/2.74/4.88 | 85/2.81/4.51 | 81/1.86/2.99 |
| English (web) | 86/8.71/26.93 | 83/7.64/26.18 | 78/7.74/23.54 | 66/7.21/25.37 |
| French | 86/5.06/10.57 | 85/3.08/5.71 | 81/2.99/5.00 | 74/2.14/3.82 |
| German | 88/3.90/10.57 | 86/2.26/7.14 | 83/2.26/5.14 | 79/1.40/3.32 |
| Hebrew | 93/0.00/0.00 | 91/0.15/0.37 | 92/0.00/0.00 | 94/0.00/0.00 |
| Hindi | 90/2.01/9.64 | 93/0.00/0.00 | 92/0.00/0.01 | 94/0.00/0.00 |
| Indonesian | 87/5.05/9.92 | 86/2.77/4.54 | 82/2.72/4.46 | 76/1.98/2.95 |
| Italian | 88/3.64/8.56 | 87/2.04/5.08 | 84/1.86/3.66 | 80/1.46/2.76 |
| Japanese | 87/1.56/2.21 | 89/1.18/1.97 | 89/0.39/0.56 | 92/0.12/0.17 |
| Korean | 92/0.56/0.69 | 91/0.42/0.65 | 90/0.40/0.51 | 93/0.04/0.05 |
| Persian | 92/0.46/1.36 | 92/0.11/0.25 | 90/0.17/0.67 | 94/0.02/0.02 |
| Polish | 90/2.46/4.77 | 88/1.62/2.84 | 87/1.45/2.08 | 85/0.95/1.20 |
| Portuguese | 87/4.21/10.18 | 86/2.29/5.09 | 83/2.13/4.19 | 79/1.46/2.58 |
| Russian | 90/1.52/3.43 | 90/0.32/1.58 | 88/0.46/1.15 | 93/0.06/0.08 |
| Spanish | 86/3.58/10.16 | 86/1.58/4.65 | 83/1.54/3.74 | 79/1.04/2.10 |
| Thai | 91/1.59/3.43 | 89/0.92/1.98 | 87/1.05/1.78 | 86/0.28/0.49 |
| Turkish | 90/3.56/6.75 | 88/2.18/4.80 | 85/2.67/5.52 | 84/1.31/2.07 |
| Ukrainian | 91/1.11/1.71 | 91/0.24/0.47 | 88/0.56/1.19 | 92/0.06/0.08 |
| Urdu (2k) | 92/0.79/3.61 | 92/0.09/0.35 | 91/0.21/0.47 | 94/0.01/0.01 |
Table 16. Corpus rule with 4,000 trimming lines (2,000 for Urdu and Chinese): rows removed with merge closure (%) / real re-split rate with closure (%) / real re-split rate without closure (%).
Ablations
Keeping Latin. Trimming to the Thai script alone, without Latin, loses 6.6% (Gemma), 5.0% (Qwen), 5.7% (Llama 3), 5.4% (Tekken), 4.2% (Whisper), 3.6% (Mistral v3), 3.5% (Llama 2) and 1.8% (GPT-2) of the tokens in held-out Thai text; with Latin kept, none.
Rank prior. Nominal re-split rates, before closure. Keeping the first pieces by vocabulary order in addition to the corpus rule, with 1,000 trimming lines: for Qwen and French the re-split rate is 8.6, 7.5, 6.6 and 5.3% for , 4k, 8k and 16k, with 88.9, 87.5, 85.6 and 81.3% of rows removed. Increasing the trimming text to 4,000 lines with gives 3.2% at 85.5%. The pattern is the same for the other families and languages.
Reproducibility
The results were produced with version 1.0.7 of the package on PyPI. The package computes the tax with the vocab_tax method of explore(repo), given the languages and optionally the user’s text, and trims with trim_vocab, which also takes an output folder; both close the keep set under BPE merges (merge_closure_added) and return the certificate fields predicted_resplit_rate, lines_needed_for_target and certificate. The browser tool (Where to cut, on the playground) encodes text with tokenizer.json (BPE and Unigram), tiktoken, Tekken, SentencePiece and GGUF vocabularies; against tokenizers, tiktoken, sentencepiece and llama.cpp it matches token for token on all 605 test sentences for 19 of the 24 vocabularies we checked and on at least 603 for four more (SentencePiece normalization rules), with DeepSeek-LLM’s pre-tokenizer the exception (564), and it computes the same keep set and certificate as the package. Model quality is measured with quality_eval.py, released with the experiment scripts. The tables are published at https://huggingface.co/datasets/JoesSattes/ai-vocabulary-tax: models and model_language (tax per model, language and rule), tokenizers (rows removed and real re-split rate per family, language and amount of text), certificate, merge_closure and quality. The command line equivalent is
ai-model-explorer tax REPO -lang th -corpus mine.txt -target 0.01
-export out/
Treebank text is used under the licenses of the individual treebanks.
References
- [1]Orevagheogne Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
- [2]Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
- [3]Ofir Ben Shoham. Balancing coverage and draft latency in vocabulary trimming for faster speculative decoding. arXiv preprint arXiv:2603.05210, 2026.
- [4]Nikolay Bogoychev, Pinzhen Chen, Barry Haddow, and Alexandra Birch. The ups and downs of large language model inference with vocabulary trimming by language and by frequency. In Proceedings of the Fifth Workshop on Insights from Negative Results in NLP, 2024.
- [5]Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. Universal dependencies. Computational Linguistics, 47(2):255–308, 2021.
- [6]Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall, 1993.
- [7]William A. Gale and Geoffrey Sampson. Good-Turing frequency estimation without tears. Journal of Quantitative Linguistics, 2(3):217–237, 1995.DOI
- [8]Georgi Gerganov and ggml-org contributors. llama.cpp: LLM inference in C/C++. https://github.com/ggml-org/llama.cpp, 2023. Vocabulary files in models/, read November 2026.
- [9]I. J. Good. The population frequencies of species and the estimation of population parameters. Biometrika, 40(3–4):237–264, 1953.DOI
- [10]Daniil Gurgorov, Michal Gregor, Josef van Genabith, and Simon Ostermann. On multilingual encoder language model compression for low-resource languages. arXiv preprint arXiv:2505.16956, 2025.
- [11]Harold Stanley Heaps. Information Retrieval: Computational and Theoretical Aspects. Academic Press, 1978.
- [12]Hugging Face. safetensors: Simple, safe way to store and distribute tensors. https://github.com/huggingface/safetensors, 2022.
- [13]Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2018.arxiv.org/abs/1808.06226
- [14]Eugene Kwek and Wenpeng Yin. COMPACT: Common-token optimized model pruning across channels and tokens. arXiv preprint arXiv:2509.06836, 2025.arxiv.org/abs/2509.06836
- [15]Sander Land and Max Bartolo. Fishing for Magikarp: Automatically detecting under-trained tokens in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
- [16]Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.15425
- [17]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI technical report, 2019.
- [18]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, 2023.arxiv.org/abs/2212.04356
- [19]Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021.
- [20]Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016.arxiv.org/abs/1508.07909
- [21]Sattaya Singkul. Reading the architecture off the weights: Manifest-only model anatomy and a census of 12,974 Hugging Face repositories. https://doi.org/10.13140/RG.2.2.19473.19048, 2026. Preprint, ResearchGate.
- [22]SqueezeBits. Vocabulary trimming: An easy and effective method for SLM acceleration. https://blog.squeezebits.com/vocabulary-trimming-methods, 2025.
- [23]Arthit Suriyawongkul, Ekapol Chuangsuwaniсh, Pattarawat Chormai, and Charin Polpanumas. Wisesight sentiment corpus. https://github.com/PyThaiNLP/wisesight-sentiment, 2019.DOI
- [24]The Unicode Consortium. Unicode standard annex #24: Unicode script property. https://www.unicode.org/reports/tr24/.






