betaFeedback

WorksComputer Science

A Corpus-Driven SentencePiece Unigram Tokenizer for Sindhi: Design and Benchmark Against Fourteen Existing Tokenizers

Abstract

Sindhi is a low-resource Indo-Aryan language written mainly in an extended Perso-Arabic script. Several Sindhi tokenizers have been released on public model hubs, but they have not been compared with one another on shared data. We train SentencePiece Unigram tokenizers with byte fallback at four vocabulary sizes (32k to 150k) on about 1.10 million Sindhi sentences and a smaller, repeated Urdu, Persian and Arabic slice, from four public sources, and compare them with fourteen existing tokenizers on four evaluation sets: FLORES+, Sindhi Wikipedia, a news-derived sentiment corpus, and a contemporary news corpus. The metrics are fertility, single-token retention, UNK rate, byte-fallback rate, and round-trip character loss. At 64k, our tokenizer averages 1.147 tokens per word, against 1.208 for the best existing tokenizer and 1.512 for XLM-RoBERTa, and keeps 88.6% of words as single tokens. On fertility, single-token retention and UNK rate it is best or tied for best in all four evaluation sets at 64k and above; at 32k it trails one baseline on FLORES+. Margins over the best Sindhi-adapted baselines are small (0.002 fertility on FLORES+ at 64k), and we report no significance tests. A round-trip check finds no character loss for our tokenizer and silent character deletion (up to 0.85% of characters) in two existing ones; seven others lose less than 0.1%. We evaluate intrinsic metrics only, so effects on downstream tasks remain untested. Code and tokenizer files are public (DOI: 10.5281/zenodo.23962474).

Supervisor: Dr. Attaullah Sahito, Head of the Department of Computer Science

Introduction

Sindhi is an Indo-Aryan language spoken in the Sindh province of Pakistan and in parts of India, with about 33.5 million speakers in Sindh province alone in the 2023 Pakistani census ([12]). It is written mainly in an extended Perso-Arabic script. Sindhi is a morphologically rich language with few public text resources ([5]), and word segmentation is itself difficult because of space omission and space insertion ([6]).

Tokenization is the first step of every language model. It sets the sequence length for a given text, and with it the cost and the usable context, and the gap can be large between languages ([26]). Tokenizer design also affects downstream accuracy ([28]; [29]). A tokenizer trained mostly on other languages tends to split text in a low-resource language into many pieces, and byte-level models can need several times as many tokens for some languages ([26]).

Several Sindhi tokenizers have been released on public model hubs. Each is documented on its own page with its own test text, and we found no study that compares them (the search is described in § 4, and we do not claim it is exhaustive). This paper makes four contributions:

  1. A comparison of fourteen existing tokenizers on four evaluation sets with one set of metrics, and a documented search protocol for finding them (§ 4).

  2. A SentencePiece Unigram tokenizer with byte fallback at four vocabulary sizes, released with training code, corpus provenance and license notes ([23]) (§ 5 and § 11).

  3. A round-trip check that counts characters deleted by encoding and decoding, applied to every tokenizer. It finds silent deletion in two existing tokenizers that standard metrics do not flag (§ 7.3).

  4. Two design choices that did not help in our setting (clitic pre-marking and forcing frequent words to single tokens), and a record of the corrections we made to our own evaluation code (§ 5.3 and § 6.4).

All results are intrinsic. We train no language models and evaluate no downstream tasks, and the margins over the closest competitors are small. Section 9 lists the limitations.

Background: Sindhi Script and Text

Sindhi uses letters that have no counterpart in the Arabic or Persian alphabets: implosives (ڳ ڏ ڄ ٻ), aspirated consonants written with single letters (ڀ ڦ ٺ ڇ ٿ), and the retroflex nasal ڻ. Urdu writes aspirated consonants as digraphs rather than single letters. A tokenizer that lacks these letters must split them into bytes or drop them.

Text in the wild mixes code points for visually similar letters, for example U+0647 and U+06C1 for heh, and some text uses Arabic Presentation Forms. SentencePiece applies Unicode normalization before training; we use its default rule, nmt_nfkc, and apply no Sindhi-specific normalization. We treat all code points that occur in the evaluation text as input to be preserved, and we take no position on their status in standard orthography.

Word boundaries are unreliable in Sindhi, so word-based metrics are noisy [6]. Our fertility and single-token retention use whitespace-delimited words, and they inherit this noise. This study covers Arabic-script Sindhi only. Sindhi is also written in Devanagari in India, and we do not evaluate it.

Related Work

Subword tokenization

Byte-pair encoding (BPE) builds a vocabulary by repeatedly merging the most frequent symbol pair [30]. The Unigram language model instead starts from a large candidate vocabulary and keeps the pieces that maximize the likelihood of the training text [19]. SentencePiece implements both algorithms, treats text as a raw character stream, and is designed so that decoding the encoded normalized text returns the normalized text [20]. Bostrom and Durrett [10] report that Unigram tokenization recovers units closer to morphology than BPE and matches or outperforms BPE in pretraining experiments on English and Japanese. We use Unigram for Sindhi without testing it against BPE (Section 9).

Evaluating tokenizers

Rust et al. [28] show that the tokenizer is a major factor in the monolingual performance of multilingual models. Petrov et al. [26] measure tokenization parity on parallel FLORES text and find large differences in encoded length across languages. Schmidt et al. [29] test the hypothesis that fewer tokens give better downstream performance and find that it does not hold in general. Fertility and compression are therefore proxies, and we treat them as such.

Tokenization for Indic languages and Sindhi

Pattnayak et al. [25] compare BPE, SentencePiece and character-level tokenization for named entity recognition in low-resource Indic languages, and report results for Sindhi, which is written in Arabic script. Work on Sindhi text resources has built a web corpus of more than 61 million words for word embeddings [5] and a neural word segmenter [6]. We do not know of a published comparison of Sindhi tokenizers on shared evaluation text. The tokenizers we compare (Section 4) are distributed as model-hub repositories without accompanying papers.

Baseline Tokenizers

We evaluate fourteen existing tokenizers in three groups (Table 1). The first group has four Sindhi-specific tokenizers found at the start of the project: three released under the name SindhiLM (v1 to v3, differing in algorithm and vocabulary size) and one independent Sindhi BPE tokenizer. The second has six general-purpose multilingual tokenizers that were not trained with Sindhi as a primary target. The third has four Sindhi-adapted tokenizers found later: mBERT extended with a 20K-token Sindhi Unigram vocabulary, TinyLlama extended with 10,681 Sindhi tokens, and two tokenizers from the hellosindh organization.

Search protocol. We searched the Hugging Face Hub and GitHub with ten search terms (“Sindhi Tokenizer”, “Sindhi NLP”, “Sindhi Subword Tokenization”, “Sindhi Arabic Script”, “Sindhi BPE”, “Sindhi SentencePiece”, “Sindhi Language Model”, “Sindhi Tokenization Benchmark”, “Low Resource Language Tokenizer” and “Indic Language Tokenization”) for public tokenizer files that load without modification and are described as supporting Sindhi. The first search, on 15 September 2026, found ten tokenizers. Four more were found after the first benchmark had been run, and all results were recomputed with the full set. A final search before submission found no further tokenizers on 27 September 2026. We loaded every tokenizer from its own repository between 15 and 27 September 2026 without pinning a revision. The benchmark notebook in the repository records each tokenizer’s repository ID and a hash of its vocabulary.

TokenizerGroupVocabularyReference
SindhiLM v1Sindhi-specific (initial)43(aakashMeghwar01, n.d.a)
SindhiLM v2Sindhi-specific (initial)151,643(aakashMeghwar01, n.d.b)
SindhiLM v3Sindhi-specific (initial)32,000(aakashMeghwar01, n.d.c)
FahadQazi BPESindhi-specific (initial)13,031(fahadqazi, n.d.)
XLM-RoBERTaGeneral multilingual250,002[11]
mT5General multilingual250,100[34]
Qwen2.5General multilingual151,643[27]
BLOOMGeneral multilingual250,680[9]
Gemma 3General multilingual262,144[14]
Llama 3.2General multilingual128,000[21]
mBERT-SindhiSindhi-adapted (later search)119,547(Kashif786, n.d.)
TinyLlama-SindhiSindhi-adapted (later search)42,681(jamalimubashirali, n.d.)
hellosindh-BERTSindhi-adapted (later search)32,000(hellosindh, n.d.a)
hellosindh-NERSindhi-adapted (later search)32,001(hellosindh, n.d.b)

Table 1. Baseline tokenizers. Vocabulary sizes are as loaded by the evaluation code. The Llama 3.2 vocabulary is 128,000 regular entries, as reported by the loaded tokenizer.

SindhiLM v2 is a re-published copy of the Qwen2.5 tokenizer: fertility, single-token retention and total token counts are identical to three decimal places in every evaluation set. We keep it in the tables as one row, “Qwen2.5 / SindhiLM v2”, so the comparison has 13 distinct baseline rows for 14 tokenizers. SindhiLM v1 has a vocabulary of 43 entries. The two hellosindh tokenizers differ in vocabulary size, file format and tokenization behavior on identical input, so we treat them as distinct.

files when we visited it. Text was split into sentences at the Arabic full stop, period, exclamation mark, Arabic question mark and line breaks, and fragments of three characters or fewer were dropped. We applied no word-count limit and no language filter. A diagnostic on the shaikhsalman sample found that 11.3% of its sentences were mostly Latin script. We did not remove them, so the corpus contains some English text (Section 9).

Our Tokenizer

Training corpus

The corpus has 1,455,915 lines and about 122.4 million characters from four sources (Table 2). The Sindhi side has 1,098,119 sentences and 102.7 million characters. The Urdu, Persian and Arabic side has 89,449 unique sentences and 4.9 million characters; we repeated it four times so that it makes up 16.1% of the characters, which means that about 358,000 of the 1.46 million lines are repeats. We chose sources that do not overlap with the corpus that the SindhiLM model card names as its training data, the Sindhi Mega Corpus of ambile-official [8], so that our results would not reproduce theirs by construction. The separation is not complete: that corpus lists Sindhi Wikipedia among its seven sources, and so do we. We could not inspect it, because its dataset page showed no files when we visited it.

SourceContent usedLicense
Sindhi Wikipedia (Wikimedia Foundation, n.d.)6,000 articles (articles 2,001 to 8,000 of a seeded shuffle; the first 2,000 are held out for evaluation)CC BY-SA
Sindhi-NLP (Sindhi-NLP, n.d.)Two Twitter accounts, Awami Awaz articles, Daily Kawish articlesMIT (repository)
shaikhsalman/sindhi-llm-corpus (shaikhsalman, n.d.)20,000-row sample (seed 7), no language filter; about 57% of the Sindhi-side sentencesNone stated
Urdu, Persian and Arabic Wikipedia (Wikimedia Foundation, n.d.)1,500 articles each, for shared script and loanwords; repeated four timesCC BY-SA

Table 2. Training corpus. After repetition, the cross-lingual slice makes up 16.1% of the corpus by character count.

Training

We train with SentencePiece [20] using the Unigram model type, full character coverage and byte fallback (Table 3). Four vocabulary sizes were trained: 32,000, 64,000, 100,000 and 150,000. Byte fallback means that no character is mapped to the unknown token, and we use full character coverage so that every character seen in training, including the Sindhi letters listed in § 2, has its own piece. The 64,000 model is the default (§ 8).

SettingValue
model_typeunigram
vocab_size32,000; 64,000 (default); 100,000; 150,000
character_coverage1.0
byte_fallbacktrue
split_by_whitespacetrue
unk_id, bos_id, eos_id, pad_id0, 1, 2, disabled (−1)
normalization_rule_namenmt_nfkc (SentencePiece default)
Seed, input sentence sampling, library versionsNo seed set; input_sentence_size at its default, so all sentences are used; library versions not recorded (Kaggle default image)

Table 3. Training settings (notebooks/training.ipynb in the repository; training/train_tokenizer.py holds the same settings).

Design choices tested and dropped

Two choices motivated by work on clitic-heavy languages were tested and removed. Each is a single experiment on one corpus, so we report them as what happened in our setting, not as general findings.

Clitic pre-marking. We separated six common postpositions and clitics with spaces before training, guided by a corpus-wide diagnostic of fused forms. For three of the six, 17% to 39% of occurrences looked fused, but a control character with no plausible attachment showed 0.2%, and the apparent fusion was overlap with unrelated words. We removed the preprocessing step.

Forced single tokens. We forced the same six clitics and two frequent loanwords to be single tokens with user_defined_symbols. The final Unigram training objective at 32k was about 14.9 with forcing and about 9.6 without, and the targeted words needed more tokens in total. We removed forcing, and all target words became single tokens without it. Objective values from runs with different constraints are not strictly comparable, so this result supports only the weaker statement that forcing did not help. Held-out fertility for both runs would be a better comparison, and we did not run it.

Evaluation Setup

Evaluation sets

We evaluate on four sets (Table 4), all in full with no sampling. FLORES+ ([24]) is a translated benchmark. The Sentiment set was built from Kawish and Awami Awaz newspaper corpora (alinawazmahar, n.d.), so we call it news-derived, and we built it ourselves. The Sindh Line corpus ([22]) was downloaded manually under its CC BY-SA 4.0 license because it has no scriptable interface.

The Wikipedia set is disjoint from training by article. The Sentiment set and the Sindhi-NLP part of the training corpus draw on the same newspapers (Awami Awaz and Daily Kawish). The specific files do not overlap, but we did not run a sentence-level near-duplicate check, so results on this set may favor our tokenizer. For the Sindh Line set we found no overlap in sources, and we did not run a sentence-level check either. Removing the Sentiment set does not change the 64k result (§ 7.1).

Metrics

Fertility is the number of tokens divided by the number of whitespace-delimited words. Each word is encoded on its own, without special tokens. Single-token retention is the percentage of words encoded as one token. UNK rate is the percentage of output tokens equal to the tokenizer’s unknown token. Byte-fallback rate is the percentage of output tokens whose standalone decoding contains the replacement character U+FFFD. We report it as a diagnostic and not as a score, because a low value can mean that a tokenizer drops characters (§ 7.3).

Round-trip character loss encodes and decodes each sentence, then normalizes the original and the decoded text with NFKC and removes zero-width characters, bidirectional control characters and all whitespace, so that routine normalization is not counted. Loss is the sum of positive length differences divided by the number of characters in the

SetSourceSentencesWordsType
FLORES+snd_Arab, devtest [24]1,01225,356Translated benchmark
Wikipedia20231101.sd, first 2,000 articles, 3 to 100 words (Wikimedia Foundation, n.d.)25,480435,947Encyclopedic
SentimentSindhi_Sentiment_dataset (alinawazmahar, n.d.)3,53647,156News-derived
Sindh LineSindh Line Publishers [22]87,1581,014,998News

caption: Evaluation sets.

originals. A sentence is “affected” if its decoded text is shorter. The measure counts net deletions only, so a substituted or inserted character is not detected.

Encoding each word on its own differs from encoding it in context for byte-level BPE tokenizers that mark a leading space, and it can add a standalone word-boundary piece for some SentencePiece vocabularies. We did not correct for either. Every tokenizer was run with its own default normalization.

Aggregation

The sets differ in size by a factor of 86 (1,012 to 87,158 sentences). The main table uses an unweighted mean over the four sets. A pooled mean would be dominated by the Sindh Line set. Per-set values are in Appendix § 13.

Corrections to the evaluation code

We corrected two problems in our own code. First, byte-fallback was first detected by matching byte-piece strings of the form <0xNN>. That method reported 0% for several tokenizers whose rate was between 1% and 30%, including XLM-RoBERTa, which changed from 0.000% to above 1%. We replaced it with the standalone-decode check in § 6.2, and excluded tokens that decode to an empty string (such as the lone SentencePiece word-boundary piece), because counting them inflated the rates of SentencePiece tokenizers. Second, after the four later tokenizers were added, we re-ran everything. Two words in the word-level test (§ 7.4) that had been outright wins for our tokenizer became ties. Other corrections made during development, logged in docs/explainer.md in the repository, include a tokenizer loader that silently loaded only three pieces, a training run in a forked process that silently produced empty models, an unencoded space in a download URL that had silently left out about 9,500 Sindhi-NLP articles, and two rewrites of the round-trip check to stop it counting normalization and whitespace changes as losses.

Results

Main results

Table 5 gives the mean over the four sets, sorted by fertility. At 64k and above, our tokenizer has the lowest fertility, the highest single-token retention and the lowest UNK rate (tied at zero with several others) in every one of the 12 cells formed by three metrics and four sets. No baseline is strictly better in any of those cells. The same holds in the 9 cells that remain when the Sentiment set is removed. At 32k, our tokenizer is best or tied in 10 of 12 cells and is behind TinyLlama-Sindhi on FLORES+ fertility (1.233 against 1.193) and single-token retention (82.4% against 84.4%). We checked whether TinyLlama-Sindhi’s deletion of one character (§ 7.3) explains this by normalizing the deleted character and re-measuring. Fertility changed by less than 0.002, so the 32k result stands.

At 64k, mean fertility is 24.1% lower than XLM-RoBERTa (1.147 against 1.512) and 60.4% lower than Llama 3.2 (2.896). Those tokenizers were not built for Sindhi. The relevant comparison is with the Sindhi-adapted tokenizers. Against the nearest one, TinyLlama-Sindhi, the mean reduction is 5.0% (0.061 tokens per word). Table 6 shows the margin in each set.

SetFertility: OursBest baselineDiff.Single-token %: OursBest baselineDiff.
FLORES+1.1911.193 (TinyLlama-Sindhi)0.00284.784.4 (TinyLlama-Sindhi)+0.3
Wikipedia1.1941.272 (hellosindh-NER)0.07886.982.6 (hellosindh-NER)+4.3
Sentiment1.1101.149 (TinyLlama-Sindhi)0.03990.387.6 (TinyLlama-Sindhi)+2.7
Sindh Line1.0931.155 (hellosindh-NER)0.06292.487.9 (hellosindh-NER)+4.5
Mean1.1471.208 (TinyLlama-Sindhi)0.06188.684.4 (TinyLlama-Sindhi)+4.2

Table 6. Margin of Ours 64k over the best baseline in each set. Differences are absolute (fertility in tokens per word, single-token retention in percentage points). No confidence intervals are available.

On FLORES+ the margin at 64k is 0.002 fertility and 0.3 points of single-token retention. Without confidence intervals we treat this as parity. On the other three sets the margin is 0.039 to 0.078 tokens per word.

Vocabulary size confounds fertility, since a larger vocabulary generally lowers it. Table 7 compares tokenizers with about 32,000 entries. Ours at 32k has lower fertility than all three (1.184 against 1.218 to 1.960). The strongest baselines mostly have larger vocabularies (TinyLlama-Sindhi 42,681; mBERT-Sindhi 119,547), and Ours 32k still has lower mean fertility than TinyLlamaSindhi.

TokenizerVocabFertilitySingle-token %UNK %
Ours 32k32,0001.18486.50.000
hellosindh-NER32,0011.21883.70.194
SindhiLM v332,0001.73055.83.605
hellosindh-BERT32,0001.96035.50.122
TinyLlama-Sindhi (larger vocabulary)42,6811.20884.40.000

Table 7. Tokenizers with about 32,000 entries, with TinyLlama-Sindhi as a larger-vocabulary reference. Means over four sets.

Gains diminish with vocabulary size: mean fertility is 1.184, 1.147, 1.133 and 1.125 at 32k, 64k, 100k and 150k. Going from 64k to 150k lowers fertility by 1.9%. The 64k model is the smallest size that is best or tied in all 12 cells, and we use it as the default. The recommendation rests on intrinsic metrics only.

Per-set results

Figure 2 shows fertility in each evaluation set for our 64k tokenizer and the four strongest baselines. Ours has the lowest value in all four sets; on FLORES+, TinyLlama-Sindhi is within 0.002.

Round-trip character loss

TinyLlama-Sindhi and FahadQazi BPE both report a UNK rate of exactly zero, and TinyLlama-Sindhi also a byte-fallback rate of zero (Table 5), yet they both delete characters. We first saw this for TinyLlama-Sindhi, when emoji, Chinese, Cyrillic and mathematical-symbol strings decoded to shorter strings with neither UNK nor byte-fallback tokens. Table 8 shows the effect on real text. Five tokenizers, including all four of ours, show exactly zero loss in every set. The remaining tokenizers lose less than 0.1% of characters, though up to 1.13% of sentences are affected (SindhiLM v3).

TokenizerVocabFertilitySingle-token %UNK %Byte-fallback %Max loss %
Ours 150k150,0001.12589.70.0000.1540.0000
Ours 100k100,0001.13389.30.0000.1490.0000
Ours 64k64,0001.14788.60.0000.1380.0000
Ours 32k32,0001.18486.50.0000.1130.0000
TinyLlama-Sindhi†42,6811.20884.40.0000.0000.4147
hellosindh-NER32,0011.21883.70.1940.1970.0751
mBERT-Sindhi119,5471.27878.40.1430.0000.0021
XLM-RoBERTa250,0021.51265.40.0341.4020.0040
SindhiLM v332,0001.73055.83.6050.0000.0754
FahadQazi BPE†13,0311.91939.10.0007.1270.8461
hellosindh-BERT32,0001.96035.50.1220.1140.0089
mT5250,1002.28021.70.02212.7460.0040
Gemma 3262,1442.30826.80.0000.1050.0000
Qwen2.5 / SindhiLM v2151,6432.58023.80.0008.6650.0000
BLOOM250,6802.62724.70.00031.6600.0000
Llama 3.2128,0002.89618.20.00028.4720.0000
SindhiLM v1434.3060.114.0170.0000.0790

Table 5. Mean over the four evaluation sets, sorted by fertility (lower is better); single-token retention is higher-is-better. Byte-fallback is a diagnostic. “Max loss” is the largest round-trip character loss in any set (Table 8). Rows for our tokenizer are shaded. † Zero UNK rate comes with silent character deletion (§ 7.3).

FahadQazi BPE deletes individual Latin letters in code-switched text. For example, the sentence “An update on my situation, stay safe everyone” decodes with the letters A, p, f and v, among others, removed. This affects up to 16.15% of Wikipedia sentences, where citations and transliterations are common. TinyLlama-Sindhi deletes every occurrence of U+06C1 (ARABIC LETTER HEH GOAL), which occurs in common function words such as ہ, بہ and نہ, and it corrupts text written in the Arabic Presentation Forms block, in the worst case reducing a sentence to seven isolated characters. U+06C1 appears in 24.8% of FLORES+ sentences and 6.2% of Wikipedia sentences, which explains why its loss is largest on FLORES+.

Zero loss for our tokenizer is expected from the design, since byte fallback keeps every character representable, and NFKC is applied to both sides before comparison. The informative result is for the baselines. Lossless decoding is a design goal of SentencePiece ([20]). We use round-trip loss as a reported metric for comparing tokenizers, not as a unit test, and we recommend reporting it next to fertility and UNK rate.

Mean fertility against vocabulary size

Figure 1. Mean fertility against vocabulary size (log scale). Our tokenizers (line) lie below the baselines at every size. SindhiLM v1 (vocabulary 43, fertility 4.306) is omitted.

Word-level test

We tokenized 22 hand-selected words (four categories: genitive constructions, inflectional forms, aspirated digraphs, and loanwords) with every tokenizer and compared token counts (Appendix § 14). Our tokenizer ties the best competitor on 21 of 22 words and is behind on one (هنجي, 2 tokens against 1 for FahadQazi BPE and mBERT-Sindhi). It wins outright on none. The list was fixed before the four later tokenizers were found and re-run unchanged. This test counts tokens only. It does not check where the boundaries fall, so it is not a morphological evaluation. It shows that several competitors are already near-optimal on common, well-behaved words.

Discussion

The results support three statements. First, the new tokenizer has lower mean fertility and higher single-token retention than all fourteen existing tokenizers in this benchmark, by a large margin over general multilingual tokenizers and by a small one (5%) over the best Sindhi-adapted tokenizer. Second, the advantage over the closest competitors is not uniform: it is 0.002 on FLORES+ and 0.04 to 0.08 tokens per word elsewhere. Third, UNK and byte-fallback rates can look excellent because a tokenizer discards what it cannot represent, and the round-trip check is what exposes this.

We do not know why our tokenizer does better. Plausible factors are a larger and more varied training corpus than the baselines used, and the Unigram algorithm, but we did not train a BPE tokenizer on the same corpus or vary corpus size, and the baselines differ in vocabulary size, algorithm and training data that is mostly undocumented. These remain hypotheses.

Lower fertility does not by itself make a better language model ([29, 28]). A 5% reduction in tokens per word may or may not matter downstream. The byte-fallback correction in § 6.4 is a reminder that a plausible detection heuristic can silently misreport. A heuristic that reported 0% for XLM-RoBERTa was wrong, and the corrected value is above 1%. We have not broken the corrected value down by piece for XLM-RoBERTa or mT5.

Figure 2. Fertility by evaluation set for Ours 64k and the four strongest baselines. Full per-set results for all tokenizers are in Table A1.

TokenizerFert.Single %UNK %Byte-fb. %TokenizerFert.Single %UNK %Byte-fb. %
Ours 150k1.16785.90.0000.199Ours 150k1.15788.60.0000.179
Ours 100k1.17685.50.0000.198Ours 100k1.17188.00.0000.173
Ours 64k1.19184.70.0000.172Ours 64k1.19486.90.0000.167
Ours 32k1.23382.40.0000.099Ours 32k1.24984.00.0000.158
TinyLlama-Sindhi1.19384.40.0000.000TinyLlama-Sindhi1.29280.50.0000.000
hellosindh-NER1.26179.70.1060.109hellosindh-NER1.27282.60.4950.411
mBERT-Sindhi1.34173.90.0560.000mBERT-Sindhi1.37373.50.4400.000
XLM-RoBERTa1.55261.80.0101.303XLM-RoBERTa1.58963.10.1111.774
SindhiLM v31.82452.74.6500.000SindhiLM v31.72655.44.9490.000
FahadQazi BPE1.94037.70.0007.199FahadQazi BPE2.00237.40.0008.613
hellosindh-BERT1.98334.50.0680.070hellosindh-BERT1.98836.10.3170.246
mT52.31819.90.00713.040mT52.28522.10.07713.079
Gemma 32.34125.60.0000.000Gemma 32.31728.00.0000.259
Qwen2.5 / SindhiLM v22.61722.30.0009.223Qwen2.5 / SindhiLM v22.56825.50.0007.498
BLOOM2.69223.40.00032.700BLOOM2.56925.30.00029.959
Llama 3.22.95117.00.00030.015Llama 3.22.85618.50.00026.716
SindhiLM v14.3070.014.0190.000SindhiLM v14.2710.215.1370.000
TokenizerFert.Single %UNK %Byte-fb. %TokenizerFert.Single %UNK %Byte-fb. %
Ours 150k1.09991.00.0000.137Ours 150k1.07893.30.0000.100
Ours 100k1.10390.80.0000.133Ours 100k1.08493.00.0000.091
Ours 64k1.11090.30.0000.126Ours 64k1.09392.40.0000.085
Ours 32k1.13089.00.0000.126Ours 32k1.12490.50.0000.068
TinyLlama-Sindhi1.14987.60.0000.000TinyLlama-Sindhi1.19885.20.0000.000
hellosindh-NER1.18284.60.0840.068hellosindh-NER1.15587.90.0900.201
mBERT-Sindhi1.22381.20.0400.000mBERT-Sindhi1.17685.00.0380.000
XLM-RoBERTa1.43469.30.0091.133XLM-RoBERTa1.47567.40.0071.400
SindhiLM v31.73955.32.9410.000SindhiLM v31.63059.91.8810.000
FahadQazi BPE1.83142.20.0006.318FahadQazi BPE1.90339.30.0006.376
hellosindh-BERT1.91536.20.0520.020hellosindh-BERT1.95335.20.0530.118
mT52.25922.40.00112.120mT52.25922.30.00412.746
Gemma 32.27627.00.0000.162Gemma 32.29926.70.0000.000
Qwen2.5 / SindhiLM v22.56223.50.0008.968Qwen2.5 / SindhiLM v22.57423.90.0008.969
BLOOM2.64524.70.00032.653BLOOM2.60025.50.00031.328
Llama 3.22.90718.40.00029.384Llama 3.22.87119.00.00027.774
SindhiLM v14.2580.112.5420.000SindhiLM v14.3880.014.3680.000

Table A1. Per-set results for 17 tokenizer rows (four sizes of ours and 13 baseline rows, with SindhiLM v2 merged into Qwen2.5). Fertility is tokens per word; the other columns are percentages.

TokenizerMax loss %SetMax sentences affected %Set
FahadQazi BPE0.8461Wikipedia16.15Wikipedia
TinyLlama-Sindhi0.4147FLORES+24.90FLORES+
SindhiLM v10.0790Wikipedia0.69Wikipedia
SindhiLM v30.0754Wikipedia1.13Wikipedia
hellosindh-NER0.0751Wikipedia0.49Wikipedia
hellosindh-BERT0.0089Wikipedia0.09Wikipedia
XLM-RoBERTa0.0040Wikipedia0.04Wikipedia
mT50.0040Wikipedia0.04Wikipedia
mBERT-Sindhi0.0021Wikipedia0.04Wikipedia

Table 8. Round-trip character loss: the largest value over the four sets, with the set where it occurs. Tokenizers with zero loss in every set: Ours (all sizes), Qwen2.5 / SindhiLM v2, BLOOM, Gemma 3, Llama 3.2. Per-set values are in Table A2.

TokenizerFLORES+ lossFLORES+ aff.Wiki lossWiki aff.Sent. lossSent. aff.Sindh Line lossSindh Line aff.
Ours (all sizes)0.00000.000.00000.000.00000.000.00000.00
Qwen2.5 / SindhiLM v20.00000.000.00000.000.00000.000.00000.00
BLOOM0.00000.000.00000.000.00000.000.00000.00
Gemma 30.00000.000.00000.000.00000.000.00000.00
Llama 3.20.00000.000.00000.000.00000.000.00000.00
XLM-RoBERTa0.00000.000.00400.040.00000.000.00300.02
mT50.00000.000.00400.040.00000.000.00300.02
mBERT-Sindhi0.00000.000.00210.040.00000.000.00010.00
hellosindh-BERT0.00000.000.00890.090.00000.000.00310.02
SindhiLM v10.00000.000.07900.690.00000.000.00050.01
SindhiLM v30.00100.100.07541.130.00860.310.00290.04
hellosindh-NER0.00210.100.07510.490.00170.060.00720.11
TinyLlama-Sindhi0.414724.900.31146.950.04091.670.04131.44
FahadQazi BPE0.410515.810.846116.150.08471.870.22726.08

Table A2. Round-trip character loss by set. “Loss” is the percentage of characters lost; “Aff.” is the percentage of sentences with at least one lost character. Sentiment is abbreviated “Sent.”

Limitations

  1. Intrinsic evaluation only. We train no language model and run no downstream task. The default vocabulary size and the ranking of tokenizers rest on proxy metrics.

  1. No significance tests. We report no confidence intervals. The 64k margin on FLORES+ (0.002) should be read as parity.

  1. No ablation. We did not compare Unigram with BPE on the same corpus, and did not vary corpus size, so we cannot attribute the advantage to either factor.

  1. Evaluation-set independence. The Sentiment set shares newspaper sources with part of the training corpus and was built by the authors. We ran no sentence-level near-duplicate check on it or on the Sindh Line set.

  1. Training corpus quality. The training text was not filtered by language or length. The shaikhsalman sample contains English text (11.3% of its sentences were mostly Latin script in our diagnostic), the Urdu, Persian and Arabic slice is repeated four times, and Sindhi Wikipedia may also appear in the SindhiLM training corpus. We did not measure how these choices affect the results.

  1. Baseline coverage and protocol. The search for tokenizers may be incomplete. We know of two further public Sindhi tokenizers that we did not evaluate: the Sindhi Wikipedia tokenizers in the wikilangs/sd repository (8k to 64k) and the SindhiFormer tokenizer by the author of SindhiLM. Baselines were run with default normalization and with words encoded in isolation, and their vocabulary sizes differ.

  1. Metric limits. Whitespace-delimited words are noisy in Sindhi [6]. The byte-fallback detector counts any token whose standalone decoding contains U+FFFD, and we did not inspect which pieces trigger it for each tokenizer. Round-trip loss counts net deletions only. The unweighted mean over four sets is a choice, not a given.

  1. No classical or literary Sindhi. Five candi- @page 8/13 starts_mid=1 ends_mid=1

Figure 2 (PDF p. 8). Fertility by evaluation set for Ours 64k and the four strongest baselines. Full per-set results for all tokenizers are in Table A1.

TokenizerMax loss %SetMax sentences affected %Set
FahadQazi BPE0.8461Wikipedia16.15Wikipedia
TinyLlama-Sindhi0.4147FLORES+24.90FLORES+
SindhiLM v10.0790Wikipedia0.69Wikipedia
SindhiLM v30.0754Wikipedia1.13Wikipedia
hellosindh-NER0.0751Wikipedia0.49Wikipedia
hellosindh-BERT0.0089Wikipedia0.09Wikipedia
XLM-RoBERTa0.0040Wikipedia0.04Wikipedia
mT50.0040Wikipedia0.04Wikipedia
mBERT-Sindhi0.0021Wikipedia0.04Wikipedia

Table 8 (PDF p. 8). Round-trip character loss: the largest value over the four sets, with the set where it occurs. Tokenizers with zero loss in every set: Ours (all sizes), Qwen2.5 / SindhiLM v2, BLOOM, Gemma 3, Llama 3.2. Per-set values are in Table A2.

wrong, and the corrected value is above 1%. We have not broken the corrected value down by piece for XLM-RoBERTa or mT5.

Limitations

  1. Intrinsic evaluation only. We train no language model and run no downstream task. The default vocabulary size and the ranking of tokenizers rest on proxy metrics.

  1. No significance tests. We report no confidence intervals. The 64k margin on FLORES+ (0.002) should be read as parity.

  1. No ablation. We did not compare Unigram with BPE on the same corpus, and did not vary corpus size, so we cannot attribute the advantage to either factor.

  1. Evaluation-set independence. The Sentiment set shares newspaper sources with part of the training corpus and was built by the authors. We ran no sentence-level near-duplicate check on it or on the Sindh Line set.

  1. Training corpus quality. The training text was not filtered by language or length. The shaikhsalman sample contains English text (11.3% of its sentences were mostly Latin script in our diagnostic), the Urdu, Persian and Arabic slice is repeated four times, and Sindhi Wikipedia may also appear in the SindhiLM training corpus. We did not measure how these choices affect the results.

  1. Baseline coverage and protocol. The search for tokenizers may be incomplete. We know of two further public Sindhi tokenizers that we did not evaluate: the Sindhi Wikipedia tokenizers in the wikilangs/sd repository (8k to 64k) and the SindhiFormer tokenizer by the author of SindhiLM. Baselines were run with default normalization and with words encoded in isolation, and their vocabulary sizes differ.

  1. Metric limits. Whitespace-delimited words are noisy in Sindhi [6]. The byte-fallback detector counts any token whose standalone decoding contains U+FFFD, and we did not inspect which pieces trigger it for each tokenizer. Round-trip loss counts net deletions only. The unweighted mean over four sets is a choice, not a given.

  1. No classical or literary Sindhi. Five candi-

date sources were investigated, including Bhittaipedia and the Shah Jo Risalo labeled dataset (Abdul Majid Bhurgri Institute of Language Engineering, n.d.). They failed for different reasons: a corrupted CSV export, an unrelated SQL dump, two unusable OCR scans (one with an incompatible license), and a subscription-gated platform.

  1. Scope. Arabic-script Sindhi only. Dialects are not examined, and the tokenizer is not validated for speech recognition or text-to-speech.

Data, Code and License

The code is released under the MIT license and the tokenizer files under CC BY-SA 4.0 [23]. We chose CC BY-SA because part of the training data is share-alike Wikipedia text; this is a cautious default and not legal advice. The repository redistributes no training text. The shaikhsalman corpus states no license and supplied about 57% of the Sindhi-side training sentences. The Sindhi-NLP repository is MIT-licensed, but the tweets and newspaper articles in it may carry other rights. A tokenizer vocabulary can contain frequent strings from its training text, including personal and place names. Users should check these terms for their own use. Training and evaluation sources, sizes and licenses are listed in DATA_SOURCES.md in the repository. The evaluation code, per-set results and the exact release used here are in the repository at release v1.0.0 (Zenodo DOI: 10.5281/zenodo.23962474). All Hugging Face, GitHub and Mozilla Data Collective resources were accessed between 15 and 27 September 2026, except the Sindhi Mega Corpus [8] (4 November 2026).

Author contributions. A.N.: conceptualization, methodology, software, data curation, investigation, formal analysis, writing – original draft. M.F. and S.S.: validation, visualization, writing – review and editing. A.S.: supervision.

Acknowledgments. The authors thank Dr. Attaullah Sahito for supervision and guidance, Kaggle for the compute used for training and evaluation, and the creators of the datasets we used: the Sindhi-NLP project, shaikhsalman, Sindh Line Publishers, Open Language Data (FLORES+) and the Wikimedia Foundation.

Funding. This research received no external funding.

Competing interests. The authors declare no competing interests.

Conclusion

A SentencePiece Unigram tokenizer trained on about 1.10 million Sindhi sentences and a smaller, repeated Urdu, Persian and Arabic slice, from four public sources reaches a mean fertility of 1.147 tokens per word at 64k, against 1.208 for the best of fourteen existing tokenizers. It is best or tied for best on fertility, single-token retention and UNK rate in every evaluation set at 64k and above, with a margin of 0.002 on FLORES+ and 0.04 to 0.08 elsewhere. It loses no characters in a round-trip check, while two existing tokenizers silently delete characters (up to 0.85% of characters) and seven others lose less than 0.1%. These are intrinsic results on four evaluation sets and carry the limits in Section 9. The next steps are to train small language models with the leading tokenizers on identical data and compare them, to add confidence intervals and near-duplicate checks, to run a BPE and corpus-size ablation, to measure tokenization parity against English on FLORES+, to build a small gold morpheme-segmentation set, and to add the unevaluated tokenizers and a literary evaluation set.

Per-Set Results

Word-Level Test

CategoryWordBest customBest competitorResult
Genitive/possessiveجو دنيا32k (2)2 (5 tokenizers)Tie
Genitive/possessiveجو خدا32k (2)2 (6 tokenizers)Tie
Genitive/possessiveجي پاڻي32k (2)2 (5 tokenizers)Tie
Genitive/possessiveجو ماڻهن32k (2)2 (5 tokenizers)Tie
Inflectional suffixڪتابن32k (1)1 (5 tokenizers)Tie
Inflectional suffixشهرن32k (1)1 (3 tokenizers)Tie
Inflectional suffixتوهانڪي64k (1)1 (hellosindh-NER)Tie
Inflectional suffixمنهنجو32k (1)1 (5 tokenizers)Tie
Inflectional suffixهنجي32k (2)1 (FahadQazi BPE, mBERT-Sindhi)FahadQazi BPE
Aspirated digraphٻار32k (1)1 (5 tokenizers)Tie
Aspirated digraphڄاڻ32k (1)1 (5 tokenizers)Tie
Aspirated digraphڏينهن32k (1)1 (4 tokenizers)Tie
Aspirated digraphڳوٺ32k (1)1 (7 tokenizers)Tie
Aspirated digraphڀاءُ32k (1)1 (4 tokenizers)Tie
Aspirated digraphٿورو32k (1)1 (5 tokenizers)Tie
Aspirated digraphڇوڪرو32k (1)1 (4 tokenizers)Tie
Loanwordعلم32k (1)1 (8 tokenizers)Tie
Loanwordتعليم32k (1)1 (9 tokenizers)Tie
Loanwordدنيا32k (1)1 (6 tokenizers)Tie
Loanwordاسڪول32k (1)1 (5 tokenizers)Tie
Loanwordڪمپيوٽر32k (1)1 (5 tokenizers)Tie
Loanwordانٽرنيٽ32k (1)1 (5 tokenizers)Tie

Table B1. Token counts for 22 hand-selected words. “Best custom” is the best of our four sizes (token count in brackets). “Best competitor” is the best of the 13 non-duplicate baselines (SindhiLM v2 and Qwen2.5 are identical). The result is 21 ties, 0 outright wins and 1 loss.

References

  1. [5]W. Ali et al. 2019. Word embedding based new corpus for low-resourced language: Sindhi. arXiv:1911.12579.arxiv.org/abs/1911.12579
  2. [6]W. Ali et al. 2020. A subword guided neural word segmentation model for Sindhi. arXiv:2012.15079v1. Retitled in v2 (2024): “Enhancing Sindhi word segmentation using word subword representation learning and position-aware self-attention”.
  3. [8]A. M. Bhurgiri. 2025. Sindhi language mega corpus – 118M tokens. ambile-official, Hugging Face dataset repository (compiler); LGPL-3.0. Accessed 4 November 2026.
  4. [9]BigScience Workshop. 2022. BLOOM: A 176B-parameter open-access multilingual language model. arXiv:2211.05100. Tokenizer: bigscience/bloom-560m.
  5. [10]Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In Findings of ACL: EMNLP 2020, pages 4617–4624.arxiv.org/abs/2004.03720
  6. [11]A. Conneau et al. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL. ArXiv:1911.02116. Tokenizer: FacebookAI/xlm-roberta-base.arxiv.org/abs/1911.02116
  7. [12]Express Tribune. 2024. Census shows rich lingual tapestry in Sindh. 22 July 2024.
  8. [14]Gemma Team, Google DeepMind. 2025. Gemma 3 technical report. arXiv:2503.19786. Tokenizer: google/gemma-3-1b-pt.
  9. [19]Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple sub-word candidates. In Proceedings of ACL.arxiv.org/abs/1804.10959
  10. [20]Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of EMNLP: System Demonstrations.arxiv.org/abs/1808.06226
  11. [21]Llama Team, AI @ Meta. 2024. The Llama 3 herd of models. arXiv:2407.21783. Tokenizer: meta-llama/Llama-3.2-1B (Llama 3.2 model card, 2024).
  12. [22]Mozilla Data Collective. 2026. Sindh line publishers corpus. CC BY-SA 4.0.
  13. [23]A. Nawaz et al. 2026. sindhi-tokenizer: A corpus-driven SentencePiece Unigram tokenizer for Sindhi. GitHub (https://github.com/alinawazcode/sindhi-tokenizer) and Hugging Face (https://huggingface.co/alinawazmahar/sindhi-tokenizer). Code MIT; tokenizer files CC BY-SA 4.0. Release v1.0.0, Zenodo DOI https://doi.org/10.5281/zenodo.23962474.
  14. [24]NLLB Team et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv:2207.04672. FLORES+ is maintained by Open Language Data: https://huggingface.co/datasets/openlanguagedata/flores_plus.arxiv.org/abs/2207.04672
  15. [25]P. Pattnayak, H. L. Patel, and A. Agarwal. 2025. Tokenization matters: Improving zero-shot NER for Indic languages. arXiv:2504.16977.arxiv.org/abs/2504.16977
  16. [26]Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. In Advances in Neural Information Processing Systems 36. ArXiv:2305.15425.arxiv.org/abs/2305.15425
  17. [27]Qwen Team, Alibaba Group. 2024. Qwen2.5 technical report. arXiv:2412.15115. Tokenizer: Qwen/Qwen2.5-0.5B.
  18. [28]Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? On the monolingual performance of multilingual language models. In Proceedings of ACL-IJCNLP. ArXiv:2012.15613.
  19. [29]C. W. Schmidt et al. 2024. Tokenization is more than compression. In Proceedings of EMNLP, pages 678–702. ArXiv:2402.18376.arxiv.org/abs/2402.18376
  20. [30]Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of ACL, pages 1715–1725.arxiv.org/abs/1508.07909
  21. [34]L. Xue et al. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of NAACL-HLT. Tokenizer: google/mt5-base.

Paper details

vlnc:W_xbg2vrnt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents