betaFeedback

WorksComputer Science

FIDEL: Abugida-Aware Pre-tokenization for Ethiopic Languages

Abstract

Subword tokenizers assume that meaningful linguistic units can be recovered by splitting text at learned boundaries, an assumption better suited to concatenative morphology than to Semitic root-and-pattern systems. The challenge is acute for the Ethiopic languages, whose abugida script encodes consonant-vowel units as single characters and hides consonantal structure. We introduce FIDEL (FIdäl DEcomposition and Linearization), a family of reversible abugida-aware pre-tokenization schemes that expose consonant and vowel-order structure, extend SPLINTER reductions from abjad scripts to the Ethiopic abugida, and evaluate them on three Ethiopic languages under controlled conditions, holding corpus, architecture, budget and protocol fixed so that tokenization is the only variable. We report intrinsic tokenizer statistics for BPE and UnigramLM across vocabulary sizes alongside downstream NLP tasks, including transfer scenarios. We find that FIDEL reduction improves news classification consistently and named-entity recognition at larger capacity, while sentiment analysis performance declines under the more aggressive schemes, and extractive question answering performance is preserved.

Introduction

Every language model begins its operation with tokenization, the mapping from character sequences to integer sequences that determine what the model can learn about word structure. The dominant algorithms (Schuster and Nakajima, [24]; Sennrich et al., [25]; Kudo, [17]) share a concatenative assumption: an ID only ever represents runs of adjacent characters, so the units a tokenizer can discover are contiguous substrings. In cases where morphology is expressed by interleaving rather than concatenation, no concatenative schema can recover morphemes. Such is the case with Semitic languages, which build words from a consonantal

root combined with a mostly vocalic template. In an abjad script, such as that representing Hebrew or Arabic, vowels are largely unwritten, so the written form approximates the skeleton directly, and recent work exploits this to improve tokenization (Gazit et al., [12]).

The Ethiopic languages present a harder case. Amharic, Tigrinya and Ge’ez are written in the fidäl, an abugida of 33 base consonants each realizable in seven vowel orders. In Unicode, each character is represented as one indivisible codepoint fusing a consonant with a vowel: ሰ is not “b” followed by “e”, but a single symbol denoting the combined form. In the example in Table 1, both verbs meaning “broke” derive from the root ሰ–በ–ረ, but in ሰባበሩ the frequentative template inserts በ and raises the vowel order of the final consonant (ረ → ሩ). A tokenizer can only be exposed to the syllabic characters of each form and never the common consonantal sequence, because that skeleton is not a subsequence of the written string.

(a) ልጁ መስታወቱን ሰበረ። “The boy broke the mirror.”
(b) ተማሪዎቹ ወንበሮቹን ሰባበሩ። “The students broke the chairs into pieces.”
ScriptChallenge and solution
AlphabetSkeleton visible; with concatenative morphology, subwords generally suffice.
AbjadSkeleton visible but intermixed with morphology; exposed by SPLINTER.
AbugidaSkeleton fused into consonant–vowel codepoints; exposed by FIDEL, after which SPLINTER applies.

Table 1. Examples of Amharic text exemplifying its nonconcatenative morphology. The root ሰበረ “break” appears in both sentences, but in (a) it comprises a contiguous segment whereas in (b) it is broken by the templatic ሰ and by the vowel order of its final consonant (ረ → ሩ). The lower block compares abugida with related script types.

This is also why SPLINTER (Gazit et al., [12]), which isolates roots in abjads by deleting templatic material, cannot be applied directly to Ethiopic scripts. Consequently, morphologically related words may be segmented into unrelated subword units, limiting the model’s ability to generalize across forms derived from the same root. Existing Ethiopic resources treat tokenization as fixed infrastructure [13, 4, 28], so no study isolates its effect for these languages. We therefore introduce FIDEL, a reversible pre-tokenization framework for the Ethiopic abugida that decomposes each fidäl character into a canonical consonant and an explicit vowel-order marker, exposing the hidden consonantal structure and enabling SPLINTER-style reductions to recover shared root patterns that are inaccessible while consonants and vowels remain fused. As shown in Figure 1, the resulting representation can be processed by any ordinary subword tokenizer and decoded back into standard fidäl thanks to systematic recording of the vowel-order decomposition and reduction operations. We evaluate several ways of combining the two operations, varying both representation and operation order, and find that the resulting representations improve token efficiency at larger vocabularies and benefit news classification and named-entity recognition downstream.

An Amharic language model pipeline showing standard and FIDEL flows

Figure 1. An Amharic language model pipeline for ሰባበሩ (säbabbäru, “they shattered”): the standard flow for a 16k-vocabulary BPE tokenizer, with and without incorporating FIDEL. The standard flow yields pieces that share nothing with the segmentation of the related ሰበረ, so the root is never exposed; FIDEL decomposes and reduces the word to that root, which the tokenizer then emits as a single shared piece.

Our contributions are: FIDEL, a reversible pre-tokenization framework for abugida scripts with a formal decoding procedure; an adaptation of SPLINTER from abjads to an abugida; a controlled comparison across three languages, two model scales, two vocabulary sizes and four task families; and intrinsic tokenizer analyses for two subword algorithms across a range of vocabulary sizes, reported alongside the downstream results they are often used to predict.

Background and Related Work

Subword tokenization. The three main tokenization algorithms: byte-pair encoding (BPE; [25]) which builds a vocabulary by iteratively merging frequent adjacent token pairs; WordPiece [24] which merges by likelihood gain; and UnigramLM [17] which prunes a large candidate vocabulary, all share a concatenative assumption. Only adjacent symbols form tokens, either through merges or by initial vocabulary construction. This makes them less effective at capturing morphological structures in which related elements are interleaved rather than concatenated. In addition, standard tokenizers may violate script-specific orthographic constraints in abugida writing systems, producing invalid token boundaries [15]. Tokenizer quality affects downstream performance [22], and subwords align poorly with morphemes in morphologically complex languages [16].

The fidäl. The Ethiopic (Ge’ez) script is an abugida in which each fidäl character encodes a consonant–vowel sequence rather than a bare phoneme [6]. The consonantal skeleton of Ge’ez admits seven vowel orders, realized as modifications of a consonant’s base shape [8, 19]. Thus, consonant and vowel information are fused within a single character. The consonantal skeleton is therefore not directly represented in the written string; the verbs ሄደ (“he went”) and ሄዱ (“they went”) differ only in the vowel order of the final character but share no symbol in that position. This fusion of consonants and vowels prevents structure from being directly accessible to a tokenizer.

SPLINTER. Gazit et al. [12] address nonconcatenative morphology through a reversible input transformation that relinearizes text by iteratively pruning characters from words and re-appending them at the end. The learned pruning operations specify both a character and its location in the word, and the transformed text is processed by an unmodified subword tokenizer. The authors report improvements on downstream tasks for Hebrew and Arabic, both of which exhibit nonconcatenative root-and-pattern morphology. In these languages, roots are represented by separable consonantal characters, while vowels may be unwritten or represented separately (Daniels and Bright, [6]), allowing SPLINTER’s character-deletion operations to expose the roots. However, SPLINTER’s character-deletion operations cannot be applied directly to Ethiopic abugida, where consonant and vowel information are fused within a single fidäl character. Deleting a whole character would therefore remove both consonant and vowel information, rather than isolating the consonantal root. Extending SPLINTER to Ethiopic therefore requires a prior decomposition step that transforms each fidäl into separable consonant and vowel elements before reduction can apply.

Ethiopic script representations and Resources. Representing an abugida as consonant–vowel pairs is long established outside tokenization. The SERA scheme (Firdyiwek and Yaqob, 1997) maps each fidäl character to an ASCII digraph for interchange, and finite-state morphological analysers operate over decomposed representations (Gasser, [10, 11]). Beyond representation and morphological analysis, computational treatments of Ethiopic languages include stemming (Alemu Argaw and Asker, [3]) and corpus construction (Gezmu et al., [14]), while EthioLLM (Tonja et al., [28]) provides encoders with a 256k subword vocabulary and benchmarks, including MasakhaNER (Adelani et al., [1]) and AfriSenti (Muhammad et al., [21]).

Tokenization for Ge’ez-script languages. Closest to our work is MoVoC (Teklehaymanot et al., [27]), which builds a morpheme-aware subword vocabulary for Ge’ez-script languages by combining supervised morphological analysis with BPE over annotated morpheme data. It modifies the vocabulary and depends on annotation, whereas we modify the input and require none, leaving the subword algorithm unmodified, so that differences are attributable to the representation rather than to the segmenter. They find that morphology-aware segmentation improves intrinsic measures without yielding significant translation gains, consistent with our observation that intrinsic statistics do not straightforwardly predict downstream behavior.

Method

Every Fidel scheme operates as a pre-tokenization step: raw text is transformed and passed to a tokenizer built using SentencePiece (Kudo and Richardson, [18]). We compare six pre-tokenization schemes that combine text representations and SPLINTER-based reduction in different ways. We describe the text representations in § 3.1, the reduction applied in § 3.2, and the reversible decoding procedure in § 3.3. Table 2 presents the conditions we compare and shows each applied to the same input.

SchemeDecompositionReductionDefinition
baseline——Raw fidäl, no transformation: ሰባበሩ.
splinter_only—ሰበሩ RRReduction only, applied directly to fidäl.
syllabicሰበV3V_{3}በረV1V_{1}—Decomposition only, no reduction.
syllabic_then_splinterሰበV3V_{3}በረV1V_{1}⟶\longrightarrow ሰበረ R1R2R_{1}R_{2}Interleaved (CVCV) decomposition, then reduction.
cccvvv_then_splinterሰበበረ V3,2V1,4V_{3,2}V_{1,4}⟶\longrightarrow ሰበረ R1R2R_{1}R_{2}Grouped (CCCVVV) decomposition, then reduction.
splinter_then_syllabicሰበረV1V_{1} RR⟵\longleftarrow ሰበሩ RRReduction first, then decomposition (operation-order ablation).

Table 2. The pre-tokenization schemes, each applied to ሰባበሩ (säbabbäru, “they broke into pieces ”). The arrow gives the order in which the two operations are applied, and a dash marks an operation the scheme does not perform. R1R_{1} and R2R_{2} are reduction markers, each a single codepoint recording one operation. Appendix § 13 presents the same conditions as pipelines.

Consonant–Vowel Representations

The fidäl is decomposed into a base consonant and a separate vowel marker, giving two elements per syllable. The interleaved form (CVCV) places each vowel marker directly after its consonant, preserving syllable boundaries; the grouped form (CCCVV) emits all consonants first and all vowel markers afterwards:

ቃል→ከ V3 እ V1(interleaved)ቃል \rightarrowከ\ V_{3}\ እ\ V_{1}\qquad\text{(interleaved)}
ቃል→ቃልV3,2V1,4(grouped)ቃል \rightarrowቃል V_{3,2} V_{1,4}\qquad\text{(grouped)}

In the interleaved representation, each marker follows its consonant, so adjacency determines the pairing. In the grouped representation, markers follow all consonants and explicitly identify their positions. In Vo,iV_{o,i}, oo denotes the vowel order and ii the one-based consonant position. Thus, V3,2V1,4V_{3,2}V_{1,4} assigns vowel order 3 to the second consonant and vowel order 1 to the fourth; unmarked consonants retain the first order. In both, the skeleton እ, እ, ክ becomes explicit, and the frequentative template and plural inflection that distinguish ቀደም from ቀደ are isolated in separate markers rather than buried in a distinct codepoint. We implement markers using unused CJK codepoints, following Gazit et al. [12]. Appendix § 12 presents all s-b-r forms under this decomposition.

First-order canonicalization. The consonantal component is always written in the canonical first-order fidäl form, with the vowel order carried separately. A vowel order is the position of a character within the seven-member series that a base consonant forms: ሰ is the first order of the consonant ss. The remaining six orders realize the same consonant with each of the other six vowels, giving the seven-member series ሰ, ሱ, ሲ, ሳ, ሴ, ስ, ሶ. Vowel orders are indexed from zero, V0V_{0} through V6V_{6}; V0V_{0} is canonical, meaning that a consonant already in first order carries no marker. Thus, as shown in the decomposition column of Table 2, the first and third consonants are already in first order and therefore written bare. The absence of a marker is itself informative, which is what makes the decoding rule (see § 3.3) well-defined: an unmarked consonant is reconstructed in first order on both sides of the mapping.

The marker V1V_{1} is lowered to first order, yielding the root ሰበረ (säbbära, “he broke”), which is substantially more frequent in the corpus. Each operation is recorded as a reduction marker containing the removed material and its position, so that R1R_{1} restores በV3V_{3} and R2R_{2} restores V1V_{1}. The encoded form therefore consists of the consonantal skeleton together with the reduction markers; decoding applies the recorded operations in reverse order and reconstructs ሰባበሩ exactly. In the grouped form, markers follow all bases: ሰበበረV3,2V1,4V_{3,2}V_{1,4} undergoes the same reductions, yielding ሰበረR1R2R_{1}R_{2}. Decoding is likewise the same replay in reverse: R2R_{2} returns order 1 to the third consonant, and R1R_{1} reinserts በ with order 3 as the second, recomposing ሰባበሩ. Both representations use the same reduction and reconstruction procedure; they differ only in how vowel markers are associated with consonants: by adjacency in the interleaved form and by explicit position in the grouped form.

Adapting SPLINTER to an Abugida

In our main setup, SPLINTER is applied over decomposed representations rather than over raw fidäl. Table 2 presents the splinter_only scheme, which reduces ሰባበሩ only to ሰበሩ, whereas both composed schemes reach the root ሰበረ. Reaching the root requires manipulating vowel-order information (ሩ →\to ረ), which is impossible while consonants and vowels remain fused. Only after decomposition separates consonants from vowel orders can SPLINTER remove templatic material and recover the root.

The transformation in FIDEL proceeds in two stages. Decomposition rewrites the word over the consonant alphabet, giving ሰ በV3V_{3} በ ረV1V_{1}. Reduction then operates on this sequence. The frequentative element በV3V_{3} is removed and the final vowel marker V1V_{1} is lowered to first order, yielding the root ሰበረ (säbbära, “he broke”), which is substantially more frequent in the corpus. Each operation is recorded as a reduction marker containing the removed material and its position, so that R1R_{1} restores በV3V_{3} and R2R_{2} restores V1V_{1}. The encoded form therefore consists of the consonantal skeleton together with the reduction markers; decoding applies the recorded operations in reverse order and reconstructs ሰባበሩ exactly.

Reductions are applied iteratively, allowing words with multiple affixes to undergo several reduction steps. The process halts when no more frequent attested intermediate form exists, preventing spurious root recovery.

Two constraints keep reductions morphological. First, a candidate is accepted only if the result is itself an attested word that is more frequent than the input, following Gazit et al. [12]. Second, the result must retain at least three units, preventing short words from collapsing into single frequent characters that are orthographic rather than morphological relatives. Reduction statistics are computed per language; otherwise, a form frequent in one language may incorrectly attract reductions from another. For example, the Tigrinya word ሰብእ (“people”, frequency 23) would reduce to the Amharic ሰባ (“seventy”, frequency 54) under pooled statistics, even though the two are unrelated.

The decomposition schemes differ in what they can expose. syllabic separates vowel-order information from the consonants, increasing overlap between related forms but leaving internal templatic material intact. splinter_only applies reduction while vowels remain fused, so it may halt at an intermediate form. Only the two schemes that both decompose and reduce, syllabic_then_splinter and cccvvv_then_splinter, reach the root, because decomposition makes the consonants explicit before reduction removes templatic material.

Reversibility and decoding

Because the representation separates consonant identities from vowel information, a deterministic decoding procedure is required to reconstruct valid Ethiopic words. Encoding is injective, and decoding its output is a deterministic inverse: each consonant takes the vowel order specified by its marker, unmarked consonants remain in first order, and reduction markers are undone in reverse order. We require of every scheme that

Decode⁡(Encode⁡(W))=W\operatorname{Decode}(\operatorname{Encode}(W)) = W

for every valid Ethiopic abugida word WW, so that preprocessing loses no information and predictions over the transformed stream can be scored against the original surface form. When a language model generates malformed sequences, such as markers referring to nonexistent consonant positions or missing vowel instructions, the decoder uses the projection procedure described in Appendix § 14 to handle these cases.

Verification. Reversibility is checked by round-trip reconstruction rather than assumed. On the Amharic evaluation split, cccvvv_then_splinter reconstructed 652,966 of 652,966 whitespace-delimited tokens exactly. Three classes of edge case are handled explicitly: Ethiopic punctuation (U+1360–U+1368), trimmed before encoding and reattached afterwards; foreign-script material whose codepoints overlap the CJK marker blocks, passed through untouched; and labiovelars, which fall outside the regular seven-order grid. For any form the encoder cannot verify, it returns the input unchanged, so identity holds by construction.

Intrinsic Tokenizer Analysis

We first evaluate each representation independently of any trained model. We report five measures, each calculated over a corpus: Rényi efficiency (Zouhar et al., [30]) penalizes token distributions skewed towards very high- or very low-frequency tokens (higher is better).1 Tokens per word, also called subword fertility (Rust et al., [22]), is the mean number of tokens a whitespace-delimited word is split into; 4+ token words is the proportion of words segmented into four or more tokens; and 1-char tokens the proportion of tokens consisting of a single character. All three are indicators of compression efficiency (lower is better). Finally, distinct neighbors measures contextual coherence (Yehzelkel and Pinter, [29]), the number of distinct tokens each token co-occurs with within a distance of two, averaged over the vocabulary. Fewer contexts make a token’s environments more distinguishable, so lower is better. Together, these measures capture different and partly conflicting aspects of tokenizer quality.

Table 3 summarises both BPE and UnigramLM at 64k vocabulary, and Table 4 focuses on vocabulary size effects in BPE; UnigramLM numbers are reported in Appendix § 15. The interleaved scheme syllabic_then_splinter is the most token-efficient at the larger vocabularies, slightly ahead of the untransformed baseline, for both BPE and UnigramLM. At 16k and below, the baseline regains the advantage, so any benefit of decomposition depends on there being vocabulary capacity to absorb the additional symbols. syllabic, which marks vowel order without applying reduction, is the least token-efficient scheme at every size, producing roughly 1.8 times as many tokens per word as the baseline; because its markers encode position as well as vowel order, its marker inventory is large and each marker is individually rare. It nevertheless yields the fewest distinct neighbors at every size, so the two measures point in opposite directions for this scheme. More generally, no single measure orders the schemes as the others do, which is consistent with the caution of Schmidt et al. [23] against assessing tokenizers by compression alone.

SchemeRényi efficiencyTokens per word4+ token words1-char tokensDistinct Neighbors
BPE
baseline0.6541.3531.98%1.89%7286
syllabic_then_splinter0.6431.3232.13%1.19%6329
splinter_then_syllabic0.6541.3622.33%1.21%7315
splinter_only0.6551.3672.41%1.95%7304
cccvvv_then_splinter0.6681.4985.34%2.40%8421
syllabic0.5712.3915.12%1.60%5628
UnigramLM
baseline0.5471.3822.67%2.53%6988
syllabic_then_splinter0.5741.3462.49%1.32%5865
splinter_then_syllabic0.6001.3772.59%1.43%7104
splinter_only0.5981.3822.81%2.43%7082
cccvvv_then_splinter0.5961.5036.17%3.14%7572
syllabic0.5482.4225.36%1.69%5766

Table 3. Intrinsic statistics at 64k vocabulary for both subword algorithms. Bold marks the best value per column within each block; lower is better for all measures except Rényi efficiency.

SchemeRényi efficiencyTokens per word4+ token words1-char tokensDistinct Neighbors
64k vocabulary
baseline0.6541.3531.98%1.89%7286
syllabic_then_splinter0.6431.3232.13%1.19%6329
splinter_then_syllabic0.6541.3622.33%1.21%7315
splinter_only0.6551.3672.41%1.95%7304
cccvvv_then_splinter0.6681.4985.34%2.40%8421
syllabic0.5712.3915.12%1.60%5628
32k vocabulary
baseline0.6881.4953.33%2.84%6480
syllabic_then_splinter0.6751.4713.89%1.74%5980
splinter_then_syllabic0.6881.5124.09%1.77%6464
splinter_only0.6881.5194.22%2.93%6439
cccvvv_then_splinter0.7111.7259.57%3.37%7319
syllabic0.5822.4967.82%1.98%4713
16k vocabulary
baseline0.7221.6945.68%4.18%5207
syllabic_then_splinter0.7081.6967.13%2.68%4900
splinter_then_syllabic0.7201.7317.29%2.63%5069
splinter_only0.7201.7447.48%4.51%5027
cccvvv_then_splinter0.7492.04015.84%4.79%5686
syllabic0.5902.63012.04%2.69%3562
8k vocabulary
baseline0.7451.98110.15%6.75%3655
splinter_then_syllabic0.7452.08613.73%4.76%3351
syllabic_then_splinter0.7352.09014.32%4.31%3393
splinter_only0.7422.10414.07%7.60%3322
cccvvv_then_splinter0.7732.49625.84%6.82%3877
syllabic0.5902.81018.04%4.14%2410

Table 4. Intrinsic statistics for BPE across vocabulary sizes, following the protocol of Gazit et al. [12]. Bold marks the best value per column within each block; lower is better for all measures except Rényi efficiency.

Downstream Setup

We further evaluate the FIDEL pre-tokenization schemes by training language models from scratch on each representation and fine-tuning them on downstream tasks. Holding the corpus, architecture, optimization budget and evaluation protocol fixed leaves the tokenization as the only variable, so any difference in downstream performance is attributable to the representation.

Corpus and models. We pretrain encoders with pre-layer-normalization from scratch using masked language modeling (Devlin et al., [7]) on a 385 MB Ge’ez-script corpus (662 files, 1,447,588 lines), identical across conditions. Two scales are used, small (∼\sim91M parameters) and large (∼\sim165M), each with vocabularies of 16k and 64k. We optimize using AdamW [20] with warmup and cosine decay. Downstream fine-tuning updates the full encoder under an identical budget, and the best-validation checkpoint is evaluated. Further details are provided in Appendix § 17. Experiments were conducted on an HPC cluster using NVIDIA RTX 3090 GPUs (24 GB), one GPU per run.2

Tasks. Evaluation covers named-entity recognition (EthioNER, MasakhaNER; [28, 1]), sentiment analysis (AfriSenti; [21]), news classification (MasakhaNEWS; [2]) and part-of-speech tagging (UD_Amharic-ATT; [26]). Named-entity recognition is reported as entity-F1, and the remaining tasks as weighted F1. finding no entities would still exceed 0.90 weighted-F1.

Controlled comparison. All causal claims are made against our internal baseline, in which corpus, architecture, budget and protocol are identical and only tokenization differs. Published systems differ in data, architecture and training regime simultaneously, so no difference against them could be attributed to tokenization; their figures are cited as context only. Reported values are medians over the runs available for each configuration.

Downstream Results

We report two configurations of the large model. The 64k vocabulary gives the subword algorithm enough capacity to absorb the additional symbols that decomposition introduces, so marker-bearing schemes are not penalized for inventory size; this is the setting in which the intrinsic statistics of § 4 favor the transformed representations. The 16k vocabulary is the constrained setting, where every marker competes with a stem for vocabulary slots, and where the baseline regains its intrinsic advantage. Both are realistic, corresponding roughly to monolingual model budgets and script-within-multilingual model vocabulary budgets, respectively.

SchemeNERSent.NewsPOS
baseline61.1054.8790.9894.26
splinter_only62.9852.7692.0994.20
syll_then_spl63.6049.0492.3194.00
cccvvv_then_spl61.5156.4392.0394.28
spl_then_syll55.4041.0189.9191.60
syllabic53.1660.7092.1794.22

Table 5. Amharic, large model, 64k vocabulary. NER is entity-F1; other tasks weighted-F1.

Downstream Results

Amharic

Tables 5 and 6 report the large model; the small-model results appear in Appendix § 16. Across tasks, splinter_then_syl, which reverses decomposition and reduction, is the weakest condition on every task, confirming that decomposition must precede reduction.

Within tasks we encounter nuanced findings. On news classification, every reduction scheme exceeds the baseline at both vocabulary sizes, by one to two points. This is the most stable effect we observe. For named-entity recognition, both vocabulary sizes present advantage for Fidel but the magnitude and the leading scheme differ. At 16k, splinter_only reaches 65.80 against a baseline of 62.79; while at 64k, syllabic_then_spl reaches 63.60 against 61.10. In sentiment analysis, syllabic_then_spl falls below baseline at both sizes, while cccvvv_then_spl and syllabic exceed it at 64k. This is the least stable task in our experiments and we draw no ordering from it. The part-of-speech tagging results, when excluding the reversed-order ablation, all fall in the 93.44–94.69 range. At this level, the remaining differences are not negligible in relative terms, as the range represents a delta of roughly 20% in error rate. The ordering nevertheless varies across configurations, and the gaps fall within the spread observed across runs, so we do not rank the schemes on this task. Finally, Table 7 shows that model performance on question answering is preserved rather than harmed: syllabic_then_spl attains the highest F1, and splinter_only the highest exact-match score.

SchemeEMF1
baseline47.4960.55
splinter_only48.5060.00
syllabic_then_spl45.0061.20

Table 7. Extractive QA on AmQA. Answers are scored in raw Amharic after inverse transformation, which reversibility makes possible.

SchemeNERSent.NewsPOS
baseline62.7957.0289.9194.09
splinter_only65.8055.3191.9194.69
syll_then_spl63.1852.1491.7593.98
cccvvv_then_spl60.2050.3088.8294.43
spl_then_syll60.1543.6386.9190.83
syllabic60.2346.1587.6993.44

Table 6. Amharic, large model, 16k vocabulary.

Tigrinya and Ge'ez

Table 8 reports the large model on Tigrinya and on Ge’ez part-of-speech tagging, the latter transferred zero-shot from Amharic.

TigrinyaGe’ez
SchemeSent.NewsPOSPOS
64k vocabulary
baseline86.2785.3196.7036.59
splinter_only85.8583.8996.1237.94
syll_then_spl86.1386.1296.2137.89
cccvvv_then_spl85.3284.4096.8238.34
16k vocabulary
baseline86.8182.1096.6237.00
splinter_only86.9281.3096.4638.56
syll_then_spl86.4682.6796.4436.38
cccvvv_then_spl85.4481.4796.6136.95

Table 8. Tigrinya and Ge’ez, large model. Ge’ez is transferred zero-shot from Amharic.

The pattern differs from Amharic. On Tigrinya, transformed representations achieve top performance at 64k vocabulary, with syllabic_then_splinter leading on news classification and cccvvv_then_splinter leading on part-of-speech tagging. At 16k vocabulary, splinter_only leads on sentiment while syllabic_then_splinter leads on news classification. Overall differences across schemes remain under two points across all Tigrinya tasks. Ge’ez part-of-speech tagging sits between 36 and 39 for all conditions, which reflects the difficulty of zero-shot transfer rather than any property of the schemes. We therefore restrict our core claims to Amharic.

Analysis

Where reduction helps. Reduction improves news classification and named-entity recognition at both vocabulary sizes, while sentiment is mixed. A plausible reading is that entity and topic labels depend chiefly on stem identity, which reduction consolidates by mapping inflected variants onto shared forms, whereas sentiment depends more on affixal cues such as negation, which reduction compresses into opaque markers.

Arrangement and ordering. The two X_then_spl schemes differ only in symbol layout, and they differ downstream: the interleaved form leads on named-entity recognition at 64k, while the grouped form leads on sentiment. Intrinsically, the interleaved form is more token-efficient and the grouped form scores higher on Rényi efficiency, so the two arrangements trade off against each other. Operation order, by contrast, is not a trade-off: reversing decomposition and reduction degrades every task, since a position-indexed reduction map cannot be applied to a stream whose structure has already changed.

Intrinsic measures disagree. Tokens per word and Rényi efficiency select different schemes, and neither ordering matches the downstream ordering. Teklehaymanot et al. [27] report a comparable dissociation, with morphology-aware segmentation improving intrinsic measures without yielding significant translation gains. We therefore treat intrinsic statistics as descriptions of the representation rather than as predictors of task performance.

Conclusion

In this work, we introduced Fidel, a pre-tokenization method for subword tokenizers that decomposes Ethiopic abugida characters into explicit consonant–vowel representations, exposing linguistic structure that is otherwise fused within a single symbol. Building on this representation, we adapted the Splinter reduction algorithm from abjads to an abugida and conducted a controlled study of tokenization for Ethiopic languages, holding the corpus, architecture, training budget, and evaluation protocol fixed so that tokenization remained the sole variable.

Four findings emerge. First, exposing the consonantal skeleton helps tasks that turn on stem identity, consistently for news classification and for named-entity recognition, while tasks that depend on the inflectional material reduction com presses benefit less or not at all; the effect of pre-tokenization is therefore task-dependent rather than uniform. Second, how the two operations are composed matters as much as whether they are applied, since reversing decomposition and reduction degrades every task. Third, different vocabulary sizes provide different preferences, so intended use case is a factor in deciding on a pre-tokenization schema. Fourth, the intrinsic measures commonly used to select tokenizers disagree with one another here and do not order the schemes as downstream accuracy does, so they are better read as descriptions of a representation than as predictions of its usefulness.

Future work might extend evaluation for Tigrinya and Ge’ez beyond the scope reported here, investigate additional languages using the Ethiopic script, and evaluate this approach on other abugida writing systems. Additionally, introducing a curated affix inventory could allow corpus frequency to validate rather than define reductions, and testing generative decoding will confirm if reversibility is preserved when a model generates sequences unseen during encoding.

Limitations

Language coverage is uneven. Amharic is evaluated on four tasks; Tigrinya and Ge’ez are not fully covered in the tables reported here, which reflects dataset availability rather than design. Claims about Ethiopic languages therefore more takes on Amharic evidence. The interpretation offered in § 8, that reduction consolidates stems at the expense of affixes, is consistent with the observed pattern but was not verified against a morphological analyser. Reductions are learned from corpus frequency and have no access to morphological knowledge, so a frequent reduction need not correspond to a morpheme. Finally, our results concern more on a single corpus and the family of subword algorithms. Whether the same trade appears with different pretraining data, or under character or byte-level models that do not tokenize at all, is untested.

Ethics Statement

This work concerns tokenization for Amharic, Tigrinya and Ge’ez, languages with limited NLP resources relative to their speaker populations, and improved tokenization may lower the cost of building language technology for these communities. All data used is in these three languages, written in the Ge’ez script, and is drawn from news articles, social media posts and Wikipedia; annotation was performed by native speakers as reported by the original dataset creators. Two cautions apply. Evaluation coverage is uneven across the three languages, so benefits should not be assumed to transfer equally. And reductions are learned from corpus statistics, so a rule frequent in news text need not be linguistically general.

Decomposition of the s-b-r Family

Examples of consonant–vowel decomposition applied to the s-b-r morphological family are presented in Table 9.

FormGlossConsonantsDecomposed
ሰበረ“he broke”ሰበረሰ በ ረ
ስብርባሪ“fragments”ሰበረሰV5 በV5 ረV5 በV3 ረV2
ሰባበረ“he shattered”ሰበበረሰ በV3 በ ረ
ሰባበሩ“they shattered”ሰበበረሰ በV3 በ ረV1
ተሰበረ“it was broken”ተሰበረተ ሰ በ ረ
መሰበሪያ“instrument for breaking”መሰበረየመ ሰ በ ረV2 የV3

Table 9. The s-b-r family after consonant–vowel decomposition. Each row gives a derived form, its gloss, the base consonants it decomposes to, and the decomposed sequence, in which every consonant appears in canonical first order followed by a marker for any non-first vowel order. In the surface fid”al of the first column, the six words share almost no characters, since each template realises the same consonants in different vowel orders. After decomposition, the root consonants ሰ–በ–ረ become explicit in every form, either directly or interleaved with vowel markers and additional morphological material. Reduction can then iteratively remove this surrounding material—the passive prefix ተ, the instrumental መ with its suffix የ, and the reduplicated በ of the frequentative—and, where the resulting form is attested, reach the root ሰበረ. Forms whose consonants are all first order carry no marker at all.

All Pre-tokenization Pipelines

The six conditions are shown as pipelines in Figure 2.

The six pre-tokenization conditions applied to ሰባበሩ

Figure 2. The six pre-tokenization conditions applied to ሰባበሩ (säbabbäru, “they shattered”). VnV_n marks vowel order nn. Reduction markers are written R1R_1, R2R_2 for legibility; each is a single codepoint in the implementation, recording what was deleted and where. Here R1R_1 records the removal of the reduplicated በ together with its vowel order V3V_3, and R2R_2 that the final ረ was lowered from V1V_1 to first order, while RR in splinter_only records the removal of the syllable ባ. splinter_only reduces the surface form but cannot expose the root, since on an abugida the vowels remain fused inside the fidäl characters; only after decomposition are the root consonants ሰበረ visible as contiguous symbols. Decomposition exposes structure and reduction exploits it, which is why the two are separate contributions, and why splinter_then_syllabic—which reduces first—is included as an ablation.

Decoding Procedure

Definition 1 (CV manifold). Let M\mathcal{M} be the set of all valid Ge’ez strings that can be written as sequences of pairs (ci,vi)(c_i, v_i) with ci∈Cc_i \in C a base consonant and viv_i a valid vowel order.

Given a token sequence SS produced by the model, decoding finds the nearest valid word:

W^=arg min⁡W∈M d(S,W).\widehat{W} = \underset{W \in\mathcal{M}}{\operatorname{arg\,min}} \ d(S,W).

Let C=[c1,…,cn]C = [c_1,\ldots,c_n] be the consonants extracted from SS and V={(pj,mj)}j=1mV = \{(p_j,m_j)\}_{j=1}^{m} the vowel instructions decoded from marker tokens, with pjp_j a position and mjm_j a marker. The distance is

d(S,W)=min⁡α∈A∑j=1m∣α(j)−pj∣+λmax⁡(0,m−n)+μmax⁡(0,n−m),(1)d(S,W) = \min_{\alpha\in A} \sum_{j=1}^{m} \lvert\alpha(j)-p_j \rvert+ \lambda\max(0,m-n) + \mu\max(0,n-m), \tag*{(1)}

where AA is the set of monotone, injective assignments from instructions to positions. The first term aligns vowel instructions to consonant positions, while the second and third penalise overflow and underflow. Letting λ\lambda and μ\mu tend to infinity enforces strict handling: surplus instructions are discarded and unfilled positions take the default order. All schemes are validated by round-trip reconstruction on held-out Amharic and reproduce the original text exactly.

UnigramLM Intrinsic Statistics

Intrinsic benchmark results for UnigramLM across vocabulary sizes are presented in Table 10.

SchemeRényi efficiencyTokens per word4+ token words1-char tokensDistinct Neighbors
64k vocabulary
baseline0.5471.3822.67%2.53%6988
syllabic_then_SPLINTER0.5741.3462.49%1.32%5865
SPLINTER_then_syllabic0.6001.3772.59%1.43%7104
SPLINTER_only0.5981.3822.81%2.43%7082
cccvvv_then_SPLINTER0.5961.5036.17%3.14%7572
syllabic0.5482.4225.36%1.69%5766
32k vocabulary
baseline0.5611.5254.59%3.90%6370
syllabic_then_SPLINTER0.6021.4824.70%1.81%5636
SPLINTER_then_syllabic0.6251.5194.67%1.83%6348
SPLINTER_only0.6231.5295.09%3.91%6354
cccvvv_then_SPLINTER0.6331.70710.90%4.12%6473
syllabic0.5412.5568.64%2.10%4818
16k vocabulary
baseline0.5741.7307.87%5.17%4990
syllabic_then_SPLINTER0.6381.6978.57%2.65%4613
SPLINTER_then_syllabic0.6541.7318.45%2.56%4861
SPLINTER_only0.6511.7479.07%5.41%4845
cccvvv_then_SPLINTER0.6721.99117.22%5.48%4948
syllabic0.5302.70513.28%2.72%3654
8k vocabulary
baseline0.5832.03914.12%7.38%3335
SPLINTER_then_syllabic0.6742.08615.90%4.24%3043
syllabic_then_SPLINTER0.6732.08116.60%4.52%2977
SPLINTER_only0.6742.11817.08%8.43%2929
cccvvv_then_SPLINTER0.7012.41026.34%7.72%3368
syllabic0.5132.90220.46%3.98%2407

Table 10. Intrinsic statistics for UnigramLM across vocabulary sizes. Bold marks the best value per column within each block. The ordering reproduces the BPE results of Table 4.

Small-Model Results

Downstream evaluation results for the Amharic small model across tasks and vocabulary sizes are presented in Table 11.

SchemeNERSent.NewsPOS
16k vocabulary
baseline33.5947.5088.3692.31
SPLINTER_only32.9147.0089.9389.86
syllabic_then_SPLINTER32.2141.0389.6889.81
cccvvv_then_SPLINTER25.9940.9787.8790.74
SPLINTER_then_syllabic27.5628.6582.0587.62
syllabic21.6247.1188.3592.63
64k vocabulary
baseline36.4349.4191.4990.73
SPLINTER_only35.1246.9792.0492.28
syllabic_then_SPLINTER34.3747.2791.2590.55
cccvvv_then_SPLINTER32.8650.1889.3190.43
SPLINTER_then_syllabic27.0347.2089.4489.78
syllabic19.7346.6090.3190.32

Table 11. Amharic, small model. At this capacity the baseline is competitive on named-entity recognition, while the reduction schemes retain their advantage on news classification.

Training Hyperparameters

Pretraining. All encoders are pretrained from scratch using masked language modeling. We use AdamW with linear warmup over the first 6% of steps followed by cosine learning-rate decay. The small models use a learning rate of 1×10−41 \times10^{-4} and a batch size of 16, while the large models use a learning rate of 5×10−55 \times10^{-5} and a batch size of 32. The maximum training duration is 25 epochs for small models and 30 epochs for large models. The sequence length is 512 tokens.

Fine-tuning. For downstream tasks, we fine-tune the full encoder with a learning rate of 2×10−52 \times10^{-5} for 5 epochs, except for question answering, which uses 10 epochs. We select the checkpoint with the best validation performance. All experiments use three random seeds.

References

  1. [1]David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsudeen H. Muhammad, Chris Chinenye Emueze, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, and 42 others. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, 9:1116–1131.
  2. [2]David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, et al. 2023. MasakhaNEWS: News topic classification for African languages. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 144–159, Nusa Dua, Bali. Association for Computational Linguistics.
  3. [3]Atelach Alemu Argaw and Lars Asker. 2007. An Amharic stemmer : Reducing words to their citation forms. In Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources, pages 104–110, Prague, Czech Republic. Association for Computational Linguistics.
  4. [4]Tadesse Destaw Belay, Atnafu Erku, and Seid Muhie Yimam. 2022. The effect of normalization for bidirectional Amharic-English neural machine translation. In ICT4DA.
  5. [5]Marco Cognetta, Vilém Zouhar, Sangwhan Moon, and Naoaki Okazaki. 2024. Two counterexamples to tokenization and the noiseless channel. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16897–16906, Torino, Italia. ELRA and ICCL.arxiv.org/abs/2402.14614
  6. [6]Peter T. Daniels and William Bright, editors. 1996. The World’s Writing Systems. Oxford University Press, New York.
  7. [7]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference on the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.arxiv.org/abs/1810.04805
  8. [8]August Dillmann. 1974. Ethiopic Grammar: Orthography and Phonology, Morphology and Syntax of the Ethiopic Language. Philo Press, Amsterdam. Reprint of the 1907 edition published by Williams & Norgate, London. Translation of Grammatik der äthiopischen Sprache.
  9. [10]Michael Gasser. 2009. Semitic morphological analysis and generation using finite state transducers with feature structures. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 309–317, Athens, Greece. Association for Computational Linguistics.DOI
  10. [11]Michael Gasser. 2011. HornMorpho: a system for morphological processing of Amharic, Oromo, and Tigrinya. In Conference on Human Language Technology for Development.
  11. [12]Bar Gazit, Shaltiel Shmidman, Avi Shmidman, and Yuval Pinter. 2025. Splintering nonconcatenative languages for better tokenization. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22405–22417, Vienna, Austria. Association for Computational Linguistics.
  12. [13]Andargachew Mekonnen Gezmu, Andreas Nürnberger, and Tesfaye Bayu Bati. 2022. Extended parallel corpus for Amharic-English machine translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6644–6653, Marseille, France. European Language Resources Association.arxiv.org/abs/2104.03543
  13. [14]Andargachew Mekonnen Gezmu, Binyam Ephrem Seyoum, Michael Gasser, and Andreas Nürnberger. 2018. Contemporary Amharic corpus: Automatically morpho-syntactically tagged Amharic corpus. In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, pages 65–70, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  14. [15]Sai Hemanth Kapila and Rakshika Bagavathy. 2026. Type-driven tokenization for Brahmic scripts. In Proceedings of the ACM SIGPLAN International Workshop on Type-Driven Development (TyDe ’26), Paris, France. ACM.
  15. [16]Stay Klein and Reut Tsarfaty. 2020. Getting the #life out of living: How adequate are word-pieces for modelling complex morphology? In SIGMORPHON.DOI
  16. [17]Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics.arxiv.org/abs/1804.10959
  17. [18]Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.arxiv.org/abs/1808.06226
  18. [19]Wolf Leslau. 1987. Comparative Dictionary of Ge’ez (Classical Ethiopic): Ge’ez–English, English–Ge’ez, with an Index of the Semitic Roots. Otto Harrassowitz, Wiesbaden.
  19. [20]Ilya Loshchilov, Frank Hutter, et al. 2017. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5(5):5.
  20. [21]Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, Nedjma Ousidhoum, David Ifeoluwa Adelani, Seid Muhie Yimam, Ibrahim Sa’id Ahmad, Meriem Belouicif, Saif M. Mohammad, Sebastian Ruder, Oumaima Hourrane, Pavel Brazdil, Alipio Jorge, Felermino Dário Mário António Ali, Davis David, Salomey Osei, Bello Shehu Bello, Falalu Ibrahim, Tajuddeen Gwadabe, and 8 others. 2023. AfriSenti: A Twitter sentiment analysis benchmark for African languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13968–13981, Singapore. Association for Computational Linguistics.arxiv.org/abs/2302.08956
  21. [22]Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics.
  22. [23]Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alemeddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. Tokenization is more than compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 678–702, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2402.18376
  23. [24]Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In ICASSP, pages 5149–5152.DOI
  24. [25]Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.arxiv.org/abs/1508.07909
  25. [26]Binyam Ephrem Seyoum, Yusuke Miyao, and Baye Yimam Mekonnen. 2018. Universal dependencies for Amharic. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 2216–2222, Miyazaki, Japan. European Language Resources Association (ELRA).DOI
  26. [27]Hailay Kidu Teklehaymanot, Dren Fazlia, and Wolfgang Nejdl. 2025. MoVoC: Morphology-aware subword construction for Ge’ez script languages. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 13131–13144, Suzhou, China. Association for Computational Linguistics.DOI
  27. [28]Atnafu Lambebo Tonja, Israel Abebe Azime, Tadesse Destaw Belay, Mesay Gemeda Yigezu, Moges Ahmed Ah Mehamed, Abinew Ali Ayele, Ebrahim Chekol Jibril, Michael Melese Woldeyohannis, Olga Kolesnikova, Philipp Slusallek, Dietrich Klakow, and Seid Muhie Yimam. 2024. EthioLLM: Multilingual large language models for Ethiopian languages with task evaluation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6341–6352, Torino, Italia. ELRA and ICCL.arxiv.org/abs/2403.13737
  28. [29]Shaked Yehzekel and Yuval Pinter. 2023. Incorporating context into subword vocabularies. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 623–635, Dubrovnik, Croatia. Association for Computational Linguistics.
  29. [30]Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for Computational Linguistics.

Paper details

vlnc:W_wgnw946t
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents