WorksComputer Science
Syllable- and Word-Level Evaluation of Tibetan OCR
Abstract
Optical character recognition (OCR) for Tibetan is evaluated predominantly by character error rate (CER). Tibetan marks syllable boundaries but does not mark word boundaries, so a word error rate requires a segmenter. This study evaluates nine recognizer configurations on typeset, woodblock and two handwritten script styles, alone and in combination, and reports syllable- and word-level error rates, to our knowledge the first for Tibetan OCR. The configurations span CTC models, TrOCR encoder-decoders with and without post-training quantization, and a vision-language model, and each is scored under CER, the syllable error rate (SER) and three segmented word error rates (SWER). CER ranks systems largely as the higher-level rates do (Kendall ), with one pair ordered oppositely. On the effect of joint training, CER and SER record significant changes of opposite sign in 7 of 64 comparisons. SER needs no segmenter and ranks systems as SWER does under two of the three segmenters (). The output of each configuration is also tagged for part of speech, annotated for named entities and machine translated, and each result is compared with that obtained on the reference text. As rough guidance, translation chrF of 60 is reached at SER of 0.047 or below and part-of-speech preservation of 0.8 at 0.078 or below, while no error rate separates the systems that reach entity preservation of 0.6 from those that do not.
CCS Concepts: • Applied computing → Optical character recognition; • Computing methodologies → Natural language processing; • General and reference → Empirical studies.
Introduction
Optical character recognition (OCR) and handwritten text recognition are applied to Tibetan prints and manuscripts, including to make large collections of Buddhist texts searchable [13, 17, 28]. Recognition of continuous Tibetan text is typically evaluated by character error rate (CER) [13, 17, 42, 54]. The orthography of Tibetan makes word error rates, typical in other languages [34, 51], difficult to compute.
In scripts that delimit a unit larger than the character, CER and an error rate over that unit can rank systems differently [1, 26]. OCR error also degrades downstream tasks that use the recognized text [50]. Named-entity recognition falls sharply once word-level error reaches 8–20% [19], and corrupting named-entity spans reduces keyword search performance while CER and word error rate remain low [23].
Tibetan explicitly marks syllable boundaries, chiefly with the tsheg (་), but does not mark word boundaries. A word error rate therefore requires a segmenter. The syllable is an intermediate unit that requires none, and segmented word error rates have been applied to Tibetan speech recognition [31]. To our knowledge, no error rate above the character has been reported for Tibetan OCR. It is therefore not known whether CER leads an evaluator to the same choices as a syllable- or word-level rate would, or which rate best predicts the quality of downstream tasks.
Nine recognizer configurations are evaluated, spanning CTC models, TrOCR encoder–decoders with and without post-training quantization, and a vision-language model. They are trained on four datasets, covering modern typeset text, woodblock print and two handwritten styles, and on three mixed-domain combinations of them to compare joint with single-domain training. This gives 60 configuration condition cells. Each cell is scored under CER, the syllable error rate (SER) and three segmented word error rates (SWER). On the four single-domain conditions, recognizer output is also tagged for part of speech, annotated for named entities and machine translated into English, and each result is scored against the result obtained on the reference transcription. These preservation scores measure the effect of recognition error on each task.
This study makes three contributions.
A comparative evaluation of Tibetan OCR across architectures, quantization and script domains. Configurations are compared on page-grouped held-out splits with bootstrap intervals, under syllable- and word-level error rates that are, to our knowledge, the first publicly reported for Tibetan OCR.
A comparison of CER with syllable- and word-level rates. CER ranks systems largely as the higher-level rates do (Kendall ), but on the effect of joint training it records significant changes of opposite sign to SER in 7 of 64 comparisons. Word-level rankings agree across three segmenters at (§ 4.2, § 5.1, § 5.2).
Approximate error rates for downstream use. The SER needed for a rough preservation target ranges from 0.007 for entities to 0.078 for part-of-speech tags, so no single rate serves every task. LLM annotation also fails more often as OCR error rises (§ 4.4, § 5.3).
Sections 2 and 3 give the background and methods, Section 4 the results and Section 5 their interpretation. Section 6 states limitations and future work, Section 7 concludes, and Appendix § 9 gives the prompts used.
Background and Prior Work
Tibetan Script
Structure
Tibetan is written in an abugida, a writing system whose base unit is a consonant carrying an inherent vowel, modified by diacritics. The fundamental unit of written Tibetan is the syllable, built from a root consonant that may carry superscript and subscript letters, a vowel diacritic, and prefix and suffix consonants [22]. The root and its superscript and subscript form a single visual column, called a stack. Unicode encodes a stack as a sequence of codepoints rather than as a unit, so one visual column may consist of several codepoints.
Syllables are separated by the tsheg (་, U+0F0B) [49]. Word boundaries are not marked. Nearly every word boundary is also a syllable boundary, but words may be polysyllabic [6], as also in Dzongkha [35]. Punctuation is sparse: the only mark occurring throughout running text is the shad (།, U+0F0D), which separates units larger than the syllable, roughly sentences [49], and so also ends the syllable before it. Level tokenization therefore splits on the tsheg, the shad and whitespace. Word-level tokenization requires a more complex segmentation step.
Varieties and Media
Written Tibetan divides into two script families, distinguished by the presence or absence of a head, a short horizontal stroke along the top of many letters [49]. Uchen (‘headed script’) is the formal block script and the basis for woodblock-carved prints and modern digital typesetting. Ume (‘headless script’) is the cursive family, whose letters lack a head and are written with more connected strokes, with substantially greater variability in letterform shape and positioning. Within Ume, Drutsa is a handwritten cursive [13], and Betsug (‘book form’) is a calligraphic style [49]. The two families differ enough that recognition does not necessarily transfer between them [17].
The physical medium varies independently of script variety. In woodblock prints each character is carved by hand, so letterforms vary more than in machine print, and manual printing can merge the strokes of neighbouring characters [20]. Modern typeset text is uniform in letterform, while handwriting varies between writers [60].
Recognition Architectures
Recognizers
Early Tibetan OCR systems decoded with hidden Markov models [20, 41]. More recent Tibetan recognition is neural, including convolutional and convolutional-recurrent networks [53, 56, 60], an encoder–decoder for scene text [Ke et al. 2025], the PyLaia engine of Transkribus [13, 17] and a vision-language model adapted to Tibetan [54]. This study compares three kinds of neural recognizer. Connectionist temporal classification [16] trains a sequence model to maximize the likelihood of the target over all valid frame-to-character alignments. Convolutional-recurrent networks [44] and Easter2.0 [7], which replaces the recurrence with convolutions, are both trained with CTC. Autoregressive encoder-decoders generate over the encoder output instead, as in TrOCR [27]. Vision-language models pair a vision encoder with a language decoder pretrained on a large multilingual corpus rather than on the recognition task; Qwen3-VL [2], whose decoder is built on Qwen3 [55], is one such model.
Quantization and Adaptation
Quantization stores weights at lower numerical precision and may be applied to an already-trained model [9]. Quantized low-rank adaptation [11] quantizes a frozen base model and trains only small low-rank adapter matrices [21], which reduces the memory needed to fine-tune a large model.
Edit-Distance Error Rates
In a full OCR pipeline, layout analysis and line segmentation precede recognition. This study takes line images as given and evaluates recognition alone. The recognized text of each line is scored against a reference transcription by the number of edits needed to turn one into the other, relative to the length of the reference.
Character, syllable and word error rates share the same formula,
where , and are the substitutions, deletions and insertions in the hypothesis with respect to the reference, and is the count of units in the reference. Character error rate (CER) tokenizes over Unicode characters, syllable error rate (SER) over syllables delimited by the tsheg, the shad or white-space, and segmented word error rate (SWER) over words, which for Tibetan requires a segmenter (§ 2.1.1). Three SWER variants were introduced for Tibetan automatic speech recognition (ASR) by Moore et al. [31]. BoTok-SWER segments with the dictionary-based botok tokenizer, BERT-SWER with a Tibetan BERT token classifier (tibetan-bert-base-upos), and Gem-SWER with Gemini 2.5 Flash Lite, prompted at temperature 0.0. Their agreement with human word-boundary annotation on speech transcripts, as reported there, is Cohen’s , 0.372 and 0.656 respectively. For all of the error rates described here, lower error-rate values indicate better performance. Values above 1.0 occur when the edits outnumber the reference units, which requires insertions.
Because the three levels differ only in tokenization, they are not independent measurements, but errors falling on unit boundaries affect them differently: a single inserted or deleted delimiter counts as one character edit but corrupts both units it separates.
Prior Work
Evaluation Practice in Tibetan OCR
Early Tibetan OCR systems scored output in units of Tibetan orthography rather than codepoints. Hedayati et al. [ [20] evaluate a woodblock recognizer by Levenshtein distance over stack characters rather than codepoints, and Rowinski and Keutzer [ [41] reports isolated-character accuracy and scores each page by a syllable validity ratio, the proportion of syllables that follow Tibetan spelling rules or appear in a list of accepted exceptions.
Recent OCR and handwritten text recognition (HTR) on continuous Tibetan text typically reports CER. Much of this work is built on Transkribus, which returns CER [13, 17], and Sabbagh [ [42] reports mean and median CER on a held-out set. FTibSuite Xu et al. [54] pairs CER with Exact Match, a whole-line binary, and notes that changes in the two are not perfectly aligned. Other tasks use other measures: work on isolated characters reports classification accuracy [Wang and Wang 2024; Zhang et al. 2022; see also the survey by Gongqu et al. 2024].
Work on scene text, evaluated on photographs of street views and outdoor environments, reports line accuracy, character accuracy and a normalized edit distance [Ke et al. 2025].
To our knowledge, no syllable- or word-level error rate has been reported for Tibetan OCR, though SER has been reported in Tibetan ASR [52, 58].
Character and Word Error Rates in Other Scripts
CER requires no tokenization beyond the codepoint and is computable for any script, including those that do not delimit words with whitespace. Thennal et al. [ [47] advocate it as the preferred metric for multilingual ASR, on the grounds that for languages without word separators “the WER can only be applied after a subjective and potentially inaccurate word segmentation process”. Similar reasoning appears in OCR. Clérice [ [8] judges WER “complicated to apply” to Old French and Medieval Latin, where spaces vary in size or are absent.
Where whitespace does delimit words, word error rate is reported alongside CER and comparison across the two levels is normal. The two levels often agree. Neudecker et al. [ [34] plot CER and WER side by side, WER exceeding CER throughout, and the two systems compared by Vesalainen et al. [ [51] are ranked the same way by both, though they differ in the kind of edits they make. Agreement is not general. Agbeti-Messan et al. [ [1], benchmarking historical newspaper OCR on Latin script, report a different best system under CER than under WER on each of their two subcorpora, though under 0.1 percentage points separates the CER of their leading models.
An intermediate unit has been used in scripts where word boundaries are ambiguous: Naing et al. [ [32] propose a syllable error rate for Myanmar speech recognition. Vietnamese words often span several space-separated syllables Le et al. [ [26], so a word error rate over space-delimited tokens is in effect a syllable error rate. In that survey’s line-level results the system with the lower CER (3.24% against 4.67%) has the higher syllable-level rate (14.11% against 13.33%).
For Tibetan, the only comparison against a word-level metric is in speech recognition. Moore et al. [ [31] benchmark five ASR architectures and six adaptation strategies under CER, SER and the three SWER variants, and report ranking differences between SER and CER, the same architectural ranking under all three word-level metrics, and a fall in the CER–SER correlation under parameter-efficient fine-tuning.
Downstream Use of Recognition Output
Recognition output is often the input to a further process, and its quality has been measured by the degradation it causes there. van Strien et al. [ [50] evaluate six tasks across OCR output of varying quality and report a consistent impact across all six. Hamdi et al. [ [19] find named-entity F-score falling from roughly 90% to 60% once word-level error rates reach the 8–20% range. Jaud et al. [ [23] report that corrupting named-entity spans sharply reduces keyword search performance while CER and WER remain low, whereas corrupting the same proportion of non-entity tokens raises both rates and leaves retrieval comparatively stable.
Downstream resources for Tibetan are limited. Meelen and Hill [ [29] tag Classical Tibetan under a purpose-built tagset. A 20,215-sentence Tibetan named-entity corpus is available on request rather than publicly released, and recent entity recognition work on it fine-tunes a large language model with domain prompts [57]. Tibetan machine translation has concentrated on the Tibetan–Chinese pair [Moore and Lauren 2025; e.g. Zhou et al. 2024], with few studies of Tibetan–English [30, 33, 45].
Griffiths [ [17] notes that a CER under 10% is “considered efficient” for automatic transcription, and sets 5% as the project’s target for text mining, though neither threshold is explicitly validated.
Methods
Nine recognizer configurations are evaluated on seven data conditions, four single-domain and three mixed-domain (§ 3.1–§ 3.3). Each of the resulting 60 cells is scored on a held-out test split under five error rates (§ 3.4). Error concentration is measured per cell (§ 3.5), and the rates are compared across systems, lines and training conditions (§ 3.6). On the four single-domain conditions, recognizer output is also passed to three downstream tasks (§ 3.7).
Datasets and Splits
Four publicly available datasets from the Hugging Face Hub are used, spanning the principal script domains of historical and contemporary Tibetan OCR (§ 2.1.2): KhyentseWangpo (BDRC/KhyentseWangpo, modern typeset), Karmapa8 (BDRC/Karmapa8, woodblock print), Drutsa (openpecha/OCR-Drutsa, cursive dbu med handwriting) and Betsug (openpecha/OCR-Betsug, semi-cursive dbu med handwriting). All four supply pre-cropped line images paired with transcriptions. Statistics are given in Table 1 and example lines in Figure 1.
| Dataset | Total lines | Train lines | Test lines | Train pages | Test pages | Mean chars | Max chars | Vocab |
| KhyentseWangpo | 13,527 | 12,172 | 1,355 | 2,102 | 234 | 167 | 312 | 112 |
| Karmapa8 | 24,973 | 22,476 | 2,497 | 4,048 | 449 | 212 | 401 | 122 |
| Drutsa | 32,360 | 29,116 | 3,244 | 2,901 | 318 | 128 | 457 | 135 |
| Betsug | 28,318 | 25,485 | 2,833 | 4,337 | 480 | 148 | 314 | 131 |
| Drutsa+Betsug | 60,678 | 54,601 | 6,077 | |||||
| KhyentseWangpo+Karmapa8 | 38,500 | 34,648 | 3,852 | |||||
| All four | 99,178 | 89,249 | 9,929 |
Table 1. Dataset statistics. Mean chars and Vocab are computed on the training split. Identifiers refer to public Hugging Face repositories. The three mixed-domain conditions concatenate components, and their line counts are the sums of the component counts.

Figure 1. One test line from each script domain, with its reference transcription.
All datasets are split 90/10 by page, pages drawn randomly until the test set first reaches the target size. A mixed condition is the concatenation of its components’ splits, so each mixed test set is the union of the corresponding single-domain test sets.
Models
Five architectures are evaluated (Table 2), spanning the CTC, autoregressive encoder-decoder and vision-language classes of § 2.2.1.
Table 2. Model architectures. TrOCR checkpoints are stage-1 pretrained on synthetic printed text-line images [27].
CTC models. Both follow their published specifications with two exceptions. CRNN takes input 64 px high rather than 32 px, because Tibetan stacks up to four glyph components vertically. Easter2.0 is reimplemented without its TACo augmentation and weighted CTC loss, so that the two CTC models share one training procedure. Line images are resized to 64 × 2048 px, and both models produce about 512 output frames per line, more than the number of characters in any reference line.
TrOCR. The pretrained decoders are initialized from RoBERTa and MiniLM [27], whose subword vocabularies were not built for Tibetan. The decoder vocabulary is replaced with a Tibetan character vocabulary before fine-tuning: embed_tokens and output_projection are resized and randomly reinitialized, while all encoder and decoder weights are retained. Images are resized to 384 × 384 by the checkpoint’s processor.
Qwen3-VL. Its byte-level BPE covers the Tibetan block (U+0F00–0FFF) at 1.33 tokens per character. The model is prompted with a fixed instruction (Appendix § 9.§ 1) and the line image. Qwen3-VL is trained only on the four single-domain datasets, with 4-bit QLoRA: adapters (, , dropout 0.1) on the attention and feed-forward projections.
Adaptation and Training
Each TrOCR size is used in three configurations: standard fine-tuning, and 8-bit and 4-bit quantization (§ 2.2.2) of the fine-tuned checkpoint after training, which are not trained separately. CRNN and Easter2.0 receive standard fine-tuning only, and Qwen3-VL 4-bit QLoRA only.
Shared across every run: 4,000 / 8,000 / 16,000 total steps for 1 / 2 / 4 datasets, scaled so that the number of epochs per example is approximately constant across conditions; a linear warmup over the first 12.5% of the budget followed by linear decay to zero; and seed 42. CRNN, Easter2.0, TrOCR-small standard and TrOCR-base standard are additionally retrained at seeds 43 and 44 on Karmapa8 and Drutsa, and Qwen3-VL at both seeds on Drutsa; their quantized variants inherit the seed of the checkpoint they quantize. Decoding is greedy, with TrOCR capped at 460 generated tokens and Qwen3-VL at 768.
CRNN, Easter2.0 and TrOCR share a character-level vocabulary built from the training split. Out-of-vocabulary test characters count as errors but no test split contains more than ten of them, at most 0.0013% of test characters. Qwen3-VL keeps its pretrained vocabulary unchanged.
Where results are aggregated above individual configurations, the nine configurations are grouped into four families: CTC (CRNN, Easter2.0), TrOCR-small and TrOCR-base (the standard, 8-bit and 4-bit variants of each), and Qwen3-VL. The two TrOCR sizes are kept apart, differing by a factor of ten in parameters.
All experiments ran on a single NVIDIA RTX 4050 Laptop GPU with 6 GB of VRAM, which restricts Qwen3-VL to QLoRA.
Metrics
All metrics are defined in § 2.3 and computed directly on hypothesis and reference strings, following the tibetan-wer package1. Rates are macro-averaged, each line weighted equally, except for the paired per-component comparisons of transfer, which are micro-averaged. Leading and trailing whitespace is stripped from every reference and hypothesis before scoring. CER is computed over the codepoints of the stored strings, with no Unicode normalization or punctuation stripping. A syllable for SER is a maximal run of characters between delimiters: whitespace, the tsheg and tsheg bstar (U+0F0B–0F0C), the shad and the other head, clause and section marks (U+0F04–0F14 and U+0F34, U+0F3A–0F3D, U+0FD0–0FD4, U+0FD9–0FDA). The delimiters are not scored.
The three SWER variants use the segmenters of § 2.3. On the reference lines of the downstream subset, BoTok and BERT words average 1.3 syllables and Gemini words 2.9. Because of API cost, Gem-SWER is computed on a seeded random sample of 500 lines per condition, with the prompt of Appendix § 9.§ 2. On the mixed conditions the 500 lines are allocated across the components in proportion to their test-split sizes, and each component’s lines are drawn from that component’s single-domain sample. Before segmentation, strings whose character-type repetition ratio – length divided by number of distinct characters – exceeds 10 are excluded, because API calls time out. A line is dropped if either its reference or hypothesis trips the filter.
Error Concentration
Error concentration is the share of error-free lines and the share of summed error contributed by the worst decile of lines, computed per cell over the per-line CER array and over the per-line SER array. It requires no segmenter.
Statistical Analysis
Two systems compared on the same lines are tested with a paired bootstrap over test pages: pages are resampled 10,000 times, the difference is recomputed on the lines of the sampled pages, and it is significant when its 95% interval excludes zero. No multiple-comparison correction is applied.
Transfer between the mixed and single-domain conditions is measured per component: the mixed-trained and single-domain-trained models are scored on the identical lines the split makes common to both (§ 3.1) and the difference assessed by the paired bootstrap above.
A ranking reversal is a pair of configurations that CER and every metric above the character level order in opposite directions. Gem-SWER is excluded from the search and from the paired transfer comparisons, being measured on a different sample (§ 3.4); where it differs from the other three, this is reported. The search is exhaustive over every pair of configurations in every condition, so a pair may differ in quantization, in size or in architecture. A reversal is significant when the paired difference is significant on CER and, in the opposite direction, on at least two of SER, BoTok-SWER and BERT-SWER.
Correlations are quantified at two levels. Between per-line metric arrays we report Spearman , computed within each model–condition cell separately. At the ranking level, models are ordered by each metric within each condition and the orderings compared with Kendall’s . The same procedure across the three SWER variants measures whether the segmenter changes the ordering.
Downstream Task Evaluation
No gold English translation and no gold Tibetan entity annotation exist for these test sets, so absolute downstream quality cannot be measured. Following van Strien et al. [50], a downstream system’s output on clean text is taken as the ground truth for its output on OCR’d text. Downstream figures are therefore preservation scores, measuring the effect of recognition error on the task, not the system’s accuracy.
Evaluation uses the 500-line sample of each single-domain condition, the same lines as Gem-SWER, and all nine configurations (Table 4). Part-of-speech and entity preservation are F₁ over per-line items, repeated items counted separately, averaged over lines; translation preservation is corpus chrF [37] (, character 6-grams) and BLEU, both via sacreBLEU 2.6.0 [38] with default settings, and sentence chrF and BLEURT [43], a learned metric rather than surface overlap, from the original BLEURT checkpoint, whose scores fall below zero for inadequate translations, each averaged over lines. Where the annotator returns no usable output for a hypothesis, the hypothesis is scored as empty output rather than dropped, so the failure counts against the system that produced the line. A line whose reference the annotator could not process has no gold target and drops out. Downstream targets are set at round values of each measure (0.8 and 0.9 for part-of-speech F, 0.6 and 0.8 for entity F, 60 for chrF, 0.60 for BLEURT), chosen as reasonable levels of preservation for rough guidance; an error rate separates a target when every cell meeting it has a lower rate than every cell missing it.
| Task | Annotator | Measures reported |
| Part-of-speech | tibetan-bert-base-upos | F over (word, tag) pairs and over tags; tag-sequence error rate |
| Named entities | Gemini 2.5 Flash Lite | F over (span, type) pairs and over types |
| Translation | Gemini 2.5 Flash Lite | corpus and sentence chrF, BLEU, BLEURT |
Table 4. Downstream tasks. All are scored as preservation against reference-derived output. Entity labels are PER, LOC, ORG and WORK, the last for text titles.
Part-of-speech. Tagging uses tibetan-bert-base-upos, the model that also supplies the BERT-SWER segmentation, run locally: the word boundaries it induces are kept together with their UPOS labels, so each string becomes a sequence of (word, tag) pairs. The reference and every hypothesis are tagged this way, truncated to 800 characters and subject to the repetition filter of §3.4.
Entities and translation are annotated by Gemini 2.5 Flash Lite (§2.4.3). Requests are batched four lines at a time under the repetition filter of §3.4, each line truncated to 500 characters; prompts are in Appendix § 9.§ 3.
Annotator coverage. For every string, whether usable output was returned is recorded, with reference lines as the control, separating strings excluded by the repetition filter from strings queried without parseable output. A failing request is retried, finally without the translation field (Appendix § 9.§ 4).
Results
Error Rates
CRNN records the lowest CER in all four single-domain conditions (Table 5) and in two of the three mixed ones, Easter2.0 in the third. On Karmapa8 and Betsug CRNN leads Easter2.0 by 0.001 CER, and on Karmapa8 the two exchange places at one training seed (§4.2.1).
| Configuration | CER | SER | BoTok | BERT | Gem |
| KhyentseWangpo | |||||
| CRNN | 0.002 | 0.004 | 0.005 | 0.009 | 0.010 |
| Easter2.0 | 0.005 | 0.007 | 0.008 | 0.010 | 0.031 |
| TrOCR-base std | 0.007 | 0.017 | 0.023 | 0.054 | 0.109 |
| Qwen3-VL | 0.017 | 0.026 | 0.038 | 0.060 | 0.235 |
| TrOCR-base 4-bit | 0.028 | 0.078 | 0.119 | 0.194 | 0.507 |
| TrOCR-small std | 0.200 | 0.440 | 0.622 | 0.646 | 1.075 |
| TrOCR-small 4-bit | 0.653 | 0.989 | 1.168 | 1.110 | 1.365 |
| Karmapa8 | |||||
| CRNN | 0.013 | 0.031 | 0.043 | 0.105 | 0.290 |
| Easter2.0 | 0.014 | 0.032 | 0.045 | 0.092 | 0.358 |
| TrOCR-base std | 0.175 | 0.355 | 0.435 | 0.524 | 1.169 |
| TrOCR-base 4-bit | 0.353 | 0.633 | 0.733 | 0.756 | 1.494 |
| Qwen3-VL | 0.415 | 0.696 | 0.761 | 0.784 | 1.489 |
| TrOCR-small std | 0.551 | 0.910 | 0.958 | 0.975 | 1.614 |
| TrOCR-small 4-bit | 0.726 | 1.125 | 1.246 | 1.237 | 3.026 |
| Drutsa | |||||
| CRNN | 0.047 | 0.073 | 0.104 | 0.175 | 0.622 |
| Easter2.0 | 0.078 | 0.133 | 0.187 | 0.234 | 0.849 |
| TrOCR-base std | 0.175 | 0.254 | 0.311 | 0.393 | 0.946 |
| Qwen3-VL | 0.251 | 0.434 | 0.517 | 0.580 | 1.243 |
| TrOCR-base 4-bit | 0.282 | 0.409 | 0.479 | 0.541 | 1.320 |
| TrOCR-small std | 0.645 | 0.944 | 0.980 | 0.958 | 1.491 |
| TrOCR-small 4-bit | 0.726 | 1.021 | 1.092 | 1.037 | 1.635 |
| Betsug | |||||
| CRNN | 0.024 | 0.045 | 0.066 | 0.142 | 0.393 |
| Easter2.0 | 0.025 | 0.047 | 0.067 | 0.131 | 0.347 |
| TrOCR-base std | 0.109 | 0.171 | 0.221 | 0.321 | 0.660 |
| Qwen3-VL | 0.182 | 0.328 | 0.411 | 0.498 | 1.002 |
| TrOCR-base 4-bit | 0.212 | 0.344 | 0.425 | 0.512 | 0.843 |
| TrOCR-small std | 0.688 | 0.994 | 0.993 | 0.942 | 1.320 |
| TrOCR-small 4-bit | 0.691 | 1.151 | 1.114 | 1.061 | 1.420 |
Table 5. Macro-averaged error rates on the single-domain conditions, ordered by CER within each condition; lowest per condition in bold. The 8-bit variants, which differ from their standard counterparts by at most 0.014 in CER and 0.034 in SER, BoTok-SWER and BERT-SWER, are omitted.
Agreement Between Metric Levels
4.2.1 Model rankings. Kendall between the CER ranking and each syllable- or word-level ranking ranges from 0.833 to 1.000 across the 28 condition × metric cells of Table 6. The three segmenters produce rankings that agree at a minimum of 0.833, a median of 1.000 and a maximum of 1.000. The minimum is shared by the Karmapa8 pairing of BERT-SWER and Gem-SWER and the Drutsa pairing of BoTok-SWER and Gem-SWER; BoTok and BERT agree at 0.944–1.000 in every condition, and SER ranks systems as BoTok-SWER and BERT-SWER do at 0.889–1.000.
| Condition | CER against | Segmenter pairs | ||||||
| SER | BoTok | BERT | Gem | BoTokBERT | BoTokGem | BERTGem | ||
| KhyentseWangpo | 9 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Karmapa8 | 9 | 0.944 | 1.000 | 0.944 | 0.889 | 0.944 | 0.889 | 0.833 |
| Drutsa | 9 | 0.944 | 0.889 | 0.944 | 0.944 | 0.944 | 0.833 | 0.889 |
| Betsug | 9 | 0.944 | 0.944 | 0.889 | 0.833 | 0.944 | 0.889 | 0.944 |
| Drutsa+Betsug | 8 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| KhyentseWangpo+Karmapa8 | 8 | 0.929 | 0.929 | 0.929 | 0.929 | 1.000 | 1.000 | 1.000 |
| All four | 8 | 0.929 | 0.929 | 0.929 | 0.929 | 1.000 | 1.000 | 1.000 |
Table 6. Kendall between model rankings of the configurations: under CER against under each syllable- and word-level metric, and under each pair of segmenters.
At two further training seeds for CRNN, Easter2.0, TrOCR-small standard and TrOCR-base standard, between seeds is 1.000 on Drutsa under CER, SER and BoTok-SWER. On Karmapa8 it is 0.667 for two of the three seed pairs under all three metrics: CRNN and Easter2.0, whose CERs differ by 0.001, exchange places at one seed. Per-configuration CER varies across seeds by 0.000–0.009 for the CTC models, 0.021–0.072 for the TrOCR configurations and 0.007 for Qwen3-VL.
4.2.2 Per-line agreement and the error rate. Across the 60 configuration condition cells, a cell’s mean CER and its per-line CERSER agreement correlate at , and within each family separately at CTC (14 cells), TrOCR-small (21), TrOCR-base (21) and Qwen3-VL (4). At comparable error rates, families differ (Figure 2): the CTC cells on the conditions containing handwriting record agreement of 0.598–0.746 at mean CER 0.022–0.077, and the three TrOCR-base cells in that range 0.940–0.975. A median of 30.9% of a CTC cell’s character edits fall on whitespace, which SER does not score (§ 3.4), against 11.3% for TrOCR-base, 4.7% for TrOCR-small and 1.9% for Qwen3-VL. The correlation between mean CER and per-line agreement is −0.588 for CERBoTok-SWER, −0.500 for CERBERT-SWER, −0.693 for SERBoTok-SWER, −0.346 for SERBERT-SWER and −0.318 for BoTok-SWERBERT-SWER. Agreement among SER, BoTok-SWER and BERT-SWER is positive in all 60 cells, with a minimum of 0.256.
Figure 2. Per-line Spearman correlation between CER and SER against the mean CER of the same cell, for all 60 configuration condition cells. Legend entries give the correlation within each family. Agreement falls with mean CER and, at comparable mean CER, is lower for the CTC cells on the conditions containing handwriting, much of whose character error falls on whitespace.
Reversals and sign disagreements
One configuration pair is a significant reversal (§ 3.6). On Drutsa, CER ranks Qwen3-VL ahead of TrOCR-base 4-bit by 0.031 [0.007, 0.058]. BoTok-SWER and BERT-SWER rank TrOCR-base 4-bit ahead by 0.037 and 0.039, and SER by 0.024, whose interval includes zero. Across the nine pairings of the two configurations at seeds 42–44, CER and the three higher-level rates order the pair oppositely in all nine. On the downstream sample, sentence chrF favours the higher-level ordering by 5.37 [2.53, 8.33], and part-of-speech and entity do not differ.
Joint training was compared with single-domain training per component on the 64 configuration component cells, each a component’s full test split of 1,355–3,244 lines (§ 3.6). CER records a significant reduction in 48 cells and a significant increase in 11; SER 46 and 13, BoTok-SWER 46 and 10, BERT-SWER 43 and 15. In ten cells CER and a syllable- or word-level metric record significant changes of opposite sign. Eight are TrOCR-small 4-bit cells, in each of which CER records the reduction. The other two are TrOCR-base standard and 8-bit on the Betsug component of Drutsa+Betsug, in which CER records the increase. No CTC cell disagrees on sign. By pair, CER disagrees in sign with SER in seven cells, five of them TrOCR-small 4-bit, with BoTok-SWER in five and with BERT-SWER in eight; SER, BoTok-SWER and BERT-SWER disagree with one another in two or three cells per pair.
Error Concentration
Concentration differs by family (Table 7, Figure 3). CTC places 49.8% of its character error in the worst tenth of lines and leaves 28.2% of lines clean, TrOCR-base 42.9% and 7.9%, and Qwen3-VL 26.8% and 1.9%. TrOCR-small spreads error almost uniformly, 14.9% in the worst decile, and leaves 0.1% clean. The ordering is the same under SER, with every worst-decile share lower; the CTC clean share rises to 38.2%, since lines whose only errors fall on whitespace or punctuation are clean under SER. By domain, CTC leaves 90.9–95.0% of lines unchanged on KhyentseWangpo and 35.7–37.6% on Karmapa8; on Drutsa and Betsug every configuration has at least 79.3% of lines differing from the reference.
| Family | Per-line CER: Clean % | Per-line CER: Worst decile % | Per-line SER: Clean % | Per-line SER: Worst decile % | |
| CTC | 8 | 28.2 | 49.8 | 38.2 | 46.0 |
| TrOCR-small | 12 | 0.1 | 14.9 | 0.1 | 13.1 |
| TrOCR-base | 12 | 7.9 | 42.9 | 12.5 | 30.1 |
| Qwen3-VL | 4 | 1.9 | 26.8 | 2.8 | 22.7 |
Table 7. Concentration of per-line error, median over each family’s cells. Clean is the share of lines with no error and worst decile the share of all error falling in the worst tenth of lines.
Figure 3. Concentration of character error across lines, for the 36 single-domain cells. Each curve is the cumulative share of summed per-line CER against the cumulative share of lines, worst first; bold curves are family medians and the dashed diagonal is a uniform distribution. A curve’s value at the dotted line is its worst-decile share (Table 7). CTC and TrOCR-base concentrate error in few lines; TrOCR-small spreads it almost uniformly.
Downstream Consequences
Downstream preservation
Preservation declines as the error rate rises on all three tasks — part-of-speech tagging, named-entity recognition and translation — and declines furthest on named-entity recognition (Table 8, Figure 4). The CTC models on typeset text, at a median CER of 0.003, score 0.995 on part-of-speech preservation, 0.946 on entity preservation and 96.4 chrF. The same models on Betsug, at CER 0.024–0.025, score 0.880–0.895 on part-of-speech preservation while entity preservation falls to 0.587–0.602. At the other end, TrOCR-small on the two handwriting conditions reaches CER 0.65–0.73, with entity preservation of 0.142–0.494 and BLEURT of −1.39 to −1.18. Entity preservation is below part-of-speech preservation in 30 of the 36 measurements, by a median of 0.09.
Figure 4. Preservation of the reference annotation on three downstream tasks, against (a) CER and (b) SER on logarithmic axes, over the 36 configuration × condition cells for which downstream measures were computed. Curves are LOWESS fits on the logarithm of the rate, one per task; dashed lines mark the targets of § 4.4.1 (0.8 for part-of-speech and entity preservation, 60 for chrF). Translation chrF is divided by 100 to share the axis. SER’s association with each measure is within 0.005 of the strongest of the five rates (Table 9).
| Family | CER | SER | POS F | NER F | chrF | BLEURT | |
| KhyentseWangpo | |||||||
| CTC | 2 | 0.003 | 0.005 | 0.995 | 0.946 | 96.4 | +0.835 |
| TrOCR-small | 3 | 0.214 | 0.465 | 0.451 | 0.241 | 28.8 | -0.738 |
| TrOCR-base | 3 | 0.007 | 0.019 | 0.952 | 0.794 | 83.1 | +0.517 |
| Qwen3-VL | 1 | 0.017 | 0.026 | 0.955 | 0.688 | 75.3 | +0.339 |
| Karmapa8 | |||||||
| CTC | 2 | 0.014 | 0.032 | 0.919 | 0.704 | 70.8 | +0.250 |
| TrOCR-small | 3 | 0.552 | 0.910 | 0.271 | 0.322 | 19.8 | -1.047 |
| TrOCR-base | 3 | 0.185 | 0.372 | 0.555 | 0.429 | 37.1 | -0.637 |
| Qwen3-VL | 1 | 0.415 | 0.696 | 0.371 | 0.315 | 26.6 | -0.932 |
| Drutsa | |||||||
| CTC | 2 | 0.063 | 0.103 | 0.818 | 0.619 | 52.6 | -0.153 |
| TrOCR-small | 3 | 0.648 | 0.944 | 0.177 | 0.438 | 15.2 | -1.181 |
| TrOCR-base | 3 | 0.185 | 0.256 | 0.662 | 0.568 | 41.9 | -0.332 |
| Qwen3-VL | 1 | 0.251 | 0.434 | 0.518 | 0.501 | 35.4 | -0.713 |
| Betsug | |||||||
| CTC | 2 | 0.024 | 0.046 | 0.888 | 0.594 | 62.7 | +0.036 |
| TrOCR-small | 3 | 0.688 | 0.998 | 0.194 | 0.180 | 16.8 | -1.211 |
| TrOCR-base | 3 | 0.110 | 0.176 | 0.722 | 0.488 | 49.4 | -0.324 |
| Qwen3-VL | 1 | 0.182 | 0.328 | 0.598 | 0.397 | 39.4 | -0.602 |
Table 8. Downstream preservation, median over each family’s configurations, on the 500-line seeded random subset of each single-domain condition. POS F is over (word, tag) pairs, NER F over (surface form, type) pairs; chrF and BLEURT score the translation against the one produced from the reference. CER and SER are family medians of the full test-set rates. Bold: the downstream target of § 4.4.1 is met (POS and NER F 0.8, chrF 60).
Downstream targets are set at round values (§ 3.7). Every target except entity preservation of 0.6 is separated by CER, SER, BoTok-SWER and BERT-SWER. The seven cells reaching part-of-speech preservation of 0.9 are set apart from the remaining 29 at CER 0.017–0.024 and at SER 0.032–0.045. Entity preservation of 0.8 is reached only at SER 0.007 or below (CER 0.005–0.007), chrF 60 at SER 0.047 or below (CER 0.025–0.028), and part-of-speech preservation of 0.8 at SER 0.078 or below
(CER 0.047–0.078). BLEURT 0.60 is reached only at SER 0.007 or below (CER 0.005–0.007). No rate separates the nine cells reaching entity preservation of 0.6 from those missing it: Drutsa CRNN reaches it at SER 0.073 (CER 0.047), and Betsug CRNN misses it at SER 0.045 (CER 0.024).
Predictors of downstream degradation
The three part-of-speech measures are tagged locally and do not depend on an LLM annotator. On every measure CER, SER, BoTok-SWER and BERT-SWER correlate with preservation within 0.035 of one another (Table 9; Figure 4 plots CER SER against three of the measures). Gem-SWER, computed on the same 500 lines but with lines dropped by its repetition filter, is the weakest rate on all nine measures.
Annotator coverage
The entity and translation scores reported above come from Gemini 2.5 Flash Lite; part-of-speech preservation is tagged locally and does not depend on it. A line the annotator cannot process is scored as empty output (§ 3.7), so every measure covers at least 493 of the 500 lines per cell; the lines that drop out are those whose reference it could not process. The annotator returns no usable output, an empty or unparseable response, for up to 1.4% of a cell’s lines, at a rate that correlates with CER at .
Discussion
Ranking Agreement
In general, rankings of systems agree across the metric levels. Kendall between the CER ranking and each syllable- or word-level ranking never falls below 0.833 (§4.2.1). This is consistent with Vesalainen et al. [ [51], but contrasts with the ranking reversals seen elsewhere [1, 26, 31]. We find only one reversal, which holds across training seeds. On it, downstream translation favours the ordering given by the segmented word error rates (§ 4.2.1).
Rankings computed over the three segmenters agree at a median pairwise of 1.000 (§4.2.1). Which segmenter is used does not change which system a word-level rate judges better, as Moore et al. [ [31] found for automatic speech recognition.
Line by line, CER and SER diverge for two reasons. Across cells, per-line CER∼SER agreement falls with mean CER at (§4.2.2). At a given error rate, it also depends on how much of a system’s character error falls on whitespace, which CER counts and SER does not: the CTC models, with a median of 30.9% of character edits on whitespace, agree at 0.598–0.746 on the conditions containing handwriting, where TrOCR-base cells at comparable CER agree at 0.940–0.975. A CER-only evaluation of such a system therefore ranks its lines partly by whitespace errors, which SER does not score. Agreement among SER, BoTok-SWER and BERT-SWER is positive in every cell. Moore et al. [ [31] report the CER–SER correlation falling under parameter-efficient fine-tuning; here it falls with the error rate within the CTC and TrOCR-small families.
Disagreement Between the Rates
On a decision about a single system, CER can give the opposite answer from the rates above it. Comparing joint with single-domain training on identical lines, CER and a syllable- or word-level rate record significant changes of opposite sign in ten of the 64 cells, eight of them TrOCR-small 4-bit cells, where CER records an improvement (§ 4.2.1).
Conveniently, SER requires no segmenter, but ranks systems as BoTok-SWER and BERT-SWER do at Kendall (§4.2.1, § 3.4); the words of those two segmenters average 1.3 syllables. The same reasoning led Naing et al. [ [32] to a syllable error rate for Myanmar speech recognition, where the definition of a word is ambiguous.
Error Rate and Downstream Use
As rough guidance, with arbitrary but reasonable targets, part-of-speech preservation of 0.8 is reached only at SER 0.078 or below and translation chrF of 60 at 0.047 or below; the corresponding CER bands are 0.047–0.078 and 0.025–0.028 (§ 4.4). No rate separates the systems that reach entity preservation of 0.6 from those that miss it. The 5% CER that Griffiths [ [17] sets for text mining falls within the tagging band; the 10% described there as efficient for automatic transcription lies above it.
Across the three tasks, the SER needed ranges from 0.007 for entity preservation of 0.8 to 0.078 for part-of-speech preservation of 0.8, so no single acceptable rate serves every downstream use of Tibetan OCR at either the character or the syllable level. Entity preservation degrades fastest of the three tasks and is the hardest to predict, at for CER and for SER (Table 9). Hamdi et al. [ [19] also report entity extraction degrading sharply under OCR error, and Jaud et al. [ [23], using simulated OCR noise, find the loss concentrated in errors that fall on entity spans.
| Downstream measure | CER | SER | BoTok | BERT | Gem |
| POS (word, tag) F | -0.986 | -0.993 | -0.996 | -0.992 | -0.956 |
| POS tag-only F | -0.992 | -0.991 | -0.988 | -0.982 | -0.960 |
| POS tag-sequence ER | 0.982 | 0.991 | 0.993 | 0.996 | 0.963 |
| NER (form, type) F | -0.854 | -0.887 | -0.889 | -0.877 | -0.790 |
| NER type-only F | -0.932 | -0.955 | -0.956 | -0.946 | -0.886 |
| MT chrF | -0.985 | -0.993 | -0.995 | -0.992 | -0.956 |
| MT sentence chrF | -0.979 | -0.991 | -0.993 | -0.990 | -0.950 |
| MT BLEU | -0.985 | -0.994 | -0.997 | -0.990 | -0.947 |
| MT BLEURT | -0.981 | -0.994 | -0.995 | -0.988 | -0.944 |
Table 9. Spearman correlation of each error rate with each downstream preservation score over the 36 cells, strongest per row in bold. SER is within 0.005 of the strongest rate on every measure.
Entity extraction and translation are performed by an LLM, Gemini 2.5 Flash Lite, which fails to return usable output more often as its input degrades (§ 4.4.3). Unreliable behaviour of language and translation models on noisy OCR text, on perturbed source input and in low-resource translation has been reported before [18, 39] (Piryani et al. 2025). Failures are scored as empty output, so they count against the score. An evaluation that places an LLM annotator downstream of OCR ought thus to report the annotator’s coverage and how its failures are scored.
Limitations and Future Work
Limitations
Splits are grouped by page, not by text or scribe. Other pages of a test page’s text, and other lines in its scribe’s hand, can occur in training, so the error rates reported may be lower than those on unseen texts or hands, particularly on handwriting.
Training settings differ by architecture: batch size, learning rate and the number of passes over the training data were each set by hardware constraints and training stability (Table 3). Differences between architecture families therefore include differences in optimization.
| Architecture | Train batch | Eff. batch | Epochs | Learning rate | Optimizer |
| CRNN | 16 | 128 | 17.6–42.1 | AdamW | |
| Easter2.0 | 16 | 128 | 17.6–42.1 | AdamW | |
| TrOCR-small | 4 | 32 | 4.4–10.5 | AdamW (8-bit) | |
| TrOCR-base | 2 | 16 | 2.2–5.3 | AdamW (8-bit) | |
| Qwen3-VL-2B | 1 | 8 | 1.1–2.6 | AdamW (8-bit) |
Table 3. Per-architecture hyperparameters. The 8-bit optimizer is that of Dettmers et al. [ [10]. Epochs is the number of passes over the training set implied by a 4,000-step single-domain run at the given effective batch size, as a range over the four single-domain datasets.
Part-of-speech tagging and BERT-SWER use the same model, so the two are not independent. Gem-SWER, entity extraction and translation use a hosted model that cannot be pinned to a version.
The downstream figures are preservation against a fixed annotator’s output on the reference rather than accuracy, as no gold Tibetan entity annotation or English translation exists for these test sets (§ 3.7). They cover the nine configurations on a 500-line random sample of each single-domain test split (§ 3.4). Gem-SWER is computed on the same lines. The reliability of LLMs as evaluators varies across models and tasks [4], particularly in low-resource and multilingual settings [12, 14], and the same may hold for LLM annotators. A single annotator is used, so its errors are not separated from those caused by recognition. About 4% of entity spans, on references and hypotheses alike, are returned as English renderings that cannot match a Tibetan span, so the entity bands in particular are relative quantities. The downstream targets are round values chosen for rough guidance.
Future Work
Future work should extend this analysis to other scripts that leave word boundaries unmarked, such as other Tibetic languages (e.g. Dzongkha) or the Brahmic abugidas of mainland Southeast Asia (Thai, Lao, Khmer and Burmese).
Evaluation should also be extended to the full pipeline, with layout detection and line segmentation before recognition, where the page-level measures of Bourne et al. [ [5] apply; this study assumes correct line segmentation (§ 2.3). Scoring one recognizer on gold and on predicted line crops would separate segmentation error from recognition error, particularly on handwriting.
Post-editing effort should be measured directly and related to per-line error concentration. Correction time has been measured for OCR [40] and transcription time with ASR output visible for speech [46], and for machine translation edit distance to the post-edited output has been compared with perceived effort [25].
Conclusion
This study reports syllable- and word-level error rates for Tibetan OCR, to our knowledge for the first time. Nine configurations from three architecture classes, CTC recognizers, TrOCR encoder–decoders with and without post-training quantization and a vision-language model adapted with QLoRA, were evaluated on seven conditions drawn from typeset, woodblock and two handwritten script styles. Each was scored under CER, SER and three segmented word error rates on page-grouped held-out splits with bootstrap intervals. The rates were compared to one another, to the distribution of error over lines, and to the preservation of part-of-speech tags, named entities and translations computed from the output.
Tibetan OCR evaluations should report SER alongside CER. SER requires no segmenter, lexicon or model. CER broadly agrees with the higher-level rates on rankings, with one reversal, on which translation quality favours the ordering given by the segmented rates (§ 5.1). On decisions about a single system, such as whether joint training helped, CER and SER can disagree in sign (§ 5.2).
Where an LLM is used to annotate OCR output for downstream evaluation, we recommend that its coverage be reported and its failures scored rather than dropped. Gemini 2.5 Flash Lite returned no usable output for up to 1.4% of lines, more often as its input degraded (§ 5.3).
As rough guidance for downstream use, the SER needed ranges from 0.078 or below for part-of-speech preservation of 0.8 to 0.007 or below for entity preservation of 0.8, so no single acceptable rate serves every downstream use. No rate separates the systems that reach entity preservation of 0.6 from those that do not (§ 5.3).
Acknowledgments
We thank the Buddhist Digital Resource Center (BDRC) and WeBuddhist (formerly OpenPecha) for making the KhyentseWangpo, Karmapa8, Drutsa and Betsug datasets publicly available.
Claude (Anthropic), used through Claude Code, assisted with writing code for training and analysis, generating figures and tables, and drafting and revising the text.
Data Availability
The four datasets are public Hugging Face repositories (Table 1). TrOCR is fine-tuned from the public checkpoints microsoft/trocr-small-stage1 and microsoft/trocr-base-stage1, and Qwen3-VL from Qwen/Qwen3-VL-2B-Instruct; CRNN and Easter2.0 are trained from random initialization. BERT-SWER segmentation and part-of-speech tagging use the public KoichiYasuoka/tibetan-bert-base-upos. CER, SER and the three SWER variants are computed with the public tibetan-wer package (https://pypi.org/project/tibetan-wer/). All prompts are given in Appendix A. Training and evaluation code, trained checkpoints and recognizer outputs are available from the authors on reasonable request.
Prompts
All prompts are reproduced verbatim. Placeholders in braces are filled at call time. The Gemini prompts are issued at temperature 0.0 under a constrained JSON response schema.
Line transcription (Qwen3-VL)
Issued in the model’s native chat format alongside the line image (§ 3.2):
Transcribe the Tibetan text in this line image. Output only the text.
Word segmentation for Gem-SWER
Segment the following Tibetan text into words. Output ONLY the segmented
text with words separated by a pipe character (|). Keep the original
Tibetan characters and tsek marks (་) intact within each word. Do not add
any explanation or punctuation beyond the pipe separators. Text: {text}
Translation and entity extraction
One call returns both fields for a batch of four numbered lines (§ 3.7):
You are processing lines of Tibetan text, some of which are noisy OCR
output and may be garbled or nonsensical.
For EACH numbered line below, return:
"translation": a literal English translation, AT MOST 40 WORDS. Never
repeat a word, phrase, whitespace or punctuation mark to pad the output. If
the line is too garbled to translate, give the best partial translation you
can in a few words; return "" only if nothing at all is recoverable.
"entities": named entities appearing in that line, each as {"text": <the
Tibetan substring>, "type": one of "PER" (person, including teachers,
lineage holders and deities), "LOC" (place), "ORG" (monastery, school,
institution), "WORK" (title of a text or scripture)}. Return an empty list
if there are none.
Do not merge or reorder lines. Return exactly one result object per input
line, in order.
Lines:
{lines}
Entity-only fallback
Issued to a single line when the combined request above returns no parseable output. It is the prompt of Appendix § 9.§ 3 with the translation field and its instruction removed, so only the entity list is requested; this recovers entity coverage on degraded lines whose translation runs to the output-token cap.
References
- [1]Merveilles Agbeti-Messan, Pierrick Tranouez, Stéphane Nicolas, Clément Chatelain, and Thierry Paquet. 2026. A Benchmark of State-Space Models vs. Transformers and BiLSTM-based Models for Historical Newspaper OCR. arXiv:2604.00725 [cs.CV] https://arxiv.org/abs/2604.00725
- [2]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Binyuan Hui, Junyang Lin, Dayiheng Liu, Rui Men, Xuancheng Ren, Peng Wang, Haiyang Xu, An Yang, Bowen Yu, Jingren Zhou, et al. 2025. Qwen3-VL Technical Report. arXiv:2511.21631 [cs.CV] https://arxiv.org/abs/2511.21631
- [3]Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022. BEiT: BERT Pre-Training of Image Transformers. In International Conference on Learning Representations (ICLR). OpenReview.net, Virtual Event. arXiv:2106.08254 [cs.CV] https://arxiv.org/abs/2106.08254
- [4]Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Nenploev, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K. Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Vienna, Austria, 238–255. doi:10.18653/v1/2025.acl-short.20
- [5]Jonathan Bourne, Wiza Simbeve, and Joseph Nicholas. 2026. Character Error Vector: Decomposable Errors for Page-Level OCR Evaluation. arXiv:2604.06160 [cs.CV] https://arxiv.org/abs/2604.06160
- [6]Collin J. Brown. 2024. Improved Neural Word Segmentation for Standard Tibetan. In Proceedings of the 2nd Workshop on Resources and Technologies for Indigenous, Endangered and Lesser-resourced Languages in Eurasia (EURALI) @ LREC-COLING 2024. ELRA and ICCL, Torino, Italia, 12–17. https://aclanthology.org/2024.eurali-1.2DOI
- [7]Kartik Chaudhary and Raghav Bali. 2022. Easter2.0: Improving Convolutional Models for Handwritten Text Recognition. arXiv:2205.14879 [cs.CV] https://arxiv.org/abs/2205.14879
- [8]Thibault Clérice. 2022. Ground-truth Free Evaluation of HTR on Old French and Latin Medieval Literary Manuscripts. In Proceedings of the Computational Humanities Research Conference (CHR 2022) (CEUR Workshop Proceedings, Vol. 3290). CEUR-WS.org, Antwerp, Belgium, 1–24. https://ceur-ws.org/Vol-3290/
- [9]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022a. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. In Advances in Neural Information Processing Systems (NeurIPS 2022), Vol. 35. Curran Associates, Inc., Red Hook, NY, USA, 30318–30332. arXiv:2208.07339 https://arxiv.org/abs/2208.07339
- [10]Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022b. 8-bit Optimizers via Block-wise Quantization. In 10th International Conference on Learning Representations (ICLR 2022). OpenReview.net, Virtual Event. arXiv:2110.02861 https://arxiv.org/abs/2110.02861
- [11]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS 2023), Vol. 36. Curran Associates, Inc., Red Hook, NY, USA, 10088–10115. arXiv:2305.14314 https://arxiv.org/abs/2305.14314
- [12]A. Seza Doğruöz, Xixian Liao, Verena Blaschke, Jakob Prange, Senyu Li, and David Ifeoluwa Adelani. 2026. Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages. arXiv:2607.02235 [cs.CL] https://arxiv.org/abs/2607.02235
- [13]Franz Xaver Erhard. 2025. Text and Information Extraction for Modern (post-1950) Tibetan Newspapers and Print Publications with Transkribus. Revue d’Études Tibétaines 74 (Feb. 2025), 128–171. https://d1ljdgw69xsqx0.cloudfront.net/digitalhimalaya/collections/journals/ret/pdf/ret_74_04.pdf
- [14]Xiyan Fu and Wei Liu. 2025. How Reliable is Multilingual LLM-as-a-Judge? arXiv:2505.12201 [cs.CL] https://arxiv.org/abs/2505.12201
- [16]Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In Proceedings of the 23rd International Conference on Machine Learning (ICML 2006). ACM, Pittsburgh, Pennsylvania, USA, 369–376. doi:10.1145/1143844.1143891DOI
- [17]Rachael M. Griffiths. 2024. Handwritten Text Recognition (HTR) for Tibetan Manuscripts in Cursive Script. Revue d’Études Tibétaines 72 (July 2024), 43–51. https://archivevnepal.s3.amazonaws.com/digitalhimalaya/collections/journals/ret/pdf/ret_72_03.pdf
- [18]Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. Hallucinations in Large Multilingual Translation Models. Transactions of the Association for Computational Linguistics 11 (2023), 1500–1517. doi:10.1162/tacl_a_00615
- [19]Ahmed Hamdi, Axel Jean-Caurant, Nicolas Sidère, Mickaël Coustaty, and Antoine Doucet. 2019. An Analysis of the Performance of Named Entity Recognition over OCRed Documents. In 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL). IEEE, Champaign, IL, USA, 333–334. doi:10.1109/JCDL.2019.00057
- [20]Fares Hedayati, Jike Chong, and Kurt Keutzer. 2010. Recognition of Tibetan Wood Block Prints with Generalized Hidden Markov and Kernelized Modified Quadratic Distance Function. Technical Report UCB/EEC-2010-138. Electrical Engineering and Computer Sciences, University of California at Berkeley. https://www2.eecs.berkeley.edu/Pubs/TechRpts/2010/EEC S-2010-138.html
- [21]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In 10th International Conference on Learning Representations (ICLR 2022). OpenReview.net, Virtual Event. arXiv:2106.09685 https://arxiv.org/abs/2106.09685
- [22]Cheng Huang, Nyima Tashi, Fan Gao, Yutong Liu, Jiahao Li, Hao Tian, Siyang Jiang, Thupten Tsering, Ban Ma-bao, Renzeg Duojie, Gadeng Luosang, Rinchen Dongrub, Dorje Tashi, Jin Zhang, Xiao Feng, Hao Wang, Jie Tang, Guojie Tang, Xiangxiang Wang, Jia Zhang, Tsendgar Lee, and Yongbin Yu. 2025. Tibetan Language and AI: A Comprehensive Survey of Resources, Methods and Challenges. arXiv:2510.19144 [cs.CL] https://arxiv.org/abs/2510.19144
- [23]Alexandre Jaud, Ahmed Hamdi, Antoine Doucet, Adam Jatowt, and Mickaël Coustaty. 2025. Beyond CER and WER: How Does OCR Really Impact Information Retrieval?. In Proceedings of the 2025 ACM/IEEE Joint Conference on Digital Libraries (JCDL ’25). IEEE, DeKalb, IL, USA, 30–39. doi:10.1109/JCDL67857.2025.00014DOI
- [25]Maarit Koponen. 2012. Comparing Human Perceptions of Post-Editing Effort with Post-Editing Operations. In Proceedings of the Seventh Workshop on Statistical Machine Translation. Association for Computational Linguistics, Montréal, Canada, 181–190. https://aclanthology.org/W12-3123/
- [26]Anh Le, Thanh Lam, and Dung Nguyen. 2025. A Survey on Vietnamese Document Analysis and Recognition: Challenges and Future Directions. arXiv:2506.05061 [cs.CV] https://arxiv.org/abs/2506.05061
- [27]Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2023. TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models. In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI 2023), Vol. 37. AAAI Press, Washington, DC, USA, 13094–13102. arXiv:2109.10282 doi:10.1609/aaai.v37i11.26538DOI
- [28]Queenie Luo and Yung-Sung Chuang. 2024. Cleansing Jewel: A Neural Spelling Correction Model Built On Google OCR-ed Tibetan Manuscripts. ACM Transactions on Asian and Low-Resource Language Information Processing 23, 5, Article 73 (May 2024), 11 pages. doi:10.1145/3654811DOI
- [29]Marieke Meelen and Nathan W. Hill. 2017. Segmenting and POS Tagging Classical Tibetan Using a Memory-Based Tagger. Himalayan Linguistics 16, 2 (2017), 64–89. doi:10.5070/H916234501DOI
- [30]Jacob Moore and Paula Lauren. 2025. Optimizing T5 for Lightweight Tibetan-English Translation. Research Square preprint. doi:10.21203/rs.3.rs-7409829/v1
- [31]Jacob Moore, Sheng Li, and Paula Lauren. 2026. Evaluating Tibetan ASR With Segmented Word Error Rate: Beyond Character-Level Metrics. IEEE Access 14 (2026), 101790–101805. doi:10.1109/ACCESS.2026.3709206DOI
- [32]Hay Mar Soe Naing, Aye Mya Hlaing, Win Pa Pa, Xinhua Hu, Ye Kyaw Thu, Chiori Hori, and Hisashi Kawai. 2015. A Myanmar Large Vocabulary Continuous Speech Recognition System. In Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, Hong Kong, 320–327.DOI
- [33]Sebastian Nehrdich and Kurt Keutzer. 2026. MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for Pāli, Sanskrit, Buddhist Chinese, and Tibetan. arXiv:2601.06400 [cs.CL] https://arxiv.org/abs/2601.06400
- [34]Clemens Neudecker, Konstantin Baierer, Mike Gerber, Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. 2021. A survey of OCR evaluation tools and metrics. In Proceedings of the 6th International Workshop on Historical Document Imaging and Processing (HIP ’21). ACM, Lausanne, Switzerland, 13–18. doi:10.1145/3476887.3476888DOI
- [35]Sithar Norbu, Pema Choejey, Tenzin Dendup, Sarmad Hussain, and Ahmed Muaz. 2010. Dzongkha Word Segmentation. In Proceedings of the 8th Workshop on Asian Language Resources (COLING 2010). CoLing 2010 Organizing Committee, Beijing, China, 95–102. https://aclanthology.org/W10-3213/
- [37]Maja Popović. 2015. chrF: Character n-gram F-score for Automatic MT Evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Lisbon, Portugal, 392–395. doi:10.18653/v1/W15-3049DOI
- [38]Matt Post. 2018. A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers. Association for Computational Linguistics, Brussels, Belgium, 186–191. doi:10.18653/v1/W18-6319DOI
- [39]Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The Curious Case of Hallucinations in Neural Machine Translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 1172–1183. doi:10.18653/v1/2021.naacl-main.92DOI
- [40]Shruti Rijhwani, Daisy Rosenblum, Michayla King, Antonios Anastasopoulos, and Graham Neubig. 2023. User-Centric Evaluation of OCR Systems for Kwak’wala. In Proceedings of the Sixth Workshop on the Use of Computational Methods in the Study of Endangered Languages. Association for Computational Linguistics, Remote, 19–29. https://aclanthology.org/2023.computel-1.4/arxiv.org/abs/2302.13410
- [41]Zach Rowinski and Kurt Keutzer. 2016. Namsel: An Optical Character Recognition System for Tibetan Text. Himalayan Linguistics 15, 1 (2016), 12–30. doi:10.5070/H915129937DOI
- [42]Christina Sabbagh. 2025. Enhanced HTR Accuracy for Tibetan Historical Texts: Optimising Image Pre-processing for Improved Transcription Quality. Revue d’Etudes Tibétaines 74 (Feb. 2025), 81–127. https://doi.org/10.2143/RET.74.03. pdf
- [43]Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. BLEURT: Learning Robust Metrics for Text Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 7881–7892. doi:10.18653/v1/2020.acl-main.704DOI
- [44]Baoguang Shi, Xiang Bai, and Cong Yao. 2017. An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 11 (2017), 2298–2304. doi:10.1109/TPAMI.2016.2646371DOI
- [45]Peng Shu, Junhao Chen, Zhengliang Liu, Hui Wang, Zihao Wu, Tianyang Zhong, Yiwei Li, Huaqin Zhao, Hanqi Jiang, Yi Pan, Yifan Zhou, Constance Owl, Xiaoming Zhai, Ninghao Liu, Claudio Saunt, and Tianming Liu. 2024. Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translation. arXiv:2411.11295 [cs.CL] https://arxiv.org/abs/2411.11295
- [46]Matthias Sperber, Gharan Neubig, Jan Niehues, Satoshi Nakamura, and Alex Waibel. 2017. Transcribing Against Time. Speech Communication 93 (2017), 20–30. https://arxiv.org/abs/1709.05227
- [47]D. K. Thennal, Jesin James, Deepa Padmini Gopinath, and Muhammed Ashraf K. 2025. Advocating Character Error Rate for Multilingual ASR Evaluation. In Findings of the Association for Computational Linguistics: NAACL 2025. Association for Computational Linguistics, Albuquerque, New Mexico, 4941–4950. arXiv:2410.07400 doi:10.18653/v1/2025.findings-naacl.277DOI
- [48]Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training Data-Efficient Image Transformers & Distillation Through Attention. In Proceedings of the 38th International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research, Vol. 139). PMLR, Virtual Event, 10347–10357. https://proceedings.mlr.press/v139/touvron21a/touvron21a.pdfarxiv.org/abs/2012.12877
- [49]Sam van Schaik. 2014. Towards a Tibetan Palaeography: Developing a Typology of Writing Styles in Early Tibet. In Manuscript Cultures: Mapping the Field, Jörg Quenzer, Dmitry Bondarev, and Jan-Ulrich Sobisch (Eds.). De Gruyter, Berlin, 299–338. doi:10.1515/9783110225631.299DOI
- [50]Daniel van Strien, Kaspar Beelen, Mariana Coll Ardanuy, Kasra Hosseini, Barbara McGillivray, and Giovanni Colavizza. 2020. Assessing the Impact of OCR Quality on Downstream NLP Tasks. In Proceedings of the 12th International Conference on Agents and Artificial Intelligence (ARTIDIGH). SciTePress, Valletta, Malta, 484–496. doi:10.5220/0009169004840496DOI
- [51]Ari Vesalainen, Eetu Mäkelä, Laura Ruotsalainen, and Mikko Tolonen. 2026. Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision–Language Model. arXiv:2602.14524 [cs.CV] https://arxiv.org/abs/2602.14524
- [52]Hui Wang, Fei Gao, Yue Zhao, Li Yang, Jianjian Yue, and Huilin Ma. 2020. Multitask Learning with Local Attention for Tibetan Speech Recognition. Complexity 2020 (2020), 8894566. doi:10.1155/2020/8894566
- [53]Xiaojuan Wang and Weilan Wang. 2024. An Unsupervised Character Recognition Method for Tibetan Historical Document Images Based on Deep Learning. Applied Sciences 14, 5 (2024), 2142. doi:10.3390/app14052142DOI
- [54]Guixian Xu, Yide Liang, Zeli Su, Xuexian Song, Ziyin Zhang, Yushuang Dong, Ting Zhang, and Xu Han. 2026. TibFSuite: A Comprehensive Resource Suite for Tibetan Vision–Language Modeling. arXiv:2605.26601 [cs.CL] https://arxiv.org/abs/2605.26601
- [55]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenggen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https://arxiv.org/abs/2505.09388
- [56]Ce Zhang, Weilan Wang, and Guowei Zhang. 2022. Construction of a Character Dataset for Historical Uchen Tibetan Documents under Low-Resource Conditions. Electronics 11, 23 (2022), 3919. doi:10.3390/electronics11233919DOI
- [57]Jin Zhang, Fan Gao, Lobsang Yeshi, Dorje Tashi, Xiangshi Wang, Nyima Tashi, and Gadeng Luosang. 2025. Cross-Domain Tibetan Named Entity Recognition via Large Language Models. Electronics 14, 1 (2025), 111. doi:10.3390/electronics14010111
- [58]Yue Zhao, Jianjian Yue, Xiaona Xu, Licheng Wu, and Xiali Li. 2019. End-to-End-Based Tibetan Multitask Speech Recognition. IEEE Access 7 (2019), 162519–162529. doi:10.1109/ACCESS.2019.2952406DOI
- [60]Maoke Zhou, Xiaoke Qi, Wei Bao, and Xiaobing Zhao. 2026. TibHCR: A Large-Scale Dataset for Offline Handwritten Tibetan Character Recognition. IEICE Transactions on Information and Systems E109-D, 1 (2026), 193–205. doi:10.1587/transinf.2025EDP7098


