WorksComputer Science
Do Rule-Based Tools Help LLMs Translate Endangered Uralic Languages in the Prompt or over MCP?
Abstract
Komi-Zyrian, Moksha, Erzya and Udmurt have mature rule-based resources, among them finite-state analysers, constraint grammars and bilingual dictionaries, which language models do not use. We ask whether giving a model access to these resources improves machine translation from the four languages into English, and whether it matters how the access is provided: as analyses computed in advance and written into the prompt, or as tools that the model calls itself through the Model Context Protocol (MCP). We evaluate manually: an expert linguist read every English output of the strongest model under three conditions, 816 sentences in all, accepted or rejected each one, and named the main error in every rejected translation. With the analyses in the prompt, the number of accepted translations rises from 13 to 43 out of 272; with the tools attached it rises to 55. The gain comes from the bilingual dictionary. For Udmurt, whose dictionary has no English side, the prompt-based condition brings no gain at all, while for the other three languages an English gloss reaches the prompt for 74 to 80% of the source words. Tool calling is also much harder to get working than prompt injection. Of 31 open-weight models, 14 never call a tool, 11 of them because they cannot produce a usable call, and half of the calls that are made name a language the toolkit does not support. We release the data set, every model output and all tool-call logs.
Introduction
Komi-Zyrian, Moksha, Erzya and Udmurt are poorly supported by current language models [18]. At the same time they are well described. Decades of finite-state and constraint-grammar work [22, 20] have produced morphological analysers, generators and bilingual dictionaries for each of them. These resources are maintained in the GiellaLT infrastructure [13] and are available through the UralicNLP Python library [7]. UralicMCP [9] proposes to bring the two together by exposing the resources to a language model as tools that the model can call.
So far the proposal has been demonstrated rather than evaluated, and it leaves a design question open. There are two ways to bring linguistic knowledge to a model. The analyses can be computed in advance and written into the prompt, which requires nothing of the model beyond reading them. Alternatively, the tools can be attached to the model, which then decides what to look up; the Model Context Protocol [3] makes this easy to set up. The second option is the more attractive one, since it scales to inputs that nobody has pre-processed and since it is how agentic systems are being built today.
We compare the two options on a 68-sentence parallel test set in the four languages, with 31 open-weight instruction-tuned models of up to 27B parameters and seven experimental conditions. We decided early on to rely on manual evaluation. An expert linguist read every English output of the strongest model, Qwen3.8-27B, under three conditions, accepted or rejected each output, and assigned an error category to every rejected one. The reason is that the only reference translation we have is a free literary one, and against such a reference an automatic metric mostly measures how closely a system happens to match one translator’s wording. We still report chrF++ and BLEU in Appendix § 10 so that readers can compare with other work, but none of our claims rests on them.
Writing the analyses into the prompt raises the number of accepted translations from 13 to 43 out of 272. The gain is confined to the languages whose dictionary has an English side; for Udmurt, where no English gloss reaches the prompt, there is none. Giving the same model the tools instead raises the count further, to 55, but this comparison is only possible for the one model in the roster that calls the tools on every sentence and still answers in prose. Across the roster the picture is different. 14 of the 31 models never emit a tool call, 11 of them because they cannot produce one that a parser reads, and even the models that do call send an unsupported language code in half of their calls. Translation into the four languages, finally, fails under every condition, with 23% of the outputs in the wrong language and 13% stuck in a repetition loop. What is missing in that direction is a model of the target language itself, and the rule-based resources cannot supply it.
The rest of the paper is organised as follows. § 2 reviews related work, § 3 describes the data, § 4 the conditions and models, § 5 the manual evaluation, § 6 the tool-calling behaviour of the full roster, and § 7 concludes.
Related work
The Finno-Ugric languages are severely underrepresented in language model development [18], and the gap is visible already at the level of the tokeniser [5]. Hämäläinen [10] argues that calling these languages “low-resource” is misleading, since they lack text corpora rather than linguistic description. Two kinds of response have been proposed. The first changes the model: Purason et al. [18] continue pretraining and instruction-tune models for Võro, Livonian and Komi. The second leaves the model unchanged and changes what it receives at inference time. Hämäläinen [8] take this route by putting the dictionary translations of candidate lemmas into a ChatGPT prompt in order to disambiguate Erzya and Skolt Sami sentences, reaching 41% and 50% accuracy, and UralicMCP [9] goes one step further by letting the model fetch the material itself.
The study closest to ours is Tereshchenko et al. [25], who evaluate GPT models on translation between Finnish and the same four languages, compare reasoning and non-reasoning architectures, and report a 16-point difference in refusal rates. That work uses frontier APIs and no linguistic tools. Earlier work on machine translation of Uralic languages includes rule-based systems [11, 21, 15] and neural systems [27, 6, 1].
Token fertility as a measure of how well a tokeniser represents a language follows Rust et al. [23]. For Uralic languages, Chelombitko and Komissarov [5] quantify the imbalance and show that it can be remedied, reducing the compression ratio for Finnish from 3.41 under LLaMA-2 to 1.18 with a monolingual BPE tokeniser; Xu and Kim [26] find that morphologically better-aligned segmentation improves downstream tagging across six Uralic languages; and Chelombitko et al. [4] derive per-language vocabulary-size recommendations from the lexicon.
Data
We use the 68-sentence parallel set of literary prose introduced by Alnajjar et al. [2], which covers English, Finnish, Komi-Zyrian, Moksha, Erzya and Udmurt. The set is small, but to our knowledge it is the only one that is parallel across all four endangered languages with a single English source and checked translations. One English text was translated independently into each language, so the English side is identical across the five Uralic pairs. Each sentence carries a sentiment polarity label (35 negative, 33 positive); we do not use these labels in this paper, and the Limitations explain why.1
The morphological analysers cover the set well. They return at least one reading for 91.1% of the Udmurt word tokens and for 95.4% to 98.5% of the tokens in the other four languages, so the results below cannot be explained by the tools failing to parse the input.
Experimental setup
Every condition sends the same request to the same model: a system prompt that asks for a translation and nothing else, and a user message with the source sentence. The conditions differ only in what is added to this request. Figure 1 shows the shared request and the additions made by the INJECT and MCP branches, and Table 1 lists all seven conditions. Models often wrap their answer in code fences, commentary or tool transcripts, so the translation is first recovered from the raw output; Appendix § 15 describes this step and how it is checked.

Figure 1. How conditions are built. THE SAME REQUEST EVERY TIME. “You are a professional translator. . . Output ONLY the translation.” system prompt. “Translate from Komi-Zyrian into English:” Керкаас вöлi пöдтымöн жар. user message. AND THEN ONE BRANCH ADDS. ZEROSHOT – nothing at all. INJECT – to the user message, what the tools would have returned: Керкаас lemma: керка analysis: кер-ка+N+Sg+Ine+PxSg3 dictionary: керка → house. MCP – to the system prompt: “your own knowledge . . . may be unreliable”, the list of the five tools, and “1. Analyse. . . 2. Look the lemmas up. . . 3. use the inflector”. And six tool schemas, attached to the call. MCP_PROMPT is this branch without the schemas. UralicNLP tools six, one, or thirteen. model. result. call. only the MCP branch can reach these. “The air in the cottage was hot and suffocating.” THE OUTPUT.
| Condition | What is added to the request | Tools | Models |
| ZEROSHOT | nothing | – | 31 |
| MCP_PROMPT | the tool-use system prompt, with no tools attached (control) | – | 31 |
| INJECT | lemma, reading and dictionary entry of every source word, computed in advance, in the user message | 6 | 31 |
| INJECT_RICH | the same, computed with the thirteen-tool surface | 13 | 14 |
| MCP | the tool-use system prompt, with the tool schemas attached; the model decides whether to call | 6 | 31 |
| MCP_COARSE | as MCP, with one sentence-level tool | 1 | 31 |
| MCP_RICH | as MCP, with the thirteen-tool surface | 13 | 16 |
Table 1. The seven conditions. Tools is the surface that supplies the linguistic material; Models is how many of the 31 models ran the condition for Komi-Zyrian. The other four languages ran ZEROSHOT, MCP_PROMPT, INJECT, MCP and MCP_COARSE on ten models.
Resources and tool surfaces. All linguistic information comes from UralicNLP [7], which wraps the finite-state analysers and generators, the constraint grammars and the bilingual dictionaries of the GiellaLT infrastructure. We expose the library to the models through three tool surfaces. The six-tool surface is the reference surface of UralicMCP [9]: an analyser, a lemmatiser, an inflector, a morpheme segmenter, a dictionary lookup and a list of supported languages, each of which operates on one word. The one-tool surface is a coarse interface we built for this study: it takes a whole sentence and returns the readings and glosses of every word in a single call. It tests the hypothesis that composable per-word primitives suit a programmer but not a small model, which would otherwise have to plan ten calls where one would do. The thirteen-tool surface extends the six tools with sentence-level glossing, resolution of ISO language codes and dictionary search by inflected form. The last of these cannot be done on the per-word surface, which requires the caller to know the lemma already.
Conditions. Table 1 lists the seven conditions; the task instruction and the user message are identical in all of them. Three design decisions need a word of explanation. MCP_PROMPT is a control rather than a system: MCP changes the system prompt and attaches tools at the same time, so a drop in score under MCP could be caused by either change, and keeping the prompt while removing the tools separates the two. INJECT and INJECT_RICH contain exactly the information that the six-tool and the thirteen-tool surfaces return, computed in advance and written into the user message: every source word with its lemma, its reading and its dictionary translations. These two conditions separate the information from the act of fetching it, and they are open to models that cannot call tools.
Models and decoding. We use open-weight instruction-tuned models only, with at most 27B parameters. Decoding is greedy with no repetition penalty, the weights are in bf16, and the context window is 32,768 tokens. The roster is not a full grid. Ten models from 0.6B to 27.8B parameters ran every direction; a further 21 models ran Komi-Zyrian only and serve the analysis of tool-calling behaviour in § 6. Four models fine-tuned for function calling are included on purpose, since tool calling is the behaviour under study. All 62,920 records are released with their tool-call transcripts and usage counters,2 so that Appendix § 15 can be checked without a GPU.
Evaluation. We evaluate manually, as described in § 5. We also computed chrF++ and BLEU for every record, but we do not use them as evidence: the only reference is a free literary translation, and on this data the scores mainly reward agreement with that translator’s wording. Appendix § 10 reports them, together with the reasons for this decision.
Human evaluation and error analysis
An expert linguist read every English output of Qwen3.8-27B against the reference, sentence by sentence, accepted or rejected it, and assigned one error category to every rejected output. The analysis covers the 68 source sentences in each of the four languages under three conditions, ZEROSHOT, INJECT and MCP, which gives 816 judgements. We chose this model because it is the only one in the roster that calls the tools on practically every sentence and still answers in prose, so it is the only model for which the three conditions really are three different treatments of the same sentence. What the other models do when the tools are attached is the subject of § 6.
An output was accepted when it conveyed the content of the reference sentence: the same participants, the same event, no invented or omitted material. Fluency and wording were not criteria, and a loose but faithful rendering was accepted. Table 2 reports the result. MCP has the largest number of accepted outputs (55 of 272), followed by INJECT (43) and ZEROSHOT (13). The advantage is not uniform across languages: INJECT and MCP tie for Moksha and Erzya, and INJECT brings no gain over ZEROSHOT for Udmurt.
| Language | ZEROSHOT | INJECT | MCP |
| Komi-Zyrian | 8 (11.8%) | 17 (25.0%) | 23 (33.8%) |
| Moksha | 1 (1.5%) | 11 (16.2%) | 11 (16.2%) |
| Erzya | 0 (0.0%) | 11 (16.2%) | 11 (16.2%) |
| Udmurt | 4 (5.9%) | 4 (5.9%) | 10 (14.7%) |
| Total | 13 (4.8%) | 43 (15.8%) | 55 (20.2%) |
Table 2. Outputs accepted as conveying the content of the English reference, by language and condition. Every output not accepted carries exactly one primary error category, defined in § 5.1 and counted in Table 3.
The reason Udmurt behaves differently lies in the dictionaries. The dictionaries in UralicNLP are
directional, and the Udmurt dictionary has no English side. Under INJECT, an English gloss reaches the prompt for 80.3% of the Komi-Zyrian source tokens, 73.9% of the Moksha tokens and 78.5% of the Erzya tokens, but for none of the Udmurt tokens, whose prompt therefore contains lemmas and morphological readings only. The gain appears where the glosses arrive, which separates the effect of the dictionary from the effect of having any linguistic material in the prompt.
Error taxonomy
Every rejected output received one primary error category from a taxonomy shared across the four languages. Table 3 defines the categories and gives the counts.
| Primary error | Definition | Komi-Zyrian | Moksha | Erzya | Udmurt |
| Wholesale replacement | the situation of the reference is replaced, or several independent major changes are made to it | 39/10/9 | 48/21/26 | 54/24/25 | 42/41/23 |
| Lexical substitution | an action, entity, property or idiomatic meaning is replaced while part of the sentence structure survives | 10/16/15 | 6/12/9 | 9/11/11 | 11/14/13 |
| Participant, referent | who acts or who experiences the event changes: person, gender, kinship or the direction of an interaction | 3/9/9 | 2/8/13 | 1/10/11 | 3/2/7 |
| Polarity, modality, act | negation, propositional force or the status as statement, question or command changes; incompatible event states included | 3/4/3 | 2/5/2 | 0/2/4 | 1/2/4 |
| Omission | an important proposition or semantic component of the reference is missing | 2/7/3 | 8/8/5 | 4/9/5 | 5/4/6 |
| Unsupported addition | a substantive detail that the English reference does not establish is introduced | 2/5/4 | 0/2/2 | 0/1/1 | 1/0/4 |
| Temporal, quantitative | a salient age, duration or temporal relation between events is altered | 0/0/1 | 0/1/0 | 0/0/0 | 0/0/0 |
| Untranslated content | a non-English or name-like form is kept where an English meaning is required | 1/0/1 | 1/0/0 | 0/0/0 | 1/1/1 |
| Rejected in total | 60/51/45 | 67/57/57 | 68/57/57 | 64/64/58 |
Table 3. The error taxonomy, and the primary error category of every rejected output of Qwen3.8-27B as ZEROSHOT/INJECT/MCP counts out of 68 sentences per cell. With the accepted outputs of Table 2 every column sums to 68.
Under ZEROSHOT, wholesale replacement is the most frequent category in every language, with 39, 48, 54 and 42 of the 68 outputs in Komi-Zyrian, Moksha, Erzya and Udmurt. The injected analyses move most of these outputs out of that category, to 10, 21 and 24 in the three languages whose dictionary has an English side, while in Udmurt the count stays at 41. The narrower categories do not decrease; lexical substitutions and participant errors in particular become more frequent in every language with an English dictionary side. In other words, the analyses turn an unrelated output into a nearly correct one, and the errors that remain are word-level errors, such as a wrong choice among the glosses of a dictionary entry or a person or possessor misread from the morphological analysis. The subsections below give one example for every category that occurs in each language, each headed by the condition and the sentence identifier.
Komi-Zyrian
Wholesale replacement. ZEROSHOT, sentence neg-006:
Source: Синъясыс пемдiсны; гор¨одт¨ог Татьяна сь¨окыда уси джоджас. Output: The sun had set; without the city, Tatiana felt very lonely.
Reference: His vision went dark and without making a sound he fell heavily to the floor.
The blackout and collapse in the reference are replaced by sunset and loneliness, changing the setting, participants, and event.
Lexical and conceptual substitution. INJECT, sentence neg-003:
Source: - Тай¨о дедыд томналiс, - ск¨орысь шуис Федя. Output: “Your grandfather was young,” Feda said angrily.
Reference: - Grandpa locked it, - Fedya said angrily.
The central predicate changes from locking something to being young, even though the speaker and angry delivery are partly preserved.
Participant and referent error. INJECT, sentence neg-011:
Source: - Ой, ёртъяс, пропадитi! Output: Oh, my friends, you’re lost!
Reference: - Oh, comrades, I’m done for!
The first-person predicament becomes a second-person predicament: the threatened individual is no longer the speaker.
Polarity, modality and speech act. INJECT, sentence neg-014:
Source: Василий Потупчик лӧгалiс Кулуканов вылӧ да некод водзын эз дзеб ассьыс лӧглунсӧ. Output: Vasily Potupchik was angry at Kulukanov and never showed his anger in front of anyone.
Reference: Vasili Potupchik hated Kulukanov and did not hide his hatred from anyone.
The reference explicitly states that the hatred was not concealed, whereas the output says that the anger was never shown. The output also weakens hatred to anger.
Omission. INJECT, sentence neg-018:
Source: Керкаас вӧлi пӧдтӧм жар. Output: It was very hot in the house.
Reference: The air in the cottage was hot and suffocating.
The output retains the hot interior but omits its suffocating quality, an important component of the reference description.
Unsupported addition. MCP, sentence pos-006:
Source: Дона, радейтана мама! Output: Donna, dear mother!
Reference: Dear, precious mother!
Komi дона is the adjective ‘dear’. Capitalised at the start of the exclamation it looks like a personal name, and the model transliterated it as one, which adds a participant and drops the affectionate description. The same mechanism produces Musa below, from муса ‘beloved’, and Kelmoma in Moksha, from кельгома ‘beloved’.
Temporal and quantitative error. MCP, sentence neg-027:
Source: Эрд выйӧ бӧрдiгтырий котӧрӧн петiс Клава. Output: After crying, Klava came out at a run to the top of the clearing.
Reference: Klava ran into the clearing crying.
The reference presents crying and running as simultaneous. The output makes the crying precede the running and also adds an unsupported spatial detail.
Untranslated content. ZEROSHOT, sentence pos-007:
Source: Муса¨ой! Output: Musa! Reference: Dear mother!
The vocative Муса¨ой ‘my beloved’ is read as the name Musa, so the output does not convey a maternal address to an English reader.
Moksha
For Moksha, INJECT and MCP each yield 11 accepted outputs against one under ZEROSHOT. The equal totals hide different error profiles: MCP has more wholesale replacements and more participant errors than INJECT, and fewer lexical substitutions and omissions.
Wholesale replacement. ZEROSHOT, sentence neg-025:
Source: Сонь тарсь улаксоц. Output: The sun is setting. Reference: His chin trembled.
An event involving a trembling chin is replaced by an unrelated statement about sunset.
Lexical and conceptual substitution. INJECT, sentence pos-009:
Source: - Цебярьхть каготкятне, Трофим Сергеевич! Output: What lovely coats, Trofim Sergeievich! Reference: - Good papers, Trofim Sergejevitsh.
The evaluated objects change from papers to coats; the positive evaluation and the addressee alone do not preserve the proposition.
Participant and referent error. INJECT, sentence neg-025:
Source: Сонь тарсь улаксоц. Output: She is trembling. Reference: His chin trembled.
The output changes the male referent to a female referent and generalizes the trembling from a chin to the whole person.
Polarity, modality and speech act. INJECT, sentence pos-010:
Source: Аляц рахазевсь: Output: Father was silent: Reference: Father began to laugh.
Beginning to laugh is replaced by being silent, producing an incompatible event state.
Omission. INJECT, sentence neg-022:
Source: Татьянась гневондазь ювади: Output: Tatiana’s anger flared up: Reference: Tatjana exclaimed angrily:
The emotional state is retained, but the act of exclaiming is omitted.
Unsupported addition. MCP, sentence pos-006:
Source: Кельгома родной тядяй! Output: My dear mother, I love you! Reference: Dear, precious mother!
The output adds an explicit declaration of love. Although compatible with the affectionate address, it is not asserted by the reference. This is a flagged borderline rejection under the conservative addition policy.
Temporal and quantitative error. INJECT, sentence neg-019:
Source: Татьянась ульсь осал и лофтана, эрязень эсь колмогемонь ветие кизонзон фалу трудендазь, радостьта няйсь кржа. Output: Tatiana was thin and pale; in her thirtieth year of life she was always working hard and showed little joy. Reference: Tatjana was thin and pale, she had lived all thirty-five years of her life in constant work. She had seen little joy.
The age and duration change from an entire thirty-five-year life of work to work in the thirtieth year. The reference’s experience of little joy is also recast as showing little joy.
Untranslated content. ZEROSHOT, sentence pos-006:
Source: Кельгома родной тядяй! Output: My dear Kelmoma! Reference: Dear, precious mother!
The adjective кельгома ‘beloved’ is read as the name Kelmoma, which replaces the maternal address instead of conveying it.
Erzya
No ZEROSHOT output is accepted for Erzya under the reference-equivalence criterion, against 11 under each of INJECT and MCP. This does not mean that every ZEROSHOT output lacked useful content; partially correct sentences still fail the sentence-level criterion.
Wholesale replacement. ZEROSHOT, sentence neg-002:
Source: Ледьс руужясто ды наймунонть прязо лазовсь пельксэн-пелькс. Output: The cold wind and the rain poured down in sheets. Reference: He fired his shotgun and smashed the merchant’s head.
The shotgun attack and injury are replaced by a weather description, leaving no equivalent account of the reference event.
Lexical and conceptual substitution. INJECT, sentence neg-025:
Source: Улоозо ноцквтевсь. Output: The chin has been dislocated. Reference: His chin trembled. Trembling is replaced by dislocation, changing the described physical condition.
Participant and referent error. INJECT, sentence pos-010:
Source: Тетязо пейдезевсь: Output: The aunt laughed. Reference: Father began to laugh.
The action of laughing remains, but the father is replaced by an aunt.
Polarity, modality and speech act. MCP, sentence pos-016:
Source: Весе мекс-бути пейдсть. Output: Why are they all laughing? Reference: Everyone was laughing about something.
A statement reporting laughter is converted into a question asking for its cause.
Omission. INJECT, sentence neg-022:
Source: Татьяна кежейстэ рангстась: Output: Tatiana was furious. Reference: Tatjana exclaimed angrily:
The output expresses anger but omits the speech event specified by exclaimed.
Unsupported addition. MCP, sentence neg-005:
Source: Куростонть маряви авань кежей рангома: Output: The mother’s angry scream was heard from the grove: Reference: From outside came an angry woman’s voice:
The output narrows the unspecified woman to a mother and outside to a grove. These details are unsupported by the English reference; source-language adjudication would be needed to determine whether they are genuine translation errors.
Udmurt
For Udmurt, ZEROSHOT and INJECT each yield four accepted outputs and MCP ten. INJECT therefore brings no gain in the one language whose prompt carries no English glosses. MCP halves the number of wholesale replacements, while lexical, participant and omission errors remain.
Wholesale replacement. ZEROSHOT, sentence pos-007:
Source: Дуное! Output: Damn it! Reference: Dear mother!
An affectionate maternal address becomes profanity, replacing both the referent and the communicative function.
Lexical and conceptual substitution. MCP, sentence pos-010:
Source: Атай серекъяны ¨одъяз: Output: Father began to mock: Reference: Father began to laugh.
Laughing becomes mocking, adding a derisive meaning that the reference does not establish.
Participant and referent error. MCP, sentence neg-020:
Source: Пинал дыръяз солы дышетскыны ¨оз кылды: узыр бускельзы дорын медъяськыса ужалляз. Output: In his youth, he was unable to study: he was hired to work at the home of a wealthy neighbor. Reference: As a child she had had no opportunity to study: she had worked as a servant for a wealthy neighbor.
The reference’s female participant becomes male. Much of the education and employment content survives, but the participant identity is not equivalent.
Polarity, modality and speech act. INJECT, sentence neg-033:
Source: Одйгезлэн кышкаме мукет бёсь кылек-тъ: Output: One person’s fear does not affect the others: Reference: Cowardice is contagious.
The proposition that cowardice spreads is reversed into a claim that one person’s fear does not affect others.
Omission. INJECT, sentence neg-005:
Source: Урамсь лек ныкышко куаpа кылъиськыз: Output: A woman’s voice was heard from the street: Reference: From outside came an angry woman’s voice:
The location and the woman’s voice remain, but the angry quality is omitted.
Unsupported addition. MCP, sentence neg-008:
Source: Печатай вожез потыса киоснынэ шона, шиег: Output: The old man, having come out of the river, shook his hands and hissed: Reference: Grandpa waved his hands furiously and hissed:
The output introduces a prior event in which the old man emerges from a river, which is absent from the reference. It also omits the explicitly furious manner.
Untranslated content. MCP, sentence pos-028:
Source: - Э! Кампет! - шумпотӥса черекя Федя. Output: - Hey! Kampet! - Fedya called out, rejoicing. Reference: - Candies! - Fedja squealed happily.
The common noun кампет ‘candies’, capitalised after the interjection, is kept as the name-like form Kampet. The joyful delivery does not compensate for the untranslated content word.
Tool calling across the roster
The manual evaluation covers one model. What did the other 30 models do when the tools were attached? That question concerns the behaviour of the models rather than the quality of their translations, and it can be answered by counting.
Call rates. Attaching a tool does not make a model use it. Whether to call is the model’s own decision, taken sentence by sentence. Table 4 gives the share of sentences on which each model emitted at least one tool call under the three MCP conditions. Across the study, 57.6% of the records in a condition with tools attached contain no call, and 14 of the 31 models never call at all. For these models the MCP condition is the ZEROSHOT condition with a longer system prompt. Model size does not predict the call rate. Gemma never calls at 1B, 4B, 12B or 27B parameters, Qwen3 calls at every size from 1.7B upwards, and Mistral-7B never calls while Ministral-8B always does. The behaviour is stable within a model family and ranges from 0% to 100% across families of the same size.
| Model | Size | Calls | Model | Size | Calls |
| Qwen3-0.6B | 0.6 | 0% | Mistral-7B-Instruct-v0.3 | 7.0 | 0% |
| gemma-3-1b | 1.0 | 0% | OLMo-2-1124-7B | 7.0 | 0% |
| xLAM-2-1b | 1.0 | 23% | Qwen2.5-7B | 7.0 | 3% |
| Llama-3.2-1B | 1.2 | 71% | Hermes-3-Llama-3.1-8B | 8.0 | 7% |
| Hammer2.1-1.5b | 1.5 | 0% | Llama-3.1-8B | 8.0 | 98% |
| Qwen2.5-1.5B | 1.5 | 50% | Ministral-8B | 8.0 | 100% |
| EuroLLM-1.7B | 1.7 | 0% | Qwen3-8B | 8.0 | 46% |
| Qwen3-1.7B | 1.7 | 97% | c4ai-command-r7b-12-2024 | 8.0 | 0% |
| granite-3.3-2b | 2.0 | 42% | granite-3.3-8b | 8.0 | <1% |
| Hammer2.1-3b | 3.0 | 0% | watt-tool-8B | 8.0 | 0% |
| Qwen2.5-3B | 3.0 | 58% | EuroLLM-9B | 9.0 | 0% |
| xLAM-2-3b | 3.0 | 100% | Mistral-Nemo | 12.0 | 100% |
| Llama-3.2-3B | 3.2 | 85% | gemma-3-12b | 12.0 | 0% |
| Phi-4-mini | 3.8 | 0% | gemma-3-27b | 27.0 | 0% |
| Qwen3-4B | 4.0 | 96% | Qwen3.8-27B | 27.8 | 100% |
| gemma-3-4b | 4.0 | 0% |
Table 4. Share of sentences on which each model emitted at least one tool call under MCP, MCP_COARSE and MCP_RICH. Fourteen models never call, and one more calls on fewer than one sentence in two hundred. Size is in billions of parameters.
Capability and willingness. A model that never calls may be unable to produce a usable call or may choose not to. Appendix § 11 separates the two cases with three tests: a direct instruction to call, an exhaustive search over the tool-call parsers of the serving framework, and the sweep itself. Nine of the 31 models fail all three tests, and two more emit call syntax that no parser reads; for these eleven models the call format is absent rather than unused. The remaining three silent models are able to call and do not. All our conditions are zero-shot, and a worked example of a tool call might change the behaviour of these three models, which we have not tested.
What the calls contain. Every call names the language to be processed by an ISO code, and most of these codes are wrong. Of the 315,910 codes sent during the study, 42.5% name the language actually being processed, 49.1% name no language that UralicNLP supports, and 8.4% name a supported but wrong language; 26,630 calls ask for koi, Komi-Permyak, on Komi-Zyrian text. As a consequence, 52.4% of all calls return an empty result. Appendix § 12 gives these figures per model. The injected prompt, by contrast, never has to name the language and delivers the same material for every sentence.
Function-calling specialists. Four models in the roster are fine-tuned for function calling. They call readily, but they answer in call syntax rather than in prose, so that often no translation can be found in their output. This is the case for 42.4% of the outputs of Hammer2.1-3b and 17.2% of the outputs of xLAM-2-1b-fc-r, against 4.3% for the models that are not specialised in this way.
Translation into the endangered languages. Table 9 in Appendix § 13 classifies every output in the study by its form. Out of the four languages, 98.1% of the zero-shot outputs are well-formed English. Into them, only 72.0% of the zero-shot outputs are well-formed text in the target language, and the share falls to 42.5% under MCP, with 22.9% of the outputs in the wrong language and 13.1% caught in a repetition loop under greedy decoding. The injected analyses keep the share at 67.1% and change nothing else. The rule-based resources supply lemmas, readings and glosses; what is missing in this direction is a model of the target language itself, which neither an analyser nor a dictionary can provide.
| Task | Condition | Well-formed | Copied | Wrong lang. | Degenerate | Empty |
| Into the endangered languages | no tools | 72.0 | 3.8 | 11.0 | 13.1 | 0.1 |
| Into the endangered languages | tool prompt | 50.1 | 7.8 | 24.0 | 9.5 | 8.5 |
| Into the endangered languages | inject | 67.1 | 5.9 | 10.4 | 15.8 | 0.8 |
| Into the endangered languages | inject+ | 66.3 | 6.8 | 3.9 | 23.0 | 0.0 |
| Into the endangered languages | mcp | 42.5 | 18.5 | 22.9 | 9.2 | 6.8 |
| Into the endangered languages | mcp coarse | 43.3 | 11.7 | 25.0 | 11.3 | 8.7 |
| Into the endangered languages | mcp rich | 38.9 | 10.4 | 23.0 | 14.3 | 13.5 |
| Out of the endangered languages | no tools | 98.1 | 0.3 | 0.1 | 0.6 | 1.0 |
| Out of the endangered languages | tool prompt | 89.3 | 1.5 | 0.1 | 1.1 | 8.1 |
| Out of the endangered languages | inject | 93.8 | 0.5 | 0.1 | 0.6 | 5.1 |
| Out of the endangered languages | inject+ | 96.2 | 0.0 | 0.0 | 0.4 | 3.4 |
| Out of the endangered languages | mcp | 90.3 | 1.3 | 0.4 | 1.2 | 6.9 |
| Out of the endangered languages | mcp coarse | 90.8 | 1.1 | 0.1 | 2.2 | 5.8 |
| Out of the endangered languages | mcp rich | 87.8 | 1.1 | 0.5 | 1.5 | 9.2 |
Table 9. Share of outputs (%) by failure mode, over every model and sentence in each group of directions. Well-formed is none of the other four, a test of form and not of adequacy: such an output can still be a wrong translation. Copied reproduces the input, wrong lang. is in the source language, degenerate loops or runs past four times the reference length, empty returned nothing. Rows are conditions; the five columns sum to 100.
Conclusions
Our main result is that rule-based analyses written into the prompt improve translation from Komi-Zyrian, Moksha and Erzya into English. On the model we evaluated manually, the number of accepted translations rises from 13 to 43 out of 272, and the remaining errors shift from wholesale replacement of the sentence to word-level substitutions. The gain depends on the bilingual dictionary and is absent for Udmurt, whose dictionary has no English side. The same analyses delivered as tools over MCP help the one model that calls the tools reliably and answers in prose, with 55 accepted translations, but we cannot recommend this route in general: 14 of the 31 models never call, 11 of them cannot, and half of the calls that are made name a language the toolkit does not know. Translation into the four languages fails under every condition, because the resources do not supply a model of the target language.
These conclusions rest on a narrow basis. The manual judgements cover one model, one annotator and 68 sentences from one text, and the comparison between the two delivery routes rests on that model alone. Whether the advantage of the tools over the injected analyses holds for other models that call reliably is a question we cannot answer with this data.
Limitations
The test set consists of 68 sentences from one literary text. The English reference is a free translation made independently of the Uralic ones, which is why we evaluate manually rather than against the reference. The manual judgements come from one annotator and cover one model, so they describe the best case in the roster rather than a typical member of it. Recovering the translation from each raw output relies on a language model; the step is checked by verbatim containment on every record, but not by a person.
The test set is balanced for sentiment polarity, with a gold label on every sentence, because a translation can match its reference closely and still invert the emotional tone of the source. Our evaluation does not use these labels. Measuring polarity preservation would require a sentiment classifier for the target language, and none exists for Komi-Zyrian, Moksha, Erzya or Udmurt; the alternative is annotation by an expert linguist, which is what § 5 does for one model and which does not scale to seven conditions. Until such a measure exists, our evaluation cannot distinguish a system that conveys the content of a sentence from one that also conveys its tone.
The four languages tokenise badly in every model of the roster, at 3.50 to 4.98 subword tokens per Komi-Zyrian word against 1.36 to 1.59 per English word (Table 10 in Appendix § 14). This does not affect the comparison between conditions, since the analyses reach the model through the same tokeniser in every condition, but it means that the same context window holds two to three times less text in an endangered language, which neither a better prompt nor a better tool can change.
| Tokenizer | eng | kpv | mdf | myv | udm |
| Qwen3-0.6B | 1.51 | 3.96 | 3.65 | 3.77 | 3.52 |
| Qwen3-1.7B | 1.51 | 3.96 | 3.65 | 3.77 | 3.52 |
| Qwen3-4B | 1.51 | 3.96 | 3.65 | 3.77 | 3.52 |
| Qwen3-8B | 1.51 | 3.96 | 3.65 | 3.77 | 3.52 |
| Llama-3.2-1B-Instruct | 1.51 | 4.24 | 3.44 | 3.62 | 3.48 |
| Llama-3.2-3B-Instruct | 1.51 | 4.24 | 3.44 | 3.62 | 3.48 |
| Llama-3.1-8B-Instruct | 1.51 | 4.24 | 3.44 | 3.62 | 3.48 |
| gemma-3-1b-it | 1.42 | 3.50 | 3.21 | 3.27 | 3.10 |
| gemma-3-4b-it | 1.42 | 3.50 | 3.21 | 3.27 | 3.10 |
| gemma-3-12b-it | 1.42 | 3.50 | 3.21 | 3.27 | 3.10 |
| Ministral-8B-Instruct-2410 | 1.58 | 4.13 | 3.33 | 3.37 | 3.33 |
| Phi-4-mini-instruct | 1.48 | 4.01 | 3.24 | 3.39 | 3.21 |
| OLMo-2-1124-7B-Instruct | 1.52 | 4.98 | 4.41 | 4.53 | 4.17 |
| EuroLLM-1.7B-Instruct | 1.36 | 4.09 | 3.16 | 3.37 | 3.36 |
| EuroLLM-9B-Instruct | 1.59 | 4.13 | 3.22 | 3.34 | 3.38 |
| Viking-7B | 1.55 | 4.67 | 4.26 | 4.34 | 3.91 |
Table 10. Subword pieces per word, measured on the parallel set itself, so the six columns describe the same 68 sentences in six languages. A value of 1.0 would mean one token per word; English costs 1.36 to 1.59 and the endangered languages 2.0 to 3.3 times that.
All models ran in bf16, whereas a local deployment typically runs 4-bit or 8-bit weights. Emitting a well-formed tool call is a format-sensitive behaviour that quantisation is known to disturb early ((Siniaev et al., 2026)), so the call rates reported here are an optimistic estimate of what a deployed system would show.
Appendix
Automatic scores, for reference
We computed chrF++ (Popović, [16]) and BLEU (Papineni et al., [14]), with sacreBLEU (Post, [17]),3 for every record, and chrF++ on text lemmatised with the same FSTs on both sides for the directions into the Uralic languages. We do not use these scores as evidence in the paper, for two reasons. First, the only reference is a free literary translation, so a score mainly rewards agreement with that translator’s wording. Second, trained metrics such as COMET (Rei et al., [19]) require a pretrained encoder for the target language, and none exists for these four languages. We report the scores here so that readers can compare with other work. Every score should be read against a floor, the same metric computed on the references paired with the wrong sentence: 11.5 chrF++ for English targets and 9.3 to 10.8 for the Uralic ones. Character overlap rewards any string in the right alphabet, and a score below the floor means that the output resembles the reference less than an unrelated sentence does.
Table 5 gives chrF++ per direction and condition, averaged over the ten models that ran every direction. Table 6 puts the scores beside the manual judgements on the twelve cells where both exist. On these cells the two agree at (Pearson) and (Spearman) for chrF++, and at 0.95 and 0.79 for BLEU. This is agreement at the level of whole conditions and says nothing about agreement on individual sentences.
| Direction | no tools | tool prompt | inject | mcp | mcp coarse |
| Komi-Zyrian→English | 19.8 | 15.6 (-4.3) | 25.3 (+5.5) | 18.5 (-1.3) | 18.9 (-1.0) |
| Moksha→English | 17.4 | 16.0 (-1.4) | 22.1 (+4.7) | 15.2 (-2.2) | 15.2 (-2.2) |
| Erzya→English | 15.1 | 11.9 (-3.2) | 20.0 (+4.9) | 14.5 (-0.6) | 14.5 (-0.6) |
| Udmurt→English | 16.3 | 14.0 (-2.2) | 16.4 (+0.2) | 14.8 (-1.4) | 15.6 (-0.7) |
| English→Komi-Zyrian | 4.4 | 3.3 (-1.1) | 3.7 (-0.7) | 4.5 (+0.1) | 2.6 (-1.8) |
| English→Moksha | 3.4 | 2.3 (-1.1) | 2.1 (-1.4) | 3.5 (+0.1) | 2.0 (-1.5) |
| English→Erzya | 4.0 | 3.3 (-0.7) | 2.9 (-1.0) | 4.4 (+0.5) | 2.7 (-1.2) |
| English→Udmurt | 5.4 | 2.9 (-2.4) | 4.0 (-1.3) | 3.1 (-2.3) | 3.7 (-1.7) |
Table 5. chrF++ averaged over the ten models that ran every direction, 68 sentences each. Parentheses give the change from the zero-shot baseline. The floor is 11.5 for English targets and 9.3 to 10.8 for the Uralic ones.
| Language | ZEROSHOT acc. | ZEROSHOT chrF | INJECT acc. | INJECT chrF | MCP acc. | MCP chrF |
| Komi-Zyrian | 8 | 29.9 | 17 | 37.8 | 23 | 40.5 |
| Moksha | 1 | 22.4 | 11 | 30.2 | 11 | 31.6 |
| Erzya | 0 | 20.2 | 11 | 29.7 | 11 | 30.9 |
| Udmurt | 4 | 24.5 | 4 | 26.2 | 10 | 29.7 |
Table 6. The twelve cells where both kinds of evidence exist: Qwen3.8-27B, four languages, three conditions. acc. is how many of the 68 outputs an expert linguist accepted; chrF is chrF++ on the same outputs. The two agree at (Pearson) and (Spearman).
Across the roster, INJECT raises chrF++ for every one of the ten models in Komi-Zyrian, Moksha and Erzya, by 1.2 to 8.4 points; under a paired bootstrap over sentences with 1,000 resamples (Koehn, [12]) the gain is significant in 25 of the 30 model-language cells, with no significant loss. For Udmurt the median change is −0.6. Under MCP the four models that call reliably diverge: out of Komi-Zyrian, Qwen3.8-27B gains 10.5 points, Llama3.1-8B loses 0.4, Mistral-Nemo-12B loses 3.2 and Qwen3-4B loses 11.8. Collapsing the six per-word tools into one sentence-level call (MCP_COARSE) changes little: 30 model-language cells improve, 29 get worse, and the median change is zero. The thirteen-tool surface (MCP_RICH, 16 models) raises the reliable callers by 7.0 points on average, on four cells. The tool-use system prompt alone (MCP_PROMPT) lowers chrF++ by 2.5 points out of Komi-Zyrian and by 1.6 into it.
Into the four languages, every model scores between 2.0 and 5.4 chrF++ against a floor of 9.3 to 10.8. Lemmatising both sides raises every cell by a few tenths of a point and raises the floor by as much, which indicates that the failure in this direction is not a matter of inflection. BLEU and chrF++ differ by an order of magnitude in level, 25.3 chrF++ against 5.7 BLEU out of Komi-Zyrian, and they order the conditions in the same way in every direction. Appendix G lists every model and direction.
Tool-calling probe
Table 7 records, for each model, whether it can emit a tool call that succeeds, and on what evidence. A model counts as capable if the positive control passes, or an exhaustive parser search finds a serving configuration under which it passes, or it made successful calls during the sweep. All three are needed: everything was served with vLLM, which wants the right tool-call parser per model, and the wrong one silently yields zero parsed calls from a model that is emitting them, while the control alone gives false negatives because whether a model calls depends on the prompt. Twenty of the 31 are capable, nine fail all three tests, and two emit call syntax that no parser in the search reads.
| Model | Size | Tool calling | Evidence |
| Qwen3-0.6B | 0.6 | works | probe |
| xLAM-2-1b-fc-r | 1.0 | works | probe |
| gemma-3-1b-it | 1.0 | none | probe, parser search, runs |
| Llama-3.2-1B-Instruct | 1.2 | works | 3899 calls in the runs |
| Hammer2.1-1.5b | 1.5 | unreadable | emits calls no parser reads |
| Qwen2.5-1.5B-Instruct | 1.5 | works | probe |
| Qwen3-1.7B | 1.7 | works | probe |
| EuroLLM-1.7B-Instruct | 1.7 | none | probe, parser search, runs |
| granite-3.3-2b-instruct | 2.0 | works | probe |
| Hammer2.1-3b | 3.0 | unreadable | emits calls no parser reads |
| Qwen2.5-3B-Instruct | 3.0 | works | probe |
| xLAM-2-3b-fc-r | 3.0 | works | probe |
| Llama-3.2-3B-Instruct | 3.2 | works | probe |
| Phi-4-mini-instruct | 3.8 | works | 137 calls in the runs |
| Qwen3-4B | 4.0 | works | probe |
| gemma-3-4b-it | 4.0 | none | probe, parser search, runs |
| Qwen2.5-7B-Instruct | 7.0 | works | probe |
| OLMo-2-1124-7B-Instruct | 7.0 | none | probe, parser search, runs |
| Mistral-7B-Instruct-v0.3 | 7.0 | works | probe |
| c4ai-command-r7b-12-2024 | 8.0 | none | probe, parser search, runs |
| Hermes-3-Llama-3.1-8B | 8.0 | works | probe |
| Qwen3-8B | 8.0 | works | probe |
| granite-3.3-8b-instruct | 8.0 | works | parser llama3_json |
| Llama-3.1-8B-Instruct | 8.0 | works | probe |
| Ministral-8B-Instruct-2410 | 8.0 | works | probe |
| watt-tool-8B | 8.0 | none | probe, parser search, runs |
| EuroLLM-9B-Instruct | 9.0 | none | probe, parser search, runs |
| gemma-3-12b-it | 12.0 | none | probe, parser search, runs |
| Mistral-Nemo-Instruct-2407 | 12.0 | works | probe |
| gemma-3-27b-it | 27.0 | none | probe, parser search, runs |
| Qwen3.8-27B | 27.8 | works | parser qwen3_xml |
Table 7. Whether each model can emit a tool call that succeeds. A model counts as capable if the positive control passes, or if an exhaustive parser search finds a parser under which it passes, or if it made successful calls during the sweep – the control alone gives false negatives, since whether a model calls a tool depends on the prompt.
Language codes sent to the tools
Table 8 breaks the figures in § 6 down by model. The share of codes naming something Uralic-NLP supports for the text in hand ranges from 11.5% to 74.5%, and no model sends a supported code more than three times in four. The invalid ones are not random: en is sent 74,449 times, and the attempts at Komi-Zyrian run through kv, kom, komi-zyrian, kz, kmz and kmm before any of them reaches kpv.
| Model | Codes | Correct | Unsupported |
| Llama-3.1-8B-Instruct | 53513 | 48.4% | 50.3% |
| Llama-3.2-3B-Instruct | 40906 | 35.0% | 36.4% |
| Qwen3-1.7B | 40682 | 41.9% | 33.7% |
| Ministral-8B-Instruct-2410 | 36292 | 38.9% | 61.1% |
| Llama-3.2-1B-Instruct | 30530 | 21.8% | 78.2% |
| Qwen3-8B | 30207 | 45.1% | 54.9% |
| Qwen3.8-27B | 22100 | 74.5% | 23.5% |
| Qwen3-4B | 20790 | 24.6% | 57.1% |
| Mistral-Nemo-Instruct-2407 | 18079 | 70.7% | 29.0% |
| xLAM-2-3b-fc-r | 7642 | 42.6% | 57.0% |
| Qwen2.5-3B-Instruct | 6241 | 38.9% | 60.7% |
| Qwen2.5-7B-Instruct | 5478 | 20.1% | 79.8% |
| granite-3.3-2b-instruct | 1797 | 45.0% | 55.0% |
| Qwen2.5-1.5B-Instruct | 1106 | 45.8% | 54.2% |
| Hermes-3-Llama-3.1-8B | 304 | 16.8% | 83.2% |
| Phi-4-mini-instruct | 156 | 11.5% | 87.8% |
| xLAM-2-1b-fc-r | 87 | 11.5% | 88.5% |
| All | 315910 | 42.5% | 49.1% |
Table 8. Language codes the models sent to the tools, counted per call. Correct is a code UralicNLP supports for the language actually being processed; unsupported means nothing the library knows. The two do not sum to 100: the remainder, 8.4% over the whole study, is a code the library supports for the wrong language, which returns an answer about a language nobody asked for.
Failure modes in full
Table 9 classifies every output by what went wrong with it, rather than by how far it scored from the reference. Copied reproduces the input instead of translating it; wrong lang. is decided by script where the scripts differ and by langdetect where they do not, believing only the claim that a string is in the source language; degenerate loops or runs past four times the reference length; empty returned nothing. A well-formed output is none of those, which is a test of form and not of adequacy: it can still be a wrong translation.
Tokeniser fertility
Table Table 10 gives subword pieces per word for each tokeniser in the roster, measured on the parallel set itself, so the six columns describe the same 68 sentences in six languages.
Extraction
Of 62,920 records, 40,783 raw outputs were already a single clean line under 400 characters with no markup and were scored unchanged; 4,072 were empty; the remaining 18,065 were read by a language model instructed to return the translation inside each one as a verbatim substring, stripping code fences, XML tags, JSON envelopes, lead-ins, trailing commentary, role markers, emphasis and enclosing quotes, while keeping dialogue dashes, in-sentence quotes and degenerate repetition. All 18,065 returned values passed verbatim containment against their source, so the extractor removed packaging and never wrote text of its own. Replacing all 4,541 empty strings with the whole raw output moves the headline gap by at most 0.5 chrF++, so nothing in the paper turns on how an answerless output is treated. The full instruction, the inputs, the returned values and the validation code are released with the data.
Every model, every direction
These tables list every cell behind the averages in Appendix § 10. Every cell the paper averages over is listed here: ten models, five directions each way, under the conditions that carry the argument.
| Model | zs | inj | mcp | zs | inj | mcp |
| kpv–eng | ||||||
| granite-3.3-8b-instruct | 18.8 | 22.7 | 17.1 | 1.6 | 1.9 | 1.6 |
| Hermes-3-Llama-3.1-8B | 23.0 | 27.8 | 22.5 | 5.2 | 6.2 | 4.9 |
| Llama-3.1-8B-Instruct | 21.7 | 29.5 | 21.2 | 3.7 | 9.5 | 3.8 |
| Mistral-7B-Instruct-v0.3 | 17.3 | 24.8 | 17.7 | 1.8 | 3.7 | 2.2 |
| Mistral-Nemo-Instruct-2407 | 16.9 | 22.5 | 13.7 | 2.2 | 3.5 | 1.6 |
| Qwen2.5-7B-Instruct | 19.2 | 27.9 | 17.9 | 3.4 | 6.9 | 2.6 |
| Qwen3-0.6B | 15.2 | 14.5 | 10.8 | 1.4 | 1.1 | 0.5 |
| Qwen3-4B | 17.4 | 19.9 | 5.7 | 1.6 | 2.1 | 0.1 |
| Qwen3-8B | 18.9 | 25.9 | 17.8 | 4.0 | 5.2 | 2.9 |
| Qwen3.8-27B | 29.9 | 37.8 | 40.5 | 10.2 | 16.3 | 19.6 |
| mdf–eng | ||||||
| granite-3.3-8b-instruct | 17.8 | 21.4 | 16.8 | 2.1 | 2.8 | 1.8 |
| Hermes-3-Llama-3.1-8B | 19.2 | 24.0 | 17.4 | 1.9 | 3.9 | 2.0 |
| Llama-3.1-8B-Instruct | 19.5 | 22.9 | 17.0 | 2.7 | 3.5 | 2.7 |
| Mistral-7B-Instruct-v0.3 | 17.6 | 21.4 | 16.7 | 1.4 | 2.1 | 1.6 |
| Mistral-Nemo-Instruct-2407 | 17.7 | 22.7 | 13.2 | 1.4 | 3.2 | 1.1 |
| Qwen2.5-7B-Instruct | 17.6 | 22.8 | 18.9 | 2.0 | 3.2 | 3.3 |
| Qwen3-0.6B | 5.9 | 13.3 | 4.5 | 0.2 | 2.0 | 0.1 |
| Qwen3-4B | 18.6 | 19.0 | 9.3 | 3.3 | 3.5 | 0.2 |
| Qwen3-8B | 18.1 | 23.2 | 6.7 | 3.2 | 4.3 | 0.3 |
| Qwen3.8-27B | 22.4 | 30.2 | 31.6 | 4.9 | 7.8 | 11.2 |
| myv–eng | ||||||
| granite-3.3-8b-instruct | 16.0 | 19.9 | 15.4 | 1.2 | 1.3 | 1.1 |
| Hermes-3-Llama-3.1-8B | 10.6 | 21.8 | 17.1 | 0.3 | 2.8 | 1.2 |
| Llama-3.1-8B-Instruct | 17.9 | 21.9 | 15.7 | 1.9 | 3.0 | 2.2 |
| Mistral-7B-Instruct-v0.3 | 14.7 | 20.1 | 13.9 | 1.1 | 1.4 | 1.3 |
| Mistral-Nemo-Instruct-2407 | 14.8 | 19.2 | 13.7 | 1.4 | 1.6 | 0.7 |
| Qwen2.5-7B-Instruct | 15.8 | 20.6 | 15.5 | 1.7 | 1.9 | 1.5 |
| Qwen3-0.6B | 9.3 | 9.1 | 4.2 | 0.2 | 0.3 | 0.1 |
| Qwen3-4B | 16.0 | 16.7 | 8.2 | 1.5 | 3.7 | 0.5 |
| Qwen3-8B | 15.9 | 21.3 | 10.5 | 1.5 | 3.1 | 0.3 |
| Qwen3.8-27B | 20.2 | 29.7 | 30.9 | 3.8 | 8.7 | 9.7 |
| udm–eng | ||||||
| granite-3.3-8b-instruct | 16.6 | 16.9 | 15.7 | 1.7 | 1.7 | 1.3 |
| Hermes-3-Llama-3.1-8B | 19.0 | 17.3 | 18.3 | 3.3 | 2.3 | 1.4 |
| Llama-3.1-8B-Instruct | 18.4 | 16.8 | 14.6 | 3.6 | 2.4 | 1.6 |
| Mistral-7B-Instruct-v0.3 | 15.5 | 14.4 | 15.6 | 1.4 | 1.0 | 1.9 |
| Mistral-Nemo-Instruct-2407 | 15.3 | 14.8 | 12.0 | 1.5 | 1.4 | 0.3 |
| Qwen2.5-7B-Instruct | 15.3 | 15.4 | 15.3 | 2.1 | 1.6 | 1.9 |
| Qwen3-0.6B | 6.0 | 12.7 | 5.8 | 0.2 | 1.3 | 0.2 |
| Qwen3-4B | 15.8 | 15.2 | 5.3 | 1.9 | 1.7 | 0.5 |
| Qwen3-8B | 16.2 | 14.8 | 15.8 | 2.5 | 1.4 | 2.0 |
| Qwen3.8-27B | 24.5 | 26.2 | 29.7 | 7.3 | 7.2 | 10.2 |
Table 11. Every model and every direction into English, scored on the model extraction. The first three columns are chrF++ and the last three BLEU. The chrF++ floor is 11.5.
| Model | zs | inj | mcp | crs | zs | inj | mcp | crs |
| eng–kpv | ||||||||
| granite-3.3-8b-instruct | 5.9 | 3.3 | 1.8 | 2.2 | 6.2 | 3.4 | 1.9 | 2.3 |
| Hermes-3-Llama-3.1-8B | 7.0 | 7.8 | 6.6 | 5.7 | 7.2 | 8.2 | 6.8 | 6.2 |
| Llama-3.1-8B-Instruct | 2.4 | 2.1 | 2.9 | 2.0 | 2.4 | 2.3 | 3.2 | 2.1 |
| Mistral-7B-Instruct-v0.3 | 3.0 | 3.6 | 2.9 | 1.5 | 3.2 | 3.5 | 3.1 | 1.6 |
| Mistral-Nemo-Instruct-2407 | 3.9 | 4.3 | 1.8 | 2.2 | 4.0 | 4.5 | 2.0 | 2.7 |
| Qwen2.5-7B-Instruct | 8.6 | 3.9 | 3.8 | 5.1 | 8.8 | 3.9 | 4.1 | 5.3 |
| Qwen3-0.6B | 0.4 | 1.8 | 0.5 | 0.1 | 0.4 | 2.2 | 0.5 | 0.1 |
| Qwen3-4B | 1.4 | 1.8 | 1.3 | 1.9 | 1.5 | 1.8 | 1.4 | 2.0 |
| Qwen3-8B | 2.4 | 1.7 | 0.5 | 0.3 | 2.4 | 1.7 | 0.6 | 0.3 |
| Qwen3.8-27B | 9.0 | 6.8 | 22.6 | 5.0 | 9.2 | 6.9 | 26.3 | 5.2 |
| eng–mdf | ||||||||
| granite-3.3-8b-instruct | 1.6 | 0.8 | 1.2 | 0.8 | 1.6 | 0.8 | 1.2 | 0.8 |
| Hermes-3-Llama-3.1-8B | 3.1 | 4.5 | 3.5 | 2.4 | 3.1 | 4.6 | 3.3 | 2.4 |
| Llama-3.1-8B-Instruct | 0.4 | 0.5 | 2.1 | 1.7 | 0.4 | 0.5 | 2.1 | 1.6 |
| Mistral-7B-Instruct-v0.3 | 1.3 | 0.7 | 2.5 | 1.4 | 1.3 | 0.8 | 2.6 | 1.4 |
| Mistral-Nemo-Instruct-2407 | 3.7 | 1.9 | 1.4 | 1.9 | 3.9 | 2.1 | 1.5 | 1.9 |
| Qwen2.5-7B-Instruct | 9.6 | 4.1 | 3.2 | 2.3 | 10.1 | 4.2 | 3.3 | 2.4 |
| Qwen3-0.6B | 1.4 | 1.6 | 2.5 | 2.5 | 1.4 | 1.6 | 2.6 | 2.6 |
| Qwen3-4B | 1.4 | 1.2 | 1.1 | 0.6 | 1.4 | 1.3 | 1.2 | 0.6 |
| Qwen3-8B | 1.8 | 0.7 | 0.5 | 0.4 | 1.9 | 0.7 | 0.5 | 0.4 |
| Qwen3.8-27B | 10.2 | 4.8 | 17.6 | 5.8 | 10.7 | 5.3 | 20.7 | 5.9 |
| eng–myv | ||||||||
| granite-3.3-8b-instruct | 1.6 | 1.6 | 1.4 | 1.7 | 1.5 | 1.5 | 1.4 | 1.6 |
| Hermes-3-Llama-3.1-8B | 5.8 | 5.3 | 3.0 | 3.3 | 5.7 | 5.3 | 3.1 | 3.2 |
| Llama-3.1-8B-Instruct | 1.7 | 1.6 | 5.0 | 2.0 | 1.6 | 1.6 | 4.8 | 2.0 |
| Mistral-7B-Instruct-v0.3 | 2.4 | 2.8 | 1.8 | 0.8 | 2.4 | 2.9 | 1.7 | 0.7 |
| Mistral-Nemo-Instruct-2407 | 5.5 | 2.5 | 2.5 | 1.4 | 5.6 | 2.4 | 2.7 | 1.5 |
| Qwen2.5-7B-Instruct | 9.1 | 3.7 | 6.5 | 8.5 | 9.5 | 3.6 | 7.7 | 8.8 |
| Qwen3-0.6B | 1.1 | 1.2 | 0.9 | 0.7 | 1.2 | 1.3 | 1.0 | 0.7 |
| Qwen3-4B | 1.9 | 2.8 | 1.1 | 2.0 | 1.9 | 2.9 | 1.1 | 2.0 |
| Qwen3-8B | 1.2 | 1.3 | 0.5 | 0.3 | 1.2 | 1.4 | 0.5 | 0.4 |
| Qwen3.8-27B | 9.3 | 6.4 | 21.7 | 6.5 | 9.5 | 6.7 | 26.0 | 6.8 |
| eng–udm | ||||||||
| granite-3.3-8b-instruct | 3.6 | 3.1 | 1.1 | 2.8 | 3.8 | 3.1 | 1.2 | 2.9 |
| Hermes-3-Llama-3.1-8B | 5.8 | 4.0 | 5.0 | 4.7 | 6.2 | 4.2 | 5.5 | 5.0 |
| Llama-3.1-8B-Instruct | 3.4 | 2.0 | 3.4 | 3.0 | 3.5 | 2.1 | 3.6 | 3.3 |
| Mistral-7B-Instruct-v0.3 | 2.5 | 3.9 | 2.1 | 1.1 | 2.5 | 4.1 | 2.1 | 1.1 |
| Mistral-Nemo-Instruct-2407 | 5.2 | 3.0 | 3.3 | 3.0 | 5.7 | 3.2 | 3.4 | 3.1 |
| Qwen2.5-7B-Instruct | 11.6 | 7.6 | 5.8 | 8.8 | 12.3 | 8.2 | 5.9 | 9.4 |
| Qwen3-0.6B | 0.1 | 0.1 | 0.4 | 0.3 | 0.1 | 0.1 | 0.4 | 0.3 |
| Qwen3-4B | 8.0 | 4.4 | 1.3 | 4.6 | 8.1 | 4.5 | 1.4 | 4.7 |
| Qwen3-8B | 1.5 | 1.3 | 0.7 | 0.7 | 1.6 | 1.4 | 0.7 | 0.7 |
| Qwen3.8-27B | 12.0 | 10.8 | 7.8 | 7.7 | 12.7 | 12.0 | 8.2 | 7.9 |
Table 12. Every model and every direction into a Uralic language, scored on the model extraction. The first four columns are chrF++ and the last four the same measure on lemmatised text. Floors are 9.3 to 10.8 chrF++ and 9.9 to 11.5 lemmatised.
References
- [1]Khalid Alnajjar, Mika Hämäläinen, and Jack Rueter. 2023a. Bootstrapping Moksha-Erzya neural machine translation from rule-based apertium. In Proceedings of the Joint 3rd International Conference on Natural Language Processing for Digital Humanities and 8th International Workshop on Computational Linguistics for Uralic Languages, pages 213–218, Tokyo, Japan. Association for Computational Linguistics.
- [2]Khalid Alnajjar, Mika Hämäläinen, and Jack Rueter. 2023b. Sentiment analysis using aligned word embeddings for Uralic languages. In Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023), pages 19–24, Tórshavn, the Faroe Islands. Association for Computational Linguistics.arxiv.org/abs/2305.15380
- [3]Anthropic. 2024. Model context protocol. https://modelcontextprotocol.io.
- [4]Iaroslav Chelombitko, Ekaterina Chelombitko, and Aleksey Komissarov. 2025. SampoNLP: A self-referential toolkit for morphological analysis of sub-word tokenizers. In Proceedings of the 10th International Workshop on Computational Linguistics for Uralic Languages, pages 57–67, Joensuu, Finland. Association for Computational Linguistics.
- [5]Iaroslav Chelombitko and Aleksey Komissarov. 2024. Specialized monolingual BPE tokenizers for Uralic languages representation in large language models. In Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages, pages 89–95, Helsinki, Finland. Association for Computational Linguistics.
- [6]David Dale. 2022. The first neural machine translation system for the Erzya language. In Proceedings of the First Workshop on NLP Applications to Field linguistics, pages 45–53, Gyeongju, Republic of Korea. International Conference on Computational Linguistics.arxiv.org/abs/2209.09368
- [7]Mika Hämäläinen. 2019. UralicNLP: An NLP library for Uralic languages. Journal of Open Source Software, 4(37):1345.DOI
- [8]Mika Hämäläinen. 2024. DAG: Dictionary-augmented generation for disambiguation of sentences in endangered Uralic languages using ChatGPT. In Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages.arxiv.org/abs/2411.01531
- [9]Mika Hämäläinen and Jack Rueter. 2025. UralicMCP: Turning LLMs into experts in endangered languages with MCP. In Lightning Proceedings of the 10th International Workshop on Computational Linguistics for Uralic Languages.
- [10]Mika Hämäläinen. 2021. Endangered Languages are not Low-Resourced!, page 1–11. University of Helsinki.
- [11]Ryan Johnson, Tommi A Pirinen, Tiina Puolakainen, Francis Tyers, Trond Trosterud, and Kevin Unhammer. 2017. North-Sámi to Finnish rule-based machine translation system. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 115–122, Gothenburg, Sweden. Association for Computational Linguistics.
- [12]Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain. Association for Computational Linguistics.
- [13]Sjur Nørstebø Moshagen, Flammie Pirinen, Lene Antonsen, Børre Gaup, Inga Mikkelsen, Trond Trosterud, Linda Wiechetek, and Katri Høivan-Aasikainen. 2023. The GiellaLT infrastructure: A multilingual infrastructure for rule-based NLP. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa).
- [14]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
- [15]Flammie A Pirinen and Linda Wiechetek. 2022. Building an extremely low resource language to high resource language machine translation system from scratch. In Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022), pages 150–155, Potsdam, Germany. KONVENS 2022 Organizers.
- [16]Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.DOI
- [17]Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.arxiv.org/abs/1804.08771
- [18]Taido Purason, Hele-Andra Kuulmets, and Mark Fishel. 2025. LLMs for extremely low-resource Finno-Ugric languages. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6692–6712, Albuquerque, New Mexico. Association for Computational Linguistics.
- [19]Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.arxiv.org/abs/2009.09025
- [20]Jack Rueter, Mika Hämäläinen, and Niko Partanen. 2020. Open-source morphology for endangered Mordvinic languages. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 94–100.DOI
- [21]Jack Rueter and Mika Hämäläinen. 2021. Prerequisites for shallow-transfer machine translation of mordvin languages: Language documentation with a purpose. Preprints.DOI
- [22]Jack Rueter, Niko Partanen, Mika Hämäläinen, and Trond Trosterud. 2021. Overview of open-source morphology development for the Komi-Zyrian language: Past and future. In Proceedings of the Seventh International Workshop on Computational Linguistics of Uralic Languages, pages 29–39, Syktyvkar, Russia (Online). Association for Computational Linguistics.DOI
- [23]Philip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics.
- [25]Yehor Tereshchenko, Mika Hämäläinen, and Svitlana Myroniuk. 2025. Evaluating OpenAI GPT models for translation of endangered UralicLanguages: A comparison of reasoning and non-reasoning architectures. In Proceedings of the 10th International Workshop on Computational Linguistics for Uralic Languages, pages 131–139, Joensuu, Finland. Association for Computational Linguistics.arxiv.org/abs/2512.16287
- [26]Nuo Xu and Ahrii Kim. 2026. Tokenization and morphological fidelity in Uralic NLP: A cross-lingual evaluation. In Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoReLM 2026), pages 493–503, Rabat, Morocco. Association for Computational Linguistics.DOI
- [27]Lisa Yankovskaya, Maali Tars, Andre Tättar, and Mark Fishel. 2023. Machine translation for low-resource Finno-Ugric languages. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 762–771, Tórshavn, Faroe Islands. University of Tartu Library.