betaFeedback

WorksComputer Science

MITRA-Swarm: Injecting Explicit Linguistic Structure into Foundation Models for Classical Machine Translation

Abstract

Classical Sanskrit poses three challenges unresolved by single-call monolithic models: morphophonemic fusion Sandhi, dense nominal compounding Samāsa, and discourse-level zero-anaphora forced by poetic meter. We present MITRA-Swarm, a multi-agent architecture injecting explicit linguistic structure into an otherwise monolithic pipeline: a Sandhi-splitting agent, a parallel array of generalist gemma-4-26B-A4B-it and domain-specialized gemma-2-mitra-it translators, and a discourse-aware Ellipsis Restorer that consults corpus-derived register tags to avoid inventing active-voice subjects in impersonal legal and philosophical registers. Two separated judges evaluate quality: a routing judge (JrouteJ_{\text{route}}, same model family as the generators) driving an internal critique loop, and an independent reporting judge (Jreport=J_{\text{report}}=gpt-4o-mini, no shared vendor lineage) scoring every output exactly once. On a stratified benchmark of 1,000 verses across five registers (23,990 scored verdicts, 12-condition pre-registered ablation), the full swarm reaches 85.9085.90 MQM against a monolithic baseline of 85.2785.27 (Δ=+0.63Δ=+0.63, padj=0.0018p_{\text{adj}}=0.0018, 95% CI [+0.30,+0.97][+0.30,+0.97]), and isolated discourse restoration alone accounts for +0.44+0.44 MQM (padj=0.0288p_{\text{adj}}=0.0288), outperforming a call-count-matched self-refinement baseline by +0.51+0.51 MQM (padj=0.0288p_{\text{adj}}=0.0288). Severe per-verse failures (MQM<80<80) fall from 9.0%9.0\% to 7.6%7.6\%, at roughly 9×9\times the per-verse inference cost of the baseline. Additionally, JrouteJ_{\text{route}} scores its own family's outputs 22-77 MQM higher than JreportJ_{\text{report}} on every one of the 12 conditions, empirically motivating the two-judge split; and a preflight screen of a 192,151-entry retrieval index against all 12 evaluation works verifies 0 reference leaks across 269 runtime checks. These are small gains on a metric saturating near 85-86.

Introduction

Sanskrit is the primary linguistic repository for a vast corpus of classical Indian philosophy, science, and literature, yet remains severely under-resourced in modern NLP: the lack of large parallel corpora and the language’s grammatical complexity have historically limited MT quality [8, 3]. Foundation models trained overwhelmingly on high-resource, concatenative languages do not, by default, carry the inductive biases classical Sanskrit requires, i.e. no mechanism for undoing phonetic fusion at word boundaries, no structured representation of deeply nested compounds, and no discourse memory built to restore an argument elided clauses earlier for metrical reasons, which in turn produces translations marred by omission, semantic drift, and hallucination [12].

Three linguistic properties make this failure systematic rather than incidental:

  • Sandhi (Morphophonemic Fusion): phonetic mergers at word boundaries obscure lexical boundaries without pre-translation segmentation, causing out-of-vocabulary and dictionary-lookup failures.

  • Samāsa (Nominal Compounding): compounds, sometimes spanning entire clauses, leave the syntactic and semantic relations between their constituents implicit.

  • High Ellipsis (Zero Anaphora): metrical verse systematically omits subjects, objects, even verbs, requiring inference from the surrounding discourse.

A monolithic model attempts segmentation, reordering, lexical translation, and discourse-level argument restoration in a single generation call; without explicit modular guidance it loses local morphosemantic structure, hallucinates relations between compound members, and invents target-language arguments rather than interrogating the source. MITRA-Swarm instead injects each linguistic principle as an explicit, separately-verifiable pipeline stage:

  1. A morphological Sandhi Parser decomposes phonetic mergers and compounds into a structured morphosemantic guide before translation.

  2. A parallel generator array of a generalist frontier model (gemma-4-26B-A4B-it) and a domain-specialized fine-tune (gemma-2-mitra-it) compares specialization against raw scale rather than assuming it.

  3. A discourse-level Ellipsis Restorer with register-aware voice guarding (§ 3.4) restores zero-anaphora referents in narrative registers while suppressing invented active-voice subjects in impersonal legal (Dharmasāstra) and philosophical (Sūtra) injunctions.

  4. A two-judge quality architecture: a routing judge JrouteJ_{\mathrm{route}} (shares a model family with the generators) drives an internal critique loop; an independent reporting judge JreportJ_{\mathrm{report}} (shares no vendor lineage with any generator or JrouteJ_{\mathrm{route}}) scores every finished output exactly once, off the critical path, for the numbers in this paper.

When JrouteJ_{\mathrm{route}}’s estimate falls below threshold, a Critique Agent diagnoses errors, a Tool Selector queries whatever retrieval tools are genuinely backed by data, and a Refinement Agent rewrites the translation.

Our contributions:

  • Architecture. Agents that each target one linguistic failure mode of monolithic models: fusion, compounding, and discourse ellipsis, rather than leaving all three to a single unguided generation call.

  • Empirical scale and significance. A pre-registered 12-condition ablation on N=1,000N = 1{,}000 verses (23,990 scored verdicts): full swarm beats the monolithic baseline by +0.63 MQM (padj=0.0018p_{\mathrm{adj}} = 0.0018) and a call-count-matched self-refinement baseline by +0.51 MQM (padj=0.0288p_{\mathrm{adj}} = 0.0288), via paired approximate randomization with Holm–Bonferroni correction and a measured noise floor rather than an uncorrected bootstrap-tail probability.

  • An isolated discourse-syntax finding. Ablating only the Ellipsis Restorer costs −0.44-0.44 MQM (padj=0.0288p_{\mathrm{adj}} = 0.0288), representing the single largest individually-significant component effect, and the clearest evidence that a linguistically-motivated discourse module, not scale or generic self-critique, is doing the work.

  • Evaluation and decontamination rigor. We directly quantify LLM-judge self-preference (JrouteJ_{\mathrm{route}} scores its own family’s outputs 2–7 MQM higher than JreportJ_{\mathrm{report}} on every condition, which is why only JreportJ_{\mathrm{report}} is reported) and screen a 192,151-entry retrieval index for reference leakage (0 leaks / 269 runtime checks).

Related Work

Sanskrit Computational Linguistics Prior work builds on foundational lexical-database and morphological-analysis resources [11], and more recently on neural sequence-to-sequence splitters [4] and transformer tokenizers such as TransLIST (Sandhan et al., 2022), integrated into tools like SanskritShala (Sandhan et al., 2023) and corpora such as the Digital Corpus of Sanskrit [9] (Hellwig 2021).

LLMs for Machine Translation General-purpose LLMs excel on high-resource languages [10] but struggle where parallel data is scarce [8]. Two-stage fine-tuning, monolingual pretraining followed by a small amount of high-quality parallel data (ALMA, Xu et al. (2024)), motivates our domain-aligned gemma-2-mitra-it backbone [1]; scaling efforts like NLLB-200 [3] expand coverage without addressing classical-language-specific needs like zero-anaphora resolution (Wang et al., 2023).

Multi-Agent Systems and Self-Refinement Multi-agent debate among independent LLM instances improves factuality and reasoning [5]; TransAgents (Wu et al., 2025) showed multi-role literary translation with specialized agents preferred by humans despite lower n-gram scores. Self-refinement [17] is directly implemented, call-count-matched, as our self_refine ablation. Tool-augmented reasoning (Yao et al., 2023; Schick et al., 2023) and retrieval augmentation [13] motivate § 3.6, where we report what happens when tool-use is restricted to tools actually backed by data.

Reference-Free Evaluation and Judge Reliability Reference-based metrics have documented validity limits (Reiter, 2018); reference-free QE (Rei et al., 2022) and fine-grained error-span detection [7] address this, as do LLM-as-judge frameworks [15 a] aided by chain-of-thought prompting (Wei et al., 2022). Such judges also exhibit documented biases like self-preference (Zheng et al., 2023): a judge sharing a model family with the system it evaluates is one instance, and § 5.2 measures it directly rather than assuming it away.

Methodology

The MITRA-Swarm pipeline decomposes translation into chunking, morphosemantic preprocessing, parallel drafting, discourse-aware argument restoration, automated quality routing, and agentic refinement, illustrated in Figure 1 and detailed with exact prompts in Appendix § 9.

System architecture of the MITRA-Swarm pipeline, showing source Sanskrit verse flowing through semantic chunking, Sandhi parsing, parallel translators, ellipsis restoration, and a Jroute critique loop with refinement and final English translation

Figure 1. System architecture of the MITRA-Swarm pipeline. JrouteJ_{\mathrm{route}} (same model family as the generators) drives the internal critique loop; JreportJ_{\mathrm{report}} (independent vendor, gpt-4o-mini) scores the finished output exactly once, off the critical path, and is the only judge whose scores are reported as results (§ 5.2).

Document Segmentation (Chunking)

To manage context-window limits and preserve local discourse cohesion [16], the source document is segmented at classical sentence/verse terminators (daṇḍa∣daṇḍa |, dvaṇḍa∣∣dvaṇḍa ||), a sliding token window (W=200W = 200, overlap O=40O = 40) for unpunctuated prose, or an LLM-driven semantic chunker for narrative boundaries.

Morphological Pre-processing (Sandhi Parser)

A Sandhi Parser Agent decomposes phonetic mergers and compounds (Samāsa; [4] (Sandhan et al. 2022)) into a structured morphosemantic guide, threaded through a typed TranslationRequest into every downstream translator call. gemma-4-26B-A4B-it accepts the guide via its chat template; gemma-2-mitra-it uses a legacy completion-style prompt for which guide injection was evaluated empirically and disabled, so the guide_only/guide_ellipsis contrasts reflect that per-model choice rather than a uniform assumption.

Parallel Translation Array

The chunk is translated concurrently by a parallel array of models,

T={tspecialized,tfrontier,…},(1)\mathcal{T} = \{t_{\mathrm{specialized}}, t_{\mathrm{frontier}}, \ldots\}, \tag*{(1)}

comprising the domain-specialized gemma-2-mitra-it ([1]) and the generalist frontier gemma-4-26B-A4B-it ([6]), both served locally via vLLM. Each model generates a draft translation independently, and the routing judge selects among candidates.

Ellipsis and Zero-Anaphora Argument Restoration with Register-Aware Voice Guarding

The Ellipsis Restorer operates over a sliding discourse window of the preceding/succeeding chunks, Cdiscourse={ci−k,…,ci}C_{\mathrm{discourse}} = \{c_{i-k}, \ldots, c_i\}, identifying zero-anaphora and interpolating the implied argument. Naively restoring an elided subject risks inventing a third-person actor in passages that are impersonal or injunctive by design, where restoration could convert a Dharmaśāstra injunction into an active clause with a hallucinated subject. MITRA-Swarm mitigates this with a voice-preservation guard: every source work ID is mapped, via a corpus-derived, manually verified register table (registers.json; six registers (vedic, epic, legal, sutra, kavya, shastra), covering over 100% of the corpus’s 52 work IDs, rather than inferred from a keyword heuristic), to a voice_preserving flag. Legal and sutra are flagged voice-preserving as characteristically impersonal/passive by genre; the Ellipsis Restorer receives the literal genre hint “legal” for any work so flagged, and the raw register name otherwise, which measurably changes its restoration behavior (Section A) without a separate model or prompt per register.

Two-Judge Quality Architecture

Quality estimation is split into two roles, enforced at startup by an automated model-family check. JrouteJ_{\mathrm{route}} (Routing Judge): a locally-hosted gemma-4-26B-A4B-it instance, sharing a model family with the generators, scores intermediate drafts Sj∈[0,100]S_j \in[0, 100] against threshold Tjudge=80T_{\mathrm{judge}} = 80 to decide whether the critique loop fires; because it shares a family with what it evaluates and is queried repeatedly during generation, its scores are an internal control signal only, never reported as a result. JreportJ_{\mathrm{report}} (Reporting Judge): an independent offline judge (openai: gpt-4o-mini), sharing no vendor lineage with either generator or JrouteJ_{\mathrm{route}}, invoked exactly once per finished output on a separate machine from the generation cluster. Every number in Section 5 is a JreportJ_{\mathrm{report}} score.

Both judges use a GEMBA-style rubric returning structured error spans (category, severity, verbatim span, explanation) rather than a bare number. The headline MQM is recomputed deterministically from that list using fixed severity weights (critical =−25=-25, major =−5=-5, minor =−1=-1, floored at 00), not the model’s own top-line arithmetic, which is retained separately for the judge-validation diagnostic (§ 5.2). A failed judge call yields a missing score, excluded from all aggregates, rather than a sentinel 0.00.0 that would otherwise depress the mean and minimum and spuriously trigger the critique loop.

Agentic Feedback and Tool Retrieval

If Sj<TjudgeS_j < T_{\mathrm{judge}}, a Critique Agent diagnoses errors, a Tool Selector queries whichever tools are genuinely backed by data, and a Refinement Agent synthesizes the draft, critique, and retrieved context into a revised translation, re-scored up to MM iterations. The inventory is reported as offered: a tool with no backing index is absent rather than present and silently returning a canned string. On the air-gapped GH200 cluster used here, that inventory reduces to TRANSLATION_MEMORY, which performs fuzzy character-6-gram matching against a decontaminated corpus index (§ 4.2). A DICTIONARY tool (offline lexicon lookup, with entries OCR-sanitized to strip 19th-century scanning artifacts before reaching the Refinement Agent) exists but was not backed by a snapshot in this run; ALIGNMENT and live SEARCH require, respectively, an alignment index and network access unavailable offline. A Dedup Agent stitches overlapping segments pairwise, tmerged=Stitch⁡(ti−1,ti)t_{\mathrm{merged}} = \operatorname{Stitch}(t_{i-1}, t_i), feeding only the tail of the running translation (last L=600L = 600 characters) so per-step input stays bounded rather than growing with the number of merged segments.

Experimental Setup

Evaluation Dataset and Stratification

Experiments are conducted on a stratified sample of N=1,000N = 1{,}000 verses drawn, proportionally by register, from the 5,552-verse test partition (12 works) of the MitraSamgraha corpus ([2]). Table 1 reports the realised sample against the full test-partition population for each of the five registers represented in this partition; the register-to-work-ID mapping is manually verified (100% work-ID coverage, 0 unmapped) rather than inferred from a keyword heuristic.

RegisterDescriptionPop.SampleVoice-preserv.
epicItihāsa/Purāṇa narrative1,720310–
sutraPhilosophical sūtra, commentary1,488268✓
vedicVedic saṃhitā/upaniṣad; archaic1,000180–
kavyaOrnate metrical poetry844152–
legalDharmaśāstra, nibandha50090✓
Total5,5521,000

Table 1. Register composition of the N=1,000N = 1{,}000 evaluation sample, proportional to its population in the 5,552-verse test partition. Voice-preserving marks registers routed to the impersonal-voice guard in the Ellipsis Restorer (§ 3.4).

This sample size balances statistical power with the cost of a 12-condition ablation (12,000 full pipeline runs). Because the corpus is drawn from 12 works with substantial internal stylistic variation, register-level results are an average over that variation, not a claim of uniformity within it.

Data Decontamination and Preflight Reference-Echo Screening

To ensure rigorous evaluation and prevent data leakage, the TRANSLATION_MEMORY index contains 192,151 segment-level entries, with all 12 evaluation work IDs excluded at build time (125,665 entries excluded; 0 evaluation work IDs reachable in the live index). A runtime reference-echo detector computes character 6-gram F-score similarity between every retrieved string and the gold reference that triggered it, flagging anything ≥0.6\ge0.6; across a 200-item preflight check and 269 runtime retrievals, the maximum observed similarity is 0.375, with 0 echoes flagged. We separately audit translator outputs against the gold reference for suspected training-data memorization (independent of retrieval): 0.21% of gemma and 0.70% of mitra outputs exceed a 0.6 "suspect" threshold, with 0.15% of mitra outputs exceeding a stricter 0.8 threshold; inspection shows short sentences (e.g., "He makes offering(s) to four deities") with only a one-word paraphrase gap, not retrieval-time leakage.

Models and Hardware

The generator array comprises gemma-4-26B-A4B-it (26B-parameter MoE, 4B active) and gemma-2-mitra-it, both served locally via vLLM on NVIDIA GH200 nodes; JrouteJ_{\mathrm{route}} runs on the same endpoint. JreportJ_{\mathrm{report}} runs offline, after generation, against the OpenAI API (gpt-4o-mini), since the generation cluster is air-gapped.

Hyperparameters and Compute Budget

Translation agents decode at temperature 0.0; critique/judge/tool-selector agents at 0.1; serving context window is capped at 4,096 tokens (with a 2,048-token output cap) to optimize batch throughput across verse segments, well within the models’ native context windows. Table 2 compares per-verse token budget and latency for the three conditions central to the main results (full per-condition table in Appendix Table 10), measured directly from run instrumentation, not estimated. Across the full run, generation consumed 17.61M tokens (10.71M prompt, 6.90M completion) across 85,509 LLM calls at a 54.9% cache hit rate; the routing judge alone accounts for 17,713 calls (12.57M tokens), since JrouteJ_{\mathrm{route}} is queried once per candidate per stage rather than once per verse.

ConfigurationPrompt tok.Compl. tok.Total/verseLatency (s)
baseline91491402.5
self_refine164892534.9
bo_n45525871312.8
full_swarm82846212907.1

Table 2. Per-verse compute, key conditions (measured, not estimated). Full swarm costs 9.2× baseline tokens for +0.63 MQM.

Ablation Matrix

We pre-register a 12-condition ablation matrix (full active-component flags in Appendix Table 8), isolating each architectural component against a monolithic baseline, a paired test-retest control, a placebo re-judging control, and two compute-matched controls (best-of-N and generic self-refinement): baseline (monolithic single-pass, no guide/ellipsis/loop); baseline_dup (different seed; the noise floor); placebo_rejudge (baseline re-judged; judge-side noise only); bo_n (compute-matched best-of-5, temperature 0.7); self_refine (call-count-matched generic self-critique, no role specialization); guide_only (Sandhi guide alone); guide_ellipsis (guide + Ellipsis Restorer, no loop/tools); no_guide, no_ellipsis (the isolated discourse-restoration ablation), no_critique, no_tools (full swarm minus one component each); and full_swarm, the complete architecture (Section 3.4–3.6).

Metrics and Statistical Plan

Quality is MQM, a reference-free [0, 100] score from JreportJ_{\mathrm{report}} (§ 3.5). Significance uses a paired approximate randomization test (the preferred test for MT system comparison rather than a bootstrap-tail probability; Riezler and Maxwell, 2005) at R=10,000R = 10{,}000, with 95% BCa bootstrap CIs (B=10,000B = 10{,}000); all 10 pre-registered contrasts are corrected jointly with Holm–Bonferroni. Because mean MQM is compressed near 85–86 for every condition, we also report a measured noise floor: the mean absolute difference between baseline and baseline_dup is 0.21 MQM, and a contrast is reportable only if it is both significant and exceeds this floor; a significant result smaller than the chance disagreement between two identical systems is a measurement of the harness, not the method.

Results and Analysis

Main Ablation Results

Table 3 reports the full 12-condition matrix; Table 4 reports the 10 pre-registered contrasts derived from it.

ConfigurationNNMQMSDP10P25Min%<80
baseline100085.275.6780.084.0359.0
baseline_dup100085.275.5380.084.0358.5
placebo_rejudge100085.265.7180.084.0358.9
bo_n100085.585.5180.084.0408.7
self_refine100085.395.5680.084.0358.3
guide_only100085.235.6580.084.0408.9
guide_ellipsis99985.915.0380.084.0607.4
no_guide99985.915.2080.084.0607.1
no_ellipsis100085.475.2280.084.0558.3
no_critique99985.915.0380.084.0607.4
no_tools99985.895.2980.084.0407.6
full_swarm99985.905.2980.084.0407.6

Table 3. Ablation results on the N=1,000N = 1{,}000 stratified MitraSamgraha benchmark. MQM from JreportJ_{\mathrm{report}} (gpt-4o-mini); P10/P25 are the 10th/25th percentiles; %< 80 is the severe-failure rate.

ContrastNNΔ\DeltaMQM95% CIpadjp_{\mathrm{adj}}dzd_zW/L/T
Full swarm vs monolithic baseline999+0.63*[+0.30, +0.97]0.002+0.12397/294/308
Re-judging alone (placebo)1000-0.01†[-0.07, +0.01]1.000-0.022/2/996
Full swarm vs best-of-N (compute matched)999+0.32[-0.05, +0.67]0.460+0.06363/333/303
Full swarm vs self-refine (specialisation)999+0.51*[+0.16, +0.85]0.029+0.09389/301/309
Morphosemantic guide999-0.01†[-0.27, +0.23]1.000-0.00170/179/650
Ellipsis restoration999+0.44*[+0.12, +0.73]0.029+0.09323/232/444
Tool retrieval999+0.01†[+0.00, +0.03]1.000+0.042/0/997
Critique loop (isolated)999-0.02†[-0.14, +0.09]1.000-0.0124/27/948
Baseline -> guide1000-0.04†[-0.38, +0.27]1.000-0.01252/256/492
Guide -> guide + ellipsis999+0.69*[+0.39, +1.00]0.001+0.14324/221/454

Table 4. Pre-registered contrasts, paired approximate randomization test, R=10,000R = 10{,}000, Holm–Bonferroni adjusted over the family of 10. * significant and exceeds the 0.21 MQM noise floor. † below the noise floor (not distinguishable from run-to-run variation regardless of nominal pp-value). * significant after Holm–Bonferroni over 10 pre-registered contrasts. † below the measured noise floor of 0.21 MQM (the difference between two identical pipelines under different decoding seeds) and therefore not distinguishable from run-to-run variation.

Four trends are reportable under the criterion in § 4.6 (significant and above the noise floor).

(1) The full swarm beats the monolithic baseline by +0.63 MQM (padj=0.0018p_{\mathrm{adj}} = 0.0018, CI [+0.30, +0.97], dz=0.12d_z = 0.12, 397/294/308 W/L/T), which is real but small in absolute terms (both means sit at 85.3–85.9 of 100), which we report plainly. (2) Removing only the Ellipsis Restorer costs −0.44 MQM relative to the full swarm (padj=0.0288p_{\mathrm{adj}} = 0.0288, [+0.12, +0.73], 323/232/444 W/L/T), constituting the single largest individually-significant component effect, confirming that discourse restoration operates effectively in isolation. (3) The swarm beats a call-count-matched generic self-refinement baseline by +0.51 MQM (padj=0.0288p_{\mathrm{adj}} = 0.0288, 389/301/309 W/L/T), attributable to role specialization and discourse/morphology structure rather than extra inference calls alone. (4) Adding the Ellipsis Restorer on top of the Sandhi guide alone yields +0.69 MQM (padj=0.001p_{\mathrm{adj}} = 0.001, [+0.39, +1.00], dz=0.14d_z = 0.14), the largest effect size in the matrix, consistent with (2).

The other six contrasts are not reportable, and a null from a well-controlled comparison is itself informative: the Sandhi guide alone, against the full swarm (−0.01) or the baseline (−0.04, guide_only); the critique loop in isolation (−0.02); tool retrieval (+0.01; § 3.6 explains why, given the tool inventory actually available); the placebo re-judging control (−0.01, confirming judge stability under repeat scoring); and full swarm vs. best-of-NN (+0.32, padj=0.460p_{\mathrm{adj}} = 0.460, numerically positive and above the floor but not significant, though a large enough compute-matched sampling budget may close some of this gap).

These findings replicate under a fully deterministic, reference-based metric. To check they do not depend on a single judge’s preferences, we additionally compute chrF++ (Popović, 2017) via sacrebleu (word order 2) over the same selected outputs. The full swarm’s advantage over baseline holds under this judge-independent metric (29.27 → 30.33 chrF++, padj=0.0010p_{\mathrm{adj}} = 0.0010), as does isolated ellipsis restoration (+0.73, padj=0.0010p_{\mathrm{adj}} = 0.0010) and the gain over self-refine (+0.78, padj=0.0010p_{\mathrm{adj}} = 0.0010); the same noise-floor check that gates the MQM contrasts flags marginal seed-to-seed chrF variance on its own (p=0.033p = 0.033), smaller than the reported effects but disclosed rather than omitted.

Restricting to baseline, the isolated ellipsis ablation, and the full swarm (full 12-condition breakdown in Appendix Table 9), the full-swarm gain over baseline is largest in vedic (83.6 → 84.5, +0.9; heaviest ellipsis, most archaic morphology), moderate in kavya (+0.4), legal, and sutra (+0.3 each), and flat in epic (+0.0). This pattern is consistent with, though not definitive proof of, the discourse-restoration mechanism in § 3.4: the voice-preserving registers (legal, sutra) do not show the largest numeric gains, since the voice guard changes how arguments are restored (Appendix § 7) rather than guaranteeing the largest score delta.

Error-category counts (mean per segment by MQM category, Appendix Table 11) are dominated by mistranslation and omission in every condition; the full swarm shows a modest omission reduction relative to no_ellipsis/baseline, alongside a small rise in flagged mistranslation spans, since the critique loop rewrites text and not every rewrite is judged an improvement. The tail of the per-verse score distribution (Appendix Figure 2) confirms the system’s effect is concentrated in the low tail (P10, minimum, %< 80 in Table 3) rather than a uniform shift of the whole distribution, since the mean is compressed near 85–86 for every condition.

Judge Validation: Self-Preference and Score-Arithmetic Reliability

Table 5 summarizes the evaluation-methodology findings motivating the two-judge split (full per-condition self-preference in Appendix Table 12). Self-preference is large and universal: on every one of the 12 conditions, JrouteJ_{\mathrm{route}} scores the output 1.9–6.8 MQM higher than JreportJ_{\mathrm{report}}. Reporting JrouteJ_{\mathrm{route}}’s own scores, as a single-judge design would, would inflate every headline number by an amount that itself varies by condition, biasing cross-condition comparison; to our knowledge, this is one of the more direct in-pipeline measurements of LLM-judge self-preference reported for a low-resource MT system. The judge’s own arithmetic disagrees with its stated rubric on 96.4% of scored items (mean ∣Δ∣=16.37|\Delta| = 16.37 between the self-reported score and the score recomputed from its own error spans), which is why we report the recomputed score as the metric and both scores as a diagnostic. Recomputing MQM is a deliberate departure from the published GEMBA-MQM protocol, so the two are not numerically interchangeable with prior work using the standard prompt. Test–retest reliability has a within-item SD of 1.45 MQM, smaller than the smallest reportable effect in Table 4; the reporting judge’s failure rate is 0.41%.

Judge diagnosticValue
Reporting judgeopenai:gpt-4o-mini
Judge failure rate0.41%
Arithmetic disagrees with rubric96.4% (mean ∣Δ∣=16.37|\Delta| = 16.37)
Test–retest SD (repeated calls)1.45
Self-preference range (min–max, 12 conditions)+1.94 to +6.79

Table 5. Judge validation summary under the zero-shot judge prompt.

Few-shot prompting reduces, but does not eliminate, this arithmetic gap. Motivated by the concern that a zero-shot judge may simply be miscalibrated, we re-score a stratified subsample, the same 100 items per condition, paired across all twelve, with a three-shot variant of the judge prompt. This reduces the arithmetic-disagreement rate from 96.4% to 79.3% (mean ∣Δ∣|\Delta|: 16.37 → 8.46) but leaves most items still disagreeing, reinforcing that recomputing the score from error spans is not an optional refinement. Zero-shot and few-shot scores on the same items correlate only moderately (Pearson r=0.47r = 0.47, Spearman ρ=0.60\rho= 0.60), indicating the judge’s output is itself non-trivially prompt-sensitive. At n=100n = 100 per condition the pre-registered contrasts do not reach significance here (n≈146n \approx146 is needed to detect a 2-point effect), so we treat this as a calibration diagnostic, not a second significance test.

Table 5: Judge validation summary under the zero-shot judge prompt (the few-shot comparison above is a separate, n=100n = 100-per-condition subsample). Self-preference range is Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}} over the 12 conditions.

Tool Utility

Table 6 reports usage for the TRANSLATION_MEMORY tool: 5,513 queries, 4.9% hit rate, zero transport errors. Its measured effect (+0.01 MQM, Table 4) is below the noise floor, representing a genuine negative result rather than an artifact: fuzzy translation-memory lookup rarely finds a close precedent for an unseen verse, and when it does, the 26B-parameter generalist model already appears to carry comparable lexical knowledge parametrically.

ToolCallsHit rateErrorsLatency (ms)
TRANSLATION_MEMORY55130.050513

Table 6. Tool utility, aggregated over all conditions in which the tool inventory was active.

Conclusion

We introduced MITRA-Swarm, a multi-agent architecture injecting explicit linguistic structure (Sandhi splitting, register-aware discourse ellipsis restoration, and a two-judge quality architecture) into an otherwise monolithic foundation-model pipeline for classical Sanskrit. On a pre-registered, stratified N=1,000N = 1{,}000 benchmark, the swarm produces small but statistically robust gains over a monolithic baseline and a call-count-matched self-refinement baseline, with the isolated discourse-restoration ablation as the single largest component effect. Equally central is what the benchmark says about evaluation itself: LLM-judge self-preference is large and universal (2–7 MQM, every condition), and a preflight decontamination sweep is directly falsifiable and, here, clean. A foundation-model system for a morphologically rich, low-resource classical language should be evaluated with rigorous self-scrutiny; we release the pipeline, matrix, and diagnostics so that findings can be independently reproduced. Future work: gold linguistic annotation to measure each agent against the phenomenon it targets rather than only aggregate MQM; extension to Classical Tibetan and Pāli; and a multi-judge reporting panel to bound the single-judge design’s residual uncertainty.

Limitations

  • Effects are small relative to the metric’s range. Every condition scores 85.2–85.9 of 100; the effect concentrates in the low tail, not a uniform mean shift, and should not be extrapolated to a metric with more headroom.

  • A single reporting judge is a single point of failure. Despite the validation in § 5.2, the headline metric rests on one model; a multi-judge panel (future work) would strengthen this.

  • The noise floor bounds generation variance, not judge variance, since baseline/baseline_dup differ only in seed and are judged at temperature 0 with caching.

  • Recomputing MQM from error spans departs from the published GEMBA-MQM protocol (96.4% disagreement with the model’s own arithmetic, § 5.2), so it is not numerically comparable to prior work reporting the raw score.

  • bo_n decodes at temperature 0.7 vs. 0.0 for every other condition: this is inherent to best-of-NN, but a genuine confound.

  • Single-prompt reasoning baselines: While self_refine and bo_n provide compute-matched controls, exploring unified single-prompt Chain-of-Thought reasoning (prompting a single frontier model to perform Sandhi analysis and discourse tracking in explicit scratchpad tokens before emitting a translation) represents a valuable direction for future baseline comparisons.

  • Ensemble aggregation vs. routing: The parallel drafting array selects the top candidate via JrouteJ_{\mathrm{route}}; evaluating multi-candidate synthesis/aggregation models rather than tournament routing remains an open architectural alternative.

Ethical considerations. Multi-agent inference raises compute cost per query relative to single-pass models (Table 2), acceptable for offline research but warranting caching and hardware-efficiency care before deployment; translating culturally and religiously loaded terminology (Dharma, Śūnya) risks interpretive drift, which the voice guard and independent evaluation aim to mitigate. We credit and respect the licenses of the open assets used: MitraSamgraha (CC-BY-SA 4.0; [2]) and the Gemma model family (Gemma Open License; [6]).

Qualitative Analysis: Register-Aware Voice Preservation

Table 7 shows an unedited pipeline output pair for a legal (Manusmṛti, voice-preserving) verse. Both conditions produce fluent, accurate English; the full swarm’s Ellipsis Restorer, informed that this work is voice-preserving, rewrites the target clause as an impersonal passive ("the mind should be turned away...") rather than an invented active third-person subject ("one should turn the mind away..."), matching the impersonal register rather than converting it to a personal directive. We report this as a mechanism demonstration, not a large per-item MQM swing: per-item routing-judge scores are a diagnostic signal, not a reported metric (§ 3.5); the statistically assessed effect of this mechanism is the register-stratified pattern (§ 5) and the isolated ablation (§ 5, trend 2), not any single verse.

Source*indriyārtheṣu sarveṣu na prasajyeta kāmataḥ / atiprasaktim cait eṣām manasā saṃnivartayet //
Gold Reference“...let him carefully obviate an excessive attachment to them, by (reflecting on their worthlessness in) his heart.”
no_ellipsis (guard inactive)“...furthermore, one should turn the mind away from excessive attachment to these.”
full_swarm (legal → passive)“...furthermore, the mind should be turned away from excessive attachment to these.”

Table 7. Register-aware voice preservation, an unedited output pair (Manusmṛti 2.98, item GSD36manu_test:130).

Supplementary Tables

Table 8: Ablation matrix: active components per configuration. Guide = Sandhi Parser; Ellip. = Ellipsis Restorer; Crit. = critique-refinement loop; Tools = tool-selector inventory; TM = translation-memory prefetch; QA = automated QA audit pass. Referenced from § 4.5.

ConfigurationGuideEllip.Crit.ToolsTMQANotes
baseline–––––––
baseline_dup––––––seed +1
placebo_rejudge––––––re-judge only
bo_n––––––N=5N = 5
self_refine––✓––✓unspecialised
guide_only✓––––––
guide_ellipsis✓✓–––––
no_guide–✓✓✓✓✓–
no_ellipsis✓–✓✓✓✓–
no_critique✓✓––✓✓–
no_tools✓✓✓–✓✓–
full_swarm✓✓✓✓✓✓–

Table 8. Ablation matrix: active components per configuration.

Table 9: Full 12-condition register breakdown, MQM by register and configuration (JreportJ_{\mathrm{report}}); the condensed discussion in § 5 is derived from this.

Registerbaselinebaseline_dupbo_nfull_swarmguide_ellipsisguide_onlyno_critiqueno_ellipsisno_guideno_toolsplacebo_rejudgeself_refine
epic86.085.885.886.085.885.985.886.086.486.086.085.9
kavya85.485.585.385.885.785.485.785.485.785.885.485.4
legal85.184.785.085.485.185.085.184.885.285.485.184.6
sutra85.785.885.986.086.085.586.085.886.286.085.786.1
vedic83.683.983.784.584.683.484.783.584.484.683.683.7

Table 9. Full 12-condition register breakdown, MQM by register and configuration (JreportJ_{\mathrm{report}}).

ConfigurationPrompt tok.Compl. tok.Total/verseLatency (s)
baseline91491402.5
baseline_dup91481402.4
bo_n45525871312.8
full_swarm82846212907.1
guide_ellipsis73843011695.1
guide_only4823898712.5
no_critique73843011695.1
no_ellipsis72647211987.3
no_guide4151145296.8
no_tools79745812556.8
placebo_rejudge91491402.5
self_refine164892534.9

Table 10. Full per-verse token budget and latency by configuration (measured); Table 2 in the main text shows the three key conditions. Referenced from § 4.4.

Configurationadditionmistranslationomissionuntranslatedawkwardgrammar
baseline0.071.621.350.020.020.13
baseline_dup0.071.621.360.020.020.13
bo_n0.071.631.370.010.020.13
full_swarm0.061.681.310.010.020.13
guide_ellipsis0.061.691.310.010.020.13
guide_only0.061.661.320.020.020.15
no_critique0.071.691.310.010.020.13
no_ellipsis0.061.701.330.010.020.14
no_guide0.071.661.340.010.020.11
no_tools0.061.681.310.010.020.13
placebo_rejudge0.071.621.350.020.020.13
self_refine0.071.681.350.010.020.12

Table 11. Errors per segment by MQM category, JreportJ_{\mathrm{report}}. Referenced from § 5.

Errors per segment, by MQM category, as judged by the independent reporting judge.

Agent System Prompts and Prompt Templates

In this appendix, we provide the exact system instructions, prompt templates, and JSON schemas used by the various subagents in MITRA-Swarm to ensure full reproducibility of the agentic behaviors.

Morphological Pre-processing (Sandhi Parser Agent)

The Sandhi Parser Agent is responsible for decomposing complex phonetic mergers and nominal compounds before translation begins. Its output is threaded into every downstream translator call via a typed TranslationRequest.guide field (§ 3.2).

System Instruction:

You are an expert Sanskrit grammarian.
Analyze the input Sanskrit text. Identify
complex compound words (Samasa) and
phonetic joins (Sandhi). Write down the
compound word splits and explain the
individual components briefly to assist a
translator. Format your output as a concise
split analysis. If there are no complex
compounds, reply: 'No complex compounds.'

Prompt Template:

Analyze compounds/sandhi in this text:
{source_chunk}
Judge diagnosticValue
Reporting judgeopenai:gpt-4o-mini
Judge failure rate0.41%
Judge arithmetic disagrees with rubric96.4% (mean ∣Δ∣|\Delta| = 16.37)
Test–retest SD (same input, repeated calls)1.45
Self-preference, baseline (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+2.84
Self-preference, baseline_dup (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+2.72
Self-preference, bo_n (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+6.79
Self-preference, full_swarm (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+5.43
Self-preference, guide_ellipsis (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+4.49
Self-preference, guide_only (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+1.94
Self-preference, no_critique (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+4.48
Self-preference, no_ellipsis (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+5.12
Self-preference, no_guide (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+5.77
Self-preference, no_tools (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+5.46
Self-preference, placebo_rejudge (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+2.90
Self-preference, self_refine (Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}})+5.65

Table 12. Full per-condition judge-validation diagnostics; self-preference is Jroute−JreportJ_{\mathrm{route}} - J_{\mathrm{report}}. Table 5 in the main text summarizes this. Referenced from § 5.2.

Figure 2. Tail of the per-verse MQM distribution, full swarm vs. monolithic baseline. Referenced from § 5.

Discourse Argument Restoration (Ellipsis Restorer Agent)

This agent operates over a sliding window to interpolate omitted arguments (zero-anaphora) into the draft translation. In production, the genre field passed below is "legal" whenever the source work ID is flagged voice_preserving in the register table (covering the legal and sutra registers), and the literal register name otherwise (§ 3.4).

System Instruction:
You are an expert classical language editor.
Sanskrit frequently omits pronouns,
subjects, or verbs (ellipses) when they are
clear from context. Review the source
Sanskrit text, the surrounding discourse
context, and the draft English translation.
If the source genre is impersonal or
injunctive (e.g. legal/sutra), prefer
impersonal or passive restorations over
inventing an active third-person subject.
Identify if any implied arguments are
missing in the draft English translation. If
so, write down the restored/improved
translation. If the draft translation is
already complete, accurate, and fluent,
return the draft translation EXACTLY as is.
Return only the final English text without

explanations or headers.

Prompt Template:
Source Sanskrit:
{source}

Source genre: {genre}

Draft English Translation:
{draft_translation}

Surrounding Discourse Context:
- Preceding Chunk: {prev_chunk}
- Succeeding Chunk: {next_chunk}

Output the restored translation:

Quality Evaluation Judge (GEMBA-MQM)

Both JrouteJ_{\mathrm{route}} and JreportJ_{\mathrm{report}} (§ 3.5) use this prompt; the reported metric is the deterministic score recomputed from errors using the fixed weights below, not the model’s own top-line number, which is retained separately as model_reported_score for the judge-validation diagnostic in § 5.2.

System Instruction:
You are an expert translation quality evaluator
trained in MQM (Multidimensional Quality
Metrics). You identify translation errors from
classical languages into English and assign
severity levels. Be rigorous but fair. Respond
strictly in the JSON schema requested.

Prompt Template:
Source language: {lang_name}
Source text:
"""
{source}
"""

Candidate translation (English):
"""
{translation}
"""

Identify every translation error. For each,
choose:
  - category: one of [accuracy/mistranslation,
    accuracy/omission, accuracy/addition,
    accuracy/untranslated, fluency/grammar,
    fluency/register, fluency/spelling,
    fluency/punctuation, style/awkward,
    terminology, other]
  - severity: one of [minor, major, critical]
  - span: the offending English substring
    (verbatim)
  - explanation: one short sentence

Then compute an overall MQM score. Use the standard weights:
  critical = -25, major = -5, minor = -1
  score = max(0, 100 + sum(weights))
Return ONLY valid JSON matching:
{
  "errors": [
    {"category": "...", "severity": "...", "span": "...",
     "explanation": "..."}
  ],
  "score": <number 0-100>,
  "rationale": "<one sentence summary>"
}

Critique and Tool-Selector Agents

When a translation falls below the quality threshold, the Critique Agent diagnoses the errors, and the Tool Selector Agent retrieves necessary context strictly from the tools reported as available in the current environment (§ 3.6); for the benchmark in this paper, that inventory reduces to TRANSLATION_MEMORY.

Critique Agent System Instruction:
You are a senior translation editor. A Sanskrit-to-English translation
has received a low evaluation score from our automated judge. Perform a
detailed critique. Identify errors, mistranslations, omissions, or
terminology mismatches in the draft compared to the Sanskrit source.
Suggest specific corrections.
Tool Selector Agent System Instruction:
You are a research coordinator routing translation tasks to tools. Review
the Sanskrit source text and the critique. Determine which tools from
the inventory below are needed to resolve the issues. Only tools listed
here are available in this environment; do not request any other tool.
Available Tools (environment-reported; varies by run):
1. TRANSLATION_MEMORY: fuzzy-match the source against a decontaminated
   parallel corpus for sentence-level precedents (query = Sanskrit text).
2. DICTIONARY: look up a headword in the offline lexicon snapshot, if
   one is configured for this run (query = single word in IAST).
3. ALIGNMENT: Tibetan/Classical Chinese parallel alignment, if an
   alignment index is configured for this run (query = Sanskrit keyword).

You must select between 0 and 3 tool calls. Return your selection ONLY
as a JSON list of dictionaries, e.g.:
[
  {"tool": "TRANSLATION_MEMORY", "query": "citraśravaḥ tama havante"}
]
Do not output markdown code blocks or any
explanation.

References

  1. [1]Anonymous. 2026a. MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for Pāli, Sanskrit, Buddhist Chinese, and Tibetan. Citation anonymized for double-blind review.
  2. [2]Anonymous. 2026b. Mitrasamgraha: A Comprehensive Classical Sanskrit Machine Translation Dataset. Citation anonymized for double-blind review.
  3. [3]Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, and 1 others. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv preprint arXiv:2207.04672.
  4. [4]Sushant Dave, Arun Kumar Singh, A. P. Prathosh, and Brajesh Lall. 2021. Neural Compound-Word (Sandhi) Generation and Splitting in Sanskrit Language. In 8th ACM IKDD CODS and 26th COMAD, pages 171–177. Association for Computing Machinery.arxiv.org/abs/2010.12940
  5. [5]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv preprint arXiv:2305.14325.arxiv.org/abs/2305.14325
  6. [6]Google. 2026. Gemma 4: Open Models for Responsible AI Development. Technical report, Google DeepMind.
  7. [7]Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luísa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection. Transactions of the Association for Computational Linguistics, 12:979–995.arxiv.org/abs/2310.10482
  8. [8]Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindřich Helcl, and Alexandra Birch. 2022. Survey of Low-Resource Machine Translation. Computational Linguistics, 48(3):673–732.
  9. [9]Oliver Hellwig. 2010–2021. DCS - The Digital Corpus of Sanskrit.
  10. [10]Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. arXiv preprint arXiv:2302.09210.arxiv.org/abs/2302.09210
  11. [11]Gérard Huet. 2004. Design of a Lexical Database for Sanskrit. In Proceedings of the Workshop on Enhancing and Using Electronic Dictionaries, pages 8–14, Geneva, Switzerland. COLING.DOI
  12. [12]Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezhen Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin J. Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12):248.DOI
  13. [13]Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest Neighbor Machine Translation. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2010.00710
  14. [15]Tom Kocmi and Christian Federmann. 2023b. Large Language Models Are State-of-the-Art Evaluators of Translation Quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation.arxiv.org/abs/2302.14520
  15. [16]Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157–173.arxiv.org/abs/2307.03172
  16. [17]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In Advances in Neural Information Processing Systems, volume 36, pages 46534–46594.arxiv.org/abs/2303.17651

Paper details

vlnc:W_j4awa2fw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents