WorksComputer Science
XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages
Abstract
Typed Decision Models such as Jev, Clef and pplx-decider return a probability for each answer an application provides. Applications use them in many languages, but are compared mostly in English. We introduce XDecideBench, a multilingual benchmark of nine suites in 122 languages. On our twelve-language suite, three open models fine-tuned for typed decisions beat their Qwen base models in every language. On new test data with local base models, their accuracy gain is larger in languages with less web text than in those with more. They fix a similar share of base-model errors in both, so the larger gain may reflect weaker base models. The best model on the twelve-language suite is best on only two other suites. The best English model on each suite, chosen in hindsight, is significantly beaten in about a quarter of suite and language combinations. On Belebele, Decision-1 is best in English but falls far behind Jev in Burmese. A confidence threshold fitted on English to allow 5% errors gives a higher error rate on XNLI, even in German and French. XDecideBench shows that results from one suite or from English are a poor guide to how a model and its threshold behave in other languages.
Introduction
Many calls that applications make to language models are decisions, such as routing a request to one of sixty intents, flagging a message as abusive or rating its anger on a four-level scale. Typed Decision Models ([37, 22]), also called System One models ([60]), answer such calls directly (Figure 1). With each request the application declares the possible answers (named options, yes or no, or a level on an ordered rubric), and the model returns a probability for each. We call these CHOICE, NOUL and SCORE decisions ([60]). Some are decision heads, built by fine-tuning a general language model (the backbone) to return such probabilities. A backbone readout asks the untuned backbone the same question and reads its option probabilities, locally or through a hosted API. An application that acts only on confident decisions also needs a threshold, which is cheapest to fit on English development data.
Figure 1. The best model in English is confidently wrong on this Burmese item. A real Belebele item, selected after the runs, with the Burmese options glossed in English. Bars show the probabilities of Microsoft’s Decision-1, the best model on English Belebele. Two options that both Decision-1 and Jev put near zero are omitted. Over all Burmese items its Brier skill is .336, against .814 for Jev (§ 6.1).
Model rankings change from one test suite to another. On Core12, our development cohort of five datasets in twelve languages, GPT-6 Luna ranks eighth of the ten models with complete runs, but it is first on the native INCLUDE exams, at .600 against .536 for the runner-up. On the other suites, Kendall with the Core12 ranking is only .33–.64. The model that wins in English is not a safe choice elsewhere. Judged in hindsight, it loses significantly in 56 of the 232 suite–language cells and reaches only .336 on Burmese Belebele. A threshold set on English data for 5% errors lets through 7.8–12.4% on German and French XNLI.
| Study | Langs | DMs | Types | Same | P/N | H–B | Transf. |
| Deußer et al. | 200+ | 1 | CNS | – | – | – | – |
| yanng981 | 8 | 5 | C | ✓ | – | – | – |
| Convai Innovations | 51 | 1 | CNS | – | – | – | – |
| Tateno | 1 | 23 | CNS | ✓ | – | – | – |
| Rafe and Das | 1 | 8 | CN | ✓ | – | ✓ | – |
| Lu and Wu | 1 | 2 | C | ✓ | – | – | task |
| XDecideBench | 122 | 11 | CNS | ✓ | ✓ | ✓ | lang. |
Langs: languages evaluated (ours: headline suites). DMs: Typed Decision Models or checkpoints. Types: CHOICE, NOUL, SCORE. Same: models on identical items. P/N: aligned and native evidence reported separately. H–B: heads against own backbones. Transf.: a confidence policy reused on another task or language. XDecideBench: this paper.
Earlier studies evaluate either one Typed Decision Model across languages or several models in English (§ 2), whereas XDecideBench evaluates several models on the same multilingual items.
RQ1 asks how the models compare across suites and label regimes, RQ2 how much open heads gain over their own backbones, and RQ3 whether confidence policies fitted in English hold in other languages. Our first contribution is a common scoring contract (§ 5). It fixes what counts as a valid answer, how refusals are scored, which items form a cell and how we resample. Within a suite, Jev in Thai and Clef in German are then scored alike.
Our second contribution is the benchmark itself (§ 3), with 88,425 texts and 153,875 questions in 122 languages, grouped into nine aligned, locally labelled or native suites. We score eleven Typed Decision Models from ten developers, the stated Qwen backbones of three open heads (two of them locally verified) and a supervised mmBERT comparator (§ 4). Each open head beats its backbone in all Core12 languages under both readouts and gains more accuracy where web text is scarce. Under the local readout, a pre-registered held-out cohort confirms this larger gain and shows that the heads remove a similar share of backbone errors in low- and high-web-share languages (§ 6.2).
Related Work
Typed Decision Models. Deußer et al. [18] evaluate Jev on 37 datasets, including Belebele in 122 varieties and SIB-200 in 205 configurations. They report calibration and selective prediction, but compare Jev only with ordinary language models read through their option probabilities. Community benchmarks compare several such models on shared items, either in English with all three decision types ((Tateno, 2026; apolinario, 2026; Benchmark Heaven, 2026)) or in eight languages with CHOICE only ((yanng981, 2026)). Laya’s report covers 51 MASSIVE languages with a 20-intent subset ([16]). Rafe and Das [52] and OpenDecider ((manjunathshiva, 2026)) compare decision checkpoints with their untuned backbones in English. Other English-centred preprints benchmark calibration ([56]), audit agent decisions ([34]) and train a looped model ([22]). Lu and Wu [37] find that a rejection policy calibrated on one task loses its operating point on another (Table 1).
| Study | Langs | DMs | Types | Same | P/N | H–B | Transf. |
| Deußer et al. | 200+ | 1 | CNS | – | – | – | – |
| yanng981 | 8 | 5 | C | ✓ | – | – | – |
| Convai Innovations | 51 | 1 | CNS | – | – | – | – |
| Tateno | 1 | 23 | CNS | ✓ | – | – | – |
| Rafe and Das | 1 | 8 | CN | ✓ | – | ✓ | – |
| Lu and Wu | 1 | 2 | C | ✓ | – | – | task |
| XDecideBench | 122 | 11 | CNS | ✓ | ✓ | ✓ | lang. |
Table 1. No prior study covers more than two of the last four columns.
Multilingual calibration and benchmark design. Ordinary multilingual models are miscalibrated outside English, and temperatures fitted in English transfer poorly ([4, 32, 65, 68]). English-heavy instruction tuning can raise low-resource confidence without raising accuracy ([28]). The closest study is Mling-Conf ([64]), which finds that English dominates LLM confidence estimation across languages. We ask whether an English-fitted policy, either a risk threshold ([23]) or a split conformal set ([5] (Sadinle et al. 2019)), keeps its operating point in other languages for several Typed Decision Models. AfroBench and BenchMAX rank general LLMs on African languages and post-edited translations ([45, 29]), and MuBench aligns items for generative models ([25]). XDecideBench adds typed decisions, matched head–backbone pairs and a pre-registered held-out confirmation. Like Belebele and XTREME-S, we fix parallel items ([8, 14]). Because translation adds its own artefacts ([7]), we report native material separately, following INCLUDE and Global-MMLU ([53, 57]). Our chance-normalised skills follow SEA-HELM and MMTEB ([58, 20]), and, as in HELM, failures stay in the score ([35]). We resample clusters ([41]) and give each score a sacreBLEU-style signature ([49]).
The Benchmark
Three label regimes. XDecideBench scores only the probabilities that a Typed Decision Model returns for the declared answers (Figure 2a). In four suites the same source items appear in all languages with one label per item, so a gap between languages reflects language and translation rather than content. RTP-LX also aligns its items across 38 locales, but three annotators from each locale label them, and the gold label is their median. The gold toxicity label differs from the English one for 4,016 of 7,400 aligned locale–item pairs (54%), and across all eight rubrics 22% of labels differ (Appendix § 11.§ 3). A language gap on RTP-LX therefore measures how well a model predicts local labels. The four native suites contain items written in each language for its speakers and test local validity. Since topic, culture and difficulty differ, we never compare their raw scores across languages ([53, 57, 30]).
Figure 2. Overview of XDecideBench. (a) What we score (§ 5). (b) Headline suites by label regime, and the held-out cohort. A question is one typed decision about a text. Multilingual NLI reuses the 6,000 Core12 XNLI texts, so the suites hold 88,425 distinct texts and 153,875 questions. (c) One tick per covered language, ordered by share of Common Crawl pages (log scale). Languages missing from the crawl use their macrolanguage’s share, and languages with neither are at the right. (d) Research questions (§ 6).
Suites. Besides Core12, the aligned suites are Wide-Belebele ([8]), MASSIVE, an NLI family ([15, 3, 19]) and RTP-LX toxicity ([17]) (Figure 2b). MASSIVE covers all 52 locales on 117 utterances unused elsewhere ([21]). The native suites are INCLUDE ([53]), InjongoIntent in 16 African languages ([67]), Multilingual HateCheck with Aya red-teaming prompts ([54, 1]) and BRIGHTER emotion intensity ([42]) (Appendix § 11.§ 2). Belebele, NLI and Injongo reach the languages with the least web text (Figure 2c).
Scoring contract. We score a suite only when a submission covers every case in every cell ([14]). Refusals, interface limits and invalid outputs count as a uniform distribution, and we print the share of valid outputs beside each score ([35, 26]). A cell’s score is skill relative to a declared reference . For CHOICE, is uniform chance, and for the skewed NOUL and SCORE datasets it is the cell’s constant base-rate predictor. With BS the mean multiclass Brier score, the primary metric is . The Brier score is strictly proper and bounded, though BSS is not bounded below. Unlike log loss, it stays finite when rounded vendor probabilities contain exact zeros ([24]). We average with equal weights at each level from items up to suites, but never across suites ([20]). Each mean has a cluster-bootstrap interval (§5) and a signature, and we never rank scores with different signatures together.
Core12. Core12 covers twelve languages from tiers 2 to 5 of Joshi et al. [33] (Arabic, German, English, Spanish, French, Chinese, Hindi, Russian, Vietnamese, Thai, Urdu and Swahili) and keeps Bold: the short name used in the text and all tables. API models are dated snapshots, except the undated hosted Clef routes. †Locally verified pair: the head’s pinned local weights and this exact backbone revision. ‡Base stated by the vendor and not verified by us. Precision: of the returned probabilities (dp: decimal places).
| Typed Decision Model | Access | Backbone | Precision |
| Clef | local, API | Qwen3.8-27B† | fp32/4 dp |
| Clef-flash | local, API | Qwen3.5-9B† | fp32/4 dp |
| pplx-decider-v1.1 | API | Qwen3.8-27B‡ | full |
| Microsoft-Decision-1 | API | Qwen3.5-9B‡ | full |
| Kev-4B | API | Qwen3.5-4B‡ | 4 dp |
| Laya-multilingual | local | mmBERT-base | full |
| Jev 1.13 | API | undisclosed | 2 dp |
| GPT-6 Luna Decisions | API | undisclosed | 2 dp |
| Liquid d1 | API | undisclosed | full |
| Mercury Decide | API | undisclosed | full |
| Solar Decide | API | undisclosed | 6 dp |
Table 2. The eleven Typed Decision Models. Three are paired with their own Qwen backbone, and a fourth pair, Decision-1, is in Appendix § 14.§ 4.
the audited items of five labelled sources that exist in all twelve. The sources are MASSIVE intents over all 60 classes [9], XNLI [15], Belebele, full MMLU-ProX [63] and SIB-200 topics [2], with 500, 500, 501, 500 and 204 items per language. For the headline Core12 requests we reuse the source labels, translations and distractors. Each request pairs target-language content with an English instruction and English option texts, as in an application with an English schema. MASSIVE is localized, XNLI and SIB-200 are translated and MMLU-ProX is machine-translated with expert review.
Development and held-out cohorts. Core12 is the development cohort, and the held-out cohort has 4,860 cases (405 source items per language) from MASSIVE, XNLI, Belebele and MMLU-ProX in the Core12 languages. None of its cases shares a source item, near-duplicate text or input hash with any other manifest. We froze it before any inference and ran it only after pre-registration. We will release pinned manifests that rebuild identical cases at https://github.com/NaiveNeuron/XDecideBench, but not the request texts (Appendices § 11 and § 17).
Panel and Readouts
Typed Decision Models. Table 2 lists the eleven models that we run on identical requests. They are TypeSafe’s Jev [60], the open-weight heads Clef, Clef-flash, pplx-decider and Kev [12, 48, 47], hosted models from Liquid, Upstage, Inception, OpenAI and Microsoft [36, 61, 31, 46, 40] and the small encoder Laya [16]. Where head training is documented, it is mostly in English. We also run Clef and Clef-flash locally on an H100 from pinned public weights, and Laya from a pinned checkpoint. The hosted Clef routes name no checkpoint, but their top answer matches the local one on 99.1% of Core12 cases for Clef and on 99.5% for Clef-flash (Appendix § 17).
Backbones and readouts. For the three primary head–backbone pairs we also score the declared base [51 a]. Two pairs, Clef with Qwen3.8-27B and Clef-flash with Qwen3.5-9B, are locally verified: we run both head and backbone from pinned local weights. For pplx-decider we rely on the vendor’s statement of the base. Our provider offers no log-probabilities for Kev-4B’s stated base, Qwen3.5-4B, so we did not run it. The local readout gives the backbone the same input, instruction and option texts through one frozen prompt (Appendix § 12.§ 1) in its own chat template with thinking disabled. We teacher-force each option text and take the softmax over mean per-token log-likelihoods, which gives an exact but uncalibrated distribution. The hosted readout sends the prompt with lettered options to a pinned bf16 (9B) or fp16 (27B) endpoint. It reads the label distribution from the top-20 log-probabilities. It is our only backbone readout outside Core12 and Wide-Belebele. The pre-registered hypotheses use the local readout, which scores the heads’ own option texts exactly, and the hosted readout is a sensitivity check.
Comparators and limits. As a supervised comparator, we fine-tune Laya’s backbone, mmBERT-base, on the training data of MASSIVE, SIB-200 and English MultiNLI [39, 62]. Laya uses the same encoder and has near-zero Brier skill on MASSIVE (.030) despite .320 accuracy. We also score the MediaPipe Decision Maker with EmbeddingGemma 2 and with GLiNER2.5-Decision as off-the-shelf comparators (MP-EG2 and MP-GliNER). Vendors return probabilities at different precisions, and each model has a fixed acceptance rule that matches its rounding (Appendix § 12.§ 2). Because two-decimal Brier skill against each suite’s reference, with failures scored as uniform. Grey subscripts: half-width of the 95% joint cluster-bootstrap interval. Bold: best hosted Typed Decision Model per column. Underlined: runner-up. †Valid-output coverage below 1. ‡Partly over an interface cap (Table 16). capped: all items exceed Solar’s option cap. Laya ran on its own copies of the suite manifests, and its model card lists MASSIVE and XNLI as evaluated in its development. Local runs (Core12 / Belebele only): Clef .674±.020 / .701±.055, Clef-flash .631±.020 / .597±.053. Local backbone readouts: Qwen3.8-27B .410±.021 / .409±.046, Qwen3.5-9B .403±.014 / .308±.040. Names: Table 2.
| (a) | Aligned, shared labels | Locale labels | ||||||||
| Model | Core12 | Belebele | MASSIVE | NLI | RTP-LX | |||||
| Fail. | Zero | Fail. | Zero | Fail. | Zero | Fail. | Zero | Fail. | Zero | |
| Hosted Typed Decision Models | ||||||||||
| pplx-decider | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 |
| Jev | 0 | 86.5 | 0 | 75.6 | 0 | 100.0 | 0 | 72.7 | 0 | 93.1 |
| Clef | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 8 | 0.0 |
| d1 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 |
| Decision-1 | 0 | 0.0 | 0 | 0.0 | 0 | 0.6 | 0 | 0.0 | 0 | 0.0 |
| Clef-flash | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 8 | 0.0 |
| Mercury | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 |
| Luna | 64 | 67.4 | 1 | 71.0 | 46 | 99.7 | 0 | 56.3 | 328 | 96.2 |
| Kev-4B | 0 | 0.2 | 0 | 0.0 | 0 | 0.3 | 0 | 0.2 | 0 | 0.0 |
| Solar | 6,000 | 0.6 | 0 | 0.0 | 6,084 | – | 0 | 0.4 | 0 | 0.0 |
| Backbone readouts, hosted | ||||||||||
| Qwen3.8-27B | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 |
| Qwen3.5-9B | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 |
| Comparators | ||||||||||
| MP-EG2 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 |
| MP-GLiNER | 2,073 | 0.0 | 2,202 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 |
| (b) | Native-authored | |||||||||
| Model | INCLUDE | Injongo | Moder. | BRIGHTER | ||||||
| Fail. | Zero | Fail. | Zero | Fail. | Zero | Fail. | Zero | |||
| Hosted Typed Decision Models | ||||||||||
| pplx-decider | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | ||
| Jev | 0 | 51.2 | 0 | 100.0 | 0 | 31.8 | 0 | 77.6 | ||
| Clef | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | ||
| d1 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | ||
| Decision-1 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | ||
| Clef-flash | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | ||
| Mercury | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | ||
| Luna | 122 | 55.9 | 12 | 84.2 | 48 | 60.5 | 0 | 96.5 | ||
| Kev-4B | 0 | 0.0 | 0 | 4.0 | 0 | 0.0 | 0 | 0.0 | ||
| Solar | 0 | 0.1 | 1,920 | – | 0 | 0.9 | 0 | 0.0 | ||
| Backbone readouts, hosted | ||||||||||
| Qwen3.8-27B | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | ||
| Qwen3.5-9B | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | ||
| Comparators | ||||||||||
| MP-EG2 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | 0 | 0.0 | ||
| MP-GLiNER | 439 | 0.0 | 321 | 0.0 | 0 | 0.0 | 0 | 0.0 |
Table 16. Fail.: items whose status is not a valid typed answer (refusals, provider errors and interface caps). Zero: share of valid answers, in percent, with an exact-zero probability.
Table 16: Failures and exact zeros per model and suite. Failed items (scored as uniform) and the share of valid answers whose probability vector contains an exact zero: (a) aligned suites; (b) native suites.
| Model | Core12 chance | Belebele chance | MASSIVE chance | NLI chance | RTP-LX base rate | INCLUDE chance | Injongo chance | Moder. mixed | BRIGHTER base rate |
| Reference | |||||||||
| pplx-decider | .712±.019 | .756±.058 | .718±.075 | .397±.022 | .051±.041 | .536±.009 | .578±.026 | .673±.022 | .144±.020 |
| Jev | .688±.022 | .785±.060 | .619±.081 | .128±.030 | −.101±.051 | .531±.009 | .499±.024 | .561±.026 | −.428±.046 |
| Clef | .674±.020 | .701±.055 | .714±.069 | .254±.023 | .045±.046 | .459±.009 | .492±.026 | .274±.032 | .167±.016 |
| d1 | .663±.020 | .711±.061 | .634±.073 | .315±.023 | .070±.046 | .488±.009 | .481±.024 | .449±.027 | .118±.022 |
| Decision-1 | .651±.018 | .716±.047 | .721±.059 | .108±.029 | −.229±.061 | .400±.011 | .417±.024 | .500±.029 | −.375±.040 |
| Clef-flash | .631±.020 | .597±.053 | .745±.061 | .127±.025 | .071±.047 | .385±.009 | .415±.026 | .379±.031 | .123±.016 |
| Mercury | .611±.025 | .677±.076 | .651±.087 | .188±.030 | −.022±.052 | .411±.011 | .416±.031 | .449±.032 | .013±.027 |
| Luna | .568±.023 | .743±.073 | .602±.065 | −.058±.036 | −.018±.047 | .600±.009 | .580±.018 | .541±.031 | −.096±.031 |
| Kev-4B | .449±.021 | .389±.091 | .371±.046 | .070±.030 | −.169±.051 | .273±.007 | .225±.014 | .278±.023 | −.275±.036 |
| Solar | .398±.024 | .334±.074 | capped | −.236±.029 | −.568±.078 | .251±.013 | capped | −.046±.058 | −1.396±.077 |
| Laya | .119±.022 | −.167±.057 | −.120±.069 | −.206±.033 | −1.647±.118 | −.085±.006 | −.096±.014 | −.415±.057 | −2.907±.128 |
| Qwen3.8-27B | .584±.024 | .678±.065 | .523±.088 | .043±.035 | −.336±.073 | .426±.010 | .405±.028 | .397±.031 | −.056±.028 |
| Qwen3.5-9B | .477±.022 | .525±.077 | .313±.091 | −.036±.029 | −.562±.086 | .310±.010 | .167±.029 | .347±.029 | −.034±.027 |
| MP-EG2 | −.084±.029 | −.284±.097 | .347±.058 | −.997±.039 | −1.363±.116 | −.242±.010 | .054±.010 | −.004±.021 | −1.166±.068 |
| MP-GLiNER | .057±.015 | −.066±.048 | .029±.037 | .010±.007 | −.984±.081 | −.034±.005 | .032±.008 | .076±.021 | −.807±.044 |
Table 2 (PDF p. 5).
Table 3: Rankings change across suites. Suites differ in reference and population, so scores should not be compared across columns.
APIs return exact zeros, we floor log loss at . Interface limits count as failures rather than missing data. Solar Decide rejects CHOICE questions with more than 26 options, so all 60-intent MASSIVE cases score as uniform. Laya refuses inputs that its description cap would truncate.
Protocol and Statistics
Evidence status. We use Core12 for development, and the held-out cohort below is the prospective test. We fixed the Core12 manifests and an accuracy-based protocol before any benchmark inference, and registered each model with its dated snapshot before its first benchmark request (Table 14). We made Brier skill the primary suite score after most Core12 runs, but before any request on the other eight suites, and pinned those suites’ manifests at the same time. On Core12, the Brier-skill and chance-corrected-accuracy rankings of the ten hosted models agree at Kendall .
Head–backbone comparison. For a pair, is the head’s metric minus its backbone’s metric on the same cases of dataset and language . Accuracy uses all cases, while Brier score and adaptive expected calibration error (ECE; (Nixon et al., 2019)) use the cases where both models return a valid answer. The per-language effect averages the five datasets with equal weight. The interaction contrast is the mean over a low-web-share group {Hindi, Swahili, Thai, Urdu} minus the mean over a high-web-share group {English, German, Spanish, French, Russian, Chinese}. Web share, our rough proxy for pretraining exposure, is at least 4.2% of Common Crawl pages in the high group and at most 0.36% in the low group ([13]), and Arabic and Vietnamese fall in between. An absolute gain cannot exceed the backbone’s error rate, so we report the share of backbone errors removed, for accuracies and .
Calibration with an English temperature. For each head and its backbone’s local readout, we fit one temperature per dataset (MASSIVE, XNLI) on English development data and apply it unchanged to all Core12 languages. We report ECE and Brier score before and after scaling, which leaves accuracy unchanged.
Transfer of English-fitted confidence policies. We split the English development data (500 MASSIVE and 500 XNLI decisions) by source cluster and fit a temperature on one half. On the other half we fit thresholds at 1%, 5% and 10% empirical risk and a split-conformal set with score at ([5] (Sadinle et al. 2019)). We apply these policies unchanged to all Core12 languages on MASSIVE and XNLI, for each hosted model with English development runs, and report risk among admitted decisions, admitted share, conformal coverage and set size. The targets are nominal, with thresholds estimated on 250 English decisions, and conformal coverage assumes exchangeability with the English data.
Uncertainty. Translations of one source item, questions on one Belebele passage and XNLI hypotheses with a shared premise are dependent, so all intervals resample whole source clusters. Contrasts use a paired percentile bootstrap in which both models and all target languages share one cluster resample. For transfer, we recompute admitted sets, risk and coverage in each replicate. For suite scores, all languages and models of a parallel dataset share one draw of clusters, while native datasets resample within each language ([41]). The intervals reflect only item sampling, for this fixed set of suites and languages. Development-cohort intervals are not adjusted for multiplicity, and we call a contrast significant when its 95% interval excludes zero.
Confirmation on the held-out cohort. Before any request on the held-out cohort (Table 14), we pre-registered the panel, the analysis code and two hypotheses on the local readout, pooled over the three head–backbone pairs. Holm correction controls the familywise error at ([27]). H1 states that the heads gain more accuracy in the low-web-share group than in the high one. H2 states that the share of backbone errors removed is the same in both groups. We test equivalence with two one-sided tests (TOST) and a margin of , about 2.8 to 3.3 accuracy points at low-web-share backbone error rates. Simulated power is 98% for H1 at the Core12 effect and 70% for H2 at a true difference of zero (Appendix § 17). We will release the registration.
Results
Decision quality across suites (RQ1)
Table 3 gives all model–suite scores. On Core12, the hosted Typed Decision Models range from .398 to .712, and all except Solar beat both local backbone readouts. The hosted readout of plain Qwen3.8-27B (.584) scores above GPT-6 Luna, Kev-4B and Solar. Only coarse ties are stable, and the rank intervals of the top models overlap (pplx-decider 1–3, Jev 1–7, Decision-1 1–7, d1 2–6 in Table 8). On the three Core12 datasets it was trained for, the supervised mmBERT comparator reaches .673 [.639, .709], and Laya, which uses the same encoder, reaches .274 [.234, .307] (.119 on all of Core12). Both MediaPipe arms trail all hosted models on Core12, Belebele, RTP-LX and INCLUIR, and Laya beats its reference on only one of its nine suites, Core1.
| Model | Brier skill | Rank | Worst | Gap to en | Cov. |
| pplx-decider | .712 [.693, .731] | 1–3 | sw .62 | −.075 | 1.000 |
| Jev | .688 [.667, .710] | 1–7 | sw .58 | −.096 | 1.000 |
| Clef, API | .674 [.655, .695] | 2–5 | sw .56 | −.103 | 1.000 |
| d1 | .663 [.642, .682] | 2–6 | sw .55 | −.102 | 1.000 |
| Decision-1 | .651 [.633, .669] | 1–7 | sw .52 | −.131 | 1.000 |
| Clef-flash, API | .631 [.611, .651] | 3–8 | sw .47 | −.133 | 1.000 |
| Mercury | .611 [.588, .638] | 5–9 | sw .45 | −.110 | 1.000 |
| Luna | .568 [.545, .591] | 5–9 | sw .50 | −.084 | .998 |
| Kev-4B | .449 [.428, .470] | 8–11 | sw .28 | −.150 | 1.000 |
| Solar | .398 [.373, .422] | 6–11 | sw .14 | −.141 | .773 |
| Local heads | |||||
| Clef, local | .674 [.655, .695] | – | sw .56 | −.103 | 1.000 |
| Clef-flash, local | .631 [.611, .651] | – | sw .47 | −.133 | 1.000 |
| Backbone readouts | |||||
| Qwen3.8-27B{sup}* | .410 [.390, .431] | – | hi .30 | −.129 | 1.000 |
| Qwen3.5-9B{sup}* | .403 [.388, .417] | 9–11 | hi .29 | −.110 | 1.000 |
| Qwen3.8-27B{sup}◦ | .584 [.562, .610] | – | sw .45 | −.134 | 1.000 |
| Qwen3.5-9B{sup}◦ | .477 [.455, .499] | – | sw .29 | −.166 | 1.000 |
Table 8. Brier skill: estimate followed by its 95% joint cluster-bootstrap interval [lo, hi]. Rank: hierarchical 95% rank interval among the eleven models ranked together, the ten hosted Typed Decision Models and the local Qwen3.5-9B backbone, with datasets and languages also resampled and bounds widened to whole ranks. –: not ranked. Worst: lowest-scoring language and its skill. Gap: mean language minus English. Cov.: share of valid typed answers. {sup}*Backbone, local readout. {sup}◦Backbone, hosted readout.
Rankings depend on the task. On the other suites, Kendall with the Core12 ranking of the ten models with complete runs is only .33 (RTP-LX) to .64 (NLI, Injongo). GPT-6 Luna is eighth on Core12 but first on the native INCLUIR exam (.600, against .536 for pplx-decider). It also shares the Injongo lead with pplx-decider (.580 against .578) but falls below chance on NLI (). On the two score-type suites, several models do no better than a base-rate predictor. On RTP-LX, skill runs from (Solar) to .071 (Clef-flash) and is significantly below zero for Jev, Kev-4B, Decision-1 and Solar. Jev, second on Core12, scores on BRIGHTER and beats only Solar. Kev-4B, Luna and Decision-1 also score below the base rate. Ordinal scoring also reorders models. On RTP-LX, Brier-skill and ranked-probability-skill rankings agree at , and Luna moves from fifth to second (Appendix § 13).
Choosing a model on English. If we pick the best English model on each parallel suite from the same test data, another model beats it in 109 of 232 suite–language cells, significantly so in 56 (Appendix § 13). On Belebele, Decision-1 is best in English and loses significantly in 37 of 105 languages (Figure 3), with the largest loss in Burmese, where it ranks seventh. Picking the best model on Core12 gives pplx-decider, which is best on only two of the eight other suites and significantly beaten on Belebele, INCLUDE and BRIGHTER. On the held-out cohort, the English pick is best in all eleven languages. There, the ranking of the ten Typed Decision Models agrees with Core12 at Kendall (exploratory, Appendix § 16), so new languages on the same datasets reorder the models much less than new suites do.
Figure 3. The best model in English is often beaten in other languages. Belebele Brier skill of Decision-1, which is best in English (line), and of the best model per language, chosen in hindsight (markers). The five languages without a web share are left out. Dark stems: significant paired losses (37 of 105 languages, 35 shown).
Label descriptions and native items. One-line English label descriptions raise MASSIVE skill by .035 to .053 for four models but barely change SIB-200. On the nine languages INCLUDE shares with MMLU-ProX, Luna’s native-exam skill exceeds its translated-exam skill by .210, while Jev’s is .082 lower (Appendix § 13.§ 2). Neither diagnostic explains the ranking changes.
Interface language. In an exploratory ablation (Appendix § 13.§ 3), claude-opus-5.5 translated the instructions and option texts of Core12 XNLI and MASSIVE into eight languages, and gpt-6.1-sol back-translated them blind, with no native-speaker check. Native instructions lower pooled XNLI Brier skill by .022 [.017, .026], by .039 in the low-web-share group and .002 in the high one. On MASSIVE, translating the texts changes nothing (−.003 [−.010, +.003]) if hosted models still see English option IDs. With a fully native interface and neutral IDs, skill falls by .055 across six models. Clef-flash’s XNLI lead over the hosted Qwen3.5-9B readout shrinks by .099 [.067, .129], so the § 6.2 head gains hold under English schemas.
Heads versus their own backbones (RQ2)
Accuracy and Brier score. Under the local readout, which H1 and H2 use, the heads win in all twelve languages on both metrics (Figure 4a and Figure 6), by 7.6 to 18.2 accuracy points and .15 to .29 in Brier score. The accuracy gain is significantly larger in the low-web-share group, by 2.2 points for Clef, 4.3 for Clef-flash and 4.7 for pplx-decider. In Brier score the interaction is significantly positive for Clef (.013) and pplx-decider (.042) but
Figure 4. Accuracy gains grow as web share falls. Heads against their own backbones per Core12 language, by decreasing web share. (a) Accuracy gain. (b) The gain as a share of the backbone’s errors, which barely changes under the local readout. Solid: local readout. Pale, dotted: hosted readout. Bars: 95% paired intervals. Segments: group means. Shaded: low-web-share group.
Figure 6. Each head beats its backbone in all Core12 languages and metrics. Head minus backbone for accuracy and backbone minus head for Brier score and ECE under the local readout, so higher is better and 0 is the backbone. Bars show 95% paired source-cluster bootstrap intervals, shading marks the low-web-share group, and open squares mark pplx-decider, the vendor-stated pair.
significantly negative for Clef-flash, −.011 [−.022, −.000]. On 106 Wide-Belebele languages, no head is significantly less accurate than its backbone (Appendix § 14.§ 3). On Core12, the heads remove similar shares of backbone errors in both groups (Figure 4b), with .33 against .36 for Clef, .30 against .31 for Clef-flash and .44 against .42 for pplx-decider. Only Clef’s difference is significant, and it favours the high-web-share group (−.032 [−.061, −.003]).
Calibration. The heads have lower raw ECE than their backbones in all languages, by .06 to .19. With English-fitted temperatures for both arms (Appendix § 14.§ 2), the heads keep the lower Brier score in all 36 MASSIVE and XNLI pair–language cells and the lower ECE in 36, significantly in 29.
Hosted backbone readout. Hosted readouts raise the backbones’ Core12 skill by .074 (9B) and .174 (27B), but the heads still win in all twelve languages on both metrics. The low-web-share group still gains significantly more accuracy, by 1.6 points for Clef, 2.0 for Clef-flash and 4.1 for pplx-decider. The difference in errors removed is −.008 for Clef, −.065 [−.096, −.034] for Clef-flash and +.052 [.017, .090] for pplx-decider. In an exploratory comparison outside H1 and H2, Decision-1 and its stated base Qwen3.5-9B also gain more accuracy in the low-web-share group under both readouts (Appendix § 14.§ 4). On Wide-Belebele, Decision-1’s accuracy gain has an interval above zero in 90 of 106 languages under the local readout and below zero in none, with a median gain of 11.0 points.
Held-out confirmation. On the held-out cohort, under the local readout, the pooled accuracy gain is 4.8 points larger in the low-web-share group, confirming H1 (Holm-level bound 2.8, ). The pooled difference in errors removed is −.034, with its 90% interval [−.073, .006] inside the margin. H2 holds at Holm’s second step (, TOST ) but not at the stricter first-step level of .025. Under the exploratory hosted readout, H1 holds with a smaller contrast (2.7 points, ), but H2 does not (TOST ).
Transfer of English-fitted confidence policies (RQ3)
On English test data, the 5% threshold keeps risk near target (Table 13), at 2.3–5.1% on XNLI and 1.7–6.6% on MASSIVE. Only Kev-4B exceeds 5% on both, and Solar admits only 3% on XNLI.
| Model | Risk at 5% (admitted %) | 90% set: coverage/size | ||||
| En | High | Low | En | High | Low | |
| XNLI | ||||||
| Jev | .035 (63) | .098 (55) | .099 (37) | .92/1.06 | .81/1.07 | .73/1.03 |
| pplx-decider | .045 (93) | .104 (89) | .133 (82) | .91/0.96 | .83/0.95 | .75/0.90 |
| d1 | .047 (89) | .100 (81) | .127 (67) | .92/0.99 | .82/0.97 | .74/0.93 |
| Mercury | .049 (85) | .096 (74) | .105 (58) | .91/1.02 | .82/1.03 | .76/1.04 |
| Kev-4B | .051 (78) | .126 (72) | .224 (52) | .90/1.04 | .80/1.04 | .65/1.07 |
| Luna | – (0) | – (0) | – (0) | .93/1.46 | .86/1.52 | .85/1.67 |
| Decision-1 | .023 (70) | .110 (64) | .174 (41) | .92/1.04 | .81/1.06 | .70/1.09 |
| Solar | .143 (3) | .225 (3) | .322 (3) | .91/1.38 | .83/1.44 | .73/1.49 |
| MASSIVE | ||||||
| Jev | .030 (47) | .029 (39) | .022 (29) | .94/1.86 | .93/2.00 | .88/2.38 |
| pplx-decider | .017 (59) | .019 (54) | .023 (43) | .93/1.20 | .92/1.23 | .87/1.27 |
| d1 | .041 (68) | .050 (64) | .060 (50) | .94/1.40 | .93/1.51 | .87/1.72 |
| Mercury | .040 (60) | .034 (57) | .044 (46) | .95/1.94 | .93/1.99 | .90/2.76 |
| Kev-4B | .066 (49) | .080 (43) | .113 (23) | .94/2.51 | .91/2.76 | .80/3.76 |
| Decision-1 | .037 (81) | .045 (78) | .052 (62) | .94/1.16 | .91/1.16 | .86/1.23 |
| Luna | .046 (48) | .060 (47) | .066 (40) | .90/2.07 | .89/2.12 | .86/2.41 |
Table 13. Table 13: English-fitted policies miss their targets on XNLI outside English. Policies fitted on English development data and applied unchanged on Core12, for every hosted model with English development runs, with a floor of . Solar cannot be fitted on MASSIVE, where its option cap leaves no valid English development decision.
More admitted errors on XNLI. For all six assessable models, risk rises significantly in German and French, by 3.0–8.0 points to 7.8–12.4% (Figure 5). Risk is higher still in the low-web-share group for all six models, and Kev-4B reaches a group mean of 22.4%. Conformal coverage for these six models is .90–.92 in English and drops to .80–.83 in the high-web-share group and .65–.76 in the low-web-share one, significantly below target. Luna keeps coverage at .85–.93 by returning larger sets. On the held-out cohort, the same thresholds give 8.4–13.2% risk in German and French in an exploratory analysis (Appendix § 16).
Figure 5. English-fitted thresholds let through more errors on XNLI and fewer decisions on MASSIVE. Error rate among admitted decisions (top) and admitted share (bottom) per Core12 language under an English-fitted 5% threshold, for the hosted models we could assess. Luna admits no English XNLI decisions, so it has no XNLI risk. Dotted box: English. Shaded: low-web-share group. Bars: 95% paired source-cluster intervals. Appendix E lists the omitted cells.
Lower admitted share on MASSIVE. On MASSIVE, risk stays near the target, but the models admit fewer decisions. No model’s mean risk changes significantly in high-web-share languages, though d1 in French (2.2 points [0.3, 4.4]) and Decision-1 in Spanish rise significantly. Outside English, risk exceeds the target by at most 1.2 points for Jev, pplx-decider, Mercury and Decision-1, and by 2.6 for Luna. The largest excesses are for d1 and Kev-4B in Swahili, at 5.6 and 9.9 points. In the low-web-share group, admitted share is 7.3 (Luna) to 25.1 (Kev-4B) points below English, and Jev admits 10.8% of Swahili decisions against 46.8% of English ones.
Conclusion
On XDecideBench, rankings change from suite to suite. In hindsight, the best model in English loses significantly in one cell of four, so neither a Core12 nor an English leaderboard reliably ranks models for deployment. Three open heads beat their Qwen backbones in all development languages and gain more where web text is scarce. Under the local readout, the held-out cohort confirms the larger gain, though the heads fix about the same fraction of their backbones’ errors in both groups. English-fitted thresholds admit more errors on XNLI, even in German and French, and fewer decisions on MASSIVE. Practitioners should pick a model on suites that match their decision type and setting, and fit its confidence policy on target-language data. We will release the manifests, scorer and run receipts at https://github.com/NaiveNeuron/XDecideBench so that new models can be scored on the same cases.
Limitations
Scope of the evidence. The head–backbone and transfer analyses use Core12, the development cohort. We repeat both on the held-out cohort, but the repeated transfer analysis is exploratory. The low-web-share group of the held-out cohort has four languages, and only Swahili lies below Joshi tier 3. The held-out cohort and the other suites are held out from our development analyses. But all sources are public, and some are well-known evaluation sets, so they may appear on the web or in vendors’ training data.
Head–backbone contrast. Head training data are undisclosed, so we cannot say which part of a gain comes from decision training. The local backbone readout is a length-normalised likelihood projection under one frozen prompt, and the hosted readout reads label tokens from a truncated top-20 list. Other prompts or readouts could make backbones stronger or weaker and change both the absolute gains and the shares of backbone errors removed. We have only three pre-registered pairs, plus the exploratory Decision-1–Qwen3.5-9B pair. Two of the three share the 27B backbone, and pplx-decider’s base is stated by its vendor. The comparison of shares concerns group averages rather than individual languages.
Translation, schema and web share. Most aligned suites translate or localize English material, and Core12 requests pair target-language content with English instructions and options. Our language effects therefore combine the effect of target-language content under an English schema with translation artefacts. In an exploratory ablation (Appendix § 13.§ 3), native instructions lower XNLI skill most in low-web-share languages, so the English schema may understate those gaps. Web share is confounded with script, language family, source quality and translation process. The XNLI–MASSIVE contrast cannot separate task from translation. RTP-LX gaps also include differences between local annotator pools.
Hosted models are snapshots. Headline API scores come from one pass over the cases, against snapshots that vendors may retire. We repeated the 8,448 Core12 MASSIVE and SIB-200 cases on the same day. The repeat keeps the top answer for 96.5 to 100% of cases per model and moves dataset Brier skill by at most .0024. Where we bracketed a run with probe requests, their probabilities drift by at most .032 between its start and end (Appendix § 17). We did not repeat the other eight suites, and the hosted Clef routes name no checkpoint. Vendors also round probabilities to different precisions. Brier skill limits the damage, but rounding still discards information.
HateCheck measures specific weaknesses. HateCheck is a functional test suite rather than a sample of real traffic, so it says little about quality in deployment.
Ethics Statement
Data and licences. The headline suites reuse existing public datasets and gold labels under their licences (Appendix § 11). For the exploratory interface-language ablation, an LLM translated the instruction and option texts (Appendix § 13.§ 3). We treat XNLI, AfriXNLI, AmericasNLI and RTP-LX as noncommercial. Belebele, SIB-200, AmericasNLI, Global PIQA and parts of RTP-LX are ShareAlike. Global PIQA forbids training, so we never use it for fitting. InjongoInt- ent carries conflicting licence declarations, none of them noncommercial. The benchmark as a whole is for research use. We will release manifests (identifiers and hashes), adapters, the scorer and aggregate results at https://github.com/NaiveNeuron/XDecideBench. We will not release source text, which users can obtain from the original distributors. The dataset authors built the datasets with their own annotator populations. We recruited no annotators, translators or human participants. The English described-label texts are new, and we wrote them from the label names.
Harmful content. RTP-LX, Multilingual Hate-Check and Aya red-teaming contain toxic, hateful or harmful text by design. RTP-LX’s terms forbid harmful or surveillance use. We send this text only to Typed Decision Models for scoring and release no payloads. The RTP-LX archive stays behind its upstream password. A high moderation skill score does not show that a model is fit for content moderation in any language.
Model access and provider terms. We accessed hosted models through a commercial decision router under the terms of the providers. We sent only public benchmark text and never sent gold labels. We ran the local models (Clef, Clef-flash, Laya and the Qwen backbones) from pinned public checkpoints under their licences.
Intended use. Passing a benchmark does not replace validation of a model or a confidence policy on target-language deployment data. On Core12 XNLI, a 5% threshold fitted in English admits 7.8 to 12.4% errors in German and French.
Compute. Hosted API spend was $78.26 over all runs. Local inference used 41.4 H100 GPU-hours and an estimated 14.1 RTX 4090 hours. The MediaPipe arms used an estimated 57.2 CPU wall-clock hours (Appendix § 17, Table 15).
| Hosted arm | Bench. | Held-out cohort | Explor. | Pilot | Total | In | Out |
| Qwen3.8-27B, hosted | 40.52 | 1.43 | 0.00 | 0.00 | 41.95 | 42.1 | 0.2 |
| Luna | 5.75 | 0.19 | 0.72 | 0.00 | 6.66 | 66.6 | 0.0 |
| Clef | 3.34 | 0.14 | 2.90 | 0.00 | 6.38 | 85.1 | 0.0 |
| Jev | 3.97 | 0.16 | 0.58 | 0.35 | 5.06 | 120.5 | 26.5 |
| Solar | 4.73 | 0.10 | 0.09 | 0.00 | 4.92 | 98.5 | 0.2 |
| Qwen3.5-9B, hosted | 4.11 | 0.14 | 0.24 | 0.00 | 4.49 | 44.4 | 0.2 |
| Clef-flash | 1.72 | 0.07 | 0.48 | 0.00 | 2.27 | 85.7 | 0.0 |
| Kev-4B | 1.76 | 0.06 | 0.11 | 0.00 | 1.93 | 46.0 | 45.6 |
| d1 | 1.49 | 0.05 | 0.16 | 0.00 | 1.70 | 42.6 | 0.0 |
| Decision-1 | 1.47 | 0.06 | 0.11 | 0.00 | 1.64 | 39.1 | 0.2 |
| pplx-decider | 1.08 | 0.04 | 0.14 | 0.00 | 1.25 | 62.6 | 0.2 |
| Mercury | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 63.7 | 0.3 |
| Total | 69.94 | 2.44 | 5.53 | 0.35 | 78.26 | 796.8 | 73.5 |
| Local arm | Device | Hours |
| Clef, local | H100 | 3.9 |
| Clef-flash, local | H100 | 2.2 |
| Qwen3.8-27B, local | H100 | 35.3 |
| Laya | RTX 4090 | 0.9 |
| Qwen3.5-9B, local | RTX 4090 | 13.2 |
| mmBERT (inference) | RTX 4090 | 0.0 |
| MP-EG2 | CPU | 11.4 |
| MP-GLiNER | CPU | 45.8 |
Table 15. Hosted API spend, tokens and local compute hours. Spend in US dollars by run group, and the input and output tokens the providers reported, in millions (top); local compute hours (bottom).
Use of generative AI. Generative AI tools assisted at several stages of preparing this paper, in addition to the translations described in Appendix § 13.§ 3. The authors checked all outputs and take full responsibility for the content, including any errors.
Appendix Conventions
In tables, n.c. means not comparable (the caption gives the reason), n.r. not run and inc. incomplete. Refusals are never excluded and count as failures. Every number in the appendix comes from an analysis report that we will release with the code.
Data and Suites
Sources and Licences
Before sampling, we dropped two MMLU-ProX questions in all languages, one because its answer letter and answer index disagree and one because a translated distractor is empty. Manifests record item identifiers, hashes, selection rules and licence flags. They grant no right to redistribute the texts, and noncommercial or ShareAlike terms carry over NC: noncommercial. SA: ShareAlike.
to any texts rebuilt from them. Each dataset’s original paper or card describes its speaker and annotator populations.
XNLI premise cluster used in Core12 or held-out cohort and share only 27 pair IDs with each other. We therefore compute gaps to English only within datasets that contain English. RTP-LX toxicity asks eight SCORE questions about each of the same 200 source items in 38 aligned locales, one on toxicity with five levels and seven on harm categories with three.
Suite Construction
Figure 2 counts each language once, even when a suite has several regional varieties of it. Counted by variety, MASSIVE has 52 locales, RTP-LX 38 and native moderation 17.
Aligned suites. The Belebele suite asks the same 100 questions in 106 language varieties. The questions come from 54 passage clusters, none of which appear in Core12 or the held-out cohort. MASSIVE uses 117 utterance IDs in all 52 MASSIVE 1.1 locales instead of 120, because one intent has no eligible test row and another has only one. The IDs are disjoint from the Core12 and held-out cohort MASSIVE IDs. The multilingual NLI suite gives equal weight to three datasets, each parallel across its own languages: Core12 XNLI with 12 languages 500 items, AfriXNLI with 18 429 and AmericasNLI with 10 150 and no English. AfriXNLI and AmericasNLI exclude every
Native suites. INCLUDE regional exams samples up to 500 exam questions per language in 44 languages and keeps all questions where fewer exist. Injongo intent classification has three utterances for each of 40 intents in 16 African languages. Its English rendering of the African utterances is only diagnostic, because it is not natively authored English. Native moderation combines HateCheck and Aya red-teaming. HateCheck has 200 N O U L cases per language in 10 varieties, stratified by functionality, and Aya red-teaming has 120 nine-way harm-category CHOICE cases per language in 8 languages. Each dataset is normalised against its own reference before we take the equal-weight mean. HateCheck’s functional tests are partly shared across languages, but its items are authored separately and have no shared IDs, so we treat it as native. BRIGHTER emotion intensity asks one four-level intensity question per annotated emotion (six emotions, or five in English) on 250 texts per language. For SCORE datasets we also report the mean absolute error of the expected level, quadratic-weighted kappa and the ranked probability score.
Diagnostic suites. SIB-205 (205 varieties 50) is drawn from a training split and its overlap with Belebele was removed only by English exact substring (Adelani et al., [2]). Global PIQA has two options in its native split (136 varieties), so well-resourced languages hit a ceiling. Its parallel split (131 varieties, four options) is never pooled with the native one (Chang et al., [11]). The remaining diagnostics include RTP-LX’s two unaligned locales, hbs and sw (400 texts, 3,200 questions, no aligned English) and InjongoIntent English. The others are the described-label variant of Core12 MASSIVE and SIB-200 with 16,896 cases, English dev with 1,000, MultiEURLEX, PAWS-X and XCOPA with 2,600, and MIRACL relevance in three languages with 614.
RTP-LX Gold Labels Across Locales
Agree: exact agreement of the median gold levels. Locale−en: mean signed level difference. Differ 2: share of pairs two or more levels apart.
RTP-LX gives each locale its own three annotators, and the gold is their median, so an aligned item can have different targets in different locales. Table 5 compares each locale’s gold with the English gold for the same source item. Toxicity agreement per locale ranges from 0.23 (Slovak) to 0.58 (Brazilian Portuguese), and locale annotators rate toxicity lower than the English annotators on average. We score against the median of the three released annotations, which differs from the released aggregate label in 92 rows.
| Rubric (levels) | Agree | Locale−en | Differ 2 |
| toxicity (5) | .457 | −0.269 | .167 |
| insult (3) | .655 | +0.076 | .149 |
| bias (3) | .729 | +0.074 | .130 |
| identity attack (3) | .740 | +0.087 | .120 |
| violence (3) | .864 | +0.090 | .058 |
| microaggression (3) | .896 | +0.152 | .048 |
| sexual content (3) | .897 | +0.068 | .039 |
| self-harm (3) | .976 | +0.031 | .013 |
| all rubrics | .777 | +0.039 | .090 |
Table 5. Locale labels often differ from the English gold. They differ most on toxicity. Each row pools the 7,400 aligned pairs of a non-English locale and the English item with the same source index (37 locales 200).
INCLUDE Sample
| Lang | Test rows | Eligible | Frozen |
| ar | 552 | 544 | 500 |
| de | 139 | 138 | 138 |
| es | 550 | 549 | 500 |
| fr | 419 | 419 | 419 |
| hi | 547 | 545 | 500 |
| ru | 552 | 542 | 500 |
| ur | 352 | 343 | 343 |
| vi | 550 | 549 | 500 |
| zh | 545 | 545 | 500 |
| Total | 3,900 |
Table 6. The INCLUDE sample. It caps the items frozen per language and keeps every eligible item where fewer exist. This nine-language subset of the INCLUDE suite serves the native-versus-translated comparison (Appendix C.2).
Model Interfaces
Backbone Prompts
The local readout uses the frozen user template below, wrapped in each backbone’s chat template with the generation prompt on and thinking disabled. {options} lists one option text per line, prefixed by a hyphen, in canonical order.
Read the state and answer the question. Reply
with exactly one of the allowed options, copied
verbatim.
STATE:
{state}
QUESTION:
{instruction}
ALLOWED OPTIONS:
{options}
The line breaks inside the first sentence are only typographic. The hosted readout keeps the state, instruction and option sections verbatim but lists options as <label> <text> and asks for the label. Labels run A–Z, a–z and 0–9, which allows up to 62 options, and each label is a single token in both Qwen tokenizers. The readout decodes one token at temperature 0 with reasoning disabled. A label’s probability is the sum of the returned top-20 probabilities of tokens that equal the label after leading whitespace is stripped. Mass that is not returned is shared equally among labels outside the top 20, and the result is renormalised. Both endpoints are pinned to one provider and quantisation without fallbacks.
Precision Rules and Interface Limits
| Rule | Models | |
| two decimals | 0.01 | Jev, Luna |
| four decimals | 0.005 | Kev-4B; Clef, Clef-flash (API) |
| six decimals | Solar | |
| strict | pplx-decider, d1, Mercury, Decision-1 | |
| fp32 | strict | Clef, Clef-flash (local) |
caption: tbl:7: Acceptance tolerances follow each interface’s precision. Accepted vectors are renormalised, and the native choice may trail the rounded maximum by one rounding unit.
:::
Two-decimal APIs return exact zeros, which leave log loss undefined without a floor. Their ties flatten risk–coverage curves. Solar Decide rejects questions with more than 26 options, which rules out the 60-intent Core12 MASSIVE cases and all of MASSIVE-52 and Injongo. The scorer flags those 52 and 16 cells as capped and scores them as uniform. Laya’s rendered-option cap refuses 372 of 26,460 Core12 cases, all of them MMLU-ProX. Laya also refuses 6,000 described-label MASSIVE cases, 189 Belebele cases in 48 of the 106 languages and 27 cases of the nine-language INCLUDE sample, 25 by the option cap and 2 by input length. GLiNER2.5-Decide reads at most 512 tokens. Its adapter refuses any case whose full question and text exceed that window instead of scoring a silently truncated input. It therefore answers 24,387 of 26,460 Core12 cases, 8,398 of 10,600 Belebele cases, 20,640 of 21,079 INCLUDE cases, 1,719 of 2,040 Injongo cases and 10,896 of 16,896 described-label cases. In 3 Belebele languages (Tibetan, Burmese and Shan) it refuses nearly every case, and the scorer flags those cells as capped. It also refuses 156 of 26,569 Global PIQA cases and answers every other configuration in full. The hosted Clef routes silently truncate inputs at 16,384 tokens, well above our longest case of about 1,960 tokens. Local Clef and Clef-flash use the vendor’s length-bucketed batching. On synthetic test inputs, its probabilities differ from those of single requests by up to 0.006, and no prediction changes. Every other model has full
Model Rankings and Label Regimes
Core12 Scores and Rankings
valid-output coverage on every suite it ran, apart from the cells marked in Table 3. Table 16 counts the failed items and the share of valid answers with an exact-zero probability for every model and suite. The exact zeros come from the two-decimal APIs (Jev, GPT-6 Luna).
Leaving out any one dataset changes the Core12 ranking little (Kendall 0.82–0.96). We report a view without Solar’s capped MASSIVE cells but do not rank it. Intervals on the other suites are wide. Clef-flash has the top MASSIVE-52 score (.745), for example, but its interval overlaps those of the next six models.
English selection under other criteria. The main text counts a cell as a loss for the English pick when the paired loss interval excludes zero (56 of 232 cells). The two models’ own intervals do not overlap in 22 cells, and the loss is at least .10 Brier skill in 47. Within English, the pick’s interval never lies above the runner-up’s, and a paired interval separates them in only one of the six English comparisons (pplx-decider against Mercury), so the English results do not settle the choice. Of the Core12 pick’s paired-significant losses on Belebele, INCLUDE and BRIGHTER, only the INCLUDE loss also has non-overlapping intervals.
Ordinal suites under RPS skill. The ten hosted Typed Decision Models all completed RTP-LX and BRIGHTER. We rank them by ranked probability skill against uniform, macro-averaged over languages, and compare this ranking with the Brier-skill ranking of Table 3. On BRIGHTER the rankings agree (Kendall ). They differ only in three adjacent pairs, where Clef and pplx-decider swap first and second place, Clef-flash and d1 third and fourth, and Kev-4B and Decision-1 seventh and eighth. On RTP-LX . Luna moves from fifth to second, d1 from second to fifth and Clef-flash from first to third, and 8 of 45 model pairs swap. Because the references differ (base rate for Brier skill, uniform for RPS skill), we compare only the orderings.
Native Versus Translated Exams and Described Labels
MMLU-ProX translates English exam questions, whereas INCLUDE collects exams written in each
Table 8: pplx-decider leads Core12. Every model is weakest in Swahili or Hindi. Per-model Core12 Brier skill against uniform chance, rank interval, worst language, gap to English and coverage. Solar’s 12 MASSIVE cells score as uniform (26-option cap), and without them it scores .497. Laya runs a separate Core12 manifest, and mmBERT has no Belebele or MMLU-ProX head, so neither has a suite score (§ 4).
language ([63, 53]). On the nine languages both cover (Appendix § 11.§ 4), the two sets differ in subject mix, option count (four against ten) and difficulty, and INCLUDE items are not parallel. We therefore compare chance-referenced skills within each arm and language and never compare raw accuracy across the two sets. Table 9 gives each arm’s INCLUDE minus MMLU-ProX Brier skill, averaged over the nine languages (3,900 INCLUDE items and 500 MMLU-ProX items per language), with a 95% bootstrap interval. The bootstrap resamples MMLU-ProX parallel groups and INCLUDE items within each language, and all arms share one draw. Only the differences between arms are therefore meaningful. Solar has the lowest gap at −.155 and Luna the highest at +.210, so Luna’s lead on INCLUDE in Table 3 is specific to native exams. Jev does worse on native exams than on translated ones, at −.082. German and Urdu have a negative gap for most arms.
| Model | Native instr. XNLI | Native instr. MASSIVE | Fully native XNLI | Fully native MASSIVE |
| Hosted Typed Decision Models | ||||
| Jev | +.009{sup}*` | −.001 | +.025{sup}*` | −.027{sup}*` |
| pplx-decider | −.002 | +.001 | +.006 | −.024{sup}*` |
| d1 | −.001 | −.016{sup}*` | −.001 | −.016{sup}*` |
| Luna | +.101{sup}*` | −.000 | +.065{sup}*` | −.171{sup}*` |
| Mercury | −.005 | −.001 | −.012{sup}*` | −.041{sup}*` |
| Kev-4B | −.015{sup}*` | +.023{sup}*` | n.r. | n.r. |
| Solar | −.059{sup}*` | – | n.r. | n.r. |
| Clef | −.078{sup}*` | −.010{sup}*` | n.r. | n.r. |
| Clef-flash | −.119{sup}*` | −.021{sup}*` | −.088{sup}*` | −.049{sup}*` |
| Decision-1 | −.046{sup}*` | −.004 | n.r. | n.r. |
| Backbone readout | ||||
| Qwen3.5-9B{sup}◦` | −.021 | +.018 | n.c. | n.c. |
Table 9. The native-exam gap changes sign across arms. Described labels move the skill of Typed Decision Models much less. Change in Brier skill against chance for native against translated exams (NAT) and described against bare labels (DESC).
The described-label diagnostic repeats the 8,448 Core12 MASSIVE and SIB-200 cases with a one-line English description per label, keeping state, gold and option identities fixed. We pair each described case with its original from the same run and report the change in Brier skill, averaged over the twelve languages, with a joint 10,000-replicate bootstrap. On MASSIVE, descriptions raise skill for Kev-4B, Jev, Mercury and Luna by .035 to .053 and lower it slightly for both Clef heads. On SIB-200 every change lies between −.048 and 0, and only the intervals of Decision-1 (−.048) and Laya (−.035) exclude zero. Laya rejects all 6,000 described MASSIVE cases because its option-description cap would truncate them, but answers all 6,000 originals. Solar’s option cap rules out MASSIVE in both variants. The local backbones lose .43 to .61, but their readout scores the whole description sentence in place of the label text, so this loss reflects the changed readout input rather than the label names.
Interface Language (Exploratory)
We registered this ablation before its first request, separately from the held-out cohort pre-registration. On Core12 XNLI and MASSIVE, all 500 items per language in Arabic, German, French, Hindi, Swahili, Thai, Urdu and Chinese get a copy with translated instruction and option texts. Option IDs, states and gold answers are unchanged, and the paired baseline is each model’s existing Core12 run. A second condition replaces the English Estimate followed by its 95% bootstrap interval [lo, hi]. NAT: native INCLUDE minus translated MMLU-ProX, mean over nine languages. DESC: described labels minus bare label identifiers on the same Core12 cases, mean over twelve languages. –: Solar’s 26-option cap rejects MASSIVE. refused: Laya rejects every described MASSIVE case, scored as a failure. n.r.: not run. n.c.: not comparable, because the local readout scores the description text itself rather than the label.
| Arm | MMLU-ProX | MASSIVE | SIB-200 |
| NAT: INCLUDE − | DESC: described − bare | ||
| Typed Decision Models | |||
| pplx-decider | .052 [−.005, .109] | .009 [−.001, .019] | −.011 [−.049, .023] |
| Jev | −.082 [−.136, −.028] | .043 [.020, .066] | .000 [−.028, .025] |
| Clef, API | .095 [.045, .144] | −.021 [−.037, −.006] | −.008 [−.051, .032] |
| Clef, local | .094 [.045, .143] | −.021 [−.037, −.005] | −.008 [−.051, .031] |
| d1 | .102 [.046, .157] | .007 [−.010, .025] | −.022 [−.058, .011] |
| Clef-flash, API | .072 [.024, .120] | −.013 [−.030, .003] | −.006 [−.039, .026] |
| Clef-flash, local | .072 [.024, .120] | −.013 [−.030, .003] | −.006 [−.039, .026] |
| Mercury | .147 [.081, .211] | .039 [.018, .061] | −.021 [−.064, .015] |
| Luna | .210 [.158, .259] | .035 [.014, .056] | −.012 [−.064, .036] |
| Kev-4B | .096 [.059, .132] | .053 [.039, .067] | −.012 [−.034, .010] |
| Solar | −.155 [−.218, −.087] | – | −.035 [−.089, .016] |
| Decision-1 | .109 [.048, .171] | −.013 [−.028, .002] | −.048 [−.084, −.015] |
| Laya | .002 [−.018, .023] | refused | −.035 [−.069, −.0005] |
| Backbone readouts | |||
| Qwen3.8-27B, hosted | .172 [.114, .229] | n.r. | n.r. |
| Qwen3.5-9B, hosted | .182 [.128, .236] | n.r. | n.r. |
| Qwen3.8-27B, local | .152 [.128, .176] | n.c. | n.c. |
| Qwen3.5-9B, local | .129 [.107, .150] | n.c. | n.c. |
Table 3.
Native instr.: translated instruction and option texts, English option IDs. Fully native: translated texts with neutral IDs. Clef and Clef-flash through the API. {sup}◦hosted backbone readout. {sup}*: significant (95% paired cluster-bootstrap interval excludes zero). n.r.: not run. –: Solar’s 26-option cap rejects MASSIVE. n.c.: the hosted readout never shows option IDs.
tbl:10: Native instructions cost the Qwen-based heads most on XNLI. Neutral option IDs cost most on MASSIVE. Change in Brier skill against chance relative to the English-schema Core12 run on the same items, mean over eight languages.
option IDs with neutral ones, in English and in translated form, for six models. We left out Kev-4B, Solar and Decision-1 to stay within budget. claude-opus-5.5 translated the 520 texts from a written brief, and gpt-6.1-sol, blind to the translator, back-translated them and mapped every translated option back to an English ID. All 504 options were recovered, and no review changed a label’s meaning. No native speaker reviewed the translations, and the XNLI state keeps its English premise and hypothesis headers.
Pooled over the Typed Decision Models, the low-minus-high difference on XNLI is .037 [.029, .046] points of Brier skill in favour of the high-web-share group. Removing the English IDs alone lowers MASSIVE skill by .018 [.014, .022], and a synthetic probe confirms that nine of the ten hosted Typed Decision Models follow the IDs when option texts are blank. Only GPT-6 Luna and, slightly, Jev gain on XNLI under native instructions. Clef-flash’s Brier-skill advantage over the hosted Qwen3.5-9B readout shrinks by .099 [.067, .129] on XNLI and by .039 on MASSIVE. The English pick on MASSIVE is wrong in five of eight languages under either schema. Under native instructions its mean loss rises from .014 to .023. The model ranking by language mean agrees with the English-schema ranking at Kendall on XNLI and .78 on MASSIVE. On MASSIVE, the best model changes in three languages, and the English pick’s largest loss, in Swahili, grows from .081 to .106.
Heads Versus Backbones
Gains per Language and Dataset
Web share and the local-readout accuracy gain. Across the twelve Core12 languages, the head’s accuracy gain over its backbone falls as web share rises. Spearman’s is for Clef, for Clef-flash and for pplx-decider (permutation , and ), and the OLS slopes are , and accuracy points per tenfold increase in web share. These correlations are not independent, since Clef and pplx-decider share a backbone and all three pairs share items.
Head–backbone interactions by dataset. Under the local readout, Belebele and MASSIVE have the largest accuracy interactions. On Belebele they are points for Clef, for Clef-flash and for pplx-decider, and on MASSIVE they are , and in the same order. On XNLI the sign depends on the pair, with Clef’s gain smaller in the low-web-share group ( [, ]) and Clef-flash’s larger ( [, ]). For both Clef heads the Brier contrasts on XNLI and MMLU-ProX favour the high-web-share group, and for Clef-flash they outweigh Belebele and MASSIVE.
English Temperature Scaling
Figure 7. Heads keep part of their ECE advantage after English temperature scaling. Clef-flash keeps the least. ECE reduction, backbone minus head, on Core12 MASSIVE and XNLI under the local readout, (a) raw and (b) after applying one temperature per dataset, fitted on English development data, to both the head and its backbone. Bars show 95% paired source-cluster intervals, and shading marks the low-web-share group.
Scaling lowers backbone ECE on MASSIVE (Table 11) from 0.132–0.199 to 0.029–0.088 for Qwen3.8-27B and from 0.124–0.254 to 0.027–0.087 for Qwen3.5-9B. On XNLI it helps much less (0.134–0.347 to 0.116–0.299, and 0.059–0.329 to 0.035–0.283). After scaling, the head’s Brier advantage ranges from to for Clef, to for Clef-flash and to for pplx-decider, and the paired interval excludes zero in all 36 language cells. For ECE, 29 of the 36 intervals lie below zero. The exceptions are Thai for Clef and German, English, Spanish, Swahili, Vietnamese and Chinese for Clef-flash. After scaling, the low-minus-high web-share contrast in the ECE difference is [, ] for Clef, [, ] for pplx-decider and [, ] for Clef-flash. Intervals use 10,000 paired source-cluster replicates.
| Pair | MASSIVE | XNLI | MASSIVE | XNLI | ECE raw lowest to highest | ECE scaled lowest to highest |
| Clef | 0.97 | 0.86 | 0.65 | 1.24 | −.218 to −.118 | −.121 to −.019 |
| Clef-flash | 0.97 | 0.78 | 0.68 | 0.22 | −.236 to +.005 | −.077 to −.003 |
| pplx-decider | 1.20 | 0.92 | 0.65 | 1.24 | −.214 to −.121 | −.130 to −.050 |
Table 11. English temperatures shrink the heads’ ECE advantage. Clef-flash loses the most. One temperature per arm and dataset, MASSIVE/XNLI, fitted by negative log-likelihood on the English development half used for temperatures, for each head () and its backbone’s local readout (). ECE differences are head minus backbone per language, as the mean over MASSIVE and XNLI, ranged over the twelve Core12 languages. On these two datasets raw Clef-flash ECE is higher than its backbone’s in Swahili (+.005).
Wide-Belebele Breadth Check (Exploratory)
We repeat the head–backbone comparison on all 106 Wide-Belebele languages under the local readout, with the same paired bootstrap (Table 12). The low- and high-web-share groups exist only for the twelve Core12 languages, so here web share is the only language variable. The accuracy gain is above zero in 95 languages for Clef, 98 for Clef-flash and 103 for pplx-decider, and no language has an interval below zero. All three heads beat their backbones in 88 languages, and in twelve of them all three intervals lie above zero. The Brier score improves significantly (interval below zero) in 102, 94 and 100 languages and worsens significantly (interval above zero) in zero, one and one. Over the 91 languages with a usable web share, the accuracy gain falls as web share rises for Clef (, ) and pplx-decider (, ), with OLS slopes of −1.0 and −2.4 points per tenfold share. Clef-flash shows no such trend (, ), unlike on Core12. Instead, its Brier difference tracks web share (, ), with larger Brier gains where web share is higher. For Clef and pplx-decider the Brier difference shows no clear trend ( and +.14, and ). With the merged-variety languages added back (), the accuracy correlations are −.39, +.03 and −.54. Each language has only a hundred questions, so single-language intervals are wide. Web shares come from the same crawl as above, recomputed from page counts. Each FLORES code maps to its own crawl (CLD2) code or, if the crawl lists only the macrolanguage, to its ISO 639-3 macrolanguage (swh_Latn to Swahili, nob_Latn to Norwegian). Bambara, Fula, Ilocano, Jingpho and Shan (five languages) have neither and get no web share. We also leave out ten languages whose crawl code also covers a FLORES+ variety without its own code, either a different language or the same language in another script. These are Arabic, Azerbaijani, Chinese, Kurdish, Latvian, Malay, Mongolian, Persian, Sindhi and Uzbek. The full mapping will be released with the code.
| Head | Gain>0 | CI>0 | Median gain | Median share | with web share () | Slope |
| Clef | 95 | 28 | 4.0 | .26 | −.41 () | −1.0 |
| Clef-flash | 98 | 44 | 7.0 | .31 | +.02 () | +0.3 |
| pplx-decider | 103 | 50 | 5.5 | .40 | −.54 () | −2.4 |
Table 12. Every head beats its backbone in most Wide-Belebele languages. Only Clef’s and pplx-decider’s gains fall with web share. Wide-Belebele breadth check (exploratory, local readout): each head against its own backbone in 106 languages.
Second Head–Backbone Pair (Exploratory)
Microsoft states that Decision-1 is post-trained from Qwen3.5-9B, which we cannot verify. We added the comparison after the held-out cohort pre-registration, so it is exploratory. Among the remaining hosted models, only Decision-1 has a declared backbone available with log-probabilities through our provider. We did not run Qwen3.5-4B, the public base of Kev-4B. We analyse the pair with the estimands and bootstrap of the three main pairs. Unlike them, its registration names the hosted readout of Qwen3.5-9B as primary and the local readout as sensitivity, and we report both. Clef-flash shares the backbone, so the two pairs are not independent. On Core12, the low-minus-high accuracy interaction is 6.8 [5.7, 8.0] accuracy points with the local backbone readout and 4.5 [3.4, 5.6] with the hosted one, and the gain’s Spearman correlation with log web share is −.83 for both. The low-minus-high difference in the share of backbone errors removed, , is +.037 [+.006, +.067] locally and −.013 [−.043, +.017] through the API. On the held-out cohort the local and hosted are 7.1 [4.2, 10.1] and 5.0 [2.3, 7.8], with = −.001 [−.063, +.065] and −.025 [−.090, +.044]. On Wide-Belebele, the head’s accuracy gain has an interval above zero in 90 of 106 languages under the local readout (72 hosted) and below zero in none, with a median gain of 11.0 points.
Gain>0: languages where the head’s accuracy is higher. CI>0: languages whose 95% paired source-cluster interval lies above zero. Median gain: in accuracy points. Median share: share of backbone errors removed. : Spearman correlation of the gain with log web share over 91 languages, with permutation . Slope: OLS, accuracy points per tenfold web share.
Confidence-Policy Transfer
Table 13 applies the English-fitted policies of every hosted model with English development runs to Core12 XNLI and MASSIVE. For models with exact zeros, we floor probabilities before temperature fitting, at in the primary analysis. Across Jev, d1 and pplx-decider, both datasets and twelve languages, a floor of moves the realised risk at the 5% threshold by at most 2.2 points and conformal coverage by at most 2.6. With the largest shifts are 0.6 and 4.6 points. Rounding every model to two decimals before fitting (mass-preserving, available for Jev and d1) changes risk and coverage by no more than 0.6 and 1.2 points. The XNLI conformal sets also become empty for up to 13% of decisions in a non-English language, against at most 4% in English.
Figure 5 omits the cells that cannot be assessed. Luna’s English XNLI threshold admits no decision and Solar’s admits 1 to 4% per language, so neither is drawn on XNLI, and Solar has no MASSIVE fit.
Held-out Cohort Results
Confirmatory tests. The paired stratified bootstrap draws 10,000 replicates over 282 parent groups. Under the local readout, pooled is 4.8 accuracy points, with a one-sided bound of 2.8 at the Holm level (, tested at .025). Pooled is −.034, with a 90% interval of [−.073, .006] (TOST , tested at .05). Under the hosted readout, which is exploratory, is 2.7 points (bound 0.9, ) and is −.039 [−.084, .006], with TOST , so equivalence is not shown.
Per pair (exploratory, local readout). The accuracy interaction is 4.8 [2.3, 7.3] points for Clef, 4.7 [1.7, 7.7] for Clef-flash and 4.9 [2.0, 7.8] for pplx-decider. The difference in the share of backbone errors removed is −.006 [−.069, .060], −.045 [−.117, .028] and −.050 [−.114, .016]. For the Brier gain, the low–high interaction is +.010 [−.013, .032], −.028 [−.057, .000] and +.039 [.016, .063]. The ECE gain is smaller in the low-web-share group for all three pairs. The interactions are −.045 [−.071, −.016] for Clef, −.106 [−.132, −.070] for Clef-flash and −.025 [−.057, −.001] for pplx-decider. Across the twelve languages, the Spearman correlation of the accuracy gain with log web share is −.77 for Clef, −.68 for Clef-flash and −.51 for pplx-decider, with permutation values of .005, .018 and .093.
Skill ranking and policy transfer (exploratory). On the held-out cohort, Brier skill against chance runs from −.214 to .730 over all complete arms, including backbones and the two MediaPipe arms. The MediaPipe embedding arm scores −.214, and GLiNER scores .010 at 91% coverage, missing only cases refused by the 512-token window. Over the ten Typed Decision Models, the held-out ranking agrees with Core12 at Kendall , with pplx-decider first on both. Decision-1 was registered after the pre-registration, so its held-out rows are exploratory. We also apply the English-fitted 5% thresholds unchanged to held-out XNLI. For the six assessable models, risk is 1.3–3.6% in English and 8.4–13.2% in German and French. Conformal coverage in the low-web-share group is .65–.75 against a target of .90.
Reproducibility and Compute
Fixed artefacts. The statistical protocol, manifests and model registrations are identified by SHA-256 hashes in the release, and Table 14 gives the order in which they were fixed. Every manifest rebuilds identical case bytes from pinned source revisions (Table 4). Each suite score comes with a signature that records the suite version and hash, manifest hashes, scorer commit, reference, floor, refusal and bootstrap settings [49]. Hosted
| Source | Revision | Licence (XDecideBench treatment) | Provenance of non-English text |
| MASSIVE 1.1 | ff6bd8e | CC-BY-4.0 | Localized from English SLURP utterances |
| XNLI | b8dd5d7 | CC-BY-NC-4.0 | Translated MultiNLI-style pairs; labels propagated |
| Belebele | 7899cdf | CC-BY-SA-4.0 | Questions on FLORES-200 passages, translated |
| MMLU-ProX (full) | 8e6106a | MIT (declared) | Machine translation of MMLU-Pro with expert review |
| SIB-200 | 38977a6 | CC-BY-SA-4.0 | English topic labels propagated through FLORES-200 |
| INCLUDE | d2e1f60 | Apache-2.0 (card only) | Native regional exams, not translated |
| AfriXNLI | e3ca06b | card Apache-2.0 (NC: XNLI-derived) | Translations of XNLI 1.0 test sentences |
| AmericasNLI | 1f3f4fa | card CC-BY-SA-4.0 (NC, SA) | Translations of XNLI 1.0 test sentences |
| RTP-LX | 805763e | MIT/Apache-2.0 + NOTICE (NC; SA for he, ko, pt-BR) | Aligned multilingual toxicity prompts with native components; extremely toxic |
| BRIGHTER | 08a5d61 | CC-BY-4.0 | Natively authored texts with emotion-intensity labels |
| Multilingual HateCheck | a03f2ca | CC-BY-4.0 | Functional tests authored separately per language; hateful by design |
| Aya red-teaming | 5a16fee | Apache-2.0 (card) | Natively written harmful prompts |
| InjongoIntent | fe4be38 | conflicting: CC-BY-4.0 / Apache-2.0 / GPL-3.0 | Natively collected intent utterances |
| Global PIQA | 6777742, b0b1851 | CC-BY-SA-4.0, no training | Native split written per variety; parallel split translated |
Table 4. Several sources have noncommercial or ShareAlike terms. Sources of the evaluation suites and Global PIQA, with pinned revisions (abbreviated), licences as declared upstream and as XDecideBench treats them, and provenance. MultiEURLEX, PAWS-X, XCOPA and MIRACL enter only diagnostic suites. Arabic in Core12 is Modern Standard Arabic and Chinese is simplified script.
High: mean over German, Spanish, French, Russian and Chinese. Low: mean over Hindi, Swahili, Thai and Urdu. Risk: error rate among admitted decisions, with the admitted share of the 500 decisions per language in parentheses. –: no decision admitted. Coverage counts all scheduled decisions, and size is the mean prediction-set size.
models are reached through one decision router and one generic adapter, and a response is accepted only if it names the registered dated snapshot. Repeat runs and hosted–local agreement. The described-label runs re-send the 8,448 Core12 MASSIVE and SIB-200 cases with unchanged case identifiers and input hashes on the same day. Against the headline answers, top-1 agreement is 1.000 for pplx-decider and Luna, .9995 for Clef, .9996 for Clef-flash, .997 for Mercury, .995 for Kev-4B, .990 for d1, .989 for Decision-1, .980 for Jev and .965 for Solar (SIB-200 only). Dataset Brier skill changes by at most .0024 (Solar). Decision-1’s API is not deterministic. Twelve identical probe requests spread by up to .03 on one option, and only 10% of its repeated probability vectors are identical, yet it keeps 98.9% of its top answers and moves dataset skill by at most .0002. The 99th percentile of the largest per-case probability difference is .005 for both hosted Clef heads, at most .148 for the other models and .973 for Solar. Hosted and local Clef agree on the top answer for 99.14% of the 26,460 Core12 cases and Clef-flash for 99.48%, with median largest probability difference .0014 for both. Eight fixed probe requests sent at the start and end of the hosted runs differ by at most .032, for Decision-1, whose probes bracket all its suites. Solar’s differ by up to .016. The hosted Clef routes have start probes only, pplx-decider’s end probes precede its last Core12 attempt, and no probes bracket the other models’ runs on the other eight suites.
Bootstrap settings. The head–backbone analysis uses 10,000 paired replicates, and suite scores use 1,000 joint replicates at the 95% level, both with seed 20261004. Transfer intervals use the same number of replicates and seed. Per dataset, one draw of source clusters (500 MASSIVE utterances, 455 XNLI premises) is shared by English, every target language and every model, and the English fit is held fixed. Resampling each language independently would understate the standard error of Jev’s Core12 Brier skill about threefold (0.0037 against 0.0110). SIB-200 and Belebele share text. Of the 204 SIB English test sentences, 176 occur verbatim in a Belebele passage, and 107 of them appear in 87 of the 276 passages of the Core12 Belebele sample. The shipped bootstrap resamples the two datasets independently. Resampling each linked passage together with its SIB sentences as one parent, with 2,000 replicates, changes the standard error of Core12 Brier skill by a factor of 0.96 to 1.02 across twelve models. These are local Clef and Clef-flash, the two Qwen backbones and ev-
| Time (UTC, 2026) | Event |
| 10-04 00:56 | Statistical protocol frozen, with accuracy as the primary endpoint |
| 10-04 01:23–03:58 | Core12 and English development manifests |
| 10-06 20:25 | Held-out cohort manifest frozen |
| 10-07 14:42 | Described-label manifest |
| 10-07 14:43–10-08 20:41 | Model registrations (Laya’s on 10-04), each before that model’s first benchmark request |
| 10-07 14:44–10-08 20:38 | Core12 inference |
| 10-07 20:14 | Head–backbone analysis code committed |
| 10-08 01:23 | First pair with both Core12 runs complete |
| 10-08 16:04–17:04 | Manifests of the other eight suites |
| 10-08 17:10 | Suite scoring contract: Brier skill primary, breadth manifest hashes pinned |
| 10-08 19:18–20:38 | Hosted Clef, Clef-flash and Qwen Core12 arms |
| 10-08 19:18–10-09 00:09 | Inference on the other eight suites |
| 10-08 22:58–23:00 | Final English temperature and transfer fits |
| 10-09 12:00 | Held-out cohort pre-registration, before the first held-out cohort request |
| 10-10 10:20 | Decision-1 registered, after the held-out cohort pre-registration |
| 10-10 10:20–15:34 | Decision-1 inference on every suite and the held-out cohort |
caption: tbl:14: Protocol, manifests and registrations were fixed before the requests they cover. Chronology from git commits, manifest creation times and run logs. Laya’s Core12 manifest (10-08 08:20) has the same case identifiers and input hashes as the 10-04 manifest, and Laya’s breadth manifests date from 10-08 19:27. Some registrations followed synthetic probe and smoke requests.
:::
ery hosted Typed Decision Model except the two hosted Clef routes. For Jev the interval moves from [.6664, .7102] to [.6655, .7076].
Power of the held-out tests. Our power simulation (10,000 draws) takes, per task, as many Core12 source clusters as the held-out cohort has parent groups, 282 in total, and shares one draw among the three pairs. Two of the pairs share the Qwen3.8-27B backbone and all share items, so the pooled estimate is one correlated test, with standard error .030 for H2. H2 uses two one-sided tests at per side, which needs the 90% interval inside . Its power is 70% at a true difference of zero and 62% at the Core12 estimate. A margin of would give 80% power, and a margin of almost none. Under Holm, H2 is tested at .05 once H1 has been rejected. If H2 has the smaller -value and is tested first at .025, its power at zero is 53%. H1 has 98% power at the Core12 effect, or 95% at .025. Per pair, H2 power at is only 23 to 35%, so we report per-pair values only descriptively. At the low-web-share group’s Core12 backbone error rates, .354 for Qwen3.8-27B and .410 for Qwen3.5-9B, the margin corresponds to about 2.8 and 3.3 accuracy points.
Scoring a new model. One command, xdecidebench eval --model model, scores a new model on the benchmark. It rebuilds the cases of every suite the chosen tracks need from the pinned sources, runs the model, audits the runs and writes a report per suite with Brier skill, valid-output coverage and a 95% interval under a signature that fixes how it was computed. It then compares the scores with the published models. The model can be a hosted route, an OpenAI-compatible server, a local Hugging Face checkpoint or a custom adapter that returns a full probability vector for each scheduled question. Every case is scored, including refusals. Guides for scoring a new model and extending the benchmark will be released with the code at https://github.com/NaiveNeuron/XDecideBench.
Compute and cost. Hosted API spend over every run in the run stores is 69.94 for the benchmark suites, 5.53 for the exploratory interface-language ablation and $0.35 for pilots (Table 15). The ten hosted Typed Decision Models account for 6.66 each, and the two hosted Qwen readouts for $46.44. Per attempt, the cost is the billed total of the provider’s generation records where these were fetched, and otherwise the cost the response reported. The providers reported 797M input and 73M output tokens. Probe requests are not in the run stores and are not counted. Local Clef, Clef-flash and the Qwen3.8-27B backbone took 41.4 H100 GPU-hours, summed from per-run wall-clock times in the run receipts. For one Qwen3.8-27B run, this is end-to-end time including an interruption. On one RTX 4090, Laya, the Qwen3.5-9B backbone and mmBERT inference took an estimated 14.1 hours and mmBERT fine-tuning 2.4 hours, and the two MediaPipe arms took an estimated 57.2 wall-clock hours on CPU. The RTX 4090 inference and CPU figures are estimated from attempt start times with gaps over ten minutes removed, since usage was not metered.
H100 hours from run receipts; RTX 4090 inference and CPU hours estimated from attempt start times (gaps over ten minutes removed). Mercury was served at no charge.
Additional Result Tables
Reading the full-results tables. Table 16 to Table 18 report failures, suite scores and per-dataset scores for fourteen arms of Table 3: the ten hosted Typed Decision Models, the hosted readouts of the two backbones and the two MediaPipe comparators. They use the groups, order and short names of that table, with hosted Typed Decision Models ordered by Core12 Brier skill. Failures are scored as uniform. Bold marks the best hosted Typed Decision Model in each column and underline the runner-up, by point estimate. Uncertainty is printed in grey at the size of the estimate: an interval in brackets is a 95% interval, never a half-width. Cells without a score are grey (–: no value). Table 17 gives each suite score under both refusal views, with the constant-predictor baseline, and Table 18 the per-dataset skills of the multi-dataset suites. Per-language scores of every arm on every suite, the per-language head–backbone differences and the per-language policy-transfer results will be released with the code.
| Model | Aligned, shared labels | Locale labels | Native-authored | ||||||
| Core12 | Belebele | MASSIVE | NLI | RTP-LX | INCLUDE | Injongo | Moder. | BRIGHTER | |
| Hosted Decision Models | |||||||||
| pplx-decider | .712 | _.756_ | .718 | .397 | .051 | .536 | .578 | .673 | .144 |
| Jev | _.688_ | .785 | .619 | .128 | −.101 | _.531_ | _.499_ | _.561_ | −.428 |
| Clef | .674 | .701 | .714 | .254 | .045 | .459 | .492 | .274 | .167 |
| d1 | .663 | .711 | .634 | .315 | .070 | .488 | .481 | .449 | .118 |
| Decision-1 | .651 | .716 | _.721_ | _.108_ | −.229 | .400 | .417 | .500 | −.375 |
| Clef-flash | .631 | .597 | .745 | .127 | .071 | .385 | .415 | .379 | .123 |
| Mercury | .611 | .677 | .651 | .188 | −.022 | .411 | .416 | .449 | .013 |
| Luna | .568 | .743 | .602 | −.058 | −.018 | .600 | .580 | .541 | −.096 |
| Kev-4B | .449 | .389 | .371 | .070 | −.169 | .273 | .225 | .278 | −.275 |
| Solar | .398 / .497 | .334 | .000 | −.236 | −.568 | .251 | .000 | −.046 | −1.396 |
| Backbone readouts, hosted | |||||||||
| Qwen3.8-27B | .584 | .678 | .523 | .043 | −.336 | .426 | .405 | .397 | −.056 |
| Qwen3.5-9B | .477 | .525 | .313 | −.036 | −.562 | .310 | .167 | .347 | −.034 |
| Comparators | |||||||||
| MP-EG2 | −.084 | −.284 | .347 | −.997 | −1.363 | −.242 | .054 | −.004 | −1.166 |
| MP-GLiNER | .057 | −.066 / −.068 | .029 | .010 | −.984 | −.034 | .032 | .076 | −.807 |
| Baseline | .013 | .024 | .000 | .001 | .000 | .007 | .000 | .001 | .000 |
Table 17. After a slash: the sensitivity view that drops capped cells, where any are dropped. Baseline: the constant predictor that answers every item with its cell’s label prior.
Table 17: Suite Brier skill under both refusal views. Each suite’s skill against its declared reference with failures scored as uniform, and the constant-predictor baseline.
| (a) | Core12 | ||||
| Model | MASSIVE | XNLI | Belebele | MMLU-ProX | SIB-200 |
| Hosted Typed Decision Models | |||||
| pplx-decider | .766 [.729, .803] | .663 [.619, .700] | .887 [.861, .910] | .450 [.402, .498] | .794 [.735, .844] |
| Jev | .651 [.608, .693] | .482 [.424, .536] | .883 [.851, .911] | .584 [.535, .634] | .839 [.783, .888] |
| Clef | .776 [.739, .811] | .572 [.525, .615] | .853 [.819, .880] | .363 [.318, .404] | .808 [.749, .868] |
| d1 | .712 [.676, .748] | .601 [.550, .645] | .856 [.824, .883] | .358 [.313, .406] | .786 [.733, .834] |
| Decision-1 | .766 [.731, .800] | .447 [.389, .500] | .922 [.903, .938] | .263 [.213, .312] | .857 [.816, .895] |
| Clef-flash | .774 [.735, .811] | .493 [.443, .538] | .786 [.751, .816] | .301 [.261, .340] | .801 [.748, .853] |
| Mercury | .684 [.640, .728] | .501 [.441, .556] | .855 [.821, .885] | .254 [.194, .312] | .764 [.689, .829] |
| Luna | .608 [.570, .643] | .243 [.168, .310] | .854 [.820, .884] | .326 [.282, .372] | .809 [.748, .864] |
| Kev-4B | .434 [.407, .458] | .397 [.337, .459] | .471 [.416, .526] | .182 [.154, .212] | .763 [.716, .808] |
| Solar | .000 [.000, .000] | .164 [.082, .244] | .683 [.638, .721] | .392 [.339, .445] | .750 [.684, .812] |
| Backbone readouts, hosted | |||||
| Qwen3.8-27B | .592 [.545, .637] | .435 [.370, .493] | .840 [.805, .870] | .238 [.190, .288] | .817 [.757, .872] |
| Qwen3.5-9B | .441 [.395, .486] | .278 [.211, .339] | .725 [.686, .759] | .125 [.081, .167] | .814 [.759, .865] |
| Comparators | |||||
| MP-EG2 | .519 [.486, .555] | −.996 [−1.112, −.876] | −.280 [−.327, −.233] | −.044 [−.060, −.029] | .379 [.321, .442] |
| MP-GLiNER | .132 [.110, .154] | .033 [.015, .052] | −.060 [−.097, −.020] | −.068 [−.086, −.051] | .248 [.191, .308] |
| (b) | NLI | Moder. | |||
| Model | XNLI | AfriXNLI | AmericasNLI | HateCheck | Aya red-teaming |
| Hosted Typed Decision Models | |||||
| pplx-decider | .663 [.619, .700] | .436 [.395, .476] | .094 [.057, .132] | .797 [.767, .823] | .549 [.513, .585] |
| Jev | .482 [.424, .536] | .254 [.204, .303] | −.351 [−.401, −.295] | .596 [.561, .629] | .527 [.487, .570] |
| Clef | .572 [.525, .615] | .298 [.252, .341] | −.109 [−.143, −.078] | .018 [−.030, .073] | .531 [.498, .570] |
| d1 | .601 [.550, .645] | .374 [.332, .414] | −.030 [−.068, .005] | .413 [.373, .452] | .486 [.451, .524] |
| Decision-1 | .447 [.389, .500] | .127 [.081, .171] | −.251 [−.296, −.205] | .509 [.462, .552] | .492 [.458, .532] |
| Clef-flash | .493 [.443, .538] | .140 [.095, .183] | −.252 [−.285, −.218] | .271 [.218, .323] | .487 [.452, .526] |
| Mercury | .501 [.441, .556] | .234 [.182, .282] | −.171 [−.220, −.124] | .418 [.373, .468] | .479 [.434, .524] |
| Luna | .243 [.168, .310] | .153 [.090, .206] | −.569 [−.628, −.501] | .605 [.555, .648] | .478 [.438, .520] |
| Kev-4B | .397 [.337, .459] | .063 [.011, .110] | −.251 [−.297, −.206] | .162 [.117, .200] | .393 [.372, .417] |
| Solar | .164 [.082, .244] | −.345 [−.381, −.307] | −.527 [−.554, −.498] | −.406 [−.518, −.307] | .315 [.262, .373] |
| Backbone readouts, hosted | |||||
| Qwen3.8-27B | .435 [.370, .493] | .156 [.101, .206] | −.461 [−.521, −.396] | .293 [.245, .347] | .500 [.461, .540] |
| Qwen3.5-9B | .278 [.211, .339] | .000 [−.046, .045] | −.387 [−.429, −.344] | .250 [.210, .296] | .444 [.404, .484] |
| Comparators | |||||
| MP-EG2 | −.996 [−1.112, −.876] | −.998 [−.998, −.998] | −.998 [−.998, −.998] | −.160 [−.205, −.124] | .151 [.139, .165] |
| MP-GLiNER | .033 [.015, .052] | .002 [−.003, .006] | −.005 [−.013, .003] | .002 [−.035, .037] | .150 [.126, .175] |
Table 18. Table 18: Per-dataset Brier skill in the multi-dataset suites. (a) The five Core12 datasets; (b) the datasets of the NLI and native moderation (Moder.) suites, each against its own reference.
References
- [1]Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. 2024. The multilingual alignment prism: Aligning global and local preferences to reduce harm. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12027–12049, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2406.18682
- [2]David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Hao-nan Gao, and En-Shiun Annie Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 226–245, St. Julian’s, Malta. Association for Computational Linguistics.
- [3]David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, et al. 2025. IrokoBench: A new benchmark for African languages in the age of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2732–2757, Albuquerque, New Mexico. Association for Computational Linguistics.
- [4]Kabir Ahuja, Sunayana Sitaram, Sandipan Dandapat, and Monojit Choudhury. 2022. On the calibration of massively multilingual language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4310–4323, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- [5]Anastasios N. Angelopoulos and Stephen Bates. 2023. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591.
- [7]Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross-lingual transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674–7684, Online. Association for Computational Linguistics.DOI
- [8]Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics.
- [9]Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020. SLURP: A spoken language understanding resource package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7252–7262, Online. Association for Computational Linguistics.arxiv.org/abs/2011.13205
- [11]Tyler A. Chang, Catherine Arnett, Abdelrahman Sadallah, et al. 2025. Global PIQA: Evaluating commonsense reasoning across 100+ languages and cultures. https://arxiv.org/abs/2510.24081. Preprint, arXiv:2510.24081.
- [12]Cloudflare. 2026. Introducing Clef: Cloudflare’s first open-source decision models, now on Workers AI. https://developers.cloudflare.com/changelog/post/2026-10-01-clef-workers-ai/. Changelog, 1 October 2026; model cards https://huggingface.co/Cloudflare/clef and https://huggingface.co/Cloudflare/clef-flash. Accessed 2026-10-08.
- [13]Common Crawl. 2026. Statistics of Common Crawl monthly archives: Distribution of languages. https://commoncrawl.github.io/ cc-crawl-statistics/plots/languages. Crawl CC-MAIN-2026-39; primary document language by CLD2. Accessed 2026-10-08.
- [14]Alexis Conneau, Ankur Bapna, Yu Zhang, et al. 2022. XTREME-S: Evaluating cross-lingual speech representations. In Proc. Interspeech 2022, pages 3248–3252.arxiv.org/abs/2203.10752
- [15]Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.arxiv.org/abs/1809.05053
- [16]Convai Innovations. 2026. Laya model card. https://huggingface.co/convaiinnovations/laya. Benchmark report: https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md. Accessed 2026-10-08.
- [17]Adrian de Wynter, Ishaan Watts, Tua Wongsangaroonsri, et al. 2025. RTP-LX: Can LLMs evaluate toxicity in multilingual scenarios? In Proceedings of the AAAI Conference on Artificial Intelligence.
- [18]Tobias Deußer, Rajkumar Ramamurthy, Lorenz Sparrenberg, and Rafet Sifa. 2026. Evaluating and bench-marking the system one model. https://arxiv.org/abs/2609.37647. Preprint, arXiv:2609.37647. ArXiv preprint, v1, 29 September 2026.
- [19]Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, et al. 2022. AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6279–6299, Dublin, Ireland. Association for Computational Linguistics.
- [20]Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, et al. 2025. MMTEB: Massive multilingual text embedding benchmark. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2502.13595
- [21]Jack Fitzgerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2023. MASSIVE: A 1M-example multilingual natural language understanding dataset with 51 typologically-universe languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4277–4302, Toronto, Canada. Association for Computational Linguistics.
- [22]Shuyu Gan, Young-Jun Lee, and Dongyeop Kang. 2026. SanSi: A looped type decision model for system 1.5 thinking. https://arxiv.org/abs/2610.07730. Preprint, arXiv:2610.07730. ArXiv preprint.
- [23]Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30, pages 4878–4887. Curran Associates, Inc.arxiv.org/abs/1705.08500
- [24]Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378.
- [25]Wenhan Han, Yifan Zhang, Zhixun Chen, Binbinliu, Mykola Pechenizkiy, Meng Fang, and Yin Zheng. 2026. MuBench: Assessment of multilingual capabilities of large language models across 61 languages. In Findings of the Association for Computational Linguistics: ACL 2026, pages 16163–16192, San Diego, California, United States. Association for Computational Linguistics.arxiv.org/abs/2506.19468
- [26]Anas Himmi, Ekhine Irurozki, Nathan Noiry, Stephan Clémençon, and Pierre Colombo. 2023. Towards more robust NLP system evaluation: Handling missing scores in benchmarks. https://arxiv.org/abs/2305.10284. Preprint, arXiv:2305.10284.
- [27]Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70.DOI
- [28]Jerry Huang, Peng Lu, Qiuhao Zeng, Yusuke Iwasawa, Yutaka Matsuo, Sarath Chandar, Edison Marrese-Taylor, and Irene Li. 2026. Investigating the multilingual calibration effects of language model instruction tuning. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–59, Rabat, Morocco. Association for Computational Linguistics.
- [29]Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, and Fei Yuan. 2025. BenchMAX: A comprehensive multilingual evaluation suite for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 16751–16774.
- [30]Dieuwke Hupkes and Nikolay Bogoychev. 2025. MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages. https://arxiv.org/abs/2504.10356. Preprint, arXiv:2504.10356.
- [31]Inception. 2026. Mercury decide. https://openrouter.ai/inception/mercury-decide. Structured decision model served as a System One endpoint; snapshot inception/mercury-decide-26060930. Accessed 2026-10-08.
- [32]Zhengping Jiang, Anqi Liu, and Benjamin Van Durme. 2022. Calibrating zero-shot cross-lingual (un-)structured predictions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2648–2674, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.DOI
- [33]Pratik Joshi, Sebastian Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.arxiv.org/abs/2004.09095
- [34]Jiawei Li. 2026. Fast models, slow evidence: A paired and self-audited evaluation of System-1 decision models for LLM agent harnesses. https://arxiv.org/abs/2610.02267. Preprint, arXiv:2610.02267. ArXiv preprint.
- [35]Percy Liang, Rishi Bommasani, Tony Lee, et al. 2023. Holistic evaluation of language models. Transactions on Machine Learning Research.DOI
- [36]Liquid AI. 2026. d1: Liquid AI’s structured decision model. https://openrouter.ai/liquid/d1. Served as a System One endpoint via OpenRouter; snapshot liquid/d1-20260930. Accessed 2026-10-08.
- [37]Jiawen Lu and Tongtong Wu. 2026. Benchmarking candidate coverage and rejection policy transfer in typed decision models. https://arxiv.org/abs/2610.03387. Preprint, arXiv:2610.03387. ArXiv preprint.
- [39]Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmBERT: A modern multilingual encoder with annealed language learning. Preprint, arXiv:2509.06888.arxiv.org/abs/2509.06888
- [40]Microsoft. 2026. Microsoft-decision-1: Our model for fast decision-making. https://commandline.microsoft.com/microsoft-decision-1-model-foundry/. Decision-scoring model in Microsoft Foundry and on OpenRouter; snapshot microsoft/microsoft-decision-1-20261009. Accessed 2026-10-10.
- [41]Evan Miller. 2024. Adding error bars to evals: A statistical approach to language model evaluations. https://arxiv.org/abs/2411.00640. Preprint, arXiv:2411.00640.
- [42]Shamsuddeen Hassan Muhammad, Nedjma Ousidoum, Idris Abdulmumin, et al. 2025. BRIGHTER: BRIdging the gap in human-annotated textual emotion recognition datasets for 28 languages. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8895–8916, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2502.11926
- [45]Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, Kelechi Ogueji, Jimmy Lin, Pontus Stenetorp, and David Ifeoluwa Adelani. 2025. AfroBench: How good are large language models on African languages? In Findings of the Association for Computational Linguistics: ACL 2025, pages 19048–19095.
- [46]OpenAI. 2026. Decisions API (public beta). https://developers.openai.com/api/docs/guides/decisions. Accessed 2026-10-08.
- [47]Jared Palmer. 2026. Kev: A Jev-like family of decision models built on top of Qwen3.5/3.8. https://github.com/jaredpalmer/kev. Kev 1.0 release; model cards at https://huggingface.co/jaredpalmer/kev-0.8b. Accessed 2026-10-08.
- [48]Perplexity AI. 2026. Decider v1.1 27b (pplx-decider-v1.1-27b). https://openrouter.ai/perplexity/pplx-decider-v1.1-27b. Predecessor model card: https://huggingface.co/perplexity-ai/pplx-decider-v1-27b. Accessed 2026-10-08.
- [49]Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.arxiv.org/abs/1804.08771
- [51]Qwen Team. 2026b. Qwen3.8-Max: A new bar for coding and cowork. https://qwen.ai/blog?id=qwen3.8. Model card: https://huggingface.co/Qwen/Qwen3.8-27B. Accessed 2026-10-08.
- [52]Amir Rafe and Subasish Das. 2026. Benchmarking system one decision models against trained classifiers and language models for automated decision gates. https://arxiv.org/abs/2610.00346. Preprint, arXiv:2610.00346. ArXiv preprint.
- [53]Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Sneha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, and 40 others. 2024. INCLUDE: Evaluating multilingual language understanding with regional knowledge. Preprint, arXiv:2411.19799.arxiv.org/abs/2411.19799
- [54]Paul Röttger, Haitham Seelawi, Debora Nozza, Zeerak Talat, and Bertie Vidgen. 2022. Multilingual Hate-Check: Functional tests for multilingual hate speech detection models. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 154–169, Seattle, Washington (Hybrid). Association for Computational Linguistics.
- [56]Rahul Sharma, Andrew B. Duncan, Gaétan Marceau Caron, and Sebastian J. Vollmer. 2026. TypedBench: A benchmark for calibration, framing sensitivity, and cost in system one decision models. https://arxiv.org/abs/2610.11392. Preprint, arXiv:2610.11392. ArXiv preprint.
- [57]Shivalika Singh, Angelika Romanou, Clémentine Fourrier, et al. 2025. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics.
- [58]Yosephine Susanto, Adithya Venkatadri Hulagadri, Jann Railey Montalan, Jian Gang Ngui, Xianbin Yong, Wei Qi Leong, Hamsawardhini Rengararajan, Peerat Limkonchotiwat, Yifan Mai, and William Chandra Tjhi. 2025. SEA-HELM: South-east Asian holistic evaluation of language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12308–12336, Vienna, Austria. Association for Computational Linguistics.
- [60]TypeSafe AI. 2026. Introducing system one models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev. TypeSafe AI Blog; API docs at https://docs.typesafe.ai/api.md. Accessed 2026-10-08.
- [61]Upstage. 2026. Solar decide. https://openrouter.ai/upstage/solar-decide. Structured decision model served as a System One endpoint; snapshot upstage/solar-decide-20260928. Accessed 2026-10-08.
- [62]Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.arxiv.org/abs/1704.05426
- [63]Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, and 13 others. 2025. MMLU-ProX: A multilingual benchmark for advanced large language model evaluation. Preprint, arXiv:2503.10497.arxiv.org/abs/2503.10497
- [64]Boyang Xue, Hongru Wang, Rui Wang, Sheng Wang, Zezhong Wang, Yiming Du, Bin Liang, Wenxuan Zhang, and Kam-Fai Wong. 2025. MlingConf: A comprehensive study of multilingual confidence estimation on large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 2535–2556.arxiv.org/abs/2402.13606
- [65]Yahan Yang, Soham Dan, Dan Roth, and Insup Lee. 2024. On the calibration of multilingual question answering LLMs. https://arxiv.org/abs/2311.08669v2. Preprint, arXiv:2311.08669. ArXiv preprint, v2, 15 April 2024.
- [67]Hao Yu, Jesujoba Oluwadara Alabi, Andiswa Bukula, et al. 2025. INJONGO: A multicultural intent detection and slot-filling dataset for 16 African languages. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9429–9452, Vienna, Austria. Association for Computational Linguistics.
- [68]Ej Zhou, Caiqi Zhang, Tiancheng Hu, Chengzu Li, Nigel Collier, Ivan Vulić, and Anna Korhonen. 2025. Beyond the final layer: Intermediate representations for better multilingual calibration in large language models. https://arxiv.org/abs/2510.03136. Preprint, arXiv:2510.03136. ArXiv preprint.






