betaFeedback

WorksComputer Science

One Confidence Threshold, Unequal Coverage: Language Gaps in Typed Decision Interfaces and How Translating Inputs Reduces Them

Abstract

Deployed classifiers often act on a prediction only when its confidence clears a threshold and pass the rest to a person. We ask whether one threshold serves all languages equally on typed decision interfaces, APIs that return a probability for each option of a fixed multiple-choice question. In a confirmatory study with a written analysis plan, three vendors' interfaces classified 1,000 MASSIVE items, each in 51 languages, with a threshold of 0.9. Even when a request is answered correctly in both English and another language, the English version clears the threshold more often, by 8.6-11.7 percentage points on average. Observed coverage, the share of requests handled automatically, is lower than English's in every other language, while the requests the gate handles are rarely wrong. The gap remains when every language is reweighted to the same accuracy, and a standard calibration metric (ECE) does not detect it on one interface. In exploratory analyses, English's coverage stays higher when each language gets its own threshold fitted to the same 5% error target. Translating inputs into English nearly closes the gap at 0.9 but leaves English ahead at that error target. Requests written natively in African languages show a gap in the same direction when English is their human translation. We recommend reporting per-language coverage at the operating threshold.

Introduction

A common way to deploy a classifier is to let it act alone only when it is confident. The application reads the probability of the top option, handles the request automatically when that probability clears a threshold, and otherwise defers it to a person. This rule, which we call a confidence gate, is selective prediction with a

AI assistance: The authors used Claude Code (Anthropic) during the preparation of this work. reject option [7, 17]. Typed decision interfaces, a kind of model API first offered with TypeSafe AI’s Jev [42] and since by OpenAI, Microsoft and Perplexity [31, 29, 34], make the rule easy to apply. The developer writes a question with a fixed set of options, and the model returns one probability per option for an input such as a user’s message. “Typed” refers to the question’s declared answer type, such as Boolean or Choice. A developer whose users write in many languages can apply one threshold to all of them. We ask how evenly that threshold serves the languages, whether the difference survives changes of accuracy, schema and threshold, and what reduces it.

Figure 1 shows the effect on one item. We measure it with two standard quantities of selective prediction: coverage, the share of requests the gate handles automatically, and selective error, the error rate among those it handles. Earlier multilingual work asks whether the probabilities are trustworthy, which is calibration [1, 46, 24, 20], or how accurate a model’s kept answers are when it may abstain [13], or how well confidence ranks errors in each language [12]. A model can be well calibrated in two languages and still be less confident in one of them. That lower confidence can be what calibration requires in a language the model answers less accurately, so we also fit each language its own threshold for the same 5% selective error and ask whether English still covers more.

One MASSIVE item in three of its 51 language versions
LanguageInput (MASSIVE’s version)Top optionProbabilityGate at 0.9
Englishhey siri what’s the timedatetime_query (correct)0.98handled automatically
Swahilihey siri saa ngapidatetime_query (correct)0.52deferred to a person
Javaneseheh siri jam piradatetime_query (correct)0.30deferred to a person

Figure 1. One MASSIVE item in three of its 51 language versions, sent to gpt-6-luna. The model picks the correct intent in all three, but only the English probability clears the threshold, so the gate defers the correct Swahili and Javanese answers to a person.

Every language receives the same items, in the datasets’ translations. The main evidence is a confirmatory study on 1,000 MASSIVE intent items [14] in each of 51 languages, sent to three typed decision interfaces from different vendors. Exploratory analyses on these items and on a separate set of 150 MASSIVE and 300 XNLI items [8] per language compare the gap with other classifiers, schemas and formats and test what reduces it. Both datasets translate English originals, so a further exploratory check uses requests written natively in 16 African languages [47], with human or machine English translations.

Findings. The first finding is the confirmatory result; the others are exploratory.

  • The gap. In all three interfaces, among items answered correctly in both English and another language, the gate handles English more often, by 8.6–11.7 points on average. Observed coverage is lower than English’s in every other language, detectably in 44–47, while the requests the gate handles are rarely wrong (§ 3).

  • A common error target. With each language’s threshold fitted to 5% selective error on tuning items, English’s held-out coverage still exceeds the mean other language’s by 17.3–18.9 points (§ 3).

  • Comparisons. The gap remains after standardizing every language to a 75% share of correct answers and with a schema written in each input’s language. A classifier fine-tuned on every MASSIVE language has a much smaller gap, at a higher error (§ 4).

  • Natively written inputs. When the non-English request is the original and English its human translation, the English version is still handled more often, by 25.0–33.4 points (§ 4).

  • What reduces it. Translating each input into English before the gate reduces the gap at 0.9 to 0.8–1.4 points; at a common 5% error target it narrows English’s lead on two interfaces (§ 5).

  • What does not. Thresholds fitted on each language’s own labels leave English’s coverage higher, and per-language conformal sets widen English’s lead in single-label sets on gpt-6-luna. A fixed-sequence test certifies per-language thresholds in more than half of language–split pairs from 300 labels per language upward, and only on two interfaces (§ 5).

Setup

Interface and gate. A request carries an input and a Choice question about it, and the response gives one probability per option. For MASSIVE the input is the user’s utterance and the question asks for the user’s intent, with the 59 intents of the test split as options and each intent’s name (“alarm set”) as its description. For XNLI the input is a premise–hypothesis pair and the question asks how the hypothesis relates to the premise, with three options and one-sentence descriptions. We call the question text and the option descriptions the schema; option values such as alarm_set are not part of it. The schema stays in English while the input changes language, as when a developer writes the schema once for users of all languages. At threshold tt the gate handles a request automatically when the top option’s probability is at least tt and defers it otherwise. The returned option is always the top option, and confidence means its probability, not the separate confidence field that the interfaces also return. The main threshold is 0.9.

Models. gpt-6-luna is served by OpenAI’s Decisions API (OpenAI, [31]), a public beta whose model name carries no snapshot version; it returned probabilities rounded to two decimals. Microsoft-Decision-1 (Microsoft, [29]) and Perplexity Decider v1.1 27B (Perplexity, [34]) are served by OpenRouter’s decision endpoint with provider fallback disabled, and every response named the same snapshot (microsoft-decision-1-20261009 and pplx-decider-v1.1-27b-20261006). They were chosen as the OpenRouter decision models from vendors other than OpenAI that have a versioned name and return full-precision probabilities over all 59 options (Appendix § 10).

Data. An item is one source utterance or sentence pair with all its language versions; a request is one language version of one item. The confirmatory study draws 1,000 MASSIVE test items at random from those present in all 51 languages, excluding every item of the exploratory study and an earlier pilot, so each model receives 51,000 requests. The exploratory study uses 150 other random MASSIVE items and 300 XNLI test pairs in all 15 XNLI languages. Tables name languages by their ISO 639-1 codes (en English, sw Swahili, jv Javanese, ur Urdu; zh-CN and zh-TW are simplified and traditional Chinese in MASSIVE, and zh is Chinese in XNLI).

Measures. Per language we report coverage, selective error, accuracy of the top option, and top-label expected calibration error (ECE) over 10 equal-width bins. Our main measure compares like with like. Take the set BLB_L of items whose top option is correct in both English and language LL. On these items, dLd_L is how much more often the gate handles the English version, and the mean paired handling gap Δ\Delta averages dLd_L over the 50 other languages:

dL=1∣BL∣∑u∈BL(Hen(u)−HL(u)),Δ=150∑L≠endL,d_L = \frac{1}{|B_L|} \sum_{u \in B_L} \left(H_{\mathrm{en}}(u) - H_L(u)\right), \qquad\Delta= \frac{1}{50} \sum_{L \ne\mathrm{en}} d_L,

where H(u)H(u) is 1 if the gate handles that version of item uu and 0 otherwise.

Refusals. A refused or unanswered request counts as deferred and wrong, and is left out of ECE. gpt-6-luna refused 42 of 7,650 exploratory MASSIVE requests and 0 of 4,500 XNLI requests and answered 99.7% of confirmatory requests and 99.6% of translated ones; the other two interfaces answered every request.

Confirmatory plan. The plan fixed the item sample, the three models, the threshold, the scoring rules above and Δ\Delta as the primary endpoint, one test per model. Inference resamples the 1,000 items 10,000 times, each resample shared by all languages, and gives a percentile 95% interval and a two-sided bootstrap pp-value; Holm’s procedure adjusts the three pp-values. The confirmatory success criterion for a model is an adjusted pp below 0.05 with Δ>0\Delta> 0. Seven secondary measures, reported in Sections 3 and 4 and Appendix A, carry no decision rule. The plan was written before the first confirmatory request, and every planned request was sent (Appendix A).

Comparing languages with English. Testing 50 languages against English one at a time would flag some differences by chance. We therefore use a simultaneous 95% band: a bootstrap interval for each language’s difference from English, widened so that all 50 intervals hold together with 95% probability (Appendix E). A difference is detected when its band excludes zero.

Exploratory analyses. Every analysis of the 150-item MASSIVE and the XNLI data, and every analysis of the confirmatory items outside the plan, is exploratory. Corrections within an analysis are stated with it; we do not correct across exploratory analyses.

Does one threshold serve the languages equally?

The mean paired handling gap favors English in all three interfaces. The primary endpoint meets the confirmatory success criterion in all three models (Table 1): on items answered correctly in both languages, the gate handles the English version more often by 8.6–11.7 points on average over the 50 other languages (Holm-adjusted p<0.001p < 0.001 for each). All 150 per-language estimates dLd_L are positive (Table 7, Appendix A).

gpt-6-lunaMicrosoft-Decision-1Perplexity Decider
Δ\Delta, primary endpoint11.2 [9.0, 13.3]11.7 [10.2, 13.2]8.6 [7.2, 9.9]
Holm-adjusted pp< 0.001< 0.001< 0.001
Languages with dL>0d_L > 0 (of 50, descriptive)505050
Smallest / largest dLd_L2 (ja) / 37 (jv)2 (de) / 39 (ta)2 (ru) / 23 (am)
Coverage, en / sw / jv59 / 36 / 2775 / 41 / 4778 / 48 / 55
Lowest coverage27 (jv)33 (ta)48 (cy)
Coverage below / above en (simult.)45 / 044 / 047 / 0
Accuracy, en / lowest78 / 6088 / 5888 / 69
Selective error, en5.13.63.6
Highest selective error8.1 (jv)7.0 (sq)8.2 (sq)
ECE, en / range4.0 / 2.2–5.73.6 / 2.4–8.83.8 / 3.5–9.4
ECE above / below en (simult.)0 / 01 / 06 / 0
Handled among correct, en / sw72 / 5182 / 5485 / 65
Handled among correct below en (simult.)443438
Correct, deferred, en / sw22 / 3216 / 3313 / 24
Standardized gap8.5 [6.6, 10.4]9.0 [7.3, 10.7]6.9 [5.1, 8.6]

Table 1. In all three interfaces, the gate handles English more often, including on items answered correctly in both languages. Confirmatory study, gate at 0.9. Δ\Delta: mean paired handling gap with 95% bootstrap interval. “simult.”: languages whose difference from English the simultaneous band detects. Correct, deferred: share of all requests deferred although the top label was correct. Standardized gap: English minus mean language in coverage at a 0.75 share of correct answers, with 95% interval. A language named at a minimum or maximum is chosen before rounding. In % or points.

langgpt-6-luna accgpt-6-luna covgpt-6-luna dLd_LMicrosoft-Decision-1 accMicrosoft-Decision-1 covMicrosoft-Decision-1 dLd_LPerplexity Decider accPerplexity Decider covPerplexity Decider dLd_L
af724218826311806410
am704317745218745023
ar714316776011796410
az7451880611181659
bn74481280601483679
cy653524724427694822
da745188669485707
de765648571285743
el775548569486726
en7859–8875–8878–
es755368470484725
fa7547148471284706
fi73461583631184697
fr775288469586725
he7651108368585716
hi775378366884725
hu7653583641084689
hy744911836311816510
id765288469585734
is724317765517786210
it765378672386744
ja785728773286753
jv642737704723735518
ka674312755712745813
km603224643729695118
kn74481178561481669
ko775648570585734
lv74501182611280668
ml76481375472685697
mn765010745420786115
ms7447138164984707
my745010583335786113
nb75491184668846810
nl775468570486725
pl785558571287744
pt765278569485743
ro7547148465883697
ru765628672285752
sl73481284611382678
sq684217755317756111
sv765288468586725
sw663625714130714822
ta744613653339826611
te7749978541983689
th74481276561781688
tl734911805617816311
tr775448567785724
ur764517826410816610
vi765478571386734
zh-CN775468471484743
zh-TW7448108167582705

Table 7. Confirmatory study per language. acc: accuracy; cov: coverage at 0.9; dLd_L: among items correct in both English and the language, the share handled in English minus the share handled in the language. All in % or points.

Observed coverage is lower than English’s in all 50 languages. With one threshold, gpt-6-luna handles 59% of English requests automatically, 36% of Swahili and 27% of Javanese, the lowest. The other two interfaces are more confident overall but show the same pattern. Observed coverage is below English’s in all 50 languages for each model, detectably so in 44–47 of them. The handled requests are almost always correct: the largest observed selective error is 8.2% (Table 1).

For gpt-6-luna, ECE does not show the gap. Each language’s observed ECE differs from English’s by at most 1.8 points, and the simultaneous band (median half-width 3.6 points) detects no difference, while coverage trails English by up to 32 points. For Microsoft-Decision-1 and Perplexity Decider the band detects higher ECE than English’s in 1 and 6 languages and lower in none, but coverage differences in many more.

Much of the gap is correct answers sent to a person. Of the correctly answered requests, gpt-6-luna’s gate handles 72% in English and 51% in Swahili; this share is below English’s in all 50 languages for each model, detectably so in 44 for gpt-6-luna. The gate defers 32% of all Swahili requests although the top label was correct, against 22% of English requests. This share follows from three standard quantities:

P(correct and deferred)=accuracy−coverage×(1−selective error).P(\text{correct and deferred}) = \text{accuracy} - \text{coverage} \times(1 - \text{selective error}).

Lower coverage outside English appears at every tested threshold. In the exploratory study the simultaneous band puts 9, 15, 23 and 35 MASSIVE languages below English at thresholds 0.7, 0.8, 0.9 and 0.95, and none above (Appendix § 15, Figure 4).

Figure 4. The language ordering changes little across thresholds. Coverage on MASSIVE and XNLI at the four tested thresholds 0.7, 0.8, 0.9 and 0.95, one gray line per language, with English, Swahili, Javanese and Urdu highlighted.

On XNLI, calibration also differs by language. Selective error rises to 29% (ur), and the band detects an ECE difference against English for 10 of 14 languages, in line with the miscalibration outside English that Yang et al. [46] report for multilingual question answering.

At a common error target, English still covers more. Part of the gap arises because 0.9 sits at a different point on each language’s risk–coverage curve: on the exploratory items, the threshold that keeps observed selective error within 5% ranges from 0.44 to 0.94 across MASSIVE languages (English 0.52). But the curves also differ (Figure 2). On the confirmatory items we fit each language’s threshold to 5% selective error on a random half and apply it to the other half, over 200 random splits (Table 3, Appendix § 17). English’s held-out coverage then exceeds the mean other language’s by 17.3–18.9 points in the three interfaces, each 95% interval above zero, at held-out selective error of 5.1–5.2%.

Figure 2. On MASSIVE, English’s empirical curve reaches the highest coverage at 5% selective error, tied with Turkish. gpt-6-luna, exploratory study: observed selective error against coverage as the threshold falls, one gray line per language; markers show the gate at 0.9. Dashed: 5% selective error (MASSIVE), 50% coverage (XNLI).

interfacethresholdenmeanen − mean [95%]below enlowest (lang)errmiss
gpt-6-lunain-sample56.940.916.0 [4.9, 27.7]49 / 10.0 (ar)––
held-out57.840.517.3 [8.5, 23.4]50 / 79.8 (jv)5.248.9
Microsoft-Decision-1in-sample80.463.017.4 [14.3, 22.8]50 / 2128.5 (my)––
held-out81.163.118.0 [15.4, 21.9]50 / 3029.4 (my)5.252.7
Perplexity Deciderin-sample86.967.919.0 [14.6, 22.5]50 / 3140.5 (km)––
held-out86.767.818.9 [15.4, 21.9]50 / 4639.6 (km)5.151.2

Table 3. With thresholds fitted to 5% selective error, English’s coverage exceeds the mean other language’s, in-sample and held-out. Confirmatory items. en, mean: coverage for English and the mean of the other 50 languages; en − mean with 95% bootstrap interval; below en: languages whose coverage is below English’s, as point estimates / detected by the simultaneous band; lowest: lowest coverage, with ties listed (0: no threshold meets 5%); err: held-out selective error, averaged over splits and then languages; miss: share of language–split pairs whose held-out selective error exceeds 5%. In % or points.

Diagnostic comparisons

The standardization and the classifier use the confirmatory items, the native-input check uses InjongoIntent, and the schema, format and open-weight checks use the exploratory items, with gpt-6-luna unless noted.

The gap remains after standardizing every language to a 75% share of correct answers. Most languages are less accurate than English, so some coverage gap is expected. This planned secondary measure recomputes each language’s coverage as if 0.75 of its answers were correct, keeping its own handling rates among correct and among wrong answers. English still leads the mean language by 6.9–9.0 points, with every 95% interval above zero (Table 1).

With natively written requests, English is handled more often even as the translation. In MASSIVE and XNLI, English is the original and every other version a translation. InjongoIntent Yu et al., [47] reverses this: native speakers of 16 African languages wrote its requests, covering 40 intents, and 622 of them come with a human English translation. We sent both versions to the three interfaces, with the MASSIVE question and these intents as options (Table 4; Appendix § 11). On items correct in both versions, the gate handles the English translation more often than the native original, by 25.0–33.4 points. Machine translation into English points the same way: over 640 native requests in each of five languages, the mean gap is 13.3–32.1 points, and every per-language interval lies above zero (smallest 5.7 [2.5, 8.9], sw on gpt-6-luna).

human pairsmachine pairs
covaccΔ\DeltacovaccΔ\Delta
gpt-6-luna41 / 7567 / 8628.5 [23.5, 33.5]60 / 7884 / 8917.6 [15.8, 19.3]
Microsoft-Decision-133 / 8053 / 8833.4 [27.8, 38.9]43 / 8170 / 9232.1 [29.9, 34.3]
Perplexity Decider48 / 8469 / 8925.0 [20.4, 29.6]66 / 8581 / 9113.3 [11.6, 14.9]

Table 4. The English version is handled more often also when it is the translation. InjongoIntent requests written natively in African languages, gate at 0.9, exploratory. Human pairs: 622 native utterances in 16 languages and their human English translations. Machine pairs: 640 native utterances in each of sw, am, ha, yo and zu and their gpt-5.4-mini English translations. cov, acc: native / English. Δ\Delta: mean paired handling gap, English minus native, on items correct in both versions, with 95% interval; for human pairs, pooled over the 622 items; for machine pairs, the mean of the five languages’ gaps. In % or points.

With a machine-translated schema in the input’s language, the gate still handles English more often. So far every non-English input came with an English schema. We translated the question and the 59 intent descriptions into each language with LLMs, mostly Qwen models (Alibaba Cloud, 2026; Qwen Team, [35]), and sent the same 150 items per language with the schema in the input’s language; option values stay English identifiers (Appendix § 13). The mean paired handling gap stays above zero in all three interfaces.

On gpt-6-luna it grows from 8.3 to 18.1 points, a detected increase, and the localized schema reorders the languages; on Microsoft-Decision-1 and Perplexity Decider it changes by less than three points, a change that is not detected (Appendix § 13).

A classifier fine-tuned on every language has a smaller gap and a higher error. We gated a public XLM-RoBERTa base model (Conneau et al., [9]), fine-tuned on MASSIVE’s training data in all its languages (haryoaw, 2024), on the same confirmatory inputs (Appendix § 11). Its mean paired handling gap is 1.8 points [0.9, 2.7], against 8.6–11.7 for the interfaces. It handles 91% of English requests, against 59–78% for the interfaces, and its handled requests are wrong up to 13.9% of the time (km), against at most 8.2%. The classifier also differs in training and architecture, so the comparison describes two kinds of system.

Wording, format and model checks. An English rewording of the schema with the same meanings keeps gpt-6-luna’s coverage ranking (Spearman 0.88), and a corrected Swahili schema changes Swahili coverage by +1.3 points [−5.3, +7.3] (Appendix § 13). Asking the gate’s question together with two other questions about the same input leaves per-language coverage nearly unchanged (Spearman 0.96; Appendix § 15.§ 1). In three open-weight models of the class, English ranks first or second in coverage at 0.9 (Appendix § 19). A gate can also fail with high coverage everywhere: a small open model, laya-multilingual (convaiinnovations, 2026), handles 71–83% of XNLI requests in every language at selective error up to 34%, a failure a coverage check alone would miss.

What reduces the gap, and what does not

Translating inputs into English

Translating each input into English reduces the gap in all three interfaces. A deployer can translate each request into English before the gate (translate-test). We translated the 50,000 non-English confirmatory inputs with gpt-5.4-mini (OpenAI, [32]) and sent them with the unchanged English schema (Appendix § 11). The mean paired handling gap falls from 8.6–11.7 to 0.8–1.4 points (Table 5), fewer languages fall detectably below English, and English’s lead over the lowest language shrinks. A small gap remains detectable only for Perplexity Decider. Accuracy rises too, and so does selective error in some languages whose coverage rises, such as Swahili on gpt-6-luna and Perplexity Decider. Khmer has the lowest coverage after translation in all three interfaces.

gpt-6-luna origgpt-6-luna TTMicrosoft-Decision-1 origMicrosoft-Decision-1 TTPerplexity Decider origPerplexity Decider TT
Δ11.2 [9.0, 13.3]0.8 [−1.1, 2.6]11.7 [10.2, 13.2]1.0 [−0.1, 2.1]8.6 [7.2, 9.9]1.4 [0.2, 2.4]
Δ on the fixed set11.00.511.30.78.20.9
Accuracy78 / 74 / 6078 / 76 / 6488 / 80 / 5888 / 85 / 7488 / 82 / 6988 / 85 / 73
Below en (of 50)453448478
Coverage, en / sw / jv59 / 36 / 2759 / 58 / 5775 / 41 / 4775 / 72 / 7378 / 48 / 5578 / 76 / 76
Lowest coverage27 (jv)45 (km)33 (ta)57 (km)48 (cy)60 (km)
en − lowest321442173017
Selective error, sw5.87.15.24.84.06.1
Highest selective error8.1 (jv)7.8 (am)7.0 (sq)6.4 (th)8.2 (sq)7.8 (km)
en − mean at 5% target17.3 [8.5, 23.4]13.7 [6.7, 20.3]18.0 [15.4, 21.9]6.7 [5.0, 9.9]18.9 [15.4, 21.9]10.8 [7.8, 12.9]

Table 5. Translating each input into English before the gate reduces the mean paired handling gap in all three interfaces.

Translation changes which answers are correct, so we also compare the gap on items correct in both languages under both conditions. The MASSIVE versions were translated from English, so translating back might simply recover the English utterance; 18.4% of translations match it after normalizing case, punctuation and whitespace. The reduction stays detectable on the fixed items, without the exact matches, and on the half of pairs least similar to the English utterance, in all three interfaces (Table 8, Appendix § 11).

gpt-6-lunaMicrosoft-Decision-1Perplexity Decider
Fixed items, orig / TT11.0 / 0.511.3 / 0.78.2 / 0.9
change−10.5 [−11.7, −9.4]−10.6 [−11.6, −9.6]−7.3 [−8.1, −6.5]
Without exact matches, orig / TT11.4 / 1.111.6 / 1.18.8 / 1.7
change−10.3 [−11.5, −9.2]−10.5 [−11.5, −9.4]−7.1 [−7.9, −6.3]
Least similar half, orig / TT11.5 / 0.912.6 / 1.410.8 / 2.7
change−10.6 [−12.0, −9.3]−11.2 [−12.4, −10.0]−8.0 [−9.0, −7.1]

Table 8. The reduction from translate-test holds on fixed items and without translations that reproduce the English utterance. Confirmatory items, gate at 0.9. orig, TT: mean paired handling gap without and with translation; change: TT minus orig with 95% interval. Fixed items: items correct in both languages under both conditions. Least similar half: pairs below the median chrF to the English utterance. In points.

At a common error target, translation narrows English’s lead on two interfaces. We repeat the held-out analysis of Table 3 on the translated inputs, fitting each language’s threshold to 5% selective

Table 5: Translating each input into English before the gate reduces the mean paired handling gap in all three interfaces. Exploratory analysis of the confirmatory items, gate at 0.9. orig: the MASSIVE input; TT: its English translation. Δ: mean paired handling gap with 95% interval; on the fixed set: on items correct in both languages under both conditions. Accuracy: English / mean of the other 50 languages / lowest language. Below en: languages the simultaneous band puts below English. en − lowest is computed before rounding, so it can differ by one from the difference of the rounded values. en − mean at 5% target: English’s held-out coverage minus the mean other language’s with each language’s threshold fitted to 5% selective error (as in Table 3), with 95% interval. In % or points.

error on the same tuning halves and applying it to the same held-out halves; held-out selective error is 5.1–5.4%. English’s lead is 6.7–13.7 points after translation, against 17.3–18.9 before (Table 5). The mean other language’s held-out coverage rises by 11.3 [9.2, 13.1] points on Microsoft-Decision-1 and 8.1 [6.4, 9.9] on Perplexity Decider; on gpt-6-luna the change, 3.6 [−3.7, 8.7], is not detected.

Thresholds set per language from about a hundred labels

A deployer can instead set thresholds from a small labeled sample, aiming at selective error of at most 5% (the error budget). We simulate this on the exploratory items, split by even or odd id into 83 tuning and 67 held-out items per language (156/144 on XNLI), and compare three thresholds:

  • English threshold: the lowest threshold whose selective error on English’s tuning items is within budget, applied to every language (0.74 on MASSIVE, 0.98 on XNLI).

  • Own threshold: the same rule, fitted on each language’s own tuning items.

  • Certified threshold: Learn-then-Test [5], in two variants. Each of 50 thresholds from 0.50 to 0.99 gets an exact one-sided binomial test of its tuning error. The Bonferroni variant corrects over all of them; the fixed-sequence variant tests from 0.99 downward and stops at the first failure. If tuning and future items are i.i.d., either gives a threshold whose population selective error is at most 5% with probability at least 0.9. If none passes, the gate defers everything.

The English threshold misses the budget in many languages. On the held-out items it exceeds the 5% budget in 12 of 51 MASSIVE languages and twice the budget in 1. The worst language (sq) reaches 11%, while English stays within it (0%). On XNLI it misses the budget in 15 of 15 languages.

Own thresholds leave coverage unequal and often exceed the budget. Under its own threshold, a MASSIVE language stays within the budget on held-out items in 34 of 51 cases; among these, coverage ranges from 31% to 79%, and the other 17 exceed the budget (Appendix F). On XNLI an own threshold exists in 3 languages and exceeds the budget in 3 of them. Over 200 random half splits, the own threshold misses the budget in 47% of MASSIVE language–split pairs on average.

Neither procedure certifies a threshold on the exploratory split. The sample is too small. Even a threshold with no tuning errors must handle 122 tuning items to pass the Bonferroni test, against 83 per MASSIVE language, and the fixed-sequence variant needs 45 handled items at 0.99, where no MASSIVE language handles more than 26. The certified gate therefore defers every request, on XNLI as well (Appendix E).

Thresholds certified with larger label budgets

From 300 labels, fixed-sequence certification succeeds in more than half of language–split pairs on two interfaces. The larger confirmatory sample allows bigger label budgets (Table 6; exploratory). The fixed-sequence procedure certifies in 63% and 55% of language–split pairs at 300 labels per language on Microsoft-Decision-1 and Perplexity Decider, against 30% and 27% at 150; gpt-6-luna reaches 18% only at 500. On a grid up to 0.99, and counting a split that certifies nothing as zero coverage, English’s held-out coverage is never below the mean other language’s; at 500 labels Perplexity Decider handles 78% of English requests against 46% for the mean other language. Adding 1.00 to the grid reverses the comparison for gpt-6-luna at 500 labels (Appendix H).

nngpt-6-lunaMicrosoft-Decision-1Perplexity Decider
certcov en / meanown misscertcov en / meanown misscertcov en / meanown miss
8300 / 061.60.20.6 / 0.157.7423 / 254.4
150211 / 0.859.13029 / 1556.32746 / 1553.1
3001114 / 451.46361 / 3252.55571 / 3150.2
5001824 / 648.98774 / 4552.77878 / 4651.2

Table 6. Certification of per-language thresholds by label budget. Exploratory analysis of the confirmatory items, 200 random splits per language into nn tuning and at least 500 held-out items. cert: share of language–split pairs in which the fixed-sequence procedure certifies, with 90% confidence, a threshold whose population selective error is at most 5%. cov en / mean: held-out coverage for English and the mean other language, counting a split that certifies nothing as zero. own miss: share of pairs in which the uncertified own threshold exceeds 5% held-out selective error. In %.

Per-language conformal thresholds

Per-language conformal thresholds bring set coverage to target but leave automatic handling unequal. We ran the conformal method of Vangala & Navya [43] on gpt-6-luna’s outputs for the exploratory items. A request is handled automatically when its prediction set holds a single label. With one threshold per language, set coverage reaches its 90% target in every language on average, but English’s lead over the lowest language in this singleton rate widens from 30 points under a pooled threshold to 51 on MASSIVE. Part of this widening follows from the two-decimal rounding of gpt-6-luna’s probabilities (Appendix I).

Related work

Work on multilingual calibration studies the probability values and how to repair them, for pretrained language models and multilingual question answering models ([1, 46]) and for LLMs ([24, 20]). Feng et al. [13] show that abstention strategies developed on English lose accuracy in low-resource languages. We instead ask how the coverage of one fixed gate on a production interface differs by language. Selective prediction and the reject option ([7, 17]) define coverage and selective risk, and Jones et al. [21] showed that confidence-based abstention can widen group disparities. Our groups are the language versions of the same items in production interfaces.

TypeSafe AI introduced the interface class with Jev ([42]). Concurrently, a benchmark of Jev reports steep per-language accuracy degradation, with risk–coverage curves per dataset pooled over languages ((Deußer et al., 2026)). Another benchmark compares the class with classifiers and language models and applies held-out selective-risk thresholds ([37]). Where Deußer et al. (2026) report accuracy by language, we report what one fixed gate does in each language. Fadli et al. [12] find on Indonesian regional languages that Jev’s calibration error does not grow relative to Indonesian while its ranking of errors (area under the risk–coverage curve) worsens. We measure the coverage of one fixed threshold on three interfaces in 51 language versions and test per-language thresholds, which they leave to future work. Also concurrently, Vangala & Navya [43] show on MasakhaNEWS and AfriSenti that a pooled conformal threshold on a TF-IDF classifier meets its 90% coverage target on average but covers Somali at 77.5%, and one threshold per language restores coverage with 100–200 labels. Gao et al. [16] show that equal coverage and equal set size across groups are in tension. Appendix § 20 covers further related work.

Limitations

The confirmatory study covers MASSIVE only. XNLI and the schema, format, and open-weight checks rest on 150 or 300 items per language, which resolves large coverage differences but not individual ranks. Translate-test used one translator. MASSIVE has been public since 2022, so its English utterances may be better represented in the models’ training data than its translations. In MASSIVE and XNLI every non-English version is a translation from English. The native-input check covers one task and 16 African languages, with 31–40 human-translated items each, and measures the direction of the gap for these languages. We measure deferral but not its cost: human-review availability, latency and outcomes are outside the data.

Conclusion

On MASSIVE, a confidence gate on three typed decision interfaces serves 51 languages unequally: answers correct in both languages are acted on more often in English, while the requests the gate handles are rarely wrong. The gap remains after standardizing every language to a 75% share of correct answers. In exploratory analyses it also remains with a schema written in each input’s language and with each language’s threshold fitted to the same 5% error target, and it holds for requests written natively in African languages, with English as the human translation. Translating inputs into English reduces the gap at 0.9 from 8.6–11.7 to 0.8–1.4 points; at a common 5% error target, English’s lead after translation is 6.7–13.7 points. At 300 labels per language, a fixed-sequence test certifies per-language thresholds in more than half of language–split pairs on two interfaces, against under a third at 150. How often it certifies, and whether English keeps the higher coverage, depend on the procedure and its threshold grid. We suggest that evaluations of typed decision interfaces report, per language, coverage at the operating threshold and the mean paired handling gap against a reference language, alongside calibration.

Broader Impact Statement

Where selective error is controlled, a confidence gate can automate routine requests and reserve human review for uncertain ones. A gate that defers more often in some languages gives their speakers less automation and, where deferred requests go to human review, can mean longer waits. Per-language coverage reports can expose this before deployment. The “deferred with a correct top label” counts are retrospective: a top label is known to be correct only in hindsight. Lowering a language’s threshold on their basis requires measuring the selective error at the new threshold.

Confirmatory study details

Plan. Before the plan was written, three requests per OpenRouter model on one already-used item (in English, Swahili and Javanese) checked the response format. The plan fixes the following. The pool is the MASSIVE test ids present in all 51 languages minus every id used in this paper’s exploratory study or in an earlier 200-item pilot (Appendix § 14), which leaves 2,633 eligible ids; the sample is 1,000 ids drawn from the pool sorted by id with NumPy’s default_rng(20261015). Each request carries the MASSIVE intent question of § 2 alone, with the 59 intents of the test split as options. Scoring, the primary endpoint, its bootstrap (10,000 resamples of the item ids with default_rng(20261016); a language whose BLB_L is empty in a resample is left out of that resample’s mean), the two-sided bootstrap p-value p=min⁡(1,2min⁡(P∗(Δ≤0),P∗(Δ≥0)))p = \min(1, 2\min(P^{*}(\Delta\le0), P^{*}(\Delta\ge0))), Holm’s procedure over the three models and the decision rule are as in § 2. Requests are retried on transport errors and HTTP 429 or 5xx, refusals are not, and all 51,000 requests of a model are sent with no interim look; an arm with fewer than 95% answered requests counts as failed and enters no test. The secondary measures, reported without a decision rule, are: per-language coverage at 0.9 with the simultaneous band of its difference from English; the same for ECE and for the share handled among correct answers; the coverage gap at a common accuracy of 0.75 (coverage recomputed as 0.75P(H∣correct)+0.25P(H∣wrong)0.75P(H \mid\text{correct}) + 0.25P(H \mid\text{wrong}), English minus the mean language, with a bootstrap interval); selective error per language with exact 95% intervals; the share of requests deferred with a correct top label; and, for gpt-6-luna, the Spearman correlation of per-language coverage with the exploratory study. The ancillary files hold every secondary measure for each model and language.

Models. Microsoft-Decision-1 and Perplexity Decider were chosen from the OpenRouter models with decision outputs, by four criteria: all 59 options returned with probabilities summing to one, full-precision probabilities, a versioned model name and a vendor other than OpenAI. Checking the response format excluded models that rejected a 59-option question, accepted at most 20 options or returned rounded probabilities. Each OpenRouter arm then passed an admission check of 20 requests on exploratory items: every response returned all 59 options, with probabilities summing to one within 0.01 and a choice among the options. Requests go to https://openrouter.ai/api/v1/systeme with the model name, the input, the question and "provider": {"allow_fallback": false}. gpt-6-luna’s unanswered requests were refused or empty after retries.

Per-language results. Table 7 gives accuracy, coverage at 0.9 and dLd_L for every language and model. The highest exact 95% upper bound on selective error is 12.0% (gpt-6-luna, jv: 22 errors among 272 handled requests). On the 150 exploratory items (Table 2), gpt-6-luna’s per-language coverage correlates with the confirmatory one at Spearman 0.84.

langaccECEautoerrpaired hand/bothEN thr cov/errown thr cov/err
en8356010 / 2872 / 072 / 0
nl83853014 / 1973 / 287 / 5†
de78458310 / 1876 / 242 / 0
zh-CN79555114 / 1667 / 272 / 4
ja83654116 / 1572 / 073 / 0
hi78554111 / 1875 / 6†78 / 6†
ar74644220 / 2060 / 269 / 4
ur76946117 / 2164 / 264 / 2
am69740025 / 1758 / 5†63 / 5
jv641124349 / 1342 / 7†45 / 7†
sw68729039 / 1746 / 337 / 0
cy70937030 / 1757 / 8†51 / 3

Table 2. The exploratory study on MASSIVE: coverage varies across languages while observed selective error stays between 0% and 8%. gpt-6-luna, selected languages (all in Appendix C), in %. acc: accuracy. auto, err: coverage and selective error at 0.9. hand/both: among items correct in both English and the language, the share handled in English but deferred in the language, and the share deferred in both. EN thr, own thr (Section 5): held-out coverage/selective error under a threshold fitted to a 5% error budget on English’s or the language’s own tuning items. “–”: no threshold meets the budget. †: error above 5%, also when it rounds to 5.

Methods of the exploratory additions

Translate-test. Each non-English confirmatory input was translated into English by gpt-5.4-mini at temperature 0, 50 utterances of one language per call, with the instruction “Translate each of the following user requests to English. Output only the translations, as a JSON object with the same keys.” and the utterances as a JSON object with numbered keys; the source language was not named. A reply was used only if it had exactly the requested keys and a non-empty string for each; otherwise the call was retried, and after three failures each utterance was translated in its own call with “Translate the following user request to English. Output only the translation.” Every interface received the translations with the unchanged English schema; English inputs and answers are those of the confirmatory study. Similarity to the original English utterance is measured as exact match after lowercasing and removing punctuation and extra whitespace (18.4% of pairs; from 8.8% for ja to 35.2% for af), and as character F-score (chrF; mean 65.8). The low-similarity half is the pairs below the median chrF; gaps on both subsets use the confirmatory bootstrap.

Natively written inputs. The data are the test splits of InjongoIntent (Yu et al., [47]), published as masakhane/InjongoIntent (revision fe4be38, Apache-2.0). Human pairs are the 622 items of its English test split, each a native utterance and its human English translation, with the language taken from the example id (31–40 items per language). Machine pairs are the full test splits of Swahili, Amharic, Hausa, Yoruba and Zulu, translated as in the translate-test arm above. Each request carries the MASSIVE intent question alone, with the 40 intents as options and each intent’s name, underscores replaced by spaces, as its description. Each interface receives 7,644 requests. gpt-6-luna refused 15 of them, scored as deferred and wrong; the other two interfaces answered every request. Intervals come from 10,000 resamples of the items, within each language for the machine pairs.

Fine-tuned classifier. The checkpoint is haryoaw/scenario-TCR-data-AmazonScience-massive-all_1.1-model-xlm-roberta-base (revision f996499f), an XLM-RoBERTa base model fine-tuned on the training split of MASSIVE in all its locales. Its 60 output classes map to MASSIVE’s intent list; the one intent absent from the test split (cooking_query) is scored as wrong when predicted. Inputs are tokenized with the checkpoint’s tokenizer and truncated to 128 tokens. The confidence is the top softmax probability, and all other quantities follow the confirmatory analysis with the same bootstrap draws. It handles 96% of correct English answers (the interfaces: 72–85%), its ECE is 9.4–14.0% in every language, and the simultaneous band puts 26 languages below English in coverage.

Label budgets. For each budget nn of 83, 150, 300 and 500, 200 random permutations of the 1,000 items are drawn once with a fixed seed and shared by all languages and interfaces; the first nn items of a permutation are the tuning items and the rest are held out, so the budgets are nested. Thresholds are fitted as in § 5. We use Learn-then-Test because it bounds selective error; conformal risk control (Angelopoulos et al., [4]) bounds the share of all requests that are handled and wrong.

Full per-language tables

Tables 9 and 10 give every language, with the columns of Table 2.

langaccECEauto errpaired hand/bothEN thr cov/errown thr cov/errlangaccECEauto errpaired hand/bothEN thr cov/errown thr cov/err
tr8055816 / 1876 / 284 / 5†fi78748119 / 1969 / 076 / 6†
vi80553014 / 1776 / 278 / 2es76447322 / 1769 / 233 / 0
ko79457213 / 1676 / 463 / 0th77651115 / 1867 / 467 / 4
de78458310 / 1876 / 242 / 0tl76448117 / 1867 / 470 / 4
ru80752016 / 1775 / 481 / 6†zh-CN79555114 / 1667 / 272 / 4
hu77551316 / 1975 / 6†76 / 8†ta79753112 / 1967 / 472 / 4
hi78554111 / 1875 / 6†78 / 6†lv73947318 / 1967 / 281 / 6†
pt79551016 / 1873 / 481 / 7†he78846320 / 2167 / 266 / 2
pl7975619 / 1973 / 461 / 0af77941024 / 2167 / 276 / 6†
nl83853014 / 1973 / 287 / 5†zh-TW76949518 / 2066 / 7†31 / 0
fr771055211 / 1973 / 278 / 2kn75845122 / 1764 / 566 / 5
id77555412 / 1672 / 454 / 0ur76946117 / 2164 / 264 / 2
bn77551317 / 1772 / 479 / 9†az74649014 / 2063 / 270 / 4
en8356010 / 2872 / 072 / 0ms72844225 / 1461 / 252 / 3
fa781047119 / 2172 / 279 / 4is75643019 / 2061 / 061 / 0
sv77655213 / 1572 / 085 / 7†el801150416 / 2261 / 070 / 0
ja83654116 / 1572 / 073 / 0hy751047017 / 1860 / 052 / 0
te75551318 / 1572 / 476 / 6†ar74644220 / 2060 / 269 / 4
sl77450317 / 1970 / 270 / 2am69740025 / 1758 / 5†63 / 5
mn77851316 / 1670 / 6†75 / 6†cy70937030 / 1757 / 8†51 / 3
ml78649421 / 1570 / 6†75 / 6†sq69840829 / 1655 / 11†31 / 5
da79647319 / 2070 / 281 / 6†ka67745416 / 1855 / 5†52 / 3
my73850115 / 1769 / 276 / 4sw68729039 / 1746 / 337 / 0
ro75745023 / 1669 / 469 / 4km65432430 / 2146 / 6†36 / 4
nb77645318 / 2269 / 7†87 / 16†jv641124349 / 1342 / 7†45 / 7†
it82555010 / 1969 / 079 / 4

Table 9. MASSIVE, all 51 languages, sorted by coverage under the English threshold. Columns as in Table 2.

langaccECEauto errpaired hand/bothEN thr cov/errown thr cov/err
en82968110 / 2645 / 9†45 / 9†
ru6920612318 / 1244 / 22†–
bg7612611516 / 1340 / 11†31 / 9†
tr6720582722 / 1338 / 29†–
de7613611713 / 1638 / 13†–
vi6720582117 / 1138 / 19†–
th6719562420 / 1237 / 21†–
es7512571617 / 1537 / 13†–
zh6820602417 / 1237 / 17†–
el7116592220 / 1336 / 21†18 / 19†
fr7215601415 / 1335 / 20†–
hi6619552826 / 1033 / 21†–
sw6521562824 / 930 / 23†–
ar6520562619 / 1529 / 17†–
ur6419472932 / 1326 / 24†–

Table 10. XNLI, all 15 languages. Columns as in Table 2, with 156 tuning and 144 held-out items.

Schema translation checks

Localized schema. The schemas are LLM translations, checked by back-translation and, for Swahili, by LLM review (below). The baseline is an English-schema run of the same items from the same session (run-to-run variation in Appendix E). On gpt-6-luna the two coverage rankings barely correlate (Spearman 0.19), mean coverage changes by −7.6 points, and English’s lead over the lowest language grows from 31 to 40 points (Figure 3). For the other two interfaces the ranking barely moves (Spearman 0.91 and 0.92), and English’s lead stays similar: 41 to 38 points for Microsoft-Decision-1 and 33 to 33 for Perplexity Decider.

Figure 3. For gpt-6-luna, localizing the schema lowers coverage in more languages than it raises it. Each point is one language: coverage at 0.9 with the English (xx) and the localized schema (yy).

Table 11 gives selected languages under both schemas. Its English-schema baseline is a run sent in the same session as the localized one; English’s lead over the lowest language there is 31 points, against 36 in the first run of Table 2, a run-to-run difference. With the localized schema, coverage ranges over 16–59%. Swahili is handled automatically 35% of the time against 56% for English, and the English–Swahili difference goes from 25 to 21 points. The ten languages lowest under the English schema change by −0.1 points on average and the other languages by −9.5, so the new bottom is made of languages that fell, not of the old bottom left behind. For gpt-6-luna, coverage rises by more than two points in 16 languages and falls by more than two in 28, and selective error under the localized schema stays at 0–6%. Under the localized schemas, accuracy moved by at most 5 points in any language and by +1 for mn, the language that lost the most coverage. The accuracy change barely correlates with the coverage change (Pearson 0.20 across the non-English languages).

langaccuracy ENaccuracy local[email protected] EN[email protected] localsel. error ENsel. error local
en8383565611
nl8281515100
de7876575123
zh-CN7879532700
ja8484562700
hi7779535614
ar7374443720
ur7774465512
am6974394901
jv6767252932
sw6771313500
cy7071353122

Table 11. In most of these languages, localizing the schema changes coverage more than accuracy or selective error. Same items with the English schema (EN) and with the schema in the input’s language (local), in %: accuracy of the intent question, coverage at 0.9 ([email protected]) and selective error at 0.9.

Mean paired handling gap. Each run’s gap uses its own English answers, with 95% intervals from 10,000 resamples of the items. With the English and the localized schema, the gap is 8.3 [2.8, 13.5] and 18.1 [12.0, 23.9] points for gpt-6-luna, 10.7 [7.3, 14.0] and 8.4 [4.2, 12.5] for Microsoft-Decision-1, and 8.1 [5.4, 10.8] and 7.4 [4.0, 10.7] for Perplexity Decider. The 95% intervals of the change are [7.6, 12.0], [−5.3, 0.6] and [−2.2, 0.7] points.

Back-translation. An automatic back-translation check with a different LLM (gpt-5.4-mini [32]) over all 60 schema strings per language finds a mean equivalence of 4.71 of 5 on a rubric from 5 (same meaning) to 1 (unrelated or wrong), with 5.8% of strings judged changed (score 2 or less; 6 of them instructions). These are spread over 42 of the 50 languages, and the ten languages with the most hold only 54% of them. A language’s judge drift is 5 minus its mean score and its lexical drift is 1 minus the mean token F1 between each English string and its back-translation. We computed 7 rank correlations between these drift measures and the per-language changes in accuracy and coverage, with nominal pp-values. Neither measure has a detected association with the coverage change under localization: Spearman −0.14 (p=0.35p = 0.35) for judge drift and −0.26 (p=0.070p = 0.070) for lexical drift (label descriptions alone −0.26, p=0.072p = 0.072, Bonferroni 0.51). Mongolian (mn), the largest coverage loss, ranks 14 of 50 on judge drift, tied with two other languages.

English rewording. The rewording restates the question text and each of the 59 intent descriptions in different English words with the same meaning; option values and items are unchanged. We say that localization reorders the languages more than the rewording when the 95% interval of the paired difference between the two Spearman correlations lies above zero. The reworded, localized and English-schema runs were sent in one session; the endpoint refused the intent question in 9 reworded requests. Accuracy changed by +0.4 points on average, and selective error at 0.9 stays at or below 10% in every language. The rewording’s Spearman correlation with the original ordering is 0.88; the paired difference between it and the localized schema’s correlation is 0.69 (95% interval [0.45, 0.76]), and the rewording raises mean coverage by +2.5 points. The reworded schema’s gate decisions agree with the baseline run’s on 85% of requests, and the localized schema’s on 81%; two sendings of the unchanged suite on different days agree on 93.8% (Appendix § 14).

Swahili review and corrected-schema run. Two automated LLM reviews of the Swahili schema rated 9 and 8 of its 80 strings wrong and 20 and 19 imprecise. The 80 strings are the 60 that the single-question request uses plus the scenario question, 18 scenario names and the Boolean template of the multi-question request. All three alarm intents, for instance, used alama (mark) for alarm. The corrected schema takes the 24 strings that both reviews proposed identically and a fresh gpt-5.4-mini translation of the other 56; the corrected-schema run sends its question text and intent descriptions with the unchanged option values. On the same 150 Swahili items, coverage went from 34.7% to 36.0% (paired difference +1.3 points, 95% bootstrap interval [−5.3, +7.3]). Accuracy went from 70.7% to 72.7% (interval of the difference [−2.0, +6.0]). ECE went from 7% to 8% (jointly answered items: [−4, +5]). The English–Swahili coverage gap went from 21.3 to 20.0 points (25.3 under the English schema).

Reproducibility details

Requests. Each gpt-6-luna request is one POST to https://api.openai.com/v1/decisions with the JSON body {"model": "gpt-6-luna", "input": <input>, "questions": [...]}. No other field is sent, so no decoding setting is chosen; the endpoint returns probabilities rounded to two decimals and reports zero output tokens. Identical requests are not fully deterministic, as reported for repeated runs of LLMs under deterministic settings (Ouyang et al., 2025; Attil et al., 2025). The exploratory MASSIVE suite, sent a second time on another day, keeps the top label in 94.5% and the gate decision at 0.9 in 93.8% of the 7,589 requests answered both times. Within one session the variation is smaller: the localized run’s 150 English requests, identical to those of the English-schema run sent in the same session, returned the same intent probabilities in 150 cases. The input is the MASSIVE utterance as plain text, or for XNLI the object {"premise": ..., "hypothesis": ...} serialized as indented JSON, since the endpoint takes text only. A Choice question is sent as {"type": "choice", "name", "instructions", "choices": [{"value", "description"}]}, a Boolean as {"type": "predicate", "name", "instructions"}. Every response names the model gpt-6-luna. All requests were sent in October 2026, with 8 concurrent requests and a 120s timeout. The OpenRouter requests of the confirmatory study are described in Appendix § 10; their probabilities are not rounded.

Questions. MASSIVE intent (Choice): “What does the user want the assistant to do?”, one option per intent label of the test split (59), with the label as value and the label with underscores replaced by spaces as description (alarm_set: “alarm set”). Scenario (Choice, multi-question format only): “Which area does the user’s request belong to?”, the 18 label prefixes as value and description. Boolean, sent only in the multi-question format: “Is the user asking the assistant to do this: ⟨target description⟩?”. XNLI (Choice): “How does the hypothesis relate to the premise?” with entailment “the hypothesis definitely follows from the premise”, neutral “the hypothesis might or might not be true given the premise” and contradiction “the hypothesis definitely contradicts the premise”. Boolean: “Does the hypothesis relate to the premise in this way: ⟨description⟩?”. The localized schemas translate these strings and keep the option values.

Items and Boolean targets. The confirmatory study’s items are drawn as in Appendix § 10. The exploratory MASSIVE items come from the test split of all 51 languages (mteb/amazon_massive_intent): the ids present in every language, in the order of the English test split, are shuffled by Python’s random with seed 20260919 and the first 150 kept. The pilot excluded from the confirmatory pool is built the same way from the ids present in 11 languages (en, zh-CN, zh-TW, ja, ko, es, de, ar, hi, vi, sw), shuffled with seed 20260917, first 200 kept. XNLI items (facebook/xnli) are the first 300 of the shuffle of the test indices 0, 1, . . . under the same seed, with a fresh generator. For each item the Boolean target is drawn once, with the same generator: the gold label with probability 0.5, otherwise a uniformly drawn other label, and the same target is used in every language. The Boolean question appears only in the multi-question format.

Available material. The ancillary files contain the aggregate values of every printed table and the generated values behind the results reported in the text: aggregate statistics per system and language, and the per-item values of the worked example in Figure 1. The items come from the public MASSIVE, XNLI and InjongoIntent test splits and are selected by the procedure above and in Appendices § 10 and § 11; the open-weight models are the public repositories named below, run at each repository’s default revision at run time.

Failures and refusals. A request gets up to 8 attempts on a connection error or HTTP 429, 529 or ≥500\ge500, waiting min⁡(60,2a)\min(60,2^{a}) seconds times a uniform factor in [0.5,1.5][0.5,1.5] after attempt aa; any other non-200 status is stored as a failure. No exploratory request failed. The endpoint refused the intent question in 42 of 7,650 MASSIVE requests (74 localized) and the XNLI question in 0 of 4,500; refusals are scored as described in § 2 (Refusals).

Open-weight models. Each model received the multi-question request and ran through the inference code in its Hugging Face repository (Mapika/decider-2b, Hanno-Labs/bosun-v3.1-1.7b, FrontiersMind/Lumma-fev-0.6b), which takes the same input and questions and returns a probability per option, so prompt formatting and probability extraction are the repositories’ own. open-decider uses the repository’s Decider.system_one, open-lumma its decide, and open-bosun its predict, called once per question with the options as candidates (it permutes their order internally). Settings are the repositories’ defaults, the XNLI input is passed as the structured pair, and the gate reads the top option probability as for the API. The comparator of § 4 is caininnovation s/laya-multilingual, read the same way.

Statistics. ECE is top-label: confidences fall into 10 equal-width bins on [0,1][0,1], each closed on the left (a confidence of 0.7 opens the bin [0.7,0.8)[0.7,0.8)), and ECE is the sum over non-empty bins of the bin’s share of items times ∣mean confidence−accuracy∣|\text{mean confidence} - \text{accuracy}|, refusals excluded. With 150 exploratory items per language the bins hold 15 items on average and typically one holds fewer than five, so per-language ECE there is noisy and biased [38]; the confirmatory study’s 1,000 items give 100 per bin on average. The confirmatory study’s simultaneous bands use the construction below on the 10,000 resamples of its primary bootstrap.

For the exploratory analyses, paired rates rest on the items correct in both English and the language, 92 for jv. Marginal and paired 95% intervals are percentile intervals from 2,000 resamples of the items (paired: of the items shared with English). The simultaneous band uses 20,000 draws, each one multiset of items shared by all languages. The fixed-threshold coverage comparisons with English and the comparison of handled correct answers use one run of these draws; the risk–coverage bands come from one run of 10,000 draws, and the single- vs multi-question format-change band from 2,000 draws. Let dkd_k be the observed difference of language kk against English, dk∗d_k^* its value in a draw and sks_k the standard deviation of dk∗d_k^* over draws. Then qq is the 95% quantile of max⁡k∣dk∗−dk∣/sk\max_k |d_k^* - d_k|/s_k, and the band is dk±qskd_k \pm qs_k (q=3.48q = 3.48 for ECE and 3.18 for coverage on MASSIVE, 2.88 and 2.82 on XNLI). In the exploratory study a language’s refused pairs drop out of its ECE contrast; in the confirmatory study each language’s ECE is computed on its own answered requests. The per-threshold bands use the same construction at each threshold. The rank-correlation intervals of Appendix § 15 resample languages and items jointly (2,000 draws); those of the English rewording resample items only, with the 51 languages fixed. The corrected-schema run uses 4,000 paired resamples. The repeated threshold splits of the 150-item exploratory study use 200 random halves drawn separately for each language, with the English threshold fitted on the same items as the language’s tuning half. The conformal comparison uses 200 splits shared by all languages. Random generators are seeded per system and language.

Threshold fits. The empirical fit is the lowest observed confidence whose selective error on the tuning items is at most 5% (integer counts). The Learn-then-Test fit tests each threshold tt of the grid 0.50, 0.51, …\ldots, 0.99 by pt=P(Bin⁡(nt,0.05)≤et)p_t = P(\operatorname{Bin}(n_t, 0.05) \le e_t), with ntn_t tuning items at or above tt and ete_t errors among them, and certifies the lowest tt with pt≤0.1/50p_t \le0.1/50 (Bonferroni over the grid). The fixed-sequence variant tests from 0.99 downward at level 0.1 and stops at the first failure. It certifies a threshold in 0 of 51 MASSIVE and 0 of 15 XNLI languages: 0.99 already fails everywhere on the exploratory even/odd split, and at 0.99 no MASSIVE language has more than 26 tuning items (45 error-free items are needed).

Threshold sweep and further checks

Marginal and unadjusted intervals at 0.9. These descriptive comparisons complement the simultaneous band of § 3. On MASSIVE every non-English marginal ECE interval overlaps English’s. The marginal coverage intervals of English [52, 68]% and Swahili [23, 37]% do not overlap, and those of 8 languages lie entirely below English’s interval. Unadjusted paired intervals exclude zero for 2 (ECE, with upper ends of up to 11 points) and 37 (coverage) languages. On XNLI unadjusted intervals detect an ECE difference for 11 languages, against 10 under the simultaneous band. Across the 15 XNLI languages, ECE and coverage rank-correlate at −0.41 (95% interval [−0.84, 0.37], resampling languages and items). The interval spans both signs, so the direction of the association is unresolved.

Threshold sweep. The coverage gap is present at every tested threshold (Figure 4). On MASSIVE, coverage ranges over 43–73% at 0.7 and 15–53% at 0.95. At 0.7, 0.8, 0.9 and 0.95 the mean English-minus-language gap is 6 (95% interval [3, 10]), 9 [5, 13], 12 [8, 17] and 15 [10, 20] points. The English–Swahili gap is 17 [4, 30], 25 [11, 38], 31 [17, 44] and 32 [19, 45] points, each interval from that threshold’s own simultaneous band. Javanese is the lowest at every threshold on MASSIVE.

Resource classes. The English-minus-language gap at 0.9 rank-correlates with the resource class of Joshi et al. [22], a rating of how much data and tooling a language has (higher is better resourced), assigned by language name on MASSIVE. The rank correlation is −0.57, with a 95% interval [−0.74, −0.22] that resamples languages and items as if the 50 languages were a sample. Better-resourced languages tend to trail English by less, but the relation is not monotonic. By class, from 1 to 5 (5 the best resourced), the mean English-minus-language gap at 0.9 is 17, 22, 12, 7 and 8 points over 12, 3, 15, 13 and 7 languages. The 3 class-2 languages trail English by more than the class-1 languages, and class 5 by more than class 4. Slovenian, listed in two classes, takes class 3, and Norwegian (nb) takes the Norwegian Bokmål class, 1.

Own-threshold misses. Table 12 lists every language whose own threshold, fitted on the tuning items, has held-out selective error above 5%, with the handled and error counts behind it. The intervals are per language, without simultaneous correction across languages. Only nb’s exact interval among MASSIVE languages, and el’s on XNLI, lie wholly above 5% (Table 12). The English threshold fitted on the tuning half, 0.74, differs from the in-sample 0.52 of § 3 because it uses the tuning half only.

suitelangthrnneeerr95% CIsuitelangthrnneeerr95% CI
MASSIVEnl0.515835.2[1.1, 14.4]MASSIVEmn0.635036.0[1.3, 16.5]
MASSIVEtr0.445635.4[1.1, 14.9]MASSIVEjv0.723026.7[0.8, 22.1]
MASSIVElv0.585435.6[1.2, 15.4]MASSIVEsv0.455747.0[1.9, 17.0]
MASSIVEda0.615435.6[1.2, 15.4]MASSIVEpt0.515447.4[2.1, 17.9]
MASSIVEru0.555435.6[1.2, 15.4]MASSIVEhu0.715147.8[2.2, 18.9]
MASSIVEhi0.625235.8[1.2, 15.9]MASSIVEbn0.545359.4[3.1, 20.7]
MASSIVEaf0.485135.9[1.2, 16.2]MASSIVEnb0.4258915.5[7.3, 27.4]
MASSIVEfi0.665135.9[1.2, 16.2]XNLIbg0.994548.9[2.5, 21.2]
MASSIVEte0.655135.9[1.2, 16.2]XNLIen0.986569.2[3.5, 19.0]
MASSIVEml0.675036.0[1.3, 16.5]XNLIel1.0026519.2[6.6, 39.4]

Table 12.

Smaller label budgets and repeated splits. The own fit is repeated on a random subset of 25 or 50 tuning items. With 25 tuning items the fit exceeds twice the budget in 5 MASSIVE languages, and with 50 Table 12: Most own-threshold misses are not distinguishable from 5%. Languages whose own threshold (fitted on the 83 MASSIVE or 156 XNLI tuning items to a 5% budget) has held-out selective error above 5%. thr: the threshold; nn: the held-out requests it handles; ee: the errors among them; err: the error rate, with its exact (Clopper–Pearson) 95% interval (CI), in %. Held-out items: 67 (MASSIVE) and 144 (XNLI) per language.

items in 1. We also repeat the fit over 200 random 75/75 splits of the same 150 items, which overlap. The own threshold misses the budget on the held-out half in 47% of language–split pairs on average (22–64% per language, English 56%), and in more than half the splits for 17 of 51 languages. Counting splits in which no threshold meets the budget on the tuning half as misses also gives 47%. The English threshold misses in 69% of pairs.

Uncorrected Learn-then-Test bounds. Without correcting for the choice of threshold, the tightest 90% bound that any single threshold reaches on the tuning items is close to the budget at best: 4.2% (pt) to 14.2% (sq) on MASSIVE, English 5.3%, and 8.1–33.9% on XNLI.

Multi-question requests

The multi-question request adds two questions to the gate’s Choice question: on MASSIVE a scenario question (“Which area does the user’s request belong to?”, 18 options) and a Boolean question asking whether the user wants one given intent, and on XNLI the Boolean question only (Appendix E). The gate reads only the answer to its own question. The open-weight models of Section 4 received this format. gpt-6-luna received it for the same 7,650 MASSIVE and 4,500 XNLI requests, and the analysis is the same. Per-language coverage at 0.9 correlates across the two formats at Spearman 0.96 on MASSIVE and 0.98 on XNLI, and the English-minus-language gaps at 0.96 and 0.98. The mean coverage change is −0.1 points on MASSIVE (95% interval [−0.4, 0.2]) and 0.1 on XNLI ([−0.3, 0.5]). Per-language changes range from −3.3 to 2.7 points on MASSIVE, and the simultaneous band detects 0 of them (XNLI: 0). The answers differ between the formats about as often as two sendings of the same multi-question request do (Table 13). In both formats English has the highest coverage at 0.9 and Javanese the lowest, no language is detected above English, the simultaneous band detects no MASSIVE ECE difference, and no Learn-then-Test threshold is certified.

MASSIVE unless notedsingle (main)multi
Coverage at 0.9, English / Swahili / Javanese (%)60 / 29 / 2459 / 31 / 23
Languages below 50% coverage at 0.92630
Below en at 0.7/0.8/0.9/0.95 (simult.)9 / 15 / 23 / 3512 / 14 / 19 / 30
Above English at the same thresholds0 / 0 / 0 / 00 / 0 / 0 / 0
Mean English-minus-language gap at 0.9 (points)12 [8, 17]11 [7, 16]
ECE differences detected (simult. / unadjusted)0 / 20 / 4
Own threshold within 5% held out (of 51)3434
Learn-then-Test thresholds certified00
XNLI coverage at 0.9, English / Urdu (%)68 / 4768 / 47
XNLI ECE above English (simult., of 14)109
Agreement with multi-question runsingle question, MASSIVE / XNLIre-sent: single / copy
Top label agrees (%)98.0 / 98.996.2 / 97.4
Gate decision at 0.9 agrees (%)98.1 / 98.098.6 / 98.6
Mean total-variation distance per item0.019 / 0.0110.021 / 0.020

Table 13. Single-question (main) and multi-question requests on the same items. Brackets are 95% bootstrap intervals; “simult.” counts languages whose paired difference against English the simultaneous 95% band detects. Top-label agreement and distance count the 7,601 MASSIVE and 4,500 XNLI requests answered in both formats; gate agreement counts all requests, with a refusal deferred. The last column compares the single-question answer and a re-sent multi-question copy with the original multi-question answer on the same 506 MASSIVE requests.

Risk–coverage details

Exploratory items. With every MASSIVE language at its own in-sample 5% threshold, English reaches 80% coverage and leads the mean language by 14 points (95% interval [6, 24]), although the simultaneous band detects no single language below English. Because those thresholds are chosen in-sample, we also fix the coverage instead. On XNLI, where no language’s full-sample curve reaches 5% observed error, selective error at 50% coverage is 8% for English and up to 29% (ur), and the band puts 13 of 14 languages above English (Table 14).

langautoerrcov at risk ≤ 5%thrlangautoerrcov at risk ≤ 5%thr
sq405/6031 [4, 59]0.94fa471/7170 [59, 80]0.57
km322/4833 [13, 60]0.89ml493/7471 [7, 82]0.67
jv241/3642 [8, 57]0.72mn512/7771 [33, 81]0.60
sw290/4444 [31, 67]0.79bn512/7671 [34, 81]0.56
zh-TW494/7347 [24, 82]0.91id553/8371 [43, 83]0.65
cy370/5551 [39, 65]0.77lv472/7071 [45, 80]0.58
ka453/6851 [15, 71]0.77my501/7571 [60, 80]0.49
nb452/6757 [39, 82]0.84af410/6172 [54, 84]0.49
am400/6059 [43, 73]0.59zh-CN551/8272 [61, 83]0.62
ms441/6659 [41, 79]0.74da472/7172 [41, 83]0.61
is430/6561 [47, 75]0.67hi541/8173 [53, 82]0.62
ro450/6762 [47, 74]0.74hy470/7073 [49, 83]0.55
es472/7063 [9, 77]0.75pt510/7673 [63, 85]0.63
tl481/7263 [45, 75]0.70el503/7573 [41, 87]0.56
kn451/6863 [50, 83]0.73sv552/8273 [57, 84]0.56
hu512/7667 [43, 81]0.73ta531/8073 [58, 85]0.60
ko572/8567 [16, 85]0.76de583/8773 [35, 85]0.70
he462/6967 [33, 82]0.65fr552/8274 [36, 82]0.57
th511/7667 [53, 79]0.68vi530/8074 [62, 84]0.57
fi481/7267 [56, 78]0.66it550/8376 [60, 87]0.56
sl502/7568 [24, 81]0.65ja541/8177 [64, 88]0.62
ar441/6668 [53, 76]0.50ru520/7877 [62, 88]0.53
az490/7369 [53, 78]0.54nl530/7978 [59, 89]0.53
ur461/6969 [53, 81]0.60en601/9080 [67, 90]0.52
te512/7669 [32, 81]0.65tr581/8780 [57, 87]0.44
pl561/8469 [56, 85]0.72

Table 14. Risk–coverage per language, gpt-6-luna. auto, err: coverage and selective error (errors/handled) at 0.9. cov at risk: the largest coverage whose observed selective error is at most 5% (MASSIVE, top) or 10% (XNLI, bottom), with its 95% bootstrap interval; thr: the threshold that attains it (“–”: none). XNLI also gives selective error at a common coverage of 50% (err at 50% cov) with its 95% bootstrap interval, and its difference from English in points with the simultaneous 95% band (− en [simult.]). 0 [0, 0]: no threshold meets the budget in the sample or in any resample. In %.

For each language, the requests are sorted by the gate’s confidence and the threshold is lowered one distinct value at a time, so tied requests enter together. Coverage at a risk level is the largest coverage whose observed selective error is at most that level (zero if no threshold reaches it), and the threshold reported is the one that attains it. Selective error at a common coverage takes the most confident share of requests, with a tied group entering pro rata (the expectation under random tie-breaking). Intervals come from 10,000 resamples of the items, each shared by all languages, and the simultaneous bands use the construction of Appendix § 14 on the differences from English. Table 14 lists every language.

On XNLI, 2 of 15 languages reach 10% observed selective error at some threshold (English at 60% coverage), and none reaches 5%. At 50% coverage on MASSIVE, selective error is 0% for English and up to 9% (sq); the mean language exceeds English by 2 points ([1, 4]), and the simultaneous band detects no single language. Among the open-weight models, open-decider reaches 5% observed selective error at 75% coverage in English and 0–75% across languages, and the simultaneous band puts 21 languages below English. For open-bosun the values are 45% and 0–50%, with 0 languages detected.

langautoerrcov at risk ≤ 10%threrr at 50% cov− en [simult.]
ar5643/1680 [0, 27]–23 [16, 30]15.1 [5.8, 24.4]
de6132/1830 [0, 46]–17 [11, 23]9.8 [1.5, 18.0]
el5938/1760 [0, 22]–19 [13, 27]11.8 [4.0, 19.6]
es5727/1710 [0, 53]–15 [10, 21]7.9 [0.4, 15.3]
fr6026/1800 [0, 60]–15 [10, 21]7.6 [0.7, 14.6]
hi5546/1660 [0, 18]–27 [20, 34]19.7 [9.9, 29.4]
ru6142/1820 [0, 0]–21 [15, 29]13.9 [4.9, 22.8]
sw5647/1670 [0, 13]–27 [20, 34]19.3 [9.0, 29.5]
th5640/1680 [0, 25]–22 [15, 29]14.0 [5.5, 22.6]
tr5847/1730 [0, 19]–26 [19, 33]18.5 [9.8, 27.1]
ur4740/1400 [0, 0]–29 [22, 37]21.6 [11.1, 32.0]
vi5836/1740 [0, 28]–18 [13, 25]10.8 [2.3, 19.2]
zh6043/1810 [0, 19]–21 [15, 29]13.9 [4.9, 22.9]
bg6127/18253 [0, 65]0.938 [4, 14]1.0 [−4.9, 6.8]
en6823/20560 [0, 81]0.958 [4, 12]

Table 18.

Thresholds fitted to a common risk target, and a certification grid up to 1.00

Both analyses are exploratory and use the confirmatory items. For coverage at a common risk target, a language’s threshold is the lowest confidence whose selective error is at most 5%, as in Appendix § 16. The in-sample threshold is fitted on all 1,000 items. The held-out threshold is the own threshold of § 5, fitted on 500 tuning items of each of the 200 splits of the label-budget analysis and applied to the other 500; held-out coverage is the mean over splits. Intervals and simultaneous bands use the confirmatory study’s item resamples: all 10,000 in-sample and the first 1,000 held-out, with every threshold fitted in each resample (Table 3). With held-out thresholds the simultaneous band puts 7–46 languages below English, and held-out selective error exceeds 5% in 48.9–52.7% of language–split pairs, scattering around the target.

The certification comparison applies the fixed-sequence procedure of Table 6 with two grids: 0.50 to 0.99, and 0.50 to 1.00, where testing starts at 1.00. A threshold of 1.00 handles only requests with probability 1.00. gpt-6-luna returns probabilities rounded to two decimals, so these exist; Microsoft-Decision-1 and Perplexity Decider return none, and their test fails at the first step (Table 15).

gpt-6-lunaMicrosoft-Decision-1Perplexity Decider
nncertcov, 0.99cov, 1.00certcov, 0.99cov, 1.00certcov, 0.99cov, 1.00
830 / 00 / 00 / 00.2 / 00.6 / 0.10 / 04 / 023 / 20 / 0
1502 / 011 / 0.80 / 030 / 029 / 150 / 027 / 046 / 150 / 0
30011 / 314 / 43 / 0.863 / 061 / 320 / 055 / 071 / 310 / 0
50018 / 1024 / 62 / 387 / 074 / 450 / 078 / 078 / 460 / 0

Table 15. A fixed-sequence grid that also includes 1.00 certifies less often than the grid up to 0.99 wherever the latter certifies. Confirmatory items, splits as in Table 6. cert: share of language–split pairs certified, grid to 0.99 / grid to 1.00. cov: held-out coverage for English / the mean other language under each grid, counting a split that certifies nothing as zero. In %.

Label budgets in detail. The Bonferroni variant certifies far less than the fixed-sequence one: at 500 labels in 23% of pairs for Microsoft-Decision-1, 19% for Perplexity Decider and 0.6% for gpt-6-luna. On the grid up to 0.99, English is certified more often than the average other language: at 500 labels, Perplexity Decider is certified for English in 100% of splits but for km in 12%. Held-out coverage of the certified gate falls both when certification fails and when the certified threshold is high; at 500 labels gpt-6-luna handles 24% of English requests and 6% of the mean other language’s, against 59% of English requests under the fixed 0.9 gate. The uncertified own threshold exceeds 5% held-out selective error in 48.9–61.6% of gpt-6-luna’s pairs, depending on the label budget, and in a similar share for the other two interfaces. A grid that also includes 1.00 certifies nothing for Microsoft-Decision-1 and Perplexity Decider, whose probabilities stay below 1.00, and certifies gpt-6-luna at 500 labels in 10% of pairs instead of 18%; English’s coverage (2%) then falls below the mean other language’s (3%) (Table 15).

Table 15: A fixed-sequence grid that also includes 1.00 certifies less often than the grid up to 0.99 wherever the latter certifies. Confirmatory items, splits as in Table 6. cert: share of language–split pairs certified, grid to 0.99 / grid to 1.00. cov: held-out coverage for English / the mean other language under each grid, counting a split that certifies nothing as zero. In %.

Conformal comparison details

Method. We ran the method of Vangala & Navya [43] on our outputs (Table 16). Split conformal prediction with the LAC score (one minus the probability of the true label) returns for each request a set of labels meant to contain the true one with 90% probability; this probability is the set coverage. A request is handled automatically when its set holds a single label, and the share of such requests is the singleton rate, the conformal counterpart of our coverage. The threshold is calibrated on all languages pooled or on each language, over 200 random half splits. Because the pooled rows are translations of the same items, the pooled threshold has no finite-sample guarantee.

langpooled conformalper-language conformalgate 0.9correct, deferred
covsingleerrcovsingleerrautoerrconf.gate
en93531915626012823
sw87†27295703006139
jv86†22396502436041
km84†30594803245734
cy89†333931803705233
am89†350932004004929
sq89†337931944085233

Table 16. Per-language conformal thresholds bring set coverage to target in every language, on average over splits, but leave singleton rates unequal. gpt-6-luna, exploratory items. The method of Vangala & Navya [43] on MASSIVE (LAC score, 90% target), calibrated on all languages pooled or on each language, next to our gate at 0.9. Averaged over 200 random half splits shared by all languages. cov: held-out set coverage. single: share of held-out items whose set holds one label. err (conformal): error among singletons. auto, err: the gate’s coverage and selective error. correct, deferred: share deferred although the top label was correct, under per-language conformal sets (conf.) and under the gate. The gate columns are held-out averages over the same splits, so they differ slightly from Table 2. †: set coverage below 90% before rounding. In %.

Results. Their finding recurs on our outputs. The pooled threshold falls below target set coverage in 16 of 51 MASSIVE languages (lowest 84%, km) and in 7 of 15 XNLI languages, while per-language thresholds bring every language to 91% or more on average (XNLI 90%). The singleton rate stays unequal: English’s lead over the lowest language widens from 30 points under the pooled threshold to 51 under per-language thresholds on MASSIVE, against 36 for our gate on the same items (XNLI: 25, 47 and 21). These leads are averaged over splits, so they differ slightly from the differences of the averaged rates in Table 16. Set coverage also flags different languages than our gate: of the 10 languages lowest in pooled set coverage, 5 are among the 10 lowest at our gate (ties included).

Per-language conformal thresholds therefore trade unequal set coverage for a wider gap in singleton rates, which parallels the tension Gao et al. [16] derive between equal coverage and equal set size. For gpt-6-luna part of this widening is mechanical: its probabilities are rounded to two decimals, and on the even/odd split 42 of the 51 per-language thresholds admit every intent with probability at least 0.01, so a singleton occurs only when all other intents round to zero. The APS score shows the same pattern with a smaller widening.

Splits and results. Each of 200 draws splits the items into halves, and the same split applies to every language, so a held-out item never enters calibration through one of its translations. One threshold is calibrated on the calibration half of all languages pooled (75 items in 51 languages, 3,825 rows on MASSIVE) or one per language (75 items). With nn calibration scores the threshold is the (n+1)×90%\left(n+1\right) \times90\%-th smallest, and a set holds every label whose score is at most the threshold. The pooled threshold covers the true intent in 91% of held-out items on average over languages. Under per-language thresholds set coverage is 91–96% in every language, while the singleton rate runs from 5% (jv) to 61% (ja), English 56%. Under the same thresholds 61% of held-out Swahili requests are deferred although the top label is correct, against 28% for English (gate: 39% and 23%). A refusal gets score 1, so its set is empty below threshold 1 and holds every label at 1. Under per-language thresholds, mean set size reaches 21.5 intents (English 3.0). On the even/odd split it is up to 20.5 (English 2.5). Mean singleton error under per-language thresholds is at most 5% in any MASSIVE language and reaches 24% on XNLI (ur), since 90% set coverage does not hold the error of the singletons alone to 10%.

APS score. The sensitivity check replaces LAC with the randomized APS score (Romano et al., [39]): the labels are ranked by probability, with tied labels in a random order, and the score is the probability of the labels ranked above the true one plus a uniform random fraction of the true label’s own probability. The order and the fraction are drawn once per request. Splits, calibration halves and the singleton rule are those of the LAC comparison. Under APS the pooled threshold falls short of 90% set coverage in 24 of 51 MASSIVE languages (lowest 85%) and in 9 of 15 XNLI languages. The English-minus-lowest singleton gap is 30 points pooled and 35 per language on MASSIVE (lowest per language: jv, 21%), and 27 and 41 on XNLI. Per-language calibration thus widens the singleton gap under both scores, by less under APS (LAC: 30 to 51 on MASSIVE).

Localized schema. With the localized schema, 6 of the 10 languages with the lowest per-language singleton rate stay among the 10 lowest (ties included), and English leads the lowest language by 43 points (51 with the English schema).

Open-weight models and the small comparator

Coverage ranking. Three open-weight models of the same interface class, open-decider [27], open-bosun [18] and open-lumma [15], received the same items in the multi-question format (Table 17). English’s rank by coverage at 0.9 is 2 for open-bosun (behind zh-CN), 1 for open-decider, 1 for open-lumma (tied with one other language) and 1 for gpt-6-luna. For open-decider and gpt-6-luna, English’s higher coverage comes at similar observed selective error to the other languages. For open-bosun and open-lumma, selective error is also higher outside English (14% and 22% pooled, against 7% and 5%). These models handle few requests in their lowest languages (Table 17), so their rankings are coarse.

systemenlowest languageother languages
autoerr (k/n)auto (lang)err (k/n)err (k/n)en − lowestρ\rho
gpt-6-luna601 (1/90)24 (jv)3 (1/36)2 (68/3,601)361.00
open-bosun587 (6/87)1 (am)100 (2/2)14 (396/2,862)570.76
open-decider450 (0/67)1 (km)0 (0/1)2 (30/1,537)440.54
open-lumma135 (1/20)0 (12 tied)– (0/0)22 (52/236)130.47

Table 17. English ranks first or second in coverage in every model, and its observed selective error is no greater than that of the other languages pooled. MASSIVE exploratory items at 0.9, in %: coverage (auto) and selective error (err, kk errors among nn handled) for English, the lowest-coverage language (tied languages pooled) and all non-English languages pooled; spread: en − lowest; ρ\rho: Spearman correlation of per-language coverage with gpt-6-luna’s. The open-weight rows use the multi-question request; the gpt-6-luna row and ρ\rho use its single-question run.

English’s lead over the median language at 0.9 is 16, 25 and 11 points for open-bosun, open-decider and open-lumma, and 11 for gpt-6-luna. A language’s marginal 95% coverage interval lies entirely below English’s interval in 25, 36 and 38 languages for the three open models, and in 8 for gpt-6-luna (its single-question run, as in Table 17. This is a descriptive count without pairing or multiplicity adjustment. The language orderings correlate with gpt-6-luna’s at Spearman 0.76, 0.54 and 0.47. Of each model’s ten lowest languages, ties included, 6, 5 and 5 are among gpt-6-luna’s ten lowest. The open-lumma model has 12 languages at zero coverage, so its bottom set is not ten, and these languages cannot be ordered among themselves.

The three models differ in overall accuracy and confidence. The open-decider model handles 45% of English automatically, and its English accuracy of 85% is above gpt-6-luna’s. The open-lumma model handles 13% (English accuracy 48%). On XNLI only one of the three clears the threshold often enough to compare languages. The open-decider model handles 60% of English (gpt-6-luna 68%), and its XNLI ordering has a Spearman correlation of 0.77 with gpt-6-luna’s. The open-bosun model clears the threshold on about 3% of requests at best, and open-lumma on a single item in one language.

Small comparator. laya-multilingual (convaiinnovations, 2026) is a small open multilingual model of the interface class (about 0.3B parameters, Apache-licensed), run on the same items. On MASSIVE it is a weak classifier (intent accuracy 6–55%, English 55%), and its coverage tracks its accuracy (1–37%). On XNLI it is overconfident everywhere: it handles 71–83% of requests automatically in every language, with selective error from 9% (English) to 34% and ECE up to 32%. On MASSIVE, gpt-6-luna’s gate shows the opposite pattern: selective error stays within 0–8% while coverage varies by language.

Further related work

Huang et al. [20] find that instruction tuning on high-resource-language data raises confidence in low-resource languages without a matching gain in accuracy, and Alfarano et al. [2] compare uncertainty estimation methods for multiple-choice question answering across 22 languages. The multilingual calibration studies cited in the main text repair the probabilities with temperature scaling on target-language data or small translated calibration sets ([1, 46]), and Feng et al. [13] propose multilingual feedback, which reduces the abstention gap. Xin et al. [45] study selective prediction for NLP models, and Kamath et al. [23] showed that abstention based on softmax probabilities fares poorly under domain shift. Conformal risk control and Learn-then-Test ([4, 5]) give finite-sample guarantees for threshold selection that our empirical fit lacks, and group-conditional risk control ([40]) would give such a guarantee per group. Gao et al. [16] also show that pooled conformal calibration can distort group-wise coverage. In concurrent work on security decisions with this interface, Liu [26] reports that a frozen policy validated under an error budget exceeds it on unseen inputs. Its groups are sets of related security-benchmark inputs, where ours are languages. Two further studies of the class report the same held-out miss ([25] (Nandakishore 2026)). Messoudi & Mansar [28] find that naive thresholds on models of the class exceed their risk target in about half of settings, and Learn-then-Test thresholds in 7.8%. Sharma et al. [41] report that a hosted model of the class is sensitive to wording and systematically underconfident. Wang et al. [44], also concurrent, address the language difference on the training side with a Chinese-language model of the class, whereas we measure what a fixed gate does with it.

References

  1. [1]Kabir Ahuja, Sunayana Sitaram, Sandipan Dandapat, and Monojit Choudhury. On the calibration of massively multilingual language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4310–4323, 2022. doi: 10.18653/v1/2022.emnlp-main.290.DOI
  2. [2]Andrea Alfarano, Andrea Bacciu, Saab Mansour, Amin Mantrach, and Marcello Federico. Estimating uncertainty from reasoning: A large-scale study of multi- and crosslingual MCQA performance in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2026, 2026. To appear. arXiv:2607.06327.arxiv.org/abs/2607.06327
  3. [4]Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In International Conference on Learning Representations, 2024.
  4. [5]Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. Annals of Applied Statistics, 19(2): 1641–1662, 2025. doi: 10.1214/24-AOAS1998.DOI
  5. [7]C. K. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41–46, 1970. doi: 10.1109/TIT.1970.1054406.DOI
  6. [8]Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475–2485, 2018. doi: 10.18653/v1/D18-1269.DOI
  7. [9]Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440–8451, 2020. doi: 10.18653/v1/2020.acl-main.747.DOI
  8. [12]Romi Fadli, Rusdah, Wahyu Ramadani Putri Utami, and Zulfa Malika Amana. Does System One transfer? Calibrated typed-decision models, label budgets, and translation for Indonesian regional languages. arXiv preprint arXiv:2610.14112, 2026.
  9. [13]Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Oreaoghene Ahia, Shuyue Stella Li, Vidhisha Balachandran, Sunayana Sitaram, and Yulia Tsvetkov. Teaching LLMs to abstain across languages via multilingual feedback. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4125–4150, 2024. doi: 10.18653/v1/2024.emnlp-main.239.
  10. [14]Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. MASSIVE: A 1M-example multilingual natural language understanding dataset with 51 typologically-diverse languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4277–4302, 2023. doi: 10.18653/v1/2023.acl-long.235.DOI
  11. [15]FrontiersMind. Lumma-fev-0.6b. https://huggingface.co/FrontiersMind/Lumma-fev-0.6b, 2026. Hugging Face model card; Apache-2.0. Accessed 2026-10.
  12. [16]Ziang Gao, Pengqi Liu, Archer Yi Yang, Mouloud Belbahri, Jesse C. Cresswell, and Masoud Ashgarian. On the burden of achieving fairness in conformal prediction. arXiv preprint arXiv:2605.14260, 2026.arxiv.org/abs/2605.14260
  13. [17]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1705.08500
  14. [18]Hanno Labs. bosun-v3.1-1.7b. https://huggingface.co/Hanno-Labs/bosun-v3.1-1.7b, 2026. Hugging Face model card; Apache-2.0. Accessed 2026-10.
  15. [20]Jerry Huang, Peng Lu, Qiuh ao Zeng, Yusuke Iwasawa, Yutaka Matsuo, Sarath Chandar, Edison Marrese-Taylor, and Irene Li. Investigating the multilingual calibration effects of language model instruction tuning. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1–59, 2026. doi: 10.18653/v1/2026.eacl-short.1.DOI
  16. [21]Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. Selective classification can magnify disparities across groups. In International Conference on Learning Representations, 2021.arxiv.org/abs/2010.14134
  17. [22]Pratik Joshi, Sebastian Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282–6293, 2020. doi: 10.18653/v1/2020.acl-main.560.DOI
  18. [23]Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5684–5696, 2020. doi: 10.18653/v1/2020.acl-main.503.DOI
  19. [24]Lea Krause, Wondimagegnue Tufa, Selene Baez Santamaria, Angel Daza, Urja Khurana, and Piek Vossen. Confidently wrong: Exploring the calibration and expression of (un)certainty of large language models in a multilingual setting. In Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pp. 1–9, 2023. URL https://aclanthology.org/2023.mmnlg-1.1/.
  20. [25]Jiawei Li. Fast models, slow evidence: A paired and self-audited evaluation of System-1 decision models for LLM agent harnesses. arXiv preprint arXiv:2610.02267, 2026.arxiv.org/abs/2610.02267
  21. [26]Yixuan Liu. Evaluating System One models for agent security decisions: Reliability, calibration, and selective automation. arXiv preprint arXiv:2609.33401, 2026.arxiv.org/abs/2609.33401
  22. [27]Mapika. decider-2b. https://huggingface.co/Mapika/decider-2b, 2026. Hugging Face model card; Apache-2.0. Accessed 2026-10.
  23. [28]Soundouss Messoudi and Youness Mansar. Calibration and risk-budgeted decisions with System One models. arXiv preprint arXiv:2610.19811, 2026.arxiv.org/abs/2610.19811
  24. [29]Microsoft. Microsoft-Decision-1. https://commandline.microsoft.com/microsoft-decision-1-model-foundry/, 2026. Announced 2026-10-09; snapshot microsoft-decision-1-20261009, used through OpenRouter. Accessed 2026-10-16.
  25. [31]OpenAI. Decisions API. https://developers.openai.com/api/docs/guides/decisions, 2026a. Public beta; model gpt-6-luna. Accessed 2026-10-10.
  26. [32]OpenAI. GPT-5.4 mini. https://developers.openai.com/api/docs/models/gpt-5.4-mini, 2026b. Accessed through Azure OpenAI, 2026-10.
  27. [34]Perplexity. pplx-decider-v1.1-27b. https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b, 2026. Hugging Face model card; Apache-2.0. Snapshot pplx-decider-v1.1-27b-20261006, used through OpenRouter. Accessed 2026-10-16.
  28. [35]Qwen Team. Qwen3.6-27B: Flagship-level coding in a 27B dense model. https://qwen.ai/blog?id=qwen3.6-27b, April 2026a.
  29. [37]Amir Rafe and Subasish Das. Benchmarking System One decision models against trained classifiers and language models for automated decision gates. arXiv preprint arXiv:2610.00346, 2026.arxiv.org/abs/2610.00346
  30. [38]Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating bias in calibration error estimation. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 4036–4054, 2022. URL https://proceedings.mlr.press/v151/roelofs22a.html.
  31. [39]Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems, volume 33, pp. 3581–3591, 2020.arxiv.org/abs/2006.02544
  32. [40]Murilo Salem, Luísa Böhm, Daniel Pontes, and Anderson Ferrugem. Hierarchical group-conditional conformal risk control for selective prediction in language models. arXiv preprint arXiv:2607.24562, 2026.arxiv.org/abs/2607.24562
  33. [41]Rahul Sharma, Andrew B. Duncan, Gaétan Marceau Caron, and Sebastian J. Vollmer. TypedBench: A benchmark for calibration, framing sensitivity, and cost in System One decision models. arXiv preprint arXiv:2610.11392, 2026.arxiv.org/abs/2610.11392
  34. [42]TypeSafe AI. Introducing System One models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev, 2026. Accessed 2026-09-17.
  35. [43]Bhanu Prakash Vangala and Vangala Navya. One threshold does not fit all languages: Language-conditional deferral for reliable and efficient low-resource text classification. arXiv preprint arXiv:2609.37861, 2026.arxiv.org/abs/2609.37861
  36. [44]Zexiao Wang, Zihao Zhang, Xudong Wang, Pan Wang, Ziyi Ye, Haoyu Zhao, Zuxuan Wu, and Shuicheng Yan. Chinese-Jev: Bringing System One model to Chinese-language tasks. arXiv preprint arXiv:2609.36965, 2026.arxiv.org/abs/2609.36965
  37. [45]Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1040–1051, 2021. doi: 10.18653/v1/2021.acl-long.84.DOI
  38. [46]Yahan Yang, Soham Dan, Dan Roth, and Insup Lee. On calibration of multilingual question answering LLMs. Transactions on Machine Learning Research, 2026. URL https://openreview.net/forum?id=4klghu2Ptj.
  39. [47]Hao Yu, Jesujoba Oluwadara Alabi, Andiswa Bukula, Jian Yun Zhuang, En-Shiun Annie Lee, Tadesse Kebede Guge, Israel Abebe Azime, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukibii, Salomon Kabongo Kabenamualu, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, Dietrich Klakow, and David Ifeoluwa Adelani. INJONGO: A multicultural intent detection and slot-filling dataset for 16 African languages. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9429–9452, 2025. doi: 10.18653/v1/2025.acl-long.464.DOI

Paper details

vlnc:W_w96258fj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents