betaFeedback

WorksComputer Science

I play of the cello: Evaluating Constructional Bleeding in Multilingual LLMs with CALQUE

Abstract

Languages use different morphosyntactic strategies to express the same meaning. Speakers of multiple languages are able to apply strategies from one language to another, either by mistake or intentionally, to produce a literal translation, or calque. Considering that large language models (LLMs) have been shown to generalize crosslingually, despite being predominantly trained on English, we investigate whether they overgeneralize constructional strategies from English to other languages. To test this, we create a dataset of 5583 sentence pairs across seven languages, matching literal translations of strategies from English into other languages, and literal translations of the other languages' strategies into English. We then quantify constructional bleeding by measuring how often strategies from English are assigned a higher probability than native strategies. We find that models often fail to assign higher probabilities to native strategies and tend to prefer English-derived strategies even when evaluating sentences in other languages. Our findings highlight the limitations of the current English-dominant training paradigm.

Introduction

Languages express the same meaning using systematically different forms. For example, in (1), English expresses the instrument as a direct object with a definite article (the cello), while French introduces it with the preposition de, fused with the definite article (du violoncelle, from de + le).

(1) a. I play the cello. 1SG play DEF cello

I play the cello. *Je joue le violoncelle. *I play of the cello. Je joue du violoncelle. transitive strategy oblique strategy

b. Je joue du violoncelle. 1SG play.1SG of.DEF cello

The English and French constructions are language-specific, but they instantiate cross-lingually identifiable morphosyntactic patterns referred to as strategies in linguistic typology ([33, 20, 11]). When using their native language, speakers can deploy these strategies productively. When using a second language, speakers sometimes produce literal translations that transfer a strategy from one language to another. This raises a fundamental question about multilingual language models: given how prevalent English is in the training data, will models overgeneralize strategies found in English to other languages?

Interference from constructions in one language on the processing of another language can generally be described as constructional bleeding. Here we are interested in the special case when the interference occurs at the level of strategies, which we call cross-linguistic strategy transfer. We refer to a strategy conventionally used to express a given meaning in a language as the native strategy, and to a strategy transferred from another language as the transferred strategy. We introduce a controlled evaluation paradigm that holds semantic content approximately constant while varying the strategy used to express it. For each cross-linguistic contrast, as Figure 1, we construct four sentences: an English sentence with the native strategy, an English sentence with the transferred strategy, a sentence in the other language with its native strategy, and a corresponding sentence with the English strategy transferred into that language.

Figure 1. English and French speakers describe playing the cello with distinct morphosyntactic strategies. We measure bleeding of English morphosyntax into French (Δlog⁡p(⋅)\Delta\log p(\cdot) between French sentences) and additionally conduct a control experiment on English (Δlog⁡p(⋅)\Delta\log p(\cdot) between English sentences).

This design provides a controlled setting for studying cross-lingual strategy transfer. Multilingual LLMs are commonly trained on highly imbalanced multilingual corpora in which English constitutes a substantial share of the data, and previous work has found evidence of English-centric biases in their internal representations ([76, 64, 71]) and output behavior ([56, 25]). Do such biases afflict constructional strategy choices as well? By testing both directions of each strategy contrast, our framework allows us to examine whether strategy-level generalization is asymmetric: in particular, whether models overgeneralize strategies from English to other languages.

Within this framework, we introduce CALQUE: Constructional Alternations across Languages: Quantifying English bias, a benchmark for evaluating constructional bleeding via cross-lingual strategy transfer in multilingual LLMs. It covers seven languages, including English, and three broad construction types, with controlled contrasts between native and transferred strategies. Following the minimal-pair paradigm used in previous behavioral evaluations of language models ([73, 27, 49, 45]), we evaluate model preferences using sentence probabilities. We benchmark 39 language models, including 28 English-centric models and 11 models with a stronger focus on non-English languages.

Using CALQUE, we first confirm that models often struggle to distinguish native from transferred strategies, exhibiting lower failure rates in English than in other languages. Furthermore, the distributions of model preferences vary depending on their focus languages.

To control for differences in the evaluation language, we then evaluate models exclusively in Japanese, comparing native Japanese strategies with strategies transferred from the other six languages. We find evidence of asymmetric constructional bleeding: models exhibit greater difficulty distinguishing native Japanese strategies from English-transferred strategies than from strategies transferred from other languages.

Our main contributions are threefold: (1) CALQUE, a controlled benchmark for evaluating language-specific constructional representation across seven languages and three construction types; (2) results offering evidence for constructional bleeding in multilingual LLMs; and (3) a large-scale evaluation of 39 models establishing systematic differences between English-centric and non-English-centric models as well as strong correlations in strategy preferences across English-centric model families.

Related Work

Crosslingual Transfer

An analogy can be made to human acquisition of a second language. An L2 speaker will often model some aspect of their L2 output after their L1. In the field of second language acquisition, this is called transfer ([30]). When transfer is facilitative (e.g., because the two languages share a strategy in common), it is regarded as a positive transfer. When, however, it results in a formulation in the L2 that is ungrammatical or unidiomatic, it is a negative transfer or interference ([16]). Here we investigate a kind of negative transfer, albeit for multilingual models (cf. [79, 51, 2]) rather than humans.

A calque is an expression resulting from hyper-literal translation ([47]). We adopt it as the name of our dataset as the term evokes the sort of fluency contrast in our evaluation (though we do not target a translation task).

Translationese is another phenomenon whereby translated language differs in systematic ways from non-translated utterances or texts in the target language. It is often a statistical phenomenon rather than a qualitative one: for example, a translated text may contain a higher than expected number of connective words, even if each individual use of those connectives sounds perfectly natural. Koppel and Ordan [36]; Jalota et al. [29]; Wein and Schneider [74] approach such a phenomenon from a computational perspective. Our study focuses on strategy-pairings where annotators find a strong qualitative difference in acceptability. Other methods would therefore be necessary to detect nuances associated with translationese.

Multilingual Representation and English Bias in LMs

Multilingual language models display remarkable ability to transfer knowledge encoded in different languages [59, 9]. These models share internal representations across languages [48, 6, 66], though there is evidence that even shared internal representations are not fully language agnostic: typologically similar languages are more likely to share representations [7, 68] and internal representations may be biased towards English [76, 71, 64, 61] or other languages overrepresented in training data [80].

In addition to representational biases, there is evidence that LLMs have English behavioural biases. Representation in multilingual pretraining data has long been observed to be a key factor in determining downstream performance in a language [78]. Some multilingual models display English word order preferences even in free word order languages [56], and overuse English-like vocabulary and syntax in non-English generated text [25]. We build on this line of work by investigating English bias across a range of controlled sentence-level preferences, asking whether models transfer English strategies to other languages.

Construction Grammar and LLMs

Construction Grammar (CxG) models linguistic knowledge as form–meaning pairings [18, 21, 10, 22]. Recent work has investigated whether LLMs represent English constructions [81, 65, 52]. However, there are comparatively few works investigating non-English constructions [72, 4, 28], and comparisons across languages are underexplored. MultiBLiMP [31] evaluates linguistic knowledge across 101 languages, but focuses on the subject–verb agreement strategy across all languages that use it, rather than contrasting different strategies used to express the same meaning across languages.

Recent work has attempted to unite typologically grounded approaches to construction grammar [12] with computational research by creating databases of typologically motivated comparative concepts [40], and investigating annotation of constructions across languages in Universal Dependencies [75, 13, 50 i.a.]. We build on these comparative concepts to test whether multilingual LLMs prefer native strategies or overgeneralize strategies from English.

Dataset Creation

We build a controlled dataset for testing whether multilingual language models exhibit cross-lingual bias in the strategies they use to express the same meaning. We focus specifically on transfer involving English: when two languages conventionally use different strategies to express comparable semantic content, does a model distinguish their language-specific preferences, or does it systematically favor the strategy associated with English?

We define the strategy contrasts underlying our evaluation (§ 3.1), then describe how we discover candidate contrasts from learner data (§ 3.2) and how the candidates are instantiated and validated by native speakers to construct CALQUE (§ 3.3).

Constructional Strategies and Cross-Lingual Transfer

We consider cases in which two languages use different constructional strategies to express the same semantic content. For each such contrast, we construct four realizations: (i) the native strategy of language AA realized in language AA, (ii) the strategy of language BB transferred into language AA, (iii) the native strategy of language BB realized in language BB, and (iv) the strategy of language AA transferred into language BB. Examples for different language pairs are shown in Table 1.

StrategySource LanguageTarget languageEnglish
GermanDas Wasser fühlt sich kalt an.The water feels itself cold.
EnglishDas Wasser fühlt kalt.The water feels cold.
SwedishJag bäddade sängen.I bedded the bed.
EnglishJag gjorde sängen.I made the bed.
FrenchJe joue du violoncelle.I play of the cello.
EnglishJe joue le violoncelle.I play the cello.
PortugueseEstá dando caimbra na nossa perna.It is giving cramp in our leg.
EnglishNós temos caimbra nas nossas pernas.We have cramps in our legs.
Chinese那茶喝起来有股烟味。That tea, when drunk, has a cigarette smell.
English那茶喝起来像香烟。That tea tastes like cigarettes.
Japaneseハエが目に入っている。A fly is entering eye.
English私は私の眼の中にハエを持っている。I have a fly in my eye.

Table 1. Examples of sentence pairs in each language.

We define a cross-linguistic strategy contrast as a pair of sentences from two languages using the same strategy, where the native strategy of one language, when transferred to the other language, makes the sentence less acceptable. For example, in Figure 1, the transitive strategy used in English for instrument playing, when transferred to French, creates the unacceptable sentence *Je joue le violoncelle.

Importantly, transferred strategies need not be categorically unacceptable. They may range from acceptable but dispreferred alternatives to strongly unacceptable realizations.

Strategy Discovery from Learner Data

To form an inventory of constructions that are likely to be realized by contrasting strategies across our target languages, we develop a semi-automatic discovery pipeline using the NAIST Lang-8 Learner Corpora [46]. We compare productions by learners with English as their reported L1 against their native corrections, using recurrent differences in function-word or syntactic realization as candidate strategy contrasts. Individual learner errors are used only for discovery and are not assumed to reflect L1 transfer.

Correction reconstruction. As Lang-8 Learner Corpora contain inconsistent correction tags, we reconstruct the corrected sentences using Llama3-70B Instruct [24] and GPT-5.1 [67] before further filtering.

Automatic candidate classification. We parse each learner–correction pair with Stanza [60], word-align by exact match and identify candidate strategy contrasts as differences in function-word realization or dependency relations that cannot be attributed solely to word order. We exclude pairs where only content words are edited, thereby retaining differences involving function words, either alone or together with content words. We further restrict the pairs to a Stanza-tokenized edit distance of at most two and group recurring patterns by UPOS and edit-operation signatures.

Candidate selection. We prioritize sufficiently frequent recurring patterns for manual inspection, using Tiny-Aya-Global [62] performance to prioritize patterns for which the model frequently fails to prefer the native correction. Model performance serves only as a prioritization signal and does not determine final construction selection. From the inspected candidates, we select constructions based on productivity, the presence of a plausible cross-lingual contrast with English, diversity, and interpretability as a strategy contrast. Detailed procedures for candidate filtering, grouping, ranking, and selection are provided in Appendix § 7.§ 5.

CALQUE

Construction inventory These constructions fall into three broader groups: (i) body-part constructions, in which a body part is realized as an affected entity (e.g., I brushed my teeth); (ii) everyday-activity constructions, covering conventionalized everyday activities (e.g., play the piano, wash the dishes, and eat dinner); and (iii) sensory-experience constructions, covering stimulus- and experiencer-oriented expressions of perception with contrasts in argument structure and intentionality (e.g., X smells like Y, hear, listen to).

Four-way sentence creation For each construction, we create instances following the four-way design in § 3.1. Starting from a natural English sentence expressing semantic content aa, native speakers produce a natural target-language realization, a target-language realization preserving the English strategy, and an English realization preserving the target-language strategy. This bidirectional design allows us to quantify constructional bias in both the target language and English. From this point on, we treat the English and target-language pairs as separate instances for evaluation unless otherwise noted.

Native preference annotation CALQUE was annotated by NLP experts who are native speakers of their respective languages. Because transferred constructions may vary gradiently in acceptability, annotators compare the language-internal and transferred realizations using a six-level preference scale (≫\gg, >>, ==, <<, ≪\ll, ≠\ne). While the arrows and == denote preference judgements on semantically matching sentences, ≠\ne indicates that both sentences are equally acceptable but do not mean the same thing (e.g., I had lunch ≠\ne I ate at midday). Because we evaluate sentence probabilities without context, and not translation, this makes them unsuitable for our purposes.

We retain instances with a native preference of ≫\gg or >> for the primary benchmark. The magnitude of this preference need not be symmetric across languages. For example, the German contrast in Table 1 may show a weaker native preference than its English counterpart. CALQUE is therefore parallel in semantic content and strategy, but not necessarily in the magnitude of native preference.

Japanese Baseline

To investigate whether models specifically favor English-derived strategies over strategies transferred from other languages, we additionally construct a Japanese control dataset.

In addition to the native Japanese strategy and the English strategy transferred into Japanese, we construct realizations in Japanese using strategies from German, Swedish, French, Portuguese, and Chinese. For each source language, we select semantic contents for which its strategy differs sufficiently from both the English and Japanese strategies. This allows us to compare transfer from English with transfer from other languages while holding the evaluation language, Japanese, constant.

Methodology

Sequence-Level Strategy Preference

We treat strategy preference as a property of the complete realization rather than of the locally contrasting tokens alone. A strategy choice can affect the probability assigned to subsequent material, even when that material is shared across the two realizations. For example, in a German contrast Ich habe mir den Arm gebrochen and Ich habe meinen Arm gebrochen, the context Ich habe does not yet identify the expression as a body-part construction, and the model’s preference between the two strategies may therefore be reflected in the probabilities of the subsequent tokens. Sequence-level scoring therefore allows us to capture the model’s preference for each strategy realization as a whole.

We quantify this preference using SLOR ([57, 38]), which normalizes sequence log probability by sequence length and the unigram probabilities of its lexical material:

SLOR⁡(S)=log⁡p(S)−log⁡puni(S)∣S∣.\operatorname{SLOR}(S) = \frac{\log p(S) - \log p_{\mathrm{uni}}(S)}{|S|}.

This normalization is particularly suitable for our setting because alternative strategy realizations can differ both in length and in the baseline frequency of their lexical material. SLOR thus allows us to compare how strongly a model favors each realization beyond these baseline differences.

We define the model’s strategy preference as

ΔSLOR=SLOR⁡(Snative)−SLOR⁡(Strans),\Delta_{\mathrm{SLOR}} = \operatorname{SLOR}(S_{\mathrm{native}}) - \operatorname{SLOR}(S_{\mathrm{trans}}),

where SnativeS_{\mathrm{native}} is the native strategy realization and StransS_{\mathrm{trans}} the transferred strategy realization. Positive values indicate a preference for the native strategy, whereas negative values indicate a preference for the transferred strategy.

Factors Associated with Strategy Preferences

We fit multiple linear regression models to identify factors associated with the models’ strategy preferences. Predictors include Model Size (total parameter count, including for MoE models), pretraining language proportion, Instruction Tuning, human preference annotations, and cross-linguistic structural features, including strategy distance, other-language strategy divergence, word-order divergence, and non-1:1 alignment rates.

For pretraining language proportion, we use the base-10 logarithm of language-specific corpus sizes from GlotCC ([32]) as a proxy for language availability in web-crawled pretraining data. Strategy distance quantifies how different the strategy instantiated in the target language is from native strategies in other languages. Other-language strategy divergence measures the variation among strategies instantiated in other languages to express the same semantic content. Word-order divergence quantifies the proportion of word-order inversions between the native strategies in the target language and English, while the non-1:1 alignment rate measures the proportion of words without one-to-one correspondences.2

Results

Cross-Lingual Asymmetries in Strategy Preferences

We first examine whether language models distinguish native strategies from transferred strategies across languages. We evaluate this ability using two complementary measures: the difference in SLOR between native and transferred strategy realizations (ΔSLOR\Delta\mathrm{SLOR}), and the failure rate, defined as the proportion of sentence pairs for which a model assigns a higher SLOR to the transferred strategy than to the native strategy. Figure 2 presents the distributions of ΔSLOR\Delta\mathrm{SLOR} for the largest model from each model family, while Figure 3 reports failure rates across models and languages.

Distribution of strategy preferences across languages

Figure 2. Distribution of strategy preferences across languages for sentence pairs with strong native-speaker preferences (≫), measured by ΔSLOR\Delta\mathrm{SLOR}. Following Zhong et al. [80], continually pretrained models (CPMs) are adapted from English-centric pretrained models through additional pretraining on target-language data, whereas balanced language models (BLMs) are pretrained from scratch with an emphasis on English and their respective focus languages. Positive values indicate a preference for the native strategy. Numbers in parentheses indicate sample sizes.

Figure 3. Failure rates (%) across languages, defined as the percentage of sentence pairs for which a model assigns a higher SLOR to the transferred strategy than to the native strategy.

Figure 2(a) reveals a systematic asymmetry in English-centric models’ strategy preferences. While these models generally assign higher SLOR to native English strategies than to strategies transferred from other languages, their preferences are substantially less consistent in non-English languages, including German, Swedish, Portuguese, Japanese, and Chinese. In these languages, the distributions exhibit more frequent negative values, indicating that models often fail to distinguish native strategies from English-transferred alternatives. This asymmetry suggests that English-centric models’ ability to distinguish strategies does not appear uniformly across languages. A linear mixed-effects model confirmed significantly higher ΔSLOR\Delta\mathrm{SLOR} in

English than the equally weighted mean across non-English languages for English-centric models (Wald tt-test, p=1.09×10−46p = 1.09 \times10^{-46}, df = 227.28).

However, Figure 2(b) shows that this asymmetry is not universal across model families. Models with a stronger training focus on non-English languages exhibit different preference patterns, often assigning larger positive ΔSLOR\Delta\mathrm{SLOR} values in their focus languages than in other languages, although these preferences do not necessarily exceed those observed in English. This contrast suggests that the linguistic composition of model training may be associated with the acquisition of strategy preferences. A linear mixed-effects model confirmed significantly higher ΔSLOR\Delta\mathrm{SLOR} in their focus languages than in other non-English languages (Wald tt-test, p=3.05×10−10p = 3.05 \times10^{-10}, df = 59.09).

Figure 3 presents the failure rates across models and languages. Both English-centric and non-English-centric models generally exhibit substantially lower failure rates in English than in other languages, indicating that the English advantage persists across model families.

We examine this asymmetry through a controlled comparison in Japanese (Figure 4). Holding the evaluation language, semantic content, and human preferences constant, we compare models’ preferences for native Japanese strategies over strategies transferred from other languages. Models often exhibit stronger preferences for native Japanese strategies when compared against transferred strategies from non-English languages than against transferred ones from English (Wald tt-test, p=7.53×10−9p = 7.53 \times10^{-9}, df = 78.84). This suggests that English-derived strategies are more likely to be favored over native Japanese strategies than transfers from other languages, providing further evidence of English-biased cross-lingual generalization.

Figure 4. Comparison of model preferences for native Japanese strategies over strategies transferred from English and other languages, measured by Δ\DeltaSLOR. All comparisons are evaluated in Japanese, with matched semantic content and restricted to sentence pairs with strong preference on the native Japanese strategy (≫\gg) over both the English-transferred and the corresponding non-English-transferred strategies. Positive values indicate a preference for the native Japanese strategy. Numbers in parentheses indicate sample sizes.

Human Preferences Are More Strongly Associated with Model Behavior Than Model Scale

To investigate which factors are associated with models’ strategy preferences, we conduct a multivariable regression analysis incorporating human preference annotations, model size, instruction tuning, and linguistically motivated features capturing cross-linguistic structural variation.

As shown in Figure 5, human preference emerges as a strong predictor among the examined features, with estimated coefficients of β=0.12−0.31\beta= 0.12 - 0.31 (p<0.001p < 0.001 for all four model families; two-way cluster-robust tt-tests). Although the magnitude of this association is modest, the positive coefficient suggests that models tend to assign larger preference differences to contrasts for which native speakers express stronger preferences for the native strategy. In contrast, model size exhibits little association with model preferences. This finding differs from the scaling-related improvements reported in multilingual linguistic evaluations such as MultiBLiMP (Jumelet et al., [31]), suggesting that increasing model capacity does not necessarily lead to corresponding improvements in distinguishing strategies. Similarly, instruction tuning and the examined linguistic features show relatively small associations with model preferences.

Figure 5. Standardized regression coefficients from multivariable linear regression models predicting Δ\DeltaSLOR. Predictors include human preference annotations, model size, instruction tuning, and cross-linguistic structural features. Positive coefficients indicate a positive association with preference for the native strategy.

English-Centric Models Share Similar Preference Patterns

We further investigate whether different models exhibit similar preferences across individual strategy contrasts. To this end, we compute pairwise Spearman rank correlations between models’ Δ\DeltaSLOR values over matched sentence pairs (Figure 6).

Figure 6. Pairwise Spearman rank correlations of Δ\DeltaSLOR across language models, computed over matched sentence pairs. Higher correlations indicate greater similarity in strategy preference distributions between models.

English-centric models exhibit high correlations with one another, with Spearman coefficients ranging from approximately 0.76 to 0.84. These correlations extend across different model families, suggesting that English-centric models share similar relative patterns of preference across strategy contrasts despite differences in architecture and model scale. In other words, sentence pairs that elicit relatively strong native-strategy preferences in one English-centric model tend to elicit similarly strong preferences in other English-centric models. In contrast, correlations between English-centric

and non-English-centric models are somewhat weaker, with coefficients around 0.36 to 0.64. This indicates that models with different linguistic training emphases do not necessarily exhibit the same relative preferences across strategy contrasts.

These findings complement the language-level asymmetries observed in § 5.1. While English-centric models vary in their overall performance, they exhibit substantial similarities in which strategy contrasts are easy or difficult. Lower correlations with non-English-centric models further suggest that these patterns are not universal properties of the evaluated constructions.

Overall, our results reveal systematic asymmetries in multilingual models’ strategy preferences. While English-centric models generally favor native strategies more strongly in English than in other languages, non-English-centric models show varying direction and magnitude of these preferences tied to their respective focus languages. Moreover, the substantial correlations across English-centric model families, together with the limited association of model size and instruction tuning with strategy preferences, suggest that shared aspects of training may be more relevant to constructional behavior than model scale. These findings highlight limitations in the robustness of language-specific constructional knowledge acquired under predominantly English-centric multilingual training.

Conclusion

We introduced CALQUE, a new benchmark grounded in the notion of constructional strategies, and evaluated 39 models across seven languages. Our results reveal systematic constructional bleeding in multilingual LLMs. Models frequently struggle to distinguish native from transferred strategies, with English-centric models exhibiting stronger preferences for native strategies in English than in other languages. Non-English-centric models show different patterns depending on their focus languages. Furthermore, our regression analysis shows that model preferences are more strongly associated with human preferences than with model size, instruction tuning, or other linguistic features. English-centric models also exhibit highly correlated preferences across model families.

Together, these findings suggest that multilingual LLMs acquire strategy knowledge unevenly, often favoring English-associated strategies even when processing non-English languages. The consistency of these patterns across model families highlights limitations of current English-dominated multilingual training practices.

Limitations

The set of languages covered in the dataset is limited by the native languages of the paper’s authors, and as such, while covering three major typological families, it is not as diverse as would be desirable to find the largest possible set of strategies for each sentence. Likewise, CALQUE only covers three groups of constructions. Finally, although we report IAA on a subset of our data, our primary annotations were mostly conducted by one native author annotator per language.

Interpretation of results. Given how the models tested are built, we are not able to report the exact pre-training data proportion for non-English-centric models, as their corpus often includes other languages. Also, we do not disentangle the effects of translationese in the pre-training data from constructional bleeding. Finally, due to time limitations, only one language—Japanese—was used as a fixed language control.

Ethical Considerations

Annotation for CALQUE was carried out primarily by the authors, with one additional contributor (American English native speaker) who voluntarily participated in the annotation process and consented to the use of their annotations for research purposes. For inter-annotator agreement, additional non-author annotators independently annotated a subset of the dataset (50 sentence pairs per language, except for Japanese, which included 100).

No annotators received monetary compensation for the primary annotation. Among the second annotators, one received monetary compensation for annotating a larger amount of Japanese sentence pairs, while the others participated voluntarily.

The non-author annotator spent approximately four hours on primary annotation, while author-annotators spent approximately 8–50 hours each on annotations. Additional non-author annotators for IAA spent approximately 20–30 minutes on the annotation task.

NAIST Lang-8 Learner Corpora were used solely to identify construction types which we focus on, and our dataset does not include sentences extracted from other corpora. Some of the code is written with the help of AI assistants, and verified by the authors.

Acknowledgments

The authors acknowledge the financial support by the Federal Ministry of Research, Technology and Space of Germany and by Sächsische Staatsministerium für Wissenschaft, Kultur und Tourismus in the programme Center of Excellence for AI-research “Center for Scalable Data Analytics and Artificial Intelligence Dresden/Leipzig”, project identification number: ScaDS.AI. Participation in this research by Georgetown affiliates was supported in part by NSF award IIS–2144881. Tiago Torrent is a Research Productivity Grantee of the Brazilian National Council for Scientific and Technological Development (CNPq grant 311241/2025–5). His participation in this research relies on funding granted to INCT-TILDIAR (CNPq grant 408490/2024–1). Marie-Catherine de Marneffe is a Research Associate of the Fonds de la Recherche Scientifique – FNRS. We thank Lori Levin for annotating English preferences and giving us constructive feedback on this paper. We are also thankful to the following colleagues for annotating for the following language preferences used to estimate Inter-Annotator Agreement: Andre Coneglian, for Brazilian Portuguese, Luke Gessler, for English, Norika Hiruta, for Japanese, Linus Högberg, for Swedish, Erika De Longueville, for French, Jacob Lee Suchardt, for German, and Wenhui Zhao, for Chinese. Part of this work was initiated by Dagstuhl Seminar 25301 “Linguistics and language models: What can they learn from each other”.

Appendix

Dataset Statistics

Our dataset consists of 669 instances of semantic content, each instantiated across multiple languages and constructional strategies.

Table 2 reports the number of sentence pairs with strong (≫\gg) and weak (>>) human preferences for native over transferred strategies, organized by construction group and the languages from which strategies are transferred.

NativeTransferredBody Part ≫\ggBody Part >>Everyday Activities ≫\ggEveryday Activities >>Sensory Experience ≫\ggSensory Experience >>
ende1627436661
ense78133922701
enfr188318213914
enpt6841848856
enzh60512271511
enja11766543468
deen2917113807312
seen6513315394634
fren6912621571060
pten311281144011518
zhen3457107653225
jaen54140177456043
jade1780043300
jase10574902038
jafr179069512412
japt2211699636
jazh34168402662

Table 2. Distribution of human preferences for native over transferred strategies across three construction groups. ≫\gg and >> indicate strong and weak preferences for the native strategy, respectively. Japanese-control comparisons use Japanese as the native strategy and a non-English language as the source of the transferred strategy.

Table 3 summarizes the dataset composition in terms of temporal interpretations, subject types, and construction groups.

CategoryValueCount
TemporalPast236
Ongoing133
Recurrent121
Current state77
Anterior54
Generic48
Subject1sg pronoun195
3sg pronoun192
1pl pronoun192
Common noun90
GroupEveryday activities279
Body part200
Sensory experience190
Total per language669

Table 3. Dataset composition by temporal interpretation, subject type, and construction group. The distribution is identical across all evaluated languages.

List of models

Table 4 reports the models used in our experiments.

FamilyModelCitation
Aya ExpanseCohereLabs/aya-expanse-8b<br>CohereLabs/aya-expanse-32bDang et al. [15]
Tiny AyaCohereLabs/tiny-aya-base<br>CohereLabs/tiny-aya-global<br>CohereLabs/tiny-aya-earth<br>CohereLabs/tiny-aya-fire<br>CohereLabs/tiny-aya-waterSalamanca et al. [62]
EuroLLMutter-project/EuroLLM-1.7B<br>utter-project/EuroLLM-1.7B-Instruct<br>utter-project/EuroLLM-9B<br>utter-project/EuroLLM-9B-Instruct<br>utter-project/EuroLLM-22B-2512<br>utter-project/EuroLLM-22B-Instruct-2512Martins et al. [42]
Gemma 4google/gemma-4-E2B<br>google/gemma-4-E2B-it<br>google/gemma-4-E4B<br>google/gemma-4-E4B-it<br>google/gemma-4-12B<br>google/gemma-4-12B-it<br>google/gemma-4-26B-A4B<br>google/gemma-4-26B-A4B-it<br>google/gemma-4-31B<br>google/gemma-4-31B-itTeam et al. [69]
Apertusswiss-ai/Apertus-8B-2509<br>swiss-ai/Apertus-8B-Instruct-2509<br>swiss-ai/Apertus-70B-2509<br>swiss-ai/Apertus-70B-Instruct-2509Hernández-Cano et al. [26]
openai/gpt-oss-120bOpenAI et al. [54]
Aleph-Alpha/Kolibri-1-BF16Alpha et al. [1]
hfl/chinese-llama-2-13bCui et al. [14]
maritaca-ai/sabia-7bPires et al. [58]
recogna-nlp/bode-13b-alpaca-pt-br-no-peftPaiola et al. [55]
OpenLLM-France/Claire-Mistral-7B-0.1Louradour et al. [41]
OpenLLM-France/Lucie-7BGouvert et al. [23]
tokyotech-llm/Swallow-13b-hfFujii et al. [19]; Okazaki et al. [53]
llm-jp/llm-jp-3-13bLLM-jp et al. [39]; Kiyomaru and Kodama [34]
llm-jp/llm-jp-3-172bLLM-jp et al. [39]; Kiyomaru and Kodama [35]
AIInstituteLNCC/manaca-1b-baseMenezes et al. [43]
AIInstituteLNCC/manaca-1b-instructMenezes et al. [43]

Table 4. Models evaluated, grouped by model family.

Annotation Scheme

We followed a set of annotation guidelines to ensure consistency and comparability across languages.

Annotation Guidelines Annotators were instructed to preserve the natural word order of each language as much as possible, including when applying strategies transferred from other languages. Differences outside the target construction were minimized to keep the sentence pairs as comparable as possible.

For constructions involving determiners, annotators were instructed not to introduce determiners within the target construction in languages that lack them, while preserving natural determiner usage elsewhere. Grammatical number was kept consistent within each pair unless directly relevant to the construction or necessary to preserve the naturalness or the intended meaning.

Preference Annotation We used a six-category preference annotation scheme:

  • ≫\gg: The native strategy is strongly preferred.

  • >>: The native strategy is weakly preferred.

  • ==: Both strategies are equally natural, or their naturalness depends on the intended nuance.

  • <<: The transferred strategy is weakly preferred.

  • ≪\ll: The transferred strategy is strongly preferred.

  • ≠\ne: Both sentences are natural but differ in meaning, preventing a direct preference comparison.

Since native strategies are defined as the conventionally preferred realizations, annotations of << or ≪\ll generally indicated that the sentence pair required revision. In such cases, annotators typically modified the sentences to ensure that the intended native and transferred strategies were correctly represented. Annotation guidelines were discussed and refined through regular meetings among contributors. To improve annotation efficiency, annotators were allowed to leave sentence pairs marked as "SELECT" when neither ≫\gg nor >> applied.

Dataset Expansion and Validation. We initially annotated 58 instances of semantic content from scratch, covering the construction groups, temporal interpretations, and subject types. Each instance was realized in four sentences across six languages, resulting in 1,392 realizations.

We subsequently expanded the dataset to 669 instances of semantic content, yielding 16,056 realizations. To facilitate this expansion, we provided GPT-5.1 [67] with previously annotated examples and used it to generate native and transferred strategy realizations. This approach reduced the need to write every sentence from scratch.

However, GPT-5.1 frequently failed to generate transferred strategies accurately. We therefore manually corrected many of the generated sentences through cross-linguistic comparison before submitting them to native speakers for final correction and validation.

Each language was annotated by at least one native speaker with expertise in NLP. We additionally conducted cross-linguistic checks to improve annotation consistency.

Japanese Control For the Japanese-control dataset, sentences using strategies transferred from languages other than English were written manually from scratch.

The Japanese-control annotations were conducted by a native Japanese speaker with at least B2-level proficiency in all evaluated languages except Swedish.

Robustness Analysis

To assess the robustness of our findings to the choice of scoring metric, we additionally evalu-

ate model preferences using ΔMeanLP [38].

Figure 7 shows a similar asymmetry in strategy preferences to that observed with ΔSLOR in Figure 2. English-centric models generally exhibit stronger preferences for native over transferred strategies in English than in other evaluated languages.

Distribution of strategy preferences across languages for English-centric and non-English-centric models

Figure 7. Distribution of strategy preferences across languages for sentence pairs with strong native-speaker preferences (≫), measured by ΔMeanLP. Positive values indicate a preference for the native strategy.

Figure 8 demonstrates that models struggle to distinguish the Japanese native strategies from the English-transferred strategies under ΔMeanLP.

Figure 8. Comparison of model preferences for native Japanese strategies over strategies transferred from English and other languages, measured by ΔMeanLP. All comparisons are evaluated in Japanese, with matched semantic content and restricted to sentence pairs with a strong preference for the native Japanese strategy (≫) over both the English-transferred and the corresponding non-English-transferred strategies. Positive values indicate a preference for the native Japanese strategy.

Candidate Selection Details

Initial filtering. We first restrict the candidate pairs to those with an edit distance of at least two and whose constructional differences (defined as difference in dependency relations or function words) involve either only function words or a combination of function and content words. After filtering, the candidate sets contain 4,879 pairs for German, 5,674 for French, 9,632 for Japanese, 551 for Portuguese, 575 for Swedish, and 8,429 for Chinese.

Chinese.

We exclude candidates whose differences involve only content words. Manual inspection showed that such cases yielded relatively few useful strategy contrasts and were predominantly attributable to spelling errors or erroneous linguistic analyses produced by Stanza. Pairs with an edit distance larger than 2 are excluded, as pairs with a larger edit distance tend to contain multiple errors and are therefore more difficult to interpret as a single strategy contrast.

Grouping and ranking. We group the remaining candidate pairs according to their operational error patterns, defined by an edit operation and the Part of Speech of the edited word. This results in 157 groups for German, 148 for French, 181 for Japanese, 102 for Portuguese, 116 for Swedish, and 188 for Chinese.

Within each language, we rank groups in ascending order of their average win rate under Tiny-Aya-Global. We define a pair as a win when the model assigns a higher ΔMeanLP to the native correction than to the corresponding learner sentence. Low average win rates therefore identify error patterns for which the model relatively often prefers the learner realization over the native correction, making them particularly relevant candidates for identifying strategy preferences that models might struggle in general.

To prioritize sufficiently frequent patterns, we define a language-specific frequency threshold as

τℓ=NℓGℓ,\tau_{\ell} = \frac{N_{\ell}}{G_{\ell}},

where NℓN_{\ell} is the total number of candidate pairs and GℓG_{\ell} is the number of groups for language ℓ\ell. The resulting thresholds are shown in Table 5. Among groups whose frequency exceeds this threshold, we manually inspect the five groups with the lowest win rates for each language.

LanguageCandidate pairsGroupsThreshold
German4,87915731.07
French5,67414838.34
Japanese9,63218153.22
Portuguese5511025.40
Swedish5751164.96
Chinese8,42918844.84

Table 5. Candidate statistics after filtering. The threshold corresponds to the mean number of candidate pairs per group within each language.

Subgroup inspection. For each group, we further define subgroups according to the surface forms involved in the corresponding edit operation. We manually inspect all subgroups whose win rate is below 100%, thereby excluding patterns for which Tiny-Aya-Global consistently prefers the native correction.

From these candidates, we select constructions according to four criteria: (i) productivity, such that the construction can be instantiated with multiple lexical items and contexts; (ii) cross-lingual contrast, such that the relative preference between the two strategies can plausibly reverse in English; (iii) diversity, to avoid selecting multiple near-duplicate phenomena; and (iv) interpretability, such that the contrast admits a clear constructional analysis.

Construction inventory. This procedure yields 51 candidate constructions across the six target languages. We instantiate these constructions in 105 sentences and annotate the resulting four-way strategy contrasts and sentence-level preferences. We then examine the inventory cross-linguistically and organize the constructions that are most relevant to our research questions into three broader construction groups:

  1. Body part as patient. Constructions involving body parts as affected entities, including expressions such as I broke my arm and I brushed my teeth.

  1. Everyday activities with salient patients. Constructions describing common activities involving a salient, often generic or superordinate patient, such as play the piano, play soccer, wash the dishes, and eat dinner. The patient may also have a specific referent in appropriate contexts (e.g., play the piano you gave me).

  1. Sensory experience. Constructions expressing sensory perception. These include stimulus-subject structures, optionally with an experiencer (e.g., something looks/smells/tastes/sounds/feels like X), and experiencer-subject structures involving verbs such as see, look at, watch, hear, and listen to. Some English verbs participate in multiple frames, and some contrasts additionally encode differences in intentionality, such as hear vs. listen to and see vs. look at.

Constructions collected during the discovery process that do not fall into these three groups are retained in the released dataset under the category OTHER.

Regression Features

Pretraining Language Proportion uses the sum of the context-length field across records ([32]).

Strategy Distance is measured by the mean normalized Levenshtein distance between the corresponding English literal translations of the target-language strategies:

Dl=1∣L∣−1∑k∈L, k≠lLev⁡(Sl,Sk)max⁡(∣Sl∣,∣Sk∣).D_l = \frac{1}{|L|-1}\sum_{k\in L,\,k\ne l}\frac{\operatorname{Lev}(S_l,S_k)}{\max(|S_l|,|S_k|)}.

where LL is the set of target languages that meet the strategy preference contrast and SlS_l and SkS_k denote the corresponding English literal translation for language ll and the other languages kk, respectively.

To distinguish language-specific divergence from the structural diversity of the remaining languages, we additionally compute Other-Language Strategy Divergence:

OLSD⁡l=∑k<mk,m∈L∖{l}Lev⁡(Sk,Sm)max⁡(∣Sk∣,∣Sm∣)×2(∣L∣−1)(∣L∣−2).(1)\operatorname{OLSD}_l = \sum_{\substack{k<m\\k,m\in L\setminus\{l\}}}\frac{\operatorname{Lev}(S_k,S_m)}{\max(|S_k|,|S_m|)} \times\frac{2}{(|L|-1)(|L|-2)}. \tag*{(1)}

Finally, using Awesome-Align ([17]) between the English and target-language native-strategy realizations (the tokenization was performed by Stanza), we compute Word-Order Divergence,

WO=# inverted aligned token pairs(naligned2),WO = \frac{\#\ \text{inverted aligned token pairs}}{\binom{n^{\text{aligned}}}{2}},

where nalignedn_{\mathrm{aligned}} is the number of aligned token pairs, and Non-1:1 Alignment Rate,

A=# non-1:1 target-language tokens# target-language tokens.A = \frac{\#\ \text{non-1:1 target-language tokens}}{\#\ \text{target-language tokens}}.

Multivariable regression was performed only on sentence pairs with complete metadata for all those predictors. Together, these predictors allow us to examine whether strategy preferences are associated with model-level properties, or structural properties of the cross-linguistic contrast.

Inter-Annotator Agreement

We sampled 50 semantically matched canonical sentence pairs from each target language, comprising 16 body-part constructions, 17 everyday-activity constructions, and 17 sensory-experience constructions. For Japanese, we sampled an additional 50 sentence pairs from the Japanese control split, including 10 contrasts for each of the five strategy-transfer languages. Canonical sentences typically feature first-person singular subjects and past-tense verbs, although some contain third-person common-noun subjects. Sentence pairs were randomly selected from this pool of canonical sentences.

We evaluate inter-annotator agreement (IAA) by distinguishing strong native preference (≫\gg) from all other labels, as well as preferences for the native strategy (≫\gg or >>) from all other labels. Table 6 reports percentage agreement and Cohen’s κ\kappa.

Lang.NN≫\gg vs others Agr.≫\gg vs others κ\kappa(≫\gg or >>) vs others Agr.(≫\gg or >>) vs others κ\kappa
en500.800.5670.780.556
de500.820.6130.780.551
se500.780.4280.940.849
fr500.820.5260.980.929
pt500.760.5250.860.451
zh500.740.4750.840.649
ja500.800.5970.860.610
ja-control500.860.7030.700.000

Table 6. Inter-annotator agreement by target language (Lang.). Strong preference distinguishes ≫\gg from all other labels, whereas preference direction distinguishes ≫\gg or >> from all other labels. Agreement (Agr) denotes the proportion of matching annotations, and κ\kappa denotes Cohen’s kappa.

The results show relatively high observed agreement across languages (0.74–0.98) for both types of IAA, while Cohen’s κ\kappa varies more substantially, partly reflecting differences in label distributions across languages in our dataset, which predominantly contains sentences favoring native strategies. In particular, the Japanese control split shows high agreement for both strong preference and preference direction, with substantial Cohen’s κ\kappa for strong preference but Cohen’s κ=0\kappa= 0 for preference direction. This stems from our dataset construction: the first annotator selected semantic contents that instantiate strategies that are distinct from Japanese and English, resulting in a split exclusively containing preferences towards native strategies.

Automatic Glossing

Our dataset includes linguistic glosses following the Leipzig Glossing Rules (Comrie et al. [8]). To reduce variation in automatic glossing, we define explicit rules for segmenting overt morphemes

and annotating grammatical properties such as case, tense, aspect, and reflexivity, with additional language-specific conventions. Function words and inflectional morphology receive detailed grammatical annotations, while lexical morphemes are generally glossed with English lexical equivalents. We generate the glosses using GPT-5.1 (Singh et al. [67]).

Implementation Details

SLOR and MeanLP Calculation. We load models using Hugging Face Transformers (Wolf et al. [77]) and run them in evaluation mode with BF16 precision on GPUs. Sentence log probabilities are computed by summing next-token log probabilities from causal language models, using each model’s tokenizer. We tokenize with add_special_tokens=True, excluding the initial token from scoring, and convert logits to FP32 before applying log-softmax. For SLOR normalization, we estimate language- and tokenizer-specific unigram distributions from the FLORES-200 development split (Team et al. [70]), using add_special_tokens=False and add-one smoothing (α=1\alpha= 1). We use greedy decoding and batch size 1. Periods were excluded during preprocessing of the dataset. We use simplified Chinese characters for Mandarin.

Metadata Construction. We tokenize sentences using Stanza and obtain word alignments using Awesome-Align with the awesome-without-co checkpoint, extracting representations from Transformer layer 8.

LMMs and Wald tt-tests. In § 5.1, we fit three linear mixed-effects models (LMMs) using the lme4 package (Bates et al., [3]) in R with restricted maximum likelihood (REML) estimation. We evaluate the corresponding fixed-effect contrasts using Wald tt-tests.

The first analysis examines whether Δ\DeltaSLOR differs significantly between English and non-English languages in English-centric models. We fit

ymℓi=β0+βℓ+um+vmℓ+wi+εmℓi,y_{m\ell i} = \beta_0 + \beta_{\ell} + u_m + v_{m\ell} + w_i + \varepsilon_{m\ell i},

where ymℓiy_{m\ell i} denotes Δ\DeltaSLOR for model mm, language ℓ\ell, and item ii. English is the reference level, such that βen=0\beta_{\mathrm{en}} = 0, β0\beta_0 represents the English intercept, and βℓ\beta_{\ell} represents the difference between language ℓ\ell and English. The terms umu_m, vmℓv_{m\ell}, and wiw_i denote random intercepts for model, model–language combination, and item, respectively, and εmℓi\varepsilon_{m\ell i} denotes the residual error. We test

H0:16∑ℓ≠enβℓ=0.H_0 : \frac{1}{6} \sum_{\ell\ne\mathrm{en}} \beta_{\ell} = 0.

The second analysis examines whether the focus language of a non-English-centric model exhibits a significant difference in Δ\DeltaSLOR. We fit

ymℓi=β0+βℓ+βfocusFmℓ+um+vmℓ+wi+εmℓi,y_{m\ell i} = \beta_0 + \beta_{\ell} + \beta_{\mathrm{focus}} F_{m\ell} + u_m + v_{m\ell} + w_i + \varepsilon_{m\ell i},

where FmℓF_{m\ell} is a binary indicator equal to one when language ℓ\ell is the focus language of model mm, and zero otherwise. The random intercepts have the same interpretation as in the first LMM. We test

H0:βfocus=0.H_0 : \beta_{\mathrm{focus}} = 0.

The third analysis examines whether the difference in Δ\DeltaSLOR between the Japanese-strategy and English-strategy conditions differs significantly from the difference between the Japanese-strategy and non-English-strategy conditions. We define

ymℓi=ΔSLORmℓiJA control−ΔSLORmℓiJA baseline,y_{m\ell i} = \Delta\mathrm{SLOR}^{\mathrm{JA\ control}}_{m\ell i} - \Delta\mathrm{SLOR}^{\mathrm{JA\ baseline}}_{m\ell i},

and fit

ymℓi=βℓ+um+vmℓ+wi+εmℓi.y_{m\ell i} = \beta_{\ell} + u_m + v_{m\ell} + w_i + \varepsilon_{m\ell i}.

Unlike the first two LMMs, this model does not include a common fixed intercept, so βℓ\beta_{\ell} represents the language-specific fixed effect. We test

H0:1L∑ℓ=1Lβℓ=0,H_0 : \frac{1}{L} \sum_{\ell=1}^{L} \beta_{\ell} = 0,

where LL denotes the number of languages. Degrees of freedom were approximated by Satterthwaite’s method (Satterthwaite, [63]), as implemented in the R package lmerTest (Kuznetsova et al., [37]).

Multivariable Regression and Two-Way Cluster-Robust tt-Tests

In § 5.2, we fit separate multivariable linear regression models for each of the four model families (Aya, EuroLLM, Gemma 4, and Apertus) using ordinary least squares (OLS). The regression model is specified as

ΔSLORi,f=β0,f+β1,flog⁡10(Sizei,f)+β2,fITi,f+β3,fPreferencei,f+⋯+εi,f,\begin{aligned} \Delta\mathrm{SLOR}_{i,f} ={}& \beta_{0,f} \\ &+ \beta_{1,f}\log_{10}(\mathrm{Size}_{i,f}) \\ &+ \beta_{2,f}\mathrm{IT}_{i,f} \\ &+ \beta_{3,f}\mathrm{Preference}_{i,f} \\ &+ \cdots+ \varepsilon_{i,f}, \end{aligned}

where ff denotes the model family, ii denotes an observation, and εi,f\varepsilon_{i,f} is the error term. Regression coefficients are estimated using OLS.

Following (Cameron et al., [5]), we assess statistical significance using two-way cluster-robust tt-tests, clustering by model and sentence-pair identity. Specifically, we implement the Cameron–Gelbach–Miller (CGM) two-way cluster-robust covariance estimator to account for dependence within both clustering dimensions.

We test the null hypothesis

H0:β3,f=0.H_0 : \beta_{3,f} = 0.

When computing the two-way cluster-robust covariance matrix, we apply a positive semidefinite (PSD) correction by replacing negative eigenvalues with zero. We use df=min⁡(Mf,Pf)−1df = \min(M_f, P_f) - 1, where MfM_f and PfP_f denote the numbers of model and sentence-pair clusters, respectively, in family ff.

Computational Resources

Model evaluations were conducted in two computing environments. The first was a server equipped with two NVIDIA RTX PRO 5000 Blackwell GPUs (48 GB VRAM each), a 32-core AMD EPYC CPU, and 64 GB of RAM. The second was an HPC cluster, where jobs were allocated 2-4 NVIDIA H100 GPUs, 16 CPU cores, and 192 GB of RAM. Evaluation times varied across models, ranging from two to over eight hours per model, including model loading and inference.

Failed Examples

Table 7 presents high-failure-rate canonical sentence examples across 28 English-centric models, along with preference annotations assigned by two native annotators.

Lang.Native strategyA1A2Transferred strategyFailed
deIch habe Zähne geputzt.>>Ich habe meine Zähne geputzt.27/28
deIch bin umgeknickt.≫≫Ich habe meinen Knöchel umgeknickt.26/28
deIch habe mir das Gesicht gewaschen.>>Ich habe mein Gesicht gewaschen.26/28
svAxeln gick ur led på mig.≫>Jag dislokerade min axel.28/28
svMaten luktade vitlök.>>Måltiden luktade som vitlök.27/28
svJag borstade tänderna.>>Jag borstade mina tänder.25/28
frJ’avais une crampe à la jambe.>≫J’avais une crampe dans ma jambe.25/28
frJ’avais une mouche dans l’oeil.≫≫J’avais une mouche dans mon oeil.22/28
frJe me suis coupé les cheveux.≫≫J’ai coupé mes cheveux.19/28
ptA voz dela parecia de criança.>≫A voz dela soava como a voz de uma criança.28/28
ptEu lavei o rosto.>≫Eu lavei meu rosto.28/28
ptEu desloquei o ombro.>≫Eu desloquei meu ombro.27/28
zh我挠了背>>我挠了我的背28/28
zh我刷了牙>>我刷了我的牙27/28
zh我昨天拉了大提琴。≫>我昨天玩了大提琴。26/28
jaチェスをした。>≫チェスを遊びました。28/28
ja背中を掻いた。>>私は私の背中を掻いた。28/28
ja髪をとかした。>>私は私の髪をとかした。28/28

Table 7. High-failure-rate canonical sentence examples across 28 English-centric models in six languages. For each language, the three sentence pairs are shown, for which both annotators preferred the native strategy (> or ≫). The A1 and A2 columns report the respective native-speaker preference annotations. The Failed column indicates the number of models assigning a lower SLOR to the native strategy than to the transferred strategy, out of 28 evaluated models.

Failure Rates

Table 8 presents the failure rates across models.

ModelITENDESVFRPTJAZH
Multilingual models
Tiny AyaBase6.635.544.322.059.652.631.8
Tiny AyaEarth✓9.548.246.622.966.849.740.9
Tiny AyaFire✓8.144.248.923.567.350.940.3
Tiny AyaGlobal✓10.248.746.121.866.649.741.2
Tiny AyaWater✓10.047.547.322.266.653.240.3
Aya Expanse8B6.151.165.024.961.950.936.2
Aya Expanse32B5.941.453.922.459.648.236.2
Gemma 4E2B5.737.850.723.363.246.231.1
Gemma 4E2B✓45.448.247.335.957.279.461.6
Gemma 4E4B5.441.147.719.959.445.937.1
Gemma 4E4B✓50.752.548.228.566.875.767.0
Gemma 412B6.240.438.625.657.843.233.0
Gemma 412B✓52.848.561.468.850.064.442.8
Gemma 426B-A4B5.732.943.018.258.347.233.0
Gemma 426B-A4B✓36.553.447.326.250.055.560.7
Gemma 431B5.137.843.018.757.445.936.5
Gemma 431B✓29.354.147.331.563.565.342.1
Apertus8B9.538.847.315.559.051.651.3
Apertus8B✓11.141.453.916.667.759.553.5
Apertus70B7.534.042.316.161.049.945.0
Apertus70B✓6.833.143.222.862.652.046.5
EuroLLM1.7B8.445.443.924.565.554.539.0
EuroLLM1.7B✓7.447.343.625.464.852.436.8
EuroLLM9B6.043.543.424.963.051.446.9
EuroLLM9B✓5.742.345.723.563.253.446.5
EuroLLM22B6.445.943.921.859.651.139.6
EuroLLM22B✓7.842.144.322.260.852.237.4
GPT-oss120B✓8.852.553.419.965.745.345.3
Non-English-centric models
Kolibri 170B✓24.839.263.629.863.974.052.2
Claire-Mistral7B6.962.258.622.671.569.448.1
Lucie7B5.537.455.928.369.566.362.3
Sabiá7B10.255.650.528.158.572.365.7
Bode13B✓10.556.754.127.270.264.761.9
Manaca1B18.154.155.538.452.060.173.9
Manaca1B✓17.449.955.241.553.857.865.1
LLM-jp 313B6.967.853.241.367.344.348.1
LLM-jp 3172B6.064.549.534.865.941.041.8
Swallow13B6.266.250.728.972.637.854.4
Chinese Llama 213B4.462.954.124.370.063.641.2

Table 8. Failure rates (%) across languages, defined as the percentage of sentence pairs for which a model assigns a higher SLOR to the transferred strategy than to the native strategy. Darker red cells indicate higher failure rates. ✓ in the IT column marks instruction-tuned models.

References

  1. [1]Aleph Alpha, Moritz Althaus, Artur Baranowski, Michael Barlow, Fabien C. Y. Bunereau, Felix Berkenkamp, Amrita Bhattacharjee, Lukas Blübaum, Duncan Blythe, Augustijn de Boer, Bastian Boll, Thomas F. Burns, Björn Deiseroth, Niko Dürr, Béni Balázs Egressy, Niklas Finken, Niklas Freymuth, Fabian Gebhart, Max Hammer, and 48 others. 2026. Kolibri: A sovereign European model on the pareto frontier. Preprint, arXiv:2610.31945.arxiv.org/abs/2610.31945
  2. [2]Tatsuya Aoyama and Nathan Schneider. 2024. Modeling nonnative sentence processing with L2 language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4927–4940, Miami, Florida, USA. Association for Computational Linguistics.DOI
  3. [3]Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1–48.
  4. [4]Bastian Bunzeck, Daniel Duran, and Sina Zarriε. 2025. Do construction distributions shape formal language learning in German BabyLMs? In Proceedings of the 29th Conference on Computational Natural Language Learning, pages 169–186, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2503.11593
  5. [5]A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller. 2011. Robust inference with multiway clustering. Journal of Business & Economic Statistics, 29(2):238–249.DOI
  6. [6]Tyler A. Chang, Zhuowen Tu, and Benjamin K. Bergen. 2022. The geometry of multilingual language model representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 119–136, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  7. [7]Rochelle Choenni and Ekaterina Shutova. 2022. Investigating language relationships in multilingual sentence encoders through the lens of linguistic typology. Computational Linguistics, 48(3):635–672.
  8. [8]Bernard Comrie, Martin Haspelmath, and Balthasar Bickel. 2015. The Leipzig glossing rules: Conventions for interlinear morpheme-by-morpheme glosses. Department of Linguistics, Max Planck Institute for Evolutionary Anthropology, and Department of Linguistics, University of Leipzig. Revised version of 31 May 2015.
  9. [9]Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.arxiv.org/abs/1911.02116
  10. [10]William Croft. 2001. Radical construction grammar: Syntactic theory in typological perspective. Oxford University Press, USA.
  11. [11]William Croft. 2016. Comparative concepts and language-specific categories: Theory and practice. Linguistic Typology, 20(2):377–393.DOI
  12. [12]William Croft. 2022. Morphosyntax: Constructions of the World’s Languages. Cambridge Textbooks in Linguistics. Cambridge University Press.
  13. [13]William Croft and Joakim Nivre. 2025. Verbal predication constructions in Universal Dependencies. In Proceedings of the Second International Workshop on Construction Grammars and NLP, pages 50–60, Düsseldorf, Germany. Association for Computational Linguistics.
  14. [14]Yiming Cui, Ziqing Yang, and Xin Yao. 2024. Efficient and effective text encoding for Chinese LLaMA and Alpaca. Preprint, arXiv:2304.08177.arxiv.org/abs/2304.08177
  15. [15]John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, and 26 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261.arxiv.org/abs/2412.04261
  16. [16]Paolo Della Putta. 2016. Do we also need to unlearn constructions? The case of constructional negative transfer from Spanish to Italian and its pedagogical implications. In Sabine De Knop and Gaëtanelle Gilquin, editors, Applied Construction Grammar, pages 237–268. De Gruyter Mouton.DOI
  17. [17]Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112–2128, Online. Association for Computational Linguistics.arxiv.org/abs/2101.08231
  18. [18]Charles J. Fillmore, Paul Kay, and Mary Catherine O’Connor. 1988. Regularity and idiomaticity in grammatical constructions: The case of let alone. Language, 64(3):501–538.DOI
  19. [19]Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Ida, Masanori Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual LLM adaptation: Enhancing Japanese language capabilities. In First Conference on Language Modeling.
  20. [20]Talmy Givón. 1979. On Understanding Grammar. Academic Press.DOI
  21. [21]Adele E. Goldberg. 1995. Constructions: A Construction Grammar Approach to Argument Structure. University of Chicago Press.
  22. [22]Adele E. Goldberg. 2006. Constructions at Work: The Nature of Generalization in Language. Oxford University Press, USA.
  23. [23]Olivier Gouvert, Julie Hunter, Jérôme Louradour, Christophe Cerisara, Evan Dufraisse, Yaya Sy, Laura Rivière, Jean-Pierre Lorré, and OpenLLM-France community. 2025. The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation. Preprint, arXiv:2503.12294.arxiv.org/abs/2503.12294
  24. [24]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783.arxiv.org/abs/2407.21783
  25. [25]Yanzhu Guo, Simone Conia, Zelin Zhou, Min Li, Saloni Potdar, and Henry Xiao. 2025. Do large language models have an English accent? Evaluating and improving the naturalness of multilingual LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3823–3838, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2410.15956
  26. [26]Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pásztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Durech, Ido Hakimi, Juan Garcia Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, and 83 others. 2026. Apertus: Democratizing open and compliant LLMs for global language environments. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 46877–46955, San Diego, California, United States. Association for Computational Linguistics.
  27. [27]Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger P. Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational Linguistics.arxiv.org/abs/2005.03692
  28. [28]Xinyao Huang, Yue Pan, Stefan Hartmann, and Yanning Yang. 2025. Assessing minimal pairs of Chinese verb-resultative complement constructions: Insights from language models. In Proceedings of the Second International Workshop on Construction Grammars and NLP, pages 144–150, Düsseldorf, Germany. Association for Computational Linguistics.
  29. [29]Rricha Jalota, Koel Chowdhury, Cristina España-Bonet, and Josef van Genabith. 2023. Translating away translationese without parallel data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7086–7100, Singapore. Association for Computational Linguistics.arxiv.org/abs/2310.18830
  30. [30]Scott Jarvis and Aneta Pavlenko. 2008. Crosslinguistic influence in language and cognition. Routledge, New York.DOI
  31. [31]Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2026. MultiBLiMP 1.0: A massively multilingual benchmark of linguistic minimal pairs. Transactions of the Association of Computational Linguistics, 14:193–216.
  32. [32]Amir Hossein Kargaran, François Yvon, and Hinrich Schütze. 2024. GlotCC: An open broad-coverage CommonCrawl corpus and pipeline for minority languages. In Advances in Neural Information Processing Systems, volume 37, pages 16983–17005. Curran Associates, Inc.arxiv.org/abs/2410.23825
  33. [33]Edward L. Keenan and Bernard Comrie. 1977. Noun phrase accessibility and Universal Grammar. Linguistic Inquiry, 8:63–99.
  34. [34]Hirokazu Kiyomaru and Takashi Kodama. 2024a. LLM-jp-3-13b: Model card. https://huggingface.co/llm-jp/llm-jp-3-13b. Accessed: 2026-10-11.
  35. [35]Hirokazu Kiyomaru and Takashi Kodama. 2024b. LLM-jp-3-172b: Model card. https://huggingface.co/llm-jp/llm-jp-3-172b. Accessed: 2026-10-30.
  36. [36]Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1318–1326, Portland, Oregon, USA. Association for Computational Linguistics.
  37. [37]Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen. 2017. lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13):1–26.
  38. [38]Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202–1241.DOI
  39. [39]LLM-jp, Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazuki Fujii, Kensuke Fukumoto, Takuya Fukushima, Namgi Han, Yuto Harada, Chikara Hashimoto, Tatsuya Hiroaka, Shohei Hisada, Sosuke Hosokawa, Lu Jie, Keisuke Kamata, Teruhito Kanazawa, and 63 others. 2024. LLM-jp: A cross-organizational project for the research and development of fully open Japanese LLMs. Preprint, arXiv:2407.03963.
  40. [40]Arthur Lorenzi, Peter Ljunglöf, Ben Lyngfelt, Tiago Timponi Torrent, William Croft, Alexander Ziem, Nina Böbel, Linnéa Bäckström, Peter Uhrig, and Ely E. Matos. 2024. MoCCA: A model of comparative concepts for aligning constructions. In Proceedings of the 20th Joint ACL - ISO Workshop on Interoperable Semantic Annotation @ LREC-COLING 2024, pages 93–98, Torino, Italia. ELRA and ICCL.
  41. [41]Jérôme Louradour, Julie Hunter, Ismaïl Harrando, Guokan Shang, Virgile Rennard, and Jean-Pierre Lorré. 2024. Claire: Large language models for spontaneous French dialogue. In Actes de la 31ème Conférence sur le Traitement Automatique des Langues Naturelles, volume 1 : articles longs et prises de position, pages 530–548, Toulouse, France. ATALA and AFPC.
  42. [42]Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. EuroLLM: Multilingual language models for Europe. Preprint, arXiv:2409.16235.arxiv.org/abs/2409.16235
  43. [43]Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, and Fabio Andre Machado Porto. 2026a. Manacá-1B: An open, reproducible Brazilian-Portuguese language model and a tokenizer-aware, paired evaluation. Preprint, arXiv:2608.30114.arxiv.org/abs/2608.30114
  44. [45]Kanishka Misra, Julia Rayz, and Allyson Ettinger. 2023. COMPS: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2928–2949, Dubrovnik, Croatia. Association for Computational Linguistics.arxiv.org/abs/2210.01963
  45. [46]Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning SNS for automated Japanese error correction of second language learners. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 147–155, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
  46. [47]Lucía Molina and Amparo Hurtado Albir. 2004. Translation techniques revisited. A dynamic and functionalist approach. Meta: Translators’ Journal, 47:498–512.
  47. [48]Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. 2021. First align, then predict: Understanding the cross-lingual ability of multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2214–2231, Online. Association for Computational Linguistics.arxiv.org/abs/2101.11109
  48. [49]Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for Computational Linguistics.arxiv.org/abs/2010.00133
  49. [50]Joakim Nivre. 2025. Constructions and strategies in Universal Dependencies. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 419–423, Tallinn, Estonia. University of Tartu Library.
  50. [51]Miyu Oba, Tatsuki Kuribayashi, Hiroki Ouchi, and Taro Watanabe. 2023. Second language acquisition of neural language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13557–13572, Toronto, Canada. Association for Computational Linguistics.
  51. [52]Miyu Oba and Saku Sugawara. 2026. CxMP: A linguistic minimal-pair benchmark for evaluating constructional understanding in language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 45949–45963, San Diego, California, United States. Association for Computational Linguistics.
  52. [53]Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. 2024. Building a large Japanese web corpus for large language models. In First Conference on Language Modeling.arxiv.org/abs/2404.17733
  53. [54]OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastian Bubeck, Che Chang, and 107 others. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  54. [55]Pedro Henrique Paiola, Gabriel Lino Garcia, and João Paulo Papa. 2024. Phi-bode.DOI
  55. [56]Isabel Papadimitriou, Kezia Lopez, and Dan Jurafsky. 2023. Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1194–1200, Dubrovnik, Croatia. Association for Computational Linguistics.
  56. [57]Adam Pauls and Dan Klein. 2012. Large-scale syntactic language modeling with treetlets. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 959–968, Jeju Island, Korea. Association for Computational Linguistics.
  57. [58]Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. 2023. Sabiá: Portuguese large language models. In Intelligent Systems, pages 226–240, Cham. Springer Nature Switzerland.DOI
  58. [59]Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.arxiv.org/abs/1906.01502
  59. [60]Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online. Association for Computational Linguistics.arxiv.org/abs/2003.07082
  60. [61]Alan Saji, Jaavid Aktar Husain, Thanmay Jayakumar, Raj Dab­re, Anoop Kunchukuttan, and Ratish Puduppully. 2025. RomanLens: The role of latent Romanization in multilinguality in LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 26410–26429, Vienna, Austria. Association for Computational Linguistics.
  61. [62]Alejandro R. Salamanca, Diana Abagyan, Daniel D’souza, Ammar Khairi, David Mora, Saurabh Dash, Viraat Aryabumi, Sara Rajaee, Mehrnaz Mofakhami, Ananya Sahu, Thomas Euyang, Brittawnya Prince, Madeline Smith, Hangyu Lin, Acyr Locatelli, Sara Hooker, Tom Kocmi, Aidan Gomez, Ivan Zhang, and 7 others. 2026. Tiny aya: Bridging scale and multilingual depth. Preprint, arXiv:2603.11510.arxiv.org/abs/2603.11510
  62. [63]F. E. Satterthwaite. 1946. An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6):110–114.pubmed.ncbi.nlm.nih.gov/20287815
  63. [64]Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do multilingual LLMs think in English? In ICLR 2025 Workshop on Building Trust in Language Models and Applications.arxiv.org/abs/2502.15603
  64. [65]Wesley Scivetti, Ethan Gotlieb Wilcox, Nathan Schneider, Kanishka Misra, and Leonie Weissweiler. 2026. Language models learn constructional semantics, not to mention syntax: Investigating LM understanding of paired-focus constructions. In Proceedings of the 30th Conference on Computational Natural Language Learning, pages 250–267, San Diego, California, USA. Association for Computational Linguistics.
  65. [66]Nadav Shani and Ali Basirat. 2025. Language dominance in multilingual large language models. In Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 137–148, Suzhou, China. Association for Computational Linguistics.DOI
  66. [67]Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksandr Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb, Alex Beutel, Alex Karpenko, and 467 others. 2026. OpenAI GPT-5 system card. Preprint, arXiv:2601.03267.arxiv.org/abs/2601.03267
  67. [68]Karolina Stanczak, Edoardo Ponti, Lucas Torroba Hennigen, Ryan Cotterell, and Isabelle Augenstein. 2022. Same neurons, different languages: Probing morphosyntax in multilingual pre-trained models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1589–1598, Seattle, United States. Association for Computational Linguistics.
  68. [69]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Alghayeres, Alek Andreev, Olivier Bachem, Ian Bal­lantyne, Cormac Brick, Victor Čarbune, Michelle Casbon, Mayank Chaturvedi, Aditya Chawla, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, and 304 others. 2026. Gemma 4 technical report. Preprint, arXiv:2607.02770.
  69. [70]NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 others. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.arxiv.org/abs/2207.04672
  70. [71]Katharina Trinley, Toshiki Nakai, Tatiana Anikina, and Tanja Baeumel. 2025. What language(s) does Aya-23 think in? How multilinguality affects internal language representations. In Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models, pages 159–171, Varna, Bulgaria. INCOMA Ltd., Shoumen, BULGARIA.arxiv.org/abs/2507.20279
  71. [72]Yu-Hsiang Tseng, Cing-Fang Shih, Pin-Er Chen, Hsin-Yu Chou, Mao-Chang Ku, and Shu-Kai Hsieh. 2022. CxLM: A construction and context-aware language model. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6361–6369, Marseille, France. European Language Resources Association.DOI
  72. [73]Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohan­aney, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392.
  73. [74]Shira Wein and Nathan Schneider. 2024. Lost in translationese? Reducing translation effect using Abstract Meaning Representation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 753–765, St. Julian’s, Malta. Association for Computational Linguistics.
  74. [75]Leonie Weissweiler, Nina Böbel, Kirian Guller, Santiago Herrera, Wesley Scivetti, Arthur Lorenzini, Nurit Melnik, Archna Bhatia, Hinrich Schütze, Lori Levin, Amir Zeldes, Joakim Nivre, William Croft, and Nathan Schneider. 2024. UCxn: Typologically informed annotation of constructions atop Universal Dependencies. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16919–16932, Torino, Italia. ELRA and ICCL.
  75. [76]Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? On the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2402.10588
  76. [77]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.DOI
  77. [78]Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics.arxiv.org/abs/2005.09093
  78. [79]Aditya Yadavalli, Alekhya Yadavalli, and Vera Tobin. 2023. SLABERT talk pretty one day: Modeling second language acquisition with BERT. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11763–11777, Toronto, Canada. Association for Computational Linguistics.
  79. [80]Chengzhi Zhong, Qianying Liu, Fei Cheng, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, and Sadao Kurohashi. 2025. What language do non-English-centric large language models think in? In Findings of the Association for Computational Linguistics: ACL 2025, pages 26333–26346, Vienna, Austria. Association for Computational Linguistics.
  80. [81]Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David R. Mortensen, and Lori Levin. 2024. Constructions are so difficult that even large language models get them right for the wrong reasons. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3804–3811, Torino, Italia. ELRA and ICCL.arxiv.org/abs/2403.17760

Paper details

vlnc:W_yj7ve9zg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents