betaFeedback

Reading v1View latest version

WorksComputer Science

Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models

Abstract

Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g. by removing all nouns or shuffling the words. Does the same hold for large language models (LLMs), and if so, is it because they have seen the benchmarks during pre-training? In this paper we revisit these perturbation experiments with eight LLMs alongside fine-tuned encoders on GLUE and the three rounds of ANLI. Regardless of model size, every system that answers directly retains 36-72% of its above-chance performance without nouns on MNLI compared to only 14-29% on ANLI. Only a model that reasons before answering departs from this pattern. We also check contamination indicators (including exact nn-gram counts in the public pre-training data of two of the models) and find no link between exposure and robustness. Our strongest evidence comes from 368 new, human-validated items that none of the models can have seen. All eight LLMs keep more of their performance without nouns on items written in MNLI's style than on items written without its cues. When the perturbation instead removes the function words carrying such cues, this advantage reverses. In light of these results, we argue that robustness to perturbation on familiar NLI benchmarks reflects dataset shortcuts rather than memorisation, and tells us more about the benchmark than about the model.

Introduction

Benchmark performance remains the primary evidence supporting claims regarding the language understanding capabilities of current models. However, achieving high scores on benchmarks does not mean that the model solves the problem as intended by the benchmark. One method to verify this is to transform the input data by removing information that humans need, and then check if the model still makes accurate predictions. While previous studies referred to this as data corruption, we use word-class ablation for word removal and word-order perturbation for changing word order. Talman et al. (2021) and Talman et al. (2022) performed this type of evaluation on fine-tuned encoders. Accuracy was largely maintained on MNLI and various GLUE tasks even when nouns, verbs, or other parts of speech were all removed or when the word order was shuffled. This was the case even though human readers found many of the resulting items difficult or impossible to label.

For large language models (LLMs), the same questions are raised again in different settings. LLMs are pre-trained on web-scale corpora that may already contain benchmark items, and are increasingly being evaluated without task-specific fine-tuning. If an LLM maintains accuracy on perturbed MNLI items, two explanations are possible. The first is memorisation: the model recognises the item even after perturbation because it encountered it during pre-training. The second is that the item can be solved from superficial cues that remain after perturbation, such as negation words in the hypothesis (shortcuts or annotation artefacts). These two explanations lead to different empirical predictions. According to the memorisation account, items present in the pre-training corpus should be more robust than absent items, and the effect should be greatest on the benchmarks with the highest potential for leakage. Conversely, if robustness stems from shortcuts, items that a model can solve solely from the hypothesis should be more robust regardless of prior exposure, and this advantage should disappear if the cue-providing words themselves are removed.

This paper tests both of these predictions. First, we reproduce previously published encoder results on the MNLI dataset within 1 percentage point, and then re-examine the perturbation experiments of Talman et al. (2021, 2022) using four open-weight LLMs, four hosted LLMs, and fine-tuned encoders. The main contributions of this paper are as follows:

  • We demonstrate that the gap between MNLI and ANLI lies in the dataset, not the model. Regardless of model size, all systems that answer directly keep much more of their performance on MNLI than on ANLI when nouns are removed (36–72% vs. 14–29%). Only the model that reasons before answering deviates from this pattern.

  • Introducing perturbations to the inputs causes the LLMs to be biased toward a single label, so that raw accuracy can underestimate how much performance is retained, and we measure robustness after correcting for these biases.

  • We find no association between the three contamination indicators and robustness. Rather, in all three of the most robust LLMs on MNLI, items solvable solely from the hypothesis are more robust, and this advantage is reversed when function words are removed.

  • We construct 368 new, human-validated NLI items written either in the MNLI style or without cues. All eight LLMs exhibit the same shortcut pattern in these unseen items.

  • We compare MNLI with BioASQ, a benchmark without hypothesis cues. BioASQ is also robust to perturbation, but for a different reason: many of its questions can be answered directly from biomedical knowledge.

In other words, robustness against perturbation alone cannot prove language understanding or memorisation. What it reveals depends on which cues remain intact even after perturbation is applied.

Code and aggregate results are available at https://github.com/aarnetalman/shortcuts-or-memorisation; the fresh items will be released on publication.

Related Work

Perturbed input. Several studies have shown that models fine-tuned on NLI and other GLUE tasks are remarkably insensitive to changes in input form. Sinha et al. (2021b) show that NLI models predict the same label across many permutations of a sentence pair, while Pham et al. (2021) and Gupta et al. (2021) report comparable findings for word order on GLUE. Even at the pre-training stage, Sinha et al. (2021a) find that training on shuffled text causes very little degradation on downstream tasks. For word removal, Talman et al. (2021) ablate entire word classes in MNLI and ANLI and find that models trained on the ablated data still reach high accuracy. Talman et al. (2022) extend these experiments across GLUE and show that the outcome varies considerably by task. Removing nouns and verbs has the largest impact, whereas some tasks barely change at all. Nearly all of this work, however, predates instruction-tuned LLMs, which are normally evaluated without task-specific training.

Shortcuts in NLI. Crowdsourced NLI corpora are known to contain annotation artefacts that allow a classifier to predict the label without looking at the premise. Gururangan et al. (2018) and Poliak et al. (2018) demonstrate that hypothesis-only models perform well above chance on both SNLI and MNLI, with negation and other function words serving as strong indicators of contradiction. In addition, McCoy et al. (2019) show that NLI models rely heavily on lexical overlap heuristics. ANLI (Nie et al., 2020) was constructed adversarially to address this issue, asking annotators to write hypotheses that fool strong models and thereby reducing the scope for such cues.

Contamination. As pre-training corpora grow in scale, evaluation benchmarks are increasingly likely to be included in them (Sainz et al., 2023). Where the pre-training data are accessible, benchmark overlap can be measured directly, e.g. through exact nn-gram counts (Liu et al., 2024). When the training data are not open, contamination has to be probed indirectly through model behaviour. Existing approaches either ask the model to complete or fill in parts of benchmark items Golchín and Surdeanu, [5]; Deng et al. [3], or inspect the likelihood assigned to an item’s least probable tokens (Shi et al., 2024; Zhang et al., 2025).

Answer bias. When LLMs are prompted to output a class label, they often favour particular answers regardless of the input, although recentring the answer scores removes this preference (Zhao et al., 2021; Holtzman et al., 2021). Our work connects these threads: on a shared set of models and items, we test whether robustness under perturbation follows pre-training contamination or dataset shortcuts, and we pull the two apart using newly written items that none of the models can have seen.

Experimental Set-up

Perturbations

We use the perturbations of Talman et al. (2022), applied independently to each text field of an item, and keep the earlier condition names (CORRUPT-TRAIN, CORRUPT-TEST). Words are tagged with the universal part-of-speech tagset (Petrov et al. (2012)) using NLTK (Bird et al. [1]). drop-X removes all words of class X (e.g. drop-NOUN); keep-X+Y keeps only words of the listed classes (e.g. keep-NOUN+VERB, which removes all function words); shuffle-kk splits each field into consecutive kk-grams and shuffles their order (Pham et al. (2021)); hyp-only removes the premise. Perturbed text is detokenised and fields that become empty are kept, as in Talman et al. (2022); for the reproduction of Talman et al. (2021) we use that paper’s conventions and, where available, its released files. In total there are 19 perturbations; for the hosted models we use a core set of ten.1

Data

We evaluate on the development sets of the GLUE tasks (Wang et al. (2018)), using the matched development set for MNLI (Williams et al. (2018)), and on the test sets of the three ANLI rounds (Nie et al. (2020)). The original splits and metrics are kept: accuracy for MNLI, QNLI, RTE, SST-2 and ANLI, F1 for MRPC and QQP, Matthews correlation for CoLA and Pearson correlation for STS-B. For the hosted models, for five-shot prompting and for the contamination checks we use a fixed label-stratified sample of 2,000 items for MNLI and QQP.

Fresh items. As a control that no model can have seen, we built 368 new NLI items (Appendix § 12). Premises are sentences from Wikipedia articles created, and from Wikipedia’s daily current-events pages written, after 1 September 2026, excluding any sentence with a 13-gram found in OLMo’s pre-training data. Two models that we do not evaluate, Llama 4 Maverick and GPT-5.5, wrote one hypothesis per label for each premise, in two styles: following MNLI’s annotation instructions with MNLI training pairs as examples (MNLI style), and with instructions that rule out the best-known hypothesis-only cues such as negation in contradictions (cue-free). A further set pairs sentences from the first and the current version of new Wikipedia articles. Fourteen volunteers labelled the items without seeing the intended label; we keep items whose labels all agree with the intended one and that nobody flagged, giving 168 MNLI-style, 148 cue-free and 52 naturally occurring pairs.

Models

Fine-tuned encoders. As a reference we fine-tune RoBERTa-base (Liu et al. (2019)), BERT-base (Devlin et al. [4]), ModernBERT-base (Warner et al. (2025)) and XLM-R-base (Conneau et al. [2]) on clean training data (3 epochs, batch size 32, learning rate 2⋅10−52 \cdot10^{-5}, three seeds). Ablated training data (CORRUPT-TRAIN) is used only for the reproduction. For ANLI we use the released RoBERTa-large model of Nie et al. (2020), trained on SNLI, MNLI, FEVER-NLI and ANLI.

Open LLMs. We use Qwen3.8-27B, Muse Glimmer 30B and the base versions of OLMo 3 7B and 32B, whose pre-training data are public.2 Each item is formatted with a task instruction that ends in a one-word answer choice (e.g. Answer with one word: entailment, neutral, or contradiction.), with the model’s chat template where it has one and reasoning switched off. The prediction is the answer word with the highest log-probability. In the five-shot setting the prompt contains five training examples, either clean or with nouns removed (drop-NOUN), the in-context analogue of CORRUPT-TRAIN.

Hosted models. Mistral Large 3, DeepSeek-V4-Pro and GPT-4o3 are prompted with the same instructions at temperature 0, with reasoning switched off. For GPT-4o and DeepSeek-V4-Pro we also record the log-probabilities of the top answer tokens. Grok 4.7, which cannot be run without reasoning, is evaluated on a subset of 400 items per task under five perturbations.

Measures

Raw accuracy conflates two things: losing the information needed for the task, and shifting towards one answer. We therefore also report centred accuracy: for each perturbation, the mean score of each answer over all items is subtracted before taking the argmax, which removes a constant preference without using gold labels (cf. Zhao et al., 2021). To compare systems with different clean performance we report the share of above-chance performance a system retains,

R=100⋅spert−csclean−c,R = 100 \cdot\frac{s_{\mathrm{pert}} - c}{s_{\mathrm{clean}} - c},

where cc is chance accuracy (1/k1/k for kk labels). R=100R = 100 means no loss and R=0R = 0 means a drop to chance. Unless stated otherwise, RR is computed from centred accuracy for the open LLMs, and from accuracy for the encoders and for hosted models without log-probabilities. Confidence intervals are 95% bootstrap intervals over items (1,000 resamples, or 500 for the group comparisons in Section 5).

Results

Before moving on to LLMs, we verify that our pipeline reproduces the previous encoder results (Appendix § 8). On MNLI, all 8 reproduced scores for RoBERTa-base and BERT-base are within 0.6 points of the published ones, the other GLUE tasks are within about 3 points with two explained exceptions, and the public ANLI model of Nie et al. (2020) is within 3 points on all three rounds, clean and perturbed.

Perturbation Effects Across Systems

Table 1 and Figure 1 summarise how much performance the systems maintain. An LLM not trained on the task at all is expected to behave differently from a fine-tuned encoder. In some respects, that is the case. Excluding nouns, open-weight LLMs maintain slightly lower MNLI performance than RoBERTa (47–70% vs. 72%), but maintain higher RTE performance (29–40% vs. 19%).

SystemMNLI CleanMNLI RR nounMNLI RR verbRTE CleanRTE RR nounRTE RR verbANLI (R1–R3 average) CleanANLI (R1–R3 average) RR nounANLI (R1–R3 average) RR verb
RoBERTa (fine-tuned)‡87.8728170.8198556.12466
Qwen3.8-27B88.9637491.3407375.22960
OLMo 3 32B89.3707990.6307661.61949
OLMo 3 7B88.8687687.0406651.41450
Muse Glimmer 30B67.2476680.9297655.62556
GPT-4o81.5567180.1487571.42664
DeepSeek-V4-Pro74.2496488.5377357.72558
Mistral Large 3†80.7365085.6176363.62347
Grok 4.7 (reasoning)†§71.8436088.8348777.93868

Table 1. Clean accuracy (%) and share of performance above chance (RR, %) maintained under the drop-NOUN and drop-VERB conditions. RR is calculated from centred accuracy for open-weight LLMs and from accuracy for others (§ 3.4). ‡For MNLI and RTE, the RoBERTa-base model fine-tuned on clean data is used, and for ANLI, the public RoBERTa-large model from Nie et al. (2020) is used. †Answer scores are unavailable, and MNLI is evaluated on a 2,000-item sample. §Reasoning model, first 400 items per task.

Figure 1. Share of performance above chance maintained after noun removal (left) and word shuffling (right). Open-weight LLMs are scored zero-shot by centred accuracy, and RoBERTa is scored by accuracy (RoBERTa-large for ANLI, not run with shuffling). Values less than 0 indicate performance below chance.

However, the more notable result is what the systems share in common. Excluding nouns, open-weight LLMs retain 47–70% of their above-chance performance on MNLI, whereas on ANLI they retain only 14–29% (averaged over the three rounds). RoBERTa shows the same contrast (72% vs. 24%), and similar results are observed in the hosted models GPT-4o (56% vs. 26%), DeepSeek-V4-Pro (49% vs. 25%), and Mistral Large 3 (36% vs. 23%), which is scored on raw answers. When word order is shuffled, the performance difference becomes even more pronounced. Open-weight LLMs maintain 32–52% on MNLI and up to 27% on ANLI rounds, whereas OLMo 3 is close to or below chance on ANLI R1 and R2 (Figure 1). Among the GLUE tasks, CoLA, where labels are determined by word order and morphology, is most affected, whereas SST-2 maintains most of its performance, similar to Talman et al. (2022).

Model size does not substantially affect robustness. OLMo 3 7B and 32B were trained on the same data, and the larger model shows 10 points higher performance on clean ANLI. However, both models maintain similar retention on MNLI without nouns (68% and 70%), while the larger model shows only slightly higher retention on ANLI (19% vs. 14%). Unlike training an encoder on ablated data, five ablated in-context examples also do not help (Appendix § 10).

Perturbed Input Shifts the Answer Distribution

Perturbed input not only removes information but also biases LLMs toward a single answer. Table 6 in Appendix § 11 shows the proportion of items receiving the most frequent answer. When nouns are removed, Muse Glimmer 30B provides the same answer for 96–99% of MNLI, ANLI R1, and RTE items, and Qwen3.8-27B for 90% of RTE items, compared to 51–64% with clean input. Therefore, part of the decrease in accuracy is due to changes in the decision threshold rather than information loss.

TaskModelΔ\DeltaAccΔ\DeltaCenΔ\DeltaAUCTop answer
MNLIQwen3.8-27B-25.9-20.4-13.435→5735 \to57
OLMo 3 32B-19.5-17.5-10.138→4438 \to44
OLMo 3 7B-19.8-17.4-10.435→4735 \to47
Muse Glimmer 30B-31.9-25.7-21.363→9663 \to96
ANLI R1Qwen3.8-27B-32.3-32.8-27.239→5939 \to59
OLMo 3 32B-31.1-31.5-29.847→5247 \to52
OLMo 3 7B-24.5-22.2-22.740→4540 \to45
Muse Glimmer 30B-31.7-29.9-24.656→9756 \to97
RTEQwen3.8-27B-35.7-25.3-24.051→9051 \to90
OLMo 3 32B-29.2-28.9-27.960→7160 \to71
OLMo 3 7B-21.7-23.5-23.362→6062 \to60
Muse Glimmer 30B-32.5-25.6-28.064→9964 \to99

Table 6. Change (points) under drop-NOUN in raw accuracy, centred accuracy and AUC, and the share of items (%) given the most frequent answer, clean →\to perturbed. Zero-shot open LLMs.

Recentering the answer scores recovers up to 10 points (Qwen3.8-27B on RTE), and threshold-independent AUC shows the same results. The OLMo 3 models are less affected, and the three measures show agreement within approximately 2 points in ANLI R1 and RTE. For three of the four models, the ranking of tasks is the same under all three measures, with the least loss on MNLI and the greatest on ANLI R1 and RTE; the exception is Muse Glimmer 30B, whose raw MNLI accuracy is already low on clean input because of its answer bias.

Hosted Models

GPT-4o, scored by centred accuracy from its answer-token log-probabilities, behaves like the open-weight LLMs (Table 1), answer bias included: without nouns, the share of MNLI items receiving the most frequent answer rises from 38% to 65%, and recentring reduces the accuracy drop from 28.2 to 20.9 points. DeepSeek-V4-Pro shows the strongest answer bias of all systems. Without nouns, it gives the same answer for 87% of MNLI items and 93% of RTE items, compared to 52% and 59% with clean input. Judged by raw accuracy alone, it would keep only 22% of its above-chance MNLI performance, which is misleading: centred accuracy gives 49%.

A reasoning model. Grok 4.7, which reasons before answering, is the only system that does not follow the pattern. It does better on clean ANLI than on clean MNLI (77.9% vs. 71.8%), and without nouns it keeps about as much of its performance

on ANLI as on MNLI (38% vs. 43%), whereas every other system keeps at least twice as much on MNLI. Since Grok 4.7 returns no answer scores, these are raw accuracies of a single model on the first 400 items of each task, and they should be read with caution.

Memorisation or Shortcuts?

Why do all systems keep so much more of their performance on MNLI than on ANLI? Given that MNLI is old and widely distributed, and thus a likely candidate for pre-training leakage, memorisation is a natural initial hypothesis. We first test this with three contamination indicators and then examine the alternative, that MNLI entries can be resolved through cues that survive the perturbations.

Contamination Indicators

Guessing masked words. Following Deng et al. [3], we masked one content word in each entry and asked Qwen3.8-27B to recover it. In the initial version of this experiment, we allowed the masked word to appear in the other paired sentence, resulting in an exact match rate of 43–70% in the sentence-pair tasks Table 9. However, most of these matches are obtained simply by copying across the pair. If masking is restricted to words that do not appear elsewhere within the item, the exact match rate drops to 23–35%, whereas there is no change for the single-sentence tasks CoLA and SST-2. In these stricter settings, specific benchmarks do not stand out, and ANLI (released in 2020) is recovered with nearly the same frequency as MNLI.

TaskTS-guessing copyTS-guessing no copyMin-K%++ QwenMin-K%++ OLMoinfini-gram Prem.infini-gram Hyp.
MNLI56.329.0-1.89-1.882118/448
RTE66.432.4-1.78-1.49243/25
ANLI R167.534.0-1.79-1.077210/273
ANLI R269.231.3-1.84-1.03715/218
ANLI R359.226.6-2.18-1.49343/228
QNLI42.634.6-3.05-1.70––
MRPC70.127.5-1.87-1.69––
SST-213.813.6-2.17-1.80––

Table 9. Contamination indicators. TS-guessing: exact-match rate (%) of Qwen3.8-27B filling in a masked word, when the word may also occur in the other field (copy) and when it may not. Min-K%++: mean score (higher suggests seen). infini-gram: premises (or first sentences) found in OLMo’s pre-training data (%), and hypotheses of at least 13 words found / all such hypotheses.

Likelihood of rare tokens. Min-KK%++ (Zhang et al. (2025)) scores each item using the normalised log probability assigned to the least likely tokens in each item. For OLMo 3 32B, the highest values appear in ANLI R1 and R2 (−1.07 and −1.03), while for the remaining tasks, they range from −1.49 to −2.25. In contrast, Qwen3.8-27B does not show a similar pattern. Since ANLI R1 and R2 are based on Wikipedia passages, these high scores appear to reflect familiarity with the source text rather than exposure to the benchmark itself.

Exact nn-gram counts. Since OLMo’s pre-training corpus is publicly available, we can also check overlap directly. Using the infini-gram API (Liu et al. (2024)), we counted two evenly spaced 13-grams of each field (and the entire field, if up to 50 words long) in the OLMo 2 pre-training mix.4 A field counts as found if one of its 13-grams appears, or if the entire field appears and has at least 13 words, since short sentences such as The man is asleep. appear in any large corpus. The counts support the Min-KK%++ reading. Premises were found in 72% of ANLI R1 entries and 71% of ANLI R2 entries, whereas in MNLI, this figure was only 21%. Conversely, long hypotheses rarely appeared (1–4% for both MNLI and ANLI). Overall, the OLMo pre-training data contain a large part of the source texts of ANLI, but the benchmark entries themselves appear only rarely.

Seen and Unseen Items

If memorisation helps the model process perturbed inputs, items that appeared as text during pre-training should have less performance degradation than unseen items. For each open-weight LLM, we compared the degradation of items with found premises and of the remaining items on MNLI and the three ANLI rounds, using centred accuracy and five perturbations. Among all 80 comparisons, 7 have 95% confidence intervals excluding zero, with differences in both directions and an overall average of −0.8 points. We therefore find no evidence that exposure to the source text increases robustness. For found hypotheses, the number of items (10–18 per task) is too small to support reliable comparisons of the same type.

Hypothesis-Only Shortcuts

Following Gururangan et al. (2018), we train a hypothesis-only classifier (logistic regression on word unigrams and bigrams) on the training hypotheses of each dataset, and split the evaluation items into correctly predicted items (easy items) and the remaining items (hard items). The classifier reaches 54.3% on MNLI, in line with Gururangan et al. (2018), 42.0–43.1% on the ANLI rounds and 47.7% (below chance) on RTE. If hypothesis cues are the factor maintaining accuracy for perturbed inputs, then easy items should have a lower drop than hard items, and such a difference should depend on whether the perturbation preserves those cues.

Both predictions hold on MNLI for the three models with the most robust MNLI performance, Qwen3.8-27B and the two OLMo 3 models (Figure 2). For OLMo 3 32B and 7B, easy items lose 7.5 and 4.0 points less than hard items when nouns are removed, 10.2 and 6.7 points less when words are shuffled, and 18.1 and 17.0 points less when the premise is removed altogether. Qwen3.8-27B shows the same pattern with a smaller gap without nouns (2.0 points), and all of these intervals exclude zero. The one perturbation that reverses the pattern is keep-NOUN+VERB: here easy items lose more, by 2.3–4.4 points for the three models. This is also the only perturbation that removes function words, including negation words such as not, no and never, which are the strongest hypothesis-only cues in MNLI ((Gururangan et al., 2018)). Easy items are also easier on clean input (93.6% vs. 87.4% for OLMo 3 32B), which leaves them more room to drop, so the comparison is, if anything, conservative.

Figure 2. MNLI degradation for items correctly classified by the hypothesis-only classifier (easy items) minus the degradation for the remaining items (hard items), expressed as centred accuracy points with 95% bootstrap intervals. Positive values indicate less degradation for easy items. GPT-4o and DeepSeek-V4-Pro were evaluated on a sample of 2,000 items.

However, this pattern does not appear in all models. For Muse Glimmer 30B, this phenomenon occurs only under the drop-VERB and shuffle-n1 conditions. For GPT-4o, easy items show greater score loss (9.2 points) when nouns are absent, while no reliable difference appears under other conditions. DeepSeek-V4-Pro shows a reliable difference (7.9 points) only when the premise is removed, and no difference appears in Mistral Large 3, which is scored based on raw answers. The same applies to Grok 4.7 (all four MNLI intervals include 0). In other words, this effect is most clearly evident in open-weight models that we score ourselves, but not in hosted models. In ANLI, results are mixed across perturbations, but when the premise is removed, most models show less R2 and R3 score loss for easier items (a difference of 8–14 points). This suggests that the weak cues remaining in ANLI are used in the same way.

Fresh Items

The new items enable the most direct test: no model has seen them, but the MNLI-style items have the same type of cues as MNLI. If memorisation causes the robustness on MNLI, it should disappear on these items; if shortcuts cause it, it should remain on MNLI-style items and shrink on cue-free ones.

Table 2 shows the second pattern in all models. In the absence of nouns, all eight LLMs maintain higher performance in MNLI-style items (50–75% vs. 37–57%) than in cue-free items, which is nearly the same level as performance in MNLI itself. When only nouns and verbs remain, the order is reversed in all models tested with this perturbation (42–59% vs. 62–74%), as on MNLI. Within MNLI-style items, items correctly labelled by the MNLI hypothesis-only classifier show 11–28 points less loss than the remaining items when nouns are absent, and for 7 of the 8 models the 95% interval excludes zero. The same effect is observed here in the reasoning model and in hosted models, even though such effects were not present on MNLI. The new items are easier than MNLI (93–100% clean accuracy), and since most were labelled by a single annotator, absolute figures should be interpreted with caution. However, the difference between the two styles is consistently observed across all models.

ModelClean MNLIClean freeRR no nouns MNLIRR no nouns freeRR nouns+verbs MNLIRR nouns+verbs freeEasy−hard
Qwen3.8-27B100.097.372474971+16.0*
OLMo 3 32B99.498.075575062+10.9
OLMo 3 7B98.294.673545974+17.8*
Muse Glimmer 30B98.898.054455068+18.5*
GPT-4o98.294.660515669+17.1*
DeepSeek-V4-Pro99.496.653494670+17.6*
Mistral Large 3†98.293.252374268+28.1*
Grok 4.7 (reasoning)†§98.293.95042––+22.7*

Table 2. New items written in MNLI style or without cues (168 and 148 items): clean accuracy (%, centred where answer scores exist), retention rate of performance above chance (RR, %) under drop-NOUN and keep-NOUN+VERB conditions, and difference between easy and hard items in drop-NOUN drop for MNLI-style items (points, with * for 95% confidence intervals excluding 0). Symbols are the same as Table 1.

A Benchmark Without Hypothesis Cues

Robustness to perturbations does not necessarily come from shortcuts: models may also answer from what they already know. To see robustness without hypothesis cues, we perturbed the evidence passages of the 1,519 yes/no questions in BioASQ ((Tsatsaronis et al., 2015)) and kept the questions, which act as hypotheses, intact. A classifier using only the questions did not outperform the majority baseline on the three most recent test sets, and we found no sign of memorisation (Appendix H). Nevertheless, the models retained 64–89% of their above-chance AUC without any evidence, compared to 26–51% on MNLI using only hypotheses (Table 10 in Appendix H). Unlike MNLI, keeping only nouns and verbs is almost no more costly than removing nouns (83–96% retention compared to 38–47% retention in MNLI). BioASQ is therefore robust for other reasons, most likely biomedical knowledge, and the reversal under keep-NOUN+VERB in MNLI appears to be specific to data whose cues lie in function words.

ModelNo evidence MNLINo evidence BioASQNo nouns MNLINo nouns BioASQNouns+verbs MNLINouns+verbs BioASQ
Qwen3.8-27B268372974495
OLMo 3 32B517779944789
OLMo 3 7B416478884383
GPT-4o278960983892
DeepSeek-V4-Pro298855984096

Table 10. Retention rate of above-chance AUC (RR, %) without evidence (MNLI: hypothesis only, BioASQ: question only) and under drop-NOUN and keep-NOUN+VERB for MNLI and BioASQ yes/no questions. Since models respond yes by default when BioASQ items are degraded, AUC was used for both datasets.

Discussion

Taken together, the results point away from memorisation and towards shortcuts as the reason why familiar NLI benchmarks survive perturbation. Four observations support this reading. First, the contrast between MNLI and ANLI is shared by every system that answers directly, including RoBERTa fine-tuned on MNLI, whose robustness comes from what it learned from MNLI’s training data rather than from having seen the evaluation items. Second, none of the contamination indicators is related to robustness: items whose text is found in OLMo’s pre-training data lose as much as items whose text is not. Third, for the open LLMs most robust on MNLI, items that can be solved from the hypothesis alone are more robust, and lose this advantage exactly when the perturbation removes the function words that carry the cues. Fourth, and most directly, every LLM shows the same signature on fresh items that it cannot have seen, as long as they are written in MNLI’s style.

The hosted models and the reasoning model show no hypothesis-only effect on MNLI itself, although they do on the fresh MNLI-style items, so they can exploit the same cues. Why the effect is absent on MNLI remains open: their training data are unknown, so memorisation of MNLI cannot be ruled out for them, and they may rely on cues our classifier does not capture, such as lexical overlap (McCoy et al., 2019).

On MNLI, a large part of the benchmark can be solved from surface properties of the hypothesis that the perturbations leave intact; ANLI was built to remove such properties, and on ANLI all systems fall close to chance once the nouns are gone. In each case the perturbation experiments measure the benchmark as much as the model.

Do models, then, need to understand language to do well on these benchmarks? Our results suggest that the answer depends on the benchmark more than on the model. On MNLI, understanding the sentence pair is sufficient for a high score but not necessary: much of the benchmark can still be solved once the words a human reader would need are gone (Talman et al., 2021). In this sense MNLI measures, in part, how its annotators wrote hypotheses rather than whether one sentence follows from another. This does not mean that the models fail to understand the pairs. On the cue-free fresh items every LLM reaches 93–98% on clean input, so the models can draw the inference when no cue is available; a benchmark on which a cue-based and an inference-based solution score alike simply cannot tell the two apart. BioASQ adds a further distinction, between understanding the evidence and already knowing the answer. We would therefore argue that a benchmark score neither confirms nor refutes understanding, which is better seen as a matter of degree, and that perturbation experiments are most useful for finding out what a given benchmark can and cannot distinguish.

Conclusion

In this paper we have revisited the effect of input perturbations on NLI and other language understanding tasks with LLMs and fine-tuned encoders. Every system that answers directly keeps 36–72% of its above-chance MNLI performance without nouns but only 14–29% on ANLI, whatever its size. Contamination indicators show no link between exposure and robustness, whereas items solvable from the hypothesis are more robust. On 368 fresh items that no model can have seen, all eight LLMs keep more of their performance on items written in MNLI’s style than on cue-free items, and lose this advantage when function words are removed. We therefore argue that robustness to perturbation on familiar NLI benchmarks reflects shortcuts in the datasets rather than memorisation. We recommend that robustness results on NLI benchmarks be reported together with a hypothesis-only analysis and, where possible, on items written after the models’ training data were collected.

Open questions remain: a larger, multiply annotated fresh set would sharpen the comparison; other cues, such as lexical overlap (McCoy et al., 2019), may explain why the hosted models show no hypothesis-only effect on MNLI; and it remains to be seen whether reasoning models in general escape the MNLI–ANLI contrast.

Limitations

Our contamination checks are partial. The nn-gram counts cover the OLMo 2 pre-training mix rather than that of OLMo 3, sample only two 13-grams per field, and detect only verbatim overlap; paraphrased or reformatted copies of a benchmark, for example in code repositories, would be missed. For the other models the training data are not available, and the indirect indicators we use are known to be noisy. The absence of a link between exposure and robustness is therefore evidence against memorisation as the main explanation, not proof that the models have not seen the benchmarks; the fresh items address this for the comparison that matters most, but not for MNLI itself.

The shortcut split depends on one simple classifier; items it labels incorrectly may still contain cues that a stronger classifier would find, so the split underestimates the number of shortcut-solvable items, and a stronger hypothesis-only model, such as a fine-tuned encoder, would give a sharper one. The hosted models are analysed on the 2,000-item MNLI sample, which gives the comparison less power than for the open models.

The LLMs are scored with a single prompt template per task and, for the open models, by answer log-probability. Other templates or free-form generation could give different absolute numbers, although the comparisons we draw are within a model and template; a second template would show how far the absolute numbers depend on it. The hosted models are evaluated on the core set of perturbations and, for MNLI and QQP, on a 2,000-item sample, and their providers may change the models behind an endpoint over time. All experiments are on English data. The BioASQ comparison covers only the yes/no questions and six models, and is scored by AUC rather than centred accuracy. The fresh items were written by two LLMs and validated mostly by a single volunteer annotator each, none a native English speaker; their hypotheses differ between the two styles in more than cues (for example, cue-free hypotheses are longer), and a larger set with several labels per item would make the comparison more reliable.

Ethics Statement

We use GLUE (Wang et al., 2018) under the licences of its component datasets, ANLI under CC BY-NC 4.0, and Wikipedia text under CC BY-SA 4.0; open models are used under their published licences and hosted models under their providers’ terms. We report results only in aggregate. The fresh items were labelled by unpaid volunteers who could stop at any time; their age and language background were collected only to describe the rater pool and are reported only in aggregate. Hosted models were accessed through their providers’ standard interfaces; a small number of items (at most 0.5% per perturbation for MNLI and ANLI, 1% for RTE) were blocked by a provider’s content filter and are counted as errors.

Acknowledgements

We thank the volunteers who labelled the fresh items. We acknowledge CSC – IT Center for Science, Finland, for computational resources on the Roihu supercomputer.

Reproducing the Earlier Results

We reproduce the drop-NOUN experiments of Talman et al. (2022) with RoBERTa-base on five GLUE tasks and those of Talman et al. (2021) with BERT-base on MNLI. As in the earlier work, there are three conditions: CORRUPT-TEST (trained on clean data, tested on perturbed data), CORRUPT-TRAIN (trained on perturbed data, tested on clean data) and CORRUPT-TRAIN-AND-TEST (perturbed in both). Table 3 gives the published and reproduced scores side by side. On MNLI, all reproduced scores are within 0.6 points of the published ones, and the other GLUE tasks are within about 3 points, with two exceptions. Our clean CoLA score is 5.2 points lower, and our CORRUPT-TRAIN models reach 62.9 on RTE instead of 47.3. The published RTE figure equals the share of entailment items in the development set (131 of 277), which suggests that the original run predicted a single class; our three seeds did not. For ANLI, we evaluate the released RoBERTa-large model of Nie et al. (2020) on the clean test sets and on the perturbed test sets released by Talman et al. (2021); all scores are within three points of the published ones (Table 4).

ModelTaskCleanCORRUPT-TESTCORRUPT-TRAINCORRUPT-TRAIN-AND-TEST
RoBERTa-baseMNLI87.9 / 87.872.8 / 72.885.6 / 85.877.5 / 77.7
RoBERTa-baseRTE70.0 / 70.853.8 / 54.047.3 / 62.947.3 / 56.7
RoBERTa-baseMRPC88.7 / 88.182.3 / 79.686.3 / 82.080.2 / 82.3
RoBERTa-baseCoLA64.0 / 58.817.8 / 15.339.7 / 41.334.3 / 34.9
RoBERTa-baseSTS-B90.1 / 89.956.1 / 55.881.7 / 81.863.5 / 63.3
BERT-baseMNLI83.7 / 83.969.8 / 70.280.7 / 81.373.4 / 73.3

caption: Published / reproduced scores for drop-NOUN. RoBERTa-base rows are compared with Talman et al. (2022), BERT-base with Talman et al. (2021). Metrics as in GLUE: accuracy (MNLI, RTE), F1 (MRPC), Matthews correlation (CoLA) and Pearson correlation (STS-B), all ×100; ours are means over three seeds.

:::

R1R2R3
Clean73.8 / 73.548.9 / 49.344.4 / 45.5
CORRUPT-TEST NOUN43.7 / 46.536.0 / 34.732.4 / 33.8
CORRUPT-TEST VERB54.6 / 57.244.7 / 42.339.3 / 39.0

caption: ANLI test accuracy (%) of the released RoBERTa-large model of Nie et al. (2020), published / ours. Clean scores are compared with Nie et al. (2020), perturbed ones with Talman et al. (2021), using that paper’s released files.

:::

Prompts

All tasks share one template per task; the NLI tasks (MNLI, ANLI, fresh items) use:

Premise: {premise}
Hypothesis: {hypothesis}
Does the premise entail the hypothesis,
contradict it, or neither?
Answer with one word: entailment,
neutral, or contradiction.

RTE asks Does the premise entail the hypothesis? with the answers yes or no; QNLI asks whether a sentence contains the answer to a question; MRPC and QQP whether two sentences (questions) mean (ask) the same thing; SST-2 for the sentiment (positive or negative); CoLA whether a sentence is grammatically acceptable; STS-B for a similarity score from 0 to 5. Base models receive the text followed by Answer:; chat models and hosted models receive it as a user message. In the five-shot setting, five training examples precede the item, each followed by its gold answer (as Answer: <word> for base models, as an assistant turn for chat models), drawn label-balanced with a fixed seed; for the ablated variant the same perturbation is applied to the examples.

Ablated In-Context Examples

For encoders, training on ablated data recovers most of the loss ((Talman et al., 2021, 2022)): RoBERTa-base reaches 85.8 on clean MNLI after training without nouns, and 77.7 when tested without nouns as well (Table 3). Is there an in-context analogue for LLMs? Table 5 suggests that there is not. Five examples help OLMo 3 32B on ANLI R1 and QNLI, raising the retained share from 23% to 31% and from 44% to 56%. Ablated examples, however, show no consistent advantage over clean ones: the two differ by at most six points in either direction, and on MNLI not at all (72% with either kind for OLMo 3 32B, 67% for OLMo 3 7B). The model does not adapt to the ablation shown in the examples; the gain comes from the examples themselves.

ModelExamplesMNLIRTEANLI R1ANLI R2ANLI R3QNLI
OLMo 3 7B0-shot88.3 (68)89.2 (40)63.0 (25)50.1 (3)47.5 (13)82.0 (56)
5-shot, clean88.4 (67)88.5 (35)66.2 (28)50.9 (18)50.2 (18)87.0 (56)
5-shot, drop-NOUN87.5 (67)88.5 (33)63.6 (25)49.4 (12)49.6 (20)86.5 (57)
OLMo 3 32B0-shot90.8 (70)91.3 (30)74.0 (23)61.1 (19)56.8 (17)88.5 (44)
5-shot, clean89.0 (72)92.1 (33)75.0 (31)64.4 (19)60.8 (17)91.0 (56)
5-shot, drop-NOUN89.2 (72)91.7 (34)75.1 (31)62.8 (22)60.8 (18)90.8 (59)

Table 5. Clean centred accuracy (%), and, in brackets, the share of above-chance performance retained under drop-NOUN (RR, %), with no examples, five clean examples or five examples with nouns removed. MNLI five-shot on the 2,000-item sample.

Full Results

The tables in this section give the results behind § 4: the effect of the answer bias (Table 6), the share retained under every perturbation for the LLMs (Table 7) and the fine-tuned encoders (Table 8).

PerturbationQwen3.8OLMo 32BOLMo 7BGlimmerGPT-4oDeepSeek
drop-NOUN63/2970/1968/1447/2556/2649/25
drop-VERB74/6079/4976/5066/5671/6464/58
drop-ADJ78/7082/7280/6477/7079/7578/69
drop-ADV74/8277/8374/8473/82–/––/–
drop-DET92/9392/9193/9391/9391/9592/95
drop-PRON97/9997/9797/9294/9799/9796/99
drop-NUM97/6898/7197/6596/70–/––/–
drop-CONJ99/10099/9899/9498/97–/––/–
drop-NOUN+PRON60/2867/1766/1343/23–/––/–
keep-NOUN+VERB34/1738/833/1330/2331/2537/23
keep-NOUN+VERB+ADJ52/3255/1852/1846/34–/––/–
keep-NOUN+ADV+VERB54/2259/957/1449/28–/––/–
keep-NOUN+PRON+VERB37/1742/937/1532/24–/––/–
keep-NOUN+VERB+ADV+ADJ76/4277/2676/2468/42–/––/–
shuffle-n146/1552/252/232/1846/2642/21
shuffle-n261/2765/1564/1240/23–/––/–
shuffle-n369/3472/2070/1950/3264/4455/33
hyp-only27/1539/3032/2417/3226/3729/33
swap57/3154/948/071/49–/––/–

Table 7. Share of above-chance performance retained (RR, %, from centred accuracy) on MNLI / mean of the three ANLI rounds, for every perturbation; hosted models were run on the core set only (–).

ModelTaskCleanNo nounsNo verbsShuffledHyp.-only
RoBERTa-baseCoLA58.815.34.41.7–
MRPC88.179.684.275.231.6
RTE70.854.067.657.248.0
STS-B89.955.885.385.25.1
BERT-baseCoLA56.416.24.41.1–
MRPC81.977.880.173.531.6
RTE64.053.561.258.447.5
STS-B88.456.684.286.13.6
ModernBERT-baseCoLA58.913.73.92.8–
MRPC84.178.981.863.931.7
RTE58.748.952.849.847.2
STS-B89.253.184.281.71.3
XLM-R-baseCoLA43.519.25.02.4–
MRPC84.277.980.076.131.5
RTE59.251.159.554.151.1
STS-B86.350.982.884.61.3

Table 8. Fine-tuned encoders trained on clean data (means over three seeds; GLUE metrics ×100).

Fresh Items

Premises were taken from Wikipedia (CC BY-SA 4.0) and kept at most two per article; the infini-gram check dropped any premise with a 13-gram found in the OLMo 2 pre-training mix. Generators alternated over premises, so each wrote half of the items in both styles. Raters (14 volunteers, none a native English speaker, median 21 seconds per item) saw premise and hypothesis only, labelled each pair as entailment, neutral or contradiction from the premise alone, and could flag ambiguous, ungrammatical or knowledge-dependent pairs. Most items received one label; on the 32 items with several, Krippendorff’s α\alpha is 0.82, and the labels agree with the intended label on 85.6% of the labelled generated items. Revision pairs that differ only in case, punctuation or word order (28) were dropped. On the kept items, the MNLI hypothesis-only classifier reaches 65.5% on MNLI-style and 41.9% on cue-free items (54.3% on MNLI itself), and 39% of MNLI-style contradictions contain a negation word, against none of the cue-free ones. The items remain private until publication so that they stay unseen.

Annotation guidelines. Raters saw the following instructions verbatim:

For each pair, choose a label judged only from the premise as written: Entailment (e): if the premise is true, the hypothesis must be true. Neutral (n): the hypothesis may or may not be true. Contradiction (c): if the premise is true, the hypothesis must be false. Near-identical sentence pairs can just get e. Under Problem, note anything that makes the item unusable (ambiguous, ungrammatical, needs outside knowledge). Leave it empty if the item is fine. Label independently and keep the items private: don’t share them or paste them anywhere online. Do as many as you like and stop whenever you want.

On each item, raters could tick ambiguous, ungrammatical or needs outside knowledge and add free text before choosing a label.

Compute and Cost

Fine-tuning, log-probability scoring and the contamination checks ran on NVIDIA GH200 nodes, about 59 GPU-hours in total. The hosted models were queried through Azure AI Foundry for USD 126, including the generation of the fresh hypotheses.

Contamination Indicators

Table 9 gives the three contamination indicators discussed in Section 5.

BioASQ

We use the yes/no questions of the BioASQ Task b training set 11b (1,271) and of the test sets of 12b, 13b and 14b (248), with their gold snippets; the perturbations are applied to the snippets only. Answers are scored by the log-probabilities of yes and no, computed by us for the open models and returned by the provider for the hosted models. One item blocked by a provider’s content filter is dropped for the hosted models. Llama-3.3-70B, evaluated on BioASQ only, retains 95% of its above-chance AUC under drop-NOUN. Measured from the question-only baseline instead of chance, the snippets keep 51–73% of their contribution under keep-NOUN+VERB for the open models and DeepSeek-V4-Pro, against −8 to 25% on MNLI measured from the hypothesis-only baseline; for GPT-4o the figure is 23% (95% interval −14 to 50), which does not separate it from MNLI (15%), since its clean and question-only AUC differ by only five points.

Question cues. A logistic regression on question uni- and bigrams reaches a macro F1 of 0.61 on the training set (five-fold cross-validation) but 64.5% accuracy on the pooled test sets, below the majority baseline of 64.9%. Splitting the gold-no items by this classifier, as for MNLI, gives intervals above zero in only 5 of 24 model–perturbation cells, and no reversal under keep-NOUN+VERB. We restrict the split to gold-no items because over all items it is confounded with the models’ default answer yes.

Answer default. Almost all of the loss falls on gold-no items, which the models increasingly answer with yes as the evidence degrades; gold-yes accuracy stays at 97–100% under every perturbation. Qwen3.8 is the exception, losing on both classes. AUC, which does not depend on a decision threshold, corrects for this default.

Memorisation. By the developers’ stated training cut-offs, the 12b–14b questions postdate the training data of GPT-4o and Llama-3.3-70B, 13b and 14b that of OLMo 3, and, by a third-party report, 14b that of DeepSeek-V4-Pro; Qwen3.8 has no published cut-off. On factoid questions, the share of performance kept without the evidence on these newer questions overlaps that on the training set for every model; yes/no questions are too few per year for this comparison. For the 1,668 factoid questions, we compared items most of whose snippets contain a 13-gram found in the OLMo 2 pre-training mix with the rest, stratified by the number of snippets; the difference in the drop without evidence is within its interval for every model in the pooled data.

References

  1. [1]Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly Media.
  2. [2]Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.arxiv.org/abs/1911.02116
  3. [3]Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8706–8719, Mexico City, Mexico. Association for Computational Linguistics.arxiv.org/abs/2311.09783
  4. [4]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.arxiv.org/abs/1810.04805
  5. [5]Shahriar Golchin and Mihai Surdeanu. 2024. Time travel in LLMs: Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2308.08493

Paper details

vlnc:W_35kg7jmx
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 12

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents