WorksComputer Science
The Shared Word: Evaluating Language Models on Dutch Word Puzzles
Abstract
Many LLM benchmarks for languages other than English are translated, multiple-choice, or difficult to score automatically. We introduce The Shared Word (TSW-NL), a Dutch benchmark built around four core principles: it is language-specific, open-vocabulary, human-curated, and deterministically scored. Each puzzle gives three clues that describe different senses or uses of the same Dutch word. The clues are individually ambiguous, requiring the solver to combine evidence across multiple clues. We construct the benchmark from Dutch Wikipedia disambiguation pages, followed by automatic filtering and extensive manual curation. The final dataset contains 222 instances, with three independent human answers per test item. The task is clearly challenging: scores in our human study range from 49.5% to 61.0% and the highest-scoring open-weight model we tested reaches only 39.3%. Many others even score below 20%. At the same time, closed-source models show great progress: GPT-4.1 (April 2025) scores 59.2%, whereas GPT-5.6 Sol with max reasoning scores 88.8%. We believe this task provides a useful addition to Dutch model evaluation, plus a clear recipe that can be applied to construct a similar benchmark for other languages.
Introduction
Reliable benchmarking is important for understanding what large language models (LLMs) can and cannot do, in particular for languages other than English. Many of the benchmarks commonly used for such languages are directly translated from English, making it unclear whether LLMs are truly capable in those languages. Similarly, many tasks are multiple-choice, which only test whether the models recognize the correct answer, instead of generating it themselves. Moreover, tasks that require generation often have unclear and fuzzy evaluation metrics, making comparison between models hard.
To address these issues, we introduce a Dutch benchmark based on four simple principles:
The benchmark is language-specific: the lexical relation between its clues and answer does not survive direct translation.
The benchmark requires open-vocabulary generation.
Each item has one short intended answer and can be scored deterministically.
All items in the benchmark are human-verified and curated.
This may sound straightforward, but finding benchmarks that meet all four requirements is surprisingly difficult.1
Task format The task we introduce consists of three clues that all point to a single Dutch word. Figure 1 shows an example. It is inspired by a popular question type in the Dutch TV quiz Met Het Mes Op Tafel. There are no answer options or predefined labels. The model must interpret the clues, combine them, and retrieve the word from an open vocabulary. The clues are curated around one intended answer and they are deliberately underspecified as not to point to only a single term. We use disambiguation pages of Dutch Wikipedia to create our dataset. After extensive filtering, we manually curated it to ensure, among other things, that the three clues collectively admit only one intended answer, the examples cannot be translated while preserving the same answer, and the clues are distinctly different and do not contain obscure words or references. Importantly, we want a task that is still doable for humans, so the LLM capabilities can be evaluated in the light of human performance as well.

Figure 1. Example puzzle. The three Dutch clues describe a football coach (Dick Advocaat), an alcoholic drink (Dutch-specific, similar to eggnog), and a profession (lawyer), all answered by advocaat.
Dutch benchmarks Dutch already has strong resources: DUMB combines nine language-understanding tasks, the EuroEval Natural Language Understanding benchmark includes Dutch, and BEIR-NL tests information retrieval ([8] (Saattrup Nielsen et al. 2025; Lotfi et al. 2025)). What our work adds is a difficult free-generation task that is easy to score. We also test whether models adapted to Dutch, such as GEITje, Fietje 2 and ChocoLlama (Vanroy, 2024, 2025; Meeus et al., 2024), perform relatively better on a Dutch-specific benchmark such as this one.
Results The final, heavily curated test set contains 200 instances, each answered by three human annotators. Experiments show that the benchmark has a useful level of difficulty. All evaluated open-weight models struggle, given that the strongest approach only reaches 39.3% accuracy. There are still clear differences between the models, indicating that our benchmark seems to distinguish well between different model capabilities. The strongest closed models perform much better: they clearly exceed the scores of our human annotators, while still being quite far from perfect.
Contributions While the benchmark itself is language-specific, the process that creates it is not. Our main contribution is therefore a novel NLP task for testing language-specific lexical knowledge, candidate retrieval, and evidence integration through an open-vocabulary task that does not transfer directly across languages. In particular, we release TSW-NL, a carefully curated set of 222 Dutch puzzle instances that serves as a useful addition to current Dutch LLM evaluation benchmarks.
Related Work
Relation to existing tasks In a sense, our benchmark runs word-sense disambiguation in reverse (Navigli, 2009). Instead of mapping a given word to a fixed sense inventory, the solver recovers the word from several senses or conventional uses. It also resembles riddle and lateral-reasoning benchmarks ([14] (Lin et al. 2021; Jiang et al. 2023b; Manna et al. 2024; Samdarshi et al. 2024; Saha et al. 2025)), but is language-specific and has no answer choices. Solving it requires lexical knowledge, candidate search, and evidence integration. The retrieval direction is close to the reverse dictionary task, where a description is used to find a word (Hill et al., 2016; Pilehvar, 2019; Yan et al., 2020; Qi et al., 2020). However, our task is quite different. For one, reverse dictionary systems typically recover a word from a description of just one sense. Moreover, here each clue is deliberately underspecified and describes a different sense. Potential solvers must therefore combine candidates across clues, with the shared word form often being their only connection. Wikipedia disambiguation pages support this construction well, since they group ordinary senses with people, places, organizations, titles, and other cultural referents. They also allow us to remove obscure references by checking page views.
Language-specific benchmarks We construct the task in Dutch rather than translate it from English. Translation is useful for cross-lingual comparison, but translation quality and artefacts can affect multilingual benchmark results (Thellmann et al., 2024). The benchmark does not transfer directly across languages: senses that share a form in Dutch generally use unrelated words in another language. There are several recent benchmarks that are also language or culture specific, such as Korean social values, Persian etiquette and customs, Arabic financial reasoning, and Turkish temporal interpretation ([10, 9] (Lee et al. 2024; Sadr et al. 2025; Sakhaeirād et al. 2026)). There are, however, not many that follow our four principles as described in the introduction. Appendix Tables 7 and 8 compare a very broad set on these principles and find that only a few tasks in larger language-specific benchmarks pass all four checks.2 Note that this is not to say that four principles are the only important qualities for a benchmark. There are many more important qualities. But it does highlight that our benchmark possesses interesting qualities that are not common.
Dutch benchmarks There are already a few well-known Dutch LLM benchmarks. DUMB {(citep}(de Vries et al., 2023)} evaluates Dutch language understanding through nine tasks, including part-of-speech tagging, named-entity recognition, word-sense discrimination, pronoun resolution, inference, sentiment, abusive-language classification, and question answering. EuroEval {(citep}(Saattrup Nielsen et al., 2025)} places Dutch in a broader European evaluation suite and covers classification, extraction, summarization, knowledge, common-sense reasoning, and simplification. BEIR-NL adds a task on Dutch information retrieval {(citep}(Lotfi et al., 2025)}. Appendix Table 9 shows that none of the existing tasks pass all four principles, meaning that we believe our task would be a valuable addition to the community.
Benchmark Construction
Task As shown in Figure 1, each puzzle has three Dutch clues and one Dutch answer. Each clue describes a different sense or referent of the answer word. Together they should point to one intended answer. We design the clues so that no single clue should identify the answer on its own. Their combination should narrow the possibilities to the intended answer. For example, in Figure 1, clues 1 and 2 could also point to Rose, clues 1 and 3 to Boer, and clues 2 and 3 to Kardinaal.3 Appendix Table 10 shows ten more puzzle examples.
Filtering and Curation
Source and initial filters We start by collecting all Dutch Wikipedia disambiguation pages.4 The page title then is a candidate answer and descriptions of linked pages are candidate clues. The initial collection has 97,707 pages. We require at least three links (since we want three clues) and remove all short abbreviations. We also only keep titles found in Open Dutch WordNet {(citep}(Postma et al., 2016)}: we want all the answers to be common Dutch words. This ensures that we remove instances for which all the clues point to a given name.5 Moreover, the puzzles are most interesting when there is a mix of clues. We also filter pages where the clues are not descriptive enough (e.g., too short or numerical).
Detailed filters At this point (4,047 examples left), a manual inspection showed us that the benchmark still has a few problems. For one, some puzzles are too easy, and a single clue often just gives away the answer. On the other hand, some puzzles are simply way too hard, as they for example reference an obscure Dutch city or person that almost no person realistically will have heard of. Moreover, clues are sometimes too similar, which does not lead to an engaging puzzle.6 Therefore, we apply a few more filters to only keep interesting and engaging puzzles. We filter puzzles by media and place categories, as they overwhelmingly reference obscure places and titles. We also exclude puzzles in which one of the clue descriptions contains the answer directly, and remove repeated descriptions. Then, we filter puzzles that have clues that are too formulaic and Wikipedia-specific, such as "region of", "taxon", and "building in". Finally, as an extra filter for too obscure answers, we filter linked pages below 3,000 views in the previous month.
Manual curation At this point (see Table 1), we have 2,087 puzzles left, but we can tell there are still clear issues with the resulting puzzles that automatic filtering cannot capture. We turn to manual filtering, with one primary curator, who discussed interesting cases with a second curator. First, we filter all instances where any one of the clues is simply deemed too obscure, or where two clues effectively point to the same sense. This is subjective to some extent: when is something obscure and when is it not? Nevertheless, this is a necessary step, and we trust humans a lot more on this task than LLMs or other automated methods.
| Stage | Left | Removed | Operation |
| Automatic collection and filtering | |||
| Dutch Wikipedia disamb. pages | 97,707 | 0 | Retrieve candidate answer titles. |
| At least three linked pages | 48,518 | 49,189 | Require enough links for three clues. |
| Title in Open Dutch WordNet | 5,723 | 42,795 | Prefer lexical answers over proper-name-only pages. |
| Title has at least four characters | 5,075 | 648 | Remove short abbreviations and acronyms. |
| At least three described linked pages | 4,159 | 916 | Require three links with descriptions and categories. |
| Informative descriptions | 4,047 | 112 | Remove generic, Wikipedia-internal, and numeric clues. |
| Non-media category mix | 3,057 | 990 | Remove candidates dominated by media and place categories. |
| No answer leakage | 2,929 | 128 | Remove descriptions containing the answer. |
| Distinct descriptions | 2,896 | 33 | Remove repeated descriptions. |
| Sufficiently viewed pages | 2,479 | 417 | Remove pages with fewer than 3,000 views in the previous month. |
| No unsuitable domain descriptions | 2,087 | 392 | Remove formulaic location, taxon, building, and related clues. |
| Manual annotation and selection | |||
| Obscurity & sense | 568 | 1,519 | Remove obscure candidates and check for distinct senses. |
| Final quality selection | 222 | 346 | Check for a single answer, the need for multiple clues, and Dutch specificity. |
Table 1. Data reduction during automatic filtering and manual annotation.
Final manual checks We then have 568 potential puzzles left, but there are still a few more checks to do. For one, we only want puzzles for which all three clues are ideally needed to arrive at the final answer. We therefore filter examples where clearly only one or two clues would be needed to solve the puzzle. This works well, but is also a fine line: we see later in Table 12 that models can still solve some of the puzzles when only seeing one or two clues. Second, we want the puzzle to be completely Dutch-specific. Third, we carefully check that the clues point to one intended answer and that no obvious alternative remains.7
| Model | 1 | 2 | 3 |
| Fietje 2 Instruct | 2.0 | 5.3 | 6.0 |
| Fietje 2 Chat | 2.7 | 7.0 | 9.5 |
| GEITje 7B Ultra | 3.7 | 9.5 | 15.0 |
| ChocoLlama 3 8B Instruct | 4.3 | 9.2 | 15.5 |
| Mistral 7B Instruct v0.1 | 2.2 | 3.2 | 3.5 |
| Llama 3 8B Instruct | 5.5 | 11.7 | 19.5 |
| Salamandra 7B Instruct | 4.8 | 9.5 | 14.0 |
| Aya Expanse 8B | 4.2 | 11.0 | 13.5 |
| Aya Expanse 32B | 8.2 | 23.0 | 33.5 |
| Llama 3.1 8B Instruct | 5.2 | 10.3 | 14.0 |
| Gemma 3 4B IT | 3.7 | 10.7 | 16.0 |
| Gemma 3 12B IT | 8.2 | 19.0 | 31.0 |
| Gemma 3 27B IT | 9.2 | 26.3 | 39.0 |
| EuroLLM-9B Instruct | 6.2 | 16.7 | 27.5 |
| EuroLLM-22B Instruct | 5.5 | 17.7 | 26.5 |
| Ministral 3 14B Instruct | 6.7 | 20.0 | 31.5 |
| Qwen3-8B | 6.8 | 21.2 | 26.0 |
| Qwen3.5-9B | 5.4 | 15.5 | 22.5 |
| Phi-4 Mini Reasoning | 1.7 | 4.3 | 11.5 |
| Phi-4 Reasoning | 0.0 | 3.2 | 9.5 |
| GPT-OSS-20B | 0.3 | 9.0 | 12.0 |
| GPT-4o (2024-11-20) | 11.0 | 35.5 | 57.5 |
| GPT-4.1 (2025-04-14) | 11.3 | 39.8 | 60.0 |
| GPT-5.6 Luna (none) | 9.0 | 34.2 | 54.5 |
| GPT-5.6 Luna (max) | 9.7 | 57.7 | 78.0 |
| GPT-5.6 Sol (none) | 9.7 | 43.0 | 68.0 |
| GPT-5.6 Sol (max) | 10.3 | 63.2 | 87.5 |
Table 12. Accuracy (%) by number of clues. Column 1 is the average of the single-clue conditions 1, 2, and 3. Column 2 averages the two-clue conditions 1+2, 1+3, and 2+3. Finally, column 3 uses all three clues in canonical order.
Rewriting For the puzzles that pass this check, we manually rewrite the clues. Wikipedia descriptions were written to identify pages, not to make puzzles. Some contain dates, locations, or parenthetical details that give away too much information, while others are too vague outside the context of the page. We shorten these clues or add just enough context to make them readable, while keeping the underlying sense unchanged. The final dataset of 222 instances (randomly split into 22 development and 200 test instances) consists entirely of manually reviewed and curated items.8
Size The test set contains 200 carefully curated items. This is relatively small for a benchmark. We prioritize item quality over a larger set containing weaker puzzles. To reduce dependence on clue order, each model is evaluated on all six permutations, producing 1,200 predictions while treating the six predictions for each item as repeated measurements. Also, as Table 3 will later show, the resulting scores still reveal substantial differences between models.
| System | Acc. | SD |
| Human annotators | 54.0 | — |
| Dutch models | ||
| Fietje 2 Instruct | 7.0 | 1.0 |
| Fietje 2 Chat | 9.3 | 0.8 |
| GEITje 7B Ultra | 14.8 | 2.1 |
| ChocoLlama 3 8B Instruct | 15.0 | 1.7 |
| Open-weight non-reasoning | ||
| Mistral 7B Instruct v0.1 | 5.1 | 1.3 |
| Llama 3 8B Instruct | 14.2 | 2.8 |
| Salamandra 7B Instruct | 13.1 | 2.8 |
| Aya Expanse 8B | 14.0 | 1.0 |
| Aya Expanse 32B | 32.9 | 1.2 |
| Llama 3.1 8B Instruct | 15.8 | 0.9 |
| Gemma 3 4B IT | 15.6 | 2.0 |
| Gemma 3 12B IT | 29.8 | 1.4 |
| Gemma 3 27B IT | 39.3 | 1.5 |
| EuroLLM-9B Instruct | 25.8 | 1.5 |
| EuroLLM-22B Instruct | 27.3 | 1.8 |
| Ministral 3 14B Instruct | 30.8 | 0.7 |
| Open-weight reasoning | ||
| Qwen3-8B | 28.5 | 1.5 |
| Qwen3.5-9B | 19.9 | 2.6 |
| Phi-4 Mini Reasoning | 10.3 | 2.0 |
| Phi-4 Reasoning | 9.0 | 1.3 |
| GPT-OSS-20B | 12.1 | 1.2 |
| Proprietary models | ||
| GPT-4o (2024-11-20) | 56.3 | 2.5 |
| GPT-4.1 (2025-04-14) | 59.2 | 0.7 |
| GPT-5.6 Luna (none) | 54.1 | 1.4 |
| GPT-5.6 Luna (max) | 79.0 | 1.3 |
| GPT-5.6 Sol (none) | 69.9 | 0.9 |
| GPT-5.6 Sol (max) | 88.8 | 1.2 |
Table 3. Test accuracy (%), averaged over the six permutations of the three clues. SD is the sample standard deviation of the six order-level accuracies.
Dataset characteristics The items mix lexical senses with names and cultural references. We manually reviewed all 600 test clues and counted the clues tied to a specifically Dutch entity or cultural context. We found 47 such clues (7.8%) across 38 test puzzles (19.0%). Note that the rest of the benchmark is still Dutch-specific through its lexical ambiguity. Across the dataset, models solving this task must handle polysemy, homonymy, metonymic or conventional uses, names of entities, titles, brands or technical senses. These categories can also overlap within an item.
Wikipedia exposure Because the candidate senses originate from Dutch Wikipedia, pretrained models may benefit from associations acquired from that source, and models may differ in the extent of their exposure. However, all systems are evaluated closed-book, and every open-weight model scores below 40%, indicating that source exposure alone is clearly insufficient to solve the task reliably.
Human Evaluation
To get a good sense of human performance, we asked nine annotators to attempt to solve the puzzles, such that each puzzle item had three attempted answers. Seven Dutch speakers answered blocks of 20 or 40 items, together covering the test set once. Two more annotators each answered all 200 items. All nine participants are native speakers and took part voluntarily. The full annotation protocol is available in Appendix § 11. It turns out that human annotators can do the task quite well, but they are also far from perfect. The pooled block of 7 annotators contains 103 correct answers (51.5%). The two full-test annotators solved 99 (49.5%) and 122 items (61.0%). There are still 43 items that no human managed to solve. This confirms that the benchmark remains challenging despite the strict manual filtering for obscure clues.
Experimental Setup
To see how well LLMs do on this benchmark, we test a large number of models. We test instruction-tuned models in a closed-book setting, without retrieval or any Wikipedia or internet access. We define the task as zero-shot: a Dutch prompt explains that the clues point to one word and asks for only that word.9 The exact prompts are available in Appendix § 15.
Models We cover Dutch-specific models and multilingual models (with and without reasoning) of varying sizes and families. The benchmark is intended to help open-weight developers evaluate their models, but we understand the curiosity to see how well state-of-the-art closed-source models do. Therefore, we test GPT-5.6 Luna and Sol and include two earlier models from around the time work on this benchmark began, to show progress. Table 2 lists every model and groups them by model type. To investigate whether further training on Dutch helped the Dutch-specific models, we add the closest instruction-tuned controls for two lineages: Mistral 7B Instruct v0.1 for GEITje 7B Ultra and Llama 3 8B Instruct for ChocoLlama 3 8B Instruct.
| Model | Size | Type | Reference |
| Fietje 2 Instruct | 2.7B | Dutch | Vanroy, 2025 |
| Fietje 2 Chat | 2.7B | Dutch | Vanroy, 2025 |
| GEITje 7B Ultra | 7B | Dutch | Vanroy, 2024 |
| ChocoLlama 3 8B Instruct | 8B | Dutch | Meeus et al., 2024 |
| Mistral 7B Instruct v0.1 | 7B | Multilingual | Jiang et al., 2023a |
| Llama 3 8B Instruct | 8B | Multilingual | Graffattoni et al., 2024 |
| Salamandra 7B Instruct | 7B | Multilingual | Gonzalez-Agirre et al., 2025 |
| Aya Expanse 8B | 8B | Multilingual | Dang et al., 2024 |
| Aya Expanse 32B | 32B | Multilingual | Dang et al., 2024 |
| Llama 3.1 8B Instruct | 8B | Multilingual | Graffattoni et al., 2024 |
| Gemma 3 4B IT | 4B | Multilingual | Gemma Team et al., 2025 |
| Gemma 3 12B IT | 12B | Multilingual | Gemma Team et al., 2025 |
| Gemma 3 27B IT | 27B | Multilingual | Gemma Team et al., 2025 |
| EuroLLM-9B Instruct | 9B | Multilingual | Martins et al., 2025 |
| EuroLLM-22B Instruct | 22B | Multilingual | Ramos et al., 2026 |
| Ministral 3 14B Instruct | 14B | Multilingual | Mistral AI, 2025 |
| Qwen3-8B | 8B | Multi/reason | Yang et al., 2025 |
| Qwen3.5-9B | 9B | Multi/reason | Qwen Team, 2026 |
| Phi-4 Mini Reasoning | 3.8B | Multi/reason | Xu et al., 2025a |
| Phi-4 Reasoning | 14B | Multi/reason | Abdin et al., 2025 |
| GPT-OSS-20B | 20B | Multi/reason | OpenAI, 2025 |
| GPT-4o (2024-11-20) | – | Multilingual | OpenAI, 2026 |
| GPT-4.1 (2025-04-14) | – | Multilingual | OpenAI, 2026 |
| GPT-5.6 Luna | – | Multi/reason | OpenAI, 2026 |
| GPT-5.6 Sol | – | Multi/reason | OpenAI, 2026 |
Table 2. Models included in the evaluation.
Prompting conditions We design the task as zero-shot: the clues should contain enough information without worked examples. Out of interest, we compared performance using 0, 1, 3, 5, 10, and 22 examples in canonical clue order for a selection of our models, shown in Appendix Table 13. Differences are generally small, and some models in fact do best with no examples. For the reasoning models, we use the highest reasoning setting available.
Scoring Our main metric is exact-match accuracy after trimming whitespace and lowercasing. This is deterministic and requires no subjective or model-based judge. In our main experiment, every model sees all six clue orders to reduce dependence on any single clue order. We report their mean accuracy and sample standard deviation, which provides an immediate indication of sensitivity to clue order. We average correctness over the six orders for each item. We obtain 95% confidence intervals by bootstrapping the 200 items. For model comparisons, we use paired sign-flip tests on these item-level scores and paired bootstrap intervals for the accuracy difference, with Holm correction across the family of comparisons. We test clue-order effects by randomly permuting the six order labels within each item. The iteration counts and further implementation details are available in Appendix § 16.
Source-retrieval diagnostic To further judge the results of the LLMs, we also run a BM25 source-retrieval diagnostic (Robertson and Zaragoza, 2009). Note that this is not a fair baseline, as it receives candidate inventories and Wikipedia source material, but it makes for an interesting comparison. Each candidate answer becomes a document containing its linked senses, the three clues form the query, and the title of the highest-ranked document is returned. We use two source-matched pools with linked-page titles and Dutch descriptions: all 2,087 automatically filtered candidates and, as an oracle-like condition, only the 200 test answers. This diagnostic basically asks how directly the clues match their source descriptions and how plausible recovery from memorized source associations would be. BM25 reaches 69.0% with the 2,087-candidate pool and 85.5% with the oracle-like 200-answer pool. This shows that the clues do contain substantial source signal, though it is also far from enough to completely solve the task.
Error annotation
To understand the remaining errors, we manually annotate every answer initially marked incorrect for a subset of models and one human annotator. We assign each response to one of the following mutually exclusive categories:
Empty: the human left the answer blank, or the model produced no final answer before reaching its token limit.
Repeats (part of) a clue: the answer copies wording from a clue instead of identifying the shared word.
Completely incorrect: none of the other categories apply and the answer does not meaningfully fit the clues.
Fits only one clue: the answer is related to one clue but not the other two.
Fits only two clues: the answer combines part of the evidence but fails on the third clue.
Inflection variant: the answer is morphologically related to an accepted answer, but the produced form does not itself satisfy all three clues.
Correct alternative: another answer that also fits all three clues.
This analysis also made us realize that checking wrong answers for only a few models is unfair. However, annotating all 18,866 unique non-empty wrong responses across all experiments is not feasible. Therefore, we additionally annotate every previously unreviewed answer that occurs at least six times across the experiments, but only to determine whether the answer should be accepted.10 Of these 532 unique answers, only three should have been accepted. Appendix Table 11 lists all these accepted alternatives, together with the four unique ones found in the more extensive manual annotation (Table 5). We then rescore all experiments with the expanded answer key, e.g. allowing some puzzles to have multiple answers. This produces only small changes, but is in our opinion the most fair setup.
| Category | Fietje | Luna | Sol | Human |
| Empty | 1 | 17 | 7 | 47 |
| Completely incorrect | 21 | 0 | 0 | 0 |
| Repeats (part of) a clue | 41 | 0 | 0 | 0 |
| Fits only one clue | 90 | 7 | 3 | 15 |
| Fits only two clues | 35 | 16 | 13 | 16 |
| Inflection variant | 0 | 4 | 2 | 0 |
| Correct alternative | 0 | 3 | 3 | 1 |
| Total | 188 | 47 | 28 | 79 |
Table 5. Manual categories for answers initially marked incorrect in zero-shot canonical-order Fietje 2 Instruct, GPT-5.6 runs with max reasoning, and one full-test human annotator.
Results
Table 3 shows the accuracy scores for the LLMs used in this study.11 Scores range from 5.1% (Mistral 7B) to an impressive 88.8% for GPT-5.6 Sol. Generally, all models except the very best ones clearly struggle with the task. We are happy to see that, as it shows that the benchmark is far from saturated, at least for open-weight and non-reasoning models. Among the open-weight models, Gemma 3 27B scores highest at 39.3%, followed by Aya Expanse 32B at 32.9% and Ministral 3 14B at 30.8%. Larger models usually do better than their smaller counterparts, except for the EuroLLM models, which do not differ significantly in score. The Dutch-specific models unfortunately do not perform as well, though it is interesting to see that GEITje and ChocoLlama perform better than Fietje 2.
Significance The statistical tests support the clearest differences in the table. GPT-5.6 Sol outperforms every other model after Holm correction, while Gemma 3 27B is significantly better than all other open-weight models. In general, we find that clue order has little systematic effect. Of the 27 model conditions shown in Figure 4, 25 have an uncorrected randomization .12
Figure 4. Test accuracy for all models included in the evaluation, averaged over the six permutations of the three clues. Dots show the mean accuracy and lines show 95% confidence intervals obtained by bootstrapping the 200 test items while keeping the six clue orders for each item together.
Does further Dutch training help? To answer this, we compare the GEITje and ChocoLlama models to the two parent models they were based on. We cannot make the same comparison for Fietje 2: its base model, Phi-2, is not instruction-tuned and could not perform the prompted task at all. The picture is mixed. For GEITje it clearly helped, as it reaches 14.8%, compared with 5.1% for its parent model Mistral 7B Instruct. However, ChocoLlama reaches 15.0%, compared with 14.2% for its parent model Llama 3 8B Instruct, which is not a significant improvement.
Does reasoning help? Allowing reasoning makes a large difference for both GPT-5.6 models (see the bottom of Table 3). Without it, Luna drops by 24.9 points and Sol by 18.9 points. Even though this is not really a “reasoning” task in the traditional sense, the models clearly benefit from being able to spend more compute on thinking through and evaluating potential answers.
How much did SOTA models improve? The dated GPT snapshots at the bottom of Table 3 give a rough view of progress: GPT-4o from November 2024 scores 56.3%, GPT-4.1 from April 2025 scores 59.2%, and GPT-5.6 Sol (July 2026) with max reasoning reaches 88.8%. Of course this is not a controlled architecture comparison, but the 29.6-point difference between GPT-4.1 and Sol shows how much performance on this task changed over roughly fifteen months.
Do models need all clues? We created this dataset with the goal that each puzzle could only be reliably solved when combining information from multiple clues, preferably even all three clues. However, in practice models could obviously still get to the right answer by looking at only two clues, or making an educated guess after only one. This ablation experiment is shown in Figure 2.13 We see that having three clues is certainly beneficial, but that models can do reasonably well with two. GPT-5.6 Sol (max) averages only 10.3% with one clue, already 63.2% with two, and all three get the model close to 90%. Most open-weight checkpoints in our selection also improve with more evidence. One clue is almost never enough (apart from some lucky guesses), but two clues already help the best models a lot. In future work, it would be interesting to run a similar experiment with human annotators, to see if it helps them equally much.
Figure 2. Model accuracy by number of clues. For clarity, only a few models are highlighted in color. The other models are shown in grey. One- and two-clue scores average over the three corresponding clue subsets.
Do humans and LLMs find the same items difficult? To answer this, Table 4 groups items by how many of the three human judgments were correct. Model accuracy generally rises with human agreement, so the two notions of difficulty are related. The pattern is not perfectly clear for smaller models, however. Interestingly, GPT-5.6 Sol (max) manages to solve 72.5% of the 43 items missed by all three human judgments. We did manually inspect these instances, but could not pinpoint a single cause. The model simply seems better at generating, selecting and evaluating possible answer candidates.
| Model | 0/3 | 1/3 | 2/3 | 3/3 |
| Fietje 2 Instruct | 4.3 | 7.4 | 5.0 | 10.5 |
| Fietje 2 Chat | 4.7 | 11.3 | 6.6 | 13.5 |
| GEITje 7B Ultra | 5.0 | 8.9 | 17.0 | 25.1 |
| ChocoLlama 3 8B Instruct | 5.4 | 12.1 | 15.1 | 24.6 |
| Mistral 7B Instruct v0.1 | 3.5 | 6.4 | 1.9 | 8.2 |
| Llama 3 8B Instruct | 10.1 | 12.8 | 11.0 | 21.3 |
| Salamandra 7B Instruct | 3.9 | 10.3 | 19.2 | 16.7 |
| Aya Expanse 8B | 9.3 | 12.1 | 6.9 | 25.7 |
| Aya Expanse 32B | 17.8 | 25.5 | 36.8 | 46.8 |
| Llama 3.1 8B Instruct | 7.8 | 13.1 | 14.8 | 24.9 |
| Gemma 3 4B IT | 5.0 | 16.3 | 14.8 | 23.7 |
| Gemma 3 12B IT | 12.8 | 32.3 | 29.6 | 40.9 |
| Gemma 3 27B IT | 21.3 | 33.3 | 41.5 | 55.8 |
| EuroLLM-9B Instruct | 10.1 | 22.3 | 27.0 | 39.5 |
| EuroLLM-22B Instruct | 11.2 | 27.7 | 22.3 | 43.9 |
| Ministral 3 14B Instruct | 15.1 | 27.0 | 29.9 | 46.5 |
| Qwen3-8B | 12.0 | 23.8 | 32.1 | 41.5 |
| Qwen3.5-9B | 9.7 | 11.7 | 24.2 | 30.4 |
| Phi-4 Mini Reasoning | 4.7 | 10.6 | 10.4 | 14.3 |
| Phi-4 Reasoning | 2.7 | 6.7 | 9.4 | 15.2 |
| GPT-OSS-20B | 5.8 | 12.4 | 8.5 | 19.9 |
| GPT-4o (2024-11-20) | 31.8 | 45.4 | 62.6 | 78.1 |
| GPT-4.1 (2025-04-14) | 31.4 | 50.0 | 68.6 | 78.9 |
| GPT-5.6 Luna (none) | 34.9 | 44.0 | 59.1 | 72.2 |
| GPT-5.6 Luna (max) | 63.2 | 72.0 | 82.7 | 93.3 |
| GPT-5.6 Sol (none) | 46.5 | 63.5 | 75.8 | 87.4 |
| GPT-5.6 Sol (max) | 72.5 | 90.4 | 93.4 | 95.3 |
Table 4. Accuracy (%) by the number of correct human judgments, averaged over six clue orders. The four groups contain 43, 47, 53, and 57 items.
Discussion
What types of mistakes are made? We manually categorize the answers initially marked incorrect from one low-performing model (Fietje 2 Instruct), two high-performing models (GPT-5.6 Luna and Sol), and the best full-test human annotator. Usually, when the human annotator or good models make a mistake, their answers still fit one or two clues but fail on the rest (Table 5). Luna and Sol even each produced three correct alternatives, and the human annotator did this once.14 Empty is also an interesting category. For the human annotator, this means that they could not come up with an answer they considered good enough and left the line empty. For a model, it means that they did not come up with an answer before running out of reasoning or generation tokens. On inspection, we saw that the models mostly looped through the same answers, and were no longer making meaningful progress when they ran out of tokens. Fietje is clearly not up to the task: it often repeats all or part of a clue instead of proposing a new answer, or comes up with an answer that only fits a single clue.
Are the instances discriminative? Figure 3 shows how many instances are solved by at least a given number of model conditions, counting an instance as solved when a condition succeeds in any clue order. Only four instances are never solved, while also only four are solved in all 27 conditions. We consider this scarcity at both extremes a desirable property for a benchmark: even though the benchmark is relatively small, many of its instances play a role in distinguishing one model from another.
Figure 3. Number of test instances solved by at least the number of model conditions on the horizontal axis. A model condition solves an instance if it produces an accepted answer in at least one of the six clue orders. Counts are cumulative, so an instance solved by, for example, ten models also contributes to every point from one to ten.
What makes an item difficult? Table 6 relates model and human accuracy to simple item characteristics.15 Frequencies are estimated from a large Dutch news corpus (Goldhahn et al., 2012). As perhaps expected, familiar answers (checked by answer frequency and page views) and disambiguation pages with more entries are easier for both models and humans, while most other characteristics have little association with accuracy. The full breakdown is shown in Appendix Table 14.
| Feature | Models | Human |
| Answer frequency | .45 | .34 |
| Average clue frequency | .10 | .19 |
| Minimum clue frequency | −.07 | .01 |
| Average clue length | .11 | .00 |
| Answer length | −.13 | −.14 |
| Named-entity clues | −.03 | −.01 |
| Original disambiguation entries | .38 | .25 |
| Wikipedia page views | .25 | .28 |
Table 6. Item-level Spearman correlations with mean model accuracy and human success. Bold values are significant at .
Conclusion
In this paper, we introduced the Dutch LLM benchmark The Shared Word (TSW-NL), which combines four properties that rarely occur together in language-model evaluation: it is language-specific, open-vocabulary, deterministically scored, and fully curated by humans. The task is simple to describe but not easy to solve. A model receives three clues for different senses or uses of one Dutch word and has to generate that word.
The results show both headroom and clear progress. Most models in our selection struggle, and the strongest open-weight checkpoint reaches only 39.3%. GPT-4o and GPT-4.1 score within the range observed for our human annotators, while GPT-5.6 Luna and Sol score substantially higher. The ablations also confirm the intended structure of the task: one clue is rarely enough, two help considerably, and all three give the strongest result.
The benchmark itself is deliberately Dutch, since its lexical relations do not survive translation. The construction process is more general, though. Other languages with similar resources could construct a similar benchmark by following our recipe, though it would require careful manual curation. We hope this work encourages more benchmarks that test language-specific knowledge directly without giving up open generation or clear evaluation.
Limitations
As indicated in the method, our benchmark is relatively small. We chose 200 carefully curated test items over a larger set of weaker puzzles. That choice limits statistical power and perhaps the ability to distinguish between models. We do believe this trade-off between quality and size is worth it in this case.
Our benchmark is also just for a single language. This is entirely on purpose: we believe we need more carefully curated, language-specific datasets instead of large, automatic datasets across multiple languages that are not manually verified. The same construction process could be applied to other languages with sufficient disambiguation resources. We sincerely hope future work will look into this.
Dataset construction required extensive manual curation. It involved judgments about familiarity, difficulty, and whether an answer would be unique. Only two curators were involved in these decisions. This introduces some subjectivity into item selection, as is unavoidable when curating this kind of puzzle benchmark. A different team might therefore select a somewhat different final set.
Since the task is derived from Wikipedia, pre-trained models may benefit from associations acquired from that source, but the low performance of most evaluated models shows that source exposure alone is not enough. Because the task is open-vocabulary, valid alternatives cannot be guaranteed to be exhaustively known in advance. Therefore, we tried our best to identify such alternatives during error analysis, and updated all scores in the paper taking them into account.
Finally, TSW-NL is mainly a test of Dutch lexical knowledge, candidate generation, and evidence integration. In particular, while it tests open-vocabulary generation, it does not test generation capabilities beyond just words. We see it as a nice addition to the current evaluation landscape, but not, of course, as a superior benchmark in testing general Dutch capabilities.
Acknowledgments
We thank the Center for Information Technology of the University of Groningen for their support and for providing access to the Hábrók high performance computing cluster. We also thank all our colleagues of the GroNLP group for the discussions and useful feedback. Finally, we thank all the annotators that took part in this study.
References
Related Language-Specific Benchmarks
Tables 7, 8, and 9 apply the same four criteria to the benchmarks included in our literature review. For benchmarks containing multiple tasks, means that only some tasks have the property. Our definition of single-language is stricter than simply evaluating text in the target language: translations of another benchmark do not pass SL.
Human Evaluation Protocol
Participants and research ethics Participants were recruited through our personal network and received no compensation. They were all native speakers of Dutch. Before participating, they were informed that their anonymized answers would be used in research and gave consent. The task involved solving ordinary lexical puzzles and did not collect personal or sensitive information. No ethical clearance from our institution was therefore deemed necessary.
Participant instructions Participants received the same instructions as the LLMs, except for the instructions about JSON output. They were not shown example puzzles. There was no time limit. They were not allowed to use browsing or any other external resource.
Licenses
The title filter uses an archived lemma list derived from Open Dutch WordNet (Postma et al., 2016), distributed under CC BY-SA 4.0. The candidate clues were derived from Dutch Wikipedia text. At the time of collection, this material was also available under CC BY-SA 4.0. We will release our benchmark under an Apache 2.0 license.
Example Test Items
Table 10 gives ten test items, with English translations for readability.
| Clue 1 | Clue 2 | Clue 3 | Answer |
| Een plant<br>A plant | Een gemeenschap van mensen<br>A community of people | Een term uit taalkunde<br>A linguistic term | Stam |
| Een Nederlandse televisiepresentator<br>A Dutch television presenter | Vuur<br>Fire | Een Nederlandse brouwerij<br>A Dutch brewery | Brand |
| Een kleur<br>A colour | Een strafkaart<br>A penalty card | Een politieke richting<br>A political orientation | Rood |
| Een reeks tekens in een postadres<br>A string of characters in a postal address | Een versleuteling van gegevens<br>An encryption of data | Een computertaal<br>A programming language | Code |
| Een fossiele hars<br>A fossil resin | Een kleur<br>A colour | Een vrouwelijke voornaam<br>A female given name | Amber |
| Een houtsoort<br>A type of wood | Een onderdeel van een kerkgebouw<br>Part of a church building | Een Nederlandse televisiekok<br>A Dutch television chef | Beuk |
| Een onderdeel van een schip<br>Part of a ship | Een onderdeel van een molen<br>Part of a windmill | Een type vloerbedekking<br>A type of floor covering | Zeil |
| Een muziekinstrument<br>A musical instrument | Een rookgerei<br>A smoking implement | Een waterafvoer<br>A drain | Pijp |
| Een Nederlandse volksvertegenwoordiging<br>A Dutch representative body | Een onderdeel van een woning<br>Part of a home | Onderdeel van het hart en vaatstelsel<br>Part of the cardiovascular system | Kamer |
| Een plant<br>A plant | Een datastructuur<br>A data structure | Een staaf horizontaal over een weg<br>A horizontal barrier across a road | Boom |
Table 10. Ten example test items. English translations are shown in italics and are not part of the benchmark input.
Correct Alternative Answers
Table 11 lists the seven items for which error annotation identified a second correct answer. Both answers are accepted by the final benchmark scorer. We added an explanation for why the alternative answer should be accepted. The final three answers can hardly be considered different answers: they are plural or verbal versions of the original answer.
| Clue 1 | Clue 2 | Clue 3 | Gold | Alternative | Who |
| Het terrein van beroepsmatig toegepaste kennis<br>A field of professionally applied knowledge | Een term in sport<br>A term in sport | Het geheel van arbeidstaken<br>The set of work duties | Discipline | Vak | Luna, Sol, human |
| Why accept? Vak perfectly fits clues 1 and 3. The second one is too vague to really rule it out: vak is also a term in sport. | |||||
| Een automodel<br>A car model | Een type vliegtuig<br>A type of aircraft | Een wild dier<br>A wild animal | Eend | Jaguar | Luna |
| Why accept? Jaguar even fits the first and third clues better than Eend and both are relatively obscure aircraft. | |||||
| Een eenheid<br>A unit | Een mannelijke voornaam<br>A male given name | Een oude computer<br>An old computer | Pascal | Mark | Sol |
| Why accept? Both general clues fit. Mark is a very obscure old computer, but Sol is still technically correct. | |||||
| Een verzorger<br>A caregiver | Een lid van een kloosterorde<br>A member of a religious order | Een familielid<br>A family member | Zuster | Broeder | Luna, Sol |
| Why accept? Broeder and Zuster are old-fashioned words for brother and sister, and both fit the descriptions. This was an oversight during data curation. | |||||
| Een temperatuurschaal<br>A temperature scale | Een academisch concept<br>An academic concept | Een wiskundig concept<br>A mathematical concept | Graad | Graden | GPT-4o, Ministral |
| Why accept? Graden is the regular plural of graad and fits the same three clues. | |||||
| Een term uit muziektheorie<br>A term from music theory | Een opmerking uit een tekst<br>A note in a text | Een vrucht<br>A fruit | Noot | Noten | Aya, EuroLLM, Qwen3, Salamandra |
| Why accept? Noten is the regular plural of noot and fits the same three clues. | |||||
| Het zoeken van kandidaten<br>The search for candidates | Een onderdeel van evolutie<br>A component of evolution | Een grafische gebruikersomgeving handeling<br>A graphical user interface action | Selectie | Selecteren | Aya, ChocoLlama, Llama 3, Llama 3.1, Qwen3, Salamandra |
| Why accept? Selecteren is the corresponding verb and fits the three described processes. |
Table 11. Items for which a model or human annotator produced a correct alternative to the gold answer. These alternatives are included in the accepted-answer sets used for all reported scores.
| Benchmark | Language | Brief description | SL | OV | HV | DE |
| ITALIC ((Seveso et al., 2025)) | Italian | Italian culture, history, civic knowledge, and language questions from public examinations (0/12 tasks pass) | ✓ | × | ✓ | ✓ |
| SOBACO ((Yamamoto et al., 2025)) | Japanese | Paired questions testing Japanese age, gender, hierarchy, and workplace norms | ✓ | × | ✓ | ✓ |
| JPharmaBench ((Ono et al., 2025)) | Japanese | Japanese pharmaceutical licensing, terminology normalization, and consistency checking (0/3 tasks pass) | ∼ | × | ✓ | ✓ |
| Japanese Financial Benchmark ((Hirano, 2024)) | Japanese | Japanese financial sentiment, knowledge, audit, and professional-exam tasks (0/5 tasks pass) | ✓ | × | × | ✓ |
| JMedBench ((Jiang et al., 2025)) | Japanese | Japanese biomedical MCQA, translation, NER, classification, and similarity tasks (0/5 tasks pass) | ∼ | ∼ | ∼ | ∼ |
| KBL ((Kim et al., 2024b)) | Korean | Korean legal knowledge and reasoning | ✓ | × | ✓ | ✓ |
| KorNAT ((Lee et al., 2024)) | Korean | Social values and common knowledge | ✓ | × | ✓ | ✓ |
| Nunchi-Bench ((Kim and Lee, 2025)) | Korean/English | Korean superstition knowledge, advice, and interpretation (0/3 tasks pass) | × | ∼ | ✓ | ∼ |
| CLIcK ((Kim et al., 2024a)) | Korean | Korean cultural and linguistic intelligence across 11 exam- and textbook-based categories (0/11 tasks pass) | ✓ | × | ✓ | ✓ |
| PRONE ((Thapa et al., 2025)) | Nepali | Nepali proverb meaning selection and thematic classification (0/2 tasks pass) | ✓ | × | ✓ | ✓ |
| NorEval ((Mikhailov et al., 2025)) | Norwegian | Bokmål and Nynorsk understanding, generation, idiom, and knowledge tasks (3/24 tasks pass) | ∼ | ∼ | ✓ | ∼ |
| TAAROFBENCH ((Sadr et al., 2025)) | Persian/English | Open-ended role play about Persian taarof | × | ✓ | ✓ | × |
| DIVANBENCH ((Sakhaeirad et al., 2026)) | Persian | Superstitions, customs, and social etiquette (0/3 tasks pass) | ✓ | × | × | × |
| FFE-Hallu ((Hosseini et al., 2026)) | Persian/English | Idiom/proverb generation, detection, and translation (0/3 tasks pass) | ∼ | ∼ | ✓ | ∼ |
| Persian LLM Evaluation ((Hosseinbeigi et al., 2025)) | Persian | Persian language understanding, academic knowledge, and culture tasks with MCQ and open-ended formats (1/9 tasks pass) | ∼ | ∼ | ✓ | ∼ |
| SweSAT-1.0 ((Kurfalı et al., 2025)) | Swedish | Swedish university entrance-exam verbal and quantitative reasoning (0/6 tasks pass) | ✓ | × | ✓ | ✓ |
| TimeRes ([9]) | Turkish | Temporal interpretation and date calculation (0/4 tasks pass) | ✓ | ✓ | × | ✓ |
| TurkBench ((Toraman et al., 2026)) | Turkish | Turkish knowledge, reasoning, language, safety, and cultural evaluation tasks (0/21 tasks pass) | ∼ | ∼ | ✓ | ∼ |
| CETVEL ([11]) | Turkish | Native, translated, and multilingual Turkish understanding, generation, and culture tasks (0/23 tasks pass) | ∼ | ∼ | ∼ | ∼ |
Table 8. Continuation of Table 7.
| Benchmark | Language | Brief description | SL | OV | HV | DE |
| TSW-NL (ours) | Dutch | Three clues identify one ambiguous Dutch word | ✓ | ✓ | ✓ | ✓ |
| EuroEval Dutch main-score tasks ((Saattrup Nielsen et al., 2025)) | ||||||
| CoNLL-NL | Dutch | Named-entity recognition | ✓ | × | ✓ | ✓ |
| DBRD | Dutch | Sentiment classification | ✓ | × | ✓ | ✓ |
| Duidelijke Taal | Dutch | Text simplification | ✓ | ✓ | ✓ | × |
| HellaSwag-NL | Dutch | Multiple-choice common-sense reasoning | × | × | × | ✓ |
| MMLU-NL | Dutch | Multiple-choice knowledge | × | × | × | ✓ |
| ScaLA-NL | Dutch | Linguistic acceptability | ✓ | × | × | ✓ |
| SQuAD-NL | Dutch | Extractive question answering | × | ✓ | ✓ | ✓ |
| WikiLingua-NL | Dutch | Abstractive summarization | × | ✓ | ✓ | × |
| BEIR-NL ((Lotfi et al., 2025)) | ||||||
| BEIR-NL | Dutch | Automatically translated information retrieval | × | × | × | ✓ |
| DUMB tasks ([8]) | ||||||
| COPA-NL | Dutch | Causal reasoning | × | × | ✓ | ✓ |
| DALC | Dutch | Abusive-language classification | ✓ | × | ✓ | ✓ |
| DBRD | Dutch | Sentiment classification | ✓ | × | ✓ | ✓ |
| DPR | Dutch | Pronoun resolution | ✓ | × | ✓ | ✓ |
| Lassy POS | Dutch | Part-of-speech tagging | ✓ | × | ✓ | ✓ |
| SICK-NL | Dutch | Natural-language inference | × | × | ✓ | ✓ |
| SoNaR-1 NER | Dutch | Named-entity recognition | ✓ | × | ✓ | ✓ |
| SQuAD-NL | Dutch | Extractive question answering | × | ✓ | ✓ | ✓ |
| WiC-NL | Dutch | Word-sense discrimination | ✓ | × | ✓ | ✓ |
Table 9. Dutch tasks in EuroEval, DUMB, and BEIR-NL, compared with TSW-NL. BEIR-NL is translated automatically from the English BEIR datasets, ranks documents from a fixed corpus, reports deterministic retrieval metrics, and does not include substantive human verification of its Dutch construction. Column definitions and symbols follow Table 7.
| Benchmark | Language | Brief description | SL | OV | HV | DE |
| TSW-NL (ours) | Dutch | Three clues identify one ambiguous Dutch word | ✓ | ✓ | ✓ | ✓ |
| JAWAHER ((Magdy et al., 2025)) | Arabic/English | Multidialectal proverb translation and explanation (0/3 tasks pass) | ∼ | ✓ | ✓ | × |
| Nahw ((Mubarak et al., 2026)) | Arabic | Grammar analysis, correction, and explanation (2/4 tasks pass) | ✓ | ∼ | ✓ | ∼ |
| ALARB ([2]) | Arabic | Legal verdict prediction and statute identification (0/6 tasks pass) | ✓ | ∼ | × | ∼ |
| SAHM ([10]) | Arabic | Financial and Shari’ah-compliant reasoning (0/7 tasks pass) | ✓ | ∼ | ✓ | ∼ |
| BALSAM ([3]) | Arabic | Arabic understanding and generation platform (0/78 tasks pass) | ∼ | ∼ | × | ∼ |
| BanglaSocialBench ((Sijan et al., 2026)) | Bangla | Address terms, kinship, and social customs | ✓ | × | ✓ | ✓ |
| CDQA ((Xu et al., 2025b)) | Chinese | Time-sensitive, human-verified short-answer questions collected from the Chinese web | ✓ | ✓ | ✓ | × |
| MEDFACT Response Checking ([5]) | Chinese | Evidence-based fact checking of LLM-generated Chinese medical responses | ✓ | × | ✓ | ✓ |
| MedFact Veracity and Localization ((He et al., 2026)) | Chinese | Chinese medical veracity classification and error localization (0/2 tasks pass) | ✓ | ∼ | ✓ | ∼ |
| CHENGYU-BENCH ([12]) | Chinese | Chinese idiom connotation, contextual appropriateness, and open-cloze use (1/3 tasks pass) | ✓ | ∼ | ✓ | ✓ |
| CTM ((Wang et al., 2026)) | Chinese | Chinese dynastic chronology QA and timeline-order reasoning (0/9 tasks pass) | ✓ | ∼ | ✓ | ✓ |
| LexGenius ((Liu et al., 2026)) | Chinese | Expert-level Chinese legal knowledge and reasoning across 11 MCQ tasks (0/11 tasks pass) | ✓ | × | ✓ | ✓ |
| LAiW ([6]) | Chinese | Chinese legal syllogism tasks from retrieval to legal application (0/14 tasks pass) | ✓ | ∼ | ∼ | ∼ |
| Danish Semantic Reasoning ((Pedersen et al., 2024)) | Danish | Curated Danish lexical-semantic inference, synonymy, relatedness, and word-sense tasks (0/25 tasks pass) | ✓ | × | ✓ | ✓ |
| CrowS-Pairs-NL ((van der Weide et al., 2026)) | Dutch | Dutch stereotype-bias preference pairs adapted from English and crowdsourced locally | × | × | ✓ | × |
| LCHAIM ((Malul et al., 2025)) | Hebrew | Long-context Hebrew natural-language inference translated from ConTRoL | × | × | ✓ | ✓ |
| Icelandic Linguistic Benchmark ([4]) | Icelandic | Icelandic syntax, morphology, anaphora, and translation diagnostics (5/21 tasks pass) | ∼ | ∼ | ✓ | ∼ |
Table 7. Recent LLM benchmarks developed for non-English languages, part 1. SL: created for one target language rather than translated from another benchmark or evaluated in parallel target-language and English versions. OV: the task requires generating an answer rather than selecting from fixed candidates. HV: construction includes substantive human checking, curation and validation. DE: answers are deterministically evaluated using a fixed answer or rule without needing an LLM judge, human rating, or fuzzy metric. For multi-task benchmarks, ✓, ∼, and × mean that all, some, or none of the constituent tasks pass, respectively.
Prompt Templates
System message
Je bent een taalexpert die Nederlandse taalkundige raadsels oplost.
Gebruik geen browser, webzoekopdracht, externe bron of tool.
Los het raadsel alleen op met je interne taalkennis.
Geef het antwoord altijd in het Nederlands, niet in het Engels.
User message
Los het volgende taalkundige raadsel op. Het bevat drie aanwijzingen, en het antwoord is één woord dat daarop van toepassing is.
Geef alleen het antwoord zelf in JSON. Geen uitleg, geen markdown, geen extra sleutels.
Het antwoord moet Nederlands zijn, niet Engels.
Het JSON-formaat moet exact zijn: {"answer": "<antwoord>"}.
Als er meerdere geldige opties zijn, kies dan je beste gok.
Vraag: {puzzel}. Wat is het?
Antwoord:
Experimental Details
Reproducibility Each full clue-order evaluation contains 1,200 predictions: all six permutations for every test item. The development set was not used for optimizing performance, and it was only used for the shot items in Table 13. No settings or prompts were tuned on the test set. Local models were run with the same seed (42) and a maximum of 2,048 generated tokens on an NVIDIA A40 48GB or A100 80GB GPU. The settings not mentioned are left at their default values.
| Model | 0 | 1 | 3 | 5 | 10 | 22 |
| Fietje 2 Instruct | 6.0 | 7.0 | 7.0 | 6.0 | 6.5 | 3.5 |
| Fietje 2 Chat | 9.5 | 7.5 | 7.0 | 6.5 | 6.0 | 8.5 |
| GEITje 7B Ultra | 15.0 | 11.0 | 16.5 | 15.0 | 17.5 | 15.5 |
| ChocoLlama 3 8B Instruct | 15.5 | 15.0 | 17.0 | 17.5 | 16.5 | 16.5 |
| Mistral 7B Instruct v0.1 | 3.5 | 10.5 | 9.5 | 10.0 | 11.0 | 11.0 |
| Llama 3 8B Instruct | 19.5 | 16.5 | 18.5 | 19.0 | 20.5 | 19.0 |
| Salamandra 7B Instruct | 14.0 | 14.5 | 15.5 | 13.5 | 13.5 | 10.5 |
| Aya Expanse 8B | 13.5 | 19.5 | 19.0 | 16.5 | 17.5 | 17.0 |
| Aya Expanse 32B | 33.5 | 35.5 | 36.5 | 35.0 | 34.0 | 33.0 |
| Llama 3.1 8B Instruct | 14.0 | 17.5 | 20.5 | 21.0 | 20.0 | 23.5 |
| Gemma 3 4B IT | 16.0 | 18.0 | 17.0 | 22.0 | 21.0 | 18.0 |
| Gemma 3 12B IT | 31.0 | 31.0 | 32.5 | 34.0 | 33.0 | 35.0 |
| Gemma 3 27B IT | 39.0 | 41.0 | 41.0 | 40.5 | 39.5 | 44.5 |
| EuroLLM-9B Instruct | 27.5 | 18.0 | 19.0 | 17.5 | 18.0 | 20.5 |
| EuroLLM-22B Instruct | 26.5 | 24.5 | 21.5 | 20.5 | 22.0 | 24.5 |
| Ministral 3 14B Instruct | 31.5 | 33.0 | 31.5 | 33.0 | 35.5 | 34.5 |
| Qwen3-8B | 26.0 | 24.0 | 22.0 | 23.5 | 23.5 | 27.0 |
| Qwen3.5-9B | 22.5 | 19.5 | 13.5 | 15.5 | 14.5 | 14.0 |
| Phi-4 Mini Reasoning | 11.5 | 7.5 | 8.5 | 9.0 | 8.5 | 9.0 |
| Phi-4 Reasoning | 9.5 | 5.5 | 5.0 | 7.0 | 7.0 | 7.0 |
Table 13. Accuracy (%) for a subset of models in one canonical-order run under each shot condition. Best result per model is shown in bold.
We extract the requested JSON answer and use fixed fallbacks for a final-answer label or a plain first line when a model violates the output format. Our scoring strips surrounding whitespace, lowercases both strings, and otherwise requires exact equality with any accepted answer. Statistical resampling keeps the six orders of an item together. We use 10,000 item-bootstrap samples for accuracy and paired-difference intervals, 100,000 sign flips for paired model comparisons, and 10,000 within-item permutations for clue-order tests, all with the same seed as in the experiments. Holm correction is applied separately to the model-comparison and order-effect families.
| Feature | All | Open | Prop. | Human |
| Answer frequency | .45 (< .001) | .44 (< .001) | .42 (< .001) | .34 (< .001) |
| Average clue frequency | .10 (.168) | .05 (.461) | .17 (.015) | .19 (.006) |
| Minimum clue frequency | −.07 (.329) | −.10 (.168) | .05 (.480) | .01 (.846) |
| Average clue length | .11 (.117) | .12 (.099) | .10 (.160) | .00 (.945) |
| Answer length | −.13 (.067) | −.08 (.283) | −.17 (.014) | −.14 (.052) |
| Named-entity clues | −.03 (.637) | −.05 (.449) | .03 (.707) | −.01 (.880) |
| Original disambiguation entries | .38 (< .001) | .38 (< .001) | .31 (< .001) | .25 (< .001) |
| Wikipedia page views | .25 (< .001) | .25 (< .001) | .20 (.005) | .28 (< .001) |
caption: tbl:14: Item-level Spearman correlations. Each cell shows , with the two-sided -value in parentheses. All denotes mean accuracy across the 27 model conditions, while Open and Prop. give the corresponding open-weight and proprietary means.
:::
OpenAI settings GPT models are queried through the OpenAI Chat Completions API. GPT-4o and GPT-4.1 are identified by dated snapshots (see the bottom of Table 3), but Sol and Luna did not have such a snapshot yet. We ran the experiments in July 2026. The experiments use either max or no reasoning effort and a maximum completion budget of 2,048 tokens, which includes hidden reasoning tokens.
Benchmark Characterization
We test whether item difficulty is associated with simple lexical and source characteristics. We compute every feature once per test item and correlate it with item-level success using Spearman’s . For each model condition, correctness is averaged over the six clue orders. The model outcome is then the mean across all 27 conditions. Human success is the mean of the three human judgments for each item. Table 14 additionally separates the open-weight and proprietary model conditions. We process answers and clues with the Dutch spaCy model nl_core_news_sm. Corpus frequencies are lemma counts from the one-million-sentence Dutch 2023 news corpus in the Leipzig Corpora Collection (Goldhahn et al., 2012), normalized per million alphabetic tokens. The features are:
Answer frequency: the frequency of the answer lemma.
Average clue frequency: the mean frequency of all lemmatized alphabetic non-stopwords across the three clues.
Minimum clue frequency: the frequency of the least frequent clue in that set.
Average clue length: the mean number of alphabetic tokens per clue, including stopwords.
Answer length: the number of characters in the answer.
Named-entity clues: the number of clues that contained at least one named entity.
Original disambiguation entries: the number of linked entries on the answer’s Dutch Wikipedia disambiguation page before filtering.
Wikipedia page views: the mean stored 30-day page-view count of the three retained source entries.
Answer frequency has the strongest association with mean model accuracy (), followed by the number of entries on the original disambiguation page () and Wikipedia page views (). The corresponding correlations are similar for open-weight and proprietary models. Minimum clue frequency, clue length, and the number of clues containing a named entity have little relation to mean model accuracy. These results suggest that familiar answers and richer source pages make an item easier, but that benchmark difficulty is not largely explained by the simple properties tested here.
AI Assistance
We used AI tools mainly for coding assistance, and a bit for proofreading. All AI-generated output was verified by the authors.
References
- [2]Harethah Abu Shairah, Somayah AlHarbi, Abdulaziz AlHussein, Sameer Alsabea, Omar Shaqaqi, Hebah AlShamlan, Omar Knio, and George Turkiyyah. 2025. ALARB: An Arabic legal argument reasoning benchmark. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 389–406, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2510.00694
- [3]Rawan Al-Matham, Kareem Darwish, Raghad Al-Rasheed, Waad Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, Norah Alzahrani, Eman AlBilali, Nizar Habash, Abdelrahman El-Sheikh, Muhammad Elm allah, Haonan Li, Hamdy Mubarak, Mohamed Anwar, Zaid Alyafeai, and 24 others. 2025. BALSAM: A platform for benchmarking Arabic large language models. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 258–277, Suzhou, China. Association for Computational Linguistics.
- [4]Bjarki Ármannsson, Finnur Ágúst Ingimundarson, and Einar Freyr Sigurðsson. 2025. An Icelandic linguistic benchmark for large language models. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 37–47, Tallinn, Estonia. University of Tartu Library.
- [5]Tong Chen, Zimu Wang, Yiyi Miao, Hoaran Luo, Yuanfei Sun, Wei Wang, Zhengyong Jiang, Procheta Sen, and Jionglong Su. 2025. MedFact: A large-scale Chinese dataset for evidence-based medical fact-checking of LLM responses. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32340–32353, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2509.17436
- [6]Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, and Hao Wang. 2025. LAiW: A Chinese legal large language models benchmark. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10738–10766, Abu Dhabi, UAE. Association for Computational Linguistics.
- [8]Wietse de Vries, Martijn Wieling, and Malvina Nissim. 2023. DUMB: A benchmark for smart evaluation of Dutch models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7221–7241, Singapore. Association for Computational Linguistics.
- [9]Habib Yağız Demir, Ümit Atlamaz, and Susan Üsküdarlı. 2026. TimeRes: A Turkish benchmark for evaluating temporal understanding of large language models. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 910–920, Rabat, Morocco. Association for Computational Linguistics.
- [10]Rania Elbadry, Sarfraz Ahmad, Ahmed Heakl, Dani Bouch, Momina Ahsan, Muhra AlMahri, Marwa Elsaid Khalil, Yuxia Wang, Salem Lahlou, Sophia Ananiadou, Veselin Stoyanov, Jimin Huang, Xueqing Peng, Preslav Nakov, and Zhuohan Xie. 2026. SAHM: A benchmark for Arabic financial and shari’ah-compliant reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 34509–34536, San Diego, California, United States. Association for Computational Linguistics.
- [11]Yakup Abrek Er, İlker Kesen, Gözde Gül Şahin, and Aykut Erdem. 2026. Cetvel: A unified benchmark for evaluating language understanding, generation and cultural capacity of LLMs for Turkish. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1052–1085, Rabat, Morocco. Association for Computational Linguistics.
- [12]Yicheng Fu, Zhemin Huang, Liuxin Yang, Yumeng Lu, and Zhongdongming Dai. 2025. CHENGYU-BENCH: Benchmarking large language models for Chinese idiom understanding and use. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2355–2366, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2506.18105
- [14]Ine Gevers and Walter Daelemans. 2026. Do you get the hint? Benchmarking LLMs on the board game concept. In Findings of the Association for Computational Linguistics: ACL 2026, pages 24358–24371, San Diego, California, United States. Association for Computational Linguistics.arxiv.org/abs/2510.13271


