betaFeedback

WorksComputer Science

Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects

Abstract

How does Jev compare with other systems on Arabic tasks, and how does its performance change between Modern Standard Arabic (MSA) and dialects? We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol. The Arabic-focused frozen suite includes English controls in three parallel resources, which we also remove in a separate descriptive comparison. Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate. Removing the recorded English controls, repeated cross-task items or both leaves the full system order unchanged. Relative performance and gains between Jev versions vary across task families. In content-matched comparisons within Jev 1.14.0, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero. Paired contrasts with Jev 1.13.0, Gemma-4-31B and Jais-2-8B show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive. These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps.

Introduction

Many language tasks require a decision with a fixed set of possible answers: assign an intent, identify a dialect, mark harmful content, or give a rating. Jev addresses such tasks through typed questions rather than generated text. An earlier study evaluated this interface across task types and compared Jev with large language models (LLMs) by scoring the probabilities of fixed answer codes [1]. This provides a starting point for comparing task decisions while allowing each system to retain its own inference interface. The question for an Arabic-focused study is how well this comparison holds across the tasks and language varieties represented in its evaluation.

Arabic evaluation already includes broad language-understanding suites, knowledge questions and dialect resources. ARLUE covers multi-dialectal language understanding, while ArabicMMLU evaluates subject knowledge through questions in Modern Standard Arabic (MSA) [2, 3]. DialectalArabicMMLU provides adapted versions of shared questions in Arabic dialects [4]. These resources support different views of performance. A broad task average summarizes the chosen suite, while a comparison of matched MSA and dialect questions examines changes for versions of the same source content. Together, these views describe performance across the selected task inventory and changes across matched language varieties.

This paper asks two questions:

RQ1: How does Jev perform on Arabic tasks compared with other decision models and LLMs? RQ2: How does Jev’s performance vary between MSA and Arabic dialects on content-matched tasks, and how do its accuracy gaps compare with those of selected LLMs?

The first question covers overall performance, task-family differences, both Jev versions and coverage. The second describes paired accuracy differences by dataset and dialect, then compares Jev 1.14.0’s gaps with Jev 1.13.0 and two LLMs selected by their overall RQ1 scores. We distinguish identifying a dialect from performing a task written in that dialect.

Both questions are descriptive. They were selected after evaluation from a broader preregistered study; the original plan and test records remain unchanged. Section C.4 explains this scope and identifies the companion records for the original analyses.

The evaluation suite contains 43 tasks in 9 families and compares 67 systems: 47 decision models and 20 LLMs. The inventory counts evaluated versions or checkpoints separately, including both Jev versions. Within this scope, the paper makes three contributions:

  • An organization of existing Arabic datasets into typed decision tasks, including parallel language varieties and a range of task label schemes.

  • A shared zero-shot evaluation protocol that documents model interfaces, scoring, task support and answered-item coverage alongside performance.

  • A comparison of models across Arabic task families and a source-aware analysis of matched MSA–dialect differences, including per-dialect results and paired contrasts with a fixed model panel.

Jev 1.14.0 has a mean task-primary score of 69.24 after multiplying by 100, above Jev 1.13.0’s 68.77 and below Gemma-4-31B’s 71.81. The English-control removal sensitivity gives 69.18 for Jev 1.14.0 and 71.75 for Gemma-4-31B, with the full system order unchanged. Its relative performance changes across task families. The matched dialect comparison also varies by dataset, with a smaller pooled gap on SIB-200 than on the other retained sets. These results motivate reporting task profiles and matched language differences alongside the aggregate mean.

The comparison is bounded by the selected tasks, available answers and implemented readouts. An average across different task metrics is not an overall accuracy, and a comparison based on different supported items needs separate coverage information. The following sections place the study in prior work, describe the benchmark and protocol, and report each research question before discussing the findings and their limits.

Related Work

Zero-Shot Classification and Decision Interfaces

Zero-shot text classification can express label meanings in natural language. Yin et al. [5] studied this setting through a textual-entailment formulation across different label types. For a generative large language model (LLM), a classification decision can instead be read from probabilities assigned to fixed answer codes. The earlier Jev evaluation used this approach to compare open LLMs with Jev on shared requests [1]. Our open-LLM comparison follows this restricted-answer approach, with the supported codes and answered-item coverage made explicit in § 4.

The earlier study also provides the methodological starting point for our use of Jev [1]. It evaluated a model that accepts a shared state and typed questions for option selection, ordered scoring and yes/no decisions, and examined both task performance and confidence. We retain these question types while building an Arabic-focused comparison across decision models and LLMs. This paper focuses on Arabic task performance and content-matched MSA–dialect differences. Returned probabilities provide the task decisions used by the shared scoring protocol.

Arabic Evaluation and Dialect Coverage

Arabic model evaluation already draws on suites that cover different tasks and language varieties. ARLUE, introduced with ARBERT and MARBERT, combines datasets for multi-dialectal language-understanding evaluation and was used to compare fine-tuned models [2]. AlGhafa provides a multiple-choice benchmark for Arabic language models [6]. These studies establish task coverage and evaluation format as distinct choices when comparing Arabic systems. Our study uses a shared zero-shot protocol for Jev, other decision models and LLMs, with task-level results and coverage reported alongside aggregate scores.

Knowledge evaluation and dialect evaluation also address different questions. ArabicMMLU uses school-examination questions in Modern Standard Arabic (MSA) across subject areas [3]. DialectalArabicMMLU adapts MMLU-Redux questions into Arabic dialects, supporting comparisons across language varieties on matched question content [4]. Dialect identification instead asks a model to recognize a text’s variety, as in the country- and province-level tasks of NADI 2021 [7]. Success at identifying a dialect therefore answers a different evaluation question from success at answering questions written in that dialect.

We build on this evaluation tradition by representing the datasets in our suite as typed questions within one evaluation protocol. The paper connects broad model and task-family comparisons to a focused within-system comparison of matched MSA and dialect content. It also preserves the difference between parallel comparisons of shared content and analyses of independently collected native text, as detailed in § 3.

Benchmark Suite

Scope and Selection

We compare Jev with other decision models and large language models (LLMs) on existing Arabic classification and rating datasets. The suite contains 43 tasks in 9 families. Parallel sets hold source content fixed across Modern Standard Arabic (MSA) and dialects, while other tasks evaluate decisions such as sentiment, intent and rating. These controlled contrasts differ from evaluation on independently collected dialect text, where both content and language variety can change.

Tasks added during suite expansion followed a selection rule recorded before evaluation: declared licensing or usage terms, available reference labels, answers determined from the supplied input, and minimum sample-size and class-count requirements. The rule excluded model-written text or labels unless model output was itself the object being evaluated. These criteria apply to the additions, rather than uniformly to every original dataset. The suite does not represent every Arabic task, dialect or application.

The frozen configurations of Belebele, SIB-200 and DialectalArabicMMLU include English controls alongside their Arabic varieties. These controls contribute to each resource’s pooled primary score and therefore to the full-suite mean. The aggregate describes this frozen Arabic-focused inventory, rather than an Arabic-only subset. We therefore also report a descriptive sensitivity excluding the recorded English configurations; it retains Arabic and romanized-MSA examples under the same primary English-instruction template (T-EN) (section B.1).

Task Families

Table 1 summarizes coverage; section A.1 lists every task, question type, primary metric and evaluation size. Tasks within the suite measure different abilities: Belebele tests reading comprehension from a passage, whereas ArabicMMLU draws on school examinations across subjects [3, 8]. We therefore report task-level results alongside family summaries. Family labels organize reporting; they are not an assumption of common difficulty or identical primary metrics.

IDFamilyTasksCoverage
AVarieties and script4Reading and topic decisions across language varieties and scripts
BNative dialect text3Dialect identification, sentiment, sarcasm, and stance
CLabel ladders5Tasks grouped around coarse-to-fine labels, including sentiment and intent
DApplication tasks12Intent, inference, relevance, similarity, moderation, and safety
EKnowledge2Multiple-choice questions across subject areas
FNews and claims6Subjectivity, propaganda, claims, authorship, and truthfulness
GCommonsense and culture4Cultural knowledge, plausibility, humor, and physical commonsense
HRating and grading4Readability and assessment of essays and short answers
IArabic language3Poetic meter, word meaning, and grammar
Total43

Table 1. Task families in the evaluation suite. Counts refer to tasks in our evaluation, not the number of labels, domains, or questions within each source dataset. Family membership organizes reporting; each research question uses its own specified tasks and conditions.

From Dataset Labels to Typed Questions

Each item has an input, called the state, and one or more named questions. Choice selects an option from a specified set, Noul asks whether a property is present, and Score assigns a value on an ordered scale. Choice and Score answers include probabilities over options or levels; Noul gives the probability of yes. The evaluation protocol describes how each system supplies these answers.

Several annotations can become separate questions about one state. Our ArSarcasm-v2 mapping asks about sarcasm, sentiment and language variety for each tweet [9]. Multilabel emotion classification uses one Noul question per emotion, allowing several emotions per item. Figure 1 shows how the shared representation passes through the system interfaces to scoring.

Figure 1. Evaluation flow from a shared task representation through distinct model interfaces to label-based scoring. The Arabic sentiment example and English gloss are self-authored illustrations; no benchmark item, model prediction or measured probability is shown. Actual requests use frozen task wording, and supported questions and coverage depend on the system and condition.

Mapping choices define the label scheme being evaluated. For BAREC readability, we use the seven-level scheme because its nineteen-level scheme exceeds Jev’s supported Score range [10]. The evaluator uses numerical level indices, distinct from the human-readable level names.

Evaluation Splits and Sampling

Each task uses the labelled evaluation split in its preregistered mapping. For NADI 2021, this is the development split, with training data used for piloting [7]. Added tasks are capped at 300 items, sampled within strata using a fixed hash order. Their sampler deduplicates texts, excludes groups with conflicting labels and, where pilots are carved from the pool, removes those pilots from evaluation. Original tasks retain their own split and sampling rules.

The frozen inventory contains 133,689 task-item entries across the suite. These are not necessarily unique texts or independent observations, and the count does not establish that every system answered every question. We report coverage with performance; the protocol and results state each comparison’s pairing and resampling scope.

The primary task comparison uses each task’s frozen evaluation inventory; paired dialect summaries retain only eligible matched records. Full task size, answered-item coverage and paired source count therefore have different meanings.

Overlap and Label Constraints

MMLU Arabic [11] and DialectalArabicMMLU [4] share 2,384 distinct requests, affecting 2,391 MMLU items because requests can recur within a dataset. The registered task-level summaries retain this overlap; a separate descriptive sensitivity analysis removes the affected MMLU items before recomputing overall means. We report that analysis alongside the relevant comparisons.

Identical requests can also have conflicting reference labels, as in MADAR city identification [12]. A predictor assigning one fixed answer per identical request achieves its highest accuracy by choosing the most frequent label in each request group. This bound helps interpret city accuracy; it does not directly bound other metrics or each realized run of a stochastic predictor. These constraints inform the coverage, pairing and uncertainty rules described next.

Systems and Evaluation Protocol

Systems and Comparison Scope

We evaluate 67 systems: 47 decision models and 20 LLMs, including pinned Jev versions jev-1.13.0 and jev-1.14.0. The comparison covers hosted decision services, Arabic and general-use open decision models, Arabic-centric LLMs, and the Qwen and Gemma families (Table 2). Section A.3 gives the systems and declared readouts; complete pinned revisions are supplied in the companion files. Group differences are descriptive and cannot isolate effects of architecture or training.

GroupSystemsScope
Hosted decision models4Jev versions and the Microsoft and OpenAI decision services
Arabic decision models5Arabic-focused releases in the pinned comparison
Other general decision models5Previously selected open decision-model releases
Arabic-centric LLMs9Jais, Jais 2, ALLaM and Fanar
Qwen LLMs6Qwen3.5 sizes and the Qwen3.8 comparison model
Gemma LLMs5Gemma 4 releases at the pinned sizes
Additional vendor decision models33Vendor releases added for the descriptive comparison
Total67

Table 2. Active systems grouped for presentation. Counts refer to evaluated versions or checkpoints, so related releases are counted separately. Arabic-centric and general-use labels follow the pinned study inventory. Group membership does not imply identical inference interfaces or coverage.

4 pinned systems were deferred: d3-turbo, h2o-lightning-4b, clef-omni, and Jais-family-30B. They do not enter the reported inventory or its means. The companion records document these deferrals.

Interfaces and Probability Readouts

Each backend presents the shared state, questions and labels through its system’s interface. Hosted decision models receive typed service requests; open decision models use backend adapters that map returned probabilities to benchmark labels. Choice selects the highest-probability option, Noul gives the probability of yes, and Score predicts the probability-weighted mean of ordered level indices. Common scoring therefore allows different request serializations.

Open LLMs receive a prompt with the state, instructions and candidate answers. The default readout scores single-token answer codes at the answer position, with thinking disabled in the chat template, then normalizes their probabilities over the allowed codes. The distribution is conditional on those codes: it excludes other continuations and need not be calibrated to correctness.

Questions exceeding the context limit or unsupported by the default readout are recorded as skipped and affect coverage.

Primary Condition and Eligibility

The primary comparison uses the frozen T-EN task templates and the default readout described above. Frozen wording examples and the distinction between task templates and LLM wrappers appear in section A.5. This fixed condition supports the reported model comparison; it does not establish performance under other prompt or answer representations.

Metrics and Coverage

Primary metrics include accuracy, macro-F1, positive-class F1, multilabel Jaccard similarity and rating Spearman correlation (Table 6). Accuracy measures the share of correct primary decisions. Positive-class F1 scores the task’s named positive label; macro-F1 gives equal weight to the classwise F1 values for labels present in gold or predictions[13]. Multilabel tasks average the item-level Jaccard ratio of label-set intersection to union[14]; we assign one when both sets are empty and mark a label positive when its Noul probability is at least 0.5. Rating tasks use Spearman correlation between the predicted Score and source gold, measuring their rank association rather than exact rating agreement[15].

FamilyTaskTypesMetricItemsSource
ABelebeleCAcc.7,200[8]
ASIB-200CAcc.11,044[16]
ADialectalArabicMMLUCAcc.21,945[4]
ATUNIZICAcc.3,000[17]
BNADI 2021C2Macro-F15,000[7]
BArSarcasm-v2N, C2Macro-F13,000[9]
BMawqifC2, NMacro-F1619[18]
CMADAR-26N, C3Acc.5,200[12]
CLABRC3, SSpearman5,000[19]
CSemEval-2018 AffectN11, S4Jaccard1,518[20]
CMASSIVE ar-SAC2Acc.2,974[21]
CSemEval-2016 ABSA hotelsCMacro-F1300[22]
DArBanking77CMacro-F115,534[23, 24]
DXNLI ArabicCAcc.5,010[25]
DMIRACL ArabicNF13,000[26]
DSemEval-2017 STSSSpearman250[27]
DEgyptian offensive and hate speechN, CMacro-F11,634[28]
DAraTrustCAcc.506[29]

Table 6. Task inventory and source attribution.

Overall and family summaries are unweighted means of available task-primary scores, omitting missing or undefined scores. These means combine different metrics and can cover different task sets across systems; per-task results and coverage are therefore needed to interpret them.

Task coverage is the number of available primary-metric item records divided by the frozen task size; mean coverage averages these fractions. It measures primary-score support, not completion of every auxiliary question. For a multilabel primary metric, an item requires every binary primary head. The main results apply this complete-primary-item rule to the two refused AraTox heads found in one system, and retain the original export separately (section C.1). We distinguish unsupported questions, missing responses and undefined metrics. Paired comparisons further restrict eligibility to shared records.

Statistical Protocol and Reproducibility

The broader preregistration is available at https://doi.org/10.5281/zenodo.23586552. The paper’s descriptive scope was chosen after evaluation and does not redefine its hypotheses or correction families (section C.4).

Both paper questions present descriptive comparisons selected after evaluation. Paper RQ1 reports overall and task-family scores, coverage and sensitivity summaries without confirmatory hypothesis-test claims. Paper RQ2 uses the same four matched datasets for both Jev versions, Gemma-4-31B and Jais-2-8B. The latter two have the highest overall available-task RQ1 means among general-use and Arabic-centric LLMs, respectively. Selection does not use dialect penalties, but remains conditional on the evaluated inventory and post-evaluation scope.

We join responses to exact frozen requests, then join systems by task and item identifier with matching source, variety, input and gold label. DialectalArabicMMLU source keys include the domain and question identifier. All four systems have valid primary answers for the same 40,618 eligible item rows, producing 31,280 source–variety pairs. ArBanking77 first averages the same available paraphrases separately on each side of each source pair for every model; the matched analysis uses accuracy rather than its RQ1 macro-F1. No unsupported or missing answer is treated as an error in this panel.

Let gm,d=accm,d−accm,MSAg_{m,d} = \mathrm{acc}_{m,d} - \mathrm{acc}_{m,\mathrm{MSA}} on the matched source population. A negative gap means lower dialect accuracy. We report each dialect separately and an equal-source–variety-pair average within each dataset. The cross-system contrast is gJev 1.14,d−gc,dg_{\mathrm{Jev\ 1.14},d} - g_{c,d} for comparator cc; positive values mean a less negative gap for Jev. We do not pool different datasets. Whole articles are resampled for Belebele and SIB-200, and whole task/source identifiers for DialectalArabicMMLU and ArBanking77. Every draw keeps linked models, varieties and paraphrases together. The expanded analysis uses 10,000 draws per dataset and pointwise 95% percentile intervals for accuracy gaps; fixed seeds and the earlier interval correction are documented in sections B.4 and C.2.

For cross-system contrasts we also report approximate simultaneous 95% bands covering the complete 69-contrast grid, including dialect-specific and pooled rows. Within each dataset, bands use the 98.75th percentile of the maximum absolute centered bootstrap contrast divided by its bootstrap standard deviation; the radius is that critical value times the contrast’s standard deviation. Bonferroni allocation across four datasets accounts for their shared content without assuming independence. These new descriptive bands do not replace the original preregistered RQ2 tests or correction family. A separate exploratory sensitivity removes pairs with identical full state/question inputs across MSA and dialect versions; its intervals are pointwise.

All current primary T-FEN scores, item counts and coverage fractions were reproduced from the existing responses (section C.3). This verifies the primary numerical reporting, rather than historical server settings or secondary confidence analyses. Original test outputs and qualifications remain in the companion archive (section C.4).

The system ranking is descriptive; these means alone do not establish statistical separation.

Frozen task mappings, templates and item identifiers specify intended inputs; the inventory records pinned model revisions. Generated tables and source hashes trace the manuscript’s statements to those records and saved outputs. Section A.4 defines the recorded per-request timing and processing rates reported beside the evaluation results. The companion files retain full model-source metadata, audit evidence and original test records.

Results

Paper RQ1: Overall and Task-Family Performance

Gemma-4-31B has the highest available-task mean at 71.81 on the scale multiplied by 100. Jev 1.14.0 scores 69.24, above Jev 1.13.0’s 68.77 and 2.57 points below Gemma. Jev 1.14.0 ranks 6th among all 67 systems. Table 3 shows both Jev versions, the other hosted decision services and the highest-mean system from each original presentation group, with recorded processing rates beside their scores; the full ranking is in Table 9. Means use available primary scores, so task and item coverage qualify this ordering.

SystemScoreRecords/sArabic-onlyTasksCoverage (%)
Gemma-4-31B71.817.8671.7543/43100.00
h2o-lightning-31b71.127.1171.0743/43100.00
jev-1.14.069.244.9369.1843/43100.00
OpenAI-Decisions-gpt-6-luna68.793.5768.7243/4399.84
jev-1.13.068.775.2268.7143/43100.00
SII-Decision-27B68.392.0668.3143/43100.00
Microsoft-Decision-166.062.7965.9743/43100.00
Qwen3.5-27B65.966.4265.8843/43100.00
Qwen3.8-27B65.156.5365.0543/43100.00
ara-jev-Faisal59.933.6559.7640/4393.02
Jais-2-8B54.5436.6854.4643/43100.00
Jais-family-13B46.841.3746.7543/4399.60

Table 3. Fixed-group summary ordered by unrounded score. Scores are mean task-primary metrics multiplied by 100. Records/s uses recorded processing time under each backend’s execution settings (section A.4). Arabic-only removes English controls; tasks and coverage describe the frozen evaluation population.

SystemScoreRecords/sTasksCoverage (%)
Gemma-4-31B71.817.8643/43100.00
h2o-lightning-31b71.127.1143/43100.00
d370.784.0543/43100.00
GLiDe-1.270.364.2343/43100.00
pplx-decider-v1.269.904.8843/43100.00
SystemScoreRecords/sTasksCoverage (%)
jev-1.14.069.244.9343/43100.00
OpenAI-Decisions-gpt-6-luna68.793.5743/4399.84
jev-1.13.068.775.2243/43100.00
Gemma-4-26B-A4B68.5526.5643/43100.00
SII-Decision-27B68.392.0643/43100.00
Kev-27B67.514.7043/43100.00
d3-flash67.049.6043/43100.00
clef66.821.9543/43100.00
strands-decider-26B-A4B-gemma466.704.0643/43100.00
Gemma-4-12B66.6418.7143/43100.00
Microsoft-Decision-166.062.7943/43100.00
Qwen3.5-27B65.966.4243/43100.00
Qwen3.8-27B65.156.5343/43100.00
strands-decider-12B-gemma464.245.5243/43100.00
d3-mini64.009.9143/43100.00
pplx-decider-flash-v1.263.4911.1343/43100.00
clef-flash60.854.0043/43100.00
strands-decider-E4B-gemma460.787.3443/43100.00
SII-Decision-4B60.394.5643/43100.00
ara-jev-Faisal59.933.6540/4393.02
Intern-Decision-4B59.755.2742/4397.67
Qwen3.5-9B59.4319.4343/43100.00
Gemma-4-E4B59.3636.0843/43100.00
Vela-2.0-9B57.654.2843/43100.00
Kev-4B56.8714.2843/43100.00
d1-3B56.8240.4743/43100.00
d3-nano56.2414.2643/43100.00
Vela-2.0-4B55.774.3243/43100.00
Qwen3.5-4B54.9921.7843/43100.00
Jais-2-8B54.5436.6843/43100.00
ALLaM-7B54.2332.6743/43100.00
Fanar-2-27B53.689.7243/43100.00
strands-decider-E2B-gemma453.668.6343/43100.00
Intern-Decision-2B51.757.1042/4397.67
d3-lite50.4314.3943/43100.00
strands-decider-2B-qwen3.550.378.9443/43100.00
h2o-lightning-4b-nemotron-nano49.4522.6643/43100.00
Jais-family-13B46.841.3743/4399.60
Gemma-4-E2B45.8460.9343/43100.00
Jais-family-6.7B45.622.5643/4399.60
Qwen3.5-2B45.6148.2343/43100.00
Vela-2.0-0.8B44.465.9443/43100.00
Intern-Decision-0.8B43.936.8242/4397.67
d3-edge39.9635.9143/43100.00
Jais-family-2.7B39.696.2043/4399.60
Jev-AR37.4261.2022/4350.86
Vela-2.0-0.3B35.9263.2143/43100.00
Qwen3.5-0.8B35.1357.5043/43100.00
jeb34.2079.1140/4393.01
jeb-typed-decisions31.8079.0540/4393.01
GLiNER2.5-Decide31.6923.8343/4399.998
GLiNER2.5-Decide-1B31.4119.2243/4399.998
GLiNER2.5-multi-Decide31.2148.9443/4399.998
Jais-family-590M30.8823.4243/4399.60
Laya-multilingual30.4355.4343/43100.00
Jais-family-256M29.3143.4243/4399.60
QARAR27.0759.0843/43100.00

Table 9. All systems ordered by the unrounded evaluation mean. Scores are mean task-primary metrics multiplied by 100. Records/s uses recorded processing time under each backend’s execution settings (section A.4). Tasks and coverage describe the scored population. Masked means are in the companion sensitivities.csv.

A descriptive sensitivity comparison after the support correction uses the same 40 tasks with finite scores for every system in the fixed summary group (section B.3). Two close comparisons change order: Jev 1.13.0 versus OpenAI Decisions, and Qwen3.5-27B versus Microsoft Decision-1. This task-level restriction does not ensure identical answered items. Removing the recorded English controls gives means of 69.18 for Jev 1.14.0 and 68.71 for Jev 1.13.0, compared with 71.75 for Gemma-4-31B. Removing repeated MMLU items changes any system’s overall mean by less than 0.021 aggregate points. The complete 67-system order is unchanged under English-control removal, overlap removal or both; these descriptive checks do not establish statistical separation (section B.1).

Figure 2 shows that the aggregate ordering varies across task families, using the same selected systems as Table 3. The detailed comparison of both Jev versions with Gemma-4-31B and h2o-lightning-31b is retained in Table 10; the two comparators have the highest overall saved means among the LLMs and non-Jev decision models, respectively. Jev 1.14.0 exceeds Gemma-4-31B in the saved knowledge mean (81.95 versus 80.19) and news-and-claims mean (70.57 versus 68.65). Its largest family-level deficit against Gemma is in Arabic language tasks, at 10.01 aggregate points. These contrasts describe variation in relative performance within families; they do not establish that one family is intrinsically harder than another.

Figure 2. Task-family means multiplied by 100 for the fixed summary population, ordered by overall mean. Families: A, varieties/script (including English controls); B, native dialect; C, label ladders; D, applications; E, knowledge; F, news/claims; G, commonsense/culture; H, rating/grading; I, Arabic language. Colors use a fixed 0–100 scale. Family means use available task scores; tasks and coverage summarize the evaluation population. Numerical validation is described in section C.3.

FamilyJev 1.14Jev 1.13Gemma 31BH2O 31B
A Varieties and script83.0082.8984.2484.13
B Native dialect text48.1448.0853.2252.56
C Label ladders52.0351.1655.1555.33
D Application tasks74.0873.4177.8778.74
E Knowledge81.9582.4780.1980.80
F News and claims70.5769.4768.6560.14
G Commonsense and culture82.2882.0386.0486.66
H Rating and grading73.4973.7173.2974.48
I Arabic language47.0846.6057.0958.50

Table 10. Task-family means multiplied by 100. Gemma 31B is Gemma-4-31B; H2O 31B is h2o-lightning-31b. All four systems have full primary-item coverage. Family composition and primary metrics differ.

The change between Jev versions is also uneven across families. The saved Jev 1.14.0 mean is lower than Jev 1.13.0 for knowledge and rating/grading, while the other family means are higher. The overall increase therefore does not represent a gain in every family.

These comparisons describe overall and task-family performance rather than testing the original hypotheses. The companion archive preserves the original RQ1 test records and their unresolved qualifications (section C.4).

Paper RQ2: Content-Matched MSA and Dialect Performance

Matched accuracy across datasets. Jev 1.14.0 has lower pooled dialect accuracy than MSA accuracy on Belebele, DialectalArabicMMLU and the corpus-linked Palestinian subset of ArBanking77. The respective gaps are -5.69, -3.63 and -7.94 percentage points, with pointwise source-aware intervals below zero. SIB-200’s pooled gap is -0.35 points and its interval includes zero (Table 4). This small average does not establish equivalence in every dialect. The table also shows absolute accuracy for the fixed model panel, so a small gap can be distinguished from strong performance on both versions.

SystemMSA (%)Dialect (%)Gap [95% interval]
Belebele: 4,500 pairs; 488 clusters
Jev 1.13.095.4489.33-6.11 [-7.24, -4.97]
Jev 1.14.095.5689.87-5.69 [-6.81, -4.57]
Gemma-4-31B94.4489.56-4.89 [-6.13, -3.65]
Jais-2-8B82.1173.20-8.91 [-10.90, -6.95]
SIB-200: 8,032 pairs; 393 clusters
Jev 1.13.088.1587.59-0.56 [-1.17, +0.07]
Jev 1.14.087.6587.30-0.35 [-1.03, +0.35]
Gemma-4-31B88.8488.05-0.80 [-1.56, -0.05]
Jais-2-8B81.1880.02-1.16 [-2.15, -0.17]
DialectalArabicMMLU: 15,675 pairs; 3,135 clusters
Jev 1.13.085.6881.10-4.58 [-5.30, -3.88]
Jev 1.14.084.7581.12-3.63 [-4.31, -2.95]
Gemma-4-31B78.9875.53-3.44 [-4.24, -2.63]
Jais-2-8B53.2449.40-3.84 [-4.92, -2.75]
ArBanking77 (Palestinian): 3,073 pairs; 3,073 clusters
Jev 1.13.075.3066.81-8.49 [-9.83, -7.19]
Jev 1.14.074.8566.91-7.94 [-9.27, -6.64]
Gemma-4-31B75.8771.48-4.39 [-5.48, -3.30]
Jais-2-8B20.3716.56-3.81 [-4.73, -2.88]

Table 4. Matched accuracy for the fixed RQ2 model panel. Gaps are dialect minus MSA in percentage points, with pointwise 95% whole-cluster percentile-bootstrap intervals. Pairs count source–variety comparisons; repeated MSA sources and paraphrases remain linked in resampling. ArBanking77 uses source-level means of paraphrase correctness. All four models use identical eligible items.

Variation between dialects. The pooled estimates conceal substantial differences (Figure 3). On Belebele, Jev 1.14.0’s point gaps range from -2.78 points for Egyptian to -8.56 for Mesopotamian Arabic. On DialectalArabicMMLU, the corresponding Saudi and Moroccan gaps are -1.66 and -7.24 points. These are descriptive within-dataset patterns, not a ranking of intrinsic dialect difficulty: varieties also differ in wording and translation quality. Table 12 reports every dialect’s absolute accuracy and gap for all four models.

Figure 3. Jev 1.14.0’s matched dialect-minus-MSA accuracy gap for every retained dataset–dialect cell. Bars are pointwise 95% source/article-clustered intervals. The zero line denotes equal matched accuracy, not a tested equivalence margin. The main table pools these cells within each dataset; it does not pool datasets.

DialectSystemPairsMSA (%)Dialect (%)Gap [95% interval]
Belebele
EgyptianJev 1.13.090095.4493.11-2.33 [-3.68, -1.01]
EgyptianJev 1.14.090095.5692.78-2.78 [-4.30, -1.32]
EgyptianGemma-4-31B90094.4492.00-2.44 [-4.01, -0.89]
EgyptianJais-2-8B90082.1175.11-7.00 [-9.64, -4.43]
NajdiJev 1.13.090095.4490.89-4.56 [-6.19, -2.99]
NajdiJev 1.14.090095.5691.33-4.22 [-5.84, -2.65]
NajdiGemma-4-31B90094.4491.33-3.11 [-4.77, -1.45]
NajdiJais-2-8B90082.1175.89-6.22 [-7.89, -3.63]
North LevantineJev 1.13.090095.4488.89-6.56 [-8.59, -4.55]
North LevantineJev 1.14.090095.5690.67-4.89 [-6.65, -3.11]
North LevantineGemma-4-31B90094.4490.00-4.44 [-6.43, -2.45]
North LevantineJais-2-8B90082.1176.00-6.11 [-8.95, -3.31]
MoroccanJev 1.13.090095.4487.89-7.56 [-9.65, -5.47]
MoroccanJev 1.14.090095.5687.56-8.00 [-10.12, -5.90]
MoroccanGemma-4-31B90094.4488.11-6.33 [-8.44, -4.31]
MoroccanJais-2-8B90082.1170.00-12.11 [-15.05, -9.13]
MesopotamianJev 1.13.090095.4485.89-9.56 [-11.84, -7.40]
MesopotamianJev 1.14.090095.5687.00-8.56 [-10.63, -6.56]
MesopotamianGemma-4-31B90094.4486.33-8.11 [-10.23, -6.08]
MesopotamianJais-2-8B90082.1169.00-13.11 [-16.02, -10.34]
SIB-200
EgyptianJev 1.13.01,00488.1587.05-1.10 [-2.18, 0.00]
DialectSystemPairsMSA (%)Dialect (%)Gap [95% interval]
EgyptianJev 1.14.01,00487.6587.45-0.20 [-1.34, +0.93]
EgyptianGemma-4-31B1,00488.8488.15-0.70 [-2.05, +0.62]
EgyptianJais-2-8B1,00481.1880.08-1.10 [-2.62, +0.41]
NajdiJev 1.13.01,00488.1587.95-0.20 [-0.51, 0.00]
NajdiJev 1.14.01,00487.6587.650.00 [-0.40, +0.40]
NajdiGemma-4-31B1,00488.8488.65-0.20 [-0.61, +0.19]
NajdiJais-2-8B1,00481.1880.68-0.50 [-1.17, +0.10]
North LevantineJev 1.13.01,00488.1587.95-0.20 [-1.36, +0.94]
North LevantineJev 1.14.01,00487.6588.15+0.50 [-0.71, +1.78]
North LevantineGemma-4-31B1,00488.8488.05-0.80 [-2.10, +0.51]
North LevantineJais-2-8B1,00481.1880.38-0.80 [-2.42, +0.86]
MoroccanJev 1.13.01,00488.1586.75-1.39 [-2.73, -0.10]
MoroccanJev 1.14.01,00487.6586.35-1.29 [-2.71, +0.10]
MoroccanGemma-4-31B1,00488.8486.95-1.89 [-3.44, -0.40]
MoroccanJais-2-8B1,00481.1879.28-1.89 [-3.88, +0.10]
MesopotamianJev 1.13.01,00488.1587.75-0.40 [-1.23, +0.41]
MesopotamianJev 1.14.01,00487.6587.650.00 [-0.89, +0.84]
MesopotamianGemma-4-31B1,00488.8488.25-0.60 [-1.55, +0.32]
MesopotamianJais-2-8B1,00481.1879.58-1.59 [-2.92, -0.29]
TunisianJev 1.13.01,00488.1587.55-0.60 [-1.76, +0.59]
TunisianJev 1.14.01,00487.6586.16-1.49 [-2.75, -0.30]
TunisianGemma-4-31B1,00488.8487.85-1.00 [-2.21, +0.10]
TunisianJais-2-8B1,00481.1879.08-2.09 [-3.68, -0.51]
South LevantineJev 1.13.01,00488.1587.65-0.50 [-1.82, +0.80]
South LevantineJev 1.14.01,00487.6586.95-0.70 [-2.00, +0.58]
South LevantineGemma-4-31B1,00488.8488.15-0.70 [-2.03, +0.61]
South LevantineJais-2-8B1,00481.1879.78-1.39 [-3.00, +0.20]
Ta’izzi-AdeniJev 1.13.01,00488.1588.05-0.10 [-0.97, +0.77]
Ta’izzi-AdeniJev 1.14.01,00487.6588.05+0.40 [-0.39, +1.19]
Ta’izzi-AdeniGemma-4-31B1,00488.8488.35-0.50 [-1.31, +0.30]
Ta’izzi-AdeniJais-2-8B1,00481.1881.27+0.10 [-1.04, +1.20]
DialectalArabicMMLU
EgyptianJev 1.13.03,13585.6882.33-3.35 [-4.27, -2.46]
EgyptianJev 1.14.03,13584.7581.66-3.09 [-4.08, -2.11]
EgyptianGemma-4-31B3,13578.9876.08-2.90 [-4.02, -1.82]
EgyptianJais-2-8B3,13553.2451.39-1.85 [-3.25, -0.38]
SaudiJev 1.13.03,13585.6882.68-3.00 [-3.99, -2.07]
SaudiJev 1.14.03,13584.7583.09-1.66 [-2.52, -0.80]
SaudiGemma-4-31B3,13578.9877.16-1.82 [-2.84, -0.83]
SaudiJais-2-8B3,13553.2449.79-3.44 [-4.85, -2.07]
MoroccanJev 1.13.03,13585.6877.16-8.52 [-9.76, -7.30]
MoroccanJev 1.14.03,13584.7577.51-7.24 [-8.42, -6.06]
MoroccanGemma-4-31B3,13578.9872.63-6.35 [-7.62, -5.07]
MoroccanJais-2-8B3,13553.2447.37-5.87 [-7.40, -4.31]
SyrianJev 1.13.03,13585.6880.19-5.49 [-6.67, -4.31]
SyrianJev 1.14.03,13584.7580.35-4.40 [-5.55, -3.29]
SyrianGemma-4-31B3,13578.9874.96-4.02 [-5.30, -2.71]
SyrianJais-2-8B3,13553.2448.01-5.23 [-6.79, -3.64]
EmiratiJev 1.13.03,13585.6883.13-2.55 [-3.51, -1.63]
EmiratiJev 1.14.03,13584.7583.00-1.75 [-2.68, -0.80]
EmiratiGemma-4-31B3,13578.9876.84-2.14 [-3.19, -1.05]
EmiratiJais-2-8B3,13553.2450.43-2.81 [-4.15, -1.47]
ArBanking77 (Palestinian)
PalestinianJev 1.13.03,07375.3066.81-8.49 [-9.83, -7.19]
PalestinianJev 1.14.03,07374.8566.91-7.94 [-9.27, -6.64]
PalestinianGemma-4-31B3,07375.8771.48-4.39 [-5.48, -3.30]

Table 12. Per-dialect matched accuracy and pointwise 95% intervals. Gaps are dialect minus MSA in percentage points; all four models use identical items.

Comparison with other systems. The pooled cross-system contrasts vary by dataset (Table 5). On Palestinian ArBanking77, the Jev 1.14.0-minus-Gemma gap contrast is -3.55 points and the Jev-minus-Jais contrast is -4.13 points. Both simultaneous bands lie below zero, indicating a larger penalty for Jev on this population. All contrasts outside these two Palestinian banking comparisons, whose single-dialect and pooled rows coincide, have simultaneous bands that include zero (Table 13). This does not establish equal penalties. Jais’s smaller banking gap accompanies only 20.37% MSA accuracy and 16.56% dialect accuracy, illustrating why a gap alone is insufficient to rank useful performance. The stronger absolute banking accuracy for Gemma and its smaller observed gap provide a more direct contrast with Jev on this matched population.

Datasetvs. Jev 1.13vs. Gemmavs. Jais
Belebele+0.42 [-1.02, +1.86]-0.80 [-2.82, +1.22]+3.22 [-0.22, +6.66]
SIB-200+0.21 [-1.15, +1.57]+0.45 [-1.11, +2.00]+0.81 [-1.36, +2.98]
DialectalArabicMMLU+0.95 [-0.33, +2.23]-0.19 [-1.87, +1.50]+0.21 [-1.97, +2.39]
ArBanking77 (Palestinian)+0.55 [-1.06, +2.17]-3.55 [-5.60, -1.50]-4.13 [-6.34, -1.92]

Table 5. Pooled gap contrasts: Jev 1.14.0 minus each comparator, in percentage points. The second line gives approximate simultaneous 95% bands accounting for all 69 dialect-specific and pooled contrasts across four datasets. Positive values mean a less negative gap for Jev. Gemma is Gemma-4-31B; Jais is Jais-2-8B. Bands spanning zero do not establish equivalence.

DialectComparatorContrastSimultaneous band
Belebele
EgyptianJev 1.13.0-0.44[-2.22, +1.33]
EgyptianGemma-4-31B-0.33[-3.13, +2.47]
EgyptianJais-2-8B+4.22[-0.32, +8.76]
NajdiJev 1.13.0+0.33[-1.61, +2.28]
NajdiGemma-4-31B-1.11[-4.05, +1.82]
NajdiJais-2-8B+2.00[-2.42, +6.42]
North LevantineJev 1.13.0+1.67[-0.96, +4.29]
North LevantineGemma-4-31B-0.44[-3.57, +2.68]
North LevantineJais-2-8B+1.22[-3.91, +6.35]
MoroccanJev 1.13.0-0.44[-3.00, +2.12]
MoroccanGemma-4-31B-1.67[-5.50, +2.17]
MoroccanJais-2-8B+4.11[-1.32, +9.55]
MesopotamianJev 1.13.0+1.00[-1.64, +3.64]
MesopotamianGemma-4-31B-0.44[-3.66, +2.77]
MesopotamianJais-2-8B+4.56[-0.90, +10.01]
SIB-200
EgyptianJev 1.13.0+0.90[-1.06, +2.85]
EgyptianGemma-4-31B+0.50[-1.96, +2.96]
EgyptianJais-2-8B+0.90[-2.57, +4.37]
NajdiJev 1.13.0+0.20[-0.29, +0.69]
NajdiGemma-4-31B+0.20[-0.65, +1.05]
NajdiJais-2-8B+0.50[-0.87, +1.87]
North LevantineJev 1.13.0+0.70[-1.50, +2.89]
North LevantineGemma-4-31B+1.29[-1.16, +3.75]
North LevantineJais-2-8B+1.29[-2.52, +5.11]
MoroccanJev 1.13.0+0.10[-2.51, +2.71]
MoroccanGemma-4-31B+0.60[-2.72, +3.92]
MoroccanJais-2-8B+0.60[-3.18, +4.38]
MesopotamianJev 1.13.0+0.40[-1.40, +2.19]
MesopotamianGemma-4-31B+0.60[-1.32, +2.52]
MesopotamianJais-2-8B+1.59[-1.18, +4.37]
TunisianJev 1.13.0-0.90[-3.22, +1.43]
TunisianGemma-4-31B-0.50[-3.42, +2.42]
TunisianJais-2-8B+0.60[-2.83, +4.03]
South LevantineJev 1.13.0-0.20[-2.48, +2.08]
South LevantineGemma-4-31B0.00[-2.52, +2.52]
South LevantineJais-2-8B+0.70[-2.90, +4.29]
Ta’izzi-AdeniJev 1.13.0+0.50[-1.41, +2.40]
Ta’izzi-AdeniGemma-4-31B+0.90[-0.98, +2.78]
Ta’izzi-AdeniJais-2-8B+0.30[-2.10, +2.70]
DialectalArabicMMLU
EgyptianJev 1.13.0+0.26[-1.46, +1.97]
EgyptianGemma-4-31B-0.19[-2.48, +2.09]
EgyptianJais-2-8B-1.24[-4.17, +1.68]
SaudiJev 1.13.0+1.34[-0.32, +3.00]
SaudiGemma-4-31B+0.16[-1.99, +2.31]
DialectComparatorContrastSimultaneous band
SaudiJais-2-8B+1.79[-1.05, +4.63]
MoroccanJev 1.13.0+1.28[-0.70, +3.25]
MoroccanGemma-4-31B-0.89[-3.54, +1.75]
MoroccanJais-2-8B-1.37[-4.67, +1.93]
SyrianJev 1.13.0+1.08[-0.82, +2.98]
SyrianGemma-4-31B-0.38[-3.05, +2.28]
SyrianJais-2-8B+0.83[-2.39, +4.05]
EmiratiJev 1.13.0+0.80[-0.83, +2.43]
EmiratiGemma-4-31B+0.38[-1.87, +2.63]
EmiratiJais-2-8B+1.05[-1.71, +3.81]
ArBanking77 (Palestinian)
PalestinianJev 1.13.0+0.55[-1.06, +2.17]
PalestinianGemma-4-31B-3.55[-5.60, -1.50]
PalestinianJais-2-8B-4.13[-6.34, -1.92]

Table 13. Jev 1.14.0 minus comparator dialect–MSA gap, by dialect. Bands account for the complete 69-contrast grid. Positive values mean a less negative gap for Jev.

Version changes and identical-input sensitivity. Jev 1.14.0’s pooled gap is less negative than Jev 1.13.0’s in each dataset, but none of the version contrasts is separated from zero by the simultaneous bands. Moreover, a smaller gap need not mean better dialect accuracy. On DialectalArabicMMLU, dialect accuracy is nearly unchanged (81.10% to 81.12%), while MSA accuracy falls from 85.68% to 84.75%. Removing pairs whose complete model inputs are identical across language versions leaves Jev’s pooled gap on SIB-200 small, at -0.39 points on 7,237 pairs; its pointwise interval still includes zero. Table 14 reports this sensitivity for all four datasets.

DatasetRemovedRetainedFull gapNonidentical gap [95% interval]
Belebele04,500-5.69-5.69 [-6.78, -4.58]
SIB-2007957,237-0.35-0.39 [-1.14, +0.40]
DialectalArabicMMLU12015,555-3.63-3.66 [-4.35, -2.96]
ArBanking77 (Palestinian)03,073-7.94-7.94 [-9.26, -6.59]

Table 14. Jev 1.14.0 after removal of identical full-input pairs. Gaps and pointwise 95% intervals are in percentage points; remaining pairs still retain source/article clustering. This is a selected nonidentical-input population, not an estimate of performance on all dialect text.

Answer to RQ2. On the evaluated matched resources, Jev 1.14.0’s dialect penalty depends on both dataset and variety. The point gaps are generally negative on reading comprehension, knowledge questions and Palestinian banking intents, while the topic-classification average is close to zero. The cross-system evidence does not support a general dialect advantage for Jev: it has a larger observed Palestinian banking penalty

than both selected LLMs, while the other contrasts remain inconclusive under simultaneous uncertainty. These comparisons concern the particular matched versions and interfaces; they do not isolate task type as the cause, establish native-text performance or measure linguistic distance from MSA.

Discussion and Limitations

Discussion

Jev 1.14.0 improves the mean task-primary score over Jev 1.13.0, but both the leading LLM and the leading non-Jev decision model score higher overall in this inventory (§ 5.1). Its position also changes across families: the overall increase between Jev versions includes lower knowledge and rating/grading means. The system order also survives removal of the recorded English controls and repeated MMMLU content, although task means remain conditional on coverage. For an application, the relevant task family and supported inputs therefore provide information that a single aggregate rank cannot supply. These comparisons describe the selected systems and protocol; they do not isolate a training or architecture effect.

The matched comparisons show that the language-version effect differs across datasets and varieties. Jev 1.14.0’s smaller pooled SIB-200 gap persists after identical-input pairs are removed, while larger losses appear for some varieties on Belebele and DialectalArabicMMLU. Cross-system contrasts add a practical qualification: on matched Palestinian banking intents, Jev loses more accuracy between MSA and dialect versions than either selected LLM. A smaller gap alone is insufficient for choosing a model, as Jais’s low banking accuracy shows. Likewise, the smaller gap in a newer Jev version can reflect lower MSA accuracy rather than an improvement on dialect text.

Holding source content fixed helps define the comparison, but datasets still differ in task, wording and included varieties. The present results therefore do not identify task complexity, model architecture or linguistic distance as the cause of a gap. Native text is a separate validation need. The companion supplies a prospective, unexecuted protocol for Egyptian and Gulf sentiment tweets with human-checked MSA and same-dialect paraphrases. The paraphrase condition would help distinguish MSA normalization from rewriting in general. Human review and new-condition evaluation are required before this can supply evidence; the current paper makes no empirical native-text robustness claim.

Limitations

The suite covers selected Arabic tasks and varieties rather than a representative sample of all Arabic use. Its task average gives equal weight to datasets with different primary metrics, sizes and purposes, so it summarizes the chosen inventory rather than a general level of Arabic ability. Label definitions also constrain interpretation: identical inputs can have different reference labels in the MADAR city task, as described in § 3.5. For a system that returns the same answer to identical inputs, some errors then reflect conflicting reference labels rather than distinguishable text differences.

The comparison concerns the specified interfaces and inference conditions. Open LLMs use restricted answer codes with thinking disabled; decision models use their supported native interfaces or adapters. These conditions allow common task scoring, but do not isolate architecture or training as the cause of a difference. They also do not establish how the systems would compare with task-specific prompt tuning or generated reasoning. The primary comparison fixes one task-template/readout condition and does not establish robustness to other prompts or answer representations.

Incomplete support can change the population behind a reported score. Coverage of the designated primary question does not certify completion of every auxiliary question, and an undefined metric is distinct from a missing response. Paired comparisons further restrict analysis to the items answered under both conditions. Different models, tokenizers and subset scopes can therefore contribute different items even when the nominal task is the same. Coverage and eligibility qualify the comparisons; they do not recover performance on unsupported inputs.

Shared content limits the independence of some observations. Parallel variety versions share source material, and the MMMLU Arabic and DialectalArabicMMLU overlap remains in the registered task summaries; the separate sensitivity analysis addresses that overlap descriptively. The exclusions used for the model-card comparisons cover documented overlap lists, while QARAR’s unpublished training corpus prevents a complete check of its training exposure. More generally, the present evaluation cannot establish that public benchmark material was absent from every evaluated model’s training data. We therefore do not interpret held-out evaluation splits as proof of unseen training content.

Primary-score reconstruction and the new source-aware matched contrasts address the numerical checks needed for the two descriptive questions. They do not repair the original RQ1/RQ2 hypothesis specifications or validate source annotations and historical execution settings. The fixed comparison panel represents selected systems, not every model in the inventory, and its simultaneous bands are approximate bootstrap summaries. Those boundaries are recorded in section C.4; descriptive results do not replace confirmatory tests.

Conclusion

We evaluated Jev alongside decision models and LLMs on the selected Arabic-focused frozen task suite, then compared matched MSA–dialect performance by variety and against a fixed model panel. Jev 1.14.0 improves the overall task mean over Jev 1.13.0, while its relative performance and version gains vary across families; it does not lead the aggregate comparison. Matched dialect gaps depend on dataset and variety. SIB-200 has a small pooled gap, whereas Palestinian banking data show a larger gap for Jev than for the selected LLMs under simultaneous uncertainty. The evidence supports reporting absolute performance, coverage and matched changes together. It does not establish a general dialect advantage, native-text robustness or the cause of the observed differences.

Ethics and Artifact Availability

The evaluation uses existing datasets and their supplied reference labels, including harmful-language, cultural and educational judgments. The mappings and exclusions in section A.2 define the measured decisions; agreement with these labels does not establish unbiased or safe behavior outside the evaluated conditions. We report task/system aggregates and use an authored example to explain the interface, rather than reproducing source tweets in the template examples.

Evaluation response archives are available from the study’s Zenodo record at https://doi.org/10.5281/zenodo.23954443. The appendices provide the task and system inventories, key mapping choices and reporting corrections. Companion files retain full revisions, detailed mappings and machine-readable results. Access to underlying datasets follows the source-specific terms and controls; source identification does not grant redistribution permission (section A.2). The preregistration deposit is identified by https://doi.org/10.5281/zenodo.23586552; its deposited access record specifies an embargo. Dataset texts and the full response archive are not bundled with this manuscript, and we make no claim that the preregistration deposit’s files are publicly downloadable.

Acknowledgements

This research has been partially funded by the Federal Ministry of Education and Research of Germany and the state of North-Rhine Westphalia as part of the Lamarr-Institute for Machine Learning and Artificial Intelligence.

Benchmark and Evaluation Details

Tasks and Sources

Table 6 identifies all tasks, their question types, primary metrics, frozen sizes and source resources. The primary metric selects one decision or combines the required binary heads; auxiliary questions do not enter the overall mean. Complete parser and sampling descriptions are supplied in the companion file task_mappings.md.

Table 6: Task inventory and source attribution. Types are Choice (C), Noul (N) and Score (S); trailing digits count questions. Items gives the frozen task size, not independent texts or completed answers. Full mappings and revisions are in the companion files.

FamilyTaskTypesMetricItemsSource
DOSACT 2020 offensiveN, CMacro-F12,000[30]
DArabic spam and ham tweetsNF1300[31, 32]
DArCyC cyberbullyingN2F1300[33]
DAraTox toxicityN6Jaccard300[34]
DMultilingual HateCheckNF1300[35]
DAya red-teamingN9Jaccard300[36]
EArabicMMLUCAcc.14,455[3]
EMMMLU ArabicCAcc.14,042[11]
FCheckThat! 2025 subjectivityNF1300[37]
FArMPro propagandaNF1300[38]
FCheckThat! 2022 tweetsN2F1300[39]
FArAIEval 2023 disinformationNF1300[40]
FAIRABICNF1300[41, 42]
FHalluTruthQA-4KNF1300[43]
GArabCultureCAcc.3,458[44]
GArabic ComVECAcc.1,000[45]
GArabic HumorNF1300[46]
GGlobal PIQACAcc.300[47]
HBAREC 2025 readabilitySSpearman300[10]
HAR-AES essay scoringSSpearman300[48]
HAR-ASAG short-answer gradingSSpearman300[49]
HCARESSpearman300[50]
IPoetry meter (MetRec)CMacro-F1300[51, 52]
IWord sense disambiguationCAcc.300[53, 54]
INahw-MCQ grammarCAcc.300[55]

Table 17.

Mapping Choices and Source Limits

Parallel and independently collected text. Belebele, SIB-200 and DialectalArabicMMLU retain shared source identifiers across language versions. Their frozen primary inventories also include English controls; SIB-200 combines the source train, validation and test splits without fitting a model. English and romanized-MSA controls do not enter the matched dialect comparison. ArBanking77’s matched analysis uses corpus-linked MSA and Palestinian sources, averaging available paraphrase correctness within each source and variety. Independently collected variety text, including Global PIQA, does not provide matched MSA–dialect pairs.

Primary targets. ArSarcasm-v2 is scored on sentiment even though requests also ask about sarcasm and variety. NADI uses country-level macro-F1; MADAR uses city accuracy; MASSIVE uses intent accuracy. MIRACL becomes a query–passage relevance decision, rather than a retrieval-ranking task. Egyptian harmful-language evaluation uses category macro-F1, with offensive text included in the auxiliary harmfulness label. AraTox scores six toxicity types, excluding the complementary non-toxic label. The CheckThat! 2025 adaptation uses positive-class F1 rather than the source task’s macro-F1, so its numbers are not directly comparable with that leaderboard[37].

Ordered scores. BAREC uses seven levels, mapped to indices 0–6; its nineteen-level scheme exceeds Jev’s supported Score range. LABR uses star rating minus one. AR-AES retains rubric maxima of five and averages the available final rater scores. AR-ASAG uses marks on levels 0–5. CARE maps the supplied 1–10 ratings to indices 0–9. Spearman correlation evaluates rank association with these supplied scores, not exact grade agreement or deployment quality.

Sampling and development. Added tasks use the deterministic, stratified sampling and duplicate/conflict filtering described in section 3.4; original tasks retain their own rules. NADI uses its source development split for evaluation. MASSIVE uses Arabic release 1.1; its publication describes 1.0, and the maintainers report that the other language data were unchanged when Catalan was added1. Missing auxiliary gold labels, such as emotion intensity, differ from missing required primary answers. The companion mapping descriptions record each exception and development/pilot rule.

Revisions and permissions. task_sources.csv records the complete declared source locators and revisions. These identify intended inputs rather than certifying every source annotation, training exclusion or redistribution right. Historical terms are distinguished from current maintainer labels. In particular, Arabic ComVE has a CC0 licence file and a CC BY-SA statement in its README2; the companion records retain this conflict. Neither dataset texts nor full model responses are bundled with the paper.

Systems and Primary Readouts

Table 7 lists every evaluated version. The companion systems.csv gives full repository identifiers, pinned revisions, declared precision and adapter settings; deferred_systems.csv identifies the four excluded systems. Declared precision and service labels do not independently establish historical execution settings or hosted weights. The analysis environment and available run provenance are recorded separately in reproducibility.md.

SystemDeclared precisionReadout
Hosted decision models
jev-1.13.0Not recordedAPI
jev-1.14.0Not recordedAPI
Microsoft-Decision-1Not recordedAPI
OpenAI-Decisions-gpt-6-lunaNot recordedAPI
Arabic decision models
QARARfloat32NH
jebfloat32NH
Jev-ARfloat32NH
jeb-typed-decisionsfloat32NH
ara-jev-Faisalbfloat16LP
Other general decision models
Laya-multilingualfloat16NH
Kev-4Bbfloat16NH
Kev-27Bbfloat16NH
SII-Decision-4Bbfloat16NH
SII-Decision-27Bbfloat16NH
Arabic-centric LLMs
Jais-family-256Mfloat32TC
Jais-family-590Mfloat32TC
Jais-family-1.3Bfloat32TC
Jais-family-2.7Bfloat32TC
Jais-family-6.7Bfloat32TC
Jais-family-13Bfloat32TC
Jais-2-8Bbfloat16TC
ALLaM-7Bbfloat16TC
Fanar-2-27Bbfloat16TC
Qwen LLMs
Qwen3.5-0.8Bbfloat16TC
Qwen3.5-2Bbfloat16TC

Table 7. Evaluated systems and declared primary readouts. API: hosted interface; TC: restricted token codes; NH: native heads; OI: own interface; LP: letter codes with its own prompt. Full repository identifiers, revisions and declared precision are in systems.csv.

SystemDeclared precisionReadout
Qwen3.5-4Bbfloat16TC
Qwen3.5-9Bbfloat16TC
Qwen3.5-27Bbfloat16TC
Qwen3.8-27Bbfloat16TC
Gemma LLMs
Gemma-4-E2Bbfloat16TC
Gemma-4-E4Bbfloat16TC
Gemma-4-12Bbfloat16TC
Gemma-4-26B-A4Bbfloat16TC
Gemma-4-31Bbfloat16TC
Additional vendor decision models
pplx-decider-v1.2bfloat16OI
pplx-decider-flash-v1.2bfloat16OI
GLiDE-1.2bfloat16OI
GLiNER2.5-Decidefloat32OI
GLiNER2.5-Decide-1Bfloat32OI
GLiNER2.5-multi-Decidefloat32OI
clefbfloat16OI
clef-flashbfloat16OI
h2o-lightning-31bbfloat16OI
h2o-lightning-4b-nemotron-nanobfloat16OI
Intern-Decision-4Bbfloat16OI
Intern-Decision-2Bbfloat16OI
Intern-Decision-0.8Bbfloat16OI
d1-3Bbfloat16OI
d1-omni-600Mfloat32OI
tasksource-decider-nanobfloat16OI
strands-decider-2B-qwen3.5bfloat16OI
strands-decider-E2B-gemma4bfloat16OI
strands-decider-E4B-gemma4bfloat16OI
strands-decider-12B-gemma4bfloat16OI
strands-decider-26B-A4B-gemma4bfloat16OI
d3bfloat16OI
d3-flashbfloat16OI
d3-minibfloat16OI
d3-nanobfloat16OI
d3-litebfloat16OI
d3-edgebfloat16OI
Vela-2.0-0.3Bfloat32OI
Vela-2.0-0.8Bbfloat16OI
Vela-2.0-4Bbfloat16OI
Vela-2.0-9Bbfloat16OI
Layafloat16NH
Laya-typed-decisionsfloat16NH

Table 18.

Per-Request Timing and Processing Rates

We match saved response timings to the exact frozen T-EN requests used in the primary audit, retaining one response record per unique request. A record contains a shared state and its question set; it can include unsupported or skipped questions, so primary coverage remains a separate result. The companion request_timing.csv reports mean seconds per record and Records/s beside each system’s evaluation score. Records/s equals the number of timed records divided by the sum of their recorded processing times, or equivalently the reciprocal of mean seconds per record. Timing is averaged over distinct records, while evaluation scores remain unweighted means over available tasks.

Jev and other decision backends time each request or prediction call, including any network, retry and adapter work. Local LLMs allocate batch processing time across records. Model startup, dataset loading and output-file writes are excluded. The resulting rates describe each backend’s recorded execution settings; hardware, batching and concurrency differ across systems. The companion retains the full timing definitions, quantiles, counts and source hashes.

Frozen Primary Wording

The T-EN source files match the recorded frozen hashes. Table 8 shows an actual task question without an evaluation tweet or model prediction. Task wording is separate from the LLM wrapper, which begins “Answer the question about the following state.” and adds answer-format instructions. The default readout normalizes the probabilities of supported single-token answer codes; context limits and unsupported representations affect coverage. The companion templates_primary.json retains the exact primary source strings, not each backend’s final chat serialization.

TemplateFrozen instruction and criterion descriptions
T-ENInstruction: What is the overall sentiment of ‘tweet‘?<br>negative: The text expresses a negative opinion.<br>neutral: The text expresses neither a positive nor a negative opinion.<br>positive: The text expresses a positive opinion.

Table 8. Actual frozen ArSarcasm-v2 sentiment question under the primary T-EN template. Internal label keys are preserved; no tweet, predicted answer or full model-specific chat serialization is shown. Exact source strings for the complete main suite are retained in the accompanying local template bundle.

Additional Descriptive Results

All-System Summaries and Exclusion Sensitivities

Table 9 reports every system’s frozen evaluation score and recorded processing rate. The companion sensitivities.csv retains all three descriptive masks. The Arabic-only mask removes 900 English items from Belebele, 1,004 from SIB-200 and 3,135 from DialectalArabicMMLU, while retaining romanized MSA and T-EN instructions. The overlap mask removes 2,391 MMMLU items whose state/question requests also occur in DialectalArabicMMLU; it removes them only from MMMLU. Each affected task metric is recomputed before unweighted task averaging. The combined mask applies both exclusions. All 67 systems retain their ordering; maximum absolute mean changes are below 0.332 and 0.021 aggregate points for the separate masks, respectively.

The full unrounded system–task matrix, item counts and coverage are supplied as primary_scores.csv. Its original export is retained separately as primary_scores_original.csv, with the single reporting correction documented below. These machine-readable tables replace an exhaustive printed score matrix.

SystemScoreRecords/sTasksCoverage (%)
Jais-family-1.3B26.7111.7743/4399.60
d1-omni-600M25.3739.4443/43100.00
tasksource-decider-nano22.9946.6837/4386.05
Laya-typed-decisions22.0434.5543/43100.00
Laya21.7635.0943/43100.00

Table 19.

Focal-System Task-Family Means

The exact focal-system means complement the main heatmap. Comparator selection uses overall means rather than selecting a different leading system within each family.

Common Task Support

The intersection of finite primary scores for all twelve fixed summary systems contains 40 tasks, excluding ArBanking77, MADAR and MASSIVE. Table 11 uses the current corrected primary copy. This restriction changes the task mix rather than ensuring identical answered items. Two close comparisons reverse: Jev 1.13.0 versus OpenAI Decisions, and Qwen3.5-27B versus Microsoft Decision-1.

SystemCorrected meanCorrected shared-task mean
Gemma-4-31B71.8172.92
h2o-lightning-31b71.1271.93
jev-1.14.069.2470.69
OpenAI-Decisions-gpt-6-luna68.7969.93
jev-1.13.068.7770.21
SII-Decision-27B68.3969.71
Microsoft-Decision-166.0667.04
Qwen3.5-27B65.9667.34
Qwen3.8-27B65.1566.37
ara-jev-Faisal59.9359.93
Jais-2-8B54.5457.76
Jais-family-13B46.8449.54

Table 11. Available-task and common-task means after the completeness correction, multiplied by 100. The common column retains the same 40 tasks; item support can still differ within them.

Matched Dialect Panel and Uncertainty

The expanded panel uses the same eligible item UIDs across both Jev versions, Gemma-4-31B and Jais-2-8B. Full state/question hashes and source/gold metadata agree across systems. All primary responses are present for the 40,618 eligible rows before source matching. Only sources with both MSA and a given dialect enter that comparison; unmatched sources are counted in the companion support report. The resulting 31,280 pairs reproduce every original Jev 1.14.0 matched correctness value. All 16 full-task primary score/count/coverage cells also reproduce, and the response-source hashes match the earlier primary audit. Banking source means retain identical paraphrase sets across systems; these means need not be binary, so an exact McNemar test is not applied to them.

For each dataset, whole-cluster multinomial resampling generates 10,000 draws with NumPy’s default PCG64 generator. Seeds are 12900 for Belebele, 12901 for SIB-200, 12902 for DialectalArabicMMLU and 12903 for ArBanking77. Each draw uses one shared cluster multiplicity vector for all models and varieties. Source–variety pairs have equal weight, so resampled clusters contribute all their pairs and the denominator is recomputed on every draw. MSA observations recur against several varieties and remain linked. The 2.5th and 97.5th percentiles give the pointwise intervals.

The simultaneous grid contains three Jev 1.14.0-minus-comparator gap contrasts for every dialect and the pooled row of each dataset, for 69 reported contrasts in total. The Palestinian-only banking dialect and pooled rows are identical and are retained for consistent reporting. For each contrast, the bootstrap standard deviation supplies a fixed scale. Within each draw we take the largest absolute centered, scaled deviation over that dataset’s entire contrast grid. The 98.75th-percentile critical value gives a two-sided band centered on the observed contrast. Allocating error probability 0.05/4 to each dataset yields approximate 95% simultaneous coverage across the grid by the union bound, even though Belebele and SIB-200 share FLORES content. These are conservative, bootstrap-based descriptive bands, not adjusted outcomes of the original hypothesis tests. The companion retains pointwise contrast intervals as well as the simultaneous bands.

Table 12: Per-dialect matched accuracy and pointwise 95% intervals. Gaps are dialect minus MSA in percentage points; all four models use identical items.

DialectSystemPairsMSA (%)Dialect (%)Gap [95% interval]
PalestinianJais-2-8B3,07320.3716.56-3.81 [-4.73, -2.88]

Table 20.

Identical-Input Sensitivity

The exploratory sensitivity excludes a source–variety pair when the sets of complete model-independent state/question hashes are identical on its MSA and dialect sides. It does not exclude a pair merely for sharing a passage or some answer options. No pairs are removed from Belebele or Palestinian ArBanking77; SIB-200 and DialectalArabicMMLU contain some identical inputs. The same mask applies to all four systems. Table 14 shows Jev 1.14.0, and the companion supplies every system’s restricted summaries. Seeds 13900–13903 and 10,000 whole-cluster draws produce separate pointwise intervals. No simultaneous or equivalence claim is based on this sensitivity.

Reporting Corrections and Reproducibility

Incomplete Primary Heads

Two OpenAI Decisions AraTox responses explicitly refused one required binary head each. The original pivot counted the incomplete items and converted missing gold to true. We reproduced that arithmetic, then applied the complete-primary-item rule used for the main reporting: 298 items remain. Table 15 shows the effect on score and coverage. Exactly one of 2,881 system–task cells changes; all other cells and original hypothesis-test outputs are preserved. The increase reflects a changed denominator, not model improvement.

ReportingAraTox itemsCov. (%)AraTox meanOverall mean
Original saved export300100.0061.6568.78
Complete primary items29899.3362.0068.79

Table 15. Original and corrected OpenAI Decisions reporting. Scores are multiplied by 100. Original coverage is the saved export’s value; the corrected row requires all primary heads.

Source-Aware Dialect Intervals

The four matched populations were reconstructed from pinned sources and existing Jev 1.14.0 responses. Request keys, required answers, per-variety means and pair counts match the saved summaries. Belebele and SIB-200 sources map to shared FLORES articles; ArBanking77 averages available paraphrases within source and variety before pairing.

The original Belebele/SIB-200 intervals grouped by question-source ID rather than article. The first descriptive correction resampled whole articles for those datasets and whole source identifiers for the other two, using 2,000 draws initialized with seed 0. Table 16 preserves those historical endpoints and their correction; point estimates were unchanged. The expanded panel in the main text now uses the separate 10,000-draw procedure in section B.4, so its Monte Carlo endpoints differ slightly. Neither procedure repairs the separate historical H2d comparison.

DatasetOriginal saved intervalRecomputed interval
Belebele[-6.73, -4.56][-6.82, -4.63]
SIB-200[-1.02, +0.35][-1.02, +0.36]
DialectalArabicMMLU[-4.29, -2.93][-4.35, -2.93]
ArBanking77 (Palestinian)[-9.27, -6.61][-9.27, -6.61]

Table 16. Historical original and first corrected Jev 1.14.0 interval endpoints, in percentage points. The expanded main-text panel uses more draws and different fixed seeds; matched populations and point estimates agree.

Numerical Validation

All 43 frozen task counts and primary gold domains were reconstructed from recorded raw files and pinned cached datasets. Exact model/state/question keys linked inputs to existing responses, with duplicate-response and required-head checks. Recomputed primary scores, counts and coverage agree for all 2,881 cells: 2,842 finite values and 39 cells with no observed primary items. The maximum finite difference was at numerical rounding precision. This made no model calls or changes to the original databases. The companion validation report preserves the checks and hashes; intermediate cache-recovery logs are kept outside the paper. The audit does not certify source annotations, historical backend settings or original hypothesis-test inference.

Scope and Original Study Records

The two paper questions are descriptive and were selected after evaluation from the broader preregistered study. The added per-dialect model panel and its uncertainty procedure were also specified after the original evaluation. This scope does not change the original hypotheses, populations or correction families. The companion legacy_rq1_rq2.json retains the original test outputs, including null and contrary outcomes, and legacy_analysis_status.md states their qualifications. Original RQ1 resampling, H1d contrast/correction rules and some original RQ2 inferential checks remain unresolved. The historical H2d effect remains withheld because its original join omitted task identity. The new task-aware descriptive panel uses independently reconstructed pairs and a separately stated comparator/uncertainty rule; it does not validate or replace that historical effect or its correction family. The deposited preregistration and original outputs are unchanged.

References

  1. [1]T. Deußer, R. Ramamurthy, L. Sparrenberg, and R. Sifa, “Evaluating and benchmarking the system one model Jev,” arXiv:2609.37647v2, 2026, version 2, 29 October 2026.
  2. [2]M. Abdul-Mageed, A. Elmadany, and E. M. B. Nagoudi, “ARBERT & MARBERT: Deep bidirectional transformers for Arabic,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 2021, pp. 7088–7105.
  3. [3]F. Koto, H. Li, S. Shatnawi, J. Doughman, A. Sadallah, A. Alraeesi, K. Almubarak, Z. Alyafeai, N. Sengupta, S. Shehata et al., “ArabicMMLU: Assessing massive multitask language understanding in arabic,” in Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 2024, pp. 5622–5640.arxiv.org/abs/2402.12840
  4. [4]M. H. Altakror, N. Habash, T. Lynn, Y. Samih, A. A. Freihat, K. Chirkunov, M. AbuOdeh, R. Florian, P. Nakov, and A. F. Aji, “DialectalArabicMMLU: Benchmarking dialectal capabilities in arabic and multilingual language models,” in Proceedings of the Fifteenth Language Resources and Evaluation Conference. ELRA Language Resource Association, 2026, pp. 3199–3219.
  5. [5]W. Yin, J. Hay, and D. Roth, “Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, 2019, pp. 3914–3923.arxiv.org/abs/1909.00161
  6. [6]E. Almazrouei, R. Cojocaru, M. Baldo, Q. Malartic, H. Alobeidli, D. Mazzotta, G. Penedo, G. Campesan, M. Farooq, M. Alhammadi et al., “AlGhafа evaluation benchmark for Arabic language models,” in Proceedings of ArabicNLP 2023. Association for Computational Linguistics, 2023, pp. 244–275.DOI
  7. [7]M. Abdul-Mageed, C. Zhang, A. Elmadany, H. Bouamor, and N. Habash, “NADI 2021: The second nuanced Arabic dialect identification shared task,” in Proceedings of the Sixth Arabic Natural Language Processing Workshop. Association for Computational Linguistics, 2021, pp. 244–259.
  8. [8]L. Bandarkar, D. Liang, B. Muller, M. Artetxe, S. N. Shukla, D. Husa, N. Goyal, A. Krishnan, L. Zettlemoyer, and M. Khabsa, “The Bebelele benchmark: a parallel reading comprehension dataset in 122 language variants,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2024, pp. 749–775.
  9. [9]I. Abu Farha, W. Zaghouani, and W. Magdy, “Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic,” in Proceedings of the Sixth Arabic Natural Language Processing Workshop. Association for Computational Linguistics, 2021, pp. 296–305.
  10. [10]CAMeL Lab, “BAREC Shared Task 2025: Sentence-level dataset,” Hugging Face, n.d., official dataset card, accessed 4 November 2026.
  11. [11]OpenAI, “MMMLU,” Hugging Face, n.d., official dataset card, accessed 4 November 2026.
  12. [12]H. Bouamor, N. Habash, M. Salameh, W. Zaghouani, O. Rambow, D. Abdulrahim, O. Obeid, S. Khalifa, F. Eryani, A. Erdmann et al., “The MADAR Arabic dialect corpus and lexicon,” in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), 2018.DOI
  13. [13]scikit-learn developers, “F1 score: Reference documentation,” Online documentation, n.d., accessed 4 November 2026.
  14. [14]——, “Jaccard similarity: Reference documentation,” Online documentation, n.d., accessed 4 November 2026.
  15. [15]SciPy developers, “Spearman rank correlation: Reference documentation,” Online documentation, n.d., accessed 4 November 2026.
  16. [16]D. I. Adelani, H. Liu, X. Shen, N. Vassilyev, J. O. Alabi, Y. Mao, H. Gao, and E.-S. A. Lee, “SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2024, pp. 226–245.
  17. [17]C. Fourati, A. Messaoudi, and H. Haddad, “TUNIZI: A tunisian Arabizi sentiment analysis dataset,” GitHub, 2020, author-maintained dataset repository and citation metadata.arxiv.org/abs/2004.14303
  18. [18]N. S. Alturayeif, H. A. Luqman, and M. A. K. Ahmed, “Mawqif: A multi-label Arabic dataset for target-specific stance detection,” in Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP). Association for Computational Linguistics, 2022, pp. 174–184.DOI
  19. [19]M. Aly and A. Atiya, “LABR: A large scale Arabic book reviews dataset,” in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 2013, pp. 494–498.
  20. [20]S. Mohammad, F. Bravo-Marquez, M. Salameh, and S. Kiritchenko, “SemEval-2018 task 1: Affect in tweets,” in Proceedings of the 12th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 2018, pp. 1–17.DOI
  21. [21]J. FitzGerald, C. Hench, C. Peris, S. Mackie, K. Rottmann, A. Sanchez, A. Nash, L. Urbach, V. Kakarala, R. Singh et al., “MASSIVE: A 1M-example multilingual natural language understanding dataset with 51 typologically-diverse languages,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2023, pp. 4277–4302.arxiv.org/abs/2204.08582
  22. [22]M. Pontiki, D. Galanis, H. Papageorgiou, I. Androutsopoulos, S. Manandhar, M. Al-SMadi, M. Al-Ayyoub, Y. Zhao, B. Qin, O. De Clercq et al., “SemEval-2016 task 5: Aspect based sentiment analysis,” in Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). Association for Computational Linguistics, 2016, pp. 19–30.DOI
  23. [23]M. Jarrar, A. Birim, M. Khalilia, M. Erden, and S. Ghanem, “ArBanking77: Intent detection neural model and a new dataset in modern and dialectical Arabic,” in Proceedings of ArabicNLP 2023. Association for Computational Linguistics, 2023, pp. 276–287.arxiv.org/abs/2310.19034
  24. [24]S. Malaysha, M. El-Haj, S. Ezzini, M. Khalilia, M. Jarrar, S. Almujaiwel, I. Berrada, and H. Bouamor, “AraFinNLP 2024: The first Arabic financial NLP shared task,” in Proceedings of the Second Arabic Natural Language Processing Conference. Association for Computational Linguistics, 2024, pp. 393–402.
  25. [25]A. Conneau, R. Rinott, G. Lample, A. Williams, S. R. Bowman, H. Schwenk, and V. Stoyanov, “XNLI: Evaluating cross-lingual sentence representations,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018, pp. 2475–2485.arxiv.org/abs/1809.05053
  26. [26]X. Zhang, N. Thakur, O. Ogundepo, E. Kamalloo, D. Alfonso-Hermelo, X. Li, Q. Liu, M. Reza-gholizadeh, and J. Lin, “MIRACL: A multilingual retrieval dataset covering 18 diverse languages,” Transactions of the Association for Computational Linguistics, vol. 11, pp. 1114–1131, 2023.DOI
  27. [27]D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia, “SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation,” in Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, 2017, pp. 1–14.DOI
  28. [28]I. Ahmed, M. Abbas, R. Hatem, A. Ihab, and M. W. Fakhr, “Fine-tuning Arabic pre-trained transformer models for Egyptian-Arabic dialect offensive language and hate speech detection and classification,” in 2022 20th International Conference on Language Engineering (ESOLEC), vol. 20. IEEE, 2022, pp. 170–174.
  29. [29]E. A. Alghamdi, R. I. Masoud, D. Alnuhait, A. Y. Alomairi, A. Ashraf, and M. Zaytoon, “AraTrust: An evaluation of trustworthiness for LLMs in Arabic,” arXiv:2403.09017, 2024.arxiv.org/abs/2403.09017
  30. [30]H. Mubarak, K. Darwish, W. Magdy, T. Elsayed, and H. Al-Khalifa, “Overview of OSACT4 Arabic offensive language detection shared task,” in Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. European Language Resources Association, 2020, pp. 48–52.
  31. [31]S. Kaddoura and S. Henno, “Dataset of Arabic spam and ham tweets,” 2024, version 4.
  32. [32]——, “Dataset of Arabic spam and ham tweets,” Data in Brief, vol. 52, p. 109904, 2024.
  33. [33]F. Shannag, “ArCyC: A fully annotated Arabic cyberbullying corpus,” 2023, version 1.DOI
  34. [34]F. Alsharqi, A. Abulohoom, S. Yagi, F. Jabr, L. Lulu, and A. Elnagar, “Aratox: a multi-dialect, multi-label Arabic dataset and model benchmark for toxicity detection,” Language Resources and Evaluation, vol. 60, p. 39, 2026, article 39; publisher records a correction and an update to Arabic word rendering.
  35. [35]P. Röttger, H. Seelawi, D. Nozza, Z. Talat, and B. Vidgen, “Multilingual HateCheck: Functional tests for multilingual hate speech detection models,” in Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH). Association for Computational Linguistics, 2022, pp. 154–169.
  36. [36]Cohere Labs, “Aya red-teaming: Dataset card,” Hugging Face, n.d., maintainer resource record, accessed 4 November 2026.
  37. [37]F. Ruggeri, A. Muti, K. Korre, J. M. Struß, M. Siegel, M. Wiegand, F. Alam, M. R. Biswas, W. Zaghouani, M. Nawrocka et al., “Overview of the CLEF-2025 CheckThat! lab task 1 on subjectivity in news articles,” in Working Notes of CLEF 2025, ser. CEUR Workshop Proceedings, vol. 4038. CEUR-WS, 2025, pp. 681–694.
  38. [38]M. Hasanain, F. Ahmad, and F. Alam, “Can GPT-4 identify propaganda? annotation and detection of propaganda spans in news articles,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). ELRA and ICCL, 2024, pp. 2724–2744.arxiv.org/abs/2402.17478
  39. [39]P. Nakov, A. Barrón-Cedeño, G. Da San Martino, F. Alam, R. Míguez, T. Caselli, M. Kutlu, W. Zaghouani, C. Li, S. Shaar et al., “Overview of the CLEF-2022 CheckThat! lab task 1 on identifying relevant claims in tweets,” in Working Notes of CLEF 2022, ser. CEUR Workshop Proceedings, vol. 3180. CEUR-WS, 2022, pp. 368–392.
  40. [40]M. Hasanain, F. Alam, H. Mubarak, S. Abd aljalil, W. Zaghouani, P. Nakov, G. Da San Martino, and A. A. Freihat, “ArAIEval shared task: Persuasion techniques and disinformation detection in Arabic text,” in Proceedings of ArabicNLP 2023. Association for Computational Linguistics, 2023, pp. 483–493.
  41. [41]Hamed1Hamed, “AIRABIC: Arabic dataset for performance evaluation of AI detectors,” Creator repository, n.d., creator repository attribution; accessed 2 November 2026.
  42. [42]H. Alshammari and A. El-Sayed, “AIRABIC: Arabic dataset for performance evaluation of AI detectors,” in 2023 International Conference on Machine Learning and Applications (ICMLA). IEEE, 2023, pp. 864–870.DOI
  43. [43]Bekhouche, “HalluTruthQA-4k: Dataset resource record,” Hugging Face, n.d., maintainer account attribution; resource record accessed 4 November 2026.
  44. [44]A. Sadallah, J. C. Tonga, K. Almubarak, S. Almheiri, F. Atif, C. Qwaider, K. Kadaoui, S. Shatnawi, Y. Alesh, and F. Koto, “Commonsense reasoning in Arab culture,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2025, pp. 7695–7710.arxiv.org/abs/2502.12788
  45. [45]S. Tawalbeh and M. AL-Smadi, “Is this sentence valid? an Arabic dataset for commonsense validation,” arXiv:2008.10873, 2020.arxiv.org/abs/2008.10873
  46. [46]H. Alkhalifa, F. Alzahrani, H. Qawara, R. Alrowais, S. Alowa, and L. Aldhubayi, “A dataset for detecting humor in Arabic text,” in Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022). Association for Computational Linguistics, 2022, pp. 219–225.
  47. [47]MRL benchmarks, “Global PIQA non-parallel: Dataset card,” Hugging Face, n.d., maintainer resource record, accessed 4 November 2026.
  48. [48]R. Ghazawi and E. Simpson, “Automated essay scoring in Arabic: a dataset and analysis of a BERT-based system,” arXiv:2407.11212, 2024.arxiv.org/abs/2407.11212
  49. [49]L. Ouahrani and D. Bennouar, “AR-ASAG-dataset,” 2020, version 1.DOI
  50. [50]G. Guo, T. Naous, H. Wakaki, Y. Nishimura, Y. Mitsufuji, A. Ritter, and W. Xu, “CARE: Multilingual human preference learning for cultural awareness,” arXiv:2504.05154, 2025.arxiv.org/abs/2504.05154
  51. [51]M. S. Al-shaibani, Z. Alyafeai, and I. Ahmad, “MetRec: A dataset for meter classification of arabic poetry,” Data in Brief, vol. 33, p. 106497, 2020.DOI
  52. [52]——, “Meter classification of Arabic poems using deep bidirectional recurrent neural networks,” Pattern Recognition Letters, vol. 136, pp. 1–7, 2020.DOI
  53. [53]S. Kaddoura and R. Nassar, “Dataset for Arabic word sense disambiguation,” 2024, version 2.DOI
  54. [54]——, “A comprehensive dataset for Arabic word sense disambiguation,” Data in Brief, vol. 55, p. 110591, 2024.DOI
  55. [55]H. Mubarak, M. Hawasly, and A. Mohamed, “Nahw: A comprehensive benchmark of Arabic grammar understanding, error detection, correction, and explanation,” in Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2026, pp. 6310–6328.

Paper details

vlnc:W_2kh2u3me
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents