WorksComputer Science
AQL-Lite: Enabling Efficient Knowledge Acquisition for Arabic LLMs via Data-Centric Continual Pretraining
Abstract
Training Arabic large language models faces challenges from noisy web data, dialectal variation, and the high cost of continual pretraining (CPT). We introduce AQL-Lite (AQL: Arabic Quality Language), a time- and resource-efficient framework for adapting a multilingual LLM to Modern Standard Arabic (MSA). AQL-Lite couples a quality-first curation pipeline (structural cleaning, heuristic and model-based filtering, MSA selection, and deduplication) with a partial-parameter CPT recipe that trains RS-LoRA adapters across the network together with the last eight transformer blocks and the embedding matrix, under layer-wise learning rates. At a matched 1B-token budget, CPT on AQL-curated data raises a four-benchmark Arabic mean by 3.51 points over CPT on raw web text, which yields no gain over the base model. In a controlled 5B-token comparison, AQL-Lite recovers 94% of the gain of full-parameter CPT (mean 73.19 vs. 73.58) in 52% of the wall-clock time and outperforms adapter-only baselines on every benchmark, and a component ablation shows that adapters, unfrozen blocks, and embedding training each contribute. Scaled to 40B MSA tokens and instruction-tuned on one million examples, the resulting 12B model scores 75.3 on the seven-benchmark Open Arabic LLM Leaderboard v2 (OALL v2) suite. These results suggest that partial-parameter CPT on a carefully curated corpus is a practical route to efficient MSA adaptation.
Introduction
Large language models (LLMs) have significantly advanced natural language processing, yet Arabic remains comparatively under-served despite its wide use and linguistic diversity. This gap is driven not only by model architecture, but also by two practical bottlenecks: data quality and training cost. Arabic web corpora contain large amounts of noisy, duplicated, templated, and low-signal text, while also reflecting variation across Modern Standard Arabic (MSA), dialects, code-switching, and inconsistent orthography. As a result, scaling raw Arabic tokens alone is insufficient for building strong Arabic LLMs.
Recent Arabic LLMs such as Jais and ALLaM show that strong Arabic performance requires careful preprocessing, filtering, deduplication, and distribution control [6] (Sengupta et al. 2023). At the same time, continual pretraining (CPT) remains costly, especially when updating all model parameters. Parameter-efficient methods such as LoRA reduce this cost (Hu et al., 2022), but adapter-only CPT may under-update the representations that a new language distribution requires [7] (Pezeshkpour and Hruschka 2025). This motivates our central question: can we approximate the benefits of full Arabic CPT while substantially reducing training time and compute?
We introduce AQL-Lite (AQL: Arabic Quality Language), a data-centric framework for efficiently adapting a strong multilingual checkpoint to MSA. AQL-Lite first builds a high-quality corpus through a five-stage pipeline: structural cleaning, heuristic filtering, MSA/dialect control, model-based quality and safety scoring, and exact/near deduplication. It then performs partial-parameter CPT: RS-LoRA adapters on all linear layers, full-rank updates to the last eight transformer blocks and the embedding matrix, and layer-wise learning rates. This design sits between adapter-only CPT, which is efficient but limited, and full-parameter CPT, which is effective but computationally expensive.
We evaluate AQL-Lite in three controlled settings. First, at a matched 1B-token budget, we vary only the training data (raw, heuristically filtered, or fully curated) to measure the effect of curation. Second, at 5B tokens, we compare the recipe against full CPT, LoRA-only, and RS-LoRA on identical data and hardware, and ablate its components. Third, we scale the recipe to 40B MSA tokens, instruction-tune the result, and evaluate the 12B model before and after instruction tuning on the seven-benchmark OALL v2 suite [12].
Contributions. Our contributions are:
Arabic quality-first data pipeline. We introduce the AQL pipeline, a five-stage MSA corpus curation pipeline combining structural cleaning, heuristic filtering, MSA/dialect control, model-based quality and safety scoring, and exact/near deduplication, and show at a matched token budget that the curated data adds 3.51 points over raw web text.
Partial-parameter CPT. We propose AQL-Lite, a CPT recipe that combines RS-LoRA, selective upper-block unfreezing, embedding training, and layer-wise learning rates to approximate full-CPT gains at lower cost, and ablate its components.
Empirical evaluation for MSA adaptation. We compare AQL-Lite against full and adapter-only CPT in a controlled 5B-token setting, then evaluate the final 12B model on the OALL v2 suite, separating the contributions of CPT and instruction tuning.
Related Work
Arabic LLMs. Recent Arabic-centric LLMs show that strong Arabic capability is feasible, but also highlight persistent bottlenecks around data curation, dialect balance, and safety. Jais emphasizes Arabic preprocessing, deduplication, and bilingual training to mitigate Arabic data scarcity (Sengupta et al., 2023). ALLaM reports strict language-ID filtering, spam/URL filtering, and exact deduplication, while noting challenges related to fuzzy deduplication, toxic content, and social bias in internet-scale corpora [6]. Broader surveys further document the rapid development of Arabic LLMs and the need for reliable Arabic evaluation practices (Mashaabi et al., 2024). Closest to our setting, AceGPT continually pretrains an English-centric base model on Arabic before localized instruction tuning (Huang et al., 2024), and SambaLingo studies how to adapt English-centric LLMs to new languages, including Arabic, through continual pretraining with vocabulary extension [11]. Unlike these works, we adapt a multilingual base model and focus on reducing the cost of the continual-pretraining stage itself.
Data quality, filtering, and deduplication. A growing body of work shows that data quality substantially improves knowledge acquisition per token, often rivaling gains from model scaling. Large-scale filtering, repetition removal, and deduplication are now standard in strong pretraining pipelines (Rae et al., 2021; Penedo et al., 2023). These findings are especially relevant for Arabic, where raw web tokens are abundant but effective tokens are reduced by boilerplate, templated pages, SEO spam, duplication, orthographic variation, and dialect/code-switching effects. For Arabic specifically, the 101 Billion Arabic Words dataset applies cleaning, normalization, and deduplication to Common Crawl [5], and FineWeb2 adapts language identification, filtering, and deduplication to more than 1,000 languages, ablating each step with small-model training runs (Penedo et al., 2025). AQL adds explicit MSA/dialect control, a learned multi-category safety scorer, and per-document audit logs to this line of work.
Continual pretraining and parameter-efficient adaptation. Continual or domain-adaptive pretraining adapts a general checkpoint to a new distribution and can improve in-domain performance without training from scratch [14]. However, full-weight CPT is costly at LLM scale. Parameter-efficient fine-tuning methods such as LoRA reduce adaptation cost by learning low-rank updates (Hu et al., 2022), but adapter-only CPT can under-update representations needed for robust knowledge acquisition. Recent work studies LoRA variants and their stability, including rank-stabilized scaling (Kalajdzievski, 2023), retention and forgetting behavior [7], and broader PEFT trade-offs (Mao et al., 2025). Most directly related, Pezeshkpour and Hruschka (2025) analyze how far LoRA-based continual pretraining can support domain-specific knowledge acquisition. The individual ingredients of our recipe have precedents: LongLoRA shows that training the embedding and normalization layers alongside LoRA is important when adapting to a new context-length regime [9], and discriminative fine-tuning assigns different learning rates to different layers (Howard and Ruder, 2018). Our contribution is to combine these with selective upper-block unfreezing for language-adaptive continual pretraining and to measure the resulting cost–quality trade-off against full CPT.
Arabic evaluation and safety benchmarks. Arabic LLMs are commonly evaluated using exam-style QA, multiple-choice knowledge benchmarks, and safety/trust stress tests. Representative benchmarks include AlGhafa [4], ArabicMMLU (Koto et al., 2024), Arabic EXAMS [15], and AraTrust [3]. The Open Arabic LLM Leaderboard v2 combines these with MadinahQA, ALRAGE, and ArbMMLU-HT into a seven-benchmark suite [12], and HELM Arabic re-implements the same suite within the HELM framework (Liang et al., 2023; Mai et al., 2025).
Proposed Method
AQL Data Pipeline
Arabic web-scale text is abundant but highly heterogeneous: boilerplate, templated pages, SEO spam, aggressive duplication, and inconsistent orthography reduce the fraction of effective training tokens. AQL Pipeline is a quality-first pipeline that converts noisy Arabic web text into a curated corpus suitable for continual pretraining (CPT). We start from 885B raw Arabic tokens and filter to a 215B-token high-quality corpus. Table 1 summarizes token retention.
| Stage | Tokens (B) | Retention |
| Raw crawl | 885 | 100.0% |
| Stages 1–2 (structural + heuristic quality) | 325 | 36.7% |
| Stage 3 (MSA selection) | 275 | 31.1% |
| Stages 4–5 (model-based + dedup) | 215 | 24.3% |
Table 1. Token accounting through the AQL pipeline.
Design goals. AQL is optimized for (i) token efficiency (maximize downstream gains per token), (ii) distribution control (explicitly manage MSA vs. dialect and code-switching), (iii) auditability (store per-stage reasons/scores), and (iv) safety-by-construction (filter risky content before CPT).
Data model and streaming execution
AQL operates on sharded JSONL where each record contains: {doc_id, url, timestamp, source, raw_text, meta}. Each stage appends derived fields (e.g., clean_text, features, msa_prob, quality_score, safety_scores, dedup_cluster) and a structured filter_log capturing pass/fail reasons. The pipeline is implemented in a streaming manner to avoid materializing full corpora in memory: each stage reads shards, emits filtered shards, and writes lightweight counters for token accounting and ablations.
Stage 1: Structural Cleaning and Arabic Normalization. Stage 1 removes structural noise that inflates token counts without adding linguistic signal, producing clean_text: Markup stripping: remove HTML tags, script/style blocks, and embedded navigation fragments when detectable. Boilerplate cleanup: drop repeated headers/footers, cookie banners, and templated lists using simple repetition heuristics. Unicode and whitespace: normalize Unicode, remove control characters, collapse whitespace/newlines, and standardize punctuation spacing. Arabic orthography normalization: remove tatweel, and normalize common punctuation variants. This stage also extracts and preserves metadata (URL/domain, language-id hints if available) to support domain-level dedup and analysis.
Stage 2: Heuristic Quality Filtering with Feature Computation. Stage 2 computes lightweight document-level features on clean_text (Stage 1 output) and removes clearly low-signal pages using predefined thresholds (decided before training and logged for reproducibility), following the general principle that large-scale web corpora benefit substantially from systematic filtering and de-dup–aware quality controls (Penedo et al., 2024; Rae et al., 2021). The intent is to eliminate common Arabic web failure modes that inflate token counts without adding learning signal, including spam/SEO artifacts, boilerplate-heavy templates, digit/symbol-heavy noise, and low-diversity repetitive text. Concretely, we enforce basic length and layout constraints (min/max tokens or characters, and mean line length), compute script/noise ratios (Arabic character ratio, digit ratio, punctuation ratio, and symbol ratio), and quantify repetition and diversity (maximum repeated-character runs, within-document repeated token -gram fraction, and lexical diversity proxies such as unique-token ratio). We also include simple lexical/template indicators, such as stopword-ratio bounds, a template score based on near-identical lines or list-like patterns, and a banned-keyword trigger that flags documents containing terms from a small curated blacklist of high-risk/low-signal keywords; this blacklist is treated as a deterministic rule and

Figure 1. AQL Pipeline overview. The AQL pipeline progressively refines raw web text via structural cleaning, heuristic filtering, dialect control, model-based quality/safety scoring, and exact+near deduplication.
| Rule (on clean_text) | Threshold |
| Min / max tokens per document | 50 / 100,000 |
| Arabic character ratio2 | |
| Digit + punct. + symbol ratio | |
| Max repeated-character run | |
| Repeated token 4-gram fraction | |
| Unique-token ratio | |
| Stopword ratio | [0.15, 0.45] |
| Banned keyword present | drop |
Table 2. Stage 2 heuristic rules. Thresholds were fixed before any training run.
is versioned and logged like other thresholds. Both the stopword list and the blacklist were built by the authors. For auditability, every rejected document is assigned a primary rejection reason (e.g., low_arabic_ratio, repetition, template, banned_keyword), and we store the computed features and the final decision, enabling debugging and corpus analysis.
Stage 3: MSA vs. Dialect Classification. Arabic performance depends strongly on dialect balance and code-switching. Stage 3 assigns each document an MSA probability using a MARBERTv2 classifier [1] fine-tuned on 70K manually annotated samples (Section A.1) and applied to all Stage 2 survivors. We apply a 100% MSA policy: only documents accepted as MSA by the classifier are retained, which keeps 84.6% of Stage 2 tokens (275B of 325B) and makes the curated 215B-token corpus MSA-only.
Stage 4: Model-Based Quality & Safety Scoring. Stage 4 applies a learned multi-task scorer to each candidate document to estimate both content quality and safety risk. Concretely, we use a Gemma 3 4B [13] model fine-tuned on 100K labeled samples to produce a 7-dimensional score vector for each document :
where is a continuous quality score (higher is better), and are category-wise risk scores (higher indicates a higher likelihood of unsafe content in that category). We retain documents that are high quality while keeping risk scores below thresholds:
with a quality threshold , per-category safety thresholds , and . Threshold values are listed in Table 9.
| Rule | Condition | Threshold |
| Quality | 0.7 | |
| Toxicity | 0.5 | |
| Hate | 0.5 | |
| Harm | 0.4 | |
| Sexual | 0.5 | |
| Violence | 0.5 | |
| Extremism | 0.5 |
Table 9. Stage 4 filtering thresholds.
Stage 5: Exact + Near-Duplicate Removal. Duplication is pervasive in Arabic web data (mirrors, reposts, scraped news, and templated pages), which wastes CPT steps and reduces effective diversity. AQL therefore applies deduplication after the strongest filtering stages so that similarity is computed over cleaner candidates and the dedup policy preserves informative variety. For exact dedup, we compute a deterministic hash over a normalized representation of clean_text (collapsed whitespace and standardized punctuation) and remove duplicates using a persistent hash index. For near-duplicate dedup, we use a standard MinHash+LSH pipeline: documents are shingled into -grams, mapped to MinHash signatures, and bucketed via locality-sensitive hashing to identify near-copies above a Jaccard threshold . Within each near-duplicate cluster, we keep a single representative according to a deterministic policy based on the Stage 4 quality score, content length, and timestamp, and drop the rest. We log cluster sizes, removal rates, and representative-selection statistics to quantify how much token reduction is attributable to duplication and to support auditability.
Algorithm 1 AQL pipeline: a streaming filtering
pass followed by global deduplication.
1: Input: raw shards $\{S_k\}$; Output: curated shards $\{S'_k\}$
2: $C \leftarrow \varnothing$ {pool shared by all shards}
3: for each document $d$ in each shard $S_k$ do
4: $x \leftarrow \operatorname{Clean}(d.\operatorname{raw})$; $f \leftarrow \operatorname{Features}(x)$
5: if not $\operatorname{HeuristicPass}(f)$ then
6: continue {Stage 2, log reason}
7: end if
8: if not $\operatorname{MSAPass}(p(\operatorname{MSA} \mid x))$ then
9: continue {Stage 3}
10: end if
11: $(q,s) \leftarrow \operatorname{Scorer}(x)$
12: if not $\operatorname{keep}(q,s)$ then
13: continue {Stage 4}
14: end if
15: add $d$ to $C$ with $x,f,q,s$
16: end for
17: $C \leftarrow \operatorname{NearDedup}(\operatorname{ExactDedup}(C))$ {Stage 5}
18: write $C$ as shuffled shards $\{S'_k\}$ with audit metadata
Final corpus format. The final corpus is stored as shuffled, length-bucketed JSONL shards for efficient training. Each record includes text plus minimal metadata (e.g., doc_id, source/domain, msa_prob, Stage 4 quality/safety scores, and dedup cluster ids) to support mixture control and reproducible token accounting. Algorithm 1 summarizes the full pipeline.
AQL-Lite: Partial-Parameter Continual Pretraining
The AQL pipeline produces a high-quality MSA corpus, but continual pretraining (CPT) with full-weight updates remains expensive at 12B scale. AQL-Lite approximates full CPT with a partial-parameter recipe that updates low-rank adapters everywhere and full-rank weights only where they are expected to matter most.
What we did end-to-end. Our training follows four phases: (1) controlled CPT experiments that vary only the data at a matched 1B-token budget and only the training recipe at a matched 5B-token budget; (2) the main CPT scale-up on 40B tokens sampled from the MSA-only AQL corpus; (3) supervised instruction tuning on 1M examples (800K core instruction-following + 200K safety/alignment); and (4) deployment with a lightweight safety classifier that screens unsafe user inputs before they reach the model. All large-scale training is executed on 8 × A100 80GB GPUs.
Training objective
Let be a pretrained base checkpoint. CPT adapts to a target distribution (here: AQL-curated MSA text) by minimizing next-token prediction loss:
Full CPT updates all parameters , which is costly, while adapter-only CPT can under-update the representations a new language distribution requires. AQL-Lite balances the two by training LoRA adapters across the network and unfreezing only the last 8 transformer blocks, together with the token embedding matrix. Within the unfrozen blocks, layer-wise learning rates assign larger steps to later layers; the blocks below them are updated only through low-rank adapters. This lets high-level representations adapt aggressively while lower-level linguistic features change little.
Partial-parameter CPT
In a single CPT run, AQL-Lite jointly trains (i) RS-LoRA adapters on all linear projections (query, key, value, output, gate, up, and down), (ii) all weights of the last 8 transformer blocks, and (iii) the token embedding matrix. We regulate adaptation with differentiated learning rates: separate rates for adapters and embeddings, and a layer-wise schedule within the unfrozen blocks that gives larger updates to later blocks.
Trainable parameter partition. Let the base model parameters be , adapter parameters be , token embeddings be , and the number of transformer blocks be ( for Gemma 3 12B). We unfreeze the last blocks, with index set
and partition base parameters as
AQL-Lite trains the set
The 8 unfrozen blocks contain 1.79B parameters and the embedding matrix, which Gemma 3 ties to the LM head and which we count once, 1.01B; together they are 2.80B, or 23.8% of the 11.77B-parameter language model. RS-LoRA adapters on all seven projections of all 48 blocks add 4.09M parameters per unit of rank, so the total is 2.80B 4.09M, where is the adapter rank. Even at a large rank such as the adapters add about 1.05B parameters, so the trainable share stays below one third of the model; Table 10 lists the full-rank groups.
| Setting | Value |
| Shared by all 5B-token runs | |
| Sequence length | 4096 (packed) |
| Precision | bfloat16 |
| Gradient clipping | 1.0 |
| Gradient accumulation | 16 |
| Parallelism | DeepSpeed ZeRO-2 |
| Hardware | 80GB |
| AQL-Lite | |
| Adapters | RS-LoRA, all linear layers |
| Unfrozen blocks | last 8 of 48 (1.79B) |
| Embeddings (tied LM head) | trained (1.01B) |
| (adapters) | |
| (top block) | |
| (layer-wise decay) | 0.7 |
| (embeddings) | |
| Wall-clock time, 5B tokens | |
| Full CPT / LoRA-only | 121h / 17h |
| RS-LoRA / AQL-Lite | 24h / 63h |
Table 10. Training configuration of the 5B-token runs in Table 3. The 40B-token run uses the AQL-Lite settings.
Objective. We optimize the standard next-token prediction (NTP) loss over sequences :
RS-LoRA adapters (stable low-rank updates). For a base linear projection , LoRA adds a low-rank update with , where , , and . Standard LoRA uses ; RS-LoRA (Kalajdzievski (2023)) uses
which keeps the scale of the update stable as the rank grows and improves CPT stability.
Selective unfreezing (fractional full-rank updates). Unlike full CPT (updates all ) or adapter-only CPT (updates only ), AQL-Lite additionally updates full-rank base weights in the upper blocks only:
This targets the blocks where high-level semantic abstractions are thought to form, while keeping lower-level lexical and syntactic processing stable.
Differentiated (layer-wise) learning rates. We use separate optimizer parameter groups for the adapters (), the embeddings (), and each unfrozen block (). For the unfrozen blocks we use a monotone schedule that gives larger steps to later layers, following discriminative fine-tuning (Howard and Ruder (2018)):
Update rule. Let denote a parameter group with learning rate . Each group is updated by the optimizer with its own rate; schematically,
where is the optimizer’s update direction for .
Embedding training. We unfreeze the embedding matrix and train it with its own learning rate . This adapts the representations of existing subwords to MSA text; the tokenizer is not extended. In our runs , larger than .
Design rationale. Full-rank updates to the upper blocks and embeddings allow larger representational changes than low-rank adapters alone, while adapters in the frozen blocks provide a cheap path for the rest of the network to adjust. § 4.1.2 tests each component.
Experiments
Continual Pretraining
We run controlled CPT experiments and one scale-up, all with the recipe of § 3.2:
Experiment 1 (5B tokens, recipe comparison): AQL-Lite against full CPT and adapter-only CPT on the same 5B-token subset, testing whether the recipe recovers most of the full-CPT gain at lower cost.
Ablations (§ 4.1.2): a data ablation at a matched 1B-token budget, varying only the training data, and a component ablation at 5B tokens, varying only the recipe.
Experiment 2 (40B tokens, scale-up): the same recipe on 40B tokens of the MSA-only AQL corpus, followed by instruction tuning.
Experiment 1: 5B-token controlled CPT
Objective. We test whether partial-parameter CPT can recover most of the gains of full CPT while substantially reducing compute. To isolate the effect of the training recipe, all methods are trained on the same fixed 5B-token subset and evaluated with the same protocol.
Data and setup. The 5B-token subset is 96% MSA text sampled from the AQL corpus, plus 2% code and 2% math data drawn from outside the AQL corpus. All runs use identical normalization and tokenization, identical hardware (8×A100 80GB), and the same configuration: sequences packed into 4096-token blocks, gradient accumulation 16, bfloat16 precision, gradient clipping 1.0, and DeepSpeed ZeRO-2 (Rasley et al. (2020)). Table 10 summarizes the configuration.
Methods. We compare five settings: Gemma 3 12B PT (Gemma Team et al. [13]) with no CPT, Full CPT with all parameters trainable, LoRA-only with adapter-only updates, RS-LoRA with rank-stabilized adapter-only updates, and AQL-Lite. AQL-Lite combines RS-LoRA on all linear layers, full-rank updates to the last eight blocks (2.80B parameters together with the embeddings), embedding training, and layer-wise learning rates with , , , and .

Figure 2. AQL-Lite phases. We validate the recipe in a 5B-token controlled CPT setting, then scale CPT to 40B tokens (100% MSA), followed by 1M-example instruction tuning and deployment-time safety filtering.
Evaluation. We evaluate all checkpoints on four Arabic benchmarks: ArabicMMLU (Koto et al., 2024), AlGhafa (Almazrouei et al., [4]), AraTrust (Alghamdi et al., [3]), and Arabic EXAMS (Hardalov et al., [15]), and report the macro-average as Mean. All checkpoints in Table 3–Table 5 are evaluated with the same protocol; these scores come from a separate evaluation of CPT checkpoints and should not be compared directly with Table 6.
| Method | Time | ArabicMMLU | AlGhafa | AraTrust | Arabic EXAMS | Mean |
| Gemma 3 12B PT | – | 65.56 | 60.68 | 87.01 | 54.38 | 66.91 |
| Full CPT | 121h | 68.92 | 77.33 | 89.22 | 58.85 | 73.58 |
| LoRA-only | 17h | 66.43 | 71.11 | 87.75 | 55.68 | 70.24 |
| RS-LoRA | 24h | 66.57 | 72.81 | 88.27 | 55.87 | 70.88 |
| AQL-Lite (ours) | 63h | 68.26 | 78.00 | 89.15 | 57.36 | 73.19 |
Table 3. Controlled CPT experiment on 5B tokens. All methods use the same data, hardware, and evaluation protocol. Scores are in percentage points; Mean is the macro-average across the four benchmarks. Time is wall-clock time on 8×A100 80GB.
| Config. (5B tok.) | ArMMLU | AlGhafa | AraTr. | EXAMS | Mean |
| RS-LoRA only | 66.57 | 72.81 | 88.27 | 55.87 | 70.88 |
| Unfreeze last 8 only | 64.74 | 63.84 | 88.42 | 54.85 | 67.96 |
| RS-LoRA + emb. | 67.28 | 75.23 | 88.12 | 55.43 | 71.52 |
| RS-LoRA + unfreeze | 66.89 | 74.91 | 88.84 | 56.26 | 71.73 |
| AQL-Lite (full) | 68.26 | 78.00 | 89.15 | 57.36 | 73.19 |
| Full CPT | 68.92 | 77.33 | 89.22 | 58.85 | 73.58 |
Table 5. Component ablation on the 5B-token subset of Table 3. “Unfreeze” trains all weights of the last eight blocks; “emb.” trains the embedding matrix. Full CPT is shown for reference.
| Model | Mean | AlGhafa | ArabicMMLU | Arabic EXAMS | MadinahQA | AraTrust | ALRAGE | ArbMMLU-HT |
| Reported by the HELM Arabic leaderboard | ||||||||
| Gemini 2.5 Flash | 81.7 | 78.0 | 81.6 | 63.2 | 85.0 | 93.9 | 89.5 | 80.6 |
| GPT-5.1 (2025-11-13) | 80.9 | 79.1 | 80.3 | 63.1 | 80.4 | 94.1 | 90.9 | 78.4 |
| Qwen3 235B A22B Instruct | 78.6 | 77.3 | 76.9 | 59.0 | 77.7 | 91.7 | 89.0 | 78.3 |
| Llama 4 Scout Instruct | 74.6 | 77.5 | 73.4 | 60.0 | 72.5 | 89.1 | 81.1 | 68.6 |
| ALLaM-7B-Instruct-preview | 69.9 | 70.0 | 69.4 | 55.0 | 68.7 | 83.7 | 85.2 | 57.1 |
| Jais-adapted-70B-chat | 66.3 | 72.9 | 63.9 | 53.1 | 60.5 | 85.4 | 71.3 | 57.1 |
| Gemma 3 12B base (evaluated by us with OALL v2) | ||||||||
| AQL-Lite-12B (CPT only, before SFT) | 74.8 | 77.5 | 74.6 | 58.8 | 75.2 | 89.2 | 74.1 | 73.9 |
| AQL-Lite-12B (after SFT) | 75.3 | 78.2 | 75.0 | 59.2 | 75.7 | 89.7 | 74.8 | 74.3 |
Table 6. Head-to-head comparison on the seven OALL v2 benchmarks, in percentage points (higher is better; Mean is the macro-average). The bottom block shares the Gemma 3 12B base and was evaluated by us with OALL v2. The top block is taken from the HELM Arabic leaderboard, which uses a different protocol, so cross-block comparisons are indicative only. Best value per column in bold.
Results and analysis. Table 3 shows that AQL-Lite reaches a mean of 73.19, compared with 73.58 for Full CPT and 66.91 for the base model. It thus recovers 94% of the full-CPT gain (6.28 of 6.67 points) while reducing wall-clock training time from 121h to 63h (52%). It outperforms LoRA-only and RS-LoRA on all four benchmarks. AQL-Lite is ahead of Full CPT on AlGhafa (78.00 vs. 77.33) and behind it on AraTrust (89.15 vs. 89.22); both margins are small, and we do not read them as differences between the methods. All rows are single training runs. Figure 3 plots score against training time.
Figure 3. Mean score on four Arabic benchmarks against wall-clock CPT time for the 5B-token runs of Table 3.
Wall-clock time alone does not isolate where the savings come from: in every method, adapters in all blocks (and, for AQL-Lite, the trainable embeddings) require gradients to flow through the whole network, so the savings come from fewer weight gradients and smaller optimizer state rather than from a shorter backward pass.
Ablations
Does curation matter? Table 4 varies only the training data at a matched 1B-token budget. CPT on raw crawled Arabic leaves the mean unchanged (66.80 vs. 66.91 for the base model). Structural cleaning and heuristic filtering (Stages 1–2) add 0.65 points, and the full pipeline adds a further 2.86, for a total of +3.51 over raw text at the same budget (Figure 4). Most of the gain comes from AlGhafa (+10.52 over raw), with smaller gains on Arabic EXAMS (+1.71), ArabicMMLU (+1.02), and AraTrust (+0.77). The mean increases at each stage, but not every benchmark does: Arabic EXAMS drops slightly after Stages 1–2. Because the last row applies MSA selection, model-based filtering, and deduplication together, this experiment does not separate their contributions.
Figure 4. Data ablation: change over the base model after CPT on 1B tokens of each data condition (values from Table 4).
| Data (1B tok.) | ArMMLU | AlGhafa | AraTr. | EXAMS | Mean |
| Base (no CPT) | 65.56 | 60.68 | 87.01 | 54.38 | 66.91 |
| Raw crawl | 65.37 | 59.32 | 88.06 | 54.46 | 66.80 |
| Stages 1–2 | 65.61 | 62.77 | 88.12 | 53.28 | 67.45 |
| Stages 1–5 (AQL) | 66.39 | 69.84 | 88.83 | 56.17 | 70.31 |
Table 4. Data ablation: CPT on 1B tokens of each data condition with an identical training recipe; only the data changes. ArMMLU = ArabicMMLU, AraTr. = AraTrust, EXAMS = Arabic EXAMS.
Which components matter? Table 5 adds the components of AQL-Lite one at a time. Unfreezing the last eight blocks without adapters is the weakest configuration (67.96), 2.92 points below RS-LoRA alone, so the network-wide low-rank path matters more than full-rank updates confined to the top blocks. Adding embedding training (+0.64) or the unfrozen upper blocks (+0.85) to RS-LoRA each helps, and the full recipe gains 2.31 points over RS-LoRA, more than the sum of the two individual gains (1.49). The full recipe also adds layer-wise learning rates, which are not ablated separately, so the remaining 0.82 points may reflect an interaction between components, the learning-rate schedule, or both. We also did not tune the number of unfrozen blocks; a sweep over is left to future work.
Experiment 2: 40B-token CPT scale-up (100% MSA)
Objective and data policy. Experiment 2 applies the same recipe to 40B tokens sampled from the MSA-only AQL corpus. The 40B budget is chosen to adapt an already capable multilingual checkpoint, not for compute-optimal pretraining, and we make no claim about the optimal token budget. The MSA-only setting reduces dialectal variance and targets the formal Arabic emphasized by our benchmark suite; MSA–dialect mixtures are left to future work.
Training configuration. We keep the configuration of Experiment 1 (Table 10) and scale the token budget. The resulting CPT checkpoint is then instruction-tuned to obtain AQL-Lite 12B (instruct).
Alignment
We convert the 40B-token CPT checkpoint into the final assistant model using supervised instruction-tuning (SFT) on 1M Arabic instruction examples: 800K core instruction-following examples and 200K safety/alignment examples, mixed 80/20. The loss is cross-entropy computed only on assistant response tokens, following standard instruction-tuning practice (Chung et al., [10]). The data covers factual QA, summarization, extraction, rewriting, translation, classification, constrained generation, structured outputs, and multi-turn dialogue.
Data provenance. The data was built from Nojoom AI’s production Arabic assistant: the prompts are human-written questions drawn from its usage logs, and the responses were crafted on the same platform.
Inference-time safety classifier. At deployment time, we add a lightweight safety classifier as an inference-time guardrail. It screens unsafe user requests before they reach AQL-Lite 12B (instruct), enabling refusal or safe redirection without retraining the LLM.
Evaluation
Benchmarks and protocol
We evaluate on the seven benchmarks of the Open Arabic LLM Leaderboard v2 (OALL v2) [12]: AlGhafa [4], ArabicMMLU (Koto et al., 2024), Arabic EXAMS [15], MadinahQA (Koto et al., 2024), AraTrust [3], ALRAGE [12], and ArbMMLU-HT (Sengupta et al., 2023). We evaluate our own checkpoints with the OALL v2 evaluation suite. For reference, Table 6 also lists scores of other models on the same seven benchmarks as reported by the HELM Arabic leaderboard (Mai et al., 2025); we did not run these models. HELM Arabic uses its own protocol (zero-shot generation with Arabic option letters, at most 1,000 instances per subset), so comparisons between our rows and the leaderboard rows are indicative rather than exact.
Results
With the protocol caveat above, AQL-Lite-12B’s mean (75.3) lies between those reported for Qwen3 235B (78.6) and Llama 4 Scout (74.6), and above ALLaM-7B (69.9) and Jais-adapted-70B (66.3). It is close to the best reported AlGhafa score (78.2 vs. 79.1 for GPT-5.1) and comparatively weakest on ALRAGE (74.8), the only open-ended, passage-grounded task in the suite.
CPT versus instruction tuning. AQL-Lite-12B already scores 74.8 before SFT; instruction tuning adds 0.5 points to the mean and between 0.4 and 0.7 points on each benchmark. On this suite, the model’s performance is therefore largely in place after CPT, and SFT mainly contributes instruction-following and safety behaviour that these benchmarks do not measure directly.
Discussion
Our results suggest that efficient MSA adaptation depends on the interaction between corpus quality, distribution control, and where model updates are applied. AQL-Lite couples a quality-first data pipeline with a CPT recipe that updates both low-rank adapters and selected full-rank parameters.
Curation, not corpus size. The pipeline keeps 24.3% of raw tokens, but the size of that reduction does not by itself show that the removed text was less useful. The controlled comparison does: at a matched 1B-token budget, raw web text leaves the base model unchanged while curated text adds 3.51 points (Table 4). For the MSA-centred benchmarks we use, effective token quality matters more than raw token volume.
Partial-parameter CPT as a practical middle ground. Adapter-only CPT is efficient but limited, while full CPT is effective but costly. AQL-Lite recovers 94% of the full-CPT gain in 52% of the wall-clock time. The component ablation indicates that the network-wide low-rank path is necessary, since full-rank updates confined to the top blocks perform worst, and that full-rank upper-block and embedding updates add to it.
Knowledge acquisition or re-alignment? Gemma 3 is a multilingual model that already handles Arabic, and our benchmarks cannot separate genuinely new knowledge from better use of knowledge the base model already has. The data ablation shows that curated text contributes more than raw text at the same budget, but it does not resolve this question. A from-scratch study on a smaller model (1B–4B parameters) would test the corpus outside the adaptation setting.
Safety as a layered design. AQL-Lite treats safety as a pipeline-level property rather than a single alignment step. Risk is reduced upstream through model-based corpus filtering, during SFT through a dedicated safety/alignment subset, and at inference time through a lightweight safety classifier. This layered design is especially relevant for Arabic deployments, where cultural, religious, regional, and political sensitivities can make unsafe or inappropriate outputs more harmful. We report AraTrust for the model alone and do not measure the contribution of each layer separately.
Future work. AQL-Lite-12B is the first model in the AQL family. Future work will study controlled MSA–dialect mixtures selected with the Stage 3 classifier, sweep the number of unfrozen blocks, run a from-scratch study on a smaller model, scale token budgets, improve morphology-aware filtering, and extend evaluation to dialect robustness, bias analysis, and deployment-oriented safety auditing.
Limitations
AQL-Lite remains an efficiency–performance trade-off rather than a full replacement for full-weight continual pretraining: in the controlled 5B-token experiment it recovers 94% of the full-CPT gain but does not match Full CPT on every benchmark. The main 40B-token scale-up uses a 100% MSA policy, which strengthens standardized Arabic performance but underrepresents dialectal Arabic, code-switching, and region-specific language use; the model should not be assumed to handle dialects well.
Several choices are not fully ablated. The number of unfrozen blocks () was not tuned, the layer-wise learning-rate schedule is not ablated separately from the other components, and the data ablation applies Stages 3–5 jointly. All results come from single training runs, so differences below about one point should not be over-interpreted. We have not yet measured -gram overlap between our training data and the benchmarks, or the effect of CPT on English capabilities; both are planned for a future version. We also did not run full CPT at 40B tokens, so the efficiency comparison with full CPT is validated only at the 5B-token scale. Our efficiency comparison counts CPT wall-clock time only; it excludes the cost of running the Stage 3 and Stage 4 classifiers over the corpus and the cost of instruction tuning. As discussed in § 6, our benchmarks also cannot distinguish new knowledge from re-alignment of knowledge the base model already has.
Our evaluation is based on automatic benchmarks and a head-to-head comparison in which the comparison rows are taken from a public leaderboard that uses a different evaluation protocol, rather than re-run by us. The MSA classifier’s 99.96% accuracy is measured in-domain; such a high value may partly reflect annotation artifacts or near-duplicates shared between the training and validation splits, and an out-of-domain check on dialect identification data such as MADAR ([8]) or NADI ([2]) is needed to confirm it. The Stage 2 requirement of at least 70% Arabic characters likely removes technical and code-switched MSA, such as programming tutorials and scientific text with Latin-script terms, which may limit the model on technical and coding prompts. The evaluation may also not capture all dimensions of Arabic instruction-following, dialect robustness, factual reliability, or deployment safety. Finally, despite corpus filtering, safety instruction tuning, and an inference-time safety classifier, the model may still hallucinate, reflect biases from web data, or fail under adversarial prompting, especially in high-stakes domains.
Ethics Statement
AQL-Lite is intended to support Arabic language technology for education, information access, writing assistance, and Arabic-centric retrieval-augmented applications. Because Arabic NLP systems operate in culturally, religiously, and politically sensitive contexts, model errors or biased outputs may cause harm. Relevant risks include misinformation, cultural or religious insensitivity, dialect marginalization caused by MSA-centred training, and unsafe responses to adversarial prompts.
Instruction data. The instruction-tuning data was built from Nojoom AI’s production Arabic assistant: the prompts come from its usage logs and the responses were crafted on the same platform. Because the prompts were written by real users, their use for training depends on user consent and on removing personal information, so this version does not release the instruction-tuning data.
Mitigations. We mitigate these risks through layered safeguards: upstream corpus filtering with model-based quality and safety scoring, safetyfocused instruction tuning, and a deployment-time safety classifier that screens user inputs before they reach the model. However, these safeguards do not eliminate all risks. We therefore recommend that AQL-Lite not be used as the sole decision-maker in high-stakes settings such as medical, legal, financial, or civic decision-making. Any release should include clear usage guidelines, staged access when appropriate, monitoring for misuse, and continued auditing for bias, toxicity, dialect coverage, and culturally sensitive failure cases.
Pipeline Details
Stage 3: MSA vs. Dialect Classifier
Stage 3 enforces distribution control by predicting whether a document is Modern Standard Arabic (MSA) or dialectal. This enables the 100% MSA policy used here and, in future work, controlled MSA/dialect mixtures.
Model and training. We fine-tune a MARBERTv2 encoder ([1]) with a classification head on 70K labeled samples.
Inference policy. Given a document , the classifier outputs , and documents accepted as MSA are kept. This retains 84.6% of Stage 2 tokens (275B of 325B).
Validation performance. Table 7 reports in-domain validation metrics. They measure agreement with our annotations on held-out samples from the same distribution, not general MSA/dialect identification ability. Because near-perfect in-domain scores can also arise from annotation artifacts or near-duplicate leakage across splits, we treat them as an upper bound; an out-of-domain evaluation on MADAR ([8]) or NADI ([2]) is left to future work.
| Metric | Value |
| Eval loss | 0.00346 |
| Accuracy | 0.99964 |
| Macro-F1 | 0.99963 |
| Macro-Precision | 0.99956 |
| Macro-Recall | 0.99970 |
Table 7. MSA vs. dialect classifier, in-domain validation metrics (final epoch).
Stage 4: Quality & Safety Scorer
Stage 4 uses a learned multi-task scorer that outputs a continuous quality score and category-wise safety risk scores. We fine-tune a Gemma 3 4B backbone with a lightweight multi-head prediction layer: one regression head for quality and six classification heads for safety categories (toxicity, sexual, violence, harm, hate, extremism). Writing for the predicted quality and for the predicted risk in category , we optimize a weighted multi-task objective:
where is the binary label for category and the weights are tuned to mitigate class imbalance and stabilize training.
Validation performance. Table 8 reports held-out validation metrics.
| Metric | Value | Notes |
| Eval loss | 0.0834 | Multi-task objective |
| Quality MSE | 0.00523 | Lower is better |
| Toxicity Acc / F1 | 0.958 / 0.958 | Binary head |
| Sexual Acc / F1 | 0.994 / 0.975 | Binary head |
| Violence Acc / F1 | 0.984 / 0.969 | Binary head |
| Harm Acc / F1 | 0.940 / 0.857 | Most ambiguous |
| Hate Acc / F1 | 0.976 / 0.939 | Binary head |
| Extremism Acc / F1 | 0.988 / 0.956 | Binary head |
| Safety Micro-F1 | 0.973 | All safety labels |
Table 8. Stage 4 scorer validation metrics (final epoch). Quality is evaluated with MSE; safety categories report accuracy and F1.
Qualitative sanity checks. We run short, controlled checks to ensure that clearly unsafe inputs yield high risk scores while unrelated categories remain low. To avoid reproducing harmful text, we report aggregate metrics only.
Score thresholds. Stage 4 keeps a document if and for every category . Table 9 lists the thresholds.
Stage 5: Deduplication
Stage 5 reduces memorization risk and improves effective data diversity with two-level deduplication: (i) exact deduplication by hashing normalized text, and (ii) near-duplicate removal with shingling and MinHash/LSH.
Exact deduplication. We compute a stable hash over the Stage 1-normalized text and remove repeated items using a global index.
Near-duplicate deduplication. We represent each document as a set of shingles, approximate Jaccard similarity with MinHash signatures, and use LSH buckets to find candidate duplicates. Pairs with similarity above are merged, keeping one representative per cluster; see § 3.1.
Training Configuration
Human Evaluation Figures
Figures 5 to 7 show per-domain results of the pairwise human evaluation.
Figure 5. Evolution of Elo ratings over the course of the pairwise comparisons.

Figure 6. Per-domain Elo scores in the human evaluation (10 two-turn prompts per domain, 3 annotators).

Figure 7. Outcomes of AQL-Lite-12B’s pairwise comparisons (win, tie, loss, both bad) in each domain.
References
- [1]Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics.
- [2]Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop. Association for Computational Linguistics.
- [3]Emad A. Alghamdi, Reem Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2025. AraTrust: An evaluation of trustworthiness for LLMs in Arabic. In Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025), pages 8664–8679. Association for Computational Linguistics.arxiv.org/abs/2403.09017
- [4]Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malaric, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. AlGhafa evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023, pages 244–275. Association for Computational Linguistics.DOI
- [5]Manel Aloui, Hasna Chouikhi, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. 2024. 101 billion Arabic words dataset. Preprint, arXiv:2405.01590.arxiv.org/abs/2405.01590
- [6]M. Saiful Bari, Yazeed Alnumay, Norah Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, et al. 2024. ALLAM: Large language models for Arabic and English. Preprint, arXiv:2407.15390.arxiv.org/abs/2407.15390
- [7]Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chilev, Jonathan Frankle, et al. 2024. LoRA learns less and forgets less. Preprint, arXiv:2405.09673.
- [8]Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdul-rahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA).DOI
- [9]Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. LongLoRA: Efficient fine-tuning of long-context large language models. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2309.12307
- [10]Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53.
- [11]Zoltan Csaki, Bo Li, Jonathan Lingjie Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. SambaLingo: Teaching large language models new languages. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 1–21. Association for Computational Linguistics.
- [12]Ali El Filali, Manel Aloui, Tarique Husaain, Ahmed Alzubaidi, Basma El Amel Boussaha, Ruxandra Cojocaru, Clémentine Fourrier, Nathan Habib, and Hakim Hacid. 2025. The Open Arabic LLM leaderboard 2. Hugging Face blog, https://huggingface.co/blog/leaderboard-arabic-v2. 10 February 2025.
- [13]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.arxiv.org/abs/2503.19786
- [14]Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360. Association for Computational Linguistics.
- [15]Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5427–5444. Association for Computational Linguistics.arxiv.org/abs/2011.03080


