WorksComputer Science
When Does Continual Pre-Training Help? A Controlled Study of Dialectal Adaptation for Québec French
Abstract
Continual pre-training (CPT) is widely used to adapt pretrained language models to dialectal and non-standard language varieties, yet it remains unclear when this additional training is necessary, what drives its benefits, and whether aggregate evaluation scores adequately capture its effects. In this paper, we present a controlled study of CPT for Québec French language understanding across nine LLMs spanning different families and scales. Using a budget-matched protocol of 30M tokens per experiment, our results show that the benefit of CPT depends strongly on the starting model's existing competence in the target variety. When CPT is beneficial, the improvements are driven more by corpus composition than by data volume, with literary text accounting for most of the gain. Paired Québec and standard French evaluations further show that CPT primarily improves expression comprehension rather than grammatical competence, and that corpus register determines which regional expressions are acquired. These gains also involve capability-specific trade-offs, including declines in reading comprehension that aggregate scores can conceal. Overall, these findings show that CPT for dialectal adaptation should not be treated as a uniformly beneficial default. Instead, adaptation strategies should begin with a careful assessment of the base model's existing competence, accompanied by source-level data analysis and task-level evaluation.
Introduction
Large Language Models (LLMs) are pretrained on massive corpora to support a wide range of applications, including summarization, natural language and code generation, and open-domain dialogue. Despite their capabilities, LLMs often perform better on the standard form of a language than on its regional varieties. This phenomenon, known as the dialect gap [25], persists even in multilingual models trained on large and diverse corpora [21, 10, 28]: when a dialect is underrepresented in the training data, its linguistic capabilities are often reduced in the resulting LLM.
Continual pre-training (CPT) [26] is widely used to address dialectal adaptation. By taking an existing model pretrained on the standard language and continuing its training on dialectal text, CPT aims to improve regional language competency while preserving the broad capabilities learned during pre-training [18, 32].
However, this approach leaves several questions unresolved: 1) Base models can differ substantially in their existing competence in the target variety, making adaptation unnecessary for models that already perform well; 2) dialectal corpora are often small and heterogeneous, combining a variety of genres and registers. Unless the amount of training data is controlled, it is difficult to determine whether performance changes are caused by specific genres and registers or by greater volume of texts; and 3) aggregate benchmark scores may not be sufficiently representative of the gains (or degradation) of an LLM’s capabilities in specific tasks. We isolate three factors that are usually confounded in dialectal adaptation studies: the base model’s existing competence, the composition of the adaptation corpus, and the granularity of evaluation. We focus on Québec French, for which paired dialectal and standard French benchmarks targeting the same capabilities are available, which makes this isolation possible. Our evaluation is restricted to language understanding; whether CPT changes the dialectal character of generated text is left for future work.
To investigate these research questions, we conduct a controlled, budget-matched study of CPT for Québec French. Our evaluation spans nine LLMs from three model families (0.5B to 14B parameters), with each experiment utilizing exactly 30M training tokens. This uniform constraint not only enables direct comparisons between various corpus compositions, but also mirrors the realistic data scarcity inherent to low-resource dialectal adaptation, where clean, target-variety text is often difficult to obtain. We compare every adapted model with its own unadapted baseline. Evaluation pairs each Québec French benchmark with a standard French benchmark measuring the same linguistic capability, drawn from the COLE suite [3], alongside standard French retention tasks and held-out perplexity scores.
Together, our findings suggest that, in the low-resource regime where dialectal adaptation is typically performed, CPT should not be treated as the default response to dialect under-representation. Under a limited adaptation budget, the impact of CPT depends on the starting model, the composition of the adaptation corpus, and trade-offs among individual capabilities that aggregate scores and perplexity may fail to reveal. Specifically, our contributions in this work are:
We show that the benefit of dialectal CPT for comprehension depends strongly on the base model’s existing competence in the target variety, and that informal-only adaptation degrades regional expression comprehension in eight of nine models.
We show that corpus composition matters more than data volume, with gains driven primarily by literary text on CroissantLLM, the model that gains most from CPT.
We show that aggregate evaluation scores and held-out perplexity provide an incomplete evaluation of CPT: task-level evaluation reveals large and opposing capability changes, including losses in reading comprehension, that remain hidden behind aggregate performance.
Related Work
Continual Pre-Training (CPT)
Continual pre-training (CPT) adapts pretrained language models to new domains via self-supervised learning on unlabeled text [18, 26]. Recent work explores strategies such as multilingual training [11, 15, 23] and data replay [17, 20] to improve adaptation while preserving prior knowledge. Crucially, CPT’s efficacy depends heavily on model scale and domain size: small or mismatched adaptation corpora can leave stronger models unchanged or even degrade them [31].
Dialectal and Low-Resource Adaptation
Because LLMs perform unevenly on underrepresented regional varieties [25, 13], a standard approach is to adapt existing models rather than train them from scratch. While low-resource adaptation often applies CPT to massive, heterogeneous multilingual corpora [28, 22, 36], dialectal adaptation is distinct because models already possess substantial knowledge of the associated standard language. Prior work employs techniques such as supervised fine-tuning, LoRA, and synthetic data to bridge these dialect gaps [29, 2, 19]. For Québec French specifically, recent work has demonstrated that adaptation is feasible under strict resource constraints, while highlighting that outcomes remain highly sensitive to corpus composition [27].
To our knowledge, no previous work has systematically isolated the role of base model competence, demonstrated how specific textual genres drive the acquisition of regional expressions rather than grammar, or studied how standard aggregate metrics mask the task-specific capability trade-offs inherent to dialectal continual pre-training.
Corpus and Adaptation
Québec French Corpus
To investigate these research questions, we conduct a controlled, budget-matched study of CPT for Québec French. We use the mixed-source Québec French corpus of [27] which covers multiple genres and registers. Table 1 shows the composition of the corpus.
| Source | Genre | Register | Tokens |
| BEQ | Literary | Formal | 15.4M |
| Québec Wikipedia | Encyclopedic | Formal | 31.2M |
| Le Soleil | News | Formal | 5.9M |
| Formal subtotal | 52.5M | ||
| CRIFUQ | Speech | Informal | 0.8M |
| Social media | Informal | 2.4M | |
| Dépotoir | Web forum | Informal | 22.8M |
| MontrealRacing | Web forum | Informal | 4.2M |
| YouTube | Comments | Informal | 2.2M |
| Social media | Informal | 1.8M | |
| Informal subtotal | 34.2M | ||
| Total | 86.7M |
Table 1. Composition of the Mixed-Source Québec French Adaptation Corpus. Token count was computed using the CroissantLLM tokenizer.1
As shown in Table 1, the complete 86.7M-token corpus is drawn from nine sources categorized into two primary registers. The formal register (52.5M tokens) consists of clean, standardized written language, including public-domain literature (Bibliothèque Électronique du Québec), Québec Wikipedia and news articles. Conversely, the informal register (34.2M tokens) captures vernacular language, comprising transcribed speech (*Centre de recherche interuniversitaire sur le français en usage au Québec) alongside colloquial user-generated web text from regional discussion forums and social media platforms.
Adaptation Setup
Models and Training. We perform CPT on nine LLMs across three families to compare adaptation across different architectures and levels of prior French exposure: CroissantLLM (1.35B) (Faysse et al., [14]), Llama-3.2/3.1 (1B, 3B, 8B) (Grattafiori et al., [16]), and Qwen2.5 (0.5B, 1.5B, 3B, 7B, 14B) (Team, [33]). To ensure a fair comparison and avoid oversampling our smallest informal sources, all experiments are strictly budget-matched to 30M tokens. We adapt all models using LoRA while keeping the base-model weights frozen. Full optimization details, hyperparameters, and checkpoint selection criteria are provided in Appendix § 10.
Evaluation. We evaluate our models on two separate axes: dialect-specific acquisition and standard French retention. To isolate Quebec dialectal competence, we use two benchmarks:
QFrBLiMP (Beauchemin et al., [5]) is a grammatical acceptability task consisting of 1,761 sentence pairs, where the model must distinguish a correct sentence from one containing a common grammatical mistake sourced from official Québec guidelines.
QFrCoRE (Beauchemin et al., [4]) is a definition-matching task containing 4,633 regional idiomatic multi-word expressions, requiring the model to identify the correct meaning from ten possible candidates.
We pair these with standard French equivalents that target the same linguistic capabilities in order to serve as the standard French control:
MultiBLiMP-Fr (Jumelet et al., [24]) is a grammatical acceptability task parallel to QFrBLiMP, consisting of 255 sentence pairs.
MFrCoE (Beauchemin et al., [4]) is a definition-matching task parallel to QFrCoRE, containing 4,938 standard French expressions.
In addition, to monitor for standard French forgetting during adaptation, we track performance on four general standard French tasks from the COLE suite (Beauchemin et al., [3]):
AlloCiné (Blard, [7]) is a binary sentiment classification task evaluating a corpus of 20,000 French movie reviews.
PAWS-X (Yang et al., [35]) is a paraphrase detection task requiring the model to identify whether two lexically overlapping sentences are semantically equivalent. We used the test set of 2,000 French samples for evaluation.
Fr-BoolQ (Clark et al., [9]) is a binary reading comprehension task consisting of yes/no questions paired with context passages. We used the French subset of 178 test samples.
MMS (Augustyniak et al., [1]) is a sentiment analysis task evaluating text drawn from diverse domains using a binary or trinary classification schema. We used the French subset of 63,190 test samples.
Experimental Setup
To investigate CPT for dialectal adaptation, we designed several experiments along three core dimensions: 1) the impact of corpus composition (see § 4); 2) the impact of base model capacity (see § 5); and 3) the masking effect of aggregate metrics (see § 6).
Across all experiments, we report macro-F1 scored via length-normalized candidate log-likelihoods, averaged over three random seeds. Additionally, we compute held-out perplexity (PPL) for both language varieties. Unless otherwise noted, all performance changes are measured against each model’s unadapted baseline.
Impact of Corpus Composition
To evaluate how register and genre influence dialectal acquisition, we run three experiments. The first (see § 4.1) examines the formal-only and informal-only register extremes across all nine LLMs. The second experiment (see § 4.2) varies the formal-to-informal ratio exclusively on CroissantLLM to measure how granular changes in register impact the acquired dialect. Finally, the third experiment (see § 4.3) conducts a leave-one-genre-out ablation relative to the full mixed-source corpus to isolate the contribution of specific genres.
Table 2 shows the full mixed-source corpus and the variants obtained by reweighting individual sources. To vary the register, we construct five variants ranging from formal-only (100/0) to informal-only (0/100). Within each register, sources are sampled proportionally to their distribution in the full corpus. If a source’s target allocation exceeds its raw availability (e.g., CRIFUQ), we do not artificially cycle the text over multiple epochs. Instead, all available tokens from that source are used exactly once, and the shortfall is backfilled with random sentences sampled from other sources within the same register. This protocol maintains the strict 30M-token budget across all experiments while preventing overfitting to the smallest domains.
Register Sweep Across all Models
To evaluate the effect of register, we first examine the formal-only and informal-only variants across all nine LLMs. Full results are in Appendix B, in Table 5.
As Figure 1 (a) and (b) show, grammatical performance fluctuates minimally, shifting at most 2.0% on QFrBLiMP and 4.8% on MultiBLiMP-Fr. This confirms that dialectal CPT safely preserves underlying syntax regardless of register.

Figure 1. Register Sweep Across All Models: Shifts in grammatical performance (top row) and expression comprehension (bottom row) at the two register extremes for all nine models. Models are ordered left to right by baseline QFrCoRE. Results are difference in Macro-F1 compared to baseline result, averaged over three seeds.
Expression comprehension, by contrast, is highly sensitive to corpus composition (see Figure 1 (c) and (d)). Formal-only training yields higher QFrCoRE scores across almost all models, averaging a 5.3-point average advantage over informal-only. Conversely, informal training degrades QFrCoRE in eight of nine models, averaging a 4.0% drop, while raising the standard French equivalent, MFrCoE (5.0%), in all nine. Formal training drives the exact opposite shift, primarily in weaker base models.
Because QFrCoRE and MFrCoE share identical multiple-choice formats, this opposing shift confirms a change in linguistic knowledge rather than mere format learning. This divergence exposes an evaluation trap: if a model trained on informal Québec data is validated on standard French
| Mixture Condition | Budget (M) | BEQ | Wiki | Soleil | CRIFUQ | FB | Dép | MR | YT | |
| Original Corpus | 86.7 | 15.4 | 31.2 | 5.9 | 0.8 | 2.4 | 22.8 | 4.2 | 2.2 | 1.8 |
| Formal-only (100/0) | 30.0 | 8.8 | 17.8 | 3.4 | – | – | – | – | – | – |
| 75% Formal (75/25) | 30.0 | 6.6 | 13.4 | 2.5 | 0.2 | 0.5 | 5.0 | 0.9 | 0.5 | 0.4 |
| Mixed (50/50) | 30.0 | 4.4 | 8.9 | 1.7 | 0.4 | 1.1 | 10.0 | 1.8 | 1.0 | 0.8 |
| 75% Informal (25/75) | 30.0 | 2.2 | 4.5 | 0.8 | 0.5 | 1.6 | 15.0 | 2.8 | 1.4 | 1.2 |
| Informal-only (0/100) | 30.0 | – | – | – | 0.7 | 2.1 | 20.0 | 3.7 | 1.9 | 1.6 |
Table 2. Corpus Variants Used in the Register Sweep. Values represent the allocated token budget (in millions) for each source. Sources within each register are sampled proportionally to their raw availability to maintain exactly 30M tokens per condition.
benchmarks, rising scores create a false illusion of success. Instead, our paired evaluation reveals that target dialect comprehension degrades.
Continuous Register Sweep on CroissantLLM
To observe the impact of the register across a continuous spectrum, we conduct a register sweep on CroissantLLM 1.35B. We chose this model as our testbed because it shows the largest gain in regional expression comprehension during CPT (see § 4.1). As Table 3 shows, under formal-only training, its QFrCoRE score rises from a 27.5% baseline to 35.7% (+8.2%). This strong capacity to absorb regional knowledge provides the clear signal needed to accurately track how incremental changes in the data mixture affect dialect acquisition.
| Condition | QFrBLiMP | QFrCoRE | AlloCiné | PAWS | Fr-BoolQ | MMS | QC | FR |
| Base (no CPT) | 89.2 | 27.5 | 50.4 | 35.5 | 56.2 | 25.4 | 17.24 | 8.31 |
| Formal-only | 89.9 | 35.7 | 56.6 | 41.0 | 37.7 | 25.5 | 15.63 | 9.89 |
| 75/25 | 89.4 | 33.9 | 54.1 | 39.1 | 40.2 | 25.4 | 15.79 | 9.73 |
| Mixed (50/50) | 90.2 | 33.7 | 50.8 | 39.0 | 40.8 | 25.3 | 15.99 | 9.62 |
| 25/75 | 91.0 | 32.2 | 57.2 | 38.6 | 41.7 | 25.3 | 16.30 | 9.53 |
| Informal-only | 88.5 | 28.2 | 60.0 | 40.7 | 38.9 | 25.1 | 23.19 | 10.70 |
| Double (60M)2 | 90.7 | 33.8 | 61.8 | 39.0 | 43.5 | 25.4 | 15.67 | 9.84 |
Table 3. Continuous Register Sweep on CroissantLLM 1.35B: Results are Macro-F1, averaged over three seeds. Perplexity (PPL) is evaluated on held-out Québec French and standard French sets.
When informal text progressively replaces formal text, this dialectal gain decreases monotonically (), while grammatical performance (QFrBLiMP) fluctuates in no particular direction. This continuous decline reinforces our broader finding that the target variety’s unique vocabulary is sparse in informal data and significantly better preserved in formal sources.
To isolate the effect of register, we verified whether dialectal adaptation relies on specific data curation or simply raw text exposure. If volume were the primary factor, doubling the budget should yield substantial gains. Therefore, we evaluate a 60M-token Mixed Double condition, a 50/50 formal–informal mixture, which reaches a QFrCoRE score of 33.8%. As shown in Table 3, this is only 0.1% above the standard 30M-token Mixed (50/50) condition (33.7%). By comparison, allocating the baseline 30M-token budget entirely to formal text increases QFrCoRE to 35.7%. Because a targeted 30M-token corpus outperforms the mixed 60M-token corpus, we conclude that data composition matters more than total volume at this scale.
Leave-One-Genre-Out Ablations
Having established that formal text drives the acquisition of regional expressions, we use a leave-one-genre-out ablation to verify which genre aids adaptation the most. When a source is removed, the remaining sources are upsampled to maintain the budget, ensuring any performance change is driven by the absence of that genre’s linguistic characteristics rather than a reduction in data.
As Table 4 shows, literary text is the clear driver of dialectal adaptation. Removing 15.4M tokens of BEQ text drops QFrCoRE from the full-corpus reference of 33.7% to 28.3%, erasing nearly all CPT gains. Conversely, removing the largest individual source, Québec Wikipedia (31.2M tokens), leaves the score unchanged at 33.8%.
| Removed Genre | QFrBLiMP | QFrCoRE | AlloCiné | PAWS | Fr-BoolQ | MMS | QC | FR |
| None (full corpus) | 90.2 | 33.7 | 53.1 | 39.0 | 40.5 | 25.4 | 15.91 | 9.68 |
| - Encyclopedic (QC Wiki, 31.2M) | 90.1 | 33.8 | 57.3 | 39.7 | 40.3 | 25.4 | 16.32 | 9.51 |
| - Literary BEQ, (15.4M) | 90.3 | 28.3 | 50.8 | 40.3 | 35.2 | 25.3 | 10.46 | 11.73 |
| - News (Le Soleil, 5.9M) | 90.2 | 34.8 | 53.5 | 35.9 | 45.8 | 25.3 | 15.85 | 9.71 |
| - Forums/social (Dépotoir, MtlRacing, FB, Reddit, 31.2M) | 89.5 | 34.9 | 65.0 | 38.8 | 41.9 | 25.5 | 15.65 | 9.85 |
| - Speech/comments (CRIFUQ, YouTube, 3.0M) | 90.0 | 33.7 | 52.9 | 39.2 | 39.7 | 25.4 | 15.89 | 9.73 |
Table 4. Leave-One-Genre-Out Ablations at fixed budget for CroissantLLM 1.35B. All rows train on the same 30M token budget. Results are the mean of three seeds.
Grammatical performance (QFrBLiMP), in contrast, is unchanged, fluctuating by less than a single point (89.5% 90.3%) across all ablations. While the acquisition of new regional expressions is highly sensitive to the genres provided during training, the model’s fundamental syntactic capabilities remain preserved regardless of the mixture. QFrBLiMP specifically evaluates the model’s robustness against common regional syntactic errors and English calques documented by official Québec linguistic guidelines. Its stability demonstrates that CPT injects divergent dialectal expressions without corrupting the model’s pre-existing, shared syntactic foundation.
Beyond dialectal acquisition, the ablation reveals that corpus composition dictates standard French retention. Removing literary text strengthens the decline in standard French reading comprehension (Fr-BoolQ: 40.5% 35.2%) and degrades standard French perplexity (9.68 11.73). Conversely, removing informal data spikes AlloCiné by 11.9% (53.1% to 65.0%), suggesting that informal text interferes with pre-existing sentiment capabilities.
The perplexity scores further expose a dissociation between language modeling metrics and downstream capability. Removing literary text yields the best Québec French PPL in the entire suite (15.91 10.46), yet this condition produces the worst QFrCoRE score (33.7% 28.3%). This inverse relationship confirms that perplexity improvements can be driven by simplifying the held-out distribution rather than by acquiring regional features.
Overall, the results in Table 4 isolate genre, not register, to demonstrate that composition matters more than raw volume. Because the budget remains controlled, the decline when removing BEQ (33.7% 28.3%) confirms that literary text supplies the regional expressions the benchmark requires, whereas removing Wikipedia, a formal encyclopedic source twice the size of BEQ, leaves performance unchanged. This supports the register-sweep result of § 4.2: the benefit of formal-heavy training comes not from formality itself, but from the literary component that formal-heavy mixtures contain more of. We offer a hypothesis: BEQ consists of public-domain Québec literary works, whose older, lexically denser vocabulary makes it the most plausible source of the traditional regional expressions QFrCoRE evaluates, which encyclopedic, news, and modern informal text may rarely contain.
Impact of Base Model Capability
Having identified formal-only text as the optimal mixture, we investigate whether dialectal adaptation benefits all LLMs equally. To answer this, we apply the same training procedure across nine LLMs, again using a fixed 30M token adaptation budget. This allows us to verify the role of each LLM’s starting competence. Full results are available in Appendix § 11, in Table 6.
| Model | Family | Base | +CPT | |
| CroissantLLM 1.35B | Croissant | 27.5 | 35.7 | +8.2 |
| Qwen2.5 0.5B | Qwen | 30.2 | 31.7 | +1.5 |
| Qwen2.5 1.5B | Qwen | 31.5 | 32.1 | +0.6 |
| Llama-3.2 1B | Llama | 31.9 | 33.3 | +1.4 |
| Llama-3.2 3B | Llama | 33.0 | 33.1 | +0.1 |
| Llama-3.1 8B | Llama | 35.9 | 34.5 | −1.4 |
| Qwen2.5 3B | Qwen | 32.7 | 34.1 | +1.4 |
| Qwen2.5 7B | Qwen | 33.7 | 35.2 | +1.5 |
| Qwen2.5 14B | Qwen | 39.1 | 36.9 | −2.2 |
Table 6. Formal-only CPT versus each model’s own untrained baseline on QFrCoRE. Scores are reported as Macro-F1, and taken as the mean of three random seeds.
Figure 2 shows a divergence between grammatical acceptability and expression comprehension based on a model’s initial competence. Plotting CPT performance against the unadapted baseline reveals that grammatical performance (QFrBLiMP) remains stable. Scores fluctuate by at most 2.0% across all models regardless of size, clustering along the diagonal and confirming that syntactic structures are preserved across the spectrum.

Figure 2. Impact of Baseline Competence: CPT performance plotted against the unadapted baseline. The diagonal dashed line represents unchanged performance.
Regional expression comprehension (QFrCoRE), in contrast, shows an inverse relationship between baseline competence and adaptation gains, where weaker base models benefit the most. The limited 30M-token budget acts as an equalizer, pushing smaller models significantly above the diagonal (+8.2% for CroissantLLM, +1.5% for Qwen 0.5B). Meanwhile, improvements in performance shrink as baseline competence increases, and the two strongest baselines fall below the diagonal (-1.4 for Llama-3.1 8B, -2.2 for Qwen2.5 14B). These results suggest that highly capable models already possess much of the regional knowledge in our corpus, and that a 30M-token signal is too sparse to update their internal representations. Budget-constrained CPT thus effectively bootstraps smaller models, but frontier models likely require larger data volumes or targeted instruction tuning to show measurable dialectal gains.
Pitfalls of Aggregate Metrics
Finally, to evaluate if aggregate scores conceal forgetting and task-specific trade-offs, we replace a portion of the training budget with standard French replay data from a held-out corpus1. This isolates severe, capability-specific regressions and recoveries that would otherwise remain hidden behind stable aggregate averages.
Standard French Replay
While § 5 showed how base model capability dictates the acquisition of new dialectal knowledge, measuring success solely by these gains ignores the potential capabilities lost in the process. In standard continual learning, adapting a model to a new domain often degrades its prior knowledge, a phenomenon known as catastrophic forgetting [30, 6]. To expose this hidden cost, we allocate a portion of our 30M-token budget (10%, 25%, and 50%) to standard French replay data. Full results are provided in Appendix § 11, in Table 7.
| Condition | Québec French BLiMP | Québec French CoRE | Standard French AlloCiné | Standard French PAWS | Standard French BoolQ | Standard French MMS | PPL ↓ QC | PPL ↓ FR |
| No replay (ref.) | 90.2 | 33.3 | 50.8 | 39.0 | 40.8 | 25.3 | 15.99 | 9.62 |
| 10% replay | 90.0 | 33.9 | 47.7 | 36.2 | 45.8 | 25.4 | 16.02 | 8.67 |
| 25% replay | 90.3 | 33.7 | 45.4 | 36.0 | 48.1 | 25.4 | 16.08 | 8.55 |
| 50% replay | 89.8 | 33.7 | 43.1 | 35.7 | 50.1 | 25.4 | 16.22 | 8.47 |
| 25% FR+EN | 90.4 | 33.7 | 45.7 | 36.0 | 48.0 | 25.4 | 16.08 | 8.55 |
Table 7. Standard French replay on a 50/50 Québec French mixture for CroissantLLM 1.35B. Deltas discussed in the text are relative to the unadapted base model in Table 3. Results are the mean of three seeds.
As Figure 3 shows, evaluating standard French capabilities through a broad aggregate metric makes the replay intervention appear ineffective. Averaging across the four general French benchmarks yields a flat trend line (see the dotted black line), sitting around 3% below the unadapted baseline (41.9%) and hovering around 38.8% regardless of the replay ratio. Relying solely on this macroscopic evaluation would mistakenly lead us to conclude that replay fails to restore any standard French capabilities degraded by dialectal CPT.

Figure 3. Standard French replay on the 50/50 Québec French mixture for CroissantLLM 1.35B.
Disaggregating these scores, however, reveals opposing capability shifts. As standard French replay increases, performance on reasoning-heavy tasks such as Fr-BoolQ (blue line) recovers monotonically (40.8% 50.1%). However, as Fr-BoolQ improves by 9.3%, the standard French sentiment task AlloCiné degrades by 7.7%. This opposing shift exposes a trade-off under a constrained token budget: re-introducing formal standard French restores logical inference but pulls the model’s distribution away from the colloquial language necessary for domain-specific sentiment tasks.
Meanwhile, performance on PAWS-X (green line) and MMS (pink line) remains static across all replay ratios. Because these flat scores are averaged with the opposing swings of Fr-BoolQ and AlloCiné, the resulting aggregate metric is artificially stabilized.
Rather than restoring a generic standard French average, replay specifically rehabilitates reading comprehension without erasing dialectal gains, as QFrCoRE holds steady at 33.7%. First, as shown in Figure 4, standard French PPL strictly improves as replay volume increases (9.62 8.47), indicating global language modeling gains despite the stable task average. Second, as reported in Table 7 (see Appendix § 11), replacing French-only replay with a balanced French-English source at the 25% ratio yields nearly identical Fr-BoolQ performance (48.0% vs. 48.1%). Because CroissantLLM is bilingual, this suggests that the recovery of its logical reasoning capabilities is not strictly tied to standard French exposure, but can be effectively stimulated by generic pre-training text in either of the model’s core languages.

Figure 4. Held-out Perplexity during standard French replay for CroissantLLM 1.35B.
Aggregate Metrics Mislead
The masking effect observed in § 6.1 exposes an important flaw in aggregate scoring: broad averages conceal severe capability trade-offs forced by replay under a constrained token budget. This masking also occurs during adaptation itself. As noted in § 4.1, the formal-only condition improves overall Québec dialect performance while hiding an 18.5% degradation on Fr-BoolQ (56.2% baseline 37.7%).
Beyond aggregate averages, held-out perplexity presents a second evaluation trap. When dedicated dialectal benchmarks are unavailable, perplexity is often used as a proxy for adaptation success [18, 34]. Our results caution against this substitution. The clearest evidence is the within-model dissociation in the genre ablation (see Table 4): the experiment with literary text removed achieves the best (lowest) PPL of any CroissantLLM condition (15.91 10.46) yet yields the worst QFrCoRE score (28.3%).
Removing lexically dense literary text from the training corpus simplifies the language modeling task, which artificially improves perplexity. However, because this literary text is the primary source of regional vocabulary, removing it erases the model’s ability to comprehend the dialectal expressions measured by the benchmark. While perplexity is not always inversely correlated with downstream performance, this dissociation proves it is an unreliable predictor of dialectal acquisition: a model can achieve "better" perplexity simply by training on standard, predictable text, even as it fails to learn the target dialect.
Echoing recent critiques of NLP leaderboards [12, 8], our findings demonstrate that shortcut metrics, whether aggregate averages or perplexity scores, are insufficient for evaluating dialectal adaptation in a comprehensive way. Validating adaptation requires the disaggregated, per-task reporting adopted throughout this paper.
Conclusion
We presented a controlled study of dialectal CPT for Québec French language understanding on nine LLMs from three families. By restricting the training budget to 30M tokens, we demonstrated that the efficacy of adaptation is dictated primarily by a model’s baseline competence rather than the process itself. While our weakest base model gained 8.2 points in regional expression comprehension, stronger models hit a capability ceiling, yielding no benefit, leaving the most capable unadapted models strictly superior to all adapted variants.
Furthermore, when CPT succeeds, its benefits are narrow: improvements are concentrated in expression comprehension rather than grammar, driven by formal literary text rather than raw volume, and highly sensitive to register. Notably, relying on informal genres actively degraded target dialect comprehension in eight of the nine models. Finally, we exposed the zero-sum nature of these gains in expression comprehension: severe degradations in reading comprehension, which standard French replay recovers but aggregate task averages conceal.
Additionally, our results caution against using held-out perplexity as a proxy for adaptation success. We reveal a dissociation where simplifying the held-out distribution, such as removing lexically dense literary text, artificially improves perplexity while destroying the model’s acquisition of regional expressions.
Ultimately, our findings challenge the assumption that dialectal CPT is a uniformly beneficial default. Effective low-resource adaptation requires assessing the base model’s competence, curating adaptation corpora by genre rather than volume, and favoring disaggregated, task-level evaluation.
Limitations
Although this study highlights important gaps in the assessment of dialectal adaptation, our empirical scope is limited. Our analysis is restricted to Québec French and relies exclusively on the COLE benchmark suite alongside its standard French counterparts. Consequently, while our methodological critiques regarding aggregate metrics and perplexity are applicable to the Québec French dialect, the specific dynamics we observe, such as the strong impact of literary text, must be validated across other dialects and languages before being generalized.
Secondly, all models are adapted with the same LoRA configuration (see Appendix § 10). We fixed these values so that differences in performance could be attributed to the base model and the corpus rather than to tuning, and because of hardware constraints. This configuration produces relatively small updates and may underestimate what CPT can achieve for larger and stronger models, so the absence of gains for these models should be read as a property of this setting rather than a theoretical upper bound. We do not evaluate full-parameter training, other LoRA ranks or learning rates, or budgets beyond 60M tokens.
Additionally, our evaluation covers language understanding only. Dialectal adaptation is often motivated by the wish for models that produce the target variety, and our results do not show whether CPT changes the dialectal character of generated text. Evaluating generation, for instance through human judgments of Québec French expressions and constructions in model outputs, is an important direction for future work.
Finally, while we show that formal literary text drives the acquisition of regional expressions, the exact reason why this occurs has not been investigated in this study. Future work should analyze these literary texts more deeply to understand why they so heavily outperform web data for dialectal adaptation.
Acknowledgments
This project was undertaken thanks to funding from NSERC, IVADO and the Canada First Research Excellence Fund. The authors would also like to thank CN2i / Le Soleil for generously giving us access to their data.
Training Details and Hyperparameters
All experiments are implemented using LoRA with rank , scaling factor , and dropout 0. LoRA is applied to all attention and feed-forward projections (, , , , gate, up, and down), while the base-model weights are completely frozen.
We train using the AdamW optimizer with a weight decay of 0.01, a peak learning rate of , and a cosine learning rate schedule with a 0.1 warmup ratio. We apply gradient clipping at 1.0, use a sequence length of 1024 with a stride of 512, and train using mixed precision (fp16) with gradient checkpointing. We reserve 5% of each source for validation and held-out perplexity evaluation; this text is never used for training. At the end of training, we select the final model checkpoint based on the lowest validation loss.
Full Result Tables
Register Sweep Across all Models
| Model | QFrBLiMP Formal | QFrBLiMP Inf. | MultiBLiMP-Fr Formal | MultiBLiMP-Fr Inf. | QFrCoRE Formal | QFrCoRE Inf. | MFrCoE Formal | MFrCoE Inf. |
| Croissant 1.35B | +0.7 | −0.8 | −1.3 | −1.3 | +8.2 | +0.7 | +2.0 | +4.8 |
| Qwen2.5 0.5B | −0.7 | −2.0 | −4.8 | +1.7 | +1.5 | −3.4 | +0.7 | +4.2 |
| Qwen2.5 1.5B | +1.4 | +0.4 | +1.7 | +1.3 | +0.6 | −4.0 | +0.3 | +5.3 |
| Llama-3.2 1B | +0.8 | −0.4 | −0.9 | −2.2 | +1.4 | −4.3 | +0.4 | +2.2 |
| Qwen2.5 3B | −0.4 | −1.3 | 0.0 | +0.9 | +1.4 | −4.3 | −1.1 | +6.6 |
| Llama-3.2 3B | −0.3 | −0.7 | −1.3 | −2.2 | +0.1 | −6.1 | +0.1 | +3.8 |
| Llama-3.1 8B | −1.4 | +0.1 | 0.0 | −0.1 | −1.4 | −3.3 | −0.3 | +5.7 |
| Qwen2.5 7B | +0.4 | 0.0 | 0.0 | 0.0 | +1.5 | −3.8 | −0.2 | +6.1 |
| Qwen2.5 14B | −0.7 | +0.1 | −1.7 | −3.2 | −2.2 | −7.8 | −0.9 | +6.5 |
Table 5. Grammatical performance and expression comprehension under the two register extremes (formal-only and informal-only CPT), for all nine models. Results are reported as the change in Macro-F1 relative to each model’s unadapted baseline.
Impact of Base Model Capability
Standard French Replay
References
- [1]Łukasz Augustyniak, Szymon Woźniak, Marcin Gruza, Piotr Gramacki, Krzysztof Rajda, Mikołaj Morzy, and Tomasz Jan Kajdanowicz. 2023. Massively multilingual corpus of sentiment datasets and multi-faceted sentiment classification benchmark. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023) Datasets and Benchmarks Track, New Orleans, Louisiana, USA.arxiv.org/abs/2306.07902
- [2]Niyati Bafna, Emily Chang, Nathaniel Romney Robinson, David R. Mortensen, Kenton Murray, David Yarowsky, and Hale Sirin. 2025. DialUp! Modeling the Language Continuum by Adapting Models to Dialects and Dialects to Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), pages 20188–20233, Vienna, Austria. Association for Computational Linguistics.
- [3]David Beauchemin, Yan Tremblay, Mohamed Amine Youssef, and Richard Khoury. 2025. COLE: A Comprehensive Benchmark for French Language Understanding Evaluation. Preprint, arXiv:2510.05046.arxiv.org/abs/2510.05046
- [4]David Beauchemin, Yan Tremblay, Mohamed Amine Youssef, and Richard Khoury. 2026a. Idiom Understanding as a Tool to Measure the Dialect Gap. In Findings of the Association for Computational Linguistics: ACL 2026, pages 505–522, San Diego, California, USA. Association for Computational Linguistics.arxiv.org/abs/2510.05026
- [5]David Beauchemin, Pier-Luc Veilleux, Richard Khoury, and Johanna-Pascale Roy. 2026b. QFrBliMP: A Quebec-French Benchmark of Linguistic Minimal Pairs. In Findings of the Association for Computational Linguistics: EACL 2026, pages 4274–4304, Rabat, Morocco. Association for Computational Linguistics.
- [6]Magdalena Biesialska, Krzysztof Biesialski, and Marta R. Costa-jussà. 2020. Continual Lifelong Learning in Natural Language Processing: A Survey. In Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020), pages 6523–6541, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- [7]Théophile Blard. 2020. French Sentiment Analysis with BERT (Allociné Dataset). https://github.com/TheophileBlard/french-sentiment-analysis-with-bert. Allociné: 200k French movie reviews for sentiment analysis.
- [8]Samuel R. Bowman and George Dahl. 2021. What Will it Take to Fix Benchmarking in Natural Language Understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2021), pages 4843–4855, Online. Association for Computational Linguistics.
- [9]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), Minneapolis, Minnesota, USA.arxiv.org/abs/1905.10044
- [10]Gheorghe Comanici and 1 others. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. ArXiv, abs/2507.06261.arxiv.org/abs/2507.06261
- [11]Zoltan Csaki, Bo Li, Jonathan Lingjie Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. Sam-baLingo: Teaching Large Language Models New Languages. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 1–21, Miami, Florida, USA. Association for Computational Linguistics.
- [12]Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the Eye of the User: A Critique of NLP Leaderboards. In Proceedings of the 25th Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 4846–4853, Online. Association for Computational Linguistics.arxiv.org/abs/2009.13888
- [13]Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastassopoulos. 2024. DIALECTBENCH: An NLP Benchmark for Dialects, Varieties, and Closely-Related Languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), pages 14412–14454, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2403.11009
- [14]Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2024. CroissantLLM: A truly bilingual french-english language model. Preprint, arXiv:2402.00786.arxiv.org/abs/2402.00786
- [15]Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities. In Proceedings of the 1st Conference on Language Modeling (COLM 2024), Philadelphia, Pennsylvania, USA.arxiv.org/abs/2404.17790
- [16]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.arxiv.org/abs/2407.21783
- [17]Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023. Continual Pre-Training of Large Language Models: How to re-warm your model? In Workshop on Efficient Systems for Foundation Models @ ICML2023 (ES-FoMO 2023), Honolulu, Hawaii, USA.arxiv.org/abs/2308.04014
- [18]Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), pages 8342–8360, Online. Association for Computational Linguistics.arxiv.org/abs/2004.10964
- [19]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In Proceedings of the 10th International Conference on Learning Representations (ICLR 2022), Online.arxiv.org/abs/2106.09685
- [20]Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothée Lesort, and Irina Rish. 2024. Simple and scalable strategies to continually pre-train large language models. Transactions on Machine Learning Research. Accepted by TMLR.arxiv.org/abs/2403.08763
- [21]Aaron Jaech and 1 others. 2026. OpenAI o1 System Card. Preprint, arXiv:2412.16720.arxiv.org/abs/2412.16720
- [22]Shaoxiong Ji, Zihao Li, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O’Brien, Hengyu Luo, Hin-rich Schütze, Jörg Tiedemann, and Barry Haddow. 2024. EMMA-500: Enhancing massively multilingual adaptation of large language models. Preprint, arXiv:2409.17892.arxiv.org/abs/2409.17892
- [23]Raviraj Joshi, Kanishk Singla, Anusha Kamath, Raunak Kalani, Rakesh Paul, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikар, and Eileen Long. 2025. Adapting multilingual LLMs to low-resource languages using continued pre-training and synthetic corpus: A case study for Hindi LLMs. In Proceedings of the 1st Workshop on Natural Language Processing for Indo-Aryan and Dravidian Languages, pages 50–57, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- [24]Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2026. MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs. Transactions of the Association for Computational Linguistics, 14:193–216.
- [25]Anjali Kantharuban, Ivan Vulić, and Anna Korhonen. 2023. Quantifying the Dialect Gap and its Correlates Across Languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7226–7245, Singapore. Association for Computational Linguistics.
- [26]Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual Pre-training of Language Models. In Proceedings of the 11th International Conference on Learning Representations (ICLR 2023), Kigali, Rwanda.arxiv.org/abs/2302.03241
- [27]Eеham Khan, Firas Saidani, Owen Van Esbroeck, Richard Khoury, and Leila Kosseim. 2026. Low-resource dialect adaptation of large language models: A french dialect case-study. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), pages 10723–10734, Palma de Mallorca, Spain. European Language Resources Association (ELRA).arxiv.org/abs/2510.22747
- [28]Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F. T. Martins, and Hinrich Schütze. 2024. Mala-500: Massive language adaptation of large language models. Preprint, arXiv:2401.13303.arxiv.org/abs/2401.13303
- [29]Yanchen Liu, William Held, and Diyi Yang. 2023. DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules. In Proceedings of the 28th Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 13776–13793, Singapore. Association for Computational Linguistics.arxiv.org/abs/2305.13406
- [30]Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motivation, 24:109–165.DOI
- [31]Machel Reid, Victor Zhong, Suchin Gururangan, and Luke Zettlemoyer. 2022. M2D2: A Massively Multi-Domain Language Modeling Dataset. In Proceedings of the 27th Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), pages 964–975, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.arxiv.org/abs/2210.07370
- [32]Soumaj yoti Sarkar, Kaixiang Lin, Sailik Sengupta, Leonard Lausen, Sheng Zha, and Saab Mansour. 2022. Parameter and Data Efficient Continual Pre-training for Robustness to Dialectal Variance in Arabic. In Proceedings of the NeurIPS 2022 Workshop on Efficient Natural Language and Speech Processing (ENLSP), New Orleans, Louisiana, USA.arxiv.org/abs/2211.03966
- [33]Qwen Team. 2024. Qwen2.5 technical report. Technical report, Alibaba Group. ArXiv preprint arXiv:2412.15115.
- [34]Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. 2024. QuRating: Selecting High-Quality Data for Training Language Models. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), Proceedings of Machine Learning Research, pages 52554–52583, Vienna, Austria. PMLR.arxiv.org/abs/2402.09739
- [35]Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In Proceedings of the 24th Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019), pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.arxiv.org/abs/1908.11828
- [36]Hao Yu, Tianyi Xu, Michael A. Hedderich, Wassim Hamidouche, Syed Waqas Zamir, and David Ifeoluwa Adelani. 2026. AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), pages 5909–5928, San Diego, California, USA. Association for Computational Linguistics.