WorksComputer Science
Show Your Work in Swahili: Reinforcement Learning for Native-Language Mathematical Reasoning
Abstract
We present Fikiria, a two-stage SFT+GRPO recipe for Swahili mathematical reasoning. Its reward multiplies a format gate by a weighted sum of answer accuracy and the ChrF++ similarity of the Swahili reasoning to an in-language reference chain of thought, so that training targets the surface fidelity of the reasoning trace and not only the final answer. We train on a 2.3k-problem Swahili dataset created by translating GSM8K with GPT-4.1 and filtering with SSA-COMET. On AfriMGSM Swahili, Fikiria-AfriqueQwen-14B reaches 79.2%. Under the same fixed prompt and strict answer-format scoring, GPT-4o reaches 68.0%, partly because it often fails the required answer format; IrokoBench reports 78.8% for GPT-4o using a best-per-model prompt, a figure numerically close to ours but not directly comparable. In a small human evaluation with co-author annotators, GPT-4o's outputs were preferred overall. ChrF++ measures surface fidelity, not the validity of individual reasoning steps.
Introduction
Recent mathematical reasoning models often produce correct answers, but their intermediate reasoning is expressed in English, making it inaccessible to many students. This creates a mismatch between correctness and usability: a correct answer without an understandable explanation does not support learning. We study the following problem: given a math question in Swahili, generate both (i) a correct answer and (ii) a step-by-step reasoning chain in Swahili that a student can follow. We define our target as native-language reasoning quality: reasoning that is well-formed, expressed in fluent target-language text, and paired with a correct final answer. Whether individual steps causally support the answer is not verified by our automated training signal, but is assessed qualitatively with human evaluation on a subset of outputs (§ 3.6).
Prior work on mathematical reasoning (e.g., TAPO [7], MathOctopus [4]) optimizes for answer accuracy, often treating reasoning as an auxiliary signal. CURE-Med [13] extends this to multilingual medical reasoning and enforces language consistency only on the final answer. Barua et al. [2026] explored how different stages of model development affect long reasoning traces beyond English. We explicitly optimize for the quality and native-language fidelity of reasoning in the target language with gated structured rewards applied to a <rationale> block.

Figure 1. Problem summary. Given a math question in Swahili, the LLM must generate both a correct answer and linguistically usable reasoning. Previous work has focused on improving accuracy while this paper considers both correctness of the final answer and usability of the reasoning trace through a two-stage SFT and GRPO pipeline described in Figure 2.
Our approach optimizes the reasoning trace directly rather than treating it as an auxiliary signal. Methods that target answer accuracy alone (TAPO, MathOctopus) or enforce language consistency only on the final answer (CURE-Med) can give full reward to a correct answer reached through English-language reasoning that is unusable to a Swahili-speaking student. More recent work rewards the language of the reasoning itself [25, 8]. Fikiria differs in its reward construction: a multiplicative format gate (tags, an equation in the rationale, minimum length) that zeroes the reward for malformed outputs, applied to two separate terms, a parameter-free ChrF++ similarity of the Swahili rationale to an in-language reference chain of thought and exact-match accuracy. We train on a Swahili math dataset we construct and evaluate on human-translated AfriMGSM.
Our contributions are as follows:
A reward for Swahili reasoning traces that combines a multiplicative format gate, a ChrF++ term against an in-language reference chain of thought, and exact-match accuracy, together with a 2.3k-problem Swahili training set.
A two-stage recipe in which cold-start SFT bootstraps output structure and Swahili instruction-following before GRPO. In ablations on Qwen3, SFT+GRPO outperforms GRPO alone by 6.4 points at 8B, 12.0 points at 14B, and 2.0 points (within noise) at 32B.
Related Work
Our work relates to three lines of research: accuracy-focused multilingual mathematical reasoning, rewards for language consistency in multilingual reasoning, and the quality of generated reasoning traces.
Multilingual mathematical reasoning. TAPO [7] and MathOctopus [4] improve multilingual accuracy on GSM8K [5] and MGSM [19]. TAPO follows an understand-then-reason paradigm: the model translates the question into English before reasoning, and uses ChrF++ [15] as a translation-quality reward. Its reasoning trace is therefore in English, whereas we target reasoning in the target language itself. MAPO [18] improves multilingual reasoning through alignment-as-preference optimization, and TRIT [12] jointly trains translation and reasoning in a self-improving loop that needs no external feedback. Neither rewards the language or quality of the reasoning trace itself.
Language-consistency rewards. Several recent methods use reinforcement learning with Group Relative Policy Optimization (GRPO; [17]) to keep reasoning in the input language. CURE-Med [13] applies GRPO to multilingual medical reasoning across 13 languages, including Swahili, with a linear reward combining correctness, language consistency and format (, 0.30 and 0.05). Language consistency is enforced on the final <answer>, not the reasoning trace, and the reward is not gated, so malformed outputs still receive partial credit. M-Thinker [25] gates its reward on a rule-based Language Consistency (LC) check over the thinking and answer, and adds a Cross-lingual Thinking Alignment (CTA) reward computed by an external LLM judge against the model’s own English reasoning. We avoid learned or judged rewards because of reward-hacking risk (§ 3.5). Hwang et al. [2025] train Qwen2.5-7B-Instruct with a joint SFT and GRPO objective on machine-translated questions in ten MGSM languages, including Swahili, using a language-identification reward (and, in a later version, mT5 embedding-similarity rewards against the original English reasoning trace). They also introduce a factual-reasoning benchmark with Swahili. On MGSM Swahili their outputs stay in Swahili but math accuracy drops from 14.0% to 2.1–3.7%. Qi et al. [2025] show that reasoning models often default to English or produce fragmented reasoning in other languages, and that forcing the reasoning language through prompting improves readability but lowers accuracy. This motivates optimizing the reasoning trace directly instead of relying on prompts.
Reasoning-trace quality. Supervising intermediate reasoning, rather than only final answers, has a long history in English-language settings [2, 11]. Barua et al. [2026] study how different stages of model development affect long chain-of-thought (CoT) reasoning beyond English, focusing on accuracy. Surface metrics such as ChrF++ measure text overlap with a reference rather than whether a reasoning trace causally supports the answer, a limitation we discuss in § 5.
Positioning. Gating a reward on format validity is not unique to our work: M-Thinker gates on format and language consistency, and TAPO masks its ChrF++ translation reward with a format reward. Our gate checks only format (tags present, an equation in the rationale, minimum length). Language fidelity enters through a separate term, the ChrF++ similarity of the Swahili rationale to an in-language reference chain of thought, alongside an exact-match accuracy term. Fikiria therefore differs from these methods in what the surface-metric reward is computed on and against what reference, and in being evaluated on human-translated AfriMGSM Swahili.
African NLP context. We build on the continued pre-training model family AfriqueLLM [23] and the benchmark AfriMGSM [1].
Method
Problem formulation
We consider the task of Swahili mathematical reasoning with native-language justification. Given a math problem posed in Swahili, the model must produce two outputs: a step-by-step reasoning trace in Swahili and a final numeric answer. We define success along two axes. A response is correct if the numeric answer matches the ground truth. A response has high native-language reasoning quality if the reasoning trace is expressed in fluent Swahili, contains mathematical steps consistent with a correct answer, and is legible to a native Swahili-speaking student. We measure this via surface-level linguistic fidelity (ChrF++) and answer correctness, and separately examine step-level causal support through human evaluation on a sample (§ 3.6). Optimizing for correctness alone is insufficient: a model can produce the right answer via English reasoning, which fails the linguistic requirement even when the answer is right.
Pipeline overview
Figure 2 shows the summary of the two-stage, SFT+GRPO, training along with additional details on the GRPO reward construction.

Figure 2. Fikiria training pipeline. Starting from AfriqueQwen (8B or 14B, African CPT from Qwen3), cold-start SFT teaches Swahili instruction-following and the <rationale>/<answer> output format. GRPO then refines the model using a gated reward: R₁ (format gate) multiplicatively gates R₂ (ChrF++ of <rationale> against a Swahili MT reference, =0.25) and R₃ (exact-match accuracy on <answer>, =0.50). The dashed SFT border and bypass arrow indicate that GRPO-only ablation variants omitting SFT are also evaluated (Table 1).
Dataset construction
The Grade School Math 8K dataset (GSM8K) contains 8.5k grade school math question-answer pairs and is a popular English-only benchmark for math reasoning [5]. Several datasets, such as MGSM [19] and AfriMGSM [1], extend GSM8K to non-English languages, including Swahili, via machine translation or human translation. However, these are too small for our purposes ().
To construct our dataset, we translated all question-answer pairs into Swahili using GPT-4.1 with a 3-shot prompt drawn from AfriMGSM (see Appendix § 9 for templates). We then assessed translation quality using SSA-COMET-MTL [10], a reference-free quality estimator (QE), and conducted a human spot check (see Figure 5 in Appendix § 7). For the remainder of this work, we retain only entries whose translated question and answer both score above 0.65 on SSA-COMET-MTL, reducing the train and test splits to roughly 2.3k and 400 entries, respectively.

Figure 5. Individual distributions of SSA-COMET-MTL QE scores for GPT-4.1 translated questions (left) and chain of thought (center) along with the joint distribution (right). Selection threshold of 0.65 is shown as dashed lines.
The filtering threshold was chosen to ensure high translation fidelity, which can significantly impact downstream applications [14]. We adopted the value recommended by the metric authors [D. Adelani, private comm.] and validated it through human spot-checks and cross-dataset comparison.
For the human spot-check, we randomly sampled 20 examples distributed around the threshold and asked two native Swahili speakers to annotate each translation for grammatical and logical consistency (see § 3.6 for annotator selection). The small sample size limits statistical power, but the human scores are well-balanced around the metric as a linear predictor: roughly half fall below the value predicted by linear regression at the 0.65 threshold (see Figure 7 in Appendix § 7).

Figure 7. Human spot check of GSM8K GPT-4.1 translations compared to the SSA-COMET-MTL metric (average of the Question and CoT).
In addition to the spot check, we compared SSA-COMET-MTL metrics for our dataset to AfriMGSM, which is an extension of MGSM [19, 1]. The GPT-4.1 translations score similarly for values greater than 0.65 while for samples with lower scores GPT-4.1 tends to score high (see Figure 6). Across all three reference datasets, for samples exceeding this threshold, at least 89% of GPT-4.1 samples also exceed the threshold. This indicates over all quality and agreement between the machine-translated entries to current standard datasets.

Figure 6. Comparison of SSA-COMET-MTL QE scores for AfriMGSM (yellow) questions on a per sample basis to the GPT-4.1 translation presented in this work.
We release the translations and corresponding SSA-COMET-MTL scores on Hugging Face. At roughly 2.3k training and 400 test entries after filtering, this dataset is substantially larger than MGSM and AfriMGSM, both of which contain 250 entries for testing and 8 for training.
Supervised fine-tuning
The first stage of our training pipeline is a supervised fine-tuning round that jointly entrains the chat template, Swahili response, and reasoning style. We train for completion over 3 epochs of the combined Swahili question and COT dataset using TRL to minimize the standard token-level cross-entropy loss [21].
GRPO with native-language rewards
Figure 2 shows the reward structure and equation, including R₁ (format gate) that gates all rewards, R₂ (ChrF++ vs Swahili machine-translated reference):Swahili–Swahili, and R₃ (exact match from <answer>).
The format gating requires that the generated content be of sufficient length, contain tags for the rationale and answer, and the rationale must contain an equation. By gating the reward function with the format, we enforce the answer can be parsed and includes a math equation in the rationale, which we found to be the case for all languages in the AfriMGSM dataset train split. See Appendix § 10 for the full prompt template.
For the language quality reward, R₂, we require a reward that incentivizes Swahili output while being robust to reward hacking. We considered SSA-COMET-MTL [10], an embedding-based quality estimator well-suited for African languages, but followed Huang et al. [ [7] in avoiding learned embedding-based rewards, which are susceptible to reward hacking via adversarial outputs that score highly without genuine quality improvement [20]. A second reason to avoid a learned metric here is signal separation: should measure language fidelity independently of answer correctness, which is already captured by . Conflating these into a single learned metric would obscure whether the model is being rewarded for reasoning in Swahili or for getting the answer right. Instead, we use ChrF++ [15], a surface-level character n-gram metric with no learned parameters. To validate that ChrF++ serves as an effective proxy for Swahili language fidelity, we examined GlotLID Swahili confidence scores [9] during initial experiments and found that optimizing ChrF++ against a Swahili–Swahili reference produced a steady increase in GlotLID Swahili confidence over training steps. This indicates that surface-level character overlap with a Swahili reference is a proxy for target-language output.
Human evaluation protocol
We conduct a blind pairwise human evaluation to compare Fikiria-AfriqueQwen-14B against GPT-4o on Swahili mathematical reasoning. For each example, annotators are shown a Swahili math question together with two anonymized model outputs presented in randomized order. Each output consists of a step-by-step solution in Swahili and a final numeric answer. To minimize stylistic bias unrelated to reasoning quality, we normalize both outputs by removing model-specific formatting (e.g., XML-style tags) while preserving the underlying reasoning content and final answer.
We use two native Swahili-speaking researchers (co-authors) with STEM backgrounds as annotators. This choice reflects the practical constraint that evaluating mathematical reasoning in a low-resource language requires both domain knowledge and high language proficiency, which are difficult to obtain simultaneously. All evaluations are conducted in a fully blind pairwise setting with randomized output ordering and normalized formatting to remove model-specific cues. While using co-authors as annotators may introduce bias, we treat this evaluation as a qualitative assessment of reasoning quality rather than a large-scale benchmark. The annotators are based in different regions (Kenya and Tanzania), reflecting dialectal and stylistic variation in Swahili, which introduces additional subjectivity but better captures real-world variation in language use. This setup enables fine-grained assessment of reasoning faithfulness that would be difficult to obtain with non-expert annotators. Each example is independently annotated by both annotators. See Appendix § 8 for a screenshot of the data collection interface.
We report inter-annotator agreement alongside randomized, pairwise preference results. We emphasize that this human evaluation is intended as a qualitative complement to automatic metrics, rather than a standalone large-scale benchmark.
Annotators are asked to select which output is better overall, with three possible labels: , , or Tie. They are instructed to base this decision on four criteria: (i) correctness of the final answer, (ii) whether the intermediate reasoning steps causally support the answer, (iii) clarity and usefulness for a Swahili-speaking student, and (iv) naturalness of the Swahili, including avoidance of code-switching or unnatural phrasing. This protocol is designed to better match our target notion of faithful native-language reasoning than scalar fluency judgments alone.
After making the pairwise preference decision, annotators assign a single categorical tag indicating the primary reason for their choice. The tag set is: correctness, reasoning quality, clarity/helpfulness, language quality, or other. These tags allow us to distinguish whether preferences are driven by mathematical validity, reasoning faithfulness, pedagogical usefulness, or linguistic quality, rather than by overall style alone. The annotator is not required to provide a reason for a tie.
Before the main evaluation, annotators complete a short calibration phase on a separate set of examples in order to align interpretation of the criteria, especially the distinction between fluent Swahili and reasoning that genuinely supports the final answer. We then report pairwise win/loss/tie rates, category-level preference breakdowns, and inter-annotator agreement. In addition, we separately summarize common error types observed during annotation, including code-switching, fabricated intermediate steps, and correct final answers supported by incomplete or misleading reasoning.
Experiments
Experimental setup
Qwen3 models support an extended reasoning mechanism (enable_thinking=ON) that generates a native <think> block before the final response. Figure 3 shows the accuracy–efficiency tradeoff across Qwen3-family models on AfriMGSM Swahili. While thinking=ON yields marginally higher accuracy, it does so at 3–6× greater completion length with no consistent gain in Swahili language fidelity, and in several models reduces it. This is likely because extended <think> blocks introduce English reasoning that lowers GlotLID Swahili confidence. Since our GRPO training uses <rationale> tags to supervise reasoning directly, the native thinking mechanism is redundant and conflicts with our output format. We therefore use enable_thinking=False throughout all SFT, GRPO, and evaluation runs.

Figure 3. Effect of thinking=ON vs thinking=OFF on Qwen3-family models (AfriMGSM Swahili test, 250 examples). Marker size is proportional to active parameter count. Grey lines connect the same model under both conditions. Left: Accuracy vs. mean completion length (characters). thinking=ON yields marginally higher accuracy at 3–6× greater completion cost. Right: Accuracy vs. mean GlotLID Swahili confidence. thinking=OFF produces higher language identification confidence, an indicator for increased fidelity. We use thinking=OFF throughout to increase native language performance and reduce compute.
Figure 3 and Table 1 additionally show that accuracy is not tied to parameter count within the Qwen3 family. Qwen3-32B scores 32.0% zero-shot, below the smaller Qwen3-8B (40.8%) and Qwen3-14B (52.8%). One possible explanation, which we did not test, is that larger models trained predominantly on high-resource languages have stronger English-reasoning priors, while smaller models may be more flexible in following Swahili-language instructions. After GRPO alone, performance of Qwen3-32B is greater than the smaller models (68.8% vs. 53.6% for Qwen3-14B and 51.2% for Qwen3-8B), suggesting that task-specific optimization helps surface multilingual capability.
| Model | Size | Base | Instruct (baseline) | GRPO | Final (SFT+GRPO) |
| AfriqueQwen (African CPT) — AfriMGSM test set | |||||
| AfriqueQwen | 8B | 36.4 (±3.0) | 74.0 (±2.8) | ||
| AfriqueQwen | 14B | 27.2 (±2.8) | 79.2 (±2.6) | ||
| Qwen3 — AfriMGSM test set | |||||
| Qwen3 | 4B | 17.6 (±2.4) | |||
| Qwen3 | 8B | 40.8 (±3.1) | 51.2 (±3.2) | 57.6 (±3.1) | |
| Qwen3 | 14B | 52.8 (±3.2) | 53.6 (±3.2) | 65.6 (±3.0) | |
| Qwen3 | 32B | 32.0 (±3.0) | 68.8 (±2.9) | 70.8 (±2.9) | |
| Qwen3 | 30B-A3B (MoE) | 56.0 (±3.1) | |||
| Reference model, our run (same prompt, decoding and scorer as our models) | |||||
| GPT-4o (our prompt) | 68.0 (±3.0) | ||||
| Prior work: IrokoBench [1] on AfriMGSM test set{sup}‡` | |||||
| Aya-101 | 13B | 6.0 | |||
| Gemma 2 | 27B | 73.6 | |||
| Llama 3.1 | 70B | 68.0 | |||
| GPT-4o | 78.8 | ||||
| Prior work: TAPO [7] on MGSM Swahili test set{sup}†` | |||||
| Qwen2.5 | 3B | 13.2 | 49.0 | ||
| Llama3.2 | 3B | 27.1 | 72.1 |
Table 1. Swahili math accuracy (%) and standard error by model and training stage. AfriqueQwen, Qwen3, and IrokoBench results are reported on the AfriMGSM test set (250 examples). {sup}‡IrokoBench values are taken from Adelani et al. [2025]. They use the best-performing prompt per model and have no reported standard error; our models use a single fixed <rationale>/<answer> prompt. {sup}†TAPO values are taken from Huang et al. [2026], have no standard error here, and are measured on MGSM Swahili, so they are not directly comparable. GPT-4o (our prompt) is a single run at temperature 0 with the same prompt, decoding and strict scorer as our models; the IrokoBench GPT-4o value used a best-per-model prompt.
We focus on AfriqueQwen-8B and AfriqueQwen-14B [23] as our primary models. These models are part of a model family that has been adapted to 20 African languages from Qwen3-8B, Qwen3-14B (respectively) through continued pre-training and have good baseline performance in both Swahili and reasoning. We also compare the performance of the base models as well as larger variants in the same model family: Qwen3-32B, and Qwen3-30B-A3B (instruct, zero-shot). Finally, we also consider metrics from TAPO-ChrF++ [7], Qwen2.5-3B, Llama3.2-3B, and IrokoBench [1] (which includes frontier models). All local inference uses enable_thinking=False.
SFT and GRPO training were conducted on a cloud instance with 8×A100 80GB GPUs (96 vCPU, 1136GB RAM, 640GB VRAM) and 4×A100 instance (48 vCPU, 568GB RAM, 320GB VRAM). Total wall-clock training time ranged from 7–15 hours per model for GRPO stage. GRPO training used group size , temperature , maximum completion length 512 tokens, and learning rate . Models were fine-tuned with LoRA [6] (rank 32, , dropout 0.05) applied to query, key, value, output, gate, up, and down projection layers. Effective batch size was 48 prompts per update step (4 per device 4 gradient accumulation steps 3 processes), each generating 8 completions, for 384 total rollouts per step. Reward weights were and with no KL penalty (, DAPO asymmetric clipping [24]). Inference used greedy decoding () with a maximum of 512 tokens (enable_thinking=False and 4096-tokens for thinking=ON for results shown in Figure 3).
Main results
Figure 4 plots AfriMGSM Swahili accuracy against total parameter count. Fikiria-AfriqueQwen-14B reaches 79.2% (2.6). Under the same prompt, decoding and strict scorer, GPT-4o reaches 68.0% (3.0), partly because it often wraps answers in extra characters (for example <answer>$18</answer>), which the strict scorer marks wrong. The 78.8% reported for GPT-4o by Adelani et al. [ [1] used best-performing prompts per model and is not directly comparable. Fikiria-AfriqueQwen-14B also exceeds Gemma 2 27B (73.6%). Table 1 shows this result in the context of training stages and baselines. Zero-shot, AfriqueQwen-14B scores 27.2%, while Qwen3-14B instruct reaches 52.8% without any task-specific training. After SFTGRPO, Fikiria-AfriqueQwen-14B gains +52.0 points over its own base and +26.4 points over the Qwen3-14B instruct baseline. Under the same SFTGRPO pipeline, the AfriqueQwen backbones outperform their Qwen3 counterparts by +13.6 points at 14B (79.2% vs. 65.6%) and +16.4 points at 8B (74.0% vs. 57.6%), suggesting that African continued pre-training provides a stronger foundation for GRPO than the original instruction-tuned model. Fikiria-AfriqueQwen-8B reaches 74.0%, +37.6 points over its base.

Figure 4. Swahili math accuracy (AfriMGSM test, 250 examples) vs. total parameter count (log scale). Marker shape: circle = base, square = instruct, star = GRPO-trained. Thick edges = MoE (total parameters active parameters during inference). Color: orange = ours, blue = IrokoBench [1], purple = TAPO [7]. GPT-4o shown as blue dashed reference line (IrokoBench best-per-model prompt; parameter count undisclosed).
Fikiria-AfriqueQwen-14B (79.2%) scores higher than TAPO-ChrF++ on Llama3.2-3B (72.1%), which targets English reasoning and is evaluated on MGSM Swahili rather than AfriMGSM. The two results differ in benchmark, model size and backbone, so we do not attribute the difference to any single factor.
IrokoBench models [1] were evaluated with the best-performing prompt template per model; our models use a fixed <rationale>/<answer> prompt consistent with GRPO training, and prompt choice can meaningfully affect accuracy on AfriMGSM. TAPO [7] reports on MGSM Swahili rather than AfriMGSM; these points situate the training paradigm but are not directly comparable. Mixture-of-Experts (MoE; thick edges) are plotted at total parameter count; effective compute is closer to active parameter count (e.g. Qwen3-30B-A3B uses 3B active parameters per forward pass). Aya-101 (13B, mT5-XXL) scores 6%, consistent with its multilingual generalist design [1].
Ablation Studies
Table 1 shows GRPO-only and SFT+GRPO training on three Qwen3 models (single training run per cell). These ablations isolate the effect of cold-start SFT; they do not isolate the reward design (see limitations in § 5), and we did not run a GRPO-only ablation on AfriqueQwen.
SFT adds the most at 8B and 14B. Without it, GRPO must simultaneously discover the output format, Swahili instruction following, and reasoning quality, which may be harder at smaller scale. Qwen3-8B reaches 51.2% with GRPO alone (+10.4 points over the 40.8% instruct baseline) and 57.6% with SFT+GRPO (+16.8 points); the 6.4-point difference between the two is modest relative to the standard errors. The effect is clearer at 14B: GRPO alone improves on the 52.8% baseline by only 0.8 points (53.6%, within noise), while SFT+GRPO reaches 65.6% (+12.8 points, +12.0 over GRPO-only).
At 32B, GRPO alone accounts for nearly all of the improvement: Qwen3-32B reaches 68.8% (+36.8 points over its 32.0% baseline), and SFT adds only 2.0 points (70.8%), within one standard error. One possible explanation, which we did not test, is that the larger instruction-tuned model can acquire the output format during GRPO without cold-start SFT, whereas the smaller models benefit from SFT first.
Human evaluation
Following the protocol in § 3.6, we collected samples rated by both annotators (108 entries total). Assuming a Dirichlet(1,1,1) prior on , the posterior probability that GPT-4o’s true preference rate exceeds Fikiria’s is 0.999, with an estimated gap of 24.5 points (95% credible interval: +9 to +40). Inter-rater agreement was 41%, and the two annotators used the "Tie" label very differently (51% vs. 0%). The direction of the preference is therefore robust, but the magnitude should be treated as approximate.
Notably, Fikiria-AfriqueQwen-14B was judged as good as or better than GPT-4o on 30–60% of samples (depending on annotator) and outright preferred 11–30% of the time. The reasons given for each preference also differed significantly (, , ): Fikiria-AfriqueQwen-14B was preferred for better Swahili (12 out of 22 cases), while GPT-4o preferences split between clearer explanation and better reasoning (with better Swahili cited in only 10% of cases). Although GPT-4o is preferred overall, Fikiria-AfriqueQwen-14B was preferred most often for naturalness of Swahili, suggesting that its main advantage over GPT-4o lies in language quality rather than reasoning clarity. Given the small sample, the co-author annotators, and moderate agreement (41%), we treat these results as suggestive.
Conclusion
We present Fikiria, a two-stage SFT+GRPO recipe for Swahili mathematical reasoning whose reward multiplies a format gate by a ChrF++ term on the Swahili rationale and an exact-match accuracy term. Fikiria-AfriqueQwen-14B achieves 79.2% on AfriMGSM Swahili versus 68.0% for GPT-4o under the same prompt and strict scorer, while a small human evaluation favors GPT-4o’s reasoning overall. Its accuracy is also higher than the 72.1% reported for TAPO-ChrF++, though this value is measured on MGSM Swahili and is not directly comparable. Our results are consistent with the view that a surface-level reward on the reasoning trace can be optimized alongside accuracy, and that African CPT provides a stronger foundation in our setting than instruction tuning alone. A limited human evaluation further examines whether reasoning steps causally support the final answer, though on a small, moderately-agreeing sample (§ 4.4). Limitations include the small AfriMGSM evaluation set (250 examples), and restriction to a single language and task. ChrF++ is a surface-level metric of linguistic overlap with a reference translation; it does not verify that individual reasoning steps causally support the final answer, so our automated reward signal should be understood as targeting native-language fluency rather than step-level logical validity. Our experiments also do not ablate the reward design (ChrF++ versus accuracy-only, or the format gate versus a plain weighted sum), so the contribution of each reward component is not isolated; all trained results are from a single training run per configuration; we did not run a GRPO-only ablation on AfriqueQwen; and because AfriMGSM is derived from GSM8K, which is possibly present in the pre-training data of the base models, some of the reported accuracy may reflect contamination rather than reasoning.
Causal support between reasoning steps and the final answer is instead assessed qualitatively by human annotators on a small sample (N=54, § 4.4); inter-annotator agreement on this sample was moderate (41%), so these judgments should be treated as suggestive rather than a validated measure of faithfulness at scale. Our training data is constructed with machine translation, which may introduce systematic artifacts, mistranslations near the quality threshold, and biases inherited from the source model (GPT-4.1); downstream effects on reasoning quality are difficult to fully characterize. Human evaluation is small-scale and inherently subjective, with annotator disagreement reflecting both dialectal variation and the difficulty of judging reasoning quality in a low-resource language. Future work includes extending to other African languages, applying the gated reward framework to other low-resource reasoning tasks, developing automated metrics that more directly assess step-level causal validity, and scaling the human evaluation protocol.
Acknowledgments and Funding
This work is based on research conducted by Institute for Disease Modeling, a research group within, and solely funded by, the Gates Foundation. The funders had no role in the analysis, or interpretation of the data; the preparation, review, or approval of the report; or the decision to submit the manuscript for publication. Some authors are employees of the Gates Foundation, however, this study does not necessarily represent the views of the Gates Foundation.
Dataset statistics and SSA-COMET-MTL score distribution
Figure 5 shows SSA-COMET-MTL QE scores for GPT-4.1 (gpt-4.1-2025-04-14) translations of GSM8K English-to-Swahili questions and CoT for the train split. Distributions between the train and test splits were indistinguishable to a KS test. The distributions between translated questions and CoT are significantly different (, KS-statistic = 0.31). Filtering for scores greater than 0.65 reduces the test split to 416 (out of 1319) entries and the train split to 2308 (out of 7473).
Figure 6 compares the SSA-COMET-MTL scores for GPT-4.1 translations compared to the AfriMGSM test split and a second-order polynomial regression [22]. GPT-4.1 tends to score higher than AfriMGSM, particularly for entries with lower scores.
Figure 7 shows the results of the human spot check of the quality of the generated text. Both annotators were given the same set of 20 question / answer pairs of GPT-4.1 translations to evaluate. Completion rates were 20/20 and 9/20. The annotators were asked to score the translation on a range of 0-100 but the resulting scalings were quite different (90 to 100 vs 70 to 100). However, when we look at the overall distribution of human scores to the SSA-COMET-MTL score we see a broad pattern of agreement without threshold. This is indicated by how a simple linear regression on the scores intersects the 0.65 threshold at a point that bisects both samples.
Taken together, this spot check corroborates the choice of 0.65 as a reasonable delineation for high-quality translation although the signal is still noisy. We prefer to err on the side of high-quality since errors will negatively impact our training objectives.
Human evaluation example
Figure 8 shows a screenshot of the tool used for human evaluation and described in § 3.6. Formatting (e.g., newlines) was removed from the text blocks and both annotators saw the same set of sample pairs drawn from the AfriMGSM test split. We used GPT-4o (gpt-4o-2024-08-06) for our comparison to Fikiria-14B.

Figure 8. Screenshot of the tool used for our human evaluation (see § 3.6).
Translation prompt templates
Translation System Prompt
You are a precise translation system.
Your task:
Translate the provided QUESTION and CHAIN-OF-THOUGHT (CoT) into the target
language.
Requirements:
- Preserve meaning exactly.
- Do NOT summarize.
- Do NOT explain.
- Do NOT add commentary.
- Preserve logical structure of the reasoning.
- Output must strictly follow the specified format.
Translation Prompt Template
Translate the following content into {TARGET_LANGUAGE}.
You must:
- Translate faithfully.
- Preserve reasoning structure.
- Keep formatting identical.
- Output ONLY the translated result.
- Do not improve, refine, or correct the reasoning.
- Translate literally while preserving logical form.
FORMAT REQUIREMENTS:
<TRANSLATED_QUESTION>
...
</TRANSLATED_QUESTION>
<TRANSLATED_COT>
...
</TRANSLATED_COT>
{SHOTS}
NOW TRANSLATE THE FOLLOWING:
Input:
<QUESTION>
{QUESTION}
</QUESTION>
<COT>
{COT}
</COT>
Shot Template
EXAMPLE {N}
Input:
<QUESTION>
{QUESTION}
</QUESTION>
<COT>
{COT}
</COT>
Output:
<TRANSLATED_QUESTION>
{TRANSLATED_QUESTION}
</TRANSLATED_QUESTION>
<TRANSLATED_COT>
{TRANSLATED_COT}
</TRANSLATED_COT>
Evaluation prompt template
Prompt template
You are a helpful assistant that solves math word problems.
Think through the problem step by step in {language_name}, showing your calculations.
Put your reasoning inside <{rationale_tag}>...</{rationale_tag}> tags and your final numeric answer inside <answer>...</answer> tags.References
- [1]David Ifeoluwa Adelani et al. IrokoBench: A new benchmark for african languages in the age of large language models. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics, 2025.
- [2]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI Feedback, December 2022. URL http://arxiv.org/abs/2212.08073. arXiv:2212.08073 [cs].
- [4]Nuo Chen, Zinan Zheng, Ning Wu, Linjun Shou, Ming Gong, Yangqiu Song, Dongmei Zhang, and Jia Li. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.
- [5]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.arxiv.org/abs/2110.14168
- [6]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685.
- [7]Xu Huang, Zhejian Lai, Zixian Huang, Jiajun Chen, and Shujian Huang. TAPO: Translation-augmented policy optimization for multilingual mathematical reasoning, 2026. URL https://arxiv.org/abs/2603.25419.
- [8]Jaedong Hwang, Kumar Tanmay, Seok-Jin Lee, Ayush Agrawal, Hamid Palangi, Kumar Ayush, Ila Fiete, and Paul Pu Liang. Learn globally, speak locally: Bridging the gaps in multilingual reasoning, 2025. URL https://arxiv.org/abs/2507.05418.
- [9]Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. GloT LID: Language Identification for Low-Resource Languages. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.410. URL https://aclanthology.org/2023.findings-emnlp.410/.DOI
- [10]Sen Yu Li, Jiayi Wang, Felermino D. M. A. Ali, Colin Cherry, Daniel Deutsch, Eleftheria Briakou, Rui Sousa-Silva, Henrique Lopes Cardoso, Pontus Stenetorp, and David Ifeoluwa Adelani. SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages? In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12979–12998, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.656. URL https://aclanthology.org/2025.emnlp-main.656/.DOI
- [11]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step, May 2023. URL http://arxiv.org/abs/2305.20050. arXiv:2305.20050 [cs].
- [12]Junxiao Liu, Zhijun Wang, Yixiao Li, Zhejian Lai, Liqian Huang, Xin Huang, Xue Han, Junlan Feng, and Shujian Huang. Self-Improving Multilingual Long Reasoning via Translation-Reasoning Integrated Training, 2026. URL https://arxiv.org/abs/2602.05940. arXiv:2602.05940 [cs.CL].
- [13]Eric Onyame, Akash Ghosh, Subhadip Baidya, Sriparna Saha, Xiuying Chen, and Chirag Agarwal. CURE-Med: Curriculum-informed reinforcement learning for multilingual medical reasoning, 2026. URL https://arxiv.org/abs/2601.13262.
- [14]Jan-Thorsten Peter, David Vilar, Tobias Domhan, Dan Malkin, and Markus Freitag. Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results, November 2025. URL http://arxiv.org/abs/2511.05162. arXiv:2511.05162 [cs].
- [15]Maja Popović. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark, 2017.DOI
- [17]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024.arxiv.org/abs/2402.03300
- [18]Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. MAPO: Advancing multilingual reasoning through multilingual alignment-as-preference optimization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
- [19]Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2210.03057.
- [20]Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking, 2025. URL https://arxiv.org/abs/2209.13085.
- [21]Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformers Reinforcement Learning, 2020. URL https://github.com/huggingface/trl.
- [22]Michael L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60): 3021, 2021. doi: 10.21105/joss.03021. URL https://doi.org/10.21105/joss.03021.
- [23]Hao Yu, Tianyi Xu, Michael A. Hedderich, Wassim Hamidouche, Syed Waqas Zamir, and David Ifeoluwa Adelani. AfricanLLM: How data mixing and model architecture impact continued pre-training for african languages, 2026. URL https://arxiv.org/abs/2601.06395.
- [24]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale, March 2025. URL https://arxiv.org/abs/2503.14476. arXiv:2503.14476 [cs.CL].
- [25]Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Kaiyu Huang, Yufeng Chen, Xu Jinan, and Jie Zhou. Thinking natively: Unlocking multilingual reasoning with consistency-enhanced reinforcement learning, July 2026. URL https://aclanthology.org/2026.acl-long.951/.