betaFeedback

WorksComputer Science

CamSpokenNLI: A Pilot Resource for Cross-Variety Transfer Evaluation in Cameroon Pidgin English and Camfranglais

Abstract

Cameroon Pidgin English (a West African Pidgin English variety) and Camfranglais are widely used in Cameroon, yet to our knowledge, neither has a dedicated spoken Natural Language Inference (NLI) resource. We introduce CamSpokenNLI, a pilot dataset of 114 scenario-matched spoken NLI items in each variety (228 in total), recorded by one native speaker from Buea. Each item pairs premise audio and text with a hypothesis, an NLI label, and English glosses. The author assigned labels, so the dataset is an exploratory resource rather than a mature benchmark. The NLI experiments use the reference text; the audio is evaluated separately. XLM-RoBERTa baselines with five-fold item-level cross-validation reach 44.7% on Pidgin and 46.2% on Camfranglais (chance 33.3%). Still, neither is statistically distinguishable from hypothesis-only input, so hypothesis wording may account for much of the accuracy. Under disjoint-item cross-variety transfer, where a test item's counterpart is never seen in training, accuracy is 40.1% (Pidgin to Camfranglais) and 45.6% (Camfranglais to Pidgin). A counterpart-leak audit raises this to 60-64%, illustrating how counterpart leakage can overstate transfer. Zero-shot Whisper transcription of the recordings performs poorly: the best word error rates are 66.6% for Pidgin and 45.5% for Camfranglais, and with automatic language identification, Whisper Large-v3 labels 93 of 114 Pidgin recordings as Yoruba or Swahili. Together, the results suggest that such resources need disjoint-item evaluation and hypothesis-only controls, and that speech recognition must be adapted to these varieties before spoken NLI can be tested end to end.

Introduction

In Cameroon, Pidgin English (Kamtok, Wes Cos; ISO 639-3: wes) and Camfranglais, which is a French–English–Pidgin contact variety shaped by local languages, are mainly spoken rather than written [9, 17]. A client who needs an electrician is more likely to say “light don go fo haus” (“the power has gone off at home”) than to type it out. Yet systems capable of reasoning over such utterances remain limited, particularly for Natural Language Inference (NLI), which determines whether a statement entails, contradicts, or is neutral with respect to another statement.

A large speaker population does not guarantee adequate computational coverage. Cameroon Pidgin is spoken in some form by an estimated half of Cameroon’s population (Lewis et al., [14]). However, its speakers are hard to count because competence ranges from native use to a trade language (Schröder, [20]). It belongs to the wider West African Pidgin English (WAPE) cluster, which Yakpo [23] estimates reaches up to 140 million speakers across six nations (Nigeria, Cameroon, Ghana, Sierra Leone, Equatorial Guinea, and The Gambia). Within this cluster, Nigerian Pidgin has benefited from growing computational resources for tasks such as sentiment analysis (Muhammad et al., [15]) and translated NLI data (Kondoro et al., [12]). Still, resources for a related regional variety cannot replace direct evaluation of Cameroon Pidgin, Camfranglais, and related Cameroonian contact varieties.

Large English NLI datasets such as SNLI (Bowman et al., [4]) and MultiNLI (Williams et al., [22]) exist, and recent efforts have extended NLI-style resources to African languages through translation of XNLI (Adelani et al., [1]; Kondoro et al., [12]). Audio-premise entailment has also begun to receive attention, for non-speech audio (Deshmukh et al., [7]) and for African-accented English (Okocha and Grant, [16]). Despite these developments, our review of the published literature and publicly available resources did not identify a dedicated NLI dataset for Cameroon Pidgin English or Camfranglais, nor a spoken-premise NLI resource for either variety. To address this gap, we introduce CamSpokenNLI, a spokenpremise NLI resource for Cameroon Pidgin and Camfranglais, and evaluate it with text-based baselines and zero-shot speech recognition.

Motivation

The need is practical as well as scientific, especially for voice-based applications. CamSpokenNLI was developed alongside LocalHands Africa, a voice-first service-exchange platform in which a spoken request such as “light don go fo haus” must be matched to an artisan skill like electrical work, whichever variety the speaker uses. Entailment offers one way to test such matching: a system should be able to judge that “The speaker needs an electrician” can reasonably be inferred from the spoken request. We disclose this connection explicitly; see the Ethics Statement.

Contributions

The contributions of this work are as follows:

(1) We introduce CamSpokenNLI, a pilot spoken-premise NLI resource containing 114 manually annotated, scenario-matched items in each variety. To the best of our knowledge, it is the first dedicated NLI resource for either variety and the first with spoken premises in these varieties.

(2) We report XLM-RoBERTa baselines under five-fold item-level cross-validation, together with a hypothesis-only analysis. The XLM-RoBERTa accuracy is not statistically distinguishable from the hypothesis-only baselines, so hypothesis wording may account for much of it.

(3) We provide a cross-variety transfer study on disjoint items, and an audit illustrating how counterpart leakage can inflate transfer accuracy.

(4) We assess audio-grounded robustness for both varieties by transcribing the premise recordings with zero-shot Whisper (Base, Small, and Large-v3) under automatic, forced-English, and forced-French decoding, analyzing failure patterns such as language misidentification and lexical substitution. This evaluation is separate from the NLI baselines, which use the reference text only.

(5) We release the recording protocol, the complete dataset, and reproducible Colab notebooks so that every NLI run, prediction, and Whisper transcript can be inspected by the community.

CamSpokenNLI is too small for a standard benchmark; we frame it as a seed resource whose protocol and baselines can guide larger-scale collection by the broader African NLP community.

Related Work

We organize prior work by relevance to spoken Cameroonian NLI, then distinguish the closely related Nigerian Pidgin resources from Cameroon Pidgin English and Camfranglais.

Prior Work in NLI and African NLP

  • Foundational English NLI: SNLI (Bowman et al., [4]) and MultiNLI (Williams et al., [22]) established large-scale NLI resources for English. XNLI (Conneau et al., [6]) subsequently extended multilingual NLI evaluation to English and 14 other languages.

  • Annotation artifacts: On English NLI datasets, classifiers that see only the hypothesis can score well above the majority-class baseline, which indicates that some labels can be predicted from hypothesis wording alone ([11, 18]). This motivates the hypothesis-only and premise-only diagnostics we run.

  • Multilingual NLI: XLM-RoBERTa (Conneau et al., [5]), pretrained on CommonCrawl text in 100 languages, provides a widely used multilingual encoder for low-resource transfer settings; Cameroon Pidgin English and Camfranglais are not identified as dedicated pretraining languages.

  • African NLU resources: Community-driven efforts have expanded computational resources for African languages, including a Twitter sentiment corpus that covers Nigerian Pidgin (Muhammad et al., [15]).

  • NLI for African languages: IrokoBench (Adelani et al., [1]) includes AfriXNLI, a human translation of XNLI into African languages; Afri-XNLI+ (Kondoro et al., [12]) adds further translated XNLI pairs, including Nigerian Pidgin. These resources are text-only translations of English items. Our review did not identify an NLI resource for Cameroon Pidgin English or Camfranglais, nor one built from original, spoken Cameroonian material.

  • Speech and entailment: A spoken corpus of Cameroon Pidgin English [17] provides recordings and transcripts, including data from Buea, for linguistic study; however, it lacks inference labels. In the domain of reasoning, Audio Entailment [7] pairs general (non-speech) audio premises with text hypotheses, while Afrispeech Semantics [16] evaluates entailment on African-accented English. To our knowledge, these entailment studies do not cover Cameroon Pidgin English or Camfranglais.

  • Sociolinguistic research: Both varieties are well documented, including work on Cameroon Pidgin English [3, 17] and on the structure and use of Camfranglais [13, 21]. The gap addressed here is therefore computational rather than descriptive, particularly for inference tasks such as NLI over spoken premises.

  • Creole-language NLI precedent: Outside Africa, JamPatoisNLI [2] provides a precedent for constructing NLI resources for English-lexified creole languages. Comparable resources for African pidgin and contact varieties remain scarce, especially spoken Cameroonian NLI data.

Translated benchmarks make results comparable across languages, but a translation inherits the wording of its English source and may miss locally used expressions. For contact varieties, whose vocabulary and constructions mix several source languages, items written directly in the variety can differ from translated ones. CamSpokenNLI takes this route: premises and hypotheses are written in each variety, and the premises are recorded.

Distinctions from Nigerian Pidgin Resources

Existing Nigerian Pidgin resources (such as NaijaSenti and Afri-XNLI+) cannot stand in for Cameroon Pidgin English. The two Pidgins belong to the same WAPE cluster [23], but they differ in ways that matter for speech and NLI data.

  • Contact languages: As contact languages, both Pidgins are English-lexified, yet they diverge in their substrate influences. While Nigerian Pidgin primarily incorporates Benue-Kwa languages of the Niger-Congo family [10], Cameroon Pidgin English draws significantly on French and indigenous Cameroonian languages [20]. Earlier estimates reported by Echu [8] put the Cameroon Pidgin English lexicon at roughly 80–85% English and 13–14% indigenous loanwords, with French contribution rising to approximately 2–5%.

  • Grammar: The imperfective/progressive marker is typically di in Cameroon Pidgin English [20], whereas Nigerian Pidgin uses dè (or dey) [10]. Examples include “How you di do?” (Cameroon) and “How you dey?” (Nigeria).

  • Camfranglais: Unlike Nigerian Pidgin, Camfranglais is a distinct mixed variety rather than an English-lexified pidgin, and it is also separate from Cameroon Pidgin English. Its lexicon comes from French, English, West African Pidgin, Cameroonian languages, Latin, and Spanish [13]. To date, no existing Nigerian Pidgin resource covers it, so it needs its own evaluation data.

These differences motivate treating Cameroon Pidgin English and Camfranglais as distinct computational targets; conflating them would obscure the unique Francophone and indigenous lexical integration central to the Cameroonian context. We reference Nigerian Pidgin to situate our work within the broader landscape of West African creoles, serving as linguistic context rather than an experimental baseline. Consequently, § 4.3 evaluates cross-variety transfer directly between the two Cameroonian varieties.

Dataset Construction

This section describes how the first release (v1.0) was constructed.

Design

Each sample is a tuple:

(id,premise_text,premise_text_en,premise_audio, hypothesis_text,hypothesis_text_en,label)

where label is one of the three NLI classes (entailment, contradiction, or neutral). The _en fields are English glosses of the premise and hypothesis, included so that readers who do not speak Cameroon Pidgin English or Camfranglais can inspect the data; they are translations of the original material, not independently authored English NLI examples. Labels are balanced within each variety: 38 entailment / 38 contradiction / 38 neutral. Labels follow a fixed rule: a hypothesis is entailment if it must be true whenever the premise is true, contradiction if the two cannot both be true, and neutral otherwise. All premises are declarative sentences; questions and imperatives are not used.

The dataset is organized as two variety subsets that follow the same schema. They are scenario-matched: the same 114 scenarios are represented in both varieties, with each variety providing its own sentences and recording, and corresponding items sharing the same numeric item identifier (wes_NNNNN / camf_NNNNN). Labels agree across all 114 corresponding items, and the English premise and hypothesis glosses are identical across the two files. Each subset keeps its own annotations and audio, so either variety can be inspected or extended independently. All 114 premises and all 114 hypotheses are unique within each variety.

Sentence Creation

Sentences reflect daily life in Buea/Molyko and cover the everyday service domains where voice-first requests arise, such as electricity, water, transport (bike/moto), markets, school, and financial transactions. They were drawn from the author’s everyday experience in these settings and resemble everyday exchanges, for example, with friends or a motorbike rider. Each premise was recorded in the target variety with a hypothesis already in mind that stood in a specific relation to it (entailment, contradiction, or neutral), and the label was fixed at that point. The premise recordings were then transcribed, the hypotheses were written as text, and English glosses were added afterward as translations of both. Only premises are recorded in the release; hypotheses are text only. The author wrote all premises and hypotheses for this dataset. Premises are short single sentences averaging 9.2 words (Pidgin) and 9.5 (Camfranglais); hypotheses average 6.9 and 7.4 words, respectively, and vary little by label (Appendix § 17).

The Pidgin items were produced first. For Camfranglais, the author expressed the same 114 scenarios in his everyday Camfranglais, with his own sentences and recordings, transcribed the same way and matched to the same English gloss. The Camfranglais sentences are therefore not word-for-word translations of the Pidgin ones: the mean word overlap between counterpart premises, measured as the Jaccard similarity of their word sets, is about 8%, and no pair is identical (Appendix § 17). Because each pair of items describes the same situation under the same label, the two subsets are scenario-matched rather than independent. A test item’s counterpart in the other variety shares its meaning and label, so cross-variety transfer is evaluated with disjoint-item splits (§ 4.1).

Example (Cameroon Pidgin English):

Premise / GlossHypothesis / GlossLabel
light no de molyko fo two days na so. (The power has been off in Molyko for two days.)molyko no get light. (Molyko doesn’t have electricity.)entailment
papa de fo haus di wait you. (Dad is home right now waiting for you.)papa de fo workplace. (Dad is at work.)contradiction
a jus take salary na na so. (I’ve just received my salary.)a go pay school fees wit de moni. (I’ll pay school fees with the money.)neutral

Table 1.

Example (Camfranglais):

Premise / GlossHypothesis / GlossLabel
la light est off à Molyko depuis deux jours. (The power has been off in Molyko for two days.)Molyko n’a pas la light. (Molyko doesn’t have electricity.)entailment
le vieux est au piol entrain de te wait. (Dad is home right now waiting for you.)le vieux est au work. (Dad is at work.)contradiction
mon salaire vient de tomber. (I’ve just received my salary.)je vais use les doo la pour pay les school fees. (I’ll pay school fees with the money.)neutral

Table 2.

Audio Protocol and Pipeline Overview

Each premise was recorded as one spoken sentence. Figure 1 shows how the data are used: NLI classification with XLM-RoBERTa-base takes the premise and hypothesis text as input, while the premise audio is used only in a separate speech check, where zero-shot Whisper transcribes it, and the transcript is scored against the premise text (§ 4.4). The recording protocol was as follows:

Overview of the two separate paths in this paper

Figure 1. Overview of the two separate paths in this paper. Top: NLI is run on the reference text of the premises and hypotheses with XLM-RoBERTa-base. Bottom: the premise audio is transcribed by zero-shot Whisper and scored against the premise text. The results are in § 4.3 (NLI) and 4.4 (speech check). Neither the audio nor the ASR transcripts are used as NLI input.

  • Format: WAV, 16 kHz, mono, 16-bit PCM

  • Length: 2.0–6.8 s per clip after trimming (mean 3.7 s for Pidgin, 3.4 s for Camfranglais), one sentence per file

  • Natural conversational pace (not read-aloud/news style), consistent volume

  • 0.1 s of silence before and 1.0 s of silence after the speech

  • Quiet room, phone ∼15 cm from mouth, minimal background noise

  • Conversion with ffmpeg in a Colab notebook (.m4a → .wav, 16 kHz, mono)

Splits and Class Balance

The dataset contains 114 examples per language variety and is released with a single training split. No dedicated development or held-out test partition is included. This design reflects the early stage of the release: the current objective is to establish a reproducible resource and baseline rather than a mature benchmark for held-out evaluation. Each language subset is balanced across the three NLI classes (38 entailment, 38 contradiction, 38 neutral).

Label distribution in each variety subset

Figure 2. Label distribution in each variety subset: 114 samples, balanced across entailment, contradiction, and neutral (38 each). Chance level: 33.3%.

Predicting one label for every item is correct for exactly one third (38 of 114), the chance level in the exact binomial tests. For the experiments, the 114 items are divided once into five stratified folds (22–23 items per fold, 7–8 per class). Both varieties share the same fold assignment: item ii is in the same fold in Pidgin and in Camfranglais. It is also fixed across all random seeds and all conditions, so seeds change only model initialization and training order. In each run, one fold is held out for testing, and the other four are used for training, so across the five runs every item is held out exactly once. In cross-variety transfer, the model is trained on one variety’s items outside the held-out fold and tested on the other variety’s items inside it, so a test item and its counterpart are never in the training set. The folds were generated once, with seed 42, and were never redrawn. The full fold item-level assignment is given in Appendix § 16.

Positionality Statement

All recordings are by a single speaker (author, male, native speaker of both varieties, based in Buea). This limits regional and demographic coverage. It is also a strength: sentences and labels reflect the author’s firsthand native-speaker judgment and lived experience of both varieties, rather than data elicited from third parties. The same person wrote, recorded, and labeled every item, so no inter-annotator agreement is available. The two subsets are scenario-matched and share the same 114 scenarios, so they are not fully independent.

Experiments and Results

We run two separate evaluations (Figure 1): NLI on the reference text of the premises and hypotheses (§ 4.1–§ 4.3), and a zero-shot Whisper check on the premise audio (§ 4.4). Each gives its setup and then its results; we then interpret the two together in § 4.5.

NLI Experimental Setup

Approach. We report a text-based NLI baseline using XLM-RoBERTa (Conneau et al., [5]) on the reference text of the premises and hypotheses. The task is a three-way classification: entailment, contradiction, or neutral. Because the dataset contains only 114 examples per variety and has no dedicated development or test partition, we evaluate with stratified five-fold cross-validation over items rather than a held-out test set.

Model and training. We fine-tune xlm-roberta-base with the following configuration (full details in Appendix § 10):

  • 15 training epochs, batch size 4, learning rate 2×10−52 \times10^{-5};

  • maximum sequence length 64;

  • random seeds 42, 123, and 2024.

Training used AdamW with a linear learning-rate schedule, weight decay 0.0, and fp16 mixed precision, with an evaluation batch size of 8, on a single NVIDIA Tesla T4 GPU in Google Colab.

Epochs and tuning. We use 15 epochs as the main setting and report 5, 10, and 20 epochs as a sensitivity analysis on the same data (Appendix § 14). The learning rate (2×10−52 \times10^{-5}) and batch size (4) were set in early exploratory runs and were not changed after the reported results were seen. No early stopping or checkpoint selection was used, so predictions come from the final epoch. Because there is no separate development set, the main accuracies may be slightly optimistic.

Conditions. We compare four input conditions:

  • Native: the premise and hypothesis in the variety itself.

  • English gloss: the same model and training setup applied to the English glosses.

  • Hypothesis-only: the model sees only the hypothesis and not the premise.

  • Premise-only: the model sees only the premise and not the hypothesis.

The glosses are shared by the two varieties and are translations of the original material, not an independently authored English NLI corpus. The English gloss condition is therefore a single approximate reference condition, not a matched English benchmark for comparison.

The hypothesis-only and premise-only conditions are diagnostics: they test whether the label can be predicted from only one of the two texts. Hypothesis-only baselines are a standard check for annotation artifacts in NLI data (Gururangan et al., [11]; Poliak et al., [18]). The hypothesis-only model is also run in the transfer setting; premise-only transfer was not run.

Cross-variety transfer. Transfer is evaluated on disjoint items. For each fold kk, the model is trained on source-variety items outside fold kk (91–92 items) and tested on target-variety items in fold kk (22–23 items). The two subsets are scenario-matched, so the counterpart of a test item expresses the same scenario and label. Because the folds are shared by item number, the counterparts of the test items fall in fold kk and are excluded from training, so every test scenario is unseen. Within-variety and transfer results therefore use the same test items and the same number of training items, and differ only in the variety of the training data.

Counterpart-leak audit. We additionally train on all 114 source-variety items and test on all 114 target-variety items. Here the counterpart of every test item is seen in training, so this setting is not a measure of transfer. We report it to illustrate how much leakage can inflate accuracy. It also trains on more items (114 instead of 91–92), so the gain over disjoint-item transfer cannot be attributed to leakage alone.

NLI Evaluation Protocol

What we report. For each experiment and seed, we report two accuracies:

  • the mean five-fold accuracy, with its standard deviation across folds;

  • the pooled accuracy over the 114 predictions made across the five folds, with the correct/total count.

The pooled accuracy is the value used in the main tables (where ±\pm is the standard deviation over seeds) and in the statistical tests.

Accuracy. Accuracy is the proportion of correct predictions among the evaluated examples. For seed ss, the mean five-fold accuracy is

As=15∑k=15As,kA_s = \frac{1}{5} \sum_{k=1}^{5} A_{s,k}

where As,kA_{s,k} is the accuracy on fold kk. The pooled accuracy is

Ps=1114∑k=15Cs,kP_s = \frac{1}{114} \sum_{k=1}^{5} C_{s,k}

where Cs,kC_{s,k} is the number of correct predictions on fold kk. The two differ slightly because the folds hold 23, 23, 23, 23, and 22 items. Chance level for three balanced classes is 33.3%.

Seeds. The three seeds are repeated runs on the same 114 examples, so they measure variation from initialization and training order, not independent evidence. We summarize them descriptively with the sample mean and standard deviation across them; the population standard deviation is given in brackets in the appendix tables. Summary statistics are computed from the underlying fold-level and item-level outputs before rounding, so recomputing from the rounded appendix values may not reproduce the displayed summaries exactly.

Above-chance test. We apply a one-sided exact binomial test of each seed’s pooled correct count against a success probability of 1/31/3, and report how many of the three seeds reach p<0.05p < 0.05. A score of 47/114 is the smallest that passes (Appendix § 15).

Paired comparison. The question is whether the premise changes accuracy relative to the hypothesis alone. We compare the full-input and hypothesis-only models on the same items with an exact two-sided McNemar test, separately for each seed. This covers the two within-variety settings and the two transfer directions, for 12 comparisons in total (Appendix § 15).

How to read the results. At this sample size, a single accuracy has about a 95% interval of roughly ±\pm9 points, so differences of a few points between conditions are within noise. “Not significantly different” should not be read as “equivalent.” Because the seeds share the same items, the count of seeds above chance describes consistency across runs, not independent replication. No correction for multiple comparisons is applied, so the pp-values should be read as descriptive.

NLI Results

Table 1 summarizes the main NLI results in a single train/test matrix. The diagonal entries are within-variety five-fold cross-validation, and the off-diagonal entries are disjoint-item cross-variety transfer. Table 2 adds the hypothesis-only and premise-only diagnostics. The text below reports the within-variety results (including the English-gloss reference), cross-variety transfer, the diagnostics, and the counterpart-leak audit, in that order.

Train dataTest: Cameroon PidginTest: Camfranglais
Cameroon Pidgin only44.7% ±\pm 2.3%40.1% ±\pm 3.3%
Camfranglais only45.6% ±\pm 3.5%46.2% ±\pm 1.8%
English glosses (reference)61.1% ±\pm 4.3%61.1% ±\pm 4.3%
Chance level (3-class)33.3%33.3%

Table 1 (PDF p. 7). NLI baseline and cross-variety transfer results

Note. Diagonal entries: pooled five-fold accuracy (114 predictions per seed). Off-diagonal entries: disjoint-item transfer, so no test item or its counterpart is seen in training. Mean ±\pm sample SD over seeds 42, 123, and 2024. The English-gloss row is a shared reference, not a cross-variety experiment.

SettingFull inputHypothesis-onlyPremise-onlyMcNemar significant
Pidgin, within44.7 ±\pm 2.341.8 ±\pm 1.837.1 ±\pm 1.80 of 3
Camfranglais, within46.2 ±\pm 1.847.1 ±\pm 5.038.3 ±\pm 5.80 of 3
Pidgin → Camfranglais40.1 ±\pm 3.344.2 ±\pm 5.6–0 of 3
Camfranglais → Pidgin45.6 ±\pm 3.548.8 ±\pm 2.7–0 of 3

Table 2 (PDF p. 7). Accuracy (%) with full input compared with hypothesis-only and premise-only input

Note. Pooled accuracy, mean ±\pm sample SD over three seeds. “McNemar significant.” is the number of seeds in which full and hypothesis-only input differ (exact McNemar test, p<0.05p < 0.05). Premise-only transfer was not run. Per-seed values and tests are in Appendices § 11, § 13 and § 15.

Within-variety results. Cameroon Pidgin English obtained a descriptive mean accuracy of 44.7% ±\pm 2.3% and Camfranglais 46.2% ±\pm 1.8%. The 1.5-point gap between the varieties is within seed variation, so we do not rank them by difficulty.

The English glosses obtained 61.1% ±\pm 4.3%. All three (Pidgin, Camfranglais, gloss) are above the 33.3% chance level in every seed (one-sided exact binomial test, p<0.05p < 0.05 in 3 of 3 seeds), which indicates a learnable signal under the evaluated setup (see the diagnostics below).

Accuracy rises most from 5 to 10 epochs, and from 15 to 20 epochs it changes by +1.2 points (Pidgin) and +2.3 points (Camfranglais), which is within the seed-to-seed variation (Appendix § 14).

English gloss reference. The English reference is about 15 to 16 points higher than the within-variety means. This comparison is treated with caution: the glosses are translations of the original material, so the result does not establish a language-specific representation gap in XLM-RoBERTa.

Cross-variety transfer. A model trained on Pidgin reached 40.1% ±\pm 3.3% on Camfranglais (above chance in 1 of 3 seeds), and a model trained on Camfranglais reached 45.6% ±\pm 3.5% on Pidgin (above chance in 3 of 3 seeds). We therefore read transfer into Pidgin as above chance, though not above hypothesis-only transfer, and transfer into Camfranglais as suggestive only (one-sided binomial tests per seed; see § 4.2).

These runs share test items and training-set size with the within-variety runs, so the two can be compared directly:

  • On Pidgin items, the Camfranglais-trained model (45.6%) is about as accurate as a model trained on Pidgin itself (44.7%).

  • On Camfranglais items, the Pidgin-trained model is 6.1 points below a model trained on Camfranglais itself (40.1% against 46.2%).

With three seeds and 114 items, we do not claim that transfer across varieties is asymmetric.

Hypothesis-only and premise-only diagnostics. A hypothesis-only model reaches 41.8% (Pidgin) and 47.1% (Camfranglais) within variety, close to the full-input models (44.7% and 46.2%). Paired exact McNemar tests find no significant difference between full and hypothesis-only input in any of the 12 seed-level comparisons (smallest p=0.212p = 0.212). The full-input model is higher in 6 comparisons and lower in 6, and the mean differences range from +2.9 to −4.1-4.1 points.

The transfer results show the same pattern: hypothesis-only transfer (44.2% and 48.8%; Table 2) is at least as accurate as full-input transfer (40.1% and 45.6%). Premise-only models stay near chance (37.1% and 38.3%; above chance in 0 and 1 of 3 seeds), so the premise alone carries little of the label. One possible cue is negation: no neutral hypothesis in either variety contains a negation word, whereas 16–29% of the entailment and contradiction hypotheses do (Appendix § 17). This does not by itself explain the hypothesis-only accuracy.

Counterpart-leak audit. When the model is trained on all 114 source items and tested on all 114 target items, accuracy is 63.7% ±\pm 5.3% (Pidgin →\to Camfranglais) and 60.5% ±\pm 8.0% (Camfranglais →\to Pidgin). These are about 24 and 15 points above disjoint-item transfer in the same directions. The gain is positive in all six seed pairs, between 5 and 31 additional correct predictions per seed. Because the counterpart of every test item has been seen in training, these values reflect leakage and a larger training set (114 instead of 91–92 items) as well as any transfer, and should not be read as evidence of cross-variety transfer.

Bar chart comparing disjoint-item transfer and full-source leak audit for Pidgin to Camfranglais and Camfranglais to Pidgin

Figure 3. Disjoint-item transfer versus the counterpart-leak audit, which trains on all 114 source items (accuracy, %). Bars show the mean over seeds 42, 123, and 2024, whiskers the sample SD, and dots the individual seeds. The dashed line is the 33.3% chance level.

Speech Recognition Robustness Check

The premises in this resource are spoken, so we also evaluated how well a general-purpose multilingual automatic speech recognition system transcribes the recordings, to see whether current tools can already handle these varieties (speech-check path in Figure 1). We used Whisper ([19]) without task-specific fine-tuning, in three sizes (Base, Small, and Large-v3, which is a later release of the same model family) and three language conditions: automatic language identification, language forced to English, and language forced to French. Each condition was run on all 114 recordings per variety (2,052 transcripts in total), and Word Error Rate (WER) was measured against the reference premise texts. This experiment serves as an audio-grounded robustness check rather than a dedicated ASR benchmark.

In general, Word Error Rate is defined as:

WER=S+D+IN(1)\mathrm{WER} = \frac{S + D + I}{N} \tag*{(1)}

where SS, DD, and II denote the number of substitutions, deletions, and insertions, and NN is the total number of words in the reference transcription. WER can exceed 100% when the accumulated errors exceed the reference word count.

Setup. For each variety vv, the speech check uses the 114 audio recordings aia_i and their reference premise texts rir_i:

Dv={(ai,ri)}i=1114D_v = \{(a_i, r_i)\}_{i=1}^{114}

For model mm (Base, Small, or Large-v3), let r^i(m)\hat{r}_i^{(m)} be its raw transcription of aia_i. We call this a transcription, not a hypothesis, to avoid confusion with the NLI hypothesis.

Text normalization. Let ϕ(⋅)\phi(\cdot) be the normalization function, applied identically to the reference and the transcription (Appendix § 12):

  • Whisper language tags are removed.

  • Text is converted to Unicode NFC and lowercased.

  • Apostrophes are kept.

  • Every other character that is not a letter, digit, or combining mark becomes a space.

Word error rate. Adapted to our data, a minimum-edit-distance alignment between ϕ(ri)\phi(r_i) and ϕ(r^i(m))\phi(\hat{r}_i^{(m)}) gives the substitutions SiS_i, deletions DiD_i, and insertions IiI_i for each recording. With NiN_i the number of words in ϕ(ri)\phi(r_i), the corpus-level WER pools errors over the whole subset:

WERv(m)=∑i=1114(Si+Di+Ii)∑i=1114Ni(2)\mathrm{WER}_v^{(m)} = \frac{\sum_{i=1}^{114}(S_i + D_i + I_i)}{\sum_{i=1}^{114}N_i} \tag*{(2)}

The reference denominator is 1,055 words for Cameroon Pidgin and 1,096 for Camfranglais. WER ranges are 95% bootstrap intervals over the 114 recordings.

Additional metrics. Because neither variety has a standard spelling, WER penalizes a correct recognition that is spelled differently from the reference, so CER is less sensitive to spelling variation.

We report CER on the same normalized text, together with MER, WIL, and WIP (definitions in Appendix § 12).

Language-identification counts. Under automatic decoding, let ℓi(m)\ell_i^{(m)} be the language tag that model mm predicts for recording aia_i. The number of distinct tags counts how many different languages the model assigns:

Tv(m)={ℓi(m):i=1,…,114}T_v^{(m)} = \left\{\ell_i^{(m)} : i = 1,\ldots,114\right\}

Note. Matched zero-shot conditions using all 114 recordings per variety. Values are corpus-level WER with CER in brackets, computed on normalized text. Full metrics and WER ranges are in Appendix § 12.

Findings. Large-v3 helps, but does not solve the problem: it has the lowest WER in every row of Table 3 (tied with Small for Pidgin with forced French). The best results are:

VarietyConditionWhisper BaseWhisper SmallWhisper Large-v3
Cameroon PidginAutomatic91.8 (61.9)95.7 (70.2)80.6 (45.6)
Cameroon PidginForced English80.3 (48.1)74.1 (39.0)66.6 (32.3)
Cameroon PidginForced French136.2 (116.3)98.3 (72.6)98.3 (68.0)
CamfranglaisAutomatic90.1 (64.5)77.1 (52.5)64.9 (33.4)
CamfranglaisForced English100.2 (76.0)98.2 (68.4)93.9 (62.4)
CamfranglaisForced French79.3 (54.8)61.8 (39.3)45.5 (26.3)

Table 3. Zero-shot ASR results (normalized WER %, with CER % in brackets)

  • Camfranglais, forced French: 45.5% WER (26.3% CER);

  • Pidgin, forced English: 66.6% WER (32.3% CER).

The ranges of the Pidgin automatic results for Base and Small overlap, so those two models cannot be ranked, and the Pidgin forced-French ranges for Small and Large-v3 include 100%.

Language identification. Under automatic decoding, Whisper rarely identifies Pidgin as English. Large-v3 tags 53 of 114 Pidgin recordings as Yoruba and 40 as Swahili, and only 9 as English. Camfranglais is most often tagged as French (54 of 114), but 24 recordings are tagged as Haitian Creole (Appendix § 12.9). Across all recordings, the number of distinct tags is 20 (Base), 11 (Small), and 9 (Large-v3) for Pidgin, and 23 (Base), 17 (Small), and 17 (Large-v3) for Camfranglais.

Script errors. Many transcripts also come out in the wrong script. For Pidgin under automatic decoding, 26 (Base), 34 (Small), and 14 (Large-v3) of the 114 transcripts contain non-Latin letters. Forcing the matching language largely removes this in both varieties (Appendix § 12.11).

Forced versus automatic language. Forcing the matching language lowers WER for every model and in both varieties, for example from 64.9% to 45.5% for Camfranglais with Large-v3. The best-per-variety results assume that the language of the recording is known in advance. Without that knowledge, the automatic setting applies, giving 80.6% WER for Pidgin and 64.9% for Camfranglais with Large-v3, the best model.

Error types. The transcripts show several kinds of error: language misidentification, output in a different script, wrong substitutions of local vocabulary, and mismatches with the non-standard reference spelling (Appendix § 12.8).

Interpretation

The NLI and ASR experiments examine two separate paths. The NLI experiments ask whether semantic inference can be learned from the available textual representations. The ASR experiments ask how hard it is for a general-purpose speech recognizer to recover the premise text from raw speech recordings.

The within-variety NLI results are above the 33.3% chance level (44.7% and 46.2%), but they do not show that the model uses the relation between premise and hypothesis. A hypothesis-only model scores about the same (41.8% and 47.1%), and paired tests do not separate the two. The most cautious reading is that, here, hypothesis wording may account for much of the above-chance accuracy. This is a property of the current items, and it points to what a larger release should control, such as having independent annotators label the hypotheses without seeing the intended label.

The cross-variety results show the same pattern. Under disjoint-item splits, transfer into Camfranglais (40.1%) is 6.1 points below within-variety Camfranglais accuracy (46.2%), transfer into Pidgin (45.6%) is close to within-variety Pidgin accuracy (44.7%), and neither exceeds hypothesis-only transfer (44.2% and 48.8%). This pattern therefore does not demonstrate that inference ability transfers between the varieties. The higher full-source numbers (63.7% and 60.5%) were obtained with each test item’s counterpart in training, which is consistent with counterpart overlap, although the audit also trains on more items (114 instead of 91–92). None of this bears on the linguistic relation between Cameroon Pidgin and Camfranglais.

The ASR results give a complementary view of the spoken path. With automatic language identification, Large-v3 reaches 80.6% normalized WER for Pidgin and 64.9% for Camfranglais. Forcing the matching language helps, but not enough (best 66.6% and 45.5%). For Pidgin, language identification rarely returns English (9 of 114 recordings with Large-v3). For Camfranglais, the largest gain comes from forcing French with the largest model (64.9% to 45.5%), which suggests that language misidentification accounts for a substantial part of the error. The remaining error may reflect code-switching, local vocabulary, and non-standard spelling, which these experiments do not separate (Appendix § 12.8). Together with the frequent language misidentification, these results indicate that recovering a reliable textual representation from speech is difficult for the evaluated off-the-shelf multilingual recognizers.

The ASR results do not explain the NLI results: the NLI baseline uses reference text rather than Whisper-generated transcripts, so the present experiments do not measure how transcription errors propagate into semantic inference. The defensible joint conclusion is that the dataset exposes two complementary challenges: semantic inference from the textual representation, including how far the labels depend on hypothesis wording, and reliable conversion of spoken input into that representation.

This separation also clarifies the role of this resource. The current release (v1.0) establishes an initial dataset, annotation procedure, and baseline evaluation framework, not an end-to-end spoken-NLI solution.

Future Work

This pilot is a stepping stone. A future version (v2.0) is envisioned as a larger, more diverse resource in which the two varieties no longer share the same scenarios. The planned extensions fall into five parts:

Speakers, Annotation and Splits. We plan to record multiple speakers across demographic groups and Cameroonian regions, using varied devices and acoustic conditions, to test robustness to accent, dialect, and recording-channel variation, and to use independent external annotators so that inter-annotator agreement (e.g., Cohen’s κ\kappa or Fleiss’ κ\kappa) can be reported under documented consent procedures. Version 2.0 also aims to provide held-out development and test partitions.

Unmatched Contexts and Cue Mitigation. To prevent models from exploiting parallel alignments, v2.0 aims to introduce distinct, unmatched scenarios for Camfranglais that decouple it from the Pidgin subset. Hypotheses will be written by several annotators and then labeled by independent annotators who do not see the intended label, and the use of negation words will be balanced across labels to reduce phrase-level artifacts.

ASR Error Propagation and Model Scope. We plan to feed predicted ASR transcripts into the NLI models to measure how word- and character-level errors affect inference. Because neither variety has a standardized script, this will require unified transcription conventions, which could also support fine-tuning speech models on these varieties. We also plan to evaluate larger multilingual encoders and prompted language models.

End-to-End Spoken NLI. We intend to record spoken hypotheses alongside spoken premises, moving from text-based NLI to inference directly from speech, without intermediate transcription.

Additional Varieties. As the collection expands, we aim to add other oral-first Cameroonian language varieties to the evaluation suite, supporting research toward voice-first technologies for under-resourced communities.

Conclusion

We presented CamSpokenNLI, a pilot spoken Natural Language Inference resource for Cameroon Pidgin English and Camfranglais, with 114 balanced scenario-matched items per variety. Fine-tuned XLM-RoBERTa-base reaches 44.7% on Pidgin and 46.2% on Camfranglais under five-fold cross-validation, above the 33.3% chance level, and disjoint-item transfer reaches 40.1% from Pidgin to Camfranglais and 45.6% in the reverse direction. Diagnostic checks show that these scores should be read with care: a hypothesis-only baseline performs about as well as full input, and letting counterpart items into training raises accuracy by about 15 to 24 points, illustrating how counterpart leakage can inflate transfer results.

Evaluated separately from NLI, zero-shot Whisper remains far from adequate on both varieties: its best normalized WER is 66.6% on Pidgin and 45.5% on Camfranglais, reached only when the matching language is forced. These results are exploratory because the release is small, single-speaker, single-annotator and scenario-matched, with no held-out test partition. It is therefore a seed resource, not a benchmark. We release it with its code to support larger, multi-speaker, independently annotated successors and, eventually, end-to-end spoken NLI for these under-resourced varieties.

Limitations

As a pilot resource, CamSpokenNLI has several limitations that bound its scope:

Scale and Coverage. Each variety has only 114 examples, so evaluation uses stratified five-fold cross-validation instead of an independent held-out test set. At this scale, a single seed needs 47 correct predictions (41.2%) to exceed chance at p<0.05p < 0.05, and repeating seeds on the same items does not give independent evidence. All items come from one person, the author, a native of Buea, who wrote the scenarios from everyday life in Buea/Molyko, recorded all the audio, and labeled every item. The dataset therefore does not capture regional, demographic, accent, or topical variation, and inter-annotator agreement cannot be computed.

Dataset Construction and Artifacts. The two subsets share the same 114 scenarios, English glosses, and gold labels. They are not independent collections, so the Camfranglais results are not an independent replication of the Pidgin findings, and the glosses are not an independent control. Hypotheses were written together with their target labels, so style may leak the label; for example, no neutral hypothesis contains a negation word. This does not by itself explain the hypothesis-only accuracy, but the hypothesis-only baseline performs comparably to the full-input model and paired tests do not separate them. The accuracies should therefore not be read as proof of semantic inference, nor as measures of mutual intelligibility or structural similarity between the two varieties.

Modeling Scope and Training. We fine-tuned only XLM-RoBERTa-base; larger multilingual encoders, prompted language models, and instruction-tuned models were not evaluated. The learning rate and batch size were set in early exploratory runs, and there is no separate development set, which may introduce a mild optimistic bias. We kept 15 epochs as the main setting; a 20-epoch run scored higher by +1.2 points for Pidgin and +2.3 points for Camfranglais, within seed variation. Training beyond 20 epochs was not explored.

Audio and Speech Recognition. The NLI baselines use the reference text and do not use the audio; the recordings are evaluated only through a separate Whisper check. Only the premises are recorded; the hypotheses exist as text only. The recordings were made on a consumer mobile phone, not across varied acoustic conditions. The Whisper results (Base, Small, and Large-v3) are an out-of-the-box zero-shot diagnostic, not a fine-tuned ASR benchmark, and only English and French decoding were forced. The best word error rates remain high: 66.6% for Pidgin (forced English) and 45.5% for Camfranglais (forced French), both with Large-v3. Because neither variety has a standardized orthography, these figures partly reflect spelling mismatch rather than incorrect word recognition. Since the NLI experiments use reference text, the effect of ASR errors on a full spoken-NLI pipeline remains unmeasured in this study.

Broader Impact and Ethics Statement

Broader Impact. This resource is intended to support research on spoken language technology for Cameroon Pidgin English and Camfranglais, two under-resourced varieties. It may inform future work on local conversational agents, spoken information retrieval, and voice-driven services. However, because this release is a single-speaker, single-annotator pilot, it is unsuited to benchmarking production systems, and should not be used to draw general sociolinguistic conclusions about the speaker communities. Systems built on it may work worse for other speakers and regions, and should not be deployed without evaluation on the intended users.

Consent, Licensing, and Responsible Data Use. All sentences, recordings, and annotations were created by the author, who consents to their release under CC BY-NC 4.0 for non-commercial research. The scenarios come from everyday life, including utilities, local transport, marketplaces, basic education, and financial transactions. As far as the author has checked, the data contain no sensitive or offensive content and no personally identifiable information. Any future multi-speaker collection will require documented consent and institutional ethical review.

Conflict of Interest. The dataset was developed alongside LocalHands Africa, a voice-first service-exchange platform created by the author, whose intended domain overlaps with the tasks studied here. No data in the v1.0 release come from LocalHands Africa, and the experiments use only general-purpose models. To support independent verification, the dataset, evaluation results, and scripts are made available for research use.

Data and Code Availability

The CamSpokenNLI dataset, including its audio recordings and linguistic annotations, was developed under LAB 237 and is available at huggingface.co/datasets/lab-237/camspoken for non-commercial use under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). The data processing pipeline, reproducible notebooks, and experimental results are available at github.com/lab-237/camspoken under the PolyForm Noncommercial License 1.0.0. Both repositories are tagged v1.0.

Acknowledgments

The author acknowledges the open-source model and software communities upon whose work this study builds, and Google Colab for providing the computational resources.

Experimental Configuration

This appendix lists the settings used for all reported NLI and ASR runs.

XLM-RoBERTa NLI Configuration

To facilitate reproducibility of the NLI experiments, we detail the fine-tuning configuration. All experiments were run on a single NVIDIA Tesla T4 GPU via Google Colab.

  • Base model: xlm-roberta-base

  • Learning rate: 2×10−52 \times10^{-5}, linear schedule

  • Optimizer: AdamW; weight decay 0.0

  • Batch size: 4 per-device train, 8 per-device eval

  • Precision: mixed precision (fp16)

  • Epochs: 15 (the main setting); 5, 10, and 20 epochs are reported as a sensitivity analysis (Appendix § 14)

  • Maximum sequence length: 64 tokens

  • Input (native condition): the premise and hypothesis exactly as in the released CSV (no lowercasing or punctuation removal)

  • Seeds: 42, 123, 2024 (three repeated runs per condition; descriptive mean and standard deviation of accuracy reported)

  • Folds: one stratified five-fold assignment on item numbers 1–114, shared by both varieties (fold seed 42; Appendix § 16)

  • Cross-variety transfer: disjoint-item. For each fold, the model trains on source-variety items outside that fold and is tested on target-variety items inside it, so the counterpart of a test item is never seen in training. Training on all 114 source items and testing on all 114 target items is reported only as a counterpart-leak audit (Appendix § 13.§ 4).

Protocol note. The learning rate and batch size were set in early exploratory runs, without a separate development set, and were not changed after the reported results were seen. No early stopping or checkpoint selection was used, so predictions always come from the final epoch. The seeds are repeated runs on the same 114 items per variety rather than independent test sets, and the fold assignment is fixed for all seeds, conditions, and experiments. Hence, the seeds vary only the model initialization and training order. In the across-seed rows, ±\pm is the sample standard deviation over the three seeds, with the population value in brackets (“pop.”); in the per-seed rows, it is the standard deviation across the five folds of that seed’s cross-validation.

Experimental Conditions and Run Counts

ExperimentSettingRuns
Within-variety, native input2 varieties ×\times 3 seeds ×\times 5 folds30
Within-variety, English gloss2 varieties ×\times 3 seeds ×\times 5 folds30
Within-variety, hypothesis-only2 varieties ×\times 3 seeds ×\times 5 folds30
Within-variety, premise-only2 varieties ×\times 3 seeds ×\times 5 folds30
Epoch sensitivity, native input (5, 10, 20 epochs)2 varieties ×\times 3 epoch settings ×\times 3 seeds ×\times 5 folds90
Disjoint-item transfer (native and hyp.-only)2 directions ×\times 2 inputs ×\times 3 seeds ×\times 5 folds60
Counterpart-leak audit2 directions ×\times 3 seeds6
Total276

Table 4.

The 15-epoch native runs serve as the 15-epoch point of the epoch study. The two English-gloss conditions use the same glosses, labels, and folds, so they give identical results; the two gloss conditions are counted separately in the table.

Whisper ASR Configuration

  • Models: OpenAI Whisper Base, Small, and Large-v3 (openai-whisper 20250625), zero-shot, with no fine-tuning

  • Conditions: automatic language identification, forced English and forced French, for both varieties

  • Scale: 18 blocks (2 varieties × 3 models × 3 language conditions) of 114 recordings, 2,052 transcripts

  • Decoding: default settings, with the random seed set to 0 before every recording

  • Audio: the 16 kHz mono recordings described in Appendix § 17

Software and Reproducibility

Python 3.13.15, PyTorch 2.11.0 (CUDA 13.0), Transformers 5.16.1, Datasets 4.8.5, scikit-learn 1.6.1, SciPy 1.16.3, jiwer 4.0.0, NumPy 2.1.3, pandas 2.2.3, and FFmpeg (Colab build, version not recorded). The release includes SHA-256 fingerprints for the two dataset files, the fold assignment and every audio file, along with a run manifest. Every NLI run stores its 114 item-level predictions.

Mean training loss over all optimization steps, for the 120 within-variety runs at 15 epochs, was 0.80 (Pidgin) and 0.67 (Camfranglais) on native text, 0.52 on the English gloss, 0.81 / 0.75 for hypothesis-only and 0.86 / 0.83 for premise-only input, against about 1.10 for uniform guessing; it was logged for these runs only. All NLI and ASR analyses were rerun identically from the saved files. The Whisper transcripts were generated once, and their exact reproducibility on a different GPU has not been tested.

NLI Robustness and Per-Seed Results

This appendix preserves the seed-level NLI evidence supporting the main-paper summaries. It functions as the audit trail for the small-sample experiments rather than as an alternative headline result table. pp is the one-sided exact binomial test of the pooled accuracy against chance (1/3). In the “Across seeds” row, the last column counts the seeds with p<0.05p < 0.05.

Cameroon Pidgin English – Primary Configuration (15 epochs)

SeedMean accuracy ± SD across foldsPooled accuracypp (vs 1/3)
4245.5% ± 8.0%52/114 (45.6%)0.0043
12346.5% ± 8.1%53/114 (46.5%)0.0024
202442.1% ± 4.8%48/114 (42.1%)0.0312
Across seeds–44.7% ± 2.3% (pop. 1.9%)3 of 3

Table 5.

The pooled accuracy is computed over the 114 examples for each seed. The three seeds are repeated runs, not independent test sets.

English Gloss Reference – Cameroon Pidgin Material

SeedMean accuracy ± SD across foldsPooled accuracypp (vs 1/3)
4264.0% ± 7.8%73/114 (64.0%)< 0.0001
12356.2% ± 5.9%64/114 (56.1%)< 0.0001
202463.2% ± 13.8%72/114 (63.2%)< 0.0001
Across seeds–61.1% ± 4.3% (pop. 3.5%)3 of 3

Table 6.

The English glosses are translations of the original Pidgin and Camfranglais material and therefore constitute an approximate reference condition rather than an independently authored English corpus. The glosses, labels and folds are the same for both varieties, so B.2 and B.4 give identical results by construction (§ 4.1).

Camfranglais – Primary Configuration (15 epochs)

SeedMean accuracy ±\pm SD across foldsPooled accuracyp (vs 1/3)
4245.5%±10.0%45.5\% \pm10.0\%52/114 (45.6%)0.0043
12344.7%±7.3%44.7\% \pm7.3\%51/114 (44.7%)0.0074
202448.2%±10.8%48.2\% \pm10.8\%55/114 (48.2%)0.0007
Across seeds–46.2%±1.8%46.2\% \pm1.8\% (pop. 1.5%)3 of 3

Table 7.

The three seeds are repeated runs on the same 114 examples, not independent test sets.

English Gloss Reference – Camfranglais Material

SeedMean accuracy ±\pm SD across foldsPooled accuracyp (vs 1/3)
4264.0%±7.8%64.0\% \pm7.8\%73/114 (64.0%)< 0.0001
12356.2%±5.9%56.2\% \pm5.9\%64/114 (56.1%)< 0.0001
202463.2%±13.8%63.2\% \pm13.8\%72/114 (63.2%)< 0.0001
Across seeds–61.1%±4.3%61.1\% \pm4.3\% (pop. 3.5%)3 of 3

Table 8.

As in Appendix § 11.§ 2, this condition is an approximate translated reference rather than an independently authored English corpus, and it gives the same results as B.2 by construction.

Hypothesis-Only Baseline – Cameroon Pidgin English

SeedMean accuracy ±\pm SD across foldsPooled accuracyp (vs 1/3)
4240.3%±3.1%40.3\% \pm3.1\%46/114 (40.4%)0.0696
12343.9%±3.2%43.9\% \pm3.2\%50/114 (43.9%)0.0123
202441.2%±4.9%41.2\% \pm4.9\%47/114 (41.2%)0.0473
Across seeds–41.8%±1.8%41.8\% \pm1.8\% (pop. 1.5%)2 of 3

Table 9.

The model sees only the hypothesis. This shows how much of the native-input accuracy can be reached without the premise.

Hypothesis-Only Baseline – Camfranglais

SeedMean accuracy ±\pm SD across foldsPooled accuracyp (vs 1/3)
4252.5%±15.5%52.5\% \pm15.5\%60/114 (52.6%)< 0.0001
12342.8%±15.8%42.8\% \pm15.8\%49/114 (43.0%)0.0199
202445.5%±8.0%45.5\% \pm8.0\%52/114 (45.6%)0.0043
Across seeds–47.1%±5.0%47.1\% \pm5.0\% (pop. 4.1%)3 of 3

Table 10.

Premise-Only Baseline – Cameroon Pidgin English

SeedMean accuracy ±\pm SD across foldsPooled accuracyp (vs 1/3)
4237.7%±2.0%37.7\% \pm2.0\%43/114 (37.7%)0.1851
12338.7%±8.5%38.7\% \pm8.5\%44/114 (38.6%)0.1376
202435.1%±7.6%35.1\% \pm7.6\%40/114 (35.1%)0.3791
Across seeds–37.1%±1.8%37.1\% \pm1.8\% (pop. 1.5%)0 of 3

Table 11.

The model sees only the premise. With balanced labels, a premise alone should carry little label information, so results near 33.3% are expected.

Premise-Only Baseline – Camfranglais

SeedMean accuracy ±\pm SD across foldsPooled accuracypp (vs 1/3)
4236.8% ±\pm 2.2%42/114 (36.8%)0.2417
12344.7% ±\pm 7.9%51/114 (44.7%)0.0074
202433.4% ±\pm 4.6%38/114 (33.3%)0.5352
Across seeds–38.3% ±\pm 5.8% (pop. 4.8%)1 of 3

Rounding note. The reported mean ±\pm SD summaries are computed from the underlying fold-level and item-level outputs before rounding. The one-decimal seed-level values in these tables are rounded for presentation and may not reproduce the displayed summary statistics exactly when recomputed from the rounded values.

ASR / WER Diagnostic Record

This appendix preserves the detailed Whisper diagnostics for all three models (Base, Small, Large-v3) under three language conditions (automatic language identification, forced English, forced French) for both varieties: 18 blocks of 114 recordings, 2,052 transcripts. Raw WER is computed on the text as written (only Whisper language tags and extra whitespace removed). Normalized scoring uses 1,055 reference words per block for Cameroon Pidgin English and 1,096 for Camfranglais. All other values use the normalization rule in Appendix § 12.§ 1; Appendix § 12.10 gives the range of each WER, and Appendix § 12.11 gives the number of transcripts with non-Latin script and the number left empty after normalization.

Text Normalization and Metric Definitions

Normalization. Normalization is applied identically to the reference and the Whisper output: (1) delete Whisper language tags of the form <|xx|>; (2) convert to Unicode NFC and lowercase; (3) convert curly apostrophes to straight ones; (4) keep apostrophes, because they belong to French words (j’ai, l’école); (5) replace every other character that is not a letter, a digit or a combining mark with a space; (6) collapse whitespace. All letters, including accented letters, are kept. Example: “Je suis au carrefour comme ça.” becomes “je suis au carrefour comme ça”.

Metrics. Raw WER is computed on the unnormalized strings split on whitespace (case and punctuation kept; only Whisper tags and extra whitespace are removed). Normalized WER is computed on the normalized strings and is the value reported in the main paper. MER, WIL, and WIP are the standard match-error, word-information-lost and word-information-preserved measures. CER is the character error rate on the normalized text. S, D and I are substitutions, deletions and insertions from a minimum-edit-distance alignment, and Correct is the number of matched words. Distinct language tags is the number of different languages Whisper predicted across the 114 clips under automatic language identification (one when the language is forced).

Final Primary ASR Record – Cameroon Pidgin English (automatic language ID)

MetricBaseSmallLarge-v3
Raw WER94.6%98.3%84.4%
Normalized WER91.8%95.7%80.6%
MER85.7%90.5%77.2%
WIL97.2%98.7%93.3%
WIP2.8%1.3%6.7%
CER (normalized)61.9%70.2%45.6%
Reference words1,0551,0551,055
Hypothesis words882801898
Correct161106251
Substitutions647634601
Deletions247315203
Insertions746146
Distinct language tags20119

Cameroon Pidgin – Forced-English Diagnostic

MetricBaseSmallLarge-v3
Raw WER84.0%79.2%72.0%
Normalized WER80.3%74.1%66.6%
MER76.2%71.2%64.2%
WIL93.1%90.3%85.4%
WIP6.9%9.7%14.6%
CER (normalized)48.1%39.0%32.3%
Reference words1,0551,0551,055
Hypothesis words9619721,000
Correct265316392
Substitutions639613568
Deletions15112695
Insertions574340

Table C.3.

Cameroon Pidgin – Forced-French Diagnostic

MetricBaseSmallLarge-v3
Raw WER137.9%101.6%101.3%
Normalized WER136.2%98.3%98.3%
MER95.9%91.7%89.8%
WIL99.7%99.1%98.7%
WIP0.3%0.9%1.3%
CER (normalized)116.3%72.6%68.0%
Reference words1,0551,0551,055
Hypothesis words1,3319541,024
Correct6194118
Substitutions827784806
Deletions167177131
Insertions44376100

Table C.4.

Forced English lowers the normalized WER at every model size compared with automatic language identification (for Large-v3, 66.6% against 80.6%), whereas forced French raises it at every size. Forcing French is a mismatched-language control for Pidgin. Base inserts many words (443 insertions), which pushes its WER above 100%. Small and Large-v3 reach the same normalized WER (98.3%) by coincidence: each makes 1,037 errors against 1,055 reference words, but with a different mix (784 substitutions, 177 deletions and 76 insertions for Small; 806, 131 and 100 for Large-v3).

Final Primary ASR Record – Camfranglais (automatic language ID)

MetricBaseSmallLarge-v3
Raw WER94.3%82.1%71.0%
Normalized WER90.1%77.1%64.9%
MER81.7%73.7%63.1%
WIL95.5%91.0%84.4%
WIP4.5%9.0%15.6%
CER (normalized)64.5%52.5%33.4%
Reference words1,0961,0961,096
Hypothesis words1,0029131,011
Correct221301416
Substitutions668562564
Deletions207233116
Insertions1135031
Distinct language tags231717

Table C.5.

Automatic language identification is the primary condition. Normalized WER falls from 90.1% (Base) to 64.9% (Large-v3), and correct words rise from 221 to 416 of 1,096 reference words. Large-v3 also makes the fewest deletions (116) and insertions (31). Base produces 23 distinct language tags, against 17 for Small and Large-v3.

Camfranglais – Forced-French Diagnostic

MetricBaseSmallLarge-v3
Raw WER85.7%69.1%55.9%
Normalized WER79.3%61.8%45.5%
MER70.9%59.5%44.2%
WIL89.7%80.8%66.2%
WIP10.3%19.2%33.8%
CER (normalized)54.8%39.3%26.3%
Reference words1,0961,0961,096
Hypothesis words1,1191,0061,071
Correct356460630
Substitutions634505408
Deletions10613158
Insertions1294133

Table 12.

Normalized WER falls from 79.3% (Base) to 45.5% (Large-v3), and correct words rise from 356 to 630 of 1,096 reference words. Large-v3 also makes the fewest insertions (33) and deletions (58). This run is diagnostic: forced French requires knowing the language in advance, and the recordings are Camfranglais rather than standard French.

Camfranglais – Forced-English Diagnostic

MetricBaseSmallLarge-v3
Raw WER101.0%100.0%95.6%
Normalized WER100.2%98.2%93.9%
MER97.4%92.7%89.3%
WIL99.9%99.3%98.6%
WIP0.1%0.7%1.4%
CER (normalized)76.0%68.4%62.4%
Reference words1,0961,0961,096
Hypothesis words775979993
Correct2985123
Substitutions715829814
Deletions352182159
Insertions316556

Table 13.

Forced English is a mismatched-language control for Camfranglais, which is largely French-based. In the forced-language tables (C.6 and C.7), the language tag is fixed, so the “distinct language tags” row is omitted. For all three models, forced French gives the lowest normalized WER, automatic identification is in the middle, and forced English is the highest (for Large-v3, 45.5%, 64.9%, and 93.9%). For Camfranglais, WER falls with model size in every condition. Under forced English, the normalized WER of Base is slightly above 100% (100.2%) because its total errors, including insertions, exceed the number of reference words, and Large-v3 recovers only 123 of the 1,096 reference words correctly, against 416 under automatic identification and 630 under forced French.

Representative Transcripts (Whisper Large-v3)

ClipReferenceAutomatic (detected language)Forced languageWhat it shows
wes_00018price fo petrol don climb again.price for petrodon climb again (en)samePidgin recognized as English; WER 50%
wes_00103president fo njangi house broke bank di end of year ma own share fo de money no comot.president Funchangi has broke bank for this end of year. Ma own share for the Molino Commod. (yo)President Funjangi has broke bank for this end of year. Maun She for the Molino Commode.Local words replaced by wrong ones (njangi → Funchangi); WER 56% → 72%
wes_00008e ask me how old a dey.হে আস্ক মে হাও অল্ড আ দে? (yo)He asked me how old I’d be.Non-Latin script for a Latin-script reference; WER 100% → 57%
wes_00086ma phone e screen crack crack very badly.ຕີ່ກາດສີ່ງ chao nđi Pritzk­er (sn)My phone is screen crack crack very badly.Four scripts in one transcript; forcing English fixes it; WER 100% → 25%
camf_00019je suis moi peace.Je suis moi, peace ! (fr)sameCorrect; WER 0%
camf_00018le prix du carburant a encore gonfle.Lo pri di kabir an a anko gonfle. (ht)Le prix du carburant a encore gonflé.Haitian Creole tag; forcing French improves WER 86% → 14%
camf_00023le grand vent a arraché les tôles.لغران فنغاشل الدول (ar)Le grand vent a acheté le tout.Arabic script for French speech; WER 100% → 43%
camf_00009Paul a travel pour Doual la semaine passée.Paul a travaillé pour Douala la semaine passée. (fr)sameLocal spelling “Doual” and the English word “travel” replaced; WER 25%

Table C.8.

Examples are copied unchanged from the saved transcripts and illustrate language misidentification, wrong script, wrong local words, and spelling mismatch. Clip-level WER is normalized; where two values are shown, the first is for automatic decoding and the second for forced decoding (English for Pidgin, French for Camfranglais).

Because neither variety has a standard spelling, a phonetic transcription counts as an error even though the sounds were recovered. Under Large-v3, no Pidgin recording is error-free in any condition; for Camfranglais, 8 of 114 recordings are error-free under automatic identification, 3 under forced English and 10 under forced French. The row-level record for all 114 recordings, every model and condition, is included in the release.

Language Identification Distribution Under Automatic Decoding

VarietyModelDistinct tagsMatching-language tag (count)Most frequent tags (of 114)
Cameroon PidginBase20en 16yo 31, en 16, sw 15, de 12
Cameroon PidginSmall11en 3yo 51, sw 35, fr 7, sn 6
Cameroon PidginLarge-v39en 9yo 53, sw 40, en 9, sn 4
CamfranglaisBase23fr 51fr 51, ar 11, es 6, de 6
CamfranglaisSmall17fr 58fr 58, yo 13, ht 12, ar 7
CamfranglaisLarge-v317fr 54fr 54, ht 24, sw 9, yo 8

Table C.9.

Whisper rarely identifies Cameroon Pidgin as English. Under automatic decoding, Large-v3 tags it as Yoruba or Swahili for 93 of 114 recordings and as English for only 9. Camfranglais is most often identified as French, but 24 recordings are tagged as Haitian Creole under Large-v3.

Tags are ISO 639-1 codes (yo = Yoruba, en = English, sw = Swahili, de = German, fr = French, ar = Arabic, es = Spanish, ht = Haitian Creole, sn = Shona).

Clips whose automatic transcript contains non-Latin script: Cameroon Pidgin (Base 26, Small 34, Large-v3 14), Camfranglais (Base 24, Small 26, Large-v3 7). The complete tag counts for all 18 blocks are in asr_summary.csv in the release.

Ranges of the WER

The WER is computed on 114 recordings, so it would change if different recordings were used. Each range shows how much it could change. Overlapping ranges neither establish nor rule out a difference, because every model is scored on the same recordings. Ranges are 95% bootstrap intervals (recordings resampled 2,000 times, seed 0).

VarietyConditionBaseSmallLarge-v3
Cameroon PidginAutomatic91.8% (85.0–101.4)95.7% (91.6–100.2)80.6% (75.6–85.7)
Forced English80.3% (76.6–84.2)74.1% (70.5–78.0)66.6% (62.7–70.6)
Forced French136.2% (117.0–160.9)98.3% (94.1–102.6)98.3% (93.5–102.7)
CamfranglaisAutomatic90.1% (82.4–99.2)77.1% (70.4–83.8)64.9% (58.3–71.6)
Forced English100.2% (98.4–102.4)98.2% (94.8–101.5)93.9% (89.9–97.6)
Forced French79.3% (71.5–87.8)61.8% (56.0–67.3)45.5% (41.0–50.4)

Table C.10.

Script and Empty-Transcript Checks

For each variety and condition, the number of the 114 transcripts that contain letters outside the Latin script (Non-Latin) and the number that are empty after normalization (Empty), listed for Base / Small / Large-v3.

VarietyConditionNon-Latin (Base / Small / Large-v3)Empty (Base / Small / Large-v3)
Cameroon PidginAutomatic26 / 34 / 141 / 0 / 0
Forced English4 / 0 / 20 / 0 / 0
Forced French27 / 14 / 10 / 0 / 1
CamfranglaisAutomatic24 / 26 / 70 / 0 / 0
Forced English10 / 5 / 21 / 1 / 0
Forced French11 / 2 / 00 / 0 / 0

Table C.11.

Cross-Variety Transfer Results

Protocols

Disjoint-item transfer (main result). Using the shared item folds (Appendix § 16), fold kk trains on source-variety items outside fold kk (91–92 items) and tests on target-variety items in fold kk (22–23 items). The five folds together give 114 test predictions per seed. A test item’s counterpart (the same scenario in the other variety) is never in training.

Full-source transfer (counterpart-leak audit). The model is trained on all 114 source items and tested on all 114 target items. Every test item’s counterpart is therefore in the training set (114/114). This protocol is reported only as an audit, and the numbers must not be read as cross-variety transfer.

Shared settings. The same XLM-RoBERTa configuration was used in both directions (Appendix § 10.§ 1). The three seeds are repeated runs on the same items and are therefore not independent test sets. Mean ± SD across seeds uses pooled accuracy and the sample SD (population SD in brackets). The last column gives the one-sided exact binomial pp-value against 1/3 (47/114 is the minimum for p<0.05p < 0.05); in the “Across seeds” rows it counts the seeds with p<0.05p < 0.05. Chance level is 33.3%.

Disjoint-Item Transfer, Full Input

Training →\to TestSeedMean ±\pm SD across foldsPooled accuracypp
Pidgin →\to Camfranglais4237.7%±10.3%37.7\% \pm10.3\%43/114 (37.7%)0.1851
12343.8%±6.4%43.8\% \pm6.4\%50/114 (43.9%)0.0123
202438.6%±3.8%38.6\% \pm3.8\%44/114 (38.6%)0.1376
Across seeds–40.1%±3.3%40.1\% \pm3.3\% (pop. 2.7%)1 of 3
Camfranglais →\to Pidgin4242.1%±4.4%42.1\% \pm4.4\%48/114 (42.1%)0.0312
12345.5%±12.2%45.5\% \pm12.2\%52/114 (45.6%)0.0043
202449.0%±7.7%49.0\% \pm7.7\%56/114 (49.1%)0.0004
Across seeds–45.6%±3.5%45.6\% \pm3.5\% (pop. 2.9%)3 of 3

Table 14 (PDF p. 22).

Disjoint-Item Transfer, Hypothesis-Only Input

Training →\to TestSeedMean ±\pm SD across foldsPooled accuracypp
Pidgin →\to Camfranglais4246.5%±6.5%46.5\% \pm6.5\%53/114 (46.5%)0.0024
12348.1%±8.1%48.1\% \pm8.1\%55/114 (48.2%)0.0007
202437.7%±6.2%37.7\% \pm6.2\%43/114 (37.7%)0.1851
Across seeds–44.2%±5.6%44.2\% \pm5.6\% (pop. 4.6%)2 of 3
Camfranglais →\to Pidgin4246.5%±5.2%46.5\% \pm5.2\%53/114 (46.5%)0.0024
12348.2%±7.3%48.2\% \pm7.3\%55/114 (48.2%)0.0007
202451.8%±5.0%51.8\% \pm5.0\%59/114 (51.8%)< 0.0001
Across seeds–48.8%±2.7%48.8\% \pm2.7\% (pop. 2.2%)3 of 3

Table 15 (PDF p. 22).

The hypothesis-only means (44.2% and 48.8%) are higher than the full-input transfer means (40.1% and 45.6%). Full-input transfer is therefore not shown to use the premise (see Appendix § 15.§ 2).

Full-Source Transfer (Counterpart-Leak Audit, Not a Transfer Result)

Training →\to TestSeedMean ±\pm SD across foldsAccuracy (all 114 test items)pp
Pidgin →\to Camfranglais42–72/114 (63.2%)< 0.0001
123–79/114 (69.3%)< 0.0001
2024–67/114 (58.8%)< 0.0001
Across seeds–63.7%±5.3%63.7\% \pm5.3\% (pop. 4.3%)3 of 3
Camfranglais →\to Pidgin42–79/114 (69.3%)< 0.0001
123–67/114 (58.8%)< 0.0001
2024–61/114 (53.5%)< 0.0001
Across seeds–60.5%±8.0%60.5\% \pm8.0\% (pop. 6.6%)3 of 3

Table 16 (PDF p. 22).

There are no folds in this protocol (one training run per seed); the “Mean ±\pm SD across folds” column is empty. Compared with disjoint-item transfer (Appendix § 13.§ 2), accuracy is about 24 points higher for Pidgin →\to Camfranglais and 15 points higher for Camfranglais →\to Pidgin. That gap reflects counterpart leakage together with the larger training set (114 instead of 91–92 items).

Side-by-Side Comparison

DirectionDisjoint, full inputDisjoint, hypothesis-onlyWithin-variety (target)Full-source (counterpart leak)
Pidgin →\to Camfranglais40.1%±3.3%40.1\% \pm3.3\%44.2%±5.6%44.2\% \pm5.6\%46.2%±1.8%46.2\% \pm1.8\%63.7%±5.3%63.7\% \pm5.3\%
Camfranglais →\to Pidgin45.6%±3.5%45.6\% \pm3.5\%48.8%±2.7%48.8\% \pm2.7\%44.7%±2.3%44.7\% \pm2.3\%60.5%±8.0%60.5\% \pm8.0\%

Table 17.

Note. Mean ±\pm sample SD over seeds 42, 123 and 2024. “Within-variety (target)” is the five-fold accuracy of a model trained and tested on the target variety (Camfranglais in the first row, Pidgin in the second).

Per seed, letting counterparts into training raises the number of correct predictions by 29, 29 and 23 (Pidgin → Camfranglais) and by 31, 15 and 5 (Camfranglais → Pidgin) relative to the disjoint run with the same seed, an average gain of 23.7 and 14.9 points, positive in all six seed pairs. Disjoint transfer is 6.1 points below within-variety accuracy on Camfranglais, and 0.9 points above it on Pidgin; the 0.9-point difference on Pidgin is within the seed-to-seed variation, whereas on Camfranglais all three transfer seeds (37.7%, 43.9%, 38.6%) lie below all three within-variety seeds (45.6%, 44.7%, 48.2%).

Epoch Sensitivity

Setup

The model, data, folds, and seeds are identical to the main within-variety runs (native text, full input); only the number of epochs changes (5, 10, 15, and 20). The 15-epoch values are the main runs, reused. Each epoch is 23 optimizer steps (115 to 460 in total). The two varieties are reported separately (Tables E.1 and E.2). Hypothesis-only and transfer runs used 15 epochs only.

These runs are a sensitivity analysis on the same data, folds, and seeds. Because there is no separate development set, the main accuracies may be slightly optimistic. Accuracy rises most from 5 to 10 epochs (+7.0 points for Pidgin, +5.5 for Camfranglais), and each further step adds at most 2.3 points. Fifteen epochs is not the highest-scoring setting for either variety, but from 15 to 20 epochs the mean changes by only +1.2 (Pidgin) and +2.3 points (Camfranglais), less than the across-seed SD at 20 epochs (5.0 and 4.1). We therefore keep 15 epochs as the main setting.

Table E.1: Accuracy by number of epochs (pooled accuracy, mean ± sample SD over seeds 42, 123, 2024)

VarietyEpochsMean accuracy ± SD (pop.)Change from 15 epochs (points)Seeds p<0.05p < 0.05
Cameroon Pidgin535.7% ± 3.1% (pop. 2.5%)−9.10 of 3
1042.7% ± 3.5% (pop. 2.9%)−2.02 of 3
15 (main)44.7% ± 2.3% (pop. 1.9%)–3 of 3
2045.9% ± 5.0% (pop. 4.1%)+1.22 of 3
Camfranglais538.9% ± 2.2% (pop. 1.8%)−7.31 of 3
1044.4% ± 6.2% (pop. 5.1%)−1.82 of 3
15 (main)46.2% ± 1.8% (pop. 1.5%)–3 of 3
2048.5% ± 4.1% (pop. 3.4%)+2.33 of 3

Table E.1.

Note. Five epochs give the lowest accuracy for both varieties (near chance for Pidgin). Mean accuracy rises from 5 to 15 epochs in both varieties, and 15 epochs is the earliest setting at which all three seeds are above chance for both of them; for Pidgin, 20 epochs is back to two of three.

Table E.2: Per-seed results by number of epochs

VarietyEpochsSeed 42Seed 123Seed 2024Seeds p<0.05p < 0.05
Cameroon Pidgin537 (0.6132)44 (0.1376)41 (0.3069)0 of 3
1045 (0.0994)53 (0.0024)48 (0.0312)2 of 3
15 (main)52 (0.0043)53 (0.0024)48 (0.0312)3 of 3
2054 (0.0013)46 (0.0696)57 (0.0002)2 of 3
Camfranglais544 (0.1376)47 (0.0473)42 (0.2417)1 of 3
1044 (0.1851)52 (0.0043)57 (0.0002)2 of 3
15 (main)52 (0.0043)51 (0.0074)55 (0.0007)3 of 3
2050 (0.0123)57 (0.0002)59 (< 0.0001)3 of 3

Table E.2.

Correct out of 114; one-sided exact binomial pp-values against 1/3 are in brackets. 47/114 is the minimum for p<0.05p < 0.05.

Statistical Tests

One-Sided Binomial Tests Against Chance

For each condition and seed, we test whether the pooled accuracy over 114 predictions exceeds 1/3 (one-sided exact binomial). The smallest passing score is 47/114 (41.2%, p=0.047p = 0.047); 46/114 gives p=0.070p = 0.070. A single run’s 95% interval is roughly ±9\pm9 points around its accuracy, so differences of a few points between conditions are within noise. Seeds are repeated runs on the same items, so “seeds above chance” describes consistency across runs, not independent replication.

ConditionSeed 42Seed 123Seed 2024Seeds p<0.05p < 0.05
Pidgin, within, full52 (0.0043)53 (0.0024)48 (0.0312)3 of 3
Camfranglais, within, full52 (0.0043)51 (0.0074)55 (0.0007)3 of 3
Pidgin, within, hypothesis-only46 (0.0696)50 (0.0123)47 (0.0473)2 of 3
Camfranglais, within, hypothesis-only60 (< 0.0001)49 (0.0199)52 (0.0043)3 of 3
Pidgin, within, premise-only43 (0.1851)44 (0.1376)40 (0.3791)0 of 3
Camfranglais, within, premise-only42 (0.2417)51 (0.0074)38 (0.5352)1 of 3
Pidgin gloss73 (< 0.0001)64 (< 0.0001)72 (< 0.0001)3 of 3
Camfranglais gloss73 (< 0.0001)64 (< 0.0001)72 (< 0.0001)3 of 3
Pidgin →\to Camfranglais, disjoint, full43 (0.1851)50 (0.0123)44 (0.1376)1 of 3
Camfranglais →\to Pidgin, disjoint, full48 (0.0312)52 (0.0043)56 (0.0004)3 of 3
Pidgin →\to Camfranglais, disjoint, hyp.-only53 (0.0024)55 (0.0007)43 (0.1851)2 of 3
Camfranglais →\to Pidgin, disjoint, hyp.-only53 (0.0024)55 (0.0007)59 (< 0.0001)3 of 3
Pidgin →\to Camfranglais, full-source (leak)72 (< 0.0001)79 (< 0.0001)67 (< 0.0001)3 of 3
Camfranglais →\to Pidgin, full-source (leak)79 (< 0.0001)67 (< 0.0001)61 (< 0.0001)3 of 3

Table 1 (PDF p. 24).

Cells give the correct out of 114 and the one-sided pp-value. The two gloss rows are identical because the glosses, labels, and folds are shared (§ 4.1). Epoch-sensitivity tests are in Appendix § 14.

Full Input Versus Hypothesis-Only (Exact McNemar Tests)

For each seed and setting, we compare item-level correctness of the full-input and hypothesis-only models on the same 114 test items. “Full only” counts items the full-input model got right and the hypothesis-only model got wrong, and “Hyp. only” the reverse. The pp-value is the exact two-sided McNemar (binomial) test on the discordant items.

SettingSeedFull correctHyp. correctFull onlyHyp. onlypp
Pidgin, within42524625190.451
123535024210.766
2024484720191.000
Camfranglais, within42526020280.312
123514921190.875
2024555218150.728
Pidgin →\to Camfranglais, disjoint42435321310.212
123505521260.560
2024444322211.000
Camfranglais →\to Pidgin, disjoint42485321260.560
123525522250.771
2024565923260.775

Table 2 (PDF p. 24).

Twelve comparisons were made and none has p<0.05p < 0.05 (the smallest is 0.212). The full-input model scores higher in 6 of 12 comparisons and the hypothesis-only model in the other 6, and the sign changes between seeds within the same setting (Camfranglais within-variety and Pidgin →\to Camfranglais). The data therefore do not show that full input outperforms hypothesis-only. They also do not show that the two are equal: with 114 items and 33 to 52 discordant pairs per test, differences of several points are not statistically detectable.

Mean Differences (Full Input Minus Hypothesis-Only, Pooled Accuracy)

SettingFullHypothesis-onlyDifference
Pidgin, within44.7%41.8%+2.9
Camfranglais, within46.2%47.1%−0.9
Pidgin →\to Camfranglais, disjoint40.1%44.2%−4.1
Camfranglais →\to Pidgin, disjoint45.6%48.8%−3.2

Table F.3.

Differences are in percentage points, averaged over the three seeds. They are small, change sign across settings, and are within the seed-to-seed variation.

Cross-Validation Folds

FoldItemsEntailmentContradictionNeutral
023878
123887
223887
323788
422778
Total114383838

Table G.1. Shared item-level fold assignment (fold seed 42)

One stratified five-fold assignment is defined on item numbers 1–114 and shared by both varieties: item ii is in the same fold in Pidgin (wes_NNNNN) and Camfranglais (camf_NNNNN). We built it from the labels, which are the same in both varieties, and fixed it across all seeds, conditions, and experiments, so seeds vary only in model initialization and training order. In disjoint-item transfer, fold kk trains on source items outside fold kk (91–92 items) and tests on target items in fold kk (22–23 items), so a test item’s counterpart is never in training.

FoldItem numbers
03, 6, 7, 11, 14, 15, 25, 29, 33, 38, 42, 48, 50, 56, 58, 60, 68, 69, 77, 84, 100, 108, 109
18, 16, 18, 20, 21, 24, 27, 32, 45, 46, 47, 61, 63, 64, 74, 88, 95, 97, 99, 105, 106, 111, 112
22, 4, 17, 19, 23, 28, 41, 43, 44, 49, 51, 66, 71, 75, 81, 82, 90, 93, 98, 101, 103, 104, 107
39, 10, 12, 13, 22, 35, 36, 37, 39, 40, 52, 53, 59, 65, 73, 78, 80, 86, 89, 91, 96, 102, 114
41, 5, 26, 30, 31, 34, 54, 55, 57, 62, 67, 70, 72, 76, 79, 83, 85, 87, 92, 94, 110, 113

Table G.2. Item numbers in each fold

Dataset and Audio Checks

Files and Format

Each variety is a UTF-8 (with byte-order mark) CSV with 114 rows and seven columns: id, premise_text, premise_text_en, premise_audio, hypothesis_text, hypothesis_text_en, label. Audio files are referenced as audio/pidgin/wes_NNNNN.wav and audio/camfranglais/camf_NNNNN.wav, where NNNNN is the item number (1–114), so item ii has the same number in both varieties. Labels are lowercase (entailment, contradiction, neutral), 38 each per variety. No field is empty, no field contains a newline or a curly apostrophe, and all 114 premises and all 114 hypotheses are unique within each variety (114 unique premise–hypothesis pairs).

Text Statistics

PidginCamfranglais
Items114114
Premise words (total / mean / range)1,053 / 9.2 / 4–191,084 / 9.5 / 4–21
Hypothesis words (total / mean / range)786 / 6.9 / 3–11845 / 7.4 / 3–15
English-gloss premise words1,0461,046
English-gloss hypothesis words816816
Premises ending in . ! or ?114114

Table 18.

Word counts are whitespace tokens in the released CSVs. The ASR reference totals in Appendix C (1,055 and 1,096) are slightly higher because normalization turns hyphens and other punctuation into spaces, which splits some tokens (for example, tip-là). No premise contains a question mark. Mean hypothesis length varies little by label (entailment, contradiction, neutral: Pidgin 6.6, 7.1, 6.9; Camfranglais 7.4, 7.2, 7.6), so length alone is not a visible label cue.

Scenario-Matched Structure

Items are scenario-matched by construction (§ 3.2). The Pidgin and Camfranglais sentences differ: the mean word overlap between counterpart premises (shared words divided by all distinct words, per pair) is about 8%, and no pair is identical. The English glosses are shared across the two files so that items can be aligned, and they are identical in every cell (premise and hypothesis, 114 of 114). Labels agree in all 114 items. For this reason the two English-gloss conditions are treated as one reference condition (§ 4.1).

Hypothesis Wording Check

To look for label cues in the hypotheses, we counted hypotheses containing a negation word using simple keyword lists (for example, no, never, nobody for Pidgin and the English glosses; ne, n’, pas, jamais, aucun, personne, rien and plus for Camfranglais). This is a heuristic, not a linguistic annotation.

Hypotheses with a negation wordEntailmentContradictionNeutral
Pidgin6 of 38 (16%)6 of 38 (16%)0 of 38 (0%)
Camfranglais10 of 38 (26%)11 of 38 (29%)0 of 38 (0%)
English glosses (shared by both varieties)7 of 38 (18%)9 of 38 (24%)0 of 38 (0%)

Table 19.

No neutral hypothesis contains a negation word, so the presence of negation is a partial label cue. This is consistent with the hypothesis-only results in § 4.3, but it does not by itself explain them.

Audio Statistics

PidginCamfranglais
Recordings114114
Final duration (mean ±\pm SD, s)3.66 ±\pm 0.943.40 ±\pm 0.79
Final duration (min / median / max, s)2.04 / 3.47 / 6.822.10 / 3.28 / 5.55
Total duration (s)417387
Mean raw clip length before trimming (s)5.415.05
Mean leading silence removed (s)1.401.25
Silence kept after speech (s)1.00 (all clips)1.00 (all clips)
Clips padded with silence to reach 1.0 s159

Table 20.

All files are 16 kHz, mono, 16-bit PCM WAV. Speech is located in each file from its own loudness: onset is detected 18 dB below the file’s loud-speech level, and each clip starts 0.10 s before the onset and ends 1.0 s after the end of speech (padded with silence if the source ends sooner). Eight clips flagged by an automatic content check were listened to and matched the written text. The preprocessing log (one row per clip, with source file, durations and flags) is included in the release. Durations are shown to two decimals; where the third decimal is exactly 5, the rounding may go either way, so exact values follow: Pidgin minimum 2.045 s, median 3.475 s, maximum 6.815 s, total 416.990 s; Camfranglais minimum 2.105 s, median 3.280 s, maximum 5.555 s, total 387.190 s.

References

  1. [1]David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Az-ime, Jian Yun Zhang, Jesuoba O. Alabi, Xuanyi He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, et al. 2025. IrokoBench: A new benchmark for African languages in the age of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2732–2757. Association for Computational Linguistics.
  2. [2]Ruth-Ann Armstrong, John Hewitt, and Christopher Manning. 2022. JamPatoisNLI: A Jamaican patois natural language inference dataset. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5307–5320, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.arxiv.org/abs/2212.03419
  3. [3]Miriam Ayafor and Melanie Green. 2017. Cameroon Pidgin English: A Comprehensive Grammar, volume 20 of London Oriental and African Language Library. John Benjamins Publishing Company, Amsterdam.
  4. [4]Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.arxiv.org/abs/1508.05326
  5. [5]Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.arxiv.org/abs/1911.02116
  6. [6]Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.arxiv.org/abs/1809.05053
  7. [7]Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, and Bhiksha Raj. 2025. Audio entailment: Assessing deductive reasoning for audio understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23769–23777.
  8. [8]George Echu. 2003. Influence of Cameroon Pidgin English on the linguistic and cultural development of the French language. IULC Working Papers, 3(1).
  9. [9]George Echu. 2006. Pidginization of the French language in Cameroon. TRANS: Internet-Zeitschrift für Kulturwissenschaften, 16.
  10. [10]Nicholas G. Faraclas. 2013. Nigerian Pidgin. In Susanne Maria Michaelis, Philippe Maurer, Martin Haspelmath, and Magnus Huber, editors, The Survey of Pidgin and Creole Languages, Volume I: English-based and Dutch-based Languages, pages 176–184. Oxford University Press, Oxford.
  11. [11]Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.arxiv.org/abs/1803.02324
  12. [12]Alfred Malengo Kondoro, Sharon Ibejih, Cynthia Amol, et al. 2026. Afri-XNLI+: Extending the XNLI dataset with new African language translations. Dataset, Tonative (Hugging Face).
  13. [13]Jean-Paul Kouega. 2003. Camfranglais: A novel slang in Cameroon schools. English Today, 19(2):23–29.DOI
  14. [14]M. Paul Lewis, Gary F. Simons, and Charles D. Fennig, editors. 2014. Ethnologue: Languages of the World, 17th edition. SIL International, Dallas, Texas.
  15. [15]Shamsuddeen Hassan Muhammad, David Ifeoluwa Adelani, Sebastian Ruder, Ibrahim Sa’id Ahmad, Idris Abdulmumin, Bello Shehu Bello, Monojit Choudhury, Chris Chinenye Emezue, Saheed Salahudeen Abdullahi, Anuoluwapo Aremu, Alípio Jorge, and Pavel Brazdil. 2022. NaijaSenti: A Nigerian Twitter sentiment corpus for multilingual sentiment analysis. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 590–602, Marseille, France. European Language Resources Association.arxiv.org/abs/2201.08277
  16. [16]Chibuzor Okocha and Christian Grant. 2026. Afrispeech semantics: Evaluating audio-semantic reasoning in spoken language models across domains and accents. In Findings of the Association for Computational Linguistics: ACL 2026, pages 6909–6928, San Diego, California, United States. Association for Computational Linguistics.
  17. [17]Gabriel Ozón, Miriam Ayafor, Melanie Green, and Sarah Fitzgerald. 2017. The spoken corpus of Cameroon Pidgin English. World Englishes, 36(3):427–447.DOI
  18. [18]Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics.arxiv.org/abs/1805.01042
  19. [19]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356.arxiv.org/abs/2212.04356
  20. [20]Anne Schröder. 2013. Cameroon Pidgin English. In Susanne Maria Michaelis, Philippe Maurer, Martin Haspelmath, and Magnus Huber, editors, The Survey of Pidgin and Creole Languages, Volume I: English-based and Dutch-based Languages, pages 185–193. Oxford University Press, Oxford.
  21. [21]Valentine Njende Ubanako. 2021. Is Camfranglais a new language? a review of current opinions. International Linguistics Research, 4(1):36–42.DOI
  22. [22]Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.arxiv.org/abs/1704.05426
  23. [23]Kofi Yakpo. 2024. West African Pidgin: World language against the grain. Africa Spectrum, 59(2):180–203.DOI

Paper details

vlnc:W_d3qbw9kk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents