WorksComputer Science
Prosodic Structure as a Critical Component in Phonemic Language Modelling at BabyLM Scale
Abstract
Human children learn language from continuous, prosodically structured speech long before they can read, yet almost all language models train on orthographic text. Building on previous phonemic BabyLM training (Goriely et al., 2024; arXiv:2410.22906), we ask whether supplying an estimate of the prosody infants exploit improves phonemic language modelling at BabyLM scale. We compare an orthographic baseline to three conditions forming a ladder of phonological richness. All conditions are trained as BERT MLMs at 10M and 100M words. Addressing the ecological validity of existing metrics in the small-data regime, we build a lexical decision task and two semantic organisation tasks. Our new lexical decision task offers improved insights into the distinction between wordlikeness, token familiarity, and lexicality as factors. Triphone phonotactic control and pseudo-log-likelihood scoring shows that these scores reflect phonotactic familiarity rather than wordhood: every model sits near or below chance on an adversarial inverted condition, revealing prior results of this kind as partly artefactual. On our new semantic tasks, a prosodically annotated phoneme stream gives the strongest taxonomic organisation at both the 10M and 100M word scales. Phonemic models with boundary structure are competitive with the orthographic baseline. These findings suggest that prosodic structure is a key component of phonemic language modelling in the small-data regime most analogous to infant learning.
Introduction
Human children reach near-adult linguistic competence from a few tens of millions of words of input, whereas state-of-the-art language models require several orders of magnitude more data [12] (Warstadt et al. 2023). The BabyLM Challenge (Warstadt et al., 2023) reframes this gap as a question of sample efficiency, capping training at developmentally plausible budgets. A second and equally basic difference concerns the modality of the signal. Children understand and produce language years before they learn to read: they acquire language from a continuous stream of speech in which word boundaries are not marked and must be inferred. Almost all language models, and most BabyLM submissions, instead train on orthographic text, in which word boundaries are marked with whitespace.
Working in phonemic space is a natural response [13]. However, a plain phoneme stream discards prosody, a source of information that infants are known to use to bootstrap word segmentation and syntactic structure from the earliest stages of language acquisition. Here, we ask whether providing an estimate of the prosodic structure can enhance word learning from phonemic input at BabyLM scale. We compare an orthographic baseline (Ortho) to three conditions of increasing phonological richness: a plain continuous-phoneme reconstruction of Goriely et al. [13] (P-GORIELY); a condition that inserts intonation break tokens at major syntactic boundaries, as estimated from a dependency parse (P-BREAKS); and one that further adds sentence-type boundary tones from a neural annotator trained on prosodically labelled speech (PROSODY-NN). Each condition is trained as a BERT masked language model Devlin et al. [9] at two corpus scales, 10M and 100M words, yielding eight models.
Standard benchmarks use orthographic input. While conventions about the use of punctuation and whitespace in orthography reflect some prosodic features, such as the ends of utterances, they also reflect norms about which sequences are meaningful words. Some norms, such as the spacing around function words and elements of compounds, are arbitrary and language-specific – clearly going beyond what human infants learn on a 10M budget, and in some respects beyond what children have learned after 100M words of language exposure. A focus on learning from phonemic input thus aligns with the goals of the BabyLM project, in building models that draw inspiration from the efficiency of language learning by children.
Focusing on learning of the mental lexicon, we propose and evaluate three new tasks over child-relevant vocabulary. A phonotactically controlled lexical decision task scored by pseudo-log-likelihood is similar to the BabySLM task in Lavechin et al. [20]. However, we redesign this task to make a stronger distinction between wordlikeness (having the phonological form of a word) and lexicality (having a meaning associated with the form). Our semantic tasks, concept boundary discrimination and unsupervised category induction, are grounded in research on language development.
Our contributions are threefold. In our redesign of the lexical decision task, we aim to support strong comparisons amongst wordlikeness, token familiarity, and lexicality. Pseudoword candidates are generated with a trigram Markov chain and scored in the phonemic domain. A filter removes all candidates that are real words in either orthographic or phonemic form. Then, pseudowords are individually selected to best match the phonotactic scores of the real words in the test. We also introduce an adversarial Inverted test condition, in which each pseudoword has a higher phonotactic score than its real word competitor. The results show that pseudo-log-likelihood scoring probes token familiarity more than lexical knowledge, and that previously reported phonemic-model results on the lexical decision task are partly artefactual. Second, we show that providing estimated prosodic markers makes phonemic models competitive with, or better than, the orthographic baseline on age-relevant semantic organisation. Taken together, these results indicate that, since the infant’s input is speech rather than text, careful phonemic representation enriched with prosody is a key component in developmentally plausible language modelling.
Background and Related Work
Acquisition from continuous speech. Children reach near-adult linguistic competence from roughly 100 million words of input by early adolescence Warstadt et al. (2023). Before they learn to read, their input is speech: a continuous acoustic stream with no whitespace, no punctuation, and no explicit word boundaries. To build their mental lexicon, they segment words from fluent speech using distributional regularities Saffran et al. [26], and exploit prosodic structure to do so. The prosodic bootstrapping hypothesis Christophe et al. [7]; Jusczyk [18] holds that phrase boundaries and boundary tones provide a partial parse of the signal before any grammar is in place. In addition, they carry information about some aspects of meaning. English yes/no questions typically carry a high (rising) boundary tone while declaratives and wh-questions carry a low (falling) tone, a contrast that child-directed speech exaggerates Soderstrom et al. [30]. Dip hone statistics in combination with utterance boundaries are already rich enough to strongly support segmentation Daland and Pierrehumbert [8]. These considerations motivate modelling language from a phoneme stream rather than from text.
Phonemic language modelling and BabyLM. Most BabyLM submissions train on orthographic text. Goriely et al. [13] is a notable exception. They convert the BabyLM corpus to IPA with Duddington and Dunn [10], remove word separators to form a continuous stream of phonemes, and train a GPT-2 model, finding phonemic models to be fairly competitive with orthographic baselines. We extend this framework in three ways: we use the masked language model BERT throughout Devlin et al. [9]. Its bidirectional objective is more data-efficient for the short utterances that are typical in the training set. We make prosodic structure a primary experimental variable rather than discarding it; and we redesign the lexical decision task to remove a confound in prior evaluations.
Evaluating the mental lexicon. A word is an associative pair of a phonological form and a meaning Needle et al. [23]. So, a lexical decision task should test whether a form is meaningful, not merely whether it is phonotactically word-like. Standard benchmarks serve this end imperfectly. The speech-oriented BabySLM (Lavechin et al., [20]) uses the pipeline in Keuleers and Brysbaert [19] and Nguyen et al. [24] to design pseudowords for the forced-choice lexical decision task reported in Goriely et al. [13]. This pipeline does not control for triphone phonotactics, which are known to play an important role in wordlikeness judgments ([1, 14, 23]).
We control for this factor, which intrinsically encompasses some aspects of syllable structure (though not all the aspects of syllable structure that BabySLM controls for). Overall, real words on average are expected to be phonotactically better than nonwords, because the encoding, memory, and production of lexical items is biased towards high phonotactic likelihood ([14]). Through repeated learning and use, forms that include less likely sequences tend to be regularized towards more typical forms. A matched-foil design that does not control for triphone likelihood thus retains a systematic asymmetry. A post-hoc analysis of the BabySLM test set shows that the real word has better triphone phonotactics than the matched pseudoword in 70.3% of the test pairs (see Appendix § 11). Thus, a model could score above chance simply by preferring the phonotactically better-formed item, without storing form-meaning associations. The key distinction between wordlikeness and lexicality is probed in our Inverted category (§ 4.1).
The GLUE (Wang et al., [32]) and BLiMP (Warstadt et al., 2020) tasks are used in BabyLM papers to probe the learnability of syntax and semantics. After remapping the materials to a phonemic representation, Goriely et al. [13] reports accuracies in the range of 64% to 78% on BLiMP and GLUE for their various conditions and LMs, trained on 100M words. It is difficult to disentangle how much the errors result from weaknesses in the LMs versus intrinsic limitations of the training set size. Some patterns probed in these datasets are obviously not learned by humans from 10M words. For example, the word any is learned only at age 4 or 5 (Tieu and Lidz, [31]), reflecting some 40M to 50M words of exposure. For a training budget of 64M tokens Warstadt et al. (2020) report accuracies of about 50% for words like any, and about 40% for syntactic island constraints. Given this situation, we judged that the development of new simple semantic tasks with high face validity would best further our aim of evaluating how well prosodic information can support learning of lexi-
cality.
Experimental Conditions
The orthographic text condition is included as a baseline. The phonological conditions form a ladder of increasing phonological richness, from the continuous phoneme stream as defined in Goriely et al. [13] to a phoneme stream annotated with phrase boundaries and sentence-type boundary tones (Table 1). All conditions are produced by a single preprocessing pipeline parameterised per condition, so only the input representation differs across them. Every condition is trained identically at both the 10M and 100M scales.
| Condition | Added structure |
| ORTHO | orthographic text (baseline) |
| P-GORIELY | continuous IPA; [UTT] at utterance ends |
| P-BREAKS | + [B2], [B4] phrase boundaries |
| PROSODY-NN | + [H%], [L%] boundary tones |
Table 1. The four conditions, organised as a baseline plus three phonemic conditions of increasing richness. The phonemic conditions carry no whitespace.
Corpus
All conditions are trained on the BabyLM 2025 corpus (Charpentier et al., [6]), which draws on six English sources (CHILDES, Project Gutenberg, OpenSubtitles, Simple Wikipedia, the spoken BNC, and Switchboard). Roughly half of the tokens are transcripts of conversational or child-directed speech, but the prosodic features were not transcribed, so our study estimates them (see § 3.3). The full composition is given in Appendix § 10.
The Four Conditions
ORTHO is lowercased orthographic text and serves as the baseline. P-GORIELY reconstructs the approach of Goriely et al. [13]: the corpus is transcribed to a continuous IPA stream with word separators and punctuation removed, leaving only phonemes with an utterance-boundary token [UTT]. P-BREAKS adds two levels of the ToBI break-index hierarchy (Beckman and Ayers Elam, [2]), an intermediate-phrase token [B2] and an intonation-phrase token [B4], derived from a syntactic parse rather than punctuation; it is the first condition in which a structural token can appear in a sentence with no surface punctuation. PROSODY-NN keeps these break tokens and additionally marks sentence-final intonation with a high (rising) boundary tone [H%] or a low (falling) tone [L%], assigned by a neural annotator. Table 2 shows all four conditions on three example sentences chosen to expose the two thresholds: S3 triggers a [B2] at a subordinate-clause onset with no comma, and S2 receives [H%] as a yes/no question.
| Condition | Rendered output |
| S1: “The dog sat on the mat.” | |
| ORTHO | the dog sat on the mat . |
| P-GORIELY | ðədɒɡsætɒnðəmæt [UTT] |
| P-BREAKS | ðədɒɡsætɒnðəmæt [B4] |
| PROSODY-NN | ðədɒɡsætɒnðəmæt [B4] [L%] |
| S2: “Did she eat?” | |
| ORTHO | did she eat ? |
| P-GORIELY | dɪdʃiːiːt [UTT] |
| P-BREAKS | dɪdʃiːiːt [B4] |
| PROSODY-NN | dɪdʃiːiːt [B4] [H%] |
| S3: “He left because it was late.” | |
| ORTHO | he left because it was late . |
| P-GORIELY | hiːlɛftbɪkɒzɪtwɒzleɪt [UTT] |
| P-BREAKS | hiːlɛft [B2] bɪkɒzɪtwɒzleɪt [B4] |
| PROSODY-NN | hiːlɛft [B2] bɪkɒzɪtwɒzleɪt [B4] [L%] |
Table 2. Three example sentences rendered under each condition. S3 shows a [B2] inserted at a subordinate-clause onset with no comma; S2 shows the [H%]/[L%] sentence-type distinction.
Phonemic Transcription and Prosodic Annotation
The phonemic conditions are transcribed with eSpeak-NG [10] through the phonemizer library [3], with the word separator set to empty so the output is a continuous IPA stream with no explicit word boundaries. Prosodic tokens are inserted by a two-pass scheme that phonemises each word separately and interleaves boundary tokens at the recorded positions. In P-BREAKS, [B2] and [B4] are derived from a spaCy dependency parse [15]; in PROSODY-NN, the boundaries and sentence-final tones ([H%]/[L%]) are predicted by a bert-base sequence labeller trained on the ToBI-annotated Boston University Radio News Corpus [25], with the tone head gated on held-out and a rule-based fallback. (For the annotator and gating thresholds, see Appendix § 10.)
Tokenisation
A Byte-Pair Encoding tokeniser [28] is trained from scratch for each condition, on its preprocessed corpus; the character inventory and merge statistics of a prosodically annotated IPA stream differ substantially from orthographic English. Vocabulary size is 4,000 at 10M and 8,000 at 100M. For the roughly 50-symbol IPA alphabet, BPE yields primarily diphone and triphone units. All structural tokens ([B2], [B4], [H%], [L%], [UTT], and the reserved BERT tokens) are filtered from the corpus before BPE training and inserted at fixed vocabulary ids afterwards. This prevents the algorithm from learning spurious merges inside token strings. Annotating the stream lengthens it: The annotated conditions carry roughly 10% more tokens than P-GORIELY, though every phonemic condition still consumes fewer tokens than ORTHO. Appendix § 15 reports the token counts and Appendix § 14 the segmentation behaviour.
Model Architecture and Training
A BERT-family masked language model [9] is trained from scratch for each condition. BERT was selected due to its data-efficiency at this scale. The model size is matched to corpus scale: a BERT-small configuration (7.7M parameters) at 10M and a BERT-base configuration (92M parameters) at 100M. All conditions share the same training recipe; full architecture and training hyperparameters are given in Appendix § 9.
Evaluation
We evaluate all eight models on three purpose-built tasks, each defined in the same representational space as training and restricted to infant-relevant vocabulary: a phonetically controlled lexical decision task (§ 4.1) and two semantic tasks: concept boundary discrimination (§ 5.1) and unsupervised category induction (§ 5.1.1). All tasks read from the frozen trained models without any fine-tuning.
Lexical Decision
The lexical decision task asks whether a model prefers a real word to a phonetically matched pseudoword (except for Inverted category pairs). Because the two items in a pair are matched on phonotactic plausibility (or, by deliberate design, favours the pseudoword for the Inverted category) under the model’s own training distribution tion, above-chance accuracy is evidence of stored lexical knowledge rather than phonotactic filtering alone.
Phonotactic scoring. Each condition has its own trigram () phonotactic model, estimated from that model’s training dictionary. Following Needle et al. [23], counts are type-based: each word type contributes once regardless of corpus frequency, so the model captures phonotactic licitness rather than lexical frequency. The phonotactic score of a word with phone sequence is the log-probability of its padded form under a Witten-Bell smoothed model (Witten and Bell, 1991):
Witten-Bell smoothing interpolates each context with its lower-order backoff so that unseen trigrams in the test set still receive probability mass. Writing for a context, for the count of following , , and for the number of distinct continuations of ,
where drops the leftmost symbol of and the recursion bottoms out at a uniform unigram floor. The backoff weight grows with the diversity of observed continuations , allocating more probability to novel phones in contexts that have shown lexical variety. For ORTHO the scorer operates on characters.
Pseudoword generation. For practical reasons, pseudowords are first generated in the orthographic space. Then they are mapped into the phonemic space using eSpeak-NG, and scored using the triphone Markov model described above. Pairing is carried out once, in the phonemic space of P-GORIELY, to ensure every condition is tested on identical items. This means that the pseudowords are created to match the real-word phonotactic distribution by construction and lets us control their score directly, which the Inverted category requires. Given a context , the next character is sampled with a temperature :
generating autoregressively from until is emitted. Each candidate is then phonemised with eSpeak-NG and scored with the model above, and it is that phonemic score, not the proposal, which decides what enters the test set. Lowering the temperature to concentrates proposals on frequent transitions, yielding a pool dense enough in high-scoring forms to supply the Inverted category. The full procedure is given in Appendix § 11.
Items are pairs of a real word and a pseudoword. Two filters guarantee that the items are valid. Filter A admits a real word only if it is a genuine English word, that is, attested in a reference word list after lemmatisation; this removes corpus artefacts, such as https or repeated-character strings that survive a purely alphabetic filter and would otherwise be labelled real words. Filter B admits a pseudoword only if neither its orthographic form nor its eSpeak IPA form (raw, and with stress and length marks stripped) occurs in the training set, so that no pseudoword is in fact a word the model has seen, whether spelled differently from a real homophone or transcribed identically to one. Rejected candidates are resampled so category sizes are held fixed. This phonemic screen is essential: without it, a large share of candidate pseudowords coincide with real words in IPA space and are wrongly counted as model errors. Contamination statistics are reported in Appendix § 12. For the ORTHO condition, the real word is paired with the original orthographic form of the pseudoword, from which the phonemic form was generated. (Using the same test items across all conditions means that the orthotactic likelihood is not fully controlled).
Pair categories. Real words fall into three categories. LexicalMemory words are a single token in the condition’s BPE vocabulary, so the model can score the whole word in one step. Composition words span multiple tokens, requiring the model to compose a representation from sub-word units. Inverted pairs are adversarial: the real word is drawn from the bottom of the phonotactic-score distribution and the pseudoword from the top, so that by construction. A model relying only on phonotactics will therefore prefer the pseudoword and score at or below chance on Inverted; only stored lexical knowledge that overrides phonotactic expectation can raise accuracy above chance. Standard, non-inverted pairs are matched to within 2.0 log-probability units. The evaluation set is built once on orthographic words and derived for the phonemic conditions by phonemising the same pairs, so every model is tested on identical word content; it contains 4,129 pairs, of which 2,000 are Inverted, with the split between LexicalMemory and Composition depending on each condition’s vocabulary.
Metric. Each model is scored by pseudo-log-likelihood (PLL; Salazar et al. [27]), the natural scoring rule for a masked language model. For a word tokenised as ,
the sum of masked-token log-probabilities across the word. A pair is judged correct when . Accuracy is reported separately for each category; chance is 0.50.
Semantic Tasks
Both semantic tasks operate on contextualised embeddings of 68 early-vocabulary words drawn from the MacArthur-Bates Communicative Development Inventories (Fenson et al. [11]), spanning six categories (animals, food, body parts, clothing, vehicles, and household objects); the full list is given in Appendix § 13. For each target word we collect up to 30 short excerpts where it occurs from the CHILDES set (MacWhinney [22]). We render each excerpt into the surface form for the condition, with the frozen model. We use the target-anchored representation: the per-excerpt vector is the mean of the final-layer hidden states at the subword positions of the target word, and the word vector is the mean over all excerpts. Writing for the target’s subword positions in excerpt with hidden states ,
Anchoring on the target word, rather than mean-pooling the whole excerpt, isolates the word’s representation from the sentence topic and is what makes the measure word-specific.
Concept boundary discrimination (CBD)
Concept boundary discrimination (CBD) is a 3-way ranking task. For each target word, a probe presents three alternatives: a same-category word, a thematic foil from a different category but linked to the target by a non-taxonomic relation, and an unrelated foil. Two words are defined as thematically related when they stand in a script or event association, a production or source relation, a containment or part-whole relation, a spatial co-occurrence, or a functional relation, and not through shared category membership (Lin and Murphy [21]; Imai et al. [17]). Each of the 68 targets is assigned a single thematic foil by a hand-crafted lookup; because the foil must itself be a CDI word to have CHILDES excerpts, roughly 30 targets yield a valid foil, producing 300 probes at ten draws per word. Table 3 gives representative foil pairs across the five relation families.
| Target | Thematic foil | Relation | CDI |
| Script / event | |||
| cat | milk | typical food | ✓ |
| fish | water | habitat | ✓ |
| dog | bone | typical food | † |
| Production / source | |||
| cow | milk | product | ✓ |
| egg | bird | source | ✓ |
| Containment / part-whole | |||
| milk | bottle | container | ✓ |
| shoe | foot | worn on | ✓ |
| hand | contains | ✓ | |
| Spatial / locational | |||
| boat | water | location | ✓ |
| car | road | location | † |
| Functional / instrumental | |||
| eye | look | function | † |
| cheese | mouse | typical eater | ✓ |
Table 3. Representative thematic foil pairs by relation family. Each foil is a different-category word tied to the target by a non-taxonomic relation and, to have CHILDES excerpts, must itself be a CDI target. Entries marked † have no CDI-resident foil and contribute no probes.
The model output is the cosine similarity of each of the three alternatives to the probe word. The measure of interest is the ranking of these three alternatives. The cosine similarity is defined by the following equation (with maximum similarity having the value 1.0).
We report three accuracies: three-way (the same-category word preferred over both foils, chance ), taxonomic-versus-thematic, and taxonomic-versus-unrelated (the same-category word preferred over the thematic or unrelated foil respectively, chance . Taxonomic-versus-thematic is the primary metric: it isolates taxonomic organisation from thematic association, a distinction distributional models often fail to make. A category neighbour (e.g. cat for dog) should be preferred to a merely thematic associate (e.g. bone). Writing , and for the cosine similarity between and the same-category word, the thematic foil and the unrelated foil respectively, a probe is scored correct when:
We also report the mean taxonomic advantage: the average margin by which the same-category similarity exceeds the thematic-foil similarity,
Semantic Category Induction (SCI)
Semantic category induction (SCI) tests whether word vectors, clustered without supervision, recover the six ground-truth categories. The aggregated vectors of all 68 words are clustered into groups with both -means (ten random initialisations) and agglomerative clustering under Ward’s linkage (Ward, 1963). Cluster quality is measured against the category labels by the Adjusted Rand Index (ARI; Hubert and Arabie, [16]) and Normalised Mutual Information (NMI), with -means ARI as the primary metric. The ARI corrects the Rand Index (RI) for chance agreement,
so that ARI corresponds to random labelling and ARI to perfect categorisation. SCI is evaluated on the words observed in training. It tests only how the model has organised the vocabulary it has actually seen.
Results
We report results for the three tasks, at both the 10M and 100M scales. All figures are read from the frozen models.
Lexical Decision
Table 4 reports PLL accuracy by pair category. On LexicalMemory, every condition is near ceiling (0.930 to 0.991). For Composition, the results range from 0.768 to 0.819; the phonemic conditions perform similarly to ORTHO, with P-GORIELY highest at 10M and ORTHO highest at 100M. On the Inverted category, performance for every condition is poor. Values sit close to the 0.50 chance level at 10M (0.479 to 0.506); at 100M ORTHO remains at chance (0.496) while the three phonemic conditions fall below it, to between 0.396 and 0.444.
| Condition | LexMem | Comp | Inv. |
| 10M | |||
| ORTHO | 0.991 | 0.768 | 0.499 |
| P-GORIELY | 0.942 | 0.819 | 0.506 |
| P-BREAKS | 0.948 | 0.799 | 0.489 |
| PROSODY-NN | 0.930 | 0.800 | 0.479 |
| 100M | |||
| ORTHO | 0.980 | 0.813 | 0.496 |
| P-GORIELY | 0.988 | 0.801 | 0.396 |
| P-BREAKS | 0.982 | 0.808 | 0.444 |
| PROSODY-NN | 0.979 | 0.807 | 0.421 |
Table 4. PLL accuracy on the lexical decision task ( pairs) by pair category. Chance is 0.50. Bold marks the best condition in each column within a scale.
Semantic Tasks
Table 5 reports Concept Boundary Discrimination accuracy over 300 probes. On the primary taxonomic-versus-thematic metric, PROSODY-NN is highest at both scales, and no other condition exceeds 0.6; PROSODY-NN also has the largest mean taxonomic advantage at both scales. ORTHO leads the easier taxonomic-versus-unrelated comparison at both scales. P-BREAKS is lowest on taxonomic-versus-thematic at 10M, at 0.490. Figure 1 plots the taxonomic-versus-thematic accuracy across scales.
| Condition | 3-way | Tv-them | Tv-unrel | Adv |
| 10M | ||||
| ORTHO | 0.463 | 0.537 | 0.693 | +0.019 |
| P-GORIELY | 0.420 | 0.560 | 0.613 | +0.028 |
| P-BREAKS | 0.400 | 0.490 | 0.600 | +0.009 |
| PROSODY-NN | 0.483 | 0.607 | 0.670 | +0.040 |
| 100M | ||||
| ORTHO | 0.533 | 0.570 | 0.860 | +0.019 |
| P-GORIELY | 0.477 | 0.583 | 0.737 | +0.022 |
| P-BREAKS | 0.443 | 0.540 | 0.703 | +0.013 |
| PROSODY-NN | 0.533 | 0.610 | 0.747 | +0.027 |
Table 5. CBD accuracy ( probes; SE ≈ 0.028 on Tv-them). -them: taxonomic vs. thematic (primary); -unrel: taxonomic vs. unrelated; : mean taxonomic advantage. Chance is for 3-way and 0.50 for the pairwise columns. Bold marks the best condition in each column within a scale.
Figure 1. Taxonomic-versus-thematic accuracy at 10M and 100M for the four conditions. The dashed line marks chance (0.50).
For Semantic Category Induction, Table 6 reports cluster recovery over the 68 words. At 10M, ORTHO leads on all three measures (K-ARI 0.483), with the phonemic conditions between 0.279 and 0.303. At 100M the ordering changes: P-BREAKS (K-ARI 0.637) and PROSODY-NN (0.617) exceed ORTHO (0.576) on both -means measures, while ORTHO leads on agglomerative ARI (0.717). Figure 2 shows the 10M-to-100M trajectories, in which P-BREAKS and PROSODY-NN cross above ORTHO. The detailed embedding geometry could be found in Appendix § 16.
| Condition | K-ARI | K-NMI | A-ARI |
| 10M | |||
| ORTHO | 0.483 | 0.601 | 0.337 |
| P-GORIELY | 0.279 | 0.439 | 0.257 |
| P-BREAKS | 0.290 | 0.417 | 0.260 |
| PROSODY-NN | 0.303 | 0.454 | 0.259 |
| 100M | |||
| ORTHO | 0.576 | 0.681 | 0.717 |
| P-GORIELY | 0.551 | 0.666 | 0.503 |
| P-BREAKS | 0.637 | 0.750 | 0.530 |
| PROSODY-NN | 0.617 | 0.732 | 0.610 |
Table 6. SCI cluster-recovery scores ( words, ). -ARI, -NMI: -means ARI and NMI (K-ARI primary); -ARI: agglomerative ARI. Bold marks the best condition in each column within a scale.
Figure 2. -means ARI at 10M and 100M for the four conditions. P-BREAKS and PROSODY-NN cross above ORTHO at 100M; the dashed line marks ARI = 0.
Discussion
Lexical Decision
Pseudo-log-likelihood is strongly influenced by phonotactic familiarity. The near-ceiling accuracy for LexicalMemory reflects only the fact that a single-token real word receives strong probability under virtually any model. The accuracy for Composition is lower, but still above chance, because the real word’s multi-token sequence is partially learned while the pseudoword’s multi-token sequence is unseen. The Inverted category is decisive: every condition sits near or below chance. This means that the task does not incisively probe lexicality; high wordlikeness can cancel out the partial ability to learn meaningfulness, that was seen in the Composition category. At 100M the
phonemic conditions fall below chance while ORTHO stays at chance: the larger model has learned its input’s phonotactics more sharply and scale deepens PLL’s phonotactic dependence rather than relieving it. These findings suggest that the high lexical accuracy in Lavechin et al. [20] for the PLL lexical decision task is partially an artefact of the BabySLM stimulus design.
Semantic Organisation
For the CBD task, PROSODY-NN beats the orthographic and plain-phonemic baselines on taxonomic-versus-thematic discrimination at both scales (Figure 1). For the SCI task, it rises above ORTHO at 100M (Figure 2). PROSODY-NN’s performance indicates that its boundary and tone tokens reshape the embedding space toward categorical rather than merely thematic similarity. ORTHO keeps the lead only on the easier taxonomic-versus-unrelated comparison, where co-occurrence alone separates a category neighbour from an unrelated word.
The behaviour of P-BREAKS is striking. It is the weakest condition on taxonomic-versus-thematic, yet on cluster quality it matches or exceeds PROSODY-NN and rises above ORTHO at 100M, where it posts the highest -means ARI. Break tokens therefore appear to supply a segmentation scaffold that yields well-separated clusters, but on some basis other than taxonomy. The categorical sharpening that CBD rewards appears to come from the sentence-type tone annotation that distinguishes PROSODY-NN from P-BREAKS, not from the break structure alone.
A second conspicuous pattern is the slope of the scale effect. Every condition improves from 10M to 100M. However, the phonemic conditions climb far more steeply than ORTHO (Figure 2), converting a ten-fold increase in data into categorical organisation more efficiently than ORTHO does. We read this as a difference in what each representation still has to gain. Whitespace gives ORTHO its word units for free, so its category structure rests on the co-occurrence with other words. By 10M, it has obtained almost as much advantage from the whitespace as is possible. In contrast, the phonemic conditions must induce the word boundaries, and prosodic tokens mark the phrase edges that shape which co-occurrences count. By 100M words, it appears that the phonemically-induced word boundaries may actually be better than those marked by whitespace in ORTHO, overcoming some of the idiosyncracies of English spelling and punctuation. The margins on any single probe set are small (the binomial standard error at is about 0.028), but the pattern is consistent across both tasks, both scales, and the more robust taxonomic-advantage measure; the tighter, isotropic clusters of the boundary-annotated conditions are visible in Appendix § 16.
Conclusion
Goriely et al. [13] established that training on continuous phonemic streams is a strong competitor to orthographic text at BabyLM scale. Our work extends that framework with explicit prosodic structure, a phonotactically controlled evaluation, and a comparison across two scales over four conditions. Two findings stand out. First, our redesigned lexical decision task shows that pseudo-log-likelihood scoring is strongly influenced by phonotactic likelihood rather than probing lexical knowledge alone: every model sits around or below chance on the adversarial Inverted category, and accuracy moves with the phonotactic balance rather than with lexicality alone. Second, on the semantic tasks, prosodic annotation makes phonemic models competitive with, or in some cases better than, the orthographic baseline.
Together these results indicate that, in the data-limited regime most analogous to infant learning, the structure and naturalness of the input representation matters. Orthographic text is not a neutral default but a cognitive artefact whose statistics a model can exploit. Because the child’s input is speech and not text, a carefully constructed phonemic representation enriched with prosody is a key component of developmentally plausible language models, and one that future BabyLM work can build on.
Limitations
An important limitation of this study is that the prosodic annotation is not observed from speech. Instead, it is estimated from text. The boundary and tone tokens are estimated using a dependency parser and a BERT sequence labeller, which was trained on the Boston University Radio News Corpus. Further, the read broadcast-news register of this corpus differs from the conversational and child-directed speech that dominates the BabyLM corpus. Our annotations are therefore an approximation of the prosody an infant hears, and the gains we report are plausibly a lower bound on what genuine speech-based training would yield. Phonemisation adds a further approximation: eSpeak-NG applies a single American-English accent and rule-based grapheme-to-phoneme conversion, which introduces systematic transcription errors shared across all phonemic conditions.
The study used BPE encoding of the input, as this encoding is used in the baselines to which we make comparisons. When the white space marking word boundaries is removed, BPE encoding of an utterance can result in single tokens that bridge a (de facto) word boundary; see Appendix § 14 for further details. We do not consider alternative character-based tokenisation schemes. These have provided promising results in some BabyLM studies [13, 5, 4]. Exploration of character-based models for our experimental conditions and tasks is an important direction for future research.
The evaluations are rather narrow. The semantic tasks use 68 high-frequency CDI words in six broad categories, with hand-curated thematic foils yielding 300 probes, so the margins on any single metric are small and the categories may miss finer-grained structure; low-frequency and out-of-vocabulary generalisation is untested. All models are trained with a single random seed, so we report no variance estimates, and because the two scales differ in both corpus size and model capacity, scale effects conflate the two and support only within-scale condition rankings. Finally, our lexical decision analysis relies on pseudo-log-likelihood alone; a discriminative probe of the same representations, and a full account of why P-BREAKS forms high-quality clusters on a non-taxonomic basis, are left to future work, as is the natural step of replacing text-derived annotation with acoustic prosody from speech or self-supervised speech units.
Acknowledgments
We are grateful to Marvin Lavechin for his prompt and helpful responses to our questions about the construction of the BabySLM dataset. We also appreciate the helpful input from members of the Pierrehumbert Language Modelling Group over the course of the project.
Model Architecture and Training
| Hyperparameter | 10M | 100M |
| Architecture | ||
| Variant | BERT-small | BERT-base |
| Hidden size | 256 | 768 |
| Layers | 8 | 12 |
| Attention heads | 8 | 12 |
| FFN size | 1,024 | 3,072 |
| Vocabulary | 4,000 | 8,000 |
| Parameters | ≈7.7M | ≈92M |
| Training | ||
| Epochs | 10 | |
| Max sequence length | 256 | |
| Batch size / device | 1,024 | 128 |
| Gradient accumulation | 1 | 4 |
| Peak learning rate | ||
| LR schedule | cosine, 10% warmup | |
| Optimizer | AdamW | |
| Weight decay | 0.01 | |
| Gradient clipping | 1.0 | |
| Precision | bfloat16 | |
| MLM masking | 15% | |
| Random seed | 42 |
Table 7. Model architecture and training hyperparameters at the two scales. Architecture dimensions differ by scale; training settings are shared except batch size, gradient accumulation, and peak learning rate.
Training Corpus and Prosodic Annotation
Corpus composition. All conditions are trained on the BabyLM 2025 corpus ([6]); Table 8 gives the composition, sampled at the same proportional weights at both scales.
| Source | 10M | 100M | Share |
| CHILDES (child-directed) | 2.8M | 28.9M | 29% |
| Project Gutenberg (prose) | 2.5M | 26.4M | 26% |
| OpenSubtitles (dialogue) | 2.0M | 20.0M | 20% |
| Simple Wikipedia | 1.5M | 14.7M | 15% |
| BNC Spoken (conversation) | 0.9M | 7.8M | 8% |
| Switchboard (telephone) | 0.1M | 1.3M | 1% |
| Total | 9.95M | 99.0M | 100% |
Table 8. BabyLM training corpus composition (approximate word counts).
ToBI inventory. The prosodic tokens are drawn from the Tone and Break Indices framework ([29, 2]). We use two levels of the break-index hierarchy: [B2] for an intermediate-phrase boundary (ToBI break index 2) and [B4] for an intonation-phrase boundary (index 4); indices 0, 1, and 3 are not used. We use two boundary tones: [H%], a high (rising) tone associated with yes/no questions, and [L%], a low (falling) tone associated with declaratives, imperatives, and wh-questions. Any annotation method must be text-only, deterministic, fully automatic, and transparent, since no paired audio exists for the BabyLM sources.
Rule-based break annotation (P-BREAKS). Each sentence is parsed by spaCy’s en_core_web_sm dependency model ([15]) (NER and lemmatisation disabled for throughput). [B4] is inserted after any token that is a sentence boundary or bears terminal punctuation (., ?, !) and takes precedence over [B2] at the same position. [B2] is inserted, when [B4] has not been assigned, under the five syntactic cases in Table 9. For sentences of three or fewer whitespace words, cases A–C and E are suppressed and only case D and the sentence-final [B4] apply, reflecting the lower parse reliability and negligible internal structure of short conversational turns. Because standard phonemisation concatenates the utterance, tokens are placed by a two-pass scheme: each word is phonemised separately with a temporary separator, then the boundary tokens are interleaved at the recorded positions.
Neural annotator (PROSODY-NN). PROSODY-NN replaces the rule-based break detector with a trained sequence labeller while keeping the
| Rule | Trigger |
| A | Coordinating clause junction: a coordinating conjunction heads a clause-level cc dependency on a verbal or auxiliary root. |
| B | Subordinate or complement clause onset: a subordinating conjunction or complementiser marks an adverbial, complement, or embedded clause. |
| C | Relative clause onset: a relative pronoun or the relcl arc. |
| D | Minor surface punctuation: comma, semicolon, colon, dash, or ellipsis. |
| E | Appositive or parenthetical nominal: a token bearing the appos dependency and not preceded by punctuation. |
caption: Syntactic cases that trigger a [B2] token in P-BREAKS.
:::
same token vocabulary and two-pass pipeline. The annotator is trained on the Boston University Radio News Corpus [25], the only large manually ToBI-annotated American English corpus, comprising roughly 3,600 sentences and 113,000 word tokens of read radio news after excluding one speaker who lacks break annotation. Break indices are collapsed to {NONE, B2, B4} and tone events (mapped to the nearest following word boundary) to {NONE, L%, H%}; splits are speaker-disjoint. The model is a bert-base-uncased backbone [9] with two linear heads over the first-subword hidden state of each word, trained to minimise a joint class-weighted cross-entropy with weights [1, 3, 3] (break) and [1, 4, 8] (tone); the elevated [H%] weight compensates for the rarity of yes/no questions in read news. Training uses AdamW at learning rate with early stopping on held-out macro-.
Each head is used for corpus annotation only if it clears a held-out gate (Table 10); the break head otherwise falls back to the rule-based P-BREAKS pipeline and the tone head to the heuristic below. The final annotator clears every threshold, so both heads are shipped. Inference over the full 100M corpus takes about an hour on a single GPU.
| Head | Ship threshold | |
| [B2] | 0.58 | |
| [B4] | 0.99 | |
| [L%] | 0.69 | |
| [H%] | 0.56 |
Table 10. Held-out of the neural annotator against the shipping thresholds. All four clear their gate, so the full annotator is used.
Rule-based tone heuristic (fallback). When the tone head fails its gate, sentence-final tone is assigned by a three-gate rule, which is also the source of the boundary tones’ definition. A sentence whose final non-bracket character is not ? receives [L%]. Otherwise, the first substantive word is found by skipping up to five leading discourse markers, fillers, and response tokens: a wh-word
receives [L%], a fronted positive auxiliary or copula receives [H%] (contracted negations are excluded as they often head echo questions), and any remaining question receives [L%] conservatively. In our run the tone head cleared its gate, so this heuristic was not triggered.
Pseudoword Generation and Phonotactic Control
Generation: a proposal stage and a selection stage. Pseudowords are produced in two stages. The proposal stage supplies candidate forms and the selection stage decides which candidates enter the test set. The proposal stage is a trigram Markov chain over orthographic characters, trained on the alphabetic types of the training vocabulary, sampled autoregressively from until with a temperature that reshapes the empirical transition counts as . Candidates are constrained to between four and seven characters. Each candidate is then phonemised by the same eSpeak-NG pipeline that produced the training corpora, and every subsequent step operates on the resulting IPA form: the collision filters, the phonotactic score , the length and score matching, and the construction of the Inverted category. The chain is therefore a proposal distribution only. None of the properties the test set relies on are inherited from it, and all of them are imposed in phonemic space. This is what separates the design from a Wuggy-style pipeline, which controls sub-syllabic substitutions in the orthographic domain and phonemises afterwards, leaving the phonemic properties of the pair uncontrolled.
Selection and pairing. Real words are drawn from the Filter-A-validated pool (Appendix § 12) by stratified sampling over ten score strata, restricted to items of four to seven IPA phones. Candidate pseudowords are generated at five times the required pool size for the standard categories and screened by Filter B, with rejections backfilled so category sizes are held fixed. Each real word is then matched to the pseudoword nearest it in phonotactic score drawn only from candidates with the same number of IPA phones, subject to . Every one of the 4,129 pairs is exactly length-matched in phones as a result; the mean absolute score difference is 0.19 for LexicalMemory and 0.16 for Composition. The set comprises 967 LexicalMemory, 1,162 Composition and 2,000 Inverted pairs.
Phonotactic control in the BabySLM benchmark: The same phonemic control is absent from the benchmark our task replaces. Wuggy controls the biphone statistics of its pseudowords, but not triphone statistics, which are known to carry wordlikeness [1, 14, 23]. Table 11 scores all 17,706 real/pseudoword pairs of the BabySLM lexical development set under both models. Under the biphone model Wuggy does control, the real word is favoured on 58.1% of trials, close to the residual asymmetry expected from real words being better formed on average. Under the uncontrolled triphone model it is favoured on 70.3%. A model that had learned nothing beyond triphone phonotactics would therefore score close to 70% here, despite having no lexical knowledge at all.
| Biphone (controlled) | Triphone (uncontrolled) | |
| Real word favoured | 58.1% | 70.3% |
| Mean difference | +0.31 | +1.21 |
| Median difference | +0.21 | +0.89 |
| Skew | +0.56 | +0.75 |
| Mean − median | +0.10 | +0.32 |
| Cohen’s | 0.13 | 0.28 |
| KS | 0.070 | 0.128 |
| Wilcoxon |
Table 11. Phonotactic asymmetry between the real words and the Wuggy pseudowords of the BabySLM lexical development set ( pairs), under a biphone model, which Wuggy controls, and a triphone model, which it does not.
Lexical Decision: Filtering and Examples
The two filters. Filter A (real-word validity) admits a candidate real word only if it is attested by wordfreq and, after WordNet lemmatisation, present in the NLTK words list. wordfreq alone admits web artefacts such as https and repeated-character strings, while the NLTK list alone lacks inflected forms; gating one on the other keeps inflected and loan words while removing non-words. Filter B (pseudoword screen) rejects a generated pseudoword if its orthographic form occurs in the training vocabulary, or if its eSpeak IPA form, either verbatim or with stress and length marks stripped, occurs in the real-word IPA lexicon (the set of IPA forms of all corpus words). Rejected candidates are resampled so category counts are held fixed.
Contamination. Filter A reduced the real-word pool from the 82,548 alphabetic corpus types to 43,947 validated English words, rejecting 38,601 (47%) as non-words. Filter B then removed a substantial number of pseudowords whose phonemic form coincided with a real word (Table 12); in the Inverted category the 3,244 IPA collisions exceed the 2,000 pairs finally retained, confirming that without the phonemic screen a large share of “pseudowords” would in fact be real words in IPA space.
| Category | Filter B IPA collisions removed |
| LexicalMemory | 959 |
| Composition | 1,053 |
| Inverted | 3,244 |
Table 12. Pseudowords rejected by the phonemic screen (Filter B) because their IPA form matched a real corpus word.
Example pairs. Table 13 shows representative pairs from each category. In LexicalMemory and Composition the phonotactic scores are matched within the 2.0 threshold; in Inverted the pseudoword’s score is far higher than the real word’s by construction.
| Real | Pseudoword | ||
| LexicalMemory | |||
| pencil /pɛnsəl/ | blaver /blɛvə/ | −18.7 | −18.7 |
| heart /hɑːt/ | pred /prɛd/ | −11.9 | −12.0 |
| Composition | |||
| reissued /ˌriːɪˈsuːd/ | pophini /pəfɪni/ | −29.1 | −29.1 |
| foppish /ˈfɑːpɪʃ/ | cenfse /sɛnfɔ/ | −25.0 | −25.0 |
| Inverted | |||
| gestalt /ɡəˈstɑːlt/ | plation /pleɪʃən/ | −39.9 | −13.2 |
| kinetic /kɪˈnɛrɪk/ | noment /ˈnəʊmənt/ | −37.5 | −13.3 |
Table 13. Example real-word / pseudoword pairs with Witten-Bell triphone phonotactic scores . Standard pairs are score-matched; Inverted pairs satisfy .
Target Vocabulary
The semantic tasks use the 68 CDI words [11] listed in Table 14, drawn from the earliest vocabulary of English-speaking children and high-frequency in the CHILDES development set of the BabyLM corpus. The six categories are concrete semantic fields whose categorical structure is well established developmentally, so the benchmark tests whether the distributional structure the model has seen is reflected in its embedding geometry rather than generalisation to novel words.
| Category | Words |
| Animal | dog, cat, bird, fish, bear, duck, horse, cow, pig, frog, mouse, sheep |
| Food | milk, juice, water, cookie, apple, banana, bread, cheese, cake, egg, soup, rice |
| Body part | eye, nose, mouth, ear, hand, foot, head, hair, finger, knee, arm, leg |
| Clothing | shoe, sock, hat, shirt, coat, dress, boot, sweater, diaper, button, pocket |
| Vehicle | car, bus, truck, bike, boat, plane, wagon, stroller, tractor |
| Household | cup, spoon, bowl, plate, bottle, chair, table, bed, blanket, door, phone, clock |
Table 14. The 68 CDI target words by semantic category [11].
Tokenisation
Training. Every condition trains its own Byte-Pair Encoding tokeniser [28] from scratch on its own preprocessed corpus, using the HuggingFace tokenizers implementation. Vocabulary size is 4,000 at 10M and 8,000 at 100M, chosen to hold the ratio of vocabulary to model parameters roughly constant across the two scales; the minimum merge frequency is left at the library default of 2, and no merge-count or vocabulary sweep was run. The 10M orthographic tokeniser learns 3,708 merges and the 10M P-GORIELY tokeniser 3,938, the balance of each vocabulary being the base character inventory. Out-of-vocabulary material falls back to [UNK], and BPE dropout is not used.
Structural tokens ([B2], [B4], [H%], [L%], [UTT], [SPEAKER], [HEADER], and the reserved BERT tokens) are filtered out of the corpus before BPE training and reserved at fixed vocabulary ids by the trainer. Without this the pre-tokeniser splits [SPEAKER] into [, SPEAKER, ] and BPE learns merges inside the token name.
Segmentation. At application time BPE begins from the character sequence and repeatedly applies the highest-priority applicable merge from the learned merge list, in the order the merges were learned. It is not a longest-match scan: a longer entry in the vocabulary will not be preferred if the merges that would build it rank below the merges that build a different segmentation.
Pre-tokenisation, and the asymmetry between conditions. All conditions use the same whitespace pre-tokeniser, but it does very different work in each. Ortho retains ordinary inter-word whitespace, so the pre-tokeniser splits the stream at every word boundary and no merge can span one: word boundaries are preserved by construction, and the model receives them for free. The phonemic conditions have inter-word whitespace removed, which is the defining property of the Goriely et al. [13] setting we build on, so the pre-tokeniser splits only around structural tokens and effectively presents BPE with the whole utterance as a single unit. Merges may therefore span word boundaries, and no word boundary is marked in the input at any point.
This asymmetry is deliberate rather than incidental: it is what modelling continuous speech without orthographic segmentation means. It does, however, mean that the orthographic baseline is handed a source of information the phonemic conditions must infer.
Token Budget by Condition
Annotating a phoneme stream with structural tokens necessarily lengthens it, which raises the question of whether any advantage of the annotated conditions is a token-budget effect rather than a representational one. Each unified corpus was tokenised with its own trained tokeniser, with add_special_tokens=False so that no [CLS] or [SEP] padding is counted.
Composition of the structural share. At 10M, P-BREAKS spends 795,812 tokens on [B2] and 1,356,806 on [B4], and PROSODY-NN spends 883,401 on [B2], 1,166,168 on [B4], 17,723 on [H%] and 36,927 on [L%]. At 100M the corresponding counts are 7,852,083 and 13,443,590 for P-BREAKS, and 8,757,921, 11,465,427, 172,011 and 385,551 for PROSODY-NN. P-GORIELY carries only [UTT], 1,010,527 at 10M and 9,875,227 at 100M. Neither annotated condition emits it: in both, the utterance-final position is carried by [B4]. The two annotators differ in how often they place [B4] utterance-internally, the rule-based annotator doing so on roughly 192,000 lines at 10M against roughly 1,700 for the neural annotator, which is why P-BREAKS carries slightly more structural tokens than PROSODY-NN despite emitting no tones.
| Condition | Tokens | Per utt. | Structural | % struct. | vs P-GORIELY | vs ORTHO |
| 10M | ||||||
| ORTHO | 14,979,765 | 12.85 | 0 | 0.00% | +22.0% | n/a |
| P-GORIELY | 12,276,352 | 10.55 | 1,010,527 | 8.23% | n/a | −18.0% |
| P-BREAKS | 13,498,252 | 11.59 | 2,152,618 | 15.95% | +10.0% | −9.9% |
| PROSODY-NN | 13,417,556 | 11.52 | 2,104,219 | 15.68% | +9.3% | −10.4% |
| 100M | ||||||
| ORTHO | 140,210,835 | 12.24 | 0 | 0.00% | +27.8% | n/a |
| P-GORIELY | 109,677,972 | 9.58 | 9,875,227 | 9.00% | n/a | −21.8% |
| P-BREAKS | 122,184,461 | 10.67 | 21,295,673 | 17.43% | +11.4% | −12.9% |
| PROSODY-NN | 121,277,565 | 10.59 | 20,780,910 | 17.13% | +10.6% | −13.5% |
Table 15. BPE token counts per condition and scale, with the share contributed by structural tokens and the budget relative to P-GORIELY and to ORTHO.
Interpretation. Prosodic annotation does inflate the budget within the phonemic family, by 9.3% to 10.0% at 10M and 10.6% to 11.4% at 100M relative to P-GORIELY, so the confound is real for comparisons between the annotated conditions and P-GORIELY. It cannot, however, explain any phonemic advantage over orthography, because every phonemic condition still consumes markedly fewer tokens than ORTHO: the annotated conditions sit 9.9% to 10.4% below at 10M and 12.9% to 13.5% below at 100M, and P-GORIELY is 18.0% to 21.8% below. The orthographic baseline has the largest token budget of all eight models and is nonetheless overtaken at 100M on unsupervised category induction. The asymmetry runs against the annotated conditions, not in their favour.
UMAP Projections - Semantic Category Structure

Figure 3. UMAP projections of the 68 CDI word embeddings, coloured by semantic category, for ORTHO and PROSODY-NN at both scales; -means ARI is annotated in each panel.
References
- [1]Todd M Bailey and Ulrike Hahn. 2001. Determinants of wordlikeness: Phonotactics or lexical neighborhoods? Journal of Memory and Language, 44(4):568–591.DOI
- [2]Mary E. Beckman and Gayle Ayers Elam. 1997. Guidelines for ToBI labelling, version 3. Technical report, Department of Linguistics, The Ohio State University.
- [3]Mathieu Bernard and Hadrien Titeux. 2021. Phonemizer: Text to phones transcription for multiple languages in python. Journal of Open Source Software, 6(68):3958.DOI
- [4]Bastian Bunzeck, Daniel Duran, Leonie Schade, and Sina Zarrieß. 2025. Small language models also work with small vocabularies: Probing the linguistic abilities of grapheme- and phoneme-based baby llamas. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6039–6048, Abu Dhabi, UAE. Association for Computational Linguistics.arxiv.org/abs/2410.01487
- [5]Bastian Bunzeck and Sina Zarrieß. 2025. Subword models struggle with word learning, but surprisal hides it. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 286–300, Vienna, Austria. Association for Computational Linguistics.
- [6]Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Hu, Jaap Jummelet, Tal Linzen, Jing Liu, Aaron Mueller, Candace Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Wilcox, and Adina Williams. 2025. BabyLM turns 3: Call for papers for the 2025 BabyLM workshop. Preprint, arXiv:2502.10645.arxiv.org/abs/2502.10645
- [7]Anne Christophe, Sara Millotte, Séverine Bernal, and Jeffrey Lidz. 2008. Bootstrapping lexical and syntactic acquisition using prosodic and phonological constraints. Language and Speech, 51(1–2):61–75.
- [8]Robert Daland and Janet B. Pierrehumbert. 2011. Learning diphone-based segmentation. Cognitive Science, 35(1):119–155.DOI
- [9]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.arxiv.org/abs/1810.04805
- [10]Jonathan Duddington and Reece H. Dunn. 2025. espeak ng: Open source speech synthesizer. https://github.com/espeak-ng/espeak-ng.
- [11]Larry Fenson, Philip S. Dale, J. Steven Reznick, Donna Thal, Elizabeth Bates, John P. Hartung, Stephen Pethick, and Judy S. Reilly. 1993. MacArthur Communicative Development Inventories: User’s Guide and Technical Manual. Singular Publishing Group, San Diego, CA.
- [12]Lex Fridman and Yann LeCun. 2024. Yann LeCun: Meta AI, open source, limits of LLMs, AGI & the future of AI. Lex Fridman Podcast #416 transcript, accessed 2026-05-20.
- [13]Zébulon Goriely, Richard Diehl Martinez, Andrew Caines, Lisa Beinborn, and Paula Buttery. 2024. From babble to words: Pre-training language models on continuous streams of phonemes. Preprint, arXiv:2410.22906.arxiv.org/abs/2410.22906
- [14]Jennifer Hay, Janet B. Pierrehumbert, and Mary E. Beckman. 2004. Speech perception, well-formedness and the statistics of the lexicon. In John Local, Richard Ogden, and Rosalind Temple, editors, Papers in Laboratory Phonology VI. Cambridge University Press, Cambridge.
- [15]Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python.
- [16]Lawrence Hubert and Phipps Arabie. 1985. Comparing partitions. Journal of Classification, 2(1):193–218.DOI
- [17]Mutsumi Imai, Dedre Gentner, and Noburo Uchida. 1994. Children’s theories of word meaning: The role of shape similarity in early acquisition. Cognitive Development, 9(1):45–75.DOI
- [18]Peter W. Jusczyk. 1997. The Discovery of Spoken Language. MIT Press, Cambridge, MA.
- [19]Emmanuel Keuleers and Marc Brysbaert. 2010. Wuggy: A multilingual pseudoword generator. Behavior Research Methods, 42(3):627–633.DOI
- [20]Marvin Lavechin, Maureen de Seyssel, Jules Lévêque, Emmanuel Dupoux, and Alejandrina Cristia. 2022. BabySLM: Language-acquisition-friendly benchmark of self-supervised spoken language models. In Proceedings of Interspeech 2022, pages 1641–1645. ISCA.DOI
- [21]Emilie L. Lin and Gregory L. Murphy. 2001. Thematic relations in adults’ concepts. Journal of Experimental Psychology: General, 130(1):3–28.pubmed.ncbi.nlm.nih.gov/11293459
- [22]Brian MacWhinney. 2000. The childes project: Tools for analyzing talk (third edition): Volume I: Transcription format and programs, volume II: The database. Computational Linguistics, 26(4):657–657.
- [23]Jeremy M. Needle, Janet B. Pierrehumbert, and Jennifer B. Hay. 2022. Phonotactic Effects in the Acceptability of Pseudowords, chapter 4. Cambridge University Press.DOI
- [24]Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Riviére, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. 2020. The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling. arXiv preprint arXiv:2011.11588.arxiv.org/abs/2011.11588
- [25]Mari Ostendorf, Patti J. Price, and Stefanie Shattuck-Hufnagel. 1995. The Boston University radio news corpus. LDC Catalog No. LDC95S5.
- [26]Jenny R. Saffran, Richard N. Aslin, and Elissa L. Newport. 1996. Statistical learning by 8-month-old infants. Science, 274(5294):1926–1928.pubmed.ncbi.nlm.nih.gov/8943209
- [27]Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712.DOI
- [28]Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725. Association for Computational Linguistics.arxiv.org/abs/1508.07909
- [29]Kim Silverman, Mary Beckman, John Pitrelli, Mari Ostendorf, Colin Wightman, Patti Price, Janet Pierrehumbert, and Julia Hirschberg. 1992. ToBI: A standard for labeling English prosody. In Proceedings of the International Conference on Spoken Language Processing (ICSLP), volume 2, pages 867–870.DOI
- [30]Melanie Soderstrom, Amanda Seidl, Dana Gerber-Kemler Nelson, and Peter W. Jusczyk. 2003. The prosodic bootstrapping of phrases: Evidence from prelinguistic infants. Journal of Memory and Language, 49(2):249–267.DOI
- [31]Lyn Tieu and Jeffrey Lidz. 2016. Npi licensing and beyond: Children’s knowledge of the semantics of any. Language Acquisition, 23(4):311–332.DOI
- [32]Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355. Association for Computational Linguistics.

