WorksComputer Science
GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport
Abstract
A child learns which words name which things from roughly 100 million words of input. Vision-language models trained at a comparable scale bind words to referents far less reliably, and we argue that part of the gap is structural: the usual next-token captioning objective has no variable standing for "this word refers to that region," so training applies no direct pressure to ground words. We add such a variable by treating each scene as a small assignment problem between the content words heard and the regions seen, solved as entropic optimal transport under two constraints. First, a word may be assigned to a special NOTHING column instead of any region, which matters because most content words in caregiver speech do not name anything visible. Second, no region may receive more than a bounded share of the assignment mass, a capacity constraint that encodes mutual exclusivity. The alignment step costs about eight Sinkhorn iterations, small next to the vision encoder. In controlled simulation, the naive version of the idea, a row-wise softmax that forces every word onto some region, collapses to 0.000 word-to-picture accuracy under realistic non-reference rates, below the 0.058 of captioning alone, while our two constraints reach 0.075 in that regime and 0.700 on clean input. A one-parameter version of the model reproduces adult cross-situational learning accuracy across three ambiguity conditions (RMSE 0.024). On the SAYCam infant video corpus, adding the alignment loss to a next-token captioning model raises word-to-picture accuracy from 32.4% to 48.7% and localization IoU from 0.18 to 0.38. A contrastive model trained on the same child's data remains stronger on its own benchmark, so our claim is that the loss improves a captioning objective under high non-reference rates, not that it sets a new best.
Introduction
Quine’s linguist hears a native speaker say “gavagai” as a rabbit runs past [9]: nothing in that single scene separates rabbit from white, running, or dinner. An infant faces this problem continuously from birth, yet has a working vocabulary of hundreds of words by the second birthday. Models trained on a comparable amount of paired image-utterance data do not solve it as cleanly. The strongest recent result in this regime, CVCL [14], uses a contrastive objective on
40th Conference on Neural Information Processing Systems (NeurIPS 2026). one child’s head-mounted video and reaches 61.6% classification accuracy on Labeled-S and 34.7% on an out-of-distribution set, well above chance but still short of a two-year-old’s word knowledge.
We treat this gap as a question of objective design rather than data scale. Two properties of real caregiver speech make grounding harder than the textbook Quine scenario, both visible in ordinary transcribed utterances (Table 1). Most content words heard in a scene do not name anything visible in it (hungry, put, filler like come on), and a few objects are almost always in frame and almost never named (hands, floor, the caregiver’s body), so they are permanently available to absorb any word a naive model cannot otherwise place.
| Utterance | Content words | What should happen |
| “look at the ball” | ball | ball → region containing the ball |
| “are you hungry” | hungry | hungry → NOTHING |
| “oh look, are you hungry, there is the ball” | hungry, ball | hungry → NOTHING; ball → ball region |
| “where did the ball go” | ball | ball → NOTHING (just left frame) |
| “put the cup on the table” | put, cup, table | cup, table → regions; put → NOTHING |
| “there we go, okay, come on” | come | come → NOTHING (filler) |
Table 1. Motivating examples: most heard content words do not name anything visible in the paired scene.
Developmental psychology names two mechanisms infants appear to use. Under cross-situational learning, no single scene fixes what “ball” means, but across many scenes the word co-occurs with balls and with little else [12]. Under mutual exclusivity, a toddler shown a familiar object and an unfamiliar one and asked to “find the dax” tends to map the new word to the new object [7]. Both are design targets for the objective in § 3.
Why the obvious fix fails. The natural way to add word-to-region grounding to a captioning model is a row-wise softmax over regions for each word, forcing every heard word onto its best-matching region. § 4 shows that this can be worse than doing nothing: when most heard words are non-referential, forcing them onto some region drives the whole vocabulary onto the few ever-present background regions (hands, floor). This failure is what the two constraints are meant to prevent.
Contributions. (i) We cast word-to-region grounding for infant-scale vision-language pretraining as a capacity-constrained optimal transport problem with an explicit non-reference (NOTHING) outcome and a capacity constraint that puts mutual exclusivity directly into the assignment. (ii) In controlled simulation the naive alignment baseline collapses to 0.000 accuracy under realistic non-reference rates, below captioning’s 0.058, while our two constraints are the only setting that beats captioning in that regime (0.075) and reaches 0.700 on clean input; the exclusivity strength has an interior optimum, consistent with the mechanism we attribute to it. (iii) A single free parameter of the model reproduces adult cross-situational learning accuracy across the three ambiguity conditions of Yu and Smith [2007] (RMSE 0.024), which we treat as a consistency check, not a development claim. (iv) On the SAYCam infant video corpus, adding the alignment loss to a next-token captioning model raises word-to-picture accuracy from 32.4% to 48.7% and localization IoU from 0.18 to 0.38.
Related Work
Infant-scale and data-efficient vision-language learning. The BabyLM program established that models trained on developmentally plausible amounts of text trail large-scale models on standard benchmarks [15], and multimodal extensions train vision-language models on caregiver-scale paired data such as SAYCam-derived corpora [13, 14]. Vong et al. [2024] show that a contrastive model (CVCL) trained on one child’s recordings learns many word-referent mappings, reaching 61.6% on Labeled-S and 34.7% on an out-of-distribution evaluation. We work in the same data regime but target the grounding sub-problem, and we add an alignment term to a next-token captioning objective rather than train contrastively.
Cross-situational and mutual-exclusivity word learning. Cross-situational statistical learning has broad behavioral support in infants and adults [12, 16], and computational accounts go back to associative and Bayesian formulations [4]. Mutual exclusivity is among the most replicated biases in early word learning [7]. Our angle is to show that it maps onto a capacity constraint on an assignment matrix, rather than entering as a separate heuristic layered on a learner.
Weakly-supervised and contrastive region-word grounding. Region-word alignment without box supervision runs from weakly-supervised grounding objectives to contrastive image-text models such as CLIP [10], which in effect take a row-wise softmax over regions or patches. Our naive-alignment condition (§ 4) belongs to this family, and its collapse under realistic non-reference rates points to a mismatch between that objective and the infant-scale regime, where non-referential speech is far more common than in curated image-caption data such as COCO [5].
Optimal transport for alignment. Entropic optimal transport, solved with Sinkhorn-Knopp scaling [3, 11], has been used for word alignment, set matching, and cross-modal retrieval [1]. Unbalanced and capacity-constrained formulations that allow partial or null assignment are the relevant tools for our NOTHING column and per-region capacities [8, 2]; we read the per-region capacity as an operationalization of mutual exclusivity in the high non-reference setting.
Method
Problem setup. For each scene we observe an utterance and a paired image. We filter the utterance to its content words ; function words and a stop-list are removed (§ 5 discusses the cost of an imperfect filter). We segment the image into candidate regions (a fixed grid or region proposals) and add one more column, NOTHING, giving columns.
Alignment as capacity-constrained optimal transport
We build a cost matrix from a learned word-region compatibility score, e.g. a dot product between a word embedding and a region embedding. Column 0 is NOTHING, with a word-conditioned cost (a small MLP head on the word embedding; § 3.3). We look for a transport plan that solves
where each word carries unit mass (the row equalities), is the entropy regularizer, and sets its strength. The visual columns carry capacities ; the NOTHING column is uncapped. The visual columns have upper-bound capacity constraints, while the NOTHING column remains uncapped. Thus, the formulation uses column inequalities rather than fixed column-marginal equalities.
Rule 1: a word may name nothing. Because NOTHING is uncapped, the optimizer never has to spend a word’s mass on a region it matches poorly. This prevents the collapse of § 1. Rule 2: no region may absorb everything. The capacity bounds the total mass any single region receives in a scene, e.g. for an exclusivity strength and average scene word count , or a plausibility-weighted schedule (§ 3.4). This keeps ever-present regions (hands, floor) from soaking up the vocabulary.
Feasibility and uniqueness. Sending every word to NOTHING satisfies all constraints for any , so the feasible set is non-empty for every . The entropic objective is strictly convex, so the minimizer is unique.
Solver. Let . We alternate the updates
and return . The capped column update is the exact maximizer of the dual in column ’s multiplier, which the inequality constraint bounds to be non-positive, so the iteration is block-coordinate ascent on a concave dual, a special case of the scaling algorithms of Chizat et al. [2018]. We run 8 iterations in the log domain with , small next to a vision-encoder forward pass.

Figure 1. The GAVAGAI alignment module. Region proposals and filtered content words form a cost matrix with an added NOTHING column. Capped Sinkhorn iterations give the plan , which updates the cross-scene surprisal prior and drives the auxiliary loss added to next-token captioning.
Relation to the baselines. No column sum can exceed , so once every and : an independent row-wise softmax over the regions plus NOTHING. This is the NOTHING-only condition in § 4. Removing the NOTHING column as well recovers the naive row-softmax baseline. All four conditions in § 4 share one code path with the capacity and the NOTHING column toggled.
Cross-scene memory
A single scene is underdetermined (§ 1), so we keep a running co-occurrence tally between words and region categories over the training stream, updated by the soft assignments from each scene. Because a few regions (hands, floor, caregiver) appear in nearly every frame, raw counts are dominated by these background categories whatever the word. We instead score pairings by surprisal, a pointwise-mutual-information correction measuring how much more often and co-occur than their marginals predict, and feed this back as a prior on the next scene’s cost matrix . The surprisal term lets rare but consistent pairings such as ball outweigh frequent but non-diagnostic regions. Concretely, the tally updates as with decay , and the prior added to is (clamped at 0 below), with the total tally mass, weighted by 0.3.
Training objective
Our model is a ViT-B/16 image encoder paired with a six-layer Transformer text decoder, trained with next-token cross-entropy. Unlike CVCL, which uses a contrastive objective, our model optimizes next-token cross-entropy with an auxiliary alignment loss on top of captioning:
where is the plan from Eq. (1). Gradients flow through only, with treated as constant (by the envelope theorem the implicit dependence of on contributes nothing at the optimum), and reach both encoders. The NOTHING cost is produced by a small two-layer MLP head on each word embedding; it is kept from collapsing to the trivial “every word → NOTHING” solution by the captioning loss, which still rewards referential words for predicting their visual context, together with a penalty on the mean NOTHING mass per scene that activates above a target fraction. Full settings are in § 3.4.
Implementation details
Every setting below backs the results that follow; ablation choices were selected on the validation split, not the test split. Backbone/init. ViT-B/16 image encoder initialized from self-supervised DINOv2 weights and fine-tuned; text side a six-layer, eight-head Transformer decoder (hidden size 512) trained from random init on the corpus vocabulary. Training recipe. AdamW (, , weight decay 0.01), peak LR with 2,000 warmup steps and cosine decay to 256, 40 epochs on the SAYCam train split; the alignment loss is enabled after a 3-epoch captioning-only warmup so is non-trivial before transport begins. Training used 4 NVIDIA A100 (40GB) GPUs, wall-clock hours. OT settings. ; 8 log-domain Sinkhorn iterations; plausibility-weighted capacity , where is a per-region plausibility weight (inverse of the region’s corpus-wide appearance frequency, normalized to mean 1) and the average scene content-word count; ; ; NOTHING cost a two-layer MLP head on the word embedding, with mean per-scene NOTHING mass penalized by a hinge above target fraction 0.6 (coefficient 0.01); gradient path through with fixed (envelope theorem). Cost matrix/regions. with unit-normalized embeddings and , plus the surprisal prior above; regions from DINOv2 ViT-B/14 self-attention seeds clustered into proposals per scene, with NOTHING appended as column 0. Content-word filter. A logistic classifier over 50-d token features (POS tag, corpus frequency, utterance position), trained on a 2,000-token hand-labeled sample of SAYCam transcripts to separate referential content words from function words and filler.
Experiments
We report two settings: § 4.1 uses controlled simulation with known word-to-object ground truth; § 4.2 evaluates transfer to the SAYCam corpus.
Controlled simulation
Setup. We generate scenes with a known word-to-object ground truth under three noise regimes, clean, moderate, and realistic, that vary the rate of non-referential words and the prevalence of ever-present background objects, calibrated to child-directed-speech statistics in CHILDES [6]. Each scene draws content words; referential words name one of 60 objects, non-referential words name nothing. The three regimes set the non-reference rate to 20%/50%/80% (clean/moderate/realistic), with ever-present background objects present in 10%/40%/70% of scenes. We compare four conditions: captioning only (no alignment term), naive alignment (row-wise softmax, every word forced onto a region), NOTHING column only, and the full method (NOTHING plus the exclusivity capacity). The metric is word-to-picture accuracy on held-out instances, the fraction of held-out word tokens whose highest-mass assignment is the correct object, scored top-1 over the full 60-object vocabulary rather than a four-alternative forced choice; this open-vocabulary scoring is why absolute values sit below the 0.25 four-way chance level used on SAYCam and should not be read across the two settings. Results are the mean over three seeds; per-cell std. dev. is within 0.006 (clean), 0.009 (moderate), 0.004 (realistic).
Table 2 shows the central result. Under the realistic regime, forcing every word onto some region (naive alignment) does worse than not aligning at all (0.000 vs. 0.058), because ≈80% of heard words are non-referential and collapse onto ever-present background regions. The NOTHING column recovers part of the gap (0.017), and the exclusivity capacity recovers the rest (0.075). The gains are small in absolute terms under this regime, as expected given how little referential signal survives, and consistent across all three seeds.
| Condition | Clean | Moderate | Realistic |
| Captioning only | 0.392 | 0.242 | 0.058 |
| + Naive alignment (row-softmax) | 0.650 | 0.358 | 0.000 |
| + NOTHING column only | 0.658 | 0.358 | 0.017 |
| + NOTHING + exclusivity (ours) | 0.700 | 0.400 | 0.075 |
Table 2. Word-to-picture accuracy under controlled simulation, by noise regime (mean of 3 seeds). Under the realistic regime the naive baseline collapses to chance; the full method is the only setting that beats captioning in every regime.
Exclusivity strength sweep. Sweeping under the realistic regime gives an interior optimum (Figure 2). Accuracy rises from 0.045 at the tightest setting () to 0.075 at the fitted optimum (). As grows the capacity constraint weakens, and in the limit the plan reduces to the NOTHING-only condition, so accuracy approaches that condition’s 0.017 rather than zero; the naive baseline’s 0.000 is reached only when the NOTHING column is also removed. The rate at which words collapse onto never-named background regions falls as moves toward the optimum.

Figure 2. Accuracy (blue, solid) and background collapse rate (red, dashed) vs. under the realistic regime. The interior optimum at (0.075, matching Table 2) supports reading exclusivity as a targeted capacity constraint. As the capacity goes inactive and the plan reduces to the NOTHING-only condition (0.017).
Comparison with human cross-situational learning. Yu and Smith [ [16] presented adult participants with a cross-situational word-learning task in which each trial paired several novel words with several novel objects and gave no within-trial cue to which word named which object. Within-trial ambiguity was varied between conditions by showing 2, 3, or 4 word-object pairs per trial (the 2×2, 3×3, 4×4 conditions), with a shared inventory of novel words and referents presented over a block of training trials, and test accuracy fell as ambiguity rose. These are task-difficulty conditions with adult participants, not developmental stages, and we do not describe them as a trajectory. We ask whether a capacity-limited learner reproduces this ordering: on each trial the simulated learner computes the plan of Eq. (1) between the presented words and objects, stores the highest-mass word-object pairs in the tally , and at test picks, for each word, the highest surprisal-corrected object among the alternatives. The single free parameter is fit by grid search over to minimize RMSE against the human accuracies. With the model gives 0.91/0.75/0.54 against human accuracies of 0.89/0.78/0.56 (RMSE 0.024; Figure 3). We read this as a consistency check rather than a strong test: one parameter is fit to three points, the human data are from adults, and the conditions vary task difficulty rather than age.

Figure 3. One-parameter fit to adult accuracy across the three ambiguity conditions of Yu and Smith [ [16] (RMSE 0.024). The conditions vary within-trial ambiguity, not participant age.
Real-corpus transfer
We evaluate on real head-mounted infant video from SAYCam [13]; dataset and region-proposal properties are in Table 3. The task is a four-way word-to-picture choice (25% chance) on our own held-out SAYCam-S test split (one target, three distractor frames); localization IoU is computed against bounding boxes we annotated on a 1,000-scene subset of that split (one box per target word, two annotators with disagreements adjudicated), since Labeled-S itself provides no boxes. The 74.2% non-reference rate is estimated by applying the content-word filter and checking, for each content word, whether its named object appears among the DINOv2 region proposals.
| Property | Value |
| Corpus | SAYCam-S (“Sam”) |
| Video hours | 64.5 |
| Paired scenes | 48,200 |
| Content-word types | 1,850 |
| Train/val/test (scenes) | 38k/5k/5.2k |
| Region proposals | DINOv2, /scene |
| Non-reference rate | 74.2% |
Table 3. SAYCam-derived dataset for transfer.
Table 4 gives the transfer results, matching simulation. The naive row-softmax term lowers the captioning baseline (32.4%→28.1%), the NOTHING column recovers it (41.5%), and the exclusivity capacity improves it further, to 48.7% accuracy and 0.38 IoU. The contrastive CVCL model, trained on the same child’s data, stays ahead on its own benchmark at 61.6%, so we claim not a new best but that the alignment loss improves a next-token captioning objective, and its localization, under a high non-reference rate. Figure 4 shows the three characteristic assignment behaviors, and Figure 5 a single-scene soft assignment in which ball maps to the ball region while the non-referential hungry routes to NOTHING.
Table 4. W2P accuracy and localization IoU on held-out SAYCam scenes. CVCL’s number is published Labeled-S accuracy.

Figure 4. Schematic illustration of three assignment behaviors: correct referral, non-reference routing to NOTHING, and an adjective-modifier failure. These are drawn schematics, not SAYCam frames, since the corpus is access-restricted under Databrary and frames cannot be reproduced here.

Figure 5. Single-scene soft assignment for “oh look, are you hungry, there’s the ball”. The target word ball maps to the ball region; the non-referential hungry routes to NOTHING.
Ablations
Ablations on SAYCam (Table 5) support the default configuration: plausibility-weighted capacity, surprisal-corrected cross-scene tallying, a learned content-word filter, and eight Sinkhorn iterations. Beyond eight iterations accuracy plateaus then dips slightly, so we keep eight. All variants were selected on the validation split.
| Variant | W2P acc. | IoU | |
| (a) Capacity schedule | Uniform | 0.431 | 0.32 |
| (a) Capacity schedule | Plausibility-weighted | 0.487 | 0.38 |
| (b) Cross-scene tally scoring | Raw co-occurrence | 0.398 | 0.28 |
| (b) Cross-scene tally scoring | Surprisal-corrected | 0.487 | 0.38 |
| (c) Content-word filter | Closed-class stop-list | 0.442 | 0.33 |
| (c) Content-word filter | Learned classifier | 0.487 | 0.38 |
| (d) Sinkhorn iterations | 2 | 0.420 | 0.31 |
| (d) Sinkhorn iterations | 8 (default) | 0.487 | 0.38 |
| (d) Sinkhorn iterations | 20 | 0.486 | 0.38 |
| (d) Sinkhorn iterations | 50 | 0.485 | 0.37 |
Table 5. Ablations on SAYCam. Bold marks the default configuration.
Conclusion
GAVAGAI adds a referential variable to an infant-scale captioning objective by reading each scene as a small assignment problem, solved by capacity-constrained optimal transport with an explicit NOTHING column and a per-region capacity that encodes mutual exclusivity. The two constraints keep the vocabulary from collapsing onto ever-present background regions under the high non-reference rate of real caregiver speech. A one-parameter version matches adult cross-situational learning accuracy across three ambiguity conditions (RMSE 0.024), a consistency check. On SAYCam, the alignment loss raises a captioning model’s word-to-picture accuracy from 32.4% to 48.7% and localization IoU from 0.18 to 0.38; a contrastive model on the same data is still stronger on its own benchmark, so the result is a gain to the captioning objective rather than a new best.
Several limitations point to the next experiments. Each word is transported independently, so a phrase like “big blue truck” splits its mass across regions instead of binding jointly; transporting noun-phrase chunks, or letting modifiers share a column with their head noun, is a natural fix. Verbs and action words carry no temporal semantics here and reach NOTHING mainly through co-occurrence, which clip-level or temporal regions could address. The content-word filter is imperfect, so some discourse filler leaks into the alignment and raises the NOTHING capacity needed. Scenes with many regions tend to over-disperse the transport mass, part of why needs tuning; Figure 2 shows accuracy stays near the optimum for in [0.5, 2.5]. Head-mounted video also brings motion blur and occlusion that degrade region proposals before transport begins. The most informative next test of generality is a second corpus such as BabyView, which we leave to future work.
Broader impact. Training on developmentally plausible input streams rather than web-scale data lowers the compute and energy cost of building grounded models, and gives cognitive science a runnable model of how structural biases shape word learning. Our evaluations use the SAYCam infant video corpus, which is access-restricted and shared through Databrary under an institutional agreement rather than public; we follow its terms for child privacy and data protection.
References
- [1]Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. In International Conference on Machine Learning (ICML), pages 1542–1553, 2020.
- [2]Lénaïc Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xavier Vialard. Scaling algorithms for unbalanced optimal transport problems. Mathematics of Computation, 87(314):2563–2609, 2018.DOI
- [3]Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems (NeurIPS), volume 26, 2013.
- [4]Michael C Frank, Noah D Goodman, and Joshua B Tenenbaum. Using speakers’ referential intentions to model early cross-situational word learning. Psychological Science, 20(5):578–585, 2009.DOI
- [5]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), pages 740–755, 2014.arxiv.org/abs/1405.0312
- [6]Brian MacWhinney. The CHILDES project: Tools for analyzing talk. Lawrence Erlbaum Associates, 2000.
- [7]Ellen M Markman and Gwyn F Wachtel. Children’s use of mutual exclusivity to constrain the meanings of words. Cognitive Psychology, 20(2):121–157, 1988.pubmed.ncbi.nlm.nih.gov/3365937
- [8]Gabriel Peyré and Marco Cuturi. Computational Optimal Transport, volume 11. 2019.arxiv.org/abs/1803.00567
- [9]Willard Van Orman Quine. Word and Object. MIT Press, 1960.
- [10]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748–8763, 2021.arxiv.org/abs/2103.00020
- [11]Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343–348, 1967.
- [12]Linda Smith and Chen Yu. Is human cognition adaptive? Cognition, 106(3):1558–1568, 2008.
- [13]Jessica Sullivan, Michelle Mei, Amy Perfors, Erica Wojcik, and Michael C Frank. SAYCam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective. Open Mind, 5:20–29, 2021.
- [14]Wai Keen Vong, Wentao Wang, Emin Orhan, and Brenden M Lake. Grounded language acquisition through the eyes and ears of a single child. Science, 383(6682):504–511, 2024.DOI
- [15]Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjape, Adina Williams, Tal Linzen, et al. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Challenge at CoNLL, 2023.
- [16]Chen Yu and Linda B Smith. Rapid word learning under uncertainty via cross-situational statistics. Psychological Science, 18(5):414–420, 2007.pubmed.ncbi.nlm.nih.gov/17576281