betaFeedback

WorksComputer Science

Before Words, Beyond Speech: Evaluating Nonverbal Social Reasoning in Early Childhood

Abstract

Children's earliest years are a period of intense social learning, during which meaning is built through gaze, gesture, and physical contact. Yet, this critical aspect of social interaction remains under-evaluated in multimodal video understanding, as existing benchmarks prioritize the conversation-driven interactions of adults. We address this gap with \bench (N\textbf{N}aturalistic E\textbf{E}arly-childhood S\textbf{S}ocial inT\textbf{T}eractions), the first child-centric benchmark for nonverbal social reasoning. \bench comprises \totalvids manually annotated clips (∼\sim12s) curated for diversity across naturalistic settings, cultural contexts, and developmental stages. Beyond a lack of data, targeted model improvement in this domain is hindered by coarse evaluation schemas that conflate basic perception with higher-order reasoning. To resolve this, \bench introduces a compositional schema that builds atomic interaction cues into progressive levels of social reasoning: contextual, behavioral, interpersonal, and open-ended descriptions. Extensive analysis of state-of-the-art Vision-Language Models (VLMs) reveals a stark performance collapse: despite strong contextual recognition, models exhibit substantial gaps in decoding precise behavioral and interpersonal cues. We trace these failures to specific biases, including poor consequence grounding and temporal sensitivity. Finally, we ground \bench in real-world applications by evaluating VLMs using domain adaptation strategies. Ultimately, \bench provides a rigorous testbed for advancing socially grounded artificial intelligence in early childhood. Project page: https://laubravo.github.io/nest_website/.

Introduction

Long before children say their first words, they navigate the social world using a sophisticated set of non-verbal tools, including gaze, gesture, and spatial proximity. Coding these behaviors is a core practice in the early childhood field, including child development research [11, 44, 1], interventions to support early development [13, 53, 20], developmental assessments [7, 58], and classroom quality rating systems [40, 16]. Despite rapid advances in computer vision and the importance of early childhood for lifelong outcomes, this task remains mostly manual, forcing researchers to trade behavior density against the scale and diversity of populations studied [14, 18, 3]. Interpreting subtle non-verbal signals in caregiver-child interaction remains a blind spot in modern video understanding. To ensure the scalability promised by recent machine learning advances reaches early childhood, we need tools to track model performance and guide ongoing work.

Existing video social reasoning benchmarks target adult, speech-centric interactions (Table 1). Large-scale datasets such as Social-IQ 2.0, Social Genome, and R3R^3-VQA are drawn from online media, where social meaning is dominated by dialogue. This contrasts with child-caregiver dynamics, where meaning emerges from a child’s sub-second non-verbal cues integrated with a caregiver’s one-sided speech [8, 51, 60] — exchanges that demand compositional reasoning across interdependent cues. The few benchmarks that include children or non-verbal behavior (e.g., MMDB [43], ChildPlay [48], MimeQA [26]) target narrow assessment protocols or scripted performances rather than diverse, naturalistic child-caregiver interaction. As a result, when models fail, it remains unclear whether errors stem from imprecise temporal grounding, missed social context, or participant-specific challenges.

Dataset# Videos# QSourceChildrenNon-verbalUniform schemaQ/VidDemog.
Social-IQ 2.0 [59]1,0006,000YouTube✕✕✕6.0✕
MMDB† [43]160–Lab study✓✓✓–✓
Social Genome [34]2721,486YouTube✕✕✕5.5✕
R3R^3-VQA [36]3165,156YouTube✕✕✕16.3✕
SIV-Bench [22]2,7928,792YouTube, TikTok✓–✕✓–3.1✕
Motion2Mind [25]4,730–TV, movies✕✓✓––✕
MimeQA [26]101806YouTube✕✓✓8.0✕
NEST (ours)1,20814,024Home, lab, classroom, YouTube✓✓✓11.6✓

Table 1. Comparison of NEST with the most directly related social and child-interaction video QA benchmarks. NEST is the only benchmark that combines child-centric, non-verbal social interaction with naturalistic multi-source video, reported demographics, and a hierarchical reasoning schema applied uniformly at high density across all videos. ✕: absent, ✓: present, ✓–: partial. Children: extent of child–adult interaction content. Non-verbal: whether the benchmark targets non-verbal social cues without relying on dialogue. Uniform schema: whether the same fixed question schema is applied to every video (vs. questions generated per clip); ✓– denotes a recurring task/dimension set that is not applied as a single dense schema. Q/Vid: mean questions per video, indicating annotation density over the schema. Demog.: whether participant demographics are reported.

To address this gap, we introduce NEST (Naturalistic Early-childhood Social inTeractions), the first child-centric benchmark for compositional social reasoning in close, naturalistic video. The clips in NEST capture everyday moments like a child pointing across a dinner table at a cup her mother reaches for, or a caregiver crouching beside a flower to follow a toddler’s gaze. NEST comprises 1,208 interaction segments (∼12\sim12s) paired with 14,024 structured questions, curated to maximize diversity across participants, developmental stages, and cultural contexts. Each segment is manually annotated using a unified, temporally grounded schema spanning four complementary axes (Figure 1): contextual attributes (e.g., setting, activity, age), behavioral cues (atomic actions), interpersonal dynamics (action-reaction events), and an open description of interactions. This dense schema resolves the diagnostic ambiguity inherent in aggregated metrics, decomposes model performance into distinct capabilities, and allows stratified evaluation across metadata.

Overview of the NEST benchmark

Figure 1. Overview of the NEST benchmark. We introduce the first dataset (1,208 clips, 14k questions) for evaluating early-childhood non-verbal social reasoning. The example shows how an interaction description can be decomposed into NEST’s four-level schema (contextual, behavioral, interpersonal, open) to isolate the fine-grained details that define the exchange. By mapping this schema to task-specific video VQA questions, NEST overcomes the diagnostic ambiguity of standard aggregated metrics and disentangles a model’s underlying capabilities.

Extensive analysis of state-of-the-art video vision-language models (video-VLMs) and specialized models on NEST reveals a stark divergence: despite strong recognition of broad contextual cues, models exhibit substantial gaps in the higher-order reasoning required to decode precise behavioral and interpersonal cues. To diagnose why, we identify three biases: models ground consequences less reliably than causes, systematically under-predict interactions in child-child dyads, and miss brief temporal cues. Finally, we examine how VLMs can be deployed in practice by comparing them against specialist social detection models, exploring fine-tuning, and showing that NEST’s diagnostic findings persist in a real coaching intervention. Together with NEST, these analyses inform design guidelines for building socially grounded AI systems for early childhood interventions.

Our contributions can be summarized as follows: 1) The NEST benchmark: the first child-centric dataset for nonverbal social reasoning in close, naturalistic interactions. 2) Compositional schema and evaluation: a hierarchical schema over three reasoning levels — contextual, behavioral, interpersonal — plus an open description of interactions, that successfully disentangles basic perception from higher-order social reasoning, revealing a stark performance collapse in state-of-the-art VLMs. *3) Comprehensive evaluation of VLM failure modes: We characterize gaps and test their practical implications on early-childhood interventions. We release annotations and metadata for the ChildPlay and CHILDES subsets, together with evaluation code. Appendix A details the released artifacts, access requirements, and reproducibility limits.

Related Work

Social video question answering. Prior work [61, 59, 15] established video question answering (VQA) over YouTube as the standard setup for evaluating social reasoning, focused on adult intentions and causal inference. Recent benchmarks add structural complexity through grounded reasoning chains [34], mental-state inference [36], and relationship-aware evaluation [22]. These build on a broader line of causal VQA [28, 27, 38, 63], whose causal question taxonomies (e.g., causal-why, causal-how) we adapt to the social-interaction setting. However, as Table 1 summarizes, these benchmarks remain predominantly adult-centric and drawn from scripted or online media, where interactions are narratively framed and linguistically explicit. They also organize evaluation as sequential reasoning chains rather than across the multiple, complementary dimensions of a single interaction. NEST instead targets child-caregiver exchanges, where social meaning depends on the nonverbal coordination of these dimensions.

Non-verbal benchmarks and structured action representations. Two structural paradigms dominate non-verbal video understanding: task-decomposed evaluation schemas and graph-based action representations. MimeQA [26] poses hierarchical questions over mime performances where dialogue is absent by construction, while Motion2Mind [25] stages non-verbal Theory-of-Mind as detection, knowledge, and explanation over scripted-media clips. Action Genome [19] represents action structure as scene graphs of objects and their spatio-temporal relations, capturing the visual scaffolding of events but not the higher-order interaction dynamics they support. In contrast, NEST locates the diagnostic signal not in mental-state depth or scene structure but in composition: a child’s gaze, a point, and a partner’s answering look are each individually identifiable, but the social act—joint attention—exists only in their combination. Mirroring how developmental coding reads lower-order cues as assembling into social acts, NEST’s contextual, behavioral, and interpersonal layers (§ 3.1) localize where in this assembly models break.

Non-verbal perception models. A complementary line of work models social behavior using static or signal-level cues, treating nonverbal communication primarily as a perceptual-detection problem. Gaze localization benchmarks [42, 5, 48] frame social perception as identifying where a person is looking, while [32, 9, 10, 49, 54, 4] extend this to mutual gaze, shared attention, and atomic gaze events such as joint attention. Other specialist models address adjacent capabilities: estimating demographics such as age [23], while [57] integrates gaze with other proxemics and haptics like gesture, posture, expression, and touch. We benchmark these specialist models against generalist VLMs on NEST to test whether general-purpose reasoning can substitute for purpose-built detection.

Child-centric vision datasets. A growing body of work targets child-centered visual learning. The Multimodal Dyadic Behavior dataset (MMDB) [43] pioneered structured assessment of child–adult interaction with frame-level behavioral annotations, but operates at single-lab scale and predates current VLM evaluation. ChildPlay [48] provides naturalistic parent–child videos with 2D gaze annotations, but lacks shared-attention labels or links to higher-level social constructs. A separate line of work targets representation learning from infant-perspective video: BabyView [30] and SAYCam [47] provide large-scale egocentric streams, and BabyVLM [55, 56] extends this toward developmentally grounded pretraining and benchmarking of vision foundation models. Together, these resources target perceptual learning, isolated signals, or single protocol behavioral coding. NEST complements them by spanning multiple corpora, cultural contexts, and developmental stages.

The NEST dataset

To bridge the gap in social reasoning, we introduce NEST: the first compositional benchmark designed to capture the nuanced, non-verbal social dynamics of early childhood. This section outlines the dataset’s conceptual ideation (3.1), construction (3.2) and description (3.3).

Deconstructing social understanding

To understand where state-of-the-art VLMs struggle with social reasoning, we cannot rely solely on overall judgments of broad concepts such as engagement or dyadic synchrony. These measures are debated even among experts, and a single accuracy score does not reveal which specific signal the model missed. Instead, diagnosis must break down evaluation into tasks that can be assessed independently [43, 33, 36]. But which basic signals should guide this breakdown? We look to developmental psychology, where years of behavioral coding have identified a small set of social signals—gaze, posture, and proximity—that combine into more complex dyad-level concepts [12, 39, 7]. Rather than adopting a single schema, we design NEST around this principle with domain experts in early childhood interventions from the FIND program [13].

Concretely, NEST evaluates social reasoning at four levels of granularity (see Figure 2). Contextual questions probe basic scene properties (age, setting, main activity). Behavioral questions test temporal signals between specific individuals (mutual gaze, shared attention, actions in a time frame). Interpersonal questions address the causal and reactive structure of a social exchange (what prompted a given reaction, how someone responded to an action directed at them). An open task asks for a free-form description of the interaction, spanning all three levels.

NEST’s hierarchical task ontology and example of dense per-task VQA annotations

Figure 2. (Left) NEST’s hierarchical task ontology for non-verbal social cues. (Right) Example of dense per-task VQA annotations (avg. ∼12\sim12 per clip) used to decompose model performance across the NEST schema.

Building NEST

NEST consists of short clips of naturalistic caregiver-child interaction, each capturing the brief, nonverbal exchanges that developmental coding relies on. Our curation strategy is guided by two goals: maximizing the density of social behavior within each clip, and maximizing diversity across settings, cultures, and developmental stages. Rather than stage new recordings, we draw on existing video, which lets us capture spontaneous interaction at a scale and breadth that controlled collection cannot reach, while spanning the range of real-world contexts in which these behaviors naturally occur. Concretely, we build NEST in three main stages (Figure 3): (1) compiling a diverse video pool from multiple corpora and applying gaze-aware filtering to select clips with high social engagement, (2) iterative manual annotation of behavioral labels, and (3) conversion of these labels into multiple-choice questions. We describe each stage below.

Annotation procedure

Figure 3. Annotation procedure. From a large pool of video datasets featuring early childhood interactions, we select videos that support diversity across settings, countries, activities, ages, etc. We emphasize manual annotation (blue arrows) and quality control with semi-automatic steps (green) to arrive at NEST.

Compiling clips with high social engagement. We source videos from three corpora of naturalistic adult-child interaction across cultural contexts, developmental stages, and recording conditions. Two are publicly available: ChildPlay [48], unconstrained interactions sourced from YouTube; and CHILDES [31], a longitudinal corpus of recordings collected across many studies, languages, and cultures. The third corpus is private and drawn from a real-world intervention: caregiver–child and classroom videos collected through Filming Interactions to Nurture Development (FIND) [13], a positive-reinforcement coaching program that promotes supportive interactions between children and caregivers. Annotation and analysis were conducted under Stanford IRB-approved protocol 66624; consent, access conditions, and privacy safeguards are detailed in Appendix § 7.§ 1. From the long-form videos of the three corpora, we extract 12-second windows centered on frames with high social engagement. We use gaze as the primary signal: it is visually grounded, unrecoverable from dialogue alone, and drives parent–child coordination from infancy [11, 13]. Specifically, we restrict NEST to clips containing mutual gaze or shared focus on a common object, identified using estimated gaze points from [50] (dist≤10dist \le10 pixels). To evaluate models on clips with limited social engagement, we also include 20% of clips that do not meet these criteria. See Section B.2 for details.

Iterative manual annotation. Annotators labeled each clip using only the visual stream (no audio), ensuring the labels relied on observable cues. They first identified individuals and validated gaze events used for selection, then labeled contextual attributes, and finally described behavioral cues and interpersonal exchanges — selecting up to two action-reaction pairs between participants, with temporal grounding of atomic actions. Annotators concluded with an open-ended description of the interaction. To ensure quality, 20% of ChildPlay was annotated by all annotators and discussed before independent work proceeded; every annotation was later reviewed by a second annotator, with rejected labels reassigned. Full annotation protocols, exclusion criteria, and inter-rater agreement statistics are provided in Section B.3.

Generating multiple choice questions. To support quantitative evaluation, natural-language labels are reformatted into multiple-choice questions (MCQs). Informative MCQs require distractors that are plausible yet incorrect, as naïvely constructed distractors can artificially inflate performance by failing to meaningfully challenge models. To address this, we employ AutoConverter [62] to generate distractors that are both visually grounded and linguistically consistent. For each MCQ, we construct a compact context C\mathcal{C} consisting of (i) five frames sampled from the corresponding video clip, (ii) the question text, and (iii) relevant human annotations, including the ground-truth answer and additional contextual labels. This context was provided to Qwen3-VL-32B-Instruct [52], which generated candidate distractors conditioned jointly on visual evidence and linguistic context. By grounding distractor generation in both modalities, the resulting options remain semantically plausible while being clearly distinguishable from the correct answer.

Dataset description

NEST comprises 1,208 segments and 14,024 questions drawn from 20+ studies, 150+ participants, and 10 countries, with developmental stages ranging from infancy to school age (Figure 4).

Figure 4. NEST Statistics. A) Child age distribution (677/1,208 clips with reported age). B) Recording country distribution (801/1,208 with reported origin). C) Duration distributions for actions (a behavior directed at another participant) and reactions (the responding behavior), covering both adult→child and child→adult pairs.

Diversity of clips. We prioritize real-world data to reflect the heterogeneity of early-childhood interactions. Clips vary substantially across demographics, recording location, and acquisition conditions: recording settings are predominantly naturalistic homes (60%+), with the remainder drawn from labs and classrooms, and most interactions are dyadic, reflecting the structure of early child-caregiver exchange. (Figure 8) This diversity supports evaluation of model robustness beyond a single collection setting (Figure 4 A and B and Sec. B.1). The dataset also captures a wide spectrum of interaction dynamics (Figure 4 C), ranging from brief, subtle responses (such as rapid gaze shifts or small posture adjustments) to longer, sustained actions such as reaching toward or passing objects. These patterns occur in both child-led and adult-led contexts, reflecting the asymmetric and temporally varied nature of natural social exchanges.

Figure 8. Overview of dataset characteristics. A) Number of segments by data source, totaling 1,208 segments across three corpora. B) Distribution of the number of annotated people per video segment, with most segments containing two individuals (typically a child and caregiver). C) Setting distribution.

Dense compositional annotations. Each segment is paired with an average of 11.6 questions spanning all four annotation axes (contextual, behavioral, interpersonal, and open-ended) yielding dense, multi-dimensional coverage of every interaction. The resulting density of annotations enables per-segment diagnostic analysis: a model’s performance on contextual recognition can be compared directly against its performance on behavioral and interpersonal reasoning for the same clip, supporting the stratified evaluation used throughout our experiments. Table 8 reports the full distribution of question types per segment.

BatchContextualBehavioralInterpersonalOpen Inter.Total
agelocationmain act.shared attn.object attn.event attn.conseq.causereact.open inter.Total
ChildPlay6573163122751547363783783153163,837
CHILDES1,5397377046403521,5926496817297368,359
FIND388154134139563461501501551561,828
Total2,5841,2071,1501,0545622,6741,1771,2091,1991,20814,024

Table 8. Question counts per data batch and task category. Categories: Contextual (age, loc., main act.), Behavioral (shared attn., object attn., event attn.), Interpersonal (conseq., cause, react.), and Open Inter. (open-ended).

No-reaction segments. To test for false positives in social reasoning, where models predict a plausible reaction from prior expectations rather than from what the clip actually shows, we include no-reaction segments. These clips contain either (i) an action by one participant that elicits no observable response from the other, or (ii) no salient action and no reaction. For example, a caregiver may hold out a toy toward the child while the child continues looking elsewhere, showing no gaze shift, reach, or change in posture. No-reaction segments account for approximately 20% of the dataset (n=265n = 265) and serve as explicit negative controls, requiring models to distinguish genuine action–reaction dynamics from prior expectations about what should follow.

Experiments

Experimental setup

Model selection. We target our evaluation on video-VLMs capable of direct video processing with raw video as input. To ensure a fair comparison given variations in native context windows, we standardize the evaluation by setting the maximum sequence length to 32,768 tokens and a maximum image side of 512 pixels to preserve aspect ratio. Rather than imposing a restrictive fixed frame rate that would bottleneck performance to the least capable model, this approach allows higher-capacity models to use their full temporal resolution for capturing fine-grained details essential in this context. We evaluate both open-source (VLLM [24]) and closed-source models (through their APIs).

Metrics. We assess model performance with accuracy (%) for MCQs and balanced accuracy (%) on classification questions. To account for varying levels of detail in open-ended responses, we follow the LLM-as-Judge paradigm from Liu et al. [29] with a 0-3 similarity score, prioritizing semantic meaning over exact phrasing. However, VLMs often favor generic summaries (e.g., ’toys’ vs. ’blue car’), causing holistic scores to obscure missing details. To enforce granular rigor, we introduce a complementary claim-level recall metric (%). Inspired by existing literature [35, 37], the judge decomposes ground-truth narratives into atomic interaction claims and scores binary matches in the generated answer.

Main results

Tables 2 and 12 report the performance of state-of-the-art video VLMs across NEST’s reasoning levels on our public and FIND subsets, respectively.1 Across both subsets, models reliably identify contextual attributes but performance collapses as tasks shift toward fine-grained behavioral and interpersonal cues (Figure 5A), with the difficulty of interpersonal reasoning compounded by the precise temporal grounding it demands (e.g., shared attention, event at time tt). The same pattern persists for open-ended descriptions: similarity to ground truth remains moderate (0.7-1.7 on a 0-3 scale) and recall stays below 50%, indicating partially aligned outputs that fail to compose behavioral cues into coherent interaction-level descriptions.

Sankey plot and model performance by input type

Figure 5. A) Sankey plot of the best open-source model Qwen 3.5-27B, showing how correct (purple) and incorrect (gray) questions change across NEST’s levels. B) Model performance by input type relative to the visual input, the results highlight the non-verbal and visual nature of NEST.

Asymmetric reasoning across causal direction. Action–reaction pairs can be probed in two directions: predicting the consequence given the action, or inferring the cause given the reaction (see Figure 2). Across models (Table 2), consequence prediction is significantly harder than causal attribution (Wilcoxon signed-rank, p=1.5×10−5p = 1.5 \times10^{-5}; Figure 16). We hypothesize that this stems from information embedded in the reaction itself: a question like "why did the child reach towards the toy?" is strongly constrained by the preceding action (the caregiver offered it), whereas the converse (predicting how a child will respond to an offered toy) admits many equally plausible reactions.

ModelSizeageloc.main act.Avgobject attn.shared attn.event at tAvgreact.causeconseq.AvgAvgsim. (0-3)recall
Human–87.990.667.882.188.793.295.392.479.386.194.086.587.5––
Open-source
LLaVA-NeXT-Video7B44.564.746.351.851.537.727.538.944.141.038.741.344.00.87.3
LLaVA-OV7B69.482.666.172.768.450.238.452.375.350.235.353.659.61.014.1
openPangu-VL7B61.885.868.772.177.548.853.860.067.953.452.858.163.40.78.2
InternVL3.54B24.268.246.446.365.729.223.039.344.040.236.240.141.90.910.0
InternVL3.58B31.972.951.352.062.346.926.445.236.146.640.541.146.10.910.9
InternVL3.538B50.280.765.165.472.054.242.156.146.559.546.550.957.40.77.7
MiniCPM-V 4.58B80.091.576.482.680.356.451.262.661.561.749.957.767.61.011.8
Molmo28B77.788.977.181.281.758.960.066.960.761.859.360.669.61.014.3
Keye-VL 1.58B47.282.552.660.862.556.227.348.772.252.839.754.954.81.317.3
Qwen3-VL4B31.073.262.255.568.650.442.353.868.954.648.657.455.51.217.4
Qwen3-VL8B54.782.564.967.473.657.451.960.941.654.651.049.159.11.319.1
Qwen3-VL32B70.886.374.377.180.355.861.265.846.967.155.356.466.40.921.2
Qwen3.54B85.592.978.285.585.456.061.567.655.465.847.156.169.71.09.9
Qwen3.59B85.892.277.785.285.860.661.769.475.669.755.667.073.91.010.8
Qwen3.527B87.394.983.488.591.762.067.373.762.476.758.365.876.01.321.9
Closed-source (API)
Gemini 3.1 Flash Lite–86.586.176.983.284.662.061.569.442.572.354.856.569.71.117.1
GPT-5 mini–86.980.480.482.684.060.763.069.375.270.559.668.473.41.743.8

Table 2. Model performance on the public subset of NEST (ChildPlay and CHILDES). Open interactions are measured in similarity to ground truth (0-3) and recall (%), MCQs in accuracy (%) and balanced accuracy (%) when categorical. Best model and _second best_ model. Darker color indicates higher performance.

Comparing performance across models for cause and consequence questions

Figure 16. Comparing performance across models for cause and consequence questions.

Difficulty in temporal grounding. We examine how performance varies across event-at-time-tt questions as a function of the action’s duration (Figure 6A). Accuracy increases monotonically with action duration, rising from a median of ∼44% for actions under 1 second to ∼60% for actions of 5 seconds or longer. This pattern suggests that brief, fast-occurring actions are systematically harder to localize in time than longer, more sustained ones. We hypothesize that shorter action spans involve fewer sampled video frames, leaving models with less visual evidence to anchor the event temporally.

Figure 6. A) Per-model accuracy on event-at-time-tt questions binned by the question’s action duration (seconds). Gray dots are individual models. nn = unique questions per bin. B) Recall and precision on (Is Reaction?) questions by aggregated initiator →\to responder dyad, averaged across all models (±\pm SEM). nrn_r = number of reaction segments; nnrn_{nr} = number of no-reaction segments.

Diagnostic analyses

NEST is non-verbal and visual in nature. To ensure NEST is not solvable via language priors or transcribed speech, we evaluate three ablations all of which receive the text prompt: blind (text-only), audio (with Whisper large-v3 transcriptions [41]), and audio+visual. Figure 5 compares these against the standard visual input. This experiment reveals that NEST is fundamentally visual and non-verbal in nature: blind and audio variants degrade performance across all levels (avg. −32% and −28%). Unlike prior adult-centric benchmarks where speech dominates, combining audio and visual inputs in NEST yields only marginal benefits (avg. +1%). Figure 17b illustrates cases where audio can be helpful, particularly for activity recognition and consequence reasoning in multilingual settings.

Bias in child-centric scenes. Do models default to assuming an interaction is occurring just because multiple people are present? To test this underexplored condition, we evaluate negative instances defined as "no-reaction" segments, where there is low social engagement between the people (Sec. 3.3). Specifically, we ask: After Person A performs [action], does Person B respond? This formulation explicitly tests whether models hallucinate false positives in passive scenarios. VLMs exhibit two systematic biases (Figure 6B). Without visual input, models default to predicting no reaction: precision reaches 51-78% but recall collapses to 15-27%. With visual input, recall recovers asymmetrically. Same-dyad interactions (adult→adult, child→child) remain harder (39-40%) than cross-dyad interactions (54-57%), suggesting that VLMs lean on coarse demographic cues to parse interaction roles.

Performance across video quality and demographics. Additional analyses (Appendix § 10) show that accuracy is broadly stable across video-quality bins, and event-at-time-tt performance varies little with child age. Differences across recording continents are modest, while setting differences depend on the task: lab clips show higher object-attention and reaction accuracy than home and classroom clips. These setting comparisons are descriptive, since setting is largely nested within source corpus.

Towards VLM-assisted child interaction analysis

Current VLMs cannot automate child interaction analysis on their own, but they can play a useful role in larger systems that speed up workflows for early-childhood researchers and practitioners. To guide the design of such systems, we compare VLMs to specialist models, test their learning capacity through finetuning, and check if NEST performance transfers to real parent-coaching footage.

The value of specialist models. We evaluate the trade-offs between generalist VLMs and specialist models across three NEST tasks (Table 3). While VLMs demonstrate strong zero-shot capabilities in age estimation and flexibility across understudied tasks (e.g., object of attention), specialist gaze models achieve superior accuracy on shared attention with a fraction of the parameters. Consequently, given sufficient annotated data, specialists deliver targeted precision that makes them ideal modular tools or priors within larger systems.

Age EstimationShared AttentionObject of Attention
ModelParamsAcc. (%)ModelParamsAcc. (%)ModelParamsAcc. (%)
MiVOLO v2 [23]0.1B62.2Gazelle [45]0.3B64.7SemGaze [49]0.2B42.0
Qwen3.54B85.5SemGaze [49]0.2B<u>62.7</u>Qwen3.54B<u>85.4</u>
Qwen3.527B87.3Qwen3.54B56.0Qwen3.527B91.7
Gemini 3.1 Flash Lite-<u>86.5</u>Qwen3.527B62.0Gemini 3.1 Flash Lite-84.6
Gemini 3.1 Flash Lite-62.0

Table 3. Specialist models compared to top-performing VLMs on the public subsets of NEST (ChildPlay and CHILDES). Best and <u>second best</u>.

Supervised finetuning improves the language of non-verbal interaction. Open-ended evaluation reveals a sharp degradation: free-form descriptions collapse to generic phrases (“interacting”, “playing with toys”) with recall below 30% across the evaluated open-source models, even on clips where the model answers the corresponding closed-form questions correctly. This raises the question of whether VLMs lack exposure to the language of non-verbal interaction or whether the gap reflects deeper perceptual limits. We test this by finetuning Qwen3.5-{4B, 9B, 27B} with LoRA [17] on a separate pool of CHILDES and FIND clips outside the NEST benchmark (train: 485 CHILDES; val: 137 FIND). We evaluate on NEST-filtered (N=683N = 683 clips), removing any source videos that overlap with the finetuning to prevent data leakage.

Finetuning yields +0.15 in similarity and +12% in recall on the 9B model, bringing it to parity with the 27B base (sim: 1.21 vs. 1.26; rec: 0.23 vs. 0.24). Gains shrink at 4B (sim: +0.11, rec: +8%) and reverse at 27B (sim: −0.08, rec: −0.5%) (Table 4). A possible explanation is that the 27B base is already strong on this vocabulary, and LoRA overfits the small training set. Figure 7 shows the finetuned 9B, which replaces generics with concrete action sequences, corrects attention targets, and adds fine-grained physical details. This points to one tractable component of the gap: the language of non-verbal interaction is learnable from modest task-specific data. The deeper perceptual failures NEST surfaces are not addressed here and remain open research targets.

Figure 7. Qwen3.5-VL 9B predictions before vs. after finetuning for two held-out NEST items (one ChildPlay, one CHILDES). Each row shows the keyframe, human ground truth, and predictions from the base and finetuned models.

ModelSimbefore\mathrm{Sim}_{\mathit{before}}ΔSim\Delta\mathrm{Sim}Recbefore\mathrm{Rec}_{\mathit{before}}ΔRec\Delta\mathrm{Rec}
Qwen3.5-VL 4B1.00+0.110.11+0.08
Qwen3.5-VL 9B1.06+0.150.11+0.12
Qwen3.5-VL 27B1.26-0.080.24-0.01

Table 4. Finetuning impact on the NEST-no-overlap open-interaction held-out set (n = 683). Subscript before denotes the un-finetuned base model; Δ\Delta columns report the change after finetuning (finetuned − base). We report similarity (0-3) and recall (0-1). A change of +0.12 corresponds to a gain of 12 percentage points.

Case-study: Parent-coaching videos. A direct application of NEST is in interventional programs, where computer vision could reduce experts’ manual labeling efforts. We study FIND, a parent-coaching program in which coaches show caregivers or educators video examples of their developmentally supportive interactions. Each example pairs a clip with a description to help caregivers recognize and reinforce the behavior. Currently, selecting examples and annotating descriptions is done manually by a trained editor. These editor hours are the main bottleneck to scaling the program to more families and educators. This case study aims to assess the feasibility of using VLMs to help editors generate descriptions for selected clips.

However, as NEST shows, fully open description generation by VLMs lacks the precision needed for this type of application. Instead, we study VLM generation on two specific elements of the description: a child serve (a child-initiated action) and adult return (the caregiver’s response). This is the same action–reaction structure NEST’s interpersonal level is built around. Thus we measure zero-shot recall for serve and return in program-annotated clips (avg. ∼11.6\sim11.6s) using Qwen3.5-{9B, 27B}. The input to the model is the video clip, the definition of the element required, and a description of the complementary element for that video. Appendix E.2 details the preprocessing which uses the knowledge gained from NEST.

Results mirror NEST’s findings. Visual-only recall is low across both roles (9B: 12.7% / 10.5%; 27B: 15.6% / 12.2%). Audio benefits adult returns substantially more than child serves (27B: +13.5% vs. +2.9%). This extends our main finding that audio provides limited signal for the visually-grounded behaviors NEST targets. The failure modes NEST surfaces persist in real coaching workflows.

Together, these findings suggest that a socially grounded AI system for early childhood needs a coordinated system rather than one end-to-end model. This includes generalist VLMs for contextual reasoning, specialist detectors for perception and temporal grounding, domain finetuning where descriptive language falls short, and human review for the hardest interpersonal cases. NEST provides the instrument to make these composition choices empirically by measuring which components to trust, where, and for what. We release it to support the development of reliable tools for researchers, clinicians, and practitioners who depend on understanding children’s earliest social interactions.

Conclusion

While automated video analysis promises to transform developmental science, current VLMs cannot reliably interpret the non-verbal social cues defining child–caregiver interaction. This gap has broad consequences because VLMs deployed in homes, schools, pediatric care, and assistive systems will routinely encounter children and families but have been evaluated almost exclusively on adult, conversation-driven social content. NEST addresses this gap as the first naturalistic, child-centric video benchmark with a compositional schema that breaks down social reasoning into contextual, behavioral, and interpersonal cues. Our evaluation reveals a sharp drop from strong contextual recognition to weak higher-order reasoning. This is traceable to specific failure modes: poor temporal grounding, weaker consequence than cause prediction, and overlooked dyadic configurations that persist in real coaching workflows. By making these failure modes measurable, NEST turns child–caregiver interaction from a blind spot into a tractable target for future work. Understanding human social behavior includes understanding children and caregivers. In this work, we aim to provide a rigorous framework for advancing socially grounded modeling.

Acknowledgments and Disclosure of Funding

This work was supported by the Stanford Clinical Excellence Research Center (CERC), the Heising-Simons Foundation (Grant #2024-5555) and the Bill and Melinda Gates Foundation (Gates Award INV-035364). We thank our annotators and collaborators, whose contributions and discussions made this work possible. The authors declare no competing interests.

Privacy and release details

Privacy

NEST draws on existing recordings of children and caregivers, whose identities may remain recognizable in the source videos. Our secondary annotation and analysis were conducted under Stanford IRB-approved protocol 66624, which covers annotator recruitment, training, and supervision, as well as handling of the reviewed segments. Table 5 summarizes the provenance, collection oversight, and access arrangements for each corpus.

ChildPlayCHILDESFIND
Segments / questions316 / 3,837736 / 8,359156 / 1,828
Source videos / studies80 / 1333 / 1758 / 4
Original collectionPublic YouTube video, released as a research datasetResearch studies contributed to TalkBankParent-coaching intervention program
Consent & oversight at collectionChildPlay dataset termsDocumented by contributing investigators; gated by TalkBank registrationProgram’s IRB-approved protocol; caregiver consent at enrollment and for research purposes
Our secondary useStanford IRB 66624Stanford IRB 66624Stanford IRB 66624; accessed under the Intervention Program protocol
We redistributeAnnotations + metadataAnnotations + metadataNothing
Video accessYouTubeTalkBank registrationIntervention program

Table 5. Provenance and oversight.

ChildPlay comprises publicly available YouTube recordings released as a research dataset. CHILDES recordings originate from research studies whose consent procedures are documented by the contributing investigators, with video access through TalkBank registration. FIND recordings were collected under the intervention program’s IRB-approved protocol, with caregiver consent at enrollment and for research purposes, and remain subject to the program’s access controls.

The public data release is limited to annotations and segment metadata for ChildPlay and CHILDES; their videos are not redistributed. FIND videos, annotations, questions, and segment metadata are excluded from the public data release. Closed-source models served through third-party APIs were not evaluated on FIND. During benchmark construction, clips containing sensitive content, including nudity, were excluded.

Release Manifest

The public benchmark release comprises 12,196 questions and reference answers covering 1,052 segments: 316 from ChildPlay and 736 from CHILDES. It includes multiple-choice options and correct-answer labels, open-ended reference descriptions, the annotation schema and protocol spreadsheet, and segment metadata specifying the source video and corresponding 12-second window. We also release evaluation code, together with utilities for extracting benchmark segments from source videos obtained by the user. Table 6 lists the released artifacts and their access requirements.

CategoryArtifactReleasedAccess
DataQuestions + reference answers (12,196)Full, public segmentsOpen
Multiple-choice options (generated distractors)YesOpen
Annotation schema + protocol spreadsheetYesOpen
Segment metadata (source video ID, 12 s window)YesOpen
ChildPlay / CHILDES videosNoYouTube / TalkBank registration
Intervention video, 156 segments, 1,828 questionsNoIntervention program
CodeEvaluation codeYes—
Segment extraction (from downloaded videos)Yes—

Table 6. Release manifest. Annotations: CC BY-NC-SA 4.0.

Annotations are released under CC BY-NC-SA 4.0. Source-video access remains subject to the original providers’ terms and procedures: YouTube for ChildPlay and TalkBank registration for CHILDES. The annotation license does not alter these access arrangements. FIND materials remain accessible only through the intervention program’s own authorization procedures and are not distributed with NEST.

Table 7 summarizes reproducibility from the public release, which covers approximately 87% of NEST’s segments and questions. Table 2 can be reproduced entirely using the released ChildPlay and CHILDES annotations and code once the corresponding source videos have been obtained. Analyses combining these public subsets with FIND are partially reproducible: the same analyses can be rerun on the public subsets, but reproducing the reported full-dataset results requires authorized access to FIND. FIND-only benchmark results and the parent-coaching case study cannot be reproduced from the public release. The finetuning experiment is also partially reproducible: its CHILDES training data and 673 public held-out examples are released, while its 137 FIND validation examples used for checkpoint selection and 10 FIND held-out examples remain restricted.

ResultData requiredPublic-only?
Table 2 — model comparison, all reasoning levelsPublic subsetsYes
Fig. 5A — level-wise collapse (Sankey)Public subsets + FINDPartial
Table 13 and Figure 5B — blind ablationsPublic subsetsYes
Figure 5B and Figure 17b — audio ablationsPublic subsets + FINDPartial
Fig. 6A — temporal grounding by action durationPublic subsets + FINDPartial
Fig. 6B — no-reaction false positives, dyad biasPublic subsets + FINDPartial
Fig. 16 — cause–consequence asymmetryPublic subsets + FINDPartial
Table 3 — specialist vs. generalist modelsPublic subsetsYes
Table 4 — LoRA finetuningTrain: 485 CHILDES; validation: 137 FIND; evaluation: 683 examples (673 public + 10 FIND)Partial
Table 12 — FIND subset resultsFINDNo
App. E.2 — parent-coaching case studyFINDNo

Table 7. What is reproducible from the public release.

Benchmark details

This section details the composition of NEST. The benchmark draws from three sources: CHILDES, ChildPlay, and FIND. FIND is a collection of videos from caregiver-child and early childhood classroom interactions collected in natural learning environments as part of the video coaching intervention program Filming Interactions to Nurture Development (FIND) [13]. Table 8 reports question counts per source and task category; Figure 8 summarizes segment-level characteristics including group size, recording setting, and source distribution.

Statistics

To verify that the multiple-choice options are not positionally biased — a known failure mode of MCQ benchmarks — we report the distribution of the correct option letter across all multiple-choice subtypes (Figure 9). Within every subtype the letters are approximately uniform, with no letter exceeding the expected 1/k1/k frequency by more than a small amount. This holds for both 3-option subtypes (e.g., shared attention, reaction) and 4-option subtypes (e.g., age, event at time t).

Figure 9. Distribution of correct-answer letters per multiple-choice subtype, organized by question family. Open-ended subtypes are excluded.

To make the task taxonomy concrete, Figure 10 shows one representative question per subtype, drawn directly from the released NEST split. Each row displays the keyframe with bounding boxes on the referenced people, the multiple-choice options (or free-text reference answer for the open-ended subtype), and the correct response highlighted in the family color.

Figure 10. Example NEST questions per task subtype. Each row shows the keyframe with bounding boxes overlaid on the people referenced by color, the question presented to the model, and the answer choices with the correct option highlighted. Subtypes are color-banded by task family: Contextual (purple — age, location, main activity), Behavioral (blue — shared attention, object of attention, event at time t) Interpersonal (tan — consequence, cause, reaction), and Open Inter. (pink — open ended).

Selecting Videos

Across our three corpora (ChildPlay, CHILDES, and FIND), we select videos to maximize diversity of cultural contexts, recording settings, participant demographics, and interaction types. Selection was guided by metadata where available (e.g., CHILDES study tags, FIND program labels) and by manual inspection. The final benchmark spans 22 studies across 10 countries, with participants of diverse ages (Figure 11).

Gaze-based clip extraction. Figure 12 illustrates our extraction pipeline. From each long-form video, we extract short clips using gaze cues as a proxy for active social engagement. We first sample frames at 0.2 fps and run the gaze estimation model from [50] on each sampled frame, producing per-frame estimates of every detected individual’s gaze target. We then identify shared focus keyframes — frames where two or more individuals are estimated to be either looking at the same location (shared attention) or at one another (mutual gaze). Each such keyframe anchors a 12-second clip centered on the frame.

Clip extraction from long-form videos

Figure 12. Clip extraction from long-form videos. We use a gaze estimation model to sample candidate frames where the people are sharing each other’s focus. The clip length is set to 12s and frame rate between clips to 0.2 fps.

Negative examples. To support evaluation on cases of limited or absent social engagement, we additionally sampled 20% of clips from windows that did not meet the gaze criterion. These segments serve as negative controls, requiring models to distinguish genuine action–reaction dynamics from common co-occurrence priors.

Annotation Protocol

Annotators completed labels in a fixed order, each step building on the previous: (1) focus group identification and gaze validation, (2) contextual attributes, (3) action description with timing, and (4) interpersonal interaction dynamics. All annotations used the visual stream only — no audio — ensuring labels rely on observable cues. For each (keyframe, clip) pair, annotators were shown the selected keyframe (see Section B.2) alongside its corresponding video clip, displayed side-by-side as in Figure 13. To support consistent reference to individuals, color bounding boxes were overlaid on each detected person’s head; annotators referred to individuals by bounding-box color in their natural-language responses. For each question, annotators were informed whether to refer to the keyframe, the segment, or both.

Video clip sample

Figure 13. Video clip sample. Left: Bounding boxes were overlaid on a keyframe from the video clip so annotators could refer to individuals by color. Right: The full video clip played alongside the keyframe.

Corpora# Studies# Videos# Participants
FIND45843
CHILDES17333111
ChildPlay180Unknown

Table 11. Data sources and participants.

Annotator recruitment and training

Annotators were trained research assistants, onboarded with the labeling protocol and pilot examples before independent annotation. Annotation labels and protocols were co-designed with domain experts in early-childhood interaction from FIND.

Exclusion criteria

Clips were excluded if they were too blurry to assess gaze reliably, or if the head detection model failed to identify valid individuals in the focus group. We additionally excluded clips containing sensitive content such as nudity. Excluded clips were not replaced; the final benchmark reflects only clips passing these criteria.

Annotations

Annotations were collected along three dimensions matching the benchmark’s task families — contextual, behavioral, and interpersonal — using a structured schema co-designed with domain experts from FIND. We release the full schema as a spreadsheet alongside our code for reproducibility.

Contextual Annotations. Contextual attributes were labeled using both the individual keyframe and the full clips. Using the keyframe, annotators were asked to identify which bounding boxes corresponded to real people rather than erroneous head detections, defining the valid group. Then, for each individual in the valid group, annotators estimated their age range as one of: <1<1 (infant), 1–3 (toddler), 3–5 (preschool aged child), 6–12 (school aged), 13–17 (adolescent), or 18 (adult) years of age. When uncertain, multiple adjacent age ranges could be selected. These age bands reflect commonly used developmental stages from infancy through adolescence. Using the full clip, the annotators specified the location of the valid group (e.g. "inside a living room, sitting on a couch") and described the main activity that the group was engaged in (e.g. "eating at a table").

Behavioral Annotations. Given the role of gaze (§ 3.1), annotators identified a focus group — a subset of individuals exhibiting one of three gaze conditions: (a) each looking at the other, (b) each looking at the same object, or (c) one or more looking at another individual who might be looking offscreen or at another entity. For cases (b) and (c), annotators provided a label for the object of focus (e.g., "yellow toy truck"). Annotators then grounded behaviors performed by members of the valid group, including a facial expression and three actions, each with start and end times.

Interpersonal Annotations. To capture the causal and reactive structure of social exchanges, annotators built on the actions and facial expressions identified above. They estimated whether the two sequential actions formed a response pair — that is, whether one was performed in reaction to the other. To support multiple-choice evaluation, annotators also provided a plausible distractor reaction: an alternative reaction consistent with the clip but not what actually occurred.

Open Interaction To capture details not covered by the structured annotations, annotators provided a plain-text description of interactions between individuals in the valid group at any point in the clip. When no interaction was observed, they instead described the individual actions of valid-group members.

Quality control

A 20% subset of ChildPlay clips was annotated by 4 annotators and discussed before independent work began, ensuring shared interpretation of the protocol. Each completed annotation was then reviewed by a second annotator or research team member, and rejected annotations were reassigned or corrected.

Inter-rater agreement

On the double-annotated subset of ChildPlay clips, we computed Krippendorff’s κ\kappa for categorical labels and semantic similarity (cosine similarity over all-mpnet-base-v2 [46] embeddings) for free-text labels. Figure 14 reports per-label agreement. Structured labels such as age and shared attention reached substantial to high agreement (α\alpha between 0.66 and 0.86); free-text labels reached a mean cosine similarity of 0.66. The low-similarity cases were reviewed manually before proceeding. Some common sources of disagreement included borderline-ambiguous gaze direction in low-resolution frames, granularity mismatch, and stylistic differences. All disagreements were discussed before the second-pass review. Open Interactions, which involve free-form descriptions of social dynamics, show fair agreement (0.69), reflecting the inherent subjectivity of these annotations.

Figure 14. Inter-rater agreement for annotations in our dataset, measured on a subset of 64 video segments rated by 4 annotators. We report Krippendorff’s α\alpha for categorical variables (Age, Shared Attention) and cosine similarity for free-text descriptions (Location, Activity, Object of Focus, Description of Sharing Focus, Open Interactions).

Descriptions for identity grounding.

Establishing social interactions in video requires consistent person identification. However, generating continuous detection trajectories can be hindered by limitations in tracking itself (occlusions, video quality, detection performance). We study the utility of VLMs in maintaining identity across frames using age as a semantic proxy on the Childplay subset which has groundtruth tracklets. Figure 15 A) shows that age serves as an effective identity marker, with task difficulty scaling with the categorization granularity. Our results (Figure 15 B) and C)) demonstrate that per-person descriptions from a single frame can effectively replace explicit tracking. Ultimately, these single-frame VLM-derived descriptions perform comparably to ground-truth tracklets for person identification, offering an alternative for challenging videos where traditional tracking fails.

Identity grounding under age category groupings, prompting strategy, and input source

Figure 15. Identity grounding. We study the two open models Qwen3-VL-8B and Mini-CPM-v4.5 under three conditions: (A) age category groupings, (B) prompting strategy, and (C) input source.

Robustness of distractors

Human vs. generated. Consequence questions provide a within-question comparison of distractor sources: each contains one correct answer, one expert-written distractor, and two distractors generated by Qwen3-VL-32B. We analyze 15,012 responses from 17 models on 896 questions, including 7,645 incorrect responses. Conditioned on an incorrect response, uniform selection among the three distractors would assign 33.3% of errors to the human-written option and 66.7% to the two generated options together.

The human-written distractor accounts for 22.1% of errors, while the two generated distractors jointly account for 77.9%, or approximately 39.0% per generated option on average. Thus, even after accounting for the number of options from each source, generated distractors attract a larger share of errors in this subset. This supports their effectiveness as challenging alternatives in the evaluated consequence questions.

Generator-family comparison. We additionally examine whether models from the generator’s family tend to avoid generated distractors, which would increase the share of their errors assigned to the human-written option. The reported human-distractor error share is 21.0% for Qwen-family models and 22.9% for other models (Table 9). These descriptive results do not show the hypothesized avoidance pattern, although they do not rule out other forms of model-family bias.

Model group / referenceHuman-written share (%)
All models22.1
Qwen-family models21.0
Other models22.9
Uniform choice among distractors33.3

Table 9. Human-written distractor share among incorrect responses. Each consequence question contains one human-written and two generated distractors. Uniform selection among incorrect options gives a human-written share of 33.3%.

Scope and limitations. This analysis measures error attraction within questions containing both human-written and generated distractors; it does not compare separate all-human and all-generated question sets or establish semantic validity for every option. Rejection and revision counts of distractors were not logged during manual review, so these rates cannot be reported retrospectively.

Experimental details

Compute Resources

All experiments were run on a local Slurm cluster with NVIDIA Ampere GPUs. Distractor generation and description generation We use Qwen3-VL-32B-Instruct [2] on a single Ampere GPU with 141 GB HBM (constraint=141G). Distractor generation additionally uses Qwen3-8B [2] as a lightweight grammar/coherence filter. Description generation runs at max_model_len = 26,496. When used as the judge model for SIM/REC scoring during evaluation, Qwen3-VL-32B [2] is run at max_model_len = 32,768 with gpu_memory_utilization = 0.7.

Finetuning. We finetune three open-weight checkpoints — Qwen3.5-4B, Qwen3.5-9B, and Qwen3.5-27B — using Unsloth [6] with bf16 LoRA [17] (rank 16, α=16\alpha= 16). Each run uses a single GPU: the 4B and 9B models fit on a 48 GB Ampere card with 64 GB system RAM; the 27B model requires an 80 GB A100 with 128 GB system RAM.

Evaluation and judge. We evaluate models and finetuned checkpoints by serving each model under vLLM [24] on a single 141 GB Ampere GPU and pairing it with the Qwen3-VL-32B [2] judge described above for SIM and REC metrics.

Human evaluation

Human evaluation of multiple-choice questions

To contextualize VLM performance and verify that ground truths are recoverable from video alone, five expert raters familiar with the questions and tasks annotated a subset of the benchmark using a browser-based viewer that presented the same inputs given to the VLMs: silent video and the question text. Raters answered 664 unique (video, question) pairs spanning all nine question types: contextual and behavioral questions over 64 videos (16 per arm), interpersonal questions over 80 videos. Per-type accuracy ranged from 0.68 to 0.95, as shown in Table 10. We computed inter-rater agreement only on the interpersonal questions, as they are the most subjective, and the other levels were already reviewed during benchmark construction; on the 83 pairs answered by three raters, Fleiss κ\kappa ranged from 0.73 to 0.79, as shown in Table 10.

TasknnAccuracyFleiss’ κ\kappanoverlapn_{\mathrm{overlap}}
Contextual
Age1410.879––
Location640.906––
Main activity590.678––
Behavioral
Shared attention590.932––
Object attention530.887––
Event at tt640.953––
Interpersonal
Consequence1080.9400.78941
Cause360.8610.73112
Reaction800.7930.73130
Overall6640.8750.76083

Table 10. Human evaluation results. For each (video, question) pair, we first average correctness across the raters who answered it, then average those per-question scores across questions within a task (macro-averaging at the question level), so each question contributes equally regardless of how many raters answered it. Inter-rater agreement (Fleiss’ κ\kappa) is reported for the interpersonal tasks. nn denotes the number of unique (video, question) pairs evaluated; noverlapn_{\mathrm{overlap}} is the size of the multi-rater subset used for κ\kappa. Reaction reports balanced accuracy under a binary framing.

Human validation of open-ended metrics

We validate the automated evaluation of open-ended interaction descriptions against expert human judgments. Two expert annotators assessed 138 atomic reference claims across 31 model responses from 62 videos used in our multiple-choice human evaluation. For each claim, annotators indicated whether the model’s response covered its content. We focus on Qwen3.5-27B, our strongest open model, and GPT-5 mini, our strongest closed model. Inter-rater reliability, measured over 35% of claims coded by both annotators, is substantial (83% agreement, Cohen’s κ=0.66\kappa= 0.66, 95% CI [0.46, 0.83]). All confidence intervals were estimated using 2,000 cluster bootstrap resamples of whole responses, preserving dependence among claims within each response.

Table 11 compares human and automated judge scores for recall and similarity. Human-assessed recall was higher than the judge’s for both models (Qwen3.5-27B: +7.2 points, 95% CI [−0.4, 14.6]; GPT-5 mini: +2.5, [−5.7, 11.0]; pooled: +4.8, [−0.8, 10.7]), but every interval includes zero, so we detect no significant bias. The consistent direction suggests the judge is slightly conservative. At the claim level for Qwen3.5-27B, errors were asymmetric: the false-positive rate was 3.8%, while the false-negative rate was 46.7%, so the judge more often missed coverage that annotators recognized than credited coverage they did not. Human-assessed recall nonetheless remained low for Qwen3.5-27B (24.7%; upper CI bound 33.5%) and moderate for GPT-5 mini (49.6%), supporting the conclusion that model descriptions omit substantial reference content. Similarity scores agreed closely, with differences of at most 0.10 on the 0-3 scale and all intervals spanning zero. We therefore find no detectable bias in the automated similarity metric and treat reported recall as a lower bound.

ModelnnJudgeHuman [95% CI]Δ\Delta [95% CI]
Recall (%), macro mean over responses
Qwen3.5-27B3117.424.7 [16.5, 33.5]+7.2 [−0.4, 14.6]
GPT-5 mini3147.249.6 [37.7, 61.8]+2.5 [−5.7, 11.0]
All6232.337.1 [29.2, 45.2]+4.8 [−0.8, 10.7]
Similarity (0–3), macro mean over responses
Qwen3.5-27B311.001.10 [0.89, 1.32]+0.10 [−0.21, 0.42]
GPT-5 mini311.681.58 [1.32, 1.84]−0.10 [−0.42, 0.24]
All621.341.34 [1.16, 1.52]+0.00 [−0.24, 0.23]

Table 11 (PDF p. 25). Human validation of the open-interaction metrics. We report human and automated judge scores for Qwen3.5-27B and GPT-5 mini (n=31n = 31 responses each), alongside combined results. Brackets indicate 95% confidence intervals; Δ\Delta denotes human − judge.

Cause and consequence asymmetry

Figure 16 shows the asymmetry in causality direction for the interpersonal questions discussed in § 4.2.

Performance on NEST for the FIND subset

Table 12 shows results across all open-source models on the FIND subset of NEST. Due to the private nature of the FIND dataset, closed-source models served by third-party companies were not evaluated on this subset. Evaluation on this held-out subset confirms the same patterns discussed for the public subsets of NEST in § 4.2.

ModelSizeContextual ageContextual loc.Contextual main act.Contextual AvgBehavioral object attn.Behavioral shared attn.Behavioral event at tBehavioral AvgInterpersonal react. (bal.)Interpersonal causeInterpersonal conseq.Interpersonal AvgAll AvgOpen Inter. sim. (0-3)Open Inter. recall
Open-source
LLaVA-NeXT-Video7B41.668.841.050.541.127.924.031.044.539.337.340.440.60.74.1
LLaVA-OV7B63.687.059.069.953.643.438.245.072.846.036.051.655.51.08.8
openPangu-VL7B55.989.669.471.673.247.746.055.669.956.759.362.063.11.110.0
InternVL3.54B18.965.636.640.348.231.817.332.432.138.726.732.535.10.76.4
InternVL3.58B29.676.044.850.150.034.124.636.236.346.041.341.242.50.97.7
InternVL3.538B46.484.459.763.558.948.837.348.348.756.746.750.754.20.97.6
MiniCPM-V 4.58B74.196.875.482.175.054.149.459.569.462.047.359.667.11.010.8
Molmo28B72.395.571.679.875.043.159.559.270.565.353.363.067.40.911.6
Keye-VL 1.58B45.481.250.759.160.742.326.343.162.852.744.753.451.91.314.8
Qwen3-VL4B32.479.258.256.657.145.238.246.862.256.755.358.153.81.214.3
Qwen3-VL8B49.884.462.765.664.350.247.153.946.165.357.356.358.61.315.4
Qwen3-VL32B63.789.076.176.373.247.358.459.652.469.358.760.165.31.423.8
Qwen3.54B85.294.879.986.675.047.561.661.452.771.346.056.768.20.97.2
Qwen3.59B86.494.871.684.376.850.355.560.970.068.050.762.969.30.98.0
Qwen3.527B89.797.482.189.787.556.662.769.069.974.756.767.175.31.214.1

Table 12. Model performance on the FIND subset of NEST. Open interactions are measured in similarity to ground truth (0-3) and recall (%), all other tasks are in accuracy (%). Best model and second best model. Darker color indicates higher performance.

Blind experiment

To verify that NEST requires visual understanding rather than language priors, we evaluate all models on question text alone, with no video or keyframe (Table 13 and Table 14). Performance drops substantially across all categories, providing evidence that NEST measures genuine visual social reasoning rather than language-only shortcuts.

ModelSizeContextual ageContextual loc.Contextual main act.Contextual AvgBehavioral object attn.Behavioral shared attn.Behavioral event at tBehavioral AvgInterpersonal react. (bal.)Interpersonal causeInterpersonal conseq.Interpersonal AvgAll AvgOpen Inter. sim. (0-3)Open Inter. recall
Open-source
LLaVA-NeXT-Video7B45.424.721.330.520.535.917.024.444.638.733.939.131.30.62.3
LLaVA-OV7B43.835.735.038.237.327.922.429.252.247.033.844.337.20.52.8
openPangu-VL7B45.558.438.447.539.632.623.932.017.345.843.135.438.30.81.2
InternVL3.54B43.150.537.343.635.128.916.126.752.940.139.644.238.20.90.5
InternVL3.58B43.651.835.043.523.338.314.125.235.047.943.342.136.90.90.6
InternVL3.538B42.647.535.241.836.151.222.136.735.553.241.243.340.60.90.6
MiniCPM-V 4.58B38.555.438.043.947.741.227.638.823.248.938.836.939.90.61.8
Molmo28B44.745.737.442.625.451.230.235.630.154.052.445.541.20.51.7
Keye-VL 1.58B45.755.733.845.133.355.317.635.464.046.734.348.342.90.61.1
Qwen3-VL4B45.049.536.643.730.633.024.129.249.049.246.448.240.40.50.3
Qwen3-VL8B44.548.432.341.739.124.325.729.721.549.949.740.437.30.60.9
Qwen3-VL32B46.860.237.948.341.429.429.133.324.256.345.041.941.10.71.4
Qwen3.54B44.852.930.442.730.029.318.325.824.444.228.532.433.60.70.4
Qwen3.59B45.559.937.847.740.428.522.130.338.552.437.842.940.30.00.0
Qwen3.527B47.371.046.254.848.326.131.635.423.260.542.342.044.10.71.5
Closed-source (API)
Gemini 3.1 Flash Lite-49.253.643.948.945.248.736.743.532.858.945.845.846.10.30.6
GPT-5 mini-47.456.140.948.144.421.232.932.824.958.248.343.841.60.85.8

Table 13. Model performance on the public subsets of NEST without visual input. Open interactions are measured in similarity to ground truth (0-3) and recall (%), all other tasks are in accuracy (%). Best model and second best model. Darker color indicates higher performance.

ModelSizeContextual ageContextual loc.Contextual main act.Contextual AvgBehavioral object attn.Behavioral shared attn.Behavioral event attBehavioral AvgInterpersonal react. (bal.)Interpersonal causeInterpersonal conseq.Interpersonal AvgAll AvgOpen Inter. sim. (0-3)Open Inter. recall
Open-source
LLaVA-NeXT-Video7B45.529.220.131.621.429.818.223.146.937.335.339.931.50.61.4
LLaVA-OV7B43.835.131.336.733.933.821.129.646.546.035.342.636.30.41.9
openPangu-VL7B44.735.132.837.544.637.723.735.416.345.344.035.236.00.90.1
InternVL3.54B41.335.732.136.430.431.813.925.445.845.333.341.534.40.90.0
InternVL3.58B42.848.732.141.230.435.811.625.932.943.345.340.535.90.90.1
InternVL3.538B41.240.337.339.642.941.019.934.638.050.744.044.239.50.90.4
MiniCPM-V4.5 8B35.345.535.138.644.638.027.736.825.450.037.337.637.70.52.3
Molmo28B42.344.232.839.826.839.633.233.229.060.054.747.940.30.60.5
Keye-VL 1.58B43.846.138.842.933.941.020.531.852.946.042.047.040.60.70.6
Qwen3-VL4B43.644.230.639.430.432.921.128.139.946.750.045.537.70.70.1
Qwen3-VL8B41.544.235.840.532.134.326.631.025.051.354.743.738.40.70.0
Qwen3-VL32B45.140.941.842.648.231.828.936.326.553.352.044.041.00.80.8
Qwen3.54B43.033.830.635.830.431.820.527.626.743.324.731.631.60.90.2
Qwen3.59B43.654.533.643.933.930.622.328.936.348.042.742.338.40.10.0
Qwen3.527B44.963.645.551.346.437.131.238.228.659.339.342.444.00.80.6

Table 14. Model performance on the FIND subset of NEST without visual input. Open interactions are measured in similarity to ground truth (0-3) and recall (%), all other tasks are in accuracy (%). Best model and second best model. Darker color indicates higher performance.

Audio Experiment

To test whether visual perception is essential for NEST or whether speech and ambient audio carry comparable signal, we evaluate models under three conditions: blind (text only), audio-only (transcribed via Whisper [41]), and audio+visual. Figure 17a reports accuracy gaps relative to the visual-only baseline. Audio alone consistently underperforms visual input across all task categories, with average drops of −36% on Contextual, −29% on Behavioral, and −20% on Interpersonal questions. Combining audio and visual yields only marginal gains over visual alone (within 2% across all subtypes), indicating that for most NEST questions the visual stream already contains the information needed to answer correctly. Qualitative examples (Figure 17b) illustrate the narrow conditions under which audio contributes meaningfully, either on its own or alongside visual input—most often when adults verbally narrate the ongoing activity. Together, these results confirm that NEST is fundamentally a visual social-reasoning benchmark.

(a) (a) Accuracy gap relative to the visual-only baseline for blind, audio-only, and audio+visual conditions, averaged across models (± Standard Error of the Mean (SEM)). Audio alone consistently underperforms visual input across all task categories, while combining both modalities yields marginal gains.

(b) (b) Qualitative examples showing cases where audio alone suffices (rows 1–2) and where combining audio and visual is necessary (row 3, consequence reasoning in Spanish).

Figure 17. Impact of audio on task performance. (a) Quantitative accuracy gaps across task subtypes. (b) Qualitative examples across modality conditions.

Reliability of LLM judge for open-ended descriptions

We assess sensitivity to judge choice by re-scoring all open-ended responses from four evaluated models using GPT-5, and comparing its scores with those of our primary judge, Qwen3-VL-32B. For each evaluated model, we compute Spearman correlation between the two judges’ per-response scores, mean absolute difference (MAD), and the difference in mean scores (Δ\Delta = Qwen judge − GPT-5 judge), separately for similarity and recall.

Table 15 shows strong overall agreement between the two judges. The relative model rankings proved stable (Spearman rank correlations = 0.60 for sim, 0.70 for recall), and the overall score deltas remained small (-0.3 for sim, 0.024 for recall), indicating that substituting the judge model preserves our findings. The results also suggest that same model bias is small, with similarity being more stringent and recall slightly higher.

ModelSize (B)SpearmanMADQwen3GPT-5Δ\DeltaSpearmanMADQwen3GPT-5Δ\Delta
SimilarityRecall
Keye-VL 1.580.60.41.31.4-0.10.6760.1040.1730.1510.022
Qwen3-VL320.30.70.91.5-0.60.7120.1130.2120.1730.039
Qwen3.5270.70.41.31.5-0.20.7270.1120.2190.2030.016
Gemini 3.1 Flash Lite-0.70.41.11.4-0.30.6830.10.1710.1510.02
Average0.60.51.21.5-0.30.7000.1070.1940.1700.024

Table 15. Open question model performance by judge. We measure inter-judge agreement for similarity and recall between Qwen3-VL-32B and GPT-5 using Spearman rank correlation, Mean Absolute Distance (MAD), and the overall delta.

Additional experiments

Performance stratified by initiator

We further analyze model performance on Cause and Consequence questions stratified by the action-reaction dyad — i.e., whether the action–reaction pair is performed by an adult–adult, adult–child, child–adult, or child–child pair (Figure 18). Across all four dyad groupings, models perform reliably better on Cause than on Consequence, with the gap reaching statistical significance in three of four conditions (adult–adult, adult–child, and child–adult; Wilcoxon signed-rank test, all p<0.001p < 0.001). The consistent Cause-over-Consequence gap suggests that models more reliably identify the antecedents of an observed reaction than predict the downstream effects of an observed action — a pattern consistent with the broader interpersonal reasoning weaknesses we report in the main results.

Figure 18. Comparing performance across models by action-reaction dyad groups for cause and consequence interpersonal questions.

Performance by video quality

Because NEST aggregates videos from three corpora collected under different recording conditions, the dataset spans a wide range of perceptual quality. We assess whether this variation drives model performance using MUSIQ [21], a no-reference image quality metric that accounts for blur, noise, compression artifacts, and resolution. Figure 19 shows accuracy stratified by MUSIQ quality bin (Poor < 30, Low 30–50, Medium ≥\ge 50). Accuracy remains relatively stable across bins for all six models, suggesting that performance limitations are not primarily driven by perceptual degradation but by challenges in structured social reasoning.

Figure 19. Accuracy stratified by MUSIQ [21] absolute. A higher MUSIQ score reflects a better perceived quality. It accounts for blur, noise, compression artifacts, resolution, etc.

Performance stratified by demographics

NEST spans a broad range of participant demographics, including a wide age range of children (from infancy to adolescence) and videos collected across multiple geographic regions. To verify that model performance is not driven by, or systematically biased against, any specific demographic group, we stratify accuracy by child age and continent of recording.

Performance by age

Figure 20 shows model accuracy on behavioral (event-at-time-t) questions stratified by the child’s age category. This indicates that model performance is broadly stable across the developmental range covered by NEST, and not concentrated in a particular age band.

Figure 20. Behavioral question (event at time t) accuracy by child age category. Visual condition (blue) substantially outperforms blind (purple) and random chance (25%) across all age groups, with no significant variation by age. Error bars show standard error of the mean (SEM) across models.

Performance by continent

Figure 21 shows model performance across task subtypes stratified by continent. Across all three categories, geographic disparities are modest. Asia tends to show slightly lower accuracy in Interpersonal tasks (56% vs. 58–62% average), while North America and Europe perform comparably.

Figure 21. Model accuracy across task subtypes and geographic regions. Bars show average accuracy across models (± standard error of the mean (SEM)) for North America, Europe, and Asia, grouped by task category (Contextual, Behavioral, Interpersonal). The dashed line indicates chance level.

Performance by setting

Performance is stable across homes, classrooms, and labs at the category level (largest gap: home 62% vs. classroom 55% on Interpersonal), but the ordering inverts by task type: lab segments are weakest on location (84% vs. 94% at home) and strongest on object attention (86% vs. 78–81% elsewhere) and reaction (71% vs. 62–66%), while main activity is unaffected (77% throughout), suggesting controlled framing trades scene context for a clearer view of the dyad. As the setting is largely nested within the data source, and the classroom (n=228n=228) and lab (n=263n=263) subsets are substantially smaller and drawn from fewer studies than home (n=718n=718), we report this as a descriptive stratification rather than a causal effect of setting (Figure 22).

Model accuracy across task subtypes and recording settings

Figure 22. Model accuracy across task subtypes and recording settings. Bars show average accuracy across models (±\pmSEM) for Home (n=718n=718), Classroom (n=228n=228), and Lab (n=263n=263 segments), grouped by task category; the dashed line indicates chance. Setting is largely nested within data source (Lab is 237/263 CHILDES 4) and the Classroom and Lab subsets are smaller, so differences are descriptive rather than causal.

Towards VLM-assisted interventions

Finetuning

Setup. We LoRA-finetune qwen3.5 at three scales (4B, 9B, 27B) on a separate pool of non-reviewed open-interaction questions, with 485 CHILDES samples for training and 137 FIND samples for validation. We evaluate on a held-out NEST subset (n=683n=683) with no source-video overlap with the training pool, ensuring reported gains reflect transfer to held-out scenes. Train and validation pools are split by source corpus to test cross-corpus transfer. Models are trained for 3 epochs and early-stopped at epoch 2 by validation loss. We use rank-16 LoRA (α=16\alpha=16, no dropout) on the language layers with the vision encoder frozen, 8-bit AdamW at a learning rate of 2×10−42 \times10^{-4} with linear decay, and an effective batch size of 4.

Per-source results. Table 16 shows both the overall and per-source performance of the open interaction task before and after finetuning. The 4B and 9B models gain on every source; the 27B model regresses on similarity for the ChildPlay/CHILDES subsets, suggesting the 27B base is already close to the LoRA’s reachable optimum on in-domain prompts and that more data would be needed to see consistent gains.

SizeSourcennSimbase_{\mathrm{base}}Simft_{\mathrm{ft}}Δ\DeltaSimRecbase_{\mathrm{base}}Recft_{\mathrm{ft}}Δ\DeltaRec
4BOverall6831.0001.113+0.1130.1110.193+0.082
Childplay3161.1491.253+0.1040.1450.236+0.090
Childes3570.8800.994+0.1150.0810.156+0.074
FIND100.6000.900+0.3000.0730.162+0.088
9BOverall6831.0601.211+0.1510.1130.230+0.118
Childplay3161.1491.241+0.0920.1300.264+0.134
Childes3570.9861.190+0.2040.0950.201+0.106
FIND100.9001.000+0.1000.1720.213+0.042
27BOverall6831.2621.186-0.0760.2360.231-0.005
Childplay3161.3421.244-0.0980.2950.265-0.030
childes3571.1961.134-0.0620.1840.199+0.016
FIND101.1001.200+0.1000.2280.260+0.032

Table 16. Finetuning impact on the NEST-no-overlap open-interaction held-out set, broken out by data source. base is the un-finetuned base model and ft is the finetuned model. Sim is the MMBench-Video judge score (0-3); Rec is claim-level recall (0-1).

Application: Parent-coaching videos

The parent-coaching videos for the case study are part of the FIND program, an intervention program that aims to support parents and educators by providing feedback on how they interact with the children in their care. FIND is based on the principle of positive feedback, where parents (and educators) are shown examples of their own developmentally supportive interactions. The examples are short (∼11.6s\sim11.6s) clips with accompanying descriptions selected from a long-form video recorded in the caregiver-child dyad’s home or other natural setting. This coaching program is designed to enhance learning and optimize engagement through the positive reinforcement. In this experiment, we build on the two main elements of the developmentally supportive interactions: the child-led serve and adult return. A child serves by doing, saying, or looking at something, while an adult returns by noticing the serve and responding. As this paired serve-return follows the same structure of NEST’s action-reaction, it is a natural application setting for the benchmark.

We preprocess the 123 edited videos by extracting the interaction description and video segments. We manually review that all the clips and descriptions align. Similarly to NEST (see B.4), we produce descriptions with Qwen3-VL-32B-Instruct for grounding the identity of each main child and main adult in the scenes and manually correct if needed. We make the distinction of main child or adult as the videos can feature siblings or other children, particularly in the classroom settings. To prompt the models for evaluation we adapt the structure of NEST’s interpersonal questions with i) the people’s descriptions, ii) definitions for key task specific terms (in this case child serve and adult return) and iii) a description of the complementary element. Figure 23 shows the template for the prompts.

Prompts for the application of NEST on parent-coaching videos

Figure 23. Prompts for the application of NEST on parent-coaching videos.

Impact

NEST targets a sensitive setting–video of young children and their caregivers in naturalistic interactions–so we discuss the expected positive and negative impacts of releasing the benchmark and the associated evaluation protocol, followed by the ethical considerations that shaped its design.

Positive impacts

Reducing the manual burden of behavioral video coding. Research and intervention in early childhood development rely heavily on manual coding of video, a process that is slow, expensive, and a common bottleneck for both basic research and applied programs. By providing a structured, multi-dimensional benchmark we make it easier for the community to identify which off-the-shelf vision systems are appropriate for which coding sub-tasks, and where they fail. This in turn supports tools that can lower the cost of observational research and broaden the contexts in which video-based intervention is feasible.

Improved evaluation of socially aware video models. Existing social-reasoning benchmarks are largely adult-centric and outcome-focused. By decomposing each interaction into contextual, behavioral, and interpersonal questions over the same segment, NEST surfaces at which level a model fails. This gives developers of vision-language models a more diagnostic signal than aggregate accuracy and encourages progress on perceptual grounding and interaction-level reasoning, not only narrative inference.

Standardization across heterogeneous corpora. Annotations span ChildPlay, CHILDES, and FIND under a single schema, enabling like-for-like evaluation across cultural contexts, recording conditions, and interaction styles. This is unusual in the child-centric vision literature and supports more transferable conclusions about model behavior.

Negative impacts and dual-use considerations

Privacy of children depicted in source videos. Because NEST indexes video of identifiable minors, any improvement we enable in social inference also lowers the cost of unwanted analysis of similar video. We address the most acute case — the FIND parent-coaching corpus — by excluding its segments from the public release entirely, and we redistribute only annotations and metadata for the remaining corpora, keeping source-video access subject to TalkBank registration and the YouTube Terms of Service. This limits, but does not eliminate, the risk that downstream users analyze child-centric video without comparable safeguards.

Surveillance and behavioral profiling. Better social-reasoning models are dual-use: they can support clinical observation as easily as they can support workplace, classroom, or home surveillance. We discourage such uses through the non-commercial license on the annotations (CC BY-NC-SA 4.0) and by framing the benchmark as a diagnostic evaluation tool rather than a deployment-ready system, but we cannot enforce downstream use.

Risk of misinterpreted model outputs in applied settings. Coding scales in early-childhood research feed into developmental assessments and intervention decisions. Models that achieve nontrivial accuracy on NEST may be tempting to plug into such pipelines, even though our results show substantial gaps in interaction-level reasoning. We therefore emphasize that NEST is an evaluation benchmark, not a clinical instrument, and that gains on NEST alone do not justify clinical or developmental use without dedicated validation.

Demographic and cultural coverage. While we explicitly diversify across studies, countries, and ages, the benchmark is bounded by the demographics of the contributing corpora. Performance and failure patterns observed on NEST should not be assumed to generalize uniformly across populations, and downstream users should treat group-level claims with caution.

Ethical considerations

Consent and data provenance. Each contributing corpus carries its own consent regime, which we respect rather than override. CHILDES videos [31] come from research studies whose consent procedures are documented by the contributing investigators and gated by TalkBank registration. ChildPlay videos [48] are publicly available on YouTube; we redistribute only annotations that reference them, not the videos themselves. FIND videos are governed by the parent-coaching program’s consent procedures and remain entirely behind the program’s access controls.

Institutional review. Annotation and analysis were conducted under Stanford’s IRB-approved protocol (66624). The protocol covers the recruitment, training, and supervision of human annotators, as well as the data-handling practices for the segments they reviewed.

Annotator welfare. Annotators were trained, compensated above the local minimum wage, and worked from a fixed protocol with explicit guidance on which frames to attend to. Inter-annotator review was built into the workflow, both as a quality-control measure and to avoid placing the burden of ambiguous cases on a single annotator. The instructions and annotation spreadsheet are included, respectively, in the annotation-protocol subsection of this supplementary (Section B.3) and code release.

Mitigations carried into the release. The release reflects the considerations above: FIND excluded; only annotations and metadata redistributed; CC BY-NC-SA 4.0 license; explicit pointers to the source corpora’s access procedures; and a dataset card documenting schema, splits, license, and known limitations (Section F.4). We expect downstream users to honor the same constraints when building on NEST.

Limitations

We discuss the main limitations of NEST along five axes, spanning data release, evaluation protocol, the role of LLMs in benchmark construction and scoring, and the heterogeneity of the underlying video material.

Restricted release of the FIND subset. Segments sourced from the FIND parent-coaching intervention corpus identify minors and are governed by the program’s consent procedures. We therefore exclude these clips entirely from the public release; the HuggingFace distribution drops 156 segments and the corresponding 1,828 NEST test questions. As a consequence, results reported on the FIND case study (§ 4) cannot be reproduced from the public release alone. Researchers wishing to reproduce or extend those results must obtain access through the original program.

Standardized evaluation context. To enable a fair comparison across models with very different native context windows, we cap the inference context at 32,768 tokens and 512×512512 \times512 maximum pixels per frame for all models. This standardization may not reflect each model’s optimal configuration: higher-capacity models with larger native context windows or higher input resolutions may achieve better performance under their default settings, and conversely some smaller models may have been tuned for shorter contexts. Reported numbers should therefore be interpreted as performance under a shared, controlled evaluation budget rather than as an absolute upper bound on each model’s capability.

LLM-in-the-loop benchmark construction and scoring. NEST uses Qwen3-VL-32B-Instruct to generate multiple-choice distractors through AutoConverter [62] and Qwen3-VL-32B [2] as the primary judge for open-ended responses. Because some evaluated models belong to the same family, these choices may introduce stylistic or family-specific biases. We manually reviewed a subset of the generated questions and examined distractor selection on 896 consequence questions containing one expert-written and two generated distractors (App. § 8.§ 5). Across 17 models, generated distractors attracted 77.9% of incorrect responses, averaging 39.0% per option compared with 22.1% for the expert-written option. Qwen-family models showed no descriptive tendency to avoid generated distractors. These results support the generated options as challenging alternatives within this subset, although residual stylistic or family-specific biases may still exist. The analysis covers only consequence questions, does not compare all-human and all-generated question sets, and lacks rejection and revision counts from manual review. We additionally assess scoring reliability through a second judge and expert validation, as discussed below.

Open-ended evaluation metrics. Our similarity and recall metrics measure semantic alignment with reference descriptions and coverage of reference claims, respectively. They capture complementary aspects of response quality, but they share known limitations of automated open-ended scoring: insensitivity to subtle factual errors, reward for surface-level fluency, and limited ability to penalize plausible but unsupported descriptions. We assess their reliability through cross-judge comparison and expert validation. Re-scoring responses from four models with GPT-5 reveals judge-dependent scores, including a 0.6-point similarity shift for one model on the 0–3 scale (App. § 9.7). This comparison does not establish invariant model rankings and excludes GPT-5 mini, the highest-scoring model on open interactions. Human validation covers 62 responses from Qwen3.5-27B and GPT-5 mini (App. § 9.2.2). For Qwen3.5-27B, claim-level false-positive and false-negative rates are 3.8% and 46.7%, respectively, suggesting that automated recall can miss coverage recognized by humans. These checks also do not establish the completeness or correctness of the automatic reference decomposition. We therefore report both metrics and recommend interpreting open-interaction results as comparative rather than absolute.

Sensitivity to source video quality. Source clips vary in resolution, lighting, compression artifacts, and camera stability, particularly across the three contributing corpora. We probe this directly by stratifying accuracy by MUSIQ [21] quality bins (Figure 19) and observe that performance is largely stable across bins, suggesting that current bottlenecks are reasoning-related rather than perceptual. We nonetheless flag this as a limitation: MUSIQ is a single perceptual-quality proxy, and other axes of acquisition variability (camera angle, motion blur during fast interactions, partial occlusion of the focus group) may affect specific question types in ways our aggregate analysis does not surface.

References

  1. [1]M. D. S. Ainsworth, M. C. Blehar, E. Waters, and S. N. Wall. Patterns of attachment: A psychological study of the strange situation. Psychology press, 2015.
  2. [2]S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
  3. [3]M. M. Black, S. P. Walker, L. C. Fernald, C. T. Andersen, A. M. DiGirolamo, C. Lu, D. C. McCoy, G. Fink, Y. R. Shawar, J. Shiffman, et al. Early childhood development coming of age: science through the life course. The lancet, 389(10064):77–90, 2017.DOI
  4. [4]X. Cao, H. Yang, V. Gunda, Z. Zhou, T. Xu, A. Kowdle, I. Kim, and J. M. Rehg. Gaze target estimation anywhere with concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 31304–31315, 2026.arxiv.org/abs/2608.11367
  5. [5]E. Chong, Y. Wang, N. Ruiz, and J. M. Rehg. Detecting attended visual targets in video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5396–5406, 2020.arxiv.org/abs/2003.02501
  6. [6]M. H. Daniel Han and U. team. Unsloth, 2023. URL https://github.com/unslothai/unsloth.
  7. [7]A. N. Esler, V. H. Bal, W. Guthrie, A. Wetherby, S. E. Weismer, and C. Lord. The autism diagnostic observation schedule, toddler module: Standardized severity scores. Journal of autism and developmental disorders, 45(9):2704–2720, 2015.DOI
  8. [8]K. Eulau and K. Hirsh-Pasek. From behavioral synchrony to language and beyond. Frontiers in Integrative Neuroscience, 18:1488977, 2024.DOI
  9. [9]L. Fan, Y. Chen, P. Wei, W. Wang, and S.-C. Zhu. Inferring shared attention in social scene videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6460–6468, 2018.DOI
  10. [10]L. Fan, W. Wang, S. Huang, X. Tang, and S.-C. Zhu. Understanding human gaze communication by spatio-temporal graph reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5724–5733, 2019.arxiv.org/abs/1909.02144
  11. [11]R. Feldman. Parent–infant synchrony and the construction of shared timing: physiological precursors, developmental outcomes, and risk conditions. Journal of Child Psychology and Psychiatry, 48(3-4):329–354, 2007.pubmed.ncbi.nlm.nih.gov/17355401
  12. [12]R. Feldman. Parenting behavior as the environment where children grow. 2012.DOI
  13. [13]P. A. Fisher, T. I. Frenkel, L. K. Noll, M. Berry, and M. Yockelson. Promoting healthy child development via a two-generation translational neuroscience framework: the filming interactions to nurture development video coaching program. Child Development Perspectives, 10(4):251–256, 2016.DOI
  14. [14]R. G. Fukkink. Video feedback in widescreen: A meta-analysis of family programs. Clinical psychology review, 28(6):904–916, 2008.DOI
  15. [15]X.-Y. Guo, Y.-F. Li, and R. Haf. Desiq: Towards an unbiased, challenging benchmark for social intelligence understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3169–3180, 2023.arxiv.org/abs/2310.18359
  16. [16]T. Harms, R. M. Clifford, and D. Cryer. Early childhood environment rating scale. ERIC, 1998.
  17. [17]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022.arxiv.org/abs/2106.09685
  18. [18]A. Imhof, S. Liu, L. Schlueter, T. Phu, S. Watanura, and P. Fisher. Improving children’s expressive language and auditory comprehension through responsive caregiving: evidence from a randomized controlled trial of a strength-based video-coaching intervention. Prevention Science, 24(1):84–93, 2023.DOI
  19. [19]J. Ji, R. Krishna, L. Fei-Fei, and J. C. Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.DOI
  20. [20]F. Juffer, M. J. Bakermans-Kranenburg, and M. H. Van Ijzendoorn. Promoting positive parenting: An attachment-based intervention. Taylor & Francis, 2023.
  21. [21]J. Ke, Q. Wang, Y. Wang, P. Milanfar, and F. Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5148–5157, 2021.arxiv.org/abs/2108.05997
  22. [22]F. Kong, W. Zu, X. Chen, Y. Yang, S.-C. Zhu, and X. Feng. Siv-bench: A video benchmark for social interaction understanding and reasoning. arXiv preprint arXiv:2506.05425, 2025.arxiv.org/abs/2506.05425
  23. [23]M. Kuprashevich and I. Tolstykh. Mivolo: Multi-input transformer for age and gender estimation. 2023.
  24. [24]W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.arxiv.org/abs/2309.06180
  25. [25]S. Lee, J. Jeong, D. Kim, Y. Son, and Y. Yu. Mind the motions: Benchmarking theory-of-mind in everyday body language. arXiv preprint arXiv:2511.15887, 2025.arxiv.org/abs/2511.15887
  26. [26]H. Li, M. Tjandrasuwita, Y. R. Fung, A. Solar-Lezama, and P. P. Liang. MimeQA: Towards socially-intelligent nonverbal foundation models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2026. URL https://openreview.net/forum?id=XYkjC08fG5.arxiv.org/abs/2502.16671
  27. [27]J. Li, L. Niu, and L. Zhang. From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21273–21282, 2022.arxiv.org/abs/2205.14895
  28. [28]J. Li, P. Wei, W. Han, and L. Fan. Intentqa: Context-aware video intent reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11963–11974, 2023.DOI
  29. [29]Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024.arxiv.org/abs/2307.06281
  30. [30]B. Long, R. Z. Sparks, V. Xiang, S. Stojanov, Z. Yin, G. E. Keene, A. W. Tan, S. Y. Feng, C. Zhuang, V. A. Marchman, et al. The babyview dataset: High-resolution egocentric videos of infants’ and young children’s everyday experiences. arXiv preprint arXiv:2406.10447, 2024.arxiv.org/abs/2406.10447
  31. [31]B. MacWhinney. The CHILDES project: The database, volume 2. Psychology Press, 2000.
  32. [32]M. J. Marin-Jimenez, V. Kalogeiton, P. Medina-Suarez, and A. Zisserman. Laeo-net: revisiting people looking at each other in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3477–3485, 2019.
  33. [33]L. Mathur, P. P. Liang, and L.-P. Morency. Advancing social intelligence in ai agents: Technical challenges and open questions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20541–20560, 2024.arxiv.org/abs/2404.11023
  34. [34]L. Mathur, M. Qian, P. P. Liang, and L.-P. Morency. Social genome: Grounded social reasoning abilities of multimodal models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 24879–24902, 2025.
  35. [35]S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, 2023.
  36. [36]L. Niu, J. Li, X. Yu, S. Wang, R. Feng, B. Wu, P. Wei, Y. Wang, and L. Fan. R^ 3-vqa: "read the room" by video social reasoning. arXiv preprint arXiv:2505.04147, 2025.arxiv.org/abs/2505.04147
  37. [37]S. Ostmeier, J. Xu, Z. Chen, M. Varma, L. Blankemeier, C. Bluethgen, A. E. Michalson, M. Moseley, C. Langlotz, A. S. Chaudhuri, and J.-B. Delbrouck. GREEN: Generative radiology report evaluation and error notation. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 374–390, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.21. URL https://aclanthology.org/2024.findings-emnlp.21/.
  38. [38]P. Parmar, E. Peh, R. Chen, T. E. Lam, Y. Chen, E. Tan, and B. Fernando. Causalchaos: dataset for comprehensive causal action question answering over longer causal chains grounded in dynamic visual scenes. Advances in Neural Information Processing Systems, 37:92769–92802, 2024.arxiv.org/abs/2404.01299
  39. [39]M. Perlman, O. Falenchuk, B. Fletcher, E. McMullen, J. Beyene, and P. S. Shah. A systematic review and meta-analysis of a measure of staff/child interaction quality (the classroom assessment scoring system) in early childhood education and care settings and child outcomes. PloS one, 11(12):e0167660, 2016.DOI
  40. [40]R. C. Pianta, K. M. La Paro, and B. K. Hamre. Classroom Assessment Scoring System™: Manual K-3. Paul H. Brookes Publishing Co., 2008.
  41. [41]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023.arxiv.org/abs/2212.04356
  42. [42]A. Recasens, C. Von drick, A. Khosla, and A. Torralba. Following gaze in video. In Proceedings of the IEEE International Conference on Computer Vision, pages 1435–1443, 2017.
  43. [43]J. Rehg, G. Abowd, A. Rozga, M. Romero, M. Clements, S. Sclaroff, I. Essa, O. Ousley, Y. Li, C. Kim, et al. Decoding children’s social behavior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3414–3421, 2013.DOI
  44. [44]L. A. Roggman, G. A. Cook, M. S. Innocenti, V. Jump Norman, and K. Christiansen. Parenting interactions with children: Checklist of observations linked to outcomes (piccolo) in diverse ethnic groups. Infant mental health journal, 34(4):290–306, 2013.DOI
  45. [45]F. Ryan, A. Bati, S. Lee, D. Bolya, J. Hoffman, and J. M. Rehg. Gaze-lle: Gaze target estimation via large-scale learned encoders. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28874–28884. IEEE, 2025.arxiv.org/abs/2412.09586
  46. [46]K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857–16867, 2020.arxiv.org/abs/2004.09297
  47. [47]J. Sullivan, M. Mei, A. Perfors, E. Wojcik, and M. C. Frank. Saycam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective. Open mind, 5:20–29, 2021.
  48. [48]S. Tafasca, A. Gupta, and J.-M. Odobez. Childplay: A new benchmark for understanding children’s gaze behaviour. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20935–20946, 2023.arxiv.org/abs/2307.01630
  49. [49]S. Tafasca, A. Gupta, V. Bros, and J.-M. Odobez. Toward semantic gaze target detection. Advances in Neural Information Processing Systems, 37:121422–121448, 2024.DOI
  50. [50]S. Tafasca, A. Gupta, and J.-M. Odobez. Sharingan: A transformer architecture for multi-person gaze following. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2008–2017, 2024.DOI
  51. [51]C. S. Tamis-LeMonda, Y. Kuchirko, and L. Song. Why is infant language learning facilitated by parental responsiveness? Current Directions in Psychological Science, 23(2):121–126, 2014.
  52. [52]Q. Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  53. [53]K. Vik and R. Rohde. Tiny moments of great importance: The marte meo method applied in the context of early mother–infant interaction and postnatal depression. utilizing daniel stern’s theory of ‘schemas of being with’ in understanding empirical findings and developing a stringent marte meo methodology. Clinical Child Psychology and Psychiatry, 19(1):77–89, 2014.
  54. [54]P. Vuillecard and J.-M. Odobez. Omnihead: A unified model for dynamic nonverbal facial behaviors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3553–3564, 2026.DOI
  55. [55]S. Wang, A. Chandra, A. Liu, V. Saligrama, and B. Gong. Babyvlm: Data-efficient pretraining of vlms inspired by infant learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1380–1390, 2025.arxiv.org/abs/2504.09426
  56. [56]S. Wang, W. Wang, Z. Wang, M. Whitton, M. Wakeham, A. Chandra, J. Huang, P. Zhu, H. Chen, D. Li, et al. Babyvlm-2: Toward developmentally grounded pretraining and benchmarking of vision foundation models. arXiv preprint arXiv:2512.10932, 2025.arxiv.org/abs/2512.10932
  57. [57]J. Wei, T. Zhou, Y. Yang, and W. Wang. Nonverbal interaction detection. In European Conference on Computer Vision, pages 277–295. Springer, 2024.arxiv.org/abs/2407.08133
  58. [58]A. M. Wetherby and B. M. Prizant. Communication and symbolic behavior scales: Developmental profile. Paul H Brookes Publishing Co., 2002.
  59. [59]A. Wilf, L. Mathur, S. Mathew, C. Ko, Y. Kebe, P. P. Liang, and L.-P. Morency. Social-iq 2.0 challenge: Benchmarking multimodal social understanding. https://github.com/abw ilf/Social-IQ-2.0-Challenge, 2023.
  60. [60]H. Yamamoto, A. Sato, and S. Itakura. Eye tracking in an everyday environment reveals the interpersonal distance that affords infant-parent gaze communication. Scientific reports, 9(1):10352, 2019.DOI
  61. [61]A. Zadeh, M. Chan, P. P. Liang, E. Tong, and L.-P. Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8807–8817, 2019.DOI
  62. [62]Y. Zhang, Y. Su, Y. Liu, X. Wang, J. Burgess, E. Sui, C. Wang, J. Aklilu, A. Lozano, A. Wei, et al. Automated generation of challenging multiple-choice questions for vision language model evaluation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29580–29590, 2025.arxiv.org/abs/2501.03225
  63. [63]T. Zhou, D. Chen, Q. Jiao, B. Ding, Y. Li, and Y. Shen. Humanbench: Probing human-centric video understanding in mllms with automatically synthesized benchmarks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4494–4504, 2026.

Paper details

vlnc:W_tx8naw35
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents