betaFeedback

WorksComputer Science

PRIMA: Pose Refined Inductive Motion Adapter for Zero Shot Social Bodily Behavior Recognition

Abstract

Recognizing fine-grained bodily behaviors, such as self-grooming, postural shifts, and backchanneling, is a useful intermediate step for computational social signal analysis. But the behaviors that matter vary across social contexts, and retraining a model for each new taxonomy is expensive. Vision-language models (VLMs) can adapt to new categories through text prompts, yet they lack sensitivity to body structure and often confuse subtle postural distinctions. We propose PRIMA (Pose Refined Inductive Motion Adapter), a lightweight module that injects a body-structure prior into frozen VLMs using human pose as a training-time supervisory signal, requiring no pose input at inference. PRIMA separates a model-specific projection head from a shared core that encodes transferable body-configuration knowledge; adapting to a new VLM requires retraining only the head. We train PRIMA on a micro-action dataset and evaluate entirely through zero-shot prompting. The adapter improves recognition on its training domain, transfers to additional VLM backbones at different scales with the core frozen, and generalizes to two unseen target datasets from social psychology and daily activity recognition. In a supporting downstream study, the improved behavior predictions also yield higher correlations with human ratings of rapport and dominance in a collaboration context. Within the scope of the tested models and indoor datasets, the results indicate that pose-guided training can produce body-structure priors that generalize across unseen behavior vocabularies and can be partially reused across VLM backbones while preserving RGB-only inference.

Figure 1. From social signals to transferable body structure priors. (a) Social attributes lack stable definitions. (b) Bodily behaviors are concrete, but (c) their label sets change with the setting. (d) PRIMA injects a pose-guided prior that transfers from RGB alone.

*Corresponding author.

CCS Concepts

• Computing methodologies → Activity recognition and understanding; Scene understanding.

Keywords

social bodily behavior, vision language models, pose estimation, zero-shot transfer, social signal processing, nonverbal communication

ACM Reference Format: Mingcong Ding, Babette Bühler, Tim Fütterer, Ulrich Trautwein, and Enkelejda Kasneci. 2026. PRIMA: Pose Refined Inductive Motion Adapter for Zero Shot Social Bodily Behavior Recognition. In Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3767308.3835063

Introduction

Understanding subtle social cues embedded in interactions from video matters for healthcare, education, and human robot interaction, but formalizing what to recognize is difficult [42, 64, 73, 74]. Most work targets high-level social attributes such as leadership or rapport, yet these lack agreed upon definitions: What counts as rapport varies across disciplines, making annotation unreliable and cross-setting generalization difficult [21, 53, 65, 71]. A different approach focuses on what people physically do: head nods, hand gestures, forward leans, postural shifts [2, 16, 38]. Psychology and nonverbal communication research has established that these fine-grained bodily behaviors function as social signals in their own right [40, 48, 58, 72]. Because they are grounded in observable movement, they are easier to define, easier to annotate, and less dependent on contested constructs [4, 56, 72]. Recent work in social signal processing has organized such behaviors into structured category systems that serve as an intermediate representation between raw video and social interpretation [4, 9, 50, 51].

This intermediate representation is useful, but the specific behaviors that matter are not universal. A conversation study may track back channeling and gaze aversion [1, 50]; a clinical assessment may focus on self-soothing and repetitive movements [23, 60]. Each context defines its own categories. Existing fine-grained action recognition and bodily behavior recognition methods train supervised classifiers on a fixed label set [4, 17, 24], so when categories change, the model must be retrained on new data. The cost scales with the number of scenarios, making the approach as expensive as directly predicting high-level attributes. This raises a concrete question: Can we learn general knowledge about how bodies move from one annotated source and transfer it to recognize behaviors under a completely different set of categories, without retraining?

Vision language models offer a path forward [34, 59, 82]. They classify visual inputs solely from natural language descriptions, so when behavior categories change, only the text descriptions need updating [59]. But their training data overwhelmingly describes objects and scenes at a coarse level, and the subtle postural and gestural differences that distinguish fine-grained behaviors are rarely captured. Fine-tuning on domain-specific action data risks impairing the general visual knowledge that makes zero-shot transfer possible [41, 63, 78].

These behaviors are defined by the spatial configuration of body parts, which VLMs do not reliably capture from RGB alone. Human pose encodes exactly this structure, is invariant to clothing and background, and can be extracted automatically from any frame. Existing pose-enhanced VLM methods, however, either require pose at inference or depend on target-task classification labels [10, 81].

We propose PRIMA (Pose-Refined Inductive Motion Adapter), which injects body structure knowledge into frozen VLMs by aligning their visual features toward pose-informed representations during training. No pose input is needed at inference, and no classification labels are used at any point. The architecture separates a shared core from a VLM-specific projection head; transferring to a new backbone requires retraining only the head while the core stays fixed.

We evaluate entirely through zero-shot VLM prompting. PRIMA improves recognition on its 52-class training domain [24, 44], transfers to a different VLM with the shared core frozen [3, 76], and improves performance on two unseen target datasets: BBSI [4] with 14 classes from group interactions and a 30-class subset of NTU RGB+D 120 [45, 66] drawn from daily and health-related activities. In a supporting analysis, the adapter’s predictions also yield higher correlations with BBSI’s human ratings of rapport and dominance.

Our goal is not state-of-the-art absolute scores, but a body-structure prior that transfers across changes in the behavior taxonomy or VLM backbone.

We make three contributions:

  • We introduce PRIMA, a lightweight adapter that uses human pose as a training-time teaching signal to inject a body-structure prior into frozen VLMs. No pose input is needed at inference, and no classification labels are used during adapter training.

  • We factor the adapter into a VLM-specific Projection Head and a shared Core, and show that the learned Core can be reused across additional VLM backbones at different scales while retraining only the Head.

  • We show that a prior learned on MMA 52 improves zero-shot recognition on two unseen target taxonomies, and we analyze where the transfer succeeds (spatial body-part relations) and where it remains limited (fine hand detail and temporal trajectory).

Related Work

From Social Attributes to Bodily Behaviors

Social signal processing has studied how nonverbal behavior encodes social information in group interactions [8, 9, 73, 74], including gaze-based joint-attention estimation from third-person social-scene video [68]. Early computational work predicted high-level social attributes from handcrafted features [21, 32, 33]: Sanchez-Cortes et al. [65] combined speaking activity and prosody for emergent leader identification, Beyan et al. [6, 7] applied pose-based features for leadership detection, and Müller et al. [52] used facial action units and hand motion for rapport detection. More recent work has estimated student engagement from attentional and affective facial-video features in authentic classroom settings [22, 69]. Müller and Bulling later attempted cross-dataset leadership transfer and observed substantial performance drops [53]. More recent multimodal work has assessed similarly context-specific constructs such as classroom discourse quality by combining transcript, audio, and video representations [28]. Each study defined its own construct, collected setting-specific annotations, and trained a model that does not transfer to other settings. More recently, annotation efforts have shifted toward mid-level bodily behaviors as a more reusable intermediate representation. Balazia et al. [4] introduced BBSI, annotating 14 body language classes from the Ethological Coding System for Interviews and evaluating supervised detectors. Liu et al. [46] annotated 32 micro gesture classes in iMiGUE, and Guo et al. [24], Li et al. [44] constructed MMA with 52 micro action classes. Each dataset targets subtle body motion, but their label sets come from different traditions and share only a few body configurations while most classes differ. The recognition models are closed-set supervised classifiers that cannot serve as a general foundation across interaction settings [49, 55, 60]. To our knowledge, the transfer of body structure priors across behavior vocabularies has not been systematically evaluated. Cross-modal distillation and pose-guided pretraining operate within fixed label sets and do not test whether learned priors remain useful when the target taxonomy changes substantially.

Pose in Action Recognition

Human pose has been widely used in action recognition as a complementary modality to RGB. Multi-stream architectures such as PoseConv3D and CTR-GCN fuse skeleton sequences with appearance features, requiring pose at both training and inference [12, 15, 67, 79]. Large-scale benchmarks such as NTU RGB+D 120 [45] provide diverse action categories spanning daily activities and medical conditions, but recognition models are typically evaluated on the same label set used for training. Pose-guided attention mechanisms such as POET and Pose-DRL similarly depend on pose at deployment [13, 14, 62]. Cross-modal distillation removes this dependency: A pose teacher trains an RGB student that operates independently at test time, as in PoseDistill [20, 27, 70] and VPN++ [13], which distills pose-driven attention into an RGB network for daily activity recognition. But these approaches transfer knowledge within a single dataset and label set, producing no representation intended to generalize across behavior vocabularies [26].

Adapting Vision Language Models

Vision-language models learn joint image-text representations that enable zero-shot classification via natural-language prompting [34, 43, 59]. This open-vocabulary capability is relevant to the cross-taxonomy problem because changing the behavior categories requires only updating the text descriptions, not retraining the model [77, 85]. Substantial work has explored adapting these models to specific visual tasks while preserving generality: CoOp and CoCoOp optimize learnable prompt tokens [39, 83, 84], LoRA injects trainable low-rank matrices [31], and adapter-based approaches such as CLIP-Adapter insert small bottleneck layers [11, 19, 30]. For video understanding, ST-Adapter and AIM add temporal modeling to image-level backbones [37, 54, 57, 80], and ViFi-CLIP [61, 75] explores fine-tuning strategies that balance temporal adaptation with retention of pretrained knowledge. These methods share a common pattern: adaptation targets a specific label set, and the resulting model is evaluated on the same or a closely related set [25, 35], not on unseen behavior vocabularies. PeVL [81] uses pose to enhance VLMs for fine-grained action recognition through cross-modality refinement and contrastive learning. It shares our intuition that VLMs lack sensitivity to body structure, but differs in its training objective and transfer scope: PeVL trains on target-task classification labels and requires pose at inference, tying the learned representation to a fixed taxonomy. PRIMA uses no classification labels, requires no pose at inference, and is evaluated on social behavior categories it has never seen.

Methodology

The goal of this work is to learn a transferable body structure prior which generalizes across behavior taxonomies without task-specific retraining. The key mechanism is feature alignment guided by pose. When a 2D skeleton is rendered on top of a video frame, a frozen VLM naturally produces features more sensitive to body structure; we treat these as teacher features. A lightweight adapter is trained to map the VLM’s features from plain RGB to the pose-informed teacher features. After training, the skeleton branch is discarded, and the adapter operates on RGB alone (Figure 2). Table 1 summarizes the notation used throughout this section.

Figure 2. PRIMA pipeline. Top: skeleton overlaid frames produce teacher features EtE_t (precomputed, cached). Middle: RGB frames pass through the trainable adapter to produce Es′E_s'; Lalign\mathcal{L}_{\mathrm{align}} closes the gap. Bottom: inference uses only the student path with the adapter frozen.

SymbolDescription
fv,fpf_v, f_pFrozen vision encoder and vision projector
A\mathcal{A}PRIMA adapter (=H+C=\mathcal{H} + \mathcal{C})
C\mathcal{C}Core (model agnostic, operates in Rd0\mathbb{R}^{d_0})
H\mathcal{H}Projection Head (model specific)
Win,WoutW_{\mathrm{in}}, W_{\mathrm{out}}Linear projections in H\mathcal{H}
ggLearnable gate scalar (init 0.01)
dvd_vVision token dimension (VLM dependent)
d0d_0Bottleneck dimension (768)
I,I+I, I^{+}RGB frame, skeleton overlaid frame
Vs,Vs′V_s, V'_sOriginal and adapted vision tokens
Et,Es′E_t, E'_sTeacher and student post projector features
δ\deltaAdapter residual before addition to VsV_s

Table 1. Notation summary.

Pose-guided Feature Alignment

A standard VLM processes a video frame in two stages: a vision encoder fvf_v converts the image into a sequence of visual tokens, and a vision projector fpf_p maps those tokens into the language model’s embedding space. We add a small trainable adapter A\mathcal{A} between the two, keeping the VLM entirely frozen so that the adapter must encode body-structure information as a small, transferable adjustment rather than a large, model-specific distortion.

For each frame II, we use MMPose [36] to estimate 2D body keypoints and render them as colored dots and bones on the original image, producing a skeleton overlaid frame I+I^{+}. Feeding I+I^{+} through the frozen VLM yields features that retain visual context but are also informed by body structure:

Et=fp(fv(I+))(1)E_t = f_p(f_v(I^{+})) \tag*{(1)}

Teacher features are computed once and cached to disk (about 150 GB per backbone). The teacher signal depends on pose estimation quality, but the dependence is mild: with synthetic keypoint noise at 20% of the person bounding box diagonal, the adapter still reaches 0.278 F1pool_{\mathrm{pool}} on MMA 52 against the 0.253 baseline, and performance is insensitive to the rendering style of the overlay.

On the student side, we feed the same frame without any skeleton overlay through the frozen vision encoder, pass the resulting tokens through the adapter, and then through the frozen projector:

Es′=fp(A(fv(I)))(2)E'_s = f_p(\mathcal{A}(f_v(I))) \tag*{(2)}

The adapter is trained to make Es′E'_s match EtE_t: it learns to reproduce the features the VLM would produce if a skeleton were visible, using only plain RGB. We align in post-projector space because these features directly drive the language model’s output.

At inference (Figure 2, bottom row), only the student path is used: each frame passes through frozen encoder, frozen adapter, and frozen projector. No pose input is needed.

Adapter Architecture

The adapter sits between the vision encoder output and the vision projector input (Figure 3). It takes in the original vision tokens Vs∈RN×dvV_s \in\mathbb{R}^{N \times d_v} produced by fvf_v and outputs adapted tokens Vs′V'_s of the same shape. The dimension dvd_v depends on the VLM (dv=1152d_v = 1152 for Qwen3 VL, dv=4096d_v = 4096 for InternVL3.5 after pixel shuffle). Gated residual. The adapter computes a residual δ\delta added to the original tokens:

Figure 3. Adapter internal structure. Blue: Head H\mathcal{H} (dv↔768d_v \leftrightarrow768, model specific). Purple: Core C (two blocks, model agnostic). Gate gg scales δ\delta before addition to VsV_s.

Vs′=Vs+g⋅δ(3)V'_s = V_s + g \cdot\delta \tag*{(3)}

where gg is a learnable scalar initialized to 0.01, keeping adapted features close to the VLM’s original representation at the start of training.

The residual δ\delta is produced by passing VsV_s through two components in sequence, a Projection Head H\mathcal{H} and a Core C\mathcal{C}:

δ=Wout(C(Win(LN⁡(Vs))))(4)\delta= W_{\mathrm{out}}(\mathcal{C}(W_{\mathrm{in}}(\operatorname{LN}(V_s)))) \tag*{(4)}

Projection Head H\mathcal{H}. Different VLMs produce vision tokens of different dimensions dvd_v; operating directly in dvd_v space would tie the adapter to one VLM. The Head maps tokens into a VLM-independent bottleneck: LayerNorm followed by WinW_{\mathrm{in}} (dv→d0=768d_v \to d_0 = 768), then after the Core, WoutW_{\mathrm{out}} (d0→dvd_0 \to d_v). Only the Head changes when switching VLMs. The bottleneck at d0=768d_0 = 768 sits below dvd_v, forcing the Core to learn structured representations.

Core C\mathcal{C}. The core operates entirely in Rd0\mathbb{R}^{d_0} and consists of two identical blocks, each applying an MLP and multi-head self-attention with residual connections:

Z=X+MLP⁡(LN⁡(X))(5)Z = X + \operatorname{MLP}(\operatorname{LN}(X)) \tag*{(5)}
X′=Z+MHSA⁡(LN⁡(Z))(6)X' = Z + \operatorname{MHSA}(\operatorname{LN}(Z)) \tag*{(6)}

The MLP expands to 4d04d_0 with GELU activation. The MHSA uses 4 heads, allowing the Core to model spatial dependencies across tokens, which matters because body configuration is relational: Whether a person leans forward depends on the torso’s position relative to the legs. Both operations use a dropout rate of 0.1 and the pre-norm convention. Because the Core operates in a fixed dimension independent of dvd_v, it is decoupled from any particular VLM (§ 3.4).

The adapter adds approximately 16M parameters on Qwen3 VL and 20.5M on InternVL3.5 (under 0.3% of either 8B VLM); the difference comes entirely from the Head.

Training Objective

The adapter is trained to minimize:

Ltotal=Lalign+λLreg(7)\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{align}} + \lambda\mathcal{L}_{\mathrm{reg}} \tag*{(7)}

The alignment loss combines MSE (feature magnitude) and cosine distance (direction):

Lalign=12MSE⁡(Es′,Et)+12(1−cos⁡(Es′,Et))(8)\mathcal{L}_{\mathrm{align}} = \frac{1}{2}\operatorname{MSE}(E'_s, E_t) + \frac{1}{2}\left(1 - \cos(E'_s, E_t)\right) \tag*{(8)}

The regularization loss prevents the adapter from modifying the original features too aggressively:

Lreg=mean⁡(δ2),λ=0.01(9)\mathcal{L}_{\mathrm{reg}} = \operatorname{mean}(\delta^2), \quad\lambda= 0.01 \tag*{(9)}

This penalizes large residuals, keeping the adapter’s adjustments small and the learned prior transferable.

Training details. We train for 20 epochs on MMA 52 (4533 clips) using AdamW [47] (lr = 10−410^{-4}, weight decay 0.01), cosine schedule with 10% warmup, gradient clipping at 1.0, and batch size 1 on a single A100 80GB GPU. Each clip is sampled at 2 fps (up to 768 frames) at a resolution of 448×448448 \times448.

Cross-Backbone Transfer

The reason we split the adapter into a Head and a Core is to enable transfer across VLM backbones (Figure 4). In Stage 1, we train the full adapter on Qwen3 VL (WinW_{\mathrm{in}}: 1152 →\to 768, WoutW_{\mathrm{out}}: 768 →\to 1152). In Stage 2, we freeze the Core and train only a new Head for InternVL3.5, whose token dimension differs (dv=4096d_v = 4096). The new Head (WinW_{\mathrm{in}}: 4096 →\to 768, WoutW_{\mathrm{out}}: 768 →\to 4096) maps InternVL tokens into the 768-dimensional space where the Core already operates, then maps the results back.

Cross-backbone transfer

Figure 4. Cross backbone transfer. Stage 1 trains the full adapter on Qwen3 VL. Stage 2 freezes the Core and trains only a new Head for InternVL3.5.

We validate the design on InternVL3.5 at both 8B and 4B scales, freezing the Core learned on Qwen3 VL and retraining only the Head while keeping the teacher embeddings fixed. Because the new Head must bridge both a dimensional mismatch and a distributional gap between the two backbones, successful transfer indicates that the Core captures body-structure information not tied to a single backbone’s feature space. A full adapter trained natively on InternVL3.5 performs slightly better (0.446 versus 0.427 F1pool_{\mathrm{pool}} on BBSI), with the frozen Core retaining 68 to 77% of the native gain across the three datasets. Transfer also holds in the reverse direction: freezing a Core trained natively on InternVL3.5 and training only a new Qwen3 VL Head reaches 0.479 F1pool_{\mathrm{pool}} on BBSI against the 0.443 baseline, a gain of 0.036 averaged over three seeds, again retaining about 77% of the corresponding native gain.

Experiments

Experimental Setup

All experiments use zero-shot VLM prompting with no target-task labels.

Datasets. We use three action recognition datasets drawn from different research areas, whose label sets share only a few body configurations while most classes differ. MMA 52 [24, 44] contains 52 fine-grained micro action classes covering everyday body movements such as scratching, waving, and nodding. The training split has 4533 video clips, and the test split has 737 samples. Each clip may contain multiple simultaneous actions, making this a multi-label recognition problem. BBSI [4] annotates 14 body language categories from the Ethological Coding System for Interviews on group interaction recordings from the MultiMediate corpus [50, 51], including behaviors such as self-grooming, leg crossing, and postural shifts. The test set contains 947 samples. BBSI is used only for evaluation; neither the adapter nor any VLM sees BBSI data during training. NTU RGB+D 120 [45] contains 120 action classes covering daily activities, medical conditions, and mutual interactions. We evaluate on a 30-class single-person subset (20 daily activity classes + 10 medical condition classes), excluding mutual interactions that require two visible people. The test set contains 1000 samples under the cross-subject protocol. NTU RGB+D 120 is used only for evaluation.

Methodmicro MAPmacro MAP
Swin + PDAN (supervised) [4]0.7450.374
Qwen3 VL baseline (zero-shot)0.4030.151
PRIMA (zero-shot)0.4480.185

Table 2. BBSI evaluation under the original frame-level MAP protocol [4]. Supervised results are from the original benchmark using Swin features with PDAN detection trained on BBSI labels. Our methods use zero-shot prompting with no BBSI training data.

BBSI is the primary transfer target because its categories directly encode subtle actions related to social behavior. NTU serves as a supplementary test whose categories are not social in origin, probing whether the prior generalizes to action recognition beyond the social domain. A few labeled actions partially overlap between MMA 52 and NTU (e.g. “nodding” and “nod head/bow”), but the visual domains and annotation granularity differ.

Evaluation protocol. We treat each clip as a multi-label classification instance. The model produces a predicted label set Y^i\hat{Y}_i for each clip ii, which is compared against the ground truth label set YiY_i. We define per-sample counts: TPi=∣Yi∩Y^i∣\mathrm{TP}_i = \lvert Y_i \cap\hat{Y}_i \rvert, FPi=∣Y^i∖Yi∣\mathrm{FP}_i = \lvert\hat{Y}_i \setminus Y_i \rvert, FNi=∣Yi∖Y^i∣\mathrm{FN}_i = \lvert Y_i \setminus\hat{Y}_i \rvert. We report two F1 variants that capture different aspects of performance.

Sample-pooled F1 (F1pool\mathrm{F1}_{\mathrm{pool}}) aggregates all counts across samples before computing a single score:

F1pool=2∑iTPi2∑iTPi+∑iFPi+∑iFNi(10)\mathrm{F1}_{\mathrm{pool}} = \frac{2\sum_i \mathrm{TP}_i}{2\sum_i \mathrm{TP}_i + \sum_i \mathrm{FP}_i + \sum_i \mathrm{FN}_i} \tag*{(10)}

This reflects overall prediction quality but is influenced more by frequent classes.

Class averaged F1 (F1cls\mathrm{F1}_{\mathrm{cls}}) computes F1 for each class cc independently, then averages over all CC classes:

F1cls=1C∑c=1C2TPc2TPc+FPc+FNc(11)\mathrm{F1}_{\mathrm{cls}} = \frac{1}{C}\sum_{c=1}^{C}\frac{2\mathrm{TP}_c}{2\mathrm{TP}_c+\mathrm{FP}_c+\mathrm{FN}_c} \tag*{(11)}

This gives equal weight to every class regardless of frequency.

The three datasets use incompatible native metrics, so we adopt clip-level F1 for cross-taxonomy comparison. We additionally report BBSI and MMA 52 under their native metrics in § 4.2, confirming that relative gains hold. NTU’s native protocol requires training data we do not use, so we rely on clip-level F1.

Zero-shot prompting. We split each dataset’s label set into three anatomical groups (head, upper body, lower body) and make one VLM call per group, presenting the relevant category names with numeric identifiers; shorter candidate lists reduce the chance that

the VLM overlooks or confuses visually similar categories. The same prompt templates, parsing rules, and retry logic are used across all models and all compared methods; only the category names change. We therefore treat prompting as part of the evaluation protocol rather than part of the proposed method. A single prompt instead of grouped prompting lowers absolute scores for all methods by a similar margin without changing their ranking. Retry and union-merging are deterministic post-processing applied identically to all methods; disabling retries changes fewer than 2% of predictions and does not alter the ranking.

Models. We use three VLMs: Qwen3 VL 8B Instruct, InternVL3.5 8B Instruct, and InternVL3.5 4B Instruct [76]. The adapter is trained on Qwen3 VL using MMA 52 training data. For both InternVL3.5 variants, we freeze the Core learned on Qwen3 VL and train only a new Projection Head (§ 3.4). All three VLMs are frozen throughout; only the adapter parameters are trainable.

Main Results

Table 3 reports zero-shot recognition results across nine evaluation settings: three VLM backbones and three datasets. Each setting compares the baseline VLM receiving only RGB against PRIMA receiving only RGB after pose-guided training.

BackboneMethodMMA 52 (source) F1poolMMA 52 (source) F1clsBBSI (transfer) F1poolBBSI (transfer) F1clsNTU (transfer) F1poolNTU (transfer) F1cls
Qwen3 VL 8BRGB baseline0.2530.1620.4430.2190.3910.243
Qwen3 VL 8BPRIMA0.3240.1860.4900.2320.4380.278
InternVL3.5 8BRGB baseline0.1740.1090.3820.1560.3120.186
InternVL3.5 8BPRIMA (frozen Core)0.2130.1580.4270.2080.3580.227
InternVL3.5 4BRGB baseline0.1480.0910.3490.1370.2820.163
InternVL3.5 4BPRIMA (frozen Core)0.1750.1210.3840.1750.3160.196

Table 3. Zero-shot recognition results. All methods use the same VLM and prompting protocol with no target-dataset labels. PRIMA uses pose only during training; at inference, it receives only RGB. Best result per backbone in bold; † values across three seeds ≤0.005\leq0.005 on all settings. InternVL3.5 4B tests cross-scale transfer with the same frozen Core.

Pose information improves the ability of VLMs to capture fine-grained body motion. An Oracle receiving skeleton overlaid frames outperforms the RGB baseline in all six settings we measured on Qwen3 VL and InternVL3.5 8B. On MMA 52 it reaches 0.355 and 0.249 F1pool\mathrm{F1}_{\mathrm{pool}} against 0.253 and 0.174 for the RGB baseline; on BBSI it reaches 0.512 and 0.463 against 0.443 and 0.382. These Oracle runs are reported in text only and are not tabulated. The F1cls\mathrm{F1}_{\mathrm{cls}} pattern is similar, and the gap confirms that VLMs do not fully capture body structure from RGB alone.

PRIMA internalizes the pose signal. PRIMA improves over the RGB baseline in all nine settings without receiving any pose input at inference (Table 3). On MMA 52, F1pool\mathrm{F1}_{\mathrm{pool}} improves by +28% on Qwen3 VL (0.253 to 0.324) and +22% on InternVL3.5 8B (0.174 to 0.213), recovering about 70% and 52% of the Oracle gap respectively.

The body structure prior transfers across taxonomies. The adapter was trained on MMA 52 only, yet it improves recognition on both unseen target datasets. On BBSI, F1pool\mathrm{F1}_{\mathrm{pool}} improves by +11% on Qwen3 VL (0.443 to 0.490) and +12% on InternVL3.5 8B (0.382 to 0.427). On NTU, F1poolF_{1\mathrm{pool}} improves by +12% and +15% respectively. The F1clsF_{1\mathrm{cls}} gains are proportionally larger, indicating that the adapter helps rare classes as well. A paired bootstrap test (10,000 resamples) confirms significance at p<0.01p < 0.01 on all settings, including InternVL3.5 4B, where the frozen Core still yields relative F1poolF_{1\mathrm{pool}} gains of +10% to +18%. A randomly initialized Core paired with a trained Head remains near the baseline, so the trained Core’s content, not the Head’s alignment procedure, drives transfer.

VariantF1poolF_{1\mathrm{pool}}F1clsF_{1\mathrm{cls}}
PRIMA (full)0.3240.186
Loss components
w/o cosine loss (MSE only)0.3050.169
w/o MSE loss (cosine only)0.2910.158
w/o regularization (λ=0\lambda= 0)0.3130.175
Teacher signal
Skeleton on blank background0.2780.147
Scrambled skeleton0.2440.113
Control
Random adapter (no pose training)0.2520.129

Table 4. Ablation on MMA 52 with Qwen3 VL. Each row modifies one component.

Which behaviors benefit and which do not. Per-class analysis on BBSI shows that behaviors defined by clear spatial joint relationships benefit most (Hand-face +8.2 pp, Legs crossed +6.5 pp on Qwen3 VL), while behaviors requiring fine hand detail (Fumble, Smearing hands) or temporal trajectory (Gesture) degrade slightly, with the same ranking on Qwen3 VL and InternVL3.5 8B. A post-hoc temporal averaging window (w=3w=3) recovers most of the Gesture drop (−2.8 to −0.9 pp) at minimal cost to the spatial categories.

Comparison under the BBSI native evaluation protocol. Table 2 evaluates BBSI under its original frame-level MAP [4]; under MMA 52’s native temporal detection mAP, PRIMA improves [email protected] from 0.112 to 0.149. The gap to supervised baselines reflects training on target labels. The relevant finding is that PRIMA improves over the zero-shot baseline under both native protocols, confirming that the gains are not artifacts of our clip-level evaluation.

Ablation Studies

We ablate key design choices on MMA 52 using Qwen3 VL (Table 4).

Loss components. Removing cosine (MSE only) drops F1poolF_{1\mathrm{pool}} to 0.305: MSE matches feature magnitude but does not constrain direction, so the adapted features can drift in orientation while staying close in norm. Removing MSE (cosine only) drops it further to 0.291, because cosine alignment ignores scale differences that the projector is sensitive to. Removing regularization yields 0.313; without the penalty on δ\delta, the adapter makes larger modifications that fit the source domain more tightly but are harder to transfer.

Teacher signal. Rendering the skeleton on a blank background drops F1poolF_{1\mathrm{pool}} to 0.278: Without visual context, teacher features diverge too far from the VLM’s natural distribution. A scrambled skeleton control (keypoints randomly reassigned across body locations) drops to 0.244, confirming that anatomically correct structure is essential.

DatasetMethodF1poolF_{1\mathrm{pool}}F1clsF_{1\mathrm{cls}}
MMA 52RGB baseline0.2530.162
MMA 52LoRA (supervised)0.3620.214
MMA 52Supervised adapter0.3410.198
MMA 52PRIMA (pose-guided)0.3240.186
BBSIRGB baseline0.4430.219
BBSILoRA (supervised)0.4510.215
BBSISupervised adapter0.4620.221
BBSIPRIMA (pose-guided)0.4900.232
NTURGB baseline0.3910.243
NTULoRA (supervised)0.4010.246
NTUSupervised adapter0.4130.257
NTUPRIMA (pose-guided)0.4380.278

Table 5. Comparison with alternative adaptation methods on Qwen3 VL. All methods use the same zero-shot prompting protocol at evaluation. Supervised methods are trained on MMA 52 classification labels. PRIMA uses no classification labels.

Random adapter. An adapter with identical architecture but random untrained weights scores 0.252, essentially matching the baseline’s 0.253. The improvement from PRIMA comes from pose-guided training, not from the adapter’s structure.

Comparison with supervised adaptation. Table 5 compares PRIMA with LoRA [31] (rank 16) and a supervised adapter (same architecture, cross-entropy loss), both trained on MMA 52’s 52-class labels. On MMA 52, supervised methods outperform PRIMA as expected: LoRA leads by +0.038 F1poolF_{1\mathrm{pool}}. The comparison reverses on transfer: on BBSI, LoRA drops to 0.451 (near the 0.443 baseline) while PRIMA reaches 0.490; on NTU, PRIMA leads by a similar margin. Methods that fit source-label boundaries lose their advantage when the taxonomy changes. We exclude PeVL [81] because it requires target-task labels and pose at inference, so a direct comparison would differ in inputs rather than isolate the effect we study. The supervised adapter in Table 5 isolates this factor: same architecture, classification loss, and its BBSI transfer (0.462) barely exceeds the baseline (0.443), confirming that the training signal, not the architecture, limits cross-taxonomy transfer.

Two further controls isolate what drives transfer. An augmentation alignment baseline with the same architecture and loss but a pose-free teacher reaches 0.454 F1poolF_{1\mathrm{pool}} on BBSI against 0.490 for PRIMA, so the alignment procedure alone does not explain the gain. A Core with temporal instead of spatial attention, all else fixed, transfers worse (+4.3% over the baseline on BBSI versus +11% for PRIMA), indicating that spatial body-part modeling rather than temporal dynamics carries the prior across taxonomies.

Supporting Downstream Study: Social Attribute Prediction

A natural follow-up question is whether the improved behavior predictions carry information useful for downstream social interpretation. Recent work has shown that multimodal features extracted from video, audio, and transcripts can support automated assessments of high-level interaction qualities, including encouragement, warmth, and broader dimensions of teaching effectiveness [18, 29].

Figure 5. Downstream social attribute prediction on BBSI (Qwen3 VL, Ridge, leave-one-recording-out CV). (a) PRIMA improves correlation with human ratings on all five attributes; ∗: p<0.05p < 0.05, paired permutation test. (b) Per-class F1 gain correlates with each behavior’s weight in the rapport regression (r=0.71r = 0.71, p<0.01p < 0.01). Hf: Hand-face, Hm: Hand-mouth, Lc: Legs crossed, Fa: Fold arms, Se: Settle, Sh: Shrug, Ge: Gesture, Gr: Groom, Sc: Scratch, Lm: Leg movement, St: Stretching, Ac: Adjusting clothing, Fu: Fumble, Sm: Smearing hands.

We treat this as supporting evidence rather than a primary benchmark, given the small dataset (204 dyadic ratings, 22 recordings) and context-dependent nature of the attributes.

BBSI was recorded from group interactions in which each participant rated other participants on rapport, perceived dominance, leadership, competence, and liking after the session [5, 65]. These ratings provide a direct test: If PRIMA produces more accurate behavior predictions, and if those behaviors genuinely encode social information, then using PRIMA’s predictions as features should yield better social attribute estimates than using the baseline’s predictions.

Setup

For each subject in each recording, we aggregate clip-level behavior predictions into a 14-dimensional frequency vector (proportion of clips predicting each behavior). For each dyadic rating (subject ii rates subject jj), we use the vector of ratings for the rated subject as features and the rating as the target. We fit Ridge regression (α=1.0\alpha= 1.0) under leave-one-recording-out CV (22 folds) and report Spearman ρ\rho between predicted and actual scores. We use rank correlation because the ratings are ordinal. We assess significance with a paired permutation test (10,000 iterations) on Δρ=ρPRIMA−ρbaseline\Delta\rho= \rho_{\mathrm{PRIMA}}-\rho_{\mathrm{baseline}}.

We compare two feature sources: RGB baseline and PRIMA predictions, both from Qwen3 VL. No social labels are used during adapter training.

Results

fig:5a shows that PRIMA’s behavior features produce higher correlations than the baseline on all five attributes, significantly on dominance (ρ\rho: 0.21 → 0.30, p=0.02p = 0.02), rapport (0.18 → 0.27, p=0.03p = 0.03), and leadership (0.16 → 0.24, p=0.04p = 0.04). Competence and liking improve (+0.06, +0.05) but do not reach significance, consistent with prior work showing these constructs depend more on facial expression than body configuration [71]. The absolute correlations are modest, as expected from zero-shot binary predictions. Applying the same Ridge predictor to ground truth behavior annotations yields ρ≈0.36\rho\approx0.36 on rapport, against 0.27 for PRIMA and 0.18 for the baseline, so better recognition alone can close only about half of the remaining gap. A permutation control and L1 normalization both confirm that the gains are not artifacts.

Behavior-Level Attribution

The overall correlations do not indicate which specific recognition improvements are responsible. To answer this, we extract the standardized Ridge regression coefficients for each of the 14 behavior predictors in the rapport model and plot them against PRIMA’s per-class F1 gain (Figure 5b). The scatter reveals a positive association (r=0.71r = 0.71, p<0.01p < 0.01): The behaviors PRIMA improves most are also the strongest predictors of rapport. The three behaviors with the largest F1 gains, Hand-face (+8.2 pp), Hand-mouth (+7.1 pp), and Legs crossed (+6.5 pp), are classified as displacement activities or closed postures in the Ethological Coding System for Interviews [72] that underlies BBSI’s annotation. Displacement activities are established negative correlates of rapport [23, 40, 72]; their regression coefficients are strongly negative (β=−0.31,−0.25,−0.21\beta= -0.31, -0.25, -0.21), so more accurate detection translates directly into better rapport prediction. Gesture (−2.8 pp) carries a positive rapport coefficient (β=+0.19\beta= +0.19) but depends on temporal trajectory, so the spatial prior does not lead to improved predictions.

Conclusion

We proposed PRIMA, a lightweight adapter that uses human pose as a training-time teaching signal to inject a body-structure prior into frozen vision-language models, with no pose input at inference. Trained on a single micro-action dataset, the prior improves zero-shot recognition on its source domain, transfers to two unseen target taxonomies, and can be partially reused across a second VLM backbone with the shared Core frozen. The transfer is selective: spatial body-part relations benefit most, fine hand detail and temporal trajectory least.

In a supporting downstream study on BBSI, PRIMA’s predictions correlate more strongly with human ratings of rapport, dominance, and leadership, suggesting that better zero-shot behavior predictions can serve as a useful intermediate signal for social interpretation.

Limitations. The prior is bounded by what the skeleton represents: major body joints are tracked, but hand and finger detail is absent, and each frame is transformed independently without cross-frame dynamics. We evaluate reuse across two additional VLM backbones at 8B and 4B scale within the same architecture family; transfer across architectural families and outdoor or unconstrained settings remains untested. The absolute zero-shot scores remain lower than those of supervised methods trained on target data, and studying PRIMA as an initialization for few-shot adaptation is a natural next step.

Acknowledgments

This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under project number 563097140 as part of the Priority Programme SPP 2481 UGAZE (Understanding Gaze).

References

References

  1. [1]Ahmed Amer, Chirag Bhuvaneshwara, Gowtham K Addluri, Mohammed M Shaik, Vedant Bonde, and Philipp Müller. 2023. Backchannel detection and agreement estimation from video with transformer networks. In 2023 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–8.arxiv.org/abs/2306.01656
  2. [2]Michael Argyle. 2013. Bodily communication. Routledge.
  3. [3]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjing Ge, et al. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025).arxiv.org/abs/2511.21631
  4. [4]Michal Balazia, Philipp Müller, Ákos Levente Tánczos, August von Liechtenstein, and François Brémond. 2022. Bodily behaviors in social interaction: Novel annotations and state-of-the-art evaluation. In Proceedings of the 30th ACM International Conference on Multimedia. 70–79.DOI
  5. [5]Frank J Bernieri, John S Gillis, Janet M Davis, and Jon E Grahe. 1996. Dyad rapport and the accuracy of its judgment across situations: a lens model analysis. journal of Personality and Social Psychology 71, 1 (1996), 110.
  6. [6]Cigdem Beyan, Francesca Capozzi, Cristina Becchio, and Vittorio Murino. 2017. Prediction of the leadership style of an emergent leader using audio and visual nonverbal features. IEEE Transactions on Multimedia 20, 2 (2017), 441–456.DOI
  7. [7]Cigdem Beyan, Vasiliki-Maria Katsageorgiou, and Vittorio Murino. 2017. Moving as a leader: Detecting emergent leadership in small groups using body pose. In Proceedings of the 25th ACM international conference on Multimedia. 1425–1433.
  8. [8]Cigdem Beyan, Alessandro Vinciarelli, and Alessio Del Bue. 2023. Co-located human–human interaction analysis using nonverbal cues: A survey. Comput. Surveys 56, 5 (2023), 1–41.DOI
  9. [9]Judee K Burgoon, Nadia Magnanat-Thalmann, Maja Pantic, and Alessandro Vinciarelli. 2017. Social signal processing. Cambridge University Press.
  10. [10]Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2019. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE transactions on pattern analysis and machine intelligence 43, 1 (2019), 172–186.
  11. [11]Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. 2022. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems 35 (2022), 16664–16678.arxiv.org/abs/2205.13535
  12. [12]Xinyu Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. 2021. Channel-wise topology refinement graph convolution for skeleton-based action recognition. In Proceedings of the IEEE/CVF international conference on computer vision. 13359–13368.
  13. [13]Srijan Das, Rui Dai, Di Yang, and Francois Bremond. 2021. Vpnn++: Rethinking video-pose embeddings for understanding activities of daily living. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 12 (2021), 9703–9717.arxiv.org/abs/2105.08141
  14. [14]Srijan Das, Saurav Sharma, Rui Dai, Francois Bremond, and Monique Thonnat. 2020. Vpn: Learning video-pose embedding for activities of daily living. In European conference on computer vision. Springer, 72–90.arxiv.org/abs/2007.03056
  15. [15]Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. 2022. Revisiting skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2969–2978.arxiv.org/abs/2104.13586
  16. [16]Paul Ekman, Wallace V Friesen, et al. 1969. The repertoire of nonverbal behavior: Categories, origins, usage and coding. Vol. 1. Mouton de Gruyter Berlin.
  17. [17]Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision. 6202–6211.arxiv.org/abs/1812.03982
  18. [18]Tim Fütterer, Ruikun Hou, Babette Bühler, Efe Bozkir, Courtney Bell, Enkeljeda Kasneci, Peter Gerjets, and Ulrich Trautwein. 2026. Validating automated assessments of teaching effectiveness using multimodal data. Learning and Instruction 101 (2026), 102264.DOI
  19. [19]Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. 2024. Clip-adapter: Better vision-language models with feature adapters. International journal of computer vision 132, 2 (2024), 581–595.arxiv.org/abs/2110.04544
  20. [20]Nuno C Garcia, Pietro Morerio, and Vittorio Murino. 2018. Modality distillation with multiple stream networks for action recognition. In Proceedings of the European Conference on Computer Vision (ECCV). 103–118.arxiv.org/abs/1806.07110
  21. [21]Daniel Gatica-Perez. 2009. Automatic nonverbal analysis of social interaction in small groups: A review. Image and vision computing 27, 12 (2009), 1775–1787.DOI
  22. [22]Patricia Goldberg, Ömer Sümer, Kathleen Stürmer, Wolfgang Wagner, Richard Göllner, Peter Gerjets, Enkeljeda Kasneci, and Ulrich Trautwein. 2021. Attentive or not? Toward a machine learning approach to assessing students’ visible engagement in classroom instruction. Educational Psychology Review 33, 1 (2021), 27–49.
  23. [23]Matthew S Goodwin, Stephen S Intille, Fahd Albinali, and Wayne F Velicer. 2011. Automated detection of stereotypical motor movements. Journal of autism and developmental disorders 41, 6 (2011), 770–782.DOI
  24. [24]Dan Guo, Kun Li, Bin Hu, Yan Zhang, and Meng Wang. 2024. Benchmarking micro-action recognition: Dataset, methods, and applications. IEEE Transactions on Circuits and Systems for Video Technology 34, 7 (2024), 6238–6252.DOI
  25. [25]Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366 (2021).arxiv.org/abs/2110.04366
  26. [26]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).arxiv.org/abs/1503.02531
  27. [27]James Hong, Matthew Fisher, Michael Gharbi, and Kayvon Fatahalian. 2021. Video pose distillation for few-shot, fine-grained sports action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9254–9263.arxiv.org/abs/2109.01305
  28. [28]Ruikun Hou, Babette Bühler, Tim Fütterer, Efe Bozkir, Peter Gerjets, Ulrich Trautwein, and Enkeljeda Kasneci. 2025. Multimodal Assessment of Classroom Discourse Quality: A Text-Centered Attention-Based Multi-Task Learning Approach. International Educational Data Mining Society (2025).arxiv.org/abs/2505.07902
  29. [29]Ruikun Hou, Tim Fütterer, Babette Bühler, Efe Bozkir, Peter Gerjets, Ulrich Trautwein, and Enkeljeda Kasneci. 2024. Automated assessment of encouragement and warmth in classrooms leveraging multimodal emotional features and ChatGPT. In International conference on artificial intelligence in education. Springer, 60–74.DOI
  30. [30]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attarian, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning. PMLR, 2790–2799.arxiv.org/abs/1902.00751
  31. [31]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. Iclr 1, 2 (2022), 3.arxiv.org/abs/2106.09685
  32. [32]Hayley Hung and Daniel Gatica-Perez. 2010. Estimating cohesion in small groups using audio-visual nonverbal behavior. IEEE Transactions on Multimedia 12, 6 (2010), 563–575.DOI
  33. [33]Dinesh Babu Jayagopi, Hayley Hung, Chuchao Yeo, and Daniel Gatica-Perez. 2009. Modeling dominance in group conversations using nonverbal activity cues. IEEE Transactions on Audio, Speech, and Language Processing 17, 3 (2009), 501–513.DOI
  34. [34]Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning. PMLR, 4904–4916.
  35. [35]Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. 2022. Visual prompt tuning. In European conference on computer vision. Springer, 709–727.arxiv.org/abs/2203.12119
  36. [36]Tao Jiang, Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Liu, Yining Li, and Kai Chen. 2023. Rtpmose: Real-time multi-person pose estimation based on mmpose. arXiv preprint arXiv:2303.07399 (2023).arxiv.org/abs/2303.07399
  37. [37]Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. 2022. Prompting visual-language models for efficient video understanding. In European conference on computer vision. Springer, 105–124.arxiv.org/abs/2112.04478
  38. [38]A Kendon. 2004. Gesture: Visible Action as Utterance, Cambridge Uni.
  39. [39]Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 19113–19122.arxiv.org/abs/2210.03117
  40. [40]Mark L Knapp, Judith A Hall, and Terrence G Horgan. 1972. Nonverbal communication in human interaction. Thomson Wadsworth.
  41. [41]Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054 (2022).arxiv.org/abs/2202.10054
  42. [42]Sangmin Lee, Minzhi Li, Bolin Lai, Wengqi Jia, Fiona Ryan, Xu Cao, Ozgur Kara, Bikram Boote, Weiyan Shi, Diyi Yang, et al. 2024. Towards social ai: A survey on understanding social interactions. arXiv preprint arXiv:2409.15316 (2024).arxiv.org/abs/2409.15316
  43. [43]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. PMLR, 19730–19742.arxiv.org/abs/2301.12597
  44. [44]Kun Li, Pengyu Liu, Dan Guo, Fei Wang, Zhiliang Wu, Hehe Fan, and Meng Wang. 2025. Mmad: Multi-label micro-action detection in videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 13225–13236.arxiv.org/abs/2407.05311
  45. [45]Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot. 2019. NTU rgb+d 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions on pattern analysis and machine intelligence 42, 10 (2019), 2684–2701.DOI
  46. [46]Xin Liu, Henglin Shi, Haoyu Chen, Zitong Yu, Xiaobai Li, and Guoying Zhao. 2021. imigue: An identity-free video dataset for micro-gesture understanding and emotion analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10631–10642.arxiv.org/abs/2107.00285
  47. [47]Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).arxiv.org/abs/1711.05101
  48. [48]Marc Mehu and Klaus R Scherer. 2012. A psycho-ethological approach to social signal processing. Cognitive processing 13, Suppl 2 (2012), 397–414.pubmed.ncbi.nlm.nih.gov/22328016
  49. [49]Philipp Müller, Michal Balazia, Tobias Baur, Michael Dietz, Alexander Heimerl, Anna Penzkofer, Dominik Schiller, François Brémond, Jan Alexandersson, Elisa-beth André, et al. 2024. Multimediate’24: multi-domain engagement estimation. In Proceedings of the 32nd ACM International Conference on Multimedia. 11377–11382.
  50. [50]Philipp Müller, Michal Balazia, Tobias Baur, Michael Dietz, Alexander Heimerl, Dominik Schiller, Mohammed Guermal, Dominike Thomas, François Brémond, Jan Alexandersson, et al. 2023. Multimedia’23: Engagement estimation and bodily behaviour recognition in social interactions. In Proceedings of the 31st ACM International Conference on Multimedia. 9640–9645.DOI
  51. [51]Philipp Müller, Michael Dietz, Dominik Schiller, Dominike Thomas, Guanhua Zhang, Patrick Gebhard, Elisabeth André, and Andreas Bulling. 2021. Multimediable: Multi-modal group behaviour analysis for artificial mediation. In Proceedings of the 29th ACM International Conference on Multimedia. 4878–4882.
  52. [52]Philipp Müller, Michael Xuelin Huang, and Andreas Bulling. 2018. Detecting low rapport during natural interactions in small groups from non-verbal behaviour. In Proceedings of the 23rd International Conference on Intelligent User Interfaces. 153–164.DOI
  53. [53]Philipp Matthias Muller and Andreas Bulling. 2019. Emergent leadership detection across datasets. In 2019 International Conference on Multimodal Interaction. 274–278.arxiv.org/abs/1905.02058
  54. [54]Bin Liu, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. 2022. Expanding language-image pretrained models for general video recognition. In European conference on computer vision. Springer, 1–18.arxiv.org/abs/2208.02816
  55. [55]Fatemeh Noroozi, Marina Marjanovic, Angelina Nje gus, Sergio Escalera, and Gholamreza Anbarjafari. 2017. Audio-visual emotion recognition in video clips. IEEE Transactions on Affective Computing 10, 1 (2017), 60–75.
  56. [56]Lya K Paas Oliveros, Javier Villanueva Valle, Susana I González Arredondo, Ana Fresán, Iván Arango de Montis, Martin Brüne, and Jairo Muñoz Delgado. 2015. Study of application and validation of the Ethological Coding System for Interviews (ECSI). Salud mental 38, 1 (2015), 41–46.
  57. [57]Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. 2022. Strategies for domain adaptation-efficient image-to-video transfer learning. Advances in Neural Information Processing Systems 35 (2022), 26462–26477.
  58. [58]Isabella Poggi and Francesca D’Errico. 2012. Social signals: a framework in terms of goals and beliefs. Cognitive Processing 13, Suppl 2 (2012), 427–445.pubmed.ncbi.nlm.nih.gov/22797857
  59. [59]Alec Radford, Jong Wook Kim, Chris Halla cy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763.arxiv.org/abs/2103.00020
  60. [60]Shyam Rajagopalan, Abhinav Dhall, and Roland Goecke. 2013. Self-stimulatory behaviours in the wild for autism diagnosis. In Proceedings of the IEEE international conference on computer vision workshops. 755–761.
  61. [61]Hanoona Rasheed, Muhammad Uza ir Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6545–6554.arxiv.org/abs/2212.03640
  62. [62]Dominick Reilly, Aman Chadha, and Srijan Das. 2023. Seeing the pose in the pixels: Learning pose-aware representations in vision transformers. arXiv preprint arXiv:2306.09331 (2023).arxiv.org/abs/2306.09331
  63. [63]Marcus Rohrbach, Michael Stark, and Bernt Schiele. 2011. Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In CVPR 2011 IEEE, 1641–1648.DOI
  64. [64]Ognjen Rudovic, Jaeryoung Lee, Miles Dai, Björn Schuller, and Rosalind W Picard. 2018. Personalized machine learning for robot perception of affect and engagement in autism therapy. Science Robotics 3, 19 (2018), eaa o6760.DOI
  65. [65]Dairazalia Sanchez-Cortes, Oya Aran, Marianne Schmid Mast, and Daniel Gatica-Perez. 2011. A nonverbal behavior approach to identify emergent leaders in small groups. IEEE transactions on multimedia 14, 3 (2011), 816–832.DOI
  66. [66]Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. 2016. N t rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1010–1019.
  67. [67]Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. 2019. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12026–12035.arxiv.org/abs/1805.07694
  68. [68]Ömer Sümer, Peter G e r j e t s, Ulrich Trautwein, and Enkeljeda Kasneci. 2020. Attention flow: End-to-end joint attention estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3327–3336.DOI
  69. [69]Ömer Sümer, Patricia Goldberg, Sidney D’Mello, Peter Gerjets, Ulrich Trautwein, and Enkeljeda Kasneci. 2021. Multimodal engagement analysis from facial videos in the classroom. IEEE Transactions on Affective Computing 14, 2 (2021), 1012–1027.arxiv.org/abs/2101.04215
  70. [70]Fida Mohammad Thoker and Juergen Gall. 2019. Cross-modal knowledge distillation for action recognition. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 6–10.arxiv.org/abs/1910.04641
  71. [71]Linda Tickle-Degnen and Robert Rosenthal. 1990. The nature of rapport and its nonverbal correlates. Psychological inquiry 1, 4 (1990), 285–293.DOI
  72. [72]Alfonso Troisi. 1999. Ethological research in clinical psychiatry: the study of human behavior during interviews. Neuroscience & Biobehavioral Reviews 23, 7 (1999), 905–913.pubmed.ncbi.nlm.nih.gov/10580305
  73. [73]Alessandro Vinciarelli, Maja Pantic, and Hervé Bourlard. 2009. Social signal processing: Survey of an emerging domain. Image and vision computing 27, 12 (2009), 1743–1759.DOI
  74. [74]Alessandro Vinciarelli, Maja Pantic, Dirk Heylen, Catherine Pelachaud, Isabella Poggi, Francesca D’Errico, and Marc Schroeder. 2011. Bridging the gap between social animal and unsocial machine: A survey of social signal processing. IEEE transactions on affective computing 3, 1 (2011), 69–87.DOI
  75. [75]Mengmeng Wang, Jiazheng Xing, and Yong Liu. 2021. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472 (2021).arxiv.org/abs/2109.08472
  76. [76]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. 2025. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025).arxiv.org/abs/2508.18265
  77. [77]Jianzong Wu, Jiangtai Li, Shil i n Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, et al. 2024. Towards open vocabulary learning: A survey. IEEE transactions on pattern analysis and machine intelligence 46, 7 (2024), 5092–5113.arxiv.org/abs/2306.15880
  78. [78]Xun Xu, Timothy M Hospedales, and Shaogang Gong. 2016. Multi-task zero-shot action recognition with prioritised data augmentation. In European Conference on Computer Vision. Springer, 343–359.
  79. [79]Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.arxiv.org/abs/1801.07455
  80. [80]Taoqiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. 2023. Aim: Adapting image models for efficient video action recognition. arXiv preprint arXiv:2302.03024 (2023).arxiv.org/abs/2302.03024
  81. [81]Haosong Zhang, Mei Chee Leong, Liyuan Li, and Weisi Lin. 2024. Pevl: Pose-enhanced vision-language model for fine-grained human action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18857–18867.DOI
  82. [82]Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE transactions on pattern analysis and machine intelligence 46, 8 (2024), 5625–5644.arxiv.org/abs/2304.00685
  83. [83]Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16816–16825.arxiv.org/abs/2203.05557
  84. [84]Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision-language models. International journal of computer vision 130, 9 (2022), 2337–2348.DOI
  85. [85]Chaoyang Zhu and Long Chen. 2024. A survey on open-vocabulary detection and segmentation: Past, present, and future. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 12 (2024), 8954–8975.arxiv.org/abs/2307.09220

Paper details

vlnc:W_tyrd4fmz
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents