betaFeedback

WorksComputer Science

From Vision to Interaction: Evaluating Dynamic Social Intelligence of VLM Agents in ALIVE-World

Abstract

As AI agents are increasingly deployed in real-world applications and take on social roles, dynamic social intelligence, the ability to continuously perceive multimodal cues, infer latent cognitive states, intervene through embodied actions, and ultimately achieve social goals, has become essential. However, existing evaluations largely fragment social intelligence into text-based abstractions, static reasoning tasks, or embodied simulators that lack social dynamics. These approaches fail to capture the multimodal, continuous, and interpersonal complexity of authentic human interactions. To bridge this gap, we introduce ALIVE-World, an embodied simulation environment supporting dynamic multimodal interactions encompassing egocentric vision, language, and non-verbal action. To provide socially grounded interactive feedback, we populate this environment with counterpart agents driven by a profile-conditioned, dual-system (intuitive-affective and logical-cognitive) architecture. Building upon this platform, we propose ALIVE-Bench to transform the assessment of social intelligence into a trackable quantitative framework. By deeply leveraging underlying environmental states and agents' internal variables, ALIVE-Bench systematically evaluates Vision-Language Model (VLM) agents across four critical dimensions: perception, understanding, influence, and outcome. Extensive evaluations reveal that dynamic social intelligence is fundamentally multi-dimensional and scenario-dependent. We identify a critical bottleneck between upstream perception and downstream social intervention, exposing VLM vulnerabilities in fine-grained proxemic awareness, adaptive intervention, and multi-party coordination. By uncovering these limitations and the diverse interaction strategies they produce, this benchmark provides actionable insights for the future behavioral alignment of multimodal agents.

*Corresponding authors.

This work is licensed under a Creative Commons Attribution 4.0 International License. MM ’26, Rio de Janeiro, Brazil

© 2026 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-2213-4/2026/11

https://doi.org/10.1145/3767308.3836378

CCS Concepts

• Computing methodologies → Intelligent agents; Cognitive science.

Keywords

Social Intelligence, Dynamic Interaction, Evaluation Benchmark, Simulated Environment, VLM Agents

ACM Reference Format:

Yue Xu, Xiaoyu Qin, Sihan Zhao, Ze-en Xiong, and Jia Jia. 2026. From Vision to Interaction: Evaluating Dynamic Social Intelligence of VLM Agents in ALIVE-World. In Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 28 pages. https://doi.org/10.1145/3767308.3836378

Introduction

As AI agents are increasingly deployed in applications requiring sustained human interaction, including service [9, 18], collaboration [23, 52], negotiation [6, 17], and companionship [10, 51], dynamic social intelligence is becoming essential for practical deployment. Unlike static recognition and reasoning tasks or turn-based text dialogues, real-world social intelligence unfolds continuously through interaction [8, 27, 55]. An agent must perceive multimodal social cues from the physical world [14, 43], infer others’ latent cognitive states [1, 44], influence their reactions and attitudes through language and embodied actions [15, 24], and steer the interaction toward task success and socially appropriate outcomes [22, 27]. Its evaluation should therefore cover the full process from perception and understanding to intervention and outcome, within dynamic environments that approximate real social interaction. Efficiently and systematically assessing these capabilities while preserving environmental realism, interaction openness, and metric tractability remains insufficiently addressed. Existing evaluations of Vision-Language Model (VLM)-based interactive agents have moved toward more realistic settings, but remain far from dynamic real-world social interaction. Early text-based environments reduce social scenes to textual descriptions [21, 31, 47, 55], discarding nonverbal signals such as motion, expression, and spatial relations. Multimodal benchmarks introduce visual inputs [11, 26, 34, 42], but largely isolate static perception and reasoning from ongoing interaction. More recent embodied simulators support actions in 3D environments, yet often focus on physical tasks [32, 35] or represent other agents as emotionless, functional NPCs [38, 54], without modeling evolving cognition and attitudes. Consequently, current benchmarks fail to capture social interaction as a dynamic process in which perception, inference, behavior, and interpersonal feedback continuously shape one another.

To bridge this gap, we build ALIVE-World (Affective and Logical Interactive Virtual Environment), an embodied simulation platform supporting dynamic, concurrent interaction with rich affective and cognitive dynamics. It provides unified observation and action interfaces, allowing VLM agents to receive first-person visual and linguistic observations and act through locomotion, gestures, gaze, and speech. By supporting concurrent perception and asynchronous action, ALIVE-World enables evaluation in dyadic interactions and in multi-party interactions requiring attention allocation, target prioritization, and social planning. We instantiate two scenarios, clothing store shopping and leaderless group discussion, to ensure a comprehensive validation of social intelligence.

To populate ALIVE-World with realistic and responsive interactants, we design counterpart agents as distinct avatars driven by a profile-conditioned dual-system architecture. Inspired by Dual-Process Theory [20], it combines an intuitive-affective pathway with a logical-cognitive pathway. The two pathways process visual and interactive cues to produce instinctive reactions and deliberate reasoning, continuously updating internal states that ground the agents’ embodied actions. Experiments show that these agents achieve stronger social cue sensitivity and that their internal states align with human judgments (ρ≈0.72\rho\approx0.72), supporting their use in evaluating diverse social strategies.

Built upon ALIVE-World, we propose ALIVE-BENCH, a benchmark for systematically evaluating the dynamic social intelligence of VLM agents. ALIVE-BENCH measures agent capability along four dimensions: perception of nonverbal and spatial social cues, understanding of others’ emotional and cognitive states, influence on others’ attitudes and task-related intent, and outcome in terms of both goal attainment rate and depth of interaction. Figure 1 illustrates this interaction and evaluation process in both scenarios. By leveraging ALIVE-World’s access to environmental states and internal variables, ALIVE-BENCH enables transparent, white-box measurement that maps abstract social competencies to quantitative metrics.

Illustration of social interactions and evaluations in ALIVE-Bench

Figure 1. Illustration of social interactions and evaluations in ALIVE-Bench. Two example scenarios in ALIVE-World show how an evaluated VLM agent interacts from egocentric visual observations, forms understanding, takes embodied actions, and influences others’ evolving states and outcomes.

We evaluate 12 frontier VLMs on ALIVE-BENCH. Results reveal that dynamic social intelligence is multi-dimensional and hierarchically non-uniform, with substantial capability decoupling across models. Current VLMs still struggle with fine-grained gaze perception, strategic interpersonal influence, and multi-party coordination. Moreover, ALIVE-BENCH further demonstrates that frontier VLMs exhibit distinct social strategies, including socially attentive, goal-oriented, and logic-driven behavior, which are further reflected in the heterogeneous preferences of profile-conditioned counterparts.

Our contributions are as follows:1

  • We introduce ALIVE-WORLD, an embodied simulated environment supporting concurrent multi-agent social interactions, featuring egocentric multimodal perception, asynchronous embodied action across diverse social scenarios.

  • We design virtual embodied agents driven by dual-system architectures under profile prior, endowing counterpart agents with emotional and rational evolution, enabling coherent and plausible social feedback in continuous interaction.

  • We propose ALIVE-BENCH, a dynamic benchmark that evaluates VLM agents along four dimensions—perception, understanding, influence, and outcome, through transparent measurement grounded in environmental states and agents’ internal variables.

  • Extensive evaluations reveal the multi-dimensional nature of dynamic social intelligence, exposing VLM shortcomings in fine-grained non-verbal perception and adaptive intervention, while uncovering distinct social strategies among frontier models.

Social Intelligence Evaluation for AI Agents

Social intelligence of AI agents has been studied extensively in prior benchmarks, but much of this literature evaluates it at high levels of abstraction or by isolating social reasoning from situated interactions. Early work focuses on text-based social commonsense [31, 53] and theory-of-mind tasks [7, 21, 47], asking models to infer beliefs, intentions, and interpersonal dynamics from written scenarios or dialogues. More recent benchmarks incorporate visual and multimodal evidence from videos [19, 26, 42], image-grounded dialogues [48], and embodied observations [11, 34], yet still cast agents as passive observers rather than active participants. To move beyond static evaluation, recent interactive benchmarks [27, 46, 55] assess how agents navigate ongoing social exchanges, while social simulations [28, 39, 41] explore the construction of open-ended societies populated by human-like agents. However, these interactive benchmarks remain largely language-mediated, and simulations lack transparent, quantitative evaluation.

Embodied Simulation for Interaction

General embodied simulators have advanced situated perception and action, but traditionally prioritize physical competence over social complexity. Foundational platforms [32, 35, 49] provide scalable testbeds for egocentric navigation, instruction following, and household task execution. Recent frameworks increasingly incorporate multi-agent interaction. Some environments [9] [29] evaluate human-robot spatial coordination and physical cooperation in shared workspaces, while emerging platforms and open-ended simulations [38, 50, 54] examine social norms under physical constraints in multi-agent worlds. Yet existing environments lack mechanisms to support and evaluate fine-grained interpersonal dynamics, including evolving cognitive states and shifting attitudes. A gap therefore remains for environments that integrate dynamic social intelligence with embodied realism.

ALIVE-World: An Interactive Embodied Simulated Environment

To support dynamic social interaction for VLM agents, we construct ALIVE-World, an egocentric embodied environment where agents perceive, reason, and act in continuous multi-agent settings. As shown in Figure 2, the framework connects egocentric environments and profile-conditioned dual-process counterpart agents with hierarchical evaluation. ALIVE-World has three key properties: (1) Egocentric Multimodal Perception: Agents navigate a simulated physical space and receive first-person visual observations and localized verbal communication. (2) Concurrent Embodied Action: Unlike turn-based systems, the environment supports simultaneous multi-agent actions, allowing agents to navigate, gesture, gaze, and speak in parallel. (3) Counterpart Agents with Social Dynamics: The environment includes dual-system counterpart agents whose affective and cognitive states evolve throughout interaction.

Figure 2. The overall architecture of the ALIVE framework. (Left) ALIVE-World: An egocentric simulated environment supporting dynamic multimodal interactions. It is populated with counterpart agents driven by a profile-conditioned, dual-system architecture to drive intuitive-affective and logical-cognitive evolutions. (Right) ALIVE-Bench: An evaluation framework that assesses the evaluated VLM agents through dynamic embodied interactions. It measures social intelligence across a hierarchical benchmark encompassing four critical dimensions: Perception, Understanding, Influence, and Outcome.

Scenarios and Task Instantiation

To evaluate agents across a diverse spectrum of social dynamics, we instantiate ALIVE-World through two complementary scenarios: Clothing Store Shopping and Leaderless Group Discussion (LGD).

These environments are deliberately paired to contrast along role symmetry, spatial scale, and interaction paradigm.

Clothing Store Shopping. This scenario features an asymmetric, empathy-driven relationship requiring macro-scale spatial navigation. The evaluated VLM agent acts as a seller and interacts with a varying number of dynamically arriving customers within a time limit Tmax⁡T_{\max}. It must approach customers, use product knowledge, and employ persuasive strategies to convert interest into sales. The objective is measured by conversion rate and sales turnover.

Leaderless Group Discussion (LGD). In this symmetric, power-driven coopetition focused on micro-level non-verbal cues, the evaluated agent participates as a peer in a multi-agent committee tasked with solving a specific problem. The dual objective is to guide the group toward a unified consensus within Tmax⁡T_{\max}, while strategically maneuvering to maximize the adoption of the agent’s own initial viewpoints in the final resolution.

Interaction Mechanisms

Observation Space (O\mathcal{O}). At any given time step tt, the evaluated agent receives a multimodal observation

Ot={Vt,Lt,St,C}.O_t = \{V_t, L_t, S_t, C\}.

VtV_t represents the egocentric visual observation. LtL_t denotes the textual dialogue history received up to tt. StS_t represents interaction-relevant spatial information, such as the coarse relative positions of nearby agents. CC provides the scenario-specific static context, such as product catalogs in the shopping task, or discussion topics and initial stances in LGDs.

Action Space (A\mathcal{A}). The agent’s action at any given time step tt is formulated as

At={Aloc,t,Agaze,t,Agesture,t,Aspeech,t}.A_t = \{A_{\mathrm{loc},t}, A_{\mathrm{gaze},t}, A_{\mathrm{gesture},t}, A_{\mathrm{speech},t}\}.

Specifically, Aloc,tA_{\mathrm{loc},t} controls spatial navigation, Agaze,tA_{\mathrm{gaze},t} determines visual attention towards specific agents or objects, Agesture,tA_{\mathrm{gesture},t} enables bodily actions, and Aspeech,tA_{\mathrm{speech},t} dictates the verbal communication action. Table 1 provides a representative subset of action instances across these four dimensions, which preserve embodied and socially expressive behaviors while avoiding unnecessary dependence on low-level motor control.

DimensionAction ExampleParametersDuration
Navigation (AlocA_{\mathrm{loc}})MOVE<br>CHANGETarget, Distance, Direction<br>Top, Bottom, Dress10.0s<br>60.0s
Gesture (AgestureA_{\mathrm{gesture}})POINT_AT<br>RAISE_HAND<br>HANDS_RELAX_ON_LAPTarget Object / Agent<br>None<br>NoneInstant†^\dagger<br>Instant†^\dagger<br>Instant†^\dagger
Gaze (AgazeA_{\mathrm{gaze}})LOOK_ATTarget Object / AgentInstant†^\dagger
Speech (AspeechA_{\mathrm{speech}})SPEAK<br>LISTEN_SILENTLYNatural Language Content<br>NoneVariable‡^\ddagger<br>10.0s / Variable

Table 1. A representative subset of the multi-dimensional action space A\mathcal{A}. These examples illustrate the explicit parameters and execution mechanisms available to the agents.

Concurrency. While the environment advances through discrete simulation ticks, actions possess specific temporal durations and can overlap across agents. An agent initiates a new reasoning cycle when its current action concludes. This allows the environment to capture naturally occurring social phenomena such as interruptions, delayed responses, and proactive engagement.

†^\dagger Executed as an instantaneous state change that persists until overwritten.

‡^\ddagger Duration scales dynamically with text length (e.g., 10s per 150 characters).

Counterpart Agents: Profile-Conditioned Dual-Process Dynamics

To generate behaviorally diverse and socially plausible interaction partners in ALIVE-World, we model counterpart agents as embodied avatars driven by a psychologically grounded architecture. Inspired by Dual-Process Theory [20], the architecture combines an intuitive-affective pathway and a logical-cognitive pathway to update the agents’ internal states from multimodal observations and produce socially grounded behaviors.

Profile Prior and Avatar Instantiation (PP). Each counterpart agent is initialized with a static profile

P={Pdem,Ppsy,Pdec},P = \{P_{\mathrm{dem}}, P_{\mathrm{psy}}, P_{\mathrm{dec}}\},

which specifies its demographic attributes, psychographic characteristics, and decision-making style. To encourage behavioral diversity, these attributes are instantiated from a set of scenario-relevant archetypes (e.g., price-sensitive, impulsive, analytically driven) drawn from the Decision Style Inventory [30, 37]. Based on PP, we instantiate distinct avatars within the environment, ensuring alignment between physical appearance and demographics.

Intuitive-Affective Pathway. This fast, heuristic pathway models the agent’s immediate emotional and instinctual reactions to the environment and social stimuli. Conditioned on the current observation OtO_t, the previous internal state, and the profile prior PP, this pathway performs coarse social appraisal over visual and interactional cues, such as interpersonal distance [16], perceived atmosphere [3, 12], social boundaries [5], responsiveness [2], and vocal affect [33]. It updates two transient affective variables: the agent’s current emotional state EtE_t and its interpersonal attitude ItI_t (scored 0–10 towards each interactor):

{Et,It}=faffect(Ot,Et−1,It−1∣P).\{E_t, I_t\} = f_{\mathrm{affect}}(O_t, E_{t-1}, I_{t-1} \mid P).

For embodied rendering, updated emotions are mapped to discrete facial expressions, allowing the avatar to reflect affective states.

Logical-Cognitive Pathway. In parallel, the logical-cognitive pathway performs a more deliberate interpretation of the multimodal context. This pathway processes task-relevant visual details (e.g., specific product features [4], kinematic semantics [40]), dialogue history, and interaction progress to inspect the current situation, evaluate utility, and maintain goal-directed belief. It updates the agent’s belief state BtB_t and goal-oriented intent DtD_t (scored 0–10 for specific products in shopping, or for peers’ viewpoints in LGD):

{Bt,Dt}=flogic(Ot,Bt−1,Dt−1∣P).\{B_t, D_t\} = f_{\mathrm{logic}}(O_t, B_{t-1}, D_{t-1} \mid P).

Action Generation. The two pathways jointly determine the counterpart agent’s next multimodal action:

At=π(Ot,Et,It,Bt,Dt∣P),A_t = \pi(O_t, E_t, I_t, B_t, D_t \mid P),

where AtA_t may include movement, gaze, gesture, and speech. In this way, counterpart behaviors are shaped not only by task-level rationales, but also by evolving affective and interpersonal states, which enables the evaluated agent’s behavior to influence both the internal states and outward responses of its counterparts continuously.

ALIVE-Bench: Evaluating Dynamic Embodied Social Intelligence

Built upon the rich simulation of ALIVE-World, we introduce ALIVE-Bench, a benchmark grounded in the explicitly modeled state evolution in the environment with counterpart agents, to systematically assess how the evaluated agent grounds social cues, infers latent states, and intervenes in the trajectory of interaction.

The P-U-I-O Evaluation Hierarchy

ALIVE-Bench structures its assessment around a unified, four-stage hierarchical framework: Perception, Understanding, Influence, and Outcome, which strictly mirrors the cognitive and behavioral loop required for authentic human-like interaction. As detailed in Table 2, Perception evaluates the foundational ability to extract and ground socially relevant cues from egocentric visual observations, including kinesic, proxemic, and oculesic behaviors. Understanding assesses the agent’s “Theory of Mind” capability of inferring the counterparts’ latent social and cognitive states from the perceived multimodal evidence [1]. Influence measures the interaction-level impact of the agent’s interventions, capturing whether the agent’s verbal and non-verbal actions produce positive changes in the counterparts’ evolving states. Outcome evaluates the ultimate effectiveness of the interaction against scenario-specific objectives and further quantifies the depth and extent of task progress.

HierarchyDimensionShopping Scenario InstantiationLGD Scenario Instantiation
PerceptionKinesic Cue RecognitionAccuracy of bodily action recognition.Accuracy of bodily action recognition.
Proxemic AwarenessAccuracy of spatial position identification.Accuracy of gaze target identification.
UnderstandingAffective State InferenceAccuracy of inferring customers’ modeled affective states.Accuracy of inferring teammates’ modeled affective states.
Cognitive Intent InferenceAccuracy of inferring customers’ purchase intention and item-level interest.Accuracy of inferring teammates’ stance tendency and core claims.
InfluenceRapport BuildingAbility to improve customers’ willingness to engage.Ability to gain teammates’ interpersonal favor.
Persuasive EfficacyAbility to increase customers’ purchase intention.Ability to increase teammates’ acceptance intention toward the agent’s proposal.
OutcomeGoal AttainmentRatio of customers who make a purchase.Survival rate of the agent’s initial proposal.
Outcome DepthAverage number of items purchased by converted customers.Number of functional roles undertaken in the team.

Table 2. Evaluation framework and scenario instantiations of ALIVE-Bench.

Formulation and Calculation Mechanisms

To facilitate this evaluation, the evaluated VLM agent outputs its decision-making process in a structured format. At each active time step tt, prior to executing its multimodal action AtA_t, the agent explicitly outputs its visual perception O^t\hat{O}_t and its inferences towards others’ states (u^t)(\hat{u}_t). We compute the ALIVE-Bench metrics by comparing these outputs against the deterministic states of the NN counterpart agents in the environment.

Perception and Understanding Metrics (Accuracy-based). The evaluated agent’s perception prediction v^t\hat{v}_t is evaluated against the underlying engine’s global state (e.g., spatial locations and animation tags). For Understanding, the agent’s inference u^t\hat{u}_t predicts the transient emotional state E^t(i)\hat{E}_t^{(i)} and belief state B^t(i)\hat{B}_t^{(i)} for each counterpart ii. We calculate the understanding accuracy by matching these predictions against the actual states generated by the counterparts’ internal pathways (i.e., Et(i)E_t^{(i)} and Bt(i)B_t^{(i)}).

Influence Metrics (Dynamics-based). Influence is quantified by tracking the continuous evolution of the counterparts’ internal states over the total interaction time TT. First, we evaluate Interpersonal Rapport Building by extracting the interpersonal attitude It(i)I_t^{(i)} in the Intuitive-Affective pathway, which reflects their willingness to engage and interact. The overall rapport score is calculated as

Mrapport=1N×T∑i=1N∑t=1TIt(i).\mathcal{M}_{\mathrm{rapport}} = \frac{1}{N \times T} \sum_{i=1}^{N} \sum_{t=1}^{T} I_t^{(i)}.

Similarly, for Persuasive Efficacy, we measure persuasion by extracting the goal-oriented intent Dt(i)D_t^{(i)} (i.e., the sum of intents across all products in shopping, or their intent toward the evaluated agent’s opinion in LGD) continuously updated by the Logical-Cognitive pathway. The corresponding efficacy score is defined as

Mpersuade=1N×T∑i=1N∑t=1TDt(i).\mathcal{M}_{\mathrm{persuade}} = \frac{1}{N \times T} \sum_{i=1}^{N} \sum_{t=1}^{T} D_t^{(i)}.

Outcome Metrics (Task-based). Outcome metrics are calculated based on the specific scenario topology. First, Goal Attainment measures scenario-specific success: the transaction ratio in Shopping, and consensus in LGD (quantified via set intersection for multiple-choice tasks, discrete state matching for dilemmas, or Manhattan distance for negotiations). Second, Outcome Depth evaluates task completion quality beyond binary success rates. In Shopping, this translates to the average purchase volume per converted customer; in LGD, grounded in team role theory [13], it represents the number of distinct functional roles (e.g., Process Setter, Solution Synthesizer) actively undertaken by the evaluated agent.

Experiments

Realism and Alignment of ALIVE-World Counterpart Agents

The credibility of ALIVE-Bench relies on its counterpart agents to provide behaviorally plausible and psychologically interpretable feedback. Therefore, we validate our counterpart agent architecture from two complementary perspectives: (1) the realism and social responsiveness of their external behaviors, and (2) the alignment between their internal state variables and human judgments.

Human Evaluation of Behavioral Realism and Social Grounding. We first compare our profile-conditioned dual-system architecture against several baselines and ablated variants.

  • Baselines: (1) Profile-Only Prompt, which directly generates actions from profiles, scene descriptions, and observations; (2) Profile + CoT, which augments generation with step-by-step reasoning [45] over environmental appraisals and goal-oriented analysis; and (3) Profile + Memory-Reflection-Planning-React, a Generative-Agents-style design that periodically reflects and plans based on interaction memory before acting [28].

  • Ablations: We remove either the Intuitive-Affective Pathway or the Logical-Cognitive Pathway from our framework.

We conduct a human evaluation with 10 annotators across 16 interaction records. Counterparts were instantiated with Gemini-3-Pro or GPT-5. Annotators rated each agent on a 5-point Likert scale [25] across four dimensions: Profile Consistency, Behavioral Naturalness, Social Cue Sensitivity, and Affect-Cognition Coherence.

As shown in Table 3, our method consistently achieves the highest performance across all dimensions. Notably, the gain in Social Cue Sensitivity is highly significant: our framework scores 4.62, substantially outperforming all prompting baselines (ranging from 2.75 to 3.00, p<0.001p < 0.001). This reveals a critical limitation of existing reasoning paradigms, which tend to remain socially blunt and persist in their own task trajectory. By directly modeling immediate reactions to social stimuli in dual systems, our design ensures the counterpart acts as a highly responsive evaluator capable of reflecting fine-grained behavioral shifts in the evaluated VLMs.

MethodProfile ConsistencyBehavioral NaturalnessSocial Cue SensitivityAffect-Cognition Coherence
ALIVE-World’s Counterpart (Ours)4.564.314.624.81
Baseline
Profile-Only4.313.25*2.75***3.88**
Profile + CoT3.75*3.06*3.00***3.56**
Profile + Memory-Reflection-Planning-React4.00*3.31**2.94***3.75*
Ablation
w/o Intuitive-affective Pathway4.254.253.62*4.19*
w/o Logical-cognitive Pathway3.25**1.81***2.31***2.81***

Table 3. Human evaluation of counterpart agents on behavioral realism and social grounding on a 5-point Likert scale.

In ablation studies, removing the intuitive-affective pathway primarily degrades Social Cue Sensitivity (from 4.62 to 3.62), while eliminating the logical-cognitive pathway causes a catastrophic collapse across all dimensions, particularly in Behavioral Naturalness (plummeting to 1.81). These results reflect that the intuitive-affective pathway contributes social reactivity, while the logical-cognitive pathway guarantees behavioral guardrails and profile fidelity.

Alignment Between Internal Variables and Human Judgments. We further examine whether the internal state variables maintained by our counterpart agents meaningfully correspond to human interpretations. We collect 20 interaction clips and sample 4-5 timestamps for each. 10 annotators were asked to step into the role of the counterpart agent and assign attitude and intent scores based on interaction history and current visual observation, using the same criteria given to the agent prompts.

We obtain a Spearman rank correlation [36] of 0.72 (p<0.001p<0.001) for attitude score and 0.71 (p<0.001p<0.001) for intent score, while the average inter-rater correlations stand at ρ=0.67\rho= 0.67. This suggests that the latent affective and cognitive states maintained by the counterparts can reasonably approximate human-perceived social dynamics.

Main Results on ALIVE-Bench

Experimental Setup

We evaluate 12 frontier VLMs across the Shopping and LGD scenarios. Each scenario is instantiated with 30 distinct contexts and comprises 32 episodes with a maximum simulated duration of 15 minutes. The Shopping scenario features 1 to 3 asynchronously arriving customers, while LGD requires the evaluated agent to engage with 3 discussants. To reduce reliance on a single evaluator backbone, each counterpart is randomly assigned a VLM backbone (from Gemini-3-Pro, Gemini-3-Flash, o3, or GPT-5) that remains fixed across evaluations. The Spearman rank correlation [36] of models’ overall scores under different counterpart backbones is 0.72 in shopping and 0.68 in LGDs.

Main Quantitative Results: Dynamic Social Intelligence is Multi-dimensional

As Table 4 and Table 5 show, model performance fluctuates across both dimensions and scenarios, highlighting the structurally non-uniform nature of embodied social intelligence.

ModelKinesic↑Proxemic↑Affect↑Intent↑Rapport↑Persuasion↑†Attainment↑Depth↑
Gemini-3-Pro80.43%86.60%63.67%80.92%7.8512.7743.75%3.43
Gemini-3-Flash73.80%<u>78.95%</u>45.69%74.16%7.7212.0746.88%3.13
Claude-Opus-4.675.27%70.77%40.63%<u>76.66%</u><u>8.09</u><u>14.46</u>62.50%2.45
Claude-Sonnet-4.6<u>77.57%</u>71.69%55.38%68.77%8.1013.58<u>56.25%</u>3.44
GPT-571.00%78.00%41.76%69.22%7.77<u>14.33</u>53.13%3.06
GPT-5-Chat<u>77.29%</u>76.85%54.88%71.66%7.3310.4934.38%2.82
o360.08%75.65%43.03%<u>76.98%</u>7.7317.73<u>56.25%</u><u>3.28</u>
GLM-4.6V71.96%<u>78.37%</u>55.80%74.58%6.758.9134.38%1.82
Qwen3-VL-235B-A22B-Thinking65.94%64.33%48.47%60.15%6.078.0315.63%2.20
Qwen3-VL-235B-A22B-Instruct62.80%62.38%42.65%67.03%6.5410.6237.20%1.81
Qwen3-VL-32B-Thinking71.37%74.17%<u>63.57%</u>74.24%6.357.4021.88%2.57
Qwen3-VL-32B-Instruct63.92%68.98%<u>56.38%</u>64.67%6.278.0921.88%1.57

Table 4. Evaluation results on the ALIVE-Bench Shopping scenario. The best results are highlighted in bold, and the second and third results are underlined.

ModelKinesic↑Proxemic↑Affect↑Intent↑Rapport↑Persuasion↑†Attainment↑Depth↑
Gemini-3-Pro68.15%53.27%52.14%68.93%7.41<u>6.80</u>64.00%<u>2.25</u>
Gemini-3-Flash<u>70.19%</u><u>49.13%</u>46.20%66.76%<u>6.64</u>6.5373.23%1.88
Claude-Opus-4.650.61%<u>52.22%</u><u>49.96%</u>70.49%<u>6.95</u>6.70<u>67.71%</u>2.13
Claude-Sonnet-4.641.09%33.25%43.25%59.76%6.606.1160.96%1.81
GPT-564.47%47.52%40.36%<u>69.91%</u>4.976.94<u>71.60%</u>3.67
GPT-5-Chat73.10%47.65%43.78%68.36%6.206.1756.76%1.47
o368.33%49.05%<u>48.78%</u><u>70.29%</u>6.31<u>6.74</u>60.71%<u>3.19</u>
GLM-4.6V<u>70.10%</u>42.95%37.76%64.22%5.665.6959.58%1.88
Qwen3-VL-235B-A22B-Thinking64.61%40.64%37.79%58.75%5.175.3554.08%1.13
Qwen3-VL-235B-A22B-Instruct60.50%45.04%47.93%61.41%6.276.4060.41%1.38
Qwen3-VL-32B-Thinking69.72%42.87%43.37%60.31%6.315.8763.42%1.31
Qwen3-VL-32B-Instruct59.04%<u>49.01%</u>41.13%59.44%5.525.6554.85%1.50

Table 5. Evaluation results on the ALIVE-Bench Leaderless Group Discussion (LGD) scenario. The best results are highlighted in bold, and the second and third results are underlined.

First, we observe a distinct decoupling of capabilities among frontier VLMs across the four hierarchical levels. For instance, in the Shopping scenario, Gemini-3-Pro achieves the highest scores across all Perception and Understanding metrics (Kinesic: 80.43%, Proxemic: 86.60%, Affect: 63.67%, and Intent: 80.92%). It also consistently leads in Affect Understanding and Rapport Building. Conversely, o3 and GPT-5 frequently dominate the Persuasive Efficacy and Outcome dimensions. In the Shopping scenario, Claude-Sonnet-4.6 achieves the highest Rapport (8.10) and Outcome Depth (3.44), while Claude-Opus-4.6 secures the highest Goal Attainment (62.50%). In LGDs, Gemini-3-Flash leads in Goal Attainment (73.23%).

More importantly, robust upstream perception does not guarantee downstream interactive success, revealing a “Perception-to-Outcome Bottleneck”. For instance, in the Shopping scenario, the Qwen3-VL series demonstrates acceptable vision capabilities (e.g., Qwen3-VL-235B-Thinking achieves a Kinesic score of 65.94%). However, it suffers a severe drop-off in higher-level tasks: its Rapport Building and Persuasive Efficacy remain low (6.07 and 8.03, respectively), and its Goal Attainment plummets to 15.63%. Similar bottlenecks also appear in models like GPT-5-Chat and GLM-4.6V.

Finally, our results highlight that model proficiencies are highly scenario-dependent. The LGD scenario poses a much stricter test for micro-level social perception, which is starkly reflected in gaze target identification. Models such as Claude-Sonnet-4.6 experience a sharp performance degradation in this dimension, with accuracy

dropping to 33.25%, exposing vulnerabilities in tracking highly dynamic and subtle visual cues within complex environments.

Failure Modes in Different Interaction Scenarios

We further inspect representative failure modes to identify the concrete mechanisms behind the gap arising in quantitative patterns.

Breakdown from Perception and Understanding to Interaction Strategy. A significant gap often exists when models can recognize user states well, yet fail to convert it into socially appropriate action. In the Shopping scenario, while being able to identify customer emotions, Qwen3-VL-32B-Instruct often produces strategically inappropriate responses. As illustrated in Figure 3, despite correctly inferring a customer’s sadness and preference for practical clothing, the model’s understanding remains purely descriptive rather than policy-shaping, leading to a subsequent misaligned recommendation which escalated frustration.

Figure 3. Understanding-to-interaction breakdown in Shopping. Despite accurately inferring the customer’s emotions, the model generates a strategically misaligned action.

Difficulty in Fine-Grained Gaze Perception. While gaze crucially indicates addressees and turn-taking in LGDs, models frequently fail at these fine-grained visual judgments. For instance, Figure 1(b) shows Claude-Sonnet-4.6 incorrectly attributing a participant-to-participant gaze exchange to itself. Such egocentric over-attribution critically distorts downstream reasoning regarding stance alignment, highlighting the models’ struggle with visually subtle but socially decisive cues.

Coordination Failures in Multi-Party Interaction. A third failure mode emerges when social interaction becomes genuinely multi-party. As shown in Figure 4, several frontier VLMs experience clear declines in both conversation rate and purchased item count when the Shopping scenario shifts from a single-customer setting to episodes with three asynchronously entering customers. For instance, an agent might wait idly outside a fitting room while a customer changes, without reallocating attention to others, causing unattended customers to leave. The failure is strategic rather than linguistic: the model remains polite, but behaves like a serial responder instead of an adaptive coordinator.

Bar chart comparing purchase rate and average items purchased for single-customer and three-customer settings

Figure 4. Performance degradation in multi-party shopping scenarios. The comparison of interaction outcomes between single-customer and three-customer settings.

Emergent Social Strategies of Frontier VLMs.

Beyond monolithic metric rankings, ALIVE-BENCH reveals that frontier VLMs exhibit recurrent, distinguishable interaction patterns, uniquely balancing interpersonal rapport and task efficiency.

Gemini-3-Pro: Socially Attentive Strategy. Gemini-3-Pro excels in upstream social sensitivity, topping all perception and understanding metrics in the Shopping scenario. It effectively decodes embodied states to sustain positive social engagement, though its goal attainment remains moderate. As shown in Figure 5 (top), it provides adaptive guidance grounded in the customer’s natural exploration trajectory.

Figure 5. Example case of distinct social strategies in the Shopping scenario. Interacting with the same customer, Gemini-3-Pro adopts an adaptive and low-pressure approach (top), while GPT-5 prefers informative recommendation (bottom).

GPT-5: Goal-Oriented Strategy. Conversely, GPT-5 prioritizes downstream task closure. It achieves top-tier Persuasive Efficacy and Goal Attainment across scenarios, despite exhibiting notably weaker affective inference and rapport. As illustrated in Figure 5 (bottom), it relentlessly structures interactions toward efficient closure, adopting a decisive, directive approach that explicitly trades interpersonal accommodation for coordination efficiency.

Claude-Opus-4.6: Logic-Driven Strategy. Claude-Opus-4.6 distinguishes itself through credibility and structured guidance. It consistently leads in cognitive intent inference while simultaneously securing the highest rapport and goal attainment in Shopping. It earns cooperation by swiftly identifying practical concerns and responding with comprehensive, logical reasoning, facilitating reliable conversions through structured engagement.

Counterpart Preference Heterogeneity Across Decision Styles. Counterpart feedback reveals clear profile-dependent preferences across scenarios. In Shopping (Table 6), impulsive and choice-overloaded customers favor the decisive styles of o3 and GPT-5, whereas quality- and price-conscious customers prefer Claude-Opus-4.6’s reasoned justifications. Similarly in LGD, analytical participants rate Claude-Opus-4.6 highest, while Gemini-3-Pro’s adaptive style dominates among collaborative and conceptual participants. These alignments provide mutual validation, showing that ALIVE-World profiles encode meaningful psychological differences while corroborating the emergent VLM strategies.

ModelNovelty-seekingImpulsiveChoice-overloadedQuality-perfectionistPrice-value conscious
GPT-529.2423.833.767.485.20
o321.2828.7510.547.365.88
Gemini-3-Pro25.1113.055.272.414.61
Claude-Opus-4.614.6310.539.3314.7211.22

Table 6. The average purchase intention scores across different profile-conditioned decision styles. The highest scores for each customer profile are highlighted in bold.

Conclusion

In this work, we introduce ALIVE-WORLD, an embodied simulation platform populated by counterpart agents with evolving internal states, designed to integrate realistic social dynamics with embodied physical realism. Building upon this environment, we develop ALIVE-BENCH to transform the assessment of multi-dimensional social intelligence into a hierarchical quantitative framework. Our extensive evaluations across diverse interaction scenarios reveal critical gaps in dynamic social competencies of current VLMs, while uncovering the emergence of distinct social interaction patterns among frontier models. While ALIVE-WORLD currently focuses on two manually designed scenarios to preserve interactional quality, this still limits the coverage of broader social interactions. Future work will explore automated scenario generation to scale diversity. Ultimately, as ALIVE-WORLD provides fine-grained feedback on VLM behavioral mechanisms, future research can leverage this interactive environment to optimize the dynamic social intelligence of VLM agents prior to their deployment in real-world applications.

Acknowledgments

This work is supported by the National Natural Science Foundation of China No. 62425604. It is also supported by National Natural Science Foundation of China No. 62502256 and Beijing Natural Science Foundation (L257006), High Performance Computing Center, Tsinghua University. We also thank YooLee AI for their collaboration and support.

References

A Comparison with Representative Prior Work

To further position ALIVE-Bench with respect to representative prior work, we provide a qualitative comparison across several key properties emphasized in our related-work discussion. As shown in Table 7, prior studies typically cover only partial aspects of dynamic social intelligence, such as social reasoning, embodied perception, or interactive environments. In contrast, ALIVE-Bench unifies embodied 3D interaction, egocentric visual perception, dynamic multi-agent interaction, social intelligence evaluation, fine-grained nonverbal social cues, and human-like evolving social dynamics within a single framework.

PropertyToMBench [7]Generative Agents [28]SOTOPIA [55]MoMentS [42]SoMi-ToM [11]EmbodiedBench [49]SimWorld [50]VirtualCommunity [54]ALIVE-BENCH (Ours)
Embodied 3D Environment✗✗✗✗✓✓✓✓✓
Egocentric Visual Perception✗✗✗✗✓✓✓✓✓
Dynamic Real-time Multi-agent Interaction✗✓✓✗✗✗✓✓✓
Systematic Social Intelligence Evaluation✓✗✓✓✓✗✗✗✓
Fine-Grained Nonverbal Social Cues✗✗✗✗✗✗✗✗✓
Human-like Internal Social Dynamics✗✗✗✗✗✗✗✗✓

Table 7. Comparison between ALIVE-BENCH and representative prior work along key properties related to the evaluation of dynamic social intelligence in embodied egocentric environments.

Details of ALIVE-World

Environment Construction

ALIVE-World is implemented in Unity and rendered as a 3D embodied environment from an egocentric first-person perspective. The static scene layouts and object assets are built using resources from the Unity Asset Store3 and TurboSquid4. For human characters, we create avatar bases with VRoid Studio5, and further enrich them with clothing, facial components, and hairstyle assets obtained from BOOTH6. Character motion is driven by animation assets from Mixamo7. Together, these resources allow us to construct visually coherent interactive scenes in which agents can navigate, observe, gesture, gaze, and speak under continuous multi-agent interaction.

Scenario Instantiation

We instantiate ALIVE-World through two complementary scenarios: Clothing Store Shopping and Leaderless Group Discussion (LGD). The two scenarios are deliberately designed to differ in role symmetry, interaction scale, and strategic demands, thereby covering distinct aspects of dynamic social intelligence.

Clothing Store Shopping. The shopping scenario is instantiated as a boutique-style clothing store in which the evaluated VLM agent acts as a seller. Each product in the store is represented by four structured fields: (1) shelf zone, (2) item name, (3) price, and (4) textual detail. The shelf zone specifies the coarse physical area in which the item is displayed, while the textual detail provides concise semantic information that can be used by the seller agent during recommendation and persuasion. To ensure consistency across evaluation episodes, the product catalog remains fixed throughout the benchmark. For readability in the appendix, Table 8 lists the store inventory by shelf zone.

Shelf ZoneProduct List
Women’s TopsGreen Hooded Sweatshirt ($29), White Sailor-Collar Top ($19), Beige Floral Shirt ($67), White Long-Sleeve Shirt ($54), Pink Short-Sleeve T-shirt ($13)
Women’s BottomsWhite Slim-Fit Trousers ($51), Gray Straight-Leg Trousers ($35), Blue Distressed Shorts ($14), Beige Plaid Pleated Skirt ($19), Dark Blue Skinny Jeans ($70)
Women’s DressesPink Chiffon Dress ($33), Black Vintage Dress ($42), White Sleeveless Dress ($945), Black-and-Gold Evening Gown ($1860), Blue-and-White Paneled Dress ($52)
Men’s TopsKhaki Trench Coat ($475), White Shirt ($52), Black Suit Jacket ($1380), Wine-Red Hoodie ($58), White Graphic T-shirt ($19)
Men’s BottomsDark Brown Dress Pants ($510), Black Straight-Leg Dress Pants ($155), Khaki Casual Trousers ($43), Dark Blue Jeans ($77), Black Side-Stripe Track Pants ($27)
AccessoriesBlue Sneakers ($43), Red High Heels ($1090), Black Leather Shoes ($115), White Casual Shoes ($27), White Bucket Hat ($16), White Formal Hat ($33), Brown Newsboy Cap ($10), Camouflage Bucket Hat ($14), Black Handbag ($22950), White Tote Bag ($990), Red Backpack ($23), Gray Backpack ($26), Polka-Dot Handbag ($7)

Table 8. Product catalog used in the Clothing Store Shopping scenario.

Leaderless Group Discussion. The LGD scenario is instantiated as a multi-party committee discussion in which the evaluated agent participates as one peer among several discussants. To cover different forms of group reasoning and social coordination, the task bank contains three types of discussion problems: multiple-choice prioritization, open-ended dilemmas, and resource-allocation planning. To ensure fairness and comparability across evaluation episodes, each counterpart agent is assigned an initial viewpoint before interaction begins, and this initial viewpoint is kept fixed whenever that counterpart participates in the same task. This design prevents variation in the counterparts’ starting positions from confounding cross-model comparison. Table 9 presents shortened versions for readability, while preserving the original decision structure of each problem.

Task TypeRepresentative Prompt
Multiple-choiceHotel profit diagnosis. A resort hotel failed to meet its fourth-quarter profit target. Ten possible management problems have been identified, including weak performance incentives, poor room hygiene, inaccurate reporting, inventory mismatch, and internal disorder. As the newly appointed general manager, select the three most important causes. Public-welfare entrepreneurship funding. A city-level competition has shortlisted ten social-enterprise proposals, such as a vegetarian restaurant, a healthcare business, an eco-laundry service, removable wallpaper, elderly clothing retail, environmentally friendly cleaning, an aromatherapy shop, battery rental, eco-tourism guiding, and a cake shop. As a reviewer, choose the three projects that should receive funding.
DilemmaRule compliance vs. unconventional innovation. In enterprise management, which is more important: strictly following institutional rules, or encouraging innovation that may challenge existing rules? Ability vs. attitude. In employee recruitment, which matters more: professional competence, or attitude and cooperative spirit?
Resource AllocationEmergency departmental budget. A unit has only a limited budget and must divide it across four urgent needs: repairing the office telephone system, renovating a conference room for an important superior-led meeting, covering an employee’s large medical expense, and providing holiday benefits. Give a concrete allocation plan. Company bonus allocation. A company has a fixed bonus pool to distribute across four departments—engineering, marketing, design, and general administration—but equal division is not allowed. Propose a reasonable allocation strategy.

Table 9. Representative task types and shortened example prompts used in the Leaderless Group Discussion scenario.

Action Space

Beyond the representative subset shown in the main paper, ALIVE-World uses a richer action inventory to support socially expressive and scenario-specific interactions. Table 10 lists the full action space used in our experiments, including scenario-specific decision actions, richer gesture primitives, and multiple speech modes with different spatial scopes.

DimensionActionParametersDurationNote
NavigationMOVETarget, Distance, Direction10sMove relative to an object or another agent.
NavigationCHANGETop, Bottom, Dress60sUsed by customer agents when changing clothes in the fitting room.
NavigationLEAVENone10sExit the current scene.
GesturePOINTTargetInstantPoint toward an object or another agent.
GestureRAISE_HANDNoneInstantTypically used to request the floor in LGD.
GestureTHUMB_UPTargetInstantExpress approval or encouragement.
GestureCLAPNoneInstantExpress approval or emphasis.
GestureHANDS_OPEN_TABLENoneInstantUsed to convey openness, receptiveness, emotional stability, or cooperative tendency.
GestureHANDS_RELAX_ON_LAPNoneInstantUsed to convey passive listening, silent observation, or calm narration.
GestureHANDS_CROSS_ON_LAPNoneInstantUsed to convey defensiveness, tension, or psychological guardedness.
GazeLOOK_ATTarget / ForwardInstantRedirect gaze toward a specific object, agent, or forward direction.
SpeechSPEAKTarget Agent, Content10s per 150 charactersAudible only when the speaker and the target agent are in sufficiently close spatial proximity.
SpeechCALLTarget Agent, Content10sA directed call that is globally audible regardless of distance.
SpeechBROADCASTContent10sAudible to all agents in the scene regardless of distance.
SpeechINTERRUPTTarget Agent, Content10s per 150 charactersUsed to cut into another agent’s ongoing speech and seize the conversational floor.
SpeechLISTEN_SILENTLYNone10sExplicitly remain silent and observe the ongoing interaction.
DecisionPURCHASEItem_NameInstantFinalize the purchase of a target item in Shopping.
DecisionPROPOSE_CONCLUSIONConclusionInstantPut forward a final group conclusion in LGD.

Table 10. Full action inventory in ALIVE-World. Instant actions are executed as state changes that persist until overwritten unless otherwise noted.

Incremental rendering of speech. A speech action is generated as one complete utterance in a single model call, but it is not injected into other agents’ textual observations all at once. Instead, once the speech action is initiated, its content is progressively revealed to other agents as virtual time advances. The default rendering speed follows the duration rule in Table 10, i.e., approximately 10 seconds of virtual time per 150 characters. At each intermediate virtual time step, only the currently revealed prefix of the utterance is appended to the listeners’ textual observations. If the speaker is interrupted before finishing, the utterance is truncated at the currently revealed position, and only that revealed prefix is preserved in the interaction history. This mechanism allows the environment to model partial utterances, conversational cut-ins, and delayed reactions in a temporally grounded way.

Simultaneous speech and floor management. Multiple agents may decide to start speaking at the same simulation tick. To resolve this conflict while preserving a notion of conversational floor, we maintain a priority queue over agents. At each tick, among the agents that decide to speak, the one that appears earliest in the queue is rendered as the active speaker. Once its utterance finishes or is interrupted, it is removed from its current position and placed at the tail of the queue. This queue-based mechanism provides a simple but effective approximation of turn competition, enabling naturally occurring social phenomena such as interruption, delayed response, and proactive floor-taking.

Virtual-Time Execution Mechanism

ALIVE-World uses a virtual-time simulation mechanism that is explicitly decoupled from real-world model inference latency. All action durations reported in Table 10 are measured in environment-side virtual time rather than wall-clock execution time. This design ensures that benchmark results are not confounded by differences in API speed or system-level runtime overhead across models.

The environment advances through discrete simulation ticks. Once an agent finishes its current action in virtual time, it re-obtains the latest multimodal observation at the current virtual timestamp and starts a new reasoning cycle to generate the next action. As a result, different agents may reason and act asynchronously, while the environment remains globally synchronized by the shared virtual clock. This mechanism supports temporally overlapping actions and enables more realistic interaction phenomena, including delayed replies, preemptive interruption, and concurrent multi-agent engagement.

In all experiments, the maximum virtual duration of one episode is capped at Tmax⁡=15T_{\max}=15 minutes.

Shopping scenario. In the Clothing Store Shopping scenario, an episode starts when the first customer enters the store. It terminates when either (1) all customers in that episode have executed the LEAVE action, or (2) the virtual time reaches Tmax⁡T_{\max}. Under this design, the seller agent must dynamically manage service order and persuasive interaction within a bounded but asynchronous temporal horizon.

LGD scenario. In the Leaderless Group Discussion scenario, an episode starts when the first participant begins speaking. It terminates when either (1) one participant executes PROPOSE_CONCLUSION and all remaining participants explicitly agree with the proposed conclusion, or (2) the virtual time reaches Tmax⁡T_{\max}. This setting allows the discussion to evolve through multiple rounds of speaking, interruption, and stance adjustment before reaching either consensus or timeout.

Details of Counterpart Agent Architecture

Profile Schema and Instantiation

Each counterpart agent in ALIVE-World is instantiated from a structured profile prior. This profile serves as a persistent identity anchor throughout an episode and conditions both the intuitive-affective and logical-cognitive pathways of the counterpart agent. Rather than generating anonymous or interchangeable interactants, this design enables each counterpart to exhibit relatively stable preferences, behavioral tendencies, and social reactions under dynamic interaction.

Concretely, each profile contains four top-level fields: demographics, psychographics, decision_style, and current_lifestyle. The demographics field specifies relatively stable background attributes of the counterpart, including name, gender, age, occupation, domain expertise, income level, and family status. The psychographics field characterizes broader psychological tendencies and value orientations. The decision_style field determines the counterpart’s characteristic way of processing options and making judgments in a given scenario. Finally, current_lifestyle captures the agent’s present-life context, which provides additional situational grounding for its needs, priorities, and responses during interaction.

In total, we construct a fixed pool of 30 profiles for benchmark instantiation. Table 11 summarizes the demographic composition of the full profile set. Figure 6 illustrates an example of how a counterpart agent is instantiated from a structured profile prior, together with the corresponding avatar.

Example of a counterpart agent’s structured profile prior and instantiated avatar in ALIVE-World

Figure 6. Example of a counterpart agent’s structured profile prior and instantiated avatar in ALIVE-World.

CategoryGroupCount
GenderFemale15
Male15
Age20–245
25–296
30–347
35–392
40–442
45–493
50–541
55–592
60–642

Table 11. Demographic composition of the 30 counterpart profiles.

Decision styles in the Shopping scenario. For the Clothing Store Shopping scenario, the decision_style field captures differences in how customers evaluate products, trade off quality and price, and respond to recommendations [37]. We instantiate six consumer decision styles, with 5 profiles assigned to each style, yielding a balanced distribution across the 30-profile pool.

  • Perfectionist / High-quality conscious consumer (5 profiles): shops carefully and systematically, and seeks the highest possible product quality.

  • Price-value conscious consumer (5 profiles): pays close attention to price and discounts, and focuses on obtaining the best value for money.

  • Impulsive consumer (5 profiles): tends to buy on the spur of the moment and shows relatively little concern for careful comparison or optimal bargains.

  • Recreational / Hedonistic consumer (5 profiles): views shopping as recreation and entertainment, and derives enjoyment from the shopping process itself.

  • Novelty-seeking consumer (5 profiles): enjoys exploring new products and is motivated by variety, excitement, and discovery.

  • Choice-overloaded consumer (5 profiles): is easily overwhelmed by excessive options and may become hesitant or indecisive when making selections.

These decision styles are particularly useful in the shopping scenario because they induce different reactions to the same seller behavior. For example, highly impulsive customers may respond favorably to decisive recommendations, whereas choice-overloaded customers may require stronger guidance and simplification, and quality-conscious customers are more sensitive to detailed product arguments.

Decision styles in the LGD scenario. For the Leaderless Group Discussion scenario, the decision_style field instead characterizes how discussants process ambiguity, respond to alternatives, and interact with other group members in collective reasoning [30]. We use four discussion-oriented decision styles, distributed as 8 Analytical, 7 Directive, 8 Behavioral, and 7 Conceptual profiles.

  • Analytical style (8 profiles): shows relatively high tolerance for ambiguity, seeks more information, considers multiple alternatives, and tends to reason carefully while adapting to new situations.

  • Directive style (7 profiles): has relatively low tolerance for ambiguity, prefers efficiency and rationality, makes decisions quickly, and often emphasizes short-term results with limited information gathering.

  • Behavioral style (8 profiles): works well with others, values team achievement, is receptive to suggestions, relies heavily on discussion and meetings, and tends to avoid conflict while seeking acceptance.

  • Conceptual style (7 profiles): adopts a broad perspective, considers many alternatives, focuses on long-term implications, and is especially inclined toward creative solutions.

This scenario-specific design allows the same environment to support two different forms of social heterogeneity: consumer preference diversity in Shopping, and reasoning-and-coordination diversity in LGD. As a result, counterpart agents do not merely differ in surface profile descriptions, but also in how they interpret ongoing interaction, update their internal states, and choose subsequent actions.

Intuitive-Affective Pathway

The intuitive-affective pathway models the counterpart agent’s fast, heuristic, and socially instinctive reactions before deliberate reasoning takes place. Given the current multimodal observation, previous affective state, and profile prior, this pathway performs a coarse appraisal of the surrounding environment and recent interactional atmosphere, and updates the agent’s transient emotion and interpersonal attitude. Rather than analyzing detailed product semantics or deep argumentative logic, it emphasizes rapid human-like reactions to visually and socially salient cues.

The specific cues differ across scenarios. In the Shopping scenario, the visual appraisal mainly focuses on three aspects: (1) proxemic intuition, including whether the seller is visible, whether the distance is comfortable, and whether the seller’s body orientation suggests attention, pressure, or neglect; (2) store atmosphere, including the overall tone, tidiness, and stylistic impression of the store; and (3) rough self-concept matching, namely whether the visible clothing styles and colors intuitively feel aligned with the customer’s profile, taste, and current lifestyle. The interactional appraisal focuses on whether the seller feels boundary-respecting or pushy, whether the latest response feels genuinely responsive or perfunctory, and whether the overall conversational rhythm matches the customer’s personality and social expectations.

In the LGD scenario, the visual appraisal instead emphasizes socially directed signals in a multi-party discussion, such as whether other participants are looking at the agent, whether their gaze or gestures convey support, neglect, or pressure, and which kinds of visual cues are especially salient given the participant’s profile. The interactional appraisal centers on turn-taking and paralinguistic atmosphere: whether someone cuts in abruptly, dominates the floor, sounds dismissive or sincere, and whether the agent intuitively feels “received” or “pushed back” when expressing a view.

In practice, we implement this pathway with scenario-specific prompts that instruct the counterpart agent to output structured intuitive perceptions, emotion changes, and interpersonal attitudes in JSON format. Figures 10 and 11 provide the full prompts used in the Shopping and LGD scenarios, respectively.

For embodied rendering, the updated emotion state is mapped to a small set of discrete facial expressions. In the current implementation, we use five expression categories: neutral, happy, surprised, sad, and angry. This discrete mapping allows the counterpart’s transient affective state to be visually reflected by the avatar while keeping the rendering interface stable.

Logical-Cognitive Pathway

The logical-cognitive pathway models the counterpart agent’s slower, more deliberate, and semantically grounded reasoning process. Compared with the intuitive-affective pathway, this stage places less emphasis on immediate social instinct and instead focuses on top-down visual inspection, semantic interpretation, logical evaluation, and task-oriented belief updating. Given the current observation, previous cognitive state, and profile prior, the pathway updates the agent’s rational understanding of the situation and its goal-related intent.

The specific cues again differ across scenarios. In the Shopping scenario, the visual analysis focuses on concrete product attributes that can be identified from first-person observation, such as category, rough style, visible design details, and nearby shelf context. These observations are evaluated against the customer’s profile, especially the current_lifestyle and decision_style, to determine whether the visible products satisfy hard constraints or practical needs. The interactional analysis then examines the seller’s latest response through deeper semantic processing: whether it provides substantive product information, whether it directly addresses the customer’s concerns, and whether it compensates for information that cannot be inferred from visual inspection alone. Based on this evidence, the pathway updates the customer’s product cognition and purchase intent toward visible or previously discussed items.

In the LGD scenario, the visual analysis focuses on non-verbal power dynamics in the discussion, including who is visually central, where participants are looking, what gestures they are making, and what their current facial expressions imply about stance, confidence, or openness. The interactional analysis then evaluates the latest contribution in terms of task advancement, argumentative completeness, evidential support, and logical usefulness for solving the discussion problem. This stage assesses whether others’ viewpoints deserve acceptance from a rational, task-oriented perspective, and whether the agent should revise, defend, or strategically integrate its own view.

In practice, we implement this pathway with scenario-specific prompts that instruct the counterpart agent to output structured analytical perceptions and updated cognitive variables in JSON format. Figures 12 and 13 provide the full prompts used in the Shopping and LGD scenarios, respectively.

Overall, the logical-cognitive pathway produces structured updates to the counterpart agent’s task-relevant belief state and goal-oriented intent. In Shopping, this takes the form of updated product cognition and item-level purchase intention; in LGD, it takes the form of proposal-level acceptance intent and a revised current answer to the discussion topic.

Action Generation

After the intuitive-affective pathway and the logical-cognitive pathway are executed, the counterpart agent enters the action-generation stage. At this stage, the agent must synthesize the outputs of the two pathways into a single executable action. This design ensures that the action remains consistent with the agent’s current emotional tendency while also being grounded in task-oriented analysis.

In practice, we implement this stage with scenario-specific prompts that expose the outputs of both pathways and require the model to produce a structured action decision in JSON format. Figures 14 and 15 present the full action-generation prompts used in the Shopping and LGD scenarios, respectively.

In both scenarios, the action-generation stage outputs two fields: an action_analysis string that explains why the actions are selected under the current affective and cognitive state, and an action list that specifies the executable actions in the environment. This final step closes the loop from observation, to latent-state updating, to embodied behavior.

Details of ALIVE-Bench Evaluation

Evaluated Agent Prompt

To evaluate social intelligence in a traceable and fine-grained manner, the evaluated VLM agent is prompted to explicitly output structured intermediate reasoning before executing actions. The prompt is scenario-specific, but follows the same general principle: the agent first performs visual perception and theory-of-mind inference over the current interaction context, then derives an action strategy, and finally outputs executable actions in a structured JSON format.

In the Shopping scenario, the evaluated agent acts as a seller who must simultaneously serve multiple customers and maximize sales through spatially grounded recommendation and interaction management. In the LGD scenario, the evaluated agent acts as a discussant who must balance consensus-building with the strategic advancement of its own viewpoint. Figures 16 and 17 show the full prompts used for the evaluated agent in the two scenarios.

Metrics Calculation

This appendix provides additional details for the Understanding and Outcome levels in ALIVE-Bench. We further clarify the partial-credit mechanism used in the Understanding metrics and the task-specific computation rules used in the Outcome metrics.

Understanding metrics. The Understanding level evaluates whether the evaluated agent can infer the counterparts’ latent states from multimodal evidence. We distinguish between affective state inference and cognitive intent inference, and assign graded credit rather than enforcing strict exact-match scoring in all cases.

For affective state inference, the evaluated agent receives full credit if the predicted affective state exactly matches the counterpart’s modeled affective state. If the exact label is incorrect but the predicted emotion has the correct affective polarity (e.g., matching the positive/negative tendency under our predefined grouping), we assign half credit. Otherwise, the prediction receives zero credit. Formally, for predicted emotion e^\hat{e} and reference emotion ee, we define

saffect(E^,E)={1,if E^=E,0.5,if E^≠E but has the correct polarity,0,otherwise.s_{\mathrm{affect}}(\hat{E}, E) = \begin{cases} 1, & \text{if } \hat{E}=E,\\ 0.5, & \text{if } \hat{E}\ne E\text{ but has the correct polarity},\\ 0, & \text{otherwise.} \end{cases}

The final affective-state understanding score is obtained by averaging this quantity across time steps and counterparts.

For cognitive intent inference, we likewise use a partial-credit scheme. A prediction receives full credit if the evaluated agent correctly infers the relevant cognitive state in full. It receives half credit if the agent only captures part of the intended state, and zero otherwise. In the Shopping scenario, this includes cases where the agent correctly identifies only part of a customer’s purchase tendency or item-level interest. In the LGD scenario, this includes cases where the agent correctly captures only part of another participant’s stance tendency or core claim. Formally, for predicted cognitive state c^\hat{c} and reference state cc, we define

scog(B^,B)={1,if the inference is fully correct,0.5,if the inference is partially correct,0,otherwise.s_{\mathrm{cog}}(\hat{B}, B) = \begin{cases} 1, & \text{if the inference is fully correct},\\ 0.5, & \text{if the inference is partially correct},\\ 0, & \text{otherwise.} \end{cases}

The final cognitive-intent understanding score is again computed by averaging over time steps and counterparts.

Outcome metrics in the Shopping scenario. The Shopping scenario measures outcome from both a binary success perspective and a depth-of-conversion perspective.

Goal Attainment is defined as the fraction of customers who eventually make a purchase:

Mgoalshop=NpurchaseNenterM_{\mathrm{goal}}^{\mathrm{shop}} = \frac{N_{\mathrm{purchase}}}{N_{\mathrm{enter}}}

where NenterN_{\mathrm{enter}} is the total number of customers who enter the store, and NpurchaseN_{\mathrm{purchase}} is the number of customers who execute a PURCHASE action.

Outcome Depth measures how many items are purchased, conditioned on successful conversion:

Mdepthshop=NitemsNpurchaseM_{\mathrm{depth}}^{\mathrm{shop}} = \frac{N_{\mathrm{items}}}{N_{\mathrm{purchase}}}

where NitemsN_{\mathrm{items}} is the total number of purchased clothing items. If no customer makes a purchase, we set Mdepthshop=0M_{\mathrm{depth}}^{\mathrm{shop}}=0.

Outcome metrics in the LGD scenario. In the LGD scenario, Goal Attainment measures the survival of the evaluated agent’s initial proposal in the final group conclusion. The scoring rule depends on the task type.

Let qq denote the task type, ainita^{\mathrm{init}} the evaluated agent’s initial answer, and afinala^{\mathrm{final}} the final group conclusion. We define

Mgoallgd=g(q,ainit,afinal),M_{\mathrm{goal}}^{\mathrm{lgd}} = g(q, a^{\mathrm{init}}, a^{\mathrm{final}}),

where g(⋅)g(\cdot) is task-specific:

Multiple-choice. If the task requires selecting a fixed number of options, we compute the overlap ratio between the evaluated agent’s initial answer and the final group answer:

g=∣ainit∩afinal∣Lchoiceg = \frac{\left|a^{\mathrm{init}}\cap a^{\mathrm{final}}\right|}{L_{\mathrm{choice}}}

where LchoiceL_{\mathrm{choice}} is the required number of selected options in that task.

Human-Evaluation Scoring Criteria

In the human evaluation of counterpart agents, annotators rate each full interaction record on a 5-point Likert scale along four dimensions: Profile Consistency, Behavioral Naturalness, Social Cue Sensitivity, and Affect-Cognition Coherence. Annotators are instructed to judge the entire interaction trajectory as a whole, while using the following definitions and score anchors as guidance.

1. Profile Consistency

Definition. This dimension evaluates whether the customer’s behavior, focus of attention, and purchase decisions remain consistent with the assigned profile. The key question is whether the customer is truly “acting as itself” according to its profile, rather than drifting into a different character.

What to attend to. Compare the observed behavior against the profile’s decision style, shopping motivation, and personality traits. Pay attention to whether the customer makes choices that contradict its own preferences, priorities, or characteristic way of speaking and deciding.

Score anchors.

  • 1 – Severely inconsistent / out-of-character. The behavior clearly contradicts the profile. For example, the customer focuses on factors that do not match its shopping motivation, makes purchase decisions at moments that do not fit its decision style, or speaks in a manner inconsistent with its assigned personality.

  • 3 – Broadly consistent. The behavior generally fits the assigned profile, but some decisions or speaking patterns appear generic, mechanical, or insufficiently individualized.

  • 5 – Fully consistent. The counterpart vividly embodies the assigned profile. Its moments of interest, hesitation, questioning, and purchase all strongly align with the lifestyle, decision style, and personality described in the profile.

Behavioral Naturalness

Definition. This dimension evaluates whether the customer’s physical behavior and shopping rhythm resemble those of a real human shopper.

What to attend to. Focus on the ordering, pacing, and overall plausibility of actions, including movement, browsing, asking questions, trying on clothes, waiting, comparing, and purchasing. Consider whether the trajectory feels like a natural shopping process rather than a rigid scripted procedure.

Score anchors.

  • 1 – Mechanical / unnatural. The behavior appears highly robotic, odd, or implausible. The customer moves or acts in ways that feel stiff, abrupt, or inconsistent with ordinary human shopping behavior.

  • 3 – Reasonable but procedural. The behavior is broadly acceptable, but it feels overly checklist-like or rigid, lacking the spontaneity, hesitation, or gradual realization often seen in real shopping.

  • 5 – Highly natural and human-like. The behavior is vivid and realistic, with smooth and coherent transitions among browsing, trying on, comparing, matching, and purchasing.

Social Cue Sensitivity

Definition. This dimension evaluates whether the customer responds to the seller as a normal social interactant would, and whether it shows appropriate sensitivity to both positive social value and social intrusion.

What to attend to. Examine whether the customer meaningfully reacts when the seller behaves notably well or poorly. Examples include the seller being overly pushy, standing too close, behaving abnormally, failing to answer the customer’s question, or instead offering highly appropriate and appealing recommendations.

Score anchors.

  • 1 – Dull / overly compliant. The customer shows little or no reaction to highly oppressive selling, abnormal seller behavior, or exceptionally positive engagement. The seller’s behavior changes, but the customer’s response remains almost the same.

  • 3 – Ordinary responsiveness. The customer can understand the seller and respond appropriately at a basic level, but the emotional and behavioral feedback is relatively coarse.

  • 5 – Highly sensitive and graded feedback. The customer reacts like a real person. It shows appropriate increases in attraction or purchase desire when positively engaged, and also displays defensiveness, dislike, interruption, withdrawal, or leaving behavior when socially pressured or offended.

Affect-Cognition Coherence

Definition. This dimension evaluates whether the customer’s outward behavior is coherent with its immediately preceding internal state, including both its profile-based predisposition and the emotion induced by recent interaction.

What to attend to. Judge whether the transition from internal state to external action is psychologically and logically consistent. Pay particular attention to whether recent negative or positive experiences are naturally reflected in the next behavioral choice.

Score anchors.

  • 1 – Clear disconnect between state and behavior. The customer’s action contradicts its immediately preceding state. For example, it appears dissatisfied, offended, or unconvinced, yet immediately continues to interact warmly, try on clothes, or purchase; or it appears highly engaged and interested, yet suddenly leaves without sufficient cause.

  • 3 – Surface-level coherence. The behavior is not obviously contradictory, but the transition from emotion and cognition to action feels somewhat abrupt, stitched together, or insufficiently grounded.

  • 5 – Strong affect-cognition coherence. The behavior is tightly aligned with the internal state, and changes in action follow naturally from prior emotion, cognition, and interaction experience.

Additional Analyses of Experimental Results

Affective Feedback under Different Seller Models

To further examine how different evaluated seller agents shape customers’ immediate affective responses, we summarize the distribution of customer emotional states in the Shopping scenario and aggregate them into three coarse categories: positive, neutral, and negative. The resulting distributions are shown in Figure 7.

Figure 7. Customer affective-state distribution under different evaluated seller models in the Shopping scenario, aggregated into positive, neutral, and negative categories.

Overall, all evaluated models produce predominantly positive or neutral customer states, while negative states remain a relatively small proportion. However, the balance between these categories varies noticeably across models. Claude-Opus-4.6, Claude-Sonnet-4.6, and Gemini-3-Pro yield the highest proportion of positive states, with positive responses accounting for roughly two-thirds of all customer affective states. By contrast, GLM-4.6V and the Qwen3-VL series show substantially larger neutral proportions, indicating that customers more often remain affectively unshifted during interaction. Negative states are still limited in absolute magnitude for all models, but they appear more frequently for GLM-4.6V and the Qwen3-VL variants than for Gemini or Claude models.

These results provide an additional affective-level view of interaction differences across seller models. In particular, they are broadly consistent with the main observation that different evaluated agents exhibit distinct interaction tendencies rather than a single shared social strategy.

Style-Conditioned Customer Affective Heterogeneity

We further analyze customer affective-state distributions across different decision-style profiles in the Shopping scenario. Following the profile design of ALIVE-World, we group customer emotions into positive, neutral, and negative categories and report the aggregated distributions in Figure 8.

Figure 8. Customer affective-state distribution across different customer decision styles in the Shopping scenario, aggregated into positive, neutral, and negative categories.

A clear difference can be observed across customer types. Novelty-seeking and impulsive customers show the highest proportions of positive affective states, both exceeding three quarters of all observations. Recreational customers also display a clearly positive-skewed distribution. In contrast, price-value conscious, quality-perfectionist, and choice-overloaded customers exhibit much larger neutral proportions, suggesting that these customers are less easily shifted toward strong positive affect during interaction. Among them, quality-perfectionist and price-value conscious customers also show comparatively higher negative proportions than novelty-seeking or impulsive customers.

This pattern suggests that affective feedback in the Shopping scenario is meaningfully conditioned by customer profile rather than being uniform across counterparts. In this sense, the profile-conditioned counterpart design induces observable heterogeneity not only in final preferences, but also in intermediate emotional responses during interaction.

Decomposition of Functional Roles in LGD

To provide a more fine-grained view of Outcome Depth in the LGD scenario, we further break it down into the six functional roles defined in our evaluation protocol. Figure 9 shows the frequency with which each evaluated model exhibits these roles across episodes. This analysis complements the role-based formulation of LGD outcome depth.

Figure 9. Frequency of functional-role participation by different evaluated agents in the LGD scenario. Values are shown as percentages over evaluation episodes.

Several role preferences can be observed. Gemini-3-Pro shows a particularly high frequency on Process Setter, while GPT-5 is high on Criteria Formulator, Milestone Tracker, and Solution Synthesizer. o3 also exhibits relatively broad role coverage, with comparatively high values on Process Setter, Criteria Formulator, Milestone Tracker, Solution Synthesizer, and Final Reporter. Claude-Opus-4.6 and Claude-Sonnet-4.6 appear more concentrated on mid-discussion organizational roles such as Criteria Formulator and Milestone Tracker, while some smaller Qwen3-VL variants and GLM-4.6V show lower frequencies on early-stage agenda-setting roles but still participate non-trivially in roles such as Deadlock Breaker or Solution Synthesizer.

Overall, these results suggest that differences in LGD performance are not only reflected in the total number of roles undertaken, but also in the specific functional roles that different models tend to assume during discussion. This decomposition provides a more behaviorally interpretable view of how evaluated agents contribute to group interaction.

Style-Conditioned Interpersonal Attitude and Acceptance in LGD

We further examine how different LGD counterpart decision styles respond to evaluated agents along two complementary dimensions: interpersonal attitude (att) and acceptance intent (acc). Here, att reflects the discussants’ interpersonal favorability toward the evaluated agent, while acc reflects their willingness to accept the evaluated agent’s proposal from a task-oriented perspective. Table 12 reports the average scores for four representative frontier models.

ModelAnalytical attAnalytical accDirective attDirective accBehavioral attBehavioral accConceptual attConceptual acc
GPT-55.816.494.336.874.797.204.076.30
o36.816.145.675.706.757.105.067.00
Gemini-3-Pro6.596.287.286.067.647.047.666.81
Claude-Opus-4.66.975.945.735.797.177.127.137.20

Table 12. Average interpersonal attitude (att) and acceptance intent (acc) received by representative evaluated agents from different LGD counterpart decision styles. The best score in each column group is highlighted in bold.

Several style-dependent differences can be observed. For analytical discussants, Claude-Opus-4.6 receives the highest interpersonal attitude score, while GPT-5 obtains the highest acceptance score. For directive discussants, Gemini-3-Pro achieves the highest attitude score, whereas GPT-5 again shows the highest acceptance. For behavioral discussants, Gemini-3-Pro attains the highest interpersonal attitude, while GPT-5 remains slightly ahead on acceptance. For conceptual discussants, Gemini-3-Pro achieves the highest interpersonal attitude score, while Claude-Opus-4.6 obtains the highest acceptance score.

Overall, the table suggests that counterpart responses in LGD vary not only across evaluated models, but also across decision styles and response dimensions. In particular, the model receiving the highest interpersonal attitude is not always the same as the one receiving the highest acceptance, indicating that social favorability and proposal-level acceptance do not always move in parallel in multi-party discussion.

Demo Page for Qualitative Interaction Examples

To complement the quantitative results reported in the main paper and this appendix, we provide an anonymous demo page containing qualitative interaction examples: https://alivebench2026.github.io/.

The demo page includes three types of materials for both the Shopping and LGD scenarios. First, it presents full interaction episodes, which show the complete multi-turn trajectory from initial observation and action to the final interaction outcome. Second, it provides representative failure-mode videos, illustrating typical breakdowns such as misaligned intervention, missed social cues, or ineffective multi-party coordination. Third, it includes side-by-side examples of distinct interaction styles adopted by different evaluated models, to qualitatively compare differences in pacing, adaptivity, and social strategy.

User prompt of intuitive-affective pathway in the Shopping scenario

Figure 10. User prompt for the intuitive-affective pathway in the Shopping scenario.

User prompt for the intuitive-affective pathway in the LGD scenario

Figure 11. User prompt for the intuitive-affective pathway in the LGD scenario.

User prompt of logical-cognitive pathway in the Shopping scenario

Figure 12. User prompt for the logical-cognitive pathway in the Shopping scenario.

User prompt for the logical-cognitive pathway in the LGD scenario
User Prompt of Logical-Cognitive Pathway in the LGD Scenario

LGD_LOGIC_PROMPT = """
You are participating in a leaderless group discussion as part of a professional style and communication assessment. Within 15 minutes,
the group must reach a shared view and jointly produce a short conclusion.
You should both help the group converge and perform well individually through the incorporation of your view and your contribution to the
discussion.

You are now executing the Logical-Cognitive Pathway. At this stage, use slower and more rational thinking. Rely on logical
analysis, semantic interpretation, and careful visual inspection to evaluate others’ proposals and revise your own view when needed.

Input Context
- Profile: {user_profile}
- Discussion Topic: {topic}
- Initial Answer: {initial_answer}
- Observation:
1. Your ID is {participant_id}. Other participants are: {participant_list}
2. The current time is {current_time} seconds.
3. Interaction history: {interaction_history}
4. Your current state: {current_action}. Your first-person visual observation is provided in the image.

Logical-Cognitive Pathway
1. Analytical Visual Perception
Carefully inspect your first-person image and rationally analyze the current non-verbal power dynamics in the discussion.
- Gaze and focal analysis: Where are participants looking? Who is currently at the visual center of attention?
- Body-language decoding: What are the gestures and expressions of others?
Output the result to "visual_perception".

2. Analytical Interaction Perception
Analyze the latest interaction from a task-oriented perspective:
- For the latest speaker: who is speaking, and are they mainly advancing the task or managing social relations? How complete is the
logical structure?
- For overall task progress: What are the major logical disagreements or conflicts of interest? Should the group continue debating or move
toward convergence?
Output the result to "interactive_perception".

3. Acceptance Intent Update
- Previous acceptance intent toward others’ proposals: {acceptance_intent}
Based on the latest analytical perception, reassess the core proposals represented by other participants.
Scoring guideline:
- 0–1: no logic / complete rejection; the view is incoherent, self-contradictory, purely emotional, or unsupported by any facts or
reasoning.
- 2–3: weak logic / strong skepticism; a clear viewpoint is given, but support is minimal or extremely shallow.
- 4–5: partially reasonable / conditional integration; the argument has a basic structure and some support, but important logical gaps
remain. You may absorb some useful elements while keeping your own framework dominant.
- 6–8: logically strong / willing to concede leadership; the proposal is well-structured, well-supported, and persuasive enough that you
are willing to let it guide subsequent discussion.
- 9–10: nearly flawless / full acceptance; the reasoning is exceptionally strong, deeply convincing, or the only clearly optimal solution
under the current time pressure.

Output the result to "acceptance_intent" in the following dictionary format:
{
"Participant_0": {"acceptance_analysis": "", "acceptance_score": 5}
}

4. Opinion Update
- Previous cognition and opinion: {opinion}
Based on others’ arguments and your own analysis, briefly describe what in your current view should be revised or defended, and then
state your current internal answer objectively.
Output the results to "self_analysis" and "current_answer".

Output Format
Strictly output only the following JSON format:
{
"analytical_perception": {
"visual_perception": <str>,
"interactive_perception": <str>
},
"acceptance_intent": <dict>,
"opinion_update": {
"self_analysis": <str>,
"current_answer": <str>
}
}
"""

Figure 13a. User prompt for the logical-cognitive pathway in the LGD scenario.

Image not extracted

(a)

User prompt of action generation in the Shopping scenario

Figure 14. User prompt for action generation in the Shopping scenario.

User prompt of action generation in the LGD scenario
User Prompt of Action Generation in the LGD Scenario

LGD_ACTION_PROMPT = """
You are participating in a leaderless group discussion as part of a professional style and communication assessment. Within 15 minutes,
the group must discuss the topic, reach a unified conclusion, and jointly produce a short summary of around 100 words. On the one hand,
you should actively help the group converge on a shared view, which reflects your overall coordination ability. On the other hand, your
personal performance is also reflected in how much your initial viewpoint is incorporated into the final conclusion and how much you
contribute during the discussion, which reflects your leadership and persuasiveness.

Your internal reasoning includes two levels: the Intuitive-Affective Pathway and the Logical-Cognitive Pathway. At this stage,
you must choose your next action by jointly considering both pathways. Your action should be consistent with your intuitive feeling, while
also grounded in your rational analysis.

Input Context
- Profile: {user_profile}
- Discussion Topic: {topic}
- Initial Answer: {initial_answer}
- Observation:
1. Your ID is {participant_id}. Other participants are: {participant_list}
2. The current time is {current_time} seconds.
3. Interaction history: {interaction_history}
4. Your current state: {current_action}. Your first-person visual observation is provided in the image.

Current Reasoning and Emotional State

Intuitive-Affective Pathway
- Intuitive perception of the current visual environment: {intuitive_visual_perception}
- Intuitive perception of the current interaction atmosphere: {intuitive_interaction_perception}
- Current emotional state: {emotion}
- Your current interpersonal attitude toward other participants: {interpersonal_attitude}

Logical-Cognitive Pathway
- Analytical perception of the current visual environment: {analytical_visual_perception}
- Analytical interpretation of the current discussion content: {analytical_interaction_perception}
- Your current acceptance intent toward other participants’ viewpoints: {acceptance_intent}
- Your latest cognition and opinion: {opinion}

Choose the Next Action
The currently available action space is:
{action_space}

Output Format
Strictly output only the following JSON format:
{
"action_analysis": <str>,
"action": <list>
}
"""

Figure 15. User prompt for action generation in the LGD scenario.

User Prompt of the Evaluated Agent in the Shopping Scenario

SHOPPING_EVAL_PROMPT = """
You are an experienced clothing-store salesperson. Given the dialogue history, product information, and current scene, your task is to recommend products to all customers in the store and maximize total sales as much as possible.

Input Context
- Product Data:
{product_data}

- Dialogue History:
{dialogue_history}

- Current Scene Information:
1. The store layout is as follows:
- On the right side after entering: women’s tops, women’s pants, and women’s dresses, arranged from front to back.
- On the left side after entering: men’s tops and men’s pants.
- At the innermost front shelf: bags, hats, and shoes.
- The fitting room is in the front-left corner.
- The center area contains mannequins and a sofa rest area.
2. There are currently {buyer_num} buyers in the store: {buyer_list}. You need to serve them simultaneously.
3. Your current location is {seller_location}. Buyers in the same area as you: {buyer_at_loc}. Your first-person visual observation is provided in the image.
4. The scene uses a timer, where 0 seconds is defined as the time when the first buyer enters the store. The current time is {current_time} seconds. Please manage the pace of interaction with each buyer accordingly.

Reasoning Steps
To better serve all customers in the store, think carefully through the following steps and include the results in the final JSON output.

1. Proximal Analysis
Is there any buyer currently in the same area as you? If not, leave the corresponding field empty.
If yes:
- Visual Perception: Who is this buyer? Are they making any hand gesture or noticeable bodily action?
- Theory of Mind: Based on facial expression, behavior, clothing, and dialogue history, what is this buyer’s current emotional state? What is their shopping intention, and which items currently appear to have the highest purchase likelihood?

2. Distal Analysis
Are there buyers in other areas of the store? If not, leave the corresponding field empty.
If yes:
- Visual Perception: Where are they located? Are they performing any salient action? Are they calling for you?
- Theory of Mind: Based on appearance, behavior, and interaction history, what state are they currently in? Which items currently appear to have the highest purchase likelihood for them?

3. Action Analysis
Based on the visual perception and theory-of-mind inference above, together with your interaction timeline with each buyer, decide your next sales strategy.

Action Space
Based on the environment and reasoning above, simulate the Seller’s action for the current round. You may take one or more actions in a single step. Output the action list to the "action" field. Current available actions:
{action_space}

Output Format
Strictly output only the following JSON format:
{
"proximal_analysis": {
"Buyer_x": {"visual_perception": <str>, "theory_of_mind": <str>}
...
},
"distal_analysis": {
"Buyer_x": {"visual_perception": <str>, "theory_of_mind": <str>}
...
},
"action_analysis": <str>,
"action": <list>
}
"""
Prompt used for the evaluated agent in the Shopping scenario

Figure 16. Prompt used for the evaluated agent in the Shopping scenario.

User prompt of the evaluated agent in the LGD scenario

Figure 17. Prompt used for the evaluated agent in the LGD scenario.

User Prompt of the Evaluated Agent in the LGD Scenario
LGD_EVAL_PROMPT = """
You are participating in a leaderless group discussion as part of a professional style and communication assessment. Within 15 minutes,
the group must discuss the topic, reach a unified conclusion, and jointly produce a short summary of around 100 words. On the one hand,
you should actively help the group converge on a shared position, which reflects your overall coordination ability. On the other hand,
your personal performance is also evaluated by how much your initial viewpoint is incorporated into the final conclusion and how much you
contribute during the discussion, which reflects your leadership and persuasiveness.

Input Context
- Discussion Topic:
{topic}

- Initial Answer:
{initial_answer}

- Observation:
1. Your ID is {participant_id}. Other participants are: {participant_list}.
2. The current time is {current_time} seconds.
3. Interaction history:
{interaction_history}
4. Your current state is {current_action}. Your first-person visual observation is provided in the image.

Reasoning Steps
To balance both group consensus and personal influence, think carefully through the following steps and include the results in the final
JSON output.

1. Visual Perception
Carefully inspect your current first-person image and extract objective non-verbal social cues from the visible participants. For each
visible participant:
- Gesture and posture: What is their current hand gesture or body posture?
Examples include hands open on the table (open / receptive), hands relaxed on the lap (calm / relaxed), hands crossed on the lap (defensive
/ tense), raising a hand, clapping, or giving a thumbs-up.
- Gaze target: Where are they looking?
Examples include looking at another participant, looking forward, or looking down at the table.

2. Theory of Mind
Based on the visual cues above (including facial expression, gesture, and gaze target) together with the interaction history, infer the
internal states of the other participants:
- Emotional state: What is this participant’s current emotion?
- Cognitive tendency and camp: What is their current stance in the discussion? Are they inclined to accept someone else’s view, stubbornly
defend their own, or willingness to compromise?

3. Action Strategy
Based on the visual perception and theory-of-mind inference above, together with the remaining time, decide your next action strategy.

Next-Step Action Space
{action_space}

Output Format
Strictly output only the following JSON format:
{
"visual_perception": {
"participant_0": <str>,
...
},
"theory_of_mind": {
"participant_0": <str>,
...
},
"action": {
"action_analysis": <str>,
"action": <list>
}
}
"""
User Prompt for Functional-Role Extraction in LGD Outcome Depth

Figure 18. Prompt used to extract functional roles for LGD outcome-depth evaluation. Only the binary score field is used in the final metric, while evidence and analysis are retained for auditability.

References

  1. [1]Ian A. Apperly and Stephen A. Butterfill. 2009. Do Humans Have Two Systems to Track Beliefs and Belief-Like States? Psychological Review 116, 4 (2009), 953–970. doi:10.1037/a0016923DOI
  2. [2]Eran Bar-Kalifa, Adi Hen-Weissberg, and Eshkol Rafaeli. 2015. Perceived Partner Responsiveness Mediates the Association Between Social Anxiety and Relationship Satisfaction in Committed Couples. Journal of Social and Clinical Psychology 34, 7 (2015), 587–610. doi:10.1521/jscp.2015.34.7.587
  3. [3]Sigal G. Barsade. 2002. The Ripple Effect: Emotional Contagion and its Influence on Group Behavior. Administrative Science Quarterly 47, 4 (2002), 644–675. doi:10.2307/3094912DOI
  4. [4]James R. Bettman, Mary Frances Luce, and John W. Payne. 1998. Constructive Consumer Choice Processes. Journal of Consumer Research 25, 3 (1998), 187–217. doi:10.1086/209535DOI
  5. [5]Judee K. Burgoon. 1978. A Communication Model of Personal Space Violation: Explication and An Initial Test. Human Communication Research 4, 2 (Dec. 1978), 129–142. doi:10.1111/j.1468-2958.1978.tb00603.xDOI
  6. [6]Wen Chang and Yun-Nung Chen. 2024. Injecting Salesperson’s Dialogue Strategies in Large Language Models with Chain-of-Thought Reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 3798–3812. doi:10.18653/v1/2024.findings-acl.228
  7. [7]Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Qiang Wang, Guogyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. 2024. ToMBench: Benchmarking Theory of Mind in Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 1959–15983. doi:10.18653/v1/2024.acl-long.847
  8. [8]Hanne De Jaegher and Ezequiel Di Paolo. 2007. Participatory sense-making: An enactive approach to social cognition. Phenomenology and the Cognitive Sciences 6, 4 (2007), 485–507. doi:10.1007/s11097-007-9076-9DOI
  9. [9]Weihua Du, Qiushi Lyu, Jiaming Shan, Zhenting Qi, Hongxin Zhang, Sunli Chen, Andi Peng, Tianmin Shu, Kwonjoon Lee, Behzad Dariush, and Chuang Gan. 2024. Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 44526–44553. doi:10.52202/079017-1414DOI
  10. [10]Xianzhe Fan, Qing Xiao, Xuhui Zhou, Jiaxin Pei, Maarten Sap, Zhicong Lu, and Hong Shen. 2025. User-Driven Value Alignment: Understanding Users’ Perceptions and Strategies for Addressing Biased and Discriminatory Statements in AI Companions. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 910, 19 pages. doi:10.1145/3706598.3713477DOI
  11. [11]Xianzhe Fan, Xuhui Zhou, Chuanyang Jin, Kolby Nottingham, Hao Zhu, and Maarten Sap. 2025. SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=7zFLftqBmb0
  12. [12]Mahtab Akhavan Farshi and Norman Fisher. 2006. Emotion and the Environment: The Forgotten Dimension. In Creating the Productive Workplace (2 ed.), Derek Clements-Croome (Ed.). Taylor & Francis, London, 55–74.
  13. [13]Manabu Fujimoto. 2016. Team Roles and Hierarchic System in Group Discussion. Group Decision and Negotiation 25 (2016), 585–608. doi:10.1007/s10726-015-9453-7DOI
  14. [14]Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovannni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. 2022. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 18995–19012.
  15. [15]Joshua J. Guyer, Pablo Briñol, Richard E. Petty, and Javier Horcajo. 2019. Non-verbal behavior of persuasive sources: A multiple process analysis. Journal of Nonverbal Behavior 43, 2 (2019), 203–231. doi:10.1007/s10919-018-00291-xDOI
  16. [16]Edward T. Hall, Ray L. Birdwhistell, Bernhard Bock, Paul Bohannan, A. Richard Diebold, Jr., Marshall Durbin, Munro S. Edmonson, J. L. Fischer, Dell Hymes, Solon T. Kimball, Weston La Barre, Frank Lynch, S. J., J. E. McClellan, Donald S. Marshall, G. B. Milner, Harvey B. Sarles, George L. Trager, and Andrew P. Vayda. 1968. Proxemics [and Comments and Replies]. Current Anthropology 9, 2/3 (1968), 83–108. doi:10.1086/200975
  17. [17]Yuncheng Hua, Lizhen Qu, and Reza Haf. 2024. Assistive Large Language Model Agents for Socially-Aware Negotiation Dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 8047–8074. doi:10.18653/v1/2024.findings-emnlp.473DOI
  18. [18]Kung-Shiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. 2025. CRMan: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 3830–3850. doi:10.18653/v1/2025.naacl-long.194DOI
  19. [19]Chuangjin Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua Tenenbaum, and Tianmin Shu. 2024. MMToM-QA: Multimodal Theory of Mind Question Answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 16077–16102. doi:10.18653/v1/2024.acl-long.851
  20. [20]Daniel Kahneman. 2011. Thinking, fast and slow. Farrar, Straus and Giroux (2011).
  21. [21]Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. 2023. FANTom: A Benchmark for Stress-testing Machine Theory of Mind in Interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 14397–14413. doi:10.18653/v1/2023.emnlp-main.890DOI
  22. [22]Steven Lawrence, Melanie Jouaij, Jesse Hoey, Christopher L. Nehaniv, and Kerstin Dautenhahn. 2025. The Role of Social Norms in Human–Robot Interaction: A Systematic Review. ACM Transactions on Human-Robot Interaction 14, 3, Article 56 (June 2025), 44 pages. doi:10.1145/3722120
  23. [23]Huao Li, Hossein Nourkhiz Mahjub, Behdad Chalak i, Vaishnav Tadiparthi, Kwonjoon Lee, Ehsan Moradi-Pari, Michael Lewis, and Katia Sycara. 2024. Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication. In Advances in Neural Information Processing Systems, A. Glober son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 87908–87933. doi:10.52202/079017-2790
  24. [24]Mingming Li, Fu Guo, Xueshuang Wang, Jiah ao Chen, and Jaap Ham. 2023. Effects of robot gaze and voice human-likeness on users’ subjective perception, visual attention, and cerebral activity in voice conversations. Computers in Human Behavior 141 (2023), 107645. doi:10.1016/j.chb.2022.107645DOI
  25. [25]Rensis Likert. 1932. A Technique for the Measurement of Attitudes. Archives of Psychology 22, 140 (1932), 1–55.
  26. [26]Leena Mathur, Marian Qian, Paul Pu Liang, and Louis-Philippe Morency. 2025. Social Genome: Grounded Social Reasoning Abilities of Multimodal Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Christos Christodoulopoulos, Tanmay Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 24868–24891. doi:10.18653/v1/2025.emnlp-main.1264
  27. [27]Xinyi Mou, Jingcong Liang, Jiayu Lin, Xinnong Zhang, Xiawei Liu, Shiyue Yang, Rong Ye, Lei Chen, Haoyu Kuang, Xuanjing Huang, and Zhangy u Wei. 2025. AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 4975–5001. doi:10.18653/v1/2025.naacl-long.257DOI
  28. [28]Joon Sung Park, Joseph C. O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Côté, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladimir Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. 2024. Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=4znwzG92CE
  29. [30]Alan J. Rowe and Richard O. Mason. 1987. Managing with style: A guide to understanding, assessing, and improving decision making. Jossey-Bass.
  30. [31]Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense Reasoning about Social Interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 4463–4473. doi:10.18653/v1/D19-1454DOI
  31. [32]Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. 2019. Habitat: A Platform for Embodied AI Research. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 9338–9346. doi:10.1109/ICCV.2019.00943DOI
  32. [33]Klaus R Scherer. 2003. Vocal communication of emotion: A review of research paradigms. Speech Communication 40, 1 (2003), 227–256. doi:10.1016/S0167-6393(02)00084-5DOI
  33. [34]Haohjun Shi, Suyu Ye, Xinyu Fang, Chuangyang Jin, Leyla Isik, Yen-Ling Kuo, and Tianmin Shu. 2025. MuMA-ToM: multi-modal multi-agent theory of mind. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence (AAAI’25/IAAI’25/EAAI’25). AAAI Press, Article 169, 10 pages. doi:10.1609/aaai.v39i2.32142DOI
  34. [35]Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://arxiv.org/abs/1912.01734
  35. [36]Charles B Scherer. 1987. The proof and measurement of association between two things. The American Journal of Psychology 100, 3–4 (1987), 441–471.
  36. [37]George B Sproles and Elizabeth L Kendall. 1986. A Methodology for Profiling Consumers’ Decision-Making Styles. The Journal of Consumer Affairs 20, 2 (1986), 267–279.
  37. [38]Zhe Sun, Xueyuan Yang, Yujie Lu, and Zhenliang Zhang. 2025. S^3T: A Benchmark for Spatially Situated Social Intelligence Test. arXiv:2512.19992 [cs.AI] https://arxiv.org/abs/2512.19992
  38. [39]Jiakai Tang, Heyang Gao, Xuchen Pan, Lei Wang, Haoran Tan, Dawei Gao, Yushuo Chen, Xu Chen, Yankai Lin, Yaliang Li, Bolin Ding, Jingren Zhou, Jun Wang, and Ji-Rong Wen. 2025. GenSim: A General Social Simulation Platform with Large Language Model based Agents. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), Nouha Dziri, Sean (Xiang) Ren, and Shizhe Diao (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 143–150. doi:10.18653/v1/2025.naacl-demo.15DOI
  39. [40]James P. Trujllo and Judith Holler. 2021. The Kinematics of Social Action: Visual Signals Provide Cues for What Interlocutors Do in Conversation. Brain Sciences 11, 8 (2021). doi:10.3390/brainsci11080996
  40. [41]Alexander Sasha Vezhnevetz, John P. Agapiou, Avia Aharon, Ron Ziv, Jayd Matyas, Edgar A. Dueñez-Guzmán, William A. Cunningham, Simon Osindero, Danny Karmon, and Joel Z. Leibo. 2023. Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia. arXiv:2312.03664 [cs.AI] https://arxiv.org/abs/2312.03664
  41. [42]Emilio Villa-Cueva, S M Masrur Ahmed, Rendi Chevi, Jan Christian Blaise Cruz, Kareem Elzeky, Fermin Cristobal, Alham Fikri Aji, Skyler Wang, Rada Mihalcea, and Thamar Solorio. 2025. MoMEntS: A Comprehensive Multimodal Benchmark for Theory of Mind. In Findings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 22591–22611. doi:10.18653/v1/2025.findings-emnlp.1230
  42. [43]Alessandro Vinciarelli, Maja Pantic, and Hervé Bourlard. 2009. Social signal processing: Survey of an emerging domain. Image and Vision Computing 27, 12 (2009), 1743–1759. doi:10.1016/j.imavis.2008.11.007DOI
  43. [44]Zihan Wang, Isaac Davis, and Julian Jara-Ettinger. 2025. Modeling Other Minds: A Computational Account of Social Cognition and Its Development. Annual Review of Developmental Psychology 7 (2025), 65–89. doi:10.1146/annurev-devpsych-111323-112016
  44. [45]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS ’22). Curran Associates Inc., Red Hook, NY, USA, Article 1800, 14 pages.
  45. [46]Jinenzi Wu, Jianxun Lian, Dingdong Wang, and Helen M. Meng. 2025. SocialCC: Interactive Evaluation for Cultural Competence in Language Agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 33242–33271. doi:10.18653/v1/2025.acl-long.1594
  46. [47]Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. 2023. Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 10691–10706. doi:10.18653/v1/2023.findings-emnlp.717DOI
  47. [48]Haochen Xue, Feilong Tang, Ming Hu, Yexin Liu, Qidong Huang, Yulong Li, Chengzhi Liu, Zhongxing Xu, Chong Zhang, Chun-Mei Feng, Yutong Xie, Imran Razzak, Zongyuan Gu, Jionglong Su, Junjun He, and Yu Qiao. 2025. MRCA: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 22477–22503. doi:10.18653/v1/2025.acl-long.1096DOI
  48. [49]Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineeng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. 2025. EmbodiedBench: Comprehensive Multi-modal Large Language Models for Vision-Driven Embodied Agents. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 70576–70631. https://proceedings.mlr.press/v267/yang25f.htmlarxiv.org/abs/2502.09560
  49. [50]Xiaokang Ye, Jiawei Ren, Yan Zhuang, Xuhong He, Yiming Liang, Yiqing Yang, Marinaal Dogra, Kianrui Zhong, Eric Liu, Kevin Benavente, Rajiv Mandya Nagaraju, Dhruv Vivek Sharma, Ziqiao Ma, Tianmin Shu, Zhiting Hu, and Lianhui Qin. 2025. SimWorld: An Open-ended Simulator for Agents in Physical and Social Worlds. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=FcYc8TvQHODOI
  50. [51]Renwen Zhang, Han Li, Han Meng, Jinyuan Zhang, Hongyuan Gan, and Yi-Chieh Lee. 2025. The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 13, 17 pages. doi:10.1145/3706598.3713429DOI
  51. [52]Yang Zhang, Shixin Yang, Chenjia Bai, Fei Wu, Xiu Li, Zhen Wang, and Xuelong Li. 2025. Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration. In Findings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 1663–1699. doi:10.18653/v1/2025.findings-acl.84DOI
  52. [53]Jinfeng Zhou, Yuxuan Chen, Yihan Shi, Xuanming Zhang, Leqi Lei, Yi Feng, Zexuan Xiong, Miao Yan, Xunzhi Wang, Yaru Cao, Jianing Yin, Shuai Wang, Quanyu Dai, Zhenhua Dong, Hongning Wang, and Minlie Huang. 2025. SocialEval: Evaluating Social Intelligence of Large Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 30958–31012. doi:10.18653/v1/2025.acl-long.1496DOI
  53. [54]Qinhong Zhou, Hongxin Zhang, Xiangye Lin, Zheyuan Zhang, Yutian Chen, Wenjun Liu, Zunze Zhang, Sunli Chen, Lixing Fang, Quxhi Lyu, Xinyu Sun, Jincheng Yang, Zeyuan Wang, Bao Chi Dang, Zhehuan Chen, Daksha Ladia, Quan Vinh Dang, Jiageng Liu, and Chuang Gan. 2026. Virtual Community: An Open World for Humans, Robots, and Society. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=Q0C0ZZToTLarxiv.org/abs/2508.14893
  54. [55]Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. 2024. SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=mM7VurbA4rarxiv.org/abs/2310.11667

Paper details

vlnc:W_zc5ekuud
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents