betaFeedback

WorksComputer Science

EgoLink: Egocentric Language-Vision Interactive Network Knowledge Challenge Summary

Abstract

The EgoLink Challenge at ACM Multimedia 2026 benchmarks egocentric multimodal intelligence linking first-person perception, social reasoning, and interactive action. Unlike benchmarks centered on visible objects and activities, EgoLink evaluates whether models can interpret emotional and social cues, resolve visually grounded references, and convert user requests into verified tool-mediated actions. Its two tracks cover social reasoning in egocentric video and interactive agents that complete service tasks through dialogue, tool use, and closed-loop execution. Together, they test whether perception, reasoning, and decision-making remain coupled under both offline question answering and state-changing interaction. We highlight a multi-stage Track 1 data pipeline for multiple-choice question (MCQ) generation, chain-of-thought generation, automatic quality grading, and video-contribution filtering. We summarize the task design, evaluation metrics, rules, leaderboards, and representative solutions. The benchmark combines objective answer scoring with process- and result-based evaluation, enabling analysis of both final correctness and execution quality. The best Track 1 system achieved 0.9263 accuracy, while the best Track 2 system achieved 0.844 joint success and 0.916 micro accuracy. Participant reports reveal a shared principle: strong systems make intermediate evidence explicit through candidate verification, option-robust reasoning, scenario-aware planning, tool canonicalization, and state-aware correction before answering or acting. These findings highlight the importance of process verification and evidence-grounded reasoning for reliable egocentric multimodal intelligence.

CCS Concepts

• Computing methodologies → Computer vision tasks; Activity recognition and understanding; Natural language processing.

Keywords

egocentric video understanding, social reasoning, embodied AI, multimodal agents

ACM Reference Format:

Renhe Sun, Yueying Feng, Bohan Yu, Tong Niu, Yunqi Liu, Zitong Wang, Haolin He, Kexin Shi, Hualei Wang, Qiuqiang Kong, Chang Yao, Yuge Huang, Weiqiang Wang, Jingyuan Chen, and Jian Liu. 2026. EgoLink: Egocentric Language-Vision Interactive Network Knowledge Challenge Summary. In Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3767308.3837706

Introduction

Egocentric multimedia intelligence requires models to interpret the world from the user’s situated point of view [3]. This setting is central to AI glasses, augmented-reality assistants, social robots, and embodied service agents, where a system must reason over camera motion, partial observations, speech, gestures, objects, and ongoing human interactions [8, 9]. In such environments, a correct response often depends on more than recognizing visible entities: the system must identify emotional states, explain social causes, and infer future intents [7, 8], while deciding when a visually grounded request should become an executable action [9].

Existing egocentric benchmarks have advanced first-person perception [3], whereas EgoLink targets integrated social and interactive intelligence beyond perception. Two recurring failure modes motivate this focus. First, social answers can be linguistically plausible but unsupported by the observed video or audio [4, 5]. Second, interactive agents can produce fluent final responses while invoking the wrong tool, using an ungrounded entity, or changing the database state before the task preconditions are verified [9, 15]. Both failures reflect weak evidence–decision coupling.

The EgoLink Challenge3 addresses these issues through two complementary tracks (Figure 1). Track 1, Social Reasoning in Egocentric Video, uses a unified multiple-choice question (MCQ) protocol to evaluate emotional perception and localization, social causal reasoning, behavioral intent prediction, and egocentric semantic summarization. Track 2, Interactive Agent Challenge, evaluates whether a service agent can combine first-person visual streams, user instructions, and external tools to complete real-world tasks in retail, kitchen, restaurant, and order scenarios [9]. Together, the tracks test whether perception, reasoning, and decision-making can remain coupled under both offline question answering and closed-loop interaction.

EgoLink challenge overview

Figure 1. EgoLink challenge overview.

The paper makes three contributions. First, we present EgoLink as a benchmark that jointly studies egocentric social cognition and interactive task execution. Second, we detail the Track 1 construction pipeline—question and choice generation, template-guided chain-of-thought generation, five-dimensional quality grading, and video-contribution filtering—together with the official data, rules, evaluation protocol, and leaderboards for both tracks. Third, we analyze representative submissions and identify a shared emphasis on explicit evidence construction before final prediction or execution.

Related Work

Egocentric Social Reasoning

Egocentric video understanding has expanded from activity recognition to social and affective interpretation. The E3E^{3} dataset provides first-person videos with embodied emotion annotations and forms the basis of EgoLink Track 1 [8]. EgoLink reformulates these videos into multiple-choice questions for social reasoning. Interpreting gestures, tone, and facial expressions requires temporal and interpersonal context, as similar observations may carry different social meanings [7]. DeSIQ further shows that social-interaction MCQs can contain shortcuts exploitable without the intended multimodal context [4]. These findings motivate EgoLink’s emphasis on social answers grounded in observable multimodal evidence.

Interactive Multimodal Agents

Interactive multimodal agents extend visual understanding to tool-mediated task execution. Prior benchmarks study multi-step multimodal planning, visually grounded execution, and stateful tool-agent-user interaction [6, 12, 15]. EgoBench formalizes the egocentric setting through a user-agent-tool sandbox with simulated users, official tools, and database-state validation [9]. EgoLink Track 2 adapts this setting to interactive service tasks.

Compared with text-only tool-use benchmarks, task entities are often specified visually or dialectically. A user may refer to a product or previously observed object without providing its canonical database name. Agents must map these references to valid tool parameters and use tool feedback to guide subsequent actions, with execution evaluated against the resulting database state.

Challenge Description

Tasks

Track 1: Social Reasoning in Egocentric Video

3.1.1 Track 1: Social Reasoning in Egocentric Video. Track 1 evaluates social reasoning in egocentric video through a unified multiple-choice question (MCQ) protocol. It covers four capability groups: emotional perception and temporal localization, social causal reasoning, behavioral intent prediction, and egocentric semantic summarization. Answering requires interpreting temporal, affective, and interpersonal cues beyond object and action recognition.

3.1.2 Track 2: Interactive Agent Challenge. Track 2 evaluates interactive service-task execution using first-person visual streams, user instructions, scenario databases, and official tools/APIs. The agent must ground user references, decide whether to answer, request clarification, or invoke tools, and use returned observations to guide subsequent actions. Database updates must be supported by visual evidence and tool observations [9].

Datasets and Data Construction

The Track 1 dataset pairs E3E^{3} videos [8] with EgoLink-specific labels and MCQs. The release provides training and validation annotations, together with an evaluation benchmark of 4,030 MCQs paired with 1,055 egocentric videos. The challenge annotations must be used in place of the original E3E^{3} labels.

MetricDescription
Track 1 accuracyOverall top-1 MCQ accuracy over the evaluation questions.
Dimension scoresScores for emotion, causality, intent, and summarization question groups.
Track 2 joint successTask-level success requiring both correct tool process and correct database result.
Micro accuracyCall-level tool accuracy for diagnosing fine-grained execution behavior.
Result/tool successSeparate result-based and tool-based scores that expose process-result mismatches.

Table 1. Evaluation signals used in EgoLink.

MCQs and answer choices are generated from multimodal clips, followed by template-guided chain-of-thought rationales [13]. Quality grading follows the LLM-as-a-judge paradigm [16], using Qwen3-235B-A22B [14]. Each candidate receives a score on a five-point scale for answer consistency, distractor quality, language fluency, reasoning logic, and simplified reasoning quality. These scores guide quality-based filtering.

Following the contribution-aware principle of Audio-Contribution Filtering [5], Video-Contribution Filtering screens candidates for dependence on visual evidence (Figure 2). This step aims to reduce modality shortcuts [1] by filtering questions answerable without viewing the video.

Video-Contribution Filtering for Track 1

Figure 2. Video-Contribution Filtering for Track 1. Candidates are screened to reduce questions answerable without visual evidence.

Track 2 uses the EgoBench user-agent-tool sandbox [9] across retail, kitchen, restaurant, and order scenarios. The final release contains 309 tasks across retail6, retail10, kitchen4, restaurant5, and order2. The official final scoring list identifies a common subset of 96 tasks selected to avoid ground-truth ambiguities: 20 each from retail6, retail10, and order2, and 18 each from kitchen4 and restaurant5.

Evaluation Metrics

Track 1 uses overall top-1 MCQ accuracy as the primary metric, with additional scores for emotion, causality, intent, and summarization.

Track 2 ranks systems by joint_success_rate: a task achieves joint success only when it passes both tool-call-based process evaluation and database-result-based evaluation [9]. Micro accuracy measures call-level tool accuracy, while separate tool-based and result-based success scores diagnose process–result mismatches. Table 1 summarizes these evaluation signals.

RankTeamAccuracy
1GXU-LIPE0.9263
2Lai0.9246
3TeleAI0.9132
4EgoMinds0.9092
5wjdd0.9089
5MR-CAS0.9089

Table 2. Official Track 1 leaderboard excerpt. Accuracy is the overall top-1 MCQ score reported on the challenge website.

TeamJointMicroResultTool
MediaLab0.8440.9160.8750.854
GML-MM Group0.8230.9140.8540.823
BigVisionAgentGoGoGo0.7810.8890.8960.802
uestc_goat0.7290.8140.7600.740
EgoMinds0.6040.7330.6460.615

Table 3. Official Track 2 leaderboard excerpt. Joint success is the primary ranking metric.

Rules

The EgoLink Challenge follows rules designed to make leaderboard differences reflect reasoning and execution methods rather than hidden data advantages.

  • For Track 1, participants must pair EgoLink labels with the E3E^3 video data rather than using the original E3E^3 labels.

  • Manual annotation of the evaluation set is prohibited.

  • External labeled or unlabeled video data cannot be used for training, fine-tuning, distillation, calibration, or pseudo-label construction in Track 1.

  • Track 2 submissions must follow the official interaction protocol and output valid natural-language responses or JSON tool-call arrays.

  • The Track 2 challenge does not allow participants to directly expose the test-set database or user instructions in their prompts.

  • The challenge materials are released for non-commercial research and education use under the stated license and data-governance policy.

Participants

We report official leaderboard scores and analyze four organizer-provided technical reports: GXU-LIPE and Lai for Track 1, and MediaLab and GML-MM Group for Track 2. These reports cover a representative subset of submissions.

Challenge Results

Tables 2 and 3 report the official rankings, while Table 4 summarizes representative methods.

Track 1: Social Reasoning in Egocentric Video

The leading Track 1 approaches address complementary risks: unsupported social interpretations and sensitivity to answer-option positions.

GXU-LIPE frames answer selection as evidence verification. It builds a question-conditioned social evidence graph and evaluates each candidate for supporting, contradictory, and missing evidence. The system combines these assessments with risk-aware re-verification and all-option evidence calibration to resolve uncertainty among socially plausible candidates.

Lai focuses on the distinction between semantic candidate identity and option position. Its structured cold-start data and EgoAffect reward incorporate question type, emotional tone, and candidate elimination. Option-Perturbed DAPO reorders answer candidates and assigns rewards by semantic identity to reduce reliance on displayed letters. The reported ablation reaches 91.98% with process-aware reward and 92.75% validation accuracy with the full option-perturbed setting. These results support further evaluation of candidate discrimination under option permutations.

Track 2: Interactive Agent Challenge

Track 2 highlights the distinction between execution outcomes and tool-process correctness. For example, BigVisionAgentGoGoGo achieves 0.896 result-based success and 0.802 tool-based success. This gap motivates reporting process correctness alongside final-state success.

MediaLab couples a scenario-aware service agent with a correction agent through a shared evidence ledger. Visual hypotheses are canonicalized through official tools before execution, while the correction agent audits high-risk database mutations and final replies against dialogue context, tool observations, and current database state. Removing correction reduces Joint from 0.844 to 0.781; removing scenario rules lowers it to 0.646. The larger drop without scenario rules indicates reliance on scenario-specific guidance in the reported setting.

GML-MM Group emphasizes explicit execution-state tracking and evidence compression. It represents user requests as objectives with dependencies, constraints, and completion status. Reporter summaries retain verified facts, state changes, intermediate computations, and unresolved requirements, while paired grounding across adjacent or cropped frames helps recover occluded object references. This design links progress tracking with visual evidence recovery during multi-step execution.

Conclusion and Future Directions

EgoLink benchmarks egocentric social reasoning and interactive service-task execution. Across the submissions summarized in Table 4, candidate verification, option-robust reasoning, tool canonicalization, and execution-state tracking highlight the role of explicit intermediate evidence. The reported ablations suggest the value of option-robust training and execution correction in the evaluated settings, while generalization beyond the released scenarios is not established by the results summarized here.

TeamTrackMain mechanismReported evidence
GXU-LIPETrack 1Question-conditioned social evidence graph, candidate-wise verification, source-traceable reranking, low-confidence retraction, and all-option evidence calibration.92.89% validation accuracy and 92.63% official test accuracy.
LaiTrack 1Cold-start structured reasoning data, elimination-guided reward design, and option-perturbed DAPO that rewards semantic candidate identity rather than displayed option letters.92.75% validation accuracy and 92.46% official challenge accuracy.
MediaLabTrack 2Scenario-aware service agent, official-tool canonicalization, evidence ledger, and scope-limited correction for state-changing calls and final replies.0.844 Joint and 0.916 Micro over five test scenarios.
GML-MM GroupTrack 2Status-aware grounded execution, Reporter summaries for evidence compression, step-specific tool views, and paired visual grounding for occluded references.0.781 Joint on released retail1; 0.823 Joint over five test scenarios.

Table 4. Representative participant teams and their main technical mechanisms.

Future work should strengthen evidence grounding and robustness. For Track 1, priorities include modality-dependence tests, option-order perturbations, and temporal evidence evaluation to distinguish grounded reasoning from shortcut-based success [2, 4]. For Track 2, evaluation should extend to unseen scenarios, longer interactions, and recoverable tool failures, using step-level progress and database-state checks to assess execution reliability [10, 11]. These directions aim to strengthen the assessment of auditable reasoning and reliable execution across scenarios.

References

  1. [1]Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4971–4980. doi:10.1109/CVPR.2018.00522DOI
  2. [2]Shangzhe Di and Weidi Xie. 2024. Grounded Question-Answering in Long Egocentric Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12934–12943. https://openaccess.thecvf.com/content/CVPR2024/html/Di_Grounded_Question-Answering_in_Long_Egocentric_Videos_CVPR_2024_paper.html
  3. [3]Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, et al. 2021. Ego4D: Around the World in 3,000 Hours of Egocentric Video. arXiv preprint. arXiv:2110.07058 [cs.CV] https://arxiv.org/abs/2110.07058
  4. [4]Xiao-Yu Guo, Yuan-Fang Li, and Reza Haf. 2023. DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Singapore, 3169–3180. doi:10.18653/v1/2023.emnlp-main.191
  5. [5]Haolin He, Xingjian Du, Renhe Sun, Zheqi Dai, Yujia Xiao, Mingru Yang, Jiayi Zhou, Xiquan Li, Zhengxi Liu, Zining Liang, Chunyat Wu, Qianhua He, Tan Lee, Xie Chen, Wei-Long Zheng, Weiqiang Wang, Mark Plumbley, Jian Liu, and Qiuqiang Kong. 2025. Measuring Audio’s Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models. arXiv preprint. arXiv:2509.21060 [ees.AS] https://arxiv.org/abs/2509.21060
  6. [6]Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvuri, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. arXiv preprint. arXiv:2401.13649 [cs.LG] https://arxiv.org/abs/2401.13649
  7. [7]Fanqi Kong, Weiqin Zu, Xinyu Chen, Yaodong Yang, Song-Chun Zhu, and Xue Feng. 2025. SY-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning. arXiv preprint. arXiv:2506.05425 [cs.CV] https://arxiv.org/abs/2506.05425
  8. [8]Wang Liu, Yunying Feng, Wenkang Han, Tao Jin, Zhou Zhao, Fei Wu, Chang Yao, and Jingyuan Chen. 2024. E³: Exploring Embodied Emotion Through a Large-Scale Egocentric Video Dataset. In Advances in Neural Information Processing Systems, Vol. 37. 118182–118197. Datasets and Benchmarks Track. doi:10.52202/097017-3753
  9. [9]Yunqi Liu, Tong Niu, Zitong Wang, Zhenlong Dai, Yuqi Qing, Weiqiang Wang, and Jian Liu. 2026. EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents. arXiv preprint. arXiv:2605.27820 [cs.AI] https://arxiv.org/abs/2605.27820
  10. [10]Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. 2025. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025. Association for Computational Linguistics, Albuquerque, New Mexico, 1160–1183. doi:10.18653/v1/2025.findings-naacl.65DOI
  11. [11]Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. In Advances in Neural Information Processing Systems, Vol. 37. 74325–74362. doi:10.52202/079017-2365
  12. [12]Xizian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, and Ranjay Krishna. 2024. m&m’s: A Benchmark to Evaluate Tool-Use for Multi-Step Multi-Modal Tasks. arXiv preprint. arXiv:2403.11085 [cs.CV] https://arxiv.org/abs/2403.11085
  13. [13]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, Vol. 35. 24824–24837. doi:10.52202/068431-1800
  14. [14]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengyan Huang, Chenyu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 Technical Report. arXiv preprint. arXiv:2505.09388 [cs.CL] doi:10.48550/arXiv.2505.09388
  15. [15]Shunyu Yao, Noah Shinn, Pranav Rajpurkar, and Karthik Narasimhan. 2024. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint. arXiv:2406.12045 [cs.AI] https://arxiv.org/abs/2406.12045
  16. [16]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Daching Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, Vol. 36. 46595–46623. doi:10.52202/075280-2020DOI

Paper details

vlnc:W_p4axpmp3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents