betaFeedback

WorksComputer Science

Open Challenges and Opportunities in Human-Agent Systems

Abstract

Agentic AI systems increasingly undertake extended tasks and act on people's behalf, making human-agent interaction central to their effectiveness. Successful interaction requires understanding human perspectives, coordinating complementary capabilities, and maintaining productive relationships within socio-technical environments. This paper examines how foundations developed by the Autonomous Agents and Multi-Agent Systems (AAMAS) community inform these demands. We consider interactions ranging from individual human-agent partnerships to teams and larger groups, where participants differ in their roles, decision authority, capabilities, and objectives. We organize computational approaches around the connected challenges of understanding others and deciding how to interact with them. Theory of Mind, belief-desire-intention architectures, goal and plan recognition, inverse planning, and recursive reasoning provide mechanisms for representing and inferring partially observable mental states. Observer-aware planning, model reconciliation, and decision-theoretic communication connect these inferences to action, explanation, and information exchange. Together with shared autonomy, argumentation, and cooperative inverse reinforcement learning, these approaches offer foundations for agentic systems that account for evolving human knowledge, intentions, and expectations. We argue that progress requires treating human interaction as an ongoing decision problem shaped by uncertainty, interdependence, and potentially conflicting objectives.

Introduction

Agentic systems operate within socio-technical environments: their behavior depends on, and affects, both people and technical infrastructure. As agents take on extended tasks, make decisions, and act on people’s behalf, successful interaction requires more than interpreting user instructions. It requires understanding human perspectives, coordinating heterogeneous capabilities, accounting for embodied interactions, and maintaining productive relationships over time. In purely multi-agent terms, a human is another agent interacting with another agent. However, humans have a particular set of computational, behavioral, and intentional constraints and capabilities.

Interactions span settings involving one person and one agent, human–agent teams, and larger groups. People may participate as supervisors, operators, peers, or bystanders, with different capabilities, responsibilities, and opportunities to influence the system. These roles help determine what the person is accountable for, what they are positioned to observe, and whether they are a party to the interaction at all. Many taxonomies in this space include an ordinal scale of how much decision authority rests with the system rather than the person. A canonical example is a 10-level scale of automation [131, 105]. Decision authority may also vary across information gathering, analysis, decision selection, and action execution. Moreover, participants’ objectives may coincide, partially overlap, or conflict. Together, these dimensions shape what effective interaction demands, from cooperative planning and shared control to negotiation and resistance to manipulation.

Across these settings, a foundational challenge is reasoning about others whose beliefs, goals, intentions, and capabilities are only partially observable. Theory of Mind (ToM) concerns attributing mental states to others [107], with cognitive neuroscience examining the mechanisms supporting such social reasoning [6]. Computational ToM provides tools to represent and infer these properties, helping agents anticipate behavior, recognize misunderstandings, and communicate in ways that account for a partner’s perspective [152, 52, 104]. For human–agent interaction, this means treating people as partners whose knowledge, objectives, and expectations evolve throughout an interaction.

The Autonomous Agents and Multi-Agent Systems (AAMAS) community and broader multi-agent systems communities have developed complementary, computational approaches to this challenge. Belief–desire–intention architectures provide explicit representations of agents’ mental attitudes and support reasoning about those of others [112, 21, 22, 123, 46, 95]. Plan, intent, and goal recognition address how these latent properties can be inferred from observed behavior [138, 91]. Inverse planning interprets actions through the goals and constraints that could explain them [12], while planning-based recognition uses existing planners to evaluate candidate goals [109, 106]. Goal mirroring similarly reuses an agent’s planning capabilities to generate and compare hypotheses about another’s behavior [149]. Although inspired by accounts of biological mirroring, this computational approach should be distinguished from claims that mirror neurons themselves explain intention understanding, whose functions remain debated [38]. Recursive models, such as interactive POMDPs, further capture how agents reason about others who may, in turn, be reasoning about them [58]. These models connect understanding others to choosing how to interact with them. Observer-aware planning considers how an agent’s behavior will be interpreted [93]; explanation through model reconciliation addresses differences between the agent’s and the human’s understanding of a task [30]; inquiries require balancing interruption costs and the value of potential information [90], and decision-theoretic communication weighs the benefits and costs of exchanging information [154]. In robotics, ToM has also been used to accommodate variability in human teammates’ behavior [65]. Together, these approaches provide foundations for shared autonomy, where assistance and control depend on the human’s intentions and capabilities, and for argumentation, where participants must identify and address differences in beliefs or objectives. Cooperative inverse reinforcement learning adds a complementary perspective: a human and an agent pursue a shared reward that the agent initially does not know, making teaching, learning, and communication part of their joint decision problem.

Challenges in Current Human-Agent Systems

Foundation models have expanded agents’ ability to interpret instructions, generate explanations, and act through tools. These capabilities make human–agent collaboration more accessible, but fluent interaction alone does not establish shared understanding, appropriate delegation, or aligned incentives. This gap between fluent behavior and coordinated agency motivates recent calls to reconnect human-agentic AI with multi-agent research [23, 79, 48]. Across the settings introduced above, the central challenge is to connect what agents say and do to explicit accounts of their partners, responsibilities, and limitations. This challenge closely relates to problems studied in traditional agent research communities, especially in the Human-Agent Interaction (HAI) research sub-area. We organize this challenge around five pain points: uncertain understanding of people, poorly managed authority and commitments, unstructured disagreement, miscalibrated reliance, and insufficient support for mutual adaptation.

Motivating Scenario: A Personal AI Assistant. Consider a person who uses an LLM-based personal assistant to manage daily activities. The assistant has access to the user’s calendar, email, and other digital services, allowing it to schedule meetings, draft messages, and coordinate with others. Over time, the assistant is expected to learn the user’s preferences and priorities, providing proactive assistance rather than merely responding to explicit requests.

Suppose the user asks the assistant to organize a busy week, balancing work commitments, family responsibilities, and personal activities. Some preferences are stated explicitly, while others must be inferred from past interactions. The assistant must distinguish persistent preferences from temporary constraints, recognize when the user’s priorities change, and decide when clarification is necessary (Challenge 1). Moreover, successful assistance requires more than generating a feasible schedule. The assistant must determine which actions it can perform autonomously and which require approval. For example, rescheduling a routine appointment may be acceptable, while canceling an important meeting may not. Excessive confirmation requests undermine the assistant’s usefulness, while acting too independently risks violating the user’s expectations (Challenge 2). Furthermore, coordinating with other people introduces competing preferences, negotiation, and decisions about what personal information to disclose (Challenge 3). Strategic influence and deception add risks: other participants may misrepresent their constraints to secure favorable arrangements, while the assistant may selectively present information to steer the user’s decisions (Challenge 4). The assistant must also communicate uncertainty and disagreements without simply agreeing with the user or encouraging unwarranted trust. Finally, the relationship evolves through repeated interactions. The assistant must learn from corrections and adapt to changing preferences, while the user learns when to rely on its recommendations and when to intervene. Effective human-agent interaction therefore requires more than completing individual tasks: it demands understanding people, managing authority, supporting disagreement, calibrating trust, and enabling mutual adaptation over time (Challenge 5).

Challenge 1: Understanding People

Human instructions don’t always specify everything needed for successful collaboration. Attributing mental states is the central concern of Theory of Mind (ToM) [107]; accounts of social cognition and mirroring offer complementary perspectives on understanding others, without making them interchangeable mechanisms [6, 38]. Agents must infer goals, recognize gaps in a partner’s knowledge, and distinguish mistakes from intentional deviations. LLMs have demonstrated success on ToM tasks, but their performance remains sensitive to task formulation, and benchmark success does not establish robust social reasoning in sustained interaction [78, 129, 67]. An agent may generate a plausible account of a person’s intentions without maintaining a consistent model of those intentions or revising it appropriately when new evidence arrives [146].

The problem becomes more pronounced in groups, where participants have different knowledge, capabilities, and objectives. A useful human model must distinguish whose belief is being represented, what supports that inference, and whether it remains current. It must also represent uncertainty: metacognition research distinguishes expressed confidence from the ability to discriminate one’s correct and incorrect judgments [54]. For agentic systems, the corresponding challenge is recognizing when their interpretation of a person is unreliable enough to require clarification, verification, or restraint.

Challenge 2: Managing Authority and Commitments

Agents increasingly execute extended workflows, yet the allocation of authority within those workflows is often much less explicit than the requested outcome [127]. A user may authorize information gathering while expecting to retain control over consequential decisions. Successful assistance therefore requires more than choosing between autonomous execution and human approval: the agent must decide when to act, ask, wait, interrupt, refuse, or transfer control, while accounting for response delays and the possibility that actions are already underway. These are established problems in adjustable autonomy [126, 128, 89].

The same issue appears at the architectural level. Contemporary analyses emphasize social context, distributed organization, and human team processes [100, 86, 113]. Infrastructure proposals highlight shared protocols and orchestration [33, 141], but connecting agents and tools alone does not establish who is responsible for a subtask, which commitments are shared, or how a plan should change when a participant cannot fulfill an obligation. Classical teamwork research treats these relationships as objects of reasoning [61, 139]; contemporary systems need comparable support across language-based interactions and heterogeneous tools. Human oversight also has practical limits [56]: repeated approval requests can burden users, while escalation is ineffective if the person lacks sufficient context, time, or expertise. Proactive assistance must consequently balance the benefit of helping against the costs of interruption and loss of control.

Challenge 3: Conveying Disagreements and Explanations

Human–agent collaboration involves disagreement about facts, plans, priorities, and values. These disagreements require different responses. Uncertainty about a fact calls for inquiry; competing proposals call for deliberation; conflicting interests may require negotiation. Treating every exchange as a fixed number of debate rounds obscures these distinctions and supplies no principled criterion for when a conversation has achieved its purpose [151, 84, 108, 98]. Likewise, majority votes or an LLM judge can produce an answer without showing which evidence supports it or why it rejected opposing claims.

Explanations face a related difficulty. A fluent rationale may increase acceptance without faithfully identifying the basis of a recommendation: chain-of-thought explanations can omit factors that influenced an answer, and explanations do not consistently improve complementary human–AI performance [143, 14, 26]. The interaction therefore needs more than additional explanatory text: explanation is a social process shaped by the recipient and the decision being supported [87, 88]. Users must be able to identify assumptions, challenge premises, and understand how a challenge changes the conclusion. Moreover, disagreement need not indicate an error. People may legitimately rank values differently, particularly in decisions involving several users; aggregating their positions into a single recommendation can conceal whose priorities prevailed.

Challenge 4: Strategic Influence and Deception

The design objective for trust is appropriate reliance, rather than maximum acceptance [80]. Natural-language agents complicate this objective because confidence, warmth, and apparent empathy can be generated independently of demonstrated competence. Users may rely on confidently expressed errors, while sycophantic behavior can reinforce agreement at the expense of useful correction [157, 130, 135]. Explanations and affective signals can thus encourage delegation without supplying evidence that it is warranted.

Reliability failures must also be distinguished from strategic deception, a longstanding concern in artificial agency [29, 120]. Participants may selectively disclose information, misrepresent their intentions, or exploit another’s beliefs. These risks arise both when an agent influences a human and when it relies on other agents’ reports. ToM can support cooperation while also enabling deception [49, 121]. Influence need not involve a false statement: selective disclosure can change a recipient’s decisions while every disclosed fact remains true [72]. Affective presentation adds another channel, since emotional displays can influence human negotiation behavior [43]. Research on emotional manipulation by AI companions and socioaffective alignment extends this concern to sustained relationships [42, 75]. Longitudinal evidence also associates heavier voluntary chatbot use with greater dependence, without establishing that assigned interaction conditions caused those outcomes [53]. Neither persuasive fluency nor apparent agreement is sufficient evidence of trustworthy collaboration.

Challenge 5: Bidirectional Learning

Collaboration should improve as humans and agents learn about the task and one another. Human feedback, however, can concern a particular action, a mistaken goal inference, a general preference, or the boundaries of acceptable assistance. Preference-based reinforcement learning offers one way to learn from human evaluations [35], but these different teaching signals should not be treated interchangeably. An agent must also recognize when its representation cannot express what the person is trying to teach [11, 20, 82, 51].

Learning in the other direction is equally important. An agent that completes a task or offers an explanation has not necessarily improved the user’s understanding. Effective teaching requires a model of the learner and decisions about when to demonstrate, explain, ask, or allow independent practice [144, 114]. This creates an evaluation challenge: immediate task success can conceal persistent misconceptions, excessive dependence, or failure to transfer learning. Assessing human–agent co-learning requires measuring changes on both sides, including whether human knowledge and skills persist beyond the assisted interaction.

Possible Solutions and Directions

Relevant research communities provide representations, decision models, and interaction protocols that anticipate many of these challenges. Applying them to agentic AI, however, requires attention to the assumptions that make them work: guarantees established for a particular model may break down when language is misinterpreted, relevant information is missing, or tools have unexpected effects. As emphasized in the AAAI report and its accompanying agents panel, the opportunity that emerges here combines foundation models’ flexible interfaces and broad knowledge with explicit mechanisms for reasoning about interaction—and to evaluate the resulting system as a whole [8, 9].

Solutions for Understanding People via Explicit, Revisable Partner Models

A promising architectural direction combines language models with structured representations of beliefs, goals, intentions, and commitments. BDI architectures, collaborative planning, and recursive partner models provide foundations for distinguishing an agent’s own state from its hypotheses about others [111, 112, 61, 58]. These foundations connect resource-bounded practical reasoning and commitment [24, 37, 74] to explicit, recursive ToM implementations [21, 22, 123, 104, 95]. Language models could propose interpretations from dialogue and observations, while an explicit representation records their evidence, uncertainty, and history. Plan and goal recognition supply methods for inferring latent objectives [138, 91], including inverse planning, planner- and parser-based recognition, and goal mirroring [12, 109, 57, 149]. Work on ToM in human–AI and human–robot interaction motivates adapting these models to individual partners [152, 65]. Such a division would make inconsistencies inspectable and allow decisions to depend on a maintained partner model.

This combination also addresses limitations of classical architectures. Acquiring structured domain knowledge and specifying partner models have historically required substantial engineering. Foundation models can help propose this knowledge and generate candidate explanations of unfamiliar behavior, although those proposals still require validation. Learned approximations and abstractions may reduce the cost of recursive reasoning [52]; observer-aware planning provides a related means of considering how people will interpret the agent’s actions [93]. Evaluation should test whether these models improve anticipation, clarification, and recovery from misunderstanding across time and across different partners, rather than only performance on isolated ToM questions.

Solutions for Managing Authority and Commitment via Assistance and Control Transfer

Shared-autonomy research offers explicit decision rules for distributing initiative and authority. Transfer-of-control strategies account for decision quality, delays, and coordination costs, while extensions address time-critical and interruptible actions [126, 128]. Assistance models similarly assess whether helping is worth its cost and when missing information justifies a query [71, 125]. Work on communication in ad hoc teamwork further considers what to ask and when, even without prior coordination between teammates [154, 90]. For modern agents, these mechanisms could govern transitions between acting, requesting information, seeking authorization, and returning control. Their models must accommodate uncertainty over tool outcomes, reversibility, permissions, and human availability. Users should also be able to constrain an agent’s choices without specifying every action, extending constraint-based autonomy management from settings such as UAV planning to resources, tools, and task scope [81]. Context-sensitive reminders, proactive task assistance, and social-situation models offer complementary ways to decide when an intervention is useful and appropriate [70, 156, 76]. An orchestration architecture should maintain responsibilities and outstanding commitments alongside execution state, so that failures trigger appropriate coordination rather than merely another model call. Although an agent can model how its behavior affects future delegation [4], gaining greater autonomy should not substitute for respecting the user’s chosen limits.

Solutions for Conveying Disagreements via Explainability and Contestability

Argumentation provides a means of turning disagreement into an inspectable process. Dialogue protocols specify permissible moves, commitments, relevance requirements, and termination conditions according to whether participants are seeking information, deliberating, or negotiating [85, 10, 19]. Argumentation frameworks supply explicit rules for resolving attacks between claims and, in suitable extensions, representing support and differences in values [50, 94, 17]. These mechanisms can replace fixed debate rounds with exchanges directed toward a defined interaction goal.

Foundation models can make such systems easier to use by extracting candidate arguments from text and expressing structured results in natural language [142, 27]. Hybrid claim verification, interactive argumentation applications, and forecasting systems illustrate this combination [55, 44, 59]. However, guarantees from the formal layer remain conditional on the accuracy and completeness of the extracted arguments. Argumentation benchmarks and solver competitions offer complementary tests of language-based construction and formal computation [1, 18]; interfaces between conversational and MAS platforms provide an implementation route [41]. Interfaces should therefore preserve links to evidence and permit users to correct premises or relations. Model reconciliation adds a complementary approach by computing explanations against differences between the human’s and the agent’s task models [30, 145, 134, 31]. Argumentative explanation and implemented assistants show how to expose justifications to users [148, 97]. Together, these approaches support explanations that users can question and revise [99, 147], while making legitimate value disagreements visible rather than forcing artificial consensus [96]. Strategic exchanges additionally require negotiation policies and models of deceptive dialogue, not just argument aggregation [5, 122].

Solutions for Strategic Influence and Deception via Trust Grounding

Socio-cognitive trust models distinguish beliefs about a partner’s competence from beliefs about its willingness to cooperate [28]. Trust-aware decision models then connect estimated reliance to the agent’s behavior, including interventions intended to reduce unearned trust [153, 34, 101]. Applied to language agents, these ideas suggest grounding confidence statements in task-specific evidence and demonstrated performance; uncertainty expressions themselves can affect reliance [73]. Success should be measured through appropriate delegation, verification, and intervention, rather than user approval alone.

For interactions involving unreliable or strategic participants, reputation models provide mechanisms for combining direct experience and third-party reports [69, 115, 140]. Signaling and persuasion models clarify how incentives and selective disclosure affect communication [40, 72], while cognitive-hierarchy approaches support defenses against more sophisticated deceivers [118, 2, 3]. Mechanism design offers another route: under their specified assumptions, strategy-proof mechanisms make truthful reporting optimal, reducing dependence on detecting lies after they occur [150, 36, 62]. Security-game analysis, regret minimization in imperfect-information games, and learned best responses contribute complementary tools for adversarial interaction [77, 158, 25, 13]. Adversary-induced MDPs formalize learning against opponents with bounded memory [32]. Evolutionary and self-governance models examine when cooperation survives deception at the societal level [119, 117]. These approaches require checking their assumptions about utilities, information, and actions; attitudes toward an agent lying on a user’s behalf remain a separate normative question [124].

Appraisal-based architectures can additionally connect expressed affect to represented goals, beliefs, and coping conditions [102, 60, 83, 45]. This provides an auditable basis for emotional presentation, but internal consistency alone cannot establish that an expression is appropriate or nonmanipulative. Evaluation must also examine its effects on users’ decisions and relationships. Surveys of artificial empathy and evaluations of LLM emotion understanding provide relevant conceptual and empirical resources [103, 155, 116]. Language models offer richer test environments for these mechanisms, including fluent adversaries and varied emotional presentations; human studies remain necessary to establish how people actually respond.

Solutions for Bidirectional Learning via Closing the Loop between Teaching, Learning, and Evaluation

Human–agent co-learning can build on decision-theoretic teaching, learning from correction, and explicit learner models. Teaching can be treated as an action whose future collaborative benefit must be weighed against the cost of explaining instead of acting [136]. Feedback interpretation should distinguish local corrections from changes to goals or preferences and detect when the current hypothesis space is inadequate [20]. Cooperative inverse reinforcement learning provides a complementary formulation in which teaching, learning, and communication jointly support a shared objective that the agent initially does not know [64].

In the agent-to-human direction, tutoring and explainable-agent research support selecting explanation content and depth according to the person’s knowledge and needs [144, 66, 7, 114]. Foundation models broaden the language and demonstrations through which this exchange can occur, potentially reducing the interface and knowledge-engineering burdens of earlier teaching systems [133, 110]. Preserving their theoretical properties nevertheless requires verifying how free-form input is mapped into the underlying learning problem.

Ad hoc teamwork offers a template for jointly adapting to unfamiliar partners without prior coordination [137, 15, 16, 92]. Longitudinal evaluation should connect these directions: does the agent retain corrections, does the human develop transferable understanding, and does the partnership adapt when goals, capabilities, or team membership change? Task performance should be assessed alongside correction burden, appropriate reliance, skill retention, and recovery from misunderstanding. Simulated partners can help explore conditions and identify failure cases, but cannot substitute for human evidence about learning, trust, or autonomy. The resulting research agenda treats effective interaction as a property of the evolving human–agent relationship, supported by explicit models and protocols throughout the system.

Conclusion

The five challenges we highlighted underscore a new reality: AI agents are being embedded in real sociotechnical systems that don’t necessarily have all the prerequisite capabilities to act reliably in such situations. The slew of recent security incidents (cf. [68]) is representative not just of growing agent capabilities, but also of a failure from the LLM-research community to look back at existing agent literature.

We are not the first to raise these concerns, and the challenges and the opportunities are not limited to the ones listed above. Following position pieces from members of the traditional agents research community, e.g. [47, 79], there is emerging recognition that agents in multi-agent systems need relational capabilities in order to be able to hold individual objectives and constraints and also to be able to interact responsibly and effectively with others. Indeed, narratives prominent in the broad AI community indicate: “lessons from Multi-agent System Research can provide a valuable foundation for designing more reliable context-aware LLM-enabled multi-agent interactions” [9]; and “agentic AI systems should be understood not only as a technical construct but as systems embedded in a social context” [100].

The developments in LLMs have also provided us with novel ways to apply and extend existing works from the traditional agents research literature. A well-recognized bottleneck for classical agent architectures has been the acquisition of knowledge to equip systems with sufficient and structured domain information to apply flexibly in dynamic open environments [23]. A gap that novel foundational models can address. The new developments within LLMs also give rise to novel opportunities in hybrid systems. Our community can already provide the formal mechanisms required to support many of the capabilities discussed in this chapter. To our existing solutions, we could integrate LLMs as interfaces to communicate with users in natural language [63], as a way to generate heuristics [39], or even to create domain-specific solvers [132].

References

  1. [1]Yamen Ajjour, Carlotta Quensel, Nedim Lipka, and Henning Wachsmuth. ArgBench: Benchmarking LLMs on Computational Argumentation Tasks. In Findings of the Association for Computational Linguistics: ACL 2026, pages 21846–21874. Association for Computational Linguistics, 2026. doi: 10.18653/v1/2026.findings-acl.1099.DOI
  2. [2]Nitay Alon, Lion Schulz, Jeffrey S. Rosenschein, and Peter Dayan. A (Dis-)information Theory of Revealed and Unrevealed References: Emerging Deception and Skepticism via Theory of Mind. Open Mind, 7:608–624, 2023. doi: 10.1162/opmi_a_00097.DOI
  3. [3]Nitay Alon, Joseph M. Barnby, Stefan Sarkadi, Lion Schulz, Jeffrey S. Rosenschein, and Peter Dayan. ℵ-IPOMDP: Mitigating deception in a cognitive hierarchy with off-policy counterfactual anomaly detection. Journal of Artificial Intelligence Research, 85, 2026. doi: 10.1613/jair.1.19204.DOI
  4. [4]Nadav Kiril Altshuler and David Sarne. Modeling Assistant’s Autonomy Constraints as a Means for Improving Autonomous Assistant-Agent Design. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, pages 1468–1476. International Foundation for Autonomous Agents and Multiagent Systems, 2018. doi: 10.65109/wbji2470.DOI
  5. [5]Leila Amgoud, Yannis Dimopoulos, and Pavlos Moraitis. A unified and general framework for argumentation-based negotiation. In Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems, pages 1–8. ACM, 2007. doi: 10.1145/1329125.1329317.
  6. [6]David M. Amodio and Chris D. Frith. Meeting of minds: The medial frontal cortex and social cognition. Nature Reviews Neuroscience, 7(4):268–277, 2006. doi: 10.1038/nrn1884.DOI
  7. [7]Sule Anjomshoae, Amro Najjar, Davide Calvaresi, and Kary Främling. Explainable Agents and Robots: Results from a Systematic Literature Review. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1078–1088. International Foundation for Autonomous Agents and Multiagent Systems, 2019. doi: 10.65109/kczb5817.
  8. [8]Association for the Advancement of Artificial Intelligence. AI Agents. In AAAI 2025 Presidential Panel on the Future of AI Research. AAAI, 2025. URL https://aaai.org/wp-content/uploads/2025/03/AAAI–2025–Presanel–Report_FINAL.pdf.
  9. [9]Association for the Advancement of Artificial Intelligence. AAAI presidential panel zoom – AI agents. https://www.youtube.com/watch?v=YEiBNl2rh–8, October 2025. Video supplement to the “AI Agents” chapter in the AAAI 2025 Presidential Panel on the Future of AI Research. See commentary by Kartic Talamadupula at 14:16.
  10. [10]Katie Atkinson, Trevor Bench-Capon, and Peter Mcburney. A Dialogue Game Protocol for Multi-Agent Argument over Proposals for Action. Autonomous Agents and Multi-Agent Systems, 11(2):153–171, 2005. doi: 10.1007/s10458-005-1166-x.DOI
  11. [11]Andrea Bajcsy, Dylan P. Losey, Marcia K. O’Malley, and Anca D. Dragan. Learning from Physical Human Corrections, One Feature at a Time. In Proceedings of the 2018 ACM/IEEE International Conference on Human–Robot Interaction, pages 141–149. ACM, 2018. doi: 10.1145/3171221.3171267.DOI
  12. [12]Chris L. Baker, Rebecca Saxe, and Joshua B. Tenenbaum. Action understanding as inverse planning. Cognition, 113(3):329–349, 2009. doi: 10.1016/j.cognition.2009.07.005.DOI
  13. [13]Bikramjit Banerjee and Jing Peng. Efficient learning of multi-step best response. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, pages 60–66. ACM, 2005. doi: 10.1145/1082473.1082483.DOI
  14. [14]Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–16. ACM, 2021. doi: 10.1145/3411764.3445714.
  15. [15]Samuel Barrett, Peter Stone, and Sarit Kraus. Empirical Evaluation of Ad Hoc Teamwork in the Pursuit Domain. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 567–574. International Foundation for Autonomous Agents and Multiagent Systems, 2011. URL https://www.cs.utexas.edu/~pstone/Papers/bib2html/b2hd–AAMAS11–barrett.html.DOI
  16. [16]Samuel Barrett, Avi Rosenfeld, Sarit Kraus, and Peter Stone. Making friends on the fly: Cooperating with new teammates. Artificial Intelligence, 242:132–171, 2017. doi: 10.1016/j.artint.2016.10.005.DOI
  17. [17]Trevor Bench-Capon and Katie Atkinson. Abstract Argumentation and Values. In Argumentation in Artificial Intelligence, pages 45–64. Springer US, 2009. doi: 10.1007/978-0-387-98197-0_3.
  18. [18]Stefano Bistarelli, Lars Kotthoff, Jean-Marie Lagniez, Emmanuel Lonca, Jean-Guy Mailly, Julien Rossit, Francesco Santini, and Carlo Taticchi. The third and fourth international competitions on computational models of argumentation: Design, results and analysis. Argument & Computation, 16 (2):236–299, 2025. doi: 10.3233/aac-230013.DOI
  19. [19]Elizabeth Black and Anthony Hunter. An inquiry dialogue system. Autonomous Agents and Multi-Agent Systems, 19(2):173–209, 2009. doi: 10.1007/s10458-008-9074-5.DOI
  20. [20]Andrea Babu, Andrea Bajcsy, Jaime F. Fisac, Sampada Deglurkar, and Anca D. Dragan. Quantifying Hypothesis Space Mispecification in Learning From Human–Robot Demonstrations and Physical Corrections. IEEE Transactions on Robotics, 36(3):835–854, 2020. doi: 10.1109/tro.2020.2971415.
  21. [21]Tibor Bosse, Zulfiqar A. Memon, and Jan Treur. A two-level BDI-agent model for theory of mind and its use in social manipulation. In Proceedings of the AISB 2007 Workshop on Mindful Environments, pages 335–342, 2007.
  22. [22]Tibor Bosse, Zulfiqar A. Memon, and Jan Treur. A recursive BDI agent model for theory of mind and its applications. Applied Artificial Intelligence, 25(1):1–44, 2011. doi: 10.1080/08839514.2010.529259.DOI
  23. [23]V. Botti. Agentic AI and Multiagentic: Are We Reinventing the Wheel?, 2025. URL https://arxiv.org/abs/2506.01463. Preprint.
  24. [24]Michael E. Bratman, David J. Israel, and Martha E. Pollack. Plans and resource-bounded practical reasoning. Computational Intelligence, 4(3):349–355, 1988. doi: 10.1111/j.1467-8640.1988.tb00284.x.DOI
  25. [25]Noam Brown and Tuomas Sandholm. Solving Imperfect-Information Games via Discounted Regret Minimization. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):1829–1836, 2019. doi: 10.1609/aaai.v33i01.3311829.arxiv.org/abs/1809.04040
  26. [26]Zana Bučinca, Maja Barbara Malaya, and Krzysztof Z. Gajos. To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–21, 2021. doi: 10.1145/3449287.DOI
  27. [27]Federico Castagna, Nadin Köčicyan, Isabel Sassoon, Simon Parsons, and Elizabeth Sklar. Computational Argumentation-based Chatbots: A Survey. Journal of Artificial Intelligence Research, 80:1271–1310, 2024. doi: 10.1613/jair.1.15407.DOI
  28. [28]C. Castelfranchi and R. Falcone. Principles of trust for MAS: cognitive anatomy, social importance, and quantification. In Proceedings International Conference on Multi Agent Systems (Cat. No.98EX160), pages 72–79. IEEE Comput. Soc, 1998. doi: 10.1109/ICMAS.1998.699034.
  29. [29]Cristiano Castelfranchi. Artificial liars: Why computers will (necessarily) deceive us and each other. Ethics and Information Technology, 2(2):113–119, 2000. doi: 10.1023/a:100025403776.DOI
  30. [30]Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, and Subbarao Kambhampati. Plan explanations as model reconciliation: Moving beyond explanation as soliloquy. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 156–163. International Joint Conferences on Artificial Intelligence Organization, 2017. doi: 10.24963/ijcai.2017/23.DOI
  31. [31]Tathagata Chakraborti, Sarath Sreedharan, and Subbarao Kambhampati. The Emerging Landscape of Explainable Automated Planning & Decision Making. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 4803–4811. International Joint Conferences on Artificial Intelligence Organization, 2020. doi: 10.24963/ijcai.2020/669.DOI
  32. [32]Doran Chakraborty and Peter Stone. Online Multiagent Learning against Memory Bounded Adversaries. In Machine Learning and Knowledge Discovery in Databases, volume 5211 of Lecture Notes in Computer Science, pages 211–226. Springer, 2008. doi: 10.1007/978-3-540-87479-9_32. URL https://doi.org/10.1007/978-3-540-87479-9_32.
  33. [33]Alan Chan, Kevin Wei, Sihao Huang, Nitarshan Rajkumar, Elija Perrier, Seth Lazar, Gillian K. Hadfield, and Markus Anderljung. Infrastructure for AI Agents. Transactions on Machine Learning Research, 2025. URL https://openreview.net/forum?id=Ckh17xN2R2.arxiv.org/abs/2501.10114
  34. [34]Min Chen, Stefanos Nikolaidis, Harold Soh, David Hsu, and Siddhartha Srinivasa. Planning with Trust for Human-Robot Collaboration. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, pages 307–315. ACM, 2018. doi: 10.1145/3171221.3171264.DOI
  35. [35]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.arxiv.org/abs/1706.03741
  36. [36]Edward H. Clarke. Multipart pricing of public goods. Public Choice, 11(1):17–33, 1971. doi: 10.1007/bf01726210.DOI
  37. [37]Philip R. Cohen and Hector J. Levesque. Intention is choice with commitment. Artificial Intelligence, 42(2-3):213–261, 1990. doi: 10.1016/0004-3702(90)90055-5.DOI
  38. [38]Richard Cook, Geoffrey Bird, Caroline Catmur, Clare Press, and Cecilia Heyes. Mirror neurons: From origin to function. Behavioral and Brain Sciences, 37(2):177–192, 2014. doi: 10.1017/S0140525X13000903.DOI
  39. [39]Augusto B. Corrêa, André G. Pereira, and Jendrik Seipp. Classical planning with llm-generated heuristics: Challenging the state of the art with python code. 38, Main Conference:42070–42108, 2025. doi: 10.52202/085713-1401. URL https://proceedings.neurips.cc/paper_files/paper/2025/file/3bf4b55960aaa23553cd2a6bdc6e1b57-Paper-Conference.pdf.DOI
  40. [40]Vincent P. Crawford and Joel Sobel. Strategic Information Transmission. Econometrica, 50(6):1431–1451, 1982. doi: 10.2307/1913390.DOI
  41. [41]Marcelo Silva Custódio, Débora C. Engelmann, Rafael H. Bordini, Roberto Rodrigues-Filho, Stefan Sarkadi, and Alison R. Panisson. Rasa4JaCa - A Communication Interface Between Multi-agent Systems and Opensource Chatbot Technologies. In Advances in Practical Applications of Agents, Multi-Agent Systems, and Computational Social Science: The PAAMS Collection, volume 16031 of Lecture Notes in Computer Science, pages 291–302. Springer Nature Switzerland, 2026. doi: 10.1007/978-3-032-07638-0_24. PAAMS 2025 revised selected papers; first online 2025.
  42. [42]Julian De Freitas, Zeliha Oguz-Uguralp, and Ahmet Kaan-Uguralp. Emotional Manipulation by AI Companions, 2025. URL https://arxiv.org/abs/2508.19258. Preprint.
  43. [43]Celso M. de Melo, Peter J. Carnevale, and Jonathan Gratch. The Effect of Expression of Anger and Happiness in Computer Agents on Negotiations with Humans. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 937–944. International Foundation for Autonomous Agents and Multiagent Systems, 2011. doi: 10.65109/rtbi5119. URL https://www.ifaamas.org/Proceedings/aamas2011/.DOI
  44. [44]Adam Deijl, Deniz Gorur, and Francesca Toni. ArgLLM-App: An Interactive System for Argumentative Reasoning with Large Language Models, 2026. URL https://arxiv.org/abs/2602.24172. Preprint.
  45. [45]João Dias and Ana Paiva. Feeling and Reasoning: A Computational Model for Emotional Characters. In Progress in Artificial Intelligence, volume 3808 of Lecture Notes in Computer Science, pages 127–140. Springer Berlin Heidelberg, 2005. doi: 10.1007/11595014_13.DOI
  46. [46]Frank Dignum, David N Morley, Liz Sonenberg, and Lawrence Cavedon. Towards socially sophisticated BDI agents. In Proceedings of the International Conference on Multi-Agent Systems, volume 2000, pages 111–118, 2000.
  47. [47]Virginia Dignum and Frank Dignum. Agentifying agentic ai. arXiv preprint arXiv:2511.17332, 2025.arxiv.org/abs/2511.17332
  48. [48]Virginia Dignum and Frank Dignum. Agentifying Agentic AI, 2026. URL https://arxiv.org/abs/2511.17332v2. Preprint.
  49. [49]Xiao Pan Ding, Henry M. Wellman, Yu Wang, Genyue Fu, and Kang Lee. Theory-of-mind training causes honest young children to lie. Psychological Science, 26(11):1812–1821, 2015. doi: 10.1177/0956797615604628.DOI
  50. [50]Phan Minh Dung. On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artificial Intelligence, 77(2):321–357, 1995. doi: 10.1016/0004-3702(94)00041-x.DOI
  51. [51]Osher Elhadad, Felipe Meneguzzi, and Reuth Mirsky. GRAIL: goal recognition alignment through imitation learning. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 3777–3786. ACM, 2026. doi: 10.65109/ZCPS6180.DOI
  52. [52]Emre Erdogan, Rineke Verbrugge, and Pinar Yolum. Computational theory of mind with abstractions for effective human-agent collaboration. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 2249–2251. International Foundation for Autonomous Agents and Multiagent Systems, 2024.DOI
  53. [53]Cathy Mengying Fang, Auren R. Liu, Valdemar Danry, Eunhae Lee, Samantha W. T. Chan, Pat Pataranutaporn, Pattie Maes, Jason Phang, Michael Lampe, Lama Ahmad, and Sandhini Agarwal. How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study, 2025. URL https://arxiv.org/abs/2503.17473. Preprint.
  54. [54]Stephen M. Fleming and Hakwan C. Lau. How to measure metacognition. Frontiers in Human Neuroscience, 8:443, 2014. doi: 10.3389/fnhum.2014.00443.DOI
  55. [55]Gabriel Freedman, Adam Dejl, Deniz Gorur, Xiang Yin, Antonio Rago, and Francesca Toni. Argumentative Large Language Models for Explainable and Contestable Claim Verification. Proceedings of the AAAI Conference on Artificial Intelligence, 39(14):14930–14939, 2025. doi: 10.1609/aaai.v39i14.33637.DOI
  56. [56]Susanne Gaube, Markus Langer, Tim Miller, Kevin Baum, Raimund Dachselt, Anna Maria Feit, Ujwal Gadiraju, Harmanpreet Kaur, Mark T. Keane, Richard N. Landers, Johann Laux, Q. Vera Liao, Brian Lim, Linda Onnasch, Tim Schrills, Liz Sonenberg, Chenhao Tan, Nava Tintarev, Ziang Xiao, and Hanwei Zhang. Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems. SSRN working paper, 2026. Preprint.arxiv.org/abs/2605.16278
  57. [57]Christopher W Geib and Robert P Goldman. A probabilistic plan recognition algorithm based on plan tree grammars. Artificial Intelligence, 173(11):1101–1132, 2009.DOI
  58. [58]Piotr J. Gmytrasiewicz and Prashant Doshi. A framework for sequential planning in multi-agent settings. Journal of Artificial Intelligence Research, 24:49–79, 2005. doi: 10.1613/jair.1579.
  59. [59]Deniz Gorur, Antonio Rago, and Francesca Toni. Retrieval- and Argumentation-Enhanced Multi-Agent LLMs for Judgmental Forecasting (Extended Version with Supplementary Material), 2026. URL https://arxiv.org/abs/2510.24303v3. Preprint.
  60. [60]Jonathan Gratch and Stacy Marsella. A domain-independent framework for modeling emotion. Cognitive Systems Research, 5(4):269–306, 2004. doi: 10.1016/j.cogsys.2004.02.002.DOI
  61. [61]Barbara J. Grosz and Sarit Kraus. Collaborative plans for complex group action. Artificial Intelligence, 86(2):369–357, 1996. doi: 10.1016/0004-3702(95)00103-4.DOI
  62. [62]Theodore Groves. Incentives in Teams. Econometrica, 41(4):617–631, 1973. doi: 10.2307/1914085.DOI
  63. [63]Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. Advances in Neural Information Processing Systems, 36:79081–79094, 2023.arxiv.org/abs/2305.14909
  64. [64]Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper/2016/hash/c3395dd46c34fa7fd8d729d8cf88b7a8–Abstract.html.
  65. [65]Laura M. Hiatt, Anthony M. Harrison, and J. Gregory Trafton. Accommodating human variability in human-robot teams through theory of mind. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 2066–2071, 2011. doi: 10.5591/978-1-57735-516-8/IJCAI11-345. URL https://www.ijcai.org/Proceedings/11/Papers/345.pdf.DOI
  66. [66]Eric Horvitz. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 159–166. ACM Press, 1999. doi: 10.1145/302979.303030.DOI
  67. [67]Jennifer Hu, Felix Sosa, and Tomer Ullman. Re-evaluating theory of mind evaluation in large language models. Philosophical Transactions of the Royal Society B: Biological Sciences, 380(1932):20230499, 2025. doi: 10.1098/rstb.2023.0499.DOI
  68. [68]Hugging Face. Security incident disclosure — July 2026. July 2026. URL https://huggingface.co/blog/security-incident-july-2026. Published July 16, 2026.
  69. [69]Audun Jøsang, Roslan Ismail, and Colin Boyd. A survey of trust and reputation systems for online service provision. Decision Support Systems, 43(2):618–644, 2007. doi: 10.1016/j.dss.2005.05.019.
  70. [70]Ece Kamar and Eric Horvitz. Jogger: models for context-sensitive reminding. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1089–1090. International Foundation for Autonomous Agents and Multiagent Systems, 2011. doi: 10.65109/klcg2272.DOI
  71. [71]Ece Kamar, Ya’akov Gal, and Barbara J. Grosz. Incorporating helpful behavior into collaborative planning. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 875–882. International Foundation for Autonomous Agents and Multiagent Systems, 2009. doi: 10.65109/jqpl4088.DOI
  72. [72]Emir Kamenica and Matthew Gentzkow. Bayesian Persuasion. American Economic Review, 101(6):2590–2615, 2011. doi: 10.1257/aer.101.6.2590.DOI
  73. [73]Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan. \"I’m Not Sure, But...\": Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust. In The ACM Conference on Fairness Accountability and Transparency, pages 822–835. ACM, 2024. doi: 10.1145/3630106.3658941.DOI
  74. [74]David Kinny, Magnus Ljungberg, Anand Rao, Elizabeth Sonenberg, Gil Tidhar, and Eric Werner. Planned team activity. In *Artificial Social Systems*, volume 830 of *Lecture Notes in Computer Science*, pages 227–256. Springer Berlin Heidelberg, 1994. doi: 10.1007/3-540-58266-5_13. Revised papers from MAAMAW 1992.DOI
  75. [75]Hannah Rose Kirk, Iason Gabriel, Chris Summerfield, Bertie Vidgen, and Scott A. Hale. Why human–AI relationships need socioaffective alignment. *Humanities and Social Sciences Communications*, 12 (1):728, 2025. doi: 10.1057/s41599-025-04532-5.DOI
  76. [76]Ilir Kola, Catholijn M. Jonker, and M. Birna van Riemsdijk. Using psychological characteristics of situations for social situation comprehension in support agents. *Autonomous Agents and Multi-Agent Systems*, 37(2):24, 2023. doi: 10.1007/s10458-023-09605-w.DOI
  77. [77]D. Korzhyk, Z. Yin, C. Kiekintveld, V. Conitzer, and M. Tambe. Stackelberg vs. Nash in Security Games: An Extended Investigation of Interchangeability, Equivalence, and Uniqueness. *Journal of Artificial Intelligence Research*, 41:297–327, 2011. doi: 10.1613/jair.3269.DOI
  78. [78]Michal Kosinski. Evaluating large language models in theory of mind tasks. *Proceedings of the National Academy of Sciences*, 121(45):e2405460121, 2024. doi: 10.1073/pnas.2405460121.DOI
  79. [79]Emanuele La Malfa, Gabriele La Malfa, Samuele Marro, Jie Zhang, Elizabeth Black, Michael Luck, Philip Torr, and Michael Wooldridge. Large language models miss the multi-agent mark. In *Advances in Neural Information Processing Systems*, volume 38. Curran Associates, Inc., 2025. doi: 10.52202/085713-4663. URL https://proceedings.neurips.cc/paper_files/paper/2025/file/cc14cadd13ca5a13ba899cb49351c139-Paper-Position_Paper_Track.pdf. Position Paper Track.DOI
  80. [80]John D. Lee and Katrina A. See. Trust in Automation: Designing for Appropriate Reliance. *Human Factors: The Journal of the Human Factors and Ergonomics Society*, 46(1):50–80, 2004. doi: 10.1518/hfes.46.1.50.30392.DOI
  81. [81]Lanny Lin and Michael A. Goodrich. Sliding Autonomy for UAV Path-Planning: Adding New Dimensions to Autonomy Management. In *Proceedings of the International Conference on Autonomous Agents and Multiagent Systems*, pages 1615–1624. International Foundation for Autonomous Agents and Multiagent Systems, 2015. doi: 10.65109/xesm5076.DOI
  82. [82]Dylan P. Losey, Andrea Bajcsy, Marcia K. O’Malley, and Anca D. Dragan. Physical interaction as communication: Learning robot objectives online from human corrections. *The International Journal of Robotics Research*, 41(1):20–44, 2022. doi: 10.1177/02783649211050958.DOI
  83. [83]Stacy C. Marsella and Jonathan Gratch. EMA: A process model of appraisal dynamics. *Cognitive Systems Research*, 10(1):70–90, 2009. doi: 10.1016/j.cogsys.2008.03.005.DOI
  84. [84]Peter McBurney and Simon Parsons. Games That Agents Play: A Formal Framework for Dialogues between Autonomous Agents. *Journal of Logic, Language and Information*, 11(3):315–334, 2002. doi: 10.1023/a:1015586128739.DOI
  85. [85]Peter McBurney and Simon Parsons. Dialogue Games for Agent Argumentation. In *Argumentation in Artificial Intelligence*, pages 261–280. Springer US, 2009. doi: 10.1007/978-0-387-98197-0_13.DOI
  86. [86]Elizabeth Mieczkowski, Katherine M. Collins, Ilia Sucholutsky, Natalia Vélez, and Thomas L. Griffiths. Language Model Teams as Distributed Systems, 2026. URL https://arxiv.org/abs/2603.12229. Preprint.
  87. [87]Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. *Artificial Intelligence*, 267:1–38, 2019. doi: 10.1016/j.artint.2018.07.007.DOI
  88. [88]Tim Miller. Explainable AI Is Dead, Long Live Explainable AI! Hypothesis-Driven Decision Support Using Evaluative AI. In *ACM Conference on Fairness Accountability and Transparency*, pages 333–342. ACM, 2023. doi: 10.1145/3593013.3594001.
  89. [89]Reuth Mirsky. Artificial intelligent disobedience: Rethinking the agency of our artificial teammates. *AI Magazine*, 46(2):e70011, 2025. doi: https://doi.org/10.1002/aaai.70011. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/aaai.70011.
  90. [90]Reuth Mirsky, William Macke, Andy Wang, Harel Yedidsion, and Peter Stone. A Penny for Your Thoughts: The Value of Communication in Ad Hoc Teamwork. In *Proceedings of the International Joint Conference on Artificial Intelligence*, pages 254–260. International Joint Conferences on Artificial Intelligence Organization, 2020. doi: 10.24963/ijcai.2020/36.DOI
  91. [91]Reuth Mirsky, Sarah Keren, and Christopher Geib. Introduction to Symbolic Plan and Goal Recognition. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2021. doi: 10.2200/S01062ED1V01Y202012AIM047.DOI
  92. [92]Reuth Mirsky, Ignacio Car lucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht. A survey of ad hoc teamwork research. In European Conference on Multi-Agent Systems, pages 275–293. Springer, 2022.arxiv.org/abs/2202.10450
  93. [93]Shuwa Miura and Shlomo Zilberstein. A unifying framework for observer-aware planning and its complexity. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 610–620. PMLR, 2021. URL https://proceedings.mlr.press/v161/miura21a.html.
  94. [94]Sanjay Modgil and Henry Prakken. The ASPIC+ Framework for Structured Argumentation: A Tutorial. Argument & Computation, 5(1):31–62, 2014. doi: 10.1080/19462166.2013.869766.DOI
  95. [95]Nieves Montes, Michael Luck, Nardine Osman, Odinaldo Rodrigues, and Carles Sierra. Combining theory of mind and abductive reasoning in agent-oriented programming. Autonomous Agents and Multi-Agent Systems, 37(2):36, 2023. doi: 10.1007/s10458-023-09613-w.DOI
  96. [96]Francesca Mosca and Jose Such. An explainable assistant for multiuser privacy. Autonomous Agents and Multi-Agent Systems, 36(1):10, 2022. doi: 10.1007/s10458-021-09543-5.DOI
  97. [97]Francesca Mosca, Stefan Sarkadi, Jose M. Such, and Peter McBurney. Agent EXPRI: Licence to Explain. In Explainable, Transparent Autonomous Agents and Multi-Agent Systems, Lecture Notes in Computer Science, pages 21–38. Springer International Publishing, 2020. doi: 10.1007/978-3-030-51924-7_2.DOI
  98. [98]Quim Motger, Marc Oriol, Jordi Marco, and Xavier Franch. Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges, 2026. URL https://arxiv.org/abs/2607.26212. Preprint.
  99. [99]Elena Musi, Nadin Kö ckiyan, Khalid Al Khatib, Davide Ceolin, Emmanuelle Dietz, Klara Maximiliane Gutekunst, Annette Hautli-Janisz, Cristián Santibáñez, Jodi Schneider, Jonas Scholz, Cor Steging, Jacky Visser, and Henning Wachsmuth. Toward Reasonable Parrots: Why Large Language Models Should Argue with Us by Design. In Proceedings of the Argument Mining Workshop, pages 24–31. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.argmining-1.3.DOI
  100. [100]OECD. The agentic AI landscape and its conceptual foundations. OECD Artificial Intelligence Papers 56, OECD Publishing, Paris, 2026. URL https://doi.org/10.1787/396cf758-en.
  101. [101]Kazuo Okamura and Seiji Yamada. Adaptive trust calibration for human-AI collaboration. PLOS ONE, 15(2):e0229132, 2020. doi: 10.1371/journal.pone.0229132.DOI
  102. [102]Andrew Ortony, Gerald L. Clore, and Allan Collins. The Cognitive Structure of Emotions. Cambridge University Press, 1988. ISBN 9780521386647. doi: 10.1017/cbo9780511571299.DOI
  103. [103]Ana Paiva, Iolanda Leite, Hana Boukricha, and Ipke Wachsmuth. Empathy in Virtual Agents and Robots: A Survey. ACM Transactions on Interactive Intelligent Systems, 7(3):1–40, 2017. doi: 10.1145/2912150.DOI
  104. [104]Alison R. Panisson, Stefan Sarkadi, Peter McBurney, Simon Parsons, and Rafael H. Bordini. On the formal semantics of theory of mind in agent communication. In Agreement Technologies, volume 11327 of Lecture Notes in Computer Science, pages 18–32. Springer International Publishing, 2019. doi: 10.1007/978-3-030-17294-7_2.DOI
  105. [105]Raja Parasuraman, Thomas B Sheridan, and Christopher D Wickens. A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 30(3):286–297, 2000.pubmed.ncbi.nlm.nih.gov/11760769
  106. [106]Ramon Fraga Pereira, Nir Oren, and Felipe Meneguzzi. Landmark-based approaches for goal recognition as planning. Artificial Intelligence, 279:103217, 2020.
  107. [107]David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4):515–526, 1978. doi: 10.1017/S0140525X00076512.DOI
  108. [108]Iyad Rahwan. Guest Editorial: Argumentation in Multi-Agent Systems. Autonomous Agents and Multi-Agent Systems, 11(2):115–125, 2005. doi: 10.1007/s10458-005-3079-0.DOI
  109. [109]Miquel Ramírez and Hector Geffner. Probabilistic plan recognition using off-the-shelf classical planners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1121–1126. AAAI Press, 2010. doi: 10.1609/aaai.v24i1.7745.DOI
  110. [110]Gonzalo Ramos, Christopher Meek, Patrice Simard, Jina Suh, and Soroush Ghorashi. Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction, 35(5-6):413–451, 2020. doi: 10.1080/07370024.2020.1743931.DOI
  111. [111]Anand S. Rao and Michael P. Georgeff. Modeling Rational Agents within a BDI-Architecture. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, pages 473–484. Morgan Kaufmann, 1991. URL https://dl.acm.org/doi/10.5555/3087158.3087205.
  112. [112]Anand S. Rao and Michael P. Georgeff. BDI agents: From theory to practice. In Proceedings of the International Conference on Multiagent Systems, pages 312–319. AAAI Press, 1995. URL https://aaai.org/papers/icmas95-042-bdi-agents-from-theory-to-practice/.
  113. [113]Brian Riordan, Peter Bosch, Arash Salari­an, Mike Saeks, and Vijoy Pandey. Multi-Agent Systems have Some More to Learn from Human Multi-Agent Systems. SSRN working paper, 2026. Preprint.DOI
  114. [114]Avi Rosenfeld and Ariella Richardson. Explainability in human–agent systems. Autonomous Agents and Multi-Agent Systems, 33(6):673–705, 2019. doi: 10.1007/s10458-019-09408-y.DOI
  115. [115]Jordi Sabater and Carles Sierra. Review on Computational Trust and Reputation Models. Artificial Intelligence Review, 24(1):33–60, 2005. doi: 10.1007/s10462-004-0041-5.DOI
  116. [116]Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, and Minlie Huang. EmoBench: Evaluating the Emotional Intelligence of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5986–6004. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.326.DOI
  117. [117]Ştefan Sarkadi. Self-Governing Hybrid Societies and Deception. ACM Transactions on Autonomous and Adaptive Systems, 19(2):1–24, 2024. doi: 10.1145/3638549.DOI
  118. [118]Ştefan Sarkadi, Alison R. Panisson, Rafael H. Bordini, Peter McBurney, Simon Parsons, and Martin Chapman. Modelling deception using theory of mind in multi-agent systems. AI Communications, 32 (4):287–302, 2019. doi: 10.3233/AIC-190615.DOI
  119. [119]Ştefan Sarkadi, Alex Rutherford, Peter McBurney, Simon Parsons, and Iyad Rahwan. The evolution of deception. Royal Society Open Science, 8(9):201032, 2021. doi: 10.1098/rsos.201032.DOI
  120. [120]Stefan Sarkadi. Deception. PhD thesis, King’s College London, 2021. URL https://kclpure.kcl.ac.uk/portal/en/studentTheses/deception/.
  121. [121]Stefan Sarkadi. An arms race in theory-of-mind: Deception drives the emergence of higher-level theory-of-mind in agent societies. In IEEE International Conference on Autonomic Computing and Self-Organizing Systems, pages 1–10. IEEE, 2023. doi: 10.1109/ACSOS58161.2023.00017.DOI
  122. [122]Stefan Sarkadi, Peter John McBurney, and Simon Dominic Parsons. Deceptive Story-telling in Artificial Dialogue Games. In Proceedings of the AAAI Spring Symposium: Story-Enabled Intelligence, 2019. URL https://kclpure.kcl.ac.uk/portal/en/publications/deceptive-storytelling-in-artificial-dialogue-games/.
  123. [123]Stefan Sarkadi, Alison R. Panisson, Rafael H. Bordini, Peter McBurney, and Simon Parsons. Towards an approach for modelling uncertain theory of mind in multi-agent systems. In Agreement Technologies, volume 11327 of Lecture Notes in Computer Science, pages 3–17. Springer International Publishing, 2019. doi: 10.1007/978-3-030-17294-7_1.DOI
  124. [124]Stefan Sarkadi, Peidong Mei, and Edmond Awad. Should my agent lie for me? a study on attitudes of US-based participants towards deceptive AI in selected future-of-work scenarios. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 345–354. International Foundation for Autonomous Agents and Multiagent Systems, 2023. URL https://www.ifaamas.org/Proceedings/aamas2023/pdfs/p345.pdf.
  125. [125]David Sarne and Barbara J. Grosz. Determining the value of information for collaborative multi-agent planning. Autonomous Agents and Multi-Agent Systems, 26(3):456–496, 2013. doi: 10.1007/s10548-012-9206-9.DOI
  126. [126]Paul Scerri, David V. Pynadath, and Milind Tambe. Why the Elf Acted Autonomously: Towards a Theory of Adjustable Autonomy. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, pages 857–864. ACM Press, 2002. doi: 10.1145/544862.544944.DOI
  127. [127]Matthias Scheutz, Ravenna Thielstrom, and Mitchell Abrams. Transparency through explanations and justifications in human–robot task-based communications. International Journal of Human–Computer Interaction, 38(18-20):1739–1752, 2022. doi: 10.1080/10447318.2022.2091086.DOI
  128. [128]Nathan Schurr, Janusz Marecki, and Milind Tambe. Improving adjustable autonomy strategies for time-critical domains. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 353–360. International Foundation for Autonomous Agents and Multiagent Systems, 2009. doi: 10.65109/nhux5628.DOI
  129. [129]Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever Hans or neural theory of mind? stress testing social reasoning in large language models. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2257–2273. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.eacl-long.138. URL https://aclanthology.org/2024.eacl-long.138/.
  130. [130]Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards Understanding Sycophancy in Language Models. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.13548.
  131. [131]Thomas B Sheridan, William L Verplank, and Thurston L Brooks. Human/computer control of undersea teleoperators. In NASA. Ames Res. Center The 14th Ann. Conf. on Manual Control, 1978.
  132. [132]Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20256–20264, 2024.
  133. [133]Patrice Y. Simard, Saleema Amershi, David M. Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Christopher Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, and John Wernsing. Machine Teaching: A New Paradigm for Building Machine Learning Systems, 2017. URL https://arxiv.org/abs/1707.06742. Preprint.
  134. [134]Sarat Sreedharan, Anagha Kulkarni, and Subbarao Kambhampati. Explainable Human-AI Interaction: A Planning Perspective. Springer International Publishing, 2022. ISBN 9783031037573. doi: 10.1007/978-3-031-03767-2.DOI
  135. [135]Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar, Catarina Belem, Sheer Karny, Xinyue Hu, Lukas W. Mayer, and Padhraic Smyth. What large language models know and what people think they know. Nature Machine Intelligence, 7(2):221–231, 2025. doi: 10.1038/s42256-024-00976-7.DOI
  136. [136]Peter Stone and Sarit Kraus. To Teach or Not to Teach? Decision Making Under Uncertainty in Ad Hoc Teams. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 117–124. International Foundation for Autonomous Agents and Multiagent Systems, 2010. URL https://www.cs.utexas.edu/~pstone/Papers/bib2html/b2hd-AAMAS10-adhoc.html.
  137. [137]Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. Ad Hoc Autonomous Agent Teams: Collaboration without Pre-Coordination. Proceedings of the AAAI Conference on Artificial Intelligence, 24(1):1504–1509, 2010. doi: 10.1609/aaai.v24i1.7529.DOI
  138. [138]Gita Sukthankar, Robert P. Goldman, Christopher Geib, David V. Pynadath, and Hung Hai Bui, editors. Plan, Activity, and Intent Recognition: Theory and Practice. Morgan Kaufmann, 2014. ISBN 9780123985323. doi: 10.1016/c2012-0-01169-6.DOI
  139. [139]M. Tambe. Towards Flexible Teamwork. Journal of Artificial Intelligence Research, 7:83–124, 1997. doi: 10.1613/jair.433.DOI
  140. [140]W. T. Luke Teacy, Jigar Patel, Nicholas R. Jennings, and Michael Luck. TRAVOS: Trust and Reputation in the Context of Inaccurate Information Sources. Autonomous Agents and Multi-Agent Systems, 12(2):183–198, 2006. doi: 10.1007/s10458-006-5952-x.DOI
  141. [141]Arnaud Tournesac, Arun Gundurao, Ling Lau, and Pankaj Sachdeva. Reimagining Tech Infrastructure for (and with) Agentic AI. Technical report, McKinsey & Company, 2026. URL https://www.mckinsey.com/~/media/mckinsey/business%20functions/technology/our%20insights/reimagining%20tech%20infrastructure%20for%20and%20with%20agentic%20ai/reimagining-tech-infrastructure-for-and-with-agentic-ai.pdf.
  142. [142]Guilherme Trajano, Débora C. Engelmann, Rafel H. Bordini, Stefan Sarkadi, Jack Mumford, and Alison R. Panisson. Translating Natural Language Arguments to Computational Arguments Using LLMs. In Computational Models of Argument, Frontiers in Artificial Intelligence and Applications, pages 289–300. IOS Press, 2024. doi: 10.3233/faia240329.DOI
  143. [143]Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. In Advances in Neural Information Processing Systems, volume 36, pages 74952–74965. Curran Associates, Inc., 2023. doi: 10.52202/075280-3275. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ed3fea9033a80fea1376299fa7863f4a-Paper-Conference.pdf.DOI
  144. [144]Kurt VanLehn. The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems. Educational Psychologist, 46(4):197–221, 2011. doi: 10.1080/00461520.2011.611369.DOI
  145. [145]Stylianos Loukas Vasileiou, William Yeoh, Tran Cao Son, Ashwin Kumar, Michael Cashmore, and Dianele Magazzeni. A Logic-Based Explanation Generation Framework for Classical and Hybrid Planning Problems. Journal of Artificial Intelligence Research, 73:1473–1534, 2022. doi: 10.1613/jair.1.13431.DOI
  146. [146]Stylianos Loukas Vasileiou, Antonio Rago, Maria Vanina Martinez, and William Yeoh. Do humans and llms diverge in belief revision? evidence from a bayesian analysis. In Proceedings of the Conference on Language Modeling, 2026.
  147. [147]Stylianos Loukas Vasileiou, Antonio Rago, Francesca Toni, and William Yeoh. Argumentative Human-AI Decision-Making: Toward AI Agents That Reason With Us, Not For Us. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 3881–3886. International Foundation for Autonomous Agents and Multiagent Systems, 2026. doi: 10.65109/ekaw8904. Blue Sky Ideas Track.DOI
  148. [148]Alexandros Vassiliades, Nick Bassiliades, and Theodore Patkos. Argumentation and explainable artificial intelligence: a survey. The Knowledge Engineering Review, 36:e5, 2021. doi: 10.1017/s0269888921000011.DOI
  149. [149]Mor Vered, Gal A. Kaminka, and Sivan Biham. Online goal recognition through mirroring: Humans and agents. In Annual Conference on Advances in Cognitive Systems, Advances in Cognitive Systems. Cognitive Systems Foundation, 2016. URL https://research.monash.edu/en/publications/online-goal-recognition-through-mirroring-humans-and-agents/.
  150. [150]William Vickrey. Counter-speculations, Auctions, and Sealed Tenders. The Journal of Finance, 16(1): 8–37, 1961. doi: 10.1111/j.1540-6261.1961.tb02789.x.
  151. [151]Douglas Walton and Erik C. W. Krabbe. Commitment in Dialogue: Basic Concepts of Interpersonal Reasoning. State University of New York Press, 1995. ISBN 9780791425862. URL https://sunypress.edu/Books/C/Commitment-in-Dialogue.
  152. [152]Qiaosi Wang, Sarah Walsh, Mei Si, Jeffrey Kephart, Justin D. Weisz, and Ashok K. Goel. Theory of mind in human-AI interaction. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–6. Association for Computing Machinery, 2024. doi: 10.1145/3613905.3636308.DOI
  153. [153]Anqi Xu and Gregory Dudek. OPTIMo: Online Probabilistic Trust Inference Model for Asymmetric Human-Robot Collaborations. In Proceedings of the Annual ACM/IEEE International Conference on Human-Robot Interaction, pages 221–228. ACM, 2015. doi: 10.1145/2696454.2696492.
  154. [154]Ping Xuan, Victor Lesser, and Shlomo Zilberstein. Communication decisions in multi-agent cooperation: Model and experiments. In Proceedings of the International Conference on Autonomous Agents, pages 616–623. Association for Computing Machinery, 2001. doi: 10.1145/375735.376469.DOI
  155. [155]Nutchannon Yongsanthiachat, Parisa Ghanad Torshizi, and Stacy Marsella. Investigating Large Language Models’ Perception of Emotion Using Appraisal Theory. In International Conference on Affective Computing and Intelligent Interaction Workshops and Demos, pages 1–8. IEEE, 2023. doi: 10.1109/ aciiw59127.2023.10388194.
  156. [156]Neil Yorke-Smith, Shahin Saadati, Karen L. Myers, and David N. Morley. Like an intuitive and courteous butler: a proactive personal agent for task management. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 337–344. International Foundation for Autonomous Agents and Multiagent Systems, 2009. doi: 10.65109/rfpx8414.
  157. [157]Kaitlyn Zhou, Jena Hwang, Xiang Ren, and Maarten Sap. Relying on the Unreliable: The Impact of Language Models’ Reluctance to Express Uncertainty. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3623–3643. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.198.DOI
  158. [158]Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper_files/paper/2007/file/08d98638c6fcd194a4b1e6992063e944-Paper.pdf.DOI

Paper details

vlnc:W_hdjbnfrv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents