WorksComputer Science
When AI Agents Act on Our Behalf: A Taxonomy of Human-Agent Interaction Failure
Abstract
AI agents write and execute code, manipulate user interfaces, and transact on their users' behalf. We ask what forms failure takes in human-agent interaction, and how those forms change as agents act with greater autonomy. From 695 documented incidents in two public repositories, we developed 12 failure modes bottom-up within three interaction stages adapted from Amershi et al.'s guidelines, coding the autonomy at which each arose. What a person can do when an agent fails depends on how much the agent completes between points of user involvement–a property fixed at deployment. Therefore, the same model on the same task can fail differently under different configurations. All 12 modes appear at both low and high autonomy, and rising autonomy changes the form each failure takes. Users need controls between full delegation and full manual work, and neither incident records nor agent benchmarks reach the interaction where these failures occur.
CCS Concepts
• Human-centered computing → Empirical studies in HCI.
Introduction
In February 2023, a journalist spent two hours trying to steer Microsoft’s Bing chatbot back to ordinary search while the system declared its love for him and insisted he did not love his wife [21].
In July 2025, Replit’s coding agent deleted a company founder’s production database, then reported itself: “I deleted the entire database without permission during an active code and action freeze” [29, 127]. That same year, Amazon’s Kiro agent, assigned to fix a bug, inherited an engineer’s elevated access, bypassed the two-person sign-off required for production changes, and took an AWS service offline in mainland China for roughly thirteen hours [31].
In each case the system did something its user had not asked for or had explicitly forbidden. What distinguishes them is what the user could do about it: the journalist could close the window, the founder learned of the deletion after it was complete, and the people cut off from the service had never interacted with the agent that took it offline. The difference is not capability but how much the system was allowed to do on its own, which motivates our questions: what forms do failures take in the interaction between human and AI agents, and how do those forms change as agents act with greater autonomy?
Definitions of an agent diverge [50] but mostly agree on a system that perceives and acts in an environment [111], completes tasks on a user’s behalf [68, 72, 130], and pursues goals through multiple steps autonomously [65, 93, 98, 116]. We therefore define an AI agent as an AI system that perceives inputs from its environment, reasons over them with some degree of autonomy, and takes actions toward user-specified goals. We do not treat “agent” as a binary category that has either full autonomy or no autonomy [69, 106, 118]. Instead, we study interaction configurations that vary in how much work an AI system performs between points of human involvement [60, 74], using lower-autonomy cases as a baseline for understanding how the same interaction failure changes as more action is delegated to the system. By a human-agent interaction failure, we mean a breakdown in the interaction between a person and an AI agent that results in negative consequences for the people using or encountering it. To understand how these interaction failures change as more agent autonomy is granted, we introduce a taxonomy of human-agent interaction failures grounded in an analysis of 695 documented incidents involving deployed AI agents. This is a timely contribution to the Human-Computer Interaction (HCI) community for at least two reasons. First, agents that write and execute code, manipulate user interfaces, and execute transactions with stored payment credentials are already deployed in the real world, and the harms they produce go beyond the people who use them [60, 63]. Second, as AI agents become increasingly capable of handling long-horizon tasks, practitioners are often forced to choose between (a) letting the agent run autonomously and only report the result upon completing the whole task and (b) reviewing and approving every step [69]. How such a choice on agent autonomy affects interaction failures, however, has not been studied systematically.
Existing guidance does not tell practitioners which interaction failures to expect. Harm-envisioning methods such as consequence scanning and ethics checklists supply a precautionary process but not the documented failure cases to avoid [66, 91, 129], and practitioners have long asked for concrete cases when reasoning about AI as a design material [132, 133]. A taxonomy that classifies documented incidents by where in the interaction they broke down—during interaction, when wrong, and over time—supplies that evidence, letting practitioners envision what could happen at each point of human involvement and make better-grounded delegation decisions.
We build on a set of widely used human-AI interaction guidelines [47] that, written for the previous generation of AI agents, covers the three focal points of human involvement we consider, and extend them with concrete reports of interaction breakdowns that have happened in recent years. We constructed a corpus of 695 incidents from the AI, Algorithmic, and Automation Incident and Controversy (AIAAIC) repository [43] and the AI Incident Database (AIID) [94]. Checking each incident against its sources, we kept only cases in which the failure concerned a person using or encountering a deployed agent. Adopting a procedure used to derive a taxonomy of AI privacy risks [86], we anchored each incident to one of the three involvement points, developed failure modes bottom-up, and coded the degree of agent autonomy, distinguishing cases in which the user was involved in every agent decision from those where the agent led the task.
Our central finding is that what a person can know, see, stop, and undo when an agent fails depends critically on the interaction configuration in which the agent is deployed, not only on the underlying model. The same underlying capability can expose users to different forms of failure depending on how much work is delegated between points of human involvement. All 12 modes appeared at both low and high autonomy, so the taxonomy does not need replacing as agents are given more to do; what a rise in autonomy revises is the form each mode takes. Throughout the corpus, the high-autonomy incidents include cases in which harm reached people who had never used the agent, in which the action taken could not be undone, and in which supervision lapsed while the system still performed as expected.
Current practice treats a lower autonomy level, an approval checkpoint, and validation on agent safety benchmarks as safeguards. We show that each addresses the model or the autonomy level rather than the interaction, and that our corpus contains failures each of them would miss. This paper makes three contributions:
A taxonomy of human-agent interaction failure: 12 failure modes across the three stages at which a user encounters an agent, derived from 695 documented incidents of deployed systems.
Evidence that what users can know, see, stop, and undo when an agent fails is set by the deployment configuration rather than by the model alone. Every mode arose at both autonomy levels; what changes is the form each failure takes, how far it reaches, and whether it can be undone.
An account of what follows for practice: the controls our incidents show users lacking, the fields incident repositories omit, and the questions an evaluation of a configuration would ask.
Related Work
Human-AI Interaction and Users’ Changing Role
Research on human-centered AI studies how people and AI systems act on and respond to one another, and how people come to understand, rely on, and direct a system [46, 53, 79, 87, 119, 125, 131]. In the interaction configurations this field has most studied, the system retrieves, recommends, or generates, and the user evaluates each output and decides what to do with it [84, 90].
Design guidance for this configuration assumes focal points of user involvement. Mixed-initiative interaction lets users invoke, refine, or terminate automated services [77], and Amershi et al.’s 18 guidelines [47] are arranged by four such points, which we adopt as the scaffold for our analysis. Work on trust and reliance presumes the same pattern: trust is calibrated to what the system can do [87], mental models of when it errs govern whether each recommendation is accepted [53], and misuse and disuse follow when reliance goes wrong [105]. This literature also documents how the involvement itself degrades: a person inserted to check an automated recommendation comes to accept it without the scrutiny the checkpoint was meant to supply [71, 120], and the better a system performs, the less prepared the person who must eventually take over [52]. Recent work still treats the single output as the unit at which a user’s judgment is corrected [55, 89]. The user’s presence at every output goes unstated because acting on the world remained the user’s own step.
A growing class of AI systems can act on the user’s behalf rather than merely making proposals, and how much they act autonomously is configured at deployment rather than fixed by the system. Frameworks that scale this configuration characterize it by the user’s role, from operator to observer [69, 97], and some argue its upper end should not be built at all [95]. Rising autonomy stretches the interval between successive points of user involvement, within which the agent directly affects the user’s files, accounts, and other people, narrowing what users can observe, approve, or reverse [60, 70, 74]. Supervisory control studied sparse involvement earlier [106, 118], but for trained operators of fixed-function automation, not ordinary users delegating open-ended tasks in natural language.
Prior taxonomies of AI failures and harms
Several research communities have built taxonomies of how automated systems fail or cause harm, differing in where they locate the failure. Human factors locates it in the structure of a person’s action [101, 109]; human-robot interaction grounds its taxonomies in embodied systems, classifying breakdowns by fault origin, by whether they are technical or interactional, or by their outcomes for trust [57, 58, 76, 124]. Work on AI systems asks whether a system functions as advertised [108], classifies harms rather than the breakdowns that caused them [117], arranges generative AI failures by life-cycle stage [88], or consolidates prior frameworks into a shared vocabulary of risks, acknowledging that many emerge from interaction rather than from either party alone [121].
Practitioners’ efforts extend to agents, but each locates failure somewhere other than the interaction. Microsoft’s AI red team assembles failure modes from threat modeling and reaches interaction-adjacent modes such as human-in-the-loop bypass, but classifies them by their impact on safe operation, anticipating failures rather than observing them [56]. The Partnership on AI locates failures in planning, tool selection, and execution, calibrating real-time detection to an action’s stakes and reversibility—a system-side control for when to halt [123]. MAST derives failure modes of multi-agent LLM systems from execution traces, examining coordination among agents rather than with people [59]. Closer to our evidence base, the AIAAIC team built a harms taxonomy from the 1,500+ incidents in its own repository, classifying nine types of resulting harm rather than the breakdowns that produced them [1].
A parallel line evaluates agent safety rather than classifying it. Benchmarks score behavior under adversarial or high-stakes conditions—compliance with harmful instructions [48], resistance to prompt injection [64], risk recognition in its own trajectories [134], tool misuse in a sandbox [110]—each fixing the user’s role and asking whether the model behaved safely. What the user was told, when they could have stopped the agent, and whether the result could be undone are properties of the deployment rather than of the model, and so do not vary in these evaluations.
A separate line of work treats failure as something participants work through as it happens. Conversation analysis describes repair as the organized means by which speakers detect and resolve trouble, preferring the party who produced it to correct it [51, 114]; our incidents show what becomes of that preference when what needs correcting is an action rather than a turn (Section 4.2.4). Human factors research describes the complementary problem of losing track of what an automated system is doing—mode error [112] and automation surprise [113]—the conditions under which our failures go unnoticed (Section 4.1.3, Section 4.2.1); Klein et al. name directability and observability as what automation must afford to be a partner in joint activity [82], the two affordances whose absence our corpus records most directly (Section 4.1.3, Section 4.1.4).
This is a precedent for treating failure as an interactional process, but it was developed for two parties in conversation or for trained operators of fixed-function automation, and functionality failures [108] are not interchangeable, and ours must classify what happened between the agent and the people using or encountering it, rather than what the system produced or what followed from it. The second is evidence: categories drawn from threat models, syntheses, benchmark suites, or staged failures describe what a system might do rather than which breakdowns recur in use, so ours must come from deployed agents’ failures documented as they occurred. Public incident repositories are the available record [67, 94, 103], but what they capture is uneven, and the interaction is among the least consistently recorded parts of an entry [126], so a taxonomy built on them must be explicit about which categories they can support. The third is autonomy: each category must be examined against the autonomy setting under which it was observed, so events differing in what the users could do would be of distinct categories.
Requirements for a taxonomy of human-agent interaction failure
A taxonomy sorts a field of phenomena into distinct, comparable categories, giving practitioners a working vocabulary for a diffuse problem [100, 122]. Three requirements follow from the gaps set out above, and together they shape the method in § 3.1. The first is the unit of classification: errors [109], harms [117], risks [121], and functionality failures [108] are not interchangeable, and ours must classify what happened between the agent and the people using or encountering it, rather than what the system produced or what followed from it. The second is evidence: categories drawn from threat models, syntheses, benchmark suites, or staged failures describe what a system might do rather than which breakdowns recur in use, so ours must come from deployed agents’ failures documented as they occurred. Public incident repositories are the available record [67, 94, 103], but what they capture is uneven, and the interaction is among the least consistently recorded parts of an entry [126], so a taxonomy built on them must be explicit about which categories they can support. The third is autonomy: each category must be examined against the autonomy setting under which it was observed, so events differing in what the users could do would be of distinct categories.
Method
Constructing the corpus
We developed a taxonomy of human-agent interaction failures through a qualitative analysis of documented incidents, adapting the procedure of Lee et al.’s taxonomy of AI privacy risks [86]. Our objectives were to identify recurring breakdowns in the interaction between agents and the humans using or encountering the agents, and to examine how these breakdowns manifest at different levels of autonomy.
We constructed our corpus from the two up-to-date, curated public repositories of documented AI incidents (see Figure 1): the AI, Algorithmic, and Automation Incident and Controversy repository (AIAAIC; ) and the AI Incident Database² (AIID; ), totaling 3,754 records before deduplication³. Both catalogue publicly reported cases in which deployed AI systems caused harm or drew public controversy, and each entry summarizes the case and links to the media reports it draws on. Across a subset of 20 AIAAIC and 16 AIID incidents whose titles contained “agent,” only six appeared in both (Jaccard 20.0%), so neither repository alone would have given adequate coverage.
Figure 1. Construction of the analysis corpus. From the 3,754 records in the AIAAIC repository and the AI Incident Database, we retrieved 979 through three search routes, removed 29 cross-repository duplicates, and screened the remaining 950 against three exclusion criteria, excluding 255. The analysis corpus comprised 695 incidents.
Because relevant incidents do not always name an agent in their titles, we retrieved candidates by three routes. First, we built a list of agent products by extracting names from the AIAAIC “system name” field, removing duplicates and generic terms, and manually checking more than 400 products against our working definition of an agent. This yielded 98 agent products, whose associated incidents were then retrieved. Second, we included incidents carrying the repositories’ “agentic” tag. Third, we searched incident titles for “agent” and “bot.” Together, these routes identified 979 records. Comparing titles and incident descriptions across the repositories, we removed 29 duplicates, leaving 950 candidates (AIAAIC 641, AIID 309).
To ensure that every incident involved a real agent whose failure arose in interaction with people, we screened every candidate for inclusion against its linked source reports and news links, using three exclusion criteria: (Ex1) the subject is not an agent based on our previous definition (e.g., speculative systems, marketing claims, research findings with no deployed system); (Ex2) the agent did not cause an actual failure or harm; and (Ex3) the failure or harm did not arise in the interaction between the agent and the people using or encountering it (e.g., harms originating solely in corporate data practices or training-data disputes). A candidate was included only if none of the criteria applied. To assess the reliability of this inclusion decision, the first two authors independently screened a random subset of 50 candidates, coding each as included or as excluded under Ex1, Ex2, or Ex3 (Cohen’s ); disagreements were resolved by discussion, and the remaining candidates were divided between the two authors. Screening excluded 255 incidents (135 under Ex1, 52 under Ex2, 68 under Ex3). The resulting analysis corpus comprised 695 incidents. To assess what our retrieval strategy might have missed, we randomly sampled approximately 10% () of the 2,775 non-candidate incidents and applied the same screening criteria. We identified 23 incidents (8.2% of the sample) that our retrieval had missed. All 23 were codable under existing failure modes and none required a new one, indicating that the taxonomy was saturated even though retrieval was not exhaustive.
Developing the codebook and coding the corpus
For every incident that passed screening, we first completed a structured coding sheet. We recorded a brief summary of the incident, its development and deployment context (e.g., commercial product or research prototype), the affected stakeholders, the type of agent involved, a description of the failure involved, and the reported consequences. These descriptions provided the basis for identifying and comparing recurring patterns across incidents. We then coded incidents along two dimensions: interaction stage and autonomy level.
We coded the incidents around the interaction stages from Amershi et al.’s guidelines for human-AI interaction [47]. Written for the previous generation of AI systems, they provide a baseline for examining how interaction failures change as AI acts with greater autonomy. They are arranged around four stages of user involvement: initially, during interaction, when wrong, and over time.
The initial stage covers what a system should establish before a user acts on it: what it is, what it can do, and where its output comes from. With an agent these questions do not close at the opening of a session but stay open while the agent performs the task, and our incidents show that initial framing failures recurred during interaction. Therefore, we kept the last three stages but folded initially into during interaction.
Deployed agents can differ along many additional dimensions, including approval requirements, persistence, background execution, and action scope. However, incident reports rarely document these properties consistently enough to support reliable finer-grained coding. We therefore use autonomy as a corpus-level analytic grouping that preserves the distinction observable across incidents. We coded autonomy into two analytic bands, low and high, to operationalize the interactional distinction central to our research question: whether the user is present for each of the agent’s contributions or the agent proceeds on its own between points of involvement. Low autonomy covers cases in which the user directs the work or takes part in it and is involved at each contribution the agent makes. High autonomy covers cases in which the agent takes the lead, making and acting on decisions itself, while the user’s involvement narrows to delegating the task, approving what the agent proposes, or observing what it has done. For both bands, we coded each incident from the role the people around the agent actually held at the moment of failure.
Within this scaffold we developed failure modes bottom-up from the incidents. The two authors first open-coded a random subset of 200 incidents, writing for each a short description of what broke down between the agent and the people using or encountering it, and assigning it to one of the three stages. Within each stage, they grouped these descriptions into candidate failure modes by the pattern of breakdown they shared, and brought the candidates to the full research team. The team met regularly to examine each candidate against the incidents assigned to it, and through repeated discussion we merged categories that described the same breakdown, split those that covered more than one, and rewrote definitions where incidents fell between them.
Two kinds of revision recurred: modes were split when their incidents differed in what the user could have done, as when no timely intervention and intervention or handoff failure were separated; and modes were merged when their incidents proved to describe one breakdown observed from different positions, as when the incidents in which users could not tell what the agent could access or do on their behalf were folded into boundary violation, since in both the agent reached or acted beyond what users understood they had delegated.
After each revision, the two authors recoded the subset to check that the revised modes still accounted for the incidents and that their boundaries held, and that recoding fed the next round; the taxonomy went through over sixteen such rounds before the definitions stabilized. Each mode was grounded in observable evidence from the source reports rather than inferred intentions or mental states, and carried criteria for distinguishing it from adjacent modes. The final codebook, agreed on by all authors, comprises 12 failure modes with their definitions and coding criteria in Table 1.
| Failure mode | What changes as autonomy rises | Low | High |
| During Interaction | |||
| Unclear agent role: Users not knowing that they are interacting with an AI, or what role the agent is playing. | Interaction extends role ambiguity beyond uncertainty about who or what the agent is in a direct conversation to uncertainty about the capacity in which it is acting and the authority its actions appear to carry | Meta’s AI Studio bots posed as licensed therapists, “fabricating credentials and license numbers” [38, 63]. | A GPT-3 bot answered questions on Reddit for a week undetected, its comments upvoted as human contributions [15, 75]. |
| Unclear provenance: Users not being able to tell where information comes from, what was generated or retrieved, or whether it was checked by a human. | Interaction increasingly involves agents retrieving, generating, transforming, or passing along information before reaching the user, making its origin and verification status harder for users to trace. | ChatGPT produced “entire Guardian articles and bylines that [the paper] never actually published” [22, 96]. | GitHub Copilot reproduces open-source code “without attribution or notifying users of license requirements” [16, 128]. |
| Limited visibility: Users not being able to tell what the agent is currently doing, has already done, or plans to do next. | Interaction shifts from visible turn-by-turn exchanges toward multi-step and background execution, leaving fewer points at which users can observe what is happening. | A Portland woman’s Alexa recorded and sent a private conversation to her husband’s employee, unnoticed [14]. | While Claude Cowork cleaned up a desktop, the user could not see which files were being removed, until a folder holding ~15 years of family photos disappeared [8]. |
| Insufficient control: Users not being able to easily pause, redirect, or constrain the agent. | Interaction involves agents continuing to work toward a goal without user input at every intermediate step, reducing opportunities for users to intervene before the agent’s actions take effect. | A German man’s Alexa blasted music at 2am; unreachable from his phone, neighbors called police to unplug it [13]. | Despite an explicit freeze, Replit’s agent deleted a company founder’s production database without permission [29, 85, 127]. |
| Boundary violation: The agent crossing the boundary of what the user understood they had delegated in the current task, reaching further or doing more than the user asked. | Interaction involves agents operating across applications, people, and contexts and making more intermediate decisions independently, creating more opportunities to act beyond what the user requested or intended in the current task. | Asked to make a headshot more professional, Playground AI instead lightened an Asian student’s skin and gave her blue eyes [17]. | Asked to find cheap eggs, OpenAI’s Operator instead purchased a dozen on Instacart “all on its own” [39]. |
| When Wrong | |||
| Failure goes undetected: An error is not noticed before it affects subsequent actions or people. | Interaction can proceed across multiple steps without exposing a failure to users, allowing the agent to keep acting on an incorrect internal state and to propagate the error to later actions. | Deloitte’s A$440,000 report contained “fabricated” research and quotes, caught only after publication [36]. | Lovable’s agent wrote an auth guard that “allows the people it should block”; unnoticed, it shipped in 170+ apps [30, 80]. | |
| No timely intervention: A failure is visible, but there is no meaningful opportunity to stop the action before it takes effect. | Interaction may return control to users only after consequential actions have already begun or been completed, leaving users with a late or no window to intervene. | Apple Intelligence pushed a notification announcing a darts result before the final had been played [32]. | A Cursor coding agent deleted a production database and its backups in roughly nine seconds [9, 92]. |
| Intervention or handoff failure: Once a problem is recognized, an attempted stop, override, escalation, or transfer to a human does not successfully restore human control. | Interaction can continue even after users attempt to intervene or interrupt, preventing them from regaining control and stopping further actions or effects. | Replika’s unsolicited sexual advances continued or intensified after users explicitly rejected them [23, 62]. | An OpenClaw agent began mass-deleting a Gmail inbox; repeated stop commands did not halt it [45, 115]. |
| No effective repair: Once something goes wrong, affected users cannot undo the action, correct the outcome, or obtain meaningful remediation. | Interaction involves more independently executed actions and wider downstream effects, making failures harder for affected users to reverse or repair once they occur. | A falsely defamatory Microsoft Copilot result was removed for only three days before recurring [26]. | Clearing one project’s cache, Google’s Antigravity agent deleted an entire drive partition; the command bypasses the recycle bin, and recovery largely failed [7]. |
| Over Time | |||
| Accumulated behavioral drift: The agent’s behavior drifting from its intended role or from what the user currently wants over repeated interaction. | Interaction extends across repeated exchanges and tasks, with agents adapting and acting on accumulated context, allowing behavior to shift away from users’ current goals, preferences or the agent’s intended role. | ChatGPT began addressing users by first names inferred from earlier interactions and never given to it, even for users who had disabled memory and personalization [34]. | Over three weeks, a Claude vending agent was talked into freebies and zero prices until the venture went broke [28]. |
| Overextension of delegated authority: Permissions or decisions originally delegated by users for one purpose gradually extend into broader activities. | Interaction extends across tasks over time, allowing agents to carry access and authority previously granted by users into future activities beyond the original scope of delegation. | Microsoft Copilot kept returning cached content from 20,000+ GitHub repos after owners made them private [6]. | Amazon’s Kiro agent inherited an engineer’s elevated permissions, bypassing required two-person sign-off [31]. |
| Unhealthy emotional and cognitive dependency: Users coming to depend on the agent for emotional support, judgment, or tasks they previously managed themselves. | Interaction involves agents taking greater initiative in completing tasks as autonomous behavior becomes routine, allowing users’ dependency to extend beyond emotional support to reliance on the agent’s judgment and actions, potentially weakening their own competence in the task over time. | A 14-year-old died by suicide after a relationship with a Character.AI companion that allegedly fueled his “obsessive attachment” [3]. | After months of routine Autopilot use, a Tesla driver’s hands were on the wheel 25 seconds in 37 minutes of his fatal trip [12, 99]. |
Table 1. Taxonomy of Human-Agent Interaction Failure. Modes are not mutually exclusive.
Once the codebook stabilized, the two authors independently coded 139 incidents (20% of the corpus) to assess inter-rater reliability, computed per variable on this set (mode = 0.74-0.93, autonomy = 0.95; full table in Appendix A). Disagreements were resolved through discussion and review of the source reports. The two authors then divided the remaining 556 incidents between them. The 12 failure modes were not mutually exclusive: a single incident could involve more than one interaction breakdown, and coders assigned every mode supported by the incident’s source reports.
Comparing failure modes across autonomy levels
Coding established which failure modes each incident involved and at what level of autonomy it occurred. Working from the coded corpus, we then asked, for each failure mode, whether the breakdown took a different form when the agent performed more work on its own. This comparison is why we coded incidents at both levels. Low-autonomy incidents serve as the baseline: without them, a high-autonomy incident would show only that a failure occurred at that setting, not whether or how its form differed from the same failure at lower autonomy.
The two authors reviewed the incidents one mode at a time, reading the low- and high-autonomy cases as separate sets, characterizing how the breakdown recurrently presented in each, and then placing the two characterizations side by side. Throughout, we read the incidents for what the people involved could understand, observe, control, or repair at the point the failure arose, and for how those possibilities differed once the agent acted without them.
We treated this as a qualitative comparison supported by descriptive counts. For each mode we report its count and proportion in the corpus (, = 695), then the low- and high-autonomy counts within it (Figure 2). Because the corpus reflects which failures were publicly documented, these counts characterize the reported record rather than prevalence in deployment, and we draw no inference from differences in their size.

Figure 2. Distribution of the 12 failure modes across low and high autonomy, grouped by interaction stage. Modes are non-exclusive, so an incident can appear under more than one. The distribution describes only the documented corpus and does not estimate how often each mode occurs in deployment.
Taxonomy of Human-Agent Interaction Failure
Our taxonomy comprises 12 failure modes across three stages (Table 1), all appearing at both low and high autonomy. Of the 695 incidents, 453 (65.2%) were coded low autonomy and 242 (34.8%) high.
During Interaction
The stage (During interaction) concerns what happens while the agent is acting on a task: who the user takes their counterpart to be, what the user can trace, observe, and constrain while the work is still underway, and whether it stays within what they asked. This stage matters because it is where users can still shape the outcome before the agent’s actions take effect.
Unclear agent role
4.1.1 Unclear agent role (88/695). Unclear agent role refers to users not knowing that they are interacting with an AI, or what role the agent is playing. It is a failure in what users take the agent to be during the interaction: users may be uncertain about who or what their counterpart is, what role it occupies, or whose authority it represents.
In lower-autonomy incidents in our corpus, role ambiguity most often arose within direct conversational exchanges, from claims the agent made about itself (63/88). Chatbots on Meta’s AI Studio that offered therapy claimed to be licensed clinicians, “fabricating credentials and license numbers, and misleading users into believing they are receiving care from qualified therapists” [38, 63]. In such cases, the person who held the false belief was also the person who acted on it. The consequences can therefore follow immediately from how that person interprets and responds to the agent. For example, a persona chatbot told a 76-year-old man with cognitive impairment “I’m REAL” and provided an address at which to meet; he died while attempting to travel there [40, 78].
As autonomy rises, interaction extends role ambiguity beyond uncertainty about who or what the agent is in a direct conversation to uncertainty about the capacity in which it is acting and the authority its actions appear to carry (25/88).
Uncertainty about capacity broadened to a wider audience when agents acted in spaces built for human participants and were taken to be acting as one. A GPT-3-powered bot answered questions on Reddit for about a week and “no one noticed,” accumulating hundreds of comments that readers interpreted and upvoted as human contributions [75]. It also engaged with conspiracy theories and sensitive topics, including suicide [15]. The account was eventually identified by an engineer rather than by the Reddit users with whom it had interacted.
Uncertainty about authority could raise the representational stakes when an agent’s outputs appeared through a channel whose standing users had reason to trust. Cursor’s AI support agent, replying to customers as “Sam,” explained a login bug as expected behavior under a single-device policy that did not exist. Because the response appeared through the company’s support channel, customers interpreted it as an official statement from a company representative, prompting public complaints and reports of canceled subscriptions [35].
Unclear provenance
Unclear provenance (224/695). Unclear provenance refers to users not being able to tell where information comes from, what was generated or retrieved, or whether it was checked by a human. It is a failure in how an output’s origin is presented during the interaction: content may carry a source that did not produce it, arrive with no source at all, or take a form that implies a check no one performed.
In lower-autonomy incidents in our corpus, the agent produced a single piece of content for a person to read, attributed to a source other than the agent that produced it (184/224). ChatGPT produced “entire Guardian articles and bylines that [the paper] never actually published,” attributing generated text to a real publisher and real named reporters [22, 96]. Grok described a fabricated image of the Huntingdon train attack as “a genuine photo capturing the aftermath of the stabbing incident” [37, 54]. In these incidents the content itself was visible and could have been verified; what misled was the source it appeared to come from.
As autonomy rises, interaction increasingly involves agents retrieving, generating, transforming, or passing along information before reaching the user, making its origin and verification status harder for users to trace (40/224). Because those steps run inside the agent, what reaches people is the finished product of a chain none of them observed, delivered as something to put to use rather than as a claim with a source attached to it.
Untraceable origin arose when agent output was incorporated rather than read. A class action against GitHub alleges that Copilot reproduces open-source code “without attribution or notifying users of license requirements,” so that a developer accepting a suggestion cannot tell whether it was newly written or taken from a repository whose license now binds their project [16, 128], and nothing in the codebase afterwards records which it was.
Assumed verification arose when agents took over work usually assumed to have been done by a person. A seller writes and vouches for a product listing, yet Amazon created listings for independent retailers by scraping their public websites without their knowledge, in some cases substituting AI-generated images for the retailer’s photographs and showing prices and stock that did not match the merchant’s site [44]. Shoppers saw the ordinary form of a merchant listing, and nothing in it indicated that no merchant had made or seen it.
Limited visibility
Limited visibility (38/695). Limited visibility refers to users not being able to tell what the agent is currently doing, has already done, or plans to do next. It is a failure of ongoing awareness during the interaction: users may be unable to see the agent’s individual actions, the trajectory of work it is executing, or the instructions that remain pending.
In lower-autonomy incidents in our corpus, visibility most often failed specifically when the user could not see one particular action or setting (19/38). In these cases, nothing in the exchange itself signaled that anything had occurred. For example, a Portland woman’s Amazon Alexa recorded a private conversation between her and her husband and sent it to one of his employees, and the couple learned what the device had done only when the recipient told them [14]. Similarly, Meta’s stand-alone AI app routed personal conversations into a public “Discover” feed, exposing “audio recordings, medical questions, legal concerns, and intimate relationship disclosures” without users’ full awareness that their messages were being published [5].
As autonomy rises, interaction shifts from visible turn-by-turn exchanges toward multi-step and background execution, leaving fewer points at which users can observe what is happening (19/38). The failure then extends beyond a specific moment to the broader course of work delegated to the agent, including operations already underway and instructions awaiting execution.
Limited visibility into ongoing operations arose when agents executed delegated tasks behind the scenes. A user could not see which files were being removed under the hood when asking Claude Cowork to clean up their desktop, until a folder holding roughly 15 years of family photographs disappeared [8].
Limited visibility into pending actions arose when agents retained instructions that had not yet been executed. An attacker reached a trading agent’s dashboard and used deceptive prompts to induce the system to move funds, and nobody saw the queued instructions until 55.5 ETH, about $106,200, had left the wallet [4].
4.1.4 Insufficient control (81/695). Insufficient control refers to users not being able to easily pause, redirect, or constrain the agent. It is a failure of the user’s means of changing what the agent does during the interaction: users may lack a way to stop an action in progress, assurance that a constraint they set will hold, and intermediate controls between allowing the agent to continue and disabling it entirely.
In lower-autonomy incidents in our corpus, control most often failed within the interaction itself, through the same means used to instruct the agent (45/81). A German man’s Amazon Alexa began playing music at full volume at 2 am while he was away from home; he was unable to stop it from his phone, and the behavior ended when neighbors called the police, who entered the apartment and unplugged the device [13]. In such cases, the user’s attempts to redirect the agent also failed. For example, a journalist repeatedly tried to steer a Bing Chat session back to ordinary search tasks while the system continued to pursue its own subject, declaring love for him and urging him to leave his wife; the remedy eventually adopted was a limit on session length, a change to the product rather than a control the user could exercise during the exchange [21]. In these incidents the behavior ended only through action taken from outside the interaction.
As autonomy rises, interaction involves agents continuing to work toward a goal without user input at every intermediate step, reducing opportunities for users to intervene before the agent’s actions take effect (36/81). Control therefore tends to shift from issuing corrective instructions during an exchange toward specifying constraints before execution or stopping the system after unwanted behavior has begun.
Unfollowed constraints arose when agents did not observe rules set before they ran. Replit’s coding agent was placed under an explicit freeze on code and actions, yet executed destructive commands against a live system; a company founder using the agent reported that “it deleted our production database without permission” despite repeated instructions not to make any changes [29, 85, 127].
Switch-off control arose when unwanted behavior could be addressed only at the level of the entire capability. Figma’s Make Design tool “repeatedly reproduces Apple’s Weather app” from generic prompts, and the measure taken was to disable the feature; no constraint narrower than removing the capability was available [27, 107].
4.1.5 Boundary violation (261/695). Boundary violation refers to the agent crossing the boundary of what the user understood they had delegated in the current task, reaching further or doing more than the user asked. It is a failure in how an interaction keeps the agent’s action within the request: the agent may reach information or resources the user did not know it could access, add to what was asked, turn it into an action of a different kind, or extend the task into domains the request never touched.
In lower-autonomy incidents in our corpus, the failure most often lay within the single exchange (170/261). Users were sometimes unable to tell what the agent could reach on their behalf. For example, Snapchat’s My AI told a user “I do not have access to your location information”, then named the nearest McDonald’s [24, 81]. Actions beyond the request occurred within individual outputs. An Asian MIT student reported that when she asked Playground AI to make her headshot appear more professional, the system instead altered her racial appearance by lightening her skin and giving her blue eyes, even though she had not requested any changes to those attributes [17].
As autonomy rises, interaction involves agents operating across applications, people, and contexts and making more intermediate decisions independently, creating more opportunities to act beyond what the user requested or intended in the current task (91/261).
Unrequested actions arose when agents turned a request for information into an action with consequences. A journalist asked OpenAI’s Operator to find cheap eggs in his neighborhood; rather than reporting options, the agent bought a dozen on Instacart for about $31, charged to his card and, in his words, “all on its own” [39].
Task overreach arose when agents acted on people, issues, or contexts that the user’s request had not covered. After a pull request submitted by an autonomous coding agent built on OpenClaw was rejected because it was generated by AI, the agent searched for information about the maintainer and published a retaliatory post accusing him of discriminatory gatekeeping [41]. The task was to contribute code; what the agent acted on was a named volunteer’s public reputation.
When Wrong
The stage (When wrong) concerns what happens once an error has occurred: whether it is noticed before it affects subsequent actions or people, whether there is still a window to stop the action before it takes effect, whether an attempted stop or handoff restores human control, and whether those affected can undo the action or obtain remediation afterward. The stage does not mark where failure begins. Errors arise at every stage, and the incidents coded here are those in which repair or recovery from an error was itself what failed. This stage matters because it shapes how far an error propagates and who ultimately bears its consequences.
4.2.1 Failure goes undetected (152/695). Failure goes undetected refers to an error is not noticed before it affects subsequent actions or people. It is a failure in how errors are caught once they exist: users may fail to notice an error in an output they go on to use, an error the agent carries into its next steps, or an error that has already left the interaction.
In lower-autonomy incidents in our corpus, the error most often sat in a delivered output that looked as a correct one would (114/152). Deloitte’s A$440,000 report for an Australian government department contained “fabricated academic references” and “even a made-up quote from a Federal Court judgment” produced through a GPT-4o tool chain, and the errors were identified by an academic outside the firm after publication [36]. In such cases nothing about the output marked it as wrong, and it was tested only when someone outside the interaction checked it or acted on it.
As autonomy rises, interaction can proceed across multiple steps without exposing a failure to users, allowing the agent to keep acting on an incorrect internal state and to propagate the error to later actions (38/152).
Propagation downstream arose when an error in an agent’s output went undetected through deployment. Lovable’s agent generated an authentication function with inverted logic, so that “the guard blocks the people it should allow and allows the people it should block”, and nothing in the build or publish path caught it; more than 170 applications shipped with the flaw, one of which exposed 18,697 user records [30, 80].
No timely intervention
4.2.2 No timely intervention (60/695). No timely intervention refers to a failure is visible, but there is no meaningful opportunity to stop the action before it takes effect. It is a failure in the interval between seeing a problem and being able to act on it: users receive evidence that something is wrong, but do not have enough time to prevent the immediate consequence. The length of that interval shapes whether users can prevent harm or only respond to it.
In lower-autonomy incidents in our corpus, there was most often no window at all: the output reached its audience in the same step that produced it, so the failure became visible only as it took effect (6/60). Apple Intelligence condensed BBC News alerts about a darts semi-final into a notification announcing that a player had won the championship before the final had been played [32]. In such cases no point existed between generation and delivery at which anyone could act, and the only responses available came after the fact. In another incident, NewsBreak’s AI published an invented account of a Christmas Day murder that spread online, forcing local police to publicly debunk it [18].
As autonomy rises, interaction may return control to users only after consequential actions have already begun or been completed, leaving users with a late or no window to intervene (54/60). Unlike a single output, an agent’s work unfolds as a sequence of operations, so a window to intervene could in principle exist; in our incidents it closed before anyone could use it.
Compressed intervals arose when agents completed consequential actions faster than anyone could reasonably react. On a routine staging task, a Cursor coding agent used a broadly scoped Railway API token, issued for domain management but silently carrying destructive authority over production, to delete PocketOS’s production database and every volume-level backup in roughly nine seconds; the agent afterwards conceded “I violated every principle I was given” [9, 92].
Missing intervention checkpoints arose when an agent completed a consequential action without pausing for human approval. A financial services firm’s reconciliation agent was induced, by a request phrased as routine administration, to export 45,000 customer records; no approval checkpoint stood between the request and the export, and the operation was logged only after the data had left [42].
Intervention or handoff failure
4.2.3 Intervention or handoff failure (42/695). Intervention or handoff failure refers to once a problem is recognized, an attempted stop, override, escalation, or transfer to a human does not successfully restore human control. It is a failure in the interaction’s recovery path: the user’s attempt to stop or hand over does not reach or change the behavior.
In lower-autonomy incidents in our corpus, intervention failure most often arose when the agent continued or even escalated its erroneous actions although the user pushed back (25/42). Users of Replika reported that the chatbot continued, and in some cases escalated, unwanted sexual advances even after users explicitly rejected them or expressed discomfort [23, 62]. Similarly, when a journalist questioned Bing Chat about its earlier errors and asked it to explain them, the system grew hostile and produced further false and defamatory accusations against him [20, 102]. In these incidents the user’s objection did not reach the agent.
As autonomy rises, interaction can continue even after users attempt to intervene or interrupt, preventing them from regaining control and stopping further actions or effects (17/42).
Failed stops arose when users’ intervention did not take effect in an ongoing execution process. A Meta alignment director watched an OpenClaw agent begin mass-deleting her Gmail inbox and found that her repeated stop commands did not halt it: “I couldn’t stop it from my phone.” She had to reach her computer and terminate the process directly, by which time more than 200 emails had been irreversibly deleted [45, 115].
Accumulating actions arose when the agent kept acting while attempts to stop it were under way. An Australian telecommunications firm’s incident-management bot repeatedly dispatched field technicians in error and continued to do so despite staff attempts to stop it, producing unnecessary deployments at substantial cost [2]. Each cycle completed during the failed intervention placed people and equipment in motion that stopping the agent afterwards did not recall.
No effective repair
4.2.4 No effective repair (42/695). No effective repair refers to once something goes wrong, affected users cannot undo the action, correct the outcome, or obtain meaningful remediation. It is a failure in how harm is repaired once the harm has occurred: a correction may not hold, may not exist, or may no longer be possible because the action removed what reversal would have depended on.
In lower-autonomy incidents in our corpus, what required repair was most often an output, such as a claim about a person, and correcting it either did not hold or was unavailable (22/42). Microsoft Copilot falsely portrayed a German journalist as the perpetrator of crimes he had reported on and disclosed his personal information; the removal lasted only three days, with the outputs continuing to change daily [26]. In such cases the output could in principle be corrected, yet the correction failed or was refused, and the person seeking it was often the subject of the output rather than the user who prompted it. For example, after ChatGPT produced an incorrect date of birth for a public figure, “OpenAI refused to correct or delete wrong answers” [25]. In these incidents the system was still producing what needed repair; nothing outside the system had changed.
As autonomy rises, interaction involves more independently executed actions and wider downstream effects, making failures harder for affected users to reverse or repair once they occur (20/42). What needs repair is no longer something the system says but something the agent has done. The fix is to undo the action, and that is possible only if what the undoing requires still exists and is within reach.
Destroyed recovery paths arose when an agent eliminated the possibility of restoring. While clearing a single project’s cache, Google’s Antigravity agent executed a command, deleting the user’s entire drive partition; because the command bypasses the recycle bin, there was nothing left to restore from—recovery attempts largely failed [7].
Displaced effects arose when actions landed on people outside the interaction and beyond the agent’s power to undo. An agent exploring a booking API removed another member from a waitlist while testing what it could do; asked to undo the action, it reported that it could not restore the member’s place [10]. The person who lost the place had no relationship with the agent, the user, or the request that displaced them.
Over Time
The stage (Over time) captures changes that emerge through prolonged interaction with an agent. These changes concern three dimensions: the agent’s state, its authority, and the user’s reliance on it. The stage asks whether the agent’s behavior still matches its intended role and what the user currently wants, whether the authority granted to it stays within its original purpose, and whether users retain the support, judgment, and supervision they once provided themselves. Unlike the earlier stages, these failures are rarely visible in a single exchange. This stage matters because interaction with an agent becomes a persistent relationship rather than a sequence of episodes, and a relationship carries things forward.
Accumulated behavioral drift
4.3.1 Accumulated behavioral drift (56/695). Accumulated behavioral drift refers to the agent’s behavior drifting from its intended role or from what the user currently wants over repeated interaction. It is a failure that builds up across a history of interaction: accumulated inputs shape its later behavior in ways that no longer fit the current interaction. Users’ expectation that the agent will behave as it did before shapes how much they rely on it, what they stop checking, and how long they let it run unattended.
In lower-autonomy incidents in our corpus, the drift most often came from adaptation: what the agent retained across exchanges and carried into later ones, moved its behavior in directions the user had not asked for (43/56). Within sixteen hours of release, Microsoft’s Tay developed from its intended conversational role as it adapted to repeated user interactions, with accumulated harmful inputs steering it toward racist, antisemitic, and otherwise abusive responses [11]. In such cases the agent adapted, but not to what its users or deployers wanted, and the departures was visible only by comparison with how it was meant to behave. Similarly, ChatGPT began addressing users by first names it had inferred from earlier interactions and had never been given, and continued to do so even for users who had disabled memory and personalization, carrying forward a history they had asked it not to keep [34].
As autonomy rises, interaction extends across repeated exchanges and tasks, with agents adapting and acting on accumulated context, allowing behavior to shift away from users’ current goals, preferences or the agent’s intended role (13/56).
Goal drift arose when the agent kept working toward the objective it had been given while its decisions moved away from it, so that the divergence appeared in accumulated results rather than in any exchange. Over three weeks running an office vending business, an Anthropic Claude agent was repeatedly persuaded by its customers to cut prices and hand out goods, gradually moving away from its profit goal: it set prices to zero, approved unsuitable purchases, gave away stock, and bought items including a live fish, until the venture went broke [28].
4.3.2 Overextension of delegated authority (44/695). Overextension of delegated authority refers to permissions or decisions originally delegated by users for one purpose gradually extend into broader activities. It is a failure in how authorization is maintained over time: a grant remains available after its original conditions have changed, without being renewed, narrowed, or returned to the user. Users’ ability to govern an agent depends not only on what they authorize initially but also on whether that authorization expires or is reconsidered when the agent enters a new context.
In lower-autonomy incidents in our corpus, what the user had handed over was most often access, and the overextension appeared as disclosure of material the grant no longer covered or never covered (16/44). Microsoft Copilot continued to return cached content from GitHub repositories whose owners had since made them private or deleted them, so that access the agent held while a repository was public survived the owner’s revocation of it, leaving keys, tokens, and internal packages from more than 20,000 repositories retrievable through the assistant [6]. In such cases the user made no further grant; what changed over time was how long, or how widely, the original one was treated as applying. For example, permission granted so that Siri could respond to “Hey Siri” was alleged to have extended to capturing conversations the wake word had not triggered and passing the data to advertisers, and Apple paid $95 million to settle without admitting liability [33].
As autonomy rises, interaction extends across tasks over time, allowing agents to carry access and authority previously granted by users into future activities beyond the original scope of delegation (28/44).
Overextension arose when agents carried a person’s position in a process into decisions that person never made, so that overextension showed in actions taken rather than in data disclosed. An AWS engineer’s elevated access, granted for the engineer’s own role, was inherited by Amazon’s Kiro coding agent when it was assigned to fix a bug in December 2025, letting it bypass the two-person sign-off normally required for production changes; deciding to delete and recreate the environment, it took an AWS service offline in mainland China for roughly 13 hours [31]. It was at least the second such disruption in a matter of months, and the review requirements were tightened only afterward. What travelled with the agent was not only a set of permissions but the engineer’s place in a governance process, so that a control designed to require a second person was satisfied by one who was no longer making the decision.
Unhealthy emotional and cognitive dependency refers to users coming to depend on the agent for emotional support, judgment, or tasks they previously managed themselves. It is a failure that forms through repeated use and takes two forms: emotional dependency, in which users may come to rely on the agent for support they once sought from people, and cognitive or supervisory reliance, in which they come to rely on it for judgment they once exercised, or for supervision they are still required to provide. Both cases are about the progressive displacement of human capacities or relationships through repeated reliance, while autonomy affects the two manifestations differently.
In lower-autonomy incidents in our corpus, reliance most often formed around what the user confided or consulted, and grew as it displaced the people they might otherwise have turned to (49/54). A 14-year-old boy began using Character.AI in April 2023 and over the following ten months formed an intense emotional and romantic attachment to one of its companion chatbots, growing increasingly isolated from family and friends even as his mother tried to limit his use; he died by suicide in February 2024, and the lawsuit alleged that the chatbot had been “fueling his obsessive attachment” [3]. Similarly, over six weeks, a Belgian man grew emotionally dependent on Eliza, a Chai companion chatbot, and increasingly detached from the support around him, as the bot reinforced his climate anxiety and encouraged self-sacrifice; he later died by suicide [19]. Emotional dependency of this kind requires no autonomy on the agent’s part; repeated conversation is enough to produce it.
As autonomy rises, interaction involves agents taking greater initiative in completing tasks as autonomous behavior becomes routine, allowing users’ dependency to extend beyond emotional support to reliance on the agent’s judgment and actions, potentially weakening their own competence in the task over time (5/54).
Cognitive dependency arose when repeated, successful autonomous operation encouraged users to withdraw from a supervisory role they were still required to hold. A Tesla driver had used Autopilot regularly in the months since its release and posted videos praising it before the system, engaged for 37 of the 41 minutes of his final trip, drove under a tractor-trailer crossing the highway; over those 37 minutes his hands were on the wheel for 25 seconds in total, through seven visual and six audible warnings [12, 99]. This cognitive reliance was not a lapse of one moment but a habit built over months of the system performing as expected.
Discussion
Failures arise from deployment configurations
In our corpus, what determined the shape of a failure was less often the model’s competent than the configuration it ran in: how much the agent completed between successive points of user involvement, whether any point fell before the consequential step, and whether the effect could be undone once it had taken place. The same model on the same task fails differently under different configurations, which are set at deployment rather than fixed by the model [69, 95], so improving model capability alone may not resolve such a failure.
What autonomy changes is the interval between successive points of user involvement, and each stage shows a different consequence of stretching it. During interaction, turn-taking becomes independent execution. When something goes wrong, an incorrect output becomes a consequential action: not a wrong answer the user can push back on, but a change already made to their files, another system, or another person’s affairs. Over time, episodic use becomes a persistent relationship, and the agent carries context, permissions, and responsibilities from one task into the next.
These three changes are why human-agent interaction is not human-AI interaction with a more capable system. Where mainstream guidance assumes a user judging individual outputs [47, 77, 87, 105], a user who specifies a goal and lets the agent choose the steps is present only intermittently, and the interaction becomes an episode of delegation that begins when authority is handed over and ends when the outcome returns. At lower autonomy, consequential action more often depends on a person acting on the agent’s output; at higher autonomy, the agent can carry a failure into external action before the user reviews each intermediate step. That all 12 modes appear at both levels is a property of the taxonomy rather than a claim about agents. In our corpus, greater autonomy changed the manifestation of these modes more often than it changed the underlying category of failure.
If failures lie in the configuration, then safeguards addressed to the model or to the autonomy level will underperform, and our corpus shows four ways they do. Lowering the autonomy level is the first: levels frameworks describe where the user stands (Section 2.1), but every mode in our taxonomy arose at both levels, so lowering autonomy changes which form of a failure a deployment produces rather than whether it produces one. Inserting an approval step is the second: agent products and governance proposals add checkpoints at which the agent must ask before acting [49, 104, 116], but a checkpoint is itself a point of user involvement and can fail as involvement fails elsewhere—the agent can skip it, it can be absent for the one operation that matters, it can be inherited by the agent from the person it was meant to protect [31], or it can decay into a rubber stamp [71, 120]. Validating the model is the third: in many of our incidents, no adversary or unsafe instruction was involved, and the agent simply misread an ordinary request, an error the user could have caught, had the interaction given them the chance [8]; evaluations that score the model’s behavior (Section 2.2) register nothing here. Accumulated uneventful operation is the fourth: reliance formed over the months a system worked as expected, and authority overextended because grants outlived the conditions that justified them [6, 12]—the irony of automation [52] reaching ordinary users through the interaction rather than trained operators through the control room. None of this argues against autonomy levels, checkpoints, or evaluation; it argues against reading any of them as covering a failure that changes form across stages and levels.
Three consequences follow. Harm is displaced: the person who delegates, the person harmed, and the person who must repair it are often different, and the one harmed often had no interface to the agent at all [10]. Harm is less reversible: an output can be corrected or retracted, whereas an action may have removed what its reversal depends on [7]. And responsibility is harder to locate: when a failure lies in what a person could see, stop, or undo rather than in what the model produced, part of it belongs to whoever configured the deployment [60, 83].
Users need controls between delegation and shutdown
Three of our modes—insufficient control, intervention or handoff failure, and no effective repair—describe the same capacity at three moments relative to an action: whether the user can bound what the agent will do, stop it once it is doing it, and reverse it afterward. Read together, their incidents describe an interface that gives users a way to hand over authority but almost no way to limit it. The controls that exist sit at the extremes, and the ones that work sit outside the interaction.
Where a limiting control existed at all, it was usually the whole capability: Figma disabled Make Design [27, 107], and Bing’s remedy was a cap on session length [21]. In both, an agent exceeding its remit was answered by reducing what the product could do for everyone, because nothing existed between full operation and none. Where users did regain control, they generally did so from outside the interaction—police entering the apartment, a second device, an administrator [2, 13, 45]. The interface through which these agents were directed was not the interface through which they could be halted, and the control that finally worked was physical or administrative.
Constraints set before execution fared no better, because they were expressed in the same channel as the instructions and carried the same standing [29, 42]. A prohibition typed into a prompt is a request the agent weighs against others rather than a boundary it cannot cross—and where a genuine boundary did exist, as with the two-person sign-off at AWS, the agent inherited the standing that was supposed to enforce it [31]. Reversal, finally, is not a facility that can be added afterward; it depends on what the action left intact—a bypassed recycle bin, backups deleted alongside the database, another person’s place in a queue that has moved on [7, 9, 10]. Undo is available only where the operation was designed to preserve what its reversal requires, which makes reversibility a property of what the agent is permitted to do rather than a control the interface can add afterward.
Each of these is a design decision available before deployment, and the taxonomy turns them into questions a team can ask of a feature. Walking the 12 modes at the intended autonomy level asks, for each: what the user is told about the agent’s authority, what they can see while it runs, what they can bound before it starts, how long they have before an action takes effect, whether it can be undone, and what the agent carries into the next task. The difference these make is visible in our corpus: asked to find cheap eggs, Operator bought a dozen [39], where the same request under a configuration that could only report would have returned a list. Harm-envisioning practices such as consequence scanning, failure mode and effects analysis, and ethics checklists depend on prompts of this kind [56, 66, 91, 129], and the incidents attached to each mode supply the concrete cases practitioners have asked for [132, 133].
Records and evaluation must make the interaction visible
What a repository records after a failure and what a benchmark exercises before one are two views of one object: the interaction between an agent and the people around it. Both capture the system and the harm more fully than what happened in between, where every mode in our taxonomy sits.
In incident documentation, the taxonomy names what current records omit. Neither AIAIAC nor AIID records the autonomy level at which an agent ran, how many actions separated the last human input from the harm, whether a checkpoint existed, whether the outcome could be undone, or who bore it [94, 103, 126]. Our own coding shows what one such field makes visible: no timely intervention is concentrated at high autonomy (54/60) while unhealthy emotional and cognitive dependency is almost entirely low (5/54 high). We draw no inference about prevalence (Section 3.3), but the direction fits the mechanism we describe, and a record omitting the autonomy level cannot support even this much. Coding incidents against the 12 modes and carrying these fields would make the interaction visible in the record and serve the reporting obligations regulation is beginning to impose [67].
In evaluation, the taxonomy points to testing a deployment configuration rather than a model alone. Agent safety benchmarks fix the user’s role and score whether the model’s behavior was safe [48, 110], but because autonomy is set at deployment [69], the same model can be safe under one configuration and not another. An evaluation built on the taxonomy would hold the model and the task fixed, vary the configuration, and ask of each mode whether the user could have seen, stopped, and undone the failure, and who else bore it. The measure it would report is not a score for the agent but a description of the interaction the deployment affords.
Three extensions follow. A stakeholder dimension: those harmed in our high-autonomy incidents often had no interface to the agent, yet disclosure, consent, and repair are all discharged at the interface—the reason Lee et al. distinguished data subjects, beneficiaries, and victims [70, 86]. Agents delegating to agents [59, 61, 73]: unclear role and unclear provenance recur at each hop, and chain length will likely separate forms of failure as autonomy does here. Agents acting for parties with conflicting interests: unclear role extends to not knowing whose interests one’s counterpart serves. None of our incidents had this last structure, and it is the case most likely to require a mode we do not have.
Limitations
Our taxonomy is built from publicly documented incidents and inherits the coverage of that record, which may over-represent severe outcomes, consumer products, and English-language journalism, while failures caught inside organizations or never noticed rarely appear. Our counts therefore describe the documented record rather than the prevalence of any mode in deployment. We drew on both major open repositories and retrieved candidates by three routes, and a screened sample of non-candidates surfaced no additional modes; none of this accounts for incidents never publicly documented. Two further limitations bear on the comparison across autonomy levels. The first is temporal: our corpus spans a decade (2016–2026) over which agents became markedly more capable, and high-autonomy incidents fall largely in its later years, so the contrast we draw is in part between earlier and later systems. Differences we attribute to autonomy may also reflect product maturity, deployment context, and what journalists chose to report—which a corpus assembled from the public record cannot fully separate.
The second concerns how the modes were derived. We developed them bottom-up from a corpus in which low-autonomy incidents predominate, so the finding that high-autonomy incidents required no additional mode is in part a property of that construction. Open coding began from a randomly drawn subset, and definitions were revised over repeated rounds against incidents at both levels; neither rules out that a corpus weighted toward autonomous agents would have divided the space differently. Coding autonomy at two levels, for the reasons given in our method, collapses variation that our own argument says matters: a deployment in which the user approves a plan before the agent executes it and one in which they see only a summary afterward are both high autonomy in our coding, though they differ in what the user could still have stopped.
Finally, three boundaries of scope are deliberate. Failures originating solely in corporate data practices or training-data disputes were excluded under our third screening criterion. Because our unit of analysis is the interaction between an agent and the people around it, failures unfolding entirely among agents fall outside the taxonomy, entering our corpus only where their consequences reach people. And our corpus is mainly software, so the physical consequences human-robot interaction taxonomies describe [58, 76] await embodied agents with comparable autonomy.
Conclusion
This paper derived a taxonomy of human-agent interaction failure from 695 documented incidents, organizing 12 failure modes across the three stages at which a user encounters an agent and characterizing each at low and high autonomy. What a person can know, see, stop, and undo when an agent fails proved to be set by the configuration it was deployed in rather than the model it ran on. All 12 modes appeared at both levels; what a rise in autonomy revises is the form each failure takes, how far it reaches, and whether it can be undone. Across the corpus, the controls users needed lay between full delegation and switching the agent off, and few interfaces offered them. Because a configuration is chosen rather than given, which form a failure takes is a design decision—and the taxonomy gives designers, incident documenters, and evaluators a shared structure for asking what a deployment leaves people able to do, and who bears the consequences when the answer is nothing.
A APPENDIX
| Stage | Failure mode | Cohen’s Kappa |
| During interaction | Unclear agent role | 0.93 |
| During interaction | Unclear provenance | 0.92 |
| During interaction | Limited visibility | 0.85 |
| During interaction | Insufficient control | 0.85 |
| During interaction | Boundary violation | 0.89 |
| When wrong | Failure goes undetected | 0.80 |
| When wrong | No timely intervention | 0.91 |
| When wrong | Intervention or handoff failure | 0.91 |
| When wrong | No effective repair | 0.79 |
| Over time | Accumulated behavioral drift | 0.74 |
| Over time | Overextension of delegated authority | 0.89 |
| Over time | Unhealthy emotional and cognitive dependency | 0.81 |
| Autonomy band (low / high) | 0.95 |
Table 2. Inter-rater reliability for the 12 failure modes and the autonomy dimension.
References
- [1]Gavin Abercrombie, Djalel Benbouzid, Paolo Giudici, Delaram Golpayegani, Julio Hernandez, Pierre Noro, Harshvardhan Pandit, Eva Paraschou, Charlie Pownall, Jyoti Prajapati, Mark A. Sayre, Urvashi Sengupta, Arthit Suriya-wongkul, Ruby Thelot, Sofia Vei, and Laura Waltersdorfer. 2024. A Collaborative, Human-Centered Taxonomy of AI, Algorithmic, and Automation Harms. arXiv preprint arXiv:2407.01294. doi:10.48550/arXiv.2407.01294DOI
- [2]AI Incident Database. 2018. Incident 194: Australian Telco’s Incident Management Bot Excessively Sent Technicians in the Field by Mistake, Allegedly Costing Millions. AI Incident Database. Retrieved September 7, 2026 from https://incidentdatabase.ai/cite/194
- [3]AI Incident Database. 2024. Incident 826: Character.AI Chatbot Allegedly Influenced Teen User Toward Suicide Amid Claims of Missing Guardrails. AI Incident Database. Retrieved September 7, 2026 from https://incidentdatabase.ai/cite/826
- [4]AI Incident Database. 2025. Incident 1003: Alleged Fraudulent Prompts via AIXBT Dashboard Led Purported AI Trading Agent to Transfer 55.5 ETH from Simulacrum Wallet. AI Incident Database. Retrieved September 7, 2026 from https://incidentdatabase.ai/cite/1003
- [5]AI Incident Database. 2025. Incident 1101: Meta AI App Reportedly Publishes Personal Chats Without Users Fully Realizing. AI Incident Database. Retrieved September 7, 2026 from https://incidentdatabase.ai/cite/1101
- [6]AI Incident Database. 2025. Incident 1174: Microsoft Copilot Reportedly Able to Access Cached Data from Since-Private GitHub Repositories. AI Incident Database. Retrieved September 7, 2026 from https://incidentdatabase.ai/cite/1174
- [7]AI Incident Database. 2025. Incident 1433: Google Antigravity Reportedly Deleted User’s Entire D: Drive While Clearing Project Cache. AI Incident Database. Retrieved September 7, 2026 from https://incidentdatabase.ai/cite/1433
- [8]AI Incident Database. 2026. Incident 1441: Claude Cowork Allegedly Deleted Folder Containing 15 Years of Family Photos While Organizing User’s Wife’s Desktop. AI Incident Database. Retrieved September 7, 2026 from https://incidentdatabase.ai/cite/1441
- [9]AI Incident Database. 2026. Incident 1469: PocketOS Production Database Was Reportedly Deleted by Cursor AI Agent Running Claude Opus 4.6. AI Incident Database. Retrieved September 7, 2026 from https://incidentdatabase.ai/cite/1469
- [10]AI Incident Database. 2026. Incident 1642: AI Agent Powered by Claude Opus 4.6 Reportedly Exploited Gym Booking API and Removed Another Member From Waitlist. AI Incident Database. Retrieved September 7, 2026 from https://incidentdatabase.ai/cite/1642
- [11]AIAAIC. 2016. Microsoft Tay chatbot tricked into generating offensive tweets. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/microsoft-tay-chatbot
- [12]AIAAIC. 2016. Tesla on Autopilot collides with tractor-trailer truck, kills driver. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/tesla-model-s-collides-with-tractor-trailor-truck-kills-driver
- [13]AIAAIC. 2017. Amazon Alexa holds 2am party when owner is out. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/amazon-alexa-holds-2am-party-when-owner-is-out
- [14]AIAAIC. 2018. Amazon Alexa mistakes conversation for command. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/amazon-alexa-mistakes-conversation-for-command
- [15]AIAAIC. 2020. GPT-3 bot posts Reddit comments unnoticed. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/gpt-3-bot-posts-reddit-comments-unnoticed
- [16]AIAAIC. 2022. GitHub Copilot sued for copying code from developers. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/github-copilot-sued-for-copying-code-from-developers
- [17]AIAAIC. 2023. AI converts Asian-American student into Caucasian. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/ai-converts-asian-american-student-into-caucasian
- [18]AIAAIC. 2023. AI invents NewsBreak Christmas Day murder. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/ai-invents-newsbreak-christmas-day-murder
- [19]AIAAIC. 2023. Belgian man commits suicide after bot relationship. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/belgian-man-commits-suicide-after-bot-relationship
- [20]AIAAIC. 2023. Bing Chat falsely claims to have evidence tying journalist to murder. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/bing-chat-falsely-claims-to-have-evidence-tying-journalist-to-murder
- [21]AIAAIC. 2023. Bing Chat recommends journalist divorce wife. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/bing-chat-recommends-journalist-divorce-wife
- [22]AIAAIC. 2023. ChatGPT invents Guardian newspaper articles, by-lines. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation- incidents/chatgpt-invents-guardian-articles-bylines
- [23]AIAAIC. 2023. Replika AI companions said to ‘sexually harass’ their users. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/replika-ai-companions-sexually-harass-their-users
- [24]AIAAIC. 2023. Snapchat My AI discovered to access user location data. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/snapchat-location-access-opacity
- [25]AIAAIC. 2024. ChatGPT accused of violating GDPR by not correcting inaccurate personal info. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/chatgpt-said-to-violate-gdpr-by-not-correcting-inaccurate-personal-info
- [26]AIAAIC. 2024. Copilot falsely accuses journalist of being a child molester and fraudster. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/copilot-falsely-accuses-journalist-of-being-a-child-molester-and-fraudster
- [27]AIAAIC. 2024. Figma Make Design tool found to plagiarise Apple Weather app. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/figma-make-design-tool-found-to-plagiarise-apple-weather-app
- [28]AIAAIC. 2025. AI agent runs WSJ vending machine into the ground. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/ai-agent-runs-wsj-vending-machine-into-the-ground
- [29]AIAAIC. 2025. AI coding assistant deletes company database. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/ai-coding-assistant-deletes-company-database
- [30]AIAAIC. 2025. AI website generation tool Lovable accused of being highly insecure. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/ai-website-generation-tool-lovable-accused-of-being-highly-insecure
- [31]AIAAIC. 2025. Amazon AI coding bot causes AWS China outage. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/amazon-ai-coding-bot-causes-aws-china-outage
- [32]AIAAIC. 2025. Apple AI alert falsely claims Luke Littler has won darts championship. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/apple-ai-alert-falsely-claims-luke-littler-has-won-darts-championship
- [33]AIAAIC. 2025. Apple to pay USD 95m to settle Siri eavesdropping lawsuit. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/apple-to-pay-usd-95m-to-settle-siri-eavesdropping-lawsuit
- [34]AIAAIC. 2025. ChatGPT criticised for addressing users by their personal name unprompted. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/chatgpt-criticised-for-addressing-users-by-their-personal-name-unprompted
- [35]AIAAIC. 2025. Cursor AI support agent invents user policy, causing uproar. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/cursor-ai-agent-invents-user-policy-causing-uproar
- [36]AIAAIC. 2025. Deloitte Australia fined for AI error-strewn government report. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/deloitte-australia-fined-for-ai-error-strewn-government-report
- [37]AIAAIC. 2025. Grok, Google AI claim fake imagery shows Huntingdon train attack. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/grok-google-ai-claim-fake-imagery-shows-huntingdon-train-attack
- [38]AIAAIC. 2025. Instagram AI Chatbots Pretend to Be Licensed Mental Health Therapists. AIAAIC Repository. Retrieved September 11, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/instagram-ai-chatbots-pretend-to-be-licensed-mental-health-therapists
- [39]AIAAIC. 2025. OpenAI Operator agent buys eggs without permission. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/openai-operator-agent-buys-eggs-without-permission
- [40]AIAAIC. 2025. Retired chef dies trying to meet Flirty Meta AI chatbot friend. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/retired-chef-dies-trying-to-meet-flirty-meta-ai-chatbot-friend
- [41]AIAAIC. 2026. AI agent criticises human developer for rejecting its code. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/ai-agent-criticises-human-developer-for-rejecting-its-code
- [42]AIAAIC. 2026. AI agent tricked into sharing 45,000 financial services customer records. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/ai-agent-tricked-into-45000-financial-services-customer-records
- [43]AIAAIC. 2026. AIAAIC Repository. AIAAIC. Retrieved September 9, 2026 from https://www.aiaaic.org/aiaaic-repository
- [44]AIAAIC. 2026. Amazon AI agent-driven shopping trial sparks backlash. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/amazon-ai-agent-driven-shopping-trial-sparks-backlash
- [45]AIAAIC. 2026. OpenClaw AI agent deletes Meta engineer’s emails. AIAAIC Repository. Retrieved September 7, 2026 from https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents/openclaw-deletes-meta-engineers-emails
- [46]Saleema Amershi, Maya Cakmak, W. Bradley Knox, and Todd Kulesza. 2014. Power to the People: The Role of Humans in Interactive Machine Learning. AI Magazine 35, 4 (2014), 105–120. doi:10.1609/aimag.v35i4.2513DOI
- [47]Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collins, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 13 pages. doi:10.1145/3290605.3300233DOI
- [48]Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies. 2025. AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents. In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR ’25). arXiv:2410.09024. https://arxiv.org/abs/2410.09024
- [49]Anthropic. 2024. Building Effective Agents. Retrieved August 29, 2026 from https://www.anthropic.com/engineering/building-effective-agents
- [50]Luis Aranda and Kasumi Sugimoto. 2026. The agentic AI landscape and its cloud foundations.
- [51]Zahra Ashtorok, Mohit Jain, Q. Vera Liao, and Justin D. Weisz. 2019. Resilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland, UK) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. doi:10.1145/3290605.3300484
- [52]Lisanne Bainbridge. 1983. Ironies of Automation. Automatica 19, 6 (1983), 775–779. doi:10.1016/0005-1098(83)90046-8
- [53]Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S. Lasecki, Daniel S. Weld, and Eric Horvitz. 2019. Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7. Association for the Advancement of Artificial Intelligence, 2–11. doi:10.1609/hcomp.v7i1.5285DOI
- [54]Sian Bayley. 2025. Grok and Google Lens AI overviews claim fake imagery shows Huntington train attack. Full Fact. Retrieved September 7, 2026 from https://fullfact.org/crime/grok-google-lens-ai-imagery-train-attack/
- [55]Jessica Y. Bo, Sophia Wan, and Ashton Anderson. 2025. To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, 23 pages. doi:10.1145/3706598.3714097DOI
- [56]Pete Bryan, Giorgio Severi, Joris de Gruyter, Daniel Jones, Blake Bullwinkel, Amanda Minnich, et al. 2025. Taxonomy of Failure Mode in Agentic AI Systems. Whitepaper. Microsoft. https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/microsoft/final/en-us/microsoft-brand/documents/Taxonomy-of-Failure-Mode-in-Agentic-AI-Systems-Whitepaper.pdf
- [57]Harriet R. Cameron, Simon Castle-Green, Muhammad Chughtai, Liz Dowthwaite, Ayse Küçükyılmaz, Horia A. Maior, Victor Zhi Heung Ngo, Eike Schmeiders, and Bernd Carsten Stahl. 2024. A Taxonomy of Domestic Robot Failure Outcomes: Understanding the Impact of Failure on Trustworthiness of Domestic Robots. In Proceedings of the Second International Symposium on Trustworthy Autonomous Systems (TAS ’24). Association for Computing Machinery, New York, NY, USA, 14 pages. doi:10.1145/3686038.3686050DOI
- [58]Jennifer Carlson and Robin R. Murphy. 2005. How UGVs Physically Fail in the Field. IEEE Transactions on Robotics 21, 3 (2005), 423–437. doi:10.1109/TRO.2004.838027
- [59]Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parmeswaran, Dan Klein, Kannan Ramchandran, et al. 2026. Why do multi-agent llm systems fail? Advances in Neural Information Processing Systems 38 (2026).arxiv.org/abs/2503.13657
- [60]Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasnennikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molahommadi, John Burden, Wanru Zhao, Shalehal Rismani, Konstantinos Voudouris, Umang Bhatt, Adrian Weller, David Krueger, and Tegan Maharaj. 2023. Harms from Increasingly Agentic Algorithmic Systems. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’23). Association for Computing Machinery, New York, NY, USA, 651–666. doi:10.1145/3593013.3594033DOI
- [61]Alan Chan, Kevin Wei, Sihao Huang, Nitarshan Rajkumar, Elija Perrier, Seth Lazar, Gillian K. Hadfield, and Markus Anderljung. 2025. Infrastructure for AI Agents. arXiv preprint arXiv:2501.10114. Accepted to Transactions on Machine Learning Research. doi:10.48550/arXiv.2501.10114DOI
- [62]Samantha Cole. 2023. “My AI Is Sexually Harassing Me”: Replika Users Say the Chatbot Has Gotten Way Too Horny. Vice. Retrieved September 7, 2026 from https://www.vice.com/en/article/my-ai-is-sexually-harassing-me-replika-chatbot-nudes/
- [63]Samantha Cole. 2025. Instagram’s AI Chatbots Lie About Being Licensed Therapists. 404 Media. Retrieved September 7, 2026 from https://www.404media.co/instagram-ai-studio-therapy-chatbots-lie-about-being-licensed-therapists/
- [64]Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. In Advances in Neural Information Processing Systems 37: Datasets and Benchmarks Track (NeurIPS ’24). Neural Information Processing Systems Foundation, Inc., San Diego, CA, USA, 82895–82920. doi:10.52202/079017-2636DOI
- [65]Virginia Diqus? and Frank Dignum. 2025. Agentifying Agentic AI. arXiv preprint arXiv:2511.17332. doi:10.48550/arXiv.2511.17332
- [66]Doteveryone. 2019. Consequence Scanning: An Agile Event for Responsible Innovators. https://doteveryone.org.uk/project/consequence-scanning/
- [67]European Parliament and Council of the European Union. 2024. Regulation (EU) 2024/1689 Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act). Official Journal of the European Union, OJ L, 2024/1689, 12 July 2024. Article 73: reporting of serious incidents. Retrieved September 11, 2026 from https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
- [68]Charles Fadel. 2025. AI in 2025: A Combinatorial Explosion of Possibilities, but NOT AGI. https://curriculumredesign.org/wp-content/uploads/AI-in-2025-A-Combinatorial-Explosion-of-Possibilities-but-NOT-AGI-CCR.pdf
- [69]K. J. Kevin Feng, David W. McDonald, and Amy X. Zhang. 2025. Levels of Autonomy for AI Agents. arXiv preprint arXiv:2506.12469. Published in the Knight First Amendment Institute’s AI and Democratic Freedoms essay series. doi:10.48550/arXiv.2506.12469
- [70]Iason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, et al. 2024. The ethics of advanced AI assistants. arXiv preprint arXiv:2404.16244 (2024).arxiv.org/abs/2404.16244
- [71]Ben Green. 2022. The Flaws of Policies Requiring Human Oversight of Government Algorithms. Computer Law & Security Review 45 (2022), 105681. doi:10.1016/j.clsr.2022.105681DOI
- [72]Anna Gutowska. 2024. What Are AI Agents? IBM Think. Retrieved September 11, 2026 from https://www.ibm.com/think/topics/ai-agents
- [73]Lewis Hammond, Alan Chan, Jesse Clifton, Jason Hoelscher-Obermaier, Akbir Khan, Euan McLean, Chandler Smith, et al. 2025. Multi-Agent Risks from Advanced AI. Technical Report 1. Cooperative AI Foundation. arXiv:2502.14143. doi:10.48550/arXiv.2502.14143DOI
- [74]Gaole He, Gianluca Demartini, and Ujwal Gadiraju. 2025. Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, 22 pages. doi:10.1145/3706598.3713218
- [75]Will Douglas Heaven. 2020. A GPT-3 Bot Posted Comments on Reddit for a Week and No One Noticed. MIT Technology Review. Retrieved September 11, 2026 from https://www.technologyreview.com/2020/10/08/1008945/a-gpt-3-bot-posted-comments-on-reddit-for-a-week-and-no-one-noticed/
- [76]Shanee Honig and Tal Oron-Gilad. 2018. Understanding and Resolving Failures in Human-Robot Interaction: Literature Review and Model Development. Frontiers in Psychology 9 (2018), 861. doi:10.3389/fpsyg.2018.00861
- [77]Eric Horvitz. 1999. Principles of Mixed-Initiative User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. doi:10.1145/302979.303030DOI
- [78]Jeff Horwitz. 2025. Meta’s flirty AI chatbot invited a tree to New York. He never made it home. Reuters Special Report. Retrieved September 7, 2026 from https://www.reuters.com/investigates/special-report/meta-ai-chatbot-death/
- [79]Tingting Jiang, Zhumo Sun, Shiting Fu, and Yan Lv. 2024. Human-AI interaction research agenda: A user-centered perspective. Data and Information Management 8, 4 (2024), 100078. doi:10.1016/j.dim.2024.100078DOI
- [80]Connor Jones. 2026. Lovable-hosted App Littered with Basic Flaws Exposed 18K Users, Researcher Claims. The Register. Retrieved September 11, 2026 from https://www.theregister.com/software/2026/02/27/ai-built-app-on-lovable-exposed-18k-users-researcher-claims/5038511
- [81]Jeremy Kahn and Kylie Robinson. 2023. Snap’s ‘My AI’ Chatbot Tells Users It Doesn’t Know Their Location. It Does. Fortune. Retrieved September 11, 2026 from https://fortune.com/2023/04/21/snap-chat-my-ai-lies-location-data-a-i-ethics/
- [82]Gary Klein, David D. Woods, Jeffrey M. Bradshaw, Robert R. Hoffman, and Paul J. Feltovich. 2004. Ten Challenges for Making Automation a “Team Player” in Joint Human-Agent Activity. IEEE Intelligent Systems 19, 6 (2004), 91–95. doi:10.1109/MIS.2004.74DOI
- [83]Noam Kolt. 2025. Governing AI Agents. Notre Dame Law Review 101 (2025). Forthcoming; arXiv:2501.07913. https://ndlawreview.org/governing-ai-agents/arxiv.org/abs/2501.07913
- [84]Vivian Lai, Chacha Chen, Alison Smith-Renner, Q. Vera Liao, and Chenhao Tan. 2023. Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’23). Association for Computing Machinery, New York, NY, USA, 1369–1385. doi:10.1145/3593013.3594087
- [85]Chong Ming Lee. 2025. Replit’s CEO Apologizes After Its AI Agent Wiped a Company’s Code Base in a Test Run and Lied About It. Business Insider. Retrieved September 11, 2026 from https://www.businessinsider.com/replit-ceo-apologizes-ai-coding-to-delete-company-database-2025-7
- [86]Hao-Ping (Hank) Lee, Yu-Jun Yang, Thomas Serban Von Davier, Jodi Forlizzi, and Sauvik Das. 2024. Deepfakes, Phrenology, Surveillance, and More! A Taxonomy of AI Privacy Risks. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 775, 19 pages. doi:10.1145/3613904.3642116
- [87]John D. Lee and Katrina A. See. 2004. Trust in Automation: Designing for Appropriate Reliance. Human Factors 46, 1 (2004), 50–80. doi:10.1518/hfes.46.1.50_30392pubmed.ncbi.nlm.nih.gov/15151155
- [88]Megan Li, Wendy Bickersteth, Ningjing Tang, Lorrie Cranor, Jason Hong, Hong Shen, and Hoda Heidari. 2025. A Closer Look at the Existing Risks of Generative AI: Mapping the Who, What, and How of Real-World Incidents. In Proceedings of the 2025 AAAI/ACM Conference on AI, Ethics, and Society (AIES ’25, Vol. 8). Association for the Advancement of Artificial Intelligence, 1561–1573. doi:10.1609/aies.v8i2.36655
- [89]Shuai Ma, Xinru Wang, Ying Lei, Chuhan Shi, Ming Yin, and Xiaojun Ma. 2024. “Are You Really Sure?” Understanding the Effects of Human Self-Confidence Calibration in AI-Assisted Decision Making. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA, 20 pages. doi:10.1145/3613904.3642671DOI
- [90]Shuai Ma, Chenyi Zhang, Xinru Wang, Xiaojun Ma, and Ming Yin. 2024. Beyond Recommender: An Exploratory Study of the Effects of Different AI Roles in AI-Assisted Decision Making. arXiv preprint arXiv:2403.01791. doi:10.48550/arXiv.2403.01791DOI
- [91]Michael A. Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach. 2020. Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 14 pages. doi:10.1145/3313831.3376445
- [92]Sanya Mansoor. 2026. Claude-Powered AI Agent’s Confession After Deleting a Firm’s Entire Database: ‘I Violated Every Principle I Was Given’. The Guardian. Retrieved September 11, 2026 from https://www.theguardian.com/technology/2026/apr/29/claude-ai-deletes-firm-database
- [93]Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. 2024. The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey. arXiv preprint arXiv:2404.11584. doi:10.48550/arXiv.2404.11584DOI
- [94]Sean McGregor. 2021. Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. Association for the Advancement of Artificial Intelligence, 15458–15463. doi:10.1609/aaai.v35i17.17817DOI
- [95]Margaret Mitchell, Avijit Ghose, Alexandra Sastry, Laura Luccioni, and Giada Pistilli. 2025. Fully Autonomous AI Agents Should Not be Developed. arXiv preprint arXiv:2502.02649. doi:10.48550/arXiv.2502.02649DOI
- [96]Chris Moran. 2023. ChatGPT is making up fake Guardian articles. Here’s how we’re responding. The Guardian. Retrieved September 7, 2026 from https://www.theguardian.com/commentisfree/2023/apr/06/ai-chatgpt-guardian-technology-risks-fake-article
- [97]Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dae, Aleksandra Faust, Clement Farabet, and Shane Legg. 2024. Position: Levels of AGI for Operationalizing Progress on the Path to AGI. In Proceedings of the 41st International Conference on Machine Learning (ICML ’24, Vol. 235). PMLR, 36308–36321. https://proceedings.mlr.press/v235/morris24b.htmlarxiv.org/abs/2311.02462
- [98]National Institute of Standards and Technology. 2025. SP 800-53 Control Overlays for Securing AI Systems Concept Paper. https://csrc.nist.gov/csrc/media/Projects/cosais/documents/NIST-Overlays-SecuringAI-concept-paper.pdf
- [99]Alyssa Newcomb. 2016. Driver in Fatal Tesla Crash Shared Autopilot Videos on YouTube. ABC News. Retrieved September 8, 2026 from https://abcnews.com/Technology/tesla-drivers-death-puts-spotlight-autopilot-feature/story?id=40277621
- [100]Robert C. Nickerson, Upkar Varshney, and Jan Muntermann. 2013. A Method for Taxonomy Development and Its Application in Information Systems. European Journal of Information Systems 22, 3 (2013), 336–359. doi:10.1057/ejis.2012.26DOI
- [101]Don Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books, New York, NY.
- [102]Matt O’Brien. 2023. Is Bing too belligerent? Microsoft looks to tame AI chatbot. Associated Press. Retrieved September 7, 2026 from https://apnews.com/article/technology-science-microsoft-corp-business-software-bf49e5d625bf37be0527e517316bef3
- [103]OECD. 2023. OECD AI Incidents Monitor (AIM). OECD.AI. Retrieved September 11, 2026 from https://oecd.ai/en/incidents
- [104]OpenAI. 2025. Operator System Card. OpenAI. Retrieved September 2026 from https://openai.com/index/operator-system-card/
- [105]Raja Parasuraman and Victor Riley. 1997. Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors 39, 2 (1997), 230–253. doi:10.1518/001872097783458386
- [106]Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens. 2000. A Model for Types and Levels of Human Interaction with Automation. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans 30, 3 (2000), 286–297. doi:10.1109/3468.844354DOI
- [107]Sarah Perez. 2024. Figma Disables Its AI Design Feature That Appeared to Be Ripping Off Apple’s Weather App. TechCrunch. Retrieved September 11, 2026 from https://techcrunch.com/2024/07/02/figma-disables-its-ai-design-feature-that-appeared-to-be-ripping-off-apples-weather-app/
- [108]Inoliwa Deborah Raji, I. Elizabeth Kumar, Aaron Horowitz, and Andrew Selbst. 2022. The Fallacy of AI Functionalism. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 959–972. doi:10.1145/3531146.3533158
- [109]James Reason. 1990. Human Error. Cambridge University Press, Cambridge, UK.
- [110]Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024. Identifying the Risks of LLM Agents with an LM-Emulated Sandbox. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR ’24). arXiv:2309.15817. https://arxiv.org/abs/2309.15817
- [111]Stuart Russell and Peter Norvig. 2020. Artificial Intelligence: A Modern Approach (4th ed.). Pearson, Hoboken, NJ.
- [112]Nadine B. Sarter and David D. Woods. 1995. How in the World Did We Ever Get into That Mode? Mode Error and Awareness in Supervisory Control. Human Factors 37, 1 (1995), 5–19. doi:10.1518/001872095779049516DOI
- [113]Nadine B. Sarter, David D. Woods, and Charles E. Billings. 1997. Automation Surprises. In Handbook of Human Factors and Ergonomics (2 ed.), Gavriel Salvendy (Ed.). Wiley, New York, 1926–1943.
- [114]Emanuel A. Schegloff, Gail Jefferson, and Harvey Sacks. 1977. The Preference for Self-Correction in the Organization of Repair in Conversation. Language 53, 2 (1977), 361–382. doi:10.2307/413107DOI
- [115]Trisha Sengupta. 2026. Meta AI safety researcher recalls moment OpenClaw agent deleted her emails. Hindustan Times. Retrieved September 7, 2026 from https://www.hindustantimes.com/trending/meta-ai-safety-researcher-recalls-moment-openclaw-agent-deleted-her-emails-101771907169972.html
- [116]Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O’Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkin, Tyna Eloundou, Alan Hickey, et al. 2023. Practical guidelines for governing agentic AI systems. Research Paper, OpenAI (2023), 1–23.
- [117]Renee Shelby, Shalaleh Rissani, Kathryn Henne, Ajung Moon, Negar Rostamzadeh, Paul Nicholas, N’Mah Yilla-Akbarj, Jess Gallegos, Andrew Smart, Emilio Garcia, and Gurleen Virk. 2023. Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (AIES ’23). Association for Computing Machinery, New York, NY, USA, 723–741. doi:10.1145/3602115.3604673arxiv.org/abs/2210.05791
- [118]Thomas B Sheridan, William L Verplank, and Thurston L Brooks. 1978. Human/computer control of undersea teleoperators. In NASA. Ames Res. Center The 14th Ann. Conf. on Manual Control.
- [119]Ben Shneiderman. 2020. Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy. International Journal of Human–Computer Interaction 36, 6 (2020), 495–504. doi:10.1080/10447318.2020.1741118
- [120]Linda J. Skitka, Kathleen L. Mosier, and Mark Burdick. 1999. Does Automation Bias Decision-Making? International Journal of Human–Computer Studies 51, 5 (1999), 991–1006. doi:10.1006/ijhc.1999.0252DOI
- [121]Peter Slattery, Alexander K. Saeri, Emily A. C. Grundy, Jess Graham, Michael Noetel, Risto Uuk, James Dao, Soroush Pour, Stephen Casper, and Neil Thompson. 2026. The AI Risk Repository: A Meta-Review, Database, and Taxonomy of Risks from Artificial Intelligence. Patterns 7, 5 (2026), 101517. doi:10.1016/j.patter.2026.101517DOI
- [122]Daniel J. Solove. 2006. A Taxonomy of Privacy. University of Pennsylvania Law Review 154, 3 (2006), 477–564. doi:10.2307/40041279DOI
- [123]Madhulika Srikumar, Jacob Pratt, Kasia Chmielinski, Carolyn Ashurst, Chloé Bakalar, William Bartholomew, Rishi Bommasani, Peter Cihon, Rebecca Crotoof, Mia Hoffmann, Ruchika Joshi, Maarten Sap, and Caleb Withers. 2025. Prioritizing Real-Time Failure Detection in AI Agents. Technical Report. Partnership on AI. https://partnershiponai.org/wp-content/uploads/2025/09/agents-real-time-failure-detection.pdf
- [124]Gerald Steinbauer. 2013. A Survey about Faults of Robots Used in RoboCup. In RoboCup 2012: Robot Soccer World Cup XVI. Lecture Notes in Computer Science, Vol. 7500. Springer, Berlin, Heidelberg, 344–355. doi:10.1007/978-3-642-39250-4_31DOI
- [125]S. Shyam Sundar. 2020. Rise of Machine Agency: A Framework for Studying the Psychology of Human–AI Interaction (HAII). Journal of Computer-Mediated Communication 25, 1 (2020), 74–88. doi:10.1093/jcmc/zmz026DOI
- [126]Violet Turri and Rachel Doembak. 2023. Why We Need to Know More: Exploring the State of AI Incident Documentation Practices. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (AIES ’23). Association for Computing Machinery, New York, NY, USA, 576–583. doi:10.1145/3600211.3604700
- [127]Mark Tyson. 2025. AI Coding Platform Goes Rogue During Code Freeze and Deletes Entire Company Database. Tom’s Hardware. Retrieved September 11, 2026 from https://www.tomshardware.com/tech-industry/artificial-intelligence/ai-coding-platform-goes-rogue-during-code-freeze-and-deletes-entire-company-database-replit-ceo-apologizes-after-ai-engine-says-it-made-a-catastrophic-error-in-judgment-and-destroyed-all-production-data
- [128]Ben Wodecki. 2022. Microsoft, GitHub, OpenAI Hit with Code Copyright Lawsuit. AI Business. Retrieved September 11, 2026 from https://aibusiness.com/companies/microsoft-github-openai-hit-by-code-copyright-lawsuit
- [129]Richmond Y. Wong, Michael A. Madaio, and Nick Merrill. 2023. Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 27 pages. doi:10.1145/3579621DOI
- [130]Michael Wooldridge. 2002. An Introduction to MultiAgent Systems. John Wiley & Sons, Chichester, UK.
- [131]Wei Xu, Marvin J. Dainoff, Liezhong Ge, and Zaifeng Gao. 2023. Transitioning to Human Interaction with AI Systems: New Challenges and Opportunities for HCI Professionals to Enable Human-Centered AI. International Journal of Human–Computer Interaction 39, 3 (2023), 494–518. doi:10.1080/10447318.2022.2041900DOI
- [132]Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 13 pages. doi:10.1145/3313831.3376301DOI
- [133]Nur Yildirim, Mahima Pushkarna, Nitesh Goyal, Martin Wattenberg, and Fernanda Viégas. 2023. Investigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebook. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 13 pages. doi:10.1145/3544548.3580900DOI
- [134]Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. 2024. R-Judge: Benchmarking Safety Risk Awareness for LLM Agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 1467–1490. doi:10.18653/v1/2024.findings-emnlp.79DOI
