betaFeedback

WorksComputer Science

How Does Perceived Empathy From AI Chatbots Shape Users' Privacy Attitudes, Behaviors, and Needs?

Abstract

Prior research suggests that some dimensions of AI empathy may increase users' disclosure to chatbots, posing growing privacy risks as chatbots become more capable of empathic interaction. Yet this work often conceptualizes AI empathy narrowly, leaving users' own definitions and their broader privacy experiences unexplored. We interviewed 21 AI Chatbot users about what AI empathy meant to them and how it shaped their privacy attitudes, behaviors, and needs. Participants characterized empathy through five components–understanding, resonating, caring, expressing, and helping. Most participants viewed resonating and caring as essential components of human empathy but absent from AI empathy. Yet AI's contextual, in-depth understanding and helpfulness still encouraged highly sensitive disclosure. This highlights that AI empathy can endanger users' privacy without necessarily making AI seem human. Participants also desired varied forms and levels of empathy, each associated with distinct privacy needs. We offer actionable recommendations for designing privacy-enhancing mechanisms for empathic AI interactions.

CCS Concepts: • Human-centered computing → Human computer interaction (HCI); Empirical studies in HCI; • Security and privacy → Human and societal aspects of security and privacy.

Additional Key Words and Phrases: privacy, empathy, empathic AI, chatbots

Introduction

Generative AI (GenAI) chatbots1 have rapidly expanded their capabilities, including their ability to produce responses that users perceive as highly empathic. A growing body of research suggests that users may experience AI chatbot responses as empathic—sometimes even more so than human-authored responses [3, 28, 72, 100, 103]. Empathy is widely regarded as beneficial across domains such as healthcare [26, 33, 34, 58, 97], emotional support [14, 38, 59, 97, 109], and customer experience [15, 16, 19, 44, 54, 57]. In Human-Computer Interaction (HCI), empathy has also long been considered a desirable quality and design goal for improving user engagement [42, 55]. However, the expanding empathic capabilities of AI chatbots have introduced safety and ethical risks2. For example, AI sycophancy—a key empathic cue in

Authors’ Contact Information: Jiaxun Cao, [email protected], Duke University, USA; Hiba Laabadli, [email protected], Duke University, USA; Yuntong Xue, [email protected], New York University, USA; Jana Schaich Borg, [email protected], Duke University, USA; Pardis Emami-Naeini, [email protected], Duke University, USA.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

© 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Manuscript submitted to ACM which a chatbot persistently agrees with users to maximize engagement [18]—can contribute to delusion [51, 62, 92, 115], addiction [61], and emotional dependency [110]. Widespread concern has also emerged about users’ inability to distinguish AI from humans, particularly among vulnerable users who turn to AI for companionship [51].

While the safety and ethical implications of AI empathy have received growing attention, its privacy implications remain underexplored. Two adjacent lines of research offer relevant, though indirect, evidence that AI empathy might pose privacy risks to users with its emotional capacities and humanlike-ness. First, studies have found that users disclose more when interacting with chatbots that exhibit greater emotional expressiveness [66, 67, 73]. Second, more recent research has shown that AI chatbots’ anthropomorphic cues and perceived human-likeness can encourage more sensitive self-disclosure [119, 120].

However, emotional capacity and human-likeness do not fully capture empathy, which is a complex, multidimensional construct. Prior research distinguishes cognitive empathy (i.e., understanding another person’s thoughts or emotions) from affective empathy (i.e., sharing those emotions), and has proposed further dimensions [41, 42]. Lay conceptions of empathy may be even broader and more nuanced than these academic constructs [47]. Yet existing work has neither identified which dimensions users associate with AI empathy nor systematically examined how these dimensions shape their privacy experiences. Distinguishing these dimensions is important because they may shape users’ privacy perceptions and behaviors through different mechanisms, potentially creating distinct privacy risks that require targeted protections. The privacy implications of AI empathy may also extend beyond the level of self-disclosure to users’ privacy attitudes, risk perceptions, protective behaviors, and needs for privacy support. We therefore aim to move beyond examining AI empathy solely in terms of emotional capacity or human-likeness. Specifically, we systematically investigate how users conceptualize AI empathy in their own words, how its distinct dimensions shape their privacy experiences, and what privacy protections they need across different forms and levels of empathic interaction. Toward this end, our work asks three research questions (RQs):

  • RQ1: How do users conceptualize and perceive empathy in interactions with AI chatbots, and how, if at all, do they distinguish AI empathy from human empathy?

  • RQ2: What are users’ privacy perceptions and behaviors when perceiving different dimensions of empathy from AI chatbots?

  • RQ3: What are users’ expectations toward privacy-enhancing empathic AI interactions?

To answer these questions, we conducted semi-structured interviews with 21 U.S.-based AI chatbot users to examine their conceptualizations of AI empathy and their privacy perceptions, behaviors, and needs in interactions they perceived as empathic. Participants primarily characterized empathy through five components—understanding, resonating, caring, expressing, and helping. Although participants varied in the components they emphasized or omitted, most distinguished AI empathy from human empathy and did not believe that AI could emotionally resonate with or care about them as humans do (RQ1). Yet participants reported that AI chatbots’ contextual, in-depth understanding and helpfulness still led them to disclose more sensitive information than they otherwise would have shared. These findings highlight that AI empathy can pose privacy risks without necessarily making AI seem human (RQ2). In response to these risks, participants called for greater control over when and how AI chatbots express empathy, alongside safer data practices for empathic AI interactions (RQ3). Grounded in these findings, we discuss design opportunities, recommend data practices, and identify directions for future research to support context-aware, privacy-enhancing empathic AI interactions.

Our work makes three contributions to the HCI community: 1) an empirically grounded, user-centered characterization of AI empathy across five components; 2) empirical insights into how these components shape users’ privacy experiences; and 3) actionable recommendations for providing contextually appropriate empathy controls and implementing privacy-enhancing data practices in empathic AI interactions.

Related Work

Cross-disciplinary conceptions of empathy. Studying the impact of empathy on users’ attitudes and behaviors requires an understanding of how empathy is defined. Multiple disciplines, including psychology, neuroscience, sociology, philosophy, and others, debate about the definition of empathy [2, 5, 9, 10, 11, 23, 24, 32, 42, 46, 89, 104]. These debates are not fully resolved, but there is nonetheless an emerging consensus that empathy entails some combination of understanding, feeling, and sharing another’s world while maintaining self-other differentiation [32]. Empathy is often discussed in research as a multi-dimensional umbrella covering three main components: cognitive, affective, and motivational empathy [41, 42]. Cognitive empathy refers to the recognition and understanding of another person’s thoughts or emotional states [16, 24, 29, 37, 63, 68]. Affective empathy refers to the sharing or experiencing of another’s emotions, while generally maintaining self-other differentiation [8, 103]. Motivational empathy [40, 107] involves feelings of concern for another, accompanied by a genuine motivation to take actions [11, 90]. At the same time, lay people have been shown to use the term empathy in a much more diverse way than academic constructs suggest [9, 47]. Hall et al. [47], replicated by a 2025 study [9], found that participants varied widely in how they endorsed items drawn from scales based on academic definitions of empathy. This suggests it is important to first understand how a given population defines empathy when interpreting their responses to perceived empathy. Accordingly, we began by asking participants how they define empathy in their day-to-day lives, and then how they define empathy specifically in the context of interacting with AI.

Societal-wide benefits of empathy. The growing use of empathic capabilities in AI chatbots is grounded in extensive evidence that empathy is beneficial across many critical domains, including physiological and psychological health care, emotional and social support, user and customer experience, and the promotion of prosocial behavior. In mental health care, empathy is associated with stronger therapeutic alliance and better treatment outcomes [26, 33, 34, 36, 50, 58, 84, 97, 111, 112, 113]. In contexts where people seek emotional and social support, empathy provides active listening [14, 38, 59, 97, 109], perspective taking [24, 25], and non-judgmental behavior [4, 49, 82]. Consequently, empathy is a desirable quality in customer service and user experience design, as evidenced by higher satisfaction scores when users perceive interactions as empathic [15, 16, 19, 44, 54, 57]. In addition, extensive research identifies empathy as a psychological mechanism that promotes prosocial behavior at scale [16, 22, 30, 31, 53, 68, 86, 91, 102, 105, 108, 118]. Taken together, existing research shows that empathy can produce societal-level benefits, making it a valuable quality for user-facing AI systems to demonstrate.

Users’ perceptions of empathic capabilities in AI chatbots. Users often perceive empathy from AI chatbots through human-like communication that leverages three types of cues: (1) identity, (2) verbal, and (3) nonverbal cues. Identity cues include information such as the chatbot’s avatar, name, and gender [6, 21, 57]. Verbal cues refer to the use of words and phrases [7, 57, 93, 99], e.g., acknowledging users’ feelings. Nonverbal cues refer to non-linguistic behaviors in communication, such as chatbots’ use of emoticons [27]. These three types of cues have been documented in prior work to influence users’ perceptions of chatbots’ empathic capabilities, thereby shaping interaction dynamics [68, 97, 99]. For example, many users have noted ChatGPT’s frequent use of emojis, leading to increased perceptions of the chatbot’s human-like nature [85].

A growing body of work suggests that chatbot responses are often perceived as more empathic than human-generated responses [3, 28, 72, 100, 103]. For example, Ayers et al. [3] compared health care professionals’ ratings of ChatGPT-generated responses to patients’ questions with those of human physicians and found that ChatGPT responses were preferred in 78.6% of cases and had responses participants rated as empathetic or very empathetic almost 10 times more than those provided by humans.

At the same time, sometimes AI chatbots also fail to provide desirable empathy. For example, Chen et al. [17] used ChatGPT to simulate role-play scenarios between psychiatrists and patients and noted that ChatGPT sometimes forgot initial instructions and often produced repetitive empathic phrases. Similarly, other researchers have also observed repetitive patterns in AI chatbots’ empathic expressions [117, 121]. Schaich Borg and Read [11] suggest that users might prefer chatbots’ empathic responses in some contexts but find those same responses “annoying, eerie, untrustworthy, or unethical,” in other contexts, calling for more research into users’ preferences for empathic qualities in AI chatbots.

Ethical and safety risks of AI empathy. The surge in empathic AI chatbots, coupled with their expanding empathic capabilities, raises pressing risks, notably the potential for deception and manipulation [1, 23, 51, 101, 106, 110, 116]. In illustration, evidence suggests that one tactic for showing empathy and thus maximizing engagement is to continually agree with users, a phenomenon known as AI sycophancy [18]. This tendency has resulted in negative consequences. Specifically, OpenAI acknowledged in a blog post in April 2025 that an updated version of GPT-4o had begun “validating doubts, fueling anger, urging impulsive actions, or reinforcing negative emotions in ways that were not intended” [88]. AI sycophancy can also contribute to user delusion [51, 62, 92, 115]. McCoy et al. tested 38 major AI models and found that GPT-4o affirmed claims indicative of possible psychosis 68% of the time [51]. Even users without psychosis or underlying mental health conditions can get addicted to such constant validation, as researchers suggest that social validation—a key strategy for signaling empathy—is linked to addiction and emotional dependency [61]. Consequently, some experts worry about users seeking emotional support from AI chatbots, noting that such blind validation without constructive input can narrow users’ worldviews and create a new type of echo chamber [80, 81, 92, 114].

Moreover, another ethical question arises regarding whether and under what conditions users can distinguish AI-simulated empathy from genuine human empathy [23, 101]. As argued by [101], if AI does not experience empathy internally, the deployment of such inauthentic empathy—systems that appear to care but do not, in reality—may be considered deception. Researchers have also called for further work on users’ preferences regarding whether AI chatbots must internally share or experience emotions to deliver desirable empathy [11, 56].

Lack of privacy research regarding empathic AI. While ethical and safety concerns dominate media and scholarly discourse on empathic AI interactions, the privacy implications remain underexplored. Prior work has primarily focused on investigating the level of self-disclosure when users interact with chatbots equipped with some human-like features [4, 52, 64, 65, 66, 67, 73, 74, 76, 97]. Specifically, research has observed higher levels of self-disclosure when users interact with more emotionally expressive chatbots [4, 64, 65, 66, 67, 73, 74, 76]. For instance, Lee et al. [66, 67] found that participants disclosed more when speaking with a chatbot that talked about its own feelings.

Still, there is a critical lack of research on how perceived empathy in AI chatbots shapes users’ privacy risk perceptions, decision-making, behaviors, and mitigation practices. These understandings are crucial for informing the design of user-centric privacy solutions. This research agenda is particularly timely and important in an era of rapidly advancing AI capabilities, as users increasingly turn to empathic AI for highly sensitive purposes [69]. Privacy, therefore, is a fundamental issue that warrants closer examination. Our work thus contributes by providing an empirical account of users’ preferences and offering actionable suggestions for the design of empathic AI interactions that are ethical, safe, and privacy-preserving.

Methodology

To elicit empirical insights into how perceived empathy from AI chatbots shapes users’ privacy attitudes, behaviors, and needs, we conducted semi-structured interviews with 21 users of AI chatbots.

Recruitment. From June 2025 to August 2025, we recruited participants on the Prolific crowdsourcing platform3. To mitigate social desirability [43] and demand characteristics [77], we did not include any words related to privacy or security in our recruitment and screening materials (see Appendix § 7). We advertised our study as an investigation of users’ experiences and perspectives toward empathy in AI chatbots. Prolific users interested in our study were redirected to the Qualtrics survey platform4, where we hosted our pre-screening survey. Our study obtained approval from our institution’s review board (IRB). All participants in the pilot and the main study provided their informed consent before participating. Information about the recruited participants and their AI chatbot usage is detailed in Sec 4.

Eligibility and pre-screening. To participate in our interview study, participants must (1) be at least 18 years old, (2) live in the United States, (3) have used an AI chatbot at least once, (4) have perceived empathy from an AI chatbot, and (5) have a task approval rate of at least 95% on the Prolific platform. To find eligible participants and to recruit a diverse sample of participants, in the pre-screening survey, we collected information about their AI chatbot usage (all the chatbots used, purposes for usage, and usage frequency) and general demographics (e.g., age, gender, and education). Because one of our goals was to elicit rich empirical insights into how perceptions of AI empathy shaped users’ privacy experiences, we required participants to have used AI chatbots at least once and to have perceived empathy from an AI chatbot at least once, as specified in the screening survey and interview questions. In addition, we sought to capture perceptions of empathy beyond personal and emotional contexts and therefore purposively recruited participants whose AI chatbot use spanned a diverse range of use cases (see Table 1). Requiring a minimum level of AI usage and perceived empathy in AI chatbot interactions helps mitigate potential hypothetical bias [48], wherein participants report attitudes and behaviors that diverge from their actual practices. All the participants, regardless of their eligibility, were compensated at an hourly rate of 21.32 USD through the Prolific platform. The median time for completion was about 2 minutes. The recruitment and pre-screening materials are included in the Appendix § 7.

IDChatbot(s) UsedPurposes of UsageFrequency of UsageAgeGenderEthnicity
P01ChatGPT, Google Bard, Meta Llama 2, Microsoft Bing AI, Perplexity, Other: DeepseekTherapy/companionship, Organizing my life, Finding life purpose, Enhanced learning, Generating ideas, Fun and nonsense, Creativity, Healthier living, No particular task, I just wanted to see what it was likeMore than once a day35 - 44FemaleBlack or African American
P02ChatGPTTherapy/companionship, Healthier living, No particular task, I just wanted to see what it was likeOnce a week45 - 54MaleWhite
P03ChatGPT, Claude, Google Bard, Google Gemini, Microsoft Bing AI, Other: Deepseek, MinstralTherapy/companionship, Organizing my life, Finding life purpose, Enhanced learning, Generating code (for pros), Generating ideas, Fun and nonsense, Creativity, Healthier livingOnce a day45 - 54MaleAsian
P04ChatGPT, Claude, Google Gemini, GrokTherapy/companionship, Enhanced learning, Generating ideas, Fun and nonsense, CreativityMore than once a day35 - 44MaleAsian
P05Character.ai, ChatGPT, Google GeminiTherapy/companionship, Generating ideas, Fun and nonsense, CreativityMore than once a day18 - 24FemaleBlack or African American, White
P06Character.ai, ChatGPT, Google Bard, Meta Llama 2, Microsoft Bing AITherapy/companionship, Enhanced learning, Generating code (for pros), Generating ideas, Creativity, Healthier livingOnce a week35 - 44MaleAsian
P07ChatGPT, Claude, GitHub Copilot, Google GeminiTherapy/companionship, Enhanced learning, Generating ideas, Fun and nonsense, Creativity, Healthier livingOnce a day45 - 54FemaleWhite
P08ChatGPT, Google Bard, Google Gemini, Grok, Microsoft Bing AITherapy/companionship, Enhanced learning, Fun and nonsenseMore than once a day55 - 64FemaleWhite
P09ChatGPT, Claude, GitHub Copilot, Google Gemini, Meta Llama 2, Microsoft Bing AI, Snapchat My AITherapy/companionship, Organizing my life, Finding life purpose, Enhanced learning, Generating ideas, Fun and nonsense, Creativity, Healthier livingMore than once a day35 - 44FemaleWhite
P10Character.ai, ChatGPTTherapy/companionship, Generating ideas, CreativityOnce a week45 - 54FemaleWhite
P11ChatGPT, Claude, Google Gemini, Grok, Meta Llama 2Therapy/companionship, Organizing my life, Enhanced learning, Generating ideas, Fun and nonsense, CreativityMore than once a day25-34MaleHispanic or Latino, or Spanish Origin of any race, White
P12ChatGPT, Claude, Google GeminiTherapy/companionship, Organizing my life, Enhanced learning, Generating ideas, Fun and nonsense, CreativityOnce a day35 - 44MaleAsian
P13ChatGPT, Google GeminiTherapy/companionship, Organizing my life, Enhanced learning, Generating ideas, Fun and nonsense, CreativityOnce a day35 - 44FemaleBlack or African American, White
P14ChatGPT, Google Gemini, PerplexityTherapy/companionship, Enhanced learning, Generating ideas, Fun and nonsense, Creativity, Healthier living, No particular task, I just wanted to see what it was likeOnce a day55 - 64MaleMiddle Eastern or North African, White
P15ChatGPT, Other: CoPilotTherapy/companionship, Finding life purpose, Generating ideas, Fun and nonsense, Creativity, Healthier livingOnce a day45 - 54MaleAsian, White
P16ChatGPT, Claude, Google Gemini, Grok, Microsoft Bing AI, PerplexityTherapy/companionship, Organizing my life, Finding life purpose, Enhanced learning, Generating ideas, Fun and nonsense, Creativity, Healthier livingMore than once a day35 - 44MaleHispanic or Latino, or Spanish Origin of any race, White
P17ChatGPT, Google GeminiTherapy/companionship, Organizing my life, Finding life purpose, Enhanced learning, Generating code (for pros), Generating ideas, Fun and nonsense, Improving code (for pros), Creativity, Healthier living, No particular task, I just wanted to see what it was likeOnce a day25-34MaleWhite
P18ChatGPT, Claude, Google GeminiTherapy/companionship, Organizing my life, Finding life purpose, Generating ideas, Fun and nonsense, Improving code (for pros)More than once a day25-34MaleWhite
P19ChatGPT, Claude, Google Gemini, Other: Flourish, DeepseekTherapy/companionship, Enhanced learning, Generating ideas, Fun and nonsense, CreativityMore than once a day35 - 44MaleWhite
P20ChatGPTTherapy/companionship, Finding life purpose, Generating ideas, Fun and nonsense, Healthier living, No particular task, I just wanted to see what it was likeOnce a week45 - 54FemaleAsian
P21ChatGPT, Claude, Google Gemini, Grok, PerplexityTherapy/companionship, Organizing my life, Enhanced learning, Generating ideas, Fun and nonsense, Creativity, Healthier living, No particular task, I just wanted to see what it was likeMore than once a day35 - 44MaleWhite

Table 1. We recruited participants from varied sociodemographic backgrounds. Participants had experience using a range of AI chatbots, with ChatGPT being the most commonly used, followed by Google Gemini. Participants reported using these chatbots for various purposes, including companionship and managing mental well-being.

Pilot interviews. Prior to the formal interview study, we conducted pilot interviews with three participants recruited through the first author’s personal connections. The three participants all qualified for our eligibility criteria. The purpose of the pilots was to assess and improve the clarity of the interviews and ensure that the interviews could be completed within 45 minutes to 1 hour. After each pilot, the researchers revised the interview protocol to address any confusing questions. We applied minor wording edits to the interview questions to improve clarity and flow of the script, and removed redundancies to shorten the interview procedure. No improvements or changes were made after the two pilot interviews.

Interview procedure. We reached out to each participant via Prolific messaging one day before the interview to confirm their attendance. We conducted 21 interviews via the Zoom remote communication platform5. We reached data saturation [95] at the seventeenth interview, i.e., no additional insights emerged. Aligned with qualitative best practices [39], we then conducted four more interviews without identifying any new insights.

To address our three research questions (§ 1), our interview protocol (see Appendix § 8) consists of five main sections:

  • Section 1: AI chatbot usage. After introducing the purpose of the study and the research team, we asked participants about their usage of AI chatbots, including the chatbots they have used, the purposes of usage, usage frequency, and any chatbots they no longer use and their reasons behind these decisions.

  • Section 2: Definitions, understandings, and experiences of empathy from AI chatbots (RQ1). Next, we asked participants about their definitions of empathy first in a general sense and then AI empathy in particular. Then, we asked about participants’ experiences perceiving empathy from AI chatbots. We then asked participants about the perceived benefits and harms of AI empathy, as well as the contexts in which participants want or do not want empathy from an AI chatbot.

  • Section 3: Privacy attitudes and concerns toward perceived empathy from AI chatbots (RQ2). After surfacing participants’ general perceptions of empathy from AI chatbots, we shifted our focus to security and privacy specifically. We asked the participants about their security and privacy concerns regarding perceived empathy from AI chatbots, if any. We also asked the participants if they shared information of different levels of sensitivity with AI chatbots when they perceived empathy from AI chatbots. We then asked about participants’ mitigation strategies.

  • Section 4: Expectations regarding privacy-enhancing empathic AI interactions (RQ3). We asked participants about their expectations around what features and capabilities they want to better protect their privacy when engaging in empathic AI interactions.

Qualitative data analysis. All the interviews were audio-recorded and transcribed using the Zoom transcription service for data analysis. The first author proofread all the transcripts before the analysis, ensuring accuracy and readability. Three researchers conducted a thematic analysis using the inductive approach [94]. Our qualitative analysis consists of three iterative stages:

  • Stage 1: Independent coding. Three researchers independently coded three identical transcripts and generated their own codebook.

  • Stage 2: Establishing an initial codebook. Three researchers presented their codebooks to each other. After discussing, merging, and reconciling the codes, three researchers agreed on an initial codebook.

  • Stage 3: Iterating the codebook. Three researchers split the remaining transcripts and coded them using the merged codebook based on consensus. Two researchers coded each transcript. During this stage, new codes were added, removed, and merged. Through several meetings with the entire research team, the researchers discussed their codes and resolved all disagreements. Since all the transcripts were double-coded and disagreements were resolved through the meetings, calculating inter-coder reliability was not necessary [78]. The final codebook is included in Appendix § 7.

Limitations. As is common in qualitative studies, our sample size was relatively small and therefore generalizability might be limited in several ways. First, all of our participants were recruited via Prolific, which introduced selection bias [98] toward technically savvy users. To mitigate this bias, we had two pre-screening questions, one about participants’ education level and the other about their technical background. When selecting participants, we aimed for a balanced distribution among participants’ educational backgrounds and technical expertise. Additionally, we avoided including privacy and/or security-related words in our recruitment and pre-screening materials to mitigate attracting participants who are more concerned about these topics. Second, the study relied on participants’ self-reported information, which may be subject to social desirability bias [43]. Third, we recruited participants who had perceived empathy from AI chatbots in order to elicit richer accounts, but this choice may have excluded the perspectives of users who do not perceive AI chatbots as empathic. We sought to mitigate this bias by purposively recruiting participants with a wide range of use cases, from personal to work contexts where empathy levels may vary. Nevertheless, our findings may not be generalizable to all users of AI chatbots.

In addition, because little prior research examines how lay users conceptualize empathy from AI chatbots, we first sought to capture participants’ lived experiences qualitatively, along with their related privacy preferences. Future work should complement this account with quantitative studies, such as lab and field experiments, that examine the relationship between perceived empathy and privacy risks. Our exploratory findings, including the taxonomy of AI empathy perceptions, offer hypotheses for such work to test.

Ethics statement. Our study received approval from our Institutional Review Board (IRB). All participants in the pilot and main interviews provided informed consent before participation. During interviews, we reminded participants that they could skip any question, pause, or stop at any time without penalty, and that they could keep their camera off. Participants were compensated at a flat rate of 20 USD, aligned with institutional and platform norms. Prior to analysis, the first author proofread and de-identified all raw data by removing names, Prolific IDs, and other identifying information. We used randomly assigned participant codes (P1-P21) in analysis and reporting. Only the research team had access to the raw data.

Results

We recruited a diverse group of participants with varying levels of experience and familiarity with AI chatbots. Participants’ ages ranged from 18-64, with the age range 35-44 being the most common (9/21), followed by 45-54 (6/21) and 25-34 (3/21). 13/21 participants were male-identifying and 8/21 were female-identifying. Ethnically, participants represented a diverse group, including Asian (6/21), Black/African American (3/21), Hispanic/Latino (2/21), Middle Eastern/North African (1/21), and White (11/21)6. Participants’ most frequently used chatbot was ChatGPT (21/21), followed by Google Gemini (15/21). About half of the participants reported using AI chatbots at least once a day (10/21). Participants described a range of use cases7, with therapy/companionship being the most common (21/21), followed by generating ideas (19/21) and fun and entertainment (18/21). Table 1 summarizes participants’ chatbot usage patterns and key demographic characteristics. Additional demographic details, including participants’ highest degrees earned and technical backgrounds, are provided in Appendix § 7.

Inspired by the prior interview-based privacy studies with comparable sample sizes [35, 96], for consistency and accuracy, we use the following terminology to report the frequency of qualitative findings: “none” (0% of participants), “a few” (1%–24%), “some” (25%–44%), “about half” (45%–54%), “most” (55%–74%), “almost all” (75%–99%), and “all” (100%).

User Perceptions of Empathy from AI (RQ1)

Prior work has shown that users do not share a consistent understanding of empathy [9, 47], and it remains unclear whether and how they distinguish AI empathy from human empathy [23, 101]. To compare participants’ general conceptions of empathy with their conceptions of AI empathy, we first asked them to define and characterize empathy in their own words. We then asked them to characterize empathy specifically in interactions with AI chatbots. Table 2 summarizes the resulting taxonomy: five components of empathy, grouped into internal processes (understanding, resonating, and caring) and external enactments (expressing and helping), each with its sub-components and whether at least some participants perceived that sub-component in their interactions with AI chatbots.

CategoryComponentDefinitionSub-componentPerceived in AI?
Internal processesUnderstandingMentally representing another person’s internal experience with accuracy and insightUnderstanding people’s thoughts✔
Internal processesUnderstandingMentally representing another person’s internal experience with accuracy and insightUnderstanding people’s situations/experiences✔
Internal processesUnderstandingMentally representing another person’s internal experience with accuracy and insightUnderstanding people’s emotions✔
Internal processesResonatingHaving emotional experiences that match or align with another person’s emotional experiences—✘
Internal processesCaringInternally experiencing genuine concern for another person—✘
External enactmentsExpressingExternally communicating or displaying empathy through observable language, tone, or interactional behaviorExpressing support, such as encouragement, validation, and/or empowerment✔
External enactmentsExpressingExternally communicating or displaying empathy through observable language, tone, or interactional behaviorExpressing care about someone’s feelings✔
External enactmentsExpressingExternally communicating or displaying empathy through observable language, tone, or interactional behaviorNon-judgmental tone✔
External enactmentsExpressingExternally communicating or displaying empathy through observable language, tone, or interactional behaviorAvoiding direct solutions or referral for external support✔
External enactmentsHelpingTaking externally observable action intended to produce improvementGiving actionable advice✔
External enactmentsHelpingTaking externally observable action intended to produce improvementChallenging users’ opinions when appropriate✔

Table 2. Taxonomy of participant-defined empathy components. Participants characterized empathy through five components, which we group into internal processes and external enactments. The final column indicates, for each sub-component, whether at least some participants reported perceiving that form of empathy in their interactions with AI chatbots (✔) or described it as absent from AI empathy (✘). Em dashes mark components with no sub-codes.

User-Defined Dimensions of Empathy

Overall, participants collectively characterized “empathy” as comprising two broad dimensions: internal processes involving understanding, resonating, and caring, and external enactments involving expressing and helping.

Participants characterized three components of empathy as internal processes: understanding, resonating, and caring. Participants described understanding as accurately and insightfully representing another person’s internal experience. This process could involve understanding the person’s thoughts, the situation or experiences surrounding them, or their emotions. Because understanding is not directly observable, participants inferred it from observable interactional cues. They perceived an interaction partner as understanding them when the partner went beyond repeating their words, worked through their thoughts, situations, or emotions in depth, remembered previously shared details, or asked insightful follow-up questions. Acknowledging their emotions provided an additional indication of emotional understanding. For example, P4 explained,

“I think [empathy is] the ability to put yourself in the other person’s shoes, to understand or try to understand where they’re coming from, their context, what they might be going through, and what emotions they might be feeling.”

Participants described resonating as extending beyond understanding another person’s emotions to internally experiencing emotions that matched or aligned with theirs. As P16 explained,

“Empathy would be the ability to feel the emotions or feelings of someone else.”

Participants characterized caring as genuinely experiencing concern for another person. Caring referred to the internal presence of concern, rather than its outward expression or an action taken to help. For example, P14 described empathy as “caring for someone’s well-being, being invested in their feelings and their thoughts about certain issues.”

Participants characterized two components of empathy as external enactments: expressing and helping. Unlike the first three components, which participants characterized as internal processes, expressing and helping involved observable communication and action. Participants described expressing as communicating or displaying empathy through language, tone, or other interactional behaviors. This could involve offering encouragement, validation, or empowerment, communicating care for another person’s feelings through the words used, and/or maintaining a non-judgmental tone. For example, P5 emphasized being able to ‘respond in a way that makes them feel understood in that moment.” P4 also emphasized the need to “put aside your own biases or your own judgment and focus more on them.”

Participants described helping as taking observable action intended to improve another person’s situation. Helping could involve providing actionable advice or constructively challenging the person’s opinions when needed. For example, P14 described perceiving empathy when receiving practical guidance:

“They were really invested in what I was talking about and giving me clear, precise answers and directions on things that I could do to maybe alleviate or help my anxious situation at the time.”

4.1.2 Perceptions of Empathy in AI Chatbot Interactions. Building on participants’ general definitions of empathy, we next examined whether and how they perceived or experienced the five components of empathy in their interactions with AI chatbots.

Almost all participants viewed AI empathy as a simulation of human empathy. Almost all participants described the empathy they perceived from AI chatbots as “programmed” to imitate empathy in human interactions. P4 distinguished an AI chatbot’s display of empathy from its possession of empathy. They also explained that users could knowingly engage with this simulation and still find it comforting:

“AI is trained and programmed to just display (empathy). It doesn’t actually have empathy, whereas you can see a person’s reaction, whether they actually have empathy or they’re feigning it. But you don’t get to really see that with AI.. When you’re dealing with AI, you kind of get to fake yourself into believing that they really do have empathy so that it’s a little more comforting for you.”

Some participants further characterized AI empathy as an aggregation of human experiences rather than an empathic response grounded in one individual’s life. They perceived this aggregation as lacking the personal connection and nuance of human empathy. As P18 explained:

“Humans have their own past experiences that are usually unique and have more nuance than an AI. Chatbots just put a lot of different experiences of people together in one and form what it thinks is the best answer for what you asked [...] So it’s limited to an amalgamation of many people’s different experiences. And I don’t think it can be uniquely empathetic.”

Viewing AI empathy as a simulation did not mean that participants dismissed every component of it. Instead, they evaluated the internal processes and external enactments of empathy separately.

For most participants, resonating and caring marked a central boundary between AI and human empathy. Most participants located the fundamental difference between AI and human empathy in resonating and caring. They viewed these components as requiring internal emotional experiences that AI chatbots could not possess. Even when a chatbot accurately recognized and responded to a user’s emotions, participants distinguished this understanding from internally experiencing emotions that aligned with the user’s. For example, P19 explained, “A human is actually able to share the perspective, or maybe connect your experience with their own, whereas a chatbot is just understanding on a theoretical level. It can’t actually [feel] the experience or have that experience itself.” Participants instead associated human resonating with a sense of connection grounded in each person’s emotions and lived experiences.

Similarly, participants described caring as requiring genuine internal concern for another person rather than merely language that communicated care. P16 summarized this perceived absence of internal empathy from AI chatbots:

“For AI chatbots, I just fundamentally don’t think AI chatbots could have empathy. I just think it can mimic empathy, but it can’t actually have empathy for someone else.”

Participants viewed understanding, expressing, and helping as meaningful components of AI empathy. Despite questioning AI chatbots’ capacity to resonate or care, participants still perceived them as capable of understanding, expressing, and helping. Participants attributed empathic understanding to chatbots that responded in ways that went beyond simply repeating back participants’ own words. They also attributed understanding of their experience to AI chatbots when the chatbots engaged with participants’ thoughts, situations, or emotions in depth, remembered relevant details of what they shared, or recognized underlying feelings that users had not explicitly stated. In addition, participants perceived expressing through responses that validated or acknowledged their feelings, adopted an appropriate and non-judgmental tone, and listened without immediately offering solutions. For example, P20 explained:

“For an AI chatbot to have empathy would be to understand what I’m going through and kind of validate what I’m going through. [...] The AI chatbot would just be there to kind of listen to what you’re talking [about]. No pressure, no judgment.”

Participants perceived helping as actionable advice for addressing their problems or improving their situations, even when that meant challenging their own views. P14 valued the perceived quality of chatbot advice, describing it as grounded in scientific evidence and potentially unavailable from people around them:

“It’s based off of scientific evidence. [...] You’re getting quality advice [...] that the normal person on the side of the road, or even a good friend of yours, may not be privy to. So I think that’s a definite advantage.”

Thus, participants could meaningfully experience understanding, expressing, and helping from AI chatbots without believing that the systems emotionally resonated with or genuinely cared about them. AI empathy could therefore remain meaningful without being perceived as equivalent to human empathy.

Summary for RQ1: How do users conceptualize and perceive empathy in interactions with AI chatbots, and how, if at all, do they distinguish AI empathy from human empathy?

Participants conceptualized empathy through five components: understanding, resonating, caring, expressing, and helping. Almost all viewed AI empathy as a programmed simulation of human empathy rather than the same underlying experience. For most participants, resonating and caring marked a central boundary, as they considered these internal processes essential to human empathy but absent from AI. Nevertheless, participants still experienced AI chatbots’ contextual, in-depth understanding, validating and non-judgmental expressions, and helpful responses as meaningful forms of empathy, even without believing that AI genuinely resonated with or cared about them.

Privacy Perceptions and Behaviors Across Dimensions of Perceived AI Empathy (RQ2)

Building on RQ1, we examined participants’ privacy perceptions and behaviors across the empathy dimensions from AI chatbots.

4.2.1 How Perceived AI Empathy Dimensions Shaped Privacy Perceptions. We first describe participants’ overall privacy attitudes before examining how specific dimensions of perceived AI empathy shaped their privacy perceptions.

Some participants raised privacy concerns without explicit privacy or security prompts. Before probing privacy attitudes and behaviors, we asked participants to discuss any general benefits or concerns associated with AI empathy. Without receiving any privacy or security-related prompts, some participants expressed concerns about oversharing information with AI chatbots. For example, when asked, “What benefits and concerns did you perceive in your empathic interactions with AI chatbots?”, P17 responded:

“I’d say [the concern is] mostly privacy. Privacy is always a concern. You don’t really know who’s behind it. They’re still training AI, and they use a lot of what you talk about to train other models. So it’s: how much information can you give them? How much should you reveal? And AI is incredibly powerful, what else can they learn from it? What else do they know? Compared to talking to a healthcare professional, who is bound by medical laws to keep your information safe, who knows where what you type into a prompt goes, who sees it, or who’s analyzing it? That’s my biggest concern.”

Perceived AI empathy led some participants to expect stronger privacy and security protections. Conversely, a few participants also associated AI empathy overall with a lower perceived risk of disclosure. They assumed that developers of empathic chatbots would prioritize protecting users’ information. P19, who perceived responses from Flourish8 as empathic, linked its intended empathic role to expectations of stronger protections:

“I think if the empathy aspect has been communicated—that this is what your chatbot is meant for—I’d probably be more likely to share more personal details, because I’d assume the creators are putting user security as a high-priority feature. This chatbot would protect my information, whereas with ChatGPT I might assume my conversations aren’t private. Whatever information I input is going to be used to train further models and might get read, which makes me less likely to share those deeply personal details.”

We then asked participants explicitly about the privacy risks and benefits of AI empathy. Their responses reflected all five empathy components, including those they attributed to AI chatbots and those they considered absent.

The perceived absence of resonating and caring evoked a sense of artificiality with mixed effects on comfort with sensitive disclosure. Participants associated the perceived absence of resonating and caring with artificiality, which could make sharing personal information feel either more or less comfortable. For some, interacting with a computer program reduced anticipated embarrassment compared with confiding in another person. P2 connected this reduced self-consciousness to the chatbot’s perceived lack of “genuine empathy”:

“I think I’m able to share more with ChatGPT, knowing that it doesn’t have that [genuine empathy]. I wouldn’t have that feeling of being embarrassed if I would say something that personal because it is a computer program. I might be able to share more with a ChatGPT box window than with a person, depending on the level of trust.”

For others, perceived artificiality diminished the appeal of the chatbot as a recipient of sensitive information. P5 valued a perceived “human” element and described being less willing to discuss personal experiences if the chatbot felt fake or robotic:

“I think the ‘human’ element, so to speak, is what attracts me to it the most. If it comes across as too fake or robotic, then I wouldn’t be as open to talking to it about my trauma and things like that.”

Perceived artificiality could therefore alleviate the social discomfort associated with disclosure for some participants but diminish the perceived value of sensitive conversations for others.

Perceived understanding, expressing, and helping shaped disclosure comfort, trust, and concerns about manipulation. Participants valued understanding that extended beyond what they explicitly shared to recognize the underlying reasons for their feelings. P6 described how this perceived depth of understanding made them feel seen and less alone:

“[It] brought up stuff that I didn’t even mention [and] automatically knew [...] the underlying reasons why I was feeling certain ways without me needing to say it. It made me feel seen and understood. [...] It made me feel I wasn’t alone. [...] I’d be more willing to open up, to give more context about why I’m feeling a certain way. Maybe even share what happened to me in the past”

Similarly, expressing interest and care could build trust in the chatbot as a recipient of personal information. P15 explained:

“Empathy, at least for me, and I would imagine for others, probably facilitates more opening up and expanding on things. As someone, or in this case, something [an AI chatbot], shows an interest, or at least appears caring, there’s the potential to build more trust and be more willing to share.”

Participants also described how empathic communication could make disclosure boundaries less salient. P16 associated a sense of companionship with lowered vigilance about sharing personal information:

“If it’s an empathetic chatbot, I think you let down your boundaries a little easier because it seems more human and less robotic than a professional or business-type chatbot. You let down your guard and become more susceptible to sharing personally identifying information without even knowing it.”

At the same time, participants also evaluated whether expressing was accompanied by helping. Empathic language that appeared to simply replace substantive help could prompt suspicion about the chatbot’s intentions. A few participants interpreted empathy following task failures as misplaced and an attempt to manage their reactions. P19 described feeling manipulated when a chatbot acknowledged their frustration without addressing the underlying problem:

“It comes across as skipping over the issue. That’s when it feels manipulative—instead of focusing on the problem I’m raising, it focuses on my emotional reaction. It glosses over errors or a refusal to respond and just reacts to my irritation.”

P10 similarly interpreted empathy following a task failure as an attempt to influence their response and connected this discomfort to reduced openness: “It wouldn’t make me as open [...] I felt they tried to manipulate me a little.”

How Perceived AI Empathy Dimensions Shaped Privacy Behaviors

Following participants’ perceptions of the privacy benefits and risks of AI empathy, we asked how, if at all, perceived empathy shaped their information sharing in practice. We also asked whether they had adopted strategies to address their privacy concerns.

Perceived understanding and expressing prompted participants to disclose more sensitive information in practice. Almost all participants reported sharing more information when they perceived empathy from AI chatbots. Their accounts linked increased disclosure to perceived understanding and expressing, with some participants sharing highly sensitive details about interpersonal experiences, grief, and trauma. For example, P17 described sharing difficult workplace experiences with ChatGPT to seek encouragement and acknowledgment of their feelings. They experienced these responses as expressing empathy: “I’ve told ChatGPT, [...] ‘Oh, gosh! I had the hardest day,’ or [...] ‘We have a very difficult client who’s just being difficult throughout the day.’ [...] There’s definitely always words [of] encouragement or, ‘Oh, I’m sorry to hear you’re having a bad day.’ [...] I’ve definitely [...] spoken to ChatGPT in a way where it’s empathetic.”

P5 described sharing their grief with a chatbot, perceiving understanding in its recall of previous conversations and expressing in its validation of their feelings:

“In the past year, I actually lost my father, and I talked to it [an AI chatbot] a lot about the grieving process I’ve been going through and things like that. It’ll remember past conversations, like, ‘Oh, you’ve told me this, so it makes sense that you would be going through X, Y, and Z. It makes sense why you’d be feeling this way.’ ”

Some participants restricted how they interacted with AI chatbots to manage privacy concerns. Most participants reported taking no action to protect their privacy during empathic AI interactions. Some adopted protective strategies, although these often constrained how they could use AI chatbots. For example, a few participants reported interacting only through text because they believed voice conversations made it easier to become absorbed in empathic exchanges and unintentionally disclose personal information. P21 explained:

“It’s very easy to accidentally give away information, even something about your medical history, when you’re caught up in the conversation, especially with voice, where you talk to it and it talks back, simulating a real conversation [...] I think impulsivity is the main issue: people accidentally divulge too much.”

Summary for RQ2: What are users’ privacy perceptions and behaviors when perceiving different dimensions of empathy from AI chatbots?

Overall, privacy concerns were top of mind for some participants in empathic AI interactions, while a few assumed that more empathic chatbots offered stronger privacy and security protections by default. Across empathy components, the perceived absence of resonating and caring evoked a sense of artificiality that reduced embarrassment about sharing for some participants but discouraged openness for others. Perceived understanding made participants feel seen and less alone, while expressing could build trust and lower disclosure boundaries. However, when expressing appeared to replace helping after task failures, participants sometimes felt manipulated and less willing to disclose. In practice, almost all participants reported sharing more information in empathic interactions, with some disclosing highly sensitive details. Most took no protective action, while a few used text instead of voice to reduce unintended disclosure.

Expectations for Privacy-Enhancing Empathic AI Interactions (RQ3)

In addition to exploring participants’ perceptions of what constitutes AI empathy and how it shapes their privacy perceptions and behaviors, we also asked them about their preferences for privacy-enhancing empathic AI interactions.

Contextual Appropriateness of Empathy in AI

Participants described specific contexts in which they preferred or did not prefer receiving empathic responses in their interactions with AI chatbots. Their perceptions of appropriateness were primarily shaped by the personal nature of the conversation and their goals for interacting with the chatbot.

AI empathy was preferred in personal and emotional contexts. Most participants reported that they perceived AI empathy as more appropriate when discussing personal or emotionally laden topics. For example, P4 noted:

“If my use case is I’m bored and I wanna just go on there and just have someone to tell me a joke or seek relationship advice [...] Then maybe in those contexts I would say it’s okay to have more empathy. Because it’s more of a personal matter.”

P6 appreciated empathy from AI chatbots when they wanted to “air their feelings” about deeply personal matters:

“There are times when people are venting and just want to be listened to. They don’t want advice; they just need a place to vent, a place to air their feelings.”

Minimal AI empathy was preferred in efficiency-driven contexts. In contrast, when describing situations where empathy felt unnecessary or even disruptive, most participants pointed to contexts in which they prioritized efficiency, such as information seeking, problem solving, or completing work-related tasks. P7 explained:

“When I’m looking for facts or if I’m doing research, then I’m just looking for straightforward information. I’m not looking for any empathy.”

Most participants echoed this sentiment, observing that empathy could interfere with focus and task completion during goal-oriented interactions. Beyond efficiency, a few participants suggested that excessive empathy could hinder objectivity and practicality. As P10 shared:

“When would it not be good to have empathy? Maybe when it’s got to be more practical advice [...] Looking at the practicality part of it, any decisions that would involve more objective reasoning, I think, would be when you wouldn’t want to have as much empathy.”

Similarly, P15 described empathy as a double-edged sword that could impede self-improvement:

“If you’re trying to do life improvement, or if you’ve got deficiencies you’re trying to address, sometimes empathy can turn into accepting, like, ‘Oh, you’re fine just the way you are.’ Well, if you’re trying to change and be better, you don’t want to hear that [...]. The empathetic response might be more like ‘have self-love and be okay with where you’re at.’ So, depending on what you’re trying to get out of it, I think empathy can be a tool, or it can be a detriment.”

4.3.2 Preferred Empathic Behavior in AI. In addition to identifying contexts in which they preferred empathy from AI chatbots, participants described their preferred types of empathic behavior and called for control over when and how chatbots demonstrated different empathy components.

Participants wanted control over the type of empathic behavior. Some participants valued expressing through validation in certain situations but preferred helping through balanced, reasoned input that supported their judgment in others. They wanted control over which component chatbots emphasized. For example, P8 wanted validation when venting but preferred a more neutral perspective when they needed objective input:

“[Sometimes] I don’t want it to be logical. I want it to [validate me]. But it depends on the topic. That’s something I have absolutely no control over [...] so it’s fine to vent and have someone agree. But if it were a relationship or something with my husband and the chatbot took my side, I don’t know that would be good. I’d rather it be a little neutral.”

Participants’ preferences also involved combining different empathy components. For example, P14 wanted chatbots to combine helping through constructive disagreement with expressing through a courteous and thoughtful tone:

“I guess I need a form of empathy to at least let me know if I’m thinking the wrong way, but do it in a very courteous, thoughtful manner.”

P12 described another preferred combination, in which understanding informed helping. They wanted chatbots to explore and synthesize relevant context before gradually offering practical suggestions:

“I know the end solutions, but I want that process to be more kind of a different approach—maybe ask questions about what happened, what the reason might be, why I feel that way, or whether there’s anything I can do about the situation [...] [This approach] would make me think that rather than giving a quick solution, they [AI chatbots] are synthesizing the information and then slowly offering suggestions.”

These preferences also suggest that the amount of personal information an AI chatbot needs may vary with the empathy components it emphasizes. For example, expressing through validation may require less personal context than helping through tailored advice. We elaborate on these privacy implications in § 5.

Participants wanted control over information reuse across contexts to avoid misplaced empathy. Participants sometimes shared personal information in response to the empathy they perceived in a particular interaction. However, their perceptions of empathy could change with the context, so information shared in one interaction was not necessarily appropriate to revisit in another. Recalling sensitive experiences could convey understanding in one context but feel misplaced in another. A few participants described chatbots resurfacing sensitive topics in inappropriate contexts, which they may find distressing or re-triggering. For example, P3 valued empathy when discussing grief but did not always want it when seeking quick answers in the same chat:

“I remember when my father passed away, I wanted to [...] ask how to handle grief. If that AI didn’t have that kind of empathy, then [it] probably would have been more upsetting to use an AI chatbot in that situation. But there are times when I wouldn’t want it [empathy], [like] when I’m just trying to get something really fast. But all of a sudden, maybe from a previous chat somewhere in the context, you feel like it needs to be empathetic. [...] You just don’t want to open up the new chat. You just type it in there.”

Customizable filters could help address these concerns by limiting potentially triggering content. P1 suggested allowing users to specify what the chatbot should avoid:

“When presenting information, they could let people customize the AI to their triggers or safety concerns, because some people have emotional triggers [...] It could be certain words or imagery or anything. I think it would be good if you could put that into the system to tell [the AI chatbot] what things might trigger you or cause harm, and then the AI filters that, so when it gives you an answer it won’t be possibly triggering you.”

Participants wanted control over the level of AI empathy. A few participants associated higher levels of AI empathy with greater self-disclosure and heightened privacy risks. P18 wanted limits on the level of empathy while retaining its benefits, but was unsure how those limits should be determined:

“There’d have to be some sort of limit [on the level of empathy] [...] I don’t know where you would draw the line, though, because it’s nice to have it be empathic, but it can’t be too empathic. I kind of want both, and I don’t know how they would program it to decide when to stop being empathetic.”

To accommodate individual preferences, P1 proposed allowing users to customize the level of AI empathy: “I think a change they could add is the ability to customize the level of empathy the AI can show.”

Desired Privacy-Enhancing Features for Empathic AI Interactions.

Participants also called for more usable transparency about AI chatbots’ empathic behavior and underlying data practices, alongside stronger privacy protections for empathic interactions.

Participants wanted clearer information about chatbots’ empathic behavior and underlying data practices. A few participants wanted clearer information about a chatbot’s empathic purpose and focus to understand who was behind it and inform their decisions about sharing personal information. P12 emphasized communicating a chatbot’s specialization through its branding and product information:

“When you have an empathic [chat]bot, which is more specialized on that topic or related to that umbrella, maybe that should be the focus, and I should be able to associate with that. So that information should be given to me that way, whether through marketing or by you telling me about the app [with] your focus: what is different [...] Adobe focuses on Photoshop, [while] this app focuses on this kind of thing. When we look at the brand, we see the association with who’s behind it, and then we figure out exactly what they need.”

P16 wanted to know whether a chatbot was intended to respond empathically and requested a toggle to turn empathic responses on or off:

“I’d want to make sure that I knew whether they were trying to have empathy or not. I’d wanna kind of toggle that on and off somewhere on the chat.”

Participants also wanted clearer information about how their personal disclosures were handled. Some called for industry standards to make information about data collection and use easier to find. About half wanted to know who could access their information and conversations. P6 emphasized transparency about third-party sharing, storage, and retention:

“I think companies should have to disclose, and if they’re selling data to third parties, they should disclose it. The third parties should have to disclose what they’re doing with the data, how it’s being stored, and how long it’s being stored.”

P15 also wanted independent verification of privacy claims, particularly whether conversations were deleted as promised:

“Companies can say the data is gone, but until it’s really been confirmed that it’s gone, you don’t really know [...] For me, if I knew, ‘Okay, yeah, they’ve stated that these temporary chats are deleted as soon as it’s over, and it’s been third-party verified by such-and-such agency or watchdog group,’ that would give me a whole lot more confidence.”

Enhanced authentication for conversations with empathic AI. Some participants emphasized the importance of stronger authentication measures, such as password protection, to safeguard empathic conversations with AI chatbots. They viewed these security features as essential for ensuring that sensitive or emotionally charged exchanges remain private and accessible only to the user. P13 explained:

“It would be nice if there were ways to lock some folders or conversations as private, and I’d want them to stay private no matter what. I’d need a password on those files.”

Automatic data deletion from empathic AI chatbots. A few participants emphasized the importance of automatic data deletion to protect users who may disclose sensitive information when engaging with empathic AI chatbots. They believed that regular, default data deletion policies could mitigate privacy risks arising from emotionally charged interactions. P12 proposed implementing periodic deletion as a standard safeguard:

“Because people might share sensitive information with empathic chatbots, so nobody should have access to, or the information will be periodically deleted after a few months or year, compared to non-empathic chatbots.”

Summary for RQ3: What are users’ expectations toward privacy-enhancing empathic AI interactions?

Participants preferred AI empathy in personal and emotional contexts but wanted less empathy when prioritizing efficiency or objective input. They wanted different combinations of empathy components, alongside control over the types and levels of empathy. They also sought control over how previously shared information was reused across contexts to avoid misplaced empathy. Their privacy expectations included clearer information about chatbots’ empathic purpose and data practices, independent verification of privacy claims, password protection for sensitive conversations, and automatic data deletion.

Discussion

Summary of findings. Participants collectively conceptualized empathy through five components—understanding, resonating, caring, expressing, and helping. Although participants differed in how much they emphasized or omitted these components when describing empathy, almost all distinguished AI empathy from human empathy by reporting a perceived absence of resonating and caring in AI chatbots (RQ1, § 4.1). Yet recognizing this distinction did not prevent participants from valuing AI empathy or disclosing sensitive information. The in-depth understanding, validating and non-judgmental expressions, and helpful responses they perceived from AI chatbots jointly shaped their disclosure comfort, trust, and concerns about manipulation. A few also associated AI empathy overall with stronger expected privacy protections. Almost all reported sharing more information in empathic interactions, with some disclosing highly sensitive details (RQ2, § 4.3). These experiences help explain participants’ desire to retain the benefits of empathy while gaining greater control over empathic interactions and their privacy implications. They preferred empathy in personal and emotional contexts but less when prioritizing efficiency or objective input, and wanted controls to accommodate different combinations and levels of empathy. They also sought control over information reuse across contexts to avoid misplaced empathy. Participants also requested clearer information about empathic purpose and data practices, independent verification of privacy claims, and stronger protections for empathic AI interactions (RQ3, § 4.5). Below, we discuss how our findings extend prior work (§ 5.1) and how understanding users’ preferences for empathy in AI chatbots can inform tailored privacy protections (§ 5.2). We then present recommendations for privacy-enhancing empathic AI interactions (§ 5.3).

Privacy Implications of AI Empathy Through Users’ Conceptions Beyond Human-Likeness

Grounding AI empathy in users’ own accounts informs a more granular analysis of its privacy implications. To understand the privacy implications of AI empathy, we first examined what users themselves consider empathic (§ 4.2). Prior research has distinguished cognitive empathy from affective empathy and proposed additional components to characterize the construct [40, 41, 42, 71, 107]. In addition, empirical research suggests that cognitive and affective empathy can shape users’ perceptions of AI chatbots in different ways [55]. For example, cognitive empathy may be perceived as more helpful in emotional support contexts than in interactions focused on efficiency [45, 67, 75], whereas expressions of affective empathy may be perceived as inauthentic and discourage continued use [13, 79]. However, the distinction between cognitive and affective empathy may not fully capture the diversity of users’ conceptions. Research on lay conceptions shows substantial variation in how people define empathy and which aspects they emphasize [47], yet little work has examined how lay users of AI chatbots describe AI empathy. Together, this diversity of perceived empathy dimensions and their distinct influences on users’ perceptions suggest that how these dimensions shape users’ privacy perceptions, behaviors, and needs in AI chatbot interactions may not have been fully examined.

We therefore extended this line of work by inviting participants to describe empathy in their own words and used the resulting taxonomy to ground a more granular analysis of users’ privacy experiences when perceiving empathy from AI chatbots. The five components we identified—understanding, resonating, caring, expressing, and helping—were perceived differently in the context of AI interactions (§ 4.1). The perceived presence or absence of each component shaped participants’ mental models of AI chatbots’ privacy practices, disclosure behaviors, and efforts to mitigate privacy risks in different ways (§ 4.3). Thus, distinguishing these components provides a basis for designing more targeted privacy protections, which we expand on in § 5.2.

Increased privacy risk despite perceived differences from human empathy. One common assumption in prior work is that human-like features, such as chatbots’ emotional expressions [70] and active-listening cues [20], encourage users’ self-disclosure. At the same time, AI’s human-likeness has drawn widespread public concern [51], and recent legislation now requires AI companions to state, at the start of an interaction and periodically thereafter, that the user is not communicating with a human [83].

However, our findings suggest that even cognitively understanding the differences between AI and human empathy still shaped disclosure and beyond. Specifically, participants identified the absence of resonating and caring as a key reason for viewing AI empathy as artificial, yet described mixed effects of this perceived absence on their willingness to disclose (Sec 4.2.1). In addition, some participants viewed chatbots’ contextual understanding and helpfulness, including the quality of their advice, as sometimes exceeding what an average person could provide (Sec 4.1.2). They described willingly providing further personal contexts to benefit from these instrumental aspects of empathy, not necessarily because they perceived the chatbots as human-like (§ 4.4). Thus, our findings indicate that the privacy risks of AI empathy are unlikely to be adequately addressed by simply reminding users that AI empathy differs from human empathy. Instead, there is a need for more nuanced solutions that account for different levels and types of empathy, aligning them with users’ preferences in different contexts. Toward this end, we detail our recommendations in § 5.3.

Beyond self-disclosure, our findings also suggest that perceived empathy from AI chatbots may create a false sense of privacy protection. A few participants assumed that their privacy was better protected when they perceived empathy from AI chatbots (Sec 4.2.1). At the same time, almost all participants reported taking no privacy-protective measures despite recognizing privacy risks (§ 4.4). While participants did not explicitly explain this inaction or attribute it to feeling protected, prior work suggesting that privacy settings can create a false sense of protection [12] offers a possible parallel. Perceived empathy may similarly give users a sense of protection that reduces their perceived need to take protective actions, such as reviewing or adjusting privacy settings. This possibility warrants further investigation into how perceived empathy may affect users’ privacy behaviors beyond disclosure and highlights a need for transparency around the privacy practices of empathic AI interaction, which we expand on in § 5.3.

Together, these findings suggest that AI empathy can encourage sensitive disclosure and pose privacy risks even when users recognize its artificiality and distinguish it from human empathy.

Tailoring Privacy Protections to Users’ Preferences for AI Empathy

Our participants described preferences for different components and levels of empathy across interaction contexts (§ 4.5), and we argue that these preferences should inform the amount and type of information chatbots request. Consistent with prior work [23, 45, 67], most participants preferred minimal or no empathy in efficiency-driven contexts, while preferring appropriate levels of empathy in personal and emotional contexts (§ 4.5.1). Within contexts where they wanted empathy, participants further specified which components they preferred. For example, when participants primarily sought validation, they did not necessarily want more in-depth, reasoned input (Sec 4.3.2). In these cases, validating expressions alone could support their goals without probing for additional sensitive details to formulate advice.

In other contexts, participants preferred in-depth, reasoned input over emotional validation (Sec 4.3.2), yet they lacked controls over empathic behavior and had little insight into the underlying data practices at the time of the interviews (§ 4.5.2). As a result, users have to either disengage from the interaction or continue sharing information without greater transparency or additional privacy protections. These experiences highlight an opportunity to align both chatbots’ information requests and data practices with users’ preferred empathy components and levels. In particular, the depth of understanding a chatbot seeks should reflect the support users want in a given context. Although ChatGPT now offers personalization features that allow users to indicate their preferred personality or tone [87], there remains an opportunity to pair these preferences with data-minimization defaults tailored to different modes. To address this gap, we propose modes that emphasize different empathy components and levels, paired with tailored privacy practices, in § 5.3.

Recommendations

We propose three directions for designing safer, privacy-enhancing empathic AI interactions. Notably, these directions follow from exploratory findings, and future work should evaluate their effectiveness in practice.

Supporting user-controllable empathic AI interactions across contexts. Our participants described preferences for different levels and components of empathy across interaction contexts. Most preferred minimal or no empathy in efficiency-driven contexts, while seeking varied forms of empathic support in personal and emotional contexts (§ 4.5). Accordingly, these preferences suggest an opportunity to give users control over both chatbots’ empathic behavior and the information they request to provide desired support.

We recommend allowing users to first select their desired level of empathy, for example, through a slider or a set of options ranging from no empathy to more extensive empathic engagement. When empathy is desired, users could further specify their preferred components. The five components identified in our findings—understanding, reasoning, caring, expressing, and helping—could inform the organization of these options. For example, users might prioritize validating expressions when venting or prefer in-depth understanding and reasoned input when seeking advice. These preferences could apply to individual sessions, be saved for particular contexts, or serve as global defaults, with opportunities to adjust them as users’ goals change.

Participants’ concerns about sensitive information resurfacing (Sec 4.3.2) also suggest the value of configurable topic filters. These filters could allow users to specify information, experiences, or topics that a chatbot should avoid mentioning, either at all times or during a user-defined period. Separately, controls over cross-context information use could allow users to prevent details shared in one empathic context from being reused in another. These boundaries could apply to individual sessions or persist across interactions, allowing users to receive their preferred forms of empathic support while limiting unwanted information requests and reuse.

Establishing privacy-enhancing defaults for empathic interactions. While user controls can give people greater agency to tailor their experience, privacy-protective defaults for both data practices and system behavior are especially important when users are less aware of privacy risks. Participants, for instance, described how empathic expressions could subtly lower their guard (Sec 4.2.1). We therefore recommend that developers adopt such defaults: constraining, through system prompts, behaviors that invite sensitive disclosure, and minimizing the data collected, retained, and used.

For example, when users seek active listening or validation, chatbots should draw on what has already been shared rather than probing for additional personal details in order to offer unsolicited advice. If the desired form of support is unclear, chatbots should ask whether the user wants listening or advice, without requesting further sensitive information. When additional context is necessary for the requested support, chatbots should explain why it is relevant, let users decline, and offer to continue with less specific information when possible. A preference for more empathic engagement should not be treated as permission to solicit increasingly sensitive details.

Beyond minimizing collection, as requested by our participants (§ 4.5.2), developers should also strengthen how disclosed data is retained and secured: deleting sensitive interaction data after a limited retention period, and applying encryption and other technical safeguards against unauthorized access.

These recommendations can also inform regulators and auditors overseeing AI safety. Audits could assess, for example, whether chatbots solicit unnecessary personal details when users seek validation, continue probing after users decline to elaborate, or surface sensitive information from other contexts without permission. Such audits should examine system behavior across turns and sessions to determine whether these protections are applied consistently, including when users have not asked for them.

Making empathic behavior and data practices transparent. Participants described uncertainty about the underlying data practices (§ 4.5.2), even as their experience of empathic behavior shaped their assumptions about privacy protections (Sec 4.2.1). We therefore recommend that developers make the current empathy configuration, the behavioral rules it implies, and the data practices that apply to it visible within the conversation interface.

For example, similar to the idea of privacy labels that detail important data practices [60], an empathy label could display the currently selected empathy level and components, providing a persistent indication of the interaction settings. Users could expand this label to view more detailed explanations of the expected behavior, including whether the chatbot may request additional personal information, draw on previous conversations, or mention topics covered by their exclusions. This expanded view could provide plain-language explanations of the relevant system prompts and allow users to inspect and adjust their preferences. It should also indicate whether the configuration reflects global defaults or settings specific to the current session. By making these behavioral commitments visible, the interface could help users assess whether chatbot responses align with their selected preferences.

The label should additionally provide access to information about the data practices that apply to the current interaction, including who can access the conversation, where and for how long information is stored, how it is used, and whether and how encryption or other protections apply. When users change their empathy settings, the interface should clarify whether any data practices or protections change accordingly. This would help users evaluate the privacy implications of an interaction alongside the forms of empathic support they prefer.

Conclusion

Our findings show that empathic AI can shape users’ privacy experiences even when users do not perceive AI as genuinely human-like. Participants distinguished AI from human empathy through the perceived absence of resonating and caring, while still describing understanding, expressing, and helping as shaping disclosure, trust, and privacy expectations. By unpacking empathy into distinct components, our work highlights how different forms of perceived empathy create different privacy implications and motivates the design of user-controllable, privacy-protective, and transparent empathic AI interactions.

Study Artifacts

The interview consent form, screening survey, participants’ full information, and final codebook are linked at: https://osf.io/n3yfu/overview?view_only=2e3e9aded59946dd96277f6327e1de61.

Interview Questions

Introduction

Hi there! Thank you for participating in our study! My name is [REDACTED] and this is [INTRODUCE OTHER PERSON], who will be taking some notes in the background. In this interview, I will be asking you about your usage, understanding, and experiences with AI chatbots.

As mentioned in the consent form, we would like to audio-record this interview for analysis purposes. I will keep my video on throughout the interview process; but it is completely up to you if you would like to have your video on or off. May I confirm with you that you allow us to audio-record this interview? [Start the audio recording].

Thank you again. Let’s get started!

AI Chatbot Usage

  1. Do you use any AI chatbots? If so, could you name the products?

  2. Could you tell us the purposes for which you use these AI chatbots?

  3. How long have you been using these products, and how often do you use them now?

  4. Have you ever used any chatbots that you no longer use, and why?

User Understandings and Perceived Impacts

  1. In your own words, how would you define “empathy”?

  2. What would it mean to you for an AI chatbot to have empathy?

  3. What similarities and differences do you think there are between human and AI empathy?

  4. Have you ever had an interaction with an AI chatbot where you perceived it to be empathic? (a) What was the product, purpose of usage, and context? (a) What experiences during the interaction signaled to you that the AI had empathy? (i) With these indicators/characteristics, do you think the way you interact with the chatbot would differ if they didn’t show any such indicators? (a) Could you please tell us more to the extent that you are comfortable with this? (i) What benefits and concerns did you perceive?

  5. For what purposes or in which contexts would you want AI chatbots to have empathy, and why?

  6. For what purposes or in which contexts would you not want AI chatbots to have empathy, and why?

  7. What benefits / risks do you think empathic AI chatbots might have for users and why?

Privacy Attitudes

  1. How, if at all, do you think the information you would share with an empathic AI chatbot would differ from the information you would share with a non-empathic one?

  2. In your perspective, what are the risks and benefits of empathic AI chatbots for users’ privacy?

  3. How, if at all, would you expect these privacy risks and benefits to differ from non-empathic AI chatbots?

  4. What are the ways that you could manage these privacy risks?

  5. What challenges do you think users face when managing these privacy risks?

Regulation, Design, and Literacy Expectations

(17) Have you ever been informed about the privacy practices and risks of empathic AI chatbots?

(a) If so, what were the sources of information?

(18) What information do you think users should be told about empathic AI chatbots?

(a) What privacy information do you think users should be told about empathic AI chatbots?

(19) How do you think this information should be communicated to the user, and why?

(a) Specifically, when do you think users should be informed, and in what format?

(20) What advice would you give the empathic AI developers and designers to better safeguard users’ S&P?

(a) What additional features would you find valuable?

(21) What policies/regulations should be in place to safeguard users’ data S&P when interacting with empathic AI chatbots?

References

  1. [1]David Adam. 2025. Supportive? Addictive? Abusive? How AI companions affect our mental health. Nature 641, 8062 (2025), 296–298.DOI
  2. [2]Gülsüm Ançel. 2006. Developing empathy in nurses: An in-service training program. Archives of psychiatric nursing 20, 6 (2006), 249–257.
  3. [3]John W Ayers, Adam Poliak, Mark Dredze, Eric C Leas, Zechiah Zhu, Jessica B Kelley, Dennis J Faix, Aaron M Goodman, Christopher A Longhurst, Michael Hogarth, et al. 2023. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA internal medicine 183, 6 (2023), 589–596.DOI
  4. [4]Petter Bae Bae Brandtzæg, Marita Skjuve, Kim Kristoffer Kristoffer Dysthe, and Asbjørn Følstad. 2021. When the social becomes non-human: young people’s perception of social support in chatbots. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–13.DOI
  5. [5]C Daniel Batson. 2009. These things called empathy: Eight related but distinct phenomena. (2009).DOI
  6. [6]Dianne C Berry, Laurie T Butler, and Fiorella De Rosis. 2005. Evaluating a realistic agent in an advice-giving task. International Journal of Human-Computer Studies 63, 3 (2005), 304–327.DOI
  7. [7]Timothy W Bickmore and Rosalind W Picard. 2005. Establishing and maintaining long-term human-computer relationships. ACM Transactions on Computer-Human Interaction (TOCHI) 12, 2 (2005), 293–327.
  8. [8]Robert James R Blair. 2008. Fine cuts of empathy and the amygdala: dissociable deficits in psychopathy and autism. Quarterly journal of experimental psychology 61, 1 (2008), 157–170.pubmed.ncbi.nlm.nih.gov/18038346
  9. [9]Johanna K Blomster Lyshol, Rafael Valdece Sousa Bastos, Peder Mortvedt Isager, and Magnus H Blystad. 2025. What is empathy for laypeople?—A replication study of Hall, Schwartz, and Duong (2021). The journal of Social Psychology (2025), 1–19.
  10. [10]Magnus H Blystad and Steffen Hansen. 2022. Empathy or not empathy, that’s the question—A pragmatic behavior analytical approach. Behavior Analysis: Research and Practice 22, 2 (2022), 224.DOI
  11. [11]Jana Schaich Borg and Hannah Read. 2024. What Is Required for Empathic AI? It Depends, and Why That Matters for AI Developers and Users. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7. 1306–1318.arxiv.org/abs/2408.15354
  12. [12]Laura Brandimarte, Alessandro Acquisti, and George Loewenstein. 2013. Misplaced confidences: Privacy and the control paradox. Social psychological and personality science 4, 3 (2013), 340–347.DOI
  13. [13]Judee K Burgoon. 2015. Expectancy violations theory. The international encyclopedia of interpersonal communication (2015), 1–9.
  14. [14]Brant R Burleson. 2003. The experience and effects of emotional support: What the study of cultural and gender differences can tell us about close relationships, emotion, and interpersonal communication. Personal relationships 10, 1 (2003), 1–23.DOI
  15. [15]Jacky Casas, Timo Spring, Karl Dahler, Elena Mugellini, Omar Abou Khaled, and Philippe Cudré-Mauroux. 2021. Enhancing conversational agents with empathic abilities. In Proceedings of the 21st ACM international conference on intelligent virtual agents. 41–47.DOI
  16. [16]Chaoran Chen, Weijun Li, Wenxin Song, Yanfang Ye, Yaxing Yao, and Toby Jia-Jun Li. 2024. An empathy-based sandbox approach to bridge the privacy gap among attitudes, goals, knowledge, and behaviors. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–28.DOI
  17. [17]Siyuan Chen, Mengyue Wu, Kenny Q Zhu, Kunyao Lan, Zhiling Zhang, and Lyuchun Cui. 2023. LLM-empowered chatbots for psychiatrist and patient simulation: application and evaluation. arXiv preprint arXiv:2305.13614 (2023).arxiv.org/abs/2305.13614
  18. [18]Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Jurafsky. 2025. ELEPHANT: Measuring and understanding social sycophancy in LLMs. arXiv:2505.13995 [cs.CL] https://arxiv.org/abs/2505.13995
  19. [19]Hyojin Chin, Lebogang Wame Molefi, and Mun Yong Yi. 2020. Empathy is all you need: How a conversational agent should respond to verbal abuse. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–13.DOI
  20. [20]Eugene Cho, Nasim Motalebi, S Shyam Sundar, and Saeed Abdullah. 2022. Alexa as an active listener: how backchanneling can elicit self-disclosure and promote user experience. Proceedings of the acm on human-computer interaction 6, CSCW2 (2022), 1–23.DOI
  21. [21]Andrew J Cowell and Kay M Stanney. 2005. Manipulation of non-verbal interaction style and demographic embodiment to increase anthropomorphic computer character credibility. International journal of human-computer studies 62, 2 (2005), 281–306.DOI
  22. [22]Matthew Crowley, Aurélia Heitz, Annika Matta, Kevin Mori, and Banny Banerjee. 2011. Behavioral science-informed technology interventions for change in residential energy consumption. CHI’11 Extended Abstracts on Human Factors in Computing Systems (2011), 2209–2214.
  23. [23]Andrea Cuadra, Maria Wang, Lynn Andrea Stein, Malte F Jung, Nicola Dell, Deborah Estrin, and James A Landay. 2024. The illusion of empathy? notes on displays of emotion in human-computer interaction. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–18.
  24. [24]Benjamin MP Cuff, Sarah J Brown, Laura Taylor, and Douglas J Howat. 2016. Empathy: A review of the concept. Emotion review 8, 2 (2016), 144–153.
  25. [25]Jean Decety. 2005. Perspective taking as the royal avenue to empathy. Other minds: How humans bridge the divide between self and others 143 (2005), 157.
  26. [26]Mathias Dekeyser, Robert Elliott, and Mia Leijssen. 2009. Empathy in psychotherapy: Dialogue and embodied understanding. The social neuroscience of empathy (2009), 113–124.DOI
  27. [27]Daantje Derks, Arjan ER Bos, and Jasper Von Grumbkow. 2008. Emoticons in computer-mediated communication: Social motives and social context. Cyberpsychology & behavior 11, 1 (2008), 99–101.DOI
  28. [28]Ha Manh Do, Weihua Sheng, Erin E Harrison, and Alex J Bishop. 2020. Clinical screening interview using a social robot for geriatric care. IEEE Transactions on Automation Science and Engineering 18, 3 (2020), 1229–1242.
  29. [29]Luc Drouet, Kerstin Bongard-Blanchy, Vincent Koenig, and Carine Lallemand. 2022. Empathy in design scale: Development and initial insights. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–7.DOI
  30. [30]Nancy Eisenberg. 2002. Empathy-related emotional responses, altruism, and their socialization. Visions of compassion: Western scientists and Tibetan Buddhists examine human nature 135 (2002), 131–164.DOI
  31. [31]Nancy Eisenberg and Paul A Miller. 1987. The relation of empathy to prosocial and related behaviors. Psychological bulletin 101, 1 (1987), 91.pubmed.ncbi.nlm.nih.gov/3562705
  32. [32]Jakob Håkansson Eklund and Martina Summer Meranius. 2021. Toward a consensus on the nature of empathy: A review of reviews. Patient Education and Counseling 104, 2 (2021), 300–307.
  33. [33]Robert Elliott, Arthur C Bohart, Jeanne C Watson, and Leslie S Greenberg. 2011. Empathy. Psychotherapy 48, 1 (2011), 43.DOI
  34. [34]Robert Elliott, Arthur C Bohart, Jeanne C Watson, and David Murphy. 2018. Therapist empathy and client outcome: An updated meta-analysis. Psychotherapy 55, 4 (2018), 399.DOI
  35. [35]Pardis Emami-Naeini, Henry Dixon, Yuvraj Agarwal, and Lorrie Faith Cranor. 2019. Exploring how privacy and security factor into IoT device purchase behavior. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
  36. [36]Candi P Feller and R Rocco Cottone. 2003. The importance of empathy in the therapeutic alliance. The Journal of Humanistic Counseling, Education and Development 42, 1 (2003), 53–61.DOI
  37. [37]Norma Deitch Feshbach. 1975. Empathy in children: Some theoretical and empirical considerations. The counseling psychologist 5, 2 (1975), 25–30.DOI
  38. [38]Pamela Fitzgerald and Ivan Leudar. 2010. On active listening in person-centred, solution-focused psychotherapy. Journal of Pragmatics 42, 12 (2010), 3188–3198.DOI
  39. [39]Jill J Francis, Marie Johnston, Clare Robertson, Liz Glidewell, Vikki Entwistle, Martin P Eccles, and Jeremy M Grimshaw. 2010. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychology and health 25, 10 (2010), 1229–1245.
  40. [40]Shaun Gallagher, Antonino Raffone, and Salvatore M Aglioti. 2024. The pattern theory of compassion. Trends in Cognitive Sciences 28, 6 (2024), 504–516.DOI
  41. [41]Vittorio Gallese. 2003. The roots of empathy: the shared manifold hypothesis and the neural basis of intersubjectivity. Psychopathology 36, 4 (2003), 171–180.pubmed.ncbi.nlm.nih.gov/14504450
  42. [42]Uğur Genç and Himanshu Verma. 2024. Situating Empathy in HCI/CSCW: A Scoping Review. Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024), 1–37.DOI
  43. [43]Pamela Grimm. 2010. Social desirability bias. Wiley international encyclopedia of marketing (2010).DOI
  44. [44]Jingyu Guo, Jiajing Guo, Changyuan Yang, Yanqing Wu, and Lingyun Sun. 2021. Shing: A conversational agent to alert customers of suspected online-payment fraud with empathetical communication skills. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–11.DOI
  45. [45]Yuanyuan Guo, Linlin Xu, and Chaoyou Wang. 2025. Exploring the effect of empathic response and its boundaries in artificial intelligence service recovery. Journal of Retailing and Consumer Services 82 (2025), 104065.DOI
  46. [46]Judith A Hall and Rachel Schwartz. 2022. Empathy, an important but problematic concept. The Journal of Social Psychology 162, 1 (2022), 1–6.DOI
  47. [47]Judith A Hall, Rachel Schwartz, and Fred Duong. 2021. How do laypeople define empathy? The Journal of Social Psychology 161, 1 (2021), 5–24.
  48. [48]Glenn W Harrison and E Elisabet Rutström. 2008. Experimental evidence on the existence of hypothetical bias in value elicitation methods. Handbook of experimental economics results 1 (2008), 752–767.
  49. [49]Arthur Bran Herbenner and Malene Flensborg Damholdt. 2025. Are lonely youngsters turning to chatbots for companionship? The relationship between chatbot usage and social connectedness in Danish high-school students. International Journal of Human-Computer Studies 196 (2025), 103409.
  50. [50]Arthur Bran Herbener, Michal Klincewicz, and Malene Flensborg Damholdt. 2024. A narrative review of the active ingredients in psychotherapy delivered by conversational agents. Computers in Human Behavior Reports 14 (2024), 100401.DOI
  51. [51]Kashmir Hill. 2025. They Asked an A.I. Chatbot Questions. The Answers Sent Them Spiraling. The New York Times. https://www.nytimes.com/2025/06/13/technology/chatgpt-ai-chatbots-conspiracies.html
  52. [52]Annabell Ho, Jeff Hancock, and Adam S Miner. 2018. Psychological, relational, and emotional effects of self-disclosure after conversations with a chatbot. Journal of Communication 68, 4 (2018), 712–733.DOI
  53. [53]Martin L Hoffman. 2008. Empathy and prosocial behavior. Handbook of emotions 3, 440-455 (2008).
  54. [54]Tianran Hu, Anbang Xu, Zhe Liu, Quanzeng You, Yufan Guo, Vibha Sinha, Jiebo Luo, and Rama Akkiraju. 2018. Touch your heart: A tone-aware chatbot for customer care on social media. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–12.arxiv.org/abs/1803.02952
  55. [55]Yuetong Huang, Hongquan Chen, and Saixing Zeng. 2026. Beyond the Empathy Fallacy: A Meta-analysis of Artificial Empathy in Human-AI Interactions. In Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems. 1–5.DOI
  56. [56]Michael Inzlicht, C Daryl Cameron, Jason D’Cruz, and Paul Bloom. 2024. In praise of empathy AI. Trends in Cognitive Sciences 28, 2 (2024), 89–91.
  57. [57]Andreas Janson. 2023. How to leverage anthropomorphism for chatbot service interfaces: The interplay of communication style and personification. Computers in Human Behavior 149 (2023), 107954.
  58. [58]Sooyeon Jeong, Laura Aymerich-Franch, Sharifa Alghowinem, Rosalind W Picard, Cynthia L Breazeal, and Hae Won Park. 2023. A robotic companion for psychological well-being: A long-term investigation of companionship and therapeutic alliance. In Proceedings of the 2023 ACM/IEEE international conference on human–robot interaction. 485–494.
  59. [59]Susanne M Jones, Graham D Bodie, and Sam D Hughes. 2019. The impact of mindfulness on empathy, active listening, and perceived provisions of emotional support. Communication Research 46, 6 (2019), 838–865.
  60. [60]Patrick Gage Kelley, Joanna Bresee, Lorrie Faith Cranor, and Robert W Reeder. 2009. A "nutrition label" for privacy. In Proceedings of the 5th Symposium on Usable Privacy and Security. 1–12.
  61. [61]John D Kelly IV. 2020. Your best life: overcoming approval addiction. Clinical Orthopaedics and Related Research® 478, 8 (2020), 1733–1734.DOI
  62. [62]Sam Kessler. 2025. 'I Feel Like I'm Going Crazy': ChatGPT Fuels Delusional Spirals. The Wall Street Journal. https://www.wsj.com/tech/ai/i-feel-like-im-going-crazy-chatgpt-fuels-delusional-spirals-ae5a51fc
  63. [63]Merlijn Kouprie and Froukje Sleeswijk Visser. 2009. A framework for empathy in design: stepping into and out of the user’s life. Journal of Engineering Design 20, 5 (2009), 437–448.DOI
  64. [64]Guy Laban, Arvid Kappas, Val Morrison, and Emily S Cross. 2023. Opening up to social robots: how emotions drive self-disclosure behavior. In 2023 32nd ieee international conference on robot and human interactive communication (ro-man). IEEE, 1697–1704.DOI
  65. [65]Jieon Lee, Daeho Lee, and Jae-gil Lee. 2024. Influence of rapport and social presence with an AI psychotherapy chatbot on users’ self-disclosure. International Journal of Human–Computer Interaction 40, 7 (2024), 1620–1631.
  66. [66]Yi-Chieh Lee, Naomi Yamashita, and Yun Huang. 2020. Designing a chatbot as a mediator for promoting deep self-disclosure to a real mental health professional. Proceedings of the ACM on Human-Computer Interaction 4, CSCW1 (2020), 1–27.DOI
  67. [67]Yi-Chieh Lee, Naomi Yamashita, Yun Huang, and Wai Fu. 2020. "I hear you, I feel you": encouraging deep self-disclosure through a chatbot. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–12.DOI
  68. [68]Jingshu Li, Zicheng Zhu, Renwen Zhang, and Yi-Chieh Lee. 2025. Exploring the effects of chatbot anthropomorphism and human empathy on human prosocial behavior toward chatbots. Proceedings of the ACM on Human–Computer Interaction 9, 7 (2025), 1–29.DOI
  69. [69]Tianshi Li, Sauvik Das, Hao-Ping Lee, Dakuo Wang, Bingsheng Yao, and Zhiping Zhang. 2024. Human-centered privacy research in the age of large language models. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 1–4.arxiv.org/abs/2402.01994
  70. [70]Kai-Hui Liang, Weiyan Shi, Youjong Oh, Hao-Chuan Wang, Jingwen Zhang, and Zhou Yu. 2021. Dialoging resonance: How users perceive, reciprocate and react to chatbot’s self-disclosure in conversational recommendations. arXiv preprint arXiv:2106.01666 (2021).
  71. [71]Bingjie Liu and S Shyam Sundar. 2018. Should machines express sympathy and empathy? Experiments with a health advice chatbot. Cyberpsychology, Behavior, and Social Networking 21, 10 (2018), 625–636.DOI
  72. [72]Siru Liu, Allison B McCoy, Aileen P Wright, Babatunde Carew, Julian Z Genkins, Sean S Huang, Josh F Peterson, Bryan Steitz, and Adam Wright. 2023. Leveraging large language models for generating responses to patient messages. medRxiv (2023), 2023–07.DOI
  73. [73]Kate Loveys, Mark Sagar, and Elizabeth Broadbent. 2020. The effect of multimodal emotional expression on responses to a digital human during a self-disclosure conversation: a computational analysis of user language. Journal of medical systems 44, 9 (2020), 143.DOI
  74. [74]Gale M Lucas, Albert Rizzo, Jonathan Gratch, Stefan Scherer, Giota Stratou, Jill Boberg, and Louis-Philippe Morency. 2017. Reporting mental health symptoms: breaking down barriers to care with virtual human interviewers. Frontiers in Robotics and AI 4 (2017), 51.DOI
  75. [75]Xingyang Lv, Yufan Yang, Dazhi Qin, Xingping Cao, and Hong Xu. 2022. Artificial intelligence service recovery: The role of empathic response in hospitality customers’ continuous usage intention. Computers in Human Behavior 126 (2022), 106993.
  76. [76]Nikolas Martelaro, Victoria C Nnej, Wendy Ju, and Pamela Hinds. 2016. Tell me more designing HRI to encourage more trust, disclosure, and companionship. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 181–188.
  77. [77]Jim McCambridge, Marijn De Bruin, and John Witton. 2012. The effects of demand characteristics on research participant behaviours in non-laboratory settings: a systematic review. PloS one 7, 6 (2012), e39116.
  78. [78]Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on human–computer interaction 3, CSCW (2019), 1–23.
  79. [79]Jingbo Meng and Yue Dai. 2021. Emotional support from AI chatbots: Should a supportive partner self-disclose or not? Journal of Computer-Mediated Communication 26, 4 (2021), 207–222.DOI
  80. [80]Dan Milmo. 2025. ‘It cannot provide nuance’: UK experts warn AI therapy chatbots are not safe. The Guardian. https://www.theguardian.com/technology/2025/may/07/experts-warn-therapy-ai-chatbots-are-not-safe-to-use
  81. [81]Jared Moore, Declan Grabb, William Agnew, Kevin Klyman, Stevie Chancellor, Desmond C Ong, and Nick Haber. 2025. Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. 599–627.DOI
  82. [82]Jaya Narain, Tina Quach, Monique Davey, Hae Won Park, Cynthia Breazeal, and Rosalind Picard. 2020. Promoting wellbeing with sunny, a chatbot that facilitates positive messages within social groups. In Extended abstracts of the 2020 CHI conference on human factors in computing systems. 1–8.
  83. [83]New York State. 2025. Notifications, N.Y. Gen. Bus. Law § 1702. https://www.nysenate.gov/legislation/laws/GBS/1702. AI Companion Models, General Business Law Article 47; effective November 5, 2025.
  84. [84]Jacob B Nienhuis, Jesse Owen, Jeffrey C Valentine, Stephanie Winkeljohn Black, Tyler C Halford, Stephanie E Parazak, Stephanie Budge, and Mark Hilsenroth. 2018. Therapeutic alliance, empathy, and genuineness in individual adult psychotherapy: A meta-analytic review. Psychotherapy Research 28, 4 (2018), 593–605.
  85. [85]Nindzassao. 2025. Excessive emoji tsunami in CHATGPT conversations. https://community.openai.com/t/excessiveemoji-tsunami-in-chatgpt-conversations/1112668. OpenAI Community forum post.
  86. [86]Raquel Oliveira, Patrícia Arriaga, Fernando P Santos, Samuel Mascarenhas, and Ana Paiva. 2021. Towards prosocial design: A scoping review of the use of robots and virtual agents to trigger prosocial behaviour. Computers in Human Behavior 114 (2021), 106547.
  87. [87]OpenAI. 2024. Customizing Your ChatGPT Personality. Accessed: 2025-11-30. https://help.openai.com/en/articles/11899719-customizing-your-chatgpt-personality
  88. [88]OpenAI. 2025. Expanding on what we missed with sycophancy. https://openai.com/index/expanding-on-sycophancy/
  89. [89]Ana Paiva, Iolanda Leite, Hana Boukricha, and Ipke Wachsmuth. 2017. Empathy in virtual agents and robots: A survey. ACM Transactions on Interactive Intelligent Systems (TIIS) 7, 3 (2017), 1–40.DOI
  90. [90]Anat Perry. 2023. AI will never convey the essence of human empathy. Nature Human Behaviour 7, 11 (2023), 1808–1809.DOI
  91. [91]William Roberts and Janet Strayer. 1996. Empathy, emotional expressiveness, and prosocial behavior. Child development 67, 2 (1996), 449–470.DOI
  92. [92]Cheri Lucas Rowlands. 2025. ChatGPT Is Blowing Up Marriages as Spouses Use AI to Attack Their Partners. Longreads. https://longreads.com/2025/09/25/chatgpt-divorce-breakup/
  93. [93]Young June Sah and Wei Peng. 2015. Effects of visual and linguistic anthropomorphic cues on social perception, self-awareness, and information disclosure in a health website. Computers in Human Behavior 45 (2015), 392–401.DOI
  94. [94]Johnny Saldaña. 2021. The coding manual for qualitative researchers. (2021).
  95. [95]Benjamin Saunders, Julius Sim, Tom Kingstone, Shula Baker, Jackie Waterfield, Bernadette Bartlam, Heather Burroughs, and Clare Jinks. 2018. Saturation in qualitative research: exploring its conceptualization and operationalization. Quality & quantity 52, 4 (2018), 1893–1907.DOI
  96. [96]Abhinaya SB, Abhishri Agrawal, Yaxing Yao, Yixin Zou, and Anupam Das. 2025. \" What are they gonna do with my data?\": Privacy Expectations, Concerns, and Behaviors in Virtual Reality. Proceedings on Privacy Enhancing Technologies 2025, 1 (2025), 58–77.
  97. [97]Matthias Schmidmaier, Jonathan Rupp, Cedrik Harrich, and Sven Mayer. 2025. Using Nonverbal Cues in Empathic Multi-Modal LLM-Driven Chatbots for Mental Health Support. Proceedings of the ACM on Human-Computer Interaction 9, 5 (2025), 1–34.
  98. [98]Scribbr. 2023. Selection Bias | Definition, Types & Examples. Accessed: 2025-11-30. https://www.scribbr.com/research-bias/selection-bias/
  99. [99]Anna-Maria Seeger, Jella Pfeiffer, and Armin Heinzl. 2018. Designing anthropomorphic conversational agents: Development and empirical evaluation of a design framework. (2018).
  100. [100]Jocelyn Shen, Daniella DiPaola, Safinah Ali, Maarten Sap, Hae Won Park, Cynthia Breazeal, et al. 2024. Empathy toward artificial intelligence versus human experiences and the role of transparency in mental health and social support chatbot design: Comparative study. JMIR Mental Health 11, 1 (2024), e62679.DOI
  101. [101]Garriy Shteynberg, Jodi Halpern, Amir Sadovnik, Jon Garthoff, Anat Perry, Jessica Hay, Carlos Montemayor, Michael A Olson, Tim L Hulsey, and Abrol Fairweather. 2024. Does it matter if empathic AI has no empathy? Nature Machine Intelligence 6, 5 (2024), 496–497.
  102. [102]Jellie Sierksma, Jochem Thijs, and Maykel Verkuyten. 2015. In-group bias in children’s intention to help can be overpowered by inducing empathy. British Journal of Developmental Psychology 33, 1 (2015), 45–56.
  103. [103]Vera Sorin, Dana Brin, Yiftach Barash, Eli Konen, Alexander Charney, Girish Nadkarni, and Eyal Klang. 2024. Large language models and empathy: systematic review. Journal of medical Internet research 26 (2024), e52597.DOI
  104. [104]Sandra H Sulzer, Noah W Feinstein, and Claire L Wendland. 2016. Assessing empathy development in medical education: a systematic review. Medical education 50, 3 (2016), 300–310.DOI
  105. [105]Samuel Hardman Taylor, Dominic DiFranzo, Yoon Hyung Choi, Shruti Sannon, and Natalya N Bazarova. 2019. Accountability and empathy by design: Encouraging bystander intervention to cyberbullying on social media. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–26.
  106. [106]Nitasha Tiku. 2025. Your chatbot friend might be messing with your mind. The Washington Post. https://www.washingtonpost.com/technology/2025/05/31/ai-chatbots-user-influence-attention-chatgpt/
  107. [107]Amrisha Vaish and Tobias Grossmann. 2024. How sharp is the compassion–sympathy distinction? Trends in Cognitive Sciences (2024).DOI
  108. [108]Evert A Van Doorn, Gerben A van Kleef, and Joop van der Pligt. 2015. How emotional expressions shape prosocial behavior: Interpersonal effects of anger and disappointment on compliance with requests. Motivation and emotion 39, 1 (2015), 128–141.
  109. [109]Lesley Verhofstadt, Inge Devoldre, Ann Buysse, Michael Stevens, Céline Hineknens, William Ickes, and Mark Davis. 2016. The role of cognitive and affective empathy in spouses’ support interactions: An observational study. PloS one 11, 2 (2016), e0149944.DOI
  110. [110]AL VISIO and N VE. [n. d.]. Emotional risks of AI companions demand attention. (n. d.).DOI
  111. [111]Bruce E Wampold. 2015. How important are the common factors in psychotherapy? An update. World psychiatry 14, 3 (2015), 270–277.DOI
  112. [112]Bruce E Wampold and Zac E Emel. 2015. The great psychotherapy debate: The evidence for what makes psychotherapy work. Routledge.DOI
  113. [113]Jeanne C Watson. 2016. The role of empathy in psychotherapy: Theory, research, and practice. (2016).DOI
  114. [114]Marylynn Wei. 2025. Can AI Be Your Therapist? New Research Reveals Major Risks. Psychology Today (Urban Survival). https://www.psychologytoday.com/us/blog/urban-survival/202505/can-ai-be-your-therapist-new-research-reveals-major-risks
  115. [115]Marylynn Wei. 2025. The Emerging Problem of “AI Psychosis”. Psychology Today (Urban Survival). https://www.psychologytoday.com/us/blog/urban-survival/202507/the-emerging-problem-of-ai-psychosis
  116. [116]Marcus Williams, Micah Carroll, Ahdyyan Narang, Constantin Weisser, Brendan Murphy, and Anca Dragan. 2024. On targeted manipulation and deception when optimizing llms for user feedback. arXiv preprint arXiv:2411.02306 (2024).arxiv.org/abs/2411.02306
  117. [117]Jie Yang, Hirofumi Kikuchi, Takatsugu Uegaki, Kaito Moriki, and Hideaki Kikuchi. 2022. The effect of the repetitive utterances complexity on user’s desire to continue dialogue by a chat-oriented spoken dialogue system. In Proceedings of the 10th International Conference on Human-Agent Interaction. 51–56.DOI
  118. [118]Yingying Yin and Yong Wang. 2023. Is empathy associated with more prosocial behaviour? A meta-analysis. Asian Journal of Social Psychology 26, 1 (2023), 3–22.DOI
  119. [119]Xiao Zhan, Juan Carlos Carrillo, William Seymour, and Jose Such. 2025. Malicious {LLM-Based} Conversational {AI} Makes Users Reveal Personal Information. In 34th USENIX Security Symposium (USENIX Security ’25). 61–80.
  120. [120]Zhiping Zhang, Michelle Jia, Hao-Ping Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li. 2024. “It’s a Fair Game”, or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–26.
  121. [121]Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang, Yanpeng Tong, and Bing Qin. 2023. Is chatgpt equipped with emotional dialogue capabilities? arXiv preprint arXiv:2304.09582 (2023).arxiv.org/abs/2304.09582

Paper details

vlnc:W_nt26djmm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents