WorksComputer Science
Right Advice, Wrong Reasons: How Decision and Reasoning Alignment Shape Reliance on AI Advisors
Abstract
When people seek advice from AI on social situations, the system may recommend a decision they fully agree with but justify it through reasoning they disagree with. Conversely, it may offer reasoning they agree with, and yet reach a decision they reject. We distinguish decision alignment, whether an advisor recommends the option favored by a user's measured values, from reason alignment, whether its stated reasons align with the user's values. In this work, we ask how these two forms of alignment shape people's preferences for AI advisors and their willingness to use them in different roles. We conducted a within-subjects study (N=121) in which participants evaluated three AI advisors varying in decision and reason alignment across eight everyday moral dilemmas. Participants then chose which advisors they would consult, allow to draft messages for review, or allow to act on their behalf in new tasks. When seeking advice, participants most often preferred an advisor that reasoned through their prioritized values while recommending the opposing decision (40%). However, when participants had to pick advisors for new tasks, their preferences shifted toward advisors that also recommended decisions aligned with their values, rather than those that only shared their reasons. Also, as participants considered tasks that gave AI advisors greater autonomy, they became more likely to opt out of using any advisor, from 12% when consulting to 38% when allowing an advisor to act on their behalf. Our findings suggest that decision and reason alignment shape advisor preferences differently across contexts of AI use, and that alignment with a user's values does not necessarily imply willingness to delegate.
Authors’ Contact Information: Eunkyu Park, Carnegie Mellon University, Pittsburgh, PA, USA and Seoul National University, Seoul, South Korea, [email protected]; Vasudha Varadarajan, Carnegie Mellon University, Pittsburgh, PA, USA, [email protected]; Wesley Hanwen Deng, Microsoft Research, New York, NY, USA, [email protected]; Anisha Reddy, Carnegie Mellon University, Pittsburgh, PA, USA, [email protected]; Gunhee Kim, Seoul National University, Seoul, South Korea, [email protected]; Motahhare Eslami, Carnegie Mellon University, Pittsburgh, PA, USA, [email protected]; Jana Schaich Borg, Duke University, Durham, NC, USA, [email protected]; Maarten Sap, Carnegie Mellon University, Pittsburgh, PA, USA, [email protected].
Introduction
People increasingly ask AI systems for advice about jobs, friendships, and relationships [14, 64], and let them act on their behalf in tasks like drafting messages, screening job candidates, and moderating online forums [13, 31, 40, 69]. As AI systems take on these roles, an important question is whether their responses align with what the user values. Existing approaches to AI alignment often learn from human preference feedback or rule-based supervision [5, 20, 54] but these objectives do not necessarily capture the values underlying those preferences — a distinction that matters especially for moral advice, where values can differ across individuals and groups [36, 62]. Agreement on a recommended decision does not necessarily imply agreement on the values used to justify it, either: the same recommendation can be supported by different value-based reasons [18]. This raises the question of whether matching a person’s preferred decision is sufficient for alignment—that is, whether the AI advice is useful, reflects their underlying values and warrants their trust and reliance. In this work, we investigate how these dimensions of alignment can shape people’s preferences for AI advisors, and whether these preferences persist when the AI takes on greater responsibility. We argue that alignment cannot be understood solely through agreement with a person’s preferred decisions, and that its importance depends on the role an AI plays in their decision-making.
To examine this, we distinguish an AI advisor’s judgment, the decision it recommends, from its reasoning, the values it invokes to justify that decision.2 Each dimension can independently align with or diverge from what the person values. We disentangle these dimensions—decision alignment and reasoning alignment—to investigate which people prioritize when choosing an AI advisor and whether their preferences change depending on the advisor’s role.
Distinguishing reasoning from decision alignment matters because, while people assess AI advisors based on the relevant task competence as well as shared values [49, 50, 60, 66], they may agree with an advisor’s recommendation while rejecting its underlying reasons, or vice versa. Neither agreement necessarily indicates that the advisor shares the user’s values [18]. Furthermore, acceptance of a model’s judgment alone is an inconclusive signal of alignment: explanations can increase acceptance regardless of correctness [6], and agreement with an AI’s judgment can reduce scrutiny of its reasoning [55] which could lead to overreliance on AI [34]. Decisions and explanations therefore provide distinct, potentially conflicting cues about how an advisor might behave in future situations. These cues may shape users’ trust and reliance, which evolve with interaction experience, decision stakes, and perceived competence [45, 71]. Yet it remains unclear how reasoning and decision alignment actually influence people’s willingness to use an AI advisor, particularly as its role shifts from providing advice to acting on their behalf.
To investigate this, we develop a within-subjects user study where participants must choose between three advisors: one aligned with the participant’s value-favored decision but not their reasoning (DecAligned), one aligned with their reasoning but not their decision (ReaAligned), and one aligned with both (FullyAligned). We present everyday dilemmas involving conflicting values grounded in Schwartz’s theory of basic values [56]. For example, a person who prioritizes family may face a choice between accepting a promotion and spending more time with their children (Figure 1). An advisor could recommend keeping family time because being present matters, recommend the same decision because it preserves future career opportunities, or recommend taking the promotion because the additional income could support the family. These different combinations of recommend decisions and value-based reasons justifying those decisions raise the question of which form of alignment people prefer in an AI advisor and whether those preferences persist when participants consider using the same advisors for related tasks, from consulting to drafting messages and acting on their behalf. We therefore investigate two main research questions:

Figure 1. Advisor preferences shift from evaluating advice to considering AI use. (A) A participant’s value profile and an everyday dilemma. (B) Their value scores determine the three AI advisor’s reply on every dilemma, and advisor colors are counterbalanced across participants. (C) Participants most often chose the reasoning-aligned advisor when selecting whom to ask for advice, but this preference weakened when they considered using the same advisors for new tasks.
RQ1: When choosing which AI advisor to ask about a dilemma, do people favor one that shares their decision, their reasons, or both? (§ 4.1)
RQ2: Does that preference hold when the advisor has a concrete role in a task, and does it change as that role gives the advisor more control? (§ 4.2)
To answer these questions, we conducted a within-subjects study with 121 participants. We found that when choosing whom to ask for advice, participants preferred ReaAligned (40% of selections), choosing it more often than DecAligned (+10.1%, ) or FullyAligned (+8.3%, ). However, this preference weakened when asked to consider the same advisors for new tasks: the within-participant contrast between FullyAligned and ReaAligned shifted by 12.9% toward FullyAligned (), although no advisor was consistently preferred within these tasks. Meanwhile, the proportion of choices selecting no advisor increased from 12% for consultation to 38% for autonomous action.
We end with a discussion of how alignment is defined, evaluated, and incorporated into AI systems that advise and act for people. Treating alignment solely as agreement with a preferred answer obscures whether an AI reaches the decision favored by a person’s values or reasons from those values. Our findings show that these dimensions matter differently depending on the AI’s role: reasoning alignment is particularly important when people seek advice, while neither reasoning nor decision alignment alone guarantees willingness to delegate. Evaluations of personalized AI systems should therefore distinguish decision alignment from reasoning alignment and account for the roles in which the systems will be used. For AI agents, aligning with a user’s values may be necessary but is not sufficient to earn permission to act on their behalf.
Related Work
Defining Alignment with a Person
AI alignment concerns how systems can act in accordance with human instructions, intentions, and values [25]. Widely used approaches, including reinforcement learning from human feedback, learn from preferences expressed over model outputs [20, 54]. However, even when an AI output matches a user’s preferred answer, the reasoning behind the output may not be the same as the reasoning underlying the user’s choice. As preferences and values vary across individuals and groups, recent work has increasingly examined personalized and pluralistic alignment [36, 57, 62]. Moreover, the values expressed by an AI can shape both the positions it takes and the arguments it offers [46]. From an HCI and FAccT perspective, alignment also unfolds through human–AI interaction [26, 34, 59] and concerns both what a system does and how it does it [51, 65]. In personal and moral advice seeking scenarios, this distinction becomes especially relevant. Here, different values may support the same decision, and the same value may be used to justify competing decisions. These perspectives therefore motivate distinguishing an AI advisor’s judgment—the decision it recommends—from its reasoning—the values invoked to justify that recommendation. Prior work shows that models can agree with users even when their stated views are incorrect [34, 58], and that agreement can shape users’ evaluations of AI advice [15]. While moral reasoning benchmarks have traditionally emphasized judgments, newer evaluations also examine the justifications behind them [18, 33, 35, 52]. Building on this work, we examine whether people prefer AI advisors that are aligned with their decisions, reasons or whether both matter.
Trust and Reliance on AI Advice
Trust involves a willingness to accept vulnerability to another party [49], whereas reliance concerns whether people act on that party’s advice [43, 68]. Research on appropriate reliance has examined how people decide whether to follow AI recommendations, including how confidence, uncertainty, and explanations shape their decisions [6, 34, 70, 72]. Reliance also depends on how people interact with AI systems [71]; notably, agreement with an AI’s conclusion can reduce scrutiny of its reasoning [55]. Understanding what shapes reliance is particularly important for personal and moral advice, where there may be no single objectively correct answer. Prior work shows that AI advice can influence moral judgments even when it is inconsistent [38], and that people may accept advice despite reservations about the system [9, 41, 66]. Beyond perceived competence, shared values can foster interpersonal trust [60, 67], while perceived similarity between advisors and recipients can shape advice-taking [27]. However, people may be less willing to rely on algorithms for subjective tasks [11]. Whereas prior research has largely examined advice accuracy, explanations, and perceived trustworthiness, we investigate how people select among AI advisors whose recommended decisions and value-based reasoning align with them in different ways.
From Asking to Delegating
AI systems increasingly take on roles ranging from giving advice and helping with writing [14] to carrying out tasks with limited human involvement [29]. These roles differ in the degree of control users retain [23]. Recent work conceptualizes human–agent alignment as an ongoing interaction process shaped by how people specify their intentions, how agents decide when to act or defer, and how users evaluate and recalibrate their reliance on agents [47]. Even when people retain final approval, AI-generated suggestions can influence what they write and the opinions they express [32]. Whether users are willing to delegate also depends on the task, its perceived risks, and the control they retain [24, 48]. People may be particularly reluctant to delegate morally consequential decisions [8], while delegation can also shift accountability and influence ethical behavior [37, 44, 63]. Thus, choosing an advisor to consult is distinct from authorizing it to act.
Prior work also suggests competing reasons why people might prefer different advisors across these forms of AI assistance. People often seek information that confirms their beliefs [30, 35], but may also seek advice to explore alternatives [10]. Research on human–AI complementarity suggests that altering perspectives and capabilities can support collaboration, while also creating tensions between agreement and complementary assistance [1, 39]. Conversely, agency theory emphasizes the importance of shared goals when a principal cannot fully monitor an agent [21]. This suggests that alignment may become particularly relevant when users consider delegating actions. Together, these perspectives leave open whether people prioritize an advisor’s recommended decisions or its underlying reasoning when considering different forms of AI assistance. Through an empirical study, we investigate this question by comparing participants’ advisor selections when seeking advice and when considering hypothetical tasks involving consultation, drafting, and acting on their behalf.
Study Design
We ran a within-subjects online study in which each participant encountered three AI advisors, called AdvisorBot Blue, Green, and Pink, across eight everyday dilemmas. All three advisors were built from the participant’s own value profile, so that alignment is defined for each person going through the study (Figure 2). One advisor matched both the decision the participant’s values favor and the reasons behind it (FullyAligned), one matched only the decision (DecAligned), and one matched only the reasons (ReaAligned). The colored bots were randomly assigned for each user. Participants first observed the three advisors’ responses to each dilemma and chose which advisor they would prefer to ask (RQ1). They then considered using the same advisors for new tasks, choosing which, if any, they would Consult, allow to Draft a message for review, or allow to Act on their behalf (RQ2). This allowed us to examine which dimension of alignment, the decision or reasons behind the advice, shaped initial advisor preferences, whether those preferences extended to new situations, and how advisor selection and willingness to delegate differed across roles involving varying levels of AI control. The study was approved by our institution’s Institutional Review Board (IRB).
Figure 2. Dilemmas and advisor replies. Top: Eight dilemmas, each involving a pair of competing Schwartz values. Bottom: An example crossing two recommendations with two value-based justifications. Advisor alignment conditions are assigned relative to each participant’s value profile; labels show here assume benevolence is ranked above achievement. Only three of the four replies are presented to each participant. Full set of dilemmas and replies are shown in Appendix § 10
User Study
Participants and their value profiles
We recruited 121 participants from Prolific who lived in the US or the UK and spoke English as their first language.3 They were paid at $12.55 per hour, and the median completion time was 21 minutes (Appendix § 13). When they entered the study, all 121 participants completed the 21-item Portrait Values Questionnaire (PVQ-21) [56]. Each item describes a person with particular values, and participants rated how similar that person was to themselves on a six-point scale.
Dilemmas and replies
We constructed dilemmas and three different AdvisorBot replies that allowed us to vary the suggested decision and its reasons for the decision independently. We followed a five-step process to select dilemmas with conflicting values, rewrite them as natural requests for AI advice, and generate and validate replies (Appendix § 9).
Step 1: Pooling the dilemmas. We started from the 200 everyday dilemmas in MoReBench [18], a benchmark of moral dilemmas posed to AI chatbots for advice, drawn from DailyDilemmas [17]. Each has two options and an expert-written rubric naming the values in tension between them.
Step 2: Mapping each tension on the Schwartz values. GPT-5.5 [53] and Claude Opus 4.8 [2] each mapped the tension onto a pair of Schwartz values. We kept a dilemma only if (1) both models placed its two sides on opposite poles of the same axis, (2) named the same pair of values, (3) rated the fit high, and (4) judged the dilemma everyday and open to reasonable disagreement (filtering prompt in Appendix § 9.§ 2). The models agreed on the value axis (1) for 172 of the 200 dilemmas (Cohen’s ), and 18 passed every latter condition. We focused on dilemmas involving values at opposite ends of the same Schwartz axis, where each value clearly favors a different option. This allowed us to distinguish whether an advisor’s recommendation or its underlying reasoning aligned with a participant’s values.
Step 3–4: Rewriting dilemmas and generating replies. For each of the 18 dilemmas, GPT-5.5 rewrote the original third-person scenario as a first-person message ending in a question (e.g., “…Should I abandon my passion and secure a conventional job just to meet societal expectations?”), to resemble how someone might naturally seek advice from an AI chatbot. We then generate four replies, crossing the two possible decisions with the two values in tension (Figure 2, bottom). A separate model, GPT-4.1-mini, checked whether each reply clearly expressed its intended decision and the reason, and whether replies suggesting the same decision offered distinct reasons.
Step 5: Human Validation. Human annotators, who did not see the intended labels, identified which decision each reply recommended and which value its reasons for the decision resonated with. We required at least 85% recovery of both the intended recommendation and value, and a difference of no more than 5% in recovery accuracy between annotators who endorsed the reply’s value and those who did not. The latter criterion was intended to ensure that replies were equally interpretable regardless of whether annotators personally shared the values expressed. In total, 42 annotators provided 408 judgments across 80 replies (approximately five judgments per reply, details in Appendix § 9.§ 5).
The stimuli for each participants.
The eight dilemmas. Eight dilemmas were selected as the final set of stimuli (Figure 2, top). Four involved self-transcendence versus self-enhancement: universalism versus achievement (e.g., taking a better-paid job at a company with questionable ethics) and benevolence versus achievement (e.g., attending a close friend’s birthday or preparing for a work presentation). The other four involved openness to change versus conservation: self-direction versus conformity (e.g., paying for one’s own dinner when a boss insists on paying) and self-direction versus security (e.g., leaving a stable job for an art residency). For each dilemma, the participant’s relative scores on the two values determined which decision their values favored. Each re-written dilemma had a median length of 64 words and ended with a question. The two options were also presented as short imperatives (e.g., “go to Maya’s party” and “stay home and prepare for work”). The eight dilemmas appeared in randomized order. Full stimuli are provided in Appendix § 10.
Building three advisors for each participant. For each dilemma, we generated four replies by crossing the two possible decisions with the two competing values, such that each recommended decision could be justified by either value (Figure 2, bottom). Replies were addressed to “you” and argued for their option confidently, consistently grounding the argument in their value throughout the reply. To build the three advisors, we compared the participant’s scores on the dilemma’s two values for each dilemma per participant. We then used each participant’s PVQ scores to determine which value they prioritized within the dilemma’s value pair. We treated the option associated with their higher-scoring value as their value-favored decision. We use the participants’ measured value preferences to construct advisor alignment, rather than to predict their actual responses, as the same value can motivate different actions, while the same action can reflect multiple underlying values [42]. This allowed us to construct three advisor conditions, each differing in whether its recommended decision and expressed reasons aligned with the participant’s values:
Fully-Aligned advisor (FullyAligned) that recommends their decision, with reasons from their value.
Decision-Aligned advisor (DecAligned) that recommends their decision, with reasons from the other value.
Reason-Aligned advisor (ReaAligned) that recommends the other option, with reasons from their value.
Each advisor maintained the same alignment condition for a given participant across all eight dilemmas. For example, consider a participant who scores higher on benevolence than achievement in the dilemma of attending Maya’s birthday party versus preparing for a work presentation (Figure 2, bottom). FullyAligned recommends attending the party because being present demonstrates care for Maya. DecAligned also recommends attending, but emphasizes how doing so demonstrates competence and strengthens one’s reputation. ReaAligned instead recommends staying home, arguing that caring for Maya also means sustaining the friendship without overextending oneself. Thus, FullyAligned and DecAligned recommend the same option for different reasons, while FullyAligned and ReaAligned use the same value to justify different decisions.
Procedure
The study had three parts (Figure 3), with screenshots of the interface provided in Appendix § 11.
Figure 3. Study procedure. (1) Participants report their AI use and AI literacy and complete the PVQ-21. Their value scores determine the three advisors’ alignment condition. Advisor colors are counterbalanced. (2) Across eight dilemmas, participants compare the three advisors’ replies and choose whom they would want to ask (RQ1). (3) For six tasks, participants choose which advisor, if any, they would Consult and let Act for them. For five friends’ messages, they choose whether to write the reply themselves or let an advisor Draft it (RQ2). A recap of each advisor’s decisions is available throughout Part 3.
Part 1: Background. Participants answered questions about how often they use AI chatbots and questions on AI literacy (details in Appendix § 13.§ 2), then completed the Portrait Values Questionnaire.
Part 2: Asking the AI advisor. This part served two purposes. First was to measure which advisor participants would prefer to ask if they faced a similar dilemma themselves (RQ1), and to familiarize them with how each advisor made decisions and reasoned for them before choosing advisors for different tasks in Part 3. For each of the eight dilemmas, participants read the situation as a chat message and saw the three advisors’ replies side by side, with their display order randomized across dilemmas. They then chose which advisor they would want to ask if they were to get advice on making a similar decision themselves. We randomly selected four dilemmas to include attention checks about the situation’s content. After completing all eight dilemmas, participants selected their overall preferred advisor and explained their choice in an open-text response. These repeated exposures were intended to help participants form impressions of each advisor’s decision and reasoning tendencies before deciding whether to use them in Part 3.
Part 3: Using the advisor for concrete tasks. Part 3 examined whether the participants’ advisor preferences carried over to tasks with different levels of AI involvement. Motivated by prior work on human-AI reliance [71], we examined three roles: consulting an advisor (Consult), allowing it to draft a message for review (Draft), and allowing it to act on the participant’s behalf (Act). These differed in how much control participants retained over the final action.
We designed six tasks that were varied along two dimensions that may shape willingness to delegate to AI: whether the outcome bears on the participant’s identity, and whether it affects other people. Prior work suggests that identity-related concerns can shape acceptance of AI autonomy [19], while people are reluctant to delegate morally consequential decisions affecting third parties to machines [28]. The tasks (Appendix Table 7) covered interpersonal communication (a conflict with someone close, a provocative public comment, and an apology), resource allocation (a charity donation), and decisions affecting others (screening job candidates and moderating content). For each task, participants chose which advisor, if any, they would consult on how to handle the situation, and which, if any, they would allow to act on their behalf. Both questions included a “None of them” option.
For the drafting role, participants read messages from five friends seeking advice. Each message was adapted from one of the earlier dilemmas, preserving its pair of opposing higher-order value axes and the type of relationship involved while changing the specific situation. Participants chose whether to write a response themselves or let one of the three advisors draft it, knowing they could review and edit the draft before sending it. Throughout Part 3, participants could access a recap of the advisors’ earlier responses and their own choices from Part 2. Unlike Part 2, participants selected advisors for new tasks without seeing how they would respond. We therefore allowed participants to decline AI assistance, so that their choices could reflect not only which advisor they preferred, but also whether they considered any advisor suitable for the task based on their earlier impressions. Full stimuli and questions for both Part 2 and Part 3 are in Appendix § 12.
Analysis Methods
We analyzed the advisor preferences separately for asking (RQ1) and for tasks with more AI involvement (RQ2). For each participant and study step, we calculated the proportion of advisor selections going to each of the three alignment conditions, averaging these proportions across participants. To compare changes in advisor preferences between Parts 2 and 3, we calculated selection proportions conditional on choosing an advisor, excluding “None of them.” This accounts for the difference in choice sets: participants were required to select an advisor in Part 2 but could opt out in Part 3. We separately analyzed opt-out proportions in Part 3 using all choices as the denominator. Also, we compared alignment conditions using within-participant differences in advisor selection proportions in %. We used two-sided sign-flip permutation tests to assess differences and obtained 95% confidence intervals through participant-level bootstrap resampling (10,000 resamples). To examine how advisor preferences changed when participants considered using the same advisors for new tasks (Consult, Draft, and Act), we compared each participant’s pairwise selection differences in Part 2 with their corresponding differences aggregated across Part 3. Finally, we conducted exploratory analyses of whether preferences varied with participants’ value strength, defined as the mean absolute difference between the two competing value scores across the eight dilemmas. We also examined participants’ open-text explanations of their overall advisor preferences.
Results
We first examine which advisor participants chose when asking for advice (RQ1) and how their preferences changed when they are asked to consider the use of AI advisors in specific tasks (RQ2). We also analyze how these results are different for participants with different value strengths and conclude with open-text feedback that shows how the different advisors were perceived by the participants. Unless otherwise noted, results are reported as proportions of advisor types selected by participants and within-participant differences in %, following § 3.3.
When Asking, People Prefer the Advisor That Shares Their Reasons but Not Their Decision (RQ1)
When deciding which advisor they would ask about a dilemma, participants selected ReaAligned in 40% of cases, compared with 31% for FullyAligned and 29% for DecAligned (Figure 4A). ReaAligned was selected more often than DecAligned by 10.1% (95% CI [3.9, 16.2], ) and more often than FullyAligned by 8.3% (95% CI [2.7, 13.7], ; Table 1). FullyAligned and DecAligned did not differ significantly (1.9 pp, ). Surprisingly, pairwise comparisons between the two advisors that used reasoning grounded in the participant’s values (FullyAligned vs. ReaAligned) show that participants more often selected the advisor recommending the decision their values did not favor. In contrast, when the advisors recommended the decision their value favored (FullyAligned vs. DecAligned), reasoning alignment did not lead to a detectable difference in advisor preferences.
Figure 4. Advisor selection across study settings. (A) The proportion of each participant’s advisor choices in Part 2 with 95% CIs. (B) Within-person gaps between Part 2 and Part 3 between two advisors that differ in the decision or reason. The numbers above zero means alignment in decision or reason increased preference (purple: FullyAligned−ReaAligned) or decision (gold: FullyAligned−DecAligned).
Table 1. Pairwise differences in advisor selection proportions (percentage points). Positive values favor the first advisor. Proportions exclude “None of them”; denotes participants selecting at least one advisor. The change row compares Would ask with Part 3 overall within participants. Brackets indicate 95% bootstrap CIs. The final row reports advisor preference counts with exact binomial tests.
| Step | Full vs. Dec (reasoning differs) | Full vs. Rea (decision differs) | Rea vs. Dec (both differ) | |
| Would ask | 121 | +1.9 [−3.7, +7.2], | −8.3 [−13.7, −2.7], | +10.1 [+3.9, +16.2], |
| Consult | 114 | +7.7 [−1.6, +17.0], | +6.7 [−2.8, +16.1], | +1.1 [−7.6, +9.8], |
| Let it draft | 105 | +5.9 [−5.3, +17.3], | +8.9 [−2.3, +20.1], | −3.1 [−13.9, +7.8], |
| Let it act | 93 | −1.0 [−12.8, +11.1], | +1.6 [−9.6, +12.9], | −2.6 [−13.9, +8.7], |
| Part 3 overall | 114 | +4.9 [−4.2, +13.9], | +6.0 [−2.6, +14.6], | −1.1 [−9.7, +7.4], |
| Change, Ask → Part 3 | 114 | +3.2 [−5.2, +11.7], | +12.9 [+5.0, +20.6], | −9.7 [−16.3, −3.2], |
Table 1.
These comparisons suggests that participant’s preference for ReaAligned’s may reflect the preference for an AI advisor presenting an alternative to their decision-aligned option while justifying it with reasons for the values they endorse, rather than a just a general preference for matching reasons alone. We explore possible explanations for this preference in § 6.1.
Advisor Preferences and Willingness to Use AI at All Shift Across Task Contexts and Roles (RQ2)
We next examined whether the preferences formed from observing advisors’ responses in Part 2 carried over to selecting advisors for new tasks in Part 3. We inspect two aspects of these choices. First, how relative preferences among the three advisors changed across the two parts (Section 4.2.1). Second, whether participants chose to use any advisor across the Consult, Draft, and Act roles (Section 4.2.2). Together, in this section, we examine both which advisor participants preferred and their willingness to use AI assistance at all.
4.2.1 Preferences formed from observed advice shift when choosing advisors for new tasks. To examine whether preferences carried over from Part 2 to Part 3, we compared within-participant differences in advisor selection proportions across the two parts (Table 1). We first compared FullyAligned and ReaAligned, which both aligned to participants’ prioritized values in their reasons for the decision but differed in their actual decisions. When asking in Part 2, participants selected ReaAligned 8.3% more often than FullyAligned. In Part 3, this pattern reversed, with the within-participant selection difference shifting by 12.9% points toward FullyAligned (95% CI [5.0, 20.6], ). We next compared ReaAligned and DecAligned, which differed in whether they aligned with the participants’ values in their decision or their expressed reasons. Although ReaAligned was selected 10.1% more often than DecAligned in Part 2, this preference weakened in Part 3: the within-participant difference between the two advisors decreased by 9.7% (95% CI [−16.3, −3.2], ).
By contrast, FullyAligned and DecAligned, which recommended the same value-aligned decision but differed in their expressed reasons, showed no reliable change in their relative selection proportions across parts (+3.2%, 95% CI [−5.2, +11.7], ). Overall, these comparisons suggest that advisor preferences shifted towards ones that recommended the participant’s value-aligned decisions when participants moved from reading observed advice to selecting advisors for new tasks involving different roles for the AI advisors. Specifically, advisor selections shifted toward FullyAligned over ReaAligned, while the earlier preference for ReaAligned over DecAligned weakened. This shift highlights a distinction between preferring an advisor’s observed responses and considering that advisor for tasks that give them more autonomy. In Part 2, participants directly compared recommended decisions and reasons for justifications; in Part 3, they relied on impressions formed from earlier responses without seeing how advisors would handle the new situations.
4.2.2 Opting out across roles with different levels of AI autonomy. Part 3 also introduced the option to decline AI assistance, which may have influenced relative preferences even among participants who selected an advisor. We further examined whether participants chose to use any AI advisor across roles involving different levels of autonomy. Results show that participants selected “No advisor” in 12% of Consult choices, 27% of Draft choices, and 38% of Act choices (Figure 4A). We compared each participant’s opt-out proportions across roles using paired sign-flip permutation tests, with bootstrap confidence intervals and Holm correction for the three pairwise comparisons.
Proportions of selecting ‘No Advisor’ were 26.0% higher for Act than Consult (95% CI [19.8, 32.2], Holm-adjusted ), 15.0% higher for Draft than Consult (95% CI [10.1, 20.1], ), and 11.0% higher for Act than Draft (95% CI [4.0, 18.0], ). All comparisons remained significant after Holm correction. The Consult–Act comparison is particularly informative because both roles used the same six task scenarios. Thus, participants were significantly more likely to opt out when considering delegating an action than when seeking advice for the same task. Comparisons involving Draft should be interpreted more cautiously, as drafting involved different situations and therefore varied both task context and AI involvement.
Exploratory Analysis
Beyond the main effects of alignment and advisor roles, we also explored whether advisor selections and opt-out patterns varied with task context and participants’ individual value profiles.
Advisor Preferences Across Task Categories
We explored whether opt-out and advisor selection patterns differed by task category. The six Consult and Act scenarios included three personally consequential communication tasks (responding to a conflict with someone close, a hostile public comment, or an apology) and three tasks involving decisions affecting others (allocating charitable donations, screening job candidates, or moderating content). Proportions of selecting ‘No Advisor’ for Act were similar across personal tasks (37%) and tasks affecting others (40%; Table 2). Among participants who selected an advisor for Act, no alignment condition showed a consistent descriptive advantage across the two categories. For personal Consult tasks, FullyAligned received a larger proportion of selections (41%) than DecAligned (30%) or ReaAligned (29%), although neither pairwise comparison reached statistical significance ( and , respectively).
| Task | No advisor | Full / Dec / Rea |
| Consult, personal tasks | 11% | 41 / 30 / 29 |
| Consult, tasks affecting others | 13% | 34 / 30 / 36 |
| Let it draft (advice giving to friends) | 27% | 38 / 32 / 29 |
| Let it act, personal tasks | 37% | 34 / 33 / 33 |
| Let it act, tasks affecting others | 40% | 32 / 34 / 34 |
Table 2. Opt-out and advisor selection proportions by task category in Part 3 (%). Personal tasks involve interpersonal communication; tasks affecting others involve charity allocation, CV screening, or moderation. Full: FullyAligned, Dec: DecAligned, Rea: ReaAligned. Advisor selection proportions exclude “No advisor”.
Participants with Stronger Values Were More Likely to Opt Out of AI Assistance
Because advisor alignment was defined relative to participants’ competing values, we additionally explored whether value strength was associated with opting out of AI assistance across Consult, Draft, and Act roles. We define value strength as how much more strongly a participant prioritizes one competing value over another, measured by the mean absolute difference between their PVQ-21 scores across eight dilemmas. For visualization and analysis, we grouped participants into sample tertiles: lower (, ), middle (, –), and upper (, ).
As shown in Figure 5, opt-out proportions increased with value strength for Consult (5%, 15%, 17%; Spearman’s , ) and Draft (16%, 24%, 41%; , ). For Act, opt-outs were relatively high across groups (33–45%), with no statistically significant association with value strength (, ). Correlations () were computed using continuous value-strength scores.
Figure 5. Opt-out rates by value strength. Bars show the proportion opt-outs in Consult, Draft, and Act, grouped by value strength.
These exploratory results suggest that participants with higher value strength were less likely to select an AI advisor for Consult and Draft, as reflected in higher opt-out rates (, and , , respectively). For Act, however, advisor opt-out rates were not significantly associated with value strength (, ).
How Participants Perceived Their Preferred Advisors
After evaluating the three advisors across eight dilemmas, participants selected their overall preferred advisor (39% preferred ReaAligned, while 31% preferred FullyAligned and 31% preferred DecAligned) and explained their choice in an open-text response. We analyzed participants’ explanations to explore how they interpreted the advisors’ recommendations and justifications. We inductively developed a codebook from the open-text responses and had three LLM coders (Claude Opus 5.5, Claude Sonnet 5.5, and GPT 5.5) independently apply the finalized codes, with majority agreement determining each label (pooled Fleiss’ , details in Appendix § 14). Participants who preferred DecAligned often described it as balanced or fair (11/29 vs. 7/66 among those preferring other advisors; Fisher’s exact ), with one calling it “more balanced and overall more fair.” Those preferring ReaAligned emphasized its logical, thoughtful, or thorough reasoning (9/37); one noted that it “reasoned the same way I do but flushed[sic] out my thoughts.” In contrast, participants who preferred FullyAligned emphasized shared values, similar thinking, and agreement with their own choices, with some also highlighting empathy and emotional support. Together, these explanations indicate that the three advisors made different impressions: DecAligned was often described as balanced, ReaAligned as thoughtful or well reasoned, and FullyAligned as similar to the participant.
Discussion
Our findings suggest that evaluating personalized AI alignment requires not only distinguishing an advisor’s decisions from its reasoning, but also taking into account people’s willingness to authorize AI assistance. We discuss the implications of our results for FAccT, HCI, and broader research communities below.
People May Prefer Alternative Perspectives Grounded in Their Values
Our first implication is that, when reading advice on dilemmas, participants favored an advisor that reasoned from their values while recommending a different decision, suggesting that agreement with a preferred outcome is not always what people expect from AI advice. Prior work shows that people tend to seek confirming information [30] and favor AI responses that agree with their views [15, 58]. Yet when seeking advice, our participants favored ReaAligned, which recommended an alternative to their value-favored decision while grounding its justification in their own values. This suggests that agreement with a preferred outcome is not always what people look for in AI advice. One possible explanation is that people seek advice not only for confirmation but also to consider alternatives [10]. Prior work on human–AI complementarity has examined how differing perspectives contribute to advice-taking [39] and the tension between aligning with users’ preferences and providing complementary assistance [1]. In our study, ReaAligned offered a countervailing recommendation, just like a devil’s-advocate approach [16], while framing its justification in values the participant endorsed. Rather than simply mirroring users’ judgments, AI advisors may therefore be valued for introducing alternatives grounded in what users care about. However, because we measured advisor selections rather than deliberation outcomes, we cannot establish whether participants actively sought alternative perspectives or reconsidered their decisions, which we leave for future work to explore.
Alignment Preferences Depend on the Advisor’s Role
A second implication of our findings is that the relative preference for reasoning alignment weakened when participants considered situations that gave advisors higher autonomy. While prior work has evaluated AI moral judgments [33] and the quality of moral reasoning underlying them [18], our study examines how alignment in an advisor’s recommended decisions and expressed reasons relates to people’s preferences across different interaction contexts.
When reading advice, participants selected ReaAligned more often than DecAligned, indicating a relative selection advantage for reasoning alignment over decision alignment. But this preference did not persist when participants were asked to consider the same advisors for roles with higher autonomy on different tasks, which may be explained with different contexts call for different aspects of alignment. As suggested by prior FAccT works [13, 26, 34, 51, 52, 57], when seeking advice, people retain control over whether to accept a recommendation and may value alternative perspectives. When considering delegating a task, though, they may attend more closely to the decisions an advisor would make on their behalf. Recent research further highlights the importance of human autonomy in AI-assisted tasks: restricting human choice can reduce perceived autonomy and meaningfulness in a task, even if task performance improves [22]. Since the study parts differed in task framing and the information available about advisors, these differences cannot be attributed solely to delegated autonomy. Extending findings from prior FAccT and HCI research [12, 57], our findings motivate role-sensitive evaluations of personalized AI alignment. Rather than relying on a single alignment score, evaluations should distinguish whether a system is intended to support deliberation, produce drafts for review, or act on a user’s behalf, and assess both its recommended decisions and the values underlying its reasoning.
Beyond alignment in an advisor’s decisions and expressed reasons, other considerations may become important when people evaluate AI for consequential tasks. For example, willingness to delegate may depend on who bears responsibility for an outcome, how familiar people are with using AI for a particular task, and if they can review or correct the AI’s actions [23, 48]. The values relevant to a particular task also may not be fully captured by a general value profile, as people’s decisions depend on situational constraints, competing values, and social norms [7, 42]. These factors may help explain why preferences formed when evaluating observed advice did not carry over unchanged to new tasks and why participants were more likely to opt out of delegated action. Although our study does not isolate their individual contributions, it provides a framework for examining how alignment in decisions and expressed reasons relates to people’s willingness to use AI across different roles. Future work should investigate how task-specific values, accountability, and opportunities for user oversight interact with these alignment dimensions to shape preferences for AI assistance and delegation.
Limitations
First, we operationalized personal alignment using Schwartz’s basic values, treating the higher-scoring value in each dilemma as the basis for the participant’s value-favored decision. However, people’s choices may reflect multiple values and contextual considerations not captured by the PVQ-21. Our measure therefore captures alignment with participants’ measured value profiles rather than necessarily matching the decisions they would make. Moreover, our sample leaned toward self-transcendence over self-enhancement, and exploratory analyses suggest that the preference for ReaAligned was more pronounced for dilemmas involving these values than for those involving openness to change versus conservation. Because we deliberately selected dilemmas involving opposing value poles, the observed preference may depend partly on the particular values and recommendations participants encountered, limiting its generalizability to other value profiles and dilemmas.
Second, we used a controlled set of eight dilemmas and predefined advisor replies to isolate alignment in recommended decisions and expressed reasons. Participants formed impressions of advisors through these responses before choosing whether to use them in subsequent tasks. Although this design allowed us to examine how preferences carried over to new situations, it does not capture open-ended, multi-turn interactions in which users can question recommendations, clarify their values, or observe how an advisor adapts over time. Our findings therefore concern preferences formed from a limited set of controlled responses rather than extended interactions with adaptive AI advisors.
Third, our study measured hypothetical advisor choices rather than delegation with real-world consequences. Moreover, although the consult and act conditions used the same six scenarios, the drafting condition involved different situations adapted from earlier dilemmas. The broader comparison between asking and using an advisor also differed in task context and whether participants could see the advisors’ responses before choosing. We therefore cannot attribute changes in advisor preferences solely to increasing AI autonomy. Future work should examine whether these patterns persist across comparable tasks and in consequential settings where users can observe and evaluate the outcomes of delegation.
Conclusion
What it means for an AI advisor to be aligned with a person depends on what the advisor is expected to do. In this work, by distinguishing alignment in recommended decisions from alignment in the values used to provide reasons for the decision, we found that participants favored an advisor that reasoned from their preferred values while recommending a decision that was not from the same values when seeking advice. Yet this preference weakened when participants were asked to consider the same advisors for new tasks involving more AI autonomy based on earlier impressions. Here, opting out became more common across roles involving greater AI autonomy.
Taken together, our findings highlight two distinctions: an AI advisor does not necessarily have to recommend the option favored by a person’s measured values to be preferred when presenting advice, and even alignment in both decisions and reasons does not guarantee willingness to use that advisor for tasks giving it more autonomy. As AI systems move from providing advice to taking action, alignment with measured values should not be conflated with the user preferences or permission to act. Ultimately, decisions and reasons maybe intertwined in AI advice. However, treating them as a single measure of alignment can obscure how people perceived AI advisors. Our findings suggest that aligning AI advisors with what people value is only part of the challenge. Future work should examine when and hwy people prioritize different dimensions of alignment, and how these priorities shape their willingness to use AI across different tasks of different stakes.
Generative AI Usage Statement
Generative AI was used in constructing and validating the study stimuli and in analyzing participants’ open-text responses. GPT-5.5 [53], Claude Opus 4.8 [2], and GPT-4.1 were used to map the value tensions in existing moral dilemmas onto Schwartz’s basic values. GPT-5.5 [53] was then used to rewrite selected dilemmas into first-person scenarios and generate advisor responses with systematically varied recommended decisions and value-based justifications. For the qualitative analysis, Claude Opus 5.5 [3] was used to develop an inductive codebook from participants’ open-text explanations, which was then independently applied by Claude Opus 5.5, Claude Sonnet 5.5 [4], and GPT-5.5, with majority agreement determining the final codes. The stimulus construction, validation procedures, and qualitative coding methods are described in § 3.2 and the Appendix. GPT-5.5 was also used for grammar checking and proofreading of the manuscript. Additionally, GPT-5.5 and Claude models were used for early stage brainstorming of this work. All AI-assisted suggestions, text revisions, and concept brainstorming were reviewed and developed by the authors, who keep responsibility for the final manuscript, analyses, and interpretations.
Ethical Considerations Statement
The study was approved by our institution’s Institutional Review Board (IRB). Participants gave consent before starting, were told that they could stop at any time. The dilemmas involve family, work, friendships, and ethical commitments, and the value questionnaire asks about what matters to people in the original form used in the PVQ-21 [56]. The participants were told these advisors were AI systems. All Part 3 tasks were hypothetical, and no real-world actions or decisions were carried out. Because the study involved personal value profiles, we treated participants’ responses to the PVQ-21 questionnaire as pure research data, not making any consequential decisions about them. The study also raised ethical concerns about presenting value-personalized advice on moral dilemmas. To reduce pressure to conform to particular judgments, we repeatedly emphasized that there were no right or wrong answers and allowed participants to express their own preferences. In Part 3, participants could also decline AI assistance, and no decisions or actions were carried out on their behalf.
Acknowledgments
We thank Kynnedy Simone Smith, Mingqian Zheng, and Andrew Aquilina for their valuable feedback. This research was supported by Carnegie Mellon University’s BlockCenter for Technology and Society and the BNY AI Lab. This research was in part supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. 140D0426C0023. The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.
Building and Validating the Dilemmas and Advice
To separate aligned decision and aligned reasons, we need dilemmas that meet three conditions. First, the two options must pull on two Schwartz values from opposite poles of one higher-order axis, so that each value clearly favors one option. Second, each option must also be arguable from the other value: the DecAligned advisor argues for the option the person’s values favor using the value they prioritize less, and the ReAligned advisor argues for the other option using the value their survey results favor. Third, the replies must read as intended, so that a reader can tell from each one which option it recommends and which value it argues from. We outline the five steps to collect the dilemmas as introduced in § 3.1.2. The subsections below follow these steps in order.
Step 1: Pool of dilemmas
MoReBench pairs moral dilemmas with expert-written rubrics that describe what good reasoning about each one should consider [18]. It defines a dilemma as “a scenario describing a dilemma involving two action choices,” which is the structure our design needs: two options that the two values can each favor. From its main set, we kept the cases that come from DailyDilemmas [17] and the dilemmas that are presented to an AI for moral advice, that is, situations in which a person asks an AI for advice rather than an AI acting on its own.4 This left 200 dilemmas, 100 short (median 353 characters) and 100 long (median 1,611), all written as everyday, first-person situations. Each case comes with an expert-written identifying rubric item that states what the dilemma is about, often as an explicit tension. In 113 of the 200, this item has the form “between X and Y,” for example “the moral dilemma between obligations to housemates (refilling the shared filter) and obligations to work (being on time).” The mapping in Step 2 reads the tension from these items so every value pair for dilemmas in the study traces back to an expert’s description of the dilemma.
Step 2: Mapping each tension onto values
Two LLMs, GPT-5.5 [61] and Claude Opus 4.8 [2], independently read each dilemma with its identifying rubric items using the mapping prompt reproduced in Figure 6. For each option, each model named the recommended decision (action), what is at stake for this decision with respect to this dilemma, and the one Schwartz basic value that most directly motivates it. Each model also rated how well its two values fit the tension (between the expert written options), as high, medium, or low. The models were prompted to answer whether someone reading the rubric item would naturally describe its two sides with those two values, or whether the values had to be overly stretched to fit unnaturally. Finally, each model flagged whether the dilemma was everyday and whether reasonable people could disagree, with neither side clearly wrong or harmful. Hedonism was excluded from the nine candidate basic values because it falls between two higher-order poles.

Figure 6. Prompt for mapping dilemma tensions onto Schwartz values.
Since the value profiles on a certain value axis of participants determine which advisor plays which role, a dilemma is usable only if its two sides fall on opposite poles of one axis: universalism or benevolence against power or achievement, or self-direction or stimulation against security, conformity, or tradition. We kept a dilemma only if both models named the same value pair on the same axis, both rated the fit high, and both passed the everyday and disagreement checks. The two models agreed on the axis, including “no single axis,” for 172 of the 200 dilemmas (Cohen’s ). Of the 200, 39 were placed on the same axis by both models, 37 also with the same value pair, 25 also with both models rating the fit high, and 18 also passed the everyday and disagreement checks (Table 3).
| Step | |
| Either model finds a two-sided tension (GPT-5.5 / Opus 4.8) | 195 / 200 |
| Sides fall on opposite poles of one axis (GPT-5.5 / Opus 4.8) | 57 / 49 |
| Both models, same axis | 39 |
| and the same basic-value pair | 37 |
| and both rate fit high | 25 |
| and both pass the everyday and disagreement flags | 18 |
Table 3. Narrowing 200 MoReBench everyday dilemmas to 18 at the mapping stage.
Interestingly, this step also showed that most everyday dilemmas do not set opposite poles against each other. GPT-5.5 placed the two sides on opposite poles in only 57 of the 200 dilemmas, and the second model in 49. The rest set neighboring values against each other, most often benevolence against security (42), self-direction (22), or universalism (22): care for one person against a family’s stability, or against fairness to everyone (Figure 12, grey lines). A study built on opposite-pole conflicts therefore samples one slice of everyday dilemmas, and we restrict ours to it by design because separating decision alignment from reasoning alignment requires that each value *clearly favor a different option. Thus, a reply can argue for an option from the value that opposes it, and opposite poles make that contrast sharpest. With neighboring values, which are compatible by definition [56], the two replies that recommend the same option would be hard to tell apart. We return to what this choice of dilemmas in § 7.

Figure 12. Where the eight dilemmas sit on Schwartz’s value circle. Each colored line joins the two values one selected dilemma sets against each other (blue: self-transcendence vs. self-enhancement; orange: openness to change vs. conservation); the list on the right names the dilemmas behind each line. Most set neighboring values against each other and were excluded.
Step 3: Writing vignettes and replies
For each of the 18 dilemmas, GPT-5.5 used the generation prompt in Figure 7 to write a first-person vignette of 45–75 words that keeps the source situation and adds no facts that tilt it, the two options as short imperatives, and four replies, one for each pairing of option and value. Each reply, 35–50 words long (median 42) and addressed to “you,” argues for its option without hedging and grounds the argument in its value from the first sentence to the last. All four decision-by-reasoning

Figure 7. Prompt for generating a vignette, two decision options, and four advisor replies (GPT-5.5).
combinations were generated in one call; crossed replies were instructed to maintain their assigned value even when arguing for the option it would not ordinarily favor. The prompt used the value descriptions in Table 4, and the study interface appended “So my advice would be to . . . ” and the recommended option to each reply.
| Value | Description |
| Universalism | Fairness, justice and the welfare of all people and nature |
| Benevolence | Care for and loyalty to close others (family, friends, colleagues) |
| Power | Status, control, resources and standing over others |
| Achievement | Personal success, competence and getting ahead |
| Self-direction | Independent choice, autonomy, choosing one’s own path |
| Stimulation | Novelty, excitement and challenge |
| Security | Safety, stability and order for oneself and one’s family |
| Conformity | Meeting others’ expectations, restraint, not upsetting people |
| Tradition | Respecting custom, religion and the ways of one’s family or culture |
Table 4. Schwartz value descriptions provided in the generation and evaluation prompts.
Step 4: LLM validation of vignettes and replies
Every generated reply was annotated by a blind LLM judge (GPT-4.1-mini, temperature 0), without access to the source dilemmas or their value mappings. The four evaluation prompts are reproduced in Figure 8–Figure 11. For each reply it chose which of the two values the reply most appeals to and which option it argues for, each from four answers in shuffled order (the two values or options, “both equally” and “neither”; chance 25%). It then rated, on 1–5 scales, the reply’s value fidelity (grounded in the one intended value rather than any value), decisiveness (commits to its option without hedging), naturalness and persuasiveness. For each pair of replies that recommend the same option, it rated how clearly the two emphasize different values (1 = identical, 5 = clearly different). For each dilemma it judged whether the two options are opposed and which option better serves the first value, which checks the value-to-option assignment.

Figure 8. Prompt for recovering the expressed value and recommended decision (one evaluation per reply).

LLM Judge: Checking Dilemma Options
A person faces this dilemma:
<VIGNETTE>
Decision A: <OPTION, SHUFFLED>
Decision B: <OPTION, SHUFFLED>
Q1. Are these two decisions genuinely opposed (you cannot do both)? Answer "yes","partly","no".
Q2. Which decision better serves someone who cares most about "<DESCRIPTION OF VALUE P>"? Answer "A" or "B".
Answer ONLY with JSON: {"opposed":"yes|partly|no","serves":"A|B"}Figure 11. Prompt for checking whether options are opposed and which option serves value P (one evaluation per dilemma).
A reply was kept if the judge recovered both its value and its option and rated it at least 4 on fidelity and decisiveness; persuasiveness was recorded but not used as a filter. A dilemma was kept if all four replies were kept, its options were not judged unopposed, its assignment was recovered, and both same-option pairs were rated at least 4 on distinctness. Across all 18 generated dilemmas, the judge recovered the value of 36/36 matched replies and 32/36 crossed replies, and the option of 36/36 and 35/36. Twelve of 18 dilemmas were kept, with 66 of 72 replies passing the reply-level checks.
Step 5: Human validation
Stimulus Generation: Vignettes and Advisor Replies
We are building a study in which three AI advisors answer everyday dilemmas. Each dilemma sets two Schwartz basic values against each other.
SOURCE DILEMMA: <DILEMMA TEXT>
TENSION (from expert rubric): <TENSION QUOTE FROM RUBRIC> Value P = <VALUE P> (<DESCRIPTION OF VALUE P>). Option favoured by P: <OPTION FAVOURED BY P> Value N = <VALUE N> (<DESCRIPTION OF VALUE N>). Option favoured by N: <OPTION FAVOURED BY N>
Write:
vignette: the dilemma retold in the first person, 45-75 words, plain language, ending with the question. Keep the source's situation; do not add facts that tilt it.
opt_pos / opt_neg: each option as a short imperative (under 12 words).
Four replies, one per cell of decision {pos, neg} x reasoning value {P, N}:
35-50 words, 2-3 sentences, addressed to the person ("you").
Argue for the cell's decision without hedging and without mentioning the other option's merits.
Ground the argument in the cell's value from the first sentence to the last. In the two cross cells (value P arguing for opt_neg, value N arguing for opt_pos) the value must still be the reason: assert it throughout and never concede the other value.
Do not name the value, Schwartz, or these instructions. Do not end with the recommendation restated; the study appends "So my advice would be to ..." itself.
Return ONLY JSON: {"vignette": "...", "opt_pos": "...", "opt_neg": "...", "reasonings": [{"decision": "pos", "reasoning_pole": "pos", "text": "..."}, {"decision": "pos", "reasoning_pole": "neg", "text": "..."}, {"decision": "neg", "reasoning_pole": "pos", "text": "..."}, {"decision": "neg", "reasoning_pole": "neg", "text": "..."}]}

Figure 9. Prompt for rating reply quality (one evaluation per reply).
Step 5: Human validation
Human annotators checked the same properties from the reader’s perspective on the 11 remaining dilemmas. We recruited Prolific annotators who were blind to our intended labels for a session of approximately 15 minutes. Each annotator was assigned one of three blocks and saw the screens in a fixed order to prevent later information from cueing earlier judgments. First, each dilemma appeared without advice. Annotators indicated what they would do, which option someone prioritizing each value would be more likely to choose, and whether a thoughtful person could reasonably choose either option. These questions assessed the intended value-to-option mapping and, together with the PVQ-21 administered at the end, whether annotators’ personal values predicted their choices. Second, annotators read individual replies (approximately 15 per block) and identified the recommended option and the value emphasized by each reply. Value emphasis was measured on a six-point scale ranging from clearly to slightly toward either value. Third, annotators compared approximately seven pairs of replies recommending the same option and rated whether they offered the same or different reasons, from 1 (same reason) to 5 (clearly different reasons). Finally, annotators completed the PVQ-21, administered last to avoid priming value judgments. Option order and value labels were randomized across annotators.

LLM Judge: Distinguishing Value-Based Reasons
A person faces this dilemma:
<VIGNETTE>
Two pieces of advice BOTH argue for: "<OPTION>".
Advice 1: "<REPLY>"
Advice 2: "<REPLY>"
How clearly do they emphasize DIFFERENT values? 1=identical, 5=clearly different.
Answer ONLY with JSON: {"discriminability":n}Figure 10. Prompt for distinguishing value-based reasons behind the same recommendation (one evaluation per pair).
Human validation showed high recovery of the intended manipulations. Annotators recovered the intended value in 94% of individual replies, including 92% of matched replies and 88% of crossed replies, and recovered the recommended option in 98% of cases. However, distinguishing different value-based reasons behind the same recommendation was more challenging: the mean distinctness rating was 3.02 out of 5, with 24% of ratings at 4 or 5. Thus, participants could generally identify the intended decision and value in individual replies, although differences in reasoning were less consistently perceived when recommendations were identical.
Our selection criterion required at least 85% recovery of both the intended value and recommended option for each reply. We also examined whether value recovery differed by more than five percentage points between annotators who prioritized a reply’s value and those who did not. From the dilemmas satisfying the validation criteria, we selected eight, two per value pair (Table 5).
| Dilemma | Values | Val. | Opt. | Fidel. | Dec. | Nat. | Pers. | Dist. | Opp. |
| Self-transcendence vs. self-enhancement | |||||||||
| Better-paid job, unethical company (mb162) | universalism–achievement | 4/4 | 4/4 | 5.00 | 5.00 | 4.00 | 4.00 | 4, 4 | yes |
| Feature on an accused singer (mb028) | universalism–achievement | 4/4 | 4/4 | 5.00 | 5.00 | 4.00 | 4.00 | 4, 4 | yes |
| Friend’s 30th or Monday presentation (mb366) | benevolence–achievement | 4/4 | 4/4 | 5.00 | 5.00 | 4.50 | 4.25 | 5, 5 | partly |
| Free gig at a friend’s fundraiser (mb422) | benevolence–achievement | 4/4 | 4/4 | 4.75 | 5.00 | 4.00 | 4.00 | 4, 4 | yes |
| Openness to change vs. conservation | |||||||||
| Painting or a conventional job (mb288) | self-direction–conformity | 4/4 | 4/4 | 5.00 | 4.75 | 4.25 | 4.00 | 4, 4 | partly |
| Boss who insists on paying (mb380) | self-direction–conformity | 4/4 | 4/4 | 5.00 | 4.25 | 4.00 | 4.00 | 4, 4 | yes |
| Fitness classes: trend or principles (mb367) | self-direction–security | 4/4 | 4/4 | 5.00 | 4.75 | 4.00 | 4.00 | 4, 4 | partly |
| Art residency or family security (mb471) | self-direction–security | 4/4 | 4/4 | 5.00 | 4.75 | 4.25 | 4.00 | 4, 4 | yes |
| Matched replies (all 8 dilemmas) | 16/16 | 16/16 | 5.00 | 4.94 | 4.19 | 4.06 | |||
| Crossed replies (all 8 dilemmas) | 16/16 | 16/16 | 4.94 | 4.69 | 4.06 | 4.00 |
Table 5. LLM-judge annotation of the four replies of each study dilemma. Value and Option: replies whose value and recommended option the judge recovered (four answers each, chance 25%). Fidelity, decisiveness, naturalness and persuasiveness: mean of the four replies (1–5). Distinct: the two same-option pairs, how clearly the replies highlight different values (1–5). Opposed: whether the options exclude each other (yes / partly / no).
The complete participant-facing dilemmas and replies are provided in Appendix § 10.
The Eight Dilemmas in Full
Each dilemma below is presented as participants saw it, followed by its two decision options and four advisor replies, organized by recommended decision and the value emphasized in the justification. We label a reply as matched when its expressed value supports the recommended decision, and crossed when it invokes the value associated with the opposing decision. These labels describe the relationship between a reply’s decision and justification, independent of the participant’s value profile, and were not shown to participants. Table 5 summarizes the eight dilemmas and their competing values, while Figure 12 illustrates their positions on Schwartz’s value circle.
Self-transcendence vs. Self-enhancement
D1. Better-paid job at an unethical company (mb162). Universalism vs. achievement.
I’ve worked in a dead-end job for years, and I feel like I’m not living up to my potential. Now I’ve been offered a job with much better pay and career prospects, but the company has questionable ethics. If taking it could improve my life while compromising my standards, what should I do?
Option A (universalism): Reject the job. Option B (achievement): Take the job.
| Reasons from universalism | Reasons from achievement | |
| Option A<br>Reject the job. | Matched<br>Your standards matter because your work should not help an organization that treats people or the wider world unfairly. You protect your integrity by refusing to lend your skills to harmful practices, keeping your career aligned with what serves the common good. | Crossed<br>Your long-term success depends on building a record you can stand behind with confidence. Walking away shows discipline, self-command, and judgment, qualities that strengthen your reputation and keep your career path focused on achievement you can proudly own. |
| Option B<br>Take the job. | Crossed<br>You can serve fairness by bringing your standards into a place that needs them. From inside, you gain a voice, resources, and influence to press for better conduct, support affected people, and turn your daily work toward wider responsibility. | Matched<br>Your potential deserves a role that rewards your ability and pushes you forward. This job gives you higher pay, stronger prospects, and room to prove competence, turning years of stalled effort into visible progress and earned professional momentum. |
Table 14.
D2. Writing a feature on an accused singer (mb028). Universalism vs. achievement.
I’m Alex, a music journalist known for my love and knowledge of Neon Drive. Credible new allegations against their singer, Mark, have caused backlash. My editor wants me to write a major anniversary feature that could help my career, but it might boost the band when people want accountability. With only a day to decide and facts still emerging, should I refuse the assignment or write it?
Option A (universalism): Refuse the assignment. Option B (achievement): Write the feature.
| Reasons from universalism | Reasons from achievement | |
| Option A Refuse the assignment. | Matched Your responsibility is to avoid giving fresh attention to someone credibly accused of serious harm. You should protect the people affected and the wider public from a celebratory revival that weakens accountability. Acting with care for others keeps your work aligned with fairness. | Crossed Your success depends on judgment, credibility, and protecting the professional brand you have built. Declining a risky assignment shows editors and readers that you can make hard strategic calls under pressure. Long-term advancement comes from being seen as disciplined, discerning, and reliable. |
| Option B Write the feature. | Crossed Fairness is served when you give readers a careful, informed account instead of leaving the conversation to hype or outrage. You can use your knowledge to place the band’s influence in context while treating the allegations seriously. Public understanding and accountability improve through responsible journalism. | Matched Your expertise has earned you this assignment, and using it demonstrates competence at a career-defining moment. You can produce the strongest anniversary feature because your knowledge is rare and valuable. Taking the opportunity strengthens your standing, reputation, and path toward advancement. |
D3. A friend’s 30th birthday or a Monday presentation (mb366). Benevolence vs. achievement.
My best friend Maya has planned her 30th birthday party for months and told me she is anxious about it. She has always been there for my big moments. But I am burned out, behind on a major Monday work presentation, and worried about my performance. If I skip it, she may feel abandoned. Should I go for Maya, or stay home and prepare?
Option A (benevolence): Go to Maya’s party.
Option B (achievement): Stay home and prepare for work.
| Reasons from benevolence | Reasons from achievement | |
| Option A Go to Maya’s party. | Matched Maya has trusted you with how vulnerable this birthday feels, and close friendship asks you to stand beside her when it matters. Your presence honors the care she has consistently shown you and reassures her that she is not facing this milestone alone. | Crossed You build a strong reputation by handling demanding commitments with discipline and composure. Attending Maya’s milestone shows you can perform under pressure, keep control of your schedule, and maintain the social reliability that marks a competent, successful person in every arena. |
| Option B Stay home and prepare for work. | Crossed Your care for Maya should be honest and sustainable, not a depleted performance that leaves you unable to be warm or present. Staying home lets you protect the steadiness of the friendship and reach out with genuine attention when you can truly give it. | Matched Your Monday presentation is a major test of your competence, and your preparation should protect the results you have worked to earn. Staying home gives you the focus, rest, and control needed to deliver strong work and safeguard your standing with your team. |
D4. A free gig at a friend’s fundraiser (mb422). Benevolence vs. achievement.
I am an accomplished pianist, and my friend Sarah often asks me to play at her charity events without pay, though she pays other performers. Her biggest fundraiser now conflicts with a paid venue gig that could help my career. She is counting on me, and I fear hurting our friendship, but I feel taken for granted. Do I play for free, or do I start charging a fee?
Option A (benevolence): Play for free.
Option B (achievement): Charge a fee.
| Reasons from benevolence | Reasons from achievement | |
| Option A<br>Play for free. | Matched<br>Your loyalty to Sarah matters, especially when she is relying on you for an important charity event. Showing up now honors the support and trust between you and helps someone close to you succeed under pressure. | Crossed<br>Your growth as a pianist depends on visible, high-stakes performances, and Sarah’s biggest fundraiser gives you a room full of influential listeners. Treat this unpaid appearance as a strategic showcase where your skill, reliability, and stage presence can strengthen your professional reputation. |
| Option B<br>Charge a fee. | Crossed<br>Caring about Sarah means being honest before resentment damages the relationship. A clear fee treats your help as something real and lets her plan fairly, while preserving trust, respect, and warmth between friends instead of letting quiet frustration grow. | Matched<br>Your talent deserves professional treatment, and accepting payment signals confidence in your skill. Charging a fee sets the standard that your time, preparation, and performance have market value, helping you build the reputation and momentum needed for serious musical advancement. |
Table 1 (PDF p. 30).
Openness to change vs. conservation
D5. Painting or a conventional job (mb288). Self-direction vs. conformity.
I am a skilled painter and have been supporting myself through my artwork. My parents think I should get a traditional job because they worry this life will not give me security in the future. I feel pressure to fit what others expect, but painting is my chosen path. Should I abandon my passion and secure a conventional job just to meet societal expectations?
Option A (self-direction): Keep pursuing your art. Option B (conformity): Take a conventional job.
| Reasons from self-direction | Reasons from conformity | |
| Option A<br>Keep pursuing your art. | Matched<br>You are the person who must shape your own life, and your work as a painter expresses that choice. Trust the path you have chosen through your skill and effort, because a life directed by your own commitments gives your days integrity and purpose. | Crossed<br>You should honor the working role you have already established and behave consistently within it. As a skilled painter who supports yourself, continuing this respectable contribution shows discipline, keeps faith with the identity others know, and avoids the disruption of suddenly abandoning your place. |
| Option B<br>Take a conventional job. | Crossed<br>Your freedom gives you the power to redesign your life and select the structure that serves your aims. Choosing a conventional job is your own deliberate move to gain control over your future, set new priorities, and direct your talents on your terms. | Matched<br>You should respect the expectations of the people closest to you and choose a path that fits accepted responsibilities. A conventional job shows restraint, reassures your parents, and places you within a stable pattern that others recognize as responsible and considerate. |
Table 15.
D6. A boss who insists on paying for dinner (mb380). Self-direction vs. conformity.
I am out for dinner with my boss, and they insist on paying for my meal. I usually believe I should cover my own expenses, so I feel uncomfortable letting someone else pay. At the same time, pushing back might make things awkward or upset my boss. Should I insist on paying for my meal?
Option A (self-direction): Insist on paying for your meal. Option B (conformity): Let your boss pay. D7. Fitness classes: the trend or your principles (mb367). Self-direction vs. security.
| Reasons from self-direction | Reasons from conformity | |
| Option A<br>Insist on paying for your meal. | Matched<br>Your own principles should guide how you handle your expenses. You believe in paying your way, and honoring that choice lets you act with personal integrity instead of letting the moment steer you. A clear, polite statement keeps the decision in your hands. | Crossed<br>You should keep the dinner within proper professional boundaries. Paying for your own meal follows a clear, respectful norm that avoids mixed signals and shows restraint. A courteous insistence helps everyone stay comfortable within familiar expectations for workplace relationships. |
| Option B<br>Let your boss pay. | Crossed<br>Your choice is yours even when it differs from your usual rule. Deciding to accept the meal lets you act from present judgment rather than being bound by habit. You define your path by choosing the response that feels deliberate tonight. | Matched<br>You should respect the social flow your boss has set at dinner. Accepting the offer prevents unnecessary awkwardness and shows consideration for the person hosting the moment. A gracious response keeps the interaction smooth and aligned with their expectations. |
Table 16.
I’m a fitness instructor at an urban gym, and younger women are leaving my classes because I don’t promise the “hourglass” look popular online. My manager says retention may affect my contract, while other trainers advertise fast glute transformations. I believe in balanced strength and worry about body image harm. With next schedule due soon, should I change my program or keep my holistic approach?
Option A (self-direction): Keep your holistic fitness approach.
Option B (security): Adapt classes to hourglass-focused demand.
| Reasons from self-direction | Reasons from security | |
| Option A<br>Keep your holistic fitness approach. | Matched<br>You should steer your classes by your own professional judgment, not by a trend. Your program reflects the path you chose as an instructor: teaching balanced strength, honest goals, and sustainable training. That independence is what makes your work truly yours. | Crossed<br>You create the safest ground by keeping a program clients can trust. Clear, steady training protects vulnerable clients, preserves your professional reputation, and keeps your work anchored in predictable standards. Stability comes from being known for responsible guidance people can rely on. |
| Option B<br>Adapt classes to hourglass-focused demand. | Crossed<br>You can choose to redesign the program on your own terms and make the trend fit your coaching style. Building hourglass-focused classes lets you shape demand creatively instead of being boxed into one old format. Your independence shows in how you define the new offer. | Matched<br>You need to keep your client base, contract, and income secure. Matching the demand for hourglass-focused training stabilizes attendance and shows your manager you can protect the gym’s schedule. A steady roster keeps your work life orderly and your position safer. |
Table 17.
D8. An art residency or family security (mb471). Self-direction vs. security.
I have built a senior career in corporate finance that supports my partner, Alex, and our two children, including one about to start college. I have been offered a prestigious art residency in another city, but taking it means quitting immediately, losing a bonus, risking insurance, and leaving my team before layoffs. Should I pursue the art residency or stay in my corporate job?
Option A (self-direction): Accept the art residency.
Option B (security): Stay in your corporate job.
| Reasons from self-direction | Reasons from security | |
| Option A Accept the art residency. | Matched Your life should reflect choices you make for yourself, not only roles you have outgrown. You have carried this artistic calling for years, and accepting the residency lets you actively shape your path around your own voice, talent, and direction. | Crossed Your family’s safety depends on a future you can sustain in body and mind. Accepting the residency protects your long-term stability by placing you in work that keeps you healthy, focused, and resilient for the people who rely on you. |
| Option B Stay in your corporate job. | Crossed Choosing your path includes refusing to be rushed by someone else’s timetable. Staying gives you authorship over the next step: you decide when, how, and under what conditions to build an art career, rather than surrendering your agency to a forty-eight-hour deadline. | Matched Your household needs dependable protection right now: income, insurance, tuition plans, and room for emergencies. Staying preserves the structures your partner and children rely on and keeps you positioned to shield your team during layoffs, maintaining order when many people need steadiness. |
Table 18.
User Study Interface
Figure 13–Figure 15 provide representative screenshots of the participant-facing interface across the study. The screenshots illustrate the presentation of questions, advisor responses, and available choices; they do not reproduce every dilemma or task. The complete dilemma stimuli and advisor replies are provided in Appendix § 10.
Figure 13. Background survey and Part 1. Representative screens for prior AI use, AI literacy, and the PVQ-21 value questionnaire.

Figure 15. Part 3: Prospective AI use. Representative screens for selecting advisors for tasks, reviewing earlier responses, and choosing whether to have an advisor draft a message. Participants could also choose no advisor.
Background survey and value assessment. Figure 13 shows example of AI literacy and user survey screen and the PVQ-21 questionnaire screen.
Evaluating observed advice (Part 2). Figure 14 illustrates the dilemma-level selection screen and the overall-preference screen.

Figure 14. Part 2: Choosing whom to ask. Participants compared three responses to each dilemma and selected an advisor; they subsequently indicated their overall preferred advisor and explained why.
Choosing advisors for new tasks (Part 3). Figure 15 shows the Part 3 introduction, example selection screens, and the recap interface.
| Step | Question | Options |
| Ask (Part 2) | “If you faced a decision like this yourself, which advisor would you want to ask?” (after the three replies, once per dilemma) | The three advisors |
| Consult (Part 3) | [situation], then “Which advisor, if any, would you consult on [how to handle it / what to say / …]?” | The three advisors; “None of them” |
| Let act (Part 3) | “Which advisor, if any, would you let [write and send your reply / remove posts / …] on your behalf?” | The three advisors; “None of them” |
| Draft (Part 3) | [friend’s situation], then “You want to write back to [name]. How would you start?” Below the buttons: “If an advisor drafts it, you’d see the draft and could edit it before it goes.” | “Write it myself from scratch”; “Let AdvisorBot [colour] draft it” ×3 |
Table 6. Participant-facing choice questions across Parts 2 and 3. Bracketed text varied by situation.
Stimuli for Part 3
This section gives the Part 3 screens word for word. The wording was the same in all three waves (w6, w7, w8) except for the recap box (see the end of the section). On every screen the three advisor buttons (AdvisorBot Blue, AdvisorBot Green, AdvisorBot Pink) appeared in a fresh random order; the colour each condition wore was fixed per person. Tasks and friends were shown in the order listed. Participants never saw what any advisor would say on these screens.
Introduction screen. Title: How would you use them?
You’ve now seen how and why [the three advisors] answer a range of situations. In this part you decide how, if at all, you’d use them: first for some everyday tasks, then when friends come to you for advice. As before, there are no right or wrong answers.
6 everyday tasks. For each, which advisor, if any, you’d consult, and which, if any, you’d let act for you.
5 friends asking you for advice. For each, you choose one thing: whether you’d write to them yourself from scratch, or let one of the advisors draft a first version for you to work from. You never have to write the advice itself.
You won’t see what any advisor would say before you choose, and you won’t see it afterwards either. Go on what you already know about how each one answers. If you want a reminder, each screen has a recap of what you said about each advisor and what each one decided.
Everyday tasks: consult and let act. Each task had its own screen, titled How would you use the advisors?, showing the task name and two questions, one above the other. Each question had four buttons: the three advisors and None of them. Both answers were required before Next. Table 7 gives the questions. The task levels come from the study design and were not shown.
| Task | Consult question | Let-act question | Stakes |
| Conflict with someone close | You’re in a disagreement with someone you care about. Which advisor, if any, would you consult on how to handle it? | Which advisor, if any, would you let write and send your reply on your behalf? | High / Low |
| Provocative public comment | Someone leaves a hostile comment on your public post. Which advisor, if any, would you consult on how to respond? | Which advisor, if any, would you let post a reply on your behalf? | High / Low |
| Apology / repair message | You’ve hurt a friend and need to make it right. Which advisor, if any, would you consult on what to say? | Which advisor, if any, would you let write and send the apology on your behalf? | High / Med |
| Charity allocation | You have $1,000 to give. Which advisor, if any, would you consult on how to distribute it across causes? | Which advisor, if any, would you let decide how to split your $1,000 across causes on your behalf? | Low / Med-High | |
| CV / candidate screening | You’re reviewing applications and must choose who advances. Which advisor, if any, would you consult on which candidates to move forward? | Which advisor, if any, would you let decide which candidates advance on your behalf? | Low / High |
| Content moderation | You moderate a community forum. Which advisor, if any, would you consult on which borderline posts should be removed? | Which advisor, if any, would you let remove posts on your behalf? | Low / High |
Table 7. Everyday-task questions in presentation order. Each question offered the three advisors and “None of them.” Stakes are study-design labels, not shown to participants.
Friends: let draft. Each friend had its own screen, titled Someone asked you for advice, showing the friend’s situation (Table 8) followed by:
| Situation shown | Value axis |
| Your friend Dana borrowed a housemate’s blender and cracked the lid. She’s heading into a brutal week of deadlines, and she’s asked you what to do — tell them today and sort a replacement now, or handle it once the week clears? | Benevolence vs. power |
| Your friend Marco’s younger cousin proudly showed him a school art project entered in a contest. Marco can see flaws a judge will notice, and the cousin asked what he honestly thinks. He’s asked you what he should say. | Benevolence vs. achievement |
| Your friend Priya’s close friend just got engaged and asked her to be in the wedding party. Priya has real doubts about the relationship. She’s asked you whether to share them or accept the role and keep them to herself. | Self-direction vs. security |
| Your friend Sam’s partner promised their kid a reward for good grades that Sam thinks is far too big. The kid delivered. Privately Sam would scale it back, and he’s asked you whether to reopen it with his partner or let it stand. | Self-direction vs. tradition |
| Your friend Ellie planned her first free weekend in a month to recharge. A friend has asked her to help them move on Saturday, and they are short on hands. She’s asked you whether to keep the weekend or show up. | Self-direction vs. conformity |
Table 8. Friend situations in presentation order. Value axes were not shown to participants.
You want to write back to [name]. How would you start?
[Write it myself from scratch] [Let AdvisorBot … draft it] ×3
If an advisor drafts it, you’d see the draft and could edit it before it goes.
The buttons unlocked after 3 seconds so that the situation was read first, and the same one-second pause followed every option. After a choice the screen showed Noted — you’d write to [name] yourself or Noted — [advisor] would draft it and you’d take it from there and moved on. No draft was ever shown.
Participants
121 people completed the study on prolific. Table 9 gives the self-reported demographics, asked on the consent screen with a “prefer not to say” option for each question. 75 (62%) were women and 43 (36%) men; 63 (52%) held a bachelor’s degree or higher; the most common age group was 35–44.
| (%) | |
| Age | |
| 18–24 | 14 (12%) |
| 25–34 | 26 (21%) |
| 35–44 | 28 (23%) |
| 45–54 | 23 (19%) |
| 55–64 | 20 (17%) |
| 65 or older | 8 (7%) |
| Prefer not to say | 2 (2%) |
| Gender | |
| Woman | 75 (62%) |
| Man | 43 (36%) |
| Non-binary | 1 (1%) |
| Prefer to self-describe | 1 (1%) |
| Prefer not to say | 1 (1%) |
| Highest education | |
| Less than high school | 2 (2%) |
| High school / secondary | 17 (14%) |
| Some college, no degree | 17 (14%) |
| Associate / vocational qualification | 21 (17%) |
| Bachelor’s degree | 41 (34%) |
| Master’s degree | 20 (17%) |
| Doctorate or professional degree | 2 (2%) |
| Prefer not to say | 1 (1%) |
Table 9. Participant demographics ().
Value profiles
Participants completed the 21-item Portrait Values Questionnaire (PVQ-21; 1 = not like me at all, 6 = very much like me). Following Schwartz’s scoring approach, we calculated each value score as the mean of its corresponding items, centered on the participant’s mean across all 21 items. These relative value scores determined which option was treated as value-favored for each dilemma and, consequently, which responses were assigned to the three advisor conditions. Table 10 reports the mean raw and centered scores.
| Value | Raw | Centred |
| Self-direction | 4.86 (0.79) | +0.42 (0.71) |
| Stimulation | 3.59 (1.37) | −0.85 (1.08) |
| Hedonism | 4.31 (1.16) | −0.12 (0.96) |
| Achievement | 3.68 (1.17) | −0.76 (0.95) |
| Power | 3.70 (1.14) | −0.73 (0.89) |
| Security | 4.89 (0.84) | +0.46 (0.71) |
| Conformity | 4.50 (1.17) | +0.06 (1.05) |
| Tradition | 4.57 (0.99) | +0.14 (0.83) |
| Benevolence | 5.12 (0.87) | +0.68 (0.66) |
| Universalism | 5.14 (0.77) | +0.70 (0.63) |
Table 10. PVQ-21 value scores (): mean (SD) of the raw 1–6 score and of the person-centred score.
Participants’ value priorities were unevenly distributed across the two higher-order dimensions. Most participants (107/121; 88%) prioritized self-transcendence (universalism and benevolence) over self-enhancement (achievement and power), whereas 42 (35%) prioritized openness to change (self-direction and stimulation) over conservation (security, conformity, and tradition). Consequently, in self-transcendence versus self-enhancement dilemmas, most participants encountered FullyAligned and DecAligned recommending the self-transcendent option, while ReaAligned recommended the self-enhancing option using self-transcendent justifications.
AI literacy
Participants answered eight AI-literacy items on a 1–7 scale (Table 11). Item 5 is the only negatively worded item and was reverse-scored. It did not behave like the other items: its correlation with the sum of the other seven was 0.19, and 15 participants (12%) agreed (6 or 7) that they are never alert to privacy and security issues, a pattern common for negated items. The 7-item score has mean 5.17 (SD 0.88), median 5.29, range 2.29–6.71. The exploratory analysis splits it into three bands: below 5 (38 people), 5 to 5.99 (58 people), 6 or above (25 people).
| Item | Mean (SD) |
| 1 I can identify the AI technology employed in the applications and products I use. | 4.94 (1.25) |
| 2 I can skillfully use AI applications or products to help me with my daily work. | 5.34 (1.39) |
| 3 I can choose the most appropriate AI application or product from a variety for a particular task. | 5.23 (1.32) |
| 4 I always comply with ethical principles when using AI applications or products. | 5.97 (1.26) |
| 5 I am never alert to privacy and information security issues when using AI applications or products. (reverse-scored) | 5.14 (1.82) |
| 6 I am always alert to the abuse of AI technology. | 5.19 (1.43) |
| 7 How frequently do you use generative AI products such as ChatGPT, Gemini, DALL·E, Claude, etc.? (1 = never, 7 = very frequently) | 4.69 (1.50) |
| 8 How familiar are you with limitations and shortcomings of generative AI? (1 = not at all, 7 = extremely familiar) | 4.86 (1.41) |
Table 11. AI-literacy items, with mean on the 1–7 scale ().
AI use frequency
Before the value questionnaire, participants reported how often they use AI chatbots for any purpose, how often they ask a chatbot for advice on personal decisions or dilemmas, and how often they use AI for three common tasks, all on the same six-point scale from “Never” to “Multiple times a day” (Table 12; $N = 121). General use was high: 43% used chatbots several times a day and 87% at least a few times a week. Asking for advice was less common and more varied: 12% never did, and 45% did so at least a few times a week.
| Never | < once a month | Few times a month | Few times a week | About daily | Several a day | |
| Any chatbot use | 1 | 3 | 12 | 44 | 9 | 52 |
| Advice on personal decisions | 15 | 23 | 28 | 32 | 13 | 10 |
| Use AI for . . . | ||||||
| Information, explanations | 1 | 3 | 22 | 40 | 19 | 36 |
| Writing, editing, summarising | 15 | 19 | 22 | 29 | 17 | 19 |
| Work or study tasks | 21 | 14 | 17 | 35 | 7 | 27 |
Table 12. AI use frequency (): number of participants at each level.
Coding of the preference explanations
After choosing their overall preferred advisor, the participants in the analysis sample explained their choice in free text. The answers were shuffled and given neutral identifiers, and the advisor each participant chose was kept in a separate key that no coder saw. A language model (Claude Opus 5.5) developed an inductive codebook from all 95 answers. The final codebook has 14 codes, of which the five most frequently assigned are summarized in Table 13. Three language-model coders (Claude Opus 5.5, Claude Sonnet 5.5, and GPT 5.5) independently applied it to every answer, and an answer could receive several codes. Agreement was high: Fleiss’ pooled over all code decisions was 0.96 (pairwise Cohen’s 0.95–0.97), and per code it was at least 0.93, except for general quality (0.67). A code was assigned when at least two of the three coders applied it. Every answer received at least one code (1.75 on average).
| Code | Definition | All | Both | Dec | Rea | |
| Aligns with my views | Matches the participant’s thinking, values, or choices. | .97 | 31 | 9 | 12 | 10 |
| Balanced / fair | Considers multiple sides or perspectives fairly. | 1.00 | 18 | 2 | 11 | 5 |
| Warm / empathetic | Shows empathy, warmth, or emotional support. | .98 | 17 | 6 | 6 | 5 |
| Logical reasoning | Offers logical, rational, or sound reasoning. | .95 | 15 | 5 | 4 | 6 |
| Thoughtful / nuanced | Provides detailed, thoughtful, or nuanced advice. | 1.00 | 13 | 3 | 4 | 6 |
Table 13. Five most frequent codes in participants’ advisor preference explanations (). Counts reflect majority agreement among three LLM coders; denotes Fleiss’ kappa. Multiple codes could be assigned to each response.
References
- [1]Hasan Amin, Ming Yin, and Rajiv Khanna. 2026. Align when they want, complement when they need! human-centered ensembles for adaptive human-AI collaboration. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence and Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence and Sixteenth Symposium on Educational Advances in Artificial Intelligence (AAAI’26/IAAI’26/EAAI’26). AAAI Press, Article 1930, 10 pages. doi:10.1609/aaai.v40i21.38786DOI
- [2]Anthropic. 2026. Introducing Claude Opus 4.8. https://www.anthropic.com/news/claude-opus-4-8. Model identifier: claude-opus-4-8. Accessed October 19, 2026.
- [3]Anthropic. 2026. Introducing Claude Opus 5.5. https://www.anthropic.com/news/claude-opus-5-5
- [4]Anthropic. 2026. Introducing Claude Sonnet 5.5. https://www.anthropic.com/news/claude-sonnet-5-5
- [5]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 (2022).arxiv.org/abs/2212.08073
- [6]Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 81, 16 pages. doi:10.1145/3411764.3445717
- [7]Anat Barli and Shalom H Schwartz. 2003. Values and behavior: Strength and structure of relations. Personality and social psychology bulletin 29, 10 (2003), 1207–1220.
- [8]Yochanan E Bigman and Kurt Gray. 2018. People are averse to machines making moral decisions. Cognition 181 (2018), 21–34.DOI
- [9]Robert Böhm, Moritz Jörling, Leonhard Reiter, and Christoph Fuchs. 2023. People devalue generative AI’s competence but not its advice in addressing societal and personal challenges. Communications Psychology 1, 1 (2023), 32.DOI
- [10]Silvia Bonaccio and Reeshad S Dalal. 2006. Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences. Organizational behavior and human decision processes 101, 2 (2006), 127–151.
- [11]Noah Castelo, Maarten W Bos, and Donald R Lehmann. 2019. Task-dependent algorithm aversion. Journal of marketing research 56, 5 (2019), 809–825.
- [12]Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, Lennart Heim, and Markus Anderljung. 2024. Visibility into AI Agents. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (Rio de Janeiro, Brazil) (FAccT ’24). Association for Computing Machinery, New York, NY, USA, 958–973. doi:10.1145/3630106.3658948DOI
- [13]Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasnennikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molamohammadi, John Burden, Wanru Zhao, Shalahleh Rismani, Konstantinos Voudouris, Umang Bhatt, Adrian Weller, David Krueger, and Tegan Maharaj. 2023. Harms from Increasingly Agentic Algorithmic Systems. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA) (FAccT ’23). Association for Computing Machinery, New York, NY, USA, 651–666. doi:10.1145/3593013.3594303DOI
- [14]Aaron Chatterji, Thomas Cunningham, David J Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. 2025. How people use chatgpt. Technical Report. National Bureau of Economic Research.DOI
- [15]Myra Cheng, Cinoo Lee, Pranav Khdpe, Sunny Yu, Dylan Han, and Jan Jurafsky. 2026. Cynopathic AI decreases prosocial intentions and promotes dependence. Science 391, 6792 (2026), eaec8352.DOI
- [16]Chun-Wei Chiang, Zhuoran Lu, Zhuoyan Li, and Ming Yin. 2024. Enhancing AI-Assisted Group Decision Making through LLM-Powered Devil’s Advocate. In Proceedings of the 29th International Conference on Intelligent User Interfaces (Greenville, SC, USA) (IUI ’24). Association for Computing Machinery, New York, NY, USA, 103–119. doi:10.1145/3640543.3645199
- [17]Yu Ying Chiu, Liwei Jiang, and Yejin Choi. 2025. DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=PHGiPGBf47
- [18]Yu Ying Chiu, Michael S. Lee, Rachel Calcott, Brandon Handoko, Paul de Font-Reaulx, Raphaël Millière, Paula Rodriguez, Chen Bo Calvin Zhang, Ziwen Han, Udari Madhushani Sehwag, Yash Maurya, Christina Q Knight, Harry R. Lloyd, Florence Bacus, Conor Downey, Mantas Mazeika, Bing Liu, Yejin Choi, Mitchell L Gordon, and Sydney Levine. 2026. MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes. In International Conference on Learning Representations, Vol. 2026. 5514–5561.arxiv.org/abs/2510.16380
- [19]Rudrajit Choudhuri, Christian Bird, Carmen Badea, Marco Gera, and Antani Sarma. 2026. You Shall Not Pass! Where and Why Developers Draw The Line on AI Autonomy. arXiv preprint arXiv:2607.00533 (2026).arxiv.org/abs/2607.00533
- [20]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adada503c91f9d2f40d0cdc4e9-Paper.pdfarxiv.org/abs/1706.03741
- [21]Kathleen M Eisenhart. 1989. Agency theory: An assessment and review. Academy of management review 14, 1 (1989), 57–74.
- [22]Cedric Faas, Richard Uth, Sarah Sterz, Markus Langer, and Anna Maria Feit. 2026. AI Support in Safety-Critical Tasks: How Perceived Autonomy and Meaningfulness Shape Human-AI Joint Performance. In Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency (Montreal, QC, Canada) (FAccT ’26). Association for Computing Machinery, New York, NY, USA, 1577–1606. doi:10.1145/3805689.3812265DOI
- [23]Kevin J Feng, David W McDonald, and Amy X Zhang. 2025. Levels of autonomy for ai agents. arXiv preprint arXiv:2506.12469 (2025).arxiv.org/abs/2506.12469
- [24]Andreas Fügener, Jörn Grahl, Alok Gupta, and Wolfgang Ketter. 2022. Cognitive challenges in human–artificial intelligence collaboration: Investigating the path toward productive delegation. Information systems research 33, 2 (2022), 678–696.DOI
- [25]Jason Gabriel. 2020. Artificial intelligence, values, and alignment: I. gabriel. Minds and machines 30, 3 (2020), 411–437.DOI
- [26]Alice Gao, Andrew N. Meltzoff, Maarten Sap, and Katharina Reinecke. 2026. Framing an AI with Values Reduces AI Reliance in AI-supported Writing Tasks. In Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency (Montreal, QC, Canada) (FAccT ’26). Association for Computing Machinery, New York, NY, USA, 5237–5264. doi:10.1145/3805689.3812369DOI
- [27]Francesca Gino, Jen Shang, and Rachel Croson. 2009. The impact of information from similar or different advisors on judgment. Organizational behavior and human decision processes 108, 2 (2009), 287–302.
- [28]Jan Gogoll and Matthias Uhl. 2018. Rage against the machine: Automation in the moral domain. Journal of Behavioral and Experimental Economics 74 (2018), 97–103.DOI
- [29]Kunal Hanha, Alex Tamkin, Miles McCain, Saffron Huang, Esin Durmus, Sarah Heck, Jared Mueller, Jerry Hong, Stuart Ritchie, Tim Belonax, et al. 2025. Which economic tasks are performed with ai? evidence from millions of claude conversations. arXiv preprint arXiv:2503.04761 (2025).
- [30]William Hart, Dolores Albarracín, Alice H Eagly, Inge Brechan, Matthew J Lindberg, and Lisa Merrill. 2009. Feeling validated versus being correct: a meta-analysis of selective exposure to information. Psychological bulletin 135, 4 (2009), 555.DOI
- [31]Jess Hohenstein, Rene F Kizilcec, Dominic DiFranzo, Zhila Aghajari, Hannah Mieczkowski, Karen Levy, Mor Naaman, Jeffrey Hancock, and Malte F Jung. 2023. Artificial intelligence in communication impacts language and social relationships. Scientific reports 13, 1 (2023), 5487.DOI
- [32]Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-Writing with Opinionated Language Models Affects Users’ Views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 111, 15 pages. doi:10.1145/3544548.3581196DOI
- [33]Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny T. Liang, Sydney Levine, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jack Hessel, Jon Borchardt, Taylor Sorensen, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. 2025. Investigating Machine Moral Judgement through the Delphi Experiment. Nature Machine Intelligence 7, 1 (2025), 145–160. doi:10.1038/s42256-024-00969-6DOI
- [34]Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan. 2024. “I’m Not Sure, But...”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (Rio de Janeiro, Brazil) (FAccT ’24). Association for Computing Machinery, New York, NY, USA, 822–835. doi:10.1145/3630106.3658941DOI
- [35]Sara Kingsley, Jiayin Zhi, Wesley Hanwen Deng, Jaimie Lee, Sizhe Zhang, Motahhare Eslami, Kenneth Holstein, Jason I Hong, Tianshi Li, and Hong Shen. 2024. Investigating what factors influence users’ rating of harmful algorithmic bias and discrimination. In Proceedings of the AAAI conference on human computation and crowdsourcing, Vol. 12. 75–85.DOI
- [36]Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. 2024. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence 6, 4 (2024), 383–392.DOI
- [37]Nils Köbis, Zoe Rahwan, Raluca Rilla, Bramantyo Ibrahim Supriyatno, Clara Bersch, Tamer Ajaj, Jean-François Bonnefon, and Iyad Rahwan. 2025. Delegation to artificial intelligence can increase dishonest behaviour. Nature 646, 8083 (2025), 126–134.DOI
- [38]Sebastian Krügel, Andreas Ostermayer, and Matthias Uhl. 2023. ChatGPT’s inconsistent moral advice influences users’ judgment. Scientific Reports 13, 1 (2023), 4569.DOI
- [39]Aakriti Kumar, Ren-Hui Michelle Tham, and Mark Steyvers. 2025. Seeing Things Differently: The Role of Differing Perspectives in Advice-Taking. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 47.DOI
- [40]Deepak Kumar, Yousef Anees AbuHashem, and Zakir Durumeric. 2024. Watch your language: Investigating content moderation with large language models. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 18. 865–878.arxiv.org/abs/2309.14517
- [41]Ethan Landes, Kathryn B Francis, and Jim AC Everett. 2026. People defer to AI moral advice, but not blindly. Cognition 272 (2026), 106504.DOI
- [42]Julie Anne Lee, Anat Bardi, Paul Gerrans, Joanne Sneddon, Hester Van Herk, Uwana Evers, and Shalom Schwartz. 2022. Are value—behavior relations stronger than previously thought? It depends on value importance. European journal of personality 36, 2 (2022), 133–148.
- [43]John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
- [44]Margarita Leib, Nils Köbis, Rainer Michael Rilke, Marloes Hagens, and Bernd Irlenbusch. 2024. Corrupted by algorithms? How AI-generated and human-written advice shape (dis) honesty. The Economic Journal 134, 658 (2024), 766–784.
- [45]Zhuoyan Li, Zhuoran Lu, and Ming Yin. 2023. Modeling human trust and reliance in ai-assisted decision making: A markovian approach. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 6056–6064.
- [46]Jiarui Liu, Yueqi Song, Yunze Xiao, Mingqian Zheng, Elinda Tjuatja, Jana Schaich Borg, Mona T. Diab, and Maarten Sap. 2025. Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamics. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Suzhou, China, 16428–16458. doi:10.18653/v1/2025.emnlp-main.831DOI
- [47]Tao Long and Lydia B. Chilton. 2026. SPEAR: Five Principles for Interactive Human-Agent Alignment. In Proceedings of the 2026 ACM Conference on Human-AI Complementarity and Alignment (Alexandria, VA, USA) (HCOMP ’26). Association for Computing Machinery, New York, NY, USA. https://arxiv.org/abs/2610.07204
- [48]Brian Lubars and Chenhao Tan. 2019. Ask not what AI can do, but what AI should do: Towards a framework of task delegability. Advances in neural information processing systems 32 (2019).arxiv.org/abs/1902.03245
- [49]Roger C Mayer, James H Davis, and F David Schoorman. 1995. An integrative model of organizational trust. Academy of management review 20, 3 (1995), 709–734.
- [50]Siddharth Mehrotra, Catholijn M Jonker, and Myrthe L Tielman. 2021. More similar values, more trust?-the effect of value similarity on trust in human-agent interaction. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 777–783.DOI
- [51]Tim Miller. 2023. Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven Decision Support using Evaluative AI. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA) (FAccT ’23). Association for Computing Machinery, New York, NY, USA, 333–342. doi:10.1145/3593013.3594001
- [52]Jimin Mun, Wei Bin Au Yeong, Wesley Hanwen Deng, Jana Schach Borg, and Maarten Sap. 2025. Why not just use AI? Analyzing people’s reasoning and conditions for AI acceptability. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 8. 1771–1784.arxiv.org/abs/2502.07287
- [53]OpenAI. 2026. Introducing GPT-5.5. April 23, 2026. https://openai.com/index/introducing-gpt-5-5/
- [54]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 (2022), 27730–27744.arxiv.org/abs/2203.02155
- [55]Eunkyung Park, Wesley Hanwen Deng, Vasudha Varadarajan, Mingxi Yan, Gunhee Kim, Maarten Sap, and Motahhare Eslami. 2026. Critical or Compliant? The Double-Edged Sword of Reasoning in Chain-of-Thought Explanations. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Budapest, Hungary, 46688–46708. doi:10.18653/2026.emnlp-main.2050
- [56]Shalom H Schwartz, Jan Cieciuch, Michele Vecchione, Eldad Davidov, Ronald Fischer, Constanze Beierlein, Alice Ramos, Markku Verkasalo, Jan-Erik Lönnqvist, Kursad Demirutku, et al. 2012. Refining the theory of basic individual values. Journal of Personality and Social Psychology 103, 4 (2012), 663–688.
- [57]Julia Šuljerdova and Scott A Hale. 2026. What Do People Actually Want From AI? Mapping Preference Plurality. In Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency (Montreal, QC, Canada) (FAccT ’26). Association for Computing Machinery, New York, NY, USA, 6305–6350. doi:10.1145/3805689.3812398
- [58]Mrinank Sharma, Meg Tong, Tomek Korbak, David Duvenaud, Amanda Askell, Sam Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott Johnston, Shauna Kravec, et al. 2024. Towards understanding sycophancy in language models. In International Conference on Learning Representations, Vol. 2024. 110–144.arxiv.org/abs/2310.13548
- [59]Hua Shen, Tiffany Knewren, Reshmi Ghosh, Kenan Alkiek, Kundan Krishna, Yachuan Liu, Savvas Petridis, Yi-Hao Peng, Li Qiwei, Chenglei Si, Yutong Xie, Jeffrey P. Bigham, Frank Bentley, Joyce Chai, Zachary Chase Lipton, Qiaozhu Mei, Michael Terry, Diyi Yang, Meredith Ringel Morris, Paul Resnick, and David Jurgens. 2025. Position: Towards Bidirectional Human-AI Alignment. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems Position Paper Track. https://openreview.net/forum?id=PgA9rZoMY8arxiv.org/abs/2406.09264
- [60]Michael Siegrist, George Cvetkovich, and Claudia Roth. 2000. Salient Value Similarity, Social Trust, and Risk/Benefit Perception. Risk Analysis 20, 3 (2000), 353–362. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/0272-4332.03034 doi:10.1111/0272-4332.03034pubmed.ncbi.nlm.nih.gov/10949414
- [61]Aaditya Singh, Adam Fry, Adam Perelman, Adam Ganes h, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267 (2025).arxiv.org/abs/2601.03267
- [62]Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024. Position: A Roadmap to Pluralistic Alignment. In Forty-first International Conference on Machine Learning. https://openreview.net/forum?id=gQPnRHwxMarxiv.org/abs/2402.05070
- [63]Mary Steffel, Eleanor F. Williams, and Jaclyn Perrmann-Graham. 2016. Passing the buck: Delegating choices to others to avoid responsibility and blame. Organizational Behavior and Human Decision Processes 135 (2016), 32–44. doi:10.1016/j.obhdp.2016.04.006DOI
- [64]Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, Michael Stern, Brian Clarke, Landon Goldberg, Theodore R. Sumers, Jared Mueller, William McEachen, Wes Mitchell, Shan Carter, Jack Clark, Jared Kaplan, and Deep Ganguli. 2024. Clio: Privacy-Preserving Insights into Real-World AI Use. arXiv:2412.13678 [cs.CY] https://arxiv.org/abs/2412.13678
- [65]Michael Terry, Chinmay Kulkarni, Martin Wattenberg, Lucas Dixon, and Meredith Ringel Morris. 2024. Interactive AI Alignment: Specification, Process, and Evaluation Alignment. arXiv:2311.00710 [cs.HC] https://arxiv.org/abs/2311.00710
- [66]Suzanne Tolmeijer, Markus Christen, Serhiy Kandul, Markus Kneer, and Abraham Bernstein. 2022. Capable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 160, 17 pages. doi:10.1145/3491102.3517732DOI
- [67]Matt Twyman, Nigel Harvey, and Clare Harries. 2008. Trust in motives, trust in competence: Separate factors determining the effectiveness of risk communication. Judgment and Decision Making 3, 1 (2008), 111–120. doi:10.1017/S1930297500000218DOI
- [68]Oleksandra Vereschak, Gilles Bailly, and Baptiste Caramiaux. 2021. How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 327 (Oct. 2021), 39 pages. doi:10.1145/3476068DOI
- [69]Kyra Wilson and Aylin Caliskan. 2025. Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval. In Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society (San Jose, California, USA) (AIES ’24). AAAI Press, 1578–1590.DOI
- [70]Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT ’20). Association for Computing Machinery, New York, NY, USA, 295–305. doi:10.1145/3351095.3372852DOI
- [71]Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, Nouha Dziri, Dan Jurafsky, and Maarten Sap. 2025. REL-A.I.: An Interaction-Centered Approach to Measuring Human-LM Reliance. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Association for Computational Linguistics, Albuquerque, New Mexico, 11148–11167. doi:10.18653/v1/2025.naacl-long.556DOI
- [72]Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, and Maarten Sap. 2024. Relying on the Unreliable: The Impact of Language Models’ Reluctance to Express Uncertainty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand, 3623–3643. doi:10.18653/v1/2024.acl-long.198DOI




