betaFeedback

WorksComputer Science

Probe What-If Worlds Before You Ship: Persona Simulation for Real-World Decision Making

Abstract

User research and system testing reveal whether products and services meet user needs, yet recruiting specialized participants, designing controlled what-if comparisons, and repeating studies as designs evolve often bottleneck design iteration and data-driven decision-making. We present a framework and execution infrastructure that translates real-world industry inquiries into controlled experiments with simulated users. Our methodology formally defines the target cohort, experimental stimuli, and decision-relevant metrics, while the pipeline supports large-scale concurrent runs over cohort-stimulus-environment combinations (survey, web, chatbot, and OS app), auditable trial-level logging, cohort-level outcome aggregation, and cross-cohort and cross-variant statistical reports. Across twelve industry partner cases spanning five domains, we show how constructing hard-to-reach or what-if persona cohorts supports exploratory studies and distills simulated responses into testable design hypotheses. Cross-model comparisons on selected cases largely preserve primary conclusions, whereas perturbing or ablating persona attributes alters service outcomes, effect sizes, and inter-group differences, indicating that grounding target cohorts in detailed personas materially shapes simulation results. Together, these studies show that persona-enabled agents can test specified cohorts at design speed, diagnose systems under test, and deliver directional evidence that focuses problem definition, steers design and strategic decisions, and guides further study.

Introduction

User testing is integral to product development: it reveals how target users interpret an experience, what hinders adoption, and which design iterations satisfy their needs ([11, 10]). Yet live studies face three structural constraints that limit how frequently teams can gather this empirical evidence ([8]). First, recruiting participants and running repeated testing cycles becomes prohibitively slow when studies require specialized cohorts defined by specific health circumstances, professional roles, or service requirements. Second, because attributes in live populations are confounded, teams struggle to isolate the effect of a single user attribute by constructing cohorts that match on all other characteristics. Third, exposing existing customers to unproven designs or policies can degrade their experience and damage client relationships during the experiment ([17]).

Two partner studies illustrate how these constraints arise in practice. In one study, a healthcare provider sought to identify care-access barriers across patients with varying treatment regimens and household resources, where recruiting specialized patient cohorts and disentangling correlated socioeconomic factors are both costly. In another, a business-service partner needed to evaluate how clients would react to revised engagement terms across alternative wordings, without risking live client relationships on untested contract language. Together, hard-to-reach cohorts, confounding across user attributes, and the risk to live traffic motivate conducting controlled what-if comparisons offline before deployment.

Persona-based user simulation removes both obstacles by replacing live cohorts with explicitly specified user profiles ([24, 4]): teams can hold background attributes fixed while varying target preferences or behaviors, and repeatedly evaluate evolving design candidates offline without exposing existing customers to untested interfaces. This capability has made simulated users a standard tool for early-stage design exploration [24, 25], recommendation testing [35, 4], and assistant evaluation [8].

A Persona-Centered Infrastructure for Scalable, Standardized Controlled Evaluation

Figure 1. The execution pipeline supports scalable, controlled simulated-user evaluation across five application domains.

Building on an existing persona corpus and execution infrastructure [20], we examine twelve industry partner case studies spanning five application domains: health and well-being, marketing and consumer behavior, technology and product design, social and organizational research, and scientific research. For each partner inquiry, we specify who the target users are, what stimuli they evaluate, and which outcomes govern the decision. These specifications are translated into at-scale controlled experiments with simulated users across four types of environments by our infrastructure.

Contributions. We contribute a common methodology for what-if comparisons of design alternatives and specified persona attributes, covering experimental design, cohort construction, stimulus preparation, and outcome measurement, together with an open-source infrastructure for concurrent, auditable, and repeatable execution; twelve industry partner case studies across four digital environment and five domains that identify participation barriers, audience differences, and design hypotheses; and controlled comparisons examining the sensitivity of selected findings to simulation models and persona information. The case studies establish concrete targets or directions for subsequent human testing with rationale rather than validated predictions of real-user behavior.

Methodology

Our methodology translates partner inquiries into controlled simulated-user studies. A persona is a set of attributes conditioning an LLM’s behavior through agent harnesses; a cohort is a collection of personas selected for the audience of interest. A stimulus is the material or experience evaluated, and an experiment arm specifies one cohort–stimulus condition.

Experimental Design

Define the experiment type. We begin with a concrete research question that specifies the target population, the stimulus to be evaluated, the comparison of interest, and the decision the experiment is intended to inform. We organize experiments along two axes for exploring what-if scenarios within the simulation: whether the cohort varies and whether the stimulus varies. A single cohort evaluating a single stimulus constitutes a descriptive evaluation. When the cohort is held fixed and the stimulus varies, the experiment evaluates changes to the product, message, or experience; we refer to this as a System Under Test (SUT) experiment. When the stimulus is held fixed and the cohort varies, the experiment compares responses across population segments; we refer to this as a Persona Under Test (PUT) experiment. When both vary, a crossed design can be used to estimate stimulus effects across cohorts and examine their interactions. Figure 2 summarizes this taxonomy.

Design taxonomy defined by cohorts and stimuli

Figure 2. Design taxonomy defined by cohorts and stimuli. SUT compares alternatives within one cohort; PUT compares cohorts under one stimulus; crossed designs vary both.

Specify cohort attributes. We build on prior work that represents personas using 1,290 attribute dimensions whose dependencies are encoded in a directed acyclic graph (DAG) [20]. We first translate the target-population description into constraints on these dimensions. When an exact match exists, we map the description directly to the corresponding attributes; for example, “people aged 18–34” maps to the 18–24 and 25–34 age brackets. When the target construct is not represented directly, we approximate it using one or more related attributes. For example, high music engagement can be approximated by listening to music “daily” or “all day,” although listening frequency is not equivalent to engagement. We make such limitations explicit and check the resulting constraints against the dependencies encoded by the DAG. Logically inconsistent combinations—for example, simultaneously specifying that a persona lives alone and belongs to a multi-person household—are excluded.

Sample the cohort. After defining the relevant attributes, we estimate the cohort size and select sampling strategy according to the experiment’s purpose. We consider five cohort-composition strategies. First, reference-data-informed sampling sets target proportions using relevant human data; for example, a healthcare study may use population and clinical-study data to specify age, body mass index (BMI), sex, and race or ethnicity distributions. Second, balanced comparison assigns equal or similar representation to the groups being compared; for example, a study of an AI research assistant may include equal numbers of senior researchers, wet-lab trainees, computational biologists, and molecular geneticists. Third, targeted oversampling increases the representation of an important but uncommon group; for example, a study of a service for remotely inspecting vehicles, properties, suppliers, or documents may oversample personas with relevant transaction experience. Fourth, broad coverage distributes personas across diverse backgrounds and circumstances without attempting to reproduce population proportions; for example, a professional-networking study may include varied demographic backgrounds, professional roles, and levels of digital experience. Finally, stress-test sampling increases the representation of personas facing task-relevant challenges; for example, a community-health study may emphasize affordability constraints, limited access to care, and difficulty understanding health information. These strategies may be adapted or combined when required by the research question. When an experiment requires multiple cohorts, each cohort is constructed using the same procedure. Balanced, exploratory, and stress-test cohorts do not estimate population proportions.

Prepare the stimulus and task. The stimulus may be an existing product, stakeholder-provided material, or an artifact constructed for the experiment. It is preserved in the form that a real user would encounter whenever possible. For controlled comparisons, only the intended experimental factor is varied. For example, a message-framing experiment may change the framing while keeping the factual content, questionnaire, question order, identifiers, and response scales constant.

Each experiment also includes a task specification describing the situation, the action expected from the simulated user, and, for interactive tasks, the available actions and stopping conditions. Instructions are written neutrally and include only information that would reasonably be available to the target user.

Define outcome metrics. Studies specify decision-relevant outcomes, diagnostic measures, and guardrails. Numerical ratings, categorical choices, and explanations retain their scales, category definitions, and valid-response denominators.

Execution Infrastructure for Large-Scale Experiments

A trial comprises one simulated persona completing a task under one arm. Each persona receives only the stimulus assigned to that arm, preserving the intended comparison. The pipeline fixes cohort membership, sampling seeds, stimulus versions, instructions, and model configurations, and executes independent trials concurrently across workers. Specifications can be reused after changing a stimulus, design, or simulation model, allowing follow-up studies to retain the original cohort and outcome definitions. Controlled model comparisons aim to retain cohort, stimulus, task, and metrics; actual endpoint and prompt-assembly differences are reported because they limit model-only or persona-only interpretations. Throughput depends on interaction format, model latency, service quotas, and computing resources.

Each trial retains the persona, rendered inputs, response or interaction trajectory, and structured outcome. Aggregation preserves links to these records and reports failed or missing trials, allowing cohort-, arm-, and subgroup-level findings to be audited.

Statistical reporting. Reports preserve the valid-response denominator for each outcome and summarize responses by experiment arm and subgroup. For numerical comparisons, the pipeline provides effect estimates and uncertainty intervals while retaining links to the underlying trials. The analysis unit and any shared-persona or group structure are specified alongside each comparison. The statistical procedures depend on the comparison design; Appendix § 10 details the statistical procedures and their interpretation.

Twelve Cross-Domain Industry Case Studies

Table 1 summarizes twelve studies across five domains; protocols and results appear in the Appendix. The cases illustrate two uses: constructing repeatable studies with hard-to-reach or matched what-if cohorts and deriving testable design hypotheses from simulated responses.

Application domainsCasesTest environmentPrincipal finding
Health & well-beingC1: Music-wellbeing onboardingSurveyAsking how users felt built more trust than emphasizing science or adopting a peer-companion tone.
C10: Healthcare access barriersSurveyCost barriers were widespread, but rural users faced greater travel barriers and adults aged 31–50 reported greater time pressure.
Marketing & consumer behaviorC3: Subscription pricing and renewalSurveyA cheaper introductory offer attracted interest, but recurring payments discouraged subscription.
C5: Remote-service marketplaceSurveyUsers were more concerned about trusting the person performing the service than about its price.
C9: Engagement-letter evaluationSurveyRevising payment and liability terms improved acceptance, but distrust remained a barrier to signing.
Technology & product designC6: Professional-network signupWebUsers understood how to sign up but hesitated over identity verification and document privacy.
C8: Industrial website engagementWebVisitors were more willing to explore than contact sales; “try for free” links led to sales forms that did not meet their expectations.
C11: Home assistantChatbot/OSComfort-first preferences were satisfied more often than shorter-runtime preferences across three simulation models.
C12: Car assistantChatbot/OSProactive preference disclosure improved service when canceling a route stop; a breakfast-stop task showed model-dependent differences.
Social & organizational researchC7: Workplace survey responsesSurveyWorkers gave more favorable answers in supportive workplaces; changing models altered responses more than repeating the same model.
Scientific researchC2: Clinical-trial screening reviewSurveyPharmacists prioritized medication compatibility, while molecular specialists prioritized genetic confirmation in the same screening review.
C4: Scientific research prioritizationSurveyScientists chose different research priorities even when their comprehension scores showed no difference.

Table 1. Twelve industry partner case studies spanning five application domains and their principal simulated-user findings. Case identifiers link to the corresponding appendix reports.

Constructing Hard-to-Reach and What-If Cohorts

The methodology supports both constructing cohorts for specialized audiences and creating matched persona cohorts that differ only in specified attributes. To reach specialized populations, the healthcare-access study (C10) evaluates 900 personas across nine region–age strata in India with specified clinical conditions, treatment arrangements, and household resources, mapping care-access barriers across intersecting socioeconomic circumstances. Similarly, the clinical-trial screening study (C2) assigns 200 personas from each of four specialist roles—including oncology pharmacists and molecular pathologists—to review the same patient record, isolating role-specific evaluation criteria without repeatedly recruiting scarce clinicians.

The car-assistant study (C12) illustrates controlled what-if cohort construction: each of 50 base personas is paired with a counterfactual version that holds background attributes and travel priorities fixed while varying whether the user discloses preferences proactively or only when prompted. In the stop-cancellation task, proactive disclosure increases preference satisfaction by 18–44 percentage points across all three simulation models, pinpointing active preference elicitation as a high-priority mechanism for live validation. Across all three studies, simulated cohorts serve to generate targeted hypotheses for human testing rather than to replace empirical evidence from live populations.

From Simulated Responses to Design Implications

Linking aggregate outcomes to individual choices, explanations, and interaction logs helps distinguish task difficulty from low perceived value or unwillingness to participate.

Professional-network signup (C6) asked whether prospective users trusted a networking website enough to join. About 98% reached the signup entrance, yet only about 1% stated that they would register. The website was generally rated as clear, while explanations identified concerns about identity verification and document retention. Improving navigation alone would therefore leave the reported concerns unresolved; the study directed attention to privacy disclosures and verification procedures.

The subscription pricing and renewal study (C3) exposed a similar gap between perceived value and willingness to join. About 98% considered a cheaper introductory offer a good deal, but only about 51% would subscribe. Non-subscribers identified the recurring commitment as the main barrier. The reported hesitation concerned the ongoing payment obligation, directing the next test toward renewal terms.

Persona attributes also helped interpret inter-group differences in priorities. Scientific research prioritization (C4) asked which unresolved immunology questions four professional groups would investigate next. Their choices differed despite no observed difference on the comprehension measure, with molecular geneticists showing the largest departure from senior investigators. A single pooled preference would conceal this disciplinary variation; separate group analyses identified perspectives to include in expert consultation.

These explanations identify hypotheses for product revision and subsequent human studies; they do not establish that the proposed changes improve real-user outcomes.

Robustness and Ablation Analysis

We examine whether selected findings recur across user-simulation models and how they change when persona attributes are varied or descriptions omitted.

Cross-Model Robustness

Figures 3 and 4 compare GPT-5.6-Sol, Claude Opus 5, and Gemini 3.7 Flash, distinguishing recurring directions from agreement in effect magnitude.

All three models yield higher preference satisfaction for comfort-first than shorter-runtime personas in both home tasks. For canceling a car route stop, proactive disclosure outperforms on-request disclosure under both travel priorities (Figure 3). Home contrasts average over disclosure conditions within matched personas, whereas car contrasts hold travel priority fixed while varying disclosure. These selected contrasts recur across the evaluated configurations; the breakfast-stop task instead shows model-dependent effects (Appendix § 9.12).

Four-panel bar chart showing persona changes affecting assistant preference satisfaction

Figure 3. Persona changes affect assistant preference satisfaction. C11: preparing the home or maintaining bedroom comfort, comparing comfort-first versus runtime-first personas (the latter prefer shorter device runtime). C12: canceling a route stop, comparing proactive versus on-request disclosure of travel preferences. Bars show first-minus-second satisfaction rates (percentage points); colors identify user-simulation models. Error bars show paired-bootstrap 95% confidence intervals.

Professional group remains strongly associated with clinical-screening review priority, and all three persona models favor combined payment-and-liability revisions over liability-only revisions (Figure 4). This supports shared hypotheses for follow-up testing, not interchangeable quantitative estimates. Home/car assistant endpoints and engagement prompt assembly differ across configurations, limiting model-only interpretations.

Two-panel bar chart showing persona information shaping simulated judgments

Figure 4. Persona information shapes simulated judgments. C2: association between professional group and clinical-screening review priority (Cramér’s VV; higher means stronger association). C9: client acceptance of combined payment-and-liability revisions minus liability-only revisions to an engagement letter (percentage points). Green bars use personas; gray denotes GPT without personas. Error bars show bootstrap 95% confidence intervals.

Effects of Persona Conditioning

Perturbing persona attributes. As shown in Figure 3, controlled persona-attribute perturbations provide what-if comparisons of support across simulated user segments: the home assistant under-serves runtime-minimizing users (C11, Table 12), while the car assistant under-serves users who disclose preferences only when asked (C12, Table 13). These gaps motivate tests of runtime-aware planning and preference clarification. Trust also strongly differentiated engagement-letter acceptance (Appendix § 9.9).

ModelDifference95% CI
Prepare home before arrival
GPT-5.6-Sol+37.0[25.0, 50.0]
Claude Opus 5+37.8[27.6, 48.0]
Gemini 3.7 Flash+45.0[34.0, 56.0]
Maintain bedroom comfort
GPT-5.6-Sol+81.0[73.0, 88.0]
Claude Opus 5+53.0[42.0, 64.0]
Gemini 3.7 Flash+72.0[63.0, 80.0]

Table 12. Home-assistant preference-satisfaction contrasts.

ModelDifference95% CI
Cancel stop: fastest-arrival priority
GPT-5.6-Sol+18.0[6.0, 30.0]
Claude Opus 5+44.0[30.0, 58.0]
Gemini 3.7 Flash+36.0[20.0, 52.0]
Cancel stop: toll-avoidance priority
GPT-5.6-Sol+30.0[16.0, 44.0]
Claude Opus 5+44.0[28.0, 60.0]
Gemini 3.7 Flash+42.0[28.0, 56.0]

Table 13. Car-assistant preference-satisfaction contrasts.

Ablating persona information. As a separate sensitivity check, omitting persona specifications substantially weakens inter-group differences and can even reverse simulated design rankings (Figure 4). In clinical-trial screening, the association between professional group and review priority collapses from Cramér’s V=.928V = .928 to .063.063 without persona descriptions (Appendix § 9.§ 2). In engagement-letter evaluation, the combined-minus-liability-only acceptance difference changes from +17.25 to −15.50 percentage points, reversing the simulated ranking of the revisions (Appendix § 9.9). Different prompt assembly prevents attributing this reversal solely to persona removal, but the comparison shows that audience specification can materially affect which design the simulation favors.

Discussion and Conclusion

Across twelve industry partner studies, simulated feedback turned broad evaluation questions into specific design choices. It distinguished interest from willingness to participate, revealed professional priorities that pooled results would conceal, and exposed gaps in assistants’ support for different user preferences and disclosure styles. Model comparisons supported shared directions for pilot testing, while persona ablations showed that audience information can change which revision is selected.

The contribution is a common methodology and infrastructure for making these comparisons repeatable and statistically traceable. By connecting specified audiences to controlled what-if comparisons of designs and persona attributes, the framework helps prioritize candidates for live testing and gives subsequent user studies a sharper focus, without establishing causal effects in real users.

Limitations

Incomplete human reference data and imperfect behavioral representation limit the precision of real-world estimates from these studies. The reported intervals quantify variation within the specified simulation designs (Appendix § 10); they do not measure discrepancies between simulated and real-user behavior.

Ethical Considerations

All reported user responses are synthetic. Partner materials and case-level records remain confidential, while the public repository provides the study infrastructure. Disclosure of case descriptions and aggregate findings follows case-specific partner authorization.

Authors and Affiliations

Julie Zhu1,∗^{1,*}, Jianheng Hou8,∗^{8,*}, Zhixu Silvia Tao2,∗^{2,*}, Shi Bo7,†^{7,\dagger}, Yilan Fan4,†^{4,\dagger}, Yixuan He3,†^{3,\dagger}, Zhiyuan Hu1,†^{1,\dagger}, Shirley Huang6,†^{6,\dagger}, Fangyu Liu8,†^{8,\dagger}, Yijun Wang6,†^{6,\dagger}, Zibu Wei5,†^{5,\dagger}, Haimingyue Xu16,†^{16,\dagger}, Zhen Xu10,†^{10,\dagger}, Zichen Yuan9,†^{9,\dagger}, Jierui Zuo17,†^{17,\dagger}, Jintao Huang11^{11}, Yifan Wang12^{12}, Qianfeng Wen13^{13}, Jiahan Li4^{4}, Zhejian Peng14^{14}, Bowen Yang10^{10}, Shushu Liang6^{6}, Yuexing Hao15,1^{15,1}, Xiaomin Li15,6^{15,6}.

Affiliations

  1. Massachusetts Institute of Technology

  2. Princeton University

  3. Arizona State University

  4. New York University

  5. University of California, Los Angeles

  6. Harvard University

  7. Boston University

  8. University of Southern California

  9. Washington University in St. Louis

  1. Columbia University

  2. The Ohio State University

  3. Purdue University

  4. University of Toronto

  5. University of Illinois Urbana-Champaign

  6. MatrAIx

  7. VEGA

  8. University of Washington

*Equal contribution.

†These authors contributed equally as case-study owners.

Acknowledgement

We thank Kevin Zhou for contributing model credits that supported the experiments in this study.

Extended Related Work

Simulated users for evaluation. Social Simulacra introduced generated interactions for early design exploration [24], while Generative Agents demonstrated interactive behavior in shared environments [25]. Subsequent studies developed simulated users for conversational recommendation [35, 22], recommender evaluation [4], and assistant benchmarking [8]. PersonaEval connects persona-based users to surveys, chatbots, and web applications while recording interaction trajectories and outcomes [21]. AgentPersonaBench benchmarks adherence to specified persona attributes during task execution [15]. Building on these evaluation capabilities, we develop a common study methodology that connects industry partners’ evaluation needs to audience specification, controlled comparisons, and statistical reporting. Twelve partner studies across five application fields show how this methodology informs concrete design decisions.

Audience specification and behavioral grounding. Turing experiments and silicon-sample studies demonstrate that LLMs can reproduce selected behavioral patterns [1, 2]. Population and survey audits further show that representation depends on the audience, task, and measurement [30, 3, 33, 13, 26]. Human-grounded agents and matched-response datasets provide references for evaluating this dependence [31] (Park et al. 2026). Human comparisons and behavioral evaluations identify task-dependent differences in simulated behavior [36, 37, 6, 9, 29, 7]. Our studies examine how audience specification shapes the resulting insights, using persona ablations and model comparisons to distinguish findings that persist from those that depend on the simulation configuration.

Simulation infrastructure and application studies. Concordia, AgentSociety, and OASIS provide reusable environments for social simulation [32, 28, 34]. MicroVerse provides an instrument for measuring identity drift in long-horizon multi-agent simulations [23]. Methodological syntheses emphasize explicit design and validation choices (Bück-Kaeffer et al., 2026; Larooi and Törnberg, 2026). Building on prior infrastructure [20], we connect cohort construction, isolated trial execution, and outcome reporting through a common study specification, allowing partner evaluations to be repeated and inspected across application settings.

Statistical interpretation. Uncertainty quantification and human-data-assisted inference provide methods for analyzing synthetic responses [14, 18]. Recent work formalizes the assumptions required to connect simulated responses and ratings to human outcomes [16, 27, 12]. Our reporting protocol links each estimate and uncertainty interval to its comparison design and analysis unit, with uncertainty interpreted within the simulation design. This supports consistent interpretation across partner studies.

Case Reports

C1: Music-Wellbeing Onboarding

Summary. A music-wellbeing provider asked which opening message would encourage young users to begin a personalized journey while preserving trust, comfort, and privacy confidence.

Cohort. The persona-conditioned studies targeted a 1,000-person young-user cohort. GPT-5.6-Sol and Opus 5 reports identify the same cohort and seed 42. A separate GPT-5.6-Sol baseline omitted individual persona descriptions.

Task and design. Each study compared science-informed, warm peer-companion, and check-in-first messages in isolated trials, with 1,000 simulation trials per framing in each study. Primary outcomes were trust, intent to begin, and anticipated journey choice; comfort and privacy confidence served as guardrails. Table 2 reports score differences calculated from the local trial records on a 1–7 scale.

Model / conditionTrustIntentPrivacy
Check-in first
Opus 5+0.230+0.151+0.861
GPT-5.6-Sol+0.252+0.272+0.488
GPT, no persona+0.035+0.351+0.722
Warm peer companion
Opus 5−0.208−0.307−0.161
GPT-5.6-Sol−0.093−0.077−0.289
GPT, no persona−0.640−0.599−0.209

Table 2. Music-wellbeing score differences from science-informed framing.

Results. GPT and Opus both favored check-in-first over science-informed framing, with trust gains of 0.252 and 0.230 points and intent gains of 0.272 and 0.151 points. Warm peer-companion framing reduced both outcomes. Without personas, the check-in-first trust gain was 0.035 points, while intent increased by 0.351 points. The warm-peer trust penalty increased from 0.093 to 0.640 points without personas.

Note. Entries are local-data mean contrasts on a 1–7 scale. Valid science/warm/check-in counts are 957/960/954 (Opus), 999/1,000/1,000 (GPT), and 1,000/1,000/1,000 (no persona). Paired contrasts use 916 check-in and 920 warm-peer complete pairs for Opus, and 999 for GPT; no-persona arms are independent. The trust contrasts and their intervals are reported here and in Appendix § 10.

Insights. Agreement between the two persona-conditioned models motivates pilot testing of optional check-ins rather than assuming that warmer language builds trust. The no-persona baseline shows a different response pattern: check-in-first trust benefits weaken without profiles, while intent benefits persist. The directly estimated difference in check-in-first trust effects was +0.217 points (95% CI [0.183, 0.251]). The results concern simulated messaging responses rather than observed uptake or therapeutic efficacy.

C2: Clinical-Trial Screening Review

Summary. This study asked how professional reviewer groups evaluate the same clinical-trial screening assistant and prioritize evidence when its eligibility assessment overlooks unresolved requirements.

Cohort. The updated four-group study uses 800 personas, with 200 in each study-assigned group: oncology pharmacists, molecular genetics or pathology specialists, clinical research coordinators, and clinical investigators. The retired trainee group is excluded from this revision. Study-specific attributes specify professional responsibilities and review dimensions.

Task and design. All groups reviewed one fictional postoperative ADAURA screening record against supplied historical rules, forming a PUT comparison. An assistant marked eligibility satisfied despite pending central EGFR confirmation and current rifampin use conflicting with an exclusion. GPT-5.6-Sol, Claude Opus 5, and Gemini 3.7 Flash each evaluated the 800-persona cohort. A GPT no-persona condition retained the task and bookkeeping IDs but omitted persona information. Each configuration yielded 800 effective responses. The primary outcome was review priority; eligibility judgment and blocking-issue identification were secondary checks.

Results. All four professional groups shared their modal review priority across the three persona models (Table 3). Pharmacists prioritized medication compatibility, molecular/pathology specialists prioritized molecular evidence, and coordinators and investigators prioritized complete eligibility checking.

Reviewer groupGPTOpusGemini
Oncology pharmacyB: 99.5B: 97.5B: 100
Molecular / pathologyA: 100A: 66.0A: 100
Research coordinationC: 97.0C: 100C: 100
Clinical investigationC: 80.5C: 100C: 99.0

Table 3. Modal screening-review priorities across three persona models.

Note. Modal option and percentage; n=200n = 200 per group and model. A: molecular evidence; B: medication compatibility; C: complete, documented eligibility checking. Results use the updated four-group aggregation (October 11, 2026), including one GPT and nine no-persona supplemental runs. Figure 4 reports group–priority association with pointwise intervals.

Persona ablation. Group–priority association weakened from Cramér’s V=.928V = .928 with GPT personas to .063 without them. The no-persona control favored complete eligibility checking in all four bookkeeping groups (58.0–63.0%), whereas persona-conditioned modes differed by professional role. These descriptive estimates replace the earlier five-group analysis; its contrast interval is not reused for the updated data.

Insights. Professional context differentiated review priorities, and all three persona models preserved the role-specific modal pattern despite differences in concentration. These results motivate role-specific evidence presentation; the study-assigned review dimensions and absence of matched human responses limit claims about real professional preferences.

C3: Subscription Pricing and Renewal

Summary. A subscription-box provider sought to understand how introductory pricing affected willingness to join and whether initial interest extended to recurring payments.

Cohort. The same 500-persona cohort represented English-speaking women aged 18–54 in North America, spanning income levels, life stages, and economic motivations.

Task and design. Using GPT-5.6-Luna, personas evaluated five introductory prices: $9.99, $29.99, $39.99, $49.99, and $59.99. Product selection, membership benefits, and the $79.99 quarterly renewal price remained fixed; advertised discounts changed with price. Each condition produced 500 completed evaluations, totaling 2,500 trials.

Results. Table 4 shows similar subscription rates at $29.99 and $39.99, with substantially lower acceptance at the two higher prices. At $39.99, 489/500 rated the offer a good deal, but only 257 would subscribe. Across all conditions, no persona affirmed that quarterly renewal was acceptable; responses were unsure or negative. Every non-subscriber identified recurring commitment as the main barrier.

Introductory priceYes / valid NNRate
$9.99 | 399/500 | 79.8%
$29.99 | 251/500 | 50.2%
$39.99 | 257/500 | 51.4%
$49.99 | 84/500 | 16.8%
$59.99 | 78/500 | 15.6%

Table 4. Stated subscription acceptance across introductory prices.

Note. The same cohort evaluates every price; the quarterly renewal price remains $79.99. Entries are descriptive simulated choices, without inferential tests or intervals.

Insights. Introductory-price approval did not imply subscription acceptance or confidence in renewal. The findings motivate separate tests of renewal terms and retention. They describe simulated stated preferences rather than observed purchases or an established optimal price.

C4: Scientific Research Prioritization

Summary. This study examined whether simulated scientists with different professional backgrounds prioritized different unresolved questions in a TCF7/CD8 research-planning task.

Cohort. Four synthesized cohorts each comprised 250 personas: senior tumor-immunology principal investigators (PIs) or research scientists, wet-lab doctoral or postdoctoral researchers, computational biologists, and molecular geneticists. Each group contributed 25% of the 1,000-persona study.

Task and design. Using GPT-5.6-Luna, all cohorts evaluated the same research-choice task, forming a PUT comparison with senior investigators as the reference cohort. The questionnaire included primary and secondary categorical choices, labeled B1 and B2, an evidence-comprehension check, and a complete-valid-response indicator. All 1,000 trials completed. The report specified 95% confidence and 12 comparisons without multiple-testing correction.

Results. Both choice distributions differed from the reference in every other cohort (all six p<.001p < .001; Table 5). Molecular geneticists showed the largest differences, with Cramér’s V=0.46V = 0.46 for B1 and 0.45 for B2, compared with 0.25–0.28 for the other groups. Evidence-comprehension contrasts were zero percentage points (p=1.000p = 1.000). Complete-valid-response rates were 0.4 percentage points lower in each nonreference cohort (p=.318p = .318).

Comparison cohortB1: VVB2: VV
Wet-lab PhD / postdoc0.260.28
Computational biologist0.250.27
Molecular geneticist0.460.45

Table 5. Research-choice differences across professional cohorts. Note. Each comparison is against the senior tumor-immunology PI/research-scientist cohort, with 250 personas per group. B1 and B2 denote the primary and secondary research choices. Entries are Cramér’s VV for categorical response distributions; all six uncorrected pp-values are below .001.

Insights. The simulated cohorts differed in research prioritization while showing no observed difference in the comprehension measure. This separates variation in scientific priorities from differences on that measured check. The strongest separation involved molecular geneticists, motivating discipline-specific analysis when designing expert consultation and subsequent research-planning studies.

C5: Remote-Service Marketplace

Summary. A remote-service marketplace asked which inspection services to prioritize, whether customer and agent payment expectations overlapped, and what prevented participation. Because simulated price estimates had no behavioural baseline to check against, the entire study was run twice, under gpt-5.6-sol and gpt-5.6-luna, on the identical cohort, so that model sensitivity could be measured rather than assumed.

Cohort. Two cohorts comprised 1,000 prospective customers and 1,000 prospective agents, drawn from synthesized personas with four quota-controlled dimensions each. Customers were aged 25–54 and distributed across North America (45%), Western Europe (35%), and Eastern Europe (20%), with upper-middle (45%), middle (35%), and high (20%) incomes, and a quota on trust disposition. Agents were distributed evenly across ten regions (10% each), aged 18–64, spanning low to upper-middle incomes, with a quota on risk tolerance. Realized marginals deviated from target by at most 0.1 percentage points for customers and 1.4 for agents. All remaining attributes varied freely.

Task and design. Each persona evaluated five scenarios: apartment inspection, used-car inspection, supplier verification, purchase verification, and document pickup. Scenario assignment was within-subject, so every persona answered all five and scenarios are compared within rather than between respondents. Questions covered clarity, trust in the individual agent, trust in the platform holding funds, participation intent, and payment expectations; agents additionally assessed personal safety. No price appeared in any scenario description, and payment expectations were elicited afterwards on a ten-rung ladder from $10 to $500 with an explicit refusal option, so that declining to transact was distinct from naming a low price. The design yielded 5,000 scenario-level responses per model per side, and 20,000 in total.

Results: scenario ordering. Table 6 compares indicative prices across models. The pooled clearing point was $85 under gpt-5.6-sol and $70 under gpt-5.6-luna. Document pickup had the lowest estimated price in both models; the highest-priced scenario was used-car inspection for Sol and apartment inspection for Luna. Scenario-price ranks had a correlation of 0.7.

ScenarioSol ($) | Luna ($)
Apartment inspection8693
Used-car inspection10088
Purchase verification9073
Supplier verification8068
Document pickup6548
Matched price-rung agreement across models
Participant groupMatched NNExact agreement
Customers4,99527.0%
Agents4,89631.5%

Table 6. Indicative service prices and matched cross-model agreement.

Note. Models are gpt-5.6-sol and gpt-5.6-luna. Prices compare stated customer ceilings and agent floors after a 10% commission; they are not observed market-clearing prices. Matched NN counts persona–scenario cells, not unique people.

Results: barriers to participation. Reported situational relevance diverged sharply. gpt-5.6-sol treated essentially the whole cohort as addressable, at 97.3% to 100% across scenarios, whereas gpt-5.6-luna placed relevance between 30.9% for document pickup and 75.2% for supplier verification. Conditioned on relevance, however, the two models agreed closely on what blocks adoption: distrust of the agent was named as the principal barrier by 92.1% of customers under gpt-5.6-sol and 95.6% under gpt-5.6-luna, while price was named by 0.04% under both. Customers trusted the platform holding their funds more than the individual performing the work in both models, at 3.99 against 3.70 under gpt-5.6-sol and 4.70 against 4.08 under gpt-5.6-luna on a five-point scale. Agents expressed greater confidence in being paid than in their personal safety, also under both models, at 4.01 against 3.15 and 4.74 against 3.05.

Results: model sensitivity. Because both models answered for the same personas in the same scenarios, agreement can be measured within each persona-scenario cell rather than only between aggregate distributions. Across 4,995 matched customer cells the two models selected the same price rung 27.0% of the time and landed within one rung 71.6% of the time, with a Spearman correlation of 0.41 and a mean gap of $25.40. The agent side agreed on 31.5% of 4,896 cells, 82.6% within one rung, with a correlation of 0.56. The matched comparisons reveal substantial sensitivity to model choice beyond differences visible in aggregate summaries.

Insights. The findings distinguish situational relevance from enthusiasm, and financial feasibility from interpersonal trust. Apartment inspection emerged as a candidate for pilot testing, alongside agent verification, accountability, and safety procedures. The results also delimit what simulation can settle. The models identify the same principal participation barrier and broadly similar price rankings, supporting pilot design; absolute prices and addressable-market estimates remain model-dependent. The latter depends largely on how many people a model judges could ever be in the situation, which varied by a factor of two. Prices represent simulated stated preferences rather than validated demand or market-clearing prices, and the relevance estimate is the quantity most worth establishing against a real audience before any figure here is scaled to a population.

C6: Professional-Network Signup

Summary. This study examined whether a professional-networking website appeared relevant and trustworthy enough for prospective users to begin registration.

Cohort. The cohort comprised 1,000 simulated professionals aged 25–74 across ten regions, diverse roles, and industries. Profiles varied in technical confidence, privacy priorities, trust, identity-verification experience, and exposure to scams or bots.

Task and design. Using GPT-5.6-Luna, personas explored public pages and chose to start signup, defer, or reject. Visits were limited to six pages and permitted inspection of the signup entrance, but prohibited account creation or submission of personal information.

Results. All 1,000 trials completed. Table 7 shows that reaching and understanding the signup entrance rarely translated into willingness to register. Mean product clarity was 7.64/10, trust was 5.06/10, and signup intent was 2.24/5.

OutcomeCount / valid NNRate
Reached signup entrance976/1,00097.6%
Would begin registration9/1,0000.9%
Deferred registration806/1,00080.6%
Rejected registration185/1,00018.5%
Next step reported clear975/1,00097.5%
Trust as primary decision basis985/1,00098.5%

Table 7. Website exploration and stated signup outcomes.

Note. Counts are descriptive. Reaching the entrance is an exploration action; willingness to register is a stated choice. Neither measures completed onboarding.

Insights. Knowing how to register did not imply willingness to do so. Generated explanations associated hesitation with unclear verification procedures, document retention, and data handling despite relevant networking features. The findings identify pre-registration disclosures as candidates for revision. Reaching the entrance measures exploration, while choosing signup records stated willingness; neither establishes completed onboarding or human conversion.

C7: Workplace Survey Responses

Summary. This study examined how simulated workers answered a workplace questionnaire, how team contexts changed responses, and whether model differences exceeded repeat variability.

Cohort. The cohort comprised 114 synthetic workers in eight teams across mining, manufacturing, transport, and other industrial services. Team sizes ranged from 5 to 38.

Task and design. Three linked experiments reused profiles and item order. GPT-4.1-mini answered an original 24-item questionnaire under mixed, supportive, and strained workplace contexts. GPT-4o-mini evaluated the baseline, while a 32-person panel completed three baseline calls per model. All 584 scheduled questionnaires were schema-valid.

Results. The favorable-coded baseline mean was 2.88/5; 64.8% of raw answers were neutral. All 114 personas gave more favorable responses under the supportive context. Table 8 reports this contrast and separates model differences from repeat variability. Baseline exact item agreement between models was 34.1%, with a mean absolute difference of 0.769 points. AI-assisted inspection identified five clear value–explanation contradictions among 192 sampled answers.

ComparisonEstimate95% interval
Supportive minus strained+3.85[3.84, 3.87]
Mean absolute item differences in the repeated panel
Within GPT-4.1-mini0.192[0.181, 0.203]
Within GPT-4o-mini0.469[0.435, 0.509]
Between models0.754[0.720, 0.791]
Between minus mean within-model0.423[0.393, 0.457]

Table 8. Workplace-context effects and repeat-panel variability.

Note. The context contrast uses 114 paired personas; the repeated panel uses 32 personas with three calls per model. Intervals use 10,000 complete-team resamples, preserving within-person responses. Effects are in questionnaire points; no significance labels are used.

Insights. Responses were strongly context-sensitive and model-dependent. Because six workplace conditions changed jointly, the context contrast does not isolate a single practice. Schema validity also did not ensure semantic consistency. The findings motivate explanation audits and repeat measurements before interpreting questionnaire summaries; the intervals describe synthetic teams, without human workforce benchmarking.

C8: Industrial Website Engagement

Summary. An industrial software provider asked whether its Swedish homepage and four product pages helped prospective customers understand the offering and choose an appropriate next step.

Cohort. Separate page cohorts comprised 1,000 homepage visitors, 300 rental-system visitors, and 200 visitors each for equipment, fleet, and driving-log products. The partner’s 90-day analytics informed the mix of prospective customers, existing customers, and other visitors. All 1,900 visits completed; 1,894 passed automated checks, including 1,088 prospect responses.

Task and design. Using Claude Opus 4.8, personas evaluated the first screen in a live browser, chose whether to continue, and inspected a next-step destination without entering personal information. They separately stated willingness to provide an email for a guide, receive a callback, attend a demo, or pilot the product on their own data. Each page was a one-cohort, one-stimulus study; browsing logs were retained alongside stated responses.

Results. Among prospects, 1,076/1,088 continued reading. Table 9 shows that continued interest did not imply readiness for sales contact, and no prospect accepted a pilot. Rental-system demo willingness exceeded callback willingness, indicating that the commitment items were not a sequential conversion funnel. On fleet and driving-log pages, “try for free” led to sales forms; destination expectation-match ratings were 2.86/5 and 2.28/5. The rental demo and equipment savings calculator scored 4.97/5 and 4.96/5, respectively.

PageNNEmailCallDemo
Homepage499377500
Rental system143872750
Equipment1505245
Fleet1461002820
Driving log15010431

Table 9. Stated commitments among prospective website visitors.

Note. NN is the valid prospect-response denominator; entries count affirmative stated intentions. Pilot acceptance was zero on every page. Items were answered separately and do not measure completed conversions. Pages used separate cohorts; cross-page comparisons are descriptive.

Insights. The study distinguishes willingness to explore from willingness to engage with sales. Requests for customer references, reported by 86–100% of prospects across pages, suggest that clearer proof and accurately labeled next steps merit testing before increasing commitment. These findings identify revisions for a subsequent pilot rather than establish human conversion rates.

C9: Engagement-Letter Evaluation

Summary. An AI-enablem ent partner supplied an illustrative accounting-service engagement letter and asked whether revised payment and liability terms would improve clients’ willingness to proceed.

Cohort. The frozen cohort contained 400 personas for a scenario involving Indian businesses registered for goods and services tax (GST) renewing an accounting-firm relationship. Four economic-motivation categories crossed with four trust categories formed a balanced reference grid, with 25 personas per cell. GST status and the existing-client relationship were scenario constants, rather than verified individual attributes; the grid did not estimate client-population proportions.

Task and design. The same personas evaluated four letters in isolated trials. Payment schedules varied from 40/30/30 to 30/35/35, and liability caps from service-specific fees to total annual professional fees; the annual fee remained INR 235,000. GPT-5.6-Sol and Claude Opus 5 each completed 1,600 valid trials. The primary outcome combined signing as-is and requesting minor changes. The combined-versus-baseline contrast used exact paired McNemar inference; 19 secondary factorial contrasts used Benjamini–Hochberg correction.

Results. Both models showed positive primary contrasts (Table 10; both exact p<1.5×10−11p < 1.5 \times10^{-11}). Favorability increased by 0.1375 points for GPT and 0.0175 for Opus on a five-point scale. Neither met the working success rules, including 70% acceptable intent and a 0.30-point favorability gain. In the combined condition, the 100-person hostile-trust group had no acceptable responses in either model.

ModelBaselineCombinedLift (95% CI)
GPT-5.6-Sol0.25%28.75%+28.50 [24.06, 32.94]
Claude Opus 511.50%20.75%+9.25 [6.40, 12.10]

Table 10. Engagement-letter acceptance across simulation models.

Note. Acceptance means signing as-is or requesting minor changes. Each model uses the same 400 personas per letter. Lifts and paired tt-based 95% intervals are in percentage points; significance uses exact McNemar tests. Intervals concern variation within the simulation design.

No-persona baseline. A separate GPT-5.6-Sol run completed 400 independent requests per letter without individual profiles, adding 1,600 valid responses. Acceptable intent was 0% for baseline and payment-only, 46.25% for liability-only, and 30.75% for the combined revision. The combined-minus-liability-only acceptance contrast was +17.25 percentage points with personas and −15.50 without them. Their directly estimated difference was +32.75 percentage points (95% CI [25.00, 40.50]).

Insights. The models agreed that joint revision improved acceptance but left the offer below the working threshold, despite differing in absolute response levels. This supports further testing of unresolved terms and low-trust audiences. The findings concern illustrative terms and simulated stated intent.

Healthcare Access Barriers

Summary. This study examined which barriers a healthcare service should prioritize in India, including care-seeking and treatment management.

Cohort. We constructed 900 simulated personas across nine region–age groups: North, Central, and South India crossed with ages 18–30, 31–50, and 51–75, with 100 personas per group. Selected demographics were aligned with Census 2011 data, and household resources with Household Consumption Expenditure Survey (HCES) 2023–24 records. The cohort included 609 rural and 291 urban personas. Selected diagnosed conditions were calibrated using the National Noncommunicable Disease Monitoring Survey (NNMS) 2017–18 and published Indian diabetes age tables.

Task and design. All personas completed the same English-language questionnaire covering eight areas, including care-seeking, medication-taking, and family influence. This was a response-characterization study with subgroup analyses.

Results. Table 11 shows common cost barriers, a pronounced rural–urban difference in distance barriers, and concentrated time constraints among ages 31–50. All 212 personas with fully scorable medication-adherence responses were classified as non-adherent under the study’s scoring rules.

Outcome / groupCount / valid NNRate
Cost barrier: all personas780/90086.7%
Time constraints: all personas691/90076.8%
Distance barrier: rural527/60986.5%
Distance barrier: urban8/2912.7%
Time constraints: ages 31–50277/30092.3%
Non-adherent: fully scorable responses212/212100.0%

Table 11. Healthcare-access barriers and subgroup patterns.

Note. Each row uses its eligible, valid-response denominator. Barriers permit multiple selections. Adherence classification applies only to fully scorable responses under the study scoring rules. These are descriptive simulated results.

Insights. The findings suggest three service priorities for further investigation: reducing financial barriers to care, improving transport accessibility for rural users, and accommodating work, household, and caregiving responsibilities when designing care services. Cost barriers were common across the cohort, distance barriers were concentrated among rural personas, and competing demands on time were most prominent among those aged 31–50. Uniform adherence classifications warrant examination of the instrument and simulation setup. These are findings from a reference-data-informed simulated cohort, without comparable human responses for behavioral benchmarking.

C11: Home Assistant

Summary. This study examined whether a home assistant supports users with different comfort and device-runtime preferences, and whether those outcomes depend on users volunteering relevant details or waiting to be asked.

Cohort. The study used 50 base personas filtered for adulthood, North American location, native US culture, and California residence. Each base persona had four matched versions crossing comfort-first versus shorter-air-conditioning-runtime priorities with proactive versus on-request disclosure. Other persona attributes were held fixed across versions; this is a controlled reference cohort, not an estimate of population proportions.

Task and design. Users interacted with an assistant controlling devices such as thermostats, fans, and blinds. Two tasks introduced a complication: the arrival time changed while preparing the home, or the bedroom heat load increased while maintaining comfort until a set time. Each of GPT-5.6-Sol, Claude Opus 5, and Gemini 3.7 Flash simulated all four persona versions on both tasks, giving 400 scheduled trials per model. A programmatic verifier separately recorded goal completion, constraints and plan satisfaction, preference satisfaction, and scope compliance. The reported outcome is the binary preference check, not overall task success. For each base persona, the comfort-first minus shorter-runtime contrast averages over both disclosure conditions. Opus and Gemini interacted with DashScope’s DeepSeek-V4.1-Flash; GPT interacted with DeepSeek’s deepseek-flash endpoint, so cross-configuration comparisons also reflect an assistant-endpoint difference.

Results. Preference satisfaction was higher for comfort-first personas under all three simulation models (Table 12). The difference was 37.0–45.0 percentage points before arrival and 53.0–81.0 points for bedroom comfort. The Opus before-arrival comparison retains 49 complete four-condition personas because one trial lacks final component scores; all other comparisons retain 50. Missing component scores are not filled with zero. An invalid-action trial remains a failure in the source’s overall-success accounting, which is distinct from this component analysis.

Note. Comfort-first minus shorter-runtime preference satisfaction, in percentage points, averaged over disclosure conditions. Intervals use 10,000 paired-persona bootstrap draws. N=50N = 50 complete personas except Opus before arrival (N=49N = 49). The scoring criterion follows each persona’s priority; Appendix § 10 describes the pairing.

Insights. The recurring priority gap identifies uneven support for users’ requirements: satisfying a comfort preference does not establish that the assistant can also minimize device runtime. Since each priority has its own preference criterion, the contrast is not a performance gain under one unchanged requirement. It motivates testing whether explicit trade-off elicitation and runtime-aware planning improve service for runtime-first users. This is a design hypothesis from simulated interactions, not evidence of measured energy savings or real-user satisfaction.

C12: Car Assistant

Summary. This study examined whether an in-car assistant preserves travel preferences when users share them proactively versus only when asked, including situations where navigation plans must change.

Cohort. Fifty base personas used the same adulthood, North America, native US culture, and California-residence filters as C11. Four matched versions crossed fastest-arrival versus toll-avoidance priorities with proactive versus on-request disclosure. Comparisons retained the base persona and travel priority while changing disclosure behavior.

Task and design. Two navigation tasks were evaluated: adding a breakfast stop within opening hours under a tighter arrival deadline, and canceling an intermediate stop while keeping the arrival deadline when the first navigation update failed. Each of the three user-simulation models evaluated 50 personas in four conditions on both tasks, giving 400 scheduled trials per model. The assistant endpoints and programmatic component checks followed the setup described for C11. The main-text comparison uses preference satisfaction for the cancellation task; each contrast compares 50 matched proactive/on-request pairs within a fixed travel priority.

Results. In the cancellation task, proactive disclosure increased preference satisfaction under both priorities for every simulation model (Table 13). Differences ranged from 18.0 to 44.0 percentage points. In the breakfast-stop task, direction depended on the simulation model: under toll avoidance, the proactive-minus-on-request differences were +4, −36, and +14 points for GPT, Opus, and Gemini, respectively. That task therefore does not support the same cross-model directional claim.

Note. Proactive minus on-request disclosure, in percentage points, for canceling an intermediate stop while retaining the arrival deadline. Each row uses 50 matched persona pairs with travel priority held fixed. Intervals use 10,000 paired-persona bootstrap draws (Appendix § 10).

Insights. The cancellation task exposes a service gap for users who wait to be asked: preserving their travel preference requires the assistant to elicit information rather than rely on unsolicited disclosure. The consistent positive contrast motivates testing preference clarification during navigation changes and recovery from failed updates. The breakfast-stop result limits the generalization: disclosure effects depend on the task and simulation configuration. These findings concern simulated navigation and preference checks, not validated driving safety or real-world travel outcomes.

Statistical Reporting Protocol

The protocol combines a common structure for trial-level evidence with analyses selected for each study’s design. Descriptive evaluations report response distributions; comparison studies additionally report contrasts and the procedures used to assess them.

Outcome records and denominators. Each trial produces structured numerical, categorical, or textual outcomes. Reports retain measurement scales and category definitions, summarize numerical outcomes and categorical counts, and preserve the valid-response denominator for each metric. Missing responses are distinguished from recorded choices. Explanations are linked to the outcomes they describe, and aggregate results remain traceable to individual trial records.

Comparisons and uncertainty. The numerical comparison module matches responses by persona or pairing identifier. Matched comparisons use paired tt-tests; unmatched comparisons use Welch’s tt-tests. Outputs include mean differences, standardized effects, and tt-based confidence intervals, with 95% confidence as the default. Categorical outcomes retain counts and proportions; the generic module also reports chi-square comparisons and Cramér’s VV. These categorical tests assume independent arm observations and do not account for shared-persona pairing, so their inferential interpretation requires verification for repeated-person designs. The engagement-letter study uses exact paired McNemar inference for its binary primary outcome. Case-specific analyses accommodate other dependencies: the workplace study uses 10,000 whole-team resamples, preserving within-person responses, to obtain percentile intervals. The analysis or resampling unit is documented alongside the result.

Direct persona-condition comparisons. We use 10,000 bootstrap draws (seed 20261005) to obtain exploratory pointwise percentile 95% intervals for differences between persona and no-persona conditions. In C1, complete persona pairs are resampled across framing arms; neutral responses are resampled independently by arm, and the two conditions independently. In C9, paired binary acceptance outcomes are resampled using the joint counts recovered from published paired sufficient statistics; no-persona responses are resampled independently by letter. Each reported difference is persona minus no-persona. The direct differences reported for C1 and C9 are 0.217 trust points in C1 (95% CI [0.183, 0.251]), and 32.75 percentage points in the combined-minus-liability acceptance contrast in C9 ([25.00, 40.50]).

Intervals for plotted estimates. Error bars in Figure 3 and Figure 4 use 10,000 pointwise percentile bootstrap draws (seed 20261005). For home assistance, each resampled base persona retains all four priority-by-disclosure conditions; the comfort-first minus short-runtime contrast is averaged across its two disclosure conditions. The Opus before-arrival comparison uses 49 complete four-condition personas because one component score is missing; all other home comparisons use 50. Car contrasts resample 50 personas with their two disclosure outcomes kept together within each priority, reporting proactive minus on-request preference satisfaction. Both cases use the recorded programmatic preference check. Clinical intervals resample categorical responses within each of four professional groups, retaining 200 responses per group. Engagement-letter differences resample 400 paired persona outcomes: Opus and Gemini pairs are matched by persona ID, and GPT uses the uniquely recovered joint counts from published sufficient statistics. The GPT no-persona arms are resampled independently, with 400 responses per letter. Home, car, and engagement intervals apply to the plotted differences; clinical intervals apply to Cramér’s VV.

Multiple comparisons and subgroup analysis. Benjamini–Hochberg correction is available for the family of tested arm–metric comparisons and is optional; the default is uncorrected inference. Reports identify the correction used. Original music-wellbeing reports apply correction to overall comparisons; the local-data contrasts and new bootstrap intervals reported here are pointwise and unadjusted. The engagement-letter study applies correction to 19 secondary factorial contrasts, with a single prespecified primary contrast; its new direct persona-condition interval is also pointwise and unadjusted. Generic subgroup significance labels use raw pp-values and are treated as exploratory unless a separate correction is documented. Outcome-linked explanations support interpretation without constituting an independent statistical test.

Scope of the intervals. The intervals describe variation within the specified simulation design, conditional on the model, cohort construction, and task. With one response per persona and condition, observed variation combines differences across personas with stochastic generation. Repeated trials with fixed personas further assess generation variability, as in the workplace study. Neither these intervals nor agreement across models quantifies discrepancies from real-user behavior; that assessment requires matched human reference data.

References

  1. [1]Gati V Aher, Rosa I. Arriaga, and Adam Tauman Kalai. Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 337–371. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/aher23a.html.arxiv.org/abs/2208.10264
  2. [2]Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis, 31(3):337–351, Feb 2023. doi: 10.1017/pan.2023.2.DOI
  3. [3]James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4):401–416, May 2024. doi: 10.1017/pan.2024.5.DOI
  4. [4]Nicolas Bougie and Narimasa Watanabe. SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation. In Georg Rehm and Yunyao Li, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 43–60, Vienna, Austria, jul 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-industry.5.DOI
  5. [6]Jiawei Chen, Ruoxi Xu, Boxi Cao, Ruotong Pan, Yunfei Zhang, Yifei Hu, Yong Du, Tingting Gao, Han Li, Yaojie Lu, Yingfei Sun, Xianpei Han, Le Sun, Xiangyu Wu, and Hongyu Lin. Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces, 2026a. URL https://arxiv.org/abs/2604.08362v3. arXiv:2604.08362v3.
  6. [7]Zihan Chen, Di Zhu, and Lei Nico Zheng. When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses, 2026b. URL https://arxiv.org/abs/2607.26348v2. arXiv:2607.26348v2.
  7. [8]Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35212–35290, Suzhou, China, nov 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.emnlp-main.1786.DOI
  8. [9]Bangde Du, Minghao Guo, Songming He, Ziyi Ye, Xi Zhu, Weihang Su, Shuqi Zhu, Yujia Zhou, Yongfeng Zhang, Qingyao Ai, and Yiqun Liu. TwinVoice: A Multi-dimensional Benchmark Towards Digital Twins via LLM Persona Simulation. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Findings of the Association for Computational Linguistics: ACL 2026, pages 19604–19628, San Diego, California, United States, jul 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.findings-acl.981.DOI
  9. [10]Laura Faulkner. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3):379–383, Aug 2003. doi: 10.3758/bf03195514.DOI
  10. [11]John D. Gould and Clayton Lewis. Designing for usability: key principles and what designers think. Communications of the ACM, 28(3):300–311, 1985. doi: 10.1145/3166.3170.
  11. [12]Luke Guerdan, Justin Whitehouse, Kimberly Truong, Kenneth Holstein, and Zhiwei Steven Wu. Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas. In The Fourteenth International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2509.22957v2.
  12. [13]Tiancheng Hu, Joachim Baumann, Lorenzo Lupo, Nigel Collier, Dirk Hovy, and Paul Röttger. SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors. In The Fourteenth International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/2fc4d6a6400ddda7493b9db258c34e0-Abstract-Conference.html.
  13. [14]Chengpiao Huang, Yuhang Wu, and Kaizheng Wang. Uncertainty Quantification for LLM-Based Survey Simulations. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 25947–25971. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/huang25am.html.
  14. [15]Jintao Huang, Yifan Wang, Hongyu Shen, Yi Daniel Lu, Shirley Huang, Minsik Oh, Yewen Wang, Muhammad Ahmed Mohsin, Zhen Xu, Yilan Fan, Zichen Yuan, Ahsan Bilal, Zibu Wei, Sankalp Jajee, Henry Gagnier, Saksham Kapoor, Jicheng Wang, Qianfeng Wen, Yixuan He, Steven Dillmann, Jiashu He, Yucheng Lu, Linqiang Guo, Danyang Zhang, Shi Bo, Raunak Mondal, Haixiang Tang, Weihang Xiao, Allen Nie, Ting Tang, Yueying Li, Yifan Simon Liu, Jianheng Hou, Dianzhuo Wang, Qianyu Zhu, Zhixu Silvia Tao, Zhejian Peng, Zihan Wang, Ishan Gupta, Jinxuan Fan, Wanting Jiang, Shushu Liang, Chenxi Qiu, Yijun Wang, Xiaomin Li, and Yuexing Hao. AgentPersonaBench: Benchmarking Persona-Driven User Simulation, 2026. URL https://arxiv.org/abs/2610.04379v1. arXiv:2610.04379v1.
  15. [16]Jessica Hullman, David Broska, Huaman Sun, and Aaron Shaw. This human study did not involve human subjects: Validating LLM simulations as behavioral evidence, 2026. URL https://arxiv.org/abs/2602.15785v1. arXiv:2602.15785v1.
  16. [17]Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’13, pages 1168–1176. ACM, Aug 2013. doi: 10.1145/2487575.2488217.DOI
  17. [18]Stefan Krsteski, Giuseppe Russo, Serina Chang, Robert West, and Kristina Gligorić. Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10887–10906, San Diego, California, United States, jul 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.acl-long.498.DOI
  18. [20]Xiaomin Li, Yuexing Hao, Jianheng Hou, Jintao Huang, Qianfeng Wen, Shirley Huang, Yifan Liu, Xiaoyi Liu, Yilan Fan, Yijun Wang, Koutian Wu, Ruoqi Gao, Muhammad Ahmed Mohsin, Jing Tang, Brihi Joshi, Heming Liu, Zheyuan Deng, Zonglin Di, Sankalp Jajee, Jiuyoo Lu, Zhiwei Zhang, Saksham Kapoor, Ishan Gupta, Yunhan Zhao, Chanwoo Park, Yucheng Lu, Bing Hu, Weihang Xiao, Aravind Mohan, Hanwen Xing, Runyu Zhang, Mihir Kulshreshtha, Yuanda Xu, Qianyu Zhu, Dianzhuo Wang, Yuxin Xiao, Bowen Jiang, Yongye Su, Wenhao Chai, Zuxin Liu, Lawrence Yunliang Chen, Xuandong Zhao, Ethan Ye, Shivam Patel, Jason Xie, Alex Martin Richmond, Weixiang Ding, Emre Okcular, Diya Mathew, Ziheng Wang, Rana M. Shahroz Khan, Zhejian Peng, Fang Wu, Fan Nie, Xinyang Han, Yubin Kim, Jiawei Zhang, Zhenting Qi, Huangyuan Su, Xu Pan, Abinitha Gourabathina, Hyewon Jeong, Hemanth Neelgund Ramesh, Kumail Alhamoud, Kimia Hamidieh, Zidi Xiong, Samuel Schmidgall, Pengrui Han, Yepeng Huang, Yongheng Wang, Bowen Yang, Alex Gu, Yuchu Wang, Akshay Parurchuri, Brenna Li, Hejie Cui, Jiayuan Ding, Chaos heng Dong, Jiahao Wang, Yixuan He, Chi Wang, Pamela Bhattacharya, Tianyi Peng, Paul Pui Liang, Mitchell Gordon, Yilin Du, Marinka Zitnik, James Zou, Prasanna Tambe, Philip Torr, Emily Fox, Asu Ozdaglar, and Dawn Song. MatrAIx: Simulating the World with 8.3 Billion Persona Agents, 2026. URL https://arxiv.org/abs/2608.04205v1. arXiv:2608.04205v1.
  19. [21]Yifan Simon Liu, Qianfeng Wen, Yilan Fan, Shirley Huang, Ruoqi Gao, Jianheng Hou, Muhammad Ahmed Mohsin, Zonglin Di, Brihi Joshi, Xincheng Tan, Yucheng Lu, Xiaoyi Liu, Heming Liu, Hanwen Xing, Guanghui Min, Zhengyang Shan, My Chiffon Nguyen, Ishan Gupta, Yunze Xiao, Hannah Collison, Jintao Huang, Jiatong Li, Sankalp Jajee, Yunhan Zhao, Bing Hu, Sky Ng, Xupeng Chen, Binghang Lu, Weihang Xiao, Aravind Mohan, Bolun Sun, Yunshu Wu, Yuanda Xu, Yun Shen, Runyu Zhang, Zheyuan Deng, Zhiwei Zhang, Qianyu Zhu, Dianzhuo Wang, Yijun Wang, Yixuan He, Yuexing Hao, and Xiaomin Li. PersonaEval: Persona-Based User Simulation for Evaluating Interactive Applications, 2026. URL https://arxiv.org/abs/2608.15838v1. arXiv:2608.15838v1.
  20. [22]Ofer Meshi, Krisztian Balog, Sally Goldman, Avi Caciularu, Guy Tennenholtz, Jihwan Jeong, Amir Globerson, and Craig Boutilier. ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders. In Vera Demberg, Kentaro Inui, and Lluís Marquez, editors, Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5270–5304, Rabat, Morocco, mar 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.eacl-long.244.DOI
  21. [23]Sky Ng, Brihi Joshi, Ishan Gupta, Shirley Huang, Zonglin Di, Yun Shen, Qianfeng Wen, Yifan Simon Liu, Ruoqi Gao, Yilan (Eliza) Fan, Zhiwei Zhang, Muhammad Ahmed Mohsin, Yucheng Lu, Xiaoyi Liu, Heming Liu, Qianyu Zhu, Hanwen Xing, Zhengyang Shan, My Chiffon Nguyen, Guanghui Min, Jianheng (Jaden) Hou, Yunze (Lorenzo) Xiao, Keyang Xuan, Hannah Collison, Jintao Huang, Jiatong Li, Sankalp Jajee, Yunhan Zhao, Bing Hu, Xupeng Chen, Binghang Lu, Weihang Xiao, Aravind Mohan, Bolun Sun, Yunshu Wu, Yuanda Xu, Runyu Zhang, Zheyuan Deng, Xinch en (Cara) Tan, Dianzhuo Wang, Yijun Wang, Yixuan He, Koutian Wu, Cheng Cheng, Xiaomin Li, and Yuexing Hao. MicroVerse: An Instrument for Measuring Self-Authored Identity Drift in Long-Horizon Multi-Agent Language-Model Simulations, 2026. URL https://arxiv.org/abs/2608.15844v1. arXiv:2608.15844v1.
  22. [24]Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Social Simulacra: Creating Populated Prototypes for Social Computing Systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–18. ACM, 2022. doi: 10.1145/3526113.3545616.DOI
  23. [25]Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22. ACM, 2023. doi: 10.1145/3586183.3606763.DOI
  24. [26]Tianyi Peng, George Gui, Melanie Brucks, Daniel J. Merlau, Grace Jiarui Fan, Malek Ben Sliman, Eric J. Johnson, Abdullah Althenayyan, Silvia Bellezza, Dante Donati, Hortense Fong, Elizabeth Friedman, Ariana Guevara, Mohamed Hussein, Kinshuk Jerath, Bruce Kogut, Akshit Kumar, Kristen Lane, Hannah Li, Vicki Morwitz, Oded Netzer, Patryk Perkowski, and Olivier Toubia. Digital Twins as Funhouse Mirrors: Five Key Distortions, 2026. URL https://arxiv.org/abs/2509.19885v5. arXiv:2509.19885v5.
  25. [27]Joel Persson, Mårten Schultzberg, and Sebastian Ankargren. Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference, 2026. URL https://arxiv.org/abs/2606.17165v3. arXiv:2606.17165v3.
  26. [28]Jinghua Piao, Yuwei Yan, Jun Zhang, Nian Li, Junbo Yan, Xiaochong Lan, Zhihong Lu, Zhiheng Zheng, Jing Yi Wang, Di Zhou, Chen Gao, Fengli Xu, Fang Zhang, Ke Rong, Jun Su, and Yong Li. AgentSociety: Large-scale simulation of LLM-driven generative agents advances understanding of human behaviors and society. iFuture, 2026. doi: 10.26599/IF.2026.9710004.DOI
  27. [29]Sebastian Pohl, Harsh Mehta, Pranav Mambayil, Abdul Ghafoor, Franziska Lesigang, Yufang Hou, and Christian Hilbe. LLMs struggle to simulate human belief updates in controlled environments, 2026. URL https://arxiv.org/abs/2607.28347v2. arXiv:2607.28347v2.
  28. [30]Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose Opinions Do Language Models Reflect? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 29971–30004. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/santurkar23a.html.arxiv.org/abs/2303.17548
  29. [31]Olivier Toubia, George Z. Gui, Tianyi Peng, Daniel J. Merlau, Ang Li, and Haozhe Chen. Database Report: Twin-2K-500: A Data Set for Building Digital Twins of over 2,000 People Based on Their Answers to over 500 Questions. Marketing Science, 44(6):1446–1455, 2025. doi: 10.1287/mksc.2025.0262.DOI
  30. [32]Alexander Sasha Vezhnevets, John P. Agapiou, Avia Aaron, Ron Ziv, Jayd Matyas, Edgar A. Dueñez-Guzmán, William A. Cunningham, Simon Osindero, Danny Karmon, and Joel Z. Leibo. Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia, 2023. URL https://arxiv.org/abs/2312.03664v2. arXiv:2312.03664v2.
  31. [33]Jia Wang, Ziyu Zhao, Tingjuntao Ni, and Zhongyu Wei. SocioBench: Modeling Human Behavior in Sociological Surveys with Large Language Models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26257–26289, Suzhou, China, nov 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.emnlp-main.1335.DOI
  32. [34]Ziyi Yang, Zaibin Zhang, Zirui Zheng, Yuxian Jiang, Ziyue Gan, Zhiyu Wang, Zijian Ling, Jinsong Chen, Martz Ma, Bowen Dong, Prateek Gupta, Shuyue Hu, Zhenfei Yin, Guohao Li, Xu Jia, Lijun Wang, Bernard Ghanem, Huchuan Lu, Chaochao Lu, Wanli Ouyang, Yu Qiao, Philip Torr, and Jing Shao. OSAS: Open Agent Social Interaction Simulations with One Million Agents, 2025. URL https://arxiv.org/abs/2411.11581v5. arXiv:2411.11581v5.
  33. [35]Se-eun Yoon, Zhankui He, Jessica Echterhoff, and Julian McAuley. Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1490–1504, Mexico City, Mexico, jun 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.83.DOI
  34. [36]Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Sherry Wu, and Maarten Sap. Mind the Sim2Real Gap in User Simulation for Agentic Tasks. In The Third Conference on Language Modeling, 2026. URL https://openreview.net/forum?id=OnVi1BETk0.arxiv.org/abs/2603.11245
  35. [37]Ming Zhu, Juntao Tan, Ritesh Murthy, Jielin Qiu, Liangwei Yang, Wenting Zhao, Silvio Savarese, Shelby Heinecke, and Huan Wang. RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation. In The Third Conference on Language Modeling, 2026. URL https://openreview.net/forum?id=1izVEEZAxT.arxiv.org/abs/2605.20204

Paper details

vlnc:W_aae78k8m
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents