WorksComputer Science
Executable Human Study Programs with Silicon Participants
Abstract
We introduce HumanStudyProgram, a framework that represents a documented human study as an executable program with configurable LLM-based silicon participants. Its computational abstraction separates study logic from participant implementation through standardized APIs for presenting information and collecting responses. We implement a researcher-facing authoring tool that reconstructs source-linked study descriptions from papers and supplementary materials, supports researcher review, and generates executable HumanStudyProgram artifacts. Researchers can then configure silicon participants, run the programs, and analyze results from aggregate findings to individual responses. We demonstrate the approach through study cases and evaluate the tool through a hands-on expert review with 12 social science researchers. All completed the workflow and identified potential uses in understanding published protocols, prototyping studies, and examining AI behavior. We aim to improve the reproducibility of human studies by reconstructing their documented procedures as programs that others can reuse and build upon.
Introduction
Major HCI venues (e.g., CHI) now encourage authors to provide study materials and ask reviewers to consider their availability[1, 2]. The proportion of CHI papers that shared interview protocols increased from 2% in 2017 to 25% in 2022[3, 4]. However, using these materials still requires substantial effort[5]. First, researchers must recover the exact wording, task order, and condition assignments from information scattered across the methods section, figures, appendices, and supplementary materials[4, 6]. Second, researchers have to reconstruct the complete protocol from high-level descriptions, such as “tasks were counterbalanced in order and randomized across conditions”. Third, to test these protocols, researchers often need to secure ethics approval, recruit and compensate participants, and administer the study.
In contrast, empirical machine learning (EML) has benefited from an infrastructure for frictionless reproducibility[7], in which researchers can build directly on one another’s digital artifacts. Researchers routinely release repositories containing datasets, model weights, training code, and evaluation scripts[8, 9]. To build on a published result, a researcher can clone its repository, reproduce the reported findings, and inspect the code to diagnose discrepancies. Once the baseline is established, they can modify individual components—substituting a dataset, adjusting a hyperparameter, or replacing a model module—and rerun the pipeline to observe the effects. Each iteration requires only a fraction of the effort needed to reconstruct the original work, while shared repositories accumulate fixes and extensions contributed by others. This capacity for rapid verification and iteration is widely regarded as a key driver of AI’s striking progress[7].
This paper presents HumanStudyProgram, a researcher-facing framework that reconstructs human study protocols documented in papers and supplementary materials as executable programs and interfaces the programs with configurable LLM-based silicon participants (Figure 1). The resulting program is a reusable artifact that researchers can inspect and modify. Running it with silicon participants generates structured synthetic response data without recruiting a new human sample, allowing researchers to exercise the procedure and analysis end to end, identify implementation problems, and prototype alternative questions or conditions. HumanStudyProgram targets interviews, surveys, and in-lab experiments whose essential procedures can be represented as exchanges of information and responses. These agent-based runs complement human-participant research by supporting protocol inspection and early prototyping[10]; they do not replace human evidence for claims about people [11, 12, 13, 14].
At the core of HumanStudyProgram is our observation that diverse human study protocols share a common interaction structure: they present information, collect responses, and use procedural logic to determine what happens next. This observation allows us to separate study logic from the mechanisms used to interact with participants. We introduce two API operations to realize this separation of concerns for participants: Show presents information to one or more participants, while Ask poses a question and returns their responses (Figure 3(b)). A HumanStudyProgram uses these operations for all participant-facing interactions and ordinary Python for condition assignment, control flow, state updates, computation, and coordination. The protocol therefore specifies what participants receive and return without depending on how a participant is implemented. This separation supports studies ranging from single-question surveys to multi-round, multi-participant experiments and allows researchers to vary silicon participants while holding the protocol fixed.

Figure 3. Participant and Study Abstractions. Example code for the two abstractions. In (a), a Participant combines a model and specification with memory enabled by default; inquiry retains an interaction when memory is enabled, whereas interrogate does not. In (b), Show and Ask apply these operations to a group, and Study binds that group to an executable procedure. In Figure 1, the highlighted procedure calls these Show and Ask operations; Study.execute invokes the function with the configured participants and returns its records. The code omits runtime details.
Figure 1. The HumanStudyProgram Artifact and Its Authoring Tool. A HumanStudyProgram represents a human study as an executable program: Show and Ask express participant-facing interactions, while ordinary Python expresses the study logic connecting them. We implement this abstraction in an authoring tool with two linked stages. ① Build Study generates a HumanStudyProgram from a paper and its supplementary materials. ② Playground configures silicon participants, runs the generated HumanStudyProgram, and analyzes its results. The two stages form an iterative workflow for generating and running HumanStudyPrograms.
Behind Show and Ask, the runtime instantiates each configured agent, assembles its context, invokes the model, parses its response, and manages its memory across interactions. Agent memory is enabled by default. Show adds presented information to the participant’s context so that later interactions can refer to it. By default, Ask similarly records both its question and the returned response after invoking the configured model. A study can instead call Ask with remember=False to collect a response without adding that exchange to subsequent context. Such probes still read existing memory; only the new question and response are excluded. These semantics support both isolated measurements and stateful, multi-round interactions without requiring each study program to implement agent memory or model-specific prompting.
We implement an authoring tool for creating and running HumanStudyProgram artifacts through two linked stages (§ 6). In Build Study, an LLM-assisted pipeline extracts Materials, Participants, Procedure, Response, and Analysis from a paper and its supplementary files, links each component to its source, and flags missing, unclear, or conflicting information for researcher review. Once the researcher approves this description, the system generates an executable package containing the confirmed study materials, participant definitions, a Python procedure module, and a measurement module for the analysis; it then validates the code and performs a simulated trial run. In Playground, researchers configure the model, participant specification, memory, population, and condition assignments, execute the generated package, and inspect results from aggregate findings to individual interactions. The runtime records each agent’s configuration, visible context, raw output, parsed response, and resulting study state.
We validate the implementation through study demonstrations and evaluate its use through a hands-on expert review. The demonstrations use four cases from three cognitive, strategic, and social-psychology studies to examine recovery of missing information, execution of a multi-round group protocol, transparency into agent context, and navigation from findings to individual responses. In the expert review, 12 doctoral-level social science researchers each selected a familiar study, corrected and approved its reconstructed description, configured and ran silicon participants, and interpreted the results. All 12 completed the workflow, collectively reviewing 98 flagged issues across their study packages. They identified value in understanding published protocols, prototyping studies before human data collection, and applying established protocols to study AI behavior, while emphasizing that simulated results cannot substitute for human evidence.
This paper makes three contributions. First, it introduces a computational abstraction for human studies that combines participant-facing Show and Ask operations with ordinary program logic, separating an executable protocol from the agents used to run it. Second, it contributes an authoring tool for creating and executing HumanStudyProgram artifacts, supporting source-linked protocol reconstruction, researcher review and approval, configurable silicon participants, and analysis from aggregate findings to individual responses. Third, it provides evidence from study demonstrations and a hands-on review with 12 social science researchers, showing the abstraction’s ability to represent varied study procedures and identifying its value for understanding protocols, prototyping studies, and examining AI behavior.
Background and Related Work
We review prior work on reproducibility and reuse in human-subject research, tools for authoring and running human studies, and LLMs in behavioral research.
Reproducibility and Reuse in Human-Subject Research
Efforts to improve the reproducibility of human-subject research initially centered on conducting more direct replications (Table 1). In HCI, RepliCHI called for the community to replicate and validate prior findings more systematically[15]. Large collaborative projects subsequently coordinated replications across laboratories and participant samples, including Many Labs, the Reproducibility Project: Psychology, and replications of social science experiments published in Nature and Science[16, 17, 18]. These projects made replicability empirically assessable, but each study still had to be recovered from the published record, reimplemented, and run with newly recruited participants. They therefore demonstrated both the value of reuse and the substantial coordination and execution costs it entails.
| Period | Approach | Reusable object | Examples |
| 2011–2018 | Direct replication | Study findings with newly recruited participants | RepliCHI [15]; Many Labs [16]; Reproducibility Project: Psychology [17]; Camerer et al. [18] |
| 2017–2023 | Artifact sharing | Study plans, materials, data, and analysis code | Open Science Framework [19]; HARK No More [20]; CHI Artifact Transparency [21]; ACM Artifact Badging [1]; Living Papers [22] |
| 2025–2026 | Agentic reproduction | Existing computational experiments and analyses | CORE-Bench [23]; Paper2Code [24]; ReplicatorBench [25] |
Table 1. Historical progression toward reusable human-subject research.
The next wave of work sought to make future studies easier to verify and reuse by preserving their plans and artifacts. The Open Science Framework provided infrastructure for registering study plans and sharing protocols, stimuli, data, and code[19]; preregistration was subsequently advocated within CHI to commit hypotheses and analyses before results were known[20]. Artifact surveys and ACM’s review and badging framework then shifted attention from whether materials were shared to whether they were documented, functional, and reusable[15, 21]. More recently, Living Papers demonstrated how scholarly documents can incorporate executable code and interactive components rather than remain static accounts[22]. Together, these efforts make research outputs increasingly available, structured, and interactive. However, they support the reuse of artifacts and documents rather than making the participant-facing study itself rerunnable.
Recent work has begun to use agents to support the reproduction of published experiments[26, 27, 28]. CORE-Bench tests whether agents can re-execute computational experiments using paper-associated code and data[23]; Paper2Code generates implementations from machine-learning papers[24]; and ReplicatorBench extends automated reproduction to statistical analyses reported in social science papers[25]. These computational approaches either target computational studies or, when applied to human-subject research, reproduce only the post-data-collection analysis over existing data. HumanStudyProgram extends reuse to the study procedure itself, reconstructing human studies as programs that can generate new responses through silicon participants.
Tools for Authoring and Running Human Studies
Behavioral experiment software has tried to make human studies programmatically executable. Tools such as PsychoPy, jsPsych, and lab.js support the implementation of experimental procedures as sequences of stimuli, trials, and response collection[29, 30, 31], while experiment repositories have packaged such implementations for reuse[32]. These tools provide precise control over experimental delivery, but their abstractions primarily center on individual human interactions: presenting stimuli through an interface and capturing responses through input devices.
Later platforms extended executable experimentation to multi-participant settings. z-Tree, oTree, Empirica, and LIONESS Lab support rounds, roles, information exchange, and payoff structures required by social and economic experiments [33, 34, 35, 36]. These platforms help researchers implement studies they plan to conduct. HumanStudyProgram instead turns studies already described in papers and supplementary materials into executable programs, without requiring researchers to implement each study from scratch.
LLMs in Behavioral Research
Researchers use LLMs as silicon participants both to model human behavior and to study AI behavior in its own right. In the former role, agents are used to reproduce findings from human studies, model the responses of particular populations, and benchmark agent–human correspondence across tasks and populations [37, 38, 39, 40, 41, 42, 43, 44]. In the latter, established human study paradigms provide controlled settings for examining how AI agents perceive, decide, and interact [45, 46, 47, 48, 49].
Whether agent–human resemblance is necessary therefore depends on the claim being made. Claims about people require correspondence to be empirically established rather than assumed [11, 12, 13, 14, 50, 51]; when AI behavior itself is the object of inquiry, both correspondence with and departure from human findings are informative. Across these purposes, silicon participants have been used in individual surveys and behavioral tasks, as well as multi-agent simulations of social interaction, collective behavior, and group coordination [52, 53, 54, 55].
Another line of work explores how silicon participants should be specified, from demographic personas and autobiographical back-stories [13, 56, 57] to behavioral fine-tuning [58, 59]. Empirical studies show that simulated behavior depends not only on the underlying model, but also on persona conditioning, prompt phrasing, option ordering, and response format [60, 61, 62, 63]. A simulated response must therefore be interpreted together with the participant configuration and experimental context that produced it [64].
HumanStudyProgram makes human study protocols available as reusable programs for investigating AI behavior, supporting the research directions reviewed above. Researchers can vary the study procedure and participant configuration independently and trace results to the interactions that produced them.
Design Process
The design of HumanStudyProgram drew on three complementary activities: examination of published study procedures, informal discussions with social science researchers, and iterative implementation of study programs. These activities addressed two design questions: which operations are needed to express different study procedures, and how to support researchers in reconstructing, executing, and interpreting those procedures with silicon participants.
Understanding Study Procedures. We reviewed more than 50 papers reporting human-subject studies. We focused on social experiments designed to test theories through participant behavior. For each study, we read the paper and available supplementary materials and traced what participants were shown, what responses were requested, how the study state changed, and how the responses were used in subsequent interactions or analysis. We also considered how conditions and participant roles altered this sequence. These relationships informed the five-part study description: Materials, Participants, Procedure, Response, and Analysis.
Across these studies, we found a stable distinction between interactions with participants and the program logic that organizes those interactions. We began with an initial API and refined it while implementing 13 studies spanning single-response tasks, multistage procedures, repeated interactions with feedback, role-based multi-participant experiments, and studies containing multiple empirical substudies. Through this process, we developed two participant-facing operations: Show, which presents information, and Ask, which poses a question and collects responses. Condition assignment, branching, computation, state updates, and participant coordination remain expressible through ordinary program logic. These abstractions are formalized in Section 4.
Understanding Researcher Needs. Over more than six months, we held informal discussions with approximately 30 doctoral-level social science researchers who conduct human studies, primarily in psychology, management, economics, and sociology. We discussed how they reconstruct published procedures, configure study runs, and interpret results, and used their feedback while iterating on the tool. Through this process, we arrived at three considerations for the surrounding workflow: source-grounded reconstruction, configurable and inspectable execution, and traceable interpretation.
For reconstruction, these discussions motivated us to support comparison of extracted study information with its sources and resolution of missing, inconsistent, or incorrectly extracted details before execution. We therefore designed a review interface that places study information alongside source evidence and requires approval of the reconstructed description.
For execution, researchers needed to configure silicon participants individually or as groups, assign them to conditions or roles, and see the complete context provided to each participant. They also needed to control participant memory: context should carry across interactions when required by the study, while isolated measurements should not affect later responses. We therefore separated study programs from reusable participant and group configurations, made execution context visible, and supported interaction-level memory control through remember=False.
For interpretation, the discussions motivated support for tracing aggregate findings back to condition-level results, analysis outputs, and individual responses. We designed linked views across these levels to help researchers examine how a reported result was produced. Together, these design choices connect source review, participant configuration, and result inspection to the executable study abstraction; Section 6 describes the corresponding interfaces.
HumanStudyProgram Abstractions
Our design process identified participant-facing interactions and study logic as a recurring structure across study procedures. We formalize that structure here and show how it becomes a program.
Study and Participant Abstractions
A HumanStudyProgram represents a study through five connected parts: Materials, Participants, Procedure, Response, and Analysis. Materials contain the information presented in the original study, and Participants describe its reported population and roles. Procedure determines how the study unfolds and produces data structured by Response; Analysis defines how those data are transformed into study findings. Figure 2 situates Procedure within these five parts.
Figure 2. Representing Study Procedure Structure. The five-part study description situates Procedure within Materials, Participants, Response, and Analysis (left). A graph illustrates Procedure using Participant Actions and Study Operations, with edges carrying responses and context between them. Different compositions express independent conditions, repeated group feedback, and information exchange between participant roles (right).
Within Procedure, we distinguish between interactions with participants and the study logic that organizes those interactions. A Participant Action invokes one or more participants to present information or collect a response. A Study Operation acts on the study without invoking a participant, for example by assigning conditions, performing computations, updating study state, or routing information. For explanatory purposes, we illustrate a procedure as a graph in which edges carry responses from Participant Actions to Study Operations and return the resulting context to later Participant Actions. The executable procedure itself remains an ordinary Python program. Figure 2 shows how these operations compose to express independent conditions in the Framing Effect [65], repeated group feedback in the -Beauty Contest [66], and information exchange between roles in the Trust Game [67].
Participant Actions operate on silicon participants that persist throughout a study run. Each participant maintains a model configuration, a participant specification, and private memory that is enabled by default across interactions. Figure 3(a) shows this participant representation and how an interaction may update or leave its memory unchanged.
We implement Participant Actions through two study API operations: Show presents information to one or more participants, and Ask poses a question and returns their responses. Ordinary Python expresses the surrounding study logic. The Study abstraction binds a configured set of participants to a procedure and executes that procedure. Figure 3(b) shows these relationships in code, and Table 2 summarizes the operations provided by the study API and by Python.
| Supported operation | Example |
| Provided by the study API | |
| Define a study. | Study(participants, procedure) -> Study |
| Execute a study with silicon participants. | study.execute() -> T; raises StudyExecutionError on failure. |
| Present content to silicon participants. | Show(participants, content) -> None; raises ParticipantInteractionError on failure. |
| Collect responses from silicon participants. Pass options for a closed response set, and remember=False to avoid updating memory. | Ask(participants, question, options=None, remember=True) -> dict[ParticipantID, Response]; raises ParticipantInteractionError on failure. |
| Provided by Python | |
| Select participants. | group = [p for p in participants if p.role == "investor"] |
| Apply configured roles or conditions. | p.condition = assignments[p.id] |
| Sequence activities. | Show(group, tutorial); answers = Ask(group, question) |
| Repeat activities or rounds. | for round in range(4): ... |
| Branch the procedure. | if answer == "yes": ... |
| Maintain and update study state. | state = {"round": 1}; state["round"] += 1 |
| Perform calculations. | target = (2 / 3) * mean(guesses.values()) |
| Represent study materials. | question = "Choose a number from 0 to 100" |
| Pass information between participants. | Show(trustees, transfers) |
| Construct returned study data. | return {"guesses": guesses, "target": target} |
| Handle failures. | try: responses = study.execute() |
| Run activities concurrently. | responses = await asyncio.gather(*tasks) |
| Analyze returned data. | results = analyze(responses) |
Table 2. Operations supported by HumanStudyProgram study API and by Python.
Expressing Different Study Procedures
We first use a stateful, multi-round procedure to show how Participant Actions and Study Operations work together in an executable program. In the -Beauty Contest [66], each participant chooses a number between 0 and 100, and the participant whose choice is closest to times the group mean wins. The study repeats this process for four rounds and shows the group result after each round, so later choices are made with the results of earlier rounds. Figure 4 traces these repeated choices, group calculation, and feedback from their source passages to a reviewed procedure and its Python implementation. Participant Actions collect estimates through Ask and present group feedback through Show; Study Operations compute the group result and control the repeated rounds. More examples show how the same operations express procedures with different structures. Figure 5 uses independent condition assignment in a framing study[65], repeated item and condition assignment in an anchoring study[68], and information transfer between roles in the Trust Game[67].
Figure 4. Representing a Multi-Round Study Procedure. The four-round -Beauty Contest is transformed from source passages into a reviewed procedure, an explanatory graph, and an executable Python program.

Figure 5. Additional Example Study Programs. Three further procedures: a between-subject framing study, an anchoring study, and the Trust Game. Participant interactions use Ask; condition assignment, iteration, computation, and information routing use Python. The illustrative assign_groups helper applies the assignment configured in Playground, including random allocation when selected; it is not an additional study API operation.
HumanStudyProgram Runtime
The abstractions in Section 4 define a study program and the silicon participants that execute it. We now describe the supporting infrastructure: how a study is packaged for execution, how the runtime carries out its procedure, and how study data and execution traces are recorded for analysis.
Generating an Executable Study Package
Once researchers approve the five-part study description, the tool generates the executable study package shown in Figure 6. Confirmed Materials are stored in materials.json. participants.json retains the reported population, roles, and attributes, together with fields for the silicon-participant configuration used in a run. From the confirmed Procedure, the tool generates procedure.py, which uses the study API for participant interactions and ordinary Python for study-specific logic. The procedure module organizes participant responses and computed outcomes according to the Response specification, and measurement.py implements Analysis over the resulting study data. The worked example in Section 4.2 illustrates this program structure.
Figure 6. Executable Study Package. A confirmed five-part study description becomes the files loaded by the shared runtime. Materials and Participants configure Procedure in procedure.py, Response defines the study data returned by Procedure, and measurement.py implements Analysis.
Before making the package available for execution, the tool analyzes the generated code structure and conducts a safe test run to identify invalid operations and failures in the study logic. Packages that pass these checks are made available to the runtime.
Executing Study Programs and Recording Results
At runtime, the system instantiates the configured participants and executes Procedure in procedure.py. Memory is enabled by default, so content presented by Show and interactions collected by Ask become available to later participant calls. To interview or probe a participant without changing its subsequent context, a study calls Ask(..., remember=False). This operation reads existing memory but does not retain the new question or response. Independent calls can run in parallel, whereas interactive and multi-round procedures wait for the responses needed to continue. For example, each round of the -Beauty Contest in Figure 4 collects participants’ guesses, computes the group result, presents that result, and begins the next round with the updated context.
Execution produces four linked data layers. An Ask call returns parsed responses keyed by participant ID. Procedure combines these responses with conditions, item or round identifiers, and computed outcomes to produce study data conforming to the Response specification. measurement.py analyzes these data to compute study-specific measures and statistical comparisons. In parallel, the runtime records an execution trace containing the participant configuration and, for each call, its assigned condition and role, visible context, raw model output, parsed response, and resulting study state. Identifiers link the analyzed study records to the calls that produced them.
Authoring Tool Implementation
Sections 4 and 5 describe the study abstractions and their runtime. The tool exposes these capabilities through two linked stages (Figure 7): Build Study supports study reconstruction and review, while Playground supports participant configuration, execution, and result analysis.

Figure 7. Detailed Tool Workflow. In Build Study, researchers reconstruct and review a source-linked study description, which the tool turns into an executable HumanStudyProgram. In Playground, researchers configure silicon participants, run the program, and examine its results from study-level findings to individual responses.
Build Study: Reconstructing and Reviewing a Study
In Build Study, a user supplies the available source materials, including a paper, supplementary files, and repository links. An LLM-assisted pipeline identifies the empirical studies described across these sources, reconstructs their Materials, Participants, Procedure, Response, and Analysis, and links each field to its source location.
The interface organizes fields requiring attention into a navigable review list and presents each field alongside its source evidence. It highlights missing, unclear, conflicting, or unsupported information rather than silently completing it. Users can compare a field with its source, edit its value, add missing information, resolve conflicting descriptions, or ask the pipeline to revise a selected part. Saved revisions and review notes remain attached to the study description, and unresolved issues remain visible until the user decides whether it is ready for approval.
The anchoring study of Jacowitz and Kahneman [68] illustrates this review process. In this experiment, participants estimate unknown quantities after being exposed to either a high or low numerical anchor. The reconstruction pipeline recovered the study structure but failed to recover several anchor values from the older PDF. Rather than supplying plausible values, the interface marked the missing information and placed it alongside the source passages that specified the percentile rules (Figure 8). The user could enter the verified values and approve the study description while retaining their source references.
Figure 8. Recovering Missing Study Information. The pipeline recovers the passage defining how the anchors were set and flags the two values it could not read from the scanned table. The researcher locates Table 1 of the source paper and enters them; the package records each value as verbatim from that table.
Playground: Configuring and Executing Silicon Participants
The Playground stage runs an approved study package with silicon participants. Users define reusable participants through a model, participant specification, and memory, following the participant abstraction in Figure 3(a). Participant specifications can also incorporate optional plug-in tools, such as a cognitive bias regulator [69], to configure agent behavior. Users then form population groups from saved participants, choose which specification variables remain fixed or vary within a group, and set the group size (Figure 9).

Figure 9. Silicon Participant Configuration. Users configure reusable silicon participants, form population groups, and assign them to study conditions.
For execution, users select an empirical substudy and assign configured groups to its conditions or roles. They can use explicit assignments or a random-allocation policy and vary exposed trial-level parameters for exploratory runs. Playground records these choices as run configuration rather than modifying the approved study program, making clear which elements were held fixed and which were varied across runs.
The anchoring study also illustrates how the system makes the context of a silicon-participant response visible. A user can add a demographic persona to the run configuration while leaving the approved study content unchanged. Before execution, the interface separates the participant template and persona from the stimulus and response schema supplied by the package (Figure 10). The source view links the study content to its source passage. The same separation is retained with each response, making clear which input belongs to the reconstructed study and which belongs to the participant used for that run.
Figure 10. Tracing Silicon Participant Context. The assembled context distinguishes the participant configuration from the study content and links the latter to its source.
Playground: Exploring Results
Playground presents the study data, measurement outputs, and execution traces produced by the runtime through linked views (Figure 11). Users can move from a study-level finding to its condition-level values, response distributions, and statistical tests, then open a contributing response to inspect its participant configuration, visible context, raw model output, and parsed response.
Figure 11. Multilevel Result Analysis. Three linked views present the results of a run: the study level summarizes findings and effects by condition, the condition level shows response distributions and item-level patterns, and the response level opens an individual record with its raw output, parsed values, and audit log.
The false-consensus study of Ross, Greene, and House[70] provides a concrete walkthrough. Participants place themselves on one side of a choice or personal description and estimate the share of their peers in a given category. The analysis compares that estimate between the participants who placed themselves in the category and those who did not. Figure 12 follows one resulting finding across three views. The study-level view lists the findings and tests computed for the run. Selecting a finding opens the condition-level quantities and statistics behind it. Selecting one of these groups opens a contributing response record, including the participant’s visible context, raw model output, and parsed response. The accompanying measurement view identifies the fields consumed by measurement.py and how they are reduced to the displayed statistic. The views aggregate the same study data at different levels and link back to the corresponding execution trace.
Figure 12. Following a Finding to Its Records. A study-level finding leads to its condition-level values and statistical tests, and then to an individual response record and the measurement steps that consume it.
Expert Review
We conducted a hands-on expert review to understand how HumanStudyProgram could support social science research practice. Each participant used a familiar study from their area of expertise, allowing them to assess its reconstruction and execution against the source materials and their domain knowledge. We observed how they reviewed and corrected the reconstructed study, configured and ran silicon participants, and interpreted the results. We also asked what value and limitations they saw and how they would use the tool in their own research.
Participants
We purposively recruited 12 doctoral-level social science researchers. Participants were pursuing or had completed doctoral training, had more than three years of experience conducting human-subjects research, and represented economics, psychology, political science, and management. All participants had experience with both quantitative and qualitative methods, as well as prior experience with LLMs and LLM-based silicon participants. Each participant received $40 for an approximately 70-minute session.
Study Design and Tasks
Each participant selected a human-subject study from their area of expertise that had not previously been processed by the tool. Some selected their own papers or papers by close collaborators; others selected studies they had examined closely. Their familiarity gave them a concrete basis for evaluating how well the tool represented and executed the study.
Participants used the complete workflow. In Build Study, they compared the reconstructed study with the submitted source materials and their own knowledge. They corrected inaccurate or missing information, addressed issues identified by the tool, and approved the reconstructed study description. The system then generated and validated its executable package. In Playground, they configured silicon participants by selecting a model, defining participant profiles, setting the sample size, and adjusting other run settings. They then ran the study and examined the results at the study, condition, and individual-response levels. Participants could freely edit the reconstructed study, choose different simulation settings, and conduct multiple runs.
Participants submitted their papers and any public supplementary materials before the session. We completed and cached the extraction in advance so that the session focused on hands-on use and evaluation of the tool. We used Gemini 3 Flash for these extractions after formative testing of four models; Appendix B describes this selection. Appendix B.1 reports extraction and review times for the submitted studies, and Appendix B.2 reports the models, number of runs, and configured silicon-participant populations used during the sessions.
Procedure
Sessions were conducted online through a videoconferencing platform. Each session involved one participant, one interviewer, and one note-taker. Participants kept their cameras on and shared their screens while using the deployed web interface on their own computers. Each session consisted of three stages.
Tutorial (approximately 10 minutes). The session began with a brief background discussion and a tutorial using the same cached example. The interviewer introduced the main workflows of HUMANSTUDYPROGRAM and answered questions about interface operation.
Tool Use and Think-Aloud (approximately 40 minutes). Participants used the tool with their selected study. We encouraged them to think aloud by describing what they noticed, expected, and considered while making decisions. Based on these comments, the interviewer asked neutral follow-up questions to clarify the reasons behind participants’ actions and judgments. The interviewer also answered questions about interface operation without influencing how participants used the tool or evaluated its output.
Interview and Survey (approximately 20 minutes). The session concluded with a semi-structured interview about the participant’s experience, possible uses of the tool, and its limitations. Participants then completed the post-study survey. The think-aloud instructions, interview guide, and survey are provided in Appendix B.
Data Collection and Analysis
With consent, we recorded participants’ screens and audio and transcribed their think-aloud comments and interviews. A note-taker documented participants’ actions, questions, and comments. We also collected post-study survey responses.
We summarized survey responses on their original seven-point scales. Two researchers independently analyzed the complete set of think-aloud and interview transcripts from all 12 participants. Each researcher marked relevant passages and identified recurring patterns across participants. They then compared their analyses, discussed differences in interpretation, and jointly developed and agreed on the final themes concerning research value, workflow use, areas for improvement, and boundaries of appropriate use. We used the screen recordings and notes to relate participants’ comments to their actions and considered cases that differed from the main patterns.
Ethical Considerations
The study received institutional ethics approval. Participants provided informed consent and could stop at any time. We collected only demographic information relevant to the analysis. Recordings and transcripts were stored in access-controlled research storage. We refer to participants only by identifiers in the paper.
Results
We refer to participants as P1 through P12. In Build Study, all 12 participants reviewed the reconstruction of their selected study, addressed the issues requiring their judgment, and approved a study description from which the system generated an executable package. Across the 12 reconstructions, the tool surfaced 98 issues for researcher confirmation, with between 4 and 15 issues per study (Appendix § 12.§ 1). Reconstruction itself ran without the researcher and took 10 to 30 minutes per study; reviewing and approving the resulting package took a participant approximately 20 minutes, ranging from 10–15 to 25–30 minutes across the 12 studies. In Playground, all 12 participants configured silicon participants, executed the generated study package, and examined results at the study, condition, and individual-response levels. We organize the findings around potential uses, workflow and usability, and post-study ratings.
Potential Uses
Potential Uses. Participants described three broad ways they would use the tool: understanding published study protocols, prototyping and piloting studies, and applying human study protocols to understand AI-agent behavior. Table 3 summarizes the applications participants described.
| Understanding Protocols | Prototyping and Piloting | Studying AI Agents |
| Reconstructing and rerunning study protocols | Piloting studies before data collection | Understanding AI-agent behavior |
| Helping researchers understand papers | Finding problems before data collection | Comparing AI and human behavior |
| Practicing study-design skills | Testing alternative study designs | |
| Teaching with executable studies | Investigating new research questions | |
| Adapting studies to new contexts |
Table 3. Applications identified by participants across the three potential uses of the tool.
Understanding Published Study Protocols. Participants saw the tool as a way to understand how a published study was actually conducted. The structured study components helped them connect the paper’s description with its stimuli, conditions, measures, and analyses, making the protocol easier to inspect, reproduce, and use for teaching (P3, P6, P8, P11). As P8 explained, “I would really like to use this kind of structured extraction to help me read papers.”
Prototyping and Piloting Studies. All participants wanted to use the tool before collecting human data. They described modifying conditions in existing protocols, testing alternative designs, adapting studies to new questions or contexts, and importing their own study designs to find problems before running a human study. As P3 explained, “I would use it after I have an idea for an experiment, but before I run the human study. It could help me see what is missing and whether the design works as I expected.”
Applying Human Study Protocols to Understand AI-Agent Behavior. Participants also wanted to apply established human study protocols to AI agents. By holding the protocol fixed and varying models, prompts, personas, or languages, they could examine agent preferences and biases and compare AI and human behavior (P2, P4, P7, P8, P11, P12). As P2 explained, “AI agents themselves have now become subjects of social science research.”
Workflow and Usability Findings
Workflow and Usability Findings. We first report participants’ overall experience with the workflow, followed by their experiences with study reconstruction, human review and correction, configuring and executing silicon participants, and interpreting results.
Overall Workflow Experience. Many participants valued having study reconstruction, agent configuration, execution, and result analysis connected in one workflow. This reduced the need to coordinate separate tools and made repeated runs across models and configurations more practical. As P11 explained, “This is more or less automated, the whole workflow. You can do a lot of iterations with different combinations, different iterations, different models.” Similar views were expressed by P1, P4, P5, P8, and P10.
Study Reconstruction. Participants valued how the tool’s structured extraction organized a paper into familiar research components, such as the study overview, stimuli, measures, and analyses. This structure helped them quickly locate the information they cared about and understand how a study could be reproduced (P11). Participants generally perceived the extraction as accurate, particularly in its representation of experimental variables, procedures, and analyses. As P3 noted, “The independent variables, dependent variables, and how the data were analyzed were all extracted very clearly.” Several participants felt that the tool’s structure better reflected what researchers look for when reading a paper than general-purpose AI tools. As P7 explained, “Your toolkit understands what I want better than my AI chatbot. With ChatGPT, I have to explain what I am looking for. Sometimes, even after many rounds of conversation, it still cannot give me the part of the paper I need, and I end up reading the paper again myself.” Others made similar comparisons with Codex and Claude Code (P4, P6). Participants also valued being able to navigate directly to specific study components rather than search through the entire paper (P2, P3, P8, P11).
Human Review and Correction. The tool surfaced missing, inconsistent, and uncertain details and showed which decisions required researcher judgment. P11 described the explanation provided for each issue as “very comprehensive” and helpful for deciding how to address it. In one case, the tool flagged a mismatch between the scale described in a paper’s method section and the scale used in the experiment, then suggested contacting the authors. As P7 explained, “Your toolkit found an error in the paper and suggested contacting the authors. The scale described in the method section did not match the one used in the experiment. When I read the paper carefully, I also felt that something was not quite right.”
Participants also valued that missing-information alerts led directly to the relevant source text and the item requiring confirmation (P3, P6, P7, P11). However, P5 and P9 found the amount of information demanding to review; each spent 20–25 minutes in Build Study and suggested a mind-map view organizing the reconstructed study components and their relationships to help them navigate the information during review.
Configuring and Executing Silicon Participants. Participants tested the tool with familiar papers spanning different languages, stimuli, participant configurations, procedures, and analyses. P2 was surprised that a bilingual study gave each participant the appropriate stimuli based on their language. P4 similarly valued how the tool handled multiple participants in the same scenario and recovered the study’s multi-turn structure without requiring them to reconstruct the sequence of prompts.
Participants valued being able to see what silicon participants actually received. Before starting a run, they could review the system and user prompts, how stimuli were presented, and how multi-turn procedures were translated into agent interactions. This visibility reduced participants’ concerns about relying on AI agents and made them more comfortable using the tool to run experiments. As P1 explained, “I can see exactly what the agent sees. That greatly reduces my distrust of AI agents and makes me feel more comfortable using your toolkit to run an experiment.”
The tool also showed the model, participant profiles, sample size, language, and reasoning settings for each run. Participants said these records helped them understand how a run was produced rather than treating it as a black box. P8 viewed them as research material for comparing runs and understanding why agents behaved differently, while P11 valued using them for systematic comparisons across models and configurations. However, P2 and P3 found the configuration process difficult to navigate and wanted clearer guidance and a simpler interface.
Interpreting Results. Participants moved from the overview to condition-level results and response distributions, then to individual prompt–response records when a result required further explanation. This connected summary findings to the evidence behind them. As P9 explained, “I did not expect that I could click into every level and keep going deeper. Being able to move from the overall result to the details behind it was very helpful for understanding where the result came from.” Similar navigation was observed for P4, P8, and
P11. Participants used visualizations and statistical results to identify patterns, and individual responses and recorded prompts to examine why those patterns emerged (P4, P9, P11).
Post-Study Ratings
All seven item medians were above the scale midpoint and reflected the qualitative findings. The tool’s value for research practice received the strongest ratings (median 6, IQR 6–7), with every participant selecting 6 or 7. Ratings for study match and clarity of the simulation context also had medians of 6; clarity ratings were the most tightly clustered, ranging from 5 to 6. Ease ratings had medians of 5. Reviewing and revising the reconstructed study and study match shared the widest range (4–7), while configuring a run ranged from 4 to 6, consistent with participants’ requests for a clearer overview and configuration guidance. Table 4 reports the complete distribution summaries, and the complete items are provided in Appendix B.
| Item | Median | IQR | Range |
| Ease of using the complete workflow | 5 | 5–6 | 5–7 |
| Match with the selected study | 6 | 6–6 | 4–7 |
| Ease of reviewing and revising the reconstructed study | 5 | 5–6 | 4–7 |
| Clarity of the simulation context | 6 | 6–6 | 5–6 |
| Ease of interpreting simulation results | 5 | 5–6 | 5–7 |
| Ease of configuring a simulation run | 5 | 5–5 | 4–6 |
| Value for research practice | 6 | 6–7 | 6–7 |
Table 4. Post-study survey responses () on the original seven-point scales. Higher values indicate greater ease, match, clarity, or value; 4 is the scale midpoint. IQR reports the first-to-third-quartile interval.
Research Deployment
We deployed HumanStudyProgram as a web application and shared it with the social science research community. Within one week, 23 research teams contacted us about using the tool and potential collaboration. We established ongoing collaborations with four teams pursuing different research goals. Together, they have converted over a hundred papers into executable study protocols. A psychology team converted studies of human ethical judgment into executable protocols to examine how AI agents respond to ethical questions. A management research team converted human studies in international business into executable protocols to examine whether AI agents serve as valid silicon participants in the field. A team studying tsunami evacuation connected empirical evidence about human behavior with agent-based simulations to model individual and collective decisions. An operations research team used the tool to prototype and debug decision-making experiments before data collection. At the time of writing, all four collaborations had continued for more than two months, and each team had begun a new research project supported by HumanStudyProgram.
Discussion
Human Study Protocols as Instruments for Studying AI Agents
Human study protocols bring established behavioral theories into the study of AI agents [45, 46]. Whether theories developed to explain human behavior also explain AI behavior remains an open question [47]. Agreement with a human finding can support further investigation, while differences can reveal where a theory may need revision. Both are informative when agents themselves are the subject of research. HumanStudyProgram could support this work across multiple tasks and settings, providing a basis for developing and testing theories of AI behavior.
Aligning Agent Specifications with Observed Behavior
Behavioral evidence could also help researchers refine agent specifications. A specification describes intended characteristics, but its value depends on whether those characteristics are reflected in behavior. HumanStudyProgram could connect these descriptions to observations across studies, allowing researchers to revise a specification and evaluate the resulting changes. Testing on additional studies would help determine whether an adjustment generalizes beyond the tasks used to make it [58, 71]. Agent specifications could then be supported by evidence of how agents behave and the settings in which that behavior has been evaluated.
Understanding Community Norms in Research Writing
Reconstructing studies could also help us understand how research communities write about their work. Papers communicate more than procedures: they frame questions, justify design choices, and connect evidence to conclusions. Comparing how papers present a study with how its components fit together could reveal conventions in this process, including what authors emphasize and what they expect readers to infer. Extending HumanStudyProgram to examine these relationships across papers could help characterize how such conventions differ across fields. This could support researchers learning to write for a new community and tools that help authors explain the reasoning behind a study. It could also make those conventions easier to question, including whether familiar ways of presenting research leave important decisions or alternative interpretations unexplained.
Toward Executable Research Publications
The same representation could also change how research papers are produced. Today, HumanStudyProgram reconstructs a study from a paper. A future workflow could reverse this process and generate a paper from the study design, execution records, data, and analyses. The system could draft the manuscript using these records and the authors’ account of the research question, while authors check whether its explanations and conclusions are supported by the study. Readers could access both the paper and the executable study, follow results back to the underlying analyses, and run modified versions to explore new questions. The publication could then become a written account of a study that remains available for further work. Automatic paper generation would be part of this broader change, keeping the manuscript and the study it describes connected as both are revised.
Scope and Limitations
HumanStudyProgram currently focuses on text-based social experiments, including single- and multistage procedures, repeated interactions, and role-based multi-participant studies. It does not yet support studies that depend on physical action, physiological sensing, or immersive and other rich-media environments. The quality of a generated study program depends on the information available in the paper and its supplementary materials. Silicon-participant behavior reflects the capabilities and characteristics of the underlying LLM. The resulting runs support protocol inspection, early-stage prototyping, and research on AI behavior, while claims about human populations continue to rely on human-participant evidence.
Conclusion
We presented HumanStudyProgram, a researcher-facing framework that reconstructs a documented human study as an executable program while separating the study procedure from the silicon participants used to run it. We implemented a researcher-facing authoring tool that supports source-linked study reconstruction and review, configurable silicon-participant execution, and navigation from aggregate findings to individual responses and their execution context. Through study demonstrations and a hands-on review with 12 social science researchers, we found value in understanding published protocols, prototyping studies, and examining AI behavior. By making documented study procedures executable and reusable, we hope HumanStudyProgram can make human study methods easier to reuse and build upon.
Build Study Model Selection
During development, we evaluated the ability of Gemini 3 Flash, GPT 5 Nano, Claude Haiku 4.5, and DeepSeek V3.2 to extract and build structured study packages on 10 older papers published between 1946 and 2007 and 20 papers published after 2015. The papers covered psychology, economics, political science, and management. Older papers often contained scanned pages, tables, and text that was difficult to recover with OCR. Newer papers more often distributed study information across longer articles, supplementary files, and external repositories. All four models produced schema-conforming candidate packages and surfaced similar types of omissions and uncertainties. Their main difference was how they described the extracted content and confirmation issues.
We used Gemini 3 Flash in the expert review because its descriptions were clearer and more natural for researchers to review, while its processing cost remained practical. Table 5 summarizes
| Criterion | Observation |
| Pipeline completion | All four models generated structured study packages for researcher review. |
| Omissions and uncertainties | The four models surfaced broadly similar types of issues. |
| Presentation | The models differed mainly in how they described reconstructed content and confirmation issues. Gemini 3 Flash produced the clearest and most natural descriptions. |
| Operational cost | Gemini 3 Flash provided a practical processing cost for complete study materials. |
| Final selection | We used Gemini 3 Flash for Build Study in the expert review. |
Table 5. Model selection criteria and observations for the Build Study pipeline.
this formative comparison. Since the four models produced similar reconstruction outcomes, we compare their use in the pipeline rather than rank their accuracy.
Researcher Study Materials
Selected Studies and Submitted Materials
The selected studies covered economics (6), psychology (2), political science (2), and management (2). Five were submitted as PDFs and seven as ZIP archives containing supplementary materials. A confirmation issue is an independently recorded item that required the researcher to verify information, supply missing information, resolve ambiguity or conflict, or accept a documented departure from the source study. Extraction time is the duration of the automated reconstruction, which we completed and cached before the session and which required no researcher involvement. Review time is the part of the hands-on block a participant spent in Build Study, from opening the reconstructed study to approving the study description. We estimated it from the screen recordings and report it in five-minute bands; it excludes the tutorial, the Playground configuration and runs, and the closing interview. Estimated JSON tokens characterize the amount of structured study information in the metadata, materials, and task specification. We divide the character count of normalized JSON by four and round to the nearest hundred tokens.
Expert Review Run Configurations
Table 7 summarizes the simulation runs completed during the hands-on sessions. DeepSeek V3.2 was the default model presented to experts when configuring silicon participants. They could retain it or select another available model, including Gemini 3 Flash, Claude Haiku 4.5, and GPT 5 Nano, and could conduct additional runs as time allowed. Population reports the total number of silicon participants configured for each run.
Table 7 | Models, runs, and configured silicon-participant populations in the expert review ().
| Participant | Model used | Runs | Population per run |
| P1 | DeepSeek V3.2 | 1 | 20 |
| P2 | DeepSeek V3.2 | 1 | 30 |
| P3 | DeepSeek V3.2; Gemini 3 Flash | 2 | 50 |
| P4 | DeepSeek V3.2; Claude Haiku 4.5 | 2 | 12 |
| P5 | DeepSeek V3.2 | 1 | 50 |
| P6 | DeepSeek V3.2; Gemini 3 Flash; GPT 5 Nano | 3 | 100 |
| P7 | DeepSeek V3.2 | 1 | 40 |
| P8 | DeepSeek V3.2; Gemini 3 Flash; Claude Haiku 4.5 | 3 | 50 |
| P9 | DeepSeek V3.2; Gemini 3 Flash | 2 | 20 |
| P10 | DeepSeek V3.2 | 1 | 30 |
| P11 | DeepSeek V3.2 | 1 | 30 |
| P12 | DeepSeek V3.2 | 1 | 40 |
Table 7.
Think-Aloud Instructions
Before participants began using the tool with their selected study, the interviewer provided the following instructions:
For the next part of the session, please use the tool with your selected study. You may work through the tool in whatever way makes sense to you. As you work, please say aloud what you are looking at, what you are trying to do, what you expect to happen, and what you are considering when making a decision. Please describe your reactions as they occur.
There are no right or wrong actions or answers. We are studying the tool and are not evaluating your performance. You may decide what to review, change, configure, or run. We will not direct those choices. You may ask questions about how the interface works at any time. We can explain its operation. Decisions about how to use the tool remain yours.
Table 6 | Participant-selected studies, submitted materials, approximate extraction and review times, and issues surfaced for researcher confirmation ().
| Study | Year | Domain | Materials | Extraction time | Review time | Estimated JSON tokens | Confirmation issues |
| P1 | 2016 | Management | 10–15 min | 20–25 min | 6,200 | 10 | |
| P2 | 2013 | Psychology | 10–15 min | 20–25 min | 8,100 | 11 | |
| P3 | 2024 | Psychology | ZIP | 25–30 min | 15–20 min | 4,000 | 7 |
| P4 | 2016 | Economics | ZIP | 25–30 min | 15–20 min | 5,300 | 5 |
| P5 | 2024 | Economics | ZIP | 25–30 min | 20–25 min | 6,200 | 11 |
| P6 | 2024 | Political science | 10–15 min | 25–30 min | 8,500 | 15 | |
| P7 | 2026 | Political science | ZIP | 25–30 min | 15–20 min | 5,200 | 7 |
| P8 | 2021 | Management | ZIP | 25–30 min | 15–20 min | 1,500 | 7 |
| P9 | 2023 | Economics | ZIP | 25–30 min | 20–25 min | 9,200 | 6 |
| P10 | 2011 | Economics | ZIP | 25–30 min | 10–15 min | 1,500 | 4 |
| P11 | 1995 | Economics | 10–15 min | 15–20 min | 6,700 | 6 | |
| P12 | 2002 | Economics | 10–15 min | 20–25 min | 6,400 | 9 |
Table 6.
If you become quiet, I may remind you to keep talking or ask what you are thinking. Do you have any questions about this process before we begin?
During the session, the interviewer used only neutral prompts when clarification was needed, including “Please keep talking,” “What are you thinking now?,” “What are you trying to do?,” “What are you noticing?,” “What did you expect to happen?,” and “What led you to that choice?”
Post-Study Survey
Participants answered seven questions. Each used a seven-point response scale with labeled endpoints. Responses were coded from 1, representing the first endpoint listed below, to 7, representing the second endpoint.
How easy or difficult was it to use the complete workflow? Very difficult to very easy.
To what extent did the reconstructed study match your understanding of the selected study? Not at all to completely.
How easy or difficult was the process of reviewing and revising the reconstructed study? Very difficult to very easy.
How clear or unclear was the information about what silicon participants received? Very unclear to very clear.
How easy or difficult was it to interpret the simulation results? Very difficult to very easy.
How easy or difficult was it to configure a simulation run? Very difficult to very easy.
Based on your experience, what value, if any, does the tool have for your research practice? No value at all to substantial value.
Semi-Structured Interview Guide
The interviewer used the following questions. The order could follow the participant’s actions. Neutral prompts such as “Can you give an example?” and “What led you to that judgment?” were used when clarification was needed.
Table 8 | Individual post-study survey responses. Q1–Q7 correspond to the seven survey questions listed in Appendix B.4.
| Participant | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 |
| P1 | 6 | 7 | 5 | 6 | 5 | 5 | 7 |
| P2 | 5 | 6 | 5 | 6 | 5 | 4 | 6 |
| P3 | 5 | 6 | 6 | 6 | 5 | 4 | 7 |
| P4 | 6 | 6 | 6 | 5 | 6 | 5 | 7 |
| P5 | 5 | 6 | 4 | 6 | 5 | 5 | 6 |
| P6 | 5 | 6 | 5 | 6 | 5 | 5 | 6 |
| P7 | 5 | 6 | 6 | 6 | 5 | 5 | 6 |
| P8 | 5 | 6 | 5 | 6 | 5 | 5 | 7 |
| P9 | 5 | 6 | 4 | 6 | 6 | 5 | 6 |
| P10 | 6 | 4 | 5 | 6 | 5 | 5 | 6 |
| P11 | 7 | 6 | 7 | 5 | 7 | 5 | 7 |
| P12 | 6 | 6 | 6 | 6 | 6 | 6 | 6 |
Table 8.
Please walk me through how you approached the task.
What did you pay attention to during the workflow? Why?
What, if anything, was easy, difficult, or unclear?
How did you assess the reconstructed study?
What did you change or leave unchanged? What informed those decisions?
How did your knowledge of the selected study affect your review?
How did you choose the run configurations? What did you do when viewing the results?
How did you interpret the results? What evidence informed that interpretation?
What uses, if any, could these artifacts and results support in your work?
What conditions or concerns, if any, would affect whether you used these artifacts or results?
What changes, if any, would improve the tool for your work?
Is there anything else about your experience that you want to discuss?
References
- [1]Association for Computing Machinery. Artifact review and badging. https://www.acm.org/publications/policies/artifact-review-and-badging-current, 2020. Accessed 2026-08-28.
- [2]ACM CHI 2026. Guide to a successful submission, 2026. URL https://chi2026.acm.org/guide-to-a-successful-submission/.
- [3]Tatiana Chakravorti, Sanjana Gautam, and Sarah M. Rajtmajer. Embracing transparency: A study of open science practices among early career hci researchers, 2025. URL https://arxiv.org/abs/2410.04286.
- [4]Kavous Salehzadeh Niksirat, Lahari Goswami, Pooja S. B. Rao, James Tyler, Alessandro Silacci, Sadiq Aliyu, Annika Aebli, Chat Wacharanamotham, and Mauro Cherubini. Changes in research ethics, openness, and transparency in empirical studies between CHI 2017 and CHI 2022. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–23. Association for Computing Machinery, 2023. doi: 10.1145/3544548.3580848. URL https://doi.org/10.1145/3544548.3580848.
- [5]Olga Yargina, Kasper Hornbæk, and Aske Mottelson. On the computational reproducibility of human-computer interaction. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, 2026. doi: 10.1145/3772318.3791129.
- [6]Jean Vanderdonckt and Radu-Daniel Vatavu. Context is key for reproducibility of empirical studies in human-computer interaction. In Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, pages 41–50. Association for Computing Machinery, 2025. doi: 10.1145/3736731.3746142. URL https://doi.org/10.1145/3736731.3746142.
- [7]David Donoho. Data science at the singularity. arXiv preprint arXiv:2310.00865, 2023.
- [8]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony M. DePierrec, Tsim Pault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6/.DOI
- [9]Quentin Lhoest, Alistair Vilanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani, Bhavya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delauney, Théo Moussatière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Heike Adel and Shuming Shi, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-demo.21. URL https://aclanthology.org/2021.emnlp-demo.21/.DOI
- [10]Jacy Reese Anthis, Ryan Liu, Sean M Richardson, Austin C. Kozlowski, Bernard Koch, Erik Brynjolfsson, James Evans, and Michael S. Bernstein. Position: LLM social simulations are a promising research method. In Forty-second International Conference on Machine Learning Position Paper Track, 2025. URL https://openreview.net/forum?id=cRBg1dtj7o.arxiv.org/abs/2504.02234
- [11]William Agnew, A. Stevie Bergman, Jennifer Chien, Mark Diab, Selim El-Sayed, Jaylen Pittman, Shakir Mohamed, and Kevin R. McKee. The illusion of artificial inclusion. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703300. doi: 10.1145/3613904.3642703. URL https://doi.org/10.1145/3613904.3642703.
- [12]Shivani Kapania, William Agnew, Motharhe Eslami, Hoda Heidari, and Sarah E Fox. Simulacrum of stories: Examining large language models as qualitative research participants. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400713941. doi: 10.1145/3706598.3713220. URL https://doi.org/10.1145/3706598.3713220.
- [13]Zichao Wang and Alexa Siu. Interview-informed generative agents for product discovery: A validation study. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, pages 1–27. Association for Computing Machinery, 2026. doi: 10.1145/3772318.3791918. URL https://doi.org/10.1145/3772318.3791918.
- [14]Xinyue Gui, Ding Xia, Mark Colley, Yuan Li, Vishal Chauhan, Anubhav Anubhav, Zhongyi Zhou, Ehsan Jayawardi, Stella Hanbyeol Seo, Chia-Ming Chang, Manabu Tsukada, and Takeo Igarashi. Peeking ahead of the field study: Exploring VLM personas as support tools for embodied studies in HCI. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, pages 1–26. Association for Computing Machinery, 2026. doi: 10.1145/3772318.3790537. URL https://doi.org/10.1145/3772318.3790537.
- [15]Max L. Wilson, Wendy Mackay, Ed Chi, Michael Bernstein, Dan Russell, and Harold Thimbleby. Replici - chi should be replicating and validating results more: discuss. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’11, page 463–466, New York, NY, USA, 2011. Association for Computing Machinery. ISBN 9781450302685. doi: 10.1145/1979742.1979491. URL https://doi.org/10.1145/1979742.1979491.
- [16]Richard A. Klein, Kate A. Ratliff, Michelangelo Vianello, Reginald B. Adams, Stépan Bahník, Michael J. Bernstein, Konrad Bocian, Mark J. Brandt, Beach Brooks, Claudia Chloe Brumbaugh, Zeynep Cemalcilar, Jesse Chandler, Winnee Cheong, William E. Davis, Thierry Devos, Matthew Eisner, Natalia Frankowska, David Furrow, Elisa Maria Galliani, Fred Hasselman, Joshua A. Hicks, James F. Hovermale, S. Jane Hunt, Jeffrey R. Huntsinger, Hans Jetzmer, Melissa-Sue John, Jennifer A. Joy-Gaba, Heather Barry Kappes, Lacy E. Krueger, Jaime Kurtz, Carmel A. Levitan, Robyn K. Mallett, Wendy L. Morris, Anthony J. Nelson, Jason A. Nier, Grant Packard, Ronaldo Pilati, Abraham M. Rutchick, Kathleen Schmidt, Jeanine L. Skorinko, Robert Smith, Troy G. Steiner, Justin Storbeck, Lyn M. van Swol, Donna Thompson, A. E. Van ’t Veer, Leigh Ann Vaughn, Marek Vranka, Aaron L. Wichman, Julie A. Woodzicka, and Brian A. Nosek. Investigating variation in replicability: A “Many Labs” replication project. Social Psychology, 45(3):142–152, 2014. doi: 10.1027/1864-9335/a000178.DOI
- [17]Open Science Collaboration. Estimating the reproducibility of psychological science. Science, 349(6251):aac4716, 2015.DOI
- [18]Colin F Camerer, Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A Nosek, Thomas Pfeiffer, et al. Evaluating the replicability of social science experiments in nature and science between 2010 and 2015. Nature human behaviour, 2(9):637–644, 2018.
- [19]Erin D. Foster and Ariel Deardorff. Open science framework (OSF). Journal of the Medical Library Association, 105(2):203–206, 2017. doi: 10.5195/jmla.2017.88.DOI
- [20]Andy Cockburn, Carl Gutwin, and Alan Dix. HARK no more: On the preregistration of CHI experiments. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–12, 2018. doi: 10.1145/3173574.3173715.DOI
- [21]Chat Wacharamanotham, Lukas Eisenring, Steve Haroz, and Florian Echtler. Transparency of CHI research artifacts: Results of a self-reported survey. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–14, 2020. doi: 10.1145/3313831.3376448.
- [22]Jeffrey Heer, Matt Conlen, Vishal Devireddy, Tu Nguyen, and Josh Horowitz. Living papers: A language toolkit for augmented scholarly communication. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023. doi: 10.1145/3586183.3606791.DOI
- [23]Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. Transactions on Machine Learning Research, 2025. URL https://openreview.net/forum?id=BsMMc4MEGS.arxiv.org/abs/2409.11363
- [24]Minju Seo, Jinheong Baek, Seongyun Lee, and Sung Ju Hwang. PaCode: Automating code generation from scientific papers in machine learning. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/41b6674c28a9b93ec8d22a53ca25bc3b-Abstract-Conference.html.
- [25]Bang Nguyen, Dominik Soós, Qian Ma, Rochana R. Obadage, Zack Ranjan, Sai Koneru, Timothy M. Errington, Shakhno Nematova, Sarah Rajtmajer, Jian Wu, and Meng Jiang. ReplicatorBench: Benchmarking LLM agents for replicability in social and behavioral sciences. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, AI4Sciences Track, 2026. URL https: //arxiv.org/abs/2602.11354.arxiv.org/abs/2602.11354
- [26]Gyeongwon J Kim, Alex Wilf, Louis-Philippe Morency, and Daniel Fried. From reproduction to replication: Evaluating research agents with progressive code masking. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=gbCHWGBnIb.arxiv.org/abs/2506.19724
- [27]Chuxuan Hu, Liyun Zhang, Yeji Lim, Aum Wadhwani, Austin Peters, and Daniel Kang. REPRO-bench: Can agentic AI systems assess the reproducibility of social science research? In Findings of the Association for Computational Linguistics: ACL 2025, pages 23616–23626. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-acl.1210. URL https://aclanthology.org/2025.findings-acl.1210/.DOI
- [28]Sebastian Pagel, Michael Jirasek, and Leroy Cronin. Verification and execution of the scientific literature via computation augmented by large language models. Communications Chemistry, 9:191, 2026. doi: 10.1038/s42004-026-01993-w. URL https://doi.org/10.1038/s42004-026-01993-w.
- [29]Jonathan Peirce, Jeremy R. Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Erik Kastman, and Jonas Kristofer Lindeløv. Psychopy2: Experiments in behavior made easy. Behavior Research Methods, 51:195–203, 2019. doi: 10.3758/s13428-018-01193-y.
- [30]Joshua R. de Leeuw. jsPsych: A JavaScript library for creating behavioral experiments in a web browser. Behavior Research Methods, 47:1–12, 2015. doi: 10.3758/s13428-014-0458-y.
- [31]Felix Henninger, Yuri Shevchenko, Ulf K. Mertens, Pascal J. Kieslich, and Benjamin E. Hilbig. lab.js: A free, open, online study builder. Behavior Research Methods, 54:556–573, 2022. doi: 10.3758/s13428-019-01283-5.DOI
- [32]Vanessa V. Sochat, Ian W. Eisenberg, A. Z. Enkavi, Jamie Li, Patrick G. Bissett, and Russell A. Poldrack. The experiment factory: Standardizing behavioral experiments. Frontiers in Psychology, 7:610, 2016. doi: 10.3389/fpsyg.2016.00610.DOI
- [33]Urs Fischbacher. z-Tree: Zurich toolbox for ready-made economic experiments. Experimental Economics, 10(2):171–178, 2007. doi: 10.1007/s10683-006-9159-4.DOI
- [34]Daniel L. Chao, Martin Schonger, and Chris Wickens. oTree—an open-source platform for laboratory, online, and field experiments. Journal of Behavioral and Experimental Finance, 9:88–97, 2016. doi: 10.1016/j.jbef.2015.12.001.
- [35]Abdullah Almaatouq, Joshua Becker, James P. Houghton, Nicolas Paton, Duncan J. Watts, and Mark E. Whiting. Empirica: A virtual lab for high-throughput macro-level experiments. Behavior Research Methods, 53(5):2158–2171, 2021. doi: 10.3758/s13428-020-01535-9.DOI
- [36]Marcus Giamattei, Kyanoush Seyed Yahosseini, Simon Gächter, and Lucas Molleman. LIONESS Lab: A free web-based platform for conducting interactive experiments online. Journal of the Economic Science Association, 6(1):95–111, 2020. doi: 10.1007/s40881-020-00087-0.DOI
- [37]Ashwini Ashokkumar, Luke Hewitt, Isaias Ghezae, and Robb Willer. Large language models can predict the results of social science experiments. Nature, 656:115–122, 2026. doi: 10.1038/s41586-026-10742-x.DOI
- [38]Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. Evaluating large language models in generating synthetic HCI research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2023. doi: 10.1145/3544548.3580688. URL https://doi.org/10.1145/3544548.3580688.
- [39]Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351, 2023.DOI
- [40]John J Horton. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research, 2023.
- [41]Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International conference on machine learning, pages 337–371. PMLR, 2023.arxiv.org/abs/2208.10264
- [42]Tiancheng Hu, Joachim Baumann, Lorenzo Lupo, Nigel Collier, Dirk Hovy, and Paul Röttger. Simbench: Benchmarking the ability of large language models to simulate human behaviors. In ACL 2025 Student Research Workshop, 2025. URL https://openreview.net/forum?id=4E3inkANXm.
- [43]Yueqi Xie, Lemeng Liang, Shuzhen Li, Yifu Lu, Zhiwen Xiao, Mengdi Shi, Junming Huang, Mengdi Wang, and Yu Xie. Evaluating the statistical realism of LLM-generated social science data. Proceedings of the National Academy of Sciences, 123(19):e2538145123, 2026. doi: 10.1073/pnas.2538145123.DOI
- [44]James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. Synthetic replacements for human survey data? the perils of large language models. Political Analysis, 32(4):401–416, 2024. doi: 10.1017/pan.2024.5.DOI
- [45]Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-François Bonnefon, Cynthia Breazeal, Jacob W. Crandall, Nicholas A. Christakis, Iain D. Couzin, Matthew O. Jackson, Nicholas R. Jennings, Ece Kamar, Isabel M. Kloumann, Hugo Larochelle, David Lazer, Richard McElreath, Alan Mislove, David C. Parkes, Alex Pentland, Margaret E. Roberts, Azim Shariff, Joshua B. Tenenbaum, and Michael Wellman. Machine behaviour. Nature, 568:477–486, 2019. doi: 10.1038/s41586-019-1138-y.DOI
- [46]Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120, 2023.DOI
- [47]Nicolas Yax, Hernán Anlló, and Stefano Palminteri. Studying and improving reasoning in humans and machines. Communications Psychology, 2, 2024. doi: 10.1038/s44271-024-00091-8. URL https://doi.org/10.1038/s44271-024-00091-8.
- [48]James W. A. Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, Michael S. A. Graziano, and Cristina Becchio. Testing theory of mind in large language models and humans. Nature Human Behaviour, 8:1285–1295, 2024. doi: 10.1038/s41562-024-01882-z. URL https://doi.org/10.1038/s41562-024-01882-z.
- [49]Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. Nature Human Behaviour, 9:1380–1390, 2025. doi: 10.1038/s41562-025-02172-y. URL https://doi.org/10.1038/s41562-025-02172-y.
- [50]Jessica Hullman, David Broska, Huaman Sun, and Aaron Shaw. This human study did not involve human subjects: Validating llm simulations as behavioral evidence. arXiv preprint arXiv:2602.15785, 2026.arxiv.org/abs/2602.15785
- [51]Angelina Wang, Jamie Morgenstern, and John P. Dickerson. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7:400–411, feb 2025. doi: 10.1038/s42256-025-00986-z. URL https://doi.org/10.1038/s42256-025-00986-z.
- [52]Joshua Ashkinaze, Emily Fry, Narendra Edara, Eric Gilbert, and Ceren Budak. Plurals: A system for guiding LLMs via simulated social ensembles. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2025. doi: 10.1145/3706598.3713675. URL https://doi.org/10.1145/3706598.3713675.
- [53]Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023. URL https://arxiv.org/abs/2304.03442.
- [54]Alexander Sasha Vezhnevets, John P. Agapiou, Avia Aharon, Ron Ziv, Jayd Matyas, Edgar A. Dueñez-Guzmán, William A. Cunningham, Simon Osindero, Danny Karmon, and Joel Z. Leibo. Generative agent-based modeling with actions grounded in physical, social, or digital space using concordia, 2023. URL https://arxiv.org/abs/2312.03664.
- [55]Jinghua Piao, Yuwei Yan, Jun Zhang, Nian Li, Junbo Yan, Xiaochong Lan, Zhihong Lu, Zhiwei Zheng, Jing Yi Wang, Di Zhou, Chen Gao, Fengli Xu, Fang Zhang, Ke Rong, Jun Su, and Yong Li. Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding of human behaviors and society, 2025. URL https://arxiv.org/abs/2502.08691.
- [56]Joon Sung Park, Carolyn Q. Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein. Generative agent simulations of 1,000 people, 2024. URL https://arxiv.org/abs/2411.10109.
- [57]Marlene Lutz, Indira Sen, Georg Ahnert, Elisa Rogers, and Markus Strohmaier. The prompt makes the person(a): A systematic evaluation of sociodemographic persona prompting for large language models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, pages 23212–23237, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.1261. URL https://aclanthology.org/2025.findings-emnlp.1261/.DOI
- [58]Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria K. Eckstein, Noëmi Elétto, Thomas L. Griffiths, Susanne Haridi, Akshay K. Jagadish, Li Ji-An, Alexander Kipnis, Sreejan Kumar, Tobias Ludwig, Marvin Mathony, Marcelo Mattar, Alireza Modirshanechi, Surabhi S. Nath, Joshua C. Peterson, Milena Rmus, Evan M. Russek, Tankred Saanum, Johannes A. Schubert, Luca M. Schulze Buschoff, Nishad Singhi, Xin Sui, Mirko Thalmann, Fabian J. Theis, Vuong Truong, Vishaal Udandarao, Konstantinos Voudouris, Robert Wilson, Kristin Witte, Shuchen Wu, Dirk U. Wulff, Huadong Xiong, and Eric Schulz. A foundation model to predict and capture human cognition. Nature, 644:1002–1009, July 2025. doi: 10.1038/s41586-025-09215-4. URL https://doi.org/10.1038/s41586-025-09215-4.
- [59]Akaash Kolluri, Shengguang Wu, Joon Sung Park, and Michael S. Bernstein. Finetuning llms for human behavior prediction in social science experiments, 2025. URL https://arxiv.org/abs/2509.05830.
- [60]Tiancheng Hu and Nigel Collier. Quantifying the persona effect in LLM simulations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10289–10307. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.554. URL https://aclanthology.org/2024.acl-long.554/.DOI
- [61]Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Andreas Krause, Emma Brunskill, Kyungyung Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 29971–30004. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/santurkar23a.html.
- [62]Lindia Tjague, Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. Do llms exhibit human-like response biases? a case study in survey design, 2024. URL https://arxiv.org/abs/2311.04076.
- [63]Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. Questioning the survey responses of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=0o7dlLgqQX.
- [64]Stephen Pilli and Vivek Nallur. Emulating aggregate human choice and biases with GPT conversational agents. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, pages 377–397. Association for Computing Machinery, 2026. doi: 10.1145/3772318.3791835. URL https://doi.org/10.1145/3772318.3791835.
- [65]Amos Tversky and Daniel Kahneman. The framing of decisions and the psychology of choice. science, 211(4481):453–458, 1981.
- [66]Rosemarie Nagel. Unraveling in guessing games: An experimental study. The American economic review, 85(5):1313–1326, 1995.
- [67]Joyce E. Berg, John Dickhaut, and Kevin McCabe. Trust, reciprocity, and social history. Games and Economic Behavior, 10:122–142, 1995. URL https://api.semanticscholar.org/CorpusID:144827131.DOI
- [68]Karen E. Jacowitz and Daniel Kahneman. Measures of anchoring in estimation tasks. Personality and Social Psychology Bulletin, 21:1161–1166, 1995. URL https://api.semanticscholar.org/CorpusID:145430539.DOI
- [69]Xuan Liu, HaoYang Shang, and Haojian Jin. CoBRA: Programming cognitive bias in social agents using classic social science experiments. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI ’26, New York, NY, USA, 2026. Association for Computing Machinery. doi: 10.1145/3772318.3790804. URL https://doi.org/10.1145/3772318.3790804.
- [70]Lee D. Ross, David Greene, and Pamela House. The “false consensus effect”: An egocentric bias in social perception and attribution processes. Journal of Experimental Social Psychology, 13:279–301, 1977. URL https://api.semanticscholar.org/CorpusID:90327175.
- [71]Tal Yarkoni. The generalizability crisis. Behavioral and Brain Sciences, 45:e1, 2022. doi: 10.1017/S0140525X200001685.







