betaFeedback

WorksComputer Science

The Butterfly Effect in LLM Social Simulations: From Architecture Sensitivity to Robustness Audits

Abstract

A butterfly effect occurs when a local perturbation propagates through a system and produces divergence in its collective outcome. In LLM social simulations, this creates a particular challenge for scientific inference: sensitivity to implementation choices can enter agent interactions and alter the collective outcomes on which scientific claims are based. We study this problem through a repeated Prisoner's Dilemma and a social-media echo chamber, spanning representation- and design-level perturbations across four LLM families. We find that controlled perturbations can substantially alter collective outcomes, with sensitivity varying across both perturbations and models. In the Prisoner's Dilemma, changing only the representation of a matched persona shifts two-agent cooperation by 75.2 percentage points for GPT-5.2, while game framing and memory representation produce smaller or less consistent effects. Multi-agent interaction further reshapes representation sensitivity, producing amplification or attenuation depending on the model and perturbation. In the echo-chamber simulation, persona representation changes interaction targeting and is accompanied by divergence in collective interaction structure, extending representation sensitivity from agent behavior to network-level outcomes. These results show that robustness cannot be inferred from a single simulation configuration and that local sensitivity can change as it propagates through interaction. We introduce TRAILS (Taxonomy for Robustness Audits In LLM Simulations) and the Social Simulation Audit Card as a framework for systematically auditing these dependencies and calibrating the scope of scientific claims to the robustness evidence that supports them.

Introduction

Until recently, simulating collective human behavior in agent-based models required encoding every social tendency by hand: how an agent decides to cooperate, who it talks to, what it remembers, and how its preferences change. Large language models (LLM) have lifted this constraint. By generating agent behavior directly from natural language, they have opened a new design space for studying a wide range of social phenomena Park et al., [27]. The field has advanced rapidly, and these simulations have been studied in a wide range of contexts. Researchers now use LLM-based social simulations to explore and explain social mechanisms such as cooperation Akata et al., [2], opinion dynamics Chuang et al., [7] and social norms emergence Ashery et al., [4]; to serve as intervention testbeds for content moderation Liu et al., [19] and platform design Puelma Touzel et al., [32]; and to inform real-world decisions in policy, public health, and epidemiology Chopra et al., [6]; PHF Science [29]. Simulations are moving from exploring what might happen to claiming why it happens, when it will happen, and what should happen next.

As the stakes have risen, so has the field’s attention to validation Anthis et al., [3]; Larooij & Törnberg [16]; Puelma Touzel et al. [33]. However, existing efforts focus mostly on whether a simulation resembles the world: whether agents behave plausibly Park et al., [27], whether outcomes match empirical patterns Larooij & Törnberg, [16]; Yang et al. [53]; Gu et al. [12], whether interaction is grounded in established theory [24, 20], or whether human evaluators find the results convincing [33]. Realism and empirical grounding address essential dimensions of validity, but they do not determine whether a simulation is robust: its conclusions should remain stable across reasonable alternative configurations, just as empirical findings are expected to survive alternative controls, operationalizations, and sensitivity analyses [43, 44]. Without robustness testing, we cannot tell whether a finding reflects the social process being modeled or the design choices used to model it.

Robustness matters because LLM-based agents introduce the complexity that traditional rule-based agent-based modeling does not. A simulation can depend on choices about model substrate, persona representation, memory, network initialization, and interaction protocols [56]. Choices that appear equivalent to researchers may elicit different decisions from an LLM [38, 42]. A small representational change can alter an agent’s decision. In a multi-agent simulation, such decisions can in turn alter subsequent interactions, memories, or social signals through repeated interaction, so the local differences can compound into substantially different macro-level outcomes [5]. We use butterfly effect [9] to describe this propagation of behavioral sensitivity through interaction. The process can amplify or attenuate initial differences, with effects appearing in both behavioral and collective measures.

To study this phenomenon, we conducted two controlled simulation regimes across four LLM families (Figure 1): a repeated Prisoner’s Dilemma comparing fixed-opponent and reciprocal LLM–LLM interaction, and a social-media echo-chamber simulation tracing sensitivity from agent-level choices to collective interaction structure. We varied representation and design choices while holding other components fixed. We found that sensitivity differed across perturbations and models: persona representation substantially shifted cooperation for several models, while other perturbations had smaller or model-dependent effects. In addition, multi-agent interaction further reshapes this sensitivity: reciprocal LLM–LLM interaction amplifies the same persona perturbation for three model families but attenuates it for a fourth, while networked interaction can carry representation sensitivity from interaction targeting to collective network structure. These results show that claims from LLM-based social simulations should match their robustness evidence. We introduce TRAILS (Taxonomy for Robustness Audits in LLM Simulations) and the Social Simulation Audit Card for the robustness audits for LLM social simulation.

Figure 1. The butterfly effect in LLM social simulations. Two representations of the same strategic persona produce a 75.2-percentage-point difference in cooperation in an otherwise matched 10-round Prisoner’s Dilemma (gpt-5.2, N=30N = 30 paired seeds, Holm-corrected p<.001p < .001). We study how such local sensitivity changes across perturbations, models, and multi-agent interaction, and use TRAILS to connect robustness audits to the scope of scientific claims.

Related Work

LLM social simulation and the validation gap. Agent-based modeling (ABM) provides a bottom-up framework for studying how collective patterns emerge from individual agents and local interactions ([41, 22]), while longstanding work highlights challenges in empirical validation, replication, and sensitivity ([51, 26, 50, 5]). LLM-based agents extend this paradigm by generating decisions, communication, and adaptation through natural language ([27]), and have been used to study cooperation and negotiation ([2, 11, 1]), economic and population dynamics ([18, 23, 52]), coordinated information operations ([25]), and open-ended social phenomena ([12, 46, 37]). Existing validation efforts examine behavioral plausibility, human evaluation, empirical correspondence, real-world grounding, and theory-guided interaction mechanisms ([27, 33, 12, 24, 20]). These approaches assess whether simulations credibly represent their targets, while robustness asks whether their scientific conclusions remain stable across reasonable alternative implementations. Although individual studies include sensitivity checks, systematic analysis across representation, interaction, and design choices remains less developed. Following sensitivity analysis in ABMs ([39, 51, 5]) and robustness analysis in empirical research ([44, 43]), we examine whether conclusions from LLM social simulations persist under controlled changes to simulation design and representation.

From local LLM sensitivity to multi-agent propagation. Small changes in LLM inputs can affect individual decisions and potentially propagate through multi-agent interaction. Prior work documents sensitivity to prompt formatting, option ordering, instruction structure, and other surface features ([38, 42, 28, 35] (Ju et al. 2024)), as well as persona construction ([17]), strategic framing, and interaction-history presentation ([2, 21, 11]). Classical ABM research similarly shows that collective outcomes can depend on initial conditions and modeling assumptions ([5, 51, 50]). In a multi-agent simulation, an LLM output becomes an action, message, memory, or social signal that can enter subsequent interactions. Prior work therefore establishes sensitivity at both the model and system levels, but leaves open how the magnitude of a controlled representation perturbation changes as LLM agents interact. We examine this gap by measuring amplification or attenuation under reciprocal LLM–LLM interaction and by tracing representation sensitivity across behavioral and collective outcomes in networked interaction.

Controlled testbeds for collective sensitivity. We study this problem in two established but complementary simulation settings. Repeated Prisoner’s Dilemma is a common testbed for strategic behavior and cooperation in LLM agents ([2, 11]). Its fixed action space, payoff structure, and interaction horizon provide a controlled setting for comparing the same representation perturbation across fixed-opponent and reciprocal LLM–LLM interaction. Echo chambers provide a complementary open-ended setting in which collective structure emerges from repeated local engagement on a social network. Prior LLM-agent simulations study the roles of network structure, homophily, recommendation and exposure, memory, and interaction dynamics in polarization and echo-chamber formation ([12, 46, 10, 30]). This setting allows a representation perturbation to be examined first in agents’ interaction choices and then in the network formed by those choices. The experiments isolate selected perturbations while holding non-targeted components fixed, allowing sensitivity and its development through interaction to be measured directly. Appendix § 9 provides the broader scope of LLM social simulation and the domain-specific motivation for the perturbations evaluated in each testbed.

Method

Conceptual Framework

Let θ\theta denote a social-simulation configuration and ξ\xi a stochastic realization. A controlled perturbation pp changes one component of θ\theta while holding the remaining components fixed, yielding θ(p)\theta^{(p)}. For an outcome YY, we define the signed perturbation effect and its sensitivity magnitude as

ΔY(p)=Eξ[Y(θ(p),ξ)−Y(θ,ξ)],SY(p)=∣ΔY(p)∣.(1)\Delta_Y(p) = \mathbb{E}_{\xi}\left[Y\left(\theta^{(p)},\xi\right) - Y(\theta,\xi)\right], \qquad S_Y(p) = \left|\Delta_Y(p)\right|. \tag*{(1)}

Sensitivity measures the change associated with a controlled perturbation.

A perturbation can change agent behavior, which changes the interaction context available to other agents. We use butterfly effect to describe the resulting interaction-mediated propagation of sensitivity:

p⟶behavioral sensitivity⏟agent level⟶interactioncollective divergence⏟system level(2)p \longrightarrow\underbrace{\text{behavioral sensitivity}}_{\text{agent level}} \mathrel{\mathop{\longrightarrow}\limits^{\text{interaction}}} \underbrace{\text{collective divergence}}_{\text{system level}} \tag*{(2)}

When local and collective sensitivity are measured on a common outcome scale, interaction amplification corresponds to Scollective(p)>Slocal(p)S_{\mathrm{collective}}(p) > S_{\mathrm{local}}(p). The corresponding difference in effect magnitude is positive under amplification and negative under attenuation. When propagation spans outcomes measured at different levels, we characterize the stages separately. We distinguish representation perturbations, which vary how a fixed simulation specification is presented to the LLM, from design perturbations, which vary components of the simulated system itself; § 5 generalizes this distinction in TRAILS.

Repeated Prisoner's Dilemma

Setup. We use a 10-round repeated Prisoner’s Dilemma with actions Cooperate (CC) and Defect (DD) and payoffs (C,C)=(3,3)(C,C)=(3,3), (C,D)=(0,5)(C,D)=(0,5), (D,C)=(5,0)(D,C)=(5,0), and (D,D)=(1,1)(D,D)=(1,1). In the single-agent regime, one LLM plays against four fixed policies, O={Tit-for-Tat,Random,Always-Cooperate,Always-Defect}\mathcal{O}=\{\text{Tit-for-Tat},\text{Random},\text{Always-Cooperate},\text{Always-Defect}\}; in the two-agent regime, two matched LLM agents play each other. Each condition contains N=30N=30 runs.

Perturbations and measures. We vary three representation dimensions: persona format (PLAIN, DESCRIPTIVE, TABULAR), game framing (CANONICAL, MORALIZED, RISK), and memory representation (table or narrative history, with or without summary statistics). We measure cooperation rate as the primary outcome and average per-round payoff as a secondary outcome. Full configurations, prompt constructions, and metric definitions are provided in Appendix § 10.§ 1.

Interaction amplification. For model mm and representation contrast p=(c0,c1)p=(c_0,c_1), let

Δm,p,o=E(Y∣c1,m,o)−E(Y∣c0,m,o)\Delta_{m,p,o} = \mathbb{E}(Y \mid c_1,m,o) - \mathbb{E}(Y \mid c_0,m,o)

denote the signed perturbation effect against fixed opponent o∈Oo \in\mathcal{O}. We summarize sensitivity across the four benchmark contexts as

Sm,pfixed=1∣O∣∑o∈O∣Δm,p,o∣.S^{\mathrm{fixed}}_{m,p} = \frac{1}{|\mathcal{O}|}\sum_{o\in\mathcal{O}}\left|\Delta_{m,p,o}\right|.

Under LLM–LLM interaction,

Sm,pLLM=∣E(Y∣c1,m,LLM)−E(Y∣c0,m,LLM)∣.S^{\mathrm{LLM}}_{m,p} = \left|\mathbb{E}(Y \mid c_1,m,\mathrm{LLM}) - \mathbb{E}(Y \mid c_0,m,\mathrm{LLM})\right|.

The interaction effect gap is

Am,p=Sm,pLLM−Sm,pfixed.A_{m,p} = S^{\mathrm{LLM}}_{m,p} - S^{\mathrm{fixed}}_{m,p}.

We refer to Am,p>0A_{m,p}>0 as amplification and Am,p<0A_{m,p}<0 as attenuation; Am,p=0A_{m,p}=0 indicates equal sensitivity across interaction regimes.

Social-Media Echo Chamber

Setup. We simulate N=100N=100 agents for T=15T=15 rounds on a fixed social network. Each agent has a persona and fixed stance si∈{−2,−1,0,1,2}s_i \in\{-2,-1,0,1,2\}. Activated agents observe recent neighbor posts and choose among POST, REPOST, REPLY, and SILENT; reposts and replies additionally select a visible target. Stances remain fixed, so collective outcomes capture the interaction structure that emerges conditional on the initial stance distribution.

Perturbations and measures. We vary persona representation (BULLET vs. PROSE) and five design dimensions: input-network homophily, hub assignment, activation probability, memory window,

Effect of persona format on Prisoner’s Dilemma outcomesVector (PDF)

caption: Effect of persona format on Prisoner’s Dilemma outcomes (gpt-5.2). Left: single-agent (fixed-opponent); right: two-agent (LLM–LLM). Bars show means with SD error bars; heatmaps show pairwise differences, with color indicating significance (p<.05p < .05) and gray non-significance.

and recommendation feed size. We measure agent-level interaction targeting and collective network structure using stance assortativity and weighted same-group edge ratio. Full configurations, implementation details, and metric definitions are provided in Appendix § 10.§ 2.

Sensitivity across interaction levels. For perturbation p=(c0,c1)p = (c_0, c_1), let qt(c)q_t^{(c)} denote the cross-group targeting rate and Yt(c)Y_t^{(c)} a collective network outcome under condition cc, with Y∈{r,H}Y \in\{r, H\}. We trace sensitivity across interaction levels using

Δtarget,t(p)=qt(c1)−qt(c0),ΔY,t(p)=Yt(c1)−Yt(c0),Y∈{r,H},(3)\begin{aligned} \Delta_{\mathrm{target},t}(p) &= q_t^{(c_1)} - q_t^{(c_0)}, \\ \Delta_{Y,t}(p) &= Y_t^{(c_1)} - Y_t^{(c_0)}, \qquad Y \in\{r,H\}, \tag*{(3)} \end{aligned}

with corresponding sensitivity magnitudes Starget,t(p)=∣Δtarget,t(p)∣S_{\mathrm{target},t}(p) = |\Delta_{\mathrm{target},t}(p)| and SY,t(p)=∣ΔY,t(p)∣S_{Y,t}(p) = |\Delta_{Y,t}(p)|.

We additionally compare action-type and target agreement across paired runs to locate behavioral sensitivity in action selection or interaction targeting. We trace propagation from agent-level behavioral sensitivity to collective network divergence. Formal metric definitions are provided in Appendix § 10.§ 2.

Models and statistical analysis. We execute both experiments on four model families: gpt-5.2, claude-haiku-4-5, gemini-2.5-flash, and deepseek11. The simulation run is the unit of analysis, with N=30N = 30 independent runs per condition unless otherwise specified. For paired comparisons, conditions share the same simulation seed. We report effect estimates with 95% confidence intervals and use two-sided tests with Holm correction for multiple pairwise comparisons. Full prompts and experimental configurations are provided in Appendices C and F.

Results

Sensitivity Varies Across Perturbations and Models

Representation choices can substantially shift simulated outcomes. Figure 2 shows a representative case in the repeated Prisoner’s Dilemma. For gpt-5.2, changing only the representation of

Note: PD reports the maximum absolute effect across prespecified pairwise contrasts within each representation dimension; EC reports the absolute effect of the binary contrast. PD cooperation effects are in percentage points; other outcomes use their native units. Superscripts denote perturbation type: R = representation-level (TRAILS-R) and D = design-level (TRAILS-D). Shading denotes significance: *** p<.001p < .001, ** p<.01p < .01, * p<.05p < .05; unshaded cells are not significant. For DeepSeek, PD experiments use deepseek-v3.2, whereas EC experiments use deepseek-v4-flash.

the matched strategic persona produces a large shift in the two-agent (LLM–LLM) setting: cooperation falls from 98.2% under PLAIN to 23.0% under DESCRIPTIVE, a decrease of 75.2 percentage points (95% CI [62.0, 88.3]). Average payoff decreases by 1.48 points per round over the same contrast (p<.001p < .001 for both outcomes). The same representation contrast produces different effects in the single-agent (fixed-opponent) setting, motivating the interaction-regime analysis in § 4.2. Complete directional contrasts are reported in Appendix § 11.

Representation sensitivity varies across perturbations and models. Persona format produces significant two-agent cooperation shifts across all four model families, with the largest within-dimension contrast ranging from 7.7 percentage points for deepseek-v3.2 to 77.7 percentage points for claude-haiku-4-5 (Table 1). Game framing produces significant cooperation shifts for a subset of models, while memory representation yields a significant maximum cooperation contrast only for gpt-5.2. The observed sensitivity therefore depends on both the representation dimension and the model family.

PerturbationMetricGPT-5.2Claude Haiku 4.5Gemini 2.5 FlashDeepSeek
Repeated Prisoner’s Dilemma (Two-Agent): max ∣Δ∣\lvert\Delta\rvert
Persona formatR^{\mathrm{R}}Average payoff1.48***1.49***0.79***0.14**
Persona formatR^{\mathrm{R}}Cooperation rate (pp)75.2***77.7***36.0***7.7**
Game framingR^{\mathrm{R}}Average payoff0.42***0.09*0.190.15***
Game framingR^{\mathrm{R}}Cooperation rate (pp)19.3***3.3*9.08.8***
Memory representationR^{\mathrm{R}}Average payoff0.10***0.050.290.01
Memory representationR^{\mathrm{R}}Cooperation rate (pp)6.2***2.713.81.0
Social-Media Echo Chamber: ∣Δ∣\lvert\Delta\rvert
Persona representationR^{\mathrm{R}}Stance assortativity0.194***0.0020.035***0.023
Persona representationR^{\mathrm{R}}Same-group edge ratio0.051***0.010*0.023***0.072***
Activation probabilityD^{\mathrm{D}}Stance assortativity0.068***0.103***0.108***0.107***
Activation probabilityD^{\mathrm{D}}Same-group edge ratio0.032***0.060***0.061***0.070***
Memory windowD^{\mathrm{D}}Stance assortativity0.0090.063***0.080***0.092***
Memory windowD^{\mathrm{D}}Same-group edge ratio0.0110.039***0.038***0.058***
Recommendation feed sizeD^{\mathrm{D}}Stance assortativity0.0010.0100.015*0.008
Recommendation feed sizeD^{\mathrm{D}}Same-group edge ratio0.0060.012**0.009**0.006

Table 1. Outcome sensitivity across perturbations and models. Δ\Delta denotes the difference in outcome means between two perturbation conditions.

Sensitivity is similarly selective in networked interaction. In the echo-chamber simulation, persona representation significantly changes stance assortativity for two of the four models, with the largest effect (0.194) observed for gpt-5.2, and significantly changes the same-group edge ratio across all four models (Table 1). Among design perturbations, activation probability significantly affects both outcomes across all four models, while memory window does so for three. Recommendation feed size produces smaller, more selective effects. Sensitivity in the networked simulation therefore varies across representation and design choices, models, and collective outcomes. Signed design-perturbation contrasts and confidence intervals are reported in Appendix D, Table 8.

Across both regimes, configuration-level sensitivity varies across perturbations and models. We next examine how multi-agent interaction reshapes representation sensitivity, through amplification or attenuation in the Prisoner’s Dilemma and across behavioral and collective levels in the echo-chamber simulation.

Multi-Agent Interaction Reshapes Representation Sensitivity

The preceding results establish configuration-level sensitivity across representation and design perturbations. We focus the interaction analysis on persona representation, which produces clear sensitivity in both simulation regimes and provides a common perturbation for examining how local differences develop through interaction. The repeated Prisoner’s Dilemma permits direct comparison under fixed-opponent and LLM–LLM interaction, while the echo-chamber simulation traces sensitivity across interaction targeting and collective network structure.

Reciprocal interaction can amplify or attenuate representation sensitivity. For the PLAIN→DESCRIPTIVE persona contrast, gpt-5.2 sensitivity increases from Sfixed=0.271S_{\mathrm{fixed}} = 0.271 against fixed opponents to SLLM=0.752S_{\mathrm{LLM}} = 0.752 under LLM–LLM interaction, yielding A=0.481A = 0.481 (95% CI [0.327, 0.622]; Figure 3). The interaction effect is also positive for claude-haiku-4-5 (A=0.462A = 0.462) and gemini-2.5-flash (A=0.287A = 0.287), while deepseek-v3.2 shows attenuation (A=−0.088A = -0.088). Round-wise estimates show different temporal trajectories across models. Reciprocal interaction therefore changes the magnitude and development of representation sensitivity, with the direction of modulation depending on the model. Complete estimates and confidence intervals are reported in Appendix § 11.

Interaction reshapes persona-representation sensitivity in the repeated Prisoner’s Dilemma

Figure 3. Interaction reshapes persona-representation sensitivity in the repeated Prisoner’s Dilemma. PLAIN vs. DESCRIPTIVE, N=30N = 30 paired seeds. (a) Cooperation-rate sensitivity under fixed-opponent and LLM–LLM interaction. (b) Interaction effect gap A=SLLM−SfixedA = S_{\mathrm{LLM}} - S_{\mathrm{fixed}}; positive values indicate amplification and negative values attenuation. (c) Round-wise effect gap AtA_t. Intervals and bands show paired-bootstrap 95% CIs.

Representation sensitivity can extend from targeting behavior to collective structure. In the echo-chamber simulation, persona representation changes interaction targets even when action selection remains comparatively stable (Figure 4). For gpt-5.2, action agreement is 99.7%, while target agreement is 55.1%. PROSE increases cross-group targeting by 5.15 percentage points (95% CI [4.22, 6.07]) and is associated with a 0.194 lower cumulative stance assortativity by round 15. The pattern varies across models: gemini-2.5-flash shows increased cross-group targeting (+2.31 pp) and lower assortativity (−0.035), while the final assortativity differences for claude-haiku-4-5 and deepseek-v4-flash are not statistically significant. Representation sensitivity can therefore appear at both behavioral and collective levels, with the relationship between the two varying across models. Complementary design-perturbation trajectories show differences in both interaction targeting and cumulative interaction structure, with patterns varying across perturbations and models. Complete estimates and confidence intervals are reported in Appendix D.

Three-panel plot showing persona-representation sensitivity across interaction levels

Figure 4. Persona-representation sensitivity across interaction levels in the echo-chamber simulation. PROSE vs. BULLET, N=30N = 30 paired seeds. (a) Agreement in action type and interaction target. (b) Cross-group targeting difference Δtarget,t\Delta_{\mathrm{target},t}; positive values indicate more cross-group targeting under PROSE. (c) Weighted stance-assortativity difference Δr,t\Delta_{r,t}; negative values indicate lower assortativity under PROSE. Error bars and bands show 95% CIs.

The two regimes capture complementary forms of interaction-mediated sensitivity. In repeated dyadic interaction, a perturbation effect on a common outcome can be amplified or attenuated. In networked interaction, representation sensitivity can appear across behavioral and collective levels.

From Architecture Sensitivity to Robustness Audits

TRAILS. LLM social simulations can be sensitive to both design-level perturbations, which change the model, network, or interaction protocol, and representation-level perturbations, which change how a fixed setup is presented through instructions, personas, labels, or memory formats. Our experiments found sensitivity to both, with effects varying across perturbations and models. We introduce TRAILS (Taxonomy for Robustness Audits In LLM Simulations) to organize these choices into representation (TRAILS-R) and design (TRAILS-D) audit dimensions (Table 2), given the levels identified in Appendix B. TRAILS-D organizes changes to the substantive simulation setup into eight dimensions across micro, meso, and macro levels. TRAILS-R organizes changes to how a matched simulation specification or context is presented to LLM agents into five dimensions. These choices can affect how agents interpret their inputs. Appendix § 8 provides the examples of TRAILS-R and TRAILS-D. TRAILS covers the simulation system; measurement and evaluation sensitivity falls within the downstream evaluation pipeline.

LevelDimensionDescriptionExamples
TRAILS-R: Representation-level perturbations
Representational formatAlternative encodings of a matched agent or task specification, including prose, lists, tables, and structured formats; persona formatSalinas & Morstatter [38]; Sclar et al. [42].
Instruction hierarchyInstruction order, placement, grouping, and authority across prompt componentsSalinas & Morstatter [38]; Sclar et al. [42].
Linguistic framingAlternative wording or labels while other simulation components are held fixed; game-instruction framingSalinas & Morstatter [38]; Sclar et al. [42].
Context representationAlternative representations or encodings of the same interaction history and contextual information; memory representationPark et al. [27]; Salinas & Morstatter [38].
Interaction sequencingPresentation or ordering of otherwise matched interactionsMou et al. [24].
TRAILS-D: Design-level perturbations
MicroModel substrateModel family, model variant, inference settings, and decoding parametersZhou et al. (2024).
Agent specificationPersona attributes, goals, beliefs, demographic or behavioral profiles, and population groundingPark et al. [27]; Li et al. [17].
Internal state & cognitionBelief and preference updating, reasoning procedures, reflection rules, and internal-state representationsLiu et al. [20]; Wang et al. [49].
Memory & temporalityMemory length, retrieval, summarization, forgetting, and temporal granularityPark et al. [27]; Wang et al. [48].
MesoInteraction protocolTurn structure, available actions, visibility, activation, and rules governing who can interact with whom; activation probabilityMou et al. [24]; Yang et al. [53].
Intervention designIntervention type, target, timing, frequency, delivery mechanism, and framingLiu et al. [19, 20].
MacroEnvironment structureNetwork topology, exposure and recommendation rules, institutional structure, and platform affordances; network homophily, hub assignment, recommendation feed sizeYang et al. [53]; Gu et al. [12]; Piao et al. [31].
Population & scalePopulation size, composition, heterogeneity, agent assignment, and real- versus synthetic-population constructionYang et al. [53]; Wang et al. [47]; Piao et al. [31].

Table 2. TRAILS: Taxonomy for Robustness Audits in LLM Simulations. TRAILS distinguishes representation-level perturbations (TRAILS-R), which vary how a fixed simulation specification or context is presented to LLM agents, from design-level perturbations (TRAILS-D), which vary components of the simulated system itself. Bolded examples indicate perturbations examined in our case studies.

The Social Simulation Audit Card

Figure 5. The Social Simulation Audit Card. The five-step workflow connects perturbation selection and sensitivity analysis to the scope of scientific claims. GPT-5.2 examples illustrate two interpretations of sensitivity: a context-irrelevant persona-format change reveals implementation dependence in the repeated Prisoner’s Dilemma, whereas context-relevant network homophily bounds the conditions under which echo-chamber-like interaction structure is supported.

The Social Simulation Audit Card. We introduce the Social Simulation Audit Card to operationalize TRAILS for a stated scientific claim. The card guides researchers through five steps: stating the claim, mapping perturbations to TRAILS and context relevance, registering perturbations and preservation criteria, reporting sensitivity, and defining the claim’s scope. Context relevance is claim-specific: context-relevant perturbations represent substantively meaningful conditions of the modeled phenomenon, whereas context-irrelevant perturbations probe implementation dependence. For example, network homophily and hub assignment are context-relevant structural conditions in the echo-chamber simulation; their sensitivity results are reported in Appendix § 11. Accordingly, context-relevant sensitivity helps scope where a claim holds, while substantial sensitivity to context-irrelevant choices indicates implementation dependence. Outcomes are classified as Robust, Scope-limited, Implementation-sensitive, or Inconclusive, bounded by the audited conditions. Appendix § 12 provides the full procedure and case-study audits.

Conclusions

LLM social simulations can be sensitive to choices that are often treated as interchangeable or left unaudited. Across repeated Prisoner’s Dilemma and echo-chamber simulations, we find that sensitivity varies across perturbations, models, and outcomes. Persona representation produces substantial shifts in several settings, while other representation and design perturbations have smaller or more selective effects. Multi-agent interaction further reshapes this sensitivity. In repeated dyadic interaction, representation effects can be amplified or attenuated under reciprocal LLM–LLM interaction; in networked interaction, representation sensitivity appears in interaction targeting and, for some models, in collective network structure. These results indicate that robustness observed under one configuration or model does not establish robustness of the resulting scientific claim.

Our findings are limited to two simulation regimes, selected perturbations, four model families, and fixed interaction horizons, so the observed sensitivity patterns should not be assumed to generalize to other settings. The echo-chamber experiment holds agent stances fixed and therefore characterizes interaction structure rather than endogenous opinion change. Its behavioral and collective analyses identify cross-level patterns of sensitivity but do not establish a causal pathway from targeting differences to network divergence. More broadly, robustness to implementation choices addresses only one dimension of simulation validity; correspondence between simulated agents and human behavior requires separate empirical validation.

The broader implication concerns the scope of scientific inference from LLM social simulations. Robustness audits should identify which conclusions persist across plausible implementations, which depend on substantively meaningful conditions, and which remain sensitive to implementation choices. TRAILS and the Social Simulation Audit Card provide a vocabulary and reporting structure for conducting such audits, with the required coverage depending on the claim and simulation context. Scientific claims from simulated societies can then be scoped to the configurations, models, and interaction conditions for which robustness evidence is available.

Relationship to Earlier Work

An earlier position paper introduced the robustness-audit perspective and TRAILS framework, with selected results from the same simulation settings serving as motivating examples [54]. The present study extends the empirical analysis through systematic sensitivity comparisons across perturbations and models, explicit measurement of interaction amplification and attenuation, and analysis of behavioral and collective outcomes in networked interaction.

TRAILS (Taxonomy for Robustness Audits In LLM Simulations)

LevelDimensionWhat to perturb
MicroModel substrateModel family, model size, base vs. instruction-tuned models, alignment, temperature, top-pp, random seed, safeguards.
Agent specificationDemographics, ideology, personality, goals, prior beliefs, real-data-grounded vs. synthetic personas, relationship attributes.
Internal state and cognitionBeliefs, attitudes, emotions, needs, moral values, reasoning style, reflection rules, belief updating, preference updating.
Memory and temporalitylast-kk memory, summarized memory, episodic memory, retrieval rules, forgetting, reflection frequency, temporal granularity.
MesoInteraction protocolTurn order, fixed or free turns, dyadic or group interaction, available actions
Intervention designModerator type, target selection, timing, frequency, rule-based vs. LLM-based intervention.
MacroEnvironment structureNetwork topology, feed ranking, recommendation system, moderation rules, platform affordances, institutional rules.
Population and scaleNumber of agents, demographic distribution, ideological balance, heterogeneity, real vs. synthetic population construction.

Table 3. TRAILS-D: Design-level perturbations

CategoryPerturbationExample
Representational formatFormattingNarrative prose vs. bullet points; paragraph vs. table; bolded keywords; numbered lists.
Delimiter choiceConversation history enclosed in code blocks, XML tags, JSON, or plain text.
Output schemaFree-form paragraph vs. short reply vs. JSON with rationale.
Instruction hierarchyInstruction order“Update your opinion, then write a reply” vs. “Write a reply, then update your opinion.”
Role/message placementThe same instruction placed in the system message vs. the user message.
Linguistic framingAgent namingAgent A/B vs. Alice/Bob.
Persona wording“Left” vs. “Democrat”; “Right” vs. “Republican.”
Moderation framingSame intervention framed as a warning, suggestion, community reminder, or bridge statement.
Few-shot examplesSame examples framed differently.
Context representationMemory representationSame history shown as transcript, JSON list, bullet summary.
Context windowFull history vs. last three turns; same content with early background removed.
Message metadataAdding timestamps, usernames.
Interaction sequencingInitial seed turnSame stance, but a slightly different opening statement.
Turn orderWhich agent speaks first.

Table 4. TRAILS-R: Representation-level perturbations

Extended Related Works

LLM-based social simulations

Scope. Following work on generative social simulation and its validation [33, 3, 16], we focus on collective human behavior arising from interacting LLM agents. This scope connects individual decisions, communication, and memory to collective outcomes through a shared social environment. We distinguish it from individual or user simulations without inter-agent interaction [36, 55], simulations centered on non-human or aggregate actors, and multi-agent systems designed primarily for task completion [14, 34].

Social simulation scenarios. Our two testbeds capture complementary interaction settings. In goal-structured scenarios, explicit incentives and rules organize strategic interaction, as in cooperation and negotiation [2, 1]. In open-ended scenarios, repeated communication and engagement produce collective patterns, including echo chambers [12, 46], social norms [37], and coordinated information operations [25]. This distinction motivates comparing a fixed strategic game with networked social interaction rather than treating either as representative of all social simulations.

Factors shaping audit strength. Audit requirements depend on the claim, the simulation’s openness, and the stakes of its use. Exploratory findings require checks of prompt, seed, and model dependence; mechanism claims require plausible alternative specifications; and intervention or policy claims require broader coverage and greater precision. Open-ended simulations expose more choices about communication, memory, and platform feedback, expanding the perturbation space. Sensitivity analysis and simulation validation therefore motivate matching audit coverage and uncertainty to the intended inference [39, 40].

Repeated Prisoner's Dilemma as a Goal-Structured Testbed

Repeated Prisoner’s Dilemma provides a controlled testbed for cooperation and adaptation in LLM agents [2, 11]. Fixed actions, payoffs, and horizon let us compare representation sensitivity across fixed-opponent and reciprocal LLM–LLM interaction.

The three perturbations target how the agent, game, and history are represented. Persona format tests whether matched strategic content yields stable behavior across presentation formats, complementing concerns about persona validity [17]. Game framing tests alternative descriptions of the same payoff structure, motivated by framing sensitivity in strategic settings [21]. Memory representation tests how tabular or narrative histories and summary statistics affect decisions, building on work examining LLMs’ use of gameplay logs [11]. These contrasts separate representation dependence from changes to the game’s rules.

Echo Chambers as an Open-Ended Social Simulation Testbed

Echo-chamber simulations provide a complementary setting in which collective structure emerges from local engagement rather than a shared payoff. Prior studies connect network structure, selective exposure, and agent behavior to polarization and echo-chamber formation [12, 46, 10, 30]. We use this setting to trace sensitivity from interaction targeting to the network formed by those choices.

We first vary persona representation while preserving persona information, testing whether presentation alone changes targeting and collective structure. The design perturbations then follow the order of the main outcome summary: activation probability changes participation opportunities; memory window changes the recent interaction context; and recommendation feed size changes visible exposure, a central component of echo-chamber models [46, 45]. Finally, input-network homophily varies initial like-minded connectivity, and hub assignment varies which stance group occupies highly connected positions while preserving the degree sequence. These structural contrasts complement the main analyses and provide context-relevant conditions for the Audit Card, rather than alternative representations of a fixed system.

Additional Experimental Details

Repeated Prisoner’s Dilemma

Setup and Configurations

Each Prisoner’s Dilemma episode runs for T=10T = 10 rounds. In each round, each player simultaneously chooses Cooperate (CC) or Defect (DD), with payoffs (C,C)=(3,3)(C,C) = (3,3), (C,D)=(0,5)(C,D) = (0,5), (D,C)=(5,0)(D,C) = (5,0), and (D,D)=(1,1)(D,D) = (1,1).

We evaluate two interaction regimes. In the single-agent regime, one LLM-controlled agent plays separately against four fixed policies: Tit-for-Tat, Random, Always-Cooperate, and Always-Defect. Tit-for-Tat cooperates in the first round and subsequently copies the opponent’s previous action; Random selects CC or DD with equal probability; Always-Cooperate and Always-Defect select the corresponding action in every round. In the two-agent regime, two LLM-controlled agents with matched configurations play one another. Following prior work, these fixed policies provide benchmark interaction contexts with distinct response structures ([2, 11, 8]).

We evaluate gpt-5.2, claude-haiku-4-5, gemini-2.5-flash, and deepseek-v3.2. Each condition contains N=30N = 30 simulation runs. In two-agent runs, both agents are queried once per round and prompt order is randomized across rounds to reduce stable positional effects. If a response cannot be parsed as a valid action after one retry, the simulation falls back to DEFECT. Decoding and parsing settings are held fixed within each comparison; unless otherwise specified, model calls use temperature =0.3= 0.3 and top-p=0.95p = 0.95.

We report condition means and signed mean differences, with 95% confidence intervals for selected contrasts and interaction-amplification estimates. Pairwise outcome comparisons use two-sided paired tests with Holm correction within each prespecified comparison family (Holm [13]). For the Prisoner’s Dilemma, correction is applied separately within each model, representation dimension, and outcome across its prespecified pairwise contrasts. Interaction-amplification analyses use matched simulation seeds and paired-seed bootstrap procedures as described in Appendix § 10.1.4.

Outcome Measures

For agent ii, let ai,t∈{C,D}a_{i,t} \in\{C,D\} denote the action selected at round tt, and let ui,tu_{i,t} denote the corresponding payoff.

Average payoff. Average payoff is the mean per-round reward:

Payoff⁡i=1T∑t=1Tui,t.\operatorname{Payoff}_i = \frac{1}{T}\sum_{t=1}^{T} u_{i,t}.

Cooperation rate. Cooperation rate is the fraction of rounds in which the agent selects CC:

CoopRate⁡i=1T∑t=1TI(ai,t=C).\operatorname{CoopRate}_i = \frac{1}{T}\sum_{t=1}^{T} \mathbb{I}(a_{i,t}=C).

Cooperation rate is the primary outcome for comparing representation sensitivity across interaction regimes, with average payoff providing the corresponding outcome under the game’s incentive structure.

Perturbation Details

We evaluate three representation dimensions while holding the payoff structure and non-targeted prompt components fixed: persona format, game-instruction framing, and memory representation.

Persona format. We present a matched strategic persona in three formats: plain prose (PLAIN), a descriptive bullet list (DESCRIPTIVE), and a structured key–value table (TABULAR). The persona specifies a strategic participant focused on maximizing long-term payoff, studying the opponent for exploitable patterns, and adapting its strategy while remaining competitive.

Plain strategic persona.

You are a strategic participant focused on maximizing your long-term payoff. Study
opponent behavior for exploitable patterns, and adjust your strategy to gain advantage
while staying competitive.

Descriptive strategic persona.

Persona profile:
- Maximize your long-term payoff
- Study opponent for exploitable patterns
- Adjust strategy to gain advantage
- Stay strategic and competitive

Tabular strategic persona.

Persona | Value
Goal | Maximize your long-term payoff
Observation focus | Study opponent for exploitable patterns
Strategy | Adjust strategy to gain advantage
Style | Stay strategic and competitive

Game-instruction framing. We hold the persona fixed to PLAIN and vary the framing of the fixed payoff structure. CANONICAL uses standard game-theoretic labels such as “Cooperate” and “Defect”; MORALIZED describes the actions using cooperation and exploitation; and RISK describes the same decision in terms of risk, vulnerability, and potential reward.

Canonical game-theory framing.

You are playing a repeated Prisoner’s Dilemma for 10 rounds.

In each round, you choose one of two actions:
- Cooperate
- Defect

The payoffs for each round are:
- If both players cooperate: both receive 3 points.
- If you cooperate and the other defects: you receive 0, the other receives 5.
- If you defect and the other cooperates: you receive 5, the other receives 0.
- If both defect: both receive 1.

Your goal is to maximize your total points over all rounds.

At each round, you will see the history of previous rounds.

Moralized framing.

You will interact with another participant for 10 rounds.

In each round, you must decide whether to:
- Cooperate fairly with the other person.
- Exploit the other person for personal gain.

The outcomes are:
- If both cooperate fairly: both receive 3 points.
- If you cooperate but they exploit you: you receive 0, they receive 5.
- If you exploit them while they cooperate: you receive 5, they receive 0.
- If both exploit each other: both receive 1.

Your goal is to maximize your total points.

At each round, you will see past interactions.

Risk framing.

You will make decisions over 10 rounds against another participant.

Each round involves a choice:
- Cooperate, which is safer but vulnerable.
- Defect, which is riskier but potentially higher reward.

Outcomes:
- Both cooperate: both get 3.
- You cooperate, other defects: you get 0, they get 5.
- You defect, other cooperates: you get 5, they get 0.
- Both defect: both get 1.
Your goal is to maximize total payoff.
At each round, you will see previous outcomes.

Memory representation. We hold persona and game framing fixed and vary how the interaction history is presented. The design crosses history format (table vs. narrative) with the presence or absence of summary statistics, yielding TABLE-RAW, NARRATIVE-RAW, TABLE+STATS, and NARRATIVE+STATS. Summary statistics include cooperation rates, cumulative payoffs, and joint outcome counts.

M1: Table, raw history only.

History:
Round | You | Opponent | Your payoff | Opp payoff
1 | [YOUR ACTION] | [OPPONENT ACTION] | [YOUR PAYOFF] | [OPPONENT PAYOFF]
2 | [YOUR ACTION] | [OPPONENT ACTION] | [YOUR PAYOFF] | [OPPONENT PAYOFF]
...

M2: Narrative, raw history only.

History:
In round 1, you chose [YOUR ACTION] and the opponent chose [OPPONENT ACTION]. You received [YOUR PAYOFF] points and the opponent received [OPPONENT PAYOFF].
In round 2, you chose [YOUR ACTION] and the opponent chose [OPPONENT ACTION]. You received [YOUR PAYOFF] points and the opponent received [OPPONENT PAYOFF].
...

M3: Table, raw history plus statistics.

History:
Round | You | Opponent | Your payoff | Opp payoff
1 | [YOUR ACTION] | [OPPONENT ACTION] | [YOUR PAYOFF] | [OPPONENT PAYOFF]
2 | [YOUR ACTION] | [OPPONENT ACTION] | [YOUR PAYOFF] | [OPPONENT PAYOFF]
...

Summary statistics:
- You cooperated [N] times and defected [N] times.
- Your cooperation rate: [RATE].
- Opponent cooperated [N] times and defected [N] times.
- Opponent cooperation rate: [RATE].
- Your cumulative payoff: [PAYOFF].
- Opponent cumulative payoff: [PAYOFF].
- Your average payoff per round: [AVG].
- Opponent average payoff per round: [AVG].
- Joint outcomes: CC=[N], CD=[N], DC=[N], DD=[N].

M4: Narrative, raw history plus statistics.

History:
In round 1, you chose [YOUR ACTION] and the opponent chose [OPPONENT ACTION]. You received [YOUR PAYOFF] points and the opponent received [OPPONENT PAYOFF].
In round 2, you chose [YOUR ACTION] and the opponent chose [OPPONENT ACTION]. You received [YOUR PAYOFF] points and the opponent received [OPPONENT PAYOFF].
...

Summary statistics:
- You cooperated [N] times and defected [N] times.
- Your cooperation rate: [RATE].
- Opponent cooperated [N] times and defected [N] times.
- Opponent cooperation rate: [RATE].
- Your cumulative payoff: [PAYOFF].
- Opponent cumulative payoff: [PAYOFF].
- Your average payoff per round: [AVG].
- Opponent average payoff per round: [AVG].
- Joint outcomes: CC=[N], CD=[N], DC=[N], DD=[N].

For the two-agent memory condition, the same four memory variants are used, but the history is role-specific. For Agent A, “You” refers to Agent A and “Other agent” refers to Agent B; for Agent B, the mapping is reversed.

Interaction-Amplification Measures

For model mm, representation contrast p=(c0,c1)p=(c_0,c_1), and interaction context oo, we define the signed perturbation effect on cooperation as

Δm,p,o=E[Y∣c1,m,o]−E[Y∣c0,m,o].\Delta_{m,p,o}=\mathbb{E}[Y\mid c_1,m,o]-\mathbb{E}[Y\mid c_0,m,o].

where YY denotes cooperation rate. For the fixed-opponent regime,

O={Tit-for-Tat,Random,Always-Cooperate,Always-Defect}.\mathcal{O}=\{\text{Tit-for-Tat},\text{Random},\text{Always-Cooperate},\text{Always-Defect}\}.

We summarize sensitivity across fixed-opponent contexts as

Sm,pfixed=1∣O∣∑o∈O∣Δm,p,o∣,S^{\mathrm{fixed}}_{m,p}=\frac{1}{|\mathcal{O}|}\sum_{o\in\mathcal{O}}|\Delta_{m,p,o}|,

and sensitivity under reciprocal LLM–LLM interaction as

Sm,pLLM=∣Δm,p,LLM∣.S^{\mathrm{LLM}}_{m,p}=|\Delta_{m,p,\mathrm{LLM}}|.

The interaction effect gap is

Am,p=Sm,pLLM−Sm,pfixed.A_{m,p}=S^{\mathrm{LLM}}_{m,p}-S^{\mathrm{fixed}}_{m,p}.

Positive Am,pA_{m,p} indicates amplification, while negative Am,pA_{m,p} indicates attenuation. The fixed-opponent measure averages the absolute opponent-specific effects before comparison with the LLM–LLM effect, preserving sensitivity that differs in direction across benchmark interaction contexts.

We estimate uncertainty in Am,pA_{m,p} using 10,000 paired bootstrap replicates over the 30 matched simulation seeds. Within each replicate, the same sampled seeds are used across representation conditions, fixed opponents, and the LLM–LLM regime. We report pointwise percentile 95% confidence intervals for Am,pA_{m,p} and compute the corresponding quantities by round to characterize temporal variation.

Social-Media Echo Chamber

Setup and Configurations

The echo-chamber simulation contains N=100N=100 agents interacting for T=15T=15 rounds on a social network. Each agent has a persona and a fixed stance si∈{−2,−1,0,1,2}s_i\in\{-2,-1,0,1,2\} toward stronger regulation of advanced AI systems. Agent stances remain fixed throughout each run, so the simulation measures interaction structure conditional on the initial stance distribution.

In each round, agents become active according to the configured activation probability. If no agent is sampled as active, one agent is selected at random so that every round contains at least one decision. Active agents observe recent posts from direct network neighbors and choose among POST, REPOST, REPLY, and DO NOTHING. Reposts and replies additionally select a visible interaction target. The default configuration uses a feed size of 5 posts and a memory window of 2 rounds. Invalid repost or reply targets are converted according to the simulation’s parsing rules while preserving the fixed action space.

We evaluate gpt-5.2, claude-haiku-4-5, gemini-2.5-flash, and deepseek-v4-flash. Unless otherwise specified, each condition contains N=30N=30 simulation runs. Decoding and parsing settings are held fixed within each comparison.

Network construction. To control overall connectivity across structural conditions, we construct networks from a fixed power-law-like degree sequence with target mean degree 8 and exponent 2.4. The sequence is adjusted until graphical, producing a heavy-tailed network in which a small number of agents occupy high-degree positions. The same degree sequence is reused across network variants, holding the degree distribution and overall connectivity fixed.

Outcome Measures

We measure representation sensitivity at both the agent and collective levels. For paired persona-representation runs, agent decisions are matched by (seed, round, agent) across the BULLET and PROSE conditions.

Action agreement. For two paired conditions c0c_0 and c1c_1, action agreement measures whether the same agent selects the same action at the same round. For paired decision kk,

Ikaction=I(ak(c1)=ak(c0)).I_k^{\mathrm{action}} = \mathbb{I}\left(a_k^{(c_1)} = a_k^{(c_0)}\right).

For seed ss, action agreement is

Gsaction=1∣Ks∣∑k∈KsIkaction,G_s^{\mathrm{action}} = \frac{1}{|\mathcal{K}_s|} \sum_{k \in\mathcal{K}_s} I_k^{\mathrm{action}},

where Ks\mathcal{K}_s contains the matched agent–round decisions for that seed. We compute agreement within each seed and then summarize across paired seeds.

Target agreement. Target agreement measures whether paired decisions select the same non-missing interaction-target author:

Iktarget=I(tk(c1)=tk(c0)≠∅),I_k^{\mathrm{target}} = \mathbb{I}\left(t_k^{(c_1)} = t_k^{(c_0)} \ne\varnothing\right),

where tkt_k denotes the selected target author. Different targets, a target in only one condition, and no target in either condition are counted as non-agreement. Seed-level target agreement is

Gstarget=1∣Ks∣∑k∈KsIktarget.G_s^{\mathrm{target}} = \frac{1}{|\mathcal{K}_s|} \sum_{k \in\mathcal{K}_s} I_k^{\mathrm{target}}.

For the persona-representation analysis, c0=BULLETc_0 = \mathrm{BULLET} and c1=PROSEc_1 = \mathrm{PROSE}.

Weighted stance assortativity. Let GI=(V,EI)G_I = (V, E_I) denote the realized interaction graph, where an edge from agent ii to agent jj records a repost or reply, and let AijA_{ij} denote the number of observed interactions from ii to jj. Let gi∈Gg_i \in\mathcal{G} denote agent ii’s stance group. For stance groups u,v∈Gu, v \in\mathcal{G}, define the fraction of weighted interactions from group uu to group vv as

puv=∑i,jAijI(gi=u,gj=v)∑i,jAij,p_{uv} = \frac{\sum_{i,j} A_{ij}\mathbb{I}(g_i = u, g_j = v)}{\sum_{i,j} A_{ij}},

with marginals

puout=∑v∈Gpuv,puin=∑v∈Gpvu.p_u^{\mathrm{out}} = \sum_{v \in\mathcal{G}} p_{uv}, \qquad p_u^{\mathrm{in}} = \sum_{v \in\mathcal{G}} p_{vu}.

Weighted stance assortativity is

r=∑u∈Gpuu−∑u∈Gpuoutpuin1−∑u∈Gpuoutpuin.r = \frac{\sum_{u \in\mathcal{G}} p_{uu} - \sum_{u \in\mathcal{G}} p_u^{\mathrm{out}}p_u^{\mathrm{in}}}{1 - \sum_{u \in\mathcal{G}} p_u^{\mathrm{out}}p_u^{\mathrm{in}}}.

Positive values indicate greater within-stance interaction than expected from the observed interaction marginals, while negative values indicate greater cross-stance interaction.

Weighted same-group edge ratio. We additionally measure the fraction of interaction volume occurring within stance groups:

H=∑i,jAijI(gi=gj)∑i,jAij.H = \frac{\sum_{i,j} A_{ij}\mathbb{I}(g_i = g_j)}{\sum_{i,j} A_{ij}}.

Larger values indicate a greater concentration of interaction within stance groups.

Perturbation Details

We evaluate one representation perturbation and five design perturbations. The representation perturbation varies persona presentation while preserving persona information. The design perturbations vary input-network homophily, hub assignment, activation probability, memory-window length, and recommendation feed size.

Persona representation. We present matched persona information either as structured bullets (BULLET) or narrative prose (PROSE), while holding the remaining simulation configuration fixed.

Structured bullet-point persona (original).

Your profile:
- agent id: {agent_id}
- bio: {bio}
- follower count: {follower_count}
- current stance on AI regulation: {stance}
  (where −2 strongly oppose, −1 somewhat oppose,
  0 mixed/undecided, 1 somewhat support,
  2 strongly support)
- neighbor count in the network: {neighbor_count}

Narrative-prose persona.

You are {agent_id}, {bio}. You have {follower_count} followers, and {neighbor_count}
people in the network whose posts you can see. On the question of whether governments
should impose stronger regulation on advanced AI systems, you currently
{stance_description}.

Here, {stance_description} is the prose equivalent of the numerical stance: strongly oppose, somewhat oppose, hold a mixed or undecided view, somewhat support, or strongly support stronger regulation.

Interaction-level design perturbations. Activation probability compares 0.3 and 0.5 for all models. Memory window compares 2 and 4 rounds, and recommendation feed size compares up to 5 and 10 recent neighbor posts.

Input-network homophily (context-relevant design-level perturbation). We vary stance homophily while preserving the fixed degree sequence. Let G0=(V,E0)G_0 = (V, E_0) denote the input network and xix_i agent ii’s stance on the five-point scale. Starting from a graph realizing the fixed degree sequence, we apply degree-preserving double-edge swaps to obtain three input-network assortativity ranges:

LOW : h(G0)∈[0.02,0.10]h(G_0) \in[0.02, 0.10], MEDIUM : h(G0)∈[0.12,0.20]h(G_0) \in[0.12, 0.20], HIGH : h(G0)∈[0.22,0.30]h(G_0) \in[0.22, 0.30].

The rewiring procedure changes which stance groups are connected while preserving each agent’s degree.

Hub assignment (context-relevant design-level perturbation). We vary which agents occupy high-degree positions while preserving the degree sequence. The Random Hub condition randomly assigns degrees across agents, while Mixed Hub distributes high-degree positions across stance groups. The Pro Hub and Anti Hub conditions preferentially assign high-degree positions to the corresponding stance group. Networks are subsequently rewired to the medium-homophily range, h(G0)∈[0.12,0.20]h(G_0) \in[0.12, 0.20], approximately holding stance homophily fixed across hub conditions.

Statistical comparisons. Conditions are matched by simulation seed. Binary design perturbations are evaluated using two-sided paired tt-tests. Holm correction is applied separately within each model and outcome across activation probability, memory window, and recommendation feed size. Persona-representation effects are evaluated using two-sided paired Wilcoxon signed-rank tests, with Holm correction across the four models and two collective outcomes. All significance markers in the outcome-sensitivity tables use the corresponding Holm-corrected pp-values.

Propagation Measures

For perturbation p=(c0,c1)p = (c_0, c_1), let qt(c)q_t^{(c)} denote the cross-group targeting rate under condition cc at round tt. The signed behavioral effect and its sensitivity magnitude are

Δtarget,t(p)=qt(c1)−qt(c0),Starget,t(p)=∣Δtarget,t(p)∣.\Delta_{\mathrm{target},t}(p) = q_t^{(c_1)} - q_t^{(c_0)}, \qquad S_{\mathrm{target},t}(p) = \left|\Delta_{\mathrm{target},t}(p)\right|.

For paired persona-representation runs, we additionally compute action-type agreement and interaction-target agreement to locate representation sensitivity in action selection or interaction targeting.

For collective network outcome Y∈{r,H}Y \in\{r, H\}, we define

ΔY,t(p)=Yt(c1)−Yt(c0),SY,t(p)=∣ΔY,t(p)∣.\Delta_{Y,t}(p) = Y_t^{(c_1)} - Y_t^{(c_0)}, \qquad S_{Y,t}(p) = \left|\Delta_{Y,t}(p)\right|.

We trace representation sensitivity across interaction levels by comparing the temporal development of behavioral effects and collective network effects.

Note: Values report estimated differences between the indicated perturbation conditions. Cooperation effects are reported in percentage points; average payoff effects use their native units. For memory contrasts, NR = Narrative-Raw, NS = Narrative+Stats, TR = Table-Raw, and TS = Table+Stats. Shading denotes Holm-corrected significance: *** p<.001p < .001, ** p<.01p < .01, * p<.05p < .05; unshaded cells are not statistically significant.

Extended Experimental Results

This section provides extended results for the Prisoner’s Dilemma (PD) and echo-chamber experiments. For the PD experiment, Table 5 reports pairwise outcome contrasts across perturbations, representations, and models, while Table 6 provides the complete interaction-amplification results for the prespecified representation contrasts. For the echo-chamber experiment, Table 7 reports representation sensitivity in both behavioral and collective network outcomes. Table 8 reports signed design-perturbation contrasts with confidence intervals across all four models. We additionally report the effects of network homophily and hub assignment in Figure 8. Although these two perturbations are not analyzed in the main text, they serve as the audit-card examples of context-relevant perturbations; we therefore include their corresponding experimental results here for completeness.

PerturbationOutcomeContrastGPT-5.2Claude Haiku 4.5Gemini 2.5 FlashDeepSeek V3.2
Repeated Prisoner’s Dilemma (Two-Agent)
PersonaAverage payoffDescriptive → Tabular+1.467***+1.223***-0.010-0.127
Plain → Descriptive-1.478***-1.487***-0.777***-0.013*
Plain → Tabular-0.012-0.263***-0.787***-0.140**
PersonaCooperation rate (pp)Descriptive → Tabular+74.000***+63.333***-0.333-6.333
Plain → Descriptive-75.167***-77.667***-35.667***-1.333*
Plain → Tabular-1.167-14.333***-36.000***-7.667**
Game framingAverage payoffCanonical → Moralized+0.417***+0.025-0.187+0.028**
Canonical → Risk+0.110+0.087*-0.142-0.120**
Moralized → Risk-0.307**+0.062+0.045-0.148***
Game framingCooperation rate (pp)Canonical → Moralized+19.333***+0.833-9.000+2.167**
Canonical → Risk+4.667+3.333*-7.500-6.667**
Moralized → Risk-14.667*+2.500+1.500-8.833***
MemoryAverage payoffNR → NS+0.038*+0.043+0.092+0.000
NR → TS+0.088***+0.048+0.017+0.005
TS → NS-0.050**-0.005+0.075-0.005
TR → NS-0.063***+0.045+0.285+0.008
TR → NR-0.102***+0.002+0.193+0.008
TR → TS-0.013*+0.050+0.210+0.013
MemoryCooperation rate (pp)NR → NS+1.833*+2.333+4.833-0.000
NR → TS+4.833***+2.500+1.667+0.500
TS → NS-3.000**-0.167+3.167-0.500
TR → NS-4.333***+2.500+13.833+0.500
TR → NR-6.167***+0.167+9.000+0.500
TR → TS-1.333*+2.667+10.667+1.000

Table 5. Pairwise contrasts in the Prisoner’s Dilemma experiment across perturbations, representations, and models. Δ\Delta denotes the difference in outcome means between two perturbation conditions.

ContrastGPT-5.2Claude Haiku 4.5Gemini 2.5 FlashDeepSeek V3.2
Persona format
Plain vs DescriptiveSfixed=0.271S_{\mathrm{fixed}} = 0.271<br>SLLM=0.752S_{\mathrm{LLM}} = 0.752<br>A=+0.481A = +0.481<br>[+0.327,+0.622][+0.327, +0.622]<br>AmplificationSfixed=0.315S_{\mathrm{fixed}} = 0.315<br>SLLM=0.777S_{\mathrm{LLM}} = 0.777<br>A=+0.462A = +0.462<br>[+0.376,+0.529][+0.376, +0.529]<br>AmplificationSfixed=0.070S_{\mathrm{fixed}} = 0.070<br>SLLM=0.357S_{\mathrm{LLM}} = 0.357<br>A=+0.287A = +0.287<br>[+0.190,+0.387][+0.190, +0.387]<br>AmplificationSfixed=0.101S_{\mathrm{fixed}} = 0.101<br>SLLM=0.013S_{\mathrm{LLM}} = 0.013<br>A=−0.087A = -0.087<br>[−0.108,−0.066][-0.108, -0.066]<br>Attenuation
Plain vs TabularSfixed=0.589S_{\mathrm{fixed}} = 0.589<br>SLLM=0.012S_{\mathrm{LLM}} = 0.012<br>A=−0.578A = -0.578<br>[−0.602,−0.556][-0.602, -0.556]<br>AttenuationSfixed=0.429S_{\mathrm{fixed}} = 0.429<br>SLLM=0.143S_{\mathrm{LLM}} = 0.143<br>A=−0.286A = -0.286<br>[−0.341,−0.223][-0.341, -0.223]<br>AttenuationSfixed=0.074S_{\mathrm{fixed}} = 0.074<br>SLLM=0.360S_{\mathrm{LLM}} = 0.360<br>A=+0.286A = +0.286<br>[+0.192,+0.384][+0.192, +0.384]<br>AmplificationSfixed=0.055S_{\mathrm{fixed}} = 0.055<br>SLLM=0.077S_{\mathrm{LLM}} = 0.077<br>A=+0.022A = +0.022<br>[−0.050,+0.121][-0.050, +0.121]<br>No detectable difference
ContrastGPT-5.2Claude Haiku 4.5Gemini 2.5 FlashDeepSeek V3.2
Descriptive vs TabularSfixed=0.318S_{\mathrm{fixed}} = 0.318<br>SLLM=0.740S_{\mathrm{LLM}} = 0.740<br>A=+0.422A = +0.422<br>[+0.295,+0.546][+0.295, +0.546]<br>AmplificationSfixed=0.244S_{\mathrm{fixed}} = 0.244<br>SLLM=0.633S_{\mathrm{LLM}} = 0.633<br>A=+0.389A = +0.389<br>[+0.291,+0.481][+0.291, +0.481]<br>AmplificationSfixed=0.004S_{\mathrm{fixed}} = 0.004<br>SLLM=0.003S_{\mathrm{LLM}} = 0.003<br>A=−0.001A = -0.001<br>[−0.006,+0.013][-0.006, +0.013]<br>No detectable differenceSfixed=0.054S_{\mathrm{fixed}} = 0.054<br>SLLM=0.063S_{\mathrm{LLM}} = 0.063<br>A=+0.009A = +0.009<br>[−0.057,+0.102][-0.057, +0.102]<br>No detectable difference
Game framing
Canonical vs MoralizedSfixed=0.117S_{\mathrm{fixed}} = 0.117<br>SLLM=0.193S_{\mathrm{LLM}} = 0.193<br>A=+0.077A = +0.077<br>[+0.024,+0.127][+0.024, +0.127]<br>AmplificationSfixed=0.058S_{\mathrm{fixed}} = 0.058<br>SLLM=0.008S_{\mathrm{LLM}} = 0.008<br>A=−0.050A = -0.050<br>[−0.067,−0.007][-0.067, -0.007]<br>AttenuationSfixed=0.025S_{\mathrm{fixed}} = 0.025<br>SLLM=0.090S_{\mathrm{LLM}} = 0.090<br>A=+0.065A = +0.065<br>[−0.022,+0.143][-0.022, +0.143]<br>No detectable differenceSfixed=0.016S_{\mathrm{fixed}} = 0.016<br>SLLM=0.022S_{\mathrm{LLM}} = 0.022<br>A=+0.006A = +0.006<br>[−0.021,+0.017][-0.021, +0.017]<br>No detectable difference
Canonical vs RiskSfixed=0.127S_{\mathrm{fixed}} = 0.127<br>SLLM=0.047S_{\mathrm{LLM}} = 0.047<br>A=−0.080A = -0.080<br>[−0.131,+0.001][-0.131, +0.001]<br>No detectable differenceSfixed=0.086S_{\mathrm{fixed}} = 0.086<br>SLLM=0.033S_{\mathrm{LLM}} = 0.033<br>A=−0.052A = -0.052<br>[−0.088,−0.017][-0.088, -0.017]<br>AttenuationSfixed=0.024S_{\mathrm{fixed}} = 0.024<br>SLLM=0.075S_{\mathrm{LLM}} = 0.075<br>A=+0.051A = +0.051<br>[−0.021,+0.116][-0.021, +0.116]<br>No detectable differenceSfixed=0.072S_{\mathrm{fixed}} = 0.072<br>SLLM=0.067S_{\mathrm{LLM}} = 0.067<br>A=−0.005A = -0.005<br>[−0.058,+0.050][-0.058, +0.050]<br>No detectable difference
Moralized vs RiskSfixed=0.243S_{\mathrm{fixed}} = 0.243<br>SLLM=0.147S_{\mathrm{LLM}} = 0.147<br>A=−0.097A = -0.097<br>[−0.186,−0.012][-0.186, -0.012]<br>AttenuationSfixed=0.112S_{\mathrm{fixed}} = 0.112<br>SLLM=0.025S_{\mathrm{LLM}} = 0.025<br>A=−0.087A = -0.087<br>[−0.120,−0.040][-0.120, -0.040]<br>AttenuationSfixed=0.022S_{\mathrm{fixed}} = 0.022<br>SLLM=0.015S_{\mathrm{LLM}} = 0.015<br>A=−0.007A = -0.007<br>[−0.035,+0.057][-0.035, +0.057]<br>No detectable differenceSfixed=0.087S_{\mathrm{fixed}} = 0.087<br>SLLM=0.088S_{\mathrm{LLM}} = 0.088<br>A=+0.001A = +0.001<br>[−0.045,+0.053][-0.045, +0.053]<br>No detectable difference
Memory representation
Table-Raw vs Narrative-RawSfixed=0.046S_{\mathrm{fixed}} = 0.046<br>SLLM=0.062S_{\mathrm{LLM}} = 0.062<br>A=+0.016A = +0.016<br>[−0.002,+0.029][-0.002, +0.029]<br>No detectable differenceSfixed=0.025S_{\mathrm{fixed}} = 0.025<br>SLLM=0.002S_{\mathrm{LLM}} = 0.002<br>A=−0.023A = -0.023<br>[−0.043,−0.011][-0.043, -0.011]<br>AttenuationSfixed=0.027S_{\mathrm{fixed}} = 0.027<br>SLLM=0.090S_{\mathrm{LLM}} = 0.090<br>A=+0.063A = +0.063<br>[−0.001,+0.128][-0.001, +0.128]<br>No detectable differenceSfixed=0.019S_{\mathrm{fixed}} = 0.019<br>SLLM=0.005S_{\mathrm{LLM}} = 0.005<br>A=−0.014A = -0.014<br>[−0.028,+0.004][-0.028, +0.004]<br>No detectable difference
Table-Raw vs Table+StatsSfixed=0.021S_{\mathrm{fixed}} = 0.021<br>SLLM=0.013S_{\mathrm{LLM}} = 0.013<br>A=−0.007A = -0.007<br>[−0.025,+0.008][-0.025, +0.008]<br>No detectable differenceSfixed=0.048S_{\mathrm{fixed}} = 0.048<br>SLLM=0.027S_{\mathrm{LLM}} = 0.027<br>A=−0.021A = -0.021<br>[−0.068,−0.003][-0.068, -0.003]<br>AttenuationSfixed=0.025S_{\mathrm{fixed}} = 0.025<br>SLLM=0.107S_{\mathrm{LLM}} = 0.107<br>A=+0.082A = +0.082<br>[+0.012,+0.147][+0.012, +0.147]<br>AmplificationSfixed=0.010S_{\mathrm{fixed}} = 0.010<br>SLLM=0.010S_{\mathrm{LLM}} = 0.010<br>A=+0.000A = +0.000<br>[−0.017,+0.016][-0.017, +0.016]<br>No detectable difference
Table-Raw vs Narrative+StatsSfixed=0.030S_{\mathrm{fixed}} = 0.030<br>SLLM=0.043S_{\mathrm{LLM}} = 0.043<br>A=+0.013A = +0.013<br>[−0.007,+0.030][-0.007, +0.030]<br>No detectable differenceSfixed=0.089S_{\mathrm{fixed}} = 0.089<br>SLLM=0.025S_{\mathrm{LLM}} = 0.025<br>A=−0.064A = -0.064<br>[−0.088,−0.043][-0.088, -0.043]<br>AttenuationSfixed=0.018S_{\mathrm{fixed}} = 0.018<br>SLLM=0.138S_{\mathrm{LLM}} = 0.138<br>A=+0.121A = +0.121<br>[+0.032,+0.200][+0.032, +0.200]<br>AmplificationSfixed=0.012S_{\mathrm{fixed}} = 0.012<br>SLLM=0.005S_{\mathrm{LLM}} = 0.005<br>A=−0.007A = -0.007<br>[−0.018,+0.010][-0.018, +0.010]<br>No detectable difference
Narrative-Raw vs Table+StatsSfixed=0.052S_{\mathrm{fixed}} = 0.052<br>SLLM=0.048S_{\mathrm{LLM}} = 0.048<br>A=−0.003A = -0.003<br>[−0.021,+0.013][-0.021, +0.013]<br>No detectable differenceSfixed=0.064S_{\mathrm{fixed}} = 0.064<br>SLLM=0.025S_{\mathrm{LLM}} = 0.025<br>A=−0.039A = -0.039<br>[−0.082,−0.013][-0.082, -0.013]<br>AttenuationSfixed=0.023S_{\mathrm{fixed}} = 0.023<br>SLLM=0.017S_{\mathrm{LLM}} = 0.017<br>A=−0.007A = -0.007<br>[−0.033,+0.080][-0.033, +0.080]<br>No detectable differenceSfixed=0.016S_{\mathrm{fixed}} = 0.016<br>SLLM=0.005S_{\mathrm{LLM}} = 0.005<br>A=−0.011A = -0.011<br>[−0.023,−0.005][-0.023, -0.005]<br>Attenuation
Narrative-Raw vs Narrative+StatsSfixed=0.037S_{\mathrm{fixed}} = 0.037<br>SLLM=0.018S_{\mathrm{LLM}} = 0.018<br>A=−0.019A = -0.019<br>[−0.042,+0.004][-0.042, +0.004]<br>No detectable differenceSfixed=0.114S_{\mathrm{fixed}} = 0.114<br>SLLM=0.023S_{\mathrm{LLM}} = 0.023<br>A=−0.091A = -0.091<br>[−0.113,−0.068][-0.113, -0.068]<br>AttenuationSfixed=0.021S_{\mathrm{fixed}} = 0.021<br>SLLM=0.048S_{\mathrm{LLM}} = 0.048<br>A=+0.028A = +0.028<br>[−0.027,+0.134][-0.027, +0.134]<br>No detectable differenceSfixed=0.016S_{\mathrm{fixed}} = 0.016<br>SLLM=0.000S_{\mathrm{LLM}} = 0.000<br>A=−0.016A = -0.016<br>[−0.023,−0.002][-0.023, -0.002]<br>Attenuation
Table+Stats vs Narrative+StatsSfixed=0.014S_{\mathrm{fixed}} = 0.014<br>SLLM=0.030S_{\mathrm{LLM}} = 0.030<br>A=+0.016A = +0.016<br>[−0.009,+0.028][-0.009, +0.028]<br>No detectable differenceSfixed=0.050S_{\mathrm{fixed}} = 0.050<br>SLLM=0.002S_{\mathrm{LLM}} = 0.002<br>A=−0.048A = -0.048<br>[−0.081,−0.025][-0.081, -0.025]<br>AttenuationSfixed=0.011S_{\mathrm{fixed}} = 0.011<br>SLLM=0.032S_{\mathrm{LLM}} = 0.032<br>A=+0.021A = +0.021<br>[−0.023,+0.099][-0.023, +0.099]<br>No detectable differenceSfixed=0.005S_{\mathrm{fixed}} = 0.005<br>SLLM=0.005S_{\mathrm{LLM}} = 0.005<br>A=−0.000A = -0.000<br>[−0.012,+0.007][-0.012, +0.007]<br>No detectable difference

Table 6. Interaction amplification across prespecified representation contrasts in the Prisoner’s Dilemma experiment.

GPT-5.2Claude Haiku 4.5Gemini 2.5 FlashDeepSeek V4 Flash
Paired behavioral agreement
Action agreement0.997<br>[0.996, 0.998]0.936<br>[0.932, 0.941]0.850<br>[0.839, 0.860]0.663<br>[0.649, 0.677]
Target agreement0.551<br>[0.541, 0.560]0.593<br>[0.585, 0.601]0.610<br>[0.602, 0.618]0.144<br>[0.137, 0.151]
Behavioral sensitivity
Cross-group targeting (pp)+5.15***<br>[4.22, 6.07]−0.93*<br>[−1.62, −0.24]+2.31***<br>[1.49, 3.14]−7.35***<br>[−8.98, −5.72]
Collective network sensitivity at round 15
Stance assortativity−0.194***<br>[−0.220, −0.167]−0.002<br>[−0.015, 0.010]−0.035***<br>[−0.050, −0.019]−0.023<br>[−0.068, 0.022]
Same-group edge ratio−0.051***<br>[−0.061, −0.041]+0.010*<br>[0.003, 0.017]−0.023***<br>[−0.031, −0.014]+0.072***<br>[0.057, 0.087]

Table 7. Representation sensitivity in the echo-chamber simulation. Results compare PROSE with BULLET persona representations across N=30N = 30 paired seeds. Differences are PROSE-BULLET.

Note: Brackets report 95% confidence intervals. Cross-group targeting is averaged over rounds 2–15 and reported in percentage points; collective network outcomes are cumulative through round 15. Significance markers use Holm-corrected paired Wilcoxon pp-values: ∗∗∗p<.001p < .001, ∗∗p<.01p < .01, ∗p<.05p < .05. Agreement measures are descriptive paired-run measures and are reported with 95% confidence intervals.

MetricGPT-5.2Claude Haiku 4.5Gemini 2.5 FlashDeepSeek V4 Flash
Activation probability: 0.5 − 0.3
Stance assortativity+0.068***<br>[0.041, 0.095]+0.103***<br>[0.085, 0.121]+0.108***<br>[0.089, 0.127]+0.107***<br>[0.080, 0.134]
Same-group edge ratio+0.032***<br>[0.019, 0.045]+0.060***<br>[0.048, 0.072]+0.061***<br>[0.050, 0.073]+0.070***<br>[0.051, 0.089]
Memory window: 4 − 2 rounds
Stance assortativity−0.009<br>[−0.035, 0.018]+0.063***<br>[0.049, 0.076]+0.080***<br>[0.065, 0.094]+0.092***<br>[0.057, 0.127]
Same-group edge ratio−0.011<br>[−0.021, −0.001]+0.039***<br>[0.031, 0.048]+0.038***<br>[0.028, 0.048]+0.058***<br>[0.040, 0.075]
Recommendation feed size: 10 − 5 posts
Stance assortativity+0.001<br>[−0.018, 0.021]+0.010<br>[−0.002, 0.021]+0.015*<br>[0.003, 0.026]+0.008<br>[−0.024, 0.041]
Same-group edge ratio+0.006<br>[−0.003, 0.015]+0.012**<br>[0.005, 0.018]+0.009**<br>[0.002, 0.016]+0.006<br>[−0.009, 0.020]

Table 8. Design sensitivity in the echo-chamber simulation. Signed differences compare the higher condition with the lower condition across N=30N = 30 paired seeds. Outcomes are cumulative through round 15.

Note: Brackets report unadjusted 95% confidence intervals for paired mean differences. Stars follow the multiplicity-adjusted paired tests described in Appendix C: ∗∗∗p<.001p < .001, ∗∗p<.01p < .01, ∗p<.05p < .05. An interval excluding zero need not imply significance after adjustment. Absolute differences correspond to Table 1.

Effects of network homophily and hub assignment on echo-chamber outcomes

Figure 8. Effects of network homophily and hub assignment on echo-chamber outcomes. Left panels vary initial stance homophily; right panels vary hub assignment while holding the degree sequence fixed. Boxplots show run-level stance assortativity and weighted same-group edge ratio. Significant pairwise differences are based on paired t-tests with Holm correction (p<0.05p < 0.05).

Note. Each model cell reports SfixedS_{\mathrm{fixed}}, SLLMS_{\mathrm{LLM}}, AA [95% CI], and the CI-based pattern. SfixedS_{\mathrm{fixed}} is the mean across four fixed opponents of the absolute opponent-specific cooperation-rate effect; SLLMS_{\mathrm{LLM}} is the absolute effect under reciprocal LLM–LLM interaction; and A=SLLM−SfixedA = S_{\mathrm{LLM}} - S_{\mathrm{fixed}}. CIs use 30 paired seeds and 10,000 replicates; they are pointwise and not adjusted for multiple comparisons.

Figure 8 shows that collective interaction structure depends on initial network homophily and on which stance group occupies hub positions, even when hub reassignment preserves the degree sequence. These context-relevant dependencies qualify the structural conditions under which an echo-chamber claim holds, rather than indicating sensitivity to representation alone.

The Social Simulation Audit Card

TRAILS identifies what can be varied in an audit. Interpreting these perturbations requires an additional, claim-specific judgment: context relevance. A dimension is context-relevant when it has a meaningful human or theoretical counterpart for the modeled phenomenon, as supported by human studies or established social-science theory. Sensitivity along this dimension identifies conditions under which the claim holds. In contrast, sensitivity along a context-irrelevant dimension reveals implementation dependence. The required audit strength is also shaped by the claim’s evidentiary role, the simulation’s openness, and the stakes of its intended use (see Appendix § 9). We introduce the Social Simulation Audit Card to operationalize TRAILS for a stated scientific claim. The following five steps connect these requirements to an auditable claim.

State the claim. Record the scientific conclusion the simulation supports, its evidentiary role, and its intended scope of generalization. Define the property that must be preserved, such as the presence of a phenomenon, an effect direction, or a quantitative effect within a stated tolerance.

Design sensitivity in interaction targeting

Figure 6. Design sensitivity in interaction targeting. Panels report round-wise cross-group targeting differences for (a) activation probability (0.5 − 0.3), (b) memory window (4 − 2 rounds), and (c) recommendation feed size (10 − 5 posts), across four models and N=30N = 30 paired seeds. Models are GPT-5.2 (pink circles), Claude Haiku 4.5 (blue squares), Gemini 2.5 Flash (orange triangles), and DeepSeek V4 Flash (teal diamonds). Differences are higher minus lower condition, in percentage points; negative values indicate less cross-group targeting under the higher condition. Targeting is measured within each round, not cumulatively. Shaded bands are pointwise 95% paired-percentile-bootstrap confidence intervals from 10,000 seed resamples, not simultaneous or multiplicity-adjusted intervals.

Design sensitivity in collective interaction structure

Figure 7. Design sensitivity in collective interaction structure. Panels use the same contrasts, models, and paired seeds as Figure 6, and report signed differences in weighted stance assortativity computed from interaction edges accumulated through each round. Positive values indicate higher assortativity under the higher condition. Bands are pointwise 95% paired-percentile-bootstrap confidence intervals from 10,000 seed resamples. Round 1 is omitted because assortativity is undefined without eligible interaction edges. These trajectories describe collective sensitivity rather than a causal mediation or interaction-amplification estimate.

Map TRAILS and context relevance. For each perturbation, identify whether it changes the simulation design (TRAILS-D) or its representation (TRAILS-R), and whether the changed dimension is context-relevant or context-irrelevant. Explain each assignment and record any uncertainty.

Choose and register perturbations. Select perturbations spanning a plausible range for the claim. For each, register the baseline and alternatives, factors held constant, models, and number of independent runs. Identify unaudited dimensions and define the claim-preservation criterion, tolerance, and uncertainty requirements before examining results.

Report sensitivity results. Report effect sizes, uncertainty, and whether the conclusion meets the claim-preservation criterion. Sensitivity to context-relevant perturbations may reflect meaningful social variation. Report the range of conditions under which the claim holds and use this range to define its scope. Substantial changes or reversals under context-irrelevant perturbations indicate imimplementation dependence. Interpret these results against the specified claim-preservation criterion; statistical significance alone does not determine robustness.

Scope the claim according to audit outcomes. Classify the claim into one of four outcomes. A claim is Robust when it meets the preservation criterion across all audited perturbations with adequate coverage and precision; Scope-limited when it holds only within part of the context-relevant conditions tested; Implementation-sensitive when context-irrelevant choices, such as persona formatting or memory encoding, substantially change the conclusion; or Inconclusive when coverage or precision is insufficient to assess robustness. Report the range supporting a scope-limited claim and bound every verdict by the audited dimensions, configurations, models, and interaction horizons.

Audit sufficiency therefore depends on coverage of the dimensions consequential to the claim and the precision required by its decision rule. Every verdict should state the tested range, model coverage, and remaining uncertainty. Generalization to human behavior additionally requires external validation.

Auditing Scientific Claims in the Two Case Studies

We applied the Social Simulation Audit Card to claims about cooperation and echo-chamber formation. Both experiments used 30 independent runs per condition, with shared seeds for paired comparisons. The audits below summarize the GPT-5.2 results; their verdicts are specific to the reported model, interaction conditions, and outcomes. Table 9 maps the reported perturbations to their audit interpretations.

Table 9: Audited perturbations and claim-level interpretations for GPT-5.2. PD reports the maximum absolute cooperation-rate difference across contrasts, in percentage points (pp). EC reports absolute differences in stance assortativity (rr) and weighted same-group edge ratio (HH). Stars indicate p<.001p < .001; unmarked estimates were not statistically significant.

PerturbationTRAILSContext relevanceOutcome sensitivityInterpretation
Repeated Prisoner’s Dilemma
Persona formatTRAILS-RIrrelevant75.2∗∗∗75.2^{***} ppCooperation depended strongly on persona formatting.
Game framingTRAILS-RRelevant19.3∗∗∗19.3^{***} ppFraming conditions should qualify the claim.
Memory representationTRAILS-RIrrelevant6.2∗∗∗6.2^{***} ppSmaller but significant change; assess claim preservation against the tolerance.
Echo-chamber simulation
Persona representationTRAILS-RIrrelevant∣Δr∣=0.194∗∗∗\lvert\Delta r\rvert= 0.194^{***}<br>∣ΔH∣=0.051∗∗∗\lvert\Delta H\rvert= 0.051^{***}Estimated collective structure depended on persona representation.
Activation probabilityTRAILS-DIrrelevant∣Δr∣=0.068∗∗∗\lvert\Delta r\rvert= 0.068^{***}<br>∣ΔH∣=0.032∗∗∗\lvert\Delta H\rvert= 0.032^{***}Significant changes in both outcomes; assess claim preservation against the tolerance.
Memory windowTRAILS-DIrrelevant∣Δr∣=0.009\lvert\Delta r\rvert= 0.009<br>∣ΔH∣=0.011\lvert\Delta H\rvert= 0.011Small estimated differences; claim preservation requires a tolerance-based assessment.
Recommendation feed sizeTRAILS-DIrrelevant∣Δr∣=0.001\lvert\Delta r\rvert= 0.001<br>∣ΔH∣=0.006\lvert\Delta H\rvert= 0.006Small estimated differences; claim preservation requires a tolerance-based assessment.

Table 9.

Repeated Prisoner’s Dilemma. We audited the claim that agents exhibited stable cooperation in the two-agent, 10-round repeated Prisoner’s Dilemma. Changing the matched persona from PLAIN to DESCRIPTIVE reduced cooperation from 98.2% to 23.0%, a decrease of 75.2 percentage points (95% CI [62.0, 88.3]), and reduced average payoff by 1.48 points per round. Because persona format was a context-irrelevant TRAILS-R perturbation, this result supported an Implementation-sensitive verdict. Game framing was a context-relevant TRAILS-R perturbation with a maximum cooperation difference of 19.3 percentage points, indicating that framing conditions should qualify the claim. Memory representation was context-irrelevant and produced a smaller but significant maximum difference of 6.2 percentage points; whether this difference preserved the claim depends on

the specified tolerance. These findings bound the verdict to the audited interaction regime and representation choices.

Echo-chamber simulation. We audited the claim that the simulation produced echo-chamber-like interaction structure among 100 agents over 15 rounds with fixed stances. Persona representation (BULLET versus PROSE) was a context-irrelevant TRAILS-R perturbation. Under PROSE, cross-group targeting increased by 5.15 percentage points (95% CI [4.22, 6.07]), and cumulative stance assortativity was 0.194 lower by round 15; the absolute difference in weighted same-group edge ratio was 0.051. These results showed implementation dependence in the measured strength of echo-chamber structure. Activation probability, memory window, and recommendation feed size were treated as context-irrelevant TRAILS-D perturbations for this claim. Activation probability significantly changed both collective outcomes, whereas memory window and recommendation feed size produced smaller, non-significant differences. For the original phenomenon-presence claim, the verdict remained Inconclusive: changes in outcome magnitude did not establish whether conditions crossed a specified criterion for echo-chamber presence.

Simulation Prompts

Prisoner’s Dilemma Prompts

Base Persona

Across the Prisoner’s Dilemma prompt-perturbation experiments reported in the paper, we use the following strategic persona:

You are a strategic participant focused on maximizing your long-term payoff. Study
opponent behavior for exploitable patterns, and adjust your strategy to gain advantage
while staying competitive.

Base Single-Agent Prompt Template

In the single-agent setting, the model plays against a fixed external opponent policy. The opponent policy is not revealed to the model.

Persona:
[BASE PERSONA]
Game setup:
- Repeated Prisoner’s Dilemma for [HORIZON] rounds.
- Each round, both players simultaneously choose one action: Cooperate or Defect.
- You cannot observe the opponent’s current-round action before choosing.
- Opponent follows a fixed external policy, unknown to you.

Payoffs:
- If both choose Cooperate: (3, 3).
- If you choose Cooperate and opponent chooses Defect: (0, 5).
- If you choose Defect and opponent chooses Cooperate: (5, 0).
- If both choose Defect: (1, 1).

Task:
Choose a reasonable action this round based on history and repeated-game context.

[HISTORY / MEMORY BLOCK]

Output format:
Return valid JSON with keys:
- action: one of ["Cooperate", "Defect"]
- reason: short string

Base Two-Agent Prompt Template

In the two-agent setting, both agents are model-driven players. Each agent receives the same prompt structure but with role-specific history.

Persona:
[STRATEGIC PERSONA]

Role:
Your role label this run: Agent [A/B].

Game setup:
- Repeated Prisoner’s Dilemma for [HORIZON] rounds.
- Each round, both agents simultaneously choose one action: Cooperate or Defect.
- You cannot observe the other agent’s current-round action before choosing.
- The other agent is another model-driven player with the same prompt framing.

Payoffs:
- If both choose Cooperate: (3, 3).
- If you choose Cooperate and the other agent chooses Defect: (0, 5).
- If you choose Defect and the other agent chooses Cooperate: (5, 0).
- If both choose Defect: (1, 1).

Task:
Choose a reasonable action this round based on history and repeated-game context.

[ROLE-SPECIFIC HISTORY / MEMORY BLOCK]

Output format:
Return valid JSON with keys:
- action: one of ["Cooperate", "Defect"]
- reason: short string

Output Schema and Parsing Instruction

All Prisoner’s Dilemma prompts end with the same JSON parsing instruction.

Output format:
Return valid JSON with keys:
- action: one of ["Cooperate", "Defect"]
- reason: short string

If a model response cannot be parsed as valid JSON, the parser attempts to extract a JSON object from the raw text. If parsing still fails, the fallback action is Defect.

Echo-Chamber Prompts

Agent Persona Template

Each echo-chamber agent is initialized with a minimal persona containing an agent ID, a short bio, a follower count, and a fixed stance toward AI regulation. Stance is represented on a five-point scale from strong opposition to strong support:

si∈{−2,−1,0,1,2}.s_i \in\{-2,-1,0,1,2\}.
{
  "agent_id": "[AGENT ID]",
  "bio": "[SHORT BIO]",
  "follower_count": [FOLLOWER COUNT],
  "ai_regulation_stance": [-2|-1|0|1|2]
}

The stance scale is:

-2 = strongly oppose stronger AI regulation
-1 = somewhat oppose stronger AI regulation
 0 = mixed or undecided
 1 = somewhat support stronger AI regulation
 2 = strongly support stronger AI regulation

Round Decision Prompt

At each round, an activated agent receives its own profile, the number of neighbors in the fixed network, and a feed consisting only of recent posts from neighboring agents.

You are a social media user discussing one issue: whether governments should impose
stronger regulation on advanced AI systems.

Your profile:
- agent_id: [AGENT ID]
- bio: [BIO]
- follower_count: [FOLLOWER COUNT]
- current stance on AI regulation: [STANCE] where -2 strongly oppose, -1 somewhat oppose,
0 mixed or undecided, 1 somewhat support, 2 strongly support.
- neighbor count in the network: [NEIGHBOR COUNT]

Simulation rules:
- This is a fixed network for this run.
- You only see recent posts from your neighbors.
- Keep behavior simple and realistic.
- Do not invent facts. Use short social-media-style language.
- You may choose exactly one action this round: POST, REPOST, REPLY, or SILENT.
- REPOST or REPLY should only target one visible post from the feed below.
- Your stance is fixed for the entire run and must not change.
- Set updated_stance equal to your current stance.

Round: [ROUND] / [TOTAL ROUNDS]

Visible feed from your neighbors:
[VISIBLE FEED]

Return valid JSON with exactly these keys:
{
  "action": "POST|REPOST|REPLY|SILENT",
  "target_index": integer or null,
  "message": "short text, or empty string if SILENT",
  "updated_stance": -2|-1|0|1|2,
  "reason": "brief reason"
}

If target_index is used, it must refer to one visible feed item by its 1-based index.

Visible Feed Format

When neighbor posts are visible, they are rendered as a numbered list.

[1] post_id=[POST ID] | author=[AUTHOR ID] | action_type=[POST/REPOST/REPLY] | text=[POST
TEXT]
[2] post_id=[POST ID] | author=[AUTHOR ID] | action_type=[POST/REPOST/REPLY] |
target_post_id=[TARGET POST ID] | text=[POST TEXT]
...

If the agent has no visible recent neighbor posts, the feed is rendered as:

No recent posts from your neighbors.

Output Schema and Parsing Instruction

All echo-chamber prompts require the following JSON schema.

{
  "action": "POST|REPOST|REPLY|SILENT",
  "target_index": integer or null,
  "message":  "short text, or empty string if SILENT",
  "updated_stance": -2|-1|0|1|2,
  "reason":  "brief reason"
}

The parser accepts only four actions:

ait∈{POST,REPOST,REPLY,SILENT}.a_i^t \in\{\mathrm{POST},\mathrm{REPOST},\mathrm{REPLY},\mathrm{SILENT}\}.

If REPOST or REPLY is selected, the target must refer to a visible feed item by its one-indexed position. The agent’s stance is frozen throughout the run, so updated_stance is constrained to equal the current stance:

sit+1=sit.s_i^{t+1} = s_i^t.

References

  1. [1]Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Cooperation, competition, and maliciousness: Llm-stakeholders interactive negotiation. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9798331314385.arxiv.org/abs/2309.17234
  2. [2]Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Betge, and Eric Schulz. Playing repeated games with large language models. Nature Human Behaviour, 9(7):1380–1390, 2025. doi: 10.1038/s41562-025-02172-y. URL https://doi.org/10.1038/s41562-025-02172-y.
  3. [3]Jacy Reese Anthis, Ryan Liu, Sean M Richardson, Austin C Kozlowski, Bernard Koch, Erik Brynjolfsson, James Evans, and Michael S Bernstein. Position: Llm social simulations are a promising research method. In Forty-second International Conference on Machine Learning Position Paper Track, 2025. URL https://openreview.net/forum?id=cRBg1dtj7o.arxiv.org/abs/2504.02234
  4. [4]Ariel Flint Ashery, Luca Maria Aiello, and Andrea Baronchelli. Emergent social conventions and collective bias in llm populations. Science Advances, 11(20):eadu9368, 2025. URL https://www.science.org/doi/abs/10.1126/sciadv.adu9368.
  5. [5]Francesco Bertolotti, Angela Locoro, and Luca Mari. Sensitivity to initial conditions in agent-based models. In Multi-Agent Systems and Agreement Technologies, volume 12520 of Lecture Notes in Computer Science, pp. 501–508. Springer, 2020. doi: 10.1007/978-3-030-66412-1_32. URL https://doi.org/10.1007/978-3-030-66412-1_32.
  6. [6]Ayush Chopra, Shashank Kumar, Nurullah Giray Kuru, Ramesh Raskar, and Arnau Quera-Bofarull. On the limits of agency in agent-based models. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pp. 500–509, 2025. URL https://dl.acm.org/doi/10.5555/3709347.3743565.arxiv.org/abs/2409.10568
  7. [7]Yun-Shiuan Chuang, Agam Goyal, Nikunj Harlalka, Siddharth Suresh, Robert D Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy Rogers. Simulating opinion dynamics with networks of llm-based agents. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3326–3346, 2024. URL https://aclanthology.org/2024.findings-naacl-211/.arxiv.org/abs/2311.09618
  8. [8]Alessandro Di Stefano, Chrisina Jayne, Claudio Angione, and The Anh Han. Recognition of behavioural intention in repeated games using machine learning. In Artificial Life Conference Proceedings 35, volume 2023, pp. 103. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA, 2023. doi: 10.1162/isal_a_00637. URL https://doi.org/10.1162/isal_a_00637.
  9. [9]Emilio Ferrara. The butterfly effect in artificial intelligence systems: Implications for AI bias and fairness. Machine Learning with Applications, 15:100525, 2024. URL https://doi.org/10.1016/j.mlwa.2024.100525.
  10. [10]Antonino Ferraro, Antonio Galli, Valerio La Gatta, Marco Postiglione, Gian Marco Orlando, Diego Russo, Giuseppe Riccio, Antonio Romano, and Vincenzo Moscato. Agent-based modelling meets generative ai in social network simulations. In International Conference on Advances in Social Networks Analysis and Mining, pp. 155–170, 2024. URL https://link.springer.com/chapter/10.1007/978-3-031-78541-2_10.arxiv.org/abs/2411.16031
  11. [11]Nicoló Fontana, Francesco Pierri, and Luca Maria Aiello. Nicer than humans: How do large language models behave in the prisoner’s dilemma? In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pp. 522–535, 2025. doi: 10.1609/icwsm.v19i1.35829. URL https://doi.org/10.1609/icwsm.v19i1.35829.
  12. [12]Chenhao Gu, Ling Luo, Zainab Razia Zaidi, and Shanika Karunasekera. Large language model driven agents for simulating echo chamber formation. arXiv preprint arXiv:2502.18138, 2025. URL https://arxiv.org/abs/2502.18138.
  13. [13]Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, pp. 65–70, 1979. URL https://www.jstor.org/stable/4615733.DOI
  14. [14]Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VtmBAGCN7o.arxiv.org/abs/2308.00352
  15. [16]Maik Larooij and Petter Törnberg. Validation is the central challenge for generative social simulation: A critical review of llms in agent-based modeling. Artificial Intelligence Review, 59(1):15, 2026. doi: 10.1007/s10462-025-11412-6. URL https://doi.org/10.1007/s10462-025-11412-6.
  16. [17]Ang Li, Haozhe Chen, Hongseok Namkoong, and Tianyi Peng. LLM generated persona is a promise with a catch. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems Position Paper Track, 2025. URL https://openreview.net/forum?id=qH9eGetMG4H.arxiv.org/abs/2503.16527
  17. [18]Nian Li, Chen Gao, Mingyu Li, Yong Li, and Qingmin Liao. Econagent: Large language model-empowered agents for simulating macroeconomic activities. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 15523–15536. Association for Computational Linguistics, 2024. URL https://aclanthology.org/2024.acl-long.829/.arxiv.org/abs/2310.10436
  18. [19]Genglin Liu, Vivian Le, Salman Rahman, Elisa Kreiss, Marzyeh Ghassemi, and Saadia Gabriel. Mosaic: Modeling social ai for content dissemination and regulation in multi-agent simulations. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 6390–6417. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.emnlp-main.325. URL https://aclanthology.org/2025.emnlp-main.325/.
  19. [20]Yuhan Liu, Xiuying Chen, Xiaoqing Zhang, Xing Gao, Ji Zhang, and Rui Yan. From skepticism to acceptance: Simulating the attitude dynamics toward fake news. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pp. 7886–7894. International Joint Conferences on Artificial Intelligence Organization, 2024. doi: 10.24963/ijcai.2024/873. URL https://doi.org/10.24963/ijcai.2024/873.
  20. [21]Nunzio Lorè and Babak Heydari. Strategic behavior of large language models and the role of game structure versus contextual framing. Scientific Reports, 14(1):18490, 2024. URL https://www.nature.com/articles/s41598-024-69032-z.
  21. [22]Michael W. Macy and Robert Willer. From factors to actors: Computational sociology and agent-based modeling. Annual Review of Sociology, 28(1):143–166, 2002. doi: 10.1146/annurev.soc.28.110601.141117. URL https://www.annualreviews.org/content/journals/10.1146/annurev.soc.28.110601.141117.DOI
  22. [23]Qirui Mi, Mengyue Yang, Xiangning Yu, Zhiyu Zhao, Cheng Deng, Bo An, Haifeng Zhang, Xu Chen, and Jun Wang. MF-LLM: Simulating population decision dynamics via a mean-field large language model framework. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=m0q8NfGWnv.
  23. [24]Xinyi Mou, Zhongyu Wei, and Xuan-Jing Huang. Unveiling the truth and facilitating change: Towards agent-based large-scale social movement simulation. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 4789–4809, 2024. URL https://aclanthology.org/2024.findings-acl.285/.
  24. [25]Gian Marco Orlando, Jinyi Ye, Valerio La Gatta, Mahdi Saeedi, Vincenzo Moscato, Emilio Ferrara, and Luca Luceri. Emergent coordinated behaviors in networked llm agents: Modeling the strategic dynamics of information operations. In Proceedings of the ACM Web Conference 2026, pp. 4805–4816, 2026.DOI
  25. [26]Paul Ormerod and Bridget Rosewell. Validation and verification of agent-based models in the social sciences. In Flaminio Squazzoni (ed.), Epistemological Aspects of Computer Simulation in the Social Sciences, volume 5466 of Lecture Notes in Computer Science, pp. 130–140. Springer, Berlin, Heidelberg, 2009. doi: 10.1007/978-3-642-01109-2_10. URL https://doi.org/10.1007/978-3-642-01109-2_10.
  26. [27]Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23. Association for Computing Machinery, 2023. doi: 10.1145/3586183.3606763. URL https://doi.org/10.1145/3586183.3606763.
  27. [28]Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2006–2017, Mexico City, Mexico, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-naacl.130. URL https://aclanthology.org/2024.findings-naacl.130/.DOI
  28. [29]PHF Science. The future is now: Revolutionising decision-making with ai-driven simulations. https://www.phfscience.nz/news-publications/the-future-is-now-revolutionising-decision-making-with-ai-driven-simulations/, December 2024. Accessed: 2026-05-02.
  29. [30]Jinghua Piao, Zhihong Lu, Chen Gao, Fengli Xu, Qinghua Hu, Fernando P Santos, Yong Li, and James Evans. Emergence of human-like polarization among large language model agents. arXiv preprint arXiv:2501.05171, 2025. URL https://arxiv.org/abs/2501.05171.
  30. [31]Jinghua Piao, Yuwei Yan, Jun Zhang, Nian Li, Junbo Yan, Xiaochong Lan, Zhihong Lu, Zhiheng Zheng, Jing Yi Wang, Di Zhou, Chen Gao, Fengli Xu, Fang Zhang, Ke Rong, Jun Su, and Yong Li. Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding of human behaviors and society. iFuture, 2026. doi: 10.26599/IF.2026.9710004. URL https://doi.org/10.26599/IF.2026.9710004.
  31. [32]Maximilian Puelma Touzel, Sneheel Sarangi, Gayatri Krishnakumar, Busra Tugce Gurbuz, Austin Welch, Zachary Yang, Andreea Musulan, Hao Yu, Ethan Kosak-Hine, Tom Gibbs, Camille Thibault, Reihaneh Rabbany, Jean-François Godbout, Dan Zhao, and Kellin Perline. Sandboxsocial: A sandbox for social media using multimodal ai agents. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, pp. 11100–11103. International Joint Conferences on Artificial Intelligence Organization, 2025. doi: 10.24963/ijcai.2025/1271. URL https://www.ijcai.org/proceedings/2025/1271.DOI
  32. [33]Maximilian Puelma Touzel, Sneheel Sarangi, Aurélien Bück-Kaeffer, Zachary Yang, Jean-François Godbout, and Reihaneh Rabbany. Position: Time to close the validation gap in llm social simulations. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. PMLR, 2026.
  33. [34]Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15174–15186. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.810. URL https://aclanthology.org/2024.acl-long.810/.
  34. [35]Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. Benchmarking prompt sensitivity in large language models. In European Conference on Information Retrieval, pp. 303–313. Springer, 2025. URL https://link.springer.com/chapter/10.1007/978-3-031-88714-7_29.arxiv.org/abs/2502.06065
  35. [36]Ruivang Ren, Peng Qiu, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Hua Wu, Ji-Rong Wen, and Haifeng Wang. Bases: Large-scale web search user simulation with large language model based agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 902–917, 2024a. URL https://aclanthology.org/2024.findings-emnlp.50/.arxiv.org/abs/2402.17505
  36. [37]Siyue Ren, Zhiyao Cui, Ruiqi Song, Zhen Wang, and Shuyue Hu. Emergence of social norms in generative agent societies: Principles and architecture. In Kate Larson (ed.), Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pp. 7895–7903. International Joint Conferences on Artificial Intelligence Organization, 8 2024b. doi: 10.24963/ijcai.2024/874. URL https://doi.org/10.24963/ijcai.2024/874. Human-Centred AI.
  37. [38]Abel Salinas and Fred Morstatter. The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 4629–4651, 2024. URL https://aclanthology.org/2024.findings-acl.275/.arxiv.org/abs/2401.03729
  38. [39]Andrea Saltelli, Marco Ratto, Terry Andres, Francesca Campolongo, Jessica Cariboni, Debora Gatelli, Michaela Saisana, and Stefano Tarantola. Global Sensitivity Analysis: The Primer. John Wiley & Sons, Chichester, UK, 2008. doi: 10.1002/9780470725184. URL https://doi.org/10.1002/9780470725184.
  39. [40]Robert G. Sargent. Verification and validation of simulation models. In Proceedings of the 2010 Winter Simulation Conference, pp. 166–183. IEEE, 2010. doi: 10.1109/WSC.2010.5679166. URL https://doi.org/10.1109/WSC.2010.5679166.
  40. [41]Thomas C. Schelling. Dynamic models of segregation. Journal of Mathematical Sociology, 1(2):143–186, 1971. doi: 10.1080/0022250X.1971.9989794. URL https://www.tandfonline.com/doi/abs/10.1080/0022250X.1971.9989794.DOI
  41. [42]Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/file/6c0e99d736da621403018ca7b32b1a4d-Paper-Conference.pdf.arxiv.org/abs/2310.11324
  42. [43]Uri Simonsohn, Joseph P Simmons, and Leif D Nelson. Specification curve analysis. Nature Human Behaviour, 4(11):1208–1214, 2020. URL https://www.nature.com/articles/s41562-020-0912-z.DOI
  43. [44]Sara Steegen, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5):702–712, 2016. URL https://pubmed.ncbi.nlm.nih.gov/2769465/.
  44. [45]Petter Törnberg, Diliara Valeeva, Justus Uitermark, and Christopher Bail. Simulating social media using large language models to evaluate alternative news feed algorithms. arXiv preprint arXiv:2310.05984, 2023. URL https://arxiv.org/abs/2310.05984.
  45. [46]Chenxi Wang, Zongfang Liu, Dequan Yang, and Xiuying Chen. Decoding echo chambers: Llm-powered simulations revealing polarization in social networks. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 3913–3923, 2025a. URL https://aclanthology.org/2025.coling-main.264/.arxiv.org/abs/2409.19338
  46. [47]Lei Wang, Heyang Gao, Xiaohe Bo, Xu Chen, and Ji-Rong Wen. Yulan-onesim: Towards the next generation of social simulator with large language models, 2025b. URL https://arxiv.org/abs/2505.07581.
  47. [48]Lei Wang, Jingsen Zhang, Hao Yang, Zhi-Yuan Chen, Jiakai Tang, Zeyu Zhang, Xu Chen, Yankai Lin, Hao Sun, Ruihua Song, Xin Zhao, Jun Xu, Zhicheng Dou, Jun Wang, and Ji-Rong Wen. User behavior simulation with large language model-based agents. ACM Transactions on Information Systems, 43(2):1–37, 2025c. URL https://dl.acm.org/doi/full/10.1145/3708985.arxiv.org/abs/2306.02552
  48. [49]Zhilīn Wang, Yu Ying Chiu, and Yu Cheung Chiu. Humanoid agents: Platform for simulating human-like generative agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 167–176. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-demo.15. URL https://aclanthology.org/2023.emnlp-demo.15/.DOI
  49. [50]Uri Wilensky and William Rand. Making models match: Replicating an agent-based model. Journal of Artificial Societies and Social Simulation, 10(4):2, 2007. URL https://www.jasss.org/10/4/2.html.
  50. [51]Paul Windrum, Giorgio Fagiolo, and Alessio Moneta. Empirical validation of agent-based models: Alternatives and prospects. Journal of Artificial Societies and Social Simulation, 10(2):8, 2007. URL https://ideas.repec.org/a/jas/jasssj/2006-40-2.html.
  51. [52]Yuzhe YANG, Yifei Zhang, Minghao Wu, Kaidi Zhang, Yunmiao Zhang, Honghai Yu, Yan Hu, and Benyou Wang. Twinmarket: A scalable behavioral and social simulation for financial markets. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=h60y6zlPyI.arxiv.org/abs/2502.01506
  52. [53]Ziyi Yang, Zaibin Zhang, Zirui Zheng, Yuxian Jiang, Ziyue Gan, Zhiyu Wang, Zijian Ling, Jinsong Chen, Martz Ma, Bowen Dong, Prateek Gupta, Shuyue Hu, Zhenfei Yin, Guohao Li, Xu Jia, Lijun Wang, Bernard Ghanem, Huchuan Lu, Chaochao Lu, Wanli Ouyang, Yu Qiao, Philip Torr, and Jing Shao. Oasis: Open agent social interaction simulations with one million agents. arXiv preprint arXiv:2411.11581, 2024. URL https://arxiv.org/abs/2411.11581.
  53. [54]Jinyi Ye, Lei Cao, Ding Chen, and Emilio Ferrara. Stop drawing scientific claims from llm social simulations without robustness audits. In NeurIPS 2026 Workshop on Large Language Models for Social Reasoning and Simulation (SocialAgent), 2026. URL https://arxiv.org/abs/2605.18890.
  54. [55]Chenxiao Yu, Jinyi Ye, Yuangang Li, Zheng Li, Emilio Ferrara, Xiyang Hu, and Yue Zhao. A large-scale simulation on large language models for decision-making in political science. arXiv preprint arXiv:2412.15291, 2024.arxiv.org/abs/2412.15291
  55. [56]Jiaxu Zhou, Jen-tse Huang, Xuhui Zhou, Man Ho Lam, Xintao Wang, Hao Zhu, Wenxuan Wang, and Maarten Sap. The pimmur principles: Ensuring validity in collective behavior of llm societies. arXiv preprint arXiv:2509.18052, 2025. URL https://arxiv.org/abs/2509.18052.

Paper details

vlnc:W_jcp2vjx4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents