WorksComputer Science
Wrong Together: Shared Bias in User Simulation
Abstract
User simulators generate responses in place of human participants, allowing researchers to estimate the result of a study. To use these estimates before collecting human responses, researchers need a way to assess their reliability. We test whether disagreement among simulators can identify unreliable estimates of a group's response proportions and of differences between groups. We evaluate this signal using 6 language-model simulators and existing GlobalOpinionQA responses for 60 questions in 12 countries. Disagreement predicts invalidity more accurately for differences between two groups than for a single group's response proportions (AUC 0.87 versus 0.71). Errors shared by the simulators remain invisible to this signal. When such an error affects both groups equally, it cancels in the comparison, helping explain the stronger result for group differences. We test this explanation by estimating the shared bias using one set of groups and subtracting it from predictions for a separate set. This correction reduces prediction error without changing disagreement among the simulators. In a retrospective analysis, we simulate referring the 20% of group comparisons with the greatest disagreement for human assessment. Among the remaining comparisons with a conclusive validity judgment, 54.8% are judged invalid, compared with 63.7% before referral.
Introduction
Researchers use language models to generate responses on behalf of study participants and estimate how a population would respond. This silicon-sampling approach [1, 18] can reduce the work of collecting human responses, provided the simulated estimates support the conclusions researchers would draw from human data. We focus on survey respondents, complementing work on simulated users of search, recommender, and dialogue systems [23, 5, 22]. A survey study may estimate the proportion choosing an option in one population, which we call a level, or the difference in that proportion between two populations, a contrast.
Existing evaluations show why these estimates need checking. Simulators for agentic tasks are more cooperative and more uniform in style than human participants, overstating measured agent performance [23]. Similar differences appear in purchase decisions, where simulated shoppers reproduce buyers’ decisions more closely than non-buyers’ decisions [5]. HumanStudy-Bench assesses simulator fidelity at the study level by comparing simulated estimates and effect sizes with reconstructed human studies while accounting for sampling uncertainty [15]. Each comparison uses an available human reference to measure simulator error.
For a new study, researchers may need to decide which simulated estimates require human assessment before that reference is available. Disagreement among several simulators could identify unreliable estimates, although agreement can conceal an error shared by all models. Can simulator disagreement predict invalid estimates, and does its usefulness differ between levels and contrasts?
We examine this question using 60 GlobalOpinionQA questions [6], 12 countries, and 6 simulators. Disagreement predicts invalidity more accurately for contrasts than for levels. The error decomposition identifies a component shared across simulators and constant across countries, which cancels when two countries are compared.
We define invalidity using an equivalence margin and uncertainty from both human samples and simulator draws (§ 3). We test the shared-error explanation by estimating a question-specific bias on calibration countries and correcting level estimates on separate test countries (§ 5.2). This correction uses human calibration data, while the disagreement score itself uses simulator outputs alone. We then evaluate whether referring high-disagreement contrasts reduces invalidity among the estimates retained for simulation (§ 5.3). All evaluations use existing survey responses.
Related Work
User simulators support the training and evaluation of dialogue systems and recommenders, using statistical user models [19], configurable probabilistic simulators (Ie et al., 2019), and prompted language models [2]. Evaluating these simulators requires comparing their behavior with that of human users. Such comparisons find greater cooperation and less stylistic variation among simulators [23], a disengagement deficit across model families [5], and a need to align simulated responses with retail users [17]. In survey simulation, the corresponding comparison concerns response distributions. Persona-conditioned LLMs approximate human response distributions with mixed success [1, 18, 3], representing some populations more closely than others [18, 6]. GlobalOpinionQA provides the response distributions we use to examine whether several simulators share these errors across countries.
For studies using simulated participants, the relevant question extends beyond the realism of individual responses to the accuracy of the quantities estimated from them. HumanStudy-Bench evaluates estimands and effect sizes against reconstructed human studies, accounting for human sampling uncertainty [15]. Response realism and treatment-effect accuracy can also diverge [14]. The distinction between response realism and effect accuracy motivates evaluating levels and contrasts separately. We examine whether simulator outputs identify invalid estimates for each, without using the target populations’ human responses.
Predicting invalidity without the target human responses requires a signal from the simulators themselves. Ensemble disagreement is an established uncertainty signal [13], and Gorbett and Jana [9] use cross-model disagreement to detect errors without labels in question answering. Using such a signal requires separating agreement from validity, a distinction also made in work on LLM judges [4]. We study the corresponding problem for simulated population estimates: models may agree because they share an error, while a between-group difference may cancel part of that error. We use an equivalence test to define acceptable error [12] and selective prediction to refer high-disagreement estimates for human assessment [7].
The referral rule and any correction of simulated estimates must themselves be checked against human evidence. Hullman et al. [10] argue that simulator corrections need these checks to support confirmatory research, while Persson et al. [16] caution that validation on past studies cannot certify performance on new ones. Our tests therefore separate countries and questions within existing surveys and limit the resulting conclusions to that setting.
Problem and Method
Our primary estimates concern response proportions. A level is the proportion choosing a particular option in a population. A contrast subtracts the corresponding proportions in two populations. Let index one such quantity, or estimand, with human population value and simulator value . We report the mean of simulators, , and assess its error:
We observe finite-sample estimates and . For each estimand, the validity decision concerns the ensemble estimate. We ask whether variation among the simulator estimates predicts that decision without using the target human responses .
Determining whether an estimate meets an error margin requires accounting for sampling uncertainty. For example, an error of 0.07 is more precisely determined when the human standard error is 0.01 than when it is 0.12. We define validity by , where is a scientifically meaningful equivalence margin. We fix the margin independently of the confidence interval. Setting it from the interval width would make a less precise human reference easier to match.
For an estimated error with standard error SE, we use a conservative 95% confidence-interval rule (). Comparing the bounds SE with the margin gives three decisions:
supported: the interval lies inside the equivalence region ( SE );
contradicted: it lies outside ( SE );
unresolved: the interval straddles .
The standard error in these decisions combines the human sampling error and the simulator Monte-Carlo error from limited draws, , treated as independent. For a proportion, and over the simulators with draws each. A contrast adds the two populations’ variances. The unresolved category retains cases where these samples do not establish whether the margin is met.
To predict these validity decisions, we measure cross-simulator disagreement as the mean absolute deviation of the simulator estimates. Within-simulator uncertainty is over repeated stochastic simulation . These measures use simulator outputs only. For the secondary estimands defined in § 4, we use the fraction disagreeing with the plurality vote for direction estimates and mean pairwise total-variation distance for full distributions. Continuous estimands use mean absolute deviation throughout.
We use disagreement to select estimates for human assessment. For a threshold , we retain the simulated estimate when and otherwise refer it for human assessment. Coverage is the fraction of estimates retained. Risk is the invalidity rate among resolved, retained estimates, with unresolved estimates reported separately. We evaluate both the ranking and the referral rule retrospectively against existing human responses. Ranking performance is measured by AUC, the frequency with which an invalid estimate receives greater disagreement than a valid estimate among resolved cases.
This referral rule uses disagreement directly, without fitting a predictor to human responses. To test the explanation for the level–contrast difference, a separate analysis estimates shared error using human responses from calibration countries and applies the correction to test countries. Appendix § 9 uses the same separation of countries to fit and evaluate probabilities of invalidity for a calibrated referral rule.
Experimental Setup
We use GlobalOpinionQA [6], which records human response distributions from the World Values Survey and Pew Global Attitudes surveys. We sample 60 questions covering 12 of the best-covered countries: Russia, Mexico, Germany, Poland, Turkey, Pakistan, Jordan, Kenya, France, Lebanon, the United States, and Japan. Each question–country response distribution supplies one primary level estimand per eligible option. Contrasts compare the same option between two countries. Table 1 reports the number of estimates in each analysis.
| Analysis | Quantity | Count |
| Study | Questions | 60 |
| Populations | 12 | |
| Simulators | 6 | |
| Model families | 4 | |
| Primary: proportions | Option-level estimates | 2616 |
| Pairwise contrasts | 14388 | |
| Secondary: ordinal | Direction | 720 |
| Mean | 720 | |
| Ordinal contrasts | 660 | |
| Full distributions | 720 |
Table 1. Study scale and numbers of evaluated estimates. Primary estimates use response proportions. Secondary analyses use questions with ordered response options.
These surveys report weighted proportions without per-question respondent counts. We approximate the human standard error using each country’s survey-design sample size and a multinomial distribution. Uncertainty from limited simulator draws is also material: median , compared with . Including the simulator term moves about a fifth of the contrast estimates that would otherwise be resolved to unresolved.
We run 6 simulators from four model families: gemma4–31b, mistral–small, gpt–4o–mini, qwen3–next–80b, gpt–4o, and mistral–large. Each is prompted to respond as a randomly sampled person from the target country. Repeated stochastic draws provide its response distribution and within-model spread . We also evaluate one model per family to assess the effect of including multiple variants from a family.
For these data, each primary evaluation concerns one ensemble estimate per question, country, and option for levels, or per question, country pair, and option for contrasts. We weight each option by , where is the question’s number of eligible options, so each question has equal total weight. Inference is clustered by question because its response proportions and country contrasts are dependent. Table 5 separately evaluates individual simulator–estimand pairs.
| Estimand family | invalidity | mean | |
| direction | 3570 | 0.391 | 0.415 |
| ordinal contrast | 3251 | 0.507 | 0.444 |
| mean | 3824 | 0.651 | 0.496 |
| distribution | 4157 | 0.942 | 0.488 |
Table 5. Estimand difficulty: invalidity and mean error by family ( over resolved simulator–estimand pairs).
We apply an equivalence margin of 0.10 to these primary proportions and contrasts, corresponding to a ten-percentage-point error. Margins are fixed before the human comparison and independently of sampling uncertainty. We report invalidity among resolved estimates and the unresolved share separately.
We extend the comparison to ordered response options through four secondary estimands: direction (the modal non-refusal option), mean (the mean response index), ordinal contrast (the difference in mean index from the United States), and distribution (total-variation distance between response distributions). For these secondary analyses, we use margins of 0.30 response-index units for means and ordinal contrasts, 0.15 total-variation distance for distributions, and exact modal agreement for direction. The index scales span about four units. Appendix § 8 reports these analyses and their sensitivity checks.
We evaluate the primary signal over all pairwise contrasts, on 24 separate test questions, and on contrasts restricted to test countries. For analyses requiring human calibration, we split the 12 countries into 7 calibration countries and 5 test countries . Correction offsets and learned probability mappings use only calibration-country human responses. Prompts, temperatures, model identifiers, draw counts, and parsing procedures will be released with the code.
Results
Error prediction for levels and contrasts
Disagreement distinguishes invalid from valid ensemble estimates more accurately for contrasts than for levels. AUC is 0.87 (95% CI [0.85, 0.90]) for contrasts and 0.71 ([0.67, 0.75]) for levels. The paired difference is 0.16 (question-clustered CI [0.13, 0.19]). The contrast result also appears on 24 separate test questions. Across 3094 corresponding contrast estimands, AUC is 0.86. Excluding the United States from both sides of a contrast gives AUC 0.87 across 11990 contrasts.
These test questions retain a level–contrast AUC difference of 0.14 (Table 2). Restricting the country pairs to the 5 test populations gives level AUC 0.72 and contrast AUC 0.88 (). Both checks use the same disagreement score without refitting.
| Evaluation | Absolute level | Contrast | Difference |
| All pairwise contrasts | 0.71 [0.67, 0.75] | 0.87 [0.85, 0.90] | +0.16 [0.13, 0.19] |
| Test-question AUC | 0.73 | 0.86 | +0.14 |
| Test-population pairs only | 0.72 | 0.88 | +0.16 |
| Reference rotated (non-US) | — | 0.87 | |
| Invalidity, full coverage | — | 63.7% | |
| Invalidity, 80% coverage | — | 54.8% |
Table 2. Level and contrast AUC on response-option proportions. Prediction is evaluated on all pairs, test questions, and test-population pairs. Referral retains the lowest-disagreement 80% of contrasts. AUC is question-weighted, with question-clustered 95% confidence intervals. Validity uses a fixed equivalence margin.
Disagreement and other uncertainty signals To assess whether disagreement captures more than response ambiguity, we compare it with other uncertainty measures. Cross-model disagreement (AUC 0.87), pooled within-model sampling spread (0.88), and leave-one-model-out instability (0.86)
have similar predictive performance, with paired AUC differences within . Each exceeds the entropy of the ensemble response distribution (0.65) and its plurality margin (0.63). Disagreement exceeds entropy by 0.23 in AUC.
The comparable performance of sampling spread depends on pooling it over the ensemble. For five of the six individual simulators, sampling spread predicts invalidity of that simulator’s contrast estimates with AUC 0.51 to 0.58. Disagreement gives 0.75 to 0.80 on the same estimates. The cross-model result also persists when the ensemble contains one model per pretraining family, giving contrast AUC 0.88, close to the full ensemble’s 0.87.
Shared error and bias correction
To examine why contrast invalidity is more predictable, we separate shared error from simulator-specific deviations. Write with . Here is the shared error and is the simulator-specific deviation. Averaging over simulators leaves ensemble error . Disagreement is unchanged by the shared offset because mean absolute deviation depends on , while observed disagreement also includes sampling noise .
Identifying the component removed by a country contrast requires a further split across populations. For question , country , and option , write signed simulator error as . The component is shared across simulators and constant across countries. The component is shared across simulators but varies across countries, and is the simulator-specific residual. A contrast cancels and retains differences in and .
We estimate the components by empirical means over the crossed (simulator country) panel, weight options by , and resample questions for intervals. Of total error variance, 58% (95% CI [53, 65]) is shared across simulators. Of that shared component, 65% ([56, 72]) is constant across countries and cancels in a contrast (Figure 2). Cancellation leaves the country-varying shared component, which accounts for 21% of original error variance. The remaining errors give mean absolute error 0.199 for levels and 0.166 for contrasts. Cancellation reduces error while retaining a shared component that disagreement cannot measure.

Figure 2. Shared and simulator-specific error. 58% of error variance is shared across simulators, and 65% of that component is constant across countries. A contrast cancels the country-constant component. Country-varying shared error and simulator-specific error remain.
Bias correction across countries We test whether the country-constant component can be estimated from other countries by calculating a question- and option-specific offset on the 7 calibration countries: . We subtract it from level estimates for the 5 test countries. Mean absolute error falls from 0.194 to 0.129 (question-clustered 95% CI on the reduction [0.044, 0.087]; Figure 1B). Shuffling offsets across questions with the same option count raises median error to 0.246. The observed reduction exceeds that of every shuffled correction.
Figure 1. Error prediction, bias correction, and selective referral. (A) Disagreement predicts error better for contrasts than absolute levels (, question-clustered 95% CI). (B) Each point is test level error (MAE) on the test countries: before correction (0.194), after subtracting a per-question bias estimated on the 7 calibration countries (0.129), and after subtracting an offset from unrelated questions as a placebo (0.246). Only the real correction lowers the error. (C) Verdicts of the equivalence test on the contrasts that simulation still answers, as the highest-disagreement share is referred to humans. Referral shrinks the contradicted share, but about half of the answered contrasts stay unresolved at every threshold, so referral orders the human checks without certifying the rest. Among resolved contrasts, invalidity falls from 63.7% when simulation answers all of them to 54.8% when it answers the lowest-disagreement 80%.
Applying the same offset to every simulator leaves disagreement scores and their ordering unchanged. AUC against the corrected validity labels changes from 0.72 to 0.71 (difference CI [−0.05, 0.05]). This correction reduces level error without a clear change in AUC, consistent with removing an error component that disagreement does not measure. For contrasts, subtracting a country-constant offset changes nothing because it already cancels in the difference.
Correction by question The decomposition also predicts which questions should benefit most from this correction. We group questions into thirds by the share of calibration error that is constant across countries. These shares are 0.26, 0.62, and 0.86 in the low, middle, and high groups. The corresponding reductions in test-country level error are 0.019, 0.052, and 0.123. Questions with more country-constant error on calibration countries therefore benefit more from correction on test countries.
Selective referral
We use the disagreement score to rank contrasts for human assessment. At full coverage, invalidity among resolved contrasts is 63.7%. Referring the 20% with the greatest disagreement and retaining the rest lowers this rate to 54.8%, a reduction of 8.9 points (question-clustered 95% CI [7.5, 10.3]). Restricting the comparison to pairs of test countries gives a reduction of 8.8 points.
This reduction concerns resolved estimates. Across all 14388 contrasts, the equivalence test supports 20%, contradicts 32%, and leaves 48% unresolved. Among the retained 80%, 51% remain unresolved. These decision shares are unweighted, whereas invalidity rates are question-weighted and computed among resolved estimates.
Disagreement therefore determines the order of human assessment. An estimate retained by the referral rule still needs an equivalence decision, and many retained estimates remain unresolved.
Sensitivity and additional comparisons
Sampling uncertainty and equivalence margins The validity labels themselves depend on sampling uncertainty: larger within-model spread widens the interval and can change which estimates are resolved. A signal correlated with could therefore appear predictive partly because uncertainty enters the decision rule. Using human sampling uncertainty alone preserves the ordering (level AUC 0.63, contrast AUC 0.78). Scaling every margin from to also preserves it, with contrast AUC in [0.81, 0.90] (Section A).
Results by survey source The level–contrast ordering also persists when we analyze Pew Global Attitudes and the World Values Survey separately. Contrast and level AUC are 0.87 and 0.71 on Pew, and 0.87 and 0.73 on World Values. The paired differences are 0.16 (95% CI [0.13, 0.20]) and 0.14 ([0.02, 0.23]). The country-constant component accounts for 58% and 88% of shared error, respectively. The World Values interval is wider because it covers fewer questions.
One option per question Our primary analysis includes all eligible options with equal total weight per question. To check sensitivity to the set of options, we select the option the simulator ensemble chooses most often, without using human responses, and retain only that option for each question. Contrast AUC is 0.61, compared with 0.35 for levels. The ordering is preserved, although the below-chance level AUC may reflect both shared bias and selection of a modal option from simulator outputs. This dependence on option selection limits what the check establishes about the mechanism.
Ordinal estimands Extending the analysis from response proportions to ordered response indices gives contrast AUC 0.72 (95% CI [0.64, 0.79]), compared with 0.51 for absolute means and 0.47 for full distributions. The ordering persists across equivalence margins, assumed human sample sizes, survey sources, ensemble sizes, and population splits (Appendix § 8).
Conclusion
Across 60 survey questions, 12 countries, and 6 simulators, disagreement predicts invalidity more accurately for between-group differences (AUC 0.87) than for a single group’s response proportions (0.71). The decomposition and correction on separate countries connect this difference to shared error: contrasts cancel a country-constant component that disagreement cannot measure. Using disagreement to refer the highest-scoring 20% of contrasts reduces invalidity among resolved, retained contrasts from 63.7% to 54.8%. This use of disagreement therefore depends on the quantity a study estimates and prioritizes the human checks that remain necessary.
Limitations
Separating calibration and test populations prevents the fitted corrections and probability mappings from using test-country responses. However, the survey responses predate our study design, and the evaluation does not constitute a preregistered prediction of newly collected outcomes. The 300 population splits assess sensitivity to the chosen countries, rather than transfer to a new human study. A prospective evaluation would fix simulator predictions and the analysis before collecting human responses. We plan to release a time-stamped protocol and a blind collection application for such a test.
Referral reduces risk without establishing validity for every retained estimate. In Figure 1C, the contradicted share falls from 32% at full coverage to 20% at half coverage, while 44% to 51% remain unresolved. At 80% coverage, the unweighted decision shares are 24% supported, 25% contradicted, and 51% unresolved. These shares describe all retained estimates, whereas the headline invalidity rate uses resolved estimates and question weighting.
The surveys report weighted response proportions without per-question respondent counts. We use each country’s survey-design sample size and do not adjust for the design effect of weighting. These approximations affect the human standard error and the validity decisions. In the ordinal sensitivity analysis, varying assumed sample sizes sixteen-fold changes the unresolved share from 22.3% to 5.4%. Invalidity among resolved estimates remains between 56% and 58%, and contrast AUC remains between 0.69 and 0.76 (Figure 5B,C).
Figure 5. The contrast signal is stable and grows with the ensemble. (A) Contrast AUC against the equivalence margin. (B) Contrast AUC against the assumed human sample size. (C) Verdicts on all estimates against the same sample size: the invalid share among resolved estimates stays between 56% and 58%, while the unresolved share falls from 22.3% to 5.4% as the assumed human study grows. An uncertain respondent count therefore changes how many estimates are resolved more than how the resolved ones split. (D) Contrast AUC against the number of simulators (bars are standard deviations over random -subsets). Filled markers are the defaults. The dashed line is 0.5.
The primary evaluation covers global opinion, with 6 simulators from four model families. Transfer to interactive tasks requires separate evidence. Appendix § 10 reports an exploratory persuasion-dialogue analysis whose outcomes do not permit evaluating the disagreement predictor. Using a simulator for adaptive optimization introduces the further problem of optimizing an imperfect proxy [8]. Our static evaluation does not assess validity under that distribution shift.
In interactive simulation, a bias that is constant across systems would cancel in their difference. Whether a simulator’s bias is sufficiently similar across systems must be tested. In conversational search, a prompted simulator over-states retrieval effectiveness on the turns where real users find little [21].
Our country-based prompts also raise a representational concern. Conditioning a simulator on a country elicits the model’s picture of that country’s population, which can reproduce stereotypes. We use these answers only to measure their error against real respondents and do not report them as descriptions of any population.
Ordinal-family analysis and robustness
We repeat the analysis on questions with ordered response options, using direction, mean response index, ordinal contrast, and full response distribution as estimands. Figure 3a and Figure 4a show how invalidity varies with disagreement and how well disagreement ranks invalid estimates on test populations. AUC is 0.67 for direction and 0.72 for ordinal contrasts (95% CI [0.64, 0.79], Spearman 0.37), compared with 0.51 for means and 0.47 for distributions.
Figure 3. Selective simulation on test populations. (a) Share of invalid estimates in each fifth of disagreement, ranked within family so the families share one axis. The trend differs by family: contrast invalidity rises from the lowest to the highest fifth, distribution invalidity does not. (b) Referring the highest-disagreement estimates lowers invalidity among the answered contrast estimates; the same rule gives a smaller reduction across all families combined. The dotted line marks 80% coverage.
Figure 4. (a) ROC of disagreement, by family. (b) Shared vs. simulator-specific error. Predictability and mechanism by family. (a) ROC curves, one per estimand family, using disagreement to catch invalid estimates: contrasts and direction rise above the diagonal, means and distributions stay near it (AUC is shown for each family, with a dotted line at 0.5). (b) Fraction of estimand-error variance shared across simulators (invisible to disagreement) versus simulator-specific. Contrast error contains a larger simulator-specific component than mean or distribution error. Family-level invalidity rates are in Table 5.
These differences make the estimand family important to interpreting the signal. Pooling all families gives AUC 0.59. Within-simulator spread gives contrast AUC 0.50 in this ordinal analysis, differing from the predictive pooled spread in the primary response-proportion analysis. We therefore report the secondary results by family.
Selective referral on ordinal items The stronger contrast prediction supports evaluating referral within that family. Invalidity falls from 40.8% at full coverage to 32.9% when the 20% highest-disagreement contrasts are referred (Figure 3b, red). Applying the same rule across all families gives a smaller reduction, from 55.2% to 50.9%, consistent with the weaker prediction for means and distributions. The fixed 80% coverage illustrates the rule’s effect. Appendix B fits probabilities of invalidity (test-population ECE 0.080) and uses assigned error and referral costs to choose a threshold, with a maximum modeled saving of 17%.
Results by topic To examine whether the contrast–mean ordering depends on subject matter, we assign questions to topics by keyword and evaluate the two estimands within each topic (Table 4). Among the 5 topics with at least 40 contrast estimands, contrast AUC exceeds mean AUC in all 5, ranging from 0.68 to 0.87. A separate three-question group falls below the 40-estimand threshold. The comparison tests consistency across topics within the survey domain.
| Topic domain | #q | contrast AUC | mean AUC | |
| religion/morals | 5 | 55 | 0.87 | 0.66 |
| economy/work | 12 | 132 | 0.69 | 0.56 |
| governance | 17 | 187 | 0.68 | 0.50 |
| international | 18 | 198 | 0.73 | 0.48 |
| society/tech2 | 3 | 33 | 0.23 | 0.38 |
| other | 5 | 55 | 0.76 | 0.46 |
Table 4. Disagreement predicts contrast error, but not mean error, within each subject-matter domain.
| Ordinal contrast family | Value |
| AUC (disagreement) | 0.72 [0.64, 0.79] |
| AUC (within-sim ) | 0.50 |
| AUC (all families) | 0.59 |
| invalidity, full cov. | 0.408 |
| invalidity, 80% cov. | 0.329 |
| 0.456 | |
| 0.673 |
Table 3. Selective simulation on the ordinal contrast family (robustness), test populations. The primary proportion result appears in Table 2.
Invalidity at low disagreement The retained risk is also apparent among low-disagreement estimates: invalidity is 0.456, compared with 0.673 among high-disagreement estimates. For distribution estimates, disagreement and error are weakly negatively related. Both findings illustrate the limitation of using agreement to assess errors shared across simulators.
Shared error and sampling variance fig:4b reports shared components of 0.58 of ordinal contrast-error variance and 0.94 of distribution-error variance. These are distinct from the primary response-proportion decomposition in § 5.2.
This shared component also limits the benefit of additional simulator draws. For a fixed estimand, mean squared error separates into squared bias and sampling variance. Additional draws reduce the sampling component but do not remove systematic bias (Table 6). More draws can therefore improve the precision of a simulated estimate without bringing it closer to the human reference.
| Simulator | RMSE | bias | std | v.frac |
| mistral-small | 0.540 | 0.515 | 0.161 | 0.089 |
| gpt-4o-mini | 0.589 | 0.586 | 0.055 | 0.009 |
| gpt-4o | 0.621 | 0.617 | 0.063 | 0.010 |
| qwen3-next-80b | 0.644 | 0.644 | 0.026 | 0.002 |
| mistral-large | 0.674 | 0.671 | 0.070 | 0.011 |
| gemma4-31b | 0.679 | 0.679 | 0.014 | 0.000 |
Table 6. Bias–variance decomposition per simulator on the same-unit families.
Invalidity by estimand Table 5 reports invalidity for direction (0.391), contrasts (0.507), means (0.651), and distributions (0.942). Contrasts have lower invalidity than the means from which they are constructed. Adjusting for human-reference signal-to-noise, effect size, survey source, and simulator preserves this ordering. On the standardized error scale , contrast error is 0.262 lower than mean error (SE 0.043, question-clustered ). These checks are consistent with shared-bias cancellation contributing to the difference.
Invalidity by model The high invalidity rates also persist across the included model groups. For gpt-4o and mistral-large, invalidity is 63.2%, compared with 63.6% for the smaller models. These comparisons do not isolate parameter count, model family, or training, and therefore do not establish a causal effect of scale.
Sensitivity to margins and sample sizes To assess dependence on the validity criteria, we vary the ordinal-contrast equivalence margin from half to twice its default. AUC remains in [0.70, 0.73] (Figure 5). Varying assumed human sample sizes from 250 to 4000 gives AUC in [0.69, 0.76], while the unresolved share falls from 22.3% to 5.4%. Separate survey-source analyses give contrast AUC 0.71 on Global Attitudes and 0.82 on World Values. Table 4 reports results by topic.
Population splits and ensemble size We also repeat the calibration/test split to assess dependence on the chosen countries. Over 300 random calibration/test splits of the populations, the test contrast AUC has median 0.73 (5–95%: [0.68, 0.78]) and exceeds 0.5 in 100% of them. Evaluating one test country at a time gives AUC between 0.62 (Lebanon) and 0.84. Resampling questions also preserves the contrast–mean AUC difference, estimated at 0.24 (95% CI [0.15, 0.34]).
Varying ensemble size shows how the signal depends on the number of simulators: contrast AUC rises from 0.66 at to 0.73 at (Figure 5D), as more simulators estimate the simulator-specific spread more precisely.
Prediction on test populations Risk scores for test populations use simulator outputs, with any learned mapping fixed on calibration populations. Their predictive performance (Figure 3) supports transfer across countries within these surveys for the estimand families where disagreement is informative.
Calibration and cost-based referral
The 80% coverage used above is an illustrative operating point. To choose a threshold from assigned error and referral costs, we fit probabilities of invalidity for ordinal contrasts on calibration populations and evaluate them on separate test populations. This supplements the main response-proportion analysis, which uses disagreement directly and gives invalidity of 63.7% at full coverage and 54.8% at 80% coverage.
Probability calibration Specifically, a logistic model estimates on calibration populations. We bin its test predictions to compare predicted and observed invalidity. Figure 6a shows the reliability curve: predicted and observed invalidity track the diagonal, with an expected calibration error of 0.080. These test-population probabilities provide the inputs to the referral decision below.
Figure 6. (a) Reliability of . (b) Expected cost vs. cost ratio. From risk score to decision (contrast, test populations). (a) Probability calibration on test populations (ECE 0.080). (b) Expected cost under the assigned error and referral costs.
Referral under assigned costs Using these predicted probabilities, let be the cost assigned to an invalid simulated estimate and the cost assigned to referral. We assume human referral resolves the estimate. Retaining an estimate has expected cost , while referral costs . Under this model, referral is optimal when . Figure 6b compares this rule with retaining every estimate or referring every estimate. The maximum modeled saving is 17%. At , the rule retains 35% of contrasts and saves 13%. The savings are calculated from the assigned costs rather than measured study expenditure.
Exploratory interactive analysis
The survey analysis concerns one-shot responses. To examine whether its level–contrast ordering also appears in interactive data, we compare simulator outputs with recorded human donation decisions in a persuasion-dialogue corpus [20].
All simulators overstate engagement, and contrast error is lower than level error. The overstatement is sufficiently uniform that the disagreement predictor cannot be scored on these outcomes. The case therefore permits a level–contrast error comparison, while evaluating disagreement as a predictor requires an interactive corpus covering a wider range of errors.
References
- [1]Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. volume 31, pages 337–351. Cambridge University Press, 2023.DOI
- [2]Krisztian Balog, Nolwenn Bernard, Saber Zerhoudi, and ChengXiang Zhai. Theory and toolkits for user simulation in the era of generative AI: User modeling, synthetic data generation, and system evaluation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 4138–4141, 2025. doi: 10.1145/3726302.3731697.DOI
- [3]Nolwenn Bernard, Krisztian Balog, Saber Zerhoudi, and ChengXiang Zhai. Large language models for the simulation of human behavior. HAL preprint hal-05484009, 2026. URL https://hal.science/hal-05484009.
- [4]Khaoula Chehbouni, Mohammed Haddou, Jackie Chi Kit Cheung, and Golnoosh Farnadi. Neither valid nor reliable? investigating the use of LLMs as judges. In Advances in Neural Information Processing Systems, volume 38, 2025. arXiv:2508.18076.arxiv.org/abs/2508.18076
- [5]Liang Chen. Simulated customers never walk away: Decision fidelity of llm user simulators measured against real purchase outcomes, 2026.arxiv.org/abs/2606.20708
- [6]Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. Towards measuring the representation of subjective global opinions in language models, 2023.arxiv.org/abs/2306.16388
- [7]Ran El-Yaniv et al. On the foundations of noise-free selective classification. volume 11, 2010.DOI
- [8]Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2023.arxiv.org/abs/2210.10760
- [9]Matt Gorbett and Suman Jana. Cross-model disagreement as a label-free correctness signal, 2026.arxiv.org/abs/2603.25450
- [10]Jessica Hullman, David Broska, Huaman Sun, and Aaron Shaw. This human study did not involve human subjects: Validating llm simulations as behavioral evidence, 2026.arxiv.org/abs/2602.15785
- [12]Daniël Lakens. Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. volume 8, pages 355–362. Sage Publications Sage CA: Los Angeles, CA, 2017.
- [13]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. volume 30, 2017.arxiv.org/abs/1612.01474
- [14]Zonghan Li and Feng Ji. When simulations look right but causal effects go wrong: Large language models as behavioral simulators. arXiv preprint arXiv:2604.02548, 2026.
- [15]Xuan Liu, Haoyang Shang, Zizhang Liu, Xinyan Liu, Yunze Xiao, Yiwen Tu, and Haojian Jin. Humanstudy-bench: Towards ai agent design for participant simulation, 2026.arxiv.org/abs/2602.00685
- [16]Joel Persson, Mårten Schultzberg, and Sebastian Ankargreen. Statistical foundations of LLM-based A/B testing: A surrogacy framework for human causal inference. arXiv preprint arXiv:2606.17165, 2026.arxiv.org/abs/2606.17165
- [17]Yada Pruksachatkun, Elaine Wan, Lyanna Chen, Kai-Wei Chang, and Chien-Sheng Wu. Salessim: Benchmarking and aligning multimodal language models as retail user simulators. arXiv preprint arXiv:2605.08334, 2026.arxiv.org/abs/2605.08334
- [18]Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International conference on machine learning, pages 29971–30004. PMLR, 2023.arxiv.org/abs/2303.17548
- [19]Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review, 21(2):97–126, 2006.DOI
- [20]Xuewei Wang, Weiyan Shi, Richard Kim, Yooyoung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 5635–5649, 2019.arxiv.org/abs/1906.06725
- [21]Saber Zerhoudi, Jelena Mitrovic, and Michael Granitzer. AutoSim: A self-improving user simulator for conversational search evaluation. In Proceedings of the 2026 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, 2026a. doi: 10.1145/3848000.3850196.arxiv.org/abs/2610.28054
- [22]Saber Zerhoudi, Adam Roegiest, and Johanne R. Trippas. Simulation of interactive information retrieval: A guided tour. In Proceedings of the 2026 Conference on Human Information Interaction and Retrieval, pages 434–436, 2026b. doi: 10.1145/3786304.3787892.DOI
- [23]Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Sherry Tongshuang Wu, et al. Mind the sim2real gap in user simulation for agentic tasks, 2026.arxiv.org/abs/2603.11245




