betaFeedback

WorksComputer Science

Silicon Samples Match the Answers but Not the Respondents

Abstract

Silicon sampling generates survey respondents by having a large language model (LLM) answer a survey as a person with given demographic attributes, and the resulting synthetic populations are increasingly used in place of human ones. Existing evaluations focus on individual-answer accuracy, per-question answer distributions, and correlations across questions. None of these compares the respondents themselves, so a generator whose respondents have combinations of answers found in no human respondent can still score well if it reproduces every answer distribution. We propose the respondent coverage gap (RCG), which represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one, relative to the distance between human samples. On two US social surveys, all six LLMs we evaluate leave human respondents uncovered, and under our main settings, they fall behind a baseline that samples each answer independently from the human marginals. Across the prompting and inference settings we test, every LLM has a higher RCG than a Gaussian copula fitted to the human data.

Introduction

Large language models (LLMs) can act as a specified person and answer questions from that person’s standpoint (Zhang et al., 2018; Shanahan et al., 2023; Wang et al., 2024). Silicon sampling [5] exploits this ability: it gives an LLM a combination of demographic attributes (e.g., female, in her thirties) as a persona, has it produce a full set of answers to a questionnaire (an answer profile) as a human respondent would, and repeats this over many personas to obtain a synthetic population. Such synthetic populations are increasingly used as substitutes for, or supplements to, human

populations in opinion polls, market research, and social science experiments [23, 1] (Santurkar et al. 2023; Park et al. 2026). Typical uses include estimating the distribution of opinions, probing the reactions of groups that are hard to recruit, and piloting a survey before fielding it.

The quality of such a synthetic population has so far been evaluated through the distribution of answers to each question. Individual accuracy [24] (Park et al. 2026) counts, question by question, whether the answer the LLM generates from a respondent’s attributes matches that respondent’s own answer. Marginal fidelity (Santurkar et al., 2023; Durmus et al., 2024 compares, for each question, the distribution of human answers with the distribution of answers generated from the same attributes. Correlational fidelity (Moon et al., 2024; Williams et al., 2026; Libovický, 2026; Ma et al., 2026) computes the correlation between answers to different questions separately on the human side and on the generated side, and compares the two. All of them aggregate answers along the axis of questions; none of them compares the respondent distribution, that is, how respondents are spread out as answer profiles.

Agreement in answer distributions does not guarantee agreement in respondent distributions. Even when the answer distributions for every question match, the synthetic population can consist of generated respondents whose answer profiles are absent from the human population; for a synthetic population to stand in for a human one, the respondent distributions must agree. We fit two generators to the World Values Survey (WVS) [19]: Marginals reproduces only the per-question answer distributions, and Copula additionally models pairwise dependence between questions. Per question, both are indistinguishable from the human respondents (Figure 1, bottom left). Per respondent, measuring the distance from each human respondent to the nearest generated one separates the two (Figure 1, bottom right; details in Appendix § 7): Marginals lies far from the human respondents, and Copula, despite modeling pairwise dependence, still falls short of the human reference. Per-question distributions and pairwise correlations do not fully characterize the respondent distribution.

Figure 1. Marginals reproduces each question’s answer distribution, and Copula additionally models pairwise dependence between questions. Top: answer profiles of human and generated respondents. Bottom: existing evaluation compares answers per question, where the three groups are identical (left); ours compares respondents, where humans are farther from the nearest generated respondent than from the nearest other human (right).

In this paper, we propose the respondent coverage gap (RCG), a metric of how well a synthetic population covers the respondent distribution of a human population. We treat each answer profile as a point and measure, from each human point, the distance to the nearest generated point; this directly quantifies how close each human respondent is to a generated respondent. Because this distance depends on the number of questions and the size of the populations, we normalize it by the same distance measured among the human points themselves. The reference is another human sample rather than the human respondents themselves: a synthetic population should not copy the human respondents but resemble them as a population. A ratio of one means the synthetic population is as close to the human respondents as another human sample would be; the more it exceeds one, the farther the least covered human respondents are from any generated respondent.

Evaluating six LLMs (two proprietary models from OpenAI and Google and four open-weight models from 9B to 70B parameters) on two real-world surveys of US residents, we find that none of them cover the human respondents. Under our main settings, all fall short of Marginals, which reproduces nothing but the per-question answer distributions. Some settings bring individual models close to or below Marginals: using the full response history, increasing reasoning effort, or using higher-tier models such as gpt-5.4 and gemini-3.1-pro-preview. However, no tested setting yields a lower RCG than Copula, which additionally models pairwise dependence between questions. Calibrating the generated answer distributions to match the human ones, question by question, reduces RCG below that of Marginals, but not below that of Copula.

This failure of LLMs is invisible to existing metrics. Marginal fidelity does not separate generators that differ in respondent coverage, and an aggregate correlation metric assigns Marginals, which samples every question independently, a score on par with the split-half ceiling of the human data. The failure also takes model-specific forms: some models contract onto typical profiles, others drift off the support of the human population. These failures surface only when the respondent distribution is examined; by that standard, synthetic populations from the evaluated LLMs are not yet adequate substitutes for human ones.

Related Work

Silicon Sampling and Its Evaluation

Silicon sampling has been applied across domains such as social simulations (Park et al., 2023, 2026), economic experiments [23], social science experiments [6, 1], and social surveys [5] (Santurkar et al. 2023; Ma et al. 2025).

Individual accuracy. This line of work scores, question by question, whether the generated answer matches the answer observed from the corresponding human respondent. Argyle et al. [5] address the task of predicting vote choice and measure the correspondence with tetrachoric correlations. Park et al. (2026) ground LLM agents in interviews with Americans and predict their held-out answers to the General Social Survey and the Big Five Inventory. Yet the scoring remains per-question. Park et al. (2026) themselves note that agents can predict a respondent’s answers well yet still misrepresent how attitudes covary across the sample, and defer sample-level correlational structure to future work.

Marginal fidelity. In contrast, this line of work compares answer distributions question by question, without pairing generated and human respondents. Santurkar et al. (2023) measure the 1-Wasserstein distance between model and human answer distributions over 1,498 Pew survey questions. Durmus et al. [13] apply the same per-question comparison across 60+ countries, scoring one minus the Jensen–Shannon distance. In all of these, the human reference is itself an aggregate distribution; no individual enters the comparison.

Beyond Marginal Evaluation

Both lines of work treat each question in isolation. A few studies (Moon et al., 2024; Williams et al., 2026; Libovický, 2026; Ma et al., 2026) take a step further and evaluate dependence across questions. To do so, they share a common two-step procedure: (1) computing inter-item associations separately within the human sample and within the generated sample, and (2) comparing the two association matrices as wholes. While these methods do capture dependence across questions, the comparison is made at the level of question pairs, not of the individual respondents that silicon sampling promises to substitute for. Williams et al. (2026) themselves defer individual-level structure to future work.

Comparing Generated and Human Respondents as Distributions

Outside silicon sampling, generated samples are compared with human ones as whole distributions, each sample being one point. Sajjadi et al. (2018) decompose this comparison into precision, how closely the generated samples resemble the human ones (fidelity), and recall, how well they span the variability of the human ones (diversity). Kynkäänniemi et al. [28] and Naeem et al. (2020) estimate both with kk-nearest-neighbor balls, and MAUVE (Pillutla et al., 2021) measures both for text as a divergence between the two distributions. These methods use discrete membership information (whether a sample lies inside a neighborhood ball or which quantization cell it occupies) rather than retaining its distance to the human samples.

ples. Prior work also compares synthetic-to-human nearest-neighbor distances with human-to-human distances. Alaa et al. [2] use such a comparison in a binary authenticity test for each generated sample, whereas Lautrup et al. [29] use a ratio of median distances in med.DCR to assess privacy risk. Our focus is instead on human respondents whose nearest generated respondent is too far away. Silicon sampling has so far been evaluated along the axis of questions (§ 2.1 and § 2.2); we instead compare distributions at the respondent level, treating each respondent’s complete answer profile as one sample.

Evaluation Framework

Let q1,…,qMq_1,\ldots,q_M be MM questions. A respondent is represented as an answer vector u=(u1,…,uM)\boldsymbol{u}=(u_1,\ldots,u_M), where umu_m is the answer chosen for question qmq_m. We write H\mathcal{H} for a set of human answer vectors and G\mathcal{G} for a set of generated answer vectors.

Conditional Generation

We generate each respondent by conditioning an LLM on a set of demographic attributes (e.g., sex and age), and compare the generated respondents G\mathcal{G} against the human respondents H\mathcal{H} who share the same attributes. The LLM answers the questions one at a time, and the prompt for each question includes a few of that respondent’s earlier answers. This sequential format, also employed in prior work (Moon et al., 2024; Libovický, 2026),2 allows later answers to depend on earlier ones within the same respondent.3

Respondent Coverage Gap

Given a distance d(u,v)d(u,v) between answer vectors, let δ(u;S)\delta(u;\mathcal{S}) be the distance from an answer vector uu to its nearest neighbor in a set S\mathcal{S}.4 Our core idea is to examine the distribution of the nearest-neighbor distance δ(u;G)\delta(u;\mathcal{G}) over the human respondents u∈Hu \in\mathcal{H} and, symmetrically, of δ(v;H)\delta(v;\mathcal{H}) over the generated respondents v∈Gv \in\mathcal{G}. The former asks, for each human respondent, how close the nearest generated respondent is; the latter asks, for each generated respondent, how close the nearest human respondent is.

Taking the maximum of the former recovers the directed Hausdorff distance (Hausdorff, [21]):

dH→G=max⁡u∈Hδ(u;G).(1)d_{\mathcal{H}\to\mathcal{G}} = \max_{u \in\mathcal{H}} \delta(u;\mathcal{G}). \tag*{(1)}

We modify this distance in two ways. First, we replace the maximum with the 95th percentile, denoted dH→G95d^{95}_{\mathcal{H}\to\mathcal{G}}, for robustness to outliers (Huttenlocher et al., [25]). Second, we normalize it by the same quantity computed within the human respondents, dH→H95d^{95}_{\mathcal{H}\to\mathcal{H}}, the 95th percentile of the distances from each human respondent to their nearest other human respondent; this makes the ratio comparable across different choices of H\mathcal{H}.

We define the respondent coverage gap (RCG) and its reverse-direction companion, the off-support ratio (OSR), which share this human-side quantity as the denominator:

RCG⁡(H,G)=dH→G95dH→H95.(2)\operatorname{RCG}(\mathcal{H},\mathcal{G}) = \frac{d^{95}_{\mathcal{H}\to\mathcal{G}}}{d^{95}_{\mathcal{H}\to\mathcal{H}}}. \tag*{(2)}
OSR⁡(H,G)=dG→H95dH→H95.(3)\operatorname{OSR}(\mathcal{H},\mathcal{G}) = \frac{d^{95}_{\mathcal{G}\to\mathcal{H}}}{d^{95}_{\mathcal{H}\to\mathcal{H}}}. \tag*{(3)}

Since δ(u;S)\delta(u;S) shrinks as SS grows, we use equally sized H\mathcal{H} and G\mathcal{G} in our main evaluation (§ 4.2 and Appendix § 10). RCG measures the extent to which human respondents are left uncovered by the generated ones; OSR reverses the direction of the numerator while keeping the same denominator, and thus measures the extent to which generated respondents lie outside the support of the human respondents. We use RCG as our main metric and OSR as a diagnostic when analyzing failure modes (§ 5.2).

Both ratios are expected to be near one when G\mathcal{G} is an independent sample of human respondents. RCG >> 1 indicates undercoverage of human respondents, whereas OSR >> 1 indicates generated respondents unusually far from the human sample (off-support). A low OSR is not necessarily better: when RCG >> 1 but OSR << 1, generated respondents stay close to some human respondents while leaving others poorly covered (contraction). Figure 2 illustrates these patterns: the off-support and contraction examples have similar RCG values (2.17 and 2.06), but their OSR values lie on opposite sides of one (2.24 and 0.55).

Toy examples of human-like, off-support, and contraction patterns

Figure 2. Toy examples of the human-like, off-support, and contraction patterns on two-dimensional synthetic data (50 human and 50 generated respondents, Euclidean distance). The panels illustrate representative configurations, not an exhaustive classification of RCG and OSR values.

OSR has a similar form to med.DCR (Lautrup et al., [29]), but addresses the opposite concern (§ 2.3). For privacy, a synthetic record that is too close to a human record poses a risk. Accordingly, med.DCR divides the median synthetic-to-real nearest-neighbor distance by its real-to-real counterpart, with values below one indicating greater concern. In contrast, OSR uses the 95th percentile to identify generated respondents that are unusually far from the human sample, with values above one indicating off-support generation.

Retaining the distances also sets our ratios apart from the density and coverage of Naeem et al. (2020), which test each nearest-neighbor distance against a threshold (§ 2.3). Our ratios stay informative no matter how far apart the two sets are: for a generated set that lies outside all human neighborhood balls, the coverage of Naeem et al. (2020) is zero however far away it lies, whereas dH→G95d^{95}_{\mathcal{H}\to\mathcal{G}} keeps growing with the distance. Our ratios also do not require the neighborhood size kk that sets that threshold; the only remaining choice is the aggregation percentile, whose effect we examine in Appendix § 14.

Evaluation Setup

Datasets

We use two surveys of US residents: the World Values Survey (WVS; Haerpfer et al., [19]), on social, religious, and political values, and the American National Election Studies (ANES; American National Election Studies, [4]), on vote choice and partisanship.5 For WVS, we use the United States sample of wave 7 (2,594 respondents) and uniformly sample 100 questions from the 144 used by Liu et al. (2026); the dataset provides 17 attribute variables, such as sex, age, education, and income. For ANES, we use the 2024 wave of its Time Series Cumulative Data File (February 2026 release; 5,453 respondents) and uniformly sample 100 questions from the 149 remaining after excluding administrative, demographic, and geographic items; the dataset provides 11 attribute variables including party identification and liberal–conservative ideology.6 Appendix § 8 lists all attributes and their binning.

In both datasets, the 100 questions are drawn once, and the same draw is used for all models and cells. Of the 100 questions, 30 (WVS) and 32 (ANES) are numeric rating scales and the rest are multiple-choice questions with ordinal or nominal options, so each respondent is represented by a vector of M=100M = 100 answers.

Conditioning Cells

A cell fixes the values of LL demographic attributes on which generation is conditioned (§ 3.1), e.g., (female, 30–39) with L=2L = 2; every cell also includes US residence, which all respondents share (§ 4.1). We keep cells with at least 100 human respondents and no missing attribute values, and sample up to 100 cells per LL, which gives 730 cells for WVS and 603 for ANES (L=0L = 0–88; Appendix § 10). Each cell cc gives a pair (Hc,Gc)(\mathcal{H}_c, \mathcal{G}_c): the answer vectors of the human respondents in cc and of the 100 generated respondents whose prompts carry the attribute values of cc (§ 4.3). All metrics in § 3.2 are computed per cell.

Prompts

Our prompt has four parts: a task instruction, a persona description, few-shot examples7 of the respondent’s earlier answers, and the question with its choices. To reduce sensitivity to wording and ordering, we prepare three instruction formats and three persona formats following prior studies ([5, 8, 3] (Santurkar et al. 2023; Taday Morocho et al. 2026; Moon et al. 2024)), and randomly assign one instruction format and one persona format to each generated respondent. We also randomize the question order and the order of attributes in the persona. For questions without a natural ordering of answer choices, we shuffle the choices to reduce LLM preferences for particular answer labels ([12] (Zheng et al. 2024)). The full templates and two example prompts are in Appendix § 9.

Models

Proprietary. We use two proprietary models: gpt-5.4-nano (OpenAI) and gemini-3.1-flash-lite (Google, hereafter gemini-flash-lite). Additionally, in an ablation study (§ 5.5), we use gpt-5.4 and gemini-3.1-pro-preview (Google, hereafter gemini-pro). We leave the sampling parameters (temperature, top_p, and top_k where available) at their vendor defaults and set only the reasoning effort to the lowest level available: none for the OpenAI models and minimal for the Google models.

Open-weight. We use four open-weight models: Qwen3.5-9B (Alibaba; Qwen Team, [14]), gemma-4-12B-it (Google, hereafter gemma-4-12B; Gemma Team, [14]), gpt-oss-20b (OpenAI, 2025), and Llama-3.3-70B-Instruct (Meta, hereafter Llama-3.3-70B; Grattafiori et al., 2024). We use their default decoding settings (Appendix § 11) and without tuning, to evaluate off-the-shelf use.

Distance

Because the questions are of mixed types (§ 4.1), we use the Gower distance (Gower, [16]) as d(u,v)d(u,v) (details in Appendix § 12). The Gower distance averages per-question distances, excluding the mm-th question when either umu_m or vmv_m is missing.8

Baselines

We select three non-LLM baselines. Marginals samples each answer umu_m independently from the marginal distribution of the human respondents H\mathcal{H}. Copula draws the whole answer vector uu from a Gaussian copula (Sklar, 1959): the marginals are estimated as in Marginals, and the questions are coupled through a correlation matrix estimated with Ledoit–Wolf shrinkage (Ledoit and Wolf, [30]), as fitting uses fewer respondents than questions in about half of the cells. LCA draws uu from a two-class latent class model (Goodman, [15]), that is, a mixture of two class-specific Marginals generators, fitted with StepMix (Morin et al., 2025). Because these baselines are fitted to the same respondents against which they are evaluated, they serve as references for what a generator can achieve with access to the cell’s respondents, rather than as direct competitors to the LLMs (fitting details in Appendix I).

Results

Main Results

For each cell, we compute RCG and OSR for 50 random draws of 100 human respondents. We first average the scores over draws within each cell, then average the resulting cell-level scores across cells. We also confirm that RCG behaves as intended; in each cell cc,9 we replace the generated set GcG_c with a separate sample of 100 human respondents drawn from HcH_c, disjoint from the 100 respondents used for evaluation, and compute RCG exactly as above. The average RCG is 1.001 (WVS) and 1.000 (ANES). The same replacement gives an average OSR of 1.001 (WVS) and 1.002 (ANES). Results under the hold-out fit setting are reported in Appendix I.

Table 1 shows the RCG and OSR for each generator and dataset (Appendix § 17 shows RCG on WVS and ANES by the number of conditioning attributes LL). All uncalibrated LLM RCG values in Table 1 exceed 1.0 and the values of all three baselines, including the simplest baseline, Marginals. For every LLM on both datasets, the 95% confidence interval for the RCG difference from Marginals excludes zero (Table 8 in Appendix § 16). Thus no LLM covers the human respondents as well as another human sample does (1.000–1.001), or even as well as Marginals, which ignores all dependence between questions. RCG shows no consistent ordering by model family, weight availability (proprietary vs. open-weight), or model size.

WVSANES
GeneratorRCGCal.OSRRCGCal.OSR
gpt-5.4-nano1.2721.1500.9941.3701.2321.173
gemini-flash-lite1.2181.1260.8401.3531.1780.838
Qwen3.5-9B1.2731.1450.9921.3471.2171.213
gemma-4-12B1.1991.1370.7931.3321.1940.942
gpt-oss-20b1.2691.1401.0131.3461.2021.144
Llama-3.3-70B1.2751.1470.7221.2771.1980.853
Marginals1.158–0.9891.250–1.078
Copula1.041–0.9461.091–0.976
LCA1.105–0.9591.143–0.950

Table 1. The respondent coverage gap (RCG) and the off-support ratio (OSR) for each generator on each dataset. Cal. is the RCG after marginal calibration. OSR is reported before calibration.

WVSANES
RCG − Marginals
gpt-5.4-nano+0.114 [+0.111, +0.117]+0.120 [+0.110, +0.130]
gemini-flash-lite+0.060 [+0.055, +0.065]+0.103 [+0.088, +0.120]
Qwen3.5-9B+0.115 [+0.112, +0.119]+0.097 [+0.087, +0.108]
gemma-4-12B+0.041 [+0.038, +0.045]+0.083 [+0.067, +0.100]
gpt-oss-20b+0.111 [+0.107, +0.115]+0.096 [+0.082, +0.111]
Llama-3.3-70B+0.116 [+0.112, +0.120]+0.028 [+0.017, +0.039]
Copula−0.118 [−0.121, −0.115]−0.159 [−0.172, −0.147]
LCA−0.053 [−0.056, −0.051]−0.106 [−0.118, −0.095]

Table 8. Cell-wise differences of the uncalibrated RCG (RCG in Table 1) from Marginals, averaged over cells. Brackets show the 95% CI from 10,000 bootstrap resamples of the sets of conditioning attributes, each with all of its cells. Every uncalibrated LLM is significantly above Marginals on both datasets.

Calibrating marginal distributions. We next ask whether correcting per-question response distributions closes the respondent coverage gap. We use the empirical human marginals in each evaluation cell as oracle targets, applying quantile mapping (Gudmundsson et al., [18]; Cannon et al., [9]) to ordered responses and minimum-rewrite remapping to nominal responses (Appendix § 22). Calibration (Cal. in Table 1) reduces RCG for all six LLMs on both datasets. The calibrated RCG values range from 1.126 to 1.150 on WVS and from 1.178 to 1.232 on ANES. They are lower than those of Marginals but higher than the human reference and those of both Copula and LCA. All differences from the three baselines are statistically significant (Table 9 in Appendix § 16). Thus, this item-wise correction improves respondent coverage but does not restore human-level coverage.

WVSANES
− Marginals− Copula− LCA− Marginals− Copula− LCA
gpt-5.4-nano−0.008<br>[−0.009, −0.007]+0.110<br>[+0.107, +0.113]+0.045<br>[+0.043, +0.048]−0.018<br>[−0.020, −0.016]+0.141<br>[+0.130, +0.153]+0.089<br>[+0.078, +0.099]
gemini-flash-lite−0.032<br>[−0.034, −0.031]+0.085<br>[+0.083, +0.088]+0.021<br>[+0.019, +0.023]−0.072<br>[−0.078, −0.065]+0.087<br>[+0.080, +0.095]+0.035<br>[+0.028, +0.042]
Qwen3.5-9B−0.014<br>[−0.015, −0.012]+0.104<br>[+0.101, +0.107]+0.040<br>[+0.037, +0.042]−0.032<br>[−0.035, −0.030]+0.127<br>[+0.116, +0.138]+0.074<br>[+0.064, +0.084]
gemma-4-12B−0.021<br>[−0.023, −0.020]+0.096<br>[+0.094, +0.099]+0.032<br>[+0.030, +0.034]−0.055<br>[−0.060, −0.050]+0.104<br>[+0.096, +0.112]+0.051<br>[+0.044, +0.059]
gpt-oss-20b−0.019<br>[−0.020, −0.017]+0.099<br>[+0.096, +0.102]+0.035<br>[+0.032, +0.037]−0.047<br>[−0.052, −0.043]+0.112<br>[+0.103, +0.121]+0.059<br>[+0.051, +0.067]
Llama-3.3-70B−0.011<br>[−0.013, −0.010]+0.106<br>[+0.104, +0.109]+0.042<br>[+0.040, +0.044]−0.052<br>[−0.057, −0.047]+0.107<br>[+0.099, +0.116]+0.055<br>[+0.047, +0.062]

Table 9. Cell-wise differences of the calibrated RCG (Cal. in Table 1) from each baseline, averaged over cells; e.g., “− Marginals” is Cal. − Marginals. Brackets show the 95% CI from 10,000 bootstrap resamples of the sets of conditioning attributes, each with all of its cells, as in Table 8. Every calibrated LLM is significantly below Marginals and significantly above Copula and LCA on both datasets.

Failure Modes

Using the uncalibrated OSR column of Table 1, we next examine how each LLM fails to cover the human respondents. The baselines (Marginals, Copula, and LCA) have OSR at or near one (0.946–1.078) on both datasets. RCG alone does not show how a model fails: on ANES, the six LLMs span 1.277–1.370 in RCG but 0.838–1.213 in OSR.

In both datasets, OSR splits the six LLMs into two groups: the upper group (gpt-oss-20b, gpt-5.4-nano, and Qwen3.5-9B) and the lower group (gemma-4-12B, gemini-flash-lite, and Llama-3.3-70B). The OSRs of the lower group are below one (0.722–0.840 on WVS and 0.838–0.942 on ANES) and the RCGs are above one. By the definitions in § 3.2, this combination indicates contraction. In the upper group, the OSRs on ANES are above one (1.144–1.213), which is the off-support pattern. By contrast, on WVS, the OSRs are close to one (0.992–1.013).

Prior work has reported both patterns under other names, though not at the level of individual respondents’ joint answer profiles: contraction as reduced variance ([8]) and flattening ((Wang et al., 2025)); and the off-support pattern as misportrayal ((Wang et al., 2025)).

Prior Metrics Can Mask Coverage Failures

RCG is not a repackaging of existing metrics: marginal fidelity does not separate generators that differ in RCG, and an aggregate correlation metric is saturated by a generator that samples answers independently across questions. We measure marginal fidelity by the mean per-question Jensen–Shannon distance (JSD; [13]) and dependence across questions by the correlation-structure score of Williams et al. (2026).

(2026). Figure 3 shows per-question JSD against RCG on WVS for the human reference and for the generators that match the human answer distributions: Marginals, Copula, LCA, and the six calibrated LLMs (the same figure for ANES is in Appendix § 18). Their JSD values lie within a narrow range (0.08–0.11), whereas their RCG ranges from 1.04 (Copula) to 1.16 (Marginals). The calibrated LLMs have the lowest JSD, as expected from calibrating to the human marginals in each evaluation cell (§ 5.1), yet their RCG remains above that of Copula and LCA. Per-question JSD depends only on each question’s answer distribution, not on how a respondent’s answers combine across questions. RCG compares whole answer profiles and therefore reflects this combination.

Figure 3. Per-question JSD (mean over questions and cells) against RCG on WVS for three baselines and six LLMs. For the LLMs, both metrics are computed after in-cell marginal calibration. Human (disjoint) uses only the 203 cells with at least 200 human respondents. Larger values are worse on both axes.

Figure 4 plots the correlation-structure mismatch (1−ρ1-\rho, where ρ\rho is the correlation-structure score of Williams et al., 2026) against RCG for all generators on WVS (the same figure for ANES is in Appendix § 18). Marginals achieves ρ=0.994\rho= 0.994 on WVS (split-half ceiling10 of the human data: 0.995) and 0.999 on ANES (split-half ceiling of the human data: 0.999). Its RCG, however, is 1.158 on WVS and 1.250 on ANES.

Figure 4. Correlation-structure mismatch (1−ρ1-\rho) against RCG for all nine generators on WVS. Larger values are worse on both axes. Triangles are LLMs; circles are the statistical baselines. The dotted horizontal line marks the permutation null (each question column permuted across cells, preserving marginals but destroying correlations). The dashed horizontal line marks the split-half ceiling of the human data (footnote 10).

Proposition 1 (Proof in Appendix § 19). For any set of generated respondents Gc\mathcal{G}_c in a cell cc, let Gc′\mathcal{G}'_c be obtained by permuting, separately for each question qmq_m, the answers to qmq_m among the respondents of Gc\mathcal{G}_c who answered it. Then Gc′\mathcal{G}'_c has the same marginal for every question as Gc\mathcal{G}_c, and the correlation-structure score of Williams et al. (2026) takes identical values on {Gc}c\{\mathcal{G}_c\}_c and {Gc′}c\{\mathcal{G}'_c\}_c.

The permutation destroys the alignment of answers within respondents, yet the score is unchanged. In particular, applying such within-cell permutations to the human data yields a correlation-structure score of exactly one against the original human data. This invariance is not a defect for measuring aggregate cross-cell structure; rather, it shows that the score cannot substitute for respondent-level evaluation.

Number of Generated Respondents

As described in § 3.2, RCG shrinks as ∣Gc∣\lvert\mathcal{G}_c\rvert grows for any generator. We ask how many generated respondents are needed for RCG to reach one.

We change ∣Gc∣|\mathcal{G}_c| from 100 to 1,000 (10 × the 100 human respondents) for the two proprietary models (gpt-5.4-nano and gemini-flash-lite)11 and to 10,000 (100 ×) for the three open-weight models (Qwen3.5-9B, gemma-4-12B, and gpt-oss-20b).12

Table 2 shows the RCG on the WVS subset of § 5.5 at different numbers of generated respondents ∣Gc∣|\mathcal{G}_c| and the slope of RCG regressed on log⁡10∣Gc∣\log_{10}|\mathcal{G}_c| (the same table for ANES is in Appendix § 20). The RCG of the two proprietary models remains above one even at ∣Gc∣=1,000|\mathcal{G}_c| = 1,000, and that of the three open-weight models remains above one even at ∣Gc∣=10,000|\mathcal{G}_c| = 10,000, whereas that of Marginals falls to 1.003 at ∣Gc∣=10,000|\mathcal{G}_c| = 10,000. OSRs change by less than 0.01 across different numbers of generated respondents (Appendix § 20).

RCG at ∣Gc∣=|\mathcal{G}_c| =Slope
1001,00010,000per decade
gpt-5.4-nano1.2701.197–−0.073†
gemini-flash-lite1.2221.159–−0.060†
Qwen3.5-9B1.2711.1931.130−0.070
gemma-4-12B1.2041.1411.094−0.056
gpt-oss-20b1.2671.1891.124−0.072
Marginals1.1611.0741.003−0.078
Copula1.0410.9310.847−0.095
LCA1.1091.0230.954−0.077

Table 2. RCG (cell mean over the WVS subset of § 5.5) at three sizes of the generated set ∣Gc∣|\mathcal{G}_c|. Slope is the median over cells of the per-cell change in RCG per tenfold increase in ∣Gc∣|\mathcal{G}_c|. –: not generated; †: fitted over one decade only, for reference.

Extrapolating the fitted trends linearly in log⁡10∣Gc∣\log_{10}|\mathcal{G}_c| from ∣Gc∣=10,000|\mathcal{G}_c| = 10,000 would place RCG = 1 at about 5×1055 \times10^{5} to 7×1057 \times10^{5} generated respondents per cell for the three open-weight models, or 5,000 to 7,000 times the 100 human respondents.13

Robustness to Prompting, Inference Settings, and Model Tier

We examine the robustness of the results in § 5.1 on a subset of the cells in § 4.2: five cells per level of LL, or all cells at levels with fewer (41 cells for WVS and 37 for ANES). The values in this subsection therefore differ from Table 1. Settings are in Appendix § 13.

Sensitivity to temperature. We vary the temperature from 0.1 to 1.5. RCG changes by at most 0.033 within each model, while OSR rises with temperature, most for gpt-5.4-nano, Qwen3.5-9B, and gpt-oss-20b (Appendix § 24).

Sensitivity to reasoning effort. We also vary the reasoning effort, which controls how much the model reasons before answering, over low, medium, and high for gpt-5.4-nano (default none) and over medium and high for gpt-oss-20b (default low). Raising the reasoning effort shifts both models toward contraction, and the RCG of gpt-5.4-nano falls mostly from none to low (Appendix § 25).

Sensitivity to model tier. We use gpt-5.4 from OpenAI, a higher-tier model of the same generation as gpt-5.4-nano, and gemini-3.1-pro-preview (gemini-pro), a frontier model from Google. Using higher-tier models reduces RCG except for gemini-pro on WVS, and both shift toward contraction. On ANES, gpt-5.4-nano at high effort and both higher-tier models have RCG point estimates 0.004–0.016 below Marginals but remain above Copula and LCA (Appendices § 25 and § 26).

Sensitivity to response history. We replace the up-to-three earlier answers in the prompt with the respondent’s full response history (see footnote 2). RCG falls in all twelve model–dataset pairs, and in two of them (gemma-4-12B on WVS and Llama-3.3-70B on ANES) significantly below Marginals, but every pair remains above Copula (Appendix § 21).

Conclusions

We proposed the respondent coverage gap to evaluate silicon sampling at the respondent level. No evaluated LLM achieves human-level coverage on either survey. Under our main settings, all trail a baseline that samples answers independently across questions. Across all tested temperatures, response histories, reasoning efforts, model tiers, and generated sample sizes, every evaluated LLM has a higher RCG than the Gaussian copula baseline at the same generated sample size. Existing per-question metrics miss this failure. Marginal calibration reduces RCG below that of Marginals, but not that of Copula. Before a synthetic population stands in for a human one, its respondents, not only its answers, should be shown to match.

Limitations

Scope of the LLM analysis. RCG and OSR are defined only through distances between respondents’ answer profiles, so they do not depend on a domain or population. Our analysis uses only US respondents from two surveys, WVS wave 7 ([19]) and the 2024 wave of ANES ([4]). Our findings on LLMs (RCG above one for all six models, and the contraction and off-support patterns) are thus shown only for these surveys; testing them in other areas requires further experiments. Such areas include psychological experiments ([7]), online purchasing ((Lu et al., 2026)), and human–computer interaction ([20]). Non-US and multilingual settings ((Ma et al., 2025; Liu et al., 2026)) are also outside our analysis.

Evaluating alternative prompting approaches with RCG. We evaluate LLMs with off-the-shelf prompting, as practitioners typically use them. Other prompting approaches, such as backstories ((Moon et al., 2024)), criteria-based prompting ([22]), Verbalized Sampling ((Zhang et al., 2026)), and String Seed of Thought ((Misaki and Akiba, 2026)), are not examined. Training-based methods, such as supervised fine-tuning ([10, 7, 26] (Suh et al. 2025)) and representation-level intervention ((Wang et al., 2026)), are also used to improve the accuracy and diversity of silicon sampling. These methods target per-question accuracy or response distributions; RCG can test whether their improvements extend to the level of individual respondents.

Ethical Considerations

Potential risk of silicon sampling. Silicon sampling could misportray or flatten identity groups ((Wang et al., 2025)) or caricature them ([11]), which may reinforce stereotypes when synthetic populations replace human respondents. Our metrics, RCG and OSR, can flag patterns consistent with off-support generation and contraction (§ 5.2) at the level of individual respondents. However, the metrics are computed only on the evaluated questions and cells.

Data and terms of use. We use the WVS wave 7 ([19]) and ANES wave 2024 ([4]). We follow their terms of use: we use the data only for research, do not redistribute the data files, and do not investigate specific survey respondents.

Use of AI. We used AI assistants to polish the manuscript’s English and assist with experimental code. The authors reviewed and verified all manuscript content and code.

Details of the Experiment in the Introduction

In § 1, we use the same 2,594 respondents of WVS ([19]) and 100 questions as in the main text (§ 4.1). We fit Marginals and Copula (§ 4.6) on all 2,594 human respondents and generate 2,594 respondents from each for the scatter plots and the nearest-neighbor distances; only the bar chart uses 50,000 generated respondents per generator. For the top panels of Figure 1, we apply principal coordinates analysis (PCoA) to the Gower distance matrix of the pooled 7,782 points (Human, Marginals, and Copula), treating the Gower distance as a squared distance. We adopt the first two principal coordinates as the axes and plot 100 randomly chosen points per group. In the bottom-right panel, the curves are kernel density estimates of the Gower distance from each human respondent to the nearest other human respondent (Human) and to the nearest generated respondent (Marginals and Copula), as defined in § 3.2.

Demographic Attributes

Table 3 shows demographic attributes and their numbers of levels.14 We bin age and the number of children as shown in the table and keep the original categories of the other attributes. As we described in § 4.1, we treat party identification and liberal–conservative ideology as demographic attributes for ANES. Ideology has eight levels because we keep “DK; haven’t thought much about it” as a level in addition to the seven-point scale.

AttributeWVSANES
sex22
age (10-year bins; 80+ top-coded)67
education65
employment status64
income level85
marital status45
religion34
immigrant status2–
citizenship1–
chief wage earner2–
family savings4–
social class4–
occupational group6–
sector of employment3–
household size6–
number of children (0, 1, 2, 3, 4+)5–
living with parents2–
race–5
Hispanic origin–3
party identification (7-point)–7
liberal–conservative ideology–8
Number of attributes1711

Table 3. Demographic attributes and their numbers of levels.

Prompts

Each query is a single user message with four blocks separated by blank lines: an instruction, a persona description, few-shot examples, and a question block. The instruction and the persona description are each drawn from three variants; the variant pair and the order of attributes in the persona description are assigned to each generated respondent at random and kept fixed across its 100 questions. Each generated respondent answers its 100 questions in a random order. The few-shot examples carry up to three of the respondent’s own previous answers, drawn at random without replacement from the questions it has already answered (§ 4.3). Figure 5 shows two full prompts from the main runs.

Two full prompts from the main run (WVS)

Figure 5. Two full prompts from the main run (WVS). Left: a multiple-choice question with the QA persona format and instruction i1. Right: a rating-scale question with the second-person persona format and instruction i2. Both personas are conditioned on sex, immigrant status, and age, and both prompts carry three few-shot examples taken from the same generated respondent’s previous answers. Texts are reproduced verbatim, including the original WVS question wording. The choice order in the left example is the shuffled order shown to the model.

Instructions. We use three instruction texts from prior work, with the adjustments described below. i1 and i2 are placed at the top of the prompt; i3 is placed between the few-shot examples and the question block. In i1, the survey name is “World Values Survey (WVS)” for WVS and “American National Election Studies (ANES) survey” for ANES. For i3, we keep only the first two sentences of the original; the remaining ones specify an answer format, which our question block replaces.

Instruction Variants

i1 ((Taday Morocho et al., 2026)): You are a survey respondent. Answer the following World Values Survey (WVS) question from the point of view of the given persona.

i2 ((Moon et al., 2024)): Below you will be asked to provide a short description of your demographic information, and then answer some questions.

i3 ([3]): Answer the following question from this perspective. Others will read what you choose; your goal is to convince them it was chosen from the perspective of the persona described above.

Persona descriptions. We use three formats: an interview transcript (QA; [5]), a second-person description ([8]), and a first-person description ([5] (Santurkar et al. 2023; Moon et al. 2024)). Residence in the United States is added to every persona

as an additional attribute. Every persona description starts with the survey year (“It is 2017.” for WVS and “It is 2024.” for ANES), following Bisbee et al. [8]. The following are the three formats for one cell (sex, immigrant status, and age) with the same attribute order.

QA Format

It is 2017.
Interviewer: What is your sex?
Me: Male
Interviewer: In what country do you live?
Me: The United States
Interviewer: Were you born in this country or
are you an immigrant to this country?
Me: Born in this country
Interviewer: What is your age in years?
Me: between 40 and 49
Second-Person Format

Description: It is 2017. Your sex is Male. You
live in the United States. You were born in the
United States. You are between 40 and 49 years
old.
First-Person Format

Description: It is 2017. My sex is Male. I live
in the United States. I was born in the United
States. I am between 40 and 49 years old.

Few-shot examples. The few-shot examples follow the persona format. The {answer} slot holds the text of the selected option for multiple-choice questions and the number for rating scales, including the don’t-know code; unparsable answers are skipped. Line breaks inside a question text are collapsed into single spaces in few-shot examples. This use of earlier answers as context follows Moon et al. (2024).

Few-Shot Example Formats

Interviewer: {question}
Me: {answer}

You answered "{answer}" to the question
"{question}".

I answered "{answer}" to the question
"{question}".

Question block. The question block uses a fixed template around the question and its answer choices. For multiple-choice questions, the numeric labels (1, 2, . . . , K) remain fixed. For each respondent–question pair, we shuffle the answer texts if the choices have no natural order; otherwise, we preserve their natural order.

Multiple-Choice Question Template

Answer the following question with a single
choice.

Choose one of the following numbers (1 2 . . .
K).
Output ONLY the answer tag in the form
<ans>N</ans> (for example, <ans>1</ans>). Do
NOT include any explanation, justification,
reasoning, or surrounding text.

# Question

{question text}

# Choices

- 1: {answer text shown at position 1}
- ...
- K: {answer text shown at position K}

Respond with only <ans>N</ans>. No other text.
Rating-Scale Question Template

Answer the following question with a single
integer.

Answer with an integer between {min} ({label
of min}) and {max} ({label of max}).
A larger number means closer to "{label of
max}".
If you cannot decide, answer {code for don't
know}.
Output ONLY the answer tag in the form
<ans>N</ans> (for example, <ans>5</ans>). Do
NOT include any explanation, justification,
reasoning, or surrounding text.

# Question

{question text}

Respond with only <ans>N</ans>. No other text.

Building Cells

As described in § 3.1, generation is conditioned on a set of demographic attributes. We build these sets from the VV attribute variables of each dataset (§ 4.1) in three steps. (i) We fix the number of conditioning attributes LL and enumerate all (VL)\binom{V}{L} subsets of the variables. For L=2L = 2 and the variables sex, age, and marital status, the subsets are (sex, age), (sex, marital status), and (age, marital status). (ii) For each subset, every combination of attribute values observed in the data defines a cell, e.g., (female, 30–39). We keep a cell only if it contains at least 100 human respondents and none of its values is a missing code; the threshold matches the 100 respondents we generate per cell. When a cell has more than 100 human respondents, we repeatedly draw 100 of them at random, so that § 3.2 always compares sets of equal size. (iii) For each LL, we sample 100 cells uniformly at random from the retained cells, or take all of them when fewer than 100 remain. All models are evaluated on the same cells. This yields 730 cells for WVS and 603 for ANES. Fewer than 100 cells remain at L=0L = 0 (one cell), at L=1L = 1 (70 and 55), and at the deepest levels (WVS: 59 at L=8L = 8; ANES: 46 at L=7L = 7 and one at L=8L = 8); every other level contributes 100 cells.15 In total, this amounts to 13.33 million queries per model (one per respondent-question pair).

All respondents in both datasets reside in the United States (§ 4.1), so every cell also includes this residence as an attribute. Each cell cc gives one pair (Hc,Gc)(\mathcal{H}_c, \mathcal{G}_c): Hc\mathcal{H}_c is the set of answer vectors of the human respondents in the cell, and Gc\mathcal{G}_c is the set of 100 answer vectors generated with the cell’s attribute values in the prompt (§ 4.3). All metrics in § 3.2 are computed per cell.

LLM Settings and Computational Cost

All queries to the proprietary models were issued between September 20 and October 1, 2026 (UTC). We leave the sampling parameters (temperature, top_p, and top_k where available) at their vendor defaults and set only the reasoning effort to the lowest level available: none for the OpenAI models and minimal for the Google models.

Open-weight models are served with vLLM 0.26.0 (Kwon et al., [27]) in bfloat16 on eight NVIDIA H200 GPUs. We send no sampling parameters, so each model runs with the values shipped in its generation config (or vLLM’s defaults when none are shipped), and we set reasoning to the lowest level available (off for Qwen3.5-9B and gemma4-12B, low for gpt-oss-20b).

Table 4 lists the six LLMs and the decoding settings that their defaults resolved to (§ 4.4). For gemini-flash-lite, Vertex AI does not expose its defaults, and we assume the values published for the same model in AI Studio (†). The exact identifiers are gemini-3.1-flash-lite for gemini-flash-lite and, for the open-weight models, Qwen/Qwen3.5-9B, google/gemma-4-12B-it, openai/gpt-oss-20b, and meta-llama/Llama-3.3-70B-Instruct. Every query is a single user message without a system prompt, with max_tokens of 4,096.

ModelWeights / accessParamsTemp.top-pptop-kkReasoningRun dates (2026, UTC)
gpt-5.4-nanoOpenAI API–1.01.0–effort noneSep 20
gemini-flash-liteGoogle Vertex AI–1.00.95166416thinking minimalSep 20
Qwen3.5-9Bopen weights, vLLM9B1.01.0–thinking offSep 20
gemma-4-12Bopen weights, vLLM12B1.00.9564thinking offSep 20 – 21
gpt-oss-20bopen weights, vLLM20B1.01.0–effort lowSep 20
Llama-3.3-70Bopen weights, vLLM70B0.60.9––Sep 21 – 24

Table 4. Decoding settings of the six LLMs in the main experiment. –: not applicable or not disclosed. †: assumed (see text).

Computational cost. For the runs reported in this paper, we spent approximately $11,160 on the proprietary models (OpenAI $4,140 and Google $7,020), and approximately 1,960 H200 GPU-hours (about 245 hours on eight GPUs) on the open-weight models.

Gower Distance

For two answer vectors uu and vv over the M=100M = 100 questions, the Gower distance is

d(u,v)=∑m=1Mwmdm(um,vm)∑m=1Mwm(4)d(u,v) = \frac{\sum_{m=1}^{M} w_m d_m(u_m,v_m)}{\sum_{m=1}^{M} w_m} \tag*{(4)}

where dmd_m depends on the type of question mm. For numeric and ordinal questions, dm=∣um−vm∣/Rmd_m = |u_m - v_m|/R_m, where RmR_m is the range of the answer scale of question mm: the maximum minus the minimum for rating scales, and the number of choices minus one for ordinal questions, whose answers are coded by their position in the choice order. For nominal questions, dm=1[um≠vm]d_m = \mathbf{1}[u_m \ne v_m]. The weight wmw_m is 1 if both respondents answered question mm and 0 otherwise, so the distance is an average over the questions that both respondents answered. We treat as missing the don’t-know code offered for rating scales (99 in WVS and −1-1 in ANES), the don’t-know and missing codes of the survey data, and any value outside the presented choices or scale.

Rating scales are treated as numeric, and multiple-choice questions as ordinal when their choices have a natural order and as nominal otherwise, giving 30, 45, and 25 questions of the three types in WVS and 32, 36, and 32 in ANES. We check the type of every question by hand, reading its wording and choices. In WVS, we treat the nine agree–disagree questions that offer “Hard to say” as nominal, because “Hard to say” has no position on the agreement scale.

Near-empty respondents. A respondent who answered very few questions is at distance zero from everyone who gave the same answers to those questions and can be the nearest neighbor of many respondents. We therefore use only human respondents with at least 10 valid answers (§ 4.1); this removes 2 WVS and 22 ANES respondents from 6 and 42 cells, and no cell falls below 100 human respondents.

Ablation Settings

The ablations in § 5.5 share the cell subset described there (up to five cells per level of LL; 41 cells for WVS and 37 for ANES). The six models and the three baselines of the main experiment are not rerun; their metrics are recomputed on this subset from the original responses. The values therefore differ slightly from Table 1. Unless stated otherwise, all settings other than the varied one are those of the main experiment (Appendix § 11).

Temperature. For the API models we send the temperature and top-pp explicitly: gpt-5.4-nano at t∈{0.1,0.5,1.0,1.5}t \in\{0.1, 0.5, 1.0, 1.5\} and gemini-flash-lite at t∈{0.1,1.0,1.5}t \in\{0.1, 1.0, 1.5\}, both with top-p=1.0p = 1.0. For gemini-flash-lite we also rerun its default (t=1.0t = 1.0, top-p=0.95p = 0.95) and confirm that the explicit setting reproduces the main run (RCG and OSR differ by at most 0.006, about as much as between two runs of the same setting). For the open-weight models we disable the released generation configurations and use the plain vLLM sampler (top-p=1.0p = 1.0, no top-kk) at t∈{0.1,0.5,1.0,1.5}t \in\{0.1, 0.5, 1.0, 1.5\} for Qwen3.5-9B, gemma-4-12B, and Llama-3.3-70B, and at t∈{0.1,1.0,1.5}t \in\{0.1, 1.0, 1.5\} for gpt-oss-20b. All temperature runs are therefore untruncated except those of gemini-flash-lite, which keep its default top-kk.

Reasoning effort. We vary the reasoning effort of gpt-5.4-nano at low, medium, and high (default none) and of gpt-oss-20b at medium and high (default low). Temperature and top-pp stay at their defaults, and max_tokens is raised to 16,384 for gpt-5.4-nano and 7,168 for gpt-oss-20b to leave room for reasoning tokens.

Model tier. We add gpt-5.4 and gemini-3.1-pro-preview (gemini-pro). At minimal, the thinking level we use for both Google models, gemini-pro still uses about 260 thinking tokens per call on average, whereas gemini-flash-lite uses none.

Sensitivity to Percentile Choices

All six LLMs have cell-averaged RCG above one at every tested percentile. We write the percentile in RCG and OSR as a fraction qq; the main text uses q=0.95q = 0.95, the 95th percentile. The choice of qq determines which part of the nearest-neighbor distance distribution is emphasized. At q=1.00q = 1.00, the metric uses the maximum distance; at q=0.95q = 0.95, it uses the 95th percentile and is less sensitive to the largest distances.

Table 5 and Table 6 show RCG and OSR for qq from 0.50 to 1.00. Our main result, that every LLM has a higher RCG than Marginals (Table 1), holds in both datasets at q=0.90,0.95,0.99,q = 0.90, 0.95, 0.99, and 1.001.00. At q≤0.75q \le0.75, only gemini-flash-lite, gemma-4-12B, and Llama-3.3-70B have lower RCG than Marginals, in three model–dataset pairs at q=0.75q = 0.75 and five at q=0.50q = 0.50. These three models show the contraction pattern (§ 5.2): their generated respondents stay close to some human respondents while leaving others poorly covered. A small qq ignores the human respondents farthest from the generated ones, which for these models are the poorly covered ones, so their RCG approaches or falls below that of Marginals. OSR decreases with qq for every generator, but the two groups of LLMs in § 5.2 do not overlap at any qq.

WVSANES
0.500.750.900.950.991.000.500.750.900.950.991.00
gpt-5.4-nano1.3251.3121.2901.2721.2561.2471.5351.4631.4041.3701.3171.282
gemini-flash-lite1.1571.1851.2101.2181.2301.2311.2931.3311.3461.3531.3611.359
Qwen3.5-9B1.3111.3071.2901.2731.2551.2371.5101.4361.3771.3471.2961.259
gemma-4-12B1.1441.1671.1881.1991.2101.2041.3701.3561.3421.3321.3211.306
gpt-oss-20b1.3181.3071.2871.2691.2491.2321.4811.4221.3721.3461.3111.284
Llama-3.3-70B1.2051.2471.2671.2751.2901.2871.2751.2781.2771.2771.2811.277
Marginals1.2291.1941.1681.1581.1501.1401.3531.2941.2611.2501.2341.219
Copula1.0821.0611.0461.0411.0341.0361.0991.0911.0891.0911.0981.102
LCA1.1451.1241.1091.1051.1031.1081.1561.1451.1421.1431.1511.154

Table 5. RCGs at various percentiles qq, dqHqdqCq\frac{d q_{\mathcal{H}}^q}{d q_{\mathcal{C}}^q}.

WVSANES
0.500.750.900.950.991.00
gpt-5.4-nano1.1851.1171.0430.9940.9200.8731.415 1.313 1.224 1.173 1.094 1.046
gemini-flash-lite0.9290.8980.8670.8400.7960.7640.966 0.912 0.865 0.838 0.794 0.766
Qwen3.5-9B1.1711.1061.0380.9920.9210.8771.442 1.347 1.262 1.213 1.135 1.090
gemma-4-12B0.9370.8840.8290.7930.7380.7041.123 1.047 0.980 0.942 0.884 0.848
gpt-oss-20b1.1811.1221.0571.0130.9490.9091.369 1.278 1.194 1.144 1.066 1.016
Llama-3.3-70B0.8650.8100.7560.7220.6720.6421.023 0.951 0.889 0.853 0.801 0.770
Marginals1.1921.1181.0400.9890.9110.8621.307 1.213 1.127 1.078 1.002 0.956
Copula1.0791.0340.9820.9460.8870.8501.072 1.047 1.005 0.976 0.926 0.893
LCA1.1121.0661.0030.9590.8890.8441.074 1.037 0.986 0.950 0.892 0.854

Table 6. OSRs at various percentiles qq, dqCqdqHq\frac{d q_{\mathcal{C}}^q}{d q_{\mathcal{H}}^q}.

Baseline Fitting

Copula transforms the answers to each question into normal scores using the ranks of their codes (tied values are assigned average ranks), estimates the correlation matrix of these scores with Ledoit–Wolf shrinkage, and maps sampled scores back to codes through the standard normal cumulative distribution function and the empirical quantile function of each question. Nominal questions go through the same steps, so their codes are ranked in numerical order although this order has no meaning. For these questions, Copula preserves the fitted marginal answer distributions, but the modeled man respondents of each cell, from which the 100 evaluated respondents are also drawn. To check whether the comparison between the LLMs and the baselines depends on this, we compare two settings on the cells with at least 200 human respondents (203 of the 730 cells for WVS and 196 of the 603 cells for ANES): In-cell fits the baselines as in § 5.1; Hold-out fits them, in each of the 50 draws, to 100 human respondents drawn from Hc\mathcal{H}_c without overlap with the 100 evaluation respondents. Both settings use the same draws of the evaluated respondents.

Table 7 shows the results. Compared with In-cell, Hold-out raises the RCG of Marginals by 0.008 on WVS and 0.009 on ANES, and that of Copula and LCA on WVS by 0.040 and 0.020. On ANES, Copula and LCA have lower RCG under Hold-out than under In-cell because 84% of their 100-respondent fitting samples require the median-imputation procedure described above. This procedure allows the models to be fitted to all 100 respondents rather than only those who answer every question. Under Hold-out, every LLM still has a higher RCG than every baseline, except that on ANES Llama-3.3-70B and Marginals have almost the same RCG (1.273 and 1.272). Under In-cell,

WVSANES
In-cellHold-outIn-cellHold-out
gpt-5.4-nano1.2611.354
gemini-flash-lite1.2211.345
Qwen3.5-9B1.2601.326
gemma-4-12B1.1861.310
gpt-oss-20b1.2551.317
Llama-3.3-70B1.2641.273
Marginals1.1561.1651.2631.272
Copula1.0471.0881.1201.109
LCA1.1081.1281.1681.151

Table 7. RCG under the In-cell and Hold-out settings. The LLMs are not fitted, so each LLM has one value for both settings.

dependence on other questions depends on this arbitrary ordering. LCA models every question as categorical and does not use the order of the codes.

Copula and LCA are fitted to the respondents who answer every question. In 147 of the 603 ANES cells, and in no WVS cell, fewer than 50 respondents do so; there we fill each missing answer with the median of the codes observed for its question in the cell (a placeholder with no meaning for nominal questions) and fit to all respondents.

In § 5.1, we fit the baselines to the huthe RCG of Llama-3.3-70B is 0.010 above that of Marginals on the 196 ANES cells used here, and 0.028 above it on all 603 ANES cells (Table 8).

95% Confidence Intervals for the Main Results

Table 8 and Table 9 show the 95% confidence intervals of § 5.1.

RCG by LL

Figure 6 and Figure 7 show RCG on WVS and ANES by the number of conditioning attributes LL. At every value of LL on WVS, all six LLMs have a higher RCG than Marginals, and all series stay above 1. On WVS, cells at different values of LL represent different subpopulations, whose median

RCG on WVS by the number of conditioning attributes L

Figure 6. RCG on WVS by the number of conditioning attributes LL. Each point is the mean over the cells at that LL (cell counts below the axis; hollow markers denote a single cell). Triangles are LLMs and circles are Marginals; the gray horizontal line marks RCG = 1.

RCG on ANES by the number of conditioning attributes L

Figure 7. RCG on ANES by the number of conditioning attributes LL. Each point is the mean over the cells at that LL (cell counts below the axis; hollow markers denote a single cell). Triangles are LLMs and circles are Marginals; the gray horizontal line marks RCG = 1.

size decreases from 2,594 respondents at L=0L = 0 to 114 at L=8L = 8. We therefore do not interpret the increase in RCG with LL as the effect of adding an attribute. Overall, on WVS the gap to Marginals is present at every conditioning depth we test, from the unconditioned population (L=0L = 0) to eight attributes; on ANES, it is present from one to six attributes.

Prior Metrics and RCG

Table 10 lists JSD, 1−ρ1-\rho, and RCG for all generators on both datasets.

WVSANES
JSD1−ρ1-\rhoRCGJSD1−ρ1-\rhoRCG
gpt-5.4-nano0.4030.3921.2720.4090.4941.370
gemini-flash-lite0.4130.2531.2180.4270.1181.353
Qwen3.5-9B0.3140.4521.2730.3430.4331.347
gemma-4-12B0.3930.3151.1990.4200.3911.332
gpt-oss-20b0.3580.3481.2690.3720.5031.346
Llama-3.3-70B0.4520.3291.2750.4480.2671.277
Marginals0.0910.0061.1580.0970.0011.250
Copula0.1000.0181.0410.1240.0051.091
LCA0.1000.0181.1050.1250.0061.143
Split-half ceiling−0.005−−0.001−
Permutation null−1.004−−1.000−

Table 10. JSD is the per-question Jensen–Shannon distance (base 2) averaged over questions and cells; 1−ρ1-\rho is the correlation-structure mismatch, where ρ\rho is the correlation-structure score of Williams et al. (2026); RCG is as in Table 1. Larger values are worse in every column. The last two rows give 1−ρ1-\rho for the split-half ceiling of the human data and for the permutation null.

Figure 8 shows mean per-question JSD against RCG for the baselines and calibrated LLMs on ANES.

Figure 8. Per-question JSD (mean over questions and cells) against RCG on ANES for three baselines and six LLMs. For the LLMs, both metrics are computed after in-cell marginal calibration. Human (disjoint) uses only the 196 cells with at least 200 human respondents. Larger values are worse on both axes.

Figure 9 shows correlation-structure mismatch (1−ρ1-\rho) against RCG for all nine generators on ANES.

Figure 9. Correlation-structure mismatch (1−ρ1-\rho) against RCG for all nine generators on ANES. Larger values are worse on both axes. Triangles are LLMs; circles are the statistical baselines. The dotted horizontal line marks the permutation null (each question column permuted across cells, preserving marginals but destroying correlations). The dashed horizontal line marks the split-half ceiling of the human data (footnote 10).

Proof of Proposition 1

The correlation-structure score of Williams et al. (2026) starts from a set of matrices {Xc}c∈C\{X_c\}_{c\in\mathcal{C}} with Xc∈Rnc×MX_c\in\mathbb{R}^{n_c\times M}, where C\mathcal{C} is the set of cells; we denote by Xc,i,mX_{c,i,m} the answer of respondent ii in cell cc to qmq_m, i∈Ici\in\mathcal{I}_c, ∣Ic∣=nc|\mathcal{I}_c|=n_c. Let XcHX_c^{H} be the matrix whose rows are the answer vectors in Hc\mathcal{H}_c, and XH={XcH}c∈CX^{H}=\{X_c^{H}\}_{c\in\mathcal{C}}; we define XGX^{\mathcal{G}} from {Gc}\{\mathcal{G}_c\} in the same way, and likewise XPX^{\mathcal{P}} and XHX^{\mathcal{H}} below.17

The score of Williams et al. (2026) is computed in three steps. (i) For each cell cc and each of the MoM_o rating-scale or ordinal questions qmq_m, let Ac,m(X)A_{c,m}(\mathcal{X}) be the mean answer to qmq_m among the respondents of cell cc who answered it, so that A(X)∈R∣C∣×MoA(\mathcal{X}) \in\mathbb{R}^{|C|\times M_o}. 17 (ii) Let R(X)∈RMo×MoR(\mathcal{X}) \in\mathbb{R}^{M_o\times M_o} be the matrix of Pearson correlations between the columns of A(X)A(\mathcal{X}), that is, between questions across cells, and let s(X)∈RMo(Mo−1)2\mathbf{s}(\mathcal{X}) \in\mathbb{R}^{\frac{M_o(M_o-1)}{2}} be the vector of its strictly upper-triangular entries. (iii) The score is the Pearson correlation ρ=corr⁡(s(XH),s(XG))\rho=\operatorname{corr}(\mathbf{s}(\mathcal{X}^{H}),\mathbf{s}(\mathcal{X}^{G})).

Proof. In each cell cc, the permutation only reorders the answers to qmq_m among the respondents in Gc\mathcal{G}_c who answered qmq_m, so the within-cell marginal of qmq_m is unchanged, which is the first claim. In particular, the mean answer to each of the MoM_o questions is unchanged, so A(XG′)=A(XG)A(\mathcal{X}^{\mathcal{G}'}) = A(\mathcal{X}^{\mathcal{G}}). Since RR, ss, and ρ\rho are computed from AA alone, ρ\rho takes the same value on {Gc}c\{\mathcal{G}_c\}_c and {Gc′}c\{\mathcal{G}'_c\}_c.

The same argument applied to the human respondents gives A(XH′)=A(XH)A(\mathcal{X}^{\mathcal{H}'}) = A(\mathcal{X}^{\mathcal{H}}), so the score of {Hc′}c\{\mathcal{H}'_c\}_c against {Hc}c\{\mathcal{H}_c\}_c is the correlation of s(XH)\mathbf{s}(\mathcal{X}^{\mathcal{H}}) with itself, which is one.

Number of Generated Respondents

This section reports the ANES results and the OSR for § 5.4. At each ∣Gc∣|\mathcal{G}_c|, the generated set is drawn at random from 1,000 (proprietary models) or 10,000 (others) respondents per cell, generated separately from the main experiment, so the values at ∣Gc∣=100|\mathcal{G}_c| = 100 in Table 2, Table 11, and Table 12 differ from those of the main experiment on the same subset (Appendix § 13) by less than 0.01.

RCG at ∣Gc∣=|\mathcal{G}_c| = 1001,00010,000Slope per decade
gpt-5.4-nano1.3911.298–−0.093†
gemini-flash-lite1.3591.259–−0.097†
Qwen3.5-9B1.3571.2391.146−0.104
gemma-4-12B1.3381.2421.167−0.083
gpt-oss-20b1.3451.2401.159−0.092
Marginals1.2711.1661.080−0.094
Copula1.0860.9640.874−0.103
LCA1.1431.0430.965−0.089

Table 11. RCG (cell mean over the ANES subset of § 5.5) at three sizes of the generated set ∣Gc∣|\mathcal{G}_c|. Slope is the median over cells of the per-cell change in RCG per tenfold increase in ∣Gc∣|\mathcal{G}_c|. –: not generated; †: fitted over one decade only, for reference.

WVSANES
OSR at ∣Gc∣\lvert G_c\rvert =SlopeOSR at ∣Gc∣\lvert G_c\rvert =Slope
1001,00010,000per decade1001,00010,000per decade
gpt-5.4-nano0.9880.992–0.003{sup}†`1.1931.197–0.005{sup}†`
gemini-flash-lite0.8280.832–0.004{sup}†`0.8700.875–0.004{sup}†`
Qwen3.5-9B0.9810.9860.9860.0021.2321.2371.2380.002
gemma-4-12B0.7830.7870.7870.0020.9670.9720.9720.002
gpt-oss-20b1.0081.0131.0130.0021.1631.1681.1690.002
Marginals0.9900.9930.9930.0021.1061.1091.1090.002
Copula0.9460.9500.9510.0020.9991.0051.0060.003
LCA0.9630.9670.9670.0020.9610.9660.9660.002

Table 12. OSR (cell mean over the subsets of Appendix § 13) at three sizes of the generated set ∣Gc∣\lvert G_c\rvert. Slope is the median over cells of the per-cell slope of OSR regressed on log⁡10∣Gc∣\log_{10}\lvert G_c\rvert, i.e., the change per tenfold increase. The two proprietary models were generated only up to ∣Gc∣=1,000\lvert G_c\rvert= 1{,}000 (–: not generated); their slopes (†) are fitted over one decade instead of two and are reported for reference.

Table 11 is the ANES counterpart of Table 2. RCG starts higher on ANES and falls faster with ∣Gc∣|\mathcal{G}_c| for every generator. Still, as on WVS, no LLM reaches one within the sizes we generate. We estimate how many generated respondents per cell are needed for RCG to reach one by extrapolating the slopes in Table 11 linearly in log⁡10∣Gc∣\log_{10}|\mathcal{G}_c| from ∣Gc∣=10,000|\mathcal{G}_c| = 10{,}000. The estimates are about 2.5×1052.5 \times10^5 to 1.0×1061.0 \times10^6 generated respondents per cell for the three open-weight models, that is, about 2,500 to 10,000 times the 100 human respondents. As on WVS, RCG falls less in the second decade than in the first, so these estimates are likely underestimates.

Table 12 shows OSR at the same sizes on both datasets. From ∣Gc∣=100|\mathcal{G}_c| = 100 to the largest size we generate, OSR rises by 0.003 to 0.007 for every generator.

Full-History Ablation

Providing the full response history lowers mean RCG in all twelve model–dataset pairs, although every mean RCG remains above one.

We check how the amount of response history included in the prompt affects RCG and OSR. We use the full history as few-shot examples (Table 13).

WVSANES
RCGRCG − MarginalsOSRRCGRCG − MarginalsOSR
gpt-5.4-nanolimited history (≤3\le3)1.270+0.109 [+0.097, +0.121]0.9901.390+0.125 [+0.087, +0.165]1.189
full history1.231+0.070 [+0.056, +0.083]1.1241.328+0.062 [+0.015, +0.113]1.304
gemini-flash-litelimited history (≤3\le3)1.218+0.057 [+0.038, +0.076]0.8291.356+0.090 [+0.039, +0.141]0.878
full history1.184†1.184^{\dagger}+0.022 [−0.002, +0.046]0.9011.249†1.249^{\dagger}−0.017 [−0.078, +0.048]0.820
Qwen3.5-9Blimited history (≤3\le3)1.268+0.107 [+0.094, +0.120]0.9811.354+0.089 [+0.053, +0.129]1.234
full history1.258+0.097 [+0.082, +0.111]1.0971.276†1.276^{\dagger}+0.010 [−0.036, +0.061]1.326
gemma-4-12Blimited history (≤3\le3)1.205+0.044 [+0.031, +0.058]0.7831.332+0.066 [+0.014, +0.124]0.968
full history1.113‡1.113^{\ddagger}−0.049 [−0.063, −0.035]0.8371.224†1.224^{\dagger}−0.042 [−0.103, +0.025]1.033
gpt-oss-20blimited history (≤3\le3)1.268+0.107 [+0.093, +0.121]1.0031.338+0.072 [+0.027, +0.123]1.163
full history1.193+0.031 [+0.017, +0.046]1.0541.276†1.276^{\dagger}+0.010 [−0.039, +0.063]1.251
Llama-3.3-70Blimited history (≤3\le3)1.279+0.118 [+0.102, +0.135]0.7091.274†1.274^{\dagger}+0.008 [−0.031, +0.048]0.877
full history1.243+0.082 [+0.063, +0.101]0.7241.156‡1.156^{\ddagger}−0.110 [−0.160, −0.056]0.938
Marginals1.1610.9911.2661.108
Copula1.0400.9471.0860.991
LCA1.1110.9641.1490.964

Table 13. RCG − Marginals is the cell-wise difference in RCG from Marginals, averaged over cells. The brackets show its 95% CI from 10,000 bootstrap resamples of the sets of conditioning attributes, each with all of its cells. The CI thus covers variation from the choice of attribute sets, with the human and generated responses fixed. † indicates that the 95% CI of RCG − Marginals includes zero, whereas ‡ indicates that the interval lies entirely below zero. With the full history, all RCG values remain above those of Copula.

† indicates that the 95% CI of RCG − Marginals includes zero, whereas ‡ indicates that the interval lies entirely below zero. All full-history RCG values are above those of Copula; the RCGs of four of six LLMs for WVS and one of six for ANES are significantly above Marginals. Only gemma-4-12B on WVS and Llama-3.3-70B on ANES have RCG values significantly below those of Marginals.

Calibration Settings

Let aa and bb denote answer choices for question qmq_m and pc,mHp^{\mathcal{H}}_{c,m} be the distribution of answers to qmq_m among respondents in Hc\mathcal{H}_c (and pc,mGp^{\mathcal{G}}_{c,m} likewise for Gc\mathcal{G}_c). We consider a transition matrix Tc,m(b∣a)T_{c,m}(b \mid a) that satisfies ∑aTc,m(b∣a)pc,mG(a)=pc,mG(b)\sum_a T_{c,m}(b \mid a)p^{\mathcal{G}}_{c,m}(a) = p^{\mathcal{G}}_{c,m}(b) for all bb and ∑bTc,m(b∣a)=1\sum_b T_{c,m}(b \mid a) = 1 for all aa. We replace each answer of a generated respondent with a draw from Tc,m(⋅∣a)T_{c,m}(\cdot\mid a), independently across questions and respondents, and leave missing answers as they are.

The constraint leaves Tc,mT_{c,m} underdetermined. For ordered questions (rating scales and ordinal questions), we use quantile mapping (Gudmundsson et al., [18]; Cannon et al., [9]), which is an optimal transport plan for the cost ∣a−b∣|a-b| (Santambrogio, 2015). With choices {1,2,3}\{1,2,3\}, pcG=(0.5,0.5,0)p^{\mathcal{G}}_c = (0.5, 0.5, 0), and pcH=(0.25,0.25,0.5)p^{\mathcal{H}}_c = (0.25, 0.25, 0.5), it moves half of the answers at 1 to 2 and all answers at 2 to 3. For nominal questions, where ∣a−b∣|a-b| has no meaning, we use minimum-rewrite remapping, an optimal transport plan for the cost 1[a≠b]\mathbb{1}[a \ne b] (Levin and Peres, 2017): it preserves a probability mass of min⁡(pc,mG(a),pc,mH(a))\min(p^{\mathcal{G}}_{c,m}(a), p^{\mathcal{H}}_{c,m}(a)) at each choice $a and reallocates the excess to choices where pc,mGp^{\mathcal{G}}_{c,m} falls short of pc,mHp^{\mathcal{H}}_{c,m}.

Missing-Value Ablation

We evaluate the impact of missing values on the RCG and OSR. In calculating RCG and OSR, the Gower distance excludes questions with missing answers; fewer compared questions make distances more variable, so nearest-neighbor distances become smaller. Human respondents have missing answers while generated respondents rarely do, so human-to-human distances are computed over fewer questions than human-to-generated distances; this asymmetry may reduce the human-to-human distance and thereby inflate RCG.

To check this, we compute a masked variant: in calculating RCG and OSR, we repeat the following process for every generated respondent v∈Gcv \in G_c. We randomly select a human respondent u∈Hcu \in H_c outside the 100 human respondents used in the calculation, and take the questions that uu left unanswered as a mask. We then apply the mask to vv by treating the answers of vv to these questions as missing. In cells with exactly 100 human respondents (11 of the 730 cells for WVS and 8 of the 603 cells for ANES), no human respondent remains outside the evaluation sample. For these cells, we randomly assign the 100 human respondents’ missingness masks to the 100 generated respondents in a one-to-one mapping. Table 14 therefore covers the same cells as Table 1. This procedure assesses sensitivity to the mismatch in missingness patterns between human and generated respondents.

WVSANES
RCGOSRRCGOSR
gpt-5.4-nanooriginal1.2720.9941.3701.173
masked1.2650.9921.3371.169
gemini-flash-liteoriginal1.2180.8401.3530.838
masked1.2090.8401.3050.833
Qwen3.5-9Boriginal1.2730.9921.3471.213
masked1.2670.9921.3181.210
gemma-4-12Boriginal1.1990.7931.3320.942
masked1.1910.7941.2870.938
gpt-oss-20boriginal1.2691.0131.3461.144
masked1.2631.0111.3141.143
Llama-3.3-70Boriginal1.2750.7221.2770.853
masked1.2610.7241.2370.851
Marginalsoriginal1.1580.9891.2501.078
masked1.1540.9911.2221.075
Copulaoriginal1.0410.9461.0910.976
masked1.0370.9471.0630.972
LCAoriginal1.1050.9591.1430.950
masked1.1010.9601.1110.947

Table 14. RCG and OSR in the original setting (Table 1) and the masked setting.

Table 14 shows the results. The masked RCGs are lower than the original RCGs, but the masked RCGs of all six LLMs remain above Marginals. RCG decreases more on ANES (by 0.03–0.05) than on WVS (by at most 0.014), because human respondents leave more answers missing on ANES (6.0% vs. 1.1%). OSR changes by less than 0.01.

Temperature Ablation

Table 15 shows RCG and OSR at each temperature. For RCG, the largest range is 0.033 (Qwen3.5-9B on WVS), and RCG stays above Marginals at every temperature. For OSR, the range on each dataset is 0.035 to 0.275 for gpt-5.4-nano, Qwen3.5-9B, and gpt-oss-20b, but at most 0.025 for the other three models; the largest movement is in Qwen3.5-9B on WVS, from 0.812 at temperature 0.1 to 1.087 at 1.5. For gpt-5.4-nano, Qwen3.5-9B, and gpt-oss-20b, OSR on WVS crosses one as the temperature increases. Wang et al. (2025) report a finding consistent with this pattern for free-response text: raising the temperature does not bring the diversity of GPT-4 responses up to the level of human participants.

WVSANES
RCGOSRRCGOSR
gpt-5.4-nano
t=0.1t = 0.11.2700.9391.4031.136
t=0.5t = 0.51.2700.9551.3951.152
t=1.0t = 1.0 (default)1.2700.9901.3901.189
t=1.5t = 1.51.2711.0441.3941.251
gemini-flash-lite
default1.2180.8291.3560.878
t=0.1t = 0.11.2210.8271.3540.868
t=1.0t = 1.01.2190.8301.3460.873
t=1.5t = 1.51.2160.8381.3490.878
Qwen3.5-9B
t=0.1t = 0.11.2900.8121.3391.107
t=0.5t = 0.51.2800.8601.3421.147
t=1.0t = 1.0 (default)1.2680.9811.3541.234
t=1.5t = 1.51.2571.0871.3461.302
gemma-4-12B
default1.2050.7831.3320.968
t=0.1t = 0.11.2030.7761.3390.963
t=0.5t = 0.51.2040.7781.3400.962
t=1.0t = 1.01.2000.7861.3350.970
t=1.5t = 1.51.2050.8011.3310.983
gpt-oss-20b
t=0.1t = 0.11.2610.9701.3591.161
t=1.0t = 1.0 (default)1.2681.0031.3381.163
t=1.5t = 1.51.2851.0761.3551.196
Llama-3.3-70B
default1.2790.7091.2740.877
t=0.1t = 0.11.2770.7081.2770.876
t=0.5t = 0.51.2760.7091.2770.877
t=1.0t = 1.01.2780.7121.2770.877
t=1.5t = 1.51.2760.7161.2760.881
Marginals1.1610.9911.2661.108

Table 15. RCG and OSR at each temperature. At t=1.5t = 1.5, 9.7% (WVS) and 9.9% (ANES) of the answers of Qwen3.5-9B cannot be parsed and are treated as missing; for the other open-weight models, the rate is at most 1.1%.

Reasoning Effort Ablation

Raising the reasoning effort shifts both models toward contraction, while RCG falls most when reasoning is turned on. We vary the reasoning effort, which controls how much the model reasons before answering, over low, medium, and high for gpt-5.4-nano (default none) and over medium and high for gpt-oss-20b (default low). Table 16 shows the results.19 Turning reasoning on for gpt-5.4-nano (none to low) lowers RCG by 0.045 on WVS and 0.110 on ANES, whereas raising the effort beyond low changes RCG by at most 0.031 in either model. OSR drops in all four model–dataset pairs, by 0.073 to 0.169 from the default to high. RCG stays above Marginals except for gpt-5.4-nano on ANES, which equals Marginals at medium (1.266) and falls below it at high (1.253).

WVSANES
RCGOSRRCGOSR
gpt-5.4-nano
none (default)1.2700.9901.3901.189
low1.2250.9451.2801.059
medium1.2140.9191.2661.025
high1.2130.9171.2531.020
gpt-oss-20b
low (default)1.2681.0031.3381.163
medium1.2370.9321.3261.096
high1.2460.9021.3381.060
Marginals1.1610.9911.2661.108
Copula1.0400.9471.0860.991
LCA1.1110.9641.1490.964

Table 16. RCG and OSR at each reasoning effort level.

gemini-pro costs approximately 33 times as much as gemini-flash-lite.

Table 17 indicates that raising the tier lowers RCG in three of the four model–dataset pairs, by 0.071 to 0.128, and in the remaining one raises it by 0.020 (gemini-pro on WVS); on ANES, both higher-tier models fall below Marginals. OSR also drops in all four pairs and crosses one only for gpt-5.4 on ANES (from 1.189 to 0.838).

Model Tier Ablation

Raising the model tier mostly lowers RCG but shifts each model toward contraction. We use gpt-5.4 from OpenAI, a higher-tier model of the same generation as gpt-5.4-nano, and gemini-3.1-pro-preview (gemini-pro), a frontier model from Google. In our runs, gpt-5.4 costs approximately 12 times as much per API call as gpt-5.4-nano, and

WVSANES
RCGOSRRCGOSR
OpenAI
gpt-5.4-nano (default)1.2700.9901.3901.189
gpt-5.41.1990.6921.2620.838
Google
gemini-flash-lite (default)1.2180.8291.3560.878
gemini-pro1.2380.6921.2500.763
Marginals1.1610.9911.2661.108
Copula1.0400.9471.0860.991
LCA1.1110.9641.1490.964

Table 17. RCG and OSR of the default and higher-tier models.

References

  1. [1]Gati Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 337–371. PMLR.arxiv.org/abs/2208.10264
  2. [2]Ahmed M. Alaa, Boris van Breugel, Evgeny Saveliev, and Mihaela van der Schaar. 2022. How faithful is your synthetic data? Sample-level metrics for evaluating and auditing generative models. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 290–306. PMLR.arxiv.org/abs/2102.08921
  3. [3]Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404–12422. Association for Computational Linguistics.arxiv.org/abs/2402.13231
  4. [4]American National Election Studies. 2026. ANES Time Series Cumulative Data File. [dataset and documentation]. February 5, 2026 version.
  5. [5]Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Polit. Anal., 31(3):337–351.
  6. [6]Ashwini Ashokkumar, Luke Hewitt, Isaias Ghezae, and Robb Willer. 2026. Large language models can predict the results of social science experiments. Nature, 656(8126):115–122.
  7. [7]Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria K Eckstein, Noémi Éltető, Thomas L Griffiths, Susanne Haridi, Akshay K Jagadish, Li Ji-An, Alexander Kipnis, Sreejan Kumar, Tobias Ludwig, Marvin Mathony, Marcelo Mattar, and 21 others. 2025. A foundation model to predict and capture human cognition. Nature, 644(8078):1002–1009.DOI
  8. [8]James Bisbee, Joshua D Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M Larson. 2024. Synthetic replacements for human survey data? The perils of large language models. Polit. Anal., 32(4):401–416.
  9. [9]Alex J Cannon, Stephen R Sobie, and Trevor Q Murdock. 2015. Bias correction of GCM precipitation by quantile mapping: How well do methods preserve changes in quantiles and extremes? J. Clim., 28(17):6938–6959.DOI
  10. [10]Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich. 2025. Specializing large language models to simulate survey response distributions for global populations. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3141–3154. Association for Computational Linguistics.arxiv.org/abs/2502.07068
  11. [11]Myra Cheng, Tiziano Piccardi, and Diyi Yang. 2023. CoMPoST: Characterizing and evaluating caricature in LLM simulations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10853–10875. Association for Computational Linguistics.arxiv.org/abs/2310.11501
  12. [12]Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. 2024. Questioning the survey responses of large language models. In Advances in Neural Information Processing Systems, volume 37, pages 45850–45878. Curran Associates, Inc.
  13. [13]Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. Towards measuring the representation of subjective global opinions in language models. In First Conference on Language Modeling.arxiv.org/abs/2306.16388
  14. [14]Gemma Team. 2026. Gemma 4 technical report. arXiv preprint arXiv:2607.02770.
  15. [15]Leo A Goodman. 1974. Exploratory latent structure analysis using both identifiable and unidentifiable models. Biometrika, 61(2):215–231.
  16. [16]J C Gower. 1971. A general coefficient of similarity and some of its properties. Biometrika, 27(4):857–871.DOI
  17. [18]L. Gudmundsson, J. B Bremnes, J. E Haugen, and T Engen-Skaugen. 2012. Technical note: Downscaling RCM precipitation to the station scale using statistical transformations – a comparison of methods. Hydrol. Earth Syst. Sci., 16(9):3383–3390.DOI
  18. [19]Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Bi Puranen, editors. 2022. World Values Survey: Round Seven – Country-Pooled Datafile Version 6.0. JD Systems Institute and WVSA Secretariat, Madrid, Spain and Vienna, Austria.
  19. [20]Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating large language models in generating synthetic HCI research data: A case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–19. ACM.DOI
  20. [21]Felix Hausdorff. 1914. Grundzüge der Mengenlehre. Veit & Comp., Leipzig.
  21. [22]Shirley Anugrah Hayati, Minhwa Lee, Dheeraj Rajagopal, and Dongyeop Kang. 2024. How far can we extract diverse perspectives from large language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5336–5366. Association for Computational Linguistics.arxiv.org/abs/2311.09799
  22. [23]John J Horton, Apostolos Filippas, and Benjamin S Manning. 2023. Large language models as simulated economic agents: What can we learn from Homo Silicus? Working Paper w31122, National Bureau of Economic Research, Cambridge, MA.
  23. [24]Tiancheng Hu, Joachim Baumann, Lorenzo Lupo, Nigel Collier, Dirk Hovy, and Paul Röttger. 2026. SimBench: Benchmarking the ability of large language models to simulate human behaviors. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2510.17516
  24. [25]D P Huttenlocher, G A Klanderman, and W J Rucklidge. 1993. Comparing images using the Hausdorff distance. IEEE Trans. Pattern Anal. Mach. Intell., 15(9):850–863.DOI
  25. [26]Stefan Krsteski, Giuseppe Russo, Serina Chang, Robert West, and Kristina Gligorić. 2026. Valid survey simulations with limited human data: The roles of prompting, fine-tuning, and rectification. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10887–10906. Association for Computational Linguistics.
  26. [27]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626. ACM.arxiv.org/abs/2309.06180
  27. [28]Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. 2019. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  28. [29]Anton D Lautrup, Tobias Hyrup, Arthur Zimek, and Peter Schneider-Kamp. 2025. SynthEval: a framework for detailed utility and privacy evaluation of tabular synthetic data. Data Min. Knowl. Discov., 39(1):6.
  29. [30]Olivier Ledoit and Michael Wolf. 2004. A well-conditioned estimator for large-dimensional covariance matrices. J. Multivar. Anal., 88(2):365–411.

Paper details

vlnc:W_aazmwcgc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 14

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents