betaFeedback

WorksEconomics, Econometrics and Finance

Decision Models as Scalable Economic Surrogates that Capture Human Variation

Abstract

Decision Models (DMs), a new class of AI models, give fast responses and return a probability distribution over a fixed set of options. These properties make them candidate human surrogates in economic simulations and surveys, a role increasingly filled by Large Language Models (LLMs), which are costly at scale and under-represent the variation in human behaviour. We present a first evaluation of DMs as human surrogates in economics, comparing 10 DMs with 8 LLMs in three economic settings: a household spending survey on consumption, several classic economic games and a labour-market field experiment. Frontier DMs, those that score highest on general decision-making benchmarks, reproduce the variation in human responses more closely than LLMs in all three settings. They capture the modal human response as well as LLMs in the spending survey, but less well in the economic games. Frontier DMs are also roughly ten times faster than the fastest LLM and can be over a hundred times cheaper. Together, these results suggest that DMs capture the heterogeneity of human behaviour better than LLMs and make large-scale economic simulations more feasible.

Introduction

AI agents are increasingly used as human surrogates in surveys or agent-based simulations, with recent applications spanning political preferences [3], consumer choices [46] and strategic economic games [32, 49]. AI agents address the core challenge in socio-economic modelling of balancing agent adaptability with behavioural realism. Traditional agent-based simulations rely on fixed, hand-written behavioural rules [22]. Because people adapt when policy changes, these rules may not generalise to new policies, a problem known as the Lucas critique [44]. To address this critique, economic models have used perfectly optimising rational expectations agents, an assumption that does not align with real human behaviour [14, 37, 57]. While surrogates such as LLM-agents are imperfect approximations of humans [27], they can capture many human behavioural biases while adapting dynamically to policy changes [32]. They therefore provide a much-needed alternative to the unrealistic behavioural assumptions of the status quo, enabling simulations that can generate more reliable answers to policy questions.

However, using LLMs as human surrogates faces two major limitations. First, while LLMs often successfully replicate the modal human response, they fail to capture the full distribution of behaviour, leading to significantly less heterogeneity than is found in human populations [9, 59, 60]. This compromises economic modelling for policy, as human heterogeneity is a driver of both macroeconomic outcomes [39] and collective social dynamics [29, 60]. Although persona prompting and demographic conditioning can recover some of this lost variation [58], they require extensive sampling, which leads to the second challenge: operational latency and computational cost. Generating autoregressive text for multiple, sequential decisions across large-scale populations is slow and computationally expensive, forcing researchers to trade off the size of their simulated economies against agent diversity.

Decision Models (DMs), a rapidly growing class of AI models [21, 42, 1], are in a good position to address these challenges. They are fine-tuned to take as input natural language questions and scenarios and make decisions between provided options. Crucially, rather than generating free-form text, DMs are designed to directly assign probabilities to the available choices. They are also faster and cheaper than LLMs, potentially allowing simulations with orders of magnitude more agents. This makes them a promising candidate for representing human behaviour as they are adaptable, provide structured choice probabilities, are fast and can be sampled cheaply to generate variation in simulated decisions.

In this paper, we test whether DMs are a viable additional tool for representing humans in economic modelling. We choose our experiments so that we explore a range of economic settings (consumption, games and labour) and question types (binary choices vs multiple choices and qualitative vs quantitative). The human data we compare to varies from surveys to economic experiments and realistic field experiments. We consider survey answers on changes in consumption in response to a future earnings shock [23], a variety of classic economic games [49] and a field experiment on labour preferences [48]. By replicating these experiments and surveys with DMs and LLMs, we assess how well DMs are able to act as human surrogates for economic research compared to LLMs.

Our results show that “frontier” DMs, those that lead on standard decision-making benchmarks, capture the variation in human behaviour better than LLMs, and do so faster and more cheaply. In all three settings, frontier DMs are closer to the human distribution than LLMs on our distributional measures. In the consumption survey, they also capture the modal human response as well as LLMs, although on the economic games, LLMs represent the modal human response better. In the labour experiment, frontier DMs reproduce the gradual shift in human choices as a wage gap changes, whereas LLMs switch abruptly from one job to the other. Taken together, we find that frontier DMs can be an effective and scalable tool for modelling human behaviour, especially where capturing the variation in responses matters.

Related work

Our work fits into a broader agenda of integrating AI models into economic research. Reinforcement Learning has been used in macroeconomics [26] and firm competition environments [10]. A growing body of work has also studied LLMs, both to understand their behaviour and to assess whether they are viable human surrogates, with encouraging results across a range of single-shot and multi-turn games [32, 49, 47]. LLMs have also been used within agent-based models (ABMs), in diverse settings such as macroeconomics [41, 11, 24], status signalling [16] and trading markets [31, 19, 56], with platforms such as Concordia allowing such simulations to be built quickly and flexibly [58].

However, the use of LLMs as human surrogates remains contested: LLMs tend to capture only a narrow, modal part of the human distribution [9, 59, 60], and the outcomes of LLM-based ABMs can be sensitive to prompting [62]. Several works have therefore aimed to make LLMs better represent human distributions. For example, conditioning LLMs on personas or backstories can widen the part of the human distribution they capture [3, 52], although this approach can be brittle to the exact wording of the prompt [45].

Work on LLM-agent simulations has also been limited in scale, typically to populations of a few hundred or a thousand agents. However, complex systems such as the economy of a city or country have millions of agents. Some progress has been made on scaling LLM-based ABMs with the number of agents going over the million mark, although this remains the exception, not the rule. For example, Chopra et al. [2025] used archetypes to achieve scale, where a single query determines the behaviour of a group of agents. Alternatively, some ABMs increase scale by not having every agent act at every time-step [61]. However, these approaches still involve a trade-off between a more faithful simulation requiring more LLM calls and the time required for the simulation.

These limitations should nonetheless be weighed against the alternatives. Economics lacks a widely accepted model of human behaviour with the “full information rational expectations” and perfectly optimising agents assumptions2 still prevalent. This is despite substantial evidence from behavioural economics suggesting these assumptions are a poor model for humans [14, 13] (DellaVigna 2009). Although LLMs are not a perfect model, they do offer a sensible alternative.

DMs may also be a viable alternative surrogate for humans. We are, to the best of our knowledge, the first to research using DMs as human surrogates for economic modelling. The closest work to ours is Huang et al. [2026], who used Jev to make decisions inside a predator-prey simulation. DMs have also been considered for use in social sciences by Ibrahim and Zaki [2026], who found that DMs can be useful as a first step in annotation pipelines. On DM evaluation, Asseri and Asseri [2026] used the Values Survey Module 2013 to find that Jev, one of the most popular DMs, defaults to American values. However, we expect the body of work on DMs to expand quickly as they become more entrenched in the AI ecosystem.

Methods

Experiments

We start with a survey of US households run every 4 months by the Federal Reserve Bank of New York. We use questions from the Household Spending Survey, which ask how an unexpected 10% increase and 10% decrease in future earnings would impact household consumption [23]. For each question, there are 7 possible answers, such as “Spend some and save some” or “Cut spending by some and cut savings by some”. Further details can be found in Appendix § 7.§ 1. We randomly select 1000 households from the 2020-2024 data in this experiment.

Economic literature suggests that stated preferences in surveys do not always accurately reflect revealed preferences, what people actually do [17, 43, 50]. Therefore we also consider two revealed preference datasets, where respondents take actions and their preference is then inferred from these actions. We start with economic games, where a short scenario is presented to a participant and they must choose their action. For example, in the “Bomb Risk Game”, the participant is given 100 boxes. They win a dollar value proportional to the number of these 100 boxes they decide to open but hidden in one of these boxes is a “bomb”; if the bomb is in a box they choose, they receive nothing. Participants then choose how many boxes to open as a number from 0-100.

We use five economic games found in Mei et al. [2024]. Two of the five games have two roles, so there are a total of 7 “positions” to test from these games. We use the following games which test the attributes in brackets: Dictator Game (altruism), Ultimatum Game (two positions testing fairness and spite), Trust Game (two positions testing trust, fairness, altruism and reciprocity), Bomb Risk Game (risk aversion) and Public Goods Game (free-riding, altruism and cooperation). Details of these games are in Appendix § 7.§ 1. In all 7 positions, participants choose between numbers 0-100 or 0-150 depending on the position.

For the third dataset, we use the natural field experiment of Mas and Pallais [ [48], which is the closest setting to real human choices we consider. In this experiment, participants applied to a call centre job and as part of the application were asked to choose between two jobs. Both jobs involved identical work but had differing working arrangements: one job was always a standard 9-5, 40 hour week job done in person at an office, while the other had an alternative working arrangement. The five alternatives were: flexible schedule (changing times and days of work), flexible number of hours (up to 40 hours a week), the option to work from home, a combined flexible option with all of these, and an arrangement where the employer chose hours and schedule one week in advance. As applicants saw these two jobs, the wages of the two jobs were also varied, so for example the standard job might be paid at $19 per hour, while having the option to work from home paid $16.5 per hour.

Once these choices are made, we fit a logistic curve to the data to give the proportion of respondents who chose the alternative schedule as the wage gap between the alternative and standard job varied. This gives “Willingness to Pay” (WTP) curves which are effectively cumulative distributions describing the proportion of respondents who would choose the alternative if the alternative paid xx dollars less (more) than the standard job. However, Mas and Pallais [ [48] found that a significant proportion (25-30%) of human participants were not paying attention and choosing at random, estimated by the number of people who chose the less flexible job even when it paid $5 less. All WTP calculations therefore first estimate and remove this inattention before fitting the curve. Details are in Appendix § 7.§ 1.

Running experiments on DMs and LLMs

DMs, also known as “System 1” models in reference to Kahneman’s intuitive decision making [Kahneman, 2011], are fast, cheap models specialised in making decisions. They take a natural language description of a situation, a question and a set of options and return a probability for each option supplied in one forward pass. These probabilities form a distribution over the supplied options by construction. The types of decisions they can make are multiple choice (“choice” format), a level on an ordered scale (“score” format) or a binary yes/no (“noul” format). Technical summaries are thin but the interested reader is referred to [64, 28, 34].

We consider 10 DMs across a range of providers. These models span capability levels in standard Decision benchmarks, and we use the Artificial Analysis Decision Index to define a group of “frontier” DMs, consisting of Jey, D1 Ultra, Mammoth and Perplexity [4, 38]. These four models are the four highest scoring DMs of those we test on the Artificial Analysis Decision Index. Model details and a discussion of DM benchmarks are in Appendix § 7.§ 2.

For experiments in Mei et al. [ [49] with numerical answers (i.e. pick a number between 0-100), we define 10 evenly spaced bins. We test DM responses as both a choice over the middle of the bins (5,15,25 etc) or a score over the bins (“0-9”, “10-19” etc). The difference between these DM response modes is that scores remain ordered and the DM knows that bin 2 is after bin 1. In a choice, the ordering of the bins does not matter. For all other experiments, we use a simple choice as in the original experiments, either between two options for Mas and Pallais [2017] or 7 options for Federal Reserve Bank of New York [2026].

For LLMs, we use Mei et al. [ [49]’s method of eliciting answers and consider a total of 8 LLMs, of which for two we also look at varying their reasoning parameters. These models span a range of sizes and include models from the GPT, Claude, Gemini, Gemma, GLM and Qwen families. We use recommended or default settings for LLMs. Details of the models and their answer formats are in Appendix § 7.§ 2.

Since neither DMs nor LLMs give deterministic outputs, we average results over repeats of the experiments. The consumption survey has demographic data attached to each answer and so we create 1000 personas using data on age, sex, race, US state, education, marriage, whether they are employed and income bracket. We prompt models with these personas as well as the question. Since we consider 1000 separate personas, we do not do repeats for each persona. For the economic games, LLMs and humans give continuous numerical answers while DMs can only answer with a choice between 10 bins. We therefore bin the LLM and human answers into these 10 bins for comparison. We do 50 repeats for each game and every DM and LLM. DM distributions are then averaged over the repeats for each game while LLM distributions are simply the binned distribution of the 50 LLM answers per game. In the labour choice experiment, every model makes each choice 6 times. We consider a high and a low wage regime and also ask questions with the binary choices in both orders. We consider 29 different wage gaps, meaning that each of the 5 treatments has 696 responses in total per model.

Measuring agreement with human results

We use a Turing Test, following Mei et al. [ [49], to measure how well a model captures the modal human answers. This test asks: what is the probability that if we pick a random model data point and a random human data point, the model data point will be more typical under the human distribution? Unlike in Mei et al. [ [49], we calculate the exact expected win rate of the random model data point rather than randomly sampling and recording the wins. The Turing Test rewards picking the modal human outcome, since a model which only ever picks the modal human outcome can never be less likely than another data point under the human distribution. However, a high win rate does not mean a good distributional fit; a model which only picks the modal human answer ignores the rest of the human distribution but scores highly on the Turing Test. This is why some models seem to “beat” humans in being more human-like.

For one question (or game and role), let the answer options be k=1,…,Kk = 1,\ldots,K. Let pkp_k be the share of humans who chose option kk, and qkq_k the share of a model’s answers on option kk. Draw one model answer a∼qa \sim q and, independently, one human answer h∼ph \sim p. The Turing Test win-rate is then:

W=Pr⁡(pa>ph)=∑k=1K∑l=1Kqkpl1{pk>pl}.(1)W = \Pr(p_a > p_h) = \sum_{k=1}^{K} \sum_{l=1}^{K} q_k p_l \mathbf{1}\{p_k > p_l\}. \tag*{(1)}

We compare the DM and LLM Turing Test scores with the human “self-play” score, where qq is set equal to pp.

We use two measures of distributional fit throughout the paper. Where options are naturally ordered, as in the economic games and the labour choices, we use the Wasserstein-1 distance, also called the “earth-movers” distance. Where appropriate, we check these results also hold with Kullback-Leibler (KL) divergence. A lower Wasserstein-1 distance denotes a closer agreement with the target human distribution, and we normalise over the width of the range of answers to make all answers comparable. By definition, the Wasserstein-1 distance of the human distribution from itself is 0. The Wasserstein-1 distance between two probability distribution functions, uu, vv, with Cumulative Distribution Functions, UU, VV, is defined as:

W1(u,v)=∫−∞∞∣U(x)−V(x)∣ dx.(2)W_1(u,v) = \int_{-\infty}^{\infty} \lvert U(x) - V(x) \rvert\,\mathrm{d}x. \tag*{(2)}

For comparison of two probability distributions where the choices have no natural distance measure between them (as in Federal Reserve Bank of New York [2026]), we use KL divergence. As with the Wasserstein-1 distance, closer to 0 is a better fit and humans have a KL divergence of 0 from themselves by definition. The KL divergence between two probability distribution functions, uu, vv, is defined as:

DKL(u∥v)=∑i=1nuilog⁡uivi.(3)D_{\mathrm{KL}}(u \mathbin{\|} v) = \sum_{i=1}^{n} u_i \log\frac{u_i}{v_i}. \tag*{(3)}

To account for the fact that the KL divergence can go to infinity if any option has zero samples, we mix all distributions with 1% of a uniform distribution.

Results

Consumer spending

Starting with the survey questions in Federal Reserve Bank of New York [2026], Figure 1 shows that frontier DMs capture human responses more closely than both LLMs and other DMs. On the mean KL divergence across the two questions, every frontier DM is closer to humans than every LLM: all 36 pairwise comparisons (4 DMs, 9 LLMs) are significant at the 1% level under a two-sided bootstrap test with Holm correction.

Figure 1. Frontier DMs better represent humans than LLMs in two consumer spending survey questions. KL divergence is compared to the Turing Test. DMs are solid circles while LLMs are hollow squares. Scores are averaged over the two consumer spending survey questions. Qwen (NT) is Qwen with no thinking, and Gemini Flash (LT) is Gemini Flash with low thinking.

The Turing test is more mixed: using the same significance tests as above, frontier DMs have significantly higher win rates in 6 tests, significantly lower in 7 and no significant difference in 23 tests compared to the 9 LLMs. On this experiment, frontier DMs and LLMs do not differ systematically in capturing the modal human, as measured by the Turing Test. Details of all significance test results, including those showing that frontier DMs are better than the Uniform baseline on both measures, are in Appendix § 8.§ 1. The Turing Test has fewer significant tests because its win rates are far more sensitive to the resampling from the bootstrap procedure than the KL divergence metric, meaning that the errors on the metrics are wider. This sensitivity is a good reason to include distributional measures like KL divergence, which are less binary than a win or no-win outcome and therefore vary more smoothly with resampling than the Turing test.

We consider a range of robustness checks, looking at minor changes to the wording of one question, asking the models to “act like a human” and exploring the impact of the system prompt for LLMs. We run these checks on the four frontier DMs (Mammoth, Jev, D1 Ultra and Perplexity) and three LLMs (Gemini Argon, GLM Flash and Sol) and find that our results are robust to these changes; full details are in Appendix B.1.

Turning to non-frontier DMs, Laya and AnyJev perform close to the Uniform benchmark on both Turing test and KL divergence. Other DMs such as Mercury or Luna Decisions, which are behind the DM frontier according to Artificial Analysis, beat the uniform baseline on the Turing test but only Luna Decisions has a lower KL divergence than the Uniform baseline. For this task, Luna Decisions is as good at capturing the human behaviour as the frontier DMs.

Most LLMs do not fit the human distribution well on this task. All LLMs except GLM Flash and Gemini Argon have a higher KL divergence than the Uniform benchmark (0.66 vs 1.22-2.47), suggesting that they tend to focus their answers more consistently on a narrower range of answers. However, GLM Flash and Argon both fit the human mode and distribution well, although they are still statistically significantly worse than frontier DMs on the KL divergence metric. On this task, frontier DMs (and Luna Decision) are therefore the most faithful human surrogates we test, matching both the spread of human answers and the typical answer.

Economic games

Figure 2 shows that frontier DMs capture the distribution of human results better than LLMs according to the mean Wasserstein-1 metric. Using a bootstrapping procedure and a two-sided test between all pairs of frontier DMs and LLMs (80 tests, with 4 DMs, 2 answer methods per DM and 10 LLMs), 79 tests show frontier DMs with statistically significantly lower mean Wasserstein-1 distance to the human distribution than LLMs (1% level with Holm correction, all 7 positions equal weights), and only 1 test shows no significant difference. As a robustness check using KL divergence rather than Wasserstein-1, all 80 tests show frontier DMs with statistically significantly lower KL divergence than LLMs; this is because the KL divergence penalises bins where a model has zero answers strongly, therefore skewing the results in favour of DMs. Details of these tests are in Appendix B.2.

Figure 2. Frontier DMs capture the full human distribution better. Mean results from the 7 positions in Mei et al. [2024]. Shows the Wasserstein-1 distance and the Turing Test results against the Human results and a Uniform benchmark. DMs are circles and LLMs are hollow squares. The frontier DMs are: Jev, D1 Ultra, Mammoth and Perplexity. Qwen (NT) is Qwen with no thinking, and Gemini Flash (LT) is Gemini Flash with low thinking.

However, LLMs have higher Turing Test win rates in these games. Using the same bootstrap procedure as above, all LLMs have statistically significantly higher Turing test win rates than the frontier DMs (full results in Appendix B.2). This shows that LLMs are better at concentrating near the modal human responses on average than DMs for these games. However, the fact that several LLMs have higher Turing Test win rates than humans also supports the results with the Wasserstein-1 distance; LLMs tend to overly concentrate near the modal human answer while DMs capture a wider spread of the human distribution.

We find that although using the score vs choice answer method does change DM results slightly, it does not do so in a consistent way across differing DMs.

Just as with LLMs, not all DMs are equal. The frontier DMs (Jev, D1 Ultra, Mammoth and Perplexity) generally have lower Wasserstein-1 distance and higher Turing test metrics than other DMs. All DMs which have statistically significantly lower Wasserstein-1 distance than the Uniform distribution are frontier DMs and frontier DMs also have a higher Turing test win rate than the baseline. As in § 3.1, Laya and AnyJev are close to the Uniform benchmark while the remaining DMs are somewhere between the Uniform benchmark and the frontier models.

The uniform baseline is a strong baseline on the Wasserstein-1 measure for this data, partially because the human results at times are also close to uniform. A good representation of human behaviour needs the combination of capturing the modal human answers as well as its broader distribution. LLMs focus on the modal response and in some senses, DMs therefore provide a middle ground between the uniform baseline and the modal-response focus of LLMs.

Labour choices

Figure 3 shows that frontier DMs outperform LLMs and other DMs on average in representing human responses on the labour field experiment of Mas and Pallais [2017].

Bootstrapped two-sided tests at 1% with Holm correction show that frontier DMs have statistically significantly lower mean Wasserstein-1 distances (averaged over the five treatments) than any LLM. The left panel from Figure 3 shows why: DMs tend to capture the smooth increase in proportion of humans choosing one job over another as the wage gap increases. In contrast, LLMs look much more like step functions, as once they decide that one option is better than another, they tend to repeat their choice consistently. Details of the statistical tests and graphs of the other treatment curves are in Appendix § 8.§ 3.

Figure 3. Frontier DMs better capture the smooth curve in the human results compared to LLMs on labour choices. Left panel shows DM results for the Work From Home treatment on top and the LLM results below, with the human data in black. The right panel shows the Wasserstein-1 distance between models and the human results across all 5 treatments. DMs are circled, with frontier DMs in bold. The full fitted curves for all five treatments are in Appendix § 8.§ 3. Qwen (NT) is Qwen with no thinking, and Gemini Flash (LT) is Gemini Flash with low thinking.

We omit Laya from this analysis as it does not give an upward sloping curve and has an inattention of near 100%, in keeping with our earlier results that Laya is roughly as good as a random uniform distribution at predicting human responses. However, AnyJev does better in these experiments than in the consumption survey and economic games: it generally gives a smooth curve which often matches the human data well (mean Wasserstein-1 distance of 1.46 vs best DM at 1.14 and best LLM at 1.55).

The above analysis compares the inattention-corrected curves between models and humans. Mas and Pallais [2017] estimated that around 25-30% of the humans in their experiments were inattentive and chose at random. However, DMs and LLMs, except Laya, all have low estimated inattention proportions, all less than 10% averaged over the five treatments with most near 0% (see Appendix § 8.§ 3). It is unsurprising that DMs and LLMs do not reproduce human inattention as these models have been fine-tuned for use in cases, such as customer ticket prioritisation, where inattention would be highly undesirable. However, the low inattention of DMs and LLMs is worth bearing in mind if considering using these models as human surrogates.

The ability of DMs to react sensibly and smoothly to small differences in prompts, in this case small wage differences, is critical to being able to use DMs as human surrogates. Since DMs do not reason in the same way that LLMs do, they need an “intuitive” understanding of numbers and this experiment shows that they are indeed able to give sensible results with numerical data, even though they cannot do extended reasoning.

Computational efficiency and its implications

DMs are faster and cheaper than LLMs; this holds across the three settings we consider and Figure 4 illustrates this with the mean seconds per call and cost per 1000 calls across all experiments. Focusing only on the frontier DMs which give the most realistic representations of humans, a DM like Jev is around 10-100 times faster across experiments compared to the LLMs we test. Similarly, the cheapest frontier DM, Perplexity, is around 10-300 times cheaper than the LLMs we test (excluding locally-run LLMs, which are “free”). However, Jev is only around 2 times cheaper than Luna LLM and these numbers are subject to evolution as pricing has already changed for several DMs since their release. Nonetheless, our results illustrate that frontier DMs give a significant boost in terms of both speed and cost compared to LLMs. Results on speed and cost for all models and all experiments are reported in Appendix § 8.§ 4.

Figure 4. DMs are faster and cheaper than LLMs. Averaged across all calls to the models across the three experiments with equal weight to each experiment. Mean seconds per call and mean dollars per 1000 calls are on log scales. Qwen (with thinking) is omitted as it was only run for the economic games, where its average time per call was 657 seconds and cost was 0 as it was run locally. Gemma is included in the locally run models, but was in fact run using an API, where it is free to use. Qwen (NT) is Qwen with no thinking, while Gemini Flash (LT) is Gemini Flash with low thinking.

These speed and cost advantages transfer into increasing the feasible size and speed of in silico surveys and simulations. In consumer research, LLMs have been used to simulate purchase intent for product concepts [46]. Because LLMs asked directly for a rating produce unrealistic response distributions, these studies elicited free text and mapped it onto a rating scale, whereas a DM assigns a probability to each rating in a single call. Assuming that the average costs and speeds in our experiments translate to other economic settings, recreating the 18,600 decisions in [46] would take under 10 minutes with DMs while LLMs take around 1-3 hours.

Just as LLMs have already started to be used in economic ABMs [41, 51], DMs offer an alternative for the agents in ABMs. Existing ABMs could be transformed to use DMs for decisions such as which occupations workers apply to [18], how suppliers ration scarce output [Pichler et al., 2022], or how firms set markups [55]. If we assume a simulation with 10,000 agents over 50 periods and a concurrency of 10 requests for all models, then in these ABMs, Perplexity would take around 4.5-13.5 hours and cost $3-9, against Luna or Haiku taking over a day for $25-70 and Sol or Argon at 1-4 days for $400-600 (see Appendix § 8.§ 4). If techniques such as archetypes were also used, then a simulation such as that in Chopra et al. [2025] could feasibly be scaled up from 8.4 million agents to around 80 million agents. Frontier DMs therefore offer a significant advantage compared to LLMs in scaling up economic surveys and computational simulations with AI agents.

Discussion and Conclusion

We examine whether Decision Models can act as human surrogates in economic research, comparing 10 DMs and 8 LLMs with human data from a household spending survey, classic economic games and a labour-market field experiment. Our experiments show that frontier DMs do a better job of representing the distribution of human behaviours than LLMs, and they do so in fractions of the time and cost. In the spending survey, frontier DMs represent the full human distribution better than LLMs without losing the ability to capture the modal human answer, while on labour choices they better capture the smooth curves seen in the human data. In the economic games, LLMs better capture the modal human answer than frontier DMs, but frontier DMs are still closer to humans on full distributional measures. Together, our results show that frontier DMs offer a scalable human surrogate which better captures the variation in human data.

While these results are promising, there are limitations to the experiments and data selected. The consumption and labour study data are US based. Moreover, since both LLMs and DMs were trained after the publication of the datasets and experiments used, there is likely data contamination. These limitations are partly addressed by the fact that the experiments in Mei et al. [2024] include data from outside the US, and that LLM predictions of experimental treatment effects can remain accurate for studies published after the model’s training cut-off [5]. Further, training after data publication does not guarantee that an LLM will replicate human behaviour [56]. Nonetheless, validating these models against unpublished, out-of-distribution choice experiments and culturally diverse populations remains essential.

As Decision Models continue to evolve, they present opportunities to develop more faithful human surrogates. For example, models such as NumericJev already support fine-grained numerical outputs alongside discrete choices [63]. However, general advances in model capabilities do not always produce better human surrogates; frontier models often become hyper-rational on economic benchmarks, making them less representative of boundedly rational humans [56]. Rather than relying on general model scaling, future work should fine-tune DMs explicitly on empirical human choice data, adapting behavioural calibration techniques previously applied to LLMs [8 Kolluri et al.,] 2025]. Because DMs have a modest parameter footprint and incur negligible inference costs, fine-tuning them specifically for behavioural fidelity offers a practical, scalable foundation for population-level economic simulation.

By addressing both the population-level scaling and restricted behavioural heterogeneity challenges of LLMs, Decision Models enable better large-scale computational simulations. While computational infeasibility has constrained LLM-based simulations, our results suggest that frontier DMs run approximately 10 to 100 times faster than LLMs. This efficiency allows existing agent-based simulations to scale by an order of magnitude, for example, expanding populations from 1 million to 10 million agents, or compressing the execution time of complex macroeconomic models from days to hours. Achieving 1-to-1 agent representations at a city or national scale at highly accessible costs enables practical applications across both public policy, such as modelling labour reallocation and supply-chain shocks, and commercial operations, where DMs can efficiently simulate heterogeneous consumer demand. In an era where AI is rapidly reshaping society, leveraging computer science to build computationally tractable and behaviourally faithful human surrogates is a promising tool for socio-economic modelling and policymaking.

Acknowledgements

We acknowledge Google for providing API credits for use in running Gemini experiments.

Disclosure of AI Usage

The authors declare the use of generative AI in the research and writing process. According to the GAIDeT taxonomy (2025), the following tasks were completed with assistance from GAI under full human supervision: code generation & optimisation, proofreading & editing. The tool used was Claude Opus 5.5 (Anthropic). After using these tools, the authors fully reviewed and edited all code and material. Responsibility for the final manuscript lies entirely with the authors.

Methods

Experiments

Wording for the Household Spending Survey:

  • Suppose next year you were to find your household with 10% more income than you currently expect. What would you do with the extra income?

    1. Save or invest all of it

    2. Spend or donate all of it

    3. Use all of it to pay down debts

    4. Spend some and save some

    5. Spend some and use part of it to pay down debts

    6. Save some and use part of it to pay down debts

    7. Spend some, save some and use some to pay down debts

  • Imagine that next year you were to find yourself with 10% less household income. What would you do?

    1. Cut spending by the whole amount

    2. Not cut spending at all, but cut my savings by the whole amount

    3. Not cut spending at all, but increase my debt by borrowing the whole amount

    4. Cut spending by some and cut savings by some

    5. Cut spending by some and increase debt by some

    6. Cut savings by some and increase debt by some

    7. Cut spending by some, cut savings by some and increase debt some

Brief explanation of each game, taken verbatim from Mei et al. [2024]:

  • A Dictator Game - given an endowment of money, one player (the dictator) chooses how much of the money to keep and how much to donate to a second player. This involves altruism [30, 25].

  • An Ultimatum Game - given an endowment of money, one player (the proposer) offers a split of the money to a second player (the responder) who either accepts the split or rejects it in which case neither player gets anything. This involves fairness and spite [30].

  • A Trust Game — given an endowment of money, one player (the investor) decides how much of the money to keep and passes the remainder to a second player (the banker), which is then tripled. The banker decides how much of that tripled revenue to keep and returns the remainder to the investor. This involves trust, fairness, altruism, and reciprocity [7].

  • A Bomb Risk Game — a player chooses how many boxes out of 100 to open and the player is rewarded for each opened box but loses everything if a randomly placed bomb is encountered. This involves risk aversion [15].

  • A Public Goods Game — given an endowment of money, a player chooses how much of the money to keep and how much to contribute to a public good and receives half of the total amount donated to the public good by all four players. This involves free-riding, altruism, and cooperation [2].

How to fit inattention and WTP logistic curves in Mas and Pallais [2017]:

Let gig_i be position A’s hourly wage minus position B’s in observation ii. Let yiy_i be 1 if A was chosen and 0 otherwise; for decision models, which return probabilities, yiy_i is the probability of choosing A.

Inattention. Following Mas and Pallais [2017], we allow a share of choices to ignore the wage gap. We regress yiy_i on gig_i by OLS using only gaps of $2 or more, yi=β0+β1gi+ϵiy_i = \beta_0 + \beta_1 g_i + \epsilon_i. Inattention is the predicted share not choosing A when it pays $5 more:

α^=1−(β^0+5β^1),\hat{\alpha} = 1 - (\hat{\beta}_0 + 5\hat{\beta}_1),

clipped to [0,0.49][0, 0.49].

Curve. Given α^\hat{\alpha}, the probability of choosing A is

P(A∣g)=α^+(1−2α^)Λ(bg+c),Λ(x)=11+e−x.P(A \mid g) = \hat{\alpha} + (1 - 2\hat{\alpha}) \Lambda(bg + c), \qquad\Lambda(x) = \frac{1}{1 + e^{-x}}.

We estimate bb and cc by maximum likelihood, maximising ∑i[yilog⁡P(A∣gi)+(1−yi)log⁡(1−P(A∣gi))]\sum_i \left[y_i \log P(A \mid g_i) + (1-y_i) \log\left(1-P(A \mid g_i)\right)\right] by Fisher scoring with step halving. For plotting, we plot inattention-corrected curves using the above inattention-corrected fitted parameters.

Models

Model NameModelProviderCost $/M in/out | Device
Jevjev-1.13.0TypeSafe0.042 / 0.00API
Perplexitypplx-decider-v1.1-27bPerplexity0.02 / 0.00API
Mammothbaseten-mammoth-27bBaseten0.10 / 0.00API
D1 Ultrad1-ultraLiquid0.042 / 0.00API
Luna Decisionsgpt-6-lunaOpenAI0.10 / 0.00API
Mercurymercury-decideInception0.02 / 0.00API
Microsoft Decision 1microsoft-decision-1Microsoft0.042 / 0.00API
Clef Omniclef-omniCloudflare0.15 / 0.00API
AnyJevQwen3.5-4BNokia-GPU
LayalayaConvAI-GPU

Table 1. Decision Models used. The GPU used was a Quadro RTX 6000.

Model NameModelProviderCost $/M in/out | Device
Gemini Flashgemini-3.8-flashGoogle0.75 / 3.75API
Gemini Argongemini-4-argonGoogle2.0 / 10.0API
Gemmagemma-4-26b-a4b-itGoogle-API
Luna LLMgpt-6-lunaOpenAI0.1 / 0.5API
Solgpt-6-solOpenAI2.0 / 10.0API
Haikuclaude-haiku-5-5Anthropic0.1 / 0.5API
GLM Flashglm-5.3-flashZ.AI0.15 / 0.50API
QwenQwen/Qwen3.5-4BAlibaba-GPU

Table 2. LLMs used. APIs are all the provider’s standard API interface. The GPU used was a Quadro RTX 6000 except for Qwen with thinking, where a mixture of a Quadro RTX 6000 and an Apple M4 were used to increase the speed of the runs. The Quadro RTX 6000 averaged 361 seconds per call and the M4 around 713 on similar length outputs. Both are several orders of magnitude slower than any of the DMs.

Gemini-3.8-Flash and GPT-6-Luna were run with their default settings, except when changing Gemini-3.8-Flash’s reasoning to “low”. Qwen3.5-4B was run using Hugging Face transformers, bfloat16. With thinking on, the settings were: temperature 1.0, top-pp 0.95, top-kk 20 (as recommended). With thinking off, the settings were: temperature 0.7, top-pp 0.8, top-kk 20 (again according to the model card recommendation).

DM benchmarks

We use the Artificial Analysis Decision Index to judge if a Decision Model is at the frontier of decision tasks. Not all models have been tested by Artificial Analysis; the scores, where available, are below:

Model NameArtificial Analysis Decision Index
Jev57
Perplexity56
Mammoth65.9
D1 Ultra65
Luna Decisions48
Mercury53
Microsoft Decision 154
Clef Omni44
AnyJevNot run
LayaNot run

Table 3. Decision Index scores. Jev score is in fact Jev 1.14, a slightly newer version than we use. Artificial Analysis have not run Jev 1.13 on their benchmark, but TypeSafe, the creators of Jev, find little difference in score between the two models on their own benchmark. We therefore make an assumption that the same holds true on the Artificial Analysis benchmark. The results for Mammoth are those reported by the creators of Mammoth rather than those reported by Artificial Analysis (since they have not been run by Artificial Analysis).

LLM prompting:

We follow Mei et al. [ [49] and ask LLMs to reply with: “Please give only one choice, and highlight the position you choose with [] (such as [Position 1]).” For consistency with Mei et al. [ [49], LLMs are given the system prompt: “You are a helpful assistant.”

Results

Consumer spending

LLMJevPerplexityMammothD1 Ultra
Panel A: KL divergence from humans (bits)
Qwen (NT)2.08*2.04*2.23*2.13*
Gemini Flash0.89*0.84*1.03*0.93*
Gemini Flash (LT)0.84*0.79*0.98*0.88*
Luna LLM1.23*1.18*1.37*1.27*
Sol1.54*1.49*1.68*1.58*
Gemma2.13*2.08*2.27*2.17*
GLM Flash0.17*0.12*0.31*0.21*
Gemini Argon0.17*0.12*0.31*0.21*
Haiku2.08*2.03*2.22*2.12*
Decision model closer: 36; LLM closer: 0; no significant difference: 0.
Panel B: Turing test win rate (%)
Qwen (NT)10.8*13.9*8.412.8*
Gemini Flash4.77.7*2.26.6
Gemini Flash (LT)6.29.2*3.88.1
Luna LLM0.33.3−2.2*2.2
Sol−8.8*−5.8*−11.3*−6.9*
Gemma5.08.02.56.9
GLM Flash−3.1*−0.0−5.5*−1.1
Gemini Argon3.26.3*0.85.2
Haiku2.75.70.34.6
Decision model closer: 6; LLM closer: 7; no significant difference: 23.

Table 4. Paired bootstrap tests of each DM against each LLM, averaged over the two SCE spending questions. Panel A tests which model has a lower KL divergence while Panel B tests which model has a higher Turing Test score. Each cell is in bits (Panel A) or percentage points (Panel B), averaged over the 2 questions with equal weights: positive if the decision model is lower in KL divergence, higher in Turing Test. * Significant at the 1% level (two-sided paired bootstrap test, 10,000 draws, Holm-corrected over each panel’s tests). Each draw resamples the 1,000 personas with replacement.

Robustness checks

The first robustness check looks at changing the wording for the question asking for a response after a 10% reduction in earnings. This is considered because in the original survey the two questions are asked one after the other. However, since the DMs and LLMs get each question separately, the exact survey wording for the second question no longer made logical sense, and had to be slightly changed for these experiments. We therefore consider two versions of the prompt, the first of which is the prompt we use throughout the paper and the second is the robustness check:

Decision modelKL divergence from humans (bits)Turing test (%)
Jev0.28*12.4*
D1 Ultra0.32*14.3*
Perplexity0.23*15.4*
Mammoth0.42*9.9*

Table 5. Paired bootstrap tests of the frontier DMs against the uniform distribution, averaged over the two SCE spending questions. Values are as above, with a higher number meaning DMs have a lower KL divergence or higher Turing test scores. * Significant at the 1% level (two-sided paired bootstrap test, 10,000 draws, Holm-corrected). Each draw resamples the 1,000 personas with replacement.

  1. “Imagine that next year you were to find yourself with 10% less household income. What would you do?”

  1. “Suppose next year you were to find your household with 10% less income than you currently expect. What would you do?”

The second check considers adding the instruction to “Act like a human” before prompts for both DMs and LLMs. The third robustness check considers only LLMs. LLMs are prompted in the same way as in Mei et al. [ [49]. However, this includes adding a system prompt of “You are a helpful assistant”. We look at taking this system prompt out for LLMs.

We find that our results are very robust to these three considered changes, with frontier DMs still closer to humans on KL divergence and mixed results with the Turing Test. Figure 5 shows the results.

Figure 5. Our results are robust to three different robustness checks. DMs are solid while LLMs are hollow. Human is the test where models are told to act like a human. Nosystem removes the system prompt for the LLMs while wording changes the wording for the loss of income question. Scores are averaged over the two consumer spending survey questions with equal weight.

Economic Games

LLMJevPerplexityMammothD1 Ultra
Panel A: Wasserstein-1 distance (% of range)
Qwen1.010.5*9.5*9.4*
Qwen (NT)9.8*19.4*18.3*18.3*
Gemini Flash8.3*17.9*16.9*16.8*
Gemini Flash (LT)10.3*19.9*18.9*18.8*
Luna LLM2.7*12.2*11.2*11.1*
Sol9.3*18.9*17.8*17.8*
Gemma6.1*15.7*14.6*14.5*
GLM Flash5.4*15.0*14.0*13.9*
Gemini Argon7.6*17.1*16.1*16.0*
Haiku3.1*12.6*11.6*11.5*
Decision model closer: 39; LLM closer: 0; no significant difference: 1.
Panel B: Turing test win rate (%)
Qwen−28.0*−29.1*−28.0*−27.1*
Qwen (NT)−9.0*−10.0*−9.0*−8.1*
Gemini Flash−20.9*−22.0*−20.9*−20.0*
Gemini Flash (LT)−22.0*−23.0*−21.9*−21.1*
Luna LLM−22.6*−23.6*−22.5*−21.6*
Sol−21.3*−22.3*−21.2*−20.3*
Gemma−24.9*−25.9*−24.8*−23.9*
GLM Flash−21.4*−22.4*−21.3*−20.4*
Gemini Argon−17.9*−19.0*−17.9*−17.0*
Haiku−21.6*−22.7*−21.6*−20.7*
Decision model closer: 0; LLM closer: 40; no significant difference: 0.

Table 6. Bootstrap tests of frontier DMs, using the choice answer, against each LLM, averaged over Mei et al. [2024] games. Each cell is how much lower the Wasserstein-1 distance or higher the Turing Test the decision model is than the LLM, in percentage points, averaged over the 7 games with equal weights: positive if the decision model is lower Wasserstein-1 or higher Turing test. * Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected over each panel’s tests). Each model’s calls are resampled with replacement in each game for the bootstrap.

LLMJevPerplexityMammothD1 Ultra
Panel A: Wasserstein-1 distance (% of range)
Qwen5.6{sup}*`9.1{sup}*`9.2{sup}*`7.5{sup}*`
Qwen (NT)14.4{sup}*`17.9{sup}*`18.1{sup}*`16.3{sup}*`
Gemini Flash13.0{sup}*`16.4{sup}*`16.6{sup}*`14.9{sup}*`
Gemini Flash (LT)15.0{sup}*`18.4{sup}*`18.6{sup}*`16.8{sup}*`
Luna LLM7.3{sup}*`10.8{sup}*`10.9{sup}*`9.2{sup}*`
Sol13.9{sup}*`17.4{sup}*`17.6{sup}*`15.8{sup}*`
Gemma10.7{sup}*`14.2{sup}*`14.4{sup}*`12.6{sup}*`
GLM Flash10.1{sup}*`13.5{sup}*`13.7{sup}*`12.0{sup}*`
Gemini Argon12.2{sup}*`15.7{sup}*`15.9{sup}*`14.1{sup}*`
Haiku7.7{sup}*`11.2{sup}*`11.4{sup}*`9.6{sup}*`
Decision model closer: 40; LLM closer: 0; no significant difference: 0.
Panel B: Turing test win rate (%)
Qwen−25.3{sup}*`−25.9{sup}*`−24.3{sup}*`−25.5{sup}*`
Qwen (NT)−6.3{sup}*`−6.9{sup}*`−5.3{sup}*`−6.5{sup}*`
Gemini Flash−18.2{sup}*`−18.8{sup}*`−17.2{sup}*`−18.4{sup}*`
Gemini Flash (LT)−19.3{sup}*`−19.8{sup}*`−18.3{sup}*`−19.5{sup}*`
Luna LLM−19.9{sup}*`−20.4{sup}*`−18.9{sup}*`−20.0{sup}*`
Sol−18.6{sup}*`−19.1{sup}*`−17.6{sup}*`−18.7{sup}*`
Gemma−22.2{sup}*`−22.7{sup}*`−21.2{sup}*`−22.3{sup}*`
GLM Flash−18.7{sup}*`−19.2{sup}*`−17.7{sup}*`−18.8{sup}*`
Gemini Argon−15.2{sup}*`−15.8{sup}*`−14.2{sup}*`−15.4{sup}*`
Haiku−18.9{sup}*`−19.5{sup}*`−17.9{sup}*`−19.1{sup}*`
Decision model closer: 0; LLM closer: 40; no significant difference: 0.

Table 7. Bootstrap tests of frontier DMs, using the score answer, against each LLM, averaged over Mei et al. [2024] games. Each cell is how much lower the Wasserstein-1 distance or higher the Turing Test the decision model is than the LLM, in percentage points, averaged over the 7 games with equal weights: positive if the decision model is lower Wasserstein-1 or higher Turing test. {sup}*` Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected over each panel’s tests). Each model’s calls are resampled with replacement in each game for the bootstrap.

LLMJevPerplexityMammothD1 Ultra
Panel A: KL divergence (bits)
Qwen2.24*2.83*2.75*2.89*
Qwen (NT)3.81*4.40*4.32*4.46*
Gemini Flash3.51*4.10*4.02*4.16*
Gemini Flash (LT)3.80*4.38*4.30*4.44*
Luna LLM2.66*3.25*3.16*3.31*
Sol3.55*4.14*4.05*4.20*
Gemma2.75*3.34*3.26*3.40*
GLM Flash2.88*3.47*3.38*3.53*
Gemini Argon3.81*4.40*4.32*4.46*
Haiku2.71*3.29*3.21*3.35*
Decision model closer: 40; LLM closer: 0; no significant difference: 0.

Table 8. Bootstrap tests of frontier DMs, using the choice answer, against each LLM, averaged over Mei et al. [ [49] games. Each cell is how much lower the KL divergence of the decision model is than the LLM, averaged over the 7 games with equal weights: positive if the decision model is lower KL divergence. * Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected). Each model’s calls are resampled with replacement in each game for the bootstrap.

LLMJevPerplexityMammothD1 Ultra
Panel A: KL divergence (bits)
Qwen2.61*2.95*2.96*2.93*
Qwen (NT)4.18*4.52*4.53*4.50*
Gemini Flash3.88*4.22*4.23*4.20*
Gemini Flash (LT)4.16*4.50*4.52*4.48*
Luna LLM3.03*3.37*3.38*3.35*
Sol3.92*4.26*4.27*4.24*
Gemma3.12*3.46*3.47*3.44*
GLM Flash3.25*3.59*3.60*3.57*
Gemini Argon4.18*4.52*4.53*4.50*
Haiku3.07*3.41*3.42*3.39*
Decision model closer: 40; LLM closer: 0; no significant difference: 0.

Table 9. Bootstrap tests of frontier DMs, using the score answer, against each LLM, averaged over Mei et al. [ [49] games. Each cell is how much lower the KL divergence of the decision model is than the LLM, averaged over the 7 games with equal weights: positive if the decision model is lower KL divergence. * Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected). Each model’s calls are resampled with replacement in each game for the bootstrap.

Decision modelWasserstein-1 distance (% of range)Turing test (%)
Jev (choice)−7.2*5.3*
Jev (score)−2.5*8.0*
D1 Ultra (choice)1.3*6.3*
D1 Ultra (score)−0.7*7.9*
Perplexity (choice)2.4*4.3*
Perplexity (score)0.9*7.5*
Mammoth (choice)1.3*5.4*
Mammoth (score)1.1*9.0*

Table 10. Bootstrap tests of frontier DMs against the uniform distribution. Each cell is how much lower the Wasserstein-1 distance or higher the Turing Test the decision model is than the uniform distribution, averaged over the 7 games with equal weights: positive if the decision model is lower Wasserstein-1 or higher Turing test. * Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected). Each model’s calls are resampled with replacement in each game for the bootstrap.

Labour choices

LLMJevPerplexityMammothD1 Ultra
Qwen (NT)0.80{sup}*0.96{sup}*0.79{sup}*0.89{sup}*
Gemini Flash0.52{sup}*0.68{sup}*0.51{sup}*0.61{sup}*
Gemini Flash (LT)0.52{sup}*0.68{sup}*0.51{sup}*0.61{sup}*
Luna LLM0.85{sup}*1.01{sup}*0.84{sup}*0.94{sup}*
Sol0.26{sup}*0.42{sup}*0.25{sup}*0.35{sup}*
Gemma0.40{sup}*0.56{sup}*0.39{sup}*0.49{sup}*
GLM Flash0.40{sup}*0.56{sup}*0.39{sup}*0.49{sup}*
Gemini Argon0.53{sup}*0.69{sup}*0.52{sup}*0.62{sup}*
Haiku0.37{sup}*0.52{sup}*0.35{sup}*0.46{sup}*
Decision model closer: 36; LLM closer: 0; no significant difference: 0.

Table 11. Bootstrap tests of frontier DMs against each LLM. Each cell is how much closer to the applicants’ fitted WTP distribution the decision model’s is than the LLM’s, in $ per hour of Wasserstein-1 distance (the area between the fitted choice curves without inattention), averaged over the 5 treatments with equal weights: positive if the decision model is closer. {sup}* Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected over the tests).

Figure 6. Fitted inattention-corrected curves for all 5 treatments. Job A is the more flexible alternative, so in 4 treatments it is the alternative and when the treatment is employer discretion, it is the standard 9-5 40 hour job. The left column shows frontier DMs, while the right column shows all LLMs.

FSEDWFHFHCFMean
Laya0.49{sup}†`0.49{sup}†`0.43{sup}†`0.49{sup}†`0.41{sup}†`0.46
Jev0.000.000.000.000.000.00
AnyJev0.000.030.000.010.000.01
Luna Decisions0.000.010.000.000.000.00
Perplexity0.000.010.000.000.000.00
Mammoth0.030.070.020.080.030.05
Mercury0.000.000.000.000.000.00
D1 Ultra0.030.020.010.020.010.02
Microsoft0.000.000.000.000.000.00
Clef Omni0.020.020.030.020.030.02
Qwen (NT)0.000.000.000.000.000.00
Gemini Flash0.000.000.000.000.000.00
Gemini Flash (LT)0.000.000.000.000.000.00
Luna LLM0.000.000.000.000.000.00
Sol0.000.000.000.000.000.00
Gemma0.000.000.000.000.000.00
GLM Flash0.000.000.000.000.000.00
Gemini Argon0.000.000.000.000.000.00
Haiku0.000.000.000.000.000.00

Table 12. Estimated inattention in each of Mas and Pallais [2017]’s treatments. Inattention α\alpha in the inattention-corrected logit P(A∣gap)=α+(1−2α)F(b gap+c)P(A \mid\mathrm{gap}) = \alpha+ (1 - 2\alpha)F(b\,\mathrm{gap} + c): the share not choosing A when it pays $5 more, from a linear fit of the choices at wage gaps of $2 to $5. Comparison to the human average of 25-30% then requires the above estimate of α\alpha to be doubled. Mean is the average over the 5 treatments with equal weights. {sup}†` Above 0.4, so no WTP is fitted. We use the following abbreviations to save space: FS (Flexible Schedule), ED (Employer Discretion), WFH (Work From Home), FH (Flexible Hours), CF (Combined Flexible).

Speed and cost analysis

We estimate the cost and time of using DMs and LLMs as human surrogates in the applications discussed in § 3.4, using the cost and time per call of each model in Table 13. A simulation with NN calls to a model with cost cc and time τ\tau per call costs NcNc and, with KK concurrent requests, takes Nτ/KN\tau/K to run. We set K=10K = 10 for all models. In an ABM, NN is the product of the number of agents, periods and calls per decision. Periods run in sequence, but each period contains 10410^{4} decisions against 10 concurrent requests, so the sequencing does not change the estimate. We assume that every agent decides in every period, which is an upper bound for models in which only some agents act, and that prompts are of similar length to those in our experiments. Calibration and Monte Carlo analysis multiply all figures by the number of runs.

Mean seconds per callMean $ per 1,000 calls | | |
SpendingGamesLabourMeanSpendingGamesLabourMean
Laya0.030.030.030.030.0000.0000.0000.000
Jev0.260.240.210.240.0240.0220.0200.022
AnyJev2.811.940.751.830.0000.0000.0000.000
Luna Decisions0.220.220.240.230.0330.0310.0280.030
Perplexity0.330.310.330.320.0060.0060.0050.006
Mammoth0.240.430.210.290.0330.0330.0260.031
Mercury0.860.710.410.660.0060.0070.0050.006
D1 Ultra0.280.270.250.270.0100.0080.0080.009
Microsoft0.340.350.330.340.0100.0090.0090.009
Clef Omni0.330.230.290.280.0590.0620.0440.055
Qwen–656.67–––0.000––
Qwen (NT)0.846.210.732.590.0000.0000.0000.000
Gemini Flash5.137.233.965.441.8442.6341.2631.914
Gemini Flash (LT)3.393.652.923.321.0761.3330.7781.062
Luna LLM1.672.901.792.120.0500.0620.0390.050
Sol1.462.331.341.710.9101.0390.6640.871
Gemma22.0433.9512.8022.930.0000.0000.0000.000
GLM Flash3.584.7312.496.930.3270.5120.2960.378
Gemini Argon3.838.915.576.100.8011.5331.1741.169
Haiku1.792.282.052.040.1250.1510.1390.139

Table 13. Mean time and cost per call in each experiment. Seconds exclude model loading and a warm-up call. Cost is worked out from each call’s tokens and the API’s list prices; models run locally cost 0, and Gemma is free. Mean is the average over the 3 experiments with equal weights, blank for a model not run in all of them.

Maier et al. [2025] simulated purchase intent on a five-point rating scale. A DM returns a probability for each rating in one call. We consider a firm testing 57 products over a total of 9,300 human choices, giving 18,600 ratings since Maier et al. [2025] repeat calls each time. For LLMs we assume one call per rating, which understates the cost of the method of Maier et al. [2025], as it generates free text and computes an embedding for each response.

For ABMs, we consider 10410^{4} agents over 50 periods, giving 5×1055 \times10^{5} decisions, and assume that a DM chooses among at most ten options per call. In the occupational mobility model of del Rio-Chanona et al. [2021], workers choose which of 464 occupations to apply to. A DM can make this choice in three nested calls by arranging the occupations in a tree with at most ten branches per node (83=512≥4648^{3} = 512 \ge464). For LLMs we assume one call per decision. This favours LLMs: they make a third as many calls as DMs. In the input–output model of Pichler et al. [2022], suppliers that cannot meet demand ration their customers by a fixed rule, and Pichler et al. [2022], Pichler and Farmer [2022] show that the estimated output losses depend on the rule used. Letting each supplier choose among a set of rules takes one DM call per period. Because Pichler et al. [2022] model industries rather than firms, our figures correspond to a firm-level version of the model. In the model of Poledna et al. [2023], firms set prices each period, and choosing among at most ten discrete price changes also takes one DM call.

Table 14 reports the estimates of time and cost for frontier DMs and LLMs across the survey and ABMs considered.

AveragePurchasesOcc. mobilityRationing & Markup
CostTimeCostTimeCostTimeCostTime
Frontier DM
Jev0.0220.240.410.1232.999.8911.003.30
Perplexity0.0060.320.110.178.6513.432.884.48
Mammoth0.0310.290.570.1545.8812.1715.294.06
D1 Ultra0.0090.270.170.1413.4011.194.473.73
LLM
Luna LLM0.0502.120.941.0925.2229.4125.2229.41
Sol0.8711.7116.200.88435.5223.73435.5223.73
Gemini Argon1.1696.1021.753.15584.7584.79584.7584.79
Haiku0.1392.042.581.0569.2828.3469.2828.34

Table 14. Projected total cost and time of each experiment. Occ. mobility refers to the occupational mobility ABM in del Rio-Chanona et al. [2021]. Average is the mean cost ($ per 1,000 calls) and time (seconds per call) over the three experiments of Table 13. Each experiment’s cost ($) is its calls times the mean cost per call, and its time (hours) its calls times the mean seconds per call divided by the calls run at once. Purchase Intent, 18,600 calls, 10 at once; Occupational mobility, 500,000 calls (3 for each by a decision model), 10 at once; Rationing, 500,000 calls, 10 at once.

References

  1. [1]Diogo Almeida. Introducing System One Models & Jev. TypeSafe AI Blog, September 2026. URL https://typesafe.ai/blog/introducing-system-one-models-and-jev. Company blog post.
  2. [2]James Andreoni. Cooperation in public-goods experiments: kindness or confusion? The American Economic Review, pages 891–904, 1995.
  3. [3]Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351, 2023.DOI
  4. [4]Artificial Analysis. Artificial analysis decision index. https://artificialanalysis.ai/evaluations/artificial-analysis-decision-index, 2026. Leaderboard; accessed 28 October 2026.
  5. [5]Ashwini Ashokkumar, Luke Hewitt, Isaias Ghezae, and Robb Willer. Large language models can predict the results of social science experiments. Nature, pages 1–8, 2026.DOI
  6. [7]Joyce Berg, John Dickhaut, and Kevin McCabe. Trust, reciprocity, and social history. Games and economic behavior, 10(1):122–142, 1995.DOI
  7. [8]Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria K Eckstein, Noémi Elétéö, et al. A foundation model to predict and capture human cognition. Nature, 644(8078):1002–1009, 2025.DOI
  8. [9]James Bisbee, Joshua D Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M Larson. Synthetic replacements for human survey data? the perils of large language models. Political Analysis, 32(4):401–416, 2024.
  9. [10]Emilio Calvano, Giacomo Calzolari, Vincenzo D enicolo, and Sergio Pastorello. Artificial intelligence, algorithmic pricing, and collusion. American Economic Review, 110(10):3267–3297, 2020.DOI
  10. [11]Zhixun Chen, Zijing Shi, Yaodong Yang, Meng Fang, and Yali Du. Hierarchical multi-agent framework for dynamic macroeconomic modeling using large language models. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2025.
  11. [13]Olivier Coibion, Yuriy Gorodnichenko, and Rupal Kamdar. The formation of expectations, inflation, and the Phillips curve. Journal of Economic Literature, 56(4):1447–1491, 2018.
  12. [14]John Conlisk. Why bounded rationality? Journal of economic literature, 34(2):669–700, 1996.
  13. [15]Paolo Crosetto and Antonio Filippin. The “bomb” risk elicitation task. Journal of risk and uncertainty, 47(1):31–65, 2013.
  14. [16]Logan Cross, Jordi Grau-Moya, William A Cunningham, Alexander Sasha Vezhnevets, and Joel Z Leibo. A generative model of conspicuous consumption and status signaling. arXiv preprint arXiv:2603.13220, 2026.arxiv.org/abs/2603.13220
  15. [17]Ronald G Cummings, Glenn W Harrison, and E Elisabet Rutström. Homegrown values and hypothetical surveys: is the dichotomous choice approach incentive-compatible? The American Economic Review, 85(1):260–266, 1995.
  16. [18]R Maria del Rio-Chanona, Penny Mealy, Mariano Beguerisse-Díaz, François Lafond, and J Doyne Farmer. Occupational mobility and automation: a data-driven network model. Journal of The Royal Society Interface, 18(174):20200898, 2021.
  17. [19]R Maria del Rio-Chanona, Marco Pangallo, and Cars Hommes. Can generative ai agents behave like humans? evidence from laboratory market experiments. arXiv preprint arXiv:2505.07457, 2025.
  18. [21]Feiyu Duan, Jiayu Lin, Jia Wang, Jun Xiang, Jialiang Wu, Xinnong Zhang, Hanqi Yan, Siyuan Wang, and Zhongyu Wei. General decision models: Benchmarking and insights beyond Jєv. arXiv preprint arXiv:2610.03935, 2026.arxiv.org/abs/2610.03935
  19. [22]J Doyne Farmer and Duncan Foley. The economy needs agent-based modelling. Nature, 460 (7256):685–686, 2009.DOI
  20. [23]Federal Reserve Bank of New York. SCE Household Spending Survey. Survey of Consumer Expectations, Center for Microeconomic Data, 2026. URL https://www.newyorkfed.org/microeconomics/sce/household-spending. Accessed: 2026-10-16.
  21. [24]Yeqi Feng, Yucheng Lu, Hongyu Su, Yixin Tao, and Tianxing He. SimCity: Multi-agent urban development simulation with rich interactions. arXiv preprint arXiv:2510.01297, 2025.arxiv.org/abs/2510.01297
  22. [25]Robert Forsythe, Joel L Horowitz, Nathan E Savin, and Martin Sefton. Fairness in simple bargaining experiments. Games and Economic behavior, 6(3):347–369, 1994.DOI
  23. [26]Federico Gabriele, Aldo Glielmo, and Marco Taboga. Heterogeneous RBCs via deep multi-agent reinforcement learning. arXiv preprint arXiv:2510.12272, 2025.DOI
  24. [27]Yuan Gao, Dokyun Lee, Gordon Burtch, and Sina Fazelpour. Take caution in using llms as human surrogates. Proceedings of the National Academy of Sciences, 122(24):e2501660122, 2025.
  25. [28]Jonathan Geuter, Andrew Cheng, and David Alvarez-Melis. Releasing the inner Jev: A simple recipe for open decision models. arXiv preprint arXiv:2610.17205, 2026.arxiv.org/abs/2610.17205
  26. [29]Mark Granovetter. Threshold models of collective behavior. American journal of sociology, 83(6):1420–1443, 1978.DOI
  27. [30]Werner Güth, Rolf Schmittberger, and Bernd Schwarze. An experimental analysis of ultimatum bargaining. Journal of economic behavior & organization, 3(4):367–388, 1982.
  28. [31]Thomas Henning, Siddhartha M Ojha, Ross Spoon, Jiatong Han, and Colin F. Camerer. LLM agents do not replicate human market traders: Evidence from experimental finance. arXiv preprint arXiv:2502.15800, 2025.arxiv.org/abs/2502.15800
  29. [32]John J Horton, Apostolos Filippas, and Benjamin S Manning. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research, 2023.
  30. [34]Archer Hume. Jev’s architecture unmasked. Personal blog, archerhume.com, September 2026. URL https://archerhume.com/posts/jevs-architecture-unmasked/. Essay.
  31. [37]Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I, pages 99–127. World Scientific, 2013.
  32. [38]Max Kirkby, Michael Psenka, Thomas Villeneuve, Aaryam Sharma, Charles O’Neill, Chloe Florit, and Ardin Lo. The top ranked decision models in the world: mammoth-1 and elephant-1. Baseten blog, https://www.baseten.co/blog/the-best-decision-models-in-the-world-mammoth-1-and-elephant-1/, October 2026. Accessed 28 October 2026.
  33. [39]Alan P Kirman. Whom or what does the representative individual represent? Journal of economic perspectives, 6(2):117–136, 1992.
  34. [41]Nian Li, Chen Gao, Mingyu Li, Yong Li, and Qingmin Liao. Econagent: large language model-empowered agents for simulating macroeconomic activities. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15523–15536, 2024.arxiv.org/abs/2310.10436
  35. [42]Guoming Ling, Muen Xue, and Zijian Ye. Jev in the wild: A data-driven analysis of the Jev model’s functionality, applications and ecosystem. arXiv preprint arXiv:2609.30216, 2026.arxiv.org/abs/2609.30216
  36. [43]John A List and Craig A Gallet. What experimental protocol influence disparities between actual and hypothetical stated values? Environmental and resource economics, 20(3):241–254, 2001.DOI
  37. [44]Robert E Lucas Jr. Econometric policy evaluation: A critique. In Carnegie-Rochester conference series on public policy, volume 1, pages 19–46. North-Holland, 1976.DOI
  38. [45]Marlene Lutz, Indira Sen, Georg Ahnert, Elisa Rogers, and Markus Strohmaier. The prompt makes the person (a): A systematic evaluation of sociodemographic persona prompting for large language models. In EMNLP (Findings), pages 23212–23237, 2025.arxiv.org/abs/2507.16076
  39. [46]Benjamin F Maier, Ulf Aslak, Luca Fiaschi, Nina Rismal, Kemble Fletcher, Christian C Luhmann, Robbie Dow, Kli Pappas, and Thomas V Wiecki. Llms reproduce human purchase intent via semantic similarity elicitation of likert ratings. arXiv preprint arXiv:2510.08338, 2025.arxiv.org/abs/2510.08338
  40. [47]Benjamin S Manning and John J Horton. General social agents. Technical report, National Bureau of Economic Research, 2026.
  41. [48]Alexandre Mas and Amanda Pallais. Valuing alternative work arrangements. American Economic Review, 107(12):3722–3759, 2017.
  42. [49]Qiaozhu Mei, Yutong Xie, Walter Yuan, and Matthew O Jackson. A Turing test of whether AI chatbots are behaviorally similar to humans. Proceedings of the National Academy of Sciences, 121(9):e2313925121, 2024.DOI
  43. [50]James J Murphy, P Geoffrey Allen, Thomas H Stevens, and Darryl Weatherhead. A meta-analysis of hypothetical bias in stated preference valuation. Environmental and Resource Economics, 30(3):313–325, 2005.
  44. [51]Georgios Papadopoulos. From equations to prompts: integrating LLMs into a hybrid macroeconomic ABM. 2026.
  45. [52]Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S Bernstein. Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10109, 2024.
  46. [55]Sebastian Poledna, Michael Gregor Miess, Cars Hommes, and Katrin Rabitsch. Economic forecasting with an agent-based model. European Economic Review, 151:104306, 2023.
  47. [56]Maxime Saxena, Marco Pangallo, Cars Hommes, Fabio Caccioli, and R Maria del Rio-Chanona. Machine Spirits: Speculation and adaptation of LLM agents in asset markets. arXiv preprint arXiv:2604.18602, 2026.arxiv.org/abs/2604.18602
  48. [57]Richard H Thaler. Behavioral economics: Past, present, and future. American economic review, 106(7):1577–1600, 2016.
  49. [58]Alexander Sasha Vezhnevets, John P Agapiou, Avia Aharon, Ron Ziv, Jayd Matyas, Edgar A Duéñez-Guzmán, William A Cunningham, Simon Osindero, Danny Karmon, and Joel Z Leibo. Generative agent-based modeling with actions grounded in physical, social, or digital space using concordia. arXiv preprint arXiv:2312.03664, 2023.
  50. [59]Angelina Wang, Jamie Morgenstern, and John P Dickerson. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7(3):400–411, 2025.
  51. [60]Zengqing Wu, Run Peng, Takayuki Ito, Makoto Onizuka, and Chuan Xiao. Position: LLM-based social simulations require a boundary. In Forty-third International Conference on Machine Learning Position Paper Track, 2026.arxiv.org/abs/2506.19806
  52. [61]Ziyi Yang, Zaibin Zhang, Zirui Zheng, Yuxian Jiang, Ziyue Gan, Zhiyu Wang, Zijian Ling, Jingsong Chen, Martz Ma, Bowen Dong, et al. Oasis: Open agent social interaction simulations with one million agents. arXiv preprint arXiv:2411.11581, 2024.arxiv.org/abs/2411.11581
  53. [62]Jinyi Ye, Lei Cao, Ding Chen, and Emilio Ferrara. Stop drawing scientific claims from LLM social simulations without robustness audits. arXiv preprint arXiv:2605.18890, 2026a.arxiv.org/abs/2605.18890
  54. [63]Weiwei Ye, Hangchen Liu, and Renhe Jiang. NumericJev: Jey-like LLM numerical decoding with multiway decision trees. arXiv preprint arXiv:2609.28587, 2026b.arxiv.org/abs/2609.28587
  55. [64]Jiamu Zhang, Tianze Yang, Yucheng Shi, Evan Chen, Zixiang Nie, Kelly Wan, Liangjie Hong, Ninghao Liu, and Liang Wu. Anyjev technical report. arXiv preprint arXiv:2610.00831, 2026.arxiv.org/abs/2610.00831

Paper details

vlnc:W_45tb86d4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Economics, Econometrics and Finance

Not recorded for this paper

View statement
Looking for related papers…
Contents