WorksComputer Science
RehearsalBench: Can AI Predict What Happens When You Change a Company?
Abstract
Organizations increasingly ask AI systems what will happen if they change how they operate: which work to automate, which team to resize, what a voice agent will do to waiting times. We introduce RehearsalBench, a benchmark that measures whether AI models can diagnose an operation, choose an intervention, and forecast its consequences. Each model is placed in a simulated bank contact center fitted to real call-level records, sees only what a manager would see, and submits a plan and forecasts that are then played out across 200 seeded futures and three plausible versions of the organization. Decisions are scored in dollars at the stakeholders' stated valuations; forecasts are scored on the effect of each change relative to doing nothing. Ten models made 280 attempts across 14 organizations. The best model, GPT-6 Astra, improved on doing nothing by 6.8% of the no-change cost and made 77% less forecast error than a forecaster who always predicts no effect. On average, every model forecast the direction of every specified effect correctly, but models erred on the size of the same effects in the same direction, overstating a voice agent's effect on repeat contacts about five-fold. No model ever recommended holding off when doing nothing was the right choice. To test external validity, twelve models forecast 33 real organizational changes with known outcomes. Under verdict rules fixed in advance, the benchmark's forecast ranking agreed with the real-world ranking (Spearman , 95% lower bound 0.50). We report the result alongside an inconclusive first test, the analysis choices made after outcomes were seen, and evidence that part of the agreement reflects how well models size their uncertainty.
Introduction
Organizations increasingly turn to artificial intelligence (AI) to reshape customer service. In July 2025, the Commonwealth Bank of Australia announced that an AI voice bot would replace 45 customer-service roles. The staff union subsequently reported rising call volumes. In August, the bank reversed its decision, acknowledging that the roles were still needed [11]. This case illustrates a general difficulty. Even when a bot resolves some calls, managers must anticipate how much staff time the remaining calls will require, and they must do so before the change is made.
Moreover, the effect of a change is difficult to measure even afterwards. A contact center, the team that handles an organization’s customer calls, observes the outcome of only the decision it actually makes. If it moves staff to an evening shift, it can measure how long callers then wait. However, the wait under the old roster remains a counterfactual, an outcome that was never observed. A simulation can test both rosters against the same calls. Nevertheless, how well such a comparison represents a real center depends on the simulator’s assumptions.
AI models are increasingly evaluated both as forecasters and as agents that run simulated businesses (§ 2). Forecasting benchmarks score predictions of world events. Business simulations score the outcomes of each agent’s own actions. Neither tests whether a model can predict the effect of an operational change before it is made. Neither compares different models’ forecasts of the same change.
We therefore propose RehearsalBench, a benchmark that asks large language models (LLMs) to rehearse operational decisions in simulated bank contact centers (Figure 1). To the best of our knowledge, RehearsalBench is the first benchmark that evaluates both the operational decisions of LLMs and their forecasts of the counterfactual effects of shared interventions. It is also the first to validate the resulting forecast ranking against real organizational changes. We examine two questions:
Figure 1. Overview of RehearsalBench. (a) Statistics estimated from Bank A’s records calibrate a multi-class queue simulator, which is validated on held-out Bank A records and on Bank B. (b) The simulator generates 14 fictional banks, each with stakeholders, costs and an operational problem. Bank parameters and the effects of the eight intervention types (icons, left to right and top to bottom: staffing, opening hours, routing, callbacks, AI voice agent, AI assistance for staff, self-service and coaching) stay hidden from the model. (c) In phase 1, the LLM chooses a plan, scored by its decision value relative to no change on the same call sequences. In phase 2, every model forecasts the effects of the same assigned interventions, scored with the interval score. (d) Models also forecast 33 disguised real organizational changes, and the two forecast rankings are compared. Example plans, forecasts and rankings are illustrative.
First, can LLMs choose a valuable intervention and predict its effects? In RehearsalBench, each model chooses a plan for a simulated center. It then forecasts the effects of a fixed list of interventions given to every model. This allows forecasts to be compared even when models recommend different plans.
Second, does the resulting forecast ranking carry over to real organizational changes? We give the models historical cases, using only information available before each change, and rank their forecasts against the observed outcomes.
The second comparison tests whether forecasting skill transfers across settings. It does not estimate the value of implementing a recommended plan in a real center.
The main contributions of this paper are as follows:
Counterfactual benchmark: We develop a simulator of bank contact centers that is built and validated on historical call records. It is combined with explicit, documented assumptions about the effects of AI interventions. Because every alternative plan faces the same simulated calls, the benchmark measures the value of each plan relative to making no change.
Comparable forecasts: We separate planning from forecasting. Every model forecasts the same assigned interventions with point estimates and prediction intervals, so forecasts can be ranked fairly even when the models’ own plans differ.
Evaluation of open-weight and closed models: We evaluate open-weight and closed LLMs on simulated contact centers and analyze where their forecasts are incorrect. On average, every model predicts the direction of each effect correctly, but all models overstate the size of several. These include the increase in repeat contacts under a voice agent, which models overstate approximately five-fold.
Retrospective validation: We test whether the benchmark’s forecast ranking agrees with the models’ ranking on real organizational changes. The test uses pre-registered criteria and checks for prior knowledge of the outcomes.
The remainder of this paper is organized as follows. § 2 reviews related work. § 3 describes the simulation environment, and § 4 the benchmark tasks and scoring. § 5 presents the experimental setup and § 6 the results. § 7 analyzes forecast errors and planning behavior. § 8 compares the benchmark ranking with performance on real organizational changes. § 9 concludes the paper and outlines future work.
Related Work
In this section, we review three lines of work: i) AI forecasting benchmarks and data memorization, ii) evaluations of AI agents in business and professional tasks, and iii) queueing models of contact centers.
Forecasting benchmarks and data memorization. Karger et al. [13] propose ForecastBench, which poses questions about future events drawn from prediction markets, forecasting platforms and public time series. It scores probability forecasts with the Brier score as the events resolve, using crowd forecasts as a provisional reference for unresolved market questions. This design prevents models from knowing the answer but requires waiting for outcomes. Notably, expert forecasters outperform the best LLM.
Retrospective tests can use known outcomes immediately. However, Lopez-Lira et al. [15] show that GPT-4o recalls exact economic and financial values from before its knowledge cutoff. Masking fails, because the model reconstructs the company, quarter and year of an anonymized earnings call from minimal context. Bordt et al. [2] find that LLMs have memorized popular tabular datasets verbatim and predict better on these datasets than on novel ones, which the authors attribute to overfitting. We address this risk with disguised events and recognition checks (§ 8.2).
In contrast to RehearsalBench, forecasting benchmarks such as ForecastBench score forecasts of events that occur regardless of any decision. They do not score forecasts of what an intervention would change relative to the counterfactual without it.
Agents in business and professional work. Backlund and Petersson [1] propose Vending-Bench, in which an agent manages inventory, orders and prices for a simulated vending business and is scored on its final net worth. Every model tested has runs that derail. Chen et al. [5] propose CEO-Bench, in which an agent runs a fictional subscription-software company for 500 simulated days and is scored on its final cash. Every model scores below a rule-based baseline. CEO-Bench also asks for four-week cash forecasts, but these concern each agent’s own company under its own decisions. Park et al. [16] place 25 generative agents in a simulated town and have human evaluators rank the believability of their behavior, without scoring their decisions against outcomes.
Other benchmarks assess professional work directly. Patwardhan et al. [17] propose GDPval, in which occupational experts blindly compare model and human deliverables on tasks drawn from 44 occupations. Jimenez et al. [12] propose SWE-bench, in which a model edits a Python repository to resolve a real GitHub issue and is scored by the repository’s tests. SWEeper-Bench [19] gives an agent only a web application’s codebase and an open-ended request. The agent must find the bug through interactive testing before repairing it, and the best of 15 agents passes 59.0% of tasks. -bench [18] has an agent use tools and follow domain policies to serve an LLM-simulated customer in retail and airline service. Success is judged by the final database state, and even the strongest agents succeed inconsistently across repeated trials. Kwa et al. [14] at METR (Model Evaluation and Threat Research) measure how long humans with relevant expertise typically take to complete tasks that agents complete with 50% success. They find that this time horizon has doubled approximately every seven months since 2019.
Nevertheless, all of these benchmarks score the outcomes of each agent’s own actions. Because agents act differently, the benchmarks cannot compare how well different models predict the effect of the same change. RehearsalBench measures this directly by having every model forecast the same assigned interventions.
Queueing models of contact centers. Gans et al. [7] review call-center operations, covering arrival forecasting, staffing, abandonment and skills-based routing. They note that the Erlang-C model common in practice ignores customer impatience. Garnett et al. [8] analyze the Erlang-A model, in which callers with exponentially distributed patience may hang up before being answered. They derive staffing rules of thumb for large call centers and show that staffing tools that ignore abandonment lead to overstaffing. Brown et al. [3] use a year of call-by-call records from Bank A (§ 3.1) to test these models against observed behavior. They find a strong linear relation between average wait and the fraction of callers who abandon, and show that Erlang-A gives useful upper bounds on the main performance measures.
Our simulator extends this line of work. It models time-varying arrivals, staffing and caller patience (§ 3.2) and draws on the same archive of call records. However, rather than using the model to design staffing, we use it as a testbed to score AI decisions and forecasts.
Simulation Environment
In this section, we describe the environment in which models are evaluated. We first summarize the statistics estimated from historical call records (§ 3.1) and the contact-center model built on them (§ 3.2). We then describe how the simulator is validated and how the validation data relate to the benchmark test set (§ 3.3). Finally, we describe the assumed effects of AI interventions (§ 3.4).
Historical Call Records
Our simulator is built on the call archive of the Service Enterprise Engineering Laboratory (SEELab) at the Technion – Israel Institute of Technology [3]. For each call, the records give its arrival time, its type and, if it was answered, how long staff spent handling it. Some callers hang up before being answered, which is known as abandonment. For these callers, the records also give how long they waited. The records also show how many staff were on duty in each interval. We use records from two banks, which we refer to as Bank A and Bank B. Both banks are anonymized in the archive. Bank A’s 1999 records are those analyzed by Brown et al. [3]. Bank B is a US bank whose records cover the consumer-loan line of its contact center from 2001 to 2003.
From these records, we estimate four statistics for each call type :
Arrival rates. The rate at which type- calls arrive in interval on weekday type ,
where denotes the set of recorded days of weekday type , represents the number of type- calls that arrived on day in interval , and corresponds to the interval length (one hour).
Handling times. The distribution of the time staff spend on answered type- calls.
Patience. The distribution of how long type- callers are willing to wait before hanging up. For an abandoned call, the recorded wait equals the caller’s patience. For an answered call, it is only a lower bound, because the caller would have waited longer. We therefore estimate with the Kaplan–Meier estimator, which accounts for such censored observations.
Retry and repeat probabilities. The probability that an abandoned caller calls again, and the probability that an answered call is followed by a repeat contact. Both are estimated as the share of callers who call again within 24 hours, using the half of the calls whose callers are identified.
Contact-Center Model
We model a contact center as a multi-class queue with time-varying arrivals, impatient callers and a staffing roster. Calls of type arrive as a non-homogeneous Poisson process with rate . Each call has an arrival time , a type , a handling time and a patience , drawn as
where denotes the handling-time distribution of type- calls and represents their patience distribution. The roster sets the number of staff on duty, , and a routing policy assigns waiting calls to available staff. Let denote the time a call i would wait before being answered under . The caller abandons if this wait exceeds their patience,
where is the indicator function. An abandoned caller calls again with probability ; otherwise, the contact is lost. An unanswered call leads to a repeat contact with probability .
Over a horizon with calls, the simulator reports five outcomes: the abandonment rate , the mean wait , the average handling time , the repeat-contact rate , and the staff cost , where is the hourly cost of one staff member.
We collect the bank’s parameters in . A plan specifies the roster, the routing policy and any interventions. For a random call sequence , which fixes the draws of arrivals, handling times and patience, the simulator returns the outcome vector
Every plan is replayed on the same call sequences, so differences between plans reflect the plans themselves rather than random variation. The effect of plan on outcome is
where denotes current operations and the expectation is taken over call sequences. Models forecast this effect, together with its variation across call sequences (§ 4.2).
Simulator Validation and Data Roles
We validated the simulator in two settings. In each, we gave it the recorded call arrival times and the number of staff on duty. It then simulated how the calls were handled, including how long callers waited and whether they hung up. We compared the simulated outcomes with the records, using criteria set before each test.
Later months at the same bank. We estimated the statistics in § 3.1 from Bank A’s January–June 1999 records. We then compared the simulator’s results with its July–December records, which we had set aside for testing.
A different bank. We tested whether the same simulator could reproduce outcomes at another center, using Bank B’s 2001–2003 records from its consumer-loan line.
In both settings, the simulator reproduced the aggregate outcomes in the records. However, two features varied over time in the records: the way calls were assigned to staff (routing) and the level of service callers received. Rather than fixing either at a single value, we sample both from ranges, so that the simulated banks reflect this variation.
Table 1 summarizes how the data sets relate to one another. The validation data test the simulator, whereas the benchmark test set tests the models. The simulator connects the two. Once validated, it generates each test bank by drawing parameters around the estimated statistics. Every test bank’s history is built from Bank A’s fit, with bank-level values drawn within a declared envelope. For example, handling times are 1.00–1.36 times and caller patience 1.0–2.4 times Bank A’s, and call volumes range from 4,700 to 17,300 calls a week. The test banks are therefore grounded in the records, but none is a copy of Bank A or Bank B. The real organizational changes in § 8 form a separate test that does not use the simulator.
| Data | Role | Used in |
| Bank A, Jan–Jun 1999 | Estimate call statistics (calibration) | Section 3.1 |
| Bank A, Jul–Dec 1999 | Held-out validation: same bank, later period | Section 3.3 |
| Bank B, consumer loans, 2001–2003 | External validation: different bank | Section 3.3 |
| 14 fictional banks | Benchmark test set, generated by the validated simulator | Sections 6–7 |
| 33 real organizational changes | Retrospective test of the forecast ranking | Section 8 |
Table 1. Data sets and their roles.
Assumed Intervention Effects
An intervention is a single operational change, such as adding evening staff or deploying an AI voice agent. A plan combines one or more interventions, or keeps current operations. Table 2 lists the eight types of intervention available to models and the part of the model in § 3.2 that each one changes.
| Intervention | What it does | Changes |
| Staffing | Changes how many staff are on duty in each interval | |
| Opening hours | Extends or shortens the hours in which calls are answered | |
| Routing | Changes how waiting calls are assigned to staff | |
| Callbacks | Offers waiting callers a call back instead of holding | , |
| AI voice agent | Resolves some calls without staff | , , |
| AI assistance for staff | Helps staff handle calls faster | |
| Self-service | Lets customers resolve requests online instead of calling | |
| Coaching | Trains staff to resolve calls faster or at the first attempt | , |
Table 2. Intervention types and the model components they change.
Staffing, opening hours and routing change only how existing calls are served, so their effects follow from the model in § 3.2. The other interventions also change the calls themselves. The records describe ordinary call handling, so simulating these interventions requires assumptions about their effects. For example, what share of calls could an AI voice agent resolve without staff? Each intervention modifies the model through a small set of effect parameters . For example, a voice agent resolves a share of type- calls without staff and changes the repeat-contact probability,
where denotes the containment share and represents the change in the repeat-contact probability. The voice agent also changes the handling times of the calls left to staff. A call that the voice agent does not resolve reaches the queue later, and its handling time is longer by 0–90 s. Containment does not depend on the call’s handling time. Similarly, AI assistance for staff shortens handling times, , where is the fractional time saving.
Self-service, callbacks and coaching act on arrival rates, patience and handling times. Self-service removes a share of the arriving calls of the targeted types (20–36% for simple types). Callbacks let callers whose expected wait exceeds a threshold keep their place in the queue without abandoning. Coaching moves each coached staff member’s handling time toward that of the fastest fifth of staff (9–25% of the gap) and takes the coaching hours off the phones.
We draw the effect parameters from documented ranges. For example, voice-agent containment is drawn from 14–36%, following a 2024 Gartner customer survey [9], and AI assistance shortens handling times for existing staff by 0–13%, following Brynjolfsson et al. [4]. Parameters with no published measurement, such as the share of callers who agree to use the voice agent, are assumed and given wide ranges. For each bank, the drawn values, together with , are kept hidden from the model being evaluated. Figure 1 illustrates how the historical records and the assumed effects combine to support the two evaluations.
Benchmark Design
In this section, we describe how test banks are constructed and selected (§ 4.1), the two tasks given to each model (§ 4.2), and how plans and forecasts are scored (§ 4.3).
Simulated Banks and Selection
Each benchmark instance is a fictional bank (for example, bank B04) whose customer contact center is simulated with the model of § 3.2. The bank provides the organizational context for the decision: stakeholders such as Finance, their goals, the monetary cost they assign to repeat and lost contacts, a minimum service requirement, and an operational problem to address. The contact center is the operation that the model changes. We therefore refer to each instance by its bank and to the operation being changed as its contact center.
For each bank, we generate 26 weeks of management reports, which the model receives together with the roster, a cost sheet and stakeholder statements. A search with a fixed budget of 540 plan evaluations then looks for the most valuable plan it can find, using separate call sequences that are never used for scoring. We call the result the best-known plan. It serves as a reference rather than an optimum, and models frequently outperform it (§ 6).
Before a candidate bank is used, it must pass three screening checks. These checks confirm that i) current operations are feasible, ii) the hidden parameters cannot be read from the material given to the model, and iii) the best-known plan gains over both making no change and a generic plan chosen without the bank’s data. We record both accepted and rejected candidates.
Candidate banks were generated in two batches. The first batch contained 24 candidates, three for each of eight intervention types, of which 21 passed screening. A pre-specified balance rule required the best-known plans to cover a mix of intervention types. However, staffing changes dominated the best-known plans, so the batch failed this rule and its two fallback rules. The final fallback therefore selected eight banks that passed screening: for each of the eight intervention types, taken in a fixed order, it selected the first accepted bank whose specification targeted that type. A second batch of 16 candidates added six banks, bringing the total to 14. Two of these six are banks where no available plan reliably outperforms making no change. They test whether models recognize when keeping current operations is the best choice. Three others impose a hiring freeze.
Model Tasks
A run is one attempt by one model on one bank. Each run consists of two phases, carried out in an environment without network access.
Phase 1: choose a plan. The model reads the bank’s management reports, roster, cost sheet and stakeholder statements. It may purchase evidence, such as a short pilot, staff-level records or vendor references, at a cost charged to its plan. A pilot is a simulated trial of one intervention, lasting two, four or eight weeks, on about a third of the staff or calls, run on call sequences separate from those used for scoring. It costs $15,000 per two weeks, and the model sees only its noisy readout. It then ranks the bottlenecks it identifies, chooses a plan and forecasts the plan’s outcomes. The plan may keep current operations or combine any of the interventions in Table 2. We then lock the submission.
Phase 2: forecast assigned interventions. In a fresh conversation, every model receives the same list of specific interventions for the bank, such as deploying an AI voice agent or adding evening staff. We call these the assigned interventions. They are generated by a fixed rule, the same for every bank: a cut of about two staff members, an AI voice agent, callbacks offered when
the expected wait exceeds five minutes, AI assistance for all staff, self-service content for the largest simple call type, two hours of coaching per staff member per month and, in banks whose staff form a single team, two staff members reserved for specialist call types. Banks in the second batch also get a roster reshape that moves two staff members. For each intervention and each of the five outcomes in Eq. (4), the model gives a point estimate of the effect in Eq. (5) and an 80% prediction interval, a range that it expects to contain the simulated effect four times in five (models also forecast outcome levels, but only the stated effects are scored).
Phase 1 is limited to 400,000 generated tokens, 60 minutes and 200 tool calls per run, and phase 2 to 300,000 generated tokens, 30 minutes and 100 tool calls.
We report two results for each model: the value of its own plan from phase 1 (decision value) and the accuracy of its assigned forecasts from phase 2 (forecast improvement). Because every model forecasts the same interventions, the phase 2 results yield the forecast ranking that we compare with real organizational changes in § 8.
Scoring
Decision value. For each bank, the evaluator simulates four alternatives: current operations (no change), the model’s plan, the assigned interventions and the best-known plan. Each alternative is simulated in three versions of the bank: as described to the model, under adverse conditions and under favorable conditions. The versions differ only in assumed parameters, such as intervention effects, caller patience and the stakeholders’ valuations. The described version draws each from its published range, the adverse version from the half less favorable to automation and AI, and the favorable version from the other half, with no cost for lost contacts. The mechanics estimated from the records are the same in all three. In each version, every alternative is replayed on the same randomly generated call sequences , each covering a 13-week horizon.
The utility of a plan is
where denotes the staff cost, represents the cost of the plan’s interventions and purchased evidence, with one-off costs spread over 24 months, and correspond to the numbers of repeat contacts and lost contacts, and and are the dollar costs that the bank’s stakeholders assign to each. The decision value of plan is its gain over making no change, expressed as a percentage of the total no-change cost:
where denotes utility averaged over the three versions and call sequences, and is the total cost of making no change, including the costs of repeat and lost contacts. A positive decision value therefore means that the plan outperforms making no change. A plan can gain value by reducing repeat or lost contacts as well as direct costs. Evidence purchases are charged to the plan. If a plan breaches the bank’s minimum service requirement in any version, it earns no credit: it is scored as making no change, so its decision value is zero.
Forecast score. Decisions depend on how much an intervention changes outcomes, so we score forecasts of effects (Eq. (5)) rather than of outcome levels.
We score each 80% prediction interval with the interval score [10],
where and denote the lower and upper bounds of the interval, represents the simulated effect, and corresponds to the share of outcomes an 80% interval is allowed to miss. The first term penalizes wide intervals, and the other two penalize intervals that miss the simulated effect, in proportion to the size of the miss. We compute the score against the effect in each of the call sequences of the version described to the model, that is, the paired difference inside the expectation in Eq. (5), and average over the sequences. To make outcomes on different scales comparable, we divide each score by , the standard deviation of outcome under no change across call sequences. We then average the scaled scores over outcomes, interventions and banks to obtain a model’s forecast score .
The forecast improvement compares this score with that of a baseline that always predicts zero effect with a zero-width interval,
where denotes the baseline’s score in this evaluation. A forecast improvement of zero matches the baseline, and one corresponds to perfect forecasts. We report it as a percentage.
Uncertainty in comparisons. A model’s score depends on which banks are in the benchmark and on how its runs vary on the same bank. To capture both sources of uncertainty, we compute 95% bootstrap intervals from 10,000 resamples. Each resample first draws banks with replacement and then draws runs within each sampled bank. We use the same draws for all models, so that every model is compared on the same resampled benchmark. We consider two models that are next to each other in a ranking to be distinguishable only when the interval for the difference between their scores excludes zero. We do not adjust these intervals for multiple comparisons, so individual distinctions should be read as indicative. The comparison with real organizational changes in § 8 has a different source of uncertainty, namely which real changes are included. There, we therefore resample changes instead of banks.
Protocol. Before running the models, we fixed the bank specifications, call sequences, evaluation software and scoring rules, and recorded their SHA-256 digests, cryptographic fingerprints that change if any file changes (listed in the public repository). We recorded all scoring amendments for the main evaluation before scoring it. Claude Opus 5.5 was used as a coding assistant in developing the benchmark software and was not among the evaluated models. Because three evaluated models (Claude Sonnet 5.5, Claude Fable 5.1 and Claude Haiku 5.5) share its developer, we disclose this use.
Remark (relation to reinforcement learning). The simulator plays a role similar to an environment in reinforcement learning (RL). The bank’s parameters form a hidden state, and the management reports are observations. Evidence purchases are information-gathering actions with a cost, and the submitted plan is a final action whose reward is its decision value. A short pilot is the closest analog of exploration: the model pays to observe a brief, noisy run of an intervention in the simulator. However, unlike in standard RL, each run is a single episode, and the model receives no reward until scoring. RehearsalBench therefore tests planning under partial observability rather than learning by trial and error. Nevertheless, the same environment could be used to train agents with RL in future work.
Experimental Setup
In this section, we describe the models evaluated, how they were accessed and which runs count toward the results.
We evaluated ten LLMs from six developers (Table 3). Two of them, DeepSeek V4.1 Flash and Mistral Small 2603, have openly available weights. The other eight, including Qwen3.8 Flash, are available only through APIs. Each model made two runs on each of the 14 banks, giving 280 runs in total, carried out on 30–31 October 2026. Each run took 40–90 minutes. The evaluator simulated each alternative on call sequences in each version of a bank. Access to the models through their application programming interfaces (APIs) cost $1,202 for this evaluation. Every model ran at its highest available reasoning setting.
| Model | Short name | Developer | Access route |
| GPT-6 Astra | Astra | OpenAI | Developer API |
| GPT-6.1 Sol | Sol | OpenAI | Developer API |
| GPT-6 Luna | Luna | OpenAI | OpenRouter |
| Claude Sonnet 5.5 | Sonnet | Anthropic | OpenRouter |
| Claude Fable 5.1 | Fable | Anthropic | Both |
| Claude Haiku 5.5 | Haiku | Anthropic | Developer API |
| Gemini 3.8 Flash | Gemini Flash | Developer API | |
| DeepSeek V4.1 Flash | DeepSeek | DeepSeek | OpenRouter |
| Qwen3.8 Flash | Qwen Flash | Alibaba | OpenRouter (own key) |
| Mistral Small 2603 | Mistral Small | Mistral | OpenRouter |
Table 3. Models evaluated and their access routes (28 runs per model). The rest of the paper refers to models by their short names. OpenRouter is a third-party service that routes requests to model providers; “own key” means that requests were billed to our own account with the provider. Claude Fable 5.1 used OpenRouter on five banks and its developer’s API on nine.
We rank the nine models that submitted a complete plan and forecasts in at least half of their runs, and report a partial result for Mistral Small 2603 separately. Four further models were evaluated for the comparison with real organizational changes, and we introduce them in § 8.
Results
In this section, we first illustrate a single run (§ 6.1). We then report decision value and forecast performance for all models (§ 6.2) and relate them to the cost of each model (§ 6.3).
Illustrative Run: A Roster Change at Bank B04
To illustrate what a run involves, we describe one of Astra’s runs. At bank B04, the Finance team requested a 10% cut in payroll. Astra found in the management reports that 15,957 weekday calls between 21:00 and midnight had reached an after-hours announcement because no staff were on duty. It proposed moving existing staff hours to evenings and to morning peaks, rather than adding staff. Astra declined to promise the payroll cut, citing the bank’s minimum service requirement and the cost that Finance assigns to unanswered calls.
The plan improved utility over the 13-week horizon by $156,683 relative to making no change and by $17,131 over the best-known plan. Its forecasts were also close to the simulated outcomes (Table 4). For example, Astra forecast an abandonment rate of 11.0%, against 10.6% in the simulation, and its staff-cost forecast was within $215 of the simulated cost. This case highlights the kind of reasoning the benchmark rewards: the model identified a mismatch between when calls arrive and when staff are on duty, and it weighed a stakeholder’s request against the bank’s service requirement.
| Outcome | Forecast | Simulated outcome |
| Abandonment rate (%) | 11.0 (9.8–12.4) | 10.6 |
| Mean wait (seconds) | 40 (33–48) | 38 |
| Staff cost (USD) | 306,185 | 305,970 |
Table 4. Astra’s forecasts for its chosen plan at bank B04, compared with the simulated outcomes. Parentheses give the model’s 80% prediction intervals where it reported them.
Decision Value and Forecast Performance
Figure 2 shows both results for each model. Seven ranked models chose plans worth 5–7% of the no-change cost on average, with overlapping bootstrap intervals. Astra had the highest point estimate (6.8%), followed closely by Sonnet (6.7%) and Sol (6.5%). Among models next to each other in the ranking, the bootstrap distinguished DeepSeek (5.3%) from Luna (0.7%). The ranked models therefore fall into two tiers: seven models whose plans add similar value, and Luna and Gemini Flash, whose plans add little. As § 7 shows, this gap is attributable mainly to plans that left operations unchanged (Gemini Flash) or cost more than making no change (Luna).

Figure 2. Main results for the nine ranked models and the partial result for Mistral Small, ordered by decision value. Dots give means and bars give 95% bootstrap intervals. (a) Decision value (Eq. (8)); the open circle marks the partial result for Mistral Small. (b) Forecast improvement over the zero-effect baseline (Eq. (10)); Mistral Small’s partial result lies below the baseline. API costs per run are shown in Figure 3.
All nine ranked models improved on the zero-effect baseline, with forecast improvements of 53–77%. Astra had the best forecasts (77%), and the bootstrap distinguished it from the next model, Sonnet (74%). It also distinguished Fable (71%) from Haiku (62%). Notably, planning and forecasting skill do not always coincide. DeepSeek and Qwen Flash chose plans of the same value (5.3%), but DeepSeek had the higher forecast improvement (62% versus 55%). Conversely, Luna’s plans added little value, yet its forecasts (59%) were comparable to Haiku’s (62%).
Remark (comparison with the best-known plan). The best-known plan is a reference rather than an upper bound. Although the search evaluates plans directly in the simulator, its budget is fixed, so it cannot explore every combination of interventions. In 50% of its runs, Astra’s plan had a higher mean simulated value than the best-known plan for that bank. Sonnet, Sol and Haiku did so in 39–43% of runs. This suggests that the strongest models can find plans that a budget-limited search misses, possibly through the many fine-grained roster changes they make (§ 7).
Model Cost
Figure 3 depicts each model’s mean decision value against its API cost per run. Decision value rises only modestly with cost. Sol reached nearly the same value as Astra (6.5% versus 6.8%) at about one-sixth of the cost ($2.47 versus $14.00 per run). DeepSeek reached 5.3%, or about 77% of Astra’s value, at $0.46 per run, about one-thirtieth of the cost. In contrast, a higher cost did not guarantee value. Fable was the most expensive model ($14.37 per run) but ranked fourth, and Gemini Flash cost about as much per run as Sol ($2.49) but added almost no value. This weak relationship stems from planning behavior rather than price: the two models whose plans added little value differ 14-fold in cost, and their low value reflects costly or unchanged plans (§ 7). Consequently, for this task, a model’s cost per run is a poor guide to the value of its plans.

Figure 3. Mean decision value against API cost per run for the nine ranked models. Squares mark Astra and Sol, which reach similar value at a six-fold difference in cost.
Forecast Errors and Planning Behavior
In this section, we analyze where models’ forecasts are incorrect and how their plans differ, in order to explain the results in § 6.
Direction of effects. On average across runs, every model forecast the correct direction for each assigned effect. This includes effects whose direction is not obvious. In the simulator, the assigned routing and coaching changes increased abandonment: coaching takes the coached staff off the phones for their coaching hours, and the routing change reserves two staff members for specialist call types, leaving fewer agents for the other calls. Similarly, the voice agent increased the average handling time of the calls left to staff.
Size of effects. However, models from all developers overstated the size of the same three effects (Figure 4). They predicted that AI assistance would shorten handling time by 9.0–12.6 seconds, against 8.6 seconds in the simulator, an overstatement of 5–47%. They expected callbacks to reduce waiting by 1.1–5.4 seconds, against 0.2 seconds, which overstates the effect between 5.5- and 27-fold. Most notably, they forecast that a voice agent would increase repeat contacts by 0.46–0.58 percentage points, against 0.10, approximately five times the simulated effect. Because these errors are shared across developers, they are unlikely to reflect the weakness of any single model. One possible explanation is that the vendor materials available to every model overstate these effects. Every bank’s stakeholder documents include a voice-agent vendor’s note claiming that up to a stated share of calls is fully handled without an agent and that the remaining calls are 30% shorter. Models can also request vendor reference figures for any intervention, including AI assistance and callbacks, and these overstate the true effect by an undisclosed factor of up to about six. We have not yet tested this, for example by removing those materials. It is worth mentioning that this explanation fits the overstated benefits of AI assistance and callbacks better than the overstated increase in repeat contacts, which is a cost of the voice agent.

Figure 4. Three effects whose size all models overstated. For each intervention, the band spans the models’ mean point forecasts, and the dot marks the simulated effect, which lies outside the band in all three cases. Changes in repeat contacts are in percentage points (pp).
Keeping current operations. No model explicitly recommended keeping current operations, even on the two banks where no available plan reliably outperformed making no change. Gemini Flash nevertheless submitted plans that left operations unchanged in 75% of its runs, including three of its four runs on those two banks. However, most of its unchanged plans were at banks where an intervention would have added value. This explains its decision value of 0.0% (Figure 2). The model rarely acted, and it did not distinguish banks where acting pays from banks where it does not.
Interval coverage. An 80% interval is well calibrated if it contains the simulated effect about four times in five. Coverage was 80–100% for the leading forecasters, so their intervals were, if anything, wider than necessary. In contrast, Haiku’s and Gemini Flash’s intervals often covered only 11–50% of the simulated effects, so they were too narrow. This overconfidence contributes to their lower forecast improvement (62% and 53%), because the interval score penalizes each miss in proportion to its size (Eq. (9)).
Plan composition. Staffing changes appeared in 93–100% of plans for every ranked model except Gemini Flash. This is consistent with staffing dominating the best-known plans (§ 4.1). Models differed most in how finely they adjusted the roster. Astra averaged 46 individual changes per plan, whereas Luna averaged five. Moreover, Luna’s plans cost more than making no change in 43% of its runs, which explains its low decision value. The models also differed in their use of automation. Astra never chose a voice agent, whereas DeepSeek did so in 36% of its plans and Mistral Small, with a partial result, in 86%.
Evidence and forecasts of own plans. Astra and Sol purchased about three items of evidence per run, whereas Haiku bought almost none. Haiku’s forecasts of its own plans were also less accurate. The leading models forecast the abandonment rate of their own plans to within 0.3 percentage points of the simulated rate, whereas Haiku’s forecasts differed by 2.2 percentage points. These analyses concern the simulated banks. In the next section, we test whether the models’ forecast ranking carries over to real organizational changes.
Forecast Rankings on Real Organizational Changes
In this section, we address the second question in § 1: does the forecast ranking from the simulated banks carry over to real organizational changes? We describe the comparison (§ 8.1), how the real changes were selected and disguised (§ 8.2), and the order of analysis decisions (§ 8.3). We then report the agreement between the two rankings (§ 8.4) and examine what drives it (§ 8.5).
The Comparison
We rank the same models in two ways. The benchmark ranking orders models by their forecast improvement on the assigned interventions (§ 4.3), with ties in rounded values broken by the unrounded scores (e.g., 74.2% for Sonnet versus 73.7% for Sol). For the real-change ranking, we give each model a set of real organizational changes, using only information available before each change, and ask it to forecast the effect of the change with a point estimate and an 80% prediction interval. Each change has one or more outcome cells, such as the average wait in the first three months, and the target in each is the change between the before and after windows, or the treated-minus-control difference for a randomized trial. We then score each forecast against the observed outcome. A model’s real-change score is
where denotes the number of real changes for which the model remained eligible, represents the model’s scaled interval score on change , averaged over its outcome cells, and corresponds to the score of a calibrated no-change forecast, a forecast of zero effect with an interval width set from the variation observed before the change. Each interval score is divided by the standard deviation of the same before-to-after change measured at placebo dates before the real change (for trials, across random splits of the control arm), and the calibrated forecast’s interval spans the 10th to 90th percentiles of those placebo changes. Lower scores are better, and a negative score means the model outperformed the calibrated no-change forecast. Unlike the ratio used for forecast improvement (Eq. (10)), Eq. (11) uses a difference, because models are scored on different sets of eligible changes after the recognition checks, and subtracting each change’s calibrated no-change score keeps their scores comparable.
The comparison covers 12 models: the nine ranked models from Section 6, together with Muse Spark 1.3, Qwen3.8 27B and Gemini 3.5 Flash-Lite, which made one benchmark run per bank. The fourth additional model, Solar Pro 4, did not meet the ranking criterion in Section 5 and is not included. We measure agreement between the two rankings with Spearman’s rank correlation, , which equals 1 when the rankings are identical and 0 when they are unrelated.
Real Changes and Recognition Checks
A retrospective test is only informative if the models do not already know the outcomes. We therefore disguised every real change and checked whether models recognized it.
We began with 38 candidate changes, of which a first recognition screen retained ten. We disguised the other changes more strongly and re-admitted them, replacing the change-level screen with a check of each model’s knowledge of each outcome. The final set has 33 changes: the ten retained changes and 23 of the re-disguised changes.
Disguising. For the ten retained changes, we replaced organization and unit names, shifted dates by whole years chosen so that weekdays were unchanged, and replaced identifiers with hashed codes. For the remaining 23, which used the revised procedure, we described time relative to the change, used generic organization types, and described how each intervention worked rather than naming the product. Within each source, we also rescaled all counts and all durations by hidden random factors.1 This hides absolute values while preserving rates such as the share of abandoned calls.
Recognition checks. In an initial probe, models frequently identified the source of a change (at least one model flagged 37 of the 38 candidate changes). However, naming a source does not imply knowing the outcome. A second, pre-registered probe therefore asked each model about the outcomes directly. It also included placebo questions: either the same organization and type of change at a date two to three years away, when no change took place, or a plausible change that never happened. The prompt did not say which items were placebos. We set aside 29 model–change pairs for which the probe indicated prior knowledge of the outcome, mostly for published trials. These pairs were not run, and each model is scored only on the changes for which it remained eligible. As a separate sensitivity check, we also removed, for each model, every change whose source that model had identified.
Final set of changes. The final set contains 33 real changes from four sources: open-source software teams adopting AI coding assistants or agents, government contact centers, randomized workplace trials, and the contact center of a US bank (Bank B).2 A third of the changes concern contact centers.
Order of Analysis Decisions
We fixed the test plan, including the agreement criteria in Table 5, on 29 October 2026, before the benchmark runs. Before seeing any real-change scores, we also fixed the revised disguising procedure described above. The 33-change set and the per-model recognition check (Table 5, marked †) were specified on 2 November 2026, after the benchmark results were available.
| Criterion | Threshold | Result |
| Rank correlation () | 0.92 | |
| Lower bound of 95% bootstrap interval | 0.50 | |
| Lowest after omitting one source | 0.73 (US bank omitted) | |
| Lowest after omitting one model | 0.90 | |
| Models differ on real changes (permutation test) | ||
| excluding recognized changes† | 0.83 | |
| Verdict | agree |
Table 5. Criteria for agreement between the benchmark and real-change rankings (12 models, 33 changes). The omission checks report the lowest correlation after removing each source or each model in turn. †Specified after the benchmark results were available (§ 8.3).
Agreement Between Rankings
The comparison met all five pre-specified criteria for agreement, as well as the recognition check added later (Table 5). Across the 12 models, . A 95% percentile bootstrap interval, obtained from 2,000 resamples of the real changes with randomized trials resampled by study, is 0.50–0.92. The upper bound coincides with the observed value because is close to its maximum, so resampling can lower the agreement much more than it can raise it. The interval holds the benchmark ranking fixed and therefore does not reflect uncertainty in the benchmark scores. A one-sided permutation test of the rank correlation, which shuffles the models’ real-change scores 10,000 times, gave . This differs from the test in Table 5, which asks whether the models’ real-change scores differ from one another at all (). That weaker result means that differences among models on the real changes are only modestly resolved, which is why ranks in the middle of the table reorder easily.
Figure 5a illustrates the two rankings. The same five models led both rankings, and the same three trailed. Most models moved by at most one rank. The largest shift was Sonnet, which ranked second on the benchmark but fifth on real changes: the real-change scores of the models ranked fourth to eighth lie within 0.2 of one another (−1.18 to −0.98), so small differences reorder them. The agreement is most sensitive to the two US bank changes: omitting them lowers to 0.73, which still meets the criterion.

Figure 5. Agreement with real organizational changes. (a) Forecast ranks on the simulated banks and on the 33 real changes. (b) Share of model pairs ordered as on real changes, for the benchmark ranking, a ranking by output price and the LMArena leaderboard of 27 October 2026 (66 pairs; 55 for LMArena, which does not list every model). ∗Post hoc (§ 8.4).
The benchmark ranking also ordered model pairs more accurately than two common alternatives (Figure 5b). It identified the better forecaster on real changes in 89% of model pairs. In contrast, ranking models by their price per output token did so in 67% of pairs, and the LMArena text leaderboard, which ranks models by crowd-sourced pairwise preferences [6], did so in 60% of the 55 pairs among the 11 models it lists. This LMArena comparison is post hoc: our pre-specified plan included LMArena only if it listed every model, and it does not list Qwen Flash.
The Role of Uncertainty Estimates
Both evaluations reward intervals that are narrow enough to be informative yet wide enough to contain the outcome. The agreement could therefore reflect how well models judge their uncertainty rather than how accurately they estimate effect sizes. Consistent with this, interval coverage was strongly correlated across the two settings ().
This suggests that interval calibration contributes to the agreement. How much of it reflects accurate estimates of effect size remains unclear. A high benchmark rank may therefore reflect well-judged uncertainty as much as accurate point forecasts, and should be read accordingly.
Conclusion and Future Work
In this paper, we proposed RehearsalBench, a benchmark that evaluates how LLMs choose and forecast operational changes in simulated bank contact centers. The simulator was built and validated on historical call records and combined with documented assumptions about the effects of AI interventions. Most models chose plans worth 5–7% of the no-change cost, and every model predicted the direction of each effect correctly on average. Nevertheless, models from all developers overstated the size of several effects, most notably the increase in repeat contacts under a voice agent. A manager deciding whether to adopt such a technology needs to know not only the direction of a change but also whether its size justifies the cost. Furthermore, the benchmark’s forecast ranking agreed closely with the models’ ranking on 33 real organizational changes, although part of this agreement may reflect how well models judge their own uncertainty.
Two design choices make these measurements possible. Because every plan is replayed on the same call sequences, decision value isolates the effect of each plan from random variation. Because every model forecasts the same assigned interventions, forecasts can be compared fairly even when models recommend different plans. Our findings suggest that simulated rehearsal is a promising way to screen AI decision-support tools before deployment. Candidate applications include contact-center management, workforce planning and other service operations that can be modeled as queues.
This study focuses on bank contact centers, and its scope suggests several directions for future research:
Other domains: The benchmark should be extended beyond contact centers, starting with the test on software teams adopting AI tools that we have designed.
Recent and observed data: The simulator is calibrated on call records from 1999–2003, and the effects of AI interventions rest on documented ranges. Future studies should calibrate it on recent records and on observed outcomes of AI deployments as these become available.
A wider mix of interventions: Staffing changes dominated the best-known plans in our banks. Future versions of the benchmark should include banks in which other interventions are the best choice, to test models’ judgment about those interventions more fully.
Value in operating centers: Our comparison tests whether forecast rankings carry over to real changes. Establishing the value of recommended plans will require evaluating them directly in
operating centers.
Sources of error and training: Future research is suggested to test whether the shared over-statements arise from the vendor materials given to the models, and to use the simulator as an environment for training agents with reinforcement learning.
Reproducibility Statement
We publish model-level results, the leaderboard and the figures at https://github.com/afterflowinc/rehearsalbench. To protect future evaluations, we keep the simulation engine, evaluator, scoring code, generated banks and real-change materials private. The call records are available from the Technion SEELab archive.
Ethics and Disclosures
The archive records are anonymized. Aggregate figures from a private contemporary contact center informed only the direction of some parameter ranges, and we withhold the center’s identity and figures. Claude Opus 5.5 was used as a coding assistant in developing the benchmark software and was not evaluated (§ 4.3).
References
- [1]Axel Backlund and Lukas Petersson. Vending-Bench: A benchmark for long-term coherence of autonomous agents. arXiv:2502.15840, 2025.arxiv.org/abs/2502.15840
- [2]Sebastian Bordt, Harsha Nori, Vanessa Rodrigues, Besmira Nushi, and Rich Caruana. Elephants never forget: Memorization and learning of tabular data in large language models. In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2404.06209
- [3]Lawrence Brown, Noah Gans, Avishai Mandelbaum, Anat Sakov, Haipeng Shen, Sergey Zeltyn, and Linda Zhao. Statistical analysis of a telephone call center: A queueing-science perspective. Journal of the American Statistical Association, 100(469):36–50, 2005.
- [4]Erik Brynjolfsson, Danielle Li, and Lindsey Raymond. Generative AI at work. The Quarterly Journal of Economics, 140(2):889–942, 2025.
- [5]Haozhe Chen, Karthik Narasimhan, and Zhuang Liu. CEO-Bench: Can agents play the long game? arXiv:2606.18543, 2026.arxiv.org/abs/2606.18543
- [6]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasiios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot Arena: An open platform for evaluating LLMs by human preference. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2403.04132
- [7]Noah Gans, Ger Koole, and Avishai Mandelbaum. Telephone call centers: Tutorial, review, and research prospects. Manufacturing & Service Operations Management, 5(2):79–141, 2003.DOI
- [8]Ofer Garnett, Avishai Mandelbaum, and Martin Reiman. Designing a call center with impatient customers. Manufacturing & Service Operations Management, 4(3):208–227, 2002.DOI
- [9]Gartner. Gartner survey finds only 14% of customer service issues are fully resolved in self-service. Press release, https://www.gartner.com/en/newsroom/press-releases/2024-08-19-gartner-survey-finds-only-14-percent-of-customer-service-issues-are-fully-resolved-in-self-service, 2024. 19 August 2024.
- [10]Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007.
- [11]Information Age. CBA reverses AI-driven job cuts, admits ‘error’. ACS Information Age, https://ia.acs.org.au/article/2025/cba-reverses-ai-driven-job-cuts--admits--error-.html, 2025. 21 August 2025.
- [12]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.06770
- [13]Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock. ForecastBench: A dynamic benchmark of AI forecasting capabilities. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2409.19839
- [14]Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, et al. Measuring AI ability to complete long software tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2503.14499
- [15]Alejandro Lopez-Lira, Yuehua Tang, and Mingyin Zhu. The memorization problem: Can we trust LLMs’ economic forecasts? arXiv:2504.14765, 2025.
- [16]Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023.arxiv.org/abs/2304.03442
- [17]Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, et al. GDPval: Evaluating AI model performance on real-world economically valuable tasks. In International Conference on Learning Representations (ICLR), 2026.
- [18]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations (ICLR), 2025.DOI
- [19]Yang Yao, Haozhe Chen, Bingyi Kang, Karthik R. Narasimhan, and Zhuang Liu. SWEEper-Bench: Can agents discover bugs in interactive software? arXiv:2610.14096, 2026.arxiv.org/abs/2610.14096
