WorksComputer Science
AutoSim: A Self-Improving User Simulator for Conversational Search Evaluation
Abstract
User simulators promise to evaluate conversational search systems at a scale human studies cannot reach, but only if their behavior is realistic enough to trust. We study AutoSim, a conversational-search user simulator whose relevance judgments rank a controlled pool of retrieval systems in agreement with NIST across three TREC iKAT years (Kendall's of , , and ; all ), an agreement that survives removing its persona. AutoSim self-improves against a held-out signal from real users, and we find that fine-tuning its weights makes it behave like real users, which in turn gives a more accurate estimate of how well a system serves them. Reaching that behavior takes more than prompting: when discriminating between real and simulated user sessions, no prompt we tried lowers a held-out real-versus-simulated classifier below AUC , whereas fine-tuning the weights drives it to , where neither the classifier nor a human reading full dialogues can tell it from a real user. That added realism is what makes the evaluation more accurate: where real users retrieve little, a prompted simulator over-estimates retrieval effectiveness by nDCG and returns a relevant passage on of the turns where real users find none, enough to change a deployment decision. Only the weight-level simulator escapes this bias. We release the framework and code.
CCS Concepts
•Information systems → Users and interactive retrieval; Evaluation of retrieval results.
Introduction
User simulation is a standard way to evaluate interactive and conversational IR systems without running a human study for every design choice. A simulator stands in for a population of users, so retrieval and response strategies can be compared reproducibly when tuning a ranker or a clarifying-question policy.
Because a simulator is fixed at design time as a set of personas, prompts, and decision rules, its behavior drifts from that of real users as the systems it evaluates change. Worse, even where it does not drift, a simulator is a documented unreliable proxy: changing only the underlying user model swings the evaluated agent’s measured task-success rate by about nine percentage points, calibration against humans is poor, and general-purpose LLM simulators systematically over-estimate success [19, 32]. The problem runs deeper than tuning: a more realistic simulator is not necessarily a more accurate evaluator [4].
This gap shows up directly in the utterances: asked to play a search user, a strong LLM writes long, polite, fully formed requests, whereas real users are brief and lean on the prior conversation for what they leave unsaid. Measuring it on TREC iKAT (§3) locates the gap in what the simulated user says and asks for rather than in how the simulator judges relevance, and shows that editing the generation prompt does not close it, which is what tells us where self-improvement has to act.
Two lines of work relate to this problem, and they have developed independently. Trained user models are more realistic than prompted ones [10, 33], and self-improving agents raise their own behavior by editing themselves, from prompts to weights [1, 34]; neither asks what a more realistic simulator does to evaluation validity. We ask whether a simulator can close its own gap to real users, whether that closing has to happen at the prompt or the weights, and whether closing it makes the simulator a more accurate evaluator. Therefore, we study AutoSim, a hybrid orchestration-based simulator that self-improves in a loop checked against a held-out signal from real users (Figure 2), and we vary what the loop edits: the generation prompt, or the model weights. The two behave differently (Figure 1). A prompt-level loop does not reduce the behavioral gap below the level a single hand-written prompt already reaches, whereas a weight-level loop reduces it until the discriminator can no longer separate the two.
Figure 1. Real-vs-simulated discriminator AUC for four simulator configurations on held-out iKAT: a default prompt, a tuned prompt, few-shot prompting, and a weight-level fine-tune (AutoSim), each scored by an embedding discriminator and a fine-tuned XLM-RoBERTa discriminator. An AUC of 0.5 means real and simulated utterances are indistinguishable; higher is more separable.

Figure 2. AutoSim drives a real conversational-search system. The loop records the simulator’s traces and proposes edits, accepting an edit only if it does not regress on a held-out split. The loop can edit two components: the relevance-judging instruction and the utterance generator.
We make three contributions, each backed by its own experiments.
(1) A validated evaluator (§ 3.4, § 4.1). AutoSim ranks real conversational-search systems in agreement with NIST across three iKAT Track-years ( on 2023[2]; on 2024 and on the multi-session 2025 [3]; all ), with a re-scoring standard deviation of 0.01, with two real participant runs added (), under a second assessor family, and on a second corpus (CAsT 2020[6], , rising to 0.58 when a neural reranking tier is added to the pool). The agreement comes from the relevance-assessment step, which is why it persists when the persona is removed.
(2) A limit that depends on where self-improvement acts (§5.2). Trained user models are known to be more realistic than prompted ones [10, 33]; we measure where the prompt reaches its limit. Prompt-level self-improvement does not lower the real-versus-simulated discriminator below an AUC near 0.85, and a single hand-written prompt matches an iterative loop and few-shot prompting. Weight-level self-improvement lowers the AUC to 0.45 with human-level lexical diversity; the drop replicates on a second model family (Mistral-7B) for communication style, and the match extends to four behavioral measures on the primary model.
(3) Realism changes the evaluation (§5.3). Where real users’ own queries retrieve few relevant passages, a prompt-level simulator over-estimates retrieval effectiveness by +0.136 nDCG. Matching length is not enough: a terse prompt still over-estimates by +0.087, and only the weight-level simulator is not detectably biased (point estimate +0.014, confidence interval including zero). The over-estimate hides retrieval failures, can change a deployment decision at a modest quality threshold, and does not appear in the summary metrics the field usually reports. We release the framework (LLMs with SPLADE, BM25, and a cross-encoder over ClueWeb22-B and NIST qrels), the evaluation protocol, and a blind human-discrimination study with its kit 1.
Section §2 positions the work; §3 measures where the gap lives; §4 describes the simulator, the two components the loop can edit, and the validity protocol; §5 answers the three questions in order.
Related Work
Self-evolving agents improve from their own experience by editing different parts of the system: prompts via reflective evolution [1]; memory via experience distillation [22]; weights via self-edits [34] or self-play [9, 30]; and the reward model itself via self-rewarding or self-taught evaluation [20, 24]. A recurring lesson is that the strongest gains come when the outer-loop signal is a real held-out performance signal, which we build in through a held-out acceptance check. These works establish that this choice matters in general; we measure, for a user simulator, whether the prompt or the weights reduces the gap to real users, and by how much.
User simulation for search ranges from rule-based interactive-IR toolkits to LLM agents that issue queries, click, give feedback, and ask clarifying questions [13, 16, 18, 28, 29], and to dedicated trained user language models [10, 23]. Discriminator and utterance analyses of conversational-search simulators find that instruction-tuned assistant models make poor users, being too cooperative, wordy, and polite [17, 21]. Trained user language models already outperform prompted ones for realism [10]. We add two results: a measured limit on prompt-level self-improvement, where a single hand-written prompt matches an iterative loop, and the effect on evaluation of removing that limit with weight updates. Our base follows a hybrid orchestration simulator [26]; the loop, the discriminator, and the validity results do not depend on that choice.
The closest prior work is DIAL [33], which improves a user simulator by adversarial preference optimization against a realism discriminator in mental-health support dialogue, driving the discriminator toward 0.5 while restoring the diversity that supervised fine-tuning removes. Our weight-level variant reproduces that effect in conversational search, using rejection-sampling SFT rather than DIAL’s DPO objective. Our addition is the contrast with prompt-level self-improvement and the effect on retrieval-effectiveness estimates; DIAL reports a related bias in dialogue-outcome rates, but tests neither a prompt baseline nor a graded retrieval metric.
Validating simulators rests on three modes, system-ranking agreement, behavioral-distribution match, and human realism [5, 27], with the training/evaluation objective divergence formalized separately [4]. Profile-conditioned simulators reach ranking correlation with human judgments [7], though that conditioning is a fixed per-run profile that does not adapt during the interaction, and others measure the gap without closing it [32]. A recent evaluation toolkit reports that the discriminator human-likeness check has little pooled predictive power for system-ranking validity [25], consistent with our finding that ranking validity is a property of the judge, so we use the discriminator to certify realism, not to predict ranking. Prior work establishes these properties separately; AutoSim combines behavioral-gap closing and ranking validity in one system on real conversational-search data, and places the prompt-versus-weight analysis beside them. Because part of AutoSim scores relevance with an LLM, we keep judge and generator in different model families, mindful of the known style, verbosity, and self-preference biases [14, 15, 31].
The Sim-to-Real Gap
We first measure how the default simulator differs from real users on TREC iKAT[2, 3]. The measurements separate into two parts: its relevance judging already agrees with NIST, while its user behavior is far from real.
The behavioral gap. A strong LLM’s persona-conditioned relevance assessments already rank a controlled pool of retrieval systems in agreement with NIST (Kendall’s up to 0.91), and the agreement survives removing the persona, so ranking validity is not a property self-improvement has to create; § 4.1 establishes it in full. What a more realistic simulator can still change is how the simulated user behaves, and there the default is far from a real searcher: asked to play a search user, the model writes long, polite, fully formed requests, at 37.5 words per message against 12.8 for real users, with a held-out classifier separating its utterances from real ones at AUC 0.88 and its queries retrieving the passages a real user retrieves almost never (overlap@5 0.01). The departure is in what the user says and asks for, not in how the simulator judges relevance.
The limit of prompting. Editing the generation prompt moves the summary statistics while the underlying behavior remains unchanged: every variant we tried, a single hand-written terse instruction, an iterative reflective loop, and few-shot prompting with real terse utterances, still leaves a held-out classifier separating real from simulated sessions at AUC (few-shot 0.94, the default’s own value) compared to a expected random baseline result of 0.5, so the fixable gap is behavioral and lies beyond the prompt.
Together these measurements point to a single design: leave the relevance judge unchanged, target the utterance generator, and edit it at the weights rather than the prompt. § 3 builds that simulator; § 4 confirms each step: the judge keeps the ranking valid (RQ1), weight-level editing closes the gap the prompt could not (RQ2), and only that closure removes an evaluation bias (RQ3).
AutoSim
Simulator and editable components
A prompted LLM simulator takes a user state and emits the next utterance under a fixed instruction. AutoSim keeps that generator and wraps it in state : persona (a personal text knowledge base of background statements), information need , dialogue history , and a latent cognitive or affective state (satisfaction, frustration, patience). An orchestration policy selects, at each turn, which component acts: deterministic simulators (query reformulation, click, stopping) for grounded routine behavior, or LLM cognitive functions (answer a clarifying question, give feedback, express satisfaction, generate the next utterance). We instantiate the base following a hybrid orchestration simulator [26]; since the components we edit exist in any LLM-based simulator, our results are not specific to this base.
This paper studies where self-improvement should act, so we expose two components the loop can edit and hold everything else fixed. The first is the relevance-judging instruction, the prompt the assessor applies; it sets the simulator’s satisfaction and thus the ranking it induces. The second is the utterance generator, the model that writes what the simulated user says. The generator can be edited in two ways: at the prompt, which changes its instruction, or at the weights, which changes its parameters. This prompt-versus-weights distinction is the variable our results depend on.
Gap metric (CSG)
We measure the simulator’s gap to real users along two of the validity modes of [5]: system-ranking agreement and behavioral-distribution match. We call the combined measurement the Conversational-search Sim2Real Gap (CSG), which lets us test whether improving one mode also improves the other. A ranking term measures evaluation validity (§ 3.4), while behavioral terms measure realism. We measure the first behavioral dimension, communication style (D1, verbosity and politeness), in two ways, and the difference between them is itself a result. The first is a set of aggregate style statistics (mean words per message, politeness rate, short-message rate), which are easy to read off and easy to write into a prompt. The second is a held-out real-versus-simulated discriminator, a classifier trained to tell a simulated utterance from a real one, reported as ROC AUC with 0.5 meaning the two are indistinguishable; this is the adversarial realism criterion from generative-adversarial training [8], and unlike the summary statistics it cannot be met by matching a few numbers. We measure D1 in full, and we examine reformulation and clarification behavior (D2–D3) through query-need alignment, the overlap of the passages the simulator retrieves, and its mix of conversational moves (§5.2); click, satisfaction, and stopping (D4–D6) are left to future work.
Self-improvement Loop
In each cycle, a controller records structured traces (every turn’s policy choice, retrieval, click, and utterance), clusters the differences between simulated and real behavior, derives per-cluster evaluators with the judge drawn from a different model family than the generator, improves the chosen component by reflective population search, accepts an edit only when it does not regress on a calibration split, and monitors a rolling held-out set. The acceptance check is the loop’s guarantee: it never keeps an edit that worsens calibration.
Validity Testbed
We test whether AutoSim ranks IR systems as humans do on three independent TREC iKAT collections [2, 3]: iKAT 2023 (176 judged turns), iKAT 2024 (116 turns), and the multi-session iKAT 2025 (45 turns), each with NIST passage qrels over the same ClueWeb22-B corpus [12]. iKAT is the only public collection with all four of the properties we need at once: multi-turn conversational structure, per-user PTKB personas, passage-level NIST qrels, and a corpus we can query at run time. On each year we build a controlled pool of twelve constructed systems (retrieval pipelines we assemble) over ClueWeb22-B, spanning a wide effectiveness range: first stages of BM25, learned-sparse SPLADE, and reciprocal-rank fusion; over raw, context-resolved, and PTKB-expanded queries; with and without a cross-encoder reranker. For iKAT 2023, where two organizer runs (a BM25+RM3 pipeline with automatic and with manual rewriting) are available with pooled judgments, we fold them in for a fourteen-system pool with real participant systems.
The reference ranking orders systems by mean condensed nDCG@5 over the NIST qrels, condensed so that non-pooled systems are not penalized for retrieving relevant but unjudged passages [5]; we report raw nDCG@5 alongside. AutoSim’s induced ranking orders systems by its persona-conditioned mean satisfaction with each system’s top-5 passages. We report Kendall’s (ties-corrected) with a Monte-Carlo permutation test and a system-level bootstrap CI that resamples the systems, since the system is the unit of the test. SPLADE and BM25 run over a 116M-passage ClueWeb22-B index, reranking and assessment use production LLMs, and the generator and judge come from different model families.
Setup and reproducibility
The loop runs reflective population search: each cycle proposes candidate edits to the chosen component (generated by the judge from the observed gap), scores each on a calibration split, selects the best, and keeps it only if it does not regress. We validate on a disjoint held-out split fixed before any run, and compare against static (no self-improvement) and NO-CALIBRATION (edits accepted without the calibration check). A strong model’s default style is verbose, so we initialize the generator prompt from a deliberately weak style, and report both the gain from that start and the distance from the model’s default. To edit the generator at the weights, we warm-start a small open generator (Qwen2.5-3B with a LoRA adapter) on real iKAT utterances, then run adversarial rejection-sampling rounds: each round samples several utterances per held-out turn, keeps the ones a frozen real-versus-simulated discriminator rates most human, fine-tunes on that set, refreshes the discriminator, and repeats for four rounds [33]. Freezing the discriminator within a round and taking the previous round as the reference keeps the target from moving.
Held-out utterances are the 82 iKAT turns disjoint from the SFT split (seed-fixed); the discriminator sees 2 samples per held-out turn against the matched real utterances, and every AUC is scored five-fold with folds grouped by turn, so a turn’s real and simulated utterances never split across train and test. The weight-level generator is Qwen2.5-3B/7B-Instruct (and, for the cross-family check, Mistral-7B-Instruct-v0.3) with a LoRA adapter (, , projections), warm-started and then improved over four adversarial rounds. The in-loop discriminator is a logistic classifier over multilingual-MiniLM embeddings; the second discriminator is a fine-tuned XLM-RoBERTa-base. The validity pipeline uses two open-weight instruction models from different families (an 80B-class Qwen generator and a 31B-class Gemma judge), both at temperature 0 and served without any proprietary API, so the whole pipeline runs on released weights. The RQ1 rank-correlation, RQ2 discriminator (Table 3), and RQ3 effectiveness-bias numbers are all reproducible from committed result files, and we release the configurations and the trained LoRA adapters .
Evaluation
The evaluation answers three questions, each building on the last. RQ1: does AutoSim rank IR systems as NIST does (§5.1)? RQ2: can self-improvement close the behavioral gap to real users, and at the prompt or the weights (§5.2)? RQ3: does closing the gap make the simulator a better evaluator (§5.3)?
RQ1: Is AutoSim a valid evaluator?
AutoSim’s persona-conditioned satisfaction agrees with the NIST ranking at on the twelve-system pool (condensed nDCG@5; on raw), Monte-Carlo permutation . The system-level bootstrap 95% CI on is [0.30, 0.90], wide as a twelve-system test must be and excluding zero; no single system drives the ranking (leave-one-out ), and re-scoring the pool five times gives (Table 1). Adding the two real participant runs (Table 1, starred) leaves the picture much the same: over fourteen systems (, leave-one-out ), stays at 0.59 even when the weak auto run is removed, and AutoSim ranks that weak official run last exactly as NIST does. The agreement is aggregate and not perfect for every pair: AutoSim under-ranks the strong manual run, placing it 10th of 14 where NIST places it 3rd, its largest single disagreement. As a sanity check, naive PTKB query-stuffing degrades retrieval, so the (ptkb) rows fall in the reference ranking, and AutoSim independently ranks them low.
Re-running the same twelve-system protocol and live ClueWeb22-B pipeline on two further collections shows the result holds beyond 2023: on iKAT 2024 (116 turns) AutoSim agrees more strongly ( condensed, 0.85 raw; ; CI [0.73, 1.00]; leave-one-out ), and on the multi-session iKAT 2025 (45 turns) it again reaches condensed (0.76 raw, , CI [0.73, 1.00]). Where validity is weakest is year-dependent: restricted to the six closest-quality systems, falls to 0.33 on 2023 (, n.s.) but recovers to 0.87 on 2024 (, suggestive rather than family-wise corrected), so near-tie resolution, not the ranking itself, is what varies. Off ClueWeb22-B, on TREC CAsT
| System | nDCG@5 | cond. nDCG | AutoSim |
| Context-resolved queries (best retrieval) | |||
| rrf(resolved) | 0.287 | 0.343 | 0.632 |
| splade(resolved) | 0.210 | 0.295 | 0.661 |
| bm25(resolved) | 0.225 | 0.276 | 0.553 |
| rrf+rerank | 0.124 | 0.228 | 0.594 |
| bm25+rerank | 0.107 | 0.200 | 0.506 |
| splade+rerank | 0.093 | 0.199 | 0.604 |
| PTKB-expanded queries (query stuffing) | |||
| splade(ptkb) | 0.090 | 0.148 | 0.553 |
| rrf(ptkb) | 0.066 | 0.133 | 0.506 |
| splade(ptkb)+rerank | 0.050 | 0.105 | 0.546 |
| bm25(ptkb) | 0.035 | 0.058 | 0.289 |
| Raw queries (no context) | |||
| splade(raw) | 0.062 | 0.096 | 0.334 |
| bm25(raw) | 0.088 | 0.095 | 0.277 |
| Real organizer runs | |||
| bm25_rm3(manual)★ | 0.289 | 0.289 | 0.505 |
| bm25_rm3(auto)★ | 0.033 | 0.033 | 0.155 |
Table 1. Twelve constructed systems grouped by query type, plus two organizer runs (★); 176 turns. Resolved rewrites the turn to stand on its own by filling in the conversational context; PTKB stuffs the query with persona statements; raw uses the turn as-is. , ; sorted by condensed nDCG within each group.
2020 [6] over MS MARCO [11], run persona-free over a ten-system pool, satisfaction still tracks NIST (, , CI [0.11, 0.85]), and it holds when a neural cross-encoder rerank tier joins the pool ( over fifteen systems, , CI [0.28, 0.79]; even the five neural systems on their own rank in the right order at ), so the agreement does not depend on a lexical-only pool; the signal is weaker than on iKAT and limited by about half corpus coverage, but positive on a corpus AutoSim was not built on.
A persona-free relevance judge over the same passages ranks the pool at , on par with the persona-conditioned 0.72 (both the re-scored mean; Table 2), so the validity does not depend on the persona; re-scoring the 2024 pool with a different-family assessor (Gemma against the Qwen default) reproduces the ranking unchanged (), the two assessors agreeing at . The agreement comes from relevance-assessment ability, which strong LLMs already have; the persona is what makes AutoSim a user and carries its behavioral realism. Across the collections we tried, AutoSim ranks systems as NIST does wherever they differ clearly in quality, an agreement its relevance judgments drive.
| Setting | note | |
| iKAT 2023 (176 turns) | ||
| 12 constructed systems | 0.66 | perm. , CI [0.30, 0.90] |
| +2 real participant runs (14) | 0.65 | , CI [0.26, 0.91] |
| re-scored ×5 (T=0) | 0.72 | ±0.01 (reproducible) |
| persona-free relevance judge | 0.79 | persona not the source of validity |
| 6 closest-quality systems | 0.33 | (n.s.): near-ties hard |
| iKAT 2024 (116 turns), same protocol | ||
| 12 constructed systems | 0.91 | perm. , CI [0.73, 1.00] |
| 6 closest-quality systems | 0.87 | : near-ties resolved |
| iKAT 2025 (45 turns), multi-session | ||
| 12 constructed systems | 0.91 | perm. , CI [0.73, 1.00] |
| cross-corpus: CAsT 2020 / MS MARCO (10 sys) | ||
| different corpus, no personas | 0.56 | , CI [0.11, 0.85] |
Table 2. Kendall- on condensed nDCG@5 under stress; CI is the system-level bootstrap on . The persona-free row shows validity does not come from the persona.
RQ2: Can self-improvement close the behavioral gap, at the prompt or the weights?
The diagnosis (§3) placed the fixable gap in the utterance generator and showed the prompt does not close it. We now hold the loop to that gap and measure how far each, prompt-level and weight-level, gets on the two measures of §4.
Figure 3. RQ1. Sim-to-real validity: AutoSim’s satisfaction tracks NIST condensed nDCG@5 across fourteen systems (Kendall , ). The two orange points are the real iKAT organizer runs (bm25+rm3 over manually vs. automatically rewritten queries); AutoSim rates the weak automatic run lowest, as NIST does.
At the prompt, across twelve seeds, the loop raises held-out communication-style fidelity above the no-evolution baseline: final-cycle CSG is 0.76±0.10 for the full loop against 0.56±0.07 for STATIC, a gain significant under a paired -test ($p=0.0002). Table 5 shows what it fixes: the default user averages 37.5 words against 12.8 for real users, self-improvement brings it to 11.0, and the length distribution moves much closer to human (Kolmogorov–Smirnov against 0.81). Two limits qualify this. The blind NO-CALIBRATION variant reaches only 0.63±0.07 and does not clear STATIC at significance (), so the calibration signal is what makes the loop reliably improve, and the acceptance check adds a guarantee against regression. And style is not matched exactly: the tuned style overshoots politeness (39% against 12%) and misses the 5% of very-short Real-versus-simulated discriminator AUC on the same base generator; 0.5 is indistinguishable from real, higher is more separable. A held-out XLM-RoBERTa discriminator and two further base models confirm both extremes §5.2.
| avg. words | polite % | very-short % | |
| Real users (human) | 12.8 [11.3,14.3] | 12 | 5 |
| AutoSim default | 37.5 [35.2,39.7] | 21 | 0 |
| AutoSim tuned | 11.0 [10.2,11.7] | 39 | 0 |
Table 5. RQ2. What the behavioral loop fixes on communication style. Message-style features against real iKAT users; of the held-out turns, brackets = bootstrap 95% CI on the mean.
Behavioral: held-out CSG-D1 final cycle, mean±SD over 12 seeds; the full loop improves on static at (paired ), the blind variant does not (). Assessor: held-out ranking ; the check finds no surviving edit.
human messages, so its distribution stays distinguishable (). The loop closes the gap that dominates and leaves the rest.
A sharper comparison is against a single hand-written prompt. On the aggregate statistics the loop optimizes, the loop does worse: a one-line prompt (“write a short message, about 12 words, and skip pleasantries”) reaches CSG-D1 0.80 against the loop’s 0.68 on the same turns, because three summary statistics are easier to write directly than to reach through search. On the discriminator the loop does better (AUC 0.85 against above 0.9 for the default and terse prompt, on both an embedding and a lexical classifier), but still does not reach 0.5. Few-shot prompting with real terse utterances, the strongest prompt baseline, matches real length (12.3 words) yet a held-out classifier still separates it at AUC 0.94, the default’s value.
Moving the same objective to the weights, the discriminator can no longer separate the weight-level simulator from real users: AUC falls from 0.88 to 0.45 and holds there across four adversarial rounds (Table 3). The utterances behind the drop stay diverse: they are as mutually distinct as real ones (mean pairwise token overlap 0.043 against 0.044, every utterance unique), match their length variety (entropy 3.1 against 3.1) and mean length (12.3 against 13.3 words), and only 1.2% copy a training utterance, though their vocabulary is somewhat narrower (word distinct-2 0.54 against 0.84). The match extends beyond phrasing to three further behavioral measures. In what the user retrieves, the weight-level queries retrieve the same ClueWeb passages as the real query an order of magnitude more often than any prompt-level variant (overlap@5 0.17 against 0.01 default and 0.015 terse). In how the user moves, classified into new request, refinement, feedback, and clarification, the weight-level move mix matches real users (total-variation distance 0.07) where the default diverges sharply (0.34–0.46 across two classifier families), taking the clarifying and feedback turns of an assistant in place of the focused refinement of a searcher; both classifiers put the weight-level gap at 0.07, though per-utterance labels agree only weakly (=0.22), so we read the move mix distributionally. In how the user reformulates, we measure query-need alignment as the word overlap between a variant’s next-turn query and its resolved full-need form: real users write need-focused queries (overlap 0.56) and the weight-level simulator matches them (0.51), while the verbose default dilutes the need with conversational padding (0.23) and even a deliberate terse prompt, the shortest variant at 9.6 words yet the least aligned, does not recover it (0.19), so a short query is not by itself need-focused.
| self-improvement | discriminator AUC |
| none (default prompt) | 0.88 |
| prompt-level (best of four methods) | 0.84 |
| weight-level, warm SFT on real | 0.45 |
| weight-level, + adversarial rounds | 0.42–0.50 |
Table 3. RQ2. Self-improvement depends on where it acts (iKAT, held-out).
The weight-level result raises two objections, which we check directly. Is the discriminator too weak? With folds grouped by turn, a fine-tuned XLM-RoBERTa separates the default almost perfectly (AUC 0.94) yet cannot separate the weight-level one (0.51±0.06), and the same discriminator tells a lightly politeness-perturbed real utterance from an unperturbed one at AUC 0.99, so it is strong, and the 0.51 shows the two are indistinguishable. Is the result specific to one model? The whole procedure on a Qwen2.5-7B base reproduces both extremes (1.00 against 0.50), and on an independent Mistral-7B base again (0.97 against 0.40). The indistinguishability even holds against strong judges: two frontier LLMs asked to pick the human beat chance on the default (0.59, 0.89) but not on the weight-level one (0.50, 0.48, n.s.). A panel reading the full conversation stays near 0.5 on it (0.55, ) while flagging the default (0.75), and three human annotators catch the default on 28–30 of 30 trials but only 4–11 of 30 for the weight-level one, all below the 15/30 chance level, so even a human reading the whole dialogue cannot separate it from a real user. The learned style also transfers: applying the 2023-tuned style to iKAT 2024 brings the default’s 38.6 words down to 11.8 against 11.7 for real users, indistinguishable in mean length (permutation against for the default).
Self-improvement therefore reaches a limit at the prompt but not at the weights: holding the base generator fixed, prompt-level self-improvement does not lower the discriminator below AUC 0.85, whereas weight-level self-improvement on that same generator lowers it to 0.5. The relevance judge shows a third pattern: editing the assessor prompt under the check produces no accepted edit (Table 4), and unchecked edits move by up to about 0.1, which is within the sampling noise, since a strong model is already a competent judge (§5.1). Self-improvement thus has room to change how the user behaves but not how it judges; ranking validity comes from a strong judge, and self-improvement is best applied to the generator. This is consistent with the training and evaluation objectives not improving together [4], though it is not the same statement, and it does not imply that judges cannot be improved: methods that retrain the judge [20] act on the component our prompt-level check leaves unchanged.
| Component / variant | held-out fidelity |
| Behavioral, full (calibration check) | 0.76 ± 0.10 |
| Behavioral, no-calibration | 0.63 ± 0.07 |
| Behavioral, static | 0.56 ± 0.07 |
| Assessor, full (calibration check, ) | 0.61 (no surviving edit) |
Table 4. RQ2. Self-improvement improves the utterance generator but finds no accepted edit for the relevance judge.
RQ3: Does closing the gap make a better evaluator?
Closing the behavioral gap matters only if it makes the simulator a better evaluator. System-ranking does not move: the ranking is already valid (§5.1) and does not depend on which simulator writes the query. The effect appears instead on a quantity the weight-level training never optimized, the absolute retrieval effectiveness a simulator’s own queries imply. Effectiveness comes from running retrieval on the same utterances, so the two are related. The training targets distributional similarity rather than nDCG, and per-turn effectiveness correlates only weakly with the real user’s (), so the match is distributional rather than mechanical. On 157 held-out judged turns we run BM25 retrieval with each simulator’s query and compare its nDCG@5 to the real user’s. The bias is largest on the harder, fully held-out 2024 collection, where real users’ own queries retrieve little (mean nDCG 0.016): a default, wordy simulator, whose retrieval query averages 28 words, over-estimates effectiveness by +0.136 (95% CI [0.10, 0.17], , Table 6).
| query source | words | iKAT 2024a | 2023 h.o.b |
| default (verbose) | 28 | +0.136 [0.10, 0.17] | +0.009 |
| terse prompt | 9 | +0.087 [0.06, 0.12] | −0.028 |
| length-matched | 11 | +0.071 | −0.042 |
| weight-level | 12 | +0.014 [−0.00, 0.03] | +0.015 |
Table 6. RQ3. Evaluation bias by collection (iKAT held-out). Bias = mean nDCG@5 of a simulator’s query minus the real user’s; 0 unbiased, + over-estimates. Brackets: bootstrap 95% CI; significance by paired Wilcoxon; a CI excluding 0 is a detectable bias. a iKAT 2024, (real mean nDCG 0.016). b 2023 held-out, (real mean 0.080); all values n.s. (CIs include 0).

Figure 4. RQ2. Behavioral self-improvement across 12 seeds (mean±SD band): the checked loop rises above static (paired , ); the blind loop’s gain is not significant ().
The simplest explanation is length, since real users are terse, and it does not account for the effect. A hand-written terse prompt (9 words) still over-estimates by +0.087 and a length-matched truncation of the verbose query by +0.071, both with confidence intervals above zero, while only the weight-level simulator is not detectably biased (+0.014, [−0.00, 0.03], ); across query sources length explains just 2% of the variance in retrieved nDCG (). Matching verbosity is not enough; matching the full query distribution removes the bias. On the easier 2023 held-out subset, where real users retrieve reasonably (mean 0.080), no simulator is detectably biased (default +0.009, n.s.), so the over-estimate appears exactly where real users struggle and an evaluation most needs to be trusted; pooled over both collections the default over-estimates by +0.103.
This over-estimate is not a harmless level shift that could be calibrated away, because it hides the failures an evaluation exists to reveal. Real users retrieve nothing relevant on 86% of these turns; on exactly those turns a default simulator’s query still returns a
relevant passage 57% of the time, while the weight-level simulator returns one only 14% of the time and fails where real users fail. The result is a wrong decision: on 2024 an evaluation run with a default simulator reads the system’s effectiveness as 0.15 nDCG, well above a modest 0.05 deployment threshold, where real users get 0.02 and the weight-level simulator reports the same failure (0.03). A realistic simulator, built at the weight level, gives a more accurate estimate of how a system serves real users, the quantity an aggregate rank correlation hides.
Limitations and threats to validity
Our pool is a constructed one plus two real organizer runs, short of the full participant field, and the behavioral reference is organizer-authored iKAT utterances, a proxy for logged end-users; the human evidence rests on three annotators (§5.2), who agree with the discriminators (28–30 of 30 on the default, 4–11 of 30 on the weight-level one); we release the protocol for a larger study. The validity holds for systems that differ clearly in quality: near-tie resolution is reliable on 2024 but only marginal on 2023, whose is wide, and the 2025 result rests on 45 judged turns; we read the borderline tests in Table 2 (–0.03) as descriptive rather than family-wise corrected. The behavioral results cover four axes across D1–D3, leaving click, satisfaction, and stopping (D4–D6) for future work, and the cross-corpus step (CAsT 2020, –0.58, ) is limited by about half corpus coverage. We control for contamination with disjoint calibration and validation splits fixed in advance, and apply weight-level self-improvement only to generation; the relevance judge, near its ceiling under our prompt-level check, may still move under methods that retrain it.
Conclusion
Editing the prompt cannot close a simulator’s behavioral gap to real users; editing the weights can; and that closure removes an evaluation bias. We show an AutoSim, a conversational-search user simulator that is first a valid IR evaluator ( across three iKAT years, reproducible, stable with real participant runs, and holding across corpora and assessor families) whose validity comes from relevance-assessment competence rather than the persona. The limit depends on where self-improvement acts: prompt-level self-improvement, hand-written, tuned, or few-shot, plateaus at a real-versus-simulated discriminator AUC near 0.85, while weight-level self-improvement drives it to 0.5 across two model families and four behavioral axes. The closure matters because a default simulator over-estimates retrieval effectiveness by +0.136 nDCG where real users struggle and hides 57% of their retrieval failures, while the weight-level simulator is not detectably biased; a realistic simulator, built at the weight level, is a more valid evaluator. Both effects are visible only on a learned discriminator and an effectiveness-bias test, a caution for anyone measuring simulator quality on summary metrics. Future work extends the behavioral axes to satisfaction and stopping, replaces the organizer-authored reference with logged users, and broadens the cross-corpus step to fuller corpus coverage. We release the framework, the protocol, and the models1.
References
- [1]Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Jon Stoica, Daniel Klein, Matei Zaharia, and Omar Khattab. 2025. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. CoRR abs/2507.19457 (2025). arXiv:2507.19457 doi:10.48550/ARXIV.2507.19457
- [2]Mohammad Aliannejadi, Zahra Abbasianbtaeb, Shubham Chatterjee, Jeffrey Dalton, and Leif Azzopardi. 2024. TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview. In Proceedings of the Thirty-Second Text Retrieval Conference (TREC). NIST. arXiv:2401.01330
- [3]Mohammad Aliannejadi, Zahra Abbasianbtaeb, Simon Lupart, Shubham Chatterjee, Jeffrey Dalton, and Leif Azzopardi. 2025. TREC iKAT 2024: The Interactive Knowledge Assistance Track Overview. In Proceedings of the Thirty-Third Text Retrieval Conference (TREC). NIST.
- [4]Krisztian Balog, Nolwenn Bernard, Saber Zerhoudi, and ChengXiang Zhai. 2025. Theory and Toolkits for User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, 4138–4141. doi:10.1145/3726302.3731697DOI
- [5]Nolwenn Bernard, Krisztian Balog, Saber Zerhoudi, and ChengXiang Zhai. 2026. Large Language Models for the Simulation of Human Behavior. (2026). https: //hal.science/hal-0548009 Working paper, HAL preprint hal-0548009.
- [6]Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020. CAsT 2020: The Conversational Assistance Track Overview. In Proceedings of the Twenty-Ninth Text Retrieval Conference (TREC). NIST.
- [7]Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. 2025. SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 35212–35290. doi:10.18653/V1/2025.EMNLP-MAIN.1786DOI
- [8]Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27 (NIPS), 2672–2680.
- [9]Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. 2025. R-Zero: Self-Evolving Reasoning LLM from Zero Data. CoRR abs/2508.05004 (2025). arXiv:2508.05004 doi:10.48550/ARXIV.2508.05004
- [10]Tarek Naous, Philippe Laban, Wei Xu, and Jennifer Neville. 2025. Flipping the Dialogue: Training and Evaluating User Language Models. CoRR abs/2510.06552 (2025). arXiv:2510.06552 doi:10.48550/ARXIV.2510.06552
- [11]Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated Machine Reading Comprehension Dataset. CoRR abs/1611.09268 (2016). arXiv:1611.09268arxiv.org/abs/1611.09268
- [12]Arnold Overwijk, Chenyan Xiong, and Jamie Callan. 2022. ClueWeb22: 10 Billion Web Documents with Rich Information. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, 3360–3362. doi:10.1145/3477495.3536321DOI
- [13]Paul Owoicho, Ivan Sekulić, Mohammad Aliannejadi, Jeffrey Dalton, and Fabio Crestani. 2023. Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, 632–642. doi:10.1145/3539618.3591683
- [14]Arjun Panikkersy, Samuel R. Bowman, and Shi Feng. 2024. LLM Evaluators Recognize and Favor Their Own Generations. In Advances in Neural Information Processing Systems 37 (NeurIPS). arXiv:2404.13076
- [15]Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity Bias in Preference Labeling by Large Language Models. CoRR abs/2310.10076 (2023). arXiv:2310.10076arxiv.org/abs/2310.10076
- [16]Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2022. Evaluating Mixed-initiative Conversational Search Systems via User Simulation. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM). ACM, 88–96. doi:10.1145/3488560.3498440DOI
- [17]Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2024. Analysing Utterances in LLM-based User Simulation for Conversational Search. ACM Transactions on Intelligent Systems and Technology 15, 3 (2024), 62:1–62:22. doi:10.1145/3650041DOI
- [18]Ivan Sekulić, Lili Lu, Navdeep Singh Bedi, and Fabio Crestani. 2024. Simulating Conversational Search Users with Parameterized Behavior. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP). ACM, 72–81. doi:10.1145/3673791.3698425DOI
- [19]Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde, Sameer Singh, and Seraphina Goldfarb-Tarrant. 2026. Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations. CoRR abs/2601.17087 (2026). arXiv:2601.17087 doi:10.48550/ARXIV.2601.17087
- [20]Tianlu Wang, Yilun Kuiklov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024. Self-Taught Evaluators. CoRR abs/2408.02666 (2024). arXiv:2408.02666 doi:10.48550/ARXIV.2408.02666
- [21]Zhenduo Wang, Zhichao Xu, Qingyao Ai, and Vivek Srikumar. 2024. An In-depth Investigation of User Response Simulation for Conversational Search. In Proceedings of the ACM Web Conference 2024 (WWW). ACM, 1407–1418. doi:10.1145/3589334.3645447DOI
- [22]Rong Wu, Xiaoman Wang, Jianbiao Mei, Pinlong Cai, Daocheng Fu, Cheng Yang, Licheng Wen, Xuemeng Yang, Yufan Shen, Yuxin Wang, and Botian Shi. 2025. EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle. CoRR abs/2510.16079 (2025). arXiv:2510.16079 doi:10.48550/ARXIV.2510.16079DOI
- [23]Shirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang, Joy He-Yueya, Tharindu Cyril Weerasooriya, Wei Wei, Diyl Yang, Jure Leskovec, and James Zou. 2026. HumanLM: Simulating Users with State Alignment Beats Response Imitation. CoRR abs/2603.03303 (2026). arXiv:2603.03303 doi:10.48550/ARXIV.2603.03303
- [24]Weize Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-Rewarding Language Models. CoRR abs/2401.10020 (2024). arXiv:2401.10020 doi:10.48550/ARXIV.2401.10020DOI
- [25]Saber Zerhoudi. 2026. SimEval-IR: A Unified Toolkit and Benchmark Suite for Evaluating User Simulators and Search Sessions. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, 3537–3543. doi:10.1145/3805712.3808635DOI
- [26]Saber Zerhoudi and Michael Granitzer. 2025. User Simulators: Towards a Hybrid User Search Simulation. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM). ACM, 6570–6574. doi:10.1145/3746252.3761640DOI
- [27]Saber Zerhoudi, Michael Granitzer, Christin Seifert, and Jörg Schlötterer. 2022. Evaluating Simulated User Interaction and Search Behaviour. In Advances in Information Retrieval – 44th European Conference on IR Research (ECIR). Springer, 240–247. doi:10.1007/978-3-030-99739-7_28DOI
- [28]Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Yankai Lin, and Jiaxin Mao. 2024. USimAgent: Large Language Models for Simulating Search Users. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, 2687–2692. doi:10.1145/3626772.3657963DOI
- [29]Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Zixuan Yang, and Jiaxin Mao. 2025. Exploring Human-Like Thinking in Search Simulations with Large Language Models. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, 2669–2673. doi:10.1145/3726302.3730193DOI
- [30]Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. 2025. Absolute Zero: Reinforced Self-play Reasoning with Zero Data. CoRR abs/2505.03335 (2025). arXiv:2505.03335 doi:10.48550/ARXIV.2505.03335 Also in NeurIPS 2025.arxiv.org/abs/2505.03335
- [31]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhou, Zhang Hao, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS). arXiv:2310.05685
- [32]Xuhui Zhou, Weiwei Sun, Qianmu Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Sherry Tongshuang Wu, and Maarten Sap. 2026. Mind the Sim2Real Gap in User Simulation for Agentic Tasks. CoRR abs/2603.11245 (2026). arXiv:2603.11245 doi:10.48550/ARXIV.2603.11245DOI
- [33]Ziyi Zhu, Olivier Tielemann, Caitlin A. Stamatis, Luka Smyth, Thomas D. Hull, Daniel R. Cahn, Jinghong Chen, and Matteo Malgaroli. 2025. DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulation. arXiv:2512.20773arxiv.org/abs/2512.20773
- [34]Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, and Pulkit Agrawal. 2025. Self-Adapting Language Models. CoRR abs/2506.10943 (2025). arXiv:2506.10943 doi:10.48550/ARXIV.2506.10943arxiv.org/abs/2506.10943

