betaFeedback

WorksComputer Science

Many Solutions, One Answer: Search Order Shapes the Default Solutions of Language Models

Abstract

When a constraint puzzle has several correct answers, language models tend to give the same one.We study four open-weight models from three families on symbolic puzzles and multi-solution ZebraLogic variants with exactly known solution sets.Repeated samples concentrate on one default solution far beyond uniform sampling, and different models often share it.Yet models usually produce even rarely sampled solutions when added clues uniquely identify them.On held-out symbolic puzzles, we compare visiting entities in name order with visiting those with the fewest initial candidates first. Both searches try values in the stated order. Where their pre-specified predictions differ, the latter matches a larger share of responses in all four models, by 26 to 40 percentage points.Reversing the stated value order or specifying a search order shifts responses towards the solution that the new order favours, on both testbeds. Randomising the presentation or supplying a random search order raises breadth (the effective share of solutions in use) by 26 to 42 percentage points on the symbolic task and 11 to 26 on ZebraLogic. The largest observed validity losses are 3 and 4 percentage points, respectively. Randomised presentation broadens answers more than a random-choice instruction in each of the three main models.

Introduction

Many reasoning problems have more than one correct answer: a schedule can satisfy every constraint in several ways, and a puzzle can admit several completions. Which one a language model (LM) returns matters beyond a single response: many people put the same question to the same few models, so a model that always returns one of the valid answers narrows what all of them see, and a user asking for several feasible schedules gains an option only when a different schedule appears. Yet LMs tend to return the same answer, and different models often return the same one [12], a homogeneity shaped by training and decoding choices [11, 13, 23]. Such a narrow output has two readings: the alternatives may be lost, or the model may prefer one answer while it can still produce the others, much as models encode factual answers they rarely generate [6]. Separating them, and finding what decides the preferred answer, shows whether models explore the space of valid answers or converge on one, and whether convergence can be undone without losing validity.

Open-ended generation makes this hard to test, because its set of correct answers is unknown. We therefore study constraint puzzles whose solution sets are known exactly (Sec. 3.1): a controlled symbolic task that assigns five entities to five values under exclusion constraints leaving two, four or six solutions, and multi-solution variants of ZebraLogic [18]. Every valid output maps to one solution and every solution can be targeted, so we separate lost from preferred alternatives by cued recoverability: whether a model produces a particular solution when added facts leave only that solution. Because these facts also simplify the problem, success shows that an alternative is reachable with that information, not that the model weighed it on the original prompt. We test four open-weight models from three families, Qwen3-1.7B (Yang et al., 2025), Olmo-3.1-32B-Instruct (Team Olmo, 2025) and two sizes of gpt-oss [22], as well as three training stages of Olmo 3 7B Think.

Repeated samples concentrate on one default solution far more than uniform sampling over the valid solutions would, and different models frequently share the default (Sec. 3). Yet a model cued towards a rarely produced solution usually gives it, and OLMo does so even after an earlier answer kept in the conversation, greedy or sampled, has made it repeat that answer; the concentration thus does not reflect an inability to produce the alternatives. Which solution becomes the default is predicted by an ordered-search explanation (§ 4). In the problem of Figure 2A, a search that fills the entities in name order and one that first fills the entity with the fewest candidate values both start with E1E_1 and E2E_2; the first then fills E3E_3, the second E4E_4, and they end in different valid assignments. The models behave as if they followed the second rule, tried values in their stated order and reported the first complete solution. We call this rule MRV-like, after the minimum-remaining-values (MRV) heuristic of constraint solvers ([8]); it describes outputs, not internal computation. On held-out problems, outputs match this rule more often than name order and shift as predicted when we reverse the value order or specify a search order.

Figure 2. Testing the ordered-search explanation. (A) Example: excluded cells grey; columns: remaining candidates and each procedure’s visit order; markers: first solutions, outlined if different. (B) Same requests: share matching the MRV-like minus the name-order prediction; open: gpt-oss-120b. (C, D) Change in a predicted solution’s share when the prompt reverses (C) or specifies (D) an order. Percentage points; 95% intervals.

Varying the order across requests broadens the answers (§ 5). On the Symbolic task, randomising the presentation or supplying a random search order usually keeps validity within the pre-specified margin. Randomised presentation broadens answers more than a random-choice instruction or a modest temperature increase, which lowers validity in some models. On ZebraLogic, the three main models follow a specified search order in a re-run with the answer reader fixed in advance. Both randomisations broaden answers. Randomised presentation meets the validity criterion in three of four models and random search order in two of three. Qwen3-1.7B is the exception in both.

Recent work compares whole solution distributions ((Wang et al., 2026; d’Aliberti et al., 2026)). Here we predict the favoured assignment before generation, test the prediction by changing value, listing and search orders, and cue each alternative. On ZebraLogic we test which predictions transfer and whether changing the prompt alone broadens answers.

Related Work

Output diversity and homogenisation. Sampling choices strongly affect the variety of generated text ([11]), preference tuning reduces output diversity relative to supervised fine-tuning ([13]), and writing with a model can homogenise content ([23]). Different models converge on similar open-ended responses ([12]), although the strength of this evidence is debated ((Schaeffer et al., 2026)), and aligned models are less random than base models ((West and Potts, 2025)). Models are also less diverse than human writers on prompts without a known answer set ((Zhang et al., 2025)). With known answer spaces, Renda et al. [26] score LM samples against target distributions, and on puzzles with three to eight solutions, model solution distributions are less diverse than human ones, more alike across models, and shifted by solution-preserving changes such as reflecting the board ((Wang et al., 2026)). We add a prediction of which assignment each model favours, fixed before generation and tested by changing orders, and a cue for every non-preferred solution. In the reasoning tasks of Yue et al. (2025), reinforcement learning with verifiable rewards (RLVR) improves single-sample accuracy, while base models solve more problems at large sampling budgets. On ModeBench, RLVR concentrates probability on fewer correct modes even when accuracy holds or improves. Its task-specific verifiers can count different routes to the same answer as distinct modes ((d’Aliberti et al., 2026)). Our units are distinct complete assignments. Such collapse can be countered in training, for example by teaching a model to give several answers with confidences ([25]), whereas we change only the prompt. Instruction-tuned models asked for random outputs produce skewed distributions, which fine-tuning can make more diffuse ((Zhang et al., 2024)), and verbalising a distribution over several responses can recover some diversity ((Zhang et al., 2026)); we test only one instruction to choose uniformly at random, so its failure here does not show that diversity prompting fails in general. On multiple-choice problem solving, temperatures from 0 to 1 did not change accuracy significantly ([27]). Unlike semantic-equivalence measures of uncertainty ([14]), exact solution sets also separate variety from validity.

Order and presentation effects. LMs are sensitive to the order of few-shot examples ([20]), multiple-choice options ([24] (Zheng et al. 2024)), premises ([1]) and information in long contexts ([19]). These effects are usually treated as robustness failures to be averaged away, or, for premises, predicted to favour the order of a proof ([1]). We predict instead which valid answer a change of order will favour. The ordered-search explanation specifies which order should matter (that of the stated values), which should matter less (that of the constraints) and how a change should move the answer, so that randomising the order becomes a way to control which answers are produced.

Search in language-model reasoning. Prompting and training methods make search explicit, by expanding and evaluating several partial solutions (Yao et al., 2023), sampling several reasoning paths (Wang et al., 2023), or training on serialised search traces [4, 17]; these methods generate and compare reasoning paths, some on tasks with many acceptable outputs, whereas we measure how often each valid final assignment is produced. Constraint-satisfaction solvers have long ordered variables by how few options remain [8, 28]. We use such procedures as behavioural predictors of the reported solution, not as claims about internal computation, which written reasoning need not reflect (Turpin et al., 2023) and which may be partly latent [7, 5].

Commitment and revision. LMs can struggle to revise conclusions when new information arrives (Wilie et al., 2024), become anchored to early answers in multi-turn conversations [16] and commit prematurely as agents [21]; such findings fit both a model that can no longer use an alternative and one that merely prefers its earlier answer, and our cueing and history experiments separate the two.

Concentrated outputs, recoverable alternatives

Setting and measurements

In the Symbolic task, five entities E1,…,E5E_1,\ldots,E_5 each take a different value from V1,…,V5V_1,\ldots,V_5, and a list of exclusion facts (“E1E_1 is not in V2V_2”) leaves exactly BB valid assignments (Figure 2A; Appendix § 8 gives a complete prompt). We use 72 problems, 24 for each of B∈{2,4,6}B \in\{2,4,6\}: one from each of 24 structural classes at B=4B = 4 and 6 (of 27 and 66), and relabelled instances of three classes at B=2B = 2. Problems can be shown relabelled or reordered, and answers are mapped back to a common solution set. ZebraLogic variants pose the same questions in natural language: from 48 logic grid puzzles [18] of four sizes from 3×33 \times3 to 4×44 \times4, we removed clues until two, three or four solutions remained (24, 8 and 16 variants).

In free generation, the prompt states that any solution satisfying the facts is acceptable, and we analyse the first 32 executed requests per problem and condition. In cued generation, added facts leave only a target solution: on the Symbolic task the smallest set of assignments, outside the queried entity, that does so (one assignment for 197 of the 288 targets, two or three for the rest), on ZebraLogic one added clue. Success requires producing the target (8 requests per target, 4 for the training stages; Appendix § 9 lists two exceptions). An output is valid if it states one complete assignment that satisfies every constraint, and validity is the percentage of all requests with a valid output, so invalid, ambiguous and truncated responses count against it; the separately stated answer to the Symbolic question is checked only for cued success. Let q(w)q(w) be the share of valid outputs that give solution ww among the BB solutions of a problem. The modal share max⁡wq(w)\max_w q(w) is compared with the share the most frequent solution would reach if the same number of valid outputs were drawn uniformly from the BB solutions; for 32 draws with B=2B = 2 this is about 57%, not 50%. Breadth is the effective number of solutions as a share of all BB solutions,

N~eff=1B∑wq(w)2,(1)\widetilde{N}_{\mathrm{eff}} = \frac{1}{B\sum_w q(w)^2}, \tag*{(1)}

which is 100% when valid outputs are spread evenly over all solutions and 100/B%100/B\% when they all coincide [9]. Because breadth is computed among valid outputs, we report it with validity whenever we compare interventions. We report shares and breadth as percentages and differences in percentage points, or points for short.

The main models are Qwen3-1.7B with thinking enabled, Olmo-3.1-32B-Instruct (OLMo below) and gpt-oss-20b; gpt-oss-120b tests a larger model, and the SFT, DPO and R LVR checkpoints of Olmo 3 7B Think test training stages. All use their recommended sampling settings except OLMo, sampled at temperature 1.0 and top-pp 1.0 (Appendix § 9). Intervals are 95% bootstrap intervals over clusters of related problems [3]: structural classes at B=4B = 4 and 6 and single problems at B=2B = 2 on the Symbolic task, and source puzzles on ZebraLogic (Appendix § 10). Before each confirmatory run, we fixed the analysis plan and model predictions and committed a record. of both to version control. We correct confirmatory tests within each model (Holm, [10]) and use pre-specified margins to test whether a change is negligible (Schuirmann, 1987). The main ZebraLogic comparisons use a re-run with the pre-specified order instruction and an answer reader fixed before generation. Results for declared value order, cueing, higher temperature and preference for the source puzzle’s solution come from the original runs (Appendix § 13).

Concentration within and across models

We call the most frequent valid solution in 32 requests the model’s default (Figure 1A). On the Symbolic task, the default accounts for 7878–88%88\% of valid outputs at B=2B = 2 and 5858–72%72\% at B=6B = 6. Under uniform sampling the corresponding shares are 57%57\% and 26%26\%. The 95%95\% interval for the excess lies above zero for every model and BB. The excess is larger for six-solution problems (3131–4545 points) than for two-solution problems (2121–3131 points). Each BB uses different problems, and the increase is not monotonic in every model. On ZebraLogic the default takes 6565 to 78%78\% of valid outputs at B=4B = 4, against 35%35\% under uniform sampling, and its excess over all 48 puzzles is 2424 to 3333 points, with intervals above zero.

Figure 1. Concentrated defaults and recoverable alternatives. (A) Modal share of valid outputs above uniform sampling (57%57\% at B=2B = 2), by BB; 95%95\% intervals (ZebraLogic: all puzzles). (B) Shared defaults across model pairs (filled) and the uniform-choice level (open). (C) Free frequency (solid) and cued success (dashed) by rank: name order at B=6B = 6 (Symbolic), free frequency at B=4B = 4 (ZebraLogic, original runs).

Pairs of the three main models share a default on 6363–81%81\% of Symbolic problems and 6666–77%77\% of ZebraLogic puzzles, counting ties fractionally (Figure 1B). If one model in each pair chose uniformly among each puzzle’s valid solutions, agreement would be about 30%30\% and 39%39\%, respectively. gpt-oss-120b shares its default with each of the other three models on 7676–82%82\% of Symbolic problems and 7272–95%95\% of ZebraLogic puzzles, against the same chance levels, and its agreement with gpt-oss-20b on the Symbolic task (82%82\%) also passed a pre-specified test.

Changing the presentation can also move a pair’s defaults to the same new solution. Relabelling values to place the original name-first solution as late as possible in the new name order does so on 6464–75%75\% of problems for the three main pairs (chance 2222–27%27\%). Reversing the order of the entities’ groups of facts does so on 1313–17%17\% (chance 55–6%6\%). For every pair, the 95%95\% intervals for both excesses lie above zero (Appendix § 14). Across these three model families, value labels have a stronger effect on the shared default than the order of the entities’ facts (§ 4).

Low frequency does not imply low recoverability

Figure 1C compares how often a model gives each solution freely with how often it gives that solution when cued towards it, and Appendix Figure A1 shows every BB. Symbolic solutions are ranked in name order, so rank 1 is the first solution of a name-order search, not necessarily the default; ZebraLogic solutions are ranked by observed frequency, so their free curve falls by construction and only its comparison with cued success is informative. On the Symbolic task, mean free-generation frequency falls steeply with rank, to 44–10%10\% at the last rank for B=6B = 6, whereas mean cued success stays between 95%95\% and 100%100\% at every rank and BB; these are averages over problems, not guarantees. for every target. For OLMo and gpt-oss-20b, a pre-specified equivalence test confirms that at every BB the 90% interval for the slope of cued success lies within ±5 points per step in name-order rank, although this slope test does not compare default and non-default targets directly. On ZebraLogic that direct comparison was pre-specified: cued success is lower overall (77–100%), but non-default solutions are cued about as successfully as the default (−2.0, −0.3 and +0.7 points for the three models, all within the ±10-point margin). A rarely chosen solution is therefore not out of reach: facts that identify it are enough for the model to produce it. This holds even when the conversation pushes the model towards one answer: keeping an earlier answer in the conversation raises OLMo’s free-generation share of that answer by 15 points when it was its greedy answer and by 33 points when it was a sampled alternative, yet cued success for the remaining solutions changes by less than 1 point (90% intervals inside ±2 points; Appendix § 12.§ 3). Repetition thus strengthens a preference without eroding the alternatives. Because a cue adds information, this shows recoverability with that information, not that the model weighed these solutions during free generation.

Ordered search predicts the default solution

If rarely produced solutions remain recoverable, something other than ability decides which one becomes the default. We test whether the model behaves as if it searched in a fixed order and reported the first complete solution it found.

Predicting the first solution

A depth-first search fills one entity at a time, tries the entity’s remaining values in their stated order, backs up when no value fits, and stops at the first complete solution. Such searches differ mainly in the order in which they visit entities. Name order visits E1,…,E5E_1,\ldots,E_5 in turn, whereas fewest candidates first visits entities in order of how many values the facts leave them, counted once before the search starts: a static form of MRV, whose dynamic form would recount after each assignment (Figure 2A shows how the two diverge). Because each exclusion fact removes one of an entity’s five values, the entity with the fewest candidates is also the one named in the most exclusion facts, so our tests cannot separate counting candidates from attending to the most-mentioned entity.

The hypothesis came from earlier free-generation data on the 72 main problems: on the 51 and 37 presentations where static and dynamic MRV, respectively, disagreed with name order, the share of outputs matching the MRV prediction exceeded the share matching name order by 30–44 points. We used these data to form the hypothesis, to select each model’s primary procedure among eight candidates (static MRV in every case) and to fit the procedure mixture described below (Appendix § 11). We then drew 48 new problem instances (24 at B=4B = 4 and 24 at B=6B = 6; 20 of them from structural classes not used before), showed each in four presentations, and fixed every procedure’s predicted solution before generation (Appendix § 11). The test uses the presentations on which a model’s primary procedure and name order predict different solutions (154 of 192 for Qwen3-1.7B, 149 for OLMo and gpt-oss-20b). On these, the share of all requests, invalid ones included, that match the MRV-like prediction exceeds the share that match the name-order prediction by 37, 36 and 26 points (Figure 2B; intervals exclude zero), and by 40 points for gpt-oss-120b ([33, 47]; 149 presentations). On the same presentations, 48%, 48% and 42% of outputs match the MRV-like and 11%, 12% and 16% the name-order prediction; over all 192, 49%, 49% and 44% match the MRV-like prediction and 19%, 22% and 23% the name-order prediction. In a post-hoc check across all 192 presentations, we asked whether each prediction matched the most frequent valid answer among the 32 requests, splitting credit equally across ties. The MRV-like rule matched on 64.6%, 68.8% and 63.0% of presentations for Qwen3-1.7B, OLMo and gpt-oss-20b, respectively. Name order matched on 19.3%, 22.1% and 25.0%, against 20.8% for a uniform choice. The MRV-like match rate was 73–79% on the presentations as generated and 32–55% on those designed to distinguish tie rules (Appendix § 11).

The rule thus often picks out the default on new problems, but finer distinctions between procedures were not consistent across models. Where static and dynamic MRV predict different solutions, Qwen3-1.7B’s outputs matched the static prediction more often (by 23 points, [14, 32]), whereas OLMo (8 points) and gpt-oss-20b (1 point) showed no resolved difference, and breaking ties by listing order mattered only for Qwen3-1.7B. We therefore call the default MRV-like and read the procedures as predictors of behaviour, not as the computation inside the model. Nor do they describe every answer: a mixture of the eight procedures and uniform choice gives the procedures about half to three-quarters of the probability of the free-generation answers (Appendix § 11).

Declaration order steers solution choice

An ordered search tries values in their stated order, so stating them in a different order should move the default. To separate this order from familiar label orders such as V1<V2V_1 < V_2, we replaced the value labels with arbitrary words and declared them in one order or its reverse, on the 48 problems with B=4B = 4 or 6, listing each entity’s exclusions in the declared order. Reversing the two orders together raised the share on the solution the primary procedure reaches first under the new order by 42, 49 and 43 points for the three models, averaged over both directions; here and in Figure 2, shares count invalid outputs in the denominator. A factorial design then crossed the two orders (Figure 2C), each effect averaging four paired changes in share when one order is switched and the other held fixed (Appendix § 14). Declaration order has a large effect (38–42 points) and the within-entity listing order a small but reliable one (4–7 points, all intervals above zero). This asymmetry is what the explanation predicts: a search must check every constraint wherever it is listed, but it tries values in the order in which they are declared. On ZebraLogic, reversing the declared order of each attribute’s values likewise raised the share on the correspondingly predicted solution, by 41, 33 and 36 points.

Specifying the search order

The most direct test is to tell the model which order to use. We instructed the models to consider the entities in a given order, to try each entity’s values in a given order, or both, and to keep the first choice that leads to a complete solution (Appendix § 8). The orders were chosen so that the solution they reach first differs from both the MRV-like and the name-order prediction. Relative to the unmodified prompt, every instruction moved the Symbolic outputs towards the specified solution in all three models (Figure 2D): by 39–72 points when both orders were specified, 38–83 points for the entity order alone and 13–31 points for the value order alone. Entity-only instructions shifted more responses towards their predicted assignment than value-only instructions did, in all three models.

On ZebraLogic, where the instruction fixes the house order, an attribute order within each house and a value order for each attribute, it moved the outputs towards the specified solution by 27, 38 and 54 points (all intervals above zero). In the original runs the instruction was ambiguous about the starting house. Only Qwen3-1.7B showed a clear shift towards the intended solution (Appendix § 13). Across the six model pairs, 215 re-run comparisons had both a shared default and a common search prediction. A house-by-house search predicted that shared answer in 155 of the 215 (72.1%). A uniform guess among each puzzle’s valid solutions would match in 39.8% of the same comparisons. However, searches that proceed house by house, attribute by attribute, or by the fewest remaining candidates return the same solution on 41 of the 48 unmodified puzzles. This limits how well the defaults can distinguish the three procedures. On both testbeds a stated order thus shifts which solution the models return.

Across scale, training stages and generation protocols

gpt-oss-120b shares the signatures tested at both sizes. Its Symbolic outputs are concentrated (modal share 88%, 67% and 66% at B=2B = 2, 4 and 6), it shares defaults with gpt-oss-20b (82% agreement), and it matches its own predicted procedure more often than gpt-oss-20b does (by 9 points, [6, 12]). This shows persistence at the largest size we tested, not a scaling trend. Across the training stages of Olmo 3 7B Think (Appendix Table A2), the supervised fine-tuning (SFT) checkpoint already concentrates its answers (modal share 64%) and succeeds almost always when cued (99.9%), so neither the default nor its recoverability requires preference tuning or reinforcement learning. The checkpoint after preference tuning (DPO) has a higher modal share (by 6 points, [3, 9]) and a larger MRV-like contrast (by 8 points, [2, 15]), but it also uses a different default system prompt, so this comparison changes checkpoint and prompt together and does not isolate a contribution of preference tuning. After RLVR, neither quantity shows a statistically resolved change (−0.1 and −3 points, intervals including zero).

Answering without written reasoning, in 48 tokens after a prefilled FINAL: E1=, cut validity by 57–86 points and weakened the MRV-like contrast, but it also changes the budget, the start of the answer and the supporting problems (Appendices § 12.§ 4 and § 12.§ 5).

Broadening answers by varying order

If a model tends to return the first solution it finds, varying the order across requests should broaden its answers. We compare the effects of order, a random-choice instruction and sampling temperature on breadth and validity.

Prompt and sampling conditions

Randomised presentation (prompts in Appendix § 8) shows each request with the values randomly relabelled and the entities’ facts in a random order (on ZebraLogic: random orders of each attribute’s values, the attribute lines and the clues), and maps the answers back before scoring. It changes several things at once, so it is a presentation intervention rather than an isolated change of search order; Figure 3 also shows a variant that renames the entities as well. A random search order adds the instruction of § 4.3, with entity and value orders that we draw uniformly at random for each request and write into the prompt, so the model is not asked to invent an order. To request randomness, a random-choice instruction asks the model to choose one of the valid solutions uniformly at random, and a higher temperature adds 0.25 to the sampling temperature. On the Symbolic task we also asked the model to enumerate, then choose one solution at random, which changes the solving procedure and is not a like-for-like comparison.

Figure 3. Breadth and validity under each intervention. Change from the unmodified prompt, in percentage points, on Symbolic (A) and ZebraLogic (B). Intervals are 95% for breadth and 90% for validity; shading marks the validity margins. ZebraLogic random-choice estimates are descriptive (no interval). Higher-temperature estimates on ZebraLogic come from the original runs.

For randomised presentation and the random search order we fixed a test in advance: it passes when breadth (Eq. (1)) increases and the 90% interval for the change in validity lies within ±3 points on the Symbolic task and ±5 points on ZebraLogic. The other comparisons are descriptive unless stated otherwise. Random orders need not spread the answers uniformly, because a solution that more orders reach first is still chosen more often. Before the runs we therefore forecast the breadth each intervention should reach, with different assumptions: for randomised presentation by averaging the procedure mixture of § 4.1 over the random presentations, and for the random search order by assuming that the model follows the order in the prompt (Appendix § 11).

Breadth and validity on the Symbolic task

Both order interventions broaden the answers substantially (Figure 3A). Randomised presentation raises breadth by 28–39 points in all four models while validity stays within ±3 points, so it passes the test in every model, and renaming the entities as well gives similar gains (29–41 points). A random search order raises breadth by 26–42 points and passes for Qwen3-1.7B and gpt-oss-20b; for OLMo, validity rose by 3.7 points, and its 90% interval ([2.6, 4.9]) crosses the upper margin of +3 points, so the two-sided test fails although validity improved. On the 24 six-solution problems, for example, randomised presentation raised Qwen3-1.7B’s breadth from 41.2% to 76.7%, as if its valid answers had spread from about 2.5 to 4.6 of the six solutions, while validity went from 98.4% to 97.0% (Table A3). The forecast fixed before the run, which assumes 32 valid draws, predicted gains on these problems of 34.3, 45.2 and 31.8 points for the three main models, against observed gains of 35.5, 44.5 and 36.1, although it put both breadth levels too high; averaged over the three values of BB, predicted and observed gains agree within 3 points.

TestbedModelBBUnmodified promptRandomised presentationRandom search orderRandom-choice instructionTemperature +0.25†Enumerate, then choose
SymbolicQwen3-1.7B275 / 9898 / 9995 / 9775 / 9877 / 9662 / 97
461 / 9886 / 9886 / 9860 / 9964 / 9752 / 99
641 / 9877 / 9773 / 9837 / 9941 / 9934 / 100
OLMo-3.1-32B265 / 9398 / 9197 / 9669 / 8873 / 8978 / 76
449 / 9285 / 9391 / 9551 / 9056 / 9176 / 71
634 / 9178 / 9285 / 9635 / 9141 / 9066 / 67
gpt-oss-20b273 / 10096 / 10097 / 9974 / 10079 / 10093 / 98
461 / 10087 / 10089 / 9962 / 10068 / 10081 / 98
645 / 10081 / 10084 / 9946 / 10054 / 10070 / 97
ZebraLogicQwen3-1.7B263 / 8689 / 7876 / 8462 / 8661 / 82–
362 / 8972 / 9164 / 9371 / 9166 / 88–
441 / 8975 / 8754 / 9442 / 9142 / 91–
OLMo-3.1-32B276 / 8288 / 8194 / 8082 / 7981 / 74–
362 / 8059 / 7872 / 8664 / 7755 / 71–
452 / 9074 / 8880 / 8457 / 8157 / 78–
gpt-oss-20b276 / 9691 / 9695 / 9873 / 9584 / 88–
373 / 9677 / 9885 / 9780 / 9671 / 90–
449 / 9872 / 9885 / 9959 / 9662 / 94–

Table A3. † On ZebraLogic, the higher-temperature column uses the original runs. The other columns use the re-run. Table A3: Absolute breadth and validity by condition. Each cell gives breadth (Eq. (1)) / validity, both in percent, over the first 32 requests per problem. Means cover 24 Symbolic problems per BB and 24, 8 and 16 ZebraLogic variants at BB = 2, 3 and 4. Dashes mark a condition that was not run.

Asking for randomness does much less. The random-choice instruction changes breadth by −2 to +2 points (all intervals include zero), in line with the skewed outputs of instruction-tuned models asked for random choices (Zhang et al., 2024), and a pre-specified test confirms that randomised presentation broadens the answers more than this instruction, by 29.5, 35.7 and 27.4 points (all 72 problems; 95% intervals above 23 points; Holm-corrected). A higher temperature adds 2–9 points and lowers OLMo’s validity by 2.1 points, and a larger increase left OLMo and gpt-oss-20b with almost no valid answers (Appendix § 12.§ 6). Enumerating before choosing raises breadth for OLMo and gpt-oss-20b (by 20 and 23 points) but lowers it for Qwen3-1.7B (by 10 points), and it costs OLMo 19.9 points of validity. Apart from that test, these comparisons are descriptive (Appendix § 14).

This contrast fits the ordered-search explanation. Randomising the order changes the prompt itself, so even a model that gave the same answer whenever it saw a given prompt could answer differently across requests; an instruction to choose at random leaves the order unchanged, so a model that reports the first solution of its search still reaches the same one. Randomised presentation broadened the answers and met the validity criterion in all four models; the modest temperature increase also had positive breadth estimates in all three main models, but smaller ones, with descriptive intervals.

Transfer to ZebraLogic

Randomised presentation raises breadth by 26, 13 and 16 points in the three main models and by 20 points in gpt-oss-120b (Figure 3B), and it passes the pre-specified test in all but Qwen3-1.7B, whose validity falls by 4.0 points, with a 90% interval reaching −7.7 points. A pre-specified comparison shows that it broadens more than the random-choice instruction in all three main models, by 25, 8 and 13 points. The instruction itself changes breadth by +1 to +5 points, and in the original runs a higher temperature added up to 9 points while lowering validity by 8.7 points for OLMo and 6.7 points for gpt-oss-20b. A random search order raises breadth by 11, 20 and 24 points with validity changes of +1.7, −1.9 and +0.8 points and passes the test for OLMo and gpt-oss-20b; for Qwen3-1.7B the 90% interval of the validity change reaches +5.1 points, just beyond the margin.

After randomised presentation, the share of valid answers on the default exceeds the procedure mixture’s forecast by 12, 13 and 11 points in the three main models (descriptive, no interval). At the observed numbers of valid answers, breadth falls 12 to 15 points below forecast, and all nine model-by-BB intervals for observed minus forecast lie below zero. Under the random search order, observed breadth comes within 4 points of forecast for OLMo and gpt-oss-20b (Appendix § 12.7).

Since the models follow a specified search order on ZebraLogic (§ 4.3), this remaining concentration suggests that the ZebraLogic defaults depend on more than the declared orders that randomised presentation permutes. In the original runs, all three models returned the source puzzle’s unique solution within 0.6 percentage points of the share forecast under randomised presentation, with all 95% intervals for the differences including zero (Appendix § 13).

Discussion and conclusion

In our puzzles, the models frequently favoured the same answer when asked freely, yet could usually produce other valid answers when clues identified them, and on the Symbolic task the same ordered-search rule often predicted which answer each of the three main models favoured. For the debate on homogenisation, this separates two things that output diversity alone conflates: free answers converge while the alternatives stay within reach, and the convergence has a simple behavioural source shared by models from three families. It follows that measures of output diversity should be read alongside tests of what a model can still produce, and that some of the homogeneity across models may come from shared problem-solving conventions, such as filling the most constrained entity first, applied to the same presented structure, rather than from shared knowledge alone. The convergence can also be partly undone: when diverse valid answers are wanted, changing the order in which a problem is presented or searched is a lever that needs no access to model weights, and in our Symbolic tests it broadened answers more than the random-choice instruction and the temperatures we tried; on ZebraLogic, both kinds of randomisation also broadened answers, with validity within the margin in all models but Qwen3-1.7B. Their costs differ: on ZebraLogic, randomised presentation lengthened the outputs by 2–6% and a specified search order by 14–72% (mean output tokens).

The explanation is behavioural, and on natural-language puzzles its order effects transferred while part of the default remained unexplained; open-ended tasks with many valid answers, such as code or plans, are a natural next step.

Limitations

Our tasks are small constraint problems with two to six enumerable solutions, chosen so that every answer can be checked and every alternative cued; whether the same explanation holds for open-ended generation, where valid answers are many and unknown, is untested. The search procedures predict which answer a model reports, not how it computes it, and a successful cue shows only that a solution is reachable once added facts single it out. We test four open-weight models from three families and a separate Olmo 3 7B training series, without a pre-trained base checkpoint, so other families, sizes and closed models may behave differently. Finally, some intervals were computed after the main analyses and are descriptive, the ZebraLogic results for declared order, cueing and higher temperature rest on original runs read with an answer reader chosen after data collection (Appendix § 13), and at B=2B = 2 our intervals treat relabelled instances of three structural classes as separate problems, so they do not generalise across classes (Appendix § 10 reports the results on the B=4B = 4 and 66 problems alone).

Acknowledgments

The authors wish to acknowledge CSC – IT Center for Science, Finland, for computational resources.

Tasks and conditions

Symbolic problems. We enumerated sets of B∈{2,…,6}B \in\{2,\ldots,6\} assignments of five entities to five values, retained those whose allowed entity–value pairs yield exactly those assignments, and grouped the resulting problems into structural classes that are equivalent under renaming entities and values. For B=2B=2 the main set uses eight differently labelled instances of each of three classes in which the two solutions differ in at least three entities. For B=4B=4 and B=6B=6 it uses one instance from each of 24 classes, out of the 27 and 66 classes in the census. Problems with B=3B=3 and B=5B=5 were used only for admission checks before main runs. The exclusion facts of a problem exclude exactly the entity–value pairs that occur in none of its solutions, so every pair still allowed occurs in at least one solution. A problem was shown in its canonical form, with relabelled values, with its facts listed in reverse entity order, or with both changes. Every answer is mapped back to the canonical solution set before scoring. The free-generation prompt for the first B=2B=2 problem reads in full:

Each of E1, E2, E3, E4, E5 occupies a
different value from V1, V2, V3, V4,
V5.
E1 is not in V2. E1 is not in V4. E1 is
not in V5. E2 is not in V2. E2 is not
in V3. E2 is not in V4. E3 is not in V1.
E3 is not in V3. E3 is not in V4. E4 is
not in V1. E4 is not in V2. E4 is not
in V3. E4 is not in V5. E5 is not in V1.
E5 is not in V4. E5 is not in V5.
No additional fact is supplied.
Any complete world satisfying the
original facts is acceptable.
Question: What value does E2 have?
Reconstruct the complete world.
You may explain your reasoning before
answering. Finish with one clearly
marked final answer in this form:
FINAL: E1=<value>, E2=<value>,
E3=<value>, E4=<value>, E5=<value>;
consequence=<value>
Give the complete assignment and the
answer to the question. Do not give
multiple alternative final answers.

(Each fact is on its own line in the prompt.) The two solutions are (E1,…,E5)=(V1,V5,V2,V4,V3)(E_1,\ldots,E_5)=(V_1,V_5,V_2,V_4,V_3) and (V3,V1,V5,V4,V2)(V_3,V_1,V_5,V_4,V_2). A cued prompt replaces the two lines “No additional fact is supplied.” and “Any complete world …” with one line, “Real additional fact: E1 is in V1.” for the first solution or “Real additional fact: E1 is in V3.” for the second, and is otherwise identical. Here the fact fixes one entity, and the model must still derive the other four. In general a cue is the smallest set of assignments, none of them for the queried entity, that leaves only the target, and several assignments share the line (“Real additional fact: E1 is in V3. E4 is in V1.”). Of the 288 Symbolic targets, 197 are cued by one assignment, 84 by two and 7 by three; at B=6B=6, half of the 144 targets need more than one. Each ZebraLogic target is cued by one added clue.

The consequence field asks for the answer to the question, the value of the queried entity (here E2E_2). In free generation, validity depends only on the complete assignment: a stated consequence that disagrees with the assignment does not make the output invalid. Cued success additionally requires that the consequence, if stated, equals the target’s value for the queried entity.

ZebraLogic variants. From 48 source puzzles of ZebraLogic (Lin et al., [18]) (the grid_mode test split of allenai/ZebraLogicBench at revision 2f94a44), we removed clues until B=2B=2, 3 or 4 solutions remained (24, 8 and 16 variants; 12 for each of the sizes 3×33 \times3, 3×43 \times4, 4×34 \times3 and 4×44 \times4). Pilot puzzles were disjoint from these. Each of the 136 solutions has a cue: one additional clue that leaves only that solution. An audit found that an earlier rendering of these prompts still displayed removed clues. All ZebraLogic data reported here were therefore generated after the repair. Every displayed prompt was re-parsed and solved, and its solution set was required to match the intended one.

The free-generation prompt for the 3×33 \times3, B=2B=2 variant of source puzzle lgp-test-3x3-5, from which one clue was removed, reads:

There are 3 houses, numbered 1 to 3
from left to right, as seen from across
the street. Each house is occupied by
a different person. Each house has
a unique attribute for each of the
following characteristics:
- Each person has a unique name:
‘Eric’, ‘Peter’, ‘Arnold’
- The people are of nationalities:
‘dane’, ‘brit’, ‘swede’
- People have unique favorite sports:
‘basketball’, ‘tennis’, ‘soccer’
## Clues:
1. Eric is in the second house.
2. The Dane is somewhere to the right of
Arnold.
3. The British person is somewhere to
the right of the person who loves tennis.
4. The British person is not in the third
house.
Some clues have been removed, so more

than one arrangement may satisfy every clue. Any arrangement that satisfies every clue is acceptable.

Finish with one clearly marked final grid line in this form:

FINAL: House 1: <Attribute>=<value>, . . . ; House 2: . . .

The two solutions agree on Arnold (Swede, tennis) in house 1, Eric (Brit) in house 2 and Peter (Dane) in house 3, and differ only in whether Eric or Peter loves soccer. Each cued prompt adds one line before the answer instruction, for example “Real additional clue: The person who loves soccer is in the third house.” (or “. . . basketball . . . ” for the other solution). This is the prompt used for this puzzle in the original runs.

Intervention prompts. The examples quote the prompts shown to the models. For the ZebraLogic order instruction we give the re-run wording. Randomised presentation applies a random value relabelling and a random order in which the entities’ facts are listed, drawn independently for each request; on ZebraLogic it independently permutes each attribute’s declared value list, the order of the attribute lines and the order of the clues (renumbered), and leaves house numbering and clue wording unchanged. Answers are mapped back before scoring. The Symbolic random-choice instruction adds, after “Any complete world satisfying the original facts is acceptable.”, the line “If several complete worlds satisfy the facts, choose one of them uniformly at random.”; on ZebraLogic it is “If several arrangements satisfy every clue, choose one of them uniformly at random.” Enumerate-then-choose replaces the question with “Enumerate every complete world satisfying the original facts, one complete world per line. Then choose one admissible world uniformly at random and give it as the single FINAL answer.” The Symbolic specified-order instruction is inserted at the same place, for example:

Solve the problem by considering the
entities in this order: E3, E1, E4, E2,
E5.
For each entity, try its possible values
in this order: V2, V1, V3, V5, V4.
Keep the first choice that leads to a
complete world, and give that world.

The entity-only and value-only variants keep the first or second sentence and the last one. For the fixed orders, the entity and value orders are the first permutations in a fixed hash order whose first solution differs from both the MRV-like and the name-order prediction.

Models and decoding

For the random search order, we draw the entity order and the value order independently and uniformly from all 120×120120 \times120 pairs for each request and freeze them before generation. On ZebraLogic the order instruction follows the sentence explaining that several arrangements are acceptable. One re-run prompt reads as follows.

Fill the grid house by house, from
House 1 to House 3. Within each
house, decide the attributes in this
order: FavoriteSport, Nationality, Name.
For each attribute, try its values
in this order: FavoriteSport: soccer,
basketball, tennis; Nationality: dane,
swede, brit; Name: Arnold, Peter, Eric.
Keep the first choice that leads to an
arrangement satisfying every clue.

For the fixed order, the attribute order and each attribute’s value order are chosen by a fixed hash rule so that their first solution differs from the plain house-by-house prediction; for the random order they are drawn for each request. The original runs said “from House 3.” (“from House 4.” for four houses) in place of “from House 1 to House 3.” (Appendix F).

Models were served with vLLM 0.19.1 [15] from the Hugging Face revisions in Table A1, using each model’s native chat template. We added no system message; templates that supply a default one keep it. The gpt-oss models use their native reasoning format at medium reasoning effort. The Olmo 3 7B SFT template supplies a default assistant system message, whereas the DPO and RLVR templates share a different one that describes function calling with an empty function list; the user text, sampling settings and thinking prefix are identical.

ComponentQwen3 1.7BOLMo 3.1-32Bgpt-oss 20b
Name order0.000.000.00
Listing order0.030.020.00
Static MRV, ties by name0.060.190.16
Static MRV, ties by listing0.270.270.17
Dynamic MRV, ties by name0.110.170.10
Dynamic MRV, ties by listing0.040.030.02
Query first0.010.040.05
Value-major0.000.000.02
Uniform0.470.280.48
Procedures in total0.530.720.52

Table A1.

Allowances and retries. If a response stops at its allowance without an end-of-text token, the same request is retried once with the same random seed at the retry allowance; we keep the retry’s response, never the better-scoring of the two. Responses still truncated count as invalid. The Olmo 3 7B retry allowance is computed per request as the 65,536-token context minus the prompt length.

Sample counts. Free generation and every comparison condition: the first 32 executed requests per problem and condition (Symbolic: 72 problems, 2,304 requests per condition; ZebraLogic: 48 variants, 1,536). Cued generation: 8 requests

ModelRevisionSamplingFirst allowanceRetry allowance
Qwen3-1.7B (thinking)70d244cT=0.6T=0.6, top-p 0.95, top-k 2016,38432,768
Olmo-3.1-32B-Instructac0587eT=1.0T=1.0, top-p 1.08,19216,384
gpt-oss-20b6cee5e8T=1.0T=1.0, top-p 1.0, effort medium16,38432,768
gpt-oss-120bb5c939dT=1.0T=1.0, top-p 1.0, effort medium16,38432,768
Olmo 3 7B Think SFT6ff8575T=0.6T=0.6, top-p 0.9532,76865,536 — prompt
Olmo 3 7B Think DPO7b18bf9T=0.6T=0.6, top-p 0.9532,76865,536 — prompt
Olmo 3 7B Think RLVRd97e442T=0.6T=0.6, top-p 0.9532,76865,536 — prompt

caption: Table A1: Models, revisions (first seven characters of the Hugging Face commit), sampling settings and new-token allowances. All models except Olmo-3.1-32B-Instruct use the vendor-recommended sampling settings (0.6/0.95 for that model). The answer-only condition uses 48 new tokens without retry for every model.

:::

per target solution for the main models and gpt-oss-120b (Symbolic: 288 targets; ZebraLogic: 136 targets) and 4 per target for the Olmo 3 7B training stages. Prospective test: 48 problems ×\times 4 presentations ×\times 32 requests, 6,144 per model. Order manipulations: 32 requests per problem and condition on the 48 problems with B=4B=4 or 66 (47 for the instruction that fixes both orders), and on all 72 problems for the random search order. Exceptions: Qwen3-1.7B’s Symbolic rank profiles (Figure 1C and Figure A1) come from an earlier run with 48 free requests per problem and 16 cued requests per target, and OLMo’s history experiment uses 32 free and 16 cued follow-up requests per problem and target (Appendix § 12.§ 3).

Figure A1. Free and cued generation for every BB. Symbolic (A; B=2,4,6B = 2, 4, 6) and ZebraLogic (B; B=2,3,4B = 2, 3, 4). Share of requests giving each solution in free generation (solid, filled) and in cued generation towards it (dashed, open), by solution rank (Symbolic: name order; ZebraLogic: free-generation frequency), over all requests including invalid ones. Profiles are descriptive (no intervals).

Statistical analysis

Intervals are percentile intervals from 2,000 bootstrap resamples ([3]) of clusters of related problems. On the Symbolic task, a cluster is a structural class (problems equivalent under renaming entities and values) at B=4B=4 and B=6B=6, where each class occurs once in a problem set. At B=2B=2, where the 24 problems are eight relabelled instances of each of three classes, each problem is its own cluster. The cluster counts are therefore: 72 (24 classes at each of B=4B=4 and 6, plus 24 problems at B=2B=2) for analyses on all 72 problems, including concentration, cross-model agreement, cued success, the interventions and the training stages; 48 classes for analyses on the B=4B=4 and 6 problems only, namely the declared-order and listing-order effects and the specified orders (47 for both orders together); 48 classes for the prospective test, with the four presentations of a problem resampled together; 46 and 43 problems for the history comparison; and 48 source puzzles, one variant each, on ZebraLogic. Intervals at B=4B=4 and 6 therefore generalise over structural classes like those sampled, whereas at B=2B=2 they describe relabelled instances of only three classes and do not generalise across classes; three clusters would be too few for a bootstrap. On the B=4B=4 and 6 problems alone, every Symbolic estimate whose 95% interval excludes zero on all 72 problems keeps its direction and an interval excluding zero, with two exceptions among the descriptive comparisons: OLMo’s validity drops under the random-choice instruction and the higher temperature (1.7 and 2.1 points) become 0.8 and 1.3 points, with intervals including zero. Conversely, the changes in cued success between the Olmo 3 training stages, both below one point, have intervals excluding zero on these problems.

Equivalence is tested with two one-sided tests ((Schuirmann, 1987)), reported as 90% intervals. The margins, all fixed in advance, are: ±3\pm3 points for Symbolic validity; ±5\pm5 points for ZebraLogic validity; ±10\pm10 points for cued success of non-default versus default solutions; and ±5\pm5 points per rank step for the slope of cued success. The prompts for the confirmatory tests were fixed before generation. Decisions are corrected with the Holm procedure ([10]) within each model’s family of pre-specified tests for each study. We also report descriptive summaries without multiplicity correction. These include the by-BB comparisons in Figure 1A, the ZebraLogic agreement and chance levels in Figure 1B, and the intervals shown for the random-choice instruction, higher temperature and enumerate-then-choose in Figure 3. The ZebraLogic concentration test over all 48 puzzles was pre-specified. We calculated the modal matches in § 4.1 and the enumeration-rank shares in Appendix § 12.§ 5 from retained responses after the main results were finalised. Appendices § 11 and § 12.§ 5 give the counts for both summaries. Values that are undefined, for example breadth when a problem has no valid output, are left undefined rather than set to zero.

Search procedures

Seven procedures choose an entity and then try its remaining values in the declared order, in a depth-first search that returns the first complete solution. They differ in which entity they fill next:

  • name order, E1E_1 to E5E_5;

  • listing order, the order in which entities’ facts are listed;

  • static MRV, fewest initial candidates first, with ties broken by name or by listing order;

  • dynamic MRV, fewest remaining candidates given the assignments so far, with ties broken by name or by listing order;

  • query first, the queried entity and then name order.

The eighth, a value-major procedure, instead takes values in declared order and gives each to the first entity, by name, that can take it. Each model’s primary procedure was selected on earlier data and fixed before the prospective test. It is static MRV with ties broken by listing order for Qwen3-1.7B, and static MRV with ties broken by name for OLMo, gpt-oss-20b and gpt-oss-120b.

Prospective test. The 48 problems were drawn by a fixed rule that took structural classes and their instances in hash order and kept, for each class, the first instance not used in any earlier run for which all four presentations exist (24 classes at B=4B = 4, all those with such an instance, and the first 24 at B=6B = 6). We amended the selection rule before generation because the initial rule yielded only 13 four-solution instances from nine classes. Twenty-eight of these classes also occur among the 72 main problems (22 at B=4B = 4, 6 at B=6B = 6), so the test is held out at the level of problem instances, and at the level of classes for the other twenty. Each instance is shown as generated (facts in name order) and with two value relabellings, one that changes the MRV prediction but not name order’s and one the reverse; the fourth presentation reverses the listing order of all 48 instances, and also relabels the values of 11 of them, so that breaking ties by name and by listing order give different predictions. We fixed all eight procedures’ predictions for the 192 presentations before generation. The test uses the presentations on which a model’s primary procedure and name order predict different solutions; because the primary procedures differ in tie-breaking, this gives 154 presentations for Qwen3-1.7B and 149 for OLMo and gpt-oss-20b. gpt-oss-120b’s primary procedure was selected on its own earlier free-generation data, before its prospective outcomes were read. For the post-hoc modal check in § 4.1, a predicted solution earned 1/t1/t credit when it was one of tt tied modes, and zero otherwise. Out of 192 presentations the MRV-like rule earned 124, 132 and 121 matches, and name order earned 37, 42.5 and 48 (Qwen3-1.7B, OLMo and gpt-oss-20b). Because 96 presentations had four solutions and 96 had six, a uniform guess would match on average (1/4+1/6)/2≈20.8%(1/4 + 1/6)/2 \approx20.8\% of presentations. When the main models are asked instead to list every solution, their first listed solution matches the MRV-first solution in 70–80% of lists and the name-order-first solution in 57–64%.

Procedure mixture. For a presentation xx of a problem with solution set SS, procedure kk reaches a first solution sk(x)∈Ss_k(x) \in S. The mixture predicts solution ww with probability

p(w∣x)=∑k=18πk1sk(x)=w+πU/∣S∣,p(w \mid x) = \sum_{k=1}^{8} \pi_k \mathbf{1}_{s_k(x)=w} + \pi_U/|S|,

with weights πk,πU≥0\pi_k,\pi_U \ge0 summing to one and shared across problems. For each model, the weights were fitted by expectation–maximisation to maximise the likelihood of the valid free-generation answers to the 72 main problems in four presentations (canonical, relabelled values, reversed listing, both), with up to 48 requests per presentation for Qwen3-1.7B and 32 for the other models. In leave-one-cluster-out evaluation on those data (clusters as in Appendix § 10), the mixture’s log-likelihood per valid answer is −1.09-1.09, −0.92-0.92 and −1.11-1.11 for the three main models, against −1.28-1.28, −1.22-1.22 and −1.26-1.26 for name order plus uniform choice and −1.29-1.29 for uniform choice alone. On the new problems of the prospective test, with the weights frozen, it is −1.36-1.36, −1.20-1.20 and −1.40-1.40, against −1.62-1.62, −1.65-1.65 and −1.61-1.61 for name order plus uniform choice. The weights are frozen after fitting. To predict a new condition, each procedure is rerun on the presentation actually shown, so relabelled values, a different declared or listing order change sk(x)s_k(x) and hence p(w∣x)p(w \mid x) without refitting; answers are mapped back to the canonical solutions. For randomised presentation, the forecast averages p(w∣x)p(w \mid x) over the random presentations, by exact enumeration on the Symbolic task and over 4,000 sampled presentations per variant on ZebraLogic, and the forecast breadth is the expected normalised effective number of 32 valid draws from that distribution. For the random search order, the forecast instead assumes that the model follows the specified order, enumerating all 120×120120 \times120 orders. The ZebraLogic forecast uses ZebraLogic analogues of the procedures and a mixture fitted in the same way on the free-generation ZebraLogic answers collected before the intervention runs. The weights describe the distribution of reported solutions; they do not measure how often a model runs each procedure.

The fitted weights on the Symbolic task are:

Additional results

Recoverability at every number of solutions

Figure A1 repeats Figure 1C for every BB. The pattern of § 3.3 holds at each of them. On the Symbolic task, free-generation frequency at the last rank falls from 15–25% at B=2B = 2 to 4–10% at B=6B = 6, whereas cued success stays at 95–100%. On ZebraLogic, cued success ranges from 77% (OLMo, second-ranked of three solutions) to 100%, far above the free-generation frequency at every rank after the first (at most 25%).

Training stages

Table A2 gives the three Olmo 3 7B Think checkpoints of § 4.4 on the 72 Symbolic problems.

StageModal share (%)Breadth (%)Cued success (%)
SFT64.4 [60.4, 68.2]58.7 [54.2, 63.2]99.9 [99.7, 100.0]
DPO70.2 [65.9, 74.5]53.4 [48.8, 58.5]99.3 [98.7, 99.8]
RLVR70.1 [66.0, 74.3]53.4 [48.7, 58.2]99.7 [99.4, 100.0]

Table A2. Olmo 3 7B Think across training stages. Modal share, breadth (Eq. (1)) and cued success in percent, with 95% intervals, on 72 Symbolic problems per stage (cued success: 288 targets, 4 requests each). No base checkpoint was evaluated. The SFT and DPO checkpoints also differ in their default system prompt.

History

For OLMo, we kept an earlier answer in the conversation and then sampled again, on the 24 problems at each of B=2B = 2 and 4. The earlier answer was either the model’s greedy answer or a sampled alternative, the first valid answer among 16 sampled ones that differed from the greedy answer. Of the 48 problems, two had no valid greedy answer, and for three more none of the 16 sampled answers was a different valid solution, so the comparisons use 46

and 43 problems. Compared with a fresh conversation, the free-generation share of the earlier answer rose by 15.1 points (95% interval [8.7, 21.2]) for the greedy answer and by 33.0 points ([25.7, 40.3]) for the sampled alternative. Cued success for the remaining solutions changed by +0.9 points (90% interval [−0.1, 1.9]) and −0.3 points ([−1.2, 0.5]), within the pre-specified ±5-point margin.

Answering without written reasoning

In this condition the prompt asks for the final answer line only, any reasoning channel is closed, and the answer line is prefilled with FINAL: E1 in every model, so the first entity of the answer is supplied; the model then has 48 new tokens and no retry. Validity fell by 86 points (Qwen3-1.7B), 57 (OLMo), 57 (gpt-oss-20b) and 74 (gpt-oss-120b) compared with free generation. The MRV-like contrast dropped by 49, 50 and 98 points for OLMo, gpt-oss-20b and gpt-oss-120b, with all intervals above zero. For Qwen3-1.7B it could not be estimated reliably, because only six problems had enough valid answers.

Process evidence

These three exploratory analyses fall outside the pre-specified family of tests and its multiplicity correction. They describe the written reasoning and do not establish how the models compute their answers. Reasoning is measured in native tokens over Qwen3-1.7B’s thinking, gpt-oss-20b’s analysis channel and OLMo’s response up to the FINAL line.

Commitment along the reasoning. For each of 24 problems per model (12 at each of B=4B = 4 and 6), we took the first valid free-generation response that ended normally, cut its reasoning at 10%, 20%, …, 90% of its length, and sampled 16 continuations from each cut. The commitment at a cut is the share of continuations whose final answer is the source response’s solution. The median commitment first reaches 90% at 50% (B=4B = 4) and 70% (B=6B = 6) of the reasoning for OLMo, at 80% and 80% for gpt-oss-20b, and at 40% and 70% for Qwen3-1.7B. We also annotated the first point at which a response introduces an assignment as a trial rather than a deduction (for example, “Suppose E4 = V2”), with model assistance and an author checking 20% of the annotations. Between the last cut before this first trial and the first cut after it, commitment rises by 18 points for OLMo (95% interval [8, 28]; 23 responses), 11 for gpt-oss-20b ([1, 22]; 24) and 44 for Qwen3-1.7B ([32, 57]; 24). The first trial assignment is the one the primary procedure would try first in 18 of 23 OLMo responses, 10 of 24 gpt-oss-20b responses and 13 of 24 Qwen3-1.7B responses. Commitment to the final solution therefore builds up over much of the reasoning, with a step at the first trial assignment, rather than being fixed at the start.

Reasoning length and search effort. For each presentation, we counted the dead ends (failed partial assignments) that the model’s primary procedure visits before its first solution, and the same count for name order. On valid answers that match the primary procedure’s prediction (31,589 responses from the main free-generation data in four presentations, the prospective test and the value-label data of § 4.2), log reasoning length rises with the primary procedure’s dead-end count, adjusting for BB, by 0.023 per dead end for OLMo ([0.007, 0.050]) and 0.024 for gpt-oss-20b ([0.009, 0.042]); for Qwen3-1.7B the interval includes zero (0.013, [−0.004, 0.027]). Name order’s dead-end count shows no association in any model. Predictions of length from the two counts have almost the same error, and in pairs of presentations that differ only in value labels the association with the primary procedure’s count is resolved in only one of six model and pair-type combinations (OLMo, 0.057, [0.000, 0.107]). This is therefore weak evidence that the length of the written reasoning follows the primary procedure’s search effort.

Answers outside the predictions

We pooled the ranks of 48,421 valid answers from the four free-generation presentations, the prospective test and the value-order experiment. For each ordering, we counted how often an answer outside the first rank fell at rank 2. The table gives the rank-2 count divided by the number of answers outside rank 1, separately for each ordering. This post-hoc summary is descriptive (no interval).

ModelBBMRV-likeName order
Qwen3-1.7B41,630/3,6041,921/5,087
61,532/3,8041,397/5,519
OLMo41,147/2,4261,425/3,436
6814/2,4521,292/4,062
gpt-oss-20b41,251/3,0541,554/3,892
6965/3,3381,169/4,554

Table 1.

Under uniform sampling over the remaining B−1B - 1 solutions, the rank-2 share would be 33.3% at B=4B = 4 and 20.0% at B=6B = 6. All three models exceeded these references under both orderings. The MRV-like ordering had the higher rank-2 share in every comparison, with the largest difference for Qwen3-1.7B at B=6B = 6 (40.3% versus 25.3%). We also fitted a noisy version of the primary procedure, which with probability δ\delta replaces the next entity by a random unassigned one and with probability ε\varepsilon replaces the next value by a random remaining consistent value; its first-solution distribution is computed exactly and fitted by maximum likelihood to the data used for the procedure mixture. In leave-one-cluster-out evaluation, its log-likelihood per valid answer is −1.07, −1.00 and −1.15 for Qwen3-1.7B, OLMo and gpt-oss-20b, against −1.09, −0.92 and −1.11 for the mixture (Appendix § 11); on the prospective-test problems it is −1.28, −1.29 and −1.44, against −1.36, −1.20 and −1.40. Noisy search had the higher held-out log-likelihood for Qwen3-1.7B, and the mixture for OLMo and gpt-oss-20b; a ranking of fits alone does not show which mechanism produced the answers outside the predictions.

Larger temperature increase

Raising the temperature by 0.5 instead of 0.25 reduced validity by 87 points for OLMo and 91 points for gpt-oss-20b and did not change it for Qwen3-1.7B. This condition was not completed for all problems, so we report it only here.

Remaining concentration on ZebraLogic

After randomised presentation on ZebraLogic, the share of valid answers on each model’s default exceeds the procedure mixture’s prediction by 11.9, 13.2 and 11.2 points for Qwen3-1.7B, OLMo and gpt-oss-20b, respectively (re-run means over 48 puzzles, descriptive, no interval). In the original runs the excess was 14, 12 and 13 points, with all three 95% intervals above zero. In a separate analysis of the original runs, the intervals for excess share on the source puzzle’s solution, an alphabetical solution and a control solution all included zero (Appendix § 13). The forecast fixed before the run assumes 32 valid answers per puzzle, whereas observations use 32 requests, not all of which are valid. The effective number computed from a finite sample is biased downwards, more so for fewer answers, so we compare observed breadth with the forecast recomputed at the observed numbers of valid answers (Figure A2): observed breadth remains 12.0, 15.3 and 12.1 points below the forecast

Figure A2. Observed breadth under randomised presentation and its forecast on ZebraLogic. Observed breadth of the re-run (filled; 32 requests per variant) and the forecast fixed before the run, recomputed at the observed numbers of valid answers (open), by model and BB. The forecast is not refitted; all nine 95% intervals of the difference lie below zero.

(13.2, 16.2 and 12.2 points at 32 valid answers), and all nine 95% intervals by model and BB are below zero, so this sampling effect explains little of the gap. For the random search order, whose forecast is the distribution of the frozen per-request predictions, the gaps at the observed numbers are −23.1, −3.7 and 0.0 points. The excess share on the default is not affected by this sampling effect.

Absolute breadth and validity

Table A3 gives absolute levels for the three main models. All Symbolic conditions use the same 72 problems. For the ZebraLogic temperature comparison, Appendix § 13 gives the unmodified-prompt levels from the same original runs. The final reader was used throughout the ZebraLogic block.

Original ZebraLogic runs

The ZebraLogic results for the unmodified prompt, randomised presentation, the random-choice instruction and the specified orders in the main text come from a re-run with new seeds. This section reports why these conditions were re-run and how the original runs were read. Declared value order, cueing and the higher temperature were not re-run; their results come from the original runs and hold under both readers below, except that the higher-temperature comparison is descriptive.

Deviation in the order instruction. The prespecified instruction began “Fill the grid house by house, from House 1 to House nn,” and all specified-order predictions assume that the search starts from

House 1. Because of a template error, every original prompt with a fixed or random search order instead read “Fill the grid house by house, from House nn.”, where nn is the number of houses (3 or 4). This wording can be read as asking the model to start from the last house. The re-run used the pre-specified wording (Appendix § 8). Where the two readings predict different solutions, 60.7% of gpt-oss-20b’s answers to the original fixed-order prompt matched the last-house prediction, up from 47.9% under the unmodified prompt. Its share on the first-house prediction changed from 23.0% to 23.7%. OLMo’s share rose by 5–6 points for both predictions, and Qwen3-1.7B’s share on the first-house prediction rose from 14.8% to 34.9%. With the corrected wording, the first-house prediction matched 39%, 59% and 77% of answers to the fixed-order prompt, and the last-house prediction matched 31%, 12% and 11%. The gain in first-house prediction share over the unmodified prompt was 6, 32 and 53 points larger than in the original runs. These differences combine the change of wording with run-to-run variation; for the conditions whose prompts did not change, every difference between the runs has a 95% interval that includes zero.

Original-run levels. Under the final reader, the original runs gave breadth and validity of 56% and 86%, 65% and 83%, and 66% and 97% for the unmodified prompt (Qwen3-1.7B, OLMo, gpt-oss-20b), against which the higher-temperature levels in Table A3 compare. Between the original and the final reader, breadth changes by at most 4 points except for Qwen3-1.7B under the specified orders (20 points) and gpt-oss-120b under randomised presentation (12 points), where the original reader accepted fewer than a fifth of the answers. Validity rises by 8 to 22 points in most conditions and by 39 to 83 points for gpt-oss-120b, for the specified orders in Qwen3-1.7B and OLMo and for the higher temperature in gpt-oss-20b; a reader with the vocabulary step alone gives intermediate validity.

Did models favour the source puzzle’s solution?

Each public source puzzle has a unique solution, which we recovered from the full set of clues and which remains valid after clue removal. We scored the original runs with the original reader and weighted all 48 variants equally. After randomised presentation, the observed share of valid answers giving the source solution minus the procedure mixture’s forecast was +0.51 points for Qwen3-1.7B (95% interval [−4.59, 5.72]), −0.05 for OLMo ([−6.33, 6.18]) and +0.60 for gpt-oss-20b ([−4.58, 5.65]). Of 1,536 requests per model, 1,099, 1,142 and 1,152 gave valid answers, respectively. Under the unmodified prompt, the three model pairs agreed on a default in 119 of 144 pair–puzzle comparisons after splitting credit across ties. The source solution accounted for 39 of those 119 agreements (32.8%, 95% interval [18.9, 47.5]%). Independent uniform choices would put 42.3% of expected agreements on the source solution, accounting for the different numbers of solutions per puzzle. Adding both the source solution and a solution obtained by ordering attributes and values alphabetically to the mixture closed 0.6%, 1.5% and 3.6% of the gap between forecast and observed breadth, respectively. All three 95% intervals for gap closure included zero. These tests measure preference for the original solution. The models’ exposure to the puzzles during training remains unknown.

Reading ZebraLogic answers

The re-run uses the final reader described here, fixed before its generation. For each ZebraLogic response, the final answer is read from the text marked FINAL. The reader first removes one attached end-of-text token and maps value spellings through a fixed vocabulary. Complete assignments are then put in a canonical form by house, attribute and value. Repeated statements of the same assignment count once, and two or more different assignments make the answer ambiguous. If no complete FINAL assignment can be read, a fixed rule reads the answer from other layouts. Both parts of this reader were fixed before a manual review sample; their combination was chosen after that review and fixed before a final audit. The layout part alone had not passed its own pre-specified acceptance check (3 errors among 60 newly read responses, with at most 1 allowed). We therefore report both readers.

Why the readers differ. The original, pre-specified reader rejected many answers only because of value spellings or layout. About a third of the answers it marked invalid were complete answers under the fixed vocabulary. In a post-hoc audit against 300 manually read responses, the final reader agreed with the manual reading in 282. In 13 of the 18 disagreements it read no answer where the manual reading found one. In 4 it read one answer where the manual reading found none or several, and in 1 it read the answer as ambiguous. The audit samples were stratified towards responses whose reading changed, so this agreement rate is not an estimate of overall accuracy. What the audit does establish is the direction of the remaining errors: most are missed answers, which lower measured validity. The same reader is applied to both arms of each comparison, so a miss rate common to both arms largely cancels in a validity difference. A miss rate that differs between arms does not cancel, and the specified-order instruction changes how answers are written. The audit was not designed to bound such differential errors at the scale of the ±5\pm5-point margin, so equivalence decisions that change between readers are conditional on the final reader.

Which results change. Table A4 compares the original and final readers on the concentration, cueing and intervention results from the original runs. Concentration, declared-order following, cued recoverability, the breadth gains under randomised presentation, its pre-specified test for OLMo and its comparison with the random-choice instruction hold under both. Five pre-specified decisions change, all from not supported to supported: the test for randomised presentation for gpt-oss-20b, fixed specified-order following for Qwen3-1.7B, and the test for the random search order for all three main models. The changes for the random search order are the largest. Under the original reader, it appeared to reduce validity by 63–76 points for Qwen3-1.7B and 26–34 points for OLMo, and Qwen3-1.7B also showed no breadth gain, on only the 31 puzzles that still had valid answers. Under the final reader, its validity changes by less than 4 points in every model. The specified-order instruction changes how models write their final grids, and the original reader failed to read many of those answers. Conclusions about specified search orders on ZebraLogic therefore rest on the final reader. Because the final reader was selected after the responses were inspected, and its layout component failed its own acceptance check, we treat the five changed decisions as sensitivity results under revised scoring, not as pre-specified confirmations; they are marked † in Table A4.

ResultReaderQwen3-1.7BOLMo-3.1-32Bgpt-oss-20bgpt-oss-120b
Concentration above uniformorig.33 [28, 37]24 [20, 28]25 [22, 29]22 [17, 27]
final33 [28, 37]24 [20, 29]25 [22, 29]34 [30, 38]
Declared-order followingorig.35 [27, 44]30 [24, 37]29 [23, 36]9 [7, 11]
final41 [33, 51]33 [25, 40]36 [28, 44]44 [34, 54]
Cued: non-default − defaultorig.−0.8 [−4.1, 2.5]−3.9 [−7.5, 0.0]1.2 [−2.3, 4.7]–
final−2.0 [−5.2, 1.1]−0.3 [−3.6, 3.0]0.7 [−0.8, 2.4]–
Randomised presentation
breadth gainorig.25 [18, 31]12 [6, 18]17 [12, 23]14 [8, 21]
final25 [18, 31]12 [6, 18]17 [12, 22]23 [17, 29]
validity changeorig.−2.3 [−5.5, 0.7]−0.1 [−2.4, 2.3]−2.6 [−5.1, −0.1][−2.9, 2.2]
final−2.5 [−5.7, 1.0]−0.3 [−2.5, 2.0]−0.3 [−1.8, 1.1]0.0 [−0.8, 0.8]
pre-specified test metorig.noyesno–
finalnoyesyes{sup}†–
gain over random-choice instructionorig.22 [15, 29]5 [−1, 13]12 [6, 18]–
final24 [17, 31]6 [−1, 13]13 [7, 19]–
Fixed specified order
gain on the specified solutionorig.−10 [−16, −5]−4 [−10, 1]1 [−5, 6]–
final20 [13, 28]{sup}†5 [−1, 11]1 [−6, 7]–
Random search order
breadth gainorig.−1 [−7, 5]23 [17, 28]26 [21, 30]–
final14 [10, 18]25 [20, 31]24 [20, 29]–
validity changeorig.[−76.0, −62.8][−33.7, −25.5][0.8, 9.4]–
final0.1 [−3.2, 3.6]0.4 [−2.1, 2.8]0.4 [−0.5, 1.3]–
pre-specified test metorig.nonono–
finalyes{sup}†yes{sup}†yes{sup}†–
After randomised presentation
default share above forecastorig.13 [7, 19]11 [5, 18]12 [7, 17]–
final14 [8, 20]12 [6, 19]13 [8, 18]–
breadth − forecastorig.−16 [−21, −11]−18 [−24, −14]−13 [−18, −9]–
final−14 [−19, −9]−18 [−23, −13]−12 [−16, −8]–

Table A4. ZebraLogic results under the original and the final answer reader. All values in points, with 95% intervals (90% for validity changes and the cued-success difference). Brackets alone give the interval for an original-reader estimate. orig.: original reader. Bold: pre-specified test supported after correction; †: decision differs from the original reader. gpt-oss-120b values are secondary comparisons without a pre-specified test.

Figure notes

Figure 1. Panel A shows each model’s modal share minus the modal share expected from uniform draws at the same valid sample size. Both shares use problems with complete records for the first 32 requests. The intervals apply to their difference. On ZebraLogic, the points by BB are descriptive means (no interval). The point labelled “all” gives the pre-specified estimate over all 48 puzzles with its 95% interval. On the Symbolic task each point uses 24 problems for Qwen3-1.7B and gptoss-120b, 15 for gpt-oss-20b, and 21, 20 and 20 for OLMo at B=2B = 2, 4 and 6, which matters when comparing models. In panel B the chance level comes from relabelling one model’s answers uniformly over each problem’s valid solutions (1,000 permutations per problem). For the gpt-oss pair on the Symbolic task and for ZebraLogic, this reference was computed after the original analyses. The pre-specified test for the gpt-oss pair used instead a baseline that pairs one model’s modal answer with another problem’s (2%); a shuffled answer need not be valid for its destination problem, so that baseline does not test independent choice within a problem and is not drawn. Panel C on ZebraLogic comes from the original runs, because cueing was not re-run. Joint movement (§ 3.2) is the share of problems on which both models’ most frequent solutions differ between the unmodified presentation and the manipulated one and coincide in the manipulated one (ties counted fractionally), over the 72 problems on which both models have at least eight valid outputs in both conditions; its null relabels one model’s answers by the same random permutation of the solution set in both conditions (1,000 permutations per problem). In panel C, Symbolic solutions are ranked in name order (rank 1 is the solution a name-order search reaches first, which is not always the default), and ZebraLogic solutions by observed free-generation frequency, with ties shared. Free-generation frequency and cued success keep all requests, including invalid ones, in the denominator.

Figure 2. Panel B uses 154, 149 and 149 pre-sentations for the three main models and 149 for gpt-oss-120b; the values are differences in prediction share, not accuracy. All shares in panels B–D count invalid outputs in the denominator. Panel B compares the shares on two predicted solutions within the same outputs. Panel C reports factorial contrasts on 48 problems: with scs_c the share on the solution predicted for declaration order dd minus the share on that predicted for its reverse d′d' in condition cc, the declaration effect is 14{[s(d∣L)−s(d′∣L)]+[s(d∣L′)−s(d′∣L′)]}\frac{1}{4}\{[s(d\mid L)-s(d'\mid L)]+[s(d\mid L')-s(d'\mid L')]\}, where LL and L′L' are the two listing orders. This equals the average of four paired changes in share on a predicted solution when its declaration order replaces the reverse, with the listing order fixed; the listing effect is defined analogously. Listing order here is the order of each entity’s excluded values, not of whole facts. Panel D reports the share on the specified solution under the instruction minus that under the unmodified prompt. In panel D, the Symbolic conditions use 47 (both orders) or 48 problems and ZebraLogic uses 48 puzzles. gpt-oss-120b was not run with specified orders.

Figure 3. The pre-specified test requires a breadth gain and validity within the margin, with Holm correction within each model’s family of tests; in every case shown, not passing means that the validity interval leaves the margin. This includes OLMo with renamed entities, whose validity rose by 1.8 points with a 90% interval reaching 3.0 points. On the Symbolic task, the intervals for the random-choice instruction, the higher temperature and enumerate-then-choose were computed on the problems complete in the first run of these conditions (72, 60 and 45 problems for Qwen3-1.7B, OLMo and gpt-oss-20b); the order interventions use all 72, and Table A3 gives every condition on all 72. The validity axes are clipped at −12.5-12.5 points; OLMo’s validity with enumerate-then-choose is −19.9-19.9 points (90% interval [−22.5,−17.2][-22.5, -17.2]) and is drawn as an arrow. gpt-oss-120b was run with randomised presentation only; on ZebraLogic its test belongs to the re-run’s pre-specified family. In panel B the re-run estimates for the random-choice instruction are descriptive (no interval). The higher-temperature comparison uses the original runs.

References

  1. [1]Xinyun Chen, Ryan Andrew Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 6596–6620. PMLR.arxiv.org/abs/2402.08939
  2. [3]Bradley Efron and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. Chapman & Hall.
  3. [4]Kanishk Gandhi, Denise H. Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D. Goodman. 2024. Stream of search (SoS): Learning to search in language. In First Conference on Language Modeling.arxiv.org/abs/2404.03683
  4. [5]Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. 2025. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. Advances in Neural Information Processing Systems, 38:43140–41391.arxiv.org/abs/2502.05171
  5. [6]Zorik Gekhman, Eyal Ben-David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart. 2025. Inside-Out: Hidden Factual Knowledge in LLMs. In Second Conference on Language Modeling.
  6. [7]Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2025. Training Large Language Models to Reason in a Continuous Latent Space. In Second Conference on Language Modeling.arxiv.org/abs/2412.06769
  7. [8]Robert M. Haralick and Gordon L. Elliott. 1980. Increasing tree search efficiency for constraint satisfaction problems. Artificial Intelligence, 14(3):263–313.
  8. [9]Mark O. Hill. 1973. Diversity and evenness: A unifying notation and its consequences. Ecology, 54(2):427–432.DOI
  9. [10]Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70.DOI
  10. [11]Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations.
  11. [12]Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, and Yejin Choi. 2025. Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond). Advances in Neural Information Processing Systems, 38. NeurIPS 2025, Datasets and Benchmarks Track.arxiv.org/abs/2510.22954
  12. [13]Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. Understanding the Effects of RLHF on LLM Generalisation and Diversity. In International Conference on Learning Representations.arxiv.org/abs/2310.06452
  13. [14]Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. In International Conference on Learning Representations.arxiv.org/abs/2302.09664
  14. [15]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles.arxiv.org/abs/2309.06180
  15. [16]Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2026. LLMs Get Lost In Multi-Turn Conversation. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2505.06120
  16. [17]Lucas Lehnert, Sainbayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul McVay, Michael Rabbat, and Yuandong Tian. 2024. Beyond A*: Better planning with transformers via search dynamics bootstrapping. In First Conference on Language Modeling.arxiv.org/abs/2402.14083
  17. [18]Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. 2025. ZebraLogic: On the scaling limits of LLMs for logical reasoning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 37889–37905. PMLR.arxiv.org/abs/2502.01100
  18. [19]Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157–173.arxiv.org/abs/2307.03172
  19. [20]Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.arxiv.org/abs/2104.08786
  20. [21]Aman Mehta. 2026. When Agents Commit Too Soon: Diagnosing Premature Commitment in LLM Agents. arXiv preprint. arXiv:2606.22936 [cs.AI].arxiv.org/abs/2606.22936
  21. [22]OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv:2508.10925 [cs.CL].
  22. [23]Vishakh Padmakumar and He He. 2024. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2309.05196
  23. [24]Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico. Association for Computational Linguistics.
  24. [25]Isha Puri, Mehul Damani, Idan Shenfeld, Marzyeh Ghassemi, Jacob Andreas, and Yoon Kim. 2026. Escaping the mode: Multi-answer reinforcement learning in LMs. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 100127–100146. PMLR. arXiv:2603.24844.
  25. [26]Alex Renda, Aspen Hopkins, and Michael Carbin. 2023. Can LLMs generate random numbers? evaluating LLM sampling in controlled domains. In Sampling and Optimization in Discrete Space Workshop at ICML 2023.
  26. [27]Matthew Renze and Erhan Guven. 2024. The effect of sampling temperature on problem solving in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7346–7356, Miami, Florida, USA. Association for Computational Linguistics.
  27. [28]Stuart Russell and Peter Norvig. 2020. Artificial Intelligence: A Modern Approach, 4 edition. Pearson.

Paper details

vlnc:W_k473tz4v
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents