betaFeedback

WorksComputer Science

On the Fragility of Tasks under Composition: Implicit Budgeting in Language Models

Abstract

Modern language models are increasingly asked to handle several tasks within a single prompt. Doing so requires solving each task correctly while integrating the resulting answers into a coherent response. In this paper, we identify a fundamental limitation in this process, where models make substantially more errors when tasks are composed, even when they can solve every constituent task reliably in isolation. This degradation cannot be fully explained by the accumulation of independent errors. Across both open-weight models and the frontier model Claude Opus 5, our findings instead point to an implicit thinking budget underlying this effect. In particular, under composition, models use substantially fewer tokens per task than in isolation, even when the full sequence remains well within the context window. Tasks that require longer solution traces in isolation are consequently more fragile\textit{fragile} under composition, i.e., models make more errors on prompts containing these tasks. In isolation, non-fragile tasks receive more reasoning than they need and waste compute. However, by grouping tasks according to their fragility, the implicit budgeting can be manipulated to reduce token usage by 4343% across a range of math and linguistic tasks while also preserving accuracy.

Project page: https://chenpanli.github.io/fragility

Code: https://github.com/chenpanli/fragility

Figure 1. Composition reduces every task’s reasoning by the same factor. When solved in isolation (top), each task receives more reasoning than it requires. When composed into a single prompt (bottom), the reasoning allocated to each task shrinks proportionally (yellow). Tasks with sufficient slack absorb this reduction and are still solved correctly, whereas the task that requires long reasoning falls below its requirement and thus fails.

Introduction

A single prompt often asks a language model to do several things at once, such as answering a handful of questions, solving a set of math problems, or following a list of instructions before writing one response. Common benchmarks, however, still test models one task at a time [29, 13, 9]. As scores on these benchmarks saturate, they may overstate how robust the underlying capabilities are. We ask whether a model that reliably solves a task on its own can still solve it when other tasks share the prompt.

One natural explanation is error accumulation. If each task fails with nonzero probability, solving many at once fails more often. Prior work has found this phenomenon when models are tasked to do familiar reasoning steps [25, 7, 32] or follow several instructions at once [12, 35]. However, accumulation is only half of the story. Because composition in these settings also changes the structure and token demands of the task, it remains unclear whether the model’s ability to solve each constituent task degrades as well, and what mechanism drives that degradation.

To investigate the question, we analyze the effect of composition on each constituent task of a composition. We start by establishing that this per-task degradation exists at all. For that, we select tasks the model solves correctly in isolation, which establishes that the requisite capability is present. We then leave those tasks unchanged and simply concatenate them within a single prompt. This design separates degradation under increased task load from baseline solvability on the constituent tasks.

Across state-of-the-art open models, as well as the frontier model Claude Opus 5, we find substantial performance degradation when more previously solved tasks are concatenated within the same prompt. The drop in accuracy is substantially larger than what the constituent tasks’ isolated error rates would predict, and is consistent across both mathematical and linguistic domains. We characterize this degradation along two dimensions. 1) Failures concentrate on a subset of fragile tasks that, for a given model, remains stable across sampling noise and composition sizes. These tasks also have longer reasoning traces in isolation. 2) As more tasks are presented together, models systematically reduce the reasoning tokens devoted to each task, even though no explicit per-task token constraint is imposed. This reduction follows a clear scaling pattern, with reasoning per task decreasing approximately as a power law with composition size. Together, these findings reveal an implicit thinking budget under composition, under which fragile tasks disproportionately lose accuracy as per-task reasoning decreases.

This mechanism also creates an opportunity for more efficient inference than solving every task separately. Non-fragile tasks can be grouped into larger batches, where the implicit thinking budget reduces per-task reasoning and token usage without impacting accuracy. Fragile tasks, in contrast, are placed in smaller batches so that each receives more reasoning and remains accurate. We demonstrate this with fragility-aware batching, a strategy that composes each task based on its fragility. By using empirical measurements of the fragility of tasks, it reduces token usage by 43% on average with no drop in performance compared to solving each task individually. Because it is entirely black-box, it also applies directly to proprietary models, reducing generated tokens by 24% on GPQA-Diamond with Claude Opus 5 without degrading performance.

Taken together, these results show that the failure of LLMs to generalize their reasoning budget management is a double-edged sword. Under composition, it causes fragile tasks to fail, whereas in isolation, it makes models overspend reasoning on non-fragile tasks, thus wasting compute during inference. Our key contributions and findings are:

  • We show that simply concatenating tasks that are already solved correctly in isolation causes substantial accuracy degradation across both frontier and open models (§ 3).

  • We identify an implicit per-task thinking budget that shrinks as a power law in the number of

composed tasks (§ 3.1).

  • We show that this uniform reduction concentrates degradation on tasks with long isolated reasoning, a set that is stable within each model but differs between models (§ 3.1 and § 3.2). We refer to this set of tasks as fragile.

  • We turn these findings into fragility-aware batching, a strategy that groups tasks according to their measured fragility. It reduces generated tokens by ∼43%\sim43\% across all datasets while preserving accuracy on both state-of-the-art open-weight models and proprietary frontier models (§ 5).

Preliminaries

Our central goal is to investigate whether an LLM that can solve a task in isolation keeps this ability when several such tasks are presented together. To answer this, we use a controlled setting in which the constituent tasks are unchanged and only the number of tasks in the prompt varies. We then measure how each task’s accuracy changes relative to its performance in isolation. All experiments share the setup described below.

Compositions. We compose independent tasks drawn from the same dataset D\mathcal{D}. A composition of size kk is a tuple C=(t1,…,tk)C = (t_1, \ldots, t_k) with ti∈Dt_i \in\mathcal{D}, rendered into a single prompt by a fixed per-dataset template that asks the model to answer all kk tasks in order. We split the model’s single response into kk answers and grade each for correctness (further details in Appendix § 11.2.6).

Accuracy drop. For a model mm, the accuracy drop of each individual task tit_i at composition size kk is Δim(k)=acc⁡im(k)−acc⁡im(1)\Delta_i^m(k) = \operatorname{acc}_i^m(k) - \operatorname{acc}_i^m(1), where negative values indicate degradation. Unless stated otherwise, accuracy is estimated from 5 independent trials.

Datasets. We use eight datasets whose tasks vary widely in the reasoning they require. We build two datasets ourselves: 1,000 indefinite integrals with randomized coefficients across diverse function classes, verified symbolically, and OneWordQA, a set of factual questions with single-word or numeric answers that require minimal reasoning. The remaining six are established benchmarks: GPQA-Diamond [29], GSM8K [5], MATH-500 [13, 19], MMLU-Redux [9], StrategyQA [10] and AIME (further details in Appendix § 11.2.6).

Models. We evaluate three Qwen3 models (4B Thinking, 8B, 14B) [40], trained by strong-to-weak distillation from larger RL-trained Qwen3 models, and Qwen3.8 27B [27]. We also evaluate three DeepSeek-R1-distilled models (Qwen 7B, Llama 8B, Qwen3 8B) [11], obtained by supervised fine-tuning on R1 reasoning traces, and the frontier model Claude Opus 5 with medium reasoning effort. The first two groups let us compare two different post-training recipes at similar scale. Open-weight models are sampled at temperature 0.6 and use a context limit of at least 131,072 tokens (further details in Appendix § 11.2.2).

Composing tasks causes performance degradation

Prior work has repeatedly reported that accuracy falls when several tasks share a single prompt. However, these evaluations do not isolate whether a model fails because it cannot solve a task reliably or purely because the task is composed with others, which leaves the underlying mechanism unclear.

Building on these observations, we ask whether degradation can arise from composition alone. Answering this requires separating a model’s ability to solve a task from its ability to solve the same task in composition. For that, we restrict each model MM to a subset DM⊆D\mathcal{D}_M \subseteq\mathcal{D} of confidently solved tasks: a task enters DM\mathcal{D}_M only if MM solves it in isolation in three out of three independent attempts. We then build compositions CC of size k∈{3,6,10,20}k \in\{3, 6, 10, 20\}1 exclusively from DM\mathcal{D}_M, sampling task categories uniformly within each dataset, and evaluate the accuracy of each task within them. The accuracies at k=1k = 1 come from runs independent of those used for selection.

As Figure 2 shows, simply concatenating tasks that are solvable in isolation causes substantial accuracy degradation across all models on GPQA-Diamond [29]. This observation also holds for all other datasets we tested, except OneWordQA and StrategyQA, whose tasks require little reasoning. Furthermore, this degradation cannot be explained by context-length issues, since nearly all responses stay within the context limit (see Figure 3 and Appendix § 11.1.1). Compositional degradation is therefore driven by effects inherent to composition and cannot be predicted from the solvability of tasks in isolation alone.

Figure 2. Accuracy degradation on confidently solved GPQA-Diamond tasks. Left: accuracy drop per model at k=10k = 10. Right: accuracy drop as a function of composition size kk. All tasks were solved in three out of three isolated attempts, yet accuracy drops once they are composed.

Figure 3. Context usage across datasets. Most responses use under 25% of the context window; in most datasets, under 1% hit the limit.

The extent of this degradation varies strongly across models. The R1-distilled models degrade the most, while the Qwen3 models degrade moderately less. Frontier models are not immune either: Claude Opus 5, despite its leading benchmark results, loses about 7 pp at k=10k = 10 on GPQA-Diamond, on tasks it solved in all three runs individually.

Contrary to explanations for compositions with dependent tasks, where models either match solution patterns seen in training instead of learning systematic procedures [7] or fail to combine facts they know individually [25], we propose that compositional degradation arises from a misallocation of the implicit thinking budget.

Degradation as a misallocation of the implicit thinking budget

Reasoning models solve tasks by thinking stepwise, a process shown to be crucial for task success [37]. We therefore ask whether the accuracy degradation under composition can be explained by changes in thinking behavior. A central property of this behavior is the number of thinking tokens the model spends on a task, i.e., how much effort it invests in it. This effort can strongly affect whether a task is solved [33], and prior work has shown that overall reasoning shortens when batching tasks [34, 6, 3]. What remains unclear is how each constituent task is affected, that is, how its thinking length under composition compares to its thinking length in isolation and how that change affects the task’s success.

Thinking traces shrink under composition. We close this gap by measuring the thinking tokens spent on each task within a composition of kk tasks, denoted Ti(k)T_i(k). To this end, we prefix each task with a unique tag (Q01, Q02, …) in the prompt that does not appear in the task description itself. These tags let us attribute each segment of the reasoning trace to the task it addresses (details in Appendix § 11.2.1). We quantify the reduction for each task by the token ratio ri(k)=Ti(k)/Ti(1)r_i(k) = T_i(k)/T_i(1) between its thinking tokens under composition and in isolation. Figure 4 shows how this token ratio evolves with composition size. With one exception (R1 Qwen 7B on MATH-500), the thinking tokens of almost all tasks decrease monotonically as compositions grow (further models and datasets in Appendix § 11.1.2); a weak positional bias is discussed in Appendix § 11.1.4.

Figure 4. Token ratio ri(k)=Ti(k)/Ti(1)r_i(k) = T_i(k)/T_i(1) as a function of composition size kk, where Ti(k)T_i(k) is the number of thinking tokens spent on task tit_i within a composition of size kk and Ti(1)T_i(1) the number spent in isolation. Per-task token counts in compositions are obtained by locating task tags in the reasoning trace. Tasks are binned by the quantile of their isolated trace length into Low (≤20%\le20\%), Medium (20–80%), and High (>80%> 80\%). On log-log axes, the ratio decreases approximately linearly in log⁡k\log k, indicating that LLMs shrink the relative token usage of each task under composition by roughly kmk^m, with no systematic dependence on Ti(1)T_i(1). Results are from Qwen3 4B Thinking.

Long-reasoning tasks degrade more. This reduction has important implications, since prior work has established that the number of thinking tokens spent on a task can influence its accuracy. In particular, interrupting the thinking process externally at earlier stages causes a larger degradation [21]. Composing tasks has an analogous effect on thinking length, as it implicitly constrains the per-task thinking tokens but without any external enforcement. This naturally raises the question of whether the implicit token reduction under composition is related to the accuracy degradation. Figure 5 shows how the reduction affects tasks with different isolated thinking lengths. Tasks with longer isolated traces degrade substantially more when thinking tokens are reduced, whereas tasks with shorter traces are largely unaffected. We refer to tasks that break under composition in this way as fragile.

Figure 5. Accuracy drop as a function of the relative thinking token reduction for Qwen3 4B Thinking on MMLU-Redux (left) and MATH-500 (right) below the dashed line, the y-axis is logarithmic for presentation purposes. Tasks are binned by the quantile of their isolated trace length into Low (≤ 20%), Medium (20–80%), and High (> 80%). As the reduction through composition (comp.) grows, accuracy drops, and tasks with high reasoning requirements degrade substantially more than low-effort tasks. Forcing a fixed budget externally (forced) generally causes stronger degradation, but the degradation follows the same patterns (more details in Appendix § 11.1.3).

Thinking length follows a power law. One possible source of the uneven degradation in Figure 5 could be that long-reasoning tasks lose a larger share of their thinking tokens. The token ratio approximately follows a power law in kk,

log⁡ri(k)≈mlog⁡k⟺ri(k)=Ti(k)Ti(1)≈km.(1)\log r_i(k) \approx m \log k \quad\Longleftrightarrow\quad r_i(k) = \frac{T_i(k)}{T_i(1)} \approx k^m . \tag*{(1)}

Since ri(1)=1r_i(1) = 1 by definition, we fit a line through the origin on log-log axes for each model and dataset. The fitted slopes have a median of m=−0.54m = -0.54 (m∈[−0.88,−0.04]m \in[-0.88, -0.04]), with a median R2R^2 of 0.91; averaged per model, R2R^2 ranges from 0.55 to 0.94 (Appendix § 11.1.2). Under composition, the reasoning trace of each task thus shrinks by approximately the same factor kmk^{m}, without systematic dependence on its isolated length Ti(1)T_i(1), so long-reasoning tasks do not systematically lose a larger share of their reasoning than short ones. The model preserves the relative allocation across tasks, assigning more reasoning to tasks that required more in isolation, but applies roughly the same proportional reduction to all of them. Considering that every task loses roughly the same fraction of its reasoning, the uneven degradation is best explained by how much reasoning each task can afford to lose. Short isolated traces seem to contain redundancy in reasoning beyond what the task requires, whereas long traces lie closer to the minimum reasoning needed to solve the task reliably. Hence, the same relative reduction pushes long-reasoning tasks below their minimal requirement while leaving short ones above it, concentrating degradation on tasks with long isolated traces. Since kmk^{m} decreases with kk (m<0m < 0), the implicit thinking budget shrinks as more tasks are composed, and degradation grows accordingly. This reveals a systematic failure of implicit thinking budget allocation under composition. The way models allocate their implicit thinking budget does not generalize well to composed tasks.

Fragile tasks are stable within a model

It remains to establish whether the implicit thinking budget estimation fails consistently on the same tasks across models and composition sizes. This tells us how the proposed misallocation changes as the conditions, i.e., model and composition size, vary. Since a failure in implicit thinking budget estimation manifests as accuracy degradation, we can test this consistency directly on the accuracy drops. For two conditions AA and BB, we obtain the accuracy drops across tasks, ΔA\Delta_A and ΔB\Delta_B, and measure their Pearson correlation, applying Spearman’s correction for attenuation to reduce the impact of noisy accuracy estimates (details in Appendix § 11.2.3).

At a fixed composition size, the mean correlation across models is only 0.24±0.030.24 \pm0.03. Different models thus largely disagree on which tasks degrade, even when evaluated under identical conditions. The exception is models trained with the same recipe, such as Qwen3 8B and 14B, which correlate at 0.56±0.090.56 \pm0.09. Within a single model, the correlation across composition sizes reaches 0.81±0.040.81 \pm0.04. When the same model faces compositions of different sizes, groupings and orderings, the same set of fragile tasks tends to break. Fragility is therefore a stable property of a task for a given model. Measuring a task’s accuracy drop at some composition sizes lets us anticipate how it will degrade in new compositions presented to the same model, which makes fragility predictable.

Exploiting implicit budgeting for efficient inference

Exploiting implicit budgeting for efficient inference

So far, our results suggest that LLMs fail on compositions mainly because their estimate of the implicit thinking budget does not generalize, rather than because they cannot transfer their problem-solving skills. If so, controlling how much reasoning each task receives should prevent the drop. The composition size kk offers a practical control mechanism on implicit budgeting, since the token reduction ratio follows a power law in kk. Hence, assigning robust tasks to larger compositions and fragile tasks to smaller ones could save tokens while limiting accuracy degradation.

Fragility-aware batching

To test whether this idea is viable, we first ask how many tokens could be saved if each task’s fragility were known. We estimate fragility from the measured accuracy drops and compose tasks accordingly, which gives a sense of what fragility-aware batching can achieve. In § 5.2, we replace these measurements with a linear predictor that predicts fragility, turning the approach into a method that can be deployed. Formally, we seek to assign each task a composition size kik_i that maximizes the expected token savings while bounding the expected accuracy degradation by BB. We express this trade-off as a Lagrangian relaxation,

ki(λ)=arg⁡max⁡k∈K[E[Δi(k)]+λ(1−cost⁡k)],(2)k_i(\lambda) = \mathop{\arg\max}_{k \in\mathcal{K}} \left[\mathbb{E}[\Delta_i(k)] + \lambda(1-\operatorname{cost}_k)\right], \tag*{(2)}

where K\mathcal{K} is the set of available composition sizes, E[Δi(k)]\mathbb{E}[\Delta_i(k)] is the expected accuracy drop of task tit_i at composition size kk, and cost⁡k∈(0,1]\operatorname{cost}_k \in(0,1] is the model-specific token cost of solving a task within a composition of size kk, relative to solving it in isolation. The parameter λ≥0\lambda\ge0 acts as an exchange rate between saved tokens and accuracy. We then choose λ\lambda such that the mean expected accuracy drop over the dataset, Δˉ(λ)=1n∑i=1nE[Δi(ki(λ))]\bar{\Delta}(\lambda) = \frac{1}{n}\sum_{i=1}^{n}\mathbb{E}[\Delta_i(k_i(\lambda))], conservatively stays within the budget BB,

λ∗=max⁡{λ≥0:Δˉ(λ)−SE⁡(Δˉ(λ))≥−B}.(3)\lambda^* = \max\left\{\lambda\ge0 : \bar{\Delta}(\lambda) - \operatorname{SE}(\bar{\Delta}(\lambda)) \ge-B\right\}. \tag*{(3)}

We find λ∗\lambda^* by a grid search on held-out splits, using repeated nested 5-fold cross-validation [1] to reduce the effect of noise in the empirical accuracy estimates. Given the assignments ki(λ∗)k_i(\lambda^*), we greedily group tasks into as few compositions as possible such that every task tit_i lands in a composition CC with ∣C∣≤ki(λ∗)|C| \le k_i(\lambda^*) (details in Appendix § 11.2.4). Throughout our experiments, we use the composition sizes K={1,3,6,10,20}\mathcal{K} = \{1, 3, 6, 10, 20\}, where k=1k = 1 solves a task alone, and an accuracy drop budget of B=0.5%B = 0.5\%.

Table 1 compares the accuracy degradation and token savings of different batching strategies relative to solving tasks in isolation. The common practice of batching tasks randomly into fixed-size compositions [3], here of six tasks, saves the most tokens on average but degrades accuracy strongly, particularly on GPQA-Diamond, whose tasks require many reasoning steps. Fragility-aware batching almost entirely avoids this degradation while still achieving substantial token savings. Averaged over all models and over every dataset but AIME, where fragility-aware batching leaves nearly every task in isolation (Appendix § 11.1.6), it saves 43% of tokens while keeping accuracy constant (+0.6 pp). How these savings arise depends on the dataset. On OneWordQA and StrategyQA, fragility-aware batching matches random batching, saving 68–74% of output tokens without losing accuracy relative to solving each task individually. On the remaining five datasets, where random batching degrades accuracy, fragility-aware batching preserves it while still saving roughly 32%.

Isolated, k=1k=1Random batching, k=6k=6Fragility-aware (measured)Fragility-aware (predictor)
Acc.Tok.Acc.Tok.Acc.Tok.Acc.Tok.
GPQA-Diamond
Qwen3 4B T.66.7%766356.4% (−10.3)2470 (−67.8%)66.8% (+0.2)5200 (−32.1%)67.3% (+0.7)7534 (−1.7%)
Qwen3 8B58.7%597353.4% (−5.3)1950 (−67.4%)60.0% (+1.3)3520 (−41.1%)56.9% (−1.8)5106 (−14.5%)
Qwen3 14B61.5%459751.8% (−9.7)1500 (−67.4%)57.9% (−3.7)2111 (−54.1%)61.3% (−0.2)3665 (−20.3%)
Qwen3.8 27B83.2%430379.2% (−4.0)2740 (−36.3%)84.0% (+0.8)4102 (−4.7%)85.3% (+2.1)4324 (+0.5%)
R1 Qwen 7B49.1%668623.6% (−25.5)1305 (−80.5%)49.6% (+0.5)4037 (−39.6%)46.7% (−2.4)5953 (−11.0%)
R1 Llama 8B47.7%742432.1% (−15.6)1771 (−76.2%)49.6% (+1.9)4526 (−39.0%)48.6% (+0.9)5816 (−21.7%)
R1 Qwen3 8B60.8%908448.1% (−12.6)2398 (−73.6%)61.9% (+1.2)6595 (−27.4%)59.8% (−1.0)8877 (−2.3%)
Claude Opus 589.5%68988.7% (−0.9)564 (−18.2%)89.2% (−0.4)521 (−24.3%)87.6% (−2.0)686 (−0.5%)
Average64.7%5802−10.5−60.9%+0.2−32.8%−0.5−8.9%
MATH-500
Qwen3 4B T.97.6%406496.4% (−1.1)2148 (−47.2%)97.7% (+0.1)3716 (−8.6%)96.8% (−0.8)3605 (−11.3%)
Qwen3 8B95.9%344995.6% (−0.3)1751 (−49.2%)95.1% (−0.8)2173 (−37.0%)95.9% (+0.1)2775 (−19.5%)
Qwen3 14B96.9%260495.9% (−1.0)1375 (−47.2%)96.6% (−0.2)2454 (−5.8%)96.6% (−0.2)2444 (−6.2%)
Qwen3.8 27B98.1%101496.7% (−1.5)748 (−26.2%)97.8% (−0.4)943 (−7.1%)97.2% (−0.9)994 (−2.0%)
R1 Qwen 7B93.0%238369.8% (−23.2)1658 (−30.4%)93.2% (+0.2)2205 (−7.5%)93.4% (+0.4)2381 (−0.1%)
R1 Llama 8B88.7%301979.4% (−9.3)1569 (−48.0%)88.5% (−0.2)2948 (−2.3%)88.8% (+0.2)3333 (+10.4%)
R1 Qwen3 8B96.8%676793.0% (−3.8)3705 (−45.2%)96.6% (−0.2)6608 (−2.4%)96.7% (−0.2)6657 (−1.6%)
Average95.3%3329−5.7−41.9%−0.2−10.1%−0.2−4.3%
MMLU-Redux
Qwen3 4B T.84.7%194482.9% (−1.8)527 (−72.9%)85.3% (+0.6)527 (−72.9%)84.9% (+0.2)1663 (−14.5%)
Qwen3 8B87.2%92184.0% (−3.3)306 (−66.7%)86.1% (−1.1)285 (−69.0%)86.9% (−0.3)922 (+0.2%)
Qwen3 14B89.0%71885.8% (−3.3)212 (−70.5%)88.2% (−0.9)319 (−55.6%)88.3% (−0.7)678 (−5.5%)
Qwen3.8 27B92.4%48891.9% (−0.5)262 (−46.4%)91.8% (−0.6)196 (−59.8%)92.1% (−0.3)352 (−27.9%)
R1 Qwen 7B70.1%107960.6% (−9.5)397 (−63.2%)69.7% (−0.3)601 (−44.3%)70.4% (+0.3)1127 (+4.5%)
R1 Llama 8B75.5%145858.6% (−16.9)431 (−70.4%)75.9% (+0.3)896 (−38.5%)75.3% (−0.2)1430 (−1.9%)
R1 Qwen3 8B81.6%171164.5% (−17.1)279 (−83.7%)82.1% (+0.5)1273 (−25.6%)82.2% (+0.6)1714 (+0.2%)
Average82.9%1188−7.5−67.7%−0.2−52.2%−0.1−6.4%

Table 1. Task accuracy (Acc.) and output tokens (Tok., includes thinking and answer) per task under different batching policies. For each policy, the left column reports accuracy and the right column tokens, with the change to solving in isolation (k=1k=1) in parentheses, given as an absolute difference in percentage points for accuracy and as a relative change in % for tokens. The Average row gives the mean change over the models shown. Standard fixed-size random batching causes substantial accuracy degradation, which fragility-aware batching largely mitigates. With a model-specific linear predictor, part of these token savings carries over to unseen tasks while still avoiding compositional degradation.

These results show that much of the thinking an LLM spends on a task in isolation is not needed to solve it. Specifically, on GPQA-Diamond, GSM8K and MMLU-Redux, grouping tasks by fragility saves a substantial share of tokens at no cost in accuracy.

Predicting fragility on held-out tasks

Our results show that the set of fragile tasks is stable for a given model, which hints at the existence of steady properties that define fragility. If such properties exist, fragility should be predictable for tasks the model has never seen under composition. Hence, we extend the intervention to unseen tasks by predicting their fragility with a simple linear predictor. We encode each task description with Qwen3 Embedding 8B [44], keep the embeddings fixed, and train a separate linear predictor for each evaluated language model. To generate labels, we evaluate each task in the training split with 10 trials at every composition size k∈{3,6,10,20}k \in\{3, 6, 10, 20\}, placing it in a different composition each time. Then, for each composition size kk the predictor outputs the probability that the task performs at least as well in composition as in isolation. To obtain predictions on unseen tasks, we use 5-fold cross-fitting [4], so that the held-out folds together cover the entire dataset (details in Appendix § 11.2.5). Finally, we feed the predicted accuracy drops into the same batching algorithm as in § 5.1.

Table 1 reports the results of the linear predictor. Guided by the predictor, fragility-aware batching limits the accuracy loss to at most 0.5 pp on every dataset and saves 31.0% of tokens on average. Because few training examples are available, the predictor is simple, and it works best on tasks with short, plain descriptions such as Integrals, GSM8K and OneWordQA. This simple demonstration thus shows that fragility is partly predictable from task properties encoded in the embeddings. Scaling such predictors further could enable saving tokens on real-world unseen tasks, just by batching them well.

Fragility-aware batching on a frontier model. We further test whether fragility-aware batching carries over to flagship frontier LLMs. Since the intervention requires neither fine-tuning nor access to internal states and works purely by restructuring compositions, we can apply it directly to Claude Opus 5 on GPQA-Diamond (Table 1). In isolation, Claude Opus 5 reaches an accuracy of 89.5% with 689 output tokens per task. Random composition with k=6k=6 lowers accuracy by 0.9 pp and saves 18% of output tokens. Batching by measured fragility does better on both metrics, losing just 0.4 pp while saving 24% of generated tokens. The linear predictor, trained on fewer trials for Opus than for the open-weight models, transfers less well: it saves almost no tokens and loses 2.0 pp. At least in its consequences, the mechanism we found in open-weight models carries over to a frontier model.

Effects of post-training on compositional degradation

The models in our analysis degrade very differently under composition, most clearly between the two model families, which differ in their post-training strategy. This hints that post-training strongly influences how well a model handles compositions. To investigate this, we repeat our analysis on OLMo 3 7B [22] at three stages of its training curriculum: SFT [23], DPO [28] and RLVR [18, 11].

Figure 6 summarizes the results on OLMo 3 7B. All three stages show very similar implicit budgeting under composition, and their token counts shrink at similar rates. Nevertheless, the same reduction affects accuracy differently. At similar token ratios, the final RLVR stage loses substantially less accuracy than the SFT and DPO stages. These degrade by similar amounts and also tend to fail on the same fragile tasks. Correlating their accuracy degradation on GPQA-Diamond, MATH-500 and MMLU-Redux as in § 3.2 gives a disattenuated Pearson ρ\rho of 1.03 ±\pm 0.35. RLVR’s fragile set tends to overlap less with theirs, with ρ=0.69±0.23\rho= 0.69 \pm0.23 against SFT and ρ=0.89±0.26\rho= 0.89 \pm0.26 against DPO.

Figure 6. Effects of implicit budgeting across three post-training stages of OLMo 3 7B on GPQA-Diamond. Left: Token ratio ri(k)=Ti(k)/Ti(1)r_i(k) = T_i(k)/T_i(1) as a function of composition size kk. Right: Accuracy drop as a function of the token ratio.

We conclude that RLVR post-training mitigates the accuracy drop incurred by implicit budgeting under composition, but it does not eliminate the misallocation of the thinking budget in the first place, which remained similar across the stages.

Related work

Previous work has examined both how task composition affects accuracy and reasoning length and how batching can improve efficiency or reliability. Several studies ask whether models can solve tasks as reliably together as they do separately [24, 17, 2]. Related studies examine whether failures to meet multiple response requirements can be explained by independent errors [12, 35]. These studies document accuracy losses under composition, although their causes are not yet fully understood. At the same time, batching methods have shown promise for improving efficiency and reliability [Xu et al., 2026a, Ji et al., 2025], with accuracy gains reported in some settings [34, 43, 20] (consistent with our observations on light-reasoning datasets OneWordQA and StrategyQA, see Appendix § 11.1.6). Batching has also been observed to reduce the number of reasoning tokens used per task [34, 6]. The effects of added context also vary across questions. For factual questions, models tend to resist misleading information more successfully when they can also answer related questions correctly, suggesting that accuracy in isolation does not fully capture robustness [Xu et al., 2026b]. The number of steps needed to solve a task also matters. Tasks requiring more steps show larger accuracy drops when irrelevant information is added [42]. More broadly, researchers distinguish task difficulty from sensitivity to prompt changes [31] and examine how that sensitivity varies with model size, training and architecture [26]. Overall accuracy can remain largely unchanged even when irrelevant context changes which questions a model answers correctly [45]. Estimates of this sensitivity also depend on how answers are graded. Rigid matching rules can treat differently worded correct answers as errors, exaggerating the apparent sensitivity [15]. Alongside these studies, work on reasoning allocation has shown that adding other tasks or irrelevant text can shorten reasoning traces and reduce self-checking [30]. Researchers have also trained models to solve groups of tasks more efficiently under a shared token limit [41]. However, studies of budget allocation find that models often devote more reasoning to earlier questions than to those worth more points [8]. In a single-task setting, model confidence has also been used to guide a search for shorter reasoning paths [16].

These findings can be difficult to reconcile within a common explanation, given the differences in experimental settings, interventions and datasets. Our work offers a unifying perspective on several of these observations. Budget misallocation is consistent with the greater sensitivity of deeper reasoning tasks to irrelevant context [42], self-compression [6], composition errors that cannot be explained by independent error rates alone [12], and the token savings achieved through batching [34, 3]. We identify reasoning trace length in isolation as a key variable that helps connect these findings and informs simple, effective interventions. Using this information, we are able to show how to mitigate degradation under composition, even in frontier models.

Discussion and conclusions

In this work, we show that language models fail on tasks they solve reliably in isolation once those tasks are combined in a single prompt. The resulting accuracy drop is larger than expected from isolated error rates alone. As more tasks are combined, models use fewer reasoning tokens per task, following an approximate power-law decay with composition size, and the largest failures occur on tasks that require longer reasoning in isolation. This reveals an implicit per-task budget that acts as a double-edged sword. Under composition, fragile tasks receive too little reasoning and fail. In isolation, non-fragile tasks receive more reasoning than they need and waste compute. By batching tasks according to their fragility, however, the implicit budgeting can be manipulated to reduce token usage by about 43% on average while preserving accuracy, and since it does not require internal states of the model, it also applies to proprietary models such as Claude Opus 5.

Previous context can cause degradation too. We deliberately compose independent tasks of the same type, isolating the budget effect in its simplest form. An important question is how this effect extends to richer contexts. Prior work finds that reasoning traces also shorten when the same task is embedded in irrelevant context, conversation history, or a larger task [30], suggesting that implicit thinking budgeting may extend beyond same-type composition.

Long-running agents in production. This is particularly relevant to agentic workflows, where tool outputs, interaction histories, and intermediate results continually accumulate. Agents are widely observed to degrade as their context grows, a phenomenon known as context rot [14]. Our results point to a degradation that sets in long before the context fills up. In an agentic workflow, this means a high-effort step can be unknowingly misbudgeted, thus possibly leading to error propagation in subsequent tasks. As a direct practical intervention, agents can keep their context lean and hand reasoning-intensive steps to subagents that start from a fresh context.

Post-training as a potential mitigation. Our results suggest that reasoning allocation contributes to the difficulty of composed tasks, even when the constituent tasks are solvable in isolation. Post-training may therefore help mitigate degradation under composition. For instance, integrating budget management into RL pipelines through explicit training on composed tasks could help models allocate reasoning more effectively across a shared prompt.

Limitations. While we show that composition systematically reduces per-task reasoning and disproportionately harms fragile tasks, we do not yet know what drives this reduction. Our study is limited to independent, same-dataset compositions, and detailed reasoning-length analysis is restricted to open-weight models. Extending these results to mixed-domain, dependent, and proprietary-model settings remains important future work.

Acknowledgments

This project was funded by NSF award CCF-2112665 (TILOS) and an Alexander von Humboldt professorship. Sharut Gupta was supported by a MathWorks Engineering Fellowship. Thies Brockmöller and Chenpan Li are supported by the scholarship of the Studienstiftung des deutschen Volkes, and Chenpan Li is also supported by the scholarship of the Thomas-Weiland Wissenschaftsstiftung.

Appendix

Further results

Compositional degradation is observable over a wide range of datasets and models

Figure 7. Accuracy drop as a function of composition size kk for all datasets and models. All tasks were solved in three out of three isolated attempts, yet accuracy drops once they are composed, on every dataset except OneWordQA and StrategyQA, whose tasks require little reasoning. The R1-distilled models degrade the most, and the extent of degradation varies strongly between datasets.

Compositional degradation is not caused by context length or parsing. Since composed tasks require more context and use a different answer format than isolated tasks, two natural confounders of compositional degradation are truncation due to context overflow and failures in parsing the composed answer. The results in Table 3 show that neither factor accounts for the observed degradation. Context length poses little difficulty: in most datasets, fewer than 1% of responses reach the context limit, even at the largest kk we evaluate. Parsing failures are likewise rare across all datasets except AIME. There, the majority of these failures arise because the model returns fewer answers than there are tasks, which itself constitutes a failure to solve the composition.

DatasetkkResponse at largest kk: tokensResponse at largest kk: of budgetResponses (%): hit limitResponses (%): unparseable
GSM8K202.8k3%0.0 [0]0.2 [2]
MATH-5002019.6k29%0.9 [10]2.4 [32]
AIME1057.6k35%4.8 [39]10.7 [47]
Integrals207.4k17%0.0 [0]1.4 [15]
MMLU-Redux†^{\dagger}201.6k4%0.2 [2]3.9 [44]
GPQA-Diamond108.9k8%0.2 [1]2.3 [20]
OneWordQA200.8k2%0.1 [0]0.1 [1]
StrategyQA201.1k1%0.0 [0]0.0 [0]

Table 3. Context limits and parsing failures in the composed sets. Each response has a token budget equal to the context window minus the prompt, capped at the model’s maximum generation length; a response that exhausts it is truncated. Response tokens are the mean at the largest kk. Truncation and parsing-failure rates are pooled over all models and kk, with the worst model and kk in brackets. †^{\dagger}Most unparseable responses come from R1 Qwen3 8B, which often breaks the answering contract under composition.

Composition size influences token ratio

Figure 8–Figure 14 show the token ratio ri(k)=Ti(k)/Ti(1)r_i(k) = T_i(k)/T_i(1) as a function of composition size kk for all seven open-weight models. Tasks are binned by the quantile of their isolated trace length into Low (≤20%\le20\%), Medium (20–80%), and High (> 80%). On log-log axes, the ratio decreases approximately linearly in log⁡k\log k for nearly all models and datasets, indicating that LLMs shrink the relative token usage of each task under composition by kmk^m. The exception is R1 Qwen 7B on MATH-500, where the ratio of the Low bin rises above 1 and the fit over all tasks is flat (m=−0.04m = -0.04, R2=−0.43R^2 = -0.43). This reduction shows no systematic dependence on Ti(1)T_i(1), since the ordering of the length bins varies across models and datasets and no bin is consistently reduced more than the others. We fix each power-law fit to pass through (1,1)(1, 1), since a composition of a single task has a token ratio of 1 by construction. Each caption summarizes the six fits of a model, one per dataset over all of its tasks, by the median and range of the slope mm and the mean R2R^2.

Figure 8. Token ratio on Qwen3 4B Thinking. Power-law fits through (1,1)(1, 1), one per dataset: median slope m=−0.46m = -0.46 (range [−0.79,−0.34][-0.79, -0.34]), mean R2=0.94R^2 = 0.94.

Figure 14. Token ratio on R1 Qwen3 8B. Power-law fits through (1, 1), one per dataset: median slope m=−0.61m = -0.61 (range [−0.88,−0.46][-0.88, -0.46]), mean R2=0.55R^2 = 0.55.

Six-panel plots of token ratio on Qwen3 8B across Integrals, MMLU-Redux, OneWordQA, GPQA-Diamond, GSM8K, and MATH-500

Figure 9. Token ratio on Qwen3 8B. Power-law fits through (1, 1), one per dataset: median slope m=−0.56m = -0.56 (range [−0.72,−0.45][-0.72, -0.45]), mean R2=0.94R^2 = 0.94.

Figure 10. Token ratio on Qwen3 14B. Power-law fits through (1, 1), one per dataset: median slope m=−0.55m = -0.55 (range [−0.73,−0.42][-0.73, -0.42]), mean R2=0.87R^2 = 0.87.

Figure 11. Token ratio on Qwen3.8 27B. Power-law fits through (1, 1), one per dataset: median slope m=−0.15m = -0.15 (range [−0.26,−0.07][-0.26, -0.07]), mean R2=0.72R^2 = 0.72.

Figure 12. Token ratio on R1 Llama 8B. Power-law fits through (1, 1), one per dataset: median slope m=−0.68m = -0.68 (range [−0.78,−0.35][-0.78, -0.35]), mean R2=0.59R^2 = 0.59.

Figure 13. Token ratio on R1 Qwen 7B. Power-law fits through (1, 1), one per dataset: median slope m=−0.62m = -0.62 (range [−0.78,−0.04][-0.78, -0.04]), mean R2=0.62R^2 = 0.62.

Token ratio influences accuracy degradation

Figure 15–Figure 21 extend the analysis of Figure 5 to further models and datasets. We restrict this analysis to a subset of six datasets, chosen to span a wide range of tasks from math, logical inference and linguistics, as running it on all datasets would be prohibitively expensive.

Figure 15. Accuracy drop versus relative thinking token reduction for Qwen3 4B Thinking.

Figure 21. Accuracy drop versus relative thinking token reduction for R1 Qwen3 8B.

In each plot, tasks are binned by the quantile of their isolated thinking length Ti(1)T_i(1) into Low (≤20%\le20\%), Medium (20–80%), and High (>80%> 80\%), and the accuracy drop is shown as a function of the relative thinking token reduction. Below the dashed line, the y-axis is logarithmic. The pattern holds across all models. Accuracy drops as the reduction grows, and in most cases tasks with long isolated thinking traces degrade substantially more than tasks with short ones.

Figure 16. Accuracy drop versus relative thinking token reduction for Qwen3 8B.

Figure 17. Accuracy drop versus relative thinking token reduction for Qwen3 14B.

Accuracy drop versus relative thinking token reduction for Qwen3.8 27B

Figure 18. Accuracy drop versus relative thinking token reduction for Qwen3.8 27B.

Accuracy drop versus relative thinking token reduction for R1 Llama 8B

Figure 19. Accuracy drop versus relative thinking token reduction for R1 Llama 8B.

Figure 20. Accuracy drop versus relative thinking token reduction for R1 Qwen 7B.

Budget forcing causes a degradation analogous to composition. To rule out possible context effects, we take the budget a task receives when it is solved in composition and apply it to the same task solved in isolation. Following Muennighoff et al. [ [21], we inject </think> once the thinking trace reaches this budget. In composition, the model also spends tokens on restating the question and checking the output format. To cover this constant overhead, we add the full share of unattributed tokens (tokens not attributed to any task) from the composed solution to the budget. Figure 5 and Figure 15–Figure 21 show the results as “forced”. We find that stopping the thinking abruptly causes substantially larger degradation than the planned reduction of thinking tokens under composition. Still, the same pattern emerges. Degradation rises monotonically as the budget shrinks, and tasks with higher reasoning requirements, measured by their thinking length in isolation, consistently degrade more than tasks with low requirements. In terms of its effect on accuracy, composing tasks therefore appears to act analogously to forcing a budget.

Budgeting is slightly position dependent

We also report results on how the implicit budgeting depends on position, where a task’s position in a composition is simply its position in the prompt. Previous work has observed a position bias in thinking tokens [24], so we ask whether this bias also influences the accuracy degradation through composition.

Figure 22 plots the budgeting and accuracy drop by position on MATH-500 across all models. We observe a similar budgeting effect as previously described in the literature: models use fewer thinking tokens towards the end of a composition. Remarkably, as a result, the accuracy drop grows towards the end in most models, in line with our earlier observations.

Figure 22. Positional dependence of thinking budget allocation and compositional degradation on MATH-500 at composition size k=20k = 20. The upper panels show thinking tokens (yellow-red-purple) and the lower panels show the accuracy drop (yellow-green-purple). The last position in each composition holds the dummy task, which we exclude from the analysis because our method cannot reliably determine its token usage (Appendix § 11.2.1).

Qwen3.8 27B does not show a positional bias, and thus its accuracy stays constant across positions, which hints that the training recipe is the main driver of this bias. The original Qwen3 8B and its R1-distilled version point the same way. They share the same architecture, yet R1 Qwen3 8B reduces thinking tokens substantially more at later positions, which again suggests that post-training has a large influence on budgeting behavior. Future work should investigate this phenomenon further, since it can impact performance when high-effort tasks are placed at the end of a prompt.

Correlating accuracy drop across models and composition sizes

Figure 23. Disattenuated Pearson rr correlation of accuracy degradation across different conditions, i.e. model and composition size. Each pair of conditions (a cell in the heatmap) averages over datasets and, on the left, over models, on the right, over composition sizes. Left: The correlation is substantially larger when comparing accuracy degradation within the same model across different composition sizes kk. Right: By contrast, the correlation between different models is low, especially across model families, suggesting different architectures and training recipes lead to different tasks being fragile.

Fragility-aware batching

Table 2 extends Table 1 to GSM8K, Integrals, OneWordQA and StrategyQA. We leave out AIME, where nearly every task degrades beyond the 0.5% budget already at k=3k = 3, so fragility-aware batching falls back to k=1k = 1 throughout. We note that the pattern observed in the selected datasets in the main text also holds for the other datasets we tested. Random batching at k=6k=6 saves many tokens but loses accuracy, most visibly on reasoning-heavy datasets such as GPQA-Diamond, MMLU-Redux and MATH-500. Fragility-aware batching keeps accuracy close to the isolated baseline while still using substantially fewer tokens, and the linear predictor recovers this gain partially without access to measured fragility. Claude Opus 5 appears only on GPQA-Diamond, and to limit cost it is measured with fewer trials than the open-weight models: random batching rests on 5 trials per task instead of 10, the measured-fragility assignment on 5 isolated and 5 composed trials per task instead of 10 each, and the predictor’s labels on 10 isolated trials and 5 per composition size.

Isolated, k=1k=1 Acc.Isolated, k=1k=1 Tok.Random batching, k=6k=6 Acc.Random batching, k=6k=6 Tok.Fragility-aware (measured) Acc.Fragility-aware (measured) Tok.Fragility-aware (predictor) Acc.Fragility-aware (predictor) Tok.
Task accuracy (Acc.) and output tokens (Tok., includes thinking and answer) per task under different batching policies for all models and datasets. The k=1k=1 columns report absolute values. All other columns report absolute values with the change with respect to k=1k=1 in parentheses, as an absolute difference in percentage points for accuracy and as a relative change in % for tokens. Within each policy, the left column is accuracy and the right column is tokens. The Average row reports the mean over all models shown, eight on GPQA-Diamond and seven on the other datasets: absolute values for k=1k=1 and the mean change for all other policies. On OneWordQA, the Average leaves out R1 Qwen3 8B (grey), whose answers in composition do not follow our result contract, which leads to increased parsing errors.
GPQA-Diamond
Qwen3 4B T.66.7%766356.4% (−10.3)2470 (−67.8%)66.8% (+0.2)5200 (−32.1%)67.3% (+0.7)7534 (−1.7%)
Qwen3 8B58.7%597353.4% (−5.3)1950 (−67.4%)60.0% (+1.3)3520 (−41.1%)56.9% (−1.8)5106 (−14.5%)
Qwen3 14B61.5%459751.8% (−9.7)1500 (−67.4%)57.9% (−3.7)2111 (−54.1%)61.3% (−0.2)3665 (−20.3%)
Qwen3.8 27B83.2%430379.2% (−4.0)2740 (−36.3%)84.0% (+0.8)4102 (−4.7%)85.3% (+2.1)4324 (+0.5%)
R1 Qwen 7B49.1%668623.6% (−25.5)1305 (−80.5%)49.6% (+0.5)4037 (−39.6%)46.7% (−2.4)5953 (−11.0%)
R1 Llama 8B47.7%742432.1% (−15.6)1771 (−76.2%)49.6% (+1.9)4526 (−39.0%)48.6% (+0.9)5816 (−21.7%)
R1 Qwen3 8B60.8%908448.1% (−12.6)2398 (−73.6%)61.9% (+1.2)6595 (−27.4%)59.8% (−1.0)8877 (−2.3%)
Claude Opus 589.5%68988.7% (−0.9)564 (−18.2%)89.2% (−0.4)521 (−24.3%)87.6% (−2.0)686 (−0.5%)
Average64.7%5802−10.5−60.9%+0.2−32.8%−0.5−8.9%
GSM8K
Qwen3 4B T.95.2%115191.9% (−3.3)472 (−59.0%)93.4% (−1.8)467 (−59.5%)94.9% (−0.3)793 (−31.2%)
Qwen3 8B95.4%76995.2% (−0.2)371 (−51.8%)95.0% (−0.4)279 (−63.7%)95.1% (−0.3)537 (−30.1%)
Qwen3 14B96.1%58795.9% (−0.2)253 (−56.9%)95.9% (−0.2)200 (−66.0%)95.7% (−0.3)314 (−46.6%)
Qwen3.8 27B97.4%26397.5% (+0.1)200 (−24.0%)97.5% (+0.1)158 (−40.1%)97.5% (0.0)157 (−40.1%)
R1 Qwen 7B91.4%83487.5% (−3.9)187 (−77.6%)90.8% (−0.7)308 (−63.1%)91.6% (+0.1)824 (−1.2%)
R1 Llama 8B89.5%68581.8% (−7.8)278 (−59.4%)89.3% (−0.2)672 (−1.9%)89.4% (−0.1)797 (+16.4%)
R1 Qwen3 8B92.8%237791.8% (−1.0)795 (−66.6%)93.0% (+0.2)911 (−61.7%)92.8% (0.0)2163 (−9.0%)
Average94.0%952−2.3−56.5%−0.4−50.8%−0.1−20.2%
MATH-500
Qwen3 4B T.97.6%406496.4% (−1.1)2148 (−47.2%)97.7% (+0.1)3716 (−8.6%)96.8% (−0.8)3605 (−11.3%)
Qwen3 8B95.9%344995.6% (−0.3)1751 (−49.2%)95.1% (−0.8)2173 (−37.0%)95.9% (+0.1)2775 (−19.5%)
Qwen3 14B96.9%260495.9% (−1.0)1375 (−47.2%)96.6% (−0.2)2454 (−5.8%)96.6% (−0.2)2444 (−6.2%)
Qwen3.8 27B98.1%101496.7% (−1.5)748 (−26.2%)97.8% (−0.4)943 (−7.1%)97.2% (−0.9)994 (−2.0%)
R1 Qwen 7B93.0%238368.9% (−23.2)1658 (−30.4%)93.2% (+0.2)2205 (−7.5%)93.4% (+0.4)2381 (0.0%)
R1 Llama 8B88.7%301979.4% (−9.3)1569 (−48.0%)88.5% (−0.2)2948 (−2.3%)88.8% (+0.2)3333 (+10.4%)
R1 Qwen3 8B96.8%676793.0% (−3.8)3705 (−45.2%)96.6% (−0.2)6608 (−2.4%)96.7% (−0.2)6657 (−1.6%)
Average95.3%3329−5.7−41.9%−0.2−10.1%−0.2−4.3%
Integrals
Qwen3 4B T.99.9%191499.2% (−0.7)1029 (−46.3%)99.7% (−0.2)1411 (−26.3%)99.6% (−0.3)908 (−52.6%)
Qwen3 8B99.7%288798.8% (−0.9)946 (−67.2%)99.8% (+0.1)2717 (−5.9%)98.8% (−0.9)1192 (−58.7%)
Qwen3 14B99.8%232199.0% (−0.8)670 (−71.1%)99.5% (−0.3)1335 (−42.5%)99.6% (−0.2)756 (−67.4%)
Qwen3.8 27B99.8%72498.8% (−1.0)564 (−22.1%)99.5% (−0.3)624 (−13.8%)99.6% (−0.2)550 (−24.0%)
R1 Qwen 7B96.4%251990.8% (−5.5)547 (−78.3%)96.0% (−0.4)2277 (−9.6%)96.3% (0.0)2291 (−9.0%)
R1 Llama 8B93.7%246286.7% (−7.0)613 (−75.1%)93.1% (−0.6)2161 (−12.2%)93.6% (−0.1)1926 (−21.8%)
R1 Qwen3 8B98.5%408296.7% (−1.8)1339 (−67.2%)97.9% (−0.6)4344 (+6.4%)98.5% (0.0)2701 (−33.8%)
Average98.3%2416−2.5−61.0%−0.3−14.8%−0.3−38.2%
OneWordQA
Qwen3 4B T.92.0%66892.1% (+0.1)383 (−42.6%)91.7% (−0.3)179 (−73.2%)91.8% (−0.2)185 (−72.4%)
Qwen3 8B91.7%21292.0% (+0.3)71 (−66.7%)91.2% (−0.4)46 (−78.2%)92.2% (+0.5)46 (−78.1%)
Qwen3 14B92.4%20792.8% (+0.4)73 (−64.6%)93.2% (+0.7)48 (−76.6%)93.3% (+0.9)47 (−77.1%)
Qwen3.8 27B93.3%7194.3% (+1.0)40 (−43.8%)93.4% (+0.2)36 (−50.0%)93.4% (+0.2)35 (−51.3%)
R1 Qwen 7B72.0%45476.5% (+4.4)105 (−76.8%)76.5% (+4.4)98 (−78.3%)77.1% (+5.0)89 (−80.4%)
R1 Llama 8B78.0%48484.0% (+6.0)86 (−82.2%)85.3% (+5.5)74 (−84.6%)84.0% (+6.0)81 (−83.3%)
R1 Qwen3 8B81.5%32828.6% (−52.9)136 (−58.6%)82.1% (+0.6)354 (+7.8%)82.3% (+0.8)322 (−1.7%)
Average86.6%349+1.9−62.8%+1.8−73.5%+2.1−73.8%
StrategyQA
Qwen3 4B T.63.2%83667.7% (+4.5)509 (−39.1%)68.2% (+5.0)248 (−70.4%)67.4% (+4.1)246 (−70.6%)
Qwen3 8B70.1%39472.1% (+1.9)96 (−75.6%)71.8% (+1.7)57 (−85.4%)70.8% (+0.7)52 (−86.8%)
Qwen3 14B70.0%36673.2% (+3.3)84 (−77.1%)73.5% (+3.5)60 (−83.6%)73.4% (+3.4)54 (−85.3%)
Qwen3.8 27B79.4%22480.2% (+0.8)151 (−32.4%)81.8% (+2.4)145 (−35.0%)81.0% (+1.6)144 (−35.5%)
R1 Qwen 7B56.9%65158.9% (+2.0)84 (−87.1%)58.5% (+1.6)57 (−91.2%)57.8% (+0.8)47 (−92.8%)
R1 Llama 8B58.9%44665.6% (+6.7)81 (−81.9%)65.2% (+6.3)95 (−78.7%)64.6% (+5.6)51 (−88.5%)
R1 Qwen3 8B70.0%60667.7% (−2.3)89 (−85.3%)71.1% (+1.1)425 (−29.9%)69.9% (−0.1)629 (+3.8%)
Average66.9%503+2.4−68.4%+3.1−67.7%+2.3−65.1%
MMLU-Redux
Qwen3 4B T.84.7%194482.9% (−1.8)527 (−72.9%)85.3% (+0.6)527 (−72.9%)84.9% (+0.2)1663 (−14.5%)
Qwen3 8B87.2%92184.0% (−3.3)306 (−66.7%)86.1% (−1.1)285 (−69.0%)86.9% (−0.3)922 (+0.2%)
Qwen3 14B90.0%71885.8% (−3.3)212 (−70.5%)88.2% (−1.9)319 (−55.6%)88.3% (−1.7)678 (−5.5%)
Qwen3.8 27B92.4%48891.9% (−0.5)262 (−46.4%)91.8% (−0.6)196 (−59.8%)92.1% (−0.3)352 (−27.9%)
R1 Qwen 7B70.1%107960.6% (−9.5)397 (−63.2%)69.7% (−0.3)601 (−44.3%)70.4% (+0.3)1127 (+4.5%)
R1 Llama 8B75.5%145858.6% (−16.9)431 (−70.4%)75.9% (+0.3)896 (−38.5%)75.3% (−0.2)1430 (−1.9%)
R1 Qwen3 8B81.6%171164.5% (−17.1)279 (−83.7%)82.1% (+0.5)1273 (−25.6%)82.2% (+0.6)1714 (+0.2%)
Average82.9%1188−7.5−67.7%−0.2−52.2%−0.1−6.4%

Table 2. Task accuracy (Acc.) and output tokens (Tok., includes thinking and answer) per task under different batching policies for all models and datasets. The k=1k=1 columns report absolute values. All other columns report absolute values with the change with respect to k=1k=1 in parentheses, as an absolute difference in percentage points for accuracy and as a relative change in % for tokens. Within each policy, the left column is accuracy and the right column is tokens. The Average row reports the mean over all models shown, eight on GPQA-Diamond and seven on the other datasets: absolute values for k=1k=1 and the mean change for all other policies. On OneWordQA, the Average leaves out R1 Qwen3 8B (grey), whose answers in composition do not follow our result contract, which leads to increased parsing errors.

R1 Qwen3 8B is the one model that collapses on OneWordQA under random batching, from 81.5% to 28.6%. Contrary to asking questions in isolation, in composition R1 Qwen3 8B often fails to adhere to the answering contract under composition, leading to parsing errors.

When testing on the Integrals dataset, fragility-aware batching with held-out predictions from the linear predictor outperforms the measured variant. This is because even comparably fragile integrals lose very little accuracy, so estimates from a limited number of observations are noisy. The trained predictor also has access to the structure of each integral, which makes it more robust to this measurement noise.

On OneWordQA and StrategyQA, batching even raises accuracy for most models, by up to 6.7 points (R1 Llama 8B on StrategyQA under random batching). We observe that both datasets need little reasoning. OneWordQA only asks the model to recall a fact, and StrategyQA to chain a few commonsense inferences. On such tasks, models seem to think more than the task needs, which can degrade performance too, consistent with Srivastava et al. [ [34].

Methodological details

Measuring per-task thinking length

To measure how a model divides its thinking across the tasks of a composition, we prefix each task with a unique tag (Q01, Q02, . . . ) and instruct the model to name the tag whenever it turns to that task. The tags mark where the reasoning on one task ends and the next begins. We assign every token after a tag to the corresponding task, up to the next tag or the end of the thinking trace. The thinking length Ti(k)T_i(k) of task tit_i in a composition of size kk is the total number of tokens assigned to it, and the model may return to a task more than once. A task that is never named receives no tokens.

<think> Q01:  The dye emits light with an energy of 2.34 eV ... maybe A.
Q02:  A dual catalyst system for ethylene polymerization ... tentatively D.
Back to Q01: 2.34 eV is about 530 nm, so the absorbed light ... B.
Q03:  ... </think>

Here the tokens after both Q01 tags count towards t1t_1, while t2t_2 receives the tokens between its tag and the second Q01.

This rule cannot reliably find where the trace for the last question in a composition ends, since no next tag marks its end, so its length is often overestimated. To fix this, we append an ungraded dummy question at the end of each composition.

In our experiments, this tagging strategy works quite well: LLMs name every task in 99.5% of all tagged thinking traces. GPQA-Diamond is the exception at 92.9%, mostly because R1 Qwen 7B names every task in only 61.5% of its traces at k≥3k \ge3.

Sampling parameters and context length

Sampling parameters and context length

Every open-weight model is decoded with the same settings: temperature 0.6 and top-p 0.95, with no top-k cutoff and no repetition penalty. These are the values Qwen recommends for its models in thinking mode [40], and the values DeepSeek recommends for the R1-distilled models. Each trial is an independent sample; we set no per-request seed. The context window is shared by the prompt and the model’s response, including its thinking. It is 131,072 tokens for Qwen3 8B, Qwen3 14B, Qwen3.8 27B and the three R1-distilled models, and 262,144 for Qwen3 4B Thinking. Claude Opus 5 is queried through the OpenRouter API with reasoning effort set to medium, a 200,000-token context window and at most 128,000 output tokens, thinking included. With these limits, fewer than 1% of responses reach the context limit in most datasets (Table 3).

Correlating accuracy degradation across conditions

Conditions and degradation variables. We define a condition as a tuple c=(m,k)c = (m, k) of a model mm and a composition size kk, e.g. (Qwen3 8B,k=20)(\mathrm{Qwen3\ 8B}, k = 20). For a condition A=(mA,kA)A = (m_A, k_A) we write ΔA\Delta_A for the random variable that assigns to each task tit_i its accuracy degradation ΔimA(kA)\Delta_i^{m_A}(k_A) under that condition, and analogously ΔB\Delta_B for the second condition B=(mB,kB)B = (m_B, k_B) that we correlate against. All correlations below are computed over all tasks of a dataset ti∈Dt_i \in\mathcal{D}.

Correlation and disattenuation. For each pair of conditions we compute the Pearson correlation rr between ΔA\Delta_A and ΔB\Delta_B over all tasks of a dataset. Because ΔA\Delta_A and ΔB\Delta_B are estimated from a finite number of trials, measurement noise attenuates rr toward zero. We therefore apply Spearman’s correction for attenuation to every pair of conditions separately,

r∗=rrAArBB(4)r^* = \frac{r}{\sqrt{r_{AA}r_{BB}}} \tag*{(4)}

where the reliability rAAr_{AA} is the test-retest correlation of ΔA\Delta_A, that is, the correlation between the accuracy drops at condition AA measured in the first and in the second pass. We then average r∗r^* arithmetically over all pairs of conditions and datasets that enter a reported value. We do not clip r∗r^*.

Avoiding shared measurement noise. When AA and BB share a model, both drops subtract the same isolated accuracy, so its estimation error would inflate their correlation. We therefore measure every task in two independent passes and take ΔA\Delta_A from one pass and ΔB\Delta_B from the other.

Confidence intervals. We report half the width of a 95% percentile interval from a bootstrap over tasks. In each of 5000 replicates we resample the tasks of a dataset with replacement at the original size and recompute every correlation and every reliability on the resampled tasks.

Implementation details of the fragility-aware batching algorithm

Estimating E[Δi(k)]\mathbb{E}[\Delta_i(k)]. For the fragility-aware batching intervention in § 5.1, each task has 10 isolated trials and 10 trials at every composition size. We split these trials at random into two halves, A and B. For each kk, an isotonic map from the drop measured on half A to the drop measured on half B gives E[Δi(k)]\mathbb{E}[\Delta_i(k)].

Estimating cost⁡k\operatorname{cost}_k. The ratio cost⁡k\operatorname{cost}_k does not depend on the individual task and is estimated once per model and dataset as the mean number of output tokens attributable to a single task inside a composition of size kk, divided by the mean number of output tokens used for isolated solving:

cost⁡k=1kT‾(k)T‾(1)(5)\operatorname{cost}_k = \frac{\frac{1}{k}\overline{T}(k)}{\overline{T}(1)} \tag*{(5)}

where T‾(k)\overline{T}(k) is the mean total output length of a composition of size kk. In each experiment, we estimate it over the entire dataset.

Selecting λ∗\lambda^* by repeated cross-fitting. Per-task accuracy estimates are noisy and our pool of tasks is limited, so a λ∗\lambda^* selected on the same data used to estimate E[Δi(k)]\mathbb{E}[\Delta_i(k)] would be optimistically biased. We therefore split the task pool into 5 outer folds and determine λ∗\lambda^* for each fold only on its training part.

Within a training slice, we split the tasks into 5 blocks, four times under different random shuffles, which gives 20 held-out blocks. For each block, we fit the isotonic maps on the remaining tasks, apply them to the block and assign every task in it a composition size by Equation (2). We do this for a grid of 121 exchange rates (λ=0\lambda= 0 together with 120 values spaced logarithmically over [10−4,3.0][10^{-4}, 3.0]) and record the accuracy and token ratio realized on half B.

Averaging the 20 blocks results in one accuracy curve over λ\lambda, which we make non-increasing by isotonic regression, since a larger exchange rate can only move tasks to larger composition sizes. The term SE⁡(Δ‾(λ))\operatorname{SE}(\overline{\Delta}(\lambda)) in Equation (3) is the standard error of the realized accuracy across the 20 blocks. We select the cheapest rate whose accuracy, minus one standard error, stays above acc⁡(1)−B\operatorname{acc}(1) - B, where acc⁡(1)\operatorname{acc}(1) is the mean isolated accuracy. If no rate clears this threshold, we take the rate with the highest accuracy minus one standard error.

Finally, we fit the isotonic maps on the whole training slice, apply them to the accuracy drops measured on half A of the held-out fold and assign each of its tasks a composition size at λ∗\lambda^*. Only these held-out assignments enter the reported results.

Forming compositions. The assignment yields a per-task upper bound ki(λ∗)k_i(\lambda^*) on the size of the composition that task may be placed in, and compositions must be formed such that ∣C∣≤min⁡ti∈Cki(λ∗)|C| \le\min_{t_i \in C} k_i(\lambda^*) for every composition CC. Placing a task in a composition smaller than its assigned size is admissible, since its expected drop is then no larger, so the realized mean drop should not exceed the planned one. We use the greedy procedure in Algorithm 1: proceeding from the smallest assigned size upwards, we form full compositions from the tasks assigned to the current size and fill any remaining partial composition with tasks assigned to strictly larger sizes, which are by construction more robust and whose bound is therefore not violated. Filler tasks come from the nearest larger size first and are drawn at random within it. Compositions are formed separately within each outer test fold, so tasks from different folds never share a prompt.

If fewer than k−∣P∣k - |P| tasks with a larger assigned size remain, the leftover tasks form a composition smaller than kk. Since filler tasks are consumed from the pools of larger assigned sizes, the realized mean composition size is slightly below the assigned one, so the reported token savings are a conservative estimate of what the assignment permits.

Algorithm 1 Forming compositions from a size assignment

Require: Tasks t1,…,tnt_1,\ldots,t_n with assigned sizes ki∈Kk_i \in\mathcal{K} Ensure: A set of compositions B\mathcal{B}

  1. B←∅\mathcal{B} \leftarrow\varnothing; U←{t1,…,tn}U \leftarrow\{t_1,\ldots,t_n\}

  2. for k∈Kk \in\mathcal{K} in increasing order do

  3. P←{ti∈U:ki=k}P \leftarrow\{t_i \in U : k_i = k\}

  4. while ∣P∣≥k|P| \geq k do

  5. remove kk randomly chosen tasks from PP and from UU, and add them as composition B←B∪{(t1,…,tk)}\mathcal{B} \leftarrow\mathcal{B} \cup\{(t_1,\ldots,t_k)\}

  6. end while

  7. if P≠∅P \neq\varnothing then

  8. F←F \leftarrow up to k−∣P∣k - |P| tasks from {ti∈U:ki>k}\{t_i \in U : k_i > k\}, nearest larger size first, at random within a size

  9. remove P∪FP \cup F from UU and add P∪FP \cup F as a composition to B\mathcal{B}

  10. end if

  11. end for

  12. return B\mathcal{B}

Predicting fragility from the task

Architecture. The predictor takes the task description and embeds it with Qwen3 Embedding 8B using last-token pooling. A linear head maps the embedding to a single logit ℓi(k1)\ell_i(k_1), the logit for the smallest composition size k1=3k_1 = 3. The logits for the remaining sizes are obtained by accumulating non-negative offsets along increasing kk,

ℓi(k1)=w⊤ei+b,ℓi(kj)=ℓi(kj−1)−softplus⁡(θj),j=2,…,J,(6)\ell_i(k_1) = w^\top e_i + b,\qquad\ell_i(k_j) = \ell_i(k_{j-1}) - \operatorname{softplus}(\theta_j),\qquad j = 2,\ldots,J, \tag*{(6)}

where eie_i is the embedding of task tit_i, k1<⋯<kJk_1 < \cdots< k_J are the composition sizes in K∖{1}={3,6,10,20}\mathcal{K} \setminus\{1\} = \{3,6,10,20\} ({3,6,10}\{3,6,10\} for GPQA-Diamond), and θj\theta_j are trainable scalars shared across all tasks. The softplus constrains every offset to be non-negative, so the predicted probability is non-increasing in kk by construction, matching the behavior established in § 3. Accuracy measurements under composition are expensive, so the predictor has to learn from little data. We therefore use a linear head, which has few parameters and builds monotonicity in kk into its architecture.

Training. For every task and every k∈K∖{1}k \in\mathcal{K} \setminus\{1\} we form a binary target indicating that the task performs no worse under composition than in isolation, using the 10 composed and 10 isolated trials measured for that task. The head is trained with a class-balanced binary cross-entropy loss averaged over all tasks and composition sizes, so a single coefficient vector is shared across the ladder and the sizes differ only through the learned offsets. We optimize with AdamW at a learning rate of 3×10−33 \times10^{-3}, weight decay 1.0 and a cosine schedule for 2000 steps with batch size 64 and gradient clipping at 1.0. Every 100 steps we evaluate the head on its inner held-out fold and keep the best checkpoint.

Nested cross-fitting. The predictor and the exchange rate λ∗\lambda^* must both be estimated without access to the data on which the intervention is evaluated, so we nest the fitting of the head inside the cross-fitting scheme of Appendix § 11.2.4.

We split the dataset into 5 outer folds. For each outer fold, the 4/5 training slice is split again into 4 inner folds; one head is trained on each inner training split and predicts on the corresponding inner test split. Concatenating these predictions yields an out-of-training prediction for every task in the outer training slice, which takes the place of the half-AA accuracy drops used in Appendix § 11.2.4. The isotonic maps then run from this prediction to the accuracy drop measured on all 10 trials. On the held-out fold, we average the predicted accuracy drops of the four heads, and the remainder of the intervention is unchanged.

Further details on datasets

Indefinite integrals. We generate the integral pool ourselves, to ensure that every problem is known to be solvable and simple enough for most small open-weight models to solve confidently. It holds 1,000 atomic integrands, 200 from each of five function classes:

  • polynomial: a xna\,x^n;

  • exponential: a ebxa\,e^{bx};

  • trigonometric: a f(bx+c)a\,f(bx+c) with f∈{sin⁡,cos⁡,tan⁡}f \in\{\sin,\cos,\tan\};

  • logarithmic and rational: aln⁡(bx+c)a\ln(bx+c) or a/(bx+c)a/(bx+c);

  • integration by parts: a x f(bx+c)a\,x\,f(bx+c) with f∈{exp⁡,sin⁡,cos⁡,ln⁡,arctan⁡}f \in\{\exp,\sin,\cos,\ln,\arctan\}.

The coefficient aa is a nonzero integer in [−5,5][-5,5]. The exponent nn and the frequency bb are nonzero integers in [−15,15][-15,15], and the shift cc is an integer in [−15,15][-15,15]. All parameters are drawn with a fixed seed. Structural duplicates are discarded.

An integrand is kept only if SymPy finds an elementary antiderivative for it, and only if that antiderivative passes the same checker that grades the models. The checker works symbolically rather than by string matching. It parses the boxed LaTeX answer, differentiates it, and tests whether the derivative equals the integrand, first structurally, then numerically at random points, and finally with a symbolic proof under a timeout. An additive constant is accepted, but any other free symbol is rejected. Answers that cannot be parsed, or whose check times out, are recorded separately rather than counted as wrong.

OneWordQA. OneWordQA is a set of 1,548 questions that we generated using Claude Opus 5 and checked manually. They cover 23 categories of simple factual lookups whose answers leave little room for interpretation. Examples include country capitals, dialing and ISO codes, chemical symbols and atomic numbers, Roman numerals, polygon sides and the number of players per team in a sport. Each answer is a single word, a number or a short code. Grading normalizes only how an answer is written (case, diacritics and surrounding punctuation). Where a question has more than one correct answer, the alternatives are listed explicitly in the dataset.

Benchmarks. The six public benchmarks are loaded from the Hugging Face Hub at pinned revisions. We use the error-free questions of MMLU-Redux, drawing the same number from each of 19 subjects (74 per subject, 1,406 questions), and keep the answer options in their published order. GPQA-Diamond contributes 196 of its 198 questions; we drop the two that repeat an answer option. From GSM8K we take the 1,319 problems of the test split, from MATH-500 all 500 problems, and from AIME the 60 problems of 2024 and 2025. StrategyQA contributes the 687 questions of its test split, and an answer counts as correct when it is the gold yes or no.

Prompts. We now list the prompt every model receives, for each of the eight datasets used in the main experiments: indefinite integrals, MMLU-Redux, GPQA-Diamond, OneWordQA, GSM8K, MATH-500, AIME and StrategyQA.

Conventions shared by all datasets:

  • An atomic task is treated as a composition of size k=1k=1 and uses the same template as a kk-task composition, with the plural wording made singular.

  • The prompt is sent as a single user message through the model’s chat template, with no few-shot examples and no system message.

  • Models must answer in the order the tasks are shown, which is the order in which we grade them.

Indefinite integrals

A composition of kk atomic integrands is shown as one integral whose integrand is their sum, rendered in LaTeX by SymPy. ⟨integral⟩\langle\mathit{integral}\rangle is that LaTeX string, for example:

\int 3 x^{2} + 2 \sin{\left(4 x + 1 \right)} \, dx.

The prompt shows the whole sum unchanged, but the model reports one antiderivative per term. The terms are listed in the order in which they appear in the rendered integral. Answer ii is checked symbolically against term ii.

Indefinite integrals

 1 Find an antiderivative for each term of the following indefinite integral: ⟨integral⟩.

 2
 3 The integrand is a sum of ⟨k⟩ terms, listed below in the order they appear in the integral:
 4 1. \int ⟨term 1⟩ \, dx
 5 2. \int ⟨term 2⟩ \, dx
 6 ...
 7 ⟨k⟩. \int ⟨term k⟩ \, dx
 8 Do not add the antiderivatives together, and do not give an antiderivative of the whole sum. Solve each
   term on its own and report ⟨k⟩ separate antiderivatives, one per term, in the order above.

 9
10 Give symbolic answers only in standard LaTeX format. Do not give a decimal approximation. The constant
   of integration may be omitted.

11 Return your answers in the following format in the last ⟨k⟩ lines, with no additional text: one line per
   term, in the order above, each line holding exactly one \boxed{}.

12
13 1. \boxed{<antiderivative of term 1>}
14 2. \boxed{<antiderivative of term 2>}
15 ...
16 ⟨k⟩. \boxed{<antiderivative of term ⟨k⟩>}

17
18 Before you finish, re-read your answer block and check that it holds exactly ⟨k⟩ lines, each with
   exactly one \boxed{}, in the order the terms are listed above.

MMLU-Redux and GPQA (multiple choice)

Both datasets use the same multiple-choice layout. Every question is followed by its lettered options, and consecutive questions are separated by a blank line:

Question block (⟨question block⟩) for both datasets

1 1. ⟨question 1⟩
2 A. ⟨option A⟩
3 B. ⟨option B⟩
4 C. ⟨option C⟩
5 D. ⟨option D⟩

7 2. ⟨question 2⟩
8 A. ...

In GPQA, the four options of each question are shuffled once, with a fixed seed, when the dataset is built, so the correct answer does not always sit in the same position. ⟨example⟩\langle\mathit{example}\rangle is the letters offered, cycled to length kk (for example A, B, C, D, A, B for k=6k = 6). For k=1k = 1 it is A.

MMLU-Redux (k>1k > 1)

Answer the following multiple-choice questions.

⟨question block⟩\langle\textit{question block}\rangle

For each question, give only the letter of the single best option.

End your response with a line of the following form, and write nothing after it: a comma-separated list of ⟨k⟩\langle k\rangle letters, one per question, in the order the questions were asked.

Answer: ⟨example⟩\langle\textit{example}\rangle

The GPQA prompt is the MMLU-Redux prompt word for word, with one extra closing sentence. GPQA questions are long and hard, and its thinking traces are several times longer than those on MMLU-Redux. After such long reasoning, models often lose the answer format they were given at the start, so we ask them to re-check their answer line. The sentence concerns only the format of the answer line, so it does not change what the two prompts ask about the questions:

GPQA: closing sentence appended to the MMLU-Redux prompt

Before you finish, re-read that final line and check that it holds exactly ⟨k⟩\langle k\rangle letters, in the order the questions were asked, with nothing written after them.

For k=1k = 1 this sentence reads “. . . holds exactly 1 letter, with nothing written after it.”

OneWordQA and StrategyQA

Both datasets use the same layout. Each question sits on a single line, numbered, and consecutive questions are separated by a single newline (⟨question block⟩\langle\textit{question block}\rangle). The answer is a numbered block of kk lines under the heading Answer:.

OneWordQA (k>1k > 1)

Answer the following questions. Each answer is short -- usually a single word or a number.

⟨question block⟩\langle\textit{question block}\rangle

Give the answer only: no explanation, and no units unless the question asks for them.

End your response with an answer block: the line "Answer:" followed by exactly ⟨k⟩\langle k\rangle lines, one answer per line, numbered, in the order the questions were asked. Write nothing after the block.

Answer:

  1. ⟨your answer to question 1⟩\langle\textit{your answer to question 1}\rangle

  2. ⟨your answer to question 2⟩\langle\textit{your answer to question 2}\rangle ... ⟨k⟩\langle k\rangle. ⟨your answer to question k⟩\langle\textit{your answer to question }k\rangle

Before you finish, re-read your answer block and check that it holds exactly ⟨k⟩\langle k\rangle answer lines, in the order the questions were asked, with nothing written after them.

The StrategyQA prompt is the OneWordQA prompt word for word, apart from two sentences that restrict the answer to yes or no:

StrategyQA: the two sentences that differ from OneWordQA

Answer the following questions. Each answer is either yes or no.

...

Give the answer only, as the single word yes or no: no explanation, and no answer other than those two words.

The instruction is not enforced strictly. Case and punctuation are ignored, and true and false are accepted alongside yes and no. For k=1k = 1 the first sentence reads “Answer the following question.”

GSM8K, MATH-500 and AIME (short mathematical answers)

The three datasets use the same answer contract as OneWordQA: a numbered block of kk lines under the heading Answer:. The difference is in the problems themselves, which are much longer (several sentences, displayed equations, and for 42 MATH-500 problems an Asymptote figure). Consecutive problems are therefore separated by a blank line rather than a single newline. Blank lines inside a problem are collapsed when the pool is built, so a blank line in ⟨problemblock⟩\langle problem block \rangle always marks the start of the next problem. For k=1k = 1 the answer block has the single line 1. <your answer> (for MATH-500, 1. \boxed{<your answer>}), and the closing sentence reads “. . . holds exactly one answer line, with nothing written after it.” The three prompts differ only in their first line and in the sentence that describes the answer format.

GSM8K (k>1k > 1)

Solve the following problems. Each answer is a single number.

⟨problem block⟩

Give the final answer to each problem as a number only: no units, no currency symbol, no thousands
    separator and no explanation.
End your response with an answer block: the line "Answer:" followed by exactly ⟨k⟩ lines, one answer per
    line, numbered, in the order the problems were asked. Write nothing after the block.

Answer:
1. <your answer to problem 1>
2. <your answer to problem 2>
...
⟨k⟩. <your answer to problem ⟨k⟩>
Before you finish, re-read your answer block and check that it holds exactly ⟨k⟩ answer lines, in the
    order the problems were asked, with nothing written after them.

In MATH-500 each answer is boxed. Many answers contain spaces or commas, such as tuples, intervals, or 10, ⁣08010,\!080, so the box marks exactly where an answer begins and ends. The instruction “in the form the problem asks for” matters because many MATH-500 problems set their own answer format (for example “Express your answer as a common fraction”), and the reference answer follows that format.

MATH-500 ($k > 1)

Solve the following problems.

⟨problem block⟩

Give the final answer to each problem only, with no explanation. Write each answer in LaTeX, in the form
    the problem asks for, wrapped in \boxed{...}.
End your response with an answer block: the line "Answer:" followed by exactly ⟨k⟩ lines, one answer per
    line, numbered, in the order the problems were asked. Write nothing after the block.

Answer:
1. \boxed{<your answer to problem 1>}
2. \boxed{<your answer to problem 2>}
...
⟨k⟩. \boxed{<your answer to problem ⟨k⟩>}
Before you finish, re-read your answer block and check that it holds exactly ⟨k⟩ answer lines, in the
    order the problems were asked, with nothing written after them.

AIME (k>1k > 1)

Solve the following competition problems. Each answer is an integer between 0 and 999.

⟨problem block⟩

Give the final answer to each problem as an integer between 0 and 999: digits only, no units, no
thousands separator and no explanation.
End your response with an answer block: the line "Answer:" followed by exactly ⟨k⟩ lines, one answer per line, numbered, in the order the problems were asked. Write nothing after the block.
Answer:
1. <your answer to problem 1>
2. <your answer to problem 2>
...
⟨k⟩. <your answer to problem ⟨k⟩>
Before you finish, re-read your answer block and check that it holds exactly ⟨k⟩ answer lines, in the order the problems were asked, with nothing written after them.

For k≤3k \le3 the example answer block in all three prompts lists every line, with no ellipsis.

References

  1. [1]G. C. Cawley and N. L. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. J. Mach. Learn. Res., 11:2079–2107, Aug. 2010. ISSN 1532-4435.
  2. [2]J. Chen, M. T. Pilehvar, and J. Camacho-Collados. Understanding llm performance degradation in multi-instance processing: The roles of instance count and context length, 2026. URL https://arxiv.org/abs/2603.22608.
  3. [3]Z. Cheng, J. Kasai, and T. Yu. Batch prompting: Efficient inference with large language model apis, 2023. URL https://arxiv.org/abs/2301.08721.
  4. [4]V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins. Double/de-biased machine learning for treatment and causal parameters, 2024. URL https://arxiv.org/abs/1608.00060.
  5. [5]K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.
  6. [6]J. Deng, S. Liang, J. Li, H. Li, and Y. Xie. ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure. In International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2602.01472.
  7. [7]N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y. Lin, P. West, C. Bhagavatula, R. L. Bras, J. D. Hwang, S. Sanyal, S. Welleck, X. Ren, A. Ettinger, Z. Harchaoui, and Y. Choi. Faith and fate: Limits of transformers on compositionality, 2023. URL https://arxiv.org/abs/2305.18654.
  8. [8]C. Fan, Y. Cheng, M. Li, Y. Liang, T. Zhou, and S. Feizi. Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions, 2026. URL https://arxiv.org/abs/2608.07968.
  9. [9]A. P. Gema, J. O. J. Leang, G. Hong, A. Devoto, A. C. M. Mancino, R. Saxena, X. He, Y. Zhao, X. Du, M. R. G. Madani, C. Barale, R. McHardy, J. Harris, J. Kaddour, E. van Krieken, and P. Minervini. Are we done with mmlu?, 2025. URL https://arxiv.org/abs/2406.04127.
  10. [10]M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies, 2021. URL https://arxiv.org/abs/2101.02235.
  11. [11]D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Ding, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Chen, J. Yuan, J. Tu, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. You, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Zhou, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. Li, S. Zhou, S. Wu, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zhou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Yang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, Sept. 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http://dx.doi.org/10.1038/s41586-025-09422-z.
  12. [12]K. Harada, Y. Yamazaki, M. Taniguchi, E. Marrese-Taylor, T. Kojima, Y. Iwasawa, and Y. Matsuo. When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 16506–16526, 2025. doi: 10.18653/v1/2025.findings-emnlp.896. URL https://aclanthology.org/2025.findings-emnlp.896/.DOI
  13. [13]D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874.
  14. [14]K. Hong, A. Troynikov, and J. Huber. Context rot: How increasing input tokens impacts llm performance. Technical report, Chroma, July 2025. URL https://trychroma.com/research/context-rot.
  15. [15]A. Hua, K. Tang, C. Gu, J. Gu, E. Wong, and Y. Qin. Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19889–19899, 2025. doi: 10.18653/v1/2025.emnlp-main.1006. URL https://aclanthology.org/2025.emnlp-main.1006/.DOI
  16. [16]Z. Huang, X. Xia, Y. Ren, J. Zheng, X. Wang, Z. Zhang, H. Xie, S. Liang, Z. Chen, X. Xiao, F. Zhuang, J. Li, D. Wang, and Y. Ban. Does Your Reasoning Model Implicitly Know When to Stop Thinking? In International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2602.08354.
  17. [17]Z. Ji, X. Wang, Z. Luo, Z. Xie, and M. Zhang. Optimized Batch Prompting for Cost-effective LLMs. Proceedings of the VLDB Endowment, 18(7):2172–2184, 2025. doi: 10.14778/3734839.3734853. URL https://www.vldb.org/pvldb/vol18/p2172-zhang.pdf.DOI
  18. [18]N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, G. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2025. URL https://arxiv.org/abs/2411.15124.
  19. [19]H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step, 2023. URL https://arxiv.org/abs/2305.20050.
  20. [20]J. Lin, M. Diesendruck, L. Du, and R. Abraham. Batchprompt: Accomplish more with less, 2024. URL https://arxiv.org/abs/2309.00384.
  21. [21]N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. Hashimoto. s1: Simple test-time scaling, 2025. URL https://arxiv.org/abs/2501.19393.
  22. [22]T. Olmo, ;, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. L. Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi. Olmo 3, 2026. URL https://arxiv.org/abs/2512.13961.
  23. [23]L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback, 2022. URL https://arxiv.org/abs/2203.02155.
  24. [24]Z. Pan, Q. Pei, Y. Li, Q. Sun, Z. Tang, H. V. Zhao, C. He, and L. Wu. Rest: Stress testing large reasoning models by asking multiple problems at once, 2025. URL https://arxiv.org/abs/2507.10541.
  25. [25]O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis. Measuring and Narrowing the Compositionality Gap in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, 2023. doi: 10.18653/v1/2023.findings-emnlp.378. URL https://aclanthology.org/2023.findings-emnlp.378/.DOI
  26. [26]R. Qin, Q. Wang, T. Wang, Z. Wei, and W. Shen. Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions. In International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2608.18539.
  27. [27]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, August 2026. URL https://qwen.ai/blog?id=qwen3.8.
  28. [28]R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL https://arxiv.org/abs/2305.18290.
  29. [29]D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022.
  30. [30]G. Rodionov, R. Garipov, and G. Yakushev. Reasoning Shift: How Context Silently Shortens LLM Reasoning, 2026. URL https://arxiv.org/abs/2604.01161.
  31. [31]A. Romanou, M. Ibrahim, C. Ross, C. Shaib, K. Oktar, S. J. Bell, A. Ovalle, J. Dodge, A. Bosselut, K. Sinha, and A. Williams. Brittlebench: Quantifying LLM Robustness via Prompt Sensitivity, 2026. URL https://arxiv.org/abs/2603.13285.
  32. [32]A. Sinha, A. Arun, S. Goel, S. Staab, and J. Geiping. The illusion of diminishing returns: Measuring long horizon execution in llms, 2026. URL https://arxiv.org/abs/2509.09677.
  33. [33]C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv.org/abs/2408.03314.
  34. [34]S. Srivastava, J. Bidhan, H. Yan, A. Dey, T. Kansal, P. Kath, S. Mansouri, M. Marvania, V. S. Simhadri, and G. Singh. Reasoning Under Constraint: How Batch Prompting Suppresses Overthinking in Reasoning Models. ICLR Workshop on Logical Reasoning of Large Language Models, 2026. URL https://iclr.cc/virtual/2026/poster/10017459. Non-archival workshop presentation; preprint first posted in 2025.
  35. [35]M. I. Vasileva. Large language models can follow instructions, but not many at once: Phase transitions in compositional constraint satisfaction, 2026. URL https://arxiv.org/abs/2608.12426.
  36. [37]J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903.
  37. [40]A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu. Qwen3 technical report, 2025a. URL https://arxiv.org/abs/2505.09388.
  38. [41]B. Yang, H. Ma, J. Fan, and G. Liu. Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning, 2026a. URL https://arxiv.org/abs/2604.02322v1. arXiv version 1.
  39. [42]M. Yang, E. Huang, L. Zhang, M. Surdeanu, W. Y. Wang, and L. Pan. How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 13329–13347, 2025b. doi: 10.18653/v1/2025.emnlp-main.674. URL https://aclanthology.org/2025.emnlp-main.674/.DOI
  40. [43]X. Yang, F. Jia, R. Xie, X. Xi, H. Bian, J. Li, and M. Agrawal. Batch-of-Thought: Cross-Instance Learning for Enhanced LLM Reasoning, 2026b. URL https://arxiv.org/abs/2601.02950.
  41. [44]Y. Zhang, M. Li, D. Long, X. Zhang, H. Lin, B. Yang, P. Xie, A. Yang, D. Liu, J. Lin, F. Huang, and J. Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https://arxiv.org/abs/2506.05176.
  42. [45]Y. Zhang, S. Koyejo, and D. Yang. The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context, 2026. URL https://arxiv.org/abs/2607.12963.

Paper details

vlnc:W_vrvmnsd4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents