WorksComputer Science
The Silent Cost of Conversation History in LLM Reasoning
Abstract
Large language models (LLMs) in multi-turn information systems accumulate their own earlier replies and a growing task state. We asked whether this conversation history lowers accuracy, and whether process-level quality metrics warn before it does. Each analysis was fixed before its outcomes were computed. In Study 1, on 48 sharded math tasks, retaining its replies lowered final success by 0.21 for GPT-4o-mini and 0.29 for Mistral-7B (Holm-adjusted p = 0.026 and 0.004); Llama-4 Scout showed the same direction, not significant. Study 2 introduces CDR-Stream, 100-turn state-tracking dialogues with an exact answer per turn and per-model difficulty calibration. With the full history, accuracy fell from about 0.6 to 0.1 within twenty turns at matched difficulty, whereas a solver-verified state summary kept it 0.52-0.59 higher (Holm-adjusted p less than 0.001). Consistency and logical coherence fell far less than accuracy, and stability rose. None of these metrics, a paired readiness measure or calibrated single-stream drift detectors gave robust, model-general early warning, although in Study 2 the repeat-based metrics were complete only for GPT-4o-mini. In the monitors we tested, a single 100-turn stream had power only for very large drops. Systems built on LLMs should keep task state outside the transcript and monitor across sessions.
Introduction
LLMs are increasingly used as components of information and decision-support systems that keep a task open over many turns. A procurement assistant tracks a budget as invoices arrive; a logistics assistant follows stock across warehouses. The user supplies information piece by piece and asks questions whose answers depend on everything said so far, so the model’s context fills with its own earlier replies and with a task state that changes at every turn. We call this accumulated context the conversation history, and the experimental condition that keeps it is dialogue history.
Why this is a reliability problem, not only an accuracy problem. A growing conversation history can let a model lock onto a wrong task state: once an early turn is answered incorrectly, every later reply can be generated consistently from that wrong state, without the model producing any of the usual symptoms of failure. If a deployed system does not see the gold answer, and the responses it does see remain fluent, mutually consistent and locally coherent, the failure is invisible unless some other signal flags it. The paper asks two questions that follow from this chain: does retaining history cost accuracy, and does that cost compound as the conversation grows; and if it does, can the cost be seen before it shows up in the final answer, using signals a deployed system can actually compute?
What is known. Multi-turn use is known to be costly. Models lose accuracy when a task is revealed over several turns because they commit to premature answers [30]; earlier assistant replies can propagate errors [24]; and the effective context is shorter than the nominal one [33, 22].
Two gaps these results leave open. First, a measurement gap: sharded-task studies such as Laban et al. [30] score a single answer after at most a handful of turns, so they show that history can hurt but not how the loss develops over a long horizon. Over tens of turns, the questions themselves may also become harder, so a falling accuracy curve can reflect the questions rather than the context unless difficulty is explicitly held fixed. Second, a monitoring gap: none of these studies treats reasoning-quality metrics as a stream, applies a detector calibrated on a stationary baseline, or reports a false-alarm rate. Showing that performance eventually drops is not the same as showing that the drop can be seen coming. ERGO is the closest exception, but it monitors token-entropy spikes rather than response-level quality metrics, and it is not evaluated against a long, difficulty-controlled horizon [27]. To our knowledge, no prior study jointly (i) holds difficulty fixed over a long multi-turn horizon and (ii) tests established quality metrics as calibrated early-warning signals with a controlled false-alarm rate.
This paper. We close both gaps with two experiments, each analysed under a protocol fixed before its outcomes were computed. Study 1 addresses the short-horizon measurement gap directly: it compares, turn by turn, the same user information with and without the model’s own earlier replies. Study 2 addresses the long-horizon and monitoring gaps together with CDR-Stream, a benchmark of 100-turn state-tracking dialogues with an exact answer at every turn, difficulty calibrated per model, and a control arm that replaces the history with a solver-verified state summary. The two studies differ in tasks, models and controls, so we read them as two related failure modes rather than as one dose–response curve.
The paper makes three contributions:
a prospectively frozen, paired estimate of the cost of retaining assistant replies on short sharded tasks, which replicates the direction reported by Huang et al. [24];
CDR-Stream, a long-horizon, difficulty-calibrated benchmark in which accuracy under full history collapsed within about twenty turns at matched difficulty, while a verified state summary preserved it;
a prospectively defined evaluation of whether established reasoning-quality metrics and calibrated single-stream detectors can serve as early warnings of this collapse – assessing not only whether they move before accuracy fails, but whether any such lead survives a controlled false-alarm rate and repeats across models. None did: coherence and stability moved the wrong way, staying flat or rising as accuracy fell, and a closed-form bound together with a power simulation show that, for the detectors we tested, much of this reflects what a single 100-turn conversation can reveal rather than a property of the metrics themselves.
Related work
Multi-turn and long-context degradation. Benchmarks such as MT-Bench, MT-Eval and MINT evaluate conversational ability over several exchanges [57, 29, 50]. Laban et al. [30] sharded fully specified tasks across turns and traced large losses to premature answers on which models then anchored. Huang et al. [24] omitted or condensed assistant turns, including on the Lost in Conversation math tasks, and found that earlier replies can propagate errors, and Sinha et al. [47] reported self-conditioning on earlier errors in long-horizon execution. Related mechanisms include information lost in the middle of long inputs ([33]), effective context shorter than advertised ([22]), distraction by irrelevant context ([46]), sycophancy ([45]), unreliable self-correction ([23]) and difficulty tracking entity states ([28]). On the mitigation side, memory architectures summarise or page earlier content and are tested on whether facts survive across sessions ([38, 58, 34] (Wang et al. 2025; Wu et al. 2025)), and ERGO rebuilds the prompt after spikes in token entropy ([27]); our verified state summary is an upper reference for such compact memory, not a deployable summariser.
This work documents three distinct reasons accuracy might fall – lost mid-input information, inconsistent task-state tracking, and anchoring on one’s own earlier replies – but not whether any of the three can be seen, without the gold answer, before it shows up in the final answer; none of the cited studies calibrates a detector against such a signal with a controlled false-alarm rate.
Reasoning quality and drift monitoring. Reasoning quality has been characterised by agreement among sampled solutions ([51]), faithfulness of chains of thought ([53, 48, 31]) and models’ knowledge of their own correctness ([26]); sample checks ([35]) and semantic entropy ([12]) estimate uncertainty without labels. Recent work adapts clinical cognitive tests to LLMs ([40]) and optimises the steps of mathematical reasoning ([32]). These approaches judge one response at a time; we instead ask whether the same signals, aggregated per turn, anticipate a later failure. We use the six RQEval dimensions ([44]) unchanged, rather than proposing new ones, because three of them are already label-free candidate precursors: CS and SS compare repeated samples of the same reply, and LS scores coherence within one reasoning trace, none needing the gold answer (unlike CQ and ES, the outcome itself, and RS, which needs the gold to judge a reworded reply against a correct primary answer). CUSUM ([39]), Page–Hinkley ([20]), EWMA ([42]), DDM ([13]) and ADWIN ([6]) are standard stream detectors whose delay and false alarms have been compared extensively ([14, 52, 16, 4, 5]); recent methods classify drift types ([10]) and couple detection with adaptation ([1]). Such detectors usually watch input distributions or labelled performance ([43 Rabanser] et al., 2019), and commercial LLM behaviour is known to drift [7]. We apply them instead to per-turn reasoning-quality streams within a conversation.
Table 1 places our studies against the closest work. We do not claim to be the first to show that assistant history can hurt; we add a prospectively frozen paired estimate, exact per-turn answers over 100 turns at calibrated difficulty, and a direct test of quality metrics and detectors as early warnings.
| Study | Setting | Role of history | Signals for monitoring |
| Laban et al. [30] | sharded single-turn tasks | cause of premature, unreliable answers | not studied |
| Huang et al. [24] | real and sharded conversations | assistant turns omitted or condensed | not studied |
| Sinha et al. [47] | synthetic long-horizon execution | self-conditioning on own errors | not studied |
| ERGO (Khalid et al. [27]) | sharded tasks | reset after uncertainty spikes | token entropy as a trigger |
| Study 1 (ours) | sharded math tasks, paired arms | net effect of retaining replies | CS, LS, SS and a paired readiness measure |
| Study 2 (ours) | 100-turn state tracking, calibrated difficulty | full history versus verified state | six RQEval metrics, five detectors |
Table 1. Positioning against the closest prior work.
Monitoring framework and evaluation design
Both studies share the context policies, the per-turn quality signals, the streaming monitor and the evidence standards defined here (Figure 1); Table 2 summarises the two benchmarks before the details below. Full model identifiers, gateway aliases and acquisition dates are in supplementary Table S18.

Figure 1. The monitoring framework. A context policy decides what the model sees at turn . Responses are scored into a per-turn quality vector, smoothed over a window and passed to drift detectors whose thresholds are calibrated on stationary dialogues. Exact gold is needed for CQ, RS and ES and for evaluating alarms; CS, LS and SS do not use it.
Table 2: The two benchmarks at a glance. K/L: same-context repeats / reworded instructions per turn. Metric coverage lists which of the six dimensions were computed where; CQ, RS and ES always need a final answer and the gold (§ 3).
| Study 1 (Lost in Conversation) | Study 2 (CDR-Stream) | |
| Source | GSM8K math, sharded ([9, 30]) | purpose-built, solver-checked |
| Units | 48 tasks, 4–9 turns (median 6) | 30 dialogues/model, 100 turns each |
| Domains | arithmetic word problems | budget, inventory, lab, milestones (4) |
| Difficulty | not controlled | 12-level ladder, calibrated per model |
| Scenarios | single ordered reveal | 5 (2 stationary null, 3 scheduled-change) |
| Models | GPT-4o-mini, Llama-4 Scout, Mistral-7B | GPT-4o-mini, Llama-3 70B, Gemma 3 12B |
| / | 5 / 2, all models | 3 / 2, GPT-4o-mini only |
| Metric coverage | CQ/RS/ES at the final turn; CS/LS/SS at pre-final turns (gold not yet available) | CQ/LS/ES every turn, all three models; CS/RS/SS every turn, GPT-4o-mini only (repeat-based) |
Table 2. The two benchmarks at a glance.
Context policies
At turn the user sends and the model returns a primary reply from a context set by a policy :
where is the exact task state after turn , rendered by a solver. Dialogue history versus fresh prefix estimates the net effect of keeping assistant replies; context length and message positions also differ, so the effect is not attributed to the content of the replies alone. The verified state summary shortens the context, removes earlier replies and supplies a correct state at once, so it tests a mitigation, not a mechanism. Every context is packed into the request with server-side memory, retrieval and prompt rewriting disabled, which makes it reproducible from the stored records.
A worked example (CDR-Stream). Suppose turn 7 reports “warehouse B received 40 crates of part X,” and turn 8 asks how many crates of part X warehouse B now holds. Under dialogue history, is the full transcript of turns 1–8, including whatever the model itself answered at every earlier turn, right or wrong and unflagged either way. Under the verified state summary, instead has only the solver’s current ledger line for warehouse B/part X plus turn 8; the model never sees turns 1–7 or its own earlier replies. The gold answer comes from the same solver and ledger, independently of what either arm is shown (full generator and prompt in § 5).
Per-turn quality signals
Each primary reply comes with further samples from the same context and replies to frozen rewordings of the instruction. RQEval [44] computes six dimensions in , used unchanged: accuracy (CQ, the primary answer equals the gold), consistency (CS, pairwise agreement of the answers), robustness (RS, agreement of reworded replies with a correct primary answer), logical coherence (LS, absence of NLI contradictions between consecutive steps; (He et al., 2021)), efficiency (ES, correctness combined with token economy) and stability (SS, BERTScore similarity of the traces; [56]). CQ, RS and ES need a final answer and the gold, CS needs at least two answers, and LS and SS need only reasoning text. Only LS and SS can therefore serve as label-free warnings before a model is ready to answer. LS splits a reply’s reasoning into consecutive steps and reports one minus the mean NLI-scored contradiction probability between adjacent pairs, so a reply that never contradicts itself scores 1 regardless of whether its conclusion is right; SS instead BERTScores the sampled replies against each other, so near-duplicate phrasing raises SS even when the shared answer is wrong. Both need parseable reasoning text; missing values are never imputed and are reported with their support (supplementary Section S1.2 gives the exact splitting and scoring rules).
The streaming monitor
Let be the value of metric at turn , or missing.
Definition 1 (Window mean). Let be the set of the last turns at which is observed. Once , the monitored statistic is
with primary. Missing values neither enter nor advance the window.
Five detectors, CUSUM, Page–Hinkley, EWMA, DDM and ADWIN, are assigned to metrics by three strategies fixed in the funded project design: CUSUM on every metric, a distribution-specific mapping, and ADWIN on every metric (supplementary Table S8).
Definition 2 (Alarm, calibration and detection). For a detector with threshold reading , the alarm time in a stream of turns is
with if no alarm occurs. The threshold is the most sensitive grid value whose first-alarm rate on stationary streams, , is at most ; drift data are never used. For a change at turn , an alarm is a false alarm if and a detection if .
Study 1 estimates from permutation nulls with family-wise , Study 2 from its stationary dialogues. Detecting a scheduled change within 33 turns is not yet a warning. We call a signal an early warning only if, in addition, it alarms before the accuracy (CQ) alarm in the same dialogue, keeps the null calibration, and does both across models.
A paired readiness measure
Sharded tasks allow an answer-free paired signal. Let and be 1 if the fresh-prefix and dialogue-history arms attempt an answer at turn , and 0 if they ask for more information. The Cumulative Resolution Lag of a task is
over the pre-final turns at which both arms responded validly. It measures how persistently the history arm lags an information-matched counterfactual in becoming ready to answer, and it was fixed as an auxiliary measure before the Study 1 data were collected.
Evidence standards
Each result carries one of four labels: confirmatory (a hypothesis fixed before its outcomes were computed), prespecified secondary, exploratory (rules fixed before computation on data already seen) or post hoc. In Study 1 the whole protocol, including the analysis, was frozen before data collection. In Study 2 the design, sample, difficulty mappings, prompt and truncation policy were frozen before the main run; the analysis protocol was frozen when a budget pause had left 25,039 of 54,000 calls complete, before any main-run metric was computed (dated timeline in supplementary Table S19). Raw responses are stored before scoring and never overwritten, failed gates are reported rather than relabelled, and every number, table and figure is regenerated by scripts from hashed inputs.
Study 1: the paired effect of retaining assistant replies
Design
We used the math subset of Lost in Conversation (Laban et al., [30]): 103 GSM8K-derived problems (Cobbe et al., [9]), each split into information shards revealed one per turn. Before any model call, an input audit excluded 17 tasks that duplicated our earlier inputs or appeared to change meaning when sharded, and a hash-based draw that used no model output selected 48 tasks with 4–9 shards (median 6). GPT-4o-mini [37], Llama-4 Scout and Mistral-7B [25] answered every turn in both the dialogue-history and fresh-prefix arms. Each reply had to end with FINAL_ANSWER: <number> or STATUS: NEED_MORE_INFORMATION. Each cell received samples and rewordings at temperature 0.7; only the primary reply entered later history. All 11,382 calls succeeded without truncation; six empty Mistral-7B primaries were kept as invalid.
The primary endpoint was paired final-turn success, tested by exact McNemar tests [36] with task-bootstrap 95% intervals (10,000 resamples; [11]) and Holm correction across models [21]. Before the last shard most tasks have no determined answer, so CQ, RS and ES cannot be observed; the pre-final warning candidates were CS, LS and SS. Each was tested by a task-clustered partial correlation with final failure, adjusted for model, turn, revealed information and log context length, with Holm correction across the three metrics. CRL (defined in § 3) measures delay in becoming ready to answer, not accuracy; it targeted a positive final readiness gap, in which the fresh prefix answers while the history arm is still waiting for information.
Results
Retaining assistant replies lowers final success (confirmatory). Final success fell from 0.896 to 0.688 for GPT-4o-mini (difference 0.208, 95% CI [0.063, 0.354], Holm ) and from 0.354 to 0.063 for Mistral-7B (0.292, CI [0.146, 0.438], Holm ). Llama-4 Scout moved in the same direction (0.104, CI [0.000, 0.208], Holm ) without significance (Table 3, Figure 2). Across models, only the fresh prefix succeeded in 34 model–task pairs and only the history arm in 5. The GPT-4o-mini and Mistral-7B effects remained when tasks with format-invalid final replies were excluded (supplementary Table S1). For Mistral-7B the harm took a visible form: with its own replies in context it tended to stay in the “need more information” state at the last turn, whereas without them it answered.
| Model | Dialogue history | Fresh prefix | Diff. [95% CI] | F/D | ||
| GPT-4o-mini | 33/48 (0.688) | 43/48 (0.896) | 0.208 [0.063, 0.354] | 12/2 | 0.013 | 0.026 |
| Llama-4 Scout | 41/48 (0.854) | 46/48 (0.958) | 0.104 [0.000, 0.208] | 6/1 | 0.125 | 0.125 |
| Mistral-7B | 3/48 (0.063) | 17/48 (0.354) | 0.292 [0.146, 0.438] | 16/2 | 0.001 | 0.004 |
Table 3. Study 1 paired final-turn success over 48 tasks per model. F/D: tasks in which only the fresh prefix, or only the dialogue history, succeeded; : exact McNemar test.
Figure 2. Study 1 primary endpoint ( paired tasks per model). Left: final-turn success with 95% Wilson intervals [54]. Right: paired difference, fresh prefix minus dialogue history, with 95% task-bootstrap intervals and Holm-adjusted McNemar -values.
The measurable quality metrics do not warn (confirmatory). CS, LS and SS showed no statistically detectable association with final failure beyond turn, revealed information and context length: all partial correlations were close to zero and every Holm-adjusted -value was 1.0 (Table 4). This does not mean that they carry no information; CS was measurable on only 29 source tasks, so small associations cannot be excluded.
| Metric | Turns | Units | Tasks | |||
| CS | 101 | 60 | 29 | -0.047 | 0.571 | 1.000 |
| LS | 637 | 144 | 48 | +0.017 | 0.365 | 1.000 |
| SS | 627 | 144 | 48 | -0.021 | 0.653 | 1.000 |
Table 4. Study 1 pre-final metrics as predictors of final failure under dialogue history. A unit is one (task, model) pair (up to 3 × 48); turns are the pre-final turns of those units pooled together, the level at which the metric is observed; tasks are the distinct source tasks contributing at least one turn, the level at which the permutation test clusters observations. : task-clustered partial correlation; : one-sided task-clustered permutation test.
The paired readiness measure reduces to persistence. CRL discriminated its target modestly (pooled model-stratified AUROC 0.627, permutation , CI [0.493, 0.760]; supplementary Table S2). Post hoc, the last pre-final readiness gap alone did better (AUROC 0.672, ), and within its strata CRL added nothing (AUROC 0.468, ). Against accuracy loss, a post-hoc target, the association was borderline (AUROC 0.571, ) and came from Mistral-7B alone. Even a paired, counterfactual measure thus recorded the failure once the history arm had fallen behind rather than anticipating it.
Study 2: long-horizon degradation under calibrated difficulty
Study 1 tasks span at most nine turns, and their difficulty is not controlled. In long dialogues the questions themselves may become harder, so a falling accuracy curve could reflect the questions rather than the context. Study 2 therefore uses a purpose-built benchmark with an exact answer at every turn and difficulty set by design.
The CDR-Stream benchmark and design
Dialogues. CDR-Stream contains 100-turn dialogues in four state-tracking domains: a budget ledger, inventory across warehouses, a laboratory protocol with container volumes, and chains of project milestones. Each user turn states events that update a shared state, such as a delivery, a transfer or a correction, possibly with distractor sentences, and asks one question: a sum, a difference, a count above a threshold, a maximum, a what-if or a share. An exact solver computes the gold answer of every turn.
Calibrated difficulty and scenarios. Difficulty follows a 12-level ladder that changes one axis per step: query terms, events per turn, distractors or event complexity (supplementary Table S4). Before any stream existed, a single-turn calibration mapped easy, medium and hard regimes to ladder levels for each model and domain, targeting accuracy bands of 0.75–0.92, 0.60–0.78 and 0.35–0.65, and a frozen rule (isotonic fit, [3]; one-sided Cochran–Armitage trend test, [2]; minimum regime gaps) excluded domains with a ceiling, a floor or no trend (supplementary Table S5). Each model is thus evaluated in its own informative range, at the cost of models not all being tested on the same set of domains; CDR-Stream is not designed to rank models against each other, and we do not compare raw accuracy across models for this reason. Five scenarios fix the regime sequence: stationary easy (S0-E) and hard (S0-H), which serve as monitoring nulls; a staircase from easy to hard with changes at turns 34 and 68 (S1); a sudden easy-to-hard shift at turn 51 (S2); and easy to hard to easy with changes at 34 and 67 (S3); within a regime, levels are mixed around the centre level (supplementary Section S2.2).
Models, arms and analysis. GPT-4o-mini, Llama-3 70B ([18]) and Gemma 3 12B ([15]) were retained because they passed calibration and a technical pilot, before any stream outcome existed; replaced candidates and two disclosed deviations for Llama-3 70B are documented in the supplement. Each model received 30 held-out dialogues, six per scenario, in two arms: dialogue history, and the verified state summary with the current turn. Both arms used the same frozen prompt and response contract (verbatim in supplementary Section S2.1), temperature 0.7 and a 2,048-token ceiling. The dialogue arm also drew samples and rewordings per turn; because the research budget staged acquisition, these were completed only for GPT-4o-mini. Truncated replies were kept and scored as missing under a policy frozen before the main run. Dialogues are the independent units. A prespecified secondary contrast compares accuracy at turns 1–50 and 51–100 at the same ladder level within each dialogue – holding question-side difficulty fixed (query terms, events per turn, distractors and event complexity), not the conversation length or the accumulated state, since those grow across turns under both arms by design. The confirmatory hypothesis H-C1 states that, for Llama-3 70B and Gemma 3 12B, per-dialogue accuracy over turns 1–100 is higher under the verified state summary, tested by one-sided sign-flip tests over 30 dialogues (20,000 draws; [17]) with bootstrap intervals and Holm correction. GPT-4o-mini control data had been seen before this protocol and are reported outside the family.
Results
Accuracy collapses under accumulated history. In all three models, dialogue-history accuracy fell from about 0.6 in the first five turns to about 0.1 after turn 20 (Table 5, Figure 3a). The curves of the three models were nearly identical despite different sizes and mappings, and the collapse appeared in every scenario. In stationary easy dialogues, where the questions never became harder, accuracy over turns 21–100 was 0.10–0.14 under dialogue history against 0.82–0.86 under the verified state summary, both restricted to S0-E (Figure 3b); Table 5’s state-summary column is pooled over all five scenarios, not S0-E alone, so the two numbers are not read from the same column. At the same ladder level, late turns were less accurate than early turns by -0.118 (prespecified secondary, , dialogues; -0.185 in S0-E). Because most of the decline occurs in the first half, this contrast understates the total loss. The collapse also hides the scheduled task drifts: it is present in stationary dialogues and begins before any detector could form a baseline, so pre/post contrasts around the scheduled changes mix the two (supplementary Table S7).
| Model | Dialogue history, turns 1–5 | Dialogue history, turns 6–20 | Dialogue history, turns 21–100 | S0-E, turns 21–100 | State summary, turns 21–100 |
| GPT-4o-mini | 0.61 | 0.25 | 0.09 | 0.10 | 0.65 |
| Llama-3 70B | 0.60 | 0.31 | 0.11 | 0.14 | 0.69 |
| Gemma 3 12B | 0.61 | 0.31 | 0.08 | 0.10 | 0.71 |
Table 5. Study 2 accuracy (CQ) by turn range under dialogue history, pooled and in the stationary easy scenario (S0-E), and under the verified state summary (turns 21–100). Thirty dialogues per model.
Figure 3. Study 2 accuracy over 100 turns in 5-turn bins: (a) all 30 dialogues per model; (b)–(f) the five scenarios, six dialogues per model each. Solid: dialogue history; dashed: verified state summary; dotted vertical lines: scheduled difficulty changes.
A verified state summary preserves accuracy (confirmatory). Under the verified state summary, accuracy stayed high and followed the difficulty schedule, stepping down at the changes of S1 and S2 and recovering in the final easy regime of S3, a recovery that dialogue history never showed (Figure 3). H-C1 was supported for both members of the family (Table 6): over turns 1–100 the gap was 0.545 for Llama-3 70B and 0.588 for Gemma 3 12B (Holm $p < 0.001 and <0.001), and over turns 21–100 it was 0.575 and 0.630. GPT-4o-mini showed the same direction.
| Model | Mean gap | 95% bootstrap CI | Holm | Dialogues |
| GPT-4o-mini | 0.521 | [0.476, 0.566] | – | 30 |
| Llama-3 70B | 0.545 | [0.513, 0.576] | <0.001 | 30 |
| Gemma 3 12B | 0.588 | [0.557, 0.619] | <0.001 | 30 |
Table 6. Frozen test H-C1: per-dialogue accuracy of the verified state summary minus dialogue history over turns 1–100. GPT-4o-mini was observed before the protocol and is outside the Holm family.
Can the degradation be detected early?
A deployed system does not observe accuracy, so we asked whether the framework of § 3 would have warned. “Stationary” here is the task-difficulty schedule (S0-E, S0-H), not the metric: a metric can still drift on a stationary task, and that drift is the false-alarm risk a calibrated detector must control. We separate three evaluations. (i) The initial collapse: under dialogue history, accuracy falls within the first 20 turns even on stationary tasks, before any detector has a reference period. (ii) Scheduled-drift detection: under the verified-state arms, which remove that background collapse, can a detector flag the experimenter-imposed changes of S1–S3 within a bounded delay without alarming on S0-E/S0-H? (iii) Relative timing: where both a candidate signal and CQ alarm, does the candidate alarm first, and does any such lead replicate across models? No test passed any of these three.
Quality metrics do not fall with accuracy
Coherence and stability moved in the wrong direction. In a post-hoc replay of Study 1, in which completed sessions were streamed with a switch from fresh prefix to dialogue history, both rose while accuracy fell: at in the sudden scenario, the median Cliff’s ([8]) between the last pre-change and the last window was +0.46 for SS in GPT-4o-mini, +0.42 in Llama-4 Scout, and +0.31 for LS in GPT-4o-mini (Figure 4b). Study 2 shows the same pattern over 100 turns (Figure 4a, descriptive). For GPT-4o-mini under dialogue history, accuracy fell from 0.61 (turns 1–5) to 0.08 (turns 81–100), while stability rose from 0.94 to 0.98 and coherence moved only from 0.54 to 0.47. Consistency fell from 0.79 to 0.64, but slowly and long after most accuracy was lost; over turns 6–15 it even rose as accuracy halved. As the model fails, its replies become more alike and remain locally coherent, so a monitor that alarms on decreases cannot see the failure, and a two-sided one would see a change whose sign says nothing about correctness.
Figure 4. Quality metrics do not track accuracy. (a) Study 2, GPT-4o-mini, dialogue history, 5-turn means over 30 dialogues (descriptive); dashed: accuracy under the verified state summary; CS is supported by 2976 of 3,000 turns. (b) Post-hoc replay of Study 1: median and 10–90% band over 1,000 orderings of window means () when the regime switches from fresh prefix to dialogue history at session 49.
Calibrated single-stream detectors
In the Study 1 replay, with thresholds set on 2,000 permutation nulls at family-wise and checked on 2,000 disjoint nulls, the best detection rate of any configuration was 0.067 (supplementary Table S3; there the stream items are sessions, and a detection is any alarm after the switch). In the Study 2 dialogue arm the stationary dialogues that should serve as nulls collapsed themselves, so for CQ no threshold met the 5% false-alarm target in most configurations, and the other metrics detected at most 0.11 of drifts within 33 turns (supplementary Table S9). This is the central difficulty of monitoring under accumulated history: the decline starts at turn 1, before any reference period, in the very streams that define normal behaviour.
The verified-state arms remove this background collapse and leave only the scheduled drift. Rules developed exploratorily on GPT-4o-mini and applied unchanged to Llama-3 70B and Gemma 3 12B failed the frozen validation in both models: CUSUM on the CQ window mean detected 6/18 and 1/18 drift dialogues within 33 turns, and a baseline threshold rule on LS signalled in 11/18 and 16/18 drift dialogues but also in 6/12 and 3/12 stationary ones (supplementary Section S2.7). Our generic ADWIN baseline never alarmed because the implementation omitted the variance term of the cut bound of Bifet and Gavaldà [6]; re-running the affected analyses with a corrected ADWIN and with CUSUM changed no conclusion (supplementary Section S4).
The frozen monitor forms a window and then ignores the first 20 window values, so it cannot alarm before turn 40, after the first scheduled change at turn 34. A post-hoc audit removed this floor with a CUSUM on raw values standardised against turns 1–30 (supplementary Section S5). On synthetic streams it caught a shift injected at turn 31 by turn 33 in 45% of streams for a shift of 0.3 and in nearly all for 0.5, where the frozen monitor caught none. Table 7 applies it to the control arms, with thresholds set on GPT-4o-mini and transferred unchanged. The CQ detector did not transfer, and LS, the label-free candidate, detected few drift dialogues and was usually later than CQ. Its earlier alarms in GPT-4o-mini (3 of the 5 dialogues in which both alarmed) were not repeated in the other models. ES, which needs the gold answer, did no better (supplementary Table S17). The result is therefore not an absence of early alarms but low coverage, mostly late alarms and poor transfer. It concerns the scheduled changes only; the early collapse under dialogue history begins inside the audit’s own reference period.
| Stationary alarms | Drift detected | |||||
| Model | CQ | LS | CQ | LS | LS first | Offset |
| GPT-4o-mini | 0/12 | 0/12 | 13/18 | 5/18 | 3/5 | −1 |
| Llama-3 70B | 3/12 | 1/12 | 14/18 | 5/18 | 0/2 | +27 |
| Gemma 3 12B | 2/12 | 0/12 | 15/18 | 8/18 | 1/7 | +10 |
Table 7. Post-hoc fixed-baseline CUSUM on the control arms (18 drift and 12 stationary dialogues per model; thresholds set on GPT-4o-mini). LS first: dialogues in which LS alarmed before CQ, among those in which both alarmed; offset: median LS alarm turn minus CQ alarm turn.
Why single streams are underpowered
A signal can fail to warn for two unrelated reasons: it may carry no information about the coming failure, as CS, LS and ES did above, or it may carry some information that a single 100-turn stream is simply too short to detect at a controlled false-alarm rate. The bound and simulation below quantify the second reason only, for CQ itself; they do not rehabilitate the other metrics, whose failure was a direction and transfer problem, not a sample-size one.
A simple bound shows how little one stream can reveal. Consider the accuracy window mean (4) pooled over dialogues at the same turn, so that it averages binary outcomes.
Proposition 1 (Smallest detectable drop). Let the outcomes in the window be independent Bernoulli variables with success probability before a change and after it, and write . A one-sided test that alarms when the window mean falls below a threshold with false-alarm probability under detects the drop with probability at least , under the normal approximation, only if
where is the -quantile of the standard normal distribution.
Proof. Under the window mean is approximately normal with mean and standard deviation . A false-alarm probability requires the threshold , and the detection probability under is . This is at least exactly when , which gives (7) after substituting and .
With , , and , the bound is about 0.24 for one dialogue, 0.10 for six and 0.05 for 18. A monitor that looks at many turns needs a smaller per-look level; splitting over the 67 looks from turn 34 to 100 raises the bound to 0.38, 0.15 and 0.09, and positive dependence between turns raises it further. The proposition covers only a threshold on the window mean, so it explains the scale and the gain from pooling rather than the exact power of CUSUM or the other detectors.
For the detectors actually used, we simulated correctness streams with pre-change accuracy 0.80 and a drop at turn 34 or 51, pooling dialogues per turn, with thresholds from 12 null streams at a 5% false-alarm target (Figure 5). A single stream reached 80% detection within 33 turns only for drops of about 0.5, whereas the task drifts in our data were 0.1–0.3; six dialogues per turn lowered the detectable drop to 0.2, and 18 to 0.1, with similar results for other windows and calibration sets (supplementary Table S13). On the control arms, a CQ CUSUM on six-dialogue aggregates detected all three drift scenarios in both validation models, with 0/2 and 1/2 alarms in the stationary Llama and Gemma aggregates. Pooling sessions is therefore promising, but two null aggregates per model cannot verify a 5% false-alarm rate: with no alarm in null streams, the exact one-sided 95% upper bound is , which is 0.78 for and 0.22 for .
Figure 5. Simulated detection within 33 turns versus the accuracy drop from 0.80 and the number of dialogues pooled per turn (, thresholds from 12 null streams, 5% false-alarm target). Left: change at turn 34; right: at turn 51. Dotted line: 80% detection.
Discussion
What the two studies show. Study 1 isolates the net cost of keeping the model’s own replies, with the user information held fixed. Study 2 shows a far larger loss in long state tracking, but it does not separate history length, the number of state updates and accumulated errors, because the verified summary changes all three at once. Its result is therefore a joint failure of state tracking under a growing transcript, not evidence that the assistant’s replies alone cause the collapse; that narrower claim rests on Study 1. Either way, a raw transcript as the only memory of a stateful task has a large and avoidable cost.
Why the quality metrics stay silent. Consistency, coherence and stability describe local properties of responses, and a model that has adopted a wrong state can reason coherently and repeatably from it. In our data these metrics fell far less than accuracy, and stability rose, which suggests that accumulated context makes responses more anchored rather than more erratic, in line with reports of self-conditioning and sycophancy [47, 45]. Two classes of signal remain untested. Token entropy, used by ERGO [27], needs token probabilities, which our protocols did not collect. Semantic entropy [12] can be computed from sampled answers alone; our consistency metric also summarises agreement among samples, but it is a pairwise agreement rate rather than an entropy, and we did not compute semantic entropy. Counterfactual signals, which compare the deployed context with a history-free or state-grounded re-query, target anchoring directly; CRL tracked the current readiness gap rather than anticipating it, and comparing answers rather than readiness is a natural next step.
Implications for LLM-based information systems. Where the task state can be represented explicitly, as in ledgers, inventories, protocols and schedules, it should be kept in a verified store and sent with each request; in our setting this kept accuracy 0.52–0.59 above full history and shrank the prompt at turn 100 from tens of thousands of tokens to a few hundred; summaries written by the model itself should be expected to do less well than our solver-verified one. A dashboard of one conversation’s consistency, coherence or stability is likely to stay reassuring while accuracy collapses, so pooling sessions or comparing against counterfactual re-queries is the better route. Any detector should first be shown to fire on streams of the intended length, with a reference period free of the degradation it is meant to detect.
Limitations. Both studies use arithmetic and state-tracking tasks with exact answers, which make accuracy observable but are narrower than open-ended dialogue. All models were accessed through one academic gateway, the two studies use different model sets, and the models are not the strongest available. This is partly a property of the design: calibration needs an informative accuracy range, and a stronger open model, Gemma 4 31B, stayed above the hard band even at the contingency levels of the ladder (supplementary Section S2.3), so testing frontier models requires a harder ladder rather than only more calls. In Study 2, the repeat-based metrics were complete only for GPT-4o-mini, and no user-only arm was run, so the share of the collapse due to the assistant’s replies is not estimated. Monitoring analyses beyond the frozen tests are exploratory or post hoc, two deviations for Llama-3 70B followed gate results and are disclosed, and one investigator performed the task audit and calibration.
Conclusion
Keeping an LLM’s own replies lowered success on short multi-turn tasks, and in 100-turn state tracking the full history collapsed accuracy within about twenty turns at matched difficulty, while a verified state summary preserved it. The loss was silent to the monitors we tested: no quality metric, paired readiness measure or calibrated single-stream detector gave robust, model-general early warning; coherence changed little and stability rose as accuracy fell. Systems built on LLMs should keep task state outside the transcript and monitor across sessions or against counterfactual re-queries. Open questions include uncertainty-based and counterfactual signals, the separate roles of history length, state load and error accumulation, and state files that models keep and verify themselves.
Data and code availability
All prompts, frozen protocols, configurations, analysis code and the scripts that generate every number, table and figure in this article are publicly available at https://github.com/senolali/reasoning-monitor-3501.
Acknowledgements
This work was supported by the Scientific and Technological Research Council of Türkiye (TÜBİTAK) under the 3501 programme, project number 126E663.
Declaration of competing interest
The author declares that he has no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
CRediT authorship contribution statement
Ali Şenol: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Visualization, Writing–original draft, Writing–review and editing, Project administration, Funding acquisition.
Declaration of generative AI and AI-assisted technologies in the manuscript preparation process
During the preparation of this work the author used Claude (Anthropic) in an assistive capacity, for analysis code, figure scripts and language editing. The author designed the studies, carried out the research and reviewed and edited all resulting content, and takes full responsibility for the content of the published article. The language models evaluated in Studies 1 and 2 are objects of study, not writing aids.
References
- [1]Abu-Shaira, M., Shi, W., 2027. SCCM: Stream cruise control method for automated drift detection and adaptation. Information Sciences 762, 124224. doi:10.1016/j.ins.2026.124224.
- [2]Armitage, P., 1955. Tests for linear trends in proportions and frequencies. Biometr-ics 11, 375–386. doi:10.2307/3001775.DOI
- [3]Barlow, R.E., Bartholomew, D.J., Bremner, J.M., Brunk, H.D., 1972. Statistical Inference under Order Restrictions: The Theory and Application of Isotonic Regression. Wiley Series in Probability and Mathematical Statistics, Wiley, Lon-don.
- [4]Barros, R.S.M., Santos, S.G.T.C., 2018. A large-scale comparison of concept drift detectors. Information Sciences 451–452, 348–370. doi:10.1016/j.ins.2018.04.014.
- [5]Bayram, F., Ahmed, B.S., Kassler, A., 2022. From concept drift to model degradation: An overview on performance-aware drift detectors. Knowledge-Based Systems 245, 108632. doi:10.1016/j.knosys.2022.108632.DOI
- [6]Bifet, A., Gavaldà, R., 2007. Learning from time-changing data with adaptive windowing, in: Proceedings of the 2007 SIAM International Conference on Data Mining, pp. 443–448.DOI
- [7]Chen, L., Zaharia, M., Zou, J., 2024. How is ChatGPT’s behavior changing over time? Harvard Data Science Review 6.arxiv.org/abs/2307.09009
- [8]Cliff, N., 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin 114, 494–509. doi:10.1037/0033-2909.114.3.494.DOI
- [9]Cobbe, K., et al., 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.arxiv.org/abs/2110.14168
- [10]Deng, Z., Feng, Q., Lin, B., Yen, G.G., 2025. Zilean: A modularized framework for large-scale temporal concept drift type classification. Information Sciences 712, 122134. doi:10.1016/j.ins.2025.122134.
- [11]Efron, B., 1979. Bootstrap methods: Another look at the jackknife. The Annals of Statistics 7, 1–26. doi:10.1214/aos/1176344552.DOI
- [12]Farquhar, S., Kossen, J., Kuhn, L., Gal, Y., 2024. Detecting hallucinations in large language models using semantic entropy. Nature 630, 625–630. doi:10.1038/s41586-024-07421-0.DOI
- [13]Gama, J., Medas, P., Castillo, G., Rodrigues, P., 2004. Learning with drift detection, in: Advances in Artificial Intelligence – SBIA 2004, pp. 286–295.
- [14]Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., Bouchachia, A., 2014. A survey on concept drift adaptation. ACM Computing Surveys 46, 44:1–44:37.
- [15]Gemma Team, 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786.arxiv.org/abs/2503.19786
- [16]Gonçalves, Jr., P.M., de Carvalho Santos, S.G.T., Barros, R.S.M., Vieira, D.C.L., 2014. A comparative study on concept drift detectors. Expert Systems with Applications 41, 8144–8156. doi:10.1016/j.eswa.2014.07.019.
- [17]Good, P.I., 2005. Permutation, Parametric and Bootstrap Tests of Hypotheses. Springer Series in Statistics. 3rd ed., Springer, New York. doi:10.1007/b138696.DOI
- [18]Grattafiori, A., et al., 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 .arxiv.org/abs/2407.21783
- [20]Hinkley, D.V., 1971. Inference about the change-point from cumulative sum tests. Biometrika 58, 509–523.DOI
- [21]Holm, S., 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6, 65–70.DOI
- [22]Hsieh, C.P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., Ginsburg, B., 2024. RULER: What’s the real context size of your long-context language models?, in: Proceedings of the First Conference on Language Modeling (COLM).
- [23]Huang, J., Chen, X., Mishra, S., Zheng, H.S., Yu, A.W., Song, X., Zhou, D., 2024. Large language models cannot self-correct reasoning yet, in: International Conference on Learning Representations.
- [24]Huang, J.Y., Choshen, L., Sun, W., Khattab, O., Fernandez Astudillo, R., Damani, M., Broderick, T., Andreas, J., 2026. Do LLMs benefit from their own words? arXiv preprint arXiv:2602.24287 .
- [25]Jiang, A.Q., et al., 2023. Mistral 7b. arXiv preprint arXiv:2310.06825 .arxiv.org/abs/2310.06825
- [26]Kadavath, S., et al., 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 .arxiv.org/abs/2207.05221
- [27]Khalid, H.M., Jeyaganthan, A., Do, T., Fu, Y., Sharma, V., O’Brien, S., Zhu, K., 2025. ERGO: Entropy-guided resetting for generation optimization in multi-turn language models, in: Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertainNLP 2025).
- [28]Kim, N., Schuster, S., 2023. Entity tracking in language models, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics. pp. 3835–3855. doi:10.18653/v1/2023.acl-long.213.DOI
- [29]Kwan, W.C., Zeng, X., Jiang, Y., Wang, Y., Li, L., Shang, L., Jiang, X., Liu, Q., Wong, K.F., 2024. MT-Eval: A multi-turn capabilities evaluation benchmark for large language models, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20153–20177. doi:10.18653/v1/2024.emnlp-main.1124.
- [30]Laban, P., Hayashi, H., Zhou, Y., Neville, J., 2026. LLMs get lost in multi-turn conversation, in: International Conference on Learning Representations (ICLR). ArXiv:2505.06120.
- [31]Lanham, T., et al., 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702 .arxiv.org/abs/2307.13702
- [32]Lee, B., Lv, Y., Zhang, T., 2026. FGRPO: Fine-grained reasoning preference opti- mization with iterative stabilization for mathematical reasoning in large language model. Information Sciences 741, 123232. doi:10.1016/j.ins.2026.123232.
- [33]Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P., 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12, 157–173.arxiv.org/abs/2307.03172
- [34]Maharana, A., Lee, D.H., Tulyakov, S., Bansal, M., Barbieri, F., Fang, Y., 2024. Evaluating very long-term conversational memory of LLM agents, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics. pp. 13851–13870. doi:10.18653/v1/2024.acl-long.747.DOI
- [35]Manakul, P., Liusie, A., Gales, M.J.F., 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics. pp. 9004–9017. doi:10.18653/v1/2023.emnlp-main.557.DOI
- [36]McNemar, Q., 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 153–157.pubmed.ncbi.nlm.nih.gov/20254758
- [37]OpenAI, 2024. GPT-4o system card. arXiv preprint arXiv:2410.21276.
- [38]Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S.G., Stoica, I., Gonzalez, J.E., 2023. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560.arxiv.org/abs/2310.08560
- [39]Page, E.S., 1954. Continuous inspection schemes. Biometrika 41, 100–115.DOI
- [40]Park, C., Kim, C., 2027. LMMSE: A clinically inspired framework for cognitive evaluation of large language models. Information Sciences 758, 123956. doi:10.1016/j.ins.2026.123956.DOI
- [42]Roberts, S.W., 1959. Control chart tests based on geometric moving averages. Technometrics 1, 239–250.
- [44]Şenol, A., Agrawal, G., Liu, H., 2026. Measuring reasoning quality in LLMs: A multi-dimensional behavioral framework. Big Data and Cognitive Computing 10, 300. doi:10.3390/bdcc10090300.DOI
- [45]Sharma, M., et al., 2024. Towards understanding sycophancy in language models, in: International Conference on Learning Representations (ICLR).arxiv.org/abs/2310.13548
- [46]Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E.H., Schärli, N., Zhou, D., 2023. Large language models can be easily distracted by irrelevant context, in: Proceedings of the 40th International Conference on Machine Learning.arxiv.org/abs/2302.00093
- [47]Sinha, A., Arun, A., Goel, S., Staab, S., Geiping, J., 2026. The illusion of di minishing returns: Measuring long horizon execution in LLMs, in: International Conference on Learning Representations (ICLR).
- [48]Turpin, M., Michael, J., Perez, E., Bowman, S.R., 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting, in: Advances in Neural Information Processing Systems.arxiv.org/abs/2305.04388
- [50]Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., Ji, H., 2024. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback, in: International Conference on Learning Representations.arxiv.org/abs/2309.10691
- [51]Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D., 2023. Self-consistency improves chain of thought reasoning in language models, in: International Conference on Learning Representations.arxiv.org/abs/2203.11171
- [52]Webb, G.I., Hyde, R., Cao, H., Nguyen, H.L., Petitjean, F., 2016. Characterizing concept drift. Data Mining and Knowledge Discovery 30, 964–994. doi:10.1007/s10618-015-0448-4.DOI
- [53]Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q.V., Zhou, D., 2022. Chain-of-thought prompting elicits reasoning in large language models, in: Advances in Neural Information Processing Systems (NeurIPS), pp. 24824–24837. doi:10.52202/068431-1800.DOI
- [54]Wilson, E.B., 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22, 209–212. doi:10.1080/01621459.1927.10502953.DOI
- [56]Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y., 2020. BERTScore: Evaluating text generation with BERT, in: International Conference on Learning Representations.arxiv.org/abs/1904.09675
- [57]Zheng, L., et al., 2023. Judging LLM-as-a-judge with MT-Bench and chatbot arena, in: Advances in Neural Information Processing Systems (Datasets and Benchmarks Track).arxiv.org/abs/2306.05685
- [58]Zhong, W., Guo, L., Gao, Q., Ye, H., Wang, Y., 2024. MemoryBank: Enhancing large language models with long-term memory, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 19724–19731. doi:10.1609/aaai.v38i17.29946.DOI



