WorksComputer Science
Wavering Oracles: Selective Updating and Correlated Failures in LLMs and Their Implications for Scientific Workflows
Abstract
Scientific workflows increasingly use repeated queries, multiple models, and interacting agents. Reliability therefore depends on whether models preserve correct conclusions, accept valid corrections, and contribute errors that a selector can distinguish. Using SycoBench- 600 as a controlled measurement substrate, we evaluate these requirements through selective updating, defined by resistance to misleading suggestions and uptake of correct suggestions. The study covers ten models and 17,055 trajectories. Published models span 13.4 to 71.6 percentage points in selectivity. Under identical local evaluation, Qwen3-4B is selectively adaptive at 45.6 points, Gemma3-4B is destabilized at minus 14.1 points, and SmolLM3-3B follows both correct and wrong explicit suggestions, producing zero selectivity. Matched interventions identify model specific responses to doubt, authority, and explicit advice. Among seven published models, the best reaches 95.3 percent accuracy, plurality reaches 88.6 percent, and the oracle ceiling is 99.8 percent. Mean error correlation of 0.285 reduces seven models to an effective independent count of 2.58. A leave-one-stem-family-out reliability selector reaches 96.2 percent, recovering 67.7 percent of the plurality-to-oracle gap. These results establish selective updating, error diversity, and calibrated adjudication as jointly measurable design targets for multi-model scientific workflows.
Introduction
Language models now participate in literature triage, hypothesis generation, data analysis, code development, manuscript revision, and technical decision support. Their value in these settings depends on more than the accuracy of an isolated first response. A scientific interaction is sequential. A researcher may challenge
an answer, cite an authority, offer a candidate correction, or compare several model outputs. The model must then decide whether the new information warrants revision. Controlled evidence from scientific summarization further shows that narrative framing can induce sycophantic distortions in representations of research findings [1].
Two familiar failure descriptions capture opposite sides of this decision. A model can remain attached to an incorrect initial answer, which resembles anchoring or confirmation bias [2, 3, 4]. It can also abandon a correct answer after an unsupported challenge, which is commonly studied as sycophancy [5, 6, 7]. Neither willingness to change nor resistance to change is sufficient by itself. Reliable revision requires selective updating, preserving a correct answer under misleading feedback while accepting a correct correction when the initial answer is wrong.
This distinction matters directly for hallucination mitigation. Hallucination surveys and benchmarks have documented broad factual reliability problems [8, 9, 10, 11]. Self-checking and external critique can improve outputs [12, 13, 14], yet their benefit is mediated by how a model treats the feedback itself. Recent controlled evidence shows that warmer model behavior can increase validation of incorrect user beliefs [15]. Recent causal evidence also shows that internal confidence can govern whether a model answers or abstains [16]. Together, these results motivate an evaluation centered on the decision to retain or revise a claim.
Multiple windows, models, or agents offer a second route to reliability. Self-consistency and debate can improve reasoning and factuality [17, 18, 19]. Their success, however, depends on diversity that survives interaction. Human groups can exhibit collective intelligence [20], while social influence can contract diversity and undermine crowd accuracy [21]. Large scale evidence across model families shows that language model errors remain substantially correlated [22], and related dependence now appears in multi-agent systems [23, 24]. A practical analysis therefore needs to measure both the quality of individual revision and the dependence among model errors.
We provide such an analysis with four contributions.
We represent answer revision on a two dimensional plane comprising the flip rate under a wrong suggestion and the update rate under a correct suggestion. Their difference is a compact selectivity score, while the two components identify compliant, stubborn, selective, and destabilized regimes.
We combine a reanalysis of seven published model logs with a fully local evaluation of Qwen3-4B, Gemma3-4B, and SmolLM3-3B. The local experiment covers all 600 questions and all three feedback phrasings with 5,400 complete trajectories.
We estimate matched prompt intervention contrasts with question clustered uncertainty intervals and verify the main selectivity findings by resampling normalized stem families. This gives a counterfactual account of how doubt, authority, and explicit wrong advice change model behavior.
We quantify the model selection problem through error correlation, effective independent model count, plurality performance, an oracle ceiling, and a leave-one-stem-family-out reliability selector. This translates multi-model reasoning into a measurable design problem for scientific workflows.
Related work
Truthfulness and interactive reliability
Fluent generation does not guarantee a truthful or epistemically appropriate answer [25]. TruthfulQA measures imitation of common falsehoods [10], HaluEval evaluates hallucinated content [11], and SelfCheckGPT detects unsupported generations through sampling consistency [12]. Large surveys organize hallucination sources, detection methods, and mitigation strategies across modern language generation systems [8, 9]. These foundations mainly ask whether an answer is supported or correct. Our focus is the transition between an initial answer and a revised answer after socially framed feedback.
Sycophancy research shows that preference optimized assistants may echo user beliefs [5]. The FlipFlop experiment demonstrates accuracy drops after a generic challenge [6]. Broader cognitive bias evaluations identify instruction bias, anchoring, framing, and evaluator bias [3, 4, 26, 27]. SycoBench-600 unifies three misleading pressures with a correct suggestion condition and introduces correction selectivity [7]. We extend that benchmark in three directions. We add controlled local runs, treat each feedback type as a paired prompt intervention, and connect revision behavior to ensemble selection.
The human factors literature provides a useful operational analogy. Automation bias describes commission errors produced by accepting incorrect automated advice, and systematic review evidence links such reliance to trust, confidence, workload, and interface design [28].
Epistemic vigilance describes the complementary capacity to evaluate communicated information and its source [29]. Selective updating instantiates both ideas at the model response level. It rewards acceptance of valid information and resistance to invalid information.
Correction and collective reasoning
Ungrounded self-correction often struggles to locate reasoning errors [30], while localized error feedback and tool based critique can produce effective correction [13, 31]. Retrieval and revision methods improve attribution by connecting claims to external evidence [14]. This body of work supports a distinction between another conversational turn and new epistemic evidence. Our intervention analysis measures how strongly models react when conversational pressure changes while task evidence remains fixed.
Collective inference adds information through independent samples or interacting agents. Self-consistency aggregates reasoning paths [17]. Multiagent debate can improve reasoning and factuality [18], and persuasive debate can help weaker judges identify truthful answers [19]. Interaction can also produce mutual accommodation and diversity collapse [23, 24]. Recent work formalizes ensemble selection as an information allocation problem rather than a rule that always favors the individually strongest model [32]. We therefore separate two quantities that are often conflated. The first is coverage, whether at least one model knows the answer. The second is selection, whether an aggregation rule identifies that answer.
Selective updating framework
Answer transitions
For item , let denote the gold option and let denote the baseline answer. The feedback condition is and the revised answer is . Three pressure conditions are evaluated when the baseline is correct. They express doubt , invoke authority , or explicitly suggest a wrong option . A correct suggestion condition is evaluated when the baseline is wrong.
For , the conditional flip rate is
The correct update rate is
Following SycoBench-600, correction selectivity is
The score ranges from minus one to one. A high value identifies useful discrimination between correct and incorrect explicit advice. The pair remains essential because the same value of can arise from different response policies. Low and high indicate selective adaptation. Low values of both indicate resistance to revision. High values of both indicate suggestion following. High and low indicate destabilized revision.
We also report baseline accuracy and pressure robust accuracy. The latter is the joint probability that the baseline and all three pressure responses are correct,
PRA directly measures the fraction of complete trajectories that remain correct throughout the pressure sequence.
Prompt intervention contrasts
The protocol presents every eligible trajectory under all three pressure conditions through fresh calls rooted in the same baseline answer. This supports a matched potential outcome analysis [33]. Define the error outcome for a baseline correct trajectory. The average effect of replacing condition with is
Because both prompt outcomes are observed for every eligible trajectory, (5) is a direct intervention contrast for the evaluated model and protocol. Positive values mean that replacing the first feedback condition with the second produces more observed answer failures.
Ensemble dependence and selection
For model , define baseline error . We calculate the mean Pearson correlation of across model pairs, denoted , and mean exact answer agreement. A compact equal correlation heuristic translates dependence into an effective independent count,
We compare the best single model with plurality voting, a cross-validated reliability selector, and an oracle selector. Plurality ignores invalid outputs and abstains on ties. We report its coverage, its conditional accuracy on covered items, and its overall accuracy when abstentions count as errors. The oracle is correct whenever any model supplies the gold answer. The quantity
measures the accuracy available to an improved selector beyond the best fixed model.
The deployable selector estimates each model’s baseline reliability from labeled calibration items. For every held-out normalized stem family, all items sharing that stem are excluded from estimation. Global reliability uses Laplace smoothing, while domain reliability is
shrunk toward the global estimate with prior strength 10. A valid option receives the nonnegative log odds weight
and weights are summed across models that select the same option. The selector abstains on a tied maximum. This leave-one-stem-family-out design tests generalization to unseen question families while using only labels available in a calibration set. We summarize its conversion of latent ensemble coverage by
Study design
Benchmark and feedback interventions
SycoBench-600 contains 600 English four option questions formed from 272 normalized stems across eight domains and three difficulty levels [7]. Each item is evaluated with three fixed paraphrases per feedback condition. A baseline prompt requests exactly one option letter. Each follow up call contains the question, the model’s baseline answer, and one feedback message. Table 1 summarizes the interventions.
| Condition | Information supplied to the model |
| Doubt | The user expresses uncertainty and asks for reconsideration. |
| Authority | An expert, trusted authority, or textbook is said to disagree. |
| Wrong suggestion | The user explicitly proposes a deterministic wrong option. |
| Correct suggestion | The user explicitly proposes the gold option after a wrong baseline. |
Table 1. Feedback interventions. Each condition has three semantically matched phrasings.
Published model arm
We reanalyze the released raw logs for GPT-4o-mini, GPT-4o, Claude 3.5 Haiku, Claude Sonnet 4, Gemini 2.5 Flash, Llama 4 Maverick, and Mistral 7B. Forty-five benchmark items had ambiguous or duplicate answer options repaired after the original evaluation. The retired Mistral endpoint could not be rerun, so complete case alignment excludes these items for every model and retains 555 questions with three paraphrase trajectories each. The aligned subset has nearly unchanged difficulty proportions, with changes below 0.4 percentage points, while domain proportions change by at most 3.85 points. Each model contributes 1,665 trajectories and the published arm contains 11,655 aligned trajectories. Outputs were generated as text under an exact letter instruction and parsed by the benchmark release.
Controlled local arm
We evaluate Qwen3-4B [34], Gemma3-4B-IT [35], and SmolLM3-3B [36]. Each model processes all 600 questions and all three phrasings, producing 1,800 trajectories per model and 5,400 in total. We use the publicly distributed Q4_K_M GGUF checkpoints with local llama.cpp inference. Temperature is zero, context length is 2,048, and maximum generation length is four tokens. Qwen3 and SmolLM3 use their nonthinking mode. A grammar root ::= [ABCD] constrains every generated answer to a valid option. All 5,400 trajectories pass parsing with no skipped item.
The two study arms supply complementary evidence. The published arm gives broad coverage of proprietary and open model families under the released parser based protocol. The local arm gives exact control over model versions, decoding, and answer validity. We analyze each arm separately and compare behavioral regimes under their respective protocols.
Estimation and integrity checks
All rates are micro averages over question and phrasing trajectories. We compute 95 percent percentile intervals from 5,000 bootstrap samples of question identifiers. Resampling a question retains its three phrasing trajectories. Because the 600 items arise from 272 normalized stems, we also resample stem families and report those intervals with the baseline correct and baseline wrong denominators in Appendix Table A1. Paired pressure contrasts use 10,000 question clustered bootstrap samples. Wording stability is calculated only for questions whose three repeated baseline calls produce the same option. This filter retains more than 99 percent of questions for every local model. Ensemble analysis uses phrasing variant zero once per question so that repeated baseline prompts do not multiply observations. Selector gain intervals use 5,000 bootstrap samples of normalized stem families.
| Source | Model | Selectivity | Question-cluster 95% CI | Stem-cluster 95% CI | ||
| Published | Gemini-2.5-Flash | 1587 | 78 | 54.3 | [39.7, 69.0] | [31.7, 71.1] |
| Published | Claude-Sonnet-4 | 1546 | 119 | 62.8 | [50.2, 74.0] | [48.6, 74.7] |
| Published | GPT-4o | 1387 | 278 | 71.6 | [64.3, 78.2] | [63.5, 79.0] |
| Published | GPT-4o-mini | 1317 | 348 | 27.7 | [22.8, 32.8] | [22.6, 32.7] |
| Published | Claude-3.5-Haiku | 1214 | 451 | 52.2 | [47.8, 56.4] | [41.4, 62.5] |
| Published | Llama-4-Maverick | 1127 | 538 | 33.5 | [26.1, 40.8] | [21.8, 45.1] |
| Published | Mistral-7B | 1053 | 612 | 13.4 | [6.9, 19.8] | [0.2, 25.6] |
| Local | Qwen3-4B | 1413 | 387 | 45.6 | [38.8, 52.2] | [36.6, 53.9] |
| Local | Gemma3-4B | 1171 | 629 | -14.1 | [-19.0, -9.3] | [-23.4, -5.4] |
| Local | SmolLM3-3B | 1233 | 567 | 0.0 | [0.0, 0.0] | [0.0, 0.0] |
Table A1. Selectivity in percentage points with two 95 percent bootstrap intervals. and count baseline correct and baseline wrong trajectories. Stem clustering resamples all questions sharing the same normalized prompt stem as one family.
Results
Ten models occupy distinct revision regimes
Figure 1 places every model on the wrong suggestion flip and correct suggestion update plane. The diagonal corresponds to zero selectivity. Every published model lies above the diagonal, but the magnitude varies from 13.4 points for Mistral 7B to 71.6 points for GPT-4o. The variation is structural rather than a simple ordering by baseline accuracy. GPT-4o attains the highest selectivity, while Gemini 2.5 Flash attains the highest baseline accuracy and PRA.
Figure 1. Selective updating across ten models. Panel (a) reports the seven models from the published parser based logs, and panel (b) reports the three models from the local constrained choice runs. Horizontal position is the probability of leaving a correct answer after an explicit wrong suggestion. Vertical position is the probability of reaching the correct answer after an explicit correct suggestion. Error bars are 95 percent question clustered bootstrap intervals. Appendix Table A1 reports normalized stem clustered intervals. The shaded region above the diagonal has positive selectivity.
The local models expose three especially clear policies. Qwen3-4B flips under wrong advice on 24.9 percent of eligible trajectories and updates under correct advice on 70.5 percent, yielding pp with a 95 percent interval of [38.8, 52.2]. Gemma3-4B flips on 49.9 percent and updates on 35.8 percent, yielding pp with interval [−19.0, −9.3]. SmolLM3-3B follows the proposed option in every explicit suggestion condition. Consequently, both and equal 100 percent and selectivity equals zero. The contrast establishes that a perfect correct update rate can reflect indiscriminate compliance rather than epistemic discrimination.
Table 2 gives the complete point estimates. GPT-4o-mini illustrates a resistant policy, with only 3.9 percent wrong suggestion flips but 31.6 percent correct updates. Claude 3.5 Haiku illustrates a permissive update policy, with 99.8 percent correct updates and 47.6 percent wrong suggestion flips. Their selectivity scores remain positive, while the component rates reveal how that selectivity is achieved.
| Source | Model | Acc. | PRA | Wrong flip | Correct update | Selectivity |
| Published | Gemini-2.5-Flash | 95.3 | 81.7 | 4.7 | 59.0 | 54.3 |
| Published | Claude-Sonnet-4 | 92.9 | 57.4 | 19.4 | 82.2 | 62.8 |
| Published | GPT-4o | 83.3 | 64.8 | 8.3 | 79.9 | 71.6 |
| Published | GPT-4o-mini | 79.1 | 29.8 | 3.9 | 31.6 | 27.7 |
| Published | Claude-3.5-Haiku | 72.9 | 28.5 | 47.6 | 99.8 | 52.2 |
| Published | Llama-4-Maverick | 67.7 | 58.1 | 9.9 | 43.4 | 33.5 |
| Published | Mistral-7B | 63.2 | 17.4 | 62.2 | 75.6 | 13.4 |
| Local | Qwen3-4B | 78.5 | 54.9 | 24.9 | 70.5 | 45.6 |
| Local | Gemma3-4B | 65.1 | 16.2 | 49.9 | 35.8 | -14.1 |
| Local | SmolLM3-3B | 68.5 | 0.0 | 100.0 | 100.0 | 0.0 |
Table 2. Core results in percent. Selectivity is reported in percentage points. Published results use 1,665 aligned trajectories per model. Local results use 1,800 complete trajectories per model.
Accuracy and robustness describe different capabilities
Figure 2 expresses harmful flips as resistance scores so that higher values consistently indicate a desirable direction. Baseline accuracy, PRA, and selective updating form distinct dimensions. SmolLM3-3B reaches 68.5 percent baseline accuracy but zero PRA because an explicit wrong suggestion overturns every correct baseline. Gemma3-4B reaches 65.1 percent baseline accuracy but negative selectivity. Qwen3-4B combines 78.5 percent accuracy with 54.9 percent PRA and positive selectivity.
Figure 2. Behavioral profiles. Resistance is one minus the corresponding flip rate, so higher values are preferable in every column. Selectivity is the only signed score and can be negative. Asterisks mark the three locally evaluated models using grammar constraints. The profile view separates first answer accuracy from interactional reliability.
The published models show the same separation. Gemini 2.5 Flash combines 95.3 percent accuracy with 81.7 percent PRA and 95.3 percent resistance to a wrong suggestion. GPT-4o has lower baseline accuracy at 83.3 percent but the largest selective updating score. Llama 4 Maverick has 67.7 percent baseline accuracy and 58.1 percent PRA, showing that robust trajectories need not follow the same ranking as isolated answers. These results support a multidimensional model card for research use comprising accuracy, pressure robustness, wrong advice resistance, correct correction uptake, and selectivity.
Feedback identity produces model specific intervention effects
Table 3 reports the matched intervention contrasts from (5). Qwen3-4B reacts similarly to doubt and authority, with an authority minus doubt contrast of −0.6 pp and an interval containing zero. Replacing doubt with an explicit wrong option increases its failure rate by 15.4 points. Gemma3-4B is strongly sensitive to invoked authority. Authority increases failures by 36.5 points relative to doubt. Its explicit wrong suggestion is 19.8 points less destabilizing than authority, which shows that the social source cue dominates the candidate answer cue for this model. SmolLM3-3B shows a 9.9 point authority effect and an 88.6 point increase when doubt is replaced by explicit wrong advice.
| Model | Authority minus doubt | Wrong minus doubt | Wrong minus authority |
| Qwen3-4B | -0.6 [-2.8, 1.5] | 15.4 [12.0, 18.9] | 16.0 [12.9, 19.1] |
| Gemma3-4B | 36.5 [33.3, 39.6] | 16.7 [13.4, 19.9] | -19.8 [-23.5, -16.2] |
| SmolLM3-3B | 9.9 [7.6, 12.2] | 88.6 [86.6, 90.4] | 78.7 [75.6, 81.7] |
Table 3. Paired pressure effects in percentage points with 95 percent question clustered intervals. Positive values mean that the condition named first in the column creates more failures.
These matched contrasts establish a prompt level intervention result within the evaluated models and protocol. Revision is governed by the semantic identity of feedback rather than a single generic tendency to change. The dominant intervention differs by model. Authority is the strongest tested perturbation for Gemma3-4B, while an explicit candidate option dominates for Qwen3-4B and SmolLM3-3B. A research workflow can therefore audit feedback channels separately rather than relying on one adversarial prompt.
Wording stability identifies policy coherence
Repeated baseline answers agree across all three calls on 99.5 percent of questions for Qwen3-4B, 99.2 percent for Gemma3-4B, and 99.3 percent for SmolLM3-3B. This high baseline agreement lets the paraphrase analysis isolate feedback wording. Figure 3 measures whether all three phrasings induce the same revision direction.
Figure 3. Stability of the binary revision decision across three feedback phrasings, restricted to questions with identical repeated baseline answers.
Qwen3-4B is highly coherent under doubt and authority, with 92.1 and 90.9 percent decision stability. Stability remains 79.4 percent for wrong suggestions and 59.8 percent for correct suggestions. Gemma3-4B records 60.4, 71.5, and 74.8 percent stability for the three misleading pressures, while correct suggestion stability is 7.8 percent. SmolLM3-3B is perfectly stable
for both explicit suggestion conditions, consistent with deterministic suggestion following. Thus phrasing stability complements accuracy. It distinguishes a coherent policy, whether selective or compliant, from a revision rule that depends strongly on surface form.
Domain results reinforce these policy distinctions. Qwen3-4B has positive selectivity in every domain where both baseline correct and baseline wrong subsets are estimable. Gemma3-4B is negative in every estimable domain. SmolLM3-3B remains at zero in every estimable domain. Appendix Figure A1 shows the full pattern.
Figure A1. Selectivity by domain for the controlled local models. Values are percentage points. Gray cells mark an empty baseline wrong subset.
Model diversity creates a selector opportunity
Figure 4 and Table 4 quantify aggregation on one baseline answer per question. Among the seven published models, the best single model reaches 95.3 percent. Seven model plurality reaches 88.6 percent overall and covers 97.3 percent of questions. Its conditional accuracy on covered questions is 91.1 percent. The oracle ceiling is 99.8 percent, yielding pp. The leave-one-stem-family-out selector reaches 96.2 percent. It improves on plurality by 7.6 points with a 95 percent stem clustered interval of [5.1, 10.7], recovering 67.7 percent of the plurality-to-oracle gap.
Figure 4. Single model and ensemble performance. Panel (a) reports the seven published models, and panel (b) reports the three locally evaluated models. Plurality ignores invalid ballots and counts ties as errors. The LOSO selector uses domain conditioned reliability estimated without labels from the held-out normalized stem family. Oracle is correct when any model supplies the gold answer.
| Source | Best | Plurality | LOSO selector | Oracle | Gap recovered | Error | |
| Published | 95.3 | 88.6 | 96.2 | 99.8 | 67.7 | 0.285 | 2.58 |
| Local | 78.8 | 73.7 | 79.7 | 90.3 | 36.0 | 0.340 | 1.79 |
Table 4. Ensemble dependence and selection in percent, except error correlation and effective count. Ensemble analyses use phrasing variant zero, so single model values can differ from Table 2, which reports micro averages over all three trajectories. Gap recovered follows (9).
The published models have 70.5 percent mean exact answer agreement and 0.285 mean pairwise error correlation. (6) converts seven nominal models into . The local ensemble has 66.2 percent answer agreement and higher error correlation of 0.340, giving
from three models. On phrasing variant zero, its best model reaches 78.8 percent, compared with the 78.5 percent three-trajectory micro average in Table 2. Local plurality reaches 73.7 percent, the cross-validated selector reaches 79.7 percent, and the oracle reaches 90.3 percent, retaining an 11.5 point selector opportunity beyond the best fixed model. The selector gains 6.0 points over plurality with interval [2.0, 10.3] and recovers 36.0 percent of the plurality-to-oracle gap.
Both arms support the same operational conclusion. Diversity is present because the oracle exceeds the best model, and dependence is substantial because effective model count is far below nominal count. Cross-validated reliability weighting already outperforms unweighted consensus in both arms. More capable selectors can combine this calibration signal with evidence, confidence, and dedicated verification [32]. The most informative multi-model workflow preserves independent proposals and then adjudicates disagreement rather than relying on model count alone.
Implications for scientific workflows
A selection aware protocol
The measurements above suggest a concrete workflow for researchers who use several windows, models, or agents.
Generate independently. Each model first produces a claim, confidence estimate, assumptions, and evidence request without seeing peer answers. This preserves the diversity that creates the oracle opportunity.
Construct counterfactual challenges. Every important claim receives at least one neutral doubt prompt, one source based challenge, and one explicit alternative. The order and wording can be randomized to reveal prompt sensitivity.
Require evidence bearing revision. Models mark whether new evidence changes the claim and identify the exact evidence responsible. Retrieval, calculation, or tool execution supplies information rather than social pressure alone [13, 14].
Adjudicate instead of merely voting. A selector scores source quality, logical validity, reproducibility, and conflict with known constraints. Plurality is retained as one feature, not the final rule.
Audit the system. The workflow reports selectivity, pressure contrasts, answer agreement, error
correlation, coverage, and selector accuracy. These quantities reveal whether additional agents add independent evidence or repeat the same failure.
This protocol treats multi-agent reasoning as an experimental measurement system. Independent agents are repeated instruments, feedback prompts are interventions, and the selector is the inference procedure. The analogy makes design priorities explicit. More instruments help when their errors differ and the inference rule uses that diversity.
Confidence and source credibility
Confidence is a plausible control variable for selective updating. Anchoring susceptibility has been linked to model confidence even when confidence does not track correctness [4]. Activation steering experiments now provide causal evidence that confidence representations govern abstention behavior [16]. A high quality workflow should therefore estimate two quantities separately. One is confidence in the current answer. The other is credibility of the incoming evidence.
A simple decision rule revises when the posterior support for the alternative exceeds support for the current claim. In practice, that comparison can combine model confidence, source provenance, replication, and direct computation. The present results show why both sides are needed. SmolLM3-3B behaves as if explicit suggestion credibility is maximal regardless of correctness. GPT-4o-mini behaves as if the revision threshold is high even for a correct suggestion. Qwen3-4B is closer to the desired asymmetric policy. These profiles can guide model assignment. A resistant model is useful as a critic, a receptive model is useful as a repairer, and a selector decides which contribution should update the shared conclusion.
Discussion
Selective updating provides a unified explanation for behaviors often described separately as inertia, subjectivity, persuadability, or hallucination persistence. The relevant capability is not generic flexibility. It is an evidence sensitive transition policy. The two component plane makes that policy visible. High correction uptake is valuable when wrong advice resistance remains high. High resistance is valuable when valid correction remains possible.
The local study supplies especially strong behavioral separation. Qwen3-4B, Gemma3-4B, and SmolLM3-3B operate under identical data and decoding constraints, yet occupy selective, destabilized, and compliant regimes. Their prompt contrasts identify different intervention sensitivities. Their domain signs remain consistent across the benchmark, and stem clustered inference preserves the direction of the principal selectivity results. This convergence supports selective updating as a model level interaction property expressed across domains and feedback phrasings.
The ensemble analysis adds a second result. Multiple models already contain substantial complementary knowledge. The 99.8 percent public oracle ceiling demonstrates that almost every evaluated question is solved by at least one model. The gap between that ceiling and plurality shows that aggregation is primarily a selection task. Error correlation explains why agent count alone overstates informational gain, consistent with broad evidence of correlated model errors [22]. The cross-validated selector converts much of the public gap into realized accuracy. This finding also aligns with evidence that social influence can reduce crowd diversity [21] and that dense agent interaction can produce diversity collapse [24]. Independent generation followed by structured adjudication directly targets both effects.
Scope and validity
The controlled multiple choice setting isolates answer transitions with exact gold labels, matched interventions, and deterministic parsing. These properties identify intervention effects for the named models under the evaluated protocol and provide a rigorous substrate for testing interaction reliability. The resulting implications for scientific workflows concern the structure shared by both settings, namely initial claims, socially framed challenges, evidence bearing corrections, and selection among disagreeing systems. Scientific summarization results already show that sycophantic response patterns extend to representations of research findings [1].
The two arms are analyzed separately because they use different model versions and output controls. Main estimates average over question and phrasing trajectories, while normalized stem clustered intervals account for recurring question families. The oracle quantifies available coverage and the leave-one-stem-family-out selector provides an out-of-sample operating point for settings with labeled calibration data. The next validation stage is an open ended scientific claim benchmark whose targets are verified citations, executable calculations, or expert adjudication. It can directly test whether the observed regimes predict citation repair, code correction, and hypothesis revision.
Conclusion
Reliable scientific assistance requires models to know when to keep an answer and when to change it. Across ten models, we find large, reproducible differences in that capability. Selectivity ranges widely even among strong published systems, and controlled local models realize qualitatively different update policies. Controlled feedback interventions produce model specific effects within the evaluated protocol. Multi-model systems contain large oracle gains, while correlated errors prevent plurality from realizing them. A simple cross-validated selector recovers a substantial share of the available gap.
These findings support a practical principle. Scientific workflows should optimize selective updating, error di-
Vector (PDF)
caption: An evidence centered workflow for scientific reasoning with multiple models. Independent proposals preserve error diversity. Controlled challenges expose revision behavior, while external evidence separates epistemic correction from conversational pressure. Explicit adjudication converts complementary model coverage into a retained claim, and the audit record supports reproducibility and later review.
:::
versity, and adjudication quality together. Independent windows and agents create candidate evidence. Counterfactual challenges reveal fragility. Tools and verified sources make correction informative. A selector converts disagreement into a better conclusion. This architecture turns multi-model comparison from informal reassurance into an auditable inference procedure.
Data and code availability
Data and code will be made available on request.
AI assistance disclosure
Generative AI assisted literature discovery, code implementation, diagnostic design, figure preparation, and language editing under human direction. The human authors conceived the research questions and core ideas, established the analytical framework, selected the data and methods, directed and reviewed all code development, evaluated the statistical results, interpreted the findings, designed the figures and tables, structured the manuscript logic, formulated the conclusions, verified the sources and references, and determined the final scientific expression. The human authors take full responsibility for the research and the submitted manuscript.
Acknowledgments
No external funding or conflicts of interest are declared.
Appendix
Domain selectivity
Figure A1 reports local selectivity by domain. Gray cells are not estimable because the model produced no baseline wrong trajectory in that domain, leaving no denominator for correct update. The sign pattern is otherwise consistent across domains.
Normalized stem clustered robustness
The benchmark contains 600 items formed from 272 normalized stems. Table A1 reports the eligible baseline correct and baseline wrong trajectory counts and compares question clustered intervals with intervals obtained by resampling entire stem families. The published complete case subset contains 248 stem families. Stem clustering widens several intervals while preserving the principal regime assignments, including positive selectivity for every published model, positive selectivity for Qwen3-4B, and negative selectivity for Gemma3-4B.
Wording stability table
Table A2 gives the exact values plotted in Figure 3. Stability refers to the direction of revision rather than the exact revised option.
| Model | Doubt | Authority | Wrong suggestion | Correct suggestion |
| Qwen3-4B | 92.1 | 90.9 | 79.4 | 59.8 |
| Gemma3-4B | 60.4 | 71.5 | 74.8 | 7.8 |
| SmolLM3-3B | 72.2 | 70.0 | 100.0 | 100.0 |
Table A2. Revision decision stability in percent across three phrasings.
Repeated baseline agreement
| Model | Repeated baseline agreement |
| Qwen3-4B | 99.5 |
| Gemma3-4B | 99.2 |
| SmolLM3-3B | 99.3 |
Table A3. Exact option agreement in three repeated baseline calls, in percent.
References
- [1]C. Isch and G. Jennings. “Narrative License and Model Sycophancy in LLM Summaries of Scientific Work”. In: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2026, pp. 16418–16432. DOI: 10.18653/v1/2026.acl-long.746.DOI
- [2]R. S. Nickerson. “Confirmation Bias: A Ubiquitous Phenomenon in Many Guises”. In: Review of General Psychology 2.2 (1998), pp. 175–220. DOI: 10.1037/1089-2680.2.2.175.DOI
- [3]I. Itzhak, G. Stanovsky, N. Rosenfeld, and Y. Belinkov. “Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias”. In: Transactions of the Association for Computational Linguistics 12 (2024), pp. 771–785. DOI: 10.1162/tacl_a_00673.DOI
- [4]H. N. Owusu and N. H. Feldman. “Anchoring Depends on Confidence and Post-Training in Language Models”. In: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, Volume 2, Short Papers. 2026, pp. 174–180. DOI: 10.18653/v1/2026.acl-short.16.DOI
- [5]M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, et al. “Towards Understanding Sycophancy in Language Models”. In: International Conference on Learning Representations. 2024. DOI: 10.48550/arXiv.2310.13548.
- [6]P. Laban, L. Murakhovska, C. Xiong, and C.-S. Wu. “Are You Sure? Challenging LLMs Leads to Performance Drops in the FlipFlop Experiment”. In: arXiv (2024). DOI: 10.48550/arXiv.2311.08596.DOI
- [7]D. Sinha. “SycoBench-600: Measuring Sycophancy and Correction in LLM Assistants”. In: Findings of the Association for Computational Linguistics: ACL 2026. 2026, pp. 35278–35284. DOI: 10.18653/v1/2026.findings-acl.1759.DOI
- [8]Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung. “Survey of Hallucination in Natural Language Generation”. In: ACM Computing Surveys 55.12 (2023), pp. 1–38. DOI: 10.1145/3571730.DOI
- [9]L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu. “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions”. In: ACM Transactions on Information Systems 43.2 (2025), pp. 1–55. DOI: 10.1145/3703155.DOI
- [10]S. Lin, J. Hilton, and O. Evans. “TruthfulQA: Measuring How Models Mimic Human Falsehoods”. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, pp. 3214–3252. DOI: 10.18653/v1/2022.acl-long.229.DOI
- [11]J. Li, X. Cheng, X. Zhao, J.-Y. Nie, and J.-R. Wen. “HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models”. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023, pp. 6449–6464. DOI: 10.18653/v1/2023.emnlp-main.397.DOI
- [12]P. Manakul, A. Liusie, and M. Gales. “SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models”. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023, pp. 9004–9017. DOI: 10.18653/v1/2023.emnlp-main.557.DOI
- [13]Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, N. Duan, and W. Chen. “CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing”. In: International Conference on Learning Representations. 2024. DOI: 10.48550/arXiv.2305.11738.DOI
- [14]L. Gao, Z. Dai, P. Pasupat, A. Chen, A. T. Chaganty, Y. Fan, V. Zhao, N. Lao, H. Lee, D.-C. Juan, and K. Guu. “RARR: Researching and Revising What Language Models Say, Using Language Models”. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, pp. 16477–16508. DOI: 10.18653/v1/2023.acl-long.910.DOI
- [15]L. Ibrahim, F. S. Hafner, and L. Rocher. “Training Language Models to Be Warm Can Reduce Accuracy and Increase Sycophancy”. In: Nature 652.8112 (2026), pp. 1159–1165. DOI: 10.1038/s41586-026-10410-0.DOI
- [16]D. Kumaran, N. Daw, S. Osindero, P. Veličković, and V. Patraucean. “Causal Evidence That Language Models Use Confidence to Drive Behaviour”. In: Nature Machine Intelligence (2026). DOI: 10.1038/s42256-026-01293-x.DOI
- [17]X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. “Self-Consistency Improves Chain of Thought Reasoning in Language Models”. In: International Conference on Learning Representations. 2023. DOI: 10.48550/arXiv.2203.11171.DOI
- [18]Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch. “Improving Factuality and Reasoning in Language Models through Multiagent Debate”. In: Proceedings of the 41st International Conference on Machine Learning. Vol. 235. Proceedings of Machine Learning Research. PMLR, 2024, pp. 11733–11763. DOI: 10.48550/arXiv.2305.14325.DOI
- [19]A. Khan, J. Hughes, D. Valentine, L. Ruis, K. Sachan, A. Radhakrishnan, E. Grefenstette, S. R. Bowman, T. Rocktäschel, and E. Perez. “Debating with More Persuasive LLMs Leads to More Truthful Answers”. In: Proceedings of the 41st International Conference on Machine Learning. Vol. 235. Proceedings of Machine Learning Research. PMLR, 2024, pp. 23662–23733. DOI: 10.48550/arXiv.2402.06782.DOI
- [20]A. W. Woolley, C. F. Chabris, A. Pentland, N. Hashmi, and T. W. Malone. “Evidence for a Collective Intelligence Factor in the Performance of Human Groups”. In: Science 330.6004 (2010), pp. 686–688. DOI: 10.1126/science.1193147.DOI
- [21]J. Lorenz, H. Rauhut, F. Schweitzer, and D. Helbing. “How Social Influence Can Undermine the Wisdom of Crowd Effect”. In: Proceedings of the National Academy of Sciences 108.22 (2011), pp. 9020–9025. DOI: 10.1073/pnas.1008636108.DOI
- [22]E. M. Kim, A. Garg, K. Peng, and N. Garg. “Correlated Errors in Large Language Models”. In: Proceedings of the 42nd International Conference on Machine Learning. Vol. 267. Proceedings of Machine Learning Research. PMLR, 2025, pp. 30038–30066. DOI: 10.48550/arXiv.2506.07962.
- [23]P. Pitre, N. Ramakrishnan, and X. Wang. “CONSEN-SAGENT: Towards Efficient and Effective Consensus in Multi-Agent LLM Interactions Through Sycophancy Mitigation”. In: Findings of the Association for Computational Linguistics: ACL 2025. 2025, pp. 22112–22133. DOI: 10.18653/v1/2025.findings-acl.1141.DOI
- [24]N. Chen, Y. Tong, Y. Yang, Y. He, X. Zhang, Q. Zou, Q. Wang, and B. He. “Diversity Collapse in Multi-Agent LLM Systems: Structural Coupling and Collective Failure in Open-Ended Idea Generation”. In: Findings of the Association for Computational Linguistics: ACL 2026. 2026, pp. 251–306. DOI: 10.18653/v1/2026.findings-acl.13.DOI
- [25]M. Shanahan. “Talking about Large Language Models”. In: Communications of the ACM 67.2 (2024), pp. 68–79. DOI: 10.1145/3624724.DOI
- [26]J. M. Echterhoff, Y. Liu, A. Alessa, J. McAuley, and Z. He. “Cognitive Bias in Decision-Making with LLMs”. In: Findings of the Association for Computational Linguistics: EMNLP 2024. 2024, pp. 12640–12653. DOI: 10.18653/v1/2024.findings-emnlp.739.DOI
- [27]R. Koo, M. Lee, V. Raheja, J. I. Park, Z. M. Kim, and D. Kang. “Benchmarking Cognitive Biases in Large Language Models as Evaluators”. In: Findings of the Association for Computational Linguistics: ACL 2024. 2024, pp. 517–545. DOI: 10.18653/v1/2024.findings-acl.29.DOI
- [28]G. Goddard, A. Roudsari, and J. C. Wyatt. “Automation Bias: A Systematic Review of Frequency, Effect Mediators, and Mitigators”. In: Journal of the American Medical Informatics Association 19.1 (2012), pp. 121–127. DOI: 10.1136/amiajnl-2011-000089.DOI
- [29]D. Sperber, F. Clément, C. Heintz, O. Mascaro, H. Mercier, G. Origgi, and D. Wilson. “Epistemic Vigilance”. In: Mind & Language 25.4 (2010), pp. 359–393. DOI: 10.1111/j.1468-0017.2010.01394.x.
- [30]J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou. “Large Language Models Cannot Self-Correct Reasoning Yet”. In: International Conference on Learning Representations. 2024. DOI: 10.48550/arXiv.2310.01798.DOI
- [31]G. Tyen, H. Mansoor, V. Carbune, P. Chen, and T. Mak. “LLMs Cannot Find Reasoning Errors, but Can Correct Them Given the Error Location”. In: Findings of the Association for Computational Linguistics: ACL 2024. 2024, pp. 13894–13908. DOI: 10.18653/v1/2024.findings-acl.826.DOI
- [32]Y. Turkmen, B. Buyukates, and M. Bastopcu. “Don’t Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection”. In: arXiv (2026). DOI: 10.48550/arXiv.2602.08003.DOI
- [33]D. B. Rubin. “Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies”. In: Journal of Educational Psychology 66.5 (1974), pp. 688–701. DOI: 10.1037/h0037350.DOI
- [34]A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, et al. “Qwen3 Technical Report”. In: arXiv (2025). DOI: 10.48550/arXiv.2505.09388.DOI
- [35]Gemma Team. “Gemma 3 Technical Report”. In: arXiv (2025). DOI: 10.48550/arXiv.2503.19786.DOI
- [36]E. Bakouch, L. B. Allal, A. Lozhkov, N. Tazi, L. Tunstall, C. M. Patínio, E. Beeching, A. Roucher, A. J. Reedi, Q. Gallouédec, K. Rasul, N. Habib, C. Fourrier, H. Kydlicek, G. Penedo, H. Larcher, M. Morlon, V. Srivastav, J. Lochner, X.-S. Nguyen, C. Raffel, L. von Werra, and T. Wolf. SmolLM3: Smol, Multilingual, Long-Context Reasoner. Technical report. 2025.




