Quantifying Claim Smoothing by LLM Research Agents

Introduction

Large Language Models (LLMs) are often deployed within systems that read across many sources of information and return long-form reports with citations [9, 10, 12, 11]. These systems, either general-purpose agents paired with Internet access or special-purpose “deep research” agents, retrieve large volumes of documents and analyze and synthesize this information into coherent reports [5, 13]. The skills required to do this go beyond simple multi-document summarization  [21]: agents must weigh evidence, integrate conflicting findings, and determine what the evidence warrants [14, 15, 17]. The resulting reports are typically fluent, well-organized, and densely cited, which makes them read as confident, authoritative syntheses. As a result, users increasingly treat these outputs as substitutes for reading the underlying literature, even when the synthesis misrepresents or overstates what the cited sources actually support [16, 6].

In this work, we study the prevalence of smoothing in LLM generated reports. We define smoothing as the tendency of LLMs to synthesize claims that are more decisive than supported by the documents they cite. Such claims may, for example, fail to portray mixed evidence or overclaim a result to hold in all circumstances when it has only been shown more narrowly. By contrast, a faithful claim should hedge when the evidence is mixed and narrow its scope when the support is narrow.

To identify instances of this behavior, we propose SmoothScore, a framework for measuring whether the claims in a synthesized report are calibrated to the cited documents. As shown in Figure 1, we find that an agent’s report can present a claim as well supported by some of its citations, even though other retrieved evidence shows the claim only holds under certain conditions. The bulk of prior work on claim verification and long-form factuality [1, 19, 18] can check whether individual claims are supported by their cited evidence, but breaks down for synthesized claims: these systems usually return binary judgments at the claim or subclaim level, but do not handle calibration and scoping across a large body of potentially conflicting evidence.

To compute SmoothScore, we first decompose each report into its main conclusive claims and identify the documents the report cites. As shown in Figure 2, for each claim, we run two agents over the report’s cited documents: (a) Support Agent: which surfaces evidence consistent with the claim, and (b) Skeptic Agent: which actively searches for evidence that qualifies, limits, or contradicts the claim. Evidence from both these passes are clustered into aspects reflecting different dimensions of the claim, such as viewpoint, strength of argument, and certainty [20]. Given the claim and its mapped aspect-evidence pairs, we score the claim’s calibration on a 1–5 scale, reflecting how well its stated strength and scope match the evidence collected across both passes.

Across four benchmarks and seven systems, we find that smoothing is pervasive. On average, for each system at least 25% of generated claims received a SmoothScore $\geq 3$. For most systems over half of claims show at least material smoothing, and up to 35% of Gemini Deep Research’s claims show severe smoothing ($\geq 4$). We also find that limited interaction with the cited documents contributes to smoothing: Luna opens only 19% of the pages it cites, and actively engaging with the full documents can help reduce smoothing.

Contributions Our main contributions are: (a) We introduce SmoothScore, a framework for measuring whether the claims in a synthesized report are calibrated to the strength and scope of their cited evidence, rather than merely supported by it and (b) we find this issue to be widespread across frontier systems on 4 different benchmarks.

Smoothing in LLM Research Agents

Defining Smoothing

Setup and Definitions

Given a prompt $p$, a retrieval-augmented agent $M$ produces a long-form response or report $y$ with inline citations. We collect the full text of the cited documents in the report $D = \{d_1,\ldots,d_m\}$. Our goal is to measure if the synthesized conclusive claims in $y$ faithfully represent what $D$ supports.

We call a claim smoothed if it satisfies three criteria:(1) it reflects a conclusion drawn from aggregated evidence; (2) some of the evidence supports at least part of the conclusion; and (3) the conclusion as a whole is stronger or broader than the body of the evidence warrants.

This goal is closely related to hallucination detection [37, 1, 18]; we view it as a special case of hallucination detection. However, typical methods for hallucination detection assume binary factuality judgments. Past work has looked at more nuanced judgments for narrative summarization [35] and opinion summarization [36], but these methods are designed for simpler settings and do not handle the complexity of smoothing in deep research systems.

Evaluating whether a claim is smoothed requires more than checking whether a report’s statements are supported by individual citations. For instance, in Figure 2, the extracted claim ‘discrete image-token modeling appears more promising than a Stable-Diffusion-centered architecture’ is a conclusion made by the deep research system (satisfying condition (1) above). It is supported by the cited documents, $D_1$ Emu3 paper [25]) and $D_3$ (the Show-o paper [24]) (satisfying condition (2)). However, we find that the same sources along with other documents, namely $D_2$ (the Transfusion paper [26]), also contains results that show that the statement is only true under certain conditions present counter evidence (satisfying condition (3)). Therefore, smoothing can occur even when a claim appears supported, and methods producing binary judgments from small evidence sets cannot comprehensively capture the distribution to identify it.

Identifying smoothing via aspects

We propose SmoothScore as a way of measuring the degree of smoothing a conclusion shows. Figure 2 outlines the pipeline. At a high level, first, we extract such claims from a generated report that represent the key takeaways, then find evidence that supports and refutes those claims. However, it is not straightforward to combine this evidence into a final score: counting supporting documents vs. refuting documents does not account for the nuances of how a claim is supported or refuted.

Instead, for the claim $c$ in the report $y$ and evidence from a cited document $d \in D$, we identify a set of dimensions $A(c,d)=\{a_1,\ldots,a_k\}$ that we call aspects, following [20]. These aspects are consolidated across documents, to capture fine-grained ways in which a claim might be refuted. For instance, in Figure 2, step 3, we are able to find ‘performance and efficiency of tokens vs diffusion models’, under which we see some evidence state that ‘tokens match or beat diffusion’ but we also see evidence for ‘diffusion or hybrid set ups are competitive’. As aspects are discovered per claim and tailored to the claim and its evidence, we use them to score claim against the aspect-organized evidence to quantify its degree of smoothing.

Measuring Smoothing with SmoothScore

Step 1: Claim Extraction

To focus on the agents’ ability to synthesize from aggregated evidence, we extract conclusion-bearing claims (condition (1)). These are cases where report takes a stance or presents a takeaway. For example, in Figure 2, ‘image-token modeling appears more promising than a Stable-Diffusion-centered architecture’ is a takeaway from the report which we extract as a claim.

Using prompt, Appendix § 10.1, we extract,

Each claim preserves its direction, confidence, scope, and qualifications as stated in the report.

Step 2: Evidence Extraction

In Figure 2, we see that a cited document, for example, $D1$ can carry information relevant to the claim that can both support or undermine it. To identify information relevant to the claim from the cited documents throughout the report, we extract evidence using a ‘support agent’ and a ‘skeptic’ agent.

• Support Agent (Appendix § 10.2): Given (claim, document), this agent extracts evidence that supports the assertions made by the claim. We also include stated concession as evidence when they confirm limitations or conditions present within the claim.

  $$S(c) = \bigcup_{d \in D} \operatorname{Extract}_{+}(c,d)$$

• Skeptic Agent (Appendix § 10.3): Given (claim, document), this agent extracts evidence that contains any information against the claim. To capture the “jaggedness” we go beyond just finding counters to the assertions made by the claim. The agent also extracts evidence to support limitations and conditions stated in the claims. Additionally, we include evidence on alternate approaches and their success related to the claims where relevant. This is to provide context for scope.

  $$R(c) = \bigcup_{d \in D} \operatorname{Extract}_{-}(c,d)$$

For each agent, the evidence is composed of: a source quote from the document, aspect detailing the assertion, a note on how the evidence bears on the claim, and strength rating.

Step 3: Aspect Induction

Different cited documents can be relevant to different parts of a claim, so judging the claim against the raw documents forces the judge to compare evidence that is not directly comparable. If we look across the cited documents, however, the relevant information naturally clusters into topics that bear on the claim. For example, in Figure 2, some documents discuss comparing the performance and efficiency of image tokens vs. diffusion approaches, while others discuss fit with LLM-style next-token scaling. We refer to these topics as aspects [20].

To construct aspects, for each claim–document pair $(c_i,d_j)\in C\times D$ we generate a set of document-specific aspect proposals,

Each proposal identifies a dimension discussed in $d_j$ that bears on $c_i$, and consists of a short name, a brief description, and a set of keywords. If $d_j$ contains nothing relevant to $c_i$, then $A_{ij}=\varnothing$ (see Appendix § 10.4 for the prompt). Because proposals are generated independently for each document, different documents will often propose overlapping or near-duplicate aspects. For each claim, we therefore merge the proposals across documents into a single, deduplicated set of aspects, keeping track of which documents contributed to each one (see Appendix § 10.5).

Organizing evidence by aspect places findings that address the same part of the claim side by side, regardless of which document they came from. This allows the judge to weigh evidence both supporting and opposing the claim on a like-for-like basis. For example, along a comparative dimension there may be supporting and countervailing evidence for either stance, as seen in aspect A2 of Figure 2.

Step 4: Evidence Assignment

We assign every extracted evidence item in $E(c_i)=S(c_i)\cup R(c_i)$ to the aspect it discusses, so that findings addressing the same part of the claim are grouped together and can be compared directly (prompt in Appendix § 10.6). Assignment depends only on what the evidence discusses, not on whether it supports or counters the claim, so an item’s stance, source, and strength rating are left unchanged. For example, in Figure 2, the support-agent finds that Emu3 beats SDXL in human evaluation and the skeptic-agent finding that Emu3’s own table shows a lower GenEval score than SDXL are placed under the same aspect, even though they point in opposite directions.

Step 5: Computing Smoothing

Finally, a judge $J$ produces a SmoothScore $\sigma_i$ for each claim. The judge receives the input question $q$, the claim $c_i$, and all extracted evidence $E_i$ organized under the claim’s aspects $T_i$, including evidence that is not tied to any single aspect:

The judge assesses how faithfully the claim compresses the supplied evidence, focusing on whether it omits or understates assertions, conditions, or scope in a way that changes its central takeaway. Using the five-point rubric in  § 10.7 we score degree of smoothing (condition (3)). The criteria for receiving a score of 5 includes cases that are pure hallucination and don’t match condition (2). However, we rarely observe cases of 5 (§ 4) and therefore view our system as primarily a method for analyzing smoothing rather than broader hallucination detection (even though it can identify such cases). Figure 2 and Table 13 show examples of the generated outputs for each stage of the pipeline.

Experimental Setup

Datasets.

We calculate SmoothScore on the following datasets: (1) ResearcherBench [7], (2) ScholarQA-CS2 (Expert-Written) [8] (3) DeepResearch Bench II [22] (4) ResearchRubrics [23]. These provide a range of tasks curated for research report generation. To ensure all queries require a degree of multi document synthesis we manually select queries. Details on this selection process is available in Appendix  § 9.2

Systems.

To check if overclaiming persists across architectures with varying design considerations, we evaluate the following 7 systems: (1) GPT Luna 5.6 + Websearch, (2) GPT Sol 5.6 + Websearch, (3) Claude Sonnet 5 + Websearch, (4) Claude Opus 5.5 + Websearch, (5) Gemini Deep Research, (6) DR Tulu [9], (7) and QUEST [10]. Together, these systems span several current approaches to generating research reports, including agents augmented with Websearch, proprietary end-to-end deep-research systems, and open research agents with specialized retrieval and reasoning pipelines. Addtional details provided in Appendix § 9.3

Metrics

We report SmoothScore in two ways: (1) the mean SmoothScore across claims for each system and benchmark, and (2) the fraction of claims in each SmoothScore bin. We believe the fraction of claims $\geq$3 indicates how often systems generate materially misleading claims.

Results

SmoothScore Results

Smoothing is widespread.

Table 1 shows that smoothing is pervasive across systems and datasets. Mean SmoothScore ranges from 1.66–1.95 for Luna and 1.82–1.93 for Sol to 3.05–3.27 for Gemini, with every other system falling between 2.20 and 2.88. The severity of smoothing is also reflected in the score distribution breakdown: smoothing of $\geq$3 is present on every system and dataset, with at least 22% of claims affected in every case, but severe smoothing ($\geq$4) is where systems diverge sharply. Gemini smooths severely on 22–35% of its claims, roughly 3–12 times the rate of other closed-source agents and 13-26× the rate of Sol.

		Sample	SmoothScore	% of claims w/given SmoothScore
Dataset	System	Reports	Claims	Mean [95% CI]	1	2	3	4	5
ResearcherBench	Luna 5.6	65	443	1.95 [1.85, 2.06]	37.0%	34.1%	25.7%	3.2%	0.0%
	Sol 5.6	64	430	1.93 [1.81, 2.05]	37.0%	35.6%	25.1%	2.3%	0.0%
	Sonnet 5	65	435	2.52 [2.42, 2.61]	13.8%	26.7%	53.3%	6.2%	0.0%
	Opus 5.5	65	445	2.36 [2.28, 2.44]	19.1%	29.4%	47.9%	3.6%	0.0%
	QUEST	63	404	2.50 [2.35, 2.65]	18.1%	26.7%	43.6%	10.1%	1.5%
	DR Tulu	65	408	2.57 [2.47, 2.67]	12.0%	26.7%	54.2%	6.9%	0.2%
	Gemini	64	434	3.27 [3.21, 3.34]	1.6%	5.1%	58.1%	34.8%	0.5%
CS2-Expert	Luna 5.6	25	164	1.93 [1.75, 2.11]	44.5%	23.8%	27.4%	3.0%	1.2%
	Sol 5.6	25	165	1.82 [1.66, 1.98]	43.6%	31.5%	23.6%	1.2%	0.0%
	Sonnet 5	25	167	2.30 [2.12, 2.48]	21.6%	32.3%	40.7%	5.4%	0.0%
	Opus 5.5	25	188	2.20 [2.05, 2.33]	28.7%	26.6%	42.0%	1.6%	1.1%
	QUEST	25	156	2.67 [2.46, 2.87]	17.3%	18.6%	47.4%	13.5%	3.2%
	DR Tulu	25	168	2.70 [2.53, 2.86]	12.5%	19.0%	55.4%	12.5%	0.6%
	Gemini	24	175	3.18 [3.09, 3.28]	4.0%	5.7%	59.4%	29.7%	1.1%
DRB-II	Luna 5.6	18	106	1.66 [1.46, 1.86]	58.5%	18.9%	20.8%	1.9%	0.0%
	Sol 5.6	18	111	1.83 [1.62, 2.04]	45.0%	29.7%	23.4%	0.9%	0.9%
	Sonnet 5	18	107	2.46 [2.24, 2.68]	15.0%	30.8%	47.7%	6.5%	0.0%
	Opus 5.5	18	106	2.39 [2.24, 2.53]	21.7%	26.4%	43.4%	8.5%	0.0%
	QUEST	16	95	2.88 [2.54, 3.25]	9.5%	22.1%	46.3%	14.7%	7.4%
	DR Tulu	18	102	2.49 [2.27, 2.71]	19.6%	21.6%	50.0%	7.8%	1.0%
	Gemini	18	120	3.05 [2.89, 3.23]	2.5%	12.5%	62.5%	22.5%	0.0%
ResearchRubrics	Luna 5.6	17	111	1.83 [1.57, 2.10]	45.0%	31.5%	19.8%	2.7%	0.9%
	Sol 5.6	17	108	1.93 [1.73, 2.12]	36.1%	38.9%	22.2%	1.9%	0.9%
	Sonnet 5	17	110	2.55 [2.37, 2.72]	12.7%	29.1%	49.1%	8.2%	0.9%
	Opus 5.5	17	100	2.42 [2.21, 2.61]	17.0%	29.0%	49.0%	5.0%	0.0%
	QUEST	17	104	2.88 [2.61, 3.16]	12.5%	14.4%	50.0%	19.2%	3.8%
	DR Tulu	17	99	2.63 [2.38, 2.86]	14.1%	24.2%	49.5%	9.1%	3.0%
	Gemini	17	113	3.23 [3.05, 3.40]	6.2%	1.8%	56.6%	33.6%	1.8%

Open-source agents produce severely smoothed claims

Systems follow similar patterns across datasets. The mean SmoothScore in open-source Deep Research systems like DR Tulu [9] and QUEST [10] remains comparable or slightly above that Sonnet-5. However, at least $\sim$59% and $\sim$55% of their claims score $\geq 3$, and 7–23% score $\geq 4$. From Table 9, QUEST draws from among the smallest evidence pools (roughly 10 documents and 40–55k words) so it has the least amount of evidence to compress. Despite this, while closed source systems only generate 1–2% of claims that score $= 5$, QUEST generates up to $\sim$7% and $\sim$4% such claims on DeepResearchBench II and ResearchRubrics. Appendix § 15.1 shows that QUEST often generates outputs that read as summarization rather than synthesis. The resulting claims preserve specifics faithfully while overstating scope enough to drive inflated conclusions.

Closed-source systems vary widely in smoothing severity.

Among closed-source agents with Websearch, we see that Sonnet 5 and Opus 5.5 produces almost 1.4–2.5$\times$ the fraction of claims that achieve a SmoothScore of 3–4 relative to Luna and Sol in Table 1. These systems differ in the middle of the distribution rather than at its extremes and neither of the systems produce fundamentally misleading claims (Scored 5) at a meaningful rate, at most 1.1% on any dataset. On the other hand, Luna and Sol generate at least 36% claims that show no smoothing (score $= 1$), 1.5–2.7x more than any other system across datasets.

Across all systems, Gemini performs the worst, with 85–93% of claims scoring $\geq 3$ and an average score of 3.05–3.27. Gemini also draws from significantly more documents (roughly 2–5x) than Luna or Sol with websearch, and produces the longest reports on every dataset (Table 9). This gives the agent a much larger surface to synthesize, and despite the added length, the report does not appear to compress this information more effectively. Appendix § 15.2 lists some examples of claims from the generated reports. They contain language that turns a strong result into a universal or final conclusion.

Generation features show limited within-system associations with smoothing.

From Table 9 we observe that the number of documents and report length vary across systems, which can influence the agents’ ability to synthesize and therefore SmoothScore. We check if these features of an agent predict SmoothScore in Table 10. Across systems, systems that cite more documents ($\rho = 0.37$, $p = .056$) or write longer claims ($\rho = 0.43$, $p = .02$) smooth more, with Gemini at the extreme, but within a system both relationship are negligible ($\rho = 0.03$ and $0.02$; per-system values are small and mixed in sign). Only report length shows a within system association, with a small ($\rho = 0.12$) but positive correlation in all systems, though not significant in any one alone. Number of claims shows no such connection within or across systems (except a modest negative correlation for Gemini, $\rho = -0.23$). Overall, while these features show associations with smoothing, within-system behavior suggests that other factors could also influence smoothing.

Qualitative Observations

Query framing influences SmoothScore.

We find that query framing influences how the model writes its claims. As seen in Appendix § 15.1 and  § 15.2, there are certain categories of questions where smoothing is more pronounced: (1) Comparative questions: if the query is comparative in nature, for example, ‘In multimodal pretraining, the paradigms are based on image tokens and stable diffusion, which appears more promising and why?’ or ‘Which paradigm between RL and distillation appears more promising and why?’, it can presuppose a choice from the agent where to motivate or justify a choice the agent drops important qualifications, (2) Topic overview questions: queries that require an overview of a topic, for example, ‘Describe the Sunflower conjecture, progress towards a solution, and related problems’, the agent can assemble multiple results a single conclusion or just generalize from a single finding to the whole topic.

Recurring Smoothing patterns across systems

Table 2 shows three patterns that we observe across systems: (1) Unsupported composition: Agents combine individual fragments of evidence into a stronger conclusion. Although each component may be supported, the claim presents its unsupported composition as a conclusive statement. (2) Scope expansion: A claim takes an established result for a particular setup or experimental condition and removes certain boundaries, so that a the scope for an empirical observation expands into a general conclusion. (3) Premature Closure: Claims often adopt conclusive language, using phrases like ‘strictly preserving,’ ‘the best approach,’ ‘definitively proves’ and ‘most optimal.’.

Pattern	Claim	Summarized Evidence
Unsupported composition	The aggressive deployment of duplication-aware token pruning, Modality-Aware Mixture of Experts, and adaptive halting ensures that vast, world-class parameter capacities can be maintained without triggering corresponding explosions in inference latency. : 3	Token pruning, sparse expert routing, and adaptive halting can each reduce inference cost in particular settings. However, the evidence does not evaluate their simultaneous deployment or show that their benefits compose. Each can introduce routing, pruning, synchronization, or kernel overheads, so the combined system does not guarantee bounded latency at frontier scale.
Scope expansion	Self-attention-based models are highly parallelizable and require substantially fewer parameters, making them much more computationally efficient, less prone to overfitting, and easier to fine-tune for domain-specific tasks : 4	Some ViTs achieve strong performance with lower pre-training compute in particular comparisons, but these advantages do not generalize across models or regimes. ViTs often have comparable or larger parameter counts, global attention has quadratic cost, and ViTs can be more prone to overfitting in low-data settings without large-scale pre-training or careful regularization.
Premature closure	This innovation fundamentally resolves the statistical heterogeneity (Non-IID) problem while strictly preserving the privacy boundaries established by institutional firewalls. : 5	Federated distillation can mitigate heterogeneity and reduce direct data sharing, but it does not eliminate either problem. Performance continues to degrade as heterogeneity increases, theoretical analyses retain residual bias, and privacy attacks remain possible.

Validating SmoothScore

We validate SmoothScore by running a human study as well as checking the quality of evidence retrieved by our proposed pipeline.

Human Validation

To validate SmoothScore, two annotators independently scored the same 73 claims using the same rubric given to the LLM judge (Appendix § 10.7), without access to the identity of the system that produced each claim. We find that agreement between the judge and Annotator 1 is in a similar range to inter-annotator agreement, Table tab:agreement. Disagreements are predominantly small: the judge is within one point of Annotator 1’s score on 93% of claims and differs by two or more points on only 7%. Additionally, of the claims the judge scores $\geq 3$, 62% are also scored $\geq 3$ by Annotator 1, indicating that most judge-flagged claims reflect genuine issues. Since the two annotators themselves agree at $\kappa = 0.62$ at this threshold, the judge disagreement is expected given the subjective nature of the task.

Relevance of evidence collected by SmoothScore

	Luna	Sonnet 5
(lr)5-7 Condition	Context	Length	$\boldsymbol{\Delta}$ ($p$)	Context	Length	$\boldsymbol{\Delta}$ ($p$)
WebSearch draft (score)	–	$1.0\times$	68.3	–	$1.0\times$	64.4
Revise, no context	–	$1.6\times$	$+0.4$ ($.22$)	–	$1.5\times$	$+1.1$ ($.38$)
Revise + cited documents	117k	$2.2\times$	$+4.1$ ($.004$)	324k	$2.4\times$	$+9.7$ ($<.001$)
Revise + SmoothScore evidence	11k	$2.0\times$	$+4.4$ ($<.001$)	65k	$2.9\times$	$+12.0$ ($<.001$)

While human agreement validates the SmoothScore judge’s final scores, we separately check whether the evidence SmoothScore surfaces (Steps 2–4) is useful. We consider an independent measure, using gold rubrics provided by ResearcherBench created for judging the overall quality of the generated report. We take Luna and Sonnet 5 WebSearch reports on the benchmark, and perform an additional revision step with no web search under three context settings: (1) no additional context, (2) providing all cited documents, and (3) the collected SmoothScore evidence as context. We find that both context settings rise well above the no-context control ($+0.4$ and $+1.1$), with the extracted evidence improving rubric scores for both agents (Luna $+4.4$, Sonnet $+12.0$, both $p<.001$), matching or even exceeding revision with the full documents. The evidence achieves this with roughly one-fifth to one-tenth of the tokens of the full documents. While this does not test the quality of the evidence relative to the generated claims, it shows that the evidence captures information that matters to an external, expert-written evaluation independent of the judge’s scores.

Lack of engagement with document content

The systems we evaluate generate reports with citations from the documents they retrieve. However, the agent’s interaction with the cited material is hard to control. For Luna, the trace shows that the agent works almost entirely from search snippets truncated by the tool (median 585 characters), opening a cited page for only 19% of the documents it cites and in just 20 of 65 reports. Sonnet 5 engages even less, opening only 2% of the links it cites (Table tab:document-access, Appendix § 14). As agents rarely read the full documents, an agent may never see the qualifications and countervailing findings in the body of a source, which suggests that smoothing is partly shaped by what information the agent chooses to access.

To test whether engaging with the documents reduces smoothing, we regenerate Luna and Sonnet 5 reports under varying setups. We vary (i) evidence access: the original WebSearch setting, or the original cited documents provided in context (cited docs in context), given in context with web search off; and (ii) instructions: the original prompt, or the same prompt plus calibration instructions. The instructions are identical across both evidence settings (prompts in Appendix § 13.4, § 13.3). We score all reports with SmoothScore on ResearcherBench.

					% of claims with SmoothScore
6-10 Model	Condition	Length	Claims	Mean [95% CI]	1	2	3	4	5
Luna	WebSearch	1.0$\times$	416	1.95 [1.84, 2.07]	37.3	33.4	26.0	3.4	0.0
WebSearch + Instr.	1.0$\times$	397	1.88 [1.78, 1.98]	40.3	34.0	23.2	2.5	0.0
Docs	1.6$\times$	390	1.80 [1.70, 1.91]	45.6	31.0	21.3	2.1	0.0
Docs + Instr.	1.4$\times$	378	1.52 [1.44, 1.60]	61.1	26.7	11.6	0.5	0.0
Sonnet 5	WebSearch	1.0$\times$	407	2.54 [2.44, 2.63]	13.5	25.6	54.8	6.1	0.0
WebSearch + Instr.	1.1$\times$	391	2.23 [2.12, 2.34]	25.6	30.2	40.4	3.8	0.0
Docs	1.2$\times$	394	2.38 [2.27, 2.49]	18.3	31.0	45.4	5.3	0.0
Docs + Instr.	1.3$\times$	367	1.78 [1.67, 1.88]	49.9	25.1	22.9	1.9	0.3

We find that neither intervention is sufficient on its own (Table 4). For Luna, the instructions without the documents barely change smoothing (mean $\sigma$ 1.95 to 1.88), and the documents alone reduce it modestly (1.80). Sonnet 5 responds more to instructions alone (2.54 to 2.23) but little to the documents alone (2.38). Combined, however, documents and instructions reduce smoothing far more than either does separately: mean $\sigma$ falls to 1.52 for Luna and 1.78 for Sonnet 5, and the share of materially smoothed claims ($\sigma \geq 3$) drops from 29% to 12% and from 61% to 25%, removing more than half in both cases. Severe smoothing ($\sigma \geq 4$) nearly disappears (0.5% and 2.2%). Instructions to calibrate are thus most effective when the agent has the evidence needed to act on them.

Reports written from the full documents are longer (1.2 -1.6$\times$) but contain slightly fewer extracted claims, so the reduction does not come from shorter or emptier reports. We also observe this qualitatively, the claims do not necessarily change the takeaway. Instead they tend to drop conclusive language supporting the scope of the evidence in documents and provide the required limitations as needed. Table 5 shows that the central takeaway remains the same and smoothing tracks the added reasoning.

Condition	Claim
WebSearch SmoothScore: 4	Protection should be soft and probabilistic, not permanently frozen: an important parameter may eventually be revised if sufficient contradictory evidence accumulates.
Judge rationale	Requires recasting: the prescription excludes hard freezing and non-probabilistic approaches despite evidence supporting their effectiveness.
Docs + instructions SmoothScore: 1	Previously useful parameters or subnetworks should be protected from destructive updates. This can be implemented through frozen columns, masks, structural isolation, or soft importance-weighted constraints [doc 2][doc 3][doc 5].

Related Work

Claim decomposition and aspect-based verification

A standard recipe for verifying long-form text is to decompose it into smaller checkable units and score each against a knowledge source  [1, 2, 3]. These methods generally return a binary decision per claim, which is a good granularity for factual precision, but it does not work for synthesized claims that can have varying degrees of smoothing. ConflictScore [4] is the closest to our setting, and it extends this atomic claim recipe in a direction close to ours. They mark a claim as supported, contradicted or irrelevant per document retrieved, and aggregate these into claim-level conflict measures. However, a claim can be labeled supported by every one of its cited documents and still smooth over information it omits. Our work instead follows [20] in organizing evidence along aspects (e.g., viewpoint, strength, certainty), so that supporting and qualifying evidence on the feature of a claim can be considered against each other.

Multi-Document Summarization

In science communication, prior work measures certainty in how findings are stated [27] and models how information changes as findings move from papers to news and social media [28]. For LLM summaries of scientific text, [29] test ten widely used LLMs on about 4,900 summaries and find that most produce broader generalizations than the source texts, and that LLM summaries are nearly five times more likely than human-written ones to overgeneralize. In medicine, neural multi-document summaries of clinical trials are fluent but often factually inaccurate [30], and LLM summaries of medical evidence can misstate the certainty of the evidence or overgeneralize by omitting attributes such as a disease subtype [31]. These studies summarize a single document or a small set of abstracts. We extend the concern to long-form, multi-document research reports written by agents, and give a claim-level measure grounded in the documents a report cites.

Correctness evaluation for deep research systems

Agentic deep-research systems [5, 7, 6, 22, 23] pair rubric-based insight coverage with citation grounded factuality checks. Several works audit the reliability of these systems. DeepTRACE [14] decomposes answers into statements and finds that deep-research agents remain one-sided on debate queries and include many statements unsupported by their listed sources. [13] find that citations that resolve and are topically relevant often still fail a factual check against the cited content. [17] show that a single misleading document can induce false conclusions in final reports. [15] argue for claim-level auditability, including whether conflicts in the evidence are surfaced or suppressed. Across this literature, factuality is largely operationalized as citation validity: whether a claim is supported by some retrieved source. Our results show that this is not enough. Reports whose claims are individually supported can still present mixed or conditional evidence as a decisive conclusion, so assessing how faithfully the underlying evidence is presented requires the claim-level, evidence-calibrated analysis we propose.

Conclusion

In this work, we studied the propensity of LLMs to smooth their conclusions when generating reports based on sources. They frequently generate oversimplified statements or conclusions that aren’t really backed up by the evidence cited. Our pipeline, SmoothScore, allows us to quantify this and establish its frequency across several state-of-the-art agents. Finally, we find that this models fail to read with their own cited documents, and suggest smoothing can improved by more thorough agents engaging wih their sources.

AI Use Statement

Generative AI formed the backbone of the systems studied in this work. In addition, code written with generative AI was used to conduct and carry out the analyses here. However, the research hypotheses in this work and the design of the experiments were both human-originated. AI tools were used for assistance with revisions, but not for writing in a substantive way. The authors take responsibility for the final content of this work.

Acknowledgments

This work was supported by NSF CAREER Award IIS-2145280, NSF grant IIS-2433071, the NSF AI Institute for Foundations of Machine Learning (IFML), and the NSF under Cooperative Agreement 2421782 and the Simons Foundation grant MPS-AI-00010515 awarded to the NSF-Simons AI Institute for Cosmic Origins — CosmicAI. This work was also partially supported by the Sloan Foundation and grants from Amazon and Open Philanthropy. The research has been supported by a compute grant from NVIDIA and computing support from the Torch cluster at NYU. We also thank the members of the TAUR lab at NYU for their helpful discussions and feedback.

SmoothScore

Method Details

Aspect Induction

As mentioned in § 2, the third step in our pipeline is Aspect Induction. Since aspect proposals are generated separately for each claim–document pair, the same underlying topic is often proposed by several documents under different names or with different levels of specificity. Merging is needed so that evidence on the same topic is grouped together, rather than being split across near-duplicate aspects.

For each claim $c_i$, we consolidate the document-level proposals $\{A_{ij}\}_{j=1}^{m}$ into a hierarchy

where $V_i$ is the set of aspect and subaspect nodes and $L_i$ is the set of parent–child links between them. The merge step (i) combines semantically overlapping proposals across documents into a single node, and (ii) introduces subaspects when one aspect covers distinct but related findings. Each node retains the set of documents that proposed it, so that every piece of evidence can be traced back to its source.

Pipeline Models

Table 6 shows the models used for every stage of the pipeline for generating SmoothScore. All pipeline stages use GPT-5-mini except SmoothScore judge (Step 5), which use GPT-5; every call uses reasoning_effort=medium with default sampling.

Dataset Curation

We use all 65 ResearcherBench questions and the complete 25-question expert-written ScholarQA-CS2 set. These by construction contained open ended questions that required multi document engagement. For DeepResearch Bench II and ResearchRubrics:

• Candidate pool. We take the 66 English DeepResearch Bench II tasks and all 101 ResearchRubrics tasks.

• Annotation. We label each query on synthesis (High / Med / Low), meaning whether a good answer must synthesize a body of literature. We also label openness of question type (High / Med / Low), meaning whether the question requires one definitive correct response.

• Filter. We keep queries with synthesis $=$ High and Question Type $=$ High (Table tab:query-selection).

Step	Stage	Model	Prompt
1	Claim extraction	GPT-5-mini	App. § 10.1
2	Evidence extraction: support agent	GPT-5-mini	App. § 10.2
2	Evidence extraction: skeptic agent	GPT-5-mini	App. § 10.3
3	Aspect induction: per-document proposals	GPT-5-mini	App. § 10.4
3	Aspect induction: cross-document merge	GPT-5-mini	App. § 10.5
4	Evidence assignment	GPT-5-mini	App. § 10.6
5	SmoothScore judge	GPT-5	App. § 10.7

Benchmark	Candidates	Kept
DeepResearch Bench II (English)	66	18
ResearchRubrics	101	17

Models

Table 8 contains model IDs of the evaluated systems. We evaluate 7 systems consisting of open sourced and close source agents including frontier models.

System	Model ID	Access
GPT Luna + Websearch	gpt-5.6-luna	OpenAI Responses API
GPT Sol + Websearch	gpt-5.6-sol	OpenAI Responses API
Claude Sonnet 5 + Websearch	claude-sonnet-5	Anthropic Messages API
Claude Opus 5 + Websearch	claude-opus-5-5	Anthropic Messages API
Gemini Deep Research	deep-research-preview-04-2026	Gemini Interactions API
DR Tulu	rl-research/DR-Tulu-8B	local (vLLM + MCP server)
QUEST	QUEST-35B-RL	local

Relationship between SmoothScore and generation features

Table 9 contains report length and retrieved evidence by dataset and system and Table 10 shows if such features can predict SmoothScore with and across systems. We find that while systems show connection to different features across systems these relationships are weak and vary in direction, with only report length showing a small association.

Dataset	System	Report words	Docs/report	Evidence words
ResearcherBench	Luna 5.6	1,828	8.6	77,898
Sol 5.6	1,710	9.9	92,880
Sonnet 5	2,125	21.6	155,148
Opus 5.5	1,328	18.8	138,441
QUEST	1,332	9.8	55,481
DR Tulu	1,948	17.4	114,027
Gemini	4,860	39.2	254,916
CS2-Expert	Luna 5.6	1,999	10.4	34,821
Sol 5.6	1,514	10.8	69,483
Sonnet 5	2,177	18.5	175,065
Opus 5.5	1,431	19.7	232,211
QUEST	1,210	8.6	46,954
DR Tulu	2,156	17.8	92,667
Gemini	5,122	48.6	253,233
DRB-II	Luna 5.6	3,886	14.6	82,187
Sol 5.6	3,979	20.2	162,523
Sonnet 5	3,432	32.4	192,348
Opus 5.5	2,786	31.2	205,332
QUEST	2,397	11.2	40,119
DR Tulu	2,884	14.4	110,388
Gemini	4,844	32.7	235,720
ResearchRubrics	Luna 5.6	3,128	10.0	60,175
Sol 5.6	2,736	11.9	83,795
Sonnet 5	3,031	28.6	199,191
Opus 5.5	2,010	22.7	134,858
QUEST	1,930	9.6	52,607
DR Tulu	2,732	15.0	106,390
Gemini	4,687	31.2	196,485

A. Pooled
	Across systems	Within system	Raw pooled
2-3 Variable	$\rho$	$r$	$\rho$	$\rho$
Cited documents	$+0.37$ ($.056$)	$+0.59$ ($.001$)	$+0.03$ ($.35$)	$+0.38$
Report length	$+0.24$ ($.23$)	$+0.39$ ($.04$)	$+0.12$ ($.001$)	$+0.33$
Number of claims	$+0.08$ ($.70$)	$+0.10$ ($.63$)	$-0.02$ ($.61$)	$0.00$
Claim length	$+0.43$ ($.02$)	$+0.56$ ($.002$)	$+0.02$ ($.16$)	$+0.12$

SmoothScore Prompts

Claim Generation

Supporting Pass

Skeptic Pass

Aspect generation

Aspect Merge

Evidence Assignment

SmoothScore Judge

Judge Validation

Two authors independently scored 73 claims on the same 1–5 scale (Table 11). GPT-5, our judge of record, agrees with Annotator 1 as closely as Annotator 2 does (ordinal $\alpha$ 0.62 vs. 0.64). Gemini 2.5 Pro and Claude Sonnet 4.5 agree with the annotators less well than GPT-5 does ($\alpha$ 0.36 and 0.04), although they still rank claims similarly ($\rho$ 0.60 and 0.51).

Judge	Exact	Within 1	Spearman $\rho$	Krippendorff $\alpha$
Annotator 2	58%	96%	0.72	0.71
GPT-5 (judge of record)	56%	93%	0.69	0.68
Gemini 2.5 Pro	27%	74%	0.63	0.41
Claude Sonnet 4.5	19%	56%	0.51	0.11

All judges preserve the same system ranking as the annotator (Luna $<$ DR Tulu $<$ Gemini; Table 12), with Gemini 2.5 Pro and Sonnet 4.5 scoring systematically higher.

System	Annotator 1	GPT-5	Gemini 2.5 Pro	Sonnet 4.5
Luna	1.96	1.71	2.33	2.88
DR Tulu	2.16	2.60	2.96	3.32
Gemini Deep Research	2.71	3.04	3.83	4.42

Step by Step outputs of the SmoothScore pipeline

Table 13 shows a running example of our framework processing a claim. Each step and the corresponding output for the example is shown.

Pipeline step	Output for the example
Table 13 continued
Pipeline step	Output for the example
Input question	Will RL-based methods become the mainstream approach for improving reasoning models after DeepSeek-R1, and can reward models and tree search for chain-of-thought post-training be abandoned?
1. Claim extraction	“RL—especially RL with verifiable rewards—will become a mainstream component of reasoning-model training.”
Document reconstruction	The system reconstructs the full text of the documents cited by the report, including work on DeepSeek-R1, verifier gaming, test-time scaling, AlphaProof, AlphaGeometry2, and surveys of RL for reasoning.
2a. Supporting-evidence extraction	The supporting pass extracts findings that substantiate the claim as stated. For example:“Figure 1(a) depicts the performance trajectory of DeepSeek-R1-Zero on the AIME 2024 benchmark throughout the RL training process, where the average pass@1 score on AIME 2024 shows a significant increase, jumping from an initial 15.6% to 77.9%.”“As reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for scaling reasoning capabilities in LLMs.”Across all claim–document pairs, the pass extracts $61$ supporting items.
2b. Countervailing-evidence extraction	The countervailing pass extracts findings that contradict, weaken, or materially restrict the claim. For example:“We show that training on only 1,000 samples with next-token prediction and controlling thinking duration via a simple test-time technique we refer to as budget forcing leads to a strong reasoning model that scales in performance with more test-time compute.”“The rule-based RL training stage of DeepSeek-R1-Zero is narrowly focused on reasoning tasks, resulting in limited performance in broader areas such as writing and open-domain question answering.”Across all claim–document pairs, the pass extracts $77$ countervailing items.
3a. Local aspect induction	For each claim–document pair, the system proposes dimensions discussed by that document that bear on evaluating the claim. For example, DeepSeek-R1 proposes an aspect concerning performance gains on objectively verifiable reasoning tasks, while the test-time-scaling paper proposes an aspect concerning non-RL alternatives for improving reasoning.
3b. Aspect consolidation	Overlapping document-level proposals are merged into the following claim-level hierarchy:
4. Evidence assignment	Supporting and countervailing items are assigned to the most specific applicable aspect. For example:A1.1: Large gains on tasks with objective checksSupporting evidence:Countervailing evidence:This node contains $31$ supporting and $9$ countervailing items. Across the hierarchy, this is the only node in which supporting items predominate; the remaining dimensions contain substantial countervailing evidence.
5. Smoothing judgment	Claim role: General forecast about a field-wide training trend; it can be defeated by evidence that RL with verifiable rewards is unlikely to be widely adopted or that strong, persistent limitations preclude mainstream use.Takeaway: Reinforcement learning—particularly when rewards are verifiable—will be widely adopted as a standard part of training pipelines for reasoning models, though not necessarily the only or exclusive approach.Objections: Important constraints temper the breadth of the forecast. RL scales poorly where no reliable verifier exists; it can underperform or fail to improve smaller models; compute and evaluation costs limit practicality; and verifier imperfections introduce shortcutting and misalignment risks. Strong non-RL alternatives can also match or surpass leading RL-trained systems. These findings show that RLVR will not be universally applicable or singularly dominant across reasoning tasks.Net resolution: The claim largely holds but needs an explicit limitation: RL with verifiable rewards is poised to be mainstream for verifier-evaluable settings, yet adoption will be constrained by verifier availability, safety and shortcut risks, model and compute regimes, and viable non-RL alternatives that will continue to coexist.Repair: Append Smoothing score: $2$

Report regeneration

Setup for WebSearch

Setup for WebSearch, with Instructions

Setup for Docs in Context, with Instructions

Setup for Docs in Context, No Instructions

Improved claims under document engagement

Table 14 shows how engaging with documents can repair claims. We manually select cases where the same claims persist across setups but change in terms of scope and strength representing the cited documents accurately.

Pattern	Question	WebSearch ($\sigma$)	Docs in context ($\sigma$)	Docs + instructions ($\sigma$)
A result from one setup stated without its conditions	Analyze the differences between the LIMO and S1 papers: objectives, methodology, key findings, contributions.	LIMO appears stronger in broad benchmark performance, particularly on GPQA Diamond and MATH500. (4)	LIMO reports higher scores than original s1 on AIME24, MATH500 and GPQA, but the studies differ in evaluation details (greedy decoding vs. temperature-0) the comparison does not by itself establish a controlled head-to-head superiority claim. (1)	LIMO is the more explanatory and curation-focused paper [doc 1]. (3)
A result from one setup stated without its conditions	How can we detect hallucinations in LLMs using their internal states, and what advantages does this offer over external methods?	Lower latency and computational cost: internal detectors can operate in the same forward pass and use a lightweight probe. MIND targets real-time detection, and HIDE reports substantially lower computation time than multi-pass methods. (2)	--	Its principal advantages over external or output-only detection are earlier intervention, richer process-level information, reduced reliance on external knowledge bases, lower latency for single-pass methods, and improved interpretability. (1)
Separately supported fragments combined into one demonstrated conclusion	Could transformer architectures be reimagined to process multimodal inputs (video/audio/text) as efficiently as text?	The more promising principle is: use modality-specific perception, adaptive semantic tokenization, sparse cross-modal attention, persistent latent memory, and on-demand access to raw evidence. (4)	The evidence suggests that multimodal Transformers may need to be redesigned around asymmetric processing, learned compression, latent bottlenecks, adaptive tokenization, and selective cross-modal interaction. (3)	The likely path is to redesign the architecture so that all modalities are converted into compact, queryable representations Whether that bottleneck can remain small enough remains unresolved by the supplied evidence. (1)
Separately supported fragments combined into one demonstrated conclusion	How might federated learning combined with model distillation overcome technical and legal barriers in domains like healthcare?	The most credible approach is therefore a layered architecture: local training + distillation + secure aggregation + differential privacy + strong governance and clinical validation. (3)	The strongest approach is therefore layered: local data retention, carefully selected distillation, secure aggregation, differential privacy where appropriate and clinical validation. (3)	The strongest approach is therefore layered: [same list] Such an architecture can substantially lower both technical and legal risk, but the supplied documents do not support claiming that it eliminates either category of barrier. (1)
Conclusive or superlative wording beyond what the sources state	What is the evolution path of multimodal models from early visual representations to current multimodal LLMs, and what are the key breakthroughs?	Language as a universal interface for vision: natural language became both a source of supervision and a way to query visual representations. (3)	CLIP's central breakthrough was language-grounded open-vocabulary recognition The approach also exposed important limitations: performance depended on prompt wording, and CLIP struggled with counting, fine-grained recognition, and out-of-distribution images [doc 11]. (1)	CLIP and ALIGN demonstrated that web-scale paired text can train general visual representations, support retrieval, and produce zero-shot classifiers through natural language [docs 11, 13]. (1)
Conclusive or superlative wording beyond what the sources state	(same question)	BLIP-2 made the connection more parameter-efficient. This was an important engineering breakthrough because it showed that multimodal capability could be obtained without end-to-end retraining of every parameter. (3)	--	Flamingo and BLIP-2 showed that strong pretrained vision and language models can be connected through relatively small trainable modules rather than fully retrained end-to-end [docs 15--16]. (1)

Document Engagement Details

We measure how often each system opens the documents it cites. A cited link counts as opened if the system retrieved that page, in any form, during generation. URLs are canonicalized before matching (arXiv /abs/ and /pdf/ variants merged; tracking parameters and trailing slashes removed), and each cited link is counted once per dataset. Table 15 reports all systems and datasets.

System	ResearcherBench	CS2-Expert	DRB-II	ResearchRubrics
Luna 5.6	19.2% (95/496)	13.3% (34/256)	9.3% (27/291)	8.9% (17/190)
Sol 5.6	27.7% (169/611)	30.0% (88/293)	16.1% (68/422)	28.2% (70/248)
QUEST	73.8% (217/294)	64.9% (63/97)	30.5% (36/118)	29.4% (25/85)
Sonnet 5	2.3% (34/1,458)	3.5% (19/536)	1.1% (8/750)	1.1% (6/559)
Opus 5.5	0.9% (12/1,277)	2.3% (13/561)	1.3% (10/751)	0.2% (1/471)
DR Tulu	0.5% (5/1,082)	0% (0/445)	0% (0/303)	0% (0/290)
Gemini Deep Research	–	–	–	–

Luna, Sol, Sonnet 5, Opus 5.5, and DR Tulu.

Luna and Sol use the OpenAI Responses API web search tool, whose calls are recorded as search, page-open, or find-in-page actions [32]. We count a cited link as opened if the returned search results contain that page’s opened content, marked by a Source: open(…) header. These markers capture every page-open and find-in-page action, as well as opens returned inside other calls’ results, so counting the actions alone would miss about three quarters of them. Sonnet 5 and Opus 5.5 run with Anthropic’s web search and web fetch tools. Search results return page content in encrypted form, so we cannot observe what the model sees from search alone, while web fetch retrieves a page’s full text [33, 34]. We therefore count a cited link as opened if the system successfully fetched it. As search results are encrypted, this measures full-page retrieval only; we cannot observe how much page content the model receives from search. For DR Tulu, we count calls to its browsing tool [9]. All five systems cite mostly from search results rather than pages they open. Luna and Sol open 9–19% and 16–30% of the links they cite (45 of Luna’s 65 ResearcherBench reports open no cited page), Sonnet 5 and Opus 5.5 fetch 1.1–3.5% and 0.2–2.3%, and DR Tulu calls its browsing tool only 5 times across all datasets.

QUEST.

QUEST’s visit tool does not return the raw page content to the agent: a separate model (GPT-5-mini) reads the page and returns an extract conditioned on a goal QUEST states for the visit, consisting of selected evidence and a short summary [10]. We count a cited link as opened if QUEST issued a visit to it. Its high open rate therefore does not mean QUEST reads what it cites: the agent sees only extracts filtered toward what it was looking for, which is consistent with its high smoothing despite frequent visits.

Gemini Deep Research.

The Deep Research API returns only the final report and its citation links, with no browsing or tool steps, so engagement cannot be measured.

Examples of Smoothing in Model Generated Claims

Open-source systems

Examples of generations by open sources systems are provided DR Tulu and Quest are in Table tab:opensource-exhibits.

Pattern	System	Question	Claim (score)	Judge verdict
Unsupported composition	DR Tulu	DRB-II ex16: I am researching the challenges faced by SRAM operating at low voltages, and I hope you can help me survey and compile a technical report on mainstream solutions. The report needs to include the following …	Alternate Cells: 7T adds explicit precharge–equalize sequence, 8T virtual-ground/zero-precharge, 9T conflict-free write with equalized leakage, 10T quasi-differential read, 11T DDPS for ultra-low leakage, 12T gating and two-stage read. (4)	The claim over-compresses heterogeneous literature into a fixed one-to-one mapping and misattributes key mechanisms (notably 7T and 11T), while several other pairings are only variant-specific. It requires substantial revision to a qualified, example-based mapping rather than categorical assignments.
Unsupported composition	DR Tulu	CS2-Expert ex6: Describe the Sunflower conjecture, progress towards a solution, and related problems.	Decision trees, DNF/branching programs, and communication: Variants of the sunflower conjecture directly constrain the size/width/depth of Boolean formulas and decision trees, with tight results (e.g., $\tilde{\Omega}(k^2)$) for unambiguous DNF; these bounds, proved via explicit constructions and sunflower arguments, carry over to communication protocols and algebraic complexity analogs (e.g., matrix multiplication exponent connections surveyed by Alon). (4)	The claim over-compresses heterogeneous results: it correctly cites tight unambiguous DNF bounds and connections to communication, and separately, sunflower-variant tools and matrix-multiplication connections, but it incorrectly attributes the tight DNF bound to sunflower arguments, overstates direct constraints on decision trees/branching programs, and overstates transfer to algebraic complexity. It needs substantial narrowing and reattribution.
Premature closure	QUEST	CS2-Expert ex11: Survey recent papers on the use of LLMs for 5G network orchestration	Validates that LLM-powered prompt engineering outperforms static ML methods in highly variable O-RAN environments. (5)	The claim overstates what the evidence shows. The record supports that LLM prompting can beat simple heuristics and be useful in hybrid schemes, but it does not validate outperformance over static ML in highly variable O-RAN environments.
Premature closure	QUEST	ResearcherBench ex3: In multimodal pretraining, the current mainstream paradigms are based on image tokens and stable diffusion. Analyzing the latest advancements in these two technical approaches, with reference …	SOTA Across Multiple Benchmarks: DDT-based MLLMs outperform all previously reported models in zero-shot/non-zero-shot evaluation for image-text matching, captioning, question answering (VQA, GQA), object detection, and metric scores (GenEval 0.66, NOCAPs 124.2, EVR L1 14.5, CVS 85.5%). Even compared to diffusion specialists trained on similar data, DDT shows clear superiority in compositional edits and counterfactual interpolation, owing to its recursive, attribute-rich tokens that mimic natural language structure. (4)	The claim needs substantial revision: while the quoted metric values are accurate and DDT-based MLLMs are competitive, the evidence shows they do not outperform all previously reported models on several listed benchmarks and provides no support for object detection.

Closed-source systems

Table 17 shows examples of claims generated by closed source systems.

Pattern	System	Question	Claim (score)	Judge verdict
Scope expansion	Sonnet 5	ResearcherBench ex49: From a technical perspective, how to understand the similarities and differences between Reinforcement Learning (RL) algorithms and Supervised Fine-Tuning (SFT) in Large Language Models (LLMs), as well as	RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and visual variants, while SFT, in contrast, tends to memorize training data and struggles to generalize out-of-distribution scenarios. (4)	The claim overstates a mixed and conditional landscape. While there is solid evidence that outcome-based RL can improve OOD generalization in both textual and visual tasks and that SFT often overfits, there is also substantial, direct evidence that SFT can generalize and sometimes surpass RL, and that RL's gains depend on reward design, initialization, and compute. The conclusion needs rebalancing and explicit conditions.
Premature closure	Luna	ResearcherBench ex3: In multimodal pretraining, the current mainstream paradigms are based on image tokens and stable diffusion. Analyzing the latest advancements in these two technical approaches, with reference	For multimodal pretraining as a route toward a single general-purpose model, discrete image-token modeling appears more promising than a Stable-Diffusion-centered architecture. It fits naturally with the scaling machinery of language models: images, text, and video can be serialized into one sequence and trained with a common next-token objective. The strongest evidence comes from Emu3, Janus-Pro, Chameleon, and the rapidly improving tokenizers TiTok/FlexTok. (4)	The claim needs substantial revision. The evidence supports discrete-token, next-token training as a viable and rapidly improving path, but not that it is more promising overall than diffusion-centered approaches; multiple studies show diffusion/flow or hybrid models scaling better and being more compute-efficient, and the `natural fit' is tempered by instability and modality-specific shortcomings.
Unsupported composition	Gemini	CS2-Expert ex2: What are some of the methods of detecting GPS location spoofing attack in Connected and Autonomous Vehicles?	At the mechanical layer, the tight fusion of GNSS data with highly resilient IMU kinematics and LiDAR map-matching ensures that even if a spoofed signal perfectly replicates the cryptographic and spatial signatures of a satellite, it fundamentally cannot alter the physical reality of the road ahead. (4)	The claim needs substantial revision. The evidence supports that GNSS--IMU--LiDAR/map fusion can greatly harden localization and often detect/mitigate spoofing, but it does not ensure immunity---even against sophisticated, spatially and cryptographically convincing attacks---and has documented failure modes and delays.