betaFeedback

WorksComputer Science

Towards Full-Text Scholarly Argument Mining using Entity-Anchored Claim Extraction

Abstract

Argument mining in full-text scholarly articles remains an important yet challenging task. This difficulty stems not only from the need to model long-distance dependencies but also from the substantial effort required to annotate sentence-level argumentative components throughout lengthy documents. To address these challenges, we introduce a novel approach that leverages scientific entity mentions (e.g., tasks, datasets, and models) to identify and structure claims in scholarly literature. We validate our approach through a case study involving 50 full-text research papers on sentiment analysis. We first establish a reliable human gold standard (Krippendorff's α=0.80α= 0.80), identifying 113 claim-bearing sentences and extracting 199 structured claim records from 300 candidate sentences. To address the annotation bottleneck, we employ a majority-voting ensemble of large language models (LLMs) to generate a silver-standard dataset comprising 1,487 claims. Encoders fine-tuned exclusively on the silver-standard dataset outperform those trained on a small human-annotated gold-standard dataset and demonstrate strong zero-shot transfer to NLP papers outside the training corpus.

Introduction

Scientific literature is a rich source of complex reasoning, making it a particularly relevant domain for scientific argument mining (Teufel et al., [36]; Al Khatib et al., [4]). The central objective of scientific argument mining (SAM) is to identify argumentative components and reconstruct their relations in scholarly texts (Lawrence and Reed, [22]; Al Khatib et al., [4]). Scholarly text analysis often draws on established argumentation frameworks, such as Toulmin’s model (Toulmin, 1958) and Walton’s argumentation schemes (Walton et al., 2008), to identify claims, supporting evidence, and their argumentative relationships. However, existing research has largely focused on abstracts or individual sections of scientific articles (Accuosto and Saggion, [3]; Mayer et al., [28]; Fergadis et al., [10]; Accuosto et al., [2]). One reason for this focus is the substantial effort required for full-text annotation, as illustrated by the Sci-Arg corpus (Lauscher et al., [21]). Furthermore, identifying argumentative components and their relationships across entire documents poses substantial challenges for neural models (Binder et al., [5]).

To address these two challenges, we draw on approaches from general-domain argument mining, particularly those applied to political discourse and social media, where arguments are often identified in relation to specific controversial topics (Levy et al., [23]; Ein-Dor et al., [9]; Verma et al., 2026). In these domains, topics serve as semantic anchors that facilitate sentence-level annotation independently of broader document contexts and enable systematic comparisons of claims across different sources. Although scientific discourse exhibits comparable structures, previous research on SAM has paid relatively little attention to this anchoring strategy. Building on recent advances in scientific entity extraction (Jain et al., [18]; Otto et al., [30]), we propose a novel approach to claim extraction from scholarly articles based on scientific entity anchoring. In this approach, scientific entities (e.g., tasks, methods, and datasets) serve as structural anchors, playing a role analogous to that of “topics” in general-domain argument mining. Anchoring arguments to scientific entities reformulates full-text SAM in two fundamental ways. First, it reduces the scope of annotation to individual sentences, enabling claims to be identified and structured without requiring prior resolution of long-range discourse dependencies. Second, it facilitates cross-document comparability by providing a common structural reference for claims concerning the same method and dataset, regardless of differences in their surface formulations. We present a pipeline for entity-anchored claim extraction from full-text scientific publications. Rather than exhaustively annotating entire papers, we combine a carefully curated, independently human-annotated gold-standard dataset with a large-scale silver-standard dataset generated through consensus among three large language models (LLMs). In a case study involving 50 research papers on sentiment analysis, we annotated 294 valid candidate sentences, identifying 113 claim-bearing sentences to establish a gold-standard dataset, which we subsequently used to validate a silver-standard dataset comprising 1,487 claims. We demonstrate that encoders fine-tuned exclusively on the silver-standard dataset yield more stable claim identification than those trained on the smaller human-annotated gold-standard dataset or on data generated by a single LLM. Furthermore, encoders trained on the silver-standard dataset demonstrate strong zero-shot generalization to NLP papers outside the case study corpus. Figure 1 provides an overview of the complete pipeline.

Figure 1. Pipeline of entity-anchored claim extraction.

Related Work

Full-Text Scientific Argument Mining

Teufel [36] introduced Argumentative Zoning (AZ) to classify sentences in scholarly articles by rhetorical function, such as background, aim, and contrast. Teufel and Moens (2002) implemented AZ using hand-crafted features and Naive Bayes classifiers, while subsequent extensions captured domain-specific rhetorical functions in chemistry (AZ-II) (Teufel et al., 2009) and biology ([29]). Similarly, Liakata et al. [24] introduced the Core Scientific Concepts (CoreSC) scheme to classify sentences in full-text biochemistry articles into 11 categories using SVMs and CRFs, while Fisas et al. [11] developed the Dr. Inventor corpus to annotate discourse structure and novelty in computer graphics papers. While rhetorical zoning captures the discourse structure of scholarly articles, identifying fine-grained argumentative components, particularly claims and supporting evidence, requires more specialized annotation schemes. Blake [6] introduced a rule-based framework for distinguishing explicit, implicit, and comparative claims. Building on the Dr. Inventor corpus, Lauscher et al. [21] introduced Sci-Arg, extending its annotations with argumentative components (Own Claim, Background Claim, Data). Following this line of work, we adopt Own Claim and Reported Claim as the primary argumentative components in our schema.

Scientific Argument Mining using LLMs

Binder et al. [5] proposed a BERT-based pipeline for full-text argument mining, highlighting challenges in identifying non-contiguous argumentative units and their relations. Although LLMs offer extended context windows that may help address these limitations, their application to scientific argument mining has primarily focused on claim verification and evidence-based reasoning. Specifically, benchmarks such as SciFact-Open (Wadden et al., 2022) and SciTab ([25]) evaluate scientific claim verification against large literature collections and evidence from scientific tables, respectively. Related approaches extend to peer-review analysis, assessing the evidential support for claims in reviews [13] and linking reviewer feedback to subsequent manuscript revisions [8]. More recently, benchmarks such as CLAIM-BENCH [19] and CLAIMCHECK [31] have evaluated LLMs on scientific claim-evidence reasoning and the grounding of peer-review critiques in scholarly articles, respectively. Thus, while LLMs are increasingly used to verify and critique previously identified scientific claims, their potential for entity-anchored claim extraction from full-text scholarly articles remains comparatively underexplored.

Entity-Anchored Scientific Information Extraction

Scientific entity extraction has evolved from broad terminology extraction [26] to fine-grained annotation schemes covering tasks, datasets, and metrics (TDMSci) [16], machine learning models and architectures (GSAP-NER) [30], and subsequently to the extraction of relations between scientific entities (SciER) (Zhang et al., 2024). In parallel, argument mining has increasingly incorporated topic-based approaches that anchor arguments to specific subjects of debate [17]. Building on these advances, MetaLead (Timmer et al., 2026) introduces a richer annotation scheme linking tasks, training and test datasets, metrics, scores, and experiment types to support transparent comparisons of experimental results across scientific publications. Structured scientific knowledge increasingly supports advanced tasks such as research idea generation. For example, SciMON (Wang et al., 2024) leverages scientific concepts and prior literature to generate and iteratively refine novel research ideas. While these extraction strategies have primarily served structured knowledge organisation or idea generation, their integration into full-text SAM remains limited. We address this gap by using fine-grained scientific entities as explicit anchors for claim extraction.

Entity-Anchored Claim Extraction

We introduce Entity-Anchored Claim Extraction as the task of extracting claim structures from scholarly articles by anchoring them to scientific entities. Traditional approaches to SAM often treat every sentence or phrase as a potential claim [36, 21]. In contrast, our approach focuses exclusively on sentences containing explicit mentions of scientific entities. We define an entity-anchored claim as a contestable and verifiable assertion about one or more scientific entities (e.g., tasks, datasets, metrics, or methods). A sentence qualifies as claim-bearing if it makes an assertion about an entity’s performance, capabilities, limitations, comparative advantages, or intrinsic properties, among others.

As illustrated in Figure 2, we represent each claim as a relational tuple in which scientific entities assume explicit roles (e.g., subject or object) linked by a predefined predicate. Specifically, our controlled predicate inventory comprises four categories: performance comparisons (outperforms, underperforms, is-comparable-to), performance changes (improves, degrades, achieves), applicability and transfer (generalizes-to, fails-on, is-suited-for, is-standard-for), and dependencies and efficiency (requires, is-more-efficient-than). An additional OTHER label covers assertions outside this taxonomy. We provide detailed predicate definitions and argument constraints in Table 8 in the appendix. Furthermore, following established argument mining schemes [36, 21], we distinguish Own Claims, concerning the authors’ work, from Reported Claims, attributed to prior research.

Figure 2. Extraction of claim tuples from full text, anchored to task, dataset, and method entities.

PredicateDefinitionConstraint
outperformsSubject achieves better task performance than the object on a shared task, dataset, or metric.object
underperformsSubject achieves worse task performance than the object on a shared task, dataset, or metric.object
achievesSubject attains a stated performance level, with no named comparison object.no object
improvesSubject raises performance relative to a prior state, baseline, or component it modifies.object
degradesSubject lowers performance relative to a prior state, baseline, or component it modifies.object
generalizes-toSubject transfers to the object domain, task, language, or dataset while retaining performance.object
fails-onSubject performs unacceptably on the object input class or condition.object
requiresSubject depends on the object (resource, annotation, component) to function or to reach its reported performance.object
is-suited-forSubject is appropriate or preferable for the object setting.object
is-standard-forSubject is the conventional or default choice for the object task or setting.object
is-comparable-toSubject performs equivalently to the object on a stated task, dataset, or metric; asserts no meaningful difference, not superiority.object
is-more-efficient-thanSubject costs less than the object on a stated non-accuracy dimension; asserts nothing about output quality.object + dimension{sup}†
OTHERThe sentence is a claim but no predicate above fits; the annotator writes the predicate they would have used.free text

Table 8. Predicate inventory for describing a claim. Object: the claim must name what the subject is compared against or directed at (an entity or free text). {sup}†`Dimension: one of speed, memory, parameters, data, compute, simplicity, or other.

Case Study: Sentiment Analysis

This case study evaluates the effectiveness of our entity-anchored approach to claim extraction from scholarly articles. We investigate whether this approach enables standardized claim representations that lays foundation for the identification of implicit argumentative relations across scholarly articles beyond those captured by citation-based analysis. This evaluation requires a corpus of densely interconnected scientific publications. We therefore focus on sentiment analysis, a well-established research area in NLP with an extensive citation network.

Corpus Construction

To construct a densely interconnected corpus of sentiment analysis papers, we seeded our search with three foundational benchmark publications: IMDb ([27]), SST ([35]), and SemEval-14 ABSA ([32]). Using OpenAlex ([34]), we retrieved papers citing these benchmarks and filtered them by domain-specific keywords and open-access availability. To ensure dense conceptual connectivity, we represented the retrieved papers as a graph, connecting papers through direct citations or shared scientific entities drawn from a curated list of datasets and methods. Using community detection ([7]), we identified a densely connected cluster of 209 papers. From this community, we selected 50 papers using stratified sampling by publication year to ensure temporal diversity. We parsed the documents using GROBID ([12] (GROBID contributors 2026)). Our preprocessing pipeline preserves all in-text citations (2,274 author-year spans), which serve as essential anchors for identifying Reported Claims. We provide further details on document retrieval, community detection, and text normalization in Appendix § 11.

Scientific Entity Extraction

To anchor claims within our corpus, we first require robust extraction of scientific entity mentions. We utilize GSAP-NER ([30]), which provides a fine-grained taxonomy specifically tailored to machine learning models, datasets, and tasks. Following the original architecture, we train two SciDeBERTa-CS taggers ([20]), one for specific entity types and one for generic references, on the complete set of 100 GSAP-NER publications.

For our 50-paper corpus, we retain entity mentions from six categories: Task, Dataset, Datasource, Method, MLModel, and ModelArchitecture. We exclude generic mentions and URLs but retain citation markers (ReferenceLink) to link reported claims to cited publications. Our analysis shows that 59.3% of body-text sentences contain at least one scientific entity, providing a substantial candidate pool for entity-anchored claim extraction. We provide extraction metrics, post-processing rules, and corpus-level entity statistics in Appendix § 12.

Annotation Details

Four expert annotators (two doctoral candidates and two postdoctoral researchers), all specializing in computational linguistics, conducted the annotation. We sampled a subset of 300 entity-anchored sentences for the claim detection task. To balance workload distribution and enable reliable inter-annotator agreement (IAA) estimation, we assigned 120 instances to each annotator. All four annotators independently annotated a shared set of 60 instances, which we used to compute IAA. To ensure reproducibility, we sampled instances using a fixed random seed (20260819) and developed a dedicated annotation tool (Appendix § 13) to support the process.

Claim Identification and Structuring. Before the main annotation phase, we conducted a pilot study involving seven papers to familiarize annotators with the annotation scheme and tool. From the remaining 43 papers, we identified 4,171 eligible candidate sentences, each containing at least one scientific entity mention in the body text. We stratified the sampling by document structure and selected 300 sentences. Specifically, we oversampled sentences from the abstract, conclusion, and results sections. Conversely, we undersampled sentences from the introduction, related work, and methodology sections, which often contain procedural descriptions, standard definitions, and generic contribution statements that fall outside our claim definition.

The annotation process proceeds in two stages. First, annotators determine whether the target sentence contains a claim. If the sentence contains a claim, annotators classify it as either own_claim or reported_claim. Second, annotators structure each claim by identifying its subject and object and selecting a predicate from a predefined inventory of 13 predicates (see Appendix § 13). For reported claims, annotators additionally link each claim to its corresponding citation marker.

Experiments

Although entity-anchored claim extraction reduces annotation effort, it still requires considerable time and domain expertise. To reduce this annotation burden, we investigate automated entity-anchored claim extraction using zero-shot LLMs and fine-tuned pretrained encoders.

LLM-Based Extraction Pipeline

We benchmark three LLMs: Qwen3.6-27B, Gemma-4-31B-it, and Mistral-Small-3.2-24B. To ensure comparability, we use a common prompt template across all models, iteratively refined using Qwen3.6-27B. We extract claims in three passes:

  1. Document Profiling (Paper Card): The LLM processes the title, abstract, introduction, section headings, and conclusion to generate a paper card summarizing the paper’s major claims, proposed methods, tasks, datasets, and core contributions.

  1. Sentence-Level Extraction: The model processes target sentences in the context of their full sections to facilitate anaphora resolution. The LLM determines whether a sentence contains a claim and, if so, extracts its subject, object, and predicate according to our manual annotation scheme.

  1. Rule-Based Coreference: A deterministic post-processing step replaces generic self-references (e.g., “we”, “our model”) with the explicit method name extracted from the paper card.

Fine Tuning via Silver Data

While our zero-shot LLM pipeline identifies claims, we further investigate whether smaller, specialized encoders can achieve comparable performance more efficiently. To this end, we aggregated sentence-level claim predictions from all three LLMs across the 50-paper corpus. Using majority voting, we constructed a silver-standard dataset for claim identification from the three LLMs’ predictions. We then fine-tuned two pretrained masked language models, SciBERT (Beltagy et al. [1]) and DeBERTa-v3-large (He et al. [14]), with sequence classification heads using this dataset. We employ five-fold cross-validation at the paper level, fine-tuning a separate model in each fold on silver-standard sentences from the training papers and evaluating it on gold-standard sentences from held-out papers.

External Validation on ClaimFlow

To determine whether our entity-anchored claim extraction strategy generalizes beyond our initial case study, we sought an external dataset for validation. Existing full-text argument mining corpora are either relatively dated (Teufel [36]; Teufel et al. (2009)) or focus on domains outside NLP (Mizuta et al. [29]; Blake [6]; Liakata et al. [24]; Lauscher et al. [21]), limiting their suitability for evaluating our scientific entity extraction model. Recent resources also provide full-text claim annotations, including CLAIM-BENCH (Javaji et al. [19]), which links scientific claims to supporting evidence, and ClaimFlow (Pramanick et al. [33]), which traces claims and their relationships across NLP publications. We select ClaimFlow for its scale, NLP domain coverage, and sentence-level annotations, which align with our annotation granularity. ClaimFlow identifies central claims in NLP papers, focusing on their abstracts, introductions, and conclusions. The dataset provides paraphrased claims alongside supporting evidence sentences from the source text. We use ClaimFlow claims to evaluate our fine-tuned encoder models.

Results and Discussion

We present the results of our annotation study and evaluate the performance of LLMs and fine-tuned encoders. Unless otherwise specified, we report 95% confidence intervals (CIs) for all annotation and model evaluation results, using 10,000 bootstrap resamples.

Annotation Analysis

We evaluate inter-annotator agreement for claim detection on 60 core sentences. Excluding the annotator who worked on annotation guidelines, the three independent annotators achieved a reliable Krippendorff’s α\alpha of 0.80 (95% CI: 0.66-0.92). Across the full dataset of 300 sentences we use annotation from all four annotators. Out of 300, 6 sentences were discarded as unusable, 113 (38.4%) were identified as claims, predominantly concentrated in the results sections and beyond. For the claim structure extraction, annotators produced 199 records across 113 distinct gold claim sentences, heavily anchored on scientific entities for both subjects (90.5%) and objects (66.3%). Annotators agreed on the number of claims per sentence in 93% of shared cases. However, agreement was moderate for subjects (α=0.64\alpha= 0.64, 95% CI: 0.46–0.78) and objects (α=0.65\alpha= 0.65, 95% CI: 0.45–0.81), but low for predicates (α=0.46\alpha= 0.46, 95% CI: 0.23–0.66$).

Disagreement in subject and object annotations primarily arose from different interpretations of complex, multi-claim sentences. Additionally, subject mismatches frequently resulted from differing preferences in entity span granularity (e.g., selecting simply PBAN versus the more expansive PBAN + fine-grained + bidirectional attention). Conversely, the majority of predicate disagreements (56%) were driven by the OTHER class. For example, given the sentence “syntactic target information derived from parses complements well with the left-target-right context representation”, one annotator selected OTHER (specifying complements), while the remaining two attempted to utilize the predefined taxonomy, selecting is-suited-for and is-comparable-to, respectively. Overall, outperforms (24%) and achieves (22%) were the most frequent predicates, while negative or limiting predicates accounted for only 11%. Additionally, 95% of claim records were classified as own_claim.

Model Evaluation

We evaluated the three LLMs and their majority-vote ensemble, denoted as maj3, on both the 60 shared sentences (Figure 3) and the full 294-sentence gold-standard set (Table 1). On the core subset, maj3 achieved the highest F1 score (0.889) among the LLMs and showed strong agreement with the majority human annotation (κ=0.82\kappa= 0.82), comparable to that of the least-consistent annotator with the other two (κ=0.81\kappa= 0.81). On the full gold-standard set, maj3 achieved the highest agreement (κ=0.78\kappa= 0.78) and F1 score (0.864) among the LLMs, although its F1 score was statistically indistinguishable from Gemma’s and significantly higher only than Qwen’s. Overall, all models tended to identify more claims per sentence than human annotators, resulting in lower precision. Despite moderate human agreement on claim structure, we evaluated the generated structures against gold-standard annotations for all correctly identified claim-bearing sentences. Qwen achieved the highest F1 scores for all three structural components: subjects (0.79), predicates (0.60), and objects (0.55). Mistral followed with F1 scores of 0.74, 0.53, and 0.47, respectively, while Gemma performed worst (0.71, 0.51, and 0.43).

Figure 3. Performance on the 60 core sentences evaluated against the human majority label. Whiskers indicate 95% bootstrap CIs, and grey curves denote constant F1 contours. Open circles (A1–A3) show individual annotator agreement against the majority of the other two.

Given the strong performance of maj3, we applied it to all 9,073 sentences in our 50-paper corpus, yielding a silver-standard dataset comprising 1,487 claims (Table 2). These sentences include all annotated entities (generic or specific) and indicate whether a specific entity occurs in either of the two preceding sentences. We train BERT-based encoders on this silver-standard dataset and evaluate them against the smaller human-annotated gold-standard dataset (Table 1). Averaged over three random seeds, DeBERTa-v3-large achieves an F1 score of 0.877 (±0.008\pm0.008), slightly exceeding SciBERT (0.865 ±0.014\pm0.014), although the difference is not statistically significant. Its performance also does not differ significantly from that of the individual LLMs or the teacher ensemble, maj3. Despite the small gold-standard evaluation set, these results suggest that a 0.4B-parameter encoder can approximate the performance of an ensemble of three 24–31B-parameter LLMs on claim identification at substantially lower inference cost.

Error Analysis

All three LLMs jointly missed only 2 of the 113 gold-standard claims and jointly misclassified only 2 of the 181 non-claims, allowing majority voting to mitigate model-specific errors. Qwen, the most conservative model (Figure 3), showed substantially lower recall for claims in abstracts and

ModelPRF1κ\kappa [95% CI]
Qwen3.6-27B.962.673.792.70 [.61, .78]
Gemma-4-31B.771.956.854.75 [.67, .82]
Mistral-3.2-24B.832.832.832.73 [.64, .81]
maj3.888.841.864.78 [.71, .85]
SciBERT†.926.811.865.79 [.71, .85]
DeBERTa-v3-large†.939.823.877.81 [.75, .88]

caption: tbl:1: Claim identification on the 294 gold sentences. F1 score is binary F1 for the claim class, κ\kappa is Cohen’s κ\kappa against the gold label. maj3 denotes the majority label of the three LLMs. †Fine-tuned only on maj3 silver labels.

:::

SetPapersSentencesClaim Ratio
Core326038.3%
Gold4329438.4%
Silver509,07316.4%
ClaimFlow38814,41814.8%

caption: tbl:2: Dataset statistics. Claim ratio is the proportion of claim sentences. Gold includes human labels, with Core labeled by all annotators. Silver uses 3-LLM majority voting (excluding the 300 gold sentences). Claim-Flow preview dataset.

:::

conclusions (0.42) than for those in results and subsequent sections (0.76). Relying on Qwen alone could therefore lead to underrepresentation of authors’ main findings. In contrast, DeBERTa shows high agreement with its teacher ensemble, maj3, matching its predictions on 272 of the 294 gold-standard sentences. Of the 30 sentences misclassified by maj3, DeBERTa corrects 13, primarily procedural or factual statements incorrectly classified as claims by Gemma and Mistral, while introducing only 9 new errors among the 264 sentences correctly classified by maj3. On the 60 core sentences, DeBERTa achieves perfect agreement with the annotator majority (κ=1.00\kappa= 1.00), compared with κ=0.81\kappa= 0.81–1.001.00 (mean: 0.89) for individual annotators against the majority of their peers. Finally, the encoder appears to inherit its teacher’s section-specific biases, achieving higher recall in results and subsequent sections (0.89) than in abstracts and conclusions (0.62).

Out of Corpus Evaluation

To assess cross-corpus generalization, we evaluate our DeBERTa model, fine-tuned exclusively on silver-standard data, on a preview version of ClaimFlow [33] without additional training. The dataset contains 1,365 human-annotated central claims (1-5 per paper) and their corresponding evidence sentences from 388 ACL Anthology papers. While ClaimFlow does not require claims to be entity-anchored, we observe substantial overlap: 91.7% of its claims contain at least one task, dataset, or method entity in either their summary or evidence (rising to 98.6% when generic mentions are included). Furthermore, entities are highly indicative of evidence sentences; 77% of ClaimFlow’s evidence sentences contain an entity, compared to only 63% of unselected sentences in the same sections. When evaluated zero-shot, our model flags 53.6% of ClaimFlow’s evidence sentences as claims. Because a single claim can be supported by multiple evidence sentences, the model successfully identifies 62.3% of the central claims via at least one evidence sentence, a figure that increases to 70.4% when evaluated alongside the claim summary text.

Blind Validation Our model identifies claims in 24.1% of entity-bearing sentences and 14.0% of sentences without entity mentions that ClaimFlow did not label as claim-bearing. Because ClaimFlow is restricted to a few central claims per paper, measuring strict precision against its labels is not meaningful. To interpret the discrepancies, a human annotator performed a blind manual review of 100 sentences: 50 false positives, and 50 false negatives. Among the false positives, 78% (95% CI: 64%–87%) are genuine claims under our guidelines.1 Conversely, of the false negatives, only 6% (95% CI: 2%–17%) qualify as claims under our schema. The vast majority of these misses are contribution statements (e.g., “we propose . . .”, 20 of 47) or procedural descriptions (17 of 47), which our guidelines intentionally exclude. Therefore, the divergence between the datasets reflects a difference in definitions rather than model error. ClaimFlow identifies a paper’s central claims, whereas our model extracts an exhaustive set of entity-anchored, evaluative claims.

Discussion

Does entity anchoring make claim annotation focused and reliable?

In general, every sentence in a scholarly paper can be considered as an assertion. For example, in Sci-Arg, full-text annotation produced an average of 205 claim spans per paper [21]. However, entity anchoring reframes the annotation objective from a vague is this a claim? to a specific does this sentence make a contestable assertion about this entity? This constraint significantly reduces the search space, as only 59% of body sentences in our corpus contain scientific entities. It also maintains strictness, filtering out 62% of these candidate sentences as non-claims to yield a manageable average of 32 claims per paper. More importantly, this localized focus enables highly reproducible annotations (α=0.80\alpha= 0.80). Furthermore, the likelihood of a sentence constituting a claim increases with entity density (31%, 37%, and 51% for one, two, and ≥3\ge3 entities, respectively), indicating that entities act as strong signals of argumentative content rather than arbitrary filters. Although this approach inherently bounds the scope of extractable claims (see Limitations), it successfully transforms open-ended full-text annotation into a focused, reliable sentence-level decision.

Can LLMs Replace Costly Full-Text Annotation?

While entity anchoring reduces the annotation search space, manual full-text labeling remains impractical. Relying on a single LLM is also prone to errors due to varying model strictness and biases. However, as noted in the previous section, using an ensemble of models largely offsets these biases because their mistakes rarely coincide. Consequently, our majority-vote ensemble (maj3) achieved strong human agreement (κ=0.78\kappa= 0.78 on gold, κ=0.82\kappa= 0.82 on core). We further validated this by training encoders on individual LLM outputs versus maj3. Encoders trained on single-LLM outputs inherited their respective teacher’s bias (Gemma: P 0.78 / R 0.95; Qwen: P 0.95 / R 0.71) and scored lower (F1 0.853 and 0.812) relative to the maj3-trained DeBERTa (0.877). Furthermore, training encoders exclusively on our small human-annotated gold set via 5-fold cross-validation produced weaker and less stable models (F1: 0.812 ±\pm 0.019). Therefore, the LLM-generated silver data provides a highly scalable alternative that yields a stable encoder performance.

Conclusion

We introduced entity-anchored claim extraction, an approach for extracting structured claims from full-text scholarly articles. By utilizing scientific entity mentions as anchors, our approach replaces fuzzy, document-level discourse modeling with well-defined, sentence-local claim extraction. We validated our methodology through a case study of 50 sentiment analysis papers published between 2013 and 2023. Our rigorous annotation pipeline yielded a reliable human gold standard of 113 claim sentences, which we used to validate a large-scale silver dataset of 1,487 claims generated via a majority-vote LLM ensemble. We demonstrated that encoders fine-tuned exclusively on our silver-standard dataset achieve more consistent claim identification performance than those trained on the smaller human-annotated gold-standard datasets or single-LLM-generated data. These models also demonstrate strong zero-shot generalization to NLP papers outside the training corpus.

Limitations

We acknowledge several limitations in our current study. First, our extraction pipeline relies exclusively on the GSAP-NER model, incorporating alternative or more exhaustive scientific entity recognition frameworks could potentially alter recall profiles. Second, our primary empirical validation is constrained to a densely linked case study of 50 sentiment analysis papers. Although our zero-shot experiments on the ClaimFlow dataset demonstrate generalizability across the broader NLP domain, the framework remains to be evaluated across more diverse scientific disciplines. Fundamentally, we recognize an inherent limitation of the entity-anchored paradigm: not every scientific argumentative component explicitly contains a typed scientific entity. Consequently, our methodology does not guarantee an exhaustive extraction of all argumentative units within a publication. However, our primary objective is not exhaustive recall, but rather to drastically reduce the manual effort required to structure full-text arguments. Furthermore, our structured schema partially mitigates this restriction by capturing non-entity-bearing evidence within optional fields, such as the rationale. Finally, the restriction to a 50-paper corpus was a deliberate choice to facilitate the dense, high-quality manual annotation of interlinked papers. Building on this foundation, our ongoing future work aims to model the complete argumentative graph, encompassing complex intra-document and cross-document relations.

Ethical Considerations

Our annotations are released under a CC BY-NC-SA 4.0 license, whereas quoted text fragments retain the original open access or publisher licenses of the respective venues (ACL Anthology, AAAI, ICWSM, IJCAI) for research use. Since transformer based models or LLMs can produce errors, automated claims should serve as assistive tools rather than definitive replacements for reading the original papers. LLM-generated silver labels may also inherit latent model biases. Compute requirements were minimal, limited to inference with three open-weight LLMs on a single GPU node and fine tuning lightweight encoders.

Use of AI: We utilized AI assistants (Gemini and Claude) for code generation to develop our annotation tool, data analysis and to compute corpus statistics. All generated code and outputs were thoroughly reviewed, executed, and verified by the authors.

Data and Code Release

Our code, datasets, LLM prompts, and annotation guidelines are available on Github. A preview version of the ClaimFlow dataset was used with the authors’ permission.

Corpus Construction and Preprocessing Details

Retrieval and Filtering. The seed papers IMDb, SST, and SemEval-14 ABSA had 3,299, 6,781, and 1,886 forward citations in OpenAlex, respectively. We retrieved the most-cited citing papers for each (600, 400, and 600), yielding 1,508 unique papers after deduplication and the removal of review and survey articles. Keyword filtering (requiring sentiment, opinion, polarity, aspect, or emotion in the title or abstract) reduced this set to 876 articles. We downloaded the 397 available open-access PDFs, extracting up to the first 40 pages using pdfplumber, and retained only documents containing at least 1,500 words.

Graph-Based Sampling. In our graph construction, an entity edge was formed if two papers shared at least two keyword entities (from a curated list of 16 datasets and 21 methods shown in Table 3), with at least one being a dataset. Using greedy modularity maximization (implemented via NetworkX), we selected the largest community, comprising 209 papers. To select the final 50 papers, we ranked them first by within-community citation edges, then by entity edges. We stratified the selection across four publication year bands: 2014 and before, 2015-2017, 2018-2020, and 2021+, targeting 12-13 papers per band. As the 2014 and before band contained only 8 papers, we backfilled by rank, resulting in 17 papers for 2015-2017 and 13 for 2018-2020. More information about the select 50 papers are shown in Table 4.

Entity TypeKeywords
DatasetsIMDb, SST, SST-2, SST-5, Stanford Sentiment Treebank, SemEval-2014, SemEval-2015, SemEval-2016, Sentiment140, Yelp, Amazon Reviews, Twitter US Airline, MR dataset, Rotten Tomatoes, Laptop14, Restaurant14
MethodsSVM, Naïve Bayes, LSTM, BiLSTM, GRU, CNN, RNN, BERT, RoBERTa, XLNet, DistilBERT, ALBERT, Transformer, Word2Vec, GloVe, FastText, TF-IDF, Attention, Memory Network, Capsule Network

Table 3. Curated keyword entities used for establishing cross-document entity edges.

PropertyValue
Papers50
Publication years2013-2023
Full-text length (median words)4,533
Citations received (median)191
Papers citing each seed benchmark
SemEval-14 ABSA30
SST19
IMDb10
Paper pairs linked by citation176
Papers linked within the set48

Table 4. The 50 selected papers. All are open access with full text. A paper may cite more than one seed benchmark (nine cite two), so the seed counts sum to more than 50. Citation counts are from OpenAlex.

Text Cleaning. Following GRO bid (v0.8.2) extraction, we applied localized text normalization. Specifically, we corrected hyphenation introduced by line breaks (e.g., “inter- nal” → “internal”) only when bounded by lowercase letters, deliberately preserving acronym compounds (e.g., “RAM- training”). We also repaired artifact hyphens stemming from PDF extraction errors (e.g., “Contex- tAVG” → “ContextAVG”) and recurring tokeniza- tion splits (e.g., “sof tmax” → “softmax”). Space formatting prior to punctuation was also standard- ized.

Entity Extraction and Normalization Details

Model Training and Performance. We trained the SciDeBERTa-CS taggers using the original au- thors’ hyperparameters. As a baseline validation, on fold 0 of the GSAP-NER dataset, our specific- type tagger achieved a micro-F1 score of 64.6 under exact match and 73.4 under partial match criteria.

Post-Processing and Span Resolution. Applied to our 50-paper corpus, the taggers initially pro- duced 27,050 spans. We implemented a determin- istic post-processing step to remove sub-word frag- ments, merge adjacent spans of the same entity type, and resolve overlapping duplicates. This re- duction resulted in 23,832 valid mentions (detailed in Table 5).

LabelUseMentions
Methodentity5,035
ModelArchitectureentity2,849
Taskentity2,656
MLModelentity2,080
Datasetentity980
Datasourceentity120
ReferenceLinkcitation link3,454
MLModelGenericdiscarded3,313
DatasetGenericdiscarded3,313
URLdiscarded32
Specific / all mentions13,720 / 23,832

Table 5. GSAP-NER mentions in the 50 papers after post-processing.

Filtering and Normalization. From the resolved mentions, we retained 13,720 instances correspond- ing to the six specific entity types and 3,454 ReferenceLink markers. To address surface-level vari- ations of identical concepts (e.g., semeval 2014, semeval-2014, and semeval), we applied a normal- ization heuristic. Following this normalization, the corpus encompasses 3,627 distinct scientific enti- ties (see Table 6).

StatisticValue
Specific entity mentions13,720
Per paper (median; range)250; 94-605
Body sentences with ≥1 entity6,220
Distinct entities (normalised)3,627
Method / Task2,027 / 488
MLModel / ModelArchitecture330 / 587
Dataset / Datasource172 / 23
Entities in ≥2 / ≥5 / ≥25 papers674 / 161 / 11

Table 6. Entity-layer statistics. The most widespread entities are sentiment analysis (40 papers), word embeddings (38), and ABSA (33).

Annotation Interface and Schema Details

We developed a custom, lightweight web-based tool (Flask/SQLite) for blind annotation. To pro- vide necessary context without overwhelming the reader, candidate sentences are presented in read- ing order embedded within a local context win- dow (two preceding and two succeeding sentences), alongside the paper title and section heading. Dis- tribution of stratified sampling of the sections for final annotation is shown in Table 7. To assist with structuring, automatically extracted scientific enti- ties are highlighted in distinct colors by type, while generic entities (resolvable via the surrounding con- text) are underlined.

SectionsPoolSampledRateCore
Results onward21,29317513.5%35
Abs. + Conc.3817519.7%15
Intro. + Rel. Work1,101353.2%7
Method + Exp.1,396151.1%3
Total4,1713007.2%60

Table 7. Stratified sentence sampling for claim annotation across document sections. Core denotes the overlapping subset used for inter-annotator agreement. Abbreviations: Abs. (Abstract), Conc. (Conclusion), Intro. (Introduction), Rel. Work (Related Work), Exp. (Experiment). 2Includes Results and all subsequent pre-conclusion sections (e.g., discussion, error analysis).

The annotation process proceeds in two steps:

  • Identification: Annotators classify the tar- get sentence as argumentative (claim) or non- argumentative (non-claim). A Cannot Judge option is provided to gracefully handle and exclude garbled text resulting from PDF ex- traction errors or instances lacking sufficient context.

  • Structuring: For identified claims, annota-

tors specify the claim type (Own or Reported) and assign Subject and Object arguments by clicking the highlighted entities. If an appropriate entity was missed by the automated extraction, annotators are permitted to input free text. Finally, they select from a predefined inventory of 13 predicates (12 specific predicates plus an OTHER category), which are fully detailed in Table 8.

To accommodate complex sentences, the interface allows annotators to capture multiple independent claim records per sentence. Furthermore, optional fields including the claim’s scope, condition, rationale, attribution, and specific evaluation metrics alongside their numeric results, can be explicitly extracted when stated in the text.

Implementation Details

For experimental reproducibility, we detail our computational setup and hyperparameters. Entity extraction via the GSAP-NER tagger was executed on a single NVIDIA RTX 4060 Ti (16GB VRAM) utilizing CUDA 13, Python 3.13, PyTorch 2.13, and Transformers 5.14. For silver data generation via LLMs, we deployed Qwen3.6-27B utilizing vLLM 0.17.1, alongside Mistral-Small-3.2-24B and Gemma-4-31B-it utilizing vLLM 0.29.0, hosted on an NVIDIA H200 NVL (141GB) node. Inference utilized OpenAI-compatible chat completions with prompts formulated as a single user message, employing greedy decoding (temperature 0), a 2,048 maximum token limit, a 16,000 maximum context length (max_model_len), and with internal thinking modes strictly disabled. Finally, the encoder fine-tuning was performed on an NVIDIA L40S GPU. We run 5-fold cross-validation split by paper. The 43 papers that contain human-annotated sentences are divided into five folds balanced by annotated-sentence count (8-9 papers, nearly 60 sentences per fold). The remaining 7 papers are always used for training. In each fold, the model is trained on silver labels (majority vote of three LLMs) from the other papers, with 10% of them (4 papers) held out as a development set for early stopping. It is evaluated on the human labels of the held-out fold. No paper appears in more than one of train, dev and test, and no human-annotated sentence is used in training, even with a silver label. We optimized the models using AdamW for a maximum of 5 epochs with a batch size of 32, a maximum sequence length of 128, a learning rate of 1×10−51 \times10^{-5}, a weight decay of 0.01, a 10% linear warmup schedule, and a gradient clipping norm of 1.0. Training was regulated via early stopping based on the development set positive-class F1 score, utilizing a patience of 2 epochs and restoring the best-performing weights.

References

  1. [1]Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3615–3620.arxiv.org/abs/1903.10676
  2. [2]Pablo Accuosto, Mariana Neves, and Horacio Saggion. 2021. Argumentation mining in scientific literature: From computational linguistics to biomedicine. In Frommholz I, Mayr P, Cabanac G, Verberne S, editors. BIR 2021: 11th International Workshop on Bibliometric-enhanced Information Retrieval; 2021 Apr 1; Lucca, Italy. Aachen: CEUR; 2021. p. 20-36. CEUR Workshop Proceedings.
  3. [3]Pablo Accuosto and Horacio Saggion. 2019. Transferring knowledge from discourse to arguments: A case study with scientific abstracts. In Proceedings of the 6th Workshop on Argument Mining, pages 41–51.
  4. [4]Khalid Al Khatib, Tirthankar , Ghosal, Yufang , Hou, Anita , de Waard, and Dayne , Freitag. 2021. Argument mining for scholarly document processing: Taking stock and looking ahead. In Proceedings of the Second Workshop on Scholarly Document Processing, pages 56–65, Online. Association for Computational Linguistics.DOI
  5. [5]Arne Binder, Leonhard Hennig, and Bhuvanesh Verma. 2022. Full-text argumentation mining on scientific publications. In Proceedings of the first Workshop on Information Extraction from Scientific Publications, pages 54–66.arxiv.org/abs/2210.13084
  6. [6]Catherine Blake. 2010. Beyond genes, proteins, and abstracts: Identifying scientific claims from full-text biomedical articles. Journal of biomedical informatics, 43(2):173–189.DOI
  7. [7]Aaron Clauset, Mark EJ Newman, and Christopher Moore. 2004. Finding community structure in very large networks. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 70(6):066111.DOI
  8. [8]Mike D’Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope, and Doug Downey. 2024. Aries: A corpus of scientific paper edits made in response to peer reviews. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6985–7001.
  9. [9]Liat Ein-Dor, Eyal Shnarch, Lena Dankin, Alon Halfon, Benjamin Sznajder, Ariel Gera, Carlos Alzate, Martin Gleize, Leshem Choshen, Yufang Hou, and 1 others. 2020. Corpus wide argument mining—a working solution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7683–7691.
  10. [10]Aris Fergadis, Dimitris Pappas, Antonia Karamolegkou, and Harris Papageorgiou. 2021. Argumentation mining in scientific literature for sustainable development. In Proceedings of the 8th Workshop on Argument Mining, pages 100–111.DOI
  11. [11]Beatriz Fisas, Francesco Ronzano, and Horacio Saggion. 2016. A multi-layered annotated corpus of scientific papers. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3081–3088.
  12. [12]GROBID contributors. 2008-2026. GROBID: GeneRation Of Bibliographic Data. https://github.com/grobidOrg/grobid. GitHub repository.
  13. [13]Yanzhu Guo, Guokan Shang, Virgile Rennard, Michalis Vazirgiannis, and Chloé Clavel. 2023. Automatic analysis of substantiation in scientific peer reviews. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10198–10216.arxiv.org/abs/2311.11967
  14. [14]Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021a. DebertaV3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543.arxiv.org/abs/2111.09543
  15. [16]Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2021. TDMSci: A specialized corpus for scientific literature entity tagging of tasks datasets and metrics. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 707–714.arxiv.org/abs/2101.10273
  16. [17]Arman Irani, Ju Yeon Park, Kevin Esterling, and Michalis Faloutsos. 2024. WIBA: What is being argued? a comprehensive approach to argument mining. arXiv preprint arXiv:2405.00828.
  17. [18]Sarthak Jain, Madeleine Van Zuylen, Hannahah Hajishirzi, and Iz Beltagy. 2020. Scirex: A challenge dataset for document-level information extraction. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 7506–7516.arxiv.org/abs/2005.00512
  18. [19]Shashidhar Reddy Javaji, Yupeng Cao, Haoანგ Li, Yangyang Yu, Nikhil Muralidhar, and Zining Zhu. 2025. Can ai validate science? benchmarking llms on claim→ evidence reasoning in ai papers. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 2355–2379.
  19. [20]Yuna Jeong and Eunhui Kim. 2022. Scideberta: Learning deberta for science technology documents and fine-tuning information extraction tasks. IEEE Access, 10:60805–60813.
  20. [21]Anne Lauscher, Goran Glavaš, and Simone Paolo Ponzetto. 2018. An argument-annotated corpus of scientific publications. In Proceedings of the 5th Workshop on Argument Mining, pages 40–46.
  21. [22]John Lawrence and Chris Reed. 2019. Argument mining: A survey. Computational Linguistics, 45(4):765–818.DOI
  22. [23]Ran Levy, Ben Bogin, Shai Gretz, Ranit Aharonov, and Noam Slonim. 2018. Towards an argumentative content search engine using weak supervision. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2066–2081.
  23. [24]Maria Liakata, Shyamasree Saha, Simon Dobnik, Colin Batchelor, and Dietrich Rebholz-Schuhmann. 2012. Automatic recognition of conceptualization zones in scientific articles and two life science applications. Bioinformatics, 28(7):991–1000.
  24. [25]Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min-Yen Kan. 2023. Scitab: A challenging benchmark for compositional reasoning and claim verification on scientific tables. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7787–7813.
  25. [26]Yi Luan. 2019. Information Extraction from Scientific Literature for Method Recommendation. Ph.D. thesis, University of Washington. ArXiv:1901.00401.arxiv.org/abs/1901.00401
  26. [27]Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150.
  27. [28]Tobias Mayer, Elena Cabrio, and Serena Villata. 2020. Transformer-based argument mining for healthcare applications. In ECAI 2020: 24th European Conference on Artificial Intelligence, 29 August–8 September 2020, Santiago de Compostela, Spain–Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), pages 2108–2115. SAGE Publications 1 Oliver’s Yard, 55 City Road, London, EC1Y 1SP.
  28. [29]Yoko Mizuta, Anna Korhonen, Tony Mullen, and Nigel Collier. 2006. Zone analysis in biology articles as a basis for information extraction. International journal of medical informatics, 75(6):468–487.
  29. [30]Wolfgang Otto, Matthäus Zloch, Lu Gan, Saurav Karmakar, and Stefan Dietze. 2023. GSAP-NER: A novel task, corpus, and baseline for scholarly entity extraction focused on machine learning models and datasets. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8166–8176.
  30. [31]Jiefu Ou, William Gantt Walden, Kate Sanders, Zhengping Jiang, Kaiser Sun, Jeffrey Cheng, William Jurayj, Miriam Wanner, Shaobo Liang, Candice Morgan, and 1 others. 2025. Claimcheck: How grounded are llm critiques of scientific papers? In EMNLP (Findings), pages 21712–21735.
  31. [32]Maria Pontiki, Dimitris Galanis, John Papalouopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35, Dublin, Ireland. Association for Computational Linguistics.DOI
  32. [33]Aniket Pramanick, Yufang Hou, Saif M Mohammad, and Iryna Gurevych. 2026. Claimflow: Tracing the evolution of scientific claims in nlp. arXiv preprint arXiv:2603.16073.
  33. [34]Jason Priem, Heather Piwowar, and Richard Orr. 2022. Openalex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv preprint arXiv:2205.01833.
  34. [35]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.DOI
  35. [36]Simone Teufel. 1999. Argumentative zoning: Information extraction from scientific text. Ph.D. thesis, University of Edinburgh Edinburgh.

Paper details

vlnc:W_2thb7s72
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents