betaFeedback

WorksComputer Science

Fact or Fiction: Verifying Scientific Claims

Abstract

We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that SUPPORTS or REFUTES a given scientific claim, and to identify rationales justifying each decision. To study this task, we construct SciFact, a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales. We develop baseline models for SciFact, and demonstrate that simple domain adaptation techniques substantially improve performance compared to models trained on Wikipedia or political news. We show that our system is able to verify claims related to COVID-19 by identifying evidence from the CORD-19 corpus. Our experiments indicate that SciFact will provide a challenging testbed for the development of new systems designed to retrieve and reason over corpora containing specialized domain knowledge. Data and code for this new task are publicly available at https://github.com/allenai/scifact. A leaderboard and COVID-19 fact-checking demo are available at https://scifact.apps.allenai.org.

Introduction

Due to rapid growth in the scientific literature, it is difficult for researchers – and the general public even more so – to stay up to date on the latest findings. This challenge is especially acute during public health crises like the current COVID-19 pandemic, due to the extremely fast rate at which new findings are reported and the risks associated with making decisions based on outdated or incomplete information. As a result, there is a need for automated tools to assist researchers and the public in evaluating the veracity of scientific claims.1

Fact-checking – a task in which the veracity of an input claim is verified against a corpus of documents that support or refute the claim – has been studied to combat the proliferation of misinformation in political news, social media, and on the web (Thorne et al., 2018; Hanselowski et al., [20]). However, verifying scientific claims poses new challenges to both dataset construction and effective modeling. While political claims are readily available on fact-checking websites and can be verified by crowd workers, annotators with extensive domain knowledge are required to generate and verify scientific claims.

In addition, NLP systems for scientific claim verification must possess additional capabilities beyond those required to verify factoid claims. For instance, to verify the claim shown in Figure 1, a system must have the ability to access scientific background knowledge, reason over increases and decreases in quantities or measurements, and make sense of specialized statistical language.

A scientific claim, supported by evidence identified by our system

Figure 1. A scientific claim, supported by evidence identified by our system. To correctly verify this claim, the system must possess background knowledge that troponin is a protein found in cardiac muscle and that elevated levels of troponin are a marker of cardiac injury. In addition, it must be able to reason about directional relationships between scientific processes: replacing higher with lower would cause the rationale to REFUTE the claim rather than SUPPORT it. Finally, the system should interpret p<0.001p < 0.001 as an indication that the reported finding is statistically significant.

In this paper, we introduce the task of scientific claim verification to evaluate the veracity of scientific claims against a scientific corpus. Table 1 presents some examples. To facilitate research on this task, we construct SCIFACT, an expert-annotated dataset of 1,409 scientific claims accompanied by abstracts that support or refute each claim, and annotated with rationales [24] justifying each SUPPORTS / REFUTES decision. To create the dataset, we develop a novel annotation protocol in which annotators re-formulate naturally occurring claims in the scientific literature – citation sentences – into atomic scientific claims. Using citation sentences as a source of claims both speeds the claim generation process and guarantees that the topics discussed in SCIFACT are representative of the research literature. In addition, citation links indicate the exact documents likely to contain evidence necessary to verify a given claim.

Claim 1: Lopinavir / ritonavir have exhibited favorable clinical responses when used as a treatment for coronavirus.
Supports: … Interestingly, after lopinavir/ritonavir (Kaletra, AbbVie) was administered, β-coronavirus viral loads significantly decreased and no or little coronavirus titers were observed.
Refutes: The focused drug repurposing of known approved drugs (such as lopinavir/ritonavir) has been reported failed for curing SARS-CoV-2 infected patients. It is urgent to generate new chemical entities against this virus . . .
Claim 2: The coronavirus cannot thrive in warmer climates.
Supports: …most outbreaks display a pattern of clustering in relatively cool and dry areas…This is because the environment can mediate human-to-human transmission of SARS-CoV-2, and unsuitable climates can cause the virus to destabilize quickly…
Refutes: …significant cases in the coming months are likely to occur in more humid (warmer) climates, irrespective of the climate-dependence of transmission and that summer temperatures will not substantially limit pandemic growth.

Table 1. Evidence identified by our system as supporting and refuting two claims concerning COVID-19.

We establish performance baselines on SCIFACT with an approach similar to DeYoung et al. [12], which achieves strong performance on the FEVER claim verification dataset Thorne et al. (2018). Our baseline is a pipeline system which retrieves abstracts related to an input claim, uses a BERT-based Devlin et al. [11] sentence selector to identify rationale sentences, and labels each abstract as SUPPORTS, REFUTES, or NOINFO with respect to the claim. We demonstrate that our baseline can benefit from training on claims from domains including Wikipedia articles and politics.

We showcase the ability of our model to verify expert-written claims concerning the novel coronavirus COVID-19 against the newly-released CORD-19 corpus Wang et al. (2020). Expert annotators judge retrieved evidence to be plausible for 23 of 36 claims.2 Our results and analyses demonstrate the importance of the new task and dataset to support significant future research in this domain.

In summary, our contributions include: (1) We introduce and formalize the scientific claim verification task. (2) We develop a novel annotation protocol to generate and verify 1.4K naturally-occurring claims about scientific findings. (3) We establish strong baselines on this task, and identify substantial opportunities for improvement at all stages of the modeling pipeline. (4) We demonstrate the efficacy of our system in a real-world case study verifying claims about COVID-19 against the research literature.

Background and task definition

As illustrated in Figure 1, scientific claim verification is the task of identifying evidence from the research literature that SUPPORTS or REFUTES a given scientific claim. Table 1 shows the results of our system applied to claims about the novel coronavirus COVID-19. For each claim, the system identifies relevant scientific abstracts, and labels the relation of each abstract to the claim as either SUPPORTS or REFUTES. Verifying scientific claims is challenging and requires domain-specific background knowledge – for instance, in order to identify the evidence supporting Claim 1 in Table 1, the system must determine that a reduction in coronavirus viral load indicates a favorable clinical response, even though this fact is never mentioned.

Scientific claims In SCIFACT, a scientific claim is an atomic verifiable statement expressing a finding about one aspect of a scientific entity or process, which can be verified from a single source.3 For instance, “The R0R_0 of the novel coronavirus is 2.5” is valid, but opinion-based statements like “The government should require people to stand six feet apart to stop coronavirus” are not. Compound claims like “Aerosolized coronavirus droplets can travel at least 6 feet and can remain in the air for 3 hours” should be split into two atomic claims.

Claims in SCIFACT are natural – they are derived from citation sentences, or citances (Nakov et al., 2004), that occur naturally in scientific articles. This is similar to political fact-checking datasets such as UKP Snopes (Hanselowski et al., [20]), which use political fact-checking websites as a source of natural claims. On the other hand, claims in the popular FEVER dataset (Thorne et al., 2018) are synthetic, since they are created by annotators by mutating sentences from the Wikipedia articles that will serve as evidence.

Supporting and refuting evidence In most fact-checking work, claims are assigned a global truth label based on the entirety of the available evidence. For example in FEVER, the claim “Barack Obama was the 44{sup}th` President of the United States” can be verified using Wikipedia as an evidence source.

While SCIFACT claims are indeed verifiable assertions about scientific findings, accurately assigning a global truth label to a scientific claim (given a fixed scientific corpus) requires a systematic review by a team of experts. In this work we focus on the simpler task of assigning SUPPORTS or REFUTES relations to individual claim-abstract pairs.

Each SUPPORTS or REFUTES relation between claim and abstract must be justified by at least one rationale. A rationale is a minimal collection of sentences which, taken together as premises in the context of the abstract, can reasonably be judged by a domain expert as implying the claim. Rationales facilitate the development of interpretable models which not only have the ability to make label predictions, but can also identify the exact sentences that are necessary for their decisions.

The SCIFACT dataset

The SCIFACT dataset consists of 1,409 scientific claims4 verified against a corpus of 5,183 abstracts.

Abstracts that support or refute each claim are annotated with rationales. We describe our corpus creation and annotation process.

Data source and corpus construction

To construct SCIFACT, we use S2ORC (Lo et al., [26]), a publicly-available corpus of millions of scientific articles. To ensure that documents in our dataset are of high quality, we randomly sample articles from a manually curated collection of well-regarded journals spanning domains from basic science (e.g., Cell, Nature) to clinical medicine (e.g., JAMA, BMJ). The full list of journals is included in Appendix § 9.§ 1. We restrict to articles with at least 10 citations. The resulting collection is referred to as our seed set. We use the S2ORC citation graph to sample source citances from citing articles which cite these seed articles. If a citance cites other articles not in the seed set, we refer to these as co-cited articles and add them to the corpus, as depicted in Figure 2. The content of the cited abstracts encompasses a diverse array of topics within biomedicine, as shown in Figure 3. The majority of citances used for SCIFACT cite only the seed article (no co-cited articles), as we found in initial annotation experiments that these citances tended to yield specific, easy-to-verify claims.

Figure 2. Corpus construction. Citing abstracts are identified for each seed document. A claim is written based on the source citance in the citing abstract.

Figure 3. Most frequently occurring Medical Subject Headings (MeSH) terms (y-axis) among cited abstracts. MeSH is a controlled vocabulary used for indexing articles in PubMed. Topics range from clinical trial reports (“Humans”, “Risk Factors”) to molecular biology (“Cell Line”, “RNA”).

To expand the corpus, we identify five papers cited in the same paper as each source citance but in a different paragraph, and add these to the corpus as distractor abstracts. These abstracts often have 1,000 questions, and information extraction (e.g. SciERC (Luan et al., 2018) has 500 annotated abstracts). discuss similar topics to the evidence documents, increasing the difficulty of abstract retrieval and making our metrics more accurately reflect the system’s performance on a large research corpus.

Claim writing

Annotation Annotators are shown a source citance in the context of an article, and are asked to write up to three claims based on the content of the citance; see Appendix § 9.§ 1.1 for an example. This results in natural claims because the annotator does not see the cited article’s abstract – the cited abstract – at the time of claim writing. Annotators are asked to skip citances that do not make statements about specific scientific findings.

The claim writers included four experts with background in scientific NLP, fifteen undergraduates studying the life sciences, and four graduate students (doctoral or medical) in the life sciences. Detailed information on the annotator training process can be found in Appendix § 9.§ 2. The claim-writing interface is shown in Appendix § 10.

Claim negation Unless the authors of the source citance were mistaken, cited articles should provide supporting evidence for the claims made in a citance. To obtain examples where an abstract REFUTES a claim, an NLP expert wrote negations of existing claims, taking precautions not to bias the negations by using obvious keywords like “not” [19] (Schuster et al. 2019). In § 5.1, we demonstrate that a “claim-only” verification model performs poorly, suggesting that the negation process did not introduce severe artifacts.

Claim verification

Annotation For each claim, all of the claim’s cited abstracts are annotated for evidence. Annotators are shown a single claim - cited abstract pair, and asked to label the pair as SUPPORTS, REFUTES, or NOINFO. Although our task definition allows for a single claim to be both supported and refuted (by different abstracts) – an occurrence we observe on real-world COVID-19 claims (§ 5.3) – this never occurs in our dataset. Each claim has a single label. Counts for each label are shown in Table 2a. Overall, the annotators found evidence in 63% of cited abstracts. If the annotator assigns a SUPPORTS or REFUTES label, they must also identify all rationales as defined in § 1.1. Table 2b provides statistics on the number of sentences per rationale, the number of rationales per claim / abstract pair, and the number of evidence abstracts per claim. No abstract has more than 3 rationales for a given claim, and all rationales consist of at most three sentences. Rationales in SciFACT are mutually exclusive. 28 rationales contain non-contiguous sentences.

FoldSUPPORTSNOINFOREFUTESAll
Train332304173809
Dev12411264300
Test100100100300
All5565163371409
0123+
Cited abstracts per claim-12788645
Evidence abstracts per claim5168303726
Rationales per abstract-552290153
Sentences per rationale-15429211

Table 2. (a) Distribution of claim labels in SCIFACT. (b) Evidence counts at various levels of granularity. For example, Column 2 of the row “Rationales / abstract” indicates that 290 claim / abstract pairs are supported by 2 distinct rationales. Table 2: Statistics on claim labels, and the number of evidence abstracts and rationales per claim.

The verifiers included three NLP experts, five life science undergraduates, and five graduate students studying life sciences. Annotators verified claims that they did not write themselves. Annotation guidelines are provided in Appendix § 10.

SciFACT claims are verified against abstracts rather than full articles since (1) abstracts can be annotated more scalably, (2) evidence is found in the abstract in more than 60% of cases, and (3) previous attempts at full-document annotation suffered from low annotator agreement (§ 6).

Quality We assign 232 claim-abstract pairs for independent re-annotation. The label agreement is 0.75 Cohen’s κ\kappa, comparable with the 0.68 Fleiss’ κ\kappa reported in Thorne et al. (2018), and 0.70 Cohen’s κ\kappa reported in Hanselowski et al. [20]. To measure rationale agreement, we treat each sentence as either classified as “part of a rationale” or “not part of a rationale” and compute sentence-level agreement. The resulting Cohen’s κ\kappa is 0.71.

The SciFACT task

Task Formulation The inputs to our task are a scientific claim cc and a corpus of abstracts A\mathcal{A}. All abstracts a∈Aa \in\mathcal{A} are labeled as y(c,a)∈{SUPPORTS,REFUTES,NOINFO}y(c,a) \in\{\text{SUPPORTS}, \text{REFUTES}, \text{NOINFO}\} with respect to a claim cc.

The abstracts that either SUPPORT or REFUTE cc are referred to as evidence abstracts for cc, denoted as E(c)\mathcal{E}(c). Each evidence abstract a∈E(c)a \in\mathcal{E}(c) is annotated with rationales. A single rationale RiR_i is a collection of sentences {r1(c,a),…,rm(c,a)}\{r_1(c, a), \ldots, r_m(c, a)\}, where mm is the number of sentences in rationale RiR_i. We denote the set of all rationales as R(c,a)={R1(c,a),…,Rn(c,a)}\mathcal{R}(c, a) = \{R_1(c, a), \ldots, R_n(c, a)\}.

Given a claim cc and a corpus A\mathcal{A}, the system must predict a set of evidence abstracts E^(c)\widehat{\mathcal{E}}(c). For each abstract a∈E^(c)a \in\widehat{\mathcal{E}}(c), it must predict a label y^(c,a)\widehat{y}(c, a), and a collection of rationale sentences S^(c,a)={s^1(c,a),…,s^ℓ(c,a)}\widehat{\mathcal{S}}(c, a) = \{\widehat{s}_1(c, a), \ldots, \widehat{s}_{\ell}(c, a)\}. Note that although the gold annotations may contain multiple separate rationales, to simplify the prediction task we only require the model to predict a single collection of rationale sentences; these sentences may encompass multiple gold rationales.

Task Evaluation We evaluate the task at two levels of granularity. For abstract-level evaluation, we assess the model’s ability to identify the abstracts that support or refute the claim. For sentence-level evaluation, we evaluate the model’s performance at identifying the sentences sufficient to justify the abstract-level predictions. We conduct evaluations in both the “Open” FEVER-style (Thorne et al., 2018) setting where the evidence abstracts must be retrieved, and the “Oracle abstract” ERASER-style (DeYoung et al., [12]) setting where the gold evidence abstracts E(c)\mathcal{E}(c) are provided.

Abstract-level evaluation is inspired by the FEVER score. Given a claim cc, a predicted evidence abstract a∈E^(c)a \in\widehat{\mathcal{E}}(c) is correctly labeled if (1) aa is a gold evidence abstract for cc, and (2) The predicted label is correct: y^(c,a)=y(c,a)\widehat{y}(c, a) = y(c, a). It is correctly rationalized if, in addition, the predicted rationale sentences contain a gold rationale, i.e., there exists some gold rationale Ri(c,a)⊆S^(c,a)R_i(c, a) \subseteq\widehat{\mathcal{S}}(c, a).

Like FEVER, which limits the maximum number of predicted rationale sentences to five, SCIFACT limits to three predicted rationale sentences. Overall performance is measured by the micro-F1 of the precision and recall over the correctly-labeled and correctly-rationalized evidence abstracts. We refer to these evaluations as AbstractLabel-Only_{\text{Label-Only}} and AbstractLabel+Rationale_{\text{Label+Rationale}}, respectively.

Sentence-level evaluation measures performance in identifying individual rationale sentences. Unlike the abstract-level metrics, this evaluation penalizes the prediction of extra rationale sentences.

A predicted rationale sentence s^(c,a)\widehat{s}(c, a) is correctly selected if (1) It is a member of some gold rationale Ri(c,a)R_i(c, a), (2) all other sentences from the same gold rationale Ri(c,a)R_i(c, a) are among the predicted S^(c,a)\widehat{\mathcal{S}}(c, a), and (3) y^(c,a)≠NoINFO\widehat{y}(c, a) \ne\text{NoINFO}.5 It is correctly labeled if, in addition, the abstract aa is correctly labeled: y^(c,a)=y(c,a)\widehat{y}(c, a) = y(c, a).

Overall performance is measured by the micro-F1 of the precision and recall of correctly-selected and correctly-labeled rationale sentences, denoted SentenceSelection-Only_{\text{Selection-Only}} and SentenceSelection+Label_{\text{Selection+Label}}. For sentence-level evaluation, we do not limit the number of predicted rationale sentences, since the evaluation penalizes models that over-predict.

Verisci: Baseline model

We develop a baseline (referred to as VERISCI) that takes a claim cc and corpus A\mathcal{A} as input, identifies evidence abstracts E^(c)\widehat{\mathcal{E}}(c), and predicts a label y^(c,a)\widehat{y}(c, a) and rationale sentences S^(c,a)\widehat{\mathcal{S}}(c, a) for each a∈E^(c)a \in\widehat{\mathcal{E}}(c). Following the “BERT-to-BERT” model presented in DeYoung et al. [12]; Soleimani et al. (2019), VERISCI is a pipeline of three components:

  1. ABSTRACTRETRIEVAL retrieves kk abstracts with highest TF-IDF similarity to the claim.

  2. RATIONALESELECTION identifies rationale sentences S^(c,a)\widehat{\mathcal{S}}(c, a) for each abstract.

  3. LABELPREDICTION makes the final label prediction y^(c,a)\widehat{y}(c, a).

Rationale selection Given a claim cc and abstract aa, we train a model to predict z^i≜\widehat{z}_i \triangleq 1[ai is a rationale sentence]\mathbb{1}[a_i \text{ is a rationale sentence}] for each sentence aia_i in aa. For each sentence, we encode the concatenated sequence wi=[ai,SEP,c]w_i = [a_i, \mathrm{SEP}, c] using a BERT-style language model and predict a score z~i=σ[f(CLS(wi))]\tilde{z}_i = \sigma[f(\mathrm{CLS}(w_i))], where σ\sigma is the sigmoid function, ff is a linear layer and CLS(wi)\mathrm{CLS}(w_i) is the CLS token from the encoding of wiw_i. We train the model on pairs of claims and their cited abstracts and minimize cross-entropy loss between ziz_i and z~i\tilde{z}_i. For each claim, we use cited abstracts labeled NOINFO, as well as non-rationale sentences from abstracts labeled SUPPORTS and REFUTES as negative examples. To make predictions, we select all sentences aia_i with z~i>t\tilde{z}_i > t as rationale sentences, where t∈[0,1]t \in[0,1] is tuned on the dev set (Appendix § 7.§ 1).

Label prediction Sentences identified by the rationale selector are passed to a separate BERT-based model to make the final labeling decision. Given a claim cc and abstract aa, we concatenate the claim and the predicted rationale sentences u=[s^1(c,a),…s^ℓ(c,a),SEP,c]u = [\hat{s}_1(c,a), \ldots\hat{s}_\ell(c,a), \mathrm{SEP}, c],6 and predict y~(c,a)=ϕ[f(CLS(u))]\tilde{y}(c,a) = \phi[f(\mathrm{CLS}(u))], where ϕ\phi is the softmax function, and ff is a linear layer with three outputs representing the {SUPPORTS, REFUTES, NOINFO} labels. We minimize the cross-entropy loss between y~(c,a)\tilde{y}(c,a) and the true label y(c,a)y(c,a).

We train the model on pairs of claims and their cited abstracts using gold rationales as input. For cited abstracts labeled NOINFO, we choose the kk sentences from the cited abstract with highest TF-IDF similarity to the claim as input rationales. For prediction, we use the predicted rationale sentences S^(c,a)\hat{S}(c,a) as input and predict y^(c,a)=argmax⁡y~(c,a)\hat{y}(c,a) = \operatorname{argmax} \tilde{y}(c,a). NOINFO is predicted for abstracts with no rationale sentences.

We experimented with a label prediction model which encodes entire abstracts via the Longformer ([5]), and makes predictions using the document-level CLS token. Performance was not competitive with our pipeline setup, likely because the label predictor struggles to identify relevant information when given full abstracts.

Experiments

In our experiments, we (1) analyze the performance of each individual component of VERISCI, (2) evaluate full task performance in both the “Oracle abstract” and “Open” settings, (3) present promising results verifying claims about COVID-19 using

VERISCI, and (4) discuss some modeling challenges presented by the dataset.

Pipeline components

We examine the effects of different training datasets, sentence encoders, and model inputs on the performance of the RATIONALESELECTION and LABELPREDICTION modules. The RATIONALESELECTION module is evaluated on its ability to select rationale sentences given gold abstracts.7 The LABELPREDICTION module is evaluated on its 3-way label classification accuracy given gold rationales from cited abstracts. Cited abstracts labeled NOINFO are included in the evaluation. These abstracts have no gold rationale sentences; as in §5, we provide the kk most similar sentences from the abstract as input (more details in Appendix § 7).

Training Data We train on (1) FEVER, (2) UKP Snopes, (3) SCIFACT, and (4) FEVER pretraining followed by SCIFACT fine-tuning. RoBERTa-large ([25]) is used as the sentence encoder.

Sentence encoder We fine-tune SciBERT ([4]), BioMedRoBERTa ([18]), RoBERTa-base, and RoBERTa-large. SCIFACT is used as training data.

Model Inputs We examine the performance of “claim-only” and “abstract-only” models trained on SCIFACT, using RoBERTa-large as the sentence encoder. The claim-only model makes label predictions based on the claim text alone, without access to evidence abstracts. The abstract-only model selects rationale sentences and makes label predictions without access to the claim.

Results The results are shown in Table 3. For LABELPREDICTION, the best performance is achieved by training first on the large FEVER dataset and then fine-tuning on the smaller in-domain SCIFACT training set. To understand the benefits of FEVER pretraining, we examined the claim / evidence pairs where the FEVER + SCIFACT-trained model made correct predictions but the SCIFACT-trained model did not. In 36 / 44 of these cases, the SCIFACT-trained model predicts NOINFO. Thus pretraining on FEVER appears to improve the model’s ability to recognize textual entailment relationships between evidence and claim – particularly relationships indicated by non-domain-specific cues like “is associated with” or “has an important role in”.

Training dataPRF1ACC.
FEVER41.557.948.467.6
UKP Snopes42.562.350.571.3
SCIFACT73.770.572.175.7
FEVER + SCIFACT72.467.269.781.9
Sentence encoderPRF1ACC.
SciBERT74.574.374.469.2
BioMedRoBERTa75.369.972.571.7
RoBERTa-base76.166.170.862.9
RoBERTa-large73.770.572.175.7
Model inputsPRF1ACC.
Claim-only---44.5
Abstract-only60.160.960.553.3

Table 3. Comparison of different training datasets, encoders, and model inputs for RATIONALESELECTION and LABELPREDICTION, evaluated on the SCIFACT dev set. The claim-only model cannot select rationales.

For RATIONALESELECTION, training on SCIFACT alone produces the best results. We examined the rationales that the SCIFACT-trained model identified but the FEVER-trained model missed, and found that they generally contain science-specific vocabulary. Thus, training on additional out-of-domain data provides little benefit.

RoBERTa-large exhibits the strongest performance on label prediction, while SCIBERT has a slight edge on rationale selection. The “claim-only” model exhibits very poor performance, which provides some reassurance that the claim negation procedure described in § 2.2 does not introduce obvious statistical artifacts. Similarly, the poor performance of the “abstract-only” model indicates that the model needs access to the claim being verified in order to identify relevant evidence.

Full task

Experimental setup Based on the results from § 5.1, we use the RATIONALESELECTION module trained on SCIFACT only, and the LABELPREDICTION module trained on FEVER + SCIFACT for our final end-to-end system VERISCI. Although SCIBERT performs slightly better on rationale selection, using RoBERTa-large for both RATIONALESELECTION and LABELPREDICTION gave the best full-pipeline performance on the dev set, so we use RoBERTa-large for both components. For the ABSTRACTRETRIEVAL module, the best dev set full-pipeline performance was achieved by retrieving the top k=3k = 3 documents.

Model comparisons We report performance of three model variants. For the “Oracle rationale” setting, the RATIONALESELECTION module is replaced by an oracle which outputs gold rationales for correctly retrieved documents, and no rationales for incorrect retrievals. The “Zero-shot” setting reports the zero-shot generalization performance of a model trained on FEVER (the results on UKP Snopes were slightly worse). VERISCI reports the performance of our best system.

Results The results are shown in Table 4. In the oracle abstract setting, the abstract-level F1 scores are roughly comparable to label classification accuracies, and the AbstractLabel+Rationale score in Row 3 implies an end-to-end classification accuracy of roughly 70%, given gold abstracts.

RetrievalModelSentence-level Selection-Only PSentence-level Selection-Only RSentence-level Selection-Only F1Sentence-level Selection+Label PSentence-level Selection+Label RSentence-level Selection+Label F1Abstract-level Label-Only PAbstract-level Label-Only RAbstract-level Label-Only F1Abstract-level Label+Rationale PAbstract-level Label+Rationale RAbstract-level Label+Rationale F1
Oracle abstractOracle rationale100.080.589.22.189.2_{2.1}89.672.279.93.079.9_{3.0}90.177.583.32.483.3_{2.4}90.177.583.32.483.3_{2.4}
Oracle abstractZero-shot42.545.143.82.043.8_{2.0}36.138.437.22.337.2_{2.3}86.953.666.33.166.3_{3.1}67.941.951.83.451.8_{3.4}
Oracle abstractVERISCI76.163.869.42.669.4_{2.6}66.555.760.63.160.6_{3.1}87.365.374.72.874.7_{2.8}84.963.572.72.972.7_{2.9}
OpenOracle rationale100.056.572.23.372.2_{3.3}87.649.563.23.763.2_{3.7}88.954.167.23.267.2_{3.2}88.954.167.23.267.2_{3.2}
OpenZero-shot28.737.632.52.332.5_{2.3}23.731.126.92.326.9_{2.3}56.042.348.23.348.2_{3.3}42.332.036.43.336.4_{3.3}
OpenVERISCI45.047.346.13.046.1_{3.0}38.640.539.53.039.5_{3.0}47.547.347.43.147.4_{3.1}46.646.446.53.146.5_{3.1}

Table 4. Test set performance on SCIFACT, according to the metrics from § 3. For the “Oracle abstract” rows, the system is provided with gold evidence abstracts. “Oracle rationale” rows indicate that the gold rationales are provided as input. “Zero-shot” indicates zero-shot performance of a verification system trained on FEVER. Additionally, standard deviations are reported as subscripts for all F1 scores. See Appendix § 8 for standard deviations on all reported metrics.

Access to in-domain data during training clearly improves performance. Despite the small size of SCIFACT, training on these data leads to relative improvements of 47% on open SentenceSelection+Label, and 28% on open AbstractLabel+Rationale over FEVER alone (Row 6 vs. Row 5). The three pipeline components make similar contributions to the overall model error. Replacing RATIONALESELECTION with an oracle leads to a roughly 20-point rise in SentenceSelection+Label F1 (Row 6 vs. Row 4). Replacing ABSTRACTRETRIEVAL with an oracle as well leads to a gain of roughly 20 more points (Row 4 vs. Row 1).

Nearly all correctly-labeled abstracts are supported by at least one rationale. There is only a two-point difference in F1 between AbstractLabel-Only and AbstractLabel+Rationale in the oracle setting (Row 3), and a one-point difference in the open setting (Row 6). The differences between SentenceSelection-Only and SentenceSelection+Label are larger, caused by examples where the model finds the evidence but fails to predict its relationship to the claim. We examine these in § 5.4.

We evaluate the statistical robustness of our results by generating 10,000 bootstrap-resampled versions of the test set ([14]) and computing the standard deviation of all performance metrics. Table 4 shows the standard deviations in F1 score. Uncertainties on all metrics for both the dev and test set can be found in Appendix § 8. The results indicate that the observed differences in model performance are statistically robust and cannot be attributed to random variation in the dataset.

Verifying claims about COVID-19

We conduct exploratory experiments using our system to verify claims concerning COVID-19. We tasked a medical student to write 36 COVID-related claims. For each claim cc, we used VERISCI to predict evidence abstracts E^(c)\hat{\mathcal{E}}(c). The annotator examined each (c,E^(c))(c, \hat{\mathcal{E}}(c)) pair. A pair was labeled plausible if E^(c)\hat{\mathcal{E}}(c) was nonempty, and at least half of the evidence abstracts in E^(c)\hat{\mathcal{E}}(c) were judged to have reasonable rationales and labels. For 23 / 36 claims, the response of VERISCI was deemed plausible by our annotator, demonstrating that VERISCI is able to successfully retrieve and classify evidence in many cases. Two examples are shown in Table 1. In both cases, our system identifies both supporting and refuting evidence.

Error analysis

To better understand the errors made by VERISCI, we conduct a manual analysis of test set predictions where an evidence abstract was correctly retrieved, but where the model failed to identify any relevant rationales or predicted an incorrect label. We identify five modeling capabilities required to correct these mistakes (Table 5 provides examples):

Reasoning typeExample
Science backgroundClaim: Rapamycin slows aging in fruit flies. Evidence: … feeding rapamycin to adult Drosophila produces life span extension … Gold Verdict: SUPPORTS Reasoning: Drosophila is a type of fruit fly.
DirectionalityClaim: Inhibiting glucose-6-phospate dehydrogenase impairs lipogenesis Evidence: … suppression of 6PGD increased lipogenesis Gold Verdict: REFUTES Reasoning: A decrease (not increase) in lipogenesis would indicate lipogenesis impairment.
Numerical reasoningClaim: Bariatric surgery improves resolution of diabetes. Evidence: Strong associations were found between bariatric surgery and the resolution of T2DM, with a HR of 9.29 (95% CI 6.84-12.62)... Gold Verdict: SUPPORTS Reasoning: A HR (hazard ratio) that is greater than 1 with 95% confidence indicates improvement.
Cause and effectClaim: Major vault protein (MVP) functions to decrease tumor aggression. Evidence: Knockout of MVP leads to miR-193a accumulation...inhibiting tumor progression Gold Verdict: REFUTES Reasoning: Knocking out (removing) MVP inhibits tumor progression → MVP increases tumor aggression.
CoreferenceClaim: Low saturated fat diets have adverse effects on the development of infants Evidence: Neurological development of children in the intervention group was at least as good as ... the control group Gold Verdict: REFUTES Reasoning: The intervention group in this study was placed on a low saturated fat diet.

Table 5. Reasoning types required to verify SciFACT claims which are classified incorrectly by our modeling baseline. Words crucial for correct verification are highlighted.

Science background includes knowledge of domain-specific lexical relationships.

Directionality requires understanding increases or decreases in scientific quantities.

Numerical reasoning involves interpreting numerical or statistical findings.

Cause and effect requires reasoning about counterfactuals.

Coreference involves drawing conclusions using context stated outside of a rationale sentence.

Related work

Fact checking and rationalized NLP models

Fact-checking datasets include PolitiFact (Vlachos and Riedel, 2014), Emergent ([16]), LIAR (Wang, 2017), SemEval 2017 Task 8 RumorEval ([10]), Snopes (Popat et al., 2017), CLEF-2018 CheckThat! ([2]), Verify ([1]), Perspectrum ([7]), FEVER (Thorne et al., 2018), and UKP Snopes ([20]). Hanselowski et al. [20] provides a thorough review. To our knowledge, there are no existing data sets for scientific claim verification. We refer to our task as “claim verification” rather than “fact-checking” to emphasize that our focus is to help researchers make sense of scientific findings, not to counter disinformation.

Fact-checking is one of a number of tasks where a model is required to justify a prediction via rationales from the source document. The ERASER dataset ([12]) provides a suite of benchmark datasets (including SCIFACT) for evaluating rationalized NLP models.

Related scientific NLP tasks The citation contextualization task ([8, 21]) is to identify spans in a cited document that are relevant to a particular citation in a citing document. Unlike SCIFACT, these citations are not re-written into atomic claims and are therefore more difficult to verify. Expert annotators achieved very low (21.7%) inter-annotator agreement on the BioMedSumm dataset ([9]), which contains 314 citations referencing 20 papers.

Biomedical question answering datasets include BioASQ (Tsatsaronis et al., 2015) and PubMedQA ([22]), which contain 855 and 1,000 “yes / no” questions respectively (Gu et al., 2020). Claim verification and question answering are both-knowledge intensive tasks which require an understanding of the relationship between an input query and relevant supporting text.

Automated evidence synthesis ([3] (Marshall and Wallace 2019; Tsafnat et al. 2014; Marshall et al. 2017)) seeks to automate the process of creating systematic reviews of the medical literature8 – for instance, by extracting PICO snippets (Nye et al., 2018) and inferring the outcomes of clinical trials ([23, 13]). We hope that systems for claim verification will serve as components in future evidence synthesis frameworks.

Conclusion and future work

Claim verification allows us to trace the sources and measure the veracity of scientific claims. These abilities have emerged as particularly important in the context of the current pandemic, and the broader reproducibility crisis in science. In this article, we formalize the task of scientific claim verification, and release a dataset (SciFACT) and models (VERiSci) to support work on this task. Our results indicate that it is possible to train models for scientific fact-checking and deploy them with reasonable efficacy on real-world claims related to COVID-19.

Scientific claim verification presents a number of promising avenues for research on models capable of incorporating background information, reasoning about scientific processes, and assessing the strength and provenance of various evidence sources. This last challenge will be especially crucial for future work that seeks to verify scientific claims against sources other than the research literature – for instance, social media and the news. We hope that the resources presented in this paper encourage future research on these important challenges, and help facilitate progress toward the broader goal of scientific document understanding.

Acknowledgments

This research was supported by the ONR MURI N00014-18-1-2670, ONR N00014-18-1-2826, DARPA N66001-19-2-4031, NSF (IIS 1616112), Allen Distinguished Investigator Award, and the Sloan fellowship. We thank the Semantic Scholar team at AI2, UW-NLP, and H2lab at UW for helpful comments and feedback.

Model implementation details

All models are implemented using the Huggingface Transformers package (Wolf et al., 2019).

Parameters for the final VERISCI system

For the ABSTRACTRETRIEVAL module, VERISCI retrieves the top k=3k = 3 documents ranked by TF-IDF similarity using unigram + bigram features. These parameters are tuned on the SCIFACT development set.

When making predictions using the RATIONALESELECTION module described in § 4, we find that the usual decision rule of predicting z^i=1\hat{z}_i = 1 when z~i≥0.5\tilde{z}_i \geq0.5 works well for models trained on SCIFACT. However, for models trained on FEVER and UKP Snopes, we achieve better performance by tuning the classification threshold tt, such that z^i=1\hat{z}_i = 1 when z~i≥t\tilde{z}_i \geq t, on the SCIFACT dev set. The best threshold was t=0.025t = 0.025 when training on FEVER, and t=0.75t = 0.75 when training on UKP Snopes.

Training the RATIONALESELECTION module

We experiment with various learning rates when training SciBERT, BioMedRoBERTa, RoBERTa-base, and RoBERTa-large. Below we describe the setting for training RoBERTa-large.

For models trained on SCIFACT, we use an initial learning rate of 1e−51\mathrm{e}{-5} on the transformer base and 1e−31\mathrm{e}{-3} on the linear layer. For FEVER + SCIFACT, the learning rate is set to 1e−51\mathrm{e}{-5} for the entire model for pre-training on FEVER and fine-tuning on SCIFACT. We use a batch size of 256 through gradient accumulation and apply cosine learning rate decay over 20 epochs to find the best performing model on the dev set.

For models trained on FEVER, we set the learning rate to 5e−65\mathrm{e}{-6} for the transformer base and 5e−55\mathrm{e}{-5} for the linear layer. For models trained on UKP Snopes, we set the learning rate 1e−51\mathrm{e}{-5} for the transformer base and 1e−41\mathrm{e}{-4} for the linear layer. We find that these learning rates help the models converge. We only train the model for 3 epochs on FEVER and 5 epochs on UKP Snopes because they are larger datasets and the models converged within early epochs.

Training the LABELPREDICTION module

We adopt similar settings as we used for the RATIONALESELECTION module and only change the learning rate to 1e−51\mathrm{e}{-5} for the transformer base and 1e−41\mathrm{e}{-4} for the linear layer for models trained on SCIFACT, FEVER, and UKP Snopes. When training on claim / cited abstract pairs labeled NOINFO, we use the kk sentences in the abstract with greatest similarity to the claim as rationales (§ 4). kk is sampled from {0,1}\{0, 1\} with uniform probability.

Additional training details

All models are trained using a single Nvidia P100 GPU on Google Colabortory Pro platform.9 For the RATIONALESELECTION module, it takes about 150 minutes to train on SCIFACT for 20 epochs, 120 minutes on UKP Snopes for 5 epochs, and 700 minutes on FEVER for 3 epochs. For the LABELPREDICTION module, it takes about 130 minutes to train on SCIFACT for 20 epochs, 160 minutes on UKP Snopes for 5 epochs, and 640 minutes on FEVER for 3 epochs.

Hyperparameter search

The learning rate, batch size, and number of epochs are the most important hyperparameters. We perform manual tuning and select the hyperparameters that produce the highest F1 on the development set. For the learning rate, we experiment with 1e−31\mathrm{e}{-3}, 1e−41\mathrm{e}{-4}, 5e−55\mathrm{e}{-5}, 1e−51\mathrm{e}{-5}, and 5e−65\mathrm{e}{-6}. For batch size, we experiment with 64 and 256. The number of epochs are cutoff after the model converges.

Statistical analysis

We assess the uncertainty in the results reported in the main results (Table 4) using a simple bootstrap approach [6, 15] (Drotar et al. 2018). Given our test set with ntest=300n_{\mathrm{test}} = 300 claims, we generate nboot=10,000n_{\mathrm{boot}} = 10{,}000 bootstrap-resampled test sets by resampling (uniformly, with replacement) ntestn_{\mathrm{test}} claims from the test set. For each resampled test set, we compute the metrics in Table 4. Table 6 reports the mean and standard deviation of these metrics, computed over the bootstrap samples. Table 7 reports dev set metrics. Our conclusion that training on SCIFACT improves performance is robust to the uncertainties presented in these tables.

RetrievalModelRowPRF1PRF1
Oracle abstractOracle rationale1100.00.0100.0_{0.0}80.53.380.5_{3.3}89.22.189.2_{2.1}89.62.789.6_{2.7}72.23.772.2_{3.7}79.93.079.9_{3.0}
Zero-shot242.62.242.6_{2.2}45.23.245.2_{3.2}43.82.043.8_{2.0}36.22.536.2_{2.5}38.43.038.4_{3.0}37.22.337.2_{2.3}
VERISCI376.22.976.2_{2.9}63.93.663.9_{3.6}69.42.669.4_{2.6}66.53.466.5_{3.4}55.73.755.7_{3.7}60.63.160.6_{3.1}
OpenOracle rationale4100.00.0100.0_{0.0}56.64.056.6_{4.0}72.23.372.2_{3.3}87.63.587.6_{3.5}49.53.949.5_{3.9}63.23.763.2_{3.7}
Zero-shot528.72.328.7_{2.3}37.63.437.6_{3.4}32.52.332.5_{2.3}23.82.323.8_{2.3}31.13.131.1_{3.1}26.92.326.9_{2.3}
VERISCI645.03.045.0_{3.0}47.43.847.4_{3.8}46.13.046.1_{3.0}38.53.038.5_{3.0}40.63.640.6_{3.6}39.53.039.5_{3.0}

(a) Sentence-level results.

RetrievalModelRowPRF1PRF1
Oracle abstractOracle rationale190.12.290.1_{2.2}77.52.877.5_{2.8}83.32.483.3_{2.4}90.12.290.1_{2.2}77.52.877.5_{2.8}83.32.483.3_{2.4}
Zero-shot286.92.986.9_{2.9}53.63.453.6_{3.4}66.33.166.3_{3.1}67.93.967.9_{3.9}41.93.241.9_{3.2}51.83.451.8_{3.4}
VERISCI387.32.687.3_{2.6}65.33.265.3_{3.2}74.72.874.7_{2.8}84.92.884.9_{2.8}63.53.263.5_{3.2}72.62.972.6_{2.9}
OpenOracle rationale488.92.788.9_{2.7}54.13.554.1_{3.5}67.23.267.2_{3.2}88.92.788.9_{2.7}54.13.554.1_{3.5}67.23.267.2_{3.2}
Zero-shot556.03.956.0_{3.9}42.33.342.3_{3.3}48.23.348.2_{3.3}42.34.042.3_{4.0}32.03.232.0_{3.2}36.43.336.4_{3.3}
VERISCI647.53.347.5_{3.3}47.33.547.3_{3.5}47.43.147.4_{3.1}46.63.346.6_{3.3}46.43.546.4_{3.5}46.43.146.4_{3.1}

(b) Abstract-level results.

Table 6. Test set results as in Table 4, reporting mean and standard deviation over 10,000 bootstrap samples. Standard deviations are reported as subscripts. Some means reported here are slightly different from Table 4 due to sampling variability.

RetrievalModelRowPRF1PRF1
Oracle abstractOracle rationale1100.00.0100.0_{0.0}81.93.281.9_{3.2}90.01.990.0_{1.9}91.42.591.4_{2.5}74.93.674.9_{3.6}82.32.982.3_{2.9}
Zero-shot240.72.140.7_{2.1}48.13.448.1_{3.4}44.02.144.0_{2.1}36.12.536.1_{2.5}42.63.442.6_{3.4}39.02.539.0_{2.5}
VERISCI379.42.779.4_{2.7}59.03.659.0_{3.6}67.72.867.7_{2.8}71.43.571.4_{3.5}53.03.653.0_{3.6}60.83.360.8_{3.3}
OpenOracle rationale4100.00.0100.0_{0.0}58.44.358.4_{4.3}73.73.473.7_{3.4}90.23.390.2_{3.3}52.74.352.7_{4.3}66.43.966.4_{3.9}
Zero-shot528.62.028.6_{2.0}38.53.638.5_{3.6}32.82.332.8_{2.3}24.82.224.8_{2.2}33.43.433.4_{3.4}28.42.428.4_{2.4}
VERISCI652.53.552.5_{3.5}43.83.743.8_{3.7}47.73.247.7_{3.2}46.93.746.9_{3.7}39.23.639.2_{3.6}42.63.242.6_{3.2}

(a) Sentence-level results.

RetrievalModelRowPRF1PRF1
Oracle abstractOracle rationale191.42.291.4_{2.2}76.13.076.1_{3.0}83.02.583.0_{2.5}91.42.291.4_{2.2}76.13.076.1_{3.0}83.02.583.0_{2.5}
Zero-shot288.92.888.9_{2.8}58.33.758.3_{3.7}70.43.270.4_{3.2}69.23.969.2_{3.9}45.43.545.4_{3.5}54.83.554.8_{3.5}
VERISCI391.02.391.0_{2.3}67.43.367.4_{3.3}77.42.777.4_{2.7}85.22.985.2_{2.9}63.23.563.2_{3.5}72.53.172.5_{3.1}
OpenOracle rationale491.02.691.0_{2.6}53.13.853.1_{3.8}67.03.467.0_{3.4}91.02.691.0_{2.6}53.13.853.1_{3.8}67.03.467.0_{3.4}
Zero-shot552.73.752.7_{3.7}41.63.741.6_{3.7}46.53.446.5_{3.4}43.63.743.6_{3.7}34.43.534.4_{3.5}38.43.338.4_{3.3}
VERISCI655.43.755.4_{3.7}47.53.647.5_{3.6}51.03.351.0_{3.3}52.63.752.6_{3.7}45.13.645.1_{3.6}48.53.348.5_{3.3}

(b) Abstract-level results.

Table 7. Dev set results as in Table 4, reporting mean and standard deviation over 10,000 bootstrap samples.

JournalCount
BMJ60
Blood8
Cancer Cell8
Cell51
Cell Metabolism10
Cell Stem Cell41
Circulation12
Immunity33
JAMA79
Molecular Cell27
Molecular Systems Biology5
Nature29
Nature Cell Biology26
Nature Communications19
Nature Genetics8
Nature Medicine89
Nature Methods1
Nucleic Acids Research10
Plos Biology36
Plos Medicine38
Science7
Science Translational Medicine2
The Lancet22
Other120
Total741

Table 8. Number of cited documents by journal. Some co-cited articles (§ 2.1) come from journals outside our curated set; these are indicated by “Other”.

Dataset collection and corpus statistics

Corpus

Source journals Table 8 shows the number of cited abstracts from each of our selected journals. The “Other” category includes “co-cited” (§ 2.1) abstracts that came from journals not among our pre-defined set.

Distractor abstracts In § 2.1, we mention how we increase the size of the corpus by adding distractor abstracts. The reason why we do not use the entirety of a large research corpus like S2ORC as our fact-checking corpus is that doing so would introduce many false negative retrievals: abstracts containing evidence relevant to a given claim, but not mentioned in the claim’s source citance. This can occur either because the citance authors simply were not aware of these abstracts, or because the abstracts were published after the citance was written. These retrievals would be incorrectly marked wrong by our evaluation metrics.

Distractor abstracts as defined in § 2.1 have two qualities that make them a good addition to the SciFact corpus: (1) They are cited in the same articles as our evidence abstracts, meaning that they often discuss similar topics and increase the difficulty of abstract retrieval methods based on lexical similarity. (2) The authors of our citances were aware of the distractor abstracts, and chose not to mention them in the citances used to generate claims. This makes them unlikely to be a source of false negative retrievals.

Annotation examples

Converting citances to claims Figure 4 shows an example of a citance re-written as a claim. The citance discusses the relationship between “atherosclerotic CVD” and “IL-6”, and cites two papers (44, 45) as evidence. To convert to a claim, the acronym “CVD” is expanded to “cardiovascular disease”, irrelevant information is removed, and the claim is written as an atomic factual statement.

Figure 4. A claim written based on a citance. Material unrelated to the citation is removed. The acronym “CVD” is expanded to “cardiovascular disease”.

Multiple rationales Figure 5 shows a claim supported by two rationales from the same abstract. The text of each rationale on its own is sufficient to entail the claim.

Claim supported by two rationales

Figure 5. A claim supported by two rationales from the same abstract. The text of each rationale on its own provides sufficient evidence to verify the claim.

Annotators and quality control

Claim writing Student claim writers attended an in-person training session where they were introduced to the task and received in-person feedback from the four experts. Following training, student annotators continued writing claims remotely. The expert annotators monitored claims for quality during the remote annotation process, and provided feedback when necessary; low-quality claims were returned to the annotators for re-writing. As a final check, all submitted claims were proofread (and edited if necessary) by an undergraduate whose claims were deemed especially high-quality by the expert annotators.

Claim negations As mentioned in § 2.2, an expert annotator wrote claim negations to introduce cases where an abstract REFUTES a claim. The annotator skipped claims that could only be negated by adding obvious triggers like “not”. The majority of claim negations involved a reversal of effect direction; for instance “A high microerythrocyte count protects against severe anemia” can be negated as “A high microerythrocyte count raises vulnerability to severe anemia”.

Claim verification Annotations were performed remotely through a web interface. Annotators were required to pass a 10-question “quiz” before annotating their own claims. After passing the quiz, subsequent submissions were reviewed by an NLP expert until that expert deemed the annotator reliable. Approved annotators were then assigned to review each others’ submissions. In general, graduate students were assigned to review annotations from undergraduates.

Annotation interfaces and guidelines

We show a screenshot of the claim writing interface in Figure 6, and the claim verification interface in Figure 7. The complete annotation guide for claim verification is available at the following URL: https://scifact.s3-us-west-2.amazonaws.com/doc/evidence-annotation-instructions.pdf.

The claim-writing interface

Figure 6. The claim-writing interface. The citation sentence is highlighted in blue on the top left. Additional context is provided on bottom left. The right side shows two claims that could be written based on this citation sentence.

The evidence collection interface

Figure 7. The evidence collection interface.

References

  1. [1]Ramy Baly, Mitra Mohtarami, James Glass, Lluís Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact checking in a unified corpus. In NAACL.arxiv.org/abs/1804.08012
  2. [2]Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Lluís Màrquez i Villodre, Pepa Atanasova, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the clef-2018 checkthat! lab on automatic identification and verification of political claims. task 2: Factuality. In CLEF.
  3. [3]Elaine Beller, Justin Clark, Guy Tsafnat, Clive Elliott Adams, Heinz Diehl, Hans Lund, Mourad Ouzzani, Kristina Thayer, James Thomas, Tari Turner, J. S. Xia, Karen A. Robinson, and Paul P. Glasziou. 2018. Making progress with the automation of systematic reviews: principles of the international collaboration for the automation of systematic reviews (icasr). Systematic Reviews, 7.DOI
  4. [4]Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In EMNLP.arxiv.org/abs/1903.10676
  5. [5]Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. ArXiv, abs/2004.05150.arxiv.org/abs/2004.05150
  6. [6]Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in nlp. In EMNLP.
  7. [7]Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing things from a different angle: Discovering diverse perspectives about claims. In NAACL.arxiv.org/abs/1906.03538
  8. [8]Arman Cohan, Luca Soldaini, and Nazli Goharian. 2015. Matching citation text and cited spans in biomedical literature: a search-oriented approach. In NAACL.DOI
  9. [9]Kevin Bretonnel Cohen, Hoa Trang Dang, Anita de Waard, Prabha Yadav, and Lucy Vanderwende. 2014. Tac 2014 biomedical summarization track. https://tac.nist.gov/2014/BiomedSumm/.
  10. [10]Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for rumours. In SemEval.arxiv.org/abs/1704.05972
  11. [11]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL.arxiv.org/abs/1810.04805
  12. [12]Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020a. Eraser: A benchmark to evaluate rationalized nlp models. In ACL.arxiv.org/abs/1911.03429
  13. [13]Jay DeYoung, Eric Lehman, Ben Nye, Iain James Marshall, and Byron C. Wallace. 2020b. Evidence inference 2.0: More data, better models. In BioNLP@ACL.arxiv.org/abs/2005.04177
  14. [14]Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In ACL.
  15. [15]Bradley Efron and Robert Tibshirani. 1993. An introduction to the bootstrap.
  16. [16]William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In NAACL.DOI
  17. [18]Suchin Gururangan, Ana Marasovic, Swabha Swamyadipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In ACL.arxiv.org/abs/2004.10964
  18. [19]Suchin Gururangan, Swabha Swamyadipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In NAACL.arxiv.org/abs/1803.02324
  19. [20]Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. 2019. A richly annotated corpus for different tasks in automated fact-checking. In CoNLL.arxiv.org/abs/1911.01214
  20. [21]Kokil Jaidka, Muthu Kumar Chandrasekaran, Devanshu Jain, and Min-Yen Kan. 2017. The clef-sisumm shared task 2017: Results and key insights. In BIRNDL@JCDL.
  21. [22]Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In EMNLP.arxiv.org/abs/1909.06146
  22. [23]Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C. Wallace. 2019. Inferring which medical treatments work from reports of clinical trials. In NAACL.arxiv.org/abs/1904.01606
  23. [24]Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2016. Rationalizing neural predictions. In ACL.arxiv.org/abs/1606.04155
  24. [25]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.arxiv.org/abs/1907.11692
  25. [26]Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S. Weld. 2020. S2ORC: The Semantic Scholar Open Research Corpus. In ACL.arxiv.org/abs/1911.02782

Paper details

vlnc:W_mvfv3jya
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents