betaFeedback

WorksComputer Science

Overview of the NTCIR-19 SciClaimEval Task: Cross-modal Claim Verification in Scientific Papers

Abstract

This paper presents an overview of SciClaimEval, a pilot task at NTCIR-19 on verifying scientific claims against the tables and figures of the papers in which they appear. Our dataset pairs authentic claims from 180 papers in machine learning, natural language processing, and medicine with contrastive versions of their tables and figures, one supporting the claim and one minimally modified so that it does not. The task comprises two subtasks: deciding whether a piece of evidence supports a claim, and selecting which of two evidence versions does. Twenty-two teams registered, the highest number among NTCIR-19 tasks, and 14 submitted 177 runs. Most systems built on multimodal large language models, relying on prompting and multi-stage inference rather than task-specific fine-tuning. On the first subtask, the best systems reached 98.1% pair accuracy, but all six top-ranked systems relied on the contrastive pairs; the best system that judged each claim-evidence pair on its own scored 24.3 points lower. Open-weight models, embedded in carefully designed pipelines, matched proprietary ones. Yet verifying a single claim against its evidence, the setting that matters in practice, remains an open problem.

Introduction

Scientific papers make claims that should be supported by the evidence they present, typically results reported in tables and figures. Scientific claim verification asks whether a claim is adequately supported by this evidence [5, 8, 11]. Unlike general fact-checking [3, 7], which verifies claims against external knowledge, it focuses on the internal consistency between a paper’s claims and its own evidence. Checking this consistency is a fundamental part of peer review, yet it is time-consuming and increasingly difficult as submission volumes grow.

Existing datasets for scientific claim verification differ mainly in the evidence they consider. Tables are the most common, either in structured formats such as JSON or XML, as in SciTab [5], SEM-TAB-FACTs [11], and SciAtomicBench [13], or as images, as in MMSCi-Eval [12]. Others turn to figures, such as MuSciClaims [4], whose figures often contain complex subplots and embedded tables, while SciVer [10] considers multiple charts, tables, and text paragraphs as evidence. Despite this progress, three limitations remain. Refuted examples are usually created by editing the claim, for instance by flipping its meaning, which can leave surface cues that models exploit instead of reasoning over the evidence. Many datasets also rely on claims that are not taken directly from scientific papers. Finally, tables are typically available in a single format, so it remains unclear how well models handle the same content across different representations.

With these limitations in mind, we organized SciClaimEval, a pilot task at NTCIR-19 on verifying scientific claims against the tables and figures of the papers in which they appear. It complements earlier shared tasks on scientific claim verification, which used tables (SemEval-2021 Task 9 [11]), abstracts (SCIVER [9]), or social media posts linked to their source papers (CheckThat! [6]). Rather than editing claims, we create refuted examples by having annotators modify the evidence. Each claim is thus paired with two versions of its evidence, one that supports it and one that does not, so the wording of the claim carries no cue about the label. All claims are authentic sentences taken from published papers, and tables are released in four formats: images, LaTeX, HTML, and JSON.

The SciClaimEval dataset is built in three steps. We first collect peer-reviewed papers in natural language processing (NLP) and machine learning (ML), accepted at ACL venues and NeurIPS, and in medicine, published in PeerJ. From each paper, we extract the text, tables, and figures. We then identify sentences that explicitly reference a table or figure (e.g., “Table 1”), as well as other sentences in the same paragraph, and pair each sentence individually with the referenced evidence. Finally, annotators check whether each claim is supported by its evidence and, for supported claims, edit the evidence to create an unsupported counterpart. The resulting dataset contains 1,633 claims from 180 papers, with both table- and figure-based evidence across all three domains (Section 2.1).

PropertyValTestAll Data
Labels
#Supported389474863
#Refuted338432770
Modality Type
Table469518987
Figure258388646
Context Use
No4776111,088
Short Context142188330
Full Paper108107215
Domain
NLP379384763
ML157236393
Medicine191286477
Total examples7279061,633

Table 1. Dataset statistics of SciClaimEval.

The pilot task comprises two subtasks. In Subtask 1, systems decide whether a given piece of evidence supports a claim. In Subtask 2, systems receive a claim with its two evidence versions and must select the one that supports it. Twenty-two teams registered, the highest number among NTCIR-19 tasks, and 14 of them submitted 177 runs in total: 106 for Subtask 1 and 71 for Subtask 2.

Most systems built on multimodal large language models (MLLMs). For Subtask 1, they followed two broad strategies. Some verified each claim-evidence pair independently, often with claim decomposition, structured extraction of evidence from images, or executable checks. Others exploited the paired structure of the data by comparing the two evidence versions of the same claim. For Subtask 2, systems primarily adopted direct candidate comparison, independent verification followed by selection, or hybrid approaches combining both strategies. Common techniques included ensembles, multi-agent debate, evidence extraction, and consistency checks under reversed candidate order.

We draw the following lessons from this pilot task:

  • High scores rely on paired evidence. On Subtask 1, the best systems reached 98.1% pair accuracy on the test set by comparing the two evidence versions of each claim. Teams that also evaluated their systems without this information observed substantial performance drops. On the development set, for instance, removing paired information cost the two top systems 12 and 21 points of pair accuracy.

  • Open-weight models can match proprietary ones. A structured pipeline built around an open-weight model, with selective human review, matched direct prompting of a proprietary model on Subtask 1 (98.1% pair accuracy for both) and came close on Subtask 2 (98.1% versus 98.4% accuracy).

  • Verifying a single claim against its evidence remains an open problem. In practice, reviewers and authors face one claim and one piece of evidence, with no contrasting version to compare against. This is precisely the setting in which performance drops, and one that our paired design did not measure directly.

Task Design and Dataset

Dataset Creation

Figure 1 illustrates our dataset construction pipeline, which consists of three steps: data preparation, automatic claim-evidence extraction, and human annotation.

Figure 1. Our dataset construction pipeline consists of three main steps: data preparation, automatic claim-evidence extraction, and human annotation. The human annotation process involves two tasks: claim-evidence verification and evidence modification. After collecting all samples, we design two subtasks in our dataset: claim-label prediction and claim-evidence prediction.

Data Preparation. We collect papers from three domains. For NLP, we take papers from the ACL Anthology and map them to their arXiv versions when available. For ML, we select arXiv papers whose comments indicate acceptance at conferences such as NeurIPS. For both domains, we retrieve the HTML-rendered papers from ar5iv [1] and parse them into JSON containing the title, abstract, and section paragraphs. Figures are extracted from ar5iv as PNG files, and tables are extracted from the LaTeX source and kept in LaTeX format. For medicine, we collect papers published in PeerJ between 2024 and 2025 in the Medicine Articles category. We parse their HTML versions into JSON, storing figures as PNG files and tables in HTML format.

Claim-Evidence Extraction. We identify paragraphs that refer to a table or figure using regular expressions that match references such as “Table 1”, “Tab. 1”, or “Fig. 2”. Each such paragraph is split into sentences, and each sentence is paired with the referenced table or figure, yielding a set of candidate claim-evidence pairs. Not every sentence is a verifiable claim, so these candidates are filtered during human annotation. We consider only the main text, excluding appendices and their tables and figures.

Human Annotation. Annotation was carried out by 11 graduate students and researchers, in two stages: claim-evidence verification and evidence modification.

In the first stage, annotators see a candidate claim, its table or figure, and the preceding sentences of the same paragraph as context (empty if the claim is the first sentence). They label each pair as Supported, Subjectively Supported (the claim relies on subjective terms such as “large margin” that cannot be checked against the evidence), Unsupported, or Skip (the claim is purely descriptive, problematic, or requires knowledge the annotator lacks). They also indicate whether the claim can be understood on its own, needs the short context, or requires the full paper.

In the second stage, annotators work only on Supported pairs. They modify the table or figure using predefined operations so that the claim is no longer supported by the modified evidence. For tables, they change cell values, swap row or column names, or add and remove rows or columns. For figures, they flip graphs, swap legends or category labels, or exchange subgraphs from the same paper. Annotators may also propose other edits, such as adding spurious data points (Figure 2). For a few claims, no meaningful modification was possible, and only the supported pair was kept. Further details are given in Ho et al. [2].

Figure 2. An example from our dataset involves creating an unsupported claim by adding spurious data points to the figure. The context provides the necessary information to understand the plot in the evidence.

Dataset Statistics

Table 1 reports statistics for the validation set, the test set, and the full dataset, broken down by modality (table or figure), context use, and domain. The split into validation and test sets is balanced with respect to context use and evidence modification operations. The dataset contains 863 supported and 770 refuted samples. Although each supported claim is meant to have a refuted counterpart, 93 have none, because no modification of the evidence could be found that was both logical and challenging.

Note that the statistics in Table 1 differ slightly from those reported in Ho et al. [2]. Following feedback from participants and further analysis, we removed a small number of samples to improve the quality of the dataset. Both the original version (v1.0) and the updated version (v1.1) are available on our Hugging Face page2 for reproducibility.

Dataset Analysis

In addition to the analyses of Text Lengths, Evidence-Modifying Operations, and Pixel Changes in Figure Evidence Modification presented in Ho et al. [2], we extend the analysis to include chart types. We manually categorized all figures in the development and test sets. First, we classified figures into four categories based on their plot structure: Single (a single plot), Multiple (multiple unlabeled plots), Subplots (multiple labeled plots, e.g., (a), (b)), and Multiple Subplots (a combination of multiple plots and subplots). Each individual plot, including those within multiple plots or subplots, may contain a combination of different chart types. The most common combinations are bar charts with error bars or margins. For example, such a figure is categorized as single in terms of plot structure but assigned both bar chart and error plot as chart types. Figure 3 shows the distribution of plot structures and chart types across the entire SciClaimEval dataset (development and test splits). We observe a clear bias toward single plots, with line charts, bar charts, and error plots being the most prevalent chart types. Error plots rank among the top three primarily because they frequently appear alongside line and bar charts, but rarely occur independently.

Figure 3. Distribution of all figure types in test and dev split.

Task Design

SciClaimEval comprises two subtasks.

Subtask 1: Claim-Label Prediction. This is the main subtask. Given a claim, a piece of evidence (a table or a figure) with its caption, and the contextual information, systems predict whether the claim is supported or refuted by the evidence. This subtask evaluates whether systems can understand a claim, interpret the evidence, and judge whether the two are consistent.

Subtask 2: Claim-Evidence Prediction. Given a claim, its context, and the two versions of its evidence (the original and the modified one), systems select the version that supports the claim. The two versions often differ by a single cell or legend entry, so this subtask tests whether systems can find the detail on which the claim depends.

Evaluation Metrics

Subtask 1. We report precision, recall, macro-F1, accuracy, and pair accuracy, and use the last two as primary metrics. Pair accuracy counts a claim as correct only if both of its samples, supported and refuted, are classified correctly. Claims without a refuted counterpart are excluded from this metric.

Subtask 2. We report accuracy, the proportion of claims for which the supporting evidence is correctly selected. The random baseline is 50%.

Participating Systems

Fourteen teams submitted 177 runs to SciClaimEval, with thirteen teams taking part in each subtask (Table 2). This section summarizes the main approaches for each subtask, based on the teams’ system description papers.

Team IDGroup nameCountrySubtasks# Runs
SIBSIB Text MiningSwitzerland1, 28
bit-bias9Bonn-Juelich InformaticsGermany1, 244
tabr25Black SocksAustralia1, 212
ECS-UoSUniversity of SouthamptonUK1, 22
ditlabDenso IT LaboratoryJapan1, 25
trace29SEBIS TUM teamGermany1, 22
tlabTakasuLabJapan1, 27
procr astPro²CrastJapan1, 227
SciNexSciNexJapan1, 28
ProvideQProvideQGermany1, 230
SCIAI19VerifyAIGermany1, 24
TUMNLP26LMU & TUM NLP teamGermany27
SciTrueSciTrueNew Zealand1, 218
CAISAL26CAISALabGermany13
Total177

Table 2. Participating teams, the subtasks they submitted runs to, and their number of runs.

Subtask 1: Claim-Label Prediction

Approaches to Subtask 1 followed three main directions.

Exploiting the Paired Structure. The most prominent strategy coordinated predictions across the two samples of each claim. Black Socks, Southampton, and ditlab compared the two evidence versions associated with the same claim and assigned them complementary labels. SciTrue combined model support scores with a pairing constraint, while Bonn-Juelich mapped its Subtask 2 selections back to Subtask 1 labels. By drawing on information beyond a single claim-evidence pair, these approaches achieved the highest scores. Their gains, however, do not transfer to real-world verification, where a claim is assessed without a second, contrasting version of its evidence.

Separating Extraction from Judgment. Several teams first turned the evidence into text and then judged the claim against it. VerifyAI transcribed images into text for a separate evaluator. ProvideQ converted tables into HTML and extracted structured descriptions of figure elements before verification. SciNex explored structured prompting and a two-stage variant that supplemented figure images with descriptions generated without seeing the claim. Strengthening Verification. Other teams made the verification step itself more rigorous. Bonn-Juelich combined claim decomposition, context retrieval, and iterative checks. Takasu Lab checked verifiable conditions with Python code, and CAISALab explored executable Prolog queries. Pro²Crast re-verified low-confidence predictions against the full paper, and SIB Text Mining used contradiction-focused prompting and fine-tuning on synthetic data.

Subtask 2: Claim-Evidence Prediction

Approaches to Subtask 2 also followed three main directions.

Direct Comparison. Most teams presented both candidates to the model at once. SciTrue prompted individual models to compare the candidates directly, or fused their scores in an ensemble. Bonn-Juelich built a more elaborate joint pipeline, combining claim decomposition, discrepancy reports, iterative verification, and candidate-order checks, with selective human review. Pro²Crast supplemented an initial comparison with opposing arguments or interactive debate before a judge selected the evidence, while ditlab explored direct selection across evidence formats and context settings.

Independent Verification, then Selection. Other teams first assessed each candidate on its own. Takasu Lab generated separate verification traces, including deterministic checks and intermediate reasoning, which a text-based selector then compared. Southampton and SIB Text Mining used conditional combinations: they initially verified candidates independently and escalated ambiguous cases to joint comparison. This allowed them to retain independent judgments when these clearly distinguished the candidates and reserve additional comparison for uncertain decisions.

Separating Extraction from Judgment. As in Subtask 1, several teams first extracted information from the evidence, but used it to support a joint decision rather than to label each candidate. VerifyAI compared independently generated transcripts, while SciNex jointly evaluated claim-blind descriptions alongside the original images. LMU & TUM NLP explored selection from structured figure descriptions, and ProvideQ separately grounded each figure before comparing the extracted information. These approaches used separate extraction to support a joint decision, rather than independently assigning support labels to the candidates.

Experimental Results

Fourteen teams submitted 177 runs: 106 for Subtask 1 and 71 for Subtask 2, with thirteen teams participating in each subtask. The official ranking is based on runs using the PNG version of the evidence, since structured formats such as JSON, LaTeX, and HTML are likely easier for language models to interpret. Participants could also submit optional runs using these structured formats. Note that all scores reported here are evaluated on the test set of our dataset.

Subtask 1: Claim-Label Prediction

Table 3 shows the best-performing runs² of each team for Subtask 1, where systems are ranked by pair accuracy. Bonn-Juelich Informatics and SciTrue achieved the highest pair accuracy of 98.1%, with SciTrue slightly ahead on all other metrics. Both systems clearly outperformed the other participants, with the third-ranked team trailing by 8.1 percentage points in pair accuracy.

TeamPRMacro-F1Acc.Pair Acc.
SciTrue†98.098.098.098.098.1
Bonn-Juelich Informatics†97.297.397.297.298.1
Black Socks†94.193.793.893.890.0
Pro²Crast†87.287.287.287.286.8
University of Southampton†85.885.985.885.985.9
Denso IT Laboratory†74.174.174.174.175.9
TakasuLab86.786.486.586.573.8
SEBIS TUM72.772.772.772.772.9
VerifyAI85.385.485.385.372.2
ProvideQ†84.884.984.984.971.1
SIB Text Mining81.381.381.181.164.1
Baselines80.479.579.879.461.1
SciNex77.577.577.477.457.4
CAISALab62.359.858.260.826.9

Table 3. Best run of each team on Subtask 1 (PNG evidence), ranked by pair accuracy. † indicates runs that use the paired structure of the test data.

For Subtask 1, the task design did not explicitly prohibit systems from exploiting the paired structure of the test data. Based on our review of the system descriptions, the six top-ranked systems used this information, typically by assigning complementary labels to the two evidence versions of each claim. Therefore, their results reflect a setting where both evidence versions are available, rather than the intended setting where each claim-evidence pair is evaluated independently. Among systems that did not exploit paired information, TakasuLab achieved the highest pair accuracy of 73.8%, which is 24.3 percentage points lower than the top-performing systems. A similar performance drop was observed by the two leading teams on the development set when paired information was removed, with pair accuracy decreasing from 95.7% to 74.4% and from 94.3% to 82.7%, respectively. Additionally, Bonn-Juelich Informatics incorporated selective human review, meaning its runs were not fully automatic. In Table 3, runs that exploited paired information are marked with †.

Tables 4 and 5 show the results for Subtask 1 using the optional structured formats, JSON and LaTeX/HTML, respectively. Surprisingly, runs using these formats did not clearly outperform those using PNG evidence. One possible reason is that most teams focused on multimodal models, in line with the primary goal of the task, and optimized their systems for images. Additionally, fewer runs were submitted using these formats, limiting direct comparisons. However, other factors may also contribute to these results, and no firm conclusions can be drawn.

TeamPRMacro-F1Acc.Pair Acc.
SciTrue98.198.198.198.198.4
Black Socks93.193.293.293.288.7
Pro2Crast83.683.783.683.783.6
TakasuLab85.084.984.985.071.3
SciNex78.378.177.877.859.7
CAISALab65.764.764.465.232.6

Table 4. Subtask 1 results (JSON evidence).

TeamPRMacro-F1Acc.Pair Acc.
SciTrue98.198.198.198.198.4
Bonn-Juelich Informatics96.796.796.796.797.7
ProvideQ89.990.090.090.082.4

Table 5. Subtask 1 results (TeX/HTML evidence).

Subtask 2: Claim-Evidence Prediction

Table 6 presents the best-performing runs of each team for Subtask 2, where systems are ranked by accuracy. SciTrue achieved the highest accuracy, outperforming Bonn-Juelich Informatics by a narrow margin of 0.3 percentage points. The gap between the top-performing systems and the other participants was smaller than in Subtask 1, with the third-ranked team, the University of Southampton, achieving 96.3%, 2.1 percentage points behind SciTrue.

TeamAccuracy
SciTrue98.4
Bonn-Juelich Informatics98.1
University of Southampton96.3
TakasuLab94.0
ProvideQ91.7
SIB Text Mining90.5
SciNex88.2
Baselines81.9
LMU & TUM NLP81.5
Pro2Crast80.8
Denso IT Laboratory61.3
SEBIS TUM60.6

Table 6. Best run of each team on Subtask 2 results (PNG evidence).

The baseline achieves 81.9% accuracy on Subtask 2, compared to 61.1% pair accuracy on Subtask 1, with the six top-performing systems all achieving scores above 90%. This performance difference is expected, as selecting the better-supported evidence from two nearly identical pieces of evidence requires only a relative judgment. Tables 7 and 8 present the results for Subtask 2 using JSON and LaTeX/HTML formats, respectively. Similar to Subtask 1, runs using structured formats did not show a clear advantage over those using PNG evidence. This may be because most participating teams prioritized multimodal approaches and optimized their systems for image-based evidence. The smaller number of submissions using structured formats also limits the comparison. Therefore, further investigation is needed to determine whether structured representations provide advantages over image-based evidence for this task.

TeamAccuracy
TakasuLab93.5
SciNex89.3
Black Socks84.5
LMU & TUM NLP83.6
VerifyAI66.7
Pro2Crast66.7
Denso IT Laboratory65.5

Table 7. Subtask 2 results (JSON evidence).

TeamAccuracy
Bonn-Juelich Informatics97.7

Table 8. Subtask 2 results (TeX/HTML evidence).

Analyses

Accuracy Across Evidence Modifications

Figure 4 shows the accuracy across all modifications applied to tables and figures. We would expect certain modifications to be more difficult to detect than others. For example, swapping rows or columns is likely easier to identify than modifying individual cell values. Interestingly, however, teams struggled with different types of modifications and evidence. While some teams performed better on tables, as expected given their more structured nature compared to figures, others showed a performance drop on tables.

Figure 4. Accuracy on Subtask 1 across evidence modification categories. The red and green percentages indicate performance drops and gains, respectively, compared to the overall average (black triangle marker). The brown and turquoise triangle markers indicate the average accuracy for tables and figures, respectively.

Figure 4 also reveals substantial variation in performance across modification types, with no single modification consistently challenging all teams. For example, the table modification others caused VerifyAI and ProvideQ to perform 45% below their respective averages, and CAISALab nearly 60% below its average. In contrast, SciTrue, Bonn-Juelich Informatics, University of Southampton, SIB Text Mining, and SEBIS TUM all achieved 100% accuracy on the same modification. Similarly, graph swaps particularly challenged the top-performing teams, while Takasu Lab and SciNex achieved 100% accuracy. Notably, performance on graph swaps and category swaps likely contributed to SciTrue outperforming Bonn-Juelich Informatics in the overall ranking.

Figures 5 and 6 present the same analysis as Figure 4, but separately for figures and tables, respectively. The results reveal that the top two teams, SciTrue and Bonn-Juelich Informatics, outperformed other teams primarily on tables. While most teams struggled with tables and exhibited large performance variations across modification types, the top two teams remained relatively unaffected, with performance spreads of only around 2%. Notably, Bonn-Juelich Informatics made only three mistakes across all tables.

For figures, ProvideQ and VerifyAI achieved the smallest performance spreads of 9.6% and 10.4%, respectively, followed by SciTrue at 11.1%. This difference likely played a key role in determining the winner of this year’s SciClaimEval task, as Bonn-Juelich Informatics exhibited a much larger spread of 22.3%, including a performance drop of 16.6% on graph swaps alone.

Figure 5. Accuracy on Subtask 1 across figure modification categories. The red and green percentages indicate performance drops and gains, respectively, compared to the average accuracy across all figures (black triangle marker).

These large performance variations further highlight that no team developed a solution that performed consistently well across all evidence types. Instead, teams demonstrated strong performance on specific types of claim–evidence pairs.

Figure 7 presents the same Cleveland plot as Figure 4, but for Subtask 2 submissions. While the results largely align with those of Subtask 1, some interesting outliers emerge. For example, SciTrue, Bonn-Juelich, and Southampton achieved identical scores on graph swap modifications. In contrast, on Subtask 1, SciTrue performed significantly better, while Southampton performed significantly worse on this modification type.

Figure 7. Accuracy on Subtask 2 across evidence modification categories. The red and green percentages indicate performance drops and gains, respectively, compared to the overall average accuracy (black triangle marker). The brown and turquoise triangle markers indicate the average accuracy for tables and figures, respectively.

Accuracy Across Figure Types

Figure 8 shows the accuracy of each team’s best run across figure categories. Note that the categorization was performed independently of the claims. In other words, low performance on a particular chart type, such as a violin plot, does not necessarily indicate that an approach struggles to interpret that type. Instead, it indicates lower performance when that chart type is present in the figure, regardless of whether the claim refers to it. Furthermore, unlike in Figure 3, each chart type was counted only once per figure for evaluation.

Figure 8. Best-run accuracy on Subtask 1 across figure types. Single refers to a single plot, which may combine multiple chart types (e.g., a bar chart with error bars). Multiple refers to multiple unlabeled plots, Subplots refers to multiple labeled plots (e.g., (a), (b)), and Multiple Subplots refers to a combination of labeled and unlabeled plots. Italicized categories (middle part) represent chart types across all figure structures (Single, Multiple, Subplots, and Multiple Subplots). The bottom category, Combination, refers to figures containing two or more chart types, regardless of their structure.

Figure 8 reveals a clear and consistent performance gap between the bottom six runs and the top-performing teams. However, no clear correlation between chart types and performance emerges. Stacked bar charts generally lead to lower performance but pose no difficulty for SciTrue. Similarly, heatmaps prove challenging for most teams except Pro2Crast.

Figure 6. Accuracy on Subtask 1 across table modification categories. The red and green percentages indicate performance drops and gains, respectively, compared to the average accuracy across all tables (black triangle marker).

Another observation from Figure 8 is that visual noise (additional irrelevant information) appears to have little impact, as single figures outperform multiple subplots and subplots by only 4% and 5% on average, respectively. Although figures containing multiple plots show a substantial average performance drop of up to 9%, this is primarily driven by lower-performing teams. These findings suggest that no single approach performs equally well across all chart types, further supporting the observations in Figure 4 and Figure 7.

Label and Evidence Selection Bias

Figure 9 and Figure 10 show the biases toward selected labels in Subtasks 1 and 2, respectively. In Subtask 1, no clear bias toward either supported or refuted labels emerges across teams and submitted runs. Only a few outliers, particularly several baselines, exhibit a clear bias toward supported cases. CAISALab’s best-performing run also shows a substantial bias toward predicting supported labels.

Figure 9. Label prediction bias across all runs in Subtask 1.

Figure 10. Evidence selection bias across all runs in Subtask 2.

Surprisingly, almost all teams and submitted runs, except for one run by Pro²Crast, favored the first provided evidence in Subtask 2. The reason for this bias remains unclear. One possible explanation is that processing two images may affect the attention mechanisms of MLLMs, causing them to default to the first input. Notably, no team’s best-performing run was able to overcome this bias, despite the wide variety of MLLMs and pipeline approaches employed.

Sample Difficulty and Run Similarity

Figure 11 shows the evaluation results of all submitted runs across all claim–evidence samples. We observe that most teams struggled with similar samples and performed well on the same easier samples, as indicated by the red regions in the bottom-left and gray regions in the top-right of the matrix. However, only a relatively small proportion of samples were incorrectly labeled by most submitted runs. In fact, no single sample was incorrectly labeled by all runs.

Figure 11. Full evaluation results for each submitted run and claim–evidence sample. Red and gray markers indicate incorrect and correct predictions, respectively. Queries are sorted by consensus from hardest (left, most runs predicted incorrectly) to easiest (right, most runs predicted correctly), with figure samples on the left and table samples on the right. The rightmost gray labels show the average accuracy of each run. At the bottom, P(correct)P(\mathrm{correct}) represents the fraction of runs with correct predictions for each sample, while Agreement indicates the agreement among all runs on the predicted label. Majority correct is marked red if more than half of the runs predicted incorrectly and gray otherwise.

This pattern is consistent with what we would expect from a high-quality dataset without strong systematic biases.

One notable outlier is the University of Southampton’s run, which does not show an increasing number of incorrect predictions toward the harder samples on the left. In other words, this run performed relatively well on samples that most other teams struggled with. This distinction becomes even more apparent in Figure 12, which presents a pairwise agreement matrix based on Cohen’s κ\kappa. The University of Southampton is again a clear outlier, showing its third-highest agreement with Denso IT Laboratory despite outperforming it by 11.8% in accuracy (and 10% in pair accuracy). Figure 8 also showed this team outperformed most of the other team’s in its vicinity on the two most challenging figure types stacked bar charts and violin plots. However, it remains unclear why the University of Southampton’s approach exhibits relatively low agreement with other similarly high-performing teams.

Figure 12. Similarity (Cohen’s κ\kappa) between each team’s best-performing run, with runs sorted by accuracy.

Impact of Pair Information on Performance

As earlier pointed out, one of the most common themes to tackle subtask 1 was the exploitation of pair information. This is a clear shortcoming of our task design that we overlooked in favour of a more robust evaluation. However, it is immediately obvious that this feature made the task significantly easier. In fact, we designed subtask 2 specifically to illustrate the different challange between subtask 1 and 2. As humans, analyzing an example, such as in Figure 2, is also significantly easier with both evidence figures present as it allows us to determine changes and guide our judgement visually first before understanding the claim.

Most teams acknowledged that exploiting pair information leads to unrealistic evaluations and self-reported results without utilizing pairs at least for some earlier iterations of their run submissions. Figure 13 shows these self-reported performance gains. Using this information made the task substantially easier for all teams and, in extreme cases, effectively reduced Subtask 1 to Subtask 2. However, these findings highlight an important conclusion for SciClaimEval: scientific claim verification remains challenging, and the task itself is far from solved.

Figure 13. Reported performance gains from exploiting pair information. Not all teams reported these gains, and some reported results only on the development (dev) split. SciTrue reported multiple gains depending on the underlying LLM.

For example, SciTrue reported that the proprietary frontier models Claude’s Fable 5 achieved only 74.4% pair accuracy and OpenAI’s GPT-5.6 Sol only 70.2% pair accuracy on the development set. In fact, all teams that exploited the pair structure and reported results without using pair information observed substantial drops in pair accuracy, ranging from 8.7% (ProvideQ on the test set) to 25.8% (SciTrue’s evaluation with GPT-5.6 Sol on the development set).

These findings suggest that the high performance achieved by some teams was not necessarily due to stronger verification capabilities, but rather to access to pair information that would typically be unavailable in real-world scenarios. Consequently, these results provide a strong motivation for future iterations of SciClaimEval that remove pair information.

Conclusion

This paper presented an overview of SciClaimEval, a pilot shared task at NTCIR-19 on cross-modal scientific claim verification using evidence from tables and figures. We introduced a new benchmark dataset covering multiple scientific domains, with contrastive evidence pairs, and summarized the participation and system performance across two subtasks. The results demonstrate the strong potential of MLLMs for this task, particularly when combined with carefully designed inference pipelines and effective use of contrastive evidence pairs. However, the substantial performance gap between systems using contrastive pairs and those evaluating individual claim–evidence pairs highlights the remaining challenges in practical scientific claim verification. We hope that SciClaimEval will serve as a valuable benchmark for advancing research on reliable multimodal scientific claim verification.

Discussion & Future Directions

Overall, we consider SciClaimEval a success. This initial edition attracted significant community interest and the highest number of participants at NTCIR-19. However, it also revealed shortcomings in the dataset design that allowed teams to exploit information, making the task substantially easier than intended. Although the resulting high scores may raise questions about the need for further scientific claim verification tasks, the self-reported performance drops without pair information demonstrate that real-world scientific claim verification remains challenging.

Most claim–evidence mismatches in SciClaimEval are relatively easy for humans to identify, even without paired evidence, and their prevalence in actual scientific writing remains unclear. In practice, more challenging cases arise in three scenarios. First, verification may require domain expertise, such as interpreting equations, medical scans, or crystal structures. Second, evidence may be difficult to access or scattered across lengthy documents. For example, verifying claims against cited works often requires retrieving and examining entire papers. Third, claim–evidence relationships are rarely strictly binary and may require more nuanced judgments than our current task allows.

Future iterations of SciClaimEval will address these limitations. First, we will eliminate access to pair information at test time to prevent the shortcuts observed in this edition. Second, we will expand the dataset beyond AI, NLP, and medicine to domains such as materials science and physics, incorporating additional evidence types, including equations, medical scans, and crystal structure visualizations. Third, we will introduce cited papers as evidence sources, addressing the time-consuming and error-prone process of verifying citations, particularly relevant with the rise of AI-assisted scientific writing. Finally, we plan to move beyond binary classification by introducing more nuanced labels or reformulating the task, reducing reliance on shortcuts and enabling a more comprehensive evaluation of scientific reasoning.

Acknowledgments

This work was supported by JSPS KAKENHI Grant Number 24K03231, the French National Research Agency (ANR) through the FAB-ULEUX project (ANR-26-CE23-6589), and by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 554559555. F. Boudin was also supported by a visiting professor position at the National Institute of Informatics (NII), Japan.

References

References

  1. [1]Deyan Ginev. 2024. ar5iv:04.2024 dataset, an HTML5 conversion of arXiv.org. https://s2.arxiv.org/html/2304.2024/ar5iv-dataset-2024/ SIGMathLing – Special Interest Group on Math Linguistics.
  2. [2]Xanh Ho, Yun-An Wu, Sunish Kumar, Tian Cheng Xia, Florian Boudin, Andre Greiner-Petter, and Akiko Aizawa. 2026. SciClaimEval: Cross-modal Claim Verification in Scientific Papers. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), Stelios Pipersidis, Núria Bel, Henk van den Heuvel, Nancy Ide, Simon Krek, and Antonio Toral (Eds.). European Language Resources Association (ELRA), Palma, Mallorca, Spain, 11060–11071. doi:10.63317/4ap9g2gnwmf
  3. [3]Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A Dataset for Many-Hop Fact Extraction and Claim Verification. In Findings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 3441–3460. doi:10.18653/v1/2020.findings-emnlp.309DOI
  4. [4]Yash Kumar Lal, Manikanta Bandham, Mohammad Saqib Hasan, Apoorva Kashi, Mahnaz Koupaee, and Niranjan Balasubramanian. 2025. MusiClaims: Multi-modal Scientific Claim Verification. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, Kentaro Inui, Sakriani Sakti, Haofen Wang, Derek F. Wong, Pushpak Bhattacharyya, Biplab Banerjee, Asif Ekbal, Tanmoy Chakraborty, and Dhirendra Pratap Singh (Eds.). The Asian Federation of Natural Language Processing and The Association for Computational Linguistics, Mumbai, India, 3285–3307. doi:10.18653/v1/2025.ijcnlp-long.175
  5. [5]Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min-Yen Kan. 2023. SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 7787–7813. doi:10.18653/v1/2023.emnlp-main.483DOI
  6. [6]Sebastian Schellhammer, Salim Hafid, Yavuz Selim Kartal, Katarina Boland, Dimitar Dimitrov, Konstantin Todorov, and Stefan Dietze. 2026. Overview of the CLEF-2026 CheckThat! lab task 1 on source retrieval for scientific web claims. Salido et al.[53] (2026).
  7. [7]James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Marilyn Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, New Orleans, Louisiana, 809–819. doi:10.18653/v1/N18-1074
  8. [8]David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannehan Hajishirzi. 2020. Fact or Fiction: Verifying Scientific Claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 7534–7550. doi:10.18653/v1/2020.emnlp-main.609DOI
  9. [9]David Wadden and Kyle Lo. 2021. Overview and Insights from the SCIVER shared task on Scientific Claim Verification. In Proceedings of the Second Workshop on Scholarly Document Processing, Iz Beltagy, Arman Cohan, Guy Feigenblat, Dayne Freitag, Tirthankar Ghosal, Keith Hall, Drahomira Herrmannova, Petr Knoth, Kyle Lo, Philipp Mayr, Robert M. Patton, Michal Shmueli-Schuer, Anita de Waard, Kuansan Wang, and Lucy Lu Wang (Eds.). Association for Computational Linguistics, Online, 124–129. https://aclanthology.org/2021.sdp-1.16/arxiv.org/abs/2107.08188
  10. [10]Chengye Wang, Yifei Shen, Zexi Kuang, Arman Cohan, and Yilun Zhao. 2025. SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 8562–8579. doi:10.18653/v1/2025.acl-long.420DOI
  11. [11]Nancy X. R. Wang, Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. 2021. SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS). In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, and Xiaodan Zhu (Eds.). Association for Computational Linguistics, Online, 317–326. doi:10.18653/v1/2021.semeval-1.39DOI
  12. [12]Bohao Yang, Yingji Zhang, Dong Liu, André Freitas, and Chenghua Lin. 2025. Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning. arXiv:2501.13042 [cs.CL] https://arxiv.org/abs/2501.13042
  13. [13]Yuji Zhang, Qingyun Wang, Cheng Qian, Jianteng Liu, Chenkai Sun, Denghui Zhang, Tarek Abdelzaher, Chengxiang Zhai, Preslav Nakov, and Heng Ji. 2025. Atomic Reasoning for Scientific Table Claim Verification. arXiv:2506.06972 [cs.CL] https://arxiv.org/abs/2506.06972

Paper details

vlnc:W_epktrauz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents