betaFeedback

WorksComputer Science

Rethinking the Evaluation of LLM Fact-Checkers

Abstract

Large language models (LLMs) are widely used to answer factual questions, but their answers can contain errors and must be fact-checked claim by claim. Yet the resulting score depends on three design decisions: which questions the claims answer, under what conditions each claim is judged, and which sources count as evidence. No existing benchmark specifies all three. We introduce JSFactCheckBench, a Japanese benchmark that specifies all three, with 1,913 claims from LLM answers to questions about misleading information that circulated on Japanese social media, each annotated for factuality at a reference time and for importance to the question, using evidence from preselected Web sources. Across six LLMs, all verify claims directly answering the question less accurately than supplementary claims, a gap that aggregate scores obscure. Scores also drop when models must locate the relevant content in longer or multiple sources and, for all four open models, when the surrounding question and response are added. These weaknesses are visible only because the benchmark records the conditions behind each label. A benchmark that leaves them implicit cannot reveal them.

Introduction

Large language models (LLMs) are increasingly used as knowledge sources to answer factual questions [15, 12]. However, flawed training data and missing or outdated knowledge can lead them to produce false or misleading information [9, 8]. Automated fact-checking of LLM responses is therefore important. Because a response may contain several claims, fact-checking means verifying each claim it contains, and evaluating such systems requires reliable benchmarks.

Three design decisions determine what a benchmark measures, and if any is left uncontrolled, the labels depend on it implicitly. (1) Which claims to verify. A benchmark’s questions should come from the real world, since questions constructed for evaluation may not reflect what arises in practice. Its scores then describe performance for the population those questions come from. (2) Under what conditions to judge them. The truth of many claims changes, so labels should be tied to a reference time. Without one, a label holds as of whenever the annotator happened to search, a date the verifier is never told. An error in a claim directly answering the question also matters more than one in a peripheral detail, so labels should be tied to the claim’s importance. Without it, a wrong answer to the question weighs the same as a wrong aside. (3) Which sources to accept as evidence. Web pages vary in reliability and may fact-check the claim, leaking the verdict, so sources should be controlled. Otherwise the evidence is whatever page the annotator found, reliable or not. No existing benchmark controls all three (Table 1).

We introduce JSFactCheckBench, a Japanese fact-checking benchmark designed to make all three decisions explicit (Figure 1). For (1), we draw on JSocialFact Nakazato et al., [14], whose questions derive from misleading information that actually circulated on Japanese social media, and extract 1,913 claims from the answers of eleven LLMs to them. For (2), each claim is annotated for factuality at a specified reference time and for importance to the question. For (3), factuality is grounded in evidence from preselected Web sources, and three professional annotators independently judge each claim.

Decision (1) is settled once the data are collected. The other two remain open at evaluation time, and the benchmark shows what each changes. For (2), only a quarter of claims directly answer the question, and all six LLMs we evaluate verify those claims less accurately than the supplementary ones, a gap that aggregate scores hide where it matters most. Answers to 46% of the questions can also change over time, and stating the reference time shifts the scores of some models but not others. For (3), because every candidate page is recorded, we can ask models to select the source rather than read a given passage, and the spread between models triples when they must. None of this is visible from a single score under a single condition, and none of it can be measured on a benchmark that leaves these decisions uncontrolled.

JSFactCheckBench

Dataset Design

JSFactCheckBench evaluates LLM answers to factual questions. Because a response may contain several claims differing in factuality and importance to the question, the unit of evaluation is an individual claim rather than the response as a whole.

The benchmark follows three design principles corresponding to the decisions introduced in § 1: (1) Claims come from LLM answers to questions of real-world origin. (2) Factuality is judged at a stated reference time, and each claim is labeled for its importance to the question. (3) Web evidence is collected under a controlled procedure. Figure 1 gives an overview of the construction process that implements these principles, and Table 2 shows examples from our dataset.

Overview of the dataset construction pipeline

Figure 1. Overview of the dataset construction pipeline.

Question 1Is it free to call an ambulance in Japan?
ResponseCalling an ambulance in Japan normally incurs a fee. However, · · · to request an ambulance, call 119. · · ·
ClaimCalling an ambulance in Japan normally incurs a fee. ✗ Refuted Main
Source IDd61417498a, Time: 2026-02-15
Evidence spanIn Japan, ambulance users currently bear no charge and can use the service free of cost. · · ·
Question 2Will the ¥1.1 million gift tax exemption be abolished starting in 2024?
ResponseNo decision to abolish the annual gift tax exclusion of ¥1.1 million in 2024 has been announced. The tax-free threshold for gift tax is set under Japanese tax legislation. · · ·
ClaimNo decision to abolish the ¥1.1 million gift tax exemption in 2024 has been announced. ✓ Supported Main
Source IDb3655be520, Time: 2026-02-15
Evidence span· · · it was decided that the annual gift tax system would remain in place. · · ·

Table 2. Example instances from JSFactCheckBench. The Japanese text is translated into English.

(1) LLM claims to real questions. To evaluate the errors that LLMs actually make, a benchmark should draw its claims from LLM responses to questions of real-world origin, rather than from claims constructed for evaluation. The benchmark is built for users in Japan, so we take questions about misleading information that actually circulated on Japanese social media and extract claims from LLM answers to them.

(2) Labels conditioned on time and the question. To make factuality judgments well-defined and distinguish errors by their consequence for the answer, we condition each claim on two aspects.

(2)-A. Factuality at a specified time. To assign a well-defined label to claims whose truth may change, we define factuality with respect to a specified reference time. For example, for the claim “Alice is the CEO of Company X” and the reference time “January 2025,” the label depends on whether it was true at that time, not at the time of evaluation. We therefore provide the reference time to the verifier and label each claim ✓ Supported if the evidence supports it at that time, ✗ Refuted if it contradicts it, and ? NotEnoughInfo if it does neither.

(2)-B. Importance to the question. To distinguish errors in the direct answer itself from errors in supplementary content, we label each claim by its role in answering the question. For the question “Is it free to call an ambulance in Japan?”, a claim is Main if it directly answers the question or states the core or part of the answer (“Calling an ambulance is generally free of charge.”), Sub if it provides related information without directly answering the question (“Medical treatment at the receiving hospital may incur a fee.”), and Irrelevant if it has no direct relation to the question or answer (“Fire engines in Japan are red.”).

(3) Controlled Web evidence. To verify open-domain claims while keeping gold labels grounded in accountable evidence, the benchmark must combine broad Web coverage with control over which sources can determine factuality. We therefore restrict evidence candidates to a predefined domain allowlist of accountable sources and exclude two fact-checking-site domains to reduce verdict leakage. For each claim, all annotators inspect the same pre-collected candidate set and record the URL and passage used for their judgment.

Dataset Construction

We build the dataset in three steps: collecting claims from LLM responses, constructing web evidence candidates, and human annotation.

Step 1: Collecting claims.

We randomly sampled 85 of the 385 questions in JSocialFact ([14]), a benchmark built from misleading information that circulated on Japanese social media. For each question, we generated two responses, 170 in total, using two distinct models sampled at random from a pool of eleven (Appendix § 6.§ 1). This enables us to cover a broad range of factual errors.

Following Liu et al. [10], we decompose each response into atomic claims, that is, statements that contain a single proposition and can be verified independently of the rest of the response. We apply their prompt with Meta-Llama-3-8B-Instruct. The procedure (1) splits each sentence into atomic claims; (2) resolves ambiguous references such as pronouns and incomplete entity names; (3) restores omitted temporal, causal, and comparative relations; (4) adds contextual information such as presuppositions; and (5) checks and revises each claim for concrete, objective verifiability. The LLM performs all five steps automatically.

Step 2: Constructing evidence candidates.

Selecting trusted domains. Since evidence determines gold labels, which sources may determine them must be fixed in advance rather than left to each annotator. We therefore admit only sources that an institution outside the benchmark has vetted. This excludes anonymous and unvetted material and makes the choice reproducible across annotators. Because this cannot be judged page by page, we apply it at the domain level and compile an allowlist, fixed before retrieval, from four groups, each defined by a different vetting institution:

  • Core. Domains of government (go.jp), local government (lg.jp), academic institutions (ac.jp), companies (co.jp), and incorporated organizations (or.jp), which the registry issues only to verified organizations.

  • Media. Domains of 73 organizations listed by the Japan Newspaper Publishers & Editors Association, including newspapers, broadcasters, advertising companies, and publishers,2 which are bound by its editorial standards.

  • Companies. Domains of 1,029 companies listed on the Tokyo Stock Exchange, collected from Wikipedia. These add official corporate sites outside co.jp, whose statements about their own products, services, and announcements are subject to listing rules.

  • Wikipedia references. 322 domains that appeared at least twice as references on Japanese Wikipedia pages about politics, incidents, disasters, and elections. These are sources that Wikipedia’s sourcing policy, enforced by its editors, has repeatedly admitted, and they add what the other groups miss.

In compiling these groups, we excluded two fact-checking sites, whose articles may state an existing verdict for a target claim. We also excluded domains listed on Wikipedia pages for blogs and microblogging services, as well as note.com, judging these sources unsuitable as benchmark evidence because author identity and editorial responsibility are not consistently established. The resulting allowlist contains 1,397 domains and is included in the released dataset. A domain belonging to multiple groups is counted under the highest-priority group, in the order listed above; the same rule applies to the per-category counts in Table 12. The allowlist is specific to Japan because the benchmark targets users there. The procedure itself is not. For another target, one compiles an allowlist of accountable sources for that region similarly.

Domain categoryCandidatesSelected
Core7,931539
go.jp2,408205
co.jp2,758172
or.jp1,16075
lg.jp94954
ac.jp65633
Media61048
Companies2948
Wikipedia ref.1,17672
Total10,011667

Table 12. Domain categories of candidate and selected evidence set. Numbers indicate the counts of unique links.

Searching evidence candidates. For each context-completed claim from § 1.2, an LLM generated a keyword query meeting three requirements: (1) it should retrieve information to verify concrete facts in the claim, including dates, quantities, named entities, and events; (2) it should contain expressions likely to occur in government releases, news articles, or official statistics; and (3) it should include enough context to distinguish the target from similarly named entities or events. The prompt appears in Appendix § 6.§ 2. We submitted each query to the Custom Search JSON API,3 set the search region to Japan, and retrieved the top 30 results. We retained only results from allowlisted domains, excluded .pdf URLs, and selected up to 10 top-ranked evidence candidates per claim.

Step 3: Human Annotation.

Annotators assign the factuality and importance labels defined in (2)-A and (2)-B of § 1.1.1. Claims that are not truth-evaluable, such as metalinguistic statements or subjective opinions, receive the auxiliary label Unevaluable; malformed or uninterpretable claims receive Unclear. Full guidelines are in Appendix § 6.§ 3.

Three annotators used Label Studio,4 to read the question and full response, inspect the evidence candidates, and label each target claim. For Refuted claims, annotators additionally marked the erroneous span; for all other labels, the annotation covered the full claim. Annotators also recorded the evidence URL and the passage used for the factuality judgment. We then consolidated the three annotations into one gold factuality label and one gold importance label per claim. For factuality, a Supported or Refuted judgment takes precedence over NotEnoughInfo because it is accompanied by evidence supporting or contradicting the claim. When annotations contain both NotEnoughInfo and either Supported or Refuted, we retain the latter; when Supported and Refuted conflict, we use majority vote. Unclear and Unevaluable are retained only when all three annotators assign them; otherwise, they are ignored. Importance labels are consolidated by majority vote. This automatic consolidation procedure resolved 98% of factuality labels and 97% of importance labels. For the remaining claims, two additional annotators discussed the case and assigned the final label. Because claims in LLM answers have no existing verdicts, each was verified from scratch, at about USD 7.50 per claim; Appendix A.5 reports the cost.

To validate the labels, two additional annotators independently relabeled 50 sampled claims, given the claim and the passage used in the original judgment; all three label sets agreed on 94% of the claims for factuality and 96% for importance.

Dataset Statistics

JSFactCheckBench contains 1,913 claims from 170 responses to 85 questions, with 11.3 claims per response on average (range 1–24; Figure 4). These responses broadly cover the outputs of the eleven models, as shown in Appendix A.1.

Bar chart showing the number of claims per response

Figure 4. Number of claims per response.

Label distribution. Table 3 shows the label distribution of factuality and importance. Among claims that received either Supported or Refuted, 21% are Refuted, indicating that roughly one in five verifiable claims in these responses is incorrect. On the importance axis, Sub claims account for 65.3% of the data and Main claims for 25.2%. Most of an LLM answer therefore consists of supplementary material rather than statements that directly answer the question, which is precisely the situation in which an unweighted claim-level error rate becomes difficult to interpret.

MainSubIrrel.All
Supported25062554929 (48.6%)
Refuted701728250 (13.1%)
NotEnoughInfo14034574559 (29.2%)
Unevaluable2310844175 (9.1%)
All483 (25.2%)1,250 (65.3%)180 (9.4%)1,913

Table 3. Label distribution of factuality and importance. Percentages are relative to all 1,913 claims; the 20 claims labeled Unclear are excluded from the dataset.

Question attributes. We classified the 85 questions along four dimensions: topic domain, geographic scope, question stance, and temporal stability. Tables 14–17 define the taxonomy.

Domain. The questions span 11 domains (Table 4). Medicine and health is the largest at 28.2%, followed by nuclear energy and radiation and economy, industry, and business (14.1% each). This concentration likely reflects the topics dominating misleading information on Japanese social media.

Topic domainNTopic domainN
Medicine and health24Science and technology4
Nuclear energy12Justice and law4
Economy and industry12Society and gender4
Culture and history8Politics3
Government services7International affairs2
Media and broadcasting5

Table 4. Distribution of question domains.

Geography. Of the questions, 63.5% concern a specific country, and 47.1% overall concern Japan (Table 5). Questions about Japanese institutions and events may require domestic evidence such as government publications and news reports. JSFactCheckBench restricts its questions by origin (Japanese-language claims), not by subject matter, so foreign and non-geographic topics are included.

Geographic scopeN%
Japan4047.1
Universal/non-geographic2428.2
Foreign country1416.5
International/global78.2

Table 5. Distribution of geographic scope.

Stance. Most questions are posed neutrally (83.5%; e.g., “Did Azerbaijan originate from Turkey?”), while 10.6% attribute the proposition to another source (e.g., “Is it true that the vegan meal in JAL business class was only one banana?”) and 5.9% presuppose it and invite agreement (e.g., “Isn’t it strange to say that ordinary water molecules and tritiated water molecules are the same?”).

Temporal stability. Temporally dynamic questions account for 46%. Their answers change with company policy (“Does Apple Vision Pro have an F1 app?”) or regulatory action (“Has the Consumer Affairs Agency recognized the effectiveness of Cleverin?”), unlike static scientific questions (“*Does drinking baking soda cure cancer?”).

Evidence distribution. Each claim has 9.42 evidence candidates on average, and only 2 claims (0.1%) have no candidate from an allowlisted domain. ? NotEnoughInfo labels therefore rarely arise from empty retrieval; in almost all cases annotators had candidates to inspect but could not establish the claim’s truth or falsity from them. Because the same page can appear as a candidate for multiple claims, the candidate sets comprise 10,011 unique pages, of which annotators selected 667 unique pages (6.7%) as evidence for their judgments. Government sources are selected disproportionately often: within the Core group, go.jp accounts for 30.4% of candidates but 38.0% of selected links, whereas co.jp supplies a comparable share of candidates (34.8%) and a smaller share of selections (31.9%). The most frequently selected domain was the Ministry of Health, Labour and Welfare (mhlw.go.jp; 23 pages), followed by the Asahi Shimbun (asahi.com; 20) and the Ministry of Foreign Affairs (mofa.go.jp; 15). Table 12 describes the distribution of domain categories.

Comparison with Existing Benchmarks

Table 1 compares dataset construction along the three design decisions. JSFactCheckBench combines all six properties defined in the caption.

BenchmarkLang.LLM responsesReal-world originReference timeHuman importanceOpen webSource control
FEVER (2018)En–––––✓
TruthfulQA (2021)En△–––✓–
CHEF (2022)Zh–✓△–✓△
FactDrill (2022)Multi–✓△–✓–
FActScore (2023)En✓––––✓
AVeriTeC (2023)En–✓△–✓–
FELM (2023)En✓△––✓–
JSocialFact (2024)Ja–✓––––
Factcheck-Bench (2023)En✓△–△✓–
Mu-SHROOM (2024)Multi✓––––✓
VITALERRORS (2025)En△△–△–✓
Yao et al. [25]En✓–△–✓–
VeriTaS (2024)Multi–✓△–––
JAD-AFC (2027)Ja–✓△–✓–
JSFactCheckBenchJa✓✓✓✓✓✓

Table 1. Comparison of construction. ✓: satisfied; △: partially satisfied; –: not satisfied. LLM responses: factuality or hallucination annotations of LLM-generated responses at claim, segment, or span level (△: answer-level labels or perturbation conditions only). Real-world origin: provenance of the source questions or claims (△: mixed provenance). VeriTaS and JAD-AFC include truth-altering claim variants. Reference time: a recorded reference time explicitly anchoring gold factuality judgments (△: temporal metadata or constraints only). Human importance: manually annotated claim importance that is incorporated into the benchmark’s evaluation (△: annotated but not used in evaluation, or automatically assigned). Open web: Web-derived evidence or reference sources not restricted to a fixed reference corpus. Source control: predefined restrictions on eligible evidence sources (✓: fixed reference sources or an allowlist; △: blocklist-based restrictions). AVeriTeC excludes only the originating fact-checking article and merely flags unreliable sites.

Claim collection. FEVER and TruthfulQA use annotator-written claims or reference answers, while FactDrill, CHEF, AVeriTeC, VeriTaS, and JAD-AFC draw claims from fact-checking articles rather than LLM answers. Among LLM-answer datasets, FActScore focuses on biographies, Mu-SHROOM constructs questions from Wikipedia, and FELM, Factcheck-Bench, and VITALERRORS mix prompt sources. Yao et al. [25] extract claims from news reports generated from deliberately inconsistent scenarios rather than real-world questions. The other Japanese resources do not release claim-level labels on LLM answers; our claims come from LLM answers to JSocialFact questions derived from misleading information in Japan.

Judgment conditions. None of the compared datasets records a reference time that anchors gold labels. AVeriTeC’s evidence cutoff and VeriTaS’s dated, quarterly claims use the date a claim was made to keep later information out, CHEF writes missing times into the claim, and Yao et al. [25] give the verifier a fixed date without recording one for the human labels. We instead fix when the label must hold: the evidence-collection date, unless the claim specifies a time, with only evidence available by then, and we give that date to the verifier.

Experiments

We use JSFactCheckBench to investigate how evaluation settings affect the measured fact-checking performance of LLMs. Specifically, we examine the effects of contextual information, evidence presentation, and claim importance.

Task and Setup

The task is factuality classification: given a claim and the information available under a particular setting, the model predicts Supported, Refuted, or NotEnoughInfo. Claims labelled Unevaluable or Unclear are excluded, leaving 1,738 claims. We evaluate four open Japanese models in the 8B–12B range, Llama-3.1-Swallow and Qwen3-Swallow ([4]), LLM-jp-4 ([11]), and Mistral-Nemo-Ja ([7]), and two commercial models, GPT-5.1 ([19]) and Sonnet 5 ([1]) (checkpoints in Appendix § 7.§ 1).

Input Settings

We evaluate a claim by crossing three context levels, what surrounds the claim, with four evidence conditions. This gives twelve settings; prompts and full details are in Appendix § 7.§ 1.1.

Context. At the narrowest level, the model receives the target claim only. The next level adds the reference time at the head of the prompt: the date on which that claim’s evidence was collected. The widest level further adds the source question and the full response, so that the claim appears in the context a deployed fact-checker would encounter it in. We refer to these as the Claim, Time, and Q&R settings. Comparing the first two isolates the effect of specifying a reference time; comparing the second and third isolates the effect of retaining the surrounding question and response.

Evidence. The evidence conditions correspond to three situations a fact-checking system can find itself in, differing in how much of the retrieval pipeline has already been carried out.

(a) Closed book. No external text is supplied and the model must rely on internal knowledge alone. This is the situation of a system asked to judge a claim without search access, and it measures what the model already knows.

(b) Source given. The page the claim was checked against is supplied, so the model is spared retrieval and left with reading and judgment. Two conditions differ in how much reading is required. Reference supplies only the passage annotators quoted from that page. Full supplies the page’s body text containing the Reference, truncated to a 6,000-character window centred on it. The model must therefore read a passage of page-representative length and identify the part of it that bears on the claim.

(c) Search-like. Pages supplies the body text of every candidate page shown to annotators (up to ten), each reduced to a 600-character window containing the paragraph nearest the search snippet, so the model must decide which candidates to rely on. For the page the annotators cited, the window is instead centred on their quoted passage, so (c) contains the evidence available in (b), and the comparison between them isolates the cost of selection. It is the closest of our three scenarios to a retrieval-augmented system, though the retrieval itself is fixed in advance.

Results

Table 6 reports macro-F1 scores in the factuality classification task. Significance is assessed with a paired bootstrap over questions (10,000 resamples, p<0.05p < 0.05). Scenario (a) measures what a model can do from internal knowledge alone, and it provides the reference point for everything that follows. Given the claim by itself, the open models score between 0.444 and 0.490 and the commercial models between 0.549 and 0.600. Internal knowledge carries these models past a uniform-random baseline (0.313), but the absolute scores remain low: judging factuality from parametric knowledge alone remains difficult. We therefore vary the input setting in what follows, in order to examine the models’ fact-checking capabilities more closely.

Input given to the modelMacro-F1
ClaimTimeQ & REvidenceLlama-3.1-Sw.Qwen3-Sw.LLM-jp-4Mistral-Nemo-JaGPT-5.1Sonnet 5
* (a) Closed book: no evidence supplied
✓✗✗None0.4440.4630.4850.4900.5490.600
✓✓✗None0.4180.4370.4740.4890.5570.593
✓✓✓None0.2650.3270.3070.2560.5440.586
* (b) Source given: the cited page is supplied
✓✗✗Ref.0.6910.6950.7520.7240.7700.753
✓✓✗Ref.0.6840.6660.7410.7450.7760.761
✓✓✓Ref.0.3500.3520.3700.3340.7140.750
✓✗✗Full0.5690.6700.6700.6290.7010.718
✓✓✗Full0.5820.6540.6750.6640.7160.726
✓✓✓Full0.3440.3920.4220.3400.7070.714
* (c) Search-like: the source must be selected
✓✗✗Pages0.4580.6220.6240.5810.7000.705
✓✓✗Pages0.5210.6220.6180.5840.7050.707
✓✓✓Pages0.3530.3860.3890.3240.6900.691

Table 6. Factuality classification results. See § 2.2 for the input settings. Best score per row in bold.

Surrounding question and response. Adding the question and response has comparatively little effect on the two commercial models but substan tially reduces performance for all four open models across every evidence condition. Inspection of the predictions shows that this degradation is accompanied by a strong bias toward the Supported label. For example, with Reference evidence, adding the question and response raises the proportion of claims that LLM-jp-4 labels Supported from 54% to 93%, even though the prompt explicitly instructs the model to judge only the target claim and to use the surrounding context solely for resolving references. This suggests that, rather than judging the target statement in isolation, the open models tend to treat the surrounding answer as evidence that the target statement is factual.

Form of the evidence. How evidence is presented changes not only the scores but also how far apart the models appear. When the passage that decides the label is given (Reference), the six models fall within 0.08 macro-F1 of one another (0.691–0.770); when the same passage must be found among candidate pages (Pages), the range triples to 0.25 (0.458–0.705). The loss comes in two steps. Reading a page instead of a passage (Reference to Full) costs almost every model (0.025–0.122, significant for all but Qwen3-Swallow). Choosing among pages (Full to Pages) costs four of the six (0.046–0.111, significant), whereas GPT-5.1 and Sonnet 5 lose almost nothing (0.001–0.013, not significant). A benchmark that supplies only the deciding passage would therefore rate these models as similar and miss the ability that separates them: selecting the source. JSFactCheckBench can separate the two steps because it records both the deciding passage and the full candidate set for every claim. Evidence still helps in every form: Pages outperforms the closed-book condition for all six models once the reference time is stated.

Reference time. The answers to 46% of the questions can change over time (§ 1.3), so their gold labels are defined only as of a date. We therefore state in the prompt the date on which each claim’s evidence was collected. Doing so changes macro-F1 by up to 0.063, but the effect is model-specific. The change is significant for three of the six models in at least one evidence condition (Qwen3-Swallow, Mistral-Nemo-Ja, and Llama-3.1-Swallow) and is not significant in any condition for the other three. Models thus differ in how they respond to a stated reference time. One possible explanation is that the three affected models hold their internal knowledge in association with time

ModelAcc (all)MainSubIrrel.
Llama-3.1-Sw.0.6480.5630.6870.615
Qwen3-Sw.0.7410.6970.7540.785
LLM-jp-40.7410.7100.7460.807
Mistral-Nemo-Ja0.6990.6480.7250.644
GPT-5.10.7410.7070.7470.807
Sonnet 50.7460.6700.7700.800

Table 7. Accuracy by claim importance (Claim + Evidence Full). Acc (all) is accuracy over all claims; the importance columns are accuracy within each group. We report accuracy rather than the macro-F1 of the main results because the Irrelevant group contains only 8 Refuted claims, too few for a macro-averaged score.

to some extent and adjust their judgments to the given date, whereas the others judge from the same knowledge regardless of the date. JSFactCheckBench can reveal these differences because it ties every label to a recorded reference time.

Importance. Table 7 breaks accuracy down by the annotated importance of each claim. Sub claims outnumber Main claims by about 2.5 to 1 among the evaluated claims (1,142 vs. 460), so aggregate accuracy mostly reflects Sub claims. Yet every model verifies Main claims less accurately than Sub claims, by 0.036 to 0.124; the gap is significant for three of the six (Llama-3.1-Swallow, Mistral-Nemo-Ja, and Sonnet 5). The two groups have nearly identical label distributions, so the gap is not an artifact of label imbalance. An aggregate score can therefore hide errors on the claims that directly answer the question: among the four best models, no difference in aggregate accuracy is significant (all are within 0.005 of each other), yet Sonnet 5, which has the highest aggregate accuracy, has the lowest accuracy on Main claims among the four (0.670 vs. 0.697–0.710), a significant difference from GPT-5.1. Claim importance is thus needed to read these results correctly.

Conclusion

We introduced JSFactCheckBench, a Japanese benchmark of 1,913 claims from LLM answers that makes explicit three design decisions behind a fact-checking score: which questions the claims answer, under what conditions they are judged, and which sources count as evidence. Experiments with six LLMs show that each condition exposes a different weakness: accuracy is lower on the claims that directly answer the question, models diverge once evidence must be found rather than given, some collapse when a claim is judged within its answer, and only some respond to a stated reference time. No single score under a single condition captures fact-checking ability; it must be evaluated from multiple aspects.

Limitations

Evidence source constraints. Source quality is controlled at the domain level, so the benchmark does not model quality differences among pages within the same domain. The allowlist is built on institutions specific to Japan, including organizational JP domains and membership in the Japan Newspaper Publishers & Editors Association. Applying the design to another language requires locally grounded question sources and evidence-selection criteria.

Fixed reference time. Because each label is explicitly tied to a reference time, the benchmark avoids the ill-defined gold labels discussed in sub-section 2.1; the cost is that the reference time is fixed to the evidence retrieval date, so the benchmark cannot reproduce judgments for an arbitrary past or future date. For rapidly changing facts, such as statistics and company policies, the gold label may diverge from the current state of the world as time passes. Continued use will require periodic updates to both evidence and labels.

Scale. The benchmark covers 85 questions and 1,913 claims in Japanese, fewer than benchmarks such as FEVER or AVeriTeC. Unlike datasets whose claims are sourced from existing fact-checking articles, our claims are extracted from LLM responses and therefore lack pre-existing claim-specific verdicts and evidence trails. No such verdicts exist for claims in LLM answers, so each of ours was verified from scratch against Web evidence by three annotators. Benchmarks built this way are of the same order or smaller (Factcheck-Bench: 678 claims; world-knowledge subset of FELM: 532 segments). Even so, small differences between models should not be read as rankings, so our experiments also report statistical significance, showing that the dataset is large enough to yield reliable results.

Ethics Statement

Data provenance and license. The questions are drawn from the publicly available JSocialFact benchmark [14] and are included with the permission of its authors. JSFactCheck-Bench is released under custom terms of use: it may be used, including commercially, for the purpose of improving the safety of LLMs; redistribution of the dataset itself is not allowed, while non-overlapping derivative data may be distributed with attribution. The release contains URLs and the passages annotators quoted, not full page contents; these passages were collected and included for data analysis under Article 30-4 of the Copyright Act of Japan, and copyright remains with the respective rights holders. The scope of the release was determined in consultation with a lawyer.

Annotation work. Annotation was outsourced to a Japanese professional provider and completed by three workers. The total contract cost was approximately JPY 2.3 million, corresponding to about JPY 390 for each of the 5,856 individual annotations.

Intended use and misuse. The dataset is intended to evaluate methods for fact-checking LLM responses. Because many responses contain false statements, the dataset must not be used as a source of factual knowledge. Its labels are tied to evidence collected at a particular time and do not guarantee that the same claim remains true or false at present. The questions also cover consequential topics such as medicine and nuclear energy; individual labels should not be used as the sole basis for real-world decisions.

Dataset Construction Details

Models Used for Response Generation

Table 8 lists the models used to generate the responses together with their IDs, and Figure 2 shows how many responses each model contributed.

IDModel
aopenai/gpt-35-turbo-1106
bllm-jp/llm-jp-13b-instruct-full-ac_001_16 x-dolly-ichikara_004_001_single-oasst-oasst2-v2.0
ccyberagent/calm3-22b-chat
danthropic/anthropic.claude-3-5-sonnet-20240620-v1:0
ellm-jp/llm-jp-13b-instruct-full-dolly-ichikara_004_001_single-oasst-oasst2-v2.0
ftokyotech-llm/Llama-3-Swallow-70B-Instruct-v0.1
gmeta-llama/Meta-Llama-3-8B-Instruct
hopenai/gpt-4-0613
ielyza/Llama-3-ELYZA-JP-8B
jtokyotech-llm/Llama-3-Swallow-8B-Instruct-v0.1
lgoogle/gemma-2-27b-it

Table 8. Models used to generate the answers.

Distribution of models used for response generation and the number of responses in the dataset

Figure 2. Distribution of models used for response generation and the number of responses in the dataset.

Evidence-Link Selection

Pilot study. Before full annotation, we ran a pilot in which three annotators verified 219 LLM responses, some of which overlap with the 170 in the final dataset. The sources they needed that the Core and Media groups did not cover led us to add the Companies and Wikipedia-reference groups.

We excluded URLs ending in .pdf from retrieval because documents spanning dozens of pages would impose excessive annotation cost.

Query Generation

We generated one search query per context-completed claim using Meta-Llama-3-8B-Instruct with temperature 0.3 and a 50-token limit. The system prompt states the three requirements given in § 1.2 and is followed by four fixed input–output examples supplied as user and assistant turns; the claim to be processed forms the final user turn. Figure 3 shows the prompt. The examples pair a claim with a space-separated keyword string rather than a question, which is the form the model follows in practice.

Prompt for search-query generation

Figure 3. Prompt for search-query generation. The Japanese original is translated into English; slashes mark the spaces separating keywords.

Annotation Guidelines

This section summarizes the guidelines given to the annotators. The unit of work is a single claim: annotators read the source question and the full response, inspect the evidence candidates for the claim, and assign one factuality label and one importance label.

Materials and workflow. Annotators received (i) the question and its two responses, already segmented into claims, (ii) up to ten evidence candidates per claim, distributed as a spreadsheet keyed by question ID, and (iii) the label set. Annotation was carried out in Label Studio, and the two resources were linked by question ID. For each claim, annotators were asked to record the factuality and importance labels, the single evidence link used for the judgment, the passage quoted from that link, and the time of annotation. Annotators inspected the candidates in the given order and stopped once the judgment could be made; claims with no assigned candidates were skipped.

Factuality labels. A claim is Supported if the selected evidence shows it to be correct, Refuted if the evidence shows it to be incorrect, and NotEnoughInfo if it asserts a fact but the candidate set does not settle whether that fact holds (Ohtani was born at 21:03 on July 5, 1994; according to sources close to the matter, there was a non-public agreement). A fourth label, Unevaluable, applies to claims that are not truth-evaluable at all; its subtypes are given below. Under NotEnoughInfo and Unevaluable, no evidence link is recorded, since no passage supports or contradicts the claim. Claims that cannot be read as a proposition because of segmentation or grammatical errors receive Unclear; annotators were asked to resolve omitted subjects and other ellipses from the question and response before resorting to this label.

Importance labels. Importance is defined relative to the question, not to the response. Table 9 gives the criteria and the decisive test for each label.

LabelDefinitionDecision criterion and examples (Q: What is the capital of Japan?)
MainA claim that directly answers the question, or that states the core or a part of the answer. It need not be a complete or correct answer, only a direct response to the question.Shown on its own, the claim stands as an answer to the question, whether full or partial. It is Tokyo. / I do not know.
SubA claim that supplies information related to the subject of the question or to the answer, but does not itself contain the specific value or name being asked for.On its own it does not answer the question; it mentions the topic of the question but not the value asked for. It became the capital in 1869.
IrrelevantA claim with no direct relation to the question or to the content of the answer. It contributes neither to the correctness nor to the understanding of the answer.The claim is unrelated to the topic or purpose of the question, and fits neither Main nor Sub. Osaka is famous for its food culture.

Table 9. Criteria for the importance axis. Examples are translated from the Japanese guidelines.

Subtypes of Unevaluable. A claim is assigned Unevaluable when it has the surface form of an System

Generate a search query for verifying the given claim.

Requirements:

  1. The query should retrieve the verifiable facts in the claim, such as dates, quantities, named entities, and events.

  2. It should use expressions found in trusted sources such as government releases, news articles, and official statistics.

  3. It should include enough context to distinguish the claim from similarly named events.

Examples (four pairs, given as alternating user and assistant turns)

Claim: After the Noto Peninsula earthquake, Ishikawa Prefecture published a request that individuals refrain from travelling to the Noto area.

Query: Noto Peninsula earthquake / Ishikawa Prefecture / non-essential travel / request to refrain

Claim: The 2024 Noto Peninsula earthquake had a magnitude of 7.6.

Query: Reiwa 6 Noto Peninsula earthquake / magnitude

Input

Claim: <TARGET CLAIM>

assertion but cannot be verified in principle. Table 10 lists the four subtypes. The first subtype has an explicit exception: when a concrete figure, definition, or basis of comparison is stated, the claim becomes objectively checkable and is judged on the factuality axis as usual.

SubtypeDefinitionExample
Relative evaluation onlyUses relative terms such as many, few, sufficient, or excessive without stating the standard or the object of comparison, so that the assessment depends on the reader’s values.The Japanese government allocates a large budget to scholarship programs for international students. With a figure supplied (a budget of JPY 22.2 billion), the claim is verifiable and is judged on the factuality axis.
Hypothetical conditionalStates a prospect or a generality conditional on an event, without asserting that the event occurred.If such an incident really did occur, the truth will come to light.
Metalinguistic statementStates the speaker’s own position, capability, constraints, or willingness to respond.I cannot provide detailed information about specific incidents or individuals.
Value or normative statementAsserts what should be done or what is desirable, important, or recommended.The government should be more transparent.

Table 10. Subtypes of the Unevaluable label. Examples are translated from the Japanese guidelines.

Annotation span. For Supported, NotEnoughInfo, Unevaluable, and Unclear, the label is applied to the whole claim. For Refuted, annotators instead marked the minimal erroneous span, defined as the smallest region such that rewriting it would make the entire claim correct. When a claim contained several independent errors, each was marked separately.

Evidence recording. Three constraints applied to the choice of evidence. Only the content of the linked page itself could be used, not the content of pages it links to; pages requiring a login or a payment to read were not used; and when the relevant passage appeared in a table, annotators recorded it in header: cell value form so that the quoted string remains interpretable outside the table. The following case receives special treatment. When the claim is a negative existential and the candidate set contains no statement of the asserted event (for example, the Japan Meteorological Agency has not announced that extreme heat will begin in April 2024), the claim is labelled Supported with no evidence link, and the annotator records in the comment field that the event could not be found anywhere in the candidate set. A free-text comment field was available throughout for borderline cases.

Recording format. Links and passages were entered in a fixed template so that they could be extracted automatically:

[Evidence link used]
URL: ⟨url⟩
Passage: ⟨quoted text⟩
[Link from which the claim could not be decided]
URL: ⟨url⟩

Annotation Cost and Effort

Annotation was outsourced to a Japanese professional annotation provider and carried out by three native speakers of Japanese residing in Japan. Each annotator labeled all 1,952 claims produced by decomposition, giving 5,856 individual annotations. We then removed 19 duplicate claims and the 20 claims labeled Unclear, leaving 1,913. For each claim, an annotator read the question and the full response, inspected up to ten candidate Web pages in the given order, and recorded the factuality and importance labels, the evidence URL, and the quoted passage. Annotation took 294 person-hours in total, about three minutes per annotation and nine minutes per claim across the three annotators. Table 11 summarizes the cost and effort.

JPYUSD
Total cost2,302,30014,632
Cost per annotation3932.50
Cost per claim1,1807.50
Annotators3
Claims1,952
Individual annotations5,856
Candidate pages per claim (mean)9.42
Total annotation time (person-hours)294
Time per annotation (min., mean)3.0

Table 11. Cost and effort of annotation. Cost per claim covers the three independent annotations of that claim. USD amounts use the rate of JPY 157 per USD.

Domain categories of evidence set

Table 12 reports the number of unique links in the candidate set shown to annotators and in the subset they actually used for their judgments, broken down by domain category.

Additional Statistics

Number of claims per response. Figure 4 shows the distribution of the number of claims per response.

Question stance and temporal stability. Table 13 shows the distribution of the question stance and temporal stability.

Question stanceNTemporal stabilityN
Neutral71Static46
Reported9Long-term dynamic18
Leading5Short-term dynamic21

Table 13. Distribution of question stance and temporal stability.

Question attributes taxonomy. Table 14–Table 17 give the criteria used to classify the questions along each axis.

CodeNameDefinition and inclusion/exclusion criteria
D1Medicine and healthDiseases, treatments, pharmaceuticals, vaccines, nutrition and food components, and physiology. Excludes the management or corporate history of pharmaceutical companies (D8) and the operation of health-care systems (D4).
D2Science and technologyNatural phenomena such as earthquakes, weather, and astronomy, and scientific, technical, or infrastructure topics such as chemistry, biology, communications, and energy facilities. Includes events, policies, and facilities in these fields, not only scientific propositions. Nuclear energy and radiation take D3 priority; questions centered on health effects take D1.
D3Nuclear energyNuclear power plants, radioactive materials, treated water, and nuclear fuel. If human health effects are central, assign D1; if materials, facilities, or regulation are central, assign D3.
D4Government servicesTaxation, social security, benefits, disaster administration, government organizations, and public expenditure. Excludes legal interpretation itself (D5) and party politics (D6).
D5Justice and lawCourts, police, and other justice or law-enforcement institutions.
D6PoliticsPolitical parties, politicians, elections, and policy disputes. Excludes the content of public systems themselves (D4).
D7International affairsDiplomacy, military affairs, and relations among states. Domestic affairs within a foreign country are assigned to the relevant topic domain and marked as Foreign on Axis 2.
D8Economy and industryCompanies, products, services, agriculture, forestry, fisheries, tourism, and trade. Information-distribution functions of platform operators are assigned D9.
D9Media and broadcastingSocial-media companies, broadcasters, and information-distribution services when the relevant sector is itself the subject.
D10Society and genderSex and gender norms, foundations of human rights, and discrimination. Legal definitions of rights are assigned D5.
D11Culture and historyHistorical facts, place names, architecture, entertainment, sports, and claims about individuals whose topical sector cannot be identified from the question. The People component is entity-based rather than sector-based. If the topic of a person’s statement is explicit, assign the corresponding topical domain.

Table 14. Annotation scheme for Axis 1: Question domains.

CodeNameDefinition and inclusion/exclusion criteria
T1StaticThe conclusion does not change in any substantial way, as with past events, established findings, definitions, and norms.
T2Long-term dynamicThe conclusion may change over a period of more than a year to a decade or so, as with laws and institutions, official standards, and scientific findings that are still accumulating.
T3Short-term dynamicThe conclusion may change within days to a year, as with current conditions, ongoing cases, private-sector specifications, and future plans.

Table 17. Annotation scheme for Axis 4: Temporal stability.

Experimental Details

Models Used for Evaluation

Llama-3.1-Swallow tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3

Qwen3-Swallow tokyotech-llm/Qwen3-Swallow-8B-RL-v0.2

LLM-jp-4 llm-jp/llm-jp-4-8b-instruct

Mistral-Nemo-Ja cyberagent/Mistral-Nemo-Japanese-Instruct-2408

GPT-5.1 gpt-5.1-2025-11-13

Sonnet 5 us.anthropic.claude-sonnet-5 (Amazon Bedrock)

The open models were run on single NVIDIA A40 GPU (48 GB).

Evaluation Prompt

Figure 5–Figure 7 show three representative prompts; the remaining settings differ only in which components are present. The reference-time note appears under Time and Q&R. The Question and Answer blocks, together with the instruction fixing the unit of judgement to the target claim, appear only under Q&R. The Evidence block holds the quoted passage under Reference, the cited page under Full, and the candidate set under Pages; under None it is absent and the open-book instruction is replaced by its closed-book counterpart.

Prompt template for the Claim only setting

Figure 5. Prompt template for the Claim only setting (no evidence). Angle brackets mark the slots filled per claim.

Prompt template for the Claim + Time + Q&amp;R + Pages setting

Figure 7. Prompt template for the Claim + Time + Q&amp;R + Pages setting. Angle brackets mark the slots filled per claim.

Evidence windows. Under Full, the cited page’s body text is truncated to a 6,000-character window centred on the annotators’ quoted passage. Under

Pages, each candidate page is reduced to a 600-character window containing the paragraph nearest its search snippet; for the cited page only, the window is centred on the quoted passage, guaranteeing that the evidence used for the gold label is present in the input.

CodeNameDefinition and inclusion/exclusion criteria
G1JapanInstitutions, events, or people in Japan are the referent.
G2Foreign countryDomestic affairs within a specific foreign country are the referent.
G3International or globalThe claim has geographic referents but is not restricted to one country, including international organizations, cross-border events, global distributions, and global companies or platforms.
G0Universal or non-geographicThe proposition has no geographic referent, as in natural laws, general biomedical facts, or normative propositions. Universal validity alone does not imply G3; the distinction depends on whether the proposition refers to a particular place or institution.

Table 15. Annotation scheme for Axis 2: Geographic scope.

CodeNameDefinition and inclusion/exclusion criteria
Q1NeutralPresents a proposition and asks whether it is true without marking the speaker’s prior commitment, as in ordinary yes/no questions.
Q2ReportedMarks the proposition as information attributed to another source, using expressions such as “apparently,” “is it true that,” “I heard that,” or “there is a story that.”
Q3LeadingPresupposes the proposition and seeks agreement, using forms equivalent to “isn’t it?,” “right?,” or “wouldn’t you say?”

Table 16. Annotation scheme for Axis 3: Question stance.

Prompt template for the Claim + Time + Q&R + Reference setting

Figure 6. Prompt template for the Claim + Time + Q&R + Reference setting. Angle brackets mark the slots filled per claim.

You are a factchecker system.

Reference time: <TIME(YYYYY-MM-DD)>. Below, the question, the answer, the target statement and the evidence were collected on <TIME(YYYYY-MM-DD)>. Judge the target statement as of that time, not as of your training data.

Task:

Classify the target statement by factuality.

Factuality axis (Supported / Refuted / NotEnoughInfo):

  • Supported:

The evidence shows the statement to be correct.

  • Refuted:

The evidence shows the statement to be incorrect.

  • NotEnoughInfo:

The evidence is not sufficient to decide whether the statement is correct or incorrect.

Candidate labels:

  • Supported

  • Refuted

  • NotEnoughInfo

Instructions:

  • Choose exactly one label from the candidate labels above.

  • Output nothing except the return format below.

  • The target statement is a single sentence extracted from the answer in the Answer block below. The Question and Answer blocks are given only to resolve pronouns, ellipsis, and topic. Judge the target statement alone: do not judge the answer as a whole, and do not let the overall plausibility of the answer affect your label.

  • The answer is provided as context only. It is not guaranteed to be correct, so do not treat agreement with the answer as evidence that the target statement is correct.

  • Base your judgement on the evidence text. Do not rely on outside knowledge beyond it.

Question:

<QUESTION>

Answer:

<RESPONSE>

Evidence:

<QUOTED PASSAGE>

Target statement:

<TARGET CLAIM>

Return format (one line, no explanation):

Label: <Supported | Refuted | NotEnoughInfo>

References

  1. [1]Anthropic. 2026. Claude sonnet 5. https://www.anthropic.com/news/claude-sonnet-5. Accessed: 2026-10-22.
  2. [4]Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Ida, Masanari Oi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities. In arXiv.org, COLM, page (to appear).
  3. [7]Ryosuke Ishigami. 2024. Mistral-nemo-japanese-instruct-2408.
  4. [8]Saurav Kadavath, Tom Conerly, Amanda Askell, T. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv.org, arXiv:2207.05221.arxiv.org/abs/2207.05221
  5. [9]Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. TruthfulQA: Measuring how models mimic human falsehoods. In Annual Meeting of the Association for Computational Linguistics, pages 3214–3252. Association for Computational Linguistics.arxiv.org/abs/2109.07958
  6. [10]Xin Liu, Lechen Zhang, Sheza Munir, Yiyang Gu, and Lu Wang. 2025. VeriFact: Enhancing long-form factuality evaluation with refined fact extraction and reference facts. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17919–17936. Association for Computational Linguistics.arxiv.org/abs/2505.09701
  7. [11]LLM-jp, : , Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazuki Fujii, Kensuke Fukumoto, et al. 2024. Llm-jp: A cross-organizational project for the research and development of fully open japanese llms. Conference on Empirical Methods in Natural Language Processing, arXiv:2407.03963.
  8. [12]Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanah Hajishirzi. 2022. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Annual Meeting of the Association for Computational Linguistics, pages 9802–9822. Association for Computational Linguistics.arxiv.org/abs/2212.10511
  9. [14]Tomoka Nakazato, Masaki Onishi, Hisami Suzuki, and Yuya Shibuya. 2024. SocialFact: a Misinformation dataset from Social Media for Benchmarking LLM Safety . In BigData Congress [Services Society], pages 3017–3025, Los Alamitos, CA, USA. IEEE.DOI
  10. [15]F. Petroni, Tim Rocktäschel, Patrick Lewis, A. Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? In Conference on Empirical Methods in Natural Language Processing, pages 2463–2473. Association for Computational Linguistics.arxiv.org/abs/1909.01066
  11. [19]Aaditya K. Singh, A. Fry, Adam Perelman, A. Tart, A. Ganesh, Ahmed El-Kishky, A. McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. 2026. Openai gpt-5 system card. arXiv, arXiv:2601.03267.arxiv.org/abs/2601.03267
  12. [25]Jiayi Yao, Haibo Sun, and Nianwen Xue. 2025. Fact-checking ai-generated news reports: Can llms catch their own lies? arXiv.org, arXiv:2503.18293.arxiv.org/abs/2503.18293

Paper details

vlnc:W_kav5p2pj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents