betaFeedback

WorksComputer Science

Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval

Abstract

Coverage-evaluation benchmarks quantify whether retrieved evidence covers the different pieces of information needed to satisfy an information need. Existing benchmarks mostly consider factual claims that may already be known to pretrained retrieval components, leaving open whether retrievers find corpus evidence when it contradicts those priors. We study how retrievers behave when their parametric knowledge and the information in a corpus disagree. To do so, we develop a process to augment a nugget-based coverage benchmark with controlled contradictions. For each nugget, we create a negated counterpart and edit its supporting documents to support that negation while preserving the context. Applying this process to CoverageBench, a benchmark measuring information coverage, we create CoverageBench-Negate, spanning six datasets and roughly 63,000 original-edited evidence pairs (one per nugget-document link). We attach rich construction metadata to every pair and additionally provide out-of-family LLM judgments over all pairs and a human audit of a sample. In baseline experiments, BM25 separates original from edited near-twin evidence substantially better than dense retrievers (Qwen3 and Jina), which stay near chance. We release the benchmark on Hugging Face Datasets (nuggets, documents, pairs) together with the construction materials, including the prompts used to create it.

This work is licensed under a Creative Commons Attribution 4.0 International License.

CIKM ’26, Rome, Italy

© 2026 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-2539-5/2026/11

https://doi.org/10.1145/3799682.3840200

ACM Reference Format: Pasha Abrishamchian, Eugene Yang, Andrew Yates, Nazli Goharian, and Ophir Frieder. 2026. Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), November 07–11, 2026, Rome, Italy. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3799682.3840200

Introduction

Retrieval-augmented generation (RAG) systems rely heavily on the information they retrieve. If the retrieved documents repeat the same point or leave out an important claim, the final answer can be incomplete even when every document is relevant. Retrieval quality, thus, depends not only on finding relevant documents, but on finding a complete set of distinct evidence to support an answer.

Rather than focusing on whether each retrieved document is relevant, coverage evaluation asks whether the retrieved set contains the information needed to answer a query. This information is often represented as nuggets: small, atomic claims that a complete answer should cover [8, 17]. Existing coverage benchmarks typically evaluate ordinary historical or world facts that commonly appear in the data used to train language models. Meanwhile, modern retrieval pipelines increasingly build on those same language models. Many dense retrievers, neural rerankers, and LLM-based query expansion methods inherit, directly or indirectly, the world knowledge absorbed during pretraining. Consequently, they may favor the more familiar version of a claim over the evidence that is actually present in the corpus [18]. This raises the question: when the corpus supports a claim that conflicts with what these language models have seen during pretraining, will the retrieval system still surface that evidence?

To study this, we develop a general process for augmenting a nugget-based coverage benchmark with controlled contradictions. The process keeps the original topics fixed, and for each nugget, it constructs a negated counterpart. Then, it edits the supporting documents so they support the negation while preserving the surrounding context. Since the topics, entities, and contexts stay fixed, this paired design isolates the polarity flip from confounds such as topic or entity shift. Applying it to CoverageBench [20], we create CoverageBench-Negate. It covers six datasets: 284 topics and roughly 63,000 edited document pairs, spanning every topic and 2,975 nuggets. Each evidence pair is released with quality metadata so researchers can apply strict or permissive filters depending on the diagnostic setting.

Our contributions are: (1) We develop a reusable process for augmenting nugget-based coverage benchmarks with controlled contradictions to test whether retrievers stay faithful to corpus evidence, and use it to create CoverageBench-Negate; (2) We release the full benchmark on Hugging Face Datasets (negated nuggets, edited supporting documents, a paired evidence set, quality metadata, and our human and LLM-based validations), making it auditable and reusable; and (3) We provide a baseline characterization over retriever-visible top-1000 subsets, showing that BM25 distinguishes original from edited near-twin evidence better than dense retrievers (Qwen3 and Jina).

Background

Information coverage evaluates a retrieved set of documents by how many distinct pieces of information (aspects) it covers. This underlies recent nugget-based evaluations, where answers are scored for completeness and grounding against nuggets: ARGUE for machine-generated reports [13], the AutoNuggetizer framework for TREC RAG [17], the Ragnarök RAG evaluation framework [16], and nugget-grounded generation pipelines such as GINGER [8]. CoverageBench [20] provides a reusable resource for evaluating retrieval by standardizing topics, nuggets, nugget-level relevance labels, document collections, and baseline rankings across multiple retrieval settings.

Crucially for a retrieval-focused benchmark, retrieval-side coverage metrics were shown to correlate strongly with the information coverage of downstream RAG responses [19], so evaluating coverage at the retrieval stage is a meaningful proxy for end-to-end RAG behavior.

A related line of work probes whether RAG systems follow the corpus or fall back on parametric knowledge. Closest to our setting, CRUMQs augments RAG evaluation with unanswerable multi-hop queries, testing whether systems invoke parametric knowledge when the required evidence is absent [11]. Building on the CoverageBench structure, CoverageBench-Negate poses a complementary question of whether systems still cover corpus-supported claims when those claims conflict with parametric knowledge. CRUMQs probes reliance on outside knowledge by removing support, while we probe it by adding contradicting, corpus-supported evidence.

Work on factual evaluation states that atomic claims must be explicitly supported, not merely topically related [14, 22]. Yet dense retrievers, rerankers, and LLM-based query expansion tend to reward semantic similarity, fluent reformulation, and learned associations, and can be brittle under subtle, adversarially constructed perturbations [12].

Negation is a particularly hard case. Recent analysis shows that dense retrievers may treat two passages as similar because they share a topic even when one states “X” and the other “not X” [18]. Negation-sensitive ranking benchmarks such as NevIR find that retrievers struggle to separate a claim from its negation [15, 24, 25]. In a corpus-grounded setting, however, the desired behavior is not to retrieve what is globally likely but what the corpus actually supports [7]. Testing this cleanly requires paired evidence in which the context stays fixed while only the supported claim changes.

Negation offers a practical way to construct such pairs, flipping the supported claim while leaving the topic, entities, and much of the document intact. Prior work shows that models struggle when negation changes entailment in controlled NLI settings [5, 21, 23], precisely the meaning-changing edit we exploit. This mirrors contrast sets, where small label-changing edits test whether a model responds to the changed meaning rather than surface form [4]. We use negation as a controlled augmentation rather than a target in itself: each original nugget and supporting document is paired with a negated nugget and an edited document, with construction details in § 3.

Benchmark Construction

CoverageBench-Negate1 applies a reusable, negation-based augmentation process to six CoverageBench [20] datasets: CasT 2020 [2], NeuCLIR 2024 [9], RAG 2024 [17], CRUX-DUC04 [6], CRUX-MultiNews [6], and RAGTIME 2025 [10]. We omit Fair Ranking 2022 [3] because its nuggets are fairness facets rather than factual claims.

During augmentation, we keep the original topics, nuggets, and documents unchanged and add a paired negation layer on top. Figure 1 illustrates the pipeline. We generate and validate one negated nugget per original nugget, and then edit and validate a supporting document for each surviving link.2 Since the same model both produces and judges these artifacts, we additionally validate the released pairs with an independent, out-of-family judge and a human audit. The resource consists of the following components:

Figure 1. Construction pipeline for CoverageBench-Negate. The nugget lane (top, yellow) produces one negated nugget per original nugget; the document lane (bottom, green) edits one supporting document per nugget–document link, funneling 116,842 links down to 63,372 released pairs, spanning 2,975 nuggets.

  • Topics, original nuggets, and links. We reuse all 284 CoverageBench topics without modification, and retain the 3,496 nuggets and 116,842 nugget–document support links. A link is a tuple (t,n,d)(t, n, d): document dd supports nugget nn under topic tt, and 3,234 nuggets have at least one supporting document. Keeping the originals lets the benchmark continue to measure ordinary information coverage alongside the negated coverage.

  • Negated nuggets. For every nugget nn, we create one negated nugget nˉ\bar{n} that contradicts the original claim while preserving the topic scope, entities, and event context (§ 3.1).

  • Edited documents. For each support link, we attempt to create an edited document dˉ\bar{d} that supports nˉ\bar{n} instead of nn, changing only the target claim while keeping the context intact. Editing is per link rather than per unique document since one source document can support several nuggets. After pre-filtering and failed edits, the 116,842 links yield 63,372 edited documents (§ 3.2).

  • Paired evidence set. The main artifact is a link-level paired evidence set of 63,372 pairs spanning every topic, and 2,975 original nuggets. Each pair connects an original topic, nugget, and supporting document with a negated nugget, edited document, and construction metadata.

  • Metadata. At every stage of the pipeline, we record diagnostics (edit-size and embedding-similarity measurements), validation outcomes, and categorical labels (negation type), so that users can filter the benchmark to their needs. Some checks determine what enters the benchmark; the rest are labels users can filter on (§ 3.3).

Negated Nugget Construction

For each nugget, we prompt the LLM to write a negated version such that the original and the negation cannot both be true. Two LLM checks then refine the result. A single validation call scores three criteria: whether the negation reverses the original meaning, reads naturally as a standalone claim, and preserves the relevant entities. A separate mutual-exclusivity call asks whether the original and negated nuggets could both be true, regenerating the negation from the failed attempt up to three times. Separately, an embedding check labels each negation by whether it stays on-topic yet meaningfully different from the original and whether it collides with another gold nugget in the same topic. These checks label quality rather than filter, and no nugget is removed at this stage.

Edited Document Construction

Editing operates per nugget–document link (116,842 in total). We first remove links whose negated nugget was flagged as colliding with another gold nugget in the same topic. Next, the LLM localizes the verbatim span that supports the original nugget; if it finds none, the link is excluded. The model then rewrites only that span so the document supports the negated nugget instead. Deterministic checks verify the edit’s length, the absence of synthetic-edit language, and the removal of the original span. A separate LLM judge confirms that the edited document no longer supports the original nugget, does support the negated nugget, and preserves the surrounding context. An edit that fails any check receives one failure-aware retry. For example, to support the negated nugget “WBC count often remains elevated even after the infection has cleared,” the editor rewrites the supporting sentence “…the child’s WBC count will return to normal.” into “…the child’s WBC count often remains elevated,” flipping only the target claim while leaving the rest of the document intact. The pipeline yields 63,372 evidence pairs (54.2% of the links) as three outcomes were ineligible links: nugget collisions (5.7%), missing supporting spans (35.9%), and empty or rejected edits after retry (4.2%).

Metadata

At the nugget creation stage, the LLM tags each negation with one of six types: event negation, outcome inversion, relationship negation, quantitative inversion, antonym substitution, and directional reversal. We also record the cosine similarity between the original and negated nuggets. At the document creation stage, alongside the judge’s verdict, we record the edit size and three cosine similarities of the edited document to the negated nugget, the original nugget, and the source document. From these comparisons, we derive a three-tier confidence signal (pass, review, or fail): a pair passes when the edited document is closer to the negated nugget than to the original and stays similar enough to the source.

Validation

We re-judge all 63,372 pairs with Gemma-4-31B-it under the same criteria outlined in § 3.2 and release the full judgments. Gemma confirms all three on 78% of pairs (87–97% per criterion). On the nugget layer, Gemma reproduces the negation validation for 82% of the negated nuggets, but agrees with Llama on mutual exclusivity about 53% of the time (Cohen’s k=0.05k = 0.05). Judging whether two claims can both be true is subjective, so we record mutual exclusivity as a label rather than filtering based on it.

Two annotators independently rated a stratified 120-pair sample (60 pass / 30 review / 30 fail by the confidence tiers of Section 3.3, all judge-approved) on five criteria mirroring the construction checks. Both confirm the core properties: the negation is a genuine contradiction (90% and 78%) and unrelated context is preserved (86% and 91%). They are stricter than the automatic judges on the two edit criteria (e.g., the edited document no longer supports the original, 88% and 58%), where they also agree less (64%). The human bar for a flawless pair is thus higher and more variable than the judges’.

Baseline

We provide baseline results for three first-stage retrievers (BM25, Qwen3–Embedding–8B, and jina-embeddings-v5–text-small) to illustrate how CoverageBench-Negate can be used both as an augmented coverage benchmark and as a paired diagnostic resource.

The benchmark-style evaluation follows CoverageBench directly. We add the edited documents to the collection and the negated nuggets with their qrels, then score top-20 rankings with α\alpha-nDCG [1] and Subtopic Recall (StRecall) [26]. Both are computed at rank 20: α\alpha-nDCG (standard α=0.5\alpha= 0.5) discounts the jj-th document covering a nugget by (1−α)j−1(1-\alpha)^{j-1}, while StRecall is the fraction of a topic’s nuggets covered in the top 20. We use the paired evidence structure to analyze why the dense and lexical retrievers degrade differently.

Augmented Coverage Evaluation

Table 1 reports the per-dataset change from the original CoverageBench evaluation to the augmented evaluation. The augmented benchmark reduces coverage for all three retrievers, but the degradation is larger for the two dense ones. Averaged over datasets, BM25 drops by 0.083 in α\alpha-nDCG@20 and 0.145 in StRecall@20, while the two dense retrievers degrade almost identically, dropping 0.140–0.146 in α\alpha-nDCG@20 and 0.180–0.181 in StRecall@20. Since the dense retrievers also start from higher coverage, we compare drops relative to their original scores. The α\alpha-nDCG gap persists (mean relative drop 16.8% for BM25 vs. 22.3% and 24.5% for Qwen3 and Jina), while the StRecall gap nearly closes (21.4% vs. 22.9% and 23.6%). The dense retrievers’ larger α\alpha-nDCG sensitivity is thus robust, whereas their larger absolute StRecall drop partly reflects their higher starting coverage.

DatasetΔα\Delta\alpha-nDCG BM25Δα\Delta\alpha-nDCG Qwen3Δα\Delta\alpha-nDCG JinaΔ\DeltaStRecall BM25Δ\DeltaStRecall Qwen3Δ\DeltaStRecall Jina
CAsT 2020+0.020-0.01-0.08-0.08-0.12
CRUX-DUC04-0.08-0.17-0.17-0.16-0.25-0.27
CRUX-MultiNews-0.05-0.09-0.10-0.05-0.07-0.07
NeuCLIR 2024-0.11-0.13-0.17-0.15-0.20-0.18
RAG 2024-0.13-0.27-0.25-0.23-0.24-0.20
RAGTIME-0.15-0.18-0.18-0.20-0.23-0.24

Table 1. Coverage change after augmentation (rank 20, augmented-original); larger negative values indicate larger degradation.

CAsT is the exception for α\alpha-nDCG, which remains approximately flat for all three retrievers because some edited documents enter the top-20 as non-redundant gain. StRecall still drops for all three, so we treat CAsT as metric-specific rather than evidence against the degradation pattern. Overall, these results indicate that, in most cases, the retrieval models do not rank documents supporting negated claims the way they rank the originals.

Paired Score-Gap Diagnostic

For each retriever, query, and original–edited document pair, we compare the raw score assigned to the original document, sorigs_{\mathrm{orig}}, with the raw score assigned to the edited negation-supporting document, sedits_{\mathrm{edit}}. We define the score gap as Δs=sedit−sorig\Delta_s = s_{\mathrm{edit}} - s_{\mathrm{orig}}. A negative value means the retriever prefers the original document, while a positive value means it prefers the negated document. This analysis uses retriever-specific subsets of the paired evidence set (28,952 pairs for BM25, 47,054 for Qwen3, and 43,942 for Jina) where both sorigs_{\mathrm{orig}} and sedits_{\mathrm{edit}} are available in the top-1000 retrieved documents. The two scores are compared within the same retriever and query setting, so the sign of Δs\Delta_s gives a direct pairwise preference signal, though the raw magnitudes are comparable only within a retriever.

We stratify pairs by the document edit size (character-level difference between the original and edited documents) and assign each pair to a bucket ranging from tiny (<2% change) to vast (>25% change) edits. Small edits are the cleanest test since most wording and topic remain unchanged. Any shift in a retriever’s scores can then only come from the flipped claim.

Table 2 explains the coverage deltas. BM25 prefers the original in about 84% of tiny-edit pairs, decreasing to about 61% as edits grow. In contrast, the two dense retrievers stay near chance across every bucket (Qwen3 54–61%, Jina 49–56%), with near-zero median gaps on their scale, so they cannot reliably tell the original from its negation. The contrast is sharpest on near-twin pairs, where BM25 discriminates far better than either dense retriever. The same pattern carries over to the actual rankings a downstream system consumes. When both documents fall in the same with-negated top-1000 list (26,824 BM25, 44,034 Qwen3, 42,140 Jina pairs), BM25 ranks the original above the edit in 83% of tiny-edit pairs, falling to 57% for vast edits, while Qwen3 (roughly 51–55%) and Jina (48–55%) stay near chance across the sub-25% buckets.3

RetrieverMetricTinySmallMediumLargeVastAll
BM25PREF.83.9072.1065.7061.3061.4068.80
BM25Med.0.130.160.240.440.630.24
Qwen3PREF.58.1054.0054.2055.2060.5056.00
Qwen3Med.0.010.010.010.020.020.01
JinaPREF.56.3050.9049.9050.9055.2052.20
JinaMed.0.000.010.010.010.020.01

Table 2. Score-gap by edit length over retriever-specific top-1000 subsets. PREF. is the percentage of pairs where the retriever scores the original document above its negated counterpart; Med. is median ∣Δs∣|\Delta_s|.

Conclusion

CoverageBench-Negate tests whether first-stage retrievers cover corpus evidence that contradicts pretrained expectations. Its negated nuggets, edited documents, paired evidence, and quality metadata enable studies of corpus faithfulness under controlled contradiction. Our initial characterization separates lexical and dense retrievers, making this a useful diagnostic for coverage-oriented retrieval.

Its premise is that a negated claim conflicts with parametric knowledge, but we do not verify this conflict directly, and we do not analyze whether retained links differ systematically from those excluded. Independent edits may introduce higher-order contradictions. Construction and quality checks are LLM-based, and we mitigate this with an independent out-of-family judge and a human audit.

GenAI Usage Disclosure

Generative AI is central to the construction of CoverageBench-Negate. Llama-3.3-70B-Instruct generates every negated nugget, localizes the supporting span, rewrites each edited document, and performs all LLM-based quality checks (semantic validation, mutual exclusivity, and the document judge). Qwen3-Embedding-8B produces all embedding-based checks and confidence tiers, and Gemma-4-31B-it serves as the independent, out-of-family validation judge. The released negated nuggets, edited documents, and paired evidence set are therefore LLM-generated and LLM-judged, as detailed in Section 3. The authors additionally used generative AI tools to assist with writing and editing, code debugging, and figure creation. The authors reviewed and take full responsibility for all content and analyses presented in this paper.

References

  1. [1]Charles L.A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan Büttcher, and Ian MacKinnon. 2008. Novelty and Diversity in Information Retrieval Evaluation. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, Singapore Singapore, 659–666. doi:10.1145/1390334.1390446DOI
  2. [2]Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2021. CaST 2020: The Conversational Assistance Track Overview. In The Twenty-Ninth Text REtrieval Conference Proceedings (TREC 2020) (NIST Special Publication, Vol. 1266). National Institute of Standards and Technology (NIST). https://trec.nist.gov/pubs/trec29/papers/OVERVIEW.C.pdf
  3. [3]Michael D. Ekstrand, Graham McDonald, Amifa Raj, and Isaac Johnson. 2023. Overview of the TREC 2022 Fair Ranking Track. arXiv:2302.05558 [cs.IR] doi:10.48550/arXiv.2302.05558DOI
  4. [4]Gordon V. G. Gao, Yao Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanneke Hajishirzi, Gabriel Illarco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating Models’ Local Decision Boundaries via Contrast Sets. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 1307–1323. doi:10.18653/v1/2020.findings-emnlp.117
  5. [5]Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, Online, 163–173. doi:10.18653/v1/2020.blackboxnlp-1.16DOI
  6. [6]Jia-Huei Ju, Susan Verberne, Maarten de Rijke, and Andrew Yates. 2026. Controlled Retrieval-augmented Context Evaluation for Long-form RAG. arXiv:2506.20051 [cs.IR] doi:10.48550/arXiv.2506.20051arxiv.org/abs/2506.20051
  7. [7]Ahmed Rayane Kebir, Vincent Guigue, Lynda Said Lhadj, and Laure Soulier. 2026. RAC: Retrieval-Augmented Clarification for Faithful Conversational Search. arXiv:2601.11722 [cs] doi:10.48550/arXiv.2601.11722
  8. [8]Weronika Łajewska and Krzistian Balog. 2025. GINGER: Grounded Information Nugget-Based Generation of Responses. arXiv:2503.18174 [cs] doi:10.48550/arXiv.2503.18174arxiv.org/abs/2503.18174
  9. [9]Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, and Eugene Yang. 2025. Overview of the TREC 2024 NeuCLIR Track. arXiv:2509.14355 [cs.IR] doi:10.48550/arXiv.2509.14355DOI
  10. [10]Dawn Lawrie, Sean MacAvaney, James Mayfield, Luca Soldaini, Eugene Yang, and Andrew Yates. 2026. Overview of the TREC 2025 RAGTIME Track. arXiv:2602.10024 [cs.IR] doi:10.48550/arXiv.2602.10024DOI
  11. [11]Gabrielle Kaili-May Liu, Bryan Li, Arman Cohan, William Gantt Walden, and Eugene Yang. 2025. Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries. doi:10.48550/ARXIV.2510.11956DOI
  12. [12]Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten De Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023. Black-Box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Method. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. ACM, Birmingham United Kingdom, 1647–1656. doi:10.1145/3583780.3614793DOI
  13. [13]James Mayfield, Eugene Yang, Dawn Lawrie, Sean MacAvaney, Paul McNamee, Douglas W. Oard, Luca Soldaini, Ion Sobo r off, Orion Weller, Efsun Kayi, Kate Sanders, Marc Mason, and Noah Hibbler. 2024. On the Evaluation of Machine-Generated Reports. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1904–1915. arXiv:2405.00982 [cs.CL] doi:10.1145/3626772.3657846DOI
  14. [14]Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FactScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Singapore, 12076–12100. doi:10.18653/v1/2023.emnlp-main.741DOI
  15. [15]Roxana Petcu, Samarth Bhargav, Maarten De Rijke, and Evangelos Kanoulas. 2025. A Comprehensive Taxonomy of Negation for NLP and Neural Retrievers. In Findings of the Association for Computational Linguistics: EMNLP 2025.
  16. [16]Ronak Pradeep, Nandan Thakur, Sahel Shafigh Mohammadhadi, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track. arXiv:2406.16828 [cs.IR] doi:10.48550/arXiv.2406.16828arxiv.org/abs/2406.16828
  17. [17]Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework. arXiv:2411.09607 [cs] doi:10.48550/arXiv.2411.09607DOI
  18. [18]Soumya Ranjan Sahoo, Gagan N., Sanand Sasidharan, and Divya Bharti. 2026. Negation Is Not Semantic: Diagnosing Dense Retrieval Failure Modes for Trade-offs in Contradiction-Aware Biomedical QA. doi:10.48550/ARXIV.2603.17580
  19. [19]Saron Samuel, Alexander Martin, Eugene Yang, Andrew Yates, Dawn Lawrie, Laura Dietz, and Benjamin Van Durme. 2026. Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage. doi:10.48550/ARXIV.2603.08819DOI
  20. [20]Saron Samuel, Andrew Yates, Dawn Lawrie, Ian Soboroff, Trevor Adriaanse, Benjamin Van Durme, and Eugene Yang. 2026. CoverageBench: Evaluating Information Coverage across Tasks and Domains. doi:10.48550/ARXIV.2603.20034DOI
  21. [21]Jingyuan S. She, Christopher Potts, Samuel R. Bowman, and Atticus Geiger. 2023. Scone: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Toronto, Canada, 1803–1821. doi:10.18653/v1/2023.acl-short.154
  22. [22]James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, New Orleans, Louisiana, 809–819. doi:10.18653/v1/N18-1074
  23. [23]Thanh-Huyen Tuong, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Jey Han Lau, and Karin Verspoor. 2022. Not Another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online only, 883–894. doi:10.18653/v1/2022.aacl-main.65
  24. [24]Coen Van Den Elsen, Francien Barkhof, Thijmen Nijdam, Simon Lupart, and Mohammad Aliannejadi. 2025. Reproducing NevIR: Negation in Neural Information Retrieval. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, Padua Italy, 3346–3356. doi:10.1145/3726302.3730294
  25. [25]Orion Weller, Dawn Lawrie, and Benjamin Van Durme. [n.d.]. NevIR: Negation in Neural Information Retrieval. [n.d.].
  26. [26]Chengxiang Zhai, William W. Cohen, and John Lafferty. 2003. Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. (2003), 10–17. doi:10.1145/860435.860440DOI

Paper details

vlnc:W_hr6437mj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents