WorksComputer Science
Defining Page-Centric Visual Document Retrieval and Evaluating Existing Benchmarks
Abstract
In modern information retrieval (IR), documents are increasingly encountered as structured sequences of visually rich, paginated units, rather than as flat text. Example documents encountered in our daily life include Word files, PDFs, and slide decks. Free text-centric retrieval systems either ignore this structure entirely or destroy it through arbitrary passage chunking. In this paper, we formulate page-centric Visual Document Retrieval (VDR) with pages as the fundamental indexing units and document context available for assessing page relevance. The formulation distinguishes two retrieval granularities, page and document, and three retrieval paradigms: independent-item, evidence-set, and iterative retrieval. We then assess 42 benchmarks using eight criteria covering corpus characteristics, query construction, relevance annotation, and ranking support. Our analysis shows two main findings on current VDR research. First, evaluation remains concentrated on page-level independent-item retrieval, with fewer resources supporting document-level, evidence-set, and iterative settings. Second, queries originating from real user information needs independently of target corpus content and graded relevance judgments are uncommon, while the coverage and verification of relevance judgments vary substantially. Both findings reveal gaps between the capabilities applications require and those existing evaluation protocols can establish. We provide benchmark selection guidance and identify priorities for future benchmark design, emphasizing alignment between task requirements, relevance semantics, and evaluation protocols.
CCS Concepts
• Information systems → Test collections; Relevance assessment; Document filtering.
Keywords
Visual document retrieval, benchmark analysis, multi-modal retrieval, retrieval-augmented generation
Introduction
Real-world document corpora such as technical reports, slide decks, scanned forms, financial filings, are rarely flat text. They are organized as ordered sequences of visually formatted pages. The layout, typography, and embedded figures or tables carry meaning that is inseparable from the surrounding text in each or a few pages in sequence. Traditional text-centric IR systems handle such corpora poorly: they either treat a document as an unstructured stream of text or rely on arbitrary passage chunking, both of which discard the spatial pagination and visual structure that document authors use to convey relevance-bearing information [43].
In fact, this points to a more basic mismatch. In classical IR, a "document" is a logical unit of information defined for the purposes of indexing and retrieval: a convenient abstraction, not a claim about how information is actually packaged. That abstraction often diverges from the documents we produce and read day to day, which are physically paginated1, visually structured artifacts. This calls for serious consideration of a page-centric view of information retrieval: treating the page rather than the document as the fundamental unit of indexing and retrieval. Accordingly, we examine the applications enabled by this view and ask how well page- and document-level retrieval serve end users’ information-seeking needs, particularly when documents span multiple pages and users need to refer to individual pages or entire documents.
Recent work on Visual Document Retrieval (VDR) has begun to address this gap by treating a page image itself as the unit of retrieval, most notably through late-interaction vision-language models evaluated on the ViDoRe benchmark family [10, 29, 31]. However, the literature has grown organically around whatever a given benchmark happens to measure, and no work to date separates the formal task of page-centric visual document retrieval from the benchmark design choices used to evaluate it. This conflation makes it difficult to answer two questions that matter for both research and deployment: (1) what precisely is being retrieved and ranked, for example, individual pages, whole documents, or both, and (2) which of the many existing benchmarks are actually fit to evaluate which kinds of application. This paper makes three contributions:
A page-centric task formulation (§ 3). We explicitly define page- and document-level relevance, and organize retrieval settings by granularity and retrieval paradigm, with motivating applications detailed in Appendix § 7.
A structured analysis of benchmark readiness (§ 4–§ 5). We define 8 criteria and apply them to 42 benchmarks, clarifying which retrieval capabilities their corpus construction, annotations, and evaluation protocols can meaningfully assess.
Task-specific benchmark guidance (§ 5, and detailed in Appendix § 7). We structure our benchmark recommendations across two levels of granularity (i.e., page- and document-level relevance) and three retrieval paradigms (individual items, evidence set, and iterative retrieval), yielding six distinct granularity-paradigm combinations. For each combination, we outline application-based selection rules and discuss evaluation gaps and design considerations for future benchmark design.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Related Work
In IR, documents are often represented and retrieved as whole documents or smaller passages using sparse or dense representations. Passage-level retrieval commonly relies on chunking documents into fixed-length token spans, disregarding their original layout. In VDR, pages serve a similar functional role as passages, providing sub-document units suitable for embedding, retrieval, and downstream processing, such as answer generation.
However, the two approaches differ in how retrieval boundaries are determined. Token-based chunking imposes artificial boundaries, often dictated by model constraints, which may split semantically coherent elements such as tables or figure captions. In contrast, page boundaries are inherent to fixed-layout documents, such as PDFs and slide decks, and often reflect layout decisions intended to preserve the semantic coherence of visual and textual content. Page-level retrieval thus leverages the document’s existing layout structure rather than imposing arbitrary token-based boundaries. This makes pages a natural retrieval unit for visual documents, as reflected in the predominance of page-level evaluation among existing VDR benchmarks (§ 4.3).
Nevertheless, page boundaries do not always strictly align with semantic boundaries. Identifying the relevance of an individual page may require contextual information from neighboring pages or the entire document. Document-aware passage retrieval provides a useful precedent for addressing such dependencies. DAPR [44] and ConTEB [7] investigate text retrieval settings where identifying relevant passages requires information beyond the passages themselves. More recently, CMDR [40] extends this perspective to visual documents by introducing context-dependent page retrieval and contextual multimodal embeddings within an intra-document retrieval setting. Building on these studies, our task formulation explicitly accounts for document context when assessing page relevance (§ 3). Accordingly, our benchmark analysis examines whether existing evaluation settings can meaningfully assess such contextual dependencies (§ 5).
Visual document retrieval benchmarks. Early VDR benchmarks often repurposed document visual question answering (VQA) datasets by treating questions as retrieval queries and their source images as relevant targets, as exemplified by ViDoRe V1 [10] and VisRAG [54]. Subsequent benchmarks have expanded evaluation along several dimensions, including multi-page and multi-document collections [3, 5, 21, 29], more realistic query construction [25, 29, 31], graded relevance judgments [6, 19, 29], and complementary evidence retrieval across reasoning hops [36]. More recent efforts further explore whole-document relevance [52] and iterative evidence acquisition [9, 18, 49]. Collectively, these developments have broadened VDR evaluation beyond conventional single-item retrieval toward more diverse retrieval objectives and settings. Building on these advances, we systematically examine benchmark design choices in § 4 and assess their suitability for different retrieval requirements in § 5.
Surveys and benchmark analysis. Recent surveys have extensively reviewed VDR and document RAG from different perspectives. Zhang [55] categorizes MLLMs by their roles in VDR as captioners, multimodal embedders, and end-to-end representers, examining retrieval granularity and fidelity–efficiency trade-offs. Gao et al. [11] organize document RAG models by domain, modality, and granularity, while summarizing benchmark characteristics, including corpus scale, content types, and reported retrieval and generation performance. Most relevant to ours, is Yan et al. [50]. In this dedicated survey of VDR, authors analyze existing benchmarks in terms of corpus and query sizes, multilingual coverage, reasoning requirements, and evaluation metrics under a page-image ranking formulation. They further distinguish retrieval methodologies from their applications in RAG and agentic systems.
Not focusing on methodological aspects of VDR, our work complements these surveys by systematically assessing benchmark suitability against an explicit VDR task formulation and its application requirements. Specifically, we distinguish page-level and document-level ranking, and three retrieval paradigms: independent-item, evidence-set, and iterative retrieval. Based on this formulation and the practical applications, we establish 8 operational criteria covering corpus structure and scale, query construction, relevance coverage, result cardinality, relevance grades, and ranking support. We then evaluate 42 benchmarks against these criteria and use the resulting assessments to provide task-specific benchmark recommendations. Rather than merely cataloging benchmark characteristics, our analysis aims to clarify which retrieval capabilities each benchmark can meaningfully evaluate, establish a common vocabulary for describing benchmark properties and identifying potential evaluation gaps and biases.
Task Formulation
Let be a corpus of structured documents. Each document is an ordered sequence of formatted pages, , where is the number of pages in . Each page is a multi-modal unit with , where denotes textual content like body text, headers, captions, and denotes visual characteristics. Examples of the latter include layout, typography, charts, tables, embedded figures, and spatial formatting.
Note that the separation of and in defining a page is for a formal description of page presentation as well as the comprehensiveness of the problem definition. It does not constrain whether a page must contain both modalities, or how a page is modeled during retrieval, e.g., as a page image capturing everything on the page or as a collection of textual and visual components parsed according to the page layout.
Page Relevance
Let denote a user’s information need, expressed as a textual, visual, or multi-modal query. For simplicity, we consider only textual queries throughout this paper, as query modality is not our primary focus. The core computational objective of VDR is to learn a joint multi-modal relevance function
estimating the relevance of page to query , conditioned on its parent document as global context.
In Eq. (1), conditioning on allows to capture cross-page dependencies, document-level discourse, and sequential semantic flow that a page-isolated scorer cannot represent. Nevertheless, the conditioning on can be relaxed, or even completely dropped, so that
Figure 1. Overview of page-centric VDR. A document-conditioned relevance function scores each page of every document in the corpus; the scores support two rankings, over pages () and over documents (), which serve the downstream applications.
is estimated purely from the alone and . This relaxed form is worth distinguishing explicitly because it is what most existing retrieval systems actually implement: treating each page as an independent unit rather than reasoning over document-level context. We return to this distinction in our discussions later.
Relevance semantics. Eq. (1) produces a score. The ground truth it is evaluated against is a relevance label , with for binary judgments. Benchmarks give these labels different meanings, and three of them matter for the analysis in § 4.
A page is sufficient if it contains the information needed to satisfy on its own. It is partially relevant if it contains necessary but incomplete evidence, such as one hop of a multi-hop question or the continuation of a table. It is contextual if it contains little direct evidence but is needed to identify or disambiguate the target page, for example the page that introduces the entity the question refers to. Treating all three as equally “relevant” may obscure what a retriever is actually capable of doing.
Once the relevance score is estimated, it is not difficult to localize and highlight some supporting evidences within a page, for example, as bounding boxes around relevant regions. We treat such highlighting as a post-processing step and not part of the essential retrieval process. The specific form of such evidence and how to present them to end users are application-dependent and not the key focus of our discussion here.
Retrieval and Ranking Granularities
Depending on the downstream application, VDR supports retrieval and ranking at two complementary granularities. Page-level ranking can surface relevant pages whose signal would be diluted or lost if only document-level scores were reported. Document-level ranking requires a non-trivial aggregation function that page-level ranking does not need.
Page-level ranking. The system returns a single global ranked list of individual pages across the entire collection,
where each retrieved page retains a deterministic pointer to its parent document, giving immediate access to broader context.
Document-level ranking. The system instead returns a ranked list of full documents,
ordered by a document-level relevance score derived from an aggregation function (e.g., max-pooling, soft-top-, or attention-based pooling) over constituent page scores. The arguments after the semicolon denote optional document attributes (e.g., length, title, metadata, authority measures) and corpus-level statistics. For each retrieved document, the system surfaces the specific attribution pages that drove its document-level rank.
Document relevance. The ground truth for document-level ranking can be defined in three ways. (a) Existential: is relevant if it contains at least one relevant page, . (b) Holistic: is relevant because the document as a whole addresses the information need, e.g., the lease agreement or the deck that introduces a product. (c) Evidence-set: is relevant because it supplies one member of the set of documents needed to answer the query jointly. Under (a) and (c), document labels are deterministic functions of page labels, so document-level ranking reduces to page-level ranking followed by aggregation. Under (b), aggregation over page scores can fail in both directions: a document-level topical match may not show up in any single page score, while max-pooling over a long document can reward one incidental page.
Retrieval Paradigms and Evaluation
Granularity specifies what is ranked. A second, orthogonal axis specifies how the ranked list is judged. Next, we distinguish three paradigms.
Independent items. Each ranked page or document is evaluated independently based on its relevance label, even when several items are relevant to the same query. This mirrors the classic IR setting, and the standard IR metrics apply: nDCG@K for graded relevance labels, and Recall@K, MAP, or MRR for binary relevance labels.
Evidence set. Answering requires a set of pages (or documents) that are jointly sufficient, for example one page per reasoning hop. The scorer may remain item-wise, but success is judged on the set: All@K requires every member of in the top , whereas Hit@K per hop requires at least one acceptable item for every hop and thereby allows substitutable evidence. We classify a benchmark under this paradigm only if it explicitly annotates complementary relationships among evidence items and adopts set-level evaluation metrics that assess their joint retrieval.
Iterative retrieval. An agent interleaves planning, retrieval, and reasoning, so retrieval at step is conditioned on the history of earlier steps, . Evaluation targets the final evidence, and a shared protocol is still missing.
Each paradigm can be applied at page level or at document level, giving a space of evaluation settings. For document-level ranking, a benchmark can also evaluate attribution, i.e., whether the pages surfaced for a retrieved document include a relevant page. Existing benchmarks mostly populate a single cell of this space, i.e., independent-item evaluation at page level (§ 4.3).
Evaluating Existing VDR Benchmarks
A benchmark’s ability to faithfully evaluate a task depends not merely on whether its input/output format matches the task’s specification, but on whether the process by which it was constructed actually reflects the task’s underlying requirements. The comparison of SQuAD2.0 [35] and Natural Questions [23] in [53] illustrates this point: SQuAD2.0 contains questions written from passages that already contain the answer, whereas Natural Questions is based on real search queries posed before the answer is known. Consequently, despite sharing the same (question, passage, answer) format, the datasets represent different tasks, reflected in differences in lexical diversity, question specificity, and syntactic variety.
This distinction generalizes into two broadly different paths for benchmark construction. The first repurposes an existing dataset built for a different task, by reformatting it to match the input/output signature of the target task. This path is inexpensive, which is why many benchmarks accumulate around a convenient format rather than a carefully specified task, but it offers no guarantee that the repurposed dataset reflects the target task’s actual constraints, query distribution, or notion of relevance. The resulting benchmark may end up testing whether a model fits the idiosyncrasies of the source dataset rather than whether it solves the intended task.
The second path works top-down from an explicit task definition. The dataset is developed by enumerating the task’s requirements first: who the intended users are, what constitutes a query, what relevance judgments should capture. Then, the collection and/or annotation process is guided to meet the task definition. This is the same discipline that motivates datasheets for datasets [12] and data statements [2], which require a dataset’s creators to document its motivation, the situation in which the underlying content was produced, and its collection and annotation process, precisely so that a downstream user can judge whether the dataset actually matches the task they intend to evaluate.
We evaluate 42 VDR-relevant benchmarks2 along three groups of dimensions: (i) document source and characteristics (§ 4.1), (ii) query and relevance annotation processes (§ 4.2), and (iii) annotation results, covering both page- and document-level retrieval (§ 4.3). Table 1 details the 8 specific criteria under the three groups.
| Sources and Structural Traceability (SS&T) |
| 1 The released data contain only isolated page images, images, figures, tables, page regions, or reassembled text blocks, and neither their source documents nor page order can be recovered. |
| 2 The document to which each page belongs can be inferred, but document identifiers, page identifiers, or page order are incomplete. Pages have been extracted, removed, translated, re-rendered, or reorganized without adequate documentation. |
| 3 Complete document boundaries are preserved, documents and pages have stable identifiers, and page order is explicit. The benchmark documents operations such as re-rendering, optical character recognition, translation, deduplication, filtering, and cropping. |
| Corpus Size (CS): total number of pages across all documents |
| 1 The underlying corpus contains fewer than 10,000 pages in total. |
| 2 The underlying corpus contains at least 10,000 pages in total. |
| Document Length (DL): number of pages per document |
| 1 Corpus mainly consists of single-page documents. |
| 2 The corpus consists mostly, if not entirely, of multi-page documents. |
| Annotation Process (AP) |
| 1 Before writing the query, the author sees a designated target page or a preselected fine-grained target evidence and constructs the query from it. |
| 2 The query is constructed from one or more complete source documents, but the author is not directed to a particular target page or local region. The query is more independent than a question written from a designated page, yet it remains driven by candidate documents. |
| 3 The query author can see only intentionally obscured corpus information, such as titles, metadata, abstracts, document summaries, descriptions of tables of contents, or the topic of the document collection, and cannot inspect specific evidence pages. Relevant items are searched for and annotated in the underlying document collection only after the query has been generated. |
| 4 The query originates from real search logs, retrieval-system usage records, or other real information needs independently formulated by a human, and it predates relevance annotation. No target-corpus information is provided when the query is posed; relevance is annotated against the corpus only after the query has been issued. |
| Coverage of Relevant Items (RC) |
| 1 No attempt is made to ensure all potentially relevant items are labeled. |
| 2 At least one of the following holds: (1) discriminative queries are designed to have a single gold candidate; or (2) checking is restricted to a limited subset, e.g., pages in the same document as the gold page. |
| 3 A rule or model automatically examines every candidate without human verification.† |
| 4 Human verification is incorporated into the workflow. At least one of the following conditions holds: (1) one or more retrieval systems filter out low-relevance items and return top-ranked candidates for human judgment; (2) after a rule or model screens all candidates, humans verify a pre-specified random or stratified sample, estimate error rates, and determine whether the results are reliable; or (3) humans inspect every page of all documents in the corpus. |
| Relevant Items per Query (RQ) |
| 1 Each query has only single annotated relevant item. |
| 2 The data allow both single-positive and multi-positive queries, but fewer than 20% of queries have ≥2 relevant items. |
| 3 At least 20% but fewer than 50% of queries have ≥2 directly annotated relevant items, and the full-dataset mean is <2 relevant items per query. |
| 4 At least 50% of queries have ≥2 directly annotated relevant items, or the full-dataset mean is ≥2 relevant items per query. |
| Relevance Level (RL) |
| 1 The data define binary relevant and non-relevant labels. |
| 2 The annotation scheme recognizes partial relevance between full relevance and non-relevance, yielding three ordered relevance grades. |
| 3 The data provide at least four ordered relevance grades, each with an operational definition and decision examples. |
| Types of Ranking Supported (RS) |
| 1 The benchmark evaluates page-level retrieval, including collections in which single-page content is treated as a document. Each page serves as an independent indexing unit, and no cross-page context modeling is required. Additional cases include document-level retrieval constructed directly from page-level annotations. |
| 2 The benchmark supports document-level retrieval, in which queries are built upon the topical information of large portions of the document. It also covers contextual page-level retrieval, where accurate retrieval requires surrounding context rather than isolated pages. |
Table 1. Dimensions for assessing VDR benchmark readiness. The framework consolidates considerations commonly encountered during benchmark development and evaluation, providing a structured basis for comparing benchmark characteristics and design choices.
Document Sources and Characteristics
Over time, the sources and domain coverage of VDR benchmarks have broadened. Early benchmarks often repurposed existing QA datasets or relied on relatively homogeneous collections [4, 10, 41, 54], whereas later benchmarks incorporate a wider range of professional materials from multiple disciplines [3, 34, 36, 51]. Multilingual coverage has also expanded beyond English to European [13, 22, 29, 31, 36], East Asian [6, 13, 22, 24, 36], Southeast Asian [13, 22, 48], and other languages [13, 22, 36].
Document structure has received increasing attention, with evaluation extending from isolated page images or cropped elements to intact multi-page documents. Early benchmarks often treated figures, tables, or other extracted elements as standalone retrieval units [10, 25, 47, 54], which is useful for evaluating understanding of specific content types but does not fully capture a model’s ability to integrate heterogeneous information within a document. There are also datasets consist of self-contained single-image documents, such as advertisements [26], handwritten pages [32], and invoices [20]. More recent benchmarks increasingly preserve complete multi-page documents, allowing retrieval to be evaluated in a more realistic structural context [3, 18, 29].
Corpus size and document length. While paper-oriented collections average around 20 pages [18, 36, 38], long-document benchmarks can be substantially longer; for example, FinMARGBench [51] reports an average of 133.5 pages per document. Corpus size varies even more widely, which can be measured either in number of pages or number of documents. Candidate pools can reach tens or hundreds of thousands when pages or extracted elements are treated as retrieval units [30, 47]. Others contain relatively few but much longer, carefully curated documents. Such collections can still create challenging retrieval settings because relevant evidence may span multiple pages or documents, as explicitly represented in KoViDoRe [6] and MADQA [3]. For consistency, corpus size is measured by the number of pages in our scoring system.
Visual dependence. Earlier datasets often assumed visual relevance indirectly through the input format or retrieval target, for example by retrieving figures, charts, or tables [10, 47, 54]. More recent benchmarks increasingly annotate, control, or measure the modality of required evidence. ViDoRe V3 [29] annotates evidence modality and provides region-level grounding in addition to page relevance. M4DocBench [9] provides document-, page-, and region-level evidence annotations and identifies 112 of its 158 instances as multimodal. MADQA [3] reports the proportion of visually dependent instances that requires understanding layout, tables, or other document artifacts. Overall, benchmark design has progressed from simply using document images, to retrieving explicitly visual content, and further toward directly characterizing the visual evidence required for successful retrieval.
However, we argue that benchmarks for generic visual document retrieval should not overemphasize non-textual visual evidence at the expense of textual content. In real-world documents, text remains a primary carrier of semantic information, and effective retrieval should leverage both textual and visual signals. Moreover, under our framework, the document page serves as the fundamental retrieval and indexing unit, with its layout, typography, and structural organization also contributing to the evidence available for retrieval. Therefore, we do not require every test case to involve non-textual visual evidence, nor do we score visual dependence. Instead, we recommend annotating evidence types (e.g., text, table, chart, and layout) to enable stratified evaluation.
Source of Queries and Annotation Process
We now focus on how query-item relevance is established and annotated. We first record the benchmark’s annotation direction: query-to-source, where existing user queries (e.g., from search logs) are matched to relevant corpus items; or source-to-query, where queries are constructed by annotators or LLMs from selected corpus content to simulate user information needs. We further record the coverage of relevant items, indicating whether the benchmark attempts to identify all relevant pages and documents for each query across the corpus.
Query-to-source vs Source-to-query. Among all evaluated benchmarks, only TaR-ViR [25] adopts a query-to-source annotation direction. Specifically, it reuses queries from the textual retrieval dataset NQ-TABLES [16], whose queries originate from the Google search logs used in Natural Questions [23], and identifies relevant items for those queries within a newly constructed multimodal corpus.
All remaining benchmarks follow a source-to-query paradigm, in which human annotators or LLMs generate queries after inspecting the corpus content. This process introduces two potential biases. First, the resulting queries may not reflect how users express information needs in practice, where queries are often underspecified, informal, and shaped by varying levels of domain knowledge [15]. As a result, benchmark performance may not accurately predict real-world retrieval effectiveness. Second, because source-to-query construction begins with a known source, it guarantees the existence of supporting evidence by design. This may underrepresent realistic scenarios in which the corpus contains only partially relevant evidence or no relevant item at all.
Although source-to-query annotation is widely used because real search queries are difficult to obtain, some benchmarks explicitly seek to reduce its associated bias. For example, ViDoRe V2 [31] and V3 [29] recognize that annotators often have access to corpus content when generating queries, unlike real users. To mitigate this discrepancy, they restrict annotators to document summaries and metadata rather than the full source content. Text and image descriptions are first extracted from PDFs, summarized by an LLM, and then clustered through embedding and dimensionality reduction. Selected single-section and cross-section summaries are provided to annotators as the basis for query creation. ViDoRe V3 also supplements human-written queries with LLM-generated synthetic queries. Future benchmarks could further reduce source-induced bias by adopting query collection procedures that more closely reflect real-world information-seeking conditions.
Coverage of relevant items. We next examine whether benchmarks attempt to identify all relevant pages or documents for each query. Under our criteria, only a small number explicitly assess relevance across the entire corpus and therefore support coverage of all relevant items. For example, REAL-MM-RAG [46] applies an MLLM to every page in the corpus to determine whether it contains sufficient information to answer a query. Queries are retained only when no page other than the annotated target is identified as answer-bearing, yielding a dataset in which each query is associated with a single verified relevant page. The benchmark further samples top-ranked retrieval results for human review to detect potential false-negative annotations.
More scalable approaches approximate corpus-wide assessment by first constructing a bounded candidate set through retrieval, clustering, knowledge-graph traversal, or model-based pre-screening, followed by human or model-assisted relevance annotation. Examples include DocViRe [52], ViDoRe V3 [29], PaperScope [49], and MultiDocR [19]. MultiDocR, for instance, uses a retrieval-reranking-annotation pipeline: a retriever generates candidate pages, a reranker refines the set, and an MLLM assigns graded relevance labels. Such approaches improve scalability and reduce the likelihood of missing relevant items.
Beyond explicit corpus-wide relevance assessment, several benchmarks adopt indirect strategies to reduce annotation ambiguity and improve label reliability. A common approach is to make queries sufficiently discriminative so that the intended target is unlikely to be confused with other candidates. Some benchmarks achieve this by incorporating multiple entities or constraints into each query [20, 26], while others rely on LLMs to enforce specificity. For example, IRPAPERS [38] prompts the model to generate questions that are answerable by a unique page and to provide supporting justification, whereas SeaDoc [48] filters generated queries based on their relevance and exclusivity to the designated target.
Other benchmarks apply post-hoc validation after query construction. IRPAPERS [38], for instance, evaluates query discriminativeness by replacing the gold document with a hard negative and measuring the resulting change in downstream RAG performance. Although performance degradation suggests that the annotated document contains more useful evidence, it does not eliminate the possibility of false negatives, since the alternative document may still contain partially relevant information. KoViDoRe [6] instead uses a consistency-based filtering strategy, retaining a retrieved page as relevant only when it belongs to the same document from which the query was generated. While this improves annotation consistency, it may overlook relevant pages from other documents.
Cardinality of results and ranking
The third group of dimensions examines the number of annotated relevant items per query, whether relevance level is binary or graded, and the types of ranking supported: page-level, document-level, or both. Together, these properties determine which aspects of our formulation a benchmark can evaluate.
To provide a clearer overview of how existing benchmarks support different retrieval settings, we present a Venn diagram in Figure 2, organized by relevance annotation and ranking evaluation.

Figure 2. Categorization of VDR benchmarks by relevance annotation and ranking evaluation. Colored regions denote different retrieval settings, while overlaps indicate benchmarks supporting multiple settings.
Number of relevant items per query. The number of relevant items per query has evolved from predominantly single-pair supervision toward multi-positive and evidence-set annotations. Many benchmarks inherit a single answer-bearing page, figure, table, or image from existing QA or caption-pairing datasets [4, 10, 14, 47, 54], or generate queries from preselected target items [1, 20, 24, 38, 46]. Benchmarks annotating multiple relevant items can be broadly divided into two categories according to how relevance is defined and evaluated, see Appendix § 8.
The first category follows the traditional IR setting, in which each query-item pair is annotated independently without explicitly modeling relationships among relevant items. Such annotations are most appropriate for the independent-item retrieval paradigm discussed in § 3.3. Representative benchmarks include DocViRe [52] and LitTraceQA [28] for document-level retrieval, and ViDoRe V2/V3 [29, 31] and MultiDocR [19] for page-level retrieval.3 This setting aligns with conventional search scenarios, such as web search, where multiple independently relevant items can serve as alternative retrieval results.
The second category focuses on complementary evidence retrieval. Benchmarks hence assess whether the retrieved results collectively provide sufficient evidence to answer the query. For example, Double-Bench [36] defines multi-hop retrieval success as retrieving at least one supporting page for every reasoning hop within the top- results, measured by Hit@. Although this setting imposes additional requirements on retrieval success, it still involves annotating multiple relevant items per query, typically spanning at least two distinct pages or documents. We therefore include it within the broader category of benchmarks supporting multiple relevant items per query.
These categories reflect two distinct retrieval paradigms. However, both fit naturally within the retrieval framework introduced in § 3.2, which returns a ranked list of relevant items. Their primary distinction lies in the evaluation criterion: retrieved items are judged either independently by individual relevance or collectively according to the completeness of the evidence they provide.
Binary or graded relevance. Most VDR benchmarks adopt binary judgments, while only few very recent benchmarks provide graded relevance annotations. This is partly attributable to the prevalence of single-positive annotations. Among benchmarks with graded relevance, ViDoRe V3 [29] and KoViDoRe [6] distinguish among irrelevant, key-relevant, and sufficient-relevant pages. Nayana-IR [22] also adopts a three-level scheme, assigning the highest relevance to the exact target page, partial relevance to structurally related pages from the same parent document, and zero relevance otherwise. MultiDocR [19] introduces a finer-grained five-level scheme, distinguishing fully, partially relevant, inference-relevant, topically relevant, and irrelevant pages. Collectively, these schemes move beyond binary relevance judgments to characterize how directly and sufficiently each item addresses a query.
We advocate a flexible annotation setting that allows a variable number of relevant items per query, with graded judgments capturing their different degrees of relevance. Such a setting avoids assuming a fixed number of relevant results or treating all relevant items as equally sufficient to satisfy the query.
Page- or document-level ranking. Page-level retrieval remains dominant in VDR benchmarks. A common special case treats self-contained single-page images as nominal “documents” [4, 10, 14, 17, 20, 22, 25, 26, 32, 47]. Although such settings technically support document-level retrieval, they do not evaluate selection among intact multi-page documents. Furthermore, most benchmarks based on multi-page documents treat pages as independent retrieval units rather than modeling cross-page dependencies. As a result, they offer limited assessment of document-level retrieval and reasoning over information distributed across multiple pages.
An emerging direction that addresses this limitation is contextual retrieval, in which information from intermediate pages is required to identify the target page, even when those pages provide little direct evidence for answering the query. CMDR [40] introduces this setting for intra-document retrieval, while VisDocAgentBench [18] extends it to VDR. In VisDocAgentBench, queries provide an initial semantic clue without directly specifying the target, requiring the retriever to navigate through supporting pages linked by latent semantic relationships. Although these pages may not contribute directly to answer generation, they act as essential intermediaries for locating the final evidence. The benchmark is explicitly designed for iterative retrieval, making it particularly suitable for agentic approaches that interleave planning, retrieval, and reasoning across multiple steps rather than relying on a single retrieval pass.
Contextual retrieval differs from conventional multi-hop evidence retrieval. In multi-hop retrieval, multiple pages jointly provide the evidence needed to answer a query. In contextual retrieval, by contrast, intermediate pages primarily help locate the target evidence. As noted in [40], many existing multi-hop queries explicitly mention concepts associated with individual evidence pages, allowing those pages to be retrieved directly from the query without exploiting dependencies between pages. Consequently, strong performance on this parallel multi-hop retrieval does not necessarily indicate contextual retrieval capability.
For document-level retrieval, native document-level supervision remains relatively uncommon, with some document-level evaluation protocols derived directly from page-level annotations. For example, MMEB-V2 [33] infers document-level relevance from page-level labels, even though its queries target localized page content rather than document-level information. More direct document-level formulations include whole-document topical retrieval in DocViRe [52] and research-paper discovery in PaperScope [49].
Beyond page- and document-level ranking, recent benchmarks have begun to incorporate finer-grained evidence localization. Vi-DoRe V3 [29] augments page relevance with region grounding, M4DocBench [9] links relevant documents and pages to supporting layouts, LitTraceQA [28] associates pages with evidence quotations, and BenthamQA and HW-SQuAD [32] combine page retrieval with answer-snippet localization.
Overall, existing VDR benchmarks remain dominated by independent page-level ranking. Native document-level supervision and fine-grained evidence localization are comparatively rare, while contextual retrieval highlights the need to evaluate dependencies across pages rather than only the direct relevance between individual pages and queries.
Benchmark Readiness Scores and Findings
We now introduce a structured framework for characterizing and comparing VDR benchmarks. The framework is not intended to define a single notion of an "ideal" benchmark, and the resulting
scores should not be interpreted as evidence of universal superiority across retrieval scenarios. Rather, its goal is to provide a systematic lens through which benchmark design choices can be examined. It serves two complementary purposes. First, it brings together considerations that are often discussed in isolation, including corpus construction, annotation methodology, relevance granularity,
and application alignment. Second, it makes explicit the trade-offs inherent in benchmark development. By synthesizing practices observed across existing VDR benchmarks, the framework provides a common vocabulary for describing benchmark characteristics, identifying potential gaps or biases, and facilitating more transparent comparisons. Accordingly, the resulting scores should be viewed as a structured summary of benchmark properties rather than as an absolute ranking, recognizing that different research objectives and deployment settings may prioritize different dimensions.
Using this framework, we evaluate 42 benchmarks against the eight criteria defined in Table 1, which operationalize the dimensions introduced in §4. Each criterion is scored on an ordinal scale comprising two to four levels. Initial assessments are generated by Claude Opus 5.5 and supported by evidence from the corresponding benchmark publication (e.g., direct quotations or section references) or from inspection of the benchmark annotation files. Two authors subsequently review all assessments independently and reach consensus on the final scores. The resulting evaluations are reported in Table 5, where benchmarks are ranked by their total score, computed as the unweighted sum of the eight criteria.
| Benchmark | SS&T | CS | DL | AP | RC | RQ | RL | RS | Total Score | Retrieval Paradigm |
| ViDoRe V3 [29] | 3 | 2 | 2 | 3 | 4 | 4 | 2 | 1 | 21 | Items |
| Double-Bench [36] | 3 | 2 | 2 | 2 | 4 | 4 | 1 | 2 | 20 | Set |
| MultiDocR [19] | 3 | 2 | 2 | 1 | 3 | 4 | 3 | 1 | 19 | Items |
| KoViDoRe [6] | 3 | 1 | 2 | 3 | 3 | 4 | 2 | 1 | 19 | Items |
| MMDocRAG [8] | 3 | 2 | 2 | 1 | 4 | 4 | 1 | 1 | 18 | Items |
| DocViRe [52] | 3 | 2 | 2 | 2 | 4 | 1 | 2 | 18 | Items | |
| PaperScope [49] | 3 | 2 | 2 | 3 | 4 | 1 | 2 | 18 | Iterative | |
| LitTraceQA [28] | 3 | 2 | 2 | 2 | 2 | 4 | 1 | 1 | 17 | Items |
| FinMRAGBench [51] | 3 | 2 | 2 | 1 | 2 | 4 | 1 | 1 | 16 | Iterative |
| VisDocAgentBench [18] | 3 | 1 | 2 | 2 | 4 | 1 | 1 | 2 | 16 | Iterative |
| ViDoRe V2 [31] | 3 | 1 | 2 | 3 | 1 | 4 | 1 | 1 | 16 | Items |
| M4DocBench [9] | 3 | 1 | 2 | 1 | 2 | 4 | 1 | 1 | 15 | Iterative |
| M3DocVQA [5] | 3 | 2 | 2 | 1 | 1 | 4 | 1 | 1 | 15 | Set |
| MADQA [3] | 3 | 2 | 2 | 2 | 2 | 2 | 1 | 1 | 15 | Iterative |
| UniDoc-Bench [34] | 3 | 2 | 2 | 1 | 1 | 4 | 1 | 1 | 15 | Items |
| ViDoSeek [45] | 3 | 1 | 2 | 1 | 3 | 3 | 1 | 1 | 15 | Items |
| SDS KoPub VDR [24] | 3 | 2 | 2 | 1 | 3 | 1 | 1 | 1 | 14 | Items |
| REAL-MM-RAG [46] | 3 | 1 | 2 | 1 | 4 | 1 | 1 | 1 | 14 | Items |
| DocCVQA [42] | 3 | 1 | 1 | 1 | 2 | 4 | 1 | 1 | 14 | Items |
| TaR-ViR [25] | 1 | 2 | 1 | 4 | 1 | 2 | 1 | 1 | 13 | Items |
| Nayana-IR [22] | 2 | 2 | 1 | 1 | 2 | 2 | 2 | 1 | 13 | Items |
| VisDoMBench [39] | 3 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 13 | Items |
| ViMDoc [21] | 3 | 2 | 2 | 1 | 1 | 2 | 1 | 1 | 13 | Items |
| CHaystack [26] | 3 | 2 | 1 | 1 | 2 | 1 | 1 | 1 | 12 | Items |
| SeaDoc [48] | 3 | 1 | 2 | 1 | 2 | 1 | 1 | 1 | 12 | Items |
| MMEB-V2 [33] | 2 | 2 | 1 | 1 | 2 | 2 | 1 | 1 | 12 | Items |
| SlideVQA-open [30] | 2 | 2 | 2 | 1 | 1 | 2 | 1 | 1 | 12 | Items |
| LëtzCross [1] | 3 | 1 | 2 | 1 | 2 | 1 | 1 | 1 | 12 | Items |
| IRPAPERS [38] | 3 | 1 | 2 | 1 | 2 | 1 | 1 | 1 | 12 | Items |
| Invoice Haystack [20] | 3 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 11 | Items |
| InfoHaystack [4] | 1 | 1 | 1 | 1 | 4 | 1 | 1 | 1 | 11 | Items |
| DocHaystack [4] | 1 | 1 | 1 | 1 | 4 | 1 | 1 | 1 | 11 | Items |
| DocMMIR [27] | 1 | 2 | 1 | 2 | 1 | 1 | 1 | 2 | 11 | Items |
| OpenDocVQA [41] | 1 | 2 | 1 | 1 | 1 | 2 | 1 | 1 | 10 | Items |
| VisRAG [54] | 1 | 2 | 1 | 1 | 1 | 2 | 1 | 1 | 10 | Items |
| BenthamQA [32] | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 9 | Items |
| SciMMIR [47] | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 9 | Items |
| JinaVDR [13] | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 9 | Items |
| DocQIR [17] | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 8 | Items |
| NL-DIR [14] | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 8 | Items |
| HW-SQuAD [32] | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 8 | Items |
| ViDoRe V1 [10] | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 8 | Items |
Table 5. Characterization of 42 VDR benchmarks through the proposed readiness dimensions. Scores should be interpreted as indicators of benchmark characteristics and design trade-offs rather than definitive rankings of benchmark quality.
†When information is insufficient for precise scoring, the dataset receives the lowest score among cases consistent with its paper’s description.
Recommendations by granularity × retrieval paradigm. As discussed in §3.2 and §3.3, VDR tasks vary along two dimensions: retrieval granularity (page-level or document-level) and retrieval paradigm (independent-item, evidence-set, or iterative retrieval). We organize existing benchmarks into six evaluation settings defined by the combinations of these dimensions and provide benchmark recommendations for each setting in Table 2.
| Granularity | Paradigm | Benchmarks |
| Page | Items | ViDoRe V3 (21), MultiDocR (19), KoViDoRe (19), MMDocRAG (18), LitTraceQA (17), ViDoRe V2 (16) |
| Page | Set | Double-Bench (20) |
| Page | Iterative | FinMRAGBench (16), VisDocAgentBench (16), M4DocBench (15), MADQA (15) |
| Document | Items | DocViRe (18), LitTraceQA (17) |
| Document | Set | M3DocVQA (15) |
| Document | Iterative | PaperScope (18), M4DocBench (15) |
Table 2. Recommended VDR benchmarks for six evaluation settings defined by retrieval granularity and paradigm. Scores (out of 24) are shown in parentheses.
Existing benchmarks are predominantly concentrated in page-level independent-item retrieval. For this setting, we recommend several of the highest-scoring benchmarks: ViDoRe V3 [29], MultiDocR [19], KoViDoRe [6], MMDocRAG [8], LitTraceQA [28], and ViDoRe V2 [31]. These benchmarks generally feature more comprehensive relevance annotations, with several supporting multiple relevant items per query. Among them, ViDoRe V3, MultiDocR, and KoViDoRe further provide graded relevance judgments, enabling more nuanced evaluation of retrieval quality.
The remaining five settings are comparatively underrepresented, reflecting the relatively recent interest in document-level, evidence-set, and iterative retrieval within VDR research. We therefore include nearly all benchmarks identified for these settings. For page-level evidence-set retrieval, Double-Bench [36] offers a well-designed evaluation protocol that requires retrieving complementary evidence across multiple reasoning hops. For page-level iterative retrieval, FinMRAGBench [51], VisDocAgentBench [18], M4DocBench [9], and MADQA [3] provide diverse evaluation settings that differ in their retrieval workflows and evidence-acquisition requirements.
Document-level retrieval benchmarks are even more limited. For independent-item retrieval, DocViRe [52] provides native document-level supervision through whole-document topical relevance. LitTraceQA derives document-level relevance from page-level evidence annotations. The latter also supports document-level evidence-set evaluation, alongside M3DocVQA [5], whose document-level evidence sets can be derived from localized annotations. In particular, M3DocVQA can be evaluated using set-level metrics such as All@K, as adopted in [37]. Finally, PaperScope [49] and M4DocBench provide evaluation environments for document-level iterative retrieval.
Overall, the recommended benchmarks cover all six evaluation settings, although their availability and annotation quality vary substantially. In particular, the scarcity of benchmarks beyond page-level independent-item retrieval highlights the need for more comprehensive evaluation resources supporting diverse retrieval granularities and paradigms. Beyond these two dimensions, we also summarize benchmarks featuring non-English queries or documents in Appendix § 9, providing additional guidance for researchers interested in evaluating retrieval performance across different languages.
Conclusion
This paper argues that visual document retrieval should be viewed as a family of retrieval tasks. We explicitly distinguish retrieval granularity (page versus document) from retrieval paradigm, i.e., independent-item, evidence-set, and iterative retrieval. We assessed 42 existing benchmarks across eight dimensions covering corpus construction, annotation methodology, relevance coverage, and evaluation design.
Our analysis reveals a clear pattern: benchmark development has focused primarily on independent page-level retrieval, while realistic query provenance, corpus-wide relevance assessment, graded relevance judgments, native whole-document supervision, evidence-set evaluation, and iterative retrieval remain comparatively rare. We believe the most important challenge for the next generation of VDR benchmarks is not increasing scale alone. Future benchmarks should move closer to authentic user information needs, explicitly characterize the modality and completeness of supporting evidence, and evaluate retrieval under settings where relevant information may be distributed across multiple pages, documents, and retrieval steps.
Applications and Benchmarks
Applications
We identify four application classes that VDR supports, and for each, justify why the underlying real-world need genuinely requires multi-page, visually-grounded retrieval, and which granularity(ies) it depends on. We also briefly justify why conventional text passage retrieval may not be the most suitable choice to support such applications. These justifications also parameterize the benchmark dimensions to be discussed shortly; Table 3 summarizes the requirements each application places on a benchmark.
| Application | Granularity | Paradigm | Selection rule | Benchmarks |
| Enterprise and knowledge-base search | Page; document for holistic requests | Independent items | SS=3, CS=2, DL=2, RC≥3, RL≥2 | ViDoRe V3, MultiDocR |
| RAG over visually-rich documents | Page; Document | All three paradigms | DL=2, RC≥3, RQ≥2 | ViDoRe V3, Double-Bench, MultiDocR, MMDocRAG, DocViRe, PaperScope |
| Financial and legal analysis | Page; Document | All three paradigms | SS=3, DL=2, RC≥2, domain-specific | ViDoRe V3, MultiDocR, DocViRe, FinMRAGBench, M4DocBench |
| Multi-document, multi-hop QA | Both, with cross-document aggregation | Evidence sets; iterative | DL=2, RC≥3, RS=2 | Double-Bench, PaperScope, VisDocAgentBench |
Table 3. Benchmark requirements for four representative applications, defined using the criteria in Table 1, and benchmarks satisfying these requirements based on the scores in Table 5.
Enterprise and knowledge-base search. Organizations store institutional knowledge in PDFs, slide decks, and scanned reports whose relevant content is frequently non-textual: a pricing table, an architecture diagram, or a chart of quarterly results. Many such reports are long documents with tens of even hundreds of pages. A user searching an internal knowledge base needs the exact page containing the answer, not a document-level pointer that still requires manual skimming. This application is primarily page-level: it requires large, heterogeneous document corpora with page-granular gold labels, and its practical value is proportional to how large and visually diverse the underlying corpus is. A complementary document-level need arises for holistic requests such as “find the deck that introduces product X”, the contract that is a lease, or the paper that surveys a topic (literature discovery, e-discovery). Here no single page need be relevant by itself, so page scores can only approximate document relevance (§ 3.2).
RAG over visually-rich documents. Retrieval-Augmented Generation (RAG) systems that answer questions over PDF corpora fail silently when their retriever discards visual evidence. A chart, a signature block, or a table cell could be the only place an answer lives, and no amount of generation quality compensates for a retriever that never surfaces it [54]. In practice this application needs both granularities jointly. Page-level retrieval supplies the generator with the specific evidence pages. Document-level retrieval determines which whole documents are even worth pulling into the retrieval pool for answer generation, which matters once a corpus grows beyond what fits in a single context window [5]. Without document-level filtering, page-level retrieval alone may not come with the necessary context for citation-backed generation, critical for enterprise corpora.
Financial and legal document analysis. Financial filings, contracts, and regulatory reports are simultaneously long, numerically dense, and high-stakes, in the sense that an incorrect or ungrounded answer has direct legal or financial consequences. Practitioners in this domain routinely need two distinct capabilities: which filings or contracts are relevant to a query, and where, within a relevant document, the specific clause, number, or table appears. The former is document-level ranking, since an analyst may need to triage hundreds of candidate filings. The latter is page-level ranking, since citing the correct source location is often a regulatory or professional requirement. Text-only retrieval is a particularly poor fit here because financial and legal documents rely heavily on tabular and structured layout to convey numerical relationships.
Multi-document, multi-hop question answering. Some information needs cannot be answered from a single page or even a single document. An example use case is comparing a metric across several companies’ annual reports, or reconciling a claim made in one document against supporting evidence in another. This application requires both granularities combined with cross-document aggregation: document-level ranking narrows a large corpus down to the relevant subset of documents, and page-level ranking then identifies the specific evidence within each. It is probably the most demanding application in terms of retrieval infrastructure.
Recommendations Based on Applications
Table 3 provides illustrative benchmark recommendations for different applications based on the scores in Table 5. The requirements are not prescriptive, and researchers may adjust the thresholds according to their specific priorities and evaluation needs.
Relationship between VDR and RAG
This section explains how the evolving requirements of RAG have shaped the three VDR retrieval paradigms introduced in § 3.3.
RAG enhances large language models (LLMs) by incorporating external information into the answer generation process. In a typical RAG pipeline, a retriever identifies relevant content from a knowledge corpus, and a generator uses the retrieved evidence to formulate a response. Because the quality of the retrieved content directly influences the relevance, completeness, and factual grounding of generated answers, retrieval is a critical component of RAG. From a page-centric indexing and search perspective, RAG can be viewed as a two-stage process comprising visual document retrieval to identify relevant pages, followed by answer generation.
Indeed, with the development of RAG, VDR has increasingly extended beyond retrieving individually relevant pages toward acquiring multiple pieces of evidence to support complex question answering. However, such multi-page retrieval can involve different objectives, leading to distinct retrieval and evaluation settings. Two key distinctions emerge: (i) whether relevance is assessed at the individual-item or evidence-set level, and (ii) whether retrieval is conducted in a single pass or through an iterative process. Based on these distinctions, we discuss three retrieval paradigms commonly considered in existing VDR benchmarks.
Conventional information retrieval (IR), where each retrieved item is evaluated independently according to its relevance to the query. Even when multiple pages or documents are relevant to the same query, their relevance labels are assigned individually, and retrieval is evaluated based on the resulting ranked list. Standard IR metrics are therefore applicable. For example, nDCG@K
† en, ar, bn, de, es, fr, hi, hu, id, it, ja, ko, my, nl, pt, ru, th, ur, vi, zh
‡ en, ar, bn, de, es, fr, gu, hi, it, ja, kn, ko, ml, mr, or, pa, ru, sa, ta, te, th, zh
is commonly used when graded relevance judgments are available, whereas Recall@, MAP, and MRR are widely adopted when relevance is treated as binary.
Complementary evidence across multiple pages. The second paradigm is motivated by RAG scenarios where answering a complex query requires combining complementary evidence distributed across multiple pages. The objective is to recover a sufficient set of evidence that collectively supports answer generation. This requirement has motivated the introduction of multi-hop queries and corresponding set-level evaluation protocols. For example, M3DocVQA [5] provides query-level evidence-set annotations, allowing evaluation through metrics such as All@, as adopted in [37], where retrieval succeeds only when the complete annotated evidence set is covered by the top- results. Notably, this metric was not part of M3DocVQA’s original evaluation protocol. Double-Bench [36] further accounts for the substitutability of evidence by defining multi-hop retrieval success as retrieving at least one supporting page for every reasoning hop within the top- results, measured by Hit@. Nevertheless, the presence of multi-hop queries does not necessarily imply set-level retrieval evaluation. Several benchmarks, including KoViDoRe [6], UniDoc-Bench [34], and ViDoRe V3 [29], incorporate multi-hop questions but continue to adopt conventional item-wise IR protocols, although their definitions of multi-hop queries differ.
Iterative retrieval. The recent emergence of agentic RAG has introduced a third paradigm: iterative retrieval. Unlike conventional RAG pipelines that typically retrieve evidence once before answer generation, agentic systems can interleave planning, retrieval, evidence inspection, and reasoning across multiple steps. Accordingly, evidence acquisition becomes a sequential and history-dependent process, where observations from earlier steps inform subsequent retrieval decisions and determine when sufficient evidence has been collected. Several VDR benchmarks explore this paradigm from different perspectives. VisDocAgentBench [18] introduces latent semantic bridges that must be resolved to locate the final target; MADQA [3] emphasizes acquiring minimal grounded evidence from heterogeneous collections; M4DocBench [9] incorporates conversational dependencies and annotated evidence chains; FinMRAGBench [51] focuses on iterative financial analysis; and PaperScope [49] provides a tool-based environment for scientific information seeking. Additionally, these benchmarks do not share a unified evaluation protocol. For instance, VisDocAgentBench evaluates final-target retrieval using Recall and MRR, MADQA measures the quality of retrieved citation sets using F1, and PaperScope separately reports direct-retrieval Recall for Topic Induction and downstream reasoning accuracy.
Overall, RAG has broadened the objectives and evaluation settings of VDR, from ranking independently relevant items to recovering complementary evidence sets and supporting iterative evidence acquisition. These distinctions highlight the importance of selecting benchmarks that align with the intended retrieval objective, considering not only the document domains and corpus characteristics but also whether relevance is defined at the item or evidence-set level, whether relevant items are interchangeable or complementary, and whether evidence is acquired through single-shot or iterative retrieval.
Benchmarks with Non-English Settings
Table Table 4 summarizes VDR benchmarks featuring non-English queries or documents. Some benchmarks focus on individual non-English languages without multilingual or cross-lingual evaluation [6, 24, 26]. The remaining benchmarks support diverse language settings, including translated queries [1, 17, 29, 31], multilingual document collections [48], and language-specific subsets [13, 22]. Researchers can select benchmarks according to their target languages and evaluation settings.
| Benchmarks | Query | Document |
| SDS KoPub VDR [24] | ko | ko |
| KoViDoRe [6] | ko | ko |
| CHaystack [26] | zh | zh |
| ViDoRe V1 [10] | en, fr | en, fr |
| DocQIR [17] | en, hi, ja, ko, pt | en |
| JinaVDR [13] | en & 19 langs.† | en & 19 langs.† |
| LëtzCross [1] | en, de, fr, lb | lb |
| MMEB-V2 [33] | en, de, es, fr | en, fr |
| SeaDoc [48] | en | th, vi, ms, lo |
| Nayana-IR [22] | en & 21 langs.‡ | en & 21 langs.‡ |
| ViDoRe V2 [31] | en, de, es, fr | en |
| M4DocBench [9] | en, zh | en, zh |
| Double-Bench [36] | en, ar, es, fr, ja, zh | en, ar, es, fr, ja, zh |
| ViDoRe V3 [29] | en, de, es, fr, it, pt | en, fr |
Table 4. Benchmarks with non-English queries/documents. Languages are represented by ISO 639-1 codes.
References
- [1]Omar El Bachyr, Fred Philippy, Laura Maria Bernardy, Saad Ezzini, Jacques Klein, and Tegawendé F. Bissyandé. 2026. LetzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents. arXiv:2608.21714arxiv.org/abs/2608.21714
- [2]Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Trans. Assoc. Comput. Linguistics 6 (2018), 587–604.DOI
- [3]Lukasz Borchmann, Jordy Van Landeghem, Michal Turski, Shreyansh Padar, Ryan Othniel Kearns, Adam Mahdi, Niels Rogge, Clémentine Fourrier, Siwei Han, Huaxiu Yao, Artemis Llabrés, Yiming Xu, Dimosthenis Karatzas, Hao Zhang, and Anupam Datta. 2026. Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections. In ICML. PMLR, 9178–9241.arxiv.org/abs/2603.12180
- [4]Jun Chen, Hannong Xu, Junjie Fei, Chun-Mei Feng, and Mohamed Elhoseiny. 2025. Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents. In CVPR. CVF/IEEE, 24817–24826.
- [5]Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. 2025. M3DocVQA: Multi-Modal Multi-Page Multi-Document Understanding. In ICCV Workshops. IEEE, 6237–6247.
- [6]Yongbin Choi, Yongwoo Song, and Mujeen Sung. 2026. KoViDoRe: Korean Visual Document Retrieval. arXiv:2608.20840arxiv.org/abs/2608.20840
- [7]Max Conti, Manuel Faysse, Gautier Viaud, Antoine Bosselut, Céline Hudelot, and Pierre Colombo. 2025. Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings. In EMNLP. ACL, 22594–22608.arxiv.org/abs/2505.24782
- [8]Kuicai Dong, Yujing Chiang, Shijie Huang, Yasheng Wang, Ruiming Tang, and Yong Liu. 2025. Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering. In NeurIPS.arxiv.org/abs/2505.16470
- [9]Kuicai Dong, Shurui Huang, Fangda Ye, Wei Han, Zhi Zhang, Dexun Li, Wenjun Li, Qu Yang, Gang Wang, Yichao Wang, Chen Zhang, and Yong Liu. 2026. Doc-Researcher: A Unified System for Multimodal Document Parsing and Deep Research. In WWW. ACM, 2349–2360.arxiv.org/abs/2510.21603
- [10]Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2025. ColPali: Efficient Document Retrieval with Vision Language Models. In ICLR. OpenReview.net.arxiv.org/abs/2407.01449
- [11]Sensen Gao, ShanShan Zhao, Xu Jiang, Lunhao Duan, Yong Xien Chng, Qing-Guo Chen, Weihua Luo, Kaifu Zhang, Jia-Wang Bian, and Mingming Gong. 2026. Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding. In ACL. ACL, 4458–4489.
- [12]Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Commun. ACM 64, 12 (2021), 86–92.arxiv.org/abs/1803.09010
- [13]Madhumita Ghosh, Kunj Je Ou Yang, Peng Zhang, Gaganjan Zeng, Yubo Li, and Hailun Lin. 2025. jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval. In MRL Workshop. ACL, 531–550.
- [14]Hao Guo, Xugong Qin, Jun Jie Ou Yang, Peng Zhang, Gaganyan Zeng, Yubo Li, and Hailun Lin. 2025. Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark. In CVPR. CVF/IEEE, 29722–29732.
- [15]Ashutosh Hathidara, Julien Yu, Vaishali Senthil, Sebastian Schreiber, and Manil Babu Ankisettipalli. 2026. MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness. In KDD. ACM, 8978–8988.
- [16]Jonathan Herzig, Thomas Müller, Syrine Krichene, and Julian Martin Eisenschlos. 2021. Open Domain Question Answering over Tables via Dense Retrieval. In NAACL-HLT. ACL, 512–519.
- [17]Chih-Hui Ho, Giovanna Carreira Marinho, Felipe Viana, Varad Pimpalkhute, Rodolfo Luis Tonoli, and Andre Von Zuben. 2026. DocQIR-Emb: Document Image Retrieval with Multi-lingual Question Query. In UniReps Workshop. PMLR, 183–198.
- [18]Lexiang Hu, Yanzhao Zhang, Mingxin Li, Dingkun Long, Yikang Li, Fuwei Zhang, Yisen Wang, and Zhouchen Lin. 2026. VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval. arXiv:2608.17889
- [19]Ruofan Hu, Menghui Zhu, Jieming Zhu, Bo Chen, Shengyang Xu, Minjie Hong, Xiaoda Yang, Sashuai Zhou, Li Tang, Tao Jin, and Zhou Zhao. 2026. DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark. In KDD. ACM, 1780–1791.DOI
- [20]Heethanjan Kanagalingam, Thenukan Pathmanathan, Mokeeshan Vathanakumar, Basim Azam, Sarah Monazam Erfani, and Naveed Akhtar. 2026. Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity. arXiv:2606.25343arxiv.org/abs/2606.25343
- [21]Juyeon Kim, Geon Lee, Dongwon Choi, Taeuk Kim, and Kijung Shin. 2026. Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracy. In Findings of ACL. ACL, 1073–1089.
- [22]Adithya S. Kolavi and Vyoman Jain. 2025. M3DR: Towards Universal Multilingual Multimodal Document Retrieval. arXiv:2512.03514arxiv.org/abs/2512.03514
- [23]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A Benchmark for Question Answering Research. Trans. Assoc. Comput. Linguistics 7 (2019), 452–466.DOI
- [24]Jaehon Lee, Sohyun Kim, Wanggeun Park, Geon Lee, Seungkyung Kim, and Minyoung Lee. 2025. DS KOpub VDR: A Benchmark Dataset for Visual Document Retrieval in Korean Public Documents. arXiv:2511.04910arxiv.org/abs/2511.04910
- [25]Da Li, Keping Bi, Jiafeng Guo, Wei Yuan, Fan Yang, Tingting Gao, and Xueqi Cheng. 2026. Beyond Text-Only: Towards Multimodal Table Retrieval in Open-World. In ICLR. proceedings.iclr.cc.
- [26]Hanxi Li. 2026. CHaystack: Benchmarking Chinese Document Retrieval and VQA. arXiv:2607.24760arxiv.org/abs/2607.24760
- [27]Zirui Li, Siwei Wu, Yizhi Li, Xingyu Wang, Yi Zhou, and Chenghua Lin. 2025. DocMMIR: A Framework for Document Multi-modal Information Retrieval. In Findings of EMNLP. ACL, 13117–13130.
- [28]Xuye Liu, Yimu Wang, Peng Shi, Bo Xue, Xiangrui Ke, Songcheng Cai, Kath Choi, Di Wu, Freda Shi, and Krzysztof Czarnecki. 2026. LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering. arXiv:2608.07370arxiv.org/abs/2608.07370
- [29]Antônio Loison, Quentin Macé, Antoine Edy, Victor Xing, Tom Baugh, Gabriel de Souza Pereira Moreira, Bo Liu, Manuel Faysse, Céline Hudelot, and Gautier Viaud. 2026. ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios. In ACL. ACL, 16570–16600.
- [30]Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. 2024. Unifying Multimodal Retrieval via Document Screenshot Embedding. In EMNLP. ACL, 6492–6505.arxiv.org/abs/2406.11251
- [31]Quentin Macé, António Loison, and Manuel Faysse. 2025. ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval. arXiv:2505.17166arxiv.org/abs/2505.17166
- [32]Minesh Mathew, Lluís Gómez, Dimosthenis Karatzas, and C. V. Jawahar. 2021. Asking questions on handwritten document collections. Int. J. Document Anal. Recognit. 24, 3 (2021), 235–249.DOI
- [33]Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Raghuveer Thirukovalluru, Xuan Zhang, Zeyuan Chen, Ran Xu, Caifeng Xiong, Yingbo Zhou, Wenhu Chen, and Semih Yavuz. 2026. VLM2Vec-V2: Advancing Multimodal Embeddings for Videos, Images, and Visual Documents. Trans. Mach. Learn. Res. 2026 (2026).arxiv.org/abs/2507.04590
- [34]Xiangyu Peng, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, and Chien-Sheng Wu. 2025. UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG. arXiv:2510.03663arxiv.org/abs/2510.03663
- [35]Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In ACL (Short Papers). 784–789.arxiv.org/abs/1806.03822
- [36]Wenxuan Shen, Mingjia Wang, Yaochen Wang, Dongping Chen, Junjie Yang, Yao Wan, and Weilin Wen. 2026. Are we on the right way to assess document retrieval-augmented generation? In AAAI. AAAI Press, Article 3673, 9 pages.
- [37]Joongmin Shin, Gyuho Shim, Jeongbae Park, Jaehyung Seo, and Heuisoek Lim. 2026. HKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering. In ACL. ACL, 17967–17987.
- [38]Connor Shorten, Augustas Skaburskas, Daniel M. Jones, Charles Pierse, Roberto Esposito, John Trengrove, Etienne Dilocker, and Bob van Luijt. 2026. IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering. arXiv:2602.17687arxiv.org/abs/2602.17687
- [39]Manan Suri, Puneet Mathur, Franck Dernoncourt, Kanika Goswami, Ryan A. Rossi, and Dinesh Manocha. 2025. VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation. In NAACL. ACL, 6088–6109.
- [40]Ryota Tanaka, Taku Hasegawa, and Kyosuke Nishida. 2026. CMDR: Contextual Multimodal Document Retrieval. In ECCV. Springer, 77–95.arxiv.org/abs/2607.05927
- [41]Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. 2025. Visual Document Retrieval-Augmented Generation over Visually-Rich Documents. In CVPR. CVF/IEEE, 24827–24837.arxiv.org/abs/2504.09795
- [42]Rubén Tito, Dimosthenis Karatzas, and Ernest Valveny. 2021. Document Collection Visual Question Answering. In ICDAR. Springer, 778–792.DOI
- [43]Rubén Tito, Dimosthenis Karatzas, and Ernest Valveny. 2023. Hierarchical multimodal transformers for Multipage DocVQA. Pattern Recogn. 144 (2023), 11 pages.
- [44]Kexin Wang, Nils Reimers, and Iryna Gurevych. 2024. DocPR: A Benchmark on Document-Aware Passage Retrieval. In ACL. ACL, 4313–4330.arxiv.org/abs/2305.13915
- [45]Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, and Feng Zhao. 2025. ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents. In EMNLP. ACL, 9113–9134.arxiv.org/abs/2502.18017
- [46]Navve Wasserman, Roi Pony, Oshri Naparstek, Adi Raz Goldfarb, Eli Schwartz, Udi Barzelay, and Leonid Karinsky. 2025. REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark. In ACL. ACL, 31660–31683.
- [47]Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, Wenhao Huang, Noura Al Moubayed, Jie Fu, and Chenghua Lin. 2024. SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval. In Findings of ACL. ACL, 12560–12574.arxiv.org/abs/2401.13478
- [48]Chenghao Xiao, Hou Peng Chan, Hao Zhang, Weixen Xu, Mahani Aljunied, and Yu Rong. 2025. Scaling Language-centric Omnimodal Representation Learning. In NeurIPS.arxiv.org/abs/2510.11693
- [49]Lei Xiong, Huaying Yuan, Zheng Liu, Zhao Cao, and Zhicheng Dou. 2026. Paper-Scope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers. In Findings of ACL. ACL, 8015–8040.
- [50]Yibo Yan, Jiahao Huo, Guanbo Feng, Mingdong Ou, Yi Cao, Xin Zou, Shuliang Liu, Yuanhui Lyu, Yu Huang, Jungang Li, Kening Zheng, Xu Zheng, Philip S. Yu, James Kwok, and Xuming Hu. 2026. Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval. arXiv:2602.19961arxiv.org/abs/2602.19961
- [51]Shouqing Yang, Qi Zhang, Yuhang Yang, Ruikang Xu, Yuwei Hou, Zhulin Jia, Lirong Gao, Haobo Wang, Jinglei Chen, Jiexian Wang, Sheng Guo, Bo Zheng, and Gang Chen. 2026. FinMRAGBench: A Realistic and Complex Benchmark for Multi-Modal RAG in Financial Document Analysis. In Findings of ACL. ACL, 3822–3867.DOI
- [52]Chengsong You, Qiyi Jiang, Junwei Zhou, Xiaoyu Cao, Weiyao Wang, Yiwei Xu, Ziyan Zhao, Zhen Sun, Qicheng Zhu, Xuany Zhu, Yufan Chen, Yilun Li, Rongkang Xiong, Yunhai Hu, and Nan Du. 2026. DocPC: Document-Level Visual Retrieval Mengying Yu and Aixin Sun. 2023. Dataset versus reality: Understanding model performance from the perspective of information need. J. Assoc. Inf. Sci. Technol. 74, 11 (2023), 1293–1306.
- [54]Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025. VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents. In ICLR. OpenReview.net, 21074–21098.arxiv.org/abs/2410.10594
- [55]Xiantao Zhang. 2025. Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey. In IJCNLP–AACL. AFNLP/AACL, 19–36.
