betaFeedback

WorksComputer Science

Pix2Evidence: A Multimodal RAG System for Text and Visual Evidence

Abstract

Retrieval-augmented generation (RAG) over visually rich documents must find both relevant content and the evidence that supports an answer. Screenshot retrieval preserves images and page structure, but written facts within screenshots also need to be searchable. Moreover, topical relevance alone does not identify which retrieved tile contains evidence for the answer. We introduce Pix2Evidence, a multimodal RAG system that encodes each screenshot tile together with its available transcription into a single fused image–text representation, so that layout and readable text are considered jointly during retrieval. To guide evidence selection, a vision-language teacher describes the visible content that supports each training question. We use these descriptions to adapt the retriever, then rerank the retrieved candidates before answer generation. Extensive experiments demonstrate that Pix2Evidence outperforms state-of-the-art methods. Across five widely-used Wikipedia QA benchmarks, the non-reranked Pix2Evidence leads the main comparison in answer accuracy on four. On MMSearch, it reaches 26.0%, 4.4 percentage points above PixelRAG. We will release the code and share derived data where licensing and privacy reviews permit.

Introduction

Retrieval-augmented generation (RAG) grounds answers in an external document collection (Guu et al., 2020; Lewis et al., 2020). A retriever selects candidate sources, and a reader generates an answer from the material it receives (Karpukhin et al., 2020; Izacard and Grave, 2021). Open-domain QA often uses web content: Natural Questions pairs queries with Wikipedia pages (Kwiatkowski et al., 2019), while WebQA draws on both visual and textual web sources [2].

Vision-based retrieval can preserve information that is difficult to capture as plain text, including images, tables, and page layout. Document image retrievers represent rendered pages rather than relying only on extracted text [6] (Ma et al. 2024; Yu et al. 2024). PixelRAG applies this approach to Wikipedia QA by indexing screenshot tiles at scale (Wang et al., 2026). However, many questions still depend on written facts such as names, dates, and numbers. When these facts are embedded in an image, a retriever must recognise the text visually before it can match it to the question. This motivates encoding the text together with the visual tile in a single fused representation, so that written facts become directly available for retrieval without discarding visual and layout cues.

Retrieving relevant content is only the first step. A tile may match a question’s topic without containing the fact needed for the answer, especially when the reader sees only a few tiles [11]. Standard question and document supervision (Xiong et al., 2021) identifies a relevant tile but does not directly describe the content within it that supports the answer. We therefore generate question-conditioned descriptions of visible answer evidence during training. A vision-language teacher uses the question, a positive tile, and the reference answer to describe the supporting content. These descriptions provide evidence-focused supervision for adapting the retriever.

Pix2Evidence addresses these two needs with fused multimodal retrieval and evidence-guided training. Our retrieval component integrates textual and visual information: it jointly encodes each screenshot and its available transcription into a shared multimodal representation, retrieves candidates from this fused view, and reranks them before passing them to the reader. During training, the teacher descriptions make the supporting evidence explicit. We use them to align the candidate rankings induced by a question and its evidence description, alongside contrastive learning. In this way, retriever adaptation attends not only to which tile is relevant, but also to the evidence that makes it useful for answering. Figure 2 illustrates the inference setting; § 4 and § 5 detail the data and training procedure.

Figure 1. An overview of our data construction pipeline, which consists of two parts. (1) Screenshot Text Transcription: Text is extracted and transcribed from image patches using Qwen3-VL-4B-Instruct. (2) Question-Conditioned Evidence Descriptions: A high-capacity model serves as a teacher to generate evidence description text, facilitating alignment during training.

We evaluate the complete system on five Wikipedia QA benchmarks over approximately 7.1 million pages and 28.2 million screenshot tiles. Our non-reranked configuration numerically exceeds PixelRAG in answer accuracy on four benchmarks. On MMSearch it reaches 26.0%, an improvement of 4.4 percentage points over PixelRAG. Additional configurations examine how the input representation, candidate fusion, and reader depth affect these outcomes. Our contributions are:

  • Starting from PixelRAG’s rendered screenshot tiles, we create transcriptions for indexing and teacher descriptions of answer evidence for retriever training.

  • We develop a multimodal RAG system that fuses image and text into a single retrievable representation and adapts its encoder with evidence-guided training.

  • We evaluate the system across five QA benchmarks and multiple configurations. Its non-reranked configuration scores above PixelRAG on four and exceeds both text-extraction baselines on all five.

Retrieval-augmented generation. REALM and RAG incorporate external retrieval into language modelling and knowledge-intensive prediction ((Guu et al., 2020; Lewis et al., 2020)). DPR learns dense question–passage representations ((Karpukhin et al., 2020)), and Fusion-in-Decoder combines multiple passages during generation ((Izacard and Grave, 2021)). Beyond initial retrieval, instruction-following language models can rerank candidate passages, as demonstrated by RankGPT ((Sun et al., 2023)). Readers do not always handle distracting retrieved contexts reliably ((Shen et al., 2024)). We study this selection problem in multimodal RAG, where text and visual content may offer different routes to the same answer.

Visual documents and screenshot retrieval. Document understanding models combine text, layout, and image features, as in LayoutLMv3 ([10]), or generate directly from document images without a separate OCR module, as in Donut ((Kim et al., 2022)). WebSRC pairs page screenshots with HTML and questions that require structural understanding ([4]). VisualMRC studies answer generation from document images collected across web domains ((Tanaka et al., 2021)). These tasks show why reading web content involves both text and visual organisation, although neither evaluates our web-

@page 2/17 starts_mid=1 ends_mid=1

Figure 1 (PDF p. 2). An overview of our data construction pipeline, which consists of two parts. (1) Screenshot Text Transcription: Text is extracted and transcribed from image patches using Qwen3-VL-4B-Instruct. (2) Question-Conditioned Evidence Descriptions: A high-capacity model serves as a teacher to generate evidence description text, facilitating alignment during training.

description, alongside contrastive learning. In this way, retriever adaptation attends not only to which tile is relevant, but also to the evidence that makes it useful for answering. Figure 2 illustrates the inference setting; § 4 and § 5 detail the data and training procedure.

We evaluate the complete system on five Wikipedia QA benchmarks over approximately 7.1 million pages and 28.2 million screenshot tiles. Our non-reranked configuration numerically exceeds PixelRAG in answer accuracy on four benchmarks. On MMSearch it reaches 26.0%, an improvement of 4.4 percentage points over PixelRAG. Additional configurations examine how the input representation, candidate fusion, and reader depth affect these outcomes. Our contributions are:

  • Starting from PixelRAG’s rendered screenshot tiles, we create transcriptions for indexing and teacher descriptions of answer evidence for retriever training.

  • We develop a multimodal RAG system that fuses image and text into a single retrievable representation and adapts its encoder with evidence-guided training.

  • We evaluate the system across five QA benchmarks and multiple configurations. Its non-reranked configuration scores above PixelRAG on four and exceeds both text-extraction baselines on all five.

Retrieval-augmented generation. REALM and RAG incorporate external retrieval into language modelling and knowledge-intensive prediction ((Guu et al., 2020; Lewis et al., 2020)). DPR learns dense question–passage representations ((Karpukhin et al., 2020)), and Fusion-in-Decoder combines multiple passages during generation ((Izacard and Grave, 2021)). Beyond initial retrieval, instruction-following language models can rerank candidate passages, as demonstrated by RankGPT ((Sun et al., 2023)). Readers do not always handle distracting retrieved contexts reliably ((Shen et al., 2024)). We study this selection problem in multimodal RAG, where text and visual content may offer different routes to the same answer.

Visual documents and screenshot retrieval. Document understanding models combine text, layout, and image features, as in LayoutLMv3 ([10]), or generate directly from document images without a separate OCR module, as in Donut ((Kim et al., 2022)). WebSRC pairs page screenshots with HTML and questions that require structural understanding ([4]). VisualMRC studies answer generation from document images collected across web domains ((Tanaka et al., 2021)). These tasks show why reading web content involves both text and visual organisation, although neither evaluates our webscale retrieval setup. For retrieval, DSE encodes screenshots into dense representations (Ma et al., 2024); ColPali extends late interaction from Col-BERT to visual pages ([6] (Khattab and Zaharia 2020)). VisRAG retrieves document images for vision-language generation (Yu et al., 2024), and PixelRAG scales screenshot retrieval to web QA (Wang et al., 2026). SERVAL indexes model-generated page descriptions (Nguyen et al., 2025). We instead index transcribed text from screenshot tiles and reserve question-conditioned descriptions for training. ViDoRe V3 evaluates retrieval, grounding, and generation in visually rich documents (Loison et al., 2026). Our setting keeps web tiles as retrieval units and studies how both textual and visual views affect evidence selection.

Figure 2. Overview of Pix2Evidence. Top: training candidate set for question qq with positive, negative, and in-batch tiles; dashed distributions align question ranking PqP_q to evidence ranking PtP_t. Bottom: tiles ei=(xi,oi,ui)e_i = (x_i, o_i, u_i) are encoded by fθf_\theta into a fused image–text index; query qq retrieves top-NN candidates, a vision-language reranker selects top-kk, and a reader answers. The example’s highlighted row gives US$155 million.

Hybrid retrieval and multimodal evidence fusion.

WebQA asks systems to select and combine visual and textual web sources ([2]). MultiModalQA studies questions that require reasoning across text, tables, and images (Talmor et al., 2021). Both illustrate the need for evidence beyond a single text passage, but they use different task settings from our Wikipedia tile index. MuRAG retrieves from a multimodal memory ([3]), while VisRAG passes retrieved document images to a vision language generator (Yu et al., 2024). In our system, the retriever encodes each tile together with its available transcription in one fused representation, so that readable text and layout are considered jointly rather than in separate indexes.

Evidence-guided retrieval supervision. Contrastive learning distinguishes positive documents from negatives (van den Oord et al., 2018; Karpukhin et al., 2020). RocketQA improves dense retrieval with additional negatives and data augmentation (Qu et al., 2021). Other work uses generated supervision: GPL combines synthetic queries with relevance labels from a cross encoder to adapt dense retrieval (Wang et al., 2022). ReAlign generates question-focused descriptions for selected regions of a document page, then aligns document rankings induced by the question and those descriptions ([11]). We keep a distributional alignment objective but change what the supervision is attached to. Our teacher describes evidence inside a fixed screenshot tile, using the @page 3/17 starts_mid=1 ends_mid=1

Figure 2 (PDF p. 3). Overview of Pix2Evidence. Top: training candidate set for question qq with positive, negative, and in-batch tiles; dashed distributions align question ranking PqP_q to evidence ranking PtP_t. Bottom: tiles ei=(xi,oi,ui)e_i = (x_i, o_i, u_i) are encoded by fθf_\theta into a fused image–text index; query qq retrieves top-NN candidates, a vision-language reranker selects top-kk, and a reader answers. The example’s highlighted row gives US$155 million.

scale retrieval setup. For retrieval, DSE encodes screenshots into dense representations (Ma et al., 2024); ColPali extends late interaction from Col-BERT to visual pages ([6] (Khattab and Zaharia 2020)). VisRAG retrieves document images for vision-language generation (Yu et al., 2024), and PixelRAG scales screenshot retrieval to web QA (Wang et al., 2026). SERVAL indexes model-generated page descriptions (Nguyen et al., 2025). We instead index transcribed text from screenshot tiles and reserve question-conditioned descriptions for training. ViDoRe V3 evaluates retrieval, grounding, and generation in visually rich documents (Loison et al., 2026). Our setting keeps web tiles as retrieval units and studies how both textual and visual views affect evidence selection.

Hybrid retrieval and multimodal evidence fusion.

WebQA asks systems to select and combine visual and textual web sources ([2]). MultiModalQA studies questions that require reasoning across text, tables, and images (Talmor et al., 2021). Both illustrate the need for evidence beyond a single text passage, but they use different task settings from our Wikipedia tile index. MuRAG retrieves from a multimodal memory ([3]), while VisRAG passes retrieved document images to a vision language generator (Yu et al., 2024). In our system, the retriever encodes each tile together with its available transcription in one fused representation, so that readable text and layout are considered jointly rather than in separate indexes.

Evidence-guided retrieval supervision. Contrastive learning distinguishes positive documents from negatives (van den Oord et al., 2018; Karpukhin et al., 2020). RocketQA improves dense retrieval with additional negatives and data augmentation (Qu et al., 2021). Other work uses generated supervision: GPL combines synthetic queries with relevance labels from a cross encoder to adapt dense retrieval (Wang et al., 2022). ReAlign generates question-focused descriptions for selected regions of a document page, then aligns document rankings induced by the question and those descriptions ([11]). We keep a distributional alignment objective but change what the supervision is attached to. Our teacher describes evidence inside a fixed screenshot tile, using the training question and reference answer as guidance, and these descriptions supervise the same encoder that produces our fused tile representation. Unlike ReAlign’s region selection, our pipeline does not predict bounding boxes.

Data Construction

We use the Wikipedia screenshots already rendered and tiled by PixelRAG (Wang et al., 2026) as our visual corpus. The corpus used here comprises approximately 7.1 million pages and 28.2 million tiles, each linked to its source page. On top of these images, we construct two distinct kinds of text. As shown in Fig. 1, query-independent transcriptions serve as index content, whereas query-focused evidence descriptions provide training supervision. The latter depend on a training question and reference answer and are not available as test-time document fields. Appendix § 8 reports corpus counts.

Screenshot Text Transcription

We process every indexed tile xix_i with Qwen3-VL-4B-Instruct (Bai et al., 2025) to transcribe visible text as oio_i in reading order. The prompt discourages completion of words cut off at tile boundaries. Transcriptions are cached independently of retrieval questions and reference answers. We then perform quality, length, and format checks on the resulting transcriptions. About 85% of indexed tiles have nonempty transcriptions; tiles with no visible text retain an empty text field and use image-only inputs.

Question-Conditioned Evidence Descriptions

Inspired by ReAlign [11], we use Qwen3-VL-32B-Instruct (Bai et al., 2025) as a teacher to generate KK descriptions of the evidence that supports the question for each training triple (q,x+,a)(q, x^+, a). The question defines what information is needed; the reference answer aa helps the teacher locate that information in the positive tile x+x^+. The generation instruction asks for supporting content visible in the tile, rather than an unsupported restatement of the answer. The resulting descriptions supply answer-conditioned training signals intended to focus the multimodal retriever on useful evidence. It differs from ReAlign’s region descriptions [11] because we describe fixed screenshot tiles and do not select bounding boxes.

Every training question has a supervision text tqt_q. When generated descriptions are available, we sample one uniformly at each training use; otherwise, we concatenate the question and its reference answer to form tqt_q. The reference answer helps the teacher locate supporting content during data construction. This supervision is used only for training and is not introduced at inference.

Pix2Evidence Framework

Overview

Let D={ei}\mathcal{D} = \{e_i\} with ei=(xi,oi,ui)e_i = (x_i, o_i, u_i) denote a screenshot tile, its transcription, and its source identifier. Each tile is encoded together with its transcription into a fused image–text representation. An overview of Pix2Evidence is shown in Fig. 2.

At inference, Pix2Evidence retrieves the top-NN candidates from this fused index (§ 5.2.1), reranks them with a vision-language reranker (§ 5.3), and passes the top-kk tiles to the reader. The retriever is trained offline with evidence-guided supervision: a contrastive term pulls the question toward its positive tile (§ 5.4.1), and an alignment term makes the question-side ranking follow the ranking induced by an evidence supervision text (§ 5.4.3). Evidence descriptions are used only during training; inference relies solely on the question and indexed document content.

Inference Pipeline

Multimodal Retrieval

We adapt Qwen3-VL-Embedding-2B (Li et al., 2026) with LoRA [9] in its language and vision components. For tile ei=(xi,oi,ui)e_i = (x_i, o_i, u_i), xix_i is its screenshot, oio_i is its transcription, and uiu_i identifies the source page. A shared encoder fθf_\theta maps an input to a normalised vector; θ\theta denotes its adapted parameters. We apply query and document input templates according to their roles. The notation fθ(q)f_\theta(q) is the vector for question qq, and fθ(xi,oi)f_\theta(x_i, o_i) encodes the image with its available text. If oio_i is empty, the input contains only xix_i.

The retrieval score is

s(q,ei)=⟨fθ(q),fθ(xi,oi)⟩,(1)s(q, e_i) = \langle f_\theta(q), f_\theta(x_i, o_i) \rangle, \tag*{(1)}

where angle brackets denote a vector inner product. A FAISS index supports search over these normalised vectors (Johnson et al., 2021). Appendix § 8.§ 4 describes the input roles and the availability of their prompt templates.

For a question qq, let HN(q)H_N(q) be the ranked list of at most NN hits returned by the fused retriever:

HN(q)=Top⁡Nlist(s(q,⋅)),H_N(q) = \operatorname{Top}^{\mathrm{list}}_N\left(s(q,\cdot)\right),
C(q)=HN(q),C(q) = H_N(q),
∣C(q)∣≤N.(2)|C(q)| \leq N. \tag*{(2)}

The operator Top⁡Nlist\operatorname{Top}^{\mathrm{list}}_N returns the highest-scoring hits in rank order. ∣C(q)∣|C(q)| counts hit positions in the resulting list. We pass C(q)C(q) to the reranking stage; Appendix § 8.§ 6 specifies candidate input formats.

Cross-Modal Reranking

Qwen3-VL-Reranker-2B (Li et al., 2026) jointly scores each question–candidate pair without task-specific reranker training. Given the candidate list C(q)C(q) from the fused retrieval stage, we compute for each candidate input vjv_j

rj=gϕ(q,vj),r_j = g_{\phi}(q,v_j),
Ik(q)=Top⁡kindices(r1,…,r∣C(q)∣).(3)I_k(q) = \operatorname{Top}^{\mathrm{indices}}_k\left(r_1,\ldots,r_{|C(q)|}\right). \tag*{(3)}

Here gϕg_{\phi} is the vision-language reranker with parameters ϕ\phi, rjr_j is the relevance score of candidate input vjv_j, and Top⁡kindices\operatorname{Top}^{\mathrm{indices}}_k returns the indices Ik(q)I_k(q) of the kk highest-scoring candidates. We derive rjr_j from the yes/no logits and pass those candidates to the reader with the question. The reranker input format for the fused retrieval path is detailed in Appendix § 8.§ 6.

Training

Contrastive Retrieval Objective

For question qq, let e+e^{+} denote its positive tile and D~q\widetilde{D}_q the training candidate set containing e+e^{+}, mined negatives, and in-batch candidates gathered across devices. We define the question-side ranking distribution

Pq(e)=exp⁡(sθ(q,e)/τ)∑d∈D~qexp⁡(sθ(q,d)/τ),(4)P_q(e) = \frac{\exp(s_{\theta}(q,e)/\tau)}{\sum_{d\in\widetilde{D}_q}\exp(s_{\theta}(q,d)/\tau)}, \tag*{(4)}

where sθ(q,e)s_{\theta}(q,e) is the encoder similarity between the question and the fused image–text representation of candidate ee, and τ\tau is the softmax temperature. Following standard contrastive retrieval training, we minimise

LNCE=−log⁡Pq(e+).(5)\mathcal{L}_{\mathrm{NCE}} = -\log P_q(e^{+}). \tag*{(5)}

Evidence-Guided Alignment

The contrastive term above only uses the question side and does not exploit evidence supervision. Let tqt_q be the evidence supervision text for question qq from § 4.2, and let Pt=Pθ(⋅∣tq)P_t = P_{\theta}(\cdot\mid t_q) be the ranking distribution induced by tqt_q over the same candidate set. We align the question-side ranking with this evidence-side ranking by

Lalign=DKL(sg⁡[Pt] ∥ Pq),(6)\mathcal{L}_{\mathrm{align}} = D_{\mathrm{KL}}\left(\operatorname{sg}[P_t] \,\middle\|\, P_q\right), \tag*{(6)}

where DKLD_{\mathrm{KL}} is Kullback–Leibler divergence and sg⁡\operatorname{sg} stops gradients through PtP_t, so that PqP_q is updated toward the evidence-induced target. This makes the evidence text act as a target for the question-side ranking, which is the side used at inference time.

Overall Training Objective

The total objective combines the contrastive and alignment terms:

L=LNCE+λLalign.(7)\mathcal{L} = \mathcal{L}_{\mathrm{NCE}} + \lambda\mathcal{L}_{\mathrm{align}}. \tag*{(7)}

λ\lambda balances the two terms; we set λ=0.05\lambda= 0.05 in all experiments, and both distributions use τ≈\tau\approx 0.07. Our training signal comes from descriptions of answer evidence in fixed screenshot tiles. We adapt the ranking alignment objective of ReAlign Yang et al., [11] to these tile-level descriptions rather than selected visual regions. The adapted encoder builds the fused image–text index used in the main retrieval pipeline.

Experiments

Evaluation Setting

Benchmarks. We evaluate Natural Questions (NQ) (Kwiatkowski et al., 2019), NQ-Tables, SimpleQA (Wei et al., 2024), MMSearch (Jiang et al., 2024), and Encyclopedic-VQA (EVQA) (Mensink et al., 2023) on Wikipedia QA. The main comparison reports answer accuracy and gold-article evidence recall; MMSearch has no gold articles for recall evaluation.

Baselines. We compare text extraction with mwparserfromhell and Trafilatura [1], screenshot retrieval with PixelRAG (base) and PixelRAG (Wang et al., 2026), and Pix2Evidence with and without reranking. Table 1 presents the main comparison.

MethodNQ RecallNQ AccNQ-Tables RecallNQ-Tables AccSimpleQA RecallSimpleQA AccMMSearch RecallMMSearch AccEVQA RecallEVQA Acc
No retrieval13.110.96.413.527.3
Text-based retrieval
mwparserfromhell48.627.735.018.372.958.4–12.93.230.7
Trafilatura45.829.437.721.979.369.0–13.53.229.2
Pixel-based retrieval
PixelRAG (base)53.531.045.524.379.970.3–19.917.039.4
PixelRAG58.931.150.725.883.172.8–21.633.240.1
Fused retrieval
Pix2Evidence60.632.156.825.786.073.4–26.030.840.6
Pix2Evidence (+reranker)57.833.255.827.186.977.6–23.033.040.3

Table 1. Main results on five Wikipedia QA benchmarks. We use LLM-judged for SimpleQA, MMSearch, EVQA; EM for NQ and NQ-Tables. MMSearch has no gold articles, so its recall is not reported. Bold marks the best value in each column.

Configuration. The main experiment uses Qwen3-VL-4B-Instruct as the reader and GPT-4.1 (OpenAI, 2025) to judge answer accuracy. Scores use a 0–100 scale unless stated otherwise. NQ and NQ-Tables are evaluated with either LLM-judged answer accuracy or exact match (EM), depending on the experiment. We identify the measure used in each table and figure; further protocol details appear in Appendix § 9.

Main Results

Table 1 presents the main comparison. Without reranking, Pix2Evidence already achieves better results than both text-extraction and screenshot-retrieval baselines on most benchmarks, especially

on MMSearch and EVQA, reaching 26.0 and 40.6, respectively. This shows that fusing visual and textual information in a single retrieval view provides stronger evidence than using either modality alone. Adding the vision-language reranker further improves answer accuracy on NQ, NQ-Tables, and SimpleQA, reaching 33.2, 27.1, and 77.6, respectively. Compared with PixelRAG, these gains are 2.1, 1.3, and 4.8 percentage points. The reranked configuration also raises SimpleQA evidence recall from 83.1 to 86.9 and NQ-Tables recall from 50.7 to 55.8. The non-reranked configuration remains strongest on MMSearch at 26.0 and EVQA at 40.6, which allows the reranking stage to be enabled or disabled according to the target scenario.

Evidence-Alignment Variants

Table 2 examines how we train with teacher-generated evidence descriptions. Following Pix2Evidence (Wang et al., 2026), we use a development subset for evaluation. Contrastive only uses LNCE\mathcal{L}_{\mathrm{NCE}}. Forward KL aligns the question-side ranking PqP_q to the evidence-side ranking PtP_t via DKL(sg⁡[Pt]∥Pq)D_{\mathrm{KL}}(\operatorname{sg}[P_t] \mathbin\Vert P_q), which is the direction used in our method, while reverse KL uses the opposite direction. Description alignment instead applies a cosine penalty between the question and description embeddings. The weight λ\lambda is the coefficient of the alignment term in L=LNCE+λLalign\mathcal{L}=\mathcal{L}_{\mathrm{NCE}}+\lambda\mathcal{L}_{\mathrm{align}} (Eq. (7)).

AlignmentWeight λ\lambdaQA
Contrastive only–0.7725
Forward KL0.200.7825
Reverse KL0.200.7800
Description (cosine)0.100.7750
Reverse KL0.050.7750
Description (cosine)0.200.7725
Forward KL0.050.7900

Table 2. Ablation on evidence-alignment objectives. Weight λ\lambda is the coefficient of the alignment term in Eq. (7). “Description (cosine)” applies a cosine penalty between question and description embeddings.

Among the recorded variants, forward KL with λ=0.05\lambda=0.05 achieves the highest QA score, 0.7900, versus 0.7725 for contrastive only. It also exceeds forward KL at λ=0.20\lambda=0.20 (0.7825) and reverse KL at λ=0.05\lambda=0.05 (0.7750). We therefore adopt forward KL with λ=0.05\lambda=0.05 in the final system.

Retrieval–Reader Modality Ablation

Table 3 varies the document embedding and reader inputs on SimpleQA; Text denotes corpus transcriptions. The best configuration uses Image+Text at both stages, reaching 73.4.

Embedding inputReader inputSimpleQA
Image + TextImage + Text73.4
Image + TextImage67.9
Image + TextText71.9
TextImage60.6
TextImage + Text66.2
ImageImage69.0

Table 3. Retrieval–reader modality ablation on SimpleQA. We vary the input modality at both stages. Bold indicates the highest score.

We then fix one stage and vary the other. With the embedding fixed to Image+Text, the multimodal reader scores 73.4, compared with 71.9 for text-only reading and 67.9 for image-only reading. With the multimodal reader fixed, joint embedding scores 73.4, compared with 66.2 for text-only embedding. These comparisons show that combining visual and textual information benefits both retrieval and reading.

Dual-Branch Fusion Ablation

Our main pipeline uses a single fused image–text retrieval view. As an extension, we analyze how a dual-branch design would behave if a text-only branch were added, and compare four fusion rules without reranking at k=3k = 3 (Table 4). All variants use the same image–text and text branches, with candidates deduplicated by tile. Max keeps the larger raw score for each tile; RRF sums reciprocal ranks with offset 60 ([5]); z-score and min–max normalise each branch’s scores within a query before summation.

FusionSimpleQAEVQANQ (EM)NQ-Tables (EM)MMSearch
Max64.630.429.921.719.0
RRF63.730.431.624.422.0
z-score67.333.529.824.420.3
Min–max66.030.830.724.220.3

Table 4. Fusion-rule ablation for a dual-branch extension without reranking (k=3k = 3). The two branches are an image–text retriever and a text-only retriever, with candidates deduplicated by tile. NQ and NQ-Tables use EM. Bold marks column-wise best scores, including ties.

Fusion changes how the two retrieval views prioritise shared candidates. Relative to Max, z-score raises SimpleQA from 64.6 to 67.3 and EVQA from 30.4 to 33.5. RRF achieves the highest NQ

EM at 31.6 and the highest MMSearch score at 22.0, and ties z-score on NQ-Tables EM at 24.4. No single rule dominates across all benchmarks, but every fusion rule outperforms Max on at least one benchmark, showing that combining normalised scores or branch rankings is more effective than taking the maximum raw score.

Effect of Reading Depth

Figure 3 varies kk from one to three and compares both Pix2Evidence configurations with PixelRAG, PixelRAG (base), and Trafilatura at different reading depth.

Figure 3. Effect of reading depth and reranking across four benchmarks. Pix2Evidence and Pix2Evidence (reranker) are evaluated at k∈{1,2,3}k \in\{1, 2, 3\}. NQ and NQ-Tables use EM in the follow-up runs; kk denotes the number of items supplied to the reader.

Across all evaluated depths, both Pix2Evidence configurations consistently outperform the baselines, and the margin is often large. For example, on MMSearch, the non-reranked Pix2Evidence reaches 26.0 at k=3k = 3, compared with 21.6 for PixelRAG, a gain of 4.4 points. The reranked configuration also reaches 23.0, still above the baseline. This advantage remains stable across the evaluated reading depths.

Reader Model Ablation

Table 5 compares readers from the Qwen3-VL (Bai et al., 2025) and Qwen3.5 (Qwen Team, 2026) families under the same single-branch image–text

ReaderParamsSimpleQAEVQANQ (EM)NQ-Tables (EM)MMSearch
Qwen3-VL family
Qwen3-VL-2B2.3B0.7000.3240.2440.2510.213
Qwen3-VL-4B4B0.7760.4030.3320.2710.230
Qwen3-VL-8B8B0.7940.4330.3140.2890.263
Qwen3.5 family
Qwen3.5-2B2B0.7520.4210.2350.2290.213
Qwen3.5-4B4B0.7980.4350.3350.2710.273
Qwen3.5-9B9B0.8130.4670.2720.3050.300

caption: tbl:5: Reader-model ablation with single-branch image–text retrieval and vision-language reranking (N=5N = 5, k=3k = 3). Values are on a 0–1 scale; NQ and NQ-Tables use EM. Bold marks each column’s observed maximum.

retrieval and vision-language reranking setting. The Qwen3.5 family performs better overall than Qwen3-VL at comparable or smaller parameter scales. Qwen3.5-9B gives the best observed results on four benchmarks: SimpleQA at 0.813, EVQA at 0.467, NQ-Tables at 0.305, and MMSearch at 0.300. Qwen3.5-4B achieves the highest NQ at 0.335 and also reaches 0.798 on SimpleQA. These results indicate that stronger readers generally bring greater benefits to the system. The system remains effective across all evaluated reader sizes, and its performance generally improves as the reader becomes stronger.

Image Downsampling

Table 6 evaluates image downsampling. We apply Lanczos downsampling, which reduces each image side by c\sqrt{c} and its nominal area by a factor of cc. Text is kept unchanged. Additional text-length, candidate-pool, and reading-depth results appear in Appendix § 10.

SettingPixel fractionSimpleQANQ (EM)NQ-Tables (EM)MMSearch
Baseline177.633.227.123.0
Downsample c=2c = 21/21/276.433.326.522.0
Downsample c=3c = 31/31/374.433.427.724.0
Downsample c=4c = 41/41/474.033.326.820.3

Table 6. Image downsampling on the single-branch Pix2Evidence (reranker) configuration (N=5N = 5, k=3k = 3), without compression-specific fine-tuning. Text is unchanged.

Moderate downsampling largely preserves answer quality. At c=2c = 2, SimpleQA only drops from 77.6 to 76.4, while NQ slightly improves from 33.2 to 33.3 and NQ-Tables changes from 27.1 to 26.5. Stronger compression at c=4c = 4 reduces SimpleQA to 74.0 and MMSearch to 20.3, while NQ remains stable at 33.3. Overall, these results show that moderate image downsampling generally retains most of the measured answer quality with fewer input pixels, and that SimpleQA and MMSearch are more sensitive to stronger compression than NQ and NQ-Tables in these runs.

Conclusion

We presented Pix2Evidence, a multimodal RAG system for question answering over Wikipedia pages. It treats each page as a collection of screenshot tiles and encodes each tile together with its available transcription into a fused image–text representation, so that layout and readable text are considered jointly rather than as separate retrieval views. At inference time, the fused retriever returns candidate tiles, a vision-language reranker selects the most relevant ones, and a reader generates the answer from the question and the selected evidence. For training, a teacher model produces descriptions of answer-supporting evidence, which guide the retriever through a contrastive objective and an evidence-alignment term. Across five Wikipedia QA benchmarks, the non-reranked system scores above PixelRAG in answer accuracy and leads the main comparison on four. Its 26.0% MMSearch accuracy exceeds PixelRAG by 4.4 percentage points. The results underline a practical distinction: retaining rich document content is useful, but a RAG system must also deliver the right evidence to the reader. Extensive experiments demonstrate the effectiveness of Pix2Evidence.

Limitations

Our evaluation focuses on English Wikipedia QA and a limited set of model configurations. Although we test readers from two model families, the sweep does not cover other architectures, larger scales, or non-English domains. A broader reader-model evaluation, including multilingual and domain-specific settings, remains future work. The fused image–text index and the vision-language reranker introduce additional storage and computation compared with a text-only pipeline. Our input-budget study varies pixel area and text length, but it does not measure peak memory and end-to-end latency. These factors are important for practical deployment, especially when the reader context is large. A complete cost evaluation and more efficient indexing and reranking designs therefore remain future work. For example, one could explore compressed index representations, lightweight rerankers, or adaptive reranking that is applied only when the initial retrieval is uncertain.

Ethical Considerations

We adhere to the ethical guidelines of the research community. All experiments use publicly available datasets and open-source models, ensuring transparency and reproducibility. We have not collected new data involving human subjects or personally identifiable information.

The datastore contains rendered web pages and may include copyrighted material or personal information. Retrieved evidence improves traceability, but it does not establish that a source is correct or appropriate to disclose. Reference answers condition training descriptions only and are not provided during test-time retrieval or generation. Our method focuses on retrieval and evidence selection and does not modify the underlying safety alignment of the reader models.

System and Data Construction

This appendix distinguishes the PixelRAG screenshot source (Wang et al., 2026) from our transcription, evidence supervision, and retrieval implementation. The corpus counts below describe the subset indexed for Pix2Evidence, not a new rendering effort.

Rendered Wikipedia Corpus

Screenshot source. We use the Wikipedia images already rendered and tiled by PixelRAG (Wang et al., 2026), rather than rerendering the pages ourselves. Its rendering procedure uses an 875-pixel viewport width and divides long pages into nonoverlapping tiles of up to 1024 pixels in height. Each tile retains a link to its source article. The image corpus used in our index contains 7,123,459 Wikipedia pages and 28,244,361 tiles (Table 7); these counts differ from those reported for PixelRAG’s full corpus. We will release the exact source snapshot and our indexed subset together with the indexed data.

ItemCount
Wikipedia pages7,123,459
Screenshot tiles28,244,361
Tiles with nonempty transcriptions23,948,959
Training questions39,482
Training tile occurrences118,446
Questions with descriptions39,205
Hard negatives per question2

Table 7. Corpus and selected training data. Transcription covers the indexed tiles; the transcription count includes only nonempty text. Training occurrences include one positive and two negatives per question.

Counting units. All indexed tiles are processed for transcription, but a tile without visible text may have an empty transcription. The 23,948,959 figure counts distinct tiles with nonempty text, approximately 85% of the indexed corpus, not transcription coverage. Training tile occurrences instead count one positive and two negatives per question; they are not distinct tile records.

Text Sources and Supervision

Corpus transcription. Qwen3-VL-4B-Instruct processes every indexed tile for visible text, with a 2048-token output limit. The prompt discourages completion of words cut off at tile boundaries. A [NO TEXT] response is stored as an empty string. Transcriptions are cached independently of the question and answer; when no text is visible, the fused retriever uses an image-only input. Text is truncated to a fixed character cap before encoding.

Training evidence descriptions. For each training question, positive tile, and reference answer, Qwen3-VL-32B-Instruct is prompted to generate KK descriptions of visible answer evidence; KK is configurable and the reported run uses K=3K = 3. Generation uses temperature 0.7, a 1024-token output limit, and images resized to a maximum side of 2048 pixels. Generated descriptions are available for 39,205 of 39,482 questions. One is sampled uniformly at each training use; for questions without generated descriptions, the question and answer are concatenated. Thus every training question has supervision text. The answer helps the teacher locate evidence during data construction. The supervision is used only during training, not at inference. We apply quality, length, and format checks before sampling the descriptions.

Model Inputs and Data-Production Prompts

Input roles. Table 8 lists the input fields used at each stage. In § 5.2.1, fθ(q)f_{\theta}(q) denotes the embedding of a query-template input, and fθ(xi,oi)f_{\theta}(x_i,o_i) denotes a document-template input containing the tile image and available text, or the image alone when oio_i is empty. The text-only input fθ(oi)f_{\theta}(o_i) is used only in the text-only ablation.

StageInput fields
TranscriptionTile image; text-visible transcription instruction
Evidence teacherTraining question, positive tile, reference answer
Query embeddingQuestion in query role
Document embeddingTile image and available transcription
RerankingQuestion and retrieved candidate payload
Answer generationQuestion and selected evidence items

Table 8. Inputs used at each stage. The document embedding uses the fused image–text input; a text-only document input is used only in the text-only ablation.

Data-production prompts. Qwen3-VL-4B-Instruct produces the corpus transcription oio_i of § 4.1 from the tile image, and Qwen3-VL-32B-Instruct produces the training supervision tqt_q of § 4.2 from the positive tile. Figure 4 reproduces both prompts verbatim; line breaks are added for display only. For the evidence description, {k}\{k\} is the number of requested descriptions, set to three in the reported run, and {query}\{query\} is the training question. The final line appends {answer_hint}\{answer\_hint\}, a training-only marker of the form \textbackslash nGolden answer (use only to locate the evidence; do NOT quote it verbatim as the whole description): followed by the reference answer.

The two data-production prompts

Figure 4. The two data-production prompts, reproduced verbatim: (top) tile transcription for the corpus text view and (bottom) question-conditioned evidence descriptions for retriever training.

Retrieval path / hitReranker input
Fused path: image hitImage; text if available
Dual-view ablation: image hitImage only
Dual-view ablation: text hitText

Table 10. Reranker inputs. The fused path is the main pipeline; the dual-view entries belong to the fusion ablation in § 6.5.

Retriever Training

Optimisation. The recorded launch uses batch size 32, gradient-cache chunks of two (Gao et al., [7]), learning rate 10−510^{-5}, 20 warm-up steps, and a cosine schedule capped at 350 steps. A text warm-up stage uses about 15K question and answer pairs for 50 steps. The selected retriever checkpoint is at step 100. The visual-token limit is 4096. Three H800 80GB GPUs train the retriever; a fourth serves the reader through vLLM (Kwon et al., 2023).

Alignment objective. Question and evidence-description distributions share a temperature of approximately 0.07. The forward objective in § 5.4.3 weights the KL term by 0.05 and stops gradients through the question-induced target. For the reverse-KL variant, the description-induced distribution is detached and the query branch can still receive gradients.

Development subset. The development subset, adopted from PixelRAG, is a compact minidatastore for embedding ablations, containing 400 questions and 7,426 tiles. For each question, it includes all tiles from the gold article plus the top-100 tiles retrieved by the base embedding model, mixing relevant and irrelevant candidates. We use this subset for our training alignment ablation.

Candidate Processing and Reader Inputs

Candidate lists. Our main fused retrieval path returns a single ranked list and passes the top-NN hits to the reranker. In the ablation study, we use a dual-view recall of text and fused image–text to obtain the candidate list, and deduplicate documents that appear in both views. This variant is used only for the fusion ablation in § 6.5. The fusion ablation deduplicates by tile and compares Max, RRF, z-score, and min–max without reranking. We apply per-query score normalisation before summation.

Evaluation Protocol

Benchmark Details

Benchmark groups. NQ, NQ-Tables, and SimpleQA mainly test factual answers in text or structured page content. MMSearch and Encyclopedic VQA add image-grounded questions. Table 9 lists the benchmarks used in our evaluation and their subset sizes. We report these subset sizes.

BenchmarkTask familySubset sizeMain-table measure
Natural QuestionsText and Wikipedia QA1,000EM, article recall
NQ-TablesStructured Wikipedia QA1,000EM, article recall
SimpleQAFactual Wikipedia QA1,000GPT-4.1 judged accuracy, article recall
MMSearchMultimodal QA300GPT-4.1 judged accuracy; no article recall
Encyclopedic VQAImage-grounded QA749GPT-4.1 judged accuracy, article recall

Table 9. Wikipedia benchmark subsets used in our evaluation, together with their task families, subset sizes, and main-table evaluation measures. All five tasks query our Wikipedia tile index, while NQ and NQ-Tables use exact match and the remaining benchmarks use GPT-4.1 judged accuracy.

Natural Questions (NQ). We use 1,000 questions associated with Wikipedia articles (Kwiatkowski et al., 2019). Answers are checked against the available reference aliases. This is a text-centric task, but the supporting content can include structured page elements.

NQ-Tables. We evaluate 1,000 table-oriented Wikipedia questions. Tables and infoboxes make the presentation of structured facts relevant to retrieval and reading.

SimpleQA. We use 1,000 short factual questions (Wei et al., 2024). We use the Wikipedia datastore to retrieve evidence for them. The answer-bearing fact can appear in a paragraph, list, infobox, or table; a retrieved article is not necessarily the correct tile.

MMSearch. We include all 300 end-to-end examples: 171 include a query image and 129 contain text only. Although their topics extend beyond Wikipedia, the retrieval backend in this comparison is the Wikipedia tile index. MMSearch does not supply gold article URLs for the article-recall column, which is therefore marked unavailable.

Encyclopedic VQA (EVQA). We use 749 automatic questions from the landmarks portion of Encyclopedic VQA (Mensink et al., 2023). The iNaturalist portion is excluded because some query images are unavailable in its official release. This is a visual question answering task grounded in fine-grained entity information. The shorthand EVQA in the main table denotes this subset, not every category of the full benchmark.

MoNaCo. The separate agent evaluation contains 1,315 multi-hop Wikipedia questions per recorded run, all completed (Section C.3) (Wolfson et al., 2025). PixelRAG’s reference protocol uses a GPT-5 ReAct controller with a single search tool, up to 20 search calls per question and five hits per call (Wang et al., 2026). MoNaCo Judge-F1 must not be read as the Wikipedia QA accuracy in Table 1.

Scoring and Grading

Answer accuracy. The main table uses Qwen3-VL-4B-Instruct as the reader and GPT-4.1 to judge generated answers, reporting the share judged correct on a 0–100 scale. PixelRAG’s reference grading protocol classifies answers as CORRECT, INCORRECT, or NOT_ATTEMPTED and scores

only CORRECT as one (Wang et al., 2026). For NQ and NQ-Tables, that protocol accepts up to ten gold answer aliases joined as alternatives. Its grader configuration specifies temperature zero, seed 42, and a maximum of 1,000 output tokens.

Evidence recall. Article recall at reading depth kk records whether an item in the selected top kk comes from a gold source article, matched by URL, not whether the item contains the answer. The main comparison uses a three-item reader context unless specified otherwise. The MMSearch recall column is unavailable because it has no annotated gold articles in this protocol.

Other evaluation scales. The later fusion and input-budget studies use exact match (EM) for NQ and NQ-Tables, without the main table’s LLM judging. Alignment variants report a QA value on a 0–1 scale. The MoNaCo appendix reports Judge-F1 on a 0–1 scale. The Qwen3.5 reader batch also uses a 0–1 scale; NQ and NQ-Tables there are EM-only, while the remaining benchmarks retain their own judging protocols. Scores from these different records cannot be compared by simply subtracting table cells.

Baselines and Configuration

Baselines. The main comparison includes no retrieval, two text extraction pipelines (mwparserfromhell and Trafilatura [1]), PixelRAG (base) and PixelRAG (Wang et al., 2026), and Pix2Evidence with and without reranking. The PixelRAG reference text indexes use 1,024-token chunks to balance the approximately 875 visual tokens in a screenshot tile; we follow the same chunk size for the local text baselines.

Configuration. We evaluate the system mainly with two readers, Qwen3-VL-4B-Instruct and Qwen3.5-4B, whereas PixelRAG’s main table reports a single Qwen3.5-4B reader. Our main pipeline uses the fused image–text retriever and passes its top-NN hits to reranking. The text-only branch and the dual-view retrieval that unions it with the image–text branch are evaluated in the ablation studies rather than in the main configuration.

Additional Experiments

Input and Candidate Budgets

Text truncation. Table 11 starts from the full transcription and truncates it to a fixed number of characters, keeping native-resolution images, N=5N = 5 reranker candidates, and k=3k = 3 reader items with the fused image–text retrieval view and reranking. The untruncated run is the baseline, and each remaining row is an ablation that shortens the text supplied to the encoder. At 2048 characters the scores stay close to the baseline: SimpleQA is 76.7 versus 77.6 (−0.9 points) and NQ EM is 32.9 versus 33.2 (−0.3 points), while NQ-Tables EM falls from 27.1 to 25.4 (−1.7 points). Truncating more aggressively lowers SimpleQA to 73.9 at 1024 characters and to 72.1–72.8 at 128–512 characters. NQ and NQ-Tables respond differently: NQ EM is essentially unchanged at 1024 characters (+0.1 points) but drops to 32.3 and below at 512 characters and shorter, whereas NQ-Tables EM stays within 24.7–25.4 at every truncation length, so its loss is not recovered by keeping more text. Answer quality therefore declines as more text is removed, with the penalty concentrated in SimpleQA; a mild truncation such as 2048 characters retains most of the measured quality and is a reasonable setting.

Transcription cap (chars)SimpleQANQ (EM)NQ-Tables (EM)
Baseline77.633.227.1
12872.129.724.7
25672.631.825.4
51272.832.324.7
102473.933.325.3
204876.732.925.4

Table 11. Transcription truncation using the fused image–text retrieval view with reranking, native-resolution images, N=5N = 5, and k=3k = 3. NQ and NQ-Tables report exact match; other reported scores follow their respective benchmark protocols.

Candidate-pool size. Table 12 fixes the reader depth at k=3k = 3 and varies the reranker pool NN, again with the fused image–text retrieval view and reranking. Relative to N=5N = 5, a pool of 20 raises SimpleQA by 4.4 points to 82.0, NQ EM by 1.8 to 35.0, NQ-Tables EM by 1.0 to 28.1, and MMSearch by 2.0 to 25.0. At N=50N = 50, SimpleQA reaches 82.2, the highest score in this sweep, but increases only 0.2 points over N=20N = 20. NQ EM instead falls from 35.0 to 33.7; NQ-Tables EM stays at 28.1. MMSearch peaks at N=10N = 10 with 25.7 and is 25.0 at both larger pools. The gains against N=5N = 5 are therefore positive on all four reported benchmarks, but their size and best tested pool differ. Scoring 20 or 50 candidates requires four or ten times as many query–candidate scoring calls as scoring five, assuming one forward evaluation per pair. The pool size should therefore be chosen to balance answer quality against these additional scoring calls.

SettingNNkkSimpleQANQ (EM)NQ-Tables (EM)MMSearch
Baseline5377.633.227.123.0
Larger pool10380.934.227.425.7
Larger pool20382.035.028.125.0
Larger pool50382.233.728.125.0
Longer context5577.733.626.425.3

Table 12. Candidate-pool size and reading-depth extensions for the fused image–text retrieval view with reranking. NN is the number of candidates supplied to the reranker; kk is the number of items supplied to the reader. Images and text retain the baseline input settings. NQ and NQ-Tables report EM.

Reader context. At N=5N = 5, increasing kk from three to five changes SimpleQA by +0.1 points, NQ EM by +0.4, NQ-Tables EM by −0.7, and MMSearch by +2.3. This complements the reading-depth study of § 6.6: the reader can benefit from a longer context, but the change is small and not positive on every benchmark, whereas each additional item enlarges the reader’s input. A larger kk therefore trades more reading cost for a modest change in answer quality.

More Ablations with the Qwen3.5 Reader

Relation to the reader sweep. The Qwen3.5-4B row for the fused image–text view with reranking at k=3k = 3 in Table 13 is also reported in the main reader-model comparison (Table 5).

RetrievalReranker (k=3k = 3)EVQASimpleQANQ (EM)NQ-Tables (EM)MMSearch
Fusedno0.4350.7590.3350.2470.253
Fusedyes0.4350.7980.3350.2710.273
Dual-viewno0.3260.6880.3050.2250.250
Dual-viewyes0.3990.8100.3550.2980.273

Table 13. Qwen3.5 reader results at k=3k = 3 for the four retrieval settings. Fused is the single image–text view and dual-view adds the text-only view. Values are on a 0–1 scale. NQ and NQ-Tables are EM-only; EVQA, SimpleQA, and MMSearch follow their own judging protocols.

Configurations. Table 13 and Table 14 report Qwen3.5 reader runs at k=3k = 3 and k=1k = 1 under four retrieval settings. Fused retrieves through the single view that encodes each tile together with its available transcription; dual-view adds a second view that indexes the transcriptions as text. Each setting is run with and without reranking, which uses Qwen3-VL-Reranker-2B (Li et al., 2026) to rescore the retrieved candidates before the top-kk items are passed to the reader. The four rows therefore differ only in the retrieval views and whether reranking is applied.

RetrievalReranker (k=1k = 1)EVQASimpleQANQ (EM)NQ-Tables (EM)MMSearch
Fusedno0.3750.6320.2770.2040.257
Fusedyes0.4310.7660.3130.2290.257
Dual-viewno0.2630.5680.2570.1700.187
Dual-viewyes0.3770.7840.3160.2520.263

Table 14. Qwen3.5 reader results at k=1k = 1 for the same four settings. Scale and benchmark-specific metrics follow Table 13.

Interpretation. Without reranking, the fused view is stronger than the dual-view on every benchmark at k=3k = 3, and the same holds at k=1k = 1: at k=3k = 3, SimpleQA is 0.759 versus 0.688 and NQ EM is 0.335 versus 0.305, while MMSearch is 0.253 versus 0.250. Reranking changes this ordering. It raises the fused setting from 0.759 to 0.798 on SimpleQA and from 0.247 to 0.271 on NQ-Tables EM, and lifts the dual-view further, from 0.688 to 0.810 on SimpleQA and from 0.225 to 0.298 on NQ-Tables EM. With reranking, the dual-view therefore becomes the stronger choice on most benchmarks at k=3k = 3, leading on SimpleQA (0.810), NQ EM (0.355), and NQ-Tables EM (0.298), whereas the fused view keeps the best EVQA score (0.435). Increasing the reading depth from one item to three also improves both settings on most benchmarks: for the dual-view with reranking, SimpleQA rises from 0.784 to 0.810, NQ EM from 0.316 to 0.355, and NQ-Tables EM from 0.252 to 0.298.

Agentic Search on MoNaCo

Setting and metric. We evaluate our retrieval backends as the search tool of a GPT-5 ReAct agent on the full MoNaCo benchmark, 1,315 multi-hop Wikipedia questions. Only the search backend varies across conditions; the agent loop, the prompt, and the GPT-5 reader are identical, and every run completed all 1,315 questions. The agent issues up to 15 search queries per question with three results per query, a shorter budget than the 20 queries with five hits each of PixelRAG’s reference protocol (Wang et al., 2026). MoNaCo answers are frequently multi-value lists, so we report judge-F1, computed from an LLM judge that scores each predicted answer against the reference answers. Table 15 reports judge-F1 on a 0–1 scale together with agent API cost and retrieval GPU time.

Retrieval view(s)IndexRerankJudge-F1API cost (USD)GPU time (h)
Text–no0.4755$57.43 | 18.2
Imagebaseno0.4673–91.6
Dual-viewbaseno0.4708$57.26 | 25.2
Dual-viewbaseyes0.4792$57.82 | 24.7
Dual-viewalignedno0.4816$57.44 | 16.3
Dual-viewalignedyes0.4688–18.1

Table 15. MoNaCo results, 1,315 questions per run. Text uses the transcription index and image the base screenshot index; dual-view adds a text path to either screenshot index, and ours is the index that encodes each tile with its transcription. Judge-F1 is on a 0–1 scale; bold marks the best value.

Backends. Text-only search uses the transcription index, and image-only search the base screenshot index. Dual-view search unions a screenshot path with a second path that searches the transcriptions as text; the screenshot path uses either the base index or ours, which encodes each tile together with its transcription so that a single retrieved item carries both the rendered evidence and its text.

Results. Judge-F1 ranges from 0.4673 to 0.4816 across the six runs. The best score comes from dual-view search on our index without reranking (0.4816). At the same retrieval design, our index adds 0.0108 over the base index (0.4708 to 0.4816), the largest effect in the table. Text-only search is competitive at 0.4755 and exceeds image-only search on the base index (0.4673), which matches our expectation that the transcription view contributes evidence a screenshot-only index can miss; combining the two views on our index then gives the best result. Reranking behaves differently on the two indexes: it raises the base-index dual-view setting from 0.4708 to 0.4792, but it does not help on our index (0.4816 to 0.4688), so its benefit is largest where first-stage retrieval is weaker.

Cost. Agent-side cost varies little across the backends with complete usage: text-only search and the dual-view runs each accumulate 9.3–9.8M prompt tokens and cost $57.26–$57.82 per run. Adding the text view therefore leaves the agent’s conversation length essentially unchanged, because every retrieved item remains a screenshot tile rather than an extracted passage. Each reranker invocation adds about 1.2–1.4 seconds. On the retrieval side, our index is cheaper than the base index for the same design: 16.3 versus 25.2 hours for dual-view, and 18.1 versus 24.7 hours with reranking.

References

  1. [1]extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 122–131, Online. Association for Computational Linguistics.
  2. [2]Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. 2022. WebQA: Multihop and multimodal QA. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16495–16504.arxiv.org/abs/2109.00590
  3. [3]Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William Cohen. 2022. MuRAG: Multimodal retrieval-augmented generator for open question answering over images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5558–5570. Association for Computational Linguistics.arxiv.org/abs/2210.02928
  4. [4]Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021. WebSRC: A dataset for web-based structural reading comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4173–4185. Association for Computational Linguistics.arxiv.org/abs/2101.09465
  5. [5]Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. 2009. Reciprocal rank selection outperforms Condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 758–759. Association for Computing Machinery.DOI
  6. [6]Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2024. ColPali: Efficient document retrieval with vision language models. Preprint, arXiv:2407.01449.arxiv.org/abs/2407.01449
  7. [7]Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. 2021. Scaling deep contrastive learning batch size under memory limited setup. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 316–321. Association for Computational Linguistics.arxiv.org/abs/2101.06983
  8. [9]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
  9. [10]Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. LayoutLMv3: Pre-training for document AI with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4083–4091. Association for Computing Machinery. ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8988–9003. Association for Computational Linguistics.
  10. [11]Hao Yang, Yifan Ji, Zhipeng Xu, Zhenghao Liu, Yukun Yan, Zulong Chen, Shuo Wang, Yu Gu, and Ge Yu.

Paper details

vlnc:W_ffnne58q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents