betaFeedback

WorksComputer Science

Faithful in Steps: Improving Generalization and Citation in RAG via Query Decomposition

Abstract

Retrieval-augment generation is a prevalent strategy to mitigate hallucinations of LLMs. The attributable RAG (RAGQ) generates quotes for its answers. The quotes indicate which input contexts support the RAG to derive the answers, enhancing the answer's verifiability and trustworthiness. However, existing RAGQs exhibit significant degradation when dealing with questions that require multi-hop reasoning and multi-modal understanding, suffering from over-citation, implicit entity identification failure, and poor generalization. In this paper, we propose a novel RAGQ framework, namely QDRAG. QDRAG breaks down the input question into atomic subquestions to identify the implicit entities. Then, the reranker prunes context distractors to eliminate the downstream over-citation. To facilitate query decomposition, we propose two zero-shot approaches: QD-C and QD-R, which guide the QD MLLM to decompose the question based on context knowledge and retrieval rewards, respectively. One interesting finding is that finetuning on the QD task shows better generalizability compared to directly finetuning on the downstream RAGQ task. Experiments on four multi-modal QA benchmarks demonstrate QDRAG's efficacy in grounding answers and generating faithful citations. The framework significantly outperforms all the baselines on both in-domain and out-of-domain tests, even surpassing Gemini-Pro.

Introduction

Retrieval-augmented generation (RAG) ([1]) is a common method to mitigate hallucinations in large-language models. RAG retrieves external data and grounds its answer on the retrieved contexts. Recent studies ([27] (Huang et al. 2024)) propose RAG with quotes (RAGQ) to improve answer faithfulness. RAGQ outputs include answers and verifiable source references that indicate which retrieved data support the LLM’s outputs. Users can verify answers via cited sources, and this alignment between responses and sources is key to answer integrity.

Despite recent advancements, current RAGQs still face challenges for multi-hop and multi-modal questions.

C-1: Over-citation of incorrect contexts. The RAGQ cites multi-modal contexts where the local segments align with the answer but neglects to evaluate the broader relevance of these contexts. C-2: Implicit entity retrieval failure. The post-hoc retrieval approaches ([12]) rely on retrievers to search for supporting contexts, but simply using the original text of the question as a query often fails due to the key entities being absent in the multi-hop question. C-3: Poor generalization ability. Instruction fine-tuning methods ([15, 27]) finetune LLMs for downstream tasks. They build training data to enhance LLMs’ attribution capability, leading to difficulties in maintaining reference accuracy for dynamic multimedia knowledge (Bi et al. 2024).

One intuitive solution is to use sparse RAG ([38]) to pre-filter contexts to a shorter list before MLLM generation, which assesses context relevance in parallel. However, for multi-hop questions, sparse RAG misjudges the contexts with partial knowledge as “irrelevant” due to C-2.

Another research line decomposes the question into a reasoning plan by few-shot in-context learning. The domain and difficulty gaps between examples and real questions make the MLLM hard to learn how to effectively simplify the multi-hop question [35, 18].

To address the above challenges, we propose a novel RAGQ framework named QDRAG, which consists of three main modules: query decomposer (QD), context reranker, and generator. Different from the static contexts used in existing RAGQs, QDRAG filters distractors before answer generation. Sparse contexts help MLLMs focus on correct contexts and generate better quotes (C-1). The QD-reranking paradigm assesses context relevance to single-hop subquestions, thereby resolving C-2. For query decomposition, we propose zero-shot QD-C and QD-R, two new QD models that eliminate manually written in-context examples. QD-C breaks the QD task into two clearly defined operators: entity replacement and open decomposition. QD-C is plug-and-play for existing RAGs. QD-R uses the recall rate of golden contexts as the reward of a decomposition plan to perform direct preference optimization on the QD task. The retriever feedback significantly boosts the automatic annotation process compared to LLM feedback used by RAGQs. Moreover, the inputs of the decomposition task (a multi-hop question) are context-agnostic and answer-agnostic. The finetuned QDRAG thus does not suffer from the massive and dynamic multimedia contexts. The existence of common decomposition logic among multi-hop questions makes QDRAG more generalizable than existing RAGQs (C-3). Our contributions are summarized as:

  • We highlight the close relationship between context sparsity and MLLM context reference capability. It motivates our new QD-Reranking-based RAGQ pipeline.

  • We propose two query decomposition methods to identify implicit entities on multi-hop questions, and subquestion-based context reranking to address the over-citation problem on multi-modal contexts. We enhance the model generalizability by finetuning QDRAG on the QD task.

  • Extensive experiments demonstrate that QDRAG reduces answer and citation hallucinations and demonstrates remarkable generalization across different base models.

RAG with Quotes (RAGQ).

The training-free RAGQs use post-hoc retrieval [13], chain-of-thought prompting (Ji et al. 2024) or dynamic programming [9] to select supporting documents. For training-based methods, [2, 32] perform supervised-finetuning on RAG LLMs to enable context citations. [27] (Huang et al. 2024) employs reinforcement learning with human feedback or direct preference optimization. Self-Cite [10] utilizes the change of LLM’s confidence on an answer when removing the cited documents to calculate the context’s importance. In contrast to previous work, QDRAG addresses the RAGQ task by performing query decomposition and context filtering to reduce the citation candidates and hallucinations.

Figure 2. QDRAG finetunes the MLLM for the QD task, leveraging its fast training and better generalizability.

LLM for Reranking.

There are four categories in this research line: pointwise, pairwise, listwise, and setwise. The pointwise approach evaluates context relevance one by one. The score can either be a binary label “yes/no”, or the normalized likelihood of generating a “yes” response [38] (Yu et al. 2024). Listwise methods prompt an LLM to output the ordered sequence [28]. Pairwise prompts let the LLM decide which of two contexts is more relevant [25]. Setwise approach directly outputs the relevant set of contexts [39] (Yu et al. 2024). Among the four categories, the pointwise method is the only one that is parallelizable and also has comparable performance with others [39].

LLM for Query Decomposition.

Least-to-Most [36], Decomp [18], Dynamic Least-to-Most [11], and Self-Ask [24] use in-context demonstrations to iteratively decompose the question and answer the subquestions step by step. Recently, SearChain (Xu et al. 2024) adopts the decomposition plan as the generation unit and iteratively refines the plan, surpassing PS [31] on multi-hop question answering.

Methodology

Preliminaries

Vanilla Workflow of RAGQ. Given a user query QQ, the multi-modal RAGQ retrieves visual and text knowledge pieces from external databases. Then, it merges the knowledge pieces with QQ as the input context. Next, a multi-modal large language model (MLLM) is prompted to produce an answer with in-line citations in the form of [1][2]. The goal of RAGQ (RAG with quotes) is to ensure that the answer is supported by the cited knowledge sources.

Multi-hop Question Answering

Answering a multi-hop question needs several reasoning steps Trivedi et al. [30]. The question can be decomposed repeatedly until it can be answered by a single external document. A subquestion QiQ_i can depend on the answer to another subquestion QjQ_j. In Figure 1, if QiQ_i = “Who is the voice artist?”, and QjQ_j = “⟨Answer of QiQ_i⟩ has what color hair?”, then QjQ_j relies on QiQ_i. There are also independent subquestions. For instance, the question “Which film was released first: ⟨aa⟩ or ⟨bb⟩?” has two independent subquestions: QiQ_i = “When was ⟨aa⟩ released?” and QjQ_j = “When was ⟨bb⟩ released?”.

Figure 1. The massive retrieved data in RAG misleads the MLLM to cite the incorrect but similar contexts; Sparse RAG drops key contexts in the reranking step for multi-hop questions; QDRAG filters the distractors while maintaining the key contexts for accurate downstream citations.

Sparse Context Benefits RAG Quotes

We explore the impact of context sparsity on MLLM’s reference performance through two tasks. Our findings show that the sparse RAGs generate better citations than dense RAGs.

The first task is to cite images from an expression of interleaved images and texts. It simulates the scenario that a RAGQ is about to generate citations after the answers. Specifically, the input template is “Compared with the image IRI_R, the image has ⟨aa⟩” and a list of images “[1] Image; [2] Image ...”. The outputs are image IDs (e.g., [1][3]) that support the statement. We test on two image retrieval datasets [4, 21]. In Figure 3, the results show that reducing input distractors helps RAGQ reduce the risk of citing wrong images (higher precision) and cite a more complete context set (higher recall). Therefore, sparse RAG is a promising strategy to enhance RAG citations.

Precision and recall of image citation on CIRCO and CIRR

Figure 3. Evaluate MLLM’s capability to cite images from a multi-modal statement on two datasets: CIRCO and CIRR.

The second experiment compares a state-of-the-art sparse RAG [38] with dense RAG (w/o context filtering) on simultaneously answering the question and citing the multi-modal contexts. Figure 4a and Figure 4b show that the sparse RAG produces better answers and citations than dense RAG on single-hop questions. However, in Figure 4c and Figure 4d, directly applying sparse RAG for multi-hop questions leads to lower answer quality. It implies that sparse RAG drops golden contexts and suffers from hallucinations.

Dense RAG vs Sparse RAG on answer accuracy and citation attribution

Figure 4. Dense RAG vs Sparse RAG in terms of answer accuracy and attribution capability. (a),(b): Sparse RAG shows superiority on both metrics; (c),(d): Sparse RAG still excels on attribution performance, but has worse answer quality.

Discussion. We emphasized that sparse RAGs generate better quotes than dense RAGs. We found that the existing sparse RAGs (a.k.a. reranking models) excel in single-hop ques-

tion answering and citation generation, but face challenges for multi-hop questions. Thus, we propose a novel RAGQ, namely QDRAG, which is superior in both single-hop and multi-hop scenarios.

QDRAG Overview

As shown in Figure 1, QDRAG comprises three main modules: query decomposer, context reranker, and answer generator. The decomposer derives multiple subquestions, and the reranker purifies the retrieved data by their relevance to subquestions. Afterwards, the answers and citations are generated auto-regressively by the MLLM.

QD-C: Context-Guided Question Decomposition

QD-C defines two operators for query decomposition: entity replacement and open decomposition for inter-dependent and independent subquestions, respectively.

Entity Replacement. We instruct the MLLM to identify any referential expressions (e.g., “The voice artist of the girl mammoth in Ice Age: Continental Drift”) within the question. Then, the MLLM utilizes the external knowledge to substitute references with entities (e.g., “Keke Palmer”). The operator produces two subquestions for each replacement. One is “Is ⟨reference⟩ equivalent to ⟨entity⟩?”, the other is the new question after substitution.

Open Decomposition. The MLLM is tasked with allocating queried entities to multiple subquestions to enable answering with a single context. For instance, for question “Which one released first, A or B?”, the operator produces two subquestions regarding the released year of two entities. This approach safeguards against the reranker disregarding contexts that contain partial information (e.g., the release year) without explicit comparison.

QD-C eliminates manually-written in-context examples by the well-defined operators. Moreover, the two operators can be implemented into a single MLLM inference for each multi-hop question, which makes QD-C cost-effective and plug-and-play.

QD-R: Efficient and Generalizable QD Alignment

Motivation. Similar to other single-round retrieval RAGs, QD-C assumes that the retrieved documents contain sufficient information to decompose the question. However, a single round of lightweight retrieval often fails to recall the complete set of golden contexts (Yu et al. 2024), which leads to cascading errors that degrade answer and citation quality.

QD-R. We propose retrieval-guided QD (QD-R), a new query decomposition method with multi-round retrievals. Firstly, QD-R takes only the query as input and performs zero-shot query decomposition. The output of QD-R is simply a decomposition plan comprising multiple subquestions. QD-R allows the special token “⟨Ans_of_Qi⟩” in subquestions, which means that the content of the subquestion depends on the answer of previous subquestions. Then, QD-R resolves the placeholders via RAG and replaces the special tokens with the answers to that question. Finally, different from QD-C, each resolved subquestion retrieves its own set of contexts for subsequent reranking.

QDARF. QD-R regularizes its decompositions by Aligning MLLM from Retrieval Feedback (QDARF). As shown in Figure 5, we sample multiple decomposition plans from a teacher MLLM (Qwen2.5VL-32B) and a student MLLM (Qwen2.5VL-7B). The reward of a decomposition plan is defined as the recall of golden contexts by all its subquestions. We adopt the direct preference optimization (DPO) loss function to finetune the MLLM:

Figure 5. QD-R samples multiple MLLM outputs and uses the recall of golden contexts as reward to train the model.

L=−E(x,yw,yl)∼D[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))](1)\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right)\right] \tag*{(1)}

where ywy_w are yly_l are the positive (winning) and the negative (losing) decomposition plans, respectively. The decompositions with the highest reward values are deemed as positive, while those with the lowest reward values are treated as negative. πref\pi_{\mathrm{ref}} is the initial MLLM and πθ\pi_\theta is the MLLM to optimize. Optimizing Eq. (1) increases the margin between the positive and negative plans. Consequently, the subquestions would have a larger recall rate of golden contexts after finetuning, which benefits the downstream reranking task.

Context Reranking

QDRAG evaluates the context’s relevance to each subquestion and the raw question. The relevance score of a context CC to a subquestion QiQ_i is defined as the probability of “Yes” as the next token under the reranking prompt RR:

S(C∣Qi)=P(“Yes”∣Qi,C,R)S(C\mid Q_i)=\mathcal{P}(\text{“Yes”}\mid Q_i,C,R)

Each subquestion retains a certain number of top-relevant contexts.

Adaptive Budget Allocator

Given a budget NN, how to distribute the number of reserved contexts for each subquestion? We propose an entropy-based method. The idea is that higher entropy implies that the MLLM is less certain about selecting the best supporting document, hence requiring more contexts to be retained for that subquestion. Formally,

P(C∣Qi)=exp⁡(S(C∣Qi))∑C′∈Cexp⁡(S(C′∣Qi))\mathcal{P}(C\mid Q_i)=\frac{\exp(S(C\mid Q_i))}{\sum_{C'\in\mathcal{C}}\exp(S(C'\mid Q_i))}
E(Qi)=−∑C′∈CP(C′∣Qi)log⁡P(C′∣Qi)\mathcal{E}(Q_i)=-\sum_{C'\in\mathcal{C}}\mathcal{P}(C'\mid Q_i)\log\mathcal{P}(C'\mid Q_i)

Then, the number of reserved contexts allocated to QiQ_i is

n(Qi)=N⋅E(Qi)∑Qj∈QE(Qj)n(Q_i)=N\cdot\frac{\mathcal{E}(Q_i)}{\sum_{Q_j\in\mathcal{Q}}\mathcal{E}(Q_j)}

Experiment

Experimental Setup

Datasets. We conduct evaluations on four multi-modal question answering datasets: MMQA [29], WebQA [6], InfoSeek [7] and E-VQA [23]. MMQA and WebQA consist of 1,196 and 4,967 testing queries in text form, respectively. Each query necessitates one or more image or text fragments for a complete answer. InfoSeek and E-VQA questions task the models with entity recognition in images and retrieval of relevant textual information for answering.

Evaluation Metrics. We evaluate the RAG outputs from two dimensions: answer quality and context reference faithfulness. For answer quality, we use exact match (EM) and scores graded by LLM-judge. EM measures the ratio of keywords in the ground-truth answer that are exactly mentioned in the model’s response. LLM-judge prompts an MLLM ψ\psi to grade the correctness and comprehensiveness of RAG outputs given the golden answers. For context reference capability, following existing work [27], we perform AutoAIS evaluation using an off-the-shelf MLLM ψ\psi. The inputs to ψ\psi include the user question, the model-generated

Baselines. As shown in Table 1, the main experiment contains four categories of baselines. The first category comprises dense RAGs with recent MLLM backbones, including the commercial Google Gemini2.5-Pro Kavukcuoglu [17] and the open-sourced Qwen2.5-VL Bai et al. [3] and InternVL-3 Zhu et al. [37]. Following (Huang et al. 2024), this category also contains post-hoc retrieval baselines that select reference contexts with the CLIP retriever. The second category uses multi-modal rerankers to implement sparse RAGs, including CLIP Radford et al. [26], RagVL Chen et al. [8], MMEmbed Lin et al. [20], and SparseRAG Zhu et al. [38]. The third category is QD models with single-round retrieval, including LlamaIndex2 and the proposed QDRAG-C. The last category compares the query decomposition methods under the QDRAG-R pipeline. The baselines include Self-Ask Press et al. [24] and SearChain (Xu et al. 2024). In ablation study, we further compare QDRAG with rewriting models Mao et al. [22] and the instruction-finetuned RAQGs (Huang et al. 2024).

ModelModel SizeMMQA EMMMQA LLMMMQA CiteWebQA EMWebQA LLMWebQA CiteInfoSeek EMInfoSeek LLMInfoSeek CiteE-VQA EME-VQA LLME-VQA Cite
Gemini-Pro-43.234.268.455.126.664.69.80.00.96.13.423.6
Gemini-ProCLIP_{\mathrm{CLIP}}---60.4--57.9--0.4--27.9
Qwen2.5-VL7B42.440.638.257.841.533.042.536.759.610.47.818.5
Qwen2.5-VLCLIP_{\mathrm{CLIP}}7B--47.2--35.5--58.4--26.5
InternVL314B49.150.150.958.343.644.536.030.552.28.57.315.9
InternVL3CLIP_{\mathrm{CLIP}}14B--47.6--39.7--52.2--16.6
CLIP428M/7B35.032.439.550.922.936.840.732.456.09.26.516.1
MMEmbed7B43.641.251.455.936.744.741.835.660.99.56.922.0
Sparse-RAG7B45.946.148.157.646.139.044.535.158.914.611.615.7
RagVL13B/7B42.340.850.956.138.943.4------
LlamaIndex7B43.342.150.056.440.844.110.42.13.213.110.720.0
QDRAG-C (ours)7B43.342.252.656.740.245.842.235.462.19.67.521.2
QDRAG-C (ours)14B46.846.955.857.439.748.636.928.756.56.55.515.3
Self-Ask7B42.941.949.956.240.143.842.734.560.69.78.421.1
SearChain7B43.742.649.356.340.645.142.434.364.010.08.021.7
QDRAG-R (ours)7B49.246.357.561.748.059.946.038.372.916.212.833.8
w/o DPO7B46.745.453.861.048.356.842.035.266.313.410.628.9
QDRAG-R (ours)14B53.152.865.662.653.570.939.832.468.715.712.340.4

Table 1. Overall in-domain and out-of-domain test, where best and second-best results are highlighted in bold and _underlined_, respectively. The retriever for all models is CLIP-ViT-L/14@336px. Single model size means that QD/Reranking/QA use the same MLLM, while multiple model sizes (e.g., 13B/7B) means the reranker model size and the QD/QA model size, respectively.

Implementation Details. If not specified, the backbone of 7B RAGs and 14B RAGs are Qwen2.5-VL-7B Bai et al. [3] and InternVL3-14B Zhu et al. [37], respectively. The reranking models reserve top-5 contexts. We adopt low-rank adaptation (LoRA) (Hu et al. 2022) when finetuning the MLLM. The model inference is implemented using the vLLM v0.9.2 framework Kwon et al. [19]. The experiments are conducted on a Ubuntu server with 2×2 \times A100 80B NVIDIA GPUs. For detailed hyperparameters and prompts, we refer the reader to the appendix.

Main Results

Context Filtering Boosts Citation Quality. As shown in Table 1, reranking RAGs consistency show improvements over both prompt-based and post-hoc retrieval baselines. This highlights our proposition that the sparse context helps MLLMs better attribute their answer to supporting contexts. Comparing CLIP with other rerankers reveals that relying solely on the retrieval score for reranking is suboptimal.

QDRAG-R Achieves Significant Improvements on In-domain and Out-of-Domain Tests. Our QDRAG-R models are trained on the training splits of MMQA and WebQA. InfoSeek and E-VQA have image inputs in all user queries, while MMQA and WebQA only contain pure-text queries. The RAG knowledge bases are also different. Therefore, our training data exhibits out-of-domain characteristics. Nonetheless, QDRAG-R demonstrates a substantial performance advantage on both answer quality and attribution capability over existing reranking RAGs and QD-reranking RAGs. The QDRAG-R-14B outperforms the proprietary MLLM Gemini2.5-pro in most cases.

QDRAG-C Excels with More Powerful Retrievers. In Table 1, QDRAG-C shines at resource attribution but is less splashy for answer quality. This highlights our proposition that QDRAG-C is potentially sensitive to initial retrieval quality. In Table 2, we replace the lightweight CLIP retriever with MMEmbed-7B. Then QDRAG-C demonstrates improvements in both answer quality and attribution capability.

MMQA EMMMQA LLMMMQA CiteWebQA EMWebQA LLMWebQA Cite
RAG67.764.151.370.764.060.9
MMEmbed69.765.171.568.057.169.4
SparseRAG67.966.171.968.560.965.3
LlamaIndex68.065.769.969.661.272.1
QDRAG-C70.766.474.770.763.775.1

Table 2. Comparison of single-round RAGs with 7B retriever response, and the cited contexts. The AutoAIS output is a True/False label indicating whether the aggregated contexts support the model response. In this paper, we set ψ\psi as Qwen2.5VL-32B Bai et al. [3].

Analysis

Rewriting-R vs QD-R Rewriting is another important query processing strategy. We compare QD-reranking with the recent query rewriting method RaFe [22] by adapting it for RaFe-reranking. As shown in Table 3, in

MMQAMMQAMMQAWebQAWebQAWebQA
EMLLMCiteEMLLMCite
RaFe-C43.342.249.955.640.344.4
QD-C43.342.252.656.740.245.8
RaFe-R43.642.049.355.738.742.3
RaFe-RDPO_{\mathrm{DPO}}43.842.750.257.441.748.4
QD-R46.745.453.861.048.356.8
QD-RDPO_{\mathrm{DPO}}49.246.357.561.748.059.9

Table 3. Compare QD with rewriting models

single-round retrieval setup, QD-C shows improvement compared to RaFe-C. In multi-round retrieval setup, we construct the RaFe training dataset similar to that of QD-R. The only difference is that RaFe instructs the MLLM to rewrite the query. The results show that query decomposition is indeed an essential step for RAG trustworthiness and answer quality.

Finetuning Task We compare QDRAG with a representative RAGQ finetuning method: Front (Huang et al. 2024). Front retrieves contexts for training questions and uses a teacher model and a student model to generate positive and negative answers with citations, respectively. For fair comparison, we use the same teacher model and student model as QDRAG. The results are shown in Table 4. Similar to published results [27] (Huang et al. 2024), we observe descending answer accuracy of RAGQ on out-of-domain datasets. QDRAG shows a significantly better generalization capability on out-of-domain datasets.

PipeFT taskInfoSeekInfoSeekEVQAEVQA
LLMCiteLLMCite
-36.759.67.818.5
RAGRAGQ32.955.66.418.7
QD-R36.662.37.819.1
-35.266.310.628.9
QDRAG-RRAGQ33.068.48.626.0
QD-R38.372.912.833.8

Table 4. Out-of-domain test: Front vs QDRAG

In Table 5, we compare the annotation cost of Front and QDRAG. The results show that the retrieval feedback is significantly more efficient than MLLM feedback.

FT taskFeedbackAutoCost
RAGQLLM-judge✓28h
QD-RRetrieval✓2h

Table 5. Cost of feedback when building training data

What Makes QD-R Better? In Figure 6, we divide the inputs into four cases for each model. Our key findings are: (1) most incorrect references are caused by incorrect context inputs, which highlights the importance of QD-based context reranking; (2) QD-R significantly reduces the failure cases.

Failure case analysis for MMQA and WebQA

Figure 6. Failure case analysis. IWCW ↓: Input wrong context and cite incorrectly; IWCC ↑: Input wrong context but cite correctly; ICCW ↓: Input correct context but cite incorrectly; ICCC ↑: Input correct context and cite correctly.

Ablation Study We conduct a set of ablations of QDRAG to identify which factors play key roles. We examine the benefits of the replacement (R) and the decomposition (D) operators in QD-C. From Table 6, we observe that both of them are critical for effective query decomposition. Compared with no query decomposition, QD-C shows a large performance gain. We test the contributions of QD-R training datasets in Table 7. The result shows QD-R’s scalability w.r.t. the training data size. In both models, the adaptive context number assignment module (S) provides performance gains.

DRSMMQA EMMMQA LLMMMQA CiteWebQA EMWebQA LLMWebQA Cite
✓✓✓43.342.252.656.740.245.8
✓✓43.341.851.056.840.845.4
✓✓43.841.450.956.939.445.2
✓✓44.042.052.056.540.745.8
43.542.048.855.839.841.9

Table 6. Ablation on QDRAG-C

MWSMMQA EMMMQA LLMMMQA CiteWebQA EMWebQA LLMWebQA Cite
✓✓✓49.246.357.561.748.059.9
✓✓46.944.656.961.148.156.9
✓✓46.445.454.361.648.358.3
✓✓48.846.056.561.947.860.6
46.945.054.861.048.156.9

Table 7. Ablation on QDRAG-R Traing Data

Efficiency & Cost In Figure 8a, we analyze the efficiency and scalability w.r.t. the number of retrieved contexts at each retrieval round. The dashed lines indicate the linear growth of running time. Compared with baselines, our method substantially boosts task performance without significantly increasing time consumption compared to the baselines. Notably, QDRAGs generate more effective subquestions with less running time.

Analysis of efficiency, scalability, and query decomposition costs

Figure 8. Efficient & cost analysis

In Figure 8b, we analyze the costs of query decomposition in dollars. The reported metrics include the average number of input tokens (nn), output tokens (mm), and MLLM calls (rr). The overall cost is calculated by ∑(nrp1+mrp2)\sum(nrp_1 + mrp_2), where p1p_1 and p2p_2 denote the unit price of input and output tokens of Gemini2.5-pro, respectively. QDRAG-C only incurs a single MLLM call to make it cost-effective. QDRAG-R has a slightly higher cost but has higher performance.

Case Study In Figure 7, the baseline decompositions have almost the same level of difficulty with the input question. Our QDRAG successfully simplifies the question step by step. The baselines prompt an MLLM to decompose the query in the open exploration space. This may lead to unsolvable subqueries or rewritings rather than really reducing the question complexity. In our method, we enable MLLM-IR interaction by preference optimization. The MLLM in QDRAG tends to generate reasonable subquestions that have large rewards, i.e., the recall rate of golden contexts. This benefits downstream tasks of question answering and contextual referencing.

Case study showing query decompositions by SearChain and QDRAG

Figure 7. Case study: QDRAG generates better query decompositions. GTR: Golden context Recall.

In Figure 9, we visualize the image context attention that has the largest attention values on the content ID (e.g., Document [4]) when the QA MLLM generates references. The dense RAG spends large attention on a photo of an incorrect person who also has white hair, while QDRAG is more confident and wise when generating reference IDs.

Visualization of image contexts with the largest attention values

Figure 9. Visualization of image contexts with the largest attention values when MLLM generates the references after “What color is Giorgio Armani’s hair? White. According to Document _.”

Conclusion

In this work, we propose QDRAG, a novel RAG framework to enhance attributed RAG. We present that the sparse context helps LLM concentrate on important contexts and point out the limitation of existing sparse RAGs. We propose two new query-decomposition methods. In particular, QD-C is guided by initial contexts, while QD-R is guided by retrieval rewards. QDRAG successfully addresses the three challenges of existing RAGQs. Extensive experiments are conducted to validate the generalizability and scalability of QDRAG.

Acknowledgments

The research work described in this paper was supported by Hong Kong Research Grants Council (grant# T43-513/23-N, T22-607/24N). It was partially conducted in JC STEM Lab of Data Science Foundations funded by The Hong Kong Jockey Club Charities Trust. Shimin Di’s work is supported by National Science Foundation of China (NSFC) under Grant No. 62506075.

References

  1. [1]Abootorabi, M. M.; Zobeiri, A.; Dehghani, M.; Mohammadkhani, M.; Mohammadi, B.; Ghahroodi, O.; Baghshah, M. S.; and Asgari, E. 2025. Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation. arXiv preprint arXiv:2502.08826.arxiv.org/abs/2502.08826
  2. [2]Asai, A.; Wu, Z.; Wang, Y.; Sil, A.; and Hajishirzi, H. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2310.11511
  3. [3]Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923.arxiv.org/abs/2502.13923
  4. [4]Baldrati, A.; Agnolucci, L.; Bertini, M.; and Del Bimbo, A. 2023. Zero-shot Composed Image Retrieval with Textual Inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 15338–15347.arxiv.org/abs/2303.15247
  5. [6]Chang, Y.; Narang, M. B.; Suzuki, H.; Cao, G.; Gao, J.; and Bisk, Y. 2021. WebQA: Multihop and Multimodal QA. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 16474–16483.arxiv.org/abs/2109.00590
  6. [7]Chen, Y.; Hu, H.; Luan, Y.; Sun, H.; Changpinyo, S.; Ritter, A.; and Chang, M.-W. 2023. Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 14948–14968. Singapore: Association for Computational Linguistics.arxiv.org/abs/2302.11713
  7. [8]Chen, Z.; Xu, C.; Qi, Y.; and Guo, J. 2024. MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training. arXiv:2407.21439.arxiv.org/abs/2407.21439
  8. [9]Choi, N.; Byun, G.; Chung, A.; Paek, E. S.; Lee, S.; and Choi, J. D. 2025. Trustworthy Answers, Messier Data: Bridging the Gap in Low-Resource Retrieval-Augmented Generation for Domain Expert Systems. arXiv preprint arXiv:2502.19596.
  9. [10]Chuang, Y.-S.; Cohen-Wang, B.; Shen, Z.; Wu, Z.; Xu, H.; Lin, X. V.; Glass, J. R.; Li, S.-W.; and tau Yih, W. 2025. SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models. In Forty-second International Conference on Machine Learning.arxiv.org/abs/2502.09604
  10. [11]Drozdov, A.; Schärli, N.; Akyürek, E.; Scales, N.; Song, X.; Chen, X.; Bousquet, O.; and Zhou, D. 2023. Compositional Semantic Parsing with Large Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.arxiv.org/abs/2209.15003
  11. [12]Gao, L.; Dai, Z.; Pasupat, P.; Chen, A.; Chaganty, A. T.; Fan, Y.; Zhao, V.; Lao, N.; Lee, H.; Juan, D.-C.; and Guu, K. 2023a. RARR: Research and Revising What Language Models Say, Using Language Models. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 16477–16508. Toronto, Canada: Association for Computational Linguistics.arxiv.org/abs/2210.08726
  12. [13]Gao, T.; Yen, H.; Yu, J.; and Chen, D. 2023b. Enabling Large Language Models to Generate Text with Citations. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 6465–6488. Singapore: Association for Computational Linguistics.arxiv.org/abs/2305.14627
  13. [15]Huang, L.; Feng, X.; Ma, W.; Gu, Y.; Zhong, W.; Feng, X.; Yu, W.; Peng, W.; Tang, D.; Tu, D.; and Qin, B. 2024. Learning Fine-Grained Grounded Citations for Attributed Large Language Models. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics: ACL 2024, 14095–14113. Bangkok, Thailand: Association for Computational Linguistics.arxiv.org/abs/2408.04568
  14. [17]Kavukcuoglu, K. 2025. Gemini 2.5: Our Most Intelligent AI Model. Accessed: 2025-07-21.
  15. [18]Khot, T.; Trivedi, H.; Finlayson, M.; Fu, Y.; Richardson, K.; Clark, P.; and Sabharwal, A. 2023. Decomposed Prompting: A Modular Approach for Solving Complex Tasks. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2210.02406
  16. [19]Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C. H.; Gonzalez, J. E.; Zhang, H.; and Stoica, I. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.arxiv.org/abs/2309.06180
  17. [20]Lin, S.-C.; Lee, C.; Shoeybi, M.; Lin, J.; Catanzaro, B.; and Ping, W. 2025. MM-Embed: Universal Multimodal Retrieval With Multimodal LLMs. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2411.02571
  18. [21]Liu, Z.; Rodriguez-Opazo, C.; Teney, D.; and Gould, S. 2021. Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2125–2134.arxiv.org/abs/2108.04024
  19. [22]Mao, S.; Jiang, Y.; Chen, B.; Li, X.; Wang, P.; Wang, X.; Xie, P.; Huang, F.; Chen, H.; and Zhang, N. 2024. RaFe: Ranking Feedback Improves Query Rewriting for RAG. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Findings of the Association for Computational Linguistics: EMNLP 2024, 884–901. Miami, Florida, USA: Association for Computational Linguistics.arxiv.org/abs/2405.14431
  20. [23]Mensink, T.; Uijlings, J.; Castrejon, L.; Goel, A.; Cadar, F.; Zhou, H.; Sha, F.; Araujo, A.; and Ferrari, V. 2023. Encyclopedic VQA: Visual Questions about Detailed Properties of Fine-grained Categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3113–3124.arxiv.org/abs/2306.09224
  21. [24]Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N.; and Lewis, M. 2023. Measuring and Narrowing the Compositionality Gap in Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 5687–5711. Singapore: Association for Computational Linguistics.arxiv.org/abs/2210.03350
  22. [25]Qin, Z.; Jagerman, R.; Hui, K.; Zhuang, H.; Wu, J.; Yan, L.; Shen, J.; Liu, T.; Liu, J.; Metzler, D.; Wang, X.; and Bendersky, M. 2024. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting. In Duh, K.; Gomez, H.; and Bethard, S., eds., Findings of the Association for Computational Linguistics: NAACL 2024, 1504–1518. Mexico City, Mexico: Association for Computational Linguistics.arxiv.org/abs/2306.17563
  23. [26]Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 8748–8763. PMLR.arxiv.org/abs/2103.00020
  24. [27]Song, M.; Sim, S. H.; Bhardwaj, R.; Chieu, H. L.; Majumder, N.; and Poria, S. 2025. Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2409.11242
  25. [28]Sun, W.; Yan, L. M.; X. X.; Wang, S.; Ren, P.; Chen, Z.; Yin, D.; and Ren, Z. 2023. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 14918–14937. Singapore: Association for Computational Linguistics.arxiv.org/abs/2304.09542
  26. [29]Talmor, A.; Yoran, O.; Catav, A.; Lahav, D.; Wang, Y.; Asai, A.; Ilharco, G.; Hajishirzi, H.; and Bérant, J. 2021. Multi-ModalQA: Complex Question Answering over Text, Tables and Images. In International Conference on Learning Representations.arxiv.org/abs/2104.06039
  27. [30]Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2022. MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10: 539–554.arxiv.org/abs/2108.00573
  28. [31]Wang, L.; Xu, W.; Lan, Y.; Hu, Z.; Lan, Y.; Lee, R. K.; and Lim, E. 2023. Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. In Rogers, A.; Boyd-Graber, J. L.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, 2609–2634. Association for Computational Linguistics.arxiv.org/abs/2305.04091
  29. [32]Xia, S.; Wang, X.; Liang, J.; Zhang, Y.; Zhou, W.; Deng, J.; Yu, F.; and Xiao, Y. 2025. Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generation. In Chiruzzo, L.; Ritter, A.; and Wang, L., eds., Findings of the Association for Computational Linguistics: NAACL 2025, 969–988. Albuquerque, New Mexico: Association for Computational Linguistics. ISBN 979-8-89176-195-7.arxiv.org/abs/2407.01796
  30. [35]Zhou, D.; Schärli, N.; Hou, L.; Wei, J.; Scales, N.; Wang, X.; Schuurmans, D.; Cui, C.; Bousquet, O.; Le, Q. V.; and Chi, E. H. 2023a. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2205.10625
  31. [36]Zhou, D.; Schärli, N.; Hou, L.; Wei, J.; Scales, N.; Wang, X.; Schuurmans, D.; Cui, C.; Bousquet, O.; Le, Q. V.; and Chi, E. H. 2023b. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.arxiv.org/abs/2205.10625
  32. [37]Zhu, J.; Wang, W.; Chen, Z.; Liu, Z.; Ye, S.; Gu, L.; Tian, H.; Duan, Y.; Su, W.; Shao, J.; et al. 2025a. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv preprint arXiv:2504.10479.
  33. [38]Zhu, Y.; Gu, J.-C.; Sikora, C.; Ko, H.; Liu, Y.; Lin, C.-C.; Shu, L.; Luo, L.; Meng, L.; Liu, B.; and Chen, J. 2025b. Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2405.16178
  34. [39]Zhuang, S.; Zhuang, H.; Koopman, B.; and Zuccon, G. 2024. A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, 38–47. New York, NY, USA: Association for Computing Machinery. ISBN 9798400704314.DOI

Paper details

vlnc:W_yv59vbe2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents