WorksComputer Science
RE-SciVQA: Improving Scientific VQA With Context-Relevant Retrieval and Lightweight MLLM Adaptation
Abstract
A deep understanding of scientific literature often relies on interpreting figures that contain key information beyond the main text. Leveraging Large Language Models (LLMs) to extract insights from these figures can significantly accelerate knowledge discovery. Existing works on Scientific Visual Question Answering rely on resource-intensive training pipelines that are often inaccessible and require large computational resources. Instead of designing complex architectures, we propose RE- SciVQA, which focuses on improving the quality of retrieved context and fine-tuning small LLMs with parameter-efficient LoRA training. Using a simple query expansion step, relevant section headers are appended to the original question to improve the retrieval coverage. In addition, a simple adjustment of similarity scores is applied using figure references and caption alignment. On the SciVQA benchmark, RE-SciVQA achieves a ROUGE-1 F1 score of 73.9%, outperforming chart-focused models such as UniChart (14.7%), ChartGemma (6.9%), and Matcha-ChartQA (6.3%). Our relevance-enhanced RAG setup also improves the exact match by 11.9% and the fuzzy match by 29.5% compared with a standard RAG baseline, resulting in an overall gain of 14.3%. We also examine closed-ended infinite visual questions, a category in which chart-understanding models often struggle. Our experiments show that using plot digitization instead of synthetic captions leads to an 18.5% improvement for this question type.
Introduction
In a Visual Question Answering (VQA) system [1, 2, 3], a text-based question and an image are provided to a vision-language model (VLM) to help it extract and interpret relevant information from the visual content [4]. The image is often paired with a caption, which enhances the model’s ability to understand the visual context by integrating multiple modalities. Scientific Visual Question Answering (SciVQA) involves answering questions based on figures from scientific papers [5, 6, 7]. In this context, Multimodal Large Language Models (MLLMs) are given a scientific figure, its caption, and a question. The model’s task is to select the correct answer from a set of predefined, closed-ended options.
In recent years, many studies have focused on Chart Question Answering (ChartQA) [8, 9, 10]. However, understanding scientific visual questions remains relatively underexplored [9, 11]. In scientific papers, the main idea is typically demonstrated through visuals such as charts, figures, and diagrams. The information in these scientific papers is often spread across different sections of a paper [12]. As a result, tasks in the scientific domain require precise numerical reasoning and the ability to interpret complex visual layouts alongside detailed textual descriptions.
Recently, there has been a clear shift towards smaller and more efficient Multimodal Large Language Models (MLLMs) [13, 14], such as TinyChart (3B) and ChartLlama [15]. This trend is driven by the high computational cost and resource demands of very large models (often exceeding 70B parameters). However, using smaller models typically comes with a performance trade-off, as they still lag behind larger models in accuracy. Moreover, when it comes to scientific diagrams, even with the latest model and a large dataset, researchers are still struggling [5].
LLM visualizes a chart or a graph as an image patch, sometimes it can not find the bar that is located at exact coordinates corresponding to the label on its and axes [15]. Moreover, MLLM often guesses while not calculating the actual result. Thus, previous work, such as TinyChart [14] demonstrated “Program-of-Thought” instead of “Chain-of-Thought” to solve this. Also, for most scientific charts, we have images but not the sources (i.e., Excel/CSV files) to generate the image [16]. That also makes it difficult to get accurate results.
In addition, recent research reveals that Retrieval-Augmented Generation (RAG) is found to be effective for enhancing static models. Bonomo et al. [18] proposed Visual RAG, compared to the most recent state of the art which can obtain an accuracy higher while using a much smaller set of demonstrating examples. In addition, we can see the RAG-grounded VLM performed better, whereas the same VLM without retrieval performed worse in both accuracy and precision [19]. Bhat et al. [20] introduced RAVQA-VLM, which achieved significant improvements targeting closedended visual, nonvisual questions over scientific figures and efficient computing which gave us significant proof of RAG’s potential for Scientific Visual Question Answering (SciVQA).
In this study, we propose an architecture RE-SciVQA that achieves a good accuracy for SciVQA with a resource effective model. We adopt a retrieval centric architecture to maximize our performance with fine-tuning Qwen-3 4B with QLoRA. Figure 1 presents the performance comparison between RE-SciVQA and other MLLMS. Furthermore, we carried out experiments to demonstrate whether visual grounding could be alleviated by captioning through ChartGemma and Deplot. The key contributions of this study are listed as follows:

Figure 1. Comparative analysis of model performance on the RE-SciVQA benchmark. The chart plots the performance of various models based on the ROUGE-1 F1 score [17] (%). RE-SciVQA achieves a peak performance of 73.9%, significantly outperforming other evaluated models such as UniChart (14.7%), ChartGemma (6.9%), and Matcha-ChartQA (6.3%).
We propose RE-SciVQA, which focuses on improving the quality of retrieved context and fine-tuning small LLMs with parameter-efficient LoRA training.
RE-SciVQA(4B) outperforms smaller models such as ChartGemma (R-1 F1 = 0.069) and Matcha-Chartqa (R-1 F1 = 0.063), improving their performance by 10.1x and 11.7x, respectively. This demonstrates the effectiveness of our retrieval-centric fine-tuning approach.
RE-SciVQA(4B) model achieves an R-1 F1 score of 0.739 and R-L F1 of 0.738, which is comparable to larger models such as InternVL3 (8B) (R-1 F1 = 0.740) and Qwen2.5-VL (7B) (R-1 F1 = 0.695), despite being much smaller.
Related Work
Recent developments in chart based question answering pivoted towards instruction tuning on diverse chart based datasets to enhance interpretation of charts. In particular, ChartGemma [21] achieved competitive performance by integrating a specialized encoder along with an LLM based decoder. However, these chart understanding models often exhibit good performance on benchmark datasets, but their performance degrades in the scientific domain as shown in Figure 1.
To address these domain specific challenges, prior work on Scientific Visual Question Answering involved the use of advance prompting techniques such as chain of thought alongside designing an ensemble technique to select the most suitable VLM for figure type [22]. However, this prompting technique is resource intensive and also results in inference latency.
Integrating Retrieval-Augmented Generation (RAG) with Vision-Language Models is a promising direction for handling knowledge-intensive tasks. Bhat et al. [20] successfully combined a fine-tuned InternVL-3-14B model with dense retrieval to answer scientific questions. However, Bhat et al. reported that their RAG system still suffered from hallucinations due to the retrieval of irrelevant or insufficient context. A critical limitation of this retrieval mechanism is the reliance on regular expression (Regex) to parse the scientific paper. This text based extraction ignores the variance in scientific layouts.
Preserving the semantic structure of scientific documents is a prerequisite for effective multimodal reasoning. Pfitzmann et al. [23] argue that accurate layout analysis is critical, as traditional parsers often disrupt the logical flow between text and figures [24], [25]. To address this, recent visual-layout detection models such as DocLayNet [23] have allowed precise identification of elements such as headers, captions, and figures. To address the need for structured inputs, we utilize Docling [26], [27], an open-source toolkit built upon DocLayNet, which enables us to parse different formats of documents to a specific concise layout efficiently.
Beyond structural accuracy, it is also important to consider efficiency. Most of the current research on the scientific domain conducted with massive multimodal backbones, for instance, [20] used InternVL3 with 14B parameters. These architectures are resource intensive and remain computationally prohibitive for many academic and industrial applications.
To address these constraints, our framework utilizes a compact Qwen3-4B model. However, aligning this model with the scientific domain requires Supervised Fine-tuning (SFT). A recent study [28] shows that Quantized Low Rank Adaptation (QLoRA) [29]is effective for VQA task, allowing the compact model to achieve competitive reasoning abilities by freezing the pretrained weights in a compressed 4-bit format while training other smaller adapter layers. By utilizing this technique, we have adapted the model to the scientific domain while maintaining computational efficiency.
Methodology
In this section, we present RE-SciVQA, a multi-stage framework designed for scientific figure understanding. Our approach consists of three key stages: (a) preprocessing step, (b) context-aware retrieval, and (c) parameter-efficient LoRA fine-tuning. The algorithm of RE-SciVQA is demonstrated in Algorithm 1.
Pre-processing
Image Up-scaling: Influence of image quality is observed on the performance of multimodal large language models (MLLMs) [30]. Prior work has shown that increasing image resolution can significantly improve MLLM performance [30].
Algorithm 1 RE-SciVQA: Retrieval-Aware QLoRA Training
1: Phase 2: Metadata-Aware Re-ranking 2: for candidate do 3: ▷ Calculate metadata bonuses 4: ▷ Combine scores 5: end for 6: ▷ Final Context Selection
Phase 3: Parameter-Efficient Optimization (QLoRA)
7: ▷ Freeze Base Model in 4-bit 8: Initialize where , 9: ▷ Cross-Entropy Loss for batch 10: while not converged do 11: for batch do 12: ▷ Forward Pass with Adapters 13: 14: Update ▷ Backprop only on Adapters 15: end for 16: end while 17: return
Motivated by these findings, we enhance the visual clarity of the figures by applying Lanczos interpolation for image upscaling.
2) Parsing: We employ a custom parsing pipeline built on Docling [27] to extract structured text blocks from scientific papers. The parsed content, along with relevant metadata, is stored in JSON format and later used as retrieval units in our RAG framework. Each paper is first processed by Docling’s document converter, which extracts the main document body and associated section headers. We define a set of target semantic types such as abstract, introduction, caption, and related sections to guide the parsing process. The extracted text blocks are normalized and then classified into one of these predefined semantic categories before being saved accordingly.
Context Retrieval - Phase 1
The primary goal of RAG is to retrieve relevant text blocks from the parsed paper to enhance contextual sufficiency when answering questions. This enables the MLLM to better interpret figures when the visual content and caption alone do not provide sufficient information. To achieve this, we embed both the question and the caption, then locate the most relevant text blocks through cosine similarity search, as illustrated in the Figure 2.

Figure 2. RE-SciVQAArchitecture diagram
To maximize context, the two top relevant blocks are selected. Initially, we find the vector representation of the parsed blocks with an embedding model. The figure number is appended to the initial question to ensure that the retrieval system focuses on the text blocks that refer to this specific figure. we also concatenate section names in the query; for
| Notation | Description |
| Original query, single expanded query, and the full set of expanded queries. | |
| Text embedding function and the pre-computed document block embeddings. | |
| Initial retrieved candidate pool and the final selected top- context blocks. | |
| Count of initially retrieved blocks () and final selected blocks (). | |
| The -th candidate text block in the retrieval pool. | |
| Cosine similarity score between the expanded query and candidate . | |
| Metadata-based heuristic bonus score for candidate . | |
| Final re-ranked score (combining similarity and metadata bonuses). | |
| Set of metadata attributes, their scalar weights, and the indicator function. | |
| Pre-trained base weight matrix and its 4-bit quantized (NF4) state. | |
| LoRA trainable low-rank decomposition matrices. | |
| LoRA rank (intrinsic dimension) and the adapter scaling factor. | |
| Complete set of trainable parameters within the LoRA adapter. | |
| Formatted prompt (input context) and answer (ground truth) for a training batch. |
Table I. Summary of key notations used in the RE-SciVQA framework.
instance, a section header like ‘analysis’ is concatenated with the query in case there are keywords like ‘show’ or ‘display’ in the question. This expansion would result in an enriched semantic representation, increasing the possibility of finding the correct block. The query expansion strategy is illustrated in Figure 2.
The retrieval process consists of two stages: candidate pooling and re-ranking among candidates [31]. In the first stage, a candidate pool is generated by retrieving the top- most relevant blocks for each query in the expanded query set using cosine similarity search.
where represents the query embedding and denotes the document block embeddings. It is to be noted that the notations are demonstrated in tbl:1.
In the second stage, candidates are re-ranked using a light weight heuristic scoring system that combines similarity scores with metadata-based bonuses, and the top- candidates are selected.
Here, denotes the cosine similarity score for the -th candidate from the pool, is an indicator function that equals 1 if candidate contains the -th metadata attribute (e.g., figure matching or specific section type), and 0 otherwise, and is the scalar weight assigned to the -th metadata attribute where we added bonus score for the figure ID match +0.15, caption match +0.10 and section priority +0.08 for enhancing relevance. The re-ranking weights were heuristically selected. The Figure ID match has the dominant weight due to the direct figure reference, which indicates strong relevance, followed by the caption overlap and section priority. We have tried to find the optimal weight through grid search experimentation; however, the resulted weight lead to negligible improvement in retrieval. Therefore, we retained the heuristic weights to maintain a light-weight computationally efficient re-ranking.
Finally, we select the top- candidates based on . This scoring system allows us to perform re-ranking without an additional model and helps us achieve a resource-efficient architecture.
LoRA Finetuning
Low Rank Adaptation (LoRA) [32] refers to training only a small number of new matrices by freezing the remaining weights. We used Quantized Low-Rank Adaptation (QLoRA) [29] to fine-tune the 4-billion parameter model for the task of SciVQA to avoid the high cost of standard fine-tuning. To reduce computational overhead during training, a JSONL file is compiled offline with all necessary data, like retrieved context using RAG, question, caption, ground truth and image path. This method freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture.
To fit the training process within limited VRAM constraints while maintaining gradients, the base model was loaded using the NF4 (NormalFloat 4-bit) data type, a theoretically optimal quantization for normally distributed weights and also used Double Quantization to save an additional 0.4 bits per parameter on average.
Mathematical Formulation
Following the QLoRA formulation proposed by Hu et al. [32], for a pre-trained weight matrix , the modified forward pass for a linear layer mapping input to output is defined as:
Here, remains frozen (4-bit quantized), is initialized to zeros, and is initialized with a random Gaussian distribution, ensuring at the start of training, is a scaling factor that regulates the magnitude of the adapter’s influence.
Target Module Selection
We froze the vision encoder parameters to preserve the model’s pre-trained visual grounding capabilities. The QLoRA adapters were injected only into the LLM.
Standard implementations often target only the query () and value () projections but we applied adapters to all linear layers within the Transformer blocks. The specific target modules were the attention modules, feed-forward modules, and additional three projection specific to SwigLU.
Prompt Formulation
To effectively ground the MLLM in the scientific context, we employ a structured, dynamic prompting strategy.

Figure 3. Structure of the prompt used in RE-SciVQA.
The prompt is constructed by concatenating distinct semantic blocks as System Instructions, Visual Grounding, Retrieved Context, Task and Output Constraints, depending on the data available for each sample.
The prompt construction follows the template illustrated here: System Instruction: We explicitly define the model’s role as a “scientific visual question answering assistant” to condition it for domain-specific reasoning. Crucially, we enforce a strict negative constraint (“without showing your reasoning process”) to prevent the model from generating verbose Chain-of-Thought (CoT) explanations, which simplifies parsing the final answer.
Adaptive Output Constraints: The tail of the prompt adapts to the question types. For multiple choice questions, we append the option set and enforce a single-token constraint (“Answer with ONLY the letter”) to ensure valid evaluation against ground truth. For open-ended questions, we reinforce the “direct answer only” constraint to mitigate hallucinated conversational filler.
Results
Datasets
TABLE II: Train, validation and test split for the SciVQA dataset.
The SciVQA dataset [33] consists of scientific figures from arXiv and ACL. The dataset is split into 3 parts: train, test and validation. Each figure in the dataset is associated with 7 questions. The dataset contains seven different types of questions both visual and non-visual. The description of dataset is shown in Table II.
| Dataset | Number of Samples |
| Train | 15,100 |
| Validation | 1,680 |
| Test | 4,200 |
| Total | 20,980 |
Table II. Train, validation and test split for the SciVQA dataset.
Implementation Details
Due to resource constraint, we used a small model named Qwen3-VL-4B-Instruct. We choose the Docling [27] parser to parse scientific papers, These parsed blocks were then embedded using E5-Large-V2 model to create FAISS indexing [34] for the vector database.
Moreover, we have finetuned Qwen 3 using QLoRa [29] with rank 32 along with alpha 64. The training was conducted with a learning rate of with a cosine scheduler. The loss converged within a single epoch, and all these experiments were conducted in Google cloud.
Efficiency Analysis with varying Model Size
To analyze efficiency with model size, we benchmark our method with various model sizes ranging from 140M to 14B as shown in Table III in SciVQA dataset. We use the ROUGE-1 F1 [17] and ROUGE-L F1 [17] score as our primary evaluation metrics to measure the ability for scientific reasoning.
| Model | Size | R-1 F1 | R-L F1 |
| RE-SciVQA | 4B | 0.739 | 0.738 |
| ChartGemma | 3B | 0.069 | 0.066 |
| Matcha-Chartqa | 282M | 0.063 | 0.063 |
| UniChart | 140M | 0.147 | 0.147 |
| With Chain of Thought (CoT) | |||
| InternVL3 [22] | 8B | 0.740 | 0.740 |
| Qwen2.5-VL [22] | 7B | 0.695 | 0.694 |
| Bespoke [22] | 7B | 0.709 | 0.708 |
| Phi-4 [22] | 5.4B | 0.562 | 0.561 |
| Without Chain of Thought (CoT) | |||
| InternVL3 [22] | 8B | 0.727 | 0.727 |
| Qwen2.5-VL [22] | 7B | 0.633 | 0.633 |
| Bespoke [22] | 7B | 0.652 | 0.651 |
| Phi-4 [22] | 5.4B | 0.544 | 0.543 |
| Large Model | |||
| InternVL3 LORA Fine-tuned + RAG [20] | 14B | 0.798 | 0.798 |
Table III. Performance comparison of RE-SciVQAvs other models with various sizes, comprising with and without Chain-of-Thought (CoT) on SciVQA dataset.
To find the capabilities of existing compact models, we evaluated Matcha-ChartQA (282M) and UniChart (140M). As shown in Table III, we observe that RE-SciVQA substantially dominates these small models by a large margin.
ChartGemma 3B has similarly exhibited weak scientific reasoning, highlighting a substantial performance disparity (R1-F1 0.069) relative to RE-SciVQA, even though it is a 3B-parameter model.
Remarkably, RE-SciVQA achieves near-parity with the top-performing InternVL3 8B, trailing by a negligible margin of merely 0.13%. This performance is achieved despite our model having 50% fewer parameters and without the computationally expensive Chain-of-Thought (CoT) reasoning.
In the CoT included configurations, RE-SciVQA remains competitive despite not using explicit reasoning chains. It out-performs Qwen2.5-VL (7B) by 6.3%, Bespoke (7B) by 4.2%, and Phi-4 (5.4B) by 31.5%. Excluding CoT, the advantage of RE-SciVQA becomes even more evident. It achieved accuracy gains of 1.7% over InternVL3, 16.7% over Qwen 2.5 VL, 13.3% over Bespoke, and 35.8% over Phi-4.
We also compared RE-SciVQAwith a model Bhat et al. [20] employed, a configuration closely aligned to our approach but with approximately 3.5 times more parameters. Despite this substantial increase in model capacity, the performance of RE-SciVQAis trailed by only 7.4%, highlighting its strong efficiency.
Result Analysis
Table IV compares the zero-shot performance of the Qwen-3 4B model against our proposed architecture, RE-SciVQA, which is fine-tuned using QLoRA with RAG context.
| Question Type | Qwen-3 4B (Zero-Shot + caption) | RE-SciVQA | ||||||
| Baseline | with RAG | without RAG | with RAG | |||||
| R1-F1 | RL-F1 | R1-F1 | RL-F1 | R1-F1 | RL-F1 | R1-F1 | RL-F1 | |
| Closed-ended infinite (visual) | 0.488 | 0.488 | 0.497 | 0.496 | 0.518 | 0.517 | 0.541 | 0.541 |
| Closed-ended infinite (non-visual) | 0.655 | 0.653 | 0.647 | 0.643 | 0.672 | 0.669 | 0.676 | 0.673 |
| Closed-ended finite (binary visual) | 0.735 | 0.735 | 0.736 | 0.736 | 0.738 | 0.738 | 0.771 | 0.771 |
| Closed-ended finite (binary non-visual) | 0.761 | 0.761 | 0.761 | 0.761 | 0.793 | 0.793 | 0.821 | 0.821 |
| Closed-ended finite (non-binary visual) | 0.679 | 0.679 | 0.676 | 0.676 | 0.727 | 0.727 | 0.736 | 0.736 |
| Closed-ended finite (non-binary non-visual) | 0.644 | 0.644 | 0.669 | 0.669 | 0.762 | 0.762 | 0.718 | 0.718 |
| Unanswerable | 0.661 | 0.658 | 0.093 | 0.079 | 0.915 | 0.915 | 0.908 | 0.908 |
| Overall | 0.660 | 0.660 | 0.583 | 0.580 | 0.732 | 0.732 | 0.739 | 0.738 |
Table IV. RE-SciVQAvs. Qwen-3 4B Zero-Shot performace comparison across different question types using R1-F1 and RL-F1 Metrics.
For the zero-shot setup, we evaluate the model using image captions, achieving an R1-F1 score of 0.660. To test whether contextual sufficiency would improve performance, we use the RAG context, which eventually degrades performance by
11.7%. Here, the base model treats the RAG context as a distraction, leading to increased hallucinations.
We infer RE-SciVQAwithout RAG to find out the RAG context’s effect on the fine-tuned model. RE-SciVQA(without RAG) results in a notable improvement of 10.9% over the baseline, raising the score of R1-F1 to 0.732. Moreover, we achieve peak performance in RE-SciVQA(with RAG) improving 12% over the baseline with a score of R1-F1 0.739.
The introduction of the RAG context to the model with zero-shot configuration results in an overall performance decrease of approximately 12% in the R1-F1 metric. As evidenced in Table IV, the most significant degradation occurs within the unanswerable question type. The R1-F1 score drops by 86% and RL-F1 by 88%. This substantial decrease in performance can be attributed to contextual noise [35] or distraction effect [36], where the model attempts to derive an answer from irrelevant RAG Context rather than correctly identifying the question as unanswerable. For closed-ended finite (binary non-visual), for both R1 and RL score 0.761, with and without RAG context, implies that the RAG context does not have any significance for binary answers. We observe some minor change in other question type.
RE-SciVQAachieves R1-F1 score improvement over our baseline in closed-ended infinite (visual) by 10.9%, closed-ended infinite (non-visual) by 3.2%, closed-ended finite (binary visual) by 4.8%, closed-ended finite (binary non-visual) by 7.9%, closed-ended finite (non-binary visual) by 8.4%, closed-ended finite (non-binary non-visual by 18.3% and unanswerable by 38.4%.
We further observe R1-F1 improvement of RE-SciVQAwith RAG over RE-SciVQAwithout RAG in closed-ended infinite (visual) by 4.4%, closed-ended infinite (non-visual) by 0.59%, closed-ended finite (binary visual) by 4.5%, closed-ended finite (binary non-visual) by 3.5%, closed-ended finite (non-binary visual) by 1.2%.
But in closed-ended finite (non-binary non-visual) and unanswerable type question without RAG context RE-SciVQAwithout RAG achieves 6.1% and 0.7% improvement respectively over RE-SciVQAwith RAG.
In RE-SciVQAwith RAG, a slight drop in unanswerable questions proves that even a trained model try to answer from the RAG context in the absence of an answer. In conclusion, the RAG context can hinder the performance of a base model due to increased hallucinations. Conversely, the RE-SciVQAapproach demonstrates that fine-tuning by integrating RAG context can enhance performance by leveraging the context and distinguishing noise.
Qualitative Comparison
We demonstrate the quality of response generation by RE-SciVQAin Figure 4.

Figure 4. Qualitative comparison of RE-SciVQAwith other baselines.
The first example shows the comparison between RE-SciVQAand ChartGemma, as demonstrated by correctly estimating the red line’s value at x=60 (0.62), whereas the baseline model ChartGemma returned an incorrect value of 0.65. In contrast, in the second example, while other baselines succeed in identifying the sections, e.g., identifying the red section as the highest percentage, they often struggle with output format, providing numerical values (like 4.1 or 5.0) instead of the correct categorical options (A, B, C, D).
We observe an improvement of 11.9% in exact match and 29.5% in fuzzy match for our approach compared to baseline RAG, resulting in a total of 14.3% improvement. This indicates that our mechanism helps preserve the correct answer spans.
Ablation Study
To assess how effective our retrieval mechanism is, we compare the ground truth answers with the retrieved context blocks. Table V, represents the presence of the answer in contexts, providing a comparison between baseline RAG and our proposed RAG architecture consisting of heuristic re-ranking and query expansion. We use exact match and fuzzy match count to measure context relevance. The exact match measures the appearance of ground truth answers in a complete phrase with word boundaries within the RAG context, and if answer tokens appear somewhere in the contexts we call it fuzzy match.
| Metric | Baseline RAG | Heuristic Scoring |
| Answer Presence in Contexts | ||
| Exact Match | 27.8% | 31.1% |
| Fuzzy Match | 4.4% | 5.7% |
| Total Found | 32.2% | 36.8% |
Table V. Comparison of retrieval quality between baseline RAG vs our proposed retrieval mechanism with heuristic scoring along with query expansion. Answer presence measures the frequency of the ground truth answer string appearing within the retrieved context.
Our primary focus was addressing the model’s initial struggle with Closed-ended infinite (visual) questions, where purely chart understanding models typically fail. We conduct an ablation study analyzing the impact of synthetic captions generated by ChartGemma vs. plot digitization using DePlot with RE-SciVQA. We speculate from Table III that the model’s inability to identify visual information in complex scientific charts is the cause of this failure. This is similar to research by Liu et al. [37] and Masry et al. [9], who observe that generalized chart models frequently fail to achieve the precision required for domain-specific scientific figures.
We also evaluate RE-SciVQAwith ChartGemma generated captions and plot digitization using DePlot in Table VI. RE-SciVQAwith the ChartGemma configuration yields an R1-F1 and RL-F1 score of 0.719, representing a 2.7% performance degradation relative to the RE-SciVQA. We choose this approach to provide visual grounding for infinite visual questions through synthetic captions, but the accuracy of closed-ended infinite (visual) drops by 23.1%. This demonstrates that captions through ChartGemma are not effective to provide the visual grounding required for scientific reasoning.
| Question Type | ChartGemma with | DePlot with | ||||
| RE-SciVQA(with RAG) | RE-SciVQA(with RAG) | RE-SciVQA(without RAG) | ||||
| R1-F1 | RL-F1 | R1-F1 | RL-F1 | R1-F1 | RL-F1 | |
| Closed-ended infinite (visual) | 0.416 | 0.416 | 0.493 | 0.493 | 0.486 | 0.486 |
| Closed-ended infinite (non-visual) | 0.662 | 0.659 | 0.654 | 0.650 | 0.663 | 0.661 |
| Closed-ended finite (binary visual) | 0.745 | 0.745 | 0.736 | 0.736 | 0.746 | 0.746 |
| Closed-ended finite (binary non-visual) | 0.788 | 0.788 | 0.806 | 0.806 | 0.811 | 0.811 |
| Closed-ended finite (non-binary visual) | 0.734 | 0.734 | 0.735 | 0.735 | 0.744 | 0.744 |
| Closed-ended finite (non-binary non-visual) | 0.751 | 0.751 | 0.762 | 0.762 | 0.757 | 0.757 |
| Unanswerable | 0.939 | 0.939 | 0.909 | 0.909 | 0.900 | 0.900 |
| Overall | 0.719 | 0.719 | 0.728 | 0.727 | 0.730 | 0.729 |
Table VI. Performance comparison of RE-SciVQA utilizing generated captions (ChartGemma) versus plot digitization (DePlot).
Inferring RE-SciVQAwith plot digitization using DePlot achieves an accuracy R1-F1 of 0.730 and RL-F1 of 0.729. Comparing this approach with the former ChartGemma setup, we observe an improvement of 18.5% in the Closed-ended infinite (visual) question. From this we can conclude that plot digitization provides better visual grounding through precise number extraction.
Interestingly RE-SciVQA(RAG) with the ChartGemma context configuration achieves the highest score on Unanswerable questions (0.939). So we can conclude that ChartGemma’s abstract captions are less specific than DePlot’s data tables, making the model more conservative and less likely to force an answer from the text.
Conclusion
We present a resource-efficient architecture that enhances scientific reasoning through context-aware retrieval. Moreover, we address limitations of current state-of-the-art chart-understanding models in the scientific domain. Our results show that MLLMs struggle to answer closed-ended infinite visual questions that require precise visual grounding. To mitigate this limitation, our ablation study demonstrates that synthetic captions generated by chart-understanding models further reduce accuracy on visual questions, whereas plot digitization with DePlot improves accuracy. In future work, we will explore open-ended scientific visual question answering by integrating context-aware RAG with attention mechanisms.
References
- [1]C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li, “Drivelm: Driving with graph visual question answering,” in European conference on computer vision. Springer, 2024, pp. 256–274.arxiv.org/abs/2312.14150
- [2]M. F. Ishmam, M. S. H. Showon, M. F. Mridha, and N. Dey, “From image to language: A critical analysis of visual question answering (vqa) approaches, challenges, and opportunities,” Information Fusion, vol. 106, p. 102270, 2024.
- [3]R. Tanaka, K. Nishida, K. Nishida, T. Hasegawa, I. Saito, and K. Saito, “Slidevqa: A dataset for document visual question answering on multiple images,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 11, 2023, pp. 13 636–13 645.
- [4]S. Ando, A. Agrawal, J. Lu, M. Mitchel, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2425–2433.
- [5]E. Borisova, N. Rauscher, and G. Rehm, “Scivqa 2025: Overview of the first scientific visual question answering shared task,” in Proceedings of the Fifth Workshop on Scholarly Document Processing (SDP 2025), 2025, pp. 182–210.
- [6]S. Pramannick, R. Chellappa, and S. Venugopalan, “Spiga: A dataset for multimodal question answering on scientific papers,” Advances in Neural Information Processing Systems, vol. 37, pp. 118 807–118 833, 2024.
- [7]S. Li and N. Tajbakhsh, “Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs,” arXiv preprint arXiv:2308.03349, 2023.
- [8]E. Hoque, P. Kavehzadeh, and A. Masry, “Chart question answering: State of the art and future directions,” in Computer Graphics Forum, vol. 41, no. 3. Wiley Online Library, 2022, pp. 555–572.
- [9]A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque, “Chartqa: A benchmark for question answering about charts with visual and logical reasoning,” in Findings of the association for computational linguistics: ACL 2022, 2022, pp. 2263–2279.
- [10]Y. Wu, L. Yan, L. Shen, Y. Wang, N. Tang, and Y. Luo, “Chartsights: Evaluating multimodal large language models for low-level chart question answering,” arXiv preprint arXiv:2405.07001, 2024.
- [11]N. Methani, P. Ganguly, M. Khapra, and P. Kumar, “Plotqa: Reasoning over scientific plots,” in Proceedings of the ieee/cvf winter conference on applications of computer vision, 2020, pp. 1527–1536.arxiv.org/abs/1909.00997
- [12]A. Head, K. Lo, D. Kang, R. Fok, S. Skjonsberg, D. S. Weld, and M. A. Hearst, “Augmenting scientific papers with just-in-time, position-sensitive definitions of terms and symbols,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp. 1–18.
- [13]Z. Xu, B. Qu, Y. Qi, S. Du, C. Xu, C. Yuan, and J. Guo, “Chartmoe: Mixture of diversely aligned expert connector for chart understanding,” arXiv preprint arXiv:2409.03277, 2024.
- [14]L. Zhang, A. Hu, H. Xu, M. Yan, Y. Xu, Q. Jin, J. Zhang, and F. Huang, “Tinychart: Efficient chart understanding with visual token merging and program-of-thoughts learning,” arXiv preprint arXiv:2404.16635, 2024.
- [15]X. Zhang, D. Li, L. Xiang, Y. Zhang, C. Zhong, and Z. He, “Do mllms really understand the charts?” arXiv preprint arXiv:2509.04457, 2025.arxiv.org/abs/2509.04457
- [16]A. Masry, A. Puri, M. Hashemi, J. A. Rodriguez, M. Thakkar, K. Mahajan, V. Yadav, S. T. Madhusudhan, A. Piché, D. Bahdanaut et al., “Bigcharts-r1: Enhanced chart reasoning with visual reinforcement fine-tuning,” arXiv preprint arXiv:2508.09804, 2025.
- [17]C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013/
- [18]M. Bonomo and S. Bianco, “Visual rag: Expanding mllm visual knowledge withoute fine-tuning,” arXiv preprint arXiv:2501.10834, 2025.
- [19]Y. Zhang, Q. Zhou, F. Imani, and J. Tang, “Seeing the unseen: Towards zero-shot inspection for wind turbine blades using knowledge-augmented vision language models,” arXiv preprint arXiv:2510.22868, 2025.
- [20]N. N. Bhat, J. Mondal, and S. Sarkar, “Expertneurons at scivqa-2025: Retrieval augmented vqa with vision language model (ravqa-vlm),” in Proceedings of the Fifth Workshop on Scholarly Document Processing (SDP 2025), 2025, pp. 221–229.
- [21]A. Masry, M. Thakkar, A. Bajaj, A. Kartha, E. Hoque, and S. Joty, “Chartgemma: Visual instruction-tuning for chart reasoning in the wild,” in Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, 2025, pp. 625–643.
- [22]P. Moyva and N. H. Marupaka, “Enhancing scientific visual question answering through multimodal reasoning and ensemble modeling,” in Proceedings of the Fifth Workshop on Scholarly Document Processing (SDP 2025), 2025, pp. 252–262.
- [23]B. P. K. Auer, M. Dolfin, A. S. Nassar, and P. Staar, “Doclaynet: A large human-annotated dataset for document-layout segmentation,” in Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, 2022, pp. 3743–3751.
- [24]Z. Shen, K. Lo, L. L. Wang, B. Kuehl, D. S. Weld, and D. Downey, “Vila: Improving structured content extraction from scientific pdfs using visual layout groups,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 376–392, 2022.
- [25]A. Nassar, N. Livathinos, M. Lysak, and P. Staar, “Tableformer: Table structure understanding with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4614–4623.
- [26]C. Auer, M. Lysak, A. Nassar, M. Dolfi, N. Livathinos, P. Vagenas, C. B. Ramis, M. Omenetti, F. Lindlbauer, K. Dinkla et al., “Dockling technical report,” arXiv preprint arXiv:2408.09869, 2024.
- [27]N. Livathinos, C. Auer, M. Lysak, A. Nassar, M. Dolfi, P. Vagenas, C. B. Ramis, M. Omenetti, K. Dinkla, Y. Kim et al., “Dockling: An efficient open-source toolkit for ai-driven document conversion,” arXiv preprint arXiv:2501.17887, 2025.arxiv.org/abs/2501.17887
- [28]P. Yang, H. Ying, J. Duan, L. Shi, and C. Yang, “Quantized low-rank adaptation based parameter-efficient tuning for low-resource visual question answering,” in 2024 6th International Conference on Electronic Engineering and Informatics (EEI). IEEE, 2024, pp. 1318–1322.DOI
- [29]T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in neural information processing systems, vol. 36, pp. 10 088–10 115, 2023.arxiv.org/abs/2305.14314
- [30]Y. Lu, C. Li, H. Liu, J. Yang, J. Gao, and Y. Shen, “An empirical study of scaling instruct-tuned large multimodal models,” arXiv preprint arXiv:2309.09958, 2023.arxiv.org/abs/2309.09958
- [31]M. Glass, G. Rossiel lo, M. F. M. Chowdhury, A. Naik, P. Cai, and A. Gliozzo, “Re2g: Retrieve, rerank, generate,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022, pp. 2701–2715.
- [32]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022.arxiv.org/abs/2106.09685
- [33]E. Borisova, N. Rauscher, and G. Rehm, “SciVQA 2025: Overview of the first scientific visual question answering shared task,” in Proceedings of the Fifth Workshop on Scholarly Document Processing (SDP 2025), T. Ghosal, P. Mayr, A. Singh, A. Naik, G. Rehm, D. Freitag, D. Li, S. Schimmler, and A. De Waard, Eds. Vienna, Austria: Association for Computational Linguistics, Jul. 2025, pp. 182–210. [Online]. Available: https://aclanthology.org/2025.sdp-1.18/
- [34]M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazare, M. Lomeli, L. Hosseini, and H. Jégou, “The faiss library,” IEEE Transactions on Big Data, 2025.
- [35]M. Yang, E. Huang, L. Zhang, M. Surdeanu, W. Y. Wang, and L. Pan, “How is LLM reasoning distracted by irrelevant context? an analysis using a controlled benchmark,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 13 340–13 358. [Online]. Available: https://aclanthology.org/2025.emnlp-main.674/
- [36]F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Schärli, and D. Zhou, “Large language models can be easily distracted by irrelevant context,” in International Conference on Machine Learning. PMLR, 2023, pp. 31 210–31 227.arxiv.org/abs/2302.00093
- [37]F. Liu, J. Eisenschlos, F. Piccinno, S. Krichene, C. Pang, K. Lee, M. Joshi, W. Chen, N. Collier, and Y. Altun, “Deplot: One-shot visual language reasoning by plot-to-table translation,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 10 381–10 399.