EDMA: Entropy-Driven Multimodal Answering

Introduction

Multimodal question answering (MMQA) requires the integration of heterogeneous data sources such as text, images, and tables [1, 2, 3]. For example, answering the question “What is the state in the Contour Airlines city destinations, that has a tall metal arch on a river?” requires identifying visual evidence of a metal arch in images while simultaneously retrieving structured information from tables to determine the destinations of the airline, and combining the evidence.

Recent Multimodal Large Language Models (MLLMs) have shown strong performance on such tasks by leveraging unified representations across modalities [4, 5], and are currently being employed to solve domain-specific problems [6, 7, 8, 9, 10]. Several techniques have been proposed to improve multimodal inference, most notably Chain-of-Thought (CoT) prompting [11, 12, 13], which decomposes complex tasks into intermediate steps. Building on this paradigm, extensions such as compositional prompting [14], knowledge-augmented reasoning [15], PEAR-CoT [16], frameworks such as Chain-of-Action (CoA) [17] and Visualization-of-Thought (VoT) [18] have been proposed to address limitations in compositionality, grounding, and spatial awareness. To further improve the inference process, recent work has incorporated uncertainty quantification, enabling models to adaptively seek information or refine intermediate steps [19, 20, 21]. However, while these approaches improve robustness by structuring the inference process in intermediate steps, augmenting contextual information, and guaranteeing higher robustness, they do not provide explicit stepwise quantification of how individual modalities and their fusion contribute to reducing uncertainty over the space of candidate answers. As a result, it remains unclear how different modalities contribute to the inference process, limiting interpretability and controllability in multimodal settings.

To address these limitations, we propose Entropy-Driven Multimodal Answering (EDMA), a methodology that models QA as a process of iterative entropy reduction over candidate answers. As Figure 1 shows, EDMA quantifies the impact of incorporating a specific modality and fusing modalities by measuring how this refines the space of possible answers. Compared to CoT, which splits the answering process in steps without quantifying the contribution of each modality, EDMA builds quantifiable answering trajectories, where each step and each modality is associated with a measurable contribution in the reduction of entropy. At the core of EDMA lies logical entropy [22], which measures distinctions between possible outcomes. This formulation is particularly well-suited for multimodal QA, where the objective is to progressively separate correct from incorrect candidate answers based on evidence from different modalities. By framing answering as partition refinement, EDMA treats each piece of evidence as a condition that splits the current answer space into two subsets of answers: those that satisfy the evidence and those that do not. This systematic partitioning provides a natural and interpretable mechanism for tracking exactly how information from each modality contributes to narrowing down the final prediction.

We evaluate EDMA on challenging multimodal QA benchmarks [1, 23] and show that it outperforms best-practice prompting baselines and, on average, state-of-the-art multi-agent systems, while providing traceable uncertainty estimates that guide modality selection and the answering process. Our contributions are as follows:

• EDMA, a methodology that formalizes multimodal QA as modality-conditioned entropy minimization over the space of alternatives.

• A modality-conditioned entropy reduction objective that enables explicit measurement of each modality’s contribution.

• We establish the relation between entropy minimization and lower false positives (FPs) levels, thus validating entropy minimization as a proxy for performance.

Entropy-Driven Multimodal Answering (EDMA)

Problem formulation

The problem setting involves one question $Q$, the set of modalities available to answer the question denoted by $M$ and, for each $m \in M$, the elements associated with that modality denoted by $\Omega_m$, and one Answerer denoted by $LLM$. To answer each question, the $LLM$ must find the modalities of interest, find the elements within those modalities that contain the necessary information to answer the question, and then combine the information to provide a cross-modal answer, denoted by $a^*$.

Overview of EDMA

Figure 2 shows that EDMA is composed of four stages: (a) prediction of the modalities to be used and extraction of answer requirements from the question, (b) generation of unimodal partitions and calculation of unimodal logical entropy, (c) generation of the multimodal partitions with paired multimodal logical entropy, and (d) answer generation by selecting the answering trajectory that minimized the most the logical entropy. The process allows to reduce at each step the number of elements which represent the possible alternatives by checking which ones fit the requirements extracted from the question, returning an answer based on the answering trajectory that minimised the most the logical entropy. In Algorithm 1 in Appendix B, we describe EDMA’s answering process.

Extraction of Answer Requirements

Supposing we want to answer the question “What is the state in the Contour Airlines city destinations, that has a tall metal arch on a river?”, EDMA begins by extracting the set of answer requirements, representing the characteristics that the final answer $a^*$ must satisfy. We denote the answer requirements as $R$. As shown in Figure 2, the $LLM$ generates $R$ by identifying the key elements of the question. In this specific case, the final answer has to: (i) be a State, (ii) be a destination of Contour Airlines, and (iii) be characterized by a Metal Arch. We denote the extraction of $R$ as $R = LLM(Prompt_R, Q)$, where the $LLM$ receives in input the user instructions $Prompt_R$ and the question $Q$. In this case, $R = \{S, C, A\}$, where we use $S$ to point to State, $C$ for Contour Destination, and $A$ for Metal Arch.

Prediction of Starting Modalities

EDMA predicts the subset of modalities $M_s \subseteq M$ representing the modalities with which to start the analysis. This phase does not exclude the possibility of incorporating additional modalities later in the answering process. The $LLM$ receives the input, the user instructions $Prompt_{M_s}$, the question $Q$, and for each modality $m$, the corresponding set of elements $\Omega_m$. It is then tasked with predicting the combination of modalities that is most likely to contain the elements needed to reach the correct answer. This operation can be summarized as $M_s = LLM(Prompt_{M_s}, Q, \{\Omega_m, m \in M\})$. The modalities $M_s$ are then added to $M_a$, which is the list that keeps track of the modalities used for the analysis. In Figure 2, the predicted modalities are Table ($T$) and Image ($I$).

Entropy-Driven Answer Search

To identify the correct answer in a multimodal setting, EDMA models inference as an entropy-driven search over the space of candidate answers. It quantifies how individual modalities and their interactions reduce uncertainty over the answer space. To this end, EDMA first induces partitions over the answer space using unimodal evidence and then refines these partitions through sequential cross-modal conditioning. This process enables systematic measurement of entropy reduction from both unimodal and multimodal evidence.

Quantifying Uncertainty of Answer Partitions We adopt logical entropy [22], which measures uncertainty in terms of distinctions induced by a partition of a candidate set. Let $S$ denote a finite set of candidate elements and let $P_S = \{B_1, \ldots, B_n\}$ be a partition of $S$ into disjoint non-empty blocks. To ensure compatibility across open-source and closed-source models without requiring calibrated probability estimates from the underlying model, we assume a uniform distribution over $S$, such that each block has probability mass $p_i = \frac{|B_i|}{|S|}$. Logical entropy is defined as $H(P_S) = 1 - \sum_{i=1}^{n} p_i^2$, and can be interpreted as the probability that two independently sampled elements from $S$ belong to different blocks.

For example, consider $|S| = 10$ and a binary partition $P_S = \{S_r, S_{\neg r}\}$ induced by a requirement $r$. The set $S_r$ contains elements whose information overlaps with $r$, while $S_{\neg r}$ contains elements that do not. If $|S_r| = 5$ and $|S_{\neg r}| = 5$, then $H(P_S) = 0.50$, corresponding to maximal entropy for a binary partition under a uniform prior. If instead $|S_r| = 1$ and $|S_{\neg r}| = 9$, then $H(P_S) = 1 - (0.1^2 + 0.9^2) = 0.18$, reflecting reduced uncertainty due to the increased concentration of mass.

In our multimodal QA setting, the answering process is modeled as a sequence of partition refinements over $S$, where information extracted from the elements of each modality introduces distinctions that separate relevant from irrelevant candidates. We restrict attention to valid modality-induced refinements where the predicate-satisfying subset is non-empty, $S_r \ne\varnothing$, ensuring that entropy reduction is grounded in non-vacuous evidence.

Logical entropy thus quantifies the effectiveness of each modality in refining the answer space, allowing to prioritize modalities that maximize entropy reduction, guiding the system toward increasingly discriminative evidence.

Generation of Unimodal Partitions EDMA analyses the elements $\Omega_m$ of each modality $m \in M_a$. For each element $\omega\in\Omega_m$, the $LLM$ evaluates, for each answer requirement $r \in R$, whether $\omega$ contains information overlapping with $r$. This operation can be formally expressed as $LLM(Prompt_{Check}, \omega, r)$. Based on this evaluation, EDMA constructs, for each set $\Omega_m$ and requirement $r$, a partition $P_m^r = \{\Omega_m^r, \Omega_m^{\neg r}\}$, where $\Omega_m^r$ contains elements whose information overlaps with $r$, and $\Omega_m^{\neg r}$ contains elements that do not. Under the assumption of uniform distribution over elements in $\Omega_m$, the logical entropy of $P_m^r$ is:

This formulation expresses logical entropy as a function of partition cardinalities, enabling estimation without relying on confidence estimates from the $LLM$. Logical entropy measures the distinction between elements that satisfy a given condition and those that do not. It attains its global minimum of 0 when either $\Omega_m^r = \varnothing$ or $\Omega_m^{\neg r} = \varnothing$. Otherwise, it is strictly positive and is minimized when one subset is highly concentrated (i.e., when a single candidate is clearly distinguished).

In Figure 2, unimodal partitions illustrate how different modalities contribute to distinguishing relevant from irrelevant candidates. The image-based partition $P_I^A$ yields a relatively low entropy (0.38), indicating that the visual modality already provides a strong separation by isolating the candidate satisfying the requirement $A$. In contrast, the table-based partitions exhibit higher entropy values (e.g., 0.49 for the $S$ requirement), reflecting a weaker discriminative effect when considered in isolation. Since lower entropy corresponds to sharper distinctions among candidates, this comparison indicates that the image modality is more informative at this stage for narrowing the answer space. EDMA leverages this signal to prioritize modalities that induce stronger reductions in uncertainty, guiding the answering process toward partitions that more effectively constrain the set of plausible answers.

Generation of Multimodal Partitions Once unimodal partitions $P_I^A$, $P_T^S$, $P_T^C$ are constructed, EDMA proceeds to generate multimodal partitions. This step leverages information extracted from multiple modalities to refine candidate elements, progressively reducing logical entropy, with the aim of isolating the correct candidate(s), filtering out the alternatives that are inconsistent with cross-modal evidence, which is information extracted from one modality used to reduce the answer space in another modality. In a multimodal setting, the final answer typically resides in a specific modality, referred to as the target modality, denoted by $m_t$. However, the target modality is not known in advance. To identify it, EDMA systematically treats each modality in turn as a candidate target modality $m_t$, while the remaining modalities $M_c = M_a \setminus\{m_t\}$ serve as conditioning modalities.

For each candidate $m_t$, EDMA applies information from all conditioning modalities sequentially. For each conditioning modality $m \in M_c$ and the subset $\Omega_m^{r'}$ containing elements that satisfy a requirement $r' \in R$, EDMA extracts the cross-modal evidence set:

Each piece of evidence represents information from the conditioning modality $m$ and has the same function as the answer requirements. EDMA applies the elements of this set sequentially to iteratively refine the candidates in the target modality $m_t$ that satisfy requirements $r \in R$, where $r \ne r'$. This allows evidence from one modality (e.g., visual) that satisfies one requirement to be used to filter candidates in another modality (e.g., tabular) that satisfy a distinct requirement.

Specifically, for a given requirement $r \in R$, the target subset $\Omega_{m_t}^r$ is updated sequentially for each conditioning modality. At each step, EDMA generates a partition $P_{m_t|m}^{r,E_m} = \{\Omega_{m_t}^{r,E_m}, \Omega_{m_t}^{r,\neg E_m}\}$ of the elements of modality $m_t$ consistent with requirement $r$, where $\Omega_{m_t}^{r,E_m}$ contains the elements that are also consistent with the evidence $E_m$ from the current conditioning modality, and $\Omega_{m_t}^{r,\neg E_m}$ contains those that are not. Only the subset $\Omega_{m_t}^{r,E_m}$ is carried forward to the next iteration, while $\Omega_{m_t}^{r,\neg E_m}$ is excluded from further conditioning, as it cannot lead to the correct answer. This sequential conditioning progressively narrows the answer space of $m_t$, increasing the distinction among the remaining candidates, thus decreasing logical entropy.

By evaluating the multimodal logical entropy after each step, EDMA identifies which $m_t$ ultimately yields the partition with the lowest entropy, where distinctions among candidates are maximized, and uncertainty is minimized. Conditional or multimodal logical entropy is computed analogously to the unimodal case:

In the example in Figure 2, cross-modal conditioning yields asymmetric effects depending on the direction of information flow. When table-derived constraints ($E_T$) are applied to the image modality ($P_I^A$), entropy remains unchanged (0.38), indicating limited additional discriminative power beyond visual cues. In contrast, conditioning the table modality ($P_T^C$) using visual evidence ($E_I$) significantly reduces entropy (0.44 $\to$ 0.20), showing that the image provides high-value filtering information that effectively disambiguates table entries, and showing that EDMA does not treat modalities symmetrically, but instead identifies directional entropy reduction, where certain modalities act as stronger selectors of relevant candidates if the derived information is applied in the right order.

Answer Generation

Trajectory Selection via Entropy Minimization The systematic generation of unimodal and multimodal partitions effectively maps a set of quantifiable answering trajectories ($\mathcal{T}$), representing paths in the search process. Each trajectory provides a quantifiable history of how information is fused and how logical entropy changed after the modality fusion. This allows for a modality-level attribution of the logical entropy variation. A trajectory may remain within a unimodal setting ($m \rightarrow m$) or bridge modalities to form a multimodal trajectory ($m \rightarrow m_t$), where the first modality conditions the second ($m_t|m$). We denote $\Delta H = \frac{H(P^{r}_{m_t})-H(P^{r,E_m}_{m_t|m})}{H(P^{r}_{m_t})}$, the relative change between unimodal entropy and multimodal entropy for modality $m_t$. If $\Delta H > 0$, modality fusion increases the distinguishability among candidates, thereby reducing uncertainty. If $\Delta H \leq0$, the conditional modality provides redundant or non-discriminative information for that specific answering path.

In Figure 2, EDMA identifies four answering trajectories, of which two are unimodal and two are multimodal. The unimodal trajectories are represented by $P_I^A$ (initial $H = 0.38$) and $P_T^C$ (initial $H = 0.44$). The Table-Image Trajectory starts with partition $P_I^A$ and ends with partition $P_{IT}^{A,E_T}$, where table data regarding destinations conditions the image set, resulting in a final entropy of $H = 0.38$ with $\Delta H = 0\%$. In contrast, the Image-Table Trajectory, where visual evidence of the Metal Arch conditions the table rows, starts with partition $P_T^C$ and ends with partition $P_{TI}^{C,E_I}$, reducing logical entropy from $H = 0.44$ to $H = 0.20$, achieving a significant entropy decrease of $\Delta H = 54\%$, a reduction attributed to the image modality’s influence on the table modality.

As EDMA prioritizes the trajectory whose final partition yields the lowest logical entropy ($t^*$), it selects the Image–Table Trajectory, isolating the table entry corresponding to the city satisfying requirement $C$, which is also associated with an image from which EDMA extracts visual evidence of the arch. The multimodal partition at the endpoint of $t^*$ is chosen as the basis for the final answer.

Final Inference Once the optimal trajectory, modality, and partition are selected, the LLM receives the question $Q$, the satisfied requirements, the cross-modal evidence $E^*$ (e.g., the context of the arch), and the relevant elements $\Omega^*$ that satisfy both the requirements and the cross-modal evidence (e.g., the specific row for St. Louis). Based on this information, the system returns the final answer $a^*$ (e.g., Missouri). If the LLM cannot produce an answer (i.e., $a^* = \text{None}$), EDMA selects a new set of modalities $M_s \subseteq M \setminus M_a$, where the newly selected modalities are not contained in the already analysed modality set. EDMA then repeats the process: generating new partitions, recomputing entropy deltas, and attempting to produce the answer with the newly acquired information. The iteration continues until a valid answer is obtained or all modalities are exhausted.

Crucially, by constructing and selecting among quantifiable answering trajectories, EDMA provides a principled mechanism to guide cross-modal inference. Even when SOTA approaches arrive at the same final answer, EDMA makes explicit how that answer is obtained, enabling quantifiable modality-level attribution.

Experiments

Experimental Setup

We evaluate our methodology across a diverse set of SOTA LLMs, including both closed source models, such as GPT-5.2 [24], Kimi-K2.5 [25], Nova-2-Lite-v1 (Nova-2-Lite) [26], and Claude-Sonnet-4.6 (Claude-4.6) [27], and open source models, such as Qwen3-VL-235B-A22B (Qwen3) [28], and Nemotron-Nano-12B-v2 (Nemotron) [29]. Appendix E reports how to access models and the resources needed to run them, while Appendix K reports the prompts to implement EDMA’s operations.

Baselines We compare our approach against SOTA multimodal QA system MAMMQA [30] and against several common best-practice prompting strategies: Direct Prompting (DP), where the model generates an answer without explicit intermediate steps, Chain-of-Thought (CoT) [11], which encourages step-by-step generation of intermediate steps within a single generation, and Planning Prompting (PP) [31], where the model first generates a plan and subsequently executes it, CoT-SC [32] and PP-SC which improve the greedy versions of CoT and PP by generating multiple reasoning chains and returning the majority-vote final answer.

Datasets We evaluate our methodology on MultiModalQA [1] and ManyModalQA [23] datasets which remain, to the best of our knowledge, among the most complex benchmarks for multimodal QA, requiring cross-modal integration across text, tables, and images. They both require deep joint inference over diverse data formats to arrive at a final answer, and thus provide a rigorous testbed for evaluating a model’s ability to integrate and reason over heterogeneous information sources.

Evaluation Metrics We evaluate all methods using Exact Match (EM) and F1 score, following standard practice in multimodal QA [1, 33, 34]. EM is a strict binary metric that assigns a score of 1 if the predicted answer exactly matches the ground-truth string and 0 otherwise. The F1 score measures token-level overlap between the predicted and reference answers, balancing precision and recall. Because EM and F1 can be overly penalizing for semantically correct open-ended generations, in our evaluation we also include an LLM-Judge (LLM-J) metric [35].

EDMA Performance Evaluation

Here we report the performance of EDMA across benchmarks. We first compare EDMA to standard prompting baselines (DP, CoT, and PP) and then we evaluate EDMA against budget-matched baselines.

Comparison between EDMA and Prompting Baselines

Table 1 summarises the performance of EDMA alongside standard prompting baselines across six representative MLLMs. EDMA consistently outperforms Direct Prompting (DP), Chain-of-Thought (CoT), and Planning Prompting (PP) across all evaluated backbones and datasets. On MultiModalQA, EDMA achieves an average score of 80.1 EM, 81.1 F1, and 79.3 LLM-J, outperforming the strongest prompting baseline (PP) by +22.7 EM, +14.0 F1, and +8.5 LLM-J points. On ManyModalQA, EDMA reaches 83.0 EM, 83.3 F1, and 89.1 LLM-J, improving over PP by +23.0 EM, +11.5 F1, and +0.6 LLM-J. Across both benchmarks, EDMA achieves average gains of +22.9 EM, +12.8 F1, and +4.6 LLM-J over the strongest prompting baseline. The smaller LLM-Judge gain on ManyModalQA (+0.6 vs +8.5 on MultiModalQA) stems from lower benchmark complexity, limiting EDMA’s possible improvement space. Nevertheless, EDMA’s strong leads in EM (+23.0) and F1 (+11.5) on ManyModalQA confirm that entropy-guided decomposition remains essential for precise entity extraction even on simpler queries.

Method	Claude-4.6 EM	Claude-4.6 F1	Claude-4.6 LLM-J	Kimi-K2.5 EM	Kimi-K2.5 F1	Kimi-K2.5 LLM-J	Qwen3 EM	Qwen3 F1	Qwen3 LLM-J	GPT-5.2 EM	GPT-5.2 F1	GPT-5.2 LLM-J	Nemotron EM	Nemotron F1	Nemotron LLM-J	Nova-2-Lite EM	Nova-2-Lite F1	Nova-2-Lite LLM-J	Avg EM	Avg F1	Avg LLM-J
EDMA	82.8	83.5	88.9	80.4	81.0	83.2	79.4	80.9	82.5	80.0	80.3	80.5	79.0	80.5	75.1	79.0	80.4	65.9	80.1	81.1	79.3
PP	61.4	73.1	85.6	64.0	74.0	82.0	60.6	69.4	75.4	56.6	68.4	75.5	51.8	60.0	64.6	50.0	57.8	41.8	57.4	67.1	70.8
CoT	61.4	72.3	87.4	63.8	72.7	79.9	61.0	70.7	76.4	46.2	56.8	54.4	46.4	54.2	57.1	43.4	50.2	53.4	53.7	62.8	69.8
DP	60.8	70.5	82.7	39.4	47.2	52.1	60.6	68.9	74.3	58.2	68.8	75.7	42.2	50.0	52.3	51.8	59.6	64.6	52.2	60.8	66.9
ManyModalQA [23]
EDMA	82.0	82.4	90.0	86.0	86.6	94.5	84.0	84.0	90.0	88.0	88.5	88.0	78.0	78.0	84.5	80.0	80.0	87.5	83.0	83.3	89.1
PP	56.0	70.2	90.0	62.0	74.4	94.5	64.0	75.8	89.5	48.0	63.4	86.5	64.0	71.3	84.0	66.0	75.7	87.5	60.0	71.8	88.5
DP	70.0	81.5	93.0	70.0	80.1	93.5	62.0	71.2	86.0	56.0	67.8	87.5	48.0	56.7	74.5	58.0	66.7	80.5	60.7	70.7	85.8
CoT	56.0	68.2	88.0	62.0	70.4	83.0	64.0	73.3	86.5	42.0	56.2	56.2	40.0	50.2	69.0	60.0	67.9	81.0	54.0	64.4	82.1

In Appendix F and Appendix G, we respectively report detailed analyses regarding EDMA’s intermediate operation accuracy and statistical variance for EDMA and baselines results.

Comparison between EDMA and Budget Matched Baselines

To determine whether EDMA’s gains result merely from increased inference computation, we evaluate EDMA on MultiModalQA against budget-matched self-consistency baselines (CoT-SC and PP-SC with $N = 6$ samples) and the multi-agent framework MAMMQA. As shown in Table 5 (Appendix § 12), EDMA is approximately $6\times$ computationally more expensive than single-pass CoT and PP, making $N = 6$ a direct budget-matched baseline. Furthermore, EDMA’s computational footprint is comparable to MAMMQA, establishing it as an additional budget-matched baseline. We evaluate MAMMQA using exclusively LLM-J, as its verbose answer style would unfairly penalize it under exact-matching metrics like EM and F1.

Model	Approach	Avg. Total Tokens	Avg. # Calls	Avg. Cost ($) | Latency (s)
Nova-2-Lite	CoT	19,775.20	1.00	0.006	$\sim5.04$
Nova-2-Lite	DP	19,576.12	1.00	0.006	$\sim3$
Nova-2-Lite	EDMA	89,909.88	143.35	0.027	$\sim30$
Nova-2-Lite	PP	20,070.06	1.00	0.007	$\sim5$
Nova-2-Lite	MAMMQA	79,972.14	7.00	0.024	$\sim10$
GPT-5.2	CoT	4,616.74	1.00	0.009	$\sim3$
GPT-5.2	DP	4,511.48	1.00	0.008	$\sim1.50$
GPT-5.2	EDMA	68,574.99	181.75	0.120	$\sim120$
GPT-5.2	PP	4,935.78	1.00	0.013	$\sim2.75$
Kimi-K2.5	CoT	19,812.28	1.00	0.012	$\sim2$
Kimi-K2.5	DP	19,587.54	1.00	0.012	$\sim3.5$
Kimi-K2.5	EDMA	91,275.13	165.43	0.055	$\sim90$
Kimi-K2.5	PP	20,021.98	1.00	0.013	$\sim4.07$
Kimi-K2.5	MAMMQA	80,172.47	7.00	0.048	$\sim11$
Nemotron	CoT	19,751.52	1.00	0.003	$\sim3$
Nemotron	DP	19,602.88	1.00	0.003	$\sim1$
Nemotron	EDMA	90,438.65	151.03	0.014	$\sim90$
Nemotron	PP	19,828.78	1.00	0.003	$\sim3$
Nemotron	MAMMQA	79,951.83	7.00	0.012	$\sim5$
Qwen3	CoT	19,836.08	1.00	0.005	$\sim20$
Qwen3	DP	19,574.92	1.00	0.004	$\sim5$
Qwen3	EDMA	90,314.85	149.31	0.020	$\sim120$
Qwen3	PP	20,047.36	1.00	0.005	$\sim18.37$
Claude-4.6	CoT	19,871.12	1.00	0.063	$\sim18$
Claude-4.6	DP	19,586.50	1.00	0.059	$\sim4$
Claude-4.6	EDMA	91,322.96	165.35	0.274	$\sim180$
Claude-4.6	PP	19,937.36	1.00	0.063	$\sim16.01$
Claude-4.6	MAMMQA	81,253.41	7.00	0.244	$\sim12$

As detailed in Table 2, the effect of self-consistency depends on model capacity. Under PP-SC, Nova-2-Lite gains +5.5 EM, +9.5 F1, and +17.8 LLM-J over single-pass PP, whereas Nemotron improves slightly on EM (+0.7) and F1 (+1.6) but drops −4.4 LLM-J. In contrast, the larger backbones degrade: Claude-4.6 drops −4.9 EM, −3.2 F1, and −3.1 LLM-J, and Kimi-K2.5 drops −3.5 EM, −0.3 F1, and −1.2 LLM-J. Under CoT-SC, Nova-2-Lite again benefits across all metrics (+3.1 EM, +5.4 F1, +9.8 LLM-J over single-pass CoT), while Nemotron gains +3.9 LLM-J but loses −3.9 EM and −1.5 F1. The larger backbones again degrade on all metrics: Claude-4.6 drops −2.9 EM, −0.3 F1, and −3.1 LLM-J, and Kimi-K2.5 drops −5.3 EM, −3.1 F1, and −3.5 LLM-J.

Method	Claude-4.6			Kimi-K2.5			Nemotron			Nova-2-Lite			Avg
	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J
EDMA	82.8	83.5	88.9	80.4	81.0	83.2	79.0	80.5	75.1	79.0	80.4	65.9	80.3	81.4	78.2
MAMMQA	–	–	83.3	–	–	88.4	–	–	65.7	–	–	65.7	–	–	75.7
CoT-SC	58.5	72.0	84.3	58.5	69.6	76.4	42.5	52.7	61.0	46.5	55.6	63.2	51.5	62.5	71.2
PP-SC	56.5	69.9	82.5	60.5	73.7	80.8	52.5	61.6	60.2	55.5	67.3	59.6	56.3	68.1	70.8

Across the four backbones, SC yields only modest average LLM-J gains over single-pass prompting (+2.3 for PP-SC and +1.8 for CoT-SC) and does not improve EM. In contrast, EDMA (78.2 LLM-J, 80.3 EM, 81.4 F1) outperforms CoT-SC (71.2 LLM-J) and PP-SC (70.8 LLM-J) by +7.0 and +7.4 LLM-J, respectively, and MAMMQA (75.7 LLM-J) by +2.5 LLM-J on average, although MAMMQA is stronger on Kimi-K2.5 (88.4 vs. 83.2 LLM-J). The importance of EDMA’s core mechanisms is further supported by two ablation studies, which isolate the contributions of entropy minimization and cross-modal conditioning (Table 10–Table 11, Appendix § 15).

Model	EDMA	EDMA (Random Trajectory)	Difference
GPT-5.2	80.5	45.8	-34.7
Nova-2-Lite	65.9	47.3	-18.6
Kimi-K2.5	83.2	37.1	-46.1
Nemotron	75.1	51.6	-23.5
Qwen3	82.5	48.2	-34.3
Claude-4.6	88.9	53.0	-35.9
Average	79.3	47.2	-32.2

Model	EDMA	EDMA (Unimodal-only)	Difference
GPT-5.2	80.5	65.5	-15.0
Nova-2-Lite	65.9	48.0	-17.9
Kimi-K2.5	83.2	73.1	-10.1
Nemotron	75.1	56.8	-18.3
Qwen3	82.5	68.4	-14.1
Claude-4.6	88.9	77.8	-11.1
Average	79.3	64.9	-14.4

Qualitative Example

We attribute the performance gains achieved by EDMA to its staged reasoning process. Specifically, EDMA first decomposes the answer space into unimodal partitions before extracting explicit crossmodal evidence to construct combined multimodal partitions. This progressive refinement allows the model to isolate and verify relevant information within individual modalities prior to cross-modal integration and trajectory selection, which may improve the reliability of evidence aggregation and reduce ambiguity in cross-modal alignment. In contrast, standard prompting paradigms such as CoT and PP depend on the model capacity to implicitly manage decomposition and multimodal integration within a single inference stream, which may limit their ability to consistently isolate and align evidence across modalities.

Figure 3 illustrates this contrast on two multi-hop examples. In the first, CoT misidentifies the winner of the 33{sup}rd` TCA Award for Outstanding New Program. EDMA instead isolates the tabular context to identify “This Is Us” as the winner, and then queries the secondary modality to retrieve the correct broadcasting network (NBC). In the second example, CoT retrieves the tabular context (the series “Talking Heads”) but its prediction is not grounded in the visual modality. EDMA conditions its visual query on the tabular entry and correctly aligns the image, identifying the depicted activity as writing.

False Positives Correlate with Higher Entropy

Here we analyse the relationship between entropy and prediction errors to assess the effectiveness of EDMA, examining how entropy correlates with the occurrence of FPs across target modalities. Figure 4a shows that across all modalities, FPs are consistently associated with higher entropy values compared to true positives (TPs). This indicates that incorrect predictions tend to occur in regions of higher uncertainty, where the candidate answer space remains insufficiently constrained. In contrast, correct predictions are more likely to arise when entropy is lower, reflecting a more decisive partitioning of the answer space. See Table 14 in Appendix § 16 for data regarding statistical significance.

Table 14: Statistical comparison of entropy in True Positives (TP) vs. False Positives (FP). $\overline{H}$ denotes mean entropy; $t$ denotes the t-test statistic; PB denotes the point-biserial correlation between entropy and a binary outcome variable (TP = 0, FP = 1). Tick (✓) indicates significance at $p < 0.05$.

Modality	$\overline{H}(TP)$	$\overline{H}(FP)$	Stat (t)	Sig. (p)	Corr. (r)	Sig. (p)
Text	0.20	0.24	-8.72	✓	0.26	✓
Image	0.21	0.26	-5.78	✓	0.26	✓
Table	0.21	0.27	-8.66	✓	0.23	✓

Answering Trajectories with Lower Entropy Yield Better Answers

To further show the connection between entropy level and FPs, we perform a controlled intervention analysis to assess whether entropy directly impacts prediction performance. We generate three candidate answers per question by selecting answering trajectories corresponding to the lowest, median, and highest entropy levels. The low-entropy trajectory corresponds to the one selected by EDMA in the main experiments. This design keeps the input question fixed while varying the selected trajectory, and hence its entropy. We evaluate prediction reliability using the False Discovery Rate (FDR), defined as the proportion of FPs among all positive predictions. We compute FDR separately for each entropy group across modalities. Figure 4b shows a consistent monotonic trend: low-entropy trajectories yield the lowest FDR, while medium- and high-entropy trajectories exhibit progressively higher error rates. This pattern is consistent across all target modalities, indicating a strong association between higher logical entropy and reduced prediction reliability. By systematically varying the selected trajectory while keeping the input query fixed, we observe a robust degradation in performance as entropy increases. These results support the importance of entropy minimization in EDMA: lower-entropy trajectories are associated with fewer false positives and improved prediction performance. Tables 15–16 in Appendix § 16, report data regarding statistical significance.

Related work

MultimodalQA Multimodal question answering (MMQA) has been introduced as a challenging task requiring the integration of heterogeneous data sources such as text, tables, and images [1]. Subsequent work has focused on improving representation and integration across modalities, including unified architectures such as UniMMQA [2], Solar [33], retrieval-augmented approaches such as UniRAG [34], and conversational extensions like MMConvQA [36]. More recent approaches leverage LLMs and MLLMs to enhance multi-step inference capabilities, for instance by combining language and vision models [37], improving cross-modal alignment through similarity computation [38], or framing information synthesis as a multi-agent process, as in MAMMQA [30].

LLMs in MultimodalQA LLMs have been widely adopted to enhance multimodal QA, often through extensions of Chain-of-Thought (CoT) prompting. Prior work includes Chain-of-Action (CoA) [17], knowledge-augmented CoT (KAM-CoT) [15], duty-distinct CoT (DDCoT) [13], and frameworks for structured CoT such as UniCoTT [39]. These approaches improve inference by decomposing complex tasks into intermediate steps or incorporating external knowledge. Additional adaptations target specific modalities, such as AVQA-CoT for audio-visual QA [40] and methods that leverage textual signals to guide multimodal understanding [41]. Despite these advances, existing CoT-based approaches do not provide explicit mechanisms to quantify the extent to which each modality contributes to the final prediction.

Uncertainty as Guidance Recent work has explored incorporating uncertainty quantification into LLM reasoning to improve robustness and decision-making. Hu et al. [19] propose Uncertainty of Thoughts, which evaluates alternative planning paths based on their expected uncertainty reduction. Yin et al. [20] introduce uncertainty-aware adaptive guidance, an approach that relies on uncertainty signals from the LLM to guide the reasoning process. Liang et al. [21] propose introspective planning to align model uncertainty with task ambiguity, while Bayat et al. [42] leverage uncertainty estimates to guide answer selection. While these approaches treat uncertainty as a global signal for improving reasoning, they do not provide fine-grained, modality-level attribution of uncertainty reduction, limiting their effectiveness in multimodal QA settings.

Limitations

EDMA’s worst-case complexity is $O(n^{3})$ when modalities, requirements, and elements grow together. In practice this translates into many LLM calls per question, costing between 5 and 6.4× more and taking between 20 and 46× longer than the prompting baselines (Appendix B). Our budget-matched baselines match its monetary cost, but not its latency or number of calls, and a full accuracy–cost analysis is left to future work.

Logical entropy is computed under a uniform distribution over elements, which avoids calibrated probabilities but ignores differences in the relevance of individual elements.

Finally, our methodology does not explicitly address scenarios in which different modalities provide conflicting information. One possible extension would be to partition the answer space based on parallel interpretations of the available evidence, treating them as distinct answering trajectories, which would let the model explore potentially conflicting paths and evaluate their outcomes independently.

Conclusion

This paper introduces Entropy-Driven Multimodal Answering (EDMA), a novel methodology for multimodal QA based on logical entropy minimization and the construction of quantifiable answering trajectories.

By measuring how each step reduces logical entropy, EDMA quantifies the contribution of every modality, improving traceability. Our analyses show that false positives are associated with higher entropy, and that choosing the lowest-entropy trajectory tends to lead to better answers, making entropy a useful proxy for performance. Overall, on the same backbones, EDMA improves over the baselines it is compared against.

Acknowledgments and Disclosures

This work was partially supported by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO) under the KIC HEWSTI Project under grant no. KIC1.VE01.20.004.

The authors declare that they have no competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Notation

Symbol	Description
$Q$	Input question
$M$	Set of all modalities
$m \in M$	A specific modality
$M_s \subseteq M$	Set of selected starting modalities
$M_a$	Modalities under analysis
$m_t$	Candidate target modality
$M_c$	Set of conditioning modalities ($M_a \setminus\{m_t\}$)
$\Omega_m$	Set of elements for modality $m$
$\omega\in\Omega_m$	A specific element of modality $m$
$R$	Set of answer requirements
$r \in R$	A specific answer requirement
$\Omega_m^r$	Elements of modality $m$ satisfying requirement $r$
$\Omega_m^{\neg r}$	Elements of modality $m$ not satisfying $r$
$P_m^r$	Partition of modality $m$ w.r.t. requirement $r$
$H(P_m^r)$	Logical entropy of partition $P_m^r$
$\mathcal{H}_{\mathrm{uni}}[m,r]$	Cache storing $(H(P_m^r), \Omega_m^r)$ for each $(m,r)$ pair
$E_m$	Set of cross-modal evidence extracted from a modality
$E[m,r]$	Cached cross-modal evidence extracted from modality $m$ w.r.t. requirement $r$
$\Omega_{m_t}^{r,E_m}$	Target elements consistent with requirement $r$ and cross-modal evidence $E_m$
$\Omega_{m_t}^{r,\neg E_m}$	Target elements consistent with $r$ but inconsistent with $E_m$
$P_{m_t\mid m}^{r,E_m}$	Conditional partition of target modality given $E_m$
$H(P_{m_t\mid m}^{r,E_m})$	Multimodal logical entropy after applying $E_m$
$\Omega_{\mathrm{curr}}$	Elements of $m_t$ still consistent with $r$, refined across conditioning steps
$E_{\mathrm{curr}}$	Cross-modal evidence accumulated over the conditioning steps
$\mathcal{H}_{\mathrm{multi}}[m_t,r]$	Cache storing $(H(P_{m_t\mid m}^{r,E_m}), \Omega_{\mathrm{curr}}, E_{\mathrm{curr}})$ for each $(m_t,r)$ pair
$\Delta H$	Relative change between $H(P_m^r)$ and $H(P_{m_t\mid m}^{r,E_m})$
$\mathcal{T}$	Set of candidate unimodal and multimodal trajectories
$t^*$	Optimal trajectory selected via entropy minimization
$\Omega^*$	Subset of elements in $m^*$ consistent with $r^*$ and $E^*$, as selected by $t^*$
$E^*$	Cross-modal evidence used for answer generation
$a^*$	The final generated answer

Algorithm

The EDMA algorithm has a worst-case computational complexity of $O(M \cdot R \cdot K + M^2 \cdot R)$, where $M$ denotes the number of modalities, $R$ the number of requirements, and $K$ the maximum number of elements per modality. Stage 1 (Unimodal Partitioning) contributes $O(M \cdot R \cdot K)$, evaluating each of the $K$ elements per modality against each requirement, and also caches the cross-modal evidence extracted for each $(m,r)$ pair. Stage 2 (Cross-Modal Sequential Conditioning) contributes $O(M^2 \cdot R)$, refining each of the $M \cdot R$ target pairs by conditioning on the remaining $M$ modalities via $O(1)$ lookups into this cache. Under $M \sim R \sim K \sim n$, the overall complexity is $O(n^3)$.

Algorithm 1 Entropy-Driven Multimodal Answering (EDMA)
Require: Q, M, {Ω_m, m ∈ M}
Ensure: Final answer a*
1: R ← LLM(Prompt_R, Q)                         // Extraction of answer requirements
2: M_s ← LLM(Prompt_M_s, Q, {Ω_m, m ∈ M})       // Prediction of Starting Modalities
3: a* ← None
4: M_a ← ∅
5: while a* = None and M_s ≠ ∅ do
6:     M_a ← M_a ∪ M_s
7:     M ← M \ M_s

        // Stage 1: Unimodal Partitioning (with Extraction Caching)
8:     for (m, r) ∈ M_a × R do
9:         Generate P_m^r = {Ω_m^r, Ω̄_m^r} and compute H(P_m^r)
10:        Store H(P_m^r) and Ω_m^r in H_uni[m, r]
11:        E[m, r] ← {LLM(Prompt_Extract, Q, ω) | ω ∈ Ω_m^r}   // Cached once per (m, r)
12:    end for

        // Stage 2: Cross-Modal Sequential Conditioning
13:    for (m_t, r) ∈ M_a × R do
14:        M_c ← M_a \ {m_t}, Ω_curr ← Ω_m_t^r, E_curr ← ∅
15:        for m ∈ M_c do
16:            E_m ← E[m, r]                                      // O(1) lookup
17:            Generate P_{m_t|m}^{r,E_m} = {Ω_{m_t}^{r,E_m}, Ω_{m_t}^{r,¬E_m}} using Ω_curr and E_m
18:            Ω_curr ← Ω_{m_t}^{r,E_m}
19:            E_curr ← E_curr ∪ E_m
20:        end for
21:        H(P_{m_t|m}^{r,E_m}) ← H(P_{m_t|m}^{r,E_m})
22:        Store H(P_{m_t|m}^{r,E_m}), Ω_curr, E_curr in H_multi[m_t, r]
23:    end for

        // Stage 3: Trajectory Selection
24:    T ← ∅
25:    for (m_t, r) ∈ M_a × R do
26:        H_u, Ω_u ← H_uni[m_t, r]
27:        H_m, Ω_m, E_m ← H_multi[m_t, r]
28:        T ← T ∪ {(m_t, r, H_u, Ω_u, ∅)}                         // Unimodal path
29:        if H_u > 0 then ΔH ← (H_u − H_m)/H_u else ΔH ← 0
30:        T ← T ∪ {(m_t, r, H_m, Ω_m, E_m, ΔH)}                   // Multimodal path
31:    end for
32:    Select t* = (m*, r*, H*, Ω*, E*) from T that minimizes H

        // Stage 4: Final Inference
33:    a* ← LLM(Q, Ω*, E*)                                      // Inference from optimal subset and evidence
34:    if a* = None and M ≠ ∅ then
35:        M_s ← LLM(Prompt_M_s, Q, {Ω_m, m ∈ M})
36:    else
37:        M_s ← ∅
38:    end if
39: end while
40: return a*

Experimental Setup

Baselines

In this section, we provide detailed descriptions of the baselines used in our experiments. We categorize these into multimodal architectures and prompting-based strategies.

Multimodal QA Systems

MAMMQA [30] A multi-agent QA framework for multimodal inputs spanning text, tables, and images. The system comprises two Vision-Language Model (VLM) agents and one text-based LLM agent. The first VLM decomposes the query into sub-questions to sequentially retrieve partial evidence across modalities, the second VLM synthesizes intermediate insights via cross-modal reasoning, and the LLM aggregates the findings into a cohesive final answer.

MultiModalQA [1] The baseline pipeline introduced alongside the MultiModalQA benchmark. It employs modality-specific information extractors combined with an ImplicitDecomp module to perform multi-step question decomposition and answer synthesis across text, tables, and images.

UniMMQA [2] A unified multimodal QA framework designed to reduce the modality gap by projecting heterogeneous data (text, tables, and images) into a shared representation space, enabling integrated cross-modal retrieval and answer generation.

Solar [33] A sequence-to-sequence multimodal framework that unifies visual and textual token processing within a single transformer architecture, optimized for cross-modal alignment through joint pre-training objectives.

UniRAG [34] A retrieval-augmented generation (RAG) framework that unifies evidence retrieval across textual, visual, and tabular corpora before feeding the aggregated context into a Large Multimodal Model (LMM) for answer generation.

ManyModalQA [23] The baseline framework introduced with the ManyModalQA benchmark. It incorporates an explicit modality routing mechanism to determine the most relevant data source (text, image, or table) prior to applying specialized extraction modules to return the final answer.

Prompting Strategies

Direct Prompting (DP) A standard zero-shot prompting approach where the model receives the input context and query, and directly outputs the final answer without explicit intermediate reasoning steps.

Chain-of-Thought (CoT) [11] A prompting method that prompts the model to generate a sequence of intermediate natural language reasoning steps before outputting the final answer within a single generation stream.

Planning Prompting (PP) [31] A two-stage prompting technique that decouples reasoning planning from execution: the model first generates a high-level plan of sub-tasks, and subsequently executes each sub-task to derive the final answer.

CoT Self-Consistency (CoT-SC) [32] An ensemble strategy that samples $N$ independent Chain-of-Thought reasoning paths and selects the final answer via majority voting.

PP Self-Consistency (PP-SC) An extension of self-consistency [32] applied to Planning Prompting, sampling $N$ independent plan-and-execute reasoning trajectories and selecting the answer via majority voting.

Datasets

MultiModalQA [1] is a comprehensive benchmark designed to evaluate a model’s ability to perform joint integration across three distinct data modalities: structured tables, natural language text, and images. Unlike earlier multimodal datasets where the answer usually resides within a single source (e.g., just an image or just a paragraph), MultiModalQA is specifically engineered to require multi-hop reasoning. We report results on the validation set since ground-truth annotations for the test set are not publicly accessible.

To further show the generalizability of our methodology, we test it also on ManyModalQA [23], which is a multimodal question-answering benchmark designed to evaluate an agent’s ability to reason across three distinct data types: textual passages, structured tables, and visual images. Unlike standard multimodal datasets where the target modality is often implied by the question, ManyModalQA contains ambiguous questions where the source of the answer cannot be determined by the question text alone.

Comparison with Previous Multimodal QA Systems

Table 4 compares EDMA with prior multimodal QA systems. EDMA achieves the best performance across both datasets. On MultiModalQA, it obtains 82.8 EM and 83.5 F1, improving over the strongest prior system (UniMMQA) by +9.1 EM and +3.4 F1. On ManyModalQA, the gains are substantially larger, with improvements of +35.9 EM and +36.1 F1 over UniMMQA. We note that these comparisons reflect both methodological improvements and differences in underlying foundation models.

Approach	Model	EM	F1
	MultiModalQA [1]
EDMA	Claude-4.6	82.8	83.5
UniMMQA [2]	T5-3B [43]	73.7	80.1
UniRAG [34]	LLaVA-v1.5-7B [44]	67.4	71.3
Solar [33]	T5 [43] + BLIP [45]	59.8	66.1
MultiModalQA [1]	RoBERTa-large [46] + ViLBERT [47]	49.3	55.9
	ManyModalQA [23]
EDMA	GPT-5.2	88.0	88.5
UniMMQA [2]	T5-3B [43]	52.1	52.4
ManyModalQA [23]	ELMo [48] + BERT [49]	39.7	39.7

Models

Access to Models

We access GPT-5.2 via its official API and all remaining models through Amazon Bedrock. For all experiments, we use greedy decoding (temperature = 0) with a maximum generation length of 1000 tokens.

Computational Resources

Table 5 details the raw average token consumption, API call counts, monetary cost, and latency per model across evaluated approaches (Nova-2-Lite, GPT-5.2, Kimi-K2.5, Nemotron, Qwen3, and Claude-4.6). Furthermore, Table 6 summarizes the relative computational overhead of EDMA compared to each baseline strategy.

Approach	Avg. Absolute Cost Difference vs. EDMA ($) | Avg. Cost Ratio ($\frac{\mathrm{EDMA}}{\mathrm{Approach}}$) | Avg. Latency Ratio ($\frac{\mathrm{EDMA}}{\mathrm{Approach}}$)
CoT	0.069	$5.91\times$	$22.83\times$
DP	0.070	$6.40\times$	$45.79\times$
PP	0.068	$5.06\times$	$19.92\times$
MAMMQA	0.011	$1.14\times$	$11.05\times$

As shown in Table 6, EDMA is on average 22.83$\times$ slower than CoT, 45.79$\times$ slower than DP, 19.92$\times$ slower than PP, and 11.05$\times$ slower than MAMMQA. This reflects the higher computational complexity of the algorithm. Regarding financial cost, EDMA is on average 5.06$\times$ more expensive than PP, 5.91$\times$ more expensive than CoT, and 6.40$\times$ more expensive than DP. However, its cost is directly comparable to MAMMQA (1.14$\times$). Although MAMMQA executes only seven API calls per query, it repeatedly inputs all data modalities (text, images, and tables) into the LLM context, making its token consumption and cost comparable to EDMA. Note that estimates for MAMMQA were computed across all models except GPT-5.2 and Qwen3, where evaluations were omitted due to budget restrictions.

Analysis of Intermediate Operations

Table 7 reports the average accuracy across each intermediate phase of EDMA, consisting in modality selection, criteria extraction, intermediate answer extraction, and multimodal fusion, evaluated specifically on questions where EDMA produced an incorrect final answer.

Model	Modality Selection	Criteria Extraction	Intermediate Answers	Multimodal Fusion
Nova-2-Lite	90%	94%	92%	4%
GPT-5.2	96%	98%	96%	6%
Kimi-K2.5	96%	96%	96%	8%
Nemotron	90%	94%	30%	2%
Qwen3	96%	96%	96%	8%
Claude-4.6	96%	98%	96%	8%

Across most models, errors primarily originate during multimodal fusion, where the system must cross-reference information extracted from one modality (e.g., text) with another (e.g., images). In these cases, incorrect final answers stem from over-filtered or wrongly filtered partitions, resulting in the selection of a low-entropy yet incorrect multimodal partition. Nemotron is the sole exception: its failures are driven by low accuracy at the earlier Intermediate Answers stage (30%), causing trajectories to fail before reaching the fusion step.

Table 8 presents the corresponding accuracy metrics across all evaluated questions, both correctly and incorrectly answered. Early-stage accuracies (modality selection, criteria extraction, and intermediate answer extraction) remain consistently high across the full dataset. The critical divergence appears in the multimodal fusion phase, confirming that cross-modal fusion constitutes the main point of failure in the methodology.

Model	Modality Selection	Criteria Extraction	Intermediate Answers	Multimodal Fusion
Nova-2-Lite	95%	96%	93%	70%
GPT-5.2	98%	98%	98%	82%
Kimi-K2.5	98%	97%	98%	86%
Nemotron	93%	94%	89%	76%
Qwen3	97%	96%	97%	85%
Claude-4.6	98%	98%	97%	91%

EDMA Statistical Variance

Table 9 reports the mean and standard deviation calculated across ten independent runs. The analysis confirms that the performance advantages observed in single-point comparisons remain stable under repeated evaluation: EDMA’s lead over MAMMQA on Claude-4.6, Nemotron, and Nova-2-Lite, as well as MAMMQA’s advantage on Kimi-K2.5, hold consistently across runs.

Model	Metric	EDMA	PP	DP	CoT	MAMMQA
Claude-4.6	EM	$\mathbf{82.7 \pm1.9}$	57.9 ± 1.9	56.1 ± 2.5	57.8 ± 2.0	–
Claude-4.6	F1	$\mathbf{83.0 \pm1.1}$	70.7 ± 1.3	68.1 ± 1.3	70.8 ± 0.8	–
Claude-4.6	LLM-J	$\mathbf{88.8 \pm0.5}$	85.4 ± 0.4	82.6 ± 0.5	87.5 ± 0.5	82.6 ± 1.3
Kimi-K2.5	EM	$\mathbf{80.7 \pm1.8}$	60.2 ± 2.1	61.9 ± 0.6	58.1 ± 3.1	–
Kimi-K2.5	F1	$\mathbf{81.3 \pm0.3}$	72.4 ± 1.0	73.7 ± 0.3	68.7 ± 2.3	–
Kimi-K2.5	LLM-J	83.3 ± 0.5	81.7 ± 0.4	52.0 ± 0.4	79.3 ± 0.4	$\mathbf{88.3 \pm0.9}$
Nemotron	EM	$\mathbf{80.0 \pm0.8}$	51.9 ± 0.9	41.2 ± 0.7	46.4 ± 0.2	–
Nemotron	F1	$\mathbf{80.7 \pm0.8}$	61.3 ± 0.9	50.1 ± 0.2	54.8 ± 0.4	–
Nemotron	LLM-J	$\mathbf{74.9 \pm0.6}$	64.9 ± 0.6	52.2 ± 0.4	57.0 ± 0.5	66.2 ± 1.1
Nova-2-Lite	EM	$\mathbf{79.1 \pm0.6}$	47.2 ± 1.8	51.1 ± 0.6	42.3 ± 0.6	–
Nova-2-Lite	F1	$\mathbf{80.4 \pm1.1}$	57.1 ± 1.5	60.3 ± 0.7	50.6 ± 0.3	–
Nova-2-Lite	LLM-J	$\mathbf{66.6 \pm0.4}$	42.1 ± 0.3	64.4 ± 0.4	53.3 ± 0.3	65.9 ± 1.4
Average	EM	$\mathbf{80.7 \pm1.2}$	54.3 ± 1.7	52.6 ± 1.1	51.2 ± 1.5	–
Average	F1	$\mathbf{81.3 \pm0.8}$	65.4 ± 1.2	63.1 ± 0.6	61.2 ± 0.9	–
Average	LLM-J	$\mathbf{78.4 \pm0.5}$	68.5 ± 0.4	62.8 ± 0.4	69.3 ± 0.4	75.8 ± 1.2

Ablation Studies

This section presents two ablation studies on the MultiModalQA dataset to isolate the contributions of EDMA’s key components: entropy minimization and cross-modal conditioning.

Ablation 1: Random Trajectory Selection

Table 10 shows the performance impact of disabling entropy minimization during trajectory selection. Instead of choosing the candidate trajectory that minimizes logical entropy, a trajectory is selected uniformly at random while keeping all other pipeline components identical. To account for the stochasticity of random selection, results are averaged over 10 runs using the LLM-Judge metric. Across all evaluated models, selecting trajectories uniformly at random causes a severe performance drop ($-32.2$ percentage points on average) compared to entropy-guided selection. This shows that EDMA’s performance gains do not merely stem from task decomposition into intermediate partitions, but critically rely on selecting the trajectory that minimizes logical entropy.

Ablation 2: Removing Cross-Modal Conditioning

To isolate the contribution of cross-modal conditioning, we disable multimodal partitioning. Specifically, we execute requirement extraction, unimodal partitioning, and entropy-based trajectory selection, but omit the construction of multimodal partitions via cross-modal conditioning. Final answers are thus generated solely from the lowest-entropy unimodal partition.

As shown in Table 11, removing cross-modal conditioning leads to an average performance drop of 14.4 percentage points across all models. This drop is of the same order as the share of questions that require cross-modal reasoning: approximately 35% of questions in MultiModalQA strictly require it, whereas $\sim65\%$ can be answered using a single modality. This is consistent with the framework failing to resolve cross-modal dependencies without cross-modal conditioning, and confirms that multimodal partitioning is fundamental for handling multimodal queries.

Statistical significance

EDMA vs PP

We establish the statistical reliability of the performance gains achieved by EDMA over PP by conducting paired statistical significance testing across all evaluated backbone architectures.

Statistical significance is assessed using both a parametric test (paired two-tailed $t$-test) and a non-parametric test (two-tailed Wilcoxon signed-rank test) at a significance threshold of $\alpha= 0.05$.

Table 12 reports the statistical comparison between EDMA and PP across backbones. EDMA achieves statistically significant improvements ($p < 0.05$) over PP across every model and metric evaluated under both parametric and non-parametric tests, demonstrating robust gains across diverse model families.

Backbone	Metric	EDMA (mean)	PP (mean)	$\Delta$	Sig. ($t$-test)	Sig. (Wilcoxon)
Claude-4.6	EM	82.7	57.9	+24.9	✓	✓
Claude-4.6	F1	83.0	70.7	+12.4	✓	✓
Claude-4.6	LLM-judge	88.8	85.4	+3.3	✓	✓
Kimi-K2.5	EM	80.7	60.2	+20.5	✓	✓
Kimi-K2.5	F1	81.3	72.4	+8.8	✓	✓
Kimi-K2.5	LLM-judge	83.3	81.7	+1.6	✓	✓
Nemotron	EM	80.0	51.9	+28.1	✓	✓
Nemotron	F1	80.7	61.3	+19.4	✓	✓
Nemotron	LLM-judge	74.9	64.9	+10.0	✓	✓
Nova-2-Lite	EM	79.1	47.2	+31.9	✓	✓
Nova-2-Lite	F1	80.4	57.1	+23.3	✓	✓
Nova-2-Lite	LLM-judge	66.6	42.1	+24.5	✓	✓

Modality Fusion Decreases Entropy

We analyse the role of modality fusion in EDMA through the lens of logical entropy reduction over the candidate answer space. Figure 5 compares the distribution of entropy values before and after modality fusion, which occurs when cross-modal evidence is extracted from one modality and used to condition the target modality.

Conditioning the target modality with complementary information from a secondary modality decreases logical entropy, thereby reducing uncertainty. This shows that modality fusion is associated with lower entropy, mitigating uncertainty arising from noisy or ambiguous unimodal signals. Table 13 reports the statistical significance of these reductions across all directional modality pairs.

Table 13: Statistical Evaluation of Entropy Reduction via Information Fusion. $\overline{H}$ represents mean entropy; $t$ represents the t-statistic; Point-Biserial (PB) is the correlation between a binary modality indicator (unimodal vs. multimodal) and entropy values. Negative PB values reflect lower entropy under multimodal representations. Tick (✓) indicates significance at $p < 0.05$.

Combination direction	$\overline{H}(Uni)$	$\overline{H}(Multi)$	Stat ($t$)	Sig. ($p$)	Corr. ($r$)	Sig. ($p$)
Image → Table	0.29	0.27	8.84	✓	-0.16	✓
Image → Text	0.27	0.21	5.80	✓	-0.60	✓
Table → Image	0.25	0.21	6.82	✓	-0.36	✓
Table → Text	0.25	0.20	12.16	✓	-0.43	✓
Text → Image	0.33	0.27	4.11	✓	-0.47	✓
Text → Table	0.27	0.24	9.07	✓	-0.29	✓

Higher Entropy Is Associated with More False Positives

To investigate the relationship between entropy and error rates, we compare entropy levels between correct answers (True Positives, TP) and incorrect answers (False Positives, FP). As detailed in Table 14, false positives consistently exhibit higher mean entropy across all modalities.

We further assess the impact of entropy on False Discovery Rates (FDR) by grouping entropy into Low, Medium, and High level. Table 15 confirms significance via the Friedman test, while Table 16 provides pairwise Wilcoxon signed-rank comparisons across all entropy levels.

Table 15: Statistical comparison across modalities using Mean False Discovery Rates (FDR) and the Friedman Test for Related Samples. $Q$ denotes the Friedman test statistic. Tick (✓) indicates significance at $p < 0.05$.

Modality	$\mathrm{FDR}_{\mathrm{Low}}$	$\mathrm{FDR}_{\mathrm{Medium}}$	$\mathrm{FDR}_{\mathrm{High}}$	Stat (Q)	Sig. (p)
Text	0.081	0.475	0.503	130.72	✓
Image	0.189	0.766	0.782	108.10	✓
Table	0.133	0.459	0.491	40.36	✓

Modality	Low	Med.	Sig. ($p$)	Low	High	Sig. ($p$)
Low vs Medium				Low vs High
Text	0.081	0.475	✓	0.081	0.503	✓
Image	0.189	0.766	✓	0.189	0.782	✓
Table	0.133	0.459	✓	0.133	0.491	✓

Table 17 breaks down the False Discovery Rate (FDR) controlling for modality paths and candidate set size. We split candidate set sizes (ranging from 13 to 26) into three buckets containing approximately 33% of the queries each. Even after controlling for modality path and candidate set size, FDR increases monotonically moving from low to medium and high entropy.

Modality Pair	Candidate Bucket	Low Entropy FDR	Medium Entropy FDR	High Entropy FDR
Text	13–19	0.161	0.479	0.480
Text	20–23	0.084	0.359	0.482
Text	24–26	0.071	0.426	0.461
Text → Image	13–19	0.250	0.933	1.000
Text → Image	20–23	0.267	0.917	1.000
Text → Table	13–19	0.200	0.350	0.533
Text → Table	20–23	0.167	0.428	0.514
Text → Table	24–26	0.130	0.258	0.481
Image → Text	13–19	0.200	0.678	1.000
Image → Text	20–23	0.167	0.333	0.433
Image → Text	24–26	0.173	0.851	1.000
Image	13–19	0.273	0.758	0.766
Image	20–23	0.185	0.678	0.784
Image	24–26	0.158	0.799	0.860
Image → Table	13–19	0.175	0.650	0.823
Image → Table	20–23	0.272	0.550	0.821
Image → Table	24–26	0.217	0.769	0.879
Table → Text	13–19	0.111	0.565	0.620
Table → Text	20–23	0.125	0.600	0.702
Table → Text	24–26	0.045	0.432	0.436
Table → Image	13–19	0.131	0.889	0.933
Table → Image	20–23	0.067	0.675	0.711
Table → Image	24–26	0.200	0.583	0.875
Table	13–19	0.131	0.235	0.475
Table	20–23	0.077	0.107	0.177
Table	24–26	0.036	0.414	0.414

Impact Assessment

The introduction of the EDMA methodology enhances the ability of LLMs to solve cross-modal problems by guiding the answering process through entropy minimization and producing a quantifiable answer trajectory. This traceability allows users to understand which modalities contributed most to the final answer, improving the interpretability and reliability of LLM predictions in real-world scenarios. By providing structured answering trajectories, EDMA also makes LLM-based systems more robust and trustworthy. While privacy and trustworthiness are general concerns associated with LLMs, our methodology does not introduce additional risks beyond those already inherent to LLMs.

Prompts

Prompts EDMA

Prompt Modality Decision

System Prompt

You are a Question Answering assistant. Your task is to determine which modality or combination of modalities is required to answer a given question. You are provided with: text paragraphs, images (with titles), and tables.

Select exactly one of the following labels:

• image: the answer can be derived from images alone<br>• text: the answer can be derived from text alone<br>• table: the answer can be derived from tables alone<br>• image_text: the answer requires both images and text<br>• image_table: the answer requires both images and tables<br>• text_table: the answer requires both text and tables<br>• text_table_image: the answer requires text, tables, and images

Guidelines:<br>• Use only the provided information<br>• Output exactly one label, with no additional text

User Prompt

Question: {question}<br>Images: {images_text}<br>Text Paragraphs: {paragraphs_text}<br>Tables: {tables_text}

Select the required modality or modality combination.

Prompt Extraction of Answer Requirements

System Prompt

You are a Question Answering assistant. Your task is to convert natural-language questions into structured query frames.

Given a question, extract a structured representation with the following components:<br>• Topic. Select exactly one topic from the predefined set: {topics}. Choose the most dominant topic; if unclear, use Other.<br>• Expected Answer Type. Identify both: (i) a specific answer type, and (ii) a general entity category. Focus on the type of entity the answer represents.<br>• Target Entity. Identify the main entity referenced in the question.<br>• Constraints. Extract atomic, independently verifiable conditions from the question. Each constraint should include:<br>1. a constraint type (e.g., relation, award, time, role, qualifier)<br>2. the exact supporting phrase from the question<br>3. a normalized version of that phrase

User Prompt

Question: {question_text}

Prompt Checking Answer Requirement Paragraph

System Prompt

You are a Question Answering assistant. Your task is to determine whether a text paragraph (or its title) contains any concept specified in a given answer requirement.

User Prompt

Title: {metadata} Paragraph: {text} Answer requirement: {r}

Determine whether the paragraph contains any concept from the answer requirement. If at least one concept from the answer requirement is explicitly supported by the paragraph or its title, respond yes; otherwise, respond no.

Prompt Checking Answer Requirement Image

System Prompt

You are a Question Answering assistant. Your task is to determine whether an image or its title contains any concept specified by an answer requirement.

User Prompt

Title: {metadata} Image: {image data} Answer requirement: {r}

If at least one concept from the answer requirement is explicitly supported by the image content or its title, respond yes; otherwise, respond no.

Prompt Checking Answer Requirement Table Row

System Prompt

You are a Question Answering assistant. Your task is to determine whether a table row contains any concept specified a given answer requirement.

User Prompt

Table Row: {table row} Answer requirement: {r}

If at least one concept from the answer requirement is explicitly supported by the row content, respond yes; otherwise, respond no.

Prompt to Extract Information Text

System Prompt

You are a Question Answering assistant. Your task is to extract relevant information from a given paragraph in relation to a question. You are given the question, the paragraph title, and the paragraph. Follow the instructions to extract the relevant information:

1. Step 1. Question analysis: determine whether the paragraph is related to the question using only the provided inputs.

2. Step 2. Evidence extraction: if the paragraph is related to the question, extract the minimal information relevant to answering the question. The description must include only the entity or value that directly connects the paragraph to the question.

User Prompt

Question: {question_text}<br>Paragraph Title: {title}<br>Paragraph: {text}<br><br>Provide the relevant information according to the instructions.

Prompt to Extract Information Image

System Prompt

You are a Question Answering assistant. Your task is to extract relevant information from a given image in relation to a question. You are given the question, the image title, and the image. Follow the instructions to extract the relevant information:

• Step 1. Question analysis: determine whether the image is related to the question using only the provided inputs.

• Step 2. Evidence extraction: if the image is related to the question, produce a short textual description of the image that captures the minimal information relevant to answering the question. The description must include only the entity or value that directly connects the image to the question.

User Prompt

Question: {question_text}<br>Image Title: {title}<br>Image: {text}<br><br>Provide the relevant information according to the instructions.

Prompt to Extract Information Table Row

System Prompt

You are a Question Answering assistant.

Your task is to extract relevant information from a given table row in relation to a question.

You are given the question and the table row. Follow the instructions to extract the relevant information:

• Step 1. Question analysis: determine whether the table row is related to the question using only the provided inputs.

• Step 2. Evidence extraction: if the table row is related to the question, produce a short textual description of the table row that captures the minimal information relevant to answering the question. The description must include only the entity or value that directly connects the table row to the question.

User Prompt

Question: {question_text}<br>Paragraph Title: {title}<br>Paragraph: {text}<br><br>Provide the relevant information according to the instructions.

Prompt to Check Mutual Information Between Paragraphs

System Prompt

You are a Question Answering assistant.

You are given a question and two paragraphs. Your task is to determine whether any entity, concept, or fact mentioned in one paragraph is also mentioned in the other paragraph.

User Prompt

Question: {question_text}<br>Title Paragraph 1: {title_1}<br>Paragraph 1: {content_paragraph_1}<br>Title Paragraph 2: {title_2}<br>Paragraph 2: {content_paragraph_2}<br><br>Determine whether any entity, concept, or fact mentioned in Paragraph 1 is also present in Paragraph 2. If any shared entity, name, or concept appears in both paragraphs, respond yes. Otherwise respond no.

Prompt Check Mutual Information Between Textual Description and Image

System Prompt

You are a Question Answering assistant. You are given a question, one image, and an image title and a textual description. Your task is to determine whether any entity, concept, or fact mentioned in the textual description is also present in the image or image title.

User Prompt

Question: {question_text}<br>Text: {content}<br>Image Title: {image_title}<br>Image: {image content}<br><br>Determine whether any entity, concept, or fact mentioned in the Text is also present in the Image Title or in the Image. If any shared entity, name, or concept appears in both the Text and the Image Title or in the Text and the Image, respond yes. Otherwise respond no.

Prompt to Check Mutual Information Between Textual Description and Table Row

System Prompt

You are a Question Answering assistant. You are given a question, paragraph (textual description), a table description, and a table row. Your task is to determine whether any entity, concept, or fact mentioned in the textual description is also present in the table row. Use the table description to understand the context.

User Prompt

Question: {question_text}<br>Paragraph: {content}<br>Table Description: {table_description}<br>Table Row: {table_row}<br><br>Determine whether any entity, concept, or fact mentioned in the Paragraph is also present in the Table Row. If any shared entity, name, or concept appears in both the Paragraph and the Table Row, respond yes. Otherwise respond no.

Check Answer in Paragraph

System Prompt

You are a Question Answering assistant. You are given a question, a paragraph, the expected answer type. Your task is to determine whether the paragraph contains an entity that completely or partially matches the expected answer type required by the question. Guidelines: • This is not keyword matching. Perform semantic reasoning using synonyms, paraphrases, and implicit mentions • Only return TRUE if the paragraph contains a specific entity that could fully or partially answer the question. Partial answers are acceptable if relevant. • If the paragraph only provides context without an answer entity, return FALSE. If uncertain, return FALSE. • If TRUE, extract the exact text span from the paragraph without paraphrasing or normalization. If FALSE, return NONE

User Prompt

Question: {question_text} Expected Answer Type: {answer_class} Paragraph: {paragraph_text} Check the presence of the answer based on the instructions provided above.

Check Answer in Image

System Prompt

You are a Question Answering assistant. Your task is to determine whether either the caption or the image contain an entity that matches the expected answer type required by the question. You are given the question, the image title, and an image. You must evaluate both the image title and the image. Guidelines: • First check whether the caption explicitly contains an entity that satisfies the expected answer type. If not, check whether the image contains a visually identifiable entity that satisfies the answer type. If neither modality satisfies either answer type, return FALSE. • If uncertain, return FALSE. • If evidence comes from the caption, extract the exact text span. If evidence comes from the image, provide a concise visual description. If no match, the returned entity must be NONE. • Critical instructions for the Image: Examine all parts of the image carefully, including background, corners, and small or partially obscured objects. Pay attention to subtle details such as tiny text, small objects, fine patterns, colours, positions, or partially visible entities. Do not rely only on large, obvious objects. Small details may be crucial to answer the question. Only return TRUE for contains if the entity is clearly present in the image. If uncertain, return FALSE.

User Prompt

Question: {question_text} Expected Answer Type: {answer_class_specific} Image Title: {caption_text} Check the presence of the answer in the image and its title based on the instructions provided above.

Check Answer in Table Row

System Prompt

You are a Question Answering assistant. Your task is to determine whether a table row contains a value that directly answers a question. You are given the question, the expected answer type, the table description, and one table row. Guidelines: • Use the description of the table to interpret the row correctly • Do not use only literal meaning. Also use synonyms and contextual meaning • Only return TRUE if one of the row’s values directly answers the question • If the row contains related information but no direct answer, return FALSE • If uncertain, return FALSE • If TRUE, extract the exact cell value from the row that contains the answer. If FALSE, return NONE

User Prompt

Question: {question_text} Expected Answer Type: {answer_class_specific} Table Description: {table_description} Table Row: {row} Check the presence of the answer in table row based on the instructions provided above.

Baselines Prompts

Prompt for Direct Prompting

System Prompt

You are a multimodal question answering assistant. Your task is to answer a given question using three types of provided data sources: text paragraphs, images (with titles), and tables. Carefully review all sources and derive the answer by combining relevant information across modalities. The final response should be concise and contain only the answer, without any explanation. If the necessary information is not available, output None.

User Prompt

Question: {question_text} Images: {images_text} Text Paragraphs: {paragraphs_text} Tables: {tables_text} Provide the final answer to the question.

Prompt for Chain-of-Thought

System Prompt

You are a multimodal question answering assistant. Your task is to answer a given question using only the provided information sources: text paragraphs, images (with titles), and tables. You must not rely on any external or prior knowledge. If the provided information is insufficient, output None.

Procedure to follow:

• Step 1. Extract information: Identify relevant facts from each modality: text (key statements), images (objects, labels, relationships), and tables (rows, columns, values).

• Step 2. Align information: Match entities and concepts that appear across different modalities and explicitly link them.

• Step 3. Integrate information: Combine the aligned evidence to form a complete understanding, ensuring that all parts of the question are addressed.

• Step 4. Derive the answer: Formulate the final answer, clearly grounded in the combined evidence from the modalities.

• Step 5. Verify: Ensure that the answer is fully supported by the provided information and that no relevant modality has been ignored.

Output the reasoning separated from the final response. The final response should be concise and contain only the answer, without any explanation.

User Prompt

Question: {question_text}<br>Images: {images_text}<br>Text Paragraphs: {paragraphs_text}<br>Tables: {tables_text}

Answer the question following the procedure above.

Prompt for Planning Prompting

System Prompt

You are a multimodal question answering assistant. Your task is to answer a given question using the provided data sources: text paragraphs, images (with titles), and tables. You must rely only on this information and not use any external knowledge. If the provided information is insufficient, output None.

Before producing the final answer, follow the three phases below:<br>Phase 1: Strategic Planning<br>Analyse the question and available data sources, and devise a step-by-step plan. The plan should:

• Identify which data sources are most relevant

• Determine the order in which they should be processed

• Specify how to handle potential conflicts across sources

Phase 2: Execution<br>Follow the plan to extract and combine information. Every piece of evidence must be explicitly grounded in a source (e.g., table, image title, or paragraph).

Phase 3: Final Answer<br>Provide a concise answer based on the executed plan. If the required information is not available, output None.

User Prompt

Question: {question_text}<br>Images: {images_text}<br>Text Paragraphs: {paragraphs_text}<br>Tables: {tables_text}

Follow the planning and execution procedure to answer the question.