WorksComputer Science
NaviGATE: Noise-Aware Dual-View Generation for Multimodal Knowledge Graph Reasoning
Abstract
Multimodal knowledge graph reasoning (MKGR) aims to infer missing facts by jointly exploiting relational structures, textual descriptions, and visual evidence. Existing embedding-based methods effectively capture graph topology, but their fixed multimodal representations are less effective for query-specific fine-grained semantic reasoning, while generation-based methods are vulnerable to noisy retrieval and may produce invalid entities. To address these challenges, we propose NaviGATE, a noise-aware generative framework for MKGR. NaviGATE first aligns graph-derived structural representations with textual semantics, then performs structure-aware internal knowledge activation by fine-tuning the multimodal language model with multimodal entity summaries and injected structural tokens. In the noise-aware dual-view instruction tuning stage, NaviGATE jointly optimizes retrieval-free and retrieval-aware views: reliable retrieved candidates are reinforced through contrastive logit sharpening, while misleading candidates are suppressed through parametric consistency distillation. In inference, structure-constrained decoding restricts generation to valid entities in the target knowledge graph. Experiments on standard benchmarks show that NaviGATE consistently outperforms strong unimodal, multimodal, and generation-based baselines, and remains robust under noisy retrieval scenarios.
Introduction
Multimodal knowledge graphs (MMKGs) integrate structured relational facts with multimodal information, such as textual descriptions and visual images, and have become important knowledge resources for artificial intelligence applications, including recommendation systems [25, 9], question answering (Li, Wang, and Zhu 2020), and information retrieval [14, 8]. However, real-world knowledge graphs are often incomplete, with many valid facts missing from the observed graph. Multimodal knowledge graph reasoning (MKGR) aims to predict missing entities by jointly exploiting graph structure and entity-associated multimodal context, and has attracted increasing attention in recent years.
Existing MKGR methods can be broadly categorized into embedding-based and generation-based approaches. Embedding-based methods encode entities, relations, and multimodal attributes into continuous representation spaces, and predict missing facts by scoring candidate triples. With graph neural networks [18] (Li et al. 2023), translation-based constraints [7] (Li et al. 2026), or multimodal fusion modules [30, 31, 33], these methods effectively preserve structural topology and enrich entity representations with textual and visual evidence. However, their dependence on fixed multimodal representations and predefined scoring functions limits their ability to perform query-specific fine-grained semantic reasoning. As a result, when multiple candidates are semantically similar, embedding-based models may narrow down the search space but remain insufficient to distinguish the exact target entity.
Generation-based methods [29, 32] offer a promising alternative by formulating MKGR as a text generation problem. Leveraging the semantic understanding and parametric knowledge of multimodal large language models (MLLMs), these methods can generate missing entities from textualized queries and multimodal contexts. Compared with embedding-based models, they provide greater flexibility in contextual reasoning and can better exploit relation semantics, entity descriptions, and visual information. Nevertheless, direct generation may produce answers that are not valid entities in the target knowledge graph or are inconsistent with the graph schema, which limits their completion performance. Recent retrieval-augmented frameworks [16, 24, 15] mitigate this issue by using an embedding-based retriever to obtain a top- candidate set, which is then provided to the MLLM for reranking or generation. By combining structural recall with semantic reasoning, these methods have achieved notable performance improvements. However, this pipeline is fundamentally constrained by retriever recall: if the ground-truth entity is absent from the top- candidate set, MLLMs may fail to reliably generate the correct answer; meanwhile, feeding the entire entity set into the model is impractical due to their limited context windows and attention budgets.
In realistic MKGR scenarios, retrieved candidates are not always reliable, since the ground-truth entity may be absent from the candidate set, making the retrieved candidate set noisy. Over-reliance on retrieval may force the generator to select an incorrect entity from the candidate set. This raises a key challenge: **how can a generative MKGR model rely on retrieved evidence when it is reliable, while falling back on the LLM’s internal knowledge when the retrieved candidate set is noisy?
To address this challenge, we propose NaviGATE, a progressively optimized generative framework for MKGR. First, structure-semantic collaborative alignment coordinates structural entity representations with textual semantics in a shared space, enabling graph-derived knowledge to be integrated into the MLLM. Second, structure-aware internal knowledge activation trains the model with multimodal entity summaries and structural cues under a retrieval-free view, preserving its basic completion ability when retrieval is noisy. Third, noise-aware dual-view instruction tuning jointly optimizes retrieval-free and retrieval-aware predictions: reliable candidates are strengthened through contrastive logit sharpening, while misleading candidates are suppressed through parametric consistency distillation. During inference, structure-constrained decoding restricts outputs to valid KG entities. The main contributions of this work are summarized as follows:
We propose structure-semantic collaborative alignment and internal knowledge activation to inject graph-derived knowledge into the MLLM, building retrieval-free completion ability as a reliable fallback under noisy retrieval.
We introduce noise-aware dual-view instruction tuning, which strengthens reliable candidates via contrastive logit sharpening and suppresses misleading ones via parametric consistency distillation. Structure-constrained decoding further ensures valid entity generation.
Experiments on MKG-W and DB15K show that NaviGATE achieves the best MRR and Hits@1 among fifteen baselines and remains robust under noisy retrieval.
Related Work
Embedding-based knowledge graph reasoning
Embedding-based knowledge graph reasoning (KGR) learns entity and relation representations and predicts missing facts by scoring candidate triples. Classical models, such as TransE [2], DistMult [28], ComplEx [21], RotatE [20], and TuckER [1], mainly exploit structural triples to model relational patterns. MKGR methods further incorporate images and textual descriptions into entity representation learning, including IKRL [26], RSME [22], OTKGE [4], IMF (Li et al. 2023), QEB [23], VISTA [10], NativE [30], MyGO [31], and APKGR [7]. These methods improve structural retrieval through modality fusion, interaction, adaptive learning, or noise modeling, but still rely on fixed representations and predefined scoring functions. As fine-grained multimodal clues are compressed before inference, their query-specific reasoning ability remains limited when candidates are semantically similar.
Generation-based knowledge graph reasoning
Generation-based KGR formulates link prediction as a text generation task, where a generative language model produces the missing entity from a textualized query. Early methods, such as GenKGR [27] and KG-S2S [29], convert triples into natural language sequences and apply sequence-to-sequence models for entity generation. With the emergence of large language models, recent studies further explore instruction tuning, in-context learning, and retrieval-augmented generation for KGR. For instance, KG-LLM [19] and KoPA [32] finetunes LLMs on KG completion instructions, while KICGPT [24], SAT [15], and DIFT [16] combine embedding-based retrieval with LLM-based reranking or finetuning. These methods demonstrate the potential of language models in leveraging semantic priors for KGR. Nevertheless, they often remain sensitive to retrieval quality. When retrieved candidates are noisy or the ground-truth entity is absent, generation models can be misled by irrelevant evidence. Moreover, directly generated answers may not always correspond to valid entities in the KG. In contrast, our framework targets multimodal KGR by combining noise-aware dual-view instruction tuning with structure-constrained inference, achieving robust and valid entity generation under noisy multimodal retrieval.
Methodology
As shown in Figure 1, NaviGATE progressively performs structure-semantic collaborative alignment, retrieval-free internal knowledge activation, and noise-aware dual-view instruction tuning to exploit reliable candidates while mitigating misleading ones. Finally, structure-constrained decoding ensures valid entity generation.
Figure 1. Overview of NaviGATE. The framework consists of three progressive training stages: structure-semantic collaborative alignment, structure-aware internal knowledge activation, and noise-aware dual-view instruction tuning. At inference time, structure-constrained decoding ensures valid entity generation.
Problem Formulation
We denote a multimodal knowledge graph as
where and denote the entity set and relation set, respectively. is the set of observed triples. Each entity may be associated with a textual description and a visual image . Given a query triple with one missing entity, MKGR aims to infer the missing entity from the entity set. Depending on whether the missing entity occurs in the tail or head position, the completion query can be formulated as or , respectively, where denotes the unknown entity.
Unlike conventional embedding-based MKGR methods that score candidate triples with fixed representation functions, NaviGATE formulates MKGR as a constrained generation problem. Given the query context , the model directly generates the surface name of the missing entity:
where denotes the token sequence of entity under the tokenizer vocabulary . During inference, the generation space is constrained to valid entity names in .
Structure-Semantic Collaborative Alignment
A central difficulty in generative MKGR is that graph topology and language semantics are represented in fundamentally different spaces. To bridge this gap, NaviGATE first learns structure-aware entity representations and aligns them with textual semantic representations.
Structure-aware entity encoding. We use a graph encoder to encode each entity into a structure-aware representation. For entity , its representation is obtained by aggregating information from its local neighborhood :
where is the initial feature of entity , is a learnable transformation matrix, is the attention weight between and , and is a activation function.
To preserve relational geometry, we further impose a translation-based structural objective [3]. For each positive triple , we construct a corrupted negative triple by replacing either the head or the tail entity. The structural loss is defined as
where denotes the relation embedding, is implemented as the distance, is the margin, and . This objective encourages valid triples to have lower energy than corrupted triples, thereby preserving graph topology in the structural embedding space.
Contrastive structure–text alignment. Although captures graph topology, it resides in a geometry-driven space that is misaligned with the semantic space of language models. To bridge this gap, we ground the structural representation of each entity in its textual semantics. Concretely, we encode the entity description with a frozen language encoder to obtain the textual representation , and map both representations into a shared space:
where and are learnable projection heads, and denotes normalization.
We adopt a symmetric contrastive objective to align matched graph-text pairs while separating mismatched pairs within the same mini-batch. Let denote a mini-batch, and define the InfoNCE loss as
where denotes cosine similarity and is a temperature parameter. The symmetric graph-text contrastive loss is written in expectation form as
The overall alignment objective is
where balances the contributions of the structural and contrastive objectives.
Structure-Aware Internal Knowledge Activation
Retrieval-augmented methods depend heavily on the quality of the retrieved candidate set. However, in realistic MKGR scenarios, the ground-truth entity may be absent due to limited retriever recall. To reduce such over-dependence on retrieval, NaviGATE first activates the model’s retrieval-free completion ability before exposing it to retrieved candidates.
Query-aware multimodal entity summarization. Raw entity descriptions, images, and neighboring triples can be lengthy, redundant, and noisy. Directly concatenating them into the prompt may cause context dilution and make it difficult for the MLLM to identify query-relevant evidence. Therefore, for each entity involved in query , we construct a compact query-aware multimodal summary:
where is the textual description, is the associated image, and denotes the local structural neighborhood of . The summary converts heterogeneous evidence into a concise textual form, highlighting entity attributes, visual cues, and graph-neighborhood semantics relevant to query.
Structural token injection. To explicitly provide the MLLM with graph structural information, we project the aligned structural representation into the input embedding space of the MLLM. Specifically, for each entity , we compute
where is a lightweight multilayer perceptron that matches the dimension of to that of the MLLM token embeddings. The projected vector is treated as a soft structural token and inserted into the input embedding sequence.
For a tail prediction query as an example, the retrieval-free context is constructed as
where provides query-aware multimodal evidence for the observed entity, while encodes its structural context. The context excludes retrieved candidate entities, encouraging the model to predict the missing entity based on its internal parametric knowledge and the provided multimodal and structural evidence.
Retrieval-free warm-up objective. The model is optimized to generate the ground-truth entity name from the retrieval-free context:
This stage equips the model with retrieval-independent completion ability by learning to predict the missing entity from its parametric knowledge and the provided multimodal-structural evidence. It also establishes a retrieval-free prediction distribution that later serves as a stable reference for suppressing misleading candidate evidence.
Noise-Aware Dual-View Instruction Tuning
Building on the retrieval-free completion ability established during internal knowledge activation, NaviGATE is further optimized with retrieved candidates. Let denote the top- candidate entities retrieved for query by an embedding-based retriever. These candidates may provide useful non-parametric evidence, but the ground-truth entity may be absent and the remaining candidates may be misleading. Therefore, the model should learn to exploit retrieved evidence when it is reliable, while preserving its internal parametric knowledge when retrieval is noisy.
To this end, we construct two complementary views for each training query:
The retrieval-free view captures the model’s internal belief without external candidates, while the retrieval-aware view incorporates retrieved entity evidence.
Let denote the token sequence of the ground-truth entity. At each decoding position , under teacher forcing with prefix , the model produces vocabulary logits for each view:
where is the hidden state conditioned on context and prefix , and denotes the vocabulary.
Sharpened generative supervision for reliable retrieval.
When the ground-truth entity is included in the retrieved set, i.e., , the retrieved candidates provide useful external evidence. In this case, the retrieval-aware logits should be encouraged to amplify the preference gain induced by the retrieved context.
Specifically, for each decoding position , we first compute the retrieval-induced logit shift:
where denotes the stop-gradient operation. We then construct sharpened retrieval-aware logits:
where controls the sharpening strength. Equivalently,
This operation amplifies the logit-level preference shift from the retrieval-free view to the retrieval-aware view, thereby strengthening the useful signal introduced by reliable retrieval.
The reliable-retrieval loss is computed as the token-level negative log-likelihood under the sharpened logits:
This objective encourages the model to enlarge the relative advantage of the ground-truth entity under the retrieval-aware view, while suppressing candidate preferences that mainly originate from the retrieval-free prior.
Parametric consistency distillation for noisy retrieval. When the ground-truth entity is absent from the retrieved set, i.e., , applying logit sharpening to the retrieval-aware prediction may bias the model toward plausible but incorrect candidates. Therefore, we use the retrieval-free branch as a parametric anchor and constrain the retrieval-aware branch at each decoding position.
For each output position , we minimize the KL divergence between the retrieval-free token distribution and the retrieval-aware token distribution:
where
The stop-gradient operation prevents the consistency objective from updating the retrieval-free branch, while encouraging the retrieval-aware branch to resist misleading retrieval evidence and remain consistent with the model’s internal parametric knowledge. Finally, the dual-view tuning objective is
where balances the parametric consistency objective.
Structure-Constrained Inference
Unconstrained generation may produce entity names that do not exist in the target knowledge graph. Since MKGR requires each prediction to correspond to a valid entity in , we introduce a structure-constrained decoding strategy to ensure entity validity.
We first construct a prefix trie from the tokenized names of all entities in , with an end-of-sequence token appended to each entity name. At decoding step , given the generated prefix , the set of valid next tokens is defined as
where denotes token concatenation. We then define the vocabulary constraint mask as
During inference, NaviGATE computes the retrieval-free and retrieval-aware token logits, denoted by and , respectively. Both predictions are conditioned on the generated prefix . We define the retrieval-induced logit shift as
Following the logit-level adjustment used during training, the inference logits are computed as
where controls the adjustment strength. The final decoding distribution is defined as
The prefix constraint ensures that each completed output corresponds to a valid entity name in the target KG. When retrieved candidates provide useful evidence, the logit adjustment strengthens the retrieval-induced preference. Under noisy retrieval, parametric consistency distillation reduces the discrepancy between the retrieval-free and retrieval-aware predictions, thereby limiting the influence of misleading candidates.
Experiments
Experimental Setup
Datasets. We evaluate NaviGATE on two widely used multimodal knowledge graph reasoning benchmarks, namely DB15K and MKG-W. DB15K contains 12,842 entities, 279 relations, 79,222 training triples, 9,902 validation triples, and 9,904 test triples. MKG-W contains 15,000 entities, 169 relations, 34,196 training triples, 4,276 validation triples, and 4,274 test triples. Both datasets provide structural triples together with multimodal entity information, making them suitable for evaluating the ability of MKGR models.
Baselines. We compare NaviGATE with three categories of baseline methods. First, we include representative unimodal knowledge graph embedding models that rely only on structural triples, including TransE [2], DistMult [28], ComplEx [21], RotatE [20], and TuckER [1]. Second, we compare with discriminative MKGR methods that incorporate visual or textual information into entity representation learning, including OTKGE [4], IMF (Li et al. 2023), QEB [23], AdaMF [34], SnAg [5], NativE [30], MyGO [31], and APKGR [7]. Third, we include generation-based KGR methods, including DIFT [16] and SAT [15], to evaluate whether NaviGATE can leverage generative reasoning while maintaining robustness to noisy retrieval results.
Implementation Details
We conduct all experiments on an Intel Xeon Gold CPU with 4 NVIDIA 4090 GPUs, each with 24 GB memory, under Ubuntu 18.04 LTS. We select TuckER and RotatE as representative embedding-based retrievers. Each retriever is first pretrained on the training set, and the top-20 predicted entities are then retrieved for each query as candidate entities. For the multimodal generative backbone, we use Qwen3-VL-8B as the base model and adopt LoRA (Hu et al. 2022) for parameter-efficient finetuning. The LoRA rank, scaling factor, and dropout rate are set to 64, 16, and 0.1, respectively. We train the model for 2 epochs in the internal knowledge activation stage and 4 epochs in the dual-view instruction tuning stage. During inference, we use prefix-trie constrained decoding to generate a valid entity, place it at the top of the retriever’s original ranking, remove any duplicate occurrence, and preserve the relative order of all remaining entities. We then report standard filtered MRR and Hits@1/3/10.
Main Results
Table 1 reports the overall results. Compared with unimodal KGR methods, NaviGATE consistently achieves better performance on both datasets, demonstrating the effectiveness of integrating multimodal information with generative reasoning for MKGR.
| Models | MKG-W | DB15K | ||||||
| MRR | Hits@1 | Hits@3 | Hits@10 | MRR | Hits@1 | Hits@3 | Hits@10 | |
| Unimodal KGR Methods | ||||||||
| TransE ([2]) | 0.292 | 0.211 | 0.332 | 0.442 | 0.249 | 0.128 | 0.315 | 0.471 |
| DistMult ([28]) | 0.210 | 0.159 | 0.223 | 0.309 | 0.230 | 0.148 | 0.263 | 0.396 |
| ComplEx ([21]) | 0.249 | 0.191 | 0.267 | 0.367 | 0.275 | 0.184 | 0.316 | 0.454 |
| RotatE ([20]) | 0.334 | 0.270 | 0.362 | 0.451 | 0.291 | 0.184 | 0.355 | 0.479 |
| TuckER ([1]) | 0.301 | 0.243 | 0.323 | 0.408 | 0.340 | 0.258 | 0.374 | 0.501 |
| Discriminative MKGR methods & Generative Models | ||||||||
| OTKGE ([4]) | 0.344 | 0.289 | 0.363 | 0.449 | 0.239 | 0.185 | 0.259 | 0.342 |
| IMF ((Li et al. 2023)) | 0.345 | 0.288 | 0.366 | 0.454 | 0.323 | 0.242 | 0.360 | 0.482 |
| QEB ([23]) | 0.324 | 0.255 | 0.351 | 0.453 | 0.282 | 0.148 | 0.367 | 0.516 |
| AdaMF ([34]) | 0.343 | 0.272 | 0.379 | 0.472 | 0.325 | 0.213 | 0.397 | 0.517 |
| NativE ([30]) | 0.366 | 0.296 | 0.395 | 0.494 | 0.362 | 0.273 | 0.411 | 0.529 |
| SnAg ([5]) | 0.359 | 0.284 | 0.394 | 0.492 | 0.342 | 0.250 | 0.391 | 0.515 |
| MyGO ([31]) | 0.361 | 0.300 | 0.383 | 0.469 | <u>0.372</u> | 0.296 | <u>0.413</u> | 0.520 |
| APKGR ([7]) | 0.374 | 0.306 | 0.404 | <u>0.501</u> | 0.364 | 0.282 | <u>0.413</u> | <u>0.527</u> |
| DHNS ([17]) | 0.346 | 0.268 | 0.392 | 0.512 | 0.364 | 0.297 | 0.398 | 0.492 |
| DIFT+TuckER ([16]) | 0.361 | 0.317 | 0.390 | 0.442 | 0.367 | 0.304 | 0.402 | 0.505 |
| DIFT+RotatE ([16]) | <u>0.390</u> | <u>0.351</u> | <u>0.416</u> | 0.468 | 0.332 | 0.267 | 0.366 | 0.482 |
| SAT+TuckER ([15]) | 0.363 | 0.320 | 0.388 | 0.441 | 0.369 | <u>0.305</u> | 0.401 | 0.503 |
| SAT+RotatE ([15]) | 0.389 | 0.347 | 0.415 | 0.471 | 0.336 | 0.271 | 0.369 | 0.485 |
| NaviGATE+TuckER | 0.378 | 0.330 | 0.403 | 0.453 | 0.383 | 0.313 | 0.415 | 0.516 |
| NaviGATE+RotatE | 0.408 | 0.361 | 0.430 | 0.487 | 0.348 | 0.279 | 0.379 | 0.491 |
Table 1. Overall results on MKG-W and DB15K. The best results are shown in bold, and the second-best results are underlined. We reproduce the results of RotatE, TuckER, DIFT, and SAT using their released code. Results for the remaining baselines are taken from their respective original papers.
Compared with discriminative MKGR baselines, NaviGATE achieves stronger and more consistent performance on the key top-ranking metrics across both datasets. On MKG-W, NaviGATE+RotatE improves the best MRR and Hits@1 from 0.374 and 0.306 to 0.408 and 0.361, respectively, demonstrating a clear advantage in ranking the correct entity at the top. On DB15K, NaviGATE+TuckER reaches an MRR of 0.383 and a Hits@1 of 0.313, also surpassing the strongest discriminative baselines.
Compared with LLM-based methods such as DIFT and SAT, NaviGATE achieves further improvements under both retrievers. On MKG-W, NaviGATE+RotatE improves MRR by 1.8 percentage points over DIFT+RotatE, while NaviGATE+TuckER outperforms SAT+TuckER by 1.5 percentage points. On DB15K, NaviGATE+TuckER further improves MRR by 1.4 percentage points over SAT+TuckER. These results show that noise-aware tuning effectively exploits reliable candidates while mitigating noisy retrieval.
Ablation Study
Table 2 presents the ablation results on MKG-W. Removing stage-1 pretraining or the structural contrastive loss decreases MRR from 0.408 to 0.401 and 0.405, respectively, confirming that structure-semantic alignment provides a useful foundation for subsequent generative reasoning. In the dual-view tuning stage, removing stage-2 finetuning causes a substantial drop in both MRR and Hits@1, while removing the retrieval-aware branch results in the lowest Hits@1 of 0.248. These results indicate that retrieved evidence remains essential when properly incorporated through noise-aware optimization. The multimodal summary, structure-constrained decoding, KL loss, and retrieval-free branch also yield consistent improvements, demonstrating their complementary roles in semantic enrichment, valid entity generation, and robustness to noisy candidates. Among the three modalities, removing textual information leads to the largest degradation, whereas visual and structural information provide smaller but stable gains. To directly evaluate robustness under retrieval failure, we further report results on the subset where the ground-truth entity is absent from the candidate set. As shown in Table 2, NaviGATE improves Hits@1 by 1.1 percentage points over the variant without parametric consistency, confirming its ability to suppress misleading retrieval evidence.
| Variants | MRR | H@1 | H@3 | H@10 |
| Stage-1 Structural Design | ||||
| w/o stage1 pretrain | 0.401 | 0.357 | 0.421 | 0.478 |
| w/o structural contrastive loss | 0.405 | 0.357 | 0.430 | 0.486 |
| Stage-2 Reranking Design | ||||
| w/o stage2 finetune | 0.338 | 0.256 | 0.390 | 0.468 |
| w/o multimodal text summary | 0.387 | 0.334 | 0.411 | 0.478 |
| w/o structure-const. inference | 0.395 | 0.348 | 0.415 | 0.481 |
| w/o KL loss | 0.397 | 0.352 | 0.416 | 0.477 |
| w/o retrieval-free | 0.398 | 0.352 | 0.416 | 0.480 |
| w/o retrieval-aware | 0.338 | 0.248 | 0.398 | 0.478 |
| Modality Ablation | ||||
| w/o visual modality | 0.399 | 0.350 | 0.423 | 0.481 |
| w/o textual modality | 0.364 | 0.307 | 0.389 | 0.466 |
| w/o structural modality | 0.401 | 0.350 | 0.427 | 0.485 |
| Noisy-Retrieval Subset () | ||||
| w/o parametric consistency | 0.396 | 0.350 | 0.423 | 0.482 |
| Full Model | 0.408 | 0.361 | 0.430 | 0.487 |
Table 2. Ablation results on MKG-W across training stages, dual-view tuning components, and modality configurations. “w/o” denotes removing the corresponding component from the full model. Best results are shown in bold.
Impact of the Number of Retrieved Candidates. Figure 2 analyzes the effect of candidate number on MKG-W. Increasing candidates from 10 to 20 improves MRR from 0.391 to 0.408 and Hits@1 from 0.347 to 0.361, showing that a moderate candidate set provides more useful retrieval evidence. However, further increasing the number to 30 or

Figure 2. Impact of the number of candidates in prompt inputs on model performance and training time on the MKG-W dataset.
40 brings no additional gain and slightly degrades performance, likely due to more noisy distractors in the prompt. Since training time also increases steadily with more candidates, we use 20 candidates as the default setting for better accuracy-efficiency trade-off.
Hyperparameter Sensitivity. Figure 3 shows the sensitivity of NaviGATE to and on MKG-W. For , the model performs stably when it ranges from 0.01 to 0.5, while a larger value of 1.0 slightly degrades both MRR and Hits@1. This suggests that moderate parametric consistency helps suppress noisy retrieval, whereas overly strong regularization may limit the benefit of retrieval-aware evidence. For , the best performance is achieved around 0.2–0.5. A small provides insufficient logit sharpening, while a large value over-amplifies retrieval-induced shifts and introduces noise. Therefore, we set and by default.
Figure 3. Hyperparameter sensitivity analysis on the MKG-W dataset: (a) Impact of the KL divergence weight under noisy retrieval conditions, and (b) Impact of the exploration rate in contrastive logit sharpening.
Case Study. Figure 4 shows representative cases under different retrieval conditions. When the ground-truth entity appears in the candidate list, NaviGATE exploits retrieval evidence to generate the correct answer. More importantly, even when the ground-truth entity is absent from the candidates, NaviGATE can still recover it using activated internal knowledge. The failure cases further show that distinguishing highly confusable entities under ambiguous multimodal cues remains challenging, highlighting the inherent difficulty of noise-aware reasoning in MKGR.
Figure 4. Case study of NaviGATE under different retrieval conditions. The upper part shows examples where the ground-truth entity is included in the candidate list, while the lower part shows examples where it is absent.
Conclusion
In this paper, we propose NaviGATE, a noise-aware generative framework for multimodal knowledge graph reasoning. NaviGATE aligns structural representations with textual semantics, activates retrieval-free completion through multimodal summaries and structural token injection, and employs dual-view instruction tuning to exploit reliable candidates while suppressing misleading ones via parametric consistency. Structure-constrained decoding further ensures valid entity generation. Experiments demonstrate its effectiveness and robustness. In future work, we will extend NaviGATE to incorporate more modality information beyond text and images, and explore robust designs for missing-modality scenarios to improve real-world MKGR reliability.
References
- [1]Balazevic, I.; Allen, C.; Hospedales, T.; and abc. 2019. TuckER: Sentence Factorization for Knowledge Graph Completion. In Inui, K.; Jiang, J.; Ng, V.; and Wan, X., eds., Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5185–5194. Hong Kong, China: Association for Computational Linguistics.DOI
- [2]Bordes, A.; Usunier, N.; Garcia-Durán, A.; Weston, J.; and Yakhnenko, O. 2013a. Translating embeddings for modeling multi-relational data. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, 2787–2795. Red Hook, NY, USA: Curran Associates Inc.
- [3]Bordes, A.; Usunier, N.; Garcia-Durán, A.; Weston, J.; and Yakhnenko, O. 2013b. Translating embeddings for modeling multi-relational data. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, 2787–2795. Red Hook, NY, USA: Curran Associates Inc.
- [4]Cao, Z.; Xu, Q.; Yang, Z.; He, Y.; Cao, X.; and Huang, Q. 2022. OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport. In Advances in Neural Information Processing Systems, volume 35, 39090–39102.DOI
- [5]Chen, Z.; Fang, Y.; Zhang, Y.; Guo, L.; Chen, J.; Pan, J. Z.; Chen, H.; and Zhang, W. 2025. Noise-powered Multi-modal Knowledge Graph Representation Framework. In Rambow, O.; Wanner, L.; Apidianaki, M.; Al-Khalifa, H.; Eugenio, B. D.; and Schockaert, S., eds., Proceedings of the 31st International Conference on Computational Linguistics, 141–155. Abu Dhabi, UAE: Association for Computational Linguistics.arxiv.org/abs/2403.06832
- [7]Jian, Y.; Luo, X.; Li, Z.; Zhang, M.; Zhang, Y.; Xiao, K.; and Hou, X. 2025. APKGC: Noise-enhanced Multi-Modal Knowledge Graph Completion with Attention Penalty. In Walsh, T.; Shah, J.; and Kolter, Z., eds., Thirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2025, Philadelphia, PA, USA, February 25 - March 4, 2025, 15005–15013. AAAI Press.DOI
- [8]Kim, J.; Lee, G.; Kim, T.; and Shin, K. 2025. KGMEL: Knowledge Graph-Enhanced Multimodal Entity Linking. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’25, 3015–3019. New York, NY, USA: Association for Computing Machinery. ISBN 9798400715921.DOI
- [9]Kim, Y.; Rome, S.; Foley, K.; Nankani, M.; Melamed, R.; Morales, J.; Yadav, A.; Peifer, M.; Hamidian, S.; and Huang, H. H. 2024. Improving Content Recommendation: Knowledge Graph-Based Semantic Contrastive Learning for Diversity and Cold-Start Users. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, 8743–8755.arxiv.org/abs/2403.18667
- [10]Lee, J.; Chung, C.; Lee, H.; Jo, S.; and Whang, J. J. 2023. VISTA: Visual-Textual Knowledge Graph Representation Learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, 7314–7328. Singapore: Association for Computational Linguistics.
- [14]Li, Z.; Ao, Y.; and He, J. 2024. SpherE: Expressive and Interpretable Knowledge Graph Embedding for Set Retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2629–2634.arxiv.org/abs/2404.19130
- [15]Liu, Y.; Cao, Y.; Lin, X.; Shang, Y.; Wang, S.; and Pan, S. 2025a. Enhancing Large Language Model for Knowledge Graph Completion via Structure-Aware Alignment-Tuning. In Christodoulopoulos, C.; Chakraborty, T.; Rose, C.; and Peng, V., eds., Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 20970–20984. Suzhou, China: Association for Computational Linguistics. ISBN 979-8-89176-332-6.arxiv.org/abs/2509.01166
- [16]Liu, Y.; Tian, X.; Sun, Z.; and Hu, W. 2025b. Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion. In Demartini, G.; Hose, K.; Acosta, M.; Palmonari, M.; Cheng, G.; Skaf-Molli, H.; Ferranti, M.; Hernández, D.; and Hogan, A., eds., The Semantic Web – ISWC 2024, 199–217. Cham: Springer Nature Switzerland. ISBN 978-3-031-77844-5.arxiv.org/abs/2407.16127
- [17]Niu, G.; and Zhang, X. 2026. Diffusion-Based Hierarchical Negative Sampling for Multimodal Knowledge Graph Completion. In Zhu, F.; Yu, P. S.; Nadamoto, A.; Lim, E.-P.; Shim, K.; Ding, W.; and Zhang, B., eds., Database Systems for Advanced Applications, 479–495. Singapore: Springer Nature Singapore. ISBN 978-981-95-3906-2.arxiv.org/abs/2501.15393
- [18]Peng, J.; Shan, Y.; Zha, Y.; and Wang, X. 2025. Adaptive multimodal graph learning for knowledge graph completion. Data Min. Knowl. Discov., 39(6): 70.DOI
- [19]Shu, D.; Chen, T.; Jin, M.; Zhang, C.; Du, M.; and Zhang, Y. 2025. Knowledge Graph Large Language Model (KG-LLM) for Link Prediction. In Nguyen, V.; and Lin, H.-T., eds., Proceedings of the 16th Asian Conference on Machine Learning, volume 260 of Proceedings of Machine Learning Research, 143–158. PMLR.arxiv.org/abs/2403.07311
- [20]Sun, Z.; Deng, Z.-H.; Nie, J.-Y.; and Tang, J. 2019. RotatE: Knowledge Graph Embedding by Relational Rotation* in Complex Space. In International Conference on Learning Representations.arxiv.org/abs/1902.10197
- [21]Trouillon, T.; Welbl, J.; Riedel, S.; Gaussier, E.; and Bouchard, G. 2016. Complex Embeddings for Simple Link Prediction. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, 2071–2080. PMLR.arxiv.org/abs/1606.06357
- [22]Wang, M.; Wang, S.; Yang, H.; Zhang, Z.; Chen, X.; and Qi, G. 2021. Is Visual Context Really Helpful for Knowledge Graph? A Representation Learning Perspective. In Proceedings of the 29th ACM International Conference on Multimedia, 2735–2743. Association for Computing Machinery.DOI
- [23]Wang, X.; Meng, B.; Chen, H.; Meng, Y.; Lv, K.; and Zhu, W. 2023. TIVA-KG: A Multimodal Knowledge Graph with Text, Image, Video and Audio. In Proceedings of the 31st ACM International Conference on Multimedia, 2391–2399. Association for Computing Machinery.DOI
- [24]Wei, Y.; Huang, Q.; Zhang, Y.; and Kwok, J. 2023. KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 8667–8683. Singapore: Association for Computational Linguistics.DOI
- [25]Xian, Y.; Fu, Z.; Muthukrishnan, S.; de Melo, G.; and Zhang, Y. 2019. Reinforcement Knowledge Graph Reasoning for Explainable Recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 285–294.DOI
- [26]Xie, R.; Liu, Z.; Luan, H.; and Sun, M. 2017. Image-embodied Knowledge Representation Learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, 3140–3146.arxiv.org/abs/1609.07028
- [27]Xie, X.; Zhang, N.; Li, Z.; Deng, S.; Chen, H.; Xiong, F.; Chen, M.; and Chen, H. 2022. From Discrimination to Generalization: Knowledge Graph Completion with Generative Transformer. In Companion Proceedings of the Web Conference 2022, WWW ’22, 162–165. New York, NY, USA: Association for Computing Machinery. ISBN 9781450391306.DOI
- [28]Yang, B.; Yih, W.-t.; He, X.; Gao, J.; and Deng, L. 2015. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In International Conference on Learning Representations.
- [29]Yang, R.; Zhu, J.; Man, J.; Liu, H.; Fang, L.; and Zhou, Y. 2025. GS-KGC: A generative subgraph-based framework for knowledge graph completion with large language models. Information Fusion, 117: 102868.arxiv.org/abs/2408.10819
- [30]Zhang, Y.; Chen, Z.; Guo, L.; Xu, Y.; Hu, B.; Liu, Z.; Zhang, W.; and Chen, H. 2024a. Native: Multi-modal Knowledge Graph Completion in the Wild. In SIGIR, 91–101. ACM.
- [31]Zhang, Y.; Chen, Z.; Guo, L.; Xu, Y.; Hu, B.; Liu, Z.; Zhang, W.; and Chen, H. 2025. Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation. In AAAI, 13322–13330. AAAI Press.arxiv.org/abs/2404.09468
- [32]Zhang, Y.; Chen, Z.; Guo, L.; Xu, Y.; Zhang, W.; and Chen, H. 2024b. Making Large Language Models Perform Better in Knowledge Graph Completion. In Proceedings of the 32nd ACM International Conference on Multimedia, MM ’24, 233–242. New York, NY, USA: Association for Computing Machinery. ISBN 9798400706868.arxiv.org/abs/2310.06671
- [33]Zhang, Y.; Chen, Z.; Liang, L.; Chen, H.; and Zhang, W. 2024c. Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion. In Calzolari, N.; Kan, M.-Y.; Hoste, V.; Lenci, A.; Sakti, S.; and Xue, N., eds., Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 17120–17130. Torino, Italia: ELRA and ICCL.arxiv.org/abs/2402.15444
- [34]Zhang, Y.; Chen, Z.; Liang, L.; Chen, H.; and Zhang, W. 2024d. Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion. In Calzolari, N.; Kan, M.-Y.; Hoste, V.; Lenci, A.; Sakti, S.; and Xue, N., eds., Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 17120–17130. Torino, Italia: ELRA and ICCL.arxiv.org/abs/2402.15444


