betaFeedback

WorksComputer Science

Evaluating Cultural Knowledge Editing in Vision-Language Models

Abstract

Current Vision-Language Models (VLMs) frequently fail to recognize culture-specific concepts, especially from underrepresented regions. Effectively injecting this long-tail knowledge into existing models remains a challenge. In this work, we investigate knowledge editing as a targeted alternative by introducing CulturalBind, a benchmark comprising 3.2K images of 1K concepts from 6 domains, designed to evaluate culture-aware cross-modal knowledge as well as concept-level knowledge editing in VLMs. Pre-edit evaluation reveals that textual and visual knowledge is decoupled: models often demonstrate text-based knowledge of cultural concepts but fail to identify them from images. We then evaluate editing methods in single and sequential scenarios. Although current methods successfully correct individual visual recognition errors, they exhibit low cross-modal portability, failing to integrate edits into the model's broader parametric knowledge. Furthermore, performance degrades rapidly under sequential editing, highlighting the need for comprehensive adaptation approaches that better consolidate multimodal knowledge over time.

Introduction

Current Vision-Language models (VLMs) exhibit broad visual and linguistic capabilities [4, 10] (Wang et al. 2025), yet frequently fail to recognize culture-specific concepts such as traditional musical instruments, regional dishes, and local festivals, particularly those from underrepresented cultures. This reflects a systematic imbalance in pretraining corpora that fail to capture the long tail of global cultural knowledge [40, 33, 27]. As VLMs are increasingly deployed in global contexts, this limitation risks perpetuating cultural blind spots.

One approach to addressing this cultural gap is by curating more diverse pretraining datasets [29, 1]. However, improvements in long-tail cultural concepts are expected to diminish at scale [13] (Udandarao et al. 2024), making full retraining an expensive and uncertain fix. Moreover, subpar performance on cultural benchmarks may not reflect only limited data coverage. Work on general entity knowledge has shown that VLMs can showcase factual knowledge, yet fail to access it through visual input [3, 8], suggesting a similar modality gap may underlie cultural recognition failures.

Knowledge editing offers an efficient alternative, where localized parameter updates address gaps identified post training without affecting the model’s existing knowledge (Wang et al., 2024b). Unlike prior work on entity-level editing [14, 18] (Zhang et al. 2025), where the target is a unique referent, we target concept-level knowledge: the goal is to acquire a generalizable visual category that transfers across instances of the same concept and integrates with the model’s broader parametric knowledge. Whereas an entity edit is validated on the same referent under varying conditions (e.g., the Golden Gate Bridge from different viewpoints), a concept edit must transfer to different instances of the category (e.g., nadaswarams of different makes, photographed in different settings).

To study this problem, we construct CulturalBind, a cross-modal evaluation benchmark of 1K concepts spanning 14 regions and 6 cultural domains, which pairs each concept with encyclopedic and visual attribute knowledge documented in Wikipedia. Before editing, we find that models demonstrate moderate parametric knowledge when probed through language, but recognition from images lags by large margins that do not close with scale. The correlation between textual recall and visual recognition is weak, indicating that the two are largely decoupled. This indicates that the bottleneck is not the absence of factual knowledge but the failure to access it through visual referents.

To evaluate whether editing methods can support cultural adaptation, we assess a range of parameter-modifying and memory-based approaches under two editing scenarios. Single edits evaluate whether methods can fix individual recognition failures, while sequential edits reflect the more practical setting where a model must acquire new concepts incrementally over time. Both settings are evaluated across standard editing metrics and two portability dimensions specific to cross-modal knowledge integration (Figure 1): i) Text-Portability, whether a recognition edit transfers to text-only recall of visual attributes, and ii) Image-Portability, whether it enables visually grounded access to encyclopedic knowledge.

Figure 1. Structure of the CulturalBind editing dataset. Each concept is paired with an edit sample and six auxiliary probes targeting generalization (Text and Image Generality), knowledge preservation (Text and Image Locality), and cross-modal knowledge integration (Text and Image Portability).

Results demonstrate that while editing methods reliably achieve recognition, portability remains uniformly low: editing establishes associations between images and concept labels without grounding them in the model’s broader parametric knowledge, indicating that the visual-linguistic binding gap identified in the cross-modal evaluation persists after editing. In the sequential editing setting, this limitation is exacerbated by rapid performance degradation. Overall, these results indicate that cultural concept acquisition remains a challenge that requires more comprehensive continual adaptation techniques that consolidate multimodal knowledge.

Related Work

Cultural Evaluation of VLMs Recent work has expanded the scope of cultural evaluation for VLMs, including largely multilingual benchmarks ([40, 34]) and diverse task settings ([37, 33, 6] (Yin et al. 2021)).

Cultural Adaptation of VLMs Fewer works have explored approaches to improving cultural awareness in VLMs at different stages of model development. Prior work has proposed geo-diverse pretext objectives ([36]), data augmentation through code-switching and image editing ([19]), and interventions in data compositions ([16]) during pretraining. At the post-training level, recent work has explored pipelines for culturally-aware data construction ([27]). At inference time, retrieval-augmented approaches have been proposed to inject cultural knowledge without modifying model parameters ([2, 15, 26, 28]). These methods target different points in the model lifecycle and can be considered as complementary strategies to improve cultural awareness of VLMs. In this work, we explore knowledge editing as a potential direction for injecting cultural knowledge through parameter updates that are computationally efficient and localized by design.

Multimodal Editing Evaluations Early work by Cheng et al. [7] expanded text knowledge editing frameworks to captioning and VQA tasks, but relied on synthetic images for evaluating edit generalization and offered no assessment of portability. Subsequent work has extended this framework in several directions. VLKEB (Huang et al., [14]) extends the standard reliability, locality, and generality metrics with a portability dimension. MIKE (Li et al., [18]) focuses on fine-grained entity recognition and introduces a multi-task editing evaluation. MC-MKE (Zhang et al., 2025) defines multimodal knowledge in terms of visual and textual triples and evaluates portability from both text and visual edits. More recently, MMKE-Bench (Du et al., [9]) broadens the scope beyond structured knowledge triplets and includes visual semantic and user-specific editing scenarios. Rather than counterfactual edits that overwrite accurate associations, we focus on concept acquisition: the model learns visual categories that it previously failed to recognize. Edit images in CulturalBind are also less visually similar to their generality images than those in entity-based benchmarks (Section A.6), reflecting the greater visual variation across instances of a concept. Additionally, while existing benchmarks assess whether encyclopedic knowledge is accessible from visual input after editing (Huang et al., [14]), we further evaluate whether the model acquires descriptive knowledge of a concept’s visual attributes accessible through text alone.

memorizing a single visual exemplar, the goal is to establish a concept-level association that generalizes across instances of the same category.

Table 1 summarizes the evaluation dimensions. Reliability measures edit success, while Text-Generality and Image-Generality capture generalization to paraphrases of the input question or held-out images of the same concept, respectively. Text-Locality and Image-Locality measure preservation of unrelated text-only and multimodal knowledge. Portability evaluates whether the edited model can apply the acquired knowledge beyond recognition. We focus on cross-modal portability. Text-Portability probes for knowledge transfer by testing whether a model that has learned to visually recognize a concept can also recall its visual attributes without image input. Image-Portability assesses whether recognition can be combined with encyclopedic factual recall through visually grounded questions, requiring the model to bind the concept from an image with factual knowledge. Figure 1 demonstrates an example across evaluation dimensions.

DimensionVisual InputText InputTests
ReliabilityEdit imageRecognition questionEdit success
Text-GeneralityEdit imageParaphrased questionText generalization
Image-GeneralityHeld-out imageRecognition questionImage generalization
Text-Locality—Unrelated text questionPreservation of text knowledge
Image-LocalityUnrelated imageUnrelated visual questionPreservation of VL knowledge
Text-Portability—Visual attribute questionTextual attribute recall
Image-PortabilityEdit imageEncyclopedic questionVisually-grounded factual recall

Table 1. Evaluation dimensions for knowledge editing. Generality targets the generalization of the edit, Locality focuses preservation of unrelated knowledge, and Portability tests cross-modal knowledge integration.

Knowledge Editing Formulation

Knowledge editing modifies a specific piece of knowledge kk within a model with parameters θt−1\theta_{t-1} through an editing process EϕE_{\phi}:

θt=Eϕ(θt−1,k)\theta_t = E_{\phi}(\theta_{t-1}, k)
θt(k′)=θt−1(k′)∀k′≠k(1)\theta_t(k') = \theta_{t-1}(k') \quad\forall k' \ne k \tag*{(1)}

Each edit sample k=(v,q,a∗,a^)k = (v, q, a^*, \hat{a}) consists of an input image vv, a recognition question qq (e.g., “What is this instrument called?”), the correct answer a∗a^*, and the model’s incorrect pre-edit answer a^\hat{a}. Unlike factual editing, where the target is a specific entity or proposition, we target concept-level knowledge (Wang et al., 2024c): rather than memorizing a single visual exemplar, the goal is to establish a concept-level association that generalizes across instances of the same category.

CulturalBind Edit Dataset

We construct a dataset to test VLMs’ knowledge of cultural concepts that refer to generalizable categories with multiple real-world instances, as opposed to named entities, whose recognition requires memorizing a unique referent. The full resource, CulturalBind, comprises 3.2K images from 1K concepts for cross-modal knowledge evaluation (§ 5). For editing experiments, we derive the editing samples by filtering CulturalBind based on model failures, and pairing samples with text and image locality probes (§ 6).

CulturalBind Construction

Concept Selection We collect concepts from two sources: 1) CulturalGround (Nyandwi et al., [27]) is a large-scale cultural VQA dataset covering 42 countries and sourced from Wikidata, and 2) CROPE (Nikandrou et al., [26]), a smaller probing dataset containing culture-specific concepts across five regions validated by native-speaker annotators.

Image Collection and Filtering Evaluating generality across images requires at least two images per concept. For concepts with a single image, we retrieve up to 10 Creative Commons-licensed images using the Openverse API1. Perceptual hashing2 is applied to deduplicate across all images. We then apply a three-stage filtering process to ensure that retained images are relevant and unambiguous. First, we use GPT-5.23 to dismiss images that contain the concept name in text, thereby preventing VLMs from bypassing visual recognition. Second, GPT-5.2 scores each image against the concept’s Wikipedia summary to retain only representative images. Third, we manually inspect all retained Openverse and CulturalGround images to remove remaining ambiguous cases, discarding 96 images. Concepts with fewer than two images are excluded.

Knowledge QA Generation We extract two types of questions from each concept’s Wikipedia page: visual attribute questions, covering features observable in images, and encyclopedic questions, covering knowledge that cannot be inferred from images alone. Table 10 reports the distribution across question categories. For visual attribute knowledge, we prompt GPT-5.2 to extract visual descriptions explicitly stated in the Wikipedia text across three observable categories: physical attributes, actions, and contextual cues, yielding 5,321 questions converted into QA format.

QuestionDescriptionCount
HistoricalDates, events, historical figures1,322
GeographicRegional origins, specific locations1,163
LinguisticEtymology, meaning, language of origin868
CulturalEthnic groups, practices, traditions633
FunctionPurpose, use case, role449
ComparisonSimilarities, equivalents, related items441
CompositionNon-visible ingredients or components372
PhysicalShape, color, size, visible components3,043
ContextualSettings, accompanying objects1,341
ActionMovements, gestures, interactions937

Table 10. Distribution of 5,298 encyclopedic (top) and 5,321 visual attribute (bottom) knowledge questions.

For encyclopedic knowledge, we prompt GPT-5.2 to generate question-answer pairs from a Wikipedia text using a taxonomy of eight knowledge categories (Table 10), excluding information present in the visual descriptions. To evaluate visually-grounded encyclopedic QA, the concept name in each question is replaced with a referring expression, requiring the model to identify the concept from an image before retrieving the associated background knowledge. Following Figure 1, “In which region is the nadaswaram commonly played at weddings?” becomes “In which region is this wind instrument commonly played at weddings?”. We filter out questions that GPT-5.2

can answer correctly without images, resulting in a total of 5,298 encyclopedic questions. We manually audit a sample of 200 QA pairs for question clarity and answer faithfulness (Section A.2).

Dataset Summary The final dataset comprises 1,090 unique concepts paired with 3,216 images, spanning 14 geographic regions and 6 cultural domains. The number of concepts and total samples per region are listed in Table 2. Geographically, coverage is broadest in Asian regions, reflecting the cultural diversity of the source datasets, shown in Figure 4. African sub-regions are underrepresented, mirroring known gaps in Wikidata and Wikipedia coverage. Samples across America are collectively sparse for different reasons: North American concepts are de-prioritized in long-tail cultural evaluations, while Central and South American regions reflect sparser Wikidata coverage. By domain (see Table 9), Foods & Beverages account for nearly 40% of concepts, while the remaining domains are broadly balanced across concrete categories and human activities.

RegionConceptsSamples
Eastern Asia199571
South-eastern Asia164533
Southern Asia168519
Western & Central Asia118366
Southern Europe149356
Northern Europe81166
Eastern Africa22136
Eastern Europe45108
Western Europe26100
Central Am. & Caribbean3392
Northern America1985
Western Africa2464
Southern America2362
Northern Africa1958
Total1,0903,216

Table 2. CulturalBind distribution by region.

DomainConceptsSamples
Foods & Beverages4321,235
Animals & Plants177508
Music & Performing Arts152543
Artefacts135462
Traditions126300
Sports69168
Total1,0903,216

Table 9. CulturalBind distribution by domain.

Figure 4. Geographic distribution of concepts in CulturalGround (after filtering, left) and CROPE (after retrieving images for distractor concepts, right).

Converting to KE Format

Existing multimodal KE benchmarks construct edit samples either from model failures ([7]) or counterfactually ([14, 9]), replacing the true label with a plausible but incorrect target. Counterfactual construction produces a model-agnostic dataset suited to simulating knowledge updates, but can introduce logical inconsistencies: after relabeling images of a nadaswaram as “oboe”, the expected answer for genuine oboe images becomes undefined. By selecting edit samples only for concepts the model fails to recognize, the edit target is always factually correct and consistent with image content, grounding evaluation in practical knowledge acquisition scenarios.

Each edit sample is linked to auxiliary samples that evaluate metrics beyond reliability. For text generality, the original question is paraphrased using Qwen3-32B (Yang et al. (2025)) and paired with the original image, while for image generality, a held-out image of the same concept is paired with the original question. Following MMEdit (Cheng et al. [7]), text locality samples are drawn from Natural Questions (Kwiatkowski et al. [17]). For image locality, VQA-v2 (Goyal et al. [11]) samples are selected by averaging the image-image and text-text similarities with the edit sample using SigLIP2 embeddings (Tschannen et al. [38]). For text portability, we use the generated visual attribute questions posed without image input, while for image portability, visually-grounded encyclopedic questions are paired with the edit image.

Cross-Modal Knowledge Evaluation

A concern when sourcing concepts from Wikipedia is that VLMs may already have parametric knowledge of them, given Wikipedia’s prominence as a pretraining source (Penedo et al. [29]). However, even if a model has knowledge of a concept, it might fail to access it through visual inputs. For example, a VLM may learn from text data that the nadaswaram is a wind instrument with a wooden body and a flaring bell, yet fail to identify one from an image. Following recent work surfacing gaps between text-based and image-based performance on factual recall tasks ([3, 8]), we investigate whether a similar disconnect underlies failures on cultural concepts.

To establish if a model has a baseline parametric knowledge of a concept, we evaluate Open-Ended Description: given only a name, the model is asked to provide a short definition. Text-based knowledge is further probed through the Encyclopedic QA and Attribute QA pairs in CulturalBind. On the image-based side, Recognition QA tests whether VLMs can identify the concepts from images, and Visually-Grounded Encyclopedic QA additionally requires composing recognition with factual recall.

Experimental Setup

We evaluate open-weights models from three families: InternVL3.5 (Wang et al. (2025)), Qwen3-VL (Bai et al. [4]), and Gemma 3 (GemmaTeam et al. [10]) with sizes between 4B and 38B parameters. Additionally, we report the performance of GPT-5.2 as an upper-bound closed-source model. All models are evaluated using greedy decoding to ensure deterministic, reproducible outputs. Responses are evaluated using Qwen3-32B (Yang et al. (2025)) as the LLM judge against reference answers ensuring future reproducibility. Open-Ended Description responses are scored with a binary rubric, and all QA tasks are evaluated using a graded rubric adapted from Mañas et al. [22]. Following best practices (Bavaresco et al. [5]), we validate judge reliability against human scores, finding strong agreement (average QWK = 0.86). Further details provided in the Section B.

To assess the coupling between textual knowledge and visual access, we divide concepts into Known and Unknown groups based on their Open-Ended Description scores, and compare Visual Recognition accuracy across groups. We measure the correlation between the binary Description and graded Visual Recognition scores using point-biserial correlation (rpbr_{pb}).

Results

Table 3 presents text-based and image-based performance across evaluated models. Models achieve the highest performance on Open-Ended Description, with several models reaching scores above 70%. This suggests that models possess substantive basic knowledge of the examined cultural concepts. Performance on Encyclopedic and Attribute QA drops considerably, indicating that background knowledge coverage is more limited. Models score approximately 7 points higher on Attribute QA than Encyclopedic QA across smaller models, likely because salient perceptual properties such as shape, color, and material are more broadly represented in pretraining text than specific factual knowledge. This advantage narrows for stronger models, as Encyclopedic QA improves.

ModelOE DescriptionEncyclopedicAttributeRecognitionVG Encyclopedic
InternVL3.5 (4B)35.724.831.618.615.5
InternVL3.5 (8B)46.230.037.218.716.4
InternVL3.5 (14B)54.033.142.319.817.3
InternVL3.5 (38B)60.938.947.325.720.1
Qwen3-VL (4B)56.232.439.534.623.9
Qwen3-VL (8B)67.739.346.734.924.3
Qwen3-VL (32B)77.447.155.736.328.1
Gemma 3 (4B)54.231.137.331.921.2
Gemma 3 (12B)73.843.150.638.226.8
Gemma 3 (27B)82.852.256.843.133.7
GPT 5.290.567.769.344.640.2

Table 3. Cross-modal knowledge evaluation. Text-based measures (Open-Ended Description, Encyclopedic QA, Attribute QA) probe parametric cultural knowledge via text, while image-based measures (Recognition, Visually-Grounded Encyclopedic QA) test whether knowledge about the same concepts is accessible through visual input.

All models struggle to identify the same concepts from images, with recognition scores consistently lagging behind text-based performance by a large margin. Within model families, scaling improves text-based performance more than visual recognition: Gemma 3 4B→27B gains approximately 29 points on Description but only 11 on Recognition, while for Qwen3-VL 4B→32B, the corresponding gains are 21 and fewer than 2 points. This suggests the modality gap is not merely a capacity limitation that larger models overcome.

Across models, the drop from text-only Encyclopedic QA to Visually-Grounded Encyclopedic QA ranges from approximately 8 to 28 points, with the largest gap observed for GPT-5.2 despite its overall strength. This indicates that visual grounding is a bottleneck: models may possess the relevant background knowledge but cannot access it from visual referents. This aligns with Cohen et al. [8]’s findings on knowledge extraction gaps in VLMs. Results vary substantially across regions, while domain-level variance remains limited across all dimensions, as shown in Section D.1.

Table 4 breaks down the Recognition accuracy grouped according to their Description scores. Models consistently achieve higher recognition accuracy for known concepts with differences of up to 26.5 percentage points, and the correlation of

ModelKnownUnknownΔ\Deltarpb∗r_{pb}^{*}
InternVL3.5 (4B)25.810.815.00.267
InternVL3.5 (8B)22.910.512.40.221
InternVL3.5 (14B)22.310.611.70.215
InternVL3.5 (38B)29.511.917.60.265
Qwen3-VL (4B)42.415.926.50.374
Qwen3-VL (8B)37.216.221.10.286
Qwen3-VL (32B)35.515.220.30.241
Gemma 3 (4B)37.916.621.30.314
Gemma 3 (12B)37.917.220.70.256
Gemma 3 (27B)39.824.015.80.158
GPT-5.241.018.822.20.177

Table 4. Recognition accuracy for correctly versus incorrectly recalled concepts. Known: model accurately described the concept; Unknown: model failed to describe it. Correlation (rpbr_{pb}) indicates relationship strength. ∗p<.01^{*}p < .01.

Description and Recognition scores are statistically significant across all models. However, correlations remain weak (rpb=.16r_{pb} = .16–.37), indicating that textual recall and visual recognition are largely decoupled. A model’s ability to generate a textual description for a cultural concept does not serve as a reliable predictor of its ability to identify it in an image. We note again that the correlation does not strengthen with scale. Gemma 3-27B (rpb=.158r_{pb} = .158) and GPT-5.2 (rpb=.177r_{pb} = .177) show the weakest associations despite being among the strongest models overall. These results suggest that current multimodal pretraining does not encourage cross-modal consistency, even as general text and visual capabilities improve.

Knowledge Editing Evaluation

Experimental Setup

To address the identified recognition gap, we apply representative knowledge editing methods alongside finetuning baselines. We select Qwen3-VL (4B, 8B) and Gemma 3 (12B) as base models, given their stronger cross-modal performance. As baselines, we apply finetuning of the module which connects the visual encoder to the language model (FT-V), and the final language model layer (FT-T) using standard cross-entropy on response tokens, and Direct Preference Optimization (DPO) (Rafailov et al., [31]) using the model’s incorrect pre-edit predictions as negative samples.

We evaluate four knowledge editing methods: IKE (Zheng et al., 2023), an in-context editing approach that retrieves demonstration examples at inference time without modifying model parameters; MEND ([24]), a meta-learning approach that trains a hypernetwork to predict parameter updates for new edits; GRACE ([12]), a memory-based approach that stores edited representations in a learned codebook; and WISE (Wang et al., 2024a), a dual-memory approach that routes inputs between original and edited parameter sets. For WISE, we implement two routing variants: text-based locality following the original formulation (WISE-T), and image-based locality (WISE-V), which uses visual representations for routing{su}4.

We consider two editing scenarios. Single edits evaluate whether methods can fix individual recognition failures, while sequential edits are more practical editing use cases that evaluate whether a model can acquire new concepts over time. Sequential edits are applied cumulatively in a random order, shared across all methods to ensure comparability. IKE and MEND are excluded from sequential evaluation: IKE does not modify model parameters and has no mechanism for accumulating edits, while MEND’s hypernetwork is trained against the model’s initial weights, and, as shown by Lin et al. [20], its predictions become unreliable under sequential application. After editing NN concepts sequentially, we report average metrics across all NN edited concepts. Results are reported after N={10,50,100,150,200,250}N = \{10, 50, 100, 150, 200, 250\} edits. Experiments are repeated for five orders.

Edit sets are derived from each model’s recognition failures and therefore cover different concept subsets per model. These sets largely overlap (90.3–94.3% pairwise IoU), but we base our conclusions on within-model comparisons across methods rather than comparing a method across models. Additionally, as each method targets a different component, parameter budgets vary across methods and models (Table 12). Results should, therefore, be read as comparisons between editing strategies under their standard configurations, rather than between updates of equal capacity.

MethodEdited ComponentModified ParamsAdded ParamsAuxiliary Component
Qwen3-VL (4B), 36 LM layers
FT-VVL connector27M0—
FT-TWdownW_{\mathrm{down}}, layer 3625M0—
DPOWdownW_{\mathrm{down}}, layer 3625M0—
IKE—00SentenceBERT
MENDFFN, layer 3675M0Hypernet. (189M)
GRACEWdownW_{\mathrm{down}} out, layer 280key: 9,728 value: 2,560—
WISEWdownW_{\mathrm{down}}, layer 22025M—
Qwen3-VL (8B), 36 LM layers
FT-VVL connector40M0—
FT-TWdownW_{\mathrm{down}}, layer 3650M0—
DPOWdownW_{\mathrm{down}}, layer 3650M0—
IKE—00SentenceBERT
MENDFFN, layer 36150M0Hypernet. (252M)
GRACEWdownW_{\mathrm{down}} out, layer 280key: 12,288 value: 4,096—
WISEWdownW_{\mathrm{down}}, layer 26050M—
Gemma 3 (12B), 48 LM layers
FT-VVL connector4.4M0—
FT-TWdownW_{\mathrm{down}}, layer 4859M0—
DPOWdownW_{\mathrm{down}}, layer 4859M0—
IKE—00SentenceBERT
MENDFFN, layer 48177M0Hypernet. (300M)
GRACEWdownW_{\mathrm{down}} out, layer 310key: 15,360 value: 3,840—
WISEWdownW_{\mathrm{down}}, layer 44059M—

Table 12. Configuration of editing methods per model. Modified Params count model parameters updated in place; Added Params count new parameters. IKE retrieves k=32k = 32 in-context examples per edit, embedded with all-MiniLM-L6-v2 SentenceBERT [32]. MEND’s hypernetwork is trained before editing. GRACE stores a key-value entry per edit: when the key (the FFN intermediate state) is matched, the learned value replaces the WdownW_{\mathrm{down}} output. WISE adds a copy of WdownW_{\mathrm{down}}, updated sparsely per edit (20% at random).

Results

Single Edits

As shown in Table 5, all methods achieve successful single edits with high reliability, confirming that fixing individual recognition errors is feasible across architectures. However, reliability does not guarantee generalization: methods consistently trade off generality and locality, with this tension most pronounced for image-based metrics. Among post-training methods, finetuning the visual connector (FT-V) negatively impacts both image generality and image locality, suggesting that direct updates to the visual connector disrupt cross-modal alignment rather than supporting targeted concept acquisition. Finetuning the language model (FT-T) improves image generality substantially, and DPO yields comparable results.

ModelMethodReliabilityT-GenI-GenT-LocI-LocT-PortI-PortAvg
Qwen3-VL (4B) N=871FT-V98.579.725.193.077.715.716.058.0
FT-T100.079.863.475.784.214.014.161.6
DPO99.878.962.875.484.314.014.261.3
IKE99.798.999.823.431.715.817.455.2
MEND92.388.185.986.182.312.712.565.7
GRACE99.52.44.693.097.415.715.046.8
WISE-T100.097.798.193.038.47.28.063.2
WISE-V83.885.579.692.972.912.710.562.6
Qwen3-VL (8B) N=864FT-V98.779.822.693.678.221.320.659.3
FT-T100.084.570.379.884.419.215.664.8
DPO100.084.770.179.183.919.115.464.6
IKE96.896.496.431.629.920.723.256.4
MEND93.189.285.989.680.817.813.867.2
GRACE100.02.06.493.699.221.317.048.5
WISE-T100.097.096.193.557.417.314.267.9
WISE-V86.686.578.293.679.819.315.265.6
Gemma 3 (12B) N=796FT-V99.487.822.596.773.222.020.760.3
FT-T100.094.162.491.382.720.717.466.9
DPO100.094.563.491.682.620.617.567.2
IKE99.499.199.430.526.521.228.157.8
MEND86.682.062.793.280.220.615.963.0
GRACE98.72.99.996.791.222.019.448.7
WISE-T99.597.688.770.144.012.17.659.9
WISE-V99.497.989.780.451.113.78.262.9

Table 5. Method performance on single edits to correct N model errors on cultural concept recognition.

Comparing editing methods, MEND achieves the most consistent performance across models. WISE also achieves competitive performance, though with a consistent weakness in image locality. WISE-T, which uses text-based locality samples to learn routing, achieves stronger text generality and text locality, while WISE-V, which uses image-based locality samples, recovers image locality at the cost of slightly lower generality. Together, MEND and WISE represent the most viable approaches for single-concept editing, though neither achieves strong portability.

IKE and GRACE demonstrate opposite failure modes. IKE achieves near-perfect generality by retrieving demonstration examples at inference time, but this comes at a severe cost to locality. This inability to leverage in-context text-based demonstration is in line with the findings of Nikandrou et al. [26]. GRACE is at the opposite extreme with minimal generality and near-perfect locality. This is a consequence of its activation-matching mechanism, as stored edits are retrieved only when input activations fall within a narrow neighborhood, preventing generalization across instances.

Text and image portability are uniformly low across all methods and models, ranging from 7 to 28%, and do not meaningfully differentiate method families. Even when edits are successful on other measures, the new associations between images and concept labels do not propagate to the model’s broader parametric knowledge. To better understand model behavior, we examine randomly sampled outputs before and after editing. We classify each outcome as corrected (prediction changed

from incorrect to correct), damaged (prediction changed from correct to incorrect), or unchanged (prediction matched the pre-edit answer, regardless of correctness). Table 6 reports the distribution of these outcomes. The overwhelming majority of questions fall in the unchanged category, showing that editing does not meaningfully alter model behavior on portability. Only 12-23% of questions show any change and among those, corrected and damaged cases occur at comparable rates, yielding near-zero net improvement. These suggest that the low portability scores are not because of errors introduced by editing but reflect pre-existing gaps consistent with the results in § 5.

ModelPortabilityCorrectedDamagedUnchanged
Qwen3-VL (4B)Text4861367
Qwen3-VL (4B)Image2329396
Qwen3-VL (8B)Text4728398
Qwen3-VL (8B)Image3521366
Gemma 3 (12B)Text4334393
Gemma 3 (12B)Image4025370

Table 6. Portability outcomes before and after editing.

One possible explanation for the low image portability comes from mechanistic work on factual recall from visual inputs ([8, 41]): even when recognition succeeds, recall can fail when visual entity information becomes available at deeper model layers than those supporting factual recall. Most examined methods edit the second half of the language model (Table 12), so an edit may fix recognition while leaving the concept inaccessible at the depth of factual recall. FT-V, the only method that edits a component before the language model, improves image portability over FT-T only slightly (2–5 points).5

Sequential Edits

Figure 2 shows average accuracy for sequential editing across methods and models. All methods exhibit a decline as the number of edits increases, with performance dropping sharply even at N=50N=50. Between N=50N=50 and N=200N=200, methods cluster within a narrow accuracy range, with no single approach maintaining a consistent advantage. However, at N=250N=250, FT-T and DPO collapse to near-zero for both Qwen3-VL variants, reflecting the compounding effect of sequential finetuning of the last language model layer. GRACE and WISE-T appear more stable at scale, but per-metric results (Section D.2) show that this reflects preserved locality rather than editing success, as reliability and generality decline for both methods.

Figure 2. Average accuracy of editing metrics as a function of the number of sequential edits.

Forgetting vs. Retention To separate forgetting of earlier edits from failure to acquire edits in the first place, we evaluate the final Qwen3-VL (4B) checkpoint and compare performance on the earliest and last 25% of edits (Table 7). For parametric methods (FT-V, FT-T, DPO), recent edits score considerably higher than early ones, indicating forgetting. WISE shows no such recency effect: performance is similarly low on early and recent edits, including reliability. Its degradation therefore does not stem from forgetting, but is consistent with interference among accumulated edits.

MethodReliabilityT-GenI-Gen
FT-V8.8→53.68.8 \to53.66.7→43.26.7 \to43.22.7→5.62.7 \to5.6
FT-T19.2→54.419.2 \to54.416.0→39.216.0 \to39.210.4→35.210.4 \to35.2
DPO16.8→56.816.8 \to56.816.8→44.016.8 \to44.015.3→34.715.3 \to34.7
WISE-T29.6→24.829.6 \to24.828.0→26.028.0 \to26.020.8→25.620.8 \to25.6
WISE-V29.6→31.229.6 \to31.232.0→28.732.0 \to28.726.4→28.026.4 \to28.0

Table 7. Accuracy of Qwen3-VL (4B) after sequential editing, on the earliest →\to last 25% of edits.

Conclusion

We introduce CulturalBind, a multidimensional benchmark for assessing textual and visual knowledge gaps of cultural concepts. Our cross-modal evaluation reveals that VLMs demonstrate stronger parametric knowledge in text-only settings than when identifying the same concepts from images, a gap that widens for visually-grounded QA, which requires factual knowledge recall through visual referents. Additionally, even when conditioning on concepts a model can describe, visual recognition remains weak, indicating that textual and visual knowledge are largely decoupled.

We then use CulturalBind as an editing benchmark targeting VLM recognition failures. While editing methods can reliably correct individual recognition errors without disrupting unrelated model knowledge, portability remains uniformly low. This indicates that edited associations do not propagate to the model’s broader parametric knowledge of the concept. Under sequential editing, performance degrades sharply: parametric methods forget earlier edits, while memory-based methods preserve locality but fail to reliably recall even recent edits. Overall, our findings point to cross-modal consistency as a key missing objective, both during pretraining and in post-hoc adaptation. We hope CulturalBind motivates future work on methods that consolidate multimodal knowledge rather than patch isolated associations, moving towards VLMs that are adaptable across the diversity of global cultural concepts.

Limitations

First, experiments are conducted exclusively in English. Inclusive cultural evaluation ultimately requires multilingual assessment, as the visual-linguistic binding gap measured in English may not reflect the experience of speakers from the concept’s origin language. At the same time, known performance disparities across languages in VLMs ([35] (Yue et al. 2025)) make it difficult to disentangle gaps attributable to cultural concept coverage from those attributable to language-level representation more broadly. Additionally, we acknowledge the limitations of our region-based operationalization of culture, following the geographic metadata available in source datasets, and the uneven cultural coverage in CulturalBind with South American and African sub-regions underrepresented, reflecting known inequalities in Wikidata and Wikipedia coverage ([23]). Finally, the evaluated editing methods represent a selection of the current landscape that spans varying editing mechanisms and target layers across the visual projector and language model layers. However, these methods do not explicitly target cross-modal integration, which remains an important research direction for future work.

Ethical Considerations

This work diagnoses failures in the recognition of cultural concepts and the integration of knowledge in VLMs, with the goal of motivating more effective adaptation methods. The concepts evaluated represent cultural artefacts and practices, and absent or inaccurate model knowledge can disadvantage users from underrepresented groups [30]. Our evaluation anchors cultural knowledge to English Wikipedia, whose documentation of cultural knowledge is itself shaped by WEIRD biases. While performance on our benchmark should not be interpreted as a comprehensive measure of cultural competence, it allows us to separate between concepts a model genuinely lacks knowledge of and those it fails to access through visual referents. We hope this work contributes to advancing the cultural adaptability and multimodal grounding of VLMs.

Acknowledgements

This work was supported by the Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Program at the University of Edinburgh. In addition, the authors acknowledge the use of the HWU high-performance computing facility (DMOG) and associated support services in the completion of this work.

Dataset Details

Data Sources

CulturalGround ([27]) is a multilingual dataset of 22M culturally relevant VQA questions covering 42 countries and 39 languages, with cultural entities and images sourced from Wikidata. For the purpose of this study, it provides an initial pool of culturally relevant entities. However, the majority are named entities: as shown in Figure 3, the distribution of Wikidata instance of (P31) types is dominated by individual people (109K items), followed by geographic administrative units and infrastructure, none of which correspond to generalizable visual categories. We therefore filter at the group level, retaining only P31 types whose label denotes a concrete visual concept category (e.g., type of musical instrument) and excluding proper-noun categories such as human, business, or railway station. We further restrict to items that link to English Wikipedia pages, from which background knowledge can be extracted.

Figure 3. Distribution of the 15 largest Wikidata instance of (P31) groups among CulturalGround entities with English Wikipedia pages and multiple images. Before filtering, the majority of items belong to named-entity categories, which do not satisfy the concept criterion and are excluded at the group level.

Note that while CulturalGround provides a VQA dataset, we follow a different QA generation approach suited to our evaluation objectives. CulturalGround prioritises multilinguality and broad coverage of cultural entities, providing on average 1.5 encyclopedic questions per entity. Our QA generation instead targets concept-level knowledge evaluation, generating on average 4.86 encyclopedic and 4.88 visual attribute questions per concept.

CROPE ([26]) is a dataset for probing in-context adaptation to culture-specific concepts, building on top of existing multimodal reasoning work, MaRVL ([21]) and MCC ([19]). By definition, the provided concepts satisfy the concept criterion, but images cover only 340 concepts from five language origins (Indonesian, Swahili, Tamil, Turkish, and Mandarin Chinese). CROPE frames concept recognition as a binary question, pairing each target concept with an adversarially selected distractor, visually or functionally similar, to test fine-grained discrimination. We consider both positive and negative concepts, retrieving additional images for each distractor. The region mapping is completed by querying Wikidata for COUNTRY OF ORIGIN (P495), COUNTRY (P17), and LOCATION (P276).

Human Validation

All Openverse and CulturalGround images retained after automated filtering were manually inspected by the authors, removing 96 images and yielding the final 3,216. CROPE images had already been validated by native-speaker annotators ([26]). For the generated QA pairs, we sample 20 pairs per question category (200 in total, see Table 10) and annotate each pair on two criteria: (i) question clarity, whether the question is well-formed with a single correct answer, and (ii) answer faithfulness, whether the answer is supported by the source Wikipedia text.

As shown in Table 8, all audited QA pairs were grammatically correct and coherent. Question clarity errors mostly result from underspecified questions (e.g., “Where” questions that do not specify whether the answer should be a region, country, or state), while faithfulness errors mostly result from partial answers (e.g., the reference answer includes only “coconut” instead of “lentils and coconut”).

Question ClarityAnswer Faithfulness
Encyclopedic QA89.394.6
Visual Attribute QA90.596.5

Table 8. Manual audit of 200 generated QA pairs (%).

Dataset Construction Prompts

Below, we report the prompts for 1) filtering CulturalGround for concept selection (Figure 5), 2) filtering images (Figure 6), 3) extracting visual descriptions (Figure 7), 4) extracting encyclopedic question-answer pairs (Figure 8).

Prompt for filtering the CulturalGround entities

Figure 5. Prompt for filtering the CulturalGround entities.

Prompt for image filtering

Figure 6. Prompt for filtering the dataset images.

Prompt for Visual Description Extraction

Figure 7. Prompt for extracting visual descriptions from the Wikipedia page.

Prompt for Encyclopedic Knowledge QA Generation

Figure 8. LLM prompt to generate encyclopedic questions.

Additional Dataset Statistics

Table 9 reports the distribution of concepts and images across the six cultural domains in CulturalBind. Foods & Beverages is the largest domain, accounting for nearly 40% of concepts, while Sports is the smallest, reflecting the filtering criterion that excludes specific competitions and athletic events. The remaining domains are broadly balanced across concrete object categories (Artefacts,

Animals & Plants) and performative or ceremonial human activities (Music & Performing Arts, Traditions).

Table 10 reports the distribution of knowledge questions across encyclopedic and visual attribute categories. Among encyclopedic questions, Historical, Geographic, and Linguistic categories together account for over 60% of questions. Cultural and Functional questions provide complementary coverage of context and use. Following in terms of number, Composition and Comparison questions, probe knowledge that is particularly resistant to visual inference, such as non-visible ingredients or cross-cultural equivalents. Among visual attribute questions, Physical characteristics are the most frequent, followed by Contextual and Action questions, which capture relational and motion-related visual features. Together, the ten question categories span a diverse set of knowledge types.

Dataset Editing Examples

Figure 9 illustrates representative editing examples from CulturalBind across four cultural domains and regions, showing the edit sample and its associated generality, locality, and portability probes.

Figure 9. CulturalBind editing examples.

Concept-Level vs. Entity-Level Edits

We compute the SigLIP2 Tschannen et al., [38] cosine similarity between each edit image and its paired generality and locality images in CulturalBind and two entity-based editing benchmarks, VLKЕB Huang et al., [14] and the entity editing subset of MMKE-Bench Du et al., [9], using 2,000 samples each. As shown in Table 11, CulturalBind edit images are less visually similar to their generality images (mean 0.756) than those of VLKЕB (0.853; Mann-Whitney U test p<10−5p < 10^{-5}, Cliff’s δ=−0.48\delta= -0.48) and MMKE-Bench (0.845; p<10−5p < 10^{-5}, δ=−0.44\delta= -0.44), while remaining more similar than the locality images. This supports the hypothesis that generalizing across concept instances involves greater visual variation than identifying the same entity under different conditions.

GeneralityLocality
CulturalBind0.756±0.1240.756 \pm0.1240.548±0.0950.548 \pm0.095
VLKЕB0.853±0.0730.853 \pm0.0730.688±0.1130.688 \pm0.113
MMKE-Bench0.845±0.0770.845 \pm0.0770.425±0.0840.425 \pm0.084

Table 11. SigLIP2 cosine similarity (mean ±\pm sd) between edit images and their paired generality and locality images.

License

CulturalBind is released publicly upon publication under CC BY-SA 4.0. Images are sourced from Wikimedia Commons and Openverse under CC0, CC BY, and CC BY-SA licenses. Wikipedia text used for knowledge QA generation is licensed under CC BY-SA 4.0.

Task

Determine whether the image provides sufficient visual evidence to confidently answer the recognition question using the image alone. You must respond with either "keep" or "remove".

Decision Rule

Could a knowledgeable person correctly answer the question from the visual appearance alone, without relying on any text in the image?

Instructions

Keep if ALL the following criteria are met:

  • The image shows a clear example of the concept

  • The visual appearance is prototypical and visually distinctive enough to support confident recognition

  • The concept can be identified from visual features alone (no need to read any text)

Remove if ANY of the following criteria are met:

  • The concept cannot be confidently identified from visual features alone

  • The example is atypical, ambiguous, or heavily occluded

  • Recognition depends on reading text present in the image (labels, captions, logos, signage)

Output Format

Respond with a JSON object containing the following fields:

{"reason": "<short reasoning>", "decision": "keep" | "remove"}

Context

Concept: {concept}

Wikipedia Summary: {summary}

Question: {question}

Image: {image}

LLM-Judge Evaluation

Below, we report the prompts used for the evaluation of Open-Ended Concept Description (Figure 10), factual QA tasks (Encyclopedic, Attribute, and Visually-Grounded Encyclopedic QA, Figure 11), and Recognition QA (Figure 12).

Open-Ended Concept Description Evaluation Prompt

Task

You are a cultural knowledge expert. Given a question asking a model to describe a cultural concept, a reference Wikipedia summary, and the model’s prediction, determine if the model demonstrates knowledge of the concept.

Goal

Decide if the model is correctly identifying the SAME concept described in the reference Wikipedia summary.

Scoring Criteria

  • Score 1: The model’s prediction clearly refers to the same cultural concept as the Wikipedia summary. Minor omissions, factual errors, or additional true details are acceptable, as long as the prediction captures the key definition of the concept.

  • Score 0: The model fails to demonstrate knowledge of the concept. This includes:

    • Hallucinating a completely different definition.

    • Describing a different concept with the same name.

    • Refusing to answer or claiming not to know the concept.

    • Providing generic text that does not specifically describe the concept.

Output Format

Respond only with the following JSON structure and nothing else:

{ "explanation": "<brief explanation for the score>", "score": <0 | 1> }

Figure 10. Evaluation rubric for Open-Ended Description.

Factual QA Evaluation Prompt

Figure 11. Evaluation rubric for factual QA tasks.

Visual Recognition Evaluation Prompt

Figure 12. Evaluation rubric for visual recognition QA tasks, adapted from LAVE ([22]).

To validate the reliability of automated evaluation, we independently annotated N=150N=150 samples blind to the judge scores, stratified by score across all three evaluation dimensions: Open-ended Description (N=40N=40), Text-based Factual QA (60 samples), and Image-based Recognition QA (50 samples). We use Qwen3-32B (Yang et al., 2025) as the LLM judge, as it is openly available and results are therefore fully reproducible. We compute Quadratic Weighted Kappa (QWK) between our annotations and the judge scores, which penalizes larger ordinal discrepancies more heavily. Agreement is strong across all dimensions with an overall QWK of 0.856. These results support the use of Qwen3-32B as a reliable proxy for reference-based evaluation.

Implementation Details

For each model, we construct stratified train, development, and test splits at the concept level with a 55/10/35 ratio, ensuring that all images of a given concept appear in the same split. Edit sets are then derived per model by filtering each split for the concepts the model fails to recognize, and results are reported on the test split. Methods that require additional data draw it from the train split: MENΔ\Delta’s hypernetwork is trained on it, and IKE retrieves its in-context demonstrations from it.

We allow up to 30 updates per edit, with early stopping to prevent overfitting to the edit sample. We follow the default hyperparameters of each method and grid-search over the learning rate and, where applicable, the edit layer, selecting the configuration with the highest average accuracy on single edits on the development split. The selected configurations are reused unchanged in the sequential setting. For FT-V, FT-T, and DPO, we search over learning rates in {1,3,5,8}×{10−5,10−4}\{1, 3, 5, 8\} \times\{10^{-5}, 10^{-4}\}. For GRACE and WISE, which apply edits at a single intermediate layer, we additionally sweep over mid-to-late layers. Table 12 details the edited component and parameter counts for each method per model. Budgets vary with both the method and the model architecture. This is most visible for FT-V: Gemma 3 reduces image patches with 2D average pooling followed by a linear projection, whereas Qwen3-VL concatenates 2×22 \times2 patch groups along the feature dimension before a 2-layer MLP, giving its connectors six to nine times as many trainable parameters.

All editing evaluations are run on a single NVIDIA H100 80GB GPU, with a total computational budget of approximately 360 GPU hours.

Additional Results

Cross-Modal Evaluation Results

Figure 13 and Figure 14 present the breakdown of the cross-modal evaluation results per domain and per region. Across domains, variability is modest for text-based measures but more pronounced for image-based measures, where Foods & Beverages consistently scores lowest (mean 24.6% for Recognition QA), and Animal & Plant and Artefacts score highest (mean 41.4% and 46.6% respectively). Across regions, we observe consistent patterns. North America is a persistent outlier, with high performance across models and evaluation dimensions. Western Europe and East Asia are the next strongest regions, consistent with their higher representation in web-scale pretraining data. The gap relative to lower-resource regions, such as North Africa, Southeast Asia, and Eastern Europe, is substantial, but the individual weakest regions vary per model family.

Figure 13. Text-based cross-modal evaluation results per domain and per region.

Figure 14. Image-based cross-modal evaluation results per domain and per region.

Isolating Visual Recognition To verify that the drop from Encyclopedic to Visually-Grounded Encyclopedic QA is due to recognition rather than the image input acting as a distractor, we evaluate encyclopedic questions that keep the concept name alongside the image (Table 13). Adding the image to a question that names the concept increases accuracy (+0.4 to +2.1 points), but replacing the name with a referring expression results to a large drop, supporting that the bottleneck is visual recognition.

ModelName (no image)Name + imageRef. expr. + image
Qwen3-VL (4B)32.434.523.9
Qwen3-VL (8B)39.339.724.3
Gemma 3 (12B)43.145.226.8

Table 13. Encyclopedic QA accuracy when the concept is referenced by name without the image, by name with the image, or by a referring expression with the image (Visually-Grounded Encyclopedic QA).

Sequential Edits

As the aggregate view in Figure 2 conceals how each method trades off reliability, generality, and locality, we report per-metric results in Figure 15. Reliability and generality are the most impacted across all methods. Neither finetuning-based nor memory-based approaches sustain either metric beyond the first few edits. Locality, which requires maintaining unrelated model outputs, is better re-

Figure 15. Results per editing metric for sequential edits.

tained by some methods and varies with the edited component. Portability is omitted, as it remains uniformly low across methods and edit counts.

FT-T and DPO remain competitive in terms of reliability and generality. Their text locality declines steadily, potentially due to updates being applied to the final language model layer. FT-V shows the complementary pattern: editing the vision-language connector leaves text locality intact, as text-only inputs bypass the connector, while image locality is the lowest across methods.

Memory-based methods (GRACE and WISE) preserve higher locality across edits. GRACE preserves near-perfect locality but shows near-zero generality from the first edit onwards. Its flat average performance in Figure 2 is, therefore, an artifact of this trade-off rather than evidence of sustained editing success. The WISE variants perform similarly to direct finetuning on reliability and generality, but each better preserves the locality metric matching the modality used for routing: WISE-T maintains text locality above 85%, whereas WISE-V retains image locality more consistently. Note that image locality samples are selected based on their embedding similarity to the edit sample, making this a particularly challenging test of knowledge preservation.

Prompt for Encyclopedic Knowledge QA Generation

You are a specialized QA generator creating a dataset for closed-book knowledge testing.
## Task
Given a Wikipedia extract and visual descriptions, generate up to 5 QA pairs testing for background knowledge that is
NOT present in the visual description and cannot be inferred by looking at an image of the concept.
## Instructions

1. REFERENCE: All questions must name the concept explicitly.

2. CLARITY: Questions must be unambiguous, with a single correct answer. Avoid questions that could have
   multiple valid answers or require subjective interpretation.

3. NO VISUALS: If a person could see the answer in a photograph, do not ask the question.

4. SOURCE GROUNDING: Answers must be found within the provided ’EXTRACT’.

5. BREVITY: Answers must be a single word or a very short phrase (max 5 words).

6. CATEGORIZATION: Every question must map to one of the specific categories below.

## Question Categories

1. cultural association (ethnic groups, religions, traditions, festivals, cultural practices)

2. historical (dates, events, historical figures)

3. geographic (geographic or regional origins, locations)

4. function (purpose, use case, or role)

5. composition (ingredients, preparation methods, or components that are not visually identifiable)

6. linguistic (etymology, name meaning, language of origin)

7. comparison (equivalents or related items)

8. other (any other non-visual background knowledge)

## Output Format

Output only a valid JSON object:
"questions": [ "question": "...", "answer": "...", "category": "cultural association" | "historical" | "geographic" | "function"
| "composition" | "linguistic" | "comparison" | "other" ]

## Examples

{examples}

References

  1. [1]Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. 2024. A survey on data selection for language models. Trans. Mach. Learn. Res., 2024.arxiv.org/abs/2402.16827
  2. [2]David Anugraha, Patrick Amadeus Irawan, Anshul Singh, En-Shiun Annie Lee, and Genta Indra Winata. 2026. M4-RAG: a massive-scale multilingual multicultural multimodal rag. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23083–23094.arxiv.org/abs/2512.05959
  3. [3]Dhananjay Ashok, Ashutosh Chaubey, Hirona Jacqueline Arai, Jonathan May, and Jesse Thomason. 2025. Can VLMs recall factual associations from visual references? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15691–15708, Suzhou, China. Association for Computational Linguistics.
  4. [4]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, arXiv:2511.21631.arxiv.org/abs/2511.21631
  5. [5]Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzele, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 238–255, Vienna, Austria. Association for Computational Linguistics.
  6. [6]Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, Eu-nJeong Hwang, and Vered Shwartz. 2024. From local concepts to universals: Evaluating the multicultural understanding of vision-language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6763–6782, Miami, Florida, USA. Association for Computational Linguistics.
  7. [7]Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang, Huajun Chen, and Ningyu Zhang. 2023. Can we edit multimodal large language models? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13877–13888, Singapore. Association for Computational Linguistics.arxiv.org/abs/2310.08475
  8. [8]Ido Cohen, Daniela Gottesman, Mor Geva, and Raja Giryes. 2025. Performance gap in entity knowledge extraction across modalities in vision language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29095–29108, Vienna, Austria. Association for Computational Linguistics.
  9. [9]Yuntao Du, Kailin Jiang, Zhi Gao, Chenrui Shi, Zilong Zheng, Siyuan Qi, and Qing Li. 2025. MMKE-bench: A multimodal editing benchmark for diverse visual knowledge. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2502.19870
  10. [10]GemmaTeam, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Riviѐre, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Cason, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.
  11. [11]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  12. [12]Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems, volume 36, pages 47934–47959. Curran Associates, Inc.arxiv.org/abs/2211.11031
  13. [13]Sara Hooker. 2025. On the slow death of scaling. Available at SSRN 5877662.DOI
  14. [14]Han Huang, Haitian Zhong, Tao Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2024. VLKEB: A large vision-language model knowledge editing benchmark. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.arxiv.org/abs/2403.07350
  15. [15]George Ibrahim, Rita Ramos, and Yova Kementchedjhieva. 2025. CONCAP: Seeing beyond english with retrieval-augmented captioning. In CVPR 2025 Workshop Vision Language Models For All.arxiv.org/abs/2507.20411
  16. [16]Oana Ignat, Longju Bai, Joan C. Nwatu, and Rada Mihalcea. 2024. Annotations on a budget: Leveraging geo-data similarity to balance model performance and annotation cost. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1239–1259. ELRA and ICCL.arxiv.org/abs/2403.07687
  17. [17]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.DOI
  18. [18]Jiaqi Li, Miaozeng Du, Chuanyi Zhang, Yongrui Chen, Nan Hu, Guilin Qi, Haiyun Jiang, Siyuan Cheng, and Bozhong Tian. 2024. MIKE: A new benchmark for fine-grained multimodal entity knowledge editing. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5018–5029, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2402.14835
  19. [19]Zhi Li and Yin Zhang. 2023. Cultural concept adaptation on multimodal reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 262–276. Association for Computational Linguistics.DOI
  20. [20]Zihao Lin, Mohammad Beigi, Hongxuan Li, Yufan Zhou, Yuxiang Zhang, Qifan Wang, Wenpeng Yin, and Lifu Huang. 2024. Navigating the dual facets: A comprehensive evaluation of sequential memory editing in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13755–13772, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2402.11122
  21. [21]Fangyu Liu, Emanuele Bugliarello, EdoardoMaria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  22. [22]Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. 2024. Improving automatic vqa evaluation using large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4171–4179.arxiv.org/abs/2310.02567
  23. [23]Marc Miquel-Ribé and David Laniado. 2018. Wikipedia culture gap: Quantifying content imbalances across 40 language editions. Frontiers in Physics, 6.
  24. [24]Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022. Fast model editing at scale. In International Conference on Learning Representations.arxiv.org/abs/2110.11309
  25. [26]Malvina Nikandrou, Georgios Pantazopoulos, Nikolas Vitsakis, Ioannis Konstas, and Alessandro Suglia. 2025. CROPE: Evaluating in-context adaptation of vision and language models to culture-specific concepts. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7917–7936, Albuquerque, New Mexico. Association for Computational Linguistics.
  26. [27]Jean De Dieu Nyandwi, Yueqi Song, Simran Khanuja, and Graham Neubig. 2025. Grounding multilingual multimodal LLMs with cultural knowledge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 24187–24231, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2508.07414
  27. [28]ChaeHun Park, Yujin Baek, Jaeseok Kim, Yu-Jung Heo, Du-Seong Chang, and Jaegul Choo. 2025. Evaluating visual and cultural interpretation: The k-viscut benchmark with human-VLM collaboration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21960–21974, Vienna, Austria. Association for Computational Linguistics.
  28. [29]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.arxiv.org/abs/2406.17557
  29. [30]Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. 2022. Cultural incongruencies in artificial intelligence. arXiv preprint arXiv:2211.13069.arxiv.org/abs/2211.13069
  30. [31]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc.arxiv.org/abs/2305.18290
  31. [32]Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.arxiv.org/abs/1908.10084
  32. [33]Megan Richards, Polina Kirichenko, Diane Bouchacourt, and Mark Ibrahim. 2024. Does progress on object recognition benchmarks improve generalization on crowdsourced, global data? In The Twelfth International Conference on Learning Representations.
  33. [34]David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, and 57 others. 2024. CVQA: Culturally-diverse multilingual visual question answering benchmark. In Advances in Neural Information Processing Systems, volume 37, pages 11479–11505. Curran Associates, Inc.arxiv.org/abs/2406.05967
  34. [35]IsrafeI Salazar, Manuel Fernández Burda, Shaye­kh Islam, Arshia Soltani Mookhar, Shivalika Singh, Fabian Farestam, Angelika Romanou, Danylo Boiko, Dipika Khullar, Mike Zhang, Dominik Krzemiński, Jekaterina Novikova, Luisa Shimabucoro, Joseph Marvin Imperial, Rishabh Maheshwary, Sharad Duwal, Alfonso Amayuelas, Swati Rajwal, Jebish Purbey, and 26 others. 2026. Kaleidoscope: In-language exams for massively multilingual vision evaluation. In International Conference on Learning Representations, volume 2026, pages 81112–81164.arxiv.org/abs/2504.07072
  35. [36]Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley. 2017. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. In NeurIPS 2017 workshop: Machine Learning for the Developing World.arxiv.org/abs/1711.08536
  36. [37]Ashish V. Thapliyal, Jordi Pont Tuset, and Radu Chen, Xiand Soricut. 2022. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715–729. Association for Computational Linguistics.
  37. [38]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Abdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, and 1 others. 2025. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786.arxiv.org/abs/2502.14786
  38. [40]Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadgign Admetew, Yahya Hmaiti, Amandeep Kumar, Kartik Kukreja, Mykola Maslych, Wafa Al Ghallabi, Mihail Minkov Mihaylov, Chao Qin, Abdelrahman M. Shaker, Mike Zhang, Mahardika Krisna Ihsani, Amiel Gian Esplana, Monil Gokani, and 50 others. 2025. All languages matter: Evaluating lmms on culturally diverse 100 languages. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19565–19575.
  39. [41]Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, and Neel Nanda. 2025. Too late to recall: Explaining the two-hop problem in multimodal knowledge retrieval. In Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 48328–48355. Curran Associates, Inc.arxiv.org/abs/2512.03276

Paper details

vlnc:W_eemnwu8x
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents