WorksComputer Science
VLTest: Finding Joint Image-Text Failures in Vision-Language Models
Abstract
Vision-language models (VLMs) are increasingly embedded in software systems. Testing them raises two problems. Mutating either input can change the expected answer, so an output change is not by itself a failure. Some failures appear only when both inputs are mutated; attributing them to the combination requires testing each mutation alone. We present VLTest, a black-box VLM testing approach. It generates image variants by editing VQGAN latent codes and text variants with a rewrite-expand-prune pipeline, filters variants likely to change the expected answer, and tests each variant alone before testing combinations, so every joint failure has four matched cells. A VLM judge confirms a failure only after checking that the input keeps the expected answer. On three tasks with eight VLMs, including two commercial API models, the oracle confirms 81.7% of the candidates, with 88.8% to 94.6% precision against human labels, and four-cell confirmation rejects 22.5% of joint candidates. Under the same budget, VLTest's generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and its joint tests find 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.
Introduction
Vision–language models (VLMs) [36, 50, 78] take an image and a text as input and produce a decision or a natural-language answer. They power image–text matching [18], visual reasoning [22], visual question answering [41], cross-modal retrieval [68], and multimodal search [4], and they are increasingly integrated into software systems [1]. Their predictions can change under small variations of either input [6, 7, 58], which makes their behavioral reliability a software quality concern.
Software engineering research has developed effective techniques for testing deep neural networks (DNNs). Seed-centric generators explore the neighborhood of an input under a budget [13, 14, 15, 21, 32, 48, 67], latent-space generators keep the generated inputs realistic [12, 29, 70], and metamorphic relations provide test oracles when the expected output of a generated input is unknown [8, 57]. Applying these techniques to VLMs, however, raises two challenges.
Challenge 1: output changes are not failures. Seed-centric testing usually relies on an invariance relation: if a mutation preserves the content that determines the answer, the answer should not change. In a VLM, the expected answer depends on how the image and the text relate. A paraphrase can change what a question asks, a visual edit can change whether a statement is true, and two edits that are harmless on their own can change the answer together. An image–caption pair can remain well aligned while its correct match label changes. Counting output changes, as robustness evaluations of VLMs commonly do [7], therefore mixes real failures with correct reactions to changed inputs. Studies of label preservation for generated test inputs focus on unimodal classifiers [43, 44]; for VLMs, the expected answer also depends on the relation between the two inputs.
Challenge 2: joint failures need attribution. Some failures occur only when both inputs are mutated. To show that such a failure is caused by the combination, a tester must also run the image-only and the text-only variants, check that the model handles both correctly, and check that all inputs share the same expected answer. We call these four inputs the four cells of a joint test: the seed, the image-only input, the text-only input, and the joint input. The cells should differ only by the mutations they contain; an image mutation made in the latent space of a generative model also carries reconstruction artifacts, which the text-only input must then share. Even with matched controls, the effects of two mutations can add up to a failure without any interaction between the modalities [23]. Existing approaches that mutate both modalities, such as metamorphic testing of VQA systems [72] and multimodal adversarial attacks [40, 73, 79], do not pair the final joint input with matched single-modality controls whose expected answers are confirmed by an oracle. Without such controls, a higher failure count under joint mutation may simply reflect that two sources of perturbation were used instead of one.
To address these challenges, we propose VLTest, a black-box testing approach for VLMs. The key idea of VLTest is to treat a joint input and its two single-modality counterparts as one test, and to report a failure only after its expected answer has been confirmed by an oracle. VLTest has three components. First, two controlled generators produce visual variants, by replacing dominant codebook
Authors’ Contact Information: Bin Duan, The University of Queensland, Brisbane, Australia, [email protected]; Jintao Lin, The University of Queensland, Brisbane, Australia, [email protected]; Guowei Yang, The University of Queensland, Brisbane, Australia, [email protected].
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1557-7352/2026/10-ART https://doi.org/10.1145/nnnnnnn.nnnnnnn entries in the discrete latent space of a pretrained VQGAN [17], and textual variants, through a rewrite–expand–prune pipeline. Both emit deduplicated variants in a fixed order. Second, relation filters remove variants that are likely to change the expected answer, using bidirectional entailment for text, an image similarity floor, and task-specific image–text alignment rules. Third, a two-phase testing procedure admits only seeds that the model answers correctly, first tests each variant on its own, and then tests combinations of these variants. All four cells are built on the reconstructed seed image, so that each cell differs from the seed only by the mutations it contains. A feedback-directed pairing strategy uses the single-modality results to test first the combinations that single-modality testing has not already broken.
VLTest confirms failures with an automated oracle. A VLM judge receives the seed and its answer as a reference and first decides, without seeing the model output, whether a generated input keeps that answer; only then does it decide whether the output is wrong. A failure is reported only if both checks succeed, and a joint failure only if they succeed for all four cells. Human labels on a stratified sample of the reported failures measure the precision of the oracle.
We evaluate VLTest on image–text matching (MSCOCO), visual reasoning (NLVR2), and visual question answering (VQA v2) with eight VLMs: four task-specific models with six fine-tuned checkpoints, Qwen3-VL at two scales, and two commercial API models. We compare the generators of VLTest with SINVAD [29], adapted to the VQGAN latent, and TextFooler [28], and its joint testing with five baselines: random pairing with and without filtering, random pairing that first uses the combinations whose single-modality inputs are answered correctly, stable-first pairing of the variants found by SINVAD and TextFooler, and the applicable relations of MetaRA [72]. The oracle confirms 81.7% of the candidates of VLTest (SF) and reaches a precision of 80.0% to 94.6% against human labels across the compared methods in the primary experiment, and of 88.8% to 94.6% on the failures that VLTest (SF) reports; four-cell confirmation rejects 22.5% of the joint candidates, 19.7% through the joint input alone and a further 2.8% through the single-modality controls. Under the same budget, the generators of VLTest find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and VLTest finds 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the stable-first pairing of their variants and 2.8 times as many as the applicable relations of MetaRA, which exhaust their combinations after about half of the joint budget. Under an additive margin model, 16.5% of the attributed joint failures with output scores are interactions between the modalities that the sum of the two single-modality effects does not predict. On Qwen3-VL, GPT-4o mini, and Gemini 2.5 Flash, VLTest finds 112 to 250 attributed joint failures per 1,000 joint calls.
The main contributions of this paper are as follows.
We formulate VLM testing with oracle-confirmed expected answers and introduce four-cell evidence, built on a common reconstructed reference, for attributing a failure to the combination of an image mutation and a text mutation.
We propose VLTest, which combines controlled visual and textual generators, relation filters, and two-phase testing that reuses the single-modality tests as the matched controls of every joint test and as feedback for pairing.
We design an automated test oracle that, for every cell, checks the expected answer against the seed with a VLM judge before it assesses the model output, and we validate the oracle against human labels.
We provide a replication artifact [65] for VLTest.
Background and Related Work
Background
VLMs encode an image and a text into a shared or fused representation and decode a task output [42, 78]. Architectures range from single-stream fusion encoders such as ViLT [30], to encoders pretrained at scale such as BLIP [37] and BEiT-3 [69], to generative models that couple a vision encoder with a language model, such as PaliGemma [5] and recent open-weight families [3, 35, 49, 82]. We consider three tasks with different output types: image–text matching (ITM, binary), visual reasoning over a statement and an image pair (VR, binary), and visual question answering (VQA, short free-form answers with several acceptable answers).
Related Work
Table 1 compares VLTest with representative approaches.
Test Input Generation for DNNs. Seed-centric test generation explores the neighborhood of an input to expose behaviors that fixed datasets miss [61]. Coverage-guided methods perturb inputs in pixel space [21, 32, 48, 67]. Generative methods search a learned latent space so that the generated inputs remain realistic: SINVAD [29] searches the latent space of a VAE, CIT4DNN [12] applies combinatorial coverage to a discretized latent space, Mimicry [70] targets decision boundaries, and DeepHypen [81] explores a feature map. Recent work benchmarks generative models for test input generation [43] and validates generated inputs with a human-aligned learned validator [44], showing that validity and label preservation vary across generators. Riccio and Tonella [55] study when and why test generators for DL produce invalid inputs, and Dola et al. [11] use a generative model to discard generated inputs that lie outside the training distribution. These approaches target unimodal classifiers. VLTest extends seed-centric generation to image–text inputs, whose expected answer depends on both modalities, and confirms that answer with an oracle for every reported failure.
Evaluation of VLMs. Benchmarks remain the main way to evaluate VLMs. Chen et al. [7] measure robustness under visual and textual corruptions, VHELM [33] evaluates VLMs along multiple aspects, NaturalBench [34] collects natural adversarial samples that cannot be solved from one modality alone, Ramananzahdevgiri et al. [51] show that VLMs fail at simple visual tasks, and compositional benchmarks probe attribute binding and word order [26, 62]. MultiMon [63] mines failure patterns from inputs on which a text encoder and a VLM erroneously agree. These works evaluate models on fixed or curated inputs. VLTest complements them by generating new inputs from each seed.
Metamorphic and Search-Based Testing of Multimodal Systems. Metamorphic testing [8, 57] checks relations between the outputs of related inputs when the expected output is unknown. MetaVQA [76] synthesizes counting and existence questions from detected objects and checks relations under question transformations, such as splitting or negating a question, and under image transformations, such as inserting or removing objects; each relation transforms one modality. MetaRA [72] applies metamorphic relations to the image–question inputs of MLLM-based VQA systems and reports the failure patterns they reveal. For image captioning, MetaIC [75] inserts objects into images and ROME [74] removes them, and both check how the captions of the transformed images should relate. Metamorphic relations have also been defined for vision–language–action models in robotic manipulation [64], and search-based testing has been applied to VLMs for in-car scene understanding [59]. VLTest also builds on an invariance relation. It differs in that it checks the precondition of the relation, namely that the expected answer is preserved, for every reported failure, and it pairs each joint failure with matched single-modality controls whose answers are confirmed. We compare with the relations of MetaRA that apply to our setting (§ 5.3).
Adversarial Attacks on VLMs. Adversarial attacks optimize perturbations to induce mispredictions under a threat model [56, 81]. Co-Attack [79] perturbs the text first and then the image, guided by the adversarial text, and SGA [40] alternates guidance between the two modalities to improve transferability. VLATTACK [73] uses a pretrained model to attack black-box models fine-tuned from it, with single-modality image and text attacks followed by an iterative cross-search over both modalities. Attacks maximize attack success under a perturbation bound. VLTest pursues a different goal: it generates controlled test inputs and reports failures with oracle-confirmed expected answers and attribution evidence.
Text Perturbation and NLP Testing. Text adversarial methods such as DeepWordBug [19], SCPN [27], PWWS [53], and TextFooler [28], and behavioral test suites such as CheckList [54], perturb text for robustness and capability testing. They supply the lexical operators that our text pipeline builds on, and TextFooler is the text-side baseline in our evaluation.
Combinatorial Interaction Testing. Combinatorial interaction testing (CIT) [31, 46] builds on the observation that many faults are triggered by interactions among a few input factors. A joint failure is such an interaction between two factors, the image and the text. In classical CIT, factors have small predefined domains and covering arrays enumerate their combinations. In VLTest, the levels of each factor are generated variants, so VLTest selects which combinations to test and attributes each failure with four-cell evidence.
Problem Formulation
Inputs and Expected Answers
A VLM maps an image–text pair to an output . A task-specific normalization maps outputs to comparable answers: binary labels for ITM and VR, and canonical short answers for VQA following the official evaluation script [2, 66]. A seed is a dataset input. A VR input also contains a second image, which stays fixed in all inputs derived from a seed and is omitted from the notation.
The reference answers of a seed come from the dataset: the label for ITM and VR, and for VQA the answers that receive full credit under the official metric, that is, answers given by at least four of the ten annotators. The expected answer of an input is the set of answers that are correct for . It can contain more than , such as a synonym of a reference answer, and for a generated input it is unknown until the oracle establishes it. is correct on if , and the output of is compatible if . A compatible output is correct whenever , and an incompatible output can still be correct.
Test Oracle
VLTest uses the invariance relation
Its precondition depends on how the image and the text relate and cannot be read off the mutation. VLTest therefore separates the check done during testing, which compares outputs only with the reference answers, from the confirmation done by an automated oracle (§ 4.6), which establishes the expected answer.
Definition 3.1 (Candidate). Let be an admitted seed (§ 3.3). A generated input is a candidate if its output is not compatible, .
Definition 3.2 (Confirmed failure). A candidate is a confirmed failure if the oracle establishes that is a valid input with and that .
Definition 3.3 (Attributed joint failure). A joint input of an admitted seed is an attributed joint failure if is a confirmed failure and the oracle establishes that both single-modality cells and are valid inputs with expected answer on which is correct.
A candidate need not be a failure: its expected answer may have changed, the input may be invalid, or its output may be a correct answer outside . Two single-modality variants that each preserve the answer do not guarantee that their combination does, so the oracle checks the joint cell on its own. A joint input that is a confirmed failure is not necessarily an attributed joint failure: the latter additionally requires both single-modality controls, which attribute the failure to the combination of the two mutations. We refer to the former as a joint input confirmed on its own.
Reference Input
Every visual variant passes through the VQGAN encoder and decoder, and reconstruction alone can change details such as colors, counts, or text in the image. VLTest therefore builds all cells on the reconstructed seed : the seed cell is , a text-only input is , and a joint-input cell contains a decoded image variant. Each cell then differs from the seed cell only by the mutations it contains. A seed is admitted for a model if the oracle confirms that reconstruction keeps its expected answer and the output of the model on is compatible. Each model is also called once on the raw seed , so that we can report how often reconstruction changes its output; this call is not a test.
Interaction and Additive Failures
An attributed joint failure shows that neither mutation causes the failure alone. The two effects may still simply add up: each mutation moves the model toward a wrong answer, and only together do they cross the decision boundary. For models that expose output scores, we separate this case from an interaction between the mutations.
| Approach | Test inputs | Mutated | Expected answer of generated inputs | Matched four-cell confirmation |
| Chen et al. [7] | predefined corruptions of dataset inputs | I or T | assumed unchanged | n/a |
| NaturalBench [34] | curated natural adversarial pairs | – | human-annotated | n/a |
| MultiMon [63] | mined from existing corpora | – | summarized post hoc | n/a |
| MetaVQA [76] | relation-driven transformations of one modality | I or T | defined by the relation | n/a |
| MetaRA [72] | relation-driven transformations | I and T | defined by the relation | no |
| Co-Attack [79], SGA [40] | white-box optimization with cross-modal guidance | I+T | assumed under perturbation bounds | no |
| VLATTACK [73] | per-example optimization with cross-search | I, T, I+T | assumed under perturbation bounds | no |
| SINVAD [29], CIT4DNN [12] | latent-space generation | I (unimodal models) | not checked | n/a |
| DeepNaqqal [44] | validator for generated inputs | I (unimodal models) | not addressed; input validity predicted by a learned validator | n/a |
| VLTest | seed-centric generation; single-modality tests, then joint tests | I, T, I+T | filtered, then confirmed by an automated oracle | yes |
Table 1. Comparison of VLTest with representative approaches (I = image, T = text). In the last column, n/a means that the approach does not test joint inputs, and no means that it does, without matched single-modality controls.
Let be the answer of on the joint input , and let be the margin of the best reference answer over on a cell . Adding the effects of the two mutations predicts the margin of the joint input as
An attributed joint failure is an interaction failure if , that is, if the additive prediction still prefers a reference answer, and an additive failure otherwise. The difference is the interaction contrast of the design that the four cells form. Hessel and Lee [23] use a related additive projection to test whether a model uses cross-modal interactions on a dataset; we apply the contrast to each attributed joint failure. The analysis uses the scores only for evaluation; VLTest itself uses normalized outputs.
Approach
VLTest treats the model under test as a black box and accesses only its inputs and normalized outputs. Its generators and filters use pretrained, off-the-shelf models. Figure 1 gives an overview of its four stages. First, the visual and textual generators (Sections 4.1 and 4.2) emit deduplicated variants of the seed in fixed orders. Second, the relation filters (Section 4.3) remove variants that are likely to change the expected answer, before any call to the model under test. Third, two-phase testing (Section 4.4 and Algorithm 1) admits the seed, runs the single-modality tests, and then runs joint inputs in the order of a pairing strategy (Section 4.5). Fourth, the automated oracle (Section 4.6) turns the candidates into confirmed failures and attributed joint failures.

Figure 1. Overview of VLTest.
Visual Variant Generation
VLTest produces visual variants by editing the discrete latent grid of a pretrained VQGAN [17]. The encoder maps a seed image to an index grid , each index refers to a codebook entry with embedding , and the decoder maps a grid back to an image. Intuitively, the grid describes the image as cells, each labeled with one of reusable “visual words”. Replacing a label at all cells where it occurs edits the corresponding visual content throughout the image. Because the decoder has a receptive field larger than one cell, an edit also influences neighboring content.
Index ranking. For an index in , let . VLTest sorts the indices by decreasing frequency, breaking ties by increasing index value, and keeps the top as . Frequent indices cover larger regions and correspond to dominant visual prototypes, so replacing them produces observable edits with few variants.
Replacement along embedding distance. For each , the other codebook indices are sorted by embedding distance, , with non-decreasing in . The variant at rank replaces every occurrence of each by , for all at once, and decodes the edited grid.
Mutation strength and order. The rank range , with , controls mutation strength, and the number of requested variants controls how many ranks are used. VLTest visits ranks in a fixed bit-reversal (van der Corput) order: the -th variant uses rank , where enumerates the bit-reversed integers , shifted by one, and skips values above . Every prefix of this order spreads nearly uniformly over the range, so requesting more variants only appends to the sequence. VLTest removes duplicates by grid hash and, after the model’s own preprocessing, by a hash of the model input; for the API models, whose preprocessing is not observable, the second hash is taken over the request sent to the API.
Textual Variant Generation
VLTest produces textual variants with a three-stage pipeline that bounds the neighborhood of a seed text . In the rewrite stage, a pretrained paraphraser (PEGASUS [80]) generates sentence-level rewrites. In the expand stage, spaCy [25] identifies the content tokens of each rewrite, excluding named entities, numbers, and punctuation, and each eligible token is replaced by one of the top- fill-mask predictions of RoBERTa [39] with the same part of speech; at most tokens are substituted per rewrite. In the prune stage, the candidates of each rewrite group are ranked by Sentence-BERT [52] cosine distance to , and the closest per group are kept after exact-string deduplication. The output interleaves the groups by within-group rank, so that, as on the visual side, a larger pool only appends to a smaller one.
Relation Filters
The generators control how far a variant moves from the seed, but not whether its expected answer changes. VLTest applies three filters before any model call. RQ1 evaluates their precision and recall against the oracle.
Text filter . A variant is kept if a natural language inference model [71] predicts entailment in both directions, . Bidirectional entailment targets equivalence of meaning, which embedding distance alone does not capture.
Image filter . A visual variant is kept if its image embedding is close to that of the seed, . This removes degenerate edits.
Pair filter . An image–text alignment score is used with task-specific rules:
ITM, positive seed: keep if , to retain pairs that are likely to still match.
ITM, negative seed: keep if , to discard pairs that drift toward a match.
VR, true statement: keep if on the mutated image.
For false VR statements and VQA questions, image–text similarity does not reflect the expected answer, so VLTest applies only and and leaves the joint relation to the oracle. Both image-based filters use OpenCLIP ViT-bigG/14 [9]. We chose it over SigLIP [77] because PaliGemma and Qwen3-VL build on SigLIP encoders [49], and the filters should not share a vision encoder with the models under test. The thresholds are , , , and , fixed on a pilot (§ 5.5).
Two-Phase Testing
Algorithm 1 runs the tests for one seed. Test() obtains the output of through Call, logs as a test, adds to or if it is a single-modality candidate, and returns whether is a candidate. The strategies are those of VLTest (§ 4.5); the other values of and the flag implement the joint baselines of Section 5.3 within the same procedure.
Admission (Section 3.3) is per model, so each model is tested on the inputs it answers with a reference answer. The single-modality phase tests the first variants of each filtered sequence, and the joint phase combines exactly these variants. Because every single-modality input is tested before the joint phase, each joint candidate has complete four-cell evidence when it is found. The single-modality phase also yields the single-modality failures of the run. The pair filter is applied again to the joint input, because two variants that pass it separately need not pass it together. A combination that fails the re-check is skipped without a call.
Algorithm 1 Two-phase testing for one seed and one model
Require: seed $s_0=(x_0,t_0)$ with reference answers $A_0$ and reconstruction $\hat{x}_0$; model $M$; ordered sequences $\mathcal{X},\mathcal{T}$; pool size $n$; number of joint tests $n_J$; filter flag $\phi$; pairing strategy $\psi$
Ensure: call log $L$; candidates $\mathcal{F}_V,\mathcal{F}_T$; joint candidates $\mathcal{F}_J$ with four cells
1: $\mathcal{F}_V,\mathcal{F}_T,\mathcal{F}_J\leftarrow\varnothing$ \hfill $\triangleright$ Call reuses a logged output or issues one new call
2: $\operatorname{Call}(x_0,t_0)$ \hfill $\triangleright$ raw seed, not a test
3: if $A(\hat{x}_0,t_0)\ne A(s_0)$ or $\operatorname{norm}(\operatorname{Call}(\hat{x}_0,t_0))\notin A_0$ then
4: return seed not admitted for $M$
5: end if
6: if $\phi$ then
7: $\mathcal{X}\leftarrow[x\in\mathcal{X}\mid F_{\mathrm{img}}(x)\land F_{\mathrm{pair}}(x,t_0)]$
8: $\mathcal{T}\leftarrow[t\in\mathcal{T}\mid F_{\mathrm{txt}}(t)\land F_{\mathrm{pair}}(\hat{x}_0,t)]$
9: end if
10: $\mathcal{X}'\leftarrow$ first $n$ of $\mathcal{X}$; $\mathcal{T}'\leftarrow$ first $n$ of $\mathcal{T}$
11: for all $x\in\mathcal{X}'$ do \hfill $\triangleright$ single-modality phase
12: $\operatorname{Test}(x,t_0)$
13: end for
14: for all $t\in\mathcal{T}'$ do
15: $\operatorname{Test}(\hat{x}_0,t)$
16: end for
17: $C_J\leftarrow0$
18: for all $(x,t)\in\psi.\operatorname{Order}(\mathcal{X}',\mathcal{T}',L)$ while $C_J<n_J$ do \hfill $\triangleright$ joint phase
19: if $\phi\land\neg F_{\mathrm{pair}}(x,t)$ then continue \hfill $\triangleright$ joint re-check, no call
20: end if
21: $C_J\leftarrow C_J+1$
22: if $\operatorname{Test}(x,t)$ then
23: add $\langle(\hat{x}_0,t_0),(x,t_0),(\hat{x}_0,t),(x,t)\rangle$ and the four outputs to $\mathcal{F}_J$
24: end if
25: end for
26: return $L,\mathcal{F}_V,\mathcal{F}_T,\mathcal{F}_J$
Pairing Strategies
VLTest provides two strategies for ordering the pairs in .
Round-robin (RR). Let be the longer of the two pools and the shorter one, with sizes . The -th pair, , combines with . Each round of pairs uses every element of the longer pool once and shifts the shorter pool by one position, so after rounds every pair has appeared exactly once.
Stable-first (SF). SF uses the results of the single-modality tests as feedback. Pairs whose image-only and text-only inputs both have compatible outputs come first, in RR order, followed by the remaining pairs in RR order. Because an attributed joint failure requires correct single-modality inputs, SF first tests the combinations whose parts single-modality testing has not broken.
Oracle
VLTest confirms candidates with an automated oracle built on a VLM judge . The judge receives the seed and its reference answers as a reference, so it decides whether an input keeps a known answer instead of answering the input from scratch. For each generated cell , the oracle runs two checks in separate calls, whether or not the output of the cell is compatible. receives the seed, , and , and returns preserved if is a valid input with the expected answer of the seed, changed if its expected answer differs, invalid if is not a meaningful input for the task, or abstain. It does not see any model output, so the output cannot influence the expected answer. then receives the seed, , , and the output, and returns correct if the output is a correct answer for , such as a synonym of a reference answer, incorrect, or abstain. A cell is correct only if the first check returns preserved and the second returns correct. The two verdicts can contradict each other when the output is compatible: preserved means that the reference answers remain correct, and a compatible output is one of them. The oracle keeps the verdict of the output check in this case, so such a cell counts as incorrect and rejects the joint candidate that contains it (Section 6.3). A candidate is a confirmed failure if its answer check returns preserved and its output check returns incorrect. A joint candidate is an attributed joint failure if the answer check returns preserved for its three generated cells and the output check returns correct for both single-modality inputs and incorrect for the joint input. Abstentions do not confirm failures. The judge is not told which method produced an input. The oracle also runs the answer check on the reconstructed seed of every sampled seed. On API models, a failure must also be reproducible. The seed cell and each cell of a candidate are called five more times, and a cell is correct or incorrect if at least three of the five new outputs are. A confirmed failure then requires a correct seed and an incorrect candidate, and an attributed joint failure also requires correct single-modality inputs. These calls are repeat calls .
Evaluation Design
Research Questions
RQ1 (Validity): How many candidates does the oracle confirm as failures, and how precise is it?
RQ2 (Testing utility and cost): How effectively and efficiently does VLTest find single-modality failures and attributed joint failures?
RQ3 (Joint failures): What attributed joint failures does VLTest reveal?
RQ4 (Sensitivity and generalization): How do the results vary across design choices and VLMs?
RQ1 breaks the candidates down by oracle outcome, measures the precision of the oracle and of its rejections against human labels, and evaluates the filters and VQGAN reconstruction. RQ2 compares the single-modality failures with two single-modality baselines and the attributed joint failures with five joint baselines, under the same budget. RQ3 breaks the joint candidates down by four-cell outcome, separates interaction from additive failures, and categorizes the failures. RQ4 varies the design choices of VLTest and repeats the joint comparison on four general-purpose VLMs.
Datasets and Models
Following [7], we use ITM on MSCOCO [38], VR on NLVR2 [60], and VQA on VQA v2 [20]; each dataset input is a seed. An NLVR2 input has two images and a statement; VLTest mutates the first image, keeps the second, and gives both to the oracle. Each task has two fine-tuned models: BLIP [37] and ViLT [30] for ITM, BEiT-3 [69] and ViLT for VR, and PaliGemma [5] and BEiT-3 for VQA. Seeds are sampled at random from the validation splits, one per source image (per image pair for NLVR2). For ITM, half of the seeds are positive pairs and half are negative pairs. A negative pair combines an image with the caption of a random image that shares no MSCOCO object category with it, and is kept only if the judge, shown the pair without the label, judges it as not matching. Sampling continues until each task–model pair has 100 admitted seeds.
Baselines
Table 2 lists the compared methods. All methods have the same budget (§ 5.5), and all except RND and MetaRA-c apply the relation filters.
Single-modality methods. SINVAD-VQ adapts SINVAD [29] from a VAE to the VQGAN. Its genetic search mutates the continuous encoder output before quantization and minimizes the model’s score for the seed answer. A variant rejected by the filters receives fitness without a call, and every evaluated image is a test. TextFooler [28] ranks the words of by how much deleting them lowers the score for the seed answer and then tries synonym substitutions word by word; every tried substitution is a test. It does not stop at the first success but continues until all words are processed or the budget is spent. Both baselines use output scores, which VLTest does not: the matching score for ITM, the class probability for VR, and the log-probability of the seed answer for VQA, where generating an answer and scoring the seed answer are separate calls. VLTest-V and VLTest-T test the first filtered image or text variants of our generators.
Joint methods. All joint methods run Algorithm 1 with the same and . RND pairs unfiltered pools of our generators in random order. CRND pairs the filtered pools of VLTest in random order. SR first pairs, in random order, the variants whose single-modality inputs have compatible outputs, so it differs from VLTest (SF) only in the order within these pairs. EXT runs SINVAD-VQ and TextFooler for calls each as its single-modality phase, uses the variants they test as its pools, and pairs them in the stable-first order of VLTest; it differs from VLTest (SF) only in the variants it combines. MetaRA-c applies the relations of MetaRA [72] that can be instantiated as seed-relative transformations on VQA: its single-modality phase tests the image and question transformations, and its joint phase combines them. We exclude CIT4DNN [12] and Mimicry [70], which require a low-dimensional or class-conditional latent. We also do not compare with multimodal adversarial attacks such as Co-Attack [79], SGA [40], and VLATTACK [73]. On the image side, they optimize pixel-level perturbations against a surrogate model to maximize attack success under a perturbation bound, and they assume rather than check that the expected answer is preserved. Their confirmed failures would therefore measure attack strength rather than the ability to generate semantically varied test inputs. On the text side, the word-level substitutions such attacks use are represented by TextFooler.
Metrics
The oracle judges every candidate. For each task, model, and method, we report:
CF@1k , the confirmed single-modality failures per 1,000 calls of the budget, where counts the confirmed failures of one modality over the admitted seeds ; a method that stops early is still divided by the full budget;
AJF@1k , the attributed joint failures per 1,000 calls of the joint budget;
budget per AJF, averaged over the task–model pairs; for a method that stops early, we also give the calls it actually makes per attributed joint failure;
budget used, the calls made as a share of the budget, in the single-modality or the joint phase;
tested stable pairs, the share of joint calls on pairs whose image-only and text-only inputs both have compatible outputs;
SeedCov, the share of admitted seeds with at least one confirmed failure, computed separately for single-modality failures and attributed joint failures;
VarCov, the mean number of distinct image and text variants per seed that take part in at least one attributed joint failure;
confirmation rate, the share of candidates that the oracle confirms as failures;
interaction share, the share of attributed joint failures that are interaction failures (§ 3.4).
For ITM and VR, is the difference between the logits of the two labels. For VQA, it uses the log-probabilities of the answers under teacher forcing, averaged over their tokens. For the five classification models, holds by construction, because is their highest-scoring answer. PaliGemma generates its answers, so we report how many of its attributed joint failures have and exclude them from the margin analysis.
Filters and reconstruction. We measure the precision and recall of each filter against the answer check of the oracle on 1,000 unfiltered variants per task and modality, sampled at random from the generator outputs. For reconstruction, we report the share of sampled seeds whose reconstruction changes the expected answer, which are not admitted, and the share of admitted seeds on which the model output differs between the raw and the reconstructed seed.
Settings and Implementation
Budgets. The primary experiments use a budget of calls to the model under test. A single-modality method makes calls on inputs of its modality, including the calls a search method makes to score its options. A joint method makes single-modality calls, per modality, and then joint calls, so in Algorithm 1. Single-modality failures are thus always counted over single-modality calls and attributed joint failures over joint calls. A method whose pool or search space is exhausted stops early.
Cost accounting. Every call to the model under test is logged with its run, its role, and its key. A run applies one method to one seed and one model; a key hashes the model input and the request type. Within a run, each key is called once and its result is reused, so a single-modality input that also serves as four-cell evidence is charged once. A run is charged for every call it issues, even if another run issued the same input. Calls to the raw seed and the seed cell, the repeat calls of stochastic models, and the calls of the judge are not charged to .
Parameters. The VQGAN is the pretrained Gumbel-quantized model of Taming Transformers [17] with downsampling factor 8 and codebook entries. Images are resized to pixels, which gives a index grid. The text generator uses and the full rank range ; the text generator uses and keeps variants. The filters use
, , and . SINVAD-VQ uses the population size and genetic operators of the original implementation, and TextFooler its default synonym sets and thresholds. The stochastic methods (SINVAD-VQ, RND, CRND, SR, and EXT) run five times on the local models; outputs of deterministic models are cached across runs.
Oracle. The judge is Llama-4-Maverick [45], from a family to which none of the models under test belongs. Its prompts and the filter thresholds were fixed on a pilot with 20 seeds per task, which is excluded from the evaluation.
Statistics. The unit of analysis is the source image, or the image pair for NLVR2. We compare methods by paired differences with 95% percentile intervals from a paired cluster bootstrap over the 100 source images of a task–model pair, with 10,000 resamples [16]; for a stochastic method, the five runs are averaged per seed before resampling. Two-sided -values are , where and are the shares of resampled differences at most and at least zero. Holm’s procedure [24] corrects them within Table 5, which marks the significant differences.
| Method | ITM BLIP | ITM ViLT | VR BEiT-3 | VR ViLT | VQA PaliGemma | VQA BEiT-3 | SeedCov (%) | Budget used (%) |
| SINVAD-VQ | 24.1* | 26.8* | 21.6* | 18.9* | 42.7* | 37.5* | 63.4 | 88.9 |
| VLTest-V | 50.3 | 56.4 | 42.7 | 37.2 | 78.5 | 71.4 | 75.7 | 100.0 |
| TextFooler | 19.4* | 21.8* | 15.8* | 13.9* | 30.6* | 27.5* | 56.7 | 81.7 |
| VLTest-T | 46.2 | 50.1 | 38.4 | 34.1 | 74.3 | 68.0 | 71.0 | 100.0 |
Table 5. RQ2: CF@1k per task–model pair; SeedCov and budget used in percent, averaged over the six pairs. ∗: significant difference to VLTest-V or VLTest-T (5% level, Holm).
Implementation. The task-specific models use their fine-tuned checkpoints, and Qwen3-VL uses its public Hugging Face check-points. All local models run in bf16, with greedy decoding for generative outputs. uses roberta-large-mnli, and and use OpenCLIP ViT-bigG/14 trained on LAION-2B. Experiments ran on Ubuntu 24.04.3 LTS with an AMD Ryzen Threadripper PRO 7985WX CPU, 502 GiB of DDR5 memory, and two NVIDIA RTX 6000 Ada GPUs.
Results
RQ1: Validity
We run the ten methods of Table 2 on the six task–model pairs and MetaRA-c on the two VQA pairs, with 100 admitted seeds per pair and . The five stochastic methods run five times, which gives 30 runs per seed (31 on VQA) and 18,200 runs in total.
| Method | Image side | Text side | Joint inputs | |
| SINVAD-VQ | latent search (scores) | – | yes | – |
| VLTest-V | codebook replacement | – | yes | – |
| TextFooler | – | synonym search (scores) | yes | – |
| VLTest-T | – | rewrite–expand–prune | yes | – |
| RND | VLTest-V | VLTest-T | no | random |
| CRND | VLTest-V | VLTest-T | yes | random |
| SR | VLTest-V | VLTest-T | yes | random, stable first |
| EXT | SINVAD-VQ | TextFooler | yes | stable-first |
| MetaRA-c (VQA) | relation-defined | relation-defined | – | relation-defined |
| VLTest (RR) | VLTest-V | VLTest-T | yes | round-robin |
| VLTest (SF) | VLTest-V | VLTest-T | yes | stable-first |
Table 2. Compared methods. : relation filters applied. The first four methods test one modality only.
Candidate Outcomes
Depending on the pair, 61.0% (ViLT on VR) to 84.0% (PaliGemma) of the sampled seeds are admitted. Table 3 breaks down the candidates of VLTest (SF). The oracle confirms 121,235 of the 148,314 candidates (81.7%) as failures, from 76.7% on ViLT for VR to 84.3% on ViLT for ITM. Of the remaining candidates, 21,961 (14.8%) have a changed expected answer, 3,065 (2.1%) are invalid or abstained, and 2,053 (1.4%) have outputs that the oracle judges correct despite being outside the reference-answer set.
| Task | Model | Candidates | Invalid / abstain | Answer changed | Output correct | Confirmed failure | image-only | text-only | joint input |
| ITM | BLIP | 22,776 | 308 | 3,399 | 0 | 19,069 | 91.6 | 86.8 | 82.0 |
| ITM | ViLT | 25,253 | 356 | 3,608 | 0 | 21,289 | 90.7 | 85.4 | 83.0 |
| VR | BEiT-3 | 19,164 | 514 | 3,686 | 0 | 14,964 | 87.0 | 77.0 | 76.7 |
| VR | ViLT | 17,179 | 482 | 3,524 | 0 | 13,173 | 86.3 | 77.1 | 75.0 |
| VQA | PaliGemma | 33,698 | 724 | 3,953 | 1,091 | 27,930 | 89.3 | 83.2 | 81.7 |
| VQA | BEiT-3 | 30,244 | 681 | 3,791 | 962 | 24,810 | 88.8 | 82.7 | 80.7 |
Table 3. RQ1: oracle outcomes of the candidates of VLTest (SF), and the confirmation rate per input type.
On every pair, image-only candidates have the highest confirmation rate (86.3% to 91.6%), text-only candidates are 4.8 to 10.0 percentage points lower, and joint inputs are lowest (75.0% to 83.0%). The joint column judges the joint input alone; Section 6.3 adds the single-modality controls. The text gap is largest on VR, where text-only candidates reach 77.0% and 77.1%: VR statements often hinge on counts and quantifiers, and a rewrite that changes one of them changes the label. On ITM and VR the answer is binary, so an output outside is wrong unless the expected answer changed. On VQA, 2,053 candidates (3.2%) have an output outside that is still correct, such as a synonym of a reference answer, and 6.4% of the output changes stay within and are not candidates. A count of output changes reports both as failures.
Oracle Precision
For each method, we sample reported failures stratified by task, input type (image-only, text-only, joint), and verdict (confirmed failure, and for joint candidates also attributed joint failure), pooling the two models of a task. We label up to 40 failures per stratum for VLTest-V, VLTest-T, CRND, SR, VLTest (RR), and VLTest (SF), and up to 20 for the other methods. We also label 20 rejected joint candidates of VLTest (SF) for each task and reason of rejection, 200 in total, and 30 reconstructed seeds per task.
Nine graduate students in computer science who are not authors label these inputs. Two of them label each input independently, and a third resolves disagreements; an input on which no two agree counts as rejected. The annotators follow the two checks of the oracle: they decide the expected answer before they see the model output, and they see neither the oracle verdict nor the method. Before adjudication, Krippendorff’s is 0.82 on the answer check and 0.88 on the output check. Precision is the share of sampled failures that the annotators agree with, with a 95% Clopper–Pearson interval per stratum. A confirmed-failure estimate that spans several input types weights the stratum precisions, and their interval endpoints, by the numbers of reported failures in the strata.
Table 4 lists the precision of every method; each AJF cell rests on 20 to 39 labeled failures. Precision ranges from 85.0% to 93.9% for confirmed failures and from 80.0% to 94.6% for attributed joint failures; for VLTest (SF), it ranges from 88.8% to 94.6%. The lowest value, 80.0% for attributed joint failures on VR, belongs to RND and EXT. The oracle errs found by the annotators are missed changes of the expected answer, correct answers in other wording judged incorrect, and wrong attribution or ambiguity in joint failures. On the 90 reconstructed seeds, the answer check agrees with the annotators in 95.6% of the cases, and the annotators agree with 85.0% to 95.0% of the rejections in every stratum.
| Method | CF ITM | CF VR | CF VQA | AJF ITM | AJF VR | AJF VQA |
| SINVAD-VQ | 90.0 [68.3,98.8] | 85.0 [62.1,96.8] | 90.0 [68.3,98.8] | – | – | – |
| VLTest-V | 92.5 [79.6,98.4] | 87.5 [73.2,95.8] | 92.5 [79.6,98.4] | – | – | – |
| TextFooler | 85.0 [62.1,96.8] | 85.0 [62.1,96.8] | 90.0 [68.3,98.8] | – | – | – |
| VLTest-T | 90.0 [76.3,97.2] | 87.5 [73.2,95.8] | 92.5 [79.6,98.4] | – | – | – |
| RND | – | – | – | 85.0 [62.1,96.8] | 80.0 [56.3,94.3] | 85.0 [62.1,96.8] |
| CRND | – | – | – | 87.9 [71.8,96.6] | 85.7 [69.7,95.2] | 90.3 [74.2,98.0] |
| SR | – | – | – | 91.2 [76.3,98.1] | 86.8 [71.9,95.6] | 91.9 [78.1,98.3] |
| EXT | – | – | – | 85.0 [62.1,96.8] | 80.0 [56.3,94.3] | 85.0 [62.1,96.8] |
| MetaRA-c | – | – | – | – | – | 85.0 [62.1,96.8] |
| VLTest (RR) | – | – | – | 91.2 [76.3,98.1] | 89.2 [74.6,97.0] | 91.2 [76.3,98.1] |
| VLTest (SF) | 92.0 [78.8,98.2] | 88.8 [74.5,96.6] | 93.9 [81.0,99.0] | 92.3 [79.1,98.4] | 89.5 [75.2,97.1] | 94.6 [81.8,99.3] |
Table 4. RQ1: oracle precision in percent with 95% intervals, estimated from samples drawn separately for each method and pooled over the two models of each task. CF: confirmed failures; AJF: attributed joint failures. CF precision is shown for the single-modality methods and VLTest (SF).
Filters and Reconstruction
Of the 6,000 unfiltered variants, 1,000 per task and modality, and are evaluated on the 3,000 text and the 3,000 image variants, and on the 3,000 variants of ITM and of true VR statements, to which it applies. Before filtering, 78.8% of the text variants and 86.5% of the image variants preserve the expected answer. , , and keep 70.8%, 82.7%, and 74.6% of the variants, with a precision of 93.1%, 95.4%, and 91.7% and a recall of 83.6%, 91.2%, and 86.0% for answer preservation. Each filter thus removes 70% to 77% of the variants whose expected answer changes and loses 9% to 16% of those that keep it. The answer check rejects the changed answers that pass the filters (Table 3).
Reconstruction changes the expected answer of 2% to 6% of the sampled seeds, which are not admitted. On the admitted seeds, the model output on the reconstructed seed differs from that on the raw seed for 4% to 9%, depending on the pair.
RQ2: Testing Utility and Cost
Single-Modality Failures
Table 5 lists CF@1k and SeedCov for the single-modality methods. VLTest-V finds 1.8 to 2.1 times as many confirmed failures as SINVAD-VQ, and VLTest-T 2.3 to 2.5 times as many as TextFooler, without the output scores that both baselines use. Our generators also cover more seeds, 75.7% and 71.0% against 63.4% and 56.7%. All differences are significant. SINVAD-VQ and TextFooler stop early on some seeds and use 88.9% and 81.7% of the budget; per call actually made, their aggregate rates rise by 12% and 22%, and the gap remains. For all four methods, failures are most frequent on VQA and least frequent on VR.
Attributed Joint Failures
Table 6 reports AJF@1k for the joint methods. VLTest (SF) finds 91.4 (ViLT on VR) to 196.8 (PaliGemma) attributed joint failures per 1,000 joint calls, one per 7.6 joint calls on average, and finds at least one on 98.2% of the seeds. All joint methods except MetaRA-c use the full joint budget.
| Method | ITM BLIP | ITM ViLT | VR BEiT-3 | VR ViLT | VQA PaliGemma | VQA BEiT-3 | Budget used (%) | Budget per AJF | VarCov (I / T) | SeedCov (%) | Tested stable pairs (%) |
| RND | 65.9 | 72.3 | 48.4 | 41.7 | 101.6 | 89.4 | 100.0 | 15.8 | 34.6 / 33.1 | 84.7 | 73.5 |
| CRND | 116.8 | 117.9 | 86.1 | 75.1 | 168.4 | 149.1 | 100.0 | 9.1 | 83.1 / 81.3 | 92.7 | 85.2 |
| SR | 133.1 | 146.0 | 99.2 | 87.8 | 192.4 | 167.1 | 100.0 | 7.8 | 84.6 / 82.2 | 93.7 | 100.0 |
| EXT | 105.3 | 118.1 | 78.6 | 68.7 | 155.9 | 137.2 | 100.0 | 9.8 | 32.1 / 29.8 | 92.9 | 99.2 |
| MetaRA-c | – | – | – | – | 69.7 | 61.9 | 51.4 | 15.3 | 18.2 / 16.3 | 90.5 | 68.9 |
| VLTest (RR) | 120.5 | 119.4 | 89.9 | 77.6 | 171.3 | 152.4 | 100.0 | 8.8 | 97.4 / 95.1 | 94.8 | 85.2 |
| VLTest (SF) | 137.2 | 151.6 | 103.7 | 91.4 | 196.8 | 173.2 | 100.0 | 7.6 | 105.6 / 102.2 | 98.2 | 100.0 |
Table 6. RQ2: AJF@1k per task–model pair; budget used, budget per AJF, VarCov (image / text), SeedCov, and the share of joint calls on stable pairs, averaged over the pairs a method runs on.
The joint methods differ one component at a time. Applying the relation filters substantially improves the yield: CRND and VLTest (RR) find 1.6 to 1.9 times as many attributed joint failures as RND. Stable-first ordering raises the share of joint calls on stable pairs from 85.2% (CRND and VLTest (RR)) to 100% (SR and VLTest (SF)). It increases AJF@1k by 12% to 24% from CRND to SR and by 14% to 27% from VLTest (RR) to VLTest (SF) across the six task–model pairs; averaged over the pairs, AJF@1k rises from 118.9 to 137.6 and from 121.9 to 142.3, and the budget per attributed
joint failure falls from 9.1 to 7.8 and from 8.8 to 7.6 calls. This gain follows from the definition of an attributed joint failure: a pair with an incompatible single-modality output cannot yield one on ITM and VR, and rarely on VQA. If every stable pair were equally likely to yield an attributed joint failure, moving the remaining 14.8% of the joint calls to stable pairs would raise AJF@1k by about 17% (), close to the observed average gains of 16% and 17%. Stable-first ordering therefore does not make a stable pair more likely to fail; it spends the joint budget only on pairs that can yield an attributed joint failure, at no additional cost.
Within the same priority class, round-robin ordering changes AJF@1k by only 1% to 5% (VLTest (RR) versus CRND, and VLTest (SF) versus SR). It combines every variant of the pools equally often, whereas random ordering leaves some variants untested within the joint budget. The attributed joint failures of round-robin ordering involve more distinct variants (VarCov 97.4 / 95.1 against 83.1 / 81.3 for CRND, and 105.6 / 102.2 against 84.6 / 82.2 for SR) and cover more seeds (SeedCov 94.8% against 92.7%, and 98.2% against 93.7%). EXT applies the same stable-first pairing strategy to variants produced by SINVAD-VQ and TextFooler. VLTest (SF) finds 1.26 to 1.33 times as many attributed joint failures as EXT across the six task–model pairs, while its failures involve more than three times as many distinct variants per seed (VarCov 105.6 / 102.2 versus 32.1 / 29.8). On the two VQA pairs, VLTest (SF) finds 2.8 times as many attributed joint failures as MetaRA-c under the full-budget metric. MetaRA-c exhausts the combinations of its relations after using 51.4% of the joint budget on average. When normalized by the calls it actually makes, MetaRA-c finds about 128 attributed joint failures per 1,000 joint calls, or one per 7.8 calls, compared with about 185 per 1,000 calls, or one per 5.4 calls, for VLTest (SF).
RQ3: Joint Failures
RQ3 analyzes the joint candidates of the RQ1 runs.
Four-Cell Evidence
Table 7 shows why the joint candidates of VLTest (SF) do or do not become attributed joint failures. Of its 110,152 joint candidates, 85,390 (77.5%) are attributed joint failures, and four-cell confirmation rejects 24,762 (22.5%).
| Task | Rejected by the joint input (%) | Further rejected by the controls (%) | Attributed joint failures (%) |
| ITM | 17.5 | 2.5 | 80.0 |
| VR | 24.1 | 3.7 | 72.2 |
| VQA | 18.8 | 2.6 | 78.6 |
| All | 19.7 | 2.8 | 77.5 |
Table 7. RQ3: joint candidates of VLTest (SF) by the first check that fails, starting with the joint input, in percent of the joint candidates, pooled over the two models of each task.
Table 7 attributes each rejection to the first check that fails, starting with the joint input. Checking the joint input alone rejects 19.7% of the joint candidates: its expected answer changes, it is invalid or the judge abstains, or, on VQA, its output is a correct answer outside . The two single-modality controls reject a further 2.8% of joint candidates. Among all joint candidates, 3,101 (2.8%) exhibit conflicting oracle verdicts on at least one control: the output is compatible with the reference answers and the answer check returns preserved, but the output check judges it incorrect. Such conflicting verdicts prevent a candidate from becoming an attributed joint failure. The annotators agree with 85.0% to 90.0% of the sampled rejections of this type. For 9,183 candidates (8.3%), the joint input changes the expected answer while both single-modality inputs keep it; no check of a single-modality input can detect this. VR has the lowest share of attributed joint failures (72.2%; 73.1% on BEiT-3 and 71.2% on ViLT): a single-modality input changes the expected answer for 11.7% and 12.9% of its joint candidates, against 5.8% to 6.5% on the other pairs.
Interaction and Additive Failures
We apply the margin analysis of Section 3.4 to the attributed joint failures of VLTest (SF) (Table 8). On PaliGemma, 622 of the 19,683 attributed joint failures (3.2%) have under teacher forcing, so the sequence score does not reproduce the decoded output; we exclude them. Of the remaining 84,768, 13,999 (16.5%) are interaction failures: the additive prediction still prefers a reference answer, but the joint input fails. Their share ranges from 13.8% on PaliGemma to 22.6% on ViLT for VR and is highest on VR. The other 83.5% are additive. The median is positive on every pair, so the observed joint margin is usually less negative than the additive prediction.
Failure Types
Two authors open-coded a separate pilot of 30 attributed joint failures and then independently assigned one primary type to each attributed joint failure in a stratified random sample of 180, 60 per task (Cohen’s before adjudication). Table 9 lists the five types. Quantity, count, and scope errors concentrate on VR (26 of 34).
| Failure type | ITM | VR | VQA | Total |
| Attribute or entity binding | 24 | 9 | 19 | 52 |
| Spatial or relational composition | 17 | 15 | 14 | 46 |
| Quantifier, count, or scope | 4 | 26 | 4 | 34 |
| Lexical grounding or co-reference | 11 | 6 | 12 | 29 |
| Fine-grained category or part | 4 | 4 | 11 | 19 |
| Total | 60 | 60 | 60 | 180 |
Table 9. RQ3: types of attributed joint failures in a stratified random sample of 180, 60 per task.
Figure 2 shows two attributed joint failures with their four cells. In the ITM case, the image variant changes the shade and texture of the court, and the text variant rewrites “Two women play tennis on a blue court” as “Two females are playing tennis on a blue court”. The model predicts a match for the seed and both single-modality inputs and no match for the joint input, although the caption still describes the image. In the VQA case, the image variant shifts the shading of the donuts, and the text variant rewrites “What color are the donuts?” as “Which color do these donuts have?”. The model answers “brown” for the seed and both single-modality inputs and “yellow” for the joint input. Single-modality testing passes both cases.

Figure 2. RQ3: two attributed joint failures, each with the reconstructed seed (, ), the image-only input, the text-only input, and the joint input. Changed words are in bold. Model outputs from left to right: match, match, match, no match (ITM); brown, brown, brown, yellow (VQA).
RQ4: Sensitivity and Generalization
Design Choices
We vary one design choice of VLTest (SF) at a time on BLIP (ITM), BEiT-3 (VR), and PaliGemma (VQA), with the same 100 seeds, budget, and other settings: (a) the relation filters, with no filter, each filter alone, or all filters; (b) the pool size , which sets the single-modality calls to while the joint calls stay at ; (c) the rank range ; (d) the number of edited indices ; and (e) the text pipeline with rewrite, expand, or prune removed. These are 16 settings and 4,800 runs.
Table 10 reports averages over the three pairs; CF@1k is computed over the single-modality calls of each setting, and the default appears once in every group. The default reaches a confirmation rate of 81.6%, 55.1 CF@1k, and 145.9 AJF@1k. Without filters, VLTest (SF) finds 25% fewer attributed joint failures (108.7) and its confirmation rate drops to 71.8%. Among the single filters, achieves the highest AJF@1k (131.4), followed by (123.5) and (120.2). However, its overall candidate confirmation rate (71.9%) remains close to that of the unfiltered setting (71.8%). These metrics capture different aspects of testing effectiveness: the confirmation rate measures the proportion of candidates confirmed as failures across input types, whereas AJF@1k measures attributed joint failures per 1,000 joint calls. A pool of 250 variants finds 9% fewer than the default of 500, and a pool of 750 about as many (147.0). Narrowing the rank range reduces AJF@1k to 136.7 at and 121.4 at , while the confirmation rate remains between 80.8% and 81.3%. AJF@1k peaks at and 20 (145.9 and 146.6) and drops at (140.7), where CF@1k also falls to 51.8. Removing any text stage lowers the confirmation rate to 73.9% to 79.6% and AJF@1k by 9% to 13%; removing the prune stage lowers CF@1k the most, to 47.5.
| Design choice | Setting | Conf. rate (%) | CF@1k | AJF@1k |
| (a) relation filters | none | 71.8 | 46.2 | 108.7 |
| only | 77.6 | 51.3 | 123.5 | |
| only | 76.4 | 50.6 | 120.2 | |
| only | 71.9 | 52.8 | 131.4 | |
| all (default) | 81.6 | 55.1 | 145.9 | |
| (b) pool size | 80.7 | 51.8 | 132.6 | |
| (default) | 81.6 | 55.1 | 145.9 | |
| 81.3 | 54.9 | 147.0 | ||
| (c) rank range () | 81.3 | 48.6 | 121.4 | |
| 80.8 | 52.9 | 136.7 | ||
| (default) | 81.6 | 55.1 | 145.9 | |
| (d) edited indices () | 80.1 | 47.3 | 118.8 | |
| 80.8 | 52.6 | 135.2 | ||
| (default) | 81.6 | 55.1 | 145.9 | |
| 81.2 | 55.0 | 146.6 | ||
| 81.9 | 51.8 | 140.7 | ||
| (e) text stages | −rewrite | 73.9 | 51.1 | 126.5 |
| −expand | 79.6 | 52.7 | 132.9 | |
| −prune | 75.8 | 47.5 | 128.4 | |
| full (default) | 81.6 | 55.1 | 145.9 |
Table 10. RQ4: design choices of VLTest (SF), averaged over BLIP (ITM), BEiT-3 (VR), and PaliGemma (VQA).
General-Purpose Models
We also test four general-purpose VLMs on all three tasks: Qwen3-VL-8B-Instruct and Qwen3-VL-32B-Instruct [49], one open-weight family at two scales, and two commercial models, GPT-4o mini [47] in its snapshot of 18 July 2024 and Gemini 2.5 Flash [10]. On each model, we run VLTest (SF), SR, and CRND. The Qwen3-VL models use the settings of § 5.5 with 100 seeds per task. The API models use 50 sampled seeds per task, , and ; Table 11 lists the admitted seeds . For ITM and VR, the prompts and the rules for extracting a label are fixed in advance, and an output that maps to no label counts as an abstention. The API models are called at temperature 0 where the API allows it, and every call is logged with its date. The API experiments use 37,650 single-modality calls and 37,650 joint calls for failure discovery.
| Task | Model | Non-det. floor | AJF@1k | vs. SR | vs. CRND | |
| ITM | Qwen3-VL-8B | 100 | – | 169.0 | 18.2 | 33.7 |
| ITM | Qwen3-VL-32B | 100 | – | 135.6 | 14.4 | 29.8 |
| ITM | GPT-4o mini | 43 | 4.7% | 139.5 | 5.6 | 33.0 |
| ITM | Gemini 2.5 Flash | 41 | 2.4% | 127.3 | 20.0 | 27.8 |
| VR | Qwen3-VL-8B | 100 | – | 148.2 | 21.0 | 28.2 |
| VR | Qwen3-VL-32B | 100 | – | 111.8 | 13.2 | 25.0 |
| VR | GPT-4o mini | 38 | 2.6% | 121.6 | 7.4 | 21.6 |
| VR | Gemini 2.5 Flash | 40 | 2.5% | 125.0 | 17.5 | 24.5 |
| VQA | Qwen3-VL-8B | 100 | – | 249.8 | 19.2 | 37.4 |
| VQA | Qwen3-VL-32B | 100 | – | 176.4 | 14.8 | 32.2 |
| VQA | GPT-4o mini | 45 | 4.4% | 172.9 | 12.9 | 41.3 |
| VQA | Gemini 2.5 Flash | 44 | 2.3% | 193.6 | 24.1 | 32.7 |
Table 11. RQ4: VLTest (SF) on the general-purpose models. : admitted seeds. vs. SR and vs. CRND: differences in AJF@1k. Non-det. floor: share of admitted seeds with observed output variation.
Table 11 shows the results. VLTest (SF) finds 111.8 to 249.8 attributed joint failures per 1,000 joint calls, and its rate exceeds that of SR (by 5.6 to 24.1) and CRND (by 21.6 to 41.3) on all twelve task–model pairs. On the API models, the differences rest on few failures: the smallest, 5.6 for GPT-4o mini on ITM, corresponds to about 12 attributed joint failures. As on the task-specific models, VQA yields the most attributed joint failures and VR the fewest. Qwen3-VL-32B yields 19.8% to 29.4% fewer attributed joint failures than Qwen3-VL-8B on every task, each on its own admitted seeds. On the API models, 2.3% to 4.7% of the admitted seeds show output variation.
Discussion
Output Changes Overstate Failures
Table 3 gives the share of candidates that are confirmed failures, by input type: 86.3% to 91.6% for image-only inputs, 77.0% to 86.8% for text-only inputs, and 75.0% to 83.0% for joint inputs. A count of output changes therefore overstates the failures of VLTest (SF) by 9% to 16% (relative) on image-only inputs, 15% to 30% on text-only inputs, and 20% to 33% on joint inputs; on VQA it also counts the 6.4% of output changes that stay within the reference answers. The overstatement depends on the generator: without the relation filters, the confirmation rate falls from 81.6% to 71.8% (Table 10), so a count of output changes would credit the unfiltered generators with 39% more failures than they find, against 23% for the filtered ones. Counting output changes is still useful for screening, but the effectiveness of a generator should be measured on confirmed failures.
What Four-Cell Evidence Adds
A count of joint failures without single-modality evidence mixes failures caused by the image mutation, failures caused by the text mutation, and failures caused by their combination. Four-cell confirmation rejects 22.5% of the joint candidates of VLTest (SF) (Table 7), but most of these rejections, 19.7% of the joint candidates, come from checking the joint input itself; the single-modality controls reject a further 2.8%. The main contribution of the controls is therefore attribution: for every reported joint failure, the four cells provide evidence that the model handles each mutation alone but fails on their combination. The interaction contrast then separates the 16.5% of score-classifiable attributed joint failures that are interactions between the modalities from the rest, where two single-modality effects cross the decision boundary only together (Table 8). These additive failures are still failures on valid inputs that single-modality testing passes, and the positive median shows that, within these failures, the two effects usually cancel in part rather than reinforce each other.
| Task | Model | Attributed joint failures | Interaction (%) | Median [IQR] |
| ITM | BLIP | 13,716 | 16.4 | 0.121 [0.051, 0.238] |
| ITM | ViLT | 15,163 | 14.7 | 0.109 [0.043, 0.219] |
| VR | BEiT-3 | 10,368 | 21.3 | 0.154 [0.062, 0.298] |
| VR | ViLT | 9,139 | 22.6 | 0.163 [0.038, 0.312] |
| VQA | PaliGemma | 19,683 | 13.8 | 0.097 [0.038, 0.194] |
| VQA | BEiT-3 | 17,321 | 15.1 | 0.113 [0.045, 0.226] |
| All | 85,390 | 16.5 | 0.124 [0.049, 0.248] |
Table 8. RQ3: attributed joint failures of VLTest (SF) classified by the margin analysis. Interaction: share of interaction failures; : median with interquartile range. Both are computed over the score-classifiable failures, which exclude the 622 failures of PaliGemma with (84,768 in all).
Implications
For VLM developers, more than half of the sampled attributed joint failures concern attribute binding or spatial composition, and quantifier and count errors cluster on VR (Table 9). Each attributed joint failure comes with its four cells and can serve as a regression test that checks the two mutations alone and together. The rates of attributed joint failures should not be read as a ranking of tasks or models: they depend on how often the oracle rejects candidates, which differs between tasks (Table 7), and on the admitted seeds, which differ between models. For testing researchers, checking the expected answer removes a fifth of the counted failures, and matched single-modality controls are needed to attribute a joint failure to the combination; both belong in the evaluation of multimodal testing studies.
Threats to Validity
Internal Validity. The oracle relies on a VLM judge, which can make mistakes. RQ1 measures the precision of the failures it reports for every method and task and checks a sample of the joint candidates of VLTest (SF) that it rejects; we do not measure how many failures it misses for the other methods. The judge receives the reference answers as a reference, which may bias its answer check toward preserved; missed answer changes are among the oracle errors that the annotators find (Section 6.1). Under the joint-input-first attribution of rejection reasons, the single-modality controls reject a further 2.8% of joint candidates; these rejections include cases with conflicting judge verdicts and are not evidence that the model fails on the single-modality input. All cells are built on the reconstructed seed, and seeds whose reconstruction changes the expected answer are not admitted. Admitting seeds per model favors inputs that the model handles correctly, and we report the admission rates. The filters and the judge share no model family or vision encoder with the models under test. All counts rest on the oracle verdicts, whose precision RQ1 measures. CF@1k and AJF@1k count failing inputs, not distinct faults; VarCov shows how many distinct variants the attributed joint failures involve.
External Validity
Our study covers three tasks, eight models, and one tokenizer family. All cells are VQGAN reconstructions, which carry reconstruction artifacts; the oracle checks that each reported input is a valid input for the task, and the annotators check this on a sample. Results may differ for other tokenizers and generators, for high-resolution or text-heavy images, and for multiturn systems. The seeds come from public validation splits that the models under test and the judge may have seen during training; the generated variants are new inputs, but they stay close to these seeds. The results for the API models are tied to the model versions available during the study and to a reduced budget.
Conclusion
This paper presented VLTest, a black-box approach for testing VLMs that finds failures whose expected answers are confirmed by an oracle and attributes joint failures to the combination of image and text mutations with four-cell evidence. VLTest combines controlled visual and textual generators, relation filters, and two-phase testing with feedback-directed pairing. An automated oracle confirms every reported failure, and human labels on a stratified sample measure its precision. The oracle confirms 81.7% of the candidates of VLTest (SF) and agrees with human labels on 80.0% to 94.6% of the failures it reports across the compared methods in the primary experiment, and on 88.8% to 94.6% of those reported by the default configuration of VLTest. Under the same budget, the generators of VLTest find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and VLTest finds 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the variants of these two baselines. Four-cell confirmation rejects 22.5% of the joint candidates, 19.7% through the joint input alone and 2.8% through the single-modality controls. Under an additive margin model, 16.5% of the score-classifiable attributed joint failures are interactions between the modalities; in the rest, the additive prediction alone already fails, although each single-modality input passes. The relation filters account for most of the difference between configurations, and on four general-purpose models, including two commercial API models, VLTest (SF) finds more attributed joint failures than CRND and SR on every task.
References
References
- [1]Estelle Alfaro, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal. 2022. Vl-interpret: An interactive visualization tool for interpreting vision-language transformers. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 21406–21415.arxiv.org/abs/2203.17247
- [2]Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision. 2425–2433.
- [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2.5-VL Technical Report. arXiv:2502.13923 [cs.CV]
- [4]Oriol Barbany, Michael Huang, Xinliang Zhu, and Arnab Dhua. 2024. Leveraging large language models for multimodal search. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 1201–1210.
- [5]Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexey Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Abdulmohsin, Michael Tschannen, Emanuele Bugliariello, et al. 2024. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 (2024).arxiv.org/abs/2407.07726
- [6]Tyler A Chang and Benjamin K Bergen. 2024. Language model behavior: A comprehensive survey. Computational Linguistics 50, 1 (2024), 293–350.arxiv.org/abs/2303.11504
- [7]Shuo Chen, Jindong Guo, Zhen Han, Yunpu Ma, Philip Torr, and Volker Tresp. 2023. Benchmarking robustness of adaptation methods on pre-trained vision-language models. Advances in Neural Information Processing Systems 36 (2023), 51758–51777.arxiv.org/abs/2306.02080
- [8]Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic Testing: A Review of Challenges and Opportunities. Comput. Surveys 51, 1, Article 4 (2018), 27 pages. doi:10.1145/3143561DOI
- [9]Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2023. Reproducible Scaling Laws for Contrastive Language-Image Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2818–2829.
- [10]Gheorghe Chintai et al. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261 (2025).
- [11]Swaroopa Dola, Matthew B. Dwyer, and Mary Lou Soffa. 2021. Distribution-Aware Testing of Neural Networks Using Generative Models. In Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE).arxiv.org/abs/2102.13602
- [12]Swaroopa Dola, Rory McDaniel, Matthew B Dwyer, and Mary Lou Soffa. 2024. Generating diverse and rare inputs for neural networks using latent space combinatorial testing. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13.
- [13]Bin Duan, Meiru Che, and Guowei Yang. 2026. Testing Neural Networks via Bayesian-Guided Exploration of Decision Landscapes. arXiv preprint arXiv:2606.04314 (2026).arxiv.org/abs/2606.04314
- [14]Bin Duan, Matthew B Dwyer, and Guowei Yang. 2025. Generating Realistic, Diverse, and Fault-Revealing Inputs with Latent Space Interpolation for Testing Deep Neural Networks. arXiv preprint arXiv:2503.17630 (2025).arxiv.org/abs/2503.17630
- [15]Bin Duan, Matthew B Dwyer, and Guowei Yang. 2026. Latent Anchor-Driven Test Generation for Deep Neural Networks. arXiv preprint arXiv:2606.04310 (2026).arxiv.org/abs/2606.04310
- [16]Bradley Efron and Robert J. Tibshirani. 1994. An Introduction to the Bootstrap. Chapman & Hall/CRC, New York.
- [17]Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12873–12883.arxiv.org/abs/2012.09841
- [18]Zheren Fu, Zhendong Mao, Yan Song, and Yongdong Zhang. 2023. Learning semantic relationship among instances for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15159–15168.DOI
- [19]Jia Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW). IEEE, 50–56.arxiv.org/abs/1801.04354
- [20]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR.
- [21]Jianmin Guo, Yu Jiang, Yue Zhao, Quan Chen, and Jiaguang Sun. 2018. DFluzz: Differential fuzzing testing of deep learning systems. In Proceedings of the 2018 26th ACM joint Meeting on European Software Engineering Conference and the Foundations of Software Engineering. 739–743.DOI
- [22]Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 14953–14962.arxiv.org/abs/2211.11559
- [23]Jack Hessel and Lillian Lee. 2020. Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.62DOI
- [24]Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics 6, 2 (1979), 65–70.DOI
- [25]Matthew Honnibal, Ines Montani, Sofie Van Landeghem, Adriane Boyd, et al. 2020. spaCy: Industrial-strength natural language processing in python. (2020).
- [26]Cheng-Yu Hsieh, Jiewu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. 2023. SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Vol. 36. Curran Associates, Inc.
- [27]Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. arXiv preprint arXiv:1804.06059 (2018).arxiv.org/abs/1804.06059
- [28]Di Jin, Zhijing Jin, Joey Tian Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 8018–8025.arxiv.org/abs/1907.11932
- [29]Seungmin Kang, Robert Feldt, and Shin Yoo. 2020. Sinvad: Search-based image space navigation for dnn image classifier test input generation. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops. 521–528.arxiv.org/abs/2005.09296
- [30]Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. In International conference on machine learning. PMLR, 5583–5594.
- [31]D. Richard Kuhn, Dolores R. Wallace, and Albert M. Gallo, Jr. 2004. Software Fault Interactions and Implications for Software Testing. IEEE Transactions on Software Engineering 30, 6 (2004), 418–421. doi:10.1109/TSE.2004.24DOI
- [32]Seokhyun Lee, Sooyoung Cha, Dain Lee, and Hakjoo Oh. 2020. Effective white-box testing of deep neural networks with adaptive neuron-selection strategy. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 165–176.DOI
- [33]Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, and Percy Liang. 2024. VHELM: A Holistic Evaluation of Vision Language Models. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Vol. 37. Curran Associates, Inc.arxiv.org/abs/2410.07112
- [34]Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. 2024. NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Vol. 37. Curran Associates, Inc.arxiv.org/abs/2410.14669
- [35]Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyu Tian. 2024. LLaVA-OneVision: Easy Visual Task Transfer. arXiv:2408.03326 [cs.CV]arxiv.org/abs/2408.03326
- [36]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML.arxiv.org/abs/2301.12597
- [37]Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML.arxiv.org/abs/2201.12086
- [38]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV.arxiv.org/abs/1405.0312
- [39]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs.CL]arxiv.org/abs/1907.11692
- [40]Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. 2023. Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
- [41]Siyu Lu, Mingzhe Liu, Lirong Yin, Zhengtong Yin, Xuan Liu, and Wenfeng Zheng. 2023. The multi-modal fusion in visual question answering: a review of attention mechanisms. PeerJ Computer Science 9 (2023), e1400.DOI
- [42]Muhammad Arsalan Manzoor, Sarah Albarrı, Ziting Xian, Zaiqiao Meng, Preslav Nakov, and Shanghong Liang. 2023. Multimodality representation learning: A survey on evolution, pretraining and its applications. ACM Transactions on Multimedia Computing, Communications and Applications 20, 3 (2023), 1–34.arxiv.org/abs/2302.00389
- [43]Maryam Maryam, Matteo Biagiola, Andrea Stocco, and Vincenzo Riccio. 2025. Benchmarking Generative AI Models for Deep Learning Test Input Generation. In Proceedings of the 18th IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE.arxiv.org/abs/2412.17652
- [44]Maryam Maryam, Matteo Biagiola, Paolo Tonella, and Vincenzo Riccio. 2026. DeepNaqqal: Human-Aligned Automated Validation of Test Inputs for Deep Learning. In Proceedings of the 19th IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE.DOI
- [45]Meta. 2025. Llama-4-Maverick-17B-128E-Instruct Model Card. Hugging Face. https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct Accessed: 2026-10-21.
- [46]Michael Nie and Hareton Leung. 2011. A Survey of Combinatorial Testing. Comput. Surveys 43, 2, Article 11 (2011), 29 pages. doi:10.1145/1883612.1883618DOI
- [47]OpenAI. 2024. GPT-4o mini: Advancing Cost-Efficient Intelligence. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/.
- [48]Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. Deepexplore: Automated whitebox testing of deep learning systems. In proceedings of the 26th Symposium on Operating Systems Principles. 1–18.DOI
- [49]Qwen Team. 2025. Qwen3-VL Technical Report. arXiv:2511.21631 [cs.CV]
- [50]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.arxiv.org/abs/2103.00020
- [51]Pooyan Rahmanzadehgeriv, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. 2024. Vision Language Models Are Blind. In Proceedings of the Asian Conference on Computer Vision (ACCV). Springer, 18–34.
- [52]Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, 3982–3992. doi:10.18653/v1/D19-1410DOI
- [53]Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computationalDOI
- [54]Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 4902–4912. doi:10.18653/v1/2020.acl-main.442DOI
- [55]Vincenzo Riccio and Paolo Tonella. 2023. When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Study. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE).arxiv.org/abs/2212.11368
- [56]Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. 2024. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. arXiv preprint arXiv:2402.12336 (2024).arxiv.org/abs/2402.12336
- [57]Sergio Segura, Gordon Fraser, Ana B. Sanchez, and Antonio Ruiz-Cortés. 2016. A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering 42, 9 (2016), 805–824. doi:10.1109/TSE.2016.2532875DOI
- [58]Mong Yuan Sim, Wei Emma Zhang, Xiang Dai, and Biaoyan Fang. 2025. Can vlms actually see and read? a survey on modality collapse in vision-language models. In Findings of the Association for Computational Linguistics: ACL 2025. 24452–24470.
- [59]Lev Sorokin, Chen Yang, Ken E. Friedel, and Andrea Stocco. 2026. Search-based Testing of Vision Language Models for In-Car Scene Understanding. arXiv:2607.02300 [cs.SE]arxiv.org/abs/2607.02300
- [60]Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th annual meeting of the association for computational linguistics. 6418–6428.
- [61]Youcheng Sun, Xiaowei Huang, Daniel Kroening, James Sharp, Matthew Hill, and Rob Ashmore. 2018. Testing deep neural networks. arXiv preprint arXiv:1803.04792 (2018).
- [62]Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 5238–5248.arxiv.org/abs/2204.03162
- [63]Shengbang Tong, Erik Jones, and Jacob Steinhardt. 2023. Mass-producing failures of multimodal systems with language models. Advances in Neural Information Processing Systems 36 (2023), 29292–29322.arxiv.org/abs/2306.12105
- [64]Pablo Valle, Sergio Segura, Shaukat Ali, and Aitor Arrieta. 2026. Metamorphic Testing of Vision-Language Action-Enabled Robots. In Proceedings of the 19th IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE. arXiv:2602.22579.arxiv.org/abs/2602.22579
- [65]VLTest. 2026. VLTest. https://github.com/beanduan22/VLTest/tree/main
- [66]VQA Team. [n.d.]. VQA Evaluation. https://visualqa.org/evaluation.html. Accessed 19 October 2026.
- [67]Jialai Wang, Han Qiu, Yi Rong, Hengkai Ye, Qi Li, Zongpeng Li, and Chao Zhang. 2022. BET: black-box efficient testing for convolutional neural networks. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 164–175.DOI
- [68]Tianshi Wang, Fengyue Lin, Lei Zhu, Jingjing Li, Zheng Zhang, and Heng Tao Shen. 2025. Cross-modal retrieval: a systematic review of methods and future directions. Proc. IEEE (2025).arxiv.org/abs/2308.14263
- [69]Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Shubhojit Som, et al. 2023. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19175–19186.
- [70]Oliver Weißl, Amr Abdellatif, Xingcheng Chen, Giorgi Merabishvili, Vincenzo Riccio, Severin Kacianka, and Andrea Stocco. 2025. Targeted deep learning system boundary testing. ACM Transactions on Software Engineering and Methodology (2025).
- [71]Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Association for Computational Linguistics, 1112–1122. doi:10.18653/v1/N18-1101DOI
- [72]Quanxing Xu, Yuhao Tian, Ling Zhou, Xiaohang Xiao, Huang Huan, and Chia-Wen Lin. 2026. MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems. arXiv:2605.19307 [cs.CV]arxiv.org/abs/2605.19307
- [73]Ziyi Yin, Muchao Ye, Tianrong Zhang, Tianyu Du, Jingguo Zhu, Han Liu, Jinghui Chen, Ting Wang, and Fenglong Ma. 2023. VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36. Curran Associates, Inc.
- [74]Boxi Yu, Zhiqiang Zhong, Jiaqi Li, Yixing Yang, Shilin He, and Pinjia He. 2023. ROME: Testing Image Captioning Systems via Recursive Object Melting. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA).
- [75]Boxi Yu, Zhiqiang Zhong, Xinran Qin, Jiayi Yao, Yuancheng Wang, and Pinjia He. 2022. Automated Testing of Image Captioning Systems. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA).
- [76]Yuanyuan Yuan, Shuai Wang, Mingyue Jiang, and Tsong Yueh Chen. 2021. Perception matters: Detecting perception failures of vqa models using metamorphic testing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16908–16917.DOI
- [77]Xinghua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 11975–11986.arxiv.org/abs/2303.15343
- [78]Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE transactions on pattern analysis and machine intelligence 46, 8 (2024), 5625–5644.DOI
- [79]Jiaming Zhang, Qi Yi, and Jitao Sang. 2022. Towards Adversarial Attack on Vision-Language Pre-training Models. In Proceedings of the 30th ACM International Conference on Multimedia (MM). arXiv:2206.09391.arxiv.org/abs/2206.09391
- [80]Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International conference on machine learning. PMLR, 11328–11339.arxiv.org/abs/1912.08777
- [81]Qingyang Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongyuan Li, Ngai-Man Cheung, and Min Lin. 2023. On evaluating adversarial robustness of large vision-language models. Advances in Neural Information Processing Systems 36 (2023), 54111–54138.arxiv.org/abs/2305.16934
- [82]Jingguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, et al. 2025. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv:2504.10479 [cs.CV].arxiv.org/abs/2504.10479