WorksComputer Science
MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation
Abstract
Image generation models have rapidly advanced, with many models now supporting generation conditioned on multiple reference images and textual instructions. Such models enable a wide range of applications, including entity fusion and attribute transfer across images. However, existing benchmarks for multi-reference image generation rely on limited evaluation dimensions and fail to capture the diverse challenges of the task. To address this limitation, we propose MDMR-Bench, a Multi-Dimensional benchmark for Multi-Reference image generation. MDMR-Bench is organized around representative capability settings and consists of multiple evaluation subsets spanning diverse dimensions of multi-reference image generation, including (1) common entity fusion and attribute transfer, (2) multi-element extraction from a single image, (3) infographic manipulation, and (4) visual reasoning-intensive generation. This design enables systematic analysis of model capabilities and failure modes across different challenge dimensions. Furthermore, to achieve more reliable and interpretable evaluation, we introduce a checklist-based evaluation protocol that decomposes generation quality into fine-grained evaluation aspects. Instead of relying solely on holistic scoring, our framework uses structured checklists composed of both global-level and atomic-level queries, enabling more diagnostic assessment of model performance. By benchmarking both open-source and proprietary models, we demonstrate that MDMR-Bench provides fine-grained insights into model performance across different dimensions. The dataset and code are publicly available.
Figure 1. The overview of MDMR-Bench. It covers diverse challenge dimensions for multi-reference image generation and introduces a checklist-based evaluation framework across five evaluation aspects with global-level and atomic-level queries.
fine-grained evaluation aspects. Instead of relying solely on holistic scoring, our framework uses structured checklists composed of both global-level and atomic-level queries, enabling more diagnostic assessment of model performance. By benchmarking both open-source and proprietary models, we demonstrate that MDMR-Bench provides fine-grained insights into model performance across different dimensions. The dataset is publicly available at Hugging Face3. The code is released at GitHub4.
Introduction
Text-to-image [23, 46, 7, 11, 43] and instruction-based single-image editing models [44, 27, 2, 41, 42] have achieved remarkable progress. They support high-quality image generation from text and fine-grained, controllable edits on a single image. However, text alone is often insufficient to precisely specify complex visual attributes, such as the exact appearance of clothing, or abstract properties like hairstyle, pose, and artistic style. In practice, users naturally rely on multiple reference images to convey such details, leading to a growing demand for multi-reference image generation. In response, recent models, such as Qwen-Image-Edit [46] and FLUX.2 [3], as well as proprietary models like GPT-Image-1.5 [30], Seedream [38, 5], and Nano-Banana Pro [13], support multi-image inputs and demonstrate promising capabilities in integrating information from them.
While benchmarks for text-to-image generation (e.g., GenEval [12, 20], DPGbench [16]) and instruction-based image editing (e.g. GEdit [27, 19], ImgEdit [52]) have become increasingly rich and comprehensive, benchmarks for multi-reference image generation remain scarce. Representative efforts have begun to explore this setting, with several benchmarks recently introduced. However, as shown in Table 1, existing benchmarks remain limited in several aspects. First, some existing benchmarks are limited in scale, often containing only a few hundred cases [48, 47], and provide relatively narrow task coverage. Many primarily focus on simple fusion or transfer scenarios, while some are restricted to specific applications, such as virtual try-on in VTEdit-Bench [25]. Second, reference images are often overly simple and contain many visually unnatural samples, typically dominated by salient foreground objects, and thus do not reflect the complexity of real-world scenes [47, 45, 48, 33]. Most importantly, existing benchmarks lack a multi-dimensional evaluation settings, typically organizing tasks along only a single aspect, such as the number of references or coarse task categories. While some benchmarks vary reference properties (e.g., photo fused with anime) [33], such variations mainly reflect superficial input styles rather than the diverse capabilities required in real-world multi-reference image generation.
To address these limitations, we propose MDMR-Bench, a Multi-Dimensional benchmark for Multi-Reference image generation, containing 2,390 cases collected from visually realistic sources, including both real-world images and high-quality synthetic images, as illustrated in Figure 1. Its main contribution is capability diversity: dedicated coverage of under-evaluated task families, global- and atomic-level checklist evaluation across five diagnostic dimensions, and detailed analysis of model capabilities and failure modes. Although MDMR-Bench is smaller than MultiBanana and VTEdit-Bench by raw sample count, it provides broader dedicated task coverage than MultiBanana, as shown in Table 1. In addition to varying the number of reference images, we include scenarios requiring multi-element extraction from a single reference image, infographic manipulation involving visual elements such as artistic text and logos, and reasoning-intensive generation that requires implicit relationship understanding across multiple references.
Figure 2. Inter-judge agreement measured by Krippendorff’s alpha. Settings are provided in the Appx. A.1
As for evaluation, most existing benchmarks [33, 47] adopt score-based evaluation methods similar to VIEScore [22], assigning a scalar score (e.g., 0-10) for each evaluation aspect of a generated result. However, such scoring remains highly dependent on the evaluator model and often overlooks finegrained details, such as subtle texture inconsistencies in clothing. Recently, several works [17, 51, 8] have proposed checklist-based evaluation, where a set of structured questions is generated and answered by VLMs to assess specific aspects of the output, demonstrating better alignment with human judgment. To further evaluate reliability, we compare checklist-based and score-based evaluation using multiple evaluator models on MDMR-Bench. As shown in Figure 2, checklist-based evaluation achieves substantially higher inter-judge agreement indicating more consistent assessment across judges. Building on this insight, we adopt a checklist-based evaluation protocol, where the checklist includes both global-level and atomic-level questions. Global-level questions evaluate whether the generated image satisfies the overall prompt requirements, while atomic-level questions focus on fine-grained details. We evaluate MDMR-Bench on eleven representative open-source [46, 47, 10, 3] and proprietary models [38, 5, 13, 14, 30]. Some findings include: current models exhibit a large gap between open-source and proprietary systems, perform substantially better on entity fusion than on infographic manipulation, and obtain lower scores on subsets with more reference images. In summary, our key contributions are as follows:
| Benchmark | Size | Entity Fusion | Multi-Element Extraction | Attribute Transfer | Infographic Manip. | Visual Reasoning | Global Eval. | Atomic Eval. | Metrics |
| DreamBooth [35] | 75 | ✓ | ✗ | △ | ✗ | ✗ | ✓ | ✗ | CLIP Fidelity |
| DreamOmni2 [48] | 319 | ✓ | ✗ | ✓ | △ | ✗ | ✓ | ✗ | Score-based |
| MICo-Bench [45] | 897 | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | Score-based |
| MultiBanana [33] | 3769 | ✓ | ✗ | ✓ | △ | ✗ | ✓ | ✗ | Score-based |
| VTEdit-Bench [25] | 24220 | ✓ | ✗ | △ | ✗ | ✗ | ✓ | ✗ | Score-based |
| MDMR-Bench (Ours) | 2390 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Checklist-based |
Table 1. Comparison with other benchmarks. ✓ denotes dedicated coverage, △ denotes partial or incidental coverage, and ✗ denotes no coverage.
We introduce MDMR-Bench, a capability-diverse benchmark for multi-reference image generation with 2,390 cases, emphasizing broader dedicated task coverage rather than a larger number of cases.
Our benchmark covers diverse challenge settings, including entity fusion, attribute transfer, multi-element extraction, infographic manipulation, and reasoning-intensive generation.
To the best of our knowledge, we are the first to adopt a checklist-based evaluation protocol for multi-reference image generation, enabling fine-grained diagnosis of model capabilities and failures.
Related Works
From Text and Single-Reference to Multi-Reference Image Generation
Text-to-image generation has achieved remarkable progress since the emergence of stable diffusion [34]. Subsequent works such as Imagen [36], Flux [23], Qwen-Image [46] and Z-Image [43] have significantly improved generation quality and text alignment. Building on these advances, research has further explored single-reference image generation. These approaches mainly fall into two categories: subject-driven generation, which preserves the identity of a reference subject (e.g., DreamBooth [35] and OmniControl [50]), and editing-based generation, which modifies an input image according to instructions (e.g., SeedEdit3 [44], FLUX-Kontext [2], Emu-Edit [39], and DreamOmni [49]). However, textual instructions alone are often insufficient to precisely describe complex visual attributes, such as hairstyles, poses, artistic styles, or specific identities. To address this limitation, recent works have begun to incorporate multiple reference images as complementary visual guidance. Representative models include Qwen-Image-Edit [46], DreamOmni2 [48], the GPT-Image series [29, 30], the NanoBanana series [14, 13], and the Seedream series [38, 5]. However, it remains unclear how well these models perform on multi-reference image generation, due to the lack of reliable and fine-grained evaluation benchmarks.
Benchmarks for Reference-Based Image Generation
Existing benchmarks can be broadly categorized into single-reference and multi-reference settings. Single-Reference Benchmarks Early benchmarks such as MagicBrush [54] and EMU-Edit [39] introduce large-scale, instruction-driven image editing datasets covering diverse tasks, but suffer from relatively low-quality input images and limited coverage of task types. Subsequent works [52, 53, 27, 28, 19] focus on improving realism and coverage. For example, GEdit-Bench [27] proposes a general-purpose benchmark grounded in real user instructions, while its follow-up version GEdit-v2 [19] further improves human-centric coverage and consistency evaluation. Reason-Edit [18] focuses on reasoning-intensive instruction-based image editing, which we also incorporate as a visual reasoning dimension in our multi-reference benchmark.
Multi-Reference Benchmarks OmniContext [47] introduces a multi-reference benchmark with around 400 cases focusing on scene, character, and object fusion, while MICo-Bench [45] covers 897 cases across object-centric, human-centric, human-object interaction, and De&Re tasks; however, both lack coverage of fine-grained attribute transfer. DreamOmni2-Bench [48] convers a relatively comprehensive set of tasks, but contains only 319 cases, limiting its suitability for large-scale and fine-grained evaluation, and lacks a systematic evaluation protocol. MultiBanana [33] extends the number of reference images up to eight and systematically explores variations such as domain mismatch (e.g., photo vs. anime), scale mismatch, rare concepts. However, it primarily varies the properties of input references and lacks explicit task-level difficulty modeling. In addition, its visual styles are relatively homogeneous, limiting the realism and diversity of evaluation scenarios.
Evaluation Metrics and Protocols
Many early single-image editing benchmarks [53, 54, 39, 18] provide ground-truth images, enabling the use of similarity-based metrics such as L1 distance, CLIP [15], and DINO [6] for evaluation. However, the target output is often not uniquely defined. As VLMs become increasingly powerful, evaluation can move beyond traditional similarity-based metrics toward more semantic and instruction-aware assessment. Recent works adopt VLM-based evaluators with score-based approaches, such as VIEScore [22], ImgEdit-Bench [52], and GEdit-Bench [27], which produce one or a few scalar scores (e.g., per-dimension ratings), yet remain coarse-grained and unstable cross-evaluation consistency. More recently, an increasingly popular remedy in T2I evaluation is checklist-based evaluation, which decomposes a prompt into atomic yes/no questions answered by VLMs. This paradigm is pioneered by TIFA [17], and some following works [26, 24, 40, 8] have been shown to yield finer granularity, better interpretability, and stronger alignment with human judgment than holistic scoring. However, in multi-reference image generation, where fidelity must be verified against multiple heterogeneous references simultaneously, checklist-based evaluation remains largely unexplored. We therefore adopt a checklist-based evaluation philosophy for multi-reference image generation, which provides more fine-grained, interpretable, and stable assessment than score-based evaluation.
MDMR-Bench
In this section, we introduce MDMR-Bench as a multi-dimensional benchmark for multi-reference image generation, covering diverse and realistic generation scenarios such as entity fusion, attribute transfer, multi-element extraction, infographic manipulation, and reasoning-intensive generation.
Challenging Dimensions
We organize the benchmark around representative capability settings in multi-reference image generation. Entity Fusion and Attribute Transfer are established settings in prior benchmarks, while Multi-Element Extraction tests selecting several elements from one source, Infographic Manipulation tests structured content and layout editing, and Visual Reasoning tests implicit relationships. These settings cover complementary capabilities. Specifically, we construct several challenge settings spanning different aspects of multi-reference generation, as described below.
Entity Fusion and Attribute Transfer. We include common multi-reference generation scenarios involving entity fusion and attribute transfer to evaluate models’ ability to accurately combine identities, attributes, and visual concepts from multiple reference images. In this subset, we vary the number of reference images from 2 to 4 to further assess multi-reference composition capability.
Multi-Element Extraction. Existing benchmarks [33, 48] often assume that each reference image provides only a single relevant element. In contrast, we include scenarios where multiple relevant
elements must be extracted and integrated from a single reference, evaluating models’ ability to precisely localize and compose multiple visual attributes simultaneously.
Infographic Manipulation. Recent models [13, 30] have demonstrated strong capabilities in infographic-oriented image generation. We therefore include infographic manipulation scenarios involving visual elements such as artistic text and logos to evaluate models’ ability in design-oriented editing and structured visual composition.
Visual Reasoning. We include reasoning scenarios where the required modifications are not explicitly specified in the instructions and must be inferred from references, evaluating higher-level reasoning and implicit relationship understanding across multiple inputs.
Sample Construction
The samples in MDMR-Bench are primarily constructed via an automatic and scalable pipeline, consisting of three stages: (1) Image Gallery Construction, (2) Task Bucket Filling, and (3) Sample Validation and Refinement, as illustrated in Figure 3. Although most samples are generated through this unified pipeline, certain settings require specialized construction procedures, which are introduced in their corresponding subsections.
Figure 3. Sample construction pipeline of MDMR-Bench.
Image Gallery Construction. In the first stage, we construct a large-scale image gallery that serves as the foundation for subsequent benchmark construction. Images are primarily collected from the LAION [37] with resolution above 1024. To complement web-collected images, we additionally generate synthetic images using Nano Banana Pro and GPT-Image-1.5. Specifically, captions are first extracted from real images and then used as prompts for image generation, aiming to retain the depicted content while obtaining cleaner images. In the final benchmark, 48.2% of cases contain at least one synthetic reference image, and 29% of reference images are synthetic. A paired real-versus-synthetic experiment is reported in Appendix A.5. Each image is automatically annotated using Gemini-3-Pro [9], which assigns structured semantic tags, including high-level content categories (e.g., people, animals, objects, scenes, and posters), fine-grained visual attributes (e.g., hairstyle, texture, facial expression, filter, and painting style), and auxiliary metadata such as the number of individuals, age, body framing (e.g., full-body or half-body), and facing direction. The resulting image pool contains approximately 500K images.
Task Bucket Filling. In the second stage, we define a set of task buckets for sample construction. Each bucket specifies a task type together with the corresponding reference image sampling specifications and requirements. For example, a three-image fusion bucket requires three reference images sampled according to predefined entity and attribute constraints. To ensure diversity within each task type (e.g., three-image fusion), we divide tasks into multiple buckets based on variations in image categories and compositions. Each bucket is populated with 10–50 samples. We organize sampling using two categories of annotations from the image gallery: entity tags, including person, animal, object, scene, text, outfit, and decoration, and attribute tags, including artistic style, filter, era, texture, color, hair, expression, makeup, and pose. To further improve diversity, we incorporate auxiliary tags during sampling, such as the number of people in an image and the amount of text in infographic images.
Sample Validation and Refinement. In the final stage, we perform sample validation and refinement through a multi-step process. First, given the task description, we use Gemini-3-Pro [9] to verify whether the sampled image combinations are valid and consistent with the intended task. Next, Gemini-3-Pro is used to generate corresponding instructions in both English and Chinese. We then conduct human verification to filter out low-quality or ambiguous cases, including those that violate task constraints, contain low-quality images (e.g., heavy cropping or artifacts), or result in semantically or visually inconsistent compositions.
With the above pipeline, we construct 936 entity fusion samples and 982 attribute transfer samples, each containing 2 to 4 reference images. We further construct 101 multi-element extraction cases, where multiple entities or attributes are extracted from one reference image and applied to another. For infographic manipulation, we construct 278 cases (155 Chinese and 123 English) covering operations such as addition, replacement, color modification, font change, texture/material editing, and style transfer. For addition and replacement tasks, target elements are extracted from or generated based on the original image to construct realistic and controllable editing scenarios. For visual reasoning, we construct cases spanning temporal (e.g., time and season), stylistic, state, spatial, and layout transformations, as well as operations such as addition, replacement, removal, and extraction. We first use VLMs to generate candidate reasoning formulations, which are then instantiated into concrete cases using GPT and Nano Banana Pro. The resulting cases are further divided into two categories: analogical reasoning, where reference images demonstrate the target transformation pattern, and logical reasoning, where the target output must be inferred from multiple reference images. We provide key statistics of MDMR-Bench, with additional analysis included in the appendix. Figure 4a shows the distribution of different challenge settings in the benchmark. Figure 4b reports the image type distribution, organized into normal entities, global attributes, and local attributes.
Figure 4. Benchmark statistics of MDMR-Bench.
Evaluation Metrics
We adopt a binary checklist-based protocol rather than asking a judge model for a single holistic score. Each generated image is evaluated along five dimensions: Following, whether the output strictly follows the prompt instruction; Consistency, whether reference elements and fine-grained details are faithfully preserved; Composition, whether generated elements are arranged and interact in a plausible way; Minimality, whether the output avoids unrequested modifications; and Quality, whether the image is realistic, aesthetically coherent, and free of obvious artifacts.
For each sample, every evaluation aspect contains two equally weighted groups of yes/no questions: global-level questions, which evaluate overall prompt-level requirements, and atomic-level questions, which focus on fine-grained object- and attribute-level details. These questions are automatically generated by Gemini-3-Pro based on the instruction and reference images. We manually audit a subset of the generated checklist questions; the audit protocol and results are provided in Appendix A.4. Figure 5 shows the distribution of the number of questions in the checklist. During evaluation, Gemini-3-Pro serves as the default VLM judge and answers each question with “Y” or “N” and provides a short

Figure 5. Checklist distribution.
reason. We score each dimension by its yes rate and compute the item-level overall score with weights 0.20, 0.25, 0.25, 0.20, and 0.10 for Following, Consistency, Composition, Minimality, and Quality, respectively. To reduce bias from uneven subtask sizes, Entity Fusion, Visual Reasoning, Attribute Transfer, and Infographic Manipulation are first averaged within each subtask and then across subtasks; Multi-Element Extraction is averaged directly because it has no subtask split. The final score is the equal-weight mean over the five task families, using judgments from the default evaluator alone. Consequently, task sample counts do not directly determine their contribution to Overall. The dimension weights are manually specified: Consistency and Composition emphasize preserving and integrating multiple references; Following measures instruction adherence; Minimality penalizes unrequested changes; and Quality retains a basic visual-quality check with lower weight to emphasize reference-specific capabilities.
Evaluation Results
Experimental Setup
Baseline Models We evaluate leading proprietary systems that support multi-reference generation, including GPT-Image-2 [32], Nano Banana 2 [14], Nano Banana Pro [13], GPT-Image-1.5 [30], Seedream 4.0 [38], and Seedream 5.0-Lite [5]. These models represent current high-performing commercial image generation systems with strong instruction-following and multi-image conditioning capabilities. We also include prevalent open-source baselines, including BAGEL [10], DreamOmni2 [48], OmniGen2 [47], Qwen-Image-Edit [46], and FLUX.2 [3]. These models cover a diverse set of unified multimodal generation and editing architectures. Evaluating them alongside proprietary systems allows us to compare the capability gap between closed and open models, and to identify which difficulty dimensions remain challenging for the broader research community.
Output Resolution. Each sample is annotated with a target output resolution. During evaluation, we select the closest supported resolution for each model according to its native resolution settings, using the same selection rule across systems.
Experimental Results
Table 2 reports scores from Gemini-3-Pro, the default evaluator. GPT-Image-2 ranks first (88.79), followed by Nano Banana Pro (85.1) and Nano Banana 2 (81.2). FLUX.2 leads the open-source models (51.3). Proprietary models outperform open-source models across all five dimensions. GPT-5.4 results are reported separately as a robustness analysis in Appendix § 7.§ 2.
| Model | Following | Consistency | Composition | Minimality | Quality | Overall |
| Closed-source Models | ||||||
| GPT-Image-2 [32] | 90.1 | 87.4 | 90.8 | 84.5 | 93.2 | 88.8 |
| Nano Banana Pro [13] | 91.2 | 84.6 | 83.9 | 80.4 | 87.0 | 85.1 |
| Nano Banana 2 [14] | 88.0 | 82.0 | 81.2 | 72.1 | 84.1 | 81.2 |
| Seedream 5.0 [5] | 86.2 | 74.0 | 81.1 | 71.5 | 84.3 | 78.8 |
| Seedream 4.0 [38] | 85.3 | 73.4 | 80.7 | 69.7 | 82.8 | 77.8 |
| GPT-Image-1.5 [30] | 90.3 | 72.1 | 85.9 | 56.8 | 85.6 | 77.5 |
| Open-source Models | ||||||
| FLUX.2 [3] | 46.5 | 50.0 | 51.1 | 51.5 | 63.8 | 51.3 |
| Qwen-Image-Edit [46] | 47.7 | 46.4 | 51.5 | 49.1 | 61.3 | 49.9 |
| BAGEL [10] | 37.3 | 40.7 | 34.7 | 58.1 | 39.8 | 41.9 |
| DreamOmni2 [48] | 43.9 | 31.0 | 49.8 | 33.1 | 63.0 | 41.9 |
| OmniGen2 [47] | 30.1 | 22.2 | 34.6 | 21.1 | 42.1 | 28.6 |
Table 2. Primary leaderboard under the default Gemini-3-Pro judge, in percent. GPT-5.4 results are reported separately in Appendix § 7.§ 2.
The dimension-level results further reveal different failure modes. Proprietary models generally obtain high Following and Quality scores, but their Consistency, Composition, and especially Minimality scores are lower, suggesting that even strong models may copy the wrong details, compose reference elements imperfectly, or introduce unintended changes. For open-source models, Quality is often higher than Following and Consistency, indicating that they can generate visually plausible images but frequently fail to satisfy the precise multi-reference constraints. These results show that MDMR-Bench does not merely rank models by image realism, but also exposes whether models can faithfully bind, preserve, and compose information from multiple references.
Task-Level Results
Table 3 breaks down performance by task type and reveals that different tasks stress different capabilities. We have the following findings: 1) Proprietary models exhibit relatively balanced performance across all task types, while open-source models show substantial performance variation across different settings. This suggests that current proprietary systems have developed more comprehensive multi-reference generation capabilities, whereas open-source models still rely more heavily on task-specific strengths. 2) Interestingly, Multi-Element Extraction, which can be viewed as a special case of Entity Fusion and Attribute Transfer, is often easier for current models than using multiple separate reference images. A possible explanation is that consolidating multiple reference elements within a single image reduces the effective multi-reference complexity, whereas increasing the number of reference images introduces longer visual token sequences and greater compositional difficulty, leading to degraded model performance. 3) Except for Nano Banana Pro, all models achieve noticeably lower performance on the Infographic Manipulation subset, suggesting that precise infographic editing remains a major challenge for current multi-reference image generation systems. 4) For reasoning-intensive generation, models with stronger vision-language understanding capabilities achieve substantially better performance. In particular, Nano Banana Pro and Seedream 4 outperform most other systems on reasoning-related tasks. Interestingly, BAGEL also attains relatively strong reasoning performance despite weaker overall image quality. As a unified mixture-of-transformers architecture that jointly models visual understanding and generation, BAGEL suggests that effective multi-reference reasoning depends heavily on integrated visual comprehension capabilities.
| Model | Entity Fusion | Multi-Element Extraction | Visual Reasoning | Attribute Transfer | Infographic Manip. | Overall |
| Closed-source Models | ||||||
| GPT-Image-2 [32] | 91.5 | 89.6 | 84.8 | 90.2 | 87.6 | 88.7 |
| Nano Banana Pro [13] | 84.9 | 87.2 | 85.6 | 83.3 | 84.7 | 85.1 |
| Nano Banana 2 [14] | 82.5 | 85.6 | 75.9 | 82.4 | 79.8 | 81.2 |
| Seedream 5.0 [5] | 77.9 | 81.3 | 83.6 | 78.6 | 72.5 | 78.8 |
| Seedream 4.0 [38] | 77.9 | 83.1 | 83.7 | 76.6 | 67.9 | 77.8 |
| GPT-Image-1.5 [30] | 81.8 | 78.4 | 76.5 | 82.3 | 68.5 | 77.5 |
| Open-source Models | ||||||
| FLUX.2 [3] | 73.5 | 54.1 | 36.7 | 47.9 | 44.0 | 51.3 |
| Qwen-Image-Edit [46] | 66.9 | 55.3 | 44.3 | 44.1 | 39.2 | 49.9 |
| BAGEL [10] | 41.5 | 39.9 | 52.1 | 35.6 | 40.5 | 41.9 |
| DreamOmni2 [48] | 50.0 | 44.2 | 34.2 | 43.1 | 38.0 | 41.9 |
| OmniGen2 [47] | 44.9 | 29.0 | 22.7 | 26.9 | 19.7 | 28.6 |
Table 3. Task-level results for eleven models under Gemini-3-Pro. Scores are percentages.
Reliability of VLM Judge
To validate our checklist-based evaluation protocol, we compare VLM judge decisions with human judgments. As shown in Table 4, the three proprietary judges achieve similar agreement ratios with humans, with Gemini-3-Pro [9], GPT-5.4 [31], and Qwen3.5 [4] reaching 0.853, 0.854, and 0.849, respectively. Their rank correlations are also strong, especially Gemini-3-Pro [9], which obtains the highest Spearman correlation (0.862) and Cohen’s (0.642). These results support the use of VLM judges as scalable proxies for human evaluation in MDMR-Bench.
Table 4. Correlation between human and VLM judges.
The human study comprises 5,924 checklist decisions from 200 benchmark cases and 800 outputs from Nano Banana Pro, Seedream 5.0, FLUX.2, and Qwen-Image-Edit. Five annotators independently answer Y/N while viewing the instruction, all references, and the generated output. Generator identities are hidden and output order is randomized; final human labels use majority voting. Human inter-annotator Krippendorff’s is 0.88. Task counts and further protocol details are given in Appendix A.3.
Among the evaluated judges, Gemini-3-Pro [9] provides the most balanced reliability across agreement ratio, Pearson correlation, Spearman correlation, and Cohen’s , and serves as the default evaluator for the primary leaderboard and diagnostic analyses. Qwen3-VL-32B [1] is less aligned with human ratings, but still achieves a reasonable agreement ratio of 0.800, suggesting that open-source VLMs can serve as useful alternatives when proprietary APIs are unavailable.
Diagnostic Analysis
We conduct two diagnostic analyses on the Attribute Transfer subset: (1) the impact of different reference content types, and (2) observed performance across subsets with different numbers of input images.
Impact of Reference Content Types. We observe as shown in Figure 6 that current models perform substantially worse on human-related attribute transfer tasks, such as facial expression, makeup, outfit, pose, and hairstyle, likely because these tasks require precise manipulation of human appearance and identity. The dimension-wise heatmaps also reveal trade-offs hidden by the aggregate score. For example, GPT-Image-1.5 achieves strong Following, indicating that it can satisfy the requested attribute-transfer instructions across content types, but this comes with a much lower Minimality score. In other words, it often follows the requested transfer by changing content that should have remained untouched, which limits its Overall score.

Figure 6. Diagnostic heatmaps for the original ten models under Gemini-3-Pro.
Impact of the Number of Reference Images. Figure 7 shows lower performance on Attribute Transfer subsets containing more reference images. From two to four references, average Overall is lower by roughly 0.106, and Composition by roughly 0.122, on the plotted 0–1 scale.
Figure 7. Diagnostic curves for the original ten models under Gemini-3-Pro across different numbers of images. Overall measures aggregate performance, Consistency measures reference fidelity, and Composition measures multi-reference integration.
Conclusion
We introduced MDMR-Bench, a 2,390-case, multi-dimensional benchmark for evaluating multi-reference image generation. MDMR-Bench covers realistic multi-reference scenarios across entity fusion, attribute transfer, multi-element extraction, infographic manipulation, and visual reasoning-aware generation. To move beyond coarse holistic scores, we further adopt a checklist-based evaluation protocol that decomposes each generation result into interpretable and visually verifiable judgments. GPT-Image-2 leads the primary leaderboard evaluated by Gemini-3-Pro. Our evaluation of representative proprietary and open-source models shows that multi-reference image generation remains challenging. Proprietary systems substantially outperform open-source models, but even strong models exhibit non-trivial failures in reference consistency, composition, and minimality.
The task-level results further show that models are relatively stronger at visible entity fusion, while infographic manipulation, visual reasoning-aware generation, and fine-grained attribute transfer remain difficult. Diagnostic analyses reveal that subsets with more references have lower performance, and that instruction following can come at the cost of unintended modifications. We hope MDMR-Bench provides a useful foundation for measuring these capability gaps and guiding future progress toward more faithful, controllable, and reliable multi-reference generation systems.
Dataset and Code Availability
The MDMR-Bench dataset is publicly available at https://huggingface.co/datasets/MDMR-Bench/MDMR-Bench. The evaluation code and release assets are publicly available at https://github.com/LHRYANG/MDMR-Bench.
Appendix
Inter-Judge Agreement Analysis Setting
To evaluate the reliability of different evaluation protocols, we conduct an inter-judge agreement analysis using four evaluator models: GPT-5.4 [31], Doubao-Seed-1.8 [4], Qwen3-VL-32B [1], and Gemini-3-Pro [9]. We apply these evaluator models to assess outputs generated by four representative image generation systems, including Nano Banana Pro [13], FLUX.2 [3], Seedream5 [5], and Qwen-Image-Edit [46]. For score-based evaluation, we adopt a scoring prompt similar to the evaluation protocol used in MultiBanana [33], where evaluator models assign scalar scores for each evaluation aspect. For our checklist-based evaluation, evaluator models answer structured global-level and atomic-level questions. We compute Krippendorff’s alpha [21] to measure inter-judge agreement across different evaluator models. As shown in Figure 2, checklist-based evaluation consistently achieves substantially higher agreement than conventional score-based evaluation across most evaluation aspects, indicating improved evaluation consistency across judges. Separately, Gemini score-based evaluation achieves Krippendorff’s against human scores using the interval metric.
Cross-Judge Evaluation
The primary leaderboard in Table 2 uses Gemini-3-Pro alone. However, evaluator independence is important. Therefore we conducted an additional non-Gemini judge analysis to directly test whether our conclusions depend on the Gemini judge. This supplementary analysis is not required to compute the benchmark score. Benchmark cases, generated outputs, Gemini-generated checklists, and hierarchical aggregation are held fixed; only the final evaluator changes. Table 5 reports GPT-5.4 scores. GPT-Image-2, Nano Banana Pro, and Nano Banana 2 remain the top three under both judges, while Seedream 4.0, GPT-Image-1.5, and Seedream 5.0 reorder. Together, these results indicate that the main conclusions are stable under a non-Gemini evaluator and are not explained by same-family coupling. Empirically, the coupling between the VLM evaluator and the image generator appears limited: model-quality differences dominate over generator–judge family membership.
| Model | Following | Consistency | Composition | Minimality | Quality | Overall |
| Closed-source Models | ||||||
| GPT-Image-2 | 89.99 | 81.90 | 92.04 | 76.59 | 95.67 | 86.37 |
| Nano Banana Pro [13] | 86.99 | 76.15 | 86.01 | 72.04 | 88.29 | 81.18 |
| Nano Banana 2 [14] | 85.80 | 74.34 | 83.25 | 65.24 | 88.00 | 78.41 |
| Seedream 5.0 [5] | 83.21 | 66.48 | 80.67 | 63.76 | 85.50 | 74.73 |
| Seedream 4.0 [38] | 84.38 | 66.99 | 81.46 | 64.10 | 87.35 | 75.54 |
| GPT-Image-1.5 [30] | 85.28 | 69.17 | 83.08 | 56.18 | 87.98 | 75.15 |
| Open-source Models | ||||||
| FLUX.2 [3] | 45.69 | 42.99 | 52.16 | 46.14 | 62.90 | 48.44 |
| Qwen-Image-Edit [46] | 47.42 | 41.05 | 53.45 | 42.83 | 61.94 | 47.87 |
| BAGEL [10] | 39.18 | 35.33 | 37.87 | 47.48 | 39.51 | 39.58 |
| OmniGen2 [47] | 33.00 | 19.07 | 34.40 | 19.45 | 41.56 | 28.01 |
Table 5. Evaluation results using GPT-5.4 as the judge.
Human Evaluation Protocol
The human evaluation uses 200 benchmark cases and four generators: Nano Banana Pro, Seedream 5.0, FLUX.2, and Qwen-Image-Edit. Each generator contributes one output per case, for 800 generated outputs. The evaluation set contains 5,924 checklist decisions. Five annotators independently judge each evaluated checklist item. They inspect the textual instruction, all reference images, and the generated output, and answer “Y” or “N” according to whether the checklist item is satisfied, using the same evaluation instructions as the VLM judges. Generator names and identities are hidden, and the order of model outputs is randomized. The final human label is the majority vote of the five annotators. Human inter-annotator agreement is Krippendorff’s .
| Task family | Cases |
| Entity Fusion | 70 |
| Attribute Transfer | 60 |
| Multi-Element Extraction | 20 |
| Infographic Manipulation | 30 |
| Visual Reasoning | 20 |
| Total | 200 |
Table 6. Task distribution of the 200 human-evaluated benchmark cases.
Checklist Validity Audit
Five PhD-level human auditors examine 2,962 checklist items from 100 benchmark cases covering all five task families. The audit checks whether each item is incorrect, ambiguous, or unanswerable; an item is valid only if it falls into none of these failure categories. Table 7 gives the reported item counts. Of the 2,962 items, 2,925 are valid and 37 are invalid, giving a validity rate of 98.75% and an error rate of 1.25%.
| Dimension | Total items | Invalid | Error rate |
| Following | 491 | 7 | 1.43% |
| Consistency | 780 | 11 | 1.41% |
| Composition | 634 | 6 | 0.95% |
| Minimality | 584 | 9 | 1.54% |
| Quality | 473 | 4 | 0.85% |
| Total | 2,962 | 37 | 1.25% |
Table 7. Checklist validity audit: reported counts by evaluation dimension.
Controlled Real-versus-Synthetic Inputs
We select 100 cases whose original references are all real images. Each reference is captioned, and Nano Banana Pro synthesizes a corresponding image from the caption. This yields paired real-input and synthetic-input versions with the same task structure and instruction semantics. Nano Banana Pro and Seedream 5.0 are evaluated with the same checklist protocol and Gemini-3-Pro judge.
| Model | Inputs | Following | Consistency | Composition | Minimality | Quality | Overall |
| Nano Banana Pro | Real | 95.37 | 88.39 | 80.19 | 85.64 | 86.67 | 87.01 |
| Synthetic | 95.18 | 88.68 | 85.54 | 80.01 | 88.48 | 87.44 | |
| Seedream 5.0 | Real | 91.88 | 77.16 | 84.62 | 78.83 | 83.37 | 82.93 |
| Synthetic | 90.78 | 76.37 | 87.53 | 75.76 | 88.82 | 83.16 |
Table 8. Paired input-source experiment on 100 cases. Scores are percentages under Gemini-3-Pro.
Overall increases by 0.43 points for Nano Banana Pro and 0.23 for Seedream 5.0; the gap changes from 4.08 to 4.28 points. Dimension-level changes are mixed. Both models show small Overall gains, with the performance gap changing by 0.20 points.
Repeated Runs on Multi-Element Extraction
We repeat generation and judging independently three times on all 101 Multi-Element Extraction cases. Table 9 reproduces the reported per-run scores, means, and standard deviations. Nano Banana Pro and Nano Banana 2 lead in all three runs; Seedream 4.0 and Seedream 5.0 exchange positions in the third run.
| Model | Run 1 | Run 2 | Run 3 | Mean | Std |
| Nano Banana Pro | 87.2 | 86.8 | 84.7 | 86.2 | 1.37 |
| Nano Banana 2 | 85.6 | 85.7 | 83.7 | 85.0 | 1.12 |
| Seedream 4.0 | 83.1 | 83.2 | 83.2 | 83.2 | 0.05 |
| Seedream 5.0 | 81.3 | 82.5 | 84.3 | 82.7 | 1.51 |
Table 9. Reported Multi-Element Extraction scores for three independent generation and judge runs, in percent.
Limitations
Although MDMR-Bench covers diverse multi-reference generation settings, constructing realistic and semantically coherent multi-reference cases remains challenging for certain tasks, especially reasoning-intensive and infographic-oriented scenarios. As a result, some challenge settings may still have limited coverage compared with common entity fusion and attribute transfer tasks.
In addition, except for Entity Fusion, Attribute Transfer, and Analogical Reasoning, most subsets in MDMR-Bench are constructed using only two reference images. This design choice is intentional: rather than maximizing the number of reference inputs, we focus on evaluating the atomic capabilities of current models, such as fine-grained attribute extraction, symbolic editing, and reasoning-aware generation, under controlled settings.
The benchmark construction pipeline also relies on substantial manual filtering. Current multimodal models still make mistakes when verifying subtle visual differences, distinguishing source-specific details across multiple references, or assessing fine-grained attributes. Manual filtering improves the reliability and authenticity of the benchmark, but also limits the scalability of dataset construction.
Finally, although our checklist-based evaluation improves evaluation consistency and interpretability, automatic evaluation cannot fully replace human judgment. Certain aspects of image quality, such as artistic preference, creativity, and cultural nuance, remain difficult to evaluate using deterministic yes/no questions alone.
Compute Resources
MDMR-Bench primarily evaluates model inference rather than training new generative models. The local computational cost therefore comes mainly from generating outputs for open-source baselines and organizing the benchmark artifacts. For open-source models, inference was conducted on GPUs with 80GB of memory to support high-resolution image generation and editing. The exact memory footprint varies across models because the evaluated systems differ in parameter count, precision, and reference-image handling. For proprietary generation models, we accessed the official APIs provided by the model developers; the underlying hardware, memory allocation, and serving details are not disclosed by the providers.
Checklist generation and checklist-based judging were also performed through API calls to the judge model rather than through locally hosted judge inference. As a result, these two stages do not require additional local GPU resources; their practical cost is dominated by API latency, rate limits, and the number of multimodal requests. The evaluation pipeline is embarrassingly parallel across benchmark samples and models. In practice, wall-clock runtime can be reduced by distributing open-source model inference across multiple GPU workers and distributing API-based stages across available API workers.
The benchmark itself does not require model training or hyperparameter search. Reproducing the main evaluation requires storing the benchmark images, generated model outputs, checklist files, and judge responses. We will release scripts that record model identifiers, prompts, input image order, generation parameters when available, and judge outputs, so that future users can audit both the generated images and the checklist-level decisions.
Broader Impacts
MDMR-Bench can have positive societal impact by improving the transparency and reliability of multimodal image generation systems. Multi-reference image generation is increasingly used in design, education, advertising, entertainment, and personal creative workflows. A benchmark that diagnoses whether models preserve identities, transfer attributes correctly, follow instructions, render text accurately, and compose references coherently can help developers understand failure modes before deploying such systems in user-facing settings. More granular evaluation can also reduce over-reliance on single holistic scores, which may hide important errors.
The benchmark may also benefit the research community by making model capabilities easier to compare. Because MDMR-Bench decomposes evaluation into interpretable checklist items, researchers can identify whether a model fails due to reference neglect, poor text rendering, inconsistent style transfer, weak visual reasoning, or low-level artifacts. This can guide safer and more targeted model improvement, and can help users choose appropriate systems for different creative tasks.
At the same time, better evaluation of image generation models can indirectly accelerate progress in systems that may be misused. More capable multi-reference generation could be used to create misleading media, impersonation content, deceptive advertisements, or other synthetic images that are difficult for viewers to verify. Our work does not introduce a new generative model, and it does not provide tools for bypassing safety filters. Nevertheless, evaluation benchmarks can influence model development, so we encourage responsible use of MDMR-Bench for research, transparency, and safety-oriented model assessment.
Potential mitigations include responsible dataset release, clear documentation of intended use, exclusion of unsafe or inappropriate content, and encouraging model providers to pair improved generation capabilities with provenance tools, watermarking, content moderation, and misuse monitoring. We also encourage future work to study fairness and cultural coverage in multi-reference generation, since reference-conditioned tasks may behave differently across identities, languages, artistic traditions, and visual domains.
New Asset Documentation
MDMR-Bench introduces a new benchmark asset consisting of task metadata, input images, natural-language instructions, generated checklist items, and evaluation annotations. Each sample records the task family, subtask setting, image order, image roles, content types, and other metadata. This documentation is intended to make the benchmark auditable and to reduce ambiguity in multi-reference settings, where the ordering and role of input images are critical for reproducible evaluation.
The released benchmark will include data-format documentation, field descriptions, and scripts for running checklist generation and checklist-based evaluation. We will also document known limitations, including the reliance on manual filtering, the scope of task categories, and the fact that proprietary model outputs may depend on API versions and availability. The dataset is intended for academic evaluation and diagnostic analysis of image generation systems, not for identity verification, surveillance, or deployment as an automated moderation tool.
Before release, we apply filtering to remove unsafe, inappropriate, or clearly low-quality samples. The benchmark is constructed to avoid collecting private user data, and the annotation process is conducted internally by the authors rather than through public crowdsourcing. We will provide the benchmark with usage guidelines and license information appropriate for research use.
The code is publicly available at GitHub, and the benchmark is publicly available at Hugging Face.
Prompt Templates for Checklist Generation and Evaluation
MDMR-Bench follows a two-stage checklist-based evaluation protocol. First, a checklist generation prompt asks a multimodal judge to produce atomic, visually verifiable questions for each benchmark sample. Second, an evaluation prompt asks the judge to answer each checklist item for a model output. This design follows the same motivation as binary checklist evaluation: complex multi-reference requirements are decomposed into independent yes/no judgments, making model failures easier to localize.
Checklist Generation Prompt
Task: Unified Checklist Generation for Multi-Reference Image Generation / Editing Benchmark
Objective:
You are an expert evaluator designing checklist-based metrics for a multi-reference image generation benchmark.
Given one benchmark sample, generate fine-grained yes/no checklist items that can later be used to evaluate a generated image.
The benchmark includes both editing-like tasks and non-editing multi-reference generation tasks.
In editing-like tasks, one content image is edited using information from reference images.
In non-editing tasks, all input images are references, and the model must extract required elements, attributes, styles, or semantics from them to generate a new image.
Generate objective, visually verifiable checklist items under four dimensions:
following
consistency
composition
minimality
quality
Scoring Direction:
Every checklist item must be phrased so that answer "Yes" means this item is satisfied and receives credit.
Answer "No" means this item is not satisfied and receives no credit.
Do not write negatively phrased items where "Yes" indicates failure.
Input JSON fields:
task_idin {fusion, attribute_transfer, infographic_manipulation, multiple_extraction, reasoning}subtask_idinstruction_zhinstruction_enclear_instruction(for reasoning task only)num_imagesimages: list with {order, image_name, role, content_type}role=content: image to preserve/editrole=reference: source image to extract details fromrolemay not be accurate.if all
role=reference: this is non-editing multi-reference generation
Dimension guidance:
following: instruction execution completeness, no missing required operation
consistency: elements and attributes must match their intended source images (content or reference); no source confusion or attribute mismatch
composition: multi-source integration should be coherent; spatial relationships, lighting, occlusion, and interactions must be physically and visually plausible
minimality: no introduction of elements not specified in the instruction; no unintended changes to irrelevant regions or objects; no over-editing beyond the intended scope
quality: realism, visual coherence, absence of artifacts, readability, and structural integrity
Construction rules:
Return checklist questions only, no evaluation result.
Each item must be a yes/no question with one atomic criterion.
Ground all items in provided instruction + metadata + attached images.
Use
instruction_zhandinstruction_enjointly to resolve ambiguity.For reasoning tasks, please use
clear_instruction,instruction_zhandinstruction_entogether.Distinguish editing vs all-reference generation via role.
Reference and composition should be detailed for multi-reference tasks.
Avoid subjective questions.
Avoid redundancy.
Checklist question types:
For each dimension, generate a single list of questions:
The first 2 questions MUST be global questions (VERY IMPORTANT)
High-level, holistic yes/no questions summarizing the dimension
The remaining questions MUST be atomic checklist items

- Fine-grained, objective, visually verifiable yes/no questions
Target number of questions (total per dimension):
- following: 4-7
- consistency: 6-12
- composition: 5-10
- minimality: 4-7
- quality: 3-6
The number of questions may be reduced when the task is simple or low-complexity.
Output format:
Return only a valid JSON object with exactly these keys:
{
"following": ["..."],
"consistency": ["..."],
"composition": ["..."],
"minimality": ["..."],
"quality": ["..."]
}
No markdown, no extra keys, no extra text.Figure 8. Checklist generation prompt used in MDMR-Bench. The prompt asks Gemini-3-Pro to generate global- and atomic-level yes/no questions under five dimensions for each multi-reference generation or editing sample.
Checklist Evaluation Prompt
Task: Unified Checklist-Based Evaluation for Multi-Reference Image Generation /
Editing Benchmark
Objective:
You are an expert evaluator for checklist-based assessment.
Your task is to evaluate whether the output image satisfies each checklist item, and
return a structured Y/N decision with concise, evidence-based reasoning.
You must evaluate strictly based on:
1) the instruction and metadata,
2) the Input images,
3) the output image,
4) the provided checklist items.
Do not generate new checklist items. Only evaluate the given checklist.
Scoring Rule:
- "Y" means the checklist item is satisfied.
- "N" means the checklist item is not satisfied.
- Each decision must be independent and evidence-based.
- Do not use uncertain labels. If uncertain, choose the more conservative answer based
on visible evidence.
Input JSON fields:
- task_id, instruction_zh, instruction_en, num_images, checklist
- checklist: a dict with exactly five keys:
- following: list of checklist items
- consistency: list of checklist items
- composition: list of checklist items
- minimality: list of checklist items
- quality: list of checklist items
Critical constraints:
1) Evaluate checklist items in their original order within each dimension.
2) Output list lengths must exactly match the corresponding checklist list lengths.
3) Reasoning must be concise, specific, and tied to visible evidence.
4) Do not rely on style preference or subjective taste.
5) Do not output extra keys, summaries, markdown, or explanations outside the required
JSON object.
Output format:
Return one valid JSON object with exactly three keys:
{
"Answer Dict": {
"following": ["Y" or "N", ...],
"consistency": ["Y" or "N", ...],
"composition": ["Y" or "N", ...],
"minimality": ["Y" or "N", ...],
"quality": ["Y" or "N", ...]
},
"Reason Dict": {
"following": ["reason for item 1", ...],
"consistency": ["..."],
"composition": ["..."],
"minimality": ["..."],
"quality": ["..."]
}
}
Return JSON only. No markdown fence, no extra text.

Figure 9. Checklist evaluation prompt used in MDMR-Bench. The prompt asks the judge model to answer every checklist item with a binary decision, and concise evidence-based reasoning.
Figure 10 (PDF p. 21). fusion / F2_0647. Input. Image 1. Image 2. BAGEL. DreamOmni2. flux2. GPT-Image-1.5. Nano Banana 2. Nano Banana Pro. OmniGen2. Qwen-Image-edit. Seedream4.0. Seedream5.0. Instruction: Replace the red glove on the man's left hand in Image 1 with the brown fur-trimmed leather gloves shown in Image 2, ensuring the angle and shape are adjusted to fit his hand naturally.
Figure 11 (PDF p. 21). fusion / F3_0028. Input. Image 1. Image 2. Image 3. BAGEL. DreamOmni2. flux2. GPT-Image-1.5. Nano Banana 2. Nano Banana Pro. OmniGen2. Qwen-Image-edit. Seedream4.0. Seedream5.0. Instruction: Place the woman from Image 1 into the bathroom scene of Image 3, and equip her with the suede boots shown in Image 2. Ensure natural lighting, accurate attribute transfer of the boots, and realistic proportions within the complex environment.
Qualitative Visualization Results
Visualization Set 1

Figure 1 (PDF p. 22). fusion / F3_0233
Input
Image 1 Image 2
Image 3
BAGEL DreamOmni2 flux2 GPT-Image-1.5 Nano Banana 2
Nano Banana Pro OmniGen2 Qwen-Image-edit Seedream4.0 Seedream5.0
Instruction
Place the chair from Image 3 on the grass in the foreground of Image 2. Then, extract the girl from Image 1 and adjust her pose so that she is sitting on the chair, blending into the natural scene with the giraffes.

Figure 2 (PDF p. 22). fusion / F4_0002
Input
Image 1 Image 2
Image 3 Image 4
BAGEL DreamOmni2 flux2 GPT-Image-1.5 Nano Banana 2
Nano Banana Pro OmniGen2 Qwen-Image-edit Seedream4.0 Seedream5.0
Instruction
The person in Image 1 is carrying the backpack from Image 2 on one shoulder, wearing the pants from Image 3, and standing indoors. There are no other objects in the room, only the chair from Image 4.

Figure 12. multiple_extraction / ME_0041. Input: Image 1; Image 2. Models: BAGEL; DreamOmni2; flux2; GPT-Image-1.5; Nano Banana 2; Nano Banana Pro; OmniGen2; Qwen-Image-edit; Seedream4.0; Seedream5.0. Instruction: Replace the cup on the left side of Image 2 with the one held by the leftmost person in Image 1, and replace the cup on the right side of Image 2 with the one held by the rightmost person in Image 1.

Figure 13. multiple_extraction / ME_0098. Input: Image 1; Image 2. Models: BAGEL; DreamOmni2; flux2; GPT-Image-1.5; Nano Banana 2; Nano Banana Pro; OmniGen2; Qwen-Image-edit; Seedream4.0; Seedream5.0. Instruction: Replace the material of the two vases in Image 1 with the material of the carpet in Image 2, and adjust the color of the bag in Image 1 to match the color of the plant in Image 2.

Figure 14. reasoning / AR_0010. Input: Image 1; Image 2; Image 3. Models: BAGEL; DreamOmni2; flux2; GPT-Image-1.5; Nano Banana 2; Nano Banana Pro; OmniGen2; Qwen-Image-edit; Seedream4.0; Seedream5.0. Instruction: Analyze the visual transformation applied from Image 1 to Image 2, and apply this exact same environmental effect to the scene in Image 3.

Figure LR_00014. reasoning / LR_00014. Input: Image 1; Image 2. BAGEL; DreamOmni2; flux2; GPT-Image-1.5; Nano Banana 2; Nano Banana Pro; OmniGen2; Qwen-Image-edit; Seedream4.0; Seedream5.0. Instruction: Replace the clothing of the person in Image 2 with attire that corresponds to the historical period in which the poem in Image 1 was created.

Figure LR_00029. reasoning / LR_00029. Input: Image 1; Image 2. BAGEL; DreamOmni2; flux2; GPT-Image-1.5; Nano Banana 2; Nano Banana Pro; OmniGen2; Qwen-Image-edit; Seedream4.0; Seedream5.0. Instruction: Reposition the three people in Image 2 so that their spatial arrangement follows the same height order as the three individuals in Image 1.

Figure C1S1_0376. style_transfer / C1S1_0376. Input: Image 1; Image 2. BAGEL; DreamOmni2; flux2; GPT-Image-1.5; Nano Banana 2; Nano Banana Pro; OmniGen2; Qwen-Image-edit; Seedream4.0; Seedream5.0. Instruction: Please refer to the visual style of Image 1 and transform the plant and table in Image 2 into this effect.

Figure 15 (PDF p. 25). style_transfer / C1S1_0515. Input: Image 1, Image 2. BAGEL, DreamOmni2, flux2, GPT-Image-1.5, Nano Banana 2, Nano Banana Pro, OmniGen2, Qwen-Image-edit, Seedream4.0, Seedream5.0. Instruction: Referring to the material of the grenade in Image 1, change the two stone structures by the pool in Image 2 to have the same material.

Figure 16 (PDF p. 25). style_transfer / C2S1_0068. Input: Image 1, Image 2, Image 3. BAGEL, DreamOmni2, flux2, GPT-Image-1.5, Nano Banana 2, Nano Banana Pro, OmniGen2, Qwen-Image-edit, Seedream4.0, Seedream5.0. Instruction: Please extract the woman from Image 1 and place her on the stone path in Image 2 to make her the main subject of the scene. After that, modify the pose of the woman in the merged image to match the pose of the woman shown in Image 3.

Figure 17 (PDF p. 25). text_logo / TCN_0001. Input: Image 1, Image 2. BAGEL, DreamOmni2, flux2, GPT-Image-1.5, Nano Banana 2, Nano Banana Pro, OmniGen2, Qwen-Image-edit, Seedream4.0, Seedream5.0. Instruction: Place the object from Image 2 into the sector of Image 1 labeled “脂质”, positioning it below the words “脂质”.

Figure 18 (PDF p. 26).

Figure 19 (PDF p. 26).
Figure 27.
fusion / F3_0084
Nano Banana 2 vs DreamOmni2
Input Images
Image 1 | Image 2 | Image 3
Nano Banana 2 | DreamOmni2
Instruction
Extract the woman from Image 1 and the man from Image 2, and composite them together into the coastal living room scene of Image 3. Ensure natural coexistence, realistic lighting, and appropriate scaling within the room, while retaining the 'Bay Collection' text prominently at the top of the composition.
| Dimension | Selected Question | Nano Banana 2 Answer | DreamOmni2 Answer |
| Following | Does the final image successfully combine the woman from Image 1, the man from Image 2, and the living room background from Image 3? | The final image successfully combines the woman, the man, and the coastal living room background. | The final image combines the woman and man, but completely replaces the living room background from Image 3 with a new one. |
| Consistency | Are the identities, features, and clothing of both inserted individuals visually consistent with their respective source images? | The identities, features, and clothing of both subjects visually match their source images accurately. | The identities and features match, but the man's jacket color has changed to green. |
| Composition | Is the overall composition visually cohesive, integrating the two inserted subjects seamlessly into the living room scene? | The overall composition is visually cohesive and integrates the subjects seamlessly into the scene. | The subjects are integrated cohesively into the generated living room scene. |
| Quality | Is the generated image of high overall visual quality, appearing completely realistic and free of obvious generative modeling artifacts? | The image is of high overall visual quality but contains a noticeable generative artifact on the woman's hand holding the fork. | The image has obvious generative artifacts, such as the deformed object in the man's hand and gibberish text on the pillow. |
| No unintended modification | Did the model preserve the exact original state of the unedited interior areas of the living room from Image 3 without introducing unrequested alterations? | The unedited interior areas of the living room are preserved without unrequested alterations. | The entire interior area of the living room from Image 3 was replaced. |
Table 27.
Qualitative Visualization Results
multiple_extraction / ME_0020
Figure 28a.
Input Images
Image 1
Image 2
Nano Banana
Seedream4.0
Instruction
Place the rabbit from Image 1 in the little girl’s arms in Image 2, with her gently petting it. Then position the cat from Image 1 on the edge of the wooden box on the right side of Image 2.
| Dimension | Selected Question | Nano Banana Answer | Seedream4.0 Answer |
| Following | Does the generated image successfully integrate both the rabbit and the cat from Image 1 into the scene from Image 2 as instructed? | The rabbit is successfully integrated, but the cat is placed on the armchair instead of the wooden box. | Both the rabbit and the cat from Image 1 are successfully integrated into the scene from Image 2. |
| Consistency | Do the inserted rabbit and cat accurately retain their unique visual identities and characteristics from Image 1? | Both the inserted rabbit and cat accurately retain their visual identities from Image 1. | The rabbit lost its distinctive long ears, and the cat's face and age appear slightly altered compared to Image 1. |
| Composition | Are the newly added animals naturally integrated into the scene with believable scale, positioning, and physical grounding? | The rabbit is naturally integrated, but the cat appears to be floating unnaturally off the edge of the armrest without proper physical grounding. | The animals are scaled reasonably and grounded, though the cat's back paw appears to float slightly. |
| Quality | Is the overall generated image of high visual quality, appearing completely realistic without obvious signs of digital manipulation? | The image suffers from obvious signs of digital manipulation, particularly the distorted hands of the girl and the awkward placement of the cat. | The image suffers from obvious digital artifacts, particularly in the rendering of the girl's left hand and the beads. |
| No unintended modification | Does the image successfully avoid introducing any unrecognized objects, animals, or elements not explicitly specified in the instructions? | The image successfully avoids introducing any unrecognized objects or elements. | No unrequested objects, animals, or elements were introduced into the scene. |
Table 28.
Figure 1 (PDF p. 29).
reasoning / AR_0006
Seedream5.0 vs Qwen-Image-edit
Input Images
Image 1 Image 2
Image 3
Seedream5.0 Qwen-Image-edit
Instruction
Apply the seasonal transformation observed between Image 1 and Image 2 to the scene in Image 3.
| Dimension | Selected Question | Seedream5.0 Answer | Qwen-Image-edit Answer |
| Following | Does the generated image successfully apply the seasonal transformation inferred from the reference images (transitioning to a winter scene) to the garden scene in Image 3? | The generated image successfully applies the seasonal transformation, changing the summer garden into a winter scene. | The generated image does not apply the seasonal transformation; it remains a summer scene. |
| Consistency | Are the structural layout and primary elements of the garden from Image 3 (e.g., path, rock formations, background house) clearly preserved and recognizable in the generated image? | The structural layout of the garden, including the path, rock formations, and background house, is clearly preserved. | The structural layout, including the path, rock formations, and background house, is clearly preserved. |
| Composition | Is the integration of the winter elements (e.g., snow on vegetation and structures) into the garden scene physically plausible and visually coherent? | The integration of winter elements into the garden scene is physically plausible and visually coherent. | There are no winter elements integrated into the garden scene. |
| Quality | Does the generated image exhibit high visual quality, appearing as a realistic and convincing photograph of a winter garden? | The generated image exhibits high visual quality, appearing as a realistic and convincing photograph. | The image does not appear as a photograph of a winter garden. |
| No unintended modification | Has the model strictly altered only the seasonal aspects of the scene without introducing unrelated objects or elements? | The model strictly altered only the seasonal aspects without introducing unrelated objects. | The model introduced the couple and blanket instead of altering the seasonal aspects. |
Table 1 (PDF p. 29).
style_transfer / C1S1_0578
Nano Banana 2 vs DreamOmni2
Input Images
Image 1
Image 2
Instruction
Please refer to the action of the girl in Image 1 and modify the posture of the woman in Image 2.
| Dimension | Selected Question | Nano Banana 2 Answer | DreamOmni2 Answer |
| Following | Has the posture of the woman from Image 2 been successfully modified based on the instruction? | The woman's posture has been successfully modified to match the girl's action. | The generated image shows the girl from Image 1, completely omitting the woman from Image 2. |
| Consistency | Is the identity and overall appearance of the woman from Image 2 preserved in the generated image? | The woman retains her overall identity and facial appearance. | The identity of the woman is entirely replaced by the girl from Image 1. |
| Composition | Does the woman look naturally integrated and physically balanced in her new pose? | The woman looks physically balanced and naturally integrated in her new stance. | The woman is not present in the image. |
| Quality | Is the overall visual quality of the generated image high, resembling a professional photograph? | The overall visual quality is high and resembles a professional photograph. | The image has decent lighting and resolution, resembling a photograph, despite failing the prompt. |
| No unintended modification | Are the background and overall setting free from unintended alterations? | The background and overall setting remain free from unintended alterations. | The subject was completely changed, which is a massive unintended alteration. |
caption:
:::
Figure 1 (PDF p. 31).
Instruction Transform the style of "HELPDESK" in Image 1 to match the style of "personality" in Image 2.
| Dimension | Selected Question | flux2 Answer | BAGEL Answer |
| Following | Did the model successfully alter the visual style of the text "HELPDESK" in Image 1 according to the instruction? | The model replaced the text with a new word entirely instead of just altering the style of 'HELPDESK'. | The model failed to retain the word 'HELPDESK', replacing it with illegible text. |
| Consistency | Is the overall content and structure of Image 1 consistently preserved in the generated image? | The silhouettes, speech bubbles, and reflections are consistently preserved. | All human silhouettes and speech bubbles from Image 1 are completely missing. |
| Composition | Is the overall composition and spatial layout of the generated image visually logical and coherent? | The overall spatial layout is logical and visually coherent. | The overall composition is lost because all graphical elements from Image 1 have been deleted. |
| Quality | Is the overall aesthetic quality and visual fidelity of the generated image high? | The overall aesthetic quality and visual fidelity are high. | The image consists only of jagged, illegible text on a white background, lacking high aesthetic quality. |
| No unintended modification | Are all regions and graphical elements of Image 1 unrelated to the text "HELPDESK" left entirely unmodified? | All regions and elements unrelated to the text are left unmodified. | All unrelated graphical elements from Image 1 (silhouettes, bubbles) have been unintendedly deleted. |
Table 1 (PDF p. 31).
References
- [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
- [2]Black Forest Labs. Flux.1-kontext-dev. 2025. https://blackforestlabs.ai/flux-1-kontext/.
- [3]Black Forest Labs. FLUX.2: Frontier Visual Intelligence. https://bfl.ai/blog/flux-2, November 2025. Accessed: 2026-04-30.
- [4]ByteDance Seed. Seed1.8. https://seed.bytedance.com/en/seed1_8, 2025. Accessed: 2026-05-07.
- [5]ByteDance Seed. Seedream 5.0 Lite. https://seed.bytedance.com/en/seedream5_0_lite, 2026. Accessed: 2026-04-30.
- [6]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.arxiv.org/abs/2104.14294
- [7]Jiuhai Chen, Zhi yang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip-3: A family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505.09568, 2025.arxiv.org/abs/2505.09568
- [8]Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. arXiv preprint arXiv:2310.18235, 2023.arxiv.org/abs/2310.18235
- [9]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.arxiv.org/abs/2507.06261
- [10]Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025.arxiv.org/abs/2505.14683
- [11]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024.arxiv.org/abs/2403.03206
- [12]Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152, 2023.arxiv.org/abs/2310.11513
- [13]Google DeepMind. Gemini 3 pro image (nano banana pro). https://blog.google/technology/ai/nano-banana-pro, November 2025. Accessed: 2026-04-30.
- [14]Google DeepMind. Gemini 3.1 flash image (nano banana 2). https://blog.google/innovation-and-ai/technology/ai/nano-banana-2, February 2026. Accessed: 2026-04-30.
- [15]Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 7514–7528, 2021.arxiv.org/abs/2104.08718
- [16]Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024.arxiv.org/abs/2403.05135
- [17]Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897, 2023.arxiv.org/abs/2303.11897
- [18]Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8362–8371, 2024.arxiv.org/abs/2312.06739
- [19]Zhangqi Jiang, Zheng Sun, Xianfang Zeng, Yufeng Yang, Xuanyang Zhang, Yongliang Wu, Wei Cheng, Gang Yu, Xu Yang, and Bihan Wen. Getditbench v2: A human-aligned benchmark for general image editing, 2026.
- [20]Amita Kamath, Kai-Wei Chang, Ranjay Krishna, Luke Zettlemoyer, Yushi Hu, and Marjan Ghazvininejad. Geneval 2: Addressing benchmark drift in text-to-image evaluation. arXiv preprint arXiv:2512.16853, 2025.arxiv.org/abs/2512.16853
- [21]Klaus Krippendorff. Bivariate agreement coefficients for reliability data. In Edgar F. Borgatta and George W. Bohrnstedt, editors, Sociological Methodology 1970, volume 2, pages 139–150. Jossey-Bass, San Francisco, CA, 1970.DOI
- [22]Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viscore: Towards explainable metrics for conditional image synthesis evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12268–12290, 2024.arxiv.org/abs/2312.14867
- [23]Black Forest Labs. Flux. https://github.com/black-forest-labs/flux, 2024.
- [24]Qixiang Li, Yuan Wang, Xinting Hu, Huijuan Huang, Rui Chen, Jiarong Ou, Xin Tao, Pengfei Wan, Xiaojun Qi, and Fuli Feng. Easier painting than thinking: Can text-to-image models set the stage, but not direct the play? arXiv preprint arXiv:2509.03516, 2025.arxiv.org/abs/2509.03516
- [25]Xiaoye Liang, Zhiyuan Qu, Mingye Zou, Jiaxin Liu, Lai Jiang, Mai Xu, and Yiheng Zhu. Vtedit-bench: A comprehensive benchmark for multi-reference image editing models in virtual try-on. arXiv preprint arXiv:2603.11734, 2026.arxiv.org/abs/2603.11734
- [26]Jintao Lin, Bowen Dong, Weikang Shi, Chenyang Lei, Suiyun Zhang, Rui Liu, and Xihui Liu. Aegis: Exploring the limit of world knowledge capabilities for unified multimodal models. arXiv preprint arXiv:2601.00561, 2026.arxiv.org/abs/2601.00561
- [27]Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, Guopeng Li, Yuang Peng, Quan Sun, Jingwei Wu, Yan Cai, Zheng Ge, Ranchen Ming, Lei Xia, Xianfang Zeng, Yibo Zhu, Binxing Jiao, Xiangyu Zhang, Gang Yu, and Daxin Jiang. Step1x-edit: A practical framework for general image editing. arXiv preprint arXiv:2504.17761, 2025.arxiv.org/abs/2504.17761
- [28]Yiwei Ma, Jiayi Ji, Ke Ye, Weihuang Lin, Zhibin Wang, Yonghan Zheng, Qiang Zhou, Xiaoshuai Sun, and Rongrong Ji. I2ebench: A comprehensive benchmark for instruction-based image editing. Advances in Neural Information Processing Systems, 37:41494–41516, 2024.arxiv.org/abs/2408.14180
- [29]OpenAI. GPT-Image-1 Model. https://platform.openai.com/docs/models/gpt-image-1, 2025. Accessed: 2025-10-16.
- [30]OpenAI. GPT-Image-1.5 Model. https://developers.openai.com/api/docs/models/gpt-image-1.5, December 2025. Accessed: 2025-12-16.
- [31]OpenAI. GPT-5.4 Model. https://developers.openai.com/api/docs/models/gpt-5.4, March 2026. Accessed: 2026-05-07.
- [32]OpenAI. GPT-Image-2 Model. https://developers.openai.com/api/docs/models/gpt-image-2, April 2026. Accessed: 2026-05-27.
- [33]Yuta Oshima, Daiki Miyake, Kohsei Matsutani, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo, and Hiroki Furuta. Multibanana: A challenging benchmark for multi-reference text-to-image generation. arXiv preprint arXiv:2511.22989, 2025.arxiv.org/abs/2511.22989
- [34]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022.arxiv.org/abs/2112.10752
- [35]Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500–22510, 2023.arxiv.org/abs/2208.12242
- [36]Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Wang, Emily L Denton, Kamyar Ghasempour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022.arxiv.org/abs/2205.11487
- [37]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2022), 2022.arxiv.org/abs/2210.08402
- [38]Team Seedream, Yunpeng Chen, Yu Gao, Lixue Gong, Meng Guo, Qiushan Guo, Zhiyao Guo, Xiaoxia Hou, Weilin Huang, Yixuan Huang, et al. Seedream 4.0: Toward next-generation multimodal image generation. arXiv preprint arXiv:2509.20427, 2025.
- [39]Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8871–8879, 2024.arxiv.org/abs/2311.10089
- [40]Kaiyue Sun, Rongyao Fang, Chengqi Duan, Xian Liu, and Xihui Liu. T2i-reasonbench: Benchmarking reasoning-informed text-to-image generation. arXiv preprint arXiv:2508.17472, 2025.arxiv.org/abs/2508.17472
- [41]Meituan LongCat Team, Hanghang Ma, Haoxian Tan, Jiale Huang, Junqiang Wu, Jun-Yan He, Lishuai Gao, Songlin Xiao, Xiaoming Wei, Xiaoqi Ma, Xunliang Cai, Yayong Guan, and Jie Hu. Longcat-image technical report. arXiv preprint arXiv:2512.07584, 2025.
- [42]Super Intelligence Team. Firered-image-edit: A general-purpose image editing model. 2026.
- [43]Z-Image Team. Z-image: An efficient image generation foundation model with single-stream diffusion transformer. arXiv preprint arXiv:2511.22699, 2025.arxiv.org/abs/2511.22699
- [44]Peng Wang, Yichun Shi, Xiaochen Lian, Zhonghua Zhai, Xin Xia, Xuefeng Xiao, Weilin Huang, and Jianchao Yang. Seededit 3.0: Fast and high-quality generative image editing, 2025.arxiv.org/abs/2506.05083
- [45]Xinyu Wei, Kangrui Cen, Hongyang Wei, Zhen Guo, Bairui Li, Zeqing Wang, Jinrui Zhang, and Lei Zhang. MIC-150k: A comprehensive dataset advancing multi-image composition. arXiv preprint arXiv:2512.07348, 2025.arxiv.org/abs/2512.07348
- [46]Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang, Yongqiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. Qwen-image technical report, 2025.arxiv.org/abs/2508.02324
- [47]Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng, Jiahao Wang, Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, and Zheng Liu. Omnigen2: Exploration to advanced multimodal generation. arXiv preprint arXiv:2506.18871, 2025.
- [48]Bin Xia, Bohao Peng, Yuechen Zhang, Junjia Huang, Jiyang Liu, Jingyao Li, Haoru Tan, Sitong Wu, Chengyao Wang, Yitong Wang, et al. Dreamomni2: Multimodal instruction-based editing and generation. arXiv preprint arXiv:2510.06679, 2025.arxiv.org/abs/2510.06679
- [49]Bin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, and Jiaya Jia. Dreamomni: Unified image generation and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28533–28543, June 2025.arxiv.org/abs/2412.17098
- [50]Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. arXiv preprint arXiv:2310.08580, 2023.arxiv.org/abs/2310.08580
- [51]Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. arXiv preprint arXiv:2305.10400, 2023.arxiv.org/abs/2305.10400
- [52]Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan. Imgedit: A unified image editing dataset and benchmark. arXiv preprint arXiv:2505.20275, 2025.arxiv.org/abs/2505.20275
- [53]Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea, 2025.arxiv.org/abs/2411.15738
- [54]Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems, 36:31428–31449, 2023.arxiv.org/abs/2306.10012










