betaFeedback

WorksComputer Science

Structural Self-Teaching for Compositional Generation in Unified Multimodal Models

Abstract

We propose Structural Self-Teaching (SST), a novel framework that enhances compositional image generation in diffusion-based unified multimodal models (UMMs) by leveraging their internal understanding path as a source of structural supervision. The generation path of UMMs often struggles with dense compositional prompts, leading to missing objects, attribute binding failures, and spatial errors. In contrast, their understanding path localizes the same phrases in images that satisfy these prompts. SST converts this phrase-level grounding into training-time structural signals for the generation path. Specifically, we extract phrase-conditioned support masks from internal attention maps derived from the understanding path and employ them through two mechanisms. Phrase-local contrastive alignment uses these masks as pooling regions and aligns each generated phrase region with the same region of the target image. A structural scaffold builds a phrase-level layout from the same masks and modulates an intermediate spatial state of the generation path during training. The scaffold is phased out during training and removed at inference. SST requires no external structural annotations for training and no additional input at test time. Extensive experiments across multiple UMMs demonstrate that SST improves compositional fidelity over LoRA fine-tuning on the same data, with no inference-time cost. Project page is available at https://bellos1203.github.io/sst

Introduction

Unified multimodal models (UMMs) integrate image understanding and generation within a single architecture, and recent systems have shown strong perception performance together with competitive generation quality [1, 2, 3, 4, 5, 6]. Despite recent progress, UMMs still struggle to generate images that faithfully satisfy dense compositional prompts. Given prompts that specify multiple objects, attributes, and relations, state-of-the-art UMMs often miscount instances, swap attributes across entities, and drop referenced phrases entirely. These failures intensify as prompts become compositionally denser, motivating training methods that explicitly account for phrase-level structure.

UMMs also provide an internal source of phrase-level structure through their understanding path. Given a training image and its caption, attention between caption and image tokens in the understanding path often localizes individual noun phrases, such as “three flowers,” “a checkered car,” or “a chair,” in spatially coherent image regions (see Figure 1), consistent with prior observations in vision-language models [7, 8]. These phrase-conditioned attention maps provide soft spatial supports over visual tokens. § 3 evaluates whether such supports remain usable under increasing compositional density. We use this internal grounding as a source of structural supervision during generation training, without relying on external annotations.

Figure 1. Unified multimodal models can localize phrases they fail to generate. Given an image-text pair that satisfies a compositional prompt, the understanding path produces phrase-wise attention maps that localize the requested concepts. However, generation from the same prompt can omit objects or entangle attributes, as shown by the base UMM. Structural self-teaching transfers these internal supports during training, improving phrase coverage and compositional fidelity.

To expose this internal structure to the generation path, we propose structural self-teaching for diffusion-based UMMs, a framework that turns the understanding path into an internal structural teacher for its generation path. The central idea is to extract phrase-level grounding from the understanding path and convert it into structural signals for training the generation pathway. All structural cues are derived from the model itself, requiring neither external structural annotations nor inference-time intervention.

Phrase-level structure reaches the generation path through two routes. The first route is phrase-local contrastive alignment. It uses phrase-conditioned support masks as pooling regions for representation learning. It pulls each phrase-specific region in the generated representation toward the corresponding region in the target representation, while pushing it away from other phrase-specific regions in the same prompt. The second is a structural scaffold, which converts the same support masks into a dense phrase-level canvas and modulates intermediate spatial states during training. Scaffold dropout prevents reliance on the canvas. The scaffold branch is removed at inference. Together, the two mechanisms transfer understanding-side structure through both phrase-level representation alignment and spatial modulation during denoising.

The main contributions of our paper are summarized as follows:

  • We propose structural self-teaching, a training framework that extracts phrase-conditioned support masks from the understanding path of a UMM and reuses them to supervise its generation path without external structural annotations or test-time inputs.

  • We introduce phrase-local contrastive alignment, which leverages the extracted masks to align phrase-specific generated and reference visual representations while preventing semantic collapse among distinct phrase-level features within the same prompt.

  • We develop an attenuated structural scaffold that converts these supports into a dense phrase-level canvas for training-time modulation, with a keep probability that decays over training.

  • The proposed approach demonstrates consistent improvements in compositional fidelity across multiple UMMs, effectively addressing complex object, attribute, and spatial constraints without modifying the inference-time interface.

The rest of the paper is organized as follows. § 2 reviews the related work, followed by our motivation in § 3. The details of our approach are described in § 4, and the experimental results are presented in § 5. Finally, we conclude this paper in § 6.

Related Works

Unified Multimodal Models

Unified multimodal models (UMMs) integrate visual understanding and generation within a single backbone, and generate images either autoregressively over discrete tokens [2, 3, 4, 9] or through a diffusion process [1, 10, 5, 6]. Despite competitive performance on both pathways, UMMs exhibit a persistent asymmetry in which they understand more reliably than they generate, prompting a body of post-training work aimed at closing this gap. HermesFlow [11] first quantifies this asymmetry and proposes preference optimization (Pair-DPO) using self-curated preference pairs from both pathways, aligning the two capabilities at the prompt level. SRUM [12] uses the model’s own understanding module to derive global and local rewards for reward-weighted generation training. RecA [13] instead conditions the UMM on its own understanding embeddings and reconstructs the input image, providing dense pixel-level supervision without captions. Our method also leverages internal signals from the understanding path, but targets compositional structure rather than reward-based post-training or pixel-level reconstruction: we transfer phrase-conditioned spatial supports through a phrase-local contrastive objective, directly addressing the entanglement and omission failures that emerge as compositional complexity grows.

Visual Grounding and Attention in MLLMs

Multimodal transformers encode useful structural signals in their attention patterns [14, 7, 15, 8]. In multimodal encoders, word-conditional attention localizes caption words well enough to guide representation learning without region annotations [7]. In frozen LVLMs, Kang et al. [8] show that a small subset of attention heads acts as localization heads and yield strong visual grounding without grounding-specific training, indicating that phrase-conditioned structure already emerges inside the model. At the same time, Visual Attention Sink [16] observes that a few uninformative tokens absorb disproportionate attention mass, meaning raw maps mix signal with systematic artifacts. The practical question is therefore how to extract this structure selectively. We adopt head scoring and sink suppression from previous works [8, 16], and extend them by scoring and selecting heads per phrase rather than globally—since different phrases in the same image are grounded by different heads—and by replacing hard thresholding with soft, temperature-controlled aggregation so that informative sub-threshold maps still contribute to a dense training target.

Compositional Text-to-Image Generation

Even strong generators struggle with compositional prompts, exhibiting missing entities, incorrect attribute binding, unreliable counting, and broken spatial relations—failures that recent benchmarks such as TIFA [17], GenEval [18], and GenEval2 [19] make quantitatively visible. Existing remedies act either during sampling or through added inputs. Prompt-to-Prompt [20] and Attend-and-Excite [21] manipulate cross-attention in each sampling run and leave the trained model unchanged. ControlNet [22] and GLIGEN [23] condition generation on external structure such as edges, depth, or bounding boxes, and LayoutGPT [24] produces such layouts with an LLM planner. These methods require a structural input beyond the prompt, from the user or from a separate planner. Our approach instead regularizes the generation path during training with structure from the model’s own understanding path. It adds no inference-time step and no input beyond the prompt.

Motivation: Using Understanding Path as Structure Supervision

Our framework uses the understanding path of a UMM as an internal source of phrase-level structural supervision. We therefore first examine whether this path provides usable compositional cues under dense prompts, considering two axes. The first is whether prompt-specified visual facts remain measurable when the corresponding image evidence is available, and the second is whether these facts are spatially grounded enough to define phrase-level supports.

We use the atomic questions from GenEval2 [19], which decompose each prompt into object, attribute, count, and spatial facts. Since GenEval2 provides no oracle images, we curate them through model-based generation and editing to satisfy all atomic conditions (see Appendix § 10.§ 3 for details). These oracle images serve only as diagnostic inputs for probing the understanding path, and are not used for training or as a generation benchmark. Given each oracle image and prompt, the understanding path answers the corresponding atomic questions, and we separately report prompt-only generation scores under the GenEval2 protocol to indicate the compositional difficulty of the same prompts. We emphasize that Figure 2 serves as a diagnostic of the source signal used by our method, not as a direct comparison between perception and generation.

Diagnostic plots of understanding-path supervision across four UMM backbones

Figure 2. Diagnostic of the understanding-path supervision source across four UMM backbones. Top row. Prompt-only generation scores (Gen) decrease sharply with atomicity, while understanding-path scores on oracle images (Und) remain comparatively stable. Bottom row. Activation-inside ratio (AIR) and center-of-mass pointing accuracy (Pt) of extracted phrase supports against Grounded-SAM pseudo-references remain within narrow bands across atomicity. The top row is not a direct comparison between Und and Gen, see § 3.

Across BLIP-3o [6], Show-o2 [5], and BAGEL [1], prompt-only generation scores decrease sharply as atomicity increases, dropping to near-zero on the densest prompts (Figure 2, top row). In contrast, understanding-path scores on the corresponding oracle images remain comparatively stable across the same atomicity range, with only mild degradation at high atomicity. This stability holds across all evaluated backbones, despite their differences in architecture and scale. The understanding path therefore retains measurable compositional information even under high atomicity, supporting its use as an internal source of structural supervision.

We next examine the second axis, whether the source signal is spatially grounded and stable across compositional density. We extract phrase-conditioned support masks from the understanding path (following the procedure detailed in § 4.1) and compare them against pseudo-reference masks obtained by running Grounded-SAM [25] on the same noun phrases. Across atomicity levels 3–10, the activation-inside ratio and center-of-mass pointing accuracy remain within narrow bands across all four backbones, with no systematic degradation as compositional density grows (Figure 2, bottom row). Since Grounded-SAM pseudo-references are themselves imperfect, we focus on stability under increasing atomicity rather than on absolute localization accuracy. These supports form spatially coherent regions over visual tokens, which we use as pooling regions for phrase-local alignment and as spatial carriers for a structural scaffold during generation training.

Structural Self-Teaching for Compositional Generation

The proposed Structural Self-Teaching (SST) framework transfers phrase-level structural cues from the understanding path of a UMM to its generation path during training (Figure 3). Given an image and its corresponding caption, we first extract noun phrases and derive phrase-conditioned support masks from internal text-to-image attention maps within the understanding path (§ 4.1). The support masks reach the generation path through two routes. Phrase-local contrastive alignment uses them as pooling regions for a representation-level target (§ 4.2). The structural scaffold uses them as a spatial canvas that modulates an intermediate generation state during training (§ 4.3). Attenuated scaffold dropout phases the scaffold out during training. The scaffold branch is removed at inference, so the original generation interface requires no additional input. The final objective combines the model’s native denoising loss with the phrase-local contrastive loss (§ 4.4).

Figure 3. Overview of structural self-teaching. Phrase-conditioned support masks {Ap}\{A_p\} extracted from the frozen understanding path provide two training-time signals for the LoRA-tuned generation path: phrase-local contrastive alignment between reference and generated phrase representations, and a structural scaffold that modulates an intermediate spatial feature FtℓF_t^\ell. The scaffold is applied with sample-wise cosine-decayed dropout and removed at inference.

Phrase-Conditioned Support Mask Extraction

To construct a reliable teacher signal from the understanding path, we extract a soft spatial support mask for each noun phrase in the caption. Our extraction procedure builds upon the attention-head scoring criteria of [8], originally proposed for zero-shot visual grounding, but we introduce two critical adaptations for compositional generation. First, we perform candidate scoring and selection per phrase rather than identifying single head set for the entire prompt; this reflects the observation that different phrases within the same complex scene are often grounded by distinct sets of heads. Second, we replace rigid threshold-based selection with a soft, temperature-controlled aggregation, ensuring that informative sub-threshold maps still contribute to the final spatial support.

Given an image I\mathcal{I} and its corresponding caption cc, we extract noun phrases {pn}n=1N\{p_n\}_{n=1}^{N} from cc and collect text-to-image attention from every layer-head pair of the understanding path. This produces a candidate pool Ap,ℓ,h∈RMA_{p,\ell,h} \in\mathbb{R}^{M} for each phrase pp and layer-head pair (ℓ,h)(\ell,h), where MM denotes the number of image tokens. Following [16], we preprocess each candidate map to suppress background activations and visual attention sinks, then rank these resulting candidate maps within each phrase based on a combined score sp,ℓ,hs_{p,\ell,h} of relevance and spatial plausibility as:

sp,ℓ,h=sp,ℓ,hsum+sp,ℓ,hent(1)s_{p,\ell,h} = s_{p,\ell,h}^{\mathrm{sum}} + s_{p,\ell,h}^{\mathrm{ent}} \tag*{(1)}

where the relevance term sp,ℓ,hsums_{p,\ell,h}^{\mathrm{sum}} and the spatial plausibility term sp,ℓ,hents_{p,\ell,h}^{\mathrm{ent}} are both normalized within the phrase-specific candidate pool to ensure comparability.

Let Aˉp,k\bar{A}_{p,k} be the kk-th ranked candidate map after the aforementioned normalization. The final phrase-conditioned support mask ApA_p is then constructed by softly aggregating the top-KK candidate maps with temperature-controlled weights:

Ap=Norm⁡(∑k=1Kwp,kAˉp,k),wp,k=exp⁡(sp,k/τ)∑k′=1Kexp⁡(sp,k′/τ)(2)A_p = \operatorname{Norm}\left(\sum_{k=1}^{K} w_{p,k}\bar{A}_{p,k}\right), \qquad w_{p,k} = \frac{\exp(s_{p,k}/\tau)}{\sum_{k'=1}^{K}\exp(s_{p,k'}/\tau)} \tag*{(2)}

where the temperature τ>0\tau> 0 controls the sharpness of the aggregation, and Norm⁡(⋅)\operatorname{Norm}(\cdot) denotes a final sum-to-one renormalization step to ensure the mask values stay within a valid range for the subsequent alignment stage.

The resulting ApA_p serves as a phrase-conditioned soft spatial support, rather than a binary segmentation or instance mask. By reliably providing coarse localization and separating spatial responsibilities across phrases, ApA_p acts as a frozen, structurally-grounded teacher that guides the generation path. Implementation details and exact forms of ssums^{\mathrm{sum}} and sents^{\mathrm{ent}} are provided in Appendix § 9. Example visualizations are shown in Appendix § 12.§ 1.

Phrase-local Contrastive Alignment

We leverage the phrase-conditioned support masks ApA_p as pooling operators to provide representation-level supervision. Given a caption cc, its corresponding ground-truth image II, and a sampled timestep tt, the generation path predicts a denoised latent x^0\hat{x}_0 from xtx_t conditioned on cc. We encode II into reference visual features v∈RM×D\mathbf{v} \in\mathbb{R}^{M \times D}. For each noun phrase pp in the caption, we derive a reference visual representation zpUz_p^U from the target image and a generated visual representation zpGz_p^G from the denoised latent via mask-weighted pooling:

zpU=Pool⁡(v,Ap),zpG=gϕ(Pool⁡(x^0,Ap)),(3)z_p^U = \operatorname{Pool}(\mathbf{v}, A_p), \qquad z_p^G = g_\phi(\operatorname{Pool}(\hat{x}_0, A_p)), \tag*{(3)}

where Pool⁡(⋅,Ap)\operatorname{Pool}(\cdot, A_p) denotes mask-weighted spatial averaging, with ApA_p resized to match the spatial resolution of the respective feature maps. The projection head gϕg_\phi maps the pooled features from the generation path into the visual feature space of v\mathbf{v}. The choice of features and projection is backbone-specific (Appendix § 8.§ 2).

The alignment objective pulls each generated representation toward its own reference and away from the references of other phrases in the same prompt:

Lphrase=−1N∑p=1Nlog⁡exp⁡(sim⁡(zpG,zpU)/τalign)∑q=1Nexp⁡(sim⁡(zpG,zqU)/τalign),(4)\mathcal{L}_{\mathrm{phrase}} = -\frac{1}{N}\sum_{p=1}^{N}\log\frac{\exp(\operatorname{sim}(z_p^G,z_p^U)/\tau_{\mathrm{align}})}{\sum_{q=1}^{N}\exp(\operatorname{sim}(z_p^G,z_q^U)/\tau_{\mathrm{align}})}, \tag*{(4)}

where sim⁡(⋅,⋅)\operatorname{sim}(\cdot,\cdot) denotes cosine similarity and τalign>0\tau_{\mathrm{align}} > 0 is a temperature. The positive term matches the content generated in each phrase region to the corresponding region of the reference image. The in-prompt negatives keep distinct phrases from collapsing onto one representation, which targets the attribute leakage and object fusion observed in dense prompts.

Attenuated Structural Scaffold

The structural scaffold modulates an intermediate spatial state of the generation path with a phrase-level layout during training. The layout assigns each phrase’s reference representation to its support region. Phrase-local contrastive alignment specifies what each phrase region should contain, but it reaches the generation path only through the gradient of a pooled loss. We attenuate the scaffold over training and remove it at inference, so the generation path at inference is unchanged.

Scaffold construction Given the support masks {Ap}p=1N\{A_p\}_{p=1}^{N} and reference representations {zpU}p=1N\{z_p^U\}_{p=1}^{N}, we first resize each mask to the target resolution and normalize it to A~p(u)∈[0,1]\tilde{A}_p(u) \in[0,1], ensuring the peak activation of each mask is scaled to unity. Each reference representation is projected into a scaffold-specific feature space:

ep=WszpU,(5)e_p = W_s z_p^U, \tag*{(5)}

where WsW_s is a learnable linear projection matrix that aligns the reference representation with the scaffold feature dimension. The scaffold canvas C\mathcal{C} at spatial location uu is then defined as a weighted sum of these features:

C(u)=∑p=1NA~p(u)ep.(6)\mathcal{C}(u) = \sum_{p=1}^{N}\tilde{A}_p(u)e_p. \tag*{(6)}

By not normalizing the sum across phrases, C(u)\mathcal{C}(u) naturally retains the support confidence, where regions with stronger phrase support produce higher-magnitude scaffold features.

Scaffold injection The scaffold canvas is injected into an intermediate spatial feature map Ftℓ∈RH×W×DF_t^\ell\in\mathbb{R}^{H \times W \times D} in the generation path through a lightweight residual adapter:

Ftℓ(u)←Ftℓ(u)+m⋅α⋅WcC(u),(7)F_t^\ell(u) \leftarrow F_t^\ell(u) + m \cdot\alpha\cdot W_c\mathcal{C}(u), \tag*{(7)}

where WcW_c is a projection layer, α\alpha is a learnable scalar initialized to zero, and m∼Bernoulli⁡(ρ)m \sim\operatorname{Bernoulli}(\rho) is a scaffold dropout variable with the keep probability ρ∈[0,1]\rho\in[0,1]. The residual form exposes the canvas without changing the architecture of the generation path. Since α\alpha is initialized to zero, the injection leaves the forward computation unchanged at the start of training.

We attenuate the injection by decreasing ρ\rho over training with a cosine schedule. Because mm is sampled per example and independently of the timestep, scaffolded and unscaffolded steps update the same generation parameters. As ρ\rho decays, unscaffolded steps dominate, so the parameters shaped with the scaffold converge toward the path used at inference.

Training Objective and Inference

The training objective comprises the phrase-local contrastive loss and the standard denoising loss:

L=Lgen+λLphrase,(8)\mathcal{L} = \mathcal{L}_{\mathrm{gen}} + \lambda\mathcal{L}_{\mathrm{phrase}}, \tag*{(8)}

where Lgen\mathcal{L}_{\mathrm{gen}} represents the native denoising objective of the base model and λ\lambda is a weighting hyperparameter. During training, the understanding path and the support extraction module remain frozen, while we jointly update the generation path, the projection head gϕg_{\phi}, and the scaffold adapter. The scaffold does not introduce an explicit loss term; instead, it modulates the training process solely through the forward computation when active. At inference, both the contrastive objective and the scaffold branch are discarded. The final model thus operates with its original generation interface and requires no additional inputs or computational overhead during deployment. Together, these components define our Structural Self-Teaching (SST) framework, which transfers phrase-level structure from the understanding path to the generation path entirely through internal signals.

Experiments

Setup

We train on the Instruction-Following subset of Echo-4o [26] (70K synthetic image-caption pairs) and evaluate on GenEval2 [19], which contains 800 prompts stratified by atomicity from 3 to 10 and uses Soft-TIFA, an atom-level VQA judge based on Qwen3-VL-8B. GenEval2 measures compositional fidelity, that is, whether the image satisfies each atom specified in the prompt. We report arithmetic mean (AM), geometric mean (GM), category-level scores, and atomicity-stratified GM, using GM as the primary metric since it penalizes missing visual atoms. GenEval++ results are reported in Appendix D.1. We validate four UMM backbones (BLIP-3o 4B/8B, Show-o2 7B, BAGEL) under LoRA-based tuning of the generation modules with the understanding path frozen. Backbone-specific instantiations and training details are provided in Appendix A.

Quantitative Results

We compare SST with the base UMMs and with LoRA FT, which applies the same LoRA tuning on the same data with the native denoising loss alone Table 1. For BAGEL, we also report CoT prompting and RecA as reference baselines, since they differ from our setting in inference procedure or tuning protocol.

MethodAMGMObj.Attr.CountPos.Verb345678910
BLIP-3o 4B53.8711.7877.9452.8730.7235.0615.4434.7828.2114.226.143.843.532.251.26
+ LoRA FT54.70±0.1854.70_{\pm0.18}13.00±0.3513.00_{\pm0.35}76.6254.2431.6436.3313.3739.3428.2414.6510.714.333.461.901.44
+ SST (Ours)58.29±0.1558.29_{\pm0.15}16.13±0.2816.13_{\pm0.28}81.2954.4937.1741.5317.0347.4133.1916.9214.435.724.533.932.92
BLIP-3o 8B59.3513.5085.0857.7933.4647.1320.3431.6330.9015.5912.306.795.612.672.52
+ LoRA FT60.48±0.2360.48_{\pm0.23}14.86±0.4814.86_{\pm0.48}86.3158.0034.8446.9421.2233.0532.7716.2513.877.496.925.233.27
+ SST (Ours)61.82±0.4561.82_{\pm0.45}16.47±0.3916.47_{\pm0.39}87.8858.8635.2250.1717.1435.7835.9219.8415.838.636.885.143.77
Show-o2 7B63.4315.3288.3660.5144.0144.0527.5831.9933.4416.0117.097.527.695.743.00
+ LoRA FT63.42±0.3863.42_{\pm0.38}16.03±0.4116.03_{\pm0.41}88.5261.3443.9143.4418.1237.4432.2218.8216.668.856.025.332.88
+ SST (Ours)64.28±0.3264.28_{\pm0.32}17.11±0.5817.11_{\pm0.58}89.4462.7144.1246.2720.8939.3634.6719.8117.828.457.565.863.31
BAGEL (14B)66.3619.6589.2761.9049.7344.5622.9651.4832.9421.8520.5110.5110.755.933.18
+ CoT70.8123.2991.9771.7053.0751.5117.9951.9441.5830.7425.5511.9910.977.456.10
+ RecA72.6625.5294.2171.6355.6551.5527.3458.8944.0428.7130.2015.1210.847.778.58
+ LoRA FT73.01±0.3173.01_{\pm0.31}26.47±0.2126.47_{\pm0.21}93.2368.4357.3952.7125.3259.8948.7130.7428.8615.8911.508.937.20
+ SST (Ours)74.06±0.2574.06_{\pm0.25}27.33±0.4227.33_{\pm0.42}93.3971.6958.2354.2126.8260.5252.1631.8628.7414.8913.729.337.44

Table 1. Performance comparison on GenEval2. We report arithmetic mean (AM), geometric mean (GM), category-level scores, and atomicity-wise GM. LoRA FT and SST report means over three training runs, with sample standard deviations for AM and GM. Shaded rows are reference baselines under different settings. CoT modifies inference, and RecA tunes the generation parameters directly, without LoRA, with caption-free reconstruction supervision. Bold marks the best value in each backbone group.

The gains of SST come from its structural supervision rather than from the training data. LoRA FT trains on the same data, and SST improves over it in AM and GM on all four backbones, which span three generation architectures. The gains exceed the standard deviation across three training runs. On BLIP-3o 4B and BAGEL, paired bootstrap intervals over prompts exclude zero (Appendix § 11.§ 3), and a blinded human evaluation also favors SST (Appendix § 11.§ 4).

The category-level scores follow what the supports cover. The supports localize noun phrases, and SST scores higher than LoRA FT on Object, Attribute, Count, and Position on all four backbones. Verb, which the supports do not encode and only 82 of the 800 prompts contain, is the exception. On Show-o2, LoRA fine-tuning lowers Verb sharply and SST recovers only part of the drop; on BLIP-3o 8B, SST falls below both baselines. On BAGEL, SST exceeds both reference baselines in AM and GM without changing the inference procedure, while RecA leads on Object and Verb.

Ablation Studies and Analysis

We analyze the contribution of each component, the source of the support masks, and the design choices of the scaffold, and we examine whether SST reduces generation diversity. Table 2 reports means over three training runs on BLIP-3o 4B and BAGEL. Unless noted otherwise, the remaining ablations use a single run on BLIP-3o 4B, with the same training data, number of optimization steps, and evaluation protocol as the main experiments.

MethodAMGMObjectAttributeCountPositionVerb
BLIP-3o 4B
LoRA FT54.70±0.1813.00±0.3576.62±0.4254.24±0.6831.64±0.4436.33±0.5913.37±0.66
Phrase-local only56.04±0.3415.42±0.4179.88±0.5254.23±0.3735.77±0.6637.98±0.3117.28±0.84
Scaffold only55.24±0.2814.43±0.3377.42±0.3453.94±0.2433.12±0.5638.67±0.4414.42±0.83
Full SST58.29±0.1516.13±0.2881.29±0.2554.49±0.2037.17±0.5941.53±0.4217.03±0.78
BAGEL
LoRA FT73.01±0.3126.47±0.2193.23±0.4168.43±0.4957.39±1.2752.71±0.5525.32±1.09
Phrase-local only73.69±0.3027.12±0.3993.47±0.3570.24±0.5858.20±0.4152.78±0.3626.47±0.71
Scaffold only73.44±0.3226.79±0.3793.33±0.2269.98±0.2458.42±0.6653.92±0.3826.14±0.62
Full SST74.06±0.2527.33±0.4293.39±0.2871.69±0.7658.23±0.7454.21±0.5126.82±0.67

Table 2. Results of main ablation studies on the phrase-local contrastive alignment (Lphrase\mathcal{L}_{phrase}) and structural scaffold on BLIP-3o 4B and BAGEL. Means and sample standard deviations over three training runs. Bold marks the best value in each column for each backbone.

Effect of each component Table 2 ablates Phrase-Local Contrastive Alignment and the Structural Scaffold on BLIP-3o 4B and BAGEL. Phrase-local alignment improves GM over LoRA FT on both backbones, and the full model achieves the highest GM. The two mechanisms overlap in the categories they improve but differ in where their gains concentrate. On BLIP-3o 4B, phrase-local alignment accounts for most of the gain on Object, Count, and Verb. The scaffold contributes most clearly on Position: scaffold only ranks below phrase-local only in GM on both backbones, yet above it on Position. Adding the scaffold to phrase-local alignment raises Position beyond run-to-run variation on both backbones, and on BLIP-3o 4B it also raises Object and Count. On BAGEL, phrase-local alignment alone leaves Position at the level of LoRA FT.

Support masks Table 3 replaces the attention-derived supports with masks from an external segmentation model, Grounded-SAM, keeping all other settings fixed. The attention-derived supports come within 0.79 GM of the external masks and achieve 77% of their gain over LoRA FT. Coarse supports from the understanding path suffice for SST, without an external segmentation model.

Mask sourceAMGMObjectAttributeCountPositionVerb
None (LoRA FT)54.5613.3576.2253.6031.4036.2913.60
Grounded-SAM (external)59.5816.8483.0955.6835.2840.5218.64
Attention-derived (SST)58.3416.0581.4354.3437.3241.5716.29
Δ (SST − external)−1.24−0.79−1.66−1.34+2.04+1.05−2.35

Table 3. Mask source on BLIP-3o 4B (single run each). Grounded-SAM masks serve as an external reference that replaces the attention-derived supports during training, with all other settings fixed. The last row reports the difference between the attention-derived supports and the external reference.

Loss weighting We ablate λ\lambda, the weighting of Lphrase\mathcal{L}_{\mathrm{phrase}}. The optimal scale differs across backbones, with BLIP-3o variants preferring λ∼10\lambda\sim10 and Show-o2 and BAGEL preferring λ∼0.1\lambda\sim0.1 (Table 4). The split follows the generation target space: BLIP-3o denoises in the EVA-CLIP feature space, whereas Show-o2 and BAGEL predict velocities in a VAE latent space. We use the best λ\lambda per backbone in all main experiments.

BLIP-3o variantsShow-o2 / BAGEL
λ\lambdaλ\lambda
1510150.010.10.20.5
BLIP-3o 4B14.1115.3616.0514.71Show-o2 7B15.6616.5516.1215.44
BLIP-3o 8B14.5516.1816.8915.23BAGEL26.1427.2126.4025.76

Table 4. Ablation on λ\lambda, the weighting of Lphrase\mathcal{L}_{\mathrm{phrase}}. The optimal scale of λ\lambda depends on the backbone. We report Overall GM (%) on GenEval2 and bold the best λ\lambda per backbone (single run each).

Scaffold injection layer Table 5(a) evaluates different injection layers for the structural scaffold. Since the scaffold modulates spatial generation states, we report Position as the most directly relevant category together with Overall GM. The earliest layer performs best, and Position decreases as the scaffold is moved to later layers, suggesting that scaffold modulation is most effective before the generation path has committed to a spatial layout within each forward pass.

(a) Scaffold injection layer(b) Scaffold dropout schedule
LayerPositionOverall (GM)SchedulePositionOverall (GM)
No scaffold38.2715.48No scaffold38.2715.48
041.5716.05Fixed ρ=1.0\rho= 1.038.1414.28
340.7915.86Fixed ρ=0.5\rho= 0.540.1515.76
740.0515.45Fixed ρ=0.25\rho= 0.2539.2415.32
1139.4615.52Decay of ρ\rho41.5716.05

Table 5. Scaffold design ablations on GenEval2. We evaluate scaffold injection layer and dropout schedule while keeping all other settings fixed. Since the scaffold modulates spatial generation states, we report Position and Overall GM. Bold numbers indicate the best performance in each column.

Scaffold scheduling Table 5(b) compares scaffold dropout schedules. Since the scaffold branch is removed at inference, the schedule determines how much the generation path is exposed to the privileged scaffold signal during training. The decay schedule achieves the best Position and Overall GM, outperforming fixed keep probabilities. This suggests that the scaffold should provide stronger guidance early in training and be gradually phased out as the unscaffolded generation path is optimized. The result supports scaffold dropout as a training-time curriculum rather than a persistent conditioning pathway.

Scaffold mechanism We isolate the source of the scaffold benefit through canvas controls. Replacing the phrase-grounded canvas with a Uniform canvas, which removes spatial structure, yields performance comparable to no-scaffold, while a Shuffled canvas, which preserves spatial structure but randomizes phrase–region binding, falls below it. The full canvas outperforms both controls (Table 6), indicating that the benefit derives from correctly binding phrase content to spatial support, rather than from added conditioning capacity alone.

CanvasGM
No scaffold15.48
Uniform15.37
Shuffled14.69
Full (Ours)16.05

Table 6. Canvas controls. Uniform canvas removes structure; Shuffled randomizes binding.

Generation diversity Table 7 measures generation diversity among four samples per GenEval2 prompt with two distances, LPIPS [27] and the cosine distance between DINOv2 [28] features. Diversity depends mainly on the backbone, where BAGEL yields roughly twice the distance of BLIP-3o 4B under both metrics. Within each backbone, SST yields higher diversity than LoRA FT under both metrics on BLIP-3o 4B. On BAGEL, LPIPS is similar between the two methods, while DINOv2 distance is higher with SST.

MethodBLIP-3o 4B LPIPSBLIP-3o 4B DINOv2BAGEL LPIPSBAGEL DINOv2
Base0.321 [0.316, 0.326]0.127 [0.121, 0.133]0.586 [0.579, 0.592]0.300 [0.293, 0.308]
LoRA FT0.283 [0.275, 0.291]0.098 [0.093, 0.102]0.628 [0.618, 0.638]0.243 [0.235, 0.251]
SST (Ours)0.317 [0.308, 0.326]0.123 [0.117, 0.128]0.626 [0.614, 0.638]0.256 [0.249, 0.263]

Table 7. Generation diversity on GenEval2 (800 prompts). For each prompt, we generate four samples and average the distance over all six pairs: LPIPS, and the cosine distance between DINOv2 features. Higher is more diverse. Subscripts are 95% bootstrap intervals over prompts.

Qualitative Results

Figure 4 presents qualitative comparisons across the four UMM backbones, illustrating how SST resolves distinct compositional failure modes of the base UMMs. The selected cases span attribute mis-binding, object omission, object entanglement, and spatial violation, covering the major error types observed in dense compositional prompts. Across all cases, the base UMMs either drop, fuse, or mis-relate the prompt-specified concepts, and LoRA fine-tuning provides only partial mitigation while preserving similar failure structures. SST instead restores the prompt-specified compositional structure, generating distinct objects with correct attributes and spatial relations. These cases illustrate the failure modes that SST reduces: omission, fusion, and misplacement of prompt-specified concepts.

Qualitative comparison across four UMM backbones

Figure 4. Qualitative comparison across four UMM backbones. Each group of three images shows one failure mode of the base model: attribute mis-binding for BLIP-3o 4B (cows fused into penguin-like hybrids), object omission for BLIP-3o 8B (the toy is missing), object entanglement for Show-o2 7B (violin and bicycle fused; LoRA FT also fails to fully separate them), and spatial violation for BAGEL (the clock is held rather than placed on top of the bear). SST restores the prompt-specified compositional structure across all cases, while LoRA fine-tuning provides only partial mitigation.

Conclusion

We present Structural Self-Teaching (SST), a framework that improves compositional generation in diffusion-based unified multimodal models. SST leverages the model’s own understanding path as an internal source of phrase-level structural supervision, converting phrase-conditioned attention into support masks without requiring external annotations. The masks reach the generation path through two complementary mechanisms. Phrase-local contrastive alignment aligns each generated phrase region with its target region and separates distinct phrases within a prompt. A training-only structural scaffold modulates an intermediate spatial state with a phrase-level layout built from the same masks. Experiments across representative UMMs demonstrate the effectiveness of SST, improving compositional fidelity across object, attribute, count, and spatial constraints.

Acknowledgement

This work was partly supported by Samsung Electronics Co., Ltd. (IO250418-12669-01), and the Institute of Information & communications Technology Planning & Evaluation (IITP) grants [No.RS-2022-II220959 (No.2022-0-00959), (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making, No.RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University), RS-2025-25442338, AI star Fellowship Support Program (Seoul National University)], and the National Research Foundation of Korea (NRF) grant [No.RS-2022-NR070855, Trustworthy Artificial Intelligence], funded by the Korean government (MSIT).

Implementation Details

Training Configuration

Table 8 summarizes the training configuration. All backbones are trained for 5,000 steps with a per-device batch size of 1 across 4 GPUs and 8 gradient accumulation steps, yielding an effective batch size of 32. We use the AdamW optimizer with 200 warmup steps. LoRA rank, alpha, and learning rate vary across backbones as listed below; all other hyperparameters are shared.

BackboneLoRA rrLoRA α\alphaLearning rate
BLIP-3o 4B64161×10−41 \times10^{-4}
BLIP-3o 8B64165×10−55 \times10^{-5}
Show-o2 7B32161×10−51 \times10^{-5}
BAGEL64161×10−41 \times10^{-4}

Table 8. Training configuration across backbones.

Architecture-Specific Instantiations

We instantiate SST on three unified multimodal models: BLIP-3o, Show-o2, and BAGEL. Each backbone extracts its own support masks {Ap}\{A_p\} offline from its understanding path, following the same procedure (Appendix § 9). The canvas construction and the zero-initialized residual adapter of § 4.3 are also identical across backbones.

The backbones differ in three choices, each of which follows from the native generation pathway. The student feature is the generation-side state that we pool for zpGz_p^G. The reference feature vv is the feature of the target image that we pool for zpUz_p^U. The generation module is the submodule that we tune with LoRA and whose first layer receives the scaffold. Table 9 summarizes these choices. In every case, we L2-normalize the pooled representations and apply the objective of Equation (4).

BLIP-3oShow-o2BAGEL
Student featurex^0\hat{x}_0 (EVA-CLIP)patchified x^0\hat{x}_0 (VAE latent)final LLM hidden state
Reference featureEVA-CLIP featureunified visual representationSigLIP feature
Trainable modules (LoRA)DiT cross-attentiondiffusion headgeneration MoE (all LLM layers)
Scaffold injection pointfirst NextDiT blockfirst diffusion_head_a blockfirst LLM layer (latent positions)

Table 9. Backbone-specific instantiation of phrase-local contrastive alignment and structural scaffold injection. All other components, including the support masks, canvas construction, and residual adapter, are shared across backbones.

BLIP-3o (cascaded DiT) BLIP-3o’s generation path operates natively in the EVA-CLIP [29] feature space, and the DiT directly outputs x^0\hat{x}_0 in that space. We attach a projection head gϕg_\phi that maps x^0\hat{x}_0 into the EVA-CLIP prototype space used for contrastive alignment. The DiT cross-attention is tuned with LoRA. The scaffold is injected as a residual on the output of the first NextDiT [30] block, whose token grid is already aligned with the latent spatial layout.

Show-o2 (LLM with diffusion head) Show-o2’s diffusion head predicts velocities in a VAE latent space. We recover x^0\hat{x}_0 from the predicted velocity, patchify it, and attach a linear projection head gϕg_\phi that maps each patch into the space of the reference feature, Show-o2’s unified visual representation of the clean latent of the target image. The diffusion head is tuned with LoRA and gϕg_\phi is trained from scratch. The scaffold is injected on the first diffusion_head_a block. Because Show-o2 packs [text | spatial | . . .] in a single sequence, the injection hook slices the spatial sub-region using the recorded modality positions, applies the adapter, and writes the modified state back into the corresponding positions.

BAGEL (LLM-as-DiT) BAGEL has no separate DiT module; the LLM itself predicts velocities for the image latents through a generation-specific MoE branch within each LLM layer. We attach a projection head gϕg_\phi that maps LLM hidden states to the SigLIP [31] space, producing the student feature for contrastive alignment. We tune the generation MoE of every LLM layer with LoRA, and train gϕg_\phi from scratch. The scaffold is injected on the first LLM layer. Since BAGEL packs [text | latent | . . .] in a single sequence, the injection hook acts only on the latent-token positions and leaves text positions untouched.

Compute Resources

We train BLIP-3o 4B on 4×4 \times NVIDIA A5000 GPUs, requiring approximately 11 hours of training. The remaining backbones (BLIP-3o 8B, Show-o2 7B, and BAGEL) are trained on 4×4 \times NVIDIA A6000 GPUs, each requiring approximately 10 hours.

Support extraction runs once per backbone and dataset. On BLIP-3o 4B, it takes about 10 hours on 4 A5000 GPUs for the 70K pairs and produces 4.2 GB of cached supports, which all training runs reuse. Extraction materializes attention maps, which precludes fused attention kernels such as FlashAttention. During training, the frozen encoder computes the reference features without gradients, and on BLIP-3o the denoising target already provides them.

Phrase-Conditioned Support Mask Extraction

This appendix provides the full specification of the mask extraction pipeline summarized in § 4.1. The scoring criteria follow [8], while the per-phrase application and soft aggregation are introduced in this work.

For each training pair (I,c)(\mathcal{I}, c), we extract noun phrases {pn}n=1N\{p_n\}_{n=1}^{N} from cc using spaCy, and collect text-to-image attention from every layer-head pair of the understanding path. We denote the resulting candidate map for phrase pp at layer ℓ\ell and head hh as Ap,ℓ,h∈RMA_{p,\ell,h} \in\mathbb{R}^{M}, where MM is the number of image tokens. Candidate scoring and selection are performed independently for each phrase, since different phrases in the same image are typically grounded by different heads.

Attention Map Preprocessing

Each candidate map is preprocessed in two steps before scoring. Both steps are adopted from prior work and used as filtering only.

Mean-centered rectification We suppress weak background activations by subtracting the per-map mean and applying a non-negative rectification:

A~p,ℓ,h(i)=ReLU⁡(Ap,ℓ,h(i)−1M∑i′=1MAp,ℓ,h(i′)).(9)\widetilde{A}_{p,\ell,h}(i) = \operatorname{ReLU}\left(A_{p,\ell,h}(i) - \frac{1}{M}\sum_{i'=1}^{M} A_{p,\ell,h}(i')\right). \tag*{(9)}

This step retains only above-average activations within each candidate map.

Visual sink removal Following [16], we account for visual attention sinks: a small set of image tokens that consistently absorb large attention mass irrespective of semantic content. We compute a global sink score g(i)g(i) for each image token ii by averaging A~p,ℓ,h(i)\widetilde{A}_{p,\ell,h}(i) across the entire candidate pool, and define the sink set SS as the top qq fraction of tokens with the largest g(i)g(i). For each candidate, we measure the non-sink mass ratio

rp,ℓ,h=∑i∉SA~p,ℓ,h(i)∑i=1MA~p,ℓ,h(i)+ϵ.(10)r_{p,\ell,h} = \frac{\sum_{i\notin S}\widetilde{A}_{p,\ell,h}(i)}{\sum_{i=1}^{M}\widetilde{A}_{p,\ell,h}(i)+\epsilon}. \tag*{(10)}

Maps with rp,ℓ,hr_{p,\ell,h} below a threshold η\eta are discarded as sink-dominated. The remaining maps are sink-zeroed:

A‾p,ℓ,h(i)={0i∈S,A~p,ℓ,h(i)i∉S.(11)\overline{A}_{p,\ell,h}(i) = \begin{cases} 0 & i \in S,\\ \widetilde{A}_{p,\ell,h}(i) & i \notin S. \end{cases} \tag*{(11)}

The preprocessed maps {A‾p,ℓ,h}\{\overline{A}_{p,\ell,h}\} form the surviving candidate pool CpC_p for each phrase pp.

Per-Phrase Scoring

After preprocessing, each surviving candidate is scored on two criteria: relevance and spatial plausibility. Both scores are computed per phrase and standardized within the corresponding candidate pool CpC_p, so that scores remain comparable across phrases.

Attention-sum score The attention-sum measures how much mass the candidate places on image tokens for the given phrase. We compute the raw sum

up,ℓ,h=∑i=1MAˉp,ℓ,h(i),(12)u_{p,\ell,h} = \sum_{i=1}^{M} \bar{A}_{p,\ell,h}(i), \tag*{(12)}

and standardize within CpC_p:

sp,ℓ,hsum=up,ℓ,h−μpsumσpsum+ε,(13)s^{\mathrm{sum}}_{p,\ell,h} = \frac{u_{p,\ell,h} - \mu^{\mathrm{sum}}_p}{\sigma^{\mathrm{sum}}_p + \varepsilon}, \tag*{(13)}

where μpsum\mu^{\mathrm{sum}}_p and σpsum\sigma^{\mathrm{sum}}_p are the mean and standard deviation of {up,ℓ′,h′}(ℓ′,h′)∈Cp\{u_{p,\ell',h'}\}_{(\ell',h')\in C_p}.

Entropy score The entropy term penalizes overly diffuse maps. We compute the spatial entropy of the map normalized to unit mass, so that the score depends on spread and not on mass:

Pp,ℓ,h(i)=Aˉp,ℓ,h(i)up,ℓ,h+ε,Hp,ℓ,h=−∑i=1MPp,ℓ,h(i)log⁡(Pp,ℓ,h(i)+ε)(14)P_{p,\ell,h}(i) = \frac{\bar{A}_{p,\ell,h}(i)}{u_{p,\ell,h} + \varepsilon}, \qquad H_{p,\ell,h} = -\sum_{i=1}^{M} P_{p,\ell,h}(i) \log\left(P_{p,\ell,h}(i) + \varepsilon\right) \tag*{(14)}

and standardize the negative entropy within the same candidate pool:

sp,ℓ,hent=−Hp,ℓ,h−μpentσpent+ε,(15)s^{\mathrm{ent}}_{p,\ell,h} = \frac{-H_{p,\ell,h} - \mu^{\mathrm{ent}}_p}{\sigma^{\mathrm{ent}}_p + \varepsilon}, \tag*{(15)}

with μpent\mu^{\mathrm{ent}}_p and σpent\sigma^{\mathrm{ent}}_p defined analogously over {−Hp,ℓ′,h′}\{-H_{p,\ell',h'}\}. This formulation favors candidates that are concentrated relative to other candidates for the same phrase, without imposing an absolute compactness assumption.

Final score We sum the two standardized terms with equal weighting:

sp,ℓ,h=sp,ℓ,hsum+sp,ℓ,hent.(16)s_{p,\ell,h} = s^{\mathrm{sum}}_{p,\ell,h} + s^{\mathrm{ent}}_{p,\ell,h}. \tag*{(16)}

Once candidate scoring is performed per phrase, the two standardized terms operate on comparable scales and play complementary roles, with the sum term capturing relevance and the entropy term penalizing diffuse maps.

Soft Top-KK Aggregation

For each phrase pp, we rank candidates by sp,ℓ,hs_{p,\ell,h} and retain the top KK maps {Aˉp,k}k=1K\{\bar{A}_{p,k}\}_{k=1}^{K} with corresponding scores {sp,k}k=1K\{s_{p,k}\}_{k=1}^{K}. The phrase-conditioned support mask is obtained by temperature-controlled softmax aggregation:

Ap=∑k=1Kwp,kAˉp,k,wp,k=exp⁡(sp,k/τ)∑k′=1Kexp⁡(sp,k′/τ).(17)A_p = \sum_{k=1}^{K} w_{p,k}\bar{A}_{p,k}, \qquad w_{p,k} = \frac{\exp(s_{p,k}/\tau)}{\sum_{k'=1}^{K}\exp(s_{p,k'}/\tau)}. \tag*{(17)}

The aggregated mask is renormalized so that ∑i=1MAp(i)=1\sum_{i=1}^{M} A_p(i) = 1. Compared to hard top-KK averaging, soft aggregation preserves informative sub-maximal candidates and produces a denser support, which is particularly useful in the per-phrase regime where the set of useful heads varies across phrases.

Hyperparameter Values

These values are fixed across all backbones in our experiments. Per-phrase candidate selection allows the pipeline to use a single set of hyperparameters without backbone-specific tuning.

HyperparameterSymbolValue
Sink fractionqq0.05
Non-sink mass thresholdη\eta0.7
Top-KK candidatesKK8
Aggregation temperatureτ\tau0.07
Numerical stabilizerε\varepsilon10−810^{-8}

Table 10. Hyperparameters for mask extraction.

Output Interpretation

The extracted ApA_p should be understood as a phrase-conditioned soft spatial support, not as a hard segmentation or an exact instance decomposition. The masks are reliable at coarse localization and at separating spatial responsibility across phrases, and less suitable for recovering finer structural properties such as exact cardinality. We treat ApA_p as a frozen teacher signal derived from the understanding path, and use it to supervise the generation path through phrase-local contrastive alignment (§ 4.2) and structural scaffolding (§ 4.3).

Dataset and Evaluation Protocol

Training Dataset

We use the Instruction-Following subset of Echo-4o [26], a synthetic dataset generated by GPT-4o that contains approximately 70K image-caption pairs. The captions are constructed through template-driven composition over curated base concepts, covering compositional instructions involving multiple objects, attributes (color, material, pattern), counts, sizes, and spatial relations. The synthetic and template-controlled nature of Echo-4o provides clean, diverse compositional supervision that complements the underrepresentation of dense compositional prompts in real-world image-caption corpora. We use the dataset under its publicly released license and apply no additional preprocessing beyond noun phrase extraction with spaCy as described in Appendix B.

GenEval2 Protocol

GenEval2 [19] contains 800 prompts stratified by atomicity from 3 to 10, with 100 prompts per level. Atomicity counts the number of visual atoms (objects, attributes, relations) specified in a prompt. Each prompt is decomposed into atomic VQA questions covering five skill categories (Object, Attribute, Count, Position, Verb), and generated images are evaluated by Soft-TIFA, an atom-level VQA judge based on Qwen3-VL-8B [32]. We follow the standard protocol and generate one image per prompt with default sampling hyperparameters of each backbone. We report arithmetic mean (AM), geometric mean (GM), category-level scores, and atomicity-stratified GM as defined in the original benchmark.

Oracle Image Set Curation for Section 3

The understanding-path diagnostic in § 3 requires images that satisfy every atomic condition of a given prompt. GenEval2 [19] provides prompts and atomic questions but does not release paired images, so we construct an oracle image set ourselves. The set is used solely as input to the understanding path during the diagnostic in § 3, and is not used to train any of the compared models or to evaluate generation quality.

Prompts We use the GenEval2 prompts unchanged. The set spans atomicity levels from 3 to 10 with 100 prompts at each level, for a total of 800 prompts. Each prompt is paired with a single oracle image, yielding a final oracle set of 800 image-prompt pairs.

Candidate generation For each prompt, we generate candidate images using two strong image generation models, GPT-Image-1 and Nano Banana. Using two independent generators reduces bias toward the stylistic and compositional preferences of a single model, and increases the likelihood that at least one candidate is close to satisfying the prompt before any editing is applied.

Atomic condition verification Each candidate is evaluated by the same oracle MLLM judge used in the GenEval2 protocol, Qwen3-VL-8B [32], against all atomic questions of the corresponding prompt. Candidates that satisfy every atomic condition are retained as oracle images. Candidates that fail one or more conditions enter the editing stage.

Model-based editing We edit partially satisfying candidates using the same two models (GPT-Image-1 and Nano Banana) in their image-editing modes. Edits are conditioned on a description of the failing atomic condition. After each edit, the result is re-evaluated by the MLLM judge, and the editing loop terminates when all atomic conditions are satisfied or after a small number of unsuccessful iterations. Images that cannot be brought to full satisfaction are discarded rather than retained with partial coverage. No human editing is performed at any stage.

Visual inspection by the authors The authors visually inspect every retained image in the final oracle set. The role of inspection is to catch failure modes that the MLLM judge may miss, such as subtle attribute leakage, unnatural artifacts introduced by editing, or duplicated objects that nominally satisfy a count condition. Images that pass MLLM verification but fail authorial inspection are returned to the editing stage or replaced with a freshly generated candidate. No external workers are involved in either editing or inspection.

Final set statistics The final oracle set contains 800 images, distributed evenly across atomicity levels with 100 images per level. Each retained image satisfies all atomic conditions of its paired prompt under both the MLLM judge and the authors’ visual inspection. Figure 5 shows representative examples across atomicity levels.

Representative oracle images across atomicity levels

Figure 5. Representative oracle images across atomicity levels. Each image satisfies all atomic conditions of its paired GenEval2 prompt under both the MLLM judge and the authors’ visual inspection.

Scope and limitations. The oracle set is a diagnostic instrument rather than a benchmark. It is constructed at a scale appropriate for probing understanding-path behavior across atomicity levels, not for training or for benchmark-grade evaluation. Both the MLLM judge and the editing models are fallible, and a small number of retained images may exhibit subtle inconsistencies that escape both verification stages. We accept this as the same noise floor that affects the GenEval2 generation evaluation, and the diagnostic in § 3 should be interpreted in this context. The oracle set is not used as training data or as a generation benchmark anywhere in the paper.

Additional Quantitative Results

GenEval++

GenEval++ [26] contains 280 prompts spanning seven task types of increasing difficulty, evaluated by a GPT-4.1-based judge that marks an image correct only if all compositional constraints are satisfied. Table 11 reports per-task and overall accuracy across four UMM backbones. SST improves overall accuracy over both the base UMM and LoRA fine-tuning on all four backbones. The ranking Base < LoRA FT < SST holds across all backbones, indicating that the improvements observed on GenEval2 transfer consistently to this independent compositional benchmark.

Table 11: Performance comparison on GenEval++ [26]. We report per-task accuracy (%) and overall accuracy. Task abbreviations: CA (color attribute), SCA (spatial-count attribute), CSA (color-spatial attribute), CC (color-count), MC (multi-count), SS (size/spatial), Ct (counting). Bold indicates the best result per backbone group.

MethodCASCACSACCMCSSCtOverall
BLIP-3o 4B
Base12.5017.5040.0015.0015.0052.5027.5025.71
+ LoRA FT12.5020.0042.5015.0012.5055.0027.5026.43
+ SST (Ours)20.0025.0040.0017.5017.5062.5032.5030.71
BLIP-3o 8B
Base15.0022.5045.0010.0025.0062.5035.0030.71
+ LoRA FT17.5020.0047.5012.5030.0060.0035.0031.79
+ SST (Ours)17.5022.5050.0015.0030.0067.5032.5033.57
Show-o2 7B
Base27.507.5017.5012.5012.5030.0010.0016.79
+ LoRA FT30.005.0017.5015.0017.5030.0012.5018.21
+ SST (Ours)32.505.0020.0015.0017.5035.0012.5019.64
BAGEL
Base45.007.5037.5027.5052.5050.0047.5038.21
+ LoRA FT55.0017.5050.0037.5042.5055.0057.5045.00
+ SST (Ours)60.0020.0050.0040.0047.5055.0055.0046.79

Table 11.

Ablation Studies for Support Mask

Support Mask Quality. We verify mask quality on a uniformly sampled 1,000 subset of Echo-4o, using Grounded-SAM [25] on the same noun phrases as pseudo-references. We report activation-inside ratio (AIR) and center-of-mass pointing (Pt), which align with the use case of our supports as soft pooling regions, and focus on relative improvements since the pseudo-references are themselves imperfect. As shown in Table 12, both per-phrase head selection and soft top-KK aggregation provide consistent improvements in AIR and pointing accuracy over global head selection [8].

Table 12: Mask quality ablation on a 1K Echo-4o subset (BLIP-3o 4B). We progressively add per-phrase head selection and soft top-KK aggregation to the global head selection baseline of [8]. Sink suppression [16] is applied to all variants. Bold indicates the best result in each column.

Per-phrase selectionSoft scoringAIRPt
0.3210.603
✓0.3710.645
✓✓0.4110.718

Table 12.

Sensitivity to extraction hyperparameters. We retrained BLIP-3o 4B with supports extracted under different values of τ\tau and KK (Table 13). Aggregation must draw on more than one candidate map. Setting K=1K = 1, or τ=0.01\tau= 0.01, which concentrates the softmax on the top-ranked map, lowers GM by more than one point. For τ≥0.07\tau\ge0.07 and K≥8K \ge8, GM varies by at most 0.12. AIR and pointing accuracy keep improving in this range, but generation scores do not.

AIRPtAMGM
*Temperature τ\tau (K=8K = 8)
τ=0.01\tau= 0.010.3410.64356.2814.87
τ=0.07\tau= 0.070.4110.71858.3416.05
τ=1\tau= 10.4330.72658.2616.01
*Top-KK (τ=0.07\tau= 0.07)
K=1K = 10.3730.66155.5814.37
K=4K = 40.4030.69357.6215.82
K=8K = 80.4110.71858.3416.05
K=16K = 160.4320.73258.3816.13

Table 13. Sensitivity to the extraction hyperparameters on BLIP-3o 4B (single run each). AIR and Pt are measured against Grounded-SAM pseudo-references on the 1K Echo-4o subset of Table 12. AM and GM are GenEval2 scores after retraining with the resulting supports. Our setting is marked in gray.

Paired Bootstrap over Prompts.

We test whether the gains of SST over LoRA FT exceed prompt sampling variation. For one training run per backbone, we resample the 800 GenEval2 prompts with replacement (B = 10,000) and compute paired per-prompt differences. All four intervals exclude zero (Table 14). The win rate is the fraction of prompts on which SST scores higher; no prompt is tied.

BackboneMetricΔ\Delta (SST − LoRA FT)95% CIWin rate
BLIP-3o 4BAM+3.78[+2.69, +4.86]59.4%
BLIP-3o 4BGM+2.70[+1.11, +4.28]60.9%
BAGELAM+1.04[+0.27, +1.81]56.4%
BAGELGM+0.82[+0.11, +1.53]53.9%

Table 14. Prompt-level paired bootstrap of SST over LoRA FT on GenEval2 (800 prompts, B=10,000B = 10{,}000, percentile intervals), for one training run per backbone. Win rate is the fraction of prompts on which SST scores higher.

Human Evaluation

We validate Soft-TIFA and the improvement of SST with a blinded human evaluation on BLIP-3o 4B and BAGEL. For each backbone, we sample 80 GenEval2 prompts, 10 per atomicity level, with a fixed seed before inspecting any scores. Three non-author annotators judge each image of LoRA FT and SST against the unchanged GenEval2 atom questions. Each image is presented independently, with model identity hidden and presentation order randomized. We take the majority vote per atom.

Soft-TIFA agrees with the human majority on 89% of the 2,448 atom questions (Cohen’s κ=0.76\kappa= 0.76), on both LoRA FT and SST outputs. Human judgments favor SST at the atom level by 4.4 points on BLIP-3o 4B and 7.0 points on BAGEL, and both paired bootstrap intervals exclude zero (Table 15). The all-atom rate also increases on both backbones, but its intervals include zero at this sample size.

BackboneMetricLoRA FTSSTΔ\Delta [95% CI]
BLIP-3o 4BAM (human)58.362.6+4.4 [+1.5, +7.1]
BLIP-3o 4BAM (Soft-TIFA)56.159.4+3.3
BLIP-3o 4BAll-atom rate (human)6.38.8+2.5 [−1.3, +5.8]
BLIP-3o 4BGM (Soft-TIFA)9.511.5+2.0
BLIP-3o 4BAgreement88.287.8−0.5
BAGELAM (human)70.076.9+7.0 [+2.7, +11.1]
BAGELAM (Soft-TIFA)66.475.4+9.0
BAGELAll-atom rate (human)12.518.8+6.3 [−1.3, +14.2]
BAGELGM (Soft-TIFA)20.228.8+8.6
BAGELAgreement90.089.9−0.2

Table 15. Human evaluation of LoRA FT and SST on 80 GenEval2 prompts per backbone, 10 per atomicity level, sampled with a fixed seed before any scores were inspected. Human AM is the mean majority-voted satisfaction rate over atom questions. All-atom rate is the fraction of prompts on which the human majority marks every atom question as satisfied. Soft-TIFA scores are computed on the same subset and therefore differ from Table 1. On BAGEL, this subset shows a larger Soft-TIFA gap (+9.0 AM) than the full set (+1.04; Table 14). Agreement is the fraction of atom questions on which Soft-TIFA matches the human majority, computed separately on the images of each variant. Brackets are 95% prompt-level paired bootstrap intervals. Δ\Delta is computed before rounding.

Additional Qualitative Results

Phrase-Conditioned Support Mask Visualizations

Figure 6 illustrates the phrase-conditioned support masks extracted from the understanding path, complementing the schematic illustration in Figure 1. The masks are overlaid on the corresponding images, with brighter regions indicating stronger phrase support.

Phrase-conditioned support masks for baseball glove, three tigers, three pizzas, purple cow, six dogs, and croissant prompts

Figure 6. Phrase-conditioned support masks on images from the training distribution (Echo-4o; top) and the evaluation distribution (GenEval2 oracle image; bottom). The masks form spatially coherent regions across both distributions, supporting their use as pooling regions for phrase-local alignment.

Failure Cases

We present a representative failure case at high atomicity to complement the quantitative trends reported in § 5. Across all backbones, generation under dense compositional prompts at high atomicity remains challenging even with our framework. Figure 7 illustrates this regime through one such case.

Representative failure case at atomicity 8 on BAGEL

Figure 7. Representative failure case at atomicity 8 on BAGEL. Prompt: “three checkered cows, and two bicycles behind four pigs.” The base UMM and LoRA fine-tuning omit the “four pigs” phrase entirely, while SST generates pigs but with an incorrect count (three instead of four).

Limitations

SST inherits its limits from the components it relies on. It tunes LoRA parameters only, so its gains are bounded by the generative quality of the backbone. Its supervision comes from support masks that contain errors we cannot identify without phrase-level ground truth, and all reported results include these errors. Its scaffold modulates a continuous intermediate state, so purely autoregressive image-token UMMs would require a different design.

Two questions remain open. Generation at high atomicity stays difficult on all backbones (Appendix § 12.§ 2), which we attribute in part to the scarcity of dense prompts in current post-training data. Each training pair already includes an image that satisfies its caption, so denser prompts can be constructed and paired the same way. We also schedule the scaffold over training progress and do not analyze its contribution across diffusion timesteps.

Broader Impact

Our work focuses on improving compositional fidelity in unified multimodal models, a foundational research direction with no immediate deployment path. Improvements in text-to-image generation more broadly carry well-documented risks, including the potential for misinformation, deepfakes, and copyright or privacy violations. We do not introduce new generative capabilities beyond those of the base UMMs we build upon, and we do not release new model weights; our framework only modifies the training procedure. Standard mitigations such as content provenance, watermarking, and access controls remain applicable to any downstream deployment of the base models.

References

  1. [1]Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., et al.: Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683 (2025)arxiv.org/abs/2505.14683
  2. [2]Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., Luo, P.: Janus: Decoupling visual encoding for unified multimodal understanding and generation. In CVPR. (2025)arxiv.org/abs/2410.13848
  3. [3]Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C.: Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 (2025)arxiv.org/abs/2501.17811
  4. [4]Xie, J., Mao, W., Bai, Z., Zhang, D.J., Wang, W., Lin, K.Q., Gu, Y., Chen, Z., Yang, Z., Shou, M.Z.: Show-o: One single transformer to unify multimodal understanding and generation. In ICLR. (2025)arxiv.org/abs/2408.12528
  5. [5]Xie, J., Yang, Z., Shou, M.Z.: Show-o2: Improved native unified multimodal models. In NeurIPS. (2025)arxiv.org/abs/2506.15564
  6. [6]Chen, J., Xu, Z., Pan, X., Hu, Y., Qin, C., Goldstein, T., Huang, L., Zhou, T., Xie, S., Savarese, S., et al.: Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505.09568 (2025)arxiv.org/abs/2505.09568
  7. [7]Park, J., Han, B.: Multi-modal representation learning with text-driven soft masks. In CVPR. (2023)arxiv.org/abs/2304.00719
  8. [8]Kang, S., Kim, J., Kim, J., Hwang, S.J.: Your large vision-language model only needs a few attention heads for visual grounding. In CVPR. (2025)arxiv.org/abs/2503.06287
  9. [9]Team, C.: Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 (2024)arxiv.org/abs/2405.09818
  10. [10]Zhou, C., Yu, L., Babu, A., Tirumalas, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., Levy, O.: Transfusion: Predict the next token and diffuse images with one multi-modal model. In ICLR. (2025)arxiv.org/abs/2408.11039
  11. [11]Yang, L., Zhang, X., Tian, Y., Shang, C., Xu, M., Zhang, W., Cui, B.: Hermesflow: Seamlessly closing the gap in multimodal understanding and generation. In NeurIPS. (2025)arxiv.org/abs/2502.12148
  12. [12]Jin, W., Niu, Y., Liao, J., Duan, C., Li, A., Gao, S., Liu, X.: Srum: Fine-grained self-rewarding for unified multimodal models. In ECCV. (2026)arxiv.org/abs/2510.12784
  13. [13]Xie, J., Darrell, T., Zettlemoyer, L., Wang, X.: Reconstruction alignment improves unified multimodal models. In ICLR. (2026)arxiv.org/abs/2509.07295
  14. [14]Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.: Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS. (2021)arxiv.org/abs/2107.07651
  15. [15]Park, J., Choi, J.Y., Park, J., Han, B.: Hierarchical visual feature aggregation for ocr-free document understanding. In NeurIPS. (2024)arxiv.org/abs/2411.05254
  16. [16]Kang, S., Kim, J., Kim, J., Hwang, S.J.: See what you are told: Visual attention sink in large multimodal models. In ICLR. (2025)arxiv.org/abs/2503.03321
  17. [17]Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., Smith, N.A.: TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV. (2023)arxiv.org/abs/2303.11897
  18. [18]Ghosh, D., Hajishirzi, H., Schmidt, L.: GenEval: An object-focused framework for evaluating text-to-image alignment. In NeurIPS. (2023)arxiv.org/abs/2310.11513
  19. [19]Kamath, A., Chang, K.W., Krishna, R., Zettlemoyer, L., Hu, Y., Ghazvininejad, M.: Geneval 2: Addressing benchmark drift in text-to-image evaluation. arXiv preprint arXiv:2512.16853 (2025)arxiv.org/abs/2512.16853
  20. [20]Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. In ICLR. (2023)arxiv.org/abs/2208.01626
  21. [21]Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. In SIGGRAPH. (2023)arxiv.org/abs/2301.13826
  22. [22]Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In ICCV. (2023)arxiv.org/abs/2302.05543
  23. [23]Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: Gligen: Open-set grounded text-to-image generation. In CVPR. (2023)arxiv.org/abs/2301.07093
  24. [24]Feng, W., Zhu, W., Fu, T.J., Jampani, V., Akula, A., He, X., Basu, S., Wang, X.E., Wang, W.Y.: Layoutgpt: Compositional visual planning and generation with large language models. In NeurIPS. (2023)arxiv.org/abs/2305.15393
  25. [25]Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., Zhang, L.: Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024)arxiv.org/abs/2401.14159
  26. [26]Ye, J., Jiang, D., Wang, Z., Zhu, L., Hu, Z., Huang, Z., He, J., Yan, Z., Yu, J., Li, H., et al.: Echo-4o: Harnessing the power of gpt-4o synthetic images for improved image generation. arXiv preprint arXiv:2508.09987 (2025)arxiv.org/abs/2508.09987
  27. [27]Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In CVPR. (2018)arxiv.org/abs/1801.03924
  28. [28]Ouqab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: DINOv2: Learning robust visual features without supervision. TMLR (2024)
  29. [29]Sun, Q., Fang, Y., Wu, L., Wang, X., Cao, Y.: Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 (2023)arxiv.org/abs/2303.15389
  30. [30]Zhuo, L., Du, R., Xiao, H., Li, Y., Liu, D., Huang, R., Liu, W., Zhao, L., Wang, F.Y., Ma, Z., et al.: Lumina-next: Making lumina-t2x stronger and faster with next-dit. In NeurIPS. (2024)arxiv.org/abs/2406.18583
  31. [31]Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In ICCV. (2023)arxiv.org/abs/2303.15343
  32. [32]Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)arxiv.org/abs/2511.21631

Paper details

vlnc:W_yr9exdmq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents