WorksComputer Science
Structural Self-Teaching for Compositional Generation in Unified Multimodal Models
Abstract
We propose Structural Self-Teaching (SST), a novel framework that enhances compositional image generation in diffusion-based unified multimodal models (UMMs) by leveraging their internal understanding path as a source of structural supervision. The generation path of UMMs often struggles with dense compositional prompts, leading to missing objects, attribute binding failures, and spatial errors. In contrast, their understanding path localizes the same phrases in images that satisfy these prompts. SST converts this phrase-level grounding into training-time structural signals for the generation path. Specifically, we extract phrase-conditioned support masks from internal attention maps derived from the understanding path and employ them through two mechanisms. Phrase-local contrastive alignment uses these masks as pooling regions and aligns each generated phrase region with the same region of the target image. A structural scaffold builds a phrase-level layout from the same masks and modulates an intermediate spatial state of the generation path during training. The scaffold is phased out during training and removed at inference. SST requires no external structural annotations for training and no additional input at test time. Extensive experiments across multiple UMMs demonstrate that SST improves compositional fidelity over LoRA fine-tuning on the same data, with no inference-time cost. Project page is available at https://bellos1203.github.io/sst
Introduction
Unified multimodal models (UMMs) integrate image understanding and generation within a single architecture, and recent systems have shown strong perception performance together with competitive generation quality [1, 2, 3, 4, 5, 6]. Despite recent progress, UMMs still struggle to generate images that faithfully satisfy dense compositional prompts. Given prompts that specify multiple objects, attributes, and relations, state-of-the-art UMMs often miscount instances, swap attributes across entities, and drop referenced phrases entirely. These failures intensify as prompts become compositionally denser, motivating training methods that explicitly account for phrase-level structure.
UMMs also provide an internal source of phrase-level structure through their understanding path. Given a training image and its caption, attention between caption and image tokens in the understanding path often localizes individual noun phrases, such as “three flowers,” “a checkered car,” or “a chair,” in spatially coherent image regions (see Figure 1), consistent with prior observations in vision-language models [7, 8]. These phrase-conditioned attention maps provide soft spatial supports over visual tokens. § 3 evaluates whether such supports remain usable under increasing compositional density. We use this internal grounding as a source of structural supervision during generation training, without relying on external annotations.
Figure 1. Unified multimodal models can localize phrases they fail to generate. Given an image-text pair that satisfies a compositional prompt, the understanding path produces phrase-wise attention maps that localize the requested concepts. However, generation from the same prompt can omit objects or entangle attributes, as shown by the base UMM. Structural self-teaching transfers these internal supports during training, improving phrase coverage and compositional fidelity.
To expose this internal structure to the generation path, we propose structural self-teaching for diffusion-based UMMs, a framework that turns the understanding path into an internal structural teacher for its generation path. The central idea is to extract phrase-level grounding from the understanding path and convert it into structural signals for training the generation pathway. All structural cues are derived from the model itself, requiring neither external structural annotations nor inference-time intervention.
Phrase-level structure reaches the generation path through two routes. The first route is phrase-local contrastive alignment. It uses phrase-conditioned support masks as pooling regions for representation learning. It pulls each phrase-specific region in the generated representation toward the corresponding region in the target representation, while pushing it away from other phrase-specific regions in the same prompt. The second is a structural scaffold, which converts the same support masks into a dense phrase-level canvas and modulates intermediate spatial states during training. Scaffold dropout prevents reliance on the canvas. The scaffold branch is removed at inference. Together, the two mechanisms transfer understanding-side structure through both phrase-level representation alignment and spatial modulation during denoising.
The main contributions of our paper are summarized as follows:
We propose structural self-teaching, a training framework that extracts phrase-conditioned support masks from the understanding path of a UMM and reuses them to supervise its generation path without external structural annotations or test-time inputs.
We introduce phrase-local contrastive alignment, which leverages the extracted masks to align phrase-specific generated and reference visual representations while preventing semantic collapse among distinct phrase-level features within the same prompt.
We develop an attenuated structural scaffold that converts these supports into a dense phrase-level canvas for training-time modulation, with a keep probability that decays over training.
The proposed approach demonstrates consistent improvements in compositional fidelity across multiple UMMs, effectively addressing complex object, attribute, and spatial constraints without modifying the inference-time interface.
The rest of the paper is organized as follows. § 2 reviews the related work, followed by our motivation in § 3. The details of our approach are described in § 4, and the experimental results are presented in § 5. Finally, we conclude this paper in § 6.
Related Works
Unified Multimodal Models
Unified multimodal models (UMMs) integrate visual understanding and generation within a single backbone, and generate images either autoregressively over discrete tokens [2, 3, 4, 9] or through a diffusion process [1, 10, 5, 6]. Despite competitive performance on both pathways, UMMs exhibit a persistent asymmetry in which they understand more reliably than they generate, prompting a body of post-training work aimed at closing this gap. HermesFlow [11] first quantifies this asymmetry and proposes preference optimization (Pair-DPO) using self-curated preference pairs from both pathways, aligning the two capabilities at the prompt level. SRUM [12] uses the model’s own understanding module to derive global and local rewards for reward-weighted generation training. RecA [13] instead conditions the UMM on its own understanding embeddings and reconstructs the input image, providing dense pixel-level supervision without captions. Our method also leverages internal signals from the understanding path, but targets compositional structure rather than reward-based post-training or pixel-level reconstruction: we transfer phrase-conditioned spatial supports through a phrase-local contrastive objective, directly addressing the entanglement and omission failures that emerge as compositional complexity grows.
Visual Grounding and Attention in MLLMs
Multimodal transformers encode useful structural signals in their attention patterns [14, 7, 15, 8]. In multimodal encoders, word-conditional attention localizes caption words well enough to guide representation learning without region annotations [7]. In frozen LVLMs, Kang et al. [8] show that a small subset of attention heads acts as localization heads and yield strong visual grounding without grounding-specific training, indicating that phrase-conditioned structure already emerges inside the model. At the same time, Visual Attention Sink [16] observes that a few uninformative tokens absorb disproportionate attention mass, meaning raw maps mix signal with systematic artifacts. The practical question is therefore how to extract this structure selectively. We adopt head scoring and sink suppression from previous works [8, 16], and extend them by scoring and selecting heads per phrase rather than globally—since different phrases in the same image are grounded by different heads—and by replacing hard thresholding with soft, temperature-controlled aggregation so that informative sub-threshold maps still contribute to a dense training target.
Compositional Text-to-Image Generation
Even strong generators struggle with compositional prompts, exhibiting missing entities, incorrect attribute binding, unreliable counting, and broken spatial relations—failures that recent benchmarks such as TIFA [17], GenEval [18], and GenEval2 [19] make quantitatively visible. Existing remedies act either during sampling or through added inputs. Prompt-to-Prompt [20] and Attend-and-Excite [21] manipulate cross-attention in each sampling run and leave the trained model unchanged. ControlNet [22] and GLIGEN [23] condition generation on external structure such as edges, depth, or bounding boxes, and LayoutGPT [24] produces such layouts with an LLM planner. These methods require a structural input beyond the prompt, from the user or from a separate planner. Our approach instead regularizes the generation path during training with structure from the model’s own understanding path. It adds no inference-time step and no input beyond the prompt.
Motivation: Using Understanding Path as Structure Supervision
Our framework uses the understanding path of a UMM as an internal source of phrase-level structural supervision. We therefore first examine whether this path provides usable compositional cues under dense prompts, considering two axes. The first is whether prompt-specified visual facts remain measurable when the corresponding image evidence is available, and the second is whether these facts are spatially grounded enough to define phrase-level supports.
We use the atomic questions from GenEval2 [19], which decompose each prompt into object, attribute, count, and spatial facts. Since GenEval2 provides no oracle images, we curate them through model-based generation and editing to satisfy all atomic conditions (see Appendix § 10.§ 3 for details). These oracle images serve only as diagnostic inputs for probing the understanding path, and are not used for training or as a generation benchmark. Given each oracle image and prompt, the understanding path answers the corresponding atomic questions, and we separately report prompt-only generation scores under the GenEval2 protocol to indicate the compositional difficulty of the same prompts. We emphasize that Figure 2 serves as a diagnostic of the source signal used by our method, not as a direct comparison between perception and generation.

Figure 2. Diagnostic of the understanding-path supervision source across four UMM backbones. Top row. Prompt-only generation scores (Gen) decrease sharply with atomicity, while understanding-path scores on oracle images (Und) remain comparatively stable. Bottom row. Activation-inside ratio (AIR) and center-of-mass pointing accuracy (Pt) of extracted phrase supports against Grounded-SAM pseudo-references remain within narrow bands across atomicity. The top row is not a direct comparison between Und and Gen, see § 3.
Across BLIP-3o [6], Show-o2 [5], and BAGEL [1], prompt-only generation scores decrease sharply as atomicity increases, dropping to near-zero on the densest prompts (Figure 2, top row). In contrast, understanding-path scores on the corresponding oracle images remain comparatively stable across the same atomicity range, with only mild degradation at high atomicity. This stability holds across all evaluated backbones, despite their differences in architecture and scale. The understanding path therefore retains measurable compositional information even under high atomicity, supporting its use as an internal source of structural supervision.
We next examine the second axis, whether the source signal is spatially grounded and stable across compositional density. We extract phrase-conditioned support masks from the understanding path (following the procedure detailed in § 4.1) and compare them against pseudo-reference masks obtained by running Grounded-SAM [25] on the same noun phrases. Across atomicity levels 3–10, the activation-inside ratio and center-of-mass pointing accuracy remain within narrow bands across all four backbones, with no systematic degradation as compositional density grows (Figure 2, bottom row). Since Grounded-SAM pseudo-references are themselves imperfect, we focus on stability under increasing atomicity rather than on absolute localization accuracy. These supports form spatially coherent regions over visual tokens, which we use as pooling regions for phrase-local alignment and as spatial carriers for a structural scaffold during generation training.
Structural Self-Teaching for Compositional Generation
The proposed Structural Self-Teaching (SST) framework transfers phrase-level structural cues from the understanding path of a UMM to its generation path during training (Figure 3). Given an image and its corresponding caption, we first extract noun phrases and derive phrase-conditioned support masks from internal text-to-image attention maps within the understanding path (§ 4.1). The support masks reach the generation path through two routes. Phrase-local contrastive alignment uses them as pooling regions for a representation-level target (§ 4.2). The structural scaffold uses them as a spatial canvas that modulates an intermediate generation state during training (§ 4.3). Attenuated scaffold dropout phases the scaffold out during training. The scaffold branch is removed at inference, so the original generation interface requires no additional input. The final objective combines the model’s native denoising loss with the phrase-local contrastive loss (§ 4.4).
Figure 3. Overview of structural self-teaching. Phrase-conditioned support masks extracted from the frozen understanding path provide two training-time signals for the LoRA-tuned generation path: phrase-local contrastive alignment between reference and generated phrase representations, and a structural scaffold that modulates an intermediate spatial feature . The scaffold is applied with sample-wise cosine-decayed dropout and removed at inference.
Phrase-Conditioned Support Mask Extraction
To construct a reliable teacher signal from the understanding path, we extract a soft spatial support mask for each noun phrase in the caption. Our extraction procedure builds upon the attention-head scoring criteria of [8], originally proposed for zero-shot visual grounding, but we introduce two critical adaptations for compositional generation. First, we perform candidate scoring and selection per phrase rather than identifying single head set for the entire prompt; this reflects the observation that different phrases within the same complex scene are often grounded by distinct sets of heads. Second, we replace rigid threshold-based selection with a soft, temperature-controlled aggregation, ensuring that informative sub-threshold maps still contribute to the final spatial support.
Given an image and its corresponding caption , we extract noun phrases from and collect text-to-image attention from every layer-head pair of the understanding path. This produces a candidate pool for each phrase and layer-head pair , where denotes the number of image tokens. Following [16], we preprocess each candidate map to suppress background activations and visual attention sinks, then rank these resulting candidate maps within each phrase based on a combined score of relevance and spatial plausibility as:
where the relevance term and the spatial plausibility term are both normalized within the phrase-specific candidate pool to ensure comparability.
Let be the -th ranked candidate map after the aforementioned normalization. The final phrase-conditioned support mask is then constructed by softly aggregating the top- candidate maps with temperature-controlled weights:
where the temperature controls the sharpness of the aggregation, and denotes a final sum-to-one renormalization step to ensure the mask values stay within a valid range for the subsequent alignment stage.
The resulting serves as a phrase-conditioned soft spatial support, rather than a binary segmentation or instance mask. By reliably providing coarse localization and separating spatial responsibilities across phrases, acts as a frozen, structurally-grounded teacher that guides the generation path. Implementation details and exact forms of and are provided in Appendix § 9. Example visualizations are shown in Appendix § 12.§ 1.
Phrase-local Contrastive Alignment
We leverage the phrase-conditioned support masks as pooling operators to provide representation-level supervision. Given a caption , its corresponding ground-truth image , and a sampled timestep , the generation path predicts a denoised latent from conditioned on . We encode into reference visual features . For each noun phrase in the caption, we derive a reference visual representation from the target image and a generated visual representation from the denoised latent via mask-weighted pooling:
where denotes mask-weighted spatial averaging, with resized to match the spatial resolution of the respective feature maps. The projection head maps the pooled features from the generation path into the visual feature space of . The choice of features and projection is backbone-specific (Appendix § 8.§ 2).
The alignment objective pulls each generated representation toward its own reference and away from the references of other phrases in the same prompt:
where denotes cosine similarity and is a temperature. The positive term matches the content generated in each phrase region to the corresponding region of the reference image. The in-prompt negatives keep distinct phrases from collapsing onto one representation, which targets the attribute leakage and object fusion observed in dense prompts.
Attenuated Structural Scaffold
The structural scaffold modulates an intermediate spatial state of the generation path with a phrase-level layout during training. The layout assigns each phrase’s reference representation to its support region. Phrase-local contrastive alignment specifies what each phrase region should contain, but it reaches the generation path only through the gradient of a pooled loss. We attenuate the scaffold over training and remove it at inference, so the generation path at inference is unchanged.
Scaffold construction Given the support masks and reference representations , we first resize each mask to the target resolution and normalize it to , ensuring the peak activation of each mask is scaled to unity. Each reference representation is projected into a scaffold-specific feature space:
where is a learnable linear projection matrix that aligns the reference representation with the scaffold feature dimension. The scaffold canvas at spatial location is then defined as a weighted sum of these features:
By not normalizing the sum across phrases, naturally retains the support confidence, where regions with stronger phrase support produce higher-magnitude scaffold features.
Scaffold injection The scaffold canvas is injected into an intermediate spatial feature map in the generation path through a lightweight residual adapter:
where is a projection layer, is a learnable scalar initialized to zero, and is a scaffold dropout variable with the keep probability . The residual form exposes the canvas without changing the architecture of the generation path. Since is initialized to zero, the injection leaves the forward computation unchanged at the start of training.
We attenuate the injection by decreasing over training with a cosine schedule. Because is sampled per example and independently of the timestep, scaffolded and unscaffolded steps update the same generation parameters. As decays, unscaffolded steps dominate, so the parameters shaped with the scaffold converge toward the path used at inference.
Training Objective and Inference
The training objective comprises the phrase-local contrastive loss and the standard denoising loss:
where represents the native denoising objective of the base model and is a weighting hyperparameter. During training, the understanding path and the support extraction module remain frozen, while we jointly update the generation path, the projection head , and the scaffold adapter. The scaffold does not introduce an explicit loss term; instead, it modulates the training process solely through the forward computation when active. At inference, both the contrastive objective and the scaffold branch are discarded. The final model thus operates with its original generation interface and requires no additional inputs or computational overhead during deployment. Together, these components define our Structural Self-Teaching (SST) framework, which transfers phrase-level structure from the understanding path to the generation path entirely through internal signals.
Experiments
Setup
We train on the Instruction-Following subset of Echo-4o [26] (70K synthetic image-caption pairs) and evaluate on GenEval2 [19], which contains 800 prompts stratified by atomicity from 3 to 10 and uses Soft-TIFA, an atom-level VQA judge based on Qwen3-VL-8B. GenEval2 measures compositional fidelity, that is, whether the image satisfies each atom specified in the prompt. We report arithmetic mean (AM), geometric mean (GM), category-level scores, and atomicity-stratified GM, using GM as the primary metric since it penalizes missing visual atoms. GenEval++ results are reported in Appendix D.1. We validate four UMM backbones (BLIP-3o 4B/8B, Show-o2 7B, BAGEL) under LoRA-based tuning of the generation modules with the understanding path frozen. Backbone-specific instantiations and training details are provided in Appendix A.
Quantitative Results
We compare SST with the base UMMs and with LoRA FT, which applies the same LoRA tuning on the same data with the native denoising loss alone Table 1. For BAGEL, we also report CoT prompting and RecA as reference baselines, since they differ from our setting in inference procedure or tuning protocol.
| Method | AM | GM | Obj. | Attr. | Count | Pos. | Verb | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| BLIP-3o 4B | 53.87 | 11.78 | 77.94 | 52.87 | 30.72 | 35.06 | 15.44 | 34.78 | 28.21 | 14.22 | 6.14 | 3.84 | 3.53 | 2.25 | 1.26 |
| + LoRA FT | 76.62 | 54.24 | 31.64 | 36.33 | 13.37 | 39.34 | 28.24 | 14.65 | 10.71 | 4.33 | 3.46 | 1.90 | 1.44 | ||
| + SST (Ours) | 81.29 | 54.49 | 37.17 | 41.53 | 17.03 | 47.41 | 33.19 | 16.92 | 14.43 | 5.72 | 4.53 | 3.93 | 2.92 | ||
| BLIP-3o 8B | 59.35 | 13.50 | 85.08 | 57.79 | 33.46 | 47.13 | 20.34 | 31.63 | 30.90 | 15.59 | 12.30 | 6.79 | 5.61 | 2.67 | 2.52 |
| + LoRA FT | 86.31 | 58.00 | 34.84 | 46.94 | 21.22 | 33.05 | 32.77 | 16.25 | 13.87 | 7.49 | 6.92 | 5.23 | 3.27 | ||
| + SST (Ours) | 87.88 | 58.86 | 35.22 | 50.17 | 17.14 | 35.78 | 35.92 | 19.84 | 15.83 | 8.63 | 6.88 | 5.14 | 3.77 | ||
| Show-o2 7B | 63.43 | 15.32 | 88.36 | 60.51 | 44.01 | 44.05 | 27.58 | 31.99 | 33.44 | 16.01 | 17.09 | 7.52 | 7.69 | 5.74 | 3.00 |
| + LoRA FT | 88.52 | 61.34 | 43.91 | 43.44 | 18.12 | 37.44 | 32.22 | 18.82 | 16.66 | 8.85 | 6.02 | 5.33 | 2.88 | ||
| + SST (Ours) | 89.44 | 62.71 | 44.12 | 46.27 | 20.89 | 39.36 | 34.67 | 19.81 | 17.82 | 8.45 | 7.56 | 5.86 | 3.31 | ||
| BAGEL (14B) | 66.36 | 19.65 | 89.27 | 61.90 | 49.73 | 44.56 | 22.96 | 51.48 | 32.94 | 21.85 | 20.51 | 10.51 | 10.75 | 5.93 | 3.18 |
| + CoT | 70.81 | 23.29 | 91.97 | 71.70 | 53.07 | 51.51 | 17.99 | 51.94 | 41.58 | 30.74 | 25.55 | 11.99 | 10.97 | 7.45 | 6.10 |
| + RecA | 72.66 | 25.52 | 94.21 | 71.63 | 55.65 | 51.55 | 27.34 | 58.89 | 44.04 | 28.71 | 30.20 | 15.12 | 10.84 | 7.77 | 8.58 |
| + LoRA FT | 93.23 | 68.43 | 57.39 | 52.71 | 25.32 | 59.89 | 48.71 | 30.74 | 28.86 | 15.89 | 11.50 | 8.93 | 7.20 | ||
| + SST (Ours) | 93.39 | 71.69 | 58.23 | 54.21 | 26.82 | 60.52 | 52.16 | 31.86 | 28.74 | 14.89 | 13.72 | 9.33 | 7.44 |
Table 1. Performance comparison on GenEval2. We report arithmetic mean (AM), geometric mean (GM), category-level scores, and atomicity-wise GM. LoRA FT and SST report means over three training runs, with sample standard deviations for AM and GM. Shaded rows are reference baselines under different settings. CoT modifies inference, and RecA tunes the generation parameters directly, without LoRA, with caption-free reconstruction supervision. Bold marks the best value in each backbone group.
The gains of SST come from its structural supervision rather than from the training data. LoRA FT trains on the same data, and SST improves over it in AM and GM on all four backbones, which span three generation architectures. The gains exceed the standard deviation across three training runs. On BLIP-3o 4B and BAGEL, paired bootstrap intervals over prompts exclude zero (Appendix § 11.§ 3), and a blinded human evaluation also favors SST (Appendix § 11.§ 4).
The category-level scores follow what the supports cover. The supports localize noun phrases, and SST scores higher than LoRA FT on Object, Attribute, Count, and Position on all four backbones. Verb, which the supports do not encode and only 82 of the 800 prompts contain, is the exception. On Show-o2, LoRA fine-tuning lowers Verb sharply and SST recovers only part of the drop; on BLIP-3o 8B, SST falls below both baselines. On BAGEL, SST exceeds both reference baselines in AM and GM without changing the inference procedure, while RecA leads on Object and Verb.
Ablation Studies and Analysis
We analyze the contribution of each component, the source of the support masks, and the design choices of the scaffold, and we examine whether SST reduces generation diversity. Table 2 reports means over three training runs on BLIP-3o 4B and BAGEL. Unless noted otherwise, the remaining ablations use a single run on BLIP-3o 4B, with the same training data, number of optimization steps, and evaluation protocol as the main experiments.
| Method | AM | GM | Object | Attribute | Count | Position | Verb |
| BLIP-3o 4B | |||||||
| LoRA FT | 54.70±0.18 | 13.00±0.35 | 76.62±0.42 | 54.24±0.68 | 31.64±0.44 | 36.33±0.59 | 13.37±0.66 |
| Phrase-local only | 56.04±0.34 | 15.42±0.41 | 79.88±0.52 | 54.23±0.37 | 35.77±0.66 | 37.98±0.31 | 17.28±0.84 |
| Scaffold only | 55.24±0.28 | 14.43±0.33 | 77.42±0.34 | 53.94±0.24 | 33.12±0.56 | 38.67±0.44 | 14.42±0.83 |
| Full SST | 58.29±0.15 | 16.13±0.28 | 81.29±0.25 | 54.49±0.20 | 37.17±0.59 | 41.53±0.42 | 17.03±0.78 |
| BAGEL | |||||||
| LoRA FT | 73.01±0.31 | 26.47±0.21 | 93.23±0.41 | 68.43±0.49 | 57.39±1.27 | 52.71±0.55 | 25.32±1.09 |
| Phrase-local only | 73.69±0.30 | 27.12±0.39 | 93.47±0.35 | 70.24±0.58 | 58.20±0.41 | 52.78±0.36 | 26.47±0.71 |
| Scaffold only | 73.44±0.32 | 26.79±0.37 | 93.33±0.22 | 69.98±0.24 | 58.42±0.66 | 53.92±0.38 | 26.14±0.62 |
| Full SST | 74.06±0.25 | 27.33±0.42 | 93.39±0.28 | 71.69±0.76 | 58.23±0.74 | 54.21±0.51 | 26.82±0.67 |
Table 2. Results of main ablation studies on the phrase-local contrastive alignment () and structural scaffold on BLIP-3o 4B and BAGEL. Means and sample standard deviations over three training runs. Bold marks the best value in each column for each backbone.
Effect of each component Table 2 ablates Phrase-Local Contrastive Alignment and the Structural Scaffold on BLIP-3o 4B and BAGEL. Phrase-local alignment improves GM over LoRA FT on both backbones, and the full model achieves the highest GM. The two mechanisms overlap in the categories they improve but differ in where their gains concentrate. On BLIP-3o 4B, phrase-local alignment accounts for most of the gain on Object, Count, and Verb. The scaffold contributes most clearly on Position: scaffold only ranks below phrase-local only in GM on both backbones, yet above it on Position. Adding the scaffold to phrase-local alignment raises Position beyond run-to-run variation on both backbones, and on BLIP-3o 4B it also raises Object and Count. On BAGEL, phrase-local alignment alone leaves Position at the level of LoRA FT.
Support masks Table 3 replaces the attention-derived supports with masks from an external segmentation model, Grounded-SAM, keeping all other settings fixed. The attention-derived supports come within 0.79 GM of the external masks and achieve 77% of their gain over LoRA FT. Coarse supports from the understanding path suffice for SST, without an external segmentation model.
| Mask source | AM | GM | Object | Attribute | Count | Position | Verb |
| None (LoRA FT) | 54.56 | 13.35 | 76.22 | 53.60 | 31.40 | 36.29 | 13.60 |
| Grounded-SAM (external) | 59.58 | 16.84 | 83.09 | 55.68 | 35.28 | 40.52 | 18.64 |
| Attention-derived (SST) | 58.34 | 16.05 | 81.43 | 54.34 | 37.32 | 41.57 | 16.29 |
| Δ (SST − external) | −1.24 | −0.79 | −1.66 | −1.34 | +2.04 | +1.05 | −2.35 |
Table 3. Mask source on BLIP-3o 4B (single run each). Grounded-SAM masks serve as an external reference that replaces the attention-derived supports during training, with all other settings fixed. The last row reports the difference between the attention-derived supports and the external reference.
Loss weighting We ablate , the weighting of . The optimal scale differs across backbones, with BLIP-3o variants preferring and Show-o2 and BAGEL preferring (Table 4). The split follows the generation target space: BLIP-3o denoises in the EVA-CLIP feature space, whereas Show-o2 and BAGEL predict velocities in a VAE latent space. We use the best per backbone in all main experiments.
| BLIP-3o variants | Show-o2 / BAGEL | ||||||||
| 1 | 5 | 10 | 15 | 0.01 | 0.1 | 0.2 | 0.5 | ||
| BLIP-3o 4B | 14.11 | 15.36 | 16.05 | 14.71 | Show-o2 7B | 15.66 | 16.55 | 16.12 | 15.44 |
| BLIP-3o 8B | 14.55 | 16.18 | 16.89 | 15.23 | BAGEL | 26.14 | 27.21 | 26.40 | 25.76 |
Table 4. Ablation on , the weighting of . The optimal scale of depends on the backbone. We report Overall GM (%) on GenEval2 and bold the best per backbone (single run each).
Scaffold injection layer Table 5(a) evaluates different injection layers for the structural scaffold. Since the scaffold modulates spatial generation states, we report Position as the most directly relevant category together with Overall GM. The earliest layer performs best, and Position decreases as the scaffold is moved to later layers, suggesting that scaffold modulation is most effective before the generation path has committed to a spatial layout within each forward pass.
| (a) Scaffold injection layer | (b) Scaffold dropout schedule | ||||
| Layer | Position | Overall (GM) | Schedule | Position | Overall (GM) |
| No scaffold | 38.27 | 15.48 | No scaffold | 38.27 | 15.48 |
| 0 | 41.57 | 16.05 | Fixed | 38.14 | 14.28 |
| 3 | 40.79 | 15.86 | Fixed | 40.15 | 15.76 |
| 7 | 40.05 | 15.45 | Fixed | 39.24 | 15.32 |
| 11 | 39.46 | 15.52 | Decay of | 41.57 | 16.05 |
Table 5. Scaffold design ablations on GenEval2. We evaluate scaffold injection layer and dropout schedule while keeping all other settings fixed. Since the scaffold modulates spatial generation states, we report Position and Overall GM. Bold numbers indicate the best performance in each column.
Scaffold scheduling Table 5(b) compares scaffold dropout schedules. Since the scaffold branch is removed at inference, the schedule determines how much the generation path is exposed to the privileged scaffold signal during training. The decay schedule achieves the best Position and Overall GM, outperforming fixed keep probabilities. This suggests that the scaffold should provide stronger guidance early in training and be gradually phased out as the unscaffolded generation path is optimized. The result supports scaffold dropout as a training-time curriculum rather than a persistent conditioning pathway.
Scaffold mechanism We isolate the source of the scaffold benefit through canvas controls. Replacing the phrase-grounded canvas with a Uniform canvas, which removes spatial structure, yields performance comparable to no-scaffold, while a Shuffled canvas, which preserves spatial structure but randomizes phrase–region binding, falls below it. The full canvas outperforms both controls (Table 6), indicating that the benefit derives from correctly binding phrase content to spatial support, rather than from added conditioning capacity alone.
| Canvas | GM |
| No scaffold | 15.48 |
| Uniform | 15.37 |
| Shuffled | 14.69 |
| Full (Ours) | 16.05 |
Table 6. Canvas controls. Uniform canvas removes structure; Shuffled randomizes binding.
Generation diversity Table 7 measures generation diversity among four samples per GenEval2 prompt with two distances, LPIPS [27] and the cosine distance between DINOv2 [28] features. Diversity depends mainly on the backbone, where BAGEL yields roughly twice the distance of BLIP-3o 4B under both metrics. Within each backbone, SST yields higher diversity than LoRA FT under both metrics on BLIP-3o 4B. On BAGEL, LPIPS is similar between the two methods, while DINOv2 distance is higher with SST.
| Method | BLIP-3o 4B LPIPS | BLIP-3o 4B DINOv2 | BAGEL LPIPS | BAGEL DINOv2 |
| Base | 0.321 [0.316, 0.326] | 0.127 [0.121, 0.133] | 0.586 [0.579, 0.592] | 0.300 [0.293, 0.308] |
| LoRA FT | 0.283 [0.275, 0.291] | 0.098 [0.093, 0.102] | 0.628 [0.618, 0.638] | 0.243 [0.235, 0.251] |
| SST (Ours) | 0.317 [0.308, 0.326] | 0.123 [0.117, 0.128] | 0.626 [0.614, 0.638] | 0.256 [0.249, 0.263] |
Table 7. Generation diversity on GenEval2 (800 prompts). For each prompt, we generate four samples and average the distance over all six pairs: LPIPS, and the cosine distance between DINOv2 features. Higher is more diverse. Subscripts are 95% bootstrap intervals over prompts.
Qualitative Results
Figure 4 presents qualitative comparisons across the four UMM backbones, illustrating how SST resolves distinct compositional failure modes of the base UMMs. The selected cases span attribute mis-binding, object omission, object entanglement, and spatial violation, covering the major error types observed in dense compositional prompts. Across all cases, the base UMMs either drop, fuse, or mis-relate the prompt-specified concepts, and LoRA fine-tuning provides only partial mitigation while preserving similar failure structures. SST instead restores the prompt-specified compositional structure, generating distinct objects with correct attributes and spatial relations. These cases illustrate the failure modes that SST reduces: omission, fusion, and misplacement of prompt-specified concepts.

Figure 4. Qualitative comparison across four UMM backbones. Each group of three images shows one failure mode of the base model: attribute mis-binding for BLIP-3o 4B (cows fused into penguin-like hybrids), object omission for BLIP-3o 8B (the toy is missing), object entanglement for Show-o2 7B (violin and bicycle fused; LoRA FT also fails to fully separate them), and spatial violation for BAGEL (the clock is held rather than placed on top of the bear). SST restores the prompt-specified compositional structure across all cases, while LoRA fine-tuning provides only partial mitigation.
Conclusion
We present Structural Self-Teaching (SST), a framework that improves compositional generation in diffusion-based unified multimodal models. SST leverages the model’s own understanding path as an internal source of phrase-level structural supervision, converting phrase-conditioned attention into support masks without requiring external annotations. The masks reach the generation path through two complementary mechanisms. Phrase-local contrastive alignment aligns each generated phrase region with its target region and separates distinct phrases within a prompt. A training-only structural scaffold modulates an intermediate spatial state with a phrase-level layout built from the same masks. Experiments across representative UMMs demonstrate the effectiveness of SST, improving compositional fidelity across object, attribute, count, and spatial constraints.
Acknowledgement
This work was partly supported by Samsung Electronics Co., Ltd. (IO250418-12669-01), and the Institute of Information & communications Technology Planning & Evaluation (IITP) grants [No.RS-2022-II220959 (No.2022-0-00959), (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making, No.RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University), RS-2025-25442338, AI star Fellowship Support Program (Seoul National University)], and the National Research Foundation of Korea (NRF) grant [No.RS-2022-NR070855, Trustworthy Artificial Intelligence], funded by the Korean government (MSIT).
Implementation Details
Training Configuration
Table 8 summarizes the training configuration. All backbones are trained for 5,000 steps with a per-device batch size of 1 across 4 GPUs and 8 gradient accumulation steps, yielding an effective batch size of 32. We use the AdamW optimizer with 200 warmup steps. LoRA rank, alpha, and learning rate vary across backbones as listed below; all other hyperparameters are shared.
| Backbone | LoRA | LoRA | Learning rate |
| BLIP-3o 4B | 64 | 16 | |
| BLIP-3o 8B | 64 | 16 | |
| Show-o2 7B | 32 | 16 | |
| BAGEL | 64 | 16 |
Table 8. Training configuration across backbones.
Architecture-Specific Instantiations
We instantiate SST on three unified multimodal models: BLIP-3o, Show-o2, and BAGEL. Each backbone extracts its own support masks offline from its understanding path, following the same procedure (Appendix § 9). The canvas construction and the zero-initialized residual adapter of § 4.3 are also identical across backbones.
The backbones differ in three choices, each of which follows from the native generation pathway. The student feature is the generation-side state that we pool for . The reference feature is the feature of the target image that we pool for . The generation module is the submodule that we tune with LoRA and whose first layer receives the scaffold. Table 9 summarizes these choices. In every case, we L2-normalize the pooled representations and apply the objective of Equation (4).
| BLIP-3o | Show-o2 | BAGEL | |
| Student feature | (EVA-CLIP) | patchified (VAE latent) | final LLM hidden state |
| Reference feature | EVA-CLIP feature | unified visual representation | SigLIP feature |
| Trainable modules (LoRA) | DiT cross-attention | diffusion head | generation MoE (all LLM layers) |
| Scaffold injection point | first NextDiT block | first diffusion_head_a block | first LLM layer (latent positions) |
Table 9. Backbone-specific instantiation of phrase-local contrastive alignment and structural scaffold injection. All other components, including the support masks, canvas construction, and residual adapter, are shared across backbones.
BLIP-3o (cascaded DiT) BLIP-3o’s generation path operates natively in the EVA-CLIP [29] feature space, and the DiT directly outputs in that space. We attach a projection head that maps into the EVA-CLIP prototype space used for contrastive alignment. The DiT cross-attention is tuned with LoRA. The scaffold is injected as a residual on the output of the first NextDiT [30] block, whose token grid is already aligned with the latent spatial layout.
Show-o2 (LLM with diffusion head) Show-o2’s diffusion head predicts velocities in a VAE latent space. We recover from the predicted velocity, patchify it, and attach a linear projection head that maps each patch into the space of the reference feature, Show-o2’s unified visual representation of the clean latent of the target image. The diffusion head is tuned with LoRA and is trained from scratch. The scaffold is injected on the first diffusion_head_a block. Because Show-o2 packs [text | spatial | . . .] in a single sequence, the injection hook slices the spatial sub-region using the recorded modality positions, applies the adapter, and writes the modified state back into the corresponding positions.
BAGEL (LLM-as-DiT) BAGEL has no separate DiT module; the LLM itself predicts velocities for the image latents through a generation-specific MoE branch within each LLM layer. We attach a projection head that maps LLM hidden states to the SigLIP [31] space, producing the student feature for contrastive alignment. We tune the generation MoE of every LLM layer with LoRA, and train from scratch. The scaffold is injected on the first LLM layer. Since BAGEL packs [text | latent | . . .] in a single sequence, the injection hook acts only on the latent-token positions and leaves text positions untouched.
Compute Resources
We train BLIP-3o 4B on NVIDIA A5000 GPUs, requiring approximately 11 hours of training. The remaining backbones (BLIP-3o 8B, Show-o2 7B, and BAGEL) are trained on NVIDIA A6000 GPUs, each requiring approximately 10 hours.
Support extraction runs once per backbone and dataset. On BLIP-3o 4B, it takes about 10 hours on 4 A5000 GPUs for the 70K pairs and produces 4.2 GB of cached supports, which all training runs reuse. Extraction materializes attention maps, which precludes fused attention kernels such as FlashAttention. During training, the frozen encoder computes the reference features without gradients, and on BLIP-3o the denoising target already provides them.
Phrase-Conditioned Support Mask Extraction
This appendix provides the full specification of the mask extraction pipeline summarized in § 4.1. The scoring criteria follow [8], while the per-phrase application and soft aggregation are introduced in this work.
For each training pair , we extract noun phrases from using spaCy, and collect text-to-image attention from every layer-head pair of the understanding path. We denote the resulting candidate map for phrase at layer and head as , where is the number of image tokens. Candidate scoring and selection are performed independently for each phrase, since different phrases in the same image are typically grounded by different heads.
Attention Map Preprocessing
Each candidate map is preprocessed in two steps before scoring. Both steps are adopted from prior work and used as filtering only.
Mean-centered rectification We suppress weak background activations by subtracting the per-map mean and applying a non-negative rectification:
This step retains only above-average activations within each candidate map.
Visual sink removal Following [16], we account for visual attention sinks: a small set of image tokens that consistently absorb large attention mass irrespective of semantic content. We compute a global sink score for each image token by averaging across the entire candidate pool, and define the sink set as the top fraction of tokens with the largest . For each candidate, we measure the non-sink mass ratio
Maps with below a threshold are discarded as sink-dominated. The remaining maps are sink-zeroed:
The preprocessed maps form the surviving candidate pool for each phrase .
Per-Phrase Scoring
After preprocessing, each surviving candidate is scored on two criteria: relevance and spatial plausibility. Both scores are computed per phrase and standardized within the corresponding candidate pool , so that scores remain comparable across phrases.
Attention-sum score The attention-sum measures how much mass the candidate places on image tokens for the given phrase. We compute the raw sum
and standardize within :
where and are the mean and standard deviation of .
Entropy score The entropy term penalizes overly diffuse maps. We compute the spatial entropy of the map normalized to unit mass, so that the score depends on spread and not on mass:
and standardize the negative entropy within the same candidate pool:
with and defined analogously over . This formulation favors candidates that are concentrated relative to other candidates for the same phrase, without imposing an absolute compactness assumption.
Final score We sum the two standardized terms with equal weighting:
Once candidate scoring is performed per phrase, the two standardized terms operate on comparable scales and play complementary roles, with the sum term capturing relevance and the entropy term penalizing diffuse maps.
Soft Top- Aggregation
For each phrase , we rank candidates by and retain the top maps with corresponding scores . The phrase-conditioned support mask is obtained by temperature-controlled softmax aggregation:
The aggregated mask is renormalized so that . Compared to hard top- averaging, soft aggregation preserves informative sub-maximal candidates and produces a denser support, which is particularly useful in the per-phrase regime where the set of useful heads varies across phrases.
Hyperparameter Values
These values are fixed across all backbones in our experiments. Per-phrase candidate selection allows the pipeline to use a single set of hyperparameters without backbone-specific tuning.
| Hyperparameter | Symbol | Value |
| Sink fraction | 0.05 | |
| Non-sink mass threshold | 0.7 | |
| Top- candidates | 8 | |
| Aggregation temperature | 0.07 | |
| Numerical stabilizer |
Table 10. Hyperparameters for mask extraction.
Output Interpretation
The extracted should be understood as a phrase-conditioned soft spatial support, not as a hard segmentation or an exact instance decomposition. The masks are reliable at coarse localization and at separating spatial responsibility across phrases, and less suitable for recovering finer structural properties such as exact cardinality. We treat as a frozen teacher signal derived from the understanding path, and use it to supervise the generation path through phrase-local contrastive alignment (§ 4.2) and structural scaffolding (§ 4.3).
Dataset and Evaluation Protocol
Training Dataset
We use the Instruction-Following subset of Echo-4o [26], a synthetic dataset generated by GPT-4o that contains approximately 70K image-caption pairs. The captions are constructed through template-driven composition over curated base concepts, covering compositional instructions involving multiple objects, attributes (color, material, pattern), counts, sizes, and spatial relations. The synthetic and template-controlled nature of Echo-4o provides clean, diverse compositional supervision that complements the underrepresentation of dense compositional prompts in real-world image-caption corpora. We use the dataset under its publicly released license and apply no additional preprocessing beyond noun phrase extraction with spaCy as described in Appendix B.
GenEval2 Protocol
GenEval2 [19] contains 800 prompts stratified by atomicity from 3 to 10, with 100 prompts per level. Atomicity counts the number of visual atoms (objects, attributes, relations) specified in a prompt. Each prompt is decomposed into atomic VQA questions covering five skill categories (Object, Attribute, Count, Position, Verb), and generated images are evaluated by Soft-TIFA, an atom-level VQA judge based on Qwen3-VL-8B [32]. We follow the standard protocol and generate one image per prompt with default sampling hyperparameters of each backbone. We report arithmetic mean (AM), geometric mean (GM), category-level scores, and atomicity-stratified GM as defined in the original benchmark.
Oracle Image Set Curation for Section 3
The understanding-path diagnostic in § 3 requires images that satisfy every atomic condition of a given prompt. GenEval2 [19] provides prompts and atomic questions but does not release paired images, so we construct an oracle image set ourselves. The set is used solely as input to the understanding path during the diagnostic in § 3, and is not used to train any of the compared models or to evaluate generation quality.
Prompts We use the GenEval2 prompts unchanged. The set spans atomicity levels from 3 to 10 with 100 prompts at each level, for a total of 800 prompts. Each prompt is paired with a single oracle image, yielding a final oracle set of 800 image-prompt pairs.
Candidate generation For each prompt, we generate candidate images using two strong image generation models, GPT-Image-1 and Nano Banana. Using two independent generators reduces bias toward the stylistic and compositional preferences of a single model, and increases the likelihood that at least one candidate is close to satisfying the prompt before any editing is applied.
Atomic condition verification Each candidate is evaluated by the same oracle MLLM judge used in the GenEval2 protocol, Qwen3-VL-8B [32], against all atomic questions of the corresponding prompt. Candidates that satisfy every atomic condition are retained as oracle images. Candidates that fail one or more conditions enter the editing stage.
Model-based editing We edit partially satisfying candidates using the same two models (GPT-Image-1 and Nano Banana) in their image-editing modes. Edits are conditioned on a description of the failing atomic condition. After each edit, the result is re-evaluated by the MLLM judge, and the editing loop terminates when all atomic conditions are satisfied or after a small number of unsuccessful iterations. Images that cannot be brought to full satisfaction are discarded rather than retained with partial coverage. No human editing is performed at any stage.
Visual inspection by the authors The authors visually inspect every retained image in the final oracle set. The role of inspection is to catch failure modes that the MLLM judge may miss, such as subtle attribute leakage, unnatural artifacts introduced by editing, or duplicated objects that nominally satisfy a count condition. Images that pass MLLM verification but fail authorial inspection are returned to the editing stage or replaced with a freshly generated candidate. No external workers are involved in either editing or inspection.
Final set statistics The final oracle set contains 800 images, distributed evenly across atomicity levels with 100 images per level. Each retained image satisfies all atomic conditions of its paired prompt under both the MLLM judge and the authors’ visual inspection. Figure 5 shows representative examples across atomicity levels.

Figure 5. Representative oracle images across atomicity levels. Each image satisfies all atomic conditions of its paired GenEval2 prompt under both the MLLM judge and the authors’ visual inspection.
Scope and limitations. The oracle set is a diagnostic instrument rather than a benchmark. It is constructed at a scale appropriate for probing understanding-path behavior across atomicity levels, not for training or for benchmark-grade evaluation. Both the MLLM judge and the editing models are fallible, and a small number of retained images may exhibit subtle inconsistencies that escape both verification stages. We accept this as the same noise floor that affects the GenEval2 generation evaluation, and the diagnostic in § 3 should be interpreted in this context. The oracle set is not used as training data or as a generation benchmark anywhere in the paper.
Additional Quantitative Results
GenEval++
GenEval++ [26] contains 280 prompts spanning seven task types of increasing difficulty, evaluated by a GPT-4.1-based judge that marks an image correct only if all compositional constraints are satisfied. Table 11 reports per-task and overall accuracy across four UMM backbones. SST improves overall accuracy over both the base UMM and LoRA fine-tuning on all four backbones. The ranking Base < LoRA FT < SST holds across all backbones, indicating that the improvements observed on GenEval2 transfer consistently to this independent compositional benchmark.
Table 11: Performance comparison on GenEval++ [26]. We report per-task accuracy (%) and overall accuracy. Task abbreviations: CA (color attribute), SCA (spatial-count attribute), CSA (color-spatial attribute), CC (color-count), MC (multi-count), SS (size/spatial), Ct (counting). Bold indicates the best result per backbone group.
| Method | CA | SCA | CSA | CC | MC | SS | Ct | Overall |
| BLIP-3o 4B | ||||||||
| Base | 12.50 | 17.50 | 40.00 | 15.00 | 15.00 | 52.50 | 27.50 | 25.71 |
| + LoRA FT | 12.50 | 20.00 | 42.50 | 15.00 | 12.50 | 55.00 | 27.50 | 26.43 |
| + SST (Ours) | 20.00 | 25.00 | 40.00 | 17.50 | 17.50 | 62.50 | 32.50 | 30.71 |
| BLIP-3o 8B | ||||||||
| Base | 15.00 | 22.50 | 45.00 | 10.00 | 25.00 | 62.50 | 35.00 | 30.71 |
| + LoRA FT | 17.50 | 20.00 | 47.50 | 12.50 | 30.00 | 60.00 | 35.00 | 31.79 |
| + SST (Ours) | 17.50 | 22.50 | 50.00 | 15.00 | 30.00 | 67.50 | 32.50 | 33.57 |
| Show-o2 7B | ||||||||
| Base | 27.50 | 7.50 | 17.50 | 12.50 | 12.50 | 30.00 | 10.00 | 16.79 |
| + LoRA FT | 30.00 | 5.00 | 17.50 | 15.00 | 17.50 | 30.00 | 12.50 | 18.21 |
| + SST (Ours) | 32.50 | 5.00 | 20.00 | 15.00 | 17.50 | 35.00 | 12.50 | 19.64 |
| BAGEL | ||||||||
| Base | 45.00 | 7.50 | 37.50 | 27.50 | 52.50 | 50.00 | 47.50 | 38.21 |
| + LoRA FT | 55.00 | 17.50 | 50.00 | 37.50 | 42.50 | 55.00 | 57.50 | 45.00 |
| + SST (Ours) | 60.00 | 20.00 | 50.00 | 40.00 | 47.50 | 55.00 | 55.00 | 46.79 |
Table 11.
Ablation Studies for Support Mask
Support Mask Quality. We verify mask quality on a uniformly sampled 1,000 subset of Echo-4o, using Grounded-SAM [25] on the same noun phrases as pseudo-references. We report activation-inside ratio (AIR) and center-of-mass pointing (Pt), which align with the use case of our supports as soft pooling regions, and focus on relative improvements since the pseudo-references are themselves imperfect. As shown in Table 12, both per-phrase head selection and soft top- aggregation provide consistent improvements in AIR and pointing accuracy over global head selection [8].
Table 12: Mask quality ablation on a 1K Echo-4o subset (BLIP-3o 4B). We progressively add per-phrase head selection and soft top- aggregation to the global head selection baseline of [8]. Sink suppression [16] is applied to all variants. Bold indicates the best result in each column.
| Per-phrase selection | Soft scoring | AIR | Pt |
| 0.321 | 0.603 | ||
| ✓ | 0.371 | 0.645 | |
| ✓ | ✓ | 0.411 | 0.718 |
Table 12.
Sensitivity to extraction hyperparameters. We retrained BLIP-3o 4B with supports extracted under different values of and (Table 13). Aggregation must draw on more than one candidate map. Setting , or , which concentrates the softmax on the top-ranked map, lowers GM by more than one point. For and , GM varies by at most 0.12. AIR and pointing accuracy keep improving in this range, but generation scores do not.
| AIR | Pt | AM | GM | |
| *Temperature () | ||||
| 0.341 | 0.643 | 56.28 | 14.87 | |
| 0.411 | 0.718 | 58.34 | 16.05 | |
| 0.433 | 0.726 | 58.26 | 16.01 | |
| *Top- () | ||||
| 0.373 | 0.661 | 55.58 | 14.37 | |
| 0.403 | 0.693 | 57.62 | 15.82 | |
| 0.411 | 0.718 | 58.34 | 16.05 | |
| 0.432 | 0.732 | 58.38 | 16.13 |
Table 13. Sensitivity to the extraction hyperparameters on BLIP-3o 4B (single run each). AIR and Pt are measured against Grounded-SAM pseudo-references on the 1K Echo-4o subset of Table 12. AM and GM are GenEval2 scores after retraining with the resulting supports. Our setting is marked in gray.
Paired Bootstrap over Prompts.
We test whether the gains of SST over LoRA FT exceed prompt sampling variation. For one training run per backbone, we resample the 800 GenEval2 prompts with replacement (B = 10,000) and compute paired per-prompt differences. All four intervals exclude zero (Table 14). The win rate is the fraction of prompts on which SST scores higher; no prompt is tied.
| Backbone | Metric | (SST − LoRA FT) | 95% CI | Win rate |
| BLIP-3o 4B | AM | +3.78 | [+2.69, +4.86] | 59.4% |
| BLIP-3o 4B | GM | +2.70 | [+1.11, +4.28] | 60.9% |
| BAGEL | AM | +1.04 | [+0.27, +1.81] | 56.4% |
| BAGEL | GM | +0.82 | [+0.11, +1.53] | 53.9% |
Table 14. Prompt-level paired bootstrap of SST over LoRA FT on GenEval2 (800 prompts, , percentile intervals), for one training run per backbone. Win rate is the fraction of prompts on which SST scores higher.
Human Evaluation
We validate Soft-TIFA and the improvement of SST with a blinded human evaluation on BLIP-3o 4B and BAGEL. For each backbone, we sample 80 GenEval2 prompts, 10 per atomicity level, with a fixed seed before inspecting any scores. Three non-author annotators judge each image of LoRA FT and SST against the unchanged GenEval2 atom questions. Each image is presented independently, with model identity hidden and presentation order randomized. We take the majority vote per atom.
Soft-TIFA agrees with the human majority on 89% of the 2,448 atom questions (Cohen’s ), on both LoRA FT and SST outputs. Human judgments favor SST at the atom level by 4.4 points on BLIP-3o 4B and 7.0 points on BAGEL, and both paired bootstrap intervals exclude zero (Table 15). The all-atom rate also increases on both backbones, but its intervals include zero at this sample size.
| Backbone | Metric | LoRA FT | SST | [95% CI] |
| BLIP-3o 4B | AM (human) | 58.3 | 62.6 | +4.4 [+1.5, +7.1] |
| BLIP-3o 4B | AM (Soft-TIFA) | 56.1 | 59.4 | +3.3 |
| BLIP-3o 4B | All-atom rate (human) | 6.3 | 8.8 | +2.5 [−1.3, +5.8] |
| BLIP-3o 4B | GM (Soft-TIFA) | 9.5 | 11.5 | +2.0 |
| BLIP-3o 4B | Agreement | 88.2 | 87.8 | −0.5 |
| BAGEL | AM (human) | 70.0 | 76.9 | +7.0 [+2.7, +11.1] |
| BAGEL | AM (Soft-TIFA) | 66.4 | 75.4 | +9.0 |
| BAGEL | All-atom rate (human) | 12.5 | 18.8 | +6.3 [−1.3, +14.2] |
| BAGEL | GM (Soft-TIFA) | 20.2 | 28.8 | +8.6 |
| BAGEL | Agreement | 90.0 | 89.9 | −0.2 |
Table 15. Human evaluation of LoRA FT and SST on 80 GenEval2 prompts per backbone, 10 per atomicity level, sampled with a fixed seed before any scores were inspected. Human AM is the mean majority-voted satisfaction rate over atom questions. All-atom rate is the fraction of prompts on which the human majority marks every atom question as satisfied. Soft-TIFA scores are computed on the same subset and therefore differ from Table 1. On BAGEL, this subset shows a larger Soft-TIFA gap (+9.0 AM) than the full set (+1.04; Table 14). Agreement is the fraction of atom questions on which Soft-TIFA matches the human majority, computed separately on the images of each variant. Brackets are 95% prompt-level paired bootstrap intervals. is computed before rounding.
Additional Qualitative Results
Phrase-Conditioned Support Mask Visualizations
Figure 6 illustrates the phrase-conditioned support masks extracted from the understanding path, complementing the schematic illustration in Figure 1. The masks are overlaid on the corresponding images, with brighter regions indicating stronger phrase support.

Figure 6. Phrase-conditioned support masks on images from the training distribution (Echo-4o; top) and the evaluation distribution (GenEval2 oracle image; bottom). The masks form spatially coherent regions across both distributions, supporting their use as pooling regions for phrase-local alignment.
Failure Cases
We present a representative failure case at high atomicity to complement the quantitative trends reported in § 5. Across all backbones, generation under dense compositional prompts at high atomicity remains challenging even with our framework. Figure 7 illustrates this regime through one such case.

Figure 7. Representative failure case at atomicity 8 on BAGEL. Prompt: “three checkered cows, and two bicycles behind four pigs.” The base UMM and LoRA fine-tuning omit the “four pigs” phrase entirely, while SST generates pigs but with an incorrect count (three instead of four).
Limitations
SST inherits its limits from the components it relies on. It tunes LoRA parameters only, so its gains are bounded by the generative quality of the backbone. Its supervision comes from support masks that contain errors we cannot identify without phrase-level ground truth, and all reported results include these errors. Its scaffold modulates a continuous intermediate state, so purely autoregressive image-token UMMs would require a different design.
Two questions remain open. Generation at high atomicity stays difficult on all backbones (Appendix § 12.§ 2), which we attribute in part to the scarcity of dense prompts in current post-training data. Each training pair already includes an image that satisfies its caption, so denser prompts can be constructed and paired the same way. We also schedule the scaffold over training progress and do not analyze its contribution across diffusion timesteps.
Broader Impact
Our work focuses on improving compositional fidelity in unified multimodal models, a foundational research direction with no immediate deployment path. Improvements in text-to-image generation more broadly carry well-documented risks, including the potential for misinformation, deepfakes, and copyright or privacy violations. We do not introduce new generative capabilities beyond those of the base UMMs we build upon, and we do not release new model weights; our framework only modifies the training procedure. Standard mitigations such as content provenance, watermarking, and access controls remain applicable to any downstream deployment of the base models.
References
- [1]Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., et al.: Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683 (2025)arxiv.org/abs/2505.14683
- [2]Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., Luo, P.: Janus: Decoupling visual encoding for unified multimodal understanding and generation. In CVPR. (2025)arxiv.org/abs/2410.13848
- [3]Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C.: Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 (2025)arxiv.org/abs/2501.17811
- [4]Xie, J., Mao, W., Bai, Z., Zhang, D.J., Wang, W., Lin, K.Q., Gu, Y., Chen, Z., Yang, Z., Shou, M.Z.: Show-o: One single transformer to unify multimodal understanding and generation. In ICLR. (2025)arxiv.org/abs/2408.12528
- [5]Xie, J., Yang, Z., Shou, M.Z.: Show-o2: Improved native unified multimodal models. In NeurIPS. (2025)arxiv.org/abs/2506.15564
- [6]Chen, J., Xu, Z., Pan, X., Hu, Y., Qin, C., Goldstein, T., Huang, L., Zhou, T., Xie, S., Savarese, S., et al.: Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505.09568 (2025)arxiv.org/abs/2505.09568
- [7]Park, J., Han, B.: Multi-modal representation learning with text-driven soft masks. In CVPR. (2023)arxiv.org/abs/2304.00719
- [8]Kang, S., Kim, J., Kim, J., Hwang, S.J.: Your large vision-language model only needs a few attention heads for visual grounding. In CVPR. (2025)arxiv.org/abs/2503.06287
- [9]Team, C.: Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 (2024)arxiv.org/abs/2405.09818
- [10]Zhou, C., Yu, L., Babu, A., Tirumalas, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., Levy, O.: Transfusion: Predict the next token and diffuse images with one multi-modal model. In ICLR. (2025)arxiv.org/abs/2408.11039
- [11]Yang, L., Zhang, X., Tian, Y., Shang, C., Xu, M., Zhang, W., Cui, B.: Hermesflow: Seamlessly closing the gap in multimodal understanding and generation. In NeurIPS. (2025)arxiv.org/abs/2502.12148
- [12]Jin, W., Niu, Y., Liao, J., Duan, C., Li, A., Gao, S., Liu, X.: Srum: Fine-grained self-rewarding for unified multimodal models. In ECCV. (2026)arxiv.org/abs/2510.12784
- [13]Xie, J., Darrell, T., Zettlemoyer, L., Wang, X.: Reconstruction alignment improves unified multimodal models. In ICLR. (2026)arxiv.org/abs/2509.07295
- [14]Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.: Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS. (2021)arxiv.org/abs/2107.07651
- [15]Park, J., Choi, J.Y., Park, J., Han, B.: Hierarchical visual feature aggregation for ocr-free document understanding. In NeurIPS. (2024)arxiv.org/abs/2411.05254
- [16]Kang, S., Kim, J., Kim, J., Hwang, S.J.: See what you are told: Visual attention sink in large multimodal models. In ICLR. (2025)arxiv.org/abs/2503.03321
- [17]Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., Smith, N.A.: TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV. (2023)arxiv.org/abs/2303.11897
- [18]Ghosh, D., Hajishirzi, H., Schmidt, L.: GenEval: An object-focused framework for evaluating text-to-image alignment. In NeurIPS. (2023)arxiv.org/abs/2310.11513
- [19]Kamath, A., Chang, K.W., Krishna, R., Zettlemoyer, L., Hu, Y., Ghazvininejad, M.: Geneval 2: Addressing benchmark drift in text-to-image evaluation. arXiv preprint arXiv:2512.16853 (2025)arxiv.org/abs/2512.16853
- [20]Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. In ICLR. (2023)arxiv.org/abs/2208.01626
- [21]Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. In SIGGRAPH. (2023)arxiv.org/abs/2301.13826
- [22]Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In ICCV. (2023)arxiv.org/abs/2302.05543
- [23]Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: Gligen: Open-set grounded text-to-image generation. In CVPR. (2023)arxiv.org/abs/2301.07093
- [24]Feng, W., Zhu, W., Fu, T.J., Jampani, V., Akula, A., He, X., Basu, S., Wang, X.E., Wang, W.Y.: Layoutgpt: Compositional visual planning and generation with large language models. In NeurIPS. (2023)arxiv.org/abs/2305.15393
- [25]Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., Zhang, L.: Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024)arxiv.org/abs/2401.14159
- [26]Ye, J., Jiang, D., Wang, Z., Zhu, L., Hu, Z., Huang, Z., He, J., Yan, Z., Yu, J., Li, H., et al.: Echo-4o: Harnessing the power of gpt-4o synthetic images for improved image generation. arXiv preprint arXiv:2508.09987 (2025)arxiv.org/abs/2508.09987
- [27]Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In CVPR. (2018)arxiv.org/abs/1801.03924
- [28]Ouqab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: DINOv2: Learning robust visual features without supervision. TMLR (2024)
- [29]Sun, Q., Fang, Y., Wu, L., Wang, X., Cao, Y.: Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 (2023)arxiv.org/abs/2303.15389
- [30]Zhuo, L., Du, R., Xiao, H., Li, Y., Liu, D., Huang, R., Liu, W., Zhao, L., Wang, F.Y., Ma, Z., et al.: Lumina-next: Making lumina-t2x stronger and faster with next-dit. In NeurIPS. (2024)arxiv.org/abs/2406.18583
- [31]Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In ICCV. (2023)arxiv.org/abs/2303.15343
- [32]Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)arxiv.org/abs/2511.21631

