betaFeedback

WorksComputer Science

Training-free Conditional Object Composition for Multi-object Image Generation with Diffusion Models

Abstract

Multi-object image generation in text-to-image diffusion models remains challenging, as existing approaches struggle to reliably generate all objects specified in a prompt. Compositional approaches that generate objects independently often fail to capture the interactions among them, whereas joint generation methods tend to suffer from inter-object interference. In this work, we propose a training-free framework, termed conditional object composition, that relaxes joint generation methods by taking one step back toward compositional approaches. Our framework decomposes the generation of NN objects into two stages: it first generates a subset of N−1N-1 objects and then composes the remaining object conditioned on the generated content. To enable effective composition, we adopt an inpainting-based formulation with layout guidance to preserve existing regions and encourage spatially coherent placement. To address the challenge of selecting which objects to generate separately, we introduce an attention-based scoring mechanism based on cross-attention maps and further explore the use of off-the-shelf object detectors. We evaluate our method on multi-object image generation using the widely adopted MS COCO dataset. Experimental results show that our approach achieves competitive or improved performance compared to existing methods, particularly as the number of objects increases. Further studies highlight the importance of the proposed design choices. The source code is available at https://github.com/ShampooWang/condComp_imageGen/tree/main

Introduction

Recent advances in text-to-image diffusion models [21, 22, 19, 6, 23] have significantly improved the realism and diversity of generated images. However, generating images that contain all the objects specified in the text prompt remains a challenging problem [25]. In particular, when the number of objects increases, current models often fail to faithfully generate all required objects, as shown in Figure 1(a). This suggests that multi-object image generation poses challenges beyond those addressed by standard text-to-image generation. To better understand this limitation, we conduct a pilot study on the widely adopted multi-label dataset, MS COCO dataset [13], as shown in Figure 2a.

Comparison of multi-object image generation with required objects

Figure 1. (a) SDXL [21] misses the required object "cat" (b) Our method: adding the dog conditioned on generated objects (person, apple, and cat). Comparison of multi-object image generation with required objects: person, apple, cat, and dog. The base diffusion model fails to generate all specified objects, while our method improves completeness by composing objects conditioned on previously generated content. In this example, the model first generates a subset of objects (person, apple, and cat) and then adds the remaining object (dog) in a subsequent composition step.

Prompts are constructed using ground-truth object labels, and a generation is deemed successful only if all specified object categories appear in the image. By varying the number of objects in each prompt, we observe that the success rate drops substantially as the number of categories increases.

Multi-object image generation can be addressed by either training-based or training-free approaches. Training-based methods learn additional models or require fine-tuning on multi-label datasets, e.g., MS COCO [13], which can improve controllability but may require additional annotation processes [25] and incur extra training costs, and limit applicability to existing systems. In contrast, training-free approaches operate directly on pretrained diffusion models during inference, offering a more practical and flexible alternative. In this work, we focus on training-free approaches that operate on pretrained diffusion models without additional training.

Existing approaches [7, 4, 3] primarily address this problem within a joint generation framework, where all objects are generated jointly in a single diffusion process by introducing refined cross-attention control or latent refinement techniques to improve object-level fidelity. For example, Structure Guidance [7] parses prompts using syntactic trees and applies additional cross-attention to the parsed components. Layout Guidance [4] encourages cross-attention maps to focus within predefined bounding boxes through latent refinement. Attend-andExcite [3] enhances object presence by increasing the minimum cross-attention saliency across objects, also via latent refinement.

However, despite these improvements, they remain fundamentally joint generation methods, which can lead to inter-object competition, where attention maps of different objects interfere with each other, making them increasingly difficult to reliably generate all objects as the number of objects grows. On the other hand, compositional methods [16] generate all objects individually to alleviate such competition. While this strategy circumvents disentanglement, it often fails to capture non-trivial interactions, e.g., spatial relationships between objects, leading to inconsistent or implausible compositions between objects.

To address these limitations, we adopt a strategy that relaxes joint generation methods by taking one step back toward compositional methods: instead of generating all NN objects jointly, we first generate N−1N-1 objects and then compose the remaining object conditioned on the existing content. This formulation alleviates the difficulty of controlling multiple objects within a single generation process while allowing the model to leverage information from previously generated objects to capture interactions between objects. We refer to this framework as conditional object composition. Such an approach transforms the original multi-object generation problem into two components: (1) composition, which integrates new objects into an existing scene, and (2) decomposition, which determines which objects should be generated separately rather than jointly.

However, this formulation introduces several challenges. First, object composition conditioned on previously generated content is non-trivial, as general image editting method [18] tend to reuse or overwrite existing regions, as shown in Figure 2b, leading to object overlap or spatial conflicts. To address this, we perform composition in an inpainting-like manner [17] by constructing a binary mask from cross-attention saliency, which preserves previously generated regions while encouraging new objects to occupy the remaining free space. Second, selecting an appropriate decomposition is challenging, as it requires identifying objects that are difficult to generate jointly. We observe that the salient regions of cross-attention maps capture spatial correspondences between text tokens and image regions, consistent with observations in prior works [24, 14]. We leverage the spatial distribution of cross-attention values to guide the selection of objects for decomposition, and further explore the use of off-the-shelf object detectors as an alternative estimation mechanism.

We evaluate the proposed method on multi-object image generation using the widely adopted MS COCO dataset [13]. Experimental results demonstrate competitive or superior performance over existing methods. Ablation studies show that effective decomposition is essential, while simply enforcing free-space composition during joint generation is ineffective. They further highlight the importance of considering both joint generation and subsequent composition during decomposition.

Performance vs Number of Objects

(a) Success rate of multi-object image generation on MS COCO with varying numbers of objects.

Failure cases of image editing

(b) Failure cases of image editing with increasing editing strength.

Figure 2. (a) Success rate on MS COCO as the number of target objects increases. Our method consistently outperforms the base diffusion model, particularly in challenging multi-object scenarios. Success requires all specified object categories to be generated. (b) Failure cases of general image editing for object composition. Increasing the editing strength shifts the behavior from under-editing (missing the target object) to over-editing (modifying previously generated content).

Related Work

Finer control of text-to-image diffusion models. Recent works have focused on achieving finer control over off-the-shelf text-to-image diffusion models [21, 22, 19, 6, 23] by refining cross-attention during inference. Methods such as Structure Guidance [7] parse input prompts using syntactic trees and apply additional cross-attention computations to the parsed components. Meanwhile, Layout Guidance [4] and Attend-and-Excite [3] introduce latent refinement techniques to manipulate cross-attention maps. In particular, Layout Guidance [4] enforces spatial constraints by encouraging attention maps to align with predefined bounding boxes, while Attend-and-Excite [3] enhances object presence by maximizing the minimum attention weight across target objects. Although these methods provide control at the object or token level, they operate within a joint generation framework, where cross-attention is computed and normalized over all objects simultaneously. As the number of objects increases, this shared attention mechanism introduces inter-object interference and competition for attention, which can limit the effectiveness of control. In contrast, our proposed method adopts a decomposed generation strategy, where objects are generated in different stages. By preserving previously generated regions (e.g., via masking), subsequent object generation is guided to occupy the remaining space, thereby reducing interference with existing content.

An alternative line of work decomposes multi-object generation into independent subproblems. Early works are based on energy-based models [5, 15, 20] and require additional training. Compositional Generation [16] extends this idea to text-to-image diffusion models by factorizing the generation process, modeling each object separately, and combining their conditional predictions via direct summation. While this strategy alleviates interference among objects, it often neglects their interactions, which may lead to inconsistent or implausible compositions when objects exhibit strong spatial or semantic dependencies [3]. In contrast, our formulation considers interactions by generating a subset of objects first and composing the remaining objects conditioned on them.

We note that some works [25, 8] achieve finer control by training on additional datasets. However, these approaches fall outside the scope of this work, as we specifically focus on training-free methods.

Image Editing and Inpainting. Image editing methods provide a complementary perspective for object composition by modifying or extending existing images while preserving their structure. Representative approaches for text-to-image diffusion models, such as SDEdit [18], introduce perturbations to an input image and perform the diffusion process conditioned on the editing target. Prompt-to-Prompt [9] preserves image structure by reusing cross-attention maps during generation. However, general editing methods may struggle to precisely preserve object-level structure, as modifications can affect previously generated regions, as shown in Figure 2b.

To achieve more controlled preservation, we adopt an inpainting-based formulation [17], which uses spatial masks to specify regions to be preserved or modified during generation. This enables object composition while preserving previously generated content. However, we find that inpainting alone is insufficient, as newly added objects may be only partially generated or overlap with existing content. To better encourage objects to occupy available space, we incorporate the spatial control mechanism from layout guidance [4] into the inpainting framework.

Proposed method

Problem overview

Given a text prompt that specifies a set of object categories O={o1,o2,…,oN}\mathcal{O} = \{o_1, o_2, \ldots, o_N\}, the goal is to generate an image that contains all objects in O\mathcal{O}. Existing methods primarily adopt two strategies within a single-stage denoising process. [7, 4, 3] perform joint generation with refined attention control, where all objects O\mathcal{O} are generated simultaneously based on a single conditional noise prediction. In contrast, compositional approaches [16] generate objects o1,o2,…,oNo_1, o_2, \ldots, o_N by computing object-specific noise predictions separately and combining them via direct summation without explicitly modeling interactions between all objects [3].

In this work, we consider an alternative formulation that decomposes the generation process into two stages. Specifically, we first generate an image conditioned on a subset of objects O′⊂O\mathcal{O}' \subset\mathcal{O}, and then compose the remaining object or∈O∖O′o_r \in\mathcal{O}\setminus\mathcal{O}' conditioned on the generated image. This reformulation alleviates the difficulty of jointly generating all objects while still enabling the model to capture interactions among them. It separates the problem into two components: (1) composition, which integrates new objects into an existing scene, and (2) decomposition, which determines which objects should be generated separately rather than jointly.

This formulation introduces two key challenges. First, object composition requires integrating a new object into an existing image while preserving previously generated regions. Second, the choice of decomposition, i.e., selecting which object to generate separately, affects the overall generation quality. To describe our method, let A∈RH×W×MA \in\mathbb{R}^{H \times W \times M} denote the aggregated cross-attention map, where MM is the number of text tokens, and let A:,:,j∈RH×WA_{:,:,j} \in\mathbb{R}^{H \times W} denote the attention map associated with token jj. Unless otherwise specified, AA is aggregated over the selected cross-attention layers and diffusion timesteps. We focus on attention maps corresponding to object tokens. We next describe our approach to composition and decomposition.

Composition

Given an image xO′x^{O'} generated from a subset of objects O′O' in the first stage, our goal is to compose an additional object oro_r into the existing scene while preserving previously generated content in the second stage. To achieve this, we perform composition in an inpainting-like manner [17], where the generation of the new object oro_r is conditioned on a partially preserved image.

Mask Construction. To preserve previously generated objects O′O' during composition, we construct a spatial mask M∈{0,1}H×WM \in\{0,1\}^{H \times W}, where M=0M = 0 denotes preserved regions and M=1M = 1 denotes regions available for generating the new object oro_r. We observe that cross-attention saliency provides a useful signal for spatial correspondences between text tokens and image regions, which is also suggested by prior works [14, 24]. Motivated by this observation, we construct the mask to preserve generated objects based on cross-attention maps during the generation of O′O'.

For each object oi∈O′o_i \in O', let A:,:,j∈RH×WA_{:,:,j} \in\mathbb{R}^{H \times W} denote the cross-attention map corresponding to its associated token index jj. We first apply min-max normalization to each map:

A:,:,j′=A:,:,j−min⁡(A:,:,j)max⁡(A:,:,j)−min⁡(A:,:,j).(1)A'_{:,:,j} = \frac{A_{:,:,j} - \min(A_{:,:,j})}{\max(A_{:,:,j}) - \min(A_{:,:,j})}. \tag*{(1)}

We then aggregate the normalized maps across all objects:

A′=1∣O′∣∑oi∈O′A:,:,j′.(2)A' = \frac{1}{|O'|} \sum_{o_i \in O'} A'_{:,:,j}. \tag*{(2)}

Finally, we obtain a binary mask MM by thresholding the aggregated attention map A′A', where salient regions are assumed to correspond to object locations. Let u∈{1,2,…,H}×{1,2,…,W}u \in\{1,2,\ldots,H\} \times\{1,2,\ldots,W\} denote a spatial location. The mask is defined as

Mu={0,if Au′≥τ,1,otherwise,(3)M_u = \begin{cases} 0, & \text{if } A'_u \ge\tau,\\ 1, & \text{otherwise}, \end{cases} \tag*{(3)}

where τ\tau is a threshold corresponding to the top-k%k\% values of A′A'.

Inpainting-based Composition with Layout Guidance. Given the mask MM, we compose the remaining object oro_r onto the generated image using an inpainting-based formulation [17] combined with Layout Guidance [4]. The mask M∈{0,1}H×WM \in\{0,1\}^{H \times W} defines preserved and editable regions, where M=0M = 0 denotes preserved regions and M=1M = 1 denotes regions available for generating the new object oro_r.

At each denoising timestep tt from TT to 0, we first apply layout guidance to encourage the new object to occupy available space and avoid overlapping with existing content. Following [4], we define an energy function based on cross-attention maps that penalizes placing the target object in regions already occupied by existing objects. Specifically, given the cross-attention map AA, the mask MM, and the token indices j′j' corresponding to the object oro_r, the energy function is defined as

E(A,M,j′)=(1−∑uAuj′Mu∑uAuj′)2.(4)E(A,M,j') = \left(1-\frac{\sum_u A_{u j'} M_u}{\sum_u A_{u j'}}\right)^2 . \tag*{(4)}

The latent is then updated via gradient descent on this energy:

zt←zt−η∇ztE(A,M,j′),(5)z_t \leftarrow z_t-\eta\nabla_{z_t}E(A,M,j'), \tag*{(5)}

where η\eta controls the step size.

After the latent update, we perform inpainting to integrate the new object oro_r while preserving existing content. The preserved region (defined by M=0M = 0) is diffused from the latent of the previously generated image xO′x^{O'}, denoted by zO′z^{O'}, while the editable region (defined by M=1M = 1) is updated using the diffusion model. The resulting latent at timestep tt is obtained by combining the two regions:

zt=M⊙zt+(1−M)⊙ztO′,(6)z_t = M \odot z_t + (1-M) \odot z_t^{O'}, \tag*{(6)}

where ⊙\odot denotes element-wise multiplication.

This process preserves previously generated objects while allowing the new object to be synthesized in the remaining regions. We then proceed with the standard denoising update, and repeat Eqs. (5) and (6) at each timestep tt from TT to 0.

Decomposition

A key component of our framework is to determine which object should be generated separately. Intuitively, objects that are difficult to generate jointly often exhibit strong interference with other objects, which can be reflected in their cross-attention distributions.

To quantify this effect, we define an attention-based scoring function that measures the degree of interference between objects. Let A:,:,i∈RH×WA_{:,:,i} \in\mathbb{R}^{H \times W} denote the cross-attention map corresponding to object oio_i. We use the cross-attention maps at diffusion timesteps tjointt_{\mathrm{joint}} and tcompt_{\mathrm{comp}} for the joint generation and composition stages, respectively. The attention map is then normalized using min-max normalization to obtain A:,:,i′A'_{:,:,i} in Eq. (1). To suppress noisy responses, we further apply a threshold τ′\tau' and set normalized attention values below τ′\tau' to zero. We denote the resulting attention value at spatial location uu as Aui′A'_{u_i}.

For a given subset O′\mathcal{O}', we compute a score for each object oi∈O′o_i \in\mathcal{O}':

si=∑u(Aui′−1∣O′∣−1∑oj∈O′j≠iAuj′)∑uAui′.(7)s_i = \frac{\sum_u \left(A'_{u_i} - \frac{1}{|\mathcal{O}'|-1}\sum_{\substack{o_j\in\mathcal{O}'\\j\ne i}} A'_{u_j}\right)}{\sum_u A'_{u_i}}. \tag*{(7)}

This score sis_i measures how well the attention of object oio_i is separated from that of other objects. A lower score indicates that the attention of oio_i overlaps significantly with other objects, suggesting that it is more difficult to generate jointly.

To obtain a global measure of the joint generation difficulty of the subset O′\mathcal{O}', we take the minimum score across all objects:

sjoint=min⁡isi,(8)s_{\mathrm{joint}} = \min_i s_i, \tag*{(8)}

After generating the subset O′\mathcal{O}', we further evaluate whether the newly added object oro_r is placed in appropriate regions. Let AurA_{u_r} denote the attention map of the added object. We define a composition score as

scomp=∑uAurM(u)∑uAur,(9)s_{\mathrm{comp}} = \frac{\sum_u A_{u_r} M(u)}{\sum_u A_{u_r}}, \tag*{(9)}

which measures the proportion of attention mass located in the editable region defined by the mask MM.

Finally, for each candidate object or∈Oo_r \in\mathcal{O}, we define an overall score as

sor=sjoint⋅scomp.(10)s_{o_r} = s_{\mathrm{joint}} \cdot s_{\mathrm{comp}}. \tag*{(10)}

We select the object oro_r with the highest score sors_{o_r} for decomposition. While our main formulation relies on attention-based estimates, the decomposition strategy is not restricted to this choice. One may also incorporate external object detection models [11, 12, 2, 26] to guide the selection of oro_r, for example, by estimating object presence or localization confidence. We explore such a variant in the experiments.

Experiments

Experimental Setup

Evaluation Setup. We evaluate our method on multi-object image generation using the SDXL base model [21]. Prompts are constructed from ground-truth object categories in the MS COCO dataset [13]. Specifically, we sample object sets O={o1,o2,…,oN}\mathcal{O} = \{o_1, o_2, \ldots, o_N\} from COCO images, ensuring that all objects in O\mathcal{O} co-occur in the same image, rather than forming arbitrary combinations of categories. For each set O\mathcal{O}, we construct prompts of the form: “A photo with realistic lighting, showing a o1,o2,…,oNo_1, o_2, \ldots, o_N”. Evaluation is conducted across varying numbers of objects per prompt to study scalability. For each setting, we generate 100 samples per seed across 5 random seeds. All methods are evaluated under the same sampling configuration for a fair comparison. Additional configuration details are provided in the Appendix § 7.

Evaluation Metric. We measure performance using success rate, defined as the percentage of generated images that contain all specified object categories in the input prompt. To automatically determine object presence, we employ a pre-trained object detector, YOLOv11-L [11], to detect objects in the generated images. A generation is considered successful if all target object categories are detected.

Baselines. We compare our method against representative approaches from two categories. (1) Joint generation methods with finer control, including Structure Guidance [7], Layout Guidance [4], and Attend-and-Excite [3], which generate all objects with refined attention control. (2) Compositional methods, such as Compositional Generation [16], which decompose each object independently and compose them without spatial control. All baselines are implemented using the same underlying diffusion model, SDXL [21].

Results on Multi-Object Generation

Comparison with Existing Methods. Table 1 reports the success rates of different methods on multi-object generation using MS COCO across varying numbers of objects. As the number of objects increases, all methods exhibit a noticeable decline in performance, reflecting the increasing difficulty of generating multiple objects. Among the baselines, Compositional Generation performs substantially worse than other baselines across all settings, highlighting the limitations of independently combining object-specific predictions without modeling interactions, particularly in more challenging scenarios with larger numbers of objects and complex real-world categories. We note that prior Compositional

MethodN=4N = 4N=5N = 5N=6N = 6
SDXL40.8±2.840.8 \pm2.829.0±3.029.0 \pm3.019.2±3.319.2 \pm3.3
Compositional Generation1.2±1.21.2 \pm1.20.4±0.50.4 \pm0.50.00.0
Structure Guidance33.0±0.933.0 \pm0.922.8±3.022.8 \pm3.015.2±1.715.2 \pm1.7
Layout Guidance42.6±3.942.6 \pm3.926.8±6.026.8 \pm6.019.0±2.519.0 \pm2.5
Attend-and-Excite42.6±3.742.6 \pm3.729.0±5.329.0 \pm5.321.2±2.721.2 \pm2.7
condComp (attnScore)48.4±4.348.4 \pm4.334.6±4.434.6 \pm4.425.2±4.725.2 \pm4.7
condComp (objDetect)59.8±3.259.8 \pm3.245.0±6.045.0 \pm6.035.8±5.335.8 \pm5.3

Table 1. Success rate (%) of multi-object image generation on MS COCO with varying number NN of objects. Results are reported as mean ±\pm standard deviation over 5 seeds.

Generation was primarily evaluated on simpler settings, such as two to three objects or toy object categories (e.g., spheres and cubes).

We further compare two variants of our method: condComp (attnScore), which relies on attention-based estimation (Eq. (10)) for decomposition, and condComp (objDetect), which uses an external object detector. For the latter, we adopt Faster R-CNN [12] to simulate a moderately accurate detector in practical scenarios. The attnScore variant achieves competitive performance when the number of objects is medium (e.g., 4 objects), and demonstrates clearer improvements as the number of objects increases. This trend suggests that decomposing the generation process and performing composition in a staged manner can be beneficial in more challenging multi-object settings. The objDetect variant consistently achieves higher success rates, indicating that a more accurate estimation for decomposition can further improve performance.

Overall, these results show that our approach provides a competitive or improved alternative to existing methods, particularly in settings with a larger number of objects where joint generation becomes more challenging.

Trade-off Between Success Rate and Realism. We further analyze the trade-off between object completeness and image realism. Since the number of reference multi-object images is limited, we adopt Kernel Inception Distance (KID) [1], which is more suitable for small-sample evaluation than widely adopted FID [10]. Figure 3 reports the success rate and KID under different probabilities of switching to joint generation using Attend-and-Excite [3]. Specifically, at each generation, we perform conditional composition (objDetect) with probability 1−p1 - p and joint generation with probability pp. As the probability of conditional composition increases (smaller pp), the success rate consistently improves, while the KID score gradually degrades. Notably, in several intermediate settings, such a hybrid strategy achieves substantially higher success rates while maintaining KID scores comparable to Attend-and-Excite. This suggests that conditional composition can improve object completeness without severely sacrificing image realism in some cases.

Success Rate vs. KID Trade-off

Figure 3. Trade-off between success rate and realism measured by KID. Higher success rate and lower KID correspond to better performance. We interpolate between Attend-and-Excite [3] and our method by randomly switching to it with probability pp. Lower pp corresponds to using conditional composition more frequently.

The transition between different probabilities is relatively smooth, indicating that hybrid generation provides a controllable trade-off between realism and object completeness. These results further suggest that conditional composition and joint generation provide complementary advantages for multi-object generation.

Composition Objective in Joint Generation

As joint generation methods with finer control, such as Attend-and-Excite [3], are strong baselines, we first investigate whether the composition objective—i.e., placing an object in free space suggested by other N−1N - 1 objects—can be achieved within a joint generation framework with similar mechanisms. Specifically, at each timestep of joint generation, we apply the same layout guidance objective (Eq. (5)) to the object with the lowest composition score (Eq. (9)). The mask MM is constructed using the attention maps of the other objects. We incorporate this objective into baseline models, including the SDXL [21] and Attend-and-Excite. As shown in Table 2, adding the composition objective consistently degrades performance across different numbers of objects. This suggests that directly enforcing spatial placement constraints within a joint generation process is challenging. One possible reason is that the notion of free space is not well-defined during joint generation, as attention maps of different objects may interfere with each other and evolve throughout the denoising process.

MethodN=4N = 4N=5N = 5N=6N = 6
SDXL40.8 ±\pm 2.829.8 ±\pm 3.019.2 ±\pm 3.3
SDXL + composition40.8 ±\pm 2.928.0 ±\pm 3.618.6 ±\pm 3.4
Attend-and-Excite42.6 ±\pm 3.729.0 ±\pm 5.321.2 ±\pm 2.7
Attend-and-Excite + composition40.6 ±\pm 6.626.8 ±\pm 6.119.0 ±\pm 2.5

Table 2. Effect of applying the composition objective within joint generation frameworks. Results are reported as success rate (%) (mean ±\pm standard deviation over 5 seeds).

Ablation Studies

Effect of Decomposition Strategy. We next examine the importance of selecting an appropriate object for decomposition. To this end, we compare our method with a variant, denoted as condComp (random) in Table 3, which randomly selects the object to be generated in the second stage. The results show that random decomposition leads to noticeably worse performance, in some cases even below the base model. This indicates that the choice of decomposition plays a critical role, and that not all objects are equally suitable to be generated separately. In contrast, our attention-based scoring method provides a more reliable way to identify objects that are difficult to generate jointly, leading to improved performance.

MethodN=4N = 4N=5N = 5N=6N = 6
condComp (random)34.8 ±\pm 2.927.2 ±\pm 3.618.0 ±\pm 7.1
condComp (attnScore)48.4 ±\pm 4.334.6 ±\pm 4.425.2 ±\pm 4.7

Table 3. Effect of decomposition strategy. Results are reported as success rate (%) (mean ±\pm standard deviation over 5 seeds).

Effect of Joint Generation Strategy. We further investigate whether replacing the joint generation stage with Attend-and-Excite [3] can be improved. Table 4 compares using the base diffusion model and Attend-and-Excite for the initial joint generation stage of N−1N - 1 objects. Incorporating Attend-and-Excite improves the KID score, while slightly reducing the overall success rate. This observation is consistent with the trade-off between realism and object completeness discussed in paragraph 4.2, suggesting that stronger joint generation method can improve image quality but may interfere with subsequent composition.

MethodOverallStage-wise
Success RateKIDJoint (N−1N-1)Composition
Attend-and-Excite21.2±2.721.2 \pm2.70.025±0.0040.025 \pm0.004––
condComp (SDXL)35.8±5.335.8 \pm5.30.034±0.0060.034 \pm0.00646.3±4.246.3 \pm4.257.5±7.357.5 \pm7.3
condComp (Attend-and-Excite)32.8±2.232.8 \pm2.20.032±0.0050.032 \pm0.00545.4±3.445.4 \pm3.454.0±2.654.0 \pm2.6

Table 4. Effect of using Attend-and-Excite in the joint generation stage for N=6N = 6. Results are reported as mean ±\pm standard deviation over 5 seeds.

To better understand where failures occur in the proposed framework, we separately analyze the success rates of the joint generation and composition stages in Table 4. Replacing the joint generation stage with Attend-and-Excite leads to only a minor reduction in the joint generation success rate (−0.89%-0.89\%), while the degradation is more pronounced in the composition stage (−3.51%-3.51\%). This suggests that, in our proposed method, jointly generating the initial N−1N - 1 objects should also account for the subsequent object-composition stage.

Conclusion

In this work, we study multi-object image generation in text-to-image diffusion models from a training-free perspective. Existing compositional approaches that generate objects independently often fail to capture interactions among objects, while joint generation methods tend to suffer from inter-object interference. To address these limitations, we propose conditional object composition, a framework that relaxes joint generation by taking one step toward compositional generation. Specifically, the proposed framework decomposes the generation of NN objects into two stages: it first jointly generates N−1N-1 objects and then composes the remaining object conditioned on the generated content. To enable effective composition, we adopt an inpainting-based formulation with layout guidance to preserve existing regions and encourage spatially coherent placement of newly added objects. To address the challenge of selecting suitable objects for decomposition, we further introduce an attention-based scoring mechanism based on cross-attention maps and explore the use of off-the-shelf object detectors. Experiments on MS COCO demonstrate that our method achieves competitive or improved performance compared to existing methods, particularly as the number of objects increases. Further studies highlight the importance of the proposed design choices.

Limitations

Our method introduces additional computational overhead due to its sequential formulation. In practice, it requires approximately 3×3 \times–6×6 \times longer inference time than single-stage methods such as Attend-and-Excite [3] in our experiments. In addition, the quality of composition is not always ideal: newly added objects may appear partially generated, and their placement can be spatially unnatural in some cases. Future work may explore more efficient generation strategies and improved composition mechanisms to produce more coherent and natural results.

Appendix

Hyper-parameters in Experiments We use T=50T = 50 diffusion timesteps for all methods. For the proposed method, each generation stage also uses T=50T = 50 timesteps. The classifier-free guidance scale is set to the default value of 5 for all methods.

For methods based on latent refinement, including Layout Guidance [4], Attend-and-Excite [3], and the proposed method, we optimize cross-attention layers during the denoising process. Following Layout Guidance [4], we optimize the cross-attention layers in the middle and up blocks for Layout Guidance and the proposed method. In Eq. (5), we set the guidance scale η\eta to 50 for the proposed method, while for Layout Guidance and Attend-and-Excite we tuned the refinement scale factor to η=30\eta= 30. All experiments are conducted on a machine equipped with four NVIDIA V100 GPUs, each with 32GB of memory. We found that optimizing both the middle and up blocks for Attend-and-Excite frequently resulted in out-of-memory errors under this setting. Therefore, for Attend-and-Excite, we optimize only cross-attention layers 70–81, which yielded the best performance in our preliminary experiments.

For decomposition, we use the cross-attention maps at diffusion timestep tjoint=20t_{\mathrm{joint}} = 20 for the joint generation stage and tcomp=50=Tt_{\mathrm{comp}} = 50 = T for the composition stage. After min-max normalization, we apply a threshold τ′=0.6\tau^{\prime} = 0.6 and set attention values below the threshold to zero.

References

  1. [1]Bíńkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying MMD GANs. In: Int. Conf. Learn. Represent. (2018)arxiv.org/abs/1801.01401
  2. [2]Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-End Object Detection with Transformers. In: Eur. Conf. Comput. Vis. pp. 213–229 (2020)arxiv.org/abs/2005.12872
  3. [3]Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models. ACM Trans. Graph. 42(4), 1–10 (2023)arxiv.org/abs/2301.13826
  4. [4]Chen, M., Laina, I., Vedaldi, A.: Training-Free Layout Control with Cross-Attention Guidance. In: Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. pp. 5343–5353 (2024)arxiv.org/abs/2304.03373
  5. [5]Du, Y., Li, S., Sharma, Y., Tenenbaum, J., Mordatch, I.: Unsupervised Learning of Compositional Energy Concepts. Adv. Neural Inf. Process. Syst. 34, 15608–15620 (2021)arxiv.org/abs/2111.03042
  6. [6]Esser, P., Kular, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al.: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. In: 41st Int. Conf. Mach. Learn. (2024)arxiv.org/abs/2403.03206
  7. [7]Feng, W., He, X., Fu, T.J., Jampani, V., Akula, A.R., Narayanan, P., Basu, S., Wang, X.E., Wang, W.Y.: Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. In: 11th Int. Conf. Learn. Represent. (2023)arxiv.org/abs/2212.05032
  8. [8]Gaudi, S., Sreekumar, G., Boddeti, V.: ColnD: Enabling Logical Compositions in Diffusion Models. In: 13th Int. Conf. Learn. Represent. (2025)arxiv.org/abs/2503.01145
  9. [9]Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-Prompt Image Editing with Cross-Attention Control. In: 11th Int. Conf. Learn. Represent. (2023)arxiv.org/abs/2208.01626
  10. [10]Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Adv. Neural Inf. Process. Syst. 30 (2017)
  11. [11]Khanam, R., Hussain, M.: YOLOv11: An Overview of the Key Architectural Enhancements. arXiv (2024)arxiv.org/abs/2410.17725
  12. [12]Li, Y., Xie, S., Chen, X., Dollar, P., He, K., Girshick, R.: Benchmarking Detection Transfer Learning with Vision Transformers. arXiv (2021)arxiv.org/abs/2111.11429
  13. [13]Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common Objects in Context. In: Eur. Conf. Comput. Vis. pp. 740–755 (2014)arxiv.org/abs/1405.0312
  14. [14]Liu, B., Wang, C., Cao, T., Jia, K., Huang, J.: Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. pp. 7817–7826 (2024)arxiv.org/abs/2403.03431
  15. [15]Liu, N., Li, S., Du, Y., Tenenbaum, J., Torralba, A.: Learning to Compose Visual Relations. Adv. Neural Inf. Process. Syst. 34, 23166–23178 (2021)arxiv.org/abs/2111.09297
  16. [16]Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B.: Compositional Visual Generation with Composable Diffusion Models. In: Eur. Conf. Comput. Vis. pp. 423–439 (2022)arxiv.org/abs/2206.01714
  17. [17]Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., Van Gool, L.: RePaint: Inpainting Using Denoising Diffusion Probabilistic Models. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. pp. 11461–11471 (2022)arxiv.org/abs/2201.09865
  18. [18]Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In: Int. Conf. Learn. Represent. (2022)arxiv.org/abs/2108.01073
  19. [19]Nichol, A.Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Mcgrew, B., Sutskever, I., Chen, M.: GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In: Int. Conf. Mach. Learn. pp. 16784–16804. PMLR (2022)arxiv.org/abs/2112.10741
  20. [20]Nie, W., Vahdat, A., Anandkumar, A.: Controllable and Compositional Generation with Latent-Space Energy-Based Models. Adv. Neural Inf. Process. Syst. 34, 13497–13510 (2021)arxiv.org/abs/2110.10873
  21. [21]Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In: 12th Int. Conf. Learn. Represent. (2024)arxiv.org/abs/2307.01952
  22. [22]Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-Resolution Image Synthesis with Latent Diffusion Models. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (2022)arxiv.org/abs/2112.10752
  23. [23]Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. Adv. Neural Inf. Process. Syst. 35, 36479–36494 (2022)arxiv.org/abs/2205.11487
  24. [24]Tang, R., Liu, L., Pandey, A., Jiang, Z., Yang, G., Kumar, K., Stenetorp, P., Lin, J., Türe, F.: What the DAAM: Interpreting Stable Diffusion Using Cross Attention. In: Proc. 61st Annu. Meet. Assoc. Comput. Linguist. pp. 5644–5659 (2023)arxiv.org/abs/2210.04885
  25. [25]Wang, Z., Sha, Z., Ding, Z., Wang, Y., Tu, Z.: TokenCompose: Text-to-Image Diffusion with Token-Level Supervision. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. pp. 8553–8564 (2024)arxiv.org/abs/2312.03626
  26. [26]Zhao, Y., Lv, W., Xu, S., Wei, J., Wang, G., Dang, Q., Liu, Y., Chen, J.: DETRs Beat YOLOs on Real-Time Object Detection. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. pp. 16965–16974 (2024)arxiv.org/abs/2304.08069

Paper details

vlnc:W_kkb5fzpb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents