WorksComputer Science
Why Multi-Source Audio Fails to Compose: A Geometric Diagnosis and Its Remedy
Abstract
Multi-source audio-to-image generation exhibits consistent compositional failures when audio tokens occupy geometrically displaced, semantically unanchored positions in the CLIP text-token space, where the diffusion U-Net lacks the prior to reliably attend to them or ground them in distinct spatial regions. We formalize this as the geometric displacement phenomenon and expose it through a two-level clustering analysis comparing BEATs and CLAP embeddings at the raw encoder and projected CLIP token levels. Three geometric properties emerge as important: intra-class compactness, angular alignment with the CLIP text manifold, and inter-token spatial separation. We show that CLAP, whose contrastive language-audio pretraining aligns embeddings with natural language semantics, substantially outperforms BEATs on all three properties, improving the silhouette score by more than 3x and the Audio-Image Content score as well. Building on this analysis, we introduce a multi-token extension that requires no diffusion-backbone retraining and assigns each audio source to a distinct placeholder token in the CLIP tokenizer, complemented by a mixed inference-time objective that combines Attend-and-Excite with an A* spatial separation loss to resolve residual cross-token interference. For evaluation, we construct multi-audio compositional benchmarks from VGGSound (VGG2, VGG3, and VGGN) and extend six standard audio-image metrics to the multi-source setting. Across five benchmarks and four strong baselines, our method achieves strong results on the metrics most directly tied to compositional faithfulness, with AIC improvements of up to 2.25x over the strongest baseline and FID reduced by nearly 2x on two-source generation. We propose geometric displacement as a diagnostic framework for cross-modal generation systems that project learned embeddings into a pretrained contrastive space.
Introduction
Sound in real-world environments is naturally multi-source. A street scene may contain traffic, conversation, and birdsong, while a kitchen scene may include running water, footsteps, and appliance noise. Despite recent progress in audio-conditioned image synthesis [2, 3, 7], most existing methods operate on only a single audio clip and therefore generate single-object scenes. A natural extension is to generate a coherent image from multiple audio inputs, where each sound source is represented in the final scene. We refer to this task as compositional audio-to-image generation (Figure 1).

Figure 1. Compositional audio-to-image generation with our method. The top row progressively adds four audio sources (snare drum, underwater bubbling, volcano explosion, church bell ringing), while the bottom row shows a three-source example (church bell, singing choir, drum kit). Each new object is added to a separate, spatially consistent region while preserving earlier concepts. Our analysis suggests that this behavior is strongly influenced by the geometry of audio tokens in CLIP space, which we quantify in § 3.3, use to redesign the embedder in § 3, and further refine through our mixed inference-time objective.
Extending single-audio methods directly to multiple audio sources results in consistent failure modes. Concatenating multiple audio embeddings into a single conditioning token collapses the sources into an averaged representation. Using multiple audio-derived tokens in the prompt introduces a different problem: the diffusion model often ignores all but one token, generating only one object or merging multiple objects into a semantically unclear result. This behavior is related to the catastrophic neglect problem observed in text-conditioned diffusion models [3]. Figure 2 illustrates this behavior in AudioToken [17], where the cross-attention maps of two audio tokens collapse into the same spatial region during denoising. Our mixed objective explicitly separates these attention maps.

Figure 2. Cross-attention failure modes in compositional generation. Final denoising step () for the prompt “a photo of a and a ”, where represents fire truck siren and represents cat purring. The retrained AT (column 1) produces overlapping attention maps. alone (column 2) separates the attention regions but weakens activation. Attend-and-Excite (column 3) increases token activation but still produces overlap. The mixed objective (column 4) generates strong, spatially separated attention maps.
Our analysis suggests that these failures are strongly influenced by the geometry of the conditioning space. In diffusion models, cross-attention computes similarities between image features and conditioning tokens, which are typically CLIP text embeddings drawn from a well-structured semantic space [11]. When audio-derived tokens are inserted into this space, three geometric properties become important: i) Intra-class compactness. Audio embeddings from the same semantic class should form compact clusters. Otherwise, the diffusion model receives inconsistent conditioning signals across samples. ii) Angular alignment with the CLIP text manifold. Cross-attention depends more strongly on cosine similarity than Euclidean distance. Tokens that preserve angular alignment with the CLIP text space can still behave like text tokens even if their absolute magnitude differs. iii) Spatial separation of
attention maps. When multiple audio tokens are present, each should activate strongly, and their cross-attention maps should occupy different image regions. Weak activation causes neglect, while overlapping maps cause object fusion (Figure 2).
These properties can be directly measured and used to guide model design. We show that the BEATs encoder used in prior work [17] yields weak cluster geometry, with low intra-class compactness and poor separation in both the raw embedding space and the projected CLIP token space. In contrast, CLAP [15] produces more compact and better-aligned embeddings, improving silhouette score by more than . More importantly, CLAP embeddings preserve angular alignment with the CLIP text manifold, a key property for stable cross-attention.
Based on this geometric view, we make the following contributions: i) We provide a geometric analysis of how existing audio-to-image models fail in the compositional setting, using clustering in both the raw audio embedding space and the projected CLIP token space. ii) We introduce a CLAP-based embedder redesigned around CLAP’s global 512-dimensional representation, producing significantly improved cluster geometry compared to BEATs. iii) We propose a multi-token extension (MTE) that assigns each audio source to a separate placeholder token, enabling compositional conditioning without retraining the diffusion backbone. iv) We introduce a mixed inference-time objective that combines Attend-and-Excite with the spatial loss, jointly enforcing strong token activation and spatial separation of attention maps. v) We construct three new compositional benchmarks, VGG2, VGG3 and VGGn, together with multi-source extensions of standard audio-to-image evaluation metrics.
Across five benchmarks (VGG, IS3, VGG2, VGG3, and VGGn) and against four strong baselines (AudioToken, SonicDiffusion, Sound2Scene, and MACS), our variants achieve the best performance on the metrics most closely related to compositional generation, including AIC, IIS, and FID. The CLAP variant is strongest in distribution, improving AIC by up to , while the BEATs variant leads on the out-of-distribution IS3 benchmark.
Our method operates on separate audio sources, which naturally align with settings such as instrument recordings, studio audio tracks, or labeled datasets. When only mixed audio is available, any source separation method such as Demucs [6] can be used as a preprocessing step. Our framework is independent of the specific source separation method used.
Related Work
Audio-to-image generation. Early audio-to-image (A2I) methods were mainly based on conditional GANs. Sound2Scene [3] trains a conditional GAN on a self-supervised image latent space and then aligns an audio encoder with that space. More recent diffusion-based approaches achieve much higher image quality. SonicDiffusion [7] maps CLAP audio features. to tokens and injects them into a frozen Stable Diffusion U-Net via gated audio-image cross-attention layers. AudioToken [17], which our work builds upon, formulates A2I generation as textual inversion [8], projecting BEATs [5] audio features into a learned token inserted into the CLIP text prompt of a frozen Stable Diffusion model [12]. The most closely related prior work is MACS [18], which tackles multi-source A2I by first separating mixed audio using a weakly supervised CLAP-aligned module and a ranking loss, and then mapping each separated signal to the generator via a lightweight adapter. In contrast, we assume separate audio streams and instead address the geometry of audio tokens in CLIP space, thereby repairing cross-attention behavior directly within the denoising loop.
Compositional text-to-image generation. Diffusion models are known to suffer from catastrophic neglect [3], where one or more subjects in a multi-object prompt are ignored during generation. Attend-and-Excite [3] addresses this issue by monitoring the cross-attention activation of subject tokens during denoising and updating the latent representation when activations become weak. Structured Diffusion Guidance [7] decomposes prompts into separate noun phrases routed through different attention pathways. Other approaches, such as [1] and BoxDiff [16], introduce spatial constraints that reduce overlap between attention maps or enforce bounding-box consistency. Our work extends this line of research to the audio-conditioned setting. Audio-derived tokens can occupy a different geometric region in CLIP space from natural text tokens, which can make compositional conditioning more challenging (§ 3.3).
Audio representation learning. The effectiveness of audio-conditioned generation can depend strongly on the geometry of the underlying audio embeddings. BEATs [5] is a self-supervised audio transformer trained using masked acoustic-token prediction and produces frame-level embeddings of dimension 768. In contrast, CLAP [15] is a contrastive language-audio model trained on approximately 633K audio-text pairs, producing a global 512-dimensional embedding aligned with natural language semantics. We use CLAP because its embeddings preserve important geometric properties such as compact clustering and angular separation, which remain stable after projection into the CLIP text-token space and improve cross-attention behavior.
Audio source separation. Audio source separation [6] is a separate and well-studied problem. Our method assumes that individual audio sources are already separated and does not rely on any specific source-separation model. When only mixed audio is available, any existing separation method can be used as a preprocessing step.
Method
Problem Setting
Let be a set of audio waveforms, where each and all inputs are standardised to . Let denote a prompt template such as “a photo of a and a ” and a containing placeholder tokens corresponding to the audio sources. The goal of compositional audio-to-image generation is to learn a function that produces an image that is faithful to every audio source. We formalize this through three desiderata. The first is audio fidelity (C1), which requires the object generated for each source to match its semantic content. The second is completeness (C2), which requires all sources to be visibly represented in the image. The third is spatial separation
(C3), which requires the generated objects to occupy distinct, non-overlapping regions. Our method addresses these three desiderata through three corresponding geometric mechanisms, summarised in Figure 3.
Figure 3. Method overview. Each audio clip is encoded using a frozen CLAP [15] encoder to obtain a global 512-dimensional embedding. A shared embedder projects this embedding into a 768-dimensional CLIP token (Eq. (1)), which is assigned to a unique placeholder token in the multi-token prompt (Eq. (2)). The prompt is then used to condition the frozen Stable Diffusion U-Net. During denoising, two inference-time losses improve cross-attention behavior: (Eq. (3)) increases token activations, while (Eq. (4)) encourages spatial separation between tokens. Their combined objective (Eq. (5)) addresses the failure modes shown in Figure 2. The embedder is the only trainable component in the framework.
Background
AudioToken [17] performs single-audio synthesis through a textual inversion pipeline: a frozen encoder produces an acoustic feature sequence , a small embedder (attention pooling followed by an MLP, M parameters) projects this sequence to a single embedding in the CLIP text-token space, and a frozen Stable Diffusion backbone is conditioned on a prompt with in the subject slot. The embedder is trained with the standard diffusion denoising loss while keeping both and the U-Net frozen.
The Geometry of Audio Tokens in CLIP Space
Before describing our method, we first present the main observation that motivates it. In the diffusion backbone, all conditioning tokens are processed identically. Cross-attention is computed using inner products between visual queries and conditioning keys. As a result, whether an audio token is attended to depends on where that token lies in the CLIP text-token space, relative to both (i) other tokens from the same semantic class and (ii) the natural text embedding manifold on which the diffusion model was trained. We study these relationships at two levels.
Level 1: Raw Audio Embeddings. We first extract embeddings from a 50-class subset of
VGGSound (§ 3.8) using BEATs and CLAP. We then evaluate the -normalized embeddings using standard clustering metrics: within-cluster variance (WCV, lower is better), Davies–Bouldin index (DB, lower is better), silhouette score (higher is better), and mean intra-cluster cosine similarity (higher is better).
As shown in Table 1, CLAP produces significantly better cluster structure than BEATs. The silhouette score improves by more than 3× (0.0295 vs. 0.0090), DB decreases by 21%, and WCV decreases from 0.545 to 0.470. These results indicate that BEATs places audio clips from the same class in less consistent geometric locations, which can make the downstream embedding mapping more difficult to learn.
| Raw embeddings | CLAP | BEATs |
| Global | ||
| Silhouette | 0.0295 | 0.0090 |
| Davies–Bouldin | 3.9079 | 4.9387 |
| Per-class (mean std) | ||
| WCV | 0.470 | 0.545 |
| Intra cos. sim. | 0.520 | 0.450 |
| Silhouette | 0.020 | 0.005 |
Table 1. Cluster geometry of raw embeddings (left) and projected CLIP tokens (right) on the 50-class VGGSound subset. indicate higher/lower is better, and violet marks the better result in each row. CLAP consistently produces more compact clusters and higher silhouette scores than BEATs. Although its centroids are farther from text embeddings in L2 distance, CLAP preserves cosine alignment, which is more relevant for cross-attention. These geometric improvements are preserved and further strengthened after projection into CLIP space.
Projected Tokens in CLIP Space
Next, we pass the same embeddings through the trained embedder and analyze the resulting projected tokens in CLIP space (on the right side of Table 1). The advantage of CLAP becomes even stronger after projection. The silhouette score improves from 0.16 (BEATs) to 0.35 (CLAP), while DB decreases from 2.49 to 1.28.
We further measure the distance between each class centroid and the CLIP text embedding of the corresponding class label. Two observations are important. First, CLAP tokens are farther from the text embeddings in distance (0.825 vs. 0.298), which may initially suggest weaker alignment. However, the cosine distance remains nearly identical (0.639 vs. 0.661). Since cross-attention depends on normalized inner products, angular alignment is more important than Euclidean distance. This suggests that CLAP tokens retain comparable angular alignment with the text manifold while also being more clearly separated from each other. Figure 4 visualizes this behavior.
Figure 4. UMAP visualization of audio tokens in CLIP space. For Elk Bugling (additional classes are shown in supplementary Figure S9), circles and triangles represent BEATs and CLAP tokens, respectively. Colored points denote the highlighted class, while grey points denote the remaining 49 classes. Gold and cyan stars show the BEATs and CLAP class centroids, respectively, and the red diamond shows the CLIP text embedding; dashed lines connect each centroid to the text embedding. CLAP tokens form a tighter, more separated cluster, consistent with Table 1, while its centroid is farther from the text embedding in L2 distance but similarly aligned in direction to BEATs, which is more important for cross-attention.
This geometric property offers a useful explanation for the consistent advantage of CLAP over BEATs in our experiments. More generally, it suggests that an audio encoder for diffusion conditioning does not need to exactly match text embeddings in distance. Instead, it should produce class-consistent tokens that preserve angular alignment with the text embedding.
CLAP-Based Embedder
CLAP produces a single global 512-dimensional embedding for each audio clip, unlike BEATs, which outputs a frame-level feature sequence. As a result, the original BEATs-based embedder design (attention pooling followed by an MLP) is not directly suitable for CLAP. We therefore redesign the embedder to align with CLAP’s global representation. Given an audio clip , the CLAP encoder first produces a 512-dimensional embedding. This embedding is projected into a 2304-dimensional hidden space using . The projected feature is then passed through a residual block consisting of LayerNorm, Linear, GELU, and Dropout layers. Finally, the feature is projected into the 768-dimensional CLIP token space using and -normalized to match the scale of natural CLIP text-token embeddings. A lightweight attention refinement module (FGA) is then applied for final refinement in CLIP space:
The -normalization step is important because cross-attention depends on normalized inner products and is therefore primarily sensitive to angular similarity rather than feature magnitude, as discussed in § 3.3. The embedder is the only trainable component in the pipeline (approximately 8.3M parameters), while both CLAP and the diffusion backbone remain frozen (Figure 3).
Multi-Token Extension (MTE)
To support compositional generation from multiple audio inputs, we introduce a Multi-Token Extension (MTE). We define a set of unique placeholder tokens \mathcal{T} = \{\langle* \rangle,\langle\hat{} \rangle,\langle! \rangle,\langle@ \rangle,\langle\$ \rangle\} and add them to the CLIP tokenizer by resizing the text-encoder embedding table. For each audio input , the embedder produces an audio token , which is assigned to the corresponding placeholder token . The inference prompt is then constructed as:
This allows the text encoder to process a sequence of standard text tokens together with multiple audio-derived subject tokens. MTE does not require retraining the diffusion backbone or modifying its architecture, while the embedder remains the only trainable component.
Repairing Cross-Attention Geometry at Inference Time
Even when multiple audio tokens are well separated in CLIP space, standard diffusion sampling still suffers from two common problems during generation (Figure 2): neglect, where some tokens receive very weak cross-attention, and overlap, where multiple tokens attend to the same spatial region, causing objects to merge together. We address these two issues during denoising by combining two complementary losses into a single mixed objective.
Attend-and-Excite activation (). Let denote the cross-attention map for token , averaged across attention heads and selected U-Net layers. For the set of audio tokens , the Attend-and-Excite loss [3] encourages every token to strongly activate at least one spatial location:
During sampling, the noisy latent is updated as , which increases the activation strength of weak tokens. This ensures that all audio sources are represented in the generated image, addressing criterion (C2).
A losses ().* Strong activation alone is not sufficient. Different tokens may attend to the same region, merging objects (Figure 2, column 3), and high-response regions from early denoising may drift by the end. We use soft-mask variants of the two A-STAR losses [1]. For each token we form a soft spatial mask by Gaussian-smoothing its attention map, subtracting the spatial mean, and applying a temperature-scaled sigmoid, , where is the Gaussian-smoothed attention map, its spatial mean, and . A segregation term penalizes overlap between the masks of different tokens, and a retention term keeps each token’s mask consistent across consecutive timesteps:
Together these two terms, which we denote , address C3 and reduce region collapse.
Mixed objective. The two losses serve different purposes. encourages strong activation of every token, while controls where each token appears. We combine them as:
where in all experiments. Optimization is applied during the first of 50 denoising steps, since early denoising stages determine the global scene layout, whereas later stages primarily refine texture details. As shown in § 5, the mixed objective consistently performs better than using either loss alone.
Training on a Focused 50-Class Subset
We train the embedder on a curated 50-class subset of VGGSound (25K audio–image pairs) instead of the full 309-class dataset. The main reason is representational capacity. The embedder contains only 8.3M trainable parameters, and learning mappings for all 309 classes, many of which are visually similar or acoustically ambiguous, can make optimization unstable and lead to weaker token separation. By restricting training to 50 acoustically and visually distinct classes, the task becomes easier for the embedder to learn. This helps produce cleaner gradients, more stable optimization, and tighter token clusters in CLIP space. The effect is also reflected quantitatively in the clustering results shown in Table 1.
Experiments
Datasets
VGGSound [4] is a large-scale audio-visual dataset containing approximately 200K audio clips of length 10 seconds across 309 classes, each paired with a sampled image frame. We train our embedder on a curated 50-class subset and use its test split for single-audio evaluation, which we refer to as VGG. Classes outside this training vocabulary are treated as out-of-distribution. IS3 [17] (Interactive Synthetic Sound Source) is a compositional benchmark that includes two audio clips from different classes, along with a synthesized composite reference image. We use IS3 as an out-of-distribution multi-audio benchmark because its compositions are drawn from a broader, partially non-overlapping vocabulary.
New compositional benchmarks (VGG2, VGG3, VGGn). To evaluate compositional generation under controlled settings, we construct three multi-audio benchmarks from the VGGSound test split. VGG2 contains 2500 groups, each consisting of exactly two audio-image pairs sampled from different classes. VGG3 contains 1000 groups of three audio-image pairs and supports a more detailed analysis of the three-audio setting. VGGn contains 500 groups with variable group size . To ensure balanced class coverage, we sample a group size uniformly, then select acoustically distinct classes without replacement, giving preference to classes that have not yet appeared in earlier groups. Audio-image pairs within each class are drawn sequentially, ensuring all samples are used before being reused. All splits are generated with a fixed random seed.
Metrics
We extend six standard audio-to-image metrics to the multi-source setting. For groups with audio inputs, each metric is aggregated across all objects in the group. IIS (Image–Image Similarity) measures how often the generated image is more similar, in CLIP ViT-L/14 space, to its ground-truth reference than to random references; higher is better. AIS (Audio–Image Similarity) is the analogous win rate between the generated image (CLIP ViT-B/32) and the wav2CLIP [14] embedding of each input audio, ranked against random audios. AIC (Audio–Image Content) is the recall of the ground-truth labels within the top- predictions of a zero-shot CLIP ViT-L/14 classifier over the 50 candidate classes, reducing to top-1 accuracy when . CLS (Caption-Label Similarity) is the geometric mean, over labels, of the cosine similarity between a BLIP [10]-generated caption and each class label. CCS (Caption–Caption Similarity) is the same aggregation between captions of the generated and reference images. FID [9] measures the distributional distance between generated and real images in InceptionV3 feature space, where lower values are better. Five of these six metrics use CLIP- or BLIP-derived representations, which our method also aligns to, so we report FID alongside them as a CLIP-independent check. Moreover, the coupling is indirect, since our inference-time objective acts on cross-attention maps and never optimizes CLIP image–text similarity.
Baselines and Variants
We compare our method with four existing approaches: AudioToken [7] trained on the full 309-class VGGSound dataset (denoted as ), SonicDiffusion [2], Sound2Scene [13], and MACS [18]. For a fair comparison, each baseline receives the same pre-separated audio sources as our method, so that differences reflect the generation stage rather than source-separation quality. For MACS, which includes a source separation stage, we bypass this stage and provide the same separated sources directly to its generation module. AT (309) uses the released 309-class AudioToken weights with our multi-token extension at inference, so it also produces one combined image per group and remains directly comparable. We denote our variants as follows: AT, which uses the multi-token extension with the BEATs encoder trained on the curated 50-class subset; AT Mixed, which adds the mixed inference objective ; and AT Mixed (Clap), which further replaces BEATs with the CLAP encoder. This final variant (AT Mixed (Clap)) is referred to as “Ours” in the figures. In all tables, the best result in each column is highlighted in violet, while the second-best result is shown in bold. Rows corresponding to our variants are shaded.
Main Results
The central prediction of the geometry hypothesis is that encoder substitution should substantially improve compositional fidelity, beyond the effects of inference-time optimization. The ablation results support this hypothesis. On VGGSound, AT (Clap) without any inference optimization achieves , compared to AT Mixed with the best inference strategy at AIC = 0.4477, a gap of +0.131 associated with the improved token geometry. The same ordering holds on VGG2 (0.4010 vs. 0.3926). Since SonicDiffusion and Sound2Scene are single-audio methods, only AudioToken and MACS appear on the compositional benchmarks.
Single-audio (VGG). AT Mixed (Clap) achieves the best score on 4 of 6 metrics, including the most class-discriminative one (AIC: 0.5904 vs. 0.4477 for AT Mixed and 0.2616 for MACS, a 2.3 margin). The AIC margin is particularly informative in this block, as it measures whether the generated image is recognizable as the correct audio class and gives an AIC approximately 2.3× higher for our method. IIS, CLS, and CCS also favor AT Mixed (Clap), supporting the observation that the improved CLAP-based token representation translates into more visually and semantically faithful generations. MACS retains an advantage on AIS (0.7940 vs. 0.7484), which we attribute to its joint training objective that directly optimizes audio-visual alignment. Our method trades raw alignment for class-discriminative fidelity, reflected in the substantially higher AIC scores, which we consider the more informative metric for compositional generation.
Two-audio (VGG2). The introduction of a second audio token makes attention neglect a real concern, and the value of geometric alignment becomes more visible. AT Mixed (Clap) achieves the best score on IIS, AIS, AIC, and FID, while AT Mixed is second on IIS, AIC, and FID and best on CLS and CCS, indicating that the mixed loss already produces a strong compositional baseline before CLAP is added. Against MACS, the AIC margin is (0.4332 vs. 0.2126); we improve IIS by 5.2 points and FID by 3.7. Figure 5 (middle rows) shows the corresponding qualitative pattern: our method renders both a cat and a scuba diver (row 2) or a laughing baby and a skier (row 3) in distinct regions, while AT(309) drops one source and MACS produces a single ambiguous object.

Figure 5. Qualitative comparison across different compositional settings. The first row shows on VGG with fire truck siren; the second and third rows show on VGG2 with cat purring & scuba diving and baby laughter & skiing, respectively; and the fourth row shows on VGG3 with dog barking & slot machine & hedge trimmer running. Columns are ordered from our method to the baselines. Our method generates all requested objects in separate, coherent spatial regions, whereas AT (309) and MACS often miss one source or merge multiple objects. This difference becomes more pronounced at , where the cross-attention budget is shared among more tokens.
IS3 out-of-distribution results. AT Mixed (Clap) scores below AT Mixed on AIC (0.1688 vs. 0.2131) and on CLS. This does not contradict our geometric analysis, since it follows from how each encoder behaves on unseen classes. CLAP produces a single global embedding aligned to a fixed language vocabulary, so audio outside that vocabulary may be mapped toward familiar concepts, potentially reducing the detail needed to render the correct object. BEATs is instead trained by masked acoustic-token prediction over the broad AudioSet distribution, with no text alignment, and it exposes a frame-level feature sequence rather than a single compressed vector, so it degrades more gracefully on unseen classes. The angular alignment that benefits CLAP in-distribution may therefore be less effective when the test audio falls outside the learned vocabulary, as on IS3, where the broader features of BEATs provide better robustness. Extending the embedder to broader training vocabularies is a direction for future work, as discussed in Appendix § 15.
Three-audio and variable-. The relative ranking established at is stable: AT Mixed (Clap) wins IIS, AIS, AIC, and FID on VGG3 and IIS, AIC, and FID on VGGn. Absolute scores degrade with , as expected, because more objects must share the same canvas and more tokens compete for a fixed cross-attention budget. Crucially, the AIC margin between our method and MACS remains pronounced across all values ( on VGG3, on VGGn), indicating that the benefits of CLAP-based token geometry and the mixed objective remain substantial as compositional complexity increases. The bottom row of Figure 5 shows the three-audio case: our method renders the dog, slot machine, and hedge trimmer as identifiable objects, whereas baselines collapse the scene into one or two of them.
Progressive compositions. Figure 6 grows the prompt one audio source at a time, adding one new source per panel, consistent with the quantitative drop-off in Table 2. Object retention is preserved through and remains qualitatively legible at and , even though new sources increasingly compete with prior ones for the shared attention budget. The teaser (Figure 1) shows the same pattern on the snare-drum-to-church-bell series and the church-bell-to-drum-kit series.
| Method | IIS | AIS | AIC | CLS | CCS | FID |
| VGG — single audio (in-distribution) | ||||||
| SonicDiffusion [2] | 0.4955 | 0.5137 | 0.0574 | 0.4719 | 0.4348 | 86.0037 |
| Sound2Scene [13] | 0.4873 | 0.4877 | 0.0794 | 0.5242 | 0.4776 | 112.3871 |
| AudioToken (309) [17] | 0.6310 | 0.6830 | 0.1394 | 0.5286 | 0.5067 | 93.6600 |
| MACS [18] | 0.7780 | 0.7940 | 0.2616 | 0.5615 | 0.5344 | 58.6611 |
| AT (ours) | 0.7330 | 0.7200 | 0.4337 | 0.5837 | 0.5343 | 70.0351 |
| AT Mixed (ours) | 0.7372 | 0.7215 | 0.4477 | 0.5846 | 0.5330 | 62.3452 |
| AT Mixed (Clap) (ours) | 0.8047 | 0.7484 | 0.5904 | 0.5878 | 0.5496 | 60.0736 |
| VGG2 — two audios (in-distribution compositional) | ||||||
| AudioToken (309) [17] | 0.5654 | 0.5957 | 0.1214 | 0.5005 | 0.4740 | 93.9700 |
| MACS [18] | 0.6476 | 0.6625 | 0.2126 | 0.5122 | 0.4917 | 57.8110 |
| AT (ours) | 0.6553 | 0.6550 | 0.3822 | 0.5312 | 0.4929 | 60.4968 |
| AT Mixed (ours) | 0.6701 | 0.6611 | 0.3926 | 0.5315 | 0.4938 | 55.5968 |
| AT Mixed (Clap) (ours) | 0.6993 | 0.6677 | 0.4332 | 0.5237 | 0.4902 | 54.1177 |
| IS3 — two audios (out-of-distribution, broader vocab.) | ||||||
| AudioToken (309) [17] | 0.6815 | 0.6103 | 0.1313 | 0.5240 | 0.4915 | 77.8841 |
| MACS [18] | 0.7251 | 0.6516 | 0.1337 | 0.5116 | 0.4908 | 89.3090 |
| AT (ours) | 0.7490 | 0.6174 | 0.2110 | 0.5205 | 0.5101 | 69.6100 |
| AT Mixed (ours) | 0.7520 | 0.6188 | 0.2131 | 0.5079 | 0.5035 | 66.4075 |
| AT Mixed (Clap) (ours) | 0.7291 | 0.6221 | 0.1688 | 0.4983 | 0.4914 | 69.0440 |
| VGG3 — three audios (in-distribution compositional) | ||||||
| AudioToken (309) [17] | 0.5447 | 0.5615 | 0.1233 | 0.4867 | 0.4395 | 120.7110 |
| MACS [18] | 0.5944 | 0.6098 | 0.1957 | 0.4978 | 0.4764 | 77.8314 |
| AT (ours) | 0.6237 | 0.6215 | 0.3523 | 0.5147 | 0.4783 | 76.3441 |
| AT Mixed (ours) | 0.6298 | 0.6210 | 0.3527 | 0.5111 | 0.4742 | 74.8436 |
| AT Mixed (Clap) (ours) | 0.6528 | 0.6266 | 0.3703 | 0.5026 | 0.4730 | 74.0027 |
| VGGn — variable (in-distribution, scaling) | ||||||
| AudioToken (309) [17] | 0.5351 | 0.5546 | 0.1411 | 0.4787 | 0.4326 | 122.0682 |
| MACS [18] | 0.5731 | 0.5930 | 0.1926 | 0.4982 | 0.4748 | 77.2468 |
| AT (ours) | 0.6094 | 0.5946 | 0.3366 | 0.5040 | 0.4697 | 78.1004 |
| AT Mixed (ours) | 0.6124 | 0.5973 | 0.3381 | 0.5026 | 0.4684 | 75.8900 |
| AT Mixed (Clap) (ours) | 0.6183 | 0.5898 | 0.3381 | 0.4931 | 0.4661 | 76.9760 |
Table 2. Table 2: Quantitative Comparison. Each block reports one benchmark (VGG, VGG2, IS3, VGG3, VGGn); rows are methods (baselines above, our variants shaded below) and columns are the six metrics, with / indicating whether higher or lower is better. In each column the best result is in violet and the second-best in bold.
Figure 6. Additional progressive compositions generated by our method. Each row extends a scene by adding one audio source at a time. The top row shows a four-source progression starting from scuba diving and successively adding baltimore oriole, woodpecker, and train whistling, with almost all four concepts remaining visible in the final image. The middle and bottom rows show three-source examples. Across all cases, our method preserves previously generated objects instead of overwriting them as new audio sources are added.
Ablations
We ablate the two components of the mixed loss, attend-and-excite () and the spatial A* loss (), over the BEATs and CLAP variants of the embedder. Table 3 reports the consistent trend across the four compositional benchmarks, and per-dataset breakdowns are in the supplementary Table S1–Table S5.
| Method | IIS | AIS | AIC | CLS | CCS | FID |
| AT | 0.7330 | 0.7200 | 0.4337 | 0.5837 | 0.5343 | 70.0351 |
| AT + AnE | 0.7271 | 0.7226 | 0.4333 | 0.5744 | 0.5275 | 61.5910 |
| AT + A | 0.7327 | 0.7228 | 0.4347 | 0.5833 | 0.5359 | 70.5935 |
| AT + AnE + A | 0.7372 | 0.7215 | 0.4477 | 0.5846 | 0.5330 | 62.3452 |
| AT (Clap) | 0.8052 | 0.7624 | 0.5788 | 0.5994 | 0.5600 | 55.9076 |
| AT + AnE (Clap) | 0.8064 | 0.7588 | 0.5737 | 0.5961 | 0.5574 | 60.8501 |
| AT + A (Clap) | 0.8044 | 0.7593 | 0.5723 | 0.5995 | 0.5606 | 61.6084 |
| AT + AnE + A (Clap) | 0.8047 | 0.7484 | 0.5904 | 0.5878 | 0.5496 | 60.0736 |
Table S1. Per-dataset ablation on VGGSound (VGG), single audio.
| Method | IIS | AIS | AIC | CLS | CCS | FID |
| AT | 0.6094 | 0.5946 | 0.3366 | 0.5040 | 0.4697 | 78.1004 |
| AT + AnE | 0.6084 | 0.5936 | 0.3343 | 0.4953 | 0.4637 | 74.8646 |
| AT + A | 0.6055 | 0.5982 | 0.3313 | 0.5089 | 0.4700 | 80.3390 |
| AT + AnE + A | 0.6124 | 0.5973 | 0.3381 | 0.5026 | 0.4684 | 75.8900 |
| AT (Clap) | 0.6186 | 0.5888 | 0.3308 | 0.4945 | 0.4654 | 78.4695 |
| AT + AnE (Clap) | 0.6135 | 0.5827 | 0.3368 | 0.4840 | 0.4590 | 81.0473 |
| AT + A (Clap) | 0.6122 | 0.5850 | 0.3298 | 0.5003 | 0.4685 | 80.1423 |
| AT + AnE + A (Clap) | 0.6183 | 0.5898 | 0.3381 | 0.4931 | 0.4661 | 76.9760 |
Table S5. Per-dataset ablation on VGGn, variable .
(i) primarily improves perceptual quality. Adding consistently yields the largest single-component reduction in FID, while ranking metrics change little. encourages each token to claim a meaningful spatial region, which improves perceptual coherence without necessarily affecting which class is recognized at that region.
(ii) preserves caption alignment. On compositional splits, achieves the highest CLS and CCS among single-loss variants. By forcing distinct tokens onto disjoint spatial supports, it produces cleaner per-object regions, which captioners describe more accurately.
(iii) The mixed objective is consistently best on AIC. Across VGG2, VGG3, and VGGn, AT + achieves the joint-highest AIC within each encoder family. Activation alone is insufficient (tokens fuse); separation alone is insufficient (weak tokens stay weak). Both contribute to class-correct compositional generation, which is directly reflected in the AIC metric, and the same pattern is visualized in Figure 2.
(iv) CLAP dominates in-distribution; BEATs is more robust out-of-distribution. Every CLAP variant achieves higher IIS and AIC than its BEATs counterpart with the same loss on VGG, VGG2, and VGG3. On IS3, however, BEATs variants outperform CLAP variants on
| Encoder / Loss | IIS | AIS | AIC | CLS | CCS | FID |
| BEATs encoder (AT) | ||||||
| no inference loss | 0.6593 | 0.6221 | 0.3205 | 0.5176 | 0.4878 | 71.1378 |
| + | 0.6604 | 0.6154 | 0.3159 | 0.5036 | 0.4776 | 67.4298 |
| + | 0.6597 | 0.6199 | 0.3182 | 0.5209 | 0.4822 | 71.8018 |
| + (mixed) | 0.6661 | 0.6246 | 0.3241 | 0.5133 | 0.4850 | 68.1845 |
| CLAP encoder (AT-Clap) | ||||||
| no inference loss | 0.6670 | 0.6132 | 0.3123 | 0.5104 | 0.4831 | 69.9187 |
| + | 0.6654 | 0.6148 | 0.3171 | 0.4949 | 0.4729 | 71.9798 |
| + | 0.6625 | 0.6127 | 0.3148 | 0.5162 | 0.4866 | 70.9196 |
| + (mixed) | 0.6749 | 0.6266 | 0.3276 | 0.5044 | 0.4802 | 68.5351 |
caption: tbl:3: Loss ablation averaged across compositional benchmarks (VGG2/IS3/VGG3/VGGn). For each encoder-loss combination, we report the average score across all four benchmarks. Per-dataset ablation results are provided in supplementary Table S1 − S5. Best indicates the best result within each encoder block. Two trends are consistent across experiments: (i) the mixed objective performs best or tied-best on AIC, the metric most directly related to compositional correctness, and (ii) CLAP-based variants consistently outperform their BEATs counterparts on in-distribution benchmarks.
:::
AIC and CLS, for the reasons discussed in § 4.4.
(v) The CLAP advantage narrows as grows. On VGGn, the gap between CLAP and BEATs variants is much smaller than on VGG2 and VGG3. At high the attention budget, rather than per-token quality, appears to become the limiting factor, which points to a clear direction for future work.
Due to space constraints, we provide an in-depth interpretation and discussion of these results in Supplementary Appendix § 15.
Conclusion
We addressed the problem of compositional audio-to-image generation by studying the geometry of audio tokens in the CLIP text-token space. Our analysis identifies three key properties spanning intra-class compactness, angular alignment with the text manifold, and inter-token spatial separation. We show that these properties are closely associated with common failure modes in existing methods, and we design our framework to address each. Although our method builds on existing components such as textual inversion, Attend-and-Excite, and A* separation, it combines them within a unified geometric framework. The proposed approach achieves strong performance on compositional metrics such as AIC, IIS, and FID, with AIC improvements of up to over prior methods. We also introduce three new multi-source benchmarks, VGG2, VGG3, and VGGn, to support future research in compositional audio-to-image generation. While our study focuses on audio, the proposed geometric perspective may extend to other modalities that project learned embeddings into pre-trained text spaces for cross-attention.
Limitations. Our method assumes separated audio streams and inherits the limitations of any external separation model. The embedder is trained on a curated 50-class subset, so broader vocabularies require additional training. Finally, the mixed objective acts during denoising, raising per-image cost by roughly – (Appendix § 14).
Acknowledgment
This work was supported by the Prime Minister Research Fellowship (PMRF 2122-2557) awarded to Prajwal Singh and by the Dr. Vilas Mujumdar Chair held by Shanmuganathan Raman.
Supplementary Material
This supplementary provides material that complements the main paper: (A) full implementation details and hyperparameters, (B) per-dataset ablation tables (the main paper reports the benchmark-averaged summary, Table 3), (C) additional qualitative rows for single-, two-, three-, and large- inputs (the main paper shows one representative row per regime in Figure 5), (D) cross-attention map evolution across denoising timesteps for several audio pairs and triples (the main paper shows the snapshot for one pair in Figure 2), and (E) the remaining UMAP panels of the audio-token geometry (one panel is in the main paper as Figure 4), (F) an inference-time cost analysis of the mixed objective, and (G) an extended discussion of the results.
Implementation Details
Diffusion backbone. We use Stable Diffusion v1.5 [12] as the frozen latent diffusion backbone throughout. All experiments use 50 DDIM sampling steps with a classifier-free guidance scale of 7.5, an image resolution of , and a fixed random seed per evaluation group for reproducibility.
Audio encoders. The BEATs [5] variant uses the BEATs_iter3_plus_AS2M_finetuned checkpoint and consumes raw 16 kHz waveforms of 10 s duration, producing a frame-level feature sequence of shape which is then attention-pooled by the original AudioToken [17] embedder. The CLAP [15] variant uses the official laion-clap-htsat-fused checkpoint, which produces a single 512-dimensional global embedding per clip via the audio encoder branch.
Embedder architecture (CLAP variant). The redesigned embedder (§ 3.5, Eq. (1)) operates on the global CLAP embedding and produces a single token :
Linear projection to a hidden space.
A residual block: LayerNorm Linear GELU Dropout (rate 0.05).
Linear projection to the CLIP token space.
-normalisation.
A lightweight attention refinement module (FGA), which performs a single multi-head attention pass over the normalised token using a small set of learnable CLIP-space queries.
The embedder has M trainable parameters; both CLAP and the diffusion backbone are frozen. The BEATs embedder is the original AudioToken architecture (attention pooling + MLP) and is left unchanged in its mechanics but retrained on our 50-class subset.
Training. We train the embedder with the standard diffusion denoising loss on paired (audio, image) data. Optimiser: AdamW, learning rate , weight decay , batch size 4. We train for 30,000 steps on the 50-class VGGSound subset (K pairs). Training takes hours on a single NVIDIA A100 (40 GB).
Multi-token registration (MTR). Register five placeholder tokens \mathcal{T} = \{\langle\star\rangle, \langle\wedge\rangle, \langle\vee\rangle, \langle@ \rangle, \langle\$ \rangle\} in the CLIP tokenizer (handling ). The text-encoder embedding table is resized accordingly, and the new rows are zero-initialized and overwritten with audio-derived embeddings at inference time. No fine-tuning of the text encoder is performed.
Inference-time optimization hyperparameters. Following [3], we apply optimisation in the first of 50 denoising steps. At steps we additionally run an iterative refinement of up to 20 inner iterations, with per-token activation targets of respectively. The latent step size has a base value of 20.0 and is scaled by , where decreases linearly from 1.0 to 0.5 across the schedule. The mixed-loss weighting is . Cross-attention maps are collected from the resolution U-Net layers and averaged across heads.
Clustering metrics. All cluster-geometry metrics (Table 1 in the main paper) are computed on -normalised embeddings using the implementations from scikit-learn (silhouette_score, davies_bouldin_score). Per-class WCV is computed as the mean squared distance to the class centroid in -normalized space. Intra-class cosine similarity is the mean pairwise cosine similarity within each class. UMAP visualisations (Figure 4 and Figure S9) use n_neighbors = 15, min_dist = 0.1, and the cosine metric.
Figure S9. UMAP projections of BEATs and CLAP tokens in CLIP space on the 50-class VGGSound subset. Circles represent BEATs tokens and triangles represent CLAP tokens. Colored points correspond to the highlighted class, while grey points represent the remaining classes. The gold star denotes the BEATs centroid, the cyan star denotes the CLAP centroid, and the red diamond represents the CLIP text embedding of the class label. CLAP tokens form tighter, better-separated clusters while maintaining angular alignment with the corresponding text embeddings.
Benchmark construction. VGG2 consists of 2500 groups of two audio–image pairs, each pair drawn from a distinct VGGSound class. VGG3 consists of 1000 groups of three audio–image pairs, constructed under the same constraints. VGGn consists of 500 groups of variable size , where the group size is drawn uniformly before the classes are selected. Within each group, classes are sampled without replacement so that the sources are distinct, and within each class, samples are drawn sequentially so that all clips of that class are exhausted before reuse. All three benchmarks are generated with random seed 42.
Metric computation. Let be the generated image for a group of sources. We denote by , the CLIP ViT-L/14 and ViT-B/32 image encoders, the CLIP ViT-L/14 text encoder, the wav2CLIP [14] audio encoder, and the BLIP [10] captioner (one caption per image). All embeddings are -normalised and is cosine similarity. For slot , , and are the matching reference image, audio and class label, and the negative banks and (of sizes ) are pooled from the corresponding slots of the other groups in the evaluation batch, excluding the correct item and capped at 256 entries per slot.
and are retrieval win rates. AIS compares the generated image (ViT-B/32, to match wav2CLIP’s joint space) against each input audio relative to random audios, while IIS compares the generated image (ViT-L/14) against each reference image relative to random references. We define them as
is a zero-shot CLIP classifier over the 50 curated classes with the prompt “a photo of a ”. Writing for the highest-scoring classes and for the ground-truth labels, AIC is the recall
which reduces to top-1 accuracy when .
CLS and CCS aggregate caption similarities with a geometric mean over the slots, using . CLS compares the generated-image caption(s) to each class label, while CCS compares them instead to the BLIP caption of each reference image, giving
FID [9] is accumulated across the whole benchmark rather than computed per group. Each group contributes its reference images to the real set and, paired with each reference, its single generated image to the generated set, so the generated image enters with multiplicity and the two sets stay balanced (real, gen, real, gen, ). The Fréchet distance is then computed once from the InceptionV3 statistics accumulated over all groups, with lower values indicating closer distributions.
Baseline reproduction. AudioToken [17], SonicDiffusion [2], Sound2Scene [13], and MACS [18] are run from their official public implementations with default hyperparameters. For multi-audio evaluation, AT (309) uses the released 309-class AudioToken weights together with our multi-token extension, so that a group of sources is registered to placeholder tokens in a single prompt and produces one combined image, exactly as for our own variants. Its embedder architecture is identical to that of our AT variant, so the two differ only in the training vocabulary of the embedder. SonicDiffusion and Sound2Scene accept a single audio input and are therefore reported on VGG only. Since our benchmarks already provide separated sources, we bypass the weakly supervised separation stage of MACS and pass the separated sources directly to its adapter, matching the separated-input assumption shared by all methods and isolating the comparison to the generation stage.
Per-Dataset Ablation Tables
The main paper reports the benchmark-averaged ablation in Table 3. Here we provide the per-dataset breakdown supporting that summary. Best results per column within each table are highlighted in violet, second-best in bold. “AT” refers to our multi-token AudioToken backbone with the BEATs encoder retrained on the 50-class subset; “AT (Clap)” is the same backbone with the CLAP encoder and redesigned embedder. “AnE” is ; “A*” is ; “AnE + A*” is the mixed loss.
Additional Qualitative Results
We provide qualitative comparisons across the four input regimes considered in the main paper. Each figure uses identical layout: column headers list method variants (ours baselines, left to right), and a small line below each image indicates the audio prompt for that row. The compositional fidelity gap is most pronounced in the two- and three-audio cases.
| Method | IIS | AIS | AIC | CLS | CCS | FID |
| AT | 0.6553 | 0.6550 | 0.3822 | 0.5312 | 0.4929 | 60.4968 |
| AT + AnE | 0.6638 | 0.6537 | 0.3758 | 0.5153 | 0.4826 | 53.4250 |
| AT + A | 0.6563 | 0.6531 | 0.3770 | 0.5349 | 0.4936 | 60.9600 |
| AT + AnE + A | 0.6701 | 0.6611 | 0.3926 | 0.5315 | 0.4938 | 55.5968 |
| AT (Clap) | 0.6901 | 0.6463 | 0.4010 | 0.5212 | 0.4872 | 57.0914 |
| AT + AnE (Clap) | 0.6907 | 0.6552 | 0.4112 | 0.5125 | 0.4826 | 55.3890 |
| AT + A (Clap) | 0.6793 | 0.6492 | 0.4086 | 0.5320 | 0.4976 | 58.1220 |
| AT + AnE + A (Clap) | 0.6993 | 0.6677 | 0.4332 | 0.5237 | 0.4902 | 54.1177 |
Table S2. Per-dataset ablation on VGG2, two audios.
| Method | IIS | AIS | AIC | CLS | CCS | FID |
| AT | 0.7490 | 0.6174 | 0.2110 | 0.5205 | 0.5101 | 69.6100 |
| AT + AnE | 0.7462 | 0.6042 | 0.2006 | 0.5032 | 0.4966 | 66.4122 |
| AT + A | 0.7531 | 0.6156 | 0.2108 | 0.5237 | 0.5101 | 68.7482 |
| AT + AnE + A | 0.7520 | 0.6188 | 0.2131 | 0.5079 | 0.5035 | 66.4075 |
| AT (Clap) | 0.7206 | 0.5980 | 0.1642 | 0.5096 | 0.4969 | 69.5848 |
| AT + AnE (Clap) | 0.7150 | 0.6050 | 0.1574 | 0.4907 | 0.4840 | 71.5160 |
| AT + A (Clap) | 0.7189 | 0.6010 | 0.1674 | 0.5120 | 0.4961 | 69.5840 |
| AT + AnE + A (Clap) | 0.7291 | 0.6221 | 0.1688 | 0.4983 | 0.4914 | 69.0440 |
Table S3. Per-dataset ablation on IS3 (out-of-distribution two-audio benchmark).
Cross-Attention Visualisations
We visualize how the inference-time losses shape the cross-attention geometry across denoising timesteps. Each figure shows attention maps for both audio tokens at across four loss configurations: no optimization (AT), only (AT + A), only (AT + AnE), and the mixed loss (AT + AnE + A). The progression visualizes the two-stage mechanism described in § 3.7, early timesteps determine global layout, and the mixed loss yields the clearest spatial separation of audio tokens by . The main paper shows only the snapshot of Figure S5 (as Figure 2), and the full time-evolution is provided below. In every figure, rows are timesteps and columns are loss configurations.

Figure S5. Cross-attention maps for the two-audio input Fire Truck Siren + Cat Purring. The row is reproduced as Figure 2 in the main paper.
| Method | IIS | AIS | AIC | CLS | CCS | FID |
| AT | 0.6237 | 0.6215 | 0.3523 | 0.5147 | 0.4783 | 76.3441 |
| AT + AnE | 0.6234 | 0.6100 | 0.3527 | 0.5005 | 0.4676 | 75.0173 |
| AT + A | 0.6238 | 0.6125 | 0.3537 | 0.5160 | 0.4551 | 77.1600 |
| AT + AnE + A | 0.6298 | 0.6210 | 0.3527 | 0.5111 | 0.4742 | 74.8436 |
| AT (Clap) | 0.6387 | 0.6196 | 0.3533 | 0.5164 | 0.4830 | 74.5290 |
| AT + AnE (Clap) | 0.6425 | 0.6162 | 0.3630 | 0.4924 | 0.4659 | 79.9670 |
| AT + A (Clap) | 0.6395 | 0.6155 | 0.3533 | 0.5206 | 0.4841 | 75.8300 |
| AT + AnE + A (Clap) | 0.6528 | 0.6266 | 0.3703 | 0.5026 | 0.4730 | 74.0027 |
Table S3 (PDF p. 21).
| Method | IIS | AIS | AIC | CLS | CCS | FID |
| AT | 0.6094 | 0.5946 | 0.3366 | 0.5040 | 0.4697 | 78.1004 |
| AT + AnE | 0.6084 | 0.5936 | 0.3343 | 0.4953 | 0.4637 | 74.8646 |
| AT + A | 0.6055 | 0.5982 | 0.3313 | 0.5089 | 0.4700 | 80.3390 |
| AT + AnE + A | 0.6124 | 0.5973 | 0.3381 | 0.5026 | 0.4684 | 75.8900 |
| AT (Clap) | 0.6186 | 0.5888 | 0.3308 | 0.4945 | 0.4654 | 78.4695 |
| AT + AnE (Clap) | 0.6135 | 0.5827 | 0.3368 | 0.4840 | 0.4590 | 81.0473 |
| AT + A (Clap) | 0.6122 | 0.5850 | 0.3298 | 0.5003 | 0.4685 | 80.1423 |
| AT + AnE + A (Clap) | 0.6183 | 0.5898 | 0.3381 | 0.4931 | 0.4661 | 76.9760 |
Table S4. Per-dataset ablation on VGG3, three audios.
UMAP Visualizations of the Audio-Token Geometry
We provide the remaining UMAP projections of audio tokens in CLIP space (after passing through the trained embedder) that support the geometric claims of § 3.3; the Elk Bugling panel appears in the main paper as Figure 4. In each panel, coloured points are tokens from the highlighted class; grey points are tokens from all other classes; circles are BEATs tokens and triangles are CLAP tokens; gold and cyan stars are the corresponding class centroids; the red diamond is the CLIP text embedding of the class label, and dashed lines connect centroids to the text embedding. Two phenomena are visible across all classes: (i) CLAP token clusters are consistently tighter and better separated; (ii) the cyan (CLAP) centroid is farther from the text embedding than the gold (BEATs) centroid in absolute terms, but pointing in a similar direction, exactly the L2-vs-cosine pattern reported in Table 1.
Inference-Time Cost Analysis
The mixed objective repairs cross-attention geometry during denoising rather than at training time, so it introduces a wall-clock cost at every generation. We report that cost as the time needed to produce one image, and express it as an overhead relative to sampling without inference-time optimization. We separate how this overhead scales with the number of audio sources from how it decomposes across the individual loss terms.
Measurement protocol. We time a single call to the denoising pipeline, synchronising the device immediately before and after so that no asynchronous kernel launches are attributed to the wrong interval. The measured interval therefore covers the complete 50-step generation of one image, including the attention-map collection and the latent updates, and excludes audio encoding and the embedder forward pass. All runs use the same training conditions as mentioned earlier.
Figure S1. Qualitative comparison for single-audio inputs (VGG).

Figure S2. Qualitative comparison for two-audio inputs (VGG2).

Figure S3. Qualitative comparison for three-audio inputs (VGG3).

Figure S4. Qualitative comparison for large- inputs (VGG).
Scaling with the number of sources. Table S6 reports the per-image cost of unoptimized sampling and of the full mixed objective on the three compositional benchmarks. Unoptimized sampling stays flat as grow while the mixed objective is close to three times slower, and that ratio barely moves between VGG2 and VGG. Most of the added time goes into the backward pass used to update the latent, and the cost of that pass is set by the size of the
| Method | VGG2 () | VGG3 () | VGGn () |
| AT (Clap), no inference loss | 4.082 | 4.030 | 4.032 |
| AT + (Clap), ours | 11.408 | 12.027 | 12.356 |
| Relative overhead |
Table S6. Per-image inference cost as the number of audio sources grows. Wall-clock seconds per generated image, averaged over all groups of each benchmark. The last row reports the ratio of the mixed objective to sampling without inference-time optimization on the same benchmark.

Figure S6. Cross-attention maps for the two-audio input Scuba Diving + Playing Drum Kit.

Figure S7. Cross-attention maps for the three-audio input Baby Crying + Hail + Car Engine Knocking.

Figure S8. Cross-attention maps for the three-audio input Baby Laughter + Scuba Diving + Train Horning.
U-Net rather than by how many tokens are being optimized. The overhead therefore behaves like a constant multiplier, so the method stays practical as more sources are added.
Decomposition by loss component. Table S7 separates the two loss terms on VGGn. is much cheaper than the mixed objective, so most of the cost comes from the activation term . This is because of how is applied rather than how it is computed. Following [3], we run up to 20 inner iterations at three of the denoising steps until every token reaches its activation target, and each iteration costs another forward and backward pass through the U-Net. instead applies a single latent update per optimized step. Since the cost sits in only a few steps, it can be lowered without retraining by reducing or the number of inner iterations.
| Configuration | Time / image (s) | Relative |
| AT (Clap) | 4.032 | |
| AT + (Clap) | 7.272 | |
| AT + (Clap) | 12.356 |
Table S7. Cost decomposition by inference-time loss component, measured on VGGn under the same protocol as Table S6. Each row corresponds to the matching row of the CLAP block of the ablation (Table S5), so the accuracy of a configuration can be read directly against its cost.
Discussion
Why does the geometric framing explain the results? The ablation results closely follow the three geometric properties introduced in § 3. CLAP produces more compact clusters than BEATs at both levels of the clustering analysis (Table 1 and Figure 4), and this directly matches the improvement in AIC across both single-audio and compositional benchmarks. Angular alignment with the text manifold also explains why the larger L2 distance of CLAP tokens does not hurt performance. Although CLAP tokens are farther from text embeddings in Euclidean distance, their angular alignment remains strong, which is what matters for cross-attention.
Finally, inter-token spatial separation controlled by improves metrics related to clean object separation, such as CLS and CCS. The ablation results confirm this behavior. Overall, the geometric framing yields clear, testable predictions that are consistently supported by the experiments.
Why is the CLAP advantage smaller on IS3? The reversal on IS3 comes from how the two encoders are trained rather than from the size of the embedder’s training vocabulary. CLAP is optimized contrastively to align each clip with a fixed set of language concepts and compresses it into a single global embedding. This is what gives CLAP its tight, well-separated clusters and its strong in-distribution performance, but the alignment only holds for concepts seen during pretraining. When an IS3 clip belongs to a class outside that vocabulary, its embedding drifts toward the nearest familiar concept, and the projected token loses the discriminative detail needed to render the correct object.
BEATs is instead trained by masked acoustic-token prediction over the broad AudioSet distribution, with no commitment to any text vocabulary, and it exposes a frame-level feature sequence rather than one compressed vector. Its representations vary smoothly with acoustic content and degrade more gracefully on unseen classes, which gives the embedder more grounded signal to map an out-of-distribution sound onto a still-recognizable token. The CLAP advantage is therefore largest when the test audio lies on the learned manifold and shrinks once it moves off, as on IS3.
Why does performance drop at larger ? As the number of audio sources increases ( or 5), the cross-attention at each spatial location must be shared across more tokens. Even when tokens are well separated in CLIP space, some tokens may still receive insufficient attention during generation.
The loss helps reduce overlap by spreading attention maps across different regions, but it cannot increase the total available attention. Higher image resolutions or explicit control of per-token attention allocation may help address this issue in future work.
Why are CLS and CCS not always the highest for our method? CLS and CCS depend on BLIP-generated captions, which summarize the entire image into a single sentence. As scene complexity increases, these captions become less reliable at correctly describing every individual object. This can lower CLS and CCS scores even when the generated objects are visually accurate.
Methods that generate simpler scenes may therefore obtain artificially better scores on these metrics. For this reason, we consider AIC the most reliable metric for compositional generation, as it directly assesses whether each object class is correctly recognized in the generated image.
References
- [1]Aishwarya Agarwal, Srikrishna Karanam, K J Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. A-STAR: Test-time attention segregation and retention for text-to-image synthesis. arXiv preprint arXiv:2306.14544, 2023.arxiv.org/abs/2306.14544
- [2]Burak Can Biner, Farrin Marouf Sofian, Umur Berkay Karakaş, Duygu Ceylan, Erkut Erdem, and Aykut Erdem. SonicDiffusion: Audio-driven image generation and editing with pretrained diffusion models. arXiv preprint arXiv:2405.00878, 2024.arxiv.org/abs/2405.00878
- [3]Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (SIGGRAPH), 42(4), 2023.arxiv.org/abs/2301.13826
- [4]Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. VGG-Sound: A large-scale audio-visual dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
- [5]Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Proceedings of the International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2212.09058
- [6]Alexandre Défossez. Hybrid spectrogram and waveform source separation. ISMIR Workshop on Music Source Separation, 2021.arxiv.org/abs/2111.03600
- [7]Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun R. Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2212.05032
- [8]Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2208.01618
- [9]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
- [10]Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning (ICML), 2022.arxiv.org/abs/2201.12086
- [11]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2103.00020
- [12]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2112.10752
- [13]Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh. Sound to visual scene generation by audio-to-visual latent alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.arxiv.org/abs/2303.17490
- [14]Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2CLIP: Learning robust audio representations from CLIP. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
- [15]Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
- [16]Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. BoxDiff: Text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.arxiv.org/abs/2307.10816
- [17]Guy Yariv, Itai Gat, Lior Wolf, Yossi Adi, and Idan Schwartz. AudioToken: Adaptation of text-conditioned diffusion models for audio-to-image generation. In Proc. INTERSPEECH, pages 5446–5450, 2023.
- [18]Hao Zhou, Yuxuan Cheng, Zixuan Yu, and Yiwen Tian. MACS: Multi-source audio-to-image generation with contextual significance and semantic alignment. arXiv preprint arXiv:2503.10287, 2025.arxiv.org/abs/2503.10287




