betaFeedback

WorksComputer Science

ODC: Orthogonal Drift Correction for Improved Text-to-Image Semantic Alignment at Inference

Abstract

Text-to-image models have achieved remarkable success in generating high-quality images from textual descriptions. However, they often struggle with "semantic drift," where the generated output fails to precisely align with complex or nuanced text prompts. While recent approaches have attempted to address semantic errors regarding attribute binding or object presence, there remains a gap for a more holistic method that addresses these issues by directly refining the text embeddings of the initial user prompt. In this work, we introduce Orthogonal Drift Correction (ODC), an inference-time guidance method designed to mitigate semantic drift without requiring model retraining or additional user inputs. ODC guides image generation through a two-stage process. In the first stage, it identifies the semantic drift by evaluating the initially generated image against the user prompt in a shared vision-language embedding space. It then isolates the component of this drift vector that is orthogonal to the prompt's direction and translates it back into text via a vocabulary-based surrogate mechanism. In the second stage, it produces refined text conditioning for a second generation pass by feeding both the initial text embedding and the re-embedded drift representation into an adaptive rank-reduced concept removal module. Our experiments demonstrate that ODC consistently improves prompt-image alignment, increasing CLIP scores by up to 5.4% over the baseline on the COALBench-300 and PartiPrompts benchmarks. Furthermore, on the T2I-CompBench complex subset, ODC improves the average 3-in-1 score by 4.5% over the competing methods. As a plug-and-play module, ODC offers a practical and efficient method for improving the reliability of text-to-image models. Code is available at https://github.com/kasrakamali/ODC.

Introduction

Modern text-to-image (T2I) models have opened new creative possibilities, allowing people to generate compelling images that would have been impossible to generate just a few years ago. Models such as Stable Diffusion (Rombach et al., 2022a), Imagen (Saharia et al., 2022), and DALL-E 2 ([25]) have demonstrated an unprecedented ability to synthesize visually compelling and artistic images from natural language descriptions, unlocking wide-ranging applications across numerous domains.

However, despite these profound capabilities, current state-of-the-art T2I models frequently fall short in generating images that precisely align with the full semantic meaning and composition of their input text prompts. This lack of precise semantic alignment, a phenomenon we term “semantic drift”, pertains to the frequent failure of these models to precisely adhere to user prompts. For instance, they often misinterpret complex compositional instructions, leading to incorrect attribute binding (e.g., wrong material on objects), catastrophic neglect ([3]) (omitting subjects from the prompt), or a failure to adhere to specified spatial relationships. This alignment gap forces users into frustrating trial-and-error prompt engineering, a practice that underscores the limitations of such models in accurately interpreting semantic intent.

This difficulty largely stems from how T2I models process textual information. State-of-the-art T2I diffusion models employ a U-Net architecture with cross-attention layers (Rombach et al., 2022a), which are crucial for fusing visual and textual features. While these layers encode rich semantic relations, the interaction can sometimes skew feature representations, causing semantic misalignment between the input and output. A variety of approaches have been proposed to address this problem. Fine-tuning methods like DreamBooth (Ruiz et al., 2023) and CoMat ([16]), personalize models for specific subjects but can be computationally demanding. Architectural expansions like ControlNet (Zhang et al., 2023) and T2I-Adapter ([21]) provide explicit spatial control through auxiliary inputs but require additional inputs and specialized training. Alternatively, a growing class of methods focuses on training-free, inference-time manipulation of attention maps or latent representations. Techniques such as Attend-and-Excite ([3]), CONFORM ([20]), and ToMe ([14]) directly intervene in the generation process to ensure robust semantic binding and alignment. While effective for specific tasks such as subject presence and attribute binding, they do not address the misalignment caused by imperfect initial text embeddings. This body of work reveals a clear gap for a method that can efficiently improve text-image alignment by directly correcting the text embedding to resolve complex compositional failures.

In this paper, we address this gap by introducing Orthogonal Drift Correction (ODC), a unique inference-time technique that directly corrects the text conditioning vector to prevent semantic drift. Our approach is based on the central hypothesis that semantic drift originates from the components of the computed text embedding that are orthogonal to the prompt’s true semantic direction. ODC guides the image generation process through a two-stage mechanism. First, it generates an initial candidate image. Then, it leverages the shared embedding space of a pre-trained vision-language model, to identify the semantic error vector between the generated image and the prompt. Crucially, instead of using the entire error, it isolates its orthogonal component via vector rejection. This component, which intuitively represents the “off-topic” deviation, is then removed from the initial text embedding. The resulting purified embedding guides a second generation pass, producing a final image with substantially higher prompt-image alignment. Our contributions can be summarized as follows:

  • We identify orthogonal semantic drift in text embeddings as a significant source of prompt-image misalignment in text-to-image models, and propose a model-agnostic, training-free, inference-time method that corrects the text conditioning embeddings without modifying the diffusion process or requiring fine-tuning.

  • We propose an adaptive rank-reduced concept removal strategy that refines the text conditioning by processing the initial text embedding alongside a re-embedded drift representation, obtained via a vocabulary-based surrogate mechanism.

  • We introduce COALBench-300 (COmpositional ALignment Benchmark), 300 synthetically generated prompts curated to rigorously probe semantic drift and prompt-image alignment. To support our method, we curate a large-scale vocabulary of over 300,000 words and phrases, compiled from multiple sources and refined through several preprocessing steps, to effectively capture and mitigate semantic drift.

Background & Related Work

We provide the necessary context for understanding our proposed method below.

Prompt-Image Alignment. Prompt-image alignment refers to the faithfulness of a generated image to the full semantic meaning and compositional structure of its input text prompt. While the goal of Text-to-Image (T2I) models is to translate natural language into visually aligned results, achieving precise alignment in terms of the spatial composition, complex layouts, poses, shapes, and forms described in a prompt remains a significant challenge ([15] (Saharia et al. 2022)). This often necessitates numerous trial-and-error cycles from users to achieve their desired output. Our method contributes to improving prompt-image alignment without requiring these cycles.

Text-to-Image Models. The dominant architecture in modern T2I generation is the Latent Diffusion Model (LDM) (Rombach et al., 2022b). Unlike earlier diffusion models that operated in the high-dimensional pixel space ([11]), LDMs perform the computationally intensive denoising process in the compressed latent space of a pre-trained variational autoencoder (VAE). This significantly reduces computational overhead while maintaining high-fidelity output.

The core of an LDM’s denoising process is a U-Net architecture (Ronneberger et al., 2015), which comprises a series of downsampling and upsampling blocks with residual connections. Interspersed within this network are self-attention layers, which capture global spatial dependencies, and crucially, cross-attention layers. These cross-attention layers are the primary mechanism through which textual guidance is integrated into the image generation process. They allow the model to attend to different parts of the text prompt at each denoising step, conditioning the visual features being generated. More advanced models like Stable Diffusion XL (SDXL) [22] enhance this architecture with a larger U-Net and a dual text encoder setup to achieve superior performance.

Text Embeddings and Conditioning. The journey from a text prompt to a guiding signal begins with tokenization, where the input string is converted into a sequence of discrete tokens. These tokens are then mapped to high-dimensional embedding vectors. This sequence of vectors is processed by a powerful text encoder, such as the CLIP text encoder [23] or a larger language model like T5 [24], to produce the final conditioning embedding.

This text embedding conditions the denoising U-Net at multiple resolutions via the cross-attention mechanism. At each cross-attention layer, the intermediate visual features (the query) attend to the text embedding (the key and value). This process generates spatial attention maps that define a rich semantic relationship between image regions and prompt tokens, critically influencing the final image’s composition. Our method, intervenes at the very beginning of this pipeline. It is plugged into the process by modifying the conditioning embedding before it is passed to the U-Net, thereby correcting the guidance signal at its source.

Previous Attempts to Improve Alignment. The challenge of prompt-image misalignment has inspired a wide variety of solutions. Prior work can be broadly categorized into three categories: prompt engineering, training-based methods, and inference-time guidance.

Prompt Engineering: At the most fundamental level, users engage in manual prompt engineering, iteratively refining text to achieve desired results. More structured approaches include prompt weighting, which allows users to amplify or attenuate the semantic influence of specific words, and the use of negative prompts. Negative prompts modify the unconditional embedding used in classifier-free guidance to explicitly steer the model away from unwanted concepts [10]. The efficacy of these prompts depends heavily on the capacity of the text encoder to capture complex relationships. Saharia et al. (2022) demonstrated that scaling the frozen text encoder (e.g., T5-XXL) yields significantly larger improvements in alignment than scaling the diffusion model itself. Beyond manual refinement and architectural scaling, researchers have also explored optimizing the input space directly to represent specific concepts. Textual Inversion [5] optimizes new pseudo-words within the embedding space of a frozen model to capture unique subjects or styles, acting as a bridge between the user’s intent and the model’s vocabulary.

Training-Based Methods: A powerful but resource-intensive approach involves fine-tuning the model. DreamBooth (Ruiz et al., 2023) fine-tunes the entire diffusion model to bind a unique identifier with a specific subject, allowing for the generation of that subject in novel contexts with high fidelity. To mitigate the high computational cost of full fine-tuning, Low-Rank Adaptation (LoRA) [13] freezes pre-trained weights and injects trainable rank-decomposition matrices, enabling efficient adaptation to new tasks or styles with minimal parameter updates. While these methods focus on personalization, other works target general alignment failures such as concept ignorance. CoMat [16] introduces an end-to-end fine-tuning strategy that leverages an image-to-text captioning model to minimize the discrepancy between the prompt and the generated image’s caption, aligning the model to revisit ignored concepts and correct attribute mismapping.

Inference-Time Guidance: This category includes methods that, like ours, operate at inference time without modifying model weights. The foundational technique in this category is Classifier-Free Guidance (CFG) [10], which mixes conditional and unconditional score estimates to trade off sample diversity for higher text alignment. Building on this, Chung et al. (2024) identified that standard CFG suffers from “off-manifold” issues that harm editability and alignment, proposing CFG++ to correct the sampling trajectory to remain on the data manifold. Beyond sampling adjustments, several methods intervene specifically in the cross-attention layers or latent optimization during inference to correct semantic failures. Hertz et al. [8] introduced Prompt-to-Prompt, which injects cross-attention maps from a source generation into a new generation to maintain structural layout while editing semantic content. To address “catastrophic neglect”, Attend-and-Excite (Chefer et al., [3]) optimizes the latent at each timestep to ensure all subject tokens maximize their attention activations. Similarly, CONFORM (Meral et al., [20]) applies a contrastive objective during inference to segregate objects in attention maps while keeping attributes close to their subjects. Taking a different approach to binding, Hu et al. [14] proposed Token Merging (ToMe), which merges relevant tokens into a composite token and iteratively updates this embedding during the early denoising steps using semantic binding and entropy losses. Finally, rather than optimizing the denoising path, InitNO (Guo et al., [7]) focuses on the initialization phase. It partitions the initial latent space into valid and invalid regions based on attention conflict scores and optimizes the initial noise vector to ensure it is predisposed toward semantically faithful generation before denoising begins.

Crucially, our work differs from existing inference-time approaches in its point of intervention. Whereas prior methods act by altering the internal dynamics of the U-Net denoising process, our approach is simpler and more direct: we refine the conditioning embeddings before diffusion begins. In contrast to training-based approaches, our method is entirely training-free, requiring no fine-tuning or additional optimization. Similarly, unlike attention-manipulation techniques, our intervention operates directly on the text embeddings, making it straightforward to implement and agnostic to the choice of U-Net architecture. By correcting semantic drift at its source, ODC provides a generalizable and principled way to improve the alignment and reliability of pre-trained text-to-image models.

Orthogonal Drift Correction

Our proposed method, Orthogonal Drift Correction (ODC), is an inference-time guidance technique designed to enhance the semantic alignment of text-to-image models. The central hypothesis is that suboptimal prompt alignment, or semantic drift, stems from the components of the initial text embedding that are orthogonal to the prompt’s primary semantic direction. ODC operates in a two-stage process. An initial image is generated and then evaluated against the prompt in a shared vision-language embedding space. The identified semantic error is then used to refine the initial text embedding for a second, more accurate generation pass.

Notations

Let ptxtp_{\mathrm{txt}} denote the input text prompt. The text-to-image model consists of two text encoders, a token-level encoder Etok\mathcal{E}_{\mathrm{tok}} and a pooled encoder Epool\mathcal{E}_{\mathrm{pool}}, together with an image generator G\mathcal{G} (typically a U-Net implementing a reverse diffusion process). The standard generation process first encodes the text prompt as

Ep=Etok(ptxt),ep=Epool(ptxt),(1)\mathbf{E}_{p} = \mathcal{E}_{\mathrm{tok}}(p_{\mathrm{txt}}), \qquad\mathbf{e}_{p} = \mathcal{E}_{\mathrm{pool}}(p_{\mathrm{txt}}), \tag*{(1)}

where Ep∈RL×H\mathbf{E}_{p} \in\mathbb{R}^{L \times H} is the sequence of per-token embeddings and ep∈RH\mathbf{e}_{p} \in\mathbb{R}^{H} is a pooled sentence-level vector. The image generator then produces an initial image II conditioned on both representations:

I=G(Ep,ep).(2)I = \mathcal{G}(\mathbf{E}_{p}, \mathbf{e}_{p}). \tag*{(2)}

To evaluate and correct for semantic drift, we employ a pre-trained vision-language model (VLM), denoted EVLM\mathcal{E}_{\mathrm{VLM}}, which embeds both text and images into a shared multimodal space.

The Orthogonal Drift Correction (ODC) Algorithm

The ODC algorithm consists of two main stages, comprising six sequential steps in total. These steps are described in detail below and summarized in Algorithm 1. An overview of the process is illustrated in Figure 1.

Figure 1. The workflow of Orthogonal Drift Correction (ODC). In the first stage, vocabulary items that best capture the orthogonal drift vector are retrieved. In the second stage, these vocabulary items are used in a concept removal operation, producing embeddings that better align with the user’s text prompt and serve as input for the final image generation.

Step 1: Initial Image Generation. First, an initial image II is generated using the standard process. The text prompt ptxtp_{\mathrm{txt}} is encoded by two complementary text encoders: one produces a sequence of per-token embeddings Ep\mathbf{E}_{p}, while the other yields a pooled sentence-level representation ep\mathbf{e}_{p}. Both conditioning signals are then provided to the image generator G\mathcal{G} to synthesize the initial image II.

Step 2: Semantic Drift Vector Calculation. Next, we quantify the semantic discrepancy between the generated image II and the original prompt ptxt\mathbf{p}_{\mathrm{txt}}. We use EVLM\mathcal{E}_{\mathrm{VLM}} to project both onto a shared embedding space. The semantic drift vector, vdrift\mathbf{v}_{\mathrm{drift}}, captures the discrepancy between these two embeddings. This vector essentially encodes the semantic error, i.e., the direction and magnitude of the deviation of the image’s content from the prompt’s intent as perceived by the VLM.

vtext=EVLM(ptxt),vimg=EVLM(I),vdrift=vimg−vtext.(3)\mathbf{v}_{\mathrm{text}} = \mathcal{E}_{\mathrm{VLM}}(\mathbf{p}_{\mathrm{txt}}), \qquad\mathbf{v}_{\mathrm{img}} = \mathcal{E}_{\mathrm{VLM}}(I), \qquad\mathbf{v}_{\mathrm{drift}} = \mathbf{v}_{\mathrm{img}} - \mathbf{v}_{\mathrm{text}}. \tag*{(3)}

Step 3: Orthogonal Drift Component Isolation and Vocabulary Retrieval. For vdrift\mathbf{v}_{\mathrm{drift}}, we hypothesize that the most detrimental error components are those that introduce concepts extraneous to the prompt, which correspond to the direction orthogonal to the prompt’s own embedding, vtext\mathbf{v}_{\mathrm{text}}. To isolate those components, we estimate the vector rejection of vdrift\mathbf{v}_{\mathrm{drift}} from vtext\mathbf{v}_{\mathrm{text}}. This is achieved by subtracting the projection of vdrift\mathbf{v}_{\mathrm{drift}} onto vtext\mathbf{v}_{\mathrm{text}} from vdrift\mathbf{v}_{\mathrm{drift}} itself. The projection, vproj\mathbf{v}_{\mathrm{proj}}, is calculated as

vproj=(vdrift⋅vtext∥vtext∥2)vtext.(4)\mathbf{v}_{\mathrm{proj}} = \left(\frac{\mathbf{v}_{\mathrm{drift}} \cdot\mathbf{v}_{\mathrm{text}}}{\lVert\mathbf{v}_{\mathrm{text}}\rVert^2}\right)\mathbf{v}_{\mathrm{text}}. \tag*{(4)}

The orthogonal drift vector, vorth\mathbf{v}_{\mathrm{orth}}, is then the rejection:

vorth=vdrift−vproj.(5)\mathbf{v}_{\mathrm{orth}} = \mathbf{v}_{\mathrm{drift}} - \mathbf{v}_{\mathrm{proj}}. \tag*{(5)}

To obtain a discrete, textual surrogate for vorth\mathbf{v}_{\mathrm{orth}}, we retrieve candidate terms from a custom vocabulary. To this end, we construct a large-scale vocabulary retrieval dataset designed to cover both visually grounded concepts and general linguistic phrases. The dataset construction process involves several stages of aggregation and filtering. We provide details of the dataset and its construction in § 4.1. Each vocabulary item ww is associated with an embedding ew\mathbf{e}_w, and cosine similarities are computed as

s(w)=vorth⋅ew∥vorth∥∥ew∥.(6)s(w) = \frac{\mathbf{v}_{\mathrm{orth}} \cdot\mathbf{e}_w}{\lVert\mathbf{v}_{\mathrm{orth}}\rVert\lVert\mathbf{e}_w\rVert}. \tag*{(6)}

The top-kk terms with the highest similarity scores are selected, forming a discrete approximation of vorth\mathbf{v}_{\mathrm{orth}}. We denote this retrieved set of tokens as Sorth\mathcal{S}_{\mathrm{orth}}.

Step 4: Approximating Orthogonal Drift in Text Encoder Space. The retrieved items SorthS_{\mathrm{orth}} are then re-embedded via the text encoders of the text-to-image model, yielding a per-token embedding matrix Ei\mathbf{E}_i and a pooled embedding vector ei\mathbf{e}_i for each item wi∈Sorthw_i \in S_{\mathrm{orth}}. Since SorthS_{\mathrm{orth}} typically contains more than one item, we combine them using a softmax-weighted pooling to obtain equivalent representative embeddings:

Eorth=∑iβiEi,eorth=∑iβiei,(7)\mathbf{E}_{\mathrm{orth}} = \sum_i \beta_i \mathbf{E}_i,\qquad\mathbf{e}_{\mathrm{orth}} = \sum_i \beta_i \mathbf{e}_i, \tag*{(7)}

where the weights βi\beta_i are defined by the normalized cosine similarity scores:

βi=exp⁡(s(wi))∑jexp⁡(s(wj)).(8)\beta_i = \frac{\exp(s(w_i))}{\sum_j \exp(s(w_j))}. \tag*{(8)}

This results in a pair (Eorth,eorth)(\mathbf{E}_{\mathrm{orth}}, \mathbf{e}_{\mathrm{orth}}) that represents the surrogate set in the embedding space of the text encoders of our T2I model. These representative embeddings, in addition to the original embeddings, are supplied to the concept removal module.

Step 5: Adaptive Rank-Reduced Concept Removal. To remove undesired semantic concepts from the text embeddings, we employ an adaptive rank-reduction approach that automatically identifies and retains the meaningful semantic components while filtering out noise. We denote this operator by R\mathcal{R}, and the procedure of its computation is summarized in Algorithm 2.

Given embedding tensors E1,E2∈RB×L×H\mathbf{E}_1, \mathbf{E}_2 \in\mathbb{R}^{B \times L \times H} (batch size BB, sequence length LL, hidden dimension HH), we first determine the semantic rank of E2\mathbf{E}_2 through QR decomposition. For each batch element:

E2T=QR,(9)\mathbf{E}_2^{T} = \mathbf{Q}\mathbf{R}, \tag*{(9)}

where Q\mathbf{Q} contains orthonormal basis vectors. We identify semantically meaningful components by selecting basis vectors whose relative magnitude exceeds ζ=1%\zeta= 1\% of the primary component:

k=max⁡{i:∣rii∣∣r11∣>ζ},(10)k = \max\left\{ i : \frac{\lvert r_{ii}\rvert}{\lvert r_{11}\rvert} > \zeta\right\}, \tag*{(10)}

where riir_{ii} are the diagonal elements of R\mathbf{R}. We selected this threshold empirically to separate information from noise. Our analysis shows that while the majority of energy is concentrated in the first component, semantic distinctions are spread across several smaller components (ranks 3–15).

We then reconstruct a denoised version using only the top-kk basis vectors:

E2reduced=(E2Q:,:k)Q:,:kT.(11)\mathbf{E}_2^{\mathrm{reduced}} = \left(\mathbf{E}_2\mathbf{Q}_{:,:k}\right)\mathbf{Q}_{:,:k}^{T}. \tag*{(11)}

The concept removal operates on flattened representations to capture global semantic relationships. Let E~1=flatten⁡(E1)\widetilde{\mathbf{E}}_1 = \operatorname{flatten}(\mathbf{E}_1) and E~2=flatten⁡(E2reduced)\widetilde{\mathbf{E}}_2 = \operatorname{flatten}(\mathbf{E}_2^{\mathrm{reduced}}). We compute:

E~out=E~1−αRej⁡E~1(E~2)=E~1⋅(1+α⋅⟨E~1,E~2⟩∥E~1∥22)−α⋅E~2,(12)\widetilde{\mathbf{E}}_{\mathrm{out}} = \widetilde{\mathbf{E}}_1 - \alpha\operatorname{Rej}_{\widetilde{\mathbf{E}}_1}(\widetilde{\mathbf{E}}_2) = \widetilde{\mathbf{E}}_1 \cdot\left(1 + \alpha\cdot\frac{\langle\widetilde{\mathbf{E}}_1, \widetilde{\mathbf{E}}_2\rangle}{\lVert\widetilde{\mathbf{E}}_1\rVert_2^2}\right) - \alpha\cdot\widetilde{\mathbf{E}}_2, \tag*{(12)}

where α\alpha controls the removal strength. The output is reshaped to the original dimensions. This adaptive approach automatically adjusts to concept complexity, i.e., simple concepts require fewer ranks while complex concepts utilize more, achieving robust removal without over-filtering.

Step 6: Final Image Generation. Finally, a new image I∗I^* is generated using the refined conditioning embeddings (Erefined,erefined)(\mathbf{E}_{\mathrm{refined}}, \mathbf{e}_{\mathrm{refined}}) produced by the concept removal module. The image generator G\mathcal{G} then synthesizes:

I∗=G(Erefined,erefined).(13)I^* = \mathcal{G}(\mathbf{E}_{\mathrm{refined}}, \mathbf{e}_{\mathrm{refined}}). \tag*{(13)}

By incorporating the correction derived from the surrogate embeddings, the refined conditioning better suppresses drift components while preserving the original semantic intent of the prompt.

Algorithm 1 Orthogonal Drift Correction (ODC)

  1. Input: Prompt ptxtp_{\mathrm{txt}}, vocabulary V\mathcal{V}

  2. Output: Final image I∗I^{*}

  3. (Ep,ep)←(Etok(ptxt),Epool(ptxt))(E_p,e_p) \leftarrow(\mathcal{E}_{\mathrm{tok}}(p_{\mathrm{txt}}),\mathcal{E}_{\mathrm{pool}}(p_{\mathrm{txt}}))

  4. I←G(Ep,ep)I \leftarrow\mathcal{G}(E_p,e_p)

  5. vtxt←EVLM(ptxt)v_{\mathrm{txt}} \leftarrow\mathcal{E}_{\mathrm{VLM}}(p_{\mathrm{txt}})

  6. vimg←EVLM(I)v_{\mathrm{img}} \leftarrow\mathcal{E}_{\mathrm{VLM}}(I)

  7. vdrift←vimg−vtxtv_{\mathrm{drift}} \leftarrow v_{\mathrm{img}}-v_{\mathrm{txt}}

  8. vproj←vdrift⋅vtxt∥vtxt∥2vtxtv_{\mathrm{proj}} \leftarrow\frac{v_{\mathrm{drift}}\cdot v_{\mathrm{txt}}}{\lVert v_{\mathrm{txt}}\rVert^{2}}v_{\mathrm{txt}}

  9. vorth←vdrift−vprojv_{\mathrm{orth}} \leftarrow v_{\mathrm{drift}}-v_{\mathrm{proj}}

  10. for each w∈Vw \in\mathcal{V} do

  11. ew←embedding⁡(w)e_w \leftarrow\operatorname{embedding}(w)

  12. s(w)←vorth⋅ew∥vorth∥∥ew∥s(w) \leftarrow\frac{v_{\mathrm{orth}}\cdot e_w}{\lVert v_{\mathrm{orth}}\rVert\lVert e_w\rVert}

  13. end for

  14. Sorth←\mathcal{S}_{\mathrm{orth}} \leftarrow Top-kk items by s(w)s(w)

  15. Re-embed Sorth\mathcal{S}_{\mathrm{orth}}: {(Ei,ei)}wi∈Sorth\{(E_i,e_i)\}_{w_i\in\mathcal{S}_{\mathrm{orth}}}

  16. Compute weights αi←exp⁡(s(wi))∑jexp⁡(s(wj))\alpha_i \leftarrow\frac{\exp(s(w_i))}{\sum_j\exp(s(w_j))}

  17. Eorth←∑iβiEi,eorth←∑iβieiE_{\mathrm{orth}} \leftarrow\sum_i\beta_iE_i,\quad e_{\mathrm{orth}} \leftarrow\sum_i\beta_ie_i

  18. (Erefined,erefined)←R((Ep,ep),(Eorth,eorth))(E_{\mathrm{refined}},e_{\mathrm{refined}}) \leftarrow\mathcal{R}((E_p,e_p),(E_{\mathrm{orth}},e_{\mathrm{orth}}))

  19. I∗←G(Erefined,erefined)I^{*} \leftarrow\mathcal{G}(E_{\mathrm{refined}},e_{\mathrm{refined}})

  20. return I∗I^{*}

Algorithm 2 Adaptive Rank-Reduced Concept Removal (R\mathcal{R})

  1. Input: E1,E2∈RB×L×HE_1,E_2 \in\mathbb{R}^{B\times L\times H}, e1,e2∈RB×He_1,e_2 \in\mathbb{R}^{B\times H}, α∈[0,1]\alpha\in[0,1], ζ=0.01\zeta=0.01; bounds kmin⁡=3k_{\min}=3, kmax⁡=15k_{\max}=15

  2. Output: Eout∈RB×L×HE_{\mathrm{out}} \in\mathbb{R}^{B\times L\times H}, eout∈RB×He_{\mathrm{out}} \in\mathbb{R}^{B\times H}

  3. for b=1b=1 to BB do

  4. (Q(b),R(b))←QR_decomposition⁡((E2(b))T)(Q^{(b)},R^{(b)}) \leftarrow\operatorname{QR\_decomposition}((E_2^{(b)})^{\mathsf{T}})

  5. r←diag⁡(∣R(b)∣)r \leftarrow\operatorname{diag}(\lvert R^{(b)}\rvert)

  6. k←max⁡{i:ri/r1>ζ}k \leftarrow\max\{i:r_i/r_1>\zeta\}

  7. k←clip⁡(k,kmin⁡,kmax⁡)k \leftarrow\operatorname{clip}(k,k_{\min},k_{\max})

  8. Let Qk(b)Q_k^{(b)} be the first kk columns of Q(b)Q^{(b)}

  9. E2,reduced(b)←(E2(b))TQk(b)(Qk(b))TE_{2,\mathrm{reduced}}^{(b)} \leftarrow(E_2^{(b)})^{\mathsf{T}}Q_k^{(b)}(Q_k^{(b)})^{\mathsf{T}}

  10. E~1←flatten⁡(E1(b))\widetilde{E}_1 \leftarrow\operatorname{flatten}(E_1^{(b)})

  11. E~2←flatten⁡(E2,reduced(b))\widetilde{E}_2 \leftarrow\operatorname{flatten}(E_{2,\mathrm{reduced}}^{(b)})

  12. norm⁡1←∥E~1∥2\operatorname{norm}_1 \leftarrow\lVert\widetilde{E}_1\rVert_2

  13. s←1+α⋅⟨E~1/norm⁡1,E~2/norm⁡1⟩s \leftarrow1+\alpha\cdot\langle\widetilde{E}_1/\operatorname{norm}_1,\widetilde{E}_2/\operatorname{norm}_1\rangle

  14. E~out←s⋅E~1−α⋅E~2\widetilde{E}_{\mathrm{out}} \leftarrow s\cdot\widetilde{E}_1-\alpha\cdot\widetilde{E}_2

  15. Eout(b)←reshape⁡(E~out,[L,H])E_{\mathrm{out}}^{(b)} \leftarrow\operatorname{reshape}(\widetilde{E}_{\mathrm{out}},[L,H])

  16. end for

  17. norm⁡e←∥e1∥2\operatorname{norm}_e \leftarrow\lVert e_1\rVert_2

  18. se←1+α⋅⟨e1/norm⁡e,e2/norm⁡e⟩s_e \leftarrow1+\alpha\cdot\langle e_1/\operatorname{norm}_e,e_2/\operatorname{norm}_e\rangle

  19. eout←se⋅e1−α⋅e2e_{\mathrm{out}} \leftarrow s_e\cdot e_1-\alpha\cdot e_2

  20. return Eout,eoutE_{\mathrm{out}},e_{\mathrm{out}}

Vocabulary Retrieval Dataset and COALBench-300

Construction of the Vocabulary Retrieval Dataset

To represent the isolated orthogonal drift vector vorthv_{\mathrm{orth}} using discrete textual surrogates, we constructed a vocabulary retrieval dataset of over 300,000 words and phrases designed to cover a diverse set of visually grounded and general linguistic concepts. The data was compiled from the following sources:

  • Open Images ([17]): Object class labels provided in the official release.

  • Visual Genome ([18]): Object annotations, which supply a wide range of free-form object names contributed by annotators.

  • Wikipedia ((Wikimedia Foundation, 2022)): To augment the visual vocabulary with more general linguistic coverage, we parsed 1,000 articles extracted from English Wikipedia dataset (March 1, 2022 dump) using SpaCy’s dependency parser ([12]). From each article, we extracted noun chunks, retaining those appearing at least three times across the corpus. This yielded a complementary set of multi-word expressions such as “machine learning” or “climate change.”

In addition, we used a curated English word list (i.e., the web2 dictionary via the english-words python package) to ensure broad lexical coverage ((Wiens, 2025)).

Following data collection from these sources, we apply a comprehensive preprocessing phase to clean and refine the dataset. This procedure consists of three primary stages: normalization, heuristic filtering, and spelling correction.

Normalization. All candidate strings were lowercased, Unicode-normalized, stripped of non-alphanumeric symbols, and reduced to tokens separated by single spaces. This step removed morphological noise such as capitalization, punctuation, and diacritics.

Heuristic filtering. To eliminate uninformative entries, we applied a series of heuristic rules. First, we enforced length constraints, retaining only terms that contain 1 to 3 tokens and span between 3 and 30 characters. Next, we discarded strings dominated by digits or punctuation, as well as isolated stopword phrases (e.g., “and”, “the”). Finally, to ensure that short but meaningful concepts were not mistakenly removed by these length constraints, we implemented a whitelist to preserve essential terms such as “tv” and “pc”.

Spelling correction. Because crowdsourced annotations introduce spelling inconsistencies, we employed the SymSpell algorithm [6] with a maximum edit distance of 2 to canonicalize misspellings. This reduced duplication from typographic errors.

This multi-stage process produced a broad and diverse vocabulary containing both concrete, visually grounded object names and higher-level conceptual phrases. In total, the final vocabulary contains approximately 323,000 items, which serves as the vocabulary retrieval dataset for our experiments.

Construction of COALBench-300

In order to evaluate the semantic alignment between text-to-image models and user intent, we created COALBench-300 (COmpositional ALignment Benchmark), a curated collection of 300 imaginative prompts specifically designed to test challenging semantic alignment scenarios not emphasized in existing benchmarks. The goal was to establish a standardized set of inputs characterized by richness and diversity such that alignment failures would be clearly observable. To this end, we queried three distinct large language models: OpenAI o3 (o3-2025-04-16), Anthropic Claude Opus-4 (claude-opus-4-20250514), and Google Gemini 2.5 Pro (gemini-2.5-pro), with instructions to propose creative and highly visual text-to-image prompts. Each model produced a large pool of candidate prompts emphasizing diverse styles, scenarios, and semantic complexity.

From the generated pool, we manually reviewed and curated a final set of 300 prompts. This final collection of curated prompts provides a compact but challenging benchmark that allows us to systematically probe how well text-to-image models respect compositional semantics. Unlike generic caption corpora, these prompts emphasize stress-testing of alignment through imaginative and specific requests. The resulting dataset, alongside details of our filtering criteria and validation process is presented in Appendix § 12.

Experiments

Experimental Setup

Models. To demonstrate the model-agnostic nature of ODC, we test it across Stable Diffusion XL (SDXL) [22] and FLUX.1 [schnell] [2]. We use the BLIP2-ITM-ViT-G model [19] for all first-stage operations in ODC, namely, to compute the image and text embeddings for semantic drift estimation, to embed the vocabulary items, and to perform the vocabulary retrieval based on cosine similarity.

Inference Parameters. We used the default inference parameters recommended by the model authors for all experiments. This includes an image resolution of 1024×10241024 \times1024 pixels, with SDXL running the Euler Discrete sampler (30 steps, CFG=7.5), and FLUX.1 [schnell] running the Flow Matching Euler Discrete sampler (4 steps). To ensure statistical robustness, we generated three images per prompt using fixed seeds across all methods and report the mean values for all metrics. Our method introduces a few inference-time hyperparameters. To determine their optimal values, we conducted a hyperparameter search on a randomly sampled, held-out 100-prompt subset of COALBench-300. Hyperparameters were selected once per model backbone and kept fixed across all evaluation benchmarks. The final selected configurations, along with ablation plots detailing performance across different parameter values, are provided in Appendix § 13.

Benchmarks and Datasets. We evaluate our method using three distinct sets of prompts. First, we employ COALBench-300, introduced in § 4.2. To ensure a rigorous evaluation, the main quantitative results in Table 1 are evaluated only on the 200 prompts held out from hyperparameter tuning, while other analyses in this paper use the complete set of 300 prompts. Second, we utilize PartiPrompts (Yu et al., 2022) to probe imaginative and open-ended generation. For our experiments, we filter this dataset to retain only prompts containing ten words or more, ensuring that all evaluated prompts provide sufficient semantic grounding. Lastly, we employ T2I-CompBench (Complex) [15], a challenging benchmark subset that evaluates the generation of complex scenes involving multiple objects, attributes, and relationships.

MethodCoalBench-300PartiPrompts
SDXLFLUX.1 [schnell]SDXLFLUX.1 [schnell]
CLIP ↑BLIP ↑CLIP ↑BLIP ↑CLIP ↑BLIP ↑CLIP ↑BLIP ↑
Vanilla30.32790.607829.44990.607630.83380.603031.26650.6051
CFG-Sweep (CFG=5.0)30.00220.6070--30.44680.6021--
CFG-Sweep (CFG=12.0)30.45090.6083--31.12690.6040--
Negative Prompt30.32820.607829.46160.607430.83360.603031.18440.6051
Negative Prompt BoT31.18980.609629.89120.608231.74360.605131.78110.6064
ODC (Ours)31.79760.612129.91810.608432.50430.607431.54680.6051
ODC BoT (Ours)32.29070.612730.45490.609732.99110.608232.15380.6067

Table 1. Main quantitative results on CoalBench-300 and PartiPrompts benchmarks. We report alignment metrics across SDXL and FLUX.1 [schnell]. Best results are in bold. Blue and Orange rows highlight our methods. ODC BoT selects the best image from the two available choices—see text for details.

Baselines. We compare ODC against several baselines to better understand its performance. We use the unmodified output from the base T2I models as our Vanilla reference. We also perform a CFG-Sweep, reporting results across different scales to compare against simply strengthening prompt guidance. Additionally, we evaluate a Negative Prompt baseline, which serves as a prompt-based alternative to our conditioning space correction. Specifically, we create a tailored negative prompt for each generation by joining the retrieved drift vocabulary items (SorthS_{\mathrm{orth}}), instructing the model to avoid the identified spurious concepts. Finally, we compare against two specific inference-time guidance approaches, Attend-and-Excite [3], which refines the latent code during generation by optimizing cross-attention units to “excite” neglected subject tokens (adapted for SDXL; see Appendix § 14), and ToMe [14], which merges object and attribute embeddings into composite tokens to unify their cross-attention maps via iterative optimization and entropy regularization.

Evaluation Metrics. We use established metrics for text-image alignment and computational efficiency. To measure semantic alignment between a prompt and its generated image, we rely on both CLIPScore (ViT-L/14) [9], a widely used metric for general semantic similarity, and BLIPScore (BLIP2-ITM-ViT-G) [19], which leverages the more advanced BLIP-2 vision-language model for finer-grained evaluations. Additionally, for the T2I-CompBench (Complex) benchmark, we report the “3-in-1” score alongside the individual component scores (BLIP-VQA, UniDet, and CLIPScore) to assess performance across attribute binding, spatial arrangement, and semantic alignment. Given that our method operates during inference, we measure its computational overhead in terms of both latency, defined as the average wall-clock time required to generate a single image, and GPU memory usage during generation.

Quantitative Results

To establish the efficacy of our method, we present a thorough quantitative evaluation. As shown in Table 1 and Table 2, ODC consistently improves all baselines across semantic faithfulness metrics (CLIP and BLIP) on both Stable Diffusion XL and FLUX.1 [schnell] (statistical significance validated in Appendix § 15).

MethodCoalBench-300 CLIP ↑CoalBench-300 BLIP ↑PartiPrompts CLIP ↑PartiPrompts BLIP ↑T2I-CompBench (Complex) BLIP-VQA ↑T2I-CompBench (Complex) UniDet ↑T2I-CompBench (Complex) CLIP ↑T2I-CompBench (Complex) 3-in-1 ↑
SDXL [22]30.3050.60830.8340.6030.4420.0250.2930.3294
A&E [3]29.5420.60830.150.6040.4360.0230.2900.3259
ToMe [14]30.3970.60830.6810.6030.4680.0220.2920.3417
ODC (Ours)31.7410.61232.5030.6070.4940.0200.3000.3572

Table 2. Comparison with state-of-the-art. We report CLIP and BLIP scores for CoalBench-300 and PartiPrompts benchmarks. For T2I-CompBench, we report the metrics on the complex subset, including the final 3-in-1 average score. Best results are in bold. All methods use SDXL as the baseline.

While ODC improves semantic faithfulness metrics, it can occasionally yield trade-offs in specific spatial metrics, such as UniDet (Table 2). Although BLIP2-ITM-ViT-G is used both in our pipeline and the BLIP Score evaluation, the observed improvements are independently corroborated by CLIP Score, qualitative results, and human evaluation. Additionally, our VLM ablation study (§ 5.4) demonstrates that ODC yields consistent improvements across different VLM architectures, confirming that the method does not merely overfit to the BLIP2 embedding space. Since ODC’s pipeline inherently produces both the initial baseline image (required for drift calculation) and the corrected variant, we can adopt a best-of-two (ODC BoT) strategy. Specifically, ODC BoT computes the CLIP Score for both the baseline and the ODC-corrected image, and returns the image with the higher score. This guarantees that the final output is never worse than the baseline in terms of CLIP alignment. To ensure a fair comparison, we also evaluate a Negative Prompt BoT baseline, which applies the same selection strategy to the Negative Prompt method (since it also requires an initial generation to retrieve drift vocabulary). As shown in Table 1, the Negative Prompt BoT baseline underperforms our method as well.

Furthermore, ODC preserves visual fidelity. We verify this using both an automated metric and human evaluation. For the automated evaluation, we measure the distributional shift between the original and corrected images using Kernel Inception Distance (KID) (Bińkowski et al. [1]). Near-zero KID scores (ranging from −0.00152 to 0.00093) across SDXL and FLUX.1 confirm that the corrected distributions remain statistically indistinguishable from the baselines. Crucially, a human evaluation of visual fidelity found no statistically significant difference between baseline and ODC-corrected images (Binomial test p=0.93p = 0.93, Cohen’s h=0.13h = 0.13), with a high tie rate of 58.0%. This demonstrates that ODC surgically removes drift without shifting the generation out of the data manifold. See Appendix § 16 for details.

As shown in Table 3, ODC introduces some latency overhead while leaving memory consumption unchanged. Compared to the vanilla baseline, latency increases from 3.637 s/img to 7.56 s/img, reflecting the additional correction pass. Importantly, GPU memory usage remains constant at 23.015 GB, making the approach just as accessible in practice as the baseline methods. Overall, these results demonstrate that the gains in semantic alignment come with predictable and manageable efficiency costs.

MethodLatency (s/img)Memory (GB)
Vanilla3.63723.015
ODC7.56023.015

Table 3. Efficiency analysis on SDXL.

Ultimately, we validated our quantitative results via a human evaluation consisting of 600 blind pairwise comparisons. These were collected from 12 domain experts evaluating 50 randomly sampled prompts from CoalBench-300. As shown in Table 4, ODC is preferred in 61.8% of decisive votes (95% CI: [57.0%, 66.3%]) versus the baseline’s 38.2%. This represents a statistically significant improvement, as the CI strictly excludes 50%. Furthermore, a 30.7% tie rate indicates that ODC preserves fidelity in already well-aligned images rather than applying unnecessary alterations.

MetricValue95% CI
ODC Win Rate61.8%[57.0%, 66.3%]
Baseline Win Rate38.2%[33.7%, 43.0%]
Tie Rate30.7%—

Table 4. Human evaluation results. Win rates are computed over decisive votes (ties excluded).

Qualitative Results

Visual inspection of the generated images provides the most intuitive evidence of ODC’s effectiveness. In Figure 2, we present a side-by-side comparison of images generated by vanilla SDXL and FLUX, alongside our ODC-corrected method for a variety of challenging prompts. We observe that ODC improves compositional accuracy and attribute binding, and leads to reduced semantic drift. The baseline often struggles with correct modifier-object relationships and maintaining fine-grained attributes, while ODC generates images that align more accurately with the user description. Furthermore, for prompts that are vulnerable to spurious additions (e.g., “Victorian astronaut playing violin inside ornate greenhouse orbiting Saturn’s rings”), the baseline model introduces unintended elements, such as additional violinists, that reduce alignment with the user prompt. In contrast, ODC constrains generations to the requested content and avoids unwanted additions, yielding outputs that more faithfully reflect the intended semantics. These qualitative examples demonstrate that by refining the initial text embedding, ODC provides a more reliable conditioning for the image generation process. Additional qualitative results are provided in Appendix § 17. The list of vocabulary items (SorthS_{\mathrm{orth}}) retrieved by ODC for images in Figure 2 are included in a detailed table in Appendix G.

Figure 2. Qualitative comparison of images generated with and without Orthogonal Drift Correction (ODC). For each prompt, we show the output from the baseline model (left) and our ODC-corrected version (right).

Ablation Studies

Dependence on Vision-Language Model (EVLM\mathcal{E}_{\mathrm{VLM}}). To evaluate ODC’s dependency on the VLM, we conducted an ablation study on CoalBench-300 using three additional models of varying capacity: CLIP ViT-L/14, SigLIP2 SO400M, and EVA-Giant Plus. We also include our primary model, BLIP2-ITM-ViT-G, to serve as a reference point. For each VLM tested, all ODC first-stage operations are executed independently, with the respective model computing the semantic drift vector, embedding the vocabulary items, and performing the vocabulary retrieval to ensure the process operates entirely within its own embedding space. To ensure a rigorous comparison across these different embedding spaces, we calibrated ODC’s removal strength (α\alpha) on a held-out set of 50 prompts and evaluated performance on the remainder. As shown in Table 5, ODC generalizes across VLMs, with all models successfully improving baseline alignment. Among the evaluated VLMs, BLIP2-ITM-ViT-G achieves the highest overall alignment scores. CLIP ViT-L/14 and EVA-Giant Plus outperform SigLIP2, with CLIP slightly outperforming the larger EVA on SDXL. We attribute this to architectural parity, as SDXL and FLUX.1 natively condition on the CLIP ViT-L/14 text encoder, making its calculated drift vectors inherently better aligned with the generator’s manifold. Ultimately, the strong performance of high-capacity models like EVA-Giant Plus and BLIP2-ITM-ViT-G confirms that such VLMs can effectively approximate this drift, ensuring ODC remains robust regardless of the chosen VLM.

MethodCLIP ViT-L/14 SDXL CLIP ↑CLIP ViT-L/14 SDXL BLIP ↑CLIP ViT-L/14 FLUX.1 CLIP ↑CLIP ViT-L/14 FLUX.1 BLIP ↑SigLIP2 SO400M SDXL CLIP ↑SigLIP2 SO400M SDXL BLIP ↑SigLIP2 SO400M FLUX.1 CLIP ↑SigLIP2 SO400M FLUX.1 BLIP ↑EVA-Giant Plus SDXL CLIP ↑EVA-Giant Plus SDXL BLIP ↑EVA-Giant Plus FLUX.1 CLIP ↑EVA-Giant Plus FLUX.1 BLIP ↑BLIP2-ITM-ViT-G SDXL CLIP ↑BLIP2-ITM-ViT-G SDXL BLIP ↑BLIP2-ITM-ViT-G FLUX.1 CLIP ↑BLIP2-ITM-ViT-G FLUX.1 BLIP ↑
Vanilla30.2580.60729.3290.60730.2580.60729.3290.60730.2580.60729.3290.60730.2580.60729.3290.607
ODC31.6620.61129.6160.60831.1730.61029.5040.60831.6420.61129.6290.60831.7620.61229.7780.608
ODC BoT32.1060.61230.1050.60931.5590.61130.1410.60932.1260.61230.3540.61032.2060.61230.2780.610

Table 5. Ablation Study on Vision-Language Model Dependency (EVLM\mathcal{E}_{\mathrm{VLM}}).

The Importance of Orthogonal Correction. Our central claim is that isolating and removing the orthogonal component of the semantic drift is key. To validate this, we compare ODC against two alternative correction strategies (Table 6). First, we test a “Simple Concept Removal” baseline, which directly subtracts the embeddings of the retrieved drift keywords from the prompt embeddings. While this improves upon the vanilla baseline, it substantially underperforms compared to the other approaches. Second, we test “Full Vector Correction,” which removes the entire semantic drift vector (vdrift\mathbf{v}_{\mathrm{drift}}). Despite outperforming simple concept removal, it still underperforms our proposed ODC (Figure 3), as removing the full vector inadvertently strips away useful on-axis semantic information. In contrast, ODC surgically removes only the irrelevant, orthogonal semantic contents, achieving the highest alignment.

Figure 3. Qualitative ablation comparing baseline (SDXL), Full Vector Correction, and ODC on three prompts. Results show ODC corrects semantic discrepancies in the left and middle columns.

MethodCLIP ↑
Vanilla30.305
Simple Concept Removal30.970
Full Vector Correction31.489
Orthogonal Drift Correction31.741

Table 6. Quantitative ablation results for SDXL on CoalBench-300.

Discussion

While ODC operates in a training-free, plug-and-play manner, applying it to a new backbone requires a one-time calibration of its hyperparameters. This tuning burden is minimal and can be efficiently performed on a small held-out validation set or even calibrated intuitively based on a limited number of qualitative examples. Once these parameters are set for a given backbone, they remain fixed during evaluation.

The necessity of this per-backbone calibration stems from the fact that while ODC is model-agnostic, its magnitude of improvement is dependent on the specific model architecture’s response to embedding space modifications. For instance, the alignment gains on FLUX.1 are smaller than on SDXL. We attribute this to FLUX.1’s sensitivity to shifts in the embedding space. As demonstrated in our hyperparameter ablation (Appendix § 13), FLUX.1 is prone to catastrophic over-correction at higher removal strengths (e.g., α=0.8\alpha= 0.8), necessitating a more conservative correction strength compared to SDXL. We hypothesize that this sensitivity may stem from how FLUX.1 processes and distributes token embeddings compared to SDXL, but leave a formal investigation of this to future work.

Limitations

Despite its strong performance, ODC has some inherent limitations. First, its effectiveness is bound by the perceptual capabilities of the vision-language model (VLM) used in the first stage. If the VLM cannot see a specific semantic error, ODC may not correct it. For instance, because VLMs struggle with precise counting, ODC might fail to correct numerical discrepancies, such as generating exactly “A desert landscape with three suns setting on the horizon” (Figure 4(a)). Similarly, VLMs can miss nuanced artistic styles; if a baseline misses the intended style of a prompt like “An Art Deco robot butler serving cocktails at a party,” the VLM may not recognize this drift, causing ODC to push the image towards realism rather than the desired style (Figure 4(b)). Furthermore, VLMs can misinterpret polysemous words, misguiding ODC. For example, with the prompt “Lightning striking sand creating glass,” the VLM might anchor onto the household object rather than the material, causing ODC to insert a literal drinking glass instead of a glassy formation (Figure 4(c)). These failures represent edge cases that are dependent on the interplay between the specific prompt and the generator, stemming from a combination of drift misestimation by the VLM and the generator’s own susceptibility to hallucination. Because these factors are intertwined, simply swapping the base VLM may not reliably resolve all failures. However, future work could explore using task-specific evaluators (e.g., object-counting) to tune the hyperparameters for specific prompt types, or employing VLMs with stronger compositional reasoning to improve drift estimation. As VLM embeddings continue to improve, we expect ODC’s robustness in these edge cases to advance as well.

Representative failure cases showing ODC and Baseline

Figure 4. Representative failure cases: ODC (bottom) and Baseline (top).

Second, the two-stage correction process inherently increases the inference latency compared to a standard single pass, as shown in Table 3. While the overhead from the embedding calculations themselves is negligible, the need for an initial generation pass represents a direct trade-off between computational cost and semantic drift correction. Nevertheless, considering that T2I generation is generally an offline task, this limitation is of minor nature. Moreover, our method can potentially reduce the time associated with manual trial and error process involved in prompt engineering considerably by providing better alignment of outputs to the prompts.

Conclusion

In this paper, we addressed the critical problem of semantic drift in text-to-image models. We introduced Orthogonal Drift Correction (ODC), a training-free, inference-time method that guides the image generation through a two-stage process. It estimates the components of text embeddings causing the drift and mitigates them. Our extensive experiments demonstrate that ODC significantly enhances prompt-image alignment across multiple models and benchmarks. Without requiring any retraining or architectural changes, ODC provides a practical and generalizable solution for improving the alignment and reliability of existing pre-trained text-to-image models, making them more powerful creative tools.

Reproducibility Statement

To ensure reproducibility, we provide detailed algorithmic descriptions of our method in Section 3, along with hyperparameter configurations and experimental settings in Section 5. Our proposed prompt dataset, CoalBench-300, is provided in full in Appendix A, and all other datasets used in our experiments are publicly available. All experiments are conducted using publicly available models and libraries. Our code is publicly available at https://github.com/kasrakamali/ODC.

Broader Impact Statement

Improving prompt-image alignment improves reliability and reduces user frustration. Like any alignment technique, ODC makes models more faithful to their inputs, which could include harmful or inappropriate prompts if applied to base models with weak safety filters. Importantly, ODC does not expand the generative capabilities of the base model, but merely reduces semantic drift. To prevent misuse, ODC should be deployed in conjunction with standard content moderation and safety filtering mechanisms.

Acknowledgments

Naveed Akhtar is a recipient of the Australian Research Council Discovery Early Career Researcher Award (project #DE230101058) funded by the Australian Government. This research was supported by The University of Melbourne’s Research Computing Services and the Petascale Campus Initiative, and by Melbourne Research Scholarship from The University of Melbourne.

COALBench-300

Table A1 presents the COALBench-300 dataset used in our experiments, consisting of 300 prompts generated as described in § 4.2. To curate the final set from the initial generated prompts, we applied a filtering and validation process. First, we applied the following selection criteria: prompts were retained only if they (1) were highly imaginative and visually complex, requiring precise semantic adherence to execute correctly, and (2) contained clear descriptive constraints, such as specific visual properties, artistic styles, atmospheric conditions, or surreal combinations. Finally, to ensure independent validation, the curated list was reviewed and approved by a senior researcher, confirming that the selected prompts effectively serve as a stress test for compositional alignment. Furthermore, by providing the full list of 300 prompts in Table A1, we enable readers to directly assess and validate the benchmark.

#Prompt#Prompt
1Neon cityscape reflected in rain puddles at midnight2Ancient library filled with floating glowing books
3Steampunk octopus playing violin underwater4Cherry blossoms falling on samurai armor
#Prompt#Prompt
5Cyberpunk cat cafe with holographic menu displays6Dragon made entirely of autumn leaves
7Astronaut planting flowers on Mars surface8Victorian greenhouse overflowing with bioluminescent plants
9Robot chef preparing sushi in minimalist kitchen10Northern lights dancing above crystal ice castle
11Abandoned amusement park reclaimed by nature12Phoenix rising from coffee cup steam
13Underwater city inside giant glass bubble14Mechanical butterfly landing on child’s outstretched hand
15Storm clouds shaped like galloping horses16Art deco train station at golden hour
17Jellyfish constellation floating through space18Medieval knight riding mechanical dragon
19Sunflowers growing from piano keys20Time traveler’s workshop filled with clocks
21Magical forest where trees have doorways22Origami birds transforming into real ones
23Vintage typewriter typing rainbow colored words24Giant teacup island in cloudy sky
25Bioluminescent mushroom cave with crystal formations26Retro diner floating in outer space
27Library ladder reaching into starry sky28Glass sculpture garden during thunderstorm
29Mechanical whale swimming through clouds30Paint brushes growing like flowers in pot
31Lighthouse made of stacked books32Geometric mountains with gradient sunset colors
33Tiny fairy village inside terrarium34Steam locomotive made of flowers
35Crystal skull containing miniature galaxy36Hot air balloons shaped like planets
37Underwater ballet performed by sea-horses38Gothic cathedral made of ice
39Neon sign reflected in cyberpunk alley40Tree house connected by rope bridges
41Desert oasis with mirror-like water42Mechanical heart with gears and flowers
43Floating islands connected by waterfalls44Vintage camera capturing memories as butterflies
45Snow globe containing miniature city46Giant mushroom houses in enchanted forest
#Prompt#Prompt
47Clock tower melting like Salvador Dali painting48Paper airplane leaving trail of stars
49Underwater ruins with glowing hieroglyphs50Mechanical bee pollinating metal flowers
51Aurora borealis inside crystal cave52Sailing ship made of autumn leaves
53Robot gardener tending glass flowers54Mountain peak piercing through clouds
55Neon Tokyo street in heavy rain56Butterfly wings made of stained glass
57Ancient tree with doors to other worlds58Mechanical dragon breathing rainbow fire
59Floating market on lily pad boats60Crystal palace reflecting sunset colors
61Origami crane transforming into phoenix62Underwater volcano with lava lamp effect
63Clockwork owl perched on moon64Garden maze viewed from above
65Holographic whale swimming through city66Ice sculpture melting into butterflies
67Vintage bicycle with flower basket68Portal opening in library wall
69Mechanical spider weaving light web70Treehouse city in giant redwood forest
71Glass elevator ascending through clouds72Firefly jar illuminating dark forest
73Geometric deer made of neon lines74Floating tea party in sky
75Crystal bridge over mirror lake76Mechanical hummingbird feeding on gear flowers
77Northern lights reflected in frozen lake78Paper boat sailing on book pages
79Bonsai tree with tiny glowing lanterns80Underwater castle made of coral
81Steampunk hot air balloon city82Galaxy swirling in coffee cup
83Ice phoenix rising from frozen waterfall84Mechanical butterfly collection in display case
85Floating geometric islands at sunset86Glass greenhouse in snowy landscape
87Lightning striking sand creating glass88Origami dragon breathing paper fire
89Crystal cave with rainbow reflections90Vintage train crossing bridge at dawn
91Mechanical whale breaching ocean surface92Fairy ring mushrooms glowing at twilight
93Neon arcade in rain-soaked alley94Tree growing through abandoned car
95Holographic fish swimming through air96Ice palace under aurora borealis
97Steampunk telescope viewing distant galaxies98Garden path made of piano keys
99Crystal butterfly landing on metal flower100Lighthouse beam cutting through fog
#Prompt#Prompt
101A bioluminescent jellyfish floating through a neon-lit cyberpunk city102A lone knight overlooking a valley of giant, glowing mushrooms
103A crystal skull filled with a swirling, miniature galaxy104A serene, minimalist Japanese garden in the pouring rain
105An ancient Roman library on a terraformed, dusty Mars106A tiny fairy house built inside a broken teapot
107A grand piano with keys melting like liquid Dali-style108A cozy, cluttered bookstore on a foggy autumn evening
109A Viking longship sailing on a sea of clouds110A stained glass window depicting a cosmic horror entity
111A sleek, organic starship gliding silently past a vibrant nebula112A grizzly bear made of constellations in the night sky
113A forgotten, overgrown greenhouse filled with strange, glowing plants114A city skyline at sunset, rendered in 8-bit pixel art
115A steampunk inventor’s workshop, cluttered with brass and clockwork116A macro photograph of a single dewdrop on a spiderweb
117A majestic castle made of obsidian and jagged lightning118The concept of "silence" visualized as a physical object
119An Art Deco robot butler serving cocktails at a party120A floating market on a serene, otherworldly alien canal
121A portrait of a queen, made entirely from pressed flowers122A paper-craft, origami world with a flowing paper river
123A cat made of glass, filled with swirling, colored smoke124A haunted forest where the trees have screaming human faces
125Impossible Escher-like architecture carved from a single, giant tree126A gourmet hamburger made entirely of sparkling, edible gemstones
127A desert landscape with three suns setting on the horizon128An ancient, moss-covered golem sleeping in a sunlit forest
129A city street scene painted in the style of Van Gogh130A lone, red telephone box in the middle of nowhere
131A mechanical heart with glowing tubes and intricate gears132A tranquil koi pond that reflects a starry night sky
133An astronaut discovering an ancient alien ruin on the moon134A single, perfect rose encased in a block of ice
135An island that is the back of a giant, swimming turtle136A shadowy figure standing at the end of a long pier
137A black and white photograph of a 1940s detective’s office138A city built inside a colossal, hollowed-out crystal cavern
139A samurai warrior with glowing, cybernetic neon tattoos140A massive, ancient library where the books float between shelves
141A cup of coffee with a swirling galaxy in it142A child releasing a kite that is actually a dragon
#Prompt#Prompt
143A detailed anatomical drawing of a mythical griffin144A vaporwave sculpture garden with pink and blue neon lights
145A doorway to another dimension opening in a brick wall146A city street dissolving into musical notes in the air
147A lone lighthouse battered by a furious, cosmic storm148A portrait of "joy" as an explosion of vibrant colors
149A medieval feast with ghosts sitting amongst the living guests150An underwater city inhabited by elegant, humanoid sea creatures
151A tree whose leaves are made of delicate, colored glass152A futuristic high-speed train blurring through a bamboo forest
153A sad clown looking at his reflection in a puddle154A cinematic shot of a dinosaur roaring in a modern city
155A map of a fantasy world burned onto old leather156An ornate, Venetian mask with intricate filigree and feathers
157A car rusting in a field of blooming sunflowers158A mountain range shaped like a sleeping stone dragon
159A wizard’s spellbook lying open, glowing with powerful magic160A skeleton in a spacesuit, sitting on a desolate asteroid
161A beautiful landscape painting done on a crumbling concrete wall162A ceramic mosaic depicting the birth of a star
163A clock tower where the numbers are flying away164An elven city built seamlessly into giant, living trees
165A low-poly render of a wolf howling at the moon166A secret garden hidden behind a waterfall
167A single, burning feather falling from a dark sky168A long exposure photograph of high-way lights at night
169A palace made entirely of ice, glistening in the sun170A chessboard with living pieces preparing for battle
171A surreal landscape where the ground is a reflective mirror172A powerful phoenix rising from ashes, cinematic fantasy art
173A ukiyo-e woodblock print of a busy Tokyo intersection174An old, forgotten god statue entangled in jungle vines
175A gentle robot tending to a garden of metal flowers176An underwater submarine discovering the lost city of Atlantis
177The solar system re-imagined as a delicate, hanging mobile178A tiny mouse wearing intricate, handcrafted medieval armor
179A world where the oceans are in the sky180A close-up of a dragon’s eye, reflecting a battle
181A hyper-realistic painting of a spilled glass of milk182A towering, complex sandcastle about to be hit by a wave
183A lush, tropical beach inside a giant, open clam shell184A synthwave-style sunset over a retro-futuristic highway
#Prompt#Prompt
185A collection of magical potions, glowing on a dusty shelf186A double exposure portrait of a woman and a forest
187A pirate ship sailing through a massive electrical storm188A whimsical town where all the buildings are books
189A soldier’s helmet from the future, cracked and abandoned190A tree of life connecting the earth and the cosmos
191A blueprint-style technical drawing of a fantasy airship192A labyrinth made of tall, perfectly trimmed hedges
193An abstract sculpture representing the sound of laughter194A street market on a space station, full of aliens
195A beautiful, haunting portrait of a porcelain doll196The world balanced on the tip of a giant’s finger
197A polar bear sleeping under the vibrant aurora borealis198A vintage 1950s car flying through the clouds
199A stone staircase spiraling down into the deep, dark ocean200A single, glowing key lying on an empty city street
201Neon samurai meditating under cherry blossoms, cyberpunk city distant horizon202Golden dragon soaring above tempestuous sea during crimson twilight storm
203Victorian astronaut playing violin inside ornate greenhouse orbiting Saturn’s rings204Gigantic marble statues emerging from desert dunes beneath aurora sky
205Steam-powered whale floating over steampunk city releasing glowing lanterns peacefully206Surreal library labyrinth with endless staircases and levitating luminous books
207Winter fox spirits dancing around ancient shrine under full moon208Retro-futuristic diner on Mars serving milkshakes to robot cowboys today
209Bioluminescent jungle river reflecting twin moons, explorers paddling crystal canoes210Minimalist ink painting depicting thunderstorm over solitary mountain tea house
211Ancient turtle island carrying cherry forest drifting across tranquil ocean212Post-apocalyptic fashion runway among ruins, models wearing repurposed metallic couture
213Glass butterflies emerging from cracked sculpture in abandoned classical gallery214Cosmic carnival Ferris wheel spinning galaxies instead of passenger gondolas
215Medieval alchemist laboratory illuminated by swirling emerald and violet potions216Peaceful Zen garden floating among clouds, koi fish swimming air
217Art deco submarine cruising crystal Arctic waters beneath shimmering aurora218Gothic cathedral built from bones, candles flickering with blue flames
#Prompt#Prompt
219Whimsical cat orchestra performing symphony inside giant teacup on meadow220Autumn maple leaves transforming into origami cranes mid gentle breeze
221Futuristic Venice with glass canals and levitating gondolas at sunset222Alien botanist tending glowing succulents within transparent asteroid greenhouse sphere
223Samurai mech duel atop skyscraper during torrential rain and lightning224Childhood memories visualized as floating polaroid cubes above grassy field
225Mushroom village illuminated by fireflies, tiny gnomes celebrating midsummer feast226Surreal chessboard landscape where pieces are skyscrapers casting elongated shadows
227Golden hour lighthouse standing on clouds guiding paper boats home228Cybernetic phoenix reborn from neon flames above dystopian alleyway rooftops
229Mythical underwater library with coral shelves and manta ray librarians230Celtic wolf warrior howling beside rune-carved stone under starry sky
231Industrial robots painting classical renaissance portrait inside abandoned factory hall232Pastel vaporwave beach scene with dolphins and pixelated palm trees
233Ice cream planet orbiting candy stars within childlike cosmic dreamscape234Ethereal ballerina composed of smoke gracefully twirling through candlelit ballroom
235Ancient astronauts carved into mountain, projections igniting during equinox ceremony236Monochrome noir city where rain turns neon once striking pavement
237Giant panda knight wearing bamboo armor charging across sunflower plains238Sailing ship overhead sky ocean, clouds shaped like wandering whales
239Magnificent art nouveau dragonfly airship docking at crystal palace port240Crimson haired witch brewing constellation soup in antique cast-iron cauldron
241Time travelers picnic within looping paradox meadow filled with suns242Dreamy pastel clouds dripping like paint over sleepy coastal village
243Crystal caverns resonating music, stalactites functioning as grand pipe organ244Pharaoh cyborg sitting upon holographic throne inside starship pyramid command
245Origami dragons battling above waterfall, paper scraps becoming swirling mist246Intergalactic farmers harvesting stardust using laser scythes and hover wagons
247Snowy owl postman delivering letters between lonely mountain observatories nightly248Renaissance painting fused with glitch art, saints pixelated yet divine
#Prompt#Prompt
249Lonely astronaut gardening roses inside cracked helmet on barren exoplanet250Midsummer carnival mirrored on lake surface, reflections showing alternate dimension
251Celestial koi swimming through starry void forming luminous cosmic patterns252Gigabyte forest where data streams cascade like waterfalls between trees
253Clockwork hummingbirds pollinating gears within enormous steampunk garden mechanism today254Futuristic samurai riding holographic horse across neon rice paddies night
255Survivalists camping on colossal lily pads floating down moonlit river256Hyperrealistic microphotography style city built upon a dandelion seed closeup
257Norwegian fjord village illuminated solely by bioluminescent algae and stars258Astral projection commuters riding light beams between futuristic office towers
259Fairytale bakery selling clouds, customers bottling rainbows for frosting today260Desert caravan of holographic camels crossing dunes under binary suns
261Spaceship graveyard turned skatepark, teenagers grinding along rusted hulls today262Fantasy opera house shaped like seashell, audience wearing pearl masks
263Interactive graffiti walls repaint themselves according to passerby’s emotional aura264Robot monk studying lotus algorithm amidst tranquil chrome bamboo grove
265Abandoned amusement park reclaimed by jungle, rollercoaster entwined with vines266Crystalline wolf pack running across frozen lake under kaleidoscope aurora
267Pocket watch melting into river, time flowing downstream with salmon268Digital renaissance fair where knights duel using augmented reality dragons
269Samurai standing in field of wind turbines during golden sunrise270Galactic mail carrier surfing solar winds to deliver interstellar postcards
271Carousel horses awakening midnight, galloping across city streets leaving sparks272Northern lights shaped like dragons over mirrorlike fjord with rowboat
273Cyberpunk mermaid lounging on neon coral reef playing electric harp274Snow globe metropolis shaking, skyscraper flakes swirling around central park
275Mystical doorway inside tree trunk leading to floating candlelit staircase276Marble chess pieces crying tears, puddles reflecting parallel universe conflicts
277Steampunk octopus captain steering dirigible through thunderous cloud ocean today278Silk road marketplace set on asteroid, merchants trading gravity spices
#Prompt#Prompt
279Lone monk meditating atop newspaper stack skyscraper in bustling metropolis280Candy-coated castle melting under sunrise, rivers of caramel surrounding battlements
281Moonlit bamboo forest where shadows of past lives perform theatre282Quantum computer envisioned as cathedral, qubits glowing like stained glass
283Vibrant coral reef mirrored above water, sky filled with fish284Ghostly steam train crossing galaxy, passengers silhouettes made from stardust
285Medieval tavern hosting extraterrestrial travelers swapping holographic treasure maps tonight286Arcade cabinet forest, joysticks sprouting like mushrooms among pixel leaves
287Viking ship sailing across clouds towards colossal floating mead hall288Haunted typewriter printing poems by itself inside candlelit attic loft
289Interdimensional bus stop where travelers trade memories for tickets home290Mid-century modern house perched atop giant redwood, overlooking fog valley
291Lavender fields beneath glass dome, drones pollinating under artificial sunlight292Ancient rune compass projecting holographic maps above weathered pirate desk
293Swan boat gliding through clouds, passengers sipping tea with angels294Neo-Tokyo skyline reflected in raindrop on cybernetic geisha’s finger nail
295Prismatic desert canyon walls refracting sunlight into rainbow patterned sand296Clocktower orchard where apples tick loudly with winding golden gears
297Giraffe skyscrapers eating cloud leaves in whimsical metropolitan savanna today298Secluded monastery balanced atop waterfall edge within verdant hidden valley
299Elegant cyborg ballerinas rehearsing in abandoned warehouse filled with mirrors300Thunderstorm brewing inside snow globe resting on antique wooden desk

Table A1. Full list of prompts in COALBench-300.

Hyperparameter Ablation

To justify the empirical selections in our pipeline, we conduct an ablation study on the ODC hyperparameters using the held-out 100-prompt subset of CoalBench-300 described in § 5. The results are visualized in Figure A1 and Figure A2.

Figure A1. Figure A1: Hyperparameter ablation study on a 100-prompt validation subset of CoalBench-300. Top row: SDXL. Bottom row: FLUX.1. The dashed line represents the unmodified baseline. Red markers indicate the hyperparameters chosen for our pipeline.

Figure A2. (a) Distribution of the rank kk where the relative diagonal magnitude ∣rii∣/∣r11∣|r_{ii}|/|r_{11}| drops below ζ\zeta, derived from SDXL drift embeddings across different thresholds. The dashed lines indicate the chosen bounds [3, 15]. (b) Mean CLIP Score achieved by ODC as a function of ζ\zeta, with the red marker indicating the chosen threshold (ζ=1.0%\zeta= 1.0\%).

Number of Keywords (kk). We evaluate the number of top retrieved vocabulary terms used to construct the orthogonal drift vector vorth\mathbf{v}_{\mathrm{orth}}. As shown in Figure A1(a, Figure A1d), performance saturates rapidly from top-1 to top-3. For FLUX.1, k=3k = 3 yields the clear peak performance. For SDXL, k=3k = 3 and k=5k = 5 perform nearly identically. However, we select k=5k = 5 for SDXL as it provides a better balance of semantic coverage. We therefore adopt k=5k = 5 for SDXL and k=3k = 3 for FLUX.1 as our final parameters.

Concept Removal Weight (α\alpha). This parameter scales the magnitude of the correction. Figure A1(b, Figure A1e) demonstrates that optimal removal strength is model-dependent. SDXL requires a stronger correction (α=1.0\alpha= 1.0), whereas FLUX.1 is highly sensitive, as an α\alpha of 0.8 causes catastrophic over-correction and collapse (CLIP 24.957). We select α=0.3\alpha= 0.3 for FLUX.1, which provides a safe margin from the over-correction cliff while achieving strong alignment.

Keywords Weights Temperature (τ\tau). The temperature controls the sharpness of the softmax applied to the similarity scores. We select τ=0.50\tau= 0.50 for SDXL and τ=0.05\tau= 0.05 for FLUX.1. As shown in Figure A1(c, Figure A1f), ODC is robust to this parameter across an order of magnitude (τ∈[0.05,1.0]\tau\in[0.05, 1.0]), confirming that the method does not rely on sharp, one-hot weighting.

Adaptive Rank-Reduction Bounds and Energy Threshold (kmin⁡,kmax⁡,ζk_{\min}, k_{\max}, \zeta). The adaptive rank-reduction in R\mathcal{R} (Algorithm 2) relies on bounds kmin⁡=3k_{\min} = 3 and kmax⁡=15k_{\max} = 15 to balance concept completeness against noise inclusion. To justify these bounds and our energy threshold ζ=0.01\zeta= 0.01, we analyze the drift embeddings Eorth\mathbf{E}_{\mathrm{orth}} across our validation set. As shown in Figure A2(a), we computed the QR decomposition and recorded the rank kk where the relative diagonal magnitude ∣rii∣/∣r11∣|r_{ii}|/|r_{11}| dropped below ζ\zeta, prior to applying any clamping. Evaluated across multiple thresholds (ζ∈{0.1%,0.5%,1.0%,2.0%}\zeta\in\{0.1\%, 0.5\%, 1.0\%, 2.0\%\}), the rank consistently falls within the [3, 15] range. Ranks below 3 are too sparse to represent a complete concept, while ranks above 15 encode irrelevant noise.

Furthermore, Figure A2(b) demonstrates that the threshold ζ=1.0%\zeta= 1.0\% provides the optimal empirical performance, yielding the highest CLIP Score (31.613). Tighter thresholds (e.g., 0.1%) or looser thresholds (e.g., 2.0%) incorporate either too much noise or too little semantic information, respectively. Therefore, ζ=1.0%\zeta= 1.0\% combined with the safety rails [3, 15] ensures our concept removal operator targets only the unwanted semantic drift.

Adaptation of Attend-and-Excite for SDXL

To employ Attend-and-Excite (A&E) as a baseline for the Stable Diffusion XL (SDXL) architecture, we adapted the official implementation to account for the model’s increased latent resolution and dual text-encoder structure. While the core Generative Semantic Nursing (GSN) logic remains consistent with Chefer et al. [3], our implementation introduces the following specific changes:

  1. Attention Map Extraction: The original method operates on 16×1616 \times16 attention maps. In SDXL, we found this resolution (corresponding to a 64-pixel stride) too coarse. We instead extract maps from the 32×3232 \times32 cross-attention layers of the U-Net. We aggregate attention probabilities across heads and derive token indices using the tokenizer associated with the second text encoder (OpenCLIP ViT-bigG/14).

  1. Padding and Smoothing: To prevent the optimization from exploiting edge artifacts, we apply reflection padding to the attention maps prior to Gaussian smoothing (kernel size 3×33 \times3, σ=0.5\sigma= 0.5). This ensures that subjects generated near the image boundaries are not penalized by zero-padding in the loss calculation. Additionally, we scale the raw attention scores by a factor of 100 before the Softmax operation to sharpen the attention distribution.

  1. Optimization Schedule: We perform the latent optimization during the first 25 steps of a 30-step generation schedule. We utilize an iterative refinement strategy at timesteps t∈{0,10,20}t \in\{0, 10, 20\} with acceptance thresholds τ∈{0.1,0.5,0.8}\tau\in\{0.1, 0.5, 0.8\}. The latent code ztz_t is updated with a gradient step size αt\alpha_t that follows a linear decay schedule:

αt=S⋅(1−0.5⋅tTtotal),(14)\alpha_t = S \cdot\left(1 - 0.5 \cdot\frac{t}{T_{total}}\right), \tag*{(14)}

where the initial scale factor SS is set to 20.0. Gradients are normalized if their norm exceeds 1×10−51 \times10^{-5} to maintain stability within the SDXL latent space.

  1. Subject Selection: We automate the selection of subject tokens using a Part-of-Speech (POS) tagger to identify nouns and proper nouns within the input prompts.

Statistical Significance Analysis

To assess the robustness of our reported improvements, we compute 95% confidence intervals (CIs) and perform paired t-tests. To account for repeated measurements across seeds, we first average the scores across the 3 seeds for each prompt and then conduct our significance tests over the same CoalBench-300 prompts used for our main results in Table 1 (N=200N=200). The complete results are detailed in Table A2.

ModelMetricBaseline (Mean ± Std)ODC (Mean ± Std)ODC BoT (Mean ± Std)Δ\Delta (95% CI)pp-value
SDXLCLIP ↑30.328 ± 3.22631.798 ± 3.21332.291 ± 3.0431.470 [1.165, 1.774]< 10−1710^{-17}
BLIP ↑0.6078 ± 0.00740.6121 ± 0.00670.6127 ± 0.00640.0042 [0.0036, 0.0048]< 10−2810^{-28}
FLUX.1CLIP ↑29.450 ± 3.38629.918 ± 3.22430.455 ± 3.2270.468 [0.239, 0.698]< 10−410^{-4}
BLIP ↑0.6076 ± 0.00720.6084 ± 0.00710.6097 ± 0.00690.0008 [0.0004, 0.0012]< 10−310^{-3}

Table A2. Statistical significance analysis across the 200-prompt CoalBench-300 evaluation subset (N=200N=200). We report Mean ± Std and 95% CIs for the baseline, ODC, and ODC BoT, along with the paired t-test results for the ODC improvement.

Our analysis confirms reliable improvements in alignment scores, with robust gains observed for both CLIP Score (SDXL: p<10−17p < 10^{-17}, FLUX.1: p<10−4p < 10^{-4}) and BLIP Score (SDXL: p<10−28p < 10^{-28}, FLUX.1: p<10−3p < 10^{-3}). Crucially, across all configurations, the 95% confidence intervals for the performance delta Δ\Delta (ODC−Baseline) lie strictly above zero, demonstrating that ODC consistently improves alignment without compromising baseline fidelity. This statistical robustness is further corroborated by our human evaluation (Table 4), where domain experts consistently favored ODC-corrected images.

Quantitative Assessment of Visual Fidelity

To verify that ODC preserves visual fidelity, we conduct a quantitative evaluation using the Kernel Inception Distance (KID) ([1]) and a human evaluation. First, we use KID to measure the distributional similarity between the images generated by ODC and the unmodified baseline. As an unbiased estimator of the squared Maximum Mean Discrepancy (MMD), KID can yield slightly negative values due to sampling variance; in practice, a score near zero indicates that the two distributions are statistically indistinguishable. As summarized in Table A3, the near-zero or marginally negative scores observed for both SDXL and FLUX.1 confirm that ODC does not degrade visual fidelity.

ModelCoalBench-300PartiPrompts
SDXL0.00093 ± 0.000230.00034 ± 0.00015
FLUX.1 [schnell]−0.00152 ± 0.00008−0.00095 ± 0.00011

Table A3. Visual fidelity assessment via KID Score (Mean ± Std) ↓. Near-zero or negative scores indicate no measurable degradation.

MetricValue95% CI
ODC Win Rate43.7%[35.3%, 52.4%]
Baseline Win Rate56.3%[47.6%, 64.7%]
Tie Rate58.0%—

Table A4. Human Evaluation of visual fidelity. Win rates are computed over decisive votes (ties excluded).

Binomial test p=0.93p = 0.93, Cohen’s h=0.13h = 0.13 (negligible).

Second, we conducted a human evaluation specifically targeting visual fidelity. Three domain experts performed 300 blind pairwise comparisons between baseline and ODC-corrected images. The results show a high tie rate of 58.0%, indicating that ODC does not introduce perceptible artifacts. Furthermore, the win rates in decisive votes (Method: 43.7%, Baseline: 56.3%) show no statistically significant difference (Binomial test p=0.93p = 0.93, Cohen’s h=0.13h = 0.13). This confirms that ODC preserves visual fidelity without measurable degradation, corroborating the KID findings. These results validate the core design of ODC. The method functions as a surgical modification to the conditioning space, successfully removing semantic drift components while ensuring the generated images remain firmly within the manifold of natural image distributions.

Additional Qualitative Results

In this section, we provide further qualitative results to complement the main paper. These examples demonstrate our method’s ability to improve prompt-image alignment and preserve visual fidelity across a diverse set of prompts.

Figure A3. Additional qualitative results comparing baseline generations with Orthogonal Drift Correction (ODC). For each prompt, we show the output from the baseline SDXL model (left) and our ODC-corrected version (right).

Figure A4. Additional qualitative results comparing baseline generations with Orthogonal Drift Correction (ODC). For each prompt, we show the output from the baseline FLUX.1 model (left) and our ODC-corrected version (right).

Retrieved Drift Vocabulary SorthS_{\mathrm{orth}}

To provide examples of the retrieved vocabulary used by ODC, we list the top-5 terms (SorthS_{\mathrm{orth}}) for the exact prompts featured in our qualitative results (Figure 2) in Table A5. These terms serve as the discrete textual surrogate for the isolated orthogonal drift vector (vorth\mathbf{v}_{\mathrm{orth}}). ODC utilizes the embeddings of these top-5 terms in aggregate to define the removal direction. Consequently, while individual words may appear abstract in isolation, their combined embedding closely approximates the true semantic drift direction, allowing ODC to target the spurious concepts accurately.

IndexPrompt TextModelRetrieved Drift Words (Sorth\mathcal{S}_{\mathrm{orth}})Interpretation
151A tree whose leaves are made of delicate, colored glassSDXLstained glass art, glass art, light panels, light panel, glassworkUnintended glass/art style elements.
149A medieval feast with ghosts sitting amongst the living guestsSDXLthe abbess, illustre, preillustration, overillustration, tabernaclerMisplaced religious/illustrative terms.
30Paint brushes growing like flowers in potSDXLutensils or brushes, craft materials, brush land, petrographer, makeup kitSpurious, unrelated tools/materials.
4Cherry blossoms falling on samurai armorSDXLwallpaper, rider and horse, person horse, illustre, art wall decorMisplaced compositional elements.
46Giant mushroom houses in enchanted forestSDXLmushroom by, forest floor, agaricic, agaricales, toaderUnexpected botanical elements.
63Clockwork owl perched on moonSDXLtartary, overillustration, odontolith, nightfowl, antecavernUnrelated archaic/anatomical terms.
280Candy-coated castle melting under sunrise...SDXLenchanted landscape, castle on, fantasy scene, rapunzel, castleGeneric fantasy tropes overriding details.
203Victorian astronaut playing violin inside ornate greenhouse...SDXLthe duo, musicians, a duet, twopenny, the two brothersErroneous secondary subject.
300Thunderstorm brewing inside snow globe...SDXLin the winter, house in the, with snow, semiglobe, snow globeLiteral winter concepts overriding interior.
10Northern lights dancing above crystal ice castleFLUX.1iglulirmiut, alpenglow, the arctic ocean, arcticwards, icelidaeUnexpected arctic/celestial concepts.
24Giant teacup island in cloudy skyFLUX.1islandlike, that island, land formation, an island, a small islandDetail misinterpreted as geography.
185A collection of magical potions, glowing on a dusty shelfFLUX.1liquor bottles, small bottles, alcohol bottles, bottles of liquor, bottle liquorMundane domain overriding fantasy context.
85Floating geometric islands at sunsetFLUX.1these islands, some islands, many islands, air lane sunrise, small islandsRepetitive island concepts.
31Lighthouse made of stacked booksFLUX.1lighthouse picture, lighthouse, light house, lighthouse base, the lighthouseRedundant literal subject terms.
158A mountain range shaped like a sleeping stone dragonFLUX.1the western alps, the alps, the eastern alps, the italian alps, the fra mauro formationGeographic literalism over metaphor.
291Lavender fields beneath glass dome...FLUX.1beekeeping, lavender plants, apiary, pollenlike, the pollenReal-world agriculture over sci-fi.
150An underwater city inhabited by elegant, humanoid sea creaturesFLUX.1the new atlantis, undersea, the ocean floor, underseas, the sea bottomEnvironmental setting overriding architectural subject.
268Digital renaissance fair where knights duel using AR dragonsFLUX.1dragonlike, a dragon, their armor, dragon statue, their armourLiteral dragons/armor over AR concept.

Table A5. Table A5: Retrieved Drift Vocabulary (Sorth\mathcal{S}_{\mathrm{orth}}) for Representative Prompts

References

  1. [1]Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01401, 2018.arxiv.org/abs/1801.01401
  2. [2]Black Forest Labs. Announcing Black Forest Labs. Web Page, August 2024. URL https://bfl.ai/blog/24-08-01-bfl.
  3. [3]Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM transactions on Graphics (TOG), 42(4):1–10, 2023.arxiv.org/abs/2301.13826
  4. [5]Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022.arxiv.org/abs/2208.01618
  5. [6]Wolf Garbe. SymSpell: 1 million times faster spelling correction and fuzzy search through symmetric delete spelling correction algorithm. https://github.com/wolfgarbe/SymSpell, 2025. Version 6.7.3.
  6. [7]Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting text-to-image diffusion models via initial noise optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9380–9389, 2024.arxiv.org/abs/2404.04650
  7. [8]Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.arxiv.org/abs/2208.01626
  8. [9]Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.arxiv.org/abs/2104.08718
  9. [10]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
  10. [11]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.arxiv.org/abs/2006.11239
  11. [12]Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength natural language processing in python, 2020. URL https://doi.org/10.5281/zenodo.1212303.
  12. [13]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.arxiv.org/abs/2106.09685
  13. [14]Taihang Hu, Linxuan Li, Joost van de Weijer, Hongcheng Gao, Fahad Shabaz Khan, Jian Yang, Ming-Ming Cheng, Kai Wang, and Yaxing Wang. Token merging for training-free semantic binding in text-to-image synthesis. Advances in Neural Information Processing Systems, 37:137646–137672, 2024.DOI
  14. [15]Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.arxiv.org/abs/2307.06350
  15. [16]Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image-to-text concept matching. Advances in Neural Information Processing Systems, 37:76177–76209, 2024.arxiv.org/abs/2404.03653
  16. [17]Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://storage.googleapis.com/openimages/web/index.html, 2017.
  17. [18]Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32–73, 2017.arxiv.org/abs/1602.07332
  18. [19]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730–19742. PMLR, 2023.arxiv.org/abs/2301.12597
  19. [20]Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is all you need for high-fidelity text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9005–9014, 2024.arxiv.org/abs/2312.06059
  20. [21]Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 4296–4304, 2024.
  21. [22]Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.arxiv.org/abs/2307.01952
  22. [23]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
  23. [24]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.arxiv.org/abs/1910.10683
  24. [25]Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.arxiv.org/abs/2204.06125

Paper details

vlnc:W_eguxtv9w
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents