WorksComputer Science
Spatially Gated Demographic Steering for Debiasing Multi-Person Text-to-Image Generation
Abstract
Text-to-Image models reproduce societal stereotypes, and existing mitigation applies one demographic correction to the whole scene. In multi-person scene generation this creates a trade-off: global steering improves aggregate balance only by homogenizing the people within an image. Steering-based methods like Embedding Arithmetic drive the intra-image diversity below the unsteered baseline. Text- and latent-projection approaches shift the aggregate skew with only marginal gains in intra-image diversity. We present Spatially Gated Steering (SGS), which steers the demographics of each person independently by extracting per-person masks from the Diffusion Transformer's (DiT) internal attention maps at a critical denoising timestep and applying a distinct demographic vector within each mask for the remaining steps. SGS needs no fine-tuning, inversion or layout specifications, and uses a lightweight segmentor over frozen DiT attention features to localize subjects at early timesteps. On skewed occupational prompts, SGS reaches high intra-image racial and gender diversity while keeping aggregate balance competitive and image quality near baseline, a result we validate on two architecturally distinct diffusion transformers, FLUX.1-Dev and SD3.5-Large. Codebase and model weights are available at https://github.com/cvims/spatially-gated-steering
Introduction
Text-to-Image (T2I) generation has transformed digital content creation, yet foundational models such as FLUX [24] and Stable Diffusion [10] systematically reproduce societal stereotypes [9]. Gender-neutral prompts like a carpenter yield predominantly male-presenting images; a nurse yields predominantly female-presenting images [28]. Studies show these outputs shift user beliefs about real-world distributions [14], making fairness mitigation essential for
responsible deployment.
Several mitigation strategies have been proposed and they operate at different levels of the diffusion pipeline [37]: They modify the text embeddings [6, 22, 39, 43] or fine-tune model weights [13, 36] to shift the aggregate output distribution. While effective for scenes with a single person, global mitigation methods share a fundamental limitation: They apply identical steering to every person in a scene. A prompt like ”a group of nurses” receives one demographic vector, pushing all the persons in the image in the same direction and failing to diversify individuals independently.
This limitation is not merely theoretical: Wan and Chang [41] show that multi-person prompts expose a distinct, compounding class of bias that single-subject evaluations cannot capture. Co-occurring subjects amplify stereotypes even when the same model behaves anti-stereotypically on single-person prompts.
Attempts to address this gap remain limited. Existing approaches either rewrite the generation prompt with a single corrective signal applied to the whole scene [8, 41], optimize demographic distributions at the aggregate level without per-instance control [18], or require an explicit spatial layout as input [7]. To the best of our knowledge, no existing method localizes individuals from the model’s own internal representations and applies independent demographic steering, to achieve intra-image diversity. Figure 1 illustrates this gap: while global mitigation shifts all persons uniformly toward a single demographic mode, our approach steers each individual independently through their spatial mask, reaching distinct attractors within in a single denoising pass.

Figure 1. SGS intuition. The base trajectory (black) reaches in a stereotyped region of ; global mitigation [39] (brown dashed) shifts it to one alternative mode. SGS instead intercepts at , extracts per-person masks, and steers each individual independently (orange/blue dashed) toward distinct demographic attractors within the same image .
We introduce Spatially Gated Steering (SGS), a framework for per-instance demographic steering in multi-person T2I scene generation. SGS operates by 1) Identifying and generating the masks of instances of people during the intermediate denoising steps using a lightweight detector, 2) Steering the demographics of the detected individuals, while also maintaining concept coherence.
Our contributions are as follows:
A trade-off in multi-person fairness. Global mitigation faces an inherent tension: improving aggregate balance requires collapsing intra-image diversity, sometimes below the unsteered baseline. Since any per-instance diversification trivially raises intra-image entropy, the meaningful result is high entropy at a competitive aggregate balance. SGS reaches high intra-image diversity with a competitive aggregate balance by steering per-instance without fine-tuning, flow-inversion or layout.
Instance segmentation from DiT features. We show that self-attention maps of DiTs at a critical denoising timestep encode spatially separable person instances, and train a lightweight DETR-style [4] segmentor on these frozen features that isolates individuals across crowd sizes and scene complexities.
Benchmark and analysis. We introduce a benchmark measuring fairness along both intra-image diversity and aggregate deviation, surfacing the homogenization failure mode of global methods and showing SGS is strong on both axes, on both FLUX.1-Dev and SD3.5-Large.
Related Works
Bias and Fairness in Text-to-Image Models. T2I models have been extensively shown to amplify demographic stereotypes encoded in training data, producing outputs skewed by gender, race and cultural identity across a wide range of prompt types [1, 5, 28, 35]. Beyond documenting these skews, a parallel line of work formalizes their measurement: Dehdashtian et al. [9] quantify stereotypical attributes in the generated set and trace their origin within the model, reporting that even high-fidelity systems such as FLUX retain strong stereotypical predispositions. Mitigation strategies operate at different points in the generation pipeline: prompt engineering injects demographic descriptors at inference [2, 11, 43], embedding manipulation steers text representations prior to image generation [6, 15, 22, 25, 26, 39], and parameter-efficient fine-tuning edits model weights to shift output distributions [13, 36]. Park et al.’s Entanglement-Free Attention [30] shares our goal of preserving non-target attributes during bias mitigation, training cross-attention predictors under a mask-constrained regularization loss so background edits are suppressed. However, like other global methods, it applies one scene-level attribute correction rather than steering each person independently. Despite this breadth, all existing methods treat the generated image as a single unit, applying one steering signal to the entire scene. This is sufficient when a prompt produces one person, but breaks down entirely for multi-person scenes where distinct individuals require independent control.
Multi-person Bias and Instance-level Control. This breakdown in multi-subject generation is primarily driven by semantic leakage, a phenomenon where the attention mechanisms of diffusion architectures inadvertently blend the visual and demographic features of different subjects during the denoising trajectory. Wan and Chang [41] demonstrate the consequences of this at scale: their Paired Stereotype Test shows that over 74% of DALL-E 3 images depicting two individuals with contrastive social identities display gender-occupational biases, even when the same model produces anti-stereotypical outputs for single-person prompts. Jung et al. [18] propose a Multi-Group Proportional Representation metric to measure and optimize demographic distributions across intersectional groups, but operate at the aggregate level without per-instance control. A related direction targets stereotypes that emerge from the co-occurrence of multiple concepts within a single prompt: Zhou et al. [47] show that pairing a demographic descriptor with an unrelated object can induce stereotypical associations invisible to single-concept audits, mitigated via a trained Sensitive Transformer that constrains the output distribution toward a stereotype-free target. This targets a distinct problem from ours, stereotyped associations between a subject and its surrounding context, corrected globally over the whole generation, rather than diversifying multiple people already explicitly present in a scene. Training-free inference methods such as Bounded Attention [7] attempt to prevent semantic leakage by confining attention flows within manually predefined spatial bounding boxes, but rely on rigid user-provided layouts rather than organic scene composition. Reinforcement learning approaches such as DisCo [3] directly fine-tune models to penalize intra-image facial similarity, but use face similarity as a proxy for demographic diversity rather than steering specific attribute vectors, and incur significant computational overhead. Global steering techniques such as Embedding Arithmetic [39] modify text embeddings uniformly, which fundamentally fails to produce heterogeneous intra-image diversity.
Diffusion Features for Spatial Understanding. Intermediate diffusion activations encode rich spatial structure without additional training: DIFT [38] shows mid-denoising U-Net features support zero-shot dense correspondence; ODISE [42] uses Stable Diffusion U-Net features for open-vocabulary panoptic segmentation. This extends to Diffusion Transformers only recently: ConceptAttention [16] shows DiT attention-layer projections yield sharper concept saliency than cross-attention alone; TRACE [17] shows diffusion models isolate object boundaries at a specific timestep, enabling spatial masks from internal attention. SGS extends this line of work to intra-image demographic fairness in multi-person scenes generated by T2I models.
Preliminaries
Rectified Flow Denoising
Rectified flow [27] learns a velocity field (a neural network with parameters ) that transports samples from a Gaussian noise distribution to the data distribution along approximately straight trajectories. FLUX [24] adopts this formulation, initializing generation from and iteratively denoising via:
where decreases from 1 (pure noise) to 0 (clean image) over steps, , and is the encoded text condition [31].
Multimodal Diffusion Transformer (MM-DiT)
The MM-DiT [10] architecture patchifies the noisy image latent , where are the latent height and width and is the token dimension, into image tokens, concatenates them with text conditioning tokens [31], and processes them through a stack of transformer layers, each applying joint image-text self-attention and a feedforward MLP. FLUX.1-Dev [24] uses double-stream layers; SD3.5-Large [10] uses joint transformer blocks.
Embedding Arithmetic
Embedding Arithmetic [39] computes a demographic attribute vector as the mean difference between embeddings generated by a biased prompt () and neutral prompts () over a set of profession prompts :
where denotes the text embedding and indexes the target demographic attribute. FLUX conditions denoising on two text encoders: a pooled CLIP-L/14 embedding (768-dim) that modulates the network globally, and a T5-XXL sequence embedding that carries the prompt text via cross-attention. Embedding Arithmetic’s direction vectors are computed and validated in the CLIP pooled space – and in Eq. (1) denote this pooled CLIP embedding, not the T5 sequence. and share an identical template except for an inserted demographic descriptor (e.g. “a photo of a {demographic} {profession}” vs. “a photo of a {profession}”), so isolates the demographic direction rather than any profession-specific signal. At inference, the original method shifts the conditioning vector globally as , where controls the steering strength.
Methods
SGS performs per-instance demographic steering within the standard rectified flow denoising loop, requiring no model fine-tuning, no trajectory inversion, and no external spatial layout specification. Crucially, instance masks are predicted directly from the MM-DiT’s attention features at a critical denoising step, rather than decoding the noisy latents for an external segmentor - which would be unreliable while the latent is still far from , and redundant since these features are already computed by the base pass. Figure 1 illustrates the intuition behind our approach and Algorithm 1 summarizes the complete inference procedure. The denoising loop is divided into three phases governed by two hyperparameters: and . Before , generation proceeds without intervention. At , instance masks are extracted and cached. Between and , a separate steered forward pass is run per detected instance at each denoising step, and the predictions are spatially accumulated. After , standard denoising resumes to completion.
Instance Detection
At step , the base forward pass is run with attention extraction enabled, returning the MM-DiT self-attention maps .
Algorithm 1 SGS Inference
Require: Noisy image latent , prompt embeddings (), attribute vectors , strength , , window
Self-attention maps. The self-attention map for layer is computed independently for each attention head and averaged across heads, capturing pairwise spatial affinities between image tokens [17]; the full expression is given in supplementary Section B.
Layer binning. In order to save compute and memory usage, rather than operating on all transformer layers independently, we partition them into non-overlapping bins of approximately 5 consecutive layers each for FLUX.1-Dev and 9–11 layers each for SD3.5-Large. We then compute the layer-wise mean of the self-attention maps within each bin, yielding four spatially resolved feature maps per timestep. Through an ablation study over all bin combinations (§ 5.4), we find that concatenating bins and yields the best instance segmentation performance. We therefore define the feature representation as:
where denotes channel-wise concatenation. is passed directly to the instance segmentor described below.
Segmentor Architecture. is passed to a DETR-style [4] instance segmentor. The model comprises a three-layer CNN backbone that projects into a spatial feature map, followed by a linear pixel projection that yields a sequence of pixel embeddings where is the transformer embedding dimension. A set of learnable instance queries are refined by a -layer transformer decoder attending over . The denoising timestep is incorporated via a sinusoidal embedding concatenated to the input features, conditioning the segmentor on the noise level at which features were extracted. Each refined query produces three outputs via dedicated linear heads:
where is the instance confidence, is the predicted center coordinate, and is the instance mask produced via a dot product between the query mask embedding and the pixel embedding sequence. Queries with confidence below a threshold are discarded, yielding active instances. More details, along with the hyperparameters of the segmentor can be found in the supplementary material (Section B).
Training Data. The segmentor is trained on a purpose-built dataset of 1440 generated multi-person images spanning persons across 9 scene types and 3 difficulty levels, with instance pseudo-labels produced by SAM2 [32] automatic mask generation. Full dataset details and the train/val split are provided in the supplementary material.
Training. The segmentor is trained with a Hungarian matching loss [4]. For each image, predicted queries are optimally assigned to ground-truth instances by minimising a cost matrix.
where , , are the predicted center, confidence, and mask for query , , are the ground-truth center and mask, and , weight the center and mask loss terms (values in supplementary Section B). Unmatched queries are supervised with a no-instance label. The mask head is trained with Dice loss [29] and binary cross-entropy on the matched pairs.
Mask caching and demographic assignment. The binary masks are cached and reused for all subsequent steps in the steering window without re-extraction. Each instance is assigned a target demographic attribute by uniform sampling over the intersection of gender and racial attribute categories, which can be overridden at inference time to reflect user-specified fairness targets.
Per-Instance Steering
For each timestep in the window , the base velocity prediction is first obtained from the standard forward process:
where is the conditional embeddings from the text encoder. Rather than the single global shift Embedding Arithmetic applies (§ 3.3), SGS shifts the conditioning vector independently per detected instance:
where controls the steering strength and is the demographic attribute assigned to the instance . SGS precomputes offline for a set of gender and racial attributes and applies Eq. (6) independently per detected person within the steering window described in § 5.5.
Fair Forward Pass. For each instance , a separate fair forward pass is run with the conditioning embedding steered via Eq. (6):
Spatially Gated Accumulation The per-instance steering vector is the difference between the fair and base predictions:
The steered velocity is accumulated across all instances, with each steering vector spatially gated by the corresponding cached mask of the instance:
where is the diversity strength and denotes the element-wise multiplication with broadcast over the feature dimension. The latent is then updated via Eq. (1).
Outside the steering window, the latent is updated using the base prediction only and no attention extraction, fair forward pass, or mask application occurs.
Experiments
Implementation Details
Model and Hardware Configurations. Our primary experiments use FLUX.1-Dev1 in bfloat16 precision [19] on a single NVIDIA A6000 GPU (48GB VRAM), with 21 denoising steps and a guidance scale of 4.5. To evaluate cross-architecture generalization, we additionally port SGS and all four baselines to SD3.5-Large2 in bfloat16 precision, run on a single NVIDIA A100 GPU (39GB MIG slice), at 10241024 resolution (its native training resolution) with 28 denoising steps and a guidance scale of
SGS’s instance-detection and per-instance steering mechanism
(§ 4.1, § 3.3) makes no FLUX-specific assumption beyond the existence of a joint image-text self-attention map inside the denoising transformer, which SD3.5’s MM-DiT also provides. SD3.5-Large’s EA direction vectors live in the joint pooled embedding of its two CLIP encoders (CLIP-L and CLIP-G, concatenated to 2048-dim), rather than FLUX’s single 768-dim CLIP-L pooled space; T5 remains excluded from vector construction in both cases.
Evaluation Metrics
Intersectional Entropy () Let denote the set of intersectional demographic groups over racial categories and genders , giving groups.3 For a single generated image with detected faces, let be the ratio of faces classified as group . We compute per-image intersectional entropy as:
and report its mean over all generated images. Normalizing the entropy with ensures that a image in which both persons belong to a distinct intersectional groups achieves , independent of . This captures intra-image demographic diversity which is the core objective of Spatially Gated demographic Steering, rather than aggregated statistics.
Deviation Ratio () Following [25, 40], for an attribute with category set let be the total number of detected faces classified as category and :
where is perfectly balanced and is maximally skewed. We report (: white, black, asian, indian) and (: male, female). Unlike the original single-person formulation, we aggregate face-level counts across all persons per image, which we find more appropriate for multi-person generation.
Image Quality We report CLIP score [31] for prompt fidelity and aesthetic score delta () relative to the unsteered baseline [33]. Face crops from the generated images are extracted using MTCNN [44]. Similar to prior works [39, 41], Demographic labels are produced by FairFace [20] detector, which we validate directly against known ground truth rather than assuming its accuracy (supplementary Section E).
Instance Segmentation
Evaluation Protocol. We evaluate instance segmentation on a held-out test set of our dataset as described in § 4.1. The test set has images across all person counts and three difficulty levels. We report mean intersection-over-union (mIoU), average precision at IoU threshold (AP50) and average precision averaged over thresholds 50%–95% (AP50-95). Features are extracted at each denoising timestep and we report performance across the full temporal sweep.
Architectural Comparison We compare four segmentation approaches of increasing supervisory complexity: K-Means (naive unsupervised baseline), Spectral clustering (following MaskDiffusion [21]), CenterNet [48] (supervised, lightweight), and our DETR-style model (§ 4.1). TRACE [17], which also extracts instance boundaries from diffusion self-attention maps, is a natural comparison, but its authors had not released training or inference code at submission time, precluding a faithful reproduction.
Figure 3 reports performance across FLUX denoising timesteps. All methods improve as denoising progresses toward 0.7–0.8 (step 4 of 21), then plateau or degrade as features shift toward fine-grained texture, confirming MM-DiT features become instance-discriminative at intermediate timesteps [17, 38]; together with the steering-window ablation (§ 5.5), this motivates . DETR outperforms all baselines at every timestep, peaking at mIoU 0.664, AP50 0.797, AP50–95 0.416 (vs. CenterNet’s 0.618/0.762/0.337), reflecting the gap between supervised and unsupervised instance separation from DiT features. A DETR trained analogously on SD3.5-Large reaches a comparable peak (Section F), so this timestep-localized discriminability is not FLUX-specific. Per-difficulty and qualitative detail: supplementary Sections C, G; Section D shows the segmentor is not the bottleneck behind SGS’s steering accuracy at , but shows weaker performance at .
Figure 3. Instance segmentation analysis (=720): segmentation performance across the FLUX denoising trajectory for four architectures. All methods peak around 0.7–0.8; DETR consistently outperforms all baselines. Layer bin ablation motivating our choice of Bins 3+4 as (Eq. (3)) is in supplementary Section B.
Layer Bin Ablation. Performance improves monotonically from Bin 1 to Bin 4, reflecting the progression of spatial structure across MM-DiT layer depth - earlier layers encode coarse layout while later layers capture fine-grained instance boundaries. Concatenating Bins 3 and 4 achieves the best performance across all metrics (mIoU: 0.631, AP50: 0.747, AP50-95: 0.368, averaged across all timesteps), outperforming other bin configurations (full per-bin table in supplementary Section B). This result directly motivates our choice of in Eq. (3).
Steering Parameters Ablation
We evaluate on a set of 8 occupations: 4 male-skewed (lawyer, physician, developer, politician) and 4 female-skewed (social worker, hairdresser, cashier, receptionist), generating 10 images per occupation for persons with randomly assigned target demographics per detected instance. We report five metrics: race and gender steering accuracy (, ) against the assigned target; aesthetic score delta () [33] relative to the unsteered baseline; background preservation ( [45]) outside the instance masks; and prompt–image CLIP similarity delta (), where values closer to zero indicate less semantic drift.
Steering Hyperparameters. We ablate EA strength and spatial diversity strength , reporting steering accuracy, aesthetic delta, background preservation, and CLIP similarity delta. We select and as our steering parameters; full sensitivity analysis is provided in Table S5 of the supplementary material.
| Acc<sub>race</sub>↑ | Acc<sub>gender</sub>↑ | AesΔ↑ | LPIPS<sub>bg</sub>↑ | CLIPΔ(→ 0) | |
| (a) EA strength α [λ=4.0 fixed] | |||||
| 1.5 | 0.573 | 0.722 | +0.009 | 0.916 | −0.026 |
| 2.0 | 0.708 | 0.757 | +0.027 | 0.900 | −0.033 |
| 2.5 | 0.723 | 0.740 | +0.010 | 0.890 | −0.042 |
| 3.0 | 0.636 | 0.738 | −0.134 | 0.876 | −0.052 |
| 3.5 | 0.438 | 0.566 | −0.762 | 0.842 | −0.104 |
| (b) Spatial strength λ [α=2.0 fixed] | |||||
| 3.0 | 0.644 | 0.734 | +0.020 | 0.914 | −0.028 |
| 4.0 | 0.708 | 0.757 | +0.027 | 0.900 | −0.033 |
| 5.0 | 0.736 | 0.742 | +0.034 | 0.889 | −0.039 |
| 6.0 | 0.767 | 0.746 | −0.019 | 0.876 | −0.046 |
| 7.0 | 0.758 | 0.742 | −0.120 | 0.864 | −0.053 |
Table S5. Steering hyperparameter ablation. Bold = selected configuration. AesΔ: aesthetic score delta vs. unsteered baseline (higher = better quality). LPIPS<sub>bg</sub>: background preservation. CLIPΔ: prompt–image CLIP similarity delta (closer to 0 = less semantic drift).
Steering Window. We additionally sweep detection/steering start step and end step , with and fixed (full grid and discussion in supplementary Section H, Table S6). Early detection () yields the highest steering accuracy but degrades background preservation and aesthetic quality, since MM-DiT features this early are insufficiently structured for reliable instance segmentation. Late detection () preserves background coherence but leaves too short a window to shift demographics effectively. We select , as our operating point for FLUX, balancing steering accuracy against background preservation and semantic drift. Similar Hyperparameter testing was done for SD3.5-Large and the results can be found in Section I.
Table S6. Steering window placement ablation (FLUX.1-Dev, , ). Bold = selected operating point. CLIP: prompt–image similarity delta vs. baseline.
| Start | End | Acc | Acc | Aes | LPIPS | CLIP |
| 2 | 8 | 0.713 | 0.784 | −0.097 | 0.760 | −0.097 |
| 2 | 10 | 0.718 | 0.783 | −0.123 | 0.759 | −0.102 |
| 2 | 12 | 0.749 | 0.791 | −0.179 | 0.758 | −0.105 |
| 2 | 14 | 0.754 | 0.783 | −0.237 | 0.757 | −0.106 |
| 4 | 8 | 0.632 | 0.744 | +0.008 | 0.904 | −0.029 |
| 4 | 10 | 0.708 | 0.757 | +0.027 | 0.900 | −0.033 |
| 4 | 12 | 0.741 | 0.743 | +0.036 | 0.899 | −0.038 |
| 4 | 14 | 0.769 | 0.745 | +0.036 | 0.898 | −0.041 |
| 6 | 10 | 0.476 | 0.649 | −0.001 | 0.961 | −0.013 |
| 6 | 12 | 0.545 | 0.652 | +0.002 | 0.958 | −0.016 |
| 6 | 14 | 0.594 | 0.649 | +0.007 | 0.956 | −0.018 |
Table S6.
Benchmark Comparison
We evaluate on 30 held-out WinoBias occupations (disjoint from the 10 used to fit EA’s direction vectors) 5 prompt templates seeds, on both FLUX.1-Dev and SD3.5-Large (§ 5.1; full list in supplementary Section J), yielding 4,500 images/method/architecture. We compare against Embedding Arithmetic [39], SDID [25], RespoDiff [40], and FairImagen [12]; DisCo [3] targets a related goal but requires fine-tuning and has no public implementation, precluding a faithful comparison. Each baseline was originally designed for UNet-based diffusion models; how we faithfully reimplemented them for MM-DiT is detailed in supplementary Section N. , CLIP, and are mean CI (normal approximation); DR uses a percentile bootstrap (1000 resamples), since it is a population-level statistic rather than a per-image mean.
Table 1 reports intra-image diversity () against aggregate balance () on both architectures under one protocol. Since rewards any per-instance diversification, the two must be read jointly: global methods steer every person identically and so cannot raise intra-image diversity. Embedding Arithmetic reaches the lowest on both models, but its falls well below the unsteered baseline on both – a single demographic vector pushes everyone toward the same group, improving aggregate counts while erasing intra-image variation. SDID, RespoDiff, and FairImagen leave close to baseline on both architectures, confirming that global latent and text-embedding projections do not transfer to either model’s joint-attention formulation. SGS is the only method to break this trade-off on both: highest by a wide margin, second only to EA, and prompt fidelity near baseline throughout. This pattern replicates on a second, architecturally distinct diffusion transformer, not just FLUX.
Table 1. Quantitative results on the extended WinoBias occupation benchmark (30 occupations × 5 templates × × 10 seeds, /method/architecture), on FLUX.1-Dev and SD3.5-Large. : intra-image intersectional entropy. DR: deviation ratio from uniform (lower is better). Bold and <u>Underlined</u> values represent the best and second-best methods per metric, within each architecture.
Table 1.
Human Evaluation
The metrics above rely on an automatic race/gender classifier (§ 5.1). To check that SGS’s gains are also visible to real people, not just to one automatic tool, we ran a small anonymous online study. Raters were shown two images side by side, one from SGS and one from a baseline, and asked which has more diverse people and which better matches the prompt, plus an overall preference. We collected 65 completed sessions (657 screens, 1971 judgments in total), comparing SGS against all five baselines.
Table 2 shows the share of judgments that favored SGS. Raters preferred SGS on diversity by a wide margin against every baseline (88–90%, all differences statistically significant). On matching the prompt, results are mostly a tie, except SGS is clearly preferred over EA (89%). The overall preference question shows the same pattern: SGS is clearly favored over EA (76%) and roughly tied with the rest. Agreement between raters (Krippendorff’s alpha, where 1.0 means perfect agreement and 0 means no better than chance) was 0.73 for diversity – strong agreement – and lower, 0.43–0.44, for the more subjective faithfulness/preference questions, which is expected for personal-taste judgments. Full study design, recruiting, and per-question detail are in the supplementary material (Section O).
Table 2. Human evaluation: share of judgments preferring SGS over each baseline ( raters). : statistically significant preference ().
| vs. | Diversity | Faithful. | Preference |
| Baseline | |||
| EA | |||
| FairImagen | |||
| RespoDiff | |||
| SDID |
Table 2.
Qualitative Analysis
Figure 2’s three occupations span orthogonal stereotype directions (nurses female-skewed; CEOs, farmers male-skewed), yet base outputs are homogeneous in all three. EA’s unifordm shift is clearest for nurses, where all four figures converge on one demographic, eliminating the intra-image variation that drives its below baseline. SGS instead maintains scene coherence across all three directions, confining steering to person regions rather than leaking into occupational cues like attire or background. Additional results are in the supplementary material (§ 20).

Figure 2. Qualitative comparison across three profession types spanning different stereotype directions. Prompts contain no demographic descriptors. Base is demographically homogeneous; EA shifts everyone uniformly; FairImagen is statistically indistinguishable from baseline on FLUX; SGS steers each instance toward a distinct intersectional target while preserving occupational context and scene coherence. Rightmost column: segmentation masks (nearest-neighbor upsampled from ) with assigned demographic labels.
Compute Cost
Table 3 reports per-image denoising time measured on a single A6000 GPU (48GB) across 10 runs after 3 warm-up passes. EA introduces no additional forward passes and is within measurement noise of the baseline. SGS incurs a deterministic overhead of exactly additional forward passes. For the generation takes 14.6s versus 7.8s for the baseline. This cost is proportional to the scene complexity and can be reduced by narrowing the steering window at the expense of steering accuracy (Table S6, supplementary Section H).
Table 3. Inference time per-image (A6000 GPU, , 21 denoising steps). FPs: total forward passes through the DiT per image. Overhead relative to unsteered baseline.
| Method | FPs | Time (s) | Overhead |
| Baseline | 21 | 1.0 | |
| EA | 21 | 1.0 | |
| SGS () | 33 | 1.6 | |
| SGS () | 39 | 1.9 | |
| SGS () | 45 | 2.2 |
Table 3.
Limitations
Three limitations bound the current system. First, segmentor accuracy degrades for persons in heavily occluded scenes, this also impacts the steering accuracy. Second, the per-instance forward pass introduces a compute overhead proportional to . Batching would reduce wall-clock time at the cost of VRAM, a trade-off we could not explore within our hardware budget. Third, SGS applies steering regardless of prompt intent; a query such as “four male doctors” will still redistribute demographics. A lightweight prompt parser that zeroes the steering signal for explicitly constrained slots could address this.
Conclusion
We presented SGS, a per-instance demographic steering framework for multi-person T2I generation requiring no fine-tuning, inversion, or layout input. Global steering improves aggregate balance only by driving intra-image diversity below the unsteered baseline; SGS is the only evaluated method to break this trade-off, raising intersectional entropy to 0.794 (from 0.524) on FLUX.1-Dev and to 0.832 (from 0.549) on SD3.5-Large, while recovering near-best aggregate balance and preserving semantic coherence on both architectures. Per-instance spatial steering thus emerges as a necessary step toward fair multi-person text-to-image generation, generalizing across architecturally distinct diffusion transformers rather than being a property of any single model.
Supplementary Material
Segmentor - Training Dataset Details
The segmentor is trained on a purpose-built dataset of 1440 generated multi-person images spanning persons across 9 scene types and 3 difficulty levels. The 9 scene types are: market, bus stop, city landmark, outdoor concert, train station, backyard barbecue, city park, street food fair, and mountain trail. Difficulty encodes camera framing and occlusion: easy uses a wide shot with subjects standing apart facing the camera; medium uses a medium shot with subjects side by side; hard uses tight close-up shots with partial occlusion among subjects. All images were generated using FLUX.1-Dev at 512 × 512 resolution with guidance scale 4.5 and 21 denoising steps.
Train/val/test split The 1440 training images are split into 80% training and 20% validation images, stratified by and difficulty level. The test set () is generated from a disjoint set of 9 out-of-distribution scene types with different framing protocol to evaluate generalisation beyond the training distribution.
Pseudo-label generation Instance segmentation masks are produced by SAM2 [32] automatic mask generation applied to each generated image. Predicted masks are filtered by minimum area and matched to ground-truth person count via Hungarian matching on mask centroid distance. Duplicate predictions with pairwise IoU > 0.5 are suppressed by retaining the higher confidence mask.
Segmentor - Architecture
We describe each component of the instance segmentor with exact hyper-parameter values as used in all reported experiments.
Input Features. The segmentor receives the binned self-attention feature map , formed by concatenating layer bins and of the MM-DiT self-attention at the detection timestep . Features are min-max normalized per channel before being passed to the backbone. This choice of bins is itself the result of an ablation (Table S1, moved here from the main paper’s § 5.4 for space): performance improves monotonically from Bin 1 to Bin 4 as layer depth increases, and concatenating Bins 3+4 outperforms every single-bin or other-pair configuration tested.
| Bin | mIoU | AP50 | AP50-95 |
| Bin 1 | 0.539 | 0.581 | 0.225 |
| Bin 2 | 0.583 | 0.661 | 0.287 |
| Bin 3 | 0.606 | 0.710 | 0.333 |
| Bin 4 | 0.626 | 0.736 | 0.360 |
| 3+4 | 0.631 | 0.747 | 0.368 |
Table S1. Table S1. Layer bin ablation (). Mean mIoU, AP50, and AP50-95 averaged across all denoising timesteps using DETR. Best in bold.
Timestep Conditioning The continuous FLUX timestep is encoded via a sinusoidal embedding followed by a learned linear projection with SiLU activation. The embedding is spatially broadcast and concatenated channel-wise to before the backbone, conditioning the segmentor on the noise level at which features were extracted.
CNN Backbone Three ConvBnReLU blocks (Conv2d, BatchNorm2d, ReLU, same-padding) progressively compress the high-dimensional attention channels into a compact spatial feature map while preserving the 32 × 32 resolution throughout. A final 1×1 convolution projects to the transformer embedding dimension , yielding pixel embeddings after spatial flattening.
Transformer decoder. learnable instance queries are refined by decoder layers. Each layer applies self-attention among the queries, cross-attention from queries to , and a two-layer FFN with GELU activation, all with residual connections and LayerNorm ( attention heads throughout).
Output heads. Each refined query produces three outputs via dedicated linear heads: a scalar confidence , a center coordinate , and an instance mask computed as a dot-product between a per-query mask embedding , reshaped to . is used only during training to stabilize the Hungarian matching.
Inference Queries with are discarded and the retained masks are binarized at > 0.5, yielding binary instance masks. No NMS is applied - is determined by the confidence threshold.
Training. The model is trained for 50 epochs with Adam (, batch size 8, 5 linear warm-up epochs). The 1,440 training images are split 80/20 into 1,152 training and 288 validation samples, stratified by and difficulty. Loss weights: , . The complete segmentor has 1.2M parameters, lightweight relative to the FLUX DiT backbone it operates on.
Segmentor Results - Per Difficulty
Figure 4 breaks down DETR segmentor performance across scene difficulty (left) and person count (right). AP50 remains consistently high across all cases, while AP50-95 drops substantially, indicating that instances are reliably detected but mask boundaries lack the precision required at stricter IoU thresholds. This gap widens at higher , where spatial competition between instances further degrades boundary delineation.
Figure 4. Segmentation performance of the SGS instance segmentor (DETR variant) broken down by scene difficulty (left) and person count (right). Left: mIoU, AP50, and AP50-95 averaged over all for easy, medium, and hard scenes. Performance degrades with difficulty primarily due to increased occlusion and smaller per-person regions. Right: The same metrics averaged over difficulty levels per . All three metrics decline as increases, reflecting the greater spatial competition among instances at higher person counts. Results are reported at denoising step .
Steering Performance by Person Count and Oracle-Mask Validation
Figure 4 above shows that segmentor detection quality (mIoU/AP) degrades with ; here we ask the downstream question directly: does that degradation propagate into steering accuracy, and how much of any remaining error is attributable to the segmentor rather than the steering mechanism. Both experiments below use FLUX.1-Dev only, at the operating point , , window .
Steering accuracy by , at the operating point. Table S2 slices the steering-window ablation (Section H) by person count at , . This ablation’s evaluation set covers only, the same scope as the original steering-hyperparameter ablation it is drawn from; it does not include .
| (mean) | ||||
| 2 | 0.733 | 0.748 | 2.050 | 0.662 |
| 3 | 0.683 | 0.766 | 3.075 | 0.861 |
Table S2. Steering accuracy by person count at the selected operating point ().
Oracle-mask (SAM2) vs. DETR segmentor, matched pairs. To isolate the segmentor’s contribution specifically, we re-run SGS with SAM2-derived masks (extracted from the finished, unsteered baseline image) in place of the DETR segmentor, on the same 445 images for both (identical occupation//template/seed; only the mask source differs).
| Mask source | |||||
| 2 | DETR | 0.843 | 0.896 | 0.792 | 0.743 |
| 2 | Oracle | 0.812 | 0.849 | 0.799 | 0.750 |
| 3 | DETR | 0.722 | 0.806 | 0.835 | 0.738 |
| 3 | Oracle | 0.705 | 0.776 | 0.902 | 0.740 |
| 4 | DETR | 0.643 | 0.720 | 0.834 | 0.756 |
| 4 | Oracle | 0.701 | 0.764 | 0.874 | 0.753 |
| Overall | DETR | 0.714 | 0.808 | 0.820 | 0.746 |
| Overall | Oracle | 0.740 | 0.796 | 0.858 | 0.748 |
Table S3. DETR vs. oracle (SAM2) masks, matched pairs, overall (149/148/148 at ).
The DETR segmentor matches or exceeds oracle-mask race accuracy at and (0.843 vs. 0.812; 0.722 vs. 0.705), but falls behind it at (0.643 vs. 0.701, a gap of 0.058). The Oracle SAM2 masks recover the race-steering accuracy that DETR loses once scenes become more crowded. Gender accuracy shows a related but smaller pattern: DETR leads at and (0.896 vs. 0.849; 0.806 vs. 0.776) while the oracle mask lead at by 0.04 (0.764 vs. 0.720). Background preservation () is flat across mask source at every , so this gap is not a fidelity tradeoff. Taken together, these results indicate the live segmentor is not the limiting factor behind SGS’s residual steering error at and . The gap to perfect accuracy comes from elsewhere in the pipeline (steering strength, classifier noise).
Classifier Validation
All race/gender numbers in this paper depend on one automatic tool, FairFace [20]. To read demographics off a generated face, before trusting it, we check how accurate it actually is against a set of images where the true demographic is known for certain, and we also check two other classifiers (a CLIP [31] zero-shot classifier, and DeepFace [34]) for comparison.
Building a set of images with known demographics. We generate images where every person in the group is asked for the same demographic (e.g. “a group of three Black women”), so the prompt itself has a cue for every face in that image. This is done for both FLUX.1-Dev and SD3.5-Large, across all 8 racegender combinations and group sizes . We keep only images where a face detector actually finds exactly faces, to avoid mixing in blank or wrong-count images; this leaves 1,832 individual face crops with a known-correct label.
Results. Table S4 reports accuracy for each classifier on this set. FairFace and CLIP are both highly accurate on race (95–96%) and near-perfect on gender (>99%); DeepFace is noticeably weaker on race (88%) and gender (93%). “Abstain” means the classifier’s raw output fell into a category we do not use (e.g. Middle Eastern, Latino/Hispanic, dropped due to high confusion with other categories reported in several previous works [36, 39]); this happens occasionally for FairFace and DeepFace and never for CLIP, which only ever outputs one of our four race categories directly.
| Classifier | Race Accuracy | Race Abstain | Gender Accuracy | Gender Abstain |
| FairFace | 95.7% | 2.5% | 99.7% | 0.0% |
| CLIP | 95.3% | 0.0% | 99.9% | 0.0% |
| DeepFace | 87.7% | 7.5% | 92.7% | 0.0% |
Table S4. Classifier validation against known-correct labels (=1,832 face crops, both FLUX.1-Dev and SD3.5-Large).
Do the three classifiers agree with each other? Separately from whether each classifier is correct, we check how often the three classifiers agree with each other (Krippendorff’s alpha, where 1.0 means perfect agreement and 0 means no better than chance). Agreement is high on race (0.94) and on gender (0.94), and every pairwise comparison between two classifiers also shows strong agreement (Cohen’s kappa between 0.85 and 1.00 in every case). This gives an independent check on the results above: it is not just that FairFace happens to agree with the ground-truth labels we built, three separately-built tools broadly agree with each other too.
Segmentor Timestep Sweep – SD3.5-Large
We repeat the segmentor timestep sweep of § 5.4 on SD3.5-Large, training a separate DETR-style segmentor on SD3.5’s own self-attention features at 1024×1024 resolution (same architecture and training recipe as the FLUX segmentor, Section B), and evaluating on a held-out validation split (=288) of SD3.5-generated images. Across SD3.5’s 28-step denoising schedule, segmentation quality peaks at timestep index 17, reaching mIoU of 0.664, AP50 of 0.841, and AP50-95 of 0.413 which is comparable to FLUX’s own peak (mIoU 0.664, AP50 0.797, AP50-95 0.416, § 5.4), indicating that MM-DiT self-attention features become instance-discriminative at an analogous point in SD3.5’s denoising trajectory as well.
Figure 6. SD3.5-Large DETR segmentor, timestep sweep (val split, =288). mIoU, AP50, and AP50-95 across SD3.5’s 28-step denoising schedule, mirroring Figure 3’s FLUX sweep. Performance peaks at timestep index 17 and plateaus around it, the same qualitative shape as FLUX’s own sweep.
Segmentor Results - Qualitative
Figure 5 compares instance segmentation predictions across four scenes spanning easy to hard difficulty. In the simpler market and tourist scenes (rows 1–2), all methods recover the correct instance count, though K-Means and Spectral clustering exhibit scattered noise pixels caused by high-frequency variation in the attention feature space. The gap widens in the close-up street scene and the outdoor hikers scene (rows 3–4): K-Means and Spectral produce heavily fragmented masks that merge or split individuals, while CenterNet over-estimates the instance count due to ambiguous overlap regions. DETR-based segmentor produces coherent per-person masks with the correct instance counts, demonstrating that the cross-attention mechanism of DETR learns to separate spatially proximate instances that confound clustering-based approaches.

Figure 5. Qualitative comparison of instance segmentation methods across four scenes of increasing difficulty. Each row shows the generated image, SAM2 oracle masks, and predictions from K-Means, Spectral clustering, CenterNet, and our DETR-based segmentor. K-Means and Spectral produce fragmented, noisy masks in complex scenes (rows 3–4), while CenterNet sometimes over-predicts the instance count under occlusion. DETR consistently recovers the correct number of instances with coherent boundaries across all difficulty levels.
Steering Hyperparameter Sensitivity
Table S5 reports the full sensitivity analysis for EA strength and spatial diversity strength , using the evaluation setup described in § 5.5 of the main paper.
For (panel a), steering accuracy and aesthetic quality improve from =1.5 to =2.0, then degrade at higher values. =2.5 achieves marginally higher race accuracy (0.723 vs. 0.708) but at the cost of gender accuracy, aesthetic quality, background preservation, and semantic coherence. At ≥3.0, aesthetic quality falls below the unsteered baseline and CLIP similarity drops sharply, indicating the shifted embedding has diverged from the original prompt semantics.
For (panel b), =4.0 achieves the best joint performance. Higher values improve race accuracy marginally but incur increasing aesthetic degradation and semantic drift; =7.0 produces a negative aesthetic delta (−0.120) and a CLIP drop of −0.053. We select =2.0 and =4.0 as our operating point.
Steering window. Early detection and steering at yields the highest steering accuracy across all end points but produces consistently negative aesthetic deltas and poor background preservation (LPIPS), indicating that MM-DiT features at step 2 are insufficiently structured for reliable instance segmentation, causing the steering signal to leak beyond the intended mask regions. Late detection and steering at preserves background coherence well (LPIPS) but achieves substantially lower steering accuracy (Acc, Acc), as the remaining steering window is too short to shift demographics effectively. achieves the best balance across all metrics: extending the end step from 8 to 10 improves steering accuracy (0.632→0.708 race, 0.744→0.757 gender) with only marginal reductions in background preservation, while further extension to or 14 offers diminishing accuracy gains at the cost of increasing semantic drift. We therefore select and as our operating point, corresponding to steps 4–10 of the 21-step denoising trajectory.
Steering Hyperparameter and Window Ablation – SD3.5-Large
We repeat the steering-strength and steering-window ablations on SD3.5-Large.
Table S7 sweeps and . Unlike FLUX, accuracy improves monotonically across the tested range on both parameters, with , the best-performing configuration tested; we select this as SD3.5’s operating point.
Table S7. SD3.5-Large steering-strength ablation. LPIPS: background preservation.
| Acc | Acc | LPIPS | ||
| 1.0 | 1.0 | 0.443 | 0.620 | 0.958 |
| 1.0 | 2.0 | 0.526 | 0.646 | 0.938 |
| 1.0 | 4.0 | 0.707 | 0.693 | 0.904 |
| 1.5 | 1.0 | 0.561 | 0.653 | 0.946 |
| 1.5 | 2.0 | 0.694 | 0.677 | 0.912 |
| 1.5 | 4.0 | 0.809 | 0.722 | 0.877 |
| 2.0 | 1.0 | 0.658 | 0.660 | 0.931 |
| 2.0 | 2.0 | 0.752 | 0.682 | 0.893 |
| 2.0 | 4.0 | 0.800 | 0.759 | 0.855 |
Table S7.
Table S8 sweeps detection/steering start step and end step, out of SD3.5’s 28-step schedule. start=4, end=24 achieves the highest accuracy among the configurations tested.
Table S8. SD3.5-Large steering-window ablation, , .
| Start | End | Acc | Acc | LPIPS |
| 3 | 10 | 0.391 | 0.624 | 0.921 |
| 3 | 12 | 0.402 | 0.624 | 0.920 |
| 3 | 14 | 0.427 | 0.642 | 0.919 |
| 4 | 11 | 0.378 | 0.623 | 0.945 |
| 4 | 13 | 0.397 | 0.617 | 0.944 |
| 4 | 16 | 0.424 | 0.617 | 0.944 |
| 4 | 19 | 0.453 | 0.626 | 0.944 |
| 4 | 24 | 0.514 | 0.622 | 0.944 |
| 5 | 10 | 0.338 | 0.610 | 0.964 |
| 5 | 12 | 0.357 | 0.613 | 0.963 |
| 5 | 14 | 0.386 | 0.613 | 0.962 |
| 6 | 10 | 0.293 | 0.590 | 0.977 |
| 6 | 12 | 0.310 | 0.590 | 0.976 |
| 6 | 14 | 0.340 | 0.595 | 0.975 |
Table S8.
Extended Benchmark – Occupations and Prompt Templates
Table 1 in the main paper (§ 5.6) evaluates on 30 WinoBias occupations [46], disjoint from the 10 occupations used to fit the EA direction vectors (carpenter, mechanic, construction worker, laborer, sheriff, teacher, nurse, housekeeper, flight attendant, librarian), so all baselines are evaluated out-of-distribution with respect to their own calibration set.
Occupations. 15 male-skewed: driver, mover, developer, farmer, guard, chief, janitor, lawyer, cook, physician, CEO, analyst, manager, supervisor, salesperson. 15 female-skewed: editor, designer, accountant, auditor, writer, baker, clerk, cashier, counselor, attendant, tailor, librarian, assistant, cleaner, hairdresser.
Prompt templates. Each occupation is instantiated under 5 templates, with and substituted as in the main paper:
“A portrait photo of s”
“A photo of s”
“A group of s”
“A realistic image of s at work”
“ s standing together”
Combined with and 10 seeds, this yields images per method.
Per-Occupation Demographic Distribution
Table S9 breaks down intra-image intersectional entropy () and aggregate deviation ratio (, ) per occupation, for the baseline and SGS, across all 30 occupations of the extended WinoBias benchmark (§ 5.6 of the main paper; FLUX.1-Dev). Occupations are grouped by the male-skewed/female-skewed split used to construct the benchmark (Section J).
Entropy. SGS raises over the baseline for every one of the 30 occupations without exception – the smallest gain is Driver (0.480→0.578) and the largest is Tailor (0.395→0.861), with no occupation where SGS’s per-instance steering fails to increase intra-image diversity relative to the unsteered model.
Aggregate deviation. drops for every occupation under SGS, typically substantially (e.g. Manager: 0.916→0.280; CEO: 0.907→0.288), confirming the aggregate race-balance improvement seen in Table 1 holds occupation-by-occupation, not just on average. is noisier at the per-occupation level and does not improve uniformly – most occupations move closer to balance (e.g. Chief: 0.707→0.267; Cashier: 0.440→0.040) but five of the 30 move further from it (Lawyer: 0.840→0.880; Accountant: 0.813→0.853; Counselor: 0.080→0.200; Assistant: 0.080→0.107; Cleaner: 0.360→0.520) – consistent with gender assignment being a coarser 2-category balancing problem where per-occupation sample size () leaves more room for variance than the pooled Table 1 figures show.
Intersectional Observation. Taken together, Table S9 confirms the core motivation for holds at the level of individual occupations, not only in aggregate: the baseline is both gender-stereotyped and racially homogeneous on essentially every occupation tested, and SGS’s per-instance steering raises intra-image diversity consistently across all of them.
| Occupation | Baseline | Baseline | Baseline | SGS | SGS | SGS |
| Driver | 0.480 | 0.544 | 1.000 | 0.578 | 0.324 | 0.680 |
| Mover | 0.406 | 0.486 | 0.920 | 0.717 | 0.297 | 0.627 |
| Developer | 0.482 | 0.794 | 0.987 | 0.843 | 0.279 | 0.787 |
| Farmer | 0.426 | 0.759 | 0.987 | 0.761 | 0.324 | 0.587 |
| Guard | 0.503 | 0.333 | 0.813 | 0.661 | 0.270 | 0.720 |
| Chief | 0.505 | 0.116 | 0.707 | 0.859 | 0.315 | 0.267 |
| Janitor | 0.547 | 0.674 | 0.800 | 0.810 | 0.306 | 0.600 |
| Lawyer | 0.582 | 0.866 | 0.840 | 0.735 | 0.289 | 0.880 |
| Cook | 0.489 | 0.687 | 0.893 | 0.858 | 0.421 | 0.267 |
| Physician | 0.598 | 0.640 | 0.640 | 0.862 | 0.280 | 0.587 |
| CEO | 0.385 | 0.907 | 0.987 | 0.793 | 0.288 | 0.987 |
| Analyst | 0.563 | 0.850 | 1.000 | 0.852 | 0.297 | 0.947 |
| Manager | 0.469 | 0.916 | 0.987 | 0.859 | 0.280 | 0.840 |
| Supervisor | 0.472 | 0.935 | 0.973 | 0.852 | 0.289 | 0.867 |
| Salesperson | 0.748 | 0.917 | 0.733 | 0.888 | 0.387 | 0.627 |
| Editor | 0.605 | 0.872 | 0.573 | 0.772 | 0.298 | 0.387 |
| Designer | 0.603 | 0.732 | 0.147 | 0.777 | 0.333 | 0.133 |
| Accountant | 0.642 | 0.927 | 0.813 | 0.869 | 0.374 | 0.853 |
| Auditor | 0.492 | 0.816 | 0.973 | 0.758 | 0.324 | 0.813 |
| Writer | 0.622 | 0.813 | 0.440 | 0.826 | 0.306 | 0.200 |
| Baker | 0.420 | 0.621 | 0.787 | 0.806 | 0.298 | 0.280 |
| Clerk | 0.534 | 0.890 | 0.947 | 0.857 | 0.289 | 0.760 |
| Cashier | 0.514 | 0.854 | 0.440 | 0.783 | 0.270 | 0.040 |
| Counselor | 0.623 | 0.937 | 0.080 | 0.824 | 0.279 | 0.200 |
| Attendant | 0.686 | 0.841 | 0.413 | 0.922 | 0.289 | 0.347 |
| Tailor | 0.395 | 0.756 | 0.973 | 0.861 | 0.324 | 0.653 |
| Librarian | 0.515 | 0.776 | 0.467 | 0.722 | 0.494 | 0.040 |
| Assistant | 0.566 | 0.757 | 0.080 | 0.805 | 0.324 | 0.107 |
| Cleaner | 0.507 | 0.569 | 0.360 | 0.650 | 0.297 | 0.520 |
| Hairdresser | 0.355 | 0.911 | 0.547 | 0.667 | 0.391 | 0.333 |
Table S9. Per-occupation demographic metrics, baseline vs. SGS (FLUX.1-Dev, extended benchmark, /occupation). : intra-image intersectional entropy (higher = more diverse). , : deviation ratio from uniform (lower = more balanced). Left block: male-skewed occupations. Right block: female-skewed occupations.
Extended Qualitative Analysis
Figure 7 presents an extended qualitative comparison across four occupations spanning both stereotype directions and three person counts . Occupations include two female-dominated professions (Nurse, Housekeeping) and two male-dominated professions (Sheriffs, Construction Workers).

Figure 7. Extended qualitative comparison across four occupations and three person counts . Left: baseline FLUX outputs. Right: SGS outputs. Rows show female-dominant (Nurses, Housekeepers) and male-dominant (Sheriffs, Construction Workers) professions. Baseline outputs are demographically homogeneous across all values and both stereotype directions. SGS consistently produces intersectional demographic diversity — each person within the same scene is steered to a distinct demographic target — while also preserving occupational context (uniform, apron, badge, hard hat) and scene coherence across all conditions. Performance is consistent across , demonstrating that the segmentor reliably separates and independently steers two, three, and four persons within the same image.
Baseline. Across all four occupations and all values of , the unsteered baseline produces demographically homogeneous outputs that reflect occupational stereotypes; Nurses and Housekeepers are consistently female-presenting or White, while Sheriffs and Construction workers are consistently male-presenting and White. This homogeneity persists regardless of , confirming that the stereotyping is driven by the text prompt rather than any scene composition effect introduced by specifying multiple persons.
SGS across person counts. SGS consistently produces intersectional demographic diversity across all values. At , each pair contains two persons steered to distinct intersectional targets—for example, a Black male nurse is paired with an Asian female nurse, or an Indian male housekeeper paired with an Asian female housekeeper. At and , the segmentor reliably separates all persons and assigns independent demographic vectors to each, with the resulting groups spanning multiple racial categories and both genders within the same image. The consistency across values demonstrates that the instance segmentor remains reliable as scene complexity increases.
Occupational context preservation. A key requirement of practical fairness steering is that demographic edits do not disrupt occupational cues. Across all conditions, SGS preserves the visual markers of each profession: nurses retain scrubs and stethoscopes, sheriffs retain badges and hats, housekeepers retain aprons, and construction workers retain safety helmets and reflective vests. This confirms that the spatially gated steering is confined to person regions via the instance masks, preventing demographic steering from leaking into occupational attire or scene background.
Qualitative Comparison – SD3.5-Large
Figure 8 repeats the main paper’s qualitative comparison (Figure 2) on SD3.5-Large, across four prompts and all five baselines. Images from Base, EA, FairImagen, RespoDiff, and SDID remain demographically homogeneous or only mildly shifted across all four scenes, while SGS diversifies every scene independently per person while preserving occupational context (aprons, rice-field setting, formal attire). This confirms the pattern shown quantitatively in Table 1 holds qualitatively on a second architecture as well.
Baseline Adaptation to Diffusion Transformers
FairImagen [12], SDID [25], and RespoDiff [40] were each designed and released against older UNet-based diffusion models – FairImagen against SD1.5 and SDXL, RespoDiff against SDXL, and SDID against a UNet-based Stable Diffusion pipeline – where generation is driven by a single-stream convolutional UNet with cross-attention to a text encoder. FLUX and SD3.5-Large instead use a Multimodal Diffusion Transformer (MM-DiT, § 3.2): image and text tokens are processed jointly through a stack of self-attention blocks, with no UNet and no single bottleneck layer. None of the three methods’ released implementations run on this architecture family as-is. We faithfully reimplemented each method’s core mechanism for MM-DiT, verified directly against its own released code, rather than mechanically porting UNet-specific hooks.
FairImagen. The original method operates on the model’s pooled text-conditioning vector. For each demographic group , it defines an empirical bias direction
and obtains a projector by solving
where collects the per-group mean directions and is a one-hot group indicator; thus spans the top-variance directions within the null space of the group-dependent signal . At inference, the projector is applied orthogonally to the pooled embedding, , before it conditions the UNet. MM-DiT models expose an analogous pooled conditioning vector alongside their token-level sequence conditioning, so we fit and apply this same projector on this real pooled vector, leaving the token sequence untouched.
SDID. The original method learns a single concept offset per demographic direction by
where is an image generated from a concept-inclusive prompt and is the same prompt with the concept term removed, with the base model kept frozen throughout. At inference, the offset is added directly to the UNet’s image latent bottleneck activation, , a single injection point on the image side of the network. MM-DiT has no UNet bottleneck to target, so we carry the same design principle over: the learned offset is added to the hidden state of one representative block within the joint-transformer stack, preserving a single image-side injection point rather than broadcasting the offset into the text-token stream.
RespoDiff. The original method decomposes its responsible transformation as
training its Responsible Concept Alignment Module (our race-aware/gender-aware modules) and Semantic Alignment Module , with
and injects their combined residual once at the UNet’s bottleneck hidden state . Our first attempt at reimplementing this for MM-DiT broadcast the same residual across every block of the joint-transformer stack; this reliably broke generation, producing incoherent output regardless of prompt or seed. Injecting the residual at a single representative block instead resolved this and produced coherent, steerable output. This also brings the reimplementation closer to the original method’s own single-bottleneck design than the all-block version we started from.
Shared adaptations. All three methods are applied only to the conditional half of the classifier-free-guidance batch, never the unconditional branch, consistent with how each was originally deployed. Training uses each target architecture’s own native noise parametrization and scheduler rather than a generic substitute.
Code Availability. Reference implementations of all three adapted baselines are included in the supplementary code (baselines/flux/ and baselines/sd35/): fairimagen.py, sdid.py, and respodiff.py, each containing the fit/training function and manipulation function described above with inline comments matching this section. SGS itself is in sgs/ (FLUX) and sgs_sd3/ (SD3.5-Large).
Human Evaluation Protocol
Automatic demographic classifiers may not fully capture how a human observer perceives diversity, prompt fidelity and preference. So we conducted a blind pairwise human evaluation comparing SGS against five baselines (Baseline, EA, FairImagen, RespoDiff and SDID) across 33 scenes spanning WinoBias occupations at people. per image, using a fixed seed per scene shared across all six methods. Each rater was shown two images side by side from the same scene, generated by two different methods (one always SGS), with method identity withheld (labeled only ’Image A’/’Image B’) and left – right placement randomized per pair. For each pair, raters answered three questions (refer Figure 9):

Figure 9. The human evaluation interface. Raters see a captioned image pair with method identity and position blinded, and answer three questions per pair: perceived diversity, prompt faithfulness, and overall preference.

Figure 8. Qualitative comparison on SD3.5-Large, mirroring Figure 2. Base, EA, FairImagen, RespoDiff, and SDID remain homogeneous or only mildly shifted across all four prompts; SGS independently diversifies each detected person while preserving occupational context.
Which image shows more diversity in race and gender?
Which image better matches the caption above?
Given a choice, which image would you prefer?
The first two questions offered a three way choice (Image A / about equal / Image B), while the last was a forced binary choice used to obtain a holistic preference signal. Each participants rated 10 pairs, two per comparison, ensuring balanced coverage across all five comparisons within a session.
Sessions in which a rater selected the identical response to every question throughout were excluded prior to analysis. For each comparison and question, we report the percentage of judgments preferring SGS (ties counted as half a preference) with a 95% Wilson score confidence interval and a two-sided exact binomial test against the 50% chance level (ties excluded from the significance test), alongside Krippendorff’s (nominal) as a measure of interrater reliability[23]; results are reported in Table 2.
References
- [1]Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Avlin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM conference on fairness, accountability, and transparency, pages 1493–1504, 2023.
- [2]Sarah Bonna, Yu-Cheng Huang, Ekaterina Novozhilova, Sejin Paik, Zhengyang Shan, Michelle Yilin Feng, Ge Gao, Yonish Tayal, Rushil Kulkarni, Jialin Yu, et al. Debiaspi: inference-time debiasing by prompt iteration of a text-to-image generative model. In European Conference on Computer Vision, pages 68–83. Springer, 2024.arxiv.org/abs/2501.18642
- [3]Shubhankar Borse, Farzad Farhadzadeh, Munawar Hayat, and Fatih Porikli. Disco: Reinforcement with diversity constraints for multi-human generation. arXiv preprint arXiv:2510.01399, 2025.
- [4]Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020.arxiv.org/abs/2005.12872
- [5]Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3043–3054, 2023.arxiv.org/abs/2202.04053
- [6]Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070, 2023.arxiv.org/abs/2302.00070
- [7]Omer Dahary, Or Patashnik, Kfir Aberman, and Daniel Cohen-Or. Be yourself: Bounded attention for multi-subject text-to-image generation. In European Conference on Computer Vision, pages 432–448. Springer, 2024.arxiv.org/abs/2403.16990
- [8]Farbod Davoodi, Pooriya Safaei, Sana Harighi, Parsa Gholami, Amirali Amini, Kimia Vanaei, Emad Firoozi, Parham Abed Azad, Babak Khalaj, Siavash Ahmadi, et al. Kg-fairdiff: Knowledge graph-guided prompt refinement for demographically fair text-to-image generation. In Culture x AI: Evaluating AI as a Cultural Technology (ICML 2026), 2026.
- [9]Sepehr Dehdashtian, Gautam Sreekumar, and Vishnu Boddeti. Oasis uncovers: High-quality t2i models, same old stereotypes. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2501.00962
- [10]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the 41st International Conference on Machine Learning, pages 12606–12633. PMLR, 2024.arxiv.org/abs/2403.03206
- [11]Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hinterstorf, Patrick Schramowski, Sasha Luccioni, and Kristian Kersting. Auditing and instructing text-to-image generation models on fairness. AI and Ethics, 5(3):2103–2123, 2025.DOI
- [12]Zihang Fu, Ryan Brown, Shun Shao, Kai Rawal, Eoin Delaney, and Chris Russell. Fairimagen: Post-processing for bias mitigation in text-to-image models. Advances in Neural Information Processing Systems, 38:29488–29512, 2026.
- [13]Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 5111–5120, 2024.arxiv.org/abs/2308.14761
- [14]Douglas Guilbeault, Solène Delecourt, Tasker Hull, Bhargav Srinivasa Desikan, Mark Chu, and Ethan Nadler. Online images amplify gender bias. Nature, 626(8001):1049–1055, 2024.DOI
- [15]Boyu Han, Qianqian Xu, Shilong Bao, Zhiyong Yang, Kangli Zi, and Qingming Huang. Lightfair: Towards an efficient alternative for fair t2i diffusion via debiasing pre-trained text encoders. Advances in Neural Information Processing Systems, 38:22671–22724, 2026.arxiv.org/abs/2509.23639
- [16]Alec Helbling, Tuna Han Salih Meral, Benjamin Hoover, Pınar Yanardag, and Duen Horng Chau. Conceptattention: Diffusion transformers learn highly interpretable features. In Forty-second International Conference on Machine Learning, 2025.arxiv.org/abs/2502.04320
- [17]Sanghyun Jo, Ziseok Lee, Wooyeol Lee, Jonghyun Choi, Jaesik Park, and Kyungsu Kim. Trace: Your diffusion model is secretly an instance edge detector. arXiv preprint arXiv:2503.07982, 2025.arxiv.org/abs/2503.07982
- [18]Sangwon Jung, Alex Oesterling, Claudio Mayrink Verdun, Sajani Vithana, Taesup Moon, and Flavio P. Calmon. Multi-group proportional representations for text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2505.24023
- [19]Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellemprudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of bfloat16 for deep learning training. arXiv preprint arXiv:1905.12322, 2019.arxiv.org/abs/1905.12322
- [20]Kimmo Kärkkäinen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint arXiv:1908.04913, 2019.arxiv.org/abs/1908.04913
- [21]Yasufumi Kawano and Yoshimitsu Aoki. Maskdiffusion: Exploiting pre-trained diffusion models for semantic segmentation. IEEE Access, 12:127283–127293, 2024.arxiv.org/abs/2403.11194
- [22]Eunji Kim, Siwon Kim, Minjun Park, Rahim Entezari, and Sungroh Yoon. Rethinking training for de-biasing text-to-image generation: Unlocking the potential of stable diffusion. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13361–13370, 2025.arxiv.org/abs/2408.12692
- [23]Klaus Krippendorff. Measuring the reliability of qualitative text analysis data. Quality and quantity, 38(6):787–800, 2004.DOI
- [24]Black Forest Labs. Flux. https://github.com/black-forest-labs/flux, 2024.
- [25]Hang Li, Chengzhi Shen, Philip Torr, Volker Tresp, and Jindong Gu. Self-discovering interpretable diffusion latent directions for responsible text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3, 6, 7, 8, 18arxiv.org/abs/2311.17216
- [26]Jia Li, Lijie Hu, Jingfeng Zhang, Tianhang Zheng, Hua Zhang, and Di Wang. Fair text-to-image diffusion via fair mapping. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 26256–26264, 2025. 3arxiv.org/abs/2311.17695
- [27]Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2022. 3arxiv.org/abs/2209.03003
- [28]Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Evaluating societal representations in diffusion models. Advances in Neural Information Processing Systems, 36:56338–56351, 2023. 1, 3DOI
- [29]Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565–571. Ieee, 2016. 5
- [30]Jeonghoon Park, Juyoung Lee, Chaeyeon Chung, Jaeseong Lee, Jaegul Choo, and Jindong Gu. Fair generation without unfair distortions: Debiasing text-to-image generation with entanglement-free attention. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 17567–17576. IEEE, 2025. 3arxiv.org/abs/2506.13298
- [31]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. 3, 6, 14arxiv.org/abs/2103.00020
- [32]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. In International Conference on Learning Representations, pages 28085–28128, 2025. 5, 12arxiv.org/abs/2408.00714
- [33]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278–25294, 2022. 6, 7arxiv.org/abs/2210.08402
- [34]Sefik Ilkin Serengil and Alper Ozpinar. Boosted lightface: A hybrid dnn and gbm model for boosted facial recognition. Gazi University Journal of Science, 39(1):452–466, 2026. 14DOI
- [35]Preethi Seshadri, Sameer Singh, and Yanai Elazar. The bias amplification paradox in text-to-image generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6367–6384, 2024. 3arxiv.org/abs/2308.00755
- [36]Xudong Shen, Chao Du, Tianyu Pang, Min Lin, Yongkang Wong, and Mohan Kankanhalli. Finetuning text-to-image diffusion models for fairness. In The Twelfth International Conference on Learning Representations, 2024. 1, 3, 14
- [37]Megan Smith, Venkatesh Thirugnana Sambandam, Florian Richter, Matthias Uhl, Laura Crompton, and Torsten Schön. Operationalizing fairness in text-to-image models: A survey of bias, fairness audits and mitigation strategies. In ICLR 2026 Workshop on Algorithmic Fairness Across Alignment Procedures and Agentic Systems, 2026. 1
- [38]Luming Tang, Menglin Jia, Qianqian Wang, Cheng Peng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 3, 6arxiv.org/abs/2306.03881
- [39]Venkatesh Thirugnana Sambandam and Torsten Schön. Embedding arithmetic: A lightweight, tuning-free framework for post-hoc bias mitigation in text-to-image models. In International Conference on Pattern Recognition, pages 308–323. Springer, 2026. 1, 3, 4, 6, 7, 8, 14
- [40]Silpa Vadakkeveetil Sreelatha, Sauradip Nag, Muhammad Awais, Serge Belongie, and Anjan Dutta. Respodiff: Dual-module bottleneck transformation for responsible & faithful t2i generation. In Advances in Neural Information Processing Systems, pages 42551–42583. Curran Associates, Inc., 2025. 6, 7, 8, 18
- [41]Yixin Wan and Kai-Wei Chang. The male ceo and the female assistant: Evaluation and mitigation of gender biases in text-to-image generation of dual subjects. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9174–9190, 2025. 1, 2, 3, 6arxiv.org/abs/2402.11089
- [42]Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2955–2966, 2023. 3arxiv.org/abs/2303.04803
- [43]Cheng Zhang, Xuanbai Chen, Siqi Chai, Chenyu Wu, Dmitry Lagun, Thabo Beeler, and Fernando De la Torre. Iti-gen: Inclusive text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3969–3980, 2023. 1, 3arxiv.org/abs/2309.05569
- [44]K. Zhang, Z. Zhang, Z. Li, and Y. Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10):1499–1503, 2016. 6DOI
- [45]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 7arxiv.org/abs/1801.03924
- [46]Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, 2018. 16arxiv.org/abs/1804.06876
- [47]Junlei Zhou, Jiashi Gao, Xiangyu Zhao, Xin Yao, and Xuetao Wei. Association of objects may engender stereotypes: Mitigating association-engendered stereotypes in text-to-image generation. Advances in Neural Information Processing Systems, 37:51754–51786, 2024. 3DOI
- [48]Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint arXiv:1904.07850, 2019. 6arxiv.org/abs/1904.07850


