Structure-retained low-rank adapters for weather synthesis

Introduction

Computer vision models deployed in real-world scenarios often encounter robustness problems. In outdoor settings, interference comes not only from hardware, such as photon and thermal noise (especially in low-light settings), but also from unpredictable environmental changes, such as weather conditions. Improving model performance is important for ensuring the reliability of deployed models, especially in safety-critical applications such as autonomous driving. However, collecting high-quality, sufficiently varied image data that contains various weather changes remains a challenge.

Sourcing real-world image data from the same scene in different weather conditions requires comprehensive control over dynamic objects. Existing datasets mostly contain image pairs under different conditions, e.g., sunny-rainy and sunny-foggy, without ensuring that the original objects present in the images (e.g., cars and passengers) or camera views are preserved [12, 39]. This induces not only weather-related distribution shifts, but also temporal and spatial variations.

Generative models can produce high-quality images while offering granular control over details like style (e.g., weather) and specific scene content [22, 24]. For style manipulation, previous works extracted style information from one reference image [14], or from a pair of images with and without the target style [15]. For text-to-image generation tasks, image style can be manipulated by changing style-related text descriptions. However, for image-to-image translation, manipulating the input image style using only text prompts is insufficient, either failing to produce strong changes or disrupting the image content [1, 32].

Synthesizing adverse weather conditions is a critical frontier for improving computer vision models deployed in outdoor scenarios, e.g., autonomous driving. Previous work relied on Generative Adversarial Networks (GANs), such as CycleGAN [40] that simulated diverse weather types. These methods often require high-quality paired training data, which is expensive to obtain. To circumvent this, [16] turned to graphics engines and large-scale generative models. For instance, Stable Diffusion XL (SDXL) [21] is used to curate precisely controlled weather attributes within synthetic scenes. Despite the increased variety these models offer, the original scene structure is often not preserved during image-to-image translation. ControlNet [36] was proposed to preserve structure information of an input image, by injecting structured spatial conditions, e.g., edges and depth, into intermediate feature maps. But this often leads to a loss of fine-grained details, such as original textures and object identities [13]. Other approaches that adopt instruction-based editing, such as InstructPix2Pix [2], frequently struggle with spatial relational reasoning, i.e., the models either focus solely on local effects (e.g., rain in the sky) while neglecting global consequences (e.g., reflections on the road), or alter global image color rather than specific weather elements.

In this work, we perform an architectural analysis of SDXL on the sensitivity of attention blocks to structural information, and subsequently devise a structure-retaining Low-Rank Adaptation (LoRA) [11] strategy to synthesize weather while preserving structure details. Compared to other weather synthesis methods, our method improves the variety of weather effects while requiring light computations. Compared with other style-manipulation approaches, our method preserves more structural and content information. Effectively, we reduce the need to curate high-quality paired image data through manual collection or graphic engines.

Our contributions are as follows:

1. An architectural analysis and identification of SDXL attention blocks that capture more structural and dynamic information within scenes;

2. A weather synthesis framework using structural priors in the form of low-rank adapters (structure-LoRA), to retain the layout and dynamic elements of input images. Our method improves realism and weather-synthesis effects while better preserving scene dynamic objects and structural layout compared to existing approaches.

Related works

Style transfer Image style transfer aims to change the style of a source image to match that of a reference image. With the development of vision-language models (VLMs), works have leveraged encoded semantic knowledge. In [9], it was shown that cross-attention layers in text-conditioned diffusion models extract spatial relationships between image pixels and prompt tokens. Manipulating cross-attention maps by changing only the text prompts (e.g., replacing a word or adding a specification) allows editing an image while preserving most of the original structural information.

Similarly, the utility of self-attention for capturing spatial relationships (e.g., color layouts) and local visual features (e.g., textures) was investigated in [14]. Replacing attention matrices of style reference images with those of a content image can preserve the original content while manipulating the style of the original image. However, some content of the style reference image might leak to the generated images, since content and style information extracted in the attention layers are entangled. By enabling ‘attention sharing’ during the diffusion process, style-consistent images with the first (i.e., style reference) image in a batch were generated, without optimization [10]. Explicitly learning style information, InST [38] maps the style representation of a reference image to text tokens that do not exist in the normal language, e.g., [C], and then provides such a text prompt to manipulate the style of the input images. Instead of text prompts as conditions, IP-Adapter [35] additionally injected image information extracted from a style/structure reference image into the attention layers of diffusion models. However, this approach is limited to either style manipulation or structure generation. For text-to-image (T2I) style transfer, the authors of [3] used a style encoder to extract multi-level style embeddings aligned with textual descriptions alongside an auxiliary content adapter to separate content and style.

To manipulate style information more precisely, several works have focused on decomposing style and content. This is done by disentangling and then replacing the style or content components of the images: the generated images may change in subject or style [30, 34]. The authors of [6] explored the sensitivity of each block of StableDiffusion XL to style or content information. This analysis guides the implicit separation of a single image style and content components by jointly learning the LoRA weights of the content-sensitive and style-sensitive blocks. UnZipLoRA [17] disentangles a single stylized image into content and style information as two independent LoRAs by encouraging orthogonality through regularization of LoRA weight matrices. PairCustomization [15] learns the stylistic differences from an image pair (a content image and a style reference image with similar structure) by jointly optimizing content-LoRA to reconstruct the content image and the style-LoRA to edit the style of the content image. However, these methods require per-pair or per-image optimization, which can be computationally expensive and impractical for real-time inference. The work of [25] introduces a hypernetwork that predicts the merging coefficients of content and style LoRAs, which generalizes well to unseen content and style LoRA pairs, thus eliminating the need for costly optimization. CSGO [33] trains a style-transfer model that takes style and content images as inputs, and learns to fuse content and style features by a feature injection block. This allows for style transfer during inference without fine-tuning or optimization. Rather than learning LoRA weights for style and content information, the work of [4] found that using the same noise sampling distributions and loss weighting schedules as those in the original model’s training might not address style features during diffusion. Namely, stylistic features are more prominent in the early denoising steps, which have a low signal-to-noise ratio, than in later steps. Based on this, [4] shifted the mean of the noise sampling distribution to induce more noise during fine-tuning, allowing for better style alignment.

Our structure-LoRA eliminates the need for a style reference image and modulates the weather of original images using only text prompts, thus avoiding content leakage. Importantly, structure-LoRA preserves the structure and content of input images, whereas existing approaches to style manipulation either require additional structural conditions or do not address structure preservation altogether.

Weather synthesis Adverse weather synthesis is important for computer vision applications, such as autonomous driving [20]. It facilitates evaluating model robustness under varying weather conditions and improving vision models in adverse weather. In [20] and [40], GANs and Cycle-GANs were applied respectively, to generate images with 14 adverse weather conditions. The work of [29] uses both physics-based and GAN-based methods to augment data with rain effects. However, these methods need high-quality paired image data for model training. The work of [16] curated data using graphics engines that precisely control weather attributes of the same scene, as well as a large image generative model such as StableDiffusion XL [21]. With the development of generative vision-language models, the authors of [19] leveraged DALL-E [23] to synthesize weather effects given pre-designed text prompts. To preserve the structural layout of images, the work of [8, 18] used ControlNet [36], which takes structure-condition images, e.g., Canny edge images. However, many precise details may be lost during the generative process, such as textures, color, and identity. Rather than using ControlNet, [26, 27] generated weather-modulated images with instruction editing models, such as InstructPix2Pix[2]. These models are trained on well-crafted synthetic image pairs with diverse and extreme weather conditions generated by advanced conditional diffusion models. However, local details of input images in inference time might not be preserved.

Preliminary

We investigate how changes to the condition embeddings affect the structural layout of the generated images. SDXL [21] consists of a 70-attention block UNet, each with cross- and self-attention blocks.

SDXL is trained on a latent diffusion objective $\mathcal{L}_{\mathrm{LDM}} = \mathbb{E}_{z \sim\mathcal{E}(x),\epsilon\sim\mathcal{N}(0,1),t}\left[\lVert\epsilon-\epsilon_{\theta}(z_t,t,c)\rVert^2\right]$, where $\epsilon_{\theta}$ predicts the added noise $\epsilon\sim\mathcal{N}(0,1)$ at timestep $t$ to latents $z \in\mathbb{R}^{H/8 \times W/8 \times4}$ corresponding to image $x \in\mathbb{R}^{H \times W \times3}$ encoded with $\mathcal{E}(\cdot)$.

ControlNet [36] creates a trainable copy $\theta'$ of the UNet encoder part of SDXL, keeping parameters frozen. Zero convolutions, $1 \times1$ convolutions with zero-init weights and biases are added between the frozen and trainable copy. For a given UNet block $F_{\theta}$, the output $y_C$ is formulated as $y_C = F_{\theta}(z_F) + \mathcal{Z}(F_{\theta'}(z_F + c))$, where $z_F \in\mathbb{R}^{h \times w \times d}$ is the input feature map and $c \in\mathbb{R}^{h \times w \times d}$ is the added condition after preprocessing. We show the partial view of SDXL with ControlNet in Figure 2a.

LoRA. Low-rank adapters are efficient modules that enable fine-tuning for large pre-trained models by optimizing rank decomposition matrices [11]. Given the weights of a pre-trained text-to-image diffusion model ($W_0 \in\mathbb{R}^{d \times k}$), LoRA learns an update matrix $\Delta W$ that can be decomposed into two low intrinsic rank matrices $A \in\mathbb{R}^{r \times k}$ and $B \in\mathbb{R}^{d \times r}$, with the rank $r \ll\min(d,k)$ (i.e., $\Delta W = BA$). During training, $W_0$ is frozen, while the two low intrinsic rank matrices $A$ and $B$ are updated. During inference, the model weights are fine-tuned by $W = W_0 + \Delta W$. This approach promotes efficiency by restricting the number of tunable parameters.

Method

SDXL block structure priors injection

Prior research examined how manipulating style or content keywords in text prompts, injected into various UNet blocks, impacts the resulting visual style and content [6]. However, sole text prompt manipulation through injection and replacement does not expose effects on structural image information.

In this work, we use ControlNet on SDXL to inject structural priors to each SDXL block. The combined model then generates structure-preserved images guided by text. Following an approach similar to [6], we investigate how manipulating ControlNet-encoded condition embeddings that are injected into SDXL affects the generated structure information. Specifically, we: (1) mask one of the condition paths while keeping the others unchanged; (2) maintain only one condition path, removing all others. We replace the masked path(s) with zero values to serve as null conditions (see Figure 2). During image generation, we employ a prompt template A photo of a [category name] alongside ControlNet that uses the Canny edges of the source image as the additional condition.

Dataset. To visualize and quantify how structural information is affected, we selected images from the Caltech 256 dataset [7], with simple backgrounds and clear object contours. Visualization results. Masking out the condition embedding fed to the middle block (noted as $W_1$) of SDXL results in images that do not preserve the original structure. As shown in Figure 3a, main objects either disappear, such as the cormorant, or become heavily distorted, such as the unrecognizable teapot or the zoomed-in iguana. Similarly, we observe from Figure 3b that retaining only the condition from the middle block enables SDXL to recover most structural details. Thus, for structure-preserving weather synthesis, we target optimizing the LoRA weights of the middle block of SDXL.

Structure-LoRA for weather synthesis

Guided by these insights, we design a structure-enhanced weather-synthesis adapter, offering a more lightweight alternative to traditional ControlNet-based conditioning. We follow DreamBooth LoRA [24] and finetune SDXL on a standard reconstruction objective using a general text prompt $A\ photo\ of\ [v_i]$. We avoid using $A\ photo\ of\ a\ [category\ name]$, since prompts with category names strongly bias the image generation towards the specific category, reducing the utility of the learned LoRAs. We follow the optimization setup of [6] to learn different LoRAs to represent varying styles, content, and structural information. At inference, a prompt template realistic in [rainy/cloudy/snowy/foggy] weather is used alongside the source image to guide the generation towards target weather effects, using a specific LoRA weight; e.g., structure (struct.) LoRA, content (cont.) LoRA, style (sty.) LoRA, or a combination of them. We overview the pipeline in Figure 4.

The classes are: human skeleton, helicopter, bathtub, tuning fork, T-shirt, ladder, fighter jet, revolver, bowling pin, cormorant, spoon, tricycle, knife, frisbee, sword, eyeglasses, diamond ring, tripod ipod, teapot, iguana, yoyo, soccer ball, ak47, and penguin.

Metrics. We use the Structure Similarity Index Measure (SSIM) [31] and the Learned Perceptual Image Patch Similarity (LPIPS) [37] to evaluate the preservation and realism of the structure of the generated images. SSIM quantifies the structural fidelity between source and generated images, ranging from 0 to 1, with higher values indicating better structure preservation. LPIPS measures perceptual differences aligned with human visual judgment using deep network feature embeddings. Lower LPIPS scores indicate greater perceptual similarity to human observation and thus better image quality and realism.

To evaluate the success rate of weather effect synthesis, we deploy an LLM-as-judge approach, using the prediction output of Gemini 2.5 flash. Giving a text prompt You are an image analysis model specializing in meteorological observations. Your task is to detect and describe visible weather effects in the provided image. Respond STRICTLY according to the JSON schema provided in Listing 1. The detected dominant weather effect is taken as the prediction of the model.

Experiments

Baselines and training settings. We select B-LoRA [6] and PairCustomization [15] as baselines. These two parameter-efficient fine-tuning methods are designed for style manipulation using style and content disentangled

LoRA weights, offering a straightforward comparison to our structure-LoRA regarding structural preservation. B-LoRA and PairCustomization can generate images in two setups, which are (1) text-to-image generation (T2I) and (2) image-to-image generation (I2I). Our method uses I2I generation to preserve structural information as much as possible, while manipulating the weather style with fewer constraints from the original image input. For a comprehensive analysis of the performance, we compare our method to the baseline methods using both settings. During I2I, the hyperparameter ‘strength’ controls the guidance scale of text prompts. A lower strength indicates details of an input image are largely preserved, while a higher strength indicates significant changes applied to the input image guided by the text prompt.

In our baseline experiments, we learn the weather style LoRA from a reference image depicting rain, snow, fog, or clouds. B-LoRA extracts style LoRA weights from a reference image or a set of similarly styled reference images. PairCustomization learns style-LoRA and content-LoRA from the pairwise differences between images. For weather conditions, this can include sunny - [rainy/snowy/foggy/cloudy] image pairs in exactly the same scene. Natural image datasets containing such image pairs are hard to obtain. We thus use Gemini 2.5 Flash Image (Nano Banana) [5] to generate these image pairs with text prompts modifying only the weather description.

Dataset

We select haztruck dataset, which contains truck images with hazmat plates, sourced from the internet [28]. The dataset is curated for hazmat plate detection and optical character recognition. After filtering low-quality images, we use 174 images out of 238 images. Example images are shown in Figure 5.

{
  "type": "OBJECT",
  "properties": {
    "sky_condition": {
      "type": "STRING",
      "enum": ["clear", "partly_cloudy", "overcast", "stormy", "foggy", "hazy"],
      "description": "The dominant sky condition."
    },
    "precipitation": {
      "type": "STRING",
      "enum": ["none", "rain", "snow", "hail", "sleet", "drizzle"],
      "description": "The type of precipitation observed."
    },
    "lighting_condition": {
      "type": "STRING",
      "enum": ["bright", "dim", "sunset", "night", "artificial"],
      "description": "The observed lighting condition."
    },
    "visibility": {
      "type": "STRING",
      "enum": ["good", "moderate", "poor"],
      "description": "The general visibility level."
    },
    "surface_condition": {
      "type": "STRING",
      "enum": ["dry", "wet", "snowy", "icy", "muddy"],
      "description": "The condition of the ground surface."
    },
    "wind_effects": {
      "type": "STRING",
      "enum": ["none", "visible motion blur", "bent trees", "blowing debris"],
      "description": "Observable signs of wind."
    },
    "dominant_weather_effect": {
      "type": "STRING",
      "enum": ["sunny", "cloudy", "rainy", "snowy", "foggy", "stormy", "other"],
      "description": "The single most dominant weather phenomenon."
    },
    "confidence_score": {
      "type": "NUMBER",
      "format": "float",
      "description": "Confidence of the analysis, between 0.0 and 1.0."
    },
    "reasoning": {
      "type": "STRING",
      "description": "A brief explanation of the visual cues used for the analysis."
    }
  },
  "required": [
    "sky_condition", "precipitation", "lighting_condition", "visibility",
    "surface_condition", "wind_effects", "dominant_weather_effect",
    "confidence_score", "reasoning"
  ]
}

Listing 1. JSON schema for weather analysis.

Quantitative results

We present the results of image generation quality in Figure 6, as we evaluated structure preservation, image realism, and the effectiveness of the weather effect. Annotations in the figure refer to strength values. Generation without

LoRA yields the best overall weather-synthesis effect, except for cloudy and foggy conditions. However, it shows lower SSIM than using a LoRA-enabled model at the same strength setting, indicating worse structure preservation. Using only text prompts without LoRA weights trained for structure preservation indeed limits the controllability of adding weather effects to a given scene and its applicability for systematic robustness tests of downstream tasks. The LPIPS values obtained by SDXL without LoRA are generally higher than those with LoRA at the same strength setting, indicating that the generated images are statistically less perceptually similar, i.e., less realistic. Among the results with different LoRA weights, images generated with our structure-LoRA weights show a greater degree of weather-effect manipulation. Compared with images generated using content-LoRA weights, those generated with structure-LoRA weights produce better weather synthesis, with a slight loss in structure preservation. This might be that content-LoRA, which represents object information, also contains partial structural information, but its use might limit style/weather manipulation. The weather synthesis success rates when using style-LoRA for cloudy and foggy conditions are generally higher than those using only structure-LoRA. We inspect the original images and find that many of them exhibit a mild, cloudy, or foggy effect, which has been encoded in style-LoRA. But the realism of the generated images using style-LoRA is much lower (see the high LPIPS value when strength=0.7 for cloudy effect synthesis). Combining style and structure LoRAs results in better cloudy and foggy synthesis effect while retaining structure information.

Qualitative results

In Figure 7 and Figure 8, we show rain synthesized images using different combinations of LoRA (w/ structure, w/ content, and w/ structure plus content) or without using LoRA. Images generated with lower strength values have almost no observ-

able synthesized weather effect, while a higher value results in synthesized rain streak and reflection on the road. The usage of structure-LoRA preserves most structure information, compared to that without using LoRA (generating images under strength=0.7 configuration replaces the original truck by a car). Observed from the results of LoRA w/ content and LoRA w/ structure plus content, the structure-LoRA better preserves overall structure details of the original image under high strength configuration while allowing for more obvious rainy synthesis effect. Images of other synthesized weather effects are in Figure 9.

Comparisons with baselines.

The quantitative comparisons between our method with B-LoRA and PairCustomization are presented in Table 1. Our method balances the synthesis of weather effects with the preservation of the original structure of the input images. We achieve the second-best weather-synthesis performance, except for cloudy weather, while our method preserves the most structural information of the original images (highest SSIM). The realism of our generated images ranks second compared with that of B-LoRA (I2I) w/ C.

Method	LPIPS$\downarrow$	SSIM$\uparrow$	Rainy$\uparrow$	Cloudy$\uparrow$	Snowy$\uparrow$	Foggy$\uparrow$	Avg.
Pair (I2I)	0.541	0.547	95.38	12.14	95.95	29.48	58.24
BLoRA (T2I) w/ C	0.726	0.318	60.92	<u>77.59</u>	0	20.69	39.8
BLoRA (T2I) w/o C	0.738	0.316	47.7	100	0.57	0	37.07
BLoRA (I2I) w/o C	0.431	<u>0.563</u>	53.45	39.66	2.3	43.1	34.63
Structure LoRA	<u>0.388</u>	0.601	<u>76.44</u>	28.74	<u>72.83</u>	<u>38.51</u>	<u>54.13</u>

We qualitatively compare our weather synthesized images with those generated using B-LoRA and PairCustomization in Figure 9. Under I2I generation setting, our method maintains more structure information, while adding more realistic weather effects to the images. B-LoRA instead results in almost no synthesized weather effect, and PairCustomization completely changes the structure information of the original images. Using B-LoRA T2I generation option, the generated images demonstrate different structure and content information even using the content LoRA of the original image. These results show that our method preserves more structure information with less compensation regarding the magnitude of style manipulation.

Limitations

Although structure-LoRA preserves more structural information of images during I2I translation, not all the details of the original images are maintained. For example, the number plate and hazmat sign in the images are important for identifying a truck and the hazardous material being transported. To recover these details, incorporating text inpainting could be a reasonable approach. Moreover, we observe that if an input image contains contrasting weather condition to the target weather condition, the model might fail to synthesize the targeted weather effect. This might be attributed to the insufficient guidance of a given text prompt to cover the original weather effect when the semantic gap between them is large. For future work, disentangling explicitly the weather effect from the original image content would be a promising direction.

Conclusions

Through an architectural analysis of SDXL-ControlNet, we uncover that structural condition extracted by the middle block of ControlNet is highly correlated to structure preservation in image generation. We propose a weather synthesis method that learns the corresponding LoRA matrices to image structure, enabling the manipulation of weather effects while retaining the original image structure and content. Our method requires light computations and training time, and improves weather effect variety.