betaFeedback

WorksComputer Science

Adapters as Data: The Latent Space of LoRAs

Abstract

LoRA adapters are usually treated as finished artifacts rather than as a data modality. Prior work has shown that LoRA adapters populate a low-dimensional manifold that a linear PCA basis already organizes well enough to support different operations such as sampling, editing, and inversion. We ask what a learned, nonlinear latent adds beyond this. We train a transformer autoencoder on the same population of LoRA identity adapters, mapping each into a latent space and compare every operation against the identical operation applied directly to the raw adapter weights. The latent preserves the capabilities already available in weight space, sampling diverse identities and editing individual attributes. Beyond preservation, we show that the latent is far more robust to displacement than weight space, degrading into distorted but still valid faces rather than texture. Fixed-dimensional weight-space representations cannot natively host adapters of more than one rank, so we extend our sequential autoencoder to hold mixed-rank adapters within one shared latent. This gives a path toward treating adapter weights of varying architecture as a single, general modality rather than one tied to a fixed rank, and points to a learned, nonlinear latent as the mechanism that could carry this generalization further, toward heterogeneous populations of adapters.

Introduction

Public repositories now host hundreds of thousands of LoRA adapters, yet each is still produced by its own training run and treated as a finished artifact. A network’s weights are also points in a high-dimensional space, and once many networks exist these points form a broader distribution rather than existing in isolation. This premise underlies weight space learning, which treats the parameters of neural networks as a data modality in their own right and learns representations that predict model properties directly from parameters [30, 23, 19] or synthesize new weights without training them from scratch [24, 25, 11, 31, 1, 7].

A narrower section of this literature has begun to ask the same question of Low-Rank Adaptation (LoRA), now the default mechanism for adapting large foundation models such as Stable Diffusion [20] to new identities, styles, and tasks. Since each adapter is still trained separately, composition of existing adapters remains a heuristic and unstable blend, and an adapter’s behavior is hard to predict before generation. weights2weights (w2w) [2] was first

40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: Neural Network Artifacts as a New Data Modality. to show that a large collection of identity-encoding LoRAs occupies a low-dimensional manifold that a simple PCA basis already organizes well enough to support sampling, editing, and inversion. We build directly on this setting, using the same base model, the same identity-encoding rank-1 adapters, and the same released dataset, but explore a different direction. Other methods generate LoRA weights outright rather than characterizing the space they come from, whether by pairing an autoencoder with a diffusion prior, mapping task prompts directly to weights, or predicting an adapter from a single image within a fixed subspace, and each is evaluated mainly by the quality of the adapter it produces [10, 14, 22]. What remains open is whether the space a method samples from or restricts itself to is organized in a stronger sense than downstream performance alone can reveal. That would mean neighborhoods smooth under perturbation, operations that behave consistently rather than only within the convex hull of what was fit, and a representation flexible enough to host more than one fixed adapter architecture at once.

We study this question on a sample of w2w’s population of roughly 65,000 identity-encoding LoRA adapters, each fine-tuned on a shared frozen diffusion backbone to reproduce one synthetic face identity and annotated with binary CelebA attributes. Rank-1 adapters are our primary setting, matching w2w directly, and we later extend to rank-2 adapters as well. Using the SANE methodology [25], these weights are tokenized and an autoencoder is trained in a self-supervised manner to map them into a latent space Z\mathcal{Z}, with no contrastive objective, and no attribute conditioning. Our central question is whether Z\mathcal{Z} is a structured representation of the adapter population, rather than merely a compression of it. We probe this along three axes: (i) whether operations already known to work directly on raw adapter weights, sampling new identities and steering individual attributes, are preserved once routed through a learned, nonlinear latent; (ii) whether that latent is more robust to displacements than the raw weight space; and (iii) whether a single autoencoder can absorb adapters of more than one rank. Our contributions are as follows.

  • We introduce a methodology for tokenizing LoRA adapters into weight sequences and embedding them into a structured latent space with a self-supervised transformer autoencoder.

  • We show that the resulting latent preserves the sampling and editing capabilities already present in the raw weight space, is substantially more robust to displacement, and remains stable under repeated encode-decode cycles.

  • We extend the representation to multi-rank LoRA adapters via a padding scheme that lets a single autoencoder process adapters of different rank within one shared latent.

Related Work

Weight Space Learning. Representation learning over neural network weights spans both model analysis and weight generation. Regarding model analysis, work focuses on predicting model properties from weights [30, 3] or studies the structure of trained weights itself, motivating permutation- and equivariance-aware representations [18, 35, 12, 15]. Closer to what we build on, a separate line of work treats populations of weights themselves as a data modality independent of any specific downstream task. Schürholt et al. [2021] introduce self-supervised embeddings of model populations and subsequently show these can be used generatively, sampling unseen but performant weights for initialization, ensembling, and transfer learning [24]. SANE [25] scales this idea beyond small networks by tokenizing weight matrices into a sequence and processing it with a transformer autoencoder, which keeps the per-token dimensionality independent of overall model size, and is the architecture we adopt in this work. In parallel, weight generation has also been explored directly with diffusion and other generative priors [31, 32, 29]. A broader overview is provided in Han et al. [2026], Ettling et al. [2026]. Collectively, this body of work establishes that weight tensors can be treated as a learnable distribution with its own latent structure.

Attribute Editing and Adapters. Low-Rank Adaptation (LoRA) [9] freezes a pretrained model and learns a low-rank update for selected weight matrices, typically at a single fixed rank shared across an entire collection of adapters, and is the adapter format we study throughout. FVAE-LoRA [13] further separates task-salient from residual information through variational factorization of layer activations during fine-tuning, whereas we learn a latent over adapter weights. Existing adapters can be combined post hoc, e.g. ZipLoRA [26] merges an independently trained subject and style LoRA via a learned coefficient mask, but such methods treat each adapter as fixed and optimize a merging procedure rather than a shared latent. Attribute control is more commonly approached through a learned direction in an existing latent: Concept Sliders [5] train a dedicated LoRA per attribute on a frozen diffusion model, while InterFaceGAN [28], GANSpace [8], and SeFa [27] fit, respectively, a linear SVM boundary, an unsupervised PCA direction, and a closed-form eigendecomposition to steer a pretrained GAN’s latent. We apply the same two ideas, a fitted direction for editing and PCA for diversity, one level removed to a latent over adapter weights instead of images.

Modeling LoRA Populations. A separate line of work generates LoRA adapters directly from a population, rather than editing or combining existing ones at a time. COND P-DIFF [10] adds a conditional diffusion prior over an autoencoded LoRA latent, and Drag-and-Drop LLMs [14] uses a hypernetwork to map task prompts directly to LoRA weight updates in a single forward pass, without any latent space. We instead embed adapters without any external conditioning signal, using only the weights themselves, so the same encoder can in principle be applied to any adapter, rather than requiring a paired task description or prompt. weights2weights (w2w) [2] shows that a large collection of identity-encoding LoRAs already lies on a manifold that a simple PCA basis organizes well enough to support sampling, editing, and inversion, and is the direct baseline we build on and compare against throughout this paper. All existing methods are built around adapters of a single fixed rank and offer no way to mix ranks without being rebuilt from scratch. We address that limitation directly by extending a learned latent to hold adapters of more than one rank in a single shared representation.

Approach

We take a single LoRA identity adapter θ=(A,B)\theta= (A, B), trained on top of a frozen base model, and embed it into a structured latent space from which new adapters can be sampled and edited. Figure 1 shows the full path. Following the SANE scheme [25], the adapter matrices are reshaped and sliced row-wise into a sequence of fixed-size tokens, yielding a [312×320][312 \times320] token sequence for the rank-1 adapters that form our primary setting, together with a positional encoding and a mask marking real weights against padding. A GPT-style, non-causal transformer encoder EE maps this sequence to a latent z=E(θ)∈R312×256z = E(\theta) \in\mathbb{R}^{312 \times256}, preserving the token axis, and a symmetric decoder DD reconstructs the tokenized weights from it, trained with a masked reconstruction objective. Once an adapter is represented as a latent zz, we study a range of operations on that space. We sample new latents that decode to novel but coherent adapters and steer a latent along learned directions to edit specific attributes (§ 5). We probe the geometry of that space under perturbation and test its stability under repeated encode-decode cycles (§ 6). We extend the representation itself to host adapters of more than one rank within a single shared latent (§ 7). In every case the manipulation happens on zz and is realized by decoding and de-tokenizing the latent into an adapter θ^\hat{\theta}, applying it to the frozen base model, and rendering it to an image under a fixed prompt and seed, so that differences between renders are attributable to the adapter rather than to the rendering process. When applicable, every operation we apply to zz is additionally applied directly to θ\theta, bypassing the autoencoder entirely. This weight-space control is what lets us attribute any structure we find to the latent representation rather than to the adapter population itself. Architecture, training, and tokenization details, including the alternative scheme used for the mixed-rank setting (§ 7), are given in Appendix § 11.

Overview of our approach, showing tokenization and autoencoder, operations on LoRAs, and LoRA-conditioned generation

Figure 1. Overview of our approach. (a) The LoRA matrices AA and BB are tokenized and encoded by a transformer encoder EE into a latent zz, which a decoder DD reconstructs back into an adapter. (b) We study five operations on this latent: sampling new identities, editing individual attributes, perturbing an adapter by a controlled amount, testing stability under repeated encode-decode cycles, and extending the latent to adapters of more than one rank. (c) The reconstructed or manipulated adapter modulates a frozen latent diffusion model to generate an image conditioned on a prompt pp.

Experimental Setup

Data

Our adapters are LoRA fine-tunes of a single, shared, frozen base diffusion model, Realistic Vision v5.12, a community fine-tune of Stable Diffusion 1.5 [20] optimized for photorealistic portraiture. Sharing one base model is what makes it meaningful to compare, interpolate, and embed different adapters, since every θ\theta modifies the same underlying weights. Each adapter is trained with a DreamBooth-style objective [21] to reproduce one synthetic identity under a fixed identifier token. For our primary, rank-1 setting we follow Dravid et al. [ [2], whose released dataset and binary CelebA [16] attribute labels we use throughout. The dataset comprises 64,974 identity adapters labeled along 40 CelebA attributes with widely varying prevalence, from near-balanced (Male, 48.3%) to rare (Bald, 2.5%). One attribute, Wearing Hat, is constant at 0% across the dataset and is excluded from any attribute-conditioned experiment, leaving 39 variable attributes. From this pool we subsample 8,192 adapters, split into train, validation, and test, giving ∼\sim5,734 training adapters. The full set of released adapters corresponds to 1,256 distinct underlying identities (median 6 adapters each), and our splits are drawn per adapter rather than per identity, so near-duplicate fits of the same person can appear across different splits. For the rank-2 setting used in our mixed-rank experiments, no equivalent released population exists, so we train a smaller pool of rank-2 adapters ourselves, following the same base model, DreamBooth objective, and identifier-token convention as the rank-1 dataset. Details are given in Appendix § 11.

Evaluation

Most of our experiments ask qualitative questions, e.g. whether a sampled latent decodes to a genuinely new person, or whether an edit moved one attribute while leaving the rest of the identity unchanged, and a quantitative score can only approximate this. We nonetheless report two scores where they are informative. For weight-space fidelity, we use the coefficient of determination R2(θ,θ^)R^2(\theta, \hat{\theta}) between an adapter and its reconstruction, and for image-space fidelity, we compute the LPIPS distance [34] between the renders of an original and a reconstructed or manipulated adapter. Neither is a reliable proxy for identity preservation on its own and aggregating either can hide exactly the qualitative distinctions our experiments turn on, such as an adapter degrading into texture versus into a different but valid face. We therefore treat rendered figures as the primary evidence throughout, and use R2R^2 and LPIPS to establish trends across many checkpoints or identities, and to surface differences too small to see reliably. Across 32 checkpoint–identity pairs, the two scores are in fact tightly coupled (Pearson and Spearman ρ=−0.90\rho= -0.90, p<10−11p < 10^{-11}, also holding under SSIM [33] and pixel-wise MSE), confirming that weight-space R2R^2 and perceptual quality track each other closely.

The Latent Space Preserves Properties of the Weight Space

In this section we test whether operations already known to work directly on raw adapter weights [2] also work when routed through a learned, nonlinear latent trained with no label supervision. We examine two such operations here, sampling new identities and editing existing ones, and defer a third, arithmetic and interpolation between two identities, to Appendix B.1.

Sampling

Can a latent code drawn without reference to any training adapter be decoded into a new, coherent identity, and if so, are the draws diverse rather than collapsing onto a single average?

Setup. The simplest scheme, naive sampling, fits an independent Gaussian to each latent coordinate, using the per-dimension mean and standard deviation of the encoded training latents, and draws each coordinate independently from its own marginal. A sample is drawn as a full [312×256][312 \times256] latent sequence. Because independent per-dimension sampling discards the covariance structure of the latent, the correlations between coordinates that encode which combinations of features co-occur among real identities, we also sample in the principal-component basis of the encoded training latents. A PCA is fitted to the flattened training latents, the top kk components are retained, and a sample is drawn as

zsample=zˉ+η∑j=1kξjλj uj,ξj∼N(0,1),z_{\mathrm{sample}} = \bar{z} + \eta\sum_{j=1}^{k} \xi_j \sqrt{\lambda_j}\,u_j,\qquad\xi_j \sim\mathcal{N}(0,1),

where zˉ\bar{z} is the mean latent, uju_j and λj\lambda_j are the jj-th principal component and its variance, and η\eta is a diversity factor that trades realism against variety: η=1\eta= 1 reproduces the natural spread of the population, larger η\eta pushes samples further from the mean.

Results. Naive sampling produces valid, well-formed faces, but nearly the same face every time (Figure 2, left). Decoded identities share pose, coloring, and features to the point of looking like minor variations of one person, confirming that samples decode to coherent faces but collapsing toward an average identity rather than generating diversity. Sampling in the principal-component basis recovers the missing variety (Figure 2, right). With k=256k = 256 components and η=1\eta= 1, decoded samples are clearly distinct identities, differing in gender, age, hairstyle, and coloring, while remaining coherent and realistic. So many components are needed because the latent’s variance is spread broadly rather than concentrated in a few directions. The leading component alone explains only 3.1% of it, the top 10 only 11.9%, and even the top 256 together capture just 40.9%.

Figure 2. (a) Naive sampling collapses toward average-looking identities. (b) PCA sampling with η=1\eta= 1 and k=256k = 256 preserves substantially greater diversity.

Relation to w2w. weights2weights [2] follows the same recipe directly on raw adapter weights, fitting a PCA basis over the flattened LoRA updates and sampling coefficients along the leading components to obtain new identities. That we need the identical construction here is consistent with our autoencoder never being pushed to disentangle. Since it was trained with a reconstruction objective alone and no prior encouraging independent, unit-scale dimensions, there is no reason to expect the latent’s coordinate axes to align with the population’s true directions of variation, and naive per-dimension sampling requires exactly that alignment to work.

Editing

Given one identity, can a single named attribute be changed continuously while the rest of the identity is left intact?

Setup. We estimate an attribute direction v\mathbf{v} by ridge regression, where training latents are flattened into Z∈RN×DZ \in\mathbb{R}^{N \times D} (N=5,734N = 5{,}734, D=79,872D = 79{,}872), CelebA labels are mapped to y∈{0,1}N\mathbf{y} \in\{0, 1\}^{N}, and v\mathbf{v} is the ridge solution v=arg⁡min⁡w∥Zcw−yc∥22+λ∥w∥22\mathbf{v} = \arg\min_{\mathbf{w}}\lVert Z_c\mathbf{w} - \mathbf{y}_c\rVert_2^2 + \lambda\lVert\mathbf{w}\rVert_2^2 (λ=1\lambda= 1) on centered data, solved in its dual form since D≫ND \gg N. This serves a similar purpose to the implicit shrinkage w2w [2] gets from truncating its PCA basis, though ridge shrinks every direction smoothly rather than discarding low-variance ones outright. An identity is edited as z′=z+γv\mathbf{z}^{\prime} = \mathbf{z} + \gamma\mathbf{v} (with γ=0\gamma= 0 giving back the reconstruction), where γ\gamma is expressed as a fraction of ∥z∥\lVert\mathbf{z}\rVert so that edit strength is comparable across identities and directions. Before rendering any sweep, we also score each direction v\mathbf{v} by the held-out AUC of adapters ranked by their projection onto v\mathbf{v}, the probability that a randomly chosen positive outranks a randomly chosen negative.

Results. On Bald, one of the rarest attributes in the dataset (2.5%), the flattened edit still produces a clean, continuous sweep (Figure 3). At γ=0\gamma= 0 the reconstruction renders the unedited identity, and as γ\gamma increases the hairline recedes and then disappears into a fully bald head, with intermediate strengths producing intermediate hairlines and the rest of the identity, skin tone, expression, and framing, preserved throughout. The identical ridge direction fitted directly on the raw adapter weights and applied without the autoencoder produces the same qualitative result over a comparable sweep (Figure 3). This matches what w2w [2] already showed for a linear direction fitted post hoc on raw LoRA weights, here with a ridge in place of a truncated-PCA pseudo-inverse. Applying the same construction to four further attributes on held-out identities (Figure 4, more examples in Figure 11 in Appendix B.2) shows edit quality tracking the AUC score fairly well. Male and Eyeglasses score highest and edit the most cleanly, the former moving continuously between a feminine and a masculine rendering with hair length, jawline, and facial hair changing together, the latter adding glasses as a discrete object. Gray Hair follows close behind with a milder but still clearly visible lightening of the hair, while Oval Face, well below the others, produces an edit that barely registers.

Latent-space and weight-space edits showing a full-hair-to-bald transition

Figure 3. Editing along the fitted ridge direction produces the same full-hair-to-bald transition whether applied to the latent (top) or directly to the raw adapter weights (bottom). The two rows use different γ\gamma ranges since the direction’s scale differs between the two spaces.

Attribute edits for Male, Eyeglasses, Gray Hair, and Oval Face across ridge-direction strengths

Figure 4. Edit quality declines with the direction’s held-out AUC, from clean edits for Male and Eyeglasses to a barely visible change for Oval Face.

Additional identities for the four attributes in Figure 4, showing the effect across identities

Figure 11. Additional identities for the four attributes in Figure 4, showing that the effect is consistent across identities.

The Latent Space Improves Robustness

This section tests the latent under two kinds of displacement: a single perturbation of varying size, compared directly against the same perturbation applied to the raw weights, and repeated encode–decode passes on the same adapter, which is what happens in practice whenever a sampled or edited latent is fed back through the autoencoder.

Perturbation

How far can an adapter be perturbed before the identity breaks down, and does it fail gracefully or collapse abruptly?

Setup. We test how far an adapter can be moved, in either space, before the generated identity breaks down, and whether it degrades smoothly or collapses abruptly. Each perturbation is scaled by the standard deviation of the space it acts in, so a noise scale α\alpha denotes the same self-relative displacement, a fraction α\alpha of that space’s own spread. In the latent we form z′=z+ασ^zϵz' = z + \alpha\hat{\sigma}_z \epsilon for a real identity’s latent z=E(θ)z = E(\theta), with ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0, I) and σ^z\hat{\sigma}_z the global standard deviation of the latent. In weight space we form the analogous θ′=θ+ασ^θϵ\theta' = \theta+ \alpha\hat{\sigma}_\theta\epsilon, with σ^θ\hat{\sigma}_\theta the global standard deviation of the adapter population. We sweep α\alpha over the same range in both spaces.

Results. Weight-space perturbation degrades the identity abruptly (Figure 5, left). At α=0.5\alpha= 0.5 identities remain broadly recognizable, though some detail is already lost. By α=0.75\alpha= 0.75 most have collapsed into texture-like, off-distribution images that are no longer faces. At α=1.0\alpha= 1.0 every identity has dissolved into noise. The transition is sharp, with little intermediate regime in which the adapter is perturbed yet still valid. Perturbing the latent by the same fractional amount degrades far more gracefully (Figure 5, right). At α=0.5\alpha= 0.5 identities are essentially preserved. At α=0.75\alpha= 0.75, where the weight-space adapters have already collapsed, latent-perturbed identities remain clearly recognizable despite larger variation. Only at α=1.0\alpha= 1.0 does the identity begin to break down, and even then it tends to degrade into a different or distorted face rather than into the texture-like noise seen in weight space. The raw weight space supports valid adapters only within a narrow band around each trained adapter. The neighborhood of a real point in the latent space, by contrast, is populated by valid, related identities.

Weight-space and latent-space perturbations across four identities and increasing noise levels

Figure 5. Perturbations in (a) weight-space and (b) latent-space across increasing noise levels α\alpha. Each column depicts one identity under the same perturbation level.

Reconstruction Drift

Does the small error of a single encode–decode pass compound under repetition, or does it settle to a fixed point?

Setup. We repeatedly apply the encoder and decoder to the same adapter. Starting from a real adapter θ(0)=θ\theta^{(0)} = \theta, we form θ(k)=D(E(θ(k−1)))\theta^{(k)} = D(E(\theta^{(k-1)})) for k=1,…,10k = 1, \ldots, 10, applying each θ(k)\theta^{(k)} to the frozen base model and rendering it under the fixed prompt and seed used throughout. We track R2R^2 between θ(k)\theta^{(k)} and the original θ(0)\theta^{(0)} and the LPIPS distance between their renders. A per-step change that decays toward zero would indicate a fixed point, while a change that stays roughly constant would indicate unbounded accumulation.

Results. Identities survive all ten cycles (Figure 6). Facial structure, hair, coloring, expression, and framing remain intact, and almost all visible change happens at the first cycle alone. The two quantitative measures capture different sides of this. LPIPS saturates, rising to 0.23 after one cycle and reaching only 0.36 after ten. R2R^2 declines steadily from 1 to about 0.945, and the per-cycle loss grows rather than shrinks, from 0.004 over the first five cycles to 0.007 over the second five. Ten cycles are safe in practice, since the renders stay recognizable and the perceptual cost of each additional cycle is small, but that safety reflects the renderer’s tolerance rather than the autoencoder converging. The weight-space deviation gives no sign of stopping, so extrapolating to many more cycles is not supported by this result.

Stability under repeated encode–decode cycles

Figure 6. Stability under repeated encode–decode cycles. (a) Weight-space similarity to the original adapter decreases gradually. (b) Rendered images increasingly differ from the original, measured by LPIPS. Shaded regions denote ±1σ\pm1\sigma. (c) Representative renders after k=0,1,3,5,7,10k = 0, 1, 3, 5, 7, 10 cycles.

The Latent Space Generalizes Across Adapter Ranks

Can a single autoencoder absorb adapters of varying ranks, treating adapter weights as a general modality rather than one tied to a fixed architecture?

Setup. Doubling the LoRA rank doubles adapter dimensionality from d=99,648d = 99{,}648 to d=199,296d = 199{,}296, so a rank-2 adapter does not fit the token sequence a rank-1 model expects. Our dense tokenizer assigns each token a fixed range of weight indices. At rank 2, the same token index falls in a different part of the network than it does at rank 1. The positional encoding the encoder relies on would therefore become misleading. We instead use a boundary-respecting sparse tokenizer at width 640, sized to a matrix’s rank-2 parameter count. A rank-1 matrix then occupies only the first half of its token budget, with the remainder marked as padding. Token position refers to the same matrix at both ranks. We train one autoencoder on a pool containing both rank-1 and rank-2 adapters and compare it against the rank-1-only model used throughout the rest of our experiments, reporting reconstruction quality as LPIPS on held-out adapters of each rank.

Results. A single autoencoder, with no increase in latent size or model capacity, reconstructs recognizable identities at both ranks (Figure 7). The rank-2 half of every token is therefore genuinely learned rather than ignored, and a shared latent over two adapter ranks is achievable within the same backbone used throughout the rest of our experiments. This capacity is now split between two ranks rather than dedicated to one, and the cost shows up in fidelity. On rank-1 adapters the mixed model reaches an average LPIPS of 0.29 against 0.22 for the rank-1-only model, about 32% worse, and it reaches 0.35 on rank-2 adapters. § 4.2 shows that weight-space R2R^2 and perceptual quality track each other closely, so this gap reflects a loss of reconstruction and places the mixed model in the range we observed for rank-1 adapters before convergence.

Figure 7. A single mixed-rank autoencoder reconstructs recognizable identities at both ranks. Rows 1–3 compare rank-1 adapters reconstructed by the rank-specific model against the mixed-rank model, and rows 4–5 show rank-2 adapters reconstructed by the mixed-rank model.

Limitations

Splitting. Train, validation, and test splits are drawn per adapter rather than per identity, so near-duplicate fits of the same person can appear across different splits. This does not affect the qualitative renders that carry most of our claims, but it makes held-out numbers such as the AUC optimistic relative to a strict identity-level split, since a direction can be scored against adapters that are near-duplicates of ones it effectively saw in training. Identity-level splitting is a straightforward fix left to future work.

Reconstruction floor. The best rank-1 model reconstructs adapters to an average LPIPS of about 0.22, and the first encode-decode cycle in our drift experiment costs 0.23, almost the same number, and larger than the drift accumulated over the following nine cycles combined. Every latent operation in this paper is therefore observed through a reconstruction already perceptibly different from its source, which is why the γ=0\gamma= 0 control in each edit sweep is D(E(θ))D(E(\theta)) rather than θ\theta itself. w2w, which samples and edits directly in weight space, has no equivalent floor.

Scope. Several limits on scope apply throughout. Everything rests on rank-1 and rank-2 updates to the query and value projections of a single frozen backbone, trained by the same DreamBooth-style procedure, so no claim about adapter weights as a general modality follows from this paper alone. Attribute editing also does not strictly require the learned latent, since the same ridge direction fitted directly on raw adapters produces a comparable edit, reproducing w2w in a different basis. The attribute fits themselves are underdetermined. Training uses about 5,734 adapters against a flattened latent of 79,872 dimensions, so the ridge fits interpolate the labels even at λ=1\lambda= 1, and the fitted direction itself is not uniquely identified by the data.

Out-of-distribution generalization. We test generalization across LoRA rank but not across subject domain. The autoencoder is trained exclusively on adapters encoding human identities, and whether its latent organizes adapters more broadly, for instance the dog-breed adapters also released by w2w, or has specialized to faces specifically, is a question this paper leaves open for future work.

Discussion

A structured latent over adapters mainly changes cost rather than capability. A new identity becomes a decoder pass instead of a training run, and a new attribute becomes one closed-form solve against an existing Gram matrix instead of a dedicated LoRA. More broadly, our experiments show that the different operations possible in the LoRA weight space are preserved in the learned latent, with two distinct advantages beyond preservation. The first advantage appears under displacement. Perturbations of comparable magnitude in weight space cause considerable degradation of the identity, while the learned representations show greater robustness and still decode into recognizable identities. This behavior is particularly relevant for downstream operations such as optimization and generation, where greater robustness to perturbations is necessary. The second primary advantage is the possibility to learn representations of different rank adapters. Operations in the raw adapter weight space require a shared dimensionality, whereas our tokenized approach represents different rank adapters within a common latent space.

Taken together, these results show that a learned LoRA latent space provides a more robust and extensible representation for performing multiple different downstream operations. Whether this advantage hold across different domains and tasks remains a key question for moving towards adapter weights as a general data modality.

Broader Impact

The identities studied here are synthetic, but nothing in the method depends on that, and composing this latent with an inversion step like w2w’s, turning a single photograph into a point in weight space, would in principle extend it to real people, letting a specific individual’s apparent age or gender presentation be moved continuously while their identity is preserved. This kind of consistency is exactly what makes fake personas cheap at scale. A second concern sits in the attribute directions themselves. Several CelebA attributes are annotator judgments rather than observable properties, and fitting a direction to one turns a cultural judgment into a coordinate that can be adjusted continuously on a real person’s face. This paper uses only synthetic identities and does not implement inversion, which keeps it clear of the most direct misuse path, though that is a property of the experiments rather than a guarantee of the method. We take the position that adapters encoding real people should require consent at the level of the identity, obtained beforehand.

Implementation Details

Adapter instantiation. LoRA is inserted into the query and value projections of both self- and cross-attention in the U-Net, while the key projections and all other weights stay frozen. Each pair on an image-space projection contributes 2C2C parameters at channel width CC, and three of the four adapted projections are of this kind. Cross-attention’s value takes the 768-dimensional CLIP text embedding as input, adding a further 768 parameters. Summed over one block this gives 7C+7687C + 768 parameters, and over the 16 attention blocks spanning C∈{320,640,1280}C \in\{320, 640, 1280\} (Table 1), a total adapter dimensionality of d=99,648d = 99{,}648 at rank 1. Doubling the rank doubles the per-block count to 14C+1,53614C + 1{,}536 and the total to d=199,296d = 199{,}296. The rank-2 adapters used in Section 7 are generated with the original training pipeline released by Dravid et al. [ [2], changing only the LoRA rank from 1 to 2. The base model, DreamBooth objective, and prompt template are otherwise identical to the released rank-1 recipe. Our released repository operates on the resulting adapter weights.

Channel width CCParams per block (7C+7687C + 768)BlocksTotal
3203,008515,040
6405,248526,240
12809,728658,368
Total (16 blocks)1699,648

Table 1. Per-block and total LoRA parameter count at rank r=1r = 1.

Tokenization. Dense tokenization (Figure 8a) chunks the flattened adapter into fixed-size tokens of width 320, the smallest channel width in our U-Net. The other channel widths are integer multiples of it, so every LoRA matrix except the cross-attention value (which operates on the 768-dimensional text embedding) tiles into a whole number of tokens with no leftover. As an alternative, sparse tokenization (Figure 8b) slices each matrix into its own tokens, fully preserving channel boundaries at the cost of introducing a larger amount of padding. We use dense tokenization by default, since it already preserves the adapter’s structure well enough while avoiding significant padding. For the mixed-rank setting, where rank-1 and rank-2 adapters must share one token sequence, we switch to the sparse tokenizer. Here, each matrix’s token budget is fixed by its rank-2 size, so a rank-1 matrix occupies only the first half of that budget, with the remainder marked as padding by the mask.

Figure 8. (a) Dense tokenization. Only the final token is padded, marked in grey. (b) Sparse tokenization. The last token of each matrix is padded, marked in grey. Dense versus sparse tokenization of a single LoRA adapter.

Architecture. Our primary autoencoder is a GPT-style, non-causal Transformer encoder-decoder following Schürholt et al., [25]. The encoder projects each of the N=312N = 312 input tokens from dimension 320 to a model width of 1024, applies 8 layers with 16 attention heads, and projects down to a latent dimension of 256 per token. The decoder is symmetric.

Training. The autoencoder is trained with a masked, batch-normalized MSE computed only over real-weight positions, for 1000 epochs at batch size 32 with AdamW [17] (learning rate 10−410^{-4}, weight decay 10−410^{-4}) under a one-cycle schedule and a fixed seed of 2020. We use dropout 0.15, full precision, and gradient clipping at a max-norm of 10. The adapter pool used for training is capped at 8,192 adapters and split 70/15/15 into train, validation, and test, giving ∼\sim5,734 training adapters.

Additional Experimental Results

Arithmetic and Interpolation

If the latent is organized, then simple vector operations on latent codes should correspond to meaningful, predictable changes in the decoded identity, which we refer to as “face algebra.” We test this directly by applying arithmetic and interpolation to the latents of real identities and inspecting the decoded results.

Setup. Given two identities encoded to latents zAz_A and zBz_B, we form new latents with a fixed set of elementary operations: the directional differences zA−zBz_A-z_B and zB−zAz_B-z_A, the midpoint (zA+zB)/2(z_A+z_B)/2, the raw sum zA+zBz_A+z_B, the asymmetric combinations zA+zB/2z_A+z_B/2 and zA+2zBz_A+2z_B that weight one identity more heavily, and a linear interpolation zγ=(1−γ)zA+γzBz_\gamma=(1-\gamma)z_A+\gamma z_B between the two identities. For interpolation, we additionally apply the identical construction directly to the raw adapter weights θA,θB\theta_A,\theta_B.

Arithmetic works in the latent. Elementary arithmetic on latent codes translates into visually sensible changes in the decoded identity (Figure 9). For two encoded identities AA and BB, the two directional differences zA−zBz_A-z_B and zB−zAz_B-z_A subtract one identity’s features from the other in the expected direction. The midpoint (zA+zB)/2(z_A+z_B)/2 decodes to a clean blend of the two people, and the weighted combinations tilt the result toward the more heavily weighted identity: zA+zB/2z_A+z_B/2 gives a subtle influence of BB on AA, whereas zA+2zBz_A+2z_B amplifies BB and pushes it into AA. The raw sum zA+zBz_A+z_B overloads the combination, as expected of an operation that leaves the natural range of the latent. The same pattern holds for other identity pairs.

Figure 9. Arithmetic operations on the latents of two identities, xAx_A and xBx_B, decode to visually sensible results, with the unscaled sum zA+zBz_A+z_B as the one exception that overloads the combination.

Interpolation also works in weight space. Linearly interpolating between two encoded identities produces a smooth, gradual transition through valid intermediate identities (Figure 10, top). Features change continuously from AA to BB with no abrupt switch and no invalid faces along the way. Applying the same construction directly to the raw weights (Figure 10, bottom) yields a similarly recognizable transition between the two identities, showing that this operation is preserved even without the autoencoder.

Latent-space and weight-space interpolations between identities $z_A$ and $z_B$, and $\theta_A$ and $\theta_B$

Figure 10. Interpolating between two identities produces a smooth, valid transition whether performed in the latent (top) or directly in weight space (bottom).

Editing

γ=−0.75\gamma= -0.75 γ=−0.5\gamma= -0.5 γ=−0.25\gamma= -0.25 γ=0\gamma= 0 γ=0.25\gamma= 0.25 γ=0.5\gamma= 0.5 γ=0.75\gamma= 0.75

Male (AUC: 0.997)

Eyeglasses (AUC: 0.984)

Gray Hair (AUC: 0.975)

Oval Face (AUC: 0.620)

References

  1. [1]Aron Asefaw, Konstantinos Tzevelekakis, Damian Falk, Léo Meynent, and Damian Borth. Weight-CLIP: Aligning datasets and models for weight space learning. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2607.03551
  2. [2]Amil Dravid, Yossi Gandelsman, Kuan-Chieh Wang, Rameen Abdul, Gordon Wetzstein, Alexei A. Efros, and Kfir Aberman. Interpreting the weight space of customized diffusion models. In Advances in Neural Information Processing Systems, volume 37, pages 137334–137371, 2024.
  3. [3]Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classifying the classifier: dissecting the weight space of neural networks. In European Conference on Artificial Intelligence, 2020.
  4. [5]Rohit Gandikota, Joanna Materzyńska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: LoRA adaptors for precise control in diffusion models. In European Conference on Computer Vision, pages 172–188. Springer, 2024.arxiv.org/abs/2311.12092
  5. [7]Joëlle Hanna, Damian Falk, Stella X. Yu, and Damian Borth. Geosane: Learning geospatial representations from models, not data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27804–27814, June 2026.arxiv.org/abs/2603.23408
  6. [8]Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. GANSpace: Discovering interpretable GAN controls. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2004.02546
  7. [9]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. arXiv:2106.09685.arxiv.org/abs/2106.09685
  8. [10]Xiaolong Jin, Kai Wang, Dongwen Tang, Wangbo Zhao, Yukun Zhou, Junshu Tang, and Yang You. Conditional LoRA parameter generation. arXiv preprint arXiv:2408.01415, 2024.arxiv.org/abs/2408.01415
  9. [11]Boris Knyazev, Doha Hwang, and Simon Lacoste-Julien. Can we scale transformers to predict parameters of diverse ImageNet models? In Proceedings of the International Conference on Machine Learning, 2023.arxiv.org/abs/2303.04143
  10. [12]Miltiadis Miltos Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J Burghouts, Efratios Gavves, Cees G Snoek, and David Zhang. Graph neural networks for learning equivariant representations of neural networks. In International Conference on Learning Representations, 2024.arxiv.org/abs/2403.12143
  11. [13]Shashi Kumar, Yacouba Kaloga, John Mitros, Petr Motlicek, and Ina Kodraši. Latent space factorization in lora. Advances in Neural Information Processing Systems, 38:88111–88138, 2025.arxiv.org/abs/2510.19640
  12. [14]Zhiyuan Liang, Dongwen Tang, Yuhao Zhou, Xuanlei Zhao, Mingjia Shi, Wangbo Zhao, Zekai Li, Peihao Wang, Konstantin Schürholt, Damian Borth, Michael M. Bronstein, Yang You, Zhangyang Wang, and Kai Wang. Drag-and-drop LLMs: Zero-shot prompt-to-weights. In Advances in Neural Information Processing Systems, volume 38, pages 153258–153283, 2025.arxiv.org/abs/2506.16406
  13. [15]Derek Lim, Haggai Maron, Marc T. Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for processing diverse neural architectures. In International Conference on Learning Representations, 2024.arxiv.org/abs/2312.04501
  14. [16]Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3730–3738, 2015.arxiv.org/abs/1411.7766
  15. [17]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  16. [18]Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, pages 25790–25816. PMLR, 2023.arxiv.org/abs/2301.12780
  17. [19]Theo Putterman, Derek Lim, Yoav Gelberg, Michael M. Bronstein, Stefanie Jegelka, and Haggai Maron. GL-equivariant metanetworks for learning on low-rank weight spaces. In Proceedings of the Fourth Learning on Graphs Conference, 2025.
  18. [20]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.arxiv.org/abs/2112.10752
  19. [21]Nathaniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.arxiv.org/abs/2208.12242
  20. [22]Nathaniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. HyperDreamBooth: HyperNetworks for fast personalization of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.arxiv.org/abs/2307.06949
  21. [23]Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. In Advances in Neural Information Processing Systems, volume 34, pages 16481–16493, 2021.
  22. [24]Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper-representations as generative models: Sampling unseen neural network weights. In Advances in Neural Information Processing Systems, volume 35, pages 27906–27920, 2022.
  23. [25]Konstantin Schürholt, Michael W Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of PMLR, pages 43947–43966, 2024.arxiv.org/abs/2406.09997
  24. [26]Viraj Shah, Nathaniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. ZipLoRA: Any subject in any style by effectively merging LoRAs. In European Conference on Computer Vision, pages 422–438. Springer, 2024.arxiv.org/abs/2311.13600
  25. [27]Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in GANs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.arxiv.org/abs/2007.06600
  26. [28]Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of GANs for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.arxiv.org/abs/1907.10786
  27. [29]Bedionita Soro, Bruno Andreis, Hayeon Lee, Wonyong Jeong, Song Chong, Frank Hutter, and Sung Ju Hwang. Diffusion-based neural network weights generation. In International Conference on Learning Representations, 2025.arxiv.org/abs/2402.18153
  28. [30]Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint arXiv:2002.11448, 2020.arxiv.org/abs/2002.11448
  29. [31]Kai Wang, Dongwen Tang, Boya Zeng, Yida Yin, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion. arXiv preprint arXiv:2402.13144, 2024.arxiv.org/abs/2402.13144
  30. [32]Kai Wang, Dongwen Tang, Wangbo Zhao, Konstantin Schürholt, Zhangyang Wang, and Yang You. Scaling up parameter generation: A recurrent diffusion approach. In Advances in Neural Information Processing Systems, volume 38, pages 103214–103246, 2025.DOI
  31. [33]Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
  32. [34]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF conference on computer vision and pattern recognition, pages 586–595. IEEE, 2018.arxiv.org/abs/1801.03924
  33. [35]Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in neural information processing systems, 36:24966–24992, 2023.arxiv.org/abs/2302.14040

Paper details

vlnc:W_cj4arm9b
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 14

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents