betaFeedback

WorksComputer Science

GeoNaviDiff: Transferring Disentangled Semantic Controls to Diffusion Features via Local Riemannian Navigation

Abstract

Disentangled real-image manipulation requires factor-specific semantic control while preserving image fidelity and non-target content. StyleSpace SS provides interpretable controls but is limited by GAN inversion, whereas diffusion models offer stronger image realization without an explicit factorized control basis. We introduce GeoNaviDiff, an unsupervised cross-space framework that transfers disentangled StyleSpace interventions to local Riemannian trajectories in diffusion hh-space. GeoNaviDiff constructs reconstruction-decoupled supervision by extracting a spherical Log-map tangent from a matched StyleGAN reconstruction-edit pair and parallel-transporting it to the real-image diffusion state. A shared, factor-conditioned Navigator predicts this tangent at inference, while norm-preserving geodesic realization and source-anchored rollout stabilize the reverse process without edited references or factor-specific models. Experiments across multiple datasets show that GeoNaviDiff outperforms the compared methods in fidelity, factor selectivity, edit locality, and trajectory smoothness.

Introduction

Disentangled Representation Learning (DRL) [1] seeks to factorize complex observations into semantically meaningful variables, providing a foundation for interpretable and controllable generation. For representation manipulation, its value lies in selective intervention: a target factor, such as shape, color, or expression, should change while identity, structure, and other non-target factors remain stable. This factor-specific controllability is essential for fine-grained editing, yet it depends on both an interpretable representation and a generative model capable of faithfully realizing the prescribed change.

Early DRL-based manipulation methods primarily relied on the structured latent spaces of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) [8, 14]. In particular, StyleGAN exposes increasingly fine-grained semantic controls through its latent hierarchy, with channel-level coordinates in StyleSpace SS supporting localized and interpretable interventions [15] (Wu, Lischinski, and Shechtman 2021). However, GAN-based real-image manipulation remains constrained by inversion fidelity and the fixed generator prior [27, 2, 19]. Diffusion models offer a stronger generative prior and high-quality image reconstruction, but their internal representations do not explicitly expose a factor-indexed semantic basis. This creates a fundamental tension: structured GAN representations provide factor-specific semantic controls, whereas diffusion priors favor high-fidelity reconstruction and synthesis.

To exploit the diffusion prior despite this missing factorized control space, existing conditional diffusion manipulation methods introduce semantic controls externally through text, attribute labels, or auxiliary guidance. Their controllable granularity, however, remains tied to predefined conditions and may cause unintended changes when those conditions are coarse or misaligned [20, 26] (Hu et al. 2023). An intrinsic alternative is to manipulate the U-Net bottleneck feature space, known as hh-space, which encodes rich high-level semantics without such external controls [18, 11]. Yet hh-space does not expose these semantics as explicit, reusable control coordinates. Therefore, prior work trains a separate shift predictor for each prescribed semantic attribute and injects its output as an additive Euclidean residual [18]. This direction-specific design prevents reuse across factors and ignores the geometry and feature statistics of hh-space. As illustrated in Figure 1, its straight ambient path departs from the spherical shell of normalized bottleneck states and changes the bottleneck norm, disrupting its correlation with skip-feature statistics and producing visible artifacts [11]. Remaining on the hyperspherical shell alone is insufficient: an arbitrary geodesic may be geometrically valid while reaching a semantically incorrect endpoint. Therefore, stable disentangled manipulation requires a local tangent direction conditioned on an explicit semantic factor and adapted to the source image and diffusion timestep.

Conceptual comparison of Euclidean, arbitrary geodesic, and GeoNaviDiff trajectories in diffusion h-space

Figure 1. Conceptual comparison of Euclidean, arbitrary geodesic, and GeoNaviDiff trajectories in diffusion hh-space.

Motivated by this insight, we propose GeoNaviDiff, an unsupervised DRL framework for cross-space semantic transfer via Geodesic Navigation in Diffusion hh-space. Rather than disentangling diffusion hh-space from scratch, GeoNaviDiff transfers unsupervised SS-space controls into it, enabling high-fidelity manipulation of individual semantic factors while preserving non-target content. A shared cross-space Navigator conditions on the SS-space displacement, source bottleneck state, and timestep to predict the corresponding local tangent direction in hh-space. Geometry-aware learning and source-anchored inference then realize this prediction as a stable manifold-preserving diffusion trajectory without requiring an edited reference or a separate Navigator for each factor at inference. Our main contributions are:

  1. We introduce a cross-space paradigm that maps interpretable SS-space directions to stable semantic trajectories along the local hyperspherical geometry of diffusion hh-space. This paradigm decouples semantic representation from image realization, with SS-space defining the semantics and the diffusion prior realizing them beyond GAN reconstruction fidelity.

  1. We develop a local Riemannian navigation mechanism with reconstruction-decoupled tangent supervision, norm-preserving geodesic realization, and source-anchored inference. This design aligns the transferred semantic direction with the real-image diffusion state while suppressing radial drift and cumulative rollout error.

  1. We conduct extensive evaluations across multiple datasets and diverse object categories. The results demonstrate that GeoNaviDiff generalizes robustly across object categories and achieves state-of-the-art performance across all reported metrics of image fidelity, representation disentanglement, edit locality, and trajectory smoothness.

Related Work

Disentanglement in VAEs and GANs

Early disentangled representation learning primarily relied on variational autoencoders (VAEs), which separate generative factors by regularizing the latent distribution or encouraging factorized representations [3, 16, 22]. GAN-based approaches subsequently showed that interpretable semantic directions can be identified in pretrained generators through principal components, closed-form factorization, or unsupervised direction discovery [8, 23, 28]. Moving beyond the dense directions of W\mathcal{W} and W+\mathcal{W}^{+}, StyleSpace Analysis exposed the channel-wise modulation coefficients of StyleGAN and showed that selected coordinates provide localized controls with stronger disentanglement and completeness under its evaluation protocols [15] (Wu, Lischinski, and Shechtman 2021). Fantastic Style Channels further selects representative and diverse channel directions from their generated image changes without using attribute labels [24]. These advances provide fine-grained semantic controls, but realizing them on real images remains constrained by GAN inversion and the fixed generator prior [31, 27].

Disentanglement–Fidelity Trade-off

The tension between semantic editability and image fidelity is particularly evident in real-image manipulation. In StyleGAN, latent codes that remain close to the native distribution are generally more editable, whereas representations with greater freedom can reconstruct finer details at the cost of less stable manipulation; e4e, CLCAE, and SFE address different aspects of this trade-off through editability-aware inversion, cross-space guidance, or feature editing [27, 19, 2]. Diffusion-based DRL methods approach the complementary side of the problem by learning factorized representations while retaining the generative capacity of diffusion models: InfoDiffusion maximizes information in an auxiliary semantic latent variable, DisDiff learns factor-conditioned representations without supervision, and EncDiff uses cross-attention as an inductive bias for factor separation [29, 33, 32]. These methods require training or modifying the diffusion model to learn a disentangled representation, rather than reusing an established semantic basis with a frozen denoiser. Inspired by these complementary advances, GeoNaviDiff uses SS-space to provide factor-specific semantic controls and the high-dimensional hh-space of a pretrained U-Net denoiser as the realization space for their semantic shifts. This cross-space formulation combines the disentanglement of SS with diffusion reconstruction fidelity without explicit conditioning or predefined hh-space directions.

Semantic Manipulation in Diffusion Models

Diffusion models can be controlled through text, attribute labels, or auxiliary guidance, but the resulting editing granularity remains tied to the conditioning interface and may cause unintended changes when the condition is coarse or misaligned [20, 26] (Hu et al. 2023). Intrinsic feature manipulation provides an alternative: Asyrp identifies the U-Net bottleneck hh-space as semantically editable and learns a direction-specific predictor whose output is injected as an additive Euclidean residual [18], while subsequent work discovers semantic directions across diffusion timesteps without attribute supervision [21]. InjectFusion further demonstrates that direct replacement or addition can disrupt bottleneck feature statistics, and improves content injection through norm-matched spherical interpolation and latent calibration ([11]). These approaches establish the editability of diffusion features and the importance of preserving their geometry, but do not provide a shared factor-conditioned mechanism that maps an external disentangled control to a source- and timestep-adaptive tangent trajectory. GeoNaviDiff addresses this gap by learning a shared Navigator from SS-space displacements to local Riemannian navigation in diffusion hh-space.

Background

DDIM Sampler

The Denoising Implicit Diffusion Model (DDIM) sampler ([25]) is a deterministic sampler that maintains consistency between forward and reverse process trajectories. This property enables it to map Gaussian noise x^T∼N(0,I)\hat{x}_T \sim\mathcal{N}(0,I) back to generated images via smooth denoising trajectories:

x^t−1=αˉt−1(x^t−1−αˉtαˉtϵθ(x^t,t,c))+1−αˉt−1−σt2ϵθ(x^t,t,c)+σtϵ~t,(1)\hat{x}_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\left(\hat{x}_t - \frac{\sqrt{1-\bar{\alpha}_t}}{\sqrt{\bar{\alpha}_t}}\epsilon_\theta(\hat{x}_t,t,c)\right) + \sqrt{1-\bar{\alpha}_{t-1}-\sigma_t^2}\epsilon_\theta(\hat{x}_t,t,c) + \sigma_t\tilde{\epsilon}_t, \tag*{(1)}

where σt=η1−αˉt−11−αˉt1−αˉtαˉt−1\sigma_t = \eta\sqrt{\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t}}\sqrt{1-\frac{\bar{\alpha}_t}{\bar{\alpha}_{t-1}}}, η\eta is a hyperparameter that controls the stochasticity of the sampling process. In the deterministic form (e.g., η=0\eta=0), DDIM is expected to recover the same noise across all sampling steps as in the corresponding denoising steps, thereby achieving consistent inversion of the diffusion process.

Asymmetric Reverse Process

Asymmetric Reverse Process (Asyrp) ([18]) enables image editing by injecting offset priors into the semantically editable hh-space during the denoising process. Let hth_t be the original hh latent matrix corresponding to timestep tt. From the unconditional form of Eq. (1):

Pt(x~t,t;ht)=x~t−1−αˉt⋅ϵθ(x~t,t;ht)αˉt,(2)P_t(\tilde{x}_t,t;h_t)=\frac{\tilde{x}_t-\sqrt{1-\bar{\alpha}_t}\cdot\epsilon_\theta(\tilde{x}_t,t;h_t)}{\sqrt{\bar{\alpha}_t}}, \tag*{(2)}
Dt(x~t,t;ht)=1−αˉt−1−σt2⋅ϵθ(x~t,t;ht).D_t(\tilde{x}_t,t;h_t)=\sqrt{1-\bar{\alpha}_{t-1}-\sigma_t^2}\cdot\epsilon_\theta(\tilde{x}_t,t;h_t).

Asyrp uses a simulated diffusion process via DDIM inversion to obtain the semantically manipulated h~t\tilde{h}_t and defines the following three-stage editing process. For compactness, let Pt(h)=Pt(x~t,t;h)P_t(h)=P_t(\tilde{x}_t,t;h), Dt(h)=Dt(x~t,t;h)D_t(h)=D_t(\tilde{x}_t,t;h), and denote the three timestep intervals by I1=[tedit,T]I_1=[t_{\mathrm{edit}},T], I2=[tboost,tedit)I_2=[t_{\mathrm{boost}},t_{\mathrm{edit}}), and I3=[0,tboost)I_3=[0,t_{\mathrm{boost}}):

x~t−1={αˉt−1Pt(h~t)+Dt(ht),t∈I1,αˉt−1Pt(ht)+Dt(ht),t∈I2,αˉt−1Pt(ht)+Dt(ht)+σtϵ~t,t∈I3.(3)\tilde{x}_{t-1} = \begin{cases} \sqrt{\bar{\alpha}_{t-1}}P_t(\tilde{h}_t)+D_t(h_t), & t\in I_1,\\ \sqrt{\bar{\alpha}_{t-1}}P_t(h_t)+D_t(h_t), & t\in I_2,\\ \sqrt{\bar{\alpha}_{t-1}}P_t(h_t)+D_t(h_t)+\sigma_t\tilde{\epsilon}_t, & t\in I_3. \end{cases} \tag*{(3)}

The editing process is performed within the interval T≥t≥teditT\ge t\ge t_{\mathrm{edit}}. It injects the modified h~t\tilde{h}_t only into Pt(⋅)P_t(\cdot) to prevent the injected semantics from being canceled out by Dt(⋅)D_t(\cdot).

Methodology

Overview

As shown in Figure 2, GeoNaviDiff transfers unsupervised, factor-specific SS-space controls into diffusion hh-space to enable high-fidelity, disentangled manipulation of real images. Given a source image and an intervention Δs\Delta s, the StyleGAN branch constructs a matched reconstruction–edited reference pair, and a frozen U-Net extracts bottleneck states for the source and both references under shared noising. The Riemannian displacement between the reference states defines a semantic tangent, which is parallel-transported to the real-image source state to form the Navigator target. The Navigator predicts this target from (Δs,t,ht(N))(\Delta s,t,h_t^{(N)}); norm-preserving geodesic realization and source-anchored inference then turn the prediction into a stable diffusion trajectory.

Figure 2. Training framework of GeoNaviDiff. (a) Unsupervised semantic discovery and disentangled control construction in SS-space, followed by cross-space navigation into diffusion hh-space. (b) The four-step parallel-transport procedure is used to construct the tangent navigation target.

Disentangled Semantic Basis in SS-space

GeoNaviDiff uses StyleGAN2’s StyleSpace SS as its factorized semantic basis. Given an inverted code w+={wℓ}ℓ=1Lw^+=\{w^\ell\}_{\ell=1}^{L}, the layer-wise affine transforms map w+={wℓ}ℓ=1Lw^+=\{w^\ell\}_{\ell=1}^{L} to s=(A1(w1),…,AL(wL))∈S=∏ℓ=1LRCℓs=(A_1(w_1),\ldots,A_L(w_L))\in S=\prod_{\ell=1}^{L}\mathbb{R}^{C_\ell}, where each coordinate modulates a synthesis channel ([15] (Wu, Lischinski, and Shechtman 2021)). Selected coordinates provide localized controls; the resulting displacement Δs\Delta s both conditions the Navigator and generates the matched reconstruction–edit pair used for tangent supervision.

Since SS-space contains thousands of channels with redundant effects, we adopt the unsupervised Fantastic Style Channels method ([24]) to discover representative and diverse candidate directions. It evaluates axis-aligned channel perturbations through their generated image changes and groups channels with similar effects into candidate clusters. As shown in Figure 2(a), a pretrained vision-language model (VLM) filters these clusters by inspecting their traversal sequences to retain semantically meaningful directions; the same filtering can instead be performed manually. Let I={ik}k=1K\mathcal{I}=\{i_k\}_{k=1}^{K} denote the retained Fantastic Style Channels indices and dk=eikd_k=e_{i_k}, the corresponding coordinate basis vector. During Navigator training, a sampled direction and strength specify Δs=αdk\Delta s=\alpha d_k and s′=s+Δss'=s+\Delta s, where the sign and magnitude of α\alpha determine the traversal direction and extent, respectively.

For a real image xx, a pretrained and frozen e4e inversion encoder ([27]) produces w+w^+, and the fixed affine transforms above yield ss. The frozen StyleGAN generator then produces the matched pair X(R)=G(s)X^{(R)}=G(s) and X(V)=G(s+αdk)X^{(V)}=G(s+\alpha d_k). This pair shares the same inverted code and generator pathway and differs only in the prescribed SS-space intervention. It therefore externalizes the factor-specific change encoded by Δs\Delta s as a matched visual reference for constructing the diffusion-space navigation target in § 4.3. In GeoNaviDiff, SS thus supplies both the factor-indexed control descriptor and its visual supervision.

Cross-Space Riemannian Navigation

Existing hh-space editors commonly inject an ambient Euclidean residual, h~t=ht+λΔht\tilde{h}_t=h_t+\lambda\Delta h_t ([18], ignoring bottleneck geometry and feature statistics:

∥h~t∥F2=∥ht∥F2+2λ⟨ht,Δht⟩F+λ2∥Δht∥F2(4)\lVert\widetilde{h}_t \rVert_F^2 = \lVert h_t \rVert_F^2 + 2\lambda\langle h_t, \Delta h_t \rangle_F + \lambda^2 \lVert\Delta h_t \rVert_F^2 \tag*{(4)}

so Euclidean injection incurs radial leakage and, even for a tangent residual satisfying ⟨ht,Δht⟩F=0\langle h_t, \Delta h_t \rangle_F = 0, increases the squared norm by λ2∥Δht∥F2\lambda^2 \lVert\Delta h_t \rVert_F^2. This is consequential because bottleneck and skip-feature norms are correlated, and disrupting this relation produces visible artifacts [11]. Moreover, the linear interpolation between two distinct normalized states passes through the interior of the sphere rather than following its surface, producing an off-shell trajectory.

To the best of our knowledge, GeoNaviDiff is the first framework to perform local Riemannian navigation in diffusion hh-space conditioned on disentangled SS-space factors. It replaces this geometry-agnostic residual with cross-space navigation from SS-space to diffusion hh-space. Here, each factor is indexed by an unsupervised SS-space direction dkd_k, and its signed magnitude is specified by Δs=αdk\Delta s = \alpha d_k. Rather than claiming that the complete hh-space is globally spherical, we locally model normalized bottleneck states at each timestep on the unit hypersphere Sd−1S^{d-1}. The resulting navigation target is sample-dependent and timestep-dependent, and explicitly conditioned on Δs\Delta s.

Matched semantic triple. As described in § 4.2, we construct an unedited StyleGAN reconstruction X(R)X^{(R)} and its edited reference X(V)X^{(V)} for each real source image X(N)X^{(N)}. We apply the forward noising process to all three images using the same sampled timestep tt and noise realization ϵ\epsilon:

{Xt(N)=αˉtX(N)+1−αˉtϵ,Xt(R)=αˉtX(R)+1−αˉtϵ,Xt(V)=αˉtX(V)+1−αˉtϵ.(5)\begin{cases} X_t^{(N)} = \sqrt{\bar{\alpha}_t}X^{(N)} + \sqrt{1-\bar{\alpha}_t}\epsilon, \\ X_t^{(R)} = \sqrt{\bar{\alpha}_t}X^{(R)} + \sqrt{1-\bar{\alpha}_t}\epsilon, \\ X_t^{(V)} = \sqrt{\bar{\alpha}_t}X^{(V)} + \sqrt{1-\bar{\alpha}_t}\epsilon. \tag*{(5)} \end{cases}

Shared noising removes stochastic differences from the comparison. More importantly, X(R)X^{(R)} and X(V)X^{(V)} share the same inversion and synthesis path and differ only by Δs\Delta s; their difference therefore isolates the prescribed semantic intervention, whereas X(N)X^{(N)} supplies the real-image state encountered by the Navigator at deployment. With the frozen U-Net bottleneck extractor HψH_{\psi}, we obtain

ht(i)=Hψ(Xt(i),t),i∈{N,R,V}.(6)h_t^{(i)} = H_{\psi}(X_t^{(i)}, t), \qquad i \in\{N,R,V\}. \tag*{(6)}

Together, these three bottleneck states separate the StyleGAN semantic change from the real-to-reconstruction gap before transferring it into diffusion hh-space.

Spherical normalization and semantic Log map.

Let ri=∥ht(i)∥Fr_i = \lVert h_t^{(i)} \rVert_F. We normalize each bottleneck state as

pt(i)=ht(i)ri∈Sd−1,i∈{N,R,V}.(7)p_t^{(i)} = \frac{h_t^{(i)}}{r_i} \in S^{d-1}, \qquad i \in\{N,R,V\}. \tag*{(7)}

We first measure the semantic edit entirely within the matched reconstruction domain. Define

cRV=⟨pt(R),pt(V)⟩F,sRV=1−cRV2,c_{RV} = \langle p_t^{(R)}, p_t^{(V)} \rangle_F, \qquad s_{RV} = \sqrt{1-c_{RV}^2},
θRV=atan2⁡(sRV,cRV),eR=pt(V)−cRVpt(R)sRV.(8)\theta_{RV} = \operatorname{atan2}(s_{RV},c_{RV}), \qquad e_R = \frac{p_t^{(V)} - c_{RV}p_t^{(R)}}{s_{RV}}. \tag*{(8)}

Here eRe_R is the unit direction of the shortest great-circle arc from pt(R)p_t^{(R)} to pt(V)p_t^{(V)}, and θRV\theta_{RV} is its exact geodesic angle rather than an arbitrary Euclidean residual norm. The logarithmic map is therefore

ξR=Log⁡pt(R)(pt(V))=θRVeR∈Tpt(R)Sd−1.(9)\xi_R = \operatorname{Log}_{p_t^{(R)}}(p_t^{(V)}) = \theta_{RV}e_R \in T_{p_t^{(R)}}\mathbb{S}^{d-1}. \tag*{(9)}

When pt(V)=pt(R)p_t^{(V)} = p_t^{(R)}, we define ξR=0\xi_R = 0 by the continuous limit of Eq. (9); the local S\mathcal{S}-space interventions considered here exclude the antipodal case. By construction, ⟨ξR,pt(R)⟩F=0\langle\xi_R,p_t^{(R)}\rangle_F = 0 and ∥ξR∥F=θRV\|\xi_R\|_F = \theta_{RV}. Thus, the direction of ξR\xi_R specifies the local semantic trajectory and its norm has the explicit meaning of geodesic arc length. This avoids the ambiguity of treating an ambient residual magnitude as semantic strength.

Parallel transport to the real-image anchor. Although ξR\xi_R isolates the edit from the reconstruction gap, it belongs to the tangent space at the reconstructed reference pt(R)p_t^{(R)}, while the Navigator should act at the real source point pt(N)p_t^{(N)}. Directly reusing ξR\xi_R at pt(N)p_t^{(N)} is unreasonable because ⟨ξR,pt(N)⟩F≠0\langle\xi_R,p_t^{(N)}\rangle_F \ne0. We therefore parallel-transport ξR\xi_R along the shortest geodesic from pt(R)p_t^{(R)} to the real-image anchor pt(N)p_t^{(N)} and denote the resulting tangent vector by ξt∗\xi_t^*. On the unit sphere, it has the closed form

ξt∗=ξR−⟨ξR,pt(N)⟩F1+⟨pt(R),pt(N)⟩F(pt(R)+pt(N)).(10)\xi_t^* = \xi_R - \frac{\langle\xi_R,p_t^{(N)}\rangle_F}{1+\langle p_t^{(R)},p_t^{(N)}\rangle_F}(p_t^{(R)}+p_t^{(N)}). \tag*{(10)}

Because the locally paired source and reconstruction states are non-antipodal, the denominator in Eq. (10) is nonzero. The transported vector satisfies ⟨ξt∗,pt(N)⟩F=0\langle\xi_t^*,p_t^{(N)}\rangle_F = 0 and ∥ξt∗∥F=∥ξR∥F=θRV\|\xi_t^*\|_F = \|\xi_R\|_F = \theta_{RV}. A target defined directly at the source, Log⁡pt(N)(pt(V))\operatorname{Log}_{p_t^{(N)}}(p_t^{(V)}), would mix the intended semantic displacement with the StyleGAN reconstruction gap: even when Δs=0\Delta s = 0 and pt(V)=pt(R)p_t^{(V)} = p_t^{(R)}, it remains nonzero because pt(R)≠pt(N)p_t^{(R)} \ne p_t^{(N)}. In contrast, our target first isolates the edit as Log⁡pt(R)(pt(V))\operatorname{Log}_{p_t^{(R)}}(p_t^{(V)}) and then transports it to pt(N)p_t^{(N)}. Consequently, Δs=0⇒ξt∗=0\Delta s = 0 \Rightarrow\xi_t^* = 0, while nonzero edits remain tangent at the real-image state and retain their semantic geodesic length.

Navigator Learning and Stable Inference

Tangent-field learning. The Cross-space Navigator learns a shared conditional mapping to the geometric target constructed above. Conditioned on the factorized S\mathcal{S}-space displacement, timestep, and unedited source feature, it predicts ut=M(Δs,t,ht(N))u_t = \mathcal{M}(\Delta s,t,h_t^{(N)}). The Navigator learns a shared, factor-conditioned tangent field in diffusion hh-space without requiring an edited endpoint at inference. Whereas Asyrp learns a separate additive residual predictor for each prescribed semantic direction (Kwon, Jeong, and Uh [18]), M\mathcal{M} is shared across S\mathcal{S}-space factors and directly predicts the required local trajectory from Δs\Delta s. At inference, it requires neither an edited reference nor a direction-specific model.

The target ξt∗\xi_t^* is strictly tangent at pt(N)p_t^{(N)}, whereas the raw network output need not be. We train only M\mathcal{M} while keeping the inversion encoder, StyleGAN generator, and diffusion U-Net frozen:

Ltan=Et∼U{tedit,…,T}, ϵ∼N(0,I)[∥ut−ξt∗∥F2].(11)\mathcal{L}_{\mathrm{tan}} = \mathbb{E}_{t\sim\mathcal{U}\{t_{\mathrm{edit}},\ldots,T\},\ \epsilon\sim\mathcal{N}(0,I)}\left[\left\|u_t-\xi_t^*\right\|_F^2\right]. \tag*{(11)}

Sampling procedures for the semantic factor kk and intervention strength α\alpha are detailed in Supplementary Appendix A. Because ξt∗⊥pt(N)\xi_t^* \perp p_t^{(N)}, this loss admits the orthogonal decomposition

∥ut−ξt∗∥F2=∥Πpt(N)(ut)−ξt∗∥F2+⟨ut,pt(N)⟩F2,(12)\left\|u_t-\xi_t^*\right\|_F^2 = \left\|\Pi_{p_t^{(N)}}(u_t)-\xi_t^*\right\|_F^2+\langle u_t,p_t^{(N)}\rangle_F^2, \tag*{(12)}

where

Πpt(N)(ut)=ut−⟨ut,pt(N)⟩Fpt(N).(13)\Pi_{p_t^{(N)}}(u_t)=u_t-\langle u_t,p_t^{(N)}\rangle_Fp_t^{(N)}. \tag*{(13)}

Hence, the objective aligns the semantic tangent and penalizes radial leakage. At inference, we explicitly apply Eq. (13) and denote ξ^t=Πpt(N)(ut)\hat{\xi}_t=\Pi_{p_t^{(N)}}(u_t), ensuring that the final displacement remains tangent despite Navigator errors.

Geodesic realization. We realize the predicted tangent through the exponential map:

p~t=cos⁡(∥ξ^t∥F)pt(N)+sin⁡(∥ξ^t∥F)ξ^t∥ξ^t∥F.(14)\widetilde{p}_t=\cos(\|\hat{\xi}_t\|_F)p_t^{(N)}+\sin(\|\hat{\xi}_t\|_F)\frac{\hat{\xi}_t}{\|\hat{\xi}_t\|_F}. \tag*{(14)}

We then restore the source radius as h~t=rp~t\widetilde{h}_t=r\widetilde{p}_t, thereby preserving the source bottleneck scale and its correlation with the source skip-feature statistics and avoiding the norm-induced artifacts of unconstrained residual injection [11]. When ξ^t=0\hat{\xi}_t=0, as expected for the identity edit Δs=0\Delta s=0, the normalized direction in Eq. (14) is undefined. We therefore use the continuous limit Exp⁡pt(N)(0)=pt(N)\operatorname{Exp}_{p_t^{(N)}}(0)=p_t^{(N)}, so the bottleneck remains unchanged. For ξ^t≠0\hat{\xi}_t\ne0, ξ^t/∥ξ^t∥F\hat{\xi}_t/\|\hat{\xi}_t\|_F is a unit tangent orthogonal to pt(N)p_t^{(N)}. Together, these cases ensure ∥p~t∥F=1\|\widetilde{p}_t\|_F=1, ∥h~t∥F=∥ht(N)∥F\|\widetilde{h}_t\|_F=\|h_t^{(N)}\|_F, and dS(pt(N),p~t)=∥ξ^t∥Fd_{\mathcal{S}}(p_t^{(N)},\widetilde{p}_t)=\|\hat{\xi}_t\|_F. Therefore, GeoNaviDiff neither introduces radial norm drift nor follows a straight chord through the shell interior. No additional geometric scaling is introduced: Δs=αdk\Delta s=\alpha d_k specifies the requested semantic factor and strength, while M\mathcal{M} adapts the corresponding tangent displacement to the source state and timestep.

Source-anchored rollout. During training, we sample timesteps independently and always condition M\mathcal{M} on an unedited source feature. During a reverse process, however, recursively querying M\mathcal{M} with the edited feature would expose it to unseen states. Earlier prediction errors would alter subsequent queries, and repeated residuals would cause semantic, identity, and structural drift, as verified by the ablation in Sec. 5.4. To remove this train–inference mismatch, GeoNaviDiff uses a source-anchored rollout (Alg. 1, Appendix B). After DDIM inversion, we evolve an unedited source trajectory {Xt(src)}\{X_t^{(\mathrm{src})}\} in parallel with the edited trajectory {Xt(edit)}\{X_t^{(\mathrm{edit})}\}. At each step, the source branch extracts ht(src)=Hψ(Xt(src),t)h_t^{(\mathrm{src})}=\mathcal{H}_{\psi}(X_t^{(\mathrm{src})},t) and normalizes it as pt(src)=ht(src)/∥ht(src)∥Fp_t^{(\mathrm{src})}=h_t^{(\mathrm{src})}/\|h_t^{(\mathrm{src})}\|_F. It then queries ut(src)=M(Δs,t,ht(src))u_t^{(\mathrm{src})}=\mathcal{M}(\Delta s,t,h_t^{(\mathrm{src})}) and constructs the absolute endpoint

h~t(edit∣src)=∥ht(src)∥FExp⁡pt(src)(Πpt(src)(ut(src))).(15)\widetilde{h}_t^{(\mathrm{edit}|\mathrm{src})}=\|h_t^{(\mathrm{src})}\|_F\operatorname{Exp}_{p_t^{(\mathrm{src})}}\left(\Pi_{p_t^{(\mathrm{src})}}(u_t^{(\mathrm{src})})\right). \tag*{(15)}
MethodFFHQAFHQ-v2Stanford Cars
FID↓PPL↓OFlow↓DCI↑FID↓PPL↓OFlow↓DCI↑FID↓PPL↓OFlow↓DCI↑
β\beta-TCVAE (Chen et al. [4])96.50±2.4096.50\pm2.40128.85128.850.120.120.280.2872.00±2.1772.00\pm2.17132.69132.690.130.130.320.3292.78±2.3592.78\pm2.35133.54133.540.130.130.310.31
FactorVAE (Kim and Mnih [16])104.80±2.88104.80\pm2.88118.37118.370.110.110.360.3684.50±2.6084.50\pm2.60122.56122.560.140.140.350.35115.48±3.52115.48\pm3.52120.78120.780.120.120.400.40
e4e (Tov et al. [27])75.74±1.4975.74\pm1.49130.45130.450.300.300.360.3626.65±0.5526.65\pm0.55142.69142.690.230.230.310.3184.08±1.9384.08\pm1.93147.28147.280.340.340.420.42
SFE (Bobkov et al. [2])76.42±1.3576.42\pm1.35114.88114.880.170.170.540.5431.84±1.1231.84\pm1.12125.71125.710.190.190.490.4979.21±1.7079.21\pm1.70122.75122.750.270.270.470.47
NaviGAN (Cherepkov, Voynov, and Babenko [5])76.31±1.2476.31\pm1.2494.3394.330.100.100.600.6049.36±1.0949.36\pm1.0988.9588.950.120.120.570.5753.60±0.9253.60\pm0.9293.5793.570.130.130.620.62
Asyrp† (Kwon, Jeong, and Uh [18])53.91±0.9553.91\pm0.95–0.180.18–20.90±2.4220.90\pm2.42–0.150.15–28.52±1.6828.52\pm1.68–0.190.19–
EncDiff (Yang et al. [32])168.9±1.23168.9\pm1.23145.67145.670.170.170.610.6191.2±1.0891.2\pm1.08135.64135.640.180.180.530.53168.31±1.25168.31\pm1.25178.95178.950.210.210.420.42
InfoDiffusion (Wang et al. [29])154.1±0.87154.1\pm0.87123.59123.590.160.160.590.5987.5±1.3587.5\pm1.35138.30138.300.140.140.480.48144.9±0.89144.9\pm0.89158.14158.140.170.170.600.60
PreciseControl (Parihar et al. [20])52.64±0.8852.64\pm0.88113.56113.560.110.110.480.4821.72±0.6021.72\pm0.60140.66140.660.120.120.510.5124.60±0.6424.60\pm0.64122.31122.310.130.130.430.43
DisDiff (Yang et al. [33])160.31±0.87160.31\pm0.87145.92145.920.170.170.560.56172.24±0.71172.24\pm0.71145.32145.320.150.150.430.43150.38±0.75150.38\pm0.75175.72175.720.200.200.490.49
GeoNaviDiff (Ours)26.78±0.3926.78\pm0.3942.3642.360.080.080.760.7613.66±0.2513.66\pm0.2568.5568.550.070.070.730.7318.92±0.3818.92\pm0.3871.3971.390.060.060.690.69

Table 1. Comparison with existing methods across VAE-, GAN-, and diffusion-based generative models. † denotes our baseline, bold indicates the best result, and “–” denotes not measured. Asyrp lacks fixed latent codes for PPL and DCI.

We inject the source-anchored endpoint into the decoder processing Xt(edit)X^{(\mathrm{edit})}_{t} and apply InjectFusion’s latent calibration (Jeong, Kwon, and Uh [11]) to preserve bottleneck–state compatibility, yielding Xt−1(edit)X^{(\mathrm{edit})}_{t-1}. This update avoids recursive residual accumulation on edited features, limiting train–inference mismatch and cumulative navigation error.

Experiments

Experimental Setup

Datasets. We evaluate GeoNaviDiff on FFHQ (Karras, Laine, and Aila [14]), CelebA-HQ (Karras et al. [13]),

AFHQ-v2 (Choi et al. [6]), and Stanford Cars (Krause et al. [17]), spanning human faces, animal faces, and vehicles. Details on preprocessing, data splits, hardware, and computational cost are provided in Supplementary Appendix G.

Metrics. For all four datasets, we report FID (Heusel et al. [9]) for distributional fidelity, Perceptual Path Length (PPL) (Karras, Laine, and Aila [14]) for perceptual smoothness along semantic traversals, OFlow (Jin et al. [12]) for spatial editing leakage, and the disentanglement component of DCI (Eastwood and Williams [7]) to measure how selectively individual representation dimensions encode semantic factors. The PPL computation protocol and dataset-specific DCI factor definitions are provided in Supplementary Appendix A. Lower FID, PPL, and OFlow are better, whereas higher DCI is better.

Comparison with Existing Methods

Table 1 compares GeoNaviDiff with representative VAE-, GAN-, and diffusion-based methods, and Figure 3 provides a visual comparison on smile manipulation. GeoNaviDiff achieves the best results on all reported metrics across the evaluated datasets: lower FID and PPL indicate better image fidelity and smoother semantic traversals, while lower OFlow and higher DCI show reduced non-target leakage and stronger factor-wise disentanglement. The visual comparison is consistent with these trends, where GeoNaviDiff produces more monotonic semantic changes while better preserving identity and unrelated facial attributes. These improvements support our central design: SS supplies factor-selective semantic controls, while diffusion hh-space provides the high-fidelity realization space; Riemannian navigation and source-anchored inference further reduce feature-geometry mismatch and accumulated non-target drift. More qualitative comparisons are provided in Supplementary Appendix E.

Qualitative comparison on smile manipulation across traversal levels

Figure 3. Qualitative comparison on smile manipulation across traversal levels.

Qualitative results and Source-Space Analysis

Figure 4 shows GeoNaviDiff traversals across FFHQ, CelebA-HQ, AFHQ-v2, and Stanford Cars, covering both localized and globally coordinated semantic factors. Across traversal levels, GeoNaviDiff produces smooth semantic changes while preserving identity, structure, and unrelated content, indicating that the controls discovered in SS remain interpretable after being transferred to diffusion hh-space. Additional GeoNaviDiff traversals across datasets and semantic factors are provided in Supplementary Appendix F. A quantitative comparison of semantic source spaces and transformation strategies is provided in Supplementary Appendix C.

Figure 4. Qualitative traversals of GeoNaviDiff across datasets and semantic factors.

Ablation Study

Figure 5 presents a stepwise ablation using matched source images and semantic strengths. All GeoNaviDiff variants share the same Navigator architecture and inference settings. Starting from Asyrp’s direction-specific Euclidean residual, the variants successively introduce the SS-conditioned Navigator, Riemannian realization, and source-anchored rollout. The progression shows that the Navigator improves target-attribute consistency, Riemannian realization better preserves identity and non-target appearance, and source anchoring yields the most stable trajectories at stronger edits by suppressing cumulative drift. The corresponding quantitative ablation results are reported in Supplementary Appendix D.

Figure 5. Stepwise ablation of cross-space navigation, Riemannian realization, and source-anchored rollout.

Conclusion

We presented GeoNaviDiff, an unsupervised cross-space framework that maps interpretable SS-space directions to local Riemannian trajectories in diffusion hh-space, decoupling semantic representation from image realization. Across multiple datasets, GeoNaviDiff improves representation disentanglement, image fidelity, and trajectory smoothness while preserving non-target content, supporting semantic-basis transfer as an alternative to disentangling diffusion features from scratch.

References

  1. [1]Bengio, Y.; Courville, A.; and Vincent, P. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1798–1828.
  2. [2]Bobkov, D.; Titov, V.; Alanov, A.; and Vetrov, D. 2024. The Devil is in the Details: StyleFeatureEditor for Detail-Rich StyleGAN Inversion and High Quality Image Editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.arxiv.org/abs/2406.10601
  3. [3]Burgess, C. P.; Higgins, I.; Pal, A.; Matthey, L.; and Watters, N. 2018. Understanding disentangling in β-VAE. arXiv preprint.DOI
  4. [4]Chen, T. Q.; Li, X.; Grosse, R. B.; and Duvenaud, D. 2018. Unsupervised Sources of Disentanglement in Variational Autoencoders. In Advances in Neural Information Processing Systems, volume 31, 2615–2625.
  5. [5]Cherepkov, A.; Voynov, A.; and Babenko, A. 2021. Navigating the GAN Parameter Space for Semantic Image Editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3671–3680.arxiv.org/abs/2011.13786
  6. [6]Choi, Y.; Uh, Y.; Yoo, J.; and Ha, J.-W. 2020. StarGAN v2: Diverse Image Synthesis for Multiple Domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8188–8197.arxiv.org/abs/1912.01865
  7. [7]Eastwood, C.; and Williams, C. K. 2018. A framework for the quantitative evaluation of disentangled representations. In International conference on learning representations.
  8. [8]Härkönen, E.; Hertzmann, A.; Lehtinen, J.; and Paris, S. 2020. GANSpace: Discovering interpretable GAN controls. In Larochelle, H., ed., Advances in neural information processing systems, volume 33, 9841–9850. Curran Associates, Inc.arxiv.org/abs/2004.02546
  9. [9]Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, 6626–6637.
  10. [11]Jeong, J.; Kwon, M.; and Uh, Y. 2024. Training-Free Content Injection Using H-Space in Diffusion Models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 5151–5161.arxiv.org/abs/2303.15403
  11. [12]Jin, X.; Li, B.; Xie, B.; Zhang, W.; Liu, J.; Li, Z.; Yang, T.; and Zeng, W. 2024. Closed-Loop Unsupervised Representation Disentanglement with β-VAE Distillation and Diffusion Probabilistic Feedback. In European Conference on Computer Vision.DOI
  12. [13]Karras, T.; Aila, T.; Laine, S.; and Lehtinen, J. 2017. Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv preprint arXiv:1710.10196.arxiv.org/abs/1710.10196
  13. [14]Karras, T.; Laine, S.; and Aila, T. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4401–4410.DOI
  14. [15]Karras, T.; Laine, S.; Aittala, M.; Hellsten, J.; and Lehtinen, J. 2020. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8110–8119.arxiv.org/abs/1912.04958
  15. [16]Kim, H.; and Mnih, A. 2018. Disentangling by Factorising. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2654–2663. PMLR.
  16. [17]Krause, J.; Stark, M.; Deng, J.; and Fei-Fei, L. 2013. 3D Object Representations for Fine-Grained Categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, 554–561.DOI
  17. [18]Kwon, M.; Jeong, J.; and Uh, Y. 2022. Diffusion Models already have a Semantic Latent Space. https://arxiv.org/abs/2210.10960v2.
  18. [19]Liu, H.; Song, Y.; and Chen, Q. 2023. Delving StyleGAN Inversion for Image Editing: A Foundation Latent Space Viewpoint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.arxiv.org/abs/2211.11448
  19. [20]Parihar, R.; Sachidanand, V. S.; Mani, S.; Karmali, T.; and Venkatesh Babu, R. 2024. PreciseControl: Enhancing Text-to-Image Diffusion Models withnbsp;Fine-Grained Attribute Control. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LXXXIII, 469–487. Berlin, Heidelberg: Springer-Verlag. ISBN 978-3-031-73006-1.arxiv.org/abs/2408.05083
  20. [21]Park, Y.-H.; Kwon, M.; Jo, J.; and Uh, Y. 2023. Unsupervised Discovery of Semantic Latent Directions in Diffusion Models. arXiv preprint, (arXiv:2302.12469).arxiv.org/abs/2302.12469
  21. [22]Rolinek, M.; Zietlow, D.; and Martius, G. 2019. Variational autoencoders pursue pca directions (by accident). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12406–12415.arxiv.org/abs/1812.06775
  22. [23]Shen, Y.; and Zhou, B. 2021. Closed-form factorization of latent semantics in gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1532–1540.arxiv.org/abs/2007.06600
  23. [24]Simsar, E.; Kocasari, U.; Er, E. G.; and Yanardag, P. 2023. Fantastic Style Channels and Where To Find Them: A Sub-modular Framework for Discovering Diverse Directions in GANs. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 4731–4740.arxiv.org/abs/2203.08516
  24. [25]Song, J.; Meng, C.; and Ermon, S. 2020. Denoising Diffusion Implicit Models. In International Conference on Learning Representations, 1–12.arxiv.org/abs/2010.02502
  25. [26]Sun, T.; Jiang, Y.; Jin, Y.; Guo, H.; and Zhang, Z. 2026. Context-preserving dermoscopic editing: Mask-guided lesion-aware diffusion for attribute modification. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, 5364–5373.DOI
  26. [27]Tov, O.; Alaluf, Y.; Nitzan, Y.; Patashnik, O.; and Cohen-Or, D. 2021. Designing an Encoder for StyleGAN Image Manipulation. ACM Transactions on Graphics, 40(4).
  27. [28]Voynov, A.; and Babenko, A. 2020. Unsupervised discovery of interpretable directions in the gan latent space. In International conference on machine learning, 9786–9796. PMLR.
  28. [29]Wang, Y.; Schiff, Y.; Gokaslan, A.; Pan, W.; Wang, F.; De Sa, C.; and Kuleshov, V. 2023. InfoDiffusion: Representation Learning Using Information Maximizing Diffusion Models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 36336–36354. PMLR.arxiv.org/abs/2306.08757
  29. [31]Xia, W.; Zhang, Y.; Yang, Y.; Xue, J.-H.; and Zhou, B. 2023. GAN Inversion: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3): 3121–3138.DOI
  30. [32]Yang, T.; Lan, C.; Lu, Y.; and Zheng, N. 2024. Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement. In Advances in Neural Information Processing Systems, volume 37, 82465–82492.arxiv.org/abs/2402.09712
  31. [33]Yang, T.; Wang, Y.; Lu, Y.; and Zheng, N. 2023. DisDiff: Unsupervised disentanglement of diffusion probabilistic models. In Oh, A., ed., Advances in neural information processing systems, volume 36, 69130–69156. Curran Associates, Inc.arxiv.org/abs/2301.13721

Paper details

vlnc:W_k85tzydk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents