betaFeedback

WorksComputer Science

FlowMoDE: Coarse-to-fine Flow Matching for Detail-aware Sim-to-Real Monocular Depth Estimation

Abstract

Monocular depth estimation is a fundamental capability for 3D spatial perception in robotic and intelligent systems. While data-driven discriminative methods have achieved compelling performance for monocular depth estimation, generative approaches can reconstruct fine structural details with stronger generalization. However, most existing generative diffusion models suffer from low inference efficiency caused by multi-step sampling, whereas single-step diffusion models struggle to recover high-fidelity depth maps. To mitigate these drawbacks, this paper presents FlowMoDE, a coarse-to-fine flow-matching-based framework for efficient, structure-aware monocular depth estimation supporting sim-to-real and cross-scene generalization. The core architecture first conducts coarse-level decoding conditioned on diffused features extracted from a pre-trained semantic encoder, and subsequently restores fine-grained structural patterns via expanded decoding layers. Trained exclusively on a single synthetic indoor dataset, our proposed method outperforms prior methods across five real-world indoor and outdoor datasets, yielding an average 14.1% reduction in relative error against state-of-the-art works as well as 9.9% reduction in chamfer distance for point cloud reconstruction. We further validate the generalization capability of FlowMoDE on realistic robotic scenes, with qualitative comparisons against state-of-the-art methods. The project page is https://flowmode-shao.netlify.app/

INTRODUCTION

Monocular Depth Estimation (MDE) is a core 3D perception technique that underpins a wide range of robotic downstream tasks, including scene reconstruction, robotic manipulation, and autonomous navigation. [1, 2] Due to its practical significance, numerous MDE methods have been developed, which are generally divided into discriminative and generative paradigms. Despite delivering promising overall depth prediction performance, discriminative methods inherently suffer from inaccurate depth estimation at object boundaries and fine-grained scene details, which severely degrades the geometric consistency of predicted depth maps. In contrast, generative models alleviate such local detail degradation issues yet introduce inevitable limitations: high computational cost and poor global scene contextual information, resulting in constrained overall prediction accuracy and robustness in complex scenarios.

Discriminative architectures [3, 4, 5, 6] commonly adopt a large-patch Vision Transformer encoder, paired with a convolutional up-sampling decoder. The encoder encodes input images into a low-resolution token grid and leverages global self-attention to capture comprehensive semantic information and scene layout. Such tokenization inevitably degrades pixel-level signals and fine-grained high-frequency details in the encoding phase. [7] Once these fine details are discarded in early encoding stages, convolutional up-sampling modules alone cannot completely restore them. As illustrated by the qualitative results within the green box in Figure 1, this issue yields erroneous depth predictions for fine-scale structures including wires, holes, and edges of distant objects. Moreover, discriminative depth models typically require large-scale training datasets covering diverse scenarios to achieve competitive performance. [8, 9] This imposes substantial costs for data collection and hinders their deployment in specialized scenarios where data acquisition is

Figure 1. Comparison with existing works. Qualitative comparison in green box demonstrates the outstanding structural details from our method, compared with state-of-the-art discriminative model MoGe-2 and generative model PPD. Bubble graph with purple box denotes comparison of performance, data efficiency and computational cost of the proposed model and existing models.

challenging, such as surgical environments. The comparison of data efficiency between generative models and discriminative models is demonstrated in purple box of Figure 1.

Current generative models [10], [11] commonly fine-tune Stable Diffusion to perform depth estimation. These approaches rely on a Variational Autoencoder (VAE) to project depth maps into a latent space. Such compression inherently sacrifices edge sharpness and structural fidelity. Therefore, recent studies have attempted to conduct diffusion directly in the pixel space for better recovery of structural details.[7] Nevertheless, the multi-step pixel-wise diffusion process introduces high computational overhead. While several existing approaches [12], [13] adopt single-step Stable Diffusion for depth estimation, the heavy VAE encoder and decoder still create significant computation bottlenecks. Meanwhile, single-step denoising can limit the performance of depth estimation.

In this work, we propose FlowMoDE, a coarse-to-fine flow matching model for computationally efficient sim-to-real pixel-accurate depth estimation. A lightweight decoder named as Semantic Decoder extracts global semantic features from a pre-trained discriminative encoder. Guided by global features, our model performs efficient decoding to establish a coarse depth distribution through Coarse Decoder. A subsequent Fine-grained Decoder with few wide Transformers further recovers delicate structural details to produce the final pixel-perfect depth map. Our main contributions can be summarized as the following:

  1. We propose a novel flow matching framework for sim-to-real monocular depth estimation, which can provide structure-aware depth maps with lower computational cost.

  1. We introduce coarse-to-fine semantic-guided velocity prediction which efficiently extracts coarse features based on semantic prior and further recover delicate details.

  1. The proposed model is only trained on synthetic indoor dataset, but achieve state-of-the-art performance across five diverse datasets including both indoor and outdoor scenes. Further application of the proposed model in robotic scenes demonstrate the generalization and potential ability for spatial perception of robots.

Discriminative Monocular Depth Estimation

Early monocular depth estimation relied on hand-crafted visual priors and graphical models to infer scene layouts from single images. Deep learning redefined MDE as a supervised dense prediction task, giving rise to various CNN-based solutions such as [14]. Later relative-depth studies such as MiDaS [15] prioritized cross-dataset generalization, proving that mixing heterogeneous datasets improves zero-shot transfer. Meanwhile, DPT [16] firstly established a high-performance transformer-driven dense prediction framework, leading to a series of excellent ViT-based methods. Recent foundation models such as Depth Anything [8], [9] enhance open-domain robustness via large-scale training. In parallel, metric-depth approaches such as Metric3D [17] and UniDepth [5], [6] aim to reconstruct depth with absolute scale and have achieved steady improvements in accuracy and generalization. Despite these advances, modern discriminative models commonly adopt large-patch ViT encoders together with convolutional upsampling decoders. Many discriminative methods strengthen detail reconstruction through local refinement, high-resolution inference or multi-scale decoding. Depth-Pro [18] produces sharp metric depth via high-resolution architectures and boundary-aware training. MoGe [3] improves local geometry by predicting affine-invariant point maps under combined global and local geometric supervision, while MoGe-2 [4] further preserves fine details using refined training data. MDA [19] models per-pixel depth ambiguity with mixture-density representations to generate point maps without flying-point noise. InfiniDepth [20] formulates depth as a neural implicit field and employs local implicit decoders to support arbitrary-resolution queries and clearer depth details. Nevertheless, early tokenization in such architectures damages pixel-level cues and high-frequency components, making fine geometric structures hard to recover in decoding.

Generative Monocular Depth Estimation

Diffusion generative models have been applied in monocular depth estimation models such as Marigold [10] and MonoDiffusion [11], benefiting from rapid convergence and strong representation priors. PPD [7] introduces the injection of semantic features into late diffusion process to maintain global information and perform pixel-level denoise to provide pixel-perfect depth maps. However, the computational cost acutely increase due to multiple diffusion transformer layers. Recently, DepthMaster [13] utilize single-step UNet-based diffusion to perform generative depth estimation. However, heavy VAE still limit computational efficiency, while single-step denoise degrades the performance of the proposed model. DepthFM [21] firstly introduces flow matching into monocular depth estimation. However, lack of semantic information limits its performance, while VAE-based architecture still cause high computational cost and loss of structural details. To this end, FlowMoDE proposed in this work injects adjusted semantic features into few-step flow matching to efficiently perform structure-aware monocular depth estimation.

METHOD

Preliminaries: Flow Matching

Flow matching learns a time-dependent velocity field vθ(xt,t)v_{\theta}(x_t,t) to transport samples from a simple source distribution p0p_0 to the data distribution p1p_1 over timestep tt. The generative process is governed by the ODE:

dxtdt=vθ(xt,t), x0∼p0, x1∼p1, t∈[0,1](1)\frac{\mathrm{d}x_t}{\mathrm{d}t}=v_{\theta}(x_t,t),\ x_0 \sim p_0,\ x_1 \sim p_1,\ t \in[0,1] \tag*{(1)}

In conditional flow matching, the intermediate point is xt=(1−t)x0+tx1x_t=(1-t)x_0+tx_1. The training loss minimizes the ℓ2\ell_2 distance between the predicted velocity field and the analytical target x1−x0x_1-x_0:

LCFM(θ)=E[∥vθ(xt,t)−(x1−x0)∥22].(2)\mathcal{L}_{\mathrm{CFM}}(\theta)=\mathbb{E}\left[\left\|v_\theta(x_t,t)-(x_1-x_0)\right\|_2^2\right]. \tag*{(2)}

Overview

The overall pipeline of the proposed method is demonstrated in Figure 2. Given an input image II, the corresponding ground-truth depth map with noise DtD_t and its semantic feature map ss, the proposed coarse-to-fine velocity prediction model Fdiff\mathcal{F}_{diff} predict the velocity map v^=Fdiff([I,Dt]∣s)\hat{v}=\mathcal{F}_{diff}([I,D_t]|s) for interpolation matching at the training stage. At the inference stage, given the input image II and random noise x1∼N(0,1)x_1\sim\mathcal{N}(0,1), FlowMoDE subsequently utilizes the shared coarse-to-fine velocity prediction model for twice to predict the depth map D^\hat{D}.

Figure 2. Pipeline of the proposed method.

Coarse-to-Fine Velocity Prediction

Firstly, shown as Eq. (3), semantic feature s0s^0 of the input image II is extracted by a semantic encoder Fsem\mathcal{F}_{sem}, which can be pretrained models such as DINOv2.

s0=Linear⁡ds→dc(Norm⁡(Fsem(I)))(3)s^0=\operatorname{Linear}_{d_s\to d_c}\left(\operatorname{Norm}(\mathcal{F}_{sem}(I))\right) \tag*{(3)}

where dsd_s is the dimension of the output from semantic encoder, which means Fsem(I)∈RB×HP×WP×ds\mathcal{F}_{sem}(I)\in\mathbb{R}^{B\times\frac{H}{P}\times\frac{W}{P}\times d_s}. Moreover, dcd_c is the dimension of the condition token, which is set to 384 in this work. The semantic feature s0s^0 is further processed by a Semantic Decoder with four Transformer blocks to obtain semantic condition ss.

The timestep tt is sampled from [0,T=1000][0,T=1000] based on normal noise ϵt∼N(0,1)\epsilon_t\sim\mathcal{N}(0,1) by equation Eq. (4). Meanwhile, the ground-truth depth map DD is added by noise ϵd∼N(0,1)\epsilon_d\sim\mathcal{N}(0,1) through the process formulated as Eq. (5) and concatenated with the input image II. The concatenated image Icd=Concat⁡(Dt,I−0.5)I_{cd}=\operatorname{Concat}(D_t,I-0.5) is then embedded to 16×1616\times16 patches as the input of Coarse Decoder.

t=round⁡(T1+e−ϵt)(4)t=\operatorname{round}\left(\frac{T}{1+e^{-\epsilon_t}}\right) \tag*{(4)}
Dt=(1−tT)(Norm⁡d(log⁡(D+1))−0.5)+tTϵd(5)D_t=\left(1-\frac{t}{T}\right)\left(\operatorname{Norm}_d(\log(D+1))-0.5\right)+\frac{t}{T}\epsilon_d \tag*{(5)}

where Norm⁡d\operatorname{Norm}_d means linear normalization using 2%2\%–98%98\% quantiles.

During training stage, the time condition tt is added with the semantic condition ss, then modulated to factors of scale, shift and gate for multi-head attention βm\beta_m, γm\gamma_m, gmg_m and MLP in feed forward network βf\beta_f, γf\gamma_f, gfg_f by SiLU activation and Linear projection layer, formulated as Eq. (6).

[βm,γm,gm,βf,γf,gf]=Linear⁡dc→6dc(SiLU⁡(c))(6)[\beta_m,\gamma_m,g_m,\beta_f,\gamma_f,g_f]=\operatorname{Linear}_{d_c\to6d_c}(\operatorname{SiLU}(c)) \tag*{(6)}

The coarse outputs the coarse feature map based on the embedded input through four conditioned Transformer blocks. For each conditioned Transformer block, denoted xx as the input and yy as the output, the modulated condition is injected into Transformer by Eq. (7) and Eq. (8) to guide the flow matching process with semantic information.

ymsa=x+gm⋅Attn⁡((1+βm)⋅LN⁡(x)+γm)(7)y_{msa}=x+g_m\cdot\operatorname{Attn}((1+\beta_m)\cdot\operatorname{LN}(x)+\gamma_m) \tag*{(7)}
y=ymsa+gf⋅MLP⁡((1+βf)⋅LN⁡(ymsa)+γf)(8)y=y_{msa}+g_f\cdot\operatorname{MLP}((1+\beta_f)\cdot\operatorname{LN}(y_{msa})+\gamma_f) \tag*{(8)}

After coarse decoder, the output feature map is upsampled two times to 8×88\times8 patches. Meanwhile, the condition c=s+tc=s+t and timestep tt are also respectively upsampled to cfc_f and tft_f by Eq. (9), where NRNR means repeat the nearest element in cc at each position of cfc_f to upsample cc.

cf=Linear⁡dc→df(NR(c)), tf=Linear⁡dc→df(t)(9)c_f=\operatorname{Linear}_{d_c\to d_f}(NR(c)),\ t_f=\operatorname{Linear}_{d_c\to d_f}(t) \tag*{(9)}

The fine-grained feature map ffinef_{fine} is obtained from the coarse feature map by four conditioned Transformer block with condition cfc_f. In the final layer, the fine-grained map goes through layer normalization followed by a linear layer with modulation of time token tft_f, shown as Eq. (10) and Eq. (11).

[β,γ]=Linear⁡df→2df(SiLU⁡(tf))(10)[\beta,\gamma]=\operatorname{Linear}_{d_f\to2d_f}(\operatorname{SiLU}(t_f)) \tag*{(10)}
ffinal=Linear⁡df→p2((1+β)⋅LN⁡(ffine)+γ)(11)f_{final}=\operatorname{Linear}_{d_f\to p^2}((1+\beta)\cdot\operatorname{LN}(f_{fine})+\gamma) \tag*{(11)}

At last, we unpatchify the final feature map ffinalf_{final} from the final layer to provide the velocity map v^\hat{v}.

Training: Single-step Flow Matching

Given ground-truth (GT) depth map DD and noise ϵ\epsilon, the target velocity map is v∗=ϵ−Dv^* = \epsilon- D. The single coarse-to-fine flow matching model Fdiff\mathcal{F}_{diff} outputs predicted velocity map v^\hat{v} from the input image II and noisy GT depth map DtD_t from Eq. (5). We utilize MSE between prediction and target velocity as the loss function for training, formulated as Eq. (12).

L=∑imi∥v^−v∗∥22∑imi+ϵ(12)\mathcal{L} = \frac{\sum_i m_i \lVert\hat{v} - v^* \rVert_2^2}{\sum_i m_i + \epsilon} \tag*{(12)}

where mm denotes the mask of valid depth, where value of invalid area will be set to 0.

Two-step Inference

During training stage, semantic tokens ss are extracted once from the input RGB image. Firstly, the shared coarse-to-fine network Fdiff\mathcal{F}_{diff} predicts velocity map from the concatenation of input RGB image II and random noise x1∼N(0,1)x_1 \sim\mathcal{N}(0,1). The original noise map is further diffused by the predicted velocity map, formulated as Eq. (13).

x0.5=x1−0.5×Fdiff([x1,I−0.5]∣s)(13)x_{0.5} = x_1 - 0.5 \times\mathcal{F}_{diff}([x_1, I - 0.5] \mid s) \tag*{(13)}

As Eq. (14) shows, x0.5x_{0.5} concatenated with the input image II is further processed by the shared coarse-to-fine flow matching model to generate the final velocity map and yield the final clean depth map D^=x0+0.5\hat{D} = x_0 + 0.5.

x0=x0.5−0.5×Fdiff([x0.5,I−0.5]∣s)(14)x_0 = x_{0.5} - 0.5 \times\mathcal{F}_{diff}([x_{0.5}, I - 0.5] \mid s) \tag*{(14)}

EXPERIMENTS AND RESULTS

Implementation Details

We utilize frozen DINOv2-B from MoGe as our semantic encoder. The dimensions of semantic decoder, coarse decoder and fine-grained decoder are respectively set to 384, 384 and 768. We train the proposed model on single dataset Hypersim [28], which consists of 54K high-quality synthetic indoor frames and corresponding depth maps. The proposed model is trained for 200 epochs (2000 sampling steps for each epoch) with a batch size of 4 and a fixed learning rate of 1e-4, while all training samples are resized to 640×480. All experiments are implemented on a NVIDIA H100 GPU.

Comparison Results: Monocular Depth Estimation

All methods are evaluated on five diverse real-world datasets NYUv2 [29], KITTI [30], ETH3D [31], ScanNet [32] and DIODE [33] following Marigold [10], resizing all samples to a resolution with a long edge of 768. We compared our model with existing discriminative models and generative models. For DIODE, we directly use masks provided officially, while some works such as PPD [7] use Sobel to filter areas with high gradient and RANSAC to align pixels in their papers. We utilize absolute relative error (AR) and accuracy (δ\delta) metrics, as formulated as Eq. (15) and Eq. (16), where DD is the ground truth of the depth map and D^\hat{D} is the predicted depth map, to evaluate the performance.

AR=1∣D∣∑d^∈D∣d−d^∣d(15)\mathrm{AR} = \frac{1}{|D|} \sum_{\hat{d} \in D} \frac{|d-\hat{d}|}{d} \tag*{(15)}
δ=1∣D∣∣{d^∈D∣max⁡(dd^,d^d)<1.25}∣(16)\delta= \frac{1}{|\mathbf{D}|}\left|\left\{\hat{d} \in\mathbf{D} \mid\max\left(\frac{d}{\hat{d}},\frac{\hat{d}}{d}\right) < 1.25\right\}\right| \tag*{(16)}

The quantitative comparison results are shown in Table I. Our model outperforms existing generative models across five datasets, achieving state-of-the-art performance among all compared models with highest average rank on 10 metrics. Displaying results of two different indoor scenes and two different outdoor scenes, Figure 3 further demonstrate the outstanding performance of the proposed method.

MethodYearTraining DataFLOPs (G)NYUv2 AR↓NYUv2 δ1↑KITTI AR↓KITTI δ1↑ETH3D AR↓ETH3D δ1↑ScanNet AR↓ScanNet δ1↑DIODE AR↓DIODE δ1↑Avg Rank
MiDaS [15]20202M184.611.188.523.663.018.475.212.184.633.271.519.2
DiverseDepth [22]2021320K207.811.787.519.070.422.869.410.988.237.663.119.6
LeReS [23]2021354K190.49.091.614.978.417.177.79.191.727.176.616.7
Omnidata [24]202112.2M195.47.494.514.983.516.677.87.593.633.974.216.7
DPT [16]20211.4M449.29.890.310.090.17.894.68.293.418.275.814.6
HDN [25]2022300K195.46.994.811.586.712.183.38.093.924.678.014.4
Metric3D [17]20238M225.45.896.35.897.06.696.07.494.122.478.59.6
DA [8]202462M586.04.398.17.694.76.597.24.298.027.775.96.6
DA V2 [9]202463.5M1816.54.497.97.594.86.298.04.398.126.075.96.0
UniDepth v2 [6]202516M1925.04.298.08.192.64.597.63.798.221.678.73.7
MoGe [3]20259M3108.04.497.97.992.94.897.34.198.119.779.94.2
InfiniDepth [20]2026∼1M#1844.35.097.38.791.65.296.94.997.322.478.17.9
Marigold [10]202474K5196.55.596.49.991.66.596.06.495.130.877.312.7
GeoWizard [26]2025280K5247.35.296.69.792.16.496.16.195.329.779.210.6
DepthFM [21]202574K6192.5*5.596.38.991.35.896.26.395.422.478.510.4
GenPercept [27]202574K5237.2*5.296.69.492.36.695.75.696.535.775.612.2
Lotus [12]202559K5237.2*5.496.88.592.25.997.05.995.722.873.810.2
DepthMaster [13]202674K5241.5*5.097.28.293.75.397.45.596.721.577.66.9
PPD [7]202654K18158.84.397.48.093.14.597.74.597.324.576.96.1
FlowMoDE-54K3140.14.098.16.295.93.998.84.397.618.880.42.2

Table I. Results on Marigold benchmarks. *VAE encoder & decoder are not considered in original reported FLOPs calculation. #No reported value, estimated from the description in the original paper. Best models from all models are in bold, while second best models are underlined. Best results and second best results from Sim-to-Real Generative Models are marked by corresponding colors.

Qualitative comparison of predicted depth maps and corresponding error maps for indoor and outdoor examples

Figure 3. Qualitative comparison on indoor examples (first four rows) and outdoor examples (last four rows). Figures in the second row of each sample are GT and corresponding error maps based on Abs Rel.

Comparison Results: Point Cloud Reconstruction

To evaluate the downstream application and structural details of monocular depth estimation, we utilize NYUv2 and ETH3D, which provide camera intrinsics for 3D ground-truth, to compare point cloud reconstruction from predicted depth maps. We choose three discriminative models and two generative models with best performance on depth estimation

MethodNear-field NYUv2 Acc↓Near-field NYUv2 CD↓Far-field ETH3D Acc↓Far-field ETH3D CD↓Mean Acc↓Mean CD↓
UniDepth v286.7117.2332.4427.1209.6272.2
MoGe92.7121.4352.4448.7222.6285.1
InfiniDepth100.4130.7352.3463.3226.4297.0
DepthMaster100.0120.9321.8322.3210.9221.6
PPD172.2160.3840.7596.4506.5378.4
FlowMoDE89.3100.2269.6299.1179.5199.7

Table 2. Quantitative comparison of point cloud reconstruction based on monocular depth estimation. Best results and second best results are marked by corresponding colors.

to compare with our model. Accuracy (Acc) and Chamfer Distance (CD) formulated as Eq. (18) and Eq. (19) based on point clouds from depth maps (Eq. (17)) are utilized as the metrics for point cloud reconstruction.

P(u,v)=K−1[u,v,1]D(u,v)(17)P(u,v) = K^{-1}[u,v,1]D(u,v) \tag*{(17)}
Acc=1∣P∣∑p∈Pmin⁡g∈G∥p−g∥2(18)Acc = \frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}\min_{g\in\mathcal{G}}\lVert p-g\rVert_2 \tag*{(18)}
CD=12(1∣P∣∑p∈Pmin⁡g∈G∥p−g∥2+1∣G∣∑g∈Gmin⁡p∈P∥g−p∥2)(19)CD = \frac{1}{2}\left(\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}\min_{g\in\mathcal{G}}\lVert p-g\rVert_2+\frac{1}{|\mathcal{G}|}\sum_{g\in\mathcal{G}}\min_{p\in\mathcal{P}}\lVert g-p\rVert_2\right) \tag*{(19)}

where P\mathcal{P} is the point cloud from predicted depth map and G\mathcal{G} is the point cloud from ground-truth depth map. For ETH3D dataset, we undistort RGB images and corresponding depth maps following official document 1 to match pinhole camera hypothesis before using Eq. (17).

Based on results shown in Table. II, our model achieve the outstanding performance with best average metrics across two datasets, while UniDepth v2 can provide results with better accuracy on NYUv2. Furthermore, we perform qualitative comparison in Figure 4. We can find that the proposed model can provide better spatial perception on structural details for delicate objects, such as the faucet in the first column of Figure 4 and windows in the second column of Figure 4.

Comparison of point cloud reconstruction based on estimated depth maps

Figure 4. Comparison of point cloud reconstruction based on estimated depth maps.

Comparison Results: 3D Perception in Robotic Scenes

In contrast to conventional settings, robotic environments typically contain multiple distinct agents that interact with each other. Therefore, we choose diverse cross-embodiment robotic scenes which respectively consist of Baxter, Franka, Kuka and Da Vinci Robot, to implement the proposed model for zero-shot monocular depth estimation. Compared with the most recent methods in Figure 5, we can find that our proposed method can perceive subtle details such as the delicate structure of robot body or cables. Meanwhile, based on 3D reconstruction results compared with PPD or MoGe-2 in Figure 6, FlowMoDE can better reconstruct the robot body independent of the environment or objects, reducing flying points between them.

Qualitative comparison of monocular depth estimation in robotic scenarios

Figure 5. Qualitative comparison of monocular depth estimation in robotic scenarios.

Qualitative comparison of point cloud reconstruction based on estimated depth maps in diverse cross-embodiment robotic scenes

Figure 6. Qualitative comparison of point cloud reconstruction based on estimated depth maps in diverse cross-embodiment robotic scenes.

Ablation Studies

Firstly, we perform ablations on velocity prediction model by re-organizing semantic decoder, coarse decoder and fine-grained decoder with different numbers of layers. Meanwhile, we also use different injection methods including cross-attention, concatenation and adaptive normalization instead of our modulation to perform ablation studies. To control the computational cost of models, we keep the overall number of decoder layers as 12. According to results shown in Table III. Besides, we select several ablation settings with best performance to perform qualitative comparison in Figure 7 and compare point cloud reconstruction with our full model in Table. Table IV. Furthermore, we respectively conduct 1-step, 2-step, 4-step, 6-step and 8-step inference to demonstrate the influence of flow matching steps, results of which are displayed in Table V. We can find that 2-step inference via 4-layer decoders with the propsoed semantic injection is the best choice for monocular depth estimation.

Qualitative comparison of ablation settings

Figure 7. Qualitative comparison of ablation settings.

AblationNYUv2 Acc↓NYUv2 CD↓ETH3D Acc↓ETH3D CD↓Mean Acc↓Mean CD↓
II118.0119.9455.4383.1286.7251.5
IV132.2123.1460.2368.1296.2245.6
V135.2128.5902.5621.1518.8374.8
Ours89.3100.2269.6299.1179.5199.7

Table IV. Ablation results on point cloud reconstruction. Best models are in bold.

StepsNYUv2 ARNYUv2 δ₁KITTI ARKITTI δ₁ETH3D ARETH3D δ₁ScanNet ARScanNet δ₁DIODE ARDIODE δ₁
14.498.17.597.54.698.54.597.618.880.3
24.098.16.295.93.998.84.397.618.880.4
44.198.06.595.64.098.74.397.518.980.3
64.298.06.795.44.198.74.497.519.080.2
84.298.06.795.44.198.74.497.519.080.2
MethodsGFLOPsms/imageMem (GB)Params (M)
UniDepth v21925.023.33.98353.8
MoGe3108.056.83.65314.2
DepthMaster5241.594.76.711028.2
PPD18158.8193.74.22804.2
FlowMoDE3140.153.41.87398.7

Table V. Ablation studies on flow matching steps. Best models are in bold, while second best models are underlined.

#Semantic DecoderCoarse DecoderFine-grained DecoderNYUv2 AR↓NYUv2 δ₁ ↑KITTI AR↓KITTI δ₁ ↑ETH3D AR↓ETH3D δ₁ ↑ScanNet AR↓ScanNet δ₁ ↑DIODE AR↓DIODE δ₁ ↑
Full Model4.098.16.295.93.998.84.397.618.880.4
I4-Layer8-Layer×5.897.28.394.06.197.45.696.821.778.7
II4-Layer×8-Layer5.597.58.993.85.997.55.397.220.779.6
III×12-Layer×6.296.810.589.68.295.76.196.322.177.9
IV××12-Layer5.597.49.892.36.997.05.496.921.479.0
V×6-Layer6-Layer5.597.39.892.56.097.65.396.820.579.8
VI4-Layer6-Layer2-Layer6.096.98.893.76.397.36.096.221.678.6
VII2-Layer6-Layer4-Layer6.096.89.392.86.697.15.896.622.577.9
VIIICross Attention6.396.634.242.58.294.56.096.523.575.6
IXConcatenation5.297.58.294.15.597.95.397.120.779.6
XAdaNorm5.597.39.293.25.997.35.497.020.879.7
XINo Semantic20.665.526.056.920.169.218.769.932.659.1

Table III. Ablation Studies on Velocity Prediction Model.

Computational Cost

We compare the computational cost of our model with several existing models with relatively best performance. We use GPU memory requirement (Mem) and number of

parameters (Params) to evaluate the model size. Meanwhile, GFLOPs and inference speed (ms per image) are utilized to evaluate the efficiency of models. As shown in Table VI, the proposed FlowMoDE incurs higher latency and model weight overhead compared with UniDepth v2, but achieves significantly lower computational cost than prior generative depth models.

CONCLUSIONS

In this paper, we present FlowMoDE, a flow-matching generative model for sim-to-real monocular depth estimation to recover fine structural details. Trained solely on synthetic data, our model outperforms prior methods on five diverse benchmarks. We further validate its strong ability to reconstruct fine-grained geometry in various robotic scenarios against state-of-the-art approaches. With fewer Transformer blocks and minimal inference steps, FlowMoDE is lighter and more efficient than existing generative depth models. Still, patch embedding in our flow-matching pipeline may produce occasional flying pixels. Moreover, training only on near-field indoor synthetic data combined with patch-wise denoising can cause blurred edges for distant regions.

References

  1. [1]J. Liu, H. Ma, Y. Guo, Y. Zhao, C. Zhang, W. Sui, and W. Zou, “Monocular depth estimation and segmentation for transparent object with iterative semantic and geometric fusion,” in 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 11 162–11 168.arxiv.org/abs/2502.14616
  2. [2]R. Fan, W. Zhao, M. Lin, Q. Wang, Y.-J. Liu, and W. Wang, “Generalizable thermal-based depth estimation via pre-trained visual foundation model,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 14 614–14 621.
  3. [3]R. Wang, S. Xu, C. Dai, J. Xiang, Y. Deng, X. Tong, and J. Yang, “MoGe: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision,” in 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025, pp. 5261–5271.arxiv.org/abs/2410.19115
  4. [4]R. Wang, S. Xu, Y. Dong, Y. Deng, J. Xiang, Z. Lv, G. Sun, X. Tong, and J. Yang, “Moge-2: Accurate monocular geometry with metric scale and sharp details,” Advances in Neural Information Processing Systems, vol. 38, pp. 35 928–35 959, 2026.arxiv.org/abs/2507.02546
  5. [5]L. Piccinelli, Y.-H. Yang, C. Sakairdis, M. Segu, S. Li, L. Van Gool, and F. Yu, “UniDepth: Universal monocular metric depth estimation,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024, pp. 10 106–10 116.
  6. [6]L. Piccinelli, C. Sakairdis, Y.-H. Yang, M. Segu, S. Li, W. Abbeloos, and L. Van Gool, “Unidepthv2: Universal monocular metric depth estimation made simpler,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.DOI
  7. [7]G. Xu, H. Lin, H. Luo, X. Wang, J. Yao, L. Zhu, Y. Pu, C. Chi-, H. Sun, B. Wang et al., “Pixel-perfect depth with semantics-prompted diffusion transformers,” Advances in Neural Information Processing Systems, vol. 38, pp. 174 731–174 755, 2026.arxiv.org/abs/2510.07316
  8. [8]L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024, pp. 10 371–10 381.arxiv.org/abs/2401.10891
  9. [9]L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,” Advances in neural information processing systems, vol. 37, pp. 21 875–21 911, 2024.arxiv.org/abs/2406.09414
  10. [10]B. Ke, A. Oubokhov, S. Huang, N. Metzger, R. C. Daudt, and K. Schindler, “Repurposing diffusion-based image generators for monocular depth estimation,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024, pp. 9492–9502.arxiv.org/abs/2312.02145
  11. [11]S. Shao, Z. Pei, W. Chen, D. Sun, P. C. Chen, and Z. Li, “Monodiffusion: Self-supervised monocular depth estimation using diffusion model,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 35, no. 4, pp. 3664–3678, 2024.arxiv.org/abs/2311.07198
  12. [12]J. He, H. Li, W. Yin, Y. Liang, L. Li, K. Zhou, H. Zhang, B. Liu, and Y. Chen, “Lotus: Diffusion-based visual foundation model for high-quality dense prediction,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 89 454–89 467.arxiv.org/abs/2409.18124
  13. [13]Z. Song, Z. Wang, B. Li, H. Zhang, R. Zhu, L. R. Liu, P.-T. Jiang, and T. Zhang, “Depthmaster: Taming diffusion models for monocular depth estimation,” IEEE Transactions on Circuits and Systems for Video Technology, 2026.DOI
  14. [14]H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018, pp. 2002–2011.arxiv.org/abs/1806.02446
  15. [15]R. Ranftl, K. Lasinger, K. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 3, pp. 1623–1637, 2020.
  16. [16]R. Ranftl, A. Bochkovskiy, and V. Koltun, “Vision transformers for dense prediction,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021, pp. 12 159–12 168.arxiv.org/abs/2103.13413
  17. [17]W. Yin, C. Zhang, H. Chen, Z. Cai, G. Yu, K. Wang, X. Chen, and C. Shen, “Metric3d: Towards zero-shot metric 3d prediction from a single image,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023, pp. 9009–9019.arxiv.org/abs/2307.10984
  18. [18]A. Bochkovskiy, A. Delaunoy, H. Germain, M. Santos, Y. Zhou, S. Richter, and V. Koltun, “Depth pro: Sharp monocular metric depth in less than a second,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 75 602–75 637.
  19. [19]S. Bian, C. Xu, and J. Gao, “Modeling depth ambiguity: A mixture-density representation for flying-point-free depth estimation,” arXiv preprint arXiv:2606.02552, 2026.arxiv.org/abs/2606.02552
  20. [20]H. Yu, H. Lin, J. Wang, J. Li, Y. Wang, X. Zhang, Y. Wang, X. Zhou, R. Hu, and S. Peng, “InfInDepth: Arbitrary-resolution and fine-grained depth estimation with neural implicit fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2026, pp. 26 920–26 930.arxiv.org/abs/2601.03252
  21. [21]M. Gui, J. Schusterbauer, U. Prestel, P. Ma, D. Kotovenko, O. Grebenkova, S. A. Baumann, V. T. Hu, and B. Ommer, “Depthfm: Fast generative monocular depth estimation with flow matching,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 3, 2025, pp. 3203–3211.
  22. [22]W. Yin, X. Wang, C. Shen, Y. Liu, Z. Tian, S. Xu, C. Sun, and D. Renin, “DiverseDepth: Affine-invariant depth prediction using diverse data,” arXiv preprint arXiv:2002.00569, 2020.arxiv.org/abs/2002.00569
  23. [23]W. Yin, J. Zhang, O. Wang, S. Niklaus, L. Mai, S. Chen, and C. Shen, “Learning to recover 3d scene shape from a single image,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 204–213.arxiv.org/abs/2012.09365
  24. [24]A. Eftekhar, A. Sax, J. Malik, and A. Zamir, “Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021, pp. 10 766–10 776.arxiv.org/abs/2110.04994
  25. [25]C. Zhang, W. Yin, B. Wang, G. Yu, B. Fu, and C. Shen, “Hierarchical normalization for robust monocular depth estimation,” Advances in Neural Information Processing Systems, vol. 35, pp. 14 128–14 139, 2022.arxiv.org/abs/2210.09670
  26. [26]X. Fu, W. Yin, M. Hu, K. Wang, Y. Ma, P. Tan, S. Shen, D. Lin, and X. Long, “Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image,” in European Conference on Computer Vision. Springer, 2024, pp. 241–258.
  27. [27]G. Xu, M. Liu, C. Fan, K. Xie, Z. Zhao, H. Chen, C. Shen et al., “What matters when repurposing diffusion models for general dense perception tasks?” in International Conference on Learning Representations, vol. 2025, pp. 6786–6799.arxiv.org/abs/2403.06090
  28. [28]M. Roberts, J. Ramapuram, A. Ranjan, A. Kumar, M. A. Bautista, N. Paczan, R. Webb, and J. M. Susskind, “Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021, pp. 10 892–10 902.arxiv.org/abs/2011.02523
  29. [29]N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in European conference on computer vision. Springer, 2012, pp. 746–760.DOI
  30. [30]A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” The international journal of robotics research, vol. 32, no. 11, pp. 1231–1237, 2013.DOI
  31. [31]T. Schöps, J. L. Schönberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys, and A. Geiger, “A multi-view stereo benchmark with high-resolution images and multi-camera videos,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3260–3269.DOI
  32. [32]A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, 2017, pp. 2432–2443.arxiv.org/abs/1702.04405
  33. [33]I. Vasiljević, N. Kolkin, S. Zhang, R. Luo, H. Wang, F. Z. Dai, A. F. Daniele, M. Mostajabi, S. Basart, M. R. Walter et al., “DIODE: A dense indoor and outdoor depth dataset,” arXiv preprint arXiv:1908.00463, 2019.arxiv.org/abs/1908.00463

Paper details

vlnc:W_n8fj9hgb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents