betaFeedback

WorksComputer Science

Unlocking the Potential of Vision Foundation Models for Multi-Task Dense Prediction via Mixture of Task Adapters

Abstract

Multi-task dense prediction seeks a single, coherent representation that supports simultaneously discriminative semantic recognition and regularized geometric/structural inference. For vision foundation models, pretrained semantic invariances conflict with the frequency and smoothness biases of dense objectives; joint fine-tuning and single-route adapters therefore compress competing gradients into one update channel, causing negative transfer and unstable cross-task agreement. We frame this as two-level adaptation: task-generic alignment that preserves priors, and task-aware specialization that resolves late-stage conflicts. Based on this principle, we propose D2MoE , a Mixture of Task Adapters that explicitly factorizes these roles within a frozen DINOv3 backbone. D2MoE introduces Task-Generic Low-Rank Adaptation (TG-LoRA) in attention projections throughout the network to provide a stable, shared adaptation channel, and a Task-Adaptive Mixture of Experts (TA-MoE) injected only at the final attention block of each stage to enact sparse, task-conditioned perturbations of token mixing in a single forward pass. A Semantic Prior Injector further couples multi-level task features with frozen foundation priors to reinforce semantic coherence while preserving boundary and structural fidelity. Experiments across diverse benchmarks substantiate that this structured factorization yields robust multi-task improvements with a small trainable footprint, establishing a principled and scalable route to prior-preserving, foundation-centric multi-task dense perception.

Introduction

Multi-task dense prediction (MTDP) [1], [2], [3] unifies multiple pixel-level tasks (e.g., semantic segmentation [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], depth estimation [15], [16], [17], [18], [19], [20], [20], [21], and saliency detection [22], [23], [24], [25], [26], [27]) in a single model to share computation, couple semantic/geometric/structural cues, and reduce redundancy relative to task-specific systems. This joint setting further promotes cross-task spatial coherence, especially near boundaries and geometric discontinuities. Recent MTDP increasingly builds on vision foundation models (VFMs) [28], [29], [30], using pretrained transformers (e.g., ViT [31], [32], [33], [34], [35], [36] and Swin Transformer [37], [38], [39]). As a result, the central problem shifts from learning shared representations to coordinating adaptation of semantically organized foundation features across heterogeneous dense objectives.

Although vision foundation models provide rich, transferable features, MTDP under VFMs faces a core incompatibility: one representation must satisfy objectives with distinct spatial and frequency preferences. This yields three challenges: (i) Conflicting Task Priors. Semantic objectives depend on localized discriminative detail, whereas geometric objectives prefer smooth, globally consistent structure. (ii) Prior Erosion. Naive end-to-end optimization overwrites transferable invariances, degrading generalization and cross-task consistency. (iii) Single-route Adaptation Interference, i.e., forcing all tasks through shared prompts/adapters/decoders projects heterogeneous gradients into one subspace, amplifying negative transfer. These observations motivate conditional, task-aware adaptation that selectively updates the representation to enable task-specific refinement while preserving the shared prior. Selective routing via mixture of experts provides a principled mechanism for allocating distinct update paths to competing objectives and mitigating negative transfer [40], [41].

These observations indicate that once a single shared adaptation path bottlenecks, MTDP needs task-conditioned computation at the adaptation interface while preserving a shared foundation prior rather than hard-partitioning the backbone. (i) Unscalable Factorization. Separate encoders/decoders for each task destroys sharing and is prohibitive for foundation-scale backbones and dense outputs [31], [42]. (ii) Delayed Conflict Resolution. Late-branching heads postpone specialization, leaving upstream features exposed to the same conflicting gradients. (iii) Sparse, Task-conditioned Capacity. Effective MTDP should keep a single backbone but introduce task-dependent specialization via sparsely activated auxiliary parameters. Mixture of experts (MoE) implements this through sparse gating/routing, allocating partially disjoint subspaces per objective without sacrificing the shared representation [40], [41], [43], [44], [45]. In contrast, non-MoE MTDP either applies largely uniform encoder adaptation [46], [47] or shifts capacity into decoders whose task interaction is weakly coupled to the backbone features being adapted [37], [38].

Several recent parameter-efficient and conditional multi-task adaptation methods are closely related to this direction. Polyhistor [49] and VMT-Adapter [50] improve multi-task transfer by sharing lightweight adaptation modules while preserving task-specific knowledge, and TADFormer [51] further introduces task-adaptive dynamic filtering for input-conditioned task adaptation. DiTask [52] studies multi-task fine-tuning from a weight-geometry perspective by preserving pretrained singular-vector structures and modulating singular values. These methods validate the importance of parameter-efficient multi-task adaptation, yet most of them focus on designing general adaptation modules or task-conditioned filters, rather than explicitly locating where task conflicts should be resolved inside a frozen vision foundation model for dense prediction. In parallel, MoE-based MTL methods such as M³ViT [53], Mod-Squad [54], AdaMV-MoE [55], TaskExpert [33] and MLoRE [34] introduce conditional expert selection to improve cooperation-specialization trade-offs. However, existing MoE designs typically insert trainable experts throughout the backbone, assemble task features in the decoder, or target compact task-specific execution, which may either perturb pretrained representations or leave task conflicts insufficiently addressed.

Motivated by the above analysis, we view the MTDP bottleneck under frozen VFMs as requiring two complementary capabilities: task-generic adaptation and task-aware specialization. Task-generic adaptation aligns the foundation representation to the downstream data distribution while maintaining transferable semantic prior, whereas task-aware specialization explicitly resolves conflicts induced by heterogeneous dense objectives. A closely related foundation-model-driven approach is SAK [36], which uses a shared teacher-agnostic stem with teacher-specific adapter paths to distill complementary biases from multiple VFMs and fuses them via task routers for multi-task prediction. However, the student is optimized to satisfy heterogeneous teacher feature spaces, yielding a compromise representation that weakens domain adaptation, while specialization remains teacher-conditioned rather than explicitly resolving task-level gradient conflicts. Practically, requiring ≥3\ge3 VFM teachers incurs O(K)O(K) compute/memory (multiple frozen backbones, matching losses) during training, making it bulky for deployment (as shown in Figure 1(a)).

Motivation of D²MoE

Figure 1. Motivation of D²MoE. (a) SAK distills from multiple VFMs into a shared student, coupling heterogeneous task gradients and incurring multi-teacher overhead. (b) D²MoE instead freezes a single VFM and enforces early task decoupling through a Mixture of Task Adapter (MoTA), allocating sparse, task-conditioned low-rank update routes for specialization without eroding pretrained invariances. (c) Correlation probes (Pearson maps around sampled points) confirm that model with MoTA induces more task-consistent feature organization for both semantic segmentation and depth, improving boundary adherence without sacrificing long-range structural coherence.

To address the above limitations, we propose D²MoE, a foundation model driven framework for multi-task dense prediction. Unlike prior MoE-based MTDP methods that mainly emphasize task routing and expert specialization, D²MoE uses MoE as part of a lightweight adapter design to introduce task-adaptive updates without tuning the VFM backbone. D²MoE builds on a single frozen VFM backbone and performs multi-task decoupling through a unified Mixture of Task Adapters (MoTA) design (as shown in Figure 1(b)), which combines Task-Generic Low-rank Adaptation (TG-LoRA) with Task-Adaptive Mixture of Experts (TA-MoE). TG-LoRA is applied across backbone layers to achieve stable and parameter-efficient adaptation, while TA-MoE is selectively placed at the final layer of each stage to enable sparse, task-aware specialization within a single forward pass. In addition, D²MoE incorporates semantic priors from DINOv3 to guide cross-layer feature integration, facilitating both global context modeling and fine-grained detail preservation. As a result, D2^{2}MoE achieves state-of-the-art performance on standard multi-task dense prediction benchmarks with minimal additional parameters (as shown in Figure 2).

Performance versus model complexity

Figure 2. Performance vs. Model Complexity. Comparison of D²MoE against state-of-the-art methods on NYUD-v2 [48]. D²MoE achieves superior performance (higher mIoU and lower RMSE) despite a substantial reduction in the total number of trainable parameters. The trainable-parameter count varies with the task configuration, with 45M on NYUD-v2 (shown here) and 54M on Pascal-Context (as shown in Table 1.)

Our contributions are summarized as follows.

  1. We formalize multi-task dense prediction with frozen vision foundation models as a two-level adaptation problem, disentangling task-generic alignment from task-aware specialization. This perspective explains negative transfer under single-route adaptation and motivates sparse, task-conditioned computation.

  1. We propose D2^{2}MoE, a Mixture of Task Adapters architecture that combines Task-Generic Low-Rank Adaptation with Task-Adaptive Mixture of Experts. The design introduces sparse, late-stage task specialization while preserving semantic priors within a single backbone.

  1. Extensive experiments on PASCAL-Context, NYUD-v2, and Cityscapes show that D2^{2}MoE achieves state-of-the-art multi-task performance, delivering the highest aggregated gains across benchmarks while requiring substantially fewer trainable parameters.

The remainder of this paper is structured as follows. Sec. 2 surveys related works on multi-task dense prediction, vision foundation models, and mixture of experts. Sec. 3 introduces D2^{2}MoE, including the Mixture of Task Adapters and the Semantic Prior Injector for prior-guided decoding. Sec. 4 presents the experimental protocol and results on PASCAL-Context, NYUD-v2, and Cityscape. Sec. 5 concludes with key findings and potential implications.

Related Works

Multi-task Dense Prediction

Multi-task dense prediction, which jointly addresses semantic segmentation, depth estimation, and auxiliary tasks for scene understanding, has substantively progressed via relational modeling, efficient architectures, and generative formulations [1, 2, 47, 56, 57, 58, 59, 60, 61, 62, 63]. ATRC [64] explicitly formalizes inter-task dependencies to improve transferable generalization. MTPSL [65] tackles incomplete annotations in a partially-supervised setting, enabling resilient learning under partial labels. Transformer-based designs such as the InvPT enable hierarchical feature fusion and granular scene parsing [2, 31]. Efficiency has been advanced by cross-task distillation, epitomized by KDAM [66]. MLoRA employs a mixture of low-rank experts for parameter-efficient, per-task routing that scales to heterogeneous dense outputs [34]. Synergy- and compactness-oriented approaches include SEM, which promotes salutary task interactions beyond naive sharing [67], and Bi-MTPD, which binarizes predictors for compact models with negligible performance loss [68]. State-space innovations such as MTMamba [37] capture long-range dependencies for enhanced multi-task understanding, whereas MTMamba++ [38] streamlines decoding to improve efficiency. BridgeNet [42] introduces bridge embeddings to facilitate comprehensive cross-task feature interactions. Collectively, these lines of work converge on holistic, adaptive and resource-conscious frameworks. Collectively, these lines of work converge holistic and resource-conscious frameworks.

Parameter Efficient Multi-task Adaption

Parameter-efficient multi-task adaptation studies how to transfer pretrained models to multiple downstream tasks with limited trainable parameters, making it complementary to MTDP-specific architectures. Polyhistor [49] uses decomposed hypernetworks and layer-wise scaling kernels to share knowledge without full task-wise fine-tuning. VMT-Adapter [50] introduces a once-for-all adapter that combines shared cross-task knowledge with lightweight task-specific extraction, yielding nearly task-number-independent cost. TADFormer [51] performs task-aware adaptation through parameter-efficient prompting and dynamic task filters conditioned on task contexts. DiTask [52] preserves pretrained weight geometry via diffeomorphic transformations of singular values for shared and task-specific modulation. These methods show that separating shared adaptation from task-specific modulation is effective for multi-task transfer. Different from them, D2^{2}MoE targets frozen VFM s for MTDP by coupling global TG-LoRA alignment with sparse TA-MoE specialization at late-stage attention projections, where heterogeneous dense objectives most strongly diverge.

Vision Foundation Models

Vision foundation models (VFMs) have revolutionized computer vision by learning generalizable representations from vast unlabeled datasets, enabling transfer to diverse downstream tasks [69]. Pioneering architectures like Vision Transformers (ViT) [70] scaled transformers for image processing, achieving state-of-the-art performance on benchmarks such as ImageNet. Self-supervised approaches, including Masked Autoencoders (MAE) [71], reconstruct masked image patches to capture semantic features without annotations. DINO [28] employs self-distillation to emerge invariant representations, fostering emergent properties like object segmentation. Multimodal models such as CLIP align vision and language embeddings for zero-shot classification and retrieval. Subsequent iterations like DINOv2 [29] refine training with curated data and improved objectives, yielding robust features across scales. The Segment Anything Model (SAM) [72] introduces promptable segmentation for arbitrary objects, advancing interactive vision applications. Most recently, DINOv3 [30] extends these capabilities with enhanced multi-task potential through advanced self-supervision techniques. Accordingly, this work targets prior-preserving, parameter-efficient task-conditioned adaptation of VFMs for multi-task dense prediction.

Mixture of Experts

Mixture of experts (MoE) has emerged as a de facto paradigm for conditional computation: a router sparsely activates a small subset of experts per token/sample, increasing capacity without proportional FLOPs. Foundational designs such as Switch Transformer [40] and GLaM [41] establish scalable sparse routing, while Expert-Choice routing [45] and DeepSpeed-MoE [73] show that routing policies and systems support are central to utilization and efficiency.

In vision and multimodal foundation models, MoE is used to reduce domain/task interference while retaining scalable capacity. V-MoE [44] adapts sparse experts to ViTs, LIMoE [74] shows modality-aware partitioning under contrastive learning, and Uni-Perceiver-MoE [75] integrates conditional experts into unified generalist architectures. More recent multimodal variants route over heterogeneous expert pools for generalist MLLMs or vision experts (MoME [76], MoVA [77]), with Uni-MoE [78] further systematizing scaling in unified multimodal LLMs.

MoE is also effective for multi-task learning and parameter-efficient specialization, where conditional routing allocates capacity to tasks with higher complexity or conflict. Vision MTL approaches route along task-conditioned expert pathways with varied sharing/gating (M³ViT [53], Mod-Squad [54], AdaMV-MoE [55], TaskExpert [33]), and MLoRE [34] scales experts for dense prediction via low-rank parameterization. Recent work reframes MoE as adapter-style tuning: MoE-Adapters [79] sparsely selects from a fixed adapter pool (with OOD fallback), and MoE-Adapters++ [80] introduces dynamic adapters and learned expert selection. Inspired by adapter-style MoE, we route among lightweight task adapters atop frozen DINOv3 features, concentrating specialization in small modules while preserving backbone transferability.

Methodology

This section presents D²MoE, a parameter-efficient framework for adapting frozen vision foundation models to multitask dense prediction. § 3.1 first provides an overview of the pipeline, including the frozen DINOv3 backbone, the Mixture of Task Adapters (MoTA), and the Semantic Prior Injector (SPI). § 3.2 then details MoTA, which decouples shared adaptation from task-conditioned specialization. Finally, § 4 introduces SPI for prior-guided multi-scale decoding.

Framework Overview

Figure 3 illustrates the overall architecture of D²MoE. Given an input image I∈RH×W×3I \in\mathbb{R}^{H \times W \times3}, we employ a frozen DINOv3 backbone as the shared feature extractor. The backbone provides two types of representations: task-agnostic semantic priors {Si}i=14\{S_i\}_{i=1}^{4}, which retain the pretrained foundation knowledge, and task-adapted features {Fit}i=14\{F_i^t\}_{i=1}^{4} for each downstream task t∈{1,…,T}t \in\{1,\ldots,T\}.

Overall architecture of D²MoE

Figure 3. Overall architecture of D²MoE. § 3.1 Built upon a frozen DINOv3 backbone, D²MoE organizes MoTA across four stages to preserve shared priors through task-generic adaptation and introduce localized task-specific adaption at stage endings. SPI fuses prior and task features, followed by lightweight heads for multi-task dense predictions.

To adapt the frozen backbone without full fine-tuning, we insert the proposed Mixture of Task Adapters (MoTA) into the attention blocks. MoTA serves as the only backbone-side trainable adaptation path and produces task-aware multi-level features while preserving the shared VFM prior. The adapted features are then passed to the Semantic Prior Injector (SPI), which fuses multi-level task features with frozen semantic priors for prior-guided decoding. Finally, lightweight task-specific heads map the refined representations to the corresponding prediction spaces, yielding task-wise outputs {Ot}t=1T\{O_t\}_{t=1}^{T}.

Mixture of Task Adapters

MoTA is designed to address the single-route adaptation bottleneck in frozen VFM-based multi-task dense prediction. Instead of forcing all tasks to share an identical adaptation subspace, MoTA factorizes backbone adaptation into two complementary components: a shared low-rank pathway for stable task-generic alignment, and a sparse task-conditioned pathway for resolving task-specific conflicts. This design allows D²MoE to retain the semantic priors of the frozen DINOv3 while introducing lightweight conditional capacity only where task divergence needs to be modeled. The following sections describe the two components in detail.

Task-Generic Low-Rank Adaptation

To obtain parameter-efficient transfer while preserving the semantic geometry encoded by the DINOv3 backbone, we introduce Task-Generic Low-Rank Adaptation (TG-LoRA) inside the self-attention modules. In contrast to full fine-tuning, TG-LoRA keeps all pretrained backbone weights frozen and learns only structured low-rank increments on a small set of high-impact linear maps. This constrained re-parameterization provides a stable, task-agnostic adaptation channel that aligns the foundation representation to downstream data distributions with reduced risk of representation drift and cross-task interference.

TG-LoRA is inserted exclusively into the query, key, value projections and the attention output projection of each Transformer block. Let X∈RN×CX \in\mathbb{R}^{N \times C} denote the input token matrix (NN tokens, embedding dimension CC). Under a pre-normalization design, we first compute:

X~=LN⁡(X).\widetilde{X} = \operatorname{LN}(X).

For P∈{Q,K,V}P \in\{Q,K,V\}, we express the projection as a frozen base weight plus a rank-rr residual update:

ΔWP=BPAP,(1)\Delta W_P = B_P A_P, \tag*{(1)}
Illustration of the key components in D²MoE

Figure 4. Illustration of the key components in D²MoE. (§ 3.2 and § 4) (a) Task-Generic Adaptation inserts TG-LoRA into attention projections to provide shared, low-rank adaptation while keeping the DINOv3 backbone frozen. (b) Task-Specific Adaptation employs TA-MoE, where task routers select Top-k low-rank experts to produce sparse, task-conditioned updates for attention projections. (c) The Semantic Prior Injector fuses multi-level task features with frozen semantic priors to enhance semantic consistency and structural coherence for dense prediction tasks.

P=X~(WP+sPΔWP),(2)P = \widetilde{X}\left(W_P + s_P\Delta W_P\right), \tag*{(2)}

where WP∈RC×CW_P \in\mathbb{R}^{C \times C} is frozen, while AP∈RC×rA_P \in\mathbb{R}^{C \times r} and BP∈Rr×CB_P \in\mathbb{R}^{r \times C} are learnable with r≪Cr \ll C. The scalar sPs_P controls the effective update magnitude; a common choice is sP=αP/rs_P = \alpha_P/r to keep the perturbation scale approximately stable as rr varies. This can be interpreted as learning a constrained perturbation of the form WP↦WP+sPBPAPW_P \mapsto W_P + s_P B_P A_P without modifying the pretrained weights.

The projected features are then processed by multi-head self-attention. Let hh denote the number of heads and dh=C/hd_h = C/h the per-head dimension. After reshaping into head-wise tensors {Q(m),K(m),V(m)}m=1h\{Q^{(m)}, K^{(m)}, V^{(m)}\}_{m=1}^{h} with Q(m),K(m),V(m)∈RN×dhQ^{(m)}, K^{(m)}, V^{(m)} \in\mathbb{R}^{N \times d_h}, each head computes:

H(m)=Softmax⁡(Q(m)(K(m))Tdh)V(m),m=1,…,h,(3)H^{(m)} = \operatorname{Softmax}\left(\frac{Q^{(m)}\left(K^{(m)}\right)^{\mathsf{T}}}{\sqrt{d_h}}\right)V^{(m)}, \qquad m = 1,\ldots,h, \tag*{(3)}

and the heads are concatenated to form Z=Concat⁡(H(1),…,H(h))∈RN×dZ = \operatorname{Concat}(H^{(1)},\ldots,H^{(h)}) \in\mathbb{R}^{N \times d}.

TG-LoRA is further applied to the attention output projection that maps contextualized tokens back into the shared embedding space:

Z′=Z(WO+sOBOAO),(4)Z' = Z\left(W_O + s_O B_O A_O\right), \tag*{(4)}

where WO∈Rd×dW_O \in\mathbb{R}^{d \times d} is frozen and (AO,BO)(A_O, B_O) are learnable rank-rr adapters. The attention sub-layer output is then obtained through the residual pathway:

Y=X+Z′.(5)Y = X + Z'. \tag*{(5)}

The FFN and LayerScale follow the original DINOv3 design and are kept frozenThe TG-LoRA augmented output is then propagated to the next Transformer block. To preserve the pretrained VFM’s general-purpose representation, semantic priors {Si}i=14\{S_i\}_{i=1}^{4} are extracted from the corresponding unmodified DINOv3 blocks applied to the same input XX, ensuring that the priors reflect task-agnostic backbone features rather than adaptation-induced artifacts.

By restricting TG-LoRA to {WQ,WK,WV,WO}\{W_Q, W_K, W_V, W_O\}, the trainable degrees of freedom primarily modulate (i) the token-to-token interaction geometry via (Q,K)(Q,K) and (ii) the injection of contextual evidence back into the embedding space via (V,WO)(V,W_O), while leaving the typically more representationally disruptive FFN untouched. As a result, TG-LoRA acts as a globally shared, task-generic adaptation mechanism that improves robustness to domain shifts while preserving the semantic organization of the foundation model, providing a stable substrate for subsequent task-adaptive specialization.

Task-Adaptive Mixture of Experts

TG-LoRA provides a shared, task-agnostic adaptation route that preserves the backbone’s semantic prior. Nevertheless, heterogeneous dense prediction objectives often demand incompatible token-interaction patterns at late representation levels, where high-level semantics are consolidated and task divergence becomes most pronounced. To explicitly factorize these late-stage conflicts without fragmenting the backbone, we introduce Task-Adaptive Mixture of Experts (TA-MoE) as a conditional specialization mechanism.

In contrast to MoE designs that distribute experts broadly across backbone layers or postpone task differentiation to decoder heads [53, 54], TA-MoE is injected only into the final self-attention block of each DINOv3 stage and acts directly on the attention linear maps (Q/K/V projections and the output projection). This placement targets the token-mixing operator where task-dependent contextual aggregation is most sensitive, while keeping earlier layers fully shared for stable representation learning.

Let Hs∈RN×CH^s \in\mathbb{R}^{N \times C} denote the token features entering the last attention block of stage ss, where NN is the number of tokens and CC is the channel dimension. For a TT-task setting, we introduce task-indexed inputs {Hs,t}t=1T\{H^{s,t}\}_{t=1}^{T} as notational replicas initialized by broadcasting HsH^s, so that task-conditioned operators can be written uniformly, while the shared backbone features HsH^s are computed only once.

A task-conditioned router predicts a sparse gating distribution over EE candidate experts from a global stage summary. We first compute a compact descriptor via token-wise pooling:

hˉt=Pool⁡(Hs,t)∈RC,(6)\bar{h}^{t} = \operatorname{Pool}(H^{s,t}) \in\mathbb{R}^{C}, \tag*{(6)}

and obtain expert logits using a lightweight task-specific linear map:

at=Rthˉt∈RE,(7)a^{t} = R^{t}\bar{h}^{t} \in\mathbb{R}^{E}, \tag*{(7)}

where Rt∈RE×CR^{t} \in\mathbb{R}^{E \times C}. Sparse activation is enforced by selecting the Top-kk experts (k≪Ek \ll E) and normalizing on the active set:

πt=Softmax⁡(Top⁡-k(at))∈ΔE,(8)\pi^{t} = \operatorname{Softmax}(\operatorname{Top}\text{-}k(a^{t})) \in\Delta^{E}, \tag*{(8)}

where ΔE\Delta^{E} denotes the probability simplex with zeros on inactive experts.

Each expert is implemented as a low-rank residual update. For each attention projection P∈{Q,K,V,O}P \in\{Q, K, V, O\}, the ee-th expert is parameterized by factors AP(e)∈RC×rA_{P}^{(e)} \in\mathbb{R}^{C \times r} and BP(e)∈Rr×CB_{P}^{(e)} \in\mathbb{R}^{r \times C} with rank r≪Cr \ll C. The task-specific residual to the frozen projection WP∈RC×CW_{P} \in\mathbb{R}^{C \times C} is:

ΔWPt=αr∑e=1EπetBP(e)AP(e),(9)\Delta W_{P}^{t} = \frac{\alpha}{r}\sum_{e=1}^{E}\pi_{e}^{t}B_{P}^{(e)}A_{P}^{(e)}, \tag*{(9)}

with scaling factor α\alpha. Accordingly, task-specialized projections are computed as:

Pt=Hs,t(WP+ΔWPt),P∈{Q,K,V}.(10)P^{t} = H^{s,t}(W_{P}+\Delta W_{P}^{t}), \qquad P \in\{Q,K,V\}. \tag*{(10)}

The task-adaptive QtQ^{t}, KtK^{t}, and VtV^{t} are then processed by standard multi-head self-attention:

Zt=MSA⁡(Qt,Kt,Vt)=Softmax⁡(QtKt⊤dh)Vt,(11)Z^{t} = \operatorname{MSA}(Q^{t},K^{t},V^{t}) = \operatorname{Softmax}\left(\frac{Q^{t}K^{t\top}}{\sqrt{d_{h}}}\right)V^{t}, \tag*{(11)}

where dhd_h denotes the per-head dimension (head splitting/concatenation is omitted for brevity). TA-MoE is further applied to the attention output projection to complete a coherent task-conditioned attention pathway:

H~s,t=Zt(WO+ΔWOt),(12)\widetilde{H}^{s,t} = Z^{t}(W_{O}+\Delta W_{O}^{t}), \tag*{(12)}

where WOW_{O} is the frozen output projection and ΔWOt\Delta W_{O}^{t} is the routed low-rank residual defined above.

The resulting tokens are processed by the standard residual connections, feed-forward transformation, and LayerScale of DINOv3, yielding the task-specific stage-wise outputs {Fit}i=14\{F_{i}^{t}\}_{i=1}^{4}. Overall, TA-MoE interprets task divergence as a conditional perturbation of the token-mixing operator: the backbone projections remain frozen to preserve a shared semantic prior, while a sparse mixture of low-rank experts injects task-selective capacity precisely at the late layers where conflicts are most acute. By restricting conditional computation to the final attention block of each stage, we obtain a clean separation between globally shared representation learning and task-specific refinement, achieving effective disentanglement with minimal additional overhead.

Semantic-Prior Guided Decoding

TG-LoRA and TA-MoE adapt the shared backbone for multi-task dense prediction, yet the final outputs still depend on a decoder that maps adapted features to task-specific label spaces. Most VFM-style decoders remain shallow and task-agnostic, typically performing FPN-like fusion without explicitly exploiting the high-level semantic structure already encoded in foundation models. We thus propose a semantic-prior guided decoder that integrates task-adapted features with frozen DINOv3 representations, enabling prior-aware multi-scale aggregation and task-conditioned refinement.

Semantic Prior Injection

For each task tt, let {Xit}i=14\{X_{i}^{t}\}_{i=1}^{4} be the task-adapted features from four backbone stages, where Xit∈RC×H×WX_{i}^{t} \in\mathbb{R}^{C \times H \times W}, and let {Si}i=14\{S_{i}\}_{i=1}^{4} be the semantic priors extracted from DINOv3, with Si∈RC×H×WS_{i} \in\mathbb{R}^{C \times H \times W}. The Semantic Prior Injector (SPI) performs channel alignment, task-conditioned multi-scale fusion, and semantic prior injection.

Since features from all stages share the same spatial resolution and channel dimensions, no explicit spatial rescaling is required. We project both task features and semantic priors CC using independent 1×11 \times1 convolutions:

X~it=ϕX(Xit),(13)\widetilde{X}_{i}^{t} = \phi_{X}(X_{i}^{t}), \tag*{(13)}
S~i=ϕS(Si),(14)\widetilde{S}_{i} = \phi_{S}(S_{i}), \tag*{(14)}

where X~it,S~i∈RC×H×W\widetilde{X}_{i}^{t}, \widetilde{S}_{i} \in\mathbb{R}^{C \times H \times W}. To aggregate task features across scales, we concatenate the aligned multi-scale task features along the channel dimension:

Xcatt=Concat⁡(X~1t,X~2t,X~3t,X~4t)∈R4C×H×W.(15)X_{\mathrm{cat}}^{t} = \operatorname{Concat}\left(\widetilde{X}_{1}^{t},\widetilde{X}_{2}^{t},\widetilde{X}_{3}^{t},\widetilde{X}_{4}^{t}\right) \in\mathbb{R}^{4C \times H \times W}. \tag*{(15)}

We then employ a lightweight fusion head composed of a 1×11 \times1 convolution for channel mixing and compression, followed by batch normalization and a non-linear activation function, and a subsequent 3×33 \times3 convolution to capture local spatial interactions, corresponding to pixel-wise fusion logits for each stage:

Lt=ψ(Xcatt),Lt∈R4×H×W,(16)L^{t} = \psi(X_{\mathrm{cat}}^{t}), \qquad L^{t} \in\mathbb{R}^{4 \times H \times W}, \tag*{(16)}

where ψ(⋅)\psi(\cdot) denotes the convolutional fusion module. A softmax operation is applied along the stage dimension to obtain normalized pixel-wise fusion weights:

Ait(h,w)=exp⁡(Lit(h,w))∑j=14exp⁡(Ljt(h,w)),∑i=14Ait(h,w)=1.(17)A_{i}^{t}(h,w) = \frac{\exp(L_{i}^{t}(h,w))}{\sum_{j=1}^{4}\exp(L_{j}^{t}(h,w))}, \qquad\sum_{i=1}^{4}A_{i}^{t}(h,w)=1. \tag*{(17)}

Finally, the aggregated task feature is computed as a pixel-wise weighted sum over scales:

Xaggt(h,w)=∑i=14Ait(h,w)X~it(h,w).(18)X_{\mathrm{agg}}^{t}(h,w) = \sum_{i=1}^{4} A_{i}^{t}(h,w)\widetilde{X}_{i}^{t}(h,w). \tag*{(18)}

In parallel, we merge semantic priors across stages into a single map. Concretely, we concatenate {S~i}i=14\{\widetilde{S}_{i}\}_{i=1}^{4} along the channel dimension and apply a lightweight fusion operator:

Sagg=ψS(Concat⁡(S~1,S~2,S~3,S~4)),(19)S_{\mathrm{agg}} = \psi_{S}\left(\operatorname{Concat}(\widetilde{S}_{1},\widetilde{S}_{2},\widetilde{S}_{3},\widetilde{S}_{4})\right), \tag*{(19)}

where ψS(⋅)\psi_{S}(\cdot) is implemented as 1×11 \times1 channel mixing followed by a 3×33 \times3 convolution for local context integration.

Given the aggregated task feature XaggtX_{\mathrm{agg}}^{t}, SPI injects semantic priors by directly concatenating the aligned semantic representation with the task feature, followed by joint feature refinement. Specifically, we fuse the task feature and semantic prior as:

Ut=Concat⁡(Xaggt,Sagg),(20)U^{t} = \operatorname{Concat}(X_{\mathrm{agg}}^{t}, S_{\mathrm{agg}}), \tag*{(20)}

where the concatenation is performed along the channel dimension. The fused representation is then processed by a lightweight refinement block to produce the final prior-enhanced feature:

Zt=ψZ(Ut),(21)Z^{t} = \psi_Z(U^{t}), \tag*{(21)}

where ψZ(⋅)\psi_Z(\cdot) is implemented as a Conv–Norm–GeLU–Conv stack.

Task-Specific Decoding Head

Given ZtZ^{t}, we employ a lightweight head that maps the shared latent to the prediction space of task tt:

Y^t=U(Pt(Zt)),(22)\hat{Y}^{t} = U(\mathcal{P}^{t}(Z^{t})), \tag*{(22)}

where Pt(⋅)\mathcal{P}^{t}(\cdot) is a task-dependent 1×11 \times1 projection and U(⋅)U(\cdot) denotes bilinear upsampling to the input resolution.

Joint Optimization Losses

We train all tasks jointly with Ltotal\mathcal{L}_{\mathrm{total}}, which is defined as:

Ltotal=∑t=1TλtLt,(23)\mathcal{L}_{\mathrm{total}} = \sum_{t=1}^{T} \lambda_t \mathcal{L}_t, \tag*{(23)}

where Lt\mathcal{L}_t is task-specific (e.g., cross-entropy for segmentation, ℓ1\ell_1/scale-invariant losses for depth, and cosine/angular losses for normals) and λt\lambda_t balances tasks. We use uniform weighting λt=1\lambda_t = 1 in all experiments.

Experiments

In this section, we evaluate D²MoE for multi-task dense prediction. § 5.1 introduces the benchmarks (Pascal-Context, NYUD-v2, and Cityscapes) and the evaluation protocol, including per-task metrics and the overall multi-task gain Δm\Delta_m. § 5.2 details our training and implementation. § 5.3 reports quantitative and qualitative comparisons with recent state-of-the-art methods. § 5.4 presents ablations studies and analyze the contribution of each component. § 5.5 gives feature visualization and analysis. § 5.6 evaluates robustness.

Experimental Setups.

Datasets

Following recent MTL works [32], [37], [42], we evaluate D²MoE on three widely used benchmarks under standardized protocols, with official splits and their canonical task suites. Pascal-Context [83] contains 4998 training and 5105 test images with rich pixel-level annotations, on which we report semantic segmentation (Sem. Seg.), human parsing (H. Pars.), object edge detection (Edge), surface normal estimation (Norm.), and saliency detection (Sal.). NYUD-v2 [48] focuses on indoor scenes with 795 training and 654 test samples; we evaluate Sem. Seg., monocular depth estimation (Dep.), Norm., and Edge. Cityscapes provides urban street-view imagery with 2975 training images and 500 validation images, including 19 semantic classes and stereo-derived disparity; we follow common practice and report Sem. Seg. together with geometry-related predictions (Dep.) derived from the fine annotations. Together, these datasets cover outdoor scenes, indoor environments, and driving scenarios, enabling a broad and consistent evaluation of multi-task dense prediction under diverse conditions.

Evaluation Metrics

To enable a rigorous, fair and fully comprehensive comparison for multi-task dense prediction, we adopt standard task-specific measures and a normalized cross-task summary metric for overall assessment. Specifically, we report:

  • Sem. Seg. / H. Pars.: mean Intersection-over-Union (mIoU).

  • Dep.: root mean square error (RMSE).

  • Norm.: mean angular error (mErr).

  • Sal.: maximal F-measure (maxF).

  • Edge: Optimal Dataset Scale F-measure (odsF).

To summarize performance across heterogeneous task objectives, the overall performance gain of multi-task learning (Δm\Delta_m) is computed relative to our multi-task baseline:

Δm=1T∑t=1T(−1)ℓtMt−MtmMtm,(24)\Delta_m = \frac{1}{T}\sum_{t=1}^{T}(-1)^{\ell_t}\frac{M_t-M_t^m}{M_t^m}, \tag*{(24)}

where TT is the number of tasks, MtM_t and MtmM_t^m denote the comparing result and our multi-task baseline results for task tt, and ℓt=1\ell_t=1 for lower-is-better metrics (RMSE, mErr) and ℓt=0\ell_t=0 otherwise (mIoU, maxF, ODS-F). A positive Δm\Delta_m therefore indicates an overall improvement. We report all per-task metrics together with Δm\Delta_m on the official dataset splits.

Implementation Details

To enable parameter-efficient multi-task dense prediction, we use DINOv3 [30] as frozen backbone and adapt it with our designed modules. Models are trained with AdamW (weight decay 1×10−61\times10^{-6}) and a polynomial learning rate schedule. Base learning rates are 8×10−58\times10^{-5}, 2×10−52\times10^{-5}, and 1×10−41\times10^{-4} for Pascal-Context, NYUDv2, and Cityscapes, respectively. We train on two NVIDIA A100 GPUs with a batch size of 4 (2 per GPU). Loss functions are detailed in § 4.0.3. TG-LoRA is applied to the QQ, KK, VV, and output layers of all attention blocks with rank 16. TA-MoE is applied in the final attention layer, with each expert as a rank-4 LoRA module. The SPI module has a fused feature embedding dimension of 384. All LoRA adapters use a scaling factor of 4 and a dropout rate of 0.05.

To isolate the effect of our modules from DINOv3 itself, we build the MTL Baseline with the same frozen DINOv3 backbone and identical training protocol as D2^2MoE. This baseline trains only task-specific prediction heads, without TG-LoRA, TA-MoE, or SPI. Thus, its gap to D2^2MoE directly reflects the contribution of our proposed modules.

Benchmark Results

Quantitative Comparison Results

We conduct comprehensive quantitative comparisons on three standard benchmarks, namely Pascal-Context, NYUD-v2, and Cityscapes, using the official data splits and widely adopted multi-task settings for fair evaluation. We report task-specific metrics for each benchmark and additionally summarize overall performance using the aggregated multi-task gain Δm\Delta_m, which facilitates a holistic comparison across heterogeneous dense prediction tasks.

Pascal-Context. Table 1 summarizes results on PASCAL-Context across five dense prediction tasks including semantic segmentation (Sem. Seg.), human parsing (H. Pars.), saliency detection (Sal.), surface normal estimation (Norm.), and edge detection (Edge) under the standard multi-task setting. D2^2MoE achieves the highest aggregated multi-task gain (Δm=8.88\Delta_m=8.88). Relative to the MTL Baseline (ViT-L), it improves Sem. Seg. (82.61→84.88 mIoU), H. Pars. (76.38→78.05 mIoU) and Sal. (77.09→85.55 maxF), while reducing Norm. error (17.63→13.57 mErr) and increasing boundary accuracy (74.00→78.40 odsF). The concurrent gains across recognition-oriented and geometry-sensitive tasks suggest that D2D^{2}MoE mitigates negative transfer while preserving beneficial sharing. Compared with decoder-centric methods such as MTMamba++ and BIM (Swin-L), D2D^{2}MoE attains higher performance on segmentation and parsing and a larger overall gain (8.88 versus 6.15 and 5.99). Among ViT-L-based approaches, D2D^{2}MoE also improves upon MLoRE, TaskDiffusion and BridgeNet, consistent with the view that task-adaptive routing, coupled with the semantic priors provided by DINOv3, supports stronger task disentanglement and more effective cross-task coordination.

MethodBackboneSem. Seg. mIoU↑H. Pars. mIoU↑Sal. maxF↑Norm. mErr↓Edge odsF↑Δm\Delta_m (%) ↑Para. (M)↓Train. Para. (M)↓
MTL BaselineViT-L82.6176.3877.0917.6374.000.0032825
PAD-Net [58] [CVPR2018]HRNet-1853.6059.6065.8015.3072.50-12.118181
MTI-Net [60] [ECCV2020]HRNet-1861.7060.1884.7814.2370.80-4.32128128
ATRC [64] [ICCV2021]HRNet-1867.6762.9382.2914.2472.42-2.379696
ASTMT [81] [ECCV2024]ResNet-5068.0061.1265.7114.6872.40-7.574949
MTMamba [37] [ECCV2024]Swin-L81.1172.6284.1414.1478.805.74336336
MTMamba++ [38] [TPAMI2025]Swin-L81.9472.8785.5614.2978.606.15343343
BIM [39] [ICCV2025]Swin-L81.2573.1584.1414.1379.055.99388388
InvPT [31] [ECCV2022]ViT-L79.0367.6184.8114.1573.002.52423423
InvPT++ [32] [TPAMI2024]ViT-L80.4469.1984.8513.7074.204.12421421
TaskPrompter [2] [ICLR2023]ViT-L80.8968.8984.8313.7273.503.93401401
TaskExpert [33] [ICCV2023]ViT-L80.6469.4284.8713.5674.204.39420420
MLoRE [34] [CVPR2024]ViT-L81.4170.5284.9013.5175.425.26407407
TaskDiffusion [35] [ICLR2025]ViT-L81.2169.6284.9413.5574.894.80416416
BridgeNet [42] [TPAMI2025]ViT-L80.6470.0684.6413.8272.963.87477477
SAK [82] [ICLR2025]ViT-L84.0176.9984.6513.8276.277.40407407
D²MoE [Ours]ViT-L84.4878.0585.5513.5778.408.8835854

Table 1. Performance comparison on the PASCAL-Context datasets. ↑/↓ indicates that the higher/lower results represent better performance.

NYUD-v2. Table 2 reports quantitative results on NYUD-v2 for Sem. Seg, Dep., Norm. and Edge. D2D^{2}MoE achieves the highest aggregated multi-task gain (Δm=11.34\Delta_m = 11.34), with improvements observed across all tasks. Relative to the MTL baseline (ViT-L), performance increases from 63.15 to 65.91 mIoU for Sem. Seg and from 75.10 to 80.60 odsF for Edge, while Dep. RMSE is reduced from 0.5119 to 0.4209 and Norm. from 18.81 to 15.82 mErr. Compared with recent strong baselines (e.g., BIM, TaskDiffusion), D2D^{2}MoE shows a larger overall gain, with particularly pronounced benefits on geometry-related predictions, consistent with the importance of modelling indoor structure and fine-grained details.

MethodsBackboneSem. Seg. mIoU↑Dep. RMSE↓Norm. mErr↓Edge odsF↑Δm\Delta_m (%) ↑
MTL BaselineViT-L63.150.511918.8175.100.00
MTI-Net [60] [ECCV2024]HRNet-1845.970.536520.2777.84-9.03
PAD-Net [58] [CVPR2018]HRNet-1836.610.627020.8576.38-18.41
ATRC [64] [ICCV2021]HRNet-1846.330.536320.1877.94-8.73
ASTMT [81] [ECCV2024]ResNet-5032.160.570023.1874.50-21.11
MTMamba [37] [ECCV2024]Swin-L55.820.506618.6378.70-1.21
MTMamba++ [38] [TPAMI2025]Swin-L57.010.481818.2779.401.19
BIM [39] [ICCV2025]Swin-L57.400.473318.5578.721.16
InvPT [31] [ECCV2024]ViT-L53.560.518319.0478.10-3.42
InvPT++ [32] [ECCV2024]ViT-L53.890.509118.6578.10-2.32
TaskPrompter [2] [ECCV2024]ViT-L55.300.515218.4778.20-1.78
TaskExperts [33] [ECCV2024]ViT-L55.350.515718.5478.40-1.82
MLoRE [34] [ECCV2024]ViT-L55.960.507618.3378.43-0.89
TaskDiffusion [35] [ICLR2025]ViT-L55.650.502018.4378.64-0.80
BridgeNet [42] [TPAMI2025]ViT-L55.510.493018.4778.22-0.61
SAK [82] [ICLR2025]ViT-L63.180.431316.2579.438.79
D2^2MoE [Ours]ViT-L65.910.420915.8280.6011.34

Table 2. Performance comparison on the NYUDv2 dataset. ↑/↓ indicates that the higher/lower results represent better performance.

Cityscapes. We further evaluate D2D^{2}MoE on Cityscapes for joint Sem. Seg. and Dep. (Table 3). D2D^{2}MoE attains 82.10 mIoU and 4.22 RMSE, corresponding to a Δm=5.30\Delta_m = 5.30 overall gain with the ViT-L baseline (76.45 mIoU and 4.36 RMSE). Notably, several existing approaches yield limited or negative overall gains under the same protocol, reflecting the optimisation trade-offs between semantic recognition and geometric estimation in urban driving scenes. We further observe that the relative advantage of D2D^{2}MoE is more evident on Cityscapes than on NYUD-v2 and Pascal-Context. This may stems from its regular driving layouts and tight semantic–geometric coupling, where roads, vehicles, and buildings follow stable spatial/depth patterns, making DINOv3 priors and task-aware routing more effective.

MethodsBackboneSem. Seg. mIoU↑Dep. RMSE↓Δm\Delta_m (%) ↑
MTL BaselineViT-L76.454.360
MTI-Net [60]HRNet-1853.195.05-23.13
PAD-Net [58]HRNet-1859.855.06-18.88
MTMamba [37]Swin-L78.004.66-2.43
MTMamba++ [38]Swin-L79.134.63-1.34
InvPT [31]ViT-L71.784.67-6.61
TaskPrompter [2]ViT-L72.415.49-15.60
D2^2MoEViT-L82.104.225.30

Table 3. Performance comparison on the Cityscapes dataset.

Across Pascal-Context, NYUD-v2 and Cityscapes, D2D^{2}MoE consistently yields the largest aggregated multi-task gain Δm\Delta_m, despite substantial differences in visual domains and task compositions. This pattern strongly supports our motivation that effective MTDP with a frozen foundation backbone requires (i) task-generic adaptation to align pretrained representations with dense prediction, and (ii) sparse, task-adaptive specialization to resolve conflicts.

Qualitative Comparison

We present qualitative results on Pascal-Context and NYUD-v2 to visually assess D2D^{2}MoE. Figure 5 and Figure 6 compare D2D^{2}MoE with strong recent baselines (TaskDiffusion and MTMamba) under the same multi-task setting.

Qualitative comparison on Pascal-Context showing Image, Edge, H. Pars., Sal., Norm., and Sem. Seg. results for GroundTruth, TaskDiffusion, MTMamba, and Ours

Figure 5. Qualitative Comparison on Pascal-Context. Compared with TaskDiffusion [35] and MTMamba [37], D2D^{2}MoE produces crisper boundaries and fewer cross-task inconsistencies (e.g., aligned silhouettes and object extents across parsing/segmentation/saliency and normals), indicating improved shared structure modeling and reduced task interference in multi-objective dense prediction.

Qualitative comparison on NYUD-v2

Figure 6. Qualitative Comparison on NYUD-v2. Compared with TaskDiffusion [35] and MTMamba [37], D2^{2}MoE better preserves depth/normal discontinuities at object boundaries and yields structurally coherent geometry that agrees with segmentation, suggesting stronger cross-task geometric consistency across heterogeneous dense objectives.

Pascal-Context. As shown in Figure 5, D2D^{2}MoE yields more accurate and spatially coherent predictions across tasks. For Sem. Seg., D2D^{2}MoE better preserves object extent and thin structures (e.g., limbs), reducing the label fragmentation observed in competing methods. For Norm., D2D^{2}MoE produces smoother orientation fields with sharper transitions at geometric discontinuities, indicating stronger local shape consistency. For Edge, D2D^{2}MoE outputs sharper and more continuous contours aligned with object silhouettes. For Sal., D2D^{2}MoE improves foreground–background separation with fewer spurious activations.

NYUD-v2. Figure 6 demonstrates gains in indoor scene parsing. D2D^{2}MoE yields Depth maps with refined geometry and reliable occlusion ordering, reducing planar noise. Norm. maps exhibit spatial consistency with crisp boundary transitions. For Sem. Seg., D2D^{2}MoE successfully disambiguates adjacent objects via DINOv3 semantic priors. Finally, Edge results show improved completeness and connectivity even in cluttered environments.

PCA visualizations on Pascal-Context and NYUD-v2

Figure 7. PCA Visualizations on Pascal-Context (Left) and NYUD-v2 (Right). PCA visualizations are provided to analyze feature representations learned with TA-MoE, TG-LoRA, and the Semantic Prior Injector (SPI). As shown, features extracted from different network stages exhibit progressively clearer task-specific structures, demonstrating the effectiveness of the proposed modules in enhancing task-aware representation learning across both datasets.

Ablation Study

We conduct comprehensive ablation studies on the Pascal-Context dataset. In particular, § 5.4.1 isolates the contributions of our proposed modules and their placement, whereas § 5.4.2 examines the key hyperparameters that operationalize the intended balance between conditional specialization and parameter efficiency.

Component and Design Ablations

Overall Components Analysis. Table 4 disentangles the contributions of the Mixture of Task Adapters (MoTA) and Semantic Prior Injector (SPI) to our prior-preserving multitask framework. Relative to the MTL baseline, MoTA alone increases the aggregated gain from 0.00 to 6.10%, with marked improvements in Sal. (82.61 → 84.12 mIoU) and Norm. (15.60 → 14.01 mErr), supporting the need for task-conditioned adaptation to separate dense objectives without perturbing the frozen backbone. SPI alone provides a smaller gain (Δm=5.66%\Delta_m = 5.66\%), implying that semantic priors are insufficient to resolve cross-task interference under a single shared adaptation path. Enabling both yields the best performance, i.e.84.48 mIoU (segmentation), 78.05 mIoU (parsing), 85.55 maxF (saliency), and Δm=8.88%\Delta_m = 8.88\%, consistent with complementarity: MoTA supplies controlled specialization, whereas SPI re-injects frozen semantic structure to improve global coherence and boundary fidelity.

MoTASPISem. Seg. mIoU↑H. Pars. mIoU↑Sal. maxF↑Norm. mErr↓Edge odsF↑Δm\Delta_m (%) ↑
✗✗82.6176.3877.0917.6374.000.00
✗✓83.2777.2182.5115.2478.305.66
✓✗82.4474.9084.1214.0176.206.10
✓✓84.4878.0585.5513.5778.408.88

Table 4. Overall ablation of the proposed components (MoTA and SPI) under the unified multi-task setting.

Ablation Results of MoTA. Table 5 probes MoTA’s internal factorization. Removing TG-LoRA lowers the gain to 7.74%, indicating degradation of the task-generic alignment channel that stabilizes downstream adaptation; removing TA-MoE produces a comparable reduction (Δm=7.97%\Delta_m = 7.97\%), reflecting weaker task-specific discrimination without conditional expert routing. The full model outperforms both variants across tasks, showing that frozen-backbone multi-task learning benefits from jointly coupling a globally shared low-rank substrate (TG-LoRA) with sparse task-adaptive specialization

ModulesSem. Seg. mIoU↑H. Pars. mIoU↑Sal. maxF↑Norm. mErr↓Edge odsF↑Δm\Delta_m (%) ↑
MTL Baseline82.6176.3877.0917.6374.000.00
w/o TG-LoRA84.0777.9284.3214.1578.307.74
w/o TA-MoE83.3177.6084.3513.6578.007.97
Ours84.4878.0585.5513.5778.408.88

Table 5. Component ablation of Mixture of Task Adapters (MoTA), isolating the contributions of TG-LoRA and TA-MoE under the unified multi-task setting.

Δm\Delta_m is computed relative to the MTL baseline, which is the

(TA-MoE) rather than relying on either uniform adaptation or specialization alone.

Ablation Results of TA-MoE. Table 6 evaluates where specialization most effectively alleviates the single-route bottleneck. Injecting TA-MoE into attention QKV projections outperforms insertion solely in the FFN or projection layers, indicating that conditioning token mixing (task-dependent contextual aggregation) is more effective than modifying downstream feed-forward transformations. Combining QKV and projection layers achieves the maximum gain (8.88%), suggesting that co-conditioning attention and output projection best supports task disentanglement.

QKVProjFFNSem. Seg mIoU↑H. Pars. mIoU↑Sal. maxF↑Norm. mErr↓Edge odsF↑Δm\Delta_m (%) ↑
✗✗✗83.3177.6084.3513.6578.007.97
✓✗✗83.7477.8085.1813.5778.708.62
✗✓✗83.3177.6784.8613.7378.808.24
✗✗✓83.3477.2484.6313.5878.008.03
✓✓✗84.4878.0585.5513.5778.408.88
✓✓✓83.7577.5185.0213.5179.108.68

Table 6. Effect of inserting the MoE module at different locations within the backbone transformer block.

Backbone Variants. Table 7 quantifies task-dependent scaling across ViT backbones. ViT-S to ViT-B gives the largest segmentation jump (80.62 →\rightarrow 85.25 mIoU) with modest parameter growth. Scaling to ViT-L shifts gains to structured prediction: H. Pars. 75.66 →\rightarrow 78.05 mIoU, Sal. and Norm. peak at 85.55 maxF and 13.57 mErr, while Sem. Seg. saturates at 84.48 mIoU. ViT-H adds slightly for H. Pars./Edge (78.87 mIoU, 79.60 odsF) but degrades Sal. (83.17 maxF) and Norm. (13.62 mErr), implying diminishing or negative returns from additional backbone capacity. This degradation likely arises because the larger ViT-H encodes stronger high-level semantic invariances, while our frozen-backbone and low-rank adaptation cannot fully recalibrate its spatial-frequency bias for saliency and normals, making these geometry-sensitive tasks prone to over-smoothing. Overall, ViT-L provides the optimal accuracy-efficiency trade-off.

BackbonePara. (M)Sem. Seg. mIoU↑\uparrowPars. mIoU↑\uparrowSal. maxF↑\uparrowNorm. mErr↓\downarrowEdge odsF↑\uparrow
ViT-S62.080.6273.3685.2813.7976.40
ViT-B126.685.2575.6684.9513.6677.50
ViT-L357.684.4878.0585.5513.5778.40
ViT-H911.384.8078.8783.1713.6279.60

Table 7. Comparison of backbone variants under the same multi-task setting.

Comparison with MLoRE under a Strong Backbone. Table 8 reimplements MLoRE on DINOv3. MLoRE places experts in the decoder, improving task outputs over the MTL baseline and substantiating decoder-side specialization. D2^2MoE instead instantiates experts in the backbone, decoupling tasks during representation learning while retaining DINOv3 priors through conditional adaptation. This shift delivers larger mean gains (Δm\Delta_m=8.88% vs. 6.30%) with substantially fewer trainable parameters (54M vs. 407M). Although D2^2MoE incurs higher FLOPs/latency than MLoRE (1029.9G/224.2ms vs. 829.9G/203.2ms), the additional computation is moderate and leads to stronger multi-task perfor… mance with a 7.5×\times smaller trainable footprint, demonstrating a favorable trade-off among accuracy, trainable footprint, and computational cost.

ModulesSem. Seg. mIoU↑H. Pars. mIoU↑Sal. maxF↑Norm. mErr↓Edge odsF↑Δm\Delta_m (%) ↑Para. (M) ↓Train. Para. (M) ↓Flops (G)↓Latency (ms)↓
MTL Baseline82.6176.3877.0917.6374.000.0032825759.7198.9
MLoRE [34]83.8578.0584.9815.5178.106.30407407829.9203.2
D2^{2}MoE84.4878.0585.5513.5778.408.88358541029.9224.2

Table 8. Comparison with MLoRE using a DINOv3 backbone.

Comparison with Other Variants under Matched Budgets. Tab. 9 compares three parameter- and protocol-matched controls under the same frozen DINOv3 setting. Shared LoRA uses a single task-shared LoRA (r=171r = 171) at the same QKV/projection positions, achieving Δm=8.48%\Delta_m = 8.48\%. Task LoRA replaces routing with independent task-wise LoRA modules (r=186r = 186), yielding Δm=8.31%\Delta_m = 8.31\%. Decoder MoE moves the low-rank MoE to the decoder, using the same expert pool and Top-8 routing with 25.2M MoE parameters, but reaches only Δm=5.90%\Delta_m = 5.90\%. In comparison, D2D^{2}MoE achieves the best Δm=8.88%\Delta_m = 8.88\%. Overall, these matched controls show that the gains of D2D^{2}MoE are not explained by parameter count alone, but arise from combining shared low-rank alignment with sparse, learned task-aware specialization at appropriate backbone locations.

Hyperparameter Analysis

We quantify how hyperparameters shape the capacity–efficiency trade-off in Figure 8, varying TG-LoRA rank, TA-MoE expert count, TA-MoE LoRA rank, and Top-kk routing.

Hyperparameter analysis of TG-LoRA and TA-MoE

Figure 8. Hyperparameter analysis of TG-LoRA and TA-MoE. The results demonstrate clear performance trends across different LoRA ranks and expert configurations, highlighting an effective trade-off between model capacity and multi-task performance.

LoRA Rank of TG-LoRA. Increasing TG-LoRA rank from 2 to 16 (Figure 8(a)) improves all tasks (Sem. Seg. 83.40 → 84.48 mIoU, H. Pars. 77.66 → 78.05 mIoU, Sal. 84.73 → 85.55 maxF). Consistently, Δm\Delta_m increases from 8.07% to 8.88%, indicating stronger overall multi-task performance. All parameters rise modestly (355.5M → 357.6M). At rank 32, gains saturate or reverse (e.g., Sem. Seg. 84.35) despite higher parameters/FLOPs, indicating over-parameterization without improved alignment.

Number of Experts in TA-MoE. Varying experts (Figure 8(b)) shows a similar saturation: 2→16 experts improves segmentation (83.47 → 84.48 mIoU) and boundary detection (78.30 → 78.50 odsF) with limited parameter growth (355.9M → 357.6M). The aggregated gain peaks at 16 experts with Δm=8.88%\Delta_m = 8.88\%, suggesting the best balance between specialization capacity and efficiency. At 32 experts, performance saturates or slightly decreases (e.g., parsing 78.05 → 77.64), suggesting redundancy rather than stronger task specialization.

LoRA Rank of TA-MoE. Figure 8(c) indicates that moderate per-expert rank is sufficient: rank 4 achieves strong overall accuracy (segmentation 84.48 mIoU) and the lowest normal error (13.57 mErr) at 357.6M parameters. It also obtains the highest aggregated gain (Δm=8.88%\Delta_m = 8.88\%), confirming that moderate rank is sufficient for overall multi-task capacity. Lower ranks underfit task-specific refinement, whereas higher ranks increase parameters/FLOPs with marginal, unstable gains.

Top-kk Value in TA-MoE. Figure 8(d) shows that increasing kk from 2 to 8 improves performance, peaking at 84.48 mIoU (segmentation) and 85.55 maxF (saliency) with unchanged parameters. Δm\Delta_m also reaches the best value of 8.88% when k=8k = 8. Larger kk (e.g., 12 or 16) yields marginal gains or mild degradation, consistent with reduced sparsity-driven specialization under broader expert activation.

Collectively, these ablations identify a coherent regime where improvements arise from structured task-adaptive design, including TG-LoRA alignment, sparse TA-MoE specialization, and SPI prior injection, rather than from indiscriminate increases in trainable capacity.

Feature Visualization and Analysis

Feature Embedding Visualization (t-SNE). Figure 9 shows 3D t-SNE projections of the learned feature embeddings, directly comparing the MTL baseline (top row) with D2D^{2}MoE (bottom

3D t-SNE visualizations of feature embeddings

Figure 9. 3D t-SNE Visualizations of Feature Embeddings. We compare the MTL baseline (top) with our D2D^{2}MoE framework (bottom) across three benchmarks. (a) Semantic Segmentation: Points are colored by class label. D2D^{2}MoE achieves significantly higher intra-class compactness and inter-class separability than the baseline. (b) Depth Estimation: Points are colored by discretized depth intervals (see legend). Our model yields more structured, monotonic groupings of scene geometry. Axes denote the three primary t-SNE components.

row). D2^{2}MoE reshapes the embedding manifold into task-aligned submanifolds rather than an entangled cloud. On Sem. Seg. (Pascal-Context, NYUD-v2, Cityscapes), D2^{2}MoE forms tighter, well-separated clusters with higher intra-class compactness and reduced cross-class overlap than the MTL baseline, reflecting increased inter-class variance and less negative transfer. This behavior matches selective routing within a shared backbone: early layers preserve broadly reusable cues (edges, textures), while routed experts enhance class-conditional directions, widening margins between semantic modes while maintaining the global manifold structure. Regarding depth estimation, D2^{2}MoE yields clearer and more structured groupings, suggesting routing induces a more geometry-aware embedding where features are organized topologically according to their respective depth bins, a marked improvement over the disordered feature clouds observed in the baseline.

MethodSem. Seg. mIoU↑H. Pars. mIoU↑Sal. maxF↑Norm. mErr↓Edge odsF↑Δm\Delta m (%) ↑Backbone Train. (M)↓Decoder Train. (M)↓Train. Para. (M)↓Para. (M)↓
MTL Baseline82.6176.3877.0917.6374.000.0025.2025.2328.4
Shared LoRA83.6077.5285.1913.5178.308.4825.229.254.4357.6
Task LoRA84.2577.9584.6513.7778.308.3125.229.254.4357.6
Decoder MoE83.2278.0082.4615.1678.205.90054.454.4357.6
D2^{2}MoE84.8878.0585.5513.5778.408.8825.229.254.4357.6

Table 9. Comparison with shared LoRA, task-specific LoRA and decoder-side MoE under matched trainable-parameter budgets.

Router Correlation Analysis. Figure 10 reveals stage- and operator-specific specialization. In QKV projections, early stages display strong cross-task coupling (off-diagonal correlations), whereas deeper stages become near diagonal-dominant, implying D2^{2}MoE progressively privatizes attention access patterns (queries/keys/values) to mitigate negative transfer in information selection. Conversely, attention output projections exhibit structured, non-monotonic sharing: an early block structure (notably among Hum./Nor./Sem.) persists, and late-stage correlations remain strong for specific pairs (e.g., Nor.–Sem., Hum.–Sal.), indicating D2^{2}MoE avoids forced full separation and reuses experts when supervision

Router visualization of TA-MoE across backbone stages

Figure 10. Router visualization of TA-MoE across backbone stages. We visualize the task-wise expert selection patterns of TA-MoE inserted at different backbone stages and positions. For each task, we aggregate expert activation values over all samples in the Pascal-Context dataset and compute the cosine similarity between task-specific activation vectors. The resulting similarity matrices reveal how task relationships evolve across stages and insertion locations.

is complementary. Overall, TA-MoE specializes where tasks attend (QKV) while selectively sharing what is read out (output projection), disentangling conflicts while enabling transfer when tasks align.

Robustness to Distribution Shift

We assess domain generalization on Pascal-Context under the corruption benchmark of ImageNet [84]. Figure 11 reports corruption-wise performance across severities (1/3/5), and Figure 12 visualizes level-5 out-of-distribution (OOD) predictions against TaskDiffusion [35] and MTLTamba [37]. To separate corruption sensitivity from gains due to higher clean accuracy, Figure 13 additionally reports clean-normalized performance retention, while Table 10 reports severity-5 results averaged over all 15 corruptions.

Corruption Robustness Evaluation on Pascal-Context

Figure 11. Corruption Robustness Evaluation on Pascal-Context. Across five tasks, performance decreases with corruption severity, with zoom blur and gaussian noise inducing the largest degradations (e.g., Sem. Seg. drops from 84.48 on clean to 47.50 at Level 5). Weather/digital corruptions are comparatively mild at Levels 1–3, while normal error rises primarily under noise.

Corruption robustness comparison examples

Figure 12. Corruption Robustness Comparison with TaskDiffusion [35]. All examples are drawn from the Pascal-Context dataset under level-5 corruptions. Our method demonstrates stronger robustness to severe degradations, yielding cleaner object boundaries and more stable predictions.

MethodSem. Seg. mIoU↑H. Pars. mIoU↑Sal. maxF↑Norm. mErr↓Edge odsF↑
MTL Baseline65.3664.5470.0822.4563.07
TaskDiffusion57.0149.4867.4222.8159.04
D2^{2}MoE68.0867.0372.7721.4764.96

Table 10. Averaged robustness comparison on PASCAL-Context across all corruptions.

Figure 13. Clean-normalized Performance Retention under Corruption. Task-wise retention for the MTL baseline, D2^{2}MoE, and TaskDiffusion. Avg. Δ\Delta denotes mean relative degradation from clean performance.

Across all tasks, the dominant trend in Figure 11 is graded performance change with increasing corruption severity, indicating stable behavior under covariate shift. Noise corruptions are consistently the most damaging: Sem. Seg. drops from 82.34 (Gaussian, level-1) to 47.50 (level-5), and H. Pars. from 75.98 to 58.75 under the same shift, while surface-normal error increases smoothly from 16.52 (clean) to 33.42 (Gaussian, level-5). In the blur family, zoom blur is the primary failure case across tasks (e.g., Sem. Seg. 71.25 → 47.60, H. Pars. 65.62 → 49.67, Edge 53.20 → 40.10 from level-1 to level-5), whereas defocus/motion blur produce substantially smaller degradations. Weather and digital corruptions are comparatively benign for most tasks: for Sem. Seg., level-5 contrast remains close to clean (81.95 vs. 84.48), and saliency stays high across many level-5 weather/digital shifts, demonstrating broad OOD robustness beyond a single corruption type.

Table 10 shows that D2^{2}MoE achieves the best average corrupted performance across all five tasks, outperforming both the MTL baseline and TaskDiffusion. Figure 13 provides a complementary normalized comparison. D2^{2}MoE shows smaller average degradation than TaskDiffusion at all severities (−5.02% / −12.89% / −24.69% vs. −5.36% / −13.72% / −27.62%), with stronger retention particularly on Sem. Seg. and H. Pars., whereas the MTL baseline exhibits smaller overall degradation (−1.89% / −9.39% / −17.52%). Thus, D2^{2}MoE’s superior corrupted performance stems from both stronger clean performance and improved retention over TaskDiffusion, rather than uniformly lower corruption sensitivity.

Figure 12 qualitatively confirms these trends under severe shift. Under contrast change and defocus blur, D2^{2}MoE preserves coherent region topology in Sem. Seg. and H. Pars. and maintains continuous edge structure. Even under Gaussian noise which corresponds to the largest metric drops, D2D^2MoE retains structured masks, plausible part layouts, and recognizable boundaries, whereas TaskDiffusion [35] exhibits region fragmentation, unstable part layouts, and degraded boundary continuity. Compared with MTMamba [37], D2D^2MoE better preserves foreground completeness and task-consistent structures, avoiding over-smoothing and missing responses, especially in saliency and edge predictions.

Overall, the absolute and clean-normalized results provide complementary evidence: D2D^2MoE maintains stronger absolute OOD performance and degrades less than TaskDiffusion on average, while the MTL baseline exhibits smaller overall normalized degradation despite D2D^2MoE showing better retention on selected tasks.

Conclusion

In this work, we addressed the challenge of multi-task dense prediction in the vision foundation model paradigm, where heterogeneous tasks must be adapted without compromising the robustness of pretrained representations. We proposed D2D^2MoE, which introduces a structured Mixture of Task Adapters to decouple task-generic alignment from task-specific specialization atop a frozen DINOv3. By combining Task-Generic Low-Rank Adaptation across all attention layers with a Task-Adaptive Mixture of Experts selectively applied in later stages, D2D^2MoE resolves task conflicts while preserving a shared semantic prior within a single forward pass. In addition, semantic prior injection further improves structural coherence and cross-task consistency. Extensive experiments across diverse benchmarks demonstrate that this design effectively mitigates negative transfer and achieves strong multi-task performance with high parameter efficiency. Overall, our results highlight the importance of layer-aware, conditional adaptation for unlocking the full potential of vision foundation models in multi-task dense prediction.

References

References

  1. [1]S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, and L. Van Gool, “Multi-task learning for dense prediction tasks: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 7, pp. 3614–3633, 2021.DOI
  2. [2]H. Ye and D. Xu, “Taskprompter: Spatial-channel multi-task prompting for dense scene understanding,” in Int. Conf. Learn. Represent., 2023.
  3. [3]F. Jiang, S. Wang, and X. Gong, “Task-conditional adapter for multi-task dense prediction,” in ACM Int. Conf. Multimedia, 2024, pp. 2059–2068.DOI
  4. [4]J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3431–3440.
  5. [5]E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 4, pp. 640–651, 2017.
  6. [6]H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in IEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 2881–2890.
  7. [7]V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 12, pp. 2481–2495, 2017.DOI
  8. [8]L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834–848, 2018.DOI
  9. [9]S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. Torr et al., “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in IEEE Conf. Comput. Vis. Pattern Recog., 2021, pp. 6881–6890.arxiv.org/abs/2012.15840
  10. [10]R. Strudel, R. Garcia, I. Laptev, and C. Schmid, “Segmenter: Transformer for semantic segmentation,” in Int. Conf. Comput. Vis., 2021, pp. 7262–7272.arxiv.org/abs/2105.05633
  11. [11]E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” Adv. Neural Inform. Process. Syst., vol. 34, pp. 12 077–12 090, 2021.arxiv.org/abs/2105.15203
  12. [12]T. Zhou, W. Wang, E. Konukoglu, and L. Van Gool, “Rethinking semantic segmentation: A prototype view,” in IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 2582–2593.arxiv.org/abs/2203.15102
  13. [13]Z. Huang, X. Wang, L. Wei, L. Huang, H. Shi, W. Liu, and T. S. Huang, “Ccnet: Criss-cross attention for semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 6, pp. 6896–6908, 2023.
  14. [14]Y. Zhu, Z. Zhang, C. Wu, Z. Zhang, T. He, H. Zhang, R. Manmatha, M. Li, and A. Smola, “Improving semantic segmentation via efficient self-training,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 3, pp. 1589–1602, 2024.
  15. [15]F. Liu, C. Shen, and G. Lin, “Deep convolutional neural fields for depth estimation from a single image,” in IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 5162–5170.arxiv.org/abs/1411.6387
  16. [16]H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in IEEE Conf. Comput. Vis. Pattern Recog., 2018, pp. 2002–2011.arxiv.org/abs/1806.02446
  17. [17]A. Gordon, H. Li, R. Jonschkowski, and A. Angelova, “Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras,” in Int. Conf. Comput. Vis., 2019, pp. 8977–8986.arxiv.org/abs/1904.04998
  18. [18]S. F. Bhat, I. Alhashim, and P. Wonka, “Adabins: Depth estimation using adaptive bins,” in IEEE Conf. Comput. Vis. Pattern Recog., 2021, pp. 4009–4018.DOI
  19. [19]J.-W. Bian, H. Zhan, N. Wang, Z. Li, L. Zhang, C. Shen, M.-M. Cheng, and I. Reid, “Unsupervised scale-consistent depth learning from video,” Int. J. Comput. Vis., vol. 129, no. 9, pp. 2548–2564, 2021.DOI
  20. [20]L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 10 371–10 381.arxiv.org/abs/2401.10891
  21. [21]L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,” Adv. Neural Inform. Process. Syst., vol. 37, pp. 21 875–21 911, 2024.arxiv.org/abs/2406.09414
  22. [22]W. Wang, Q. Lai, H. Fu, J. Shen, H. Ling, and R. Yang, “Salient object detection in the deep learning era: An in-depth survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 6, pp. 3239–3259, 2022.DOI
  23. [23]N. Liu and J. Han, “Dhsnet: Deep hierarchical saliency network for salient object detection,” in IEEE Conf. Comput. Vis. Pattern Recog., June 2016.DOI
  24. [24]L. Wang, H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, and X. Ruan, “Learning to detect salient objects with image-level supervision,” in IEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 136–145.DOI
  25. [25]Q. Hou, M.-M. Cheng, X. Hu, A. Borji, Z. Tu, and P. H. S. Torr, “Deeply supervised salient object detection with short connections,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 4, pp. 815–828, 2019.
  26. [26]Y. Zhuge, Y. Zeng, and H. Lu, “Deep embedding features for salient object detection,” in Proc. AAAI Conf. Artif. Intell., vol. 33, no. 01, 2019, pp. 9340–9347.DOI
  27. [27]H. Chen, Y. Li, Y. Deng, and G. Lin, “Cnn-based rgb-d salient object detection: Learn, select, and fuse,” Int. J. Comput. Vis., vol. 129, no. 7, pp. 2076–2096, 2021.arxiv.org/abs/1909.09309
  28. [28]M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in Int. Conf. Comput. Vis., 2021, pp. 9650–9660.arxiv.org/abs/2104.14294
  29. [29]M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193, 2023.arxiv.org/abs/2304.07193
  30. [30]O. Simeoni, H. V. Mo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa et al., “Dinov3,” arXiv preprint arXiv:2508.10104, 2025.arxiv.org/abs/2508.10104
  31. [31]H. Ye and D. Xu, “Inverted pyramid multi-task transformer for dense scene understanding,” in Eur. Conf. Comput. Vis. Springer, 2022, pp. 514–530.DOI
  32. [32]H. Ye and D. Xu, “Invpt++: Inverted pyramid multi-task transformer for visual scene understanding,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 12, pp. 7493–7508, 2024.DOI
  33. [33]H. Ye and D. Xu, “Taskexpert: Dynamically assembling multi-task representations with memorial mixture-of-experts,” in Int. Conf. Comput. Vis., 2023, pp. 21 828–21 837.arxiv.org/abs/2307.15324
  34. [34]Y. Yang, P.-T. Jiang, Q. Hou, H. Zhang, J. Chen, and B. Li, “Multi-task dense prediction via mixture of low-rank experts,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 27 927–27 937.arxiv.org/abs/2403.17749
  35. [35]Y. Yang, P.-T. Jiang, Q. Hou, H. Zhang, J. Chen, and B. Li, “Multi-task dense predictions via unleashing the power of diffusion,” in Int. Conf. Learn. Represent., 2025.
  36. [36]Y. Lu, S. Cao, and Y.-X. Wang, “Swiss army knife: Synergizing biases in knowledge from vision foundation models for multi-task learning,” in Int. Conf. Learn. Represent., 2025. [Online]. Available: https://openreview.net/forum?id=eePWw5u7J3arxiv.org/abs/2410.14633
  37. [37]B. Lin, W. Jiang, P. Chen, Y. Zhang, S. Liu, and Y.-C. Chen, “Mtmamba: Enhancing multi-task dense scene understanding by mamba-based decoders,” in Eur. Conf. Comput. Vis., Springer, 2024, pp. 314–330.arxiv.org/abs/2407.02228
  38. [38]B. Lin, W. Jiang, P. Chen, S. Liu, and Y.-C. Chen, “Mtmamba++: Enhancing multi-task dense scene understanding via mamba-based decoders,” IEEE Trans. Pattern Anal. Mach. Intell., 2025.DOI
  39. [39]M. Cao, S. Zhou, Y. Li, Y. Deng, W. Huang, and L. Wang, “Enhancing mamba decoder with bidirectional interaction in multi-task dense prediction,” in Int. Conf. Comput. Vis., 2025, pp. 18 815–18 824.arxiv.org/abs/2508.20376
  40. [40]W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” J. Mach. Learn. Res., vol. 23, no. 120, pp. 1–39, 2022.arxiv.org/abs/2101.03961
  41. [41]N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat et al., “Glam: Efficient scaling of language models with mixture-of-experts,” in Int. Conf. Mach. Learn. PMLR, 2022, pp. 5547–5569.arxiv.org/abs/2112.06905
  42. [42]J. Zhang, J. Fan, P. Ye, B. Zhang, H. Ye, B. Li, Y. Cai, and T. Chen, “Bridgetnet: Comprehensive and effective feature interactions via bridge feature for multi-task dense predictions,” IEEE Trans. Pattern Anal. Mach. Intell., 2025.DOI
  43. [43]D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” in Int. Conf. Learn. Represent., 2021.
  44. [44]C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” Adv. Neural Inform. Process. Syst., vol. 34, pp. 8583–8595, 2021.arxiv.org/abs/2106.05974
  45. [45]Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Q. V. Le, J. Laudon et al., “Mixture-of-experts with expert choice routing,” Adv. Neural Inform. Process. Syst., vol. 35, pp. 7103–7114, 2022.arxiv.org/abs/2202.09368
  46. [46]A. Agiza, M. Naseem, and S. Reda, “Mtlora: Low-rank adaptation approach for efficient multi-task learning,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 16 196–16 205.arxiv.org/abs/2403.20320
  47. [47]Y. Lu, S. Sejrijeding, Y. Ding, C. Wang, and H. Lu, “Prompt guided transformer for multi-task dense prediction,” IEEE Trans. Multimedia, vol. 26, pp. 6375–6385, 2024.arxiv.org/abs/2307.15362
  48. [48]N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in Eur. Conf. Comput. Vis. Springer, 2012, pp. 746–760.DOI
  49. [49]Y.-C. Liu, C.-Y. Ma, J. Tian, Z. He, and Z. Kira, “Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks,” Adv. Neural Inform. Process. Syst., vol. 35, pp. 36 889–36 901, 2022.arxiv.org/abs/2210.03265
  50. [50]Y. Xin, J. Du, Q. Wang, Z. Lin, and K. Yan, “Vmt-adapter: Parameter-efficient transfer learning for multi-task dense scene understanding,” in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 14, 2024, pp. 16 085–16 093.arxiv.org/abs/2312.08733
  51. [51]S. Baek, S. Lee, H. Jo, H. Choi, and D. Min, “Tadformer: Task-adaptive dynamic transformer for efficient multi-task learning,” in IEEE Conf. Comput. Vis. Pattern Recog., 2025, pp. 14 858–14 868.arxiv.org/abs/2501.04293
  52. [52]K. S. I. Mantri, C.-B. Schönlieb, B. Ribeiro, C. Baskin, and M. Eliasof, “Ditask: Multi-task fine-tuning with diffeomorphic transformations,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 25 218–25 229.arxiv.org/abs/2502.06029
  53. [53]Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, Z. Wang et al., “M3vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design,” Adv. Neural Inform. Process. Syst., vol. 35, pp. 28 441–28 457, 2022.DOI
  54. [54]Z. Chen, Y. Shen, M. Ding, Z. Chen, H. Zhao, E. G. Learned-Miller, and C. Gan, “Mod-squad: Designing mixtures of experts as modular multi-task learners,” in IEEE Conf. Comput. Vis. Pattern Recog., 2023, pp. 11 828–11 837.DOI
  55. [55]T. Chen, X. Chen, X. Du, A. Rashwan, F. Yang, H. Chen, Z. Wang, and Y. Li, “Adamv-moe: Adaptive multi-task vision mixture-of-experts,” in Int. Conf. Comput. Vis., 2023, pp. 17 346–17 357.DOI
  56. [56]I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in IEEE Conf. Comput. Vis. Pattern Recog., 2016, pp. 3994–4003.arxiv.org/abs/1604.03539
  57. [57]I. Kokkinos, “Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory,” in IEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 6129–6138.arxiv.org/abs/1609.02132
  58. [58]D. Xu, W. Ouyang, X. Wang, and N. Sebe, “Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing,” in IEEE Conf. Comput. Vis. Pattern Recog., 2018, pp. 675–684.arxiv.org/abs/1805.04409
  59. [59]S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” in IEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 1871–1880.arxiv.org/abs/1803.10704
  60. [60]S. Vandenhende, S. Georgoulis, and L. Van Gool, “Mti-net: Multi-scale task interaction networks for multi-task learning,” in Eur. Conf. Comput. Vis. Springer, 2020, pp. 527–543.arxiv.org/abs/2001.06902
  61. [61]A. R. Zamir, A. Sax, W. Shen, L. J. Guibas, J. Malik, and S. Savarese, “Taskonomy: Disentangling task transfer learning,” in IEEE Conf. Comput. Vis. Pattern Recog., June 2018, pp. 3712–3722.
  62. [62]X. Xu et al., “Mtformer: Multi-task learning via transformer and cross-task reasoning,” in Eur. Conf. Comput. Vis., 2022.DOI
  63. [63]S. Yang, H. Ye, and D. Xu, “Contrastive multi-task dense prediction,” in Proc. AAAI Conf. Artif. Intell., 2023.arxiv.org/abs/2307.07934
  64. [64]D. Brüggemann, M. Kanakis, A. Obukhov, S. Georgoulis, and L. Van Gool, “Exploring relational context for multi-task dense prediction,” in Int. Conf. Comput. Vis., 2021, pp. 15 869–15 878.
  65. [65]W.-H. Li, X. Liu, and H. Bilen, “Learning multiple dense prediction tasks from partially annotated data,” in IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 18 789–18 889.arxiv.org/abs/2111.14893
  66. [66]Y. Xu, Y. Yang, and L. Zhang, “Multi-task learning with knowledge distillation for dense prediction,” in Int. Conf. Comput. Vis., 2023, pp. 21 550–21 559.DOI
  67. [67]H. Huang, Y. Huang, L. Lin, R. Tong, Y.-W. Chen, H. Zheng, Y. Li, and Y. Zheng, “Going beyond multi-task dense prediction with synergy embedding models,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 28 181–28 190.DOI
  68. [68]Y. Shang, D. Xu, G. Liu, R. R. Kompella, and Y. Yan, “Efficient multitask dense predictor via binarization,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 15 899–15 908.arxiv.org/abs/2405.14136
  69. [69]M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: A survey and outlook,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 4, pp. 2245–2264, 2025.DOI
  70. [70]A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.arxiv.org/abs/2010.11929
  71. [71]K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 16 000–16 009.arxiv.org/abs/2111.06377
  72. [72]A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., “Segment anything,” in Int. Conf. Comput. Vis., 2023, pp. 4015–4026.arxiv.org/abs/2304.02643
  73. [73]S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He, “Deepspeed-moe: Advancing mixture-of-expert inference and training to power next-generation ai at scale,” in Int. Conf. Mach. Learn. PMLR, 2022, pp. 18 332–18 346.arxiv.org/abs/2201.05596
  74. [74]B. Mustafa, C. Riquelme, J. Puigcerver, R. Jenatton, and N. Houlsby, “Multimodal contrastive learning with limoe: the language-image mixture of experts,” Adv. Neural Inform. Process. Syst., vol. 35, pp. 9564–9576, 2022.arxiv.org/abs/2206.02770
  75. [75]M.-Z. Xu, Z. Zhu, W. Wang, X. Wang, H. Li, X. Wang, and J. Dai, “Uni-perceiver-moe: Learning sparse generalist models with conditional moes,” Adv. Neural Inform. Process. Syst., vol. 35, pp. 2664–2678, 2022.
  76. [76]L. Shen, G. Chen, R. Shao, W. Guan, and L. Nie, “Mome: Mixture of multimodal experts for generalist multimodal large language models,” Adv. Neural Inform. Process. Syst., vol. 37, pp. 42 048–42 070, 2024.arxiv.org/abs/2407.12709
  77. [77]Z. Zong, B. Ma, D. Shen, G. Song, H. Shao, D. Jiang, H. Li, and Y. Liu, “Mova: Adapting mixture of vision experts to multimodal context,” Adv. Neural Inform. Process. Syst., vol. 37, pp. 103 305–103 333, 2024.arxiv.org/abs/2404.13046
  78. [78]Y. Li, S. Jiang, B. Hu, L. Wang, W. Zhong, W. Luo, L. Ma, and M. Zhang, “Uni-moe: Scaling unified multimodal llms with mixture of experts,” IEEE Trans. Pattern Anal. Mach. Intell., 2025.DOI
  79. [79]J. Yu, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He, “Boosting continual learning of vision-language models with mixture-of-experts adapters,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 23 219–23 230.arxiv.org/abs/2403.11549
  80. [80]J. Yu, Z. Huang, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He, “Moe-adapters++: Toward more efficient continual learning of vision-language models via dynamic mixture-of-experts adapters,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 12, pp. 11 912–11 928, 2025.DOI
  81. [81]K.-K. Maninis, I. Radosavovic, and I. Kokkinos, “Attentive single-tasking of multiple tasks,” in IEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 1851–1860.arxiv.org/abs/1904.08918
  82. [82]Y. Lu, S. Cao, and Y.-X. Wang, “Swiss army knife: Synergizing biases in knowledge from vision foundation models for multi-task learning,” in Int. Conf. Learn. Represent., 2025.arxiv.org/abs/2410.14633
  83. [83]X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, and A. Yuille, “Detect what you can: Detecting and representing objects using holistic models and body parts,” in IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 1971–1978.arxiv.org/abs/1406.2031
  84. [84]D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” in Int. Conf. Learn. Represent., 2019.

Paper details

vlnc:W_v7xqursk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents