WorksComputer Science
Unlocking the Potential of Vision Foundation Models for Multi-Task Dense Prediction via Mixture of Task Adapters
Abstract
Multi-task dense prediction seeks a single, coherent representation that supports simultaneously discriminative semantic recognition and regularized geometric/structural inference. For vision foundation models, pretrained semantic invariances conflict with the frequency and smoothness biases of dense objectives; joint fine-tuning and single-route adapters therefore compress competing gradients into one update channel, causing negative transfer and unstable cross-task agreement. We frame this as two-level adaptation: task-generic alignment that preserves priors, and task-aware specialization that resolves late-stage conflicts. Based on this principle, we propose D2MoE , a Mixture of Task Adapters that explicitly factorizes these roles within a frozen DINOv3 backbone. D2MoE introduces Task-Generic Low-Rank Adaptation (TG-LoRA) in attention projections throughout the network to provide a stable, shared adaptation channel, and a Task-Adaptive Mixture of Experts (TA-MoE) injected only at the final attention block of each stage to enact sparse, task-conditioned perturbations of token mixing in a single forward pass. A Semantic Prior Injector further couples multi-level task features with frozen foundation priors to reinforce semantic coherence while preserving boundary and structural fidelity. Experiments across diverse benchmarks substantiate that this structured factorization yields robust multi-task improvements with a small trainable footprint, establishing a principled and scalable route to prior-preserving, foundation-centric multi-task dense perception.
Introduction
Multi-task dense prediction (MTDP) [1], [2], [3] unifies multiple pixel-level tasks (e.g., semantic segmentation [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], depth estimation [15], [16], [17], [18], [19], [20], [20], [21], and saliency detection [22], [23], [24], [25], [26], [27]) in a single model to share computation, couple semantic/geometric/structural cues, and reduce redundancy relative to task-specific systems. This joint setting further promotes cross-task spatial coherence, especially near boundaries and geometric discontinuities. Recent MTDP increasingly builds on vision foundation models (VFMs) [28], [29], [30], using pretrained transformers (e.g., ViT [31], [32], [33], [34], [35], [36] and Swin Transformer [37], [38], [39]). As a result, the central problem shifts from learning shared representations to coordinating adaptation of semantically organized foundation features across heterogeneous dense objectives.
Although vision foundation models provide rich, transferable features, MTDP under VFMs faces a core incompatibility: one representation must satisfy objectives with distinct spatial and frequency preferences. This yields three challenges: (i) Conflicting Task Priors. Semantic objectives depend on localized discriminative detail, whereas geometric objectives prefer smooth, globally consistent structure. (ii) Prior Erosion. Naive end-to-end optimization overwrites transferable invariances, degrading generalization and cross-task consistency. (iii) Single-route Adaptation Interference, i.e., forcing all tasks through shared prompts/adapters/decoders projects heterogeneous gradients into one subspace, amplifying negative transfer. These observations motivate conditional, task-aware adaptation that selectively updates the representation to enable task-specific refinement while preserving the shared prior. Selective routing via mixture of experts provides a principled mechanism for allocating distinct update paths to competing objectives and mitigating negative transfer [40], [41].
These observations indicate that once a single shared adaptation path bottlenecks, MTDP needs task-conditioned computation at the adaptation interface while preserving a shared foundation prior rather than hard-partitioning the backbone. (i) Unscalable Factorization. Separate encoders/decoders for each task destroys sharing and is prohibitive for foundation-scale backbones and dense outputs [31], [42]. (ii) Delayed Conflict Resolution. Late-branching heads postpone specialization, leaving upstream features exposed to the same conflicting gradients. (iii) Sparse, Task-conditioned Capacity. Effective MTDP should keep a single backbone but introduce task-dependent specialization via sparsely activated auxiliary parameters. Mixture of experts (MoE) implements this through sparse gating/routing, allocating partially disjoint subspaces per objective without sacrificing the shared representation [40], [41], [43], [44], [45]. In contrast, non-MoE MTDP either applies largely uniform encoder adaptation [46], [47] or shifts capacity into decoders whose task interaction is weakly coupled to the backbone features being adapted [37], [38].
Yunzhi Zhuge and Xinzhuo Yu contribute equally to this work.
Yunzhi Zhuge, Sitong Gong, Lu Zhang, Pingping Zhang and Huchuan Lu are with the School of Information and Communication Engineering, Dalian University of Technology, Dalian 116081, China. (Email: [email protected]; [email protected]; [email protected]; [email protected]; [email protected])
Xinzhuo Yu is with the School of Computer Science and Technology, Dalian University of Technology, Dalian 116081, China. (Email: [email protected])
Several recent parameter-efficient and conditional multi-task adaptation methods are closely related to this direction. Polyhistor [49] and VMT-Adapter [50] improve multi-task transfer by sharing lightweight adaptation modules while preserving task-specific knowledge, and TADFormer [51] further introduces task-adaptive dynamic filtering for input-conditioned task adaptation. DiTask [52] studies multi-task fine-tuning from a weight-geometry perspective by preserving pretrained singular-vector structures and modulating singular values. These methods validate the importance of parameter-efficient multi-task adaptation, yet most of them focus on designing general adaptation modules or task-conditioned filters, rather than explicitly locating where task conflicts should be resolved inside a frozen vision foundation model for dense prediction. In parallel, MoE-based MTL methods such as M³ViT [53], Mod-Squad [54], AdaMV-MoE [55], TaskExpert [33] and MLoRE [34] introduce conditional expert selection to improve cooperation-specialization trade-offs. However, existing MoE designs typically insert trainable experts throughout the backbone, assemble task features in the decoder, or target compact task-specific execution, which may either perturb pretrained representations or leave task conflicts insufficiently addressed.
Motivated by the above analysis, we view the MTDP bottleneck under frozen VFMs as requiring two complementary capabilities: task-generic adaptation and task-aware specialization. Task-generic adaptation aligns the foundation representation to the downstream data distribution while maintaining transferable semantic prior, whereas task-aware specialization explicitly resolves conflicts induced by heterogeneous dense objectives. A closely related foundation-model-driven approach is SAK [36], which uses a shared teacher-agnostic stem with teacher-specific adapter paths to distill complementary biases from multiple VFMs and fuses them via task routers for multi-task prediction. However, the student is optimized to satisfy heterogeneous teacher feature spaces, yielding a compromise representation that weakens domain adaptation, while specialization remains teacher-conditioned rather than explicitly resolving task-level gradient conflicts. Practically, requiring VFM teachers incurs compute/memory (multiple frozen backbones, matching losses) during training, making it bulky for deployment (as shown in Figure 1(a)).

Figure 1. Motivation of D²MoE. (a) SAK distills from multiple VFMs into a shared student, coupling heterogeneous task gradients and incurring multi-teacher overhead. (b) D²MoE instead freezes a single VFM and enforces early task decoupling through a Mixture of Task Adapter (MoTA), allocating sparse, task-conditioned low-rank update routes for specialization without eroding pretrained invariances. (c) Correlation probes (Pearson maps around sampled points) confirm that model with MoTA induces more task-consistent feature organization for both semantic segmentation and depth, improving boundary adherence without sacrificing long-range structural coherence.
To address the above limitations, we propose D²MoE, a foundation model driven framework for multi-task dense prediction. Unlike prior MoE-based MTDP methods that mainly emphasize task routing and expert specialization, D²MoE uses MoE as part of a lightweight adapter design to introduce task-adaptive updates without tuning the VFM backbone. D²MoE builds on a single frozen VFM backbone and performs multi-task decoupling through a unified Mixture of Task Adapters (MoTA) design (as shown in Figure 1(b)), which combines Task-Generic Low-rank Adaptation (TG-LoRA) with Task-Adaptive Mixture of Experts (TA-MoE). TG-LoRA is applied across backbone layers to achieve stable and parameter-efficient adaptation, while TA-MoE is selectively placed at the final layer of each stage to enable sparse, task-aware specialization within a single forward pass. In addition, D²MoE incorporates semantic priors from DINOv3 to guide cross-layer feature integration, facilitating both global context modeling and fine-grained detail preservation. As a result, DMoE achieves state-of-the-art performance on standard multi-task dense prediction benchmarks with minimal additional parameters (as shown in Figure 2).

Figure 2. Performance vs. Model Complexity. Comparison of D²MoE against state-of-the-art methods on NYUD-v2 [48]. D²MoE achieves superior performance (higher mIoU and lower RMSE) despite a substantial reduction in the total number of trainable parameters. The trainable-parameter count varies with the task configuration, with 45M on NYUD-v2 (shown here) and 54M on Pascal-Context (as shown in Table 1.)
Our contributions are summarized as follows.
We formalize multi-task dense prediction with frozen vision foundation models as a two-level adaptation problem, disentangling task-generic alignment from task-aware specialization. This perspective explains negative transfer under single-route adaptation and motivates sparse, task-conditioned computation.
We propose DMoE, a Mixture of Task Adapters architecture that combines Task-Generic Low-Rank Adaptation with Task-Adaptive Mixture of Experts. The design introduces sparse, late-stage task specialization while preserving semantic priors within a single backbone.
Extensive experiments on PASCAL-Context, NYUD-v2, and Cityscapes show that DMoE achieves state-of-the-art multi-task performance, delivering the highest aggregated gains across benchmarks while requiring substantially fewer trainable parameters.
The remainder of this paper is structured as follows. Sec. 2 surveys related works on multi-task dense prediction, vision foundation models, and mixture of experts. Sec. 3 introduces DMoE, including the Mixture of Task Adapters and the Semantic Prior Injector for prior-guided decoding. Sec. 4 presents the experimental protocol and results on PASCAL-Context, NYUD-v2, and Cityscape. Sec. 5 concludes with key findings and potential implications.
Related Works
Multi-task Dense Prediction
Multi-task dense prediction, which jointly addresses semantic segmentation, depth estimation, and auxiliary tasks for scene understanding, has substantively progressed via relational modeling, efficient architectures, and generative formulations [1, 2, 47, 56, 57, 58, 59, 60, 61, 62, 63]. ATRC [64] explicitly formalizes inter-task dependencies to improve transferable generalization. MTPSL [65] tackles incomplete annotations in a partially-supervised setting, enabling resilient learning under partial labels. Transformer-based designs such as the InvPT enable hierarchical feature fusion and granular scene parsing [2, 31]. Efficiency has been advanced by cross-task distillation, epitomized by KDAM [66]. MLoRA employs a mixture of low-rank experts for parameter-efficient, per-task routing that scales to heterogeneous dense outputs [34]. Synergy- and compactness-oriented approaches include SEM, which promotes salutary task interactions beyond naive sharing [67], and Bi-MTPD, which binarizes predictors for compact models with negligible performance loss [68]. State-space innovations such as MTMamba [37] capture long-range dependencies for enhanced multi-task understanding, whereas MTMamba++ [38] streamlines decoding to improve efficiency. BridgeNet [42] introduces bridge embeddings to facilitate comprehensive cross-task feature interactions. Collectively, these lines of work converge on holistic, adaptive and resource-conscious frameworks. Collectively, these lines of work converge holistic and resource-conscious frameworks.
Parameter Efficient Multi-task Adaption
Parameter-efficient multi-task adaptation studies how to transfer pretrained models to multiple downstream tasks with limited trainable parameters, making it complementary to MTDP-specific architectures. Polyhistor [49] uses decomposed hypernetworks and layer-wise scaling kernels to share knowledge without full task-wise fine-tuning. VMT-Adapter [50] introduces a once-for-all adapter that combines shared cross-task knowledge with lightweight task-specific extraction, yielding nearly task-number-independent cost. TADFormer [51] performs task-aware adaptation through parameter-efficient prompting and dynamic task filters conditioned on task contexts. DiTask [52] preserves pretrained weight geometry via diffeomorphic transformations of singular values for shared and task-specific modulation. These methods show that separating shared adaptation from task-specific modulation is effective for multi-task transfer. Different from them, DMoE targets frozen VFM s for MTDP by coupling global TG-LoRA alignment with sparse TA-MoE specialization at late-stage attention projections, where heterogeneous dense objectives most strongly diverge.
Vision Foundation Models
Vision foundation models (VFMs) have revolutionized computer vision by learning generalizable representations from vast unlabeled datasets, enabling transfer to diverse downstream tasks [69]. Pioneering architectures like Vision Transformers (ViT) [70] scaled transformers for image processing, achieving state-of-the-art performance on benchmarks such as ImageNet. Self-supervised approaches, including Masked Autoencoders (MAE) [71], reconstruct masked image patches to capture semantic features without annotations. DINO [28] employs self-distillation to emerge invariant representations, fostering emergent properties like object segmentation. Multimodal models such as CLIP align vision and language embeddings for zero-shot classification and retrieval. Subsequent iterations like DINOv2 [29] refine training with curated data and improved objectives, yielding robust features across scales. The Segment Anything Model (SAM) [72] introduces promptable segmentation for arbitrary objects, advancing interactive vision applications. Most recently, DINOv3 [30] extends these capabilities with enhanced multi-task potential through advanced self-supervision techniques. Accordingly, this work targets prior-preserving, parameter-efficient task-conditioned adaptation of VFMs for multi-task dense prediction.
Mixture of Experts
Mixture of experts (MoE) has emerged as a de facto paradigm for conditional computation: a router sparsely activates a small subset of experts per token/sample, increasing capacity without proportional FLOPs. Foundational designs such as Switch Transformer [40] and GLaM [41] establish scalable sparse routing, while Expert-Choice routing [45] and DeepSpeed-MoE [73] show that routing policies and systems support are central to utilization and efficiency.
In vision and multimodal foundation models, MoE is used to reduce domain/task interference while retaining scalable capacity. V-MoE [44] adapts sparse experts to ViTs, LIMoE [74] shows modality-aware partitioning under contrastive learning, and Uni-Perceiver-MoE [75] integrates conditional experts into unified generalist architectures. More recent multimodal variants route over heterogeneous expert pools for generalist MLLMs or vision experts (MoME [76], MoVA [77]), with Uni-MoE [78] further systematizing scaling in unified multimodal LLMs.
MoE is also effective for multi-task learning and parameter-efficient specialization, where conditional routing allocates capacity to tasks with higher complexity or conflict. Vision MTL approaches route along task-conditioned expert pathways with varied sharing/gating (M³ViT [53], Mod-Squad [54], AdaMV-MoE [55], TaskExpert [33]), and MLoRE [34] scales experts for dense prediction via low-rank parameterization. Recent work reframes MoE as adapter-style tuning: MoE-Adapters [79] sparsely selects from a fixed adapter pool (with OOD fallback), and MoE-Adapters++ [80] introduces dynamic adapters and learned expert selection. Inspired by adapter-style MoE, we route among lightweight task adapters atop frozen DINOv3 features, concentrating specialization in small modules while preserving backbone transferability.
Methodology
This section presents D²MoE, a parameter-efficient framework for adapting frozen vision foundation models to multitask dense prediction. § 3.1 first provides an overview of the pipeline, including the frozen DINOv3 backbone, the Mixture of Task Adapters (MoTA), and the Semantic Prior Injector (SPI). § 3.2 then details MoTA, which decouples shared adaptation from task-conditioned specialization. Finally, § 4 introduces SPI for prior-guided multi-scale decoding.
Framework Overview
Figure 3 illustrates the overall architecture of D²MoE. Given an input image , we employ a frozen DINOv3 backbone as the shared feature extractor. The backbone provides two types of representations: task-agnostic semantic priors , which retain the pretrained foundation knowledge, and task-adapted features for each downstream task .

Figure 3. Overall architecture of D²MoE. § 3.1 Built upon a frozen DINOv3 backbone, D²MoE organizes MoTA across four stages to preserve shared priors through task-generic adaptation and introduce localized task-specific adaption at stage endings. SPI fuses prior and task features, followed by lightweight heads for multi-task dense predictions.
To adapt the frozen backbone without full fine-tuning, we insert the proposed Mixture of Task Adapters (MoTA) into the attention blocks. MoTA serves as the only backbone-side trainable adaptation path and produces task-aware multi-level features while preserving the shared VFM prior. The adapted features are then passed to the Semantic Prior Injector (SPI), which fuses multi-level task features with frozen semantic priors for prior-guided decoding. Finally, lightweight task-specific heads map the refined representations to the corresponding prediction spaces, yielding task-wise outputs .
Mixture of Task Adapters
MoTA is designed to address the single-route adaptation bottleneck in frozen VFM-based multi-task dense prediction. Instead of forcing all tasks to share an identical adaptation subspace, MoTA factorizes backbone adaptation into two complementary components: a shared low-rank pathway for stable task-generic alignment, and a sparse task-conditioned pathway for resolving task-specific conflicts. This design allows D²MoE to retain the semantic priors of the frozen DINOv3 while introducing lightweight conditional capacity only where task divergence needs to be modeled. The following sections describe the two components in detail.
Task-Generic Low-Rank Adaptation
To obtain parameter-efficient transfer while preserving the semantic geometry encoded by the DINOv3 backbone, we introduce Task-Generic Low-Rank Adaptation (TG-LoRA) inside the self-attention modules. In contrast to full fine-tuning, TG-LoRA keeps all pretrained backbone weights frozen and learns only structured low-rank increments on a small set of high-impact linear maps. This constrained re-parameterization provides a stable, task-agnostic adaptation channel that aligns the foundation representation to downstream data distributions with reduced risk of representation drift and cross-task interference.
TG-LoRA is inserted exclusively into the query, key, value projections and the attention output projection of each Transformer block. Let denote the input token matrix ( tokens, embedding dimension ). Under a pre-normalization design, we first compute:
For , we express the projection as a frozen base weight plus a rank- residual update:

Figure 4. Illustration of the key components in D²MoE. (§ 3.2 and § 4) (a) Task-Generic Adaptation inserts TG-LoRA into attention projections to provide shared, low-rank adaptation while keeping the DINOv3 backbone frozen. (b) Task-Specific Adaptation employs TA-MoE, where task routers select Top-k low-rank experts to produce sparse, task-conditioned updates for attention projections. (c) The Semantic Prior Injector fuses multi-level task features with frozen semantic priors to enhance semantic consistency and structural coherence for dense prediction tasks.
where is frozen, while and are learnable with . The scalar controls the effective update magnitude; a common choice is to keep the perturbation scale approximately stable as varies. This can be interpreted as learning a constrained perturbation of the form without modifying the pretrained weights.
The projected features are then processed by multi-head self-attention. Let denote the number of heads and the per-head dimension. After reshaping into head-wise tensors with , each head computes:
and the heads are concatenated to form .
TG-LoRA is further applied to the attention output projection that maps contextualized tokens back into the shared embedding space:
where is frozen and are learnable rank- adapters. The attention sub-layer output is then obtained through the residual pathway:
The FFN and LayerScale follow the original DINOv3 design and are kept frozenThe TG-LoRA augmented output is then propagated to the next Transformer block. To preserve the pretrained VFM’s general-purpose representation, semantic priors are extracted from the corresponding unmodified DINOv3 blocks applied to the same input , ensuring that the priors reflect task-agnostic backbone features rather than adaptation-induced artifacts.
By restricting TG-LoRA to , the trainable degrees of freedom primarily modulate (i) the token-to-token interaction geometry via and (ii) the injection of contextual evidence back into the embedding space via , while leaving the typically more representationally disruptive FFN untouched. As a result, TG-LoRA acts as a globally shared, task-generic adaptation mechanism that improves robustness to domain shifts while preserving the semantic organization of the foundation model, providing a stable substrate for subsequent task-adaptive specialization.
Task-Adaptive Mixture of Experts
TG-LoRA provides a shared, task-agnostic adaptation route that preserves the backbone’s semantic prior. Nevertheless, heterogeneous dense prediction objectives often demand incompatible token-interaction patterns at late representation levels, where high-level semantics are consolidated and task divergence becomes most pronounced. To explicitly factorize these late-stage conflicts without fragmenting the backbone, we introduce Task-Adaptive Mixture of Experts (TA-MoE) as a conditional specialization mechanism.
In contrast to MoE designs that distribute experts broadly across backbone layers or postpone task differentiation to decoder heads [53, 54], TA-MoE is injected only into the final self-attention block of each DINOv3 stage and acts directly on the attention linear maps (Q/K/V projections and the output projection). This placement targets the token-mixing operator where task-dependent contextual aggregation is most sensitive, while keeping earlier layers fully shared for stable representation learning.
Let denote the token features entering the last attention block of stage , where is the number of tokens and is the channel dimension. For a -task setting, we introduce task-indexed inputs as notational replicas initialized by broadcasting , so that task-conditioned operators can be written uniformly, while the shared backbone features are computed only once.
A task-conditioned router predicts a sparse gating distribution over candidate experts from a global stage summary. We first compute a compact descriptor via token-wise pooling:
and obtain expert logits using a lightweight task-specific linear map:
where . Sparse activation is enforced by selecting the Top- experts () and normalizing on the active set:
where denotes the probability simplex with zeros on inactive experts.
Each expert is implemented as a low-rank residual update. For each attention projection , the -th expert is parameterized by factors and with rank . The task-specific residual to the frozen projection is:
with scaling factor . Accordingly, task-specialized projections are computed as:
The task-adaptive , , and are then processed by standard multi-head self-attention:
where denotes the per-head dimension (head splitting/concatenation is omitted for brevity). TA-MoE is further applied to the attention output projection to complete a coherent task-conditioned attention pathway:
where is the frozen output projection and is the routed low-rank residual defined above.
The resulting tokens are processed by the standard residual connections, feed-forward transformation, and LayerScale of DINOv3, yielding the task-specific stage-wise outputs . Overall, TA-MoE interprets task divergence as a conditional perturbation of the token-mixing operator: the backbone projections remain frozen to preserve a shared semantic prior, while a sparse mixture of low-rank experts injects task-selective capacity precisely at the late layers where conflicts are most acute. By restricting conditional computation to the final attention block of each stage, we obtain a clean separation between globally shared representation learning and task-specific refinement, achieving effective disentanglement with minimal additional overhead.
Semantic-Prior Guided Decoding
TG-LoRA and TA-MoE adapt the shared backbone for multi-task dense prediction, yet the final outputs still depend on a decoder that maps adapted features to task-specific label spaces. Most VFM-style decoders remain shallow and task-agnostic, typically performing FPN-like fusion without explicitly exploiting the high-level semantic structure already encoded in foundation models. We thus propose a semantic-prior guided decoder that integrates task-adapted features with frozen DINOv3 representations, enabling prior-aware multi-scale aggregation and task-conditioned refinement.
Semantic Prior Injection
For each task , let be the task-adapted features from four backbone stages, where , and let be the semantic priors extracted from DINOv3, with . The Semantic Prior Injector (SPI) performs channel alignment, task-conditioned multi-scale fusion, and semantic prior injection.
Since features from all stages share the same spatial resolution and channel dimensions, no explicit spatial rescaling is required. We project both task features and semantic priors using independent convolutions:
where . To aggregate task features across scales, we concatenate the aligned multi-scale task features along the channel dimension:
We then employ a lightweight fusion head composed of a convolution for channel mixing and compression, followed by batch normalization and a non-linear activation function, and a subsequent convolution to capture local spatial interactions, corresponding to pixel-wise fusion logits for each stage:
where denotes the convolutional fusion module. A softmax operation is applied along the stage dimension to obtain normalized pixel-wise fusion weights:
Finally, the aggregated task feature is computed as a pixel-wise weighted sum over scales:
In parallel, we merge semantic priors across stages into a single map. Concretely, we concatenate along the channel dimension and apply a lightweight fusion operator:
where is implemented as channel mixing followed by a convolution for local context integration.
Given the aggregated task feature , SPI injects semantic priors by directly concatenating the aligned semantic representation with the task feature, followed by joint feature refinement. Specifically, we fuse the task feature and semantic prior as:
where the concatenation is performed along the channel dimension. The fused representation is then processed by a lightweight refinement block to produce the final prior-enhanced feature:
where is implemented as a Conv–Norm–GeLU–Conv stack.
Task-Specific Decoding Head
Given , we employ a lightweight head that maps the shared latent to the prediction space of task :
where is a task-dependent projection and denotes bilinear upsampling to the input resolution.
Joint Optimization Losses
We train all tasks jointly with , which is defined as:
where is task-specific (e.g., cross-entropy for segmentation, /scale-invariant losses for depth, and cosine/angular losses for normals) and balances tasks. We use uniform weighting in all experiments.
Experiments
In this section, we evaluate D²MoE for multi-task dense prediction. § 5.1 introduces the benchmarks (Pascal-Context, NYUD-v2, and Cityscapes) and the evaluation protocol, including per-task metrics and the overall multi-task gain . § 5.2 details our training and implementation. § 5.3 reports quantitative and qualitative comparisons with recent state-of-the-art methods. § 5.4 presents ablations studies and analyze the contribution of each component. § 5.5 gives feature visualization and analysis. § 5.6 evaluates robustness.
Experimental Setups.
Datasets
Following recent MTL works [32], [37], [42], we evaluate D²MoE on three widely used benchmarks under standardized protocols, with official splits and their canonical task suites. Pascal-Context [83] contains 4998 training and 5105 test images with rich pixel-level annotations, on which we report semantic segmentation (Sem. Seg.), human parsing (H. Pars.), object edge detection (Edge), surface normal estimation (Norm.), and saliency detection (Sal.). NYUD-v2 [48] focuses on indoor scenes with 795 training and 654 test samples; we evaluate Sem. Seg., monocular depth estimation (Dep.), Norm., and Edge. Cityscapes provides urban street-view imagery with 2975 training images and 500 validation images, including 19 semantic classes and stereo-derived disparity; we follow common practice and report Sem. Seg. together with geometry-related predictions (Dep.) derived from the fine annotations. Together, these datasets cover outdoor scenes, indoor environments, and driving scenarios, enabling a broad and consistent evaluation of multi-task dense prediction under diverse conditions.
Evaluation Metrics
To enable a rigorous, fair and fully comprehensive comparison for multi-task dense prediction, we adopt standard task-specific measures and a normalized cross-task summary metric for overall assessment. Specifically, we report:
Sem. Seg. / H. Pars.: mean Intersection-over-Union (mIoU).
Dep.: root mean square error (RMSE).
Norm.: mean angular error (mErr).
Sal.: maximal F-measure (maxF).
Edge: Optimal Dataset Scale F-measure (odsF).
To summarize performance across heterogeneous task objectives, the overall performance gain of multi-task learning () is computed relative to our multi-task baseline:
where is the number of tasks, and denote the comparing result and our multi-task baseline results for task , and for lower-is-better metrics (RMSE, mErr) and otherwise (mIoU, maxF, ODS-F). A positive therefore indicates an overall improvement. We report all per-task metrics together with on the official dataset splits.
Implementation Details
To enable parameter-efficient multi-task dense prediction, we use DINOv3 [30] as frozen backbone and adapt it with our designed modules. Models are trained with AdamW (weight decay ) and a polynomial learning rate schedule. Base learning rates are , , and for Pascal-Context, NYUDv2, and Cityscapes, respectively. We train on two NVIDIA A100 GPUs with a batch size of 4 (2 per GPU). Loss functions are detailed in § 4.0.3. TG-LoRA is applied to the , , , and output layers of all attention blocks with rank 16. TA-MoE is applied in the final attention layer, with each expert as a rank-4 LoRA module. The SPI module has a fused feature embedding dimension of 384. All LoRA adapters use a scaling factor of 4 and a dropout rate of 0.05.
To isolate the effect of our modules from DINOv3 itself, we build the MTL Baseline with the same frozen DINOv3 backbone and identical training protocol as DMoE. This baseline trains only task-specific prediction heads, without TG-LoRA, TA-MoE, or SPI. Thus, its gap to DMoE directly reflects the contribution of our proposed modules.
Benchmark Results
Quantitative Comparison Results
We conduct comprehensive quantitative comparisons on three standard benchmarks, namely Pascal-Context, NYUD-v2, and Cityscapes, using the official data splits and widely adopted multi-task settings for fair evaluation. We report task-specific metrics for each benchmark and additionally summarize overall performance using the aggregated multi-task gain , which facilitates a holistic comparison across heterogeneous dense prediction tasks.
Pascal-Context. Table 1 summarizes results on PASCAL-Context across five dense prediction tasks including semantic segmentation (Sem. Seg.), human parsing (H. Pars.), saliency detection (Sal.), surface normal estimation (Norm.), and edge detection (Edge) under the standard multi-task setting. DMoE achieves the highest aggregated multi-task gain (). Relative to the MTL Baseline (ViT-L), it improves Sem. Seg. (82.61→84.88 mIoU), H. Pars. (76.38→78.05 mIoU) and Sal. (77.09→85.55 maxF), while reducing Norm. error (17.63→13.57 mErr) and increasing boundary accuracy (74.00→78.40 odsF). The concurrent gains across recognition-oriented and geometry-sensitive tasks suggest that MoE mitigates negative transfer while preserving beneficial sharing. Compared with decoder-centric methods such as MTMamba++ and BIM (Swin-L), MoE attains higher performance on segmentation and parsing and a larger overall gain (8.88 versus 6.15 and 5.99). Among ViT-L-based approaches, MoE also improves upon MLoRE, TaskDiffusion and BridgeNet, consistent with the view that task-adaptive routing, coupled with the semantic priors provided by DINOv3, supports stronger task disentanglement and more effective cross-task coordination.
| Method | Backbone | Sem. Seg. mIoU↑ | H. Pars. mIoU↑ | Sal. maxF↑ | Norm. mErr↓ | Edge odsF↑ | (%) ↑ | Para. (M)↓ | Train. Para. (M)↓ |
| MTL Baseline | ViT-L | 82.61 | 76.38 | 77.09 | 17.63 | 74.00 | 0.00 | 328 | 25 |
| PAD-Net [58] [CVPR2018] | HRNet-18 | 53.60 | 59.60 | 65.80 | 15.30 | 72.50 | -12.11 | 81 | 81 |
| MTI-Net [60] [ECCV2020] | HRNet-18 | 61.70 | 60.18 | 84.78 | 14.23 | 70.80 | -4.32 | 128 | 128 |
| ATRC [64] [ICCV2021] | HRNet-18 | 67.67 | 62.93 | 82.29 | 14.24 | 72.42 | -2.37 | 96 | 96 |
| ASTMT [81] [ECCV2024] | ResNet-50 | 68.00 | 61.12 | 65.71 | 14.68 | 72.40 | -7.57 | 49 | 49 |
| MTMamba [37] [ECCV2024] | Swin-L | 81.11 | 72.62 | 84.14 | 14.14 | 78.80 | 5.74 | 336 | 336 |
| MTMamba++ [38] [TPAMI2025] | Swin-L | 81.94 | 72.87 | 85.56 | 14.29 | 78.60 | 6.15 | 343 | 343 |
| BIM [39] [ICCV2025] | Swin-L | 81.25 | 73.15 | 84.14 | 14.13 | 79.05 | 5.99 | 388 | 388 |
| InvPT [31] [ECCV2022] | ViT-L | 79.03 | 67.61 | 84.81 | 14.15 | 73.00 | 2.52 | 423 | 423 |
| InvPT++ [32] [TPAMI2024] | ViT-L | 80.44 | 69.19 | 84.85 | 13.70 | 74.20 | 4.12 | 421 | 421 |
| TaskPrompter [2] [ICLR2023] | ViT-L | 80.89 | 68.89 | 84.83 | 13.72 | 73.50 | 3.93 | 401 | 401 |
| TaskExpert [33] [ICCV2023] | ViT-L | 80.64 | 69.42 | 84.87 | 13.56 | 74.20 | 4.39 | 420 | 420 |
| MLoRE [34] [CVPR2024] | ViT-L | 81.41 | 70.52 | 84.90 | 13.51 | 75.42 | 5.26 | 407 | 407 |
| TaskDiffusion [35] [ICLR2025] | ViT-L | 81.21 | 69.62 | 84.94 | 13.55 | 74.89 | 4.80 | 416 | 416 |
| BridgeNet [42] [TPAMI2025] | ViT-L | 80.64 | 70.06 | 84.64 | 13.82 | 72.96 | 3.87 | 477 | 477 |
| SAK [82] [ICLR2025] | ViT-L | 84.01 | 76.99 | 84.65 | 13.82 | 76.27 | 7.40 | 407 | 407 |
| D²MoE [Ours] | ViT-L | 84.48 | 78.05 | 85.55 | 13.57 | 78.40 | 8.88 | 358 | 54 |
Table 1. Performance comparison on the PASCAL-Context datasets. ↑/↓ indicates that the higher/lower results represent better performance.
NYUD-v2. Table 2 reports quantitative results on NYUD-v2 for Sem. Seg, Dep., Norm. and Edge. MoE achieves the highest aggregated multi-task gain (), with improvements observed across all tasks. Relative to the MTL baseline (ViT-L), performance increases from 63.15 to 65.91 mIoU for Sem. Seg and from 75.10 to 80.60 odsF for Edge, while Dep. RMSE is reduced from 0.5119 to 0.4209 and Norm. from 18.81 to 15.82 mErr. Compared with recent strong baselines (e.g., BIM, TaskDiffusion), MoE shows a larger overall gain, with particularly pronounced benefits on geometry-related predictions, consistent with the importance of modelling indoor structure and fine-grained details.
| Methods | Backbone | Sem. Seg. mIoU↑ | Dep. RMSE↓ | Norm. mErr↓ | Edge odsF↑ | (%) ↑ |
| MTL Baseline | ViT-L | 63.15 | 0.5119 | 18.81 | 75.10 | 0.00 |
| MTI-Net [60] [ECCV2024] | HRNet-18 | 45.97 | 0.5365 | 20.27 | 77.84 | -9.03 |
| PAD-Net [58] [CVPR2018] | HRNet-18 | 36.61 | 0.6270 | 20.85 | 76.38 | -18.41 |
| ATRC [64] [ICCV2021] | HRNet-18 | 46.33 | 0.5363 | 20.18 | 77.94 | -8.73 |
| ASTMT [81] [ECCV2024] | ResNet-50 | 32.16 | 0.5700 | 23.18 | 74.50 | -21.11 |
| MTMamba [37] [ECCV2024] | Swin-L | 55.82 | 0.5066 | 18.63 | 78.70 | -1.21 |
| MTMamba++ [38] [TPAMI2025] | Swin-L | 57.01 | 0.4818 | 18.27 | 79.40 | 1.19 |
| BIM [39] [ICCV2025] | Swin-L | 57.40 | 0.4733 | 18.55 | 78.72 | 1.16 |
| InvPT [31] [ECCV2024] | ViT-L | 53.56 | 0.5183 | 19.04 | 78.10 | -3.42 |
| InvPT++ [32] [ECCV2024] | ViT-L | 53.89 | 0.5091 | 18.65 | 78.10 | -2.32 |
| TaskPrompter [2] [ECCV2024] | ViT-L | 55.30 | 0.5152 | 18.47 | 78.20 | -1.78 |
| TaskExperts [33] [ECCV2024] | ViT-L | 55.35 | 0.5157 | 18.54 | 78.40 | -1.82 |
| MLoRE [34] [ECCV2024] | ViT-L | 55.96 | 0.5076 | 18.33 | 78.43 | -0.89 |
| TaskDiffusion [35] [ICLR2025] | ViT-L | 55.65 | 0.5020 | 18.43 | 78.64 | -0.80 |
| BridgeNet [42] [TPAMI2025] | ViT-L | 55.51 | 0.4930 | 18.47 | 78.22 | -0.61 |
| SAK [82] [ICLR2025] | ViT-L | 63.18 | 0.4313 | 16.25 | 79.43 | 8.79 |
| DMoE [Ours] | ViT-L | 65.91 | 0.4209 | 15.82 | 80.60 | 11.34 |
Table 2. Performance comparison on the NYUDv2 dataset. ↑/↓ indicates that the higher/lower results represent better performance.
Cityscapes. We further evaluate MoE on Cityscapes for joint Sem. Seg. and Dep. (Table 3). MoE attains 82.10 mIoU and 4.22 RMSE, corresponding to a overall gain with the ViT-L baseline (76.45 mIoU and 4.36 RMSE). Notably, several existing approaches yield limited or negative overall gains under the same protocol, reflecting the optimisation trade-offs between semantic recognition and geometric estimation in urban driving scenes. We further observe that the relative advantage of MoE is more evident on Cityscapes than on NYUD-v2 and Pascal-Context. This may stems from its regular driving layouts and tight semantic–geometric coupling, where roads, vehicles, and buildings follow stable spatial/depth patterns, making DINOv3 priors and task-aware routing more effective.
Table 3. Performance comparison on the Cityscapes dataset.
Across Pascal-Context, NYUD-v2 and Cityscapes, MoE consistently yields the largest aggregated multi-task gain , despite substantial differences in visual domains and task compositions. This pattern strongly supports our motivation that effective MTDP with a frozen foundation backbone requires (i) task-generic adaptation to align pretrained representations with dense prediction, and (ii) sparse, task-adaptive specialization to resolve conflicts.
Qualitative Comparison
We present qualitative results on Pascal-Context and NYUD-v2 to visually assess MoE. Figure 5 and Figure 6 compare MoE with strong recent baselines (TaskDiffusion and MTMamba) under the same multi-task setting.

Figure 5. Qualitative Comparison on Pascal-Context. Compared with TaskDiffusion [35] and MTMamba [37], MoE produces crisper boundaries and fewer cross-task inconsistencies (e.g., aligned silhouettes and object extents across parsing/segmentation/saliency and normals), indicating improved shared structure modeling and reduced task interference in multi-objective dense prediction.

Figure 6. Qualitative Comparison on NYUD-v2. Compared with TaskDiffusion [35] and MTMamba [37], DMoE better preserves depth/normal discontinuities at object boundaries and yields structurally coherent geometry that agrees with segmentation, suggesting stronger cross-task geometric consistency across heterogeneous dense objectives.
Pascal-Context. As shown in Figure 5, MoE yields more accurate and spatially coherent predictions across tasks. For Sem. Seg., MoE better preserves object extent and thin structures (e.g., limbs), reducing the label fragmentation observed in competing methods. For Norm., MoE produces smoother orientation fields with sharper transitions at geometric discontinuities, indicating stronger local shape consistency. For Edge, MoE outputs sharper and more continuous contours aligned with object silhouettes. For Sal., MoE improves foreground–background separation with fewer spurious activations.
NYUD-v2. Figure 6 demonstrates gains in indoor scene parsing. MoE yields Depth maps with refined geometry and reliable occlusion ordering, reducing planar noise. Norm. maps exhibit spatial consistency with crisp boundary transitions. For Sem. Seg., MoE successfully disambiguates adjacent objects via DINOv3 semantic priors. Finally, Edge results show improved completeness and connectivity even in cluttered environments.

Figure 7. PCA Visualizations on Pascal-Context (Left) and NYUD-v2 (Right). PCA visualizations are provided to analyze feature representations learned with TA-MoE, TG-LoRA, and the Semantic Prior Injector (SPI). As shown, features extracted from different network stages exhibit progressively clearer task-specific structures, demonstrating the effectiveness of the proposed modules in enhancing task-aware representation learning across both datasets.
Ablation Study
We conduct comprehensive ablation studies on the Pascal-Context dataset. In particular, § 5.4.1 isolates the contributions of our proposed modules and their placement, whereas § 5.4.2 examines the key hyperparameters that operationalize the intended balance between conditional specialization and parameter efficiency.
Component and Design Ablations
Overall Components Analysis. Table 4 disentangles the contributions of the Mixture of Task Adapters (MoTA) and Semantic Prior Injector (SPI) to our prior-preserving multitask framework. Relative to the MTL baseline, MoTA alone increases the aggregated gain from 0.00 to 6.10%, with marked improvements in Sal. (82.61 → 84.12 mIoU) and Norm. (15.60 → 14.01 mErr), supporting the need for task-conditioned adaptation to separate dense objectives without perturbing the frozen backbone. SPI alone provides a smaller gain (), implying that semantic priors are insufficient to resolve cross-task interference under a single shared adaptation path. Enabling both yields the best performance, i.e.84.48 mIoU (segmentation), 78.05 mIoU (parsing), 85.55 maxF (saliency), and , consistent with complementarity: MoTA supplies controlled specialization, whereas SPI re-injects frozen semantic structure to improve global coherence and boundary fidelity.
| MoTA | SPI | Sem. Seg. mIoU↑ | H. Pars. mIoU↑ | Sal. maxF↑ | Norm. mErr↓ | Edge odsF↑ | (%) ↑ |
| ✗ | ✗ | 82.61 | 76.38 | 77.09 | 17.63 | 74.00 | 0.00 |
| ✗ | ✓ | 83.27 | 77.21 | 82.51 | 15.24 | 78.30 | 5.66 |
| ✓ | ✗ | 82.44 | 74.90 | 84.12 | 14.01 | 76.20 | 6.10 |
| ✓ | ✓ | 84.48 | 78.05 | 85.55 | 13.57 | 78.40 | 8.88 |
Table 4. Overall ablation of the proposed components (MoTA and SPI) under the unified multi-task setting.
Ablation Results of MoTA. Table 5 probes MoTA’s internal factorization. Removing TG-LoRA lowers the gain to 7.74%, indicating degradation of the task-generic alignment channel that stabilizes downstream adaptation; removing TA-MoE produces a comparable reduction (), reflecting weaker task-specific discrimination without conditional expert routing. The full model outperforms both variants across tasks, showing that frozen-backbone multi-task learning benefits from jointly coupling a globally shared low-rank substrate (TG-LoRA) with sparse task-adaptive specialization
| Modules | Sem. Seg. mIoU↑ | H. Pars. mIoU↑ | Sal. maxF↑ | Norm. mErr↓ | Edge odsF↑ | (%) ↑ |
| MTL Baseline | 82.61 | 76.38 | 77.09 | 17.63 | 74.00 | 0.00 |
| w/o TG-LoRA | 84.07 | 77.92 | 84.32 | 14.15 | 78.30 | 7.74 |
| w/o TA-MoE | 83.31 | 77.60 | 84.35 | 13.65 | 78.00 | 7.97 |
| Ours | 84.48 | 78.05 | 85.55 | 13.57 | 78.40 | 8.88 |
Table 5. Component ablation of Mixture of Task Adapters (MoTA), isolating the contributions of TG-LoRA and TA-MoE under the unified multi-task setting.
is computed relative to the MTL baseline, which is the
(TA-MoE) rather than relying on either uniform adaptation or specialization alone.
Ablation Results of TA-MoE. Table 6 evaluates where specialization most effectively alleviates the single-route bottleneck. Injecting TA-MoE into attention QKV projections outperforms insertion solely in the FFN or projection layers, indicating that conditioning token mixing (task-dependent contextual aggregation) is more effective than modifying downstream feed-forward transformations. Combining QKV and projection layers achieves the maximum gain (8.88%), suggesting that co-conditioning attention and output projection best supports task disentanglement.
| QKV | Proj | FFN | Sem. Seg mIoU↑ | H. Pars. mIoU↑ | Sal. maxF↑ | Norm. mErr↓ | Edge odsF↑ | (%) ↑ |
| ✗ | ✗ | ✗ | 83.31 | 77.60 | 84.35 | 13.65 | 78.00 | 7.97 |
| ✓ | ✗ | ✗ | 83.74 | 77.80 | 85.18 | 13.57 | 78.70 | 8.62 |
| ✗ | ✓ | ✗ | 83.31 | 77.67 | 84.86 | 13.73 | 78.80 | 8.24 |
| ✗ | ✗ | ✓ | 83.34 | 77.24 | 84.63 | 13.58 | 78.00 | 8.03 |
| ✓ | ✓ | ✗ | 84.48 | 78.05 | 85.55 | 13.57 | 78.40 | 8.88 |
| ✓ | ✓ | ✓ | 83.75 | 77.51 | 85.02 | 13.51 | 79.10 | 8.68 |
Table 6. Effect of inserting the MoE module at different locations within the backbone transformer block.
Backbone Variants. Table 7 quantifies task-dependent scaling across ViT backbones. ViT-S to ViT-B gives the largest segmentation jump (80.62 85.25 mIoU) with modest parameter growth. Scaling to ViT-L shifts gains to structured prediction: H. Pars. 75.66 78.05 mIoU, Sal. and Norm. peak at 85.55 maxF and 13.57 mErr, while Sem. Seg. saturates at 84.48 mIoU. ViT-H adds slightly for H. Pars./Edge (78.87 mIoU, 79.60 odsF) but degrades Sal. (83.17 maxF) and Norm. (13.62 mErr), implying diminishing or negative returns from additional backbone capacity. This degradation likely arises because the larger ViT-H encodes stronger high-level semantic invariances, while our frozen-backbone and low-rank adaptation cannot fully recalibrate its spatial-frequency bias for saliency and normals, making these geometry-sensitive tasks prone to over-smoothing. Overall, ViT-L provides the optimal accuracy-efficiency trade-off.
| Backbone | Para. (M) | Sem. Seg. mIoU | Pars. mIoU | Sal. maxF | Norm. mErr | Edge odsF |
| ViT-S | 62.0 | 80.62 | 73.36 | 85.28 | 13.79 | 76.40 |
| ViT-B | 126.6 | 85.25 | 75.66 | 84.95 | 13.66 | 77.50 |
| ViT-L | 357.6 | 84.48 | 78.05 | 85.55 | 13.57 | 78.40 |
| ViT-H | 911.3 | 84.80 | 78.87 | 83.17 | 13.62 | 79.60 |
Table 7. Comparison of backbone variants under the same multi-task setting.
Comparison with MLoRE under a Strong Backbone. Table 8 reimplements MLoRE on DINOv3. MLoRE places experts in the decoder, improving task outputs over the MTL baseline and substantiating decoder-side specialization. DMoE instead instantiates experts in the backbone, decoupling tasks during representation learning while retaining DINOv3 priors through conditional adaptation. This shift delivers larger mean gains (=8.88% vs. 6.30%) with substantially fewer trainable parameters (54M vs. 407M). Although DMoE incurs higher FLOPs/latency than MLoRE (1029.9G/224.2ms vs. 829.9G/203.2ms), the additional computation is moderate and leads to stronger multi-task perfor… mance with a 7.5 smaller trainable footprint, demonstrating a favorable trade-off among accuracy, trainable footprint, and computational cost.
| Modules | Sem. Seg. mIoU↑ | H. Pars. mIoU↑ | Sal. maxF↑ | Norm. mErr↓ | Edge odsF↑ | (%) ↑ | Para. (M) ↓ | Train. Para. (M) ↓ | Flops (G)↓ | Latency (ms)↓ |
| MTL Baseline | 82.61 | 76.38 | 77.09 | 17.63 | 74.00 | 0.00 | 328 | 25 | 759.7 | 198.9 |
| MLoRE [34] | 83.85 | 78.05 | 84.98 | 15.51 | 78.10 | 6.30 | 407 | 407 | 829.9 | 203.2 |
| DMoE | 84.48 | 78.05 | 85.55 | 13.57 | 78.40 | 8.88 | 358 | 54 | 1029.9 | 224.2 |
Table 8. Comparison with MLoRE using a DINOv3 backbone.
Comparison with Other Variants under Matched Budgets. Tab. 9 compares three parameter- and protocol-matched controls under the same frozen DINOv3 setting. Shared LoRA uses a single task-shared LoRA () at the same QKV/projection positions, achieving . Task LoRA replaces routing with independent task-wise LoRA modules (), yielding . Decoder MoE moves the low-rank MoE to the decoder, using the same expert pool and Top-8 routing with 25.2M MoE parameters, but reaches only . In comparison, MoE achieves the best . Overall, these matched controls show that the gains of MoE are not explained by parameter count alone, but arise from combining shared low-rank alignment with sparse, learned task-aware specialization at appropriate backbone locations.
Hyperparameter Analysis
We quantify how hyperparameters shape the capacity–efficiency trade-off in Figure 8, varying TG-LoRA rank, TA-MoE expert count, TA-MoE LoRA rank, and Top- routing.

Figure 8. Hyperparameter analysis of TG-LoRA and TA-MoE. The results demonstrate clear performance trends across different LoRA ranks and expert configurations, highlighting an effective trade-off between model capacity and multi-task performance.
LoRA Rank of TG-LoRA. Increasing TG-LoRA rank from 2 to 16 (Figure 8(a)) improves all tasks (Sem. Seg. 83.40 → 84.48 mIoU, H. Pars. 77.66 → 78.05 mIoU, Sal. 84.73 → 85.55 maxF). Consistently, increases from 8.07% to 8.88%, indicating stronger overall multi-task performance. All parameters rise modestly (355.5M → 357.6M). At rank 32, gains saturate or reverse (e.g., Sem. Seg. 84.35) despite higher parameters/FLOPs, indicating over-parameterization without improved alignment.
Number of Experts in TA-MoE. Varying experts (Figure 8(b)) shows a similar saturation: 2→16 experts improves segmentation (83.47 → 84.48 mIoU) and boundary detection (78.30 → 78.50 odsF) with limited parameter growth (355.9M → 357.6M). The aggregated gain peaks at 16 experts with , suggesting the best balance between specialization capacity and efficiency. At 32 experts, performance saturates or slightly decreases (e.g., parsing 78.05 → 77.64), suggesting redundancy rather than stronger task specialization.
LoRA Rank of TA-MoE. Figure 8(c) indicates that moderate per-expert rank is sufficient: rank 4 achieves strong overall accuracy (segmentation 84.48 mIoU) and the lowest normal error (13.57 mErr) at 357.6M parameters. It also obtains the highest aggregated gain (), confirming that moderate rank is sufficient for overall multi-task capacity. Lower ranks underfit task-specific refinement, whereas higher ranks increase parameters/FLOPs with marginal, unstable gains.
Top- Value in TA-MoE. Figure 8(d) shows that increasing from 2 to 8 improves performance, peaking at 84.48 mIoU (segmentation) and 85.55 maxF (saliency) with unchanged parameters. also reaches the best value of 8.88% when . Larger (e.g., 12 or 16) yields marginal gains or mild degradation, consistent with reduced sparsity-driven specialization under broader expert activation.
Collectively, these ablations identify a coherent regime where improvements arise from structured task-adaptive design, including TG-LoRA alignment, sparse TA-MoE specialization, and SPI prior injection, rather than from indiscriminate increases in trainable capacity.
Feature Visualization and Analysis
Feature Embedding Visualization (t-SNE). Figure 9 shows 3D t-SNE projections of the learned feature embeddings, directly comparing the MTL baseline (top row) with MoE (bottom

Figure 9. 3D t-SNE Visualizations of Feature Embeddings. We compare the MTL baseline (top) with our MoE framework (bottom) across three benchmarks. (a) Semantic Segmentation: Points are colored by class label. MoE achieves significantly higher intra-class compactness and inter-class separability than the baseline. (b) Depth Estimation: Points are colored by discretized depth intervals (see legend). Our model yields more structured, monotonic groupings of scene geometry. Axes denote the three primary t-SNE components.
row). DMoE reshapes the embedding manifold into task-aligned submanifolds rather than an entangled cloud. On Sem. Seg. (Pascal-Context, NYUD-v2, Cityscapes), DMoE forms tighter, well-separated clusters with higher intra-class compactness and reduced cross-class overlap than the MTL baseline, reflecting increased inter-class variance and less negative transfer. This behavior matches selective routing within a shared backbone: early layers preserve broadly reusable cues (edges, textures), while routed experts enhance class-conditional directions, widening margins between semantic modes while maintaining the global manifold structure. Regarding depth estimation, DMoE yields clearer and more structured groupings, suggesting routing induces a more geometry-aware embedding where features are organized topologically according to their respective depth bins, a marked improvement over the disordered feature clouds observed in the baseline.
| Method | Sem. Seg. mIoU↑ | H. Pars. mIoU↑ | Sal. maxF↑ | Norm. mErr↓ | Edge odsF↑ | (%) ↑ | Backbone Train. (M)↓ | Decoder Train. (M)↓ | Train. Para. (M)↓ | Para. (M)↓ |
| MTL Baseline | 82.61 | 76.38 | 77.09 | 17.63 | 74.00 | 0.00 | 25.2 | 0 | 25.2 | 328.4 |
| Shared LoRA | 83.60 | 77.52 | 85.19 | 13.51 | 78.30 | 8.48 | 25.2 | 29.2 | 54.4 | 357.6 |
| Task LoRA | 84.25 | 77.95 | 84.65 | 13.77 | 78.30 | 8.31 | 25.2 | 29.2 | 54.4 | 357.6 |
| Decoder MoE | 83.22 | 78.00 | 82.46 | 15.16 | 78.20 | 5.90 | 0 | 54.4 | 54.4 | 357.6 |
| DMoE | 84.88 | 78.05 | 85.55 | 13.57 | 78.40 | 8.88 | 25.2 | 29.2 | 54.4 | 357.6 |
Table 9. Comparison with shared LoRA, task-specific LoRA and decoder-side MoE under matched trainable-parameter budgets.
Router Correlation Analysis. Figure 10 reveals stage- and operator-specific specialization. In QKV projections, early stages display strong cross-task coupling (off-diagonal correlations), whereas deeper stages become near diagonal-dominant, implying DMoE progressively privatizes attention access patterns (queries/keys/values) to mitigate negative transfer in information selection. Conversely, attention output projections exhibit structured, non-monotonic sharing: an early block structure (notably among Hum./Nor./Sem.) persists, and late-stage correlations remain strong for specific pairs (e.g., Nor.–Sem., Hum.–Sal.), indicating DMoE avoids forced full separation and reuses experts when supervision

Figure 10. Router visualization of TA-MoE across backbone stages. We visualize the task-wise expert selection patterns of TA-MoE inserted at different backbone stages and positions. For each task, we aggregate expert activation values over all samples in the Pascal-Context dataset and compute the cosine similarity between task-specific activation vectors. The resulting similarity matrices reveal how task relationships evolve across stages and insertion locations.
is complementary. Overall, TA-MoE specializes where tasks attend (QKV) while selectively sharing what is read out (output projection), disentangling conflicts while enabling transfer when tasks align.
Robustness to Distribution Shift
We assess domain generalization on Pascal-Context under the corruption benchmark of ImageNet [84]. Figure 11 reports corruption-wise performance across severities (1/3/5), and Figure 12 visualizes level-5 out-of-distribution (OOD) predictions against TaskDiffusion [35] and MTLTamba [37]. To separate corruption sensitivity from gains due to higher clean accuracy, Figure 13 additionally reports clean-normalized performance retention, while Table 10 reports severity-5 results averaged over all 15 corruptions.

Figure 11. Corruption Robustness Evaluation on Pascal-Context. Across five tasks, performance decreases with corruption severity, with zoom blur and gaussian noise inducing the largest degradations (e.g., Sem. Seg. drops from 84.48 on clean to 47.50 at Level 5). Weather/digital corruptions are comparatively mild at Levels 1–3, while normal error rises primarily under noise.

Figure 12. Corruption Robustness Comparison with TaskDiffusion [35]. All examples are drawn from the Pascal-Context dataset under level-5 corruptions. Our method demonstrates stronger robustness to severe degradations, yielding cleaner object boundaries and more stable predictions.
| Method | Sem. Seg. mIoU↑ | H. Pars. mIoU↑ | Sal. maxF↑ | Norm. mErr↓ | Edge odsF↑ |
| MTL Baseline | 65.36 | 64.54 | 70.08 | 22.45 | 63.07 |
| TaskDiffusion | 57.01 | 49.48 | 67.42 | 22.81 | 59.04 |
| DMoE | 68.08 | 67.03 | 72.77 | 21.47 | 64.96 |
Table 10. Averaged robustness comparison on PASCAL-Context across all corruptions.
Figure 13. Clean-normalized Performance Retention under Corruption. Task-wise retention for the MTL baseline, DMoE, and TaskDiffusion. Avg. denotes mean relative degradation from clean performance.
Across all tasks, the dominant trend in Figure 11 is graded performance change with increasing corruption severity, indicating stable behavior under covariate shift. Noise corruptions are consistently the most damaging: Sem. Seg. drops from 82.34 (Gaussian, level-1) to 47.50 (level-5), and H. Pars. from 75.98 to 58.75 under the same shift, while surface-normal error increases smoothly from 16.52 (clean) to 33.42 (Gaussian, level-5). In the blur family, zoom blur is the primary failure case across tasks (e.g., Sem. Seg. 71.25 → 47.60, H. Pars. 65.62 → 49.67, Edge 53.20 → 40.10 from level-1 to level-5), whereas defocus/motion blur produce substantially smaller degradations. Weather and digital corruptions are comparatively benign for most tasks: for Sem. Seg., level-5 contrast remains close to clean (81.95 vs. 84.48), and saliency stays high across many level-5 weather/digital shifts, demonstrating broad OOD robustness beyond a single corruption type.
Table 10 shows that DMoE achieves the best average corrupted performance across all five tasks, outperforming both the MTL baseline and TaskDiffusion. Figure 13 provides a complementary normalized comparison. DMoE shows smaller average degradation than TaskDiffusion at all severities (−5.02% / −12.89% / −24.69% vs. −5.36% / −13.72% / −27.62%), with stronger retention particularly on Sem. Seg. and H. Pars., whereas the MTL baseline exhibits smaller overall degradation (−1.89% / −9.39% / −17.52%). Thus, DMoE’s superior corrupted performance stems from both stronger clean performance and improved retention over TaskDiffusion, rather than uniformly lower corruption sensitivity.
Figure 12 qualitatively confirms these trends under severe shift. Under contrast change and defocus blur, DMoE preserves coherent region topology in Sem. Seg. and H. Pars. and maintains continuous edge structure. Even under Gaussian noise which corresponds to the largest metric drops, MoE retains structured masks, plausible part layouts, and recognizable boundaries, whereas TaskDiffusion [35] exhibits region fragmentation, unstable part layouts, and degraded boundary continuity. Compared with MTMamba [37], MoE better preserves foreground completeness and task-consistent structures, avoiding over-smoothing and missing responses, especially in saliency and edge predictions.
Overall, the absolute and clean-normalized results provide complementary evidence: MoE maintains stronger absolute OOD performance and degrades less than TaskDiffusion on average, while the MTL baseline exhibits smaller overall normalized degradation despite MoE showing better retention on selected tasks.
Conclusion
In this work, we addressed the challenge of multi-task dense prediction in the vision foundation model paradigm, where heterogeneous tasks must be adapted without compromising the robustness of pretrained representations. We proposed MoE, which introduces a structured Mixture of Task Adapters to decouple task-generic alignment from task-specific specialization atop a frozen DINOv3. By combining Task-Generic Low-Rank Adaptation across all attention layers with a Task-Adaptive Mixture of Experts selectively applied in later stages, MoE resolves task conflicts while preserving a shared semantic prior within a single forward pass. In addition, semantic prior injection further improves structural coherence and cross-task consistency. Extensive experiments across diverse benchmarks demonstrate that this design effectively mitigates negative transfer and achieves strong multi-task performance with high parameter efficiency. Overall, our results highlight the importance of layer-aware, conditional adaptation for unlocking the full potential of vision foundation models in multi-task dense prediction.
References
References
- [1]S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, and L. Van Gool, “Multi-task learning for dense prediction tasks: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 7, pp. 3614–3633, 2021.DOI
- [2]H. Ye and D. Xu, “Taskprompter: Spatial-channel multi-task prompting for dense scene understanding,” in Int. Conf. Learn. Represent., 2023.
- [3]F. Jiang, S. Wang, and X. Gong, “Task-conditional adapter for multi-task dense prediction,” in ACM Int. Conf. Multimedia, 2024, pp. 2059–2068.DOI
- [4]J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3431–3440.
- [5]E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 4, pp. 640–651, 2017.
- [6]H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in IEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 2881–2890.
- [7]V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 12, pp. 2481–2495, 2017.DOI
- [8]L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834–848, 2018.DOI
- [9]S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. Torr et al., “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in IEEE Conf. Comput. Vis. Pattern Recog., 2021, pp. 6881–6890.arxiv.org/abs/2012.15840
- [10]R. Strudel, R. Garcia, I. Laptev, and C. Schmid, “Segmenter: Transformer for semantic segmentation,” in Int. Conf. Comput. Vis., 2021, pp. 7262–7272.arxiv.org/abs/2105.05633
- [11]E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” Adv. Neural Inform. Process. Syst., vol. 34, pp. 12 077–12 090, 2021.arxiv.org/abs/2105.15203
- [12]T. Zhou, W. Wang, E. Konukoglu, and L. Van Gool, “Rethinking semantic segmentation: A prototype view,” in IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 2582–2593.arxiv.org/abs/2203.15102
- [13]Z. Huang, X. Wang, L. Wei, L. Huang, H. Shi, W. Liu, and T. S. Huang, “Ccnet: Criss-cross attention for semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 6, pp. 6896–6908, 2023.
- [14]Y. Zhu, Z. Zhang, C. Wu, Z. Zhang, T. He, H. Zhang, R. Manmatha, M. Li, and A. Smola, “Improving semantic segmentation via efficient self-training,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 3, pp. 1589–1602, 2024.
- [15]F. Liu, C. Shen, and G. Lin, “Deep convolutional neural fields for depth estimation from a single image,” in IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 5162–5170.arxiv.org/abs/1411.6387
- [16]H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in IEEE Conf. Comput. Vis. Pattern Recog., 2018, pp. 2002–2011.arxiv.org/abs/1806.02446
- [17]A. Gordon, H. Li, R. Jonschkowski, and A. Angelova, “Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras,” in Int. Conf. Comput. Vis., 2019, pp. 8977–8986.arxiv.org/abs/1904.04998
- [18]S. F. Bhat, I. Alhashim, and P. Wonka, “Adabins: Depth estimation using adaptive bins,” in IEEE Conf. Comput. Vis. Pattern Recog., 2021, pp. 4009–4018.DOI
- [19]J.-W. Bian, H. Zhan, N. Wang, Z. Li, L. Zhang, C. Shen, M.-M. Cheng, and I. Reid, “Unsupervised scale-consistent depth learning from video,” Int. J. Comput. Vis., vol. 129, no. 9, pp. 2548–2564, 2021.DOI
- [20]L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 10 371–10 381.arxiv.org/abs/2401.10891
- [21]L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,” Adv. Neural Inform. Process. Syst., vol. 37, pp. 21 875–21 911, 2024.arxiv.org/abs/2406.09414
- [22]W. Wang, Q. Lai, H. Fu, J. Shen, H. Ling, and R. Yang, “Salient object detection in the deep learning era: An in-depth survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 6, pp. 3239–3259, 2022.DOI
- [23]N. Liu and J. Han, “Dhsnet: Deep hierarchical saliency network for salient object detection,” in IEEE Conf. Comput. Vis. Pattern Recog., June 2016.DOI
- [24]L. Wang, H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, and X. Ruan, “Learning to detect salient objects with image-level supervision,” in IEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 136–145.DOI
- [25]Q. Hou, M.-M. Cheng, X. Hu, A. Borji, Z. Tu, and P. H. S. Torr, “Deeply supervised salient object detection with short connections,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 4, pp. 815–828, 2019.
- [26]Y. Zhuge, Y. Zeng, and H. Lu, “Deep embedding features for salient object detection,” in Proc. AAAI Conf. Artif. Intell., vol. 33, no. 01, 2019, pp. 9340–9347.DOI
- [27]H. Chen, Y. Li, Y. Deng, and G. Lin, “Cnn-based rgb-d salient object detection: Learn, select, and fuse,” Int. J. Comput. Vis., vol. 129, no. 7, pp. 2076–2096, 2021.arxiv.org/abs/1909.09309
- [28]M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in Int. Conf. Comput. Vis., 2021, pp. 9650–9660.arxiv.org/abs/2104.14294
- [29]M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193, 2023.arxiv.org/abs/2304.07193
- [30]O. Simeoni, H. V. Mo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa et al., “Dinov3,” arXiv preprint arXiv:2508.10104, 2025.arxiv.org/abs/2508.10104
- [31]H. Ye and D. Xu, “Inverted pyramid multi-task transformer for dense scene understanding,” in Eur. Conf. Comput. Vis. Springer, 2022, pp. 514–530.DOI
- [32]H. Ye and D. Xu, “Invpt++: Inverted pyramid multi-task transformer for visual scene understanding,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 12, pp. 7493–7508, 2024.DOI
- [33]H. Ye and D. Xu, “Taskexpert: Dynamically assembling multi-task representations with memorial mixture-of-experts,” in Int. Conf. Comput. Vis., 2023, pp. 21 828–21 837.arxiv.org/abs/2307.15324
- [34]Y. Yang, P.-T. Jiang, Q. Hou, H. Zhang, J. Chen, and B. Li, “Multi-task dense prediction via mixture of low-rank experts,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 27 927–27 937.arxiv.org/abs/2403.17749
- [35]Y. Yang, P.-T. Jiang, Q. Hou, H. Zhang, J. Chen, and B. Li, “Multi-task dense predictions via unleashing the power of diffusion,” in Int. Conf. Learn. Represent., 2025.
- [36]Y. Lu, S. Cao, and Y.-X. Wang, “Swiss army knife: Synergizing biases in knowledge from vision foundation models for multi-task learning,” in Int. Conf. Learn. Represent., 2025. [Online]. Available: https://openreview.net/forum?id=eePWw5u7J3arxiv.org/abs/2410.14633
- [37]B. Lin, W. Jiang, P. Chen, Y. Zhang, S. Liu, and Y.-C. Chen, “Mtmamba: Enhancing multi-task dense scene understanding by mamba-based decoders,” in Eur. Conf. Comput. Vis., Springer, 2024, pp. 314–330.arxiv.org/abs/2407.02228
- [38]B. Lin, W. Jiang, P. Chen, S. Liu, and Y.-C. Chen, “Mtmamba++: Enhancing multi-task dense scene understanding via mamba-based decoders,” IEEE Trans. Pattern Anal. Mach. Intell., 2025.DOI
- [39]M. Cao, S. Zhou, Y. Li, Y. Deng, W. Huang, and L. Wang, “Enhancing mamba decoder with bidirectional interaction in multi-task dense prediction,” in Int. Conf. Comput. Vis., 2025, pp. 18 815–18 824.arxiv.org/abs/2508.20376
- [40]W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” J. Mach. Learn. Res., vol. 23, no. 120, pp. 1–39, 2022.arxiv.org/abs/2101.03961
- [41]N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat et al., “Glam: Efficient scaling of language models with mixture-of-experts,” in Int. Conf. Mach. Learn. PMLR, 2022, pp. 5547–5569.arxiv.org/abs/2112.06905
- [42]J. Zhang, J. Fan, P. Ye, B. Zhang, H. Ye, B. Li, Y. Cai, and T. Chen, “Bridgetnet: Comprehensive and effective feature interactions via bridge feature for multi-task dense predictions,” IEEE Trans. Pattern Anal. Mach. Intell., 2025.DOI
- [43]D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” in Int. Conf. Learn. Represent., 2021.
- [44]C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” Adv. Neural Inform. Process. Syst., vol. 34, pp. 8583–8595, 2021.arxiv.org/abs/2106.05974
- [45]Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Q. V. Le, J. Laudon et al., “Mixture-of-experts with expert choice routing,” Adv. Neural Inform. Process. Syst., vol. 35, pp. 7103–7114, 2022.arxiv.org/abs/2202.09368
- [46]A. Agiza, M. Naseem, and S. Reda, “Mtlora: Low-rank adaptation approach for efficient multi-task learning,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 16 196–16 205.arxiv.org/abs/2403.20320
- [47]Y. Lu, S. Sejrijeding, Y. Ding, C. Wang, and H. Lu, “Prompt guided transformer for multi-task dense prediction,” IEEE Trans. Multimedia, vol. 26, pp. 6375–6385, 2024.arxiv.org/abs/2307.15362
- [48]N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in Eur. Conf. Comput. Vis. Springer, 2012, pp. 746–760.DOI
- [49]Y.-C. Liu, C.-Y. Ma, J. Tian, Z. He, and Z. Kira, “Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks,” Adv. Neural Inform. Process. Syst., vol. 35, pp. 36 889–36 901, 2022.arxiv.org/abs/2210.03265
- [50]Y. Xin, J. Du, Q. Wang, Z. Lin, and K. Yan, “Vmt-adapter: Parameter-efficient transfer learning for multi-task dense scene understanding,” in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 14, 2024, pp. 16 085–16 093.arxiv.org/abs/2312.08733
- [51]S. Baek, S. Lee, H. Jo, H. Choi, and D. Min, “Tadformer: Task-adaptive dynamic transformer for efficient multi-task learning,” in IEEE Conf. Comput. Vis. Pattern Recog., 2025, pp. 14 858–14 868.arxiv.org/abs/2501.04293
- [52]K. S. I. Mantri, C.-B. Schönlieb, B. Ribeiro, C. Baskin, and M. Eliasof, “Ditask: Multi-task fine-tuning with diffeomorphic transformations,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 25 218–25 229.arxiv.org/abs/2502.06029
- [53]Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, Z. Wang et al., “M3vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design,” Adv. Neural Inform. Process. Syst., vol. 35, pp. 28 441–28 457, 2022.DOI
- [54]Z. Chen, Y. Shen, M. Ding, Z. Chen, H. Zhao, E. G. Learned-Miller, and C. Gan, “Mod-squad: Designing mixtures of experts as modular multi-task learners,” in IEEE Conf. Comput. Vis. Pattern Recog., 2023, pp. 11 828–11 837.DOI
- [55]T. Chen, X. Chen, X. Du, A. Rashwan, F. Yang, H. Chen, Z. Wang, and Y. Li, “Adamv-moe: Adaptive multi-task vision mixture-of-experts,” in Int. Conf. Comput. Vis., 2023, pp. 17 346–17 357.DOI
- [56]I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in IEEE Conf. Comput. Vis. Pattern Recog., 2016, pp. 3994–4003.arxiv.org/abs/1604.03539
- [57]I. Kokkinos, “Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory,” in IEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 6129–6138.arxiv.org/abs/1609.02132
- [58]D. Xu, W. Ouyang, X. Wang, and N. Sebe, “Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing,” in IEEE Conf. Comput. Vis. Pattern Recog., 2018, pp. 675–684.arxiv.org/abs/1805.04409
- [59]S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” in IEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 1871–1880.arxiv.org/abs/1803.10704
- [60]S. Vandenhende, S. Georgoulis, and L. Van Gool, “Mti-net: Multi-scale task interaction networks for multi-task learning,” in Eur. Conf. Comput. Vis. Springer, 2020, pp. 527–543.arxiv.org/abs/2001.06902
- [61]A. R. Zamir, A. Sax, W. Shen, L. J. Guibas, J. Malik, and S. Savarese, “Taskonomy: Disentangling task transfer learning,” in IEEE Conf. Comput. Vis. Pattern Recog., June 2018, pp. 3712–3722.
- [62]X. Xu et al., “Mtformer: Multi-task learning via transformer and cross-task reasoning,” in Eur. Conf. Comput. Vis., 2022.DOI
- [63]S. Yang, H. Ye, and D. Xu, “Contrastive multi-task dense prediction,” in Proc. AAAI Conf. Artif. Intell., 2023.arxiv.org/abs/2307.07934
- [64]D. Brüggemann, M. Kanakis, A. Obukhov, S. Georgoulis, and L. Van Gool, “Exploring relational context for multi-task dense prediction,” in Int. Conf. Comput. Vis., 2021, pp. 15 869–15 878.
- [65]W.-H. Li, X. Liu, and H. Bilen, “Learning multiple dense prediction tasks from partially annotated data,” in IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 18 789–18 889.arxiv.org/abs/2111.14893
- [66]Y. Xu, Y. Yang, and L. Zhang, “Multi-task learning with knowledge distillation for dense prediction,” in Int. Conf. Comput. Vis., 2023, pp. 21 550–21 559.DOI
- [67]H. Huang, Y. Huang, L. Lin, R. Tong, Y.-W. Chen, H. Zheng, Y. Li, and Y. Zheng, “Going beyond multi-task dense prediction with synergy embedding models,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 28 181–28 190.DOI
- [68]Y. Shang, D. Xu, G. Liu, R. R. Kompella, and Y. Yan, “Efficient multitask dense predictor via binarization,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 15 899–15 908.arxiv.org/abs/2405.14136
- [69]M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: A survey and outlook,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 4, pp. 2245–2264, 2025.DOI
- [70]A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.arxiv.org/abs/2010.11929
- [71]K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 16 000–16 009.arxiv.org/abs/2111.06377
- [72]A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., “Segment anything,” in Int. Conf. Comput. Vis., 2023, pp. 4015–4026.arxiv.org/abs/2304.02643
- [73]S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He, “Deepspeed-moe: Advancing mixture-of-expert inference and training to power next-generation ai at scale,” in Int. Conf. Mach. Learn. PMLR, 2022, pp. 18 332–18 346.arxiv.org/abs/2201.05596
- [74]B. Mustafa, C. Riquelme, J. Puigcerver, R. Jenatton, and N. Houlsby, “Multimodal contrastive learning with limoe: the language-image mixture of experts,” Adv. Neural Inform. Process. Syst., vol. 35, pp. 9564–9576, 2022.arxiv.org/abs/2206.02770
- [75]M.-Z. Xu, Z. Zhu, W. Wang, X. Wang, H. Li, X. Wang, and J. Dai, “Uni-perceiver-moe: Learning sparse generalist models with conditional moes,” Adv. Neural Inform. Process. Syst., vol. 35, pp. 2664–2678, 2022.
- [76]L. Shen, G. Chen, R. Shao, W. Guan, and L. Nie, “Mome: Mixture of multimodal experts for generalist multimodal large language models,” Adv. Neural Inform. Process. Syst., vol. 37, pp. 42 048–42 070, 2024.arxiv.org/abs/2407.12709
- [77]Z. Zong, B. Ma, D. Shen, G. Song, H. Shao, D. Jiang, H. Li, and Y. Liu, “Mova: Adapting mixture of vision experts to multimodal context,” Adv. Neural Inform. Process. Syst., vol. 37, pp. 103 305–103 333, 2024.arxiv.org/abs/2404.13046
- [78]Y. Li, S. Jiang, B. Hu, L. Wang, W. Zhong, W. Luo, L. Ma, and M. Zhang, “Uni-moe: Scaling unified multimodal llms with mixture of experts,” IEEE Trans. Pattern Anal. Mach. Intell., 2025.DOI
- [79]J. Yu, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He, “Boosting continual learning of vision-language models with mixture-of-experts adapters,” in IEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 23 219–23 230.arxiv.org/abs/2403.11549
- [80]J. Yu, Z. Huang, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He, “Moe-adapters++: Toward more efficient continual learning of vision-language models via dynamic mixture-of-experts adapters,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 12, pp. 11 912–11 928, 2025.DOI
- [81]K.-K. Maninis, I. Radosavovic, and I. Kokkinos, “Attentive single-tasking of multiple tasks,” in IEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 1851–1860.arxiv.org/abs/1904.08918
- [82]Y. Lu, S. Cao, and Y.-X. Wang, “Swiss army knife: Synergizing biases in knowledge from vision foundation models for multi-task learning,” in Int. Conf. Learn. Represent., 2025.arxiv.org/abs/2410.14633
- [83]X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, and A. Yuille, “Detect what you can: Detecting and representing objects using holistic models and body parts,” in IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 1971–1978.arxiv.org/abs/1406.2031
- [84]D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” in Int. Conf. Learn. Represent., 2019.
