betaFeedback

WorksComputer Science

HDVS: semi-supervised semantic segmentation via heterogeneous dual-branch voting supervision

Abstract

Semi-supervised semantic segmentation, which aims to utilize large volumes of unlabeled images to achieve accurate segmentation results with fewer human annotations, has attracted increasing attention. Prior methods were primarily based on the assumption that the pseudo-labels generated by each branch were sufficiently reliable to supervise the remaining branches. However, there is a concern that the errors in pseudo-labels could accumulate during co-training, potentially resulting in suboptimal performance. To this end, we present a novel framework called heterogeneous dual-branch voting supervision (HDVS), which is designed to enhance the reliability of pseudo-labels and mitigate the issues arising from pseudo-labeling. Specifically, based on the cross-supervision framework, we introduce a voting mechanism that correlates pseudo-labels from heterogeneous branches to produce pseudo-labels with enhanced reliability. Concurrently, we employ a feature communication module to introduce perturbations at the feature level in each branch to maximize prediction diversity across the dual branches while maintaining convergence. Comprehensive evaluations of the proposed HDVS on two benchmark datasets (PASCAL VOC 2012 and Cityscapes) demonstrate its superiority to the state-of-the-art approaches.

Introduction

Recent advancements in artificial intelligence have been driven by the development of deep neural networks and the availability of large-scale, well-annotated training datasets [1, 2, 3, 4, 5, 6]. However, collecting high-quality annotations, particularly densely pixel-wise labels required for semantic segmentation, demands considerable effort [7, 8, 9, 10]. This challenge has spurred a growing trend in research towards semi-supervised semantic segmentation, which leverages a combination of a few labeled and many unlabeled images to enhance segmentation performance [11].

In the realm of semi-supervised semantic segmentation, a significant research direction involves the use of teacher-student networks [12, 13, 14, 15, 16, 17, 18]. As illustrated in Figure 1(a), methods such as U2^{2}PL [13] utilize a teacher network to segment unlabeled data and generate segmentation maps, from which pseudo-labels are derived. The parameters are continuously updated through the exponential moving average (EMA) derived from the student network. However, relying solely on a single network for pseudo-labels reconstruction will bring noise and incur confirmation biases [19, 20]. The co-training pipeline, as proposed by Chen et al. [14] and depicted in Figure 1(b), introduces the cross pseudo supervision (CPS) strategy on two networks with identical architectures and different initializations to capture various characteristics for pseudo-labels. Subsequent advancements to CPS enhance its capabilities by introducing models with distinct initialization [21] and applying multiple perturbations [22] to expand the diversity of feature perturbation. Recently, Li et al. [15] introduced a method to enhance diversity in co-training across input domains, augmentations, and architectures, showcasing the significant impact of diversity on achieving high-performance segmentation, with the architecture illustrated in Figure 1(c). However, these methods have not fully exploited the deep collaboration among their diverse components, often resulting in unreliable pseudo-labels construction. As illustrated in Figure 1(d), our key insight is to integrate complementary cues from heterogeneous networks through dynamic, confidence-aware voting, which exposes regions of consistently high/low confidence while letting each branch’s inductive bias refine supervision.

Figure 1. Overview of common semi-supervised frameworks. (a) FixMatch [12] structure used in U2U^{2}PL [13], (b) cross pseudo supervision (CPS) [14], (c) diverse co-training with discrete cosine transform (DCT) for image processing [15], and (d) our heterogeneous dual-branch voting supervision (HDVS) method where two models use voting to generate pseudo-labels. uu denotes an unlabeled image, AA and A+A_{+} denote weak and strong data augmentations, M1M^{1} and M2M^{2} are the two segmentation models, and MTeacherM_{\mathrm{Teacher}} denotes the teacher model. Trans and Conv denote the Transformer and convolutional neural network (CNN) branches, pp denotes their predictions, pop_{o} and pcp_{c} denote the predictions from the original branch and the feature communication branch, respectively. LuL_{u}, Lu1L_{u}^{1}, Lu2L_{u}^{2} are the corresponding unsupervised losses.

The performance of semi-supervised semantic segmentation models significantly depends on the quality of the pseudo-labels, where high-quality pseudo-labels are typically characterized by minimal uncertainty [23]. Motivated by this insight, our approach aims to enhance collaborative training by integrating high-confidence predictions from heterogeneous models to produce superior pseudo-labels. We conduct a comparative visualization to highlight differences in the segmentation maps generated by CPS, U2U^{2}PL, and our method. As illustrated in Figure 2, our model demonstrates robust performance across diverse challenges, including high-quality segmentation of complex structures such as the intricate leg regions of a horse (example in the first row), effective handling of complex backgrounds such as the decorations on either side of a computer (example in the second row), accurate detection of multiple small objects, such as people, cows, and horses (example in the third row), and superior performance in scenarios involving object overlap, such as between a person and a horse or between a person and a bottle (examples in the fourth and fifth rows). This performance gain results from a virtuous cycle between the heterogeneous branches and the dynamic voting module. At each iteration, the branches produce diverse and complementary predictions, which are fused by the voting module into more reliable pseudo-labels. These pseudo-labels provide stronger supervision, further improving the branches’ predictive capabilities. As training progresses, this process of prediction refinement, pseudo-labels enhancement, and model improvement continually elevates the quality of both pseudo-labels and segmentation outputs.

Quality analysis of segmentation maps

Figure 2. Quality analysis of segmentation maps. From left to right: (a) input images, (b) manually annotated ground truth labels, (c) segmentation maps generated by CPS, (d) segmentation maps generated by U2^{2}PL, and (e) segmentation maps produced by HDVS. All models are trained using the 1/8 classical partition protocol on the PASCAL VOC 2012 [24] train dataset.

In this work, we argue that relying solely on a single network’s predictions to generate pseudo-labels is suboptimal. Most prior methods either prioritize reliability via a single teacher–student network with confidence thresholds or pursue diversity via co-training/CPS pipeline (multi-view/augmentation/architecture) that swap per-branch argmax labels without explicit confidence fusion [13, 14, 15], thus risking confirmation bias. Instead, we advocate for an approach that harnesses collective wisdom and collaborative effort. Specifically, heterogeneous dual-branch voting supervision (HDVS) introduces a voting module after the decoders and couples it with a lightweight feature communication module (FCM): the voting module fuses the two outputs into a single pseudo-labels per pixel based on prediction confidences, whereas the FCM performs bidirectional, level-aligned feature communication to preserve global–local complementarity and maintain inter-branch diversity. Taken together, the FCM and the voting module operate within a single training loop: as feature communication proceeds between the heterogeneous branches, those same branches co-decide the labels via dynamic voting, jointly improving pseudo-labels reliability and preserving inter-branch diversity, thereby stabilizing training and yielding stronger segmentation performance.

Comprehensive experiments validate the efficacy of our HDVS, showcasing notable performance improvements of 0.7∼2.9 mean intersection over union (mIoU) points across two public benchmarks, i.e., PASCAL VOC 2012 [24] and Cityscapes [25]. These results underscore the potential of HDVS as a scalable and foundational approach in semi-supervised segmentation. Our source code and model weights will be made publicly available.

Related Work

Semi-Supervised Learning

Semi-supervised learning (SSL) aims to enhance model performance by using unlabeled data when labeled data is limited, and it has rapidly developed in recent years [23, 26, 27, 28, 29]. The central challenge in SSL involves devising effective supervisory signals for a large volume of unlabeled data [30, 31]. Current approaches in SSL primarily follow two foundational principles. The first principle is entropy minimization [23, 32, 33, 34, 35, 36], which aims to boost model confidence by promoting predictions that minimize uncertainty on unlabeled data. Specifically, entropy minimization compels the model to deliver confident and decisive outputs, thereby reducing the entropy in its probability perturbations. This principle often complements self-training techniques [32], where high-confidence outputs, termed pseudo-labels, are generated for unlabeled data. These pseudo-labels, combined with manually labeled data, are then used to enhance the training process.

Another significant branch of research in SSL employs consistency regularization [12, 16, 17, 37, 38], which compels models to produce consistent predictions across various perturbations of the identical input. CSD [37] integrates consistency constraints into object classification and localization tasks while utilizing background elimination to reduce the influence of background elements on detection performance. MixMatch [17] combines data augmentation, low-entropy label guessing for unlabeled data, and a mixing strategy to enforce prediction consistency, which helps in smoothing prediction perturbations and regularizing the model. Building on the MixMatch, ReMixMatch [16] introduces perturbation alignment to improve class balance and augmentation anchoring, ensuring consistent predictions across strongly augmented versions of the same unlabeled instance. FixMatch [12] uses pseudo-labels generated from weakly augmented data to guide the training on strongly augmented versions, only incorporating pseudo-labels that meet a confidence threshold. Expanding on FixMatch, UPS [38] enhances the training process by reducing noise interference, significantly increasing the accuracy of pseudo-labels, and addressing uncertainties related to both the model and the data.

Semi-supervised Semantic Segmentation

Integrating consistency learning with pseudo-labeling strategies has significantly advanced semi-supervised semantic segmentation [39, 40, 41]. Following the consistency regularization theorem, French et al. [42] successfully applied the Cutout [43] and CutMix [44] techniques to semi-supervised semantic segmentation, demonstrating their effectiveness. CCT [40] employs cross-consistency training, which trains the model to align predictions across different augmentations or views of an image. CPS [14] uses a strategy of which pseudo-labels generated by separate but architecturally identical networks are used for cross-training, facilitating mutual enhancement and error rectification within the networks. UDA [41] enhances the consistency training framework by replacing simple noise operations with more advanced data augmentation methods, such as RandAugment [45] and back-translation [46], to improve the utilization of unlabeled data.

Another prominent research trend in semi-supervised semantic segmentation utilizes the teacher-student architecture, where the student progressively updates the teacher via EMA and then uses its generated pseudo-labels to supervise the student’s training [12, 13, 47, 48, 49]. On the one hand, various studies aim to augment pseudo-labels’ utility by learning to assess their reliability with confidence scoring techniques. U2PL [13] designates unreliable pixels as negative samples within a contrastive learning framework. ELN [50] improves the quality of pseudo-labels through the integration of a trainable correction network. On the other hand, numerous studies emphasize the critical importance of data augmentation in this field [47, 48, 49]. UniMatch [51] explores innovative augmentation techniques for creating perturbed data views. AugSeg [49] innovatively enhances less confident unlabeled samples by leveraging confidently labeled samples through the application of the CutMix technique in a flexible manner. Contrary to common methods that rely on data augmentation or the design of additional correction networks, our approach employs a simple yet effective voting mechanism to enhance the quality of pseudo-labels. Building upon the foundational cross-supervision framework, our model innovates through the integration of a heterogeneous dual-network architecture complemented by a feature communication module and the voting mechanism.

Method

Preliminary

Originally developed for semi-supervised classification tasks [52, 53], the co-training approach has been successfully extended to semi-supervised semantic segmentation by CPS [14]. As depicted in Figure 1(b), this approach employs a dual-branch architecture, with each branch comprising an encoder and a decoder. These branches are collaboratively trained using a combination of a small set of labeled data, denoted as BlB_l, and a larger pool of unlabeled data, denoted as BuB_u. We denote the two segmentation models (branches) as M0M^0 and M1M^1. The training process is calculated as follows:

L=Ls+λLu,(1)\mathcal{L} = \mathcal{L}_s + \lambda\mathcal{L}_u, \tag*{(1)}

where Ls\mathcal{L}_s signifies the supervised loss term, and Lu\mathcal{L}_u represents the unsupervised loss term. λ\lambda serves as a weighting coefficient to balance different losses. Without loss of generality, the supervised loss is formulated as follows:

Ls=1BlHW∑i=1Bl∑j=1HW(ℓce(pij0,yij∗)+ℓce(pij1,yij∗)),(2)\mathcal{L}_s = \frac{1}{B_lHW}\sum_{i=1}^{B_l}\sum_{j=1}^{HW}\left(\ell_{\mathrm{ce}}(p^0_{ij},y^*_{ij})+\ell_{\mathrm{ce}}(p^1_{ij},y^*_{ij})\right), \tag*{(2)}

where ℓce\ell_{\mathrm{ce}} denotes the pixel-wise cross-entropy loss applied to each pixel. jj represents the jj-th pixel in the image or the corresponding segmentation mask of resolution H×WH \times W, pij0p^0_{ij} and pij1p^1_{ij} are the predictions from the two network branches for the jj-th pixel in the ii-th image in the batch, and yij∗y^*_{ij} is the ground truth label for that pixel.

The unsupervised loss employs a bidirectional cross-supervision mechanism, wherein the pixel-wise one-hot label from one network serves as the supervisory signal for the other, as delineated below:

Lu=1BuHW∑i=1Bu∑j=1HW(ℓce(pij0,y^ij1)+ℓce(pij1,y^ij0)),(3)\mathcal{L}_u = \frac{1}{B_uHW}\sum_{i=1}^{B_u}\sum_{j=1}^{HW}\left(\ell_{\mathrm{ce}}(p^0_{ij},\hat{y}^1_{ij})+\ell_{\mathrm{ce}}(p^1_{ij},\hat{y}^0_{ij})\right), \tag*{(3)}

where y^ij0\hat{y}^0_{ij} and y^ij1\hat{y}^1_{ij} denote the pixel-wise one-hot pseudo-labels generated by the first and second networks, respectively.

Heterogeneous Dual-Branch Network

To increase the diversity between models and enhance the co-supervision process, our model adopts a heterogeneous dual-branch structure. For each labeled image, the loss is directly computed by utilizing the manually labeled ground truth. Specifically, we initiate the process by subjecting the data to weak data augmentation. Subsequently, the augmented data is fed into the encoders of two distinct models, generating respective image feature maps. In the original branch, the features are directly input into the decoder within each model, yielding the prediction outcome po\mathbf{p}_o. Simultaneously, in the feature communication branch, features undergo communication through the FCM, incorporating features from the counterpart model. These communication features are fed into the decoder to generate the prediction outcome pc\mathbf{p}_c. The specific details of the FCM will be elaborated in Section 3.4. Therefore, the supervised loss for our model is defined as follows:

Ls=∑α=011BlHW∑i=1Bl∑j=1HW(ℓce(poijα,yij∗)+ℓce(pcijα,yij∗)),(4)\mathcal{L}_{s}=\sum_{\alpha=0}^{1}\frac{1}{B_lHW}\sum_{i=1}^{B_l}\sum_{j=1}^{HW}\left(\ell_{\mathrm{ce}}(p_{o_{ij}}^{\alpha},y_{ij}^{*})+\ell_{\mathrm{ce}}(p_{c_{ij}}^{\alpha},y_{ij}^{*})\right), \tag*{(4)}

where the superscript α\alpha denotes the segmentation networks, with α=0\alpha=0 corresponding to the Transformer model and α=1\alpha=1 to the convolutional neural network (CNN) model.

For processing an unlabeled image, as illustrated in Figure 3, the image first undergoes weak data augmentation before being input into two segmentation models. These models produce individual prediction results, denoted as po0p_o^0 and po1p_o^1. These results are then input into the voting module to generate pseudo-labels y^\hat{y}. The entropy of these predictions reflects the uncertainty of the model outputs, with higher entropy indicating lower confidence in the generated pseudo-labels. Following the method described in Ref. [13], we calculate the entropy based on po0p_o^0 and po1p_o^1. To enhance label reliability, we discard pseudo-labels associated with the top 20% of pixel positions exhibiting the highest entropy, resulting in refined pseudo-labels y^0\hat{y}^0 and y^1\hat{y}^1. Detailed workings of the voting module are further explored in § 3.3. The entropy H\mathcal{H} is calculated as follows:

Figure 3. Main pipeline of our HDVS method. It integrates two distinct parallel architectures, i.e., CNNs and Transformers, to capture pseudo-labels from different perspectives. To improve label accuracy, a fusion and voting strategy is applied across these architectures. The feature communication module (FCM) on each branch supports implicit feature-level integration, while a specialized voting module explicitly refines the pseudo-labels generated by each branch. The refined labels, denoted as y^\hat{y}, and their filtered versions, y^0\hat{y}^{0} and y^1\hat{y}^{1}, are used to jointly train both models, promoting synergistic learning across the architectures.

Hα(poijα)=−∑c=0C−1poijα(c)log⁡poijα(c),(5)\mathcal{H}^{\alpha}(p_{o_{ij}}^{\alpha})=-\sum_{c=0}^{C-1}p_{o_{ij}}^{\alpha}(c)\log p_{o_{ij}}^{\alpha}(c), \tag*{(5)}

where CC denotes the total number of classes. Later, the image, subjected to strong data augmentation, is fed into the two models. Each model generates a prediction result on both the feature communication and the original branches, supervised by the filtered pseudo-labels. Therefore, our unsupervised loss is computed as

Lu=∑α=011BuHW∑i=1Bu∑j=1HW(ℓce(poijα,y^ij(1−α))+ℓce(pcijα,y^ij(1−α))).(6)\mathcal{L}_{u}=\sum_{\alpha=0}^{1}\frac{1}{B_uHW}\sum_{i=1}^{B_u}\sum_{j=1}^{HW}\left(\ell_{\mathrm{ce}}(p_{o_{ij}}^{\alpha},\hat{y}_{ij}^{(1-\alpha)})+\ell_{\mathrm{ce}}(p_{c_{ij}}^{\alpha},\hat{y}_{ij}^{(1-\alpha)})\right). \tag*{(6)}

Voting-based Pseudo-label Generation

The quality of pseudo-labels is fundamental for achieving high model accuracy. Noisy pseudo-labels can lead to the accumulation of inaccurate data and increase the risk of confirmation bias, as noted by Arazo et al. [19]. Most current semi-supervised semantic segmentation methods, such as the teacher-student structure [13] and cross-supervision structure [14], rely on a single segmentation network to generate pseudo-labels. This dependence on a single perspective, which is inherently susceptible to noise, typically results in lower-quality pseudo-labels. Additionally, pseudo-labels of higher quality are generally associated with greater prediction confidence. To mitigate these challenges, we introduce a voting module that evaluates prediction confidence across dual network branches. This module enables the selective incorporation of higher-quality pseudo-labels into the training process, thereby enhancing overall model performance.

The process for processing an unlabeled image uiu_i is detailed in Figure 3. Initially, the image undergoes weak data augmentation and is then input into two separate models, which generate prediction vectors poij0,poij1∈RH×W×Cp_{o_{ij}}^0,p_{o_{ij}}^1\in\mathbb{R}^{H\times W\times C}. Each vector’s highest confidence score is identified, and the corresponding probability is obtained via a softmax operation. Following this, a random variable, rr, uniformly distributed between 0 and 1, is generated to aid the decision-making process. This variable is compared with the normalized probabilities of the most confidently predicted category from each model. The result of this comparison determines which model’s output will be used as the final pseudo-labels. By employing this method, we effectively implement a soft voting mechanism that adaptively integrates predictions from the heterogeneous models while introducing an element of uncertainty into the decision-making process. The final pseudo-labels are determined based on this comparative assessment as

prob⁡ij0,label⁡ij0=torch.max⁡(softmax⁡(poij0)),(7)\operatorname{prob}_{ij}^{0},\operatorname{label}_{ij}^{0}=\operatorname{torch.max}(\operatorname{softmax}(p_{o_{ij}}^{0})), \tag*{(7)}
prob⁡ij1,label⁡ij1=torch.max⁡(softmax⁡(poij1)),(8)\operatorname{prob}_{ij}^{1},\operatorname{label}_{ij}^{1}=\operatorname{torch.max}(\operatorname{softmax}(p_{o_{ij}}^{1})), \tag*{(8)}
y^ij={label⁡ij0,if r<prob⁡ij0prob⁡ij0+prob⁡ij1,label⁡ij1,else,(9)\hat{y}_{ij}= \begin{cases} \operatorname{label}_{ij}^{0}, & \text{if }r<\dfrac{\operatorname{prob}_{ij}^{0}}{\operatorname{prob}_{ij}^{0}+\operatorname{prob}_{ij}^{1}},\\ \operatorname{label}_{ij}^{1}, & \text{else}, \end{cases} \tag*{(9)}

where prob⁡ij0\operatorname{prob}_{ij}^{0} and prob⁡ij1∈[0,1)\operatorname{prob}_{ij}^{1}\in[0,1) denote the probabilities of the most confident classes by the two segmentation networks, determined using torch.max, which returns the maximum probability and its corresponding class index along the class dimension. These probabilities correspond to the pseudo-labels label⁡ij0\operatorname{label}_{ij}^{0} and label⁡ij1\operatorname{label}_{ij}^{1}. Additionally, the variable y^ij∈{0,1,…,C−1}\hat{y}_{ij}\in\{0,1,\ldots,C-1\} signifies the high-quality pseudo-labels generated through the voting module. The specific procedural steps of this voting mechanism are detailed in Algorithm 1.

Feature Communication Module

Perturbations at the feature level have been widely applied in semi-supervised semantic segmentation to enhance model performance, as demonstrated in previous works [48, 51]. Despite their effectiveness, the full

Algorithm 1 Voting-based pseudo-label generation pipeline

Input Sample NN, unlabeled images BuB_u

Output Voting-based pseudo-labels y^∈RN×H×W\hat{y} \in\mathbb{R}^{N \times H \times W}

for ui∈Buu_i \in B_u do

&nbsp;&nbsp;&nbsp;&nbsp;# Perform weak data augmentation AwA_w.

&nbsp;&nbsp;&nbsp;&nbsp;xi←Aw(ui)x_i \leftarrow A_w(u_i);

&nbsp;&nbsp;&nbsp;&nbsp;# Get predictions from the two models M0M^0 and M1M^1.

&nbsp;&nbsp;&nbsp;&nbsp;poi0←M0(xi), poi1←M1(xi)p^0_{o_i} \leftarrow M^0(x_i),\ p^1_{o_i} \leftarrow M^1(x_i);

&nbsp;&nbsp;&nbsp;&nbsp;# Sample a uniform random number.

&nbsp;&nbsp;&nbsp;&nbsp;r∼U(0,1)r \sim U(0,1);

&nbsp;&nbsp;&nbsp;&nbsp;for j∈H×Wj \in H \times W do

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Get confidence scores.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pred⁡ij0←softmax⁡(poij0)\operatorname{pred}^{0}_{ij} \leftarrow\operatorname{softmax}(p^{0}_{o_{ij}});

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pred⁡ij1←softmax⁡(poij1)\operatorname{pred}^{1}_{ij} \leftarrow\operatorname{softmax}(p^{1}_{o_{ij}});

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Get the top-1 category (argmax) and its corresponding probability.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;prob⁡ij0,label⁡ij0←torch.max⁡(pred⁡ij0)\operatorname{prob}^{0}_{ij}, \operatorname{label}^{0}_{ij} \leftarrow\operatorname{torch.max}(\operatorname{pred}^{0}_{ij});

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;prob⁡ij1,label⁡ij1←torch.max⁡(pred⁡ij1)\operatorname{prob}^{1}_{ij}, \operatorname{label}^{1}_{ij} \leftarrow\operatorname{torch.max}(\operatorname{pred}^{1}_{ij});

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Obtaining pseudo-labels through voting.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;y^ij={label⁡ij0,if r<prob⁡ij0prob⁡ij0+prob⁡ij1,label⁡ij1,else.\hat{y}_{ij} = \begin{cases} \operatorname{label}^{0}_{ij}, & \text{if } r < \frac{\operatorname{prob}^{0}_{ij}}{\operatorname{prob}^{0}_{ij}+\operatorname{prob}^{1}_{ij}}, \\ \operatorname{label}^{1}_{ij}, & \text{else.} \end{cases}

&nbsp;&nbsp;&nbsp;&nbsp;end for

end for

potential of the feature space remains underutilized due to the limitations of features derived from a single model encoder. To address this, we developed the feature communication module FCM for each branch of our model. This module facilitates the integration of complementary information from different model encoders and optimizes feature communication. Additionally, it introduces controlled perturbations to enrich feature interactions, thereby enhancing the robustness and accuracy of our method.

Taking the Transformer branch as an example, as depicted in Figure 4, the features f1\mathbf{f}^{1} extracted from the CNN encoder are integrated with the features f0\mathbf{f}^{0} extracted from the Transformer encoder using the FCM. The multi-level features of f0\mathbf{f}^{0} and f1\mathbf{f}^{1} are then combined through a series of operations, including 1×11 \times1 convolutions, interpolation, dropout, and scaling coefficients. These operations ensure that both feature sets align in width, height, and channel dimensions at each level. This integration process can be represented as follows:

Figure 4. Pipeline of the FCM in the Transformer branch. This module is designed to facilitate feature communication between two heterogeneous segmentation models while introducing controlled perturbations. The FCM effectively aligns and integrates features from both models to enhance segmentation performance. f0f^{0} and f1f^{1} denote features from the two encoders, cf0cf^{0} is the communicated feature in the Transformer branch, and λ0\lambda^{0} controls the strength of injected features. ⊕\oplus denotes feature convolution. ⊗\otimes denotes feature summation.

cfi0=fi0+λ0⋅Dropout⁡(P0(fi1)),(10)\mathbf{cf}^{0}_{i} = \mathbf{f}^{0}_{i} + \lambda^{0} \cdot\operatorname{Dropout}\left(\mathcal{P}^{0}\left(\mathbf{f}^{1}_{i}\right)\right), \tag*{(10)}
cfi1=fi1+λ1⋅Dropout⁡(P1(fi0)),(11)\mathbf{cf}^{1}_{i} = \mathbf{f}^{1}_{i} + \lambda^{1} \cdot\operatorname{Dropout}\left(\mathcal{P}^{1}\left(\mathbf{f}^{0}_{i}\right)\right), \tag*{(11)}

where cfi0\mathbf{cf}^{0}_{i} and cfi1\mathbf{cf}^{1}_{i} represent the features after undergoing processing by the FCM in their respective models, P0\mathcal{P}^{0} and P1\mathcal{P}^{1} represent adjustments made by the modules through interpolation and convolution operations, and λ0\lambda^{0} and λ1\lambda^{1} act as scaling coefficients for feature communication. Dropout is used for regularization by randomly dropping communicated features during training.

Overall Pipeline of HDVS

HDVS integrates the FCM and a dynamic voting module, and jointly trains two heterogeneous branches. It is trained on a small labeled set BlB_l and a large unlabeled set BuB_u. For each labeled image, the supervised loss Ls\mathcal{L}_s is computed as the pixel-wise cross-entropy between each branch’s prediction and the ground truth. For each unlabeled image, as shown in Figure 3, we first apply weak augmentation AwA_w to obtain predictions from both branches. A dynamic voting module then fuses these predictions into pseudo-labels y^\hat{y}, followed by entropy-based filtering that masks the top 20% most uncertain pixels to produce y^0\hat{y}^{0} and y^1\hat{y}^{1}. Next, we apply strong augmentation AsA_s and forward both the original branch and the feature communication branch to obtain pαp^{\alpha} and pαp^{\alpha} (α∈{0,1}\alpha\in\{0,1\}), where the FCM injects controlled, cross-branch feature communication to preserve prediction diversity while stabilizing convergence. Finally, the branches are trained under cross-supervision against the filtered pseudo-labels to form the unsupervised loss Lu\mathcal{L}_u, with the overall objective L=Ls+λLu\mathcal{L} = \mathcal{L}_s + \lambda\mathcal{L}_u.

Experiment

Experiment Setup

Datasets. The PASCAL VOC 2012 dataset [24] serves as a widely recognized benchmark for semantic segmentation, featuring 20 object classes plus a background class. It includes 1464 images in the training set and 1449 images in the validation set. In line with prior works [13, 14, 49, 54, 55], we adopt two common training protocols: (1) VOC Train, a classic partition that uses only the 1464 high-quality labeled images from the VOC 2012 train set; and (2) VOC Aug, an augmented partition that expands the training data to 10582 images by including an additional 9118 images with weaker annotations from the semantic boundaries dataset (SBD) [56].

The Cityscapes dataset [25], a widely recognized benchmark for semantic segmentation, focuses on urban scene understanding across 19 semantic classes. It includes 2975 images in the training set and 500 images in the validation set, each annotated at the pixel level to enable precise object delineation in complex city environments. Every high-resolution 1024×20481024 \times2048 image spans a diverse range of urban settings from various cities.

Evaluation. To assess segmentation performance, we utilize the mean intersection over union termed as mIoU metric. Consistent with protocols described in Refs. [13, 14], we evaluate our model on the PASCAL VOC 2012 validation set by center-cropping images to a resolution of 513×513513 \times513. For the Cityscapes dataset, we apply a sliding window technique to the validation set images, which maintain a resolution of 1024×20481024 \times2048, to thoroughly assess the model’s performance across the entire image area.

Implementation details. Our approach integrates two segmentation model architectures: the first is derived from Deeplabv3+ [57], utilizing a ResNet [58] backbone, while the second employs the SegFormer [2] model. Specifically, we conduct experiments using ResNet-50 [58] with Mit-B2 [2] or ResNet-101 [58] with Mit-B3 [2]. In our experiments, we find the CNN branch less tolerant to perturbations than the Transformer branch. Thus, we set λ0\lambda^{0} to 0.10 and λ1\lambda^{1} to 0.01 as scaling coefficients for the FCM in the Transformer and CNN models, respectively. Both FCMs utilize a dropout probability of 0.5. Following Refs. [13, 14], we incorporate weak data augmentation techniques such as flips and random cropping and apply the CutMix technique [44] for strong data augmentation. We choose stochastic gradient descent (SGD) as the optimizer, complemented by synchronized batch normalization (Sync-BN).

For the PASCAL VOC 2012 dataset, we configure the initial learning rate at 0.001, weight decay at 0.0001, crop size to 513×513513 \times513, batch size to 4, and set the training duration to 150 epochs. For the Cityscapes dataset, we utilize online hard example mining cross-entropy (OHEM) loss [59], with the initial learning rate adjusted to 0.00075, weight decay to 0.0005, batch size at 2, and extend the training to 300 epochs. Consistent with approaches described in Refs. [13, 14], we employ a poly learning rate policy where the initial learning rate is scaled by (1−itermax_iter)0.9\left(1 - \frac{\mathrm{iter}}{\mathrm{max\_iter}}\right)^{0.9}, where iter denotes the current training iteration and max_iter is the total number of training iterations.

Comparison with State-of-the-art Methods

PASCAL VOC 2012. Table 1 presents the performance of our HDVS method under both classic and blended partitioning schemes, in comparison with benchmarks established by previous state-of-the-art (SOTA) methods. The results demonstrate that HDVS, regardless of using ResNet-50 or ResNet-101 backbones, consistently delivers robust performance across different amounts of labeled data. Notably, our method surpasses the current SOTA method, diverse co-training [15], which also employs a combination of Transformer and CNN architectures. HDVS outperforms this method in four out of five categories under classic settings and in all categories under blended settings. Specifically, it achieves an average accuracy improvement of 1.5% with the ResNet-50 backbone and 1.0% with the ResNet-101 backbone.

A critical analysis of these results highlights the pivotal role of pseudo-labels quality. For instance, employing ResNet-50, the performance with 2646 coarsely annotated images lags by 1.4% compared to using 732 finely annotated images, despite the fourfold increase in training data quantity. This phenomenon emphasizes the value of high-quality annotations. Additionally, it showcases the considerable effectiveness of our voting mechanism in producing high-quality pseudo-labels, significantly enhancing the overall model’s performance.

Cityscapes. Table 2 presents the results achieved with ResNet-50 as the backbone on the Cityscapes train set, spanning 1/16, 1/8, 1/4, and 1/2 data partitions. Our HDVS achieves competitive results under different partition schemes. Additionally, it surpasses the previous SOTA methods by 0.6% and 0.4% mIoU under 1/8 and 1/2 partition protocols, respectively.

1/16 (186)1/8 (372)1/4 (744)1/2 (1488)
Supervised63.368.774.176.6
CPS [14]69.874.476.978.6
U2^{2}PL [13]69.073.076.377.2
AugSeg [49]73.776.578.879.3
Co-training [15]-76.377.1-
DAW [60]75.276.377.779.2
UniMatch [51]75.076.877.578.6
Ours73.176.977.879.7

Table 2. Comparison of mIoU (%) with SOTA methods on the Cityscapes [25] validation set with ResNet-50 backbone under various partition protocols.

A visual analysis of segmentation maps is presented in Figure 5, which compares segmentation maps generated by CPS, U2PL, and our HDVS on the Cityscapes validation set. All three models, utilizing ResNet-50 as the backbone, are trained using the Cityscapes train set under the 1/2 (1488) partition protocol. The results clearly show that the HDVS model performs exceptionally well in various challenging scenes. As illustrated in Figure 5, in the first row, HDVS demonstrates remarkable capability in precisely capturing fine scene details. In the second and third rows, when confronted with scenes where the target is surrounded by complex environments, HDVS accurately delineates both the target boundaries and the surrounding background. The fourth row highlights its superior performance in scenarios involving overlapping objects, where HDVS successfully outlines complex structures such as trucks. Finally, the fifth row showcases HDVS’s strong proficiency in handling scenes densely populated with numerous small objects, underscoring its exceptional effectiveness across diverse and complex urban environments.

Qualitative results on the Cityscapes validation set

Figure 5. Qualitative results on the Cityscapes validation [25] set. From left to right, the categories are (a) input images, (b) ground truth labels annotated manually, (c) segmentation maps created by CPS, (d) segmentation maps produced by U2^{2}PL, and (e) segmentation maps generated by our HDVS framework. All models are trained following the 1/2 (1488) partition protocol on the Cityscapes train set.

Feature Visualizations

In Figure 6, we present a tt-SNE [64] visualization analysis of features extracted by CPS and our HDVS method, both using ResNet-50 as the backbone and trained on the Cityscapes training dataset with the 1/2 (1488) partition protocol. The first row shows the visualization results for all 19 categories in the Cityscapes dataset. These visualizations indicate that the HDVS model achieves superior class separation compared to CPS. Specifically, within regions outlined by the black dashed lines, CPS features exhibit significant overlap among eight categories: terrain, sidewalk, fence, wall, building, truck, bus, and train, indicating potential feature confusion. In contrast, our HDVS effectively distinguishes these categories, demonstrating exceptional precision in feature extraction and category discrimination. The second row focuses on these eight categories, further highlighting the differences. CPS features tend to converge, causing overlapping feature centers, whereas HDVS shows well-defined and dispersed feature centers for each category. This distinct separation underscores the enhanced capability of our method in discriminating features across multiple categories, showcasing its improved performance in feature discrimination.

t-SNE visualization of extracted features by CPS and HDVS

Figure 6. t-SNE [64] visualization of extracted features by CPS and HDVS. Both models are trained on the Cityscapes train set following the 1/2 (1488) partition protocol, and the features are extracted from the Cityscapes validation set for display. The first row shows the visualization results for all categories, while the second row presents the visualization analysis for eight selected categories. The first and second columns display the visualization results for the CPS and HDVS models, respectively.

Ablation Studies

Ablative experiments of different components. In Table 3, we conduct ablations of HDVS components on PASCAL VOC 2012 under the classic 1464 partition protocol and the blended 1/8 (1323) partition protocol. Specifically, we build on the CPS* dual-branch baseline (a reimplementation of CPS [14] with aligned pseudo-labels filtering, matching hyperparameters, and the same learning-rate schedule) and instantiate: 1) homogeneous baselines Res-CPS*/Seg-CPS* (dual CNN / dual Transformer); 2) their “+FCM only” variants Res-C-CPS*/Seg-C-CPS* (communication branch enabled without voting module) and “+voting only” variants Res-V-CPS*/Seg-V-CPS*; and 3) heterogeneous H-CPS* (CNN+Transformer), H-V-CPS* (voting module with heterogeneous networks), and the full HDVS. All experiments use the ResNet-50 backbone, and Mit-B2 is paired with ResNet-50 in the heterogeneous setting. Table 3 shows that 1) on homogeneous baselines, adding FCM yields no gain and can slightly degrade performance relative to the CPS* baselines; 2) adding the voting module consistently improves over the corresponding baselines; and 3) on heterogeneous pairs, H-CPS* outperforms the homogeneous baselines; adding the voting module (H-V-CPS*) yields further gains; and the full HDVS (heterogeneous networks + voting + FCM) achieves the best performance. These results show that the voting module and FCM effectively aggregate complementary cues from heterogeneous branches. FCM is effective in our heterogeneous model by transferring complementary global–local cues. However, in common homogeneous dual-branch settings it tends to over-align the branches and reduce diversity.

MethodVOC TrainVOC Aug
1/16 (92)1/8 (183)1/4 (366)1/2 (732)Full (1464)1/16 (662)1/8 (1323)1/4 (2646)
Results of using ResNet-50 [58] as backbone
Supervised44.052.361.765.972.960.767.771.9
U2^{2}PL† [13]-----72.075.176.2
UniMatch [51]71.972.576.077.478.775.876.976.8
AugSeg [49]64.272.276.277.478.874.776.077.1
DAW [60]68.573.176.378.679.776.277.677.4
Switch [61]70.774.576.477.678.1---
Co-training [15]73.174.777.178.880.276.277.077.5
Ours69.975.679.681.383.178.279.379.9
Results of using ResNet-101 [58] as backbone
Supervised43.959.165.970.975.066.670.574.5
CPS [14]64.167.471.775.9-72.275.877.6
U2^{2}PL† [13]68.069.273.776.279.574.477.678.7
UniMatch [51]75.277.278.879.981.278.178.479.2
AugSeg [49]71.175.578.880.381.477.077.378.8
DAW [60]75.577.679.880.782.278.979.280.0
Co-training [15]75.777.780.180.982.077.679.080.0
AllSpark [62]76.178.479.880.882.178.380.080.4
CorrMatch [63]76.478.579.480.681.878.479.379.6
Ours72.979.481.282.983.879.180.781.0

Table 1. Comparison of mean intersection over union (mIoU) (%) with state-of-the-art (SOTA) methods on the PASCAL VOC 2012 [24] validation set with different backbones under various proportion protocols. “Supervised” denotes the supervised baseline without using unlabeled training data. The results of U2^{2}PL [13] (marked with †) are obtained from Refs. [51], and it follows the same partition protocol as ours. “VOC Train” refers to using only the 1464 finely annotated images from the train set, “VOC Aug” includes an additional 9118 weakly labeled images from the semantic boundaries dataset [56]. Bold numbers indicate the best performance under each setting.

Figure 7 visualizes segmentation maps from the representative models described above, trained under the classic 1464 partition protocol, with HDVS showcasing increased robustness in handling complex scenes through diverse components. This is particularly evident

Visualizations of segmentation maps generated by various models

Figure 7. Visualizations of segmentation maps generated by various models, (a) input images to (g) our HDVS model, showcasing progressive enhancements at each step. All models, including the homogeneous baselines Res-CPS* / Seg-CPS* (dual convolutional neural network / dual Transformer), utilizing ResNet-50 [58] as the backbone and employing a combined approach of ResNet-50 and Mit-B2 [2] for heterogeneous structures, are trained using the classic 1464 partition protocol on the PASCAL VOC 2012 train set. H-CPS* adopts a heterogeneous CNN+Transformer pair, and H-V-CPS* further introduces a voting module on top of H-CPS*.

when dealing with multiple complex targets, such as the multiple dogs in rows 1 and 2, and in handling areas where objects overlap, such as the legs of characters in rows 3 and 4. The superior performance of H-CPS* over both Res-CPS* and Seg-CPS* demonstrates the synergy between CNNs and Transformers: CNNs capture fine-grained local structures, whereas Transformers model long-range, global semantics, yielding complementary representations. In contrast, as training progresses, homogeneous dual-branch models tend to converge toward similar representations, thereby diminishing the representational diversity critical for effective co-supervision. Moreover, the additional gains of H-V-CPS* over H-CPS* indicate that architectural heterogeneity alone is insufficient and pseudo-labels quality remains a limiting factor. By adaptively fusing branch predictions according to confidence, the voting mechanism in H-V-CPS* suppresses noisy pseudo-labels and improves supervisory consistency, enhancing co-training effectiveness. Finally, HDVS further extends H-V-CPS* by incorporating a FCM and a dedicated communication branch, enabling implicit feature-level interactions across architectures to transfer complementary knowledge and strengthen collaborative learning.

Impact of hyper-parameters. To evaluate the influence of feature communication intensity on HDVS performance, we conduct comprehensive ablation experiments on the scaling coefficients λ0\lambda^{0} and λ1\lambda^{1} in FCM for both the Transformer and CNN branches. We employ ResNet-50 and Mit-B2 as our backbone and train on the classical partition of 1464 images from the PASCAL VOC 2012 train set. As shown in Table 4, by varying one coefficient while keeping the other fixed, we observe that the best performance is achieved with λ0=0.10\lambda^{0} = 0.10 and λ1=0.01\lambda^{1} = 0.01. CNNs and Transformers naturally complement each other: CNNs excel at capturing local details, while Transformers model global dependencies. However, excessive Transformer features disrupt the CNN’s representations. Reducing λ1\lambda^{1} allows the Transformer’s global information to integrate gradually, providing guidance without destabilizing the CNN. Consequently, the optimal ratio from Transformer to CNN is set lower than the other.

Table 4 Ablation study on scaling coefficients λ0\lambda^{0} and λ1\lambda^{1} using a combination of ResNet-50 and Mit-B2 as the backbone within the classical partition of 1464 training images from the PASCAL VOC 2012 train dataset.

λ0\lambda^{0}λ1\lambda^{1}Performance (%)
1.0000.01082.4
0.1000.01083.1
0.0100.01082.5
0.0010.01082.0
0.1001.00081.9
0.1000.10082.3
0.1000.01083.1
0.1000.00181.7

Table 4.

Impact of the voting mechanism. To comprehensively evaluate the efficacy of the voting mechanism, we conduct experiments on H-CPS* and H-V-CPS* using four different quantities of labeled data under the classical setting: 183, 366, 732, and 1464. Both models use a combination of ResNet-50 and Mit-B2 as their backbone.

As illustrated in Figure 8(a), we observe a more pronounced impact of the voting module on the model as the labeled data decreases. Notably, there is a 9.4% increase in accuracy for H-V-CPS* compared to H-CPS* in the 183 dataset partition. Figure 8(b) illustrates instances where H-CPS* encounters difficulties in precisely identifying specific individual classes, as evidenced by the near-zero mIoU performance in those categories. This discrepancy stems from the single-model pseudo-label generation’s flaws, which may amplify confirmation bias and degrade performance. HDVS uses a voting mechanism for collaborative learning, merging both models’ predictions to produce superior pseudo-labels at uncertain pixels, thus effectively solving this problem.

Effectiveness of the voting module under heterogeneous networks

Figure 8. Effectiveness of the voting module under heterogeneous networks. (a) The performance of H-CPS* and H-V-CPS*, using a combination of ResNet-50 and Mit-B2 as the backbone, under different numbers of labeled images on the PASCAL VOC 2012 dataset. (b) Under the classical partition protocol with 183 training samples on the PASCAL VOC train dataset, the performance of H-CPS* and H-V-CPS* across different categories. mIoU denotes mean intersection over union.

Ablations on feature communication mechanisms. In Table 5, we compare three communication schemes under identical backbones and training using the classic 1464 and the blended 1/8 (1323) partition protocol on the VOC12 train dataset: (1) Concat+MLP, which concatenates peer features and applies a per-pixel multi-layer perceptron (MLP); (2) Cross-attention, which fuses branches via attention; and (3) FCM, as summarized, it offers the best accuracy–efficiency trade-off: it improves upon the H-CPS* baseline on both partition protocol and surpasses the two heavier alternatives while using the fewest parameters. We attribute the gains to

14641323Params (M)
H-CPS*81.478.1-
Cross-attention80.576.229.3
Concat + MLP82.178.37.0
Ours (FCM)83.179.32.9

Table 5. Cross-branch communication variants. We report mIoU on the PASCAL VOC 2012 train dataset under the classic 1464 and the blended 1/8 (1323) partitioning protocols, along with the parameter count of the feature communication components (M). H-CPS* serves as the baseline without feature communication. Concat + multi-layer perceptron (MLP) concatenates peer features and projects them with an MLP. Cross-attention fuses features via cross-attention.

Table 3 Ablation study on the effectiveness of heterogeneous dual-branch voting supervision components. CPS* replicates CPS [14] with aligned pseudo-label filtering and hyperparameters. Res-/Seg-CPS* denote dual convolutional neural network/ dual Transformer baselines; H-CPS* uses a heterogeneous CNN+Transformer pair. C-CPS* augments the model with the FCM and its communication branch, while V-CPS* adds the voting module for pseudo-label generation. HDVS combines both the feature communication and voting module. All results use a ResNet-50 backbone, and Mit-B2 [2] is paired with ResNet-50 in the heterogeneous setting. ✓ and “-” indicate that the corresponding component is enabled and disabled, respectively. FCM denotes the feature communication module.

ConfigurationsmIoU
ModelHeterogeneous networksVoting moduleFCMCommunication branch14641323
Res-CPS*----78.876.9
Seg-CPS*----80.977.8
Res-C-CPS*--✓✓78.976.6
Seg-C-CPS*--✓✓80.777.4
Res-V-CPS*-✓--81.278.3
Seg-V-CPS*-✓--81.878.4
H-CPS*✓---81.478.1
H-V-CPS*✓✓--82.278.9
HDVS✓✓✓✓83.179.3

Table 3.

FCM’s design, which confines learnable computation to a single projection with parameter-free alignment, thereby avoiding channel inflation and extra decoding. In contrast, concatenation/MLP and cross-attention add heavy projections and multi-layer mappings, increasing parameter counts and overfitting risk.

Entropy drop sensitivity. We assess the effect of discarding high-entropy pixels by sweeping drop ratios of 0%, 10%, 20%, 30%, and 40% on PASCAL VOC 2012 using the classical 1464 partition protocol. As shown in Table 6, removing a moderate fraction is beneficial: mIoU rises from 78.3% at 0% to a peak of 83.1% at 20%, and remains high at 82.2% at 30%. In contrast, both a smaller ratio (10%, 81.8%) and a larger ratio (40%, 79.4%) reduce HDVS’s performance. Overall, the curve saturates near 20% and is flat over 10~30%, indicating robustness and motivating 20% as a principled default rather than a hand-tuned choice.

Drop ratio (%)010203040
mIoU78.381.883.182.279.4
Δ vs. 20%−4.8−1.3-−0.9−3.7

Table 6. HDVS performance under different entropy-drop ratios. All experiments use a combination of ResNet-50 and MiT-B2 as the backbone, within the classical partition of 1464 training images from the PASCAL VOC 2012 train dataset. Δ is relative to 20%.

Parameters and computational overhead. Table 7 summarizes the trainable parameters. Overall, HDVS totals 71.0 M parameters (ResNet-50 [58]: 43.6

M; Mit-B2 [2]: 27.4 M). FCM contributes 2.9 M (already included), while the voting module has no learnable parameters. Under PASCAL VOC 2012 with 513 × 513 crops, FCM+voting adds 36.9 GFLOPs per image; voting itself is only approximately 1.6 × 10^{-3} GFLOPs, which is negligible relative to the backbone and decoder. The overhead is dominated by the 1 × 1 channel projections used for feature alignment.

Failure Case Analysis

Although HDVS achieves competitive performance on various benchmarks, we observe certain limitations under extreme conditions. These issues are mainly observed

when the model is trained with the 1/16 classical partition protocol on the PASCAL VOC 2012 dataset, where the heterogeneous branches may occasionally produce divergent predictions in challenging regions, leading to less satisfactory segmentation. As illustrated in Figure 9, examples include the front of a car with drastic color variations (the first row), the subtle boundary between a bird and its background (the second row), and a flowerpot region affected by strong illumination changes (the third row). This behavior stems from noisy pseudo-labels early in training. While the FCM’s cross-branch perturbations can be beneficial, under severely limited supervision they may propagate noise across branches and transiently destabilize co-training. Addressing these limitations will be an important direction of our future work.

Failure cases visualizations

Figure 9. Failure cases visualizations. From left to right, the categories are (a) input images, (b) ground truth annotations, and (c) segmentation maps generated by our HDVS framework. All models are trained using the 1/16 classical partition protocol on the PASCAL VOC 2012 train dataset.

ComponentParams (M)
Student branch (ResNet-50)43.6
Teacher branch (Mit-B2)27.4
FCM2.9
Voting module0.0
HDVS71.0

Table 7. Trainable parameter counts (M) of HDVS. The parameter count of the FCM is already included in the student/teacher branch totals and is reported separately for reference only.

Conclusion

In this paper, we present a novel approach called heterogeneous dual-branch voting supervision (HDVS). HDVS builds upon the cross-supervision architecture, by integrating a heterogeneous dual network structure to enhance the diversity of pseudo-labels, both implicitly and explicitly. Each branch incorporates a feature communication module that enables more effective label extraction at the feature level. Furthermore, a straightforward yet efficient voting mechanism is proposed to refine labels collaboratively generated by the dual branches. This mechanism enables the strengthening of co-confident labels and the weakening of noisy labels. With the collaboration of the two proposed modules, our HDVS achieves significant performance improvement against the state-of-the-art approaches on both the PASCAL VOC 2012 and Cityscapes benchmarks.

Abbreviations: CPS, cross pseudo supervision; EMA, exponential moving average; FCM, feature communication module; H-CPS, heterogeneous cross pseudo supervision; HDVS, heterogeneous dual-branch voting supervision; mIoU, mean intersection over union; Res-CPS, cross pseudo supervision with dual ResNet backbones; Seg-CPS, cross pseudo supervision with dual SegFormer backbones; SOTA, state of the art; SSL, semi-supervised learning; V-CPS, heterogeneous cross pseudo supervision with a voting module.

Data Availability: The datasets used in this study are publicly available: the Cityscapes dataset can be downloaded from the official website (https://www.cityscapes-dataset.com/) and the PASCAL VOC 2012 dataset from the PASCAL VOC project page (http://host.robots.ox.ac.uk/pascal/VOC/voc2012/); see also the corresponding citations in the paper. Our source code and trained model weights will be made publicly available upon acceptance.

Competing Interests: The authors declare that they have no competing interests.

Funding: This work was supported by the Fundamental Research Funds for the Central Universities (No. DUT24RC(3)025), and 2024-0011 (No. ZX20240867).

Author contributions: Yongqi Shan conceived the method, implemented the framework, and led the experiments and manuscript drafting. Yunzhi Zhuge contributed to experimental design, analysis, and manuscript revision. Huchuan Lu supervised the project, advised on methodology, and revised the manuscript. All authors read and approved the final manuscript.

Acknowledgements We would like to express our sincere gratitude to Drs. Pingping Zhang and Lijun Wang for their valuable discussions.

References

  1. [1]Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the 9th International Conference on Learning Representations (pp. 1–21). Retrieved December 27, 2025, from https://openreview.net/forum?id=YicbFdNTTy.
  2. [2]Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021). SegFormer: Simple and efficient design for semantic segmentation with transformers. In Proceedings of the 35th International Conference on Neural Information Processing Systems (pp. 12077–12090). Red Hook: Curran Associates.arxiv.org/abs/2105.15203
  3. [3]Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2022). PVT v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8, 415–424.DOI
  4. [4]Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023). Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4015–4026). Piscataway: IEEE.arxiv.org/abs/2304.02643
  5. [5]Xu, J., Xiong, Z., & Bhattacharyya, S. P. (2023). PIDNet: A real-time semantic segmentation network inspired by PID controllers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19529–19539). Piscataway: IEEE.arxiv.org/abs/2206.02066
  6. [6]Li, F., Zhang, H., Xu, H., Liu, S., Zhang, L., Ni, L. M., & Shum, H.-Y. (2023). Mask DINO: Towards a unified transformer-based framework for object detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3041–3050). IEEE, Piscataway.arxiv.org/abs/2206.02777
  7. [7]Ding, H., Liu, C., He, S., Ying, K., Jiang, X., Loy, C. C., & Jiang, Y.-G. (2025). MeViS: A multi-modal dataset for referring motion expression video segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47, 11400–11416.DOI
  8. [8]Ding, H., Liu, C., He, S., Jiang, X., Jiang, Y.-G., Torr, P. H., & Bai, S. (2025). MOSEv2: A more challenging dataset for video object segmentation in complex scenes. arXiv preprint. arXiv:2508.05630.arxiv.org/abs/2508.05630
  9. [9]Ding, H., Liu, C., He, S., Jiang, X., Torr, P. H., & Bai, S. (2023). MOSE: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 20224–20234). Piscataway: IEEE.arxiv.org/abs/2302.01872
  10. [10]Ding, H., Liu, C., He, S., Jiang, X., & Loy, C. C. (2023). MeViS: A large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2694–2703). Piscataway: IEEE.arxiv.org/abs/2308.08544
  11. [11]Souly, N., Spampinato, C., & Shah, M. (2017). Semi-supervised semantic segmentation using generative adversarial network. In Proceedings of the IEEE International Conference on Computer Vision (pp. 5688–5696). Piscataway: IEEE.DOI
  12. [12]Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C. A., Cubuk, E. D., Kurakin, A., & Li, C.-L. (2020). FixMatch: Simplifying semi-supervised learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsel, M.-F. Balcan, & H.-T. Lin (Eds.), Proceedings of the 34th International Conference on Neural Information Processing Systems (pp. 596–608). Red Hook: Curran Associates.
  13. [13]Wang, Y., Wang, H., Shen, Y., Fei, J., Li, W., Jin, G., Wu, L., Zhao, R., & Le, X. (2022). Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4248–4257). Piscataway: IEEE.arxiv.org/abs/2203.03884
  14. [14]Chen, X., Yuan, Y., Zeng, G., & Wang, J. (2021). Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2613–2622). Piscataway: IEEE.arxiv.org/abs/2106.01226
  15. [15]Li, Y., Wang, X., Yang, L., Feng, L., Zhang, W., & Gao, Y. (2023). Diverse cotraining makes strong semi-supervised segmentor. In Proceedings of the IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE.arxiv.org/abs/2308.09281
  16. [16]Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K., Zhang, H., & Raffel, C. (2020). ReMixMatch: Semi-supervised learning with distribution matching and augmentation anchoring. In Proceedings of the 8th International Conference on Learning Representations (pp. 1–13). Retrieved December 27, 2025, from https://openreview.net/forum?id=HklkeR4KPB.
  17. [17]Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., & Raffel, C. A. (2019). MixMatch: A holistic approach to semi-supervised learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, & R. Garnett (Eds.), Proceedings of the 33rd International Conference on Neural Information Processing Systems (pp. 1). Red Hook: Curran Associates.arxiv.org/abs/1905.02249
  18. [18]Tarvainen, A., & Valpola, H. (2017). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. V. N. Vishwanathan, & R. Garnett (Eds.), Proceedings of the 31st International Conference on Neural Information Processing Systems (pp. 1195–1204). Red Hook: Curran Associates.arxiv.org/abs/1703.01780
  19. [19]Arazo, E., Ortego, D., Albert, P., O’Connor, N. E., & McGuinness, K. (2020). Pseudo-labeling and confirmation bias in deep semi-supervised learning. In Proceedings of the International Joint Conference on Neural Networks (pp. 1–8). Piscataway: IEEE.arxiv.org/abs/1908.02983
  20. [20]Ouali, Y., Hudelot, C., & Tami, M. (2020). An overview of deep semi-supervised learning. arXiv preprint. arXiv:2006.05278.arxiv.org/abs/2006.05278
  21. [21]Fan, J., Gao, B., Jin, H., & Jiang, L. (2022). UCC: Uncertainty guided cross-head co-training for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9947–9956). Piscataway: IEEE.arxiv.org/abs/2205.10334
  22. [22]Li, S., He, Y., Zhang, W., Zhang, W., Tan, X., Han, J., Ding, E., & Wang, J. (2023). CFCG: Semi-supervised semantic segmentation via cross-fusion and contour guidance supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 16348–16358). Piscataway: IEEE.
  23. [23]Grandvalet, Y., & Bengio, Y. (2004). Semi-supervised learning by entropy minimization. In Proceedings of the 18th International Conference on Neural Information Processing Systems (pp. 529–536). Red Hook: Curran Associates.
  24. [24]Everingham, M., Van Gool, L., Williams, C. K., Winn, J., & Zisserman, A. (2010). The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88, 303–338.DOI
  25. [25]Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016). The Cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3213–3223). Piscataway: IEEE.arxiv.org/abs/1604.01685
  26. [26]Chen, L., & Yang, M. (2017). Semi-supervised dictionary learning with label propagation for image classification. Computational Visual Media, 3(1), 83–94.DOI
  27. [27]Zhang, B., Wang, Y., Hou, W., Wu, H., Wang, J., Okumura, M., & Shinozaki, T. (2021). FlexMatch: Boosting semi-supervised learning with curriculum pseudo labeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. W. Vaughan (Eds.), Proceedings of the 35th International Conference on Neural Information Processing Systems (pp. 18408–18419). Red Hook: Curran Associates.
  28. [28]Wang, Y., Chen, H., Heng, Q., Hou, W., Fan, Y., Wu, Z., Wang, J., Savvides, M., Shinozaki, T., Raj, B., et al. (2023). Freematch: Self-adaptive thresholding for semi-supervised learning. In Proceedings of the 11th International Conference on Learning Representations (pp. 1–20). Retrieved December 27, 2025, from https://openreview.net/forum?id=PDrUPTXJI_A.arxiv.org/abs/2205.07246
  29. [29]Sun, C.-Y., Yang, Y.-Q., Guo, H.-X., Wang, P.-S., Tong, X., Liu, Y., & Shum, H.-Y. (2023). Semi-supervised 3D shape segmentation with multilevel consistency and part substitution. Computational Visual Media, 9(2), 229–247.arxiv.org/abs/2204.08824
  30. [30]Oliver, A., Odena, A., Raffel, C. A., Cubuk, E. D., & Goodfellow, I. (2018). Realistic evaluation of deep semi-supervised learning algorithms. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Proceedings of the 32nd International Conference on Neural Information Processing Systems (pp. 3239–3250). Red Hook: Curran Associates.arxiv.org/abs/1804.09170
  31. [31]Ding, H., Tang, S., He, S., Liu, C., Wu, Z., & Jiang, Y.-G. (2025). Multimodal referring segmentation: A survey. arXiv preprint. arXiv:2508.00265.arxiv.org/abs/2508.00265
  32. [32]Lee, D.-H. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, International Conference on Machine Learning (pp. 896). Retrieved December 27, 2025, from https://papers.nips.cc/paper_files/paper/2013/file/1a2e19b1d7b4eae60a99b3b4e78b82-Paper.pdf.
  33. [33]Xie, Q., Luong, M.-T., Hovy, E., & Le, Q. V. (2020). Self-training with noisy student improves ImageNet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10687–10698). Piscataway: IEEE.arxiv.org/abs/1911.04252
  34. [34]Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., & Le, Q. V. (2020). Rethinking pre-training and self-training. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, & H.-T. Lin (Eds.), Proceedings of the 34th International Conference on Neural Information Processing Systems (pp. 3833–3845). Red Hook: Curran Associates.arxiv.org/abs/2006.06882
  35. [35]Pham, H., Dai, Z., Xie, Q., & Le, Q. V. (2021). Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11557–11568). Piscataway: IEEE.
  36. [36]Ding, C., Zhang, J., Ding, H., Zhao, H., Wang, Z., Xing, T., & Hu, R. (2023). Decoupling with entropy-based equalization for semi-supervised semantic segmentation. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence (pp. 663–671). Cham: Springer.DOI
  37. [37]Jeong, J., Lee, S., Kim, J., & Kwak, N. (2019). Consistency-based semi-supervised learning for object detection. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, & R. Garnett (Eds.), Proceedings of the 33rd International Conference on Neural Information Processing Systems (pp. 10759–10768). Red Hook: Curran Associates.
  38. [38]Rizve, M. N., Duarte, K., Rawat, Y. S., & Shah, M. (2021). In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In Proceedings of the 9th International Conference on Learning Representations (pp. 1–20). Retrieved December 27, 2025, from https://openreview.net/forum?id=-ODN6SbiUW.arxiv.org/abs/2101.06329
  39. [39]Miyato, T., Maeda, S.-i., Koyama, M., & Ishii, S. (2019). Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8), 1979–1993.DOI
  40. [40]Ouali, Y., Hudelot, C., & Tami, M. (2020). Semi-supervised semantic segmentation with cross-consistency training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12674–12684). Piscataway: IEEE.arxiv.org/abs/2003.09005
  41. [41]Xie, Q., Dai, Z., Hovy, E., Luong, T., & Le, Q. (2020). Unsupervised data augmentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, & H.-T. Lin (Eds.), Proceedings of the 34th International Conference on Neural Information Processing Systems (pp. 6256–6268). Red Hook: Curran Associates.
  42. [42]French, G., Aila, T., Laine, S., Mackiewicz, M., & Finlayson, G. (2020). Semi-supervised semantic segmentation needs strong, high-dimensional perturbations. In Proceedings of the British Machine Vision Conference (pp. 1–14). Swansea: BMVA Press.
  43. [43]DeVries, T., & Taylor, G. W. (2017). Improved regularization of convolutional neural networks with cutout. arXiv preprint. arXiv:1708.04552.arxiv.org/abs/1708.04552
  44. [44]Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., & Yoo, Y. (2019). CutMix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6023–6032). Piscataway: IEEE.arxiv.org/abs/1905.04899
  45. [45]Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020). RandAugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 702–703). Piscataway: IEEE.arxiv.org/abs/1909.13719
  46. [46]Sennrich, R., Haddow, B., & Birch, A. (2016). Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 86–96). Stroudsburg: ACL.arxiv.org/abs/1511.06709
  47. [47]Yuan, J., Liu, Y., Shen, C., Wang, Z., & Li, H. (2021). A simple baseline for semi-supervised semantic segmentation with strong data augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8229–8238). Piscataway: IEEE.arxiv.org/abs/2104.07256
  48. [48]Liu, Y., Tian, Y., Chen, Y., Liu, F., Belagiannis, V., & Carneiro, G. (2022). Perturbed and strict mean teachers for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4258–4267). Piscataway: IEEE.arxiv.org/abs/2111.12903
  49. [49]Zhao, Z., Yang, L., Long, S., Pi, J., Zhou, L., & Wang, J. (2023). Augmentation matters: A simple-yet-effective approach to semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11350–11359). Piscataway: IEEE.
  50. [50]Kwon, D., & Kwak, S. (2022). Semi-supervised semantic segmentation with error localization network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9957–9967). Piscataway: IEEE.arxiv.org/abs/2204.02078
  51. [51]Yang, L., Qi, L., Feng, L., Zhang, W., & Shi, Y. (2023). Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7236–7246). Piscataway: IEEE.arxiv.org/abs/2208.09910
  52. [52]Qiao, S., Shen, W., Zhang, Z., Wang, B., & Yuille, A. (2018). Deep co-training for semi-supervised image recognition. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), Proceedings of the 15th European Conference on Computer Vision (pp. 135–152). Cham: Springer.arxiv.org/abs/1803.05984
  53. [53]Ke, Z., Wang, D., Yan, Q., Ren, J., & Lau, R. W. (2019). Dual student: Breaking the limits of the teacher in semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6728–6736). Piscataway: IEEE.arxiv.org/abs/1909.01804
  54. [54]Hu, H., Wei, F., Hu, H., Ye, Q., Cui, J., & Wang, L. (2021). Semi-supervised semantic segmentation via adaptive equalization learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. W. Vaughan (Eds.), Proceedings of the 35th International Conference on Neural Information Processing Systems (pp. 22106–22118). Red Hook: Curran Associates.arxiv.org/abs/2110.05474
  55. [55]Yang, L., Zhuo, W., Qi, L., Shi, Y., & Gao, Y. (2022). ST++: Make self-training work better for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4268–4277). Piscataway: IEEE.arxiv.org/abs/2106.05095
  56. [56]Hariharan, B., Arbeláez, P., Bourdev, L., Maji, S., & Malik, J. (2011). Semantic contours from inverse detectors. In Proceedings of the IEEE International Conference on Computer Vision (pp. 991–998). Piscataway: IEEE.DOI
  57. [57]Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018). Encoder-decoder with atrous separable convolution for semantic image segmentation. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), Proceedings of the 15th European Conference on Computer Vision (pp. 801–818). Cham: Springer.arxiv.org/abs/1802.02611
  58. [58]He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770–778). Piscataway: IEEE.arxiv.org/abs/1512.03385
  59. [59]Shrivastava, A., Gupta, A., & Girshick, R. (2016). Training region-based object detectors with online hard example mining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 761–769). Piscataway: IEEE.arxiv.org/abs/1604.03540
  60. [60]Sun, R., Mai, H., Zhang, T., & Wu, F. (2023). DAW: exploring the better weighting function for semi-supervised semantic segmentation. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Proceedings of the 37th International Conference on Neural Information Processing Systems (pp. 61792–61805). Red Hook: Curran Associates.DOI
  61. [61]Na, J., Ha, J.-W., Chang, H. J., Han, D., & Hwang, W. (2023). Switching temporary teachers for semi-supervised semantic segmentation. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Proceedings of the 37th International Conference on Neural Information Processing Systems (pp. 40367–40380). Red Hook: Curran Associates.arxiv.org/abs/2310.18640
  62. [62]Wang, H., Zhang, Q., Li, Y., & Li, X. (2024). AllSpark: Reborn labeled features from unlabeled in transformer for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3627–3636). Piscataway: IEEE.arxiv.org/abs/2403.01818
  63. [63]Sun, B., Yang, Y., Zhang, L., Cheng, M.-M., & Hou, Q. (2024). CorrMatch: Label propagation via correlation matching for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3097–3107). Piscataway: IEEE.arxiv.org/abs/2306.04300
  64. [64]van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9, 2579–2605.

Paper details

vlnc:W_mfjdybhh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents