betaFeedback

WorksEngineering

Shared-Pattern Collaborative Network for Multi-Source Remote Sensing Data Classification

Abstract

Multi-source remote sensing data classification has received growing attention recently. However, existing methods still struggle to jointly model long-range contextual dependencies and shared spatial patterns across heterogeneous modalities. To address this issue, we propose a Shared-Pattern Collaborative Network (SPCNet) for HSI and SAR/LiDAR joint classification. To enhance cross-modal collaborative learning, Shared-Pattern Collaborative Block (SPCB) is designed by introducing a shared large-kernel operator and a dynamic convolution branch, enabling the network to capture common spatial structures and content-adaptive local features. Furthermore, Region-Adaptive Modality Allocator (RAMA) is developed to assign spatially varying modality weights for reliable feature fusion. Experiments on Houston2013, Augsburg, and Houston2018 datasets demonstrate that SPCNet achieves superior classification performance over several state-of-the-art methods. The code will be made publicly available at https://github.com/oucailab/SPCNet

Introduction

Remote sensing image classification has become a fundamental task in Earth observation [1]. It has attracted considerable attention in recent years owing to its wide-ranging applications in environmental monitoring [2], urban planning [3], and disaster management [4]. With the increasing availability of heterogeneous Earth observation data, the task has expanded from single-source pixel recognition toward multimodal scene understanding. However, different sensors measure distinct physical properties and exhibit different spatial, spectral, and noise characteristics, making reliable joint representation learning a persistent challenge.

Land-cover classification is not merely the recognition of isolated pixels. Natural and urban scenes contain objects at markedly different spatial scales, including compact buildings, elongated roads, fragmented vegetation, and mixed transition regions. Their spectra may exhibit high intra-class variability and strong inter-class similarity because of illumination changes, material mixtures, seasonal conditions, and sensor noise [5]. An effective classifier must therefore jointly exploit diagnostic spectral signatures, local structures, and broader spatial context [6, 7].

Hyperspectral images (HSIs) acquire hundreds of contiguous spectral bands across the electromagnetic spectrum. Such rich spectral information provides strong capability for identifying complex objects [8, 9, 10, 11], but neighboring bands are highly correlated and limited spatial resolution may blur object boundaries. The classification performance of HSI is therefore often constrained by spectral similarity among different land-cover classes, noise contamination, and spectral redundancy. Integrating HSI with other modalities has consequently become an effective strategy for improving classification performance [12]. Synthetic aperture radar (SAR) and Light Detection and Ranging (LiDAR) data are particularly attractive because of their distinct imaging mechanisms and rich structural information [13, 14, 15]. SAR captures geometric and textural characteristics under all-weather and all-day conditions, while LiDAR provides precise elevation and three-dimensional structural information. Their combination with HSI enables complementary spectral, scattering, and

structural cues to be exploited jointly, as demonstrated by large multimodal benchmark studies [16]. Motivated by these advantages, this paper investigates the joint classification of HSI and SAR/LiDAR data.

The benefit of multi-source observation does not simply arise from stacking additional channels. HSI records reflectance, SAR characterizes backscattering related to surface roughness and imaging geometry, whereas LiDAR directly measures elevation and three-dimensional structure. Their relationship is therefore established through land-cover semantics rather than homogeneous signal statistics. Moreover, source reliability is spatially nonuniform: HSI is affected by shadows and spectral mixing, SAR by speckle and layover, and LiDAR may be less informative over locally flat surfaces. Differences in spatial resolution, noise distribution, and physical response consequently create a substantial cross-modal gap [17, 18, 19, 20]. A successful fusion model should preserve modality-specific evidence while discovering structural regularities shared across sensors, rather than propagating redundant or unreliable responses through indiscriminate aggregation.

Multi-source classification methods have consequently evolved from decision-level fusion toward feature-level collaborative learning. Decision-level approaches combine predictions produced by independently trained classifiers and are relatively robust to heterogeneous feature dimensions, but their late interaction cannot fully exploit intermediate correlations between spectral, scattering, and elevation cues. Feature-level methods use parallel encoders and exchange information during representation learning, allowing complementary cues to participate directly in the construction of class-discriminative features [21], [22]. CNN-based architectures provide strong local inductive biases, while attention mechanisms and Transformer-based models improve global interaction and long-range dependency modeling [23, 24, 25]. Shared-and-specific representation learning and masked modeling further demonstrate that cross-source consistency and modality individuality should be considered jointly [15], [18]. Nevertheless, increasing network depth or introducing global attention does not automatically guarantee that heterogeneous branches learn compatible spatial structures, nor does it determine which source should dominate at every location.

As illustrated in Figure 1, existing multi-source models typically extract HSI and SAR/LiDAR features using independent operators before combining the resulting representations. Such separate processing can overlook the common spatial structures shared by co-registered observations. SPCB selectively shares a patch-spanning spatial operator across co-registered modalities and block depths, while modality-specific stems, identity channels, and adaptive local filtering retain source-dependent features. This design complements shared-specific and cross-modal learning [18].

Figure 1. Comparison between existing multi-source data classification models and the proposed SPCNet. SPCNet shares a large-kernel operator across the two modality branches to collaboratively learn common spatial patterns while retaining modality-specific features for adaptive fusion.

Deep learning-based methods have become the mainstream solution for multi-source remote sensing classification. Existing approaches are primarily based on convolutional neural networks (CNNs) [21], [26], Transformers [27], [28], or their hybrid architectures to exploit complementary information. Despite their promising performance, two challenges remain insufficiently addressed: 1) Difficulty in simultaneously capturing long-range dependencies and shared spatial patterns between heterogeneous modalities. Many existing methods process different modalities independently or simply fuse them at a late stage, which limits their ability to discover common spatial structures across modalities in feature learning. Consequently, the shared contextual patterns and complementary structural cues across heterogeneous modalities remain insufficiently explored. 2) Limited region-adaptive feature fusion. Most methods adopt fixed or globally learned fusion schemes, which fail to dynamically adjust the contribution of each modality across different regions. Consequently, region-specific complementary information may be insufficiently exploited, while redundant or unreliable modality features can be introduced into the fused representation.

These two challenges are closely coupled. If the feature extractors of different sources model context using unrelated spatial operators, subsequent fusion must align not only heterogeneous physical measurements but also incompatible spatial responses. Conversely, even well-aligned representations can be degraded when a globally fixed fusion rule assigns excessive importance to an unreliable modality in a particular region. Co-registered HSI and SAR/LiDAR observations share object boundaries, neighborhood organization, and scene-level geometry, suggesting that common spatial patterns can serve as a stable inductive prior for cross-modal learning. At the same time, the appearance of these patterns remains content dependent, and the discriminative contribution of each source changes with land-cover type and local observation conditions. Therefore, the desired framework should combine shared contextual modeling with content-adaptive local representation, and should perform fusion according to spatially varying source reliability rather than a single scene-level preference.

To overcome these limitations, this paper proposes a Shared-Pattern Collaborative Network (SPCNet). To promote effective contextual feature learning, we design Shared-Pattern Collaborative Block (SPCB), in which a shared large-kernel operator is applied to both branches to capture common spatial patterns and enhance long-range contextual modeling. Meanwhile, an input-adaptive dynamic convolution is introduced to strengthen content-aware local feature representation. Furthermore, we introduce Region-Adaptive Modality Allocator (RAMA) to dynamically estimate the contribution of each modality in different spatial regions. Instead of using fixed fusion weights, RAMA adaptively assigns region-specific modality importance, allowing the network to emphasize the more reliable modality according to local scene characteristics.

The main contributions of this work are threefold:

  • We design SPCB to enhance cross-modal collaborative contextual feature learning. By introducing a shared large-kernel operator, SPCB encourages different modalities to learn common spatial patterns, while the dynamic convolution branch provides content-adaptive feature modeling.

  • We introduce RAMA for adaptive multi-source feature fusion. It dynamically assigns modality-specific weights to different spatial regions, allowing the network to emphasize the more reliable modality according to local scene characteristics.

  • Comprehensive experiments conducted on three benchmark datasets verify the effectiveness of the proposed SPCNet. In addition, we will publicly release the source code to facilitate the development of multi-source data fusion methods.

Multi-Source Remote Sensing Image Classification

Multi-source remote sensing classification methods can be broadly grouped according to the stage at which heterogeneous information is integrated. Decision-level fusion trains separate classifiers for different sources and combines their outputs by voting, probability aggregation, or graph-based reasoning. This strategy avoids directly reconciling heterogeneous feature dimensions and can remain effective when one modality is missing or severely degraded. However, because the sources interact only after independent inference, the learned representations cannot use complementary information to resolve ambiguous samples during feature extraction. Feature-level fusion has therefore become the dominant paradigm. It usually employs modality-specific encoders followed by summation, concatenation, attention, or learned interaction modules, enabling spectral, spatial, scattering, and elevation cues to be optimized jointly [21, 22, 29].

CNN-based methods established the basic dual-stream architecture for HSI and LiDAR/SAR joint classification. Coupled CNNs exchange information between modality branches, while spatial-spectral enhancement and recurrent attention models strengthen local feature interaction at multiple scales [22, 26, 30]. FusAtNet introduces dual attention to emphasize informative spectral-spatial responses [23], and asymmetric or collaborative fusion networks model unequal source characteristics instead of treating the modalities identically [31, 32]. Depthwise feature interaction, fractional Gabor filtering, nearest-neighbor contrastive learning, and cross-scale attention further improve multiscale and cross-source discrimination [33, 34, 35, 36]. These methods provide effective local representation and relatively stable optimization. Nevertheless, standard convolutions operate within limited receptive fields. Enlarging the network depth can propagate information over wider regions, but indirect long-range modeling may weaken boundary details and increase redundant computation.

Context Modeling and Adaptive Multi-Source Fusion

Transformer and hybrid architectures address this limitation by modeling global token relationships. Multimodal Transformer networks and ExViT establish long-range dependencies through self-attention and cross-modal attention [24, 25, 37], whereas hierarchical CNN–Transformer models combine convolutional locality with global interaction [38, 39]. Deep symmetric fusion, local information interaction, multiscale mixed CNN–Transformer modeling, and head selection further tailor global representation learning to multimodal remote sensing data [40, 41, 42, 43]. Quaternion, high-order, and progressive cross-fusion mechanisms further enrich the interaction between heterogeneous representations [27, 44, 45]. These architectures are effective for modeling nonlocal context, but dense attention may introduce considerable computation and can establish correlations between semantically unrelated positions. More importantly, global interaction alone does not ensure that different branches respond to the same object structures in a consistent manner.

Recent studies have therefore investigated shared-and-specific representations, masked pretraining, mutual guidance, and iterative cross-source refinement. S2FL learns shared and modality-specific subspace projections [18]. SS-MAE uses spatial-spectral masked modeling to learn multi-source representations [15]. Mutual-guidance or spectral-correlation networks progressively refine one source with information from another [2, 46]. Style separation, frequency-domain fusion, metric learning, and multiscale state-space modeling address cross-source discrepancies from complementary perspectives [47, 48, 49, 50]. These approaches demonstrate the importance of cross-source consistency.

Context modeling determines whether a classifier can distinguish objects using both local appearance and surrounding structure. Conventional small-kernel CNNs are effective at preserving local details, but their effective receptive fields grow slowly with network depth. Large-kernel CNNs broaden spatial support, while selective large-kernel, dynamic, and cross-guided filters adapt contextual responses to input content or another modality [46, 51, 52, 53, 54]. Cross-modal interaction and selected parameter sharing also promote source consistency [22, 32]. These directions leave a joint challenge: integrating long-range dependencies with a spatial prior shared across registered modalities and network depths, while retaining input-sensitive local filtering. SPCB addresses this by embedding the shared prior in feature extraction and pairing it with filtering that adapts to each input. The shared prior captures recurring scene geometry across modalities and network depths; adaptive filtering models local patterns that vary by source and input.

The need for shared contextual modeling follows from the spatial correspondence of registered observations. Although HSI, SAR, and LiDAR describe different physical attributes, roads, buildings, vegetation parcels, and object boundaries occupy approximately the same spatial locations. Their low-level intensity patterns differ, but their neighborhood organization and larger geometric structures are semantically related.

Fusion strategy is equally important because complementary information is not uniformly reliable. Addition and concatenation assume fixed source roles and rely on subsequent convolutions to suppress irrelevant responses. Channel attention and cross-attention learn data-dependent interactions [23, 27, 32], while intensity constraints, mutual guidance, and global weighting further improve source selection [28, 46]. GMU gates modality contributions at the representation level [55]. Recent remote-sensing methods further use cross-modal attention and semantic guidance, mutual guidance, or graph-based interaction with feature-dependent fusion [32, 46, 56]. These mechanisms support adaptive interaction, but spatial saliency within one representation does not itself compare the reliability of competing sources, and feature-wise modality gates do not explicitly allocate source contributions by location. Because shared scene geometry does not make modality evidence equally discriminative in every region, fusion must retain this spatial variation. RAMA addresses this requirement by deriving spatially varying modality weights from joint HSI-auxiliary features after shared-pattern extraction, so that source contributions can adapt to local content. This regional variation is reflected in the complementary cues provided by each source: LiDAR can distinguish spectrally similar objects with different elevations, HSI can discriminate materials in regions with similar geometry, and SAR can provide structural cues when optical observations are unreliable.

Accordingly, robust multi-source fusion requires two complementary capabilities: cross-source feature extractors should learn compatible contextual patterns, and the fusion module should assign source contributions according to regional content.

Methodology

The overall framework of the proposed Shared-Pattern Collaborative Network (SPCNet) is illustrated in Figure 2. SPCNet

Figure 2. Overview of the proposed Shared-Pattern Collaborative Network (SPCNet) for multi-source remote sensing data classification. The input multi-source data are processed by stacked Shared-Pattern Collaborative Blocks (SPCB) that exchange and learn common spatial-spectral representations through shared large kernels. Region-Adaptive Modality Allocator (RAMA) dynamically integrates complementary modality information to generate the classification map.

contains two parallel modality-specific branches: the HSI branch uses PCA followed by a 3D convolutional stem, whereas the SAR/LiDAR branch uses a 2D convolutional stem. Their features are subsequently refined by stacked SPCBs and integrated by RAMA.

Let Xh∈RH×W×B\mathbf{X}_h \in\mathbb{R}^{H \times W \times B} and Xx∈RH×W×Cx\mathbf{X}_x \in\mathbb{R}^{H \times W \times C_x} denote the spatially aligned HSI and auxiliary-source patches, respectively, where BB is the number of original hyperspectral bands and CxC_x is the number of SAR/LiDAR channels. PCA retains the leading NPCN_{\mathrm{PC}} components of the HSI before 3D convolution extracts joint spectral-spatial features, whereas the 2D auxiliary branch extracts structural, scattering, or elevation features. The two stems retain modality-specific low-level statistics while projecting both sources into compatible feature spaces.

The encoded features are successively processed by stacked Shared-Pattern Collaborative Blocks (SPCBs) and the Region-Adaptive Modality Allocator (RAMA). The internal structure of SPCB is shown in Figure 3. SPCB learns common spatial patterns while preserving source-dependent features, and RAMA produces the fused representation Ffus\mathbf{F}_{fus} for pixel-wise classification. The two modules and their end-to-end optimization are described below.

Figure 3. Details of the Shared-Pattern Collaborative Block (SPCB).

Shared-Pattern Collaborative Block (SPCB)

A. Shared-Pattern Collaborative Block (SPCB) Given an input feature X∈RH×W×C\mathbf{X} \in\mathbb{R}^{H \times W \times C}, SPCB first applies a 3×33 \times3 convolution to obtain feature F\mathbf{F}. Then, F\mathbf{F} is split along the channel dimension into Fa∈RH×W×16\mathbf{F}^{a} \in\mathbb{R}^{H \times W \times16} and Fi∈RH×W×(C−16)\mathbf{F}^{i} \in\mathbb{R}^{H \times W \times(C-16)}. Here, Fa\mathbf{F}^{a} is used as the active branch for shared-pattern learning and dynamic kernel generation, while Fi\mathbf{F}^{i} is preserved as an identity branch and later concatenated with the enhanced feature. Because these channels bypass the shared and dynamic filters before rejoining the output projection, they carry shallow responses into later mixing. Cross-depth reuse therefore acts on progressively refined features that retain earlier cues, while the dynamic branch adapts local filtering to each input.

Restricting the active branch to 16 channels keeps the large-kernel and dynamic operations lightweight. The remaining C−16C-16 channels form a direct information path, reducing the risk that repeated adaptive filtering removes fine source-specific details. The channel split thus provides a practical balance between contextual enhancement and feature preservation.

Shared Large-Kernel Pattern. The upper branch uses a shared large convolution kernel Ks∈RP×P×16×16K_s \in\mathbb{R}^{P \times P \times16 \times16} to model broad spatial context, where PP denotes the side length of the input image patch. Since the same kernel KsK_s is applied to both HSI and SAR/LiDAR features, the two branches are encouraged to extract spatial structures under a unified convolutional pattern. Through kernel sharing, SPCB establishes an implicit interaction mechanism between the two modalities, allowing the network to align and compare their spatial responses in a common feature space. The shared-pattern response is computed as:

Fsa=Ks∗Fa,(1)\mathbf{F}^{a}_{s}=K_s * \mathbf{F}^{a}, \tag*{(1)}

where ∗* denotes convolution, and Fsa∈RH×W×16\mathbf{F}^{a}_{s} \in\mathbb{R}^{H \times W \times16}. The large kernel helps the network capture long-range context and shared spatial structures. With stride one, the operator acts on the patch grid. Reflection padding of (P−1)/2(P-1)/2 pixels per side preserves the output size, while the support centered on the target pixel spans the unpadded input patch. The kernel side length equals the patch side length PP.

Compared with repeatedly stacking small convolutions to approximate a large receptive field, the P×PP \times P operator directly aggregates the spatial context of the input patch within one collaborative unit. Sharing its parameters constrains the two source branches to analyze that neighborhood with the same spatial template. Reusing the operator at different depths further maintains a consistent structural prior as the features evolve from low-level responses to more semantic representations.

The shared operator does not require the two modalities to produce identical feature values. Instead, it constrains the spatial relation used to process them, while the identity and dynamic branches retain modality-dependent evidence and local appearance variation. This selective sharing is important for aligned multi-source observations, whose geometry is common but whose radiometric responses remain heterogeneous.

Dynamic Kernel Generation. SPCB introduces a dynamic kernel branch to generate an adaptive convolution kernel according to the input feature. First, global average pooling is applied to Fa\mathbf{F}^{a}, and MLP is used to predict the dynamic kernel as:

Kd=MLP⁡(GAP⁡(Fa)),(2)K_d=\operatorname{MLP}(\operatorname{GAP}(\mathbf{F}^{a})), \tag*{(2)}

where the generated lightweight dynamic kernel is Kd∈R3×3×1×16K_d \in\mathbb{R}^{3 \times3 \times1 \times16}. For each channel of Fa\mathbf{F}^{a}, a corresponding 3×33 \times3 dynamic filter is used as:

Fda=DyDConv⁡(Fa;Kd),(3)\mathbf{F}^{a}_{d}=\operatorname{DyDConv}(\mathbf{F}^{a};K_d), \tag*{(3)}

where DyDConv denotes dynamic depth-wise convolution. The dynamic depth-wise convolution is more efficient while still providing input-adaptive feature modeling capability. The global descriptor produced by GAP summarizes the current sample and allows the MLP to generate a different group of channel-wise filters for different inputs. Because each 3×33 \times3 filter operates depth-wise, the dynamic branch avoids the parameter and computation cost of a dense input-dependent convolution. It therefore complements the fixed shared large kernel with efficient local adaptation rather than duplicating its long-range modeling role.

Region-Adaptive Modality Allocator (RAMA)

Collaborative Fusion. The outputs of the shared large-kernel branch and the dynamic kernel branch are then combined by element-wise addition, and then concatenated with Fi\mathbf{F}^{i} as:

Fc=Concat⁡(Fsa+Fda,Fi).(4)\mathbf{F}^{c}=\operatorname{Concat}(\mathbf{F}^{a}_{s}+\mathbf{F}^{a}_{d},\mathbf{F}^{i}). \tag*{(4)}

Element-wise addition first combines the common-context response and adaptive local response in the same 16-channel space. Concatenation with Fi\mathbf{F}^{i} then restores the preserved identity information. Finally, a 1×11 \times1 convolution mixes the enhanced and identity channels, and a 3×33 \times3 convolution refines their local spatial consistency to generate the output feature:

Y=Conv⁡3×3(Conv⁡1×1(Fc)).(5)\mathbf{Y}=\operatorname{Conv}_{3 \times3}(\operatorname{Conv}_{1 \times1}(\mathbf{F}^{c})). \tag*{(5)}

This two-step projection enables information exchange across channels before local spatial refinement and prepares a compatible feature representation for the next SPCB or the subsequent fusion stage.

The Region-Adaptive Modality Allocator (RAMA), illustrated in Figure 4, is designed to adaptively allocate the contributions of multi-source features. RAMA learns spatially varying modality weights and performs region-aware fusion. Instead of assigning a fixed global importance to each modality, RAMA dynamically determines which modality should contribute more at each spatial location.

Figure 4. Details of the Region-Adaptive Modality Allocator (RAMA).

For a mini-batch of MM co-registered patch pairs, the input features Fh,Fx∈RM×C×H×W\mathbf{F}_{h},\mathbf{F}_{x} \in\mathbb{R}^{M \times C \times H \times W} are independently processed by 3×33 \times3 convolutions to obtain F^h\hat{\mathbf{F}}_{h} and F^x\hat{\mathbf{F}}_{x}. Their concatenation has shape M×2C×H×WM \times2C \times H \times W, and a 1×11 \times1 projection produces Fr\mathbf{F}_{r} of the same shape.

Fcat=Concat⁡(F^h,F^x),(6)\mathbf{F}_{\mathrm{cat}}=\operatorname{Concat}(\hat{\mathbf{F}}_{h},\hat{\mathbf{F}}_{x}), \tag*{(6)}
Fr=Reshape⁡(Conv⁡1×1(Fcat)).(7)\mathbf{F}_{r}=\operatorname{Reshape}(\operatorname{Conv}_{1 \times1}(\mathbf{F}_{\mathrm{cat}})). \tag*{(7)}

Flattening the spatial axes gives M×2C×NsM \times2C \times N_s, where Ns=HWN_s=HW. An MLP produces two logits per feature pixel, with grid shape M×2×NsM \times2 \times N_s. Softmax over modalities and reshaping yield pixelwise maps [Ah,Ax]∈RM×2×H×W[\mathbf{A}_{h},\mathbf{A}_{x}] \in\mathbb{R}^{M \times2 \times H \times W}, each of shape M×1×H×WM \times1 \times H \times W.

[Ah(i,j),Ax(i,j)]=Softmax⁡m(MLP⁡(Fr(i,j))),(8)[\mathbf{A}_{h}(i,j),\mathbf{A}_{x}(i,j)]=\operatorname{Softmax}_{m}(\operatorname{MLP}(\mathbf{F}_{r}(i,j))), \tag*{(8)}
Ah(i,j)+Ax(i,j)=1.\mathbf{A}_{h}(i,j)+\mathbf{A}_{x}(i,j)=1.

Each map is broadcast over CC channels to weight its refined feature. The maps allocate contributions at each patch feature pixel, while input-independent global coefficients α\alpha and β\beta calibrate their overall contributions.

F^h′=Ah⊙F^h,F^x′=Ax⊙F^x,(9)\hat{F}'_h = A_h \odot\hat{F}_h,\qquad\hat{F}'_x = A_x \odot\hat{F}_x, \tag*{(9)}
Ffus=αF^h′+βF^x′,α+β=1.(10)F_{fus} = \alpha\hat{F}'_h + \beta\hat{F}'_x,\qquad\alpha+ \beta= 1. \tag*{(10)}

Optimization Objective

For an input sample with KK land-cover categories, the classification head converts FfusF_{fus} into the predicted class-probability vector pnp_n. SPCNet is trained end-to-end using the standard cross-entropy objective:

Lce=−1N∑n=1N∑k=1Kyn,klog⁡pn,k,(11)\mathcal{L}_{ce} = -\frac{1}{N}\sum_{n=1}^{N}\sum_{k=1}^{K} y_{n,k}\log p_{n,k}, \tag*{(11)}

where NN is the number of labeled training samples in a mini-batch and yn,ky_{n,k} is the one-hot ground-truth indicator. The classification loss jointly supervises the modality-specific encoders, SPCBs, dynamic kernel generator, RAMA, and final prediction head, so the shared patterns and regional allocations are learned specifically for discriminative land-cover recognition. During back-propagation, the shared spatial operator, dynamic local filters, and allocation maps are optimized jointly rather than through separate auxiliary objectives, ensuring that each mechanism is directly guided by the final classification error.

Experimental Results and Analysis

Experimental Setup and Dataset Description

To evaluate the effectiveness and generality of SPCNet, experiments were conducted on three benchmark multi-source remote sensing datasets: Houston2013, Augsburg, and Houston2018. All methods were implemented in PyTorch and trained on an NVIDIA H100 GPU with 80 GB of memory. The same training and testing partitions were used for all compared methods to ensure a fair evaluation. The Houston datasets pair HSI with LiDAR-derived structural information, whereas Augsburg combines HSI with polarimetric SAR (PolSAR) observations. These complementary settings allow the proposed network to be evaluated under both spectral–elevation and spectral–scattering fusion conditions.

Houston2013 Dataset. The Houston2013 dataset [17] was released for the 2013 IEEE GRSS Data Fusion Contest and covers the University of Houston campus and its surrounding urban area. The airborne HSI has a spatial size of 349×1905349 \times1905 pixels and 144 spectral bands spanning 380–1050 nm, with a ground sample distance of 2.5 m. The co-registered LiDAR data provide complementary elevation cues for buildings, vegetation, and transportation infrastructure. The scene contains 15 land-cover classes, including several visually similar paved and built-up categories as well as narrow road and railway structures. It therefore provides a representative urban setting for assessing whether cross-modal contextual modeling can preserve object continuity while separating classes with similar spectral responses.

Augsburg Dataset. The Augsburg dataset [18] covers the vicinity of Augsburg, Germany, and contains HySpex HSI together with Sentinel-1 PolSAR data. Both modalities are provided at a 30-m ground sample distance with a spatial size of 332×485332 \times485 pixels. The HSI contains 180 bands from 0.4 to 2.5 μ\mum, while the SAR modality includes VV and VH intensity together with the real and imaginary components of the off-diagonal PolSAR covariance term. Its ground-truth labels are derived from OpenStreetMap and describe seven urban and peri-urban classes. The coarse spatial resolution, heterogeneous land use, and highly uneven class distribution make this dataset particularly useful for testing whether a fusion model can exploit scattering information without amplifying source-specific noise or minority-class ambiguity.

Houston2018 Dataset. The Houston2018 dataset [18] was introduced in the 2018 IEEE GRSS Data Fusion Contest and covers the University of Houston campus together with neighboring urban areas. The original release provides HSI, LiDAR, and optical observations; following the multi-source HSI–LiDAR setting, only the HSI and LiDAR modalities are used in this work. The HSI contains 48 bands over the 380–1050 nm range at 1-m spatial resolution, while the LiDAR data have a finer native spatial resolution and are spatially aligned with the HSI for classification. This larger and more diverse scene includes 20 land-cover categories spanning vegetation, buildings, roads, parking areas, railways, and small objects such as cars and trains. Its extensive and strongly imbalanced labeled set provides a demanding test of generalization across complex urban spatial layouts.

The class-wise training and testing samples of Houston2013, Augsburg, and Houston2018 are listed in Table I, Table II, and Table III, respectively. Figure 5 presents the false-color HSI, the corresponding SAR/LiDAR data, and the ground-truth maps of the three datasets.

No.NameColorTrainTest
1Healthy grass1981053
2Stressed grass1901064
3Synthetic grass192505
4Trees1881056
5Soil1861056
6Water82143
7Residential1961072
8Commercial1911053
9Road1931059
10Highway1911036
11Railway1811054
12Parking lot 11921041
13Parking lot 2184285
14Tennis court181247
15Running track187473
Total283212197

Table I. Number of training and test samples for the Houston2013 dataset.

No.NameColorTrainTest
1Forest14613361
2Residential area26430065
3Industrial area213830
4Low plants24826609
5Allotment52523
6Commercial area71638
7Water231507
Total76177533

Table II. Number of training and test samples for the Augsburg dataset.

Figure 5. Visualization of the three multi-source datasets. (a) Houston2013 dataset. (b) Augsburg dataset. (c) Houston2018 dataset.

No.NameColorTrainTest
1Healthy grass100038196
2Stressed grass1000129008
3Artificial turf10001736
4Evergreen trees100053322
5Deciduous trees100019172
6Bare earth100017064
7Water500564
8Residential buildings1000157995
9Non-residential buildings1000893769
10Roads1000182283
11Sidewalks1000135035
12Crosswalks10005059
13Major thoroughfares1000184438
14Highways100038438
15Railways100026748
16Paved parking lots100044932
17Unpaved parking lots250337
18Cars100025289
19Trains100020479
20Stadium seats100026296
Total187502000160

Table III. NUMBER OF TRAINING AND TEST SAMPLES FOR THE HOUSTON2018 DATASET.

After normalization, HSI is projected by PCA and each labeled target pixel defines a co-registered centered patch pair supervised by its center label. Reflection padding of (P−1)/2(P-1)/2 pixels per side preserves output size. Both modalities receive the same horizontal or vertical flip with probability 0.5. All methods use these input settings and the same train/test partitions and optimization settings (Adam, initial learning rate 2×10−42 \times10^{-4}, weight decay 0, batch size 128, 35 epochs, dropout 0.05, and seed 6). We compare our SPCNet with ten state-of-the-art methods, including FusAtNet [23], S2ENet [26], AsyFFNet [31], ExViT [24], HCT [38], MACN [39], MICF-Net [32], GCCQTNet [44], CHNet [45], and MGMNet [46]. To quantitatively assess the classification performance, the overall accuracy (OA), average accuracy (AA), and Kappa coefficient are employed for evaluation.

Analysis of Important Parameters

Depth of SPCB Layers. Figure 6 shows the OA variation as the number of SPCB layers increases from 2 to 10. The best results are obtained with eight layers on Houston2013 (93.53%) and Houston2018 (93.44%), whereas Augsburg reaches its highest OA of 92.34% with six layers. Increasing the depth initially improves the representation of shared spatial patterns and cross-source interactions. However, further stacking does not provide a consistent benefit, because redundant feature interactions can increase optimization difficulty and computation without contributing discriminative context. The selected depths are 8, 6, and 8 for Houston2013, Augsburg, and Houston2018, respectively.

Figure 6. Depth of SPCB Layers.

Number of Principal Components for HSI. Figure 7 evaluates the number of retained principal components. On Houston2013, OA increases sharply to 93.53% when 30 components are retained, while 40 and 50 components introduce redundant spectral information and reduce performance. Augsburg and Houston2018 achieve their best OAs of 92.35% and 93.65%, respectively, with 20 components. The different optima reflect the distinct spectral redundancy and scene composition of the three datasets. Accordingly, 30 principal components are retained for Houston2013, whereas 20 components are retained for Augsburg and Houston2018.

Figure 7. Number of Principal Components for HSI.

Size of the Input Image Patches. Figure 8 studies the input patch size. Houston2013 reaches 93.53% OA with an 11-pixel patch, Augsburg reaches 92.68% with a 9-pixel patch, and Houston2018 reaches 93.77% with a 13-pixel patch. Small patches provide insufficient contextual evidence for neighboring land-cover categories, whereas excessively large patches can include unrelated pixels and weaken boundary discrimination. The different optima therefore reflect differences in object scale and spatial heterogeneity among the three scenes.

Figure 8. Size of the Input Image Patches.

Performance Comparison

We compare SPCNet with ten representative state-of-the-art multi-source classification methods, following the comparison protocol commonly adopted in recent multi-source remote sensing studies. These baselines cover complementary design routes from attention-based CNNs to hybrid Transformers and high-order adaptive fusion. FusAtNet [23] applies self-attention to HSI spectral features and uses attention cues derived from LiDAR to enhance spatial representations. S2ENet [26] introduces a spatial-spectral cross-modal enhancement module to exchange information between HSI and LiDAR branches. AsyFFNet [31] adopts an asymmetric fusion strategy to account for the unequal information contributions of HSI and SAR. ExViT [24] extends Vision Transformer with separable convolutions and a cross-modality attention module, while HCT [38] combines hierarchical CNNs with a cross-token attention encoder for joint classification. MACN [39] mixes self-attention and convolution so that local structures and global dependencies can be modeled in a unified layer. MICF-Net [32] employs dual-branch cross-source attention to extract global context and spatial relationship similarities, together with adaptive modulation to balance the optimization of different sources. GCCQTNet [44] uses an independent squeeze-expansion-like fusion structure and a cross-memory quaternion Transformer to reduce multimodal heterogeneity and model complex intra- and inter-modality relationships. CHNet [45] progressively removes spectral-spatial redundancy and combines dynamic feature selection with factorized high-order learning to improve feature discriminability. MGMNet [46] embeds mutual guidance in a two-stream network, where context-aware filters generated from one source are used to refine the features of the other. These methods provide a representative coverage of local convolutional modeling, cross-attention, global token interaction, high-order fusion, and mutual-guidance strategies. Tables IV–VI report the class-wise and overall performance, while Figs. 9–11 show the corresponding classification maps. In addition to OA, AA, and Kappa, we examine confusing categories, boundary regions, and spatially fragmented objects to evaluate both numerical accuracy and spatial consistency.

Results on Houston2013 Dataset. The quantitative results in Table IV show that SPCNet achieves the best OA, AA, and Kappa values of 93.53%, 94.85%, and 92.98%, respectively. Compared with the strongest competing OA of 90.98% obtained by MGMNet, SPCNet provides an absolute gain of 2.55 percentage points. The improvement is particularly evident for difficult transportation-related classes: the accuracy for Highway reaches 94.79%, exceeding the second-best result by 9.65 percentage points, while Railway and Parking lot 2 reach 99.53% and 96.49%, respectively. These results indicate that the shared large-kernel patterns capture extended spatial structures and that RAMA effectively exploits the complementary elevation information from LiDAR. As shown in Figure 9, several competing methods produce fragmented predictions and blurred boundaries in the enlarged region. SPCNet generates a map that is more consistent with the ground truth, with clearer object contours and fewer isolated errors. The agreement between the quantitative and visual results demonstrates strong discriminative capability in complex urban scenes.

Figure 9. Visualized classification results of different methods for the Houston2013 dataset. (a) Ground truth. (b) FusAtNet. (c) S2ENet. (d) AsyFFNet. (e) ExViT. (f) HCT. (g) MACN. (h) MICF-Net. (i) GCCQTNet. (j) CHNet. (k) MGMNet. (l) SPCNet.

Results on Augsburg Dataset. Table V shows that SPCNet obtains the best OA and Kappa values of 92.68% and 89.37%, outperforming the second-best results by 1.50 and 2.14 percentage points, respectively. Its AA of 64.84% remains competitive with the best value of 65.51%. At the category level, SPCNet maintains strong results for Low plants, Forest, and Residential area, although the best score in each individual category is sometimes achieved by a competing method. The 2.32% Commercial-area accuracy is consistent with severe class imbalance (only seven training pixels) and the coarse 30-m resolution of Augsburg’s OpenStreetMap-derived functional labels [18], which may group mixed surfaces whose spectral and scattering responses overlap with Residential or Industrial areas. This sparse supervision and within-class variation depress minority-class recall and AA, while the common classes largely determine OA. The visual results in Figure 10 further support this observation. SPCNet preserves more continuous forest and residential structures in the enlarged region and suppresses isolated errors that are visible in several competing maps. This demonstrates that collaborative HSI-SAR

Figure 10. Visualized classification results of different methods for the Augsburg dataset. (a) Ground truth. (b) FusAtNet. (c) S2ENet. (d) AsyFFNet. (e) ExViT. (f) HCT. (g) MACN. (h) MICF-Net. (i) GCCQTNet. (j) CHNet. (k) MGMNet. (l) SPCNet.

ClassFusAtNetS2ENetAsyFFNetExViTHCTMACNMICF-NetGCCQTNetCHNetMGMNetSPCNet
Healthy grass82.5381.4882.7282.7281.1082.9082.9182.0583.1083.0082.91
Stressed grass85.1584.6883.9385.1586.09100.083.4696.3385.1585.1599.72
Synthetic grass98.8199.01100.099.2199.8099.8099.8099.4199.6099.60100.0
Trees92.9092.1491.9591.9598.0198.1197.6391.0092.6187.5993.37
Soil100.0100.0100.0100.0100.0100.0100.099.5299.90100.0100.0
Water98.60100.095.8098.6095.1095.8095.8095.8095.8089.51100.0
Residential85.4589.9396.1886.6683.4978.1783.4990.9588.2595.6285.63
Commercial81.8190.1881.1491.0586.4380.4681.9195.6781.6276.4288.41
Road83.7692.6386.1291.1286.6990.9382.1587.5489.6193.9682.34
Highway53.6764.8664.5864.8679.0555.8866.2265.1573.8485.1494.79
Railway79.6597.6086.7677.0695.9799.3395.5986.8597.4194.9199.53
Parking lot 191.2688.0990.5988.7698.4699.7194.6299.1499.9098.3799.62
Parking lot 281.0591.9392.2887.7290.8888.7781.4082.8189.4789.1296.49
Tennis court100.0100.095.1497.17100.0100.0100.0100.0100.099.60100.0
Running track98.73100.0100.098.52100.0100.099.79100.0100.0100.0100.0
OA85.3289.5687.9587.4290.6689.8188.0990.4090.3090.9893.53
AA87.5691.5089.8189.3792.0791.3389.6591.4891.7591.8794.85
Kappa84.1288.6986.9686.3889.8688.9387.1289.5989.4790.2092.98
ClassFusAtNetS2ENetAsyFFNetExViTHCTMACNMICF-NetGCCQTNetCHNetMGMNetSPCNet
Forest93.7898.1096.0190.0496.9297.9595.9895.2696.2496.2896.84
Residential area97.5899.0897.9695.4498.9698.6898.4696.8099.2297.7698.71
Industrial area26.4812.1953.2934.5833.0545.3058.3868.2044.2349.6164.10
Low plants97.6791.7891.7490.6892.3694.2491.7591.5293.8392.3297.08
Allotment52.7745.1265.3951.8254.1151.2440.7341.6854.6856.6052.39
Commercial area24.661.224.5828.636.626.7423.5313.9719.855.152.32
Water47.5124.0949.6417.6547.4049.2744.8146.8046.7449.9342.47
OA90.6288.2290.1586.6589.8391.1890.7490.2291.1490.1092.68
AA62.9253.0865.5158.4061.3463.3564.8164.8964.9763.9564.84
Kappa86.3386.4785.7480.7985.2887.2386.6586.0087.1085.7589.37

Table IV. Classification performance of different methods on the Houston2013 dataset. The bold entries denote the best results.

representation learning improves both regional consistency and robustness to scattering variations.

Results on Houston2018 Dataset. As reported in Table VI, SPCNet achieves the best OA, AA, and Kappa values of 93.77%, 96.19%, and 91.93%, respectively. Relative to the second-best results, the gains are 1.14 percentage points in OA, 0.26 percentage points in AA, and 1.45 percentage points in Kappa. SPCNet is especially effective for Stressed grass, Sidewalks, and Major thoroughfares, obtaining accuracies of 94.92%, 85.47%, and 90.96%, respectively. The consistently high scores across vegetation, building, and transportation categories show that the proposed network balances spectral discrimination with structural modeling. In Figure 11, SPCNet produces cleaner class regions and more complete linear structures than the competing methods. Roads, railways, and the boundaries between vegetation and buildings are better retained in the enlarged region, while scattered prediction noise is reduced. This consistency between visual quality and quantitative performance confirms the generalization capability of SPCNet on a larger and more diverse urban scene.

ClassFusAtNetS2ENetAsyFFNetExViTHCTMACNMICF-NetGCCQTNetCHNetMGMNetSPCNet
Healthy grass60.6999.1596.9093.3497.3889.5394.4091.7191.2492.8392.26
Stressed grass93.8584.3591.8293.4489.4093.5089.7994.2293.9493.5194.92
Artificial turf100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0
Evergreen trees99.0298.1399.2995.8798.9099.1099.2498.5898.5099.6399.40
Deciduous trees94.7386.3999.5296.8598.8998.4998.8598.7199.1799.3499.04
Bare earth99.8799.3399.98100.0100.099.9699.8499.9699.9899.9699.99
Water100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0
Residential buildings95.5197.1795.1697.3995.3095.3194.8396.4194.8197.2297.13
Non-residential buildings93.6493.9395.0293.0095.2593.8695.3994.1694.3995.6496.41
Roads70.9470.1772.7272.1872.7772.1173.3377.7982.4272.0478.13
Sidewalks79.1172.1177.1869.6973.2780.6978.1082.5577.9984.5085.47
Crosswalks86.0682.4793.1488.1298.9597.6997.6397.1197.6197.9894.15
Major thoroughfares76.1289.4484.6786.1280.9079.0577.7388.0383.8489.6490.96
Highways98.8998.5399.5399.5098.0196.8697.0498.4396.8099.4199.18
Railways99.8899.2199.9099.8799.7499.9099.7699.7799.5799.9099.76
Paved parking lots95.8895.6299.1897.0297.0797.7498.7299.3597.2297.7798.07
Unpaved parking lots100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0
Cars94.2596.7796.6498.1399.0198.7498.1897.4499.0899.2798.97
Trains99.84100.0100.099.9899.9799.7499.8899.8999.9799.9999.98
Stadium seats100.0100.099.8999.9999.9899.7499.9899.8499.95100.0100.0
OA89.0990.0591.2489.9890.5690.3190.6592.1391.7592.6393.77
AA91.9193.1495.0294.0294.7494.6194.6395.7095.3295.9396.19
Kappa85.9687.2088.7187.1487.8187.5587.9489.8689.3790.4891.93

Table VI. Classification performance of different methods on the Houston2018 dataset. The bold entries denote the best results.

Figure 11. Visualized classification results of different methods for the Houston2018 dataset. (a) Ground truth. (b) FusAtNet. (c) S2ENet. (d) AsyFFNet. (e) ExViT. (f) HCT. (g) MACN. (h) MICF-Net. (i) GCCQTNet. (j) CHNet. (k) MGMNet. (l) SPCNet.

Ablation Study

Effectiveness of Essential Components. To evaluate the contributions of SPCB and RAMA, Table VII compares the baseline, the two single-component variants, and the complete SPCNet using OA on all three datasets. The baseline obtains 88.56%, 89.75%, and 89.66% on Houston2013, Augsburg, and Houston2018, respectively. The variant without SPCB, corresponding to the RAMA-only configuration, improves these results by 1.77, 1.67, and 2.09 percentage points. The variant without RAMA, corresponding to the SPCB-only configuration, improves OA by 2.36, 0.44, and 1.47 percentage points on Houston2013, Augsburg, and Houston2018, respectively. Thus, both modules improve the baseline, although their individual gains vary across datasets. When SPCB and RAMA are jointly incorporated, OA reaches 93.53%, 92.68%, and 93.77%, which are 4.97, 2.93, and 4.11 percentage points above the baseline. The full model further exceeds the RAMA-only configuration by 3.20, 1.26, and 2.02 percentage points, and the SPCB-only configuration by 2.61, 2.49, and 2.64 percentage points. These results indicate that SPCB and RAMA

Model VariantHouston 2013AugsburgHouston 2018
Baseline88.5689.7589.66
w/o SPCB90.3391.4291.75
w/o RAMA90.9290.1991.13
Full SPCNet93.5392.6893.77

Table VII. Ablation study of the proposed SPCNet in terms of OA (%).

provide complementary benefits under the same evaluation protocol.

Shared-Pattern Collaborative Block (SPCB). Table VIII isolates the Shared Large-Kernel Pattern and Dynamic Kernel Generation while keeping the remaining network, including RAMA, unchanged. The Shared-LK-only and DK-only variants obtain 92.67%, 92.19%, and 93.24%, and 91.13%, 91.62%, and 92.12%, respectively, on Houston2013, Augsburg, and Houston2018. The Shared-LK-only variant is higher than the DK-only variant by 1.54, 0.57, and 1.12 percentage points, indicating a stronger individual contribution of the shared long-range pattern in these experiments. The complete SPCB reaches 93.53%, 92.68%, and 93.77%, outperforming the Shared-LK-only variant by 0.86, 0.49, and 0.53 percentage points and the DK-only variant by 2.40, 1.06, and 1.65 percentage points. To assess input-conditioned filtering, Table VIII replaces the dynamic branch with two static controls of comparable capacity while retaining Shared-LK and the remaining network.

Model VariantHouston 2013AugsburgHouston 2018
Shared-LK only92.6792.1993.24
DK only91.1391.6292.12
Static-DW (generator)93.2191.7893.33
Static-DW (pointwise)92.6691.9093.26
Full SPCB93.5392.6893.77

Table VIII. Ablation study of SPCB in terms of OA (%).

remaining network. Static-DW (generator) feeds a learned constant into the original generator; Static-DW (pointwise) uses a fixed depth-wise kernel with pointwise expansion and projection. Neither consistently improves on Shared-LK only: on Augsburg, OA drops by 0.41 and 0.29 percentage points, respectively, and both remain below Full SPCB on all three datasets. This may reflect the inability of fixed filters to adapt to input structures and their redundant responses with Shared-LK.

Region-Adaptive Modality Allocator (RAMA). Table IX compares four fusion strategies with different levels of source-selection adaptivity. 1×11 \times1 convolution concatenates the features and learns a linear channel mixture, but the same convolutional rule is applied at every spatial location. Global allocator predicts one pair of modality-importance weights from a globally aggregated representation and applies the weights to the entire scene. Regional allocation generates spatially varying allocation maps so that different regions can favor different sources; to isolate the contribution of regional allocation, its final fusion coefficients are fixed to α=β=0.5\alpha= \beta= 0.5. Full RAMA combines the regional allocation maps with learnable fusion coefficients, enabling both spatial source selection and adaptive final aggregation.

Fusion StrategyHouston 2013AugsburgHouston 2018
1×11 \times1 convolution91.1990.7492.58
Global allocator93.5392.2693.53
Regional allocation93.2992.3193.63
RAMA93.5392.6893.77

Table IX. Ablation study of RAMA in terms of OA (%).

The 1×11 \times1 convolution produces OA values of 91.19%, 90.74%, and 92.58% on Houston2013, Augsburg, and Houston2018, respectively. The global allocator improves the results to 93.53%, 92.26%, and 93.53%, demonstrating the benefit of learning source reliability, although one global weight pair cannot describe all local regions. Regional allocation obtains 93.29%, 92.31%, and 93.63%, showing that spatially varying source selection is useful in Augsburg and Houston2018, although it is slightly lower than the global allocator on Houston2013. Full RAMA achieves 93.53%, 92.68%, and 93.77%, tying the global allocator on Houston2013 and giving the best OA on the other two datasets. Compared with the global allocator, Full RAMA improves OA by 0, 0.42, and 0.24 percentage points, respectively; compared with regional allocation, it improves OA by 0.24, 0.37, and 0.14 percentage points. These results indicate that regional allocation and learnable final aggregation are complementary, yielding the best or tied-best OA across the three datasets.

Shared-Pattern Collaborative Block (SPCB) Sharing Strategy. Table X analyzes the effect of sharing the large-kernel operator across modalities and network depths. Its first row corresponds to an independently parameterized setting in which the large-kernel operators are not shared across modalities or network depths, rather than a setting that removes the large-kernel branch. Without either form of sharing, OA is 90.46%, 91.42%, and 92.28% on Houston2013, Augsburg, and Houston2018, respectively. Cross-modal sharing alone changes these results to 90.52%, 92.26%, and 93.63%, giving gains of 0.06, 0.84, and 1.35 percentage points. Cross-depth sharing alone obtains 91.34%, 91.76%, and 93.33%, improving OA by 0.88, 0.34, and 1.05 percentage points. When both sharing strategies are enabled, OA reaches 93.53%, 92.68%, and 93.77%, exceeding the independently parameterized variant by 3.07, 1.26, and 1.49 percentage points. The combined strategy achieves the highest OA on all three datasets. These results support the complementarity of cross-modal and cross-depth sharing, while the dataset-dependent gains suggest that their relative contributions are not uniform across scenes. On Houston2013, the gain from cross-modal sharing increases from 0.06 to 2.19 percentage points when cross-depth reuse is enabled. Similar paved/built-up classes and narrow roads and railways make spatial structure informative [17]. Reusing the shared operator across modalities and depths helps retain this geometry as features evolve, while identity channels and dynamic filters preserve early and source-specific cues.

Modality SharedDepth SharedHouston 2013AugsburgHouston 2018
×\times×\times90.4691.4292.28
✓\checkmark×\times90.5292.2693.63
×\times✓\checkmark91.3491.7693.33
✓\checkmark✓\checkmark93.5392.6893.77

Table X. Ablation study of the large-kernel sharing strategies in terms of OA (%).

Computational Complexity Analysis

Table XI compares the number of parameters and floating-point operations (FLOPs) of different models. SPCNet contains 1.0893 M parameters and requires 0.2470 GFLOPs. Its parameter count is approximately 97.02% and 94.77% lower than those of FusAtNet and CHNet, respectively, indicating that the proposed modules do not rely on a heavyweight backbone. Although several highly lightweight competitors require less computation, SPCNet still has substantially lower FLOPs than FusAtNet, MACN, and CHNet. With controllable parameter and computational costs, the model achieves the highest overall accuracy (OA) on all three datasets. These results demonstrate that SPCNet achieves a favorable tradeoff between accuracy and efficiency.

Discussion

Current experiments use registered multimodal imagery and do not quantify the effect of residual misregistration on cross-modal sharing. Future work can introduce controlled spatial offsets to measure this sensitivity and investigate registration-aware feature alignment to improve robustness.

RAMA learns location-wise weights under the classification objective to allocate branch contributions, but the correspondence between these weights and sensor quality remains to be assessed. Future work can introduce controlled modality-specific degradation to examine how allocations respond to changing input quality and develop quality-aware fusion.

Although experiments cover three datasets and HSI–LiDAR and HSI–SAR combinations, they are evaluated using dataset-specific training and test splits, leaving their ability to generalize to unseen geographic scenes unexamined. Future work can combine cross-scene evaluation with domain generalization or lightweight adaptation to assess and improve transfer across acquisition conditions.

Conclusion and Future Work

We propose SPCNet for multi-source remote sensing data classification. Specifically, the SPCB introduces a shared large-kernel mechanism and dynamic convolution to enhance common spatial-pattern learning while preserving modality-specific representations. Moreover, the RAMA adaptively assigns modality weights for different spatial regions, enabling more effective and reliable cross-modal feature fusion. Experimental results on the Houston2013, Augsburg, and Houston2018 datasets demonstrate that SPCNet achieves superior classification performance compared with several state-of-the-art methods. Future work will investigate more efficient dynamic-kernel generation and evaluate the proposed collaborative learning strategy in broader multi-source remote sensing scenarios.

MetricsFusAtNetS2ENetAsyFFNetExViTHCTMACNMICF-NetGCCQTNetCHNetMGMNetSPCNet
Params (M)36.57601.17521.06540.23395.91862.34720.13321.232220.81564.30101.0893
FLOPs (G)6.69500.27080.17470.06420.08490.33380.03770.04930.32140.14550.2470

Table XI. Comparison of Model Parameters, Floating-Point Operations (FLOPs).

References

  1. [1]M. Ahmad, S. Shabbir, S. K. Roy, D. Hong, X. Wu, J. Yao, A. M. Khan, M. Mazzara, S. Distefano, and J. Channousset, “Hyperspectral image classification—traditional to deep models: A survey for future prospects,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 968–999, 2022.DOI
  2. [2]C. Gong, F. Gao, J. Lin, J. Dong, and Q. Du, “Representative spectral correlation network for multisource remote sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 64, pp. 1–14, 2026.DOI
  3. [3]Q. Lv, Z. Qi, H. Li, W. Song, D. Huang, Z. Ding, and Q. Shi, “Evaluating the potential of multi-source remote sensing and urban big data for urban land use classification,” International Journal of Digital Earth, vol. 18, no. 1, p. 2546519, 2025.
  4. [4]X. Shu, C. Ye, Z. Xu, R. Liao, and S. Zhang, “A multiscale physically-based approach to urban flood risk assessment using abm and multi-source remote sensing data,” International Journal of Disaster Risk Reduction, vol. 119, p. 105332, 2025.DOI
  5. [5]Y. Gao, J. Li, Y. Wu, L. Zhang, D. Lei, L. Kong, and L. Yue, “PyS²CF-Mamba: A pyramid spatial–spectral competitive fusion Mamba network for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, vol. 23, pp. 1–5, 2026.DOI
  6. [6]S. Hao, W. Wang, and M. Salzmann, “Geometry-aware deep recurrent neural networks for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 3, pp. 2448–2460, 2021.DOI
  7. [7]K. S. Roy, S. Manna, T. Song, and L. Bruzzone, “Attention-based adaptive spectral–spatial kernel resnet for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 9, pp. 7831–7843, 2021.DOI
  8. [8]D. Hong, C. Li, N. Yokoya, B. Zhang, X. Jia, A. Plaza, P. Gamba, J. A. Benediktsson, and J. Channousset, “Hyperspectral imaging,” Nature Reviews Methods Primers, vol. 6, no. 1, p. 19, 2026.arxiv.org/abs/2508.08107
  9. [9]S. Li, W. Song, L. Fang, Y. Chen, P. Ghamisi, and J. A. Benediktsson, “Deep learning for hyperspectral image classification: An overview,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 9, pp. 6690–6709, 2019.DOI
  10. [10]B. Rasti, D. Hong, R. Hang, P. Ghamisi, X. Kang, J. Channousset, and J. A. Benediktsson, “Feature extraction for hyperspectral imagery: The evolution from shallow to deep: Overview and toolbox,” IEEE Geoscience and Remote Sensing Magazine, vol. 8, no. 4, pp. 60–88, 2020.
  11. [11]J. Li, K. Zheng, L. Gao, Z. Han, Z. Li, and J. Channousset, “Enhanced deep image prior for unsupervised hyperspectral image super-resolution,” IEEE Transactions on Geoscience and Remote Sensing, vol. 63, pp. 1–18, 2025.DOI
  12. [12]J. Li, K. Zheng, L. Gao, L. Ni, M. Huang, and J. Chansussot, “Model-informed multistage unsupervised network for hyperspectral image super-resolution,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–17, 2024.DOI
  13. [13]P. Ghamisi, R. Gloaguen, P. M. Atkinson, J. A. Benediktsson, B. Rasti, N. Yokoya, Q. Wang, B. Hofle, L. Bruzzone, F. Bovolo, M. Chi, and K. Anders, “Multisource and multitemporal data fusion in remote sensing: A comprehensive review of the state of the art,” IEEE Geoscience and Remote Sensing Magazine, vol. 7, no. 1, pp. 6–39, 2019.DOI
  14. [14]Z. Xue, X. Yu, A. Yu, B. Liu, P. Zhang, and S. Wu, “Self-supervised feature learning for multimodal remote sensing image land cover classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2022.DOI
  15. [15]J. Lin, F. Gao, X. Shi, J. Dong, and Q. Du, “SS-MAE: Spatial–spectral masked autoencoder for multisource remote sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–14, 2023.arxiv.org/abs/2311.04442
  16. [16]J. Hu, R. Liu, D. Hong, A. Camero, J. Yao, M. Schneider, F. Kurz, K. Segl, and X. Zhu, “MDAS: A new multimodal benchmark dataset for remote sensing,” Earth System Science Data, vol. 15, no. 1, pp. 113–131, 2023.
  17. [17]C. Debes, A. Merentitis, R. Heremans, J. Hahn, N. Frangiadakis, T. van Kasteren, W. Liao, R. Bellens, A. Pizurica, S. Gautama, W. Philips, S. Prasad, Q. Du, and F. Pacifici, “Hyperspectral and LiDAR data fusion: Outcome of the 2013 GRSS data fusion contest,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 7, no. 6, pp. 2405–2418, 2014.DOI
  18. [18]D. Hong, J. Hu, J. Yao, J. Chansussot, and X. X. Zhu, “Multimodal remote sensing benchmark datasets for land cover classification with a shared and specific feature learning model,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 178, pp. 68–80, 2021.arxiv.org/abs/2105.10196
  19. [19]K. Ding, T. Lu, W. Fu, S. Li, and F. Ma, “Global–local transformer network for HSI and LiDAR data joint classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2022.
  20. [20]D. Hong, L. Gao, N. Yokoya, J. Yao, J. Chansussot, Q. Du, and B. Zhang, “More diverse means better: Multimodal deep learning meets remote-sensing imagery classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 5, pp. 4340–4354, 2021.DOI
  21. [21]X. Xu, W. Li, Q. Ran, Q. Du, L. Gao, and B. Zhang, “Multisource remote sensing data classification based on convolutional neural network,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 2, pp. 937–949, 2018.DOI
  22. [22]H. Wang, Z. Li, P. Ghamisi, D. Hong, G. Xia, and Q. Liu, “Classification of hyperspectral and LiDAR data using coupled CNNs,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 7, pp. 4939–4950, 2020.
  23. [23]S. Mohla, S. Pande, B. Banerjee, and S. Chaudhuri, “FusAtNet: Dual attention based spectral-spatial multimodal fusion network for hyperspectral and LiDAR classification,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020, pp. 416–425.
  24. [24]J. Yao, B. Zhang, C. Li, D. Hong, and J. Chansussot, “Extended vision transformer (ExViT) for land use and land cover classification: A multimodal deep learning framework,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–15, 2023.DOI
  25. [25]Y. Zhang, S. Xu, D. Hong, H. Gao, C. Zhang, M. Bi, and C. Li, “Multi-modal transformer network for hyperspectral and LiDAR classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–17, 2023.
  26. [26]S. Fang, K. Li, and Z. Li, “S2ENet: Spatial–spectral cross-modal enhancement network for classification of hyperspectral and LiDAR data,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022.DOI
  27. [27]H. Pan, Q. Zhang, H. Ge, M. Liu, and C. Shi, “PyionNet: Pyramidal progressive cross-fusion network for joint classification of hyperspectral and LiDAR data,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 18, pp. 12 042–12 058, 2025.DOI
  28. [28]C. Liang, Y. Zhao, Y. Song, and K. Ni, “IDNet: Intensity-constrained detail-enhanced network for hyperspectral and LiDAR collaborative classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 63, pp. 1–15, 2025.DOI
  29. [29]X. Wu, D. Hong, and J. Chansussot, “Convolutional neural networks for multimodal remote sensing data classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–10, 2022.DOI
  30. [30]H.-C. Li, W.-S. Hu, W. Li, J. Li, Q. Du, and A. Plaza, “A3CLNN: Spatial, spectral and multiscale attention ConvLSTM neural network for multisource remote sensing data classification,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 2, pp. 747–761, 2022.DOI
  31. [31]W. Li, Y. Gao, M. Zhang, R. Tao, and Q. Du, “Asymmetric feature fusion network for hyperspectral and SAR image classification,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 10, pp. 8057–8070, 2023.DOI
  32. [32]X. Tang, Y. Zou, J. Ma, X. Zhang, F. Liu, and L. Jiao, “Multiple information collaborative fusion network for joint classification of hyperspectral and LiDAR data,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–16, 2024.DOI
  33. [33]Y. Gao, W. Li, M. Zhang, J. Wang, W. Sun, R. Tao, and Q. Du, “Hyperspectral and multispectral classification for coastal wetland using depthwise feature interaction network,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2022.DOI
  34. [34]X. Zhao, R. Tao, W. Li, W. Philips, and W. Liao, “Fractional gabor convolutional network for multisource remote sensing data classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–18, 2022.DOI
  35. [35]M. Wang, F. Gao, J. Dong, H.-C. Li, and Q. Du, “Nearest neighbor-based contrastive learning for hyperspectral and LiDAR data classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–16, 2023.DOI
  36. [36]Y. Gao, M. Zhang, J. Wang, and W. Li, “Cross-scale mixing attention for multisource remote sensing data fusion and classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–15, 2023.DOI
  37. [37]S. K. Roy, A. Deria, D. Hong, B. Rasti, A. Plaza, and J. Chansussot, “Multimodal fusion transformer for remote sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–20, 2023.DOI
  38. [38]G. Zhao, Q. Ye, L. Sun, Z. Wu, C. Pan, and B. Jeon, “Joint classification of hyperspectral and LiDAR data using a hierarchical CNN and transformer,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–16, 2023.DOI
  39. [39]K. Li, D. Wang, X. Wang, G. Liu, Z. Wu, and Q. Wang, “Mixing self-attention and convolution: A unified framework for multi-source remote sensing data classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–16, 2023.
  40. [40]H. Chang, H. Bi, F. Li, C. Xu, J. Chansussot, and D. Hong, “Deep symmetric fusion transformer for multimodal remote sensing data classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–15, 2024.
  41. [41]Y. Zhang, Y. Peng, B. Tu, and Y. Liu, “Local information interaction transformer for hyperspectral and LiDAR data classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 16, pp. 1130–1143, 2023.
  42. [42]L. Sun, X. Wang, Y. Zheng, Z. Wu, and L. Fu, “Multiscale 3-d–2-d mixed CNN and lightweight attention-free transformer for hyperspectral and LiDAR classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–16, 2024.DOI
  43. [43]K. Ni, D. Wang, Z. Zheng, and P. Wang, “MHST: Multiscale head selection transformer for hyperspectral and LiDAR classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 17, pp. 5470–5483, 2024.DOI
  44. [44]W.-S. Hu, W. Li, H.-C. Li, F.-H. Huang, and R. Tao, “Global clue-guided cross-memory quaternion transformer network for multisource remote sensing data classification,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 4, pp. 7357–7371, 2025.DOI
  45. [45]K. Ni, Y. Xi, G. Zhao, Z. Zheng, P. Wang, and T. Lu, “Coarse-to-fine high-order network for hyperspectral and LiDAR data classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 63, pp. 1–16, 2025.DOI
  46. [46]W. Zhang, W. Song, J. Wang, and W. Gao, “MGMNet: Mutual-guidance mechanism for joint classification of multisource remote sensing data,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 18, pp. 1085–1097, 2025.DOI
  47. [47]Y. Han, H. Zhu, L. Jiao, X. Yi, X. Li, B. Hou, W. Ma, and S. Wang, “SSMU-Net: A style separation and mode unification network for multimodal remote sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–15, 2023.DOI
  48. [48]K. Ni, D. Wang, G. Zhao, Z. Zheng, and P. Wang, “Hyperspectral and LiDAR classification via frequency domain-based network,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–17, 2024.DOI
  49. [49]W. Song, Y. Dai, Z. Gao, L. Fang, and Y. Zhang, “Hashing-based deep metric learning for the classification of hyperspectral and LiDAR data,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–13, 2023.
  50. [50]F. Gao, X. Jin, X. Zhou, J. Dong, and Q. Du, “MSFMamba: Multiscale feature fusion state space model for multisource remote sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 63, pp. 1–16, 2025.arxiv.org/abs/2408.14255
  51. [51]X. Ding, X. Zhang, J. Han, and G. Ding, “Scaling up your kernels to 31x31: Revisiting large kernel design in CNNs,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11 963–11 975.arxiv.org/abs/2203.06717
  52. [52]Y. Li, X. Li, Y. Dai, Q. Hou, L. Liu, Y. Liu, M.-M. Cheng, and J. Yang, “LSKNet: A foundation lightweight backbone for remote sensing,” International Journal of Computer Vision, vol. 133, no. 3, pp. 1410–1431, 2025.arxiv.org/abs/2403.11735
  53. [53]Y. Chen, X. Dai, M. Liu, D. Chen, L. Yuan, and Z. Liu, “Dynamic convolution: Attention over convolution kernels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 030–11 039.arxiv.org/abs/1912.03458
  54. [54]B. Yang, G. Bender, Q. V. Le, and J. Ngiam, “CondConv: Conditionally parameterized convolutions for efficient inference,” in Advances in Neural Information Processing Systems, vol. 32, 2019.arxiv.org/abs/1904.04971
  55. [55]J. Arevalo, T. Solorio, M. Montes-y Gómez, and F. A. González, “Gated multimodal units for information fusion,” arXiv preprint arXiv:1702.01992, 2017.arxiv.org/abs/1702.01992
  56. [56]H. Gao, H. Feng, Y. Zhang, S. Fei, R. Shen, S. Xu, and B. Zhang, “Interactive enhanced network based on multihead self-attention and graph convolution for classification of hyperspectral and LiDAR data,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–16, 2024.DOI

Paper details

vlnc:W_gdzg88h5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents