WorksComputer Science
SpecUltra: A Unified Foundation Model for Heterogeneous Spectral Images
Abstract
Spectral imagery, including hyperspectral image (HSI) and multispectral image (MSI), exhibits substantial heterogeneity in band numbers, wavelengths, spectral resolutions, hindering unified modeling and cross-sensor transfer. From the perspective of imaging physics, these heterogeneous observations can be regarded as discrete measurements of an underlying continuous spectrum acquired through different sampling processes. Motivated by this perspective, we propose SpecUltra, a unified foundation model for heterogeneous spectral images. Specifically, SpecUltra employs 3D convolutional tokenization and wavelength-aware 3D RoPE to encode relative wavelength and spatial relationships, enabling fine-grained spatial-spectral modeling within a shared coordinate system. We further construct MultiHyper-200K, a large-scale multi-sensor HSI dataset, and pretrain SpecUltra using auxiliary-token-assisted mask image modeling (MIM), where a level token indicates the radiometric product type and register tokens absorb sensor- and acquisition-dependent variations. To bridge the spectral-granularity gap between HSI and MSI, we introduce Spectral Kernel Transformation (SKT), which transfers pretrained knowledge in parameter space by constructing MSI embedding kernels from fine-grained HSI-pretrained kernel slices. This design preserves pretrained spectral priors while maintaining compatibility with the pretrained backbone. Experiments on HSI and MSI classification and segmentation benchmarks demonstrate state-of-the-art or competitive performance across disparate sensors and downstream tasks.

Figure 1. SpecUltra unifies heterogeneous HSI and MSI in a shared representation space and is subsequently finetuned for downstream tasks.
Code — https://github.com/kkweil/SpecUltra
Introduction
Spectral imagery jointly captures the spatial structures of ground objects and their wavelength-dependent reflectance characteristics, providing rich information for material identification, target recognition, and land-cover classification [8, 2]. Unlike RGB images with fixed channel definitions, spectral images exhibit substantial heterogeneity in band configurations. HSI densely samples the spectrum using tens to hundreds of contiguous narrow bands, whereas MSI records integrated responses over only a few discrete and relatively broad spectral intervals [1]. Moreover, even within the same modality, different sensor designs can lead to substantial variations in band number, central wavelength, bandwidth, and spectral response functions. This heterogeneity poses substantial challenges to building a unified spectral foundation model for spectral images.
A natural strategy is to embed each spectral channel independently, thereby preserving fine-grained spectral information. However, for HSI with hundreds of densely sampled bands, this approach results in prohibitively long token sequences and incurs substantial computational overhead. Spectral group embedding [11] provides a more efficient alternative by jointly encoding neighboring bands and exploiting the local spectral patterns of HSI. However, this design is less suitable for MSI, whose bands are sparse and relatively broad. Grouping adjacent MSI bands may prematurely mix distinct spectral semantics. Consequently, existing spectral foundation models typically specialize in either HSI or MSI [12], with their input embeddings and positional representations tied to specific band configurations [13]. Although sensor-agnostic methods such as SMARTIES [24] and CARL [4] employ wavelength-aware projections or adaptive spectral aggregation, they often compress spectral information early or rely on sensor-dependent mappings, limiting fine-grained spatial-spectral modeling.
From the perspective of spectral imaging physics, HSI and MSI share a common observation process. Both can be regarded as discrete measurements of an underlying continuous surface spectrum, differing primarily in their spectral sampling densities and response functions (Yokoya, Grohn- feldt, and Chanussot 2017). This shared physical origin provides a principled basis for unified modeling and suggests that knowledge learned from densely sampled HSI can be transferred to coarsely sampled MSI by adapting pretrained representations and parameters to different spectral granularities. Accordingly, two fundamental questions arise: 1) How can heterogeneous spectral observations be modeled with fine-grained spatial-spectral structures and consistent physical semantics? and 2) How can pretrained knowledge be effectively transferred across substantially different spectral granularities?
To this end, we propose SpecUltra, a unified foundation model for heterogeneous spectral remote sensing imagery. SpecUltra learns fine-grained spatial-spectral representations from multi-sensor HSI data and transfers them to diverse MSI configurations through parameter-space adaptation. Specifically, a 3D convolutional embedding layer partitions each input into spatial-spectral tokens. Wavelength-aware 3D RoPE then encodes relative physical relationships along the wavelength and spatial dimensions, enabling heterogeneous observations to be modeled in a unified coordinate system. SpecUltra is pretrained on MultiHyper-200K, which contains approximately 200,000 HSI samples from six sensors, using auxiliary-token-assisted masked image modeling. A level token indicates the radiometric product type, while register tokens capture sensor- and acquisition-dependent variations. To bridge the spectral-granularity gap between HSI and MSI, we introduce SKT which reparameterizes each MSI embedding kernel as a learnable combination of fine-grained HSI-pretrained kernel slices, thereby transferring spectral priors in parameter space while adapting them to broad-band MSI observations. Experiments on four HSI and MSI benchmarks demonstrate leading performance on three benchmarks and competitive results on EuroSAT. Ablation studies further validate the effectiveness of each component. The main contributions are summarized as follows:
We formulate heterogeneous spectral foundation modeling from a shared continuous-spectrum perspective, providing a unified view of diverse band configurations.
We introduce wavelength-aware 3D RoPE to encode physical wavelength and spatial coordinates, enabling fine-grained modeling within a unified representation space.
We propose SKT to transfer pretrained knowledge across spectral granularities in parameter space by constructing MSI embedding kernels from learnable combinations of HSI-pretrained spectral kernel slices.
We construct MultiHyper-200K, a large-scale multi-sensor HSI pretraining dataset, and develop auxiliary-token-assisted MIM to improve cross-sensor representation learning.
Related work
Spectral Foundation Models. Spectral foundation models have demonstrated strong performance in remote sensing interpretation [15]. Existing methods mainly model hyperspectral and multispectral imagery separately. For example, SatMAE [6] and SpectralGPT [14] focus on multispectral imagery, whereas SpectralEarth [5], HyperSIGMA [26], HyperFree (Li et al. 2025a), HyperSL [18], and SpecAware (Ji et al. 2026) are designed for hyperspectral imagery. More recently, sensor-agnostic methods such as SMAR-TIES and CARL have attempted to unify heterogeneous spectral observations. However, they typically compress spectral information into fixed-dimensional representations at an early stage, which may discard fine-grained spectral structures and hinder parameter transfer across different band configurations.
Masked Image Modeling. MIM is a generative self-supervised learning paradigm that masks a large portion of input patches and reconstructs the missing pixels or features from the visible context, thereby learning transferable representations without manual annotations (He et al. 2022; Bao et al. 2022). In remote sensing, MIM has been adapted to domain-specific spatial, spectral, temporal, and scale characteristics. Representative methods such as RingMo [25], ScaleMAE [21] and Selective-Mae [27]. These methods have achieved strong transfer performance across classification and segmentation, establishing MIM as a fundamental pretraining paradigm for remote sensing representation learning.
Principle of Spectral Imaging. Spectral imaging measures an underlying continuous spectrum through a set of discrete spectral intervals [28]. Let denote the continuous spectral signal at spatial position . The observation of the -th band acquired by sensor is formulated as
where and denote the spectral sampling interval and response function, respectively. Therefore, spectral images acquired by different sensors can be regarded as discrete observations of the same continuous spectrum under different spectral sampling configurations. Their spectral heterogeneity mainly arises from variations in the number, location, and width of the sampling intervals.
Rotary Positional Encoding. RoPE (Su et al. 2024) encodes relative positions through coordinate-dependent rotations of query and key features. It has been extended from 1D sequences to 2D visual coordinates and adopted in vision foundation models, such as EVA-02[7] and DINOv3 [22]. Spectral images naturally involve spatial dimensions and one spectral dimension , motivating a 3D RoPE over . By using physical wavelength instead of discrete channel indices, heterogeneous HSI and MSI bands can be represented within a consistent spatial-spectral coordinate system.
Method
Overview
As illustrated in Figure 2, SpecUltra models heterogeneous spectral images within a unified spatial-spectral representation space. It tokenizes each input with 3D convolutional and encodes relative wavelength and spatial coordinates using wavelength-aware 3D RoPE. SpecUltra is pretrained on MultiHyper-200K with auxiliary-token-assisted MIM to accommodate radiometric, sensor-dependent variations. For MSI adaptation, SKT transfers HSI-pretrained embedding parameters across spectral granularities by aggregating fine-grained spectral kernel slices into MSI kernels.

Figure 2. Overview of SpecUltra. Heterogeneous spectral images are tokenized along the spatial and spectral dimensions and encoded within a unified spatial-spectral representation space using wavelength-aware 3D RoPE. SpecUltra is pretrained on MultiHyper-200K with auxiliary-token-assisted MIM to accommodate radiometric, sensor, and acquisition variations. For MSI adaptation, SKT constructs MSI-compatible embedding kernels from learnable combinations of HSI-pretrained spectral kernel slices, enabling parameter-space transfer across spectral granularities.
Unified Spatial-Spectral Embedding
Given an input HSI with a variable number of spectral bands , we first employ a learnable 3D convolutional kernel to construct fine-grained spatial-spectral tokens, where is the embedding dimension, is the spectral patch size, and is the spatial patch size. The operation is formulated as:
where the kernel size and stride are both set to . Here, , , and denote the spectral depth, height, and width of the embedded feature , respectively. The resulting feature volume is then flattened along the spectral and spatial dimensions to obtain the spatial-spectral tokens :
where is the number of output tokens and is the embedding dimension.
Then, we assign each token a spatial-spectral coordinate based on its wavelength and spatial location. Each token is associated with a normalized coordinate , and each can be computed as follows:
where and denote the mean wavelength and spatial token-grid index of token , respectively. The constant 2500 nm is used as a common upper wavelength bound for typical HSIs.
Based on the normalized coordinates, we parameterize 3D RoPE to model spatial-spectral tokens within a unified coordinate system through self-attention. For multi-head attention with heads, each head has dimension . Since the rotary dimensions are equally divided among the spectral, height, and width axes, with each rotation operating on a channel pair, we require . The rotary periods are geometrically spaced as:
where is the number of rotary channel pairs assigned to each axis, and is set to 10000 following the original RoPE. The rotary angles are computed given the normalized coordinate as follows:
where denotes the rotary period at frequency index . Subsequently, the rotary angles are concatenated and duplicated to construct the rotation vector for paired-channel rotation:
For convenience, we define a rotation operator :
where . For each attention head, let denote the query and key vectors of token . The rotated query and key are computed as follows:
where denotes element-wise multiplication. From a geometric perspective, this operation induces an independent planar rotation within each paired-channel subspace. Specifically, for the -th channel pair , with the corresponding angle , the transformation can be written as:
Accordingly, the proposed 3D RoPE performs rotary modulation by assigning separate channel groups to the wavelength, height, and width coordinates, enabling self-attention to model relative spatial-spectral geometry and align tokens from heterogeneous sensors within a unified coordinate system.
Spectral Kernel Transformation
The large spectral-granularity gap between HSI and MSI results in incompatible embedding parameterizations. Directly replacing the HSI embedding layer discards pretrained spectral priors and disrupts its compatibility with the backbone. We therefore formulate MSI adaptation as a parameter-space transformation, constraining the MSI embedding kernel to the space induced by HSI-pretrained spectral kernel slices.
Given the HSI embedding kernel , its -th spectral slice for output channel is defined as
The shared MSI embedding kernel is parameterized as
where contains learnable parameters and is the normalized transformation matrix. This transformation transfers the spectral aggregation principle from observation space to parameter space. Since and , each MSI kernel lies in the convex hull of the corresponding HSI-pretrained slices. SKT thus preserves pretrained spectral priors while learning an MSI-compatible embedding parameterization.
For an MSI input , is applied independently to each band through convolution operation, producing:
The resulting feature volume retains explicit spectral and spatial axes, allowing each token to be indexed by its wavelength and spatial coordinates. The transformed MSI features therefore remain compatible with the pretrained backbone and can be directly processed using wavelength-aware 3D RoPE.
Auxiliary-Tokens-Assisted MIM
Multi-source HSI data share transferable spatial-spectral structures while exhibiting substantial variations in sensors, acquisition conditions, and radiometric products. To improve representation learning under such heterogeneity, we introduce two types of auxiliary tokens during masked image modeling: register tokens and a radiometric-level token.
The register tokens provide additional latent capacity for capturing sensor- and acquisition-dependent variations, thereby encouraging patch tokens to focus on intrinsic and transferable spatial-spectral patterns. The level token specifies the radiometric form of the input and is selected from a learnable embedding table:
where denotes radiance or reflectance data.
Given the visible patch tokens , the encoder input is constructed as
The decoder then reconstructs the masked patches using learnable mask tokens:
Both auxiliary tokens are used only to condition the encoder representations and are excluded from the reconstruction decoder and downstream task heads.
Following Ji et al. (2026), we optimize the masked prediction task using a hybrid objective that jointly accounts for radiometric fidelity and spectral consistency. Specifically, the Charbonnier loss mitigates the influence of noise and large reconstruction errors, while the spectral angle mapper (SAM) loss constrains the angular discrepancy between the reconstructed and target spectra, thereby preserving spectral structures under magnitude variations. The final pretraining objective is defined as
Where, denotes the set of masked patches, and and are the corresponding target and reconstructed spectral vectors, and are set to 1 by default, ensures numerical stability.
Experiments
Dataset Description
MultiHyper-200K:
To facilitate large-scale multi-sensor pretraining of SpecUltra, we construct a HSI dataset from six sensors: AVIRIS, AVIRIS-3, DESIS, EnMAP, GF-5, and HyspIRI. The collected imagery covers diverse geographic regions worldwide. After manual quality inspection, the remaining images were cropped into approximately 200K nonoverlapping samples for pretraining. The details of MultiHyper-200K are summarized in Table 1.
| Sensors | Spec. Range (nm) | Spec. Res (nm) | Spat. Res (m) | Bands (#) | Data Level | Numbers (#) |
| AVIRIS | 380-2500 | 10 | 0.5-20 | 224 | L1/L2 | 94,207 |
| AVIRIS-3 | 390-2500 | 7.4 | 0.3-10 | 224 | L2 | 29,866 |
| HyspIRI | 380-2500 | 10 | 0.3-17 | 224 | L2 | 12,012 |
| EnMAP | 420-2450 | 6.5/10 | 30 | 246 | L2 | 23,207 |
| DESIS | 402-1000 | 2.55 | 30 | 235 | L2 | 27,517 |
| GF-5 | 400-2500 | 5/10 | 30 | 330 | L1 | 13,689 |
| Total | 200,498 |
Table 1. Main characteristics of the pretraining datasets.
Downstream datasets: To evaluate SpecUltra across heterogeneous sensors, spectral modalities, and downstream tasks, we adopt four representative HSI and MSI benchmarks covering airborne and satellite platforms, semantic segmentation, and scene classification. For semantic segmentation, Qingpu Ji et al. (2025) is an AMMIS HSI of size with 251 bands and 20 classes. Following Ji et al. (2025), spatially disjoint regions are used for training and testing to avoid spatial leakage. SegMunich Hong et al. [14] is a Sentinel-2 MSI of size with 10 bands and 13 classes, evaluated using the SpectralGPT protocol. These datasets assess pixel-level transfer across airborne HSI and satellite MSI. For scene classification, HRSSC Li et al. (2025b) contains 1,445 AVIRIS-C HSI patches of size , with 224 bands and 11 classes. We use 10% of the samples for training and the remainder for testing. EuroSAT Helber et al. [10] contains 27,000 Sentinel-2 MSI patches of size , with 13 bands and 10 classes. We use 100 samples per class for training and the rest for testing. These datasets evaluate image-level generalization across spectral granularities under limited supervision.
Implementation details
SpecUltra was pretrained with Auxiliary-Token-Assisted MIM using a masking ratio of 0.75 following the common practice in MAE He et al. (2022). The embedding kernel and stride were set to along the spectral and spatial dimensions. Reflectance products retained their original physical scale, while radiance products were empirically scaled during preprocessing. A learnable level token indicated the radiometric product type, and five register tokens were introduced to capture sensor-specific variations. We trained SpecUltra-S and SpecUltra-B for 200 epochs using AdamW with cosine learning-rate decay, an initial learning rate of , an effective batch size of 256, a weight decay of 0.05, and , . Mixed-precision training and random horizontal and vertical flips were adopted. The model configurations are summarized in Table 2.
| Model | Dim. | Depth | Heads | MLP |
| SpecUltra-S | 384/192 | 12/8 | 4/4 | 4.0 |
| SpecUltra-B | 576/192 | 12/8 | 8/8 | 4.0 |
Table 2. Configurations of SpecUltra variants (Enc./Dec.).
During fine-tuning, all models were trained for 100 epochs with a backbone learning rate of . The spatial patch size was set to 16 for classification and 8 for segmentation,
where the pretrained embedding kernels were spatially down-sampled to match the smaller patch size. The batch sizes were set to 16, 16, 32, and 64 for Qingpu, HRSSC, SegMunich, and EuroSAT, respectively. The same fine-tuning protocol was applied to all comparison methods to ensure a fair evaluation.
Main Results
Main Comparison. We evaluate SpecUltra on two classification benchmarks, EuroSAT and HRSSC, and two segmentation benchmarks, SegMunich and Qingpu, covering both MSI and HSI. We compare SpecUltra with SMARTIES, CARL, and six representative remote-sensing pretraining methods: SatMAE, SpectralGPT, SpectralEarth, HyperSIGMA, HyperFree, and SpecAware. For methods incompatible with heterogeneous inputs, a prefix adapter is used for channel alignment. We report OA, F1, and mIoU for segmentation, and OA, F1, and Kappa for classification. Results are averaged over five runs with different random seeds.
Finetuning. As summarized in Table 3, SpecUltra achieves strong performance across HSI and MSI segmentation and classification benchmarks. Under the spatially disjoint protocol on Qingpu, it obtains 94.69% OA, 56.87% F1, and 47.04% mIoU, surpassing the strongest baseline by 1.18, 0.94, and 2.31 points, respectively, indicating robust spatial transfer without relying on neighboring-region leakage. On SegMunich, it further improves the best competing results by 0.45, 0.67, and 0.69 points, demonstrating effective adaptation from HSI pretraining to sparse MSI bands. For classification, SpecUltra achieves the best OA and Kappa on HRSSC, outperforming HyperSigma by 2.71 and 3.08 points, while obtaining the second-best F1 with a gap of only 0.49 points. Given the limited training split, these results reflect strong sample-efficient transfer. On EuroSAT, it remains competitive with MSI-specific models and consistently outperforms all HSI-pretrained and sensor-agnostic methods, showing that SKT and wavelength-aware modeling alleviate the HSI-MSI spectral-granularity gap. The qualitative results in Figure 3 further show more coherent regions, clearer boundaries, and fewer fragmented predictions. Overall, SpecUltra generalizes effectively across heterogeneous sensors, spectral granularities, and downstream tasks.
| Method | Pretraining Data | Segmentation (OA/F1/mIoU) Qingpu (HSI) | Segmentation (OA/F1/mIoU) SegMunich (MSI) | Classification (OA/F1/Kappa) HRSSC (HSI) | Classification (OA/F1/Kappa) EuroSAT (MSI) |
| SatMAE | fMoW-S2 | 91.57/46.81/36.93 | 82.29/63.40/50.21 | 79.01/66.53/75.61 | 94.02/93.80/93.34 |
| SpectralGPT | BigEarthNet | 91.62/47.32/37.65 | 81.63/61.27/49.21 | 82.67/70.26/79.27 | 94.69/94.49/94.48 |
| SpectralEarth | SpectralEarth | 92.23/49.74/40.26 | 82.03/62.98/49.96 | 82.41/73.79/79.58 | 92.32/83.70/91.46 |
| HyperSigma | HyperGlobal-450K | 91.96/46.99/38.47 | 80.37/58.20/46.81 | 83.02/74.63/80.34 | 91.54/82.89/90.59 |
| HyperFree | HyperSeg | 90.85/44.27/35.62 | 81.18/60.56/47.66 | 80.48/69.50/77.47 | 91.19/82.52/90.20 |
| SpecAware | Hyper-400K | 93.38/50.38/41.18 | 81.50/61.72/48.70 | 82.64/69.55/79.78 | 90.13/89.70/89.02 |
| SMARTIES | fMoW-S2 | 93.51/55.93/44.73 | 81.33/61.32/48.26 | 81.86/69.91/78.89 | 91.27/90.92/90.29 |
| CARL | HySpecNet | 92.20/49.28/39.50 | 80.05/58.80/45.95 | 79.86/69.81/76.77 | 90.28/89.93/89.19 |
| SpecUltra-B | MultiHyper-200K | 94.69/56.87/47.04 | 82.74/64.07/50.90 | 85.73/74.14/83.42 | 93.74/93.45/93.03 |
Table 3. Comparison with SOTA methods on HSI and MSI downstream benchmarks (best/second).(%).
Figure 3. Qualitative comparison of segmentation results on Qingpu and SegMunich datasets.
Backbone Probing. To fairly evaluate representation transfer across heterogeneous spectral configurations, we freeze the pretrained backbones and optimize only the adaptation modules and linear classifiers for all methods. As shown in Table 4, the proposed method achieves the best performance on HRSSC, surpassing the strongest baselines by 0.33 OA, 1.31 F1, and 1.85 Kappa points. On EuroSAT, it trails the MSI-specific methods by at most 0.98 points, while outperforming all HSI-pretrained methods across the three metrics. Compared with sensor-agnostic methods, it achieves the highest OA and Kappa while remaining competitive in F1. These results demonstrate the strong transferability of SpecUltra across heterogeneous HSI and MSI configurations
| Method | HRSSC OA | HRSSC F1 | HRSSC Kappa | EuroSAT OA | EuroSAT F1 | EuroSAT Kappa |
| SatMAE | 74.92 | 57.91 | 70.56 | 83.52 | 82.34 | 81.66 |
| SpectralGPT | 76.38 | 58.46 | 70.99 | 83.58 | 82.27 | 81.73 |
| SpectralEarth | 77.47 | 68.41 | 73.82 | 82.57 | 74.10 | 80.59 |
| HyperSigma | 80.17 | 69.63 | 76.99 | 81.66 | 73.47 | 79.59 |
| HyperFree | 80.63 | 68.32 | 77.39 | 77.45 | 69.27 | 74.92 |
| SpecAware | 81.85 | 68.45 | 78.88 | 80.50 | 80.00 | 78.30 |
| SMARTIES | 78.94 | 65.60 | 75.45 | 82.87 | 82.35 | 80.93 |
| CARL | 76.08 | 63.03 | 72.12 | 82.59 | 81.60 | 80.60 |
| SpecUltra-B | 82.18 | 70.94 | 80.73 | 82.93 | 81.37 | 80.94 |
Table 4. Backbone probing results on classification benchmarks.
Model scalability. As shown in Figure 4, scaling SpecUltra from small to base improves mIoU by 1.48 and 0.15 points on Qingpu and SegMunich, respectively, and increases accuracy by 0.32 points on EuroSAT. On HRSSC, the Base variant performs slightly worse by 0.23 points, likely because its limited training set restricts the effective optimization of the larger model. Overall, increased model capacity generally improves downstream performance, particularly for dense prediction tasks with sufficient training data.

Figure 4. Comparison of SpecUltra-S and SpecUltra-B.
Computational Efficiency. As shown in Table 5, SpecUltra uses fewer parameters than most competing models. Its
| Metric | SatMAE | SpectralGPT | SpectralEarth | HyperSigma | HyperFree | SpecAware | SMARTIES | CARL | SpecUltra-S | SpecUltra-B |
| Params | 86.09 M | 85.41 M | 236.19 M | 236.19 M | 151.14 M | 89.42 M | 88.62 M | 71.93 M | 24.69 M | 55.31 M |
| FLOPs | 523.70 G | 212.97 G | 229.44 G | 229.44 G | 57.96 G | 45.16 G | 3313.01 G | 63.02 G | 324.65 G | 727.59 G |
caption: Comparison of model complexity.
:::
higher FLOPs arise from jointly tokenizing the spectral and spatial dimensions, which produces more tokens for fine-grained spatial-spectral modeling. Thus, the additional computation mainly reflects denser token interactions.
Ablation Studies
We conduct controlled ablation studies to evaluate the individual contributions of wavelength-aware 3D RoPE, SKT, and auxiliary-token-assisted pretraining. To ensure a fair comparison while reducing computational cost, all variants adopt the tiny model and are pretrained for 30 epochs under identical settings. Performance is evaluated using mIoU for segmentation tasks and OA for scene classification tasks, with all ablation results averaged over five runs.
Effect of Wavelength-Aware 3D RoPE. Table 6 compares different positional encodings. Replacing band indices with wavelengths using Wave PE & 2D RoPE improves Index PE by 0.89, 2.14, 1.22, and 2.31 points on the four datasets, respectively. Joint wavelength–spatial encoding further gains 1.59, 0.13, 0.86, and 0.76 points, achieving the best overall performance and demonstrating the transferability of unified 3D spatial–spectral coordinates.
| Position Encoding | Qingpu | SegMunich | HRSSC | EuroSAT |
| Index PE | ||||
| Wave PE & 2D RoPE | ||||
| Wave 3D RoPE |
Table 6. Effect of wavelength-aware 3D RoPE(mean±std%).
Effect of SKT. Table 7 compares different MSI embedding strategies. Prefix Adapter provides limited gains and slightly degrades SegMunich performance, whereas Mean Aggregation improves over random initialization by 0.22 mIoU and 1.75 OA, confirming the benefit of reusing HSI-pretrained kernels. SKT achieves the best results, further surpassing Mean Aggregation by 0.40 mIoU on SegMunich and 1.68 OA on EuroSAT. These gains show that learnable kernel transformation better bridges the spectral-granularity gap between HSI and MSI.
| MSI Embedding | SegMunich | EuroSAT |
| Random Initialization | ||
| Prefix Adapter | ||
| Mean Aggregation | ||
| Random Aggregation | ||
| SKT |
Table 7. Effect of SKT on MSI transfer on SpecUltra-B.
Effect of Register Tokens. Table 8 shows that both level and register tokens consistently improve performance. Their combination achieves the best results, surpassing the token-free baseline by 1.27, 2.06, 1.17, and 2.00 points on Qingpu, SegMunich, HRSSC, and EuroSAT, respectively. This suggests that level tokens provide radiometric-level cues, while register tokens capture sensor- and acquisition-dependent variations. As shown in Figure 5, performance generally saturates at five register tokens; increasing the number to seven yields only marginal gains on Qingpu and EuroSAT while slightly degrading performance on SegMunich and HRSSC. Therefore, we use five register tokens by default.
Figure 5. Effect of the number of register tokens.
| Token Configuration | Qingpu | SegMunich | HRSSC | EuroSAT |
| None | ||||
| Level Token | ||||
| Register Tokens | ||||
| Level + Register Tokens |
Table 8. Ablation results of register and level tokens.
Conclusion
This study develops SpecUltra, a unified foundation model for learning transferable representations from heterogeneous spectral imagery. SpecUltra employs wavelength-aware 3D RoPE to preserve fine-grained spatial-spectral structure across diverse band configurations and is pretrained on MultiHyper-200K with auxiliary-token-assisted MIM to improve cross-sensor representation learning. To bridge the spectral-granularity gap between HSI and MSI, SKT performs parameter-space adaptation by constructing MSI embedding kernels from HSI-pretrained spectral kernel slices. Experiments on classification and segmentation benchmarks demonstrate strong performance across multiple sensors and spectral modalities. These results suggest that SpecUltra provides a practical foundation for unified spectral remote sensing. Future work will explore broader sensor coverage, additional downstream tasks, and more efficient spatial-spectral modeling.
References
- [1]Amigo, J. M. 2019. Hyperspectral and multispectral imaging: setting the scene. In Data handling in science and technology, volume 32, 3–16. Elsevier.DOI
- [2]Ayala, L.; Adler, T. J.; Seidlitz, S.; Wirkert, S.; Engels, C.; Seitel, A.; Sellner, J.; Aksenov, A.; Bodenbach, M.; Bader, P.; et al. 2023. Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery. Science advances, 9(10): eadd6778.
- [4]Baumann, A.; Ayala, L.; Seidlitz, S.; Sellner, J.; Studier-Fischer, A.; Özdemir, B.; Maier-hein, L.; and Ilic, S. 2026. CARL: Camera-Augmented Representation Learning for Spectral Image Analysis. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2504.19223
- [5]Braham, N. A. A.; Albrecht, C. M.; Mairal, J.; Chanussot, J.; Wang, Y.; and Zhu, X. X. 2025. Spectraleath: Training hyperspectral foundation models at scale. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.DOI
- [6]Cong, Y.; Khanna, S.; Meng, C.; Liu, P.; Rozi, E.; He, Y.; Burke, M.; Lobell, D. B.; and Ermon, S. 2022. Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. In Advances in neural information processing systems.arxiv.org/abs/2207.08051
- [7]Fang, Y.; Sun, Q.; Wang, X.; Huang, T.; Wang, X.; and Cao, Y. 2024. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149: 105171.DOI
- [8]Ghamisi, P.; Yokoya, N.; Li, J.; Liao, W.; Liu, S.; Plaza, J.; Rasti, B.; and Plaza, A. 2017. Advances in hyperspectral image and signal processing: A comprehensive overview of the state of the art. IEEE Geoscience and Remote Sensing Magazine (GRSM), 5(4): 37–78.DOI
- [10]Helber, P.; Bischke, B.; Dengel, A.; and Borth, D. 2019. EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7): 2217–2226.
- [11]Hong, D.; Han, Z.; Yao, J.; Gao, L.; Zhang, B.; Plaza, A.; and Chanussot, J. 2022. SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers. IEEE Transactions on Geoscience and Remote Sensing, 60: 1–15.DOI
- [12]Hong, D.; Li, C.; Li, X.; Camps-Valls, G.; and Chanussot, J. 2026a. Foundation Models in Remote Sensing: Evolving from unimodality to multimodality. IEEE Geoscience and Remote Sensing Magazine.DOI
- [13]Hong, D.; Li, C.; Yokoya, N.; Zhang, B.; Jia, X.; Plaza, A.; Gamba, P.; Benediktsson, J. A.; and Chanussot, J. 2026b. Hyperspectral imaging. Nature Reviews Methods Primers, 6(1): 19.arxiv.org/abs/2508.08107
- [14]Hong, D.; Zhang, B.; Li, X.; Li, Y.; Li, C.; Yao, J.; Yokoya, N.; Li, H.; Ghamisi, P.; Jia, X.; et al. 2024. SpectralGPT: Spectral remote sensing foundation model. IEEE transactions on pattern analysis and machine intelligence, 46(8): 5227–5244.DOI
- [15]Huang, Z.; Yan, H.; Zhan, Q.; Yang, S.; Zhang, M.; Zhang, C.; Lei, Y.; Liu, Z.; Liu, Q.; and Wang, Y. 2025. A survey on remote sensing foundation models: From vision to multimodality. arXiv preprint arXiv:2503.22081.arxiv.org/abs/2503.22081
- [18]Kong, W.; Liu, B.; Bi, X.; Yu, C.; Li, X.; and Chen, Y. 2025. HyperSL: A spectral foundation model for hyperspectral image interpretation. IEEE Transactions on Geoscience and Remote Sensing.DOI
- [21]Reed, C. J.; Gupta, R.; Li, S.; Brockman, S.; Funk, C.; Clipp, B.; Keutzer, K.; Candido, S.; Uyttendaele, M.; and Darrell, T. 2023. Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4088–4099.arxiv.org/abs/2212.14532
- [22]Siméoni, O.; Vo, H. V.; Seitzer, M.; Baldassarre, F.; Oquab, M.; Jose, C.; Khalidov, V.; Szafraniec, M.; Yi, S.; Ramamonjisoa, M.; et al. 2025. Dinov3. arXiv preprint arXiv:2508.10104.arxiv.org/abs/2508.10104
- [24]Sumbul, G.; Xu, C.; Dalsasso, E.; and Tuia, D. 2025. SMAR-TIES: Spectrum-aware multi-sensor auto-encoder for remote sensing images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 5569–5578.
- [25]Sun, X.; Wang, P.; Lu, W.; Zhu, Z.; Lu, X.; He, Q.; Li, J.; Rong, X.; Yang, Z.; Chang, H.; et al. 2022. RingM: A remote sensing foundation model with masked image modeling. IEEE Transactions on Geoscience and Remote Sensing, 61: 1–22.DOI
- [26]Wang, D.; Hu, M.; Jin, Y.; Miao, Y.; Yang, J.; Xu, Y.; Qin, X.; Ma, J.; Sun, L.; Li, C.; et al. 2025a. HyperSIGMA: Hyper-spectral intelligence comprehension foundation model. IEEE Transactions on Pattern Analysis and Machine Intelligence.
- [27]Wang, F.; Wang, H.; Wang, D.; Guo, Z.; Zhong, Z.; Lan, L.; Yang, W.; and Zhang, J. 2025b. Harnessing Massive Satellite Imagery with Efficient Masked Image Modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 6935–6947.arxiv.org/abs/2406.11933
- [28]Wei, Q.; Bioucas-Dias, J.; Dobigeon, N.; and Tourneret, J.-Y. 2015. Hyperspectral and Multispectral Image Fusion Based on a Sparse Representation. IEEE Transactions on Geoscience and Remote Sensing, 53(7): 3658–3668.DOI

