betaFeedback

WorksEngineering

TOP: Transmission-Oriented Remote Sensing Pansharpening

Abstract

Pansharpening has been extensively studied as a fundamental image fusion task in remote sensing. However, most existing methods focus on improving fusion quality under the implicit assumption that high-resolution observations have already been fully downlinked to the ground. As a result, existing models are highly vulnerable to perturbations introduced by lossy compression used for downlink transmission. To address this limitation, we present, to the best of our knowledge, the first transmission oriented pansharpening (TOP) framework, which unifies compact latent coding, onboard pansharpening, and efficient downlink transmission within a single satellite-toground pipeline. TOP encodes remote-sensing observations into a compact binary latent space, where a lightweight onboard fusion module can efficiently predict the fused binary latent code. Since only the compact fused code is downlinked, the transmission payload is substantially reduced. Overall, TOP enables low-payload downlink and lightweight onboard processing while maintaining high fusion quality under severe bandwidth constraints. Experiments across multispectral and hyperspectral settings show that TOP consistently achieves superior fusion quality at higher compression ratios than representative pansharpening methods paired with standard lossy codecs. For example, on GaoFen-2, TOP attains a fusion PSNR of 31.4 dB at a downlink compression ratio of 50x. This design also points toward transmissionoriented processing for other remote-sensing tasks. The code will be made available at https://github.com/JingZe-Liang/TOP.git.

Introduction

Pansharpening reconstructs a high-spatial-resolution multispectral or hyperspectral image (HR-MSI/HR-HSI) by combining the spatial structures of a panchromatic (PAN) observation with the spectral information of a low-spatial-resolution multispectral or hyperspectral image (LR-MSI/LR-HSI), and underpins remote-sensing image interpretation [1], [2]. From the explicit detail-injection rules of component-substitution and multi-resolution-analysis methods [3, 4, 5] to the nonlinear mappings learned by deep networks [6, 7, 8], these methods share an implicit assumption: the input observations have already been fully delivered to the ground station and are directly available to the fusion model.

In practice, the PAN and LR-MSI/LR-HSI observations are acquired on board and must pass through bandwidth-limited downlink channels [9]. A conventional deployment therefore compresses and transmits them independently and fuses the decompressed images on the ground (first row of Figure 1(a)). Under restrictive bitrate budgets, lossy compression introduces spatial blurring, spectral distortion, and quantization artifacts, whereas pansharpening models are commonly trained on clean observations [6, 7, 8]. This training–deployment mismatch can severely degrade the fusion result (Figure 1(b)), and the codec-based baselines in Sec. IV-C indeed degrade sharply under constrained downlink budgets. Recent work on degradation-robust fusion [10] further highlights the importance of realistic input perturbations.

Figure 1. TOP compared with existing deployment paths for pansharpening. (a) Three ways of delivering a pansharpened image to the ground, with compression ratios and PSNR ranges from WorldView-3 (WV3). (b) Ground truth, the best compress-then-fuse baseline under JPEG2000, and TOP on WV3, GaoFen-2 (GF2), and Pavia Center. (c) PSNR versus downlink compression ratio for methods under HEVC or JPEG2000; onboard fusion followed by compression is evaluated on WV3 only (Sec. S9 of the Supplementary Material).

Fusing on board (second row of Figure 1(a)) does not resolve this issue. The satellite must then generate a full HR-MSI/HR-HSI, which is computationally expensive on resource-constrained platforms, and the fused image must still be compressed with a standard codec such as JPEG2000 [11] or High Efficiency Video Coding (HEVC) [12], whose rate–distortion performance then bounds the final quality [9, 13].

These limitations motivate a different question: rather than deciding whether to compress images before or after pansharpening, can transmission and pansharpening, which have so far been handled separately, benefit from each other? To this end, we propose, to the best of our knowledge, the first transmission-oriented pansharpening (TOP) framework, which jointly designs compact latent coding, onboard pansharpening, and downlink transmission. TOP maps the PAN and LR-MSI/LR-HSI observations into binary latent codes and employs a lightweight latent fusion module to predict the fused HR-MSI/HR-HSI code on board (third row of Figure 1(a)). Only this code is downlinked; the ground station decodes it into the final HR-MSI/HR-HSI or archives it for later reconstruction. TOP thus shifts the communication target from source images to a compact fused code.

TOP is built on a transmission-oriented binary tokenizer (TOBT), which maps PAN, LR-MSI/LR-HSI, and HR-MSI/HR-HSI observations into a shared latent space through an adaptive embedding for different numbers of bands and a Swin-based encoder [14]. Binary spherical quantization (BSQ) [15] then maps each latent entry to one bit. Besides reducing the payload, this bottleneck turns continuous-latent regression into an easier per-entry binary classification for the latent fusion module (§ 5-B). The binary code is transmitted directly, without an entropy model [13] or range asymmetric numeral systems (rANS) coding [16] (§ 5-C), and bit-flip augmentation makes the ground-side decoder robust to latent prediction and transmission errors.

We compare TOP with representative pansharpening methods whose inputs are compressed by HEVC or JPEG2000 before ground-side fusion, on the WorldView-3 (WV3) and GaoFen-2 (GF2) multispectral datasets and the Pavia Center hyperspectral dataset (hereafter Pavia). On all three datasets, TOP delivers a higher PSNR than every compressed baseline while operating at a higher compression ratio (Figure 1(c)); on WV3, for example, it reaches 30.32 dB at 33×, whereas the strongest baseline reaches 27.70 dB at a lower ratio. At these ratios, 1 TB of source observations corresponds to roughly 15.2 GB of downlinked data on WV3 (at 66×), 20.0 GB on GF2, and 41.7 GB on Pavia. Our contributions are summarized as follows:

  • We firstly introduce and formalize transmission-oriented pansharpening, a system-level design that jointly considers compact binary coding, onboard fusion, downlink communication, and ground-side reconstruction instead of assuming fully available lossless observations.

  • We build TOP, which performs pansharpening in a compact binary latent space and transmits only the fused HR-MSI/HR-HSI code; its BSQ bottleneck yields a code that is reconstructable, directly transmittable, and predictable for the latent fusion module.

  • We conduct experiments on multispectral and hyperspectral datasets, showing that TOP achieves superior fusion quality at substantially higher compression ratios than representative pansharpening methods paired with codecs.

Related Work

Pansharpening

Pansharpening fuses the spatial details of a PAN image with the spectral information of an LR-MSI/LR-HSI image [1, 2] and belongs to the broader family of image fusion tasks, which also includes infrared–visible, multi-exposure, and multi-focus fusion [17, 18, 19]. Traditional methods are based on component substitution, such as GS [3], BDSD [4], GSA [20], and PRACS [21]; multi-resolution analysis, such as GLP [5], AWLP [22], and morphological filters [23]; or variational, Bayesian, and low-rank formulations [1, 2, 24, 25]. They are efficient and interpretable, but predefined models may cause spatial or spectral distortions. Deep multispectral models have introduced detail injection [6], dual-branch spatial–spectral modeling [7], frequency-aware or adaptive feature extraction [8, 26, 27], degradation-aware modeling [28], and zero-shot adaptation [29]. Hyperspectral pansharpening must further handle high spectral dimensionality and sensor variability, and relies on Transformer-based fusion and equivariant imaging [30, 31], image-specific priors and band-wise propagation [32, 33, 34], and diffusion-based zero-shot guidance [35, 36]. Degradation-robust fusion methods [10] handle generic low-quality inputs rather than downlink compression artifacts. All these methods assume that the source observations are fully available at the processing platform, whereas TOP performs pansharpening before downlink in a compact binary latent space.

Remote-Sensing Image Compression and Transmission

The growing volume of remote-sensing imagery strains onboard storage and satellite-to-ground communication [9]. Standard codecs such as JPEG2000 [11] and HEVC [12] reduce payloads through transform coding, quantization, and entropy coding. Lossless and near-lossless remote-sensing codecs such as CCSDS 123.0-B-2 [37] target low-complexity, distortion-bounded compression rather than the high-compression regime studied here, and are therefore not included in our comparison. Neural image compression instead learns nonlinear transforms with quantized latents under rate–distortion optimization [13]. Existing pipelines combine compression and pansharpening in one of two orders, either compressing the source observations and fusing on the ground or fusing on board and compressing the fused image, and in both cases treat fusion and compression as separate stages. TOP instead fuses within a compact binary latent space and downlinks the fused code as the payload.

Discrete Visual Representation and Quantization

Transmitting a learned latent requires discretizing it. VQ-VAE [38] assigns latent vectors to a learned codebook whose capacity and utilization are hard to control. Finite scalar quantization [39] maps each dimension to fixed scalar levels, lookup-free quantization [40] derives token indices directly from binary states, and BSQ [15] projects latents onto a unit hypersphere before sign quantization. Such binary codes are highly compact and thus well suited to transmission, so TOP adopts BSQ as a shared latent code for all three modalities. Unlike generative models, which predict discrete tokens with autoregressive or masked-token objectives [40], TOP predicts the target binary code conditionally from the PAN and LR-MSI/LR-HSI codes, and this prediction serves as both the downlink payload and the input to the ground-side decoder.

Methodology

Sec. III-A introduces the notation and the deployment-time pipeline. We then detail the components of TOP, namely the transmission-oriented binary encoder (TOBE; Sec. III-B), BSQ (Sec. III-C), and the transmission-oriented binary decoder (TOBD; Sec. III-E), together with the latent fusion module (Sec. III-D), the training strategy (Sec. III-F), and the overall formulation of TOP (Sec. III-G). Implementation details are provided in Sec. S3 of the Supplementary Material.

Algorithm 1: Deployment-time inference of TOP.

Input: PAN observation P\mathbf{P}; upsampled LR-MSI/LR-HSI observation M\mathbf{M}; trained encoder Eθ\mathcal{E}_{\theta}; quantizer Q\mathcal{Q}; latent fusion module Fψ\mathcal{F}_{\psi}; ground-side decoder Dϕ\mathcal{D}_{\phi}

Output: Reconstructed HR-MSI/HR-HSI image H^\widehat{\mathbf{H}}

▷ Onboard Encoding, Fusion, and Downlink

  1. BP←Q(Eθ(P))\mathbf{B}_{\mathrm{P}} \leftarrow\mathcal{Q}(\mathcal{E}_{\theta}(\mathbf{P}))  ▷ Encode PAN

  2. BM←Q(Eθ(M))\mathbf{B}_{\mathrm{M}} \leftarrow\mathcal{Q}(\mathcal{E}_{\theta}(\mathbf{M}))  ▷ Encode spectral image

  3. O←Fψ(Concat⁡[BM,BP])\mathbf{O} \leftarrow\mathcal{F}_{\psi}(\operatorname{Concat}[\mathbf{B}_{\mathrm{M}},\mathbf{B}_{\mathrm{P}}])  ▷ Predict binary-state logits

  4. B^H←(2arg⁡max⁡a∈{0,1} Oa−1)/Clat\widehat{\mathbf{B}}_{\mathrm{H}} \leftarrow(2\underset{a \in\{0,1\}}{\arg\max}\ \mathbf{O}_{a}-1)/\sqrt{C_{\mathrm{lat}}}  ▷ Normalize the fused code

  5. Sdown←BitPack⁡(B^H)\mathbf{S}_{\mathrm{down}} \leftarrow\operatorname{BitPack}(\widehat{\mathbf{B}}_{\mathrm{H}})  ▷ Pack the binary payload

  6. Downlink Sdown\mathbf{S}_{\mathrm{down}} to the ground station

▷ Ground-Side Reconstruction

  1. B~H←BitUnpack⁡(Sdown)\widetilde{\mathbf{B}}_{\mathrm{H}} \leftarrow\operatorname{BitUnpack}(\mathbf{S}_{\mathrm{down}})  ▷ Recover the received fused code

  2. H^←Dϕ(B~H)\widehat{\mathbf{H}} \leftarrow\mathcal{D}_{\phi}(\widetilde{\mathbf{B}}_{\mathrm{H}})  ▷ Decode the fused HR image

  3. return H^\widehat{\mathbf{H}}

System Overview

Let P∈R1×H×W\mathbf{P} \in\mathbb{R}^{1 \times H \times W}, M∈RC×H×W\mathbf{M} \in\mathbb{R}^{C \times H \times W}, and H∈RC×H×W\mathbf{H} \in\mathbb{R}^{C \times H \times W} denote the PAN observation, the upsampled LR-MSI/LR-HSI observation, and the target HR-MSI/HR-HSI image, respectively. During deployment, TOP divides pansharpening into onboard latent fusion and ground-side reconstruction (Figure 2). On board, the shared TOBE Eθ\mathcal{E}_{\theta} and the BSQ quantizer Q\mathcal{Q} encode P\mathbf{P} and M\mathbf{M} into normalized binary latent codes BP\mathbf{B}_{\mathrm{P}} and BM\mathbf{B}_{\mathrm{M}}, which are concatenated along the channel dimension and processed by the lightweight latent fusion module Fψ\mathcal{F}_{\psi} to predict the fused code B^H\widehat{\mathbf{B}}_{\mathrm{H}}. During training, this prediction is supervised by BH\mathbf{B}_{\mathrm{H}}, the code obtained by encoding H\mathbf{H}; during inference, B^H\widehat{\mathbf{B}}_{\mathrm{H}} is directly bit-packed and transmitted. At the ground station, the received code is unpacked as B~H\widetilde{\mathbf{B}}_{\mathrm{H}}, and the decoder Dϕ\mathcal{D}_{\phi} reconstructs the fused image H^∈RC×H×W\widehat{\mathbf{H}} \in\mathbb{R}^{C \times H \times W}, which approximates the ground-truth (GT) image H\mathbf{H}. Algorithm 1 summarizes the deployment-time inference. The variables H\mathbf{H} and BH\mathbf{B}_{\mathrm{H}} define only the training target; neither is available on board during deployment. The fused code is inferred solely from P\mathbf{P} and M\mathbf{M}, so the payload is a predicted fused representation rather than a compressed copy of a pre-existing HR-MSI/HR-HSI.

Figure 2. Overall pipeline of TOP. On board, the observations are encoded by TOBE, binarized, and fused; the fused code is bit-packed, downlinked, and decoded by TOBD or stored on the ground. Downsampling is omitted for clarity; configurations are listed in Tab. X of the Supplementary Material. GELU: Gaussian error linear unit.

Transmission-Oriented Binary Encoder

TOBE maps observations with different numbers of spectral bands into a shared, compact latent feature space before binary quantization (middle-left panel of Figure 2).

Band-Adaptive Embedding

TOBE embeds all modalities with an adaptive convolution that maintains a learnable base kernel W0\mathbf{W}_{0} with C0C_{0} input channels. For an incoming modality with CinC_{\mathrm{in}} spectral bands, the kernel is resampled along its input-channel dimension:

Winput(c)=∑j=1C0αc,jW0(j),(1)\mathbf{W}_{\mathrm{input}}^{(c)} = \sum_{j=1}^{C_{0}} \alpha_{c,j}\mathbf{W}_{0}^{(j)} , \tag*{(1)}

where the superscripts (c)(c) and (j)(j) select input-channel slices of the adapted and base kernels, c=1,…,Cinc=1,\ldots,C_{\mathrm{in}}, and αc,j\alpha_{c,j} is the bicubic resampling weight from base channel jj to adapted channel cc. PAN and spectral inputs are thus mapped to the same feature dimension with shared weights. TOBD applies the same adaptation along the output-channel dimension.

BlockModeling behaviorMain complexity ↓\downarrow
CNNFixed, localO(Ntokk2d2)\mathcal{O}(N_{\mathrm{tok}}k^{2}d^{2})
Global attentionAdaptive, all tokensO(Ntok2d+Ntokd2)\mathcal{O}(N_{\mathrm{tok}}^{2}d+N_{\mathrm{tok}}d^{2})
SwinAdaptive, windowedO(NtokM2d+Ntokd2)\mathcal{O}(N_{\mathrm{tok}}M^{2}d+N_{\mathrm{tok}}d^{2})

Table 1. Complexity of candidate TOBE backbone blocks (NtokN_{\mathrm{tok}} tokens, width dd, kernel size kk, window size MM).

Swin-Based Multi-Stage Feature Extraction

The embedded features are processed by a Swin Transformer encoder [14], whose windowed self-attention provides stronger content-adaptive spatial–spectral modeling than CNNs at a much lower cost than global self-attention [41] (Tab. I); the ablation in § 5-A verifies this choice. With F0F_0 denoting the embedded feature, the LL encoder stages compute

Fℓ=Sℓ(Rℓ(Fℓ−1)),ℓ=1,…,L,(2)F_{\ell}=S_{\ell}\left(R_{\ell}\left(F_{\ell-1}\right)\right),\qquad\ell=1,\ldots,L, \tag*{(2)}

where RℓR_{\ell} is a downsampling operator, applied only until the target factor ss is reached, and SℓS_{\ell} is a stack of Swin blocks whose depth varies with ℓ\ell (Table X of the Supplementary Material). The output FLF_L has ClatC_{\mathrm{lat}} channels and is spatially reduced by a factor of ss along each dimension.

ConfigurationWV3 33×WV3 66×GF2 50×Pavia 23.97×
Bands/ClatC_{\mathrm{lat}}/s8/8/48/16/84/16/8102/64/4
Stages/d0d_04/643/1283/1284/64
Width multipliers[1, 2, 4, 4][1, 2, 4][1, 2, 4][1, 2, 4, 4]
Encoder/decoder blocks[2, 4, 4, 2]/[3, 6, 6, 3][2, 4, 2]/[4, 2, 2][2, 4, 2]/[4, 2, 2][2, 4, 4, 2]/[4, 4, 2, 2]
Embedding kernel64×8×2×264 \times8 \times2 \times2128×8×2×2128 \times8 \times2 \times2128×4×2×2128 \times4 \times2 \times264×102×2×264 \times102 \times2 \times2
BSQ groups2224
Fusion blocks/input channels8/1612/328/3210/128
Stage-1 batch816168
Stage-1 MSE/BSQ weights1/0.054/0.14/0.11/0.1
Fusion batch2888

Table X. Top architecture and configuration-specific training settings.

Binary Spherical Quantization

Although spatially compressed, FLF_L is still continuous-valued. TOBT therefore adopts BSQ [15], a binary tokenization technique from generative visual representation learning that maps the latent vector at each location to a binary code on the unit hypersphere. BSQ reduces the payload to one bit per latent entry and provides the latent fusion module with a compact and explicit discrete space that is well suited to fusion (§ 5-B). Let fi∈RClatf_i\in\mathbb{R}^{C_{\mathrm{lat}}} denote the channel vector of FLF_L at latent location i=1,…,Ni=1,\ldots,N, with N=HW/s2N=HW/s^2. BSQ performs deterministic sign quantization

bi=sign⁡(fi)∈{−1,+1}Clat,(3)\mathbf{b}_i=\operatorname{sign}(f_i)\in\{-1,+1\}^{C_{\mathrm{lat}}}, \tag*{(3)}

where sign⁡(⋅)\operatorname{sign}(\cdot) acts elementwise, maps zero to +1+1, and is trained with a straight-through estimator. Collecting the codes over all locations gives the normalized binary latent code

B=1Clatreshape⁡([ b1,…,bN ])\mathbf{B}=\frac{1}{\sqrt{C_{\mathrm{lat}}}}\operatorname{reshape}\left([\,\mathbf{b}_1,\ldots,\mathbf{b}_N\,]\right)
∈{−1Clat,+1Clat}Clat×H/s×W/s,(4)\in\left\{-\frac{1}{\sqrt{C_{\mathrm{lat}}}},+\frac{1}{\sqrt{C_{\mathrm{lat}}}}\right\}^{C_{\mathrm{lat}}\times H/s\times W/s}, \tag*{(4)}

where reshape⁡(⋅)\operatorname{reshape}(\cdot) arranges the Clat×NC_{\mathrm{lat}}\times N matrix of column vectors into a tensor in a fixed raster order, and B\mathbf{B} denotes the code of any of the three modalities. During training, soft assignment and grouped entropy regularization encourage effective code utilization [15]. In practice, the latent channels are split into several groups, and the entropy terms are computed per group; for simplicity, the main text treats all channels as a single group. Its details are given in § 9 of the Supplementary Material.

Latent Fusion Module

From the concatenated codes BM\mathbf{B}_M and BP\mathbf{B}_P, the latent fusion module Fψ\mathcal{F}_{\psi} predicts two logits for each latent entry:

O=Fψ(Concat⁡c[BM,BP])∈R2×Clat×H/s×W/s.(5)\mathbf{O}=\mathcal{F}_{\psi}\left(\operatorname{Concat}_{c}[\mathbf{B}_M,\mathbf{B}_P]\right)\in\mathbb{R}^{2\times C_{\mathrm{lat}}\times H/s\times W/s}. \tag*{(5)}

and the fused code is obtained by selecting a binary state independently for each entry:

B^H=1Clat(2arg⁡max⁡a∈{0,1} Oa−1).(6)\widehat{\mathbf{B}}_H=\frac{1}{\sqrt{C_{\mathrm{lat}}}}\left(2\underset{a\in\{0,1\}}{\arg\max}\ \mathbf{O}_a-1\right). \tag*{(6)}

where Oa\mathbf{O}_a is the slice of O\mathbf{O} for state a∈{0,1}a \in\{0,1\}, the arg⁡max⁡\arg\max is taken elementwise, and 1\mathbf{1} is an all-ones tensor of the latent shape.

Binary Downlink and TOBD

The satellite downlinks the bitstream Sdown=BitPack⁡(B^H)\mathbf{S}_{\mathrm{down}} = \operatorname{BitPack}(\widehat{\mathbf{B}}_{\mathrm{H}}), where ‘BitPack(⋅)(\cdot)’ maps negative entries to 0 and nonnegative entries to 1 and serializes them in a fixed order, together with the latent raster shape, whose negligible size is omitted from the compression ratio in Eq. (18). Unlike the original BSQ, which further compresses the codes with an autoregressive prior and arithmetic coding [15], TOP uses no entropy model or rANS codec [16], as supported by § 5-C.

On the ground side, the received bitstream is unpacked and decoded as

B~H=BitUnpack⁡(Sdown),H^=Dϕ(B~H).(7)\widetilde{\mathbf{B}}_{\mathrm{H}} = \operatorname{BitUnpack}(\mathbf{S}_{\mathrm{down}}), \qquad \widehat{\mathbf{H}} = \mathcal{D}_{\phi}(\widetilde{\mathbf{B}}_{\mathrm{H}}). \tag*{(7)}

where ‘BitUnpack(⋅)(\cdot)’ inverts ‘BitPack(⋅)(\cdot)’ by mapping each bit z∈{0,1}z \in\{0,1\} to (2z−1)/Clat(2z-1)/\sqrt{C_{\mathrm{lat}}}; the received code B~H\widetilde{\mathbf{B}}_{\mathrm{H}} may differ from B^H\widehat{\mathbf{B}}_{\mathrm{H}} if transmission errors occur. Dϕ\mathcal{D}_{\phi} follows the stage layout of Eθ\mathcal{E}_{\theta} in reverse order, possibly with different numbers of Swin blocks per stage (lower-left panel of Figure 2), and outputs the final HR-MSI/HR-HSI reconstruction H^\widehat{\mathbf{H}}. The binary code can also be archived for deferred decoding, which yields considerable storage savings.

Optimization Objectives

We first train TOBT, including its TOBE and TOBD, and then train the latent fusion module with the frozen tokenizer (Figure 3).

Figure 3. Training pipeline of the proposed method. (a) Three-stage TOBT training: stage 1 trains TOBE and TOBD jointly (40k iterations); stages 2 and 3 refine TOBD with TOBE frozen (20k iterations each), stage 3 with random bit flips. (b) Latent fusion training with TOBT frozen. Bold outlines: trainable; hatched: frozen; gray: parameter-free; dashed arrows: loss targets.

Three-Stage TOBT Training

Optimization Objectives

a) Loss Components: For an image-domain target X\mathbf{X} and its reconstruction X^\widehat{\mathbf{X}}, the mean squared error (MSE) and Sobel-gradient losses are

LMSE=∥X^−X∥22CHW,Lgrad=∥∇SX^−∇SX∥1CHW,(8)\mathcal{L}_{\mathrm{MSE}}=\frac{\lVert\widehat{\mathbf{X}}-\mathbf{X}\rVert_2^2}{CHW}, \qquad \mathcal{L}_{\mathrm{grad}}=\frac{\lVert\nabla_S\widehat{\mathbf{X}}-\nabla_S\mathbf{X}\rVert_1}{CHW}, \tag*{(8)}

where the norms are computed over vectorized entries and ∇S\nabla_S is the channel-wise Sobel operator; Lgrad\mathcal{L}_{\mathrm{grad}} penalizes edge discrepancies when refining the ground-side decoder. For spectral fidelity, the spectral angle mapper (SAM) loss [42] averages the angle, in radians, between the reconstructed and target spectral vectors x^i,xi∈RC\widehat{\mathbf{x}}_i,\mathbf{x}_i \in\mathbb{R}^{C} over all spatial locations:

LSAM=1HW∑i=1HWarccos⁡(⟨x^i,xi⟩∥x^i∥2∥xi∥2).(9)\mathcal{L}_{\mathrm{SAM}}=\frac{1}{HW}\sum_{i=1}^{HW}\arccos\left(\frac{\langle\widehat{\mathbf{x}}_i,\mathbf{x}_i\rangle}{\lVert\widehat{\mathbf{x}}_i\rVert_2\lVert\mathbf{x}_i\rVert_2}\right). \tag*{(9)}

Unlike MSE, SAM is insensitive to uniform spectral scaling. The quantization loss follows BSQ [15]: a commitment term pulls the normalized continuous latent toward its binary code, and an entropy term prevents the encoder from collapsing onto a few binary patterns:

LQ=βLcommit+Lentropy,(10)\mathcal{L}_{Q}=\beta\mathcal{L}_{\mathrm{commit}}+\mathcal{L}_{\mathrm{entropy}}, \tag*{(10)}

where β\beta weights the commitment term (Sec. S1 of the Supplementary Material).

b) Training Stages: In stage 1 (Figure 3(a)), TOBE and TOBD are jointly trained on all three modalities to stabilize the shared binary latent space before the decoder is refined, using

LTOBT=λMSELMSE+λQLQ.(11)\mathcal{L}_{\mathrm{TOBT}}=\lambda_{\mathrm{MSE}}\mathcal{L}_{\mathrm{MSE}}+\lambda_Q\mathcal{L}_Q. \tag*{(11)}

where λMSE\lambda_{\mathrm{MSE}} and λQ\lambda_Q are weighting coefficients. Stage 2 targets the full TOP setting, in which only the HR code is transmitted: TOBE is frozen, so the quantization loss is dropped, and TOBD is fine-tuned on the HR modality with

LTOBD=LMSE+λgradLgrad+λSAMLSAM,(12)\mathcal{L}_{\mathrm{TOBD}}=\mathcal{L}_{\mathrm{MSE}}+\lambda_{\mathrm{grad}}\mathcal{L}_{\mathrm{grad}}+\lambda_{\mathrm{SAM}}\mathcal{L}_{\mathrm{SAM}}, \tag*{(12)}

where the weights λgrad\lambda_{\mathrm{grad}} and λSAM\lambda_{\mathrm{SAM}} are listed in § 11 of the Supplementary Material.

Stage 3 copes with binary latent errors caused by imperfect latent fusion or by transmission. It keeps the stage-2 setting but injects random bit flips into the HR code before decoding:

BHflip=(1−2R)⊙BH,(13)\mathbf{B}_{\mathrm{H}}^{\mathrm{flip}} = (1 - 2\mathbf{R}) \odot\mathbf{B}_{\mathrm{H}}, \tag*{(13)}

where the entries of the mask R∈{0,1}Clat×H/s×W/s\mathbf{R} \in\{0,1\}^{C_{\mathrm{lat}} \times H/s \times W/s} are independently drawn from Bernoulli(p(t))(p(t)), and an entry equal to 1 reverses the sign of the corresponding latent entry. The flip probability is linearly ramped from pstart=10−5p_{\mathrm{start}} = 10^{-5} to pend=3×10−3p_{\mathrm{end}} = 3 \times10^{-3} over the first 2000 iterations:

p(t)=pstart+min⁡(t2000,1)(pend−pstart),(14)p(t) = p_{\mathrm{start}} + \min\left(\frac{t}{2000},1\right)\left(p_{\mathrm{end}} - p_{\mathrm{start}}\right), \tag*{(14)}

where tt is the stage-3 iteration index. TOBD thereby learns to correct occasional bit errors by exploiting the redundancy of the code. Under 5% bit flips (Figure 4), the stage-3 decoder suppresses the speckle-like artifacts of the stage-2 decoder and better preserves spatial structures and colors; § 5-D quantifies this effect.

WV3 reduced-resolution MSI reconstruction under 5% bit flips

Figure 4. WV3 reduced-resolution MSI reconstruction under 5% bit flips. Left: TOBD after stage 3 (with bit-flip augmentation). Right: TOBD after stage 2 (without). White boxes mark the close-ups below.

Latent Fusion Module Training

With all tokenizer parameters frozen, the PAN, upsampled LR-MSI/LR-HSI, and HR-MSI/HR-HSI images of each training sample are encoded into BP\mathbf{B}_{\mathrm{P}}, BM\mathbf{B}_{\mathrm{M}}, and BH\mathbf{B}_{\mathrm{H}}. The latent fusion module predicts the logits O\mathbf{O} from BM\mathbf{B}_{\mathrm{M}} and BP\mathbf{B}_{\mathrm{P}} as in Eq. (5), and the fixed code BH\mathbf{B}_{\mathrm{H}} supervises it, so the training target uses the same binary representation as the transmitted payload. The module is trained with a binary cross-entropy (BCE) loss on the logits:

Lfuse=fBCEWithLogits(O,T),(15)\mathcal{L}_{\mathrm{fuse}} = f_{\mathrm{BCEWithLogits}}(\mathbf{O},\mathbf{T}), \tag*{(15)}

where the target T∈{0,1}2×Clat×H/s×W/s\mathbf{T} \in\{0,1\}^{2 \times C_{\mathrm{lat}} \times H/s \times W/s} has the same shape as O\mathbf{O}, with state slices T1=I[BH>0]\mathbf{T}_{1} = \mathbb{I}[\mathbf{B}_{\mathrm{H}} > 0] and T0=1−T1\mathbf{T}_{0} = 1 - \mathbf{T}_{1} (elementwise), and the loss averages the scalar BCE terms over both states and all latent entries (§ 10 of the Supplementary Material). No pixel-domain loss is back-propagated through TOBD during this training.

Transmission-Oriented Pansharpening Formulation

At the system level, TOP can be summarized as

min⁡θ,ψ,ϕLfid(H^,H)s.t.H^=Dϕ(B~H), ∣Sdown∣≤Bmax⁡,(16)\begin{aligned} \min_{\theta,\psi,\phi}\quad& \mathcal{L}_{\mathrm{fid}}(\widehat{\mathbf{H}},\mathbf{H}) \\ \text{s.t.}\quad& \widehat{\mathbf{H}} = \mathcal{D}_{\phi}(\widetilde{\mathbf{B}}_{\mathrm{H}}),\ |\mathbf{S}_{\mathrm{down}}| \le B_{\max}, \tag*{(16)} \end{aligned}

where the downlinked bitstream Sdown\mathbf{S}_{\mathrm{down}} and the received code B~H\widetilde{\mathbf{B}}_{\mathrm{H}} are obtained from (P,M)(\mathbf{P},\mathbf{M}) through the encoding, latent-fusion, and bit-packing steps of Alg. 1 (Eqs. (5)–(7)), Lfid(⋅)\mathcal{L}_{\mathrm{fid}}(\cdot) collectively represents the fidelity objectives used throughout TOP, and Bmax⁡B_{\max} denotes the maximum allowable downlink budget.

In contrast, conventional pansharpening mainly considers

min⁡ωLfus(H,Gω(P,M)),(17)\min_{\omega}\quad\mathcal{L}_{\mathrm{fus}}\left(\mathbf{H},\mathcal{G}_{\omega}(\mathbf{P},\mathbf{M})\right), \tag*{(17)}

where Gω\mathcal{G}_{\omega} is a conventional pansharpening model with learnable parameters ω\omega and Lfus\mathcal{L}_{\mathrm{fus}} is its fusion objective. Unlike this formulation, TOP explicitly incorporates the downlink constraint into the pansharpening problem.

Experiments

Experimental Settings

Datasets

1) Datasets: We use the WV3 (8 bands) and GF2 (4 bands) multispectral datasets from PanCollection [43] and the 102-band Pavia Center hyperspectral dataset distributed through HyperPanCollection and used in Hyper-DSNet [44], all with a spatial scale factor of four. WV3, GF2, and Pavia contain 9,714, 19,809, and 1,512 training samples and 1,080, 2,201, and 168 validation samples, respectively; the validation samples are held-out training patches distinct from the test scenes. Training samples consist of 64×6464 \times64 PAN, upsampled LR-MSI/LR-HSI, and HR-MSI/HR-HSI patches together with 16×1616 \times16 native LR-MSI/LR-HSI patches. For WV3 and GF2, we test on 20 reduced-resolution scenes of size 256×256256 \times256, simulated following Wald’s protocol [45], and on 20 full-resolution scenes of size 512×512512 \times512 without GT. For Pavia, the full test split contains two 400×400400 \times400 scenes with HR-HSI references. Details are given in § 12 of the Supplementary Material.

Compared Methods

2) Compared Methods: The multispectral baselines are GS [3], BDSD [4], GLP [5], and MF [23] (traditional); FusionNet [6], U2Net [7], W-FANet [8], ADWM [26], and ARConv [27] (supervised); and ZS-PAN [29] (zero-shot). The hyperspectral baselines are AWLP [22], BDSD-PC [46], BT-H [47], GSA [20], MF [23], and PRACS [21] (traditional); HyperTransformer [30] (supervised); and DIP-HyperKite [32], R-PNN [33], uTDSR [35], DM-ZS [36], and ρ\rho-PNN [34] (zero-shot).

Transmission-Oriented Protocol and Communication Metrics

All baselines follow the ‘compress first, fuse later’ pipeline with JPEG2000 [11] (OpenJPEG, with a search over quality layers to match the target ratio) or HEVC [12] (libx265 with constant rate factor control to match the target ratio). The compression ratio RR divides the raw sensor-side data volume by the size BtransmittedB_{\mathrm{transmitted}} of the transmitted bitstream. To avoid inflating RR, the raw volume uses the native, non-upsampled LR-MSI/LR-HSI:

R=bHW+bC(Hr)(Wr)Btransmitted(18)R = \frac{bHW + bC\left(\frac{H}{r}\right)\left(\frac{W}{r}\right)}{B_{\mathrm{transmitted}}} \tag*{(18)}

where the PAN image has size H×WH \times W, the LR-MSI/LR-HSI has CC bands and size Hr×Wr\frac{H}{r} \times\frac{W}{r}, rr is the spatial scale factor, and bb is the native sensor bit depth (11 for WV3, 10 for GF2, and 13 for Pavia, based on the official specifications). For a codec-based baseline, BtransmittedB_{\mathrm{transmitted}} is the total size of the separately encoded PAN and native LR-MSI/LR-HSI streams; for TOP, it is the size of the single bit-packed fused latent. Using the native spectral image in the numerator is essential because interpolation creates samples without adding sensor information and would otherwise overstate the apparent compression ratio. The bits per pixel per band (bppb) normalizes the payload by the size of the target HR-MSI/HR-HSI:

bppb=BtransmittedCHW.(19)\mathrm{bppb} = \frac{B_{\mathrm{transmitted}}}{CHW}. \tag*{(19)}

Evaluation Metrics

On scenes with GT, we report PSNR, SSIM [48], SAM (in degrees) [42], and ERGAS [49]. We also report the no-reference HQNR index and its spectral and spatial distortion components DλD_{\lambda} and DsD_s [50]. In the reduced-resolution tables, they compare each fused image with the LR-MSI/LR-HSI and PAN of the same scene and thus measure consistency with the source observations rather than full-resolution quality. For the lossy baselines, DλD_{\lambda} and DsD_s are computed against the original lossless observations rather than their decompressed counterparts. Otherwise, a codec could distort both the fusion input and the metric anchor in a similar direction and receive an artificially favorable consistency score. Retaining the common lossless anchors makes HQNR terms comparable across codecs and TOP, while PSNR, SSIM, SAM, and ERGAS independently measure agreement with GT. In the result tables, the best and second-best results are shown in boldface and underlined, respectively.

5) Implementation Details: TOP is implemented in PyTorch with bfloat16 mixed precision on an NVIDIA RTX 4090 GPU and optimized with AdamW at an initial learning rate of 2×10−42 \times10^{-4}, with weight decay 0.05 for TOBT and 0.01 for the latent fusion module. Architectures and training schedules are listed in Sec. S3 of the Supplementary Material.

Multimodal Reconstruction of TOBT

Table III reports validation-set reconstruction of PAN, upsampled LR-MSI/LR-HSI, and HR-MSI/HR-HSI images, and Figure 5 shows examples. These modality-wise results isolate the representation capacity available before latent fusion. On WV3, halving the HR-MSI payload from 0.063 to 0.031 bppb lowers reconstruction PSNR from 30.17 to 28.22 dB, exposing the expected rate–fidelity trade-off. LR-MSI reconstruction nevertheless remains above 37 dB and 0.95 SSIM. TOBT also reaches 33.46 dB on GF2 HR-MSI and 34.85 dB on Pavia HR-HSI, supporting the shared design across sensors and spectral dimensionalities. Fine textures become smoother as the ratio increases, while the main structures and spectral appearance remain stable. These results define the tokenizer’s reconstruction ceiling; the end-to-end experiments test how closely the predicted fused codes approach it.

Figure 5. TOBT multi-modality reconstruction results on WV3, GF2, and Pavia. Upper rows: original test crops; lower rows: independent TOBT reconstructions. Ratios are overall TOP compression ratios (Eq. (18)).

Transmission-Oriented Pansharpening

Having established that TOBT preserves the three modalities under a constrained binary capacity, we evaluate whether the complete TOP pipeline can turn that representation into a useful downlink product. The comparison therefore considers fusion fidelity and communication cost jointly: a method is favorable only if it preserves spatial–spectral information while reducing the transmitted payload relative to the codec-based pipelines.

Multispectral Pansharpening on WV3

Table II reports the WV3 reduced-resolution results under JPEG2000. The best baseline reaches 27.70 dB at 21.26×, whereas TOP achieves 30.32 dB at 33× and 28.88 dB at 66×; relative to the best baseline value of each metric, TOP also reduces ERGAS from 8.36 to 6.00 and SAM from 10.39 to 6.00 degrees. Both TOP configurations outperform all baselines on every metric. Figure 6a shows that TOP preserves sharper local structures and more stable spectral appearance, whereas the codec-based baselines suffer from edge degradation and texture loss. Comparisons with all baselines are given in Sec. S5 of the Supplementary Material.

Figure 6. Qualitative comparison under the JPEG2000 transmission-oriented protocol on (a) WV3, (b) GF2, and (c) Pavia. TOP is compared with the six baselines of highest PSNR (averaged over WV3 and GF2 for the multispectral data). Rows show the predictions with GT and the error maps (dark blue: lower error). All baselines are compared in Sec. S5 of the Supplementary Material.

2) Multispectral Pansharpening on GF2: On GF2 (Tab. IV), TOP achieves 31.37 dB PSNR, 0.83 SSIM, 2.63 ERGAS, and 2.46 degrees SAM at 50×. The strongest baseline, U2Net, reaches 28.53 dB and 4.29 degrees at 20.33×, so TOP improves PSNR by 2.84 dB at about 2.5

Implementation Details

times its compression ratio; the zero-shot ZS-PAN reaches 24.52 dB. Figure 6(b) shows that TOP avoids the codec artifacts that affect the baselines.

Tables V and VI report the corresponding reduced-resolution comparisons under HEVC. The TOP rows are identical to those in the JPEG2000 tables because neither codec is part of TOP: HEVC and JPEG2000 are applied only to the source observations transmitted by the pixel-domain baselines, whereas TOP always bit-packs and transmits the same fused binary latent. Repeating the TOP operating point in each table provides a direct comparison against the baseline pipeline induced by that codec. Although the HEVC baselines operate at only 6.69× on WV3 and 8.94× on GF2, TOP exceeds their strongest PSNR values (27.22 and 29.13 dB) by 3.10 and 2.24 dB at 33× and 50×, respectively, and leads on every metric. The persistence of these gains across two sensors and two codecs indicates that the advantage comes from changing the transmitted representation rather than adapting to the artifacts of one particular codec. The corresponding visual comparisons are provided in Sec. S6, and full-resolution results, where TOP attains the highest HQNR and lowest DλD_{\lambda} under both codecs, are given in Sec. S7 of the Supplementary Material.

3) Hyperspectral Pansharpening on Pavia: Table VII reports the Pavia results under JPEG2000. The baselines operate at 19.62×, with the best PSNR and HQNR reaching 28.09 dB and 0.83. At the higher ratio of 23.97×, TOP achieves 29.72 dB PSNR, 0.83 SSIM, 5.91 ERGAS, 0.85 HQNR, 6.39 degrees SAM, Dλ=0.14D_{\lambda} = 0.14, and Ds=0.02D_{s} = 0.02, the best result on every metric. Figure 6(c) shows that TOP also preserves hyperspectral spatial structures under a constrained downlink. The HEVC results, where TOP again outperforms all baselines, are given in Sec. S6 of the Supplementary Material. Across WV3, GF2, and Pavia, the same pattern holds for 8-, 4-, and 102-band data: TOP improves both GT-based fidelity and consistency with the source observations while using a higher compression ratio. The simultaneous gains in PSNR, SAM, DλD_{\lambda}, and DsD_{s} are important because a higher PSNR alone could mask band-dependent spectral distortion. The following profile analysis examines this balance directly at the band level.

MethodRatio ↑PSNR ↑SSIM ↑ERGAS ↓HQNR ↑DλD_\lambda ↓DsD_s ↓SAM (deg) ↓
AWLP+JPEG2000 [22]26.150.658.940.760.160.0910.91
BDSD-PC+JPEG2000 [46]27.020.658.150.830.150.039.69
BT-H+JPEG2000 [47]27.250.728.600.660.300.0610.16
GSA+JPEG2000 [20]28.090.697.540.780.160.0710.20
MF+JPEG2000 [23]26.720.678.390.780.160.0710.00
PRACS+JPEG2000 [21]26.580.628.590.810.150.059.73
uTDSP+JPEG2000 [35]19.62×25.260.4810.280.690.150.1912.40
DIP-HyperKite+JPEG2000 [32]27.400.697.800.820.150.039.68
HyperTransformer+JPEG2000 [30]19.190.2120.500.440.250.4234.43
R-PNN+JPEG2000 [33]26.620.648.460.790.150.0610.34
DM-ZS+JPEG2000 [36]26.180.679.670.750.200.0611.70
ρ\rho-PNN+JPEG2000 [34]25.140.5810.000.750.140.1311.31
TOP (ours)23.97×29.720.835.910.850.140.026.39

Table VII. Pavia Results Under JPEG2000 (Mean over Two Full Test Scenes; Std. Omitted)

4) Spectral Profile Analysis: Image-level metrics summarize the final reconstruction but do not show how compression and fusion redistribute errors across spectral bands. Figure 7 therefore compares the normalized band-wise mean over a fixed ROI with the GT profile, with all baselines paired with JPEG2000. In all four settings, the TOP curve follows the GT profile most closely and its residual strip remains nearer to zero, whereas the codec-based baselines show larger band-dependent amplitude errors. This trend persists for WV3 at both 33× and 66× and extends to GF2 and the 102-band Pavia data. The spectral-profile evidence thus complements SAM and DλD_{\lambda}: the quantitative gains are not produced by spatial sharpening at the expense of spectral consistency.

Figure 7. Spectral profile comparison on WV3, GF2, and Pavia. (a, b) WV3 with TOP at 33× and 66×, (c) GF2 at 50×, and (d) Pavia at 23.97×. Upper plots: normalized band-wise mean over a fixed ROI; lower strips: deviation from GT; best-ranked baseline of each method family.

Ablation Studies

We ablate four design choices of TOP on the WV3 reduced-resolution data: the tokenizer backbone, BSQ, an additional entropy model, and stage-3 bit-flip training. These components address different system constraints, so they cannot be judged by reconstruction accuracy alone: the backbone controls representation quality and computation, BSQ affects payload and latent predictability, entropy coding trades bitrate for sender-side overhead, and stage 3 controls sensitivity to erroneous bits. Figure 8 summarizes these accuracy–payload–complexity trade-offs, and the numerical values are listed in panels (a)–(e) of Table XVI in the Supplementary Material.

Figure 8. Ablation studies of the key design choices of TOP. WV3 reduced-resolution data; blue: design adopted in TOP, gray: alternative. (a) Backbone (HR-MSI reconstruction). (b) BSQ: reconstruction–fusion PSNR gap (left), payload and fusion-module complexity (right). (c) Additional entropy model with rANS. (d) Degradation under random bit flips after stages 2 and 3. Bars in (b) and (c) are scaled per pair. Values are listed in § 16 of the Supplementary Material.

(a) Architecture reconstruction
ArchitectureParams (M) ↓GFLOPs ↓PSNR ± Std. ↑SSIM ± Std. ↑
CNN66.721169.9627.42 ± 2.020.75 ± 0.06
ViT62.709219.4627.05 ± 1.820.75 ± 0.06
Swin65.609171.7228.92 ± 2.070.83 ± 0.04
(b) BSQ payload
Methodbppb ↓Payload/latent entry ↓
TOBT0.06251 bit
TOBT w/o BSQ2.00032 bits
(c) BSQ reconstruction, fusion, and latent-module complexity
MethodRecon. PSNR ↑Fusion PSNR ↑Params (M) ↓GFLOPs ↓
TOBT31.5930.320.610.31
TOBT w/o BSQ34.7032.1119.339.88
(d) Direct packing versus entropy coding
Methodbppb ↓Add. params ↓Add. GFLOPs/sample ↓
TOP0.0625000.00M0.00
TOBEM+rans0.06026010.329096M74.769797
(e) Bit-flip robustness on 20 scenes
StageFlipΔPSNR (dB) ↓ΔSAM (deg) ↓
Stage 23%3.240.36
Stage 33%0.76−0.11
Stage 25%5.170.64
Stage 35%1.61−0.07

Table XVI. Numerical ablations on WV3 reduced-resolution data.

Architecture Choice

We compare CNN, Vision Transformer (ViT) [41], and Swin [14] backbones with 66.721M, 62.709M, and 65.609M parameters, respectively, keeping the latent configuration, quantization, and training fixed; this Swin variant is matched to the parameter budget and differs from the dataset-specific tokenizer configurations listed in Table X of the Supplementary Material. As shown in Figure 8(a), Swin achieves the best HR-MSI reconstruction at a computational cost close to that of the CNN and lower than that of ViT, whereas ViT adds computation without a corresponding gain. Swin is therefore adopted for TOBE and TOBD.

Effectiveness of BSQ

BSQ reduces the payload to one bit per latent entry (Figure 8(b)). To test whether it also makes the target latent easier to predict, we compare the fusion PSNR with the PSNR of directly encoding and decoding the GT image with the same tokenizer; their gap indicates how closely latent fusion approaches the reconstruction capability of the latent space. Both PSNRs are computed on the 20 WV3 reduced-resolution test scenes, so the fusion PSNR with BSQ is the 30.32 dB of TOP at 33× in Table II, whereas Table III reports reconstruction on the validation set. Removing BSQ improves reconstruction, but even with a deliberately enlarged and fully trained fusion module, the continuous-latent variant leaves a 2.59 dB gap between reconstruction and fusion. BSQ reduces this gap to 1.27 dB while shrinking the fusion module from 19.33M to 0.61M parameters and from 9.88 to 0.31 giga floating-point operations (GFLOPs). BSQ thus trades part of the reconstruction capacity for a compact payload and a more predictable fusion target.

MethodRatio ↑PSNR ± Std. ↑SSIM ± Std. ↑ERGAS ± Std. ↓HQNR ± Std. ↑DλD_{\lambda} ± Std. ↓DsD_s ± Std. ↓SAM (deg) ± Std. ↓
BDSD+JPEG2000 [4]25.44 ± 3.850.55 ± 0.1610.85 ± 2.310.17 ± 0.060.68 ± 0.060.48 ± 0.1211.20 ± 2.44
FusionNet+JPEG2000 [6]26.50 ± 3.690.73 ± 0.099.39 ± 2.310.36 ± 0.080.54 ± 0.080.23 ± 0.0910.51 ± 2.34
GLP+JPEG2000 [5]26.53 ± 3.840.68 ± 0.119.56 ± 2.040.27 ± 0.060.63 ± 0.050.29 ± 0.1110.64 ± 2.42
GS+JPEG2000 [3]27.70 ± 3.740.71 ± 0.118.36 ± 1.780.47 ± 0.090.43 ± 0.080.18 ± 0.0910.52 ± 2.49
MF+JPEG2000 [23]21.26×26.59 ± 3.880.67 ± 0.129.49 ± 2.080.27 ± 0.080.62 ± 0.070.29 ± 0.1210.44 ± 2.34
U2Net+JPEG2000 [7]26.62 ± 3.630.70 ± 0.109.34 ± 2.050.33 ± 0.080.56 ± 0.080.26 ± 0.0910.39 ± 2.23
WFANet+JPEG2000 [8]25.64 ± 3.920.64 ± 0.1310.54 ± 2.450.20 ± 0.080.68 ± 0.070.38 ± 0.1110.62 ± 2.34
ZS-PAN+JPEG2000 [29]26.02 ± 3.540.62 ± 0.149.92 ± 1.700.23 ± 0.090.65 ± 0.080.37 ± 0.1411.00 ± 2.44
ADWM+JPEG2000 [26]26.02 ± 5.520.71 ± 0.1411.23 ± 5.660.38 ± 0.120.53 ± 0.100.21 ± 0.1215.19 ± 7.37
ARConv+JPEG2000 [27]26.70 ± 3.840.70 ± 0.119.33 ± 1.950.29 ± 0.080.60 ± 0.070.29 ± 0.1010.43 ± 2.30
TOP (ours)33×30.32 ± 2.830.82 ± 0.046.00 ± 1.230.76 ± 0.140.19 ± 0.110.08 ± 0.066.00 ± 1.38
TOP (ours)66×28.88 ± 3.060.75 ± 0.077.13 ± 1.480.69 ± 0.130.23 ± 0.110.11 ± 0.056.69 ± 1.56

Table II. WV3 reduced-resolution results under JPEG2000 (mean ± std.).

DatasetModalitybppb ↓PSNR ± Std. ↑SSIM ± Std. ↑
WV3 (33×)HR-MSI0.06330.17 ± 4.420.83 ± 0.08
LR-MSI0.06339.61 ± 3.180.97 ± 0.01
PAN0.50034.88 ± 5.010.91 ± 0.04
WV3 (66×)HR-MSI0.03128.22 ± 4.590.74 ± 0.12
LR-MSI0.03137.14 ± 3.730.95 ± 0.02
PAN0.25031.94 ± 5.430.83 ± 0.08
GF2HR-MSI0.06333.46 ± 3.750.87 ± 0.06
LR-MSI0.06340.31 ± 2.850.97 ± 0.02
PAN0.25033.46 ± 3.940.87 ± 0.05
PaviaHR-HSI0.03934.85 ± 3.500.92 ± 0.02
LR-HSI0.03943.90 ± 2.570.99 ± 0.00
PAN4.00040.86 ± 2.930.97 ± 0.01

Table III. TOBT Reconstruction on the Validation Sets (Mean ± Std.; BPPB: Modality-Specific Payload, Eq. (19))

Trade-Off of an Additional Entropy Model

A neural entropy model, which drives rANS [16] or arithmetic coding with estimated symbol probabilities [13], is most effective when the latent retains considerable statistical redundancy, whereas TOBT already compresses the HR information

Multispectral Pansharpening on GF2

MethodRatio ↑PSNR ± Std. ↑SSIM ± Std. ↑ERGAS ± Std. ↓HQNR ± Std. ↑DλD_\lambda ± Std. ↓DsD_s ± Std. ↓SAM (deg) ± Std. ↓
BDSD+JPEG2000 [4]20.33×24.33 ± 2.190.63 ± 0.085.78 ± 0.990.09 ± 0.020.80 ± 0.020.57 ± 0.065.32 ± 0.76
FusionNet+JPEG2000 [6]26.73 ± 2.160.81 ± 0.034.36 ± 0.740.46 ± 0.060.50 ± 0.060.08 ± 0.034.97 ± 0.80
GLP+JPEG2000 [5]24.56 ± 2.130.69 ± 0.085.62 ± 0.960.12 ± 0.030.79 ± 0.030.46 ± 0.085.28 ± 0.77
GS+JPEG2000 [3]26.99 ± 1.770.78 ± 0.054.16 ± 0.640.53 ± 0.070.36 ± 0.060.18 ± 0.055.36 ± 1.14
MF+JPEG2000 [23]25.04 ± 2.180.74 ± 0.065.32 ± 0.930.20 ± 0.040.73 ± 0.040.29 ± 0.085.20 ± 0.77
U2Net+JPEG2000 [7]28.53 ± 1.650.81 ± 0.033.41 ± 0.450.63 ± 0.070.31 ± 0.060.09 ± 0.044.29 ± 0.60
WFANet+JPEG2000 [8]25.65 ± 2.230.76 ± 0.044.88 ± 0.850.27 ± 0.030.65 ± 0.040.23 ± 0.054.99 ± 0.76
ADWM+JPEG2000 [26]25.95 ± 2.290.79 ± 0.044.74 ± 0.890.38 ± 0.050.56 ± 0.050.13 ± 0.055.10 ± 0.84
ARConv+JPEG2000 [27]24.82 ± 2.240.72 ± 0.065.41 ± 0.960.14 ± 0.020.77 ± 0.020.37 ± 0.065.10 ± 0.75
ZS-PAN+JPEG2000 [29]24.52 ± 2.050.67 ± 0.085.63 ± 0.900.12 ± 0.030.78 ± 0.030.47 ± 0.085.34 ± 0.77
TOP (ours)50×31.37 ± 1.900.83 ± 0.052.63 ± 0.500.83 ± 0.030.11 ± 0.030.07 ± 0.022.46 ± 0.42

Table IV. GF2 Reduced-Resolution Results Under JPEG2000 (Mean ± Std.).

Hyperspectral Pansharpening on Pavia

into a tight binary bottleneck. We therefore code the fused binary latent that TOP downlinks for WV3, which uses 0.0625 bppb with direct bit packing, with a recent 10.329096M-parameter dictionary-based entropy model [51] and rANS. As shown in Figure 8(c), the payload decreases only from 0.062500 to 0.060260 bppb (about 3.58%), at the cost of 74.769797 additional GFLOPs/sample for probability prediction alone, plus GPU synchronization, data transfer, CPU rANS coding,

Spectral Profile Analysis

and packet assembly on the sender side. This overhead is disproportionate to the gain, which supports the entropy-coding-free design of TOP.

Stage-3 Training Strategy on Robustness

The fused code is both the output of onboard fusion and the downlink payload, so a wrong bit may come from imperfect latent prediction or from transmission errors. We randomly flip 3% or 5% of the signs of the fused code on the 20 WV3 reduced-resolution test scenes and decode it with TOBD after stage 2 or stage 3, keeping the encoder, latent fusion module, and unperturbed code unchanged. As shown in Figure 8(d), stage-3 training reduces the PSNR drop from 3.24 to 0.76 dB at 3% flips and from 5.17 to 1.61 dB at 5% flips. Under the same flips, stage 2 increases SAM by 0.36 and 0.64 degrees, whereas after stage 3 SAM changes by only −0.11 and −0.07 degrees. The small negative values can be interpreted as negligible sample-level variation rather than a benefit from corruption; the relevant result is that spectral fidelity no longer

deteriorates systematically. Bit-flip training therefore makes TOBD markedly less sensitive to bit errors.

Discussion

Mechanism of Latent-Space Fusion

TOP works because one binary latent space serves reconstruction, fusion, and transmission. In stage 1, the shared tokenizer must reconstruct PAN, LR-MSI/LR-HSI, and HR-MSI/HR-HSI images under the same binary capacity, so it keeps the structures needed for both spatial detail and spectral recovery and discards pixel-level redundancy that is costly to transmit. In this space, the latent fusion module only has to predict a binary state at each aligned latent location, instead of synthesizing every output pixel and band.

The BSQ ablation shows the cost and the benefit of the design (§ 5-B). Without BSQ, the tokenizer reconstructs better on its own, but the fusion target is continuous. It then needs a much larger fusion module and still leaves a larger reconstruction–fusion gap. BSQ gives up part of the standalone reconstruction capacity and obtains a target that is compact and easy to predict. The training stages build on this. Stage 2 fine-tunes TOBD on the HR code, and stage 3 adds bit-flip augmentation so that TOBD tolerates wrong bits from imperfect latent prediction or from the channel. With TOBT frozen, latent-fusion training then makes the onboard prediction match the HR code. The fused code is therefore already the payload for the channel, not an intermediate image that must be compressed again.

Improvements Over Existing Pipelines

Ground-side fusion after compression and onboard pixel-domain fusion before compression are both limited by an objective mismatch. In the first, the codec discards information before fusion, and the fusion model is usually trained on clean inputs, so a better fusion model cannot recover what the downlink has already removed. In the second, the fusion model produces a full-resolution image without knowing how a generic codec will quantize it, and the codec minimizes image distortion without knowing which structures matter for pansharpening. Either way, two stages with different objectives are concatenated.

TOP removes this mismatch by making the fused latent code the payload. Tokenization sets what can be represented within the bit budget, onboard fusion predicts that representation, and the ground decoder reconstructs the image from the same code that was optimized during training. This agrees with the results against compress-then-fuse baselines in the main text and against onboard fusion followed by JPEG2000 or HEVC in Sec. S9 of the Supplementary Material. The gain therefore comes from coordinating representation, fusion, and transmission, not from one fusion network or one codec alone.

Deployment Flexibility

The binary payload keeps onboard processing simple. It ends with sign selection and bit packing, and it needs neither full-resolution reconstruction nor a learned entropy model (§ 5-C). The fixed payload size is known before transmission, which helps buffer allocation and link scheduling, and decoding stays on the ground.

The shared encoder supports two transmission policies. By default, TOP sends the fused latent. If downstream tasks need the source observations, the satellite can instead send the PAN and spectral latents, and the ground station reconstructs them and applies an existing fusion method. Table VIII shows that this policy remains viable: on TOBT-decoded inputs, FusionNet reaches 29.95 dB on GF2 at 25.0×25.0\times and 29.32 dB on WV3 at 16.5×16.5\times. These ratios are about half those of TOP because two equal-sized source latents are sent. TOP sends only the fused latent, at 50×50\times and 33×33\times, while improving PSNR by 1.42 dB on GF2 and 1.00 dB on WV3. The choice is therefore explicit: transmit the source latents for reuse, or transmit the fused latent for the best rate–quality trade-off.

DatasetMethodRatio ↑\uparrowPSNR ↑\uparrowSAM (deg) ↓\downarrowHQNR ↑\uparrow
GF2FusionNet [6]25.0×25.0\times29.95±1.6329.95 \pm1.633.33±0.483.33 \pm0.480.72±0.040.72 \pm0.04
GF2WFAENet [8]25.0×25.0\times29.19±1.7729.19 \pm1.773.48±0.473.48 \pm0.470.66±0.030.66 \pm0.03
GF2TOP (ours)50×50\times31.37±1.90\mathbf{31.37} \pm1.902.46±0.42\mathbf{2.46} \pm0.420.83±0.03\mathbf{0.83} \pm0.03
WV3FusionNet [6]16.5×16.5\times29.32±2.6229.32 \pm2.627.93±1.867.93 \pm1.860.65±0.150.65 \pm0.15
WV3WFAENet [8]16.5×16.5\times27.65±3.2427.65 \pm3.248.24±1.948.24 \pm1.940.49±0.110.49 \pm0.11
WV3TOP (ours)33×33\times30.32±2.83\mathbf{30.32} \pm2.836.00±1.38\mathbf{6.00} \pm1.380.76±0.14\mathbf{0.76} \pm0.14

Table VIII. Ground-Side Fusion After TOBT-Only Transmission (Mean ±\pm Std.)

TOP also has two practical limitations: (1) the tokenizer is trained per sensor, and (2) training takes several stages. Sec. S10 of the Supplementary Material discusses them.

Conclusion

This work presents TOP, which treats pansharpening for satellite downlink as a joint design of compact latent coding, onboard fusion, and transmission. TOBE with a BSQ bottleneck produces a binary latent code that can be reconstructed, predicted, and bit-packed directly. TOBD decodes the fused code on the ground and can tolerate occasional bit errors. On WV3, GF2, and Pavia datasets, TOP produces higher fusion quality than pansharpening methods paired with common codecs, at compression ratios 1.2 to 9.9 times those of these baselines. For example, on WV3 it reaches 30.32 dB PSNR at 33×33\times, whereas the best JPEG2000 baseline reaches 27.70 dB at 21.26×21.26\times. These findings suggest that future remote-sensing systems could benefit from co-designing task processing and transmission, rather than treating downlink transmission as a separate stage. Future work will simplify the training procedure and improve generalization across sensors.

Supplementary Material for TOP: Transmission-Oriented Remote Sensing Pansharpening

Zi-Han Cao; Jing-Ze Liang; Shao-Jie Wu; Jin-Cheng Di; Liang-Jian Deng; Gemine Vivone

This supplement details BSQ and the latent-fusion loss (Secs. S1–S2), implementation and evaluation protocols (Secs. S3–S4), and additional JPEG2000, HEVC, full-resolution, ablation, and onboard-fusion results (Secs. S5–S9). Sec. S10 discusses limitations. In all tables, bold and underlined entries are the best and second best; repeated-sample results are mean ±\pm standard deviation. Baseline references appear in the main manuscript. In reduced-resolution figures, prediction and error-map rows alternate; darker blue indicates lower error, and green boxes mark enlarged regions.

Details of Binary Spherical Quantization

Following BSQ [15], inference and downlink use only the hard normalized tensor B\mathbf{B}; the soft assignments below are training auxiliaries.

a) Hard Spherical Binary Code: For fi∈RClat\mathbf{f}_i \in\mathbb{R}^{C_{\mathrm{lat}}} at latent location ii,

bi=sign⁡(fi)∈{−1,+1}Clat,\mathbf{b}_i = \operatorname{sign}(\mathbf{f}_i) \in\{-1,+1\}^{C_{\mathrm{lat}}},
B=1Clatreshape⁡([b1,…,bN])∈{−1/Clat,+1/Clat}Clat×H/s×W/s(20)\mathbf{B} = \frac{1}{\sqrt{C_{\mathrm{lat}}}}\operatorname{reshape}([\mathbf{b}_1,\ldots,\mathbf{b}_N]) \in\{-1/\sqrt{C_{\mathrm{lat}}},+1/\sqrt{C_{\mathrm{lat}}}\}^{C_{\mathrm{lat}}\times H/s\times W/s} \tag*{(20)}

where N=HW/s2N = HW/s^2 and reshape maps the Clat×NC_{\mathrm{lat}} \times N raster-ordered vectors to the latent grid. For each of the three modalities, define

fˉi=fi∥fi∥2+ϵ,qi=biClat,(21)\bar{\mathbf{f}}_i = \frac{\mathbf{f}_i}{\lVert\mathbf{f}_i\rVert_2+\epsilon}, \qquad\mathbf{q}_i = \frac{\mathbf{b}_i}{\sqrt{C_{\mathrm{lat}}}}, \tag*{(21)}

with numerical constant ϵ>0\epsilon> 0. Positive normalization preserves the sign, and the straight-through estimator (STE) is

Q(fˉi)=fˉi+sg⁡(qi−fˉi),∂Q∂fˉi=STEIClat.(22)Q(\bar{\mathbf{f}}_i) = \bar{\mathbf{f}}_i + \operatorname{sg}(\mathbf{q}_i-\bar{\mathbf{f}}_i), \qquad\frac{\partial Q}{\partial\bar{\mathbf{f}}_i}\mathrel{\overset{\mathrm{STE}}{=}} \mathbf{I}_{C_{\mathrm{lat}}}. \tag*{(22)}

Thus, the forward value is qi\mathbf{q}_i, while the identity surrogate Jacobian is used only for back-propagation; sg denotes stop-gradient. Bit packing stores signs, and unpacking restores the two normalized values. The corresponding implicit, unlearned codebook is

CClat={cnClat | cn∈{−1,+1}Clat, n=1,…,2Clat}.(23)\mathcal{C}_{C_{\mathrm{lat}}} = \left\{\frac{\mathbf{c}_n}{\sqrt{C_{\mathrm{lat}}}} \,\middle|\, \mathbf{c}_n \in\{-1,+1\}^{C_{\mathrm{lat}}},\ n=1,\ldots,2^{C_{\mathrm{lat}}}\right\}. \tag*{(23)}

b) Geometry and Packed Payload: The spherical normalization makes binary disagreement directly measurable. For two codewords, with Hamming distance h(bi,bi′)h(\mathbf{b}_i,\mathbf{b}'_i),

qiTqi′=1−2h(bi,bi′)Clat,∥qi−qi′∥22=4h(bi,bi′)Clat.(24)\mathbf{q}_i^{\mathsf{T}}\mathbf{q}'_i = 1-\frac{2h(\mathbf{b}_i,\mathbf{b}'_i)}{C_{\mathrm{lat}}}, \qquad\lVert\mathbf{q}_i-\mathbf{q}'_i\rVert_2^2 = \frac{4h(\mathbf{b}_i,\mathbf{b}'_i)}{C_{\mathrm{lat}}}. \tag*{(24)}

A sign error therefore causes a fixed, bounded latent displacement, and latent prediction errors have the same form as transmission bit errors. The packed tensor contains NClat=ClatHW/s2NC_{\mathrm{lat}}=C_{\mathrm{lat}}HW/s^2 bits, giving bppb=Clat/(Cs2)\mathrm{bpp}_{b}=C_{\mathrm{lat}}/(Cs^2) for CC target bands. Its payload is fixed by (Clat,s)(C_{\mathrm{lat}},s) and requires no codebook indices, floating-point magnitudes, or probability tables. The dot-product identity also keeps similarity in the fixed interval [−1,1][-1,1] across latent dimensions, so changing the rate configuration does not change the scale presented to the decoder.

c) Soft Assignment for Entropy Regularization: During training, inverse temperature η>0\eta>0 defines the nearest-code distribution

pi,n=exp⁡(−η∥fˉi−c^n∥22)∑j=12Clatexp⁡(−η∥fˉi−c^j∥22)=exp⁡(2ηcnTfi)∑j=12Clatexp⁡(2ηcjTfi),c^n=cn/Clat.(25)\begin{aligned} p_{i,n}=\frac{\exp(-\eta\lVert\bar{\mathbf{f}}_i-\hat{\mathbf{c}}_n\rVert_2^2)}{\sum_{j=1}^{2^{C_{\mathrm{lat}}}}\exp(-\eta\lVert\bar{\mathbf{f}}_i-\hat{\mathbf{c}}_j\rVert_2^2)} \\ &=\frac{\exp(2\eta\mathbf{c}_n^{\mathsf{T}}\mathbf{f}_i)}{\sum_{j=1}^{2^{C_{\mathrm{lat}}}}\exp(2\eta\mathbf{c}_j^{\mathsf{T}}\mathbf{f}_i)}, \qquad\hat{\mathbf{c}}_n=\mathbf{c}_n/\sqrt{C_{\mathrm{lat}}}. \tag*{(25)} \end{aligned}

The second form follows because codeword-independent squared-norm terms cancel. The Cartesian binary codebook factorizes by channel:

qˉi,d=σ(−4ηClatfˉi,d),qˉi,d+=1−qˉi,d,(26)\bar{q}_{i,d}=\sigma\left(-\frac{4\eta}{\sqrt{C_{\mathrm{lat}}}}\bar{f}_{i,d}\right), \qquad\bar{q}^{+}_{i,d}=1-\bar{q}_{i,d}, \tag*{(26)}
σ(z)=(1+e−z)−1.\sigma(z)=(1+e^{-z})^{-1}.

In practice, split the channels into G=Clat/gG=C_{\mathrm{lat}}/g disjoint groups Ih\mathcal{I}_h of gg channels. Let fi(h)=(fi,d)d∈Ih\mathbf{f}^{(h)}_i=(f_{i,d})_{d\in\mathcal{I}_h}, ch,n∈{−1,+1}g\mathbf{c}_{h,n}\in\{-1,+1\}^{g} for n=1,…,2gn=1,\ldots,2^g, and c^h,n=ch,n/Clat\hat{\mathbf{c}}_{h,n}=\mathbf{c}_{h,n}/\sqrt{C_{\mathrm{lat}}}. Then

pi,h,n=exp⁡(2ηc^h,nTfi(h))∑j=12gexp⁡(2ηc^h,jTfi(h)),pi,h=(pi,h,n)n=12g.(27)p_{i,h,n}=\frac{\exp(2\eta\hat{\mathbf{c}}_{h,n}^{\mathsf{T}}\mathbf{f}^{(h)}_i)}{\sum_{j=1}^{2^g}\exp(2\eta\hat{\mathbf{c}}_{h,j}^{\mathsf{T}}\mathbf{f}^{(h)}_i)}, \qquad\mathbf{p}_{i,h}=(p_{i,h,n})_{n=1}^{2^g}. \tag*{(27)}

This marginal preserves the global 1/Clat1/\sqrt{C_{\mathrm{lat}}} scaling and reduces explicit assignment from 2Clat2^{C_{\mathrm{lat}}} states to G2gG2^g grouped states.

d) Entropy and Commitment Terms: For the batch and latent locations Ωe\Omega_e,

Hsample=1∣Ωe∣∑i∈Ωe∑h=1GH(pi,h),H(p)=−∑npnlog⁡(pn+ϵ).(28)\mathcal{H}_{\mathrm{sample}}=\frac{1}{|\Omega_e|}\sum_{i\in\Omega_e}\sum_{h=1}^{G}\mathcal{H}(\mathbf{p}_{i,h}), \qquad\mathcal{H}(\mathbf{p})=-\sum_n p_n\log(p_n+\epsilon). \tag*{(28)}

while the entropy of average code usage is

pˉh,n=1∣Ωe∣∑ipi,h,n,Hcode=∑h=1GH(pˉh),(29)\bar{p}_{h,n}=\frac{1}{|\Omega_e|}\sum_i p_{i,h,n}, \qquad\mathcal{H}_{\mathrm{code}}=\sum_{h=1}^{G}\mathcal{H}(\bar{\mathbf{p}}_h), \tag*{(29)}

where pˉh=(pˉh,n)n=12g\bar{\mathbf{p}}_h=(\bar{p}_{h,n})_{n=1}^{2^g}. The first term favors decisive local assignments; the second prevents collapse through broad group-code usage. TOP minimizes

Lentropy=ζη(γHsample−γHcode)(30)\mathcal{L}_{\mathrm{entropy}}=\frac{\zeta}{\eta}(\gamma\mathcal{H}_{\mathrm{sample}}-\gamma\mathcal{H}_{\mathrm{code}}) \tag*{(30)}

and the commitment term

Lcommit=1∣Ωq∣∑i∈Ωq∥sg⁡(qi)−fˉi∥22,(31)\mathcal{L}_{\mathrm{commit}}=\frac{1}{|\Omega_q|}\sum_{i\in\Omega_q}\lVert\operatorname{sg}(\mathbf{q}_i)-\bar{\mathbf{f}}_i\rVert_2^2, \tag*{(31)}

where Ωq\Omega_q is the full batch and latent grid. This moves the continuous feature toward the fixed hard code. Their combination is Eq. (10) of the main manuscript. Tab. IX gives all training coefficients; they do not change the deterministic hard code. The effective coefficients of Lcommit\mathcal{L}_{\mathrm{commit}}, Hsample\mathcal{H}_{\mathrm{sample}}, and Hcode\mathcal{H}_{\mathrm{code}} are (5×10−3,5×10−4,−5×10−4)(5 \times10^{-3}, 5 \times10^{-4}, -5 \times10^{-4}) for WV3 33×, (10−2,10−3,−10−3)(10^{-2}, 10^{-3}, -10^{-3}) for WV3 66×, and (0.1,0.1,−0.1)(0.1, 0.1, -0.1) for GF2 and Pavia.

CoefficientWV3 33×WV3 66×GF2 50×Pavia 23.97×
β\beta, γ0\gamma_0, γ\gamma, ζ\zeta0.10.111
η\eta1111
λQ\lambda_Q0.050.10.10.1

Table IX. BSQ training coefficients; training uses hard BSQ with STE, whereas inference uses hard BSQ only.

e) Boundary With Entropy Coding: The entropy regularizer is used only for optimization; it produces no probability model, side information, or arithmetic-coded stream. Deployment discards pi,np_{i,n}, qi,d±q^{\pm}_{i,d}, and pi,h,np_{i,h,n} and directly bit-packs the signs of B^H\hat{\mathbf{B}}_{\mathrm{H}}. A neural entropy model would additionally require probability inference, packet coding, and decoding.

Binary Cross-Entropy Loss of the Latent Fusion Module

Sec. III-D defines the single logit tensor O\mathbf{O}. Its first axis a∈{0,1}a \in\{0,1\} represents the negative or positive state, and Oa,d,u,vO_{a,d,u,v} is one logit. For each target entry BH,d,u,vB_{\mathrm{H},d,u,v}, define

yd,u,v=I[BH,d,u,v>0]∈{0,1}(32)y_{d,u,v} = \mathbb{I}[B_{\mathrm{H},d,u,v} > 0] \in\{0,1\} \tag*{(32)}

and lift it to two complementary channels:

Ta,d,u,v=I[a=yd,u,v],a∈{0,1}.(33)T_{a,d,u,v} = \mathbb{I}[a = y_{d,u,v}], \qquad a \in\{0,1\}. \tag*{(33)}

Thus, T0,d,u,v=1−yd,u,vT_{0,d,u,v} = 1-y_{d,u,v} and T1,d,u,v=yd,u,vT_{1,d,u,v}=y_{d,u,v}. The resulting T∈{0,1}2×Clat×H/s×W/s\mathbf{T} \in\{0,1\}^{2 \times C_{\mathrm{lat}} \times H/s \times W/s} matches O\mathbf{O}, so fBCEWithLogits(O,T)f_{\mathrm{BCEWithLogits}}(\mathbf{O},\mathbf{T}) requires no broadcasting. For scalar logit oo and target tt,

ℓ(o,t)=max⁡(o,0)−ot+log⁡(1+exp⁡(−∣o∣)),(34)\ell(o,t) = \max(o,0) - ot + \log(1+\exp(-\lvert o\rvert)), \tag*{(34)}

which evaluates the exponential only at a nonpositive argument and avoids an explicit sigmoid. The latent-fusion objective is

Lfuse=12ClatN∑a=01∑d=1Clat∑u=1H/s∑v=1W/sℓ(Oa,d,u,v,Ta,d,u,v),N=HWs2.(35)\mathcal{L}_{\mathrm{fuse}} = \frac{1}{2C_{\mathrm{lat}}N} \sum_{a=0}^{1} \sum_{d=1}^{C_{\mathrm{lat}}} \sum_{u=1}^{H/s} \sum_{v=1}^{W/s} \ell(O_{a,d,u,v},T_{a,d,u,v}), \qquad N = \frac{HW}{s^2}. \tag*{(35)}

This is complementary one-vs-rest BCE, rather than softmax cross-entropy. The frozen values −1/Clat-1/\sqrt{C_{\mathrm{lat}}} and +1/Clat+1/\sqrt{C_{\mathrm{lat}}} map to a=0a=0 and a=1a=1. At inference, ad,u,v∗=arg⁡max⁡a∈{0,1}Oa,d,u,va^{*}_{d,u,v}=\arg\max_{a\in\{0,1\}}O_{a,d,u,v} and B^H,d,u,v=(2ad,u,v∗−1)/Clat\hat{B}_{\mathrm{H},d,u,v}=(2a^{*}_{d,u,v}-1)/\sqrt{C_{\mathrm{lat}}}, preserving the training state convention; no sigmoid precedes fBCEWithLogitsf_{\mathrm{BCEWithLogits}}.

Implementation Details

All configurations use Swin-based encoders/decoders, grouped BSQ with eight channels per group, and a band-adaptive 2×22 \times2 stride-2 embedding. The kernel layout is output×input×height×width and is interpolated along the input-channel dimension. The factor ss applies to both spatial axes, and stage counts include stages without scale changes. Tab. X retains all architecture and configuration-specific training values.

The fusion module concatenates spectral and PAN codes, maps 2Clat2C_{\mathrm{lat}} channels to two logits per target bit, and uses 64-channel residual blocks with two 3×33 \times3 convolutions, residual addition, and GELU [52]. With the tokenizer frozen, it is trained to convergence using only Sec. S2 (weight 1; image-domain losses disabled), seed 2025.

All tokenizers use EMA weights and three stages. Stage 1 jointly trains encoder and decoder for 40,000 iterations on mixed PAN, upsampled LR spectral, and HR reference images. Stages 2 and 3 initialize from the preceding EMA, freeze the encoder, and refine the decoder for 20,000 iterations each with batch 16 and MSE/Sobel-gradient/SAM weights 1/0.1/0.1; quantizer loss is disabled. Stage 3 ramps bit-flip probability from 10−510^{-5} to 3×10−33 \times10^{-3} over 2,000 iterations and holds it fixed. The stage-3 EMA is final. AdamW uses learning rate 2×10−42 \times10^{-4}, 500 tokenizer warm-up iterations, and weight decay 0.05/0.01 for tokenizer/fusion. No adversarial, perceptual, or LeCAM loss is used. Training uses PyTorch, bfloat16, and one NVIDIA RTX 4090.

Payload is Clat/s2C_{\mathrm{lat}}/s^2. WV3 33× uses 0.5 bit/pixel (0.0625 bppb); WV3 66× and GF2 50× use 0.25 bit/pixel (0.03125 and 0.0625 bppb); Pavia uses 4 bits/pixel, i.e., 64/(16×102)=0.03921664/(16 \times102)=0.039216 bppb and 23.9688×. WV3 33× sets z_channel=512=512. Pavia pads to at most 128 channels while using 102 spectral and 64 latent channels.

Datasets and Evaluation Protocols

WV3 (8 bands) and GF2 (4 bands) come from PanCollection [43],1 and the 102-band ROSIS Pavia Center data come from HyperPanCollection/HSI and Hyper-DSNet [44].2 All use scale factor four. Tab. XI gives every split used in training and evaluation; HR denotes the common spatial size of the reference or upsampled spectral image and PAN, and LR denotes the native spectral size.

Arrays are channel-first (N,C,H,W)(N,C,H,W). The keys gt/GT, lms/LMS, ms/MS, and pan/PAN denote the HR reference, upsampled LR spectral image, native LR spectral image, and single-band PAN; WV3/GF2 use lowercase and Pavia uppercase. Communication accounting uses the native ms/MS. WV3/GF2 reduced-resolution tests follow Wald’s protocol [45]; their GT supports PSNR, SSIM, ERGAS, and SAM, whereas full-resolution tests lack GT. Whenever HQNR, DλD_{\lambda}, and DsD_s are reported, their anchors are the original spectral and PAN observations. Pavia Full retains GT, so both metric groups are available and are averaged over its two scenes. Its MAT variable SS has shape (400,400,102)(400, 400, 102) and is permuted to (102,400,400)(102, 400, 400) before adding the sample dimension.

DatasetSplitNNCCHRLRGT
WV3Train971486416Yes
WV3Val.108086416Yes
WV3Reduced20825664Yes
WV3Full208512128No
GF2Train1980946416Yes
GF2Val.220146416Yes
GF2Reduced20425664Yes
GF2Full204512128No
PaviaTrain15121026416Yes
PaviaVal.1681026416Yes
PaviaFull2102400100Yes

Table XI. Dataset splits and spatial dimensions.

Qualitative Comparisons With All Baselines Under JPEG2000

For clarity, Figure 6 of the main manuscript shows only the best-performing baselines of each dataset. Figure 9, Figure 10, and Figure 11 provide the comparisons with all baselines under the JPEG2000 protocol for the same WV3, GF2, and Pavia scenes.

WV3 reduced-resolution JPEG2000 comparison

Figure 9. WV3 reduced-resolution JPEG2000 comparison. Baselines: 21.26×; TOP: 33×.

GF2 reduced-resolution JPEG2000 comparison

Figure 10. GF2 reduced-resolution JPEG2000 comparison. Baselines: 20.33×; TOP: 50×.

Pavia Full JPEG2000 comparison

Figure 11. Pavia Full JPEG2000 comparison. Baselines: 19.62×; TOP: 23.97×.

Comparisons Under the HEVC Protocol

The main manuscript reports the reduced-resolution WV3 and GF2 HEVC results in Table V and Table VI. Table XII gives the Pavia results, while Figure 12–Figure 14 provide the corresponding visual comparisons. The pixel-domain baselines operate at 6.69×, 8.94×, and 7.79× on WV3, GF2, and Pavia, whereas TOP uses 33×, 50×, and 23.97×, respectively. Metric anchors follow § 12; across all three datasets, TOP provides the strongest quantitative results and preserves clearer structures and more stable spectral appearance.

MethodRatio ↑PSNR ± Std. ↑SSIM ± Std. ↑ERGAS ± Std. ↓HQNR ± Std. ↑DλD_\lambda ± Std. ↓DsD_s ± Std. ↓SAM (deg) ± Std. ↓
BDSD+HEVC6.69×26.53 ± 3.650.60 ± 0.149.56 ± 2.050.31 ± 0.080.50 ± 0.100.39 ± 0.0810.72 ± 2.52
FusionNet+HEVC26.59 ± 3.550.62 ± 0.129.31 ± 2.070.34 ± 0.090.49 ± 0.110.34 ± 0.079.71 ± 2.36
GLP+HEVC26.79 ± 3.580.62 ± 0.139.24 ± 1.970.31 ± 0.070.51 ± 0.090.37 ± 0.079.52 ± 2.30
GS+HEVC27.22 ± 3.530.64 ± 0.128.82 ± 1.910.38 ± 0.100.46 ± 0.110.31 ± 0.069.79 ± 2.36
MF+HEVC26.92 ± 3.580.62 ± 0.139.13 ± 2.000.32 ± 0.090.51 ± 0.100.35 ± 0.079.35 ± 2.26
U2Net+HEVC26.65 ± 3.540.61 ± 0.139.36 ± 2.050.33 ± 0.090.49 ± 0.110.36 ± 0.079.48 ± 2.29
WFANet+HEVC26.24 ± 3.700.59 ± 0.159.82 ± 2.170.27 ± 0.070.55 ± 0.090.41 ± 0.089.65 ± 2.35
ZS-PAN+HEVC26.57 ± 3.440.60 ± 0.149.40 ± 1.850.28 ± 0.090.54 ± 0.110.40 ± 0.089.60 ± 2.32
ADWM+HEVC26.83 ± 3.700.63 ± 0.139.18 ± 2.190.37 ± 0.090.46 ± 0.110.32 ± 0.0710.13 ± 3.12
ARConv+HEVC26.64 ± 3.590.61 ± 0.149.39 ± 2.000.30 ± 0.080.52 ± 0.100.38 ± 0.089.49 ± 2.31
TOP (ours)33×30.32 ± 2.830.82 ± 0.046.00 ± 1.230.76 ± 0.140.19 ± 0.110.08 ± 0.066.00 ± 1.38
TOP (ours)66×28.88 ± 3.060.75 ± 0.077.13 ± 1.480.69 ± 0.130.23 ± 0.110.11 ± 0.056.69 ± 1.56

Table V. WV3 Reduced-Resolution Results Under HEVC (Mean ± Std.).

MethodRatio ↑PSNR ± Std. ↑SSIM ± Std. ↑ERGAS ± Std. ↓HQNR ± Std. ↑DλD_\lambda ± Std. ↓DsD_s ± Std. ↓SAM (deg) ± Std. ↓
BDSD+HEVC8.94×28.03 ± 1.910.70 ± 0.073.77 ± 0.650.43 ± 0.060.40 ± 0.050.29 ± 0.063.95 ± 0.59
FusionNet+HEVC28.68 ± 1.840.73 ± 0.063.47 ± 0.560.50 ± 0.060.37 ± 0.050.21 ± 0.043.85 ± 0.57
GLP+HEVC27.78 ± 1.860.69 ± 0.073.88 ± 0.650.37 ± 0.060.46 ± 0.050.32 ± 0.063.91 ± 0.57
GS+HEVC28.89 ± 1.710.73 ± 0.063.36 ± 0.540.61 ± 0.050.31 ± 0.040.11 ± 0.053.81 ± 0.60
MF+HEVC28.16 ± 1.890.71 ± 0.073.72 ± 0.620.43 ± 0.060.42 ± 0.050.26 ± 0.043.87 ± 0.57
U2Net+HEVC29.13 ± 1.720.72 ± 0.063.25 ± 0.520.61 ± 0.050.29 ± 0.040.14 ± 0.043.55 ± 0.49
WFANet+HEVC28.05 ± 1.910.69 ± 0.073.72 ± 0.610.41 ± 0.050.42 ± 0.050.29 ± 0.053.82 ± 0.56
ADWM+HEVC28.52 ± 1.870.72 ± 0.063.55 ± 0.580.47 ± 0.050.38 ± 0.050.24 ± 0.043.86 ± 0.57
ARConv+HEVC27.50 ± 1.980.67 ± 0.083.98 ± 0.680.34 ± 0.060.47 ± 0.050.36 ± 0.053.91 ± 0.57
ZS-PAN+HEVC27.72 ± 1.850.69 ± 0.073.88 ± 0.640.37 ± 0.050.45 ± 0.050.33 ± 0.053.93 ± 0.57
TOP (ours)50×31.37 ± 1.900.83 ± 0.052.63 ± 0.500.83 ± 0.030.11 ± 0.030.07 ± 0.022.46 ± 0.42

Table VI. GF2 Reduced-Resolution Results Under HEVC (Mean ± Std.).

WV3 reduced-resolution HEVC comparison

Figure 12. WV3 reduced-resolution HEVC comparison. Baselines: 6.69×; TOP: 33×.

MethodRatio ↑\uparrowPSNR ↑\uparrowSSIM ↑\uparrowERGAS ↓\downarrowHQNR ↑\uparrowDλD_{\lambda} ↓\downarrowDsD_s ↓\downarrowSAM (deg) ↓\downarrow
AWLP+HEVC7.79×\times24.470.5710.790.670.160.2013.36
BDSD-PC+HEVC24.280.4911.050.640.160.2412.32
BT-H+HEVC25.110.6710.380.600.330.1112.65
GSA+HEVC26.640.648.740.740.170.1112.22
MF+HEVC24.770.5910.430.700.160.1612.67
PRACS+HEVC25.100.5610.070.730.170.1212.33
uTDSP+HEVC24.500.4411.450.580.160.3113.88
DIP-HyperKite+HEVC25.230.619.940.770.170.0812.61
HyperTransformer+HEVC18.710.1721.710.400.190.5035.16
R-PNN+HEVC24.670.5710.590.710.160.1612.87
DM-ZS+HEVC24.560.6411.950.700.230.0913.60
ρ\rho-PNN+HEVC23.860.5011.630.620.160.2713.36
TOP (ours)23.97×\times29.720.835.910.850.140.026.39

Table XII. PAVIA FULL HEVC RESULTS AVERAGED OVER TWO SCENES.

Pavia Full HEVC comparison

Figure 14. Pavia Full HEVC comparison. Baselines: 7.79×\times (displayed as 8×\times); TOP: 23.97×\times (displayed as 24×\times).

Full-Resolution Validation on WV3 and GF2

Full-resolution HR-MSI references are unavailable, so WV3/GF2 evaluation uses HQNR, DλD_{\lambda}, and DsD_{s} against the

original lossless PAN and LR-MSI. Pixel-domain baselines receive codec-decoded observations, TOP transmits its fu slightly lower DsD_s, but lower HQNR and weaker spectral consistency. Figure 15–18 show the corresponding qualitative trend.

WV3 full-resolution JPEG2000 comparison

Figure 15. WV3 full-resolution JPEG2000 comparison. Baselines: 28.43×\times; TOP: 33×\times and 66×\times.

GF2 reduced-resolution HEVC comparison

Figure 13. GF2 reduced-resolution HEVC comparison. Baselines: 8.94×\times; TOP: 50×\times.

Numerical Results of the Ablation Studies

All ablations use WV3 reduced-resolution data. Table XVI reports (a) backbone, (b)–(c) BSQ payload/quality/complexity, (d) direct packing versus entropy model+rANS on the same latent, and (e) stage-3 robustness. The continuous-latent model is fully retrained with a larger fusion module; its entries use 32-bit latents. Both PSNR columns in (c) use the same 20 test scenes, unlike the validation value in Table III. In (e), Δ\DeltaPSNR is

MethodRatio ↑\uparrowHQNR ↑\uparrowDλD_{\lambda} ↓\downarrowDsD_s ↓\downarrow
JPEG2000
BDSD+JPEG200028.43×\times0.79250.16360.0524
FusionNet+JPEG20000.73360.18390.1011
GLP+JPEG20000.73420.20430.0769
GS+JPEG20000.79830.17560.0322
MF+JPEG20000.75080.19380.0686
U2Net+JPEG20000.71430.18800.1204
WFANet+JPEG20000.64470.22160.1723
ADWM+JPEG20000.73860.18420.0947
ARConv+JPEG20000.67180.21580.1436
ZS-PAN+JPEG20000.70820.20490.1101
TOP (ours, 33×\times)33×\times0.87240.09840.0329
TOP (ours, 66×\times)66×\times0.84810.11840.0389
HEVC
BDSD+HEVC23.64×\times0.38210.41820.3502
FusionNet+HEVC0.39370.41980.3236
GLP+HEVC0.37910.44660.3175
GS+HEVC0.54200.33000.1939
MF+HEVC0.40070.42860.3033
U2Net+HEVC0.41080.40660.3111
WFANet+HEVC0.33400.46370.3826
ADWM+HEVC0.43830.38350.2900
ARConv+HEVC0.35500.45100.3583
ZS-PAN+HEVC0.37950.43820.3309
TOP (ours, 33×\times)33×\times0.87240.09840.0329
TOP (ours, 66×\times)66×\times0.84810.11840.0389

Table XIII. WV3 FULL-RESOLUTION RESULTS UNDER BOTH CODECS.

MethodRatio ↑\uparrowHQNR ↑\uparrowDλ↓D_{\lambda} \downarrowDs↓D_{s} \downarrow
JPEG2000
BDSD+JPEG200027.46×\times0.82780.13660.0416
FusionNet+JPEG20000.83040.13660.0382
GLP+JPEG20000.79590.17260.0383
GS+JPEG20000.71120.17870.1344
MF+JPEG20000.79770.16890.0406
U2Net+JPEG20000.79990.13010.0800
WFANet+JPEG20000.81500.15350.0374
ADWM+JPEG20000.83110.13820.0355
ARConv+JPEG20000.72980.18460.1048
ZS-PAN+JPEG20000.80940.15970.0367
TOP (ours)50×\times0.85110.10310.0521
HEVC
BDSD+HEVC31.69×\times0.51650.32480.2365
FusionNet+HEVC0.57040.30480.1801
GLP+HEVC0.44970.38410.2709
GS+HEVC0.66060.29050.0694
MF+HEVC0.49470.36160.2258
U2Net+HEVC0.68190.24530.0971
WFANet+HEVC0.48610.34940.2536
ADWM+HEVC0.54990.31640.1964
ARConv+HEVC0.41520.39780.3116
ZS-PAN+HEVC0.47480.35920.2600
TOP (ours)50×\times0.85110.10310.0521

Table XIV. GF2 FULL-RESOLUTION RESULTS UNDER BOTH CODECS.

WV3 full-resolution HEVC comparison

Figure 16. WV3 full-resolution HEVC comparison. Baselines: 23.64×\times; TOP: 33×\times and 66×\times.

GF2 full-resolution JPEG2000 comparison

Figure 17. GF2 full-resolution JPEG2000 comparison. Baselines: 27.46×\times; TOP: 50×\times.

no-flip minus perturbed PSNR and Δ\DeltaSAM is perturbed minus no-flip SAM; lower is better.

Panel (a) isolates the backbone choice. Swin improves reconstruction PSNR by 1.50 dB over CNN and 1.87 dB over ViT, while its 171.72 GFLOPs remain close to CNN and below ViT; the SSIM increase from 0.75 to 0.83 confirms better structural preservation. It therefore provides the strongest representation without using the most expensive candidate.

Panels (b)–(c) clarify the BSQ trade-off. Removing BSQ raises reconstruction and fusion PSNR, but changes each latent entry from one to 32 bits and requires about 31.7×\times the fusion parameters and 31.9×\times the GFLOPs. Its reconstruction–fusion gap also grows from 1.27 to 2.59 dB. BSQ therefore sacrifices some reconstruction ceiling for a target that is much smaller and easier for the onboard module to predict.

Panels (d)–(e) address the remaining communication risks. Entropy modeling saves only 3.58% bppb while adding 10.33M parameters and 74.77 GFLOPs per sample, supporting direct packing. Stage-3 training reduces the PSNR loss under 3% and 5% flips by 76.5% and 68.9%, and leaves SAM nearly unchanged. The small negative Δ\DeltaSAM values reflect sample-level variation rather than a benefit from corruption; the meaningful result is the absence of systematic spectral degradation. The three components consequently have distinct roles: BSQ controls payload and prediction complexity, direct packing avoids disproportionate sender overhead, and decoder refinement limits residual bit errors.

This comparison tests whether moving conventional pansharpening on board is sufficient without redesigning the transmitted representation. Each baseline fuses the uncompressed PAN and LR-MSI into a full-resolution WV3 MSI and then applies JPEG2000 or HEVC; TOP predicts and bit-packs one fused latent. All methods use the same 20 scenes and ratio definition, so their payloads are measured against the same native observations.

Table XV reports every quality metric and measured fusion time. Against JPEG2000, TOP improves the strongest baseline PSNR by 1.08 dB at 1.65×\times its compression ratio, while raising the best HQNR from 0.6356 to 0.7552. Against HEVC, it improves the strongest PSNR by 1.66 dB at about 4.98×\times the ratio. Its lower DλD_{\lambda} and DsD_{s} under both codecs show that the rate gain does not trade spectral consistency for spatial sharpness.

MethodRatio ↑Fusion time (ms/img) ↓HQNR ↑DλD_{\lambda} ↓DsD_s ↓PSNR ↑SSIM ↑ERGAS ↓SAM (deg) ↓
JPEG2000 Codec
WFANet20.00×185.6240.5847 ± 0.09890.2622 ± 0.10460.2103 ± 0.036529.2451 ± 3.98920.6964 ± 0.12307.1451 ± 1.48328.0026 ± 1.8765
U2Net20.03×114.4270.5830 ± 0.09990.2658 ± 0.10570.2088 ± 0.036729.2254 ± 3.98890.6960 ± 0.12307.1554 ± 1.48128.0102 ± 1.8692
FusionNet19.99×1.4440.5824 ± 0.09940.2691 ± 0.10640.2057 ± 0.036729.2050 ± 3.92940.6961 ± 0.12237.1534 ± 1.47577.9752 ± 1.9140
ADWM19.97×47.6650.5794 ± 0.09820.2676 ± 0.10350.2116 ± 0.039829.1645 ± 3.93830.6947 ± 0.12387.1916 ± 1.48418.1475 ± 1.8269
ARConv20.01×80.7100.5879 ± 0.10010.2655 ± 0.10650.2024 ± 0.036029.1799 ± 3.93590.6952 ± 0.12317.1642 ± 1.47387.8361 ± 1.8401
GS20.05×-0.5464 ± 0.08670.2990 ± 0.10640.2197 ± 0.047528.4811 ± 3.74610.6658 ± 0.13067.7017 ± 1.59867.6836 ± 1.9582
BDSD19.98×-0.5857 ± 0.09750.2670 ± 0.11300.2020 ± 0.030828.6829 ± 3.74660.6830 ± 0.11927.5294 ± 1.60087.9723 ± 1.8917
GLP19.97×-0.6212 ± 0.09390.2483 ± 0.10120.1747 ± 0.039928.9107 ± 3.78030.6869 ± 0.11967.3476 ± 1.54067.6171 ± 1.8623
MF20.02×-0.6356 ± 0.09220.2277 ± 0.09450.1786 ± 0.043528.7851 ± 3.74370.6848 ± 0.12267.4207 ± 1.52907.6305 ± 1.8427
ZS-PAN20.07×36863.6830.5629 ± 0.10900.3016 ± 0.12670.1942 ± 0.039328.5151 ± 3.56050.6838 ± 0.12167.4629 ± 1.49118.0665 ± 1.8173
TOP33×32.5110.7552 ± 0.14220.1863 ± 0.11450.0789 ± 0.056430.3237 ± 2.83390.8193 ± 0.03876.0036 ± 1.22985.9989 ± 1.3813
HEVC Codec
WFANet6.65×185.6240.5355 ± 0.13690.3004 ± 0.13210.2437 ± 0.058628.6397 ± 3.44630.6983 ± 0.09657.3943 ± 1.47078.4511 ± 1.8924
U2Net6.68×114.4270.5267 ± 0.13480.3073 ± 0.13220.2484 ± 0.057528.5693 ± 3.48320.6952 ± 0.09927.4309 ± 1.48278.5067 ± 1.9079
FusionNet6.66×1.4440.5281 ± 0.14150.3115 ± 0.14060.2423 ± 0.057728.5932 ± 3.40800.6971 ± 0.09657.4091 ± 1.45958.4481 ± 1.8467
ADWM6.68×47.6650.5232 ± 0.12900.3083 ± 0.12690.2515 ± 0.058428.5146 ± 3.48340.6923 ± 0.10287.4620 ± 1.42818.6357 ± 1.8846
ARConv6.63×80.7100.5424 ± 0.14230.3012 ± 0.13660.2332 ± 0.059628.6597 ± 3.38660.6995 ± 0.09647.3682 ± 1.45398.2661 ± 1.7866
GS6.65×-0.5099 ± 0.12860.3311 ± 0.13990.2440 ± 0.053028.1831 ± 3.40050.6690 ± 0.10477.8659 ± 1.47077.8659 ± 1.9401
BDSD6.60×-0.5598 ± 0.14530.2814 ± 0.13930.2311 ± 0.061128.4264 ± 3.32640.6910 ± 0.09007.6359 ± 1.68368.3336 ± 1.8504
GLP6.60×-0.5822 ± 0.15290.2744 ± 0.13990.2085 ± 0.064328.6502 ± 3.32780.6950 ± 0.09047.4536 ± 1.60268.0500 ± 1.8544
MF6.62×-0.5881 ± 0.14750.2635 ± 0.13390.2146 ± 0.066728.4843 ± 3.39150.6858 ± 0.09487.5802 ± 1.63038.1932 ± 1.8319
ZS-PAN6.66×36863.6830.5327 ± 0.14240.3137 ± 0.13840.2329 ± 0.063428.2744 ± 3.20000.6888 ± 0.09567.6057 ± 1.56308.5449 ± 1.8477
TOP33×32.5110.7552 ± 0.14220.1863 ± 0.11450.0789 ± 0.056430.3237 ± 2.83390.8193 ± 0.03876.0036 ± 1.22985.9989 ± 1.3813

Table XV. Onboard pansharpening followed by JPEG2000/HEVC on WV3. Full-reference metrics use GT; HQNR terms use the original LR-MSI/PAN. Dashes denote omitted CPU-only fusion times.

The payload difference follows from where compression occurs. After pixel-domain fusion, a baseline must represent all CHWCHW samples of the full-resolution MSI, and its final size depends on how strongly a generic codec can quantize that image. TOP instead transmits ClatHW/s2C_{\mathrm{lat}}HW/s^{2} binary decisions, whose size is fixed before observing image content. For WV3 at 33×, this is 0.5 bit per spatial pixel, or 0.0625 bppb across eight bands. The comparison therefore evaluates two deployment representations, rather than only two fusion networks.

TOP has the same row in both codec blocks because it uses neither codec. The changing baseline rows show that onboard pixel-domain fusion still exposes the final product to a separately optimized lossy stage. The timing column measures fusion inference rather than complete encode–transmit–decode latency; among the reported neural baselines, only FusionNet is faster than TOP, while traditional CPU timings are omitted. Thus, the table supports the rate–quality advantage and moderate onboard fusion cost, but does not claim hardware-independent end-to-end latency or energy.

Limitations

a) Sensor-Specific Training: The band-adaptive embedding accepts different band counts, but dimensions and learned parameters remain configuration-specific. Each WV3, GF2,

and Pavia setting in Tab. X therefore uses its own tokenizer and fusion module. A new sensor, or a ratio requiring a different latent configuration, requires one-time ground-side retraining without additional onboard computation. A sensor-agnostic tokenizer remains open.

b) Multi-Stage Training: TOBT uses 40,000 joint encoder–decoder iterations, two 20,000-iteration decoder-refinement stages (the last with bit flips), and separate fusion training (Tab. X). These stages learn the latent space, adapt decoding to the transmitted HR code, and improve bit-error robustness, as supported by Sec. V-D. The added training and tuning effort motivates a simpler future procedure.

References

  1. [1]L.-J. Deng, G. Vivone, W. Guo, M. Dalla Mura, and J. Chanussot, “A variational pansharpening approach based on reproducible kernel Hilbert space and Heaviside function,” IEEE Transactions on Image Processing, vol. 27, no. 9, pp. 4330–4344, 2018.
  2. [2]Z.-C. Wu, T.-Z. Huang, L.-J. Deng, J. Huang, J. Chanussot, and G. Vivone, “LRTCFPan: Low-rank tensor completion based framework for pansharpening,” IEEE Transactions on Image Processing, vol. 32, pp. 1640–1655, 2023.DOI
  3. [3]C. A. Laben and B. V. Brower, “Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening,” U.S. Patent 6,011,875, 2000.
  4. [4]A. Garzelli, F. Nencini, and L. Capobianco, “Optimal MMSE pan sharpening of very high resolution multispectral images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 46, no. 1, pp. 228–236, 2008.DOI
  5. [5]B. Aiazzi, L. Alparone, S. Baronti, and A. Garzelli, “Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis,” IEEE Transactions on Geoscience and Remote Sensing, vol. 40, no. 10, pp. 2300–2312, 2002.DOI
  6. [6]L.-J. Deng, G. Vivone, C. Jin, and J. Chanussot, “Detail injection-based deep convolutional neural networks for pansharpening,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 8, pp. 6995–7010, 2021.DOI
  7. [7]S. Peng, C. Guo, X. Wu, and L.-J. Deng, “U2Net: A general framework with spatio-spectral-integrated double U-net for image fusion,” in Proceedings of the ACM International Conference on Multimedia (ACM MM), 2023, pp. 3219–3227.DOI
  8. [8]J. Huang, R. Huang, J. Xu, S. Peng, Y. Duan, and L.-J. Deng, “Wavelet-assisted multi-frequency attention network for pansharpening,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 39, 2025, pp. 3662–3670.
  9. [9]S. Xiang, Q. Liang, and P. Tang, “Task-oriented compression framework for remote sensing satellite data transmission,” IEEE Transactions on Industrial Informatics, vol. 20, no. 3, pp. 3487–3496, 2024.DOI
  10. [10]T. Lv, H. Jiang, and Y. Liu, “STAFuse: Scene-text aggregation-guided composite degradation-robust infrared and visible image fusion,” IEEE Transactions on Image Processing, vol. 35, pp. 8603–8618, 2026.DOI
  11. [11]D. S. Taubman and M. W. Marcellin, “JPEG2000: Standard for interactive imaging,” Proceedings of the IEEE, vol. 90, no. 8, pp. 1336–1357, 2002.DOI
  12. [12]G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the high efficiency video coding (HEVC) standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649–1668, 2012.DOI
  13. [13]J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1802.01436
  14. [14]Z. Liu, Y. Lin, Y. Cao, H. Hu, W. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 10012–10022.arxiv.org/abs/2103.14030
  15. [15]Y. Zhao, Y. Xiong, and P. Krähenbühl, “Image and video tokenization with binary spherical quantization,” in International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2406.07548
  16. [16]J. Duda, “Asymmetric numeral systems: Entropy coding combining speed of Huffman coding with compression rate of arithmetic coding,” arXiv preprint arXiv:1311.2540, 2013.arxiv.org/abs/1311.2540
  17. [17]X. Li, X. Li, L. Tan, H. Li, and T. Ye, “UMCFuse: A unified multiple complex scenes infrared and visible image fusion framework,” IEEE Transactions on Image Processing, vol. 34, pp. 6231–6245, 2025.DOI
  18. [18]J. Luo, W. Ren, X. Gao, and X. Cao, “Multi-exposure image fusion via deformable self-attention,” IEEE Transactions on Image Processing, vol. 32, pp. 1529–1540, 2023.DOI
  19. [19]J. Li, X. Guo, G. Lu, B. Zhang, Y. Xu, F. Wu, and D. Zhang, “DRPL: Deep regression pair learning for multi-focus image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 4816–4831, 2020.DOI
  20. [20]B. Aiazzi, S. Baronti, and M. Selva, “Improving component substitution pansharpening through multivariate regression of MS+Pan data,” IEEE Transactions on Geoscience and Remote Sensing, vol. 45, no. 10, pp. 3230–3239, 2007.DOI
  21. [21]J. Choi, K. Yu, and Y. Kim, “A new adaptive component-substitution-based satellite image fusion by using partial replacement,” IEEE Transactions on Geoscience and Remote Sensing, vol. 49, no. 1, pp. 295–309, 2011.DOI
  22. [22]X. Otazu, M. Gonzalez-Audicana, O. Fors, and J. Nunez, “Introduction of sensor spectral response into image fusion methods: Application to wavelet-based methods,” IEEE Transactions on Geoscience and Remote Sensing, vol. 43, no. 10, pp. 2376–2385, 2005.DOI
  23. [23]R. Restaino, G. Vivone, M. Dalla Mura, and J. Chanussot, “Fusion of multispectral and panchromatic images based on morphological operators,” IEEE Transactions on Image Processing, vol. 25, no. 6, pp. 2882–2895, 2016.DOI
  24. [24]T. Wang, F. Fang, F. Li, and G. Zhang, “High-quality bayesian pansharpening,” IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 227–239, 2019.DOI
  25. [25]H. Lu, Y. Yang, S. Huang, W. Tu, and W. Wan, “A unified pansharpening model based on band-adaptive gradient and detail correction,” IEEE Transactions on Image Processing, vol. 31, pp. 918–933, 2022.
  26. [26]J. Huang, H. Chen, J. Ren, S. Peng, and L.-J. Deng, “A general adaptive dual-level weighting mechanism for remote sensing pansharpening,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7474–7456.arxiv.org/abs/2503.13214
  27. [27]X. Wang, Z. Zheng, J. Shao, Y. Duan, and L.-J. Deng, “Adaptive rectangular convolution for remote sensing pansharpening,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 17872–17881.arxiv.org/abs/2503.00467
  28. [28]B. Xu, Y. Zheng, L. Wang, S. Mazhar, Z. Huang, and J. Liu, “DTEA: Degradation-aware taylor expansion approximation network for pansharpening,” IEEE Transactions on Image Processing, vol. 35, pp. 7581–7596, 2026.DOI
  29. [29]Q. Cao, L.-J. Deng, W. Wang, J. Hou, and G. Vivone, “Zero-shot semi-supervised learning for pansharpening,” Information Fusion, vol. 101, p. 102001, 2024.DOI
  30. [30]W. G. C. Bandara and V. M. Patel, “HyperTransformer: A textural and spectral feature fusion transformer for pansharpening,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 1767–1777.arxiv.org/abs/2203.02503
  31. [31]N. Wang, A. Guo, R. Dian, and S. Li, “Equivariant high-resolution hyperspectral imaging via mosaiced and PAN image fusion,” IEEE Transactions on Image Processing, vol. 35, pp. 1246–1260, 2026.DOI
  32. [32]W. G. C. Bandara, J. M. J. Valanarasu, and V. M. Patel, “Hyperspectral pansharpening based on improved deep image prior and residual reconstruction,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–16, 2022.arxiv.org/abs/2107.02630
  33. [33]G. Guarino, M. Ciotola, G. Vivone, and G. Scarpa, “Band-wise hyperspectral image pansharpening using CNN model propagation,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–18, 2024.arxiv.org/abs/2311.06510
  34. [34]G. Guarino, M. Ciotola, G. Vivone, G. Poggi, and G. Scarpa, “Zero-shot hyperspectral pansharpening using hysteresis-based tuning for spectral quality control,” IEEE Transactions on Geoscience and Remote Sensing, vol. 63, pp. 1–19, 2025.DOI
  35. [35]H. Jiang and Z. Chen, “Hyperspectral pansharpening with transformer-based spectral diffusion priors,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, 2025, pp. 581–590.DOI
  36. [36]J.-L. Xiao, T.-Z. Huang, L.-J. Deng, G. Lin, Z. Cao, C. Li, and Q. Zhao, “Hyperspectral pansharpening via diffusion models with iteratively zero-shot guidance,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 12669–12678.DOI
  37. [37]M. Hernández-Cabronero, A. B. Kiely, M. Klimesh, I. Blanes, J. Ligo, E. Magli, and J. Serra-Sagristà, “The CCSDS 123.0-B-2 “Low-complexity lossless and near-lossless multispectral and hyperspectral image compression” standard: A comprehensive review,” IEEE Geoscience and Remote Sensing Magazine, vol. 9, no. 4, pp. 102–119, 2021.
  38. [38]A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in Advances in Neural Information Processing Systems, vol. 30, 2017.arxiv.org/abs/1711.00937
  39. [39]F. Mentzer, D. Minnen, E. Augustsson, and M. Tschannen, “Finite scalar quantization: VQ-VAE made simple,” in International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2309.15505
  40. [40]L. Yu, J. Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y. Cheng, A. Gupta, X. Gu, A. G. Hauptmann, B. Gong, M.-H. Yang, I. Essa, D. A. Ross, and L. Jiang, “Learner: A neural network-based entropy model beats diffusion—tokenizer is key to visual generation,” in International Conference on Learning Representations (ICLR), 2024.
  41. [41]A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations (ICLR), 2021.
  42. [42]F. A. Kruse, A. B. Lefkoff, J. W. Boardman, K. B. Heidebrecht, A. T. Shapiro, P. J. Barloon, and A. F. H. Goetz, “The spectral image processing system (SIPS)—interactive visualization and analysis of imaging spectrometer data,” Remote Sensing of Environment, vol. 44, no. 2–3, pp. 145–163, 1993.
  43. [43]L.-J. Deng, G. Vivone, M. E. Paoletti, G. Scarpa, J. He, Y. Zhang, J. Chanussot, and A. Plaza, “Machine learning in pansharpening: A benchmark, from shallow to deep networks,” IEEE Geoscience and Remote Sensing Magazine, vol. 10, no. 3, pp. 279–315, 2022.DOI
  44. [44]Y.-W. Zhuo, T.-J. Zhang, J.-F. Hu, H.-X. Dou, T.-Z. Huang, and L.-J. Deng, “A deep-shallow fusion network with multidetail extractor and spectral attention for hyperspectral pansharpening,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 7539–7555, 2022.DOI
  45. [45]L. Wald, T. Ranchin, and M. Mangolini, “Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images,” Photogrammetric Engineering & Remote Sensing, vol. 63, no. 6, pp. 691–699, 1997.
  46. [46]G. Vivone, “Robust band-dependent spatial-detail approaches for panchromatic sharpening,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 9, pp. 6421–6433, 2019.DOI
  47. [47]S. Lolli, L. Alparone, A. Garzelli, and G. Vivone, “Haze correction for contrast-based multispectral pansharpening,” IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 12, pp. 2255–2259, 2017.DOI
  48. [48]Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
  49. [49]G. Vivone, L. Alparone, J. Chanussot, M. Dalla Mura, A. Garzelli, G. A. Licciardi, R. Restaino, and L. Wald, “A critical comparison among pansharpening algorithms,” IEEE Transactions on Geoscience and Remote Sensing, vol. 53, no. 5, pp. 2565–2586, 2015.DOI
  50. [50]B. Aiazzi, L. Alparone, S. Baronti, R. Carlà, A. Garzelli, and L. Santurri, “Full scale assessment of pansharpening methods and data products,” in Image and Signal Processing for Remote Sensing XX, vol. 9244. SPIE, 2014, p. 924402.DOI
  51. [51]J. Lu, L. Zhang, X. Zhou, M. Li, W. Li, and S. Gu, “Learned image compression with dictionary-based entropy model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 12850–12859.arxiv.org/abs/2504.00496
  52. [52]D. Hendrycks and K. Gimpel, "Gaussian error linear units (GELUs)," arXiv preprint arXiv:1606.08415, 2016.

Paper details

vlnc:W_y2krtga7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents