betaFeedback

WorksComputer Science

Anchoring with Visual Evidence: Explicit Semantic Anchors for Joint Change Detection and Captioning

Abstract

Joint change detection and captioning seeks to localize changes and describe them from bi-temporal remote sensing images. A key challenge is translating dense visual evidence into the semantic conditions that guide description generation. We propose Explicit Semantic Anchors (ESA), a supervised interface that exposes entity presence, temporal appearance variation, and spatial extent. A prediction-guided readout combines visual features and predicted change probabilities, and three factor heads estimate the corresponding anchor activations. These activations guide generation through two complementary routes. The context route maps the three factor activations to three separate memory tokens, one per factor, and appends them to the dense visual memory, while the lexicon route produces an additive vocabulary bias at each decoding step. Dense visual memory remains available for spatial detail and information beyond the explicit factors. Anchor targets are derived from existing images and change annotations, requiring no additional anchor-level language annotation. At inference, all conditions are computed from the input image pair and the model's own predictions. Experiments on LEVIR-MCI and WHU-CDC show strong joint detection and captioning performance, supporting the use of supervised visual attributes as explicit conditions for change description generation.

Introduction

A change map and a change description offer complementary accounts of how a scene evolves. In remote sensing, change detection (CD) localizes changes at the pixel level, whereas change captioning (CC) expresses their content in natural language. Multi-level change interpretation (MCI) brings these tasks into a common framework [14], combining spatially precise predictions with descriptions that people can readily interpret. Realizing this complementarity requires more than producing two outputs from the same image pair: the description should be informed by the visual evidence underlying the change map. This motivates a central question: how can a joint model turn dense change evidence into explicit semantic conditions for language generation?

The challenge lies in the different decisions required by the two tasks. Detection assigns labels to spatial locations, while captioning selects a compact set of attributes and organizes them into a sentence. Even a category-aware change map does not uniquely determine a description. A caption may need to identify the affected entities, interpret temporal appearance differences, and distinguish a localized alteration from a more extensive transformation. Dense bi-temporal features contain evidence for these decisions, but their presence in a decoder’s visual memory does not expose which attributes have been inferred or how they influence word selection. An explicit intermediate representation would make part of this mapping available for direct supervision and inspection.

Recent work has strengthened the connection between visual perception and language through shared representations and task interaction. Joint models such as ChangeAgent and ChangeMinds support detection and captioning within a common architecture [14, 21]. LCCC introduces critique and correction between the two tasks [10], while CoDeRNet selectively transfers features according to taskspecific support and reliability cues [6]. In change captioning, HiSem explores hierarchical semantic organization [20], and STAND introduces semantic anchoring constraints [8]. These approaches motivate a complementary question about the content of semantic guidance: which attributes can be grounded in the available visual supervision, and where should their predictions enter the captioning process? We address this question through a supervised interface between predicted change regions and language generation.

Our starting point is that the visual learning problem already provides useful supervision for such an interface. Change masks identify affected regions and, when semantic labels are available, their categories. The ordered image pair provides appearance differences, and mask statistics quantify the fraction of the scene that has changed. Masks Can Talk demonstrates the value of deriving structured textual supervision from existing change annotations [23]. We use this available structure to supervise intermediate visual attributes that can subsequently condition a caption decoder. This reuses the existing images and masks without requiring additional anchor-level language annotations.

We propose Explicit Semantic Anchors (ESA), an interface that reads predicted change evidence and exposes three complementary factors: entity presence, temporal appearance variation, and spatial extent. A shared visual pathway produces multi-scale bi-temporal features and change probabilities. A prediction-guided readout then combines global differences, category-conditioned features, ordered temporal evidence, and region statistics. Three supervised heads map this evidence to named factor states. Entity states indicate which supported categories are present; temporal states provide a weak appearance-polarity cue; and extent states describe changed-area coverage. Their definitions make the predicted conditions inspectable before decoding. The temporal factor is supervised by intensity variation within changed regions, so it serves as an appearance cue rather than a direct label for events such as construction or demolition.

ESA reuses the same factor activations through a context route and a lexicon route. The context route converts the three factor activations into three separate memory tokens and appends them to the dense visual memory. This allows the decoder to attend to the predicted attributes while constructing a sentence. The lexicon route maps the same activations to an additive vocabulary bias, providing an image-dependent influence on lexical scores at every decoding step. The two routes therefore expose the same evidence at complementary points in generation: contextual decoding and word selection. Dense visual memory remains available for spatial detail and content beyond the three factors. Standard caption supervision learns how these conditions should inform the generated description.

During training, auxiliary targets are derived from the existing visual annotations and image attributes, whereas the anchor readout always receives features and predicted change probabilities. At inference, the image pair alone supplies the change map, anchor activations, and autoregressively generated caption. Experiments on LEVIR-MCI and WHU-CDC evaluate the resulting model through detection accuracy, captioning metrics, and qualitative comparisons. They show strong joint performance, with the relative advantages varying across datasets and metrics.

Our contributions are threefold:

  • We introduce an explicit semantic interface for MCI that organizes prediction-guided visual evidence into supervised entity, temporal-appearance, and spatial-extent factors, making these intermediate conditions available for inspection.

  • We design a context route and a lexicon route that reuse the same anchor activations as three separate memory tokens and an additive vocabulary bias, connecting visual attributes to both contextual decoding and lexical selection.

  • We derive anchor supervision from existing images and change annotations, enabling image-only inference without additional anchor-level language annotation, and evaluate the framework on two joint detection–captioning benchmarks.

Related work

Remote Sensing Change Captioning

Remote sensing change captioning (CC) translates changes between bi-temporal images into natural-language descriptions. A major line of research improves the encoding of temporal differences for language generation. RSIC-CFormer [12] introduced the LEVIR-CC benchmark and a dual-branch Transformer, while Chg2Cap [3] combines hierarchical attention with difference-aware feature refinement. Beyond visual difference encoding, PromptCC [13] separates change-existence classification from caption generation and connects visual representations to a pretrained language model through multiple prompts. HiSem [20] further organizes change semantics through discrepancy-aware interaction, image-level routing, and token-level mixture-of-experts modeling.

Another line strengthens the spatial or semantic evidence used for description. MaskApproxNet [17] estimates change masks through conditional diffusion and applies frequency-guided feature fusion before autoregressive caption decoding. For building-damage captioning, KGBDCNet [19] combines keyword-guided cross-modal retrieval with spatiotemporal feature modeling to emphasize damage-related content. These approaches illustrate how change recognition, spatial cues, and semantic guidance support caption generation. CC evaluates change understanding through language. Joint MCI additionally requires explicit spatial predictions and evaluation of both outputs.

Remote Sensing Change Detection

Remote sensing change detection (CD) predicts spatially explicit changes from multi-temporal imagery. Early fully convolutional Siamese networks [7] fuse features from shared-weight temporal encoders through concatenation or absolute differences in skip connections. To incorporate broader context, BIT [4] models interactions among compact semantic tokens and projects their context back to dense image features. ChangeFormer [1] instead combines a hierarchical Siamese Transformer encoder with an MLP decoder for multiscale change prediction. Extending this architectural progression, ChangeMamba [5] introduces spatiotemporal state-space modeling for binary and semantic change detection as well as building damage assessment.

Recent work also expands the semantic scope of detection. OpenDPR [9] addresses open-vocabulary CD by combining foundation-model change proposals with diffusion-guided visual prototype retrieval for category identification. Thus, CD encompasses both change localization and structured semantic prediction, including change categories or damage levels. These structured outputs complement the natural-language descriptions targeted by CC.

Multi-Level Change Interpretation

Multi-level change interpretation (MCI) unifies CD and CC to jointly predict pixel-level change maps and semantic descriptions. Change-Agent [14] introduced LEVIR-MCI and a joint perception model with bi-temporal iterative interaction, while CTMTNet [16] provided the LEVIR-CDC and WHU-CDC datasets through attention-based enhancement and bi-temporal fusion. Building on these formulations, ChangeMinds [21] uses ChangeLSTM and a unified decoder to serve both heads, and Semantic-CC [24] employs a SAM-based encoder with pixel-level guidance coordinated through a three-stage training strategy, connecting language generation to spatially grounded representations. Recent methods further address this coupling with distinct mechanisms. Change-UP [22] integrates adaptive representations and text-driven style diversification, LCCC [10] couples detection-derived change tokens with a critique–correction framework, and CoDeRNet [6] selectively routes features based on receiver-side support needs and sender-side reliability cues. A central issue across these designs is how to exchange complementary task evidence while preserving the spatial detail required by CD and the semantic selectivity required by CC.

Method

Problem Formulation and Overview

Given co-registered images X1,X2∈R3×H×WX_1, X_2 \in\mathbb{R}^{3 \times H \times W}, MCI predicts a change map Y^\hat{Y} and a caption w=(w1,…,wL)w = (w_1, \ldots, w_L). The detection ontology contains CC classes, including background indexed by 0. Training provides a change map YY and reference captions R\mathcal{R}. Figure 1 summarizes the framework. A shared visual pathway supports both outputs, while the ESA pathway reads the predicted visual evidence and supplies explicit conditions to the caption decoder. Ground-truth annotations construct auxiliary targets during training; the forward-pass anchor inputs always use model features and predictions.

Overview of Explicit Semantic Anchors (ESA)

Figure 1. Overview of Explicit Semantic Anchors (ESA). A shared visual pathway predicts the change map and supplies dense visual memory. A prediction-guided, stop-gradient readout exposes change evidence to three factor heads for entity presence, temporal appearance variation, and spatial extent. The context route maps the three factor activations to three factor-specific tokens and appends them, separately and without aggregation, to the dense visual memory; the decoder can attend to each factor independently. The lexicon route maps the same activations to an additive bias on the output vocabulary logits before the softmax. Dashed arrows indicate auxiliary supervision derived from existing masks and image intensities during training; inference requires only the image pair.

ESA exposes three interpretable factors: entity presence, temporal appearance variation, and spatial extent. These factors summarize complementary attributes that can inform a change description. Their auxiliary targets are derived from the existing masks and ordered images, without parsing captions or collecting additional anchor-level language annotations. The resulting activations condition caption generation through a context route and a lexicon route, while dense visual memory remains available for details beyond these global factors.

Multi-scale Bi-temporal Evidence

Shared encoding. A hierarchical backbone EE extracts four feature stages at strides 4, 8, 16, and 32. A stage-specific 1×11 \times1 convolution πs\pi_s projects each stage to a common channel width dd, followed by layer normalization:

Fis=LN(πs(Es(Xi))),(1)F_i^s = \mathrm{LN}\bigl(\pi_s(E_s(X_i))\bigr), \tag*{(1)}

where i∈{1,2}i \in\{1,2\} and s∈{1,2,3,4}s \in\{1,2,3,4\}. The backbone and projection parameters are shared between observations; s=4s=4 denotes the coarsest grid.

Change aggregation and enhancement. We retain both absolute and ordered differences, As=∣F1s−F2s∣A^s = \lvert F_1^s - F_2^s\rvert and Rs=F1s−F2sR^s = F_1^s - F_2^s. A convolutional gate modulates a residual on the later observation:

Gs=σ(ϕgs([F1s;F2s;As;Rs])),(2)G^s = \sigma\bigl(\phi_g^s([F_1^s; F_2^s; A^s; R^s])\bigr), \tag*{(2)}
F~1s=F1s,F~2s=F2s+ϕas(As)⊙Gs.(3)\widetilde{F}_1^s = F_1^s,\qquad\widetilde{F}_2^s = F_2^s + \phi_a^s(A^s) \odot G^s. \tag*{(3)}

Here [ ; ][\,;\,] denotes channel concatenation, ϕgs\phi_g^s is a 1×11 \times1 convolution, and ϕas\phi_a^s is a 3×33 \times3 convolution block with batch normalization and ReLU. The earlier observation provides a reference, while the later feature receives difference-conditioned information. The learned residual is not constrained to increase the distance between the two features. Multi-scale change descriptors are also aggregated through a top-down recursion:

C4=ϕc4(A4),Cs=ϕcs(As)+Up⁡(ηs(Cs+1)),(4)\begin{aligned} C^{4} &= \phi_{c}^{4}(A^{4}),\\ C^{s} &= \phi_{c}^{s}(A^{s}) + \operatorname{Up}(\eta_{s}(C^{s+1})), \tag*{(4)} \end{aligned}

for s=3,2,1s = 3, 2, 1, where ηs\eta_s is a 1×11 \times1 convolution that matches the channel width before upsampling.

Dense change prediction. At each scale, a convolutional block encodes the ordered enhanced pair and its differences:

Ds=ϕds([F~1s;F~2s;∣F~1s−F~2s∣;F~1s−F~2s]).(5)D^{s} = \phi_{d}^{s}\left([\widetilde{F}_{1}^{s};\widetilde{F}_{2}^{s};|\widetilde{F}_{1}^{s}-\widetilde{F}_{2}^{s}|;\widetilde{F}_{1}^{s}-\widetilde{F}_{2}^{s}]\right). \tag*{(5)}

A top-down decoder fuses these features with Q4=D4Q^{4}=D^{4} and Qs=ψs(Ds+Up⁡(Qs+1))Q^{s}=\psi_{s}(D^{s}+\operatorname{Up}(Q^{s+1})). A 1×11 \times1 classifier followed by resizing produces logits Zcd∈RC×H×WZ_{\mathrm{cd}} \in\mathbb{R}^{C \times H \times W}. The probability map and final prediction are

P=softmax⁡(Zcd),P=\operatorname{softmax}(Z_{\mathrm{cd}}),
Y^(u)=arg⁡max⁡c∈{0,…,C−1} Pc(u).(6)\widehat{Y}(u)=\underset{c\in\{0,\ldots,C-1\}}{\arg\max}\ P_{c}(u). \tag*{(6)}

The maximization includes background. Ordered features preserve temporal information that absolute differences alone discard.

Explicit Semantic Anchors

Prediction-guided evidence readout. The anchor readout combines global differences, category-conditioned content, ordered temporal evidence, and change coverage. We resize PP to the coarsest grid, obtaining P‾c=\overline{P}_{c}= ResizeH4,W4(Pc)_{H_4,W_4}(P_c) and Pˉfg=∑c=1C−1Pˉc\bar{P}_{\mathrm{fg}}=\sum_{c=1}^{C-1}\bar{P}_c. Normalized weighted pooling is defined as

Pool⁡(Z,q)=∑uq(u)Z(u)∑uq(u)+ϵ,(7)\operatorname{Pool}(Z,q)=\frac{\sum_u q(u)Z(u)}{\sum_u q(u)+\epsilon}, \tag*{(7)}

where ϵ>0\epsilon>0 stabilizes the denominator. Since pooling normalizes by the total mask weight, area statistics are supplied separately. The evidence components are

vg=GAP⁡(D4),vc=Pool⁡(D4,Pˉc),(8)v_g=\operatorname{GAP}(D^4),\qquad v_c=\operatorname{Pool}(D^4,\bar{P}_c), \tag*{(8)}
vT=Pool⁡(F~24−F~14,Pˉfg),(9)v_T=\operatorname{Pool}(\tilde{F}^{4}_2-\tilde{F}^{4}_1,\bar{P}_{\mathrm{fg}}), \tag*{(9)}
ac=1HW∑uPc(u),afg=∑c=1C−1ac.(10)a_c=\frac{1}{HW}\sum_u P_c(u),\qquad a_{\mathrm{fg}}=\sum_{c=1}^{C-1}a_c. \tag*{(10)}

The index cc in Eqs. (8) and (10) ranges over foreground categories. We concatenate the components and detach their visual inputs:

e=sg⁡([vg;v1;…;vC−1;vT;a1;…;aC−1;afg]).(11)e=\operatorname{sg}\left([v_g;v_1;\ldots;v_{C-1};v_T; a_1;\ldots;a_{C-1};a_{\mathrm{fg}}]\right). \tag*{(11)}

For descriptor width dd, the vector has dimension (C+1)d+C(C+1)d+C. The temporal readout uses later-minus-earlier features, whereas RsR^s uses the opposite sign convention; both are fixed as defined above. The stop-gradient operation blocks updates to the visual inputs through this auxiliary evidence path. Other paths through the shared visual representation remain trainable.

Factor activations. Three heads HrH_r, each comprising layer normalization, a linear layer, GELU, and a final linear layer, predict the factor activations:

pE=σ(HE(e))∈[0,1]C,pT=softmax⁡(HT(e))∈[0,1]3,pA=softmax⁡(HA(e))∈[0,1]3.(12)\begin{aligned} p_E&=\sigma(H_E(e))\in[0,1]^C,\\ p_T&=\operatorname{softmax}(H_T(e))\in[0,1]^3,\\ p_A&=\operatorname{softmax}(H_A(e))\in[0,1]^3. \tag*{(12)} \end{aligned}

The entity head contains independent indicators for the C−1C-1 foreground categories and a separate no-change indicator. The temporal head encodes neutral, positive, and negative appearance variation; the extent head represents small, medium, and large changed-area fractions. These are global factors, rather than separate event assignments for individual objects. Their named states and auxiliary targets make the predictions inspectable, while learnable readout tables translate them into generation conditions. Entity activations are independent Bernoulli probabilities, so their weighted readout need not be a convex combination.

Reusing Existing Visual Supervision

Auxiliary targets are constructed from the existing visual annotations and image attributes. They require neither a reference-caption parser nor externally generated textual descriptors.

Anchor-conditioned Caption Generation

Entity targets. Foreground presence and the no-change target follow from the detection map:

yE,c=1[∃u:Y(u)=c],c=1,…,C−1,y_{E,c}=\mathbf{1}[\exists u:Y(u)=c],\qquad c=1,\ldots,C-1,
yE,∅=1[∀u:Y(u)=0].(13)y_{E,\varnothing}=\mathbf{1}[\forall u:Y(u)=0]. \tag*{(13)}

The supported entity vocabulary follows the mask ontology. Road and building annotations in LEVIR-MCI support those category labels, whereas a binary mask alone does not supply arbitrary semantic categories.

Temporal-variation targets. Let M(u)=1[Y(u)>0]M(u)=\mathbf{1}[Y(u)>0], and let ℓ(Xi)(u)\ell(X_i)(u) denote the scalar image-intensity representation. We compute the mean intensity difference over changed regions:

δℓ=∑uM(u)[ℓ(X2)(u)−ℓ(X1)(u)]∑uM(u)+ϵ,(14)\delta_\ell=\frac{\sum_u M(u)[\ell(X_2)(u)-\ell(X_1)(u)]}{\sum_u M(u)+\epsilon}, \tag*{(14)}

and assign the three-state target

yT={1,δℓ>τT,2,δℓ<−τT,0,otherwise.(15)y_T= \begin{cases} 1, & \delta_\ell>\tau_T,\\ 2, & \delta_\ell<-\tau_T,\\ 0, & \text{otherwise}. \end{cases} \tag*{(15)}

This target provides a weak photometric-polarity cue. The neutral state includes both unchanged samples and changes with a small mean intensity difference. These labels do not directly supervise construction or demolition; the association between appearance evidence and captioned events is learned through caption supervision.

Extent targets. For changed-area fraction r=(HW)−1∑uM(u)r=(HW)^{-1}\sum_u M(u), we define

yA={0,r≤τS,1,τS<r≤τL,2,r>τL.(16)y_A= \begin{cases} 0, & r\le\tau_S,\\ 1, & \tau_S<r\le\tau_L,\\ 2, & r>\tau_L. \end{cases} \tag*{(16)}

The bins describe spatial coverage, rather than object count or individual object size. The thresholds are dataset-specific hyperparameters selected using training data.

Anchor-conditioned Caption Generation

For each factor r∈{E,T,A}r\in\{E,T,A\}, we learn an embedding table Er∈RKr×dE_r\in\mathbb{R}^{K_r\times d} and a vocabulary table Br∈RKr×∣V∣B_r\in\mathbb{R}^{K_r\times|\mathcal{V}|}, where (KE,KT,KA)=(C,3,3)(K_E,K_T,K_A)=(C,3,3) and V\mathcal{V} is the caption vocabulary. Both routes read the same image-dependent factor activations.

Context route. The embedding tables map the three factor activations to three factor-specific tokens:

tr=prTEr,r∈{E,T,A}.(17)t_r=p_r^\mathsf{T}E_r,\qquad r\in\{E,T,A\}. \tag*{(17)}

Let Mvis=[m1,…,mN]∈RN×dM_{\mathrm{vis}} = [m_1,\ldots,m_N] \in\mathbb{R}^{N \times d} denote the dense visual memory of the caption decoder, inherited from the base captioner [14]: it consists of the flattened features {Xs}s=14\{X^s\}_{s=1}^{4} at the four scales (HsWsH_sW_s tokens at scale ss) followed by the mask-conditioned global token gg, so that N=1+∑s=14HsWsN = 1 + \sum_{s=1}^{4} H_sW_s. The decoder cross-attends over M=[Mvis;tE;tT;tA]M = [M_{\mathrm{vis};t_E;t_T;t_A}]; the three anchor tokens therefore extend the memory from NN to N+3N+3 entries instead of replacing the spatial evidence.

The global token is

g=∑s=14GAP⁡(F^2s⊙sg⁡(P‾fgs))+GAP⁡(A4),(18)g = \sum_{s=1}^{4} \operatorname{GAP}\left(\widehat{F}_2^s \odot\operatorname{sg}\left(\overline{P}_{\mathrm{fg}}^s\right)\right) + \operatorname{GAP}\left(A^4\right), \tag*{(18)}

where P‾fgs\overline{P}_{\mathrm{fg}}^s is the predicted foreground probability resized to the grid of stage ss. The three tokens are kept separate; they are neither summed nor averaged. The context route appends them, as three additional memory entries, to the dense visual memory:

M=[Mvis;tE;tT;tA].(19)M = [M_{\mathrm{vis};t_E;t_T;t_A}]. \tag*{(19)}

The decoder can therefore attend to the entity, temporal, and extent conditions independently. This three-token construction is used in all main experiments.

Lexicon route. The vocabulary tables map the same factors to an additive bias over candidate words:

banc=pETBE+pTTBT+pATBA.(20)b_{\mathrm{anc}} = p_E^{\mathsf{T}}B_E + p_T^{\mathsf{T}}B_T + p_A^{\mathsf{T}}B_A. \tag*{(20)}

The tables are initialized to zero. The bias is added to the decoder logits at each autoregressive step:

p(wt∣w<t,X1,X2)=softmax⁡(Woht+λbbanc).(21)p(w_t \mid w_{<t}, X_1, X_2) = \operatorname{softmax}(W_oh_t + \lambda_b b_{\mathrm{anc}}). \tag*{(21)}

The context route conditions the decoder memory, and the lexicon route directly conditions lexical scores. The readout tables are shared model parameters, while their activations and outputs vary with the input pair. Since dense visual memory remains accessible, the anchors provide an explicit conditioning interface rather than a strict information bottleneck.

Optimization and Inference

Entity supervision uses binary cross-entropy, while temporal variation and spatial extent use categorical cross-entropy:

Lanc=λEBCE⁡(HE(e),yE)+λTCE⁡(HT(e),yT)+λACE⁡(HA(e),yA).(22)\mathcal{L}_{\mathrm{anc}} = \lambda_E \operatorname{BCE}(H_E(e), y_E) + \lambda_T \operatorname{CE}(H_T(e), y_T) + \lambda_A \operatorname{CE}(H_A(e), y_A). \tag*{(22)}

The loss arguments are logits. For multiple reference captions, the caption objective is

LCC=−1∣R∣∑w∈R1∣w∣∑t=1∣w∣log⁡p(wt∣w<t,X1,X2).(23)\mathcal{L}_{\mathrm{CC}} = -\frac{1}{|\mathcal{R}|}\sum_{w \in\mathcal{R}}\frac{1}{|w|}\sum_{t=1}^{|w|}\log p(w_t \mid w_{<t}, X_1, X_2). \tag*{(23)}

If label smoothing is used, soft token targets replace the token-level negative log likelihood. Its coefficient is distinct from the intensity threshold τT\tau_T. The complete objective is

L=LMCI+Lanc,\mathcal{L} = \mathcal{L}_{\mathrm{MCI}} + \mathcal{L}_{\mathrm{anc}},
LMCI=λCD(LCE+λDiceLDice)+λCCLCC,(24)\mathcal{L}_{\mathrm{MCI}} = \lambda_{\mathrm{CD}}(\mathcal{L}_{\mathrm{CE}} + \lambda_{\mathrm{Dice}}\mathcal{L}_{\mathrm{Dice}}) + \lambda_{\mathrm{CC}}\mathcal{L}_{\mathrm{CC}}, \tag*{(24)}

where LCE\mathcal{L}_{\mathrm{CE}} and LDice\mathcal{L}_{\mathrm{Dice}} supervise dense change prediction. The auxiliary losses update the anchor heads but do not reach the visual inputs through Eq. (11). Caption gradients train the heads and readout tables through both routes, and can also update the shared visual pathway through MvisM_{\mathrm{vis}}. Detaching the anchor evidence therefore does not freeze the detector. At inference, the model computes PP, predicts the three factor activations, and generates the caption autoregressively. Ground-truth masks, auxiliary targets, and reference captions are absent. The images are encoded once, followed by the decoding steps needed to generate the sentence.

Experiments

Datasets

We evaluate on LEVIR-MCI [14] and WHU-CDC [16], which provide bi-temporal image pairs, pixel-level change masks, and five reference captions per pair. Both use 256×256256 \times256 images. LEVIR-MCI uses 6,815/1,333/1,929 pairs for training/validation/testing.

Evaluation Metrics

We report mIoU for detection and BLEU-1/2/3/4 [15], METEOR [2], ROUGE-L [11], and CIDEr-D [18] for captioning. The caption summary Sm∗S_m^* averages BLEU-4, METEOR, ROUGE-L, and CIDEr-D; the joint score TS averages mIoU and Sm∗S_m^*. All metrics are higher-is-better.

Quantitative Comparison

The compared methods are grouped into detection-only, captioning-only, and joint CD–CC approaches. The first two groups provide task-specific reference points, while the joint methods permit direct inspection of both outputs.

Results on LEVIR-MCI. Table 1 shows that ESA achieves the highest BLEU-1/2/3/4, METEOR, and CIDEr-D among the compared methods. Its BLEU-4 of 70.57 exceeds the strongest competing value, 66.27 from CoDeRNet, by 4.30 points. The BLEU-1, BLEU-2, and BLEU-3 improvements over CoDeRNet are 2.25, 2.28, and 3.31 points, respectively. ESA also improves CIDEr-D from 143.40 for Change-UP to 145.05 and raises the caption summary from 82.00 to 83.33. These results indicate stronger reference agreement across several complementary captioning measures.

TasksMethodsmIoUBLEU-1BLEU-2BLEU-3BLEU-4METEORROUGE-LCIDEr-DSm∗S_m^*TS
Change DetectionBIT85.10–––––––––
Change DetectionCDxLSTM86.02–––––––––
Change DetectionChangeFormer85.26–––––––––
Change DetectionChangeMamba86.16–––––––––
Change DetectionCDMamba84.48–––––––––
Change DetectionChangeDA85.64–––––––––
Change CaptioningChg2Cap–82.0874.0167.2261.9337.9472.63127.8275.08–
Change CaptioningRSCaMa–85.8377.8070.6664.7240.1675.06136.4079.09–
Change CaptioningPromptCC–83.6675.7369.1063.5438.8273.72136.4478.13–
Change CaptioningPix4Cap–85.5677.0869.7963.7839.9675.12136.7678.91–
Change CaptioningCCExpert–86.6578.4771.3165.4941.8276.55143.3281.80–
Change Detection & CaptioningMCINet86.2886.3378.4771.4265.4840.9175.87140.5380.7083.49
Change Detection & CaptioningChangeMinds83.2583.5872.2863.3355.9639.2277.18133.2676.4179.83
Change Detection & CaptioningFST-Net85.3286.7678.8271.7165.6740.5176.15140.0480.5982.96
Change Detection & CaptioningChange-UP87.0286.8779.1071.9866.2041.8076.61143.4082.0084.51
Change Detection & CaptioningLCCC86.5087.0779.0972.0466.1941.1076.58142.7381.6584.08
Change Detection & CaptioningCoDeRNet87.7987.2579.4272.2666.2741.0876.36141.7981.3884.59
Change Detection & CaptioningOurs87.3889.5081.7075.5770.5743.1074.60145.0583.3385.36

Table 1. Comprehensive performance comparison on LEVIR-MCI. Detection-only methods report mIoU, captioning-only methods report text metrics, and joint CD-CC methods report both. All metrics are higher-is-better. Best and second-best results are highlighted with red and blue backgrounds, respectively.

For detection, ESA obtains 87.38 mIoU, the second-highest value in the table. It improves on MCINet, Change-UP, and LCCC by 1.10, 0.36, and 0.88 points, respectively, while remaining 0.41 points below CoDeRNet. The resulting TS is 85.36, compared with 84.59 for CoDeRNet. The improvements are not uniform: ESA’s ROUGE-L of 74.60 is below the best result of 77.18 from ChangeMinds. Thus, the principal advantage on LEVIR-MCI is a stronger overall captioning profile with competitive localization, rather than dominance of every detection and language metric.

Results on WHU-CDC. Table 2 shows that ESA achieves 94.86 mIoU and 169.97 CIDEr-D on WHU-CDC, the highest values among the compared methods.

TasksMethodsmIoUBLEU-1BLEU-2BLEU-3BLEU-4METEORROUGE-LCIDEr-DSm∗S_m^*TS
Change DetectionBIT89.11–––––––––
Change DetectionCDXLSTM86.66–––––––––
Change DetectionChangeFormer90.29–––––––––
Change DetectionChangeMamba90.12–––––––––
Change CaptioningRSICCFormer–80.0574.2469.6166.5442.6573.91133.4479.14–
Change CaptioningPromptCC–82.6876.9472.2068.9943.7876.14137.1081.50–
Change Detection & CaptioningMCINet91.3680.9774.5468.8563.6943.5078.37148.9483.6387.50
Change Detection & CaptioningChangeMinds89.5581.0977.2474.8973.4846.1383.41144.4086.8688.21
Change Detection & CaptioningFST-Net87.3688.1583.4679.5576.7848.7882.91160.0192.1289.74
Change Detection & CaptioningLCCC93.8389.0384.2680.3577.2950.0383.86164.9994.0493.94
Change Detection & CaptioningCoDeRNet94.2687.3182.4178.3575.1548.9083.00161.3292.0993.18
Change Detection & CaptioningOurs94.8689.9984.6980.2276.5748.9579.27169.9793.6994.28

Table 2. Comprehensive performance comparison on WHU-CDC. Detection-only methods report mIoU, captioning-only methods report text metrics, and joint CD-CC methods report both. All metrics are higher-is-better. Best and second-best results are highlighted with red and blue backgrounds, respectively.

Qualitative Comparison

Figure 2 compares predicted maps and generated descriptions with LCCC and CoDeRNet. In the first example, ESA describes the group of houses built along the roads, capturing the principal changed entity and its spatial arrangement. This description is similar to CoDeRNet’s and does not include every detail in the reference, such as the additional villas at the bottom. The corresponding detection map identifies the repeated building footprints and the road-change region. In the second example, ESA focuses on the newly appearing road at the bottom of the scene, while the magnified crops permit a closer comparison of the thin road segment and its junction. These cases illustrate how the map and sentence provide complementary accounts of the same change.

Visual comparison of joint change detection and captioning results

Figure 2. Visual comparison of joint change detection and captioning results. The top two rows are from LEVIR-MCI, and the bottom two rows are from WHU-CDC. From left to right: pre-change (T1), post-change (T2), ground truth (GT), LCCC, CoDeRNet and Ours. In GT, red and yellow denote building and road changes, respectively. Dashed boxes highlight boundary details.

The building examples expose differences in foreground coverage and boundary shape. In the last example, ESA describes a large building together with a parking lot, and the predicted map delineates the main building-change regions. Such examples support inspection of entity selection, spatial wording, and localization together; they do not establish that every generated attribute is correct. In particular, the global anchor factors do not explicitly assign an event or a count to each object, and the dense visual memory remains necessary for finer spatial and contextual details.

Conclusion

We presented Explicit Semantic Anchors (ESA) for joint remote sensing change detection and captioning. ESA reads features and predicted change regions through a supervised interface that exposes entity presence, temporal appearance variation, and spatial extent. Retaining dense visual memory allows the decoder to combine these explicit conditions with detailed scene evidence. Results on LEVIR-MCI and WHU-CDC demonstrate strong joint performance and support the value of exposing visual attributes to the caption decoder. Entity semantics follow the detection ontology, the temporal target is a weak photometric proxy, and extent measures area coverage rather than individual object change. Future work can extend this interface to object level event representations and evaluate the grounding of individual generated claims, building on ESA’s central principle of exposing and reusing supervised visual evidence within language generation.

References

  1. [1]Wele Gedara Chaminda Bandara and Vishal M. Patel. A transformer-based siamese network for change detection. In IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium, pages 207–210, 2022.arxiv.org/abs/2201.01293
  2. [2]Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72. ACL, 2005.
  3. [3]Shizhen Chang and Pedram Ghamsi. Changes to captions: An attentive network for remote sensing change captioning. IEEE Transactions on Image Processing, 32:6047–6060, 2023.DOI
  4. [4]Hao Chen, Zipeng Qi, and Zhenwei Shi. Remote sensing image change detection with transformers. IEEE Transactions on Geoscience and Remote Sensing, 60:1–14, 2022.DOI
  5. [5]Hongruixuan Chen, Jian Song, Chengxi Han, Junshi Xia, and Naoto Yokoya. ChangeMamba: Remote sensing change detection with spatiotemporal state space model. IEEE Transactions on Geoscience and Remote Sensing, 62:1–20, 2024.DOI
  6. [6]Eunki Cho, Hyeon Bae Kim, and Seong Tae Kim. CoDeRNet: Selective cross-task routing under heterogeneous supervision for change detection and captioning. arXiv preprint arXiv:2610.34876, 2026.arxiv.org/abs/2610.34876
  7. [7]Rodrigo Caye Daudt, Bertrand Le Saux, and Alexandre Boulch. Fully convolutional siamese networks for change detection. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 4063–4067, 2018.arxiv.org/abs/1810.08462
  8. [8]Yangpei Gong, Beichen Zhang, Hao Wang, Zhaobo Qi, Xinyan Liu, Yuanrong Xu, Ruiyang Gao, and Weigang Zhang. STAND: Semantic anchoring constraint with dual-granularity disambiguation for remote sensing image change captioning. arXiv preprint arXiv:2604.23309, 2026.arxiv.org/abs/2604.23309
  9. [9]Qi Guo, Jue Wang, Yinhe Liu, and Yanfei Zhong. OpenPDR: Open-vocabulary change detection via vision-centric diffusion-guided prototype retrieval for remote sensing imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20399–20409, 2026.DOI
  10. [10]Huafeng Li, Yamin Zhang, Yunbin Tu, Yafei Zhang, Wen Wang, and Liang Li. Learning to change by critique and correction: A synergistic framework for remote sensing change detection and captioning. IEEE Transactions on Image Processing, 35:7783–7798, 2026.DOI
  11. [11]Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81. ACL, 2004.
  12. [12]Chenyang Liu, Rui Zhao, Hao Chen, Zhengxia Zou, and Zhenwei Shi. Remote sensing image change captioning with dual-branch transformers: A new method and a large scale dataset. IEEE Transactions on Geoscience and Remote Sensing, 60:1–20, 2022.DOI
  13. [13]Chenyang Liu, Rui Zhao, Jianqi Chen, Zipeng Qi, Zhengxia Zou, and Zhenwei Shi. A decoupling paradigm with prompt learning for remote sensing image change captioning. IEEE Transactions on Geoscience and Remote Sensing, 61:1–18, 2023.DOI
  14. [14]Chenyang Liu, Keyan Chen, Haotian Zhang, Zipeng Qi, Zhengxia Zou, and Zhenwei Shi. Change-Agent: Toward interactive comprehensive remote sensing change interpretation and analysis. IEEE Transactions on Geoscience and Remote Sensing, 62:1–16, 2024.DOI
  15. [15]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. ACL, 2002.
  16. [16]Jingye Shi, Mengge Zhang, Yuewu Hou, Ruicong Zhi, and Jiqiang Liu. A multitask network and two large-scale datasets for change detection and captioning in remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 62:1–17, 2024.DOI
  17. [17]Dongwei Sun, Jing Yao, Wu Xue, Changsheng Zhou, Pedram Ghamisi, and Xiangyong Cao. Mask approximation net: A novel diffusion model approach for remote sensing change captioning. IEEE Transactions on Geoscience and Remote Sensing, 63:1–11, 2025.
  18. [18]Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, 2015.arxiv.org/abs/1411.5726
  19. [19]Di Wang, Guorui Ma, Xiao Wang, Yongxian Zhang, Haiming Zhang, Bin Wang, and Peng Chen. KGB-DCNet: Keyword-guided building damage captioning network for bi-temporal remote sensing images. IS-PRS Journal of Photogrammetry and Remote Sensing, 234:369–385, 2026.
  20. [20]Man Wang, Chenyang Liu, Wenjun Li, Feng Ni, Bing Jia, Baoqi Huang, Riting Xia, and Zhenwei Shi. HiSem: Hierarchical semantic disentangling for remote sensing image change captioning. IEEE Transactions on Geoscience and Remote Sensing, 64:1–17, 2026.
  21. [21]Yuduo Wang, Weikang Yu, Michael Kopp, and Pedram Ghamisi. ChangeMinds: Multi-task framework for detecting and describing changes in remote sensing. arXiv preprint arXiv:2410.10047, 2024.arxiv.org/abs/2410.10047
  22. [22]Mo Yang, Luo Chen, and Jiali Zhou. Change-UP: Advancing visualization and inference capability for multi-level remote sensing change interpretation. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 15–24, 2025.DOI
  23. [23]Kai Zheng, Hang-Cheng Dong, Jiatong Pan, Zhenkai Wu, Fupeng Wei, and Wei Zhang. Macs can talk: Extracting structured text information from single-modal images for remote sensing change detection. arXiv preprint arXiv:2605.07178, 2026.arxiv.org/abs/2605.07178
  24. [24]Yongshuo Zhu, Lu Li, Keyan Chen, Chenyang Liu, Fugen Zhou, and Zhenwei Shi. Semantic-CC: Boosting remote sensing image change captioning via foundational knowledge and semantic guidance. IEEE Transactions on Geoscience and Remote Sensing, 62: 1–16, 2024.arxiv.org/abs/2407.14032

Paper details

vlnc:W_672r3pz9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents