betaFeedback

WorksComputer Science

OCBI: Order-Sensitive Cross-Branch Interaction for Image Manipulation Localization

Abstract

Image manipulation localization combines semantic structure with forensic traces to identify modified pixels. Within-branch spatial mixing and RGB-guided cross-branch mixing can produce different messages when composed in opposite orders. We propose OCBI, an extension of Co-Transformers that models the discrepancy between these two composition orders for localization. A local cross-branch fusion module uses the commutator of RGB-guided cross-attention and forensic self-attention to add a learnable signed correction to feature transfer. Pixelwise adaptive multiscale aggregation then combines spatial contexts before prediction. OCBI uses the Co-Transformers backbone and supervision. We compare OCBI with retrained Co-Transformers using CASIA-only task training, a 50-epoch budget, and final-epoch evaluation. Across five clean test datasets, OCBI achieves a macro PixelF1 of 0.5773, exceeding the baseline by 0.43 percentage points. Under JPEG compression, Gaussian blur, and resizing on three datasets, its average PixelF1 over 21 dataset-transformation conditions exceeds the baseline by 0.95 percentage points.

Introduction

Image manipulation localization identifies the modified pixels in an image. A visually plausible edit may preserve object appearance while altering noise patterns, compression artifacts, or local processing statistics. Conversely, authentic boundaries and textured regions may resemble manipulation artifacts. Effective localization therefore requires combining semantic structure with forensic evidence while distinguishing manipulation traces from ordinary image variation.

Learned forensic representations capture complementary aspects of this evidence. Constrained convolution suppresses image content while learning manipulation-sensitive features [1]. Noiseprint extracts camera-model fingerprints [2], while forensic similarity learning compares traces across image regions [3]. The RGB-and-noise detector of Zhou et al. [4] and CAT-Net [5] combine appearance with complementary forensic cues. TruFor [6] combines RGB features with learned forensic fingerprints and estimates prediction reliability. Co-Transformers [7] models collaboration between semantic structure and microscopic traces. These studies motivate examining how semantic guidance affects the spatial propagation of forensic information.

The order of spatial mixing offers another way to examine this interaction. Within-branch mixing aggregates positions according to relationships among forensic features, whereas cross-branch mixing transfers forensic information using RGB-guided relationships. Aggregating forensic evidence before this transfer need not produce the same message as transferring it first and then applying noise-derived spatial mixing. For manipulation localization, this distinction may matter when semantic boundaries and forensic discontinuities do not coincide: the two compositions may distribute weak traces differently across neighboring positions. We test the hypothesis that their difference can provide a useful correction to cross-branch transfer for pixelwise manipulation localization.

Building on Co-Transformers, we introduce OCBI to make this composition difference explicit. At each hierarchical stage, the cross-branch fusion module, CBFI, constructs two operators on an aligned local spatial grid: AA aggregates forensic values using RGB queries and forensic keys, while NN aggregates them using forensic queries and keys. For forensic values VnV_n, A(NVn)A(NV_n) applies forensic mixing before RGB-guided mixing, whereas N(AVn)N(AV_n) applies the same operators in the reverse order. The commutator C=AN−NAC = AN - NA produces their signed message difference, which is scaled by a learnable coefficient and added to the cross-branch message before updating the RGB features. The adaptive multiscale aggregation module, AMSA, then weights spatial contexts at each pixel to accommodate variation in the extent of manipulated regions. The method uses the Co-Transformers backbone, forensic extractor, prediction head, and supervision.

We evaluate OCBI against retrained Co-Transformers using the same CASIA-only task-training split and 50-epoch budget. On five clean test datasets, OCBI improves PixelF1 on every dataset and raises the macro score by 0.43 percentage points. Under JPEG compression, Gaussian blur, and resizing, OCBI raises the average PixelF1 across the 21 tested dataset–transformation conditions by 0.95 percentage points.

Related Work

Forensic Representations and Complementary Evidence

Early learning-based localization methods focused on processing traces that conventional semantic features can overlook. Bayar and Stamm [1] introduced a constrained convolutional layer for universal manipulation detection. Bappy et al. [8] combined resampling features, recurrent modeling, and an encoder–decoder architecture. Camera-based CNN features [9] and Noiseprint [2] capture camera-related traces. Learned self-consistency [10] and forensic similarity [3] use relationships between image regions to support authenticity analysis.

Multibranch models combine these traces with image appearance. The RGB-and-noise detector of Zhou et al. [4] fuses appearance and noise features to detect manipulated regions. CAT-Net [5] combines RGB information with compression evidence in the DCT domain. MVSS-Net [11] uses multiview feature learning and multiscale supervision, while TruFor [6] combines RGB information with a learned noise-sensitive fingerprint and estimates prediction reliability. Co-Transformers [7] models collaboration between semantic structure and microscopic traces. OCBI extends this architecture by modeling the difference between two compositions of local mixing operators.

Semantic Manipulation Localization [12] studies subtle, meaning-altering edits and introduces a fine-grained benchmark with pixel-level annotations. This preprint focuses on semantic changes, complementing methods based on low-level forensic traces.

Spatial Interaction and Dense Localization

Dense localization requires spatial reasoning in addition to forensic feature extraction. Existing manipulation localization methods obtain spatial evidence primarily from local anomalies, boundary cues, or source–target relationships. ManTra-Net [13] traces manipulation artifacts and detects local anomalies; SPAN [14] organizes patch relationships through a local attention pyramid. MFCN [15] jointly models spliced regions and boundaries, while GSR-Net [16] strengthens boundary cues using generated samples. BusterNet [17] localizes source and target regions in copy-move forgeries.

Related dense prediction tasks also use spatial relationships. Co-saliency models strengthen consistency among related regions through co-attention [18] and group semantics propagation [19]. In video saliency, triplet relationship aggregation [20] extends interaction modeling to multidimensional feature relationships. These methods model relationships among regions, groups, or frames. OCBI models composition-order differences between local semantic and forensic interactions and retains the baseline edge supervision.

Hierarchical Features and Multiscale Aggregation

Hierarchical representations enable information exchange across spatial resolutions. SegFormer [21] provides a hierarchical Transformer encoder. Swin Transformer [22] demonstrates window-based attention and cross-window communication through shifted windows. DANet [23] models spatial and channel dependencies for dense prediction, while DeepLabv3+ [24] combines atrous spatial processing with encoder–decoder design.

Transformer-based representation exchange also appears in multimodal learning. UTC [25] uses a unified Transformer for visual dialog and couples answer ranking and generation through inter-task contrastive learning. In video generation, DiVA [26] injects visual conditioning features through decoupled cross-attention. OCBI uses attention to exchange semantic and forensic features for pixelwise localization. CBFI operates within local windows, and AMSA weights dilated responses before the DANet prediction head.

Datasets and Evaluation Protocols

CASIA [27] and COVERAGE [28] provide benchmarks for image tampering and copy-move forgery, respectively. Columbia [29] and IMD2020 [30] add splicing and more broadly annotated manipulation data. Fantastic Reality [31] further extends the available image-splice data. These benchmarks differ in manipulation types, imaging conditions, and annotation settings. IMDL-BenCo [32] addresses inconsistencies in implementation and evaluation through a modular benchmark and codebase. We use its data-loading and metric infrastructure and report both per-dataset scores and an equally weighted macro-average. Comparisons use a retrained baseline to match the training split and evaluation procedure.

Method

Problem Formulation and Overall Architecture

Given an RGB image II and a binary manipulation mask yy, the model predicts a pixelwise manipulation probability map pp and an auxiliary edge output. The input tensor has batch size BB, three channels, and spatial dimensions H×WH \times W. Edge-supervision targets are generated from the manipulation mask using the baseline mask-construction pipeline.

The base system is Co-Transformers [7]. Its architecture contains a hierarchical edge-supervised Transformer, a noise fusion module, and a Cross-trace Extraction Transformer. We denote the output of the forensic branch by nn and the hierarchical image features by xsx_s. For s=1,2,3,4s = 1, 2, 3, 4, the stage channel widths are 64, 128, 320, and 512, respectively. The encoder uses the MiT-B3 hierarchy of SegFormer [21]. Figure 1 summarizes the Co-Transformers base and the OCBI additions.

Overall OCBI framework

Figure 1. Overall OCBI framework. The Co-Transformers base provides the MiT-B3 image encoder, forensic branch, and DANet prediction head. Four CBFI modules independently update the encoder outputs using aligned forensic features. All four updated features feed the auxiliary edge head; only x4′x'_4 is concatenated with the forensic feature nn before AMSA and mask prediction. AMSA applies pixelwise weighting to depthwise convolution responses at dilation rates 1, 2, and 3, followed by projection and residual update. The lower panel details the window-local signed interaction in CBFI. The auxiliary edge map supplies supervision during training and is not an input to the mask head.

The MiT-B3 encoder first computes the stage features sequentially: xs=Es(xs−1)x_s = E_s(x_{s-1}) for s=1,…,4s = 1,\ldots,4, with x0=Ix_0 = I. After all four features have been extracted, independent CBFI modules produce xs′=CBFIs(xs,n)x'_s = \mathrm{CBFI}_s(x_s,n), preserving each feature’s spatial and channel dimensions. No updated feature xs′x'_s is passed to the next encoder stage. The auxiliary edge head receives all four updated features, whereas the mask path uses only x4′x'_4:

ℓe=Hedge(x1′,x2′,x3′,x4′),z=Concat⁡(x4′,n),ℓm=Hmask(AMSA⁡(z)).(1)\begin{aligned} \ell_e &= H_{\mathrm{edge}}(x'_1,x'_2,x'_3,x'_4),\\ z &= \operatorname{Concat}(x'_4,n),\\ \ell_m &= H_{\mathrm{mask}}(\operatorname{AMSA}(z)). \tag*{(1)} \end{aligned}

Here, HmaskH_{\mathrm{mask}} is the DANet prediction head [23], and zz has 1280 channels. The edge and mask logits, ℓe\ell_e and ℓm\ell_m, are separately resized to the input resolution and passed through sigmoid. The edge output is not supplied to the mask head.

CBFI-1–3 contribute through auxiliary edge supervision during training: the edge loss backpropagates through them into the shared image encoder and forensic branch, whose learned representations also support mask prediction. Their corrected features have no direct forward path to the current mask output. The unmodified features x1x_1, x2x_2, x3x_3 still contribute to x4x_4 through the encoder hierarchy; their CBFI corrections do not. CBFI-4 participates directly in both the edge and mask paths.

Window-Local Cross-Branch Fusion

At each stage, the forensic feature is first aligned to the spatial resolution of the image feature by bilinear interpolation. Independent 1×11 \times1 convolutions then project both representations to 32 channels. Related alignment strategies have been used in cross-modal retrieval [33] and video feature modeling [34]. CBFI partitions the aligned features into nonoverlapping 4×44 \times4 windows, each containing L=16L = 16 positions with 32 channels.

Within a window, QrgbQ_{\mathrm{rgb}} and QnQ_n denote the image and noise queries, while KnK_n and VnV_n denote the noise keys and values. Each matrix has 16 rows and 32 columns. A hat denotes unit ℓ2\ell_2 normalization along the feature dimension. Row-wise softmax produces two operators over the same aligned positions:

As=softmax⁡row(τsQ^rgbKnT),(2)A_s = \operatorname{softmax}_{\mathrm{row}}\left(\tau_s\widehat{Q}_{\mathrm{rgb}}K_n^{\mathsf{T}}\right), \tag*{(2)}
Ns=softmax⁡row(τsQ^nKnT).(3)N_s = \operatorname{softmax}_{\mathrm{row}}\left(\tau_s\widehat{Q}_nK_n^{\mathsf{T}}\right). \tag*{(3)}

Here, AsA_s uses image queries and noise keys, whereas NsN_s uses noise queries and the same noise keys. Both operators have shape 16×1616 \times16. Each stage has an independent learnable log-temperature tst_s, with τs=exp⁡(ts)\tau_s = \exp(t_s) initialized to an effective temperature of 10.

The order-sensitive message is defined as

Cs=AsNs−NsAs,Ms=(As+λsCs)Vn.(4)C_s = A_sN_s - N_sA_s,\qquad M_s = (A_s + \lambda_s C_s)V_n. \tag*{(4)}

Matrix products act on VnV_n from right to left: As(NsVn)A_s(N_sV_n) first aggregates forensic values with NsN_s, then applies the RGB-guided operator AsA_s. Conversely, Ns(AsVn)N_s(A_sV_n) first applies AsA_s, then mixes the resulting message with the same forensic-derived weights NsN_s on the aligned spatial grid. Both compositions use the same AsA_s and NsN_s; attention weights are not recomputed between the two operations. The learnable coefficient λs\lambda_s is initialized to 0.5. Although the surrounding computation uses fp16, attention matrix products and commutator subtraction are computed in float32.

The correction is signed. Let 1\mathbf{1} denote the all-ones vector. Row normalization gives

As1=Ns1=1,Cs1=0,(As+λsCs)1=1.(5)A_s\mathbf{1} = N_s\mathbf{1} = \mathbf{1},\qquad C_s\mathbf{1} = 0,\qquad(A_s + \lambda_s C_s)\mathbf{1} = \mathbf{1}. \tag*{(5)}

Thus, the corrected mixing operator retains unit row sums, but its entries need not be nonnegative. No additional softmax is applied, preserving the signed correction. If the two operators commute, the correction vanishes and the message reduces to AsVnA_sV_n. The zero row sums of the correction also imply that it produces zero output when applied to a spatially constant value field.

After restoring the window messages to the spatial layout, we apply GroupNorm, project them back to the stage channel width, and add a channel-scaled residual:

Δs=γs⊙Wout,s(GroupNorm⁡(Ms)),xs′=xs+Δs.(6)\Delta_s = \gamma_s \odot W_{\mathrm{out},s}\bigl(\operatorname{GroupNorm}(M_s)\bigr),\qquad x'_s = x_s + \Delta_s. \tag*{(6)}

The 1×11 \times1 projection Wout,sW_{\mathrm{out},s} maps 32 channels to the stage channel width. Its weights are initialized from a truncated normal distribution with standard deviation 0.02. The channelwise residual scaling vector γs\gamma_s is initialized to 0.1. The modified stage feature has the same shape as its input and follows the routing in Eq. (1).

Adaptive Multiscale Aggregation

AMSA operates on the 1280-channel fused representation zz. Three depthwise 3×33 \times3 convolutions use dilation rates of 1, 2, and 3 to extract responses over different spatial extents. Cross-scale weighting has also been used in truncation compensation for high-resolution segmentation [35].

ud=DWConv⁡3×3,d(z),d∈{1,2,3}.(7)u_d = \operatorname{DWConv}_{3 \times3,d}(z),\qquad d \in\{1,2,3\}. \tag*{(7)}

A 1×11 \times1 convolution GG produces three scale scores at each pixel. Softmax along the scale dimension yields αd\alpha_d; these weights are shared across channels at each spatial position. Aggregation and residual projection are defined as

αd=softmax⁡d(G(z)),u=∑d=13αd⊙ud.(8)\alpha_d = \operatorname{softmax}_d(G(z)),\qquad u = \sum_{d=1}^{3}\alpha_d \odot u_d. \tag*{(8)}
z′=z+γa⊙Wproj(u).(9)z' = z + \gamma_a \odot W_{\mathrm{proj}}(u). \tag*{(9)}

The dense 1×11 \times1 projection WprojW_{\mathrm{proj}} preserves the 1280-channel width. Its weights use truncated normal initialization with standard deviation 0.02, and the channelwise scale γa\gamma_a is initialized to 0.1. The DANet prediction head receives z′z'.

Supervision and Optimization

Loss and Optimization Strategy

Training uses the baseline edge and mask Dice losses:

L=0.8LDiceedge+0.16LDicemask.(10)\mathcal{L} = 0.8\mathcal{L}_{\mathrm{Dice}}^{\mathrm{edge}} + 0.16\mathcal{L}_{\mathrm{Dice}}^{\mathrm{mask}}. \tag*{(10)}

The edge and manipulation-mask Dice losses flatten all pixels in the local batch and reduce them jointly, using a smoothing constant of 1. The Dice loss is 1−2∑yp+1∑y2+∑p2+11 - \frac{2\sum yp + 1}{\sum y^2 + \sum p^2 + 1}. The single aggregated edge output and the mask output are each passed through sigmoid before loss calculation and then weighted by 0.8 and 0.16, respectively.

All new CBFI and AMSA parameters, including residual scales, mixing coefficients, and log-temperatures, are assigned to a separate AdamW parameter group with five times the learning rate of the shared parameters and zero weight decay. Shared parameters use the same settings in both models.

Computational Complexity

Let the spatial dimensions of stage ss be hs×wsh_s \times w_s, with P=hswsP = h_sw_s, projected width dd, and window length LL. In CBFI, both query–key multiplication and attention–value multiplication are constrained to local windows and cost O(PLd)O(PLd). Composing the two attention operators costs O(PL2)O(PL^2), and attention storage requires O(PL)O(PL). With a fixed window size, these costs grow linearly with the spatial area of the stage. For AMSA, let Q=hwQ = hw and let cc be the channel width. The three depthwise 3×33 \times3 convolutions have a total cost of O(27Qc)O(27Qc), while the dense output projection costs O(Qc2)O(Qc^2). For fixed LL, dd, and cc, both modules scale linearly with the number of spatial positions.

Algorithm 1 OCBI forward computation

Require: Image II; Co-Transformers components

  1. x0←Ix_0 \leftarrow I

  2. for s=1,…,4s = 1,\ldots,4 do

  3. xs←Es(xs−1)x_s \leftarrow E_s(x_{s-1}) ▷ Complete encoding before fusion

  4. end for

  5. n←CTE⁡(NoiseFusion⁡(I))n \leftarrow\operatorname{CTE}(\operatorname{NoiseFusion}(I))

  6. for s=1,…,4s = 1,\ldots,4 do

  7. Resize nn to the spatial resolution of xsx_s by bilinear interpolation

  8. Project both branches to 32 channels and partition into local windows

  9. Apply ℓ2\ell_2 normalization to queries and keys

  10. Compute AsA_s, NsN_s, and Cs=AsNs−NsAsC_s = A_sN_s - N_sA_s in float32

  11. Compute Ms=(As+λsCs)Vn,sM_s = (A_s + \lambda_sC_s)V_{n,s} and restore the spatial layout

  12. xs′←xs+γs⊙Wout,s(GroupNorm⁡(Ms))x'_s \leftarrow x_s + \gamma_s \odot W_{\mathrm{out},s}(\operatorname{GroupNorm}(M_s))

  13. end for

  14. ℓe←Hedge(x1′,x2′,x3′,x4′)\ell_e \leftarrow H_{\mathrm{edge}}(x'_1,x'_2,x'_3,x'_4)

  15. z←Concat⁡(x4′,n)z \leftarrow\operatorname{Concat}(x'_4,n)

  16. Compute dilated responses udu_d and pixelwise softmax weights αd\alpha_d

  17. z′←z+γa⊙Wproj(∑dαd⊙ud)z' \leftarrow z + \gamma_a \odot W_{\mathrm{proj}}\left(\sum_d \alpha_d \odot u_d\right)

  18. ℓm←Hmask(z′)\ell_m \leftarrow H_{\mathrm{mask}}(z')

  19. p←σ(Resize⁡(ℓm,H,W)); e←σ(Resize⁡(ℓe,H,W))p \leftarrow\sigma(\operatorname{Resize}(\ell_m,H,W));\ e \leftarrow\sigma(\operatorname{Resize}(\ell_e,H,W))

  20. return Mask probability map pp and auxiliary edge map $e

Experiments

Datasets and Splits

We train on 4 865 entries from CASIA v2.0 and use 258 entries for validation. Table 1 lists the training and test splits. Counts refer to split-list entries, which may include duplicate images.

SplitList entriesRole
CASIA v2.0 train4 865task training
CASIA v2.0 validation258monitoring
CASIA v1.0920clean test
COVERAGE100clean test
Columbia180clean test
NIST16 subset240clean test
IMD20202 010clean test

Table 1. Training, validation, and test splits

CASIA2 is split by source group with seed 0 into the training and validation lists. Clean testing uses CASIA v1.0 [27], COVERAGE [28], Columbia [29], a fixed 240-entry NIST16 evaluation list, and IMD2020 [30]. Evaluation lists preserve duplicate-content entries.

No identical paths or image hashes were found between the training and test lists. Masks are generally binarized at grayscale values greater than 127, while Columbia uses a threshold greater than 99. NIST16 masks are inverted to mark manipulated regions. COVERAGE masks with mismatched dimensions are aligned by nearest-neighbor interpolation.

Training Settings

All experiments are conducted on four NVIDIA A800-SXM4-80GB GPUs with a global batch size of 112. The image encoder is initialized from the ImageNet-pretrained MiT-B3 model described for SegFormer [21], while the forensic branch uses the fixed noise-sensitive initialization of the baseline extractor. We use AdamW with initial learning rates of 1.37×10−41.37 \times10^{-4} for shared parameters and 6.85×10−46.85 \times10^{-4} for newly introduced OCBI parameters. The learning rate is warmed up for two epochs and then follows a cosine decay schedule. Both models are trained for 50 epochs.

Both models are trained on the CASIA-only split at 512×512512 \times512 resolution with the baseline mask construction and augmentation pipeline. The validation split is used only for monitoring, and all reported clean and post-processing results use the final epoch-50 model. The remaining optimization and execution details are listed in Table 2.

SettingConfiguration
Training data and inputCASIA-only training split; 512×512512 \times512 inputs; baseline mask construction and geometric/photometric augmentation
Objective0.8LDiceedge+0.16LDicemask0.8\mathcal{L}_{\mathrm{Dice}}^{\mathrm{edge}} + 0.16\mathcal{L}_{\mathrm{Dice}}^{\mathrm{mask}}
OptimizerAdamW with β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, and ϵ=10−12\epsilon= 10^{-12}
Learning rate and regularizationShared parameters: 1.37×10−41.37 \times10^{-4}, weight decay 0.05; new OCBI parameters: 6.85×10−46.85 \times10^{-4}, weight decay 0
Schedule2-epoch warm-up followed by cosine decay to 5×10−75 \times10^{-7}
Hardware and batch sizeFour NVIDIA A800-SXM4-80GB GPUs; per-device/global batch size 28/112; no gradient accumulation
Numerical precisionMixed-precision training; CBFI attention computed in float32
InitializationImageNet-pretrained MiT-B3 image encoder; fixed noise-sensitive forensic branch
Training budget50 epochs, 2 200 optimizer updates, random seed 42
Evaluation checkpointValidation for monitoring only; final epoch-50 model for clean and post-processing evaluation

Table 2. Training configuration for retrained Co-Transformers and OCBI

Evaluation Protocol

Evaluation uses 512×512512 \times512 inputs and float32 inference. Manipulation probabilities are binarized at a threshold of 0.5. Pixelwise F1 for a single image is defined as

F1=2TP2TP+FP+FN,(11)F_1 = \frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}, \tag*{(11)}

where TP, FP, and FN denote the numbers of true-positive, false-positive, and false-negative pixels, respectively. Dataset-level PixelF1 is the arithmetic mean of per-image F1 scores. The primary metric is the equally weighted mean of the five test-dataset scores:

Fmacro=15∑d=15F1,d.(12)F_{\mathrm{macro}} = \frac{1}{5}\sum_{d=1}^{5} F_{1,d}. \tag*{(12)}

Each test image is evaluated exactly once, without distributed-sampling padding. Manipulation masks are nonempty for all images in the test lists used.

Main Comparison

Table 3 reports localization on clean test images. OCBI achieves higher PixelF1 on all five test datasets, while the baseline has higher macro AUC. PixelF1 measures mask accuracy at the fixed threshold; AUC measures ranking performance across thresholds.

DatasetCo-Transformers F1OCBI F1Co-Transformers AUCOCBI AUC
CASIA v1.00.74090.74520.95360.9572
COVERAGE0.28530.29940.88680.8826
Columbia0.85910.86130.92880.9313
NIST16 subset0.56050.56070.90720.8928
IMD20200.41920.41970.86410.8724
Macro average0.57300.57730.90810.9073

Table 3. Clean localization at the final checkpoint. Higher is better. Bold indicates the higher score within each metric pair. F1 is computed at threshold 0.5; AUC is averaged over valid image-level rankings. Macro averages weight the five datasets equally.

Figure 2 shows the aggregate changes relative to the retrained baseline: macro PixelF1 increases by 0.43 percentage points, while macro AUC decreases by 0.08 percentage points. On the validation split, OCBI reaches F1 0.6240 and AUC 0.9391, compared with 0.6074 and 0.9298 for the baseline.

Figure 2. Changes in five-dataset macro PixelF1 and AUC relative to the retrained Co-Transformers baseline. Bars show signed changes in the equally weighted macro scores, expressed in percentage points (pp), defined as 100×(OCBI−baseline)100 \times(\mathrm{OCBI} - \mathrm{baseline}). The difference axis is symmetric about zero.

Figure 3 shows the signed PixelF1 differences across the five test datasets. Gains range from 0.02 percentage points on NIST16 to 1.41 percentage points on COVERAGE. AUC increases on CASIA v1.0, Columbia, and IMD2020, and decreases on COVERAGE and NIST16.

Figure 3. Dataset-level changes in clean localization performance. Bars show OCBI minus retrained Co-Transformers PixelF1 in percentage points (pp) on a symmetric zero-centered axis. Labels quantify the change for each dataset; absolute scores are reported in Table 3.

Robustness to Post-Processing

We evaluate JPEG compression at quality levels 90, 70, and 50; Gaussian blur with standard deviations 1 and 2; and resizing factors of 0.75 and 0.5 on CASIA v1.0, COVERAGE, and Columbia. The same transformation pipeline is used for both models, with input resizing enabled. Scaling uses bilinear interpolation.

Across the 21 dataset–transformation conditions, OCBI improves PixelF1 in 14 conditions, raising the equally weighted average from 0.4953 to 0.5048. On Columbia, it improves under all seven transformations, with the largest gain of 9.42 percentage points at JPEG Q50. On CASIA v1.0, it improves under both blur levels and both resizing factors.

Analysis and Discussion

In the CASIA-only comparison with retrained Co-Transformers [7], OCBI improves PixelF1 on all five clean test datasets, increasing the macro score from 0.5730 to 0.5773, a gain of 0.43 percentage points. The largest improvement is on COVERAGE, at 1.41 percentage points (Table 3). The gains across datasets indicate that OCBI improves thresholded localization beyond the training dataset.

We also tested JPEG compression, Gaussian blur, and resizing on three datasets. Table 4 shows improvements in 14 of the 21 conditions, with aggregate PixelF1 rising from 0.4953 to 0.5048, or 0.95 percentage points. Columbia improves under every tested transformation, including a gain of 9.42 percentage points at JPEG Q50. These improvements extend to compressed, blurred, and resized inputs, consistent with its use of local forensic relationships and multiple spatial contexts.

TransformationCASIA v1.0COVERAGEColumbia
BaselineOCBIBaselineOCBIBaselineOCBI
JPEG Q900.70350.70730.19650.19960.81790.8402
JPEG Q700.61260.59920.20550.17070.66920.7448
JPEG Q500.48850.47460.19060.14220.57670.6709
Blur σ=1\sigma= 10.65770.66940.16010.12640.74740.7681
Blur σ=2\sigma= 20.57490.57670.14320.17680.58100.6089
Scale 0.750.67310.67880.16260.16000.79790.8086
Scale 0.50.62710.64000.13380.11770.68210.7203

Table 4. PixelF1 under post-processing. CASIA denotes CASIA v1.0. Baseline denotes retrained Co-Transformers. Bold indicates the higher PixelF1 within each dataset and transformation. Blur values are Gaussian standard deviations; scale values are scaling factors.

Conclusion

We proposed OCBI, a Co-Transformers extension for image manipulation localization. The method models the difference between two compositions of local semantic and forensic mixing operators through a signed commutator correction and combines spatial contexts through pixelwise adaptive multiscale aggregation.

OCBI improves localization over the retrained baseline on the clean test datasets and under a range of image post-processing conditions. The experiments support using composition-order differences and adaptive spatial aggregation to combine semantic and forensic evidence for manipulated-region prediction.

References

  1. [1]Belhassen Bayar, and Matthew C. Stamm. A Deep Learning Approach to Universal Image Manipulation Detection Using a New Convolutional Layer. Proceedings of the ACM Workshop on Information Hiding and Multimedia Security, pp. 5-10, 2016. doi: 10.1145/2909827.2930786.DOI
  2. [2]Davide Cozzolino, and Luisa Verdoliva. Noiseprint: A CNN-Based Camera Model Fingerprint. IEEE Transactions on Information Forensics and Security, vol. 15, pp. 144-159, 2020. doi: 10.1109/TIFS.2019.2919364.arxiv.org/abs/1808.08396
  3. [3]Owen Mayer, and Matthew C. Stamm. Forensic Similarity for Digital Images. IEEE Transactions on Information Forensics and Security, vol. 15, pp. 1331-1346, 2020. doi: 10.1109/TIFS.2019.2924552.DOI
  4. [4]Peng Zhou, Xintong Han, Vlad I. Morariu, and Larry S. Davis. Learning Rich Features for Image Manipulation Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1053-1061, 2018.arxiv.org/abs/1805.04953
  5. [5]Myung-Joon Kwon, In-Jae Yu, Seung-Hun Nam, and Heung-Kyu Lee. CAT-Net: Compression Artifact Tracing Network for Detection and Localization of Image Splicing. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 375-384, 2021.DOI
  6. [6]Fabrizio Guillaro, Davide Cozzolino, Avneesh Sud, Nicholas Dufour, and Luisa Verdoliva. TruFor: Leveraging All-Round Clues for Trustworthy Image Forgery Detection and Localization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20606-20615, 2023.
  7. [7]Jiwei Zhang, Wenbo Feng, Siwei Wang, Feifei Kou, Haoyang Yu, and Shaozhang Niu. Collaborative Transformers with Multi-Level Forensic Attention for Image Manipulation Localization. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 15, pp. 12556-12563, 2026. doi: 10.1609/aaai.v40i15.38250.DOI
  8. [8]Jawadul H. Bappy, Cody Simons, Lakshmanan Nataraj, B. S. Manjunath, and Amit K. Roy-Chowdhury. Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries. IEEE Transactions on Image Processing, vol. 28, no. 7, pp. 3286-3300, 2019. doi: 10.1109/TIP.2019.2895466.DOI
  9. [9]Luca Bondi, Silvia Lameri, David Guera, Paolo Bestagini, Edward J. Delp, and Stefano Tubaro. Tampering Detection and Localization Through Clustering of Camera-Based CNN Features. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 43-52, 2017.DOI
  10. [10]Minyoung Huh, Andrew Liu, Andrew Owens, and Alexei A. Efros. Fighting Fake News: Image Splice Detection via Learned Self-Consistency. Proceedings of the European Conference on Computer Vision, pp. 106-124, 2018. doi: 10.1007/978-3-030-01252-6_7.DOI
  11. [11]Xinru Chen, Chengbo Dong, Jiaqi Ji, Juan Cao, and Xirong Li. Image Manipulation Detection by Multi-View Multi-Scale Supervision. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14185-14193, 2021.arxiv.org/abs/2104.06832
  12. [12]Zhenshan Tan, Chenhan Lu, Yuxiang Huang, Ziwen He, Xiang Zhang, Yuzhe Sha, Xianyi Chen, Tianrun Chen, and Zhangjie Fu. Semantic Manipulation Localization. arXiv preprint arXiv:2604.10132, 2026.arxiv.org/abs/2604.10132
  13. [13]Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan. ManTra-Net: Manipulation Tracing Network for Detection and Localization of Image Forgeries With Anomalous Features. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9543-9552, 2019.DOI
  14. [14]Xuefeng Hu, Zhihan Zhang, Zhenye Jiang, Syomantak Chaudhuri, Zhenheng Yang, and Ram Nevatia. SPAN: Spatial Pyramid Attention Network for Image Manipulation Localization. Proceedings of the European Conference on Computer Vision, pp. 312-328, 2020. doi: 10.1007/978-3-030-58589-1_19.
  15. [15]Ronald Salloum, Yuzhuo Ren, and C.-C. Jay Kuo. Image Splicing Localization Using a Multi-task Fully Convolutional Network (MFCN). Journal of Visual Communication and Image Representation, vol. 51, pp. 201-209, 2018. doi: 10.1016/j.jvcir.2018.01.010.DOI
  16. [16]Peng Zhou, Bor-Chun Chen, Xintong Han, Mahyar Najibi, Abhinav Shrivastava, Ser Nam Lim, and Larry S. Davis. Generate, Segment, and Refine: Towards Generic Manipulation Segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 7, pp. 13058-13065, 2020. doi: 10.1609/aaai.v34i07.7007.DOI
  17. [17]Yue Wu, Wael Abd-Almageed, and Prem Natarajan. BusterNet: Detecting Copy-Move Image Forgery with Source/Target Localization. Proceedings of the European Conference on Computer Vision, pp. 168-184, 2018.DOI
  18. [18]Zhenshan Tan, Xiaodong Gu, and Qingrong Cheng. Co-Saliency Detection With Two-Stage Co-Attention Mining and Individual Calibration. Engineering Applications of Artificial Intelligence, vol. 127, art. no. 107201, 2024. doi: 10.1016/j.engappai.2023.107201.DOI
  19. [19]Zhenshan Tan, Cheng Chen, Keyu Wen, Yuzhuo Qin, and Xiaodong Gu. A Unified Two-Stage Group Semantics Propagation and Contrastive Learning Network for Co-Saliency Detection. Proceedings of the IEEE International Conference on Multimedia and Expo, pp. 1-6, 2022. doi: 10.1109/ICME52920.2022.9859892.DOI
  20. [20]Zhenshan Tan, Cheng Chen, and Xiaodong Gu. Triplet Spatiotemporal Aggregation Network for Video Saliency Detection. Proceedings of the IEEE International Conference on Multimedia and Expo, pp. 2393-2398, 2023. doi: 10.1109/ICME55011.2023.00408.DOI
  21. [21]Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. Advances in Neural Information Processing Systems, vol. 34, pp. 12077-12090, 2021.arxiv.org/abs/2105.15203
  22. [22]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.arxiv.org/abs/2103.14030
  23. [23]Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual Attention Network for Scene Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3146-3154, 2019.arxiv.org/abs/1809.02983
  24. [24]Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Proceedings of the European Conference on Computer Vision, pp. 801-818, 2018.arxiv.org/abs/1802.02611
  25. [25]Cheng Chen, Zhenshan Tan, Qingrong Cheng, Xin Jiang, Qun Liu, Yudong Zhu, and Xiaodong Gu. UTC: A Unified Transformer With Inter-Task Contrastive Learning for Visual Dialog. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18103-18112, 2022.arxiv.org/abs/2205.00423
  26. [26]Cheng Chen, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Wen Wang, Yihao Meng, Hanlin Wang, Yixuan Li, Jiacheng Wei, Zhenshan Tan, Yanhong Zeng, Yujun Shen, Guoshéng Lin, and Fayao Liu. DiVA: Enabling Interactive Digital Life Simulation via Video Models. arXiv preprint arXiv:2609.13830, 2026.arxiv.org/abs/2609.13830
  27. [27]Jing Dong, Wei Wang, and Tieniu Tan. CASIA Image Tampering Detection Evaluation Database. IEEE China Summit and International Conference on Signal and Information Processing, pp. 422-426, 2013. doi: 10.1109/ChinaSIP.2013.6625374.DOI
  28. [28]Bihan Wen, Ye Zhu, Ramanathan Subramanian, Tian-Tsong Ng, Xuanjing Shen, and Stefan Winkler. COVERAGE: A Novel Database for Copy-Move Forgery Detection. IEEE International Conference on Image Processing, pp. 161-165, 2016.DOI
  29. [29]Yu-Feng Hsu, and Shih-Fu Chang. Detecting Image Splicing Using Geometry Invariants and Camera Characteristics Consistency. IEEE International Conference on Multimedia and Expo, pp. 549-552, 2006. doi: 10.1109/ICME.2006.262447.DOI
  30. [30]Adam Novozamsky, Babak Mahdian, and Stanislav Saic. IMD2020: A Large-Scale Annotated Dataset Tailored for Detecting Manipulated Images. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, pp. 71-80, 2020.
  31. [31]Vladimir V. Kniaz, Vladimir A. Knyaz, and Fabio Remondino. The Point Where Reality Meets Fantasy: Mixed Adversarial Generators for Image Splice Detection. Advances in Neural Information Processing Systems, vol. 32, 2019.
  32. [32]Xiaochen Ma, Xuekang Zhu, Lei Su, Bo Du, Zhuohang Jiang, Bingkui Tong, Zeyu Lei, Xinyu Yang, Chi-Man Pun, Jiangcheng Lv, and Jizhe Zhou. IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization. Advances in Neural Information Processing Systems, vol. 37, pp. 134591-134613, 2024. doi: 10.52202/079017-4277.
  33. [33]Qingrong Cheng, Zhenshan Tan, Keyu Wen, Cheng Chen, and Xiaodong Gu. Semantic Pre-Alignment and Ranking Learning With Unified Framework for Cross-Modal Retrieval. IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 7, pp. 6503-6516, 2024. doi: 10.1109/TCSVT.2022.3182549.
  34. [34]Zhenshan Tan, Cheng Chen, Keyu Wen, Qingrong Cheng, and Zhangjie Fu. Bridging Spatiotemporal Feature Gap for Video Salient Object Detection. Knowledge-Based Systems, vol. 304, art. no. 112505, 2024. doi: 10.1016/j.knosys.2024.112505.DOI
  35. [35]Li Zhang, Zhenshan Tan, Guo Zhang, Wen Zhang, and Zhijiang Li. Learn More and Learn Usefully: Truncation Compensation Network for Semantic Segmentation of High-Resolution Remote Sensing Images. IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1-14, 2024. doi: 10.1109/TGRS.2024.3363742.DOI

Paper details

vlnc:W_kaunafrh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents